CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Authors: zongzhen yang
Binhang Qi
Hailong Sun
Wenrui Long
Ruobing Zhao
Xiang Gao
Venue: ICML 2025 poster (2025)
Keywords: Model Merging
Pruning Technique
Task Vectors
Large Language Models
Conflict-Aware Sparsity (CA)
Balanced Sparsity (BS)
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: We propose CABS, a method to address parameter overlap and weight imbalance in model merging, achieving superior results across various tasks and model sizes.

Abstract: Model merging based on task vectors, i.e., the parameter differences between fine-tuned models and a shared base model, provides an efficient way to integrate multiple task-specific models into a multitask model without retraining. Recent works have endeavored to address the conflicts between task vectors, one of the significant challenges faced by model merging, through sparsification; however, two issues significantly limit their performance: *high parameter overlap* and *unbalanced weight distribution*. To address these issues, we propose a simple yet effective framework called **CABS** (Conflict-Aware and Balanced Sparsification), consisting of **C**onflict-**A**ware Sparsification (CA) and **B**alanced **S**parsification (BS). CA reduces parameter overlap by applying masks during sequential pruning, ensuring that each task vector retains distinct, non-overlapping parameters. BS leverages $n$:$m$ pruning to preserve critical weights while maintaining an even distribution across layers. Our comprehensive experiments demonstrate that CABS outperforms state-of-the-art methods across diverse tasks and model sizes.

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Zongzhen Yang" Binhang 1
Hailong Sun 12"
Wenrui Long'
Ruobing Zhao Xiang Gao 1.2
Abstract
especially popular for merging homologous models, those derived by fine-tuning the same base model on different tasks, to create a better-performing model. Many of the best- performing models on the LLM Icaderboard (Beeching et al., 2023) are built by fine-tuning the base models and subse- quently merging them optimize task sosk-specffie performance. Additionally, major enterprises have employed model merg- ing techniques in the development of pre- pre-training models, such as Llama3 (Dubey et al., 2024) and Qwen2 (Yang et al., 2024a; Lu et al., 2024), to enhance generalization capabili- ties and improve performance across a range of tasks.
Model merging based on task vectors, i.e., the parameter differences between fine-tuned models and a shared base model. provides efficient way to integrate multiple task-specific models into a multitask model without retraining. Recent works have endeavored to address the conflicts between task vectors, one of the significant challenges faced by model merging, through sparsification; however, two issues significantly limit their per- formance: high parameter overlap and unbal- anced weight distribution. To address these is- sues, we propose simple yet effective framework called CABS Conffict-Awar and Balanced Spar- sification), consisting of Conflict-Aware Sparsi- fication (CA) and Balanced Sparsification (BS). CA can reduce parameter overlap by applying masks during sequential pruning, ensuring that each task vector retains distinct, non-overlapping parameters. BS leverages n:m pruning to pre- serve critical weights while maintaining an even distribution across layers. Our comprehensive ex- periments demonstrate that CABS outperforms state-of-the-ard methods across diverse tasks and model sizes.
Recent studies have further shown that sparsifying task vec- tors before merging can mitigate parameter conflicts be- tween different task vectors, leading measurable improve- ments in merging performance (Yu et al., 2024; Yadav et al., 2024; Davari & Belilovsky, 2023; He et al., 2024). These conflicts can be categorized into two types: (a) conflicts due to redundant parameters, where parameters that contribute little to performance are unnecessarily retained, and (b) con- flicts due overlapping parameters, where task vectors retain parameters that overlap, potentially with significantly different magnitudes or signs. Such overlaps hinder the effectiveness of the merging process.
Sparsifying task vectors, whether selectively or randomly, aims to reduce conflicts in model merging. However, shares methodological similarities with one-shot pruning, which primarily focuses model compression. Magnitude- based pruning (Liang et al., 2021) is one of the mainstream pruning techniques, which can estimate the importance of weights and selectively preserve the essential weights, thus being rightfully superior to random pruning. Inspired by pruning techniques, recent model merging studies (Yadav et al., 2024) applied magnitude- task vectors with the important weights retained. However, as pointed out by DARE (Yu al., 2024), the results counterintuitive: magnitude-basee pruning underperforms compared to random pruning methods in the context of model merging.
1. Introduction
Model merging has gained increasing attention in the deep learning community, particularly in the context of using task vectors for model merging in large language models (LLMs) (Ilharco et al., 2022; Li et al., 2023; Wortsman et al., 2022; Jin et al., 2022; Matena & Raffel, 2022; Singh & Jaggi, 2020; Akiba et al., 2024). This technique has become
pruning to sparsify
State Key Laboratory of Complex & Critical Software Envi- romment (CCSE). Beihang University, Beijing, China "Hangzhou Innovation Institute of Beihang University. Hangzhou, China e "National University Singapore, Singapore, Singapore Corre- spondence to: Hailong Sun <sunhl buaa.cdu.en..
Our research explores the reasons behind this discrepancy, especially in a setting where magnitude-basee pruning expected to perform well. Addressing these issues is key to developing high-performance merged models. Specifically,
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
CA
Thek vastor A
F with CABS
Mask
Unused Mack
Taok vector RTTal
Remaining Tn
Tn with CARS
BS
BasSing
BS
clementi shia
peoduct
(a) Pruning task voctors with CABS
CARS
TA
Teak ATT
Marged
Magel
with CABS
Tg
Task wector li
CABS
swottap
Magritake Franing
sigs canfiict
Merging task vectors with CABS
Merging Lask vectors with magnitude- pruning (MP)
Merging task vectors with and without CABS
Figure 1. Illustration of the CABS framework, which enhances model merging by addressing parameter overlap and weight imbalance. By integrating Conflict-Awer Sparsification (CA) and Balanced Sparsification (BS), CABS delivers more effective merging compared to standard merging with magnitude-bass pruning (MP), leading to improved model performance.
by analyzing the weight distribution and overlap in task vectors produced by DARE and magnitude-based pruning, we identified two key factors contributing to the underper- formance of magnitude-baseed pruning:
(yellow, TB), resulting in remaining TB. This masking tech- nique minimizes conflicts during the merging process by removing shared weights, allowing for more effective task vector merging and improving the final model performance.
After magnitude-based pruning, the retained weights of different task vectors often exhibit significant overlap. particularly compared to random meth- ods like DARE. The overlap increases conflicts between task vectors during model merging, ultimately degrading the performance of the merged model.
High Parameter Overlap:
Balaneed Sparsification (BS): BS addresses the issue of unbalanced weight distribution by applying n:m pruning, which selectively retains n weights out of every m consec- utive weights based on magnitude (Zhou et al., 2021). As demonstrated in Figure (a), BS first applied to TA, fol- lowed by another application to remaining т (derived by CA). This ensures more uniform distribution of weights across layers, reducing the adverse effects of weight con- centration in certain regions.
Unbalanced Weight Distribution: Magnitude-based prun- ing tends to distribute retained weights unevenly across the model's 's weight matrices, with some regions retaining signif- icantly more weights than others. After pruning, the model merging process applies a uniform scaling coefficient glob- ally across the model to restore performance. However, this process amplifies the existing imbalance, ultimately leading to suboptimal performance.
These strategies are effective and easy to implement. We conducted extensive experiments on decoder-based Mistral- 7B (Jiang et al., 2023) and encoder based RoBERTa- Base (Liu, 2019), using tasks from the LLM leaderboard and the GLUE (Wang al., 2018) dataset. These exper- iments demonstrate that CABS effectively mitigates the issues caused by magnitude- based pruning. On Mistral-7B, CABS achieved an average performance score of 76.50, surpassing the "ideal" virtual model (76.30), which hypo- thetically selects the best performance score for each task. CABS also exceeds the state-of-the-arr (76.02) and fine- RoBERTa-Base. CABS
To address the issues above, we propose a novel framework: Conflict-Aware and Balanced Sparsification (CABS). As illustrated in Figure 1, CABS distinguishes itself from exist- ing methods by introducing two key strategies:
Condlict-Aware (CA) Sparsification: CA addresses con- flicts between task vectors by employing sequential prun- ing approach, ensuring no overlap between the retained weights of different task vectors. As shown in Figure (a), CA first applies pruning to task vector A (blue, TA), and then masks the overlapping weights when pruning task vector B
tuned models (75.86). Similarly, achieved score of 81.70, outperforming the SOTA (79.88) by 1.82 points and the vanilla baseline (79.55) by 2.15 points. These results strongly confirm CABS's s superiority across diverse neural network architectures and various tasks.
2

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Our contributions are as follows:
costs (Liu al., 2019; Frankle & Carbin, 2018; Gale et al., 2019; Zhu & Gupta, 2018). Magnitude pruning retains pa- rameters based on their magnitude, assuming that larger magnitudes correspond to more critical information (Ko- valeva et al., 2021; Puccetti et al., 2022; Yin et al., 2023). However, when applied in the context of model merging, this approach can lead to an unbalanced distribution of retained weights, which exacerbates conflicts during the merging process and results in suboptimal performance.
We identify two key issues encountered by magnitude- based pruning in the context of task vector sparsifi- cation, i.e., high parameter overlap and unbalanced weight distribution.
We propose the CABS framework, consisting of conflict-aware sparsification and balanced sparsifica- tion strategies, which can effectively address the two identified issues.
To address this issue, while n:m pruning (Zhou et al., 2021; Xia et al., 2022) was originally designed for pruning and in- ference acceleration, we discovered that it can be repurposed to control the balance of sparsified task vectors in model merging. Although n:m pruning may not perform as well as unstructured pruning in traditional scenarios, our findings demonstrate that it effectively mitigates weight imbalance, leading to improved performance in merged models.
We conduct comprehensive experiments across a va- riety of tasks and model sizes, showing that CABS outperforms state-of-the-art methods.
We are the first to introduce an "ideal" yet rigorous baseline for evaluation, where CABS outperforms this virtual baseline while all existing methods fall short.
Our code is available at https://gittee com/ zongzhenyang/CAB.
Our proposed CABS method builds upon prior works by introducing CA, a novel approach designed to eliminate parameter overlap during model merging. Additionally, it repurposes the existing n:m pruning technique to mitigate unbalanced weight distribution. Together, CABS effectively enhances the stability and performance of model merging.
2. Related Work
Model merging has become a vital strategy for combining multiple fine-tuned models into a single multitask model without requiring additional training. The simplest merg- ing method is directly averaging the model parameters (Iz- mailov et al., 2018; Wortsman et al., 2022). However, this naive approach often fails to account for task-specific variations, leading to suboptimal performance. A more refined approach, Task Arithmetic (Ilharco et al., 2022), combines task vectors-differenee differences between fine-tuned and pre-trained parameters- using weighted sums controlled by scaling coefficients X. These scaling coefficients allow pre- cise control over the contribution of each task vector during merging, playing critical role in balancing the influence of different tasks. However, it still struggles with parameter redundancy and sign conflicts.
3. Issues in Task Vector Sparsification for Model Merging
In model merging, particularly when using sparse task vec- tors to combine models fine-tuned for different tasks, an unexpected phenomenon has emerged: magnitude- based pruning, which typically retains weights with larger ab- solute values, often underperforms compared to random pruning methods. This result contradicts the intuition that preserving critical knowledge, rather than randomly retain- ing information, within the task vectors should enhance the performance of the merged model. Our investigation into this phenomenon reveals two key issues: the overlap be- tween retained weights and their unbalanced distribution within each task vector. High Parameter Overlap. By comparing the overlap rate between magnitude-based and random pruning methods, our analysis demonstrates that magnitude-basee pruning results in significantly higher parameter overlap between task vectors compared to ran- dom pruning methods. As shown in Figure 2, although the overlap rate of magnitude-pruned task vectors decreases gradually with increasing sparsity, remains significantly higher than that of randomly pruned vectors, especially at higher sparsity levels. This disparity highlights the key issue with magnitude-basee pruning, where high overlap persists even as the model becomes sparser.
To address these issues, Merg-gggging (Yadav et al., 2024) prunes mw-gaigit parameters and resolves sign con- flicts, reducing interference and preserving critical parame- ters during merging. DARE (Yu et al., 2024), a technique inspired by Dropout (Srivastava et al., 2014), reveals the high redundancy in task vectors by randomly dropping 90% of the parameters and rescaling the remaining ones. Us- ing random pruning, DARE has been shown to outperform magnitude-based pruning methods in model merging. How- ever, DARE does not fully explain the reasons for this im- provement. Our analysis suggests that DARE helps mitigate some of the overlap and imbalance. However, the random nature of the approach can potentially sacrifice precision.
Model pruning, particularly magnitude pruning (Zhu& & Gupta, 2018), have been extensively studied for their role in optimizing model performance and reducing computational
This elevated overlap in pognitude-prunee vectors intro- duces conflicts during model merging, as overlapping pa-
3

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
tribution is particularly detrimental after sparsification, it hampers the merged model's ss ability generalize effectively. The performance implications of these unbalanced weights are discussed in detail in 5.4.
1.0
Magnitude Pruning Overiap Rate Random Pruning Overlap Rate
0.9
Overiap Rate
0.7
0.6
To comprehensively analyze this issue, we further examined the weight distributions across different layers of the model, including the query-key-value (QKV) projection and MLP layers, at various sparsity levels (e.g., 50%, 75%, and 90%). These experimental results are provided in Appendix B.2, demonstrating the pervasive nature of the imbalance across different layers and sparsity levels.
0.4
0.3
0.2
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Sparsity Ratio
Figure 2. The trend of overlap rate along the sparsity ratio shows that the overlap rate achieved by magnitude-based pruning de- creases more slowly than that of random pruning, with the gap widening progressively.
4. Methodology
To address the aforementioned issues, we propose the CABS (Conffict-Aware and Balanced Sparsification) framework. As illustrated in Figure 1, CABS resolves parameter con- flicts and ensures balanced weight distribution, thus enhane- ing the performance of the merged model. The framework integrates two core strategies: Conflict-Aware Sparsifica- tion (CA) and Balanced Sparsification (BS), which will be detailed in the following sections. The detailed implementa- tion of CABS is provided in Appendix B.3.
Mavystude Prun mg
Randim Puning
per 12032
zwo wnlphts
200
4.1. Conflict-Aware Sparsification (CA)
Figure 3. Magnitude pruning results in more concentrated and unbalanced distribution of weights compared random pruning.
Sequential Pruning and Mask Application. CA aims to eliminate parameter overlap during model merging by employing a sequential pruning strategy. The process begins with the first vector TA being pruned, producing a mask mask A that marks the positions of the retained weights, This mask is then used to guide the pruning of the second task vector TB. ensuring that there is no overlap between the parameters of TA and TB-
rameters may have significantly different magnitudes or signs between task vectors. For example, if a parameter in task vector TA has a positive value indicating its impor- tance to task A, but the same parameter in TB has a negative value, this sign conflict leads to opposing contributions when merging the two vectors. These confficts are particularly challenging because they are primarily controlled through scaling coefficients x, which serve as key parameters for determining the relative contributions of task vectors dur- ing merging. Adjusting XA for TA can inadvertently affect the contribution of TB, reducing the model's ability to per- form optimally on individual tasks and ultimately leading to suboptimal task-specific performance. The performance implications of these overlapping parameters are explored in detail in 5.4. For details on how the overlap rate is calcu- lated, please refer to Appendix B.I.
For the second task vector TB, the prior mask mask is ap- plied in inverted form to determine the remaining weights that do not overlap with the first pruned task vector. Specifi- cally, the remaining weights of Ty are calculated as:
TB remaining TB @ (1 ma
(1)
This ensures that only the non-overlapping weights in TB are retained in the subsequent pruning process. Afterward, a second round of pruning is performed on тв remaining gener- ating a new sparse mask maskB, which can then be merged with the prior pruned task vector without overlap.
Unbalanced Weight Distribution. By visualizing the weight distribution shown in Figure 3, we identified an- other critical issue: the unbalanced distribution of retained weights caused by seaniitude-basedd pruning. Magnitude pruning often leads to weight concentration in specific re- gions of the model's weights. This imbalance is further exacerbated by the rescaling process, where certain weights gain disproportionate influence over the model's output, often resulting in suboptimal performance. This uneven dis-
Minimizing Overlap When Sparsity Limits are Ex- ceeded. When the sum of the sparsity levels across all task vectors exceeds 1 (e.g., when each vector retains 75% of its parameters), it becomes impossible to achieve zero overlap. In such cases, the objective shifts from eliminating overlap to minimizing it as much as possible. Additional pruning steps are applied selectively to reduce the extent of overlap between task vectors. The detailed implementation
4

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
provided in Appendix B.3.
product of the task vectors 7A and TB is:
M
4.2. Balanced Sparsification (BS)
(BA,
(TA)
Block-Based Pruning Strategy. BS, the weight matrix divided into disjoint blocks of consecutive weights, and within each block, the weights with the largest absolute magnitude retained, while the rest are pruned. This strategy is applied uniformly across all layers to ensure a more even weight distribution within each task vector. Minimizing imbalances prevents performance degradation of the merged models. A more detailed discussion about the differences between Balanced Sparsification (BS) and n:m pruning is presented in Appendix B.4.
(4)
Under the non-overlapping condition (Ma) = each term the summation equals zero:
(A AWW
=0 Vi (5)
Thus, the inner product reduces to:
CABS can be integrated with other model merging tech- niques, where CA and BS can be applied independently or combined with other approaches to further enhance model merging. Additionally, Our analysis shows that CABS intro- duces minimal computational and memory overhead com- pared standard merging methods, ensuring efficiency and scalability in various model merging scenarios. Detailed analyses are provided in Appendix B.5 and Appendix B.6.
(TA.TH): = 0. This guarantees that тл and TB are orthogonal.
(6)
Orthogonality Reduces Interference. Consider the com- bined weight update:
AW ÀATA + ΛBTB,
4.3. Theoretical Analysis
where Aa, An E RR are the scaling coefficients for the task vectors. The squared Frobenius norm of the update is:
This section provides a theoretical analysis of how Conflict- Aware Sparsification (CA) reduces parameter overlap, en- sures orthogonality of task vectors in parameter space, and mitigates interference during model merging.
| = ||ATA|| || + (TAAA(TA,TH)* (8)
When тл and TB are orthogonal (i.e., (TA, TB) F = 0), the cross-term vanishes, and the norm simplifies to:
Sparse and Non-Overlapping Task Vectors. CA employs a sequential pruning strategy to produce sparse task vectors TA -B _ax= with non-overlapping parameters. Their binary masks MA, MB {0, 1) axx satisfy:
| ||AATAll +|ABBBI
This decoupling ensures that adjusting A affects only the contribution of FA, with minimal direct interference TB- As result, task vector contributions can be independently scaled, avoiding interference during model merging.
(MA),(MB):s 0, Vi
On Overlap and Possible Synergy. While overlap often leads to conflicts, there may be cases where overlapping co- ordinates have aligned updates, providing synergistic effects. However, identifying exactly which overlap is "helpful" can be challenging, it requires deep insights into each task's s loss surface. Figure 5 shows that excessive overlap typically impairs performance, whereas minimized overlap yields stable and predictable gains. Hence, CA adopts a simpler strategy of systematically limiting overlap, ensuring robust improvements across various tasks.
The task vectors are defined as:
TA AWA A MA, TB = AW @ MB.
where AW AWn are parameter updates from base model, and @ denotes elementwise multiplication. This ensures that тл and 73 have disjoint non-zero entries. Prior studies (Yu et al., 2024; Yadav et al., 2024) and our ex- perimental results in A.8 confirm that these sparse updates are nearly lossless in retaining task-specific information, as simple rescaling compensates for pruning-induced induced changes.
Conclusion. CA eliminates parameter overlap by projecting task vectors onto nearly lossless orthogonal subspaces. Al- though perfect functional separation cannot be guaranteed in non-linear neural network, the resulting parameter-space orthogonality ensures that cross-terms vanish during model merging, allowing independent control of each task's 's contri- bution through the scaling coefficients (A). By minimizing
Non-Overlap Implies Orthogonality. The Frobenius inner
5

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
interference and enabling precise scaling, CA improves both the stability of optimization and the overall efficiency and performance of the merged model. Thus, CA successfully tackles the central challenges of task-vector sparsification, forming a robust foundation for effective model merging.
sets. Additional details are provided in Appendix B.8.
Evaluation Metrics. Performance was evaluated primarily using accuracy for GLUE tasks. For tasks from the LLM Leaderboard, we used task-specific metrics, such as success rates and accuracy, depending on the default evaluation metric for each task. Detailed explanations of the evaluation metrics and the rationale behind these choices can be found in Appendix B.9.
5. Experiments
We conducted extensive experiments to demonstrate the ef- fectiveness of CABS in enhancing performance and stability in model merging across diverse tasks and model scales.
Baselines. We compared CABS against several baseline methods in two main categories: conflict handling and sparsification strategies. For conflict handling, we eval- uated Task Arithmetic (Ilharco et al., 2022) and TIES- Merging (Yadav et al., 2024). For sparsification, we com- pared CABS with DARE (Yu et al., 2024), Magnitude Prun- ing (Zhu & Gupta, 2018), SparseGPT (Frantar & Alistarh, 2023), and Wanda (Sun et al., 2023).
5.1. Experimental Setup
Datasets and Models for Large Language Model Ex- periments. For large-scale model evaluation, we utilized the LLM Leaderboard benchmark, encompassing six key tasks: A12 Reasoning Challenge (Clark et al., 2018), Hel-
laSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sak- aguchi et al., 2021), and GSM8K (Cobbe et al., 2021). These tasks were assessed using the Eleuther AI Language Model Evaluation Harness (Gao et al., 2024), standard- ized framework designed to test generative language models across various tasks. The models used in our experiments were based on the Mistral-7b-v0. backbone and included fine-tuned variants such as M and WestSeverus-7B-DPO--22
It is worth mentioning that, to assess how far current model merging methods are from the ideal performance expected in this rescarch field, we introduce an "ideal model"' as a strict and meaningful baseline. The ideal model represents a hypo- thetical scenario where the merged model achieves optimal performance for each task. This baseline is constructed by selecting the best-performing individual task-specifie model for each task, providing an upper bound for comparison.
Other Implementation Details. Details on the grid search strategy and exact values of À are provided in Appendices B.10 and B. respectively. Hardware setups, evaluation strategies, and hyperparameter configurations are detailed in Appendix 1.
In addition, conducted new set of experiments using the Open LLM Leaderboard 2 (Fourrier et al., 2024), which includes six tasks: IFEval (Zhou et al., 2023), BBH (Suzgun et al., 2022), MATH (Hendrycks et al., 2021), GPQA (Rein et al., 2023), MUSR (Sprague et al., 2024), and MMLU- PRO (Wang et al., 2024). For these experiments, we em- ployed the qwven-2.5-7-iinstrrect (Yang al., 2024b) model as the backbone and evaluated fine-tuned fq2.5-7b-it and Cuunami-.5--7B-Inttre to assess performance across these additional benchmarks. More details about the datasets and models are provided in Appendix B.7.
5.2. Performance of CABS on Small LMs
We conducted experiments on three task sets evaluate the effectiveness of CABS in merging small-scale models (e.g.. RoBERTa): 1) 2-task set comprising RTE and MRPC, 2) 4-task set comprising RTE, CoLA, MRPC, and SST-2, and 3) 6-task set comprising RTE, CoLA, MRPC, SST-2, RACE, and SQuAD.
Datasets and Models for Small Language Model Experi- ments. For evaluating small-scale models, we utilized the GLUE benchmark, which includes four binary classification tasks: CoLA (Warstadt et al., 2019), SST-2 (Socher et al., 2013), MRPC (Dolan & Brockett, 2005), and RTE (Da- gan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009). To increase task difficulty and diversity, we also included the multiple-choice read- ing comprehension task RACE (Lai et al., 2017) and the task SQuAD (Rajpurkar, 2016). We utilized RoBERTa (Liu, 2019) and GPT-2 (Radford et al., 2019) as pre-trained backbones, with fine-tuned models sourced from HuggingFace. Due to the unavailability of test labels, the original validation sets were repurposed as test
Overall Performance. Table presents the performance for merging four task vectors. Among the baselines, "Task Arithmetic" represents vanilla approach without pruning, while other methods incorporate pruning techniques. For our proposed CABS, the last four rows display results with different orders of sequential pruning (e.g.. "MRSC" in- dicates pruning task vectors of MRPC, RTE, SST-2, and CoLA sequentially). The last column displays the overall performance of the merged model (i.c., the average result across four tasks), with the results in brackets indicating the improvement over Task Arithmetic.
As we can see, random- based pruning methods offer limited performance improvements (e.g.. "TIES- SHEES-Merging-- + DARE"
6

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Table 1. Performance of merging four task vectors (sparsity=0.90,
Table 3. Performance comparison L.M Leaderboard using dif-
ferent methods (sparsity=0755).
Method
CoLA SST-2 RTE MRPC
Avg
Ideal Model
85.04
94.04 79.42 91.18
87.42
Method
|ARC Hella. MMLU
TQA
Wino, GSM8K
AVG
WestSeverus
71.30 88.26
63.92
72.72
83.69
74.27
75.69
Task Arithmetic
76.32 90.83 69.68
81.37
79.55
WildMarcoroni
73.63 88.67
63.96
70.07
84.34
74.48
75.86
Magnitude
82.07
87.04 65.34
79.66
78.53 (-1.02)
Ideal Model
73.63 88.67
63.96
72.72
84,34
74.48
76.30
DARF
76.99 90.14 70.76
81.13
79.76 (+0.21)
Task Arithmetic
72.52 89.25
63.39
74.00
83.46
73.38
76.02(-0.28)
TIES-Merging
82.36
86.93
61.01
79.41
77.43 (-2.12)
+ Magnitude
71.93 89,32
63.18
73.85
84.12
72.22
7-.77(-0.53)
+ DARE
77.66 90.94 69.31
81.62
79.88
(+0.33)
+ DARE
72.64 88.86
63.54
72.82
84.03
73.44
75.89(-0.41)
CABS (CSRM) (Ours)
78.24 92.32 74.37
81.62
81.64 (42.09)
TIES- Merging
71.42 89.17
63.16
73.82
84.74
73.01
75.89(-0.41)
CABS (SCMR) (Ours)
78.52 91.97 73.65
82.60
81.69 (42.14)
+ DARE
71.87 88,95
63.56
72.87
84.61
73.21
75.85(-0.46)
CABS (RCMS) (Ours)
77.76
92.09
75.09
81.62
81.64 (+2.09)
CABS (Ours)
722.92 88.89
63.50
74.41
84.63
74.65
76.50(+0.20)
CABS (MRSC) (Ours)
76.89
92.09
74.73
83.09
81.70 (+2.15)
The detailed results for each configuration are presented in Table 1, Table 9, and Table 10. Additional results for the CoLA and SST-2 tasks can be found in Table 11 (Ap- pendix A.3), and the results for the GPT-2 model are pro- vided in Table 12 (Appendix A.4).
Table 2. Impact of task number model merging performance.
Method
2 tasks
tasks
tasks
Ideal Model
85.30
87.42
83.54
Task Arithmetic
80.15
79.55
66.56
+ Magnitude
80.38 (+0.23)
78.53 (-1.02)
68.28 (+1.72)
+ DARE
80.58 (+0.43)
79.76 (+0.21)
67.23 (+0.67)
5.3. Performance of CABS on Large LMs
Merging
80.20 (+0.05)
77.43 (-2.12)
65,46 (-1.10)
+ DARK
80.65 (+0.50)
79.88 (+0.33)
66.95 (+0.39)
Overall Performance. Table 3 shows the results on large LMs. The last column, "AVG", represents the average performance of merged models across six tasks, with the numbers in parentheses indicating the gap from the "ideal model". Existing methods, whether based on magnitude pruming or random pruming, show similar performance and fail to outperform Task Arithmetic. These baselines remain notably below the "ideal model", highlighting the challenge of surpassing this strict baseline. In contrast, CABS achieves an average score of 76.50, surpassing all baselines and even exceeding the "ideal model".
CABS (Ours)
81.49 (+1.34)
81.70 (+2.15)
69.62 (+3.06)
improves by only 0.33). Signitude-based pruning even degrades performance, consistent with previous findings. CABS achieves the highest average accuracy of 81.70, sur- passing Task Arithmetic by 2.15 and delivering substantial improvements over all other methods. Additionally, the pruning order can affect the performance of the merged model on specific tasks. For instance, the best results for CoLA (78.52) and SST-2 (92.32) are achieved when these tasks are pruned first. However, the variation has minimal impact on overall performance. On average, all pruning orders achieve comparable results (81.64 to 81.70), high- lighting the robustness of CABS in handling variations in pruning order despite task-specific differences.
The result highlights the advantage of model merging in enhancing generalization. While the merged model may not surpass the "ideal model" on every individual task, it often achieves superior performance on specific tasks. For example, in the TruthfulQA task (see column "TQA" in Table 3), the fine-tuned models scored 72.72 and 70.07, while the vanilla baseline reached 74.00, and CABS further increases the score to 74.41. Overall, CABS achieved an average performance of 76.50, exceeding the "ideal model" and significantly outperforming the best baseline score of 76.02. The result underscores the effectiveness of CABS in model merging for large-scale models.
Performance Impact of Number of Tasks. Table 2 high- lights the performance impact of task number on model merging, As the number of tasks increases, overall merging performance declines due to the increasing heterogeneity of tasks. This effect is particularly evident when transitioning from 4 to tasks, as including QA and multiple-choite tasks (RACE and SQuAD) introduces additional complexity.
Notable Achievement on Open LLM Leaderboard 2. As of February 24, 2025, our CABS framework enabled the cre- ation of four merged models (qwen2.55 5-b-cbbs v0.1 through v0.4), which dominated the top four positions among mod- els with 8B parameters or fewer on the Open L.LM Leader- board, As shown in Table 4. this achievement underscores CABS' effectiveness in improving model performance.
Despite these challenges, CABS consistently outperforms baseline methods across all scenarios. Compared to Task Arithmetic, CABS achieves improvements of 1.34, 2.15, and 3.06 for 2-task, 4-task, and 6-task sets, respectively. These results highlight the robustness and scalability of CABS in handling diverse and complex task sets, maintaining signifi- cant gains even as task heterogeneity increases.
Performance Impact of Sparsity Rate. Figure 4 illus-
7

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Table Results of 7B L.Ms the Open LLM Leaderboard
COLA-ST22 AVG ACC RTE MRPC AVG ACC
80.6
COLA-S5TZ AVG ACC (%)
87.0
ATE-MRPC AVG ACC (%)
Models
1FEval BBH MATH GPQA
MUSR MMLU
AVG
87.8-
-00
Tsunami-0.5--7
0.5-7b
74.00
36.14
50.45
7.83
12.21
37.92
36.43
87.7-
DARE
fq2.5-7h
73.99
36.36
46.22
6,94
17.54
37.92
36.50
cabs-v0. ((Ous))
75.06
35.84
47.96
8,50
14.17
37.84
36.56
cabs-v0.2(Oun))
74.18
36.28
49.02
7.61
14.86
37.75
36.61
Pull Overlap
cabs-v0.3(Our))
75,70
35.96
49.32
7.61
15.24
37,80
36,94
07.3-
cabs-v0.4(Ours)
75.83
36.36
48.49
7.72
15.17
37.73
36.88
DARE
Full Overlap
Overlap Rate (%)
Figure 5. Merged model performance decreases as overlap rate increases, underscoring the importance of CA in reducing conflicts.
Average Performance (AVG) (80
76.5
76.0
75.5
Methods
WestSevar us D8-0P-22
WidMorcoren/ Varianti 78
importance of both strategies for achieving optimal results.
74.5
in
Model
Nagritude
Performance Impact of Overlap Rate (CA's Contribu- tion). We examined the impact of varying overlap rates merged model performance validate the importance of CA. The experiment was conducted on two task pairs (RTE-MRPC and CoLA-SST2) at a fixed sparsity level of 0.50, using random pruning for fair comparison. To achieve the target overlap rate ranging from 0% (no overlap, i.e., CA) 100% (full overlap), we first pruned one task vector, then adjusted the pruning of the second vector by control- ling the ratio of retained weights in the overlapping and non-overlapping regions.
74.0
DARE
TIES Merging
73.5
TIES DARE
25
75 Sparse Ratio (%)
90
Figure 4. Performance comparison across sparsity.
trates the performance of different model merging methods across varying sparsity levels. The dashed lines represent the performance of the two pre-trained models, the merged model obtained via Task Arithmetic, and the ideal model. The solid lines indicate the performance of merged models obtained using different methods at varying sparsity levels, highlighting their trends as sparsity increases.
As shown in Figure 5, a lower overlap rate generally leads to better performance. Notably, the 50% overlap rate, which corresponds to the expected overlap rate of DARE, performs worse than the non-overlapping condition achieved by CA. This result highlights the importance of minimizing parame- ter overlap, achieved by CA.
As sparsity increases, all methods experience a performance decline, with the limitations of existing methods becoming particularly pronounced at 90% sparsity. Random pruning- based methods (e.g.. "TA DARE") suffer the most signif- icant degradation due to the loss of critical weights, while magnitude-basse based pruning approaches (c.g., "TA + Magni- tude") also underperform due to imbalanced weight distri- bution. contrast, CABS consistently achieves superior performance across all sparsity levels, demonstrating its ro- bustness and ability to preserve essential information even under high sparsity constraints. More detailed results and discussions for each sparsity level are presented in Table 3, Table 13, and Table 14.
Comparisons with Magnitude-Based and Advanced Pruning Methods (BS's Contribution). Table 5 com- pares BS to magnitude- baasd pruning approaches (including layer- wise and row-wise) and advanced pruning methods (i.e., SparseGPT and WANDA). The results show clear progression in performance balance improves: layer- wise pruning achieves 80.38, row-wise pruning improves to 80.61, and BS further increases to 81.30. This demonstrates that enhancing weight distribution balance can contribute to better model merging performance.
Advanced pruning methods, while effective in traditional pruning tasks, perform similarly to the worst-performing layer-wise magnitude pruning (e.g., 80.34 for SparseGPT), This indicates that such methods are less suitable for task vector sparsification in model merging scenarios. By effec- tively addressing weight distribution imbalances, BS demon- strates its robustness and effectiveness in improving model merging performance.
5.4. Ablation Studies and Discussion
Within the CABS framework, we first analyze the indepen- dent contributions of CA and BS by examining the impact of parameter overlap and unbalanced weight distribution on model merging. Next, we perform ablation studies to isolate the contributions of CA and BS, demonstrating the
8

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
5.5. Limitations and Future Work
Table 5. Comparison of sparsity strategies (sparsity=0.9),
General Limitations. Like other task vector-based meth- ods, our approach is limited to models with identical archi- tectures due to the clement-wise operations used in merging model weights. This constraint restricts the generalization of the framework to models with homogeneous structures. Fur- thermore, reliance on manual adjustment of the parameter remains common challenge, especially for large-language models, which requires trial and error optimize model performance.
Method
RTE
MRPC
AVG
Fine-tuned RTE
79.42
25.98
52.70
Fine-tuned on MRPC
47.29
91.18
69.24
Task Arithmetic
73.29
87.01
80.15
DARE
72.92
88.24
80.58(+0.43)
+ Magnitude (layer-wise)
74.73
86.03
80.38 (+0.23)
Magnitude (row-wisc)
74.06
87.05
80.61 (+0.46)
+ SparseGPT
72.92
87.75
80.34 (+0.19) 19)
WANDA
73.29
87.50
80.40 (+0.25)
Limitations Specific to CABS. CABS introduces two new hyperparameters- the sparse sequence and the n.m ratios to its design, as discussed in Appendix A.7 and A.6. While these hyperparameters were not particu- larly sensitive our experiments, they add complexity and increase computational cost.
BS (Ours)
74.37
88.23
81.30 (+1.08)
Table Ablation study of CABS across different sparsity levels.
Sparsity Level
Method
Overlap Rate
Avg Accuracy
Future Work. Several directions could help overcome these limitations. Expanding model merging techniques to include heterogeneous architectures represents key area for future research. Additionally, improving the performance of merged models multi-task settings-where current approaches do not yet match the performance of original single-task models- priority.
0%
Task Arithmetic
100.00
76.02
TA+magnitude
80.69
76.03
25%
CA Only
66.67
76.29
BS Only
80,97
76.33
CABS
66.67
76.48
71.42
75.77
75%
CA Only
0.00
76.21
BS Only
58.63
76.24
6. Conclusion
CABS
0.00
76.50
In this work, we revealed two issues in model merging: high parameter overlap and unbalanced weight distribution in task vector sparsification. To address these issues, we pro- posed Conflict-Aware and Balanced Sparsification (CABS). CABS effectively reduces overlap and ensures balanced distribution of retained weights, thus enhancing model merg- ing across various tasks and model sizes. Extensive experi- ments on both small- and large-scale models demonstrated CABS's effectiveness improving merged models' perfor- mance and generalization.
Combined Effect of CA and BS. To validate the effec- tiveness of CA and BS, we conducted an ablation study comparing configurations with only CA, only BS, and the full CABS framework. As shown in Table 6, CABS not only benefits from CA and BS independently improving performance, but their combination also minimizes overlap across all sparsity levels and achieves the highest accuracy.
In conclusion, our ablation studies confirm the necessity of reducing overlap rates and maintaining balanced weight distribution for optimal model merging. They validate the crucial roles of CA and BS, showing that combining both strategies achieves the best performance across various tasks and sparsity settings. Furthermore, we performed a series of analyses on varying n : m ratios and provided addi- tional results on the impact of different pruning orders in Appendix A.6 and A.7. These results further demonstrate the robustness the CABS framework. Additionally, we conducted rescaling experiments and found that applying rescaling to magnitude-prunee task vectors can restore per- formance to levels comparable to the original models, simi- lar to what has been observed with DARE's random pruning method. Detailed results of these rescale experiments are included Appendix A.8.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.
Acknowledge
This work supported by National Key Research and Development Program of China under Grant No. 2024YFB3309602, National Natural Science Foundation of China under Grant N0.62472017, and Guangxi Collabora- tive Innovation Center Multi-souree Information Integra- tion and Intelligent Processing.
9

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
References
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Inter- national Conference Machine Learning, pp. 10323- 10337. PMLR, 2023.
Akiba, T., Shing, M., Tang, Y., Sun, Q. and Ha, D. Evo- lutionary optimization of model merging recipes. arXiv preprint 4rXiv:2403.13188 2024.
Gale, T., Elsen, E., and Hooker, S. The state of spansity in deep neural networks. arXiv preprint arXiv: 1902.995724 2019.
Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L.., Giampiccolo, D., Magnini, B., and Szpektor, I. The second pascal recog- nising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 1. Citeseer, 2006.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C.,, Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. framework for few-shot language model evaluation, 07 2024.
Beeching, E., Fourrier, C., Habib, N., Han, S., Lambert, N., Rajani, N., Sanseviero, O., Tunstall, L., and Wolf, T. Open Ilm leaderboard, 2023.
Bentivogli, I.., Clark, P., Dagan, I,, and Giampiccolo, D. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009.
Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, W. B. The third pascal recognizing textual entailment challenge. Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing. pp. 1 1-9, 2007.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1rXiv:1803.05455 2018.
He, Y., Hu, Y., Lin, Y., Zhang, T., and Zhao, H. Localize- and-stitch: Efficient model merging via sparse task arith- metic. arXiv preprint arXiv:2408. 13656, 2024.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166. 2021.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Dagan, I., Glickman, O., and Magnini, B. The pascal recog- nising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset, 2021. URL wwwpthtt///// //ariivaaiivv////////// .ww.wi.or............. wwtgaba/////o.ooooooo........... / 2201..00 03874.
Davari, M. and Belilovsky, E. Model breadcrumbs: Scal- ing multi-task model merging with sparse masks. arXiv preprint ar arXiv:2312.06795, 2023.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2022.
Dolan, B. and Brockett, C. Automatically constructing corpus of sentential paraphrases. In Third international workshop on paraphrasing ( (/WP2005), 2005.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. In 34th Conference on Un- certainty in Artificial Intelligence 2018, UAI 2018, pp. 876-885. Association For Uncertainty in Artificial Intel- ligence (AUAI), 2018.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama herd of models. arXiv preprint arXiv:2407.21783 2024.
Fourrier, C., Habib, N., Lozovskaya, A., Szafer, K., and Wolf, T. Open Ilm leaderboard v2. https: / / h/whhggggggaaaae wttcco///aaaaoo co/spaces/ www.r/or///ee...e...a. haap//pennnnrll
Jang, M., Kim, D., Kwon, D. S., and Davis, E. Kobest: Korean balanced evaluation of significant tasks. In Pro- ceedings of the 29th International Conference Compu- tational Linguistics, pp. 3697-3708, 2022.
leaderboard, 2024. 2024.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:23100 068658 2023.
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Interna- tional Conference on Learning Representations, 2018.
10

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Jin, X., Ren, X., Preotiuc-Pietro, D., and Cheng, P. Data- less knowledge fusion by merging weights of language models. In The Eleventh International Conference Learning Representations, 2022.
Rein, D., Hou, B. L.,, Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof qa benchmark, 2023. URL https://arxee orq/
Kovaleva, O., Kulshreshtha, S., Rogers, A., and Rumshisky, A. Bert busters: Outlier dimensions that disrupt trans- formers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3392-3405, 2021.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Lai, G., Xie, Q. Liu, H., Yang, Y., and Hovy, E. Race: Large-scale reading comprehension dataset from exami- nations. In Proceedings of the 2017 Conference on Em- pirical Methods in Natural Language Processing, pp- 785-794, 2017.
Singh, S. P. and Jaggi, M. Model fusion via optimal trans- port. Advances in Neural Information Processing Systems, 33:22045-22055, 2020.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng. A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1644 2013.
Li, W., Peng. Y., Zhang, M., Ding, L., Hu, H., and Shen, L. Deep model fusion: A survey. arXiv preprint arXiv:2309.156%0 2023.
Liang, T., Glossner, J., Wang, L., Shi, S., and Zhang, X. Pruning and quantization for deep neural network accel- 2021. Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pp. 3214- 3252, 2022.
Sprague, Z., Ye, X., Bostrom, K., Chaudhuri, S., and Durrett, G. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. 2024. URL httts:://artt org/ abs
eration: A survey. Neurocomputing, 461:370-400 36:::377000060030000-0000
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov. R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 1958,, 2014.
Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1rXhv:1907.11692 2019.
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 11695, 2023.
Liu, Z., Sun, M., Zhou, T., Huang, G., and Darrell, T. Re- thinking the value of network pruning. International Conference on Learning Representations, 2019.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery. A., Le, Q. V., Chi, H., Zhou, D., and Wei, J. Challenging big-bench tasks and whether tf can solve them, 2022. URL https: / org/ abs 22.0..22.....2..
Lu, K., Yu, B., Huang, F., Fan, Y., Lin, R., and Zhou, C. Online merging optimizers for boosting rewards and miti- gating tax in alignment. arXiv preprint arXiv:2405.17931, 2024.
Tang, A., Shen, L., Luo, Y., Hu, H., Du, B., and Tao, D. Fu- sionbench: comprehensive benchmark of deep model fusion. arXiv preprint arXiv:2406.00 03280, 2024.
Matena, M. S. and Raffel, C. A. Merging models with fisher- weighted averaging. Advances in Neural Information Processing Systems, 15:17373-17716, 2022.
Wang. A., Singh, A., Michael, J.. Hill, F., Levy, O., and
Bowman, S. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Pro- ceedings of the 2018 EMNLP Workshop BlackboxNI.P. Analyzing and Interpreting Neural Networks for NLP, pp. 353 333.355,10 2018
Puccetti, G., Rogers, A., Drozd, A., and Dell' Orletta, F. Outliers dimensions that disrupt transformers are driven by frequency. In Findings of EMNLP 2022. Association for Computational Linguistics, 2022.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Wang. Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Aruiraj. A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark, 2024. URL 240..01544. https: papps://arxivv www.ii.ogg//////eeeeeeee 2406. 01574.
Rajpurkar, P. Squad: 100,000+ questions for machine com- prehension of text. arXiv preprint arXiv:1606.052. 2016.

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Warstadt, A., Singh, A., and Bowman, R. Neural network acceptability judgments. Transactions the Association for Computational Linguistics, 7:625-641, 2019. doi: 20.1162/taclaa000200
Zhu, M. and Gupta, S. To prune, not prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-opess, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965- 23998. PMLR, 2022.
Xia, M., Zhong, Z., and Chen, D. Structured pruning learns compact and accurate models. In 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, pp. 1513-1528. Association for Computational Linguistics (ACL), 2022.
Yadav, P., Tam, D., Choshen, L.., Raffel, C. A., and Bansal, M. Ties-megging: Resolving interference when merging models. Advances in Neural Information Processing
Systems, 36, 2024.
Yang. A.. Yang, B., Hui. B., Zheng, B., Yu, B., Zhou, C., Li, Ci, Li, C., Liu, D., Huang, F. al. Qwen2 techncal report. arXiv preprint arXiv:2407.1000 10671, 2024a.
Yang, A., Yang. B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang. F., Wei, H., al. Qwen2. 5 technical report. arXiv preprint 2rXiv:2412.15115. 2024b.
Yin, L., Wu, Y., Zhang, Z., Hsieh, C.-Y., Wang, Y., Jia, Y., Li. G., JAISWAL, K.. Pechenizkiy, M., Liang, Y. al. Outlier weighed layerwise sparsity (owl): A missing cret sauce for pruning Ilms to high sparsity. In Forty-first International Conference Machine Learning, 2023.
Yu, L., Yu, B., Yu, H., Huang, F., and Li, Y. Language models super mario: Absorbing abilities from homol- ogous models as free lunch. In Forty-first International Conference on Machine Learning. 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4791-4800 2019.
Zhou, A., Ma, Y., Zhu, J., Liu, J., Zhang, Z., Yuan, K., Sun, W., and Li, H. Learning n:m fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations, 2021.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-folloiinng evalu- ation for large language models, 2023. URL https: //a rxivv /arxiivorrg////eo/ev.......o.llr abs/ 07911.
12

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
A. Additional Experiments Results
A.I. Impact of Lambda Search Grid on Performance
In this section, we analyze the impact of different lambda search grids on the performance of various model merging methods. Our experiments demonstrate the importance of using fine- ine-grained grid intervals to fairly compare the effectiveness of these methods. Table 7 provides results across different grid intervals (0.01, 0.05, and 0.1) for several methods.
For most methods, performance declines as the grid interval increases, underscoring the importance of finer grids to accurately capture optimal lambda values. Coarser grids often miss these values, leading to noticeable drops in performance.
Interestingly, the DARE method maintains stable performance even with coarser grids (0.05 and 0.1). This because the optimal lambda for DARE happens to coincide with a multiple of 0.1, resulting in no significant performance loss with coarser grids. However, when we exclude such coincidental "sweet spot" lambdas, as shown Table 8, DARE also exhibits significant performance drop. This observation reinforces the idea that fine grid intervals are crucial for a fair and thorough evaluation of all methods. A finer grid ensures that all methods have equal opportunity to find the best-perforning lambda, though this must be balanced with computational cost
On the other hand, the CABS method demonstrates robust performance across all grid intervals. It consistently outperforms other methods, and its relative insensitivity to grid coarseness suggests that CABS is more robust and reliable under varying hyperparameter settings. This robustness, combined with its superior performance, makes CABS a strong choice for model merging.
Table 7. Performance comparison across different lambda grid intervals. "TA" means "Task Arithmetic"
Grid Interval
Task
DARE
TA- Magnitude
TIES- DARE
TIES-
CABS
Arithmetic
Merging
0.01
80.15
80.58(+0.43)
80.38(+0.23) 80.65(+0.40) 80.20(+0.05) 81,49(+0,91)
0.05
79.85
80.58(+0.73)
79.90(+0.05) 79.91(+0.06) 79.84(-0.01) 81.19(+1.34) 80.58(+1.15)
0.10
79.43
79.66(40.23) 79.14(-0.29) 79.83(+0.40) 80.82(+1.39)
Table 8, Performance comparison across diffcrent lambda grid intervals excluding one pair sweet spot lambdas in DARE.
Grid Interval
Task Arithmetic
DARE
TA- Magnitude
TIES- DARE
TIES-
CABS
Merging
0.01
80.15
80.58(+0.43)
80.38(+0.23) 80.65(+0.40) 80.20(+0.05)
81.49(+0.91)
0.05
79.85
79.44(-0.41) 79.90(40.05) 79.91(+0.06) 79.84(-0.01) 79.66(+0.23) 79.14(-0.29) 79.83(+0.40) 10.82(+1.39)
81.19(+1.34)
0.10
79.43
78.55(-0.88)
A.2. Detailed results of CABS on Small LMs Merging
This section provides Detailed results for the experiments on small LMs merging in Table 2. Table 9 compares the performance on the RTE-MRPC task pair at 90% sparsity, showing that CABS outperforms all baselines, achieving the highest average score of 8..49 (+1.34). Similarly, Table 10 presents the results of merging six task vectors at the same sparsity level, where CABS also demonstrates superior performance with an average score of 69.62 (+3.06), significantly surpassing other methods. These results highlight the effectiveness of CABS in achieving robust and consistent improvements across multiple tasks, even under high sparsity constraints.
A.3. Additional Experiments other Task Pairs for Small-Scale Experiments
In this section, we present additional results for the CoLA-SST2 task pair to complement the main text's findings on RTE and MRPC. These tasks were selected to further validate the robustness and effectiveness of the proposed CABS method across different types of natural language processing tasks, particularly focusing on tasks involving linguistic acceptability and sentiment analysis.
Table 11 provides a detailed comparison of various model merging methods on the CoLAA and SST2 tasks. The CABS
13

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Table 9. Performance comparison RTE-MRPC task pair using different methods (sparsity=0.9).
Method
RTE
MRPC
AVG
Fine-tuned on RTE
79.42
25.98
52.70
Fine-tuned on MRPC
47.29
91.18
69.24
Task Arithmetic
73.29
87.01
80.15
+ Magnitude
74.73
86.03
80.38(+0.23)
+ DARE
72.92
88.24
80.58(+0.43)
TIES-Mergin
74.37
86.03
80.20(+0.05)
+ DARE
72.56
88.73
80.65(+0.50)
CABS (Ours)
74.01
88.97
81.49(+1.34)
Table 10. Performance comparison of merging six task vectors(spanstty=O.))
METHOD
RTE
MRPC
CoLA
SST2
RACE
SQuAD
AVG
Ideal Model
79.42
91.18
85.04
94.04
71.71
79.82
83.54
Task Arithmetic
67.15
79.41
72.00
85.78
56.21
38.82
66.56
+ Magnitude
72.56
81.13
75.26
87.50
56.99
36.23
68.28
(+1.72)
+ DARE
71.12
65.44
72.48
83.37
59.57
51.39
67.23 (+0.67)
TIES-Merging Merging
68.94
86.01
66.43
83.33
40.11
47.94
65.46 (-1.10)
+ DARE
74.40
83.83
72.92
56.37
60.38
53.80
66.95 (+0.39)
CABS(Ours)
68.95
82.11
73.92
90.83
58.97
42.96
69.62 (+3.06)
method demonstrates superior performance, achieving the highest average scores across both tasks. The normalized accuracy scores (COLA-N and SST2-N) further emphasize the effectiveness of the CABS method, showing consistent improvements over the baseline methods.
The modest gains observed in the CoLA-SST2 experiments, similar to those in the RTE-MRPC pair, can be attributed to the fine-grained lambda grid search. This search process, which fine-tunes the sparsification parameters, improves the overall performance across all methods, thereby reducing the performance gaps. However, CABS still outperforms other methods, indicating its robustness handling task-speeific nuances during model merging.
Table II. Performance comparison on COLA-SST2
task pair using different methoods.(sparstty-00)
METHOD
COLA SST2
AVG
COLA-N
SST2-N
AVG-N
Fine-tuned model on COLA
85.04
50.92
67.98
100.00
54.15
77.08
Fine-tuned model on SST2
68.74
94.04
81.39
80.83
100.00
90.32
Task Arithmetic
81.59
92.89
87.24
95.94
98.78
97.36
Task Arithmetic Magnitude
81.69
93.46
87.58(+0.34)
96.06
99.38
97.72(+0.36)
Task Arithmetic DARE
81.78
93.46
87.62(+0.38)
96.17
99.38
97.78 97.78(+0.42)
TIES TIES-Merging
81.21
93.58 87 17.40(+0.16)
95.5
99.51
97.51(+0.19)
TIES-Merging + DARE
81.78
93.69
87.74(+0.50)
96.17
99.63
97. 97.90(+0.54)
CABS (Ours)
82.55
93.35 87.95(+0.71)
97.07
99.27
98.17(+0.81)
The results from these additional experiments support the conclusions drawn in the main paper, highlighting CABS as robust and effective model merging technique across various tasks and evaluation metrics.
A.4. Additional Experiments on GPT-2-Based Models
we have also extended our experiments to include other architectures, specifically GPT-2-based models (Radford et al., 2019). The results, summarized in Table 12, highlight the performance of CABS and other methods on tasks derived from FusionBench (Tang et al., 2024).
The results demonstrate that CABS outperforms other methods and is the only method to surpass the Ideal Model.
14

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Table 12. Performance comparison GPT-2-based models.
Method
CoLA
MRPC
AVG
Fine-tuned on CoLA
76.80
68.40
72.60
Fine-tuned on MRPC
30.80
80.39
55.60
Ideal Model
76.80
80.39
78.60
Task Arithmetic (Dense)
75.55
77.45
76.50 (-2.10)
TA DARE
76.70
77.21
76.95 (-1.65)
TA Magnitude
76.61
79.66
78.13 (-0.47)
TIES + DARE
77.09
76.72
76.91 (-1.69)
TIES-Merging
76.89
77.94
77.42 (-1.18)
CABS (Ours)
76.41
80.88
78.65 (+0.05)
Table 13. Performance comparison on LLM Leaderhoard using different methods. (sparsity:0.25)
Method
ARC Hella. MMLU TQA
Wino. GSM8K
AVG
WestSeverus 7B- B--DPOvv
71.30
88.26
63.92
72.72
83.69
74.27
75.69
WildMarcoroni- Variant1-7B
73.63
88.67
63.96
70.07 84.34
74.48
75.86
ideal model
73.63
88.67
63.96
72.72
84.34
74.48
76.30
Task Arithmetic
72.52
89.25
63.39
74.00
83.46
73.38
76.02(-0.28)
Magnitude
71.67 89.15
63.42
74.05
84.37
73.53
76.03(-0.27)
+ DARE
72.30 88.77
63.84
72.08
84.40
74.40
75.96(-0.34)
TIES- MEr-ing
72.41 89.34
63.40
74.03
83.64
73.69
76.09(-0.21)
DARE
72.30 88.63
63.76
72.16
85.06
74.37
76.05(-0.25)
TIES-Merging + CABS
72.97
89.20
63,46
74.00
85.16
74.50
76.44(+0.14)
CABS (Ours)
72.75
89.17
63.48
74.08
84.66
74.73
76.48(+0.18)
Although the improvement margin relatively smaller due to the upper-bound constraint imposed by the Ideal Model, CABS consistently proves its effectiveness across tasks.
Interestingly, magnitude pruning shows unexpectedly strong results GPT-2- based models, surpassing DARE by significant margin. This contrasts with previous experiments on other architectures, suggesting potential architecture- specific behavior in existing pruning methods. Nevertheless, CABS maintains its advantages across different architectures, showcasing its robustness and adaptability. These findings underscore the versatility of CABS and its potential for diverse architectures.
A.5. Detailed results of CABS on Large LMs Merging
This section provides detailed results for the experiments on large LMs merging under different sparsity levels. Table 13 presents the results at 25% sparsity. CABS achieves the highest average score of 76.48 (+0.18), outperforming all baselines and closely approaching the ideal model's performance. The results demonstrate the robustness of CABS preserving task-relevant information and mitigating performance degradation, even under moderate sparsity constraints.
Table 14 shows the results at much higher sparsity level of 90%. Despite the challenging conditions, CABS maintains competitive performance with an average score of 76.10 (-0.20), surpassing other methods, including Task Arithmetic, TA-dare, and mies-mag These results highlight the effectiveness of CABS in achieving stable and high-quality model merging, even at extreme sparsity levels.
A.6. Effect of Different n:m Ratios at Fixed Sparsity Levels
This section examines how different n.m ratios impact the performance of the merged model while keeping the overall sparsity fixed at 75%. The results in Table 15 indicate that while higher n.m ratios (e.g., 64:256) tend to show slight improvements, the overall impact varying n:m ratios remains relatively subtle, suggesting that model performance not highly sensitive to these values.
15

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Table 14. Performance comparison
LLM Leaderboard using different methods. (sparsity=0.90)
METHOD
ARC
Hella.
MMLU
TQA
Wino.
GSM8K
AVG
Mistral-7B-v0.1
59.98
83.31
64.16
42.15
78.37
37.83
60.97
LestSeverus-7B-DPO.
71.30
88.26
63.92
72.72
83.69
74.27
75.69
WildMarcoroni-- -arinnit--aiaai----
73.63
88.67
63.96
70.07
84.34
74.48
75.86
Ideal Model
73.63
88.67
63.96
72.72
84.34
74.48
76.30
Task Arithmetic (Dense)
72.52
89.25
63.39
74.00
83.46
73.38
76.02
TA-dare
70.73
87.18
60.15
70.69
82.64
67.93
73.22 (-3.08)
TA- -A-magnitude
71.47
89.01
62.74
73.49
83.48
72.43
75.44 (-0.86)
Ties-dare
70.31
87.12
60.38
70.40
83.66
67.93
73.30 (-3.00)
Ties-magnitude
71.57
88.93
62.71
73.49
84.08
73,26
75.67 (-0.63)
CABS (Ours)
71.87
89.01
62.95
74.04
84.65
74.06
76.10 (-0.20)
Table 15. Impact of different n:m ratios CABS.6paarsiy=0075
METHOD
ARC
Hella.
MMLU
TQA
Wino.
GSM8K
AVG
WestSeverus-7B-DPP
71.30
88.26
63.92
72.72
83.69
74.27
75.69
WildMarcoroni- Variant
73.63
88.67
63.96
70.07
84.34
74.48
75.86
Ideal Model
73.63
88.67
63.96
72.72
84.34
74.48
76.30
Task Arithmetic(Dense)
72.52
89.25
63.39
74.00
83.46
73.38
76.02(-0.28)
CABS(16:64)
72.44
89.08
63.11
73.38
84.79
75.11
76.32(+0.02)
CABS(32:128)
72.92
88.89
63.50
74.41
84.63
74.65
76.50(+0.20)
CABS(64:256)
72.38
89.29
63.15
73.47
85.40
74.65
76.39(+0.09)
A.7. Additional Experiments on Performance Impact Sparsification Sequence
We analyze how different sparse sequences, referring to the order in which source models (e.g., "wild" and "west") undergo sparsification during the merging process, affect the merged model's performance. In this context, "wild-first" and "west-first" indicate which model is sparsified first. Our findings, summarized in Table 16, suggest that while the order of sparsification has some impact, the effect remains relatively small.
A.8. Rescale Experiments
In previous research, TIES utilized magnitude pruning reduce conflicts during task vector merging but did not include rescale step. Subsequent work on DARE introduced two-step process: random pruning followed by rescaling with factor of I-pp where is the sparsity rate. DARE demonstrated that random pruning, when combined with rescaling, could restore performance to levels comparable to the original fine-tuned models. However, DARE did not explore the effect of rescaling on magnitude-prunel task vectors.
In our experiments, we evaluated the impact of rescaling on both magnitude-bast and random pruning methods across different sparsity levels. As shown in Figure 6, rescaling allows magnitude- prruned task vectors to recover performance similar to that achieved by DARE, suggesting that rescaling is crucial step for maintaining model performance post-pruning
These findings confirm that, with appropriate rescaling, both magnitude- bbased and random pruning methods can achieve near-original performance. This insight complements the primary contributions of our work by showing that magnitude pruning, which traditionally underperformed compared to random pruning in TIES, can be equally effective when combined with rescaling. Although this experiment supports the robustness of magnitude pruning under rescale conditions, it is not the main focus of our study and therefore detailed here in the appendix.
16

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Table 16. Performance comparison across different sparse sequences L.M Leaderboard tasks, tasks (sparsity=0.75))
METHOD
ARC
Hella.
MMLU
TQA
Wino. GSM8K
AVG
WestSeverus-7B-DPOOw
71.30
88.26
63,92
72.72
83.69
74.27
75.69
WildMarcoroni- Variant1-77
73.63
88.67
63,96
70.07
84.34
74.48
75.86
Ideal Model
73.63
88.67
63.96
72.72
84.34
74.48
76.30
Task )
72.52
89.25
63.39
74.00
83.46
73.38
76.02(-0.28)
CABS(16664)-wildfinns
72.30
88.87
63.47
74.27
84.77
74.12
76.3(+0.0)
CABS(16:64)-westfinng
72.44
89.08
63.11
73.38
84.79
75.11
76.32(+0.02)
CABS(22:129-wild-ins
72.92
88.89
63.50
74.41
84.63
74.65
76.50(+0.20)
CABS(G2:123)west-nnn
72.58
89.19
63.19
74.22
85.16
74.15
76.42(+0.12)
CABS(64:25)-wiiddfiist
72.87
89.02
63,43
74.61
84.37
73.92
76.37(+0.07)
CABS(64:250-wesstfist
72.38
89.29
63.15
73.47
85.40
74.65
76.39(+0.09)
0.02
6.75
135
5.72
5.00
2.43
0.22
01c
E02
3.65
0.42
Spuos Rud2
Figure Impact rescaling on different pruning methods across various sparsity levels. Performance evaluated on RTE MRPC tasks using RoBERTa. The horizontal axis represents the sparsity ratio, while the vertical axis indicates the performance of the task vectors after rescaling.
A.9. Impact of Lambda on Performance
Figure 7 provides the average performance as a function It can be observed that within certain range, the performance relatively insensitive to variations A. This result corresponds the performance of the CABS framework on the RTE-MRPC task. For visualization purposes, the same À values used across the tasks rather than the task-specific À values reported in the paper. The À values range from I to 3, with step size of 0.01, resulting in total of 200 samples.
A.10. Multilingual Applicability of CABS
While primary experiments focused on English tasks maintain comparability with prior work, extended evaluation to include two Korean language tasks, kobest copa and kobest_book (Jang et al., 2022), to investigate the multilingual applicability of our method. These additional experiments provide insight into the performance of CABS across diverse linguistic contexts. The results are summarized in Table 17.
For these experiments, we reused the merging configuration from our previous 7B experiments to ensure consistency across evaluations and to reduce computational overhead during this phase. CABS achieves an average score of 75.41, closely matching the ideal model's performance of 75.59 difference of -0.18). In comparison, the best alternative, Task Arithmetic further behind. These results confirm that CABS delivers
+ DARE, achieves 74.63 (-0.96), with other methods falling competitive performance across both English and non-English tasks.
Additionally, these findings underscore the robustness of CABS in maintaining performance across multilingual benchmarks, highlighting its potential applicability to a wide range of languages and tasks. While the absolute improvement margins may vary due to upper-bound bound constraints imposed by the ideal model, CABS consistently demonstrates effectiveness and adaptability across diverse settings.
17

---

CABS: Conflict-Aaare and Balanced Sparsification for Enhancing Model Merging
383
0.7
3.93
1.55
2.50
2.20
2.5
2.75
2200
Avaluns
Figure 7. Average performance vs.lamhda
Table 17. Performance comparison multilingual tasks, including Korean language benchmarks.
Model
ARC Hella. MMLU TQA
Wino. GSM8K Kcopa Kboolq
Avg
Mistral-7B-v0.1
59.98
83.31
64.16
42.15
78.37
37.83
59.00
62.61
60.93
WestSeverus
71.30
88.26
63.92
72.72
83.69
74.27
63.30
81.91
74.92
WildMarcoroni
73.63
88.67
63.96
70.07
84.34
74.48
64.80
82.08
75.25
Ideal Model
73.63
88.67
63.96
72.72
84.34
74.48
64.80
82.08
75.59
(Dense)
72.52
89.25
63.39
74.00
83.46
73.38
65.60
72.58
74.27 (-1.32)
TA Magnitude
71.93
89.32
63.18
73.85
84.12
72.22
64.70
72.86
74.02 (-1.57)
TA DARE
72.64
88.86
64.53
72.82
84.03
73.44
61.40
79.34
74.63 (-0.96)
TIES-Mere Merging
71.42
89.17
63.16
73.82
84.74
73.01
64.80
73.08
74.15 (-1.44)
TIES + DARE
71.87
88.95
63.56
72.87
84.61
73.21
61.40
79.63
74.51 (-1.08)
CABS (Ours)
72.92
88.89
63.50
74.41
84.63
74.65
65.10
79.20
75.41 (-0.18)
A.11. Model soups experimental results
Merging Checkpoints of the Same Task for Better Robustness. As shown in Table 18, merging checkpoints fine-tuned on same task improves performance, with CABS achieving the highest SST-2 accuracy of 0.9472, surpassing other methods by notable margin (+ (+1.4 These two checkpoints were fine-tuned for one epoch using Adam and AdamW optimizers. respectively, with learning rate of 3 10-5 The original training set was split 9:1 into set, with the validation used the test set. This result demonstrates the effectiveness of CABS in maintaining robustness and resolving conflicts during checkpoint merging.
training and validation
A.12. Effect of Learning Rate Overlap Degree
We conducted additional experiments study the effect of learning rate the parameter overlap degree under magnitude pruning with 90% sparsity. Specifically, we fine-tuned the model using learning rates from the set
Table 18. Model soups experimental setup. CABS improves performance when merging checkpoints the same tasks.
Method
SST-2 Accuracy
Fine-tuned modell
0,9323
Fine-tuned model2
0,9289
Task Arithmetic
0.9381 (+0.58)
+ +Magnitude
0.9381 (+0.58)
+DARE
0.9346 (+0.23)
TIES-ME MIE--Mrrging
0.9404 (+0.81)
+DARE
0.9358 (+0.35)
CABS(OurS)
0.9472 (+1.49)
18

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
Overiap Rate Learning
3.50
Overlap
9.45
Overlep Rate
9.25
9.20
0.35
9.20
16:05
3o-08
36-05
36:05
Learning
Figure 8. The relationship between learning rate and parameter overlap degree under magnitude pruning with 90% sparsity. Lower learning rates result in higher overlap.
{1e-6, 3e-6, 5e-6, 1e-5, 5e-5} with both Adam and AdamW optimizers. After pruning, the parameter overlap de- gree calculated analyze the relationship between learning rate and parameter overlap.
Our observations, illustrated in Figure 8, show that lower learning rates lead higher overlap degree among parameters. This indicates that fine-tuning at lower learning rates tends to preserve shared information across tasks, even under extreme sparsity conditions. Conversely, higher learning rates result in less overlap, likely due to more significant parameter updates during optimization.
B. Detailed Experimental Settings
B.1. Overlap Rate Calculation
The overlap rate between two task vectors is metric used to quantify the extent to which the same parameters retained after pruning. This metric is particularly useful in understanding how pruning strategies impact the sharing of model parameters across different tasks, which can lead to conflicts during model merging.
The overlap rate calculated as follows: Given two task vectors тл and TB. the overlap rate defined as the ratio of the number of shared non-zero parameters to the total number of non-zero parameters the first task vector TA. Mathematicallly this can be expressed as:
ITA NTB
Overlap Rate
|TA]
where |A O TB represents the count of non-zero parameters that are common to both vectors тA and TB, and |Tal denotes total count of non-zero parameters vector TA- This calculation shows the extent overlap between two task vectors. higher overlap rate means more shared parameters, increasing the potential for conflicts during model merging.
B.2. Weight Distribution Analysis Across Layers and Sparsity Ratios
This section provides a comprehensive analysis of the heatmaps illustrating weight distributions across different lay- ers of the model and various sparsity ratios. Figures 9-11 show the weight distribution for four representative layers: self_attn. k-proj. weight (layer 6), self_attn q-projj weight (layer 12), self_attn. v-proj weight (layer 24), and mlp. up-pro weight (layer 18) sparsity ratios of 25%, 50%, 75%, and 90%.
These heatmaps demonstrate how increasing sparsity causes magnitude- pruning to concentrate weights in localized regions of the parameter space. As the sparsity level increases, this clustering becomes more pronounced, especially at 75% and 90% sparsity levels, leading to potential imbalances that can degrade model performance.
The recurring pattern across all layers further highlights the significance of strategies like Balanced Sparsification (BS),
19

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
which aim distribute weights more evenly across the model. By ensuring more uniform distribution of the retained weights, BS helps maintain model stability and performance after sparsification.
Sparsity Ratio: 25%
Sparsity Ratio: 50%
Sparsity Ratio: 75%
Sparsity Ratio: 90%
400
45
100
40
430
Number Nor Zero Weights
160
Number Nor Weights
2t
Block
Block
Number Nor
Block
Block
Number
770
4n
Black
Columns
Figure Heatmaps of weight distribution mf attn. aatttk.proo.... weight across different sparsity ratios (25%, 50%, 75%, and 90%).
B.3. Algorithm of CABS
In this section, present the detailed steps for both the CABS sparsity algorithm and the Low-Overlap Sparsity approach. Algorithm ?? outlines the process behind CABS, Algorithm 2 provide the detailed algorithm for Low-Overlap Sparsity designed to minimize direct conflicts during the model merging process. The algorithm sequentially applies sparsification to task vectors, ensuring that the non-overlapping portions of the task vectors are prioritized, thereby reducing overlap and conflict between different task vectors the final merged model.
Algorithm CABS
Input: Task vectors 7A, TB, base model W base, sparsity level n, m, scaling coefficients AA Ag Output: Parameters the merged model Winal
1: Apply n:m pruning to тa and compute mask,4
include # core step of CA
7B remaining = TB (1 m eliminate overlap with TA
3: Apply n:m pruning TB remaining compute mask
# include BS
4: Merge the pruned vectors with the base model:
W iimal
huse + Xa x mask ④ TA + AB x mask @B
5: Return W final
B.4. Comparison of n:m pruning and BS
Although both n:m pruning and BS employ the same operation--selctiig the top values out of m consecutive weights based on magnitude- their goals and use cases differ:
Goal: The primary goal of n:m pruning achieve model compression and acceleration by reducing computational and
20

---

CABS: Conflict-Aaare and Balanced Sparsification for Enhancing Model Merging
Sparsity Ratio: 25%
Sparsity Ratio: 50%
Sparsity Ratio: 75%
Sparsity Ratio: 90%
16
140
400
Block Weights
Number of Zero Weights
112
Number Nor- Monn-Zero Weight'
-70
400
20
Block
Bleck
100
Block
Block
Black
Black
Figure Heatmaps weight distribution model.layerr.122ssell wer..t/ww....iiiii.... across different sparsity ratios (25%, 50%, 75%.
and 90%).
Sparsity Ratio: 25%
Sparsity Ratio: 50%
Sparsity Ratio: 75%
Sparsity Ratio: 90%
300
730
Number of Nor- Zero Weights
Number of Nor- Zero Weights
Non
30e
Block
Block
204
Columns
Figure Heatmaps weight distribution in lader.layers.115 wtt.aas.....l......... w8..pp....pppp.... wnlp.up.proj....m 90%).
weijrt weight across different sparsity ratios (25%, 50%, 75%, and
21

---

CABS: Conflict-waaee and Balanced Sparsification for Enhancing Model Merging
Sparsity Ratio: 25%
Sparsity Ratio: 50%
Sparsity Ratio: 75%
Sparsity Ratio: 90%
326
146
140
46
Number of Weightz
of Zero Weightts
Number of Non-Zerr Zero Weights
10
Block Rowes
289
Block
Block
230
206
238
Number
226
000
666
730
12
Block Columns
Block Columns
Block Columns
Block Columns
Figure 12. Heatmaps of weight distribution in moleel.ayer.22..fffftm....y weight across different sparsity ratios (25%, 50%, 75%, and 90%).
Algorithm 2 CABS Implementation overlap rate
Input: Task vectors TA, TB, base model W Wase, sparsity level n m, scaling coefficients d Ab
Output: Merged model parameters Winnal final
1: Apply n.m pruning 7л and compute mask,1
include BS
2: Compute initial_mask,
1 mask retaining non-overlapping regions of TB
3: If initial maskk retains less than n + m of weights, update maskg by including additional weights from the overlapping
region m @ TB until the target sparsity n + m is reached
4: Merge the pruned vectors with the base model:
W fimal Whsse + Xa X mask @ TA + AB X mask @ TB
5: Return W Vinal
memory costs, In contrast, BS is designed maintain balanced distribution of task vectors while minimizing conflicts between them during merging, not to merely discard unimportant weights.
Result: n.m pruning is typically used for structured pruning in models, aiming to reduce inference time and memory usage. On the other hand, BS is applied specifically to task vectors. After the task vectors are merged with a base model, the resulting model remains dense, meaning that the practical computation and memory savings are not realized, but the model gains improved capacity.
pruning often configurations like 2:4 4:8, where the sparsity level is generally around 50%. In contrast, the sparsification of task vectors under BS can involve much higher sparsity levels, as can be seen in Table 15 (Appendix A.6), with configurations such as 64:256 at 75% sparsity.
Sparsity Ratios:
Typically, n.m pruning yields lower performance compared to magnitude pruning compression tasks, as the more strict uniform distribution of sparsity across blocks (c.g., every 4 weights) tends to hurt performance. However, in model merging, n:m sparsity can outperform row-wise or layer-wise magnitude pruning due to more balanced distribution.
Effectiieness:
22

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
B.5. Computational Overhead Analysis
This section provides a detailed analysis of the computational complexity of the CABS framework, focusing on its core components: Balanced Sparsification (BS) and Conflict-Aware (CA) pruning strategies, as well as the scalability and parallelization potential.
Balanced Sparsification (BS) operates efficiently by dividing each layer's parameters into small, fixed-size blocks of m parameters. Within each block, the top n weights are selected based on magnitude, requiring a localized sorting operation with complexity (m log m) per block. For a layer with N/m blocks, the total complexity per task vector is ( log m). significantly more efficient than global magnitude pruning with a complexity of O(N log N). When merging task vectors, the total complexity becomes OkkN log m). making BS highly scalable for large-scale model merging.
Conalict-Aware Sparsification (CA) introduces minimal computational overhead by sequentially applying a mask inversion and element- product to ensure non-overlapping pruned regions across task vectors. These operations align with standard sparsification frameworks and maintain the same order of complexity, adding negligible cost compared to traditional methods. Combined with BS, the CA strategy ensures robust conflict resolution while maintaining computational efficiency.
Scalability and Parallclization. The complexity of CABS scales linearly with the number of task vectors (k)), ensuring O(KN log m) efficiency for BS. Additionally, the block-based pruning operations in BS and the sequential processing in CA are inherently parallelizable, allowing task vector processing to occur independently across layers blocks. This parallelization potential leverages modern hardware architectures, enabling efficient execution even for large-scale models. Without full parallelization, CABS still remains computationally efficient for real-world applications.
Comparison and Conclusion. Compared to traditional global magnitude pruning (O log N)), the block-based sorting in BS (O(N m) provides substantial computational savings. CA introduces negligible overhead, ensuring efficient and robust merging across multiple task vectors. Overall, with efficient scaling and inherent parallelization, CABS maintains a low computational overhead while effectively resolving task conflicts and ensuring balanced weight distribution, making suitable for both small- and large-scale models.
B.6. Memory Overhead Analysis
This section analyzes the memory overhead of CABS during the merging process and compares it to existing methods such as DARE and TIES-Merging:
Memory Overhead of CABS. During the merging process, CABS requires memory for storing the model parameters and two additional boolcan-like like masks: one to track weight usage and another to record pruning results. For model with N parameters, the memory overhead of these masks is O . N 0.125 bytes), which is negligible compared to the memory required for storing the model parameters themselves (O(N 2 bytes)). As a result, the peak memory usage of CABS during the merging phase is comparable to other methods and remains efficient for large-scale models.
Comparison with Other Methods. DARE requires loading both source models into memory during the merging process. With lazy loading, the peak memory usage is O(2 N 2 bytes), where N is the number of parameters in a model. Merginerging, on the other hand, requires memory for all task vectors simultaneously during its election phase, resulting in O(k- N 2 bytes), where k is the number of task vectors. However, with lazy loading, TIES-Merging can reduce its memory usage to ( N 2 bytes), matching that of DARE. CABS achieves a similar peak memory usage as DARE and -ergin with lazy loading, as the additional memory required for the two boolean masks is negligible compared to the memory needed for model parameters. This makes CABS as memory-efficient as other existing methods while offering additional robustness and performance benefits.
Conclusion. CABS introduces minimal additional memory overhead, as the boolean masks required for Balanced Sparsi- fication are lightweight compared to the model parameters. Furthermore, the merging process is typically performed on CPUs, where memory constraints are less critical than on GPUs. In practice, no memory bottlenecks have been observed during experiments, confirming that CABS is memory-efficient and scalable for merging large-scale models.
B.7. Details of Datasets and Models for LLMs
Datasets: Our evaluation framework comprises two benchmark suites that collectively assess a broad spectrum of language understanding, reasoning, and groblem-solving capabilities.
23

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
(1) Open LLM Leaderboard Benchmark:
A 12 Reasoning Challenge: A of grade-school science questions designed to test fundamental reasoning skills.
HellaSwag: commonsense inference task that poses challenges for state-of-ther of-the-art models while remaining straight- forward for humans (with human accuracy around 95%).
MMLU; A multitask evaluation covering 57 subjects elementary mathematics, US history, computer science, and law-to to gauge broad-domain knowledge.
TruthfulQA: A benchmark that measures model'ss tendency avoid reproducing widely circulated falsehoods.
Winogrande: An adversarial task based on Winograd schemas, which tests nuanced commonsense reasoning.
GSM8K: A collection of grade-school math word problems that require multi-step mathematical reasoning.
(2) Open LLM Leaderboard 2 Benchmark:
IFEval: Designed evaluate inference capabilities across complex, varied scenarios.
BBH: subset of BIG-Bench hard tasks that challenges models with problems requiring deep reasoning.
MATH: A dataset comprising challenging mathematical problems that demand multi-step, non-trivial problem solving.
GPQA: A general-purpose
benchmark that spans a diverse range of topics.
MUSR: Focused assessing multi-step reasoning in intricate contexts.
MMLU-PRO: An advanced variant of MMLU that emphasizes professional and specialized domain knowledge.
Models: We evaluated two families of models corresponding to the two benchmark suites.
(1) Open LLM Leaderboard Models: These models are built on the Mistral 1-bb-v0. 1 backbone and include the following fine-tuned variants:
Wi Milchaarcorre----riiatt1---11
West.Severuu-7B-DDO-W
(2) Open Leaderboard 2 Models: For the new benchmark suite, we use Qwen/Qwen2. 5-BB----I-ni- "IB-Instuuct" as the base model, and include the following fine-tuned variants:
Tsunami- wtt.:hummii----11111 -05---0--n-t-ureee
These models were selected for their robust performance across the diverse tasks and their proven utility in prior research.
ize_fa 1se
rnct
24

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
B.8. Details of Datasets and Models for Small LMs
Tasks The GLUE benchmark includes a variety of tasks designed to evaluate different aspects of natural language under- standing. For experiments, selected the following four tasks:
CoLA (Corpus of Linguistic Acceptability), which evaluates the grammatical acceptability of sentences with perfor- mance measured using the Matthews Correlation Coefficient (MCC);
SST-2 (Stanford Sentiment Treebank), binary sentiment classification task assessing whether a sentence expresses positive negative sentiment, evaluated using accuracy;
MRPC (Microsoft Research Paraphrase Corpus), a paraphrase identification task where models predict whether two sentences have the same meaning, evaluated using both accuracy and F1 score;
RTE (Recognizing Textual Entailment), a natural language inference task where models determine whether hypothesis is truc based on given premise, evaluated using accuracy.
SQuAD (Stanford Question Answering Dataset): A inestion-answeee task that evaluates models on their ability to extract precise spans of text that answer questions from a given context, measured using F1 and exact match (EM) scores.
RACE (ReAding Comprehension from Examinations): A dataset for evaluating reading comprehension by requiring models to answer multiple-choice questions based on given passages. The dataset includes diverse linguistic phenomena, with performance measured using accuracy.
Models For each task, we utilized pre-trained and fine-tuned versions of RoBERTa, obtained from Hugging Face. Specifi- cally, we used Al/rohotta//oo as base model. textattack/roheetz-bbaeee CoL Sataatttck//ooerrtt-b-----S- textattack/roberta -arerrraaaaaeeeRp textattack/rroerta-bass-
and deepset/roberta-bbas squad2'3 also use pre-trained and fine-tuned versions of GPT-2, obtained from Hugging Face for additional experiments. Specifically, we used ppenai-community/gppp" as the base model, tanganke/gpt2-colaa and kunganke/gpt2-mrpe"
Riiidkda--robert--bss---
B.9. Evaluation Metries
For GLUE tasks, accuracy chosen as the uniform metric facilitate fair comparison across tasks. While MCC is recommended for CoLA, used accuracy to maintain consistency with other tasks. MCC typically reaches around 0.64 after fine-tuiin for CoL.A, whereas accuracy for other tasks often exceeds 0.9. This discrepancy makes it difficult to include MCC in an overall performance average.
For LLM Leaderboard tasks, the following metrics were used:
RRC: Success rate (25-shot)
HellaSwag: Accuracy (10-shot)
MMLU and Winogrande: Accuracy (5-shot)
TruthfulQA: Factual accuracy (0-shot)
GSMSK: Success rate (5-shot)
25

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
These metrics provide consistent and comparable basis for evaluating model performance across various benchmarks.
B.10. Grid Search Details
For small-scale tasks, we performed fine-grained À parameter search with an interval of 0.01 (compared to 0.1 used in previous works) ensure fair comparisons between methods. In contrast, because of the high computational cost of large-scale experiments (e.g., with 7B models), followed prior work by adopting coarser grid interval of 0.1, with equal values for all vectors. The impact of lambda grid intervals is discussed in Appendix A.I, showing how coarser intervals may lead to unfair comparisons by missing optimal values.
In our small-scale experiments, we employed a two-step grid search strategy to determine the optimal scaling coefficients that maximizes average performance across multiple tasks.
Grid Search Strategy As the sparsity level incrasse, the range of potential optimal values broadens, and performance typically follows pattern of increasing and then decreasing with respect to A. To address this, adopted two-step adaptive search strategy. First, manual search with 0.1 interval was performed identify the broader region where the optimal À is likely to reside. Based on the results of this initial search, more fine-grained search using a 0.01 interval was conducted, focusing on the identified region.
To further evaluate the method's ability to merge multiple task vectors (k > 3), additional experiments were conducted by merging four models at 90% sparsity. In these experiments, a unified À value was used across all task vectors, with a search interval of 0.01. This unified approach simplifies the process and mitigates the computational burden of searching for optimal \ combinations, which would otherwise grow exponentially with the number of models k.
Unlike fixed-range search, this adaptive strategy allowed to efficiently identify the most effective scaling coefficients for each sparsity level, ensuring precise performance optimization. The performance values presented in the main text correspond to the optimal À values found through this two-step process.
B.11. Guidelines and Experimental À Values
This section describes the guidelines for setting À values and presents experimental results using unified À across various sparsity levels for large-scale models and across different numbers of tasks for small-scale models.
Guidelines for Setting
Small-Scale Models: A fine-grained grid search with an interval of 0.01 was used ensure fair comparisons and avoid missing optimal values.
Large-Scale Models (e.g,, 7B Models): A coarser grid search with interval of 0.1 was adopted to reduce computa- tional costs, consistent with prior work.
Table 19. Unified values for large-scale models at different sparsity levels.
Task- Arithmetic
TA-
TA-
TIES-
TIES-
Sparsity Level
CABS
Magnitude
DARE
Merging
DARE
0
0.6
0.25
0.6
0.8*
0.6
0.8*
0.6
0.75
0.8
2.2*
0.8
2.2*
1.2
0.90
1.2
5.5*
1.2
5,5
1.8
Table 20. Unified À values for small-scale models different task numbers.
Task-
TA-
TA-
TIES-
TIES-
Task Number
CABS
Arithmetic
Magnitude
DARE
Merging
DARE
0.48
4.61*
1.07
1.88
5.72*
1.74
6
0.49
5.61
1.04
1.88
5.41
1.64
26

---

CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
For DARE-relate method, the reported values (e.g.. À 2.2 for 0.75 sparsity and À 5.61 for 0.90 sparsity) correspond task vectors that have already been rescaled by sparsity-adjusted factor (e.g., (1/(1 sparsity))) However, directly using these rescaled task vectors for model merging without adjusting À effectively increases the step size of the À grid search. This results in a coarser optimization for DARE, making the comparison less fair. To address this, we ensured that the DARE method underwent a finer-grained search to account for this implicit difference in grid interval and to enable a more equitable comparison with other methods.
Notes:
B.12. Hardware and Hyperparameter Configurations for Model Evaluation.
The model evaluations were performed on A100-40GB GPUs. For small-scale and discriminative tasks in GLUE, we conducted a single evaluation per model, as minimal variance was observed across repeated runs. In contrast, for generative tasks involving large models, where results can be more variable, inference was implemented via the Im-evvaluation-hamett v0.4.0. To ensure consistency and robustness, we performed three evaluations and reported the average outcome. As for the hyperparameters of generative LMs, we set the maximum generation token limit to 256, the temperature to 1.0 for sampling, and the maximum context length to 2048 tokens.
B.13. Limitations and Future Work
General Limitations. Like other task vector-based methods, our approach is limited to models with identical architectures due the clement-wise operations used in merging model weights. This constraint restricts the generalization of the framework to models with homogeneous structures. Furthermore, reliance on manual adjustment of the parameter À remains a common challenge, especially for large-language models, which requires trial and error to optimize model performance.
Limitations Specific to CABS. CABS introduces two new hyperparameters- sparse sequence and the n:m ra- tios unique its design, as discussed in Appendix A.7 and A.6. While these hyperparameters were not particularly sensitive in our experiments, they add complexity and increase computational cost.
Future Work. Several directions could help overcome these limitations. Expanding model merging techniques to include heterogeneous architectures models trained from scratch represents key area for future research. Additionally, improving the performance of merged models in multi-task settings- whhee current approaches do not yet match the performance of original single-task models-remains a priority. Automating the search for optimal hyperparameters, particularly A, would reduce complexity and improve usability, especially large-scale applications.
27


Reviews:
[{"review_id": "LPA2souPIG", "reviewer": "Reviewer_QfMa", "summary": "Authors propose a novel methodology, Conflict Aware Balanced Sparsification (CABS), for model merging based on task vectors. Previous work has shown that sparsifying task vectors before merging typically yields better performance for merged model. Authors identify two main issues:\n- High Parameter Overlap: Retained weights across task vectors overlap significantly, causing conflicts during merging.\n- Unbalanced Weight Distribution: Sparsification concentrates weights in specific regions, amplifying imbalances during merging and degrading performance\nAuthors propose to resolve this by proposing:\nA) Conflict Aware pruning: Prunes task vectors sequentially, masking already retained parameters to eliminate overlap.\nB) Balanced Sparsification: Structured n:m pruning of task vectors to enforce uniform pruning of task vectors. \n\nAuthors evaluate their approach on large LM (Mistral-7B) and small LMs (RoBERTa and GPT-2). Experiments show CABS generally outperforms existing methods when merging 2, 4, and 6 tasks language modelling tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and constructive suggestions! We've provided additional experimental tables in **[Anonymous Link](https://anonymous.4open.science/r/CABS-027B/rebuttal_tables.pdf).** Below, we address your main concerns:\n\n**Q1: Inclusion of experiments on vision modality.**\n\nWe added experiments on vision tasks (ViT-B-32 on DTD and EuroSAT, see Table 2 in the link). CABS surpasses Task Arithmetic by +2.20% average accuracy, confirming its effectiveness on vision tasks.\n\n**Q2: Justification of claims regarding outperforming the \"ideal model\".**\n\nWe agree that this claim should be more cautiously presented. We will revise the paper to clarify the contribution statement and the discussion of Table 3 results, adding appropriate qualifiers to avoid overstatement.\n\nEmpirically, the claim is supported by consistent results across multiple settings (Tables 3, 11, 12, 14, and 15), including both Mistral and GPT-2 models and sparsity levels of 0.25 and 0.75 and 0.90. In particular, the improvement in GPT-2 experiments (Table 11) is free from randomness, as both the method (CABS) and the evaluation tasks (classification) are deterministic. As noted in Q4, confidence intervals in the large-model experiments further strengthen the reliability of the observed gains.\n\n**Q3: Clarification about the novelty of the \"ideal model\".**\n\nWe agree that comparing to fine-tuned models is common for SLM, where the \"ideal baseline\" is equivalent to the ''fine-tuned models''. However, in LLM merging (e.g., multi-task LLMs), this type of upper-bound baseline is not used in prior large-model merging works such as DARE. We acknowledge that this distinction was not clearly explained and will revise the text accordingly.\n\n**Q4: Reporting confidence intervals.**\n\nOur magnitude-based merging method is deterministic for classification tasks, producing consistent results across runs. This aligns with common practice in model merging literature, where confidence intervals are often omitted. Still, for LLM evaluations, we conducted 10 runs:\n|Method|Mean Score|95% Confidence Interval|\n|-|-|-|\n|TIES-DARE|0.7606|(0.7597, 0.7614)|\n|CABS|0.7647|(0.7641, 0.7653)|\n\nThese results confirm the statistical significance of our improvements.\n\n**Q5: Discussion of related work TALL-masks.**\n\nCompared to TALL-masks, CABS distinguishes itself in both design and applicability:\n\n- TALL-masks assumes task id is available at inference, loading tuned task-specific masks—analogous to utilizing separate LoRA adapters—thus achieving near-lossless performance but requiring task-specific inference and added complexity.\n\n- TALL-masks also involves tuning the mask sparsity factor for each task and scaling coefficients λ. While effective on small models, it has not been evaluated at LLM scale. In contrast, CABS only tunes λ and has demonstrated strong scalability: on the Open LLM Leaderboard, CABS occupied the top four positions among all sub-8B models at submission time (Table 1 in the link).\n\n- TALL-masks produces a base model, a merged task vector, and k task-specific masks, resulting in >2× storage overhead. In contrast, CABS yields a single, compact model for multi-task inference.\n\nGiven these differences, direct comparison with TALL-masks is inappropriate. However, its merged variant Consensus TA is a fair baseline. We have added comparisons on merging 2,4,6 models (Table 4,5,6 in the anonymous link, Mask sparsity factor =0.5), with our results clearly demonstrating the superior performance of CABS. A discussion will be added in the Related Work section.\n\n**Q6: Effect of task ordering in CA.**\n\nWe tested shuffled task orders explicitly (SLM: Table 1 in the main text; LLM: Appendix A.9). We further added results on merging six models with shuffled orders (Table 5 in the anonymous link). Results show that task ordering has limited impact relative to overall gains, demonstrating the robustness of the CA strategy.\n\n**Q7: Impact of overlap rates.**\n\nAs discussed in the paper, the main issue is high overlap rather than all overlap. We agree some overlap can be beneficial—e.g., Figure 5 shows 20% sparsity achieving strong performance. However, determining the optimal overlap level introduces additional complexity. Considering the efficiency and generalizability, we adopt the current CA strategy. \n\nFigure 5 is intended to analyze the effect of varying overlap rates within a fixed method (DARE), and already includes the result of applying CA with DARE. Adding CABS—which uses a different method (BS)—would conflate method differences with overlap effects.\n\nInstead, we have updated Table 4 in the main paper (see Table 9 in the anonymous link) to enable a clearer comparison between BS and DARE under the same sparsity level.\n\n**Q8: Effectiveness on similar tasks.**\n\nWe discuss this in our response to Reviewer 8hif (Q5). \n\nWe hope our responses have addressed your concerns. Please let us know if there are any remaining questions or clarifications we can provide.", "claims_and_evidence": "Authors run ablation studies to verify most of their claims. \nAuthors claim that they are first to introduce an \"ideal\" baseline and CABS outperforms this ideal baseline. \n- First, the ideal baseline authors proposed is to compare the performance of the merged model to the finetuned expert. This is pretty well known and has been used as a baseline in this [1]. \n- Second, The claim that CABS outperforms this baseline is misleading because only setting is in Table 3 with sparsity =0.75. CABS has average accuracy of 76.5% and \"ideal\" model has average accuracy of 76.3%. These results are within the margin of errors and I would encourage the authors to withdraw this claim. \n\n[1] Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jiménez, François Fleuret, and Pascal Frossard. 2024. Localizing task information for improved model merging and compression. In Proceedings of the 41st International Conference on Machine Learning (ICML'24), Vol. 235. JMLR.org, Article 2057, 50268–50287.", "methods_and_evaluation_criteria": "For small LMs, authors tested on GLUE tasks (CoLA, SST-2, MRPC, RTE, SQuAD, RACE), arbitrarily chosing 2, 4, and 6 tasks to merge. \nFor large LMs, authors tested on LLM Leaderboard tasks (ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8K)\n\nThese benchmarks are standard and make sense for text data modalities. However authors have not run experiments on other data modalities, such as vision tasks.", "theoretical_claims": "There are no significant theoretical claims int he paper. Authors show that CA procedure produces non-overlapping task vectors leading to orthogonality. Thus leading to no interference. This analysis is pretty straightforward.", "experimental_designs_or_analyses": "I checked the experimental design for evaluating the CABS and the ablation studies. \n\n1. This work lack of confidence intervals in the evaluation. It appears most of the work is produced on a single seed. It'll be useful to report margin of error which could boost significance of the work. \n2. Authors argue that overlapping task vectors typically lead to performance degradation. I can imagine tasks where the data distributions are very similar where, having overlapping task vectors can indeed boost the performance. Thus, this analyses is not very convincing.", "relation_to_broader_scientific_literature": "Prior work has shown that sparsifying task vectors can lead to improved performance in model merging. In this work authors suggest that  removing overlaps in the task vectors + using structured n:m sparsity can improve the performance of model merging.", "essential_references_not_discussed": "Authors should include discussion / comparison with [1]. This is published work that also appears to improve on the baselines considered in this work and seems pretty related. \n\n[1] Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jiménez, François Fleuret, and Pascal Frossard. 2024. Localizing task information for improved model merging and compression. In Proceedings of the 41st International Conference on Machine Learning (ICML'24), Vol. 235. JMLR.org, Article 2057, 50268–50287.", "other_strengths_and_weaknesses": "Strengths:\n\n- Novel merging approach that improves the performance of model merging in evaluated tasks\n- Strong ablation studies including studying the role of sparsity rate, overlap rate, pruning approaches, and general experimental setup. \n- Paper is well structured. \n\nWeakness:\n- Only text data modality is considered. In this literature, it is common practice to also evaluate results on vision data modality. \n- Claim about comparison to \"ideal\" baseline are not justified.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. For CA approach: does the sequential nature of masking induce an implicit ordering on the \"importance\" of tasks? Have you tried shuffling this order and seen any difference in the performance?\n\n2. Figure 5 suggests that perhaps small overlap rate (20%) can be beneficial for model performance. Can you include the performance of CABS with 0% overlap rate on that plot? it'll be useful to compare the performance with varying overlap rates.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vTR9d9TDXv", "reviewer": "Reviewer_8hif", "summary": "The paper presens CABS, a method for pruning and merging different task vectors, seemingly resolving conflicts between different tasks. As the authors argue, conflicts can arise due to parameter overlap and unblanaced weight distribution - CABS is intended to address these issues. To that end, disjoint pruning masks are used, while the latter problem is resolved using n:m pruning. Experiments are conducted to show the effectiveness of the method.\n\n## Update after rebuttal\nI thank the authors for their rebuttal. However, since I find some of their arguments not really convincing, I intend to keep my score. I am not convinced that pruning, originally a method to compress models, is here a better idea than otherwise merging the task vectors. Especially the statement \"when task vectors are highly similar, their directions are nearly aligned, causing their linear combination to span only a narrow subspace\" does not really resonate with me. While this statement as is is untrue (pick any vector, add eps*canonical vectors to it, giving you a full dim subspace and all vectors are arbitrarily close\", I am not convinced why this should be an argument in the first place. Overall, I think the work needs revision to resolve also the causation/correlation issue I stated.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and constructive suggestions! Below, we address your main concerns:\n\n **Q1: Causality vs. Correlation in Experiments**\n\nWe acknowledge the reviewer’s distinction between correlation and causation. While strict causal proof is difficult, our experiments go beyond correlation. In Figure 5, by fixing sparsity and method and varying only the overlap rate, we observe performance drop—indicating a causal link.\n\nTable 4 shows that more structured pruning improves balance and performance (80.38 → 80.61 → 81.30). While full causality is not isolated, consistent trends across models highlight the practical value of addressing overlap and imbalance in training-free model merging.\n\n**Q2: why would pruning one then be any better?**\n\nPrior work shows that pruning improves model merging. TIES-Merging demonstrates that conflicting signs between task vectors cause interference, and pruning one side improves performance. DARE further shows that task vectors are highly redundant—keeping only ~5% of weights with proper rescaling still maintains near-lossless performance.\n\nUnlike standard model pruning, task vector pruning is not for compression, but for reducing conflict and removing redundant updates that may interfere with other tasks—making it a strategic, empirically supported choice.\n\n**Q3: Why merge Task Vectors Instead of Learning a Joint Vector?**\n\nTask-vector-based merging aims to reuse independently fine-tuned models in a training-free manner, without access to original task data. In contrast, learning a joint vector requires access to all task data and additional training, which significantly increases computational cost.\n\nCABS builds on TIES-Merging and DARE, proposing structured and conflict-aware pruning to enable efficient, training-free model merging.\n\n**Q4: Does CA over-prune task vectors by enforcing disjoint masks?**\n\nCA does not cause excessive pruning. In many cases, task vectors already reach 90% sparsity without CA. At the same sparsity level, CA improves merging by allocating disjoint weights, reducing interference while retaining essential updates.\n\n**Q5: If two tasks are fairly similar, why force them to orthogonality?**\n\nIn our experiments with similar models—such as the model soup setting in Table 3 and Table 17 of the main paper, and Table 1 in **[Anonymous Link](https://anonymous.4open.science/r/CABS-027B/rebuttal_tables.pdf)**—we consistently observe that enforcing orthogonality via CABS still improves merging performance.\n\nOne geometric explanation is that, when task vectors are highly similar, their directions are nearly aligned, causing their linear combination to span only a narrow subspace. In contrast, enforcing orthogonality increases the span of the merged space. This broader coverage may facilitate better exploration of the solution space, enabling access to lower-loss regions.\n\nThus, even for similar tasks, CABS effectively improves expressiveness and reduces redundancy during the merging process.\n\n**Q6: Clarifying Eq. 7–9**\n\nEquation 7 defines a linear combination of two task vectors, but does not reveal how adjusting one scaling factor (e.g., λ_B) may interfere with the other task vector’s contribution in the resulting update.\n\nFor example, if τ_A = −τ_B, even small changes in λ_B can diminish the effect of τ_A, indicating strong coupling between the scaling factors. This illustrates that without further analysis, the two task vectors' contributions cannot be assumed independent.\n\nThe squared Frobenius norm in Eq. 8 explicitly captures this interaction via the inner product ⟨τ_A, τ_B⟩_F. As shown in Eq. 9, the cross term vanishes only when τ_A and τ_B are orthogonal.\n\nThis ensures that each λ independently controls its corresponding task vector’s contribution without unintended interference. Hence, the norm-based formulation is essential to formalize the decoupling effect of orthogonality.\n\n**Q7: Density vs Sparsity**\n\nWe consistently use \"sparsity\" to refer to the percentage of pruned (zero) parameters—e.g., 90% sparsity means 90% of weights are zero. We do not use \"density\" in the paper, so there should be no ambiguity.\n\nThat said, we acknowledge that Table 2 does not explicitly mention the sparsity setting. However, the table references specific results in Tables 1, 8, and 9, where the sparsity level (90%) is clearly stated. We will revise Table 2 to include this information. We hope this clarification resolves the misunderstanding.\n\n**Q8: Why does rescaling work?**\n\nRescaling is effective because task vectors are highly redundant—shown by DARE and our Figure 6 (Appendix A.8). Pruning up to 90%, with rescaling, yields near-lossless performance. Even at 99%, the drop is minimal, showing strong robustness.\n\n**Q9: The calibration dataset for SparseGPT and Wanda**\n\nWe use C4 as the calibration dataset, following the standard setup in SparseGPT.\n\nWe hope these responses address your concerns. Please feel free to let us know if anything remains unclear.", "claims_and_evidence": "- While I like the storyline of the paper, I am not convinced that the experiments show that high parameter overlap and unbalanced weight distribution are the issues in model merging, the experiments presented are not causal, but rather show correlation. For example, the overlap is larger when doing magnitude pruning, instead of random pruning, cf. Figure 2. Why more overlap should lead to worse performance is not clear - I understand the arguments when e.g. one task is positive at a certain weight, the other one negative, but why would pruning one then be any better? I believe the numbers, but I do not think that the experiments show that these are truly the issues.\n- When merging two task vectors, why are you pruning both to then build up a linear combination - why doesn't it work to build a new task vectors that is accounts for both tasks, e.g. on a calibration dataset? I find the strategy a bit convoluted.\n- When pruning task A, then using the inverse of the mask to restrict the weights of task B, which are then in turn pruned, do I not necessarily prune either A or B too much? In the end, this guarantees that the task vectors are orthogonal, which seems to be much much more than what is needed - if two tasks are fairly similar, why force them to orthogonality?\n- From Eq. 7 to Eq. 9 it is derived that the \"decoupling\" in the norm \"ensures that adjust $\\lambda_A$ affects only the contribution of $\\tau_A$\", but this is not clear to me. This statement holds in Equation 7 already, why would the norm be needed here?", "methods_and_evaluation_criteria": "The experimental design overall seems sound, however I am not entirely sure which sparsities are used, e.g. in Section 5.2.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "The authors do a good job of relating their work to the broader literature.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "- The derivation in lines 259-274 is trivial and does add little value, I would have preferred to use this space to explain why \"these sparse updates are nearly lossless in retaining task-specific information, as simple rescaling compensates for pruning-induced changes\". This statement may hold for very little sparsities, but otherwise this will not be the case.", "other_comments_or_suggestions": "- The running title has not been updated and is still the default one.\n- I find it a bit odd to cite Liang et al. when introducing magnitude pruning in line 34. Why would you cite a 2021 paper to introduce one of the most basic pruning methods? Later, Zhu & Gupta are cited (line 156), which seems more appropriate, but still not really fitting - they introduced basically a pruning schedule which is based on magnitude pruning.\n- There are some typos in the paper, e.g. \"Remaining\" in line 83 should be lowercase, \"compare to\" should be \"compared to\" in the caption of Figure 3.\n- Here, sparsity seems to be used in the sense of density. I see this from time to time in other papers, but it is not the correct usage in my opinion. A sparsity of 80% means that 80% of the parameters are zero. A density of 80% would mean that 20% of the parameters are zero.", "questions_for_authors": "- What is used as the calibration dataset for SparseGPT and Wanda? Did I miss this or is this nowhere stated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n3B6crLDL6", "reviewer": "Reviewer_zNot", "summary": "This paper introduces a task-vector-based model merging method, CABS. The authors attribute the performance degradation of model merging to: (1) high parameter overlap, and (2) unbalanced weight distribution. The proposed CABS mainly contains two modules, CA and BS. Between them, CA aims to eliminate parameter overlap and BS is introduced to guarantee a more even weight distribution. Experiments on both LLMs and small language models (SLMs) are conducted to validate the performance of the proposed method.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and constructive suggestions! We've provided additional experimental tables in **[Anonymous Link](https://anonymous.4open.science/r/CABS-027B/rebuttal_tables.pdf).** Below, we address your main concerns:\n\n**Q1: Whether the observed performance gap is due to lack of rescaling in magnitude-based pruning.**\n\nWe agree that rescaling is a key factor contributing to DARE’s effectiveness and could enhance the performance of magnitude-based pruning methods. In our experiments, all magnitude-based pruning methods were equipped with rescaling. The λ values used for merging (e.g., λ=1.88 for 4-model TIES-Merging) are detailed in Appendix B.11, confirming that proper rescaling was employed. Additionally, we also conducted rescaling experiments for magnitude-based pruning methods, which are reported in Appendix A.8.\n\nNotably, as illustrated in Table 1, even when rescaling is applied, magnitude-based pruning can sometimes impair the model merging performance. The comparison in Table 1 underscores that factors beyond rescaling - such as weight distribution - are also important for the merging performance, as further discussed in our response to Q2.\n\n**Q2: The impact of unbalanced weight distribution on merged model performance needs clearer evidence.**\n\nWe provide supporting evidence in Table 4 in the main text. It compares different sparsification strategies for model merging: Task Arithmetic + Magnitude (layer-wise), + Magnitude (row-wise), and + BS (block-wise). As the pruning becomes more structured and the weight distribution more balanced, performance improves progressively from 80.38 to 80.61 to 81.30.\n\nFrom a theoretical perspective, weight imbalance can amplify task interference during merging. Since the merged model is typically rescaled by a λ, if weights are highly concentrated in specific regions—as often caused by magnitude-based pruning—this rescaling will disproportionately amplify those regions, worsening cross-task interference.\n\nWhile weight distribution may not be the sole cause, consistent improvements across balanced pruning provide strong empirical support. We acknowledge that deeper theoretical understanding, especially of activation dynamics, would be valuable. While our current work focuses on developing an effective, training-free merging method, we see this as a promising direction for future research.\n\n**Q3: The LLMs used for merging are too similar in performance; more diverse tasks are expected.**\n\nTo address the concern about task diversity, we additionally conducted an experiment merging a Mistral model fine-tuned on instruction-following with another fine-tuned on mathematical tasks. As shown in Table 2 in anonymous link, our method still outperforms TaskArithmetic by +0.79, achieving SOTA performance in this heterogeneous setting and demonstrating robust cross-task merging capability.\n\nWe also would like to clarify that, merging models fine-tuned on similar tasks is a standard and widely adopted setup in model merging, especially under the model soup paradigm. Our large-model experiments follow DARE’s protocol, which focuses on improving performance or generalization within similar task domains.\n\nThis setup is also practically relevant: the Open LLM Leaderboard—one of the most active community benchmarks—frequently features submissions involving merges of LLMs fine-tuned on similar tasks. Using our method, we constructed four merged models and occupied the top four positions among all models with fewer than 8B parameters at the submission time (see Table 1 in the anonymous link or the official leaderboard).\n\n**Q4: The number of tasks/models in SLM experiments is limited; merging more models is recommended.**\n\nWe appreciate the suggestion and have added an experiment merging all 7 GPT-2 models from FusionBench, following its official setup (see Table 7 in the anonymous link).\n\nWhile CABS still achieves the best performance among all methods, the overall gains are notably smaller than in our 2- or 6-model merging settings. In fact, all methods—including CABS—suffer significant performance drops when merging more models, especially on some tasks.\n\nWe investigated this and found that the task vectors have highly imbalanced average weight magnitudes—up to a 10× difference across tasks (see Table 8 in the anonymous link). Since model merging uses a shared scaling factor (λ), tasks with small-magnitude vectors (e.g., RTE, MRPC, CoLA) are overwhelmed by stronger ones (e.g., QQP, MNLI), resulting in near-random performance on weaker tasks.\n\nThis suggests that blindly merging too many heterogeneous task vectors with vastly different magnitudes introduces severe imbalance and can degrade model utility. While CABS remains robust, merging fewer well-aligned models appears more practical and effective under current frameworks. \n\nWe hope our responses have addressed your concerns. Please let us know if there are any remaining questions or clarifications we can provide.", "claims_and_evidence": "The authors attribute the performance degradation of model merging to high parameter overlap, and unbalanced weight distribution. I agree with the viewpoint that parameter overlap degrades the performance, which is intuitive. As to parameter overlap, the authors demonstrate that the pruning based on magnitude results in unbalanced distribution of weights. However, from my point of view, I believe that the success of DARE is because of its rescaling process. I guess that if magnitude-based pruning is properly rescaled, it can achieve the performance close to DARE. As a result, I think that the relationship between weight distribution and the merged model’s performance needs to be further proved.", "methods_and_evaluation_criteria": "Yes. The proposed method is simple and provides an interesting perspective for model merging. However, some of its claims and experiments are not that convincing to me.", "theoretical_claims": "Yes. I checked Eq. 2 to Eq. 9. No obvious errors are found.", "experimental_designs_or_analyses": "Yes, I checked the experiments in Section 5 and the additional results in Appendix A. I have several questions:\n(1)\tThe selected LLMs have similar performance on the target tasks. For example, the performance difference of WestSeverus and WildMarcoroni is below 1% on most tasks in Table 3. However, from my point of view, model merging targets at replacing multi-task learning by merging expert models on different kinds of tasks. Say, merging LLMs finetuned on code or math tasks. Merging models with very similar performance may reduce the soundness of the method.\n(2)\tFor SLMs, the authors selected 4 tasks from GLUE and 2 other tasks in Table 2. However, I checked the huggingface links for roberta models utilized by the authors and find that there are many other weights that can be used, including roberta models finetuned on other tasks from GLUE. Similarly, in Table 11, only two GPT-2 models are utilized for merging but the original setting for this benchmark is merging 7 GPT-2 models finetuned on different tasks. Providing experimental results for merging more models make the proposed method more convincing.", "relation_to_broader_scientific_literature": "This paper attributes the performance degradation to high parameter overlap and unbalanced weight distribution. Previous studies usually use pre-processing to reduce the parameter overlap or interference. In this paper, the proposed CABS applies pre-computed masks to avoid the overlap.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n(1) Simple and intuitive idea.\n(2) Detailed proofs.\n(3) Comprehensive experimental results.\n(4) Ablation studies provided.\n\nWeaknesses (both previous-mentioned):\n(1) The relationship between weight distribution and the merged model’s performance is not that convincing to me.\n(2) Some experimental settings could be broadened.", "other_comments_or_suggestions": "(1)\tIn Line 1120, there is a ref error.\n(2)\tIn Figure 6, the font sizes are a bit too small to read.", "questions_for_authors": "Overall, I think the proposed method is meaningful, but it may lack a motivation strong enough and convincing experimental results. This is why I submit a negative overall recommendation. However, if the authors could:\n(1)\tfurther prove the relationship between weight distribution and performance;\n(2)\texpand the experimental settings.\nI would carefully consider raising my rating score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TBDYdhR0cO", "reviewer": "Reviewer_Lsxt", "summary": "This paper introduces a model merging algorithm, CABS, to address the key issues in sparsified task vectors: high parameter overlap and unbalanced weight distribution. The proposed method consists of two components: Conflict-Aware (CA) Sparsification, which sequentially prunes task vectors to minimize overlap, and Balanced Sparsification (BS), which ensures even distribution of retained parameters. The authors conduct comprehensive experiments, demonstrating that CABS outperforms existing pruning-based merging techniques and surpasses an ideal baseline in some cases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and constructive suggestions! We've provided additional experimental tables in **[Anonymous Link](https://anonymous.4open.science/r/CABS-027B/rebuttal_tables.pdf).** Below, we address your main concerns:\n\n**Q1: Comparison with recent baselines (Localize-and-Stitch [1], TALL-masks [2]).**\n\nWe experimentally compared CABS with TALL-masks and Dataless Localize-and-Stitch, with our results clearly demonstrating the superior performance of CABS.\n\nFor TALL-masks, please refer to our detailed response to Reviewer QfMa (Q5) for a comprehensive comparison.\n\nFor Dataless Localize-and-Stitch, experimental results are presented in Tables 4, 5, and 6 of the anonymous link. When merging 2, 4, or 6 models, its performance is often close to the base model, indicating limited effectiveness.  We found it essentially equivalent to simplified TIES-Merging (fixed λ=1, no 'elect').\n\nThis setting works reasonably well in the original setup, where 12 models are merged using 5–10% sparsity, so that λ = 1 happens to act as a good approximation (e.g., 5-10% × 12 ≈ 0.6-1.2). However, when merging a smaller or larger number of models, such a fixed λ becomes suboptimal.\n\nIf desired, we are happy to include a λ-tuned version of Dataless L-and-S for fairer comparison.\n\n**Q2:Effectiveness of \"elect\" step in TIES.**\n\nWe appreciate the valuable ideas introduced by TIES-Merging, particularly its insight and solution regarding parameter redundancy and conflict. Inspired by TIES-Merging, CABS offers a more robust alternative.\n\nSpecifically, the “elect” step in TIES is designed to resolve conflicting signs between task vectors and can indeed be helpful in certain cases. For example, as shown in Table 17 of our appendix, TIES-Merging is the best-performing baseline after CABS. However, its effectiveness is inconsistent. The TIES paper itself (Appendix B.1, Table 7) shows that re-estimating the sign vector via few-shot training improves performance, indicating that its elect strategy may be suboptimal.\n\nOur experiments show similar trends. As seen in Table 1, the performance gap between Task Arithmetic+Magnitude and TIES-Merging is often small despite the difference being the elect step, and in some cases, TIES-Merging even underperforms.\n\nOne possible explanation is the significant imbalance in average magnitudes among task vectors—up to a 10× difference as illustrated in Table 8 of the anonymous link. In such cases, the sign vector may be dominated by the task with the largest magnitude, resulting in biased merged results.\n\nOverall, while TIES-Merging laid important groundwork in addressing parameter conflicts, CABS advances this direction by providing a more robust and effective solution.\n\n**Q3: Are parameter interference the right problem？**\n\nAs noted in our response to Reviewer QfMa (Q5), CABS and TALL-masks operate under fundamentally different paradigms.\n\nWhile TALL-masks separates tasks during inference and does not explicitly address parameter interference, it incorporates TIES-Merging as the first stage in its pipeline. This design choice implicitly acknowledges parameter interference as a key factor.\n\nIn such settings—where task identity is unavailable and generalization is required—parameter conflict becomes an inevitable challenge for training-free model merging methods.\n\n**Q4: motivation for choosing n:m pruning.**\n\nWe choose n:m pruning for two reasons: (1) Prior methods, like row-wise (topk), already outperform layer-wise pruning, making it natural to explore more structured forms, like block-wise (n:m), to further improve merging performance. (2) n:m pruning operates at the finest granularity (weight level), is simple to apply, and requires no changes to model architecture.\n\nOther pruning techniques are less suitable for training-free merging. Coarse-grained methods (e.g., head pruning) cannot remove redundant weights or resolve conflicts within components. Sparse fine-tuning and similar methods require training, limiting their practicality.\n\n**Q5: Potential scalability concerns for CA.**\n\nPlease refer to our detailed response to Reviewer zNot (Q4). \n\n**Q6: Analysis on the impact of pruning order.**\n\nPlease refer to our detailed response to Reviewer QfMa (Q6). \n\nBuilding on our new findings on task vector magnitudes (Table 8 in the anonymous link), we suggest that placing task vectors with smaller magnitudes earlier in the pruning sequence—or assigning them larger λ—may help improve balance. This could be a promising future direction, as exhaustive search is infeasible in LLM merging.\n\n**Q7: necessary of Validation data usage for tuning λ.**\n\nLike TA, TIES, and DARE, we use validation data for tuning λ. The specific values are provided in Appendix B.11.\n\n**Q8: Reporting statistical significance with confidence intervals.**\n\nPlease refer to our detailed response to Reviewer QfMa (Q4).\n\nWe hope these responses address your concerns. Please feel free to let us know if anything remains unclear.", "claims_and_evidence": "Yes, most of the claims are well supported by evidence.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are some analyses in Sec 4.3, but they are pretty shallow.", "experimental_designs_or_analyses": "I think the experiments are comprehensive and nicely executed in general, and I particularly appreciate the experiments on large-scale language models (up to 7b). Nevertheless, \n- The paper does not discuss whether CABS requires validation data for tuning the scaling coefficients, which is a critical factor in model merging methods\n- The performance gap between different methods is not very large. Reporting error bars in addition to accuracy would help assess the significance of the improvements\n- The paper should include comparisons with recent baselines [1,2] to strengthen its empirical claims\n\n\n[1] He, Yifei, et al. \"Localize-and-stitch: Efficient model merging via sparse task arithmetic.\" arXiv preprint arXiv:2408.13656 (2024). \n\n[2] Wang, Ke, et al. \"Localizing Task Information for Improved Model Merging and Compression.\" arXiv preprint arXiv:2405.07813 (2024).", "relation_to_broader_scientific_literature": "The paper is positioned within the broader literature on model merging. Specifically, it aims to further enhance the technique of sparsified task vectors (which could potentially mitigate the interference between different tasks) using sequential and balanced pruning.", "essential_references_not_discussed": "Most of the related work is sufficiently covered, except for a few cases (e.g., [2]). I encourage the authors to elaborate further on both [1] and [2].", "other_strengths_and_weaknesses": "**Strengths**:\n- The proposed approach is straightforward yet effective\n- The evaluation is thorough, spanning multiple tasks, models and sparsity levels\n\n**Weaknesses**: \n- Motivation: My complaint is mainly about the parameter overlap part (I think the unbalanced weight distribution is a good observation). \n  - The current discussion is more or less similar to that made in [1]; the additional theoretical justifications in Sec 4.3 are shallow\n  - There are a few points that I believe the authors should consider and discuss: 1) The authors claim that conflicting weight signs between task vectors lead to interference, but this is precisely what the \"elect\" step in TIES is designed to address. Are you suggesting that this step is ineffective? 2) [2] suggests that parameter overlap (or weight interference in their terminology) is not the primary cause of performance decline in model merging, which raises concerns about whether CABS (and similarly [1]) are targeting the right problem\n- Method: \n  - About \"CA\": The authors claim that the order of pruning does not significantly impact accuracy, but this is not necessarily true in general. A deeper analysis of the effect of ordering and some recommendations for practitioners would be a huge plus. Additionally, due to the nature of this step, there seems to be a scalability issue with respect to the number of tasks. In fact, CABS is only tested on a maximum of six tasks.\n  - About \"BS\": While simple, the motivation behind using n:m pruning is not clearly explained, and alternative structured pruning methods should be discussed", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents an approach that uses task vectors to merge models, enabling the creation of a single multitask model without requiring retraining.  The authors avoid conflicts between task vectors suffered by prior work by using masking to help ensure each task vector contains distinct, non-overlapping parameters.  Reviewers are split on the outcome of this paper, with arguments in favor citing the novelty of the proposed approach and the strong empirical results, with arguments against requesting more evidence that the cited issues with prior work (e.g., high parameter overlap) are casually related with the performance drops or additional experiments on more datasets. While the former was not completely addressed, the additional experiments were provided and satisfied reviewers who updated their reviews post-rebuttal.  As such, while there are clear concerns about the current paper, the ACs find they are not sufficient to argue for rejection, and the authors are encouraged to address issues raised about their paper in their revision.
Decision: Accept (poster)