Activation-Informed Merging of Large Language Models
Authors: Amin Heyrani Nobari
Kaveh Alim
Ali ArjomandBigdeli
Akash Srivastava
Faez Ahmed
Navid Azizan
Venue: NeurIPS 2025 poster ()
Keywords: Model Merging
Activation‑Informed Merging
Large Language Models
Primary area: deep_learning
TL;DR: AIM introduces a new scheme to improve merging performance in LLMs by combining continual learning principles and activation space-based model compression.

Abstract: Model merging, a method that combines the parameters and embeddings of multiple fine-tuned large language models (LLMs), offers a promising approach to enhance model performance across various tasks while maintaining computational efficiency. This paper introduces Activation-Informed Merging (AIM), a technique that integrates the information from the activation space of LLMs into the merging process to improve performance and robustness. AIM is designed as a flexible, complementary solution that is applicable to any existing merging method. It aims to preserve critical weights from the base model, drawing on principles from continual learning (CL) and model compression. Utilizing a task-agnostic calibration set, AIM selectively prioritizes essential weights during merging. We empirically demonstrate that AIM significantly enhances the performance of merged models across multiple benchmarks. Our findings suggest that considering the activation-space information can provide substantial advancements in the model merging strategies for LLMs with up to 40% increase in benchmark performance. Our code is publicly available at https://github.com/ahnobari/ActivationInformedMerging

Activation-nnformed Merging of Large Language Models
Amin Heyrani Nobari* Kaveh Alim "1 Ali ArjomandBigdeli" Akash Srivastava', Faez Ahmed', Navid Azizan'
Massachusetts Institute of Technology Stony Brook University
MIT-IBM Watson AI Lab & Red Hat Al Innovation
Abstract
Model merging, method that combines the parameters and embeddings of mul- tiple fine-tuned large language models (LLMs), offers promising approach enhance model performance across various tasks while maintaining computational efficiency. This paper introduces oetivation-Informet Merging (AIM), technique that integrates the information from the activation space of L.LSs into the merging process to improve performance and robustness. AIM is designed as a flexible, complementary solution that is applicable to any existing merging method. It aims to preserve critical weights from the hase model, drawing on principles from con- tinual learning (CL) and model compression. Utilizing task-agnostic calibration set, AIM selectively prioritizes essential weights during merging. We empirically demonstrate that AIM significantly enhances the performance of merged models across multiple benchmarks. Our findings suggest that considering the activation- space information can provide substantial advancements in the model merging strategies for LLMs with up 40% increase benchmark performance. Our code is publicly available at https:// www.////wobaa//iiiiiiooonnomeeeo
Introduction
Foundation models are rapidly becoming the dominant force for building Artificial Intelligence (AI) systems. many cases, researchers build their machine learning models by starting from pre-trained foundation models and fine-tuning (FT) these pre- pre-trained models for some desired target task [27]. In such a paradigm, numerous fine-tuned models are developed for various tasks. However, an important opportunity is missed, as these fine-tuned sask-specialized models typically operate in isolation without leveraging the rich features that each possesses [35]. This fact highlights the importance of growing area of research focused on combining multiple task-specialized models fine-tuned from the same base foundation model.
In particular, as large language models (LLMs) continue evolve, becomes increasingly important to develop methods that can effectively fuse the specialized knowledge of various fine-tuned models derived from the same foundation model. Model merging has shown broad applications, including enhancing accuracy and robustness [40], improving generalization [28], multi-modal models [37], and model alignment to human feedback [29, 30]. Given these benefits. a substantial amount of attention has been devoted developing more effective merging algorithms for LLMSs
In the vast majority of cases, merging LLMs done using algorithms that explore the weight space of models and do not leverage the information in the activation space. Activation space information has been widely used to develop model pruning and compression methods, both in the context of
Equal Contribution.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Pie Trained
Kath Expert
Change In Weghtt Merging
date Model
473
Code Expert
Existing Methodh
O
oollowing
mnergdd
AIM Adjusted Merged Madel Parmmeters
A pre
Qgen + (1 Apre (1 w)) A mrrged
TalentWaggtt From Accivattonn Space
Calibuaion Corpus
Figure 1: Overview of the proposed activation-informee merging (AIM) in LLMs.
general deep learning methods [12], and more specifically for large language models [23]. However, this direction has remained underexplored for developing more robust merging algorithms. We hypothesize that the activation space information may hold key insights useful for model merging and explore this in our work.
In this paper, we view the problem of merging from a continual learning perspective. Specifically, we explore how FT models and merging them significantly deviate from the pre-trained base model, potentially leading to overall performance degradation. This is analogous to the common catastrophic forgetting problem in continual learning [10, 35, 39]. Given this perspective, we incorporate the activation space information, taking into account the importance the base model in preserving its pre-trained capabilities while integrating new knowledge from fine-tuned models. To achieve this, introduce new method named dndivationIIforman Merging (AIM), which modifies the update step in the merging process to ensure that the most influential weights of the base model, identified through its activations, undergo minimal changes. Figure demonstrates this basic process of AIM.
AIM fundamentally relates the widely used approach of weight regularization in continual learning [1, 22, 31]. When merging large language models fine-tuned from the same base model, the goal is to maintain the base model's performance while improving the merged model's 'ss expertise using the fine-tuned models. Various methods have been proposed to merge fine-tuned LLMs [7, 19, 21, 25, 40, 42, 46, 47]. Although useful, these methods are fragile to outliers and low-quality fine-tune models and may perform worse than the base model. Hence, we take new perspective toward merging and adopt the continual learning view to prevent catastrophic forgetting, Our extensive experimental study shows that AIM is complementary solution that can be applied in conjunction with any prior merging methods and consistently improves their performances across all tested benchmarks-inchuue
math, code, and instruction following- by up to 40%. Despite its simplicity, AIM shows the effectiieness and importance of the activation space information for more effective model merging.
2 Background and Related Work
LLM Merging. Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. Often, fine-tuned models readily available, but their training data is not due to data privacy or intellectual property concerns [8]. With many fine-tuned checkpoints from the same pre- pre-trained model, LLM merging has emerged as complementary approach to fine- tuning, combining multiple aaeesspeda models into unified architecture. This technique offers several advantages: it reduces storage and inference costs by consolidating expertise into a single model, facilitates compositional generalization by integrating specialized capabilities from different fine-tuned models, and supports decentralized model development by allowing independently trained models to be merged efficiently [43].
Model soup, proposed by Wortsman al. [40], demonstrates that even simple averaging of the weights of multiple fine-tuned models can enhance accuracy and robustness in image classification and natural language processing applications. Further, Ramé et al. [28] show that model merging can improve out-of-distribution generalization performance. Extending this approach beyond unimodal settings, Sung et al. [37] empirically demonstrate that model merging is also effective in multimodal setups. Beyond generalization benefits, model merging has also been explored for alignment: WARP [30] and WARM [29] introduce weight merging strategies to improve alignment in reinforcement learning from human feedback. WARP demonstrates that merging rewarded policies enhances model quality and alignment, while WARM shows that weight-averaged reward models improve robustness and help mitigate reward hacking.

---

Recently, many methods have been proposed to go beyond simple weight averaging to merge fine- tuned LLMs into a multitask model by combining their capabilities Spherical Linear intERPolation (SLERP) [34], originally proposed for animating rotations, which interpolates between two check- points, can be a linear path in models' parameter space, A main shortcoming of this approach is that it does not support merging more than two models. Jang et al. [20] also leverages geometric insights, showing that merging only two fine-tuned models can provide superior in-distribution and out-o--distribution performance compared to ensembling multiple models.
modification to simple weight averaging that finds a spherical path instead of
Task Arithmetic [19] generalizes simple weight averaging by introducing task vectors. It suggests moving the base model parameters in the direction of weighted average of fine-tuned model differences with respect to the base model. Followed by the introduction of task vectors. Model Breadcrumbs [7]. Trim, Elect Sign & Merge (TIES merging) [42], and Drop And REscale (DARE) [46] leverage pruning techniques for better and more scalable ways of merging task vectors. Welght DisENtanglement based merging (WIDEN) [47] takes more sophisticated approach to model merging by disentangling and analyzing weight components. Another line of work on model merging takes advantage of the information in the model activations of the training data. Matena and Raffel [25] propose Fisher merging, which leverages the Laplace approximation by using the diagonal of each model's Fisher information. Jin et al. [21] attempt to minimize prediction differences between the merged model and the individual models and introduce the Regression Mean (RegMean) method that calculates the optimal weights with respect to Euclidean distance to model predictions.
We refer the reader to Yang et al. [44] for a more comprehensive literature overview. The study provides detailed discussion of model merging methods and theories, explores their applications in LLMs and multimodal large language models, and highlights future research directions.
Continual Learning. Continual learning strategies can be categorized into five overarching ap- proaches: regularization, where parameters are constrained or updated based on past data; replay, where past data is replayed for the model as encounters new data; optimization, where the loss function and optimizer are targeted; representation, where new data representations and learned embeddings can be exploited for less forgetting; and architecture, where models and parameters can expand as new data arrives [39]. To avoid catastrophic forgetting of the base model's abilities, we view the model merging problem through the lens of CL. primarily focusing on eegularizationbasee methods. Regularization-uun based methods penalize deviation from the base model according to some norm [33]. Various methods have been proposed to mitigate catastrophic forgetting, e.g., Aljundi al. [1], Kirkpatrick al. [22], Ritter al. [31]. In particular, Elastic Weight Consolidation (EWC) employs a Fisher Information Matrix to identify and protect parameters crucial for previous tasks by adding a quadratic penalty the parameter shifts. Integrating CL approaches into the merging framework involves defining a weighted regularization term that selectively constrains parameter updates in critical areas for retaining previously learned tasks. This integration not only mitigates the risk of catastrophic forgetting but also enhances the adaptability and utility of the merged model.
Model Compression. Using activation space information has been shown to be useful in the context of model compression. Frantar and Alistarh [12] show that using a calibration dataset, deep learning models can be quantized and/or pruned efficiently. [23] introduce Activation-awe -wware Weight Quantization (AWQ) for LLM compression and show that protecting only 1 1% salient weights can greatly reduce quantization error.
Building on these ideas, this paper presents a complementary merging approach that utilizes base model activations and principles from AWQ. Our method efficiently sketches delta parameters, ensuring the base model retains its original capabilities while incorporating expertise from fine-tuned models.
3 Methodology: ectivation-Informma Merging
As discussed in Section 2, most existing approaches for merging FT LLMS primarily focus on the weight space of the models being merged. However, it is well established that the activation space of these models contains crucial insights into the degree of importance of different parameters of L.MMs. This was shown to be the case, for instance, in the work done by Lin al. [23] on Activation-awar Weight Quantization (AWQ), outperforming traditional quantization methods by including insight from the activation space of LLMs. Given this, we hypothesize that the activation space of LLMs
3

---

likely holds useful clues for model merging as well. Inspired by AWQ, we introduce Activation- Informed Merging (AIM) for merging FT LLMs. In this section, we detail our proposed solution and discuss some of the inner workings of AIM.
3.1 The Merging Problem and Connections to Continual Learning
Consider the merging of N models with parameters 02, 02...................... Oy fine-tuned on different tasks from a common pre-trained model with parameters Ppre- For each fine-tuned LLM, we are essentially creating experts on specific tasks that move away from the generalist pre-trained model, hence usually degrading performance in some tasks while improving performance on the task for which the model is fine-tuned. In this sense, each FT model with parameters O, can be seen as a model fitted to a new task D,, [X,, Y. in continual learning scenario with the potential for catastrophic forgetting on the generalist pre- pre-trained model, which may not perform as well on the specific task but will have a more balanced performance across various tasks. As such, we hypothesize that when merging FT LLMs adapted from the same base model, emphasis on the base model can build better robustness to large performance degradation across numerous tasks while still allowing for capturing each FT expert's capabilities. AIM seeks to achieve this by relaxing the changes to the salient weights of the base model in the final merged model. In this way, AIM is analogous to weight regularization in many continual learning approaches [1, 22, 31, 33, 39]. Notably, the saliency of weights is determined by analyzing the activation space (sensitivities) of the base model rather than just regularizing based on the weight space, similar to [10, 26].
3.2 Activation Space Analysis
AIM determines the saliency of a model's weights by looking the scale of activations by passing a calibration dataset to the model and recording the scale of activations in each channel. To better understand why. we will analyze how perturbation of the weights of given model affects the model outputs. Let the original weights be w E R XM and the perturbation be dw E IR xM, such that the perturbed weights are w' = w + ow. The output of linear layer with input E RIxN and perturbed weights is
y' = rw' ax + duc) = xw + x((wr).
(1)
The magnitude of the error due to this perturbation, Error = y' - x(dw), scales with the magnitude of activation æ:
N
M
Emrorr|p p = diag(z)owll I (iag(=))6((1 S
8wrgl.
(2)
For any specific input channel Z4, the error contribution from perturbation in the i-th row of w, dw; is amplified by the magnitude of the same channel in the input. As such, one could selectively regularize the weights based on the importance of the input channels, i.e., their magnitudes. In this way, we use calibration dataset to capture the average magnitude of the input channels for each layer of the base model and determine the saliency of weights in the base model from the activation space.
3.3 Calibration Data and Robustness to Calibration Data
We choose the calibration dataset to be subset of the validation data from the pile dataset [13], which is similar in distribution to most pre-training datasets. Most notably, this dataset has been utilized in model compression for both quantization in AWQ [23] and pruning in WANDA [36]. This calibration dataset is considered to be fairly diverse and not task-specifie, which should allow to quantify weights' saliency in a robust manner.
Robustness to Calibration Data The use of activation spaces from calibration data has been thoroughly studied in model compression, by Lin et al. [23] and Sun et al. [36]. both cases, authors conduct extensive studies on the robustness of their methods, which, like ours, rely on the magnitude of activations, and both concur two important findings. 1) When using calibration data to quantify activation space magnitudes to be used by algorithms for model quantization or pruning, performance is robust dataset quality. This is in contrast methods that require fine-tuning retraining after compression. This observation is made clear in both studies and confirms that sensitivity to the quality of the data is much less in methods that require only activation space analysis without

---

training, 2) Most notably, ablation studies on the size of the calibration set also show robustness to the size of calibration data, with both studies confirming that only 8-32 sample sequences of length approximately 2048 tokens are enough for model compression algorithms to produce robust outcomes [23, 36]. Despite this robustness, we use the same 256 total sequences (approximately 524K tokens) that the authors of both studies use in their main experiments; however, this robustness to dataset size is noted as an important tool for reducing computational cost and time for running algorithms such as ours. We also study this matter an ablation study and show that, like those studies, AIM is also robust to dataset size and can be made much more efficient if need be.
3.4 Adaptable Relaxation Scheme
As discussed, we introduce an adaptable relaxation scheme based on the activations of the base model, which we wish not to stray away from significantly. To make the scheme adaptable to any merging algorithm in the weight space, we formulate our relaxation scheme in terms of the changes in the weights, Given a lask-agnostic representative calibration corpus D. We can pass this corpus through the model and accumulate the activations from each token in the dataset. This will yield the average magnitude of activations for each channel in all layers of the model. As previously noted, averaging the inputs over the calibration set yields a vector E I for each lincar layer. Constructing diagonal matrix from the absolute values of this vector gives diag (a=) E RNN By normalizing diag([=)) using its maximum value, we obtain diagonal matrix Ape E [0, 1J This matrix serves to modulate weight updates based on input saliency, allowing for consistent control of merging through a universal hyperparameter w. Next, we define the action of any given merging method by the changes that it applies to the model weights with respect to each of the fine-tuned models being merged (i.e., 0,, oz..... 0,)) Specifically, we denote the weight update contributed by fine- tuned model (with parameters 0;) to the model parameters by A; (e.g., A; 6--0m for weight averaging). Now, we propose an adaptive relaxation scheme that adjusts the final model. For simplicity of notation, let 0 refer to weights of linear layer (i.e., an N x M matrix), then the AIM relaxation scheme can be written as:
N
5 SWA = mmergad Ppc = (1 - -11-ppo (1 - -w) Σ A.A
(3)
where ÓWAIM is the relaxation change and the subscript pre refers to the pre-trained model and w the relaxation factor that controls how much relaxation is applied (an w of 0.0 would revert the most salient weight to the base weights and an w of 1.0 applies no relaxation), effectively w is scaling error/deviation from the base model, and X internal to each merging algorithm and not part of the AIM relaxation; however, in general, one could selectively apply this if desired. In this work, since we do not explicitly look at the merging algorithm's 's inner workings, we simply fuse the terms Di=1 X into single algorithm-agnoste term Amereed and simplify the relaxation scheme to:
the weight factors for each A, Note that A is
A = Ppe + (1 Apre - )
Note that in general Apr is not a single matrix, rather a mapping of activations for all model parameters obtained as we described prior. In our experiments, we apply this relaxation scheme to several different merging methods and explore how the hyperparameter affects the merged model's behavior, and present these results in Section 5.
Sensitivity- Based Formulation An alternative way of choosing the importance scores for changing the model teiights-hhih has been used, e.g., in continual learning [10, 26], out-of-distribution detection [32], and meta-learning [2]-is to use the partial derivatives of the (base) model with respect to the parameters, i.e., sensitivities, which correspond to a scaled version of the activations. This is in contrast with using the activations directly, common in model compression. Formally, let fo denote the model with parameters 6, and {[j] be one of the weights; then gobs | determines how
formulation of AIM by incorporating sensitivity scores calculated from the gradients and replacing activations with sensitivity scores. We consider fa) be the logits of the model with parameter 8 and be the entropy function, similar to Farajtabar et al. [10]'s suggested approach for classification problems. Now let Gpre = 20
sensitive the output to perturbing the weight O[j]. We develop a sensitivity-
(CC be the magnitude of the gradient for all parameters of the pretrained model for
samples in calibration corpus D. Then the sensitivity-bart -ased formulation can be written as follows:
5

---

0 Ope + Gpee (1 - w)) merged-
In other words, the sensitivity-based formulation of the problem suggests that regularization of weights through relaxation should be done based on model gradients. However, we note that computing gradients for a calibration corpus can be computationally expensive and will require significantly more memory and compute resources than storing activations, which is akin to performing inference on the model. To verify that using activations retains the same performance and fidelity this formulation, we conduct our experiments for both AIM and the sensitivity-based formulation. We present the results of the continual learning view in Appendix C, and we see that when comparing results of AIM (Table 1) and the censitivity-basee formulation (Table 4), the performance boost of both approaches is very similar, with AIM being computationally more efficient.
Experimental Setup and Evaluation Metries
We conduct two separate experiments with AIM: 1) we apply AIM different merging methods including the two latest works on the topic with different numbers of experts being merged and report the performance of the models on different benchmarks; 2) we conduct ablation study on the w parameter AIM and analyze how w affects each of the merging methods in a scenario where different experts are being merged.
4.1 Selection of FT Expert LLMs
To understand how AIM reacts with different merging methods, conduct experiments with merging different experts fine-tuned from the same base model. The set of experts we use is the same set of experts used by the two latest LLM merging algorithms in the literature, namely DARE [46] and
WIDEN [47], which use the same three experts fine-tuned from Lllam-2-133 [38]. These experts
include the WizardLM-I1 M-13B [41] model fine-tuned for instruction following, WizardMath- 13B [24] fine-tuned for superior mathematical reasoning, and llama-2- 13b-code-alpaca, which serves as the code expert [4]. See Appendix A for more details.
4.2 Merging Methods Implementations
In our experiments, we implement the latest merging methods in the literature for LLM merging. These include newly developed DARE and WIDEN [46, 47] methods well some of the long- established approaches of TIES merging [42], and task arithmetic [19]. For all merging methods except WIDEN, we use the comprehensive MergeKit implementations developed by Goddard et al. [15], and for WIDEN, we use the publicly available implementation provided by the authors of the paper.
We note that in many of the merging algorithms, many hyperparameters can be adjusted. In these cases, we use the author-reommeeddd values where available and the default parameters recommended by Goddard et al. [15]. Note that it is possible to perform grid search on these hyperparameters to find optimal values for each benchmark; however, this would essentially be over-fitting on benchmarks and does not provide any value to our analysis of the proposed complementary relaxation scheme, which applies adjustments to the merged models. For reproducibility. all of our checkpoints and code to reproduce the results will be made publicly available.
4.3 Benchmarks Used For Evaluations
Given that the expert models we use in our experiments involve fine-tuning on instruction following, mathematical reasoning, and code generation, tasks. Specifically, we measure model performance on language understanding with the MMLU [16]
several common benchmarks for each of these
benchmark, instruction following with IFEval [48] benchmark, code generation with HumanEval [5]
and MBPP [3] benchmarks. and mathematical reasoning with the MATH [17] and GSM8K [6] benchmarks. For all benchmarks, we use the latest versions and up-to-date implementations developed by Gao al. [14] except for mathematical reasoning, for which used by Luo et al. [24] to replicate the results of the original model as closely as possible. The code
the chain of thought prompting
use for these benchmark results will also be made publicly available for reproducibility.

---

In addition to the common benchmarks that we use to evaluate merged models, we also propose new evaluation metric for LLM merging (or merging of different experts in general), which we believe helps better contextualize the value added by any given merging algorithm and which we discuss in the following section.
4.4 Measuring Performance From an Optimization Perspective
We note that most cases, LLMs are not meant to operate as narrow expert models, unlike large portion of deep learning applications where models are trained to perform very specific tasks such as classification or regression. LLMs, in contrast, are generalist language models aiming to assist across a wide variety of tasks and applications. As such, LLMs can be viewed from multi-objective optimization perspective. In merging scenarios specifically, multiple expert models are brought together to create a merged model aiming to find the balance of performance across the different expertise of the fine-tuned models. In this sense, each expert can be thought of as optimized for specific objective. This perspective lends itself rather well to a multi-objective optimization view of the problem. Given this, only looking at how each model performs on each of the benchmarks does not give us a full picture of the multi-objective goal of merging.
To obtain a more comprehensive view of merging performance, we propose a hypervolume- metric that quantifies the contribution of the merged model to the multi-objective frontier of FT LLMs. Consider a performance space defined over N benchmarks, where each model's performance is represented as a point in an N-dimensional space. The performance on each benchmark is normalized to the range (0, 1]. where represents the worst-case (reference point), and represents the best possible performance with 100% accuracy on the benchmark in question.
Let = (r1 ( TN) denote the reference point in this space, which we set to (0, O..0.... 0) to ensure hypervolume calculations pre-trained base model, let 5 S S denote the subset of models that are Pareto-optimal, i.e., models that are not dominated by any other model in S. The hypervolume of this set, denoted HV(S*), is defined as:
consistently defined. Given set S of FT models and the
HV(S*)= = (ddmxxxx
(6)
XES
where X(-) denotes the Lebesgue measure (i.e., volume in IR,),, and dom r) represents the hyper- volume dominated by x with respect the reference point r.
When a merged model M introduced, the new set becomes S' U {M). and the updated Pareto-ophe Pareto-optimal optimal set is denoted as S Given this set including the merged model, we can measure the added value of the merged model from multi-objective perspective the normalized hypervolume gain (HV Gain) as result of adding this merged model:
HV Gain V - - HV(SS*
(7)
Where is the number of dimensioos/bennhnarkk. Since hypervolume is computed only over Pareto- optimal models, we have HV*S** > HV(S") ensuring that HV Gain 2 0. This metric provides an aggregated measure of merging effectiveness, capturing trade-offs across multiple benchmarks rather than focusing on isolated improvements, thus providing full picture of merging performance. In our experiments, we track HV Gain along with the 6 aforementioned benchmarks.
5 Experiments and Results
In this section, we present our results on applying AIM to different merging methods as well as an ablation study on how the w hyperparameter affects the performance of each merging method. All experiments are run using 4 H100 GPUs, and each of benchmarks takes roughly 15 minutes to run for each checkpoint.
5.1 AIM Applied to Various Merging Approaches
To demonstrate the effectiveness of AIM, we conduct experiments on 5 different merging methods under 4 different scenarios. As mentioned before, we use 3 different FT LLM experts in our experiments. As such, merge models using each merging method for all 4 possible permutations
7

---

34
WITH
24
24
MVLL
NULL
MATH
MargedPartoo
ADS
PTPeedo
Figure 2: The Pareto fronts of models under different scenarios. Note that the points in these plots represent all models benchmarked in Table 1. for better readability, we only visualize the dominating points in each case. The measured increases in HV Gain when AIM is applied can be clearly seen in the Pareto frontier shifting further forward when AIM is applied compared to when only a population of merged models is evaluated.
Table 1: Benchmark Results Across Various Merging Methods. Percentage changes are shown relative to models merged without AIM. The
fine-tuned and base models highlighted in yellow, and the best-performing merged models are marked in blue. The results demonstrate that applying AIM significantly enhances the performance of merged models.
Haruzbral
XINEL
XATH
CSHIK
27177
31.91
52.93
24.30
25.25
Iraneciee Teanl
26:3
14.00
$5481
1.00
43.40
35.7
Mergel Medels
Codz Inatrstion
36.35
35:00
$4.00
53.53
8.50L
5,80
46.20
45.00
32,00
33.45
0.29 (r2.499)
0.2
DAME
Tranutie Teant Mat
Code + Mark
No
Yea
12
15.85
20
5.49
16-66
123
26
202200448400 4265
33,00
52.70
2192))
51.08
51.55
12.0011155
(4.0101999)
66.10
56.30
(4.59)
$1.96
21.91
(1.299)
23.0
12.35
0.23
Inarectio Tannd Mars
Yea
15:85
(656 1996
2740
$3.99
13301156679)
69
122 (39)
33
(5.5511 1755
0.33
45.999)
(4.399)
DARE
Code + Mud
Teamectiee Teasl + Mads
So
23,00
5241 Ea
5143
11:0
(5.00:20195
1400
60000
63.00
56.00
21.10
71.00
33.47
0.36-111
0.33
0,30
Inarectee Yaaod
Yes 51 (e:5 1993
21.20
51.55
EASS (79)
57.0 (IOWES)
$1.50
021 (216 2W953
festrction
Code+ + Mun
No
25.00
52.3
1310
21.00
0.24
Terk Aiilancic
Iearectice Teand + Malh
Yos
No
5.51
20.30
51.55
17:00
10.0
$4.20
31.31
0.3
0L19
Ineructte Yaand
Yes
1983
13.00
Aawieet (79)
$2.00
13 13 (99)
30
25.40
2475
24.5
15
005
Code+ + Man
No
1330
26:90
51.85
$4.00
62.50
21.03
The
Ironectie Teand + Manh
Yes
27
2010
35.00
34.00
9:30 1e
6.30
06 (399
44.30
54
34.04
0.25 E4G 1883
0221
Codz Instrncionn Tanol Mads
Yes
2113 (2.45))
29:20
25:33
54.4
25 98 (3.93)
BIEE
50
25
$4.97
(3.0%)
Madi
3.07
19:40
5335
14.20
40
24.00
0.2
WINNN
Intraciie Tanol a Mads
No 24.50
Yea (3.98/22595
32.00 (4.3
30.40
54,99 8003
5:33
14.00
06.50
30.02
31 (41.299)
0,30
Code Instrntion Tanod Mads
Yes 30 (-7-70))
$3.30
14.0046 A753
5410 (03315)
32 (w41399
8:20 1001
of these expert LLMs. Then we apply AIM to all merged models and measure the performance of each model in all 6 benchmarks. We also report the HV gain for each merged model compared the population of the base model and the models being merged (in cases with 2 models, the population will only include the models used for merging). These results are presented in Table 1. For this experiment, we used w 0.4, which we found to be the best balance of performance among the various merging methods we use. This choice was informed by our analysis in Section 5.2. In Table 1, we have highlighted the gain/loss of performance for each benchmark due to AIM and we can see that in the vast majority of cases, AIM causes a significant performance boost, with an Average Change of 13% (ignoring the Inf value) and more than 40% HV Gain in 20% cases, further highlighted by the fact that the top performers on each benchmark, as well as the largest hypervolume gain, are all in models merged with AIM. We observe HumanEval (10 out of 20) and MBPP (17 out of 20) often large boosts with AIM, especially when merging Instruction Tuned models with others. Some merges also reveal small drops in GSM8K or IFEval even when other benchmarks improve, reflecting the inherent trade-offs in merging specialized models. Overall, clear majority (80% ) of merges exhibit improved HV Gain under AIM, reinforcing that the method often enhances multi-task performance overall. We can further visualize this increase in hypervolume by looking at how AIM pushes the Pareto frontier. Figure 2 shows how applying AIM to existing merging methods extends the Pareto optimal frontier, which we also quantitatively measured using HV gain. These results showcase the
8

---

efficacy of the proposed method across a variety of merging methods and reinforce the hypothesis that the activation space encompasses useful insight for merging.
To validate AIM'S generalizability beyond the 1lama-2 family, we conducted new experiment on a different architecture and modality: the Qwen 2.5-VL.-7H V1--BB vision-language model. We merged two distinct experts, Video-RI model for video reasoning [11] and CAD-Coder model for image-to-code- based CAD geometry generation [9], with their instruct base model.
We evaluated this merge on 6 benchmarks: IFEval and MMLU for instruct base model capabilities, Video MMMU [18] and VSI-Bench [45] for video reasoning, and two CAD-Coder benchmarks for code generation from rendered and real images [9]. As shown in Table 2, AIM consistently improves the underlying merge methods across these diverse tasks and improves the Hypervolume Gain under all four merging methods. This demonstrates that AIM's benefits are not architecture-specffi and generalize effectively to multimodal models.
For merging, therefore, we have two experts, one for CAD and one for video reasoning, with a base model that is instruction-tunat Therefore, to assess performance, we perform the IF-Eval and MMLU benchmarks (the base model knowledge and instruction following) as well as expert benchmarks for video reasoning. Video MMMU [18] and VSI-Bench [45], and CAD generation benchmarks of CAD-Coder benchmark (on rendered CAD images) [9] and CAD-Coder Real benchmark (on real 3D printed images) [9]. As shown in Table below, AIM consistently improves the performance of the underlying merging method across diverse tasks while also showing that the overall hypervolume gain is raised in all merging scenarios, demonstrating AIM's benefits are not limited to the Llama-2 architecture, and AIM does provide a path towards higher quality merging with little computational and data overhead.
Table 2: AIM generalizes across architectures and modalities. We evaluated merging two Qwen 2.5-VL-7B experts (Vidco-RI RI and CADCoder, with an instruction -uned base), AIM consistently improves performance across benchmarks, increasing the multi-task Hypervolume Gain in all cases.
Meebod
AIM
TrCvl
MMLU
Video VSI Base Model
Video MMMU
CAD tes100
CAD R400
IIV Gan
Instruct
0.88
0.67
0.21
0.47
0.94
0.5
Video R1
0.04
0.61
0.38
0.49
0.30
0.03
CADCwder
0.27
0.66
0.00
0.00
0.61
0.32
Merged Models
TIES
No
Yos
(+1.75%))
0.57
0.6%
0.23
0.23
0.45 (69,768)
(42268)
DARF
No
Yes
0.56 (43.7096)
0.07
0.67
(44.0009)
0.25
0:45 (47.1458)
0.4
0.55 (+1 1558)
0.54
0.32
(43.47%)
DARE Task Arithmetic
No
Yes
0.56 (-128.006))
0.25
0.67
0.64
0.26 (e0.000%))
0.25
0.45 (+1.65%)
0.43
0.54
0.54
0.34 (6030388)
0.33
0.45
(-17-42%)
0.38
WIDEN
No
Yos
0135
0.35
0.66
0.66
0.14 (675.008))
0.08
0,32 (623.0186)
0.26
0.64
0.81
(64 4249
0.33
0.39
0.36 (+13.245))
0.32
5.2
Ablation study
To understand the effects of changing w in AIM, we conduct an ablation study on the case of merging all three expert L.Mss. For this study, we apply AIM with w E (0.0, 0.2, 0.4, 0.6, 0.8) and run the benchmarks on each merged model with each value of For brevity, we do not report all benchmark results for each value here; instead, we track the hypervolume gain (The full set of results are presented in Appendix B). Specifically, to better visualize the effect of w, we measure the relative change in HV Gain compared to no AIM (i.e., 3 1.0). We present these results in Figure 3. In most merging methods, we see that decreasing to even O benefits the model performance. However, in TIES merging particularly, we see that decreasing w beyond 0.4 seems to degrade performance, and setting w to the most extreme case of 0.0 does see some degradation in WIDEN as well. Given this, it seems that in these experiments, value of 0.4 balances the performance gains in methods responding well to AIM and the potential degradation of methods that benefit less from AIM. However, given this observation that in some cases pushing w to 0 still yields benefits, there may be some value in exploring non-lineat scaling of activation magnitudes and non-lineat relaxation schemes that could further boost performance in some cases.
To empirically validate the sensitivity to the calibration set, we conducted an ablation study on the calibration set size. We applied AIM to the DARE TIES merge of all three Llama-2 experts from Table 1, varying the number of calibration blocks from to 256. As shown in Figure 4, the results demonstrate that AIM is highly robust. The Hypervolume Gain (HV Gain) increases substantially with just one block and stabilizes by 8 blocks (sourced from the Pile corpus Gao et al. [13]). This confirms that the data overhead for AIM is minimal and the method is highly practical.
9

---

60
Hypervolume Gain
0.24
Belattee HJ Calt (81
50
0.22
30
0.20
20
10-
0.18
0.16
-10
2
2-
2
2
2
2
27
2
0.0
044
0.0
1,0
Calibration Set Size (blocks)
Artthrthmeticc
DARE Tisk Arttmmetic Tics Horging
WIDEN
-a. DARE Ties
Baseline
Figure AIM's Robustness Calibration Set Size. HV-Gain is plotted against number calibration blocks (log scale) for the DARE TIES merge. The dashed line is baseline performance without AIM. Significant improvement is achieved with small data, and performance stabilizes at only 8 blocks.
Figure The Impact the Relaxation Factor w on Merged Model Performance. This figure plots the relative change in HV-Gain compared to scenarios without AIM. The x-axis represents 1 reflecting that decreasing results in more relaxation. The plot indicates that for some tasks, smaller values of continue yield benefits. An 0.4-0.6 appears to strike balance.
6 Conclusion and Outlook
In this work, we introduced Activation-Informed Merging (AIM) as a complementary algorithm to existing model merging techniques for large language models (LLMs). We hypothesized that the activation space of LLMs harbors useful information that is often overlooked in model merging, as most existing methods operate purely the weight space. To explore this potential information in the activation space, we viewed the problem from a continual learning perspective and proposed leveraging the activation space information from fask-agnostic calibration set. This approach selectively preserves critical weights from the pre-trained model, mitigating catastrophic forgetting while incorporating knowledge from fine-tuned models, yielding overall higher- higher-performing models.
Through extensive empirical evaluations across multiple merging methods and benchmark tasks, and model architectures (including the Llama-2 and Qwen-VL families), we demonstrated that AIM consistently improves performance, often yielding superior results in comparison to the original merg- ing methods it was applied to. These results empirically confirm our hypothesis on the importance of the activation space. Notably, AIM boosted merged model performance by up to 40 % in some cases, underscoring the crucial role and the potential of activation information in merging methods. Furthermore, our ablation study confirmed AIM's 's robustness, showing significant gains even with minimal calibration data (as few as 8 blocks), highlighting the method's 's practical applicability. Our findings strongly highlight the necessity and benefit of incorporating activation-inforee strategies when merging multiple fine-tuned models.
Moving forward, our findings open up several promising directions for future research. First, results indicate that even aggressively preserving salient weights across many merging scenarios. This highlights the promise for more advances activation-infomme strategies and non-linear relaxation methods to potentially further enhance performance. Beyond the pre-trained activations explored in this work, there is room to improve existing merging methods by leveraging the broader activation space of the models being merged. So far, AIM has only considered the activations of the pre-trained model, while the activations of the expert LLMS remain unexplored. Future research should focus on developing methods that also encompass information from the expert model activations. Additionally, for incorporating the activation space of LLMs in merging should be developed and tested. These integrations will hold great value in improving the quality and performance of merging methods in an increasingly competitive and ever more efficient landscape of LLMs, which could benefit from smaller and more efficient yet more powerful models.
pre-trained model effective
future works, more theoretically grounded approaches
Overall, AIM serves as robust and adaptable augmentation existing LLM merging techniques, offering a principled way to incorporate activation information for more effective model fusion. By prioritizing the activation-awar generalizable merged models, demonstrated across different architectures and modalities, that better leverage the strengths of multiple fine-tuned experts.
perspective, take step towards more stable, efficient, and
Acknowledgments and Disclosure of Funding
This work was supported in part by the MIT-IBM Watson Al Lab.
10

---

References
[1] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139-154, 2018.
[2] Cesar Almecija, Apoorva Sharma, and Navid Azizan. Uncertainty meta-learning for multimodal task distributions. arXiv preprint arXiv:2210.01881, 2022.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.00773: 077322 2021.
[4] Sahil Chaudhary. Code alpaca: An instruction following https://gitee thub. com/ coom/shiiiie 1110112/0111eeee 1220114/cooeaapaca,, 2023.
foolowigionnfolloiinng Ilama model for code generation.
[5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
[6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14166, 2021.
[7] MohammadReza Davari and Eugene Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks, 2024.
[8] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping, 2020.
[9] Anna C Doris, Md Ferdous Alam, Amin Heyrami Nobari, and Faez Ahmed. Cad-coder: An open-source lisiun-language model for computer aided design code generation. arXiv preprint arXiv:2505. 14646, 2025.
[10] Mchrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pages 7762-3773. PMLR, 2020.
[11] Kaituo Feng, Kaixiong Gong. Bohao Li, Zonghao Guo. Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms. arXiv preprint arXiv:2503.2177 2025.
[12] Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post- training quantization and pruning. Advances in Neural Information Processing Systems, 35: 4475-4488, 2022.
[13] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
[14] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou, A framework for few-shot language model evaluation, 2024.
[15] Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's MergeKit: A toolkit for merging large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 477-485, Miami, Florida, US, 2024. Association for Computational Linguistics.
[16] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0 033000 2020.

---

[17] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
[18] Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501. 13826, 2025.
[19] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
[20] Dwag-Hwan Jang. Sangdoo Yun, and Dongyoon Han. Model stock: All we need is just a few fine-tuned models. In European Conference on Computer Vision, pages 207-223. Springer, 2025.
[21] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietrr and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023.
[22] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3322 2017.
et al.
[23] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and accelcration, 2024.
[24] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2025.
[25] Michael S Matena and Colin A Raffel Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703-17716, 2022.
[26] Youngjae Min, Kwangjun Ahn, and Navid Azizan. One-pass learning via bridging orthogonal gradient descent and recursive least-squares. In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 4720-4725. IEER, 2022.
[27] Aldo Pareja, Nikhil Shivakumar Nayak, Hao Wang, Krishnateja Killamsetty, Shivchander Sudalairaj, Wenlong Zhao, Seungwook Han, Abhishek Bhandwaldar, Guangxuan Xu, Kai Xu, Ligong Han, Luke Inglis, and Akash Srivastava. Unveiling the secret recipe: A guide for supervised fine-tuning small 11ms, 2024.
[28] Alexandre Ramé, Kartik Ahuja, Jianyu Zhang. Matthieu Cord, Léon Bottou, and David Lopez- Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. In International Conference on Machine Learning, pages 28656-28679. PMLR, 2023.
[29] Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi-Tazehhe
Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the benefits of weight averaged reward models. In Proceedings of the 41st International Conference on Machine Learning. pages 42043342073. PMLR, 2024.
[30] Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. Warp: On the benefits of weight averaged rewarded policies, 2024.
[31] Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, 31, 2018.
[32] Apoorva Sharma, Navid Azizan, and Marco Pavone. Sketching curvature for efficient out-of- distribution detection for deep neural networks. In Uncertainty in artificial intelligence, pages 1958.1967. PMLR, 2021.
12

---

[33] Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang. Sayna Ebrahimi, and Hao Wang. Continual learning of large language models; A comprehensive survey, arXiv preprini arXiv:2404. 16789, 2024.
[34] Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th Annual Conference Computer Graphies and Interactive Techniques, page 245-254, New York, NY, USA, 1985. Association for Computing Machinery.
[35] Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja. Kai Xu, David D. Cox, and Akash Srivastava. Lab: Large-scale alignment for chatbots, 2024.
[36] Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models, 2024.
[37] Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. An empirical study of multimodal model merging. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[38]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams. Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[39] Liyun Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5362-5388, 2024.
[40] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Lontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965-239922 PMLR, 2022.
[41] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024.
[42] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models, 2023.
[43] Prateck Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging at scale?, 2024.
[44] Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang. Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in lmms, milms, and beyond: Methods, theories, applications and opportu- nities, 2024.
[45] Jihan Yang. Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10632-10643,, 2025.
[46] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning. PMLR, 2024.
13

---

[47] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Extend model merging from fine-tuned to pre-trained large language models via weight disentanglement. arXiv preprint 0rXiv:2408.03099, 2024.
[48] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. cestruction-folloiinn evaluation for large language models, 2023.
14

---

A Reproducibility Details
Here provide the specifie details for reproducing the results presented in the paper.
Checkpoints: Firstly, we specify the publicly available checkpoints we use in our experiments.
Below is list of checkpoints used and the links to the publicly available weights for these models:
Base Model:
http://ugggiigffccccoooo/d////mm
Code Model: http:///iinnnooccccoooooo/o///ee
Math Model: http://wwgggiaaccc.ooommolooooos
Instruction Tuned Model: http://hwgggggggoooo//////a/aaaa
The weights we use in our experiments may not be exactly identical to the weights used in the experiments by Yu et al. [46] and Yu et al. [47], since the referenced weights for WizardMath-13B are no longer available publicly, instead we use a publicly available copy of the
Note On Weights:
model.
Code and Data: Aside from the checkpoint, we provide our code and the link the publicly available calibration data we use in our work. Our code is publicly available at and the calibration data can be found http://ingggnniicccoouuusussssii vackbu. back.
15

---

B Ablation Detailed Results
Here present the full results of the ablation study conducted. Table 3 includes the granular values for all benchmarks we ran for different values of w.
Table 3: Performance metrics for different methods with varying w values.
Method
HumanEval
MBPP
MMLU
MATH
GSM8K
IFEval
HV Gain
19.51
29.0
54.32
26.24
0.1015
28.6
54.35
25.37
0.1069
20.73
29.2
54.46
25.98
0.1143
Ties Merging
20.12
27.8
54.25
0.1155
0.8
20.73
27.8
54.09
33.7
24.07
0.1131
1.0
21.34
29.2
53.97
26.95
0.1096
0.0
18.90
29.4
53.42
13.8
60.5
35.49
0.2623
0.2
16.46
29.0
52.98
12.9
61.5
34.97
0.2434
0.4
15.85
27.0
52.59
12.2
60.7
33.67
0.2257
DARE Task Arithmetic
0.6
15.24
27.0
$2.18
11.8
58.1
33.95
0.2142
0.8
14.02
22.2
51.53
9.9
54.0
32.69
0.1828
1.0
11.59
19.6
50.89
9.1
49.7
33.2
O. 1604
0.0
25.61
29.6
53.31
12.0
59.4
34.64
0.2669
0.2
21.34
29.4
53.11
12.1
58.7
36.76
0.2590
0.4
19.51
28.6
52.63
11.6
57.0
36.2
0.2426
DARE Ties
0.6
15.85
26.0
52.22
10.1
54.2
34.82
0.2019
0.8
14.02
24.0
51.6
10.0
53.1
35,51
1,19
1.0
13.41
21.2
51.15
8.7
51.5
35.75
0.1717
0.0
18.90
29.4
53.42
13.8
60.5
35,49
0.2623
0.2
15.24
27.6
52.97
13.0
59.8
35,27
0.2296
0.4
15.24
27.4
52.63
12.0
58.1
33.88
0.2165
Task Arithmetic
0.6
15.24
25.4
52.13
11.4
$7.4
33.29
0.2069
0.8
13.41
21.8
51.61
10.2
S6.2
32.74
0.1862
1.0
11.59
19.6
51.20
9.0
52.7
32.95
0. 1614
27.44
33.2
14.0
64.9
32.39
0.3027
28.05
33.0
55.16
14.2
65.6
32.39
0.3066
32.8
54.98
32.76
0.3013
WIDEN
33.4
54.77
32.06
0.2947
26.22
32.6
54.64
31.74
0.2941
1.0
25.00
33.2
31.44
0.2879
16

---

C Sensitivity-Basee Formulation Comparison
In this section, present the results of running post-merging relaxation using the gradient-based sensitivity-basee formulation discussed in the main body. We run the relaxation scheme based on gradients with across benchmarks with gradient-based relaxation. We observe that the resulting performance boost remains very close to activation based relaxation, with HV gain largely unchanged. with both gradient-based and pure activation-based boost trading blows evenly (see Table 1). Noting this and given that activations alone do not come with significant memory and computational cost, activation-basse based relaxation, which does not require computing gradients, is more memory-efftciemt and computationally inexpensive process.
0.4, which we use in AIM. Table 4 shows how performance changes
Table 4: Model performance comparison across different benchmarks, after relaxation applied using the gradients of the models with respect to the calibration data.
Bax Mecke
leamntee Teanl
76:33
7:0
41:0
25.00
Code Instreciio Texol
No
1531
36.00
To
2.00
4530
32.00
BYY
DARU Aorrae
Codc Mafs
Yo-
15.5 (33719)
18:66
25:60 (63 5000
52 10
$1.6
16.00-1999)
66.00 (4L339)
21.01
23.00
(23009)
0.23 60600)
6.73
Instaciiin
Yoa
12.30
(-122222)
23 (44-4224
12000
31.00 (-12299)
0.26 (4407719)
Instrcciio
No
20
$5.00
Inoracice Taxnd Mon
You
15:4 (+8
X334
11.00
0993
34.0
15 1963
0.26
DARE Tio
Code Iourcctiee Teead
You
10.00
$400 149
K:65
47
1 (69)
33
33.661000 (3399)
0:29
Yos
17:68 (80558
1820
62 (000TR
21.00-55005
0.24 (45000R)
Instrnciion
No
1341
21.30
SU.S
$1.50
25.15
Cade Teamectree Teral
You
29 68 (2.099)
17.4
35.0 0069
SL1D 2763
7 3099
4.10
33.00 (1 506)
0:28
Tak Anthasic
Iearenice Thand
Yo
8.54 (+100 (069)
19:20
26
(-30
$3.8 2583555500
25.00
13.00
(45 00%)
61.00 (3 1060
6420
33.62 1869
21.92
0.24
Cade feamention Terra Murk
No
$3.00
$1.11
13.00
$2.00
Cade Mark
Yo
Noi
19335
1593
$2.20
1530 1830005099)
80
22.64 (4.66)
0.3 (1) (045
I03T
The Magicz
Tearaniin Taand
No
Yo
27.44 17.44-22.194) 1790
18:06
35.0 109)
2180
$4.5
9:30
46
10
540
34.51 (11.369)
26.18
0:25 (6335))
23,67
Code Testectice Terel Mar
Yo
No
33.50
24 30
54
6:35
(1.89)
24.00
26
Intracice Vaaad Math
23:78 (1.02.00
24.30
32.00 (46
$1.00 (41
1400
08 10l
51.33 5353
0:51 S00
WIDIN
Mank
Iestrctiio Tecal
76.33
13,07
20.60 (0.85)
$3.50 (0002%)
14.00 14.20040000)
23.05 (0.299
W1.17
0.24 24:212299
Cole festrccice Texol + Muk
No
26.83-4..00 (3.39)
75.00
33,90
11.30
$1.50
13.50
61.30
(4.319)
11.11
(14.399)
0,50
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: claims made by the paper are quantitative and justified by experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: experiments clearly state outcomes and performance, and future work needed is discussed.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: [NA]
18

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: details on training procedures and experiments are made clear in the appendix and main body. Code also provided for exact replication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel urchitecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
19

---

Justification: code is provided to replicate the results, and the public checkpoints used are clearly mentioned.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
quiides/ooeeSumiie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nips. cctpccppppccccccccc guwdee/ie/eeee
guudee/Codeeuuuiie Coo///ooeeeeeeeeeeeee sesiisssnsnnn (iionnolccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Experiments and what each benchmark measures and how is measured are made clear.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Experiments in these settings not accompanied by error bars, since single model is used to run benchmarks and not stochastic process that would involve differing results across runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We indicate the type of hardware used benchmark experiments. No training is needed so that is not mentioned.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https : http///eerrp...p/ cttpiisss..ppppue cec//ublic//eee Cettic///eheeeee csQuidel ines?
Answer: [Yes]
Justification: we have verified adherence the code of conduct.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No relevant socictal impacts unique to this paper (based on existing public
LLMs).
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: based already public models applicable.
Guidelines: This paper does not contain such risks.
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All sources of external data and model are mentioned.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcone thcode. com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
22

---

13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Link anonymized code is made available.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: no crowd-sourcing is involved in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: no crowdsourcing or human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
23

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https / /ttp///eerppp//peee /c///oofeeeeccce//22/////1/ for what should or should not be described.
24


Reviews:
[{"review_id": "HRh2hQqAlK", "reviewer": "Reviewer_Ldwv", "summary": "This paper introduces Activation-Informed Merging (AIM) for merging fine-tuned LLMs derived from the same pre-trained base. AIM enhances model merging by leveraging the activation space to guide which weights should be preserved during merging. The method aims to prevent performance degradation, particularly the \"catastrophic forgetting\" of the base model’s general abilities, by constraining updates to the most salient (activation-sensitive) weights. The authors conduct empirical evaluations across multiple benchmarks (math, code, instruction following) and demonstrate that AIM consistently boosts the performance of merged models, sometimes by as much as 40% in benchmark metrics.", "questions": "1. How does AIM perform on other model families (GPT, Mistral) or different model sizes? The evaluation is limited to Llama-2-13B.\n2. While you cite robustness from other work, have you empirically validated sensitivity to calibration dataset choice in the merging context?\n3. Have you considered other importance metrics beyond activation magnitude (e.g., gradient-based measures, Fisher information)?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-grounded in the literature, thoughtfully connecting AIM to both continual learning (notably, weight regularization to prevent catastrophic forgetting) and recent advances in model compression (activation-aware quantization).\n2. Experiments cover five recent merging methods, three types of fine-tuned LLM experts (math, code, instruction), and six task benchmarks. The performance is measured both in standard metrics and a new hypervolume-based multi-task evaluation.\n\nWeaknesses:\n1. The need for access to activations and a representative calibration set may limit the method’s immediate applicability in proprietary or privacy-sensitive domains.\n2. The novelty is somewhat incremental: AIM is an adaptation of activation-based ideas from compression to the merging context. This is clearly articulated, though.\n3. Only tested on one model family and size (Llama-2-13B).", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer Ldwv for the thoughtful review and for recognizing that our work is well-grounded in the literature. We hope to address all of the points raised by the reviewer and hope that this discussion and the addition of two new experiments will convince the reviewer to increase their score.\n\n## Applicability of the Publicly Sourced Calibration Set\nThe calibration data is sourced from a small, random sample of the Pile [1], a large-scale, diverse, and publicly available text dataset created specifically for training large language models. The base models we use have already been trained on public data like this, so we are not introducing any new, private information. While the data used to fine-tune expert models may be proprietary, the pre-training data for most well-known model families (Llama, Qwen, etc.) is public, and specifically, the calibration set we propose to use is publicly available. In addition, as our experiments in response to the reviewer’s Q2 demonstrate, our method still offers performance boosts even with a very small amount of calibration data and is therefore remarkably robust.\n\n## Experimentally Validating Robustness to Calibration Dataset \nTo empirically validate the sensitivity to the calibration set, we conducted a new experiment analyzing the impact of its size on AIM's performance. To do this, we conduct an experiment on the application of AIM on the merging of all three experts in our main experiment using the DARE TIES method. We apply AIM to this merging configuration with different calibration dataset sizes and present these results here. Our results show that AIM is robust, delivering significant performance gains even with a very small number of calibration samples (as few as 8 blocks (512 tokens each) from the Pile [1] corpus). This demonstrates that the overhead of sourcing a calibration set is minimal and the method is highly practical. We will add these findings to the final manuscript.\n\n**Table: Hypervolume Gain vs Calibration Set Size (DARE TIES Merging of All Three Experts)**\n| Calibration Set Size (Blocks) | Hypervolume Gain |\n| --- | --- |\n| 0 (Without AIM) | 0.1717 |\n| 1 | 0.2265 |\n| 2 | 0.2373 |\n| 4 | 0.2348 |\n| 8 | 0.2427 |\n| 16 | 0.2375 |\n| 32 | 0.2424 |\n| 64 | 0.2385 |\n| 128 | 0.2370 |\n| 256 | 0.2433 |\n\nThese results clearly showcase that even with as few as 8 blocks, the performance boost of AIM stabilizes, and more data will not be necessary, thus showcasing how AIM can be a practical framework for improving merging quality with robustness to calibration data size, which as we discuss in the paper is expected and observed in prior activation based frameworks for compression such AWQ and WANDA.\n\n## AIM as a Novel Framework for Activation-Aware Merging\n\nThe reviewer's observation that our work is taking inspiration from recent advances in model compression is valid; however, we believe that AIM is not lacking novelty because of this, and we argue that AIM is beyond incremental. We believe the core contribution of our paper is in successfully bridging these activation-based techniques to the model merging domain, which is novel and previously not explored in merging frameworks. We believe demonstrating the importance and effectiveness of including activations in the merging process, and its effectiveness at addressing catastrophic forgetting, which has been a key challenge in merging and has not been adequately addressed by prior methods, and the development of the overall algorithm, bridging continual learning principles and the lessons learned from model compression, goes beyond simple incrementation and highlights both a new finding regarding the importance of the activation space in merging, and a methods do such an integration. We hope this convinces the reviewer that AIM offers more than a simple plug-and-play adaptation and provides significant insight into model merging and a path for more effective merging.\n\n## Going Beyond A Single Family of Models (New Experiments With Different Family/Type Of Architectures)\n\nWe thank the reviewer for bringing this limitation in our experiments to our attention and helping us improve the paper and showcase AIM's effectiveness as a general framework for merging by showing its versatility across different architectures and families of models.\nTo demonstrate AIM's broader applicability, we conducted a new set of experiments on a different model family, Qwen. Moreover, to further highlight the effectiveness of AIM, we look at not only a different language model but a vision language model, the Qwen 2.5 VL family. To assess AIM in this setup, we merged the Video-R1 model, a video reasoning model [2] finetuned from  Qwen2.5-VL-7B-Instruct, and the CAD-Coder model, also trained on the same architecture for image-to-code-based CAD geometry generation [3]. For merging, therefore, we have two experts, one for CAD and one for video reasoning, with a base model that is instruction-tuned.  Therefore, to assess performance, we perform the IF-Eval and MMLU benchmarks (the base model knowledge and instruction following) as well as expert benchmarks for video reasoning, Video MMMU[4] and VSI-Bench[5], and CAD generation benchmarks of CAD-Coder benchmark (on rendered CAD images)[3] and CAD-Coder Real benchmark (on real 3D printed images)[3].  As shown in the table below, which we will include in the final paper, AIM consistently improves the performance of the underlying merging method across diverse tasks while also showing that the overall hypervolume gain is raised in all merging scenarios. This confirms that AIM's benefits are not limited to the Llama-2 architecture, and AIM does truly provide a path towards higher quality merging with little computational and data overhead. We hope the addition of this experiment addresses this major point brought forth by the reviewer and will convince the reviewer to raise their score.\n\n| Method | AIM | IFEval | MMLU | VSI-Bench (Mean Relative Accucary) | Video MMMU | CAD-Coder | CAD-Coder Real | HV Gain |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Instruct | - | **0.685** | **0.667** | 0.210 | 0.474 | 0.040 | 0.049 | - |\n| Video-R1 | - | 0.643 | 0.610 | **0.383** | **0.492** | 0.000 | 0.025 | - |\n| CADCoder | - | 0.268 | 0.657 | 0.000 | 0.002 | **0.614** | **0.324** | - |\n| TIES | No | 0.571 | **0.677** | 0.230 | 0.406 | 0.610 | 0.324 | 0.438 |\n|  | Yes | **0.577** (+0.93%) | 0.675 (-0.19%) | 0.225 (-2.24%) | **0.454** (+12.05%) | **0.647** (+6.01%) | 0.317 (-2.28%) | 0.448 (+2.26%) |\n| DARE  TIES | No | 0.540 | 0.669 | 0.246 | 0.419 | 0.537 | 0.318 | 0.430 |\n|  | Yes | 0.559 (+3.49%) | 0.670 (+0.23%) | **0.264** (+7.21%) | 0.447 (+6.63%) | 0.551 (+2.64%) | 0.321 (+0.73%) | 0.445 (+3.47%) |\n| DARE Task Arithmetic | No | 0.246 | 0.645 | 0.252 | 0.429 | 0.542 | 0.326 | 0.380 |\n|  | Yes | 0.559 (+127.07%) | 0.670 (+3.96%) | 0.258 (+2.37%) | 0.452 (+5.44%) | 0.542 (-0.13%) | **0.336** (+2.94%) | 0.446 (+17.42%) |\n| WIDEN | No | 0.349 | 0.661 | 0.084 | 0.261 | 0.611 | 0.333 | 0.318 |\n|  | Yes | 0.349 (+0.00%) | 0.661 (+0.00%) | 0.138 (+63.57%) | 0.320 (+22.55%) | 0.642 (+4.92%) | 0.334 (+0.23%) | 0.360 (+13.24%) |\n\n## Gradient-Based Measures As An Importance Metric Yields Similar Performance Gains As The Lower Cost AIM\n\nThe point on the use of a different importance metric, specifically gradients, is an excellent one. We completely agree with the reviewer that a look at such a metric would provide great insight into AIM's effectiveness and its ties with continual learning principles. In fact the paper already discusses this nuance (starting at Line 219 of the paper), and we have already conducted experiments on the use of gradients. We investigate gradient-based importance metrics as an alternative to activation magnitudes in section 3.4, where we include a sensitivity-based Formulation for the alternative way we performed relaxation. In Appendix C, we detailed the experimental evaluation of this approach, and as discussed in the paper, our findings show that while gradient-based methods are a valid alternative, they do not yield a noticeably better performance boost than the computationally less expensive and simpler activation-scale-based approach in AIM. We observed that the use of gradients yields results nearly identical to AIM, and thus concluded that the much cheaper and simpler activations are overall a superior balance. We thank the reviewer for their insightful comment, which confirms the importance of this comparison that we have included in the paper, and hope that, in light of this, the reviewer will consider raising their score.\n\n[1] Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., ... & Leahy, S. (2020). The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.\n\n[2] Feng, K., Gong, K., Li, B., Guo, Z., Wang, Y., Peng, T., … Yue, X. (2025). Video-R1: Reinforcing Video Reasoning in MLLMs. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2503.21776.\n\n[3] Doris, A. C., Alam, M. F., Nobari, A. H., & Ahmed, F. (2025). CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generation. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2505.14646.\n\n[4] Hu, K., Wu, P., Pu, F., Xiao, W., Zhang, Y., Yue, X., … Liu, Z. (2025). Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2501.13826\n\n[5] Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025). Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2412.14171"}, {"review_id": "r5OuRbIaIT", "reviewer": "Reviewer_ue55", "summary": "This paper introduces a model merging approach to enhance large language model (LLM) merging by integrating activation-space information. Unlike traditional methods focused solely on weight space, the proposed method leverages a task-agnostic calibration set and principles from continual learning to selectively preserve critical base-model weights, mitigating catastrophic forgetting while integrating fine-tuned knowledge. Empirical results demonstrate that AIM consistently improves merged model performance across benchmarks, achieving up to a 40% increase in task accuracy.", "questions": "In Figure 3​, the experiments on varying ω show HV gains even when ω=0 in most cases, which is intriguing. However, the Ties Merging​ scenario shows a decline. What could explain this discrepancy?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths\n- The paper is well-written and logically structured, making it easy to follow the research flow.\n- Unlike previous merging models that relied on simple weighted averaging in a defined space, this work proposes quantization and pruning methodologies and demonstrates their significance.\n- While calibration datasets are required, the authors used only a small set of 256 samples, showing that the method is easily applicable.\n- Extensive experiments across multiple merging scenarios were conducted to analyze the gains from applying AIM, and most cases showed positive performance improvements. Notably, Table 1​ (MMLU benchmark) demonstrates superior performance across all scenarios, and Figure 2​ shows a Pareto-efficient upward trend, both validating the proposed method’s effectiveness.\n\n### Weaknesses\n- Although Section 3.3​ claims that 256 calibration samples are sufficient, it remains unclear how the method would perform if the calibration data is biased. Experimental validation of robustness with varying calibration dataset sizes would strengthen the claims.\n- The merging experiments were limited to models based on Llama-2-13B​. It would be beneficial to test the approach on smaller models or those fine-tuned from different base models to verify generalizability.", "was_revised": true, "final_justification": "After reviewing the rebuttal, my previous concerns and issues have been resolved, leading me to adjust my score to accept. This decision is based on two key factors: (1) the robustness of the method, which is applicable even with very small calibration datasets and (2) the generality of the method, as it can be applied regardless of the model architecture.", "rebuttal": "We thank Reviewer ue55 for the positive review, and we are pleased that the reviewer found our paper well-written and our method well-validated with significant contributions. We hope to address the weaknesses pointed out by the reviewer in detail and with new experiments, and hope that, in light of this response, the reviewer will consider raising their score.\n\n## Experimentally Validating Robustness to Calibration Data Size\nWe thank the reviewer for raising this important question. To address it, we have performed a new ablation study on the size of the calibration set. Our findings confirm our initial claim and show that AIM is highly robust to the size of the calibration set. Significant hypervolume gains are observed even with a very small, task-agnostic dataset (as few as 8 blocks, 512 tokens each, from the PILE [1] corpus). This low data requirement validates the robustness of our method to calibration data size, which makes it easily applicable and a versatile approach for improved model merging. We plan to include these results in the final manuscript. To do this, we conduct an experiment on the application of AIM on the merging of all three experts in our main experiment using the DARE TIES method. We apply AIM to this merging configuration with different calibration dataset sizes and present these results here.\n\n**Table: Hypervolume Gain vs Calibration Set Size (DARE TIES Merging of All Three Experts)**\n| Calibration Set Size (Blocks) | Hypervolume Gain |\n| --- | --- |\n| 0 (Without AIM) | 0.1717 |\n| 1 | 0.2265 |\n| 2 | 0.2373 |\n| 4 | 0.2348 |\n| 8 | 0.2427 |\n| 16 | 0.2375 |\n| 32 | 0.2424 |\n| 64 | 0.2385 |\n| 128 | 0.2370 |\n| 256 | 0.2433 |\n\nThese results clearly showcase that even with as few as 8 blocks (512 tokens each), the performance boost of AIM stabilizes and demonstrably validates our claims on robustness to calibration data size.\n\n## Going Beyond a Single Family of Models (New Experiments With Different Family/Type of Architectures)\nWe thank the reviewer for pointing out the lack of further experiments on different architectures and hope to overcome this issue by introducing a new set of experiments, which hopefully, demonstrate AIM's generalizability. To demonstrate AIM's generalizability better, we conducted a new set of experiments on a different model family, Qwen. Moreover, to further highlight the effectiveness of AIM, we look at not only a different language model but a vision language model, the Qwen 2.5 VL family. To assess AIM in this setup, we merged the Video-R1 model, a video reasoning model [2] finetuned from  Qwen2.5-VL-7B-Instruct, and the CAD-Coder model, also trained on the same architecture for image-to-code-based CAD geometry generation [3]. For merging, therefore, we have two experts, one for CAD and one for video reasoning, with a base model that is instruction-tuned.  Therefore, to assess performance, we perform the IF-Eval and MMLU benchmarks (the base model knowledge and instruction following) as well as expert benchmarks for video reasoning, Video MMMU[4] and VSI-Bench[5], and CAD generation benchmarks of CAD-Coder benchmark (on rendered CAD images)[3] and CAD-Coder Real benchmark (on real 3D printed images)[3].  As shown in the table below, which we will include in the final paper, AIM consistently improves the performance of the underlying merging method across diverse tasks while also showing that the overall hypervolume gain is raised in all merging scenarios. This confirms that AIM's benefits are not limited to the Llama-2 architecture, and AIM does truly provide a path towards higher quality merging with little computational and data overhead. We hope the addition of this experiment addresses this major point on generalizability brought forth by the reviewer and will convince the reviewer to raise their score.\n\n| Method | AIM | IFEval | MMLU | VSI-Bench (Mean Relative Accucary) | Video MMMU | CAD-Coder | CAD-Coder Real | HV Gain |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Instruct | - | **0.685** | **0.667** | 0.210 | 0.474 | 0.040 | 0.049 | - |\n| Video-R1 | - | 0.643 | 0.610 | **0.383** | **0.492** | 0.000 | 0.025 | - |\n| CADCoder | - | 0.268 | 0.657 | 0.000 | 0.002 | **0.614** | **0.324** | - |\n| TIES | No | 0.571 | **0.677** | 0.230 | 0.406 | 0.610 | 0.324 | 0.438 |\n|  | Yes | **0.577** (+0.93%) | 0.675 (-0.19%) | 0.225 (-2.24%) | **0.454** (+12.05%) | **0.647** (+6.01%) | 0.317 (-2.28%) | 0.448 (+2.26%) |\n| DARE  TIES | No | 0.540 | 0.669 | 0.246 | 0.419 | 0.537 | 0.318 | 0.430 |\n|  | Yes | 0.559 (+3.49%) | 0.670 (+0.23%) | **0.264** (+7.21%) | 0.447 (+6.63%) | 0.551 (+2.64%) | 0.321 (+0.73%) | 0.445 (+3.47%) |\n| DARE Task Arithmetic | No | 0.246 | 0.645 | 0.252 | 0.429 | 0.542 | 0.326 | 0.380 |\n|  | Yes | 0.559 (+127.07%) | 0.670 (+3.96%) | 0.258 (+2.37%) | 0.452 (+5.44%) | 0.542 (-0.13%) | **0.336** (+2.94%) | 0.446 (+17.42%) |\n| WIDEN | No | 0.349 | 0.661 | 0.084 | 0.261 | 0.611 | 0.333 | 0.318 |\n|  | Yes | 0.349 (+0.00%) | 0.661 (+0.00%) | 0.138 (+63.57%) | 0.320 (+22.55%) | 0.642 (+4.92%) | 0.334 (+0.23%) | 0.360 (+13.24%) |\n\n\n## On The Ablation Study And The Effects of $\\omega$\nThe reviewer's observation is an interesting one. Although we do not have a precise idea on why this difference in behaviour is observed with $\\omega$ on different merging methods, we speculate this is due to an interaction with the highly non-linear Ties-Merging algorithm. Ties-Merging resolves sign conflicts between parameters, and our method's approximations, when applied without the protective ω > 0 regularization, may interfere with this sensitive mechanism and thus lead to reduced performance. We hope this brief discussion helps the reviewer grasp our understanding of this observation.\n\n\n[1] Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., ... & Leahy, S. (2020). The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.\n\n[2] Feng, K., Gong, K., Li, B., Guo, Z., Wang, Y., Peng, T., … Yue, X. (2025). Video-R1: Reinforcing Video Reasoning in MLLMs. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2503.21776.\n\n[3] Doris, A. C., Alam, M. F., Nobari, A. H., & Ahmed, F. (2025). CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generation. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2505.14646.\n\n[4] Hu, K., Wu, P., Pu, F., Xiao, W., Zhang, Y., Yue, X., … Liu, Z. (2025). Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2501.13826.\n\n[5] Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025). Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2412.14171"}, {"review_id": "tVXekGAmQ4", "reviewer": "Reviewer_nmzP", "summary": "This paper introduces Activation-Informed Merging (AIM), a complementary technique for combining multiple fine-tuned Large Language Models (LLMs) that originate from the same base model. Drawing from continual learning principles to avoid catastrophic forgetting, AIM uses a task-agnostic calibration dataset to identify and protect the most critical weights of the base model by analyzing its activation space. The authors demonstrate that applying AIM to existing merging methods significantly enhances performance across various benchmarks by up to 40%. The work also proposes a new evaluation metric, hypervolume gain, to assess the multi-objective improvements of merged models.", "questions": "1. Your experiments provide evidence for AIM's effectiveness, but they are exclusively focused on models fine-tuned from the Llama-2-13b architecture. Could you comment on why you expect these results to generalize to other model architectures (e.g., Mistral, Gemma) and scales?\n2. Have you considered how activation information from the fine-tuned expert models themselves could be used to inform the merge? For instance, how would you handle a scenario where the same neurons are salient for different tasks in conflicting ways?\n3. How sensitive is the resulting performance of a merged model to the domain and quality of the calibration data? For example, how would AIM perform if the calibration set was highly domain-specific (e.g., only legal or medical texts) and misaligned with the general pre-training distribution?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The paper combines ideas from model compression (activation-aware quantization) and continual learning, and applied to model merging. This offers a fresh perspective in a field dominated by weight-space-only methods.\n- The proposed AIM method demonstrates performance gains, with improvements of up to 40% reported on standard benchmarks\n- Introduces an evaluation metric, hypervolume gain, which captures the multi-objective nature of merging.\n\n## Weaknesses\n- All experiments are conducted on a single model family (Llama-2-13b). The paper does not show if the findings generalize to other model architectures or scales\n- The method relies on a calibration dataset, adding an extra step not required by pure weight-space methods. While claimed to be robust, this is still an added dependency.", "was_revised": true, "final_justification": "After rebuttal, I am happy to see new results to show that AIM work for other model families. And since I decide to raise my score.", "rebuttal": "We thank Reviewer nmzP for their constructive feedback. To directly address the valid points raised regarding generalizability and data dependency, we have conducted new experiments and are providing further clarification on our methodology below.\n\n## Validating Generalizability Across Architectures and Modalities\nTo address the reviewer's valid point on generalization, we conducted new experiments on the Qwen 2.5 VL family, a different architecture (QWEN), with a different number of parameters (7B), and a different modality (vision-language) from the models in our original paper. To assess AIM in this setup, we merged the Video-R1 model, a video reasoning model [2] finetuned from  Qwen2.5-VL-7B-Instruct, and the CAD-Coder model, also trained on the same architecture for image-to-code-based CAD geometry generation [3]. For merging, therefore, we have two experts, one for CAD and one for video reasoning, with a base model that is instruction-tuned.  Therefore, to assess performance, we perform the IF-Eval and MMLU benchmarks (the base model knowledge and instruction following) as well as expert benchmarks for video reasoning, Video MMMU[4] and VSI-Bench[5], and CAD generation benchmarks of CAD-Coder benchmark (on rendered CAD images)[3] and CAD-Coder Real benchmark (on real 3D printed images)[3].  As shown in the table below, which we will include in the final paper, AIM consistently improves the performance of the underlying merging method across diverse tasks while also showing that the overall hypervolume gain is raised in all merging scenarios. This confirms that AIM's benefits are not limited to the Llama-2 architecture, and AIM does truly provide a path towards higher quality merging with little computational and data overhead. \n\n| Method | AIM | IFEval | MMLU | VSI-Bench (Mean Relative Accucary) | Video MMMU | CAD-Coder | CAD-Coder Real | HV Gain |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Instruct | - | **0.685** | **0.667** | 0.210 | 0.474 | 0.040 | 0.049 | - |\n| Video-R1 | - | 0.643 | 0.610 | **0.383** | **0.492** | 0.000 | 0.025 | - |\n| CADCoder | - | 0.268 | 0.657 | 0.000 | 0.002 | **0.614** | **0.324** | - |\n| TIES | No | 0.571 | **0.677** | 0.230 | 0.406 | 0.610 | 0.324 | 0.438 |\n|  | Yes | **0.577** (+0.93%) | 0.675 (-0.19%) | 0.225 (-2.24%) | **0.454** (+12.05%) | **0.647** (+6.01%) | 0.317 (-2.28%) | 0.448 (+2.26%) |\n| DARE  TIES | No | 0.540 | 0.669 | 0.246 | 0.419 | 0.537 | 0.318 | 0.430 |\n|  | Yes | 0.559 (+3.49%) | 0.670 (+0.23%) | **0.264** (+7.21%) | 0.447 (+6.63%) | 0.551 (+2.64%) | 0.321 (+0.73%) | 0.445 (+3.47%) |\n| DARE Task Arithmetic | No | 0.246 | 0.645 | 0.252 | 0.429 | 0.542 | 0.326 | 0.380 |\n|  | Yes | 0.559 (+127.07%) | 0.670 (+3.96%) | 0.258 (+2.37%) | 0.452 (+5.44%) | 0.542 (-0.13%) | **0.336** (+2.94%) | 0.446 (+17.42%) |\n| WIDEN | No | 0.349 | 0.661 | 0.084 | 0.261 | 0.611 | 0.333 | 0.318 |\n|  | Yes | 0.349 (+0.00%) | 0.661 (+0.00%) | 0.138 (+63.57%) | 0.320 (+22.55%) | 0.642 (+4.92%) | 0.334 (+0.23%) | 0.360 (+13.24%) |\n\n## Validating the Robustness of the Calibration Method\n\nWe thank the reviewer for raising the question on the dependency and sensitivity of using calibration data. To quantify the impact of this \"extra step,\" we performed an ablation study on calibration set size. Our findings confirm our initial claim and show that AIM is highly robust to the size of the calibration set. Significant hypervolume gains are observed even with a very small, task-agnostic dataset (as few as 8 blocks, 512 tokens each, from the PILE [1] corpus). This low data requirement validates the robustness of our method to calibration data size, which makes it easily applicable and a versatile approach for improved model merging. We plan to include these results in the final manuscript. To do this, we conduct an experiment on the application of AIM on the merging of all three experts in our main experiment using the DARE TIES method. We apply AIM to this merging configuration with different calibration dataset sizes and present these results here.\n\n**Table: Hypervolume Gain vs Calibration Set Size (DARE TIES Merging of All Three Experts)**\n| Calibration Set Size (Blocks) | Hypervolume Gain |\n| --- | --- |\n| 0 (Without AIM) | 0.1717 |\n| 1 | 0.2265 |\n| 2 | 0.2373 |\n| 4 | 0.2348 |\n| 8 | 0.2427 |\n| 16 | 0.2375 |\n| 32 | 0.2424 |\n| 64 | 0.2385 |\n| 128 | 0.2370 |\n| 256 | 0.2433 |\n\nThese results clearly showcase that even with as few as 8 blocks (512 tokens each), the performance boost of AIM stabilizes and demonstrably validates our claims on robustness to calibration data size.\n\n## Regarding Activations from Expert Models\n\nOur method, AIM, was designed to use the base model's activations to preserve foundational knowledge during a merge. The positive results confirm our core thesis: there are unrealized gains to be had by incorporating activation data into the merging process. Using the expert models' activations would be an interesting follow-up. That approach would focus on retaining specialized skills rather than general ones and would need to address the key challenge of how to merge when different tasks mark the same neurons as important, which could be a very interesting potential follow-up research.\n\nIn summary, our new experiments validate AIM's generalizability across architectures and demonstrate the minimal, principled nature of its data requirements. We believe these additions substantially strengthen the paper and address the primary concerns raised, and we hope they will convince the reviewer to raise their score. We thank the reviewer again for their constructive engagement with our work.\n\n[1] Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., ... & Leahy, S. (2020). The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.\n\n[2] Feng, K., Gong, K., Li, B., Guo, Z., Wang, Y., Peng, T., … Yue, X. (2025). Video-R1: Reinforcing Video Reasoning in MLLMs. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2503.21776.\n\n[3] Doris, A. C., Alam, M. F., Nobari, A. H., & Ahmed, F. (2025). CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generation. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2505.14646.\n\n[4] Hu, K., Wu, P., Pu, F., Xiao, W., Zhang, Y., Yue, X., … Liu, Z. (2025). Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2501.13826.\n\n[5] Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025). Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2412.14171"}, {"review_id": "vbq3bSCYzj", "reviewer": "Reviewer_j7VA", "summary": "This paper proposes Activation-Informed Merging (AIM), a simple yet effective technique to enhance model merging for fine-tuned LLMs by leveraging activation  information from the base model. By computing input activation magnitudes from a small calibration set, AIM identifies salient channels and constrains their updates during merging, thereby preserving critical base capabilities.", "questions": "If a particular channel exhibits high activation or gradient magnitude, it may be critical for both the base and fine-tuned models. Would overly constraining its update risk suppressing important adaptations from the fine-tuned model and thereby hurt performance?", "limitations": "The paper does not include a Limitations section. The authors should discuss potential weaknesses of the method, particularly its generalization ability across models, beyond the tested settings.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**strength**\n\n1. paper writing is clear and easy to follow\n\n2. the method be plugged into many merging methods with minimal overhead.\n\n3. The method is simple  and efficient. And it consistently improves performance across benchmarks \n\n**weakness**\n1. All experiments are conducted on  a single model (LLaMA-2-13B). It remains unclear whether AIM is robust across different model families and scales\n\n2.  Most evaluations are limited to merging two or three models . It's unclear how AIM scales when merging larger pools of fine-tuned experts in diverse domains.", "was_revised": true, "final_justification": "Thank you for the detailed response and new experiments. The added results on Qwen 2.5 VL help address some concerns about generalization. I will keep my score, as I believe it fairly reflects the current scope.", "rebuttal": "We thank Reviewer j7VA for the positive feedback and for highlighting the clarity, simplicity, and efficiency of our proposed method. We appreciate the opportunity to address the weaknesses and questions raised.\n\n## Validating Generalizability Across Architectures and Modalities\nTo address the reviewer's valid point on generalization, we conducted new experiments on the Qwen 2.5 VL family, a different architecture (QWEN), with a different number of parameters (7B), and a different modality (vision-language) from the models in our original paper. To assess AIM in this setup, we merged the Video-R1 model, a video reasoning model [1] finetuned from  Qwen2.5-VL-7B-Instruct, and the CAD-Coder model, also trained on the same architecture for image-to-code-based CAD geometry generation [2]. For merging, therefore, we have two experts, one for CAD and one for video reasoning, with a base model that is instruction-tuned.  Therefore, to assess performance, we perform the IF-Eval and MMLU benchmarks (the base model knowledge and instruction following) as well as expert benchmarks for video reasoning, Video MMMU[3] and VSI-Bench[5], and CAD generation benchmarks of CAD-Coder benchmark (on rendered CAD images)[2] and CAD-Coder Real benchmark (on real 3D printed images)[3].  As shown in the table below, which we will include in the final paper, AIM consistently improves the performance of the underlying merging method across diverse tasks while also showing that the overall hypervolume gain is raised in all merging scenarios. This confirms that AIM's benefits are not limited to the Llama-2 architecture, and AIM does truly provide a path towards higher quality merging with little computational and data overhead. \n\n| Method | AIM | IFEval | MMLU | VSI-Bench (Mean Relative Accucary) | Video MMMU | CAD-Coder | CAD-Coder Real | HV Gain |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Instruct | - | **0.685** | **0.667** | 0.210 | 0.474 | 0.040 | 0.049 | - |\n| Video-R1 | - | 0.643 | 0.610 | **0.383** | **0.492** | 0.000 | 0.025 | - |\n| CADCoder | - | 0.268 | 0.657 | 0.000 | 0.002 | **0.614** | **0.324** | - |\n| TIES | No | 0.571 | **0.677** | 0.230 | 0.406 | 0.610 | 0.324 | 0.438 |\n|  | Yes | **0.577** (+0.93%) | 0.675 (-0.19%) | 0.225 (-2.24%) | **0.454** (+12.05%) | **0.647** (+6.01%) | 0.317 (-2.28%) | 0.448 (+2.26%) |\n| DARE  TIES | No | 0.540 | 0.669 | 0.246 | 0.419 | 0.537 | 0.318 | 0.430 |\n|  | Yes | 0.559 (+3.49%) | 0.670 (+0.23%) | **0.264** (+7.21%) | 0.447 (+6.63%) | 0.551 (+2.64%) | 0.321 (+0.73%) | 0.445 (+3.47%) |\n| DARE Task Arithmetic | No | 0.246 | 0.645 | 0.252 | 0.429 | 0.542 | 0.326 | 0.380 |\n|  | Yes | 0.559 (+127.07%) | 0.670 (+3.96%) | 0.258 (+2.37%) | 0.452 (+5.44%) | 0.542 (-0.13%) | **0.336** (+2.94%) | 0.446 (+17.42%) |\n| WIDEN | No | 0.349 | 0.661 | 0.084 | 0.261 | 0.611 | 0.333 | 0.318 |\n|  | Yes | 0.349 (+0.00%) | 0.661 (+0.00%) | 0.138 (+63.57%) | 0.320 (+22.55%) | 0.642 (+4.92%) | 0.334 (+0.23%) | 0.360 (+13.24%) |\n\n## On Scaling with Larger Pools of Models\nRegarding the question of how AIM scales when merging larger pools of experts, this is a valid point about the scope of our experiments. AIM is designed as a complementary plug-in that improves existing merging methods. As such, its ability to scale is inherently tied to the scalability of the underlying algorithm (e.g., DARE, Ties-Merging) it is paired with. Since AIM consistently improves the 2- and 3-model merging scenarios tested, we believe it will provide similar benefits in larger-scale merging settings, provided the base merging algorithm is effective in that context.\n\n## Balancing Base Model Preservation and Expert Adaptation\n\nThe reviewer raises a crucial point about the risk of suppressing important fine-tuned adaptations. This is the central trade-off our method is designed to manage. The 'ω' hyperparameter in Equation 4 was introduced for this exact purpose: to control the strength of the regularization. The role of 'ω' is to allow a practitioner to find the right balance between preserving the base model's general capabilities and integrating the new, fine-tuned adaptations. Our extensive experiments, which show significant performance gains across multiple benchmarks, demonstrate that we can find a 'ω' value that strikes an effective balance, successfully mitigating catastrophic forgetting without suppressing essential knowledge from the expert models.\n\nWe believe these clarifications and new experimental results directly address the points raised and hope they will convince the reviewer to raise their score. We thank the reviewer again for their constructive engagement with our work.\n\n[1] Feng, K., Gong, K., Li, B., Guo, Z., Wang, Y., Peng, T., … Yue, X. (2025). Video-R1: Reinforcing Video Reasoning in MLLMs. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2503.21776.\n\n[2] Doris, A. C., Alam, M. F., Nobari, A. H., & Ahmed, F. (2025). CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generation. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2505.14646.\n\n[3] Hu, K., Wu, P., Pu, F., Xiao, W., Zhang, Y., Yue, X., … Liu, Z. (2025). Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2501.13826.\n\n[4] Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025). Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2412.14171"}]

Author rebuttal:

Meta-review: The paper received mixed ratings initially. There are a few minor concerns from the reviewers. The rebuttal has addressed most of them, and the reviewers raised their scores. The final decision is acceptance.
Decision: Accept (poster)