NeurIPS 2025

Activation-Informed Merging of
Large Language Models
Amin Heyrani Nobari∗1, Kaveh Alim∗1, Ali ArjomandBigdeli2
Akash Srivastava3, Faez Ahmed1, Navid Azizan1
1Massachusetts Institute of Technology
2Stony Brook University
3MIT-IBM Watson AI Lab & Red Hat AI Innovation
Abstract
Model merging, a method that combines the parameters and embeddings of mul-
tiple fine-tuned large language models (LLMs), offers a promising approach to
enhance model performance across various tasks while maintaining computational
efficiency. This paper introduces Activation-Informed Merging (AIM), a technique
that integrates the information from the activation space of LLMs into the merging
process to improve performance and robustness. AIM is designed as a flexible,
complementary solution that is applicable to any existing merging method. It aims
to preserve critical weights from the base model, drawing on principles from con-
tinual learning (CL) and model compression. Utilizing a task-agnostic calibration
set, AIM selectively prioritizes essential weights during merging. We empirically
demonstrate that AIM significantly enhances the performance of merged models
across multiple benchmarks. Our findings suggest that considering the activation-
space information can provide substantial advancements in the model merging
strategies for LLMs with up to 40% increase in benchmark performance. Our code
is publicly available at https://github.com/ahnobari/ActivationInformedMerging.
1
Introduction
Foundation models are rapidly becoming the dominant force for building Artificial Intelligence (AI)
systems. In many cases, researchers build their machine learning models by starting from pre-trained
foundation models and fine-tuning (FT) these pre-trained models for some desired target task [27]. In
such a paradigm, numerous fine-tuned models are developed for various tasks. However, an important
opportunity is missed, as these fine-tuned task-specialized models typically operate in isolation
without leve...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000017. Conference bucket: NeurIPS.