Title: Purifying Task Vectors in Knowledge-Aware Subspace for Model Merging
Primary area: foundation or frontier models, including LLMs
Keywords: pre-trained models, fine-tuning, multi-task learning, model merging

Abstract: Model merging aims to integrate task-specific abilities from individually fine-tuned models into a single model without extra training.
In recent model merging methods, task vector has become a fundamental building block, as it can encapsulate the residual information from finetuning. 
However, the merged model often suffers from notable performance degradation due to the conflicts caused by task-irrelevant redundancy in task vectors. 
Existing efforts in overcoming redundancy by randomly dropping elements in the parameter space involves randomness and lacks knowledge awareness. 
To address these challenges, in this study, we propose $\textbf{P}$urifying T$\textbf{A}$sk $\textbf{VE}$ctors (PAVE) in knowledge-aware subspace. 
Concretely, we sample some training examples from each task, and feed them into their corresponding fine-tuned models to acquire the covariance matrices before linear layers.
We then perform a context-oriented singular value decomposition, which accentuates the weight components most relevant to the target knowledge.
As a result, we can split fine-tuned model weights into task-relevant and redundant components in the knowledge-aware subspace, and purify the task vector by pruning the redundant components.  
To induce fair pruning efforts across models, we further introduce a spectral rank allocation strategy by optimizing a normalized activated pruning error. 
The task vector purification by our method as a plug-and-play scheme is applicable across various task vector-based merging methods to improve their performance.
In experiments, we demonstrate the effectiveness of PAVE across a diverse set of merging methods, tasks, and model architectures.
Remarkably, when integrated on the state-of-the-art merging method EMR-Merging with RoBERTa, our method yields a significant performance improvement of 4.1\% (from 80.18\% w/o PAVE to 84.28\%) on the GLUE benchmark, approaching to the average performance of 8 individual models, 85.55\%.

TL;DR: We propose PAVE, a plug-in method that purifies task vectors by removing redundancy via decomposition in knowledge-aware subspaces and rank allocation. It improves performance across diverse merging methods, tasks, and model architectures.