Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
Authors: Weiyu Chen
James Kwok
Venue: ICML 2025 poster (2025)
Keywords: Multi-Objective Optimization
Model Merging
Primary area: general_machine_learning->transfer_multitask_and_metalearning

Abstract: Model merging, which combines multiple models into a single model, has gained popularity in recent years. By efficiently integrating the capabilities of various models, this significantly reduces the parameter count and memory usage. However, current methods can only produce one single merged model. This necessitates a performance trade-off due to conflicts among the various models, and the resultant one-size-fits-all model may not align with the preferences of different users who may prioritize certain models over others. To address this issue, we propose preference-aware model merging, and formulate this as a multi-objective optimization problem in which the performance of the merged model on each base model's task is treated as an objective. In a single merging process, the proposed parameter-efficient structure generates a Pareto set of merged models, with each representing a Pareto-optimal solution for a preference. Users can then select merged models tailored to their preferences from this learned Pareto set. Experimental results demonstrate that the proposed Pareto Merging produces diverse trade-off models and achieves higher test accuracy compared to state-of-the-art merging baselines.

Pareto Merging:
Multi-Objective Optimization for Preference-Aware Model Merging
Weiyu Chen James T. Kwok
Abstract
Fancla Margin for ssrr
Model merging, which combines multiple mod- els into a single model, has gained popularity in recent years. By efficiently integrating the capa- bilities of various models, this significantly re- duces the parameter count and memory usage. However, current methods can only produce one single merged model. This necessitates a per- formance trade-off due to conflicts among the various models, and the resultant one-size-fits-all model may not align with the preferences of dif- ferent users who may prioritize certain models over others. To address this issue, we propose preference-aware model merging, and formulate this as multi-objective optimization problem in which the performance of the merged model on each base model's task is treated as an ob- jective. In a single merging process, the pro- posed parameter-efffcient structure generates a Pareto set of merged models, with each repre- senting Pareto-optimal solution for preference. Users then select merged models tailored to their preferences from this learned Pareto set. Ex- perimental results demonstrate that the proposed Pareto Merging produces diverse trade-off mod- cls and achieves higher test accuracy compared to state-of-the-ee merging baselines.
Accurscy on Model 8'ss Deleset
Fancts Marging
Modld
exsiting nethods
Poreto Werging
Redd
Accuracy on Model A's Datoset
Figure I. Comparison of cxisting methods and the proposed Pareto Merging for merging two models with varying user preferences.
multiple fine-tuned models results in a significant increase in parameter count and memory usage. As many of them originate from the same pre-trained model (e.g.. ViT (Doso- vitskiy et al., 2020)), it is thus desirable to combine them to a single model while still maintaining the full capabilities of these fine-tuned models, even without access to their orig- inal training data. This process, known as model merging (Matena & Raffel, 2022; Jin et al., 2023; Yang al., 2024b), offers efficient way to leverage the abundance of publicly accessible fine-tuned models.
A naive model merging approach is to average the weights of the fine-tuned models. However, this merged model often suffers from poor performance. To address this problem, various more advanced methods have been developed. For example, Matena & Raffel (2022) uses the Fisher matrix preserve key features in the various models. RegMean (Jin et al., 2023) minimizes prediction discrepancies through weight adjustments. Ilharco et al. (2023) proposes to first obtain the differences (called task vectors) between the fine- tuned models and the original pre-trained model. A merged model is then created by combining the pre-trained model together with the sum of these task vectors. Further methods have been proposed to improve performance by reducing redundancy and conflicts among task vectors (Yadav et al., 2023), and addressing gradient mismatches (Daheim et al., 2024). Recently, AdaMerging ( (Yang et al., 2024b) pro- posed that adaptively weights the task vectors based on the entropies of their model predictions.
1. Introduction
Fine-tuning pre-trained models is a widely-adopted ap- proach to create specialized models for various downstream tasks (He et al., 2022; Wortsman et al., 2022; Li al., 2024a; Hu et al., 2021; Dettmers al., 2024; Liu al., 2024). Nowadays, many fine-tune fine-tuned models are publicly available on online platforms such as HuggingFace. However, having
DDpaartment of Computer Science and Engineering, The Hong Kong University of Science and Technology. Correspondence to: Weiyu Chen <wchenbx @ ese.uss..k...
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
An important limitation of existing model merging methods is that they can only produce a single merged model. Fig- ure I shows an example of merging two fine-tuned models: model A on task A and model B on task B. Due to conflicts between the two models, typically the merged model (de- noted by the square in the figure) cannot achieve the same accuracies on both tasks as models A and B. Hence, it needs to find model with performance trade-off. However, this one-size-fits-aln approach may not meet the diverse pref- erences of different users (shown as dotted vectors). For instance, user may prefer to prioritize task B over task A, while user n prioritizes task A over task B.
datasets (Ilharco et al., 2023; Matena & Raffel, 2022; Jin et al., 2023; Yadav et al., 2023; Daheim et al., 2024; Yang et al., 2024b). In this paper, we focus on the latter.
Let the deep network be fa; 0, h2). where x is the input data, 0 is the parameter of the shared bottom, and h2 is the task-specific head for the kthh dataset. We denote the pre- trained model's parameter as Qy. and that after fine-tuning on dataset k as 0g.. Since model merging focuses on the shared bottom, we use the simplified notation fo = f( 0).
The goal of model merging is to effectively combine K fine- tuned models fe...... fex. A simple approach is weight av- eraging, but it often degrades performance. Fisher Merging (Matena & Raffel, 2022) uses the Fisher information matrix to ensure that crucial features from each task are preserved effectively. RegMean (Jin et al., 2023) minimizes the predic- tion differences between the merged model and individual models by adjusting their weights. Task Arithmetic (Ilharco et al., 2023) combines the task vectors (V22 = 02 - 0o K kmmi VE. where À E R
To overcome this limitation, we propose a preference-aweer model merging approach, referred to as Pareto Merging (PM). This formulated as multi-objective optimization (MOO) problem (Miettinen, 1999) in which the performance on each model's 's task is considered as an objective. When there are n user preferences, naive integration of exist- ing merging methods with MOO still requires storing the original fine-tuned models. Besides, it requires n merging processes, which can be time expensive for merging meth- ods that require optimization (e.g., AdaMerging). Thus, we introduce novel parameter-efficient structure consisting of
to form merged model 0g + A Σ K controls the importance of task vectors. However, this might lead to sub-optimal performance due to potential conflicts among task vectors. To address this issue, TIES Merging (Yadav et al., 2023) designs operations to reduce redun- dancy and resolve sign conflicts among the task vectors. DARE (Yu al., 2024) proposes to randomly drop some parameters and rescale the remaining parameters.
base model and a preference- dependent personalized model. In one single optimization
process, PM learns the whole Pareto set of merged mod- els, each representing the Pareto-optiman model for a user preference. From the learned Pareto set, one can provide merged models for different user preferences (denoted as circles in Figure 1). Moreover, PM can be used with model merging methods that do not require downstream data or only unlabeled data in the downstream task.
By observing that lower Shannon entropy correlates well with model performance (Grandvalet & Bengio, 2004; Roy et al., 2022; Yang et al., 2024b), AdaMerging (Yang et al., 2024b) adaptively weighs the task vectors by minimizing the entropy averaged over all datasets. AdaMergigg++ further pre-processes the task vectors with TIES Merging.
The main contributions of the paper are as follows:
Several works enhance merging performance with task- specific modules but often add inference costs. Yang et al. (2024a) address representation distribution discrepancies be- tween merged and unmerged models through representation surgery. Lu et al. (2024) propose dynamic merging using the mixture of experts (Cai et al., 2024), but it incurs extra infer- ence costs and requires labeled data. MMrgingg (Huang et al., 2024) uses task-specific masks with masking and rescaling during inference, introducing overhead. These methods are orthogonal to the proposed PM and can be com- bined with it (e.g.. learning the Pareto set for these models). Notably, PM avoids extra inference costs compared to the base method once the user preference is specified.
We formulate preference-aware model merging multi- objective optimization, and provide diverse trade-off solu- tions according to user preferences.
We introduce preference-dependeen personalized model using low-rank tensor, enabling diverse model generation with minimal parameter overhead.
Experiments demonstrate that the proposed model outper- forms existing ofte-oofthhearr and can produce diverse models aligned with various user preferences.
2. Background
2.1. Model Merging
Most related to the proposed method are rewarded soups (Rame et al., 2024) and MAP (Li et al., 2024b). The re- warded soups address multi-objective reinforcement learn- ing with human feedback (RLHF) by training separate mod- els for each objective and merging them through a simple weighted averaging of models based on preference vectors.
Model merging combines multiple deep networks to a sin- gle model. Existing methods can be divided into two cat- egories: (i) merging models trained with different initial- izations (Ainsworth et al., 2023; Jordan et al., 2023; Stoica et al., 2024), and (ii) merging models fine-tuned on different
2

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
MAP employs an evolutionary MOO algorithm search for the merging weights. A detailed comparison with these two approaches will be discussed in Section 3.3.
2024; Dimitriadis et al., 2024) and concurrent works (Zhong et al., 2024; Tang et al., 2024) is in Appendix B.
3. Methodology
2.2. Multi-Ohjective Optimization
Despite the recent advances on model merging, a signifi- cant limitation is that they can only output a single trade-off' model which cannot align with the diverse user preferences. To alleviate this problem, Section 3.1 first formulates model merging MOO problem. Section 3.2 introduces effi- cient preference-dependen tensor structure. Finally, Section 3.3 shows how to solve the resultant optimization problem.
Multi-objective optimization (MOO) (Miettinen, 1999) aims to optimize K objectives:
min g(v) =
(I)
where 14 is the variable and g& is the ktth objective function. A solution dominates another solution b if and only if vk E (1...., K,)....,,... 9u (a) S. 9k (b) and 3i E 1,..., K such that 9:(a)= < g:(b). A solution a strictly dominates another solution b and only v/k E (1,...,K} 9k (a) < 9k (b)- A solution is Pareto optimal (resp. weakly Pareto optimal) if no other feasible solution dominates (resp. strictly dom- inates) it. The Pareto set the set of all Pareto optimal solutions, while the Pareto front is the set of objective val- ues of these Pareto optimal solutions.
3.1. Model Merging as Multi-Ohjective Optimization
To accommodate different user preferences and thus achieve different trade-offs, we view model merging as a MOO problem. There are two common model merging scenarios.
3.1.1. DATA-FREE MERGING
Recall that in Task Arithmetic (Ihaacco et al., 2023), the merged model is 0 = On > >k== Vs, where X is given hyperparameter. This can be interpreted as the closed-form solution of the optimization problem: ming Σ (6 + VK 01 which minimizes the total distance of 0 to each scaled finetuned model On + XK Vs. However, it does not incorporate user preferences and is restricted to generating single model.
A variety of gradient-based MOO algorithms have been developed and used in deep learning (Chen et al., 2025).
Notable among these are MGDA (Sener& & Koltun, 2018:
Désidéri, 2012; Mukai, 1980; Fliege & Svaiter, 2000),
EPO (Mahapatra& & Rajan, 2020). CAGrad (Liu et al., 2021),
Nash-MTL (Navon et al., 2022), and Auto-À (Liu et al.,
2022). However, they can only output single solution each run.
We propose to reformulate model merging as the MOO problem with objective (x) |1(0 + XKV)) 0
To better capture the Pareto set, several methods (Lin et al., 2019; Momma et al., 2022; Chen et al., 2024; Zhang et al., 2025) decompose the problem using a set of preference vec- tors, generating single solution per run for each given pref- erence. However, handling different preferences requires multiple runs, resulting in multiple models. To address this limitation and learn continuous Pareto set, Pareto Hyper- networks (Navon et al., 2021: Lin et al., 2020; 2022) uti- lize a hypernetwork to generate network parameters based net- works build on this by integrating preferences directly into the input layer (Ruchte & Grabocka, 2021) or through the FiLM layers (Dosovitskiy & Djolonga, 2020; Chen & Kwok, 2022). PaMaL (Dimitriadis et al., 2023) intro- duces a strategy that leverages the weighted sum of a set of base networks, while LORPMAN (Chen & Kwok, 2024) enhances efficiency using low-rank matrices. These ap- proaches mainly focus on smaller models, such as the LeNet (LeCun et al., 1998) (30K parameters) or ResNet-18 18 (He et al., 2016) (1 1M parameters), and focus training from scratch using large amounts of labeled data. In contrast, we explore model merging for large models (e.g., ViT (Doso- vitskiy et al., 2020) with limited unlabeled data or no data.
ming [S] .... , (x)
(2)
Given a preference vector 7 = [n...., 7Kk] in the sim- plex A = {>l EK= 7 = 1,7k 2 0}. straightfor- ward method to solve (2) is linear scalarization, which converts (2) to the single-objective optimization problem: ming Ek= 7kk 11(0 + VKV) lution is:
0 O
Its closed-form so-
on given preference vector. cefference-conditioon
K
8(7) = Bu + AK 7A VA.
(3)
By varying 7. different trade-offs can be achieved.
Specifically, (1) When preferences are equal (> = [1/K, 1/K]T), it recovers the solution of Task Arith- metic (2) When À = 1/K, the solution 0o + + Th Vv directly weights the task vectors with the preference vector, as in Rewarded Soups (Rame et al., 2024).
Note that the solution subspace obtained in (3) is not parameter-efficien. It requires storing KQ pa- rameters', where Q is the number of parameters in the
Limitation.
Since the number of user preferences is usually larger than K, it is more space-efficintt to store the KQ task vectors and
detailed discussion on related works (Chen & Kwok,
3

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
pre-trained model. In other words, the numbers of stored parameters remains unchanged before and after merging.
preference-indlepeneen base component and a preference- dependent personalized component. For the Ith module in the backbone network (such as the attention module in transformer), the base component includes the pretrained parameter 0j 'R''' xa' and weighted combination of task vectors Σi=1 X/KZ. The personalized component, parameterized as (')) Re''d', depends on the user preference 7. For simplicity, the description below focuses on a single module, omitting the superscript! I.
3.1.2. MERGING BASED ON UNLABELED DATA
In AdaMerging (Yang et al., 2024b), instead of using a fixed À for all task vectors, a weighting vector À = [Ai....; > is learned for each task vector by optimizing the entropy averaged over batches of unlabeled test data BI,...,Br Bx sampled from K datasets:
Inspired by recent advancements parameter-efficien fine- tuning (Hu et al., 2021; Dettmers et al., 2024; Liu al., 2024), W()) has a low-rank structure. However, the popu- lar approach of using a product of low-rank matrices in LoRA (Hu et al., 2021) fails to account for user preferences. To provide a parameter-efficien way to integrate preference 7, we model ((7) as the following c x d x tensor:
K min Jemn 15
K
[H(s(=;00 SALO(A) = 00+
A Vk+ (4)
where (-) is the Shannon entropy: 2 This approach demon-
strates superior performance compared data-free methods.
However, it similarly lacks preference-awarenes.
We propose to reformulate (4) to MOO by viewing the merged model's entropy ExEB, (ff(=; 0)) on each dataset k as an objective S (0):
W(>) = Ax X3
where G E RrxexK is a core tensor, E R'xc, B E Rexd are matrices, and r is given rank parameter. Note that = G X1 A X2 B can be partitioned into K slices MI,..., MR. where each Mg E Rexd is low-rank and [Ma]a] = Eij. Eg Gqi-ga.k Ag, : These matrices share parame- ters A and B, while maintaining distinct characteristics through variations in G. W(7) in (7) is then a weighted sum of these low-rank matrices: W(>) = Σ k- 7rh Mr. By adjusting 7, we modify the influence of each matrix to better align with user preference.
min (Aiix=/Ae....... ARl
((())).,........
Sx (x(O))
c x d K tensor. This
K
S.L.
((2) 0o
Σ
Av
(5)
Similar to data-free merging, given user preference vector Y. the naive MOO approach linear scalarization:
K min Σ Sx(A)( s.t. O(A) O,- + Σ
K
Ve. (6)
The whole model is then:
Different trade-off models can be obtained by varying 7. In particular, with equal preferences, it recovers AdaMerging.
K
0(>: 0o- 4 (8)
Limitations. For n user preferences ( (*)) n runs of the optimization (6) are required to compute the corresponding 's,,,, which is computational inefficient. Furthermore, similar to data-free merging, one still requires storing KQ parameters (the same as before merging).
An illustration of the proposed structure in Figure Once the parameters (G, A, B) are learned, infinite number of models can be generated by simply varying Y
Can we learn a parameter-efficiet space of merged models in a single optimization process that can provide models tailored to varying user preferences?
Parameter Efficiency. After merging, On E d V forms single model with ed parameters, matching the original pre-trained model. Thus, (8) adds only r²K + r(c + d) parameters after merging. In the experiments, we apply the preference-deepeddent modification only to the transformer's attention modules. For example, with r = 16 in a ViT-B/32 model, the proposed approach increases parameters by just 0.5% compared to the pre-trained model.
3.2. Parameter-Effiiient Structure
To efficiently learn the Pareto set of preference-an aware model merging, we propose a parameter-efficcient structure with
compute merged models on-the-fly, rather than storing merged models (nQ parameters).
xx, indicates the u-mode tensor product, which is detailed as follows: (G X2 A) = Σr =1 Digigs494411 for
2For a M-class classification problem, the Shannon entropy of prediction ÿ E [0, 1|M is (g)) = =- EM Im log Dm.
I,..., c (G X2 B) 3) = Σrr Liz=1 Giysass Big. for d2 =
I,..., d; and (G Y3 Σ =1=1 Giidzi Bay for gs =
4

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
X d
γ (1)
C X d A single model after merging
Merged Model for User
B
K x 1
y 7(2
K
A
G
Y
Merged Model for User
O
+
X
& x X K
γ (n)
preference-independeer base model
lifference-dependerr low-rank tensor
Merged Model
for User
Figure 2. Illustration of the proposed Pareto Merging. After merging, it introduces minimal parameter overhead while providing different orf models different preferences.
3.3. Optimization
Algorithm Pareto Merging (PM).
preprocess the task vectors Vi,..., Vx (if required);
In this section, we consider the optimization problems for both data-free merging and unlabeled data-based merging. Since linear scalarization convex parts of the Pareto front (Boyd & Vandenberghe, 2004), we use instead smooth Tchebycheff scalarization (Lin et al., 2024), which can identify the entire Pareto front. For the MOO problem in (1), smooth Tchebycheff scalar- ization considers the following optimization problem:
while not converged do
only identify solutions the
sample from the Dirichlet distribution;
obtain the corresponding model from (8); if data-free then
(x(((A: = 110 + AKV, 0(A:
else
*** using unlabeled data **/
sample minibatch of unlabeled data b bi....,bry
K Σ
Th.9 (1) 2
Sy (0(( E (f x)))
min p log 14
exp
(9)
end if
P
compute the objective in (11);
where o E IR is smoothing parameter, and 2 is the ideal value of gx (14). It can be shown that fx" is weakly Pareto optimal solution of the original MOO problem (1) if and only if it is also an optimal solution of (9) for some
update G, A, B (and À if using unlabeled data) via gradient descent;
end while
In our context, we set =x = 0, as the minimum of each (g) in either (2) (for data-free merging) or (5) (for unlabeled data-based merging) zero.
adopt the symmetric Dirichlet distribution, Dir(Ip....,,p) ? with parameter p. for P(y)). By minimizing the expectation over the preference distribution, we aim to learn model that minimizes the loss for each possible user preference (as the Dirichlet distribution spans the simplex of all possible preferences). We also incorporate regularizer in (10) to control the magnitude of model modification and prevent overfitting.
We apply smooth Tchebycheff scalarization problems (2) and (5) using the model in (8). The goal is to optimize the expected Tchebycheff scalarized objective value over
preference distribution P(y):
K Σ
(=Sx(0(A;7))
MinG.A.B Bymp(y))
exp
The proposed approach can be used with most merging techniques to enable preference-awee merging. As shown (10) in Algorithm 1, preference vector is sampled from the Dirichlet distribution in each iteration. For model merging using unlabeled data, also sample minibatch of unla- beled data b ...... bx from B1, ......, BR. (11) stochastic gradient descent on (11) 1) to update the learnable tensor and matrices G. A, and B. For model merging using unlabeled data, we also update the merging parameter X.
P
+B x1 A X2 B Xx 1KllF
s.L. 0() = 0p + Vo
Next, we perform
+G 1 A X2 B
For data-free approaches, À is a user providdo hyperparam- eter. For merging with unlabeled data, we follow AdaMerg- ing and optimize with the optimization objective (10). We
Computation Complexity. During training, the per-
5

---

Pareto Merging: Multi-Objectivy Optimization for Preference-Aware Model Merging
We first obtain 01 by optimizing S1, and 0g by optimizing S2- The average (0) + 02)/2 serves as the preference- independent base model, which is then fixed. Subsequently, we optimize G. A. and B in the preference-dependee model. Figure shows models sampled from the learned Pareto set using 31 uniformly distributed preferences. As can be seen, the solutions overlap with the ground-truth PF and uniformly cover the entire PF.
0
-5-
S2(0)
-10
-15
4.2. Real-World Datasets
-20
Following (Ilharco et al., 2023; Yang et al., 2024b), we use the vision encoders (ViT-B/32 and ViT-L/14) from CLIP (Radford et al., 2021) as pre-trained backbones. Unless otherwise specified, VIT-B/32 is used. This is fine-tuned on the following eight image classification datasets as in (Ilharco et al., 2023; Yang et al., 2024b): SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD. Details can be found in Appendix C.
-20
-10
0
())
Figure 3. Solutions (red stars) sampled from the PF obtained by PM on the toy problem. The ground-truth PF is in gray.
iteration complexity of the proposed AdaMerging + PM is O(Ked + (c d)r + Kr²), whereas the original AdaMerg- ing method has a complexity of ((Ke).. Since is signif- icantly smaller than e and d, the additional computational overhead remains minimal. During inference, once the pref- erence is fixed, the low-rank tensor can be merged into the base model, resulting in no additional inference overhead.
4.2.1. MERGING TWo MODELS
In this experiment, we merge the two models fine-tuned on RESISC45 and GTSRB. Pareto Merging is applied to the baselines of Task Arithmetic (data-free) and AdaMerging (with unlabeled data). We also compare with two baselines that can obtain the Pareto set: Rewarded Soups (Rame et al., 2024) (data-free) and MAP (Li et al., 2024b) (with data). We follow MAP's official implementation that uses labeled data and population size of 100.
Remark. Rewarded Soups (Rame et al., 2024) applies meefrencce-weigheer merging to RLHF. In contrast to the proposed PM, it requires storing all task vectors. More- over, the simple weighted averaging scheme it uses is often outperformed by more advanced data-free or data based merging methods (Yadav et al., 2023; Yang et al., 2024b), as will be empirically shown in Section 4.2.1.
Figure 4 shows the test accuracies of merged models across 11 preference vectors (from [0.0, 1.0]T to [1.0, 0.0] For clarity, only a subset of the solutions is shown. The com- plete plot is shown in Appendix D. As can be seen, PM, combined with baseline methods, effectively generates di- verse trade-off models across the two datasets, enabling users to choose models that align with their preferences. For example, with preference vector of 7 = [1.0, 0.0] which prioritizes RESISC45 exclusively, the model gen- erated by AdaMerging + PM (the rightmost green star in Figure 4) achieves an accuracy of 94.8% on RESISC45, while the original AdaMerging method (without preference consideration) achieves only 93.6% accuracy on RESISC45. Similarly, Task Arithmetic PM (the rightmost blue star) achieves 92.3% accuracy RESISC45, while the original Task Arithmetic achieves 91 .2% accuracy. These clearly demonstrate the advantages of preference-aware Pareto Merging over size-fits-ar models.
Similarly, the concurrent work MAP (Li et al., 2024b) again requires storing all task vectors create trade-off models. Additionally, it can only produce discrete approximation of the Pareto set. In contrast, the proposed PM stores only single merged model and a small low-rank tensor, provid- ing a continuous approximation of the Pareto set and can directly generate models for any preference.
4. Experiments
In this section, we perform experiments on both toy problem (Section 4.1) and real-world datasets (Section 4.2). Ablation study is provided in Section 4.3.
4.1. Toy Problem
Compared to the rewarded soups, Task Arithmetic + PM (which is also data-free) achieves much better trade-off. Similarly, AdaMerging + PM outperforms MAP. Notably, while MAP relies labeled data (which may not be fea- sible in practice), PM only requires unlabeled data. Due to rewarded soups' significantly inferior performance, it is
We first demonstrate that the proposed algorithm can learn the Pareto set by using a popular toy problem (Liu et al., 2021; Navon et al., 2022) with two objectives S, (0) and (0). Due to the limited space, the detailed definition can be found in Appendix A.
6

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
Table Comparison of training time (on NVIDIA A6000).
parameter count after merging and the number of models that one can obtain after merging two models.
99.0
Method
GPU hours
# params
# models infinite
98.0
Rewarded Soups
226.7M
Accurcyy on GTSRB
Task Arithmetic
2 0
113.4M
1
Task Arithmetic PM
0.04
114.0M
infinite
MAP
0.35
226.7M
500
AdaMerging
0.15
113.4M
1
96.0
AdaMerging + PM
0.28
114.0M
infinite
Task Artthmetic+ Preference 4M Paramter Overhead)
95.0
Task Arthimetic + PM (0 6M Paramter Overhead)
Afthmetic (M Paramter Overhwad)
AdaMerging Preference (113.4M 3M4 Paremter Overhead)
excluded from further comparisons.
AdaMerging PM (0 AM Paramter Overhead)
94.0
80.0
82.0
84.0
86:0
86.0
90.0
92.0
94.0
Table compares the training time, parameter count after merging, and number of models generated. As can be seen, while PM requires slightly more time than the baselines, it adds minimal parameter overhead after merging and can generate an infinite number of trade-off models. When compared to Rewarded Soups and MAP, after merging, both Rewarded Soups and MAP result in a parameter count that is double that of the pre-trained model, as they require storing all the original models. This overhead becomes even more significant as K increases.
Accuracy on RESISC4S
Figure 5. Comparison with two straightforward methods discussed in Section 3.1 when merging two ViT-B/32 models.
to attention layers), achieves a reasonable trade-ofh off with a significantly reduced parameter overhead of only 0.6M. Furthermore, extending preferencee-dependen modifications to both the attention and MLP layers results in a broader spread of PF, while keeping the parameter overhead modest at 2.1..
A similar advantage is observed with the proposed AdaMerg- ing PM method. It achieves superior trade-off with 0.6M parameter overhead, compared to the 11..4M overhead in- curred by AdaMerging Preference. Beyond parameter efficiency, AdaMerging + PM also offers significant compu- tational savings. For instance, AdaMerging + PM requires only single execution (0.28 GPU hours), whereas AdaMerging Preference requires 11 runs (1.65 GPU hours) to handle 11 preference vectors.
98.0
96.0
94.0
Rewarded Soups
92.0
Task Arthhmetic
Task Artthmetic + Pareto Merging
MAP
90.0
Ada Merging Pareto Merging
Merging
4.2.2. MERGING EIGHT MODELS
86.0
88.0
90.0
02.0
94.0
In this experiment, we utilize all eight models. The proposed method PM is applied to two representative data-free merg- ing approaches: Task Arithmetic and - (Yadav et al., 2023). Additionally, and its enhanced version AdaMerging++ (to avoid confu- sion, this is referred to as AdaMerging+TIES in the sequel). We also compare with other baselines, including Weight Averaging (Yaday et al., 2023; Yang et al., 2024b), Fisher Merging (Matena & Raffel, 2022), RegMean (Jin al., 2023), DARE (Yu al., 2024), and MAP. For MAP, We use the nested merging variant in the official code.
Accuracy RESISC45
Figure 4. Accuracies of models obtained by different methods when merging two VIT-B/32 models.
apply PM AdaMerging
We also compare PM with two straightforward methods dis- cussed in Section 3:1: Task Arithmetic + Preference (Sec- tion 3. and AdaMerging Preference (Section 3.1.2). The results are shown in Figure For clarity, only subset of the solutions is shown. The complete plot is shown in Appendix D.I.
As shown in Figures 5, Task Arithmetic + Preference has a large parameter overhead of 113.4M and some of the extreme models it obtained are typically undesirable for users. In contrast, Task Arithmetic + PM with its default configuration (applying wefferenceddeennde modification
Figure 6 shows the accuracies of 30 models sampled from the learned Pareto set by AdaMerging + PM. Since the Pareto front with more than three objectives cannot be di- rectly visualized, each subplot in Figure 6 shows the accu- racies on 2 of the 8 tasks. Results on the other baselines +

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
Table Test accuracies when merging cight ViT-B/32 models. Results on DARE and MAP are reproduced using official implementations. Results on the remaining baselines are from (Yang et al.. 2024b). The best result among each group is in bold.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Average
Individual
75.3
77.7
96.1
99.7
97.5
98.7
99.7
79.4
90.5
Traditional MTL
73.9
74.4
93.9
98.2
95.8
98.9
99.5
77.9
88.9
Weight Averaging
65.3
63.4
71.4
71.7
64.2
52.8
87.5
50.1
65.8
Fisher Merging
68.6
69.2
70.7
66.4
72.9
51.1
87.9
59.9
68.3
RegMean
65.3
63.5
75.6
78.6
78.1
67.4
93.7
52.0
71.8
DARE
54.8
54.6
66.6
78.3
80.2
69.8
97.3
49.8
68.9
MAP
60.0
58.8
85.8
69.5
83.5
73.4
87.8
53.2
71.5
Task Arithmetic
55.2
54.9
66.7
78.9
80.2
69.7
97.3
50.4
69.1
Task Arithmetic+PM (equal)
55.2
55.0
66.7
78.9
80.2
69.7
97.3
50.4
69.2
Task Arithmetic+PM (priority)
55.5
55.9
67.6
81.8
84.5
73.3
97.9
51.5
71.0
-IES-Merging
59.5
60.0
71.7
78.2
86.3
72.9
98.2
52.8
72.4
TES-Merring++PM (equal)
59.5
60.0
71.8
78.3
86.3
72.9
52.8
72.4
IEES-Merging+PP (priority)
60.0
60.6
72.9
81.7
89.2
75.8
98.4
53.8
74.1
AdaMerging
64.5
68.1
79.2
93.8
87.0
91.9
97.5
59.1
80.1
AdaMerging-PM (equal)
70.1
74.2
87.3
96.5
90.2
95.6
98.5
66.7
84.9
AdaMerging+PM (priority)
71.1
74.2
89.0
97.6
92.1
97.4
99.0
64.0
85.5
AdaMerging-TIISS
66.6
68.3
82.2
94.2
89.6
89.0
98.3
60.6
81.1
A (equal)
70.6
73.9
87.5
96.7
90.8
96.7
98.6
67.2
85.2
AdaMerging (priority)
72.1
73.7
88.8
97.5
92.2
97.5
99.0
66.1
85.9
PM are in Appendix D.2. As can be seen, PM again offers good and diverse trade-off solutions.
For AdaMerging and AdaMerging-TTES. Pareto Merging shows even larger performance gains with the help of un- labeled data. Even with equal preference, PM achieves an average performance improvement of 4.8% and 4.1%, re- spectively. With priority preference, the performance gain is even more pronounced. For instance, on RESISC45, AdaMerging+PMM (priority) outperforms AdaMerging by 10%. These demonstrate that PM can efficiently incorporate preferences to achieve better performance for unlabeled- data based methods. We further experiment on the larger ViT-L/14 backbone. Results can be found Appendix E.
Table shows the test accuracies of Pareto Merging com- bined with four baselines, along with various others. We also compare with individual task learning and traditional multi-task learning (MTL), which serve as performance upper bounds and require labeled training data and are sig- nificantly more computationally expensive. Morcover, showing results for all users preferences is infeasible, we only show two representative types of preferences: (1) equal preference: 7 = [0.125,. ..., 0.125] T; and (2) priority pref- erence, in which one task has a weight of 0.5, while each remaining task has weight of 0.5/ K 1). This reflects the common scenario where users prioritize specific task while still expecting reasonable performance the others.
4.2.3. UNSEEN DATASETS
In this experiment, we assess the performance on unseen datasets with fine-tuned model. Following (Yang et al., 2024b), we consider two settings: (i) Merge models fine- tuned on SUN397, Cars, RESISC45, SVHN, GTSRB, and DTD, and then evaluate on MNIST and EuroSAT. (ii) Merge models fine-tuned on SUN397, Cars, EuroSAT, GTSRB, MNIST, and DTD, and then evaluate on RESISC45 and SVHN. For Pareto Merging, we randomly sample 30 models from the learned Pareto set, and select the one with the smallest Shannon entropy the target dataset.
Since the nested merging variant of MAP is used when merging more than two models, multiple runs are required to incorporate different user preferences. Thus, we only present the MAP results with equal preference.
As be seen, for Task Arithmetic and -erging Pareto Merging with priority preference significantly im- proves performance. For example, on SVHN, the accuracy increases by 4.3% and 2.9% for Task Arithmetic and TIES- Merging, respectively. Additionally, the average accuracy increases by 1.9% and 1.7%, respectively. This clearly demonstrates that PM can efficiently incorporate prefer- ences provide better performance for data-free method.
Test accuracies are shown in Tables 3 and 4. As can be seen, combining with Pareto Merging results in higher accuracies compared to using the base methods alone. This demon- strates an important advantage of learning the Pareto set, namely that we can select the most suitable model from the
8

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
Table 4. Test accuracies
RESISC45 and SVHN when
merging six VIT-B/32 models. Results on baselines are from (Yang et al., 2024b).
Method
RESISC45
SVHN
Average
445
Task Arithmetic
52.3
49.9
51.1
145
Task Arithmetic+PM
52.4
51.9
52.1
812
Ties Merging
58.7
49.2
53,9
TIES Merging+PM
59.0
51.9
55,5
102
AdaMerging
50.2
60.9
55.5
AdaMerging+PM
52.4
62.6
57.5
AdaMerging+TIIS
52.0
64.9
58.5
AGaMerging+TIESSPPM
52,9
64.7
58.8
650
A54 45.
97.8
16.1
98.5
Figure 6. Models sampled from the Iearned Pareto set by AdaMerg- ing PM when merging 8 ViT-B/32 models. Each subplot shows the accuracies on 2 of the 8 tasks. For comparison, the square denotes the model obtained by AdaMerging.
96.2
ACCUSO
07.5
Table 3. Test accuracies unseen MNIST and EuroSAT when merging six ViT-B/32 models. Results on baselines are from (Yang
al., 2024b).
Method
MNIST
EuroSAT
Average
93.0
24.0
Accaraco RESBCC4
Task Arithmetic
77.2
46.2
61.7
Figure Effects of rank
Task Arithmetic+PM
79.3
46.8
63.0
TIES Merging
75.9
43.3
59.6
TIES Mcrging+PM
80.7
43.4
62.1
may leads to overfitting. Additionally, a higher rank also introduces more parameters," though it is still minimal.
AdaMerging
84.0
56.1
70.0
AdaMerging+PM
84.3
65.1
74.7
AdaMerging+TIIS
83.9
53.5
68.7
5. Conclusion
TTSS
85.9
65.4
75.7
In this paper, we address the challenge of model merging through the lens of multi-objective optimization. We in- troduce Pareto Merging, which enables the generation of an infinite number of Pareto-optimal models based on user preferences with a single merging process and minimal pa- rameter overhead. Experimental results demonstrate that PM can generate different trade-off solutions, better aligning with user preferences compared non- non-preference-and baselines. In the future, we will consider extending the experiment to large language models.
set, while the base methods do not have this freedom.
4.3. Ablation Studies
In this section, perform ablation studies (i) the rank and (ii) using outer product instead of tensor-based model in (7). Because of the lack of space, results on the outer product model can be found in Appendix F.
The default hyperparameter settings are the same as in Sec- tion 4.2. Experiments are performed by using AdaMerging + PM on the two models fine-tuned on RESISC45 and GT- SRB. For each setting, uniformly sample 11 models from the learned Pareto set.
A cknooweegment
This research supported in part by the Research Grants Council of the Hong Kong Special Administrative Region (Grant 16202523 and HKU C7004-22G).
Figure shows the test accuracies with different ranks (r). As can be seen, increasing r improves model performance in general, as higher rank allows the model greater flexibility to adapt to various preferences. However, very large rank
'For 4/8/16/32, the total number of parameters is in- creased by 055/00%%%///5/ /1.0% compared the pre-trained model.

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
Impact Statement
Dimitriadis, N., Frossard, P., and Fleuret, F. Pareto manifold learning: Tackling multiple tasks via ensembles of single- task models. In International Conference Machine Learning, 2023.
This paper aims to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which must be specifically highlighted here.
Dimitriadis, N., Frossard, P., and Fleuret, F. Pareto low-rank adapters: Efficient multi-task learning with preferences. Preprint arXiv:2407.08005 2024.
References
Ainsworth, S., Hayase, J., and Srinivasa. S. Git Re-Basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations, 2023.
Dosovitskiy, A. and Djolonga, J. You only train once: Loss- conditional training of deep networks. In International Conference on Learning Representations, 2020.
Boyd, S. P. and Vandenberghe, L. Convex Optimization. Cambridge University Press, 2004.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.. Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Cai, W., Jiang, J., Wang, F.. Tang, J., Kim, S., and Huang, J. A survey mixture of experts. Preprint arXiv:2407.06200 2024.
Chen, L., Saif, A., Shen, Y., and Chen, T. Ferero: A flexible framework for preference-guided multi-objective learning. In Neural Information Processing Systems, pp. 18758-18805, 2024.
Fliege, J. and Svaiter, B. F. Steepest descent methods for multicriteria optimization. Mathematical Methods of Op- erations Research, 51(3):479-445 494, 2000.
Grandvalet, Y. and Bengio, Y. Semi-supervised learning by entropy minimization. In Neural Information Processing Systems, 2004.
Chen, W. and Kwok, J. Multi-objective deep learning with adaptive reference vectors. Advances in Neural Informa- tion Processing Systems, 35:32723-32777 2022.
He, K., Zhang, X., Ren, S., and Sun, Delving deep into rectifiers: Surpassing human-level performance ima- genet classification. In IEEE International Conference Computer Vision, 2015.
Chen, W. and Kwok, J. Efficient Pareto manifold learning with low-rank structure. In International Conference Machine Learning, 2024.
Chen, W. Zhang, X., Lin, B., Lin, X., Zhao, H., Zhang, Q. and Kwok, J. T. Gradient-based multi-objective deep learning: Algorithms, theories, applications, and beyond. Preprint arxiv:2501. 10945, 2025.
He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Paltern Recognition, 2016.
He, X., Li, C., Zhang, P., Yang, J., and Wang, X. E. Parameter-efficien fine-tuning for vision transformers. Preprint arXiv:2203.16322 2022.
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 16(10)):865-1888 2017.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In IEEE Conference on Computer Vision and Pattern Recognition, 2014.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Daheim, N., Möllenhoff, T., Ponti, E., Gurevych, I., and Khan, M. E. Model merging by uncertainty-base gradi- ent matching. In International Conference on Learning Representations, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. Preprint (xXiv:210600966 2021.
Désidéri, J.A.
Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. Comptes Ren- dus Mathematique, 350(5-6):313-3 2012.
Huang, C., Ye, P., Chen, T., He. T., Yue, X., and Ouyang, W. EMR-merging: Tuning- free reg-operformanc model merging. Preprint arXiv:2405. 2024.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient finetuning of quantized LLMs. In Neural Information Processing Systems, 2024.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, Editing models with task
10

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
arithmetic. In International Conference on Learning Representations, 2023.
Liu, S., James, S., Davison, A., and Johns, E. Auto-Lambda: Disentangling dynamic task relationships. Transactions Machine Learning Research, 2022.
Jin, X., Ren, X., Preotiuc-Pietro D., and Cheng, P. Data- less knowledge fusion by merging weights of language models. In International Conference on Learning Repre- sentations, 2023.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-. F., Cheng, K.-T., and Chen, M.-H. DoRA: Weight-decomposed low-rank adaptation. Preprint arXiv.2402.09353, 2024.
Jordan, K., Sedghi, H., Saukh, O., Entezari, R., and Neyshabur, B. REPAIR: Renormalizing permuted activa- tions for interpolation repair. In International Conference Learning Representations, 2023.
Lu, Z., Fan, C., Wei, W., Qu, X., Chen, D., and Cheng, Y. Twin- wwin-merging: Dynamic integration of modular expertise model merging. Preprint arXiv:2406.1547, 2024.
Mahapatra, D. and Rajan, V. Multi-task learning with user preferences: Gradient descent with controlled ascent in Pareto optimization. International Conference on Ma- chine Learning, pp. 1197-660,,2021
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. Preprint arXiv: 1412.6980, 2014.
Krause, J., Stark, M. Deng, J., and Fei-Fei, L.. 3D ob- ject representations for fine-grained categorization. In International Conference on Computer Vision Workshop, 2013.
Matena, M. S. and Raffel, C. A. Merging models with Fisher-weighted averaging. In Neural Information Pro- cessing Systems, 2022.
LeCun, Y. The MNIST database of handwritten digits, 1998. http:/ /yann/ lecun. com/ exdb/mniii
Miettinen, K. Nonlinear Multiobjective Optimization, vol- ume 12. Springer Science & Business Media, 1999.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(1 (1:21)))22772222) 1998.
Momma, M., Dong, C., and Liu, J. A multi-objective/mulii task learning framework induced by Pareto stationarity. International Conference on Machine Learning, pp. 15895-15907, 2022.
Li, J., Tang, T., Zhao, W. X., Nie, J.-Y., and Wen, J.-R. Pre- trained language models for text generation: A survey. ACM Computing Surveys, 56(9):1-39, 2024a.
Mukai, H. Algorithms for multicriterion optimization. IEEE Transactions on Automatic Control, 25(2): 1980.
Li, L., Zhang, T., Bu, Z., Wang, S., He. H., Fu, J., Wu, Y., Bian, J., Chen, Y., and Bengio, Y. MAP: Low-compute model merging with amortized Pareto fronts via quadratic approximation. Preprint ar arXivv2406.07529, 2024b.
Navon, A., Shamsian, A., Fetaya, E., and Chechik, G. Learn ing the Pareto front with hypemeeworkss. In International Conference on Learning Representations, 2021.
Navon, A., Shamsian, A., Achituve, I., Maron, H., Kawaguchi, K., Chechik, G., and Fetaya, E. Multi-task learning as a bargaining game. International Confer- ence Machine Learning, 2022.
Lin, X., Zhen, H.-L.,, Li, Z., Zhang, Q.-E,, and Kwong, S. Pareto multi-task learning. In Neural Information Processing Systems, 2019.
Lin, X., Yang, Z., Zhang, Q.. and Kwong. S. Controllable Pareto multi-task learning. Preprint arXiv:2010.06313, 2020.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsu- pervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Lin, X., Yang, Z., Zhang, X., and Zhang, Q. Pareto set learn- ing for expensive multi-objective optimization. In Neural Information Processing Systems, pp. 19231-19247, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Lin, X., Zhang, X., Yang, Z., Liu, F., Wang, Z., and Zhang, Q Smooth Tchebycheff scalarization for multi-objective optimization. In International Conference on Machine Learning, 2024.
Rame, A., Couairon, G., Dancette, C., Gaya, J.-B., Shukor, M., Soulier, L., and Cord, M. Rewarded soups: To- wards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Neural Information Processing Systems, 2024.
Liu, B., Liu, X., Jin, X., Stone, P., and Liu, Q. Conflict- averse gradient descent for multi-task learning. In Neural Information Processing Systems, 2021.

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
Roy, S., Trapp, M., Pilzer, A., Kannala, J., Sebe, N., Ricci, E., and Solin, A. Uncertainty-guidee source-free domain adaptation. In European Conference on Computer Vision, 2022.
Zhang, X., Lin, X., and Zhang, Q. PMGDA: A preference- based multiple gradient descent algorithm. IEEE Transac- tions Emerging Topics in Computational Intelligence, 2025.
Ruchte, M. and Grabocka, J. Scalable Pareto front approxi- mation for deep multi-objective learning. In International Conference on Data Mining, 2021.
Zhong, Y., Ma, C., Zhang, X., Yang, Z., Chen, H., Zhang, Q. Qi, S., and Yang. Y. Panacea: Pareto alignment via preference adaptation for Ilms. In Neural Information Processing Systems, pp. 75522-75558, 2024.
Sener, O. and Koltun, V. Multi-task learning as multi- objective optimization. In Neural Information Processing Systems, 2018.
Stallkamp, J.. Schlipsing, M., Salmen, J., and Igel, C. The German traffic sign recognition benchmark: A multi- class classification competition. In International Joint Conference on Neural Networks, 2011.
Stoica, G., Bolya, D., Bjorner, J. B., Ramesh, P., Hearn, T., and Hoffman, Ziplt! Merging models from different tasks without training. In International Conference on Learning Representations, 2024.
Tang, A., Shen, L., Luo, Y., Liu, S., Hu, H., and Du, B. Towards efficient Pareto set approximation via mixture of experts based model fusion. Preprint arXiv:2406.09770, 2024.
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G.. Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tuning of zero-shot models. IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022.
Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., and Oliva, A. Sun database: Exploring large collection of scene categories. International Journal of Computer Vision, 119:3-22, 2016.
Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M. TIES-Merging Resolving interference when merging models. In Neural Information Processing Systems, 2023.
Yang, E.,, Shen, L., Wang, 7.,, Guo, G., Chen, X., Wang, X., and Tao, D. Representation surgery for multi-task model merging. Preprint arXiv:2402.02705. 2024a.
Yang. E.,, Wang, Z., Shen, L.. Liu. S., Guo, G., Wang, X., and Tao, D. AdaMerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations, 2024b.
Yu, L., Yu, B., Yu, H., Huang, F., and Li, Y. Language models are Super Mario: Absorbing abilities from homol- ogous models as free lunch. In International Conference on Machine Learning, 2024.
12

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
A. Details of the Toy Problem
We use popular toy problem (Liu et al., 2021; Navon et al., 2022) with two objectives (0) and S2(0). The original problem has a parameter 0 RR2 To create a problem with matrix-for parameter. we slightly modify it to 0 e R The problem is then reformulated as follows:
Minimize (1) = c1 (1)))11 + (1 and S2(0) = ())
where ((1)
B and (
h1 (0) = log m (0-(().5(000) g(1) - 7) - tanh (-g(2)), 0.000005)) + 6,
(0) log (max([0.5(-00%) ((0) + 3) tanh (-g(2) + 2/, 0.000005)) + 6,
( = ((-0()) + 7) + 0.1 (- - -) 20,
(2) ((-0(1) 7) 7) 0.1 (- ( 8)2) 20,
-
C1 (0) = max(tanh (0.5 * )(2),, 0) and c2 =max(tanh (-0.5 * ((2)), 0).
We first obtain 0, by optimizing St, and Og by optimizing S2. The average (0) 1 02)/2 serves as the preference-independent base model, which is then fixed. Subsequently, optimize G, A. and B in the preference-dependent model.
The rank r is set to 2. We use the Adam optimizer with a learning rate of 0.001. We use 31 uniformly distributed preference vectors (i.e.. [1, 0J [1/30, 29/30]T, ,... space.
,0..,,,, [0, ). ). We can see that the solutions uniformly cover the entire PF in the objective
B. Detailed Discussion of some Related Works
This section provides detailed comparison of the proposed approach with several related works, highlighting the key differences and advantages of our method.
Comparison with Pareto Set Learning using Low-Rank Structures. While works like (Chen & Kwok, 2024; Dimitriadis et al., 2024) use low-rank structures for Pareto set learning, our primary aim is to improve model merging itself, not use merging for Pareto set learning. We reformulate model merging as a MOO problem to integrate user preferences and propose an efficient preference-awee
tensor structure, different from the weighted LoRA sums in (Chen & Kwok, 2024; Dimitriadis al., 2024). To demonstrate the advantage of the proposed, we adapted (Chen & Kwok, 2024; Dimitriadis et al., 2024) for model merging (eight ViT-B/32 models, setup from Section 4.2.2, Table 2). Results are in Table 5.
Table 5. Comparison with low-rank structures from Pareto set learning (Chen & Kwok, 2024; Dimitriadis et al., 2024) on merging eight V/T-B/32 models.
Method
Structure
Test Accuracy (%)
Parameter Overhead (M)
AdaMerging+PMM (equal)
Ours
84.9
0.61
AduMerging-PM (equal)
Structure from (Chen & Kwok, 2024; Dimitriadis al., 2024)
84.5
4.71
AdaMerging+PM (priority)
Ours
85.5
0.61
AdaMerging-PM (priority)
Structure from (Chen & Kwok, 2024; Dimitriadis al., 2024)
85.2
4.71
The results in Table show that our proposed tensor structure achieves superior performance while substantially reducing the number of additional parameters compared adapting structures from (Chen & Kwok, 2024; Dimitriadis al., 2024) to the model merging task.
Comparison with concurrent LLM Alignment Work. The proposed method differs from the concurrent L.M alignment
work Panacea (Zhong et al., 2024) in three main aspects: (1) Problem Formulation: Panacea (Zhong et al., 2024) targets mutli-objective LLM alignment, while we formulated model merging (data-free and data-based) as MOO, enabling preference-driven model generation and improved performance. (2) Low-Rank Structure: Panacea (Zhong et al., 2024) uses an SVD-LoRA approach. Our novel low-rank tensor structure is more flexible, adaptively learning inter-objective
13

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
relationships. Table 6 compares our structure with Panacea (Zhong et al., 2024) (rank 16 for comparable parameters) on merging eight ViT-B/32 models. Our structure outperforms Panacea, especially with priority preferences. (3) Optimization & Data: Panacea (Zhong et al., 2024) uses labeled data for LLM alignment. We focus model merging with limited (unlabeled) data or no data, addressing potential overfitting via our efficient structure and tensor regularization.
Table 6. Comparison with SVD-LoRA structure from (Zhong et al., 2024) on merging eight ViT-B/32 models.
Method
Model Structure
Test Accuracy (%)
AdaMerging+PM (equal)
Ours
84.9
AdaMerging+PM (equal)
Structure from Panacea (Zhong et al., 2024)
84.1
AdaMerging+PM (priority)
Ours
85.5
AdMMerging+PM (priority)
Structure from Panacea (Zhong et al., 2024)
84.4
Comparison with Concurrent Pareto Set Learning via MoE. Another relevant concurrent work (Tang et al., 2024) focuses on enhancing the efficiency of general Pareto set learning algorithms, goal different from our primary objective of advancing model merging through MOO. Tang et al. (2024) employs task arithmetic merge selected modules from pre-trained models while leaving other components (e.g., MLP layers, both MLP and attention layers) separate. A Mixture-of- (MoE) router is then trained using labeled data to assign weights to these unmerged parts. On the other hand, our method merges all modules into a single model with a small low-rank tensor (approximately 0.5% of pre-trained model parameters) and does not require labeled data.
To further compare with MoE-based fusion in (Tang et al., 2024), below we compare its performance on merging eight ViT-B/32 models. As reported in (Tang et al., 2024), with the use of labeled data, MoF-based fusion (with a final model size of 567M) achieves an average accuracy of 77.2% when only the MLP is unmerged, and 83.5% when all modules unmerged (with final model size 1.02B). On the other hand, method (with only 1 14M parameters and not requiring labeled data) achieves higher accuracy at 85.2%.
C. Experimental Details
C.1. Datasets
The datasets used are summarized in Table 7. All datasets are publicly available, The Cars dataset has custom license restricted non-commercial use. The DTD dataset has a custom license restricted research-only use. EuroSAT under the MIT license. The licenses for the remaining datasets unknown.
Table Summary of the datasets used.
Dataset
Domain
# Classes
# Images
SUN397 (Xiao et al., 2016)
Scene classification
397
108,754
Cars (Krause et al., 2013)
Car classification
196
16,185
RESISC45 (Cheng al., 2017)
Remote sensing scene classification classification
45
31,500
EuroSAT (Helber et al., 2019)
Satellite image classification classification
10
27,000
SVHN (Netzer et al., 2011)
House numbers classification
10
600,000
GTSRB (Stallkamp et al., 2011)
Traffic sign classification
43
51,839
MNIST (LeCun, 1998)
Handwritten digits classification
10
70,000
DTD (Cimpoi al., 2014)
Texture classification
47
5,640
C.2. Training Details
We use the checkpoints of pre-trained and fine-tuned models on eight datasets in (Ilharco et al., 2023). Following (Yang et al., 2024b), we initialize all {Ax M in (8) to 0.3. We employ the Adam optimizer (Kingma & Ba, 2014), with learning rate x 10-3 and momentum parameters 31, 32 set to 0.9 and 0.999, respectively. The batch size is 32. We set the number of optimization steps to 2000 when merging models and 4000 when merging 8 models. We initialize G in (8) to the zero
14

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
tensor, and initialize A and B using the Kaiming uniform distribution (He et al., 2015). We set rank 16, and both the regularization coefficient and distribution parameter p O.1.
For the baseline methods, we use the hyperparameters provided in the original papers and adopt their publicly available official implementations. Specifically, for Task Arithmetic, we set À to 0.3 when merging eight models, as suggested in the original paper. When merging two models, we experiment with À values from {0.1, 0.3, 0.5, 0.7, 0.9} and select À = 0.7, which achieves the best performance on the validation dataset. For experiments on the ViT-B/32 (resp. ViT-L/14) model, we use single NVIDIA A6000 (resp. H800) GPU with 48GB (resp. 80GB) memory. We use Ubuntu 22.04.1 with PyTorch 1.12.
D. Additional Figures of Models Obtained by Pareto Merging
D.I. Complete Plot When Merging Two Models
Figure 8 shows the complete plot when merging two models. Rewarded soups can achieve very diverse solutions, as they retain all original models. However, their performance in the middle region is notably poor. MAP partially improves performance this region through evolutionary optimization with labeled data, but it still falls significantly short compared to AdaMerging PM. Most solutions obtained by Rewarded soups and AdaMerging PM are dominated by solutions obtained by Adaeeggin PM, except some extreme solutions. Note that, in general, users are more likely to prefer model that performs well on priority tasks while maintaining satisfactory performance other tasks. It is unlikely that users would select an extreme solution with slightly improvement on task (e.g, from 95.8% % to 95.9%) while causing significant performance drop in another task (e.g., from 52.1% 1% to 22.4%).
100.0
90.0
Accuracy on GTSAB
70.0
60.0
50.0
Rewarded Soups
40.0
Task Arithmetic
Artthmetic Pareto Merging
MAP
aMerging
Ad AdaMengg
Parata Merging
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
Accuracy on RESISC45
Figure 8, Full plot of accuracies of models obtained by different methods when merging two VIT-B/32 models.
D.2. Figures When Merging Eight Models
Figure 10 shows 30 models sampled from the learned Pareto set by AdaMerging + TIES + Pareto Merging when merging eight VTT-B/32 models. We can see that Pareto Merging obtains a diverse model set. Most models dominate models obtained by the baselines, while none of the models is dominated by models obtained by baselines.
Figures 11 and 12 show 30 models sampled from the learned Pareto set by Task Arithmetic + Pareto Merging and TIES Merging + Pareto merging when merging eight VIT-B/32 models, respectively. We can see that Pareto Merging can achieve diverse trade-off around the base method. All of the models obtained by PM are non-dominated to each other in the 8-dimensional objective space. They are also non-dominated to the model obtained by the base method. Note that "dominance" observed in the 2D projections does not imply the real dominance in the 8-dimensional objective space. Unlike the base method, which offers only single trade-off, PM allows users to select models tailored their specific preferences.
15

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
90.0
Accurary on GTSMB
60.0
40.0
Task Arithmetic Preference (113 AM Paramter Overhead)
Ovemnad]
Task Arithmetic PM (2.1M Paramter Overhead)
30.0
MaMeiigg Preference
Paramter Overhead)
Ad aMeringg PM 10.6M Paramter Overhead)
30.0
40.0
50.0
Accuracy on RESISCA5
Figure 9. Full plot of accuracies of models obtained by two straightforward methods discussed Section 3.1 in when merging two ViT-B/32 models.
E. Test accuracies when merging eight ViT-L/14 models
We further experiment on the larger ViT-L/14 backbone. The results are shown in Table 8. As can be seen, Pareto Merging still outperforms the baselines on most datasets, and the observations on ViT-B/32 model still hold.
Table Test accuracies when merging eight ViT-L/14 models. Results baselines from (Yang al., 2024b). The best result is in hold and the second hest underlined.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Average
Individual
82.3
92.4
97.4
98.1
99.7
94.2
Traditional MTL.
80.8
90.6
96.3
96.3
97.6
99.1
99.6
84.4
93.5
Weight Averaging
72.1
81.6
82.6
91.9
78.2
70.7
97.1
62.8
79.6
Fisher Merging
69.2
88.6
87.5
93.5
80.6
74.8
93.3
70.0
82.2
RegMean
73.3
81.8
86.1
97.0
88.0
84.2
98.5
60.8
83.7
DARE
74.0
81.9
86.5
93.9
88.0
86.6
98.9
65.5
84.4
MAP
76.0
84.1
88.7
87.8
90.1
87.9
97.8
71.3
85.4
Task Arithmetic
73.9
82.1
86.6
94.1
87.9
86.7
98.9
65.6
84.5
Task Arithmetic PM (equal)
74.0
82.1
86.7
94.1
87.9
86.8
98.9
65.7
84.5
Task Arithmetic + PM (priority)
74.3
82.5
87.3
95.0
91.0
88.1
99.1
66.1
85.4
TIES Merging
75.9
85.4
89.0
95.6
89.2
99.0
68.7
86.2
TIES Merging + PM (equal)
75.9
85.4
89.0
95.6
89.1
87.1
99.0
68.7
86.2
TIES Merging PM (priority)
76.0
85.4
89.7
96.7
91.9
88.5
99.1
69.3
87.1
AdaMerging
79.0
90.3
90.8
96.2
93.4
98.0
99.0
79.9
90.8
AdaMerging + (equal)
79.8
90.7
93.5
99.1
80.6
91.2
AdaMerging PM (priority)
80.5
91.7
91.9
98.1
96.4
98.8
99.1
80.8
92.1
AdaMerging + TIES
79.4
90.3
91.6
97.4
93.4
97.5
99.0
79.2
91.0
AdaMerging TIES PM (equal)
79.9
90.6
91.2
97.8
93.4
98.1
99.0
80.3
91.2
AdaMerging + TIES + PM (priority)
80.6
91.7
92.0
98.5
96.1
99.0
99.1
80.6
92.2
F. Using Outer Product in W(r)
In this experiment, we study other possibilities to transform the preference vector 7 to a x matrix using vector outer product (denoted c):
16

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
$7.
36.3
22.0-
21.0
20--
19.0
35.0
00
100
12.0
12.
654
850
36.0
47.
Accuscc on 5.900
Acturacy
RL30044
(b)
25.0
24.0
0..0
0.00
65.0
66.5
$1.
92.5
16.1
46.6
988
Accuncyy on SAMMM
Acconaoy HUST
(c)
(d)
Figure Models sampled from the learned Pareto set by AdaMerging TIES Pareto Merging when merging eight ViT-B/31 models. Each subplot shows the accuracies on of the 8 tasks. For comparison, the square denotes the model obtained by AdaMerging TIES.
Variation 1: W(+) Ay, where E R and A E Raxk
Variation (1)) = Ay o v, where v E gd and R
The experimental settings are the same as in Section 4.3. The obtained results of merging two models fine-tuned on RESISC45 and GTSRB datasets are shown in Figure 13. We observe that both outer product-based method exhibit performance inferior to the proposed tensor-basee method.
17

---

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
333
6.00
Accuroo on EuendA
15.0
24.9
24.
542
$3.0
96.3
15.3
55.4
16.4
16.6
96.9
670
67.2
(b)
73.0
72.0
Accuracy CTD
-2.0
21.0
76-0-
69-0-
72.0
06.3
31.0
03.0
362
27.
271
272
913
50.
(c)
(d)
Figure 11. Models sampled from the Iearned Pareto by Task Arithmetic Pareto Merging when merging eight ViT-B/32 models. Each subplot shows the accuracies on 2 of the tasks. For comparison, the square denotes the model obtained by Task Arithmetic.
66.
Accra o1 Runsaa
$2.0
600
330
590
94.
19.5
56.7
71.2
31.4
71.4
72.1
(b)
25.0-
ACCuBO on C70
240-
230
72.0
60.0
105
$7.0 Accunay
965
67.5
68.
66.5
95.0
16.3
96.
06.3
98.2
66.2
662
602
Accursty
(d)
Figure 12. Models sampled from the leamed Pareto set by TIES Merging Pareto Merging when merging eight ViT-B/32 models. Each subplot shows the accuracies on 2 of the tasks, The square denotes the model obtained by TIES Merging.
18

---

Pareto Merging: Optimization for Preference-Aware Model Merging
Structure
Product (Variation 2)
CTSRa
98.0
95.0
90.0
93.0
Accuracy on RESISC4S
Figure 13. Comparison between the proposed tensor-based method and simple outer product-base based method for drefedenceedeepeddett personalization.
19


Reviews:
[{"review_id": "NvsjFTtnNP", "reviewer": "Reviewer_Pu7q", "summary": "The paper introduces a novel method named \"Pareto Merging,\" which is designed for the efficient merging of multiple pre-trained machine learning models into a single model, taking into account the preferences of different users. The approach learns a set of models, each optimized for different trade-offs or preferences among the objectives, thereby offering customized solutions for varied user preferences.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions. Below, we provide a detailed response to each point.\n\n---\n**Response to Essential References Not Discussed**\n\n**R1: MGDA Reference** Thanks. We'll include it in our final version.\n\n**R2: Relation with preference-based MOO works [1,2,3]** They differ fundamentally from ours. They aim to generate a single solution per run based on a given preference. To handle different preferences, they require multiple runs and obtain multiple models. On the other hand, our method produces a continuous Pareto set of solutions. Once trained, a MOO solution can be generated from the given preference without optimization. Currently, we use smooth Tchebycheff scalarization for its simplicity and effectiveness. But algorithms in [1,2,3] can be integrated into our framework by modifying Equation (10). Specifically, instead of directly using $\\gamma_k$ to weight $S_k$, we can use algorithms in [1, 2, 3] to compute a weighting for $S_k$. \n\n**R3: Using low-rank embedding for the preference** There might be some misunderstanding. We are not using low-rank embedding for preference. Instead, we employ a low-rank tensor to integrate the preference into a single model, which is significantly more parameter- and compute-efficient compared to generating a separate model for each preference (using methods such as [1, 2, 3]) or use a full-rank tensor structure.\n\n**R4: Difference with Pareto set learning works [4, 5]**\nWe would like to emphasize that our contributions are:\n1. Problem formulation: We address the challenges of model merging by reformulating it as a MOO problem. This requires identifying the MOO objectives for model merging, and is not trivial. In data-free merging, we observe that Task Arithmetic can be viewed as optimizing the distances between models in the weight space. This then allows us to define the MOO objectives based on these distances.\nMoreover, while traditional model merging considers finding a single solution for a specific preference, our formulation transforms the problem to the finding of a continuous space of solutions, which is novel.\n\n2. Scalable Pareto Set Learning: While works [4, 5] use hypernetworks (typically 100× larger than base networks), our low-rank tensor approach dramatically reduces the computational costs, making Pareto exploration feasible for large-scale models.\n\n**R5: Comparison with LLM alignment work [6]**\nNote that [6] is indeed a concurrent work with 3 key differences:\n1. Problem Formulation:\n[6] is used for LLM alignment, while this paper is the first to apply MOO to model merging (both data-free and data-based). As mentioned in **R4**, it is non-trivial. As can be seen from empirical results, this leads to the ability\nto generate different models for different preferences\nand improved performance compared to the baseline algorithms. \n2. Efficient Low-Rank Tensor Structure: \n[6] uses a SVD-LoRA-based approach, while we propose a low-rank tensor structure.  In the following, we adapt the approach in [6] for use in our model merging setting. We set the rank in their approach to 16, so that its number of parameters is comparable with ours.\nUsing the setup in Section 4.2.2 and Table 2, the results on merging eight ViT-B/32 models are: \n|Method| Model|Test accuracy|\n|---|---|---|\n| AdaMerging+PM (equal)|ours | 84.9|\n| AdaMerging+PM (equal) | structure in [6]|84.1| \n| AdaMerging+PM (priority)| ours | 85.5|\n| AdaMerging+PM (priority) | structure in [6]|84.4|   \n\nAs can be seen, our low-rank tensor structure outperforms [6], particularly in the priority preference setting. This is because \nwhile [6] assigns a fixed singular matrix row to each objective, our tensor $G$ adaptively learns the relationships between objectives and so our structure is more flexible.\n\n3. Optimization: [6] focuses on LLM alignment using labeled data, whereas we focus on model merging with either no data or unlabeled data. This distinction presents overfitting challenges, which we address using an efficient structure and tensor regularization.\n\nWe will include the above discussions in the final version.\n\n**Response to Weakness 1**\n\n**R6: Training and inference times** Table 1 is on training time. As can be seen, our method has small computational overhead compared to the baselines. Specifically, during training, for a layer, AdaMerging optimization has a per-iteration complexity of $O(Kcd)$, while ours is $O(Kcd + (c+d)r + Kr^2)$ (where $K$ is the number of models, $c \\times d$ the shape of the layer parameter, and $r$ is the rank.). As $r$ is much smaller than $c$ and $d$, the computational overhead is small.\n\nFor inference, once the preference is fixed, the low-rank tensor can be merged into the base model, resulting in no inference overhead.\n\n**Response to Weakness 2 and Questions For Authors**\n\nPlease refer to **R5**\n\n---\nWe hope the above responses address your concerns. If you have any additional questions or suggestions, we are more than happy to discuss them further.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "No issues.", "relation_to_broader_scientific_literature": "For model merging, this work considers preference-aware merging. The learned model with low-rank embedding can generate a Pareto set of merged models, with each representing a Pareto-optimal solution for a preference.", "essential_references_not_discussed": "- The reference for MGDA is not accurate. Check the earlier work below.\n\n[0] \"Steepest descent methods for multicriteria optimization\". Mathematical methods of operations research 2000.\n\n- It would be better if the authors could discuss the relation with some other gradient-based preference-based multi-objective optimization works, as listed below. For example, how does the preference in this paper differ from the existing works? What are the pros and cons of using low-rank embedding for the preference?\n\n[1] \"A multi-objective/multi-task learning framework induced by Pareto stationarity\" ICML, 2022.\n\n[2] \"FERERO: A flexible framework for preference-guided multi-objective learning\" NeurIPS 2024.\n\n[3] \"PMGDA: A preference-based multiple gradient descent algorithm\" arXiv:2402.09492, 2024.\n\n- A detailed comparison should be made to some existing works on Pareto set learning or preference-conditioned models, some examples are listed below. Based on the discussion in Section 2.2, it seems that the only difference compared to the prior works for preference-conditioned models is that the authors \"explore model merging for large models\".\n\n[4] \"Smooth Tchebycheff scalarization for multi-objective optimization\" ICML 2024.\n\n[5] \"Pareto set learning for expensive multi-objective optimization\" NeurIPS 2022.\n\n[6] \"Panacea: Pareto Alignment via Preference Adaptation for LLMs\" NeurIPS 2024.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well-written, and the method is clearly described.\n\n2.  The proposed method is validated through some experiments.\n\n\n**Weaknesses**\n\n1. Does Table 1 provide the training or inference time? The authors should discuss the computational complexity in both training and inference and how it compares to some baselines.\n\n2. I do not see a clear novelty compared to the reference [6] in **Essential References Not Discussed** section. This paper also seems to use a low-rank embedding for the preference. Same as [6], this paper can also generate infinite number of models.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Based on the discussion in Section 2.2, it seems that the only difference compared to the prior works for preference-conditioned models is that the authors \"explore model merging for large models\" while the prior works focus on smaller models. However, the reference [6] in **Essential References Not Discussed** section is very similar to the approach proposed in this paper and also applies to large models.\nHow is this work different from [6]?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No."}}, {"review_id": "teHNiKaWiK", "reviewer": "Reviewer_i5h5", "summary": "This paper introduces a new method for merging models with trade-offs by finding the Pareto front. They included both data-free version and using unlabeled data version of the method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions. Below, we provide a detailed response to each point.\n\n---\n\n**Response to Questions For Authors**\n\n**R1: Parameter count in Table 1**\nThe \"parameter count\" refers to the number of parameters that need to be stored after merging. \nNote that for each merging method, there are two options on what to store. For instance, with two models in the soup and 100 user preferences, one can either store the 100 merged models corresponding to these 100 preferences, or store only the two original models and merge them on-the-fly based on the user preference. Obviously, the latter is more memory-efficient, and is we adopt when calculating the parameter count. Hence, for MAP and the Rewarded Soups, the parameter counts are doubled in the two-model case. \n\n**R2: Number of solutions in Table 1**\nThank you for pointing this out. In the paper, we mentioned 100 as the original NSGA-III algorithm enforces a fixed population size by removing dominated or overcrowded solutions during evolution. Upon double checking the MAP implementation, we observed that unlike the original NSGA-III, it retains all the non-dominated solutions, resulting in around 500 solutions at the end. We will correct this in the final version. However, please note that since we used the official implementation from the authors' github in the experiments,\nthis does not affect any experimental results or discussions in the paper.\n\n---\n\nWe hope the above responses address your concerns. If you have any additional questions or suggestions, we are more than happy to discuss them further.", "claims_and_evidence": "Most of the claims are clear. I've detailed unclear points in the following comments.", "methods_and_evaluation_criteria": "Yes, they are standard baselines in the model merging literature, and the paper compared with various suitable baseline methods.", "theoretical_claims": "- There is no theoretical claims.", "experimental_designs_or_analyses": "Yes, they are standard baselines in the model merging literature, and the paper compared with various suitable baseline methods.", "relation_to_broader_scientific_literature": "This paper is relevant to the model merging community and for preference-aware model merging. Prior work in the field include MAP (model merging with amortized pareto front), and methods in MTL, such as ParetoMTL.", "essential_references_not_discussed": "I did not identify any essential references not discussed.", "other_strengths_and_weaknesses": "Strengths: \n- A new algorithm for identifying pareto fronts for model merging. \n- Compared with various existing baselines. \n- Included both data-free and unlabeled data versions of the method. \nWeakness: \n- A few minor errors as I pointed out in questions.", "other_comments_or_suggestions": "NA", "questions_for_authors": "- In Table 1, how is the # parameters defined? I'm not sure why rewarded soup and MAP would double the parameter counts. Could you please clarify?\n- In Table 1, the authors claim that the number of models MAP can generate is 100, because the initial population is 100. I don't think this is true because MAP uses NSGA III and the number of points on the final Pareto front can far exceed the initial population size.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wn5CMcubj2", "reviewer": "Reviewer_FTzt", "summary": "This paper proposes a novel preference-aware multi-objective model merging method called Pareto Merging (PM) to generate a Pareto set of merged models (the number of models might be infinite) by a single merging process. The main contributions include 1) the preference-aware multi-objective model merging formulation and 2) the LoRA-based personalized model generation method with minimal parameter overhead. Experimental results show the proposed Pareto merging method can achieve state-of-the-art performance on different model merging problems with ViT-B/32 models.\n\n\n**##After Rebuttal Comment##**\n\nThank you for your detailed response and new results. Since all my concerns have been adequately addressed, I raise my rating to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions. Below, we provide a detailed response to each point.\n\n---\n\n**Response to Claims And Evidence**\n\n**R1: Comparison with [1]** Note that [1] is indeed a concurrent work of ours but with different goal. [1] aims to improve the efficiency of general Pareto set learning algorithm. Instead of training from scratch, [1] employs task arithmetic to merge specific modules while keeping others (e.g., MLP layers or both the MLP and attention layers) separate. They then train a MOE router using labeled data to weight these unmerged components.  Note that when both MLP and attention layers are not merged, [1] needs to keep the $K$ models after training.\n\nOn the other hand, our method focuses on improving model merging and formulates it as a MOO problem. This extension has not been studied before (including [1]). Unlike [1], our method merges all modules while incurring a small low-rank tensor (with only 0.5\\% parameters of the pretrained model).\n\nNote that formulating model merging as a MOO problem requires first identifying the MOO objectives for model merging, which is not trivial. For instance, in data-free merging, we observe that Task Arithmetic can be viewed as optimizing the distances between models in the weight space. This then allows us to define the MOO objectives based on these distances. While traditional model merging considers finding a single solution based on a specific preference, our MOO formulation transforms the problem to the finding of a continuous space of solutions, which is novel.\n\nTo further compare with MoE-based fusion in [1], below we compare its performance on merging eight ViT-B/32 models using the same setup as in [1]. As reported in [1], with the use of labeled data, MoE-based fusion (with a final model size of 567M) achieves an average accuracy of 77.2\\% when only the MLP is unmerged, and 83.5\\% when all modules are unmerged (with final model size 1.02B). On the other hand, our method (with only 114M parameters and NOT requiring labeled data) achieves a higher accuracy at 85.2\\%. We will add the above discussion to the final version.\n\n**Response to Methods And Evaluation Criteria**\n\n**R2: Comparison with LoRA-based Pareto Manifold leanring [2, 3]**  Thanks for your comment but that is not the key difference. As mentioned earlier, our primary objective is to improve model merging, rather than using model merging for efficient Pareto set learning. Our first key contribution is identifying the limitations of current model merging techniques and reformulating it as a MOO problem to effectively incorporate user preferences.  Furthermore, we propose an efficient preference-aware tensor structure, which is different from the weighted sum of LoRAs used in [2, 3]. \n\nTo further illustrate our advantages, we adapt [2,3] for use in our model merging setting. Using the setup in Section 4.2.2 and Table 2, the results on merging eight ViT-B/32 models are:\n\n|Method|Structure|Test Accuracy|Parameter Overhead|\n|---|---|---|---|\n| AdaMerging+PM (equal)|ours|84.9|0.61M|\n| AdaMerging+PM (equal)|structure in [2,3]|84.5|4.71M|\n| AdaMerging+PM (priority)|ours|85.5|0.61M|\n| AdaMerging+PM (priority)|structure in [2,3]|85.2|4.71M|   \n\nAs can be seen, the proposed tensor structure has better performance while significantly reducing the number of parameters. We will add the above discussion to the final version.\n\n**R3: Real-time trade-off adjustment & Merge for each preference independently** The proposed method can be used in both scenarios: (1) As you mentioned, it is promising for real-time trade-off adjustment. (2) When different users have different preferences, the straightforward preference incorporation method, as discussed in Section 3.1, has two limitations: (i) It requires storing all $K$ original models to address $n$ different user preferences. In contrast, our approach only requires a single model and a small low-rank tensor. (ii) For methods like AdaMerging, they require $n$ separate runs for $n$ preferences. For instance, handling 100 preferences with straightforward method would require $0.15 \\times 100 = 15$ GPU hours, whereas our method requires only $0.28$ GPU hours. \n\n**Response to Experimental Designs Or Analyses**\n\nFor comparison with [1], please refer to **R1**.\n\n**R4: Comparison with other scalarization methods** Thanks for your question. Following your suggestion, we add experiment with different scalarization methods on merging eight ViT-B/32 models (as in Section 4.2.2 and Table 2). The average accuracies obtained are: (i) linear scalarization: 84.3\\%; (ii) Tchebycheff scalarization: 82.8\\%; and (iii) smooth Tchebycheff scalarization: 85.2\\%. These indicate that smooth Tchebycheff scalarization outperforms the other methods for our model merging task. We will add the discussion to the final version.\n\n---\n\nWe hope the above responses address your concerns. If you have any additional suggestions, we are more than happy to discuss them further.", "claims_and_evidence": "Most claims in this work are well supported by clear and convincing evidence. \n\nHowever, I have some concerns about the claim that \"to the best of our knowledge, we are the first to utilize gradient-based MOO for model merging.\" To my understanding, there is a closely related work [1] that also investigates gradient-based MOO for model merging. A detailed discussion and comparison with [1] is required. In addition, this claim should be modified if needed.\n\n[1] Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion, arXiv:2406.09770.", "methods_and_evaluation_criteria": "I believe the proposed method and the evaluation criteria both make sense for the multi-objective model merging problem. I have the following concerns about the proposed method.\n\n1. To my understanding, the proposed Pareto Merging method has a similar model structure to the previous work on LoRA-based Pareto manifold learning [2, 3] (see Figure 2). The key difference is that PM uses a model merging approach to find the preference-independent based model, while previous work learns the base model. The pros/cons between these two approaches are not very clear to me. A detailed discussion could be very helpful to highlight the unique advantage of the proposed Pareto Merging approach.\n \n2. The motivation of this work is to use the Pareto Merging method to find different models for different users rather than a single preference-conditioned model to adjust the trade-off among different objectives in real time. To my understanding, the proposed preference-independent based model + preference-dependent low-rank tensor structure could be promising for real-time trade-off adjustment. However, if the goal is to find different models for different users, why not just use model merging to find the whole model for each user once the user's preference is known? In what situation will we need to find a set of models for different users in parallel?\n\n[2] Efficient Pareto Manifold Learning with Low-Rank Structure, ICML 2024.\n\n[3] Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences, ICLR 2025.", "theoretical_claims": "There is no theoretical claim or proof in this work.", "experimental_designs_or_analyses": "I've checked the experimental designs and analyses, and believe most of them are reasonable. I have the following concerns about the experiments.\n\n1. As mentioned in the above sections, a comparison with the closely related work [1] could be helpful. \n\n2. This work uses the smooth Tchebycheff scalarization for multi-objective optimization. It is interesting to know whether this method can truly outperform the simple linear scalarization and the original Tchebycheff scalarization in the model merging task.", "relation_to_broader_scientific_literature": "The proposed Pareto merging method is a natural extension of the previous methods on learning the entire Pareto set by a single model. I believe this extension is meaningful and valuable since the model merging method could be very important for real-world applications, especially those with large foundation models.", "essential_references_not_discussed": "As mentioned in the previous section, a closely related work on multi-objective model merging [1] is not discussed in this work.", "other_strengths_and_weaknesses": "Strengths:\n\n1. This work is generally well-written and easy to follow.\n\n2. Multi-objective model merging is important for many real-world applications, espeically those with large foundation model. This work is a timely contribution on an important research direction. \n\n3. The proposed Pareto merging method achieves state-of-the-art performance on different experiments.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please address my concerns raised in the above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KDpBemArkl", "reviewer": "Reviewer_6Su3", "summary": "The authors use multi-objective optimization to create a framework for users to give preferences to which tasks are important to them when merging models. Given a merged model, they build another PEFT architecture that takes preference weights as input This new PEFT is trained either data-free or using unlabeled data. Experiments are done on toy examples and merging 8 image data sets.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions. Below, we provide a detailed response to each point.\n\n---\n**R1: Direct incorporation of preference to Task Arithmetic and AdaMerging**\nIn Section 3, we frame model merging as a MOO problem and propose both straightforward methods (Section 3.1) and our Pareto Merging (PM) approach (Sections 3.2-3.3). While straightforward methods incorporate preferences, they have significant limitations. Our PM method is more parameter-efficient and computationally effective, achieving better performance trade-offs.\n\nFollowing your suggestion, we tested Task Arithmetic + Preference and AdaMerging + Preference. Results (available at https://anonymous.4open.science/r/ICML-0464/) show:\n\n* Task Arithmetic + Preference has 113.4M parameter overhead and some extreme solutions obtained are typically undesirable for users. Our Task Arithmetic + PM approach with the default configuration (low-rank tensor applied only to attention layers) achieves reasonable trade-offs with 0.6M parameter overhead. Extending low-rank tensors to both attention and MLP layers shows better performance with only 2.1M parameters. Users can adjust settings to balance solution diversity.\n\n* AdaMerging + PM (0.6M parameter overhead) achieves better trade-offs than AdaMerging + Preference (113.4M parameter overhead). Moreover, our approach requires only a single run (0.28 GPU hours), whereas AdaMerging + Preference demands 11 runs (1.65 GPU hours).\n\nWe will include the above disscussion in the final version.\n \n**R2: Task arithmetic and task arithmetic+PM(equal) perform almost identical, while AdaMerging+PM(equal) significantly outperform AdaMerging** \nTask arithmetic is data-free merging, so task arithmetic+PM(equal) has no additional information, resulting in similar performance. AdaMerging+PM leverages unlabeled data, allowing PM to better resolve task conflicts. This pattern of improvement with data incorporation appears consistently across various compression techniques, such as merging, pruning, and quantization. \n\n**R3: How much training is needed**\nWe apologize for the omission of detail. We use 2,000 gradient steps. As shown in Table 1, the additional training time is small.\n\n**R3: ViT-L/14 results**\nThe improvement needs to be considered with respect to traditional MTL, which serves as a performance upper bound. As can be seen from Table 6, with ViT-L/14, AdaMerging already achieves 90.8% (vs. 80.1% with ViT-B/32), naturally limiting improvement margins. Still, AdaMerging+PM (priority) achieves a significant 1.3% improvement over AdaMerging. This smaller margin reflects approaching optimal performance rather than method limitations.\n\n**R4: Computational overhead** As shown in Table 1, our computational overhead during training is small.\nfor data-free merging (with task arithmetic), the overhead is 0.04 GPU hours; whereas for merging based on unlabeled data (with AdaMerging), the overhead is 0.13 GPU hours.\n\n**R5: Priorities in Image Classification and Extension to LLM**\nThere seems to be a misunderstanding. Setting priorities does not mean that only one task is important while all other tasks are unimportant. Rather, it means that the model should excel at a specific task while still achieving acceptable performance on others. Maintaining separate models for different preferences is inefficient in terms of parameters, as it requires storing and managing multiple full models.\n\nOur Pareto Merging enables a single model to excel at prioritized tasks while maintaining reasonable performance across all tasks, with minimal parameter overhead. Additionally, our method supports flexible adjustment of prioritized tasks. We recognize the exciting potential of this approach for LLMs and are actively working on extending our algorithm to these models. However, due to time constraints, we can only include this extension in the final version of the paper.\n\n**R5: Preference Vector Sampling** Thanks for your question. We optimize the expected loss across the entire preference space to account for all possible user preferences. Different preferences weight the parameters in the low-rank tensor differently, enabling the model to capture diverse characteristics. While our method already demonstrates strong performance, we acknowledge that developing more effective preference vector sampling techniques is an interesting future work that could further enhance efficiency.\n\n---\n\nWe hope the above responses address your concerns. If you have any additional questions or suggestions, we are more than happy to discuss them further.", "claims_and_evidence": "The results do show improvements based on Pareto Merging, but I think some of what is demonstrated by Table 2 is problematic. It sticks out that Task Arithmetic and Task Artihmetic+PM(equal) perform almost identical. The weights in Task Arithmetic and other merging algorithms can also be selected to prioritize a single task in similar way that the preference vector is set for PM. \n\nThere is also missing insight. Why does AdaMerging+PM(equal) significantly outperform AdaMerging? The framework adds an additional PEFT module on top, so there is more complexity in the model. This also requires additional training and it is not clear how much training is needed. For an 8-dimensional preference vector that is sampled once per batch, it would seem that would require many epochs to sample the space of preference vectors. I didn't find any information about how many epochs are necessary in the Appendix. \n\nAn additional problem is that Table 6 in the Appendix using ViT-L/14 does not confirm the same significant outperformance claimed in Table 2. The main difference I see is that the individual models with ViT-L/14 significantly outperform the ViT-B/32 models (on average). And with better models, all merging models perform much better and the benefit of PM is not nearly as apparent. With a smaller benefit, the additional training requires more discussion.", "methods_and_evaluation_criteria": "Most evaluations make sense, but additional evaluations does Task Arithmetic and other merging methods by using the weights to assign priority would help see if the preference gives a benefit.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "No issue", "relation_to_broader_scientific_literature": "The contributions attempt to improve the state of the art in the literature.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The main strength is that users can give preference to different models with very little overhead in terms of memory. But again, a weakness is that this could involve a lot of computational resources in terms of training. Furthermore, when considering priority, it is not clear when one would have priorities in image classification. If we have a priority in image classification, we would best served to use the individual model. The authors need to discuss when this is applicable in model merging for images. I can see where this would be useful for LLMs with model alignment but that is left for future work in the Conclusion. The approach with sampling preference vectors requires more discussion and analysis. What is the insight for optimizing the average loss with respect to preference vectors? See above for additional weaknesses in Claims and Evidence.", "other_comments_or_suggestions": "See above", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a preference-aware model merging based on multi-objective optimization problems. The experimental result for mering ViT models demonstrates that the proposed method can achieve better performance than the baseline methods. The paper is well-written, and the effectiveness of the proposed method is validated through experiments.

I would recommend that the authors include a discussion regarding related work discussed in the rebuttal. Also, it would be better to elaborate on the scope and target of the proposed method pointed out in the reviewer's comments and how to extend the proposed method to LL merging. I would also like to see the performance of the proposed method in merging LLMs if possible.

In summary, I think that this paper has a meaningful contribution to the community. I would recommend accepting this paper.
Decision: Accept (poster)