MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation
Authors: Lu Li
Tianyu Zhang
Zhiqi Bu
Suyuchen Wang
Huan He
Jie Fu
Yonghui Wu
Jiang Bian
Yong Chen
Yoshua Bengio
Venue: ICLR 2025 Poster (2025)
Keywords: model merging
transfer learning
multitask learning
task arithmetic
multi-objective optimization
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We provide a computation-efficient algorithm for finding the Pareto front representing the trade-offs during model merging caused by conflicting objectives between different tasks.

Abstract: Model merging has emerged as an effective approach to combining multiple single-task models into a multitask model. This process typically involves computing a weighted average of the model parameters without additional training. Existing model-merging methods focus on improving average task accuracy. However, interference and conflicts between the objectives of different tasks can lead to trade-offs during the merging process. In real-world applications, a set of solutions with various trade-offs can be more informative, helping practitioners make decisions based on diverse preferences. In this paper, we introduce a novel and low-compute algorithm, Model Merging with Amortized Pareto Front (MAP). MAP efficiently identifies a Pareto set of scaling coefficients for merging multiple models, reflecting the trade-offs involved. It amortizes the substantial computational cost of evaluations needed to estimate the Pareto front by using quadratic approximation surrogate models derived from a preselected set of scaling coefficients. Experimental results on vision and natural language processing tasks demonstrate that MAP can accurately identify the Pareto front, providing practitioners with flexible solutions to balance competing task objectives. We also introduce Bayesian MAP for scenarios with a relatively low number of tasks and Nested MAP for situations with a high number of tasks, further reducing the computational cost of evaluation.

Published as a conference paper at ICLR 2025
MAP: LOW-COMPUTE MODEL MERGING WITH AMORTIZED PARETO FRONTS VIA QUADRATIC APPROXIMATION
Lu Li Tianyu Zhang2.", Zhiqi Bu Suyuchen Wang Huan He', Jie Fus, Jiang Bian', Yonghui Wu', Yong Chen', Yoshua Bengio*
University of Pennsylvania MILA ServiceNow 4 "Amazon AGI 5 Université de Montréal HKUST University of Florida
lulil@sas, upenn. edu, (Huan.He ychen1231 1 upenn. edu, (tianyu. zhang, yoshua beshaa.bengio bengio @mila. quebec, (rodyx218@@maii com, suyuchen, beeyuuchn.wanguuree reng ca, fujielust. hk, (bianjiang, yonghui. wutuuiiwwwuue @u))uflledd
ABSTRACT
Model merging has emerged as an effective approach to combining multiple single- task models into a multitask model. This process typically involves computing a weighted average of the model parameters without additional training. Existing model- model-merging methods focus on improving average task accuracy. However, inter- ference and conflicts between the objectives of different tasks can lead to trade-offs during the merging process. In real-world applications, a of solutions with various trade-off's can be more informative, helping practitioners make decisions based on diverse preferences. this paper, we introduce novel and low-compute algorithm, Model Merging with Amortized Pareto Front (MAP). MAP effi- ciently identifies a Pareto set scaling coefficients for merging multiple models, reflecting the trade-off's involved. It amortizes the substantial computational cost of evaluations needed to estimate the Pareto front by using quadratic approximation surrogate models derived from preselected mental results on vision and natural language processing tasks demonstrate that MAP can accurately identify the Pareto front, providing practitioners with flexible solutions to balance competing task objectives. We also introduce Bayesian MAP for scenarios with relatively low number of tasks and Nested MAP for situations with a high number of tasks, further reducing the computational cost of evaluation.
of scaling coefficients. Experi-
INTRODUCTION
Large are-trained foundation models have become widely available for many real- world applications (Wornow et al. 2023; Thirunavukarasu er al., 2023 Cui et al.) 2024). This increasing availability has led popular practice of fine-tuning these models adapt them wide range of downstream tasks. Practitioners can independently fine-tune the same pre-trained model, such as CLIP--tyle
models (Radford et al.) 2021: Wu al. [2023] Zhai et al. 2023), large language models (Brown) et al., [2020] Rozière et al., 20233 Touvron et al, 2023) Jiang et al. 2024), etc., and then release the fine-tuned models without disclosing the training data. the deployment of such fine-tuned models increases, combining models with identical architectures and initializations has emerged as promising approach to combine their respective capabilities. This useful, especially in scenarios where the training data for each task is private and cannot be shared, such as individual-level patient data in hospitals and behavioral data social media recommendation systems.
Existing methods for merging models typically involve calculating weighted average of the pa- rameters from multiple models to enhance performance uniformly across various tasks. However, this approach often overlooks the conflicts among the diverse objectives of these tasks, which can
Equul contribution.

---

Published as a conference paper at ICLR 2025
Task vectors
Model merging
(a)
Evaluation
%/
Oare
Opca
Very few
(G,,5) (2)
-0.4
12
Byre? Σ
99
-6.8 di
125
Quadratic
Approximation
(c)
Pareto fronts
(b)
Surrogate model
122
NSGA-III
6.20
6.0
Lats of (8, R,)
6.5
3.00
215
-315 -630 -655 508
eis
abe <la
Figure 1: Illustration of the overall process of MAP for the case of two tasks, Step 1: Select tasks and compute their corresponding task vectors. Step 2: Sample few scaling coefficients and query the evaluation metrics for each task, respectively. Step 3: Use the quadratie model as a surrogate model to approximate the mapping с + metrics. Step 4: Use the NSGA-III algorithm with the surrogate objective functions to find amortized Pareto fronts. (a): Contour plot of the actual accuracy landscape for the ViT-B/32 model (Dosovitskiy et al. 2020)) obtained from 100 scaling coefficients (sampled uniformly) evaluated on the SUN397 (Xiao et al., 2016) and Cars (Krause et al.) 2013) datasets. (b): Contour plot of the fitted quadratic functions. Red lines represent the Pareto front in the decision variable (c1, C2) space. (c): Example of the resulting Pareto fronts. The Pareto front (Grid search) is regarded as the ground truth given the sufficient number of grid points evaluated. The Pareto front (MAP, predicted) is the amortized Pareto front. The Pareto front (MAP, real) corresponds to the same (()) but re-evaluated to obtain the ground truth metrics for comparison. The yellow lines indicate the evaluated performance of the fine-tuned single-task models.
lead to trade-offs in model performance across different tasks. In real-world applications, it is often more beneficial to obtain set of Pareto optimal solutions rather than a single model. Such solutions allow practitioners to choose among different trade-off's based on their specific needs. For example, hospitals specializing in certain areas might prefer a model that excels in tasks relevant to their specialty while maintaining adequate performance across a broader spectrum of diseases.
In this paper, we introduce a novel method that identifies the Pareto front without retraining the models to be merged. Our algorithm utilizes a quadratic approximation of the evaluation metric. Furthermore, we enhance with a Bayesian adaptive sampling method and a nested merging scheme, which further reduces the computational cost. We validate our method across a diverse set of tasks, spanning from vision to natural language processing, and demonstrate its applicability to a variety of architectures, including ResNets (He et al. (016). VIT (Dosovitskiy et al. 2020). and large language models (Brown et al. 22200] Rozière et al. 2023 Touvron et al. [023] Jiang ct al. 2024). Our results confirm that this novel approach supports the scamless integration of diverse model capabilities and
aligns more closely with various real-world preferences by providing a set of optimal fronts across the tasks.
Contributions
The main contributions of this paper are:
C1 We propose the MAP algorithm, which utilizes quadratic surrogate models to approximate evaluation metric functions, thereby amortizing the computation of Pareto fronts.
C2 We demonstrate the effectiveness of MAP across diverse set of architectures, including vision models (c.g.. ResNet, ViT) and large language models, highlighting its generalizability and flexibility across different domains.
2

---

Published as a conference paper at ICLR 2025
C3 We introduce two variants of MAP: the mested-merging MAP, which reduces computational complexity from O(N- 2^) to O(N log N), and the Bayesian MAP, which efficiently queries computationally expensive evaluations based on loss information.
C4 To the best of our knowledge, this paper is the first work to estimates the Pareto front for task-vector-based model-merging methods with low compute, and without relying on gradient descent for deep neural networks. Our code is available at lattps://githu. com/ -uli -gt-ggi/// /MPP
Performance of the combined model on tasks
Brute-fora force direct
MAP (cur methed)
MAP method) Single-taek model
Single-task model
100
80.79
09680
00,38
26 48
00:00
58.88
200
000
2014
Accuracy (%)
100-
5933
00.00
Accuracy
70.88
50
4000
50-
35.08
222
70
MNIST
SUN397
SVHN
GTSRB
EuroSAT
Cars
DTO
RESISC45
MNIST
SUN397
SVHN
GTSRB
EuroSAT
Cars
DTD
RESISC48
Figure 2: Left: Comparison of our method with direct search for merged ViT-B/32 models (Dosovit skiy et al. 2020), based on evaluation results of 250 combinations of scaling coefficients. Our method identifies more diverse set of solutions across eight tasks within the same computational budget. Both methods aim to maximize the performance of one task while ensuring that all other tasks meet minimum threshold of 40%. The bar plot displays the maximized accuracy for each task. Right: When the threshold is increased to 65% of the single-task model's ' performance, the brute-force direct search method fails find any feasible solutions within the same computational budget.
In Figure 2 we compare our method to direct search. It is important to note that evaluating models in problem setting is computationally expensive; as a result, our proposed method can only query the evaluation pipeline a limited number of times. We choose to compare with direct brute-force search within the scaling coefficient spaces. When the number of tasks is low, the Pareto fronts obtained through direct brute-force search can be considered ground truth because the scaling coefficient spaces have been sufficiently explored. However, when the number of tasks is high, we may not be able to derive reasonable ground truth Pareto fronts using direct brute-force search. Nonetheless, we can still demonstrate the efficiency of our method by comparing the winning ratio of MAP to that of direct brute-force search.
BACKGROUND
2.1
MODEL MERGING
Model merging aims to combine two or more trained models into a single model to leverage the
strengths of each and improve overall performance (Deep et al. 2024 Shoemake 1985 Wang
al. 2024b). recent work by Iharro et al. (2022) introduced task arithmetic as simple and effective method for model merging. The task vector for task is defined as Vn = 8% e ppe. which is the element-win wise difference between the pre-trained weights and the fine-tuned weights for task n. To perform model merging with task vectors, we can compute the merged model as O.. 0 pre + ΣN Cz Vn, where on the scaling coefficients that have been shown to be essential for the performance of the merged model (Yadav et al. 2024 Yang et al. 2023).
Denoting the metric of task n Mn. most existing approaches to model merging aim to improve an equal weight average metric + Σ Mn. This target implies that the user of the algorithm has no preferences between tasks. However, in real-world applications, users may have biased preferences for the importance of tasks, necessitating trade-offs in model performance. In such cases, the goal of model merging shifts from the simple average metric to addressing a broader question:
Given any preferences over the set of tasks, what is the best way merge the models?
3

---

Published as a conference paper at ICLR 2025
2.2 PARETO FRONTS
Definition 1 (Pareto dominance).. Let X be a set representing the solution space, where each element X is possible solution the malli-objective optimization problem. Let there be objectives, and define an evaluation loss function J X + R, where E (1.2....,T1)
Given two solutions I, y E X, we define that x Pareto dominates y, denoted by x PP y. if and only if:
Vi E (1. (z)) S f(ly) and 3j (1, f5( f5(1)
Definition 2 (Parcto optimal solutions). The Pareto front is the set of solutions in the solution space X that are not Pareto dominated by any other solutions in X. The Pareto front is given by:
PF = for X By X s.t. y > x}
Pareto optimal solutions have been studied in multi-task (multi-objecttve) learning (MTL) (Sener & Koltun 2018 Lin et al. 2019). However, in most of the studies, approximating the Pareto front is computationally ly expensive and data inefficient. We introduce our method, MAP, a computationally efficient method find the Pareto front for model merging.
3
METHODS
In this section, we discuss the proposed MAP method in detail. To enhance readability, we present a table of notations in Table
3.1 QUADRATIC APPROXIMATION OF EVALUATION METRIC
Given the task vectors (n=1,..... and the initialization Ppre Rd we denote the merged model parameters as mm(c) = Ppc + Vc = 6pc + Vn, where = concat(V1,.... vx) E R N is the task matrix and c = [c1,... CN] E R the vector of scaling coefficients for the task vectors.
Let (c) Mn (f. (c)) denote the evaluation metric for task of the merged model. We aim to optimize the evaluation metric for each task via the multi-objective optimization problem (MOOP
min Mi (c)....., Mx(c)
This problem has N variables and objectives, and we seek the Pareto optimal solutions.
Motivated by the observation that the finetuned models tend to have parameters close to the pretrained model (as shown in Table T we present the following assumption.
Assumption 1. The task vectors v., have small norms relative to the pretrained model parameters Ope. i.. v., < p0 for = 1,..., Additionally, the evaluation metrics Mn(()) are sufficiently smooth around pre. such that higher-order terms in their Taylor expansions are negligible within the region defined by the task vectors.
Denote p as the number of parameters in the pre-trained model and also as the number of parameters in each task vector. Let N represent the number of tasks.
To derive our algorithm, MAP, we utilize the second-order Taylor expansion to approximate My;
Ma(c) = M, (O. (c)) M, (0me) + (O T (0m((c)
0pre)
1
+ (mm(c) (c) ppe)) H. (pcc ) (c) 0pe)) R, (0m 0pcc)
2
= MMrr0pre) + V., (Ome))
T
(Vc) Hn
Ve + Rr
xP
ERPXN ERNXI
ER
ERPXP
ERPXI
ER
where (0 = V M, pme) Rdxd is the Hessian matrix and R, (0m(c) - 0 R,, (Vc) is the third-order remainder, which, as stated in Assumption | is negligible when ||Vc|| 3 = 11 (c)
'The evaluation metric M can be differentiahle (e.g., mean square loss cnossentrroy/peeplexitt) not. necessarily (e.g., classification accuracy, Fl score, BL.UU, Rouge)
4

---

Published as a conference paper at ICLR 2025
pre
is small. Note that the recond-order Taylor expansion is widely used (McCandlish et al.
2018
Bu et al. 2024 Kaplan et al.] [020]] Hoffmann et al. 2022) and provides a close approxımation (see
Figure (a) and (b) and more discussion in Appendix
Table 1: We compute the I norm of the weight matrices of the pretrained models and the task vectors for each of the 8 tasks using the ViT-B/32 model, and compute the ratio.
Metric
SUN397
Cars
DTD
SVHN
RESISC45
MNIST
GTSRB
EuroSAT
10peel1 [all
1,270,487
1,270,487
1,270,487
1,270.487
1,270,487
1,270,487
1,270.487
1,270,487
21,055
20,127
13,621
19,349
18,409
17,578
16,712
15,941
Walli/110pm.. (56)
(15)
1.66%
1.58%
1.07%
1.52%
1.45%
1.38%
1.32%
1.25%
To validate the approximation, we calculate the L1 norm of the parameters for the 8 task vectors and their ratios relative to the L1 norm of the pretrained model. As shown in Table the ratio between the L1 norm of the task vectors and the Lı norm of the weight matrices of the pretrained model is approximately 1% to 2% for each of the tasks.
We further examine the density plot of the absolute values of the weight matrices of the 8 task vectors (Figure 3). We find that the maximum magnitude of the task vectors is 0.00859, while the mean is 1.57336 10 - These findings motivate and validate key assumption of using a second-order Taylor expansion approximate the evaluation metrics.
Binned Density Plot Tensor Values
5000
Derdity
3000
2000
1000
Leveraging this quadratic approximation, we can define surrogate models for each task N,
0.000
0.002
0.004
0.006
0.006
Absolute Value
Figure 3: Density plot of the absolute values of the weight matrices the 8 task vectors.
Mr An, bas cn) = 2 A.c
(3)
and optimize the proxy problem of Equation 2 via
min
M1 Mr(C)
(4)
where A, N is parameter matrix approximating V H, (O b, R is parameterized vector targeting V (Opee)))); en a parameter scalar which estimates Mn, (0re)) + Rn.
Importantly, Equation (4) is parameterized contrast to Equation (2). with a total of (N+1)(++2) surrogate model parameters in (En, b,, Ar), i.e., coefficient for en, N coefficients in b, and N(N 1)/2 coefficients in A, due to symmetry (1 + N + N(N+1) (N+1X(N+2), We do not calculate the gradient or Hessian to obtain the parameters An. b en. Instead, we estimate them by minimizing the MSE between M,, and Mn = 1,..,, N:
2
2
A bi en = arg min An.
[M Mn(c; bn, en)?
(5)
CEN
where Ω = (c c. (K) Below discuss 3 extensive cases of this problem.
} is the set of and (0 (c)) is the corresponding evaluation metric.
Case 1:
If the evaluation metric spans the whole real-number axis R, we use the vanilla form of the surrogate model in Equation (3). As shown in Corollary 1 have closed-form solution for Equation (5)
Corollary 1 (Closed-form Solution for Surrogate Model Parameters). Under Assumption I for each task n = 1, N, the optimization problem 5 is equivalent to solving a linear regression where the predictors include all quadratic, interaction, linear, and constant terms of c. The closed-form solution for the parameters is given by
\ve((A)) ) by
C,) C..yn
en
5

---

Published as a conference paper at ICLR 2025
c102, ,,, C12 02. ...... CN. 1), and Yn observed metrics Mn (c)) for all c E Q. Please refer to Corollary for a more detailed corollary.
where (c) = (c,. C102
n. is the vector of
Case 2: When the evaluation metric is restricted to specific range 1l. u], e.g. accuracy is restricted to [0, 11. we can apply sigmoid transformation to the quadratic term, i.e.
Mn(c (c; A, Da, Cn) (u - [))(en + b c + 1 zcc Anc) +1.
Case 3:
Similarly, when the evaluation metric is restricted to [1, +oo), applying a softplus transfor- mation ensures non-negativity while allowing smooth growth. In this case, we define:
1 (n(c; An bn en) = softplus(e, + bb, c zc Anc) +1,
T
where softplus(x)) = In(1 +
In Cases and 3, the parameters u and / are not learnable parameters but predefined based on the feasible range of the metric. The introduction of non-linear functions in M, eliminates closed-form solutions, necessitating the use of gradient descent to solve the optimization problem [ Despite this, the approach remains computationally efficient since the number parameters in the surrogate model is (N+1)(N+2) We emphasize that gradient descent on the surrogate model is computationally efficient. Compared to training full-scale deep learning models, this method requires significantly fewer computations. For instance, when merging eight models, the parameter count is only 45, which much smaller than the millions of parameters typically found in deep learning architectures.
2
3.2 MODEL MERGING WITH AMORTIZED PARETO FRONTS
In this section, introduce our generalized algorithm for estimating the amortized Pareto fronts. As mentioned in Section 3.1 we approximate the evaluation metric M.) by a surrogate quadratic model MM, (-). We then utilize (.) to compute the amortized Pareto fronts. Please see the detailed experiments in Section 4 and the algorithm details in Algorithm [ Different from other Pareto multi-task learning algorithms, our algorithm does not require calculating the gradients or performing gradient descent on the deep learning model parameters.
Algorithm MAP
Input: Pretrained model 6pres fine-tuned models {0), b. Compute task vectors {vn 07 Ppe 1,...,N7 N).
Sample vectors of e E N Denote the set as 12 for E [ do
for c c= [l,,..,,,, E f2 do
Compute O(c) = Ppe civ1 + +...+ NNVN. Obtain the evaluation metric M, (0(c)..
end
Fit the quadratic approximation surrogate model M, by learning A by ex in Equation 5
end
Apply MOOP algorithm (e.g. NSGA-III) {M,) and find the Pareto front
3.3 FURTHER METHODS TO BRING DOWN THE COMPUTATION COST WHEN THE NUMBER OF TASKS IS HIGH
In addition to MAP, we also introduce nested merging MAP (NMMAP) and Bayesian MAP (BMAP), which are variants of MAP designed to further reduce computation costs.
For a small number of tasks (N < 3), we use Bayesian adaptive sampling, inspired by Bayesian optimization. Unlike plain MAP (Algorithm []. which samples single set of scaling weights and evaluates metrics M,, (C))). Bayesian adaptive sampling iteratively samples e across multiple rounds, with each round informed by prior evaluations.

---

Published as a conference paper at ICLR 2025
The process starts with uniform sampling of scaling coefficients f(ci..... )) from [0, 1] evalu- ating the merged models (c,) for tasks 1 to N. and calculating the Lg loss. We compute a posterior distribution based on acquisition function, like the upper confidence bound, for each bin.
We iteratively update surrogate models for each task, and after meeting the stopping criterion, generate (c, {M, (c)) m= samples. A MOOP algorithm (e.g.. NSGA-III) is then applied to compute the Pareto front from Mn.
As illustrated in Figure 4 (a), BMAP uses Bayesian optimization determine the distribution of scaling coefficients for task vectors when approximating the surrogate model in MAP. For more details and experiments, please refer Appendix E.3
When the number of tasks N is high and computational resources are also limited, NMMAP reduces the number of evaluations from O(T GTSRB, DTD, SUN397, RESISC45, MNIST, EuroSAT, and SVHN. With nested merging, in the first round, we merge (0) 0² into menge. Similarly, we merge (0),, Of Jino Oeerge. (05 fer 0g into e and (0) 00,)) into 0 in parallel. Next, we merge (0 0 into 01.228.4 and (Omerge; O into 0 simultaneously and finally into 012.3.115,6,,77 Note that the model merging order is illustrative and chosen to support the exposition.
IoN N). For example, tasks to 8 cover Cars,
This approach achieves 250x speedup when the number of tasks is 8, while also outperforming direct search methods. However, it can no longer output the complete Pareto front for all tasks, and practitioners need to be able quantify their preferences during the merging since merges models in pairs in a nested fashion, as shown in Figure (b). For more details and experiments, please refer Appendix E.2
Nested Model
MNST
Figure 4: (a): Utilize Bayesian optimization to guide the sampling of scaling coefficients according to the uncertainty distribution; (b): An example of nested model merging for N 8 models.
4 EXPERIMENTS
Datasets and models We evaluate multi-task model merging on eight zero-shot image classification datasets following (Tharco al. 2022): ; SUN397 (Xiao et al. (0166), Cars (Krause et al.) 2013), GT- SRB Stallkamp et 2011), MNIST (eCunn 1998). EuroSAT (Helber et al. 22019), SVHN (Netzer al. 2011), DTD (Cimpoi al. 2004 4), and RESISC45 (Cheng al. (017). We use the ViT-B/32 architecture in CLIP (Radford et al. (021) as the pre-trained model for the experiments on vision tasks discussed in the main text. We show the results of these experiments in the main pages.
The rest of the experiments are presented in the Appendix due to page limits. The results from the datasets and tasks we experimented on are as follows: a zero-shot medical chest X-ray image classification task to show our model merging scheme works in a real-world application in the medical domain (Wang al., 2017 7) (Appendix [D.3); four fine-tuned I.lama3 models in different languages: French, Arabic, Chinese, and Japanese (Appendix D.4 4 and three vision classification tasks using the ResNet18 (He et al. 2016) architecture: CIFAR- 10 (Krizhevsky et al. 2209), Flowers- 102 (Nilsback) & Zisserman 2008). and GTSRB (Stallkamp et al. 2011) show that our model merging scheme also works with other model architectures (.p
AAll the fine-tuned language Llama3 models can be found on Hugging Face. The IDs of the models are: French:
haapchh-Alpaaa-lllaa--aaa Instruct; Chinese; shenzhi- shenzhi-wang/lllmma Chinese-Cnhttt Japanese: haqishen//.lama-3-880 Jew.aaaa--------aeeeee-------se-
.0; Arabic:
7

---

Published as a conference paper at ICLR 2025
4.1 BASELINE AND METRICS
Pareto front-finding baselines One of our baseline methods for obtaining Pareto fronts is the brute-force direct search, which can be regarded as the gold standard when the number of tasks is low (N < 4). This is because we can sample enough grid points of and query the corresponding We can then directly use the resulting evaluation metries find
evaluation metrics (M(,)(())) the Pareto front by direct comparisons of each task's 's performance of the merged model by E c.
Another baseline method for finding Pareto fronts is the Multi-Objective Evolutionary Algorithm based on Decomposition (MOEA/D) (Zhang & Li (007). Key hyperparameters in MOEA/D include the population size and the number of generations, which significantly influence its performance by affecting the quality and diversity of the resulting Pareto front as well as the computational cost.
important to note that when the number of tasks grows, the required number of (c, (M, (O(c)) pairs grows exponentially for the brute force method, as well as for the MOEA/D method, which is much larger than the points we can evaluate given the computational constraint. Thus, when the number of tasks is high (N 2 4), the results from the brute-force direct search can no longer be considered ground truth. In Appendix [..22 we show an illustration of why the curse of dimensionality occurs when the number of tasks increases.
Single merged model baselines addition to the prute-force method and MOEA/D, we compare with other model merging methods including SLERP [Shoemake| 1985), TIES MEr-mig (Yadav et al. 2024). Task Arithmetic with single scalar [Tharco al. 2022), Task Arithmetic with preferences as scalars (Ihharco et al., (022). DARE combined with Task Arithmetic (Yu et al. 20233 Ihharco et al.) 2022
and DARE combined with -eES-merging (Yu et al. 2023 Yadav et al. 2024).
Win rate We used the win rate to measure how often the Pareto front found by MAP outperforms the Pareto front found by the baseline in terms of multiple objectives. Let MAAP and Phbselline represent the set of solutions in the Pareto fronts obtained from the MAP and the baseline methods, respectively. Each solution in these sets is a vector in R. where N is the number of objectives or tasks. We sampled K = 100 points from the decision space of each of the two Pareto fronts, denoted as cx AP and cg choseline h = 1,..., K. Then, we compared M,, (f(c)) MP)) and M,, (f( (f(ccfuueliin (easeline)) pairwise for 1. ..... K and I, I...., NN, resulting in K N comparisons. The ratio of instances where Mn. (0(c > Mn (0 (((caaeiiin) is computed as the win rate of PFMAP:
K K N ΣΣΣ
Win Rate-
(MM,(((MMP)) > (O haseline)) ))] =1j=1am
K2N
where K = 100, MAPP E Decision Space of PF i = 1,..., K, c
chsclline
E Decision Space of
PPhnseline- j = 1..... K, I[-] is the indicator function.
4.2 WIN RATE OF MAP OVER THE DIRECT SEARCH METHOD
Table 2 shows the results for the win rate of MAP over the brute force direct search method when the number of tasks is small (N < 4). such cases, the Pareto front can be regarded as the ground truth Pareto front, and we can only query 30 and 50 scaling coefficients to achieve similar performance with the ground truth Pareto front
Table JJ shows the results for the win rate of MAP over the brute force direct search method when the number of tasks is high (N 2 4). The Pareto front generated by the brute force method can no longer be considered ground truth due to the limited number of points per dimension that is covered. In this setting, MAP performs much better than the brute-force (not ground truth) Pareto solutions.
Table 2: Win rate of the amortized PF over the brute-force direct search PF, where the latter can be regarded as the ground truth when N < 4. # c is the number of scaling coefficient vectors each algorithm evaluated to find the Pareto front. # с per dim = (# с direct search) I/N measures the sparsity of points the direct search method uses, illustrated in Figure
N (direct scarch)
# per dim (MAP)
Win (MAP)
RR (MAP)
200
14.14
30
49.81% (+0.30)
0.953 (+0.018)
300
46.90% (+0.71)
0.980 (+0.003)
8

---

Published as a conference paper at ICLR 2025
Table 3: Win rate of the amortized PF over the brute-force direct search PF, where the latter can no longer be regarded as the ground truth when N 2 4. # of с is the number of scaling coefficient vectors each algorithm evaluated to find the Pareto front. # c per dim = (# с direct search) 1/N 1/NV measures the sparsity of points the direct search method uses, as illustrated in Figure
c (direct search)
per dim
c (MAP)
(MAP)
(MAP)
300
4.16
60
50.67%
(42.44)
0.984 (+40.004)
5
500
3,47
85
53.00%
(+1.88)
0.941 (+0.019)
500
100
60.71%
(+131)
0.941 (+0.030
7
1000
2.68
140
63.42% (+11
0.891 (+0.024)
1000
2.37
250
65.58% (=0.1)
0.868 (00.028)
4.3 COMPARING WITH OTHER SINGLE MERGED MODEL BASELINE METHODS
We compared the performance of MAP with TIES-mergih Yadav et al. 2023). TIES- TEES-megging with DARE (Yu et al. 2023). Task Arithmetic with DARE, Task Arithmetic with normalized preference as (Deep)
scaling coefficients (Tharcoo et al. 2222)) Ada-merging++ (Yang et al. 2024). DELLA-merghle iet al. 2024) and SLERP (Shoemake) 1985). For dimension 2, we can directly visualize the results, as shown in Figure 5 The solutions found by MOEA/D exhibited clustering and lack of adequate spread across the Pareto front. In addition, the computational cost of MOEA/D was significantly higher than that of MAP. For example, with population size of 50 and 20 generations (which produced the best results for MOEA/D in terms of diversity as shown in Figure S). the total number of evaluations amounted to 2500 for two tasks. In comparison, MAP achieved its results with only 60 total evaluations reduction of over 95%. This highlights the computational efficiency of MAP in obtaining a diverse and high-quality Pareto front.
MAP offers well-distributed set of Pareto optimal solutions None of the baseline methods can directly dominate all Pareto solutions found by MAP, and most of them lie either within or on the Pareto front found by MAP. In addition, we sampled 10 points from the predicted Pareto front generated by MAP and evaluated them (MAP Pareto solutions, real) to confirm that they indeed lic close to the predicted Pareto front. This evidence further confirms the usefulness of MAP as a general strategy for finding diverse and deel-diitti set of Pareto solutions that none of the existing model merging methods can substitute.
Pareto fronts from grid search and MAP
Pareto fronts from grid search and MAP
0.80
Ord 20 rt deta
0.7
Grid point dite
searc1 Parats frost
Grid search Fanato
accuracy
0.70
(DARE) Fareto preckited
DTD accuracy
0.6
MAP (DWRE) Peress front, predicted
MOFAD Paeno fa
MISSAO Faeen froe
TTA) froeo
MAP (TA) Parto sokriers rol
0.65
Porets solutono real
DARET
DARE-TIS
0.60
Thkk Adtimmati (TA)
0.4
TEE Artiorett (TA)
TeGGeregin
TESmengo
0.55
Addeenging
LA with proferesv
ASSMerinng
with srefercee SUgRP
SLEOP
0.4
0.5
0.7
0.45
0.50
0.55
0.50
0.65
0.70
0.75
DTD accuracy
SUN397
accuracy
Figure 5: The Pareto fronts obtained using MAP with Task Arithmetic, MAP with Task Arithmetic and DARE. We sampled 10 Pareto solutions from the predicted front by MAP and evaluated them to obtain the real values. We plotted the results obtained using TIES -TESSnnrggigg Task Arithmetic (TA) with a single scalar for all tasks, Task Arithmetic with preferences as scalars, TA combined with DARE (DARE-TA), TIES-mergine combined with DARE (DARE-TIES), and SLERP.
MAP is an out-of-the-box plugin for other task-vector based model merging methods As shown in Figure 5 MAP plug-in that can be directly combined with other model merging methods, allowing users to control their preferences over each task using scaling coefficients. For example, MAP can combined with DARF, Task Arithmetic, etc.
For dimensions higher than 2, we use preference-werr -weghttd evaluation sum to compare the Pareto front found by MAP with other baseline methods. We sample 20 preference vectors and normalize

---

Published as a conference paper at ICLR 2025
them. Then, pick the solution by MAP that maximizes the preference seefrrence-weiigted weighted of accuracies and compare it with those of the baseline methods. The results are shown in Table 4 We observe that even in higher dimensions, when users have specific preferences, MAP remains effective and can better accommodate those preferences. Note that when using MAP, users are not required to pre-specify their preference vector. In fact, users can obtain the entire Pareto front to better understand the trade-offs. This preference vector-based metric is solely for evaluation.
Table 4: We compared MAP with set of baseline methods by sampling set of 20 normalized preference vectors and computing the preference-weighted sum of accuracies. 1 indicates higher is better. The number after + is the standard deviation. Please refer to Table 2 and Table 3 for the number of scaling coefficient vectors used for different numbers of tasks.
Preference weighted sum of accuracies (7)
N tasks
2
5
7
Single task models
75 75.84-+1.76
77.05+1.86
N2.45+4.40 44
88.52-4.02
3.55
90.62++2.55
MTL.
73.63-+0.30
75.1341.00
86.78-2.94
87.40+2.56
89.1142.3
Model soups ( (Wertsman ol 2022
6..79::1116
64.25 15
66.01:13.22
67.0113342
63.11=1.99 1.99
63.35:2.117
64.3612 77
TIES merging
69.30.10.33
67.6010.55
71.79:2293
36.4913.10
73.74-2.96
72.54:2.87
12.24.11911
DARE-TIES
67.62-+1.65
66.49-2.34
71.39+4.45
34.55++4.55
73.34-4.11
Task Arithmetic [harcc et al. 2023
70.73:11884
61.15+2.33
52.69:44.33
61. 58:5622-.62
51.37-3.84
39.79+43.97
60.77442..
TA with preference ni werghts
69.22:11.4
66.88+2.37
66.73455448
71:92=555
68.13-4.69
68.14+4.2
68. 68.17-22.9
DARE-TA
70.61:10.22
61.1811.24
58.0118.19
65.39117.03
56.76_7.01
46.75:15.73
64.51113.1
Ada Merginge+ Yang et 2024
67.27:11.22
67.13::1.92
71.1914.43
76.841:.71
74.13=4.07
72.58:14.116
72.554283
DEI J.A. Menging Despre
2024
67.10+2.08
65.92+22.48
70.71+4.31
74.43+4.32
72.64-3.77
71,16+33.95
71 49++
MOEA/D
70.22 1.46
67.94 + 1.79
70.85 4.58
72.03 4.03
67.88
3,09
69.06
2.97
68.59 2.89
MAP
10.70:1.776
69.05:1.84
72.84:4.40
77.3114450
74.26=4.02
73.40:3.58
72.96:12.52
5 RELATED WORK
Due to space constraints, for more comprehensive discussion, please refer to Appendix B
Multi-objective optimization Multi-objective optimization identifies diverse Pareto solutions with different trade-offs. The multi-task problem has been approached from a MOOP perspective (Sener & Koltun 2018 Lin et al.] (019). utilizing algorithms like MGDA, IMTL, GradNorm, RLW. and scalarization. While these methods iteratively optimize Rdd model parameters with significant com- putational cost, our approach performs post-training MOOP over R- scaling coefficients, reducing computational burden. For N 2 3 objectives, MOOP becomes many-objective optimization, chal- lenging traditional algorithms (c.g. NSGA-II and SPEA2). Advanced methods such as e-MOEA (Debl ct al. 2003). MSOPS (Hughes 2005),
(Beume et al. 2007), and NSGA-III (Deb & Jain 2013), the KKT approach (Augusto al. 2014) required.
Task arithmetic Task arithmetic, a model merging method gaining attention, uses weighted averages of models for multi-task performance. Various approaches exist for selecting scaling coefficients: [lharco et al. (2022) used equal scaling, which is suboptimal and limited;; Yang et al. (2023) optimized weights using Shannon entropy but required unlabeled test data, which conflict with data privacy goals in model merging. Beyond task arithmetic, Ainsworth et a (2023) merge models via weight matrix permutations, even without shared pretraining. Jin et al. (2023) introduced dataless knowledge fusion. DDheimm et al. (2024) proposed uncertainty-base gradient matching for improved merging performance and robustness.
6 CONCLUSION AND LIMITATIONS
We introduced MAP, novel low-compute approach to efficiently merge models while accounting for trade-offs between multiple tasks. By leveraging quadratic approximation of the evaluation metrics, MAP successfully identifies amortized Pareto fronts without the need for gradient descent for deep neural networks. The algorithm's 's efficiency is further enhanced through Bayesian adaptive sampling and nested merging, enabling it scale to problems with a higher number of tasks. Moreover, MAP is an out-of-the--ol plug-in for other bosk-vector-baat model merging methods.
However, we would like to point out some limitations of our method. We do not have detector algorithm to detect if two tasks have proper trade-offs. Approximating a Pareto "front" with only a single Pareto solution can lead to degenerate solutions. In this case, practitioners should be informed.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging mod- els modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https::/openreview net ppdfiidcCsamMmm1155333
Zeyuan Allen-Zhus Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International conference machine learning, pp. 242-252. PMLR, 2019.
Oscar Brito Augusto, Fouad Bennis, Stephane Caro, et al. Multiobjective optimization involving quadratic functions. Journal of optimization, 2014, 2014.
Nicola Beume, Boris Naujoks, and Michael Emmerich. Sms-emoa:
Multiobjective selection based on dominated hypervolume. European Journal of Operational Research, 181(3):1653-165) 2007.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Zhiqi Bu, Xinwei Zhang. Mingyi Hong, Sheng Zha, and George Karypis. Pre-training differentially private models with limited public data. arXiv preprint arXiv:2402.18752, 2024.
Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends@ in Machine Learning, 8(3-4):231-357, 2015.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 186510);1865-1183, 2017.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2014.
Carlos A Coello Coello and Nareli Cruz Cortés. Solving multiohjective optimization problems using an artificial immune system. Genetic programming and evolvable machines, 6:163-190, 2005.
Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. segpt: toward building a foundation model for single-cell multi-omics using generative ai. Nature Methods, pp. 1-11, 2024.
Nico Daheim, Thomas Möllenhoff, Edoardo Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. Model merging by uncertainty-based gradient matching. In The Twelfth International Conference on Learning Representations, 2024. (tth:tt::: http:///ppeeeeee.eeeeee wew.eewww.ee.oo...ee..eeeeee ner/forum? /
[d=D7KJmfEDQP
Kalyanmoy Deb and Himanshu Jain. An evolutionary many-objective optimization algorithm using reference- ceeeencc-pointt--e point-based nondominated sorting approach, part i: solving problems with box constraints. IEEE transactions evolutionary computation, 18(4):577-601, 2013.
Kalyanmoy Deb, Manikanth Mohan, and Shikhar Mishra. Towards quick computation of well- spread pareto-optimal solutions. In Evolutionary Multi-Criteriaa Optimization: Second Interna- tional Conference. EMO 2003, Faro, Portugal, April 8-11, 2003. Proceedings pp. 222-236. Springer, 2003.
Pala Tej Deep, Rishabh Bhardwaj, and Soujanya Poria. Della-merging: Reducing interference in model merging through magnitude-batt basedd sampling. arXiv preprint arXiv: 2406. 1617, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
11

---

Published as a conference paper at ICLR 2025
Guodong Du, Jing Li, Hanting Liu, Runhua Jiang, Shuyang Yu, Yifei Guo, S. Goh, and Ho-Kin Tang. Knowledge fusion by evolving weights of language models. Annual Meeting of the Association for Computational Linguistics, 2024. doi: 01.48550/urxiv224066 12208.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pp. 1675-1685 PMLR, 2019.
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23(4):2341 1003341-2666, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
Evan Hughes. Evolutionary many maje-objective optimisation: many once or one many? In 2005 IEEE congress on evolutionary computation, volume 1, pp. 222-227. IEEE, 2005.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04088 2022.
Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference Learning Representattons, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. jr
URL /ttt:p:://ppereettt
net net.
Moksh Jain, Emmanuel Bengio, Alex Harnández-GGrrfa, Jarrid Rector- Bonaventure F. P. Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequence design with gflownets. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9786-9801. PMLR, 2022. URL httpt:::/ppppooeetttttto
i ngs ml wtt..ng.........e./eeepp.p.... 62/jain22a html
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv: 2310. 1210.11664, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril. Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. arXiv preprint arXiv: 2401.04084 2024.
Xisen Jin, Xiang Ren, Daniel PreotuucPPeetro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representattinss ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL net/ CorrmmiddFCCohhat
htttps://cpeneet
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2001 (00361 2020.
12

---

Published as a conference paper at ICLR 2025
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine ine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013. 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmistl/ 1998.
Bingdong Li, Zixiang Di, Yanting Yang, Hong Qian, Peng Yang, Hao Hao, Ke Tang, and Aimin Zhou. It's morphing time: Unleashing the potential of multiple ilms via oujti-obeective optimization. arXiv preprint Xrviv: 2407.00487, 2024.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing- Fu Zhang, and Sam Kwong, Pareto multi-task learning. Advances in neural information processing systems, 32, 2019.
Xi Lin, Zhiyuan Yang. Qingfu Zhang, and Sam Kwong. Controllable pareto multi-task learning. arXiv preprint arXiv: 2010.06313, 2020.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.0611 2018.
Henry B. Moss, David S. Leslie, Daniel Beck, Javier Gonzalez, and Paul Rayson. Boss:
Bayesian optimization over string spaces. In Advances in Neural Information Process- ing Systems, 2020. URL /ttpttttt////ppeeeediig neurips cC/ cc//paper/2222/haah/ bl 31a2222858800377233384555530--23
html
Aviv Navon, Aviv Shamsian, Gal Chechik, and Ethan Fetaya. Learning the pareto front with hypemetworks. arXiv preprint arXiv: 2010.04104, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. al. Reading digits natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 7. Granada, Spain, 2011.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008
large number
National Institutes of Health al. Nih clinical center provides of the largest publicly available chest x-ray datasets to scientific community, 2017.
E. O. Pyzer-Knapp. Bayesian optimization for accelerated drug discovery. IBM Journal of Research and Development, 62(6):2:1-277, 2018. doi: 10.1147/RRD.201..220 211.:/..001121188111 8.2881731.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Alexandre Ram'c e, Nino Vieillard, L' L'onaadd Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. International Conference on Machine Learning, 2024. doi: 10.485500arxiv.2401.. 121887
Alexandre Ramé, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean Bran-Baptiste Gaya, L. Soulier, and M. Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Neural Information Processing Systems, 2023. doi: 10.465500axxxvv2236.44442 244888888..
Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. Warp: On the benefits of weight averaged rewarded policies. arXiv preprint arXiv: 2406. 16768, 2024.
13

---

Published as a conference paper at ICLR 2025
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défosse7, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code. arXiv preprini arXiv: 12950, 2023.
Michael Ruchte and Josif Grabocka. Scalable pareto front approximation for deep multi-objective learning. 2021 IEEE International Conference on Data Mining (ICDM), pp. 1306-1311 2021. doi: 10.1 (0.1009//DM51129 20210000....
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL httpt::tttppttcc httt::::/prooceedingge neurips hwwrips..ceppppss...s.eeeeeeeepe httpppseeeeapppe.e.eeeeeeeeeeeee wtwes//appeee/111/. Ei e/ FTlee/22accaalll33333333333333 65ecee Paper..eeeppp.r......e .pdf
Ken Shoemake, Animating rotation with quaternion curves. In Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '85, pp. 245-254--4 New York. NY. USA, 1985, Association for Computing Machinery. ISBN 0897911660 doi: 10.1 00.11445/2533343222222 URL hRR.http:://dol orgiiiorggiio 10. /1/44///3335.. 3252442
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In The 201 international joint conference on neural networks, pp. 1453-1460. IEEE, 2011.
Hui Su, Zhi Tian, Xiaoyu Shen, and Xunliang Cai. Unraveling the mystery of scaling laws: Part i. arXiv preprint arXiv:2403.06566 2024.
Kei Terayama, Masato Sumita, Ryo Tamura, and Koji Tsuda. Black-box optimization for automated discovery. Accounts of Chemical Research, (6(6)::334-1344 2021.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez. Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8): 1930-1940, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arxiv: 2307.09288. 2023.
David Allen Van Veldhuizen. Multiobjective evolutionary algorithms: classifications, analyses, and new innovations. Air Force Institute Technology, 1999.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi-objective rewards. arXiv preprint arXiv: 2402. 2024a.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks fication and localization of common thorax diseases. In Proceedings of the IEEÉ conference on computer vision and patlern recognition, 2017.
classi-
14

---

Published as a conference paper at ICLR 2025
Yibo Wang. Sijia Chen, Wei Jiang, Wenhao Yang, Yuanyu Wan, and Lijun Zhang. Online com- posite optimization between stochastic and adversarial environments. In In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024b.
James T. Wilson, Riccardo Moriconi, Frank Hutter, and Mare Peter Deisenroth. The reparameteriza- tion trick for acquisition functions, 2017.
Michael Wornow, Rahul Thapa, Ethan Steinberg, Jason Fries, and Nigam Shah. Ehrshot: An ehr benchmark for few-shot evaluation of foundation models. 2023.
Mitchell Wortsman, Gabriel Ilharco, S. Gadre, R. Roclofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Y. Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. International Conference Machine Learning, 2022. doi: 00.48850/rXxv.2203... 05482.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audin pretraining with feature fusion and keyword -owoorrtoo-caation aug- mentation. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, pp. 1-5, IEEE, 2023. doi: 10. 1977SS599339 2023. 1023.90, URL WtLthtts::/ / /ooi org 10. Tww.11191ISSSSSS ICASSP 9357 2023 111...009555
Jaanxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 119: 3-22, 2016.
Prateck Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Neural Information Processing Systems, 2023.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint 10.00223 2023.
Enneng Yang, Zhenyi Wang. Li Shen, Shiwei Liu, Guibing Guo. Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, 2024. TRI.https:// hhttps://openrewe ew net/ forum?
id-n 6N3D0Y
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. arXiv preprint arXiv:231 1.03099, 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, IEEE International Conference on Computer Vision, 2023. doi: 10.1 109/ ICCV51070.2023.01100
Qingfu Zhang and Hui Li. Moea/d: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions evolutionary compulation, 11(6):712-731 2007.
Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto alignment via preference adaptation for llms. arXiv preprint arXiv: 2402 02030, 2024.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one mee-refeerenc-fie fattrsale-ftt-lll alignment: Multi-objective direct preference optimization. arXiv preprint arxiv: 2310.03708, 2023.
15

---

Contents
Notations
17
More Discussion on Related Work
17
Additional Details the Methods
18
C.1
Proof: Negligibility of the Remainder in Multivariate Taylor Series
18
Closed-torm Solution for Surrogate Model Parameters
19
D
Additional Details on Experiments
20
D.I
Experiment setup
20
D.2
Additional metric: Generational distance and inverted generational distance
20
D.3
Zero-shot Medical Image Classification
21
D.4
Merging Language Model Additional experiment results on ResNet
21
D.5
23
More Details on Algorithms
24
Plain MAP
24
F.2
Nested-Merging MAP
24
F.2.1
Overview
24
E.2.2
Time Complexity of Nested-Merging Merging MAP
24
E.2.3
Explanation curse of dimensionality
25
E.2.4
Intermediate Pareto Fronts Each Round
25
E.2.5
Results on using nested merging to merge models
27
E.3
Bayesian MAP]
27
E.3.1
Performance of Bayesian MAP
29
F
Potential Q & A
31
F1
The authors assume a setting where computationally expensive to query the model evaluations. But is this the reality?
31
F2
How is MAP different from other MOOP methods, such as linear scalarization?
31
F.3
Why does quadratic approximation have a lower cost?
31
F.4
Why not compare to Git- Rebasin (Ainsworth et al. 2023
31
F.5
Why does nested merging MAP not give complete Pareto front?
32
F6
It seems nested nested-merging MAP performs worse than MAP. What is the meaning of it?
32
F7
Why does nested- merggng MAP not output the optimal solution?
F88
How do you deal with gradient and Hessian in the second-orde Taylor expansion?
32
16

---

Published as a conference paper at ICLR 2025
A
NOTATIONS
For clarity, we summarize the notations used throughout the paper in Table
Table 5: Table of notations.
Notations
Explanation
N
Number of tasks
Vector of scaling coefficients for task vectors
Cn
Scaling coefficients for the rith task
Ope
Pretrained model parameters
mm(c)
Merged model parameters as function of scaling vector
Vn
Task vector for lask
Task matrix formed by concatenating task vectors Evaluation metric for task n as function of c Quadratic coefficient matrix in the surrogate model
(c(
An
bn
Linear coefficient vector the surrogate model
en
Constant term in the surrogate model
(c; An: Dus
Surrogate quadratic model for task
Ω
Set of scaling coefficient vectors used for estimating surrogate model parameters Number of sampled coefficient vectors in set Q
K
B MORE DISCUSSION ON RELATED WORK
Second-order Taylor expansion In deep learning, the second-order Taylor expansion and thus the quadratic approximation on the evaluation metric is an important technique, that characterizes the metric landscape. For example, the Newton-Ralphson method is derived from it: w, W
H = argmin, .M (w, - v) given that (w: v) = M (w:) - vG + fvv Hv. The quadratic approximation is also combined with Lipschitz smoothness (L) in the classic convergence analysis of
SGD Bubeck et al. 2015 Ghadimi & Lan 2013). through M(w, - -/) < M(w,) - n
Ln² ||G||2 Interestingly, although the approximation is only accurate in the local neighborhood, it can be used to indicate the global convergence over the iterations. One reason is that state-of- the-art neural networks
2
usually over-parameterized and undergo lazy training, meaning that the
converging parameters are close to the initialization (Chizat et al.] 2019 Du et al. 2019 Allen-Zhu al. 2019). Thus, the local approximation informs the global convergence behavior. In particular, this approximation has played important roles in the scaling laws of neural networks (e.g. Equation
3.3 in (Su et al. 2024)) that predict the performance and help select hyperparameters before training
actually takes place.
Pareto fronts for Multi-task Learning Recent advancements multi-task learning (MTL) have seen diverse approaches to Pareto Front learning in machine learning. |Sener & Koltun (2018) explicitly formulated multi-task learning as a multi-objective optimization problem and adapted gradient-based multi-objective optimization algorithms large-scale learning problems. Lin et al. (2019) developed a constrained optimization approach to find multiple Pareto optimal solutions representing different trade-offs among tasks in multi-task learning and solved decomposed sub- problems in parallel using gradient- optimization. It requires separate training runs for each solution. Navon al. (020)) and Lin et al. (2020) employed hypernetworks for continuous Pareto Front approximation, generating target network weights based on preferred trade-offs. However, the size of these hypernetworks can become prohibitively large and needs to be properly trained as well. Ruchte and Grabocka Ruchte & Grabocka (202T) introduced memory-efficient method by augmenting network inputs with desired trade-offs, although this may obscure the network's conditioning due to nonlinear dynamics, and it is also based on the gradient updating method.
Model-merging Applications in LLM In the realm of model merging applications for language model preferences, recent research has made significant progress. Ramé et al. (2023) and Jang et al.) (2023) introduced "rewarded soups" and "personalized soups", which utilize model soup to interpolate weights of networks fine-tuned diverse rewards to achieve Pareto-optimal generalization across
17

---

Published as a conference paper at ICLR 2025
preference spaces and address the challenge of aligning large language models with individual perspectives. WARP (Weight Averaged Rewarded Policies) (Ramé et al. 2024) and WARM (Weight Averaged Reward Models) (Ram e et al. 2024) demonstrate how merging policies or reward models in weight space can refine the trade-off between competing constraints, such as reward optimization and alignment with pre-trained knowledge human preferences. (Zhong et al., (2024) developed "Panacea", which reframes alignment as multi-dimensional preference optimization problem, using singular value decomposition- bassiidooo-oedddddd low-rank adaptation guide model behavior with low- wiwdimensional preference vector. To address the limitations of scalar rewards in RLHF, Zhou et al. (203)) introduced Multi- culti-OObecctve Direct Preference Optimization, an RL- free algorithm that extends Direct Preference Optimization to handle multiple alignment objectives efficiently. Du al. (2024) relies on "mutation" " and crossover operations typical of evolutionary algorithms,, exploring the parameter space by combining and perturbing existing solutions. Models are evaluated on development datasets, and only those that improve upon their predecessors are retained, guiding the population toward better-performe optimization with a weak to-k-to-strong approach and utilizes Fisher information to improve the selection of configurations for evaluation, aiming to find optimal merging configurations within limited computational budgets. [Wang et al. (2024b) investigated online learning with limited resources, and successfully deployed model aggregation handle changing environments, establishing optimal theoretical guarantees. Finally, [Wang et al. (2024a) proposed the Directional Preference Alignment framework, which incorporates multi-objective reward modeling to represent user preferences, offering intuitive arithmetic control over LLM generation for diverse user preferences.
solutions. [i et al. (024) employs Bayesian
Bayesian Optimization Bayesian optimization has been widely used scenarios that require efficient exploration of parameter spaces, particularly when evaluating the performance of each configuration is costly or cine-conuuming. This method is especially advantageous in machine learn- ing and hyperparameter tuming. where traditional optimization techniques may be computationally Kyzer-Knapp, 2018
prohibitive (Wilson et al. 2017 Jain et al. 2022
Moss et al. 2020
Terayama
et al. 202T
C
ADDITIONAL DETAILS ON THE METHODS
C.1 PROOF: NEGLIGIBILITY OF THE REMAINDER MULTIVARIATE TAYLOR SERIES
Corollary 2. If IR" + RR is (k + 1) times continuously differentiable in neighborhood around point a R", then the Taylor polynomial (x(x) of order l provides an accurate approximation of fz) when x is sufficiently close to a. Furthermore, the remainder term Ry(x)) becomes negligibly small as I|x all approaches zero, assuming that the (k + 1)th derivatives of f are bounded by constant M in the neighborhood between and x.
Proof Consider the Taylor series expansion of f around the point a, truncated at order k:
Tx(x) = Σ f()
(x (z-a)a
al
la|Sk
where a is a multi-index of non-negative integers, Do f(a) denotes the partial derivatives of J at corresponding to a, and (x aaaa (x) a1)at ... (Inn an) )on
Assumptions
1. Proximity: ||x - all + 0 where || denotes the Euclidean norm in R2.
2. Bounded Derivatives: There exists a constant M such that for all multi-indices a with
[a] ke + the norm of the tensor D evaluated at any point & between and is bounded by M.
D s(g)l
sup
D [() Ub+1 S M
| |1
18

---

Published as a conference paper at ICLR 2025
The remainder term of the Taylor series expansion is given by:
D (E)(E)
Rx(x)
Σ
(x a)
ad!
|a|=K+1
Given the assumptions, we estimate:
D f(E)l
M a
R& (x)) &
Σ ja|=
1
allk+1
a
k+1
a!!
a!
|a|=k+1
As | - all - O, the term || al| goes to zero. Thus, the remainder term Rx(x) becomes arbitrarily small, making negligible.
In conclusion, under the stated assumptions, the Taylor series truncated at order k. (g(x), provides an accurate approximation of (() near a, and the remainder (x( can be ignored as -æ all and the higher-order derivatives remain bounded by M.
C.2 CLOSED-FORM SOLUTION FOR SURROGATE MODEL PARAMETERS
Corollary 3 (Closed-form Solution for Surrogate Model Parameters). Under Assumption for each task 1,, I,...,N, N, the optimization problem
(A,, by (.)
arg
min Dr.
Σ CEf
Mn (0m(c)) MM(c; An, bn, ex)
is equivalent to solving the following linear regression problem:
Predictors: For each coefficient vector C2, CN) RRN. construct the predictor NIN+3)
vector Cn(c) E
(c) = [c .... ..... CN,
This vector includes:
Quadratic terms: ot for I,..,,N
Interaction terms: C,C, for < S N.
Linear terms: C, for N. N.
Constant term: 1.
2. Response Variable: Let yn R.K. be the vector of observed evaluation metries for task across all sampled coefficients Ω (c c(k)],
M, (6, (c(1))
M, 6m (c(2))
Yn
Mn (0m (c(K)))
3. Design Matrix: Construct the design matrix Cn E x (NNN+)) ((N+3) +1) corresponds to C, (ck)) for
where each row
1,..,,
NIN+3) +1 as
4. Coefficient Vector: Define the coefficient vector Bn R
(ve((An)
3, b,
en
where vec(AA,) represents the vectorization of the upper triangular part of An, including the diagonal elements.
19

---

Published as a conference paper at ICLR 2025
Optimal Solution: The parameters (Ar, byy e,) that minimize the squared error obtained via the Ordinary Least Squares (OLS) solution:
= -C, C.
This closed-form solution provides the optimal surrogate model parameters by fitting quadratic model to the observed evaluation metrics through linear regression. The design matrix C, incorporates all necessary quadratic, interaction, linear, and constant terms, enabling the surrogate model Mn Ans en) to accurutely approximate M (0 (C)) within the specified region.
Interpretation:
D ADDITIONAL DETAILS ON EXPERIMENTS
D.1 EXPERIMENT SETUP
Table 6: Experiment setup in terms of task details and models.
Task type
Metric
total tasks Model type
Zero-shot Classification (normal)
Accuracy
8
VIT-B/32 (CLIP) (Dosovitskiy et al. 2020
Zero-shot Classification (medical)
Accuracy
2
ViT-B/32 (CLIP) (Dosovilskiy al.
2020
Language Generation
Loss/Perplexity
Llama3-8B (Touvion et al. 2023
Image Classification
Accuracy
ResNet-18 (He al. 2016
D.2 ADDITIONAL METRIC: GENERATIONAL DISTANCE AND INVERTED GENERATIONAL. DISTANCE
We evaluated the quality of the Pareto front in capturing the shape of the ground truth Pareto front by measuring how much the predicted Pareto front converges to the ground truth Pareto front by calculat- ing the generational distance (GD) (Van Veldhuizen 1999) and how much the predicted Pareto front covers the ground truth Pareto front by calculating the inverted generational distance (IGD) (Coello & Cortés 2005). GD and IGD are standard measures used evolutionary multi-objective optimization to evaluate the solutions found by the evolutionary algorithms. Given predicted Pareto solution sets PF {AM} (0 mer (edegge(())) ..... My (Omer godl gea(c))) j 1,2...., K,). },). and ground truth Pareto solution sets PF2 {(M{ (8 (emergaa(C)),.... M (0merged(()) (c)) 1 1, 1,2.... ... K2} the GD and IGD metrics are defined
1/P
K
1/p
GD(PF)) =
Σ
d
and IGD(PF))
T
K
K
where d, is the minimal Euclidean distance from specific Pareto Solution in PF to PFz and d, is the minimal Euclidean distance from specific Pareto Solution in PF, to PF,
Since the Pareto front that resulted from the direct search method when the number of tasks is low (i.e. strictly smaller than 4) can be deemed as ground truth Pareto fronts, we computed GD and IGD compare the Pareto fronts produced by MAP against those obtained via brute-force grid search.
20

---

Published as a conference paper at ICLR 2025
(b)
Generational distance between the Pareto fronts found by MAP and direct search
Inverted generational distance between the Pareto fronts found by MAP and direct search
0.12
Number of tasks:
0.055
Number of tasts:
0.050
0.045
0.08
9.06
0.030
0.025
0.000
100
150
200
250
300
100
150
200
250
300
Number total points
Number total points
Figure 6: (a) Generational distance between the Pareto fronts MAP and by direct search for dimensions 2 and 3. (b) Inverted generational distance between the Pareto fronts by MAP and by direct search for dimensions 2 and 3. For both subfigures, the x-axis is the number of total scaling coefficients used by MAP. For dimension 2, direct search used 200 scaling coefficients, and 300 for dimension 3.
D.3 ZERO-SHOT MEDICAL IMAGE CLASSIFICATION
In addition to natural images, we used another dataset consisting of over 112,000 chest X-rays and 30,000 unique patients (of Health et al.) (0177). It originally contained 15 classes (14 diseases and 1 class for no finding). We split the dataset into two groups, where medical task specifically tries to classify Atelectasis, Consolidation, Infiltration, Pneumothorax, and medical task 2 tries classify Nodule, Mass, and Hernia. An example image taken from the dataset is shown in Figure (a).
(a)
(b)
Pareto fronts from grid search and MAP
0.7
Medical task accuracy
0.5
0.4
Atelectasis
Cardiomegaly
0.2
point
Pareto Front (Grid search)
Pareto Front (MAP, predicted)
Front
Mass
Nodule
0.3
0.4
Medical task accuracy
0.5
0.6
0.7
Figure 7: (a) Example figure from the NIH (of Health al. 2017) dataset. (b) Pareto fronts found by brute-force direct search using 400 points and by MAP using 30 points. We randomly sampled 25 points from the predicted Pareto front by MAP. The resulting IGD is 0.016, and GD is 0.014.
D.4 MERGING LANGUAGE MODEL
Merging Languages We merge 1lama-3-8B language models French and Arabid as well C and Japanesd] shown in Table 7 The ground truth Pareto front, however, exhibits significant limitations. contains only a sparse set of points, indicating that few merged models dominate the rest. This also suggests that the trade-off between metrics across different languages
Instruct
21

---

Published as a conference paper at ICLR 2025
is not substantial. Despite these challenges, algorithm successfully identifies the Pareto fronts. Please refer to T and Appendix [D4 GD and IGD metrics between the Pareto fronts find by Grid Search and find by MAP and a visualization of the Pareto fronts found by MAP.
Furthermore, we experimented with merging Arabic+French and Chinese+Japanese using the nested scheme described in Algorithm 2 while considering various preference settings. However, the resulting Pareto front typically contains only single model. This sparsity hinders our ability to effectively characterize a meaningful Pareto front in such scenarios.
(a)
(b)
Pareto fronts from grid search and MAP
Pareto fronts from grid search and MAP
2.03
Grid Rameto front search) Fareto Front (MSP. presiiced
Grid cata Pareto front search) Pareto Front (HMP, predicted
2.00
jspanese
2.45
2.40
183
2.35
180
2.5
1.75
2.1
2.2
2.5
2.4
2.5
2.6
2.7
2.8
2.9
3.0
Arakc
Chinaso
Figure 8: (a) Amortized Pareto front by MAP merging Llama-3 fine-tuned on French with Llama-3 fine-tuned on Arabic; (b) Amortized Pareto front on merging Llama-3 fine- tuned on Chinese with Llama-3 fine-tuned on Japanese. The yellow line represents the evaluation metric of the corresponding single task model.
Merging Language Models Specializing in Math and Code Following DARE (Yu et al. 2023
we also experimented with merging two language models each with 13B parameters specializing in mathematies (referred as the math model) and coding (referred as the code model). The base model used is Llama-2 13 the math model is WizardMath- 13 and the code model is llama-2133-code-alppp
Both the math and code models are fine-tuned versions of Llama-2 13B.
The primary reason for selecting Llama-2 as the base model, rather than newer models like Llama-3 or Qwen-2.5, is the lack of suitable math and code models fine-tuned from the same base configuration. For instance, although Qwen-2.5 provides both Qwen-2.5-Math and Qwen-2.5- differ significantly in their configurations, specifically, the rope theta parameter is set to 10,000.0 for the math model and 1,000,000.0 for the coder model. This discrepancy makes it challenging to effectively merge these models. Please refer to Figure for a visualization of the Pareto front obtained from MAP and 7 for GD and IGD metrics between the Pareto fronts obtained from Grid Search and from MAP.
the two models
Table 7: The GD and IGD metrics score for Frencc, Chinese+Japanese and Math+Code between the Pareto fronts find by Grid Search and find by MAP.
GD
IGD
GD+IGD
Arabic+French
0.023
0.035p.015
0.058,.028
Chinese+Japanese
0.014p.013
0.028p.u17
0.041p.026
Math+Code
0.0399.009
0.0.0180.002
0.057o.008
22

---

Published as a conference paper at ICLR 2025
Pareto fronts from grid search and MAP
0.45
0.40
0.35
HumanEval
0.30
0.25
0.20
0.15
Grid point dala
0.10
Pareto Front (Grid search)
Pareto Front (MAP, predicted)
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.60
GSMBK
Figure 9: Amortized Pareto front by MAP on merging WizardMath- 13B and code-alpaca-I3B. The yellow line represents the evaluation metric of the corresponding single task model.
D.5 ADDITIONAL, EXPERIMENT RESULTS ON RESNET
We performed additional experiments on ResNet18 (He et al. 2016) by merging two models finetuned
CIFARIO (Krizhevsky al. 2009) and Flowers 102 (Nilsback & Zisserman 2008) and show the obtained Pareto front in Figure T0 Unlike ViT models, which perform zero-shot classification, ResNet requires additional fine-tuning of the classification head after model merging. We demonstrate that our method still applies to those models.
Pareto fronts from empirical grid search our method
90
DOBCS6OOOK XX
80
Flowers102 accuracy
60
50
40
Grid point data
30
Pereto Front (empirical)
Poreto Front methad)
20
30
40
50
60
70
80
90
accuracy
Figure 10: Pareto front obtained for two ResNet18 models CIFAR-10 10 and Flowers-102 We perform additional finetuning of the classification head after merging the model weights.
23

---

Published as a conference paper at ICLR 2025
E MORE DETAILS ON ALGORITTMM
E PLAIN MAP
It is important note that Plain MAP (Algorithm]) D an out-of-the-box plugin-like method. Many parts of it are kept generic: there are multiple ways to sample Q in terms of the number of and the style of sampling (Bayesian or not), thus creating a tradeoff between the quality of the Pareto front and the computational time; the task vector can be computed on the more memory-capalee CPU and possibly in a parameter-efficient manner (e.g. LoRA, Adapter, and BiTFiT): for example, computing Vn via the subtraction of two full 7B models requires 2 x 23 46GB of memory capacity, but the memory cost for PEFT models may reduce to 16MB by only subtracting the non-frozen components; the for-loops in lines 2 and 4 can be computed in parallel, possibly using much smaller subset of the data, and thus enjoying a computational speed-up.
E.2 MERTED-MERGIG MAP
E.2. OVERVIEW
In this section, we explain the procedure of the algorithm in Figure detail.
Empirically, we only need 20 scaling coefficients to get Pareto front of good quality to merge 2 tasks. However, due to the curse of dimensionality, we need exponentially more pairs of (c. [M, (c))) pairs to achieve good quality of the Pareto front in the case of tasks. Theoretically, in the best case, the number of points required to obtain a good quality Pareto front for N tasks is ((N)). In the worst case, when the curse of dimensionality dominates, it could be O(N 2*). Using the nested merging scheme, we reduce the computation complexity from O(N 2 to O log N). Please refer to Table g for the detailed computational complexity comparison. Algorithm details presented in Algorithm 2
Algorithm/ 2 Nested-merging MAP
Input: A predetermined preference pref E N over the N tasks, the tuple task, loss, task vector: Gn = (taskn, Ln, Ojj))
Normalize pref to make sure the sum is
Initialize the set while |r| > 1 do
[G1,..., GN}
Find the pair of (G,, G3) that closest to each other in terms of 4, Implement Algorithm I to find the Pareto front PF.., between (Mi M,) (G) E I2 based the Pareto front PF
Select c
Merge the models by Omégge = pre + c; (6pre fj:)) + e (fp - 8%(²)
Calculate the weighted average loss on the two tasks lif = pref,4 pref,ly
Update by replacing (G,, G₁} with {Grz]. }, where G.5 (task laj. 0mdrge)
end
return 0 L.
E.2.2 TIME COMPLEXITY OF NESTED-MERGING MAP
Table 8: Computational cost of MAP variants for N models.
# evals per task
minimum # evals (total)
# evals (total)
Plain MAP Nested-merging MAP
O(N2) O(1)
O(N3) O(N N)
O(N 2N) O(N log
To estimate the computational complexity of nested-mergit meriiin MAP, we denote N as the number of tasks. The number of total evaluations needed for rested-merging MAP is: N/2 x 2 + +.... N/2" x 2m O(N log N) where 2m-1 < N < 2m
24

---

Published as a conference paper at ICLR 2025
Detailed calculations follows. When the number of tasks is 8, in the first rounds, four 2-task merging procedures are running in parallel. Each of the procedures evaluates 2 tasks. In the procedure of model-merging between 2 tasks, as discussed, evaluate 20 merged models on the 2 tasks. Thus, takes 4 x 20 x 2 = 160 times evaluation the first round. In the second round, 2 2-task merging procedures are running in parallel, each of them evaluating 4 tasks. Thus, it takes 2 x 20 x 4 160 times of evaluation. In the last round, there is only one 2-task merging and evaluation total, NMMAP takes 480 times of evaluations. Generalizing the calculation, the number of rounds can be calculated by log N. In each round, we need to evaluate T N/22 2 TN where T the number of scaling coefficient vectors needed to be evaluated when the number of tasks is 2. In the above example, 20. Thus, the time complexity O(TN log N). We rewrite as ONN log N) if ignoring 7' which is constant.
need sample 20 scaling coefficients and
8 tasks. takes 20 8 160 times of evaluation. In
E.2.3 EXPLANATION OF CURSE OF DIMENSIONALITY
Figure I1 illustrates the curse of dimensionality through the discretizations of scaling coefficients for task-specific optimization. The left plot shows a 2D grid of scaling coefficients for two tasks, while the right plot extends this to three tasks (i.e., a 3D grid). In both cases, the space of possible scaling coefficients is discretized uniformly, with resolution determined by the parameter pt __per__________ dim, which refers the number of discretized points along each axis (dimension). The spacing between adjacent points, denoted as A, is given by A pta_per_ m--- As the number of dimensions (tasks) increases, the total number of grid points grows exponentially. Specifically, the number of grid points in d dimensions is given by space. This exponential growth exemplifies the curse of dimensionality, rendering brute-force search methods computationally infeasible as number tasks (dimensions) increases beyond small threshold. Therefore, while grid search may be adequate for low- oow-iimeniiona problems, it quickly becomes impractical in nigher-dimensional settings.
S leading a combinatorial explosion in the search
Discretization when # tasks = 2
Discretization when tasks
4 1/(pts, per dim-1
1/(pts _ee__eer_____eee dim-1)
Scaling Coefficients for task
Scaling Coefficienta Tor tank
0.8
Pts dim
0.6
Pts per dim
0.4
Scating Confficient fortack
0.2
Scating Cosfficienta fortack
0.5
1
Scaling Coefficients for task
The discretization of scaling coefficients when the number of tasks is 2 and 3. As the dimension grows, preserving the same number of pts per dimension results in exponentially more grid points. This serves illustrate why when the dimension is low (e.g., < 4), we can regard the brute-force direct search method as ground truth, but it becomes insufficient when the dimension is higher.
Figure 11:
E.2.4 INTERMEDIATE PARETO FRONTS IN EACH ROUND
To determine if NMMAP affects performance negatively when compared to merging multiple models single operation. We show the intermediate Pareto fronts obtained when merging 8 models using nested-me merging MAP in Figure [2] where merge two models time. The figures illustrate the intermediate Pareto fronts obtained, where B means the model obtained by merging model A and model B.
25

---

Published as a conference paper at ICLR 2025
Pareto fronts grid search
Pareto fronts grid search
0.8
FunodATVel
0.7
MNISTAI
0.6
0.5
0.4-
Grid point data
Grid point
0.2
Fareto Front IMAP, p predicted)
Pareto front (MAP, predicted)
0.1
0.5
0.0
0.2
0.0
3.0
Cersval
GTSRBV
from grid
grid
1.0
0.7-
077746
0.6
0.6
Pereto Front (MAP, predicted)
Pereto front INAP, predicted!
0.35
0.60
0.65
0.0
SUN397VM
Pareto fronts from grid search and MAP
Pareto fronts from grid search and MAP
1.0
0.6
3.8
DTD
3.2
GUN087
0.3
2.5
3.1
Grid point data
0.1
Grid po data
Pereto front IMAP, predicted!
Pareto front (MAP, predicted)
0.0
0.0
0.2
0.4
06
0.8
1.0
0.2
0.3
0.4
0.5
0.
07
0.8
0.9
SVHN_RESI8C46
Pareto fronts from grid search and MAP
1.0
3.8
EumSAT
2.6
2.4
32-
Grid point ditts
36
Pareto front (MAP, predicted!
0.0
0.2
0.4
0.6
0.8
SVHN BECHJ_RESSSCSS.oo CTT.R MNIST SUN.TTTSTTTTOOOT DTD
Figure 12:
Illustration of the sequential steps involved in merging 8 models using -eetinng MAP (left to right, top bottom). The figures show the intermediate Pareto fronts obtained, where A_B B means the model obtained by merging model A and model B.
26

---

Published as a conference paper at ICLR 2025
E.2.5 RESULTS USING NESTED MERGING TO MERGE MODELS
We further evaluate the effectiveness of -eeging MAP. Supplemental Figure 12 shows the intermediate Pareto fronts obtained using mested-merging MAP (NMMAP) sequentially. In addition, we compared the performance of NMMAP with Plain MAP (Algorithm[]] [ The results are shown in Table 9 The results confirm that pested-merging MAP can obtain comparable results to those obtained by MAP while using much fewer evaluation runs. As discussed in Appendix E.2.2 in total, we run 40 + 80 + 160 x 480 evaluations. In contrast, when running the Plain MAP (Algorithm D we used 280 coefficients and evaluate 280 X 8 = 2240 times. In addition, while being suboptimal to MAP (Algorithm [], nested-merging MAP can still outperform other baseline methods such as TA and DARE-TA in accommodating various user preferences with relatively low computational cost, especially when the number of tasks is high.
Table 9: We compared hested-merging MAP with MAP and baseline methods in merging models using a set of 10 preference vectors in terms of preference-weighted sum of accuracies.
Metric
Singl-rask madels
Nestod-merging
TIES-merging
DARE-TIES
Tisk (TA)
DARE-TA
Preference weighted sum (1) 90.05
67.05
72.12.3.78
70.791881
70.51.33.8
59.M13.68
63.144491
E.3 BAYESIAN MAP
We further improve the efficiency of MAP by proposing adaptive Bayesian sampling algorithm. This Bayesian approach samples the points from regions with the highest level of uncertainty, where uncertainty is quantified with the Upper Confidence Bound (UCB). Please refer to Algorithm more details about the algorithm.
The discretization method shown in the Figure 13 are divided (or "binned") based on their angular components, in either 2D polar or 3D spherical coordinate systems.
illustrates how continuous decision variables
In general, a point (c1, Ca) in spherical coordinates is represented using:
Cj r cos Φ1
C2 r sin Φ1 coS
C3 sin di sin d2 d3
Cn 1 rsin Φ1 sin Φn-2 COS Φn-1
Cn = sin Φ1 ... sin on sin Φn-1
where r = ||c||2 is the radial distance (magnitude), $1...., Φn-2 E [0, Φn-1 E [0, 2T).
To discretize the direction (angular component) of a decision vector E R", we follow the steps below:
1. Normalization: Convert e unit vector:
|c||
27

---

Published as a conference paper at ICLR 2025
2. Conversion to Spherical Coordinates: Represent the direction of using (n 1) spherical angles (01. $2...., Pn-1) These can be computed recursively as follows:
Φ arccos
0f
02
2
C2
22 arccos
sin Ø1
In 3
n- = arccos
Cn-
II sin 23 j=1
R- + n-1
+ c
nn-1 arctan
Cn-1
3. Discretization of Angular Space: Divide each angle Φ, into k; uniform bins. For example, if K; 4, then:
ja
(j
1) (1+1)x
Φ
for j 0, k
2k;
2k;
where the angular domain is restricted to [0, r/2 to ensure non-negativity in all coordinates.
4. Bin Assignment:
The final bin index for is tuple of angular bin indices:
Φi
bin(c) = (b., bz bn-1), where b;
AΦ;
and ΔΦ. 25,
- denotes the bin width for ó,.
Fach bin corresponds to distinct angular sector, grouping together direction vectors with similar orientations, regardless their magnitudes.
Discretization in 2D Polar Coordination System
Discretization in 3D Spherical Coordination System
3.8
2.4
0.7
50
82
6.2
(icos(
08 Ap...
03
C) rcos
(a)
(b)
Figure 13: (a) Discretizing of two task scaling coefficients along the angular dimension in 2D polar coordinate system; (b) Discretizing three task scaling coefficients along the angular dimensions in 3D spherical coordinate system.
28

---

Published as a conference paper at ICLR 2025
Algorithm 3 Bayesian Adaptive of Surrogate Model
Input: Number of iterations J, Buffer B. Pretrained model ppee Task vectors Va. Evaluators for task N, M, (-), Discretization bin number K, sample size for every iteration nj. J = 0 to J, Bootstrap dropping rate 20%, Bootstrap sampling number Q 30.
B <-
for j = to J do
if] j 0 then
Sample scaling coefficients { from 1
else
Sample ny scaling coefficients {c, 1 based the posterior distribution
end
for i ny do
Merge the model e. (c,) = Pre +C, vn Evaluate Mn.i = MM, (f.((m B t Buf(c, mn
end
Fit the quadratic approximation surrogate model M, by learning A by e, in Equation 5 Discretize the scaling coefficients along the angular dimensions in hyper-spherical coordinates (see Figure [3 as examples)
for k 0 to K do
Calculate the mean of L2 loss between M, (c:) and Mc,c,, where c; in bin k denoted as meang
Bootstrap to estimate the standard deviation of the losses.
for q 0 to Q do
Randomly (uniformly) drop a scaling coefficient in bin & Calculate the mean of L2 loss between (c,) and (c,) with the rest points and denoted with end
Calculate the standard deviation of the f4 and denoted as std, score, meang + stdd end
Calculate probability distribution across the discretized bins by score,, as the posterior sampling strategy in the next round
end
E.3.1 PERFORMANCE OF BAYESIAN MAP
Please refer to Figure 14 for the experimental results comparing Bayesian MAP Algorithm 3 with MAP Algorithm [ The very left column shows the names of the 2 tasks being merged. Below, we define points (pts) as scaling coefficients and evaluation metrics of the corresponding merged models. Please note that the results shown in this figure the mean of 7 merging tasks that merge models: DTD+Cars, DTD+RESISC45,
GTSRB+Cars, GTSRB+RESISC4S, RESISC45+SVHN, SUN397+SVHN The Pareto front estimated by Bayesian MAP with only one iteration (6+3 pts) is shown in Figure 15
The experiments initialized with 6 pairs of c, cf [M,)a= (iter 0). In every following iteration, we sample more points following the Bayesian adaptive sampling algorithm. We compare the Bayesian adaptive sampling beginning with points and adding additional points (6 pts) with running Algorithm| [ with 9 points in a row. We also compare the Bayesian adaptive sampling beginning with points and adding 3 additional points for 2 times (6 + 3 pts) with running Algorithm [ with 12 points in row. We show that, in most cases, utilizing the same number of points, Bayesian adaptive sampling performs better than the run-in-a-row scheme Algorithm |
In conclusion, when the number of data points (scaling coefficients and evaluation metrics of the corresponding merged models) is small, Bayesian MAP Algorithm 3 performs better than MAP Algorithm[1 [T] As the number of data points increases, their performance becomes closer. Thus, recommend implementing Bayesian MAP when computational resources are very limited and the number of models (tasks) merge is not high.
29

---

Published as a conference paper at ICLR 2025
Bayesian MAP vs MAP
Bayesıan
0.35
Bayesian
MAP GD
MAP CD+XID
0.30
0.25
Ville
0.20
0.15
Royotan
0.10
Sayettnter3
Tstal number of data points collected
Figure 14: Bayesian MAP compared to MAP. The x-axis represents the number of points used by either MAP or Bayesian MAP, while the y-axis represents the value for IGD or GD. We compared MAP with 6, 9, and 12 points, and Bayesian MAP with initial points, sampling 3 more points each round for two rounds.
Pareto fronts from grid search and MAP
Pareto fronts from grid search and MAP
1.0
0.8
0.5
8 0.4
0.4
62-
Grid print
point date
Parato Front (Grid scarch) Pareto frrt (MAP, teedcctdd
0.1
Foreto Front (Grid search)
Pereto Trone (MMP, predicted)
0.4
0.2
1.0
Figure 15: (a) This plot shows failure case of the amortized Pareto front when we only have initial randomly sampled pairs of (c, { (O)) (b) After one iteration of Bayesian adaptive sampling for 3 more pairs (9 in total), the amortized Pareto front is much better than the initial Pareto front.
30

---

Published as a conference paper at ICLR 2025
F POTENTIAL Q & A
In this section, we anticipate and address some potential questions that readers might have.
F.1 THE AUTHORS ASSUME A SETTING WHERE IT IS COMPUTATIONALLY EXPENSIVE TO QUERY THE MODEL EVALUATIONS, BUT IS THIS THE REALITY?
Yes, to get the Pareto front of tasks that have trade-offs. We need to have lot of data points to show the trade-off. Here, each data point is the evaluation metric of a model. To get the ground truth evaluation metric of task (e.g. let's say classification), model to deternine the metric. If we have tasks, and we set 5 possible values (let's say 0.2, 0.4, 0.6, 0.8, 1) for the scaling coefficient vector of each model specialized for one task. We will have to evaluate 54 = 625 models on all 4 tasks. Assuming each evaluation takes 1 minute, evaluating all the models on all the tasks will take 625 models tasks 1 minute 2500 minutes 41.7 hours which is expensive. In big O notation, assuming we have T tasks, the time of evaluation for each task T. The time (computational) complexity O(T22N.
need run the evaluation script of the
F.2 How IS MAP DIFFERENT FROM OTHER MOOP METHODS, SUCH AS LINEAR SCALARIZATION?
Computational Efficiency MAP computationally efficient because it leverages surrogate model (the quadratic approximation) to estimate the evaluation metrics for different model combinations. Once the quadratic model is learned, solving the optimization problem (approximating the Pareto front) is much faster because the complexity is reduced to optimizing over the quadratic approximation, rather than needing directly evaluate the task metrics over the entire solution space.
Additionally, MAP does not require gradient descent or re- re-training when merging models and the surrogate model provides an efficient way to approximate the Pareto front without re-evaluating every model configuration.
On the other hand, linear scalarization involves computing a weighted sum of the objectives and solving it as a single-objective optimization problem. However, it often requires multiple optimization runs with different weight configurations to explore different points on the Pareto front. Since linear scalarization does not model non-linear relationships, it might require more iterations or grid searches over different weights to achieve a decent approximation the Pareto front. This can become computationally expensive, especially for large number of tasks or highly complex models.
Non-convex Pareto Fronts Many real-works multi-objective optimization problems have noncon- Vex Pareto fronts, which require more sophisticated methods. While linear scalarization can only explore the convex regions of the Pareto front, NSGA-III allows MAP to discover nonconvex regions of the Pareto front. It does this by evolving population of solutions that are non-dominated (i.e., not Pareto dominated by any other solution) and spreading these solutions across the entire Pareto front, including both convex and non-convex regions.
F.3 WHY DOES QUADRATIC APPROXIMATION HAVE A LOWER COST?
For the MAP algorithm (A (Algorithm) I the time complexity is the same as what we mentioned above: what is different that we fitted an approximation of the evaluation metrics. We only need the scaling coefficient vectors to input to a quadratic function model to be able to get the estimated evaluation score. Running the quadratic function once takes only 3.91 10 6 894 x 10 10-9s. Thus, evaluating 2500 times takes < 2500 4 x 10-68 x 10-2s.
F.4 WITY NOT COMPARE TO GIT-REBASIN (AINSWORTHI ET AL. 2023
We did not compare method to Git Re- RasBn because their study focuses the scenarios of merging the models from different initializations, whereas our background works on the same initialization of different fine-tuned models.
31

---

Published as a conference paper at ICLR 2025
F.5 WHY DOES NESTED- MAP NOT GIVE COMPLETE PARETO FRONT?
In nested- nested-merging MAP, we merge the models in pairs. For example, there are model, for task as individual fine-tuned models where 1, 2.3. 3, We first merge model and model given the preference of the user between task I and task 2. We then get model1 At the same time, we merge model 3 and model 4 given the preference of the user between task 3 and task 4 and get model3, .1- Finally, we merge the model1,2 2 and model3,1 given the preference of user to get model1, 2,33,4 We output the model1,2,3,4 as the output merged model of the algorithm. Please note that the merging order in the algorithm should be decided by the loss function clustering. It is heuristic decision and does not always dominate other merging orders. Thus, we choose not to emphasize the contribution of this order. The practitioner may use any order they think be helpful for merging.
F.6 IT SEEMS NESTED-MERGING MAP PERFORMS WORSE THAN MAP. WHAT IS THE MEANING OF IT?
In theory. the surrogate model can be easily fitted when the number of tasks is low. When it is high (e.g. 8), the fit of the surrogate model can no longer be a near-perfee near-perfect approximation (please find the R in Table 33 Thus, even if the NMMAP can only find the suboptimal solution, it is still comparable to MAP Algorithm| according to Table 9 the global optimum, but please kindly keep in mind that even if their performance is comparable, NMMAP takes way less computation for evaluation.
We understand in general that NMMAP does not find
F.7
WHY DOES
MAP NOT OUTPUT THE OPTIMAL SOLUTION?
The solution found by nested-merging MAP (NMMAP) is indeed suboptimal given the limited search space compared with merging all models at once. However, it does not mean that it is not useful in all situations. When the number of tasks is high, it has comparable performance to MAP while consuming much fewer computations for evaluation.
F,8 HOw DO YOU DEAL WITH GRADIENT AND HESSIAN IN THE SECOND-ORDER TAYLOR EXPANSION?
Notations:
p as the number of parameters in the pre- pre-trained model (also the number of parameters in each task vector).
is the matrix of task vectors of different tasks. Thus, V RRXN
c is the scaling coefficient vector E N
M,, is the metric (e.g. accuracy) for the task n.
I
Mc(c) M2
VM,
pie)
T
V
+
(c))
Hn
Ve
+
R
(6)
ER
ERixP
EREXN ERNXI
ERIXP
CRENP
ERPx1
ER
En
+
bT
T
A
(7)
ER
CRIXN
ERNXI
ERIXM
ERNXN cR
(8)
Ma(c; An, en + +bJ ATA.C
where
A V H pre) VERNNN b, = V (M, E cr. (0 + R. (9)
32

---

Published as a conference paper at ICLR 2025
Please notice that A is symmetric matrix. Specifically, when the number of tasks is 2, we have:
MI, e + bj 2 AAc
Aic
(10)
1 + 11,101 + 22200 e1
A + 1
2 A
(11)
M2(c; A2, b2, 62) 02 b yc Agc
(12)
1
2 A2 + A2 + A2,22 + 12 + 62,00 +
(13)
2
(14)
We don't calculate the gradient or Hessian to get A1, A2, bj. b2, e1 and e2. We use linear regression to estimate them. How? Given (c1, 02) pair, evaluate the merged model on task and task 2 to get the metrics (e.g. accuracy). Given 20 pairs of (c1, c2), we would be able to evaluate and get 20 corresponding, M2) which metries for fit the surrogate model M1 (c; A1, e1) and M2(c; A2, e2).
can define a merged model mergg (C1, c2). We
task 1 and task 2. Thus,
33


Reviews:
[{"review_id": "YNhjcWtIWI", "reviewer": "Reviewer_6MqJ", "summary": "When merging models (generally finetuned on different tasks), many techniques boil down to a weighted sum (generally of \"task vectors\", the difference between the finetuned model and the pre-trained model) include _per-model_ scaling parameters. This creates an exponential number of settings and makes it intractable to try all the different possible merges.\n\nNormally, merging methods are evaluated based on their average performance across many tasks, but they point out that this setting ignores the idea that a user may care more about performance on some subset of tasks than others. To capture this, they introduce the metric of the \"win rate\" how often a model on one method's Pareto frontier outperforms the models on another methods frontier.\n\nThey find that by sampling several _per-model_ scaling hyperparameters, they can use a quadratic approximation to create a better Pareto frontier with less computational resources.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper does a good job explaining the motivating the idea of using a Pareto frontier when evaluating model merging and a good job explaining their win-rate metric.\n\nThe paper gives a good overview of the quadratic approximation of the Pareto frontier.", "weaknesses": "The Pareto frontier based metric they use (win rate) is explained well, but during the comparisons to other common merging methods, it would have been nice to see another experiment that used their approach to set merging hyperparameters for those methods to see if greater average performance could be achieved. For example, comparing the avg performance of TIES with the hyperparameters from the original paper vs parameters found by their method.\n\nOften merging methods are also evaluated on if they retain the ability to generalize to new tasks, it would be nice to see some experiments to test the generalization abilities of models merged with hyperparameters found using their method.\n\nThey include some talk of using Bayesian optimization for the sampling of hyperparameters and of using nested model merging, but their discussion (intro, methods, results, etc.) for these are so sparse they should probably be cut.\n\nMAP is already a very common acronym for Maximum a Posteriori estimation. This collision will hurt adoption of their approach and is distracting as you need to keep reminding yourself is something else when you see MAP in their paper.\n\nFigure 5 is designed to demonstrate the exponential growth of having _per-model_ hyperparameters. This growth is explained well enough in the paper that such a large figure is not the most effective use of space.\n\nNit: Lots of places where references appear to be part of the text, where they shouldn't be, i.e., it is Author (year) instead of (Author, year).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DNYiVBhQvw", "reviewer": "Reviewer_97hV", "summary": "The paper introduces a new model merging method that aims to approximate the pareto front of the performance of various model merging scaling factors by a quadratic approximation of the metric that is used for performance evaluation.\nBy not requiring a full search over possible scaling factors, the amount of computation that is needed is drastically reduced.\nThe authors show that this approach is favorable especially for a larger number of tasks, where the number of possible scaling factor combinations increases exponentially.", "questions": "- In Fig 2 it is not immediately clear to me why the brute force approach of finding the best multitask scaling factor performs worst, also since you call it gold standard. Could you please explain this a bit further? What does the direct search look for exactly? Is it just over Task Arithmetic scaling factors, and if so, what grid is used?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea of focusing on trade-offs between tasks for which models are merged and Pareto fronts instead of only a single model merging solution is interesting and a useful reframing of model merging.\n\n- The method is derived from sound theory and can reduce the cost of model merging.\n\n- The method can be used as a plug-in addition to Task-Arithmetic-based merging schemes.", "weaknesses": "- The paper overall is not easy to follow. Many details are left to the reader and there is not always a clear flow in writing, requiring the reader to jump back-and-forth. In particular, the following points could be improved:\n\n- Section 2.3: it is not immediately clear why these norms are calculated, because the fact that the method uses taylor approximations is only introduced at the end of it but even then it is unclear how it ties in with the bigger picture, esp. how closeness of parameters may be related to a taylor approximation of the evaluation metric. This could be clarified. In particular, it is directly showing empirical evidence that Assumption 1 may be valid but this only comes in the section afterwards.\n\n- Section 3.1 (the main description of the method) is not written very well. For example, case 2 and 3: why is the \"warping\" by a sigmoid benificial and why does a softplus help in Case 3? Many details are left for the reader to figure out. Also, it is mentioned that you optimize Eq.5 in L252 but that you do it with gradient descent is loosely thrown in in L283. Overall, Eq.5 could be discussed more, too.\n\n- The nested MAP (nMAP) is only described in Fig. 4 of the main paper and I can not seem to find any description of bMAP at all. Could you please clarify this? While I agree that how nested merging is done is very intuitive a better description would be helpful.\n\n- It would be helpful to discuss related works more, in particular, Rame et al. 2023, who also seek to use Task-Arithmetic-based merging for Pareto fronts of multiple objectives", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mCEopRGNJy", "reviewer": "Reviewer_9bUH", "summary": "The paper introduces Model Merging with Amortized Pareto Front (MAP), a low-compute algorithm that merges multiple single-task models into a multitask model by efficiently identifying a Pareto set of scaling coefficients. MAP uses quadratic surrogate models to reduce evaluation costs while providing flexible solutions to balance competing task objectives.", "questions": "1. The paper lacks comparative experiments with other established MOOP algorithms, such as MOEA/D and NSGA-II. Including these comparisons would enhance the evaluation of both solution quality and computational efficiency, providing a clearer context for assessing the performance of the proposed method. Additionally, brute force may not be the most appropriate baseline for this type of analysis and could be replaced by a simple MOOP method like MOEA/D.\n\n2. The experiments related to large language models are somewhat limited. Typically, mainstream model fusion effectiveness is tested on benchmarks like math and code tasks, as seen in recent work such as DARE (arXiv: 2311.03099). Including comparisons on these types of benchmarks would lend stronger support to the method’s effectiveness relative to established model fusion approaches.\n\n3. The paper would benefit from additional results or comparative analysis with other state-of-the-art model merging methods, such as Adamerging (ICLR 2024) and DELLA-Merging (arXiv: 2406.11617v1). Adding these would help situate the proposed method within the current landscape and highlight any unique strengths or trade-offs.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This is the first algorithm to estimate the Pareto front for task-vector-based model merging without relying on gradient descent, which is often computationally expensive.\n  \n2. The Nested MAP variant reduces computational complexity, making it suitable for large-scale problems.", "weaknesses": "1. The motivation for applying Multi-Objective Optimization Problems (MOOP) in model merging needs further clarification. While this work represents a direct application of MOOP in this area, it lacks an in-depth explanation of why MOOP would be advantageous over traditional gradient descent-based methods. \n\n2. To enhance the clarity and impact of the paper, consider including a direct comparison with gradient descent-based optimization. Specifically, the authors could discuss MOOP’s potential benefits in terms of computational efficiency, ability to handle non-differentiable objectives, flexibility in exploring trade-offs, and its capacity to fully explore the Pareto front, which gradient-based methods may not achieve. This comparison would help elucidate the unique value of MOOP for model merging.\n\n3, The paper would benefit from a more thorough comparative analysis with recent relevant works, particularly \"Knowledge Fusion by Evolving Weights of Language Models\" and \"It's Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimization.\" Both studies propose innovative methods for model merging with evolutionary approaches.\nA direct comparison with these methods could clarify the specific advancements and trade-offs associated with the MAP approach, such as variations in fusion strategies, optimization techniques, or performance across diverse benchmarks. Discussing how MAP aligns or diverges in terms of methodology, effectiveness, or scope will provide readers with a more complete understanding of its contribution to the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)