HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models
Authors: Yu Zhou
Xingyu Wu
Jibin Wu
Liang Feng
KC Tan
Venue: NeurIPS 2025 spotlight ()
Keywords: Large language model
model merging
multi-objective optimization
architecture-level merging
Primary area: deep_learning

Abstract: Model merging is a technique that combines multiple large pretrained models into a single model, enhancing performance and broadening task adaptability without original data or additional training. However, most existing model merging methods focus primarily on exploring the parameter space, merging models with identical architectures. Despite its potential, merging in the architecture space remains in its early stages due to the vast search space and challenges related to layer compatibility. This paper designs a hierarchical model merging framework named HM3, formulating a bilevel multi-objective model merging problem across both parameter and architecture spaces. At the parameter level, HM3 integrates existing merging methods to quickly identify optimal parameters. Based on these, an actor-critic strategy with efficient policy discretization is employed at the architecture level to explore inference paths with Markov property in the layer-granularity search space for reconstructing these optimal models. By training reusable policy and value networks, HM3 learns Pareto optimal models to provide customized solutions for various tasks. Experimental results on language and vision tasks demonstrate that HM3 outperforms methods focusing solely on the parameter or architecture space.

HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models
Xingyu Wu! Jibin Wu Liang Feng* Kay Chen Tan' 'Department of Data Science and Artificial Intelligence The Hong Kong Polytechnic University, Hong Kong SAR DDepartment of Computing, The Hong Kong Polytechnic University, Hong Kong SAR
Yu Zhou'
Coolleee of Computer Science, Chongqing University, Chongqing, China zy-yu. zhouococtnn. polyu. .k (xingy. wu, jibin. actan)@polyy.. edu. hk liangf@cqu. edu. edu.
Abstract
Model merging is technique that combines multiple large pretrained models into a single model, enhancing performance and broadening task adaptability without original data additional training. However, most existing model merging methods focus primarily on exploring the parameter space, merging models with identical architectures. Despite its potential, merging in the architecture space remains in its early stages due to the vast search space and challenges related to layer compatibility. This paper designs a hierarchical model merging framework named HM3, formulating bilevel multi-objective model merging problem across both parameter and architecture spaces. At the parameter level, HM3 integrates existing merging methods to quickly identify optimal parameters. Based on these, an actor- critic strategy with efficient policy discretization employed the architecture level to explore inference paths with Markov property in the layer-gran granularity search space for reconstructing these optimal models. By training reusable policy and value networks, HM3 learns Pareto optimal models provide customized solutions for various tasks. Experimental results on language and vision tasks demonstrate that HM3 outperforms methods focusing solely on the parameter architecture space.
Introduction
Recent advancements in large pretrained models and large language models (LLMs) have demon- strated remarkable performance and strong generalization abilities across various domains, such as natural language processing 8 84, 68 Open-source communities have provided many pretrained models for various data types, as well as fine-tuned versions tailored to specific tasks. However, fine-tuning large models is often complex process that requires vast amounts of high-quality data
and computational resources 28 [6]. To address the challenge of building foundational models capable of handling diverse tasks under limited computational resources, model merging has gained increasing attention [35 Model merging leverages existing pretrained models to flcxibly transfer and integrate knowledge without requiring the original training data or additional model training (66 48 37). This approach enables the creation of new models with stronger generalization capabili- ties, suited to multiple tasks and scenarios 73 In recent years, model merging has become simple yet powerful approach for large foundational model development, with merged models showing significant potential on the Open LLM leaderboard 44 Current model merging methods primarily focus on merging models with the same architecture in the parameter space [58] 57]. They discard
"Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

most redundant parameters, and only need to design parameter adjustment strategies in the remaining
space, which often obtain moderate performance 72
80
27]. Thus, research in the parameter space
has become quite extensive and mature 24
39
20
19
18
However, focusing solely on merging models within the parameter space significantly limits their practical utility 737 Models with different architectures exhibit broader diversity in representation capabilities and task types [8 43 85 potentially expanding the performance boundaries of merged models beyond those of a single architecture. Some approaches [62
attempt to unify different architectures via knowledge distillation before performing parameter merging. However, these methods still operate within the parameter-space paradigm and typically incur substantial training costs in distillation, especially for LLMs. Recent work has explored architecture-level merging. such as Franken merging [22) and SOLAR 10.7B [29], which stitches different layers from LLMs. Nevertheless, merging models across different architectures presents several practical challenges 17 57 resulting in limited research in this area. Primarily, architecture-level merging alters the computational logic of the model, necessitating the design of coordination strategies to ensure internal compatibility and seamless information flow within the new architecture. Moreover, jointly exploring both the parameter space and architecture space increases the problem's complexity [2).. requiring well-defined search spaces and efficient search strategies to identify the optimal model configuration. Recently, evolutionary algorithm (EA) has been employed to search for optimal architectures M. However, they fail to reveal the mapping between architecture sequences and performance, making them unsuitable for handling the complex, high- dighndimensooal problem of merging multiple models. Additionally, evolutionary processes are often one-time fusions, requiring a complete restart when faced with new problems, leading to significant computational consumption [68 64).
61 7
To this end, merging large pretrain models in parameter and architecture spaces appears to be a promising approach, which can enhance the representational ability of the merged models while maintaining performance. However, research in this area is scarce, primarily because merging models in both spaces without careful consideration can undermine their internal compatibility, potentially causing a performance collapse. In addition, the complexity of the architecture space further increases the difficulty of model merging and reduces the efficiency of existing search methods M. Additionally, users may have diverse preferences and expectations for the merged model, making it crucial to weight tasks differently based on these varying preferences 34 33 33).
To merge models across both parameter and architecture levels and achieve efficient model merging schemes, this paper proposes a hierarchical model merging method (HM3) that builds a bridge for model merging in the parameter and architecture spaces. HM3 first defines a joint optimization problem for model merging that spans both the parameter level and the architecture level. Compared to existing methods, HM3 has also taken extra considerations on conflicts or trade-offs across tasks by extending this problem to a multi-objective optimization perspective. In HM3, we sample diverse preference vectors to decompose the multi-objective problem into multiple subproblems, and simultancously solve them to identify approximate Pareto-optimal merged models across tasks To relieve the strong coupling between variables and the exponentially large search space of each subproblem, HM3 transforms it into a bilevel optimization problem without compromising theoretical optimality. At the architecture level, an actor-critic reinforcement learning (RL) method is designed to explore inference paths with a Markov property in the layer- granularalll search space, enabling the reconstruction of these optimal models. To improve efficiency in the large discrete action space, HM3 incorporates a Wolpertinger strategy for policy discretization. Once training is achieved, the policy and value networks of this actor-critic strategy can be reused to predict optimal merging architectures and parameters for different tasks. The final approximate Pareto merged models meet different preferences based on specific needs and trade-offs. The main contributions of this paper are summarized as follows:
We propose the hierarchical model merging method (HM3), provide the definition of the joint model merging optimization problem that spans parameter and architect space, and transform it into bilevel optimization problem without losing theoretical optimality to relieve strong coupling and vast search space.
The proposed HM3 the first reusable model merging framework by integrating the current mergggg method and designing actor-critic-based RL method with Wolper- tinger policy discretization to guide the search, exploring the optimal model configurations in both the parameter and architecture spaces.

---

We propose to incorporate multi-objectivy optimization paradigm into model merging processing, which allows users to prioritize the importance of multiple tasks based on task needs by searching for the approximate Pareto front of merging strategy, enabling them to select the most suitable merged model.
2 Hierarchical Multi-Objective Model Merging Framework
This paper aims to jointly optimize both the parameters and architecture of pretrained models to obtain a set of approximately Pareto-optimal merged models that accommodate diverse preferences under multi-task settings. Detailed related work is provided in the appendix [ Since there is a lack of definition for architecture-level merging. we propose unified mathematical formulation for multi-objective model merging at both the parameter and architecture levels for the first time.
Problem Formulation and Challenge Discussion At the parameter level, existing works already define the merging process via optimization over θ = [6m..t, .t, 1 At the architecture level, we consider the optimization of model architecture search token traverses layers from multiple fine-tuned or merged models to identify an inference path with total length not exceeding T max- This inference path is represented by sequence ((m,, 4) )>= where (m,,4) denotes the model index and layer index selected at the 1-th search step, and the total path length. To this end, the unified optimization problem is defined as:
an inference path search problem, where
max OEPCR"O (=((m (M).)))) EM
F(O,a) = (f) f2 a)..... fx (x)
(1a)
s.t. C1: (0merh Omei, = G
( wh Ou.e. Piml k=1
(Ib)
1, if 0x,1, has the same base model as l mmyhi
C2: Wk 0, otherwise;
(1c)
C3: la
T T maxi
(1d)
C4 dii (me, Ome.e) +
0est )] 0.
(Ie)
where sa(()) for & E (1,2,..., K} denotes the performance the k-th task; P is the parameter space; is the parameters of the merged model; .M M = UT {a = {mm,, 4)) 1T) m, E {I.....KK 4 E {I...., L}} is the architecture space; C1 enforces maximum inference path length of maxx and C2 strictly ensures that the output dimension of each selected layer matches the input dimension of the next. Compared to the problem formulations of existing model merging methods, extends the formulation to the architecture level. By jointly considering both space, we elevate model merging from parameter interpolation problem to
approach
general structural composition problem.
In [.. P is constructed from multiple pretrained LLMs, which results in a high-dimensional, non- convex, and piecewise linear geometric structure. Additionally. M is discrete set of cross-model, cross-layer inference paths, whose size grows exponentially with the number of models K and the number of layers L. The strong coupling between θ and leads to an extremely large and complex joint search space. Furthermore, multi-objective function F(O, a) = (Ji,... Jx) exhibits non- smoothness, non-convexity, and non-differentiabbiiiy under such coupled variables, making it difficult to solve for traditional convex optimization or multi-objective methods. A final challenge lies in layers from different iinettuned models must be stitched together while preserving dimensional consistency across the output-input input interfaces.
Transform the Problem Formulation into A Bilevel Framework To address the strong coupling between θ and a, we reformulate ( as a bilevel optimization problem [51 [3] In this problem, the upper level searches for the optimal merged parameters O in P. while the lower level searches for the optimal inference path a* under the statie environment by the converged upper-level solution e This decomposition transforms the original joint search space of size [P] x [M into two sequential
subproblems with complexity [P] + [MM, thereby significantly mitigating the combinatorial explosion in the search process. The bilevel optimization problem is given
3

---

max OEPCR"
F(6, a' (fi(O, a² a²( fz(O, a a0))
fx(O, a'()))
(2a)
K
s.L.
θ [0mml 10,, = Σ
we Oz.t. Di=
(2b)
[1, if OE,1 has the same base model as O mm
wk 0, otherwise;
(2c)
a'(+)
E
arg max F(O,a)
(2d)
o= (mme 1 EM
s.t.
aa|
T < T maxx
(2e)
1 odi (m,, Lei mm.de) +
dim (M++1. 16-1: mm44) 0.
(2f)
Lemma 1 (Stackelberg Equilibrium 31 Thm. 3.1]). Assume the follower feasible mapping
(())== = fa E M |ai| Tmmxx dimow dimm is non-empty for all θ (since Olbase (2), and its
graph is closed due to (A]). Under Assumption] [ the bilevel optimization problem ② admits at least one Stackelberg equilibrium (0* xx).. The associated leader-follower payoff corresponds to global optimum of the original problem (T. The proof of Lemma is provided in the appendix B.T
In the appendix B.I Stackelberg game, for which an equilibrium solution exists. This ensures that problem transformation does not incur any loss of optimality. In this bilevel optimization problem, the lower-level optimization searches for the optimal merged model architecture. The resulting architecture determines the length of the inference path (i.e., the number layers to be merged). The upper-level optimization then operates on the parameter set {0 m. Y corresponding this architecture. Consequently, the optimal architecture found by the lower level dynamically determines the dimensionality and scale of the parameter search space for the upper level. This naturally forms hierarchical decision-making structure, i.e., first optimizing the model architecture, then optimizing the corresponding model parameters, which embodies the core hierarchical nature of the proposed HM3.
we further prove that the bilevel optimization problem can be modeled as a
Model the User Preference into A Multi-Objective Optimization Problem After mitigating the strong coupling between variables. we focus on the diverse user preferences, we adopt a decomposition
property of (2). To accommodate strategy that explicitly guides the solution set to cover the Pareto front boundary under controllable preference vectors. Due to the non- convexity of the search space, we employ Tehebycheff decomposition strategy, which effectively approximates non-convex Pareto fronts by transforming the original multi-objective problem into N preference-weigghee scalar subproblems. By solving these subproblems in parallel, we obtain a set of approximately Pareto-optimal merged models that satisfy varying user preferences.
Specifically, we begin by generating N preference weight vectors {A (N j from K-dimensional probability simplex A = (X E RK EK d 1}. In this paper, we sample from the Dirichlet i i= 1=1..... N, which ensures uniform coverage over A
distribution: X Dirichlet(1,
with an unbiased mean E[Ag] K'
Then, we estimate the ideal point of each objective by computing the best achievable task performance across all fine-tuned LLMs: z = aa)e ax.. Finally, for each preference vector X (0), the upper-level subproblem using Tehebycheff scalarization is defined
θ ( arg min OEP
max k=I.....K K
x
pura O
- =
(3)
where F.para (e) denotes the objective function value of the merged model on the k-th task, obtained after solving the corresponding lower-level inference path problem with fixed e.

---

Pronter
Transffomm MOP
Actor-Critic
Create Modei-Layer each prefarence vector n (Subproblem
Reuse after wellowiine
Agant
Environment
inference
path
Teak1
Opáinal mode in peomeeter leaal (Mocel K-1)
dkip anallie
Figure 1: Illustration of architecture-level model merging. For each scalarized subproblem, we construct model-layer candidate pool consisting of all layers from the parameter-level merged model, and K fine-tuned models. Then, HM3 design an actor-critic algorithm with Wolpertinger discretization search inference path. The final merged models approximate the Pareto front.
Once (+) obtained, the corresponding lower-level subproblem is defined as:
(6) ( ( =i|
arg min (E
max I==I...,K K
where (()) denotes the feasible set of inference paths that satisfy C1 and C2.
Through this transformation, 11 is reduced solving N scalarized bilevel subproblems, each corresponding to distinct preference vector (6), The set of solutions all subproblems forms an approximate Pareto-optimal set of merged models.
HM3 Algorithm
Parameter-Level Optimization After generating preference vectors (X(*)
(x(e) ..... . J
we proceed to search for the optimal merged parameters in the parameter space for each preference vector. Thanks to recent advancements, parameter-level merging methods have become relatively mature and efficient. Our framework is designed to compatible with these existing techniques, such as DARE-Teee Ties merging method. Concretely, for ( we begin by computing the residual vector: 6% θ θ base. We then apply the Drop-and- Resseae operation to obtain 8p 6x/(1 p). Next, perform the Ties Merging 772 procedure for X ( removing redundant parameters from each 5PR, generating sign-con sign-consistent aggregation mask across tasks, and merging disjoint residual fragments with consistent signs to form 8.- Finally, the optimal parameter for the i-th subproblem is:
θ + basc +
M + 6R.
(5)
where M is a binary mask that controls which elements of 6, are preserved and rescaled.
Architecture-Leve Optimization As discussed in Section 2 architccture-level optimization is formulated searching for optimal inference path across the merged model and its multiple task-specific fine-tuned models. For each (+), we have already obtained the corresponding optimal merged model at the parameter level, denoted as (6),, We assign its model index as m = K + 1. To further expand the search space and leverage external knowledge, we construct a model-layer candidate pool that consists of: (1) the optimal merged model (i)) from the parameter level, and (2) all layers from the K task task-specific fine-tuned models used in construction of (1),, The corresponding architecture-levl search space is then updated as:
M = (m) (1)) (()) m ......,,+11. Le (1,...,,1) L)].
Although prior research has demonstrated the potential of using search algorithms to optimize layer sequences and enhance model merging performance I the scalability EAs suffers significantly as the number of models and layers increases 20 Moreover, EA-based approaches require

---

training from scratch for each preference vector and incur considerable computational cost due to population- based evaluations in every generation. These inefficiencies motivate us to revisit the nature of dynamic layer selection across multiple models 66 67
This process entails selecting the optimal model-layer pair at each step, considering the long-term impact of current decisions on future layer compositions and final task performance, thereby exhibiting the characteristics of a sequential decision-making problem. Furthermore, the combinatorial nature of the layer-path space, along with its discrete, structured constraints and well-defined state transitions, naturally suggests formulating the inference path search as sapeetory-awar Markov decision process (MDP). The overall process of architecture merginn is illustrated in Fig. | define its state, action, transition, and reward components and design an RL strategy to efficiently explore optimal architecture trajectories.
Then, we formally
1) State Space Since every decision the inference path dynamically alters the feasibility of subsequent layer transitions and affects the accumulated representation distribution, we define the state to retain full trajectory history for optimal distinguishability Formally, the state at the tth step is represented trajectory:
S. {mi. lis O mmg,4)) ) my {1....., K 13,1, (I...., -... E S,
(7)
where m, denotes the model index, ly denotes the layer index, and m E Rd is the corresponding parameter of the selected layer. To enable policy gradient-based learning, encoders vm. 1h, and yp to encode model identity, layer index, and layer parameters, respectively. The full trajectory is then embedded into a fixed-dimensional vector using GRU encoder: h,
of learnable
Rdd,, The process on trajectory space S satisfies Markov GRU( ([)m)(m)) (m))) (0 (0mm,.))) )1 property.
2) Action Space transition to:
At the fth step, the action A, is defined as selecting the next model- pair to
A = (m) m (I..... K 1}, L LL.
(8)
3) Reward Function. high-quality multi-takk performance with minimal complexity:
The reward encourages the construction of efficient inference paths that yield
K 1 f.. h) -.TT
R=
where the second term penalizes path length to encourage shorter and more efficient inference paths. The reward computed only after the entire inference path is generated, and the MLP-based alignment is performed. This reward is uniformly assigned all time steps in the inference path as R, = R/T, Vt E (1....,T), This uniform assignment is implemented to facilitate efficient storage of transitions and subsequent updates of the policy and value networks in the actor-critic framework.
Actor-Critic Method To solve the MDP, HM3 employs an actor-critic-basee RL strategy. In this framework, a policy network parameterized by μ outputs probability distribution over the large discrete action space, while a value network parameterized by á serves as baseline to reduce the variance of gradients under sparse reward conditions. This design facilitates stable convergence of layer sequence search under limited sampling (8).
The policy function n. (Ac S,) defines stochastic policy conditioned the current state S., representing the probability distribution over candidate actions. The distribution is modeled using a Gaussian parameterization with mean f and variance ç², from which actions are sampled to maximize the expected cumulative reward:
T
K Σ x
max 1
EM
Σ
7 Re
= max E
Σ
fe
B1t
(10)
-
where 7 E (0, 1] is the discount factor, and Rg denotes the reward at the 9th step defined earlier.

---

The policy network generates a continuous proto-action A following a Faussian-distribune stochastic policy: Á f=(AAss:e)(S) NV (U,),,), diag o (S.)). where f=(A s:0) the state-to-action mapping under policy и.
Since the decision variables of (2) lie in discrete action space W, the proto-action A must be mapped to a discrete action A E W. Existing discretization approaches fall into two categories The simple projection method, which directly selects the nearest discrete action: A' arg min | All. However, this can result in suboptimal exploration and slow convergence. The greedy method, which selects the action with the highest Q-value: A'' = arg AeW Q(S,A), but this is often computationally expensive and prone to local optima.
[55 833:
To balance exploration and exploitation, we introduce Wolpertinger policy mapping, which improves efficiency by limiting evaluation to a local neighborhood:
arg AEW) (s, (A:),,
with probability with probability 6,
A,
(11)
where ((()) is uniform distribution, and the neighborhood set (*,)), is defined as: (*,)) := argmin AEW, d(AA,)) with W (A,...,,Ae Ar} denoting the full discrete action set, H=M
d(s;)) representing Fuclidean distance, and M' is the number of nearest neighbors.
The value function is modeled by a state-value network (St),, which estimates the expected cu- mulative reward of Sy V (St)= = EIT [こ= 7 Rets S. where Σド=o 7 Re+5 is the discounted return starting from the 1-th step.
To stabilize policy optimization, HM3 constrains the update step size and adopts policy gradient approach for training. The policy network is updated using the clipped surrogate objective by proximal policy optimization
Network Updates
(4)]
L CLIP (H) = EE, min (polu)Ae, -1p(
(12)
is the importance sampling ratio between the new and old policies, and Å, is the estimated advantage. We compute Å, using the generalized advantage estimation method: Å, = (7 (++i, G R₁ (S @ter) ((S; oierr, where Ço is the temporal difference residual at the t-th step.
where p((p)= n2 (s.))
Fuee (A,))))
The value network is updated by minimizing the value loss:
L ( EE [Vvs(S)) -.)))
(13)
The overall training objective is:
L(p, = (M) + (t) - (T))
(14)
where HTT,)) denotes the entropy of the policy, and C1, C2 are weighting coefficients.
Lemma 2 (Advantage of Wolpertinger discretization). Let Q(S,A) r(S, (V( de- note the one-step proxy score derived from the value network Va- Consider a candidate set W = {Ai,..., At...., Aw). Assume there exists constant > 0 such that: Q(S, Ac) ~ u (Q(S, - c. Q(S, + 5) ve + E', and that the proxy error is bounded as:
((SSA)) Q(S, 5., YAEW".
(15)
When M 1 and 8 <<(1= (1-
we can confirm that Wolpertinger expected to outperform simple nearext-neighhor projection in terms of the true Q-value. Moreover, by reducing the candidate space from [W] to [W*1, Wolpertinger achieves greater efficiency than full greedy search over all actions. The proof of Lemma 2 is provided in the appendix B.2
112M-1) M.2
7

---

Dimension Alignment via Statistical Matching To accommodate distributional shifts across layers from different models, we introduce feed-forward MLP network that generates a scaling matrix Wm,1- The input to the MI. consists of the layer index pair (m. !) and the current time step 4, and its output defined as:
mz (m,,,,
(16)
where MLP,, y is parameterized by μ and optimized via actor-critic method. This design is motivated by the theory of moment matching [33]. Further theoretical details are provided in the appendix B.4 It is worth that the proposed HM3 is in its early exploratory stage, and we discuss the existing
limitations and possible future directions in the appendix
3.1 Experiment Sctup
Baselines We evaluate the proposed parameter-anc parameter-and-architee and-architecture hierarchical merging framework HM= three types of baselines on both language and vision tasks: fine-tuned models, three classical parameter-level merging methods, including Task Arithmetic [27]. Mees-ng [72) and
DARE-Ties Merging [0). two SOTA parameter-level merging methods, including PCB Merging
19 and Consensus Merging 665] and an architecture-leve merging method named EA [1
Benchmarks and Metries For language tasks, we used LLAMA-2-7B 60
Qwen-2.5-115B 77
and LLAMA-2-13B [60] as backbones across four subtasks: generative task, text translation, math reasoning, and code generation. For generative tasks, used GLUE benchmark [63] to evaluate the general capability of large pretrained models. For translation, we used WMT14, WMT16 [50], and IWSLT2017 77] (WMT&ISTT), evaluated by the chrf metric as well as Xnli [13 evaluated by the accuracy metric. For math reasoning, we used GSM8K [12 with the flexible match metric, and used MathQA (3] with the accuracy metric. For code generation, HvmanEEal was used with the passê1 and pass@100 @100 metric. Additionally, Qwen-2.5-1.55 was evaluated on four 3090 GPUs (24GB each), while LLaMA-2-77 and LLaMA-2-13B were evaluated on four A6000 GPUs (48GB each). All models can also be deployed on a single GPU. For vision tasks, we adopted ViT-B/32 and ViT-L/14 from CLIP 46
p and MBPP (5]
backbones, and evaluated on eight datasets: DTD
GTSRB [52) RESISC45 [O), SUN397 (70), SVHN [5]. MNIST [32), Cars 300 and EuroSAT 23
using classification accuracy. Other settings and details are summarized in the appendix C.I
3.2 Performance of Multi-Task Scenario
Merging LLAMA-2-7B LLMS Table summarizes the performance of various merging methods across three language tasks on LLAMA-7 7B series LLMs. Among the fine-tuned models, WizardMath- 7B excelled at math due to task-specific training. while CodeLlama-7B [47] dominated code generation. Llama-2-7B 2-aa-------hatt [60) showed relatively balanced performance, particularly in translation. Across merging methods, Task Arithmetic provided moderate gains across tasks, whereas Ties Merging and DARE-Ties Merging achieved better trade-offs, especially in translation and code. However, EA underperformed, likely due to its unguided architecture search, which struggles to find optimal layer combinations with limited evaluations. Our proposed HM3 significantly outperformed all baselines, achieving top scores in all tasks. These results highlight the effectiveness of jointly optimizing both parameter fusion and architectural composition.
Merging Qwen-1.5B LLMs To assess the robustness of HM3, we conducted merging experiments using the Qwen-2.5-1.5BB series LLMs. As shown in Table 2 each fine-tuned model performed best on its own task but showed clear limitations on others, reflecting the trade-offs of single-task fine-tuning. In contrast, HM3 consistently outperformed all baselines, achieving top results in math and code, and competitive performance in translation. EA performed the worst across all tasks due to its unguided structure search. An interesting observation from Table and Table is is that the models by HM3 sometimes outperform fine-tuned models, which are typically considered performance upper bounds for their respective tasks. We discuss this in the appendix C.2 Additionally, we conducted the experiment on LLAMA-I 13B, and the results and analysis sis are provided in the appendix C22)
"The implementation of HM3 is available available this page
8

---

Table 1: Comparison of merging methods for Llama-7B series LLMs on language tasks
General
Translation
Math
Code
Merging Methods
Glue
WMT&ISWT
Xnli
GSM8K
MathQA
IvmmanBval
MBPP
Fine-tuned Model Chat
55.97
40.23
43.21
15.39
25.33
19.51
24.47
Fine-tuned Model Math
29.32
34,97
38.93
45.79
27.09
20.73
16.96
Fine-tuned Model Code
18.39
33.86
40.37
12.89
28.76
43.21
Task Arithmetic
35.32
31.30
32.92
37.83
17.30
21.36
22.20
Ties Merging
38.62
34.15
35.72
29.73
22.40
26.35
DARE-Tic Ties Merging
37.03
33.93
37.47
38,20
22.70
28.20
30.11
Consensus Merging
47.52
37.97
40.74
37.95
27.99
30.15
37,93
PCB Merging
49.11
39.29
33.35
39.25
28.14
32.53
39.05
EA IIM3
21.33
37.51
27.07
25.51
22.64
25.17
16.84
51.04
41.68
40.24
45.62
28.08
43.62
44.62
Table 2: Comparison of merging methods for Qwen-1.5BB series LLMs on language tasks
Generative
Translation
Math
Code
Merging Methods
Glue
WMT&ISWT
Xnli
GSM8k
MathQA
HumanEval
MBPP
Fine-tuned Model Chat
$7.76
39.01
41.39
14.32
28.67
12.11
40.60
Fine-tuned Model - Math
41.95
23.08
35.36
32.61
43.33
13.90
44.05
Finc-tuned Model Code
28.30
24.71
41.60
15.60
32.67
34.42
52.34
Task Arithmetic
42.76
27.40
30,90
19.73
37.71
17.63
21.45
Tics Merging
42.72
29.07
28.46
22.66
36.07
16.32
19.61
DARE- Tics Merging
38.25
27.91
30.87
20.63
40.33
19.61
24.84
Consensus Merging
46.42
29.82
38.09
23.98
37.84
22.59
34.08
PCB Merging
47.25
30.05
38.31
24.29
36.90
21.87
41.33
29.77
21.36
23.87
17.40
35.68
15.33
23.27
HM3
48.22
32.26
41.73
28.05
40.13
34.31
51.80
Merging ViT-B/32 model As shown in Table 3 HM3 outperforms all baselines with an average accuracy of 66.91%. It achieves 77.21% EuroSAT, 77.62% on SVHN, and 68.21 % on GTSRB. While slightly lower on DTD, HM3 still surpasses Ties Merging and Task Arithmetic.
Merging VIT-L/14 Table shows that HM3 consistently achieves the best results across most datasets, with 90.48% on SVHN and 83.43% GTSRB. The overall average accuracy reaches 80.30%, significantly exceeding all other methods. The detailed analysis in the appendix C.2
3.3 Performance of Multi-Objective Model Merging
HM3 generates diverse set of approximately Pareto-optimal merged models, enabling flexible adaptation to different user preferences. Unlike existing methods that output single solution, HM3 provides multiple high-quality candidates. To evaluate solution quality, we compute Pareto dominance relations by pooling all solutions. A solution is dominated by 2a if at is no worse in all objectives and strictly better in at least one. Figure? 2 shows that every bascline dominated by at least one HM3 solution (S1-S15), demonstrating HM3's superiority in objective space. We also compare HM3 with multi-objective evolutionary algorithm (MOEA) bascline using the hypervolume (HV)
Table 3: Performance of different model merging methods for ViT-B/32 series models on vision tasks.
Method
Average
SUN397
RESISC45
SVHN
GTSRB
DTD
MNIST
Cars
EuroSAT
Task Arithmetic
69.44
61.41
72.42
73.74
66.12
49.82
93.81
62.14
76.09
Ties Merging
69.00
62,34
71.49
73.68
62.69
48.52
96.91
61.06
75.30
DARE- Ties Merging
69.86
60.22
71.36
76.56
65.94
50.84
97.05
60.84
76.05
Consensus Merging
72.06
64.73
73.51
79.46
69.03
52.63
96.89
63.06
77.20
PCB Merging
73.80
63.58
75.71
82.31
72.57
54.78
97.42
64.42
79.63
EA HM3
59.45
53.27
62.14
59,32
56.16
32.97
95.34
$4.03
62.33
73.83
63.42
76.27
82.11
73.11
54.60
96.85
64.63
79.66

---

Table 4: Performance of different model merging methods for ViT-L/14 series models on vision tasks.
Method
Average
SUN397
RESISC45
SVHN
GTSRB
DID
MNIST
Cars
EuroSAT
Task Arithmetic
79.48
69.56
83.60
80.51
70.58
65.88
98.02
82.13
85.53
Ties Merging
81.28
68.53
81.89
87.42
81.72
58.07
98.89
84.97
88.77
DARE-TiSs Merging
83.72
72:07
87.19
88.03
84.50
64.49
99.01
85.93
88.53
Consensus Merging
83.75
73.39
88.05
87.43
81.16
66.04
98.88
84.26
90.81
PCB Merging
85.23
75.04
88.75
86.46
86,55
69.13
98.91
86.01
91.01
EA
69.48
61.95
58.11
76.32
66.36
50.04
96.77
75.04
71.24
HM3
85.34
74.76
88.43
90.02
85.17
70.21
98.44
86.33
89.32
51
Be Dominaned
45
54
56
Sel
59
51
Task Artivnatic
512
30
Dur Tims
Tiee
518
Usma
515
Doniiaate
25
WitasMaah
Anthmets
Tias
DARE 1i8s
No Doninsaedd
Laaa Pepoooa conma
Talk DAKE
15
Math
Figure 2: Illustration of metrics for different merging methods, where S1 represents solu- tionl obtained by HM3.
Figure 3: The illustration of different model merging methods in the math reasoning and code generation tasks.
metric 25 56]. which reflects both convergence and diversity. HM3 achieves an HV of 1.8120, significantly higher than MOEA's 1,5111, highlighting the limitations of unguided evolutionary search in complex multi-objective scenarios. Detailed analysis is provided in the appendix C.3 The effectiveness of HM3 on different numbers of objectives is provided in the appendix C.4
3.3.1 Ablation Study
To evaluate the effectiveness of jointly optimizing parameter and architecture spaces, we conduct ablation studies on three variants: (i) HM3, (ii) HM3 w.o. arch (no architecture optimization). and (iii) HM3 w.o. para (no parameter optimization), with results shown in Table in the appendix C.5 In the single-objective setting, HM3 outperforms both ablated versions on all tasks, especially in code generation, highlighting the synergy between parameter and architecture optimization. In the multi-objective setting, HM3 achieves the highest HV score, followed by HM3 w.o. arch, while HM3 w.o. para performs the worst. This demonstrates that parameter optimization is critical for overall performance, and architecture optimization further enhances solution quality. Detailed analysis provided in the appendix C.3 We also analyze the computational cost of HM3 compared to the conventional pretraining and fine-tuning paradigm in the appendix C.6 Additionally, convergence analysis of RL is provided in the appendix C.7
Conclusion
In this paper, we propose HM3, a hierarchical model merging framework that jointly optimizes parameter and architecture spaces. By leveraging an actor-critic strategy and cefference-guidee multi-objective optimization, HM3 efficiently generates customized, high bighhreeformin merged models. Extensive experiments on translation, math reasoning, and code generation tasks demonstrate HM3's superiority over existing methods. The framework learns Pareto-optimal solutions tailored to diverse user preferences, offering flexible and scalable approach to model merging. Future work will explore applying HM3 to larger-scale pretrained models for broader generalization and adaptability.
10

---

Acknowledgment
This work was partially supported by National Natural Science Foundation of China under Grant U21A20512 and in part by the Research Grants Council of the Hong Kong SAR under Grant No. C5052-23G, Grant PolyU 15229824, Grant PolyU 15218622, and Grant PolyU 15215623, This work was also partially supported the Research Grants Council of the Hong Kong SAR (Grant No. PolyU15217424, PolyU25216423) and The Hong Kong Polytechnic University (Project IDs: P0043563). This work was also in part by the Natural Science Foundation of Chongqing (Innovation and Development Joint Fund) under Grant CSTB2025NSCO-12X00)
References
[1] Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 7(2):195-204, 2025.
[2] Charalambos D Aliprantis and Kim C Border. Infinite dimensional analysis: hitchhiker's guide. Springer Science & Business Media, 2006.
[3] Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedzionss Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation- based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 2357-2367, 2019.
[4] Didier Aussel and Anton Svensson. A short state of the art on Bilevel optimization: Advances and next challenges, pages 53-76, 2020.
games.
[5] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint 2 07732, 2021.
[6] Claude Berge. Topological Spaces: Including Treatment of Multi-valued Functions, Vector Spuces, and Convexity. Courier Corporation, 1997.
[7] Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuitho Sudoh, Koichiro Yoshino, and Christian Federmann. Overview of the iwslt 2017 evaluation campaign. In Proceedings of the 14th International Workshop on Spoken Language Translation, pages 2-14, 2017.
[8] Yupeng Chang, Xu Wang. Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang. Yidong Wang, et al. A survey evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1 2024.
[9] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint rrXiv:2107.03374 2021.
[10] Gong Cheng. Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art Proceedings of the IEEE, 106(10):1865-1883, 2017.
[11] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[12] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 2021.
[13] Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, 153:235-256 2007.
[14] Tianshuo Cong, Delong Ran, Zesen Liu, Xinlei He, Jinyuan Liu, Yichen Gong, Qi Li, Anyu Wang, and Xiaoyun Wang. Have you merged my model? on the robustness of large language model ip protection methods against model merging. In Proceedings of the 1 ACM Workshop on Large Al Systems and Models with Privacy and Safety Analysis, page 69-76, 2024.
11

---

[15] Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingus cons-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2775-2485,, 2018.
[16] Xiaoyu Dong, Yujie Feng, Zexin Lu, Guangyuan Shi, and Xiao-Ming Wu. Zero-shot cross- domain dialogue state tracking via context-aware auto-prompting and -ooetioonoooooiing contrastive decoding. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pages 8527-852555 3527-8540, 2024.
[17] Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. Nats-bench: Benchmarking nas algorithms for architecture topology and size. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3634-300 2021.
[18] Guodong Du, Zitao Fang, Jing Li, Junlin Li, Runhua Jiang, Shuyang Yu, Yifei Guo, Yangneng Chen, Sim Kuan Goh, Ho-Kin Tang, al. Neural parameter search for slimmer fine-tuned models and better transfer. arXiv preprint arXiv:2505.18711 2025.
[19] Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, al. Parameter competition balancing for model merging. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pages 84746 84776, 2024.
[20] Guodong Du, Jing Li, Hanting Liu, Runhua Jiang, Shuyang Yu, Yifei Guo, Sim Kuan Goh, and Ho-Kin Tang. Knowledge fusion by evolving weights of language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 11727-11742, 2024.
[21] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023.
[22] Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403. 13257, 2024.
[23] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2177-2228 2019.
[24] Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr- Emr-merging: Tuning- free high gegh-performance model merging. In Proceedings of the 38th International Confer- ence Neural Information Processing Systems, 2024.
[25] Simon Huband, Philip Hingston, Lyndon While, and Luigi Barone. An evolution strategy with probabilistic mutation for multi objectvb optimisation. In The 2003 Congress on Evolutionary Computation, volume 4, pages 2284-2291. IEEE, 2003.
[26] Binyuan Hui, Jian Yang. Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12188 2024.
[27] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference Leurning Representations, 2023.
[28] Dong-Hwan Jang, Sangdoo Yun, and Dongyoon Han. Model stock: All we need is just few fine-tuned models. In European Conference on Computer Vision, pages 207-223. Springer, 2025.
12

---

[29] Sanghoon Kim, Dahyun Kim, Chanjun Park, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeon- woo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 6: Industry Track), pages 23-35, 2024.
[30] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
[31] Ankur A Kulkarni and Uday V Shanbhag. An existence result for hierarchical stackelberg v/s stackelberg games. IEEE Transactions on Automatic Control, 60 60(12):3379-3384 2015.
[32] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
[33] Bingdong Li, Zixiang Di, Yanting Yang, Hong Qian, Peng Yang, Hao Hao, Ke Tang, and Aimin Zhou. It's morphing time: Unleashing the potential of multiple Ilms via multi oujti-objecttve optimization. IEEE Transactions Evolutionary Computation, pages 1-1, 2025.
[34] Lu Li, Tianyu Zhang, Zhiqi Bu, Suyuchen Wang, Huan He, Jie Fu, Yonghui Wu, Jiang Bian, Yong Chen, and Yoshua Bengio. Map: Low-compute model merging with amortized pareto fronts via quadratic approximation. arXiv preprint arXiv:2406. 240.0:240000729, 2024.
[35] Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309. 15698, 2023.
[36] Zhuo Li, Guodong Du Weiyang Guo, Yigeng Zhou, Xiucheng Li, Wenya Wang, Fangming Liu, Yequan Wang. Deheng Ye, Min Zhang, et al. Multi-objective large language model alignment with hierarchical experts. arXiv preprint wwXiv:2505.2092. 2025.
[37] Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang. Merge, ensemble, and cooperate! survey on collaborative strategies in the era of large language models. arXiv preprint arXiv:2407.066089 2024.
[38] Wei Lu, Rachel K Luu, and Markus Buehler. Fine-tuning large language models for do- main adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities. npj Computational Materials, 1 11(1):84, 2025.
[39] Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin- merging: Dynamic integration of modular expertise in model merging. In Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024.
[40] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-Guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, et al. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced wvol-instruct. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025.
[41] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.099 099.8833, 2023.
[42] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with -vostinst In Proceedings of the Twelfth International Conference on Learning Representations, 2024.
[43] Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning, pages 7588-7598. PMLR, 2021.
[44] Aidar Myrzakhan, Sondos Mahmoud Bsharat, and Zhiqiang Shen. feen-llm-leaderbard From multi-choice to open-style questions for llms evaluation, benchmark, and arena. arXiv preprint arXiv:2406.07545, 2024.
13

---

[45] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits natural images with unsupervised feature learning, In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 2021.
pages
[47] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
[48] Wei Ruan, Tianze Yang, Yifan Zhou, Tianming Liu, and Jin Lu. From task-specific models to unified systems: A review of model merging approaches. arXiv preprint arXiv:5503.08999 2025.
[49] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint AXXv:1707.06347 2017.
[50] Rico Sennrich, Barry Haddow, and Alexandra Birch. Edinburgh neural machine translation systems for wmt 16. arXiv preprint 160iv 120.00299,, 2016.
[51] Ankur Sinha, Pekka Malo, and Kalyanmoy Deb. A review on bilevel optimization: From classical to evolutionary approaches and applications. IEEE transactions on evolutionary computation, 22(2):276 295, 2017.
[52] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian lgel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 2011 International Joint Conference on Neural Networks, pages 1453-1460. IEEE, 2011.
[53] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision ECCV 2016 Workshops, pages 443-450. Springer, 2016.
[54] Wenju Sun, Qingyong Li, Yangliao Geng, and Boyang Li. Cat merging: A training-free approach for resolving conflicts in model merging. In Forty-second International Conference on Machine Learning. 2025.
[55] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018
[56] Kay Chen Tan, Eik Fun Khor, and Tong Heng Lee. Multiobjectiive evolutionary algorithms and applications. Springer Science & Business Media, 2005.
[57] Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Fusionbench: A compre- hensive benchmark of deep model fusion. arXiv preprint 20X.v:2406.03280, 2024.
[58] Qiaoyu Tang, Le Yu, Bowen Yu, Hongyu Lin, Keming Lu, Yaojie Lu, Xianpei Han, and Le Sun. A unified view of delta parameter editing in post-trained large-scale models. arXiv preprini arXiv:2410. 13841, 2024.
[59] Qwen Team. Qwen2.5: A party of foundation models, September 2024.
[60] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.0922. 2023.
[61] Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowl- edge fusion of large language models. In The Twelfth International Conference on Learning Representations.
[62] Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, and Xiaojun Quan. Fusechat: Knowl- edge fusion of chat models. arXiv preprint 240.:22988.79990 2024.
14

---

[63] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the Seventh International Conference on Learning Represenlations, 2019.
[64] Chao Wang, Jiaxuan Zhao, Licheng Jiao, Lingling Li, Fang Liu, and Shuyuan Yang. When large language models meet evolutionary algorithms: Potential enhancements and challenges. Research, 8:0646, 2025.
[65] Ke Wang. Nikolaos Dimitriadis, Guillermo Ortiz-Jiménez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. In Proceedings of the 41 International Conference on Machine Learning, pages 10268-50287 2024.
[66] Tom White. Sampling generative networks. arXiv preprint 1609:1660.044400 2016.
[67] Xingyu Wu, Jibin Wu, Yu Zhou, Liang Feng, and Kay Chen Tan. Towards robustness and explainability of automatic algorithm selection. In The 42rd International Conference on Machine Learning (ICML 25), 2025.
[68] Xingyu Wu, Sae Wu, Jibin Wu, Liang Feng, and Kay Chen Tan. Evolutionary com- putation the era of large language model: Survey and roadmap. IEEE Transactions on Evolutionary Computation, 29(2):534-554, 2025.
[69] Xingyu Wu, Yan Zhong, Jibin Wu, Binghing Jiang, and Kay Chen Tan. Large language model- enhanced algorithm selection: towards comprehensive algorithm representation. In Proceedings of the Thirty-Thine International Joint Conference on Artificial Intelligence, pages 5235-5244, 2024.
[70] Jianxiong Xiao, Krista Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 119:3-22, 2016.
[71] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In Proceedings of the Twelfth International Conference on Learning Representations, 2024.
[72] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: re- solving interference when merging models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 7093-7115, 2023.
[73] Prateck Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging scale? arXiv preprint arXiv:2410.03611, 2024.
[74] Zhaoyi Yan, Yiming Zhang, Baoyi He, Yuhao Fu, Qi Zhou, Zhijie Sang, Chunlin Ji, Shengyu Zhang, Fei Wu, and Hongxia Yang. Infifusion: A unified framework for enhanced cross- model reasoning via Ilm fusion. arXiv preprint arXiv:2501. 2025.
[75] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1511 2024.
[76] Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-matt technical report: Toward mathematical expert model via self-improvement. arXiv preprint EXXi:22409.12222 2024.
[77] Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in Ilms, milms, and beyond: Methods, theories, applications and opportu- nities. arXiv preprint arXiv:2408. 07666. 2024.
[78] Jian Yao, Ran Cheng, Xingyu Wu, Jibin Wu, and Kay Chen Tan. Diversity-aware policy optimization for large language model reasoning. 2025.
15

---

[79] Jian Yao. Weiming Liu, Haobo Fu, Yaodong Yang, Stephen McAleer, Qiang Fu, and Wei Yang. Policy space diversity for non-transitive games. Proceedings of the 37th International Conference Neural Information Processing Systems, pages 67771 2023.
[80] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference Machine Learning. 2024.
[81] Yiqun Zhang, Peng Ye, Xiaocui Yang, Shi Feng, Shufei Zhang, Lei Bai, Wanli Ouyang, and Shuyue Hu. Nature-inspired population-based evolution of large language models. arXiv preprint arXiv:2503.01155, 2025.
[82] Xun Zhou, A. K. Qin, Maoguo Gong, and Kay Chen Tan. survey on evolutionary construction of deep neural networks. IEEE Transactions on Evolutionary Computation, 25(5):894-912, 2021.
[83] Yu Zhou, Lei Lei, Xiaohui Zhao, Lei You, Yaohua Sun, and Symeon Chatzinotas. Decomposi- tion and meta-drl based multi-objective optimization for asynchronous federated learning in 6g-satellite systems. IEEE Journal on Selected Areas in Communications, 125511115-1 2024.
[84] Yu Zhou, Xingyu Wu, Beicheng Huang, Jibin Wu, Liang Feng, and Kay Chen Tan. Causalbench: comprehensive benchmark for causal learning capability of large language models. arXiv preprint wwxxxv2244.063444 2024.
[85] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8697-8710, 2018
16

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in both abstract and Section accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
17

---

Justification:
Please check discussion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
The full set of assumptions and a complete (and correct) proof are detailed in Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the code for reproduction.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We specify them in our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
soreeeeeuubmsssioooooococ for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/publi///guu
CodeeSumiiii wettubbmmssiiioi ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
19

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We take some analysis.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The required computer resources are decided by the structure of the models be merged.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The required computer resources are decided by the structure of the models to be merged.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httt::///eerrrss.cc//pllle/ieeit
Answer: [Yes]
Justification:
The research is conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Not applicable to societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
21

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We cite the original papers or websites that produced the code package or dataset.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pww.::s...aaseiiiia...... thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
22

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA].
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanndaddrrd components.
Please refer to our LLM policy (https://neut) /cC//aareeeeccc//222//////////// for what should or should not be described.
23

---

Appendix of HM3
A Comprehensive Related Work about HM3
A.1 Model Merging
Model merge refers to combining the parameters and features of multiple large pretrained models to generate unified model that can perform better across multiple tasks. Existing model merging approaches rely task vectors constructed from fine-tuned models and their common base model. These approaches typically perform parameter-level interpolation (e.g., Task Arithmetic (27]. TIES merging (72), PCB Merging [19]. CAT Merging [54], and Consensus Merging (651) apply parameter manipulation strategies such as drop and rescale in DARE [80]. Formally, given a
base model with parameters O and K task-specife models fine-tuned from it with parameters
(0),
the model merging process can be expressed as [14]:
θ merge
G base: . θk)
(17)
where merge denotes the parameters of the merged model, and G(-) represents the merging method.
These methods have demonstrated significant improvements in the performance of merged models. In particular, recent works have introduced evolutionary search algorithms, such as evolutionary algorithms (EAs), to enhance model merging. For example, GENOME [8T] employs one of the classical EAS, named differential evolution, to evolve new models within shared architectural weight space through crossover, mutation, and selection operations. and further performs ensemble inference. Similarly, Evolver (0) directly applies EA to the weight spaces of multiple fine-tuned models, mutating and crossing their parameter vectors to select nigher-performing combinations, thereby achieving parameter fusion without gradient-based fine-tuning. Both methods emphasize low-cost and high-efficiency strategies that yield competitive performance across different scenarios. However, they primarily focus adjusting parameter configurations within fixed architecture. In practice, models with diverse architectures may exhibit stronger representation capacities and potentially extend performance beyond the limits of single-structure model under multi-task scenarios. Motivated by this, Akiba et al. [ recently explored the use of EA to search for optimal architectures in model merging. While promising, this approach faces scalability issues: as model size increases, the architecture search space becomes substantially more complex, often resulting in performance degradation. Furthermore, EA is population- and requires expensive evaluations in each iteration. They are also typically designed for one-shot merging, which means that the search must be restarted from scratch for every new task. This leads to prohibitively high computational costs. In this work, we unify the strengths of both parameter-leve and architecture-leve merging by designing an efficient joint framework. Importantly, we train reusable model that can generalize to new tasks without requiring full re-search from scratch.
Multi-Objective in Model Merging Existing methods typically rely the model designer's s domain knowledge or intuitive understanding to manually determine these weights, resulting in single trade-off solution for the merged model. However, task preferences may differ across users, or even for the same user at different times, thereby demanding merged models that reflect diverse preferences. Recent works [34 33) have begun to explore the flexibility of multi-objective optimization in assigning weights across tasks. Nonetheless, these efforts remain in their early stages and often fail to fully exploit the nature of multi-objective trade-offs, falling short of achieving high-quality Pareto-optimal merged models. In this paper, we design multi multi-obeective strategy to obtain approximate Pareto-optimal parameters and architectures that meet different preferences.
A.2
Multi-ohjective Optimization
A.2.1 Definition
Generally, a objeet-b optimization problem can be formulated as:
min (<x) = (())
8.l. x E
(18)
where x = (I1, T2,... ..) is decision vector, and f(-): x represents k objective functions. Here, X denotes the decision space, and Y denotes the objective space. To compare the quality of solutions obtained by the multi-objective problem, the concept of Pareto dominance is introduced.
24

---

Pareto dominance Given two solutions 21 and belonging to X, z1 is said to Pareto dominate I2 (denoted as 2 22) if and only the following two conditions are satisfied:
1. For all objectives E (1, K). fi(a1) - fi(22), meaning that æ is not worse than
22 every objective.
2. There exists least objective
m) such that fz < fzz)) indicating that 2 is strictly better than 22 in at least one objective.
solution x" X is considered Pareto optimal if no other solution X Pareto dominates x'. The set of all Pareto optimal solutions known as the Pareto set:
PS ( E X 1 x' E X, x
(19)
The collection of objective vectors corresponding to the Pareto set is referred to as the Pareto front. Multi-objective optimization aims to approximate the Pareto set by identifying solutions that achieve both strong convergence and a diverse spread within the objective space.
In multi-objective optimization methods, since the true Pareto optimal solution set is unknown, we employ the commonly used metric called hypervolume (HV) [56] to comprehensively assess the diversity and convergence of the generated approximate Pareto optimal solution set. Let point set P C I& and a reference point r E Rd, where d = 3 is the number of optimization objectives. The
HV of the set P is computed as follows:
HV(P,r)= Le (Ulalpsarr)
(20)
pEP
where (-(-) represents the Lebesgue measure of a set: Ce(S) Sses 1s(s) ds Here, 1s is the characteristic function of the objective space S. If E S, then (s(s) = 1; otherwise, 1s(s) = In the calculation of HV, the non-dominated solutions obtained by each algorithm the same reference set, and the reference point is typically set at (1.1, 1.1). It is important to note that larger HV indicates a better approximation of the Pareto optimal solution set and, consequently, improved performance of the corresponding multi-objective optimization method.
normalized using
As for multi-objective optimization in model merging, there are two early explorations. The first paper 34 introduced novel method called model merging with amortized Pareto fronts, which approximated evaluation metrics using a quadratic surrogate model derived from set of pre-selected scaling coefficients. However, while this approach primarily focuses on reducing computational complexity, it does not thoroughly explore how to accurately obtain the Pareto-optimal merged model. The second paper [33] employed parallel multi-objective Bayesian optimization to systematically explore the parameter space for optimal merging configurations. However, these works only in the early stages of exploration. They merely use multi-objective optimization methods to facilitate model merging, but do not fully consider the multi-objective and multi-task characteristies inherent in the models during the merging process.
B Detail of the proposed HM3
B.1 The Proof of Problem Transformation
To handle the K-dimensional vector-valued objective f = (fi...., fk), we adopt a standard linear scalarization approach. Specifically, for given task preference vector X A sampled from Dirichlet distribution, the scalarized objective is defined as:
K
A (x) = Σ (0,
(21)
k=1
We write F : Fx for brevity. Solving the Stackelberg game for each À produces of solutions that approximates the Pareto front. Assumption (Compactness and Continuity). and compact. The architecture space M is finite and contains least one feasible base path Olaase
1. The parameter space P C Rddo is nonempty
25

---

2. For any θ E P. the follower's feasible set
( = fa |a|| S T max: di, 4;; m,.i) 44: m det 86
(22)
is nonempty (since Olbase E ) and has a closed graph.
3. The scalarized utility F 0i) is jointly continuous in (x.
Proof. (a) Follower-level solution existence. Since M is finite, the constrained set !!@)) S EM is finite and nonempty for any fixed P. Hence, the follower-level optimization
( arg max F(O,a) aen(e)
(23)
admits at least one solution. Furthermore, by Berge Maximum Theorem [6. the best-response mapping a' (0(@) is upper hemicontinuous with compact (finite) values due the closed graph
property and continuity of F.
(ii) Continuity of the leader's objective. Define the upper-level objective:
F()): = max o(),a). aE((e))
(24)
Because a (@) is upper hemicontinuous and F is continuous, it follows from 2 that F is continuous on the compact domain P. Therefore, by Weierstrass' Theorem, there exists maximizer O arg maxee F(O).
(iii) Equilibrium construction. Select any a* E a²(@"). Then the pair (0", a²) satisfies the definition of a Stackelberg equilibrium [31 Def. 2.2], and achieves the same optimal value as the original joint objective.
Remark. Assumption (A2) is typically ensured in practice by guaranteeing at least one dimension- compatible base path (e.g., through 1x1 projections when needed). Each choice of À induces scalarized subproblem, and the collection of corresponding equilibria approximates the Pareto front.
B.2 The Proof of Wolpertinger Policy in Actor-Critic Framework
Let the discrete candidate action set generated by the Wolpertinger policy be denoted W {A1, A2,... AM }. which contains the M nearest neighbors in Fuclidean distance of the continuous proto-action E Rd, Among them, define Ag- to be the nearest discrete action to Å, i.e., the one selected by the simple projection method:
Ar- = arg min AEW-
||A-AA ||l2
(25)
Assume the action-value function Q(S,A) under fixed state S satisfies the following statistical assumptions:
For all é + E", the values At) are i.i.d. samples from a uniform distribution:
Q(S, U(Q(S, E, Q(S)Ae)) + E),
(26)
where & > O is fixed constant.
The value of the nearest action Ag is set as the reference:
Ar-) go.
(27)
Let A be the action selected by the Wolpertinger strategy, i.e., the one in W with the maximum Q-value:
A = arg max Q(S, AEW-
(28)
Then, the expected Q-value of the selected action is:
2(2M -1)
B[Q(S, [b) = go +
(29)
M 2M
26

---

Let X1. XI........ ........,..... XM-- denote the i.i.d. uniform random variables representing the Q-values of the
other M candidates:
X; ul9 9o 5),
== M - 1.
(30)
Let Xaxx [nax(X1,.... J- Then the probability density function (PDF) of Xmax is:
1
(90 {)
M-2
fx (x) (M
x
1g0 - 5. q0 + 5].
(31)
2
25
The expected maximum is:
g0
(x) go + E. (1 2(2M - -1) M -.
B max
x
(32)
90-E
Note that if max > 90, then the maximum selected action A will not be Ar,, but one of the other candidates. If all X; yo. then Ag- is still chosen. Therefore, the expected Q-value of the Wolpertinger-selecter action
E|Q(S, A.)] Egmax(go- Xi...., XM- - H] Elmax X.ax]]
(33)
By integrating over the support and using order statistics of uniform distributions, the final result is:
E[Q(S, + (1 M 2
2(2M -1)
(34)
The proxy estimation error is bounded by:
A) Q(( S
AAEW".
(35)
Then the expected Q-value of the Wolpertinger-selectt action satisfies:
2(2M - 1)
M 2 )) 8.
E [[Q(S,A Q(S, (5,)
(36)
When M > and 8 <1(1=
2(2M-1) M-2M
we obtain:
EE [Q(S,AL))) Q(S,A,),
(37)
which confirms the superiority of the Wolpertinger policy even in the presence of bounded proxy approximation error.
B.3
Algorithm Description
Based on the MDP, the execution of the RL is the following stages: Stage 1: Input and initializa- tion: The algorithm begins by sampling preference vectors, each corresponding decomposed subproblem in the multi-objective framework. For each preference vector, an existing parameter-level merging method is applied to obtain initial merged model parameter. Meanwhile, the parameters of the policy, value, and the MI.P network for alignment are initialized. Stage 2: Trajectory collection and MLP alignment: For each preference vector, an inner loop executed RL. each iteration, the parameter-less and stored in the trajectory buffer. Then, each step, the current trajectory-encodee state is used to generate proto-action, which is discretized via the Wolpertinger policy to select the next action. The new state is converted, and the transition is recorded in the buffer. Once complete path collected, the algorithm performs MLP alignment. Stage 3: Reward computation and network update: After alignment, the reward is computed and then is assigned all time steps in the trajectory. Once iteration number exceeds, the algorithm enters the network update phase. In this phase, a mini-batch sampled from the buffer to compute GAE and target returns. The policy, value network, and ML. alignment network updated. The entire process continues until M aatter is reached, yiclding the
merged model and the fine-tuned models are evaluated to compute the reward
27

---

Algorithm HM3 in the architecture space
1: Input: A set of preference vectors (A² A², ................................ AND and their corresponding optimal merged models at the parameter space.
2: for each preference vector A in {X² X² ..... AN do
3: Input A fine-tuned models and the optimal merged model in the parameter space corresponding AІ-
4: Initialize the parameters of policy network as No. of value network as Φu, of MLP network as
MLP pu.
5:
for each iteration iter = 1, 2....... Mar iter do Sample the current policy Tpter (Ar(Se) by interacting with the environment to generate a trajectory of length T as {St. Ae, Re, S
6:
7:
Obtain the state S: = (m4,4)). Select the action A, = (m++1.44+1)
8:
9:
Calculate the reward RG based on At and the merged model in the t-th step.
10:
Compute the advantage function Å, and G..
11:
Update the policy network by
maximizing
LCLIF(U)
E, min (p:(n)A,, clip)) c,1 e.
Update the value network by minimizing I. (6)) = EE (VV(()))) }.)
13:
Compute the scaling matrix Wm,,, and update the MLP network parameters MLP,,.
14:
end for
15:
end for
16:
Output: Optimal policy network parameterized μ² and the optimal inference path (i.e., the
optimal sequence of actions) corresponding to the value network.
final policy network and the optimal inference paths. The well-trained networks can be reused for new tasks or incorporated into model pools.
The overall algorithm of HM3 is summarized as Algorithm[] [ It begins by taking in collection of preference vectors and the associated best-merged models from the parameter space (Line 1). Each preference vector introduces the fine-tuned models and the corresponding best-merged model (Line 3). The initial parameters for the policy, value network, and MLP network are set up (Line 4). During each loop, the policy network interacts with the environment, creating trajectory composed of state, action, and reward information (Line 6). The state the current step and the chosen action are determined next (Lines 7-8), Subsequently, the reward for the current action is calculated based on the state within the merged model (L.ine 9). These rewards are then utilized to compute the advantage and target values (Line 10). The algorithm adjusts the policy network by enhancing the policy loss and refines the value network by minimizing the value loss (Lines 11-12). The process also involves calculating the scaling matrix and fine-tuning the MLP network through PPO (Line 13). Finally, it outputs the final set of optimized policy parameters and the sequence actions that represent the optimal inference path tied to the value network (Line 16).
After obtaining the Pareto-optimal models, HM3 assumes users typically do not provide explicit preferences and supports two practical modes: (i) Offline preference sampling: We uniformly sample preference vectors to approximate the Pareto front. Users can later select model matching their needs (no input required). (ii) Optional user preference injection: If a user specifies a preference (e.g., prioritizing translation), we select the closest model on the front or conduct a targeted search, enabling both automated and interactive use.
B.4 Detail Analysis of Dimension Alignment via Statistical Matching
To accommodate distributional shifts across layers from different models, we introduce feed-forwaat MLP network that generates a scaling matrix Wm,-- The imput to the MLP consists of the layer index pair (m. 1) and the current time step, and its output defined as:
W I (L,,,m,
(38)
where MLP,, is parameterized by μ and optimized via actor-critic method.
28

---

This design is motivated by the theory of statistical moment alignment. Suppose the hidden represen- tations from the source and target layers follow Gaussian distributions:
Zare
(Hsre Esre),
Zigt ~ N (Prgt Eige).
(39)
The squared 2- 2-Wasserstein distance between them is:
W
|| Mygil Tr EEsre + Engi 2(Eig) Ex 2 Engl)
(40)
and the optimal affine transformation (zz) = Aupt + given by:
- Aopt Sig
25r b= Hegt
Apptlasc
(41)
This whitening followed by coloring transformation achieves exact alignment of first- and second- order statistics. To enable end-to-end learning, we approximate this process using the deep CORAL loss 553
DDep-CORAL = |IC(H'))) - C(Hgg)
(42)
where H' = Wm,, ((Hare (ssce) 2 ore)) + bm. and C(-) denotes the empirical covariance matrix. If mean alignment is also desired, we add mean alignment loss:
E ||Ma
Mgull2
(43)
The final training objective for the MLP alignment layer becomes:
Zerc
Mue
W m.
min DDesp-OORAL ALu.
(44)
V /daag
W 1mm.t.m..
This allows the MLP to approximate the theoretically optimal mapping (Aopts b) via gradient descent, thereby providing robust statistical alignment for seamless composition of heterogeneous transformer layers.
C
Additional Results
C.1 Detail of Experimental Setup
The core objective of this study is to design and implement an efficient and multitask- model merging framework. To verify the generality and performance of the proposed merging method, we apply it to three popular series LLMs, namely the fine-tuned models based on Qwen- 1.5B
[59], Llama-
[60] and Llama-2- 2-11B Specifically, for Llama-2-7B [60]. include three
fine-tuned models: -lama-7B-Chan 0 for text translation [60]. WizardMath- 7 for mathematical
reasoning 44). and CodeLlama-77 7B for code generation 47 Similarly, for Awen-2.5-1.5B (59).
we include three fine-tuned models: 1-enn-2.5-1.5B-Inniiie
for text translation (59]. Qwen-2.5
Code-1.501 10 for code generation (26). and D0wen-2.5-Math-1.551
for mathematical reasoning (76
As for Llama-2-13BB we include three fine-tuned models: WizardL.M-133 12 for text translation
711, WizardMath-133 133 for mathematical reasoning [40), and WizardCooer-Pythonn-Iee 14
for code
generation 42)
-llama-BBCChtt
https//huggnngffcc.com http:/agi.naaaa..eeeeeeeeeeee//s
'httss//hugggingface www.eetacccaaaaeeaaaaeaaeaaaa Ilama/Llama-2770 2-7hhh
bhttss//hugggnngface.oo
wo/mmeta-lammmLiinneee llama/Llama-2-1 1--b-hhh
"nttss:/hugggingface.oo///
http://.....c..cttaaaaaaaaaaaaaa co/meta Baae 1lama/Llama-277-chhttt
http://inggnggaacccooaalllllllla "
13B-V V1.0
29

---

WizardMath-7B:
CodeLlama-7B: http://ungggigzecccooooooo/ooo/m
LLama-2-77: http:://uugginnfaaccccoocttlllll
By leveraging these fine-tuned LLMs, we can utilize our proposed HM3 merge LLMs across multiple tasks. To evaluate the performance of the merged LLMs by HM3, we perform three tasks, including language translation, mathematical reasoning, and code generation. To achieve these evaluations quickly and efficiently, we employed two popular large model evaluation packages: Im- evaluation-harnes (1 for text translation and mathematical reasoning tasks and big code-evaluation- harness for code generation tasks. These evaluation packages can be found at the following link:
Im- mm--evalation--hame harness: http:///wwggghooooo//////o/eoeee
biggode-evaluation-narees
To further demonstrate the effectiveness and superiority of our method compared to other model merging methods, we utilized the mergekit package [22] to merge models by using several merging methods, including Task Arithmetic, Ties, DARE-Ties and EA [ The mergekit package can be found at the following link:
mergekit: https///gthhbb.coo//rree//////ee
Additionally, for HM3, Ma is 1000 and the discount factor is configured to 0.990. We split the dataset where 70% used for RL inference evaluation, while the 30% reserved for the evaluation of the obtained merged model. Then, we introduce the specific datasets for generative, text translation, math reasoning, and code generation tasks, as well as their metrics.
C.1.1 Generative Tasks
We use GLUE benchmark 63 a widely used benchmark for natural language understanding comprising nine sentence/sentence-- classification tasks (i.e., CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE, WNLI). These tasks, drawn from established datasets, cover a range of sizes, genres, and difficulties, offering a broad and challenging evaluation of language understanding.
C.1.2 Text Translation Tasks
To evaluate the multilingual translation capabilities of LLMs, we leveraged set translation tasks in the Im-evaluation-harnes package, including WMT14 WMT1 g6 [50), and IWSLT2017 (7). and Xnli [15. These tasks evaluate the model's translation accuracy and fluency across diverse language pairs. For the first translation tasks, based on character n- gram precision and recall. For the final task, we use accuracy as the evaluation metric instead.
the "chrf" metric, which measures translation quality
C.1.3 Math Reasoning Task
In this paper, we use MathQA (3) and GSMSK [12 for the evaluation of the math reasoning capability of the obtained models. MathQA 3 is a large-scale benchmark of roughly 37K English multiple- choice math word problems covering broad range of mathematical topics. It also provides operation programs aligned with problems from the AQuA dataset. Model performance on MathQA is reported as accuracy. GSM8K [12] is a dataset meticulously designed for mathematical problem-solve comprising over 8,000 high-quality problems that span from basic arithmetic to complex algebra. The primary objective of this dataset to evaluate the model's reasoning and computational abilities when tackling structured mathematical problems. For evaluating the GSM8K dataset, we employ the "flexible match" metric, which allows for minor variations the final answer.
tasks,
"https://wwwsttammte orgg//mt http:///w/t/////aaaaaaaaaaaa 4/translation-tase html 1 "https//wwww www.ttttttttommmmmm wtt.stttttt///mmmtttt. wrt/wmt w task
30

---

Table 5: Comparison of merging methods for Llama-22 1- 1-B series LLMs on language tasks
General
Translation
Math
Code
Merging Methods
Glue
WMT&ISWT
Xnli
GSMSK
MathQA
HumanEval
MBPP
Fine-tuned Model Chat
67.05
43.29
47.11
33.82
22.67
21.32
22.56
Fine-tuned Model Math
45.08
27.02
41.23
55.74
31.33
19.21
20.83
Fine-tuned Model Code
19.27
32.49
42.61
14.75
27.09
51.82
29.63
Task Arithmetic
35.24
29.16
37.33
27.74
21.47
28.52
25.80
Ties Merging
37.67
33.23
39.11
28.02
21.53
28.74
31.40
DARE Ties Merging
48.17
33.56
40:40
38.83
23.81
30.18
24.43
Consensus Merging
45.15
32.54
43.03
36.24
24.21
38.82
25.03
PCB Merging
52.77
41.96
40.90
46.34
24.56
40.15
26.57
EA HM3
15.92
25.06
28.23
17.36
19.29
13.31
14.48
53.20
41.12
45.92
46.82
29.13
43.93
33.29
C.1.4 Code Generation Task
In the code generation domain, we evaluate on MBPP and HvmanEval. MBPP contains 974 beginner-level tasks targeting short Python program synthesis from natural-language prompts; per- formance measured with pass@1 HumanEval [9] is benchmark dataset proposed by OpenAI, specifically designed to evaluate code generation capabilities. The dataset comprises 164 program- ming problems, where each problem requires the model to generate Python function based on natural language description. The evaluation metric of HumanEval is pass @ 100. The model is allowed to generate up to 100 code solutions for each problem. This metric assesses whether least one of these generated solutions passes all test cases.
C.2 Detail of Main Results
Discussion An interesting observation from Table I and Table 2 that the models produced by HMB sometimes outperform closely match task task-specific fine-tuned models, which are typically considered performance upper bounds for their respective tasks. We attribute this to a key distinction: while fine-tuned models adapt only at the parameter level, HM3 performs both parameter-level and architecture- merging. As discussed in the introduction, models with different architectures exhibit diverse representational capacities and task preferences, which extend the performance boundary beyond that of single fixed architecture. For example, if the layers of the merged model increase, then according the scaling law, the merged model is expected have higher theoretical performance ceiling.
Merging LLAMA -2-13B-BBBBB model Table presents the performance comparison of various merging methods on the LLaMA-15B series across general, translation, math, and code tasks. The proposed HM3 framework achieves consistent and significant improvements parameter-level and architecture-level baselines. HM3 attains the highest average performance across all task categories, particularly excelling GSMSK, Xnli, and HumanEval.
existing classical and SOTA
Merging ViT-B/32 model As shown in Table the HM3 method consistently outperforms other approaches on ViT-B/32, achieving average accuracy of 66.91% 1,,, which represents significant improvement. Specifically, HM3 achieved 77.21% 1 and 77.62% on the EuroSAT and SVHN datasets, respectively, and recorded 68.21% 12.1% % accuracy on the GTSRB dataset, surpassing other methods. Although its performance on the DTD dataset is slightly lower than that of the other tasks, still outperforms the Ties and Task Arithmetic methods.
Merging ViT-L/14 model Table 4 summarizes the performance of different merging methods on the ViT-L/14 model across various vision tasks. The results indicate that the HM3 method consistently achieved best performance across most tasks, with particularly high accuracy on the SVHN and GTSRB datasets, reaching 90.48% and 83.43%, respectively. Notably, HM3 achieved an overall average accuracy of 80.30% across all datasets, significantly outperforming the other merging methods.
31

---

HBM
7B-Cah-
WicardWaath118 75
Cookl 12mo-78
Anthimetc
Tiee
20
0
60
40
10
Math
Transiation
Figure 4: The illustration of different model merging methods in the text translation, math reasoning. and code generation tasks.
C,3 Detail Analysis of Multi-Objective Model Merging
HM3 is capable of generating a of approximately Pareto-optimal merged models, which enables adaptation to different user preferences. Unlike other merging methods that produce only a single solution, HM3 yields a diverse set of candidate models. To compare their quality fairly, we compute the Pareto dominance relations by pooling all solutions from different methods together. A solution Ta is said to be dominated by another solution 26 if I6 is no worse in all objectives and strictly better in at least one. The detailed computation procedure is provided in Appendix A.2. Based on this analysis, we evaluate the dominance relations across all solution sets, as shown Figure 2 Yellow cells indicate that the solution on the vertical axis is Pareto dominated by the one on the horizontal axis, while blue cells denote no dominance. The results show that every competing method is dominated by at least one solution from the HM3 solution set (i.c., S1-S15). To compare with EA, we extend EA to a multi-objective version (MOEA) as a baseline. Since MOEAs also produce a set of solutions, we employ the hypervolume (HV) metric [25 [56]. which measures both convergence and diversity. higher HV value indicates a better overall solution set. HM3 achieves an HV of 1. 1.6824, significantly outperforming the MOEA baseline, which obtains an HV of only 1.1329. This gap highlights the inefficiency of unguided EA-based methods in complex multi multi-objective spaces.
C.4 Effect of Different Number of Tasks
In this subsection, we demonstrate the effectiveness of HM3 across different numbers of tasks. In the main text, we illustrated the effectiveness of HM3 on three tasks, and the illustration of metrics for different merging methods is shown Figure. 4 From it, our approach was capable of producing a set of Pareto-optimal merged models, along with their corresponding metrics, which provided valuable guidance for users to personalize their selection based on the specific needs of their tasks. In contrast, other methods were limited to generating only single solution.
We provide evidence of HM3's effectiveness on two tasks: code generation and mathematical reasoning. The experimental results are presented in Figure 5 (Figure 3 in the main manuscript), which clearly demonstrate the significant advantages of HM3. Specifically, HM3 is capable of generating Pareto optimal set of solutions that excel not only in parameter optimization but also in architectural configurations. The blue circles in the figure represent HM3, showing that its solutions are well-distributed across the entire performance curve. Based on the available data, we used convex hull software and Gaussian process fitting to approximate the Pareto front. The results indicate that the solutions produced by HM3 closely approximate a comprehensive Pareto front, effectively capturing the optimal trade-off's under varying conditions. In contrast, other methods, such as Task Arithmetic, Ties, and DARE Ties, are restricted to generating single optimal solution exclusively in the parameter space, as indicated by the red squares, yellow stars, and green diamonds, respectively. It is evident that the solution sets produced by these methods are confined to narrower regions of the performance spectrum, lacking the diversity and flexibility that HM3 provides. Moreover, these single solutions are noticeably farther from the approximated Pareto front.
32

---

25
BPQ
Tiee
Dore Thee
10
Math
Figure 5: The illustration of different model merging methods in the math reasoning and code generation tasks.
Table 6: Performance of HM3 in different spaces
Single Objective
Multi-Objeetive
Instance
Translation
Math
Code
HV
HM3 w.o. para, opt.
32.21
18.36
20.67
1.3506
HM3 w.o. archi. opt. HM3
34.01
38.51
28.67
1.6387
44.68
45.62
43.62
1.8120
C.5 Detailed Analysis of Ablation Study
To assess the effectiveness of HM3 jointly optimizing both the parameter and architecture spaces, conduct two ablation experiments comparing the following three variants: (i) HM3, (ii) HM3 w.o. arch (without architecture-level optimization), and (iii) HM3 w.o. para (without parameter- evaluate performance under a single-objective setting using a sampled preference vector. HM3 consistently outperforms the two ablated versions across all tasks, with especially notable gains in code generation. This suggests that the joint optimization of both spaces yields significant synergistic benefits and that architecture-leve adaptation plays crucial role in tasks with more structural complexity. The second experiment examines the HV of these methods in multi-objective setting, HM3 achieves the highest HV score, followed by HM3 results highlight the importance of parameter optimization in ensuring competitive solutions across objectives, while also demonstrating that architecture optimization further enhances the solution performance.
level optimization). The results summarized in Table 6 In the first experiment,
arch, while HM3 para performs the worst. These
C.6 Computational Cost Analysis
Compared the conventional pretraining and fine-tune fine-tuning paradigm, HM3 requires significantly less computational power and time to achieve a high- regh-performacc model with a novel architecture. Moreover, it eliminates the need for high-quality data for pretraining or fine-tuning. Specifically, the network scale of policy and value networks with 200MB parameters in HM3 is much smaller than the I.IM with 13.5GB parameters (i.e., Llama-2-7B) in the fine-tuning stage. In this manner, the computational power required for RI. training is significantly lower than for fine-tuning. In this paper, we used A6000 or 3090 GPUs for RL training of HM3, whereas full fine-tuning typically requires A A100-level GPU clusters. Additionally, the overall computation time of HM3 is significantly lower than that of fine-tuning. During RL training, only inference (i.e., forward propagation) is required, whereas full fine-tuning necessitates both forward and backward propagation of the LLM. According to time complexity analysis, the backward propagation is typically several times more consuming than the forward. Therefore, the time required for the full fine-tuning is several times greater than that for RL on the same dataset. Finally, we compare the performance of HM3 and the full fine-tuning at similar time cost, and the results are provided in Table. 7 We can observe that HM3 still obtains the best performance compared with traditional fine-tuning methods similar time cost. Given preference vector, compared HM3 not only against traditional efficiency basclines such as Full
33

---

Table Comparison between HM3 and full fine-tuning methods
Method
Translation
Math
Code
LLAMA-2-7B-CHM
42.23
24.15
21.66
Fine- Tuning Method
WizardMath-7B
37.72
45.60
22.74
Codel.Lamaa
36.12
18.37
43.11
Grid Search
29.31
21.21
23.12
Evolutionary Search
31.92
23.49
26.90
HM3
44.68
45.62
43.62
20
18
16
Mean Reward
12
Meaan Reworrd
1000
Episode
Figure 6: The convergence of RL in the HM3 at the architecture space.
Fine-tuning, but also introduced two additional search-based architecture merging baselines for more comprehensive efficiency comparison: (i) Grid Search: Under the same maximum inference path length Tmax as HM3, this method exhaustively enumerates all possible path combinations until reaching approximately the same number of evaluations HM3. Layer-wise composition is performed using the passthrough strategy implemented in Mergekit, (ii) Evolutionary Search: The individual encoding shares the same maximum path length HM3. The search operators follow the standard Differential Evolution algorithm with default hyperparameters and population size. The number of evaluations is aligned with that of HM3. Layers are composed using Mergekit's passthrough method. This approach differs from the EA baseline in the original manuscript, which restricts the search space by controlling model sequences via tokens-a design that significantly reduces complexity but at the cost of final merged model performance. The final performance comparison is summarized in the table. These results demonstrate that, under the same computational budget, HM3 achieves superior multi-task performance compared to other architecture merging methods based on search algorithms.
C.7 Effectiveness RL
In this subsection, we delve into the convergence of HM3. Specifically, we randomly sample preference vector and observe the obtained reward when merging models on text translation. mathematical reasoning, and code generation tasks. The experimental results are illustrated in Figure 6 As shown in Figure the overall reward increases progressively the number of training episodes increases. During the first 200 episodes, the growth in reward was relatively slow, which is attributed PPO's exploration phase, where HM3 had not yet accumulated sufficient experience and the policy network had not been trained. However, after episode 200, with the introduction of the experience replay mechanism, the reward begins to rise significantly, indicating that the algorithm gradually learning from past experiences and improving its policy. As training continues, the reward shows more stable upward trend and eventually converges to value close 18 around the 1000th episode. HM3 can effectively leverage past experiences optimize its policy and achieve convergence.
34

---

D Discussions and Limitations
In this section, we discuss some concerns about this work from the multi-objective, architecture-level merging, and future work perspectives.
As for the multi-objective perspective, set the number of objectives to three, namely translation, math, and code. These objectives are intentionally chosen for their conflicts and diversity, which help produce well-separated and sparse Pareto front in multi-objective optimization. Adding more objectíves that do not bring sufficiently greater diversity, such as general language understanding objectíve, would shift the setting into the many-objective regime. That shift increases front dimen- sionality, densifies the set of solutions, and weakens dominance relations, all of which are known to degrade the effectiveness of standard multiobjective methods. For these reasons, do not expand the objective set here. If expansion is necessary, pair it with objective selection and decorrelation, decomposition, or reference- vector methods, indicator-based selection, and dimensionality reduction. Future work focuses on many- many-objectivv algorithms and nigh-dimensional discrimination, as well as theoretical foundations for cross- crosstarchteectuee merging and computationally efficient evaluation.
From the architecture-Ievel merging perspective, HM3 is search-based and achieves high efficiency, though search itself can be unstable, sometimes falling into local optima producing large variance. Compared with parameter-level methods, architecture-level merging typically incurs a higher time cost, a common limitation of current techniques. HM3 performs strongly when merging models that share the same base architecture but target different tasks, extending the performance boundary beyond parameter merging restricted identical structures. This work is an initial step toward architecture- level merging, focusing on handling post-merge structural differences. This idea parallels SOLAR 10.7B, which concatenates the first twenty and last twenty layers of Mixtral- with continued pretraining, while we pursue similar architecture expansion via training free model merging. When heterogeneous architectures are placed in single candidate pool, for example, by concatenating layers from Qwen and LLaMA, it becomes nearly impossible to merge them into an effective model. Additionally, existing alignment partially addresses low-order statisties, and deeper semantic shifts remain. Without alignment under cross-architecture settings, the merged model fails on the target tasks. In summary, merging fully heterogeneous architectures a promising yet unsolved endeavor.
In the future, we plan to focus on several theoretical directions for architecture-level model merging: (i) investigating nonlinear or piecewise mode connectivity under structural variations reveal the reachability and transition paths in parameter space; (ii) analyzing how mechanisms such as glue layers guarantee intermediate representation consistency, based on representation alignment and in- formation bottleneck theories; and (iii) quantifying the effects of structural merging on generalization error and model expressiveness.
35


Reviews:
[{"review_id": "M4DFD4Hq3u", "reviewer": "Reviewer_B6zw", "summary": "The paper proposes HM3, a method for heterogeneous model merging that extends to both parameters and architecture. It is framed as a hierarchical multi-objective optimization problem.\n\nSpecifically, the framework uses a two-level optimization structure to decouple the complex joint search space:\n\n* Parameter Level: It uses existing parameter merging techniques (e.g., DARE-TIES) to efficiently generate a parameter-optimal merged model according to different task preferences, which are defined by multi-objective weight vectors.\n\n* Architecture Level: It models architecture search as a Markov Decision Process (MDP). An Actor-Critic reinforcement learning policy is then designed to search for the optimal inference path (i.e., model architecture) within a \"layer pool\" composed of layers from the original models.", "questions": "In RL, how is the reward calculated for an incomplete network?\n\nThe MLP aligns the layer dimensions. What are the specific implementation details for the layer selection?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n* The article is well-written.\n* It generates a Pareto front, giving users the ability to select the most suitable model based on specific task requirements.\n* It utilizes an Actor-Critic framework, the Wolpertinger policy, and reinforcement learning to solve complex path-search problems, which is more guided and efficient than traditional EA.\n\nWeaknesses\n\n* More details on the reinforcement learning aspect are desired.\n* The outer-loop optimization in the paper requires the same parameter initialization, which is still confined to the same architectural limitations. Therefore, we cannot refer to it as a hierarchical model fusion.\n* The bi-level optimization in the paper is more analogous to alternating optimization (as both levels are overly independent).", "was_revised": true, "final_justification": "* The authors successfully clarified that their approach is a novel architecture-level merging, which is fundamentally different from traditional parameter-level merging. This resolved the reviewer's primary confusion about the fusion process.\n\n* They effectively addressed concerns about the introduced MLP layers by explaining that their training is a lightweight process confined to the architecture search phase and that the final merged model requires no subsequent fine-tuning, making it efficient and readily usable.\n\n* The authors' firm commitment to publicly releasing the complete source code upon acceptance assured the reviewer of the work's reproducibility and potential impact on the community.", "rebuttal": "**W1/Q1:** We thank the reviewer for the suggestion regarding the reinforcement learning details. In the revised manuscript, we have provided the clarifications from the method and experiment aspects:\n\n**Method Aspect**: We first clarify the components of RL. In this work, the architecture-level model merging problem is formulated as an MDP:\n\n1)State Space: We define the state as the trajectory of selected models and layers. Then, the model index, layer index, and layer parameters are embedded by learnable encoders, and map the trajectory into a state vector via a GRU.\n\n2）Action Space: The action is defined as selecting the next discrete pair of model index and layer index from the candidate model pool. As a discrete action space, we design a **Wolpertinger discretization strategy** to enhance the efficiency of discrete sampling and assist in selecting the next action. \n\n3）Reward Function: To encourage efficient inference paths while ensuring performance, the total reward of a trajectory is defined as a weighted performance minus a penalty for the path length.\n\n**Q1**: The reward function is computed after the entire inference path (trajectory) is generated and the MLP-based alignment is performed. The resulting model is evaluated, and the obtained reward is uniformly assigned to all time steps in the trajectory as $R_t$, which is used to train both the policy and value networks. In the original version, the description of the reward function may cause a misunderstanding. \n\nTherefore, in the revised manuscript, we updated the reward formula as:\\\\[R = \\sum_{k=1}^K \\lambda_k^{(i)} f_k(\\boldsymbol{\\theta},~\\pmb{h}) - \\beta_1 T\\\\] to indicate that reward obtained by model evaluation after obtaining the complete inference path and performing MLP alignment, thus ensuring the rigor and correctness of reward calculation.  \n\nBased on the above MDP, the execution of the RL is the following stages:\n\n**Stage 1: Input and initialization**: The algorithm begins by sampling $N$ preference vectors, each corresponding to a decomposed subproblem in the multi-objective framework. For each preference vector, an existing parameter-level merging method is applied to obtain an initial merged model parameter. Meanwhile, the parameters of the policy, value, and the MLP network for alignment are initialized. \n\n**Stage 2: Trajectory collection and MLP alignment:** For each preference vector, an inner loop is executed in RL. In each iteration, the parameter-level merged model and the fine-tuned models are evaluated to compute the reward and stored in the trajectory buffer. Then, at each step, the current trajectory-encoded state is used to generate a proto-action, which is discretized via the Wolpertinger policy to select the next action. The new state is converted, and the transition is recorded in the buffer. Once a complete path is collected, the algorithm performs MLP alignment.\n\n**Stage 3: Reward computation and network update：** After the alignment, the reward is computed and then is assigned to all time steps in the trajectory. Once the iteration number exceeds $iter_0$, the algorithm enters the network update phase. In this phase, a mini-batch is  sampled from the buffer to compute GAE and target returns. The policy, value network, and MLP alignment network are updated. The entire process continues until *Max_iter* is reached, yielding the final policy network and the optimal inference paths. The well-trained networks can be reused for new tasks or model pools.\n\n**Experiment Aspect: **The hyperparameters of RL are: *Max_iter* is 1000; the networks begin updating after $iter_0 = 200$; the PPO clipping ratio is 0.1;  $\\gamma$ is 0.990; the GAE smoothing factor $\\beta_A$ is 0.95. The loss coefficients are $c_1 = 1.0$ and $c_2 = 0.15$.\nWe split the dataset where 70% is used for RL inference evaluation, while the 30% is reserved for the evaluation of the obtained merged model.\nFinally, in the revised version, we conduct 10 independent runs with different random seeds and report the mean and variance to assess the stability of the RL policy training. The detailed result of RL convergence is provided in **W3 of Reviewer wkP4**.\n\n**W2:** We sincerely thank the reviewer for the valuable feedback and fully understand the underlying concern. While it is true that parameter merging generally requires the *same parameter initialization*, this condition does not apply to the outer-loop optimization in HM3.\n\nTo elaborate, in HM3, the outer-loop optimization selects a source model for each layer based on $(m_t, l_t)$, and parameter merging is only performed among models in the pool that share the same architecture as the selected source model. If the selected layer originates from a model that has no architectural counterparts in the pool, parameter merging is simply skipped. Therefore, the outer-loop optimization itself does not rely on a unified parameter initialization across models.\n\nOur approach indeed involves optimization over both model parameters and model architecture, reflecting a two-level decision process. To clarify this point, we have revised the manuscript to provide a more direct explanation of HM3 as follows:\n\n> The lower-level optimization searches for the optimal merged model architecture. The resulting architecture determines the length of the inference path (i.e., the number of layers to be merged). The upper-level optimization then operates on the parameter set $\\\\{\\theta_{m_t,l_t}\\\\}_{t=1}^T$ corresponding to this architecture.\n>\n> Consequently, the optimal architecture found by the lower level dynamically determines the dimensionality and scale of the parameter search space for the upper level. This naturally forms a hierarchical decision-making structure—first optimizing the model architecture, then optimizing the corresponding model parameters—which embodies the core *hierarchical* nature of our HM3 method.\n\nWe hope this clarification resolves the misunderstanding and highlights the distinct roles of architecture and parameter optimization.\n\n**W3:** We greatly appreciate the reviewer’s insightful concern. We understand that the interaction between the upper- and lower-level appears to resemble an alternating optimization. However, we would like to emphasize that HM3 is fundamentally a bilevel optimization rather than an alternating optimization procedure. This distinction is evident in the following aspects:\n\n(i) Different objective functions across levels: The upper-level optimization maximizes the performance $\\mathcal{F}$ of the merged model, while the lower-level optimization is formulated as an MDP with a reward function that combines model performance and inference path length: $R = \\mathcal{F} - \\beta_1 T$. The two levels optimize different functions characterized as bilevel optimization, while alternating optimization decomposes a single objective into interleaved subproblems.\n\n(ii) Strict dependency between levels: the upper-level optimization in HM3 can only be performed *after* the lower-level has searched an optimal architecture. This requirement of solving the lower-level problem to optimality before proceeding to the upper-level is a key feature of bilevel optimization.\n\n(iii) Variable interaction: Interaction variables such as the inference path length $T$ are determined exclusively by the lower-level architecture search and are then passed to the upper-level optimizer.\n\nTo avoid confusion, we have revised the manuscript to update $\\mathcal{P}2$, making clear the essential distinction between the objectives of the two levels and their interaction structure.\n\n**Q2**: We begin with sampling preference vectors. For each preference vector, the layer selection process is modeled as an MDP, as detailed in W1/Q1. Specifically, the trajectory of selected layers is embedded into a state vector via a learnable encoder. Based on this state, the policy network outputs a continuous proto-action vector, which is mapped to a discrete action using the Wolpertinger policy: it first identifies the nearest neighbors based on the distance to the proto-action vector, constructs a candidate set, and then selects the action that maximizes the value network or performs random exploration. The final action determines the next layer to be appended to the inference path.\n\nThis process yields an inference sequence. For every adjacent layer in this sequence, we apply an MLP-based alignment to address potential mismatches in dimension and representation. Specifically, we first detect their input/output dimensions. Then, for each layer, we compute statistical descriptors of its representation, i.e., mean, variance, and distribution characteristics. The proposed MLP network generates a projection matrix that transforms the representation of the preceding layer to align it with the following one in terms of both dimensionality and distribution. The transformed features are then seamlessly passed to the subsequent layer, enabling compatible inter-layer connections across different models. Preliminary experiments indicate that incorporating MLP-based alignment leads to improvements in final performance.\n\n| LLAMA-2-7B                | Translation | Math      | Code      |\n| ------------------------- | ----------- | --------- | --------- |\n| HM3 without MLP Alignment | 38.15       | 39.83     | 38.21     |\n| **HM3**                   | **44.68**   | **45.62** | **43.62** |\n\nMLP-based alignment is applied between adjacent layers in the inference path. As in **W5 of Reviewer wkP4**, learning-based layer stitching via MLP projection remains underexplored. Therefore, in the revised version, we state that the proposed MLP-based projection is in its early exploratory stage. In the *Discussion* section, we outline possible future directions."}, {"review_id": "nTaRpDB9WE", "reviewer": "Reviewer_21Mn", "summary": "This paper proposes HM3, a novel hierarchical framework for merging multiple pretrained and fine-tuned models into a single model. The method treats model merging as a multi-objective optimization task and uses reinforcement learning to explore both parameter and architecture space. The result is a Pareto front of merged models, enabling flexible trade-offs among task-specific performance objectives. Experimental evaluations on several language tasks and vision tasks demonstrate promising performance over baseline model-merging techniques.", "questions": "1. The authors claim that HM3 supports merging in architecture space. However, all experiments are conducted using pretrained models with highly similar structures. Could the authors provide more details or ablation studies to demonstrate the effectiveness of architecture-level merging, particularly in scenarios involving structurally different components?\n2. Did the authors ensure a fair comparison between the MOEA-based baseline and the proposed method? The experimental settings and evaluation are not clearly described. Clarification is needed regarding how performance metrics, computational budgets, and parameter configurations were aligned across methods.\n3. How do the authors select a final model from the set of Pareto-optimal solutions in practice? Is there an automatic or heuristic-based selection strategy, or does it require manual validation? Please elaborate on the model selection process and whether it applies fairly across all compared methods.\n4. The paper uses (1, 1) as the reference point for hypervolume (HV) calculation. This choice may not be appropriate in practice as the boundary points may not be included in the evaluation. Have the authors considered using a more appropriate reference point, such as a slightly larger value than the estimated nadir point?\n5. Is the proposed method scalable with respect to the number of tasks or objectives? For example, how would HM3 perform when merging models fine-tuned on more than three tasks?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The combination of parameter- and architecture-level merging within a multi-objective optimization framework is novel and well-motivated.\n- The reported experimental results appear promising and demonstrate competitive performance compared to existing model merging methods.\n\nWeaknesses:\n- The experimental procedure is insufficiently explained.\n- The scalability of the proposed method is not well discussed. While the method is tested on a small number of tasks and model pairs, it is unclear how HM3 performs when scaling to more base models, larger architectures, or more objectives. \n- The paper contains typos (e.g., \"optimity\") and inconsistent symbol usage (e.g., in Appendix A.2.1, the symbols k, m, and d are used ambiguously for the number of objectives). In addition, the language can be improved throughout the paper to eliminate grammatical issues and enhance readability.", "was_revised": true, "final_justification": "The authors have addressed my concerns and improved the clarity of the paper. Therefore, I have decided to raise my score.", "rebuttal": "**W1:** We thank the reviewer for the valuable suggestions. Due to space constraints,  we provided a concise overview of the models and tasks used in the evaluation.  In the revised version, we have addressed the concern about insufficient experimental details with the following clarifications and additions:\n\n**Dataset splits and evaluation protocol:** Each task adopts a 70/30 train/test split. The RL search strictly uses training data, and final performance is evaluated on the held-out test set to avoid data leakage. Task-specific metrics are now described in full, including their computation procedures.\n\n**Hyperparameter settings and stability:** We have added a dedicated *Hyperparameter List* subsection in the appendix:\n\n- Maximum number of iterations: 1000\n- Policy and value networks begin updating after iteration $iter_0 = 200$\n- PPO clipping ratio: 0.1\n- Discount factor $\\gamma$: 0.990\n- GAE coefficient $\\beta_A$: 0.95\n- Loss coefficients: $c_1 = 1.0$ (value loss), $c_2 = 0.15$ (policy entropy)\n\nWe assess robustness via 10 runs with different seeds, reporting mean and standard deviation of rewards to illustrate training stability (**see the result in W3 of Reviewer wkP4**).\n\n**Runtime Environment:** In the *Experimental Platform* section, we now report detailed hardware and software configurations. Qwen‑2.5‑1.5B was evaluated on four 3090 GPUs (24GB each), while LLaMA‑2‑7B and LLaMA‑2‑13B were evaluated on four A6000 GPUs (48GB each). All models can also be deployed on a single GPU. The software versions, dependency libraries, and parallelism settings are listed in the appendix.\n\nTo support reproducibility, we will release the source code upon paper acceptance.\n\n\n\n**W2/Q5:** We thank the reviewer for the insightful comments on the scalability of HM3. This paper primarily focuses on architecture-level model merging and demonstrates the superiority of our approach over parameter-based baselines on several representative tasks and model scales. In response to the reviewer’s concerns, we provide a comprehensive discussion from three perspectives, i.e., **tasks**, **models**, and **objectives**, and supplement our claims with additional experiments.\n\n**Task-Level Scalability**: Beyond Translation, Math, and Code, we incorporate **GLUE benchmark** as a new task domain, **Generative/General Task** . Additionally, we further expand task diversity in the current task domains:\n\n- Translation: **XNLI**\n- Math: **MathQA**\n- Code: **MBPP**\n\n**The results are provided in W2 of Reviewer wkP4**.  HM3 maintains superior performance across all four domains, demonstrating robustness to both task number and variety.\n\n**Model-Level Scalability**: HM3 relies on layer-wise features for trajectory selection, ensuring stable complexity regardless of model pool size. We also scale to **LLaMA‑2‑13B** and observe consistent gains, confirming effectiveness on larger models. **The results of LLaMA‑2‑13B are provided in W2 of Reviewer wkP4 or W1 of  Reviewer TWdc**.\n\n**Objective-Level Scalability**: In this work, we set the number of objectives to three—translation, math, and code—chosen due to their inherent conflicts and diversity, which are essential for forming a well-separated and sparse Pareto front in multi-objective optimization. Adding more objectives without sufficiently higher diversity (e.g., general language understanding) would shift the problem into the many-objective regime. This introduces challenges such as excessive front dimensionality, dense solution distributions, and dominance relation degradation, which are known to undermine the effectiveness of traditional multi-objective methods [1].\nFor these reasons, we do not include more objectives. But we will explicitly discuss the impact of objective count on problem complexity in the *Discussion* and *Limitations* sections, and outline future directions such as designing dedicated algorithms for many-objective model merging.\n\n[1] B. Li, et al. Many-objective evolutionary algorithms: A survey. *ACM Computing Surveys*, vol. 48, no.1, pp. 1-35, 2015.\n\n\n\n**W3:**  We thank the reviewer for pointing out the spelling and notation issues. We have addressed them as follows:\n\n**Spelling**: All typos (e.g., “optimity” → “optimality”) have been corrected through a full proofreading pass.\n\n**Notation consistency:** We have standardized the notation throughout the paper as follows:\n\n- Number of objectives: $K$\n- Dimensionality of decision space: $d$\n- Layer index: $l$\n- Model index: $m$\n  All notations are now consistently used and clearly defined upon their first introduction in the main text.\n\n**Appendix:** Section A.2.1 has been revised for consistent indexing, and a *Notation Summary Table* has been added for clarity.\n\nWe believe these revisions improve both clarity and formal consistency.\n\n\n\n**Q1:**  We sincerely thank the reviewer for their thoughtful comments on the concept of *architecture-space merging*. We would first like to clarify that the core innovation of our method lies in its ability to not only merge model parameters but also to extend or reorganize the architecture of the resulting model. This architectural flexibility offers the potential to overcome performance bottlenecks of a fixed architecture by enhancing the model’s representational capacity and task adaptability.\n\nAccording to the *scaling laws*, such increases in more parameterized layers can improve model performance. For instance, SOLAR-10.7B [2] improved performance by concatenating the first 20 layers and the last 20 layers from Mixtral-7B, followed by continued pretraining. Our goal is to achieve similar architectural expansion via a training-free model merging.\n\nIn this work, we focus on architecture-level merging models via base models with similar architectures due to:\n\n- Functional alignment, preserving semantic consistency after stitching.\n- Reduced distributional shift, as models from the same base checkpoint are more compatible under MLP-based projection.\n- Practicality, since merging fully-heterogeneous architectures (e.g., Qwen with LLaMA) leads to non-functional models due to severe misalignment (**see the result in Q1 of Reviewer TWdc**).\n\nThus, HM3 represents an intermediate step between traditional parameter merging and full architectural merging, similar to SOLAR-10.7B, but without additional training.\n\nWe will clarify these constraints and future directions in the *Limitations* and *Future Work* sections.\n\n[2] D. Kim, et al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. *arXiv preprint arXiv:2312.15166*, 2023.\n\n\n\n**Q2:**  We thank the reviewer for the suggestion. We agree that including an MOEA baseline is important for fair comparison. While results were briefly reported (**Line 309**), we now provide further clarification.\n\n**Experimental setup:** Both HM3 and the MOEA baseline adopt the same decomposition-based multi-objective framework, sharing:\n\n- Scalarization strategy\n- Initialization seeds (10 per preference vector)\n- 10 independent runs per method, with averaged results\n\n**Implementation:** The MOEA baseline uses decomposed-based MOEA with differential evolution operator (pop. size = 30, crossover = 0.9, mutation = 0.1). Merging is performed via mergekit passthrough, aligned with evolutionary search method **in W1 of Reviewer wkP4**. Both methods use 4 GPUs under comparable evaluation budgets.\n\n**Results:** Using a reference point (1.1, 1.1, 1.1), HM3 consistently achieves higher hypervolume than MOEA on LLAMA-2-7B across all runs, demonstrating superior search efficiency.\n\n| Method  | HV (ref = (1.1, 1.1, 1.1)) |\n| ------- | -------------------------- |\n| MOEA    | 1.2133                     |\n| **HM3** | **1.8120**                     |\n\n\n\n**Q3:**  We thank the reviewer for the important question on preference specification. Our framework assumes users typically do not provide explicit preferences and supports two practical modes:\n\n1. Offline preference sampling (Default): We uniformly sample preference vectors to approximate the Pareto front. Users can later select a model matching their needs—no input required.\n2. Optional user preference injection: If a user specifies a preference (e.g., prioritizing translation), we select the closest model on the front or conduct a targeted search, enabling both automated and interactive use.\n\nTo ensure a fair comparison protocol, we sample the same preference vector across all methods when comparing HM3 with single-objective baselines, ensuring consistent optimization and evaluation conditions.\n\nWe will add a *“Deployment Considerations”* subsection in the appendix and clarify in the main text that all baselines are evaluated under the same preference settings.\n\n\n**Q4:**  We thank the reviewer for the insightful comment on HV calculation. We acknowledge the initial reference point was suboptimal and have corrected it during the review phase.\n\nWe now use (1.1, 1.1) for bi-objective cases, and (1.1, 1.1, 1.1) for tri-objective cases. \n\nThis follows standard practice in toolkits like PlatEMO [3], where the reference point is set slightly beyond the estimated nadir to ensure valid HV computation and full Pareto coverage.\n\nExperimental results have been updated accordingly, and Line 606 in the manuscript now states:\n\n> “The reference point is set slightly beyond the nadir (e.g., (1.1, 1.1, 1.1)) to ensure non-zero HV and full coverage of the Pareto front.”\n\nWe believe this addresses the concern and improves clarity in our evaluation.\n\n| Method               | HV     |\n| -------------------- | ------ |\n| HM3 w.o. para. opt.  | 1.3506 |\n| HM3 w.o. archi. opt. | 1.6387 |\n| **HM3**              | **1.8120** |\n\n[3] Y. Tian, et al. PlatEMO: A MATLAB platform for evolutionary multi-objective optimization. IEEE Computational Intelligence Magazine, vol. 12, no. 4, pp. 73-87, 2017."}, {"review_id": "FxUkCInFI9", "reviewer": "Reviewer_TWdc", "summary": "This paper proposes HM3, a hierarchical model merging framework that jointly optimizes both the parameter space and the architecture space. To address the complexity of the combined search space, the authors formulate the problem as a bilevel optimization task, incorporating actor-critic reinforcement learning with policy discretization. Additionally, a multi-objective optimization setup is introduced to support user preferences and task-specific trade-offs. Experiments demonstrate the framework’s ability to generate diverse, reusable, and high-performing merged models under limited resources.", "questions": "1, How does HM3 perform when merging models with significantly different architectures or diverse tasks?\n\n2, HM3 retrains and generates different merged models for each task preference vector. Have the authors evaluated whether the learned strategies (e.g., actor-critic policies) can generalize to unseen tasks or new preference vectors without re-optimization from scratch?\n\n3, Since HM3 performs merging across both parameter and architecture spaces, the resulting models and inference paths may vary significantly. Have the authors analyzed the consistency and performance variance of the merged models across multiple runs? Does the stochastic nature of the search lead to unstable final results?", "limitations": "1, No direct comparison with recent SOTA merging baselines weakens empirical validation.\n\n2, The role of hyperparameter tuning is not thoroughly addressed despite its importance in merging.\n\n3, The evaluation scope is narrow, focusing on limited model types and task domains.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1, Innovative idea of merging models across both parameter and architecture spaces.\n\n2, Clear formulation of the problem using bilevel optimization.\n\n3, Use of reinforcement learning improves efficiency in the large search space.\n\nWeaknesses:\n\n1, Lacks comparison with recent state-of-the-art model merging methods (e.g., PCB-Merging, CAT Merging), which limits the empirical strength of the work.\n\n2, Hyperparameter optimization is under-discussed; related works such as Model Evolver and GENOME are not analyzed.\n\n3, Experiments are conducted on relatively narrow model and task settings, which may reduce the generalizability of the findings.\n\nReference: \\\n1), ACL24, Model Evolver: Knowledge Fusion By Evolving Weights of Language Models \\\n2), NeurIPS24, PCB-Merging: Parameter Competition Balancing for Model Merging \\\n3), ICML25 CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging \\\n4), GENOME, Nature-Inspired Population-Based Evolution of Large Language Models", "was_revised": true, "final_justification": "The quality of this paper is improved and I had misunderstand some key contributions of this work before. The idea about bilevel optimization is quite novel.", "rebuttal": "**W1:** We thank the reviewer for the insightful suggestion. While our original focus was on classical baselines to emphasize HM3's joint parameter- and architecture-space merging capability, we acknowledge the importance of including recent SOTA methods. Accordingly, we have made the following updates:\n\n1. Inclusion of PCB Merging: We re-implemented PCB Merging and evaluated it across all tasks, including text and vision. The results are included in the revised manuscript.\n\n2. Consensus Merging as a substitute for CAT Merging: As CAT Merging was recently accepted to ICML 2025 and lacks public source code, we are working to reimplement it. Meanwhile, we added Consensus Merging [1], a strong baseline emphasized in the CAT Merging paper.\n\n3. Empirical insights: PCB Merging initially outperformed the original HM3 due to its high-quality merged model. After incorporating PCB-generated models into HM3's candidate pool, HM3 significantly outperformed PCB, highlighting its flexibility and strength in architecture-level merging.\n\n4. Updated related work: We revised the *Related Work* section to discuss PCB, CAT, and Consensus Merging, offering a more comprehensive literature review.\n\n   [1] Wang et al., Localizing Task Information for Improved Model Merging and Compression, *ICML*, 2024.\n\n**Additional experiment**\n\n1. ViT-B/32\n\n| Methods           | SUN397 | RESISC45 | SVHN  | GTSRB | DTD   | MNIST | Cars  | EuroSAT | Average   |\n| ----------------- | ------ | -------- | ----- | ----- | ----- | ----- | ----- | ------- | --------- |\n| Consensus Merging | 64.73  | 73.51    | 79.46 | 69.03 | 52.63 | 96.89 | 63.06 | 77.20   | 72.06     |\n| PCB Merging       | 63.58  | 75.47    | 82.31 | 72.57 | 54.78 | 97.12 | 64.42 | 79.63   | 73.74     |\n| Original HM3      | 63.22  | 73.27    | 77.62 | 68.21 | 52.22 | 96.94 | 64.02 | 78.06   | 71.70     |\n| **HM3**           | 63.42  | 76.27    | 82.11 | 73.11 | 54.60 | 96.85 | 64.63 | 79.66   | **73.83** |\n\n2. Vit-L/14\n\n| Methods           | SUN397 | RESISC45 | SVHN  | GTSRB | DTD   | MNIST | Cars  | EuroSAT | Average   |\n| ----------------- | ------ | -------- | ----- | ----- | ----- | ----- | ----- | ------- | --------- |\n| Consensus Merging | 73.39  | 88.05    | 87.43 | 81.16 | 66.04 | 98.88 | 84.26 | 90.81   | 83.75     |\n| PCB Merging       | 75.04  | 88.75    | 86.46 | 86.55 | 69.13 | 98.91 | 86.01 | 91.01   | 85.23     |\n| Original HM3      | 72.85  | 88.00    | 90.48 | 83.43 | 66.72 | 98.41 | 86.17 | 89.50   | 84.45     |\n| **HM3**           | 74.76  | 88.43    | 90.02 | 85.17 | 70.21 | 98.44 | 86.33 | 89.32   | **85.34** |\n\n3.LLAMA-2-13B\n\n| Methods           | General-Glue | Translation-WMT&ISWT | Translation-XNLI | Math-GSM8K | Math-MathQA | Code-HumanEval | Code-MBPP |\n| ----------------- | ------------ | -------------------- | ---------------- | ---------- | ----------- | -------------- | --------- |\n| Consensus Merging | 45.15        | 32.54                | **44.03**        | 36.24      | 24.21       | 38.82          | 25.03     |\n| PCB Merging       | 52.77        | **41.96**            | 40.90            | 46.34      | 24.56       | 40.15          | 26.57     |\n| **HM3**           | **53.20**    | 41.12                | 43.92            | **46.82**  | **29.13**   | **43.93**      | **33.29** |\n\n4.LLAMA-2-7B\n\n| Methods           | General-Glue | Translation-XNLI | Math-MathQA | Code-MBPP |\n| ----------------- | ------------ | ---------------- | ----------- | --------- |\n| Consensus Merging | 47.52        | 40.74            | 27.99       | 37.93     |\n| PCB Merging       | 49.11        | 33.35            | **28.14**   | 39.05     |\n| **HM3**           | **51.04**    | **40.24**        | 28.08       | **44.62** |\n\n5.Qwen-2.5-1.5B\n\n| Methods           | General-Glue | Translation-XNLI | Math-MathQA | Code-MBPP |\n| ----------------- | ------------ | ---------------- | ----------- | --------- |\n| Consensus Merging | 46.42        | 38.09            | 37.84       | 34.08     |\n| PCB Merging       | 47.25        | 40.23            | 36.90       | 41.33     |\n| **HM3**           | **48.22**    | **41.03**        | **40.13**   | **47.80** |\n\n\n\n**W2:**  We thank the reviewer for the valuable feedback. In original version, hyperparameters were tuned via grid search on smaller models and then transferred to larger ones. The revised manuscript includes further clarifications:\n\n- Expanded hyperparameter details: We now list key hyperparameters (e.g., RL learning rate, Wolpertinger neighborhood size, number of subproblems) with value ranges. These were tuned on Qwen‑1.5B and ViT‑B/32, then applied to LLaMA‑2‑7B and ViT‑L/14. Further experiments are ongoing and results will be provided for the final version.\n- Updated related work: We added *Model Evolver* and *GENOME* to our review. Both use evolutionary method to merge weights in fixed architectures and need to restart for each task.\n- Comparison with HM3: Unlike the above, HM3 incorporates RL for architecture-level search, supports cross-model layer stitching, and adopts multi-objective scalarization for diverse solutions. The learned policy generalizes across tasks and model pools, enhancing reuse and efficiency. We also discuss potential combinations of evolutionary search and RL to further improve joint merging capabilities.\n\n**W3:** We thank the reviewer for emphasizing the importance of generalization. Our original evaluation covered three core LLM task domains, math reasoning, code generation, and multilingual translation, following the Qwen-3 evaluation protocol (e.g., Table 3 in the Qwen-3 technique report). To strengthen generalization, we have expanded both task and model coverage:\n\nExpanded Task domain: Added **GLUE benchmark** for **generative task domain**.\n\nExpanded Tasks:\n\n- *Multilingual (Translation):* Added **XNLI** dataset.\n- *Math:* Added **MathQA** dataset.\n- *Code:* Added **MBPP** dataset.\n\nExpanded Models:\n\n- Added **LLaMA-2-13B** and fine-tuned variants.\n\nThese enhancements further demonstrate the robustness and broad applicability of our method across tasks and model sizes.\n\n**Q1:** We thank the reviewer for the insightful feedback and respond from two angles: model compatibility and task generalization.\n\n**Model Compatibility**: HM3 effectively merges models with identical architectures but different specializations. Notably, HM3 also supports architecture-level merges, unlike standard parameter merging, allowing flexible architectural extensions. This capability draws inspiration from SOLAR‑10.7B [2], which significantly improved performance by concatenating the first 20 layers and last 20 layers of Mixtral‑7B followed by continued pretraining. HM3 aims to achieve similar architecture expansion in a training-free manner through model merging, without the need for expensive retraining.\n\nHowever, merging models with fundamentally different architectures (e.g., Qwen vs. LLaMA) remains difficult. Severe mismatches in dimensions and representations can significantly degrade performance. Our MLP alignment mitigates shallow-level discrepancies but cannot fully bridge deeper semantic gaps. Without alignment, merged models fail even on simple benchmarks.\n\nWhile HM3 does not yet support fully heterogeneous model merging, it serves as a promising step toward that goal. We will clarify these limitations and discuss future extensions in the revised version.\n\n| Merging LLAMA + Qwen      | Translation | Math     | Code     |\n| ------------------------- | ----------- | -------- | -------- |\n| HM3 without MLP Alignment | 0           | 0        | 0        |\n| **HM3**                   | **11.68**   | **9.21** | **5.13** |\n\n**Task Generalization:** We initially evaluated HM3 on math, code and translation task domains, following Qwen‑3's domain grouping. In the revised version, we expand coverage to include the **General** task domain and an additional task per existing task domain. Results confirm HM3's strong performance across diverse tasks, highlighting its generalization ability.\n [2] D. Kim et al., *Solar 10.7B: Scaling Large Language Models with Simple Yet Effective Depth Up-Scaling*, arXiv:2312.15166, 2023.\n\n**Q2:** We thank the reviewer for emphasizing the importance of policy generalization. To evaluate reusability, we conducted two new experiments:\n\n- Cross-domain transfer: A policy trained on translation, math, and code tasks was tested on unseen domains (QA, WSD, sentence completion), following the out-of-domain setup in PCB Merging. Results will be added to a new *Out-of-Domain Generalization* subsection.\n- Zero-shot preference generalization: A well-trained policy network in HM3 was applied to two unseen vectors without further search, directly generating inference paths for model merging. The result is the following table.\n\n| Zero-shot preference generalization  | Translation | Math  | Code  |\n| ------------------------------------ | ----------- | ----- | ----- |\n| HM3                                  | 43.86       | 39.05 | 36.85 |\n| Re-train HM3 based on sampled vector | 45.93       | 40.21 | 38.37 |\n\nThese experiments evaluate generalization across both task domains and preference vectors. Results show that HM3’s policy generalizes effectively in a zero-shot setting, producing competitive merging paths. We will provide complete results and implementation details in the final revision to highlight the policy’s robustness and reusability.\n\n**Q3:** We thank the reviewer for highlighting the importance of result stability. As HM3 performs joint search over parameter and architecture spaces, we address randomness by running the full pipeline 10 times per preference vector using different seeds. As shown in Figure 6, we report the mean and standard deviation across runs. The final reward is shown **in W3 of Reviewer wkP4**, which demonstrates the stability of HM3."}, {"review_id": "HA2l13zprG", "reviewer": "Reviewer_wkP4", "summary": "This work introduces a novel formulation of model merging as a bilevel multi-objective optimization problem that simultaneously explores the parameter and architecture spaces—overcoming the constraint of previous methods that focus only on parameter merging within identical architectures. The upper level conducts parameter fusion guided by user-defined task preferences, while the lower level performs architecture search by identifying optimal inference paths across layers.", "questions": "I have some concerns regarding the computational efficiency of HM3, the limited set of parameter-level baselines, and the stability of the RL-based architecture search (see Weakness part). It would be helpful if the authors could address these points in their response.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- Strengths\n1. The paper introduces a novel bilevel multi-objective optimization framework that simultaneously considers both parameter-level and architecture-level merging. This formulation significantly extends the scope of model merging beyond existing parameter-only approaches.\n2. The optimization problem is clearly defined with solid theoretical foundations. The use of a Stackelberg equilibrium-based bilevel formulation, together with preference-driven decomposition, makes the overall methodology logically sound and principled.\n3. By approximating the Pareto front across multiple objectives, HM3 allows the user to select merged models based on custom task preferences, providing flexibility and interpretability not present in single-solution methods.\n\n- Weaknesses\n1. Its bilevel nature and RL-based architecture search introduce substantial computational overhead. Unlike many model merging methods that emphasize efficiency under limited resources, the paper lacks sufficient evaluation on computational cost. The supplementary material mentions a cost analysis, but it is not comprehensive or benchmarked against strong baselines. Table 6 compares only against full fine-tuning, which may not be the most appropriate reference.\n2. The paper compares HM3 against only a small subset of merging baselines (Task Arithmetic, TIES, DARE-TIES), while omitting many recent and competitive methods. This weakens the empirical validity of the claimed superiority in parameter-level performance.\n3. Figure 2 suggests that all baselines are dominated by at least one HM3 solution, but it remains unclear whether HM3 produces stable and reproducible solutions across runs, given the stochasticity of the RL-based architecture search. Although the appendix provides some analysis on RL effectiveness, it does not fully address potential convergence instability or variance across seeds.\n4. There appears to be an error in Equation (13).\n5. The feasibility and semantic consistency of stitching layers across different models via a learned MLP-based projection is an underexplored issue. A deeper analysis on representation alignment or performance degradation from misaligned layers would strengthen the paper.", "was_revised": true, "final_justification": "The proposed method shares the objectives of conventional model merging but offers novel alternative contributions, justifying the adjusted rating.", "rebuttal": "**W1:** Thank you for your attention to the time cost analysis. We acknowledge that the original manuscript lacked sufficient detail regarding computational overhead. This omission was primarily due to our view that architecture-level model merging is fundamentally a different task from most existing parameter-level merging methods. The former involves dynamically altering the inference path via search, which inevitably introduces a larger search space and higher computational complexity. However, we believe the necessity of exploring more optimal model compositions beyond fixed architectures should not be overlooked. Therefore, despite the increased complexity, architecture merging constitutes a meaningful research direction that deserves investigation.\n\nTo address this concern, we have included two additional **architecture-level baselines** under the same maximum inference path length T_{\\max} for comparison:\n\n- **Grid Search**: Exhaustively enumerates possible paths using Mergekit’s passthrough for layer composition until a similar number of evaluations as HM3 is reached.\n- **Evolutionary Search**: Uses differential evolution with standard settings and Mergekit’s passthrough for layer composition, matching HM3's evaluation budget. This differs from the EA baseline in the original submission, which restricted the search space via token constraints but at the cost of final merged model performance.\n\nThe performance comparison is shown below:\n\n| Method              | Translation | Math      | Code      |\n| ------------------- | ----------- | --------- | --------- |\n| Grid Search         | 29.31       | 21.21     | 23.12     |\n| Evolutionary Search | 31.92       | 23.49     | 26.90     |\n| **HM3**             | **44.68**   | **45.62** | **43.62** |\n\nThese results confirm that HM3 consistently outperforms other search-based merging methods under equivalent computational budgets.\n\n**W2:** As noted in W1, architecture-level merging differs fundamentally from parameter-level merging. While the latter operates on a fixed model topology by interpolating weights, architecture merging constructs new inference paths by combining layers from different pretrained models.\n\nOur original baselines focused on representative parameter-level methods to highlight the benefits of architecture merging. However, we acknowledge that this may have limited the comprehensiveness of the evaluation.\n\nTo address this, we have included two SOTA parameter merging methods—PCB-Merging [1] and Consensus Merging [2]—as baselines, recommended in **Reviewer TWdc**. Results show that HM3 consistently outperforms on language ( **see following additional experiment**) and vision tasks (**please see the result in Reviewer TWdc**), reinforcing the effectiveness of HM3.\n\n[1] PCB-Merging: Parameter Competition Balancing for Model Merging, NeurIPS 2024\n[2] Consensus Merging: Localizing Task Information for Improved Model Merging and Compression, ICML 2024\n\n**Additional experiment**\n\n1.LLAMA-2-13B\n\n| Methods           | General-Glue | Translation-WMT&ISWT | Translation-XNLI | Math-GSM8K | Math-MathQA | Code-HumanEval | Code-MBPP |\n| ----------------- | ------------ | -------------------- | ---------------- | ---------- | ----------- | -------------- | --------- |\n| Consensus Merging | 45.15        | 32.54                | **44.03**        | 36.24      | 24.21       | 38.82          | 25.03     |\n| PCB Merging       | 52.77        | **41.96**            | 40.90            | 46.34      | 24.56       | 40.15          | 26.57     |\n| **HM3**           | **53.20**    | 41.12                | 43.92            | **46.82**  | **29.13**   | **43.93**      | **33.29** |\n\n2.LLAMA-2-7B\n\n| Methods           | General-Glue | Translation-WMT&ISWT | Translation-XNLI | Math-GSM8K | Math-MathQA | Code-HumanEval | Code-MBPP |\n| ----------------- | ------------ | -------------------- | ---------------- | ---------- | ----------- | -------------- | --------- |\n| Consensus Merging | 47.52        | 37.97                | 40.74            | 37.95      | 27.99       | 30.15          | 37.93     |\n| PCB Merging       | 49.11        | 39.29                | 33.35            | 39.25      | **28.14**   | 32.53          | 39.05     |\n| **HM3**           | **51.04**    | **41.68**            | **40.24**        | **45.62**  | 28.08       | **43.62**      | **44.62** |\n\n3.Qwen-2.5-1.5B\n\n| Methods           | General -Glue | Translation-WMT&ISWT | Translation-XNLI | Math-GSM8k | Math-MathQA | Code-HumanEval | Code-MBPP |\n| ----------------- | ------------- | -------------------- | ---------------- | ---------- | ----------- | -------------- | --------- |\n| Consensus Merging | 46.42         | 29.82                | 38.09            | 23.98      | 37.84       | 22.59          | 34.08     |\n| PCB Merging       | 47.25         | 30.05                | 40.23            | 24.29      | 36.90       | 21.87          | 41.33     |\n| **HM3**           | **48.22**     | **32.26**            | **41.03**        | **28.05**  | **40.13**   | **34.31**      | **47.80** |\n\n\n\n**W3:** Thank you for your valuable comments. We address your concerns in two parts:\n\n**Regarding Figure 2:** The displayed Pareto front is not from a single run, but represents the mean performance across 10 independent runs, each using a different random seed for each preference vector. For each vector, we generate one merged model per run, evaluate it on the three tasks, and report the average performance over 10 runs.  As a result, we obtain 15 averaged performance points corresponding to the 15 preference vectors (S1–S15). All baselines are also run with 10 different seeds for fair comparison. We will clarify this setup in the revised version.\n\n**Regarding Figure 6:** We agree that the current figure lacks a depiction of variability. To improve it, we re-ran HM3 under the same preference vector for 10 independent trials. The standard deviation of final rewards is consistently below 0.6, confirming the robustness and reproducibility of our method. The updated figure in the revision will show the average reward curve with shaded confidence intervals across episodes to better illustrate convergence stability.\n\n| Episode | Mean Reward (± Std) |\n| ------- | ------------------- |\n| 0       | 0.000 (± 0.000)     |\n| 50      | 3.772 (± 0.049)     |\n| 100     | 5.093 (± 0.099)     |\n| 150     | 4.014 (± 0.046)     |\n| 200     | 4.521 (± 0.120)     |\n| 250     | 8.218 (± 0.558)     |\n| 300     | 7.140 (± 0.300)     |\n| 350     | 12.427 (± 0.391)    |\n| 400     | 10.169 (± 0.204)    |\n| 450     | 15.404 (± 1.323)    |\n| 500     | 15.936 (± 0.675)    |\n| 550     | 16.783 (± 1.539)    |\n| 600     | 16.242 (± 1.047)    |\n| 650     | 16.123 (± 1.111)    |\n| 700     | 18.189 (± 0.572)    |\n| 750     | 17.083 (± 1.497)    |\n| 800     | 16.022 (± 1.547)    |\n| 850     | 17.547 (± 0.657)    |\n| 900     | 17.256 (± 0.745)    |\n| 950     | 17.533 (± 0.683)    |\n| 1000    | 17.662 (± 0.644)    |\n\n**W4:**  Thank you for your constructive feedback on the value function formulation. While Eq. (13) is mathematically correct, we agree its original form may cause ambiguity.\n\nTo improve clarity, we revised it as:\n\n$L^{\\text{VF}}(\\phi) = E_t \\[(V\\phi(S_t)-\\hat{V_t})^2\\], \\quad \\hat{V_t} = V_\\phi(S_t) + \\hat{A_t}$\n\n\nwhere $\\hat A_t$  is the advantage estimated via generalized advantage estimation. This formulation clearly separates the value prediction $V_\\phi(S_t)$ from the target $\\hat V_t$, avoiding confusion with expressions like $V_\\phi(S_t)-\\hat A_t$.\n\nWe also include a detailed explanation of GAE in the appendix to support reproducibility and clarity.\n\n**W5:** Thank you for your thoughtful suggestions regarding the feasibility and semantic consistency of cross-architecture layer stitching. We fully agree that maintaining representational alignment is critical when merging layers from heterogeneous models. Our MLP-based projection mechanism is specifically designed to address both dimensional and distributional mismatches.\n\nWe revised the manuscript to analyze this issue from three perspectives:\n\n1. Theoretical and empirical motivation: We explicitly discuss challenges such as hidden size mismatches, differing attention head numbers, normalization statistics, and semantic drift, which are common even across structurally similar models. Prior work [3] also shows that naive layer stitching (e.g., CodeLLaMA vs. LLaMA-2-Chat) significantly degrades performance. Our appendix includes a moment-matching analysis to support how the projection matrix helps align feature distributions.\n\n2. Empirical validation: We conducted ablation experiments using LLaMA-2-7B to assess the impact of omitting MLP alignment:\n\n| Model                     | Translation | Math      | Code      |\n| ------------------------- | ----------- | --------- | --------- |\n| HM3 without MLP Alignment | 38.15       | 39.83     | 38.21     |\n| **HM3**                   | **44.68**   | **45.62** | **43.62** |\n\nThese results confirm that the projection mechanism substantially improves cross-layer semantic alignment and task performance.\n\n3. Future directions: A new *Discussion* section in the revised manuscript highlights this issue and frames our work as an initial step toward principled cross-model layer composition. We hope it motivates further exploration of lightweight and theoretically grounded alignment strategies.\n\n[3] Y. He, et al. MergeBench: A Benchmark for Merging Domain-Specialized LLMs. arXiv preprint arXiv:2505.10833, 2025."}]

Author rebuttal:

Meta-review: This paper presents an interesting model merging method that simultaneously supports parameter-level and architecture-level merges, allowing flexible architectural extensions. All the reviewers vote for acceptance.
Decision: Accept (spotlight)