Multi-objective Large Language Model Alignment with Hierarchical Experts
Authors: Zhuo Li
Guodong DU
Weiyang Guo
Yigeng Zhou
Xiucheng Li
Wenya Wang
Fangming Liu
Yequan Wang
Deheng Ye
Min Zhang
Jing Li
Venue: ICLR 2026 Poster (2026)
Keywords: large language model
multi-objective
mixture-of-expert
model fusion
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: HoE (hierarchical Mixture-of-Experts) is a Multi-objective Alignment approach that enabling LLMs to adapt across the entire Pareto frontier with minimal resources.

Abstract: Aligning large language models (LLMs) to simultaneously satisfy multiple objectives remains a significant challenge, especially given the diverse and often conflicting nature of human preferences. Existing alignment methods struggle to balance trade-offs effectively, often requiring costly retraining or yielding suboptimal results across the Pareto frontier of preferences. In this paper, we introduce HoE (Hierarchical Mixture-of-Experts), a lightweight, parameter-efficient, and plug-and-play approach that eliminates the need for model retraining, while enabling LLMs to adapt across the entire Pareto frontier and accommodate diverse user preferences. In particular, HoE consists of three hierarchical components: LoRA Experts, Router Experts and Weighting Router, reaching optimal Pareto frontiers and achieving a trade-off between parameter size, training cost, and performance. We evaluate HoE across various tasks on 16 objectives and 200 different preferences among 8 benchmarks, demonstrating superior performance over 15 recent baselines.

Published as a conference paper at ICLR 2026
MULTI-OBJECTIVE LARGE LANGUAGE MODEL ALIGNMENT WITH HIERARCHICAL EXPERTS
Zhuo Li Guodong Du'-, Weiyang Guo', Yigeng Zhou', Xiucheng Li',
Wenya Wang", Fangming Liu3.4 Yequan Wang* Deheng Ye", Min Zhang', Jing Li Harbin Institute of Technology, Shenzhen, China
"Nanyang Technological University, Singapore P Cheng Laboratory, China
"Huazhong University of Science and Technology, China
Beijing Academy of Artificial Intelligence Peking University TTencent
zuoer1 901 91 @ma i 1 www.sstc..... edu. jingli wangli.phd@heee phddeotta com
ABSTRACT
Aligning large language models (LLMs) to simultaneously satisfy multiple ob- jectives remains a significant challenge, especially given the diverse and often conflicting nature of human preferences. Existing alignment methods struggle to balance trade-offs effectively, often requiring costly retraining or yielding subopti- mal results across the Pareto frontier of preferences. In this paper, we introduce HoE (Hierarchical Mixture-of- -fepeetsss,,,))..... a lightweight, parameter-effiient, and plug-and-play approach that eliminates the need for model training, while enabling LLMs to adapt across the entire Pareto frontier and accommodate diverse user preferences. In particular, HoE consists of three hierarchical components: LoRA Experts, Router Experts and Preference Routing, reaching state-of-the- Pareto frontiers and achieving trade-off between parameter size, training cost, and per- formance. We evaluate HoE across various tasks on 16 objectives and 200 different preferences among 8 benchmarks, demonstrating superior performance over 15 recent baselines. WARNING: This paper contains potentially offensive text.
INTRODUCTION
Large language models (LLMs) have achieved remarkable success in aligning with broadly defined human values (Achiam et al., 2023; Ziegler et al., 2019; Sun al., 2023). However, human preferences in practice are highly diverse and cannot be fully captured by an universal alignment goal. Users may pursue multiple personalized objectives, and even when the objectives are the same, their relative importance often varies across individuals and contexts (Fu et al., 2024; Yang et al., 2024c; Lin et al., 2025; Ren al., 2025; Zhang al., 2025). Existing approaches (Sun al., 2023: Yang et al., 2025; Chen et al., 2025), which typically optimize for one objective fixed combination, fall short in flexibly covering this preference space. These observations highlight the necessity of multi-objective alignment (MOA) to enable scalable and preference-steerable LLMs (Lin et al., 2025; Guo et al., 2024; Ramé et al., 2023; Jang et al., 2023; Yang et al., 2024c; Lin et al., 2025; Chen et al., 2025).
The central difficulty of multi-objective alignment (MOA) lies in its inherently steerable nature: LLMs must dynamically adapt to arbitrary user preferences rather than a single fixed goal, essentially acting a taack-of-all-trase
(Lin et al., 2025; Chen al., 2025; Du et al., 2025a). This steerability introduces two major challenges. First, objectives often conftict with each other. Parameters tuned to improve one objective (e.g., helpfulness) often undermine another (e.g., harmlessness) (Chen & Kwok, 2025; Zheng& & Wang, 2024; Du et al., 2024; Yadav et al., 2023; Zhou et al., 2024a). Second, competition also even exists across preferences along the Pareto frontier (Wang et al., 2024b; Shi et al., 2024; Zhou et al., 2024b; Li et al., 2021). 1). For instance, a model trained uniformly across all weightings (black solid Pareto Frontier Fig. Left) cannot achieve optimal performance at specific weighting (e.g., [0.5, 0.5]), compared to the expert model fine-tuned exclusively for that preference (colored dashed Pareto frontier in Fig. Left).
M Corresponding author. Equal contrihution. The code is publicly available this https URI..

---

Published as a conference paper at ICLR 2026
To overcome this limitation, adopt a decomposition- bassd strategy for MOA, breaking down the multi-objective alignment problem into series of single- preference subproblems (Zhang Li. 2007), each associated with a set of specialized parameters. These parameters, referred to as experts, are each assigned to distinct preference, focus solely on their corresponding preferences and optimize within their localized subproblem regions. This strategy avoids the pitfalls of single monolithic model attempting to cover the entire Pareto Frontier, thereby circumventing the steerability bottleneck.
Jack-of-all-trddes (Our HoE) Mixture-of-Experts
pareto frontiers
(Our HoE) w/router experts
Experts
wilora experts
obj
Specialized Direction-
abj
wimodel soups
[0.4]
base models
[1,0]
ong
Figure 1: (Left) HcE decomposes the multi-objective alignment problem into a series of handled by a specialized expert. (Right) HoE employs hierar- chical experts, integrating LoRA and router experts to approach near-optimal Pareto frontier.
subproblems, each
Building this idea, we then instantiates each expert as lightweight LoRA adapter within Mixture-off Cetuure-of-Experte of-Experts framework. Preference-specine behavior is captured in small, composable LoRA modules while the backbone parameters remain shared. At inference time, a routing mechanism composes and activates the appropriate LoRA experts, adapting the model's behavior dynamically to realize arbitrary weightings. By this design, efficiently reconstruct the full Pareto frontier from collection of localized, preference-specine experts, enabling a scalable, steerable, and parameter-efficient approach for multi-objective alignment.
In this work, we propose HoE, novel hierarchical Mixture-off of Experts framework for multi-objective aligmment. HoE a lightweight, parameter-efficient, and plug-and-play solution that eliminates the need for training any models while achieves strong performance across the entire Pareto frontier. It combines the decompasition principle with the LoRA-basee MoE design to enable scalable, efficient and fine-grained control over the entire Pareto frontier. Specificaly, HoE comprises three hierarchical components: LoRA experts, router experts, and preference routing. (1) "LoRA experts" are first extracted without training from off-the-shelf single-objective models using task-vector singular value decomposition (task-SVD), capturing distinct alignment objectives in compact adapter modules. (2) imulti-objection LoRA experts" are then synthesized, also without training by merging multiple existing single-objective LoRA experts, to enable on-demand generation of alignment capabilities across arbitrary preference configurations. (3) "Router experts" are trained with negligible parameters to dynamically select and combine the appropriate experts based user-specified preferences, allowing efficient traversal of the Pareto frontier. Through this hierarchical design, HoE not only provides precise control over preference-specifi behavior but also balances alignment performance, parameter cost, and training efficiency. serves as a practical and effective solution for scalable multi-objective alignment in LLMs.
The main contributions of this study are as follows:
We investigate novel decomposition strategy that breaks down the multi-objective alignment problem into series of single sengel--reeeence subproblems, each handled by a set of specialized experts, enabling ane-grained control and full Pareto coverage.
propose HoE, a lightweight, parumeter-efsiceen and plug-and-play hierarchical Mixture-of- Experts framework that comprises three-level hierarchy, bypassing full model training.
We evaluate HoE across diverse multi-, many-objective and multi-task settings, involving 14 objectives, 6 benchmarks, and 200 preference. HoE consistently outperforms 15 recent baselines with lower training cost and parameter overhead.
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with other alignment methods. M is number of preference, N is number of objectives and M > N. Our HoE approach pareto-steerable and lightweight method with highest scalability, least storage cost and least inference cost, which eliminates the need for retraining any new models or any structed prompts. Each characteristic is empirically conformed Section B.5.
Characteristic (->)
Number of stored models
Inference cost
Number
Pareto
Multi-task
Free from prompting
Scalability
Method (
trained models
stecrable
ability
MORLHF
M
M
Retrain
MODPO (ACU
RS |NoutPS
N
0
RIC 24)
>1
Retrain Retrain
MOD [NoulRS 24]
N
O
Args (4LU
O
NN
O
V
Steering (RA
MetaAligner (NcudPS
1
x
LoraMoE [ACL24]
Retrain
PAD pCLB 25]
Extra
GenARM (CUR29)
SN
O
PARM (DCML
Retrain
(ours)
2 RELATED WORK
LLM Multi-ohjective Alignment. MORLHF (Li al., 2021) and MODPO (Zhou al., 2024b) employ linear scalarization to combine multiple reward signals into single scalar metric. applying standard RLHF or DPO training separate model for each preference. Multi-objective Decoding (MOD) (Shi et al., 2024), Aligmment as Reward-Guides Search (Args) (Khanov et al., 2024) and Personalized Alignment at Decoding-time (PAD) (Chen al., 2025) derive closed form solution of optimal preference model and perform linear fusion of logits prediction during decoding. Directional Preference Alignment (DPA) (Wang al., 2024a) and Reward-in-Contexi (RiC) (Yang et al., 2024c) typically inject user preferences into the prompt, enabling in-context preference-conditiones align- ment. Additionally, Steering (Konen et al., 2024; Rimsky et al., 2024) adding "steering vectors" to all token positions after the user's 's prompt, enabling precise control multi-objective preferences. Metanligner (Yang et al., 2024b) extends the Aligner (Ji et al., 2024) framework to MOA, refining weaker outputs to better match user preferences.
Knowledge Fusion for LLMs. Model merging (Jin et al., 2022; Matena & Raffel, 2022; Du et al., 2024; Zheng Wang, 2024: Yadav al., 2023) is a widely used fusion technique that integrates multiple task-specific models into a unified model. Task Arithmetic (TA) (Ilharco et al., 2023; Ramé al., 2023; Jang et al., 2023) linearly combines task vectors, defined as the parameter differences between task-specific models and the original pre-trained model. Then Rewarded Soups (RS) (Ramé et al., 2023) and Personalized Soups (PS) (Jang et al., 2023) firstly extend this concept to MOA. LoraMoE (Dou al., 2024), the closest work to ours, is Mixture-of-Expee that uses LoRA Adapters (Hu et al., 2022) as experts, integrating LLM knowledge by activating select experts via router network. However, it requires costly training across all LoRA experts simultaneously and limits knowledge sharing among them, thus unsuitable for MOA.
(MoE) approach
In summary, we systematically compare and analyze existing LLM alignment methods in Tab. 1.
METHODOLOGY
In this section, we present the methodology behind HoΣ, a lightweight, parameter-efficient, and plug-and-play multi-objective alignment framework. As illustrated in Fig. 2, our HoE approach consists of three Iierarchical components: LoRA experts, router experts and preference routing.
Multi-Objective Alignment Problem Setting. In the MOA setting, consider N alignment objectives with reward functions (R(.) ()) user preference is represented by a weight vector usrr (AI...,, AN) E N-1 in the N-dimensional simplex, and is specified as a preference- weighted reward: R (z. y) = Σn X₁R, (2, (2.y). The objective MOA to learn policy that aligns with arbitrary preferences X across the simplex: max R0;;; =)= (-)oa)))))))) RRx (x. y)).
3

---

Published as a conference paper at ICLR 2026
Gamaec Maybe you rould owy Joke time aezuel."
she-stell prrference roiels
muttt Zoro EXports
Fornation
lona
Sparasryy
Ferging
Tzaroforeez
Laysn
resnsformee Layer 32
Final Output
Plup-in
Plug-in
HoE
Nulti Objective
Multi -efeetivee
Preference Routing
0.0
LoRA Experts
Router Experts
Transforeer Layer 1 O 23
Top-k
Eaparta
Experts selartion
Modale
B
Sedees Wesrect
Podula (0)
Parametere
neights for experts selection
Lagiis
Ireforence
Inaut Prospt
R.S,
assghts setection
friend wao beent
Helpfal Haceluss
0.55
borting
Inference
0.33
Sote every
partiy
Hidden States Input
Hidden States Input
Perfececce
Figure 2: Illustration of our HoE approach. The left side illustrates the application scenario, where the model generates response aligned with the prompt and given preferences. The bottom-right highlights its three hierarchical components the LoRA experts, router experts, and a preference routing. The right depicts individual components, each serving as an expert for specific weightings, designed for seamless plug-and-play integration within the model.
3.1 Primary LORA EXPERTS
Single-Objectin LoRA Experts. 1) Extraction. We begin with a pre-trained model Tpre. and collection of off-the-shell single-objective optimal policies {-f, ... TV }. each fine-tuned its respective objective.
T arg E. (0) ( Tpre)]
Let 6pre denote the pretrained parameters and let 0; be the parameters of the fine-tuned model. Following the task vector paradigm model merging (Ilharco al., 2023), we define each objective vector as the parameter update between fine-tuned weights and the pre-trained weights:
Tj 0: Ppee
which inherently capture the single-objective capabilities of each single-objective model.
2) Compression. Each objective vector T; is then compressed into a low-rank adapter via a task- aware truncated SVD procedure ("task-SVD") A, B, <- task-SVD(rT). where A, E Rd
B, E R.rxd and & min min(di, d. In practice, task-SVD selects high-magnitude components of T, performs per-layer SVD truncation and rescales the parameters to form new LoRA matrices A,, B, This compression preserves the optimal single-objective performance with negligible performance loss (cf. Wang al. (2024c); Ping et al. (2024); Gu et al. (2025); Yuan et al. (2023); Ryu et al. (2023); Du et al. (2025b)). These compact adapters, referred to as LoRA Experts, are highly specialized for their corresponding objectives.
3) Plugin. We convert all linear layers in the Transformer into MoE-style plugin modules, incorpo- rating the LoRA experts. Given score weight w(1) E R from the router, the module output for input E Rdi composes the pretrained weight with weighted sum of LoRA experts:
xx(x) = Wpre + EN B.
(3)
where W pre X is the base linear weight and B, Aşa is the low-rank residual from expert i.
4

---

Published as a conference paper at ICLR 2026
Multi-Objective LoRA Experts. For preferences involving multiple objectives simultaneously. the aforementioned linear combination single-objective experts may fail to recover optimal performance, especially intermediate points on the Pareto frontier (e.g., À 00.5, 0.5]). To address this, we draw inspiration from model merging (Yang et al., 2024a; Matena & Raffel, 2022; Zheng & Wang, 2024; Du et al., 2024; Jin et al., 2022; Yadav al., 2023) which amplify parameters beneficial to all tasks while suppressing conflicting or detrimental ones, enable nonlinear and fine-grained parameter adaptation, and significantly outperform linear approaches (e.g., Task Arithmetic (Ilharco et al., 2023)).
To cover the entire Pareto frontier, we incorporate model merging into our framework derive new expert parameters tailored to arbitrary preference vectors. Given a target preference X, we specify the desired objective proportions and synthesize a merged expert with parameters:
T> = Merge( (ee
where {7e] are the objective vectors derived from single-objective model. We then reuse the same task-SVD procedure. These resulting adapters serve as multi-objective LoRA experts, and are no longer aligned with a single objective, but instead specialized in specific combinations of objectives (e.g.. [0.5.0.5). 0.5).
3.2 Secondary ROUTER EXPERTS
While increasing the number of LoRA experts can improve Pareto coverage, the overall parameter budget quickly becomes prohibitive, each adapter still adds non-trivial number of parameters. To address this, we introduce Router Experts, a lightweight and lineggraiedd decomposition mechanism secondary experts. Their parameter cost is negligible compared to LoRA adapters, yet they play a crucial role enhancing flexibility: unlike LoRA experts, which are statically combined and tied fixed preferences, router experts enable module-wise fine-grained routing and input-adaptive selection. In practice, this allows the model to dynamically determine which LoRA experts to activate finer granularity depending on the input, thereby achieving more efficient and adaptive utilization of LoRA capacity across the network.
Formation. We insert light-weight linear router layer into every Transformer block as router expert. The router layer takes the same hidden states x as the LoRA adapters and outputs a score voting over all available LoRA experts. Each router expert 7) is associated with a target preference vector (e) E N-1 and then activates only the N nearest LoRA experts to A (e) in preference space.
Optimization. A key design is that the parameters of all LoRA experts remain frozen, drastically reducing resource requirements. As each router expert optimized with respect to its specific weighting (e), it qualifies as expert tailored that particular preference.
The training goal of a router expert 7/x is to realize the mixture policy that maximizes the scalarized multi-objective reward aligned with preference (e). Let nn be HoE model comprising 7), the optimization problem is
7A
arg max Enyor [R( ?)
(5)
To address non-convex regions of the Pareto frontier, we adopt Tehebycheff' (TCH) scalarization, which focuses on the worst-performing objective relative to reference point z" E I::
J(([)) max min {}, (0) 27)}.
(6)
Intuitively, this ensures that router experts do not simply optimize for the weighted average but instead balance objectives even in difficult trade-off regions.
We solve this max-min problem via Online Mirror Descent (OMD) (Liu et al., 2024), which maintains smoothed distribution
objectives. The equivalent reformulation is
J(Ø max Σ wi (0) 2f), s.t. w arg (w; (R,, 27)), ||||] 1. (7)
The auxiliary weights w are updated online using stabilize optimization.
learning (Qiu et al., 2024) to
5

---

Published as a conference paper at ICLR 2026
Finally, we integrate this process into PPO (Schulman et al., 2017). The resulting policy gradient takes the following form, where A denotes the advantage with respect to reward R4-
V voJ(OlA) as (E N
(s, a4) V log (a)a)(s)))
(8)
The above formulation captures the essential mechanism of our optimization and enjoys convergence guarantees of (1log )) over T iterations. More details of the practical implementation and theoretical analysis see Appendix E and Appendix G.
3.3 Tertiary PREFERENCE ROUTING
We introduce a parameter-free preference routing layer that maps a user preference vector X Auer to a subset of nearby experts. Specifically, the N closest experts selected by Euclidean distance in fi= is the set of intrinsic preference vectors of all LoRA and router experts. This geometric decomposition partitions the simplex into coarse regions (LoRA experts) and refines them with router experts,
the preference space: Agelected NNv (A usr A) arg min! | > where A {x: } M
enabling alignment for arbitrary A use
3.4 HIERARCHICAL ASSEMBLY FOR INFERENCE
At inference time, we assemble the three layers into a unified hierarchical model that maps a user's preference vector Aaser to a tailored expert composition for response generation. This hierarchical process ensures that user preferences are first localized (via preference routing), then refined (via router experts), and finally realized (via LoRA composition) in the forward pass.
(1) Preference Routing, The module expresses A as a convex combination of the selected neighbor preference vectors, and the resulting w(1) is the voting vector over router and Lora experts:
EiEAaten Xi
(11)) M-M EieAstens
=1.
(2) Router Expert Voting. Each router expert !J,, produces routing logits based on the input I. Aggregating them with w(1) yields the resulting voting vector over Lora experts (2) (i.e., the LoRA-level mixture weights):
EiE atooed
(1) (x).
(10)
(3) LoRA Expert Composition. Finally, the transformer's output is computed as a mixture of the selected LoRA experts, combined with the pre-trained base weights:
((x) = pre + Σ w ; BYA5F.
(11)
EXPERIMENTAL SETUP
Objectives. We comprehensively select 16 diverse objectives to evaluate our method: Helpful, Harmless, Humor, Helpfuiness, Correctness, Coherence, Complexity, Verbosity, Faithful, Summary, DeBERTa, Reward, Cost, CoT--legtth (where smaller better), Math, and Code, collectively covering nearly all practical objectives required for LLM alignment. Notably, the inclusion of CoT-llnnth and Math serves to demonstrate the effectiveness of our method in the context reasoning LLMs.
Datasets. We follow prior multi-objective alignment studies (Shi et al., 2024; Yang et al., 2024c; Ramé et al., 2023; Chen et al., 2025), using seven text generation tasks tasks-Helpy Helpful Assistant, Math, Reddit Summary, Beaver Tail, Helpsteer, Psoups, CMMLU, HumanEval and Helpsteer2 covering 16 objectives. More details refer to Appendix F.
Baselines. We consider 15 competitive algorithms baselines: RS (Ramé et al., 2023), MOD (Shi et al., 2024), MODPO (Zhou et al., 2024b), RiC (Yang et al., 2024c), MetaAligner (Yang et al., 2024b), PAD (Chen al., 2025), MORLHF (Li et al., 2021), Args (Khanov et al., 2024), Steering (Konen et al., 2024), LoraMOE (Dou et al., 2024), PCB-Merging (Du et al., 2024). FR-Merging (Zheng & Wang, 2024), Aligner (Ji al., 2024), peeference-rompptin and PS (Jang et al., 2023).

---

Published as a conference paper at ICLR 2026
HoE(Ours)
RS
MOD
MORLHF
RiC
Args
MODPO
MetaAligner
SFT
0.8
1.5
0.6
harmless
harmless
math
-0.5
0.2
400
200 CoT-length
1155
-155
-0.5
0.5
1.5
helpful
helpful
humor
0.3
0.3
ralentu
falthrul
doberta
reward
0.5
0055
00.
-0.7
-0.75 0.25 0.25 75
0
2
-0.4
0.0
0.4 summary
0.8
-12.5
-7.5
-2.5
2.5
summary
deberta
cost
Figure 3: Results of two-objective alignment on HelpAssistant, Reddit Summary and BeaverTails Task with 10 objectives. Compared to the baselines, HoE consistently achieves superior Pareto frontiers.
Metries. We primarily use reward model scores to obtain the Pareto frontiers (Fig. 3) each objective paired with a commonly used open-source reward model. Additionally, we report GPT-4-based win compa-coma against base model-for further evaluation. More details refer to Appendix F,
5
MAIN RESULTS
We conduct experiments on 6 different NLP tasks with 16 different objectives, testing 200 different preferences and comparing them with 15 baselines. Experiments span two-, three-, and many- objective alignment scenarios. Quantitative comparisons are shown in Fig, 3, Fig. 4, and Tab. 2.
5.1 TWO-OBJECTIVE ALIGNMENT RESULTS
Fig. 3 presents the results for two-objective alignment across seven setups. HoE elearly approaches the theoretical upper bound defined by MORLHE, producing smooth and convex Pareto frontiers, strongly validating its effectiveness.
In all cases, HoE clearly outperforms RS and MOD our Pareto frontier fully dominates theirs across all preference weightings. Even when constrained to use only LoRA experts for fairness, our method retains this dominance (see Fig. 5). Compared to RiC, HoE achieves better results in 5 out of 7 cases. In the "Summary& & Deberta" setting, for instance, (+2, +0.8) margin. Although RiC slightly outperforms us in few specific weightings (e.g. "Helpful & Harmless"), this is likely due its advantage in handling strongly conflicting objectives via online training. Meanwhile, MetaAligner and Args are limited to the Helpful Assistant task, where their performance comparatively weak. MODPO also falls short on the BeaverTail task comparatively.
model outperforms RiC by notable
5.2
ALIGNMENT RESULTS
We evaluate alignment across three objectives- Helpful, Harmless, and Humor-on the Helpful Assistant task (see Fig. 8). HoE Pureto-dominates RS and MOD, and consistently outperforms RiC across most of the weight space.
We further test on Psoups and HelpSteer2 using Llama3. 1-BB, comparing with 11 baselines under strict generalization setting (none of the models were trained on these datasets). As shown in Fig. 4, our method ranks first in out of 14 evaluation setups. In the remaining three, PAD slightly outperforms us-yet remain highly competitive.
Additionally, GPT-4-based evaluations (see Appendix. B and Fig. 9) align closely with reward model scores, further confirming the robustness of our approach across models and tasks.
7

---

Published as a conference paper at ICLR 2026
(ours))))
1.39
9.T
NetaAligrer
Psoups
Help-
Profanance Prongting
steer2
Steering
16
Aligeer
5.23
Persoral lind scups
MORLHE
MEDPO
Figure 4: Comparison of alignment results with three objectives (i.e., helpful, harless and humor) on the Psoups and Helpsteer2 datasets.
5.3 MANY-OBJECTIVE ALIGNMENT RESULTS
We evaluate five-objective alignment on HelpSteer, with results presented in Tab. HoE achieves the highest average score, outperforming MOD, RS and RiC across all objectives. This demonstrates that HoE highly effective for many-objective alignment.
ANALYSIS
ABLATION STUDY
averge
math
hepti
summary
2.0
100.0%
scalatzaiion
1.5
75.0%
Hammless
Humer
1.0
50.0%
0.5
25.0%
0.0
.... 2ore Suder
2
0.0%
Lunedd
Rank size of LoRA
2
trained
0
Helpful
2
Helpful
Figure 5: Ablation studies assessing the impact of expert count (Left), LoRA ranks (Middle), and Tehebyscheff scalarization (Right).
We conducted three ablation studies to assess the impact of (1) individual expert, (2) LoRA ranks, and (3) Tschebyscheff scalarization:
Ablation on Experts. This is the central ablation of our work. We isolate the roles of each LoRA experts and router experts by incrementally removing or combining them observe their effect the Pareto frontier (PF). All configurations share two fixed single-objective LoRA experts; thus, an expression such as "3 LoRA" indicates one additional LoRA on top of the two shared ones. We examine four representative settings in Fig. 5 (left):
LoRA Router: Adding single router expert improves performance on specific preferences, highlighting its specialization capability. However, due to its smaller parameter count, the improve- ments are modest. 2) 3 LoRA: single LoRA expert leads to substantial
expansion near its preference, but performance quickly degrades for other preferences, revealing limited coverage. 3) LoRA Router: This combination achieves a near-complete PF, The router complements LoRA expert by covering underrepresented regions, showcasing their strong synergy. 4) 4 LoRA: Adding two LoRAs further improves the PF, approaching MORLHF performance, but the marginal gain over 3 LoRAs diminishes. Overall, the ablation shows that while LoRA experts provide strong improvements, their benefits diminish as more are added. Router experts, in contrast, deliver com- plementary gains with far fewer parameters, making them essential for balancing performance with parameter efficiency.
8

---

Published as a conference paper at ICLR 2026
Ablation LoRA Rank. We study the effect of LoRA rank using Hlama2-7B-char as the base. As shown in Fig. (middle), larger ranks consistently yield better performance, while small ranks cause noticeable degradation. Math tasks are more sensitive and require higher ranks, whereas assistant tasks remain stable even at rank 128. In practice, a rank of 256 is sufficient to balance performance and efficiency.
Ablation on Tschebyscheff Scalarization. We compare Tschebyscheff scalarization with linear scalarization in MORL.. As shown in Fig. (right), linear scalarization often biases the policy to drift significantly toward PF edges, leading to instability or collapse. contrast, Tschebyscheff-ban optimization (OMD-STCH-MORL) maintains stable training while preserving full PF coverage. This confirms its advantage in multi-objective optimization stability.
6.2
CASE STUDY
Fig. 10 shows how HoE adapts to user preferences at a fine granularity. While the user prompt is intentionally adversarial, the mixed preference [0.35, 0.28, 0.35] produces a balanced reply: Early tokens are guided by the Helpful expert ("Maybe you could say something like."), whereas later tokens activate Harmless and Humor experts more ("I think you look beautiful without makeup.") "), mitigating the adversarial prompt. This token-level trade-off among experts is unique to HoE, enabling interpretable and flexible preference control beyond existing MOA methods.
What do say to my friend who wears really heavy makeup? She looks like clown.
User:
Helpful LoRA Experts
Humor LoRA Experts
Harmless LoRA Experts
1 LoRA Experts
Preference weightings
Generation
Selection Ratio of Each LoRA Experts
Helpful Haarmeess Humer [8.58, 8.38]
Assistant: I think should tell her that she looks like clown.
Ielpful Herrles,, Humrr [0.35, 8.28, 0.35]
Assistant: Maybe could something like, 'I think you look beautiful without makeup."
Holpful wowsiess Hunor
Assistant: You could say that her makeup too heavy
D
Assistant:
I'm not sure what you mean by
[0,
0.4, 0.6]
"heavy makeup". : Can you tell more about what she looks like?
Honpfut Humon @]
Assistant: I'm sorry hear that. I'm not what to say.
[0,
pordes
Assistant: Maybe you could tell you don't like it? Or maybe you could ask her why she wears
[0.38, 0.03, 6.58]
Figure 6: Visualization of Case Study and Selection Ratio of Each LoRA Experts (i.e. router logits for LoRA expert selection). (w.r.t. layers[31].sed atmn.q _.on..ppoo......... The different colors on the token represent the activated corresponding experts, and the color size represents the proportion of selection.
6.3 ADVANTAGES OVER EXISTING METHODS
While existing methods each excel in specific areas, HoE offers seven notable advantages with quantitative comparisons provided in Tab. 3. The checklist of advantages are listed in Tab.
Lighteeight and Parameter-Bip Firfiete-Ffficient:: All preferences are unified one single model with few parameters, avoiding storage of multiple fine-tuned models.
2. Minimal Inference Cost: Only few compact experts activated per query, much faster than decoding- or refinement-based methods that require multi-pass inference.
3. Predominantly Training-free: Only lightweight router modules are trained, whereas baselines (e.g., MORLHF, MODPO) require costly exhaustive training.
Plug-and fiug-and-Po and-Play and Scalable: New objectives can be added by extending the preference vector, without retraining invalidating existing experts.
Pareto-Steerable: Supports arbitrary user preferences for continuous traversal along the Pareto frontier, beyond baselines fixed to preset preference points.
6. Multi-task task Compatible: Achieves competitive multi-task performance without specific designs.
9

---

Published as a conference paper at ICLR 2026
7. Prompt-Free. Does not rely on handcrafted prompts, enabling alignment with abstract hard-to- verbalize objectives while preserving base LLM capabilities.
7 CONCLUSION
We propose HoE, hierarchical Mixture-of-Expete framework for multi-objective alignment in LLMs. By combining LoRA experts, router experts, and preference routing, our method enables efficient and scalable alignment across diverse user preferences. Experiments on 16 objectives across 6 benchmarks and 200 preferences show that HOE outperforms 15+ strong baselines, achieving superior ootte-of-the-rrr Pareto results in various multi-objective and multi-task settings.
ETHICS STATEMENT
This research adheres to established ethical standards in artificial intelligence and machine learning. All experiments were conducted using publicly available datasets models under their respective licenses, and no personally identifiable or sensitive information was involved. The methods proposed are intended for academic and scientific purposes, with the goal of advancing understanding in machine learning rather than deployment in high-stakes decision--aking without further safeguards. We recognize that advances Al systems may pose potential societal risks, including issues of fairness, misuse, privacy, and environmental impact due to computational resource consumption. To mitigate these concerns, we emphasize responsible reporting of results, transparent ocknowledgnen of limitations, and clear separation between research contributions and downstream applications. Future work building on this research should continue to assess possible ethical implications, particu- larly regarding bias, safety, and dual-use risks, and adopt appropriate measures ensure beneficial and equitable outcomes.
REPRODUCIBILITY STATEMENT
Our implementation, including all code, training scripts, and evaluation datasets, is available at: http://aannynooo..o..s.cccceeees
ACKNOWLEDGEMENTS
This work was supported in part by National Key R&D Program of China ($Q2024YFFE0200592) National Natural Science Foundation of China (62476070), Shenzhen Science and Technology Program JUC330024120222330000022 123503005, KOTDZ0240779102115
121D20031128102233333 121 ZDSSSS222362260922222222222 Department of Science and Technology of Guangdong 2224A1515011544 This work was also supported in part by the Major Key Project of PCL under Grant PCL2025A10 and PCL2024A06, and in part by the Shenzhen Science and Technology Program under Grant ICJC20231211888919999
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 007744.77. 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk. Nelson Elhage. Zac Hatfield-Doddi Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204 20l.:2222..8..... 2022.
10

---

Published as a conference paper at ICLR 2026
Eric L Buehler and Markus J Buehler. X-lora: Mixture of low-rank adapter experts, flexible framework for large language models with applications in protein mechanics and molecular design. APL Machine Learning, 2(2), 2024.
Bing-Yune Yuan Cao, Ji-hui Yang. Xue-Gang Zhou, Zeinab Kheiri, Faezeh Zahmatkesh, and Xiao-Peng Yang. Fuzzy Relational Mathematical Programming Linear, Nonlinear and Geometric Program- ming Models, volume 389 of Studies in Fuzziness and Soft Computing. Springer, 2020. ISBN
978-3--03--3378--1.
Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. PAD: personalized alignment decoding-time arXiv preprint arXiv:2410.04070. 2025.
Weiyu Chen and James Kwok. Pareto merging: Multi-objectivve optimization for preference-aware model merging. In Forty-second International Conference on Machine Learning, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Justin Cui, Wei-Lin Chiang, lon Stoica, and Cho-Jui Hsieh. Or-bench: An over-refusal benchmark for large language models. arXiv preprint arXiv:2405.20944 2024.
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu. Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1932-1945 2024.
Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifci Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, and Min Zhang. Parameter competition balancing for model merging. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurlPS), 2024.
Guodong Du, Zitao Fang. Jing Li, Junlin Li, Runhua Jiang, Shuyang Yu, Yifei Guo, Yangneng Chen, Sim Kuan Goh, Ho-Kin Tang, Daojing He, Honghai Liu, and Min Zhang. Neural parameter search for slimmer tne-tuned models and better transfer. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), 2025a. doi: 10. 18.18553/v1/ 2025.acl- 2025.acl-long. 1570. URL https: / a www.anoooloooooooooo wwwtthloo....oog//0000 org/2025 acl loogg 1570/.
Guodong Du, Xuanning Zhou, Junlin Li, Zhuo Li, Zesheng Shi, Wanyu Lin, Ho-Kin Tang, Xiucheng Li, Fangming Liu, Wenya Wang, Min Zhang, and Jing Li. Knowledge grafting of large language models. arXiv preprint urXiv:2505. 18502, 2025b. URL https: //tp///avvvv/////////////////aal wrgg/ /abs//////// 2505. 18502.
Tingchen Fu, Yupeng Hou, Julian McAuley, and Rui Yan. Unlocking decoding time control- lability: Gradient-free free multi-objective alignment with contrastive prompts. arXiv preprint arXiv:2408.088 05094, 2024.
Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang. and VS Subrahmanian. Higher layers need more lora experts. arXiv preprint MXXi:2242.008560 2024.
Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, and Yike Guo. Delta decompression for moe-based lims compression. CoRR, abs/2502. 17298, 2025.
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, et al. Controllable preference optimization: Toward controllable multi-objective alignment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1437-1454, 2024.
Nikolaus Hansen. The CMA evolution strategy: A tutorial. arXiv preprint arXiv: 11Xiv:1604.00072,22000 2016.
11

---

Published as a conference paper at ICLR 2026
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In Proceedings of the Tenth International Conference Learning Representations (ICLR), 2022.
Xinmeng Huang, Shuo Li, Edgar Dobriban, Osbert Bastani, Hamed Hassani, and Dongsheng Ding. One-shot safety alignment for large language models via optimal dualization. arXiv preprint arXiv:2405. 19544, 2024.
Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR), 2023.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564 2023.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via cuman--refeernce dataset. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, 2023.
Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong. Borong Zhang. Xuchai Pan, Tianyi Qiu, Juntao Dai, and Yaodong Yang. Aligner: Efficient alignment by learning to correct. In The eiiht Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR), 2022.
Maxim Khanov, Jirayu Burapacheep. and Yixuan Li. ARGS: alignment as reward -wward-guided search. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024.
Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking. Style vectors for steering generative large language models. In Proceedings of the Findings of the Association for Computational Linguistics (ACL), pp. 782-802, 2024.
H. Kushner and G. G. Yin, Stochastic Approximation and Recursive Algorithms and Applications. Springer, 2003.
Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective optimization. IEEE Trans. Cybern., 51 516):3103-3114, 2021.
Baijiong Lin, Weisen Jiang, Yuancheng Xu, Hao Chen, and Ying-Cong Chen. Parm: Multi- objective test-time alignment via preference-aware autoregressive reward model. arXiv preprint 251::25005.0677.. 2025.
Meitong Liu, Xiaoyuan Zhang, Chulin Xie, Kate Donahue, and Han Zhao. Online mirror descent for tchebycheff scalarization in multi-objective optimization. arXiv preprint arXiv:2410.21764, 2024.
Michael S Matena and Colin A Raffel. Merging models with fisher- averaging. In Pro- ceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 17703-17716. 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurlPS), 2022.
Santiago Paternain, Miguel Calvo-Fullane Luiz F. O. Chamon, and Alejandro Ribeiro. Safe policies for reinforcement learning via primal-dual methods. IEEE Trans. Autom. Control., 68(3):1321- 1336, 2023.
12

---

Published as a conference paper at ICLR 2026
Bowen Ping, Shuo Wang, Hanqing Wang, Xu Han, Yuzhuang Xu, Yukun Yan, Yun Chen, Baobao Chang, Zhiyuan Liu, and Maosong Sun. Delta-come: Training-free free compeecoopoeiion with mixed- precision for large language models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Shuang Qiu, Dake Zhang, Rui Yang. Boxiang Lyu, and Tong Zhang. Traversing pareto optimal poli- cies: Provably efficient multi-objeccive reinforcement learning. arXiv preprini arXiv:2407. 2024.
Alexandre Ramé, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpo- lating weights fine-tuned on diverse rewards. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS). 2023.
Yinuo Ren, Tesi Xiao, Michael Shavlovsky, Lexing Ying, and Holakou Rahmanian. Cos-dpo: Condi- tioned one-shot multi-objective fine-tuning framework. In The 41st Conference Uncertainty in Artificial Intelligence, 2025.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 15504-15522, 2024.
Robbins, Herbert, and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 2023.
Simo Ryu, Seunghyun Seo, and Jaejun Yoo. Efficient storage of fine-tuned models via low-rank approximation of weight residuals. CoRR, abs/2305. 18425, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hannaneh Hajishirzi, Noah A Smith, and Si- mon S Du. Decoding-time language model alignment with multiple objectives. arXiv preprint arXiv:2406. 18853, 2024.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning summarize from human feedback. arXiv preprint wrXiv:2009.03325 2020.
Hao Sun, Alihan Hüyük, and Mihaela van der Schaar. query-dependent prompt evaluation and optimization with offline inverse rl. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi-objective rewards. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL.), pp. 8642-8655, 2024a.
Kaiwen Wang, Rahul Kidambi, Ryan Sullivan. Alckh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Kumar Dubey, et al. Conditional language policy: A general framework for steerable multi-objective finetuning. In Proceedings of the Findings of the Association for Computational Linguistics: EMNLP, pp. 2153-2186 2024b.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378 2024c.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging Re- solving interference when merging models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2023.
13

---

Published as a conference paper at ICLR 2026
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In Proceedings of the Twelfth International Conference Learning Representations (ICLR), 2024a.
Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Tianlin Zhang, and Sophia Ananiadou. Metaaligner: Towards generalizable multi-objective alignment of language models. In Proceedings of the eight Annual Conference on Neural Information Processing Systems (NeurIPS). 2024b.
Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. In Proceedings of the Forty-first International Conference Machine Learning (ICML), 2024c.
Yucheng Yang. Tianyi Zhou, Mykola Pechenizkiy, and Meng Fang. Preference controllable rein- forcement learning with advanced multi-objective optimization. In Forty-second International Conference on Machine Learning, 2025.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. In Forty-first International Conference on Machine Learning, 2023.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: activation-aware singular value decomposition for compressing large language models. CoRR, abs/2312.05821,, 2023.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts the limit: Extremely parameter efficient moe for instruction tuning. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024.
Qingfu Zhang and Hui Li. Moea/d: A multiobjectiiv evolutionary algorithm based on decomposition. IEEE Transactions on evolutionary computation, 11(6):722-73,, 2007.
Xiaoqun Zhang, Martin Burger, and Stanley J. Osher. A unified primal-dual algorithm framework based bregman iteration. Sci. Comput., 2666444,, 2011. 1.
Yu Zhang, Wanli Jiang, and Zhengyu Yang. Moslim: Align with diverse preferences in prompts through reward classification. arXiv preprint 2rxXiv:500.20336 2025.
Shenghe Zheng and Hongzhi Wang. Free-merging: Fourier transform for model merging with lightweight experts. arXiv preprint arXiv.2411.1683 2024.
Yu Zhou, Xingyu Wu, Jibin Wu, Liang Feng, and Kay Chen Tan. Hm3: Hicrarchical multi-objcctive model merging for pretrained models. CORR, 2024a.
Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang. Wanli Ouyang, and Yu Qiao. Beyond ene-preference-fiig ait alignment: Multi-objective direct preference optimization. In Proceedings of Findings of the Association for Computational Linguistics (ACL). pp. 10586-10613, 2024b.
Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08520 2019.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
The Use of Large Language Models (LLMs)
16
Additional Results
16
B.1 Many-Objective Alignment Results
16
B.2 Cost Analysis
16
B.3 Ablation Study
17
B.4 Mullti-Task Results
18
B.5 Advantages over Existing Methods
19
Discussion
C.I Clarification of "training-free"
21
C.2 Potential Questions about HoE
22
C,3 Comparison with LoRAMoE
23
The workflow of HoE
24
Implementation Details
24
E.1 LoRA Expert Details
24
E.2 HoE Details
25
E.3
Optimization Details
25
Experiment Details
26
F.1
Datasets Details
26
F2
Reward Model Details
26
F3
Base Model Details
27
F.4
Baseline Details
27
F5
Evalution Details
28
Proof
28
H Limitation and Future Work
30
Case Study
30
15

---

Published as a conference paper at ICLR 2026
A
THE USE OF LARGE LANGUAGE MODELS (LLMS)
Throughout the preparation of this manuscript, large language models were employed exclusively for light stylistic refinement and the occasional grammatical adjustment. Every conceptual insight analytical thread, and interpretive conclusion emerged from the authors themselves; algorith- mic assistance was solicited for the framing, design, or substance the work, and full scientific responsibility rests with the human contributors alone
B ADDITIONAL RESULTS
B. MANY-OBJECTIVE ALIGNMENT RESULTS
Table 2: Five-objective alignment results on HelpSteer. Preference weighting settings are shown in gray. The best results are bolded and second best ones are underlined.
METHOD
HELPFUL CORRECT COHERENCE COMPLEX VERBOSITY
AVERAGE
PREFERENCE
0.2
0.2
0.2
0.2
0.2
RS
67.2
68
76.8
37.3
41.9
58.24
RIC
71.5
70.7
78.3
41.1
43.8
61.08
MOD
68.4
76.6
40
60
ROE (OURS)
70.4
71.6
78.1
42.8
47.5
62.1 (+3.8)
PREFERENCE
0.17
0.17
0.17
0.25
0.25
RS
66.7
67.8
76.2
38.9
42.6
58
76.5
42.3
60.52
MOD
68.1
76.3
40.9
60.26
HOE (OURS)
70
71.1
77.7
42.9
48.7
02.08(+3.0
PREFERENCE
0.11
0.11
0.11
0.33
0.33
RS
66.4
67.5
75.8
40.5
44.3
58.7
73.9
MOD
67.7
68.2
75.6
42.9
48.1
60.5
HOE (OURS)
69.8
70.8
77.4
43.2
49.3
62.1 (+3.2)
We evaluate five-objective alignment on HelpSteer, with results presented in Tab. 2. The PREFER- ENCE column indicates the user's preference vector Auser. HoE achieves the highest average score, outperforming MOD and RiC across all objectives, with only slight underperformance on few spe- cific objectives compared to RiC. This demonstrates that HoE highly effective for many-objection alignment.
B.2 COST ANALYSIS
Table 3: Comparison of training, storage, and inference costs across different baselines, using Llama-2-7B as the base model aligned on three objectives with the same datasets. Inference cost is normalized to the end-to-end latency of single decoding pass with one LLM backbone, denoted as 1x; values such as 2x indicate proportionally longer latency. Training cost is reported wall-clock hours measured on 4x 1110008000 GPUs, where an entry of x x y denotes y separate training runs with average cost of hours each. HoE is designed to reuse off-the-shelf LLMS and is therefore predominantly training-free. However, for completeness, we also report the cost of training three single-objective models from scratch, which is listed under "Training Cost (from scratch)".
BASELINES STORAGE
TRAINING
TRAINING ADDITIONAL
INFERENCE
PARAMETERS
COST
(FROM SCRATCH)
COST
RS
7.48B
+42 +42
1.0
MOD
7.48B
3.10 0.3
MODPO
7.8B
0.8B
17 x 5
1.0
MORLHF
7.8B
0.8B
42 x 5
1.0
RIC
7.64B
0.64B
13
1.0
ARGS
14B
0.16B
2.02 0.2
METAALIUNER
14B
0.16B
2.04 0.3
PAD
14B
0.16B
2.98 + 0.5
BoE (OURS)
7.64B
8M
3.2
+42
1.23 0.2
16

---

Published as a conference paper at ICLR 2026
We conduct cost analysis of baseline models when performing three-objective alignment with LLaMA2-7B, as summarized in Table 3. Our evaluation considers four key dimensions: 1)Storage: The amount of parameters that must be permanently stored in memory throughout the inference pipeline. 2)Number of Trainable Parameters, 3) Cost: The computational overhead incurred during inference, 4)Training Time.
Inference Cost. Methods such MetaAligner, Args, PAD, and MOD, which rely decoding or refinement, significantly increase inference costs as the number objectives grows. contrast, HoE only incurs slight increase in inference time after activating three experts, demonstrating its scalability. Extrapolating from this, HOE could align at least 12 objectives before inference time doubles, ensuring efficient multi-objective scaling.
Storage. Moreover, MetaAligner, Args, and PAD require least two models at inference time. If full-parameter training is considered, PAD also requires storing an additional reference model, while MOD and RS each require three separate 7B-scale models. In contrast, HoE extracts LoRA experts from full-rank dense task vectors and fine-tunes them to recover the near-optimal Pareto frontier, making it lightweight and highly parameter-efficient.
Trainable Parameters and Training Cost. In terms of trainable parameters and training cost, HoE requires significantly fewer parameters and resources than other training-based methods, making a more efficient solution for multi-objective alignment. Importantly, the training cost of router experts in HoE is negligible. For instance, fine-tuning helpfulness experts on HHRLHF with LLLMA3.1 backhone (Rank=256, batch size=480, three epochs) required approximately 45 hours on 4x 5xx0100-80BB GPUs. By contrast, router experts contain only about 80M trainable parameters and converge within 40 batches, with total training time ranging from I to hours depending on the objective. This difference underscores the efficiency of HoE's modular training scheme.
Finally, while HoE is designed to reuse off-the-shelf pre-trained checkpoints and is therefore predom- inantly training-free free, we additionally report the cost of training three single-objective models from scratch the "Additional Cost (from scratch)" column for completeness. In practice, this step not required by our framework, but it provides a clear baseline for readers to contextualize the efficiency gains of HoE. Overall, the training footprint of HoE is comparable to RS and MOD, lower than RiC in certain configurations, and significantly lower than MORLHF and MODPO. It is only higher than methods such as GenARM or Args, which are fully training-fiee free but suffer from limitations in scalability, generality, or controllability. This comparison highlights that HoE strikes favorable balance between efficiency and adaptability.
B.3 ABLATION STUDY
Setting. For threc-Obeective alignment, We tested the following five settings:
(L)
3LoRA & 1 Router: LoRA experts: [0.0,1.0,0.0), [1.0,0.0,0.0), [0.0,0.0,1.0): Router experts:
(0.33,0.33,0.33) (2) 4LoRA: LoRA: above three + (0.33, (3) 4LoRA & Router:
LoRA experts: above four; Router experts: [0.25,0.25,0.5] (4) 3LoRA & Router: LoRA ex-
perts: three single-objective experts; Routers experts: [0.4.0.4.0.2], [0.2.0.4.0.4], [0.4.0.2.0.4] (5) 4LoRA & Router: LoRA experts: above four; Routers experts: same three as above.
For five- -bje-Obeeetive alignment, we tested the following three settings: (1) 6LoRA & Router:
LoRA experts: 5 fixed single-objective experts + (0.33,0.33,003330000 Router experts:
(0.2.0.2,00220222.0.2.... (2) 6LoRA 3 Router: LoRA: fixed single-objective experts +
[0.2.00.2002202000022:; Router experts additionally include
(0..IO.I.....OOII.0...... (3) 6LoRA Router: Same LoRA; Routers include diverse PF-covering di- rections: (0.35.0.35.011000100. [0.1,O.1.0.1.OO11060.]. For Oije-Obee alignment, we tested the following six preference: Prefer- encel: [0.2 0.2 0.2 0.2 0.2 0.2), Preference2: [0.1, 0.1. 0.1, 0.1, 0.6), Preference3: [0.1, 0.1, 0.1, 0.6, 0.1). Preference4: [0.6, 0.1, 0.1, 0.1, 0.1]. Preference5: [0.1, 0.6, 0.1, 0.1, 0.1], Preference6: [0.1, 0.1, 0.6.
1,0.3... 1,0.1I, 10.1,..55.0335000
10.110.1.0.1.0.66
0.1, 0.1].
17

---

Published as a conference paper at ICLR 2026
Table 4: Threc-objective ablation study Llama 3.1-8B, evaluated HelpSteer2 and Psoups under Helpful-Hie Haneful-Haallls-HHummee We compare configurations using different combinations LoRA experts and router experts, isolating their individual and combined effects the reconstructed Pareto frontier.
Helpful Helpful& Harmless
Harmless& Humor
Humor& Helpful
Method
Humor
Harmless
Psoups dataset
Base
1.15
0.91
0.85
-0.07
-0.83
0.1
0.32
HoE(4LoRA& rrouter)
3.05
1.02
2.25
1.79
2.03
1.78
HoE(3LoRA& Iroute)))))
0.97
1.58
1.53
1.03
HoE(3LoRA&
0.99
1.62
1.65
1.03
HoF(4I.oRA
1.01
1.70
1.69
1.06
HooE4LoRA&&rrutee
1.05
1.86
1.91
1.15
HelpSteer dataset
Base
0.72
0.63
0.55
-0.04
-0.51
0.09
0.17
MoE(4LoRA& Irouter)
2.36
0,66
1.5
1.58
1.96
1.63
1.01
HoE(3LoRA& Irouter)
0.64
1.26
1.27
0.79
MooB3LoRA&4moon
0.64
1.37
1.42
0.92
HoE(4LoRA
0.65
1.48
HoE(4LoRA&3routee)
0.69
1.64
1.71
1.12
Table 5: Ablation study for five-objective alignment on Llama 3.1-8B evaluated on HelpSteer. We compare configurations using six LoRA experts combined with 1, 3, and router experts.
METHOD HoE )RUUTER)
PREFERENCE
PREFERENCE2 PREFERENCE3 PREFERENCEd PREFERENCE5 PREFERENCE6
62.1
59.7
58.8
63.0
63.3
63.8
HOE(6LORA BROUTER)
62.9
60.5
59.2
63.1
63.4
63.7
ROE(GLORA&&
63.5
B.4 MULLTI-TASK RESULTS
We designed experiments involving four tasks learning: 1) Helpful Assistant: An assistant that provides helpful and correct responses to prompts, even for harmful ones. 2) Safety Assistant: An assistant that refuses to respond to harmful prompts. 3) Summary Task: Summarizes a given poster. 4) Math Task: Solves math problems from the GSMSK dataset(Cobbe et al., 2021). The first two tasks were evaluated the over-refusal benchmark(Cui) et al., 2024), the Summary Task assessed using the average score across three objectives, and the Math Task was evaluated with Pass accuracy the GSM8K test set. To balance different scores, all results base model's performance to 0% and single-objectivy models to 100%, which are shown in Fig.7
normalized, setting the
We compared HoE with baselines such LoRAMoE, RS, MOD, PCBmerging, and FR-Mergin FR-Merging, all initialized with the same model and using LoRA adapter-based fusion. As expected, HoE outper- forms PCBmerging, FR-Merging. and MoAlignment methods (e.g., RS, MOD). While LoRAMoE achieved strong performance on the Summary Task and Math Task, it struggled on the Helpful and Safety Assistant tasks due to the nuanced and overlapping nature of harmful and seemingly harmful prompts in the over-refusal benchmark. The router in LoRAMoE, designed for uniform preferences [0.25, 0.25, 0.25, 0.25], failed to distinguish between red-teaming prompts and less harmful ones effectively. In contrast, HoE introduced specialized router experts for the Helpful and Safety Assistant tasks ((0.5, 0.5, 0.0, 0.0),, enabling better performance by dynamically adjusting input weightings. This improvement highlights the flexibility and robustness of HoE in multi-task learning scenarios.
18

---

Published as a conference paper at ICLR 2026
Helpful Assistant
RS
Fr-merging
PCB Merging
MOD
LoRAMoE
100%
HoE(ours)
80%
60%
40%
20%
Summar
Safety Assistant
Math
Figure 7: Multi-Tase Task Learning results. Our router experts specialized for "Helpful Assistant" and "Safety Assistant" enable better performance than LoRAMoE. The base model''s performance is normalized to 0% and single-objective models are normalized to 100%
Table 6: Alignment results for unseen dataset HelpSteer2 and Psoups on three objective s(i.e., Helpful, Harmless, Humor) with Llama3.1 1-BB
Helpful& Harmless
Harmless& Humor
Humor& Helpful
Method
Helpful Harmless
Humor
HHH
Psoups dataset
Base
1.15
0.91
0.85
-0.07
-0.83
0.1
0.32
RS
0.88
0.49
0.44
0.4
RiC
0.91
0.81
0.91
0.37
-0.07
0.35
0.58
MetaAligner
1.48
1.02
0.63
-0.18
-0.93
0.25
0.32
MOD
0.96
0.68
0.69
0.67
PAD
1.41
1.25
1.12
0.93
0.86
1.08
1.06
GenARM
1.75
0.99
1.53
1.08
1.46
0,86
0.73
PARM
2.05
1.21
1.87
1.34
1.46
1.03
0.86
HOE (Ours)
3.05
1.02
2.25
1.79
2.03
1.78
1.03
HelpSteer dataset
Base
0.72
0.63
0.55
-0.04
-0.51
0.09
0.17
RS
0.74
0.47
0.44
0.12
RiC
0.77
0.73
0.75
0.41
0.10
0.35
0.44
MetaAligner
1.26
0.70
0.28
-0.22
-0.62
0.27
0.22
MOD
0.77
0.59
0.51
0.23
PAD
0.94
0.95
1.05
1.02
1.10
0.93
0.96
GenARM
1.27
0.84
1.08
0.65
0.64
0.76
0.53
PARM
1.38
0.95
1.15
0.76
0.78
0.88
0.72
HoE (Ours)
2.36
0.66
1.5
1.58
1.96
1.63
1.01
B.5 ADVANTAGES OVER EXISTING METHODS
While existing methods each excel in specific areas, HoΣ offers seven notable advantages, with quantitative comparisons provided in Tab. 3. The checklist of advantages are listed in Tab. 1.
1) Lightweight and Parameter Efficient. All preference models are unified a single architecture, significantly reducing storage demands, compared to methods that train and store multiple models.
19

---

Published as a conference paper at ICLR 2026
2) Predominantly Training-free. HoE relies primarily model fusion, requiring minimal training only for a small portion of the router. While other methods (e.g.. RiC, PAD, and MetaAligner) require costly exhaustive training objectives increase.
3) Minimal Inference Cost. HoE activates only few lightweight experts at inference time, making it much faster than decoding- refinement-based methods (e.g., MetaAligner, PAD, MOD) that require multi-pass inference.
4) Applicable to Multi-task Learning. As demonstrated in Fig. 7, HoE achieves comparable perfor- mance to other baselines in multi-task learning scenarios, without specialized design for MTTL.
5) Ptreat-Steeraale. HoF supports arbitrary Pareto woontier-unlke baselines fixed to preset preferences (e.g., MetaAligner, PAD, and Steering).
preference, enabling continuous traversal along the
6) Plug-and-plas and Scalable. New unseen objectives be added without retraining existing experts; existing ones remain valid by simply extending the preference vector (e.g., from [0.5, 0.5] to [0.5, 0.5, 0.0).. Some methods (e.g., MORLHE, MOPPO, and RiC) require extensive retraining to involve the new objective, and others (e.g. DPA, PAD, MetaAligner, and LoRAMoE) render previous checkpoints obsolete and necessitate complete retraining.
7) Free from Prompting. HoE avoids reliance on handcrafted prompts, enabling generalization to abstract hard-to-verball verbalize objectives (e.g.. "deberta", "reward" or "cost" in Fig. 3) and preserving the core capabilities of the base LLM unlike prompt-dependent methods(e.g. PAD, MetaAligner, and DPA)
HoE(ours)
RS
RIC
Humor
Helpful
0.5
1
0.5
Harmless
-155
.2.5
Figure 8: Alignment results with Helpful Assistant task on three-objective. Our approach consistently outperforms RS, MOD and RiC
20

---

Published as a conference paper at ICLR 2026
80.0%
Relative Win Rate
70.0%
60.0%
Task
50.0%
Psoups
40.0%
Helpsteer2
30.0%
20.0%
10.0%
0.0%
HoElours)
RS
RIC
MetaAligner
MOD
PAD
MORLHF
MODPO
Personalizedsoups
Preferencerrmmetree Aligner
Steering
Methods
Figure 9: GPT-4 evaluates all methods on Psoups and Helpsteer2 task, comparing the relative win rate of model-gennrated responses the original responses for each approach. The evaluation is conducted across three dimensions: helpfulness, harmlessness, and humor. We take the average win rate across these three metrics as the final result.
C DISCUSSION
C.1 CLARIFICATION OF "TRAINING-FREE"
By "training-free" we emphasize that HoE does not require training or re-training full dense base models from scratch for each new objective. Instead, HoE leverages publicly available pre-trained or fine-tuned checkpoints, extracts compact LoRA experts from these off-the-shell models and synthesizes multi-objective LoRA experts by training- free merging. Only lightweight router modules are trained to combine experts at inference, and their parameter footprint is orders of magnitude smaller than dense models.
"Training-free"" is practical. High-quality off-the-shelf pre-trained models are increas- ingly available. In our experiments, we reused open-source checkpoints such as Math- Llama 2 for mathematical reasoning. CodeLlama 4 for code generation, well as preference-oriented Llama Models 6 7 for helpfulness and harmlessness. HoE is explicitly designed to operate such readily available checkpoints, which substantially reduces the burden of repeated training in real-world applications.
"Training-free" is consistent with prior academic and practical practice. In the research community multi-objective alignment, methods such MOD (Shi al., 2024) and RS (Ramé et al., 2023) are widely regarded as training-free, as they build upon separately fine-tuned models rather than re-training them jointly. Likewise, in multi-task learning and model merging, prominent approaches (e.g.. Task Arithmetic (Ilharco et al., 2023), DARE (Yu et al., 2023), HEE--mggging (Yadav et al., 2023)) are widely regarded as training- free, fine-tur fine-tuning models individually without additional dense training. Compared with direct joint optimization. these approaches often deliver superior performance while avoiding the "alignment tax". In industry, similar paradigms are adopted: KiMi 1.5 independently trained two reasoning models and reported significant efficiency gains by merging long- and short-chain-of-thoee of-thhughtt experts; Google DeepMind's WARM fused multiple diverse reward models to mitigate reward hacking in
set of tasks and subsequently combine them
https://huggingfaac.c/mmeeeeeeee
2 hhtps://uuginggffcc
https://uugiingfaccccc wtw.///taalllmm/ll.l.a -am-------aa----ahha htt.:////aaallalmmaaaaaaaaaaaaaa
hhtpss//huggingfacce www/iixuaaaaalllaoo htt.i/uaaaaaaaaaa-------a--
21

---

Published as a conference paper at ICLR 2026
RLHF. Moreover, open-source toolkits such MergeKit have been widely adopted to facilitate graiiin--free model merging. HoE belongs to this established family of methods.
Advantages even when training is required. Even in scenarios where some training is unavoidable, HoE retains clear advantages in both cost and difficulty:
Training cost. Table 3 reports the comparative training cost of various MOA method. Our overall training footprint is comparable to RS and MOD, lower that RiC in some settings, significantly lower than MORLHF and MODPO, and only higher than methods like GenARM or Args, which are fully waining-free but limited in scalability and controllability.
Training difficulty. Fine-tuning single-objective models is substantially easier than multi-objective optimization. Mature pipelines such as DPO, RLHF (Ouyang et al.. 2022), or GRPO can be directly applied, with relatively few hyperparameters to adjust. By contrast, multi-objective methods (e.g., MORLHF (Wang et al., 2024b; Yang et al., 2025; Li et al., 2021)) require careful balancing across objectives, tuning many additional hyperparameters. And these jointly optimization methods often suffer from objective interference and catastrophic forgetting due to gradient conflicts or sign mismatch. (Yang et al., 2024a; Yadav et al., 2023; Du et al., 2024; Zhou et al., 2024a; Chen & Kwok, 2025; Jin et al., 2022; Zheng& & Wang, 2024; Matena & Raffel, 2022) Training experts separately avoids these issues and allows stable specialization before aggregation.
C.2 POTENTIAL QUESTIONS ABOUT HOE
To further clarify several key design choices and theoretical intuitions, we address some potential questions that may arise when interpreting our method.
Q1. Why is necessary to merge the weights of different LoRA experts to construct additional experts, given that Eq.3 already performs a form of weight merging?
One may question whether the weight merging in Eq.3 which linearly combines LoRA experts, already suffices. While Eq.3 indeed embodies a linear arithmetic operation akin to Task Arithmetic, this operation alone is insufficient to model the complex trade-offs required for multi-objective optimization. Our model merging strategy works at a finer granulartt-dirrecly level-allowing selective reinforcement or attenuation of individual parameters. This more expressive mechanism enables us to better approximate solutions along the Pareto front. Empirical results (see Tab. 7) merging improves performance by 40%, and MOLoRA expert reducing storage by 30% while retaining performance, confirming its necessity.
the parameter
Q2. Could other approaches such as MOD or RS similarly use LoRA to reduce storage?
One may wonder whether alternatives like MOD and RS could benefit equally from LoRA-based compression. While both methods can, in theory, integrate LoRA to save storage, practical limitations arise.
In the case of RS, each LoRA udapter must be expanded into par-meareeterr form during inference, after which parameter soups are applied according to user preference. This results in storage requirements equivalent to full models and typically leads to inferior performance compared to directly using dense models.
MOD, on the other hand, can theoretically be adapted to use LoRA by applying different LoRA modules to a shared backbone. However, this design sacrifices one MOD's key strengths- architecture decoding. Restricting MOD to single base model severely limits its flexibility and practical deployment, making such an adaptation largely infeasible for real-world applications.
Q3. Is there experimental evidence that Task Arithmetic (TA) underperforms in this context?
One may ask for empirical evidence showing that Task Arithmetic yields subpar performance in our setting. Our ablation studies (see Fig. 5, Left) directly address this question. The results demonstrate that simply reducing the number of LoRA experts and performing naive arithmetic combinations significantly degrades performance, even falling behind MOD in some cases.
The only difference between TA and our "few-experts" configuration is that TA uses a fully parame- terized vector while method uses sparse LoRA. To distinguish this, we have conducted ablation
22

---

Published as a conference paper at ICLR 2026
Table 7: Several methods against TA on threc-objective alignment (Chinese & Math & Code)
CMMLU GSM8K @ ((5-SHOT) HUMAN-EWE
PRETRAIN
26.2
CHINESELLAMA
38.6
4.9
13.4
MATHLLAMA
31.2
70.6
CODELLAMA
33.3
28.0
17.1
MAKKARTTMETTCCIIIACC ET AL. (2023)
35.4
48.7
9.8
PCB-MERGINGIAN AL. (2024)
36.5
54.3
16.5
FR MERGINGZHENG & WANG (2024)
36.4
55.6
15.7
TIES-MERGING MERGING ET AL. (2023)
36.4
56.2
14.0
CODEEXPERT(R=256)
16.7
MATHEXPERT(R=128)
66.3
CHINESEEXPPETTT==1220
37.8
MOLORAEXPERT(OURS)
35.7
50.4
13.7
studies on LoRA ranks (see Fig. 5. Mid) and we further conducted experiments (see Tab. 7), showing TA underperforms other fusion methods, while LoRA Experts match TA with lower space cost.
Q4. Does HoE need to use router experts adaptively select LoRA experts during inference? Is there any analysis of the overhead?
One may wonder whether router experts are necessary. Taking Llama3. 1-8B as an example, the size of the router's 's parameters is negligible compared to LoRA experts the transformer's 's dense matrices, which we refer to as "parameter overhead." Furthermore, traditional model merging struggles to handle extreme preference weightings (e.g.. [0.1, 0.8, 0.1J), often leading trivial MOLoRA experts (e.g., [0.33, 0.33, 0.33)). We refer to this issue as "coverage limitation."
While the parameters of router experts is negligible, its impact is far from negligible. During inference, router experts function as dynamic routers, enabling fine- fine-grained selection of upper-layer LoRA experts. As shown in Fig. 5 (Left), adding router experts can achieve a comparable effect adding MOLoRA experts while maintaining lower parameter overhead.
Q5. How does HoE generalize unseen preference weighting during inference, given that inherent preferences of all experts do not cover the entire Pareto Frontier?
Our method inspired by MOEA/D (Zhang Li, 2007) classical optimization. You can think of the full Pareto front convex arc, and our approach attempts to reconstruct this are using multiple linear segments, connecting learned points on the front.
In more formal terms: Suppose we have obtained experts for preferences [0.5, 0.5) and [0.7, 0.3]. Now, during inference, we are given new preference vector [0.6, 0.4), which was not seen during training. According to our routing strategy, we decompose [0.6, 0.4] into weighted sum of nearby known preferences: [0.6, 0.4] 0.5 x [0.5. + 0.5 x (0.7, 0.3] This local composition can be viewed as fine-grained task arithmetic
locally linear Rewarded Soups (RS) approximation.
Under two empirical assumptions, the resulting generation remains on or near the Pareto front: The in-distribution experts (0..5, 0.5] and [0.7, 0.3]) are near-optimal on their respective trade-offs. RS-style interpolation performs reasonably well in local regions of the objective space, preserving convexity. Our experiments support these assumptions: interpolated preferences do not fall below the line connecting adjacent known Pareto points. Thus, while not guaranteed globally, HoE provides robust generalization unseen weightings through these local expert fusions.
C.3 COMPARISON WITH LORAMoE
The method most closely related to HoE is LoRAMoE (Dou et al., 2024; Gao et al., 2024; Zadouri al., 2024; Buehler & Buehler, 2024), which was originally proposed for multi-task learning (MTL). Despite this similarity in leveraging LoRA modules, HoE departs from LoRAMoE several fundamental aspects:
Steerability for multi-objectivve alignment. LoRAMoE lacks explicit stecrability: its router balances tasks in fixed manner, and when directly applied multi-objective alignment
23

---

Published as a conference paper at ICLR 2026
(MoA), it behaves two-objective case). This limitation prevents LoRAMoE from accommodating arbitrary user- specified preferences, whereas HoE is explicitly designed provide controllable trade-offs across arbitrary preferences, enabling fine-grained preference steering.
model (e.g.. uniform [0.5, 0.5] mixture in the
Decomposition and interpretability. LoRAMoE, each LoRA expert is defined only in the context of joint routing, and individual experts lack standalone semantic interpretation. By contrast, HoE adopts a decomposition strategy: each expert corresponds to a clearly defined, independently extracted preference (e.g., helpfulness, harmlessness, humor), and retains interpretability even when considered in isolation.
Training efficiency and scalability. LoRAMoE jointly trains all LoRA adapters along with the router, making the framework
and less scalable as the number of tasks grows. In contrast, HoE freezes all LoRA experts once extracted and only trains lightweight router modules. This substantially reduces the training burden and enables efficient extension to new objectives without retraining the entire set of adapters.
In summary, aside from the shared use of LoRA adapters, HoE and LoRAMoE differ funda- mentally in their design philosophy and applicability. HoE introduces steerability, interpretability. and scalability into the tailored for multi-objective alignment.
expert paradigm, establishing a novel framework specifically
D THE WORKFLOW OF HoE
Algorithm 1 The workflow of HoE
Input:
objective number N, single-objective tin-ttmnd weights (0: HegNN pre-trained weights Oyre. N-Simplex AN, number of MO LoRA Experts L, number of MO router Experts R, HoE
model θ =
uniformly select weightings (NI)wNN+L+B) AN
for i to N do
T; f extract LoRA from (0, Ppe))
end for
for 1 N to N do
Tetl f Merging (0: (0: with weighting À
end for
θ (T (=++)
for N to I do
Tr < Train router experts A, with +
end for
insert θ (ni (T))--NN-+R)
Output: +
Algorithm show the whole pipeline of HoE.
E IMPLEMENTATION DETAILS
E LORA EXPERT DETAILS
HoE builds on recent advances in delta-compression techniques, which demonstrate that task vectors derived from fine-tuned models can be faithfully compressed using SVD-based or related meth- ods (Wang et al., 2024c; Ping al., 2024; Gu et al., 2025; Yuan al., 2023; Ryu et al., 2023). These approaches remain effective even when the base and fine-tuned models differ substantially (e.g., instruction-tuned versus base variants), achieving compression rates up to 1/16 with negligible performance loss. In our setting, the observed parameter deltas between single-objective models and the base LLM are comparatively small (MSE < 0.000001, MAE < 0.000440), further ensuring that compression preserves accuracy. Across all experiments, employ Activation-ANa (ASVD (Yuan et al., 2023)) as our primary compression method. We also evaluate alternative strate- gies: 1) Naïve SVD leads noticeable accuracy degradation; 2) ASVD achieves the best
SVD
24

---

Published as a conference paper at ICLR 2026
trade-off between fidelity and efficiency, therefore used in all reported experiments; 3) Delta-Coma offers the highest fidelity but requires GPTQ training, making incompatible with our training-free setting and thus not adopted in this paper.
For multi-objective LoRA experts, we first adopt PCB-merging (Du et al., 2024) combined with DARE (Yu al., 2023) our merging strategy, and then apply SVD decomposition.
While the number of experts naturally increases with the number of objectives, this does not com- promise deployment scalability. Recent infrastructure-leve optimizations -ucc S-LoRA (?) enable serving over one hundred LoRA experts concurrently on single GPU. In practice, we adopt S-LoRA-based deployment, which delivers 2-3x speedup
standard LoRA serving in
vLLM when running HoE on 2 x A100-80GB GPUs. This design ensures that HoE remains both parameter-effiieet and inference-efficient, even when scaling to many objectives.
E.2 HoE DETAILS
For 2-objective alignment, in addition to the two corresponding single-objective LoRA experts, we introduce an additional LoRA expert represented by [0.5, 0.5) and adaptively add one router expert based on the evaluation result. This results in a total of three LoRA experts and one router expert.
For 3-objective alignment, we include the three single-objective LoRA experts along with an addi- tional LoRA expert represented by [0.33, 0.33, 0.33). Specifically, for the Helpful Assistant Task, incorporate a router expert represented by [0.25, 0.25, 0.5] to enhance preference balancing. This results in total of four LoRA experts and router expert.
For 5-objective alignment on the HelpSteer Task, we utilize five single-objective LoRA experts along- side additional LoRA expert represented by [0.33, 0.33, 0.33, 0, 0] and router expert represented by [0.2, 0.2, 0.2, 0.2, 0.2] to improve adaptability across different preferences. This results a total of six LoRA experts and one router expert.
E.3 OPTIMIZATION DETAILS
Each router expert is optimized only with respect to its associated preference vector ( Given preference (e), the optimization objective for router expert is identical to the formulation in the main text:
7/> = arg max (-)a),((1a)) (x, ()),
(12)
"
where R (x, 3)
the scalarized multi-objective reward. Unlike LoRAMoE. (Dou et al., 2024), our method keeps all LoRA expert parameters frozen, which drastically reduces training cost and ensures plug-ami and- modularity.
(2, y) E,,
To properly handle non-convex regions of the Pareto frontier, we adopt the Tchebycheff (TCH) scalarization used in the main text. For clarity, we denote the expected reward for objective as (R) = EgwD, y><xo(-))) R2 [x, v)]. The objective can be formulated as:
J(O max mjn {): (0)
-000)=)))] =ž)}
(13)
where =* represents a reference point indicating the desired performance level for each objective, and denotes the relative importance of each objective.
Because directly solving the max-min form is unstable in reinforcement learning, we apply the standard Online Mirror Descent (OMD) (Liu et al., 2024) reformulation used in the main text:
J(0[)) = max Σ
(R) - =²²),
(14)
subject to:
arg min/ (w: (B. 2()} |w]] 1.
(15)
The original TCH formulation yields one-hot indicator vectors w which cause abrupt switching near Pareto boundaries and significantly harm RL stability. Following OMD-STCH-MORL, (Qiu et al.,
25

---

Published as a conference paper at ICLR 2026
2024), adopt the Smooth Tchebycheff (STCH) relaxation by replacing the min operator with softmax: w.
softmar IR; (O)))
(16)
yielding continuous trade-off indicator vector w.
The weights are then updated online using a mirror-descent rule via TD learning, enabling the optimization leverage online data across multiple training batches for more stable estimation:
log w <- log wf + add(2 -R((0)
(17)
Using the OMD/STCH reformulation, the multi-objective optimization decomposes into uniform linear combination of singlc-objective RL problems:
VoJ[(O|A) = Σ w. (oR.
(18)
For each objectivei, we compute its PPO (Schulman et al., 2017) advantage AT The aggregated policy gradient used update the router expert becomes:
(aJ(O aa | (E wi A (st, ax)) Vo log (a(a)())))
(19)
which matches the expression in the main text.
In practice, Policy sampling, KL estimation, and backpropagation are shared across all objectives and executed once per update step. Each objective maintains own critic model to estimate value for computing A All transformer layers of critic models are shared; Only the final linear value heads are independent.
EXPERIMENT DETAILS
F.1 DATASETS DETAILS
We utilize the following dataset for training and evaluation.
For Helpful Assistant 0222http:c/uuggggiioeoo////taaie
task, we utilize "hh-rlhf" dataset (Bai al.,
multi-round dialogue dataset.
For Reddit Summary task, we utilize the summary dataset (Stiennon al., 2020) https://
huggingface co/dataset.
BeaverTails task, we utilize PKU-SafeRLHF-IOK For
(Ji al.,
For HelpSteer task, we utilize the HelpSteer dataset. http://wwggngfaaecccaaaaaavvvvve
For Helpsteer2 task, we utilize the HelpSteer2 dataset t
For Psoups task, we utilize the same evaluation dataset as (Jang et al., 2023) wtwp:egggggggggs../poooooaaaaaie
For Math task, we utilize the GSM8k http:///gginngiaa.coooaaasssssss
datasct(Cobbe al., 2021)
For MTL task, we additionally utilize over-refusal benchmark (Cui al., 2024).
F.2 REWARD MODEL. DETAILS
The 14 distinct objectives consist of both interpretable natural language goals and names derived from reward models (RMs): Hclpful", Harmless". "Humor" on Helpful Assistant task, Psoups task and Helpsteer2 task; math" Math Task; faithful", summary". "deberta" on Reddit Summary task; reward"", " cost" on Beaver Tail task; helpfulness", "correctness", "coherence", complexity". verbosity" on Helpsteer task.
We utilize following open-sourced reward models for training and evaluations. Reddit Summary, use https: / ngfaae.aae...o
For
httt:s:/huggiinffac..omm co coco/rristene wttpe//cttttttttttt.....
26

---

Published as a conference paper at ICLR 2026
summarization for Summary, rewarc--ooel--bbtttt-----a------ for "deberta" and https: / /tt.g/ggggginc..e.e for Faithful; for Helpful As- sistant, HelpSteer2 and Psoups, we use https: htttp:://hugiii get2-larnge-ialpphhlrrrrrerddrdd for Helpfulness, https://huggt ingace........... for Harmlessness and https: Humor, for BeaverTail, for
use http:////hgggghh....e///////a//a "reward" and httpp//pp////gggg.........ee///a
7b- 0-cost for "cost' for all five objectives, helpfulness, correctness, coneetttss, coherence, complexity and verbosity.
The " helpful" and harmless" RMs are directly trained on "hh-rlhf" dataset with nearly 0.8 accuracy. The humor" RM was trained on joke dataset to detect humor with 0.95 F1 score. The five RMs for HelpSteer are directly trained on HelpSteer with over 0.75 accuracy.
F.3 BASE MODEL DETAILS
We utilize three base pre-trained models: Llama2-7B(Touvva et al., 2023),, Llama3.1-8F) 1- and MetaLlama3-8B" and main results are conducted on Llama2-7B.
To adapt the model to specific task, we first preform SFT on Llama2-7B each above tasks, getting SFT models as backbones. For Llama3. 1lama33.-8B. we directly use the open-sourced model Llama3.1-SFT. 8B which is fine-tuned Llama3.1
As to fine-tuned preference models, we have the option to directly use off--hhe-shel models, which are publicly available and fine-tuned for specific objectives, such as MathL.lma 15 for mathematical reasoning, CodeL.ama 16 17 for code generation, as well as preference-orientie Llama Models 18 19 20 for helpfulness and harmlessness, or to fine-tune the entire model or apply a parameter-efficient fine-tuning (PEFT) method on the pre-trained model. For MetaLlama3-8B, no aditional SFT training are conducted.
F.4 BASELINE DETAILS
On 6 tasks, we use the same backbone models separately to reproduce all baselines. Implementation details are as follows:
RiC, RS, MORLHF:
reproduce
RiC,
RS
and
MORLHF
according
to
MOD: we reproduce MOD according to https//github..oo//szeeet
MODPO: we reproduce MODPO according to http://gghhbbbcom///ZZZZZZ/ooop
Args: We reproduce Args accoding https:/ggttbbb.commccceeemmmmmgs and use open- sourced model
PCBmerging, TiesMerging: we reproduce PCBmerging and TiesMerging according to http:://thbb...mmmuuuuuun/n////p
https//huggingface.comomm cw/metaallamm/Lllaaa-222
12 wttp://ccccccc.aaaaaaaaaaaaaaaa. co/metta-lam//Laaaaa-33 Iaam/LLama-331-8B
" "ttps://uugggnggffc.co/////
14 http://waagnnnni..ccoooool//mllf
16 http:://ggiinnahocoooosssss/2/2m
18 http:///waaiiinnggc.cc//////o/oa
19 "https//hugginngfcc.cc
20 "htpss//huggingfacc. http://hwwggnnncccccooo/cccaaaaa
27

---

Published as a conference paper at ICLR 2026
Personalized Soups:
reproduce
Personalized
Soups
according
to
httss//gghubbbco//joljjigggn
PAD: No available code is released currently, replicated unofficial implementation according to (Chen et al., 2025) and published it on our depository.
Free-merging: reproduce Free-merging according to https//gittubb.oom////ggg2/2/2/2 Merging
We faithfully reproduced the these baselines using their code, replicating their experimental setup and benchmarks as described in the original papers. For MORLHF, we only train on a few main preferences due the high training cost. For RS and MOD,, we use the exact same model for fusion. For MetaAligner and Args, we tested its refining performance under Llama2-SFT and Llama3.1-SFT.
For RiC, we train new models for each two objectives pairs or three objectives.
For PCB-Merging and Fr-Merging, we used CMA-ES (Hansen, 2016) search for the best hyperpa- rameters.
F.5 EVALUTION DETAILS
Regarding evaluation on preferences, we select weightings from N-simplex ranging from zero to one to simulate various human preferences. We discretize the weightings space us- ing small gridsize 0.1 or 0.05. When received two rewards, we randomly select 11 pref- erences A1 E 0.1. 1.0 and A2 1 A1. When received three rewards, we uni- formly select preference point from a 3D-simplex. Preference weightings are set as {(0.0, 0.0, 1.0), (0.0, 1.0, 0.0), 0.1, (0.1, 0.1), 0.2.0.2,006)) 0.4,0..4.0...
00.2,0.6, 0.2), (0.33, 0.33, 0.33). (0.4,0.4, 0.2), (0.4, 0.2, 0.4), (0.6, 0.2, 0.2) (0.8, 0.1),
(1.0, 0.O, 0.0)} Then fusion models generate replies the prompts of corresponding test set with greedy searching, and directly use the above reward model to get scores. For reproduction, We always use greedy search during generation.
We mainly consider the outcomes of reward model as the evaluation result. Specifically, for math task,
PASS@1 1 accuracy validation dataset of GSM8K (Cobbe et al., 2021) as metrics. And for the over-refusal benchmark (Cui et al., 2024), we define the safety score as the probability that the model successfully resists jailbreak attempts from genuinely harmful prompts. Meanwhile, the helpfulness score is measured by the model's I success rate in correctly responding to seemingly harmful but actually benign prompts, representing the inverse of over-refusal. At the same time, we will also use the comparative win rate provided by GPT-4 to assist in the evaluation, and we use the same prompts for GPT-4 evaluation as PAD(Chen et al., 2025). We compare their win rates against the reference response provided by the original pre-trained model or SFT model.
G PROOF
In this section, we discuss on theoretical convergence guarantee of OMD-TCH-MORL.
Let (0) denote the expected reward gap between the current policy and the targeted reward for the i-th objective: f((0) = Ex [V."0 (x)] 2 Let II denote the policy space and θ denotes the feasible region of parameter 0 space. We Then define the TCH scalarazation
N
((
Σ (f)
and then TCH optimization then solves:
mar mán (0
We begin by establishing key assumptions required for our analysis. Assumption.
1. Convexity: Vi [N]. f:(0) convex in 0.
28

---

Published as a conference paper at ICLR 2026
2. Bounded objectives: vi [N], 40 f.) S U.
3. Bounded gradients and stochastic gradients: vi [M], 00 [V & 1,11) s
L.
4. Bounded feasible region: 40 E 1, |00 I Ro.
5. Policy feasibility: A feasible reference policy x¹ exists such that x* is feasible, that is 3m
II, Vi (=>) = 2
6. Bounded gradients variance: Vi [N], v0 oor S
We define the expected cumulative reward under policy with preference as:
(+) = E (x))
(xta
(20)
The objective function for Tchebycheff scalarization is given by:
L( E2- (x)]
(21)
We then establish that the gradient update direction of the policy gradient Don metioned in 3.2 aligns with the gradient of TCH scalarazation V (L(k++1
L(Bb+112) = E VV
(x) - E EVVD DIV (x)
(22)
-EED
1V
(x) - EnnD (x)] En
Enn VV (x)] - -EnDD (x)
(23)
(r(ss, at) + , (8++1)) - (od( (xt)) + I((O)A))
D (x)
(24)
2
E aow D.T k+ (x)
7 AA (st, az)) + ((
(25)
(26)
Thus, we have:
20
N
Voc+i ([((0)+(2)) -L(()()) -L( E
B. ++)
Ae (st, az)) ( (loga k+1 (st. (se, 0z))
(a)
(27)
= V -V
(28)
Lemma G.1. (Paternain al., 2023) Let Assumption G.5 (Policy Feasibility) hold. Then the saddle point (8*, X*) exists such that: mar mén (f) = [[(0 = min max L(0
6
9
If the convexity assumption holds, OMD-TCH-MORL is strictly convergent, as proven in (Zhang al., 2011; Cao al., 2020).
Since the feasible objective space [A(II) = (EEE- [R((T] - . E II} is convex, and all reward signals are independent, then all TCH gaps Sa are linearly independent, ensuring unique saddle point (Huang et al., 2024).
If the above conditions do not hold, but the following hold: The learning rate satisfies the Robbins- Monro condition (Robbins et al., 2023), and 2) The Assumption G.6 (Bounded gradient variance ) holds, then HOMD-TCH-MORL will still converge to a local stationary point (Kushner & Yin, 2003): lim;-oo (6, X)[I = 0
none these conditions hold but assumptions 1, 3, and 4 remain valid, we establish the following convergence guarantee:
Theorem G.2. OMD-TCH enjoys a convergence rate of Olog x) where is the number of objectives and T is the number of iteration rounds, as proven in (Liu al., 2024).
29

---

Published as a conference paper at ICLR 2026
H LIMITATION AND FUTURE WORK
Despite the strengths of our method, several limitations remain: (1) Our approach depends on off- the-shell single-objective models, which may not always be available. Training such models from scratch can be time-consuming and impractical in some settings. (2) The method relies on effective model merging and SVD-based compression. While these techniques work well for the objectives considered, they may fail in some settings.
I
CASE STUDY
What say my friend who wears really heavy makeup? She looks like a clown.
User:
Helpful LoRA Experts
Humor LoRA Experts
Harmless LoRA Experts
G LoRA Experts
Preference
Generation
Selection Ratio of Each LoRA Experts
weighlings
Assistant: think you her she looks like clown.
[9:58, 0.83, 0.38]
Harsesss Huroon [8.35, 8.28, 8.35]
Assistant: Maybe you say something like. 'I think you look beautiful without makkeu.."
sonetting
Assistant: You could say that her makeup too heavy
[1,
0]
Helpful, Hareles 8.4, 0.6]
Assistant:
I'm not what you
"heavy makeup". Can you tell me more about what looks
Helpful [0,
Hanor @]
Assistant: sorry what to say.
Helpfut, Horeless, Auson [0.38, 8.83, 0.58]
Assistant: Maybe you could just tell her that you don't it? Or maybe you could ask why she wears it?
Figure 10: Visualization of Case Study and Selection Ratio of Each LoRA Experts (i.e. router logits for LoRA expert selection). (w.r.t layers[31].sed token represent the activated corresponding experts, and the color size represents the proportion of selection.
The different colors on the
30


Reviews:
[{"review_id": "eDcmEQpSO2", "reviewer": "Reviewer_cfcq", "summary": "The authors studied the problem of multi-objective alignment problem in LLMs. To address the problem, they decompose the alignment problem into a number of single-preference subproblems, each of which handled by specialized experts. They combined LoRA experts, router experts, and preference routing to address the problem in their hierarchical MOE framework.", "questions": "Motivations of why the authors combine LoRA experts, router experts, and preference for the multi-objective alignment problem are not convincing, as there are many methods/strategies to address such tasks. \n\nBecause the proposed framework is combined by LoRA experts, router experts, and preference, there are a large number of parameters applied in the framework, although the authors tried to reduce the size of the LLM by their lightweight, parameter-efficient, and plug-and-play approach. In fact, such combination may not be appropriate to address the multi-objective alignment ask due to the size of the LLMs after the combination.\n\nNot sure if there are other ways to eliminate the need to train the proposed model beside the propose HoE? \n\nAre there any additional baselines that were published in this year to be taken as baselines in the experiments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow.\nA number of different NLP tasks were taken to evaluate the performance of the proposed framework.\nA number of datasets were used to conduct the experiments.", "weaknesses": "See the below questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uceGcZuss2", "reviewer": "Reviewer_V6P4", "summary": "The paper tackles the important and challenging problem of multi-objective alignment for LLMs. The core idea of using a hierarchical, decomposition-based MoE framework (HOE) is novel and parameter-efficient. The paper's primary strength lies in its comprehensive and very strong empirical results, demonstrating state-of-the-art performance across numerous benchmarks by dominating 15 baselines.", "questions": "Please see the Weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The core architectural idea of a hierarchical Mixture-of-Experts for MOA, inspired by decomposition methods, is novel. \n\n2. HOE achieves state-of-the-art performance, consistently dominating the Pareto frontiers of 15 competitive baselines (including RS, MOD, and RiC) in 2-objective settings. This is a very strong empirical contribution.\n\n3. The paper features high-quality ablation studies that provide clear insights into the model's components.", "weaknesses": "1. The paper's central claim is the achievement of \"optimal Pareto frontiers\" and \"superior Pareto-optimal results\". However, the paper provides no evidence that the proposed HOE method actually converges to the true, global Pareto optimal frontier. The theoretical analysis in Appendix G relies on strong assumptions, such as the convexity of the objective functions (Assumption G.1), which are well-known to not hold in the non-convex landscape of LLM optimization. While the use of Tchebycheff (TCH) scalarization is appropriate (as it can find non-convex frontiers), it does not guarantee that the frontier it finds is the optimal one. Therefore, all of the paper's claims of \"optimality\" are purely empirical and lack the foundational theoretical guarantees that the term \"Pareto optimal\" implies.\n\n2. The framework's design is confusing, as it introduces two distinct types of multi-objective components: \"Multi-Objective LoRA Experts\"  and \"Router Experts\". Both components appear to serve the exact same purpose: covering intermediate points on the Pareto frontier. The paper fails to provide a clear justification for why both are necessary.\n\n3. This paper lacks of criticial details. A core component, the Merge function used to create multi-objective experts ($\\tau_{\\lambda}=Merge(...) $), is never defined in the paper.  The \"task-SVD\" compression process is also vague. The example in Appendix E.1  suggests it may involve manual, per-objective hyperparameter tuning, which would severely undermine the \"plug-and-play\" and \"lightweight\" claims. Furthermore, the mathematical formulation for the router optimization is inconsistent between the main text and the appendix, confusing any attempt at re-implementation. \n\n4. There are several issues for the Proof in App. G as well. \n\nFirst, the proof's strongest claims of convergence rely entirely on Assumption G.1 (Convexity). The parameters $\\theta$ being optimized belong to the Router Experts, which are neural network layers within a Transformer. The optimization landscape for LLMs and deep neural networks is well-known to be highly non-convex.\n\nSecond, even if local stationary point convergence is established, a local stationary point is not, by any means, equivalent to a Pareto optimal solution.\n\nThird, Lines 1480-1494 are very confusing. For example: “If none of these conditions hold but assumptions 1, 2, 3, and 4 remain valid...\"\n\n5. Please clarify some mathematical formulations. The main text in Section 3.2 (Eq. (6)-(8)) introduces an Online Mirror Descent (OMD) method for the Tchebycheff (TCH) objective . However, Appendix E.3 (Eq. (12)-(18)) shows a different-looking formulation that is supposedly based on the same TCH and OMD principles. The relationship between Eq. (8)  and Eq. (18)  is unclear, even though both appear to describe the same PPO gradient", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4eEuOcMD94", "reviewer": "Reviewer_tavC", "summary": "This paper proposes HoE, a hierarchical mixture-of-experts framework for multi-objective alignment of large language models, enabling them to adapt to diverse and conflicting human preferences without costly retraining. HoE integrates LoRA experts, router experts, and preference routing to efficiently cover the Pareto frontier, providing scalable and fine-grained control over model behaviour.", "questions": "* Q1: The HOE integrates the PPO paradigm to optimise models. I wondering that if the reward model in PPO is the same with the reward model in evaluation?\n* Q2: In Line 1233, there may be citation error \"(??)\". Please correct it.\n* Q3: The method mainly fine-tunes the routing layer while freezing LoRA parameters, but there is no detail on where the LoRA parameters come from or whether they need to be trained.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper propose a novel alignment approach named HOE. There are some strengths:\n* **Methodology**: This paper introduces a hierarchical expert-model framework (HOE) to handle multi-objective alignment, and incorporates Pareto-optimality concepts to provide a theoretical grounding for the approach.\n* **Scalable and extensible**: leverages model fusion techniques and a lightweight routing module to enable efficient training with lower resource costs.\n* **Experiments**: The evaluation covers multiple datasets and baseline methods, yielding a comprehensive analysis of the results.", "weaknesses": "However, there are some weakness of this paper:\n* **Methodology**: multi-objective LoRA experts are expected to learn different preferences, and the experimental results also validate it. However, the design of multi-objective router expert seems to be redundant. The ablation study only discusses a single router and the role and necessity of a multi-expert router are not clearly demonstrated.\n* **Reproducibility**: The results appear to rely on the pretrained model, yet the manuscript does not specify how the pre-trained model was chosen. The method mainly fine-tunes the routing layer while freezing LoRA parameters, but there is no detail on where the LoRA parameters come from or whether they need to be trained. Taken together, these points suggest certain reproducibility gaps in the code and experimental setup.\n* **Evaluation**: The evaluation in this paper primarily relies on reward-model scores, which may limit the ability to capture objective, user-centred aspects of quality and user needs. It would be more persuasive if the authors could explain the rationale behind the chosen metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)