Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging
Authors: Jinluan Yang
Dingnan Jin
Anke Tang
Li Shen
Didi Zhu
Zhengyu Chen
Ziyu Zhao
Daixin Wang
Qing Cui
Zhiqiang Zhang
JUN ZHOU
Fei Wu
Kun Kuang
Venue: NeurIPS 2025 poster ()
Keywords: Model Merging; Large Language Models; Alignment
Primary area: deep_learning
TL;DR: This study presents the first comprehensive investigation into model merging and data mixture strategies for constructing large language models (LLMs) aligned with the 3H principles (Harmlessness, Helpfulness, Honesty).

Abstract: Achieving balanced alignment of large language models (LLMs) in terms of Helpfulness, Honesty, and Harmlessness (3H optimization) constitutes a cornerstone of responsible AI. Existing methods like data mixture strategies face limitations, including heavy reliance on expert knowledge and conflicting optimization signals. While model merging offers parameter-level conflict-resolution strategies through integrating specialized models' parameters, its potential for 3H optimization remains underexplored. This paper systematically compares the effectiveness of model merging and data mixture methods in constructing  3H-aligned LLMs for the first time, revealing previously overlooked collaborative and conflict relationships among the 3H dimensions and discussing the advantages and drawbacks of data mixture (\textit{data-level}) and model merging (\textit{parameter-level}) methods in mitigating the conflict for balanced 3H optimization. Specially, we propose a novel \textbf{R}eweighting \textbf{E}nhanced task \textbf{S}ingular \textbf{M}erging method, \textbf{RESM}, through outlier weighting and sparsity-aware rank selection strategies to address the challenges of preference noise accumulation and layer sparsity adaptation inherent in 3H-aligned LLM merging. Extensive evaluations can verify the effectiveness and robustness of RESM compared to previous data mixture (2\%-5\% gain) and model merging (1\%-3\% gain) methods in achieving balanced LLM alignment.

Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging
Jinluan Yang', Dingnan Jin Anke Tang, Li Shen, Didi Zhu', Zhengyu Chen' Ziyu Zhao', Daixin Wang Qing Cui², Zhiqiang Zhang" ,Jun Zhou? Fei Wu' Kun Kuang" Zhejiang University; ? Ant Group yangj inlgiinnunnzzu. edu en,
Abstract
Achieving balanced alignment of large language models (LLMs) terms of Help- fulness, Honesty, and Harmlessness (3H optimization) constitutes a cornerstone of responsible AI. Existing methods like data mixture strategies face limitations, including heavy reliance expert knowledge and conflicting optimization signals. While model merging offers parameter-level conffict-resolution strategies through integrating specialized models' parameters, its potential for 3H optimization re- mains undercxplored. This paper systematically compares the effectiveness model merging and data mixture methods in constructing 3H-aligned LLMs for the first time, revealing previously overlooked collaborative and conflict relation- ships among the 3H dimensions and discussing the advantages and drawbacks of data mixture (data-level) and model merging (parameter-lewe methods in miti- gating the conflict for balanced 3H optimization. Specially, propose novel Reweighting Enhanced task Singular Merging method, RESM. through outlier weighting and sparsity- -aware rank selection strategies address the challenges of preference noise accumulation and layer sparsity adaptation inherent in 3H-aligned LLM merging. Extensive evaluations of RESM compared to previous data mixture (2%-5 (2%-5% gain) and model merging (1% (1%-3% gain) methods in achieving balanced LLM alignment.
verify the effectiveness and robustness
Introduction
Large language models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks IT- 3]. However, their reliable deployment necessitates balanced optimization across three critical dimensions: Helpfulness (pro- viding accurate and task-aligned responses), Honesty (avoiding hallucinations and misinformation), and Harmlessness (prevent- ing toxic or unethical outputs), collectively termed 3H optimiza- tion 7] While recent alignment techniques such as consti- tutional AI B reinforcement learning from human feedback (RLHF) and Direct Preference Optimization (DPO) [10) have improved individual aspects of 3H, secking a balance re- mains a significant challenge. For instance, models optimized for helpfulness may inadvertently generate harmful content [II.
Masit Meriha
Mandow
Figure Illustration of trade-off's in optimizing LLMs across the 3H ob- jectives.
"Corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

while harmlessness alignment can lead to dishonest responses
[2]. This trade-off illustrated in Figure I highlighting the need for systematic approaches to harmonize 3H objectives.
Traditional methods for enhancing 3H properties often rely data mixing strategies assisted by empirically heuristic rules [13), multi-dimensional scoring via reward models [14], or aligmment conflict metrics [5]. where diverse datasets are combined to fine-tune a single model. While effective, these approaches face practical limitations: (i) data curation requires substantial domain expertise and computational resources [T] [6], and (ii) conflicting optimization signals during fine-tuning may complicate prioritization of alignment objectives without compromising others [7). As a cost-effective alternative, model merging has gathered great attention for LLM alignment through integrating parameters from specialized aligned models, addressing key challenges such as catastrophic forgetting after fine-tuning [88] and achieving robust reward models [9 20]..
However, for 3H optimization, the effectiveness and limitations of existing merging methods remain underexplored, especially considering the preference noise [21] and layer significance [22 23 for
LLM multi-objective alignment. While preliminary investigations exist [24]. these are narrowly focused on constrained scenarios (e.g.. multilingual) or employ partial evaluations of 3H dimensions
25 without systematic comparisons. This raises the central question to be explored:
Can we benchmark the model merging and data mixing techniques 3H optimization for LLM alignment and explore the overlooked optimization principles specific to model merging?
To address this question, we first establish a benchmark for 3H optimization in LLM alignment and systematically compare model merging and data mixing techniques for 3H optimization. Based on this benchmark, we reveal previously overlooked collaborative and conflicting relationships among the 3H dimensions and discuss the advantages and limitations of data mixture (data-level) and model merging (parameter-level) methods in mitigating conflicts for balanced 3H optimization. Additionally, we address the challenges of preference noise accumulation and layer sparsity adaptation in L.LM task singular merging (RESM) method via outlier weighting and sparsity-aware rank selection to further enhance balanced LLM alignment. In summary, our key contributions are as follows:
multi-objective merging, proposing novel weeeighhing-enhnneee
We create the first benchmark for 3H optimization in LLM alignment and systematically compare model merging and data mixing, including our investigations into 15 representative methods (12 training-free free model merging methods and representative data mixture methods), 10 preference datasets associated with 5 annotation dimensions, 2 classific LLMs families, and 2 training settings.
We reveal a range of previously overlooked optimization principles and insights for 3H optimization in L.LM alignment. These include: different collaborative and conflict relationships among 3H dimensions, the superiority of model merging over data mixture methods, and the factors affecting the effect of model merging considering redundant parameters updates during post-training.
Beyond holistic evaluation of existing model merging methods, we propose a novel reweighting- enhanced task singular vector merging algorithm adapted the preference noise accumulation and layer sparsity during merging through outlier weighting and sparsity- aware rank selection. Extensive experiments verify its effectiveness in achieving balanced LLM alignment.
2 Related Work
Model Merging for LLM Alignment. Model merging has emerged as a parameter-level cost- effective technique for L.LM alignment [18), addressing challenges across four aspects: (a) Stabilizing reference policies focuses on the over-optimization problems during the RL training. Weight- space averaging of models with varying initializations constructs robust policy ensembles [26). while dynamic trust-region updates (27] and online gradient fusion (28) help preserve foundational capabilities. (b) Cross-mode capability transfer resolves architectural mismatches during knowledge fusion 29 through probabilistic token alignment [30], vertical domain adaptation (31), and subspace projection 32 But persistent toxic parameter propagation [3 34 remains a critical barrier, inducing biased representation transfer during integration. (c) Avoiding forgetling after finetuning develops gradient-aware selective merging (35] heterogeneous layer-wise merging [36. (37, and subspace- based merging (8 to mitigate the alignment tax or realign the model after fine-tuning for downstream tasks. (d) Balancing multi-optimized objectives employs linear interpolation of reward-tur -tunedddddddddeeeee models

---

20] and Mixture of Experts (MoE) based expert routing [25] to approximate Pareto frontiers but lacks theoretical guarantees for subspace conflict analysis. Additionally, while location-
39 17 19
based merging 40) identifies alignment-specific weights. its efficacy is heavily data-dependent. Notably, (24] provides preliminary insights into safety-utility trade-offs in cross-lingual scenarios, but none of these studies explore model merging's 's potential and limitations for 3H optimization.
Data Mixture for LLM Alignment. Compared with the pretraining data mixture in terms of different domains, LLM alignment aims to achieve a good trade-off between Helpfulness, Honesty,
and Harmlessness (3H) regarding human preference [4T 43 71.
(a) Empirical Methods 44 45
have explored the heuristic mixture strategies between helpful and safety-related data to mitigate the safety-utility. (b) Reward Model-basee Methods train traditional Bradley- Terry models [46] 47] and multi-objective reward models. which are designed to score the data for capturing the complicated human preferences [48 51 ArmoRM [4] is a representative development aiming to promote L.L.ss aligned with human- internnntterrrtablle multi-objective demands like honesty and helpfulness. (c) New Metric Methods are initially designed to select preference data only from the quality and diversity dimensions 152 53). Hummer [15] recently quantifies the conflict among preference datasets to balance diverse alignment objectives effectively. Different from these works that resolve the conflict from the data mixture perspective, we explore the parameter-level model merging solutions and select representative data mixture methods as comparisons to discuss their effectiveness.
3 Revisting Model Merging for Multi-Object Alignment Optimization
3.1
Preliminaries
The intersection of model merging and alignment optimization presents unique challenges and opportunities that warrant dedicated investigation [19 20 Given multiple models parameterized
by 8', 62, ..... 8", where each optimizes base model 6 towards a different aligmment objective, the
alignment task vector set can be achieved by A = (A), merging methods related to L.LM multi-ohjective alignment 18 can be concluded as follows.
A") = (0) - 80 ... 8 - 8²). Existing
Linear interpolation methods, such as Rewarded Soups 17 and Weight Average based methods (WARM [19] and WARP (20)), have demonstrated that simple weighted averaging of model parame- ters can be effective in learning the Pareto frontier of multiple objectives or achieving robust reward models and reward policies. The merged model can be achieved through: 0 = Di w²O' where the w = (Wi, W2,- ,w,) is defined as the interpolation weight related to adjustable preference.
Vector (TV) based methods [54] integrate different parameter update directions (the alignment task vector A = H³ 80) rather than full model parameters like linear interpolation. Advanced merging approaches like TIES (55) DARE (56). Breadcrumbs (57. and DELLA (88) explore many nuanced ways to identify and preserve crucial subspaces that capture different objectives and resolve the objective conflicts. In general, these methods can be expressed Merged 00 + Σ tiym O (0*-6"), 89), where m; E f0, 13 [8] a binary mask and O is the element-wise multiplication. Moreover, Model stock (59] identifies that model performance correlates strongly with proximity to the center of the weight space and proposes to approximate this optimal center point geometrically.
Task Singular Vector (TSV) based methods (60-62) point out that the element-wise mask often breaks the inherent row-column correlations in the weight matrix, potentially destroying low- dimensional structure of the fine-tuned parameters critical for individual tasks. As an alternative, they exploit the low-rank structure of task vectors through layer-wise parameter conflict analysis. By performing Singular Value Decomposition (SVD) with low-rank approximation on top-k singular components of layer-win wise task vectors, we can achieve compressed or truncated task matrix through SVD -09) = [ M (OT [: kaxed, :]. where U, S, and V are the left singular vectors, singular values, and right singular vectors, the kixed represents the top-k selection of singular values.
min | - -UilF s.t. U
(I)
Un
min Vi
Vi
Ville s.t. VI I,
(2)
OMarged Layer = 0f+ Σ
Ufi [i,; knned (0 vi(i) kixeds :].
(3)
3

---

30
600
3H Nornalized Gain (%)
w/o outlier weighting
Sparse Layer
w/ outlier weighting
Rank Selection
Dense Layer
20
400
10
200
3
Helfulness
Honesty
Harmlessness
number models during merging
Aligmmentt Task Vectors
(a) Preference Noise Accumulation
(b) Effective Rank Analysis
Figure 2: (a) illustrates the impact of outlier weighting in model merging when incorporating multi- aligned models. The performance discrepancy before and after processing quantifies the degree of preference noise accumulation; (b) depicts the effective rank required to capture 95% of the total energy in singular values for task vectors derived from 3H training. Larger ranks indicate the necessity of retaining a higher proportion of singular values to preserve task-relevant information.
Assisted by decorrelat-base based whitening transformation as Eq. I and Eq. 2). we can obtain orthogonal singular vectors UL1 and V.L mitigate the interference among task matrices. Finally, the layer-wise weight for the merged model can be defined as Eq. (3) where the wy is the scaling factor.
3.2 Motivation
Despite the successes of V-basee methods outlined above, several overlooked challenges undermine the efficacy of model merging for 3H optimization:
(i) Negative Outliers from Preference Noise Accumulation: methods is to separate task-specific and task-shared parameters through element-wise mask or layer- wise SVD decomposition [8). But all of these works ignore the preference noise accumulation PI1, which is a special problem for LLM alignment, especially when more alignment objectives or models considered during merging. This introduces additional outlier weight updates that make it difficult to capture true task-specific optimization direction, so as to weaken the effect of conflict resolution.
The core design of previous merging
As shown in the Figure 2a we respectively train models for each dimension of 3H optimization. The merging results with 3/6/9/12 models represent that for 3H optimization, we respectively select 1/2/3/4 single-dimension models with different hyperparameters. We apply the 30 principle to mask the outlier weight for singular value parameter updates (the details can be shown in Section 4 while adopting the representative task-singular vector merging algorithm, TSVM [60]. From the results, we can find that masking the outlier weight within the singular value can strengthen the merging effect, and the accumulation preference noise has the same trend the increased model numbers.
(ii) Unreasonable Fixed Rank Selection for Each Layer: Conventional model merging methods employ a uniform rank selection threshold & across all layers, failing to account for layer-specific sparsity patterns and parameter importance heterogeneity in L.MMs. evidenced by layer-wise analyses [3], sparse attention activation and dense feed-forward layers exhibit fundamentally differ- ent sparsity characteristics. Moreover, recent mechanistic studies 2 64 22 reveal that alignment capabilities predominantly emerge from localized parameter updates in specific subnetwork regions rather than global changes. Both of them necessitate distinct rank selection strategies during merging.
As shown in the Figure 25 we display the difference in the rank selection between dense and sparse layers while keeping 95 percent information/energy within the singular value [65]. This divergence of rank among layers highlights the challenge of using fixed top-k selection to compress the rank for model merging as Eq. because we may either discard important components or keep unnecessary components that cause interference between different alignment objectives.
4 Reweighting-Enhance Task Singular Vector Merging for 3H Optimization
Overall: Based on the above analysis, we propose novel Reweighting Enhanced task Singular vector Merging algorithm (RESM), with theoretical foundations in outlier detection and layer- wise

---

sparsity analysis for rank selection, aiming to improve model merging for 3H optimization. Thus, we can transform Eq. to Eq. by integrating with reweighting-ban based optimization. The full framework of RESM can be shown in Algorithm !
0 RREMM 05
CUNF: Ail (a
T1
1=1
Adaptive Selection
Adaptive
Outler Aware Weighting
Selection
To address (i), compared with directly utilizing full singular values, employ layer-wise outlier detection to aggregate the Outlier-Aware Weight for reweighting the singular values. This means we should check which parts of the singular values can truly represent the optimization direction towards the alignment process. Considering the heavy-tailed distribution of LLM parameter updates, where few parameters undergo significant changes while most exhibit minor adjustments, we leverage the 3a principle, which aligns with this characteristic. Thus, we adopt statistical significance filtering to weaken the noise effect and normalize competitive weight to identify true optimization adjustments.
The A E IR denotes the weight deviation at row r, column of layer for model relative to (t) in r, quantifying central tendency and dispersion, af") [0, (Eq.77) computes layer-wise aggregation weights via Ln-nommalized sparse outlier magnitudes, and THRESHOLD(M, applies thrnhhhhesso suppress elements in matrix M with absolute values below т.
initial model, frr and ( represent the mean (Eq. 5 and standard deviation of deviations (Eq.16)
(4) n = E. EellAr (1
(5)
(1)
E. E || -(4
(6)
1r
[hreehoold (1)
(, Mi, + 30
(7)
[Threshold(A ( ( Ln Mi, +30 30
Our outlier-aware reweighting mechanism operates through complementary mechanisms: Noise Suppression: By thresholding parameter deviations via the 3o rule, we filter out low-magnitude fluc- tuations that predominantly encode noise, forcing the singular vectors ut") to align with statistically significant task features; Task Equilibrium: The layer-wise aggregation weights aj are globally normalized across all models, ensuring balanced contributions from diverse tasks and preventing dominance by high-mas magnntude updates that may obscure subtle yet critical features. More details for outlier-aware weighting for singular value can be shown in Appendix A.1
To address (ii), instead of fixed top-k strategy, we propose to adaptively decide the level rank selection based on the layer sparsity. We can first compute the sparsity consensus for all models as (q. and then achieve the dynamic rank as Eq. where "/o >> 0 and
control the base rank and sparsity-related rank reduction respectively. the k; is defined as the dynamic rank for layer / determined by sparsity Q4. We set 70 = 0.2, 7 = 0.6 and e = 0.1 by default. We can observe that for layers with high sparsity (2₂ + 1), the optimal rank selection Ω + di(c% 7/24) retains most singular, whereas for dense layers Ω + O, the rank k decreases significantly, inducing stronger dimensionality reduction through truncation.
Ω nd
T (IA <c)
(8)
1=1
n0=1
k = [d (70 + +
Our aparsity-adaptive rank selection mechanism operates through two complementary principles: Information Preservation: For dense layers, where parameter updates demonstrate relatively uniform distributions with predominantly small adjustments, employing lower-rank approximations proves effective for noise suppression while preserving principal components. However, this necessitates
5

---

Algorithm 1: Brriightinn-Enhannee Task Singular Vector Merging
Input :Initial model @ and Further Aligned models (0')=z with same layers I, Sparsity factor
E [0, (0.
Output: model f for layer / < 1 do
// Step1 Aligument Task Vector Extraction
A (1:m) - [0] 0.] // Step2: Outlier- Aware Weighting
for model + 1 to do
Compute row-wise statistics: p
(8) < ROWOUTLIERSCORE( (I
Calculate sparse aggregation weights:
E [TTmmmsooon (0) (0) +30 1,lh
25-12 THRESHOLD( ( 0 +3
// Step 3: searsity-Adaptaae Rank Selection
Compute layer sparsity consensus: Q4
nd? E Ed. T( indel <<)
Determine dynamic rank: ky <- [de(7r +724]]
// Step :: FReweiighing Optimization during Merging for model 1 do
Decompose: ( (((()) V SVD(A
Compute orthogonal projections Ui and (1 via Eq 1 via Eq.
Reweight for Outlier Weight: S aj (())
Reweight for Rank Selection: UL1 UI (4) (. kl. Vi vi k.,d-
s St
ki, ki]
Merge Components: M. < E:=,0 Un (*) ( Vi_ (+)
Update the Layer for the Merged Model: 0r + 0 Mi
careful determination the optimal rank selection threshold to balance between information retention
and noise elimination. Conversely, sparse layers display concentrated parameter updates along few dominant directions, potentially containing critical outlier components. Here, maintaining higher rank becomes essential to ensure the preservation of these salient directional features, thereby preventing substantial information loss through excessive rank truncation. Conflict Mitigation: By preserving dominant singular directions in sparse layers and enforcing orthogonality through Eq. we reduce overlaps between task-specific parameters, decoupling interference-prome optimization trajectories. More details about rank selection be shown in Appendix A.2
5 Experiments
5.1 Experimental Setup
Datasets: As shown in Table 1 select com- monly used preference data for model train- ing. These datasets can be categorized into five groups from the annotation perspective.
Table 1: Dataset statistics for our DPO training.
Annotation Perspective
Dataset
Judge
HelpSteer (49
GPT4-Turbo
Following SimPO (72), we adopt models: Llama-3-8B Instruct [73 and Beitral-7B-Instuutt--ie
Backbones:
Helpfulness
Py Py-Dpo
GPT4-Tuuiis
Distilahel Oreha
67
GPT4-Turbo
two instruction-tune
DistilabelCauyyure
68
GPT4-Turbo
(74). These serve as the SFT model, and we then per- form DPO training the full network using the preference data.
Harmlessness
UltraSafety
GPT4-Turbo
Truthy-po-vvl Dno
69
Human
Honesty
GRATH h0
Llama2
Helpfulness&Honeet Honesty
UltraFeedhack
32
GPT4-Turbo
(aselines:()) Individual Training: We respec- tively train one model for each annotation per- spective as stated by Table I These models are saved for model merging. (ii) Mixture Train-
Helpfulness& Harmlesssss
PKU--Sae- RL
M
GPT4-Turbo
Nectar
GPT4-Turbo
ing: We adopt the full datasets shown in Table and then adjust the data mixture proportion before

---

training based on the Empirical methods 45 |44 the culli--dmension score of the Reward model
AomoRM-Llama3-e [14] and the alignment conflict metric from Hummer [1]. The implementa- tion details can be shown in Appendix C (iii) Model Merging: Considering that the constraints of data availability and test data leak will limit the generalization of merging methods for L.MMs, we mainly adopt fraining-free multi-task or multi-object alignment merging strategies from MergeKit [75. which includes Weight Average [76). Task Arithmetic [54], fies-Merging (55), (AR
DELLA [58), Model Stock [59] and Model Breadcrumbs [57]. Moreover, from the perspective of
Pareto-optimal front [39
20] and singular vector decomposition [60] [T]. we select Rewarded Soup [77. TSVM [60] as two additional training-fite free merging methods for 3H optimization. We
17
provide discussions about training- based and MOE-based merging methods (25) in Appendix [B]
Settings: We construct two different settings to verify the effectiveness of model merging for 3H optimization: (i) Static Optimization for DPO Training at once as Table and Table where we aim to achieve an aligned model that simultaneously meets the 3H demands using various annotated preference data at once. (ii) Continual Optimization for Sequential DPO Training as Table TQ and Table I which refers to the continual and dynamic circumstances with newly curated preference data and more customized demands compared to previously trained models. In this case, we need to simultaneously focus on the effectiveness and efficiency of constructing an aligned model.
Evaluation: (i) For Helpfulness: we se-
lect Math, GSM8K, ARC-C, ARC-E, MMLU,
Table 2: Necessary specifications for the strategy and scaling of each method.
MBPP-Plus. HumanEval-Pinn
[77], and MT-
Bench 77 to asses the helpfulness of L.MMs;
Method
Strategy
Sealing
(ii) For Honesty: we utilize the HaluEval- Wild
Data Mixture-Baser Based Methods
79 for evaluating honesty or hallucinations;
Heurieti
B
Empiscally leuuisice-ajjueed ratio
Dato Misture Ratio
ArmoRM
Revard Medel
Mult obsect Data Selection
(iii) For Harmlessness:, we conduct safety- related (SaladBench [80]) and refusal-related (OR-Bench [8]) evaluations to measure the
Hummer
Aligrment Confict! Metric
Multsohgect Data Selection
Merging-lased Methods
Weight Averige
NO
Linear Consensus
Farameter Weight
Scup
19
Linear Consensus
Parameter Weight Coeff.
harmlessness of models. Higher values are pre- ferred for all reported results to ensure the rea- sonableness and fairness of evaluation. The nor- malized metric is calculated based on the relative gain of each dimension to avoid the imbalanced evaluation datasets for the 3H perspective. More details can be shown in the Appendix
Ties
Astlmnctic
Scaling Factor
DARR [SS
Random Spirsfication
Parameter Scaling Factor
DELLA 50
Randoms Spuriitiction
Parameter Scaling Factor
Breadrrumss
57 90
TupBotoom Sparsification
Parameter Scaling Factor
Model Stcck
Gommeric Sparsfication
Parancter Adaetne Ratio
TSVM
Singulia Vilue Decompasitio
Parametia Scaling Factor
5.2 Experimental Results
There exist different collaborative and conftict relationships in terms of 3H objectives for LLM aligmment. As shown in Figure 3, we display the trade-off through individual training comparison. Denote the results of Instruct LLMs as the grey line in the Figure, we can compare the results of Honesty and Helpfulness after performing individual Helpful, Honest, and Harmless Training to distinguish the relationship between each optimization dimension. From the results, we can observe that there exist different collaborative and conflict relationships between helpfulness, honesty, and harmlessness while performing DPO Training, exhibiting that Helpful Training benefits 3H performance simultaneously, but Honest and Harmless Training weaken each other.
Model merging can serve as good alternative for data mixture methods in mitigating the 3H conflict. As shown in Table 3 and Table 4 we compare the effectiveness of data mixture and model merging methods for 3H optimization of Llama3 (73 and Mistral 74. For data mixture methods, they mitigate the 3H conflict by collecting full training preference data and then filtering conflict samples. Compared with heuristic strategies and reward model based strategies, which rely on historical training data and human effort, Hummer is specially designed for evaluating conflict for full training preference data, which can achieve better 3H results with a norm gain from 7.68% to 10.16% on the Llama3. For model merging methods, compared with full training strategies, they advocates for phased optimization to negotiate competing alignment objectives through dimension -eppesiiio--ccciiicccc individual training. where we first conduct individual training to obtain models for five different annotation dimensions respectively, and then adopt conflict-aware parameter merging strategies, such as random sparsification, Top-k filtering, and singular value decomposition, to merge these models into an ideal one that can achieve close or superior results than full training methods. Take the experiment on Llama3 for example, compared with the best data mixture methods, Hummer( 10. 16%). model
7

---

Hamma-88-Inntruus
Distral-BB-Instruct
Harmless
Training
Honest
Training
Helpful
Training
Instruct Model
50
55
60
6.5
30
35
40
45
50
55
60
65
70
75 80
Performance (%)
Performance (%)
Helpfulness Metric
Honesty Metric
Harmlessness Metric
Figure 3: Illustration of the 3H Trade-Off, where Helpful Training benefits 3H performance simul- taneously, but Honest and Harmless Training weaken each other. The dashed line (Instruct Model) serves as the reference, where rectangles on the left represent the weakening effect, and vice versa.
Table 3: 3H Results Under Static Optimization Setting where we perform DPO training using various datasets at once. The normalized gain metric is the average value of the relative gain for each dimension compared with the results of Maama-8B-Instruce
Metaue
Beansy
B-lydall Arg Blownd Ang Harrobor Aeg Gite
Nlck GSSESS
ARC-E S0ILL NBRF Fia Rasaeriijaa MiScat Raliie Wid Saiid Sea3 (R-S0ok
1.:9
31.55
76.77
$3.05
68:34
56.47
$4.35
$4.67
92.30
21.00
$0.00
57.0
-0.99
30
59.00
01.07
83:11
68.33
56.9
15
56.00
56.00
96:06
mm
05.33
+5.700
3102
77.36
05-17
$2.77
(8:11
$U.9
$4.11
8.16
44.00
91.27
51
7457-440
+9/00
Mater Hill lwrkeng (tomnnte)
36:11
05.55
03.55
11:09
68:38
9185
$7.33
3.48
54.01
12.30
$5.96
90:0
53.871
03.20%
Stntur tnaning (Thernor
21.41
0380
02.94
9141
$8.03
35.50
12.10
30111
33 03901:10
10.24
-11
10:11
07:71
$3.32
13.5
Rearabel Sacp
5164
71:6.
13.07
02211
18.54
91.55
$7.83
9.33
$3.55
01.00
57. 53551115
00.00-10
19/60%
Ttak Anheete
2101
03.01
$1.99
18.35
1714
8.31
31.13
11.39
13.23
37.891
09:00
63.07
DARE
3643
16.39
13.07
13.11
(8:41
90.5
53.11
8.28
$1.85
36.00
09.50
17.00
05.35
46.15
DILLAA
3601
3640
01.00
8131
(8:11
$.13
54.0
41.30
03.31
51.50
75:3--0m
-99 13295
06.6
17.36
01.0
(8:1
9185
13
41.00
01.0
08 08:3
08.35
46:40
0690
07.6
0110
1212
(8:53
96.6
96.01
26
96 0m
55 30.3
76.0
075
merging methods including DELLA T1es(1,.10%), Breadcrumbs 11e5(10.15%), TSVM(13.00%
and our RESM(14.97%) can consistently achieve comparable or superior results for balanced 3H optimization. These results collectively confirm that model merging's phased optimization paradigm effectively negotiates competing alignment objectives, which provides new insights for addressing the trilemma of 3H optimization for LLM alignment.
The effect of model merging for 3H optimization closely related their conffict-resolution strategies. RESM consistently achieves better results due to its reweighting designs. As shown in T we can divide existing parameter-level strategies into three categories: linear consensus, sparsification, and singular value decomposition. Linear interpolation of full model parameters or task vectors neglects the parameter conflict, limiting their performance for 3H optimization LLM aligmment. The sparsification-ben based method holds the assumption that pruning redundant (DARE and DELLA) or outlier parameters (Breadcrumbs) that do not represent the direction of updates for task vectors can improve the effect of model merging, but the level of sparsity is difficult to control for LLM through even with different sparsification methods. From the results of Table and Table 4 we can observe that there is no fixed and stable trend for the results of sparsification based methods due to random sparsification. For example, DELLA- -iics and DARE-Ties exhibit opposite phenomena in Llama3 and Mistral. More details about the sparsity that influences the effect of merging can be shown in Appendix C.4 In contrast, both TSVM and RESM can achieve stable performance gain through decomposed task singular vectors without heavily depending sparsification. Moreover, our RESM enhances the merging effect of TSVM with normalized gain from 13.00% to 14.97% due to its reweighting optimization adapted to preference noise accumulation and fixed rank problems.
RESM can achieve robust and efficient 3H optimization in continual LLM alignment than previous methods As shown in Table T0 and Table [1 in the Appendix C.3 we sequentially perform DPO training using data with annotations about Helpfulness& Honnutllsss&oonsstyy (Stage 1), Helpful-
n (Stage2) and Helpful (Stage3) to simulate continuous optimization in real-world
8

---

Table 4: 3H Results Under Static Optimization Setting where perform DPO training using various datasets at once. The normalized gain metric is the average value of relative gain for each dimension compared with the results of Mistral-7B--. I----ntuutt-....
Mosheds
Regiddoo
Bklpshi Avg Blooot, Lag Harnkoon Aog Nerre Gale
Kich
ANC-E
ANCL
MMLLL NERV FTax Rtzaatrralaa Mi-Scat
1iiklva Wes| Saiic S0x3
GN-2ck3
Miteral-TB--e Saa
0:54
46.17
$2.36
72.95
$9.57
26-40
25.55
7.25
62.17
75077
74.88
41.70
62:37
26.35
5:44
47:35
$4.40
1323
60:33
12.47
12:31
1.30
00:00
74.56
50151
-2.40
77:20
40.20%
25:
-1.11%
0:40
+9.00
12.85
60.00
22.31
33.39
1.43
18.00
$5.43
00.54
+1.00
77:00
8.76
43.1-
12:01
29:38
17.33
76.34
71.23
41.8
81 (3
77000
15.99
3492
1.45
10.00
3.399
52 Mixie ly Turkeg (A
0711
15:30
83.05
31.0
11
7011
13:58
335
03
11.00
0/
18
61 65
+1.110
$1.85
Weglt Aoncigo
035
60:38
30
1850
11.11
01 02)
73:31
45005
13.5
31.08
5635
23.51
Muvil Shnk
6:0
06.95
07.36
34.0
60
9013
10.31
6111
21.75
13.50
Tix
08.22
41:47
$5.19
34.22
61.34
27.23
11.30
7.60
28.33
$2.55
01.13
40.37
55.75:15
11.00-30
4009%
DABE TEx
08.00
02:44
15a9
34.55
61.00
36.8
11.31
7.6
28:33
$2.33
11.01
43.49
99
62.10
+1.80%
DILLA
00.4
0171
566
15:3
61.56
3191
13333
7.0
56.18
$2.00
$4.77
54
+0400
BewukoneeeeT Tixs
00.30
11.85
1501
36.1
61
3670
33 19
7.8
18.04
11.03
30.00
03:4
03
01
+1.41
RFSMI JearO
00:44
15.0
$:35
75.6
66:33
36.00
33.53
331
68.58
$6.55
30.24
61 01
85 82:30rmm
+0.05
scenarios. Comparative analysis of continual training stages reveals that catastrophic forgetting effects paradoxically enhance large language models' (LLMs) alignment with honesty, helpfulness, and harmlessness (3H) through interactive optimization. Our method strategically initializes model merging from the foundational Instruct checkpoint rather than intermediate checkpoints, thereby eliminating hyperparameter sensitivity in continual DPO training while mitigating overfitting to prior objectives that impede adaptation to new optimization targets. Experimental results demonstrate RESM's s superior performance over final-stage models across 3H metrics, confirming its robustness for multi-objective alignment optimization.
RESM can achieve better results than tradi- tional sparse LLM works with outlier-based optimization. While outleer-based optimization can also be used for pruning LLMs [84 (85, they are two-stage methods constrained to dealing with the parameters of only one LLM, while we focus on the outlier weights of different LLMs during end-to- ro-end model merging. This means should
Table Comparison with other outlier-based
LLM works on Llama3.
Method
Helpfulines
Honesly
Harmlesaess
Noem, Gain
Llama3-88 -n-strrt
58.79
$3.50
59.07
TSVM
10:30r
20:0 saoe
60 1x00
+12,99%
M-inda 51
59.3
10.35
78.75
+13.175
59.41 t1xer
56.40
79.15 33535
+13,47%
RESM (ours)
59.62
58.20 28.200
79.60-14356
+14.97%
additionally consider the parameters conflict while merging different LLMs, rather than post-hoc process well-merged L.M., To further distinguish our contribution, as shown in Table we conduct outlier-related experiments on LJama3, including two representative outlier-based pruning methods, Wanda [82] and Owl [83]. We can observe that RESM can consistently achieve better results.
Ablation Studies. As shown in Table 6 and Table 7 we can observe that integrating both outlier weighting and adarsity-adapiive rank selection can collectively enhance the merging effect, which can verify the responsibility of our reweighting based optimization. Notably, RESM outperforms data mixture baselines, achieving gain of close to 1.5x to 2x improvement. These results validate model merging as a viable pathway for LLM alignment towards balancing multi-dimensional objectives
Table 6:
Ablation studies for Reweighting- Induced Improvements on Llama3.
Table 7: Ablation studies for Reweighting-
Induced Improvements on Mistral.
Method
Helpfhinees
Honesty
Harmllsaness Norm Gain
Method
Helphitores
Honrsty
Harmlesmno Norm Gain
Liyma' Intract
58.79
53.50
59,07
Mistral 7il instmcc-9022 90.2
41.70
62.17 13
18
thesi mixuur))
Houmer misturs)
22.50-.0s
62.05
78.57 2uar
TSVM
59 59.30
56 ais
+12,595
RISM wis Outlier Weighning
59.5 59.521130
79.5554559
+13.00%
RISM Outlee
42:903x05
6180
61.25
+2.89%
RESM wis Rank Selection
36.0 29.45
ms
79 79.05
+13.71%
RESM wio Runk Solection
43.52-
61.20. ses
81.75 -204
43.128
RFSM (oune)
50 cm
58 20 mp
79.60-14270
-14.7%
RESM (ouns)
43.15-1495
61.50, ass
62.25 ma0r
+3.36%
6 Conclusion
This paper establishes the first benchmark to systematically compare data mixture and model merging methods for balanced optimization across helpfulness, harmlessness, and honesty dimensions to en- hance LLMs' alignment. Leveraging this benchmark, we uncover a series of overlooked optimization principles and insights. Specifically, we propose a novel Reweighting- Enhanced Task Singular Merg- ing (RESM) method, which employs outlier weighting and sparsity-aware rank selection strategies to address preference noise accumulation and layer sparsity adaptation challenges during L.MM merging for 3H objectives. Our theoretical analyses and experimental results provide a promising pathway for LLM alignment, advancing the development of ethically constrained language models.
9

---

ACKNOWLEDGMENTS
This work was supported in part by the National Key Research and Development Program of China (2024YFE2003700) "Pioneer" and "Leading Goose" R&D Program of Zhejiang (2025C02037), and the Starry Night Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SN-ZJU-SIAS-0010) Moreover, this work was also supported by Ant Group Research Intern Program.
References
[1] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[2] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025.
[3] ByteDance Seed, Yufeng Yuan, Yu Yue, Mingxuan Wang, Xiaochen Zuo, Jiaze Chen, Lin Yan, Wenyuan Xu, Chi Zhang, Xin Liu, al. thonkitiniing--... 5: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504. 73914, 2025.
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2rX1v:2204.05862, 2022.
[5] Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, et al. Controllable preference optimization: Toward controllable multi-objective alignment. arXiv preprint arXiv:2402.19085 2024.
[6] Shashank Sonkar, Kangqi Ni, Sapana Chaudhary, and Richard G Baraniuk. Pedagogical alignment of large language models. arXiv preprint arXiv:2402.05000 2024.
[7] Shu Yang, Jiayuan Su, Han Jiang, Mengdi Li, Keyuan Cheng, Muhammad Asif Ali, Lijie Hu, and Di Wang. Dialectical alignment: Resolving the tension of 3h and security threats of 1lms. arXiv preprint
2024.
[8] Yuntao Bai. Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:22 0800773 2022.
[9] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12771 2023.
[10] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[11] Jiaming Ji. Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. Pku-saferlhf: Towards multi-level safety alignment for 1lms with human preference. arXiv preprint arXiv:2406. 220x::2300.15513,, 2024.
[12] Youcheng Huang, Jingkun Tang, Duanyu Feng. Zheng Zhang, Wenqiang Lei, Jiancheng Lv, and Anthony G Cohn. Dishonesty helpful and harmless alignment. arXiv preprint arXiv:2406 20Xx:2406.01131, 2024.
10

---

[13] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et T," ulu Pushing frontiers in open language model post-training. arXiv preprint 2Xiv:2411.15122 2024.
[14] Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-expert.. arXiv preprint arXiv:2406. /2845, 2024.
[15] Li Jiang, Yusen Wu, Junwu Xiong, Jingqing Ruan, Yichuan Ding, Qingpei Guo, Zujie Wen, Jun Zhou, and Xiaotie Deng. Hummer: Towards limited competitive preference dataset. arXiv preprint arxiv: 1647, 2024.
[16] Chuxue Cao, Han Zhu, Jiaming Ji, Qichao Sun, Zhenghao Zhu, Yinyu Wu, Juntao Dai, Yaodong Yang. Sirui Han, and Yike Guo. Safelawbench: Towards safe alignment of large language models. arXiv preprint arXiv:2506.066333 2025.
[17] Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
[18] Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in Ilms, milms, and beyond: Methods, theories, applications and opportu- nities. arXiv preprint vrXiv:248.006666. 2024.
[19] Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401. 12187, 2024.
[20] Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre- Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. Warp: On the benefits of weight averaged rewarded policies. arXiv preprint arXiv:2406.1677 2024.
[21] Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise the alignment performance of generative language models. arXiv preprint urXiv:2404.0982. 2024.
[22] Guangyuan Shi, Zexin Lu, Xiaoyu Dong, Wenlong Zhang, Xuanyu Zhang. Yujie Feng. and Xiao-Ming Wu. Understanding layer significance in Ilm alignment. arXiv preprint arXiv:2410. 17875, 2024.
[23] Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. Safety layers in aligned large language models: The key to Ilm security. arXiv preprint arXiv:2408. 17003, 2024.
[24] Arash Ahmadian, Seraphina Goldfarb-Tarran,, Beyza Ermis. Marzich Fadace, Sara Hooker, et al. Mix data or merge models? optimizing for diverse multi-task learning, arXiv preprint arXiv:2410. 10801, 2024.
[25] Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, Zachary Yahn, and Ling Liu. H3 fusion: Helpful, harmless, honest fusion aligned 1lms. arXiv preprint arXiv:2411. 17792. 2024.
[26] Atoosa Chegini, Hamid Kazemi, Seyed Iman Mirzadeh, Dong Yin, Maxwell Horton, Moin Nabi, Mehrdad Farajtabar, and Keivan Alizadeh. Model soup for better rlhf: Weight space averaging to improve alignment in llms. In NewrIPS 2024 Workshop on Fine- Taning in Modern Machine Learning: Principles and Scalabiliry, 2024.
[27] Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. arXiv preprint 2rXivv2404.09656. 2024.

---

[28] Keming Lu, Bowen Yu, Fei Huang, Yang Fan. Runji Lin, and Chang Zhou. Online merging opti- mizers for boosting rewards and mitigating tax in alignment. arXiv preprint (rXiv:2455.17911 2024.
[29] Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. arXiv preprint 2rX1v:2401.10491, 2024.
[30] Ziyi Yang, Fanqi Wan, Longguang Zhong, Tianyuan Shi, and Xiaojun Quan. Weighted-reward preference optimization for implicit model fusion. arXiv preprint arXiv:2412.03187 2024.
[31] Tzu-Han Lin, Chen-An Li, Hung-yi Lee, and Yun-Nung Chen. Dogerm: Equipping reward models with domain knowledge through model merging. arXiv preprint arXiv:2407.01470. 2024.
[32] Megh Thakkar, Yash More, Quentin Fournier, Matthew Riemer, Pin- Yu Chen, Amal Zouaq, Payel Das, and Sarath Chandar. Combining domain and alignment vectors achieve better knowledge-safety trade-offs in llms. arXiv preprint (tl::2411.06822. 2024.
[33] Hasan Abed Al Kader Hammoud, Umberto Michieli, Fabio Pizzati, Philiip Torr, Adel Bibi, Bernard Ghanem, and Mete Ozay. Model merging and safety alignment: One bad model spoils the bunch. arXiv preprint arXiv:2406. 14563, 2024.
[34] Jinluan Yang. Anke Tang, Didi Zhu, Zhengyu Chen, Li Shen, and Fei Wu. Mitigating the backdoor effect for multi-task model merging via safety-aware subspace. arXiv preprint arXiv:2410. 139/0, 2024.
[35] Yiming Ju, Ziyi Ni, Xingrun Xing, Zhixiong Zeng. Siqi Fan, Zheng Zhang, et al. Mitigat- ing training imbalance in Ilm fine-tuning selective parameter merging. arXiv preprint vxXiv:240.03374. 2024.
[36] Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. Speciality generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprint 230.:2309.06256. 2023.
[37] Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang. Wenbin Hu, Hanning Zhang, et al. Mitigating the alignment tax of rlhf. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pages 580-606, 2024.
[38] Xin Yi, Shunfan Zheng, Linlin Wang, Xiaoling Wang, and Liang He. A safety realignment framework via subspace-oriented model fusion for large language models. arXiv preprint 3XXv:2445.09955. 2024.
[39] Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannanch Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Per- sonalized large language model alignment via post-hoc parameter merging. arXiv preprint 1xXi:22100.1566. 2023.
[40] Weixiang Zhao, Yulin Hu, Zhuojun Li, Yang Deng, Yanyan Zhao, Bing Qin, and Tat-Seng Chua. Towards comprehensive and efficient post safety alignment of large language models via safety patching, arXiv preprint arXiv:2405. 13820, 2024.
[41] Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey, arXiv preprint 2307 12966, 2023.
[42] Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy 1lms: survey and guideline for evaluating large language models' alignment. arXiv preprint 2rXiv:2308.000 05374, 2023.
12

---

[43] Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Tianyi Alex Qiu. Juntao Dai, and Yaodong Yang. Aligner: Efficient alignment by learning correct. Advances in Neural Information Processing Systems, 37:90833-90888 2024.
[44] Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safery-tuned Ilamas: Lessons from improving the safety of large language models that follow instructions. arXiv preprint arXiv:2309.0777 07770777..7... 2023..
[45] Avinash Amballa, Durga Sandeep Saluru, Gayathri Akkinapalli, Abhishek Sureddy, and Akshay Kumar Sureddy. Safe to serve: Aligning instruction-tuned models for safety and helpfulness. arXiv preprint urXiv:24/2.00077 2024.
[46] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 394344) 1952.
[47] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
[48] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.00 09288,, 2023.
[49] Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.00 1.09528, 2023.
[50] Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with oulti-objective rewards. arXiv preprint 2rXiv:2402.1857/ 2024.
[51] Jing Yu, Yibo Zhao, Jiapeng Zhu, Wenming Shao, Bo Pang, Zhao Zhang, and Xiang Li. Text detoxification: Data efficiency, semantic preservation and model generalization. arXiv preprint 0rXi0:2507.01050 2025.
[52] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ulrrfeedback: Boosting language models with high-quality feedback, 2023.
[53] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 06:59008-59033. 2023.
[54] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint wwx.i::2212.40055 2022.
[55] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
[56] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. Forty-first International Conference on Machine Learning, 2024.
[57] MohammadReza Davari and Eugene Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks. In European Conference on Computer Vision, pages 270-287.80 Springer, 2025.
[58] Pala Tej Deep, Rishabh Bhardwaj, and Soujanya Poria. Della-merging: Reducing interference in model merging through magnitude-based sampling. arXiv preprint arXiv:2406.1161 2024.
13

---

[59] Dong- Long-Hwan Jang, Sangdoo Yun, and Dongyoon Han. Model stock: All we need is just few fine-tuned models. In European Conference on Computer Vision, pages 207-223, Springer, 2025.
[60] Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà. Task singular vectors: Reducing task interference in model merging. arXiv preprint arXiv:2412.0008/, 2024.
[61] Daniel Marczak, Simone Magistri, Sebastian Cygent, Bartlomiej Twardowski, Andrew D Bagdanov, and Joost van de Weijer. No task left behind: Isotropic model merging with common and tasc-fpe subspaces. arXiv preprint arXiv:2502. 04999, 2025.
[62] Chanhyuk Lee, Jiho Choi, Chanryeol Lee, Donggyun Kim, and Seunghoon Hong. Adarank: Adaptive rank pruning for enhanced model merging. arXiv preprint arXiv:2503.22175, 2025.
[63] Lujun Peijie Dong, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering sparsity allocation for layer-wise pruning of large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[64] Zhenhong Zhou, Haiyang Yu. Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. How alignment and jailbreak work: Explain Ilm safety through intermediate hidden states, arXiv preprint wex.vv2406.05666 2024.
[65] Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606-610. IEEE, 2007.
[66] Jondurbin. Dataset.
2024.
[67] Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorea: An open dataset of gpt augmented flan reasoning traces. https://https:// 2023.
[68] Luigi Danicle and Suphavadeeprasit. Amplify-instruelt Synthetically generated diverse multi- turn conversations for elficient I1m training. arXiv preprint arXiv:(coming soon), 2023.
[69]
Jondurbin. Dataset. 1 2024.
[70] Weixin Chen, Dawn Song, and Bo Li. Grath: gradual self-truthifying for large language models. arXiv preprint arXiv:2401. 12292, 2024.
[71] Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving Ilm helpfulness and harmlessness with rlaif, November 2023.
[72] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405. 24Xx::2405.14744 2024.
[73] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llaaa 3 herd of models. arXiv preprint mtxXi:24477.17888 2024.
[74] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[75] Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.132511 2024.
[76] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. International conference on machine learning, pages 29965-23998, PMLR, 2022.
14

---

[77] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024.
[78] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023.
[79] Zhiying Zhu, Yiming Yang, and Zhiqing Sun. Halueval-wild: Evaluating hallucinations of language models in the wild. arXiv preprint arXiv:2403.0444 04300003037... 2024.
[80] Lijun Li, Bowen Dong, Ruohui Wang. Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint arXiv:2402.05044 2024.
[81] Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. Or-bench: An over-refusal benchmark for large language models. arXiv preprint (eXxv:2405.204477 2024.
[82] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
[83] Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsich, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, al. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms to high sparsity. arXiv preprint arXlv:2310.05175, 2023.
[84] Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. Model compression and efficient inference for large language models: A survey. arXiv preprint arXiv:24022.0944. 2024.
[85] Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang. Zhihang Yuan, Xiuhong Li, et al. A survey efficient inference for large language models. arXiv preprint arXiv:2404. 14294, 2024.
[86] Anke Tang, Li Shen, Yong Luo, Shuai Xie, Han Hu, Lefei Zhang, Bo Du, and Dacheng Tao. Smile: Zero-shot sparse mixture of low-rank experts construction from pre-trained foundation models. arXiv preprint arXiv:2408. 10174, 2024.
[87] Shenghe Zheng and Hongzhi Wang. Free-merging: Fourier transform for model merging with lightweight experts. arXiv preprint arXiv:2411.16815 2024.
[88] Enneng Yang. Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:2310.02575, 2023.
[89] Amin Heyrani Nobari, Kaveh Alimohammadi, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, and Navid Azizan. Activation-informedd merging of large language models. arXiv preprint arXiv:2502.000 02421, 2025.
[90] Shuqi Liu, Han Wu, Bowei He, Xiongwei Han, Mingxuan Yuan, and Linqi Song. Sens- merging: Sensitivity-guided parameter balancing for merging large language models. arXiv preprint arXiv:2502. 2025.
[91] Ziyu Zhao, Yixiao Zhou, Zhi Zhang, Didi Zhu, Tao Shen, Zexi Li, Jinluan Yang, Xuwu Wang, Jing Su, Kun Kuang, al. Each rank could be expert: Single-ranked mixture of experts lora for multi-task learning. arXiv preprint arXiv:2501.15 1513, 2025.
[92] Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, and Yaoqing Yang. Model balancing helps low- training and fine-tuning. arXiv preprint mmXiv22410.12178 2024.
[93] Yuanzhe Hu, Kinshuk Goel, Vlad Killiakov, and Yaoqing Yang. Eigenspectrum analysis of neural networks without aspect ratio bias. arXiv preprint (rXi:22506.06280, 2025.
[94] Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. arXiv preprint arXiv:2105.0660 2021.
15

---

[95] Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell' Orletta. Outliers dimen- sions that disrupt transformers are driven by frequency. arXiv preprint arXiv:2205.11380 2022.
[96] Pengxiang Li, Lu Yin, Xiaowei Gao, and Shiwei Liu. Owlore: Outlie--wiighe layer- wise sampled low-rank projection for memory-efficient Ilm fine-tuning. arXiv preprint arXi::2405.18380 2024.
[97] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangehi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403. 13372, 2024.
[98] Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Fusionbench: A comprehensive benchmark of deep model fusion. arXiv preprint Drxiv:2400.032200 2024.
[99] Kairong Han, Kun Kuang, Ziyu Zhao, Junjian Ye, and Fei Wu. Causal agent based on large language model. arXiv preprint arXiv:2408. 06849, 2024.
[100] Kairong Han, Wenshuo Zhao, Ziyu Zhao, JunJian Ye, Lujia Pan, and Kun Kuang. Cat: Causal attention tuning for injecting iinegrained causal knowledge into large language models. arXiv preprint MrXiv:2509.01533 2025.
[101] Linxiao Yang, Yunze Tong, Xinyue Gu, and Liang Sun. Explain temporal black-box models via functional decomposition. In Forty-first International Conference Machine Learning.
[102] Xinpeng Dong, Min Zhang, Didi Zhu, Ye Jun Jian, Zhang Keli, Aimin Zhou, Fei Wu, and Kun Kuang. Erict: Enhancing robustness by identifying concept tokens in zero-shot vision language models. In Porty-second International Conference on Machine Learning.
[103] Xiangwei Lv, Mengze Li, Jingyuan Chen, Zhiang Dong. Sirui Han, and Beishui Liao. Out-of- distribution detection via I1m- Ilm-guided outlier generation for text-attributed graph. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19544-19555, 2025.
[104] Zijing Hu, Fengda Zhang, Long Chen, Kun Kuang, Jiahui Li, Kaifeng Gao, Jun Xiao, Xin Wang, and Wenwu Zhu. Towards better alignment: Training diffusion models with reinforce- ment learning against sparse rewards. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 23604-23614, 2025.
[105] Zijing Hu, Fengda Zhang, and Kun Kuang. D-fusion: Direct preference optimization for aligning diffusion models with visually consistent samples. In Forty-second International Conference on Machine Learning. 2025.
[106] Yunze Tong, Fengda Zhang, Didi Zhu, Jun Xiao, and Kun Kuang. Decoding correlation- induced misalignment in the stable diffusion workflow for text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18187- 18196, 2025.
[107] Yunze Tong, Didi Zhu, Zijing Hu, Jinluan Yang, and Ziyu Zhao. Noise projection: Closing the prompt-agnostic gap behind text-to-image misalignment in diffusion models. arXiv preprint 14.2:25/00.4556, 2025.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clarify our contribution from the benchmark establishment, phenomenon, and principle exploration, and technique contribution the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We clarify the contribution in the Appendix D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
We provide detailed proof in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the details for experiments in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
We provide the detailed implementation in Appendix C.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ Caaeaes//ooSSSuumiiseeee seipaiisssooooioyyyo. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the detailed implementation in the experimental part and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the detailed implementation in the experimental part and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the detailed implementation in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////eerrr....co//////////a
Answer: [Yes]
Justification:
We provide the detailed implementation in the experimental part and Appendix
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: provide the detailed description in the Appendix D. Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Guidelines: The paper does involve crowdsoureing nor research with human subjects.
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at Ieast the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
23

---

The appendix is structured into multiple sections, each offering supplementary information and further clarification on topics discussed in the main body of the manuscript.
More Details for Method
24
More Details for Outlier-Aware Weighting
24
A.2 More Details for Dynamic Rank Selection
25
A.3 Order of Orthogonalization and Rank Selection
25
B
More Details for Related Work
26
B.1 Discussion with the Alignment Tax.
................................
26
B.2 Discussion with the Other Model Merging Methods
................................
26
B.3 Discussion with the Buttier-Basee Sparse LLM Works
27
C More Details for Experiments
27
The Training Details for Model Constructions and Baselines
27
The Evaluation Details for the Judged Models
28
More Experiments under the Continual DPO Training Settings
28
Hyper-Parameter Analysis
29
Broad Impact and Limitation
A More Details for Method
A.1 More Details for Outlier-Aware Weighting
Interpretation of Dual Objectives for outlier weighting The mathematical framework achieves cross-model consensus and intra-model saliency through its hierarchical thresholding mechanism:
(i) Cross-Model Consensus: The denominator in Eq. (3) normalizes each model's contribution by the total sparse outlier magnitude across all n models:
[HRESHOLL(AA A ( l.o (D) 3o!"
(10)
This forces models with greater sparse deviation magnitudes (potential task conflicts) to receive proportionally reduced aggregation weights af effectively suppressing outlier-ommnated models in the merged output.
(ii) Intra-Model Saliency: The 30 threshold in THRESHOLD(A A
(i) MC
30 (i) implements statisti-
cal outlier detection within each model's parameter distribution. For chussian-distributed ( (per Central Limit Theorem), this retains only the top 0.3% extreme deviations that likely correspond to:
Task-specific knowledge carriers (A > μ + 3r)
Catastrophic interference sources (A < μ - 30)
The L1 norm aggregation Ed ||-1/a then amplifies layers containing concentrated outlier parameters.
The normalization in (i) prevents any single model's outliers from dominating the merger, while the saliency detection in (ii) preserves critical task-specific features within each model. This dual mechanism reduces interference by selectively blending statistically significant parameters across models.
Synergistic Effect:

---

A.2 More Details for Dynamic Rank Selection
Our method provides enhanced guarantees through statistical awareness and adaptive computation.
(1) Conflict Probability Bound Let Poonflicr denote the probability of directional conflicts in layer I. Our rank adaptation yields as follows. We can observe that, compared TSVM's fixed _ our bound adapts to layer sparsity.
va."
E P Peonflict
(11)
ki
Vid(rro + 7821]
(ii)Theorem Proof for Conflict Probability Bound: Let (u, v) E R be random unit vectors representing Task A's and Task B's optimization direction after projection. The conflict probability is defined as follows, assisted by similarity, where is the conflict threshold (usually set 0.3):
P P(cos 0 c) P((u, a) c)
(12)
We can calculate the concentration onthe Hypersphere by Lévy's 's concentration lemma as follows:
P[Ju'v] 2 1) S 2e 2ith/122
(13)
For previous fixed rank: We can bound the expected conflict probability:
/ P([uTv|2 l)dl
E P|
(14)
/ 2e 2e kit 12 + 1 2e 12
(15)
when E (0, c), this term S 26;
when / E (e, 1), this term < 2(1 - e)e -he?
when we choose
Ver it becomes:
2
2.5
Pconflic] <
Vkn
2//2/h
Vki
For our adaptation rank, we can substitute the adaptive rank as follows:
h [de(7r
7941
Thus, we can conclude:
2.5
E Peontlicl
[di(7 +74)
The key insight includes two parts:
On the one hand, in A unit task vectors become increasingly orthogonal (evaluated by the dot product |u v/) as k oo
On the other hand, sparsity adaptation controls this effect
A.3
Order of Orthogonalization and Rank Selection
critical design in our RESM algorithm lies in the sequential relationship between orthogonalizatton (Eq. 2 and rank selection (Eq. (9. Through theoretical analysis and empirical validation,
establish that orthogonalization should precede selection to ensure optimal subspace alignment and information preservation. This ordering stems from three fundamental considerations below:
(i) Global Orthogonality Constraints: The orthogonal projection in Eq. minimizzs the Frobenius norm difference | UillF under strict orthogonality constraints. Performing this projection
25

---

Table 8: Theoretical Comparison between our proposed RESM and TSVM.
Property
TSVM
RESM
Layer adaptivity
X
Sparsity awareness
Conflict bound
((d
Oad (n0 +
Weight concentration
Uniform O(d²)
Heavy-tailed Okkd))
Comp. complexity
before selection preserves the complete singular vector structure, enabling accurate modeling of cross-task interference patterns. Early selection would discard essential components for constructing the orthogonal basis, particularly when uusk-specific updates exhibit heterogeneous rank distributions.
(iii Dynamic Rank Adaptation: Our sparsity- rank selection (Fq. D requires layer-wise
sparsity measurement Ωr computed from the full parameter deviation matrix A (x) Truncating A (1)
prematurely would bias Ω4 by excluding contributions from low- wow-manitudee parameters, thereby undermining the adaptive rank calculation. As shown in Algorithm !| orthogonalization (Step 4)
utilizes the full-rank SVD decomposition to maintain statistical fidelity.
The outlier-aware weighting mechanism (Eq. 6) operates on the complete parameter deviation matrix to identify statistically significant updates. Rank selection prior outlier detection would risk eliminating subtle yet critical features masked within lower-rank components, particularly in layers with heavy-tailed parameter distributions.
(iii) Outlier Weighting Integrity:
B
More Details for Related Work
B.1 Discussion with the Alignment Tax.
Pre-trained LLM SFT-R
Initially Aligned Instruct LLM 0nstrruct
RLHEDPO
Further Aligned Instruct LLM
0 pre
Optimization
@ instruct
Figure 4: Illustration of Training Stage of 3H Optimization, which aims to further enhance LLMs' alignment from three perspectives based on the existing Initially Aligned LLMs.
We would like to further clarify the main difference between the 3H trade-off and the previously defined alignment tax (3). In general, the alignment tax describes the phenomenon of RLHF training leading to the forgetting of pre-trained abilities during the first alignment stuge. However, as shown in Figure 4 we mainly focus on how we can further enhance the 3H-related abilities of the existing already-aligne
model during the second or subsequent stages. The trade-off mainly comes from the conflict of different alignment objects without dealing with the pre-trained knowledge. Take the Llama3 series for example, alignment tax mainly analyzes the pre-trained ability degradation on the SFT version of the Base LLM (e.g., train the Llama-3-8B on the Ultrachat) while performing DPO training, which refers to the green arrow of the Figure 4 However, in this paper, we mainly focus how (e.g. Llama3-8B- Caamaa--B-Inntruuue IB-Instruc during the second or subsequent alignment stages (orange arrow of the Figure 4 which can meet more strict demands for specific applications.
further enhance the 3--related abilities of the existing already aligned model
B.2 Discussion with the Other Model Merging Methods
To further distinguish our work from previous ones and strengthen our contribution, we provide more detailed discussions about the other model merging methods.
MOE-based merging works need additional input data train the router: These works, such as SMILES [86), Free-Merging (87) and Twin-Merging [87), aim to balance the performance and deployment costs through modular expertise identification and integration adapted to the input data, which is not designed for 3H optimization in LLM alignment. Recently, have noticed concurrent
26

---

MOE-fusion work called H3 fusion [25] related to our theme. It includes three main steps:(i) Adopt the instruction tuning and summarization fusion as two modern ensemble learning the context of helpful- saaullhammleesooneeee (H3) alignment (ii) Merge the aligned model weights with an expert router according to the type of input instruction and dynamically select subset of experts. (iii) Utilize the gating loss and regularization terms to enhance performance. But our work mainly focuses on how we can address the conflict issue for 3H optimization to construct a multi-object aligned LLM rather than dynamically adapting to the input data. Simultancously, considering that the constraints of data availability and data leak will limit the generalization of existing merging methods for LLMs, in the paper we mainly adopt the well-known and latest training-fres and data-free merging strategies for dense LLM, while H3 fusion needs the data for training and only utilizes the merging techniques for efficiently adapting to the input data. Thus, H3 fusion is indeed different from our work from the perspective of problem and technique contributions.
Other fraining- merging works need additional data for test-time adaptation optimization: These works, such as Adamerging 88 AlM-megging [89] cense-merginge [90]. Adarank [62). DAM 34]. utilize the fest-time-adaptation techniques to search for the optimal merging coefficient or prune the rank [9]]. Their effectiveness depends heavily on the provided test data. But for 3H optimization, curating high-quality preference data that meets the demand of helpfulness, harmlessness, and honesty simultaneously is difficult due to the complex collective and conflict relationships as stated above. We also need to consider the data mixture problems during test-time adaptation optimization. In this case, we can't 't compare data mixture and model merging methods for 3H optimization fairly. That's why we only compare the training- free model merging methods in our experimental parts.
B.3 Discussion with the Outlier-Based Sparse LLM Works
To further distinguish our work from previous ones and strengthen our contribution, we provide more detailed discussions about the outlier-based based sparse LLM works 92 93].
Many works investigate the outlier weight in transformer 99 95) and propose to prune LLM assisted by input activations [82 3 or sample layer-wise weight during fine-tuning (96]. From the perspective of outlier weight source, the outlier weight updates we addressed are due to the preference noise accumulation while merging different aligned LLMs, which a special problem for merging for multi multi-objective alignment. From the perspective of the status of the training process, previous outlier-basee sparsity LLM works are only constrained to the parameters of one LLM 84 85], while we should additionally consider the parameters conflict while merging different LLMs in the process, rather than a post-hoe process on a well-merged L.M.. That's why we first perform SVD analysis to separate task-specfic parameters and only adopt outlier-weighting on the singular value.
C
More Details for Experiments
C.1 The Training Details for Model Constructions and Baselines
Training hyperparameters for model constructions: following SimPO 772,, based on Llama-3-8B- Instruct and Bessral--B-Instrrce V2, we conduct preference optimization adopting the fixed batch size 128 for I epoch training with the Adam optimizer. We set the max sequence length to 4096 and apply a cosine learning rate schedule with 10 percent warmup steps for each dataset. Specially, we adjust 3 E 10.1, 0.5, 1.0, 2.0) and learning rate Ir E [3e 7, 5e for model constructions and report the best individual training models corresponding to different annotation dimensions.
The Implementation of Baselines: For Heuristic data mixture methods, we control the ratio between Honesty& Harmlessness and Helpfulness 1/5, and 1/20 by default and report the best average score (usually 1/10 according to our experiments). For ArmoRM, we follow the process of SimPO 722 to achieve refined full mixture data. For hummer 133, we refine the alignment dimension conflict (ADC) among preference datasets, leveraging the powerful ability of AI feedback (e.g., GPT4) as the paper stated. For the full mixture datasets of Table I
we control the ADC lower than 20 percent.
Computation environment: All of our experiments in this paper were conducted on 16xA100 GPUs based on the Llama-Factory 997 MergeKit (75) and fusion bench (98).
Reproducibility: We have made significant efforts to ensure the reproducibility of our work. Upon acceptance, we will release all of the trained models and the complete training and testing code to
27

---

90
70
70
Perfrrmnce
Performence
80
68
40
30
30
Random Sead
Random Sead
(a) DARE-Ties
(b) RESM
Figure 5: Comparisons between the random sparsification strategy (e.g.DARE-Ties) and SVD-based strategy (RESM on Mistral under static optimization settings adopting different seeds. RESM can achieve more stable results than random sparsification methods.
facilitate the full reproducibility of our results. We are committed to advancing this work and will provide updates on its accessibility in the future.
C.2 The Evaluation Details for the Judged Models
We provide detailed descriptions for the evaluation that needs the judged models. For MT-Bencch, we report scores following its evaluation protocol grade single answers from to 10 scores assisted by GPT4. For HaluFval--Wild given prompts to our trained model, we utilize the judged model to check whether the output of our trained model is a hallucination or not and then calculate the no hallucination rate. Similarly, we utilize the prompts from SaladBench and OR-Bench to instruct our trained models and then let the judged models check whether the replies of our trained models are safe/unsafe or refusal/answer. Based on the check results, we can naturally calculate the safe score and refusal score by counting all results. The detailed descriptions of the evaluation can be shown in Table 9
More details can be shown in the original paper.
Table Evaluation details corresponding judge models, scoring types, and metries.
Evaluation Datasets
Examples
Judge Models
Scoring Type
Metrics
MT-Bench (78
80
GPT-4
Single Answer Grade
Rating of 1-10
HaluEval-Wiin Wild 79
500
GPT4
Classify & Calculate Ratio
Rating of 0-100
SaladBench 80
1817
MD-Judge-VO.2
Classify & Calculate Ratio
Rating of 0-100
OR- OR-Bench (8]
1319
GPT4-0
Classify & Caleulate Ratio
Rating of 0-100
C.3 More Experiments under the Continual DPO Training Settings
As shown in Table II we provide additional results under the continual training settings. Through comparison results between different training stages, we can observe that the honesty, helpfulness,
Table 10: 3H Results Llama3 Under Continuous Optimization Setting. The normalized gain metric is the average value of relative gain for each dimension compared with the results of Llama3- 8B-Instruct.
Matholk
Hripldees
InstrtL Dlowe, Harndow, Norra
Von GEMER ARC-T.
ABCIC
NIVUL MIPF, FIe. Hamustiol Fla NIT Beca
TURTTO WTd (EEV Blnck Biec))
Llaned 60 Iestrant
28.00
20:09
03.00
$2.09
$5.30
$9.59
35455
5.25
$3.90
$1.16
35.37
$8.35
$3.50
59.00
Cantieal FRQ Tuowey Stnge)
74
76.00
64 (m
81 05
68 11
$1.19
96.00
3.35
52.33
62.54
17.55
+43 011000
Slage)
Wegth Avcrus
25.5
2153
10.00
$2.37
55.43
25.41
35,85
5,35
35.33
85.28
02.55
33 3525000
78 2A122tom0
40/795
Movil fock
28
2015
09.5
13.33
10
1314
3.35
$3.90
$1.35
34.55
04 06
00.00.04
1 1.00%
Tolk Antrack
7519
55.12
25.18
13.0
$3.39
15.35
7231
$2.90
$5.55
13.59
12
45233/9
7142
11.0
3003
15
96
TARE
36
0966
03.6
03
DELLA
2748
71.9
10.12
$318
11.11
48.13
46.36
5.15
$1.90
12.35
$7.13
55 110
11 18
+22875
3590
7663
09:1
13:1
65.15
5553
500
3.85
$7.80
$135
13.59
03
30 20
+41 11/00%
TEVAL
2536
7699
11.1
15.17
51.1
3.40
$3.90
1978
15.4
5185
31.700005
+33477
2979
76.7
0966
13.3
1537
1545
9610
0.57
54.00
$035
14
+3456%
28

---

Table 11: 3H Results Mistral Under Continuous Optimization Setting. The normalized gain metric the average value of relative gain for each dimension compared with the results of Mistral-7B. Instruct- V2.
Merbade
Heiphhum VOGU
Hosaat
Hiiged Ang
Thear Arg
Taamdor Avg
Nom, Gidh
CSUNC
ABCO
AMCC
Deudi
Miraak 1e
66.17
03:36
13.16
$350
35
1.5
12.7
9:.7
54.60
+1.79
10.77
75.38
(iO Traing
6.55
63:54
13:01
1.00
25.33
1.50
0133
90.6
11.31
11
00/3
17001
01:01
12.14
9.21
14.13
1.66
11
0117
$1.05
13:08:10
11.7%
19310
8314
0112
13.33
1138
3.9
1171
1.0
8.30
08.33
01.05
1130115
611
4190
1500
F131
1.0
1131
8134
0130-90
-119
31
53.0
1818
1337
11:01
0150
1.00
1635
Ter tahe
15.56
03:3
04.0
15.0
6653
31.0
1.
1.5
83.00
70.0
01336
1185730
-993
1946
11.25
15.18
1796
(16)
31
3155
140
90.5
7634
17.00
0.26
3615 m
1382
+2895
90
90
80
80
Parformence
80
80
80
40
8
30
Sparsity Rotoo y
Sparsity Ratoo y
(a) Diee-Ties
(b) RESM
Figure 6: Parameter sensitive analysis concerning sparsity factor for model merging methods on Mistral under static optimization settings.
and harmlessness of LLMs are interactively enhanced due forgetting during continual training. Morcover, model merging methods can achieve comparable results to these continual training methods without the need to consider the optimized status specific training stage. In other words, model merging paves a new way for continual DPO training, advocating training multiple models from the same start point and then merging them, rather than continually optimizing the model from the previous optimization.
C.4 Hyper-Parameter Analysis
The sparsity- -ased strategy is closely related to the merging effect. As shown in Table and Table 4 apart from the SVD-based methods, the most effective merging methods are DARE and DELLA, both of which depend on random sparsification shown in Table 2 However, studies to check the robustness and stability with respect to random seed and sparsity factors. As observe that RESM can achieve better robust results than
conduct extended
shown in Figure 5 and Figure 6 previous random sparsification-baat based methods, further verifying the effectiveness of our methods.
D Broad Impact and Limitation
Our results demonstrate that the main improvement of RESM comes from the honest and harmless aspects. This can reflect the decrease in conflict between them, which can be defined as inter-aspect conflict reduction. But for helpfulness, RESM is still worse than data mixture methods on Llama3, and the improvement on Mistral compared with the existing merging strategy is also minimal. Though the initial goal of honest and harmless training not designed for helpfulness, modern preference datasets inherently encode helpfulness as a baseline annotation, forcing the alignment process to optimize towards this dimension regardless of their primary target (honesty/narmlesseess) This means every alignment vector can represent helpfulness and one more other dimensions' optimization directions, which may lead to conflict between alignment vectors only from the helpful dimension (e.g. code and commonsense QA abilities for LLM), which can be defined as intra-dimension conflict. This phenomenon necessitates a hierarchical conflict resolution framework to improve model merging for 3H optimization, considering these two categories of conflicts simultaneously. Moreover, deploying
29

---

model merging methods for other trustworthy concerns 99 103 across more diverse circumstances
[07 106] should also be considered.
30


Reviews:
[{"review_id": "zBeGNt3Aw4", "reviewer": "Reviewer_qyir", "summary": "This paper addresses the challenge of achieving balanced alignment of large language models (LLMs) across three critical dimensions—helpfulness, honesty, and harmlessness (3H optimization)—through model merging rather than traditional data mixture strategies. While existing methods face limitations such as reliance on expert knowledge and conflicting optimization signals, the authors propose a novel parameter-level conflict-resolution framework via model merging. Their approach, termed Reweighting-Enhanced Task Singular Merging (RESM), introduces outlier-aware weighting and sparsity-adaptive rank selection to mitigate preference noise accumulation and layer sparsity adaptation issues inherent in merging 3H-aligned LLMs. Theoretical analyses and experiments on benchmarks like Llama3 demonstrate RESM’s superiority over prior data mixture and model merging techniques, achieving notable performance gains in balanced alignment metrics.  \n\nThe work further distinguishes itself by addressing gaps in training-free merging strategies, contrasting with methods like H3 fusion that require additional data for adaptation. By emphasizing reproducibility through detailed implementation guidelines and ablation studies, the authors validate RESM’s robustness in preserving task-relevant information during merging. They also contextualize their contributions within broader discussions of societal impacts, ethical considerations, and limitations, positioning RESM as a pathway for ethically constrained LLM development without compromising technical efficacy.", "questions": "1. The theoretical analysis assumes Gaussian-distributed preference noise and layer sparsity (e.g., outlier detection via Z-scores). However, real-world LLM training often exhibits heavy-tailed or non-stationary noise (e.g., power-law distributions of gradient updates). How robust is RESM to deviations from these assumptions?\n\n2. How does RESM perform on sparse or modular architectures (e.g., mixture-of-experts models) compared to dense LLMs? The paper focuses on dense models like Llama3 and Mistral, but many modern LLMs use sparse designs to improve efficiency. Does RESM’s reliance on SVD-based sparsity adaptation and outlier-aware weighting remain effective when layer-specific parameter updates are inherently sparse or distributed across experts?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**  \n\n1. The paper introduces RESM, a novel model-merging framework that explicitly addresses preference noise accumulation and layer sparsity adaptation in balancing helpfulness, honesty, and harmlessness. This fills a critical gap in existing methods, which often struggle with conflicting optimization signals or rely on expert knowledge. The combination of outlier-aware weighting and sparsity-adaptive rank selection provides a principled approach to parameter-level conflict resolution, advancing the field of ethically constrained LLM alignment.  \n\n2. By benchmarking model merging against traditional data mixture strategies, the work reveals previously overlooked collaborative and conflicting relationships among the 3H dimensions. This systematic comparison not only highlights the limitations of data-level approaches but also establishes a foundation for understanding how parameter-level strategies like RESM can achieve balanced alignment. Such insights are valuable for both practitioners and researchers aiming to navigate trade-offs in multi-objective LLM optimization.  \n\n3. The authors demonstrate RESM’s superiority through extensive experiments on benchmarks like Llama3 and Mistral, achieving measurable gains (2%-5% over data mixture, 1%-3% over prior merging methods). The inclusion of ablation studies, theoretical analyses (e.g., effective rank analysis), and reproducibility guidelines strengthens the credibility of their claims. This emphasis on empirical rigor ensures the method’s practical applicability and transparency.  \n\n---\n\n**Weaknesses:**  \n\n1. The theoretical analysis assumes that preference noise and layer sparsity follow specific statistical distributions (e.g., Gaussian for outlier detection). However, real-world LLM training dynamics often involve heavy-tailed or non-stationary distributions, which could undermine RESM’s effectiveness. The paper does not empirically validate these assumptions, leaving gaps in understanding how robust the method is to deviations from idealized conditions.  \n\n2. Although RESM avoids data-dependent training, its implementation requires fine-grained control over parameter merging (e.g., rank selection, outlier thresholds), which may demand significant computational resources and domain expertise. The paper does not thoroughly discuss the trade-offs between merging complexity and alignment performance, such as how hyperparameter tuning scales with model size or task diversity. This could hinder adoption in resource-constrained settings or real-time applications.", "was_revised": true, "final_justification": "he authors have addressed my raised concerns, and I maintain my initial rating.", "rebuttal": "Thanks for your valuable suggestions! We are glad to address your concerns one by one.\n\n**Q1: Theoretical analysis assumes Gaussian-distributed preference noise and layer sparsity**\n\nThanks for your questions. We acknowledge the outlier detection via Z-scores have the assumption of Gaussian-distributed and 3σ  may fail in a heavy-tailed distribution. But notably, for our model merging setting, the outlier have different meaning compared with traditional training.\n\n**(1) The outlier weighting is only operated on the singular vector of each alignment objective.**\n\n**(2) The outlier weighting is calculated based on the same layer of different models rather than different layers of one trained model**.\n\n**(3) This outlier noise mainly belong to the dominant factors that weaken multi-objective alignment of 3H merging rather than traditional noise for model training**. \n\n Thus, we just assume that the same layer of different models have the Gaussian-distributed due to the few parameters undergo significant changes while most exhibit minor adjustments. The design of outlier weighting can conclude as follows: For noise compression, we filter out low-magnitude fluctuations that predominantly encode noise, forcing the singular vectors to align with statistically significant task features. ensuring balanced contributions from diverse tasks and preventing dominance by high-magnitude updates that may obscure subtle yet critical features. For task equilibrium: The layer-wise aggregation outlier weights are globally normalized across all models, ensuring balanced contributions from diverse tasks and preventing dominance by high-magnitude updates that may obscure subtle yet critical features. These contents can be shown in lines182-188.\n\n**Q2: The paper does not thoroughly discuss the trade-offs between merging complexity and alignment performance.**\n\n**(1) For rank selection(related to sparsity)**, we carefully discuss the sparsity and random seed that influences the effect of merging in Figure 5 and Figure 6, we can observe that RESM can achieve better robust results than previous random sparsification-based methods, further verifying the effectiveness of our methods. The experiments that can verify the robustness of our proposed RESM can be shown in Appendix C.4. \n\n**(2) For outlier detection**, we follow the 3σ principle to cope with the preference noise accumulation shown in Figure 2. It has a stable effect on the model merging.\n\n**(3) For task diversity**, as shown in Table 1, we have conducted experiments through 5 categories of alignment objectives across 10 different datasets. The reported results can verify the effectiveness of our method.\n\n**Q3: How does RESM perform on sparse or modular architectures (e.g., mixture-of-experts models) compared to dense LLMs? Does RESM’s reliance on SVD-based sparsity adaptation and outlier-aware weighting remain effective when layer-specific parameter updates are inherently sparse or distributed across experts?** \n\nThanks for your questions. The operated object of SVD-based sparsity adaptation and outlier-aware weighting are layer-wise parameters of different models rather than different layer of one models. Thus, we believe our proposed RESM can still achieve good performance for 3H alignment. Due to the time limit, we will add the experiments later"}, {"review_id": "5FPeQJ5VrH", "reviewer": "Reviewer_HsEK", "summary": "This paper compares data mixture and model merging for aligning LLMs on Helpfulness, Honesty, and Harmlessness. It proposes RESM, a model merging method using outlier-aware weighting and adaptive rank selection. RESM outperforms prior methods in balancing 3H alignment, showing strong results across static and continual training settings.", "questions": "Please see weaknesses section.", "limitations": "- The motivation of this paper is insufficient.  \n\n- The proposed method appears to be a mere aggregation of components, lacking clear connections between them.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 3, "strengths_and_weaknesses": "#### **Weaknesses**\n\n- The proposed method is presented with considerable complexity, but it seems to be primarily a combination of existing techniques. Moreover, the relationships between the different submodules are not clearly articulated.\n\n- There is insufficient empirical motivation to support the claim that traditional merging method performs poorly under 3H multi-objective conflicts. The experimental results appear similar to those in non-3H settings, suggesting that the so-called 3H conflict might merely reflect typical parameter space conflicts in multi-model merging scenarios.\n\n- The conclusion that merging datasets is inferior to merging models seems rather obvious, and I did not observe any novel insights specific to the 3H scenario.\n\n#### **Strengths**\n\n- The writing is fluent and clear.  \n\n- The experimental section is thorough, covering a wide range of datasets and baselines.", "was_revised": true, "final_justification": "My core concerns regarding the uniqueness of 3H alignment and the proposed model merging method remain. I'm not convinced that the task conflicts shown are fundamentally different from those in other multi-task scenarios.\n\nThe reported improvements are mostly within the margin of random fluctuation and seem to be driven primarily by the OR-Bench results. This level of performance gain is not sufficient to prove the method's effectiveness.", "rebuttal": "Thanks for your valuable suggestions! We are glad to address your concerns one by one.\n\n**Q1: Motivation. There is insufficient empirical motivation to support the claim that traditional merging method performs poorly under 3H multi-objective conflicts. The experimental results appear similar to those in non-3H settings, suggesting that the so-called 3H conflict might merely reflect typical parameter space conflicts in multi-model merging scenarios.**\n\nThanks for your questions. We would like to distinguish 3H conflict with typical parameter space conflicts in previous multi-model merging scenarios as follows:\n\n**From the perspective of problem:**\n\n(i) We would like to clarify that we are the first to systematically scale the data mixture and model merging strategies for LLM 3H optimization.The detailed experimental results of LLMs across different human preference can contribute to large-scale industrial application compared with traditional multi-model merging methods; \n\n(ii) Exactly, traditional multi-model merging scenarios mainly assume we have different task-specific models for merging, which only focus on  the task conflict. But we not only consider the parameter conflict and but also consider the fine-grained alignment promotion simultaneously. As shown in Figure 3, we observe that there exist different collaborative and conflict relationships in terms of 3H objectives for LLM alignment. These findings strengthen the understanding of LLM alignment. These contents can be shown in lines 259-256.\n\n**From the perspective of technique:**\n\n(i) Due to the demand that we need to simultaneously consider the fine-grained alignment promotion and conflict, thus we shouldn't directly follow the traditional  multi-model merging methods, such as task-vector based methods(e.g. ties and dare) and Linear interpolation methods(e.g.rewarded soup). That's because the advancement of multi-model merging methods have the same assumption of separating task-common and task-specific parameter to cope with conflict. For 3H optimization of LLM alignment, the investigation analysis of Figure 6 and Broad Impact and Limitation shows that LLM alignment is a more complicated problem, we can't directly succeed this assumption, that's why we adopt the layer-wise SVD composition for 3H merging.\n\n(ii) There exists some specific problems for 3H alignment during model merging compared with traditional multi-model merging, as shown in Motivation Section 3.2, we first analyse the preference noise accumulation when we increase the number of aligned models through adjust alignment objectives and random seed. Then, we point out that conventional multi-model merging methods employ a uniform rank selection threshold $k$ across all layers, failing to account for layer-specific sparsity patterns and parameter importance heterogeneity in LLMs. These contents can be shown in lines 139-163.\n\n**Q2: Technique Details. The proposed method appears to be a mere aggregation of components, lacking clear connections between them.**\n\nAs show in Eq. 4, the outlier weighting is operated on singular vector for orthogonalization  and the rank-selection is operated on the layer-wise.  **Both of them collectively contribute to the optimal subspace alignment and information preservation during 3H optimization.** The detailed discussions between outlier-based orthogonalization and rank selection can be shown in Appendix C.3. \n\n**The design of outlier weighting is to preventing dominance for 3H optimization:** For noise compression, we filter out low-magnitude fluctuations that predominantly encode noise, forcing the singular vectors to align with statistically significant task features. ensuring balanced contributions from diverse tasks and preventing dominance by high-magnitude updates that may obscure subtle yet critical features. For task equilibrium: The layer-wise aggregation outlier weights are globally normalized across all models, ensuring balanced contributions from diverse tasks and preventing dominance by high-magnitude updates that may obscure subtle yet critical features. These contents can be shown in lines 182-188.\n\n**The design of rank selection is adapted to the LLM sparsity:** For dense layers, where parameter updates demonstrate relatively uniform distributions with predominantly small adjustments, employing lower-rank approximations proves effective for noise suppression while preserving principal components. However, this necessitates careful determination of the optimal rank selection threshold to balance between information retention and noise elimination. Conversely, sparse layers display concentrated parameter updates along a few dominant directions, potentially containing critical outlier components. Here, maintaining a higher rank becomes essential to ensure the preservation of these salient directional features, thereby preventing substantial information loss through excessive rank truncation. \n\n**Apart from achieving the outlier-weighting orthogonalization singular vector, by controling the level of preserving dominant singular directions in sparse and dense layers , we reduce overlaps between task-specific parameters, decoupling interference-prone optimization trajectories.**\n\n\n**Q3: The conclusion that merging datasets is inferior to merging models seems rather obvious, and I did not observe any novel insights specific to the 3H scenario.**\n\nThanks for your questions. Exactly, we can address the alignment conflict both from the data and parameter perspective.\n\n(i) Notably, not all merging methods achieve better results than model merging methods. For example, in Table 3, only 3/13 merging methods achieve better results than data mixture strategies.  \n\n(ii) Apart from experimental results, we aim to provide insights to analyse the 3H alignment collaborative and conflict relationships (lines 259-266) and the mechanism​ of merging for 3H merging. We provide the detailed analysis when merging is better and the factors that influence merging effect. These conclusions can contribute to the community to have a better understanding of the merging and data mixture strategies for LLM alignment.These contents can be shown in lines 286-301"}, {"review_id": "0rzChSEIIh", "reviewer": "Reviewer_jwRS", "summary": "This paper aims to solve the inherent conflicts of large language models (LLMs) in the alignment of 3H (helpfulness, honesty, harmless) multiple objectives. To this end, it establishes a comprehensive benchmark and systematically compares 15 representative methods upon this benchmark. Additionally, the authors figure out two problems in the existing merging methods and propose a novel RESM (Reweighting-Enhanced task Singular Merging) algorithm to overcome them. Extensive experiments have shown that RESM is significantly superior to existing data mixing and model merging methods in achieving 3H balance.", "questions": "N.A.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* Strengths\n1. This work conducts a systematic review on existing methods about 3H-alignment, and designs comprehensive benchmark to evaluate these methods from multiple dimensions. The extensive experiment results offer valuable insights to the community (e.g.,  a phased \"specialize-then-merge\" paradigm may be a more effective path to multi-objective alignment than a monolithic data-mixing approach).\n2. This paper proposes a new algorithm named RESM to overcome two unsolved limitations, i.e., Preference Noise Accumulation and Layer-wise Sparsity Mismatch. The design constitutes a high-quality technical contribution and is well validated by the experiments. \n\n* Weakness\n1. While assembling public resources into a structured benchmark is a valuable engineering effort, the most selected datasets in the benchmark is widely used in the community (such as MATH, GSM8k, MMLU, HumanEval, etc.). I'm not so sure whether such common combinations deserve a critical contribution of this work.\n2. The LLM landscape evolves rapidly. Although Llama-3 and Mistral are highly representative, the experiments lack validation on more recent and capable open-source models (e.g., the Qwen-2.5, DeepSeek-R1-distilled models). Evaluating existing methods and the new-proposed RESM to these cutting-edge models would more powerfully demonstrate the generalizability of this benchmark and brings more realtime insights.", "was_revised": true, "final_justification": "The author's reply addresses my concerns about this work. And I believe my current score has shown my positive attitude on this paper and will keep it.", "rebuttal": ""}, {"review_id": "l90FA1Bm0x", "reviewer": "Reviewer_UavS", "summary": "This paper systematically compares data mixture and model merging approaches for aligning large language models (LLMs) across Helpfulness, Honesty, and Harmlessness (3H) dimensions. The authors establish the first comprehensive 3H alignment benchmark and reveal that model merging, with proper conflict mitigation, can outperform data mixture strategies. To further enhance alignment, they propose a novel Reweighting-Enhanced Task Singular Vector Merging (RESM) method, which addresses preference noise and layer-wise sparsity through outlier-aware weighting and adaptive rank selection. Extensive experiments demonstrate the effectiveness of RESM in achieving balanced 3H alignment for LLMs.", "questions": "1）Mistral-7B is no longer state-of-the-art. To strengthen generality, please add experiments using a stronger baseline (e.g., Qwen-3-7B).\n2）In Table 3, individual \"Harmlessness\" training yields lower Harmlessness scores (60.14) than RESM (79.60). Why does merging non-harmlessness-specialized models (e.g., Helpfulness-trained) improve this metric beyond dedicated training?\n3）Your method's performance improvement seems to mainly take effect on honesty and harmlessness, but the evaluation datasets corresponding to these two indicators are significantly fewer than those for helpfulness. This only seems to demonstrate the effectiveness of your method for the three evaluation datasets under honesty and harmlessness, rather than for balancing the 3H (honesty, harmlessness, helpfulness). Please supplement evaluation datasets for honesty and harmlessness respectively (5 for honesty and 5 for harmlessness).", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper addresses the critical and underexplored problem of achieving balanced alignment for LLMs across Helpfulness, Honesty, and Harmlessness (3H), which is highly relevant to the current trend of safe and responsible AI development.\nThe authors establish the first comprehensive benchmark for comparing data mixture and model merging approaches for 3H alignment, covering 15 representative methods, 10 preference datasets, and 2 mainstream LLM families. This provides valuable experimental evidence and standard evaluation criteria for the field.\nThe proposed Reweighting-Enhanced Task Singular Vector Merging (RESM) method introduces a principled way to address preference noise accumulation and layer-wise sparsity heterogeneity in model merging, which are critical yet often overlooked challenges.\nWeaknesses:\nWhile RESM introduces useful improvements, it builds heavily on existing SVD-based merging frameworks (e.g., TSVM). Also, while the empirical results are strong, the theoretical foundations for why RESM consistently improves 3H alignment remain somewhat heuristic. More formal analysis or theoretical guarantees would strengthen the work.\nThe paper lacks discussion on the computational overhead and scalability of RESM. Meanwhile, the models used are not the most state-of-the-art ones, and the evaluation metrics for Honesty and Harmlessness are too single-dimensional. (See Q1-Q2)", "was_revised": true, "final_justification": "The author's response addressed my concerns and I raised my score to 4.", "rebuttal": "Thanks for your valuable suggestions! We are glad to address your concerns one by one.\n\n**Q1: More experiments on the latest models**\n\nDue to the time limit, we will add the experiments later.\n\n**Q2: More formal analysis or theoretical guarantees would strengthen the work.**\n\nThanks for your suggestions. To verify the effectiveness, in the Appendix, we provide the interpretation of dual objectives for outlier weighting from cross-model consensus and intra-model saliency, conflict probability bound for adaptive rank selection,  and the order of orthogonalization and rank selection. We will further strengthen our work referring to your suggestions.\n\n**Q3: More balanced Evaluation. This only seems to demonstrate the effectiveness of your method for the three evaluation datasets under honesty and harmlessness, rather than for balancing the 3H (honesty, harmlessness, helpfulness). Please supplement evaluation datasets for honesty and harmlessness respectively (5 for honesty and 5 for harmlessness).**\n\nExactly, the data sources of harmlessness and honesty are also diverse enough, not limited to the shown absolute number of datasets. For example, the harmlessness evaluation saladbench includes more than 9 datasets (HH-harmless, HH-red-team, Advbench, Multilingual, Do-Not-Answer, ToxicChat, Do Anything Now, GPTFuzzer). We have tried our best to make the evaluation extensive and reasonable.\n\nMoreover, in terms of the dataset quantity, the general abilities evaluation datasets themselves are superior to the trustworthy capability evaluation datasets. That's why we propose to aggregate the results for each dimension individually for a fair and reasonable evaluation\n\n**Q4: The paper lacks discussion on the computational overhead and scalability of RESM.**"}]

Author rebuttal:

Meta-review: This paper provides a systematic comparison of data mixture versus model merging strategies for aligning Large Language Models (LLMs) with the principles of Helpfulness, Honesty, and Harmlessness (3H). The authors argue that while data-level mixture methods face challenges with conflicting optimization signals, parameter-level model merging offers a promising alternative. To address the specific issues of preference noise and layer sparsity in 3H alignment, the paper introduces a novel merging method called Reweighting-Enhanced task Singular Merging (RESM). Through extensive experiments, the authors demonstrate that RESM outperforms both existing data mixture and model merging techniques, establishing a strong benchmark for future work in multi-objective LLM alignment.

The paper's primary strength, as highlighted by a majority of the reviewers, is its pioneering role in systematically investigating and benchmarking data mixture against model merging for 3H alignment. This is a critical and timely problem for responsible AI, and the comprehensive benchmark itself is a valuable contribution to the community. The proposed RESM method is technically sound, offering a principled approach to mitigate specific challenges in merging 3H-aligned models, such as preference noise and layer sparsity. Finally, the work is supported by extensive and thorough experiments that validate the effectiveness of RESM across multiple models and datasets, showing clear performance gains over numerous baselines.

The main point of contention, raised forcefully by Reviewer HsEK, is whether the "3H conflict" is a fundamentally unique problem or simply another instance of standard multi-task parameter conflict. This questions the core motivation and significance of the work, suggesting that the insights might not be specific to the 3H scenario.

I recommend accepting this paper. While the concerns raised by Reviewer HsEK regarding the fundamental novelty of the problem are valid points for discussion, the majority of reviewers (qyir, jwRS, UavS) found the paper to be a significant and valuable contribution. The primary value of this work lies in being the first to provide a large-scale, systematic empirical study of model merging for the critical task of 3H alignment. This benchmark and the corresponding analysis provide important insights for the community, regardless of whether 3H conflicts are a completely new class of problem. The proposed RESM method is technically solid and demonstrably effective. Furthermore, the authors were diligent in their rebuttal, providing new experiments on more recent models that addressed the concerns of multiple reviewers and led Reviewer UavS to raise their score.
Decision: Accept (poster)