Title: Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging
Venue: NeurIPS 2025
Accepted: yes
Keywords: Model Merging; Large Language Models; Alignment
Abstract: Achieving balanced alignment of large language models (LLMs) in terms of Helpfulness, Honesty, and Harmlessness (3H optimization) constitutes a cornerstone of responsible AI. Existing methods like data mixture strategies face limitations, including heavy reliance on expert knowledge and conflicting optimization signals. While model merging offers parameter-level conflict-resolution strategies through integrating specialized models' parameters, its potential for 3H optimization remains underexplored. This paper systematically compares the effectiveness of model merging and data mixture methods in constructing  3H-aligned LLMs for the first time, revealing previously overlooked collaborative and conflict relationships among the 3H dimensions and discussing the advantages and drawbacks of data mixture (\textit{data-level}) and model merging (\textit{parameter-level}) methods in mitigating the conflict for balanced 3H optimization. Specially, we propose a novel \textbf{R}eweighting \textbf{E}nhanced task \textbf{S}ingular \textbf{M}erging method, \textbf{RESM}, through outlier weighting and sparsity-aware rank selection strategies to address the challenges of preference noise accumulation and layer sparsity adaptation inherent in 3H-aligned LLM merging. Extensive evaluations can verify the effectiveness and robustness of RESM compared to previous data mixture (2\%-5\% gain) and model merging (1\%-3\% gain) methods in achieving balanced LLM alignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper systematically compares data mixture and model merging approaches for aligning large language models (LLMs) across Helpfulness, Honesty, and Harmlessness (3H) dimensions. The authors establish the first comprehensive 3H alignment benchmark and reveal that model merging, with proper conflict mitigation, can outperform data mixture strategies. To further enhance alignment, they propose a novel Reweighting-Enhanced Task Singular Vector Merging (RESM) method, which addresses preference noise and layer-wise sparsity through outlier-aware weighting and adaptive rank selection. Extensive experiments demonstrate the effectiveness of RESM in achieving balanced 3H alignment for LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper addresses the critical and underexplored problem of achieving balanced alignment for LLMs across Helpfulness, Honesty, and Harmlessness (3H), which is highly relevant to the current trend of safe and responsible AI development.\nThe authors establish the first comprehensive benchmark for comparing data mixture and model merging approaches for 3H alignment, covering 15 representative methods, 10 preference datasets, and 2 mainstream LLM families. This provides valuable experimental evidence and standard evaluation criteria for the field.\nThe proposed Reweighting-Enhanced Task Singular Vector Merging (RESM) method introduces a principled way to address preference noise accumulation and layer-wise sparsity heterogeneity in model merging, which are critical yet often overlooked challenges.\nWeaknesses:\nWhile RESM introduces useful improvements, it builds heavily on existing SVD-based merging frameworks (e.g., TSVM). Also, while the empirical results are strong, the theoretical foundations for why RESM consistently improves 3H alignment remain somewhat heuristic. More formal analysis or theoretical guarantees would strengthen the work.\nThe paper lacks discussion on the computational overhead and scalability of RESM. Meanwhile, the models used are not the most state-of-the-art ones, and the evaluation metrics for Honesty and Harmlessness are too single-dimensional. (See Q1-Q2)\n\nQUESTIONS:\n1）Mistral-7B is no longer state-of-the-art. To strengthen generality, please add experiments using a stronger baseline (e.g., Qwen-3-7B).\n2）In Table 3, individual \"Harmlessness\" training yields lower Harmlessness scores (60.14) than RESM (79.60). Why does merging non-harmlessness-specialized models (e.g., Helpfulness-trained) improve this metric beyond dedicated training?\n3）Your method's performance improvement seems to mainly take effect on honesty and harmlessness, but the evaluation datasets corresponding to these two indicators are significantly fewer than those for helpfulness. This only seems to demonstrate the effectiveness of your method for the three evaluation datasets under honesty and harmlessness, rather than for balancing the 3H (honesty, harmlessness, helpfulness). Please supplement evaluation datasets for honesty and harmlessness respectively (5 for honesty and 5 for harmlessness).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to solve the inherent conflicts of large language models (LLMs) in the alignment of 3H (helpfulness, honesty, harmless) multiple objectives. To this end, it establishes a comprehensive benchmark and systematically compares 15 representative methods upon this benchmark. Additionally, the authors figure out two problems in the existing merging methods and propose a novel RESM (Reweighting-Enhanced task Singular Merging) algorithm to overcome them. Extensive experiments have shown that RESM is significantly superior to existing data mixing and model merging methods in achieving 3H balance.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths\n1. This work conducts a systematic review on existing methods about 3H-alignment, and designs comprehensive benchmark to evaluate these methods from multiple dimensions. The extensive experiment results offer valuable insights to the community (e.g.,  a phased \"specialize-then-merge\" paradigm may be a more effective path to multi-objective alignment than a monolithic data-mixing approach).\n2. This paper proposes a new algorithm named RESM to overcome two unsolved limitations, i.e., Preference Noise Accumulation and Layer-wise Sparsity Mismatch. The design constitutes a high-quality technical contribution and is well validated by the experiments. \n\n* Weakness\n1. While assembling public resources into a structured benchmark is a valuable engineering effort, the most selected datasets in the benchmark is widely used in the community (such as MATH, GSM8k, MMLU, HumanEval, etc.). I'm not so sure whether such common combinations deserve a critical contribution of this work.\n2. The LLM landscape evolves rapidly. Although Llama-3 and Mistral are highly representative, the experiments lack validation on more recent and capable open-source models (e.g., the Qwen-2.5, DeepSeek-R1-distilled models). Evaluating existing methods and the new-proposed RESM to these cutting-edge models would more powerfully demonstrate the generalizability of this benchmark and brings more realtime insights.\n\nQUESTIONS:\nN.A.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper compares data mixture and model merging for aligning LLMs on Helpfulness, Honesty, and Harmlessness. It proposes RESM, a model merging method using outlier-aware weighting and adaptive rank selection. RESM outperforms prior methods in balancing 3H alignment, showing strong results across static and continual training settings.

STRENGTHS AND WEAKNESSES:
#### **Weaknesses**

- The proposed method is presented with considerable complexity, but it seems to be primarily a combination of existing techniques. Moreover, the relationships between the different submodules are not clearly articulated.

- There is insufficient empirical motivation to support the claim that traditional merging method performs poorly under 3H multi-objective conflicts. The experimental results appear similar to those in non-3H settings, suggesting that the so-called 3H conflict might merely reflect typical parameter space conflicts in multi-model merging scenarios.

- The conclusion that merging datasets is inferior to merging models seems rather obvious, and I did not observe any novel insights specific to the 3H scenario.

#### **Strengths**

- The writing is fluent and clear.  

- The experimental section is thorough, covering a wide range of datasets and baselines.

QUESTIONS:
Please see weaknesses section.

LIMITATIONS:
- The motivation of this paper is insufficient.  

- The proposed method appears to be a mere aggregation of components, lacking clear connections between them.
TITLE: Official Review
CLARITY: 3
IMPACT: 1
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of achieving balanced alignment of large language models (LLMs) across three critical dimensions—helpfulness, honesty, and harmlessness (3H optimization)—through model merging rather than traditional data mixture strategies. While existing methods face limitations such as reliance on expert knowledge and conflicting optimization signals, the authors propose a novel parameter-level conflict-resolution framework via model merging. Their approach, termed Reweighting-Enhanced Task Singular Merging (RESM), introduces outlier-aware weighting and sparsity-adaptive rank selection to mitigate preference noise accumulation and layer sparsity adaptation issues inherent in merging 3H-aligned LLMs. Theoretical analyses and experiments on benchmarks like Llama3 demonstrate RESM’s superiority over prior data mixture and model merging techniques, achieving notable performance gains in balanced alignment metrics.  \n\nThe work further distinguishes itself by addressing gaps in training-free merging strategies, contrasting with methods like H3 fusion that require additional data for adaptation. By emphasizing reproducibility through detailed implementation guidelines and ablation studies, the authors validate RESM’s robustness in preserving task-relevant information during merging. They also contextualize their contributions within broader discussions of societal impacts, ethical considerations, and limitations, positioning RESM as a pathway for ethically constrained LLM development without compromising technical efficacy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**  \n\n1. The paper introduces RESM, a novel model-merging framework that explicitly addresses preference noise accumulation and layer sparsity adaptation in balancing helpfulness, honesty, and harmlessness. This fills a critical gap in existing methods, which often struggle with conflicting optimization signals or rely on expert knowledge. The combination of outlier-aware weighting and sparsity-adaptive rank selection provides a principled approach to parameter-level conflict resolution, advancing the field of ethically constrained LLM alignment.  \n\n2. By benchmarking model merging against traditional data mixture strategies, the work reveals previously overlooked collaborative and conflicting relationships among the 3H dimensions. This systematic comparison not only highlights the limitations of data-level approaches but also establishes a foundation for understanding how parameter-level strategies like RESM can achieve balanced alignment. Such insights are valuable for both practitioners and researchers aiming to navigate trade-offs in multi-objective LLM optimization.  \n\n3. The authors demonstrate RESM’s superiority through extensive experiments on benchmarks like Llama3 and Mistral, achieving measurable gains (2%-5% over data mixture, 1%-3% over prior merging methods). The inclusion of ablation studies, theoretical analyses (e.g., effective rank analysis), and reproducibility guidelines strengthens the credibility of their claims. This emphasis on empirical rigor ensures the method’s practical applicability and transparency.  \n\n---\n\n**Weaknesses:**  \n\n1. The theoretical analysis assumes that preference noise and layer sparsity follow specific statistical distributions (e.g., Gaussian for outlier detection). However, real-world LLM training dynamics often involve heavy-tailed or non-stationary distributions, which could undermine RESM’s effectiveness. The paper does not empirically validate these assumptions, leaving gaps in understanding how robust the method is to deviations from idealized conditions.  \n\n2. Although RESM avoids data-dependent training, its implementation requires fine-grained control over parameter merging (e.g., rank selection, outlier thresholds), which may demand significant computational resources and domain expertise. The paper does not thoroughly discuss the trade-offs between merging complexity and alignment performance, such as how hyperparameter tuning scales with model size or task diversity. This could hinder adoption in resource-constrained settings or real-time applications.\n\nQUESTIONS:\n1. The theoretical analysis assumes Gaussian-distributed preference noise and layer sparsity (e.g., outlier detection via Z-scores). However, real-world LLM training often exhibits heavy-tailed or non-stationary noise (e.g., power-law distributions of gradient updates). How robust is RESM to deviations from these assumptions?\n\n2. How does RESM perform on sparse or modular architectures (e.g., mixture-of-experts models) compared to dense LLMs? The paper focuses on dense models like Llama3 and Mistral, but many modern LLMs use sparse designs to improve efficiency. Does RESM’s reliance on SVD-based sparsity adaptation and outlier-aware weighting remain effective when layer-specific parameter updates are inherently sparse or distributed across experts?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}