Title: MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Language Models

Multi-objective Alignment

Preference Optimization
Abstract: Recent advancements in large language models (LLMs) focus on aligning to heterogeneous human expectations and values via multi-objective preference alignment. However, existing methods are dependent on the policy model parameters, which require high-cost repetition of their alignment algorithms for each new policy model, and they cannot expand to unseen objectives due to their static alignment objectives. In this work, we propose Meta-Objective Aligner (MetaAligner), the first policy-agnostic and generalizable method for multi-objective preference alignment.
MetaAligner models multi-objective alignment into three stages: (1) dynamic objectives reformulation algorithm reorganizes traditional alignment datasets to supervise the model on performing flexible alignment across different objectives; (2) conditional weak-to-strong correction paradigm aligns the weak outputs of fixed policy models to approach strong outputs with higher preferences in the corresponding alignment objectives, enabling plug-and-play inferences on any policy models, which significantly reduces training costs and facilitates alignment on close-source policy models; (3) generalizable inference method flexibly adjusts target objectives by updating their text descriptions in the prompts, facilitating generalizable alignment to unseen objectives.
Experimental results show that MetaAligner achieves significant and balanced improvements in multi-objective alignments on 10 state-of-the-art policy models, and saves up to 93.63% of GPU training hours compared to previous alignment methods. The model also effectively aligns unseen objectives, marking the first step towards generalizable multi-objective preference alignment.

MetaAligner : Towards Generalizable Multi-Objective Alignment of Language Models

Abstract

Recent advancements in large language models (LLMs) focus on aligning to heterogeneous human expectations and values via multi-objective preference alignment. However, existing methods are dependent on the policy model parameters, which require high-cost repetition of their alignment algorithms for each new policy model, and they cannot expand to unseen objectives due to their static alignment objectives. In this work, we propose Meta-Objective Aligner ( MetaAligner ), the first policy-agnostic and generalizable method for multi-objective preference alignment. MetaAligner models multi-objective alignment into three stages: (1) dynamic objectives reformulation algorithm reorganizes traditional alignment datasets to supervise the model on performing flexible alignment across different objectives; (2) conditional weak-to-strong correction paradigm aligns the weak outputs of fixed policy models to approach strong outputs with higher preferences in the corresponding alignment objectives, enabling plug-and-play inferences on any policy models, which significantly reduces training costs and facilitates alignment on close-source policy models; (3) generalizable inference method flexibly adjusts target objectives by updating their text descriptions in the prompts, facilitating generalizable alignment to unseen objectives. Experimental results show that MetaAligner achieves significant and balanced improvements in multiobjective alignments on 10 state-of-the-art policy models, and saves up to 93.63% of GPU training hours compared to previous alignment methods. The model also effectively aligns unseen objectives, marking the first step towards generalizable multi-objective preference alignment. This project is open-sourced here.

1 Introduction

The recent advancements in large language models (LLMs) have focused on generating high-quality responses that align with human expectations and values. At the final stage of alignment, LLMs are supervised on human preference data via reinforcement learning from human feedback (RLHF) [40, 22, 27], where a proxy, directly trained on human preferences data, is leveraged to provide scalar rewards for reinforcement learning (RL) on the target model [22].

However, human expectations and values include a broad spectrum of heterogeneous and multidimensional objectives, which makes scalar supervisions inefficient for aligning diverse and inclusive human preferences [3, 24]. These drawbacks motivate further exploration into multiobjective alignment algorithms. Some intuitive methods extend RLHF into multi-objective RLHF (MORLHF) [26, 19, 24]. Due to its substantial computational cost [19, 24] and the unstable nature

of the proximal policy optimization (PPO) [25, 15, 23] algorithm, other methods seek to bypass the RL paradigm with multi-objective direct preference optimization (MODPO) [39, 10] or supervised fine-tuning (SFT)-based methods [35, 10], which customized prompting strategies to incorporate multiple reward values into queries explicitly.

The above methods for multi-objective alignment bear one commonality: the dependence on the policy model's parameters. This paradigm inevitably brings two key limitations: (1) they require repetition of their high-cost alignment algorithms for each newly-introduced policy model, which is incompatible with the increasing sizes and fast iteration of current foundation models [1, 30, 6, 29]; (2) all target models are statically aligned on pre-determined (e.g. "Helpful", "Harmless", "Honest" [39, 10]) objectives, with currently no efforts in expanding and evaluating their capabilities on unseen objectives. This ignorance leads to poor generalizability of existing multi-objective alignment methods.

In this work, we propose Meta-Objective Aligner ( MetaAligner ), the first policy-agnostic and generalizable method for multi-objective preference alignment. MetaAligner models multi-objective alignment into three stages: (1) the dynamic objectives reformulation algorithm reorganizes traditional alignment datasets into dynamic-objective alignment datasets, training MetaAligner to perform flexible alignment across different objectives. It achieves this by incorporating and combining text descriptions of various alignment objectives in a prompt-based manner; (2) the conditional weak-tostrong correction paradigm aligns the weak outputs of policy models to approach strong outputs with higher preferences in the corresponding alignment objectives. During training, MetaAligner is stacked onto policy models to perform objective-aware corrections, where parameters of the policy model are fixed and MetaAligner is optimized with an SFT-based three-step training process: warming up, equal-preference alignment, and contrastive-preference alignment. This paradigm enables MetaAligner to perform plug-and-play inferences on any policy models even without access to their parameters, which significantly reduces training costs and facilitates alignment on close-source LLMs; (3) the generalizable inference method flexibly adjusts target objectives by updating their text descriptions in the prompts. This method can also adapt MetaAligner to unseen objectives and achieve new alignment strategies via in-context learning [14], a new feature with rare previous exploration in alignment of language models. The number of aligned objectives also becomes expandable, theoretically leading to unlimited simultaneous alignment objectives. Table 1 compares key features between MetaAligner and previous methods. As shown, conditional weak-to-strong correction of MetaAligner extends Aligner [12] to multi-objective alignment scenarios, which are not directly solvable by Aligner itself. MetaAligner is also the first multi-objective alignment method to achieve policy-agnostic alignment and generalization to unseen objectives, two key advantages over previous methods such as MORLHF, MODPO, and SFT-based methods.

In summary, our main contributions are: (1) we propose MetaAligner , the first policy-agnostic method for multi-objective preference alignment. It performs multi-objective alignment efficiently, without tuning the policy models or accessing their parameters. Experimental results show that MetaAligner outperforms previous alignment methods and saves up to 93.63% of GPU training hours; (2) we utilize MetaAligner to exert zero-shot preference alignment for unseen objectives. To our knowledge, this work marks the first attempt at generalizable multi-objective preference alignment. Experimental results show that MetaAligner can simultaneously perform effective alignment for six unseen objectives while maintaining performance on aligned objectives; (3) We examine MetaAligner on three preference alignment datasets. Experimental results show that MetaAligner improves

win rates on multiple objectives across 10 policy models, substantially enhancing responses of state-of-the-art foundation models such as GPT-3.5-Turbo [21] and Claude-3 [2].

Stage 1:  Dynamic Objectives Reformulation

Warm-up Subset

Model

MetaAligner

MetaAligner

Stage 2: Conditional Weak-to-Strong

Correction

Query

Unaligned

Output

Stage 3: Generalizable Inference

Human efforts

Algorithm

Weight updated

Weight fused

Harmless: the response should avoid

content that is offensive, discriminatory,

or harmful.

Helpful: the response should express

clear logic and provide consistent

evidence.

Aligned Objectives

Specificity: the response should refer

to facts and details and avoid vague

arguments.

Factuality: the response should be

factually correct and avoid hallucinated

statements.

Unseen Objectives

Warmed-Up

Model

Warming

up

Equal-Preference

Alignment

…

…

Raw

Dataset

Reformulation

algorithm

Contrastive-Preference

Alignment

Aligned

Output

2 Multi-Objective Alignment of Language Models

In real-world scenarios, human expectations of high-quality responses from AI agents involve considerable variability, with complex interplays such as contradiction (e.g. "Helpful" and "Harmless") [10, 35] and dependence (e.g. "Correct" and "Informative") [34]. Multi-objective preference alignment tackles this challenge by aiming to optimize multiple objectives simultaneously. For each query-response pair, the reward vector is formalized as: R ( q, y ) = [ r 1 ( q, y ) , ..., r N ( q, y )] T , where q, y denote a query and a corresponding response, r i denotes the reward values for i -th objective, which is defined, in most cases of preference alignment, under the Bradley-Terry [5] model of preferences. Specifically, for the same prompt q and two responses ( y 1 , y 2 ) under data distribution D , the model assumes:

where σ denotes the logistic function and P D ( y 1 ≻ y 2 ) denotes the probability that y 1 is preferred against y 2 . MORLHF aims to achieve Pareto optimal among objectives, where the policy model is optimized to maximize a linear scalarization of multiple rewards [26, 19] with a KL-divergence regularization:

where π ϕ denotes the aligned policy model parameterized by ϕ , π ref denotes the reference policy model, ω = [ ω 1 , ..., ω N ] s.t. ∑ N i =1 ω i = 1 , ω i ≥ 0 is the pre-determined heuristic target preference vector. Another paradigm directly built alignment between multiple reward values and their corresponding response by minimizing an SFT loss for the policy model:

where objectives and their corresponding reward values are described with text markers and combined into queries with a static prompting template. Compared to MORLHF, SFT-based multi-objective alignment is proven more cost-efficient and training-stable [35, 10].

3 Meta-Objective Aligner

Existing methods for multi-objective alignment generally face challenges in increasing training costs with new policy models and generalization to unseen objectives. To tackle these challenges, we introduce MetaAligner , which follows a three-stage paradigm: (1) dynamic objectives reformulation for building dynamic multi-objective datasets; (2) conditional weak-to-strong correction for model training; (3) generalizable inference for multi-objective alignment. The paradigm is illustrated in Figure 1.

3.1 Dynamic Objectives Reformulation

We propose a dynamic objectives reformulation algorithm to construct a dynamic multi-objective dataset, which triggers MetaAligner 's ability for flexible adjustment of alignment objectives. Specifically, any typical multi-objective preference alignment dataset D m with m samples and N objectives

can be re-organized as { q i , y i 1 , y i 2 , P i } i m =1 , where P i = [ p i 1 , ..., p iN ] T and p ij ∈ {≻ , ≺ , ≡} indicates the preference on j -th objective.

Algorithm 1 Dynamic objectives reformulation.

̸

We define a text description for each objective: [ ⟨ d 1 ⟩ , ..., ⟨ d N ⟩ ] , where ⟨ d j ⟩ denotes the natural language description for j -th objective. Some examples of such descriptions are in Figure 1 and a full list is in Appendix C. With a pre-defined prompting template T , we build a contrastive subset D c and another equal subset D e from D m . D c includes all contrastive response pairs where p ∈ {≻ , ≺} and D e includes all equal response pairs where p ∈ {≡} . For example, we utilize the following template T ( q, y, O , t ) in building for the IMHI [34] dataset:

[ T ( q, y, O , t ) ] Edit the following QuestionAnswer pair to make it { t } considering the following objectives { O } | Question: { q } | Answer: { y } | Edit:

where q denotes the query, y denotes a corresponding response, O denotes the concatenation of text descriptions for the target objectives, and t ∈ { equal, better } depends on the current building subset. Details of the dynamic objectives reformulation algorithm are described in Algorithm 1. For an instance within the processing loop (line 2): { q, y 1 , y 2 , P } , the algorithm performs a two-step reformulation: (1) collect the sets O ≻ , O ≺ , O ≡ that includes objectives where y 1 outperforms y 2 , y 2 outperforms y 1 , and both perform equally (lines 3-12); (2) for each objective set O , we randomly shuffle the objectives to further trigger the model's flexible alignment ability, and build query-response pairs based on the corresponding prompting template T (line 13-27). All prompting templates and examples of the algorithm are presented in Appendix C.

Training on dynamic multi-objective datasets provides three key advantages: (1) instancelevel alternation of the target objectives during training enables MetaAligner to perform flexible alignment under different combinations of objectives; (2) mutual alignment between the same response pairs on different objectives fully leverages the supervision information in the preference vectors. (3) the reward-free alignment method (no explicit preference values required) avoids complicated preference-to-reward mapping [35] process in previous SFT-based multi-objective alignment methods.

3.2 Conditional Weak-to-Strong Correction

Based on the dynamic multi-objective training datasets, we train MetaAligner in a conditional weakto-strong correction manner, which follows an SFT-based training objective and a three-step training paradigm.

3.2.1 Training Objective Derivation

MetaAligner is a standard conditional seq-to-seq model on top of the original policy model π ϕ , which re-distributes the policy model output y 0 considering objectives O as follows:

̸

̸

where δ θ denotes the MetaAligner module parameterized by θ , t depends on the training dataset. Conditional weak-to-strong correction directly trains MetaAligner to align the weak policy model output y 0 to the strong target output y , which has higher preference values in corresponding objectives O . We have the standard cross-entropy loss as the training objective:

We fix the parameters of the policy model, thus excluding ϕ from the weight update process. In practice, we use the dynamic multi-objective dataset for supervision, where the weak response in each query-response pair is directly leveraged as samples y 0 from unknown policy models. Therefore, we eliminate the second term in Eqn. 5 and simplify the training objective as:

The above action poses two advantages: (1) the computation resources required for MetaAligner training is detached from policy model size, which enables policy-agnostic and cost-efficient alignment for large policy models; (2) MetaAligner works only via outputs from the policy models, which allows training and inference for alignment on close-source policy models [1, 21, 2].

3.2.2 Three-Step Model Training

In practice, we utilize an LLM as the base model for MetaAligner , which provides domain knowledge and strong reasoning ability to support the conditional weak-to-strong correction process. We propose a three-step paradigm based on the objective function in Eqn. 6: (1) Warming up . This stage trains the model in identical response pairs with a warm-up subset, a prelude proven effective in residual correction strategies [11, 12]. We randomly sample a subset of the equal subset D e as the warm-up subset, but set an identical target response for each instance; (2) Equal-preference alignment . Due to the contrastive nature of their learning paradigm, most previous preference alignment works focus on modeling the residuals between response pairs and ignore the equalpreference response pairs. However, equal preferences are common in many scenarios [34, 7] and enclose useful information such as the principle components of preference modeling regarding each objective. Based on these intuitions, we introduce a novel equal-preference alignment step to fine-tune the warmed-up MetaAligner on the equal subset D e ; (3) Contrastive-preference alignment . This stage fine-tunes the MetaAligner on the contrastive preference subset D c , which instructs the model to perform conditional weak-to-strong correction on the specified objectives.

3.3 Generalizable Inference

During inference, MetaAligner achieves alignment following the sampling process as in Eqn. 4, where unaligned outputs, sampled from the target policy model, are used as the input for conditional weakto-strong correction. With the prompting-based paradigm, the target objectives for MetaAligner also become expandable and generalizable, a key advantage over previous alignment methods [39, 35, 10]. The generalizability is two-fold: Firstly, users can manipulate the target objectives by adjusting combinations of text descriptions in the objective set O . For example, in alignment with objectives 1, 3, and 4, we can flexibly shuffle the corresponding descriptions ⟨ d 1 ⟩ , ⟨ d 3 ⟩ , and ⟨ d 4 ⟩ as follows: O = ⟨ d 3 ⟩ ; ⟨ d 1 ⟩ ; ⟨ d 4 ⟩ . Secondly, the prompt-based objectives statement enables flexible adjustment of text descriptions for existing objectives and injections of unseen objectives. Following the last example, we have two unseen alignment objectives 5: ⟨ d ∗ 5 ⟩ and 6: ⟨ d ∗ 6 ⟩ , and an updated text description ⟨ d 3 ⟩ for aligned objective 3. We can perform zero-shot alignment on the new objectives by adjusting O as follows: O ∗ = ⟨ d 3 ⟩ ; ⟨ d 1 ⟩ ; ⟨ d 4 ⟩ ; ⟨ d ∗ 5 ⟩ ; ⟨ d ∗ 6 ⟩ . This simple pattern can theoretically lead to unlimited simultaneous alignment objectives. We expect MetaAligner to make generalizable weak-to-strong corrections under these unseen conditions via its in-context learning ability. This advancement marks a new exploration into generalizable multi-objective preference alignment.

4 Experiments

4.1 Experimental Settings

Datasets. We transfer the following three alignment datasets into dynamic multi-objective datasets: (1) HH-RLHF [3]: a large-scale dataset with 160K prompts and corresponding response pairs.

We follow Yang et al. [35] and use open-sourced reward models on three objectives: "Harmless", "Helpful", and "Humor" to score the responses; (2) UltraFeedback [7]: a multi-aspect alignment dataset with 64K prompts with preferences obtained from GPT-4, including "Instruction following", "Honest", "Truthful", and "Helpful" objectives; (3) IMHI : we create an alignment dataset on the IMHI dataset [34] targeting interpretable mental health analysis. We invite domain experts to label 7.2K response pairs considering 3 objectives: "Correct", "Informative", and "Professional". Figure 2 shows the objective distributions on two datasets. The objectives display balanced overall distributions across objective set sizes, training MetaAligner to adjust targets dynamically. Most objectives also cover considerable proportions in each column category, alleviating label imbalance problems.

HH-RLHF

UltraFeedback

Overall

0.45

0.4

0.15

0.9

1.0

Overall

0.17

0.26

0.25

0.32

0.8

0.8

Harmless

0.34

0.64

0.86

0.41

0.57

0.7

0.6

Honest

0.22

0.51

0.82

- 0.6

Helpful

0.42

0.58

0.5

Truthful

0.4

0.4

0.091

0.32

0.57

Humour

0.17

0.85

= 0.3

Helpful

0.35

0.53

0.75

0.2

0.2

2

3

1

2

3

4

Models. We train MetaAligner -(1.1B, 7B, 13B) models based on TinyLLaMA-1.1B [37] and LLaMA2-(7B, 13B) [30] foundation models. We utilize MetaAligner to perform multi-objective alignment on the following open-source policy models: LLaMA2Chat-(7B,13B,70B) [30], Gemma-instruct(2B,7B) [29], and Vicuna-(7B, 13B, 33B) [6]. We also align two advanced close-source foundation models: GPT-3.5-Turbo [21] and Claude-3-Sonnet [2], where model parameters are inaccessible.

Evaluation Metric. On each objective, we quantify the alignment performance of model outputs by comparing their win rates against the ground-truth response provided by the benchmark datasets. Considering the large amounts of test samples, we leverage GPT-4 [1], a widely utilized evaluation tool in previous works [10, 28, 18], to perform the judgments. Each target response, ground-truth response, query, and evaluated objectives are provided via prompt engineering. GPT-4 is required to compare and select the response with higher alignment on the specified objective or indicate a tied performance of the two responses.

More details about the training process, model cards, dataset statistics, IMHI dataset annotation, and evaluation settings are presented in Appendix D.

4.2 Overall Performance

MetaAligner -(1.1B, 7B, 13B) performance on 3 alignment datasets are shown in Table 2. According to the results, the MetaAligner models achieve substantial improvement for most objectives and policy models. For example, on UltraFeedback, there is an average of 11.47% advantage for MetaAligner -1.1B on "Honest", 34.39% for MetaAligner -7B, and 43.79% for MetaAligner -13B. These results show the general effectiveness of MetaAligner on various upstream models and the feasibility of plugand-play multi-objective alignment. On the mental health analysis benchmark IMHI, MetaAligner models also show remarkable win rates on all objectives, proving their effectiveness in performing multi-objective alignment in domain knowledge-intense scenarios. We further evaluate MetaAligner on each IMHI sub-task and the results are shown in Appendix E.

From the policy model scale perspective, MetaAligner provides successful alignments to open-source models with sizes ranging from 2B to 70B, significantly extending the size of MetaAligner itself. In the extreme case, MetaAligner -1.1B advances the win rates of LLaMA2-Chat-70B outputs, a policy model with 63 × more parameters, by an average of 12.19% on HH-RLHF, 13.08% on UltraFeedback, and 13.26% on IMHI. These results prove MetaAligner as a parameter-efficient alignment strategy compared to previous multi-objective alignment methods, where the policy model weights are updated, leading to an inevitable surge of computation resources as policy model sizes grow. MetaAligner also significantly improves performance on close-source LLMs: GPT-3.5-Turbo and Claude-3-Sonnet. These results prove its potential for application in close-source scenarios and effective multi-objective alignment of state-of-the-art policy models.

Within most policy model families, we observe a decreasing trend in win-rate advantage as their sizes increase. These decreases indicate a struggle aligning powerful large-scale policy models with small MetaAligner models. Fortunately, MetaAligner 's capabilities also show scalability. Increasing the size of its base model leads to a higher win-rate advantage on most policy models. For example, on UltraFeedback, MetaAligner -7B outperforms MetaAligner -1.1B on all 10 policy models, and MetaAligner -13B further surpasses MetaAligner -7B by an average of 12.58%. These observations motivate further explorations in model size-performance balance for MetaAligner .

4.3 MetaAligner vs. Baseline Methods

We compare the performance of MetaAligner with MORLHF, MODPO, SFT-based methods, and Aligner. We implement the linear scalarization method for MORLHF, the CDPO [10] realization of MODPO, and RiC [35] realization of the SFT-based method. As Aligner is not suitable for multi-objective alignment, we train Aligner-7B on "Helpful" annotations for HH-RLHF and "IF" for UltraFeedback. We compare these methods on the LLaMA2-Chat-7B policy model. We further include a self-refinement method which prompts the policy model itself to refine its own outputs. We compare self-refinement on the LLaMA2-Chat-70B policy model as it requires strong in-context

learning ability from the policy model. The results are presented in Table 3. Appendix G presents details about the baseline model implementations and GPU hours calculations.

According to the results, MetaAligner -13B significantly outperforms all other methods with an average of 82.67% win rate on HH-RLHF and 61.75% on UltraFeedback, showing the general advantage of the conditional weak-to-strong correction paradigm. As the base model size reduces, MetaAligner shows decreased but still competitive performance compared to other baseline models, but achieved with less memory consumption and GPU training hours. Impressively, the MetaAligner -1.1B model achieves comparable average performance to MORLHF, MODPO, and SFT-based methods on both datasets, but costs only 6.37%-48.71% of their GPU training hours, with a 6.36 × smaller size than the LLaMA2-Chat-7B policy model. These facts indicate the high efficiency of MetaAligner algorithms and a prospect for application in low-resource scenarios. Compared to previous methods, MetaAligner models can also achieve balanced and stable performances in objective-wise evaluations, including contradictory objectives such as "Harmless" and "Helpful", without requiring explicit hyper-parameter tuning for achieving Pareto optimal solutions [10, 35, 19]. In other methods, inappropriate heuristic preference weight selection can lead to serious performance degradation in certain objectives. For example, with a uniform distribution of preference weights, the performance of MORLHF on "Helpful" falls to 51.2%, a huge gap to other methods. Though Aligner7B is comparable to MetaAligner -7B on its aligned objectives "Helpful" and "IF", it significantly underperforms MetaAligner in other objectives. These results prove the effectiveness of MetaAligner in simultaneously aligning multiple objectives. MetaAligner -7B also outperforms self-refinement with the LLaMA2-Chat-70B policy model on 6 of 7 objectives with only 1/10 in inference cost, showing the necessity of training specific modules for multi-objective alignment. Ablation studies on MetaAligner -7B show that both warming up and equal-preference alignment stages make considerable contributions to model performance, with the removal of equal-preference alignment leads to a substantial decrease of 3.53% on HH-RLHF and 3.51% on UltraFeedback in average win rates.

MetaAligner-1.1B

MetaAligner-7B

MetaAligner-13B

1.0

Evaluated Objectives

→1IF

1

Helpful

0.8

1

Honest

Truthful

Specific

Winrate

0.6

Factual

Readable

Fair

0.4

Repeat

Length

0.2

0.0

AlignedObj.

+Specific

+Factual

+Readable+Fair

+Repeat

+Length

AlignedObj.

+Specific

+Factual

+Readable +Fair

+Repeat

+Length

Aligned Obj.

+Specific

+Factual

+Readable+Fair

+Repeat

+Length

Zero-shot Alignment Objectives

4.4 Generalizable Alignment to Unseen Objectives

In this section, we explore zero-shot preference alignment by utilizing MetaAligner to align with six unseen objectives: "Specific", "Factual", "Readable", "Fair", "Repeat", and "Length" [9]. More details about these objectives are in Appendix C. We randomly select 2,700 queries from the UltraFeedback dataset and re-align the LLaMA2-Chat-70B outputs with these unseen objectives added to the objective set O one-by-one, with 10 aligned objectives in total. Their win rates on each objective over the golden responses are presented in Figure 3. We have the following conclusions:

MetaAligner performs effective zero-shot alignment for unseen objectives. With most MetaAligner models, incorporating an unseen objective into the objective set significantly improves its corresponding win rate. For example, MetaAligner -7B improves by 25.17% on "Specific", 14.5% on "Factual", and 17.5% on "Readable" compared to each of these objectives unaligned. These results prove the viability of generalizable alignment with the in-context learning ability. However, the win rates

on supervised objectives ("Instruction following", "Helpful", "Honest", and "Truthful") generally surpass unseen objectives, showing that supervised learning remains more effective in multi-objective preference alignment compared to in-context learning.

Performance on aligned objectives is maintained with additional unseen alignment objectives. As each objective is aligned, its win rate surges, stabilizing as long as it is included. On simultaneously aligning 10 objectives, MetaAligner -7B outperforms LLaMA2-Chat-70B outputs by an average of 14.25% on unseen objectives. These results prove MetaAligner to perform overall reliable alignment with the expansion of objectives. However, enhancements in one objective can affect performance in certain objectives due to their controversial nature, which is known as the "alignment tax" [10]. For example, aligning on "Fair" (+Fair) with MetaAligner -(7B, 13B) benefits its win rates, but harms performance on objectives such as "Readable" and "Factual" compared to when "Fair" is unaligned.

MetaAligner's generalizability shows scalability. Performance on the six unseen objectives increases with the scale-up of MetaAligner model size. MetaAligner -1.1B provides limited improvement on most unseen objectives, but MetaAligner -7B extends the win rates to an average of 48.5%, and MetaAligner -13B further reaches 61.25%. MetaAligner -13B also more effectively aligns objectives such as "Length", where smaller models perform badly. This scalability is attributed to larger foundation models' growing in-context learning ability, which enables accurate interpretations of the objective descriptions and instructions. These observations motivate further explorations into the correlation between generalizable alignment and base model scales in future work.

4.5 Evaluations of Objective-Wise Alignment

Instruction following

Helpfulness

0.25

Unaligned

Aignment Objectives

0.20

0.20

Instruction following

Helpfulness

Honesty

0.15

Truthfulness

Full alignment

0.10

0.05

0.05

0.00

GPT4 score

0.00

GPT4 score

Honesty

Truthfulness

0.30

0.20

0.25

0.15

0.10

0.05

0.05

0.00

0

GPT4 score

5

10

0.00

0

GPT4 score

5

10

We evaluate the objective-wise performance of MetaAligner by decoupling the target objectives. We utilize MetaAligner to perform six levels of alignments: unaligned, aligning on each objective ("Instruction following", "Helpful", "Honest", and "Truthful"), and full alignment. We leverage GPT-4 to score the responses ranging from 0 to 10. The results are shown in Figure 4. Experimental details and more results are shown in Appendix F. We have the following observations:

Objective-wise alignment improves performance on the primary target and boosts the performance on other objectives. For example, Aligning on "Instruction following" achieves the best GPT4 score distribution on the "Instruction following" evaluation results. It also significantly increases GPT4 scores on "Helpful", "Honest", and "Truthful" over the unaligned responses.

This tendency holds with other policy models and alignment objectives. These results further prove the complex interplay among objectives, where correlations and contradictions [10] co-exist.

Full alignment on all objectives provides balanced performance. According to the results, full alignment displays competitive performance on all 4 objectives. Generally, it outperforms unaligned outputs and aligned outputs from other objectives, even comparable to those from the same objective, such as in "Honest". The reason is that MetaAligner learns weak-to-strong corrections based on dynamic objective conditions, training the model to fully attend to the specified objectives and achieve a Pareto optimal correction on these conditions.

5 Related Work

This paper focuses on advancing multi-objective alignment of language models with human values, which is mainly related to two research areas: (1) Large Language Models , including the latest development in close-source AI agents [1, 21, 2] and open-source foundation models [30, 29, 6]. (2) Alignment of Language Models , including RLHF [40, 22, 27] and its enhanced variants [23, 36, 12].

Multi-objective alignment methods include MORLHF [26, 19, 24], MODPO [39, 10], and SFT-based methods [35, 10]. A detailed review of related work is in Appendix B.

6 Discussions

Conclusion. This paper proposed MetaAligner , the first policy-agnostic and generalizable method for multi-objective preference alignment. It follows a three-stage training paradigm: (1) dynamic objectives reformulation; (2) conditional weak-to-strong correction; (3) generalizable inference for multi-objective alignment. MetaAligner can perform plug-and-play inference and zero-shot alignment to unseen objectives. Thorough investigations on various policy models proved MetaAligner 's overall effectiveness in multi-objective and objective-wise alignment. Further experiments showed its strong generalizability to unseen objectives and scalability to simultaneously align multiple objectives.

Limitations and Future Work. Firstly, stacking MetaAligner module on policy models inevitably leads to increased computational burdens during alignment inference [12], which affects model deployment, especially for scenarios such as local deployment on mobile devices. Secondly, due to limited resources, we only tested the generalizability of MetaAligner on 6 unseen objectives, which does not provide a clear landscape of its alignment performance on more objectives. In future work, we aim to explore improving MetaAligner in domain-specific alignment scenarios utilizing techniques such as retrieval-augment generation [17]. We will also dive deep into the scalability of MetaAligner to evaluate its impact on alignment performance, including the model scale-performance balance. We will also provide a clearer landscape of their generalizable alignment ability by examining larger base model sizes and aligning on much more unseen objectives (we only expanded to 10 objectives). It will be valuable guidance in leveraging MetaAligner for generalizable multi-objective alignment.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023).

Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. (2024).

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022).

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021).

Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39, 3/4 (1952), 324-345.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) 2, 3 (2023), 6.

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 (2023).

Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 (2023).

Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, and Marc Dymetman. 2023. Compositional preference models for aligning LMs. arXiv preprint arXiv:2310.13011 (2023).

Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Jiexin Wang, Huimin Chen, Bowen Sun, Ruobing Xie, Jie Zhou, Yankai Lin, et al. 2024. Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment. arXiv preprint arXiv:2402.19085 (2024).

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 770-778.

Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. 2024. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint arXiv:2402.02416 (2024).

Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. 2023. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852 (2023).

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199-22213.

Aviral Kumar, Abhishek Gupta, and Sergey Levine. 2020. Discor: Corrective feedback in reinforcement learning via distribution correction. Advances in Neural Information Processing Systems 33 (2020), 18560-18572.

Yuhang Lai, Siyuan Wang, Shujun Liu, Xuanjing Huang, and Zhongyu Wei. 2024. ALaRM: Align Language Models via Hierarchical Rewards Modeling. arXiv preprint arXiv:2403.06754 (2024).

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrievalaugmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459-9474.

Haoran Li, Yiran Liu, Xingxing Zhang, Wei Lu, and Furu Wei. 2023. Tuna: Instruction Tuning using Feedback from Large Language Models. arXiv preprint arXiv:2310.13385 (2023).

Kaiwen Li, Tao Zhang, and Rui Wang. 2020. Deep reinforcement learning for multiobjective optimization. IEEE transactions on cybernetics 51, 6 (2020), 3103-3114.

Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for abstractive text summarization. arXiv preprint arXiv:2303.15621 (2023).

OpenAI. 2023. Introducing ChatGPT. (2023). https://openai.com/blog/chatgpt

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 (2022), 27730-27744.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2024).

Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2024. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems 36 (2024).

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).

Ozan Sener and Vladlen Koltun. 2018. Multi-task learning as multi-objective optimization. Advances in neural information processing systems 31 (2018).

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems 33 (2020), 3008-3021.

Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. Salmon: Self-alignment with principle-following reward models. arXiv preprint arXiv:2310.05910 (2023).

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open Models Based on Gemini Research and Technology. arXiv preprint arXiv:2403.08295 (2024).

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).

Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2024. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems 36 (2024).

Qianqian Xie, Edward J Schenck, He S Yang, Yong Chen, Yifan Peng, and Fei Wang. 2023. Faithful ai in medicine: A systematic review with large language models and beyond. medRxiv (2023).

Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023. Towards interpretable mental health analysis with large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing .

Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. MentaLLaMA: interpretable mental health analysis on social media with large language models. In Proceedings of the ACM on Web Conference 2024 . 4489-4500.

Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. 2024. Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment. arXiv preprint arXiv:2402.10207 (2024).

Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2024. RRHF: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems 36 (2024).

Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385 (2024).

Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren's song in the AI ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219 (2023).

Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. 2023. Beyond one-preference-for-all: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708 (2023).

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 (2019).

A Ethics and Impacts

A.1 Licenses

We leveraged 3 publicly available datasets to build our dynamic multi-objective datasets: HH-RLHF, UltraFeedback, and IMHI. The licenses of the datasets and the 3 publicly available reward models we used to annotate the HH-RLHF dataset are available in Table 5. In Sec. 4.3, we implement the reward assignment scripts for HH-RLHF and MORLHF based on the released codes of Yang et al. [35], which is available at Github. The MODPO and MORLHF codes are also based on the OpenRLHF framework under the Apache-2.0 license. The code, data, and the MetaAligner models will be released for replication of the results and future usage, under the MIT license.

A.2 Broader Impacts

In this work, MetaAligner provides an effective and model-agnostic method for generalizable and expandable alignment of LLM outputs with multiple human expectations. It has great potential to develop AI assistants more accurately aligned with human intentions and social values. However, the prompt-based nature of the objective selection process facilitates the customization of new alignment objectives, which can be easily misused to align responses with malicious objectives (e.g. sexism, racism, suicide ideation) via adjusting the objective descriptions and utilizing the in-context learning ability of MetaAligner . These actions can lead to harmful outputs from MetaAligner . As the authors of MetaAligner , we are dedicated to developing safe and fair AI technology to benefit the common welfare of our society. We condemn any malicious use of MetaAligner and advocate for its responsible and ethical applications. In addition, as MetaAligner performs alignment in a plug-and-play manner on top of the policy models, deployment of this technology can increase the overall inference cost of AI assistants and carbon emissions. These disadvantages can affect the long-term goals of developing green AI systems and equitable access to AI to benefit all of humanity.

A.3 Safeguards

This released codes, data, and MetaAligner models are provided for research only. None of the material constitutes actual diagnosis or advice, and help-seekers should get assistance from professional psychiatrists or clinical practitioners. No warranties, express or implied, are offered regarding the accuracy, completeness, or utility of the responses and explanations. The authors and contributors are not responsible for any errors, omissions, or any consequences arising from the use of the information herein. Users should exercise their own judgment and consult professionals before making any clinical-related decisions. The use of the software and information contained in this paper is entirely at the user's own risk.

The collected queries to build our IMHI preference dataset are from the publicly available IMHI dataset [34], and we strictly follow the privacy protocols and ethical principles to protect user privacy and guarantee that anonymity is properly applied in all the mental health-related texts. In addition, to minimize misuse, all examples provided in our paper are paraphrased and obfuscated utilizing the moderate disguising scheme.

In addition, recent studies have indicated LLMs may introduce some potential bias, such as gender gaps. Meanwhile, some incorrect prediction results, inappropriate explanations, and overgeneralization also illustrate the potential risks of current LLMs. Therefore, there are still many challenges in applying the models to real scenarios.

By using or accessing the information in this paper, the users agree to indemnify, defend, and hold harmless the authors, contributors, and any affiliated organizations or persons from any and all claims or damages.

B Related Work

B.1 Large Language Models

Large language models (LLMs) have reached approaching-human capabilities across a wide spectrum of tasks related to understanding, generating, and reasoning with natural language [1, 30, 20]. Notable

examples include commercially available LLMs like ChatGPT [21], GPT-4 [1], and Claude-3 [2]. Due to the high inference cost of these close-source models, research trend in open-source foundation models surges, leading to cutting-edge open-source models like LLaMA2 [30], Gemma [29], and Vicuna [6]. Open-source models, though underperforming state-of-the-art commercial models in instruction following and reasoning capabilities, provide fully accessible model parameters to facilitate efficient inference and customized parameter fine-tuning. Despite the advancements of LLMs, recent studies found that they can exhibit problematic behaviors, including the generation of inaccurate information [38, 32], flattery, and deception, raising concerns about their potential negative impacts and associated risks on society [4]. To address these issues, considerable research has been dedicated to refining LLMs' outputs to better align with human values and preferences [13].

B.2 LLMAlignment on Human Values

Many studies have delved into enhancing the responses of LLMs in core characteristics of human values like "Helpful", "Harmless", and "Honest". Early efforts are largely centered on Reinforcement Learning from Human Feedback (RLHF) [40, 22, 27], where the alignment of human values is manifested by maximizing a scalar value obtained from the reward model with a KL-regularization, using RL-algorithms such as PPO [25]. However, PPO faces challenges including inefficiency and instability, driving development in simplified algorithms such as DPO [23], rank-based learning [36], and weak-to-strong correction [12]. Nonetheless, human expectations and values include a broad spectrum of heterogeneous and multi-dimensional objectives, where a scalar reward model proves inadequate for aligning LLMs with varied human preferences. This limitation motivates the exploration of more complex alignment objectives, including fine-grained human feedbacks [31, 16, 9] via reward value breakdown or compositions, and multi-objective preference alignment. Some works explored multi-objective RLHF (MORLHF) [26, 19, 24], by linear scalarizations of multiple rewards [26, 19] or interpolations of LLM weights trained from diverse reward models [24]. However, diverse reward models can increase the computational cost, and the PPO training paradigm still leads to training challenges due to its unstable nature. Recent studies further explore the multi-objective direct preference optimization (MODPO) [39, 10] without the RL paradigm. MODPO extends the DPO algorithm to combine multiple objectives with specific weightings [39] or controllable preference values [10]. Other methods further simplify the paradigm with SFT-based methods [35, 10], which use customized prompting strategies to explicitly incorporate multiple reward values and optimize with supervised fine-tuning (SFT) or DPO. These methods also facilitated objective-wise controllable generation during inference. Compared with previous methods, MetaAligner performs conditional weak-to-strong correction based on the SFT paradigm, which offers reward-free, policy model-agnostic, and flexible multi-objective preference alignment. The model also effectively aligns unseen objectives, mark- ing the first step towards generalizable multi-objective preference alignment.

C Details of Dynamic Objectives Reformulation

C.1 Objective Descriptions

The text descriptions for all tested objectives in this paper are included in Table 4. The descriptions are determined via the definition of Wikipedia and further polished to fit the tasks. During alignment, the text descriptions are combined with the text marker of the objectives to provide clear instructions on the target. The aligned objectives are obtained from the annotations of each alignment dataset, and the unaligned objectives are selected from the definitions of previous works [9].

C.2 Prompting Templates

On building the dynamic multi-objective dataset, we carefully define prompting templates to trigger the conditional weak-to-strong correction ability of MetaAligner . A full list of the used templates is shown below. Specifically, for the preference subset of the HH-RLHF dataset [3], since the model is required to improve the response considering a multi-turn dialogue history, we have:

[ T ( q, y, O , ' better ') ]

You are an assistant to human. You will be provided with a context and an answer. Consider the context, then edit the answer to improve it considering these aspects: {O} | Context: {q} | Answer: {y} | Edit:

For the equal-preference subset, we have:

[ T ( q, y, O , ' equal ') ]

You are an assistant to human. You will be provided with a context and an answer. Consider the context, then edit the answer to make it equal considering these aspects: { O } | Context: {q} | Answer: {y} | Edit:

In the UltraFeedback dataset [7], the model is required to improve the response considering a single query. For the preference subset, we have:

[

T

(

q, y,

O

,

'

better

')

]

You are an assistant to human. You will be provided with a query and an answer. Consider the query, then edit the answer to improve it considering these aspects: { O } | Query: {q} | Answer: {y} | Edit:

For the equal-preference subset, we have:

C.3 An Example of Dynamic Objectives Reformulation

We provide an example of the dynamic objectives reformulation algorithm 1 to better illustrate its working mechanism. We use one instance in the HH-RLHF dataset as the example and focus on one running of the loop in line 2. Specifically, we have

q

:

Human: What are some cool countries to visit in Asia? Assistant: Hm, it's difficult to pick just one. Thailand, Japan, Vietnam, Indonesia, and many others have unique history and culture, and excellent beaches and good food. Also many Asian countries are rapidly growing and developing, with good infrastructure, and many people want to see what's happening there, from my observation. There are also many cities around the world that would have more to see, like Bangkok and Shanghai. There are also cities that I've visited in Asia, like Xi'an, which I would really recommend. Human: Okay, which one do you like? Assistant:

y

1

:

If I was a tourist in Asia, I'd love to visit Kolkata. Maybe Xi'an, because it's just a cool place to visit. y 2 :

I like Thailand and Japan, because they have a strong culture that is still being preserved and a unique history.

We have the following reward score assignments from the reward models:

We can map the above reward values to preferences on each objective: P = [ ≺ , ≺ , ≻ ] . Based on this preference vector, we obtain the text description sets after the processing on line 5-12, which are as follows:

[

O

≺

]

Harmless: The response should avoid content that is offensive, discriminatory, or harmful; Helpful: The response should provide useful resources and suggestions to the user;

Based on the above information, we can build two pairs of weak-to-strong training samples, by fitting in the templates of HH-RLHF dataset provided in Sec. C.2:

QUERY 1 :

You are an assistant to human. You will be provided with a context and an answer. Consider the context, then edit the answer to improve it considering these aspects:

: The response should avoid content that is offensive, discriminatory, or harmful;

Harmlessness

Helpfulness : The response should provide useful resources and suggestions to the user |

Context: Human: What are some cool countries to visit in Asia? Assistant: Hm, it's difficult to pick just one. Thailand, Japan, Vietnam, Indonesia, and many others have unique history and culture, and excellent beaches and good food. Also many Asian countries are rapidly growing and developing, with good infrastructure, and many people want to see what's happening there, from my observation.

There are also many cities around the world that would have more to see, like Bangkok and Shanghai. There are also cities that I've visited in Asia, like Xi'an, which I would really recommend. Human: Okay, which one do you like? Assistant:

| Answer: If I was a tourist in Asia, I'd love to visit Kolkata. Maybe Xi'an, because it's just a cool place to visit. | Edit:

CORRECTION 1 :

I like Thailand and Japan, because they have a strong culture that is still being preserved and a unique history.

QUERY 2 :

You are an assistant to human. You will be provided with a context and an answer. Consider the context, then edit the answer to improve it considering these aspects:

Humor : The response should be cheerful and amusing |

Context: Human: What are some cool countries to visit in Asia? Assistant: Hm, it's difficult to pick just one. Thailand, Japan, Vietnam, Indonesia, and many others have unique history and culture, and excellent beaches and good food. Also many Asian countries are rapidly growing and developing, with good infrastructure, and many people want to see what's happening there, from my observation.

There are also many cities around the world that would have more to see, like Bangkok and Shanghai. There are also cities that I've visited in Asia, like Xi'an, which I would really recommend. Human:

Okay, which one do you like? Assistant:

| Answer: I like Thailand and Japan, because they have a strong culture that is still being preserved and a unique history. | Edit:

CORRECTION 2 :

If I was a tourist in Asia, I'd love to visit Kolkata. Maybe Xi'an, because it's just a cool place to visit.

Since O ≡ is empty, this data sample does not contribute to the equal subset D e . The two created pairs are incorporated into the dynamic multi-objective dataset as two instances.

D Experimental Details

D.1 Model Training

Details about the training process of MetaAligner and the building process of the 3 datasets are presented in Table 5. During the performance evaluation process, GPT-4 is leveraged to compare the responses on the corresponding objective. Specifically, we have the aligned test dataset: { q i , g i , R origin i , R aligned i } n i =1 , where q denotes the query, g denotes the ground-truth response from the dataset, R origin denotes the original response from the policy model, and R aligned denotes the aligned response from MetaAligner . We use the following prompting template and probe GPT-4 to perform judgment:

where r 1 , r 2 are the compared response pairs, and ⟨ r ⟩ denotes the text description of the target objective. With the above information and the target objective description ⟨ r t ⟩ , we obtain the win rates using Algorithm 2.

D.2 Model Cards

TinyLLaMA-1.1B [37]. Acompact 1.1B language model pre-trained on around 1 trillion tokens for approximately 3 epochs. Building on the architecture and tokenizer of LLaMA2, TinyLlama leverages various advances contributed by the open-source community (e.g., Flash-Attention), achieving better computational efficiency. Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes. We use TinyLlama-1.1B-Chat-v1.0 as the base model for MetaAligner -1.1B.

LLaMA2-(Chat)-(7B, 13B, 70B) [30]. A collection of pre-trained and fine-tuned large language models (LLMs) trained and released by Meta, ranging from 7 billion to 70 billion parameters. The fine-tuned LLMs, called LLaMA2-Chat, are optimized for dialogue use cases. The models outperform other open-source models on most benchmarks. Further human evaluations prove that LLaMA2-Chat also excels in helpfulness and safety. LLaMA2 models are among the most advanced open-source foundation models. We use LLaMA2-(7B, 13B) as base models for MetaAligner -(7B, 13B), and use LLaMA2-Chat-(7B, 13B, 70B) as policy models to evaluate the alignment performances.

Vicuna-(7B, 13B, 33B) [6]. Vicuna is a family of open-source chatbots trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT. Preliminary evaluation using GPT-4 as a judge shows Vicuna-13B achieves more than 90% quality of OpenAI ChatGPT and Google Bard while outperforming other models like LLaMA and Stanford Alpaca in more than 90% of cases. We use Vicuna-(7B, 13B)-V1.5 and Vicuna-33B-V1.3 as policy models to evaluate the alignment performances.

Gemma-instruct-(2B, 7B) [29]. A family of open-source models based on Google's Gemini models. Gemma models are pretrained on 6T tokens of text, using architectures, data, and training recipes inspired by the Gemini model family. Like Gemini, these models achieve strong generalist capabilities in text domains, alongside state-of-the-art understanding and reasoning skills at scale. Gemma-instruct models are further fine-tuned for dialogue, instruction-following, helpfulness, and

safety. Gemma-instruct is developed in two sizes: a 7B version for efficient deployment and development and a 2B version for CPU and on-device applications. We select both models as policy models to evaluate the alignment performances.

MentaLLaMA-(7B, 13B, 33B) [34]. MentaLLaMA is the first open-source instruction-following LLM series for interpretable mental health analysis. Based on LLaMA2-(7B, 13B) and Vicuna-33B foundation models, MentaLLaMA is trained on the Interpretable Mental Health Instruction (IMHI) dataset with 105K instruction samples, the first multi-task and multi-source instruction-tuning dataset for interpretable mental health analysis on social media. MentaLLaMA can perform mental health analysis on social media data and generate high-quality explanations for its predictions. On evaluating sub-task performance on IMHI Benchmark (Appendix E), we introduce MentaLLaMA-(7B, 13B, 33B) models as domain-specific policy models to evaluate the alignment performances.

GPT-3.5-Turbo [21]. GPT-3.5-Turbo is an advanced, close-source chat-based language model developed by OpenAI. It is a sibling model to InstructGPT, which is trained to follow instructions in a prompt and provide a detailed response. The model is firstly fine-tuned with SFT with conversations in which the model played both sides-the user and an AI assistant. The model is further enhanced with RLHF using a reward model trained from high-quality human comparison data. In our experiments, we use the gpt-3.5-turbo-0125 API provided by OpenAI as a strong policy model to evaluate the alignment performances.

Claude-3 [2]. Claude-3 is among the state-of-the-art foundation models for industry benchmarks across reasoning, math, coding, multi-lingual understanding, and vision quality, developed by Anthropic. The model family includes 3 models: (1) Opus, the most capable model; (2) Sonnet, which provides a combination of skills and speed; (3) Haiku, the fastest and least expensive model. All models are multi-modal and demonstrate strong performance across benchmark evaluations. Due to the budget limits, we select claude-3-sonnet-20240229 API provided by Anthropic as a strong policy model to evaluate the alignment performances.

GPT-4 [1]. Developed by OpenAI, GPT-4 is a large-scale, multimodal foundation model that can accept image and text inputs and produce text outputs. GPT-4 marks the highest level of achievement in AI industry and exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. We leverage the strong capability of GPT-4 and use it as an oracle to evaluate the large-scale test samples. Considering the high cost of evaluating large-scale test data and our limited budget, we use the cheaper GPT-4-turbo model with the gpt-4-turbo-preview API provided by OpenAI in practice.

MetaAligner -(1.1B, 7B, 13B). Our proposed MetaAligner is the first policy-agnostic and generalizable method for multi-objective preference alignment. The models are based on TinyLLaMA and LLaMA2 foundation models. We train MetaAligner models on all 3 model scales for each of the 3 benchmark datasets. Specifically, HH-RLHFMetaAligner is trained to align the responses of a general daily AI assistant with specified objectives considering multi-turn dialogue contexts. UltraFeedbackMetaAligner is trained to align responses of another general AI assistant considering a single-turn query, but the queries include professional questions such as programming language and history, and the aligned responses are usually more complicated. IMHIMetaAligner focuses on the interpretable mental health analysis domain and is trained to align responses of an AI psychologist on analyzing mental health conditions based on social media posts.

D.3 IMHI Annotation

We select 1,200 queries from the IMHI benchmark covering 9 mental health analysis tasks. We obtain 4 responses to each query from 4 different policy models: GPT-4-turbo [1], GPT-3.5-Turbo [21], MentaLLaMA-13B [34], and LLaMA2-Chat-13B [30], with human annotations on ranking different objectives of the responses. We utilize the above policy models to generate explanations for the same query simultaneously. The annotation protocol is developed through collaborative efforts with 2 domain experts (Ph.D. students majoring in quantitative psychology) and considerations of human evaluation criteria for previous mental health analysis tasks [34, 33]. Specifically, 3 objectives are assessed: (1) Correctness : the explanations should make correct label predictions in

IMHI

1.0

Overall

0.35

0.4

0.25

-0.9

-0.8

Correct

0.37

0.44

0.7

0.6

Info.

0.43

0.67

0.5

0.4

Pro.

0.2

0.73

0.3

0.2

1

2

3

the corresponding mental health analysis task; (2) Informativeness : the response should express clear logic and provide consistent evidence; (3) Professionalism : the response should provide evidence with high quality and reliability from the perspective of domain experts. Each aspect is divided into four standards rating from 0 to 3. Higher ratings reflect more satisfactory performance and 3 denotes approaching human performance. Each LLM-generated explanation is assigned a score by 2 domain experts for each corresponding objective, followed by the examination of 1 domain expert. All annotators are PhD students majoring in quantitative psychology.

Annotators will be given generated responses from the 4 policy models and need to score and annotate the responses from the following objectives:

Correctness. Correctness measures the trustworthiness of the classification results. Annotators should assess whether the classification result is based on facts, has misinformation, and wrong reasoning according to the given post.

0: Completely unreliable information with factual hallucination (e.g. non-existent symptoms).

1: Partly reliable information with wrong reasoning based on facts.

2: Mostly reliable information with non-critical misinformation or wrong reasoning.

3: Completely reliable information.

Informativeness. Whether the text builds from sentence to sentence to a coherent body of information and logic about mental health and supports the classification results. Annotators should assess if the generated explanation gives consistent supporting evidence to its classifications and is well-structured.

0: Inconsistent with the classification results.

1: Consistent with the classification results, but with poor readability and several errors.

2: Consistent with the classification results. Mostly coherent and easy to read, with few minor errors.

3: Consistent with the classification results. Completely fluent, coherent, and error-free.

Professionalism. Professionality measures the rationality of the generated explanations by evaluating the evidence that supports the classification results from the psychology perspective. Annotators

should assess whether the explanation includes the following specified common diagnosis criteria of depression. To ensure the quality of the annotation scheme, we invite our domain experts to develop a list of common symptoms related to depression and sort these symptoms by criticality. The domain experts consult the Patient Health Questionnaire (PHQ-9) on determining the symptoms and sorting these symptoms on their knowledge.

Specifically, the following symptoms are checked (sorted by criticality):

Suicide ideation: Thoughts that you would be better off dead.

Self-harm ideation: Thoughts of hurting yourself in some way.

Feeling down, depressed, or hopeless.

Self-guilt ideation: Feeling bad about yourself - or that you are a failure or have let yourself or your family down.

Symptoms above are classified as with high criticality, and symptoms below are classified as with low criticality.

Feeling tired or having little energy. Little interest or pleasure in doing things.

Poor appetite or overeating.

Trouble falling or staying asleep, or sleeping too much.

Trouble concentrating on things, such as reading the newspaper or watching television.

Moving or speaking so slowly that other people could have noticed. Or the opposite being so fidgety or restless that you have been moving around a lot more than usual

Uncontrollable sexual desire or sexual frigidity.

Other symptoms.

Based on the above symptoms, the annotators score the professionality of each explanation with the following criteria:

0: The explanation provides no supportive evidence or symptoms with high criticality are missing in the explanation.

1: The explanation provides a few supportive evidence, while some symptoms with higher criticality (than provided evidence) are missing.

2: The explanation provides several supportive evidence, while some symptoms with lower criticality (than provided evidence) are missing.

3: The explanation provides all related supportive evidence in the post.

E Sub-task Performance on IMHI Benchmark

We stack MetaAligner on different policy models to perform alignment on all 3 objectives: "Correct", "Informative", and "Professional". We include MentaLLaMA-(7B, 13B, 33B) [34], the first opensource instruction-following LLM series for interpretable mental health analysis into the policy models. Details about the 9 sub-tasks are provided in Table 6. The overall performance of MetaAligner on the IMHI benchmark and its separation into 9 different sub-tasks are shown in Table 7.

According to the results, the MetaAligner models achieve substantial improvement in overall performance on all 11 policy models, with an average of 26.89% advantage on win rates for MetaAligner -1.1B, 28.01% for MetaAligner -7B, and 36.6% for MetaAligner -13B. These results show the general effectiveness of one MetaAligner on various upstream models and the feasibility of plug-and-play multi-objective alignment. MetaAligner also greatly improves performance on each sub-task. For example, MetaAligner -7B outperforms the unaligned outputs by over 25% on 7 sub-tasks. These results indicate that MetaAligner alignment can be effectively adapted to tasks that require different knowledge and response formats.

From the policy model scale perspective, MetaAligner provides successful alignments to models with sizes ranging from 2B to 70B, significantly extending the size of MetaAligner itself. In the extreme case, MetaAligner -1.1B advances the win-rate of LLaMA2-Chat-70B outputs by 21.18%, a

Training Information

Base Library

Huggingface Transformers

Fine-tuning Platform

FastChat

4 × NVIDIA Tesla A100 80GB GPUs

CPU Hardware

8 × Intel(R) Xeon(R) Gold 6342 CPU cores per GPU

Hardware Speedup

Flash Attention 2 [8]

Quantization for training

BF16

Fine-tuning Strategy

Full fine-tuning

Optimizer

Adam

Training Epochs

2

Batch sizes

HH-RLHF: 512 / UltraFeedback: 512 / IMHI: 128

Max token for training

MetaAligner -(1.1B, 7B, 13B): 2048/4096/4096

Learning rate

1e-5

Warm-up ratio

0.05

Base Model-1.1B

TinyLLaMA-1.1B

Base Model-7B/13B

Dataset Information

Dataset Name

License

Train/Val/Test ( D p )

Train/Val ( D e )

Harmless preference source

License

Helpful preference source

License

Humor preference source

License

Test evaluator

Dataset Name

License

Train/Val/Test ( D p )

Train/Val ( D e )

Instruction_following preference source

Honest preference source

Truthful preference source

Helpful preference source

Test evaluator

Dataset Name

License

Train/Val/Test ( D p )

Train/Val ( D e )

Instruction_following preference source

Correct preference source

Informative preference source

Professional preference source

Test evaluator

Policy Models

https://huggingface.co/meta-llama

Gemma-instruct-(2B, 7B)

https://huggingface.co/google

Vicuna-(7B, 13B, 33B)

https://huggingface.co/lmsys

GPT-3.5-Turbo

https://openai.com/blog/chatgpt

Claude-3-Sonnet

https://www.anthropic.com/news/claude-3-family

HH-RLHF

MIT

262,719/15,000/15,000

16,502/1,797

Ray2333/gpt2-large-harmless-reward_model

MIT

Ray2333/gpt2-large-helpful-reward_model

MIT

Apache-2.0

UltraFeedback

MIT

252,934/15,000/15,000

82,023/5,000

IMHI

MIT

5,304/1,051/2,400

3,374/689

Human annotation

Human annotation

Human annotation

Human annotation

Algorithm 2 GPT-4 win-rate computation

policy model with 63 × more parameters. These results prove MetaAligner as a parameter-efficient alignment strategy compared to previous multi-objective alignment methods, where the policy model weights are updated, leading to an inevitable surge of computation resources as policy model sizes grow. Besides the general-domain foundation models, MetaAligner also improves the performance by an average of 28.32% on MentaLLaMA models, which are fine-tuned on mental health analysis tasks. These results show that MetaAligner can make reasonable corrections on weak responses while maintaining their expertise from domain-specific policy models.

F Objective-wise Alignment

F.1 Experimental Settings

We randomly sample 1,200 queries from the UltraFeedback test set and probe the target policy models to provide responses to all queries, which are regarded as unaligned responses. Then MetaAligner is used to align these responses under different objectives, including aligning on each objective (Instruction following, Helpfulness, Honesty, and Truthfulness), and another full alignment on all 4 objectives. During evaluation, GPT-4 is leveraged as a reward model to score these responses considering different objectives. Specifically, the following prompt is developed to obtain the scores:

where q and a denotes the target query and response, and r ∈ {Instruction following, Helpful, Honest, Truthful} is the target objective for evaluation.

F.2 Experimental Results

More experimental results are presented in Figure 6. The results are the performance of MetaAligner -7B on Gemma-instruct-7B, LLaMA2-Chat-70B, and GPT-3.5-Turbo outputs from the UltraFeedback test set. According to the results, MetaAligner can perform effective objective-wise alignment on outputs of different policy models, from the small Gemma-7B model to the competitive commercial models GPT-3.5-Turbo. Unlike full alignment which shows weaker performance as the capability of the policy model increases, objective-wise alignment provides stable improvement for different policy models. For example, aligning "Instruction following" leads to over 20% of approachinghuman responses for all policy models, and aligning "Honesty" leads to over 30% of approachinghuman responses for all policy models. The reason is that single-objective alignment doesn't involve the complex interactions between multiple objectives, which facilitates the full realization of MetaAligner 's potential to improve the response on the corresponding objective.

G Implementation Details of Baseline Models

G.1 SFT-based Methods

Existing SFT-based methods [10, 35] for multi-objective preference alignment explicitly incorporate reward values into the query via prompt engineering, which includes a text marker for each alignment objective and their corresponding value numbers in the current response. The model is trained to predict the response given the enhanced query, which enables it to learn a mapping between the reward value and its reflection in the response. During inference, we achieve alignment by assigning a higher reward value to the target objectives. Specifically, we define the following prompting template for HH-RLHF dataset:

where q denotes the query, r 1 , r 2 , r 3 denote the corresponding reward values for the current response, ranging from 1 to 5. We define the following prompting template for UltraFeedback dataset:

where q denotes the query, r 1 , r 2 , r 3 , r 4 denote the corresponding reward values for the current response, obtained from existing reward models.

During inference, we aim to simultaneously align all objectives with 1 model to enable fair comparisons to MetaAligner . For UltraFeedback, since all rewards range from 1 to 5, we set all values to 5 during inference: r 1 = r 2 = r 3 = r 4 = 5 . For HH-RLHF, there are no unified ranges for each objective as all training values are obtained from reward models. Therefore, we set each objective value to its maximum in the training dataset to enable higher alignment. The values are set to: r 1 = 4 . 19 , r 2 = 3 . 03 , r 1 = 1 . 58 .

In calculating the GPU hours, we sum the training hours for fine-tuning the LLaMA-Chat-7B policy model for HH-RLHF and UltraFeedback datasets.

G.2 MODPO

We implement MODPO by implementing the Controllable Direct Preference Optimization (CDPO) [10] algorithm based on its paper introduction. We bypass the controllable preference SFT stage by utilizing the trained model from SFT-based methods as the foundation model. In the CDPO stage, each query q is accompanied by two responses c 1 and c 2 , where each response is constructed into contrastive pairs. In HH-RLHF, we have the following prompting template:

Gemma-instruct-7B

Instruction following

Helpfulness

0.25

0.20

0.20

0.15

0.05

0.05

0.00

GPT4score

0.00

GPT4score

Honesty

Truthfulness

0.4

Unaligned

hent Objectives

0.20

Density

0.2

0.3

Helpfulness

Truthfulness

Instruction following

Honesty

Full alignment

0.10

nsity

0.15

0.1

0.05

0.0

GPT4score

0.00

GPT4score

Instruction following

LLaMA2-Chat-70B

Helpfulness

0.25

0.20

0.20

0.15

0.10

0.05

0.05

0.00

GPT4score

0.00

GPT4score

Honesty

Truthfulness

0.30

0.20

0.25

0.15

0.10

0.05

0.05

0.00

GPT4score

0.00

GPT4score

Instruction following

GPT-3.5

Helpfulness

0.25

0.20

0.20

0.15

0.10

0.05

0.05

0.00

GPT4score

0.00

GPT4score

Honesty

Truthfulness

0.20

0.3

0.15

0.1

0.05

0.0

0

5

10

0.00

0

5

10

GPT4 score

GPT4 score

where r j i denotes the reward value for the i -th response on the j -th objective. For HH-RLHF, there are no unified ranges for each objective as all training values are obtained from reward models. Therefore, we set each preference value to its maximum in the training dataset to enable higher alignment. The values are set to: r 1 = 4 . 19 , r 2 = 3 . 03 , r 3 = 1 . 58 . Similarly, on UltraFeedback we have:

For UltraFeedback, since all rewards range from 1 to 5, we set all values to 5 during the DPO training process: r 1 = r 2 = r 3 = r 4 = 5 . In determining the preference scores, we use the CDPO learning goal to transform the task into a conditional multi-objective optimization problem. Specifically, the reward value R i for response c i is calculated as follows:

where ω i represents the weight of the i -th objective and g i is calculated as follows:

where λ i represents the weight of the controlled objective and p j is a pre-defined preference value for the j -th objective. In our implementation, we set ω i = λ i = 1 . We aim to simultaneously align all objectives with 1 model to enable fair comparisons to MetaAligner . For UltraFeedback, since all rewards range from 1 to 5, we set all values to 5 during inference: p 1 = p 2 = p 3 = p 4 = 5 . For HH-RLHF, there are no unified ranges for each objective as all training values are obtained from reward models. Therefore, we set each preference value to its maximum in the training dataset to enable higher alignment. The values are set to: p 1 = 4 . 19 , p 2 = 3 . 03 , p 3 = 1 . 58 . After calculation, the response with a higher reward value R i is used as the chosen response, and the other response is used as the rejected response for MODPO training. Specifically, the model is trained via the following loss function:

where x denotes the query, y w , y l denote the chosen and rejected prompts, c denotes the corresponding condition, π θ and π ref denote the target policy model and the reference model. For implementation, we build our MODPO code based on the OpenRLHF library.

In calculating the GPU hours, we include the training hours for tuning the SFT-based policy model using the CDPO algorithm for HH-RLHF and UltraFeedback datasets. We also include the training hours for fine-tuning the original LLaMA2-Chat-7B model for fair comparisons.

G.3 MORLHF

Weuse the linear scalarization [26, 19] realization of MORLHF with the KL-divergence regularization. Specifically, the model is trained via the following objective function:

where ω = [ ω 1 , ..., ω N ] s.t. ∑ N i =1 ω i = 1 , ω i ≥ 0 is the heuristic target preference vector, q , y denote the query and the response, R denotes the reward functions for the target objectives. In implementing the MORLHF algorithm, we first train a reward model for each objective based on random samples from 50% of the training data. Following the reward models we used in building the HH-RLHF dataset, we select the GPT2-large as foundation models for the reward model, and optimize the reward models using the following pair-wise loss functions:

where σ denotes the Sigmoid function, R c and R r denote the reward output of the chosen response and the rejected response, and margin denotes the margin loss for the corresponding response pairs when multiple responses are ranked, such as in UltraFeedback. Secondly, we fine-tune the LLaMA2Chat-7B policy model with the highest-ranked responses from the HH-RLHF and UltraFeedback datasets to obtain sub-optimal starting points for RLHF. The SFT training process is formalized as follows:

where q and y denote the query and its corresponding highest-ranked response. Thirdly, following most works in RLHF, we leverage the PPO algorithm [25] to enable parameterized optimization of the policy model. In linear scalarization, we set ω 1 = ω 2 = ... = ω N = 1 N . For implementation, we build our MORLHF code based on the OpenRLHF library.

In calculating the GPU hours, we include the training hours for all reward models in HH-RLHF and UltraFeedback datasets, with a sum of seven reward model training processes. We also include the training hours for fine-tuning the original LLaMA2-Chat-7B model for reaching the sub-optimal starting points. Finally, the PPO training hours for HH-RLHF and UltraFeedback are included in the GPU hours.

G.4 Self-Refinement

We include a prompt engineering-based self-refinement method as a baseline method to further demonstrate the effectiveness of MetaAligner. Specifically, this approach involves obtaining an initial response from the policy model and then prompting the same model to further refine its own output. For the second stage, we utilize the same prompting strategies as MetaAligner , which is detailed in Appendix C.2. However, this method often requires aligner models with strong in-context learning capabilities, leading to high inference costs due to larger model sizes or expensive commercial models. Therefore, we select LLaMA2-Chat-70B, a strong policy model as the target policy model for self-refinement. As self-refinement does not involve any training procedures, we do not report its GPU hours as other methods.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: all claims accurately reflect the paper's contributions and scope and are supported by experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Appendix A. They will be moved back to the main body once extra pages are allowed.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: the paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: all experimental setting are provided in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: we release all codes, trained models, and part of the training data to facilitate the reproduction of our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: all experimental setting are provided in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We mostly used GPT-4 as judges to rate the responses, a trustworthy but costly AI system. Repeating these evaluation processes or the LLM-based tuning algorithms would be too expensive and time-consuming.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: details about computational resources are provided in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: our research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: the broader impacts are discussed in Appendix A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: the safeguards are discussed in Appendix A.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: all used assets (datasets, source codes) are properly cited and their licenses are listed in Appendix A.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: the proposed models, datasets, and codes are provided with documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the research didn't involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the research didn't involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMulti-objective alignment of language models is a significant topic for LLM community, while many prior work are either costly-to-compute or policy-dependent, restricting the further depolyment. This work is aimed at providing a retraining-free and policy-agnostic approach for multi-objective alignment, namely *MetaAligner*.\n\nThe framework is composed of 3 stages:\n- Data reorganization from multi-objective datasets, getting $D_e, D_p$;\n- Supervised model training:\n   - Warming up on a randomly downsampled dataset of $D_e$;\n   - Finetune on $D_e$, as a equal-preference alignment step;\n   - Contrastive training on $D_p$.\n- Inference with prompting for even unseen objectives.\n\nExperiments are conducted on three commonly-used datasets, and the improvements are reported as prominent. Furthermore, the authors claim that they can achieve generalizable alignment to unseen objectives, through prompting.\n\nSTRENGTHS:\nOriginality:\n- The whole 3-stage framework of *MetaAligner* is original.\n\nClarity:\n- This paper is well-written and well-organized.\n- Most figures and tables are friendly to the readers.\n\nSignificance:\n- This is the first policy-agnostic and retraining-free multi-objective alignment approach, to the knowledge of the reviewer.\n- The experiments are extensive and very solid, including multiple datasets and base models. The author also conducted extensive experiments comparing *Meta-Aligner* with other baselines.\n\nWEAKNESSES:\n*Major*\n- The whole framework is complicated and a bit engineering. There is no ablation study on each components. For example, it is better to show the performance if discarding the $D_e$ dataset, which is claimed as novel.\n- In Table3, the comparison of *MetaAligner* with other baselines is somehow unfair, since *MetaAligner* utilizes at least double parameters of *MORLHF*, *MODPO*, and *RiC* (the 1.1B version performs worse than baselines, so the reviewer is talking about >=7B version).\n- The proposed method is not able to cater to different users' taste, because it can only be aligned to multiple objectives while no weights on them are introduced. But many works[1][2][3][4][5] are able to do this, making the contribution less significant.\n\n*Minor*\n- this work is a bit marginal:\n  - The reorganization process is just preparing the prompting format, which is trivial.\n  - The SFT training is a common practice in the community[1][2][3]. The only difference is the design of a detached aligner.\n  - The inference stage is simply prompting engineering, lacking guarantees.\n- The *MetaAligner* is not a light-weight approach, as the aligner model is still large, making it hard to be deployed.\n\n[1] Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment. arXiv preprint arXiv:2402.10207.\n\n[2] Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment. arXiv preprint arXiv:2402.19085.\n\n[3] Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards. arXiv preprint arXiv:2402.18571.\n\n[4] Decoding-Time Language Model Alignment with Multiple Objectives. arXiv preprint arXiv:2406.18853\n\n[5] Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. arXiv preprint arXiv:2310.11564.\n\nQUESTIONS:\n- In Figure 3, why would the performance on some unseen objectives drop after adding more objectives? For example, \"repeat\" and \"readable\" for *MetaAligner-7B*.\n- How much efforts has the author spent on tuning hyper-parameters of *MetaAligner*?\n- How does the author set the preference weights for *MORLHF*, *MODPO*, and *RiC*? Is that fair?\n\nLIMITATIONS:\nThe limitations are well discussed in Section A.2.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposes Meta-Objective Aligner (MetaAligner), the first policy-agnostic and generalizable method for multi-objective preference alignment.\n\nSTRENGTHS:\nAs a lightweight algorithm, this is particularly meaningful as model parameters continue to grow. Conditional weak-to-strong correction extends weak-to-strong correction and weak-to-strong generalization, which are currently significant research topics.\n\nWEAKNESSES:\n1. I am not entirely clear about the difference between this paper and Aligner. The author mentions:\n> As shown, conditional weak-to-strong correction of MetaAligner extends Aligner [12] to multi-objective alignment scenarios, which are not directly solvable by Aligner itself.\n\nI do not fully understand the distinction between this paper and Aligner. After reading the Aligner paper, I noticed that the authors effectively expand on multi-objective alignment. However, the introduction seems to lack a description of Aligner, making it difficult for readers to understand.\n\n2. I did not understand what the author is trying to convey with Figure 2. Can the author explain the meaning of Figure 2? The elements it aims to express are very complex. Could there be a more streamlined figure, similar to Figure 1 in the Aligner paper? Is the extension to multi-objective alignment at the methodological level or the dataset level?\n3. In the experimental section, how did the authors generate an effective correction dataset based on HH-RLHF, UltraFeedback, and IMHI?\n4. After reading the Aligner paper, I observed that it conducted experiments on the three dimensions of \"Helpful,\" \"Harmless,\" and \"Honest\" as described by the authors in line 42. I hope the authors can add comparative experiments with Aligner in the ablation study section. From a reviewer's perspective, the contribution of this paper to LLM alignment is undeniable, but I am particularly concerned about the effective improvement and extension of MetaAligner compared to Aligner.\n\nMy questions are all included in the Weaknesses section, with point 4 being my primary concern. I hope the authors can effectively enhance the ablation study section. I believe that algorithms like MetaAligner and Aligner are more efficient compared to RLHF and DPO. I look forward to the authors' response and will consider raising the score further.\n\nQUESTIONS:\nsee above.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes MetaAligner, a plug-and-play multi-objective alignment method that can generalize to unseen objectives.\n\nSTRENGTHS:\n1. This is a well-written paper that clearly expresses its core ideas.\n2. MetaAligner is a lightweight alignment method that is easier to tune in multi-objective scenarios, with lower training overhead compared to baselines.\n3. MetaAligner is a plug-and-play alignment method that can generalize to various open-source and closed-source models, enabling multi-objective alignment.\n4. MetaAligner has generalizability, capable of aligning with zero-shot objectives.\n\nWEAKNESSES:\n1. In the experimental section, the aligned answers are compared to the ground truth answers in the dataset to calculate the win rate. This lacks a more direct performance comparison, making the argument for performance advantage insufficiently robust. For example, comparing different baselines by win rate against the ground truth answers in the dataset is inadequate to demonstrate the performance advantage over the baselines.\n2. There is a lack of performance comparison with prompt engineering-based aligners. Specifically, it is unclear whether similar effects could be achieved by prompting the chat model to modify the answers within the prompts.\n\nQUESTIONS:\n1. I am interested in understanding the authors' rationale for choosing an indirect comparison with the ground truth and how they view the validity of this choice, especially considering that many models, after being aligned through RLHF, are likely to already exhibit strong performance on the objectives of test set.\n2. In the scenario where columns contain exactly two of the three attributes: Harmless, Helpful, and Humorous, which corresponds to the position of the second column in Figure 2 HH-RLHF. Let $( x_1, x_2, x_3 )$ denote the probabilities of having two characteristics, specifically including $Harmless$, $Helpful$, and $Humorous$. According to the figure, $( x_1 = 0.57 )$, $( x_2 = 0.53 )$, and $( x_3 = 0.85 )$. Now, there are exactly three possible combinations of choosing two out of the three characteristics: $(Harmless, Helpful)$, $(Helpful, Humorous)$, and $(Harmless, Humorous)$, and these combinations are mutually exclusive. Let the probabilities of these three events be $( a_1, a_2, a_3 )$ respectively, such that $( a_1 + a_2 + a_3 = 1 )$. The event `contains two characteristics, with one of them being Harmless` can be described as the disjoint union of the events `only contains (Harmless, Helpful)` and `only contains (Harmless, Humorous)`. Therefore, $( x_1 = a_1 + a_3 )$. Similarly, we have $( x_2 = a_1 + a_2 )$ and $( x_3 = a_2 + a_3 )$. Summing both sides of these three equations, we obtain $( 1.95 = 0.57 + 0.53 + 0.85 = x_1 + x_2 + x_3 = 2(a_1 + a_2 + a_3) = 2 \\times 1 = 2 )$. I am sincerely inquiring whether my understanding of Figure 2 is mistaken.\n\nLIMITATIONS:\nThe article provides a thorough discussion of its limitations in terms of deployment and the number of objectives to be selected.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This work extends Aligner to multi-objective alignment scenarios. The main contribution is adding a textual description to each sample in the existing preference datasets to indicate the reason for the preference between chosen and rejected samples. The authors found that MetaAligner is more efficient than previously trained multi-objective alignment algorithms. However, this work is incremental, with some unclear details, insufficient evaluation benchmarks, and a lack of baselines.

STRENGTHS:
Combining meta-learning concepts with alignment and guiding the model's alignment direction through textual descriptions makes sense to me and could potentially become an important part of RMs in future RLHF works.

WEAKNESSES:
1. The innovation is incremental, essentially applying Aligner to a new domain and altering training data to change the alignment objectives.

2. The most critical detail, which is how the preference descriptions between each chosen and rejected pair in the existing preference datasets are obtained, is unclear.

3. The authors did not adequately present results on objective datasets such as MATH and HumanEval, nor on mainstream subjective evaluation sets like MT-Bench/AlpacaEval.

4. There is a lack of comparative analysis with previous multi-objective alignment methods, such as "SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling."

QUESTIONS:
1. How are the objectives in Section 3.1 obtained?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-16