MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models
Authors: Kailai Yang
Zhiwei Liu
Qianqian Xie
Jimin Huang
Tianlin Zhang
Sophia Ananiadou
Venue: NeurIPS 2024 poster (2024)
Keywords: Language Models
Multi-objective Alignment
Preference Optimization
Primary area: natural_language_processing
TL;DR: We propose MetaAligner, the first policy-agnostic and generalizable method for multi-objective preference alignment.

Abstract: Recent advancements in large language models (LLMs) focus on aligning to heterogeneous human expectations and values via multi-objective preference alignment. However, existing methods are dependent on the policy model parameters, which require high-cost repetition of their alignment algorithms for each new policy model, and they cannot expand to unseen objectives due to their static alignment objectives. In this work, we propose Meta-Objective Aligner (MetaAligner), the first policy-agnostic and generalizable method for multi-objective preference alignment.
MetaAligner models multi-objective alignment into three stages: (1) dynamic objectives reformulation algorithm reorganizes traditional alignment datasets to supervise the model on performing flexible alignment across different objectives; (2) conditional weak-to-strong correction paradigm aligns the weak outputs of fixed policy models to approach strong outputs with higher preferences in the corresponding alignment objectives, enabling plug-and-play inferences on any policy models, which significantly reduces training costs and facilitates alignment on close-source policy models; (3) generalizable inference method flexibly adjusts target objectives by updating their text descriptions in the prompts, facilitating generalizable alignment to unseen objectives.
Experimental results show that MetaAligner achieves significant and balanced improvements in multi-objective alignments on 10 state-of-the-art policy models, and saves up to 93.63% of GPU training hours compared to previous alignment methods. The model also effectively aligns unseen objectives, marking the first step towards generalizable multi-objective preference alignment.

MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models
Kailai Yang' Zhiwei Liu'
Qianqian Xie Jimin Huang*
Tianlin Zhang'
Sophia Ananiadou'
The University of Manchester 2 The Fin Al
(kailai, yang, zhiwei. zhiwei, Liu, angpiiiaaaaaaaaaaddgaanncceceee (xqq. sincere, zhangt inn Shangtianlin668 inn68 @egmail com; wiinnccacccccocccs...... com
Abstract
Recent advancements in large language models (LLMs) focus on aligning hetero- geneous human expectations and values via oulti-objective preference alignment. However, cxisting methods are dependent the policy model parameters, which require high-cost repetition of their alignment algorithms for each new policy model, and they cannot expand unseen objectives due to their static align- ment objectives. In this work, we propose Meta-Objective Aligner (MetaAligner), the first policy-agnostic and generalizable method for multi-objective preference alignment. MetaAligner models multi-objective alignment into three stages: (1) dynamic objectives reformulation algorithm reorganizes traditional alignment datasets to supervise the model on performing flexible alignment across differ- ent objectives; (2) conditional weak-to-strong correction paradigm aligns the weak outputs of fixed policy models to approach strong outputs with higher prefer- ences in the corresponding alignment objectives, enabling plug-and-pla inferences any policy models, which significantly reduces training costs and facilitates alignment on close-source policy models; (3) generalizable inference method flexibly adjusts target objectives by updating their text descriptions in the prompts, facilitating generalizable alignment to unseen objectives. Experimental results show that MetaAligner achieves significant and balanced improvements in multi- objective alignments on 10 state-of-the-ar policy models, and saves up to 93.63% of GPU training hours compared to previous alignment methods. The model also effectively aligns unseen objectives, marking the first step towards generalizable multi-objective preference alignment. This project is open-sourced here.
Introduction
The recent advancements in large language models (LLMs) have focused on generating high-quality responses that align with human expectations and values. At the final stage of alignment, LLMs are supervised human preference data via reinforcement learning from human feedback (RLHF) [40, 22, 27], where a proxy, directly trained on human preferences data, is leveraged to provide scalar rewards for reinforcement learning (RL) on the target model [22].
However, human expectations and values include a broad spectrum of heterogeneous and multi- dimensional objectives, which makes scalar supervisions inefficient for aligning diverse and in- clusive human preferences [3, 24). These drawbacks motivate further exploration into multi- objective alignment algorithms. Some intuitive methods extend RLHF into multi-objective RLHF (MORLHF) [26, 19, 24]. Due to its substantial computational cost [19, 24] and the unstable nature
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 1: Comparisons between previous alignment methods and MetaAligner different features. "Policy-Agnostic Alignment" means the alignment algorithm is independent of the target policy model parameters, and "Generalizability" denotes zero-shot alignment capability on unseen objectives.
Algorithm
Paradigm Multi-Objective Alignment Policy-Agnostic Alignment Generalizability
PPO MORLHF [19]
RLHF [22]
x
PPO
MODPO [10, 39]
SFT. DPO
XXM
RiC [35]
SFT
Aligner [12]
SFT
X
MetaAligner
SFT
of the proximal policy optimization (PPO) [25, 15, 23] algorithm, other methods seek to bypass the RL paradigm with multi-objective direct preference optimization (MODPO) [39, 10] supervised fine-tuning (SFT)-based methods [35, 10J, which customized prompting strategies to incorporate multiple reward values into queries explicitly.
The above methods for oulti-objecctve alignment bear one commonality: the dependence on the policy model's 's parameters. This paradigm inevitably brings two key limitations: (1) they require repetition of their high-cost alignment algorithms for each newly-introduced policy model, which is incompatible with the increasing sizes and fast iteration of current foundation models [1, 30, 6, 29]; (2) all target models are statically aligned on pre-determined (e.g. "Helpful". "Harmless", "Honest" [39, 10] objectives, with currently efforts in expanding and evaluating their capabilities unseen objectives. This ignorance leads to poor generalizability of existing multi-objective alignment methods.
In this work, we propose Meta- Objective Aligner (MetaAligner), the first policy-agnostic and gen- eralizable method for multi-objective preference aligmment. MeluAligner models multi-objective alignment into three stages: (1) the dynamic objectives reformulation algorithm reorganizes tradi- tional alignment datasets into crnamic--ojecctvve aligmment datasets, training MetaAligner to perform flexible alignment across different objectives. It achieves this by incorporating and combining text descriptions of various alignment objectives in a prompt-based manner; (2) the conditional weak-to- strong correction paradigm aligns the weak outputs of policy models to approach strong outputs with higher preferences in the corresponding alignment objectives. During training, MetaAligner is stacked onto policy models to perform objective-awas corrections, where parameters of the policy model are fixed and MetaAligner is optimized with an SFT-based three-step training process: warm- ing up, equal-preeerence alignment, and contrastive- contrastive-preference alignment. This paradigm enables MetaAligner to perform plug-and-play inferences on any policy models even without access to their parameters, which significantly reduces training costs and facilitates alignment on close-source LLMs; (3) the generalizable inference method flexibly adjusts target objectives by updating their text descriptions in the prompts. This method can also adapt MetaAligner to unseen objectives and achieve new alignment strategies via in-context learning [14], a new feature with rare previous explo- ration in alignment of language models. The number of aligned objectives also becomes expandable, theoretically leading to unlimited simultaneous alignment objectives. Table 1 compares key features between MetaAligner and previous methods. As shown, conditional weak-to-strong correction of MetaAligner extends Aligner [12] to multi-objective alignment scenarios, which are not directly solvable by Aligner itself. MetuAligner is also the first multi-objective alignment method to achieve policy-agnostic alignment and generalization to unseen objectives, two key advantages over previous methods such as MORLHF, MODPO, and SFT-based methods.
In summary, our main contributions are: (1) we propose MetaAligner, the first policy-agnostic method for multi-objective preference alignment. performs multi-objective alignment efficiently, without tuning the policy models or accessing their parameters. Experimental results show that MetaAligner outperforms previous alignment methods and saves up to 93.63% of GPU training hours; (2) we utilize MetaAligner to exert zero-shot preference alignment for unseen objectives. To our knowledge, this work marks the first attempt at generalizable multi-objective preference alignment. Experimental results show that MetaA can simultaneously perform effective alignment for six unseen objectives while maintaining performance on aligned objectives; (3) We examine MetaAligner on three preference alignment datasets. Experimental results show that MetaAligner improves
2

---

win rates on multiple objectives across 10 policy models, substantially enhancing responses of state-of-the-art foundation models such as Turbo [21]
and Claude-3 [2].
Stape Dyrami Objeciive Reformutation
Condttional Weal-te-Strong
Gemerstirale teference
Huran aftert
da meight @ waighteudardd
Raw
Womesg Sellurr
Correction
Reformulation sigurthhm
Sunrali
Unseen Objecthees
rodor
Alaeed CuEquE
MetaAligner
is farts and berall and anse vagge argenertt.
buo fechuill cemeer and rold full ast wiror Shothe dwrs.
Contrvation Suboot
MetaAligner
Kijamere
Haaee loso th' culcente shuid aviid coetcee offerones dur
Aligned
Warmed-Up Model
Target Palicy Model
+
skuleiib ni resperao siodl oprees
Contrattoe-rrbeeenc Aligmmnt
Figure 1: Illustrations of Meta-Objective Aligner, which follows three-stage paradigm.
Multi-Objective Alignment of Language Models
In real-world scenarios, human expectations of high-quality responses from Al agents involve considerable variability, with complex interplays such as contradiction (e.g. "Helpful" and "Harm- less") [10, 35] and dependence (e.g. "Correct" and "Informative") [34]. Multi-objective preference alignment tackles this challenge by aiming to optimize multiple objectives simultaneously. For each query-response pair, the reward vector is formalized as: R(g, v) [ri v)...., T' (g. v) where q. y denote query and corresponding response, r; denotes the reward values for i-th objective, which defined, in most cases of preference alignment, under the Bradley- Terry [5] model of preferences. Specifically, for the same prompt q and two responses (1/1. under data distribution D, the model assumes:
Po(y > vzlg, (r:((g 31) rr(g, vz))
(I)
where a denotes the logistic function and PD(y) > y2) denotes the probability that yı is preferred against 1/2. MORLHF aims to achieve Pareto optimal among objectives, where the policy model optimized maximize linear scalarization of multiple rewards [26, 19] with a KL-divergence regularization:
argina Bu-DDuur, w (..TRaaw) w) &D [To( (v14))
(2)
T
where Tu denotes the aligned policy model parameterized by ø. Tref denotes the reference policy model, [wi,..., WN] s.t. > Wi 1, w; 0 is the pre-determined heuristic target prefer- ence vector. Another paradigm directly built alignment between multiple reward values and their corresponding response by minimizing an SFT loss for the policy model:
argmin BBg.))>>> (log (vlg. R(g. v))
where objectives and their corresponding reward values are described with text markers and combined into queries with a static prompting template. Compared MORLHF, SFT-based multi-objective alignment is proven more cost-efficient and training-stable [35, 10).
3 Meta-Objective Aligner
Existing methods for multi-objective alignment generally face challenges in increasing training costs with new policy models and generalization to unseen objectives. To tackle these challenges, we introduce MetaAligner, which follows three-stage paradigm: (1) dynamic objectives reformulation for building dynamic multi-objective datasets; (2) conditional weak-to-strong correction for model training; (3) generalizable inference for multi-objective alignment. The paradigm is illustrated in Figure I.
3.1 Dynamic Objectives Reformulation
We propose dynamic objectives reformulation algorithm to construct dynamic multi-objective dataset, which triggers MetaAligner's 's ability for flexible adjustment of alignment objectives. Specifi- cally, any typical multi-objective preference alignment dataset D,,, with samples and N objectives
3

---

can be re-orge re-organized as {qs, Vi, Mez, P. where P. = [P:1 (P.1.....P.NN and Pij E (, indi- cates the preference on j-th objective.
We define a text description for each objective: (di),,.. (, where (dz) denotes the natural language description for j-th objective. Some examples of such descriptions are in Figure and a full list is in Appendix C. With a pre-defined prompting template T, we build contrastive subset De and another equal subset De from Dm- De includes all contrastive response pairs where p {> }} and D. includes all equal response pairs where p E (=). For example, we utilize the following template TT V, O,t) in building for the IMHI [34] dataset:
Algorithm 1 Dynamic objectives reformulation.
Require: Raw dataset Dm f%. M11, Viz. P. )
Objective text descriptions: [(d,) ,..,, (dw));
Prompting template: T(g. O,1)
Ensure: Contrastive subset Dr; Equal subset D..-
1: De < 23, De + Ø > Initialize the 2 subsets.
2: for {1, [L,...,]] m do
Loop on instances.
3: O₂ f Ø, Ox f 2.
O, + Ø
4: for [I, ] do
5: pij is > then
D Collect the
objectives where Y/1 outperforms V/2-
6:
O- + {(d)) else if Ptj is then > Collect the
7:
[T y, O,t)] Edit the following Question- Answer pair to make it {&} considering the following objectives {O) Question: {q}
objectives where 1/2 outperforms ya-
8:
+ Ox {(d,)}
9:
else a Collect the objectives where y1 and 1/2 performs equally.
Answer: fy Edit:
10:
O= O= ((d))
where q denotes the query, y denotes a corre- sponding response, O denotes the concatena- tion of text descriptions for the target objectives, and t {equal, better) depends on the current building subset. Details of the dynamic objec- tives reformulation algorithm are described in Al- gorithm 1. For an instance within the processing loop (line 2): {g, / P). the algorithm per- forms a two-step reformulation: (1) collect the sets O O O_ that includes objectives where y1 outperforms y2. outperforms yı, and both perform equally (lines 3-12); (2) for each objec- tive set O we randomly shuffle the objectives to further trigger the model's flexible alignment ability. and build query-response pairs based on the corresponding prompting template T (line 13-27). All prompting templates and examples of the algorithm are presented Appendix C.
11:
end end for
12:
13:
if O~ * Ø then Build the training pairs
where Yı1 is used as the target.
14:
t t better
15:
O, random vandom_shuffle(O)
16:
D. + DCU((T(9, Vi2, Oz 1), va))
17:
end if
18:
if O= * Ø then> Build the training pairs
where y:2 is used as the target.
19:
1 + better
O~ t random random_shuufteed shuffle((O))
21:
D. + D.U((T(9: Vii, O y12)
22:
end if
23:
if O= * Ø then Build -reeel--eeeerred
training pairs.
24:
t equal
O - random random_shuflle shuffle(O=))
Training on dynamic multi oujei--beecctive datasets provides three key advantages: (1) instance- level alternation of the target objectives during training enables MelaAligner to perform flexible aligmment under different combinations of objectives; (2) mutual alignment between the same re- sponse pairs on different objectives fully leverages the supervision information in the preference vectors. (3) the reward-free alignment method (no explicit preference values required) avoids compli- cated preference-to-rewee mapping [35] process in previous SFT-based mujti-objective alignment methods.
26:
D. + D. DeU((T (qis Vi2, 0),t),aa)
27:
end
28:
end for
3.2 Conditional Weak-to-Strong Correction
Based on the dynamic multi-objective training datasets, we train MetuAligner in conditional weak- to-strong correction manner, which follows an SFT-hased training objective and three-step training paradigm.
3.2.1 Training Objective Derivation
MetaAligner is standard conditional seq-to-seq model on top of the original policy model T&, which re- re-distriutes the policy model output yo considering objectives O as follows:
(vlg) do(yTT(g, yo, O, To
(4)

---

where dg denotes the MetaAligner module parameterized by 6, t depends on the training dataset. Conditional yeak-to-strong correction directly trains MetaAligner to align the weak policy model output yo to the strong target output y, which has higher preference values in corresponding objectives O. We have the standard cross-entropy loss as the training objective:
argmin C(0, Φ; D) -) (log x (yl9)] 9.6
(5)
-(o... (log 5o (y) yo. 0,t))] - -Eg-D (log (Bolg))
We fix the parameters of the policy model, thus excluding & from the weight update process. In practice, we use the dynamic multi-objective dataset for supervision, where the weak response in each query-response pair is directly leveraged as samples yo from unknown policy models. Therefore, we eliminate the second term in Eqn. 5 and simplify the training objective as:
grgnnin O.))- [Log de (v (4, Ma, O,4)]
(6)
The above action poses two advantages: (1) the computation resources required for MetaAligner training is detached from policy model size, which enables volicy-agnosti and cost-efficient align- ment for large policy models: (2) MetaAligner works only via outputs from the policy models, which allows training and inference for alignment close-source policy models [1, 21, 2].
3.2.2 Three-Step Model Training
In practice, we utilize an LLM as the base model for MetaAligner, which provides domain knowledge and strong reasoning ability to support the conditional weak-to weak-to-strong correction process. We propose three-step paradigm based on the objective function in Eqn. 6: (1) Warming up. This stage trains the model in identical response pairs with a warm-up subset, a prelude proven effective in residual correction strategies [11, 12]. We randomly sample a subset of the equal subset De as the warm-up subset, but set an identical target response for each instance; (2) Equal-preference alignment. Due to the contrastive nature of their learning paradigm, most previous preference alignment works focus modeling the residuals between response pairs and ignore the equal- preference response pairs. However, equal preferences are common in many scenarios [34, 7] and enclose useful information such as the principle components of preference modeling regarding each objective. Based on these intuitions, we introduce a novel equal-preferent alignment step to fine-tume the warmed-up up MetaAligner on the equal subset D.; (3) cuntrastiv--reefere stage tine-tunes the MetaAligner on the contrastive preference subset De, which instructs the model to perform conditional weak-to-strong correction the specified objectives.
alignment. This
3.3 Generalizable Inference
During inference, MetaAligner achieves alignment following the sampling process as in Equ. 4, where unaligned outputs, sampled from the target policy model, are used as the input for conditional weak- to-strong correction. With the prompting-bas
paradigm, the target objectives for MetaAligner also become expandable and generalizable, key advantage over previous alignment methods [39, 35, 10]. The generalizability is two-fold: Firstly, users can manipulate the target objectives by adjusting combinations of text descriptions in the objective set O. For example, in alignment with objectives 3, and 4, we can flexibly shuffle the corresponding descriptions (di). (dz), and (d4) as follows: O (da); (di); (d4). Secondly, the prompt-based objectives statement enables flexible adjustment of text descriptions for existing objectives and injections of unseen objectives. Following the last example, we have two unseen alignment objectives 5: (dī) and 6: (dz). and updated text description (dg) for aligned objective 3. We can perform zero-shot alignment on the new objectives by adjusting O as follows: O (da); (d1): (d1); (dg); (dī). This simple pattern can theoretically lead to unlimited simultancous alignment objectives. We expect MetaAligner to make generalizable weak-to- corrections under these unseen conditions via its in-context learning ability. This advancement marks a new exploration into generalizable multi-objeectve preference alignment.
Experiments
4.1 Experimental Settings
Datasets. We transfer the following three alignment datasets into dynamic multi-ohjective datasets: (1) HH-RLHF [3]: large-scale dataset with 160K prompts and corresponding response pairs.
5

---

We follow Yang et al. [35] and use nourred reward models on three objectives: "Harmless", "Helpful", and "Humor" score the responses; (2) UltraFeedback [7]: multi multi-aspect alignment dataset with 64K prompts with preferences obtained from GPT-4, including "Instruction following", "Honest", "Truthful", and "Helpful" objectives: (3) IMHI: we create an alignment dataset on the IMHI dataset [34] targeting interpretable mental health analysis. We invite domain experts to label 7.2K response pairs considering 3 objectives: "Correct", "Informative", and "Professional.. Figure two datasets. The objectives display balanced overall distributions across objective set sizes, training MetaA ligner to adjust targets dynamically. Most objectives also cover considerable proportions in each column category, alleviating label imbalance problems.
shows the objective distributions
HH-RLHF
UltraFodback
Models. We train MetaAliignr--1.1B, 7B, 13B) models based on TinyLLaM--I.IB [37] and LLaMA2- 13B) [30] foundation models. We utilize MetaAligner to perform oulti-objective alignment on the following open-source policy models: ILaMA2- Chat- 13,,7,,,,,,7000 138,70B) [30], Gemma-instruct- (2B,7B) [29], and Vicuna-(7B, 13B, 33B) [6]. We also align two advanced close-source foundation models: GPT-3.5-Turbo [21] and Claude-3-Sonnet [2], where model parameters are inaccessible.
Tape Trshkl Hooeet
Figure 2: Heatmaps of the objective distributions. The columns categorize samples according to the sizes of their objective set. For the lines, "Overall" shows their distributions in the training data. Other lines show objective- wise distributions across dif- ferent categories in the columns.
Evaluation Metric. On each objective, we quantify the aligmment performance of model outputs by comparing their win rates against the ground-truth response provided by the bench- mark datasets. Considering the large amounts of test samples, we leverage GPT-4 [1], widely utilized evaluation tool in previous works [10, 28, 18), to perform the judgments. Each target response, ground-trutt response, query, and evaluated objec- tives are provided via prompt engineering. GPT-4 is required to compare and select the response with higher alignment on the specified objective or indicate a tied performance of the two responses.
More details about the training process, model cards, dataset statistics, IMHI dataset annotation, and evaluation settings are presented in Appendix D.
4.2 Overall Performance
MetaAligner- 7B, 13B) performance on 3 alignment datasets are shown in Table 2. According the results, the MetaAligner models achieve substantial improvement for most objectives and policy models. For example, on UltraFeedback, there is an average of 11.47% advantage for MetaAligner- 1.1B on "Honest", 34.39% for MetaAlignnr--7. and 43. 43.79% for MetaAligner- 13BB. These results show the general effectiveness of MetaAligner on various upstream models and the feasibility of plug- ald-play oujti-obeeectve alignment. On the mental health analysis benchmark IMHI, MetaAligner models also show remarkable win rates on all objectives, proving their effectiveness in performing multi-objective alignment domain knowledge-intense scenarios. We further evaluate MetaAligner on each IMHI sub-task and the results are shown in Appendix E.
From the policy model scale perspective, MetaAligner provides successful alignments to open-source models with sizes ranging from 2B to 70B, significantly extending the size of MetaAligner itself. In the extreme case, Metaaligner--In I. advances the win rates of LLMAA-Chaa---
outputs, a policy model with 63x more parameters, by an average of 12. 19% on HH-RLHF, 13.08% on UltraFeedback, and 13.26% on IMHI. These results prove MetaAligner compared to previous multi-obe muuti-objeective alignment methods, where the policy model weights are updated, leading to an inevitable surge of computation resources as policy model sizes grow. MetaAligner also significantly improves performance close-source LLMs: GPT-3.5-Turbo and Claude-3-Sonned These results prove its potential for application close-source scenarios and effective multi-objective alignment of ofth-oofthe-art policy models.
parameter-effiiinn alignment strategy

---

Table 2: Performance of MetaAligner--(.IB. 7B. 13B) on 3 datasets over different policy models. The responses are simultaneously aligned on all trained objectives, then evaluated on each objective. "IF" denotes the "Instruction following" objective. "+" shows the advantage of aligned outputs over the unaligned outputs on win rates against the ground-truth responses.
HH--LHH
UitraFeedback
IMHI Informative
Mensklineer Policy Model
Harmles
Helpful
Humor
Honest
Truthful
Helpful
Correct
Professional
LLaMA2-Chut
-10.05
+20,0%
+14.75%8
+11,0%
+15.0
+14.33%
+18.33%
+31,67%
LLaMAZ Chal 13B
+10,75%
+9,00%
+13.2515
+5.66%
+15.34%
+16.33%
+7.67%
+11.13%
+4.33%
+25.0%
L1MAA-CCuu-70R
-6.58%
+7.42%
222558
+12.67%
+17.33%
+16.35
+8.35%
-14.23%
+17.25%
Gemma instruct
-12.25%
+12.33%
+1467%
+16.7%
+35.55%
+37.33%
Gemma instruc--
+25.17%
+18.%
Vicana-78
+11.5%
+10.82%
+20.3316
+11.335
+13.23%
+7.0%%
+10.0%
+7.2255
+6.33%
Vicuna
-7.42%
+13.0%
+19.1716
+11.05%
+16.34%
+15.33%
+10.01
+2.75%
Vicans-338
-3.5%
+33.3%%
+3.0%
+11,77%
+6.65
+3.34%
G.T...-TTub
-1.4256
+5.0%
+5.0%
+1.0%
+9,67%
+1.5556
+9.33%
Chadde33 Someet
-3.83%
+1.58%
+13.175
+4.67%
+2.67%
+2.87%
+3.0%
+7.0%
+2.33%
+6.66 666
LLaMA2-Chrr Caat2-Cha-730
-15.0%
+27 8%
+50.55%
+5.00%
+3.0% 0%
+37.0%
+28.00 0%
-21 67%
-32.22%
89%
LLMMA2-Cha-133
+38.5%%
-20 58%
+1.25%
34.0%
347344%
899
+3.35
+25 56%
+30.0%
89
LLaMA2 Chat 70B
+16.55%
+14,42%
+3,00%
+27.0%
+3.33%
17.00%
20..50%
-17.25%
+21.67%
Gemma instruc: 2B
-20.0%
-18.75%
+17.3%
+31.33%
-0.67%
+4..3%%
+31,33%
+25 25.0%
-30.55%
+51.67%
7B
Gemmaniastrus--7
Vicuna-78
-11.0%
-19,5%
+18.33%
-23.25%
33%
426678
+53,67%
38.005
455
43900%
+37.05
+82.355
+20.0%
+35011 %
23.335
+52.235
-22,7955
+56.11% 56
+33,3356
Vicsns- 13B
-14.92%
+31 10%
670
+5466%%
+0 0%
59.9 59.67%
+36.4%
-25.59%
-20.00
15.011
Vicuns-338
-28.05
-17.00%
+30.3%
+0.00
52 33%
+20.35
-11.116
-16.115
+8. 34.
3.5
+15.92%
+21.55
+22.80%
+29.99% 99%
+303411
+28.0%
+14.34%
+1867%
-16.33%
+14.23%
Clande 3- Somnet
+19.17%
-19.08%
+26.175
+22.33%
+2.0%
+1 21.65%
+19.0%
+11.3%
-19.37%
+11.33%
LLaMA2 Chat 70
-24.0%
+30.5%
+23.75%
+5188%
+07.5%
+5.33%
+35.07%
-28.33%
-38.3396
+50.56%
LLaMA3-Chat-11
+17.75%5
+16.58%
+15.75%
+46.33%
648667
46.83%
+41.17%
+30.56%
-37 225
+40.5695
LLaMA2-Chat-13 70B
+6..55%
+19,42%%
+26.385
+44.33%5
435.0%
+45.5%
+34.0%
+31,67%
-30,564
436, 12%6
Gemma-inttra-226
-18.5%
-17.25%
+34-338
+55.0%
+44 67%
+5.33%
+56.3%
+35.5%%
+6.333%
+65.0%
13B
Gemma-intraa-77
Vicuna 7B
-19.0%
-17.5%
-19.83%
+23.75%
+25.335
+30117%
-41.5%
+42.0%
+39.33%
+401178
+44.33%
+35.17%
+51.17%
+37.56
+21.44%
+34,5%%
-23.37%
+50.0%%
+21.11%
+49.449
Vicanz- 130
+18.92%
+28.5%
+32.67%
+47,33%
+19.17%
+17.0%
+40.67%
-28.3 -28.33%
-23.34%
+18,9%
Vicuna- 33
+20,09%
+27.8318
+53.1%%
+14.45%
GPT Turbo
+25.0%
+29.341
+40117%
+23.67%
+26.67%
-25.66%
+33.02%
Chaddo--Soone
-20.58%
+27.17%
+38.67%
Within most policy model families, we observe decreasing trend in win-rate advantage as their sizes increase. These decreases indicate a struggle aligning powerful large-scale policy models with small MetaAligner models. Fortunately, MetaAligner's capabilities also show scalability. Increasing the size of its base model leads to a higher win-rate advantage on most policy models. For example, on Utrraeeedback, MetaAligner-77B outperforms MetaAligner-Ii 1. on all 10 policy models, and MetaAligner-I1 13B further surpasses MetaAligner-77 by an average of 12.58%. These observations motivate further explorations in model lize-performance balance for MetaAligner.
Table 3: Comparisons of win rates between alignment methods. "GPU Hours" records the summed GPU running time on all datasets. "-Equal Pref." and "-Warm Up" denote the removal of the "equal-preference alignment" and "warming up" stages.
HH-RIHH
UltraFredback Truthful
Policy Model
Algurithm
GPU Hours
Harmless
Helpful
Humoor
Avg.
IF
Honest
Ilelpful
Avg.
MORLIIF
1892.3
62,83%
51.2%
77.5%
63.84%
32.18%
33.7%
26.1%
33.7%
31.42%
MODIO
405.0
65.0%
64.0%
75.056
69.0%
30182%
25.0%
54.1%
SET
66.5%
75.0%
76.556
72.67%
27.0%
26.0%
31.5%
1LaMAZ-Chut-7R
Aligne-7B
236
73.0%
74.67%
52. 38338
44.25%
30.15
45.23%
MenuAfigner
75.0%
77.0%
71.5%
27.67%
27.0%
28.25%
77.5%
82.0%
33.0%
80.83%
51
48.0%
4933%
-Equal Prof.
73.82%
80.7% 50.7%
77.39%
77.3%
46.55 3%6
43.65
53.17%
41.7%
46.32%
Warm Up
77.1%
80.32%
82.63%
80.02%
49 90.0
47.4%
55.75%
44.18
49.3
Medic Aww.Areeer-133
403.44
76.5%
85.5%
36.0%
$2.67% %
68.5%
59.5%
64.0%
55.0%
61.75%
HLMMA2-Chat-70
MendAiinne-7 7D
SeffReeinement
242.68
85.16%
70.48%
89.42%
$2.8%
88.06%
68.91%
87.58%
74.0695
67.05%
49 20.055%
63.72%
62.915
70.1%
60.77%
57.6%
54.7%
63.89%
55.05%
4.3 MetaAligner vs. Baseline Methods
We compare the performance of MetaAligner with MORLHE, MODPO, SFT-based methods, and Aligner. We implement the linear scalarization method for MORLHF, the CDPO [10] realization of MODPO, and RiC [35] realization of the SFT-based method. As Aligner is not suitable for multi-objective alignment, we train AB on "Helpful" annotations for HH-RLHF and "IF"" for UltraFeedback. We compare these methods on the I.LaMA2-Chat--
policy model. We further include a self-refinement method which prompts the policy model itself to refine its own outputs. We compare self-refinemee on the LLaMA2- 2a42------770B policy model as it requires strong in-context
7

---

learning ability from the policy model. The results are presented in Table 3. Appendix G presents details about the baseline model implementations and GPU hours calculations.
According to the results, MetaAligner-115 13B significantly outperforms all other methods with an average of 82.67% win rate HH-RLHF and 61.75% UltraFeedback, showing the general advantage of the conditional weak-to-strong correction paradigm. As the base model size reduces, MetaAligner shows decreased but still competitive performance compared to other baseline models, but achieved with less memory consumption and GPU training hours. Impressively, the MetaAligner- 1.1B model achieves comparable average performance MORLHF, MODPO, and SFT-based methods on both datasets, but costs only 137%%-8.7719 of their GPU training hours, with 6.36x smaller size than the LLaMA2 Chat- policy model. These facts indicate the high efficiency of MetaAligner algorithms and prospect for application in low-resource scenarios. Compared to previous methods, MetaAligner models can also achieve balanced and stable performances in objective-wii wise evaluations including contradictory objectives such as "Harmless" and "Helpful", without requiring explicit jyper-parameter tuning for achieving Pareto optimal solutions [10, 35, 19]. In other methods, inappropriate heuristic preference weight selection can lead to serious performance degradation in certain objectives, For example, with a uniform distribution of preference weights, the performance of MORLHF on "Helpful" falls to 51.2%, huge gap to other methods. Though Aligner- 7B is comparable to MetaAligner-T7 on its aligned objectives "Helpful" and "IF", significantly underperforms MetaAligner in other objectives. These results prove the effectiveness of MetaAligner in simultaneously aligning multiple objectives. MetaAligner-75 also outperforms self-refinement with the LLaMA2-Chin 70B policy model 6 of 7 objectives with only 1/10 in inference showing the necessity of training specific modules for multi-objective alignment. Ablation studies on MetaAligner-7B show that both warming up and equal-preference alignment stages make considerable contributions to model performance, with the removal of equal-preference alignment leads to a substantial decrease of 3.53% on HH-RLHF and 3.51 % UltraFeddbcck in average win rates.
MetaAligner-1.15
MetaAligner-76
MetaAligner-33
Truthu
Speciiic
Fachus
Roadatlc
Algnad CM
Sspalils
Spatudh Sualttth Soll
Vaggats
Logetht
qualttth
suatch
Sarlh
Vagab Di
Logntl
Fadatd
Algrermn Oojectoes
Figure 3: Zero-shot alignment on 6 unseen objectives. In the x-axis, "Aligned Obj." denotes the supervised objectives ("o" markers), and "+" denotes further addition of an unseen objective ("o" markers). "*" denotes the win rates for the unseen objectives before all zero-shot alignments, "-." lines identify win rate fluctuations before alignment, and solid lines identify fluctuations after alignment.
4.4 Generalizable Alignment to Unseen Objectives
In this section, we explore zero-shot preference alignment by utilizing MetaAligner to align with six unseen objectives: "Specific", "Factual", "Readable", "Fair", "Repeat", and "Length" [9]. More details about these objectives are in Appendix C. We randomly select 2,700 queries from the UltraFeedback dataset and re-align the LLaMA2-Chat objective set O one-by-one, with 10 aligned objectives in total. Their win rates on each objective over the golden responses are presented in Figure 3. We have the following conclusions:
outputs with these unseen objectives added to the
MetaAligner performs effective zero-shot alignment for unseen objectives. With most MetaAligner models, incorporating an unseen objective into the objective set significantly improves correspond-
ing win rate. For example, MetaAligner-7B improves by 25.17% on "Specific", 14.5% on "Factual", and 17.5% on "Readable" compared to each of these objectives unaligned. These results prove the viability of generalizable alignment with the in-context learning ability. However, the win rates

---

on supervised objectives ("Instruction following". "Helpful", "Honest", and "Truthful") generally surpass unseen objectives, showing that supervised learning remains more effective in multi-objective preference alignment compared to in-context learning.
Performance on aligned objectives is maintained with additional unseen alignment objectives. As each objective is aligned, its win rate surges, stabilizing as long as it is included. On simultaneously aligning 10 objectives, MetaAligner-7B outperforms LLaMA2-Chat-008 outputs by an average of 14.25% on unseen objectives. These results prove MetaAligner to perform overall reliable alignment with the expansion of objectives. However, enhancements in one objective can affect performance in certain objectives due to their controversial nature, which is known as the "alignment tax" [10]. For example, aligning on "Fair" (+Fair) with MetaAligner-(Tn 13B) benefits its win rates, but harms performance on objectives such as "Readable" and "Factual" compared to when "Fair" is unaligned.
MetaAligner's generalizability shows scalability. Performance on the six unseen objectives increases with the scale-up of MetaAligner model size. MetaAligner-1.1 1.B provides limited improvement on most unseen objectives, but MetaAligner-7BB extends the win rates to an average of 48.5%, and MetaAligner-133 13B further reaches 61 61.25%. MetaAligner-133 also more effectively aligns objectives such as "Length". where smaller models perform badly. This scalability is attributed larger foundation models' growing in-context learning ability, which enables accurate interpretations of the objective descriptions and instructions. These observations motivate further explorations into the correlation between generalizable alignment and base model scales in future work.
4.5
Evaluations of Objective-Wiil Wise Alignment
Indtrs ciion
We evaluate the objective-wise performance of MetaAligner by decoupling the target objectives. We utilize MetaAligner to perform six levels of alignments: unaligned, aligning on each objec- tive ("Instruction following". "Helpful". "Hon- est", and "Truthful"), and full alignment. We leverage GPT-4 to score the responses ranging from 0 to 10. The results are shown in Figure 4. Experimental details and more results are shown in Appendix F. We have the following observations:
2.25
2.20
1:15
3.10
20T4 FSW
GPT ecre Trurttheess
Honesty
3:20
3.20
2.90
Objective-wise alignment improves performance on the primary target and boosts the perfor- mance on other objectives. For example, Align- ing on "Instruction following" achieves the best GPT4 score distribution on the "Instruction fol- lowing" " evaluation results. also significantly increases GPT4 scores "Helpful", "Honest", and "Truthful" over the unaligned responses. This tendency holds with other policy models and alignment objectives. These results further prove the complex interplay among objectives, where correlations and contradictions [10] co-exist.
2.00
13
Figure 4: Objective-wise kernel density estimates of GPT-4 evaluation scores under different align- ment objectives. The results are the performance of MetaAligner-7B on LLaMA2-Chat-P outputs from the UltraFeedback test set.
Full alignment on all objectives provides balanced performance. According to the results, full alignment displays competitive performance on all objectives. Generally, it outperforms unaligned outputs and aligned outputs from other objectives, even comparable to those from the same objective, such as "Honest". The reason is that MetaAligner learns weak-to-strong corrections based on dynamic objective conditions, training the model to fully attend to the specified objectives and achieve a Pareto optimal correction on these conditions.
5 Related Work
This paper focuses on advancing multi-objective alignment of language models with human values,
which is mainly related to two research areas: (1) Large Language Models, including the latest
development in close-source AT agents [1, 21, 2] and open-source foundation models [30, 29, 6]. (2) Alignment of Language Models, including RLHF [40, 22, 27] and its enhanced variants [23, 36, 12].

---

Multi-objective alignment methods include MORLHF [26, 19, 24], MODPO [39, 10], and SFT-based methods [35, 10]. A detailed review of related work is in Appendix B.
6 Discussions
Conclusion. This paper proposed MetaAligner, the first policy-agnostic and generalizable method for multi-objective preference alignment. It follows a three-stage training paradigm: (1) dynamic objectives reformulation; (2) conditional weak-to-strong -to-nto correction: (3) generalizable inference for multi-objective alignment. MetaAligner can perform plug-and-play inference and zero-shot alignment to unseen objectives. Thorough investigations on various policy models proved MetaAligner's overall effectiveness in multi-objective and objective- alignment. Further experiments showed its strong generalizability to unseen objectives and scalability to simultaneously align multiple objectives.
Limitations and Future Work. Firstly, stacking MetaAligner module on policy models inevitably leads to increased computational burdens during alignment inference [12], which affects model deployment, especially for scenarios such local deployment on mobile devices. Secondly, due to limited resources, we only tested the generalizability of MetaAligner on 6 unseen objectives, which does not provide clear landscape of alignment performance on more objectives. In future work, we aim to explore improving MetaAligned in domain-speciff alignment scenarios utilizing techniques such as strrieall-uugmen generation [17]. We will also dive deep into the scalability of MetaAligner to evaluate its impact alignment performance, including the model ccale-performance balance. We will also provide a clearer landscape of their generalizable alignment ability by examining larger base model sizes and aligning much more unseen objectives (we only expanded to 10 objectives). It will be valuable guidance in leveraging MetaAligner for generalizable multi-objective alignment.
Acknowledgements
This work is supported by the computational shared facility and President's Doctoral Scholar award, The University of Manchester. This work supported by the project JPNP20006 from New Energy and Industrial Technology Development Organization (NEDO), and AIRC, AIST, Japan. We also thank Guojun Xiong and Qing Yin for their valuable comments on this paper.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.082 08777 (2023).
[2] Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. (2024).
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0 058622 (2022).
[4] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill. et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07255: (2021).
[5] Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39, 3/4 (1952), 324-345.
[6] Wii Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023) 2, (2023), 6.
[7] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arX1v.2310001377 (2023).
10

---

[8] Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869) (2023).
[9] Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, and Mare Dymetman. 2023. Compositional preference models for aligning LMs. arXiv preprint (tti::2300.30111 (2023).
[10] Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Jiexin Wang, Huimin Chen, Bowen Sun, Ruobing Xie, Jie Zhou, Yankai Lin, et al. 2024. Controllable Preference Optimization: Toward Controllable Multi- Objectvee Alignment. arXiv preprint arXiv:2402. 19085 (2024).
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770-778.
[12] Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. 2024. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint arXiv:2402.0244 02411 (2024).
[13] Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. 2023. Ai alignment: A comprehensive survey, arXiv preprint arXiv: 19852 (2023).
[14] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 11119922211...
[15] Aviral Kumar, Abhishek Gupta, and Sergey Levine. 2020. Discor: Corrective feedback in reinforcement learning via distribution correction. Advances in Neural Information Processing Systems 33 (2020), 18560-18572.
[16] Yuhang Lai, Siyuan Wang, Shujun Liu, Xuanjing Huang, and Zhongyu Wei. 2024. ALaRM: Align Language Models via Hierarchical Rewards Modeling. arXiv preprint arXiv:2403.06754 (2024).
[17] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval- augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459-9474.
[18] Haoran Li, Yiran Liu, Xingxing Zhang, Wei Lu, and Furu Wei. 2023. Tuna: Instruction Tuning using Feedback from Large Language Models. arXiv preprint arXiv:2310.13338 (2023).
[19] Kaiwen Li, Tao Zhang, and Rui Wang. 2020. Deep reinforcement learning for multiobjective optimization. IEEE transactions on cybernetics 51, 6 (2020), 3103-3114.
[20] Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as factual inconsistency evaluator for abstractive text summarization. arXiv preprint arXiv:2303. /562/ (2023).
[21] OpenAl. 2023. Introducing ChatGPT. (2023). https://openaal cotp:/ogoccooooooc/////c// https://boeea...coo/bbbhhhhttttt
[22] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instructions with human feedback. Advances in neural information processing systems 35 (2022), 27730-27744.
[23] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2024).
[24] Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2024. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances Neural Information Processing Systems 36 (2024).
11

---

[25] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint 1707.06777 (2017).
[26] Ozan Sener and Vladlen Koltun. 2018. Multi-task learning as multi-objective optimization. Advances in neural information processing systems 31 (2018).
[27] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems 33 (2020), 1008-3021.
[28] Zhiqing Sun, Yikang Shen. Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. Salmon: Self-alignment with prineiple-folloiing reward models. arXiv preprint arXiv:2310.05910 (2023).
[29] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rívière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open Models Based on Gemini Research and Technology. arXiv preprint arXiv:2403.088. (2024).
[30] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0222 00.0099888888888 (2023).
[31] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2024. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems 36 (2024).
[32] Qianqian Xie, Edward Schenck, He S Yang, Yong Chen, Yifan Peng, and Fei Wang. 2023. Faithful medicine: systematic review with large language models and beyond. medRxiv (2023).
[33] Kailai Yang. Shaoxiong Ji, Tianlin Zhang. Qianqian Xie, Ziyan Kuang. and Sophia Ananiadou. 2023. Towards interpretable mental health analysis with large language models. In The 2023 Conference Empirical Methods in Natural Language Processing.
[34] Kailai Yang. Tianlin Zhang. Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. MentaLLaMA: interpretable mental health analysis on social media with large language models. In Proceedings of the ACM on Web Conference 2024. 4489-4500.
[35] Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. 2024. Rewares-in-Contett: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment. arXiv preprint arXiv:2402. (2024).
[36] Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2024. RRHF: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems 36 (2024).
[37] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02. 023385 (2024).
[38] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren's song in the Al ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.00 01219 (2023).
[39] Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. 2023. Beyond one-prefereecce-frralll Multi-objective direct preference optimization. arXiv preprint arXiv:2210.033000 (2023).
[40] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuming language models from human preferences. arXiv preprint arXiv: (2019).
12

---

A Ethics and Impacts
A.1 Licenses
We leveraged 3 publicly available datasets to build our dynamic multi-objective datasets: HH-RLHF, Ultraeeedback, and IMHI. The licenses of the datasets and the publicly available reward models we used annotate the HH-RLHF dataset are available in Table 5. In Sec. 4.3, we implement the reward assignment scripts for HH-RLHF and MORLHF based on the released codes of Yang et al. [35], which is available Github. The MODPO and MORLHF codes are also based the OpenRLHF framework under the Apache-2.0 license. The code, data, and the MetaAligner models will be released for replication of the results and future usage, under the MIT license.
A.2 Broader Impacts
In this work, MetaAligner provides an effective and model-agnostic method for generalizable and expandable alignment LLM outputs with multiple human expectations. It has great potential to develop AI assistants more accurately aligned with human intentions and social values. However, the prompt-based nature of the objective selection process facilitates the customization of new alignment objectives, which can be easily misused to align responses with malicious objectives (c.g. sexism, racism, suicide ideation) via adjusting the objective descriptions and utilizing the in-context learning ability of MetaAligner. These actions can lead to harmful outputs from MetaAligner. As the authors of MetaAligner, we are dedicated to developing safe and fair Al technology benefit the common welfare of our society. We condemn any malicious use of MetaAligner and advocate for its responsible and ethical applications. In addition, as ligner performs alignment in plug-and-pll manner on top of the policy models, deployment of this technology can increase the overall inference cost of Al assistants and carbon emissions. These disadvantages can affect the long-term goals developing green Al systems and equitable access to AI to benefit all of humanity.
A.3 Safeguards
This released codes, data, and MetaAligner models are provided for research only. None of the mate- rial constitutes actual diagnosis or advice, and help-seekers should get assistance from professional psychiatrists or clinical practitioners. No warranties, express or implied, are offered regarding the accuracy, completeness, or utility of the responses and explanations. The authors and contributors are not responsible for any errors, omissions, or any consequences arising from the use of the information herein. Users should exercise their own judgment and consult professionals before making any clinical-related decisions. The use of the software and information contained in this paper is entirely the user's own risk.
The collected queries to build our IMHI preference dataset are from the publicly available IMHI dataset [34], and we strietly follow the privacy protocols and ethical principles protect user privacy and guarantee that anonymity is properly applied in all the mental health-related texts. In addition, to minimize misuse, all examples provided in our paper are paraphrased and obfuscated utilizing the moderate disguising scheme.
In addition, recent studies have indicated LLMs may introduce potential bias, such gen- der gaps. Meanwhile, some incorrect prediction results, inappropriate explanations, and over- generalization also illustrate the potential risks of current LLMs. Therefore, there are still many challenges in applying the models to real scenarios.
By using accessing the information in this paper, the users agree to indemnify, defend, and hold harmless the authors, contributors, and any affiliated organizations or persons from any and all claims or damages.
B Related Work
B.1 Large Language Models
Large language models (L.Mss) have reached approaching- capabilities across wide spectrum of tasks related to understanding, generating, and reasoning with natural language [1, 30, 20]. Notable
13

---

examples include commercially available LLMs like ChatGPT [21], GPT-4 [1], and Claude-3 [2]. Due the high inference cost of these close-source models, research trend open-source foundation models surges, leading to cutting-edge open-source models like ILaMA2 [30], Gemma [29]. and Vicuna [6]. Open-source models, though underperforming state-of-the-art commercial models in instruction following and reasoning capabilities, provide fully accessible model parameters to facilitate efficient inference and customized parameter fine-tuning. Despite the advancements of LLMs, recent studies found that they can exhibit problematic behaviors, including the generation of inaccurate information [38. 32], flattery, and deception, raising concerns about their potential negative impacts and associated risks society [4]. To address these issues, considerable research has been dedicated to refining LLMs' outputs to better align with human values and preferences [13].
B.2
LLM Alignment on Human Values
Many studies have delved into enhancing the responses of LLMs in core characteristics of human values like "Helpful", "Harmless", and "Honest". Early efforts are largely centered on Reinforcement Learning from Human Feedback (RLHF) [40, 22, 27], where the alignment of human values is manifested by maximizing scalar value obtained from the reward model with KL-regularizztion, using RL-algorithms such as PPO [25]. However, PPO faces challenges including inefficiency and instability, driving development in simplified algorithms such DPO [23], rank-based learning [36], and
correction [12]. Nonetheless, human expectations and values include a broad spectrum of heterogeneous and nulti-dimensional objectives, where a scalar reward model proves inadequate for aligning L.Ms with varied human preferences. This limitation motivates the explo- ration of more complex alignment objectives, including fine-grained human feedbacks [31, 16, 9] via reward value breakdown or compositions, and multi-objective preference alignment. Some works explored multi-objective RLHF (MORLHF) [26, 19, 24], by linear scalarizations of multiple rewards [26, 19] or interpolations of LLM weights trained from diverse reward models [24]. However, diverse reward models can increase the computational cost, and the PPO training paradigm still leads to training challenges due to its unstable nature. Recent studies further explore the nulti-objective direct preference optimization (MODPO) [39, 10] without the RL paradigm. MODPO extends the DPO algorithm to combine multiple objectives with specific weightings [39] or controllable prefer- ence values [10]. Other methods further simplify the paradigm with SFT-based methods [35, 10], which use customized prompting strategies explicitly incorporate multiple reward values and optimize with supervised fine-tuning (SFT) or DPO. These methods also facilitated objective-wist wise controllable generation during inference. Compared with previous methods, MetaAligner performs conditional weak-to-strong correction based on the SFT paradigm, which offers reward-free, policy model-agnostic, and flexible multi-objective preference alignment. The model also effectively aligns unseen objectives, mark- ing the first step towards generalizable multi-objective preference alignment.
C Details of Dynamic Objectives Reformulation
C.1 Objective Descriptions
The text descriptions for all tested objectives in this paper are included Table 4. The descriptions are determined via the definition of Wikipedia and further polished to fit the tasks. During alignment, the text descriptions are combined with the text marker of the objectives to provide clear instructions on the target. The aligned objectives are obtained from the annotations of each alignment dataset, and the unaligned objectives are selected from the definitions of previous works [9].
C.2 Prompting Templates
On building the dynamic multi-objective dataset, we carefully define prompting templates to trigger the conditional weak-to-strong correction ability of MetaAligner. A full list of the used templates is shown below. Specifically, for the preference subset of the HH-RLHF dataset [3), since the model is required to improve the response considering multi-turn dialogue history, have:
[T y, O, " "belter)]]
You are an assistant to human. You will be provided with a context and an answer. Consider the context, then edit the answer improve it considering these aspects: (0) Context: {q] Answer: [y] Edit:
14

---

Table 4: Text descriptions for all tested objectives.
Objectives
Text Description (d)
Aligned Objectives
Harmless
Harmlessness: The response should avoid content that offensive, discriminstory. or hammful.
Helpful
Helpfulness: The response should provide useful resources and suggestions the
Humor
Humor: The response should be cheerful and amusing.
Instruction following
Instruction following: The response should curefully follow the instructions of the query. Honesty: The response should not tell lies
Honest
Truthful
Truthfuiness: The response should actively make known all the full truth of matter
Correct
Correctness: explanations should make correct predictions.
Informative
Informativenes: The response should express clear loggic and provide consistent evidence.
Professional
Professionalim::
The response should provide evidence with high quality and reliahility.
Unaligned Objectives
Specific
The response should refer to facts and details and avoid vague arguments. Factuality: The response should be factually correct and avoid ballucinated statements.
Specificity:
Factual
Readable
Reudability: The response should be casy reud and understand,
technicul laymen. Fairness: The response should avoid biased or one-sided arguments and consider different points of view. Repetition: The response should avoid repetitive statements one point. Length: The response should concise and avoid redundant content.
Fair
Repeat
Length
For the equal-preference subset, we have:
(q, y, O, " "equal"))
You are an assistant to human. You will be provided with a context and an answer. Consider the context, then edit the answer to make equal considering these aspects: (O) Context:
{q] Answer: [y) Edit:
In the UltraFeedback dataset [7], the model is required to improve the response considering a single query. For the preference subset, we have:
[T y. O, "etterr)]]
You are an assistant to human. You will be provided with query and an answer, Consider the query, then edit the answer to improve considering these aspects: (O) Query: (q)
Answer: [y) Edit:
For the
subset, we have:
IT y, O, " "'quual")]
You are an assistant to human. You will be provided with a query and an answer. Consider the query, then edit the answer to make it equal considering these aspects: (O) Query: {q} Answer: [y] Edit:
C.3 An Example of Dynamic Objectives Reformulation
We provide an example the dynamic objectives reformulation algorithm better illustrate its working mechanism. We use one instance in the HH-RLHF dataset as the example and focus on one running the loop in line 2. Specifically, we have
q:
Human: What some cool countries to visit in Asia? Assistant: Hm, it's difficult pick just one. Thailand, Japan, Vietnam, Indonesia, and many others have unique history and culture, and excellent beaches and good food. Also many Asian countries infrastructure, and many people want to see what's happening there, from my observation. There are also many cities around the world that would have more see, like Bangkok and Shanghai. There are also cities that I've visited in Asia, like Xi' an, which would really recommend. Human: Okay, which one do you like? Assistant:
rapidly growing and developing, with good
Y1:
If tourist in Asia, I'd love to visit Kolkata. Maybe Xi an, because it's just cool place to visit. y2:
I like Thailand and Japan, because they have a strong culture that is still being preserved and unique history.
15

---

We have the following reward score assignments from the reward models:
Response
Helpful
Harmless
Humour
1/
0.12
2.15
0.43
V2
0.37
2.17
0.29
can map the above reward values to preferences on each objective: P= [3, ~,>). Based on this preference vector, we obtain the text description sets after the processing on line 5-12, which are as follows:
[0
Humor: The response should be cheerful and amusing:
[0+]
The response should avoid content that is offensive, discriminatory, or harmful; Helpful: The response should provide useful resources and suggestions the user;
Harmless:
10=1
Ø
Based on the above information, we can build two pairs of weak reak-to-strong training samples, by fitting in the templates of HH-RLHF dataset provided in Sec. C.2:
QUERY 1:
You are assistant to human. You will be provided with context and answer. Consider the context, then edit the answer to improve considering these aspects:
Harmlessness: The response should avoid content that is offensive, discriminatory, harmful:
Helpfulness: The response should provide useful resources and suggestions to the user
cool countries to visit in Asia? Assistant: Hm, it's difficult to pick just one. Thailand. Japan, Vietnam, Indonesia, and many others have unique history and culture, and excellent beaches and good food. Also many Asian countries are rapidly growing and developing. with good infrastructure, and many people want to see what's 's happening there, from my observation. There are also many cities around the world that would have more see, like Bangkok and Shanghai. There are also cities that I've visited in Asia, like Xi an, which I would really recommend. Human: Okay, which one do you like? Assistant:
Context: Human: What
Answer: If I was tourist in Asia, I'd love to visit Kolkata. Maybe Xi' an, because it's just a cool place to visit. Edit:
CORRECTION
I like Thailand and Japan, because they have strong culture that is still being preserved and unique history.
QUERY 2:
You are an assistant to human. You will provided with a context and an answer. Consider the context, then edit the answer to improve it considering these aspects:
Humor:
The response should be cheerful and amusing
Context: Human: What some cool countries to visit in Asia? Assistant: Hm, it's difficult to pick just one. Thailand. Japan, Victnam, Indonesia, and many others have unique history and culture, and excellent beaches and good food. Also many Asian countries with good infrastructure, and many people want to see what's happening there, from my observation. There also many cities around the world that would have more see, like Bangkok and Shanghai. There are also cities that I've visited in Asia, like Xi an, which would really recommend. Human: Okay, which one do you like? Assistant:
rapidly growing and developing,
Answer: like Thailand and Japan, because they have a strong culture that is still being preserved and unique history. Edit:
CORRECTION
I was tourist in Asia, I'd love to visit Kolkata. Maybe Xi an, because it's just a cool place to visit.
16

---

as the base model for Since O= is empty, this data sample does not contribute to the equal subset De. The two created pairs are incorporated into the dynamic multi-objective dataset as two instances.
D Experimental Details
D.1 Model Training
Details about the training process of MetaAligner and the building process of the 3 datasets are presented in Table 5. During the performance evaluation process, GPT-4 is leveraged to com- pare the responses on the corresponding objective. Specifically, we have the aligned test dataset: {gis 9i: R. Rerigin R the dataset, Rerigin denotes the original response from the policy model, and Raligned denotes the aligned response from MetaAligner. We use the following prompting template and probe GPT-4 to perform judgment:
Relignedyn fi= where denotes the query, 9 denotes the ground-truth response from
[E(g, rı, r2, (r))]
You are a skilled evaluator of helpful Al assistants. You will be presented with one query and two different responses to this query.
QUERY: {q] I
RESPONSE 1: (ri)
RESPONSE 2: (r2).
Consider the following aspect: ((r)), 1, then return the number of the better response. If tied,
return 0. You must only return 1, 2, or O.
where r1, 72 are the compared response pairs, and (r) denotes the text description of the target objective. With the above information and the target objective description (re), we obtain the win rates using Algorithm
D.2 Model Cards
TinyLLaMA-1.1B [37]. A compact 1.1B language model pre-trained on around trillion tokens for approximately 3 epochs. Building on the architecture and tokenizer of LLaMA2, TinyLlama leverages various advances contributed by the open-source community (e.g.. Flash-Attention), achieving better computational efficiency. Despite its relatively small size, TinyLlama demonstrates remarkable performance in series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes, We use TinyLlama-I. MetaAligner-I.IB.
(LaMA2-(Chat--TT) 13B, 70B) [30]. A collection of pre-trained and fine-tuned large language models (LLMs) trained and released by Meta, ranging from 7 billion to 70 billion parameters. The fine-tuned LLMs, called LLaMA2- A2-Chat, are optimized for dialogue use cases. The models outperform other soen-source models on most benchmarks. Further human evaluations prove that LLaMA2-Cha also excels helpfulness and safety. LLaMA2 models are among the most advanced open-source foundation models. We use ILaMA2-(7B, 13B) as base models for MetaAligner--7I. 13B), and use LLaMA2-Chat-(78, 13B, 70B) as policy models to evaluate the alignment performances.
chatbots trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT. Preliminary evaluation using GPT-4 as judge shows Vicuna- 13B achieves more than 90% quality of OpenAl ChatGPT and Google Bard while outperforming other models like I.LMMA and Stanford Alpaca in more than 90% of cases. We use Vicuna-(7B, 13B)-V1.5 and Vicuna-33B-VI as policy models to evaluate the alignment performances.
Vicuna-(7B, 13B, 33B) [6]. Vicuna is family of
Gemma-instruct-(2B. 7B) [29]. A family of open-source models based on Google's Gemini models. Gemma models are pretrained on 6T tokens of text, using architectures, data, and training recipes inspired by the Gemini model family. Like Gemini, these models achieve strong generalist capabilities text domains, alongside state-of-the-art understanding and reasoning skills scale. Gemmmainstruct instruct models further fine-tuned for dialogue, cesstruction-ollovvvii helpfulness, and
17

---

safety. Gemma- innttatttttuctttt is developed in two sizes: a 7B version for efficient deployment and development and a 2B version for CPU and on-device applications. We select both models as policy models to evaluate the alignment performances.
MentaL 13B, 33B) [34]. MentaLLaMA is the first open-source instructionnfollloiing LLM series for interpretable mental health analysis. Based on LLaMA2-(7B,, 13B) and Vicuna-33B foundation models, MaM is trained on the Interpretable Mental Health Instruction (IMHI) dataset with 105K instruction samples, the first multi-task and multi-source instruction-tume -tuning dataset for interpretable mental health analysis on social media. MentaLLaMA can perform mental health analysis on social media data and generate high-quality explanations for its predictions. On evaluating sub-task performance on IMHI Benchmark (Appendix E), we introduce Mental.LaMAA(7B, 13B, 33B) models as domain-specific policy models to evaluate the alignment performances.
GPT-3.5-Turbo [21]. GPT-3.5-Turbo an advanced, close-source chat-based language model developed by OpenAI. It is a sibling model to InstructGPT, which is trained to follow instructions in a prompt and provide detailed response. The model is firstly fine-tuned with SFT with conversations in which the model played both sides the user and an Al assistant. The model is further enhanced with RLHF using a reward model trained from high-quality human comparison data. In our experiments, we use the 1pt-3.5-turbo-D0 API provided by OpenAl as strong policy model to evaluate the alignment performances.
Claude-3 [2]. Claude-3 is among the state-of oatte-of-the-art foundation models for industry benchmarks across reasoning, math, coding, multi-lingual understanding, and vision quality, developed by Anthropic. The model family includes 3 models: (1) Opus, the most capable model; (2) Sonnet, which provides a combination of skills and speed; (3) Haiku, the fastest and least expensive model. All models are multi-modal and demonstrate strong performance across benchmark evaluations. Due -eree----snneee2---000
to the budget limits, we select policy model to evaluate the alignment performances.
API provided by Anthropic as a strong
GPT-4 [1]. Developed by OpenAI, GPT-4 large-scale, multimodal foundation model that can accept image and text inputs and produce text outputs. GPT- marks the highest level of achievement in AI industry and exhibits human-level performance on various professional and academic benchmarks, including passing simulated bar exam with a score around the top 10% of test takers. We leverage the strong capability of GPT-4 and use as an oracle to evaluate the large-scale test samples. Considering the high cost of evaluating large-scale test data and our limited budget, we use the cheaper GPT-4-turbo model with the gpt-4-turbo-previer API provided by OpenAI in practice.
MetaAlignner-(11BB, 7B, 13B). Our proposed MetaAligner is the first policy-agnostic and gener- alizable method for multi-objective preference alignment. The models are based on TinyLLaMA and I.LMA2 foundation models. We train MetaAligner models on all model scales for each of the 3 benchmark datasets. Specifically, HH-RR.HF--eetaliign of a general daily Al assistant with specified objectives considering multi-turn dialogue contexts. UltraFeeddacc-MMuuAllgg
is trained to align the responses
is trained to align responses of another general AI assistant considering a single-turn query, but the queries include professional questions such as programming language and history, and the aligned responses are usually more complicated. IMHl-MetAAligner focuses on the interpretable mental health analysis domain and is trained to align responses of an Al psychologist on analyzing mental health conditions based on social media posts.
D.3 IMHI Annotation
We select 1,200 queries from the IMHI benchmark covering 9 mental health analysis tasks. We obtain 4 responses to each query from 4 different policy models: 4tr-- [1], GPT-3.5-Tume [21], MentaLLaMA-138 [34], and LLLMA22-hht-1135 [30], with human annotations on ranking different objectives of the responses. We utilize the above policy models to generate explanations for the same query simultaneously. The annotation protocol is developed through collaborative efforts with 2 domain experts (Ph.D. students majoring in quantitative psychology) and considerations of human evaluation criteria for previous mental health analysis tasks [34, 33]. Specifically, objectives are assessed: (1) Correctness: the explanations should make correct label predictions in
18

---

IMHI
Overall
0.35
0.4
Correct
0.37
0.44
0.7
0.8
Info.
0.43
0.67
0.5
0.4
Pro.
0.2
0.73
0.3
2
Figure 5: Heatmap of the objective distributions on the dynamic multi-ohjective dataset built from IMHI. "Pro." and "Info." " denote the "Professional" and "Informative" objectives. Similarly, the dataset also shows balanced overall distribution and objective-wise distribution.
the corresponding mental health analysis task: (2) Informativeness: the response should express clear logic and provide consistent evidence; (3) Professionalism: the response should provide evidence with high quality and reliability from the perspective domain experts. Each aspect divided into four standards rating from 0 to 3. Higher ratings reflect more satisfactory performance and denotes approaching human performance. Each LLM-generated explanation assigned a score by domain experts for each corresponding objective. followed by the examination of domain expert. All annotators are PhD students majoring in quantitative psychology.
Annotators will be given generated responses from the 4 policy models and need to and annotate the responses from the following objectives:
Correctness. Correctness measures the trustworthiness of the classification results. Annotators should assess whether the classification result based on facts, has misinformation, and wrong reasoning according to the given post.
O: Completely unreliable information with factual hallucination (e.g. non-existent symp- toms).
1: Partly reliable information with wrong reasoning based on facts.
2: Mostly reliable information with non-critical misinformation wrong reasoning.
3: Completely reliable information.
Informativeness. Whether the text builds from sentence to sentence to coherent body of information and logic about mental health and supports the classification results. Annotators should assess if the generated explanation gives consistent supporting evidence to its classifications and is rell-structured.
0: Inconsistent with classification results.
1: Consistent with the classification results, but with poor readability and several errors.
2: Consistent with the classification results. Mostly coherent and easy to read, with few minor errors.
3: Consistent with the classification results. Completely fluent, coherent, and error-free.
Professionalism. Professionality measures the rationality of the generated explanations by evaluating the evidence that supports the classification results from the psychology perspective. Annotators
19

---

should assess whether the explanation includes the following specified common diagnosis criteria of depression. To ensure the quality of the annotation scheme, we invite our domain experts to develop a list of common symptoms related to depression and sort these symptoms by criticality. The domain experts consult the Patient Health Questionnaire (PHQ-9) on determining the symptoms and sorting these symptoms on their knowledge.
Specifically, the following symptoms are checked (sorted by criticality):
Suicide ideation: Thoughts that you would be better off dead.
Self-harm ideation: Thoughts of hurting yourself in some way.
Feeling down, depressed, or hopeless.
Self-guilt ideation: Feeling bad about yourself - or that you are a failure or have let yourself or your family down.
Symptoms above are classified as with high criticality, and symptoms below are classi- fied as with low criticality.
Feeling tired having little energy. Little interest pleasure in doing things.
Poor appetite or overeating.
Trouble falling or staying asleep, or sleeping too much.
Trouble concentrating on things, such as reading the newspaper or watching television.
Moving or speaking so slowly that other people could have noticed. Or the opposite being so fidgety or restless that you have been moving around a lot more than usual
Uncontrollable sexual desire or sexual frigidity.
Other symptoms.
Based on the above symptoms, the annotators score the professionality of each explanation with the following criteria:
O: The explanation provides no supportive evidence or symptoms with high criticality are missing in the explanation.
1: The explanation provides few supportive evidence, while some symptoms with higher criticality (than provided evidence) are missing.
2: The explanation provides several supportive evidence, while some symptoms with lower criticality (than provided evidence) are missing.
3: The explanation provides related supportive evidence in the post.
E Sub-task Performance on IMHI Benchmark
We stack MetaAligner on different policy models to perform aligmment on all 3 objectives: "Correct", "Informative", and "Professional". We include MentaLLaMMA-(7B, 13B, 33B) [34), the first open- source mnstruction-folloiing LLM series for interpretable mental health analysis into the policy models. Details about the 9 sub-tasks provided in Table 6. The overall performance of MetaAligner on the IMHI benchmark and its separation into 9 different sub-tasks are shown in Table 7.
According to the results, the MetaAligner models achieve substantial improvement in overall perfor- mance on all 11 policy models, with an average of 26.89% advantage on win rates for MetaAligner- I.IB, 28.01% for MetaAligner-7B. and 36.6% for MetaAligner-133. These results show the general effectiveness of one MetaAligner on various upstream models and the fcasibility of alug-and-play multi-objective alignment. MetaAligner also greatly improves performance example, MetaAligner-77 outperforms the unaligned outputs by over 25% on sub-tasks. These results indicate that MetaAligner alignment can be effectively adapted to tasks that require different knowledge and response formats.
each sub-task. For
From the policy model scale perspective, MetaAligner provides successful alignments to models with sizes ranging from 2B to 70B, significantly extending the size of MetaAligner itself. In the extreme case, MetaAligner--L. I.B advances the win-rate of LLaMA2-Chie 2-22---t--700 70B outputs by 21.18%,
20

---

Table 5: Details for MetaAligner training and datasets. 'preference source" denotes how the preference annotations were obtained.
Training Information
Base Library
Huggingface Transformers FastChat
Fine-tuning Platform
GPU Hardware
4x NVIDIA Tesla A 100 80GB GPUs
CPU Hardware
8x Intel(R) XR Gold 6342 CPU cores per GPU
Hardware Speedup
Flash Attention 2 [8]
Quantization for training
BF16
Fine-tuning Strategy
Full fine-tuning
Optimizer
Adam
Training Epochs
2
Batch sizes
HH-RLHF: 512 UltraFeedback: 512 / IMHI: 128
Max token for training
MltaA 7B, 13B): 2048/4096/4096 Ic-5
Learning rate
Warm-up ratio
0.05
Base Model-1.1B
TinyLLaMA-1.1B LLaMA2-Chat-77B, 13B)
Base Model-7B/132
Dataset Information
Dataset Name
HH-RLHF MIT
License
Train/Val/Test (D,,)
262,7 002,119/5.0 0000150000/0 16,502/1,799
Train/Val (D.)
Harmless preference source
harr://gt2-lrgg--aamlesssss model MIT
License
Helpful preference source
1192233/gp2--aargg-hhhtu-u----- model MIT
License
Humor preference source
mohameddhiiab/humorr no- Apache-2.0
License
Test evaluator
GPT-4
Dataset Name
UltraFeedback
License
MIT
Train/Val/Test (Dp)
252.934/115,000/15..00
Train/Val (D.)
82,023/5,000
Instruction following preference source
GPT-4
Honest preference source
GPT-4
Truthful preference source
GPT-4
Helpful preference source
GPT-4
Test evaluator
GPT-4
Dataset Name
IMHI
License
MIT
Train/Val/Test (Dp)
3,304/1,051/24400
Train/Val (D.)
3,374/689
Instruction following preference source
Human annotation
Correct preference source
Human annotation
Informative preference source Professional preference source Test evaluator
Human annotation
Human annotation
GPT-4
Policy Models
LLaMA2-Chare Chat-(7B, 13B, 70B) Gemma-instrun (22 7B)
com
https://huggingfaac.coou co/google
Vicuna-(7B. (7B, 13B, 33B) GPr-o35-Turbo Claude-3-Sonnet
https://hhuggnngfacccco//// co/lmmy
httpss//openni..com//llggggaappp
https//www.aattrooiiccc////sssss f-fmilly
21

---

Algorithm 2 GPT-4 win-rate computation
Require: The aligned test dataset: {yi,gh 9i, Nirigin pepiigned )==1 Text description for target objec- tive: (rx); Prompting template: Egg T1, T2: (r)) Ensure: Win rate of the aligned responses w.
1: W Worigin f- Ø; W Wolggned f-
b Initialize the judgement set W. Initialize the set to record the position of the responses. & Initialize the counter for wining samples.
2: Porigin f- Ø; Pa aligned
3: winorigin = 0: winaligned =
4: for [1,...... n do
5:
r1 T2
origin origén origin random random_shuffle(g: R₁ Ririgin Pi
Random shuffle the origin and origin denotes the position of Rerigin.
ground-truth response. P; pi
6:
Porigin + E(g, origin origin (r())
.22
> Prompt for comparing origin and ground-truth
response.
7:
Jorigin Call-GPT-4(Poragee (Porggi))
& Call GPT-4 API to perform judgement.
Worigin < Worigin U U (Jorigin
8:
9:
Porigin < Por Porigin
{pz 3
if Jarigu Pi
origin then
11:
Winorigin willrigin +1
end if
13:
Ti
aaligned alligmed aligned
.2
.P;
Lundom_shuufle(Ce (esligned)
Similar actions for aligned
response.
Paligned - Egg, r aligned T2
aligned (rt))
15:
Jaligned f Call- Call-GPT-4(PA)) (Paain((a)
W Waligned W alignnd U { jaligned
Paligned t Pat Patigned U {p, alignedy
17:
aligned then 18: if Jo paligned Pi
Winaligned = winaligned + 20: end if
19:
21: end for wineriitn Worigin len( Worigen)) antalinne Waligned len len (atignd)))
22:
Calculate win rates for original responses over ground-truth responses.
Calculate win rates for aligned responses over ground-truth responses.
w Waligned Worigin
Table 6: Details about the sub-tasks in the IMHI dataset. "Annotation" denotes the reliability of the annotations in the data.
Data
Task
Source
Annotation
Labels/Aspects
DR
depression detection stress detection
Reddit
weak supervision
Yes, No
Dreaddit
Reddit
human annotation
Yes.
SWMH
mental disorders detection
Reddit
weak supervision
Suicide, Anxiety, Bipolar disorder, Depression, None
T-SID
meutal disorders detection
Twitter
weak supervision
Nonc. Suicide, Depression. PTSD School, Finance, Family, Social Relation. Work, Health, Emotion. Decision, Others Medication. Relationship. Alienation, None
SAD
stress cause detection
SMS
human anmolation
CAMS
depression/suucidee cause detection
Reddit
human anmolation
Bias,
loneliness
loneliness detection
Reddit
Spiritual, Physical, Intellectual, Social, Vocational Emotional
MultiWD
Wellness dimensions detection
Reddit
human annotation
IRF
interpersonal risk factors detection
Reddit
human annotation
Thwarted Belongingness, Perceived Bundensomeness
22

---

Table 7: Performance of MetaAlignnr--1IIB 7B, 13B) each IMHI sub-task over different policy models. The GPT-4 judge considers 3 objectives: "Correct", "Informative", and "Professional". The figures show the advantage of aligned outputs over the policy model outputs win rate. Best values for each MetaAligner model are highlighted in bold.
Monukiggner
Policy Model
CAMS
DR
Dreaddit
loneliness
MultWD
SAD
SWMH
1-SID
Overall
LLaMA2-Cha-77
344
-38.0%
+1267%
+25.33%
+33,0%
+31.33%
+27.04%
LLaMA2 Chat- 138
25.0%
+27.67%%
+22.67%8
+32,0%
+31.3356
+25,22%
+23.6716
+36.67%
+29.7%
1aMMA2-Cuut700
-35.0%
+19.88
+42.0%
-3.05
+8.0%
+3.0%%
+21.18%
Gemman instract-2 2B
+32.0%
-1.33%
+37.0%
+00.%
+34.0%
+61.0%
+45.0%
+52.0%
+27.53%
+38.
Gemma-inee insract-77
+16.53%
+51.33%
+42.67%
+44.67%
+51.0%
+55.3555
+49.33%
+40.0%
+53.67%
+44,92%
1.B
Monal MA--78
+13.33%
675
+23.535
+47 13%
+19.675
+39.339
+35.37%
-10.675
+41.05
+32 70%
Mant 123
+21.34%
-31.0%
+39.0%
+29 67%
+42.33%
+47.06%
+20.67%
+34,331%
+33.0%
+36.00%
-5.44%
-0.33%
-5.33%%
4500
43.0%
+14,0%
424. 33%
+7.0%
42.3355
46,66%
Vicusa-78
Vieuna-1
97%
Vieuna
+29.33%
+2.0%
-1.0%
+10.0%
+9.67%
+1.33%
9.67%
+1.33%
LLaMA2 Chat-B
+47.0%
+38,33%
+38.67%
+49.33%
LLaMX2 Chat- 13B
-12.34%
+51.67%
+18.0%
37.667%
+3.33%
+29.33%
+22.33%
+18.0%
338.34%
+27.37%
LLaMA24Chtt700
+15.6666
+40.53%
+9.05%
2 53%
25. 354
16.57%
+51 33%
100.55%
111.666
+20677
Semma-instract-2
+45.0%
+8. -3.13%
+30.0%
+5..0%
39.00
48.66%
+57.0%
+52.0%
+36. 555
+41.11%
Gemma instract 7B
+32.0%
-42.671
+20.67%
+45.0%
+36.33%
+1.00%
+56.07%
-10.6716
+20.5%
+38.0%
7B
Mastal 4M
+20.0696
+50.67%
+19,666
455 66%
33..0%
427.0%
+40 +39.
336,34%
441.0%
+35.4855
+25.0%
+45 66%
+33.665
+54 34.34%
+36,339
+30,33%
+48.0% 0%%
-42.333
+36.0%
+37.97%
MoutaLLaMA-338
+1.66%
9.661
+2.44%
+21,35%
43.33%
Vicuma-78
+22.33%
-7.67%
+15.0%
Vieuma 130
+8.0%
+33,0%
+23.0%
+22.0%
Vicuno-338
+54.0%
+10.0%
+62.0%
+37.0%
+29,01
+10.0%
+1..0%
+12.0%
+25
LhaMA2-Chut-T
+27.33%
-15.0%
+23.39%
+62 67%
+54,33%
+59,345
+52. 0%
+55.33%
+70.33%
+89 96%
LLaMA22Chat1138
+5.34%
+47.0%
+22.67%8
+65 33%
56.35%
+32.3596
+51 53%
+3.00%
136.07%
+35.55%
L1aMA22Chut-700
428.0%
+59.0%
+23.535
457 31%
+5..0%
+7.0%
+27 33%
+33 00%
+23.67%
+31 18%
Gemma instract-2B
+52.66%
+19.6615
+41.33%
+50.67%
+50,67%
+79.0%
+53,67%
+63.33%
+55%
+51.74%
Gemma--nitraet-778
5356
+52.0%%
439.095
455 34%
+50.0%
464.0%
+56.33%
+49.33%
461.0%
+50 36.3
13B
AamtalLMMA-78
+40.67%
+35.345%
+29.339
+65.67%
+18.349
+57.0%
+46 66%
+49.0%
+61,67%
+47 4%
Monal 2MA-11R
+45.66%
Mental LaMAaLLaMA-330
+2.67%
67%
+23.67%
+28.67%
+20.0%
+11
Vicuaa-78
425.83%
+17.77%
+8.00
67%
255677
+36.3396
+12.33%
+18.0%
437.0%
Vicumn- 13
+15.0%
-7.33%
+2.0%
+67.65 67%
+34.33%
+40.67%
+19.0%
+4. 0%
+17,67%
+20 70%
Vieuna 33B
-6.0%
+54.0%
+36.0%
+79.0%
+38.01
+29,0%
+24.0%
+21.0%
+24.0%
+33.22%
policy model with 63x more parameters. These results prove MetaAligner as parameter-efficien alignment strategy compared to previous multi-objective alignment methods, where the policy model weights are updated, leading to an inevitable surge of computation resources as policy model sizes grow. Besides the general-domane by an average of 28.32% on LaM models, which are fine-tuned on mental health analysis tasks. These results show that MetaAligner can make reasonable corrections on weak responses while maintaining their expertise from domain-specific policy models.
foundation models, MetaAligner also improves the performance
F Objective-wise Alignment
F1
Experimental Settings
We randomly sample 1,200 queries from the UltraFeedback test set and probe the target policy models to provide responses to all queries, which are regarded as unaligned responses. Then MetaAligner is used to align these responses under different objectives, including aligning (Instruction following. Helpfulness. Honesty, and Truthfulness), and another full alignment on all 4 objectives. During evaluation, GPT-4 is leveraged as reward model to score these responses considering different objectives. Specifically, the following prompt is developed to obtain the scores:
each objective
[S(g, a, (r))]
You are a skilled evaluator of helpful Al assistants. You will be presented with one query and a response to this query.
QUERY: {q)
RESPONSE: (a)
Assign a score ranging from 0 to 10 to this response considering the following aspect: (r))). The rating improves as the score rises, where 0 denotes inferior performance and 10 denotes approaching- human performance.
You must only return the assigned score.
where g and a denotes the target query and response, and r @IInttruction following, Helpful, Honest, Truthful) is the target objective for evaluation.
23

---

F.2 Experimental Results
More experimental results are presented in Figure The results are the performance of MetaAligner- 7B on Gemma--nnstuct--B. LLaMA2-Chat-70B: and GPT-3.5-Turbo outputs from the UltraFeedback test set. According the results, MetaAligner can perform effective objective outputs of different policy models, from the small Gemma-7B model to the competitive commercial models GPT-3.5-Tum
alignment on
Unlike full alignment which shows weaker performance as the capability of the policy model increases, objective-wise alignment provides stable improvement for different policy models. For example, aligning "Instruction following" leads to over 20% of approaching- human responses for all policy models, and aligning "Honesty" leads to over 30% of approaching- human responses for all policy models. The reason is that single-objective alignment doesn' t involve the complex interactions between multiple objectives, which facilitates the full realization of MetaAligner's potential to improve the response on the corresponding objective.
G
Implementation Details of Baseline Models
G.1 SFT-baased Methods
Existing SFT-based methods [10, 35] for multi-objective preference aligmment explicitly incorporate reward values into the query via prompt engineering, which includes a text marker for each alignment objective and their corresponding value numbers in the current response. The model is trained to predict the response given the enhanced query, which enables it to learn mapping between the reward value and its reflection in the response. During inference, we achieve alignment by assigning higher reward value to the target objectives. Specifically, we define the following prompting template for HH-RLHE dataset:
[F(g, ri, ral <Harmlessness:: (ri); <Helpfulness>: (r2): <Humour>: (ra) (q)
where g denotes the query, Fi, r2, r' denote the corresponding reward values for the current response, ranging from I to 5, We define the following prompting template for UltraFeedback dataset:
[F F1, T2, f3, r4] <instruction foolowiigoooooooiigg::o.oo (ri : <honesty>: (rz); <honesty>: (rs); <helpfulness>: (ra) [q]
where y denotes the query, Fi, r2, r3, r's denote the corresponding reward values for the current response, obtained from existing reward models.
During inference, aim simultaneously align all objectives with I model to enable fair compar- isons to MetaAligner. For UltraFeedback, since all rewards range from to 5, we set all values to 5 during inference: T1 72 T3 T4 5. For HH-RLHF, there are unified ranges for each objective as all training values are obtained from reward models. Therefore, we set each objective value to its maximum in the training dataset enable higher alignment. The values
to:
r1 = 4.19, 72 = 3.03, 71 = 1.58.
In calculating the GPU hours, we sum the training hours for fine-tuning the LLaMA-Chat-77 policy model for HH-RLHF and UlrraFeedback datasets.
G.2 MODPO
We implement MODPO by implementing the Controllable Direct Preference Optimization (CDPO) [10] algorithm based on its paper introduction. We bypass the controllable preference SFT stage by utilizing the trained model from SFT-based methods as the foundation model. In the CDPO stage, each query is accompanied by two responses e₁ and cg. where each response is constructed into contrastive pairs. In HH-RLHE, have the following prompting template:
[F rf,
Haarnlessnesss: [rt]; <Helpfulnesss: (r?); <Humour>: (rf) (q) I (G)

---

Gemma-instruct-78
following
Helpfulnees
0.26
0.20
0.20
Denkky
0.10
0.10
0.05
0.06
GPT4
Honesty
Truthfulhess
0.4
0.20
0.8
inchecioo ovigg
Trunddives
algreent
0.06
0.0
GPT mOonn
0.00
GP14 wcor Helptuiness
Instruction following
LLaMA2-Chat-70B
0.25
0.20
0.20
0.15
0.15
Deneeh
0.10
0.10
0.08
GPT4 score
GPT4 score Truthfulness
Honesty
0.50
0:20
0.20
0.18
0.10
0.08
0.00
0.00
Instruction following
GPT-3.5
Helpfuiness
0.20
0.16
0.15
0.10
0.06
0.00
GPPT4 ROOrn
0.00
GPT4 soom
Honesty
0.20
0.3
0.16
Jereety
0.10
0.1
GPT4
Figure 6: Objective-wise kernel density estimates of GPT-4 evaluation scores under different align- ment objectives.
25

---

[F(g, C2 F2, r2. r2] <Harnlessness>: [r]): <Helpfulness>:
{r]); <Humour>:
(r]) (q) (c2)
where rf denotes the reward value for the i-th response on the j-th objective. For HH-RLHF, there are no unified ranges for each objective as all training values are obtained from reward models. Therefore, we set each preference value to its maximum in the training dataset to enable higher alignment. The values set to: ,d 4.19, p2 3.03, S 1.58. Similarly, on UltraFeedback have:
[F(g, 9 r,, rf. .r.rf1 .1] <instruction following>: (r) <honesty>: (rî); <honesty>: [rî): <helpfulness>; {nt)
(q) (G))
[F(g, 02, 72, ry. <instruction fintrrctiionfoollwiinn (r]): <honesty>: [r]]: <honesty>: (r]): <helpfulness>> (r2) (q) (
For UltraFeedhack, since all rewards range from 1 to 5, we set all values during the DPO training process: y = ,2 33 = r² 5. In determining the preference scores, we use the CDPO learning goal to transform the task into conditional multi objei-obeective optimization problem. Specifically, the reward value R2 for response e; is calculated as follows:
R.
Wifi
where represents the weight of the i-th objective and gi is calculated as follows:
9i Aolp r if i-th objective is controlled,
(8)
otherwise.
where A; represents the weight of the controlled objective and p; is pre-defined preference value for the j-th objective. our implementation, we set w; A, 1. We aim to simultaneously align all objectives with model to enable fair comparisons to MetaAligner. For UltraFeedback, since all rewards range from I to 5, we set all values during inference: Pl P2 = P3 = P1 = 5. For HH-RLHF, there are no unified ranges for each objective as all training values are obtained from reward models. Therefore, we set each preference value to its maximum in the training dataset to enable higher alignment. The values the response with a higher reward value R2 is used as the chosen response, and the other response is used the rejected response for MODPO training. Specifically, the model is trained via the following loss function:
set to: P1 = 4.19, P2 3.03, ps 1.58. After calculation,
log Te c,,)) c, x)
To( x) Terl(1
LCDPO -Bi log
3 log
Ther
c, x)
where denotes the query, yw Y/ denote the chosen and rejected prompts, edenotes the corresponding condition, and Tree denote the target policy model and the reference model. For implementation, we build our MODPO code based on the OpenRI.F library.
In calculating the GPU hours, we include the training hours for tuning the SFT-based policy model using the CDPO algorithm for HH-RLHF and Ulraakee datasets. We also include the training hours for fine-tuning the original LLaMA2-Chat-77 model for fair comparisons.
G.3 MORLHF
We use the linear scalarization [26, 19] realization of MORLHF with the KL--divergence regularizatiooooooooo Specifically, the model is trained via the following objective function:
n (vl v
argmax E.wD.
(R(gg.v) y) Glog
(10)
F (4/g)
26

---

where w = WN] s.t. Σi= = 1, 2 is the heuristic target preference vector, denote the query and the response, R denotes the reward functions for the target objectives. In implementing the HORLHF algorithm, we first train reward model for each objective based on random samples from 50% of the training data. Following the reward models we used in building the HH-RLHF dataset, we select the GPT2-large foundation models for the reward model, and optimize the reward models using the following pair-wise loss functions:
Fum -log(o (a (R. R, margin))
(11)
where a denotes the Sigmoid function, R. and R, denote the reward output of the chosen response and the rejected response, and margin denotes the margin loss for the corresponding response pairs when multiple responses ranked, such in UltraFeedback. Secondly, fine-tune the LLaMA2- Chat-7B policy model with the highest-ranked responses from the HH-RLHF and UltraFeedback datasets to obtain sub- sub-optimal starting points for RLHF. The SFT training process is formalized as follows:
argmar E B(g.))< [P [Pay (vlv)]
(12)
where q and y denote the query and its corresponding highest-ranie ranked response. Thirdly, following most works in RLHF, we leverage the PPO algorithm [25] to enable parameterized optimization of the policy model. In linear scalarization, we set W1 W2 = WN = t For implementation, we build our MORLHF code based on the OpenRLHF library.
In calculating the GPU hours, we include the training hours for all reward models in HH-RLHF and UltraFeedback datasets, with sum of seven reward model training processes. We also include the training hours for fine-tuning the original starting points. Finally, the PPO training hours for HH-RLHF and UttraFedbbcck are included in the GPU hours.
A2-Chat--BB model for reaching the sub-optimal
G.4 Gelf-Refinement
We include a prompt engineering- self-refinement method as a baseline method to further demonstrate the effectiveness of MetaAligner. Specifically, this approach involves obtaining an initial response from the policy model and then prompting the same model to further refine its own output. For the second stage, we utilize the same prompting strategies as MetaAligner, which is detailed in Appendix C.2. However, this method often requires aligner models with strong in-context learning capabilities, leading high inference costs due to larger model sizes or expensive commercial models. Therefore, we select LLaMA2-Chat-700 a strong policy model the target policy model for self-refinement. As self-refinement does not involve any training procedures, we do not report its GPU hours as other methods.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: all claims accurately reflect the paper's contributions and scope and supported by experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in Appendix A. They will be moved back to the main body once extra pages are allowed.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
28

---

Answer: [NA]
Justification: the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: all experimental setting provided in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
29

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: release all codes, trained models, and part of the training data to facilitate the reproduction of our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cc Coollic/guides/Cdeeuubbei wttpsuumiisssooooiiooooooooiooee for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht / //niss cttssccccpblllcc c/gggiiiii /ooeeSumii cwtuubisssonns ssisssoliosooooo llyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: all experimental setting are provided Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We mostly used GPT-4 as judges rate the responses, trustworthy but costly AI system. Repeating these evaluation processes or the LLM-based tuning algorithms would too expensive and time-consuming.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
30

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: details about computational resources are provided in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt////eurriig http:/:////errusecccommm httpppppppuliiiiitiiiiiii.i..dd. ines?
Answer: [Yes]
Justification: our research conducted in the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the broader impacts discussed in Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
31

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: the safeguards are discussed in Appendix A.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: all used assets (datasets, source codes) properly cited and their licenses listed in Appendix A.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
32

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: the proposed models, datasets, and codes are provided with documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the research didn't involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: the research didn't involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
33

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "gBsnb8FnnC", "reviewer": "Reviewer_XHXG", "summary": "This work extends Aligner to multi-objective alignment scenarios. The main contribution is adding a textual description to each sample in the existing preference datasets to indicate the reason for the preference between chosen and rejected samples. The authors found that MetaAligner is more efficient than previously trained multi-objective alignment algorithms. However, this work is incremental, with some unclear details, insufficient evaluation benchmarks, and a lack of baselines.", "questions": "1. How are the objectives in Section 3.1 obtained?", "limitations": "N/A", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Combining meta-learning concepts with alignment and guiding the model's alignment direction through textual descriptions makes sense to me and could potentially become an important part of RMs in future RLHF works.", "weaknesses": "1. The innovation is incremental, essentially applying Aligner to a new domain and altering training data to change the alignment objectives.\n\n2. The most critical detail, which is how the preference descriptions between each chosen and rejected pair in the existing preference datasets are obtained, is unclear.\n\n3. The authors did not adequately present results on objective datasets such as MATH and HumanEval, nor on mainstream subjective evaluation sets like MT-Bench/AlpacaEval.\n\n4. There is a lack of comparative analysis with previous multi-objective alignment methods, such as \"SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for reviewing our paper and providing valuable comments. The following parts contain our point-to-point responses to the weaknesses and questions. We hope they can help address the reviewer's concerns.\n\n**Response to Weakness 1**: We'd like to clarify our novelty and contributions compared to Aligner and other alignment works: \n1) MetaAligner is the first work to explore dynamic multi-objective alignment which enables instance-level alteration of the alignment objectives without re-training and unlimited simultaneous alignment objectives. In contrast, Aligner can only perform single-preference alignment and other previous multi-objective alignment works can only perform alignment on fixed objectives. To achieve this capability, MetaAligner is trained to approach chosen responses from rejected responses, considering dynamic target objectives. This training paradigm is not explored by Aligner or any other alignment methods.\n2) MetaAligner is the first work that can perform generalizable multi-objective alignment, which can align unseen objectives in a zero-shot manner, while Aligner or any other previous multi-objective alignment works can only perform alignment on objectives that they were trained on. To achieve this capability, we innovatively leverage the in-context learning ability of MetaAligner to understand the unseen objectives and plan for alignment, allowing the users to flexibly describe their alignment objectives in a natural language format, another key difference from the methodology of Aligner.\n3) Different from Aligner, we conduct experiments on datasets in both general (HH-RLHF, UltraFeddback) and mental health (IMHI) domains, comprehensively showing the effectiveness of MetaAligner. We also conduct experiments to prove MetaAligner’s generalizable alignment ability to unseen objectives and evaluate its accuracy in objective-wise alignment, which is not covered by Aligner. We have provided related codes, models and data in the submitted paper, and they will be released for public usage. \n\n**Response to Weakness 2**: While we did not explicitly use the term “preference descriptions” in our paper, we understand that the reviewer may refer to the dynamic combinations of multi-objective descriptions for each chosen-rejected pair. For each chosen-rejected pair, we first identify the target dynamic objectives via their preference annotations in the original dataset. Then the “preference descriptions” are obtained via a random shuffle of these target objectives, followed by a concatenation of their textual descriptions. More details are illustrated in Sec. 3.1 and Algorithm 1. We also provided examples of the building process in Appendix C.3 to further clarify the process.\n\n**Response to Weakness 3**: We appreciate the opportunity to clarify our choice of datasets: \n1) Tasks of MATH and HumanEval are not well-suited for evaluating multi-objective aspects, such as \"Harmlessness,\" \"Humor,\" and \"Fairness,\" because they do not engage with the human-centered elements that require subjective judgment and contextual interpretation. Because of these factors, objective datasets are rarely used in multi-objective preference alignment research [1][2][3][4][5]; \n2) While MT-Bench and AlpacaEval are popular for single-preference alignment, they lack support for multi-objective alignment, which is central to our work. The absence of multi-objective statistics on their public leaderboards limits their applicability to our research. Additionally, these benchmarks offer a limited number of testing queries, with only 80 for MT-Bench and approximately 800 for AlpacaEval 2.0; \n3) We utilized the testing splits from HH-RLHF and UltraFeedback, both of which are widely recognized in multi-objective preference alignment research ([1][5]). These datasets provide extensive testing queries (15,000 each), ensuring a comprehensive evaluation. We also included the IMHI benchmark to assess performance in the mental health analysis domain, with 2,400 testing queries available. All our testing data is publicly accessible, and our methods are reproducible, ensuring the reliability of our results.\n\n**Response to Weakness 4**: While SPO is indeed relevant, it was first submitted on May 21, 2024. Given that the NeurIPS 2024 main conference paper deadline was on May 22, 2024, it was not feasible to incorporate SPO into our study within such a limited timeframe. Our paper includes comparisons with representative multi-objective alignment methods such as MORLHF, CDPO, and RiC, which effectively demonstrate the strengths of our approach. We acknowledge the value of including SPO and plan to add it as a baseline in future work. However, given the timing constraints, we believe the absence of SPO should not be considered a weakness at this stage.\n\nIn our work, we selected MORLHF, MODPO, and SFT-based methods as baseline works, which are widely used baseline methods in previous works[1][3][4][5]. We also do not observe a significant decrease in the number of our compared baseline methods compared to these works.\n\n**Response to Question 1**: We guess the reviewer refers to how the objective combinations are formed, please see our response to Weakness 2, where we detail the process of creating dynamic multi-objective combinations.\n\n**References:**\n\n[1] Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment. arXiv preprint arXiv:2402.10207.\n\n[2] Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment. arXiv preprint arXiv:2402.19085.\n\n[3] Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization. arXiv:2310.03708\n\n[4] Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards. arXiv preprint arXiv:2402.18571.\n\n[5] SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling. arXiv: 2405.12739", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "chnJnb35Pe", "reviewer": "Reviewer_i5Yu", "summary": "This work proposes MetaAligner, a plug-and-play multi-objective alignment method that can generalize to unseen objectives.", "questions": "1. I am interested in understanding the authors' rationale for choosing an indirect comparison with the ground truth and how they view the validity of this choice, especially considering that many models, after being aligned through RLHF, are likely to already exhibit strong performance on the objectives of test set.\n2. In the scenario where columns contain exactly two of the three attributes: Harmless, Helpful, and Humorous, which corresponds to the position of the second column in Figure 2 HH-RLHF. Let $( x_1, x_2, x_3 )$ denote the probabilities of having two characteristics, specifically including $Harmless$, $Helpful$, and $Humorous$. According to the figure, $( x_1 = 0.57 )$, $( x_2 = 0.53 )$, and $( x_3 = 0.85 )$. Now, there are exactly three possible combinations of choosing two out of the three characteristics: $(Harmless, Helpful)$, $(Helpful, Humorous)$, and $(Harmless, Humorous)$, and these combinations are mutually exclusive. Let the probabilities of these three events be $( a_1, a_2, a_3 )$ respectively, such that $( a_1 + a_2 + a_3 = 1 )$. The event `contains two characteristics, with one of them being Harmless` can be described as the disjoint union of the events `only contains (Harmless, Helpful)` and `only contains (Harmless, Humorous)`. Therefore, $( x_1 = a_1 + a_3 )$. Similarly, we have $( x_2 = a_1 + a_2 )$ and $( x_3 = a_2 + a_3 )$. Summing both sides of these three equations, we obtain $( 1.95 = 0.57 + 0.53 + 0.85 = x_1 + x_2 + x_3 = 2(a_1 + a_2 + a_3) = 2 \\times 1 = 2 )$. I am sincerely inquiring whether my understanding of Figure 2 is mistaken.", "limitations": "The article provides a thorough discussion of its limitations in terms of deployment and the number of objectives to be selected.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This is a well-written paper that clearly expresses its core ideas.\n2. MetaAligner is a lightweight alignment method that is easier to tune in multi-objective scenarios, with lower training overhead compared to baselines.\n3. MetaAligner is a plug-and-play alignment method that can generalize to various open-source and closed-source models, enabling multi-objective alignment.\n4. MetaAligner has generalizability, capable of aligning with zero-shot objectives.", "weaknesses": "1. In the experimental section, the aligned answers are compared to the ground truth answers in the dataset to calculate the win rate. This lacks a more direct performance comparison, making the argument for performance advantage insufficiently robust. For example, comparing different baselines by win rate against the ground truth answers in the dataset is inadequate to demonstrate the performance advantage over the baselines.\n2. There is a lack of performance comparison with prompt engineering-based aligners. Specifically, it is unclear whether similar effects could be achieved by prompting the chat model to modify the answers within the prompts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for reviewing our paper, which provides careful assessments and valuable comments. We also thank the reviewer for recognizing the strengths of our work. The following parts contain our point-to-point responses to the weaknesses and questions. We hope they can help address the reviewer's concerns.\n\n**Response to Weakness 1**: We adopted the win rate evaluation method because it is a widely recognized approach in SOTA benchmarks such as  AlpacaEval [1], Arena-Hard [2], and MT-Bench [3]. For instance, Arena-Hard uses win rates against GPT-4 outputs to measure model performance, a practice followed by many recent alignment studies [4][5]. This method is considered both effective and robust.\n\nIn our evaluation of MetaAligner, we calculated win rates for each policy model's output against ground truth answers both before alignment: $W_b$ and after alignment $W_a$. The performance of MetaAligner was quantified by the difference in win rates: $W_a-W_b$, as shown in Table 2. \n\nWhen comparing MetaAligner performance based on the same policy model (Table 3 and Figure 3), we reported win rates after alignment ($W_a$) directly, because all methods used the same policy model ($W_b$ are the same), ensuring a fair comparison. We believe these methods provide a reliable assessment of MetaAligner's performance advantage.\n\n**Response to Weakness 2**: We agree that including prompt engineering-based aligners would further demonstrate MetaAligner’s effectiveness. This approach involves obtaining an initial response and then prompting the same model to refine it. However, this method often requires aligner models with strong in-context learning capabilities, leading to high inference costs due to larger model sizes or expensive commercial models. MetaAligner offers a cost-effective advantage by employing a smaller model  (e.g., MetaAligner-1.1B) for the refinement stage, reducing inference costs while maintaining competitive performance through supervised fine-tuning on the alignment dataset. One MetaAligner model can also be applied to different policy models.\n\nTo further address the concern, we take time to add a new prompt-based baseline using LLaMA2-70B-chat with one-time self-refinement. For HH-RLHF, the results are as follows:\n\n|Method| Harmless   | Helpful | Humour|\n| ----------- | ----------- | ----------- | -----------|\n| LLaMA2-70B-chat      |  +6.9%      |   +12.8%  |   +14.91%   |\n| MetaAligner-1.1B      |  +6.58%      |   +7.42%  |   +22.58%   |\n| MetaAligner-7B      |  +16.58%  |   +14.42%  |   +29.08%   |\n\nFor UltraFeedback we have:\n\n|Method| IF   | Honest | Truthful |  Helpful   |\n| ----------- | ----------- | ----------- | -----------| ----------- |\n| LLaMA2-70B-chat      |  +18.9%      |   +30.19%  |   +20.0%   |  +14.9%    |\n| MetaAligner-1.1B      |  +6.0%      |   +12.67%  |   +17.33%   |    +16.33%        |\n| MetaAligner-7B      |  +31.0%      |   +27.0%  |   +31.33%   |     +17.0%            |\n\nAccording to the results, MetaAligner-7B can significantly outperform LLaMA2-70B-chat performance on 6 out of 7 objectives, with only 10% in inference cost. MetaAligner-1.1B can also achieve better or comparable performance on 4 objectives, but with only 1.5% in inference cost. These results prove the significant advantage of MetaAligner over the refinement-based methods. We will include these results in the future version of our paper.\n\n**Response to Question 1**: As noted in our response to Weakness 1, our evaluation method measures performance based on relative differences in win rates before and after alignment, ensuring that the capability of the policy model does not affect evaluation accuracy (see Table 2). While strong models like LLaMA2-Chat-70B and ChatGPT may start with high win rates (high $W_b$), their influence is neutralized by subtracting these rates from the aligned results ($W_a-W_b$), isolating the contribution of MetaAligner. This approach ensures that our assessment reflects the genuine impact of the MetaAligner modules.\n\n**Response to Question 2**: We thank the reviewer for carefully assessing the statistics and pointing out this mistake. We mistyped one of the figures when drawing the heat map with Python codes. We have rechecked and the percentage for “helpful” should be 0.58. We will correct this error in the future version of the paper.\n\n**References**:\n\n[1] Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv: 2404.04475\n\n[2] From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. arXiv: 2406.11939\n\n[3] Judging llm-as-a-judge with mt-bench and chatbot arena. NIPS 2023.\n\n[4] SimPO: Simple Preference Optimization with a Reference-Free Reward. arXiv: 2405.14734\n\n[5] Low-Redundant Optimization for Large Language Model Alignment. arXiv: 2406.12606", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YzVTY8TM2X", "reviewer": "Reviewer_E7pS", "summary": "The author proposes Meta-Objective Aligner (MetaAligner), the first policy-agnostic and generalizable method for multi-objective preference alignment.", "questions": "see above.", "limitations": "N/A.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "As a lightweight algorithm, this is particularly meaningful as model parameters continue to grow. Conditional weak-to-strong correction extends weak-to-strong correction and weak-to-strong generalization, which are currently significant research topics.", "weaknesses": "1. I am not entirely clear about the difference between this paper and Aligner. The author mentions:\n> As shown, conditional weak-to-strong correction of MetaAligner extends Aligner [12] to multi-objective alignment scenarios, which are not directly solvable by Aligner itself.\n\nI do not fully understand the distinction between this paper and Aligner. After reading the Aligner paper, I noticed that the authors effectively expand on multi-objective alignment. However, the introduction seems to lack a description of Aligner, making it difficult for readers to understand.\n\n2. I did not understand what the author is trying to convey with Figure 2. Can the author explain the meaning of Figure 2? The elements it aims to express are very complex. Could there be a more streamlined figure, similar to Figure 1 in the Aligner paper? Is the extension to multi-objective alignment at the methodological level or the dataset level?\n3. In the experimental section, how did the authors generate an effective correction dataset based on HH-RLHF, UltraFeedback, and IMHI?\n4. After reading the Aligner paper, I observed that it conducted experiments on the three dimensions of \"Helpful,\" \"Harmless,\" and \"Honest\" as described by the authors in line 42. I hope the authors can add comparative experiments with Aligner in the ablation study section. From a reviewer's perspective, the contribution of this paper to LLM alignment is undeniable, but I am particularly concerned about the effective improvement and extension of MetaAligner compared to Aligner.\n\nMy questions are all included in the Weaknesses section, with point 4 being my primary concern. I hope the authors can effectively enhance the ablation study section. I believe that algorithms like MetaAligner and Aligner are more efficient compared to RLHF and DPO. I look forward to the authors' response and will consider raising the score further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for reviewing our paper and recognizing the strengths of our work. We also thank the reviewer for willing to consider raising the scores. The following parts contain our point-to-point responses to the weaknesses and questions. We hope they can help address the reviewer's concerns.\n\n**Weakness 1**: We apologize for not including more details about Aligner in the main body due to page limits. Aligner is an SFT-based single-preference alignment method, which trains the model to correct the weak responses to approach the strong responses. The advantages of MetaAligner over Aligner are as follows: \n1) MetaAligner is the first work to explore dynamic multi-objective alignment which enables instance-level alteration of the alignment objectives without re-training and unlimited simultaneous alignment objectives. In contrast, Aligner can only perform single-preference alignment. To achieve this capability, MetaAligner is trained to approach chosen responses from rejected responses, considering dynamic target objectives. This training paradigm is not explored by Aligner or any other alignment methods.\n2) MetaAligner is the first work that can perform generalizable multi-objective alignment, which can align unseen objectives in a zero-shot manner, while Aligner or any other previous multi-objective alignment works can only perform alignment on objectives that they were trained on. To achieve this capability, we innovatively leverage the in-context learning ability of MetaAligner to understand the unseen objectives and plan for alignment, allowing the users to flexibly describe their alignment objectives in a natural language format, another key difference from the methodology of Aligner.\n3) Different from Aligner, We conduct experiments on datasets in both general (HH-RLHF, UltraFeddback) and mental health (IMHI) domains, comprehensively showing the effectiveness of MetaAligner. We also conduct experiments to prove MetaAligner’s generalizable alignment ability to unseen objectives and evaluate its accuracy in objective-wise alignment, which is not covered by Aligner.\n\nWe will include more of these explanations in the future version of our paper.\n\n**Weakness 2**: We apologize for the complexity of Figure 2 and will revise it to provide a clearer depiction of MetaAligner. We will discard the MORLHF and MODPO illustrations, and reorganize the MetaAligner pipeline from left to right as follows: (1) dynamic objective reformulation, introducing the building process of the dynamic dataset; (2) conditional weak-to-strong correction; (3) generalizable inference. We will simplify the current complex illustration within each stage. Due to time limits during rebuttal, we will include the modified figure in the future version of our paper.\n\nThe extension to multi-objective alignment occurs at both the methodological and dataset levels. Methodologically, we are the first to propose dynamic multi-objective alignment to enable instance-level alternation of the alignment objectives, which is achieved by objective-aware weak-to-strong corrections. We are also the first to perform generalizable multi-objective alignment to align unseen objectives, which is achieved by MetaAligner’s in-context learning ability. At the dataset level, we use Algorithm 1 to reorganize the dataset, training MetaAligner to handle various combinations of alignment objectives, which is crucial for its flexible and zero-shot alignment capability.\n\n**Weakness 3**: Different from Aligner which utilized LLMs to correct the weak answers to generate correction pairs, we leveraged existing response pairs within alignment datasets (e.g. HH-RLHF, UltraFeedback) to supervise weak-to-strong corrections without needing to explicitly generate correction pairs.\n\nSpecifically, MetaAligner is trained to generate improved outputs using weak responses as references and strong responses as supervision, guided by the target objectives from the dynamic multi-objective dataset. The target objectives of each response pair are obtained via their preference annotations in the original dataset. More details are described in Section 3.1.\n\n**Weakness 4**: MetaAligner and Aligner can have fundamentally different evaluation processes. Aligner uses a single-preference approach, generating one correction per policy output and evaluating it across \"Helpful,\" \"Harmless,\" and \"Honest\" dimensions. In contrast, MetaAligner can explicitly incorporate target objectives in the input, allowing it to generate different corrections for the same output based on specified objectives, a key advantage over Aligner. We didn't include the objective-specific evaluation results because as Section 4.5 demonstrates, MetaAligner usually provides better results when controlling on the target objectives than full alignment.\n\nWe agree that including comparisons with Aligner in different evaluation settings would further enhance our arguments. We provide our latest results, comparing the performance of MetaAligner-7B and Aligner-7B on the LLaMA2-7B-chat policy model: \n\n|Method| Harmless   | Helpful | Humour|\n| --- |---|---|---|\n| Aligner-7B | 72.0%|79.9%|70.12%|\n| MetaAligner-7B (full alignment)|77.5%|82.0%|83.0%|\n| MetaAligner-7B (objective-specific alignment)|79.1%|84.29%|84.45%|\n\nFor UltraFeedback we have:\n\n|Method|IF|Honest|Truthful|Helpful|\n| --- |---|---|---|---|\n|Aligner-7B|52.38%|44.23%|37.19%|39.1%|\n|MetaAligner-7B (full-alignment)|51.33%|48.0%|55.67%|44.33%|\n|MetaAligner-7B (objective-specific alignment)|57.9%|48.6%|57.92%|48.13%|\n\nThese results show that though Aligner is comparable to MetaAligner on objectives such as “Helpful” in HH-RLHF, it significantly underperforms MetaAligner in fine-grained objectives such as “Humour” and “Truthful”. These results show the effectiveness of the dynamic multi-objective alignment method. Objective-specific alignment further improves the performance of MetaAligner.\n\nThese results will be included in future versions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q2FqoeWI6y", "reviewer": "Reviewer_vbsw", "summary": "Multi-objective alignment of language models is a significant topic for LLM community, while many prior work are either costly-to-compute or policy-dependent, restricting the further depolyment. This work is aimed at providing a retraining-free and policy-agnostic approach for multi-objective alignment, namely *MetaAligner*.\n\nThe framework is composed of 3 stages:\n- Data reorganization from multi-objective datasets, getting $D_e, D_p$;\n- Supervised model training:\n   - Warming up on a randomly downsampled dataset of $D_e$;\n   - Finetune on $D_e$, as a equal-preference alignment step;\n   - Contrastive training on $D_p$.\n- Inference with prompting for even unseen objectives.\n\nExperiments are conducted on three commonly-used datasets, and the improvements are reported as prominent. Furthermore, the authors claim that they can achieve generalizable alignment to unseen objectives, through prompting.", "questions": "- In Figure 3, why would the performance on some unseen objectives drop after adding more objectives? For example, \"repeat\" and \"readable\" for *MetaAligner-7B*.\n- How much efforts has the author spent on tuning hyper-parameters of *MetaAligner*?\n- How does the author set the preference weights for *MORLHF*, *MODPO*, and *RiC*? Is that fair?", "limitations": "The limitations are well discussed in Section A.2.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality:\n- The whole 3-stage framework of *MetaAligner* is original.\n\nClarity:\n- This paper is well-written and well-organized.\n- Most figures and tables are friendly to the readers.\n\nSignificance:\n- This is the first policy-agnostic and retraining-free multi-objective alignment approach, to the knowledge of the reviewer.\n- The experiments are extensive and very solid, including multiple datasets and base models. The author also conducted extensive experiments comparing *Meta-Aligner* with other baselines.", "weaknesses": "*Major*\n- The whole framework is complicated and a bit engineering. There is no ablation study on each components. For example, it is better to show the performance if discarding the $D_e$ dataset, which is claimed as novel.\n- In Table3, the comparison of *MetaAligner* with other baselines is somehow unfair, since *MetaAligner* utilizes at least double parameters of *MORLHF*, *MODPO*, and *RiC* (the 1.1B version performs worse than baselines, so the reviewer is talking about >=7B version).\n- The proposed method is not able to cater to different users' taste, because it can only be aligned to multiple objectives while no weights on them are introduced. But many works[1][2][3][4][5] are able to do this, making the contribution less significant.\n\n*Minor*\n- this work is a bit marginal:\n  - The reorganization process is just preparing the prompting format, which is trivial.\n  - The SFT training is a common practice in the community[1][2][3]. The only difference is the design of a detached aligner.\n  - The inference stage is simply prompting engineering, lacking guarantees.\n- The *MetaAligner* is not a light-weight approach, as the aligner model is still large, making it hard to be deployed.\n\n[1] Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment. arXiv preprint arXiv:2402.10207.\n\n[2] Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment. arXiv preprint arXiv:2402.19085.\n\n[3] Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards. arXiv preprint arXiv:2402.18571.\n\n[4] Decoding-Time Language Model Alignment with Multiple Objectives. arXiv preprint arXiv:2406.18853\n\n[5] Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. arXiv preprint arXiv:2310.11564.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and recognizing its strengths. We hope the following responses can help address the concerns.\n\n**Weakness 1**: MetaAligner has a simple methodology. It triggers 2 novel capabilities: \n1. instance-level alternation of the alignment objectives without re-training; \n2. generalizable multi-objective alignment to align unseen objectives in a zero-shot manner.\n\nWe designed dynamic objectives reformulation (Sec. 3.1) to build training data with various combinations of target objectives which are used to train MetaAligner in a conditional weak-to-strong correction manner (Sec. 3.2). With the dynamic multi-objective capability triggered by the above modules, We further extend it to generalizable inference (Sec. 3.3) with the in-context learning ability of MetaAligner. All modules are centered on achieving the two capabilities.\n\nWe didn't show the ablation results because MetaAligner has a simple and intuitive structure, and most components are essential for the models. We agree that showing the ablation results for $D_e$ can be useful evidence for proving its effectiveness. Due to time limits during rebuttal, we will add these results in the future version of our paper.\n\n**Weakness 2**: We admit that the MetaAligner can introduce extra inference costs (we discussed in limitations, appendix A.2). But during training, MetaAligner has fewer trainable parameters and lower costs:\n1) All policy model weights are fused and only MetaAligner parameters are updated. When using LLaMA2-7B as policy (Table 3),  MetaAligner-1.1B has much fewer trainable parameters than the baseline methods, but with comparable or better performance. MetaAligner-(7B,13B) have comparable or more trainable parameters, but with much better performance.\n2) MetaAligner models can be trained once and used for all policy models (see Table 2), while the baseline algorithms have to be trained from scratch for each new policy model, as they update policy parameters.\n3) The efficiency of MetaAligner improves with larger policy models. Note that we used LLaMA2-7B in Table 3 as the policy model due to limited computation resources. With larger policy models (e.g. LLaMA2-70B), the training cost for baseline methods vastly increases, while the training cost for MetaAligner remains constant.\n\n**Weakness 3**: Compared to the user-specified weights for all objectives, MetaAligner’s weight-free specification of desired objectives and zero-shot alignment on unseen objectives are more useful features that no other methods achieve. \n1) The optimal weight assignment is hard to determine because the Pareto front for multi-objective alignment is usually hard for humans to perceive and quantify[2]. We believe users are more concerned with properly improving certain objective combinations than quantifying their weights by themselves. With MetaAligner, the users can simply identify their desired objectives, and the model can modify the response accordingly with an implicit optimal Pareto front learned during training. We believe our proposed solution is more user-friendly in practical scenarios.\n2) Compared to previous methods that require users to assign weights to every trained objective, MetaAligner allows users to customize their own combinations of objectives for each instance, which is more flexible.\n3) MetaAligner allows users to specify new objectives by providing text descriptions, and perform zero-shot alignment on these unseen objectives, a feature no previous methods have achieved.\n\n**Weakness 4**:\n1) We believe the core of the reorganization process is not the prompt, but identifying target objectives for each chosen-rejected pair, the key to the dynamic multi-objective alignment.\n\n2) MetaAligner shows 2 key advantages over previous SFT methods: (1) MetaAligner achieves unlimited simultaneous alignment objectives at the instance level, with the dynamic multi-objective alignment training; (2) MetaAligner achieves zero-shot alignment for unseen objectives, with its in-context learning ability. \n\n3) Prompt-based inference is the optimal way to leverage the in-context learning ability of MetaAligner to achieve generalizable alignment. It's also widely used in previous works[1][2][4]. Following their settings, we provide empirical evaluations on generalizable alignment (Sec. 4.4). Providing theoretical guarantees is hard and beyond the scope of most related works due to the low interpretability of deep models.\n\n**Weakness 5**: MetaAligner can have extra inference costs, but these are reasonable trade-offs for savings during training. Due to word limits, please refer to our response to Weakness 1.\n\n**Question 1**: Enhancements in one objective can affect performance in certain objectives due to their controversial nature, known as \"alignment tax\"[2]. For example, aligning on \"Fair\" with MetaAligner-(7B, 13B) benefits its win rates, but harms performance on objectives such as \"Readable\" and \"Factual\" compared to when \"Fair\" is unaligned. This phenomenon is expected and unavoidable. We will include these analyses in the future paper.\n\n**Question 2**: MetaAligner training is simple. We searched learning rates between 5e-6 and 5e-5. All hyperparameters are presented in Table 5 in the appendix.\n\n**Question 3**: We tried to ensure a fair comparison. For MORLHF, we set uniform preference weights, as the weight search process for over 2 objectives can be costly. The weights determination process of MODPO is presented in appendix G.2. RiC doesn’t involve assigning weights, but a preference-to-reward mapping process (appendix G.1).\n\n**References**:\n[1] Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment.\n\n[2] Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment.\n\n[3] Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards.\n\n[4] SALMON: Self-Alignment with Instructable Reward Models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Paper presents MetaAligner, a multi-objective preference alignment of language models. It is both policy-agnostic (working with various LLMs) and generalizable to unseen objectives.

I lean towards a borderline acceptance as a poster presentation: The paper addresses a relevant problem of multi-objective alignment in a way that could be beneficial, a policy-agnostic, retrained-free approach for diverse preferences is a desirable goal in the field.
Reviewers i5Yu, E7pS, and vbsw all see merit in the work, acknowledging its potential and highlighting aspects like novelty, clarity, and experimental effort. 

Core ideas are worth exploring further despite some weaknesses.

The authors have actively engaged with reviewers, demonstrating a willingness to clarify misunderstandings and addressed concerns.
Decision: Accept (poster)