MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models

Authors: kailai  yang
zhiwei  liu
qianqian  xie
jimin  huang
tianlin  zhang
sophia  ananiadou
metaaligner  metaaligner
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 12:57:02

Abstract: Recent advancements in large language models (LLMs) focus on aligning to heterogeneous human expectations and values via multi-objective preference alignment. However, existing methods are dependent on the policy model parameters, which require high-cost repetition of their alignment algorithms for each new policy model, and they cannot expand to unseen objectives due to their static alignment objectives. In this work, we propose Meta-Objective Aligner (MetaAligner), the first policy-agnostic and generalizable method for multi-objective preference alignment.
MetaAligner models multi-objective alignment into three stages: (1) dynamic objectives reformulation algorithm reorganizes traditional alignment datasets to supervise the model on performing flexible alignment across different objectives; (2) conditional weak-to-strong correction paradigm aligns the weak outputs of fixed policy models to approach strong outputs with higher preferences in the corresponding alignment objectives, enabling plug-and-play inferences on any policy models, which significantly reduces training costs and facilitates alignment on close-source policy models; (3) generalizable inference method flexibly adjusts target objectives by updating their text descriptions in the prompts, facilitating generalizable alignment to unseen objectives.
Experimental results show that MetaAligner achieves significant and balanced improvements in multi-objective alignments on 10 state-of-the-art policy models, and saves up to 93.63% of GPU training hours compared to previous alignment methods. The model also effectively aligns unseen objectives, marking the first step towards generalizable multi-objective preference alignment.

Main paper: 
 introduction: 
 The recent advancements in large language models (LLMs) have focused on generating high-quality responses that align with human expectations and values. At the final stage of alignment, LLMs are supervised on human preference data via reinforcement learning from human feedback (RLHF) [40, 22, 27] , where a proxy, directly trained on human preferences data, is leveraged to provide scalar rewards for reinforcement learning (RL) on the target model [22] .However, human expectations and values include a broad spectrum of heterogeneous and multidimensional objectives, which makes scalar supervisions inefficient for aligning diverse and inclusive human preferences [3, 24] . These drawbacks motivate further exploration into multiobjective alignment algorithms. Some intuitive methods extend RLHF into multi-objective RLHF (MORLHF) [26, 19, 24] . Due to its substantial computational cost [19, 24] and the unstable nature Table 1 : Comparisons between previous alignment methods and MetaAligner on different features. "Policy-Agnostic Alignment" means the alignment algorithm is independent of the target policy model parameters, and "Generalizability" denotes zero-shot alignment capability on unseen objectives.Paradigm Multi-Objective Alignment Policy-Agnostic Alignment Generalizability RLHF [22] PPO MORLHF [19] PPO MODPO [10, 39] SFT, DPO RiC [35] SFT Aligner [12] SFT MetaAligner SFT of the proximal policy optimization (PPO) [25, 15, 23] algorithm, other methods seek to bypass the RL paradigm with multi-objective direct preference optimization (MODPO) [39, 10] or supervised fine-tuning (SFT)-based methods [35, 10] , which customized prompting strategies to incorporate multiple reward values into queries explicitly.The above methods for multi-objective alignment bear one commonality: the dependence on the policy model's parameters. This paradigm inevitably brings two key limitations: (1) they require repetition of their high-cost alignment algorithms for each newly-introduced policy model, which is incompatible with the increasing sizes and fast iteration of current foundation models [1, 30, 6, 29] ; (2) all target models are statically aligned on pre-determined (e.g. "Helpful", "Harmless", "Honest" [39, 10] ) objectives, with currently no efforts in expanding and evaluating their capabilities on unseen objectives. This ignorance leads to poor generalizability of existing multi-objective alignment methods.In this work, we propose Meta-Objective Aligner (MetaAligner), the first policy-agnostic and generalizable method for multi-objective preference alignment. MetaAligner models multi-objective alignment into three stages: (1) the dynamic objectives reformulation algorithm reorganizes traditional alignment datasets into dynamic-objective alignment datasets, training MetaAligner to perform flexible alignment across different objectives. It achieves this by incorporating and combining text descriptions of various alignment objectives in a prompt-based manner; (2) the conditional weak-tostrong correction paradigm aligns the weak outputs of policy models to approach strong outputs with higher preferences in the corresponding alignment objectives. During training, MetaAligner is stacked onto policy models to perform objective-aware corrections, where parameters of the policy model are fixed and MetaAligner is optimized with an SFT-based three-step training process: warming up, equal-preference alignment, and contrastive-preference alignment. This paradigm enables MetaAligner to perform plug-and-play inferences on any policy models even without access to their parameters, which significantly reduces training costs and facilitates alignment on close-source LLMs;(3) the generalizable inference method flexibly adjusts target objectives by updating their text descriptions in the prompts. This method can also adapt MetaAligner to unseen objectives and achieve new alignment strategies via in-context learning [14] , a new feature with rare previous exploration in alignment of language models. The number of aligned objectives also becomes expandable, theoretically leading to unlimited simultaneous alignment objectives. Table 1 compares key features between MetaAligner and previous methods. As shown, conditional weak-to-strong correction of MetaAligner extends Aligner [12] to multi-objective alignment scenarios, which are not directly solvable by Aligner itself. MetaAligner is also the first multi-objective alignment method to achieve policy-agnostic alignment and generalization to unseen objectives, two key advantages over previous methods such as MORLHF, MODPO, and SFT-based methods.In summary, our main contributions are: (1) we propose MetaAligner, the first policy-agnostic method for multi-objective preference alignment. It performs multi-objective alignment efficiently, without tuning the policy models or accessing their parameters. Experimental results show that MetaAligner outperforms previous alignment methods and saves up to 93.63% of GPU training hours;(2) we utilize MetaAligner to exert zero-shot preference alignment for unseen objectives. To our knowledge, this work marks the first attempt at generalizable multi-objective preference alignment.Experimental results show that MetaAligner can simultaneously perform effective alignment for six unseen objectives while maintaining performance on aligned objectives; (3) We examine MetaAligner on three preference alignment datasets. Experimental results show that MetaAligner improves training; (3) generalizable inference for multi-objective alignment. The paradigm is illustrated in Figure 1 .Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: all claims accurately reflect the paper's contributions and scope and are supported by experimental results.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The limitations are discussed in Appendix A. They will be moved back to the main body once extra pages are allowed.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
dynamic objectives reformulation: 
 We propose a dynamic objectives reformulation algorithm to construct a dynamic multi-objective dataset, which triggers MetaAligner's ability for flexible adjustment of alignment objectives. Specifically, any typical multi-objective preference alignment dataset D m with m samples and N objectives can be re-organized as {q i , y i1 , y i2 , P i } m i=1 , where P i = [p i1 , ..., p iN ] T and p ij ∈ {≻, ≺, ≡} indicates the preference on j-th objective.Algorithm 1 Dynamic objectives reformulation. Require: Raw dataset D m : {q i , y i1 , y i2 , P i } m i=1 ; Objective text descriptions: [⟨d 1 ⟩ , ..., ⟨d N ⟩]; Prompting template: T (q, y, O, t) Ensure: Contrastive subset D c ; Equal subset D e .1: D c ← ∅, D e ← ∅ ▷ Initialize the 2 subsets. 2: for i ∈ {1, ..., m} do ▷ Loop on instances.3:O ≻ ← ∅, O ≺ ← ∅, O ≡ ← ∅ 4:for j ∈ {1, ..., N } do 5:if p ij is ≻ then ▷ Collect the objectives where y i1 outperforms y i2 . 6 :O ≻ ← O ≻ ∪ {⟨d j ⟩} 7:else if p ij is ≺ then ▷ Collect the objectives where y i2 outperforms y i1 .8:O ≺ ← O ≺ ∪ {⟨d j ⟩} 9:else ▷ Collect the objectives where y 1 and y 2 performs equally. if O ≻ ̸ = ∅ then▷ Build the training pairs where y i1 is used as the target. if O ≺ ̸ = ∅ then▷ Build the training pairs where y i2 is used as the target. D e ← D e ∪{(T (q i , y i2 , O ≡ , t), y i1 )}end if 28: end for We define a text description for each objective: [⟨d 1 ⟩ , ..., ⟨d N ⟩], where ⟨d j ⟩ denotes the natural language description for j-th objective. Some examples of such descriptions are in Figure 1 and a full list is in Appendix C. With a pre-defined prompting template T , we build a contrastive subset D c and another equal subset D e from D m . D c includes all contrastive response pairs where p ∈ {≻, ≺} and D e includes all equal response pairs where p ∈ {≡}. For example, we utilize the following template T (q, y, O, t) in building for the IMHI [34] dataset:[T (q, y, O, t)] Edit the following Question-Answer pair to make it {t} considering the following objectives {O} | Question: {q} | Answer: {y} | Edit:where q denotes the query, y denotes a corresponding response, O denotes the concatenation of text descriptions for the target objectives, and t ∈ {equal, better} depends on the current building subset. Details of the dynamic objectives reformulation algorithm are described in Algorithm 1. For an instance within the processing loop (line 2): {q, y 1 , y 2 , P }, the algorithm performs a two-step reformulation: (1) (3) the reward-free alignment method (no explicit preference values required) avoids complicated preference-to-reward mapping [35] process in previous SFT-based multi-objective alignment methods.Based on the dynamic multi-objective training datasets, we train MetaAligner in a conditional weakto-strong correction manner, which follows an SFT-based training objective and a three-step training paradigm.MetaAligner is a standard conditional seq-to-seq model on top of the original policy model π ϕ , which re-distributes the policy model output y 0 considering objectives O as follows:EQUATIONwhere δ θ denotes the MetaAligner module parameterized by θ, t depends on the training dataset. Conditional weak-to-strong correction directly trains MetaAligner to align the weak policy model output y 0 to the strong target output y, which has higher preference values in corresponding objectives O. We have the standard cross-entropy loss as the training objective:EQUATIONWe fix the parameters of the policy model, thus excluding ϕ from the weight update process. In practice, we use the dynamic multi-objective dataset for supervision, where the weak response in each query-response pair is directly leveraged as samples y 0 from unknown policy models. Therefore, we eliminate the second term in Eqn. 5 and simplify the training objective as:EQUATIONThe above action poses two advantages: (1) the computation resources required for MetaAligner training is detached from policy model size, which enables policy-agnostic and cost-efficient alignment for large policy models; (2) MetaAligner works only via outputs from the policy models, which allows training and inference for alignment on close-source policy models [1, 21, 2] .In practice, we utilize an LLM as the base model for MetaAligner, which provides domain knowledge and strong reasoning ability to support the conditional weak-to-strong correction process. We propose a three-step paradigm based on the objective function in Eqn. 6: (1) Warming up. This stage trains the model in identical response pairs with a warm-up subset, a prelude proven effective in residual correction strategies [11, 12] . We randomly sample a subset of the equal subset D e as the warm-up subset, but set an identical target response for each instance; (2) Equal-preference alignment. Due to the contrastive nature of their learning paradigm, most previous preference alignment works focus on modeling the residuals between response pairs and ignore the equalpreference response pairs. However, equal preferences are common in many scenarios [34, 7] and enclose useful information such as the principle components of preference modeling regarding each objective. Based on these intuitions, we introduce a novel equal-preference alignment step to fine-tune the warmed-up MetaAligner on the equal subset D e ; (3) Contrastive-preference alignment. This stage fine-tunes the MetaAligner on the contrastive preference subset D c , which instructs the model to perform conditional weak-to-strong correction on the specified objectives.During inference, MetaAligner achieves alignment following the sampling process as in Eqn. 4, where unaligned outputs, sampled from the target policy model, are used as the input for conditional weakto-strong correction. With the prompting-based paradigm, the target objectives for MetaAligner also become expandable and generalizable, a key advantage over previous alignment methods [39, 35, 10] .The generalizability is two-fold: Firstly, users can manipulate the target objectives by adjusting combinations of text descriptions in the objective set O. For example, in alignment with objectives 1, 3, and 4, we can flexibly shuffle the corresponding descriptions ⟨d 1 ⟩, ⟨d 3 ⟩, and ⟨d 4 ⟩ as follows: O = ⟨d 3 ⟩; ⟨d 1 ⟩; ⟨d 4 ⟩. Secondly, the prompt-based objectives statement enables flexible adjustment of text descriptions for existing objectives and injections of unseen objectives. Following the last example, we have two unseen alignment objectives 5: ⟨d * 5 ⟩ and 6: ⟨d * 6 ⟩, and an updated text description ⟨d 3 ⟩ for aligned objective 3. We can perform zero-shot alignment on the new objectives by adjusting O as follows: O * = ⟨d 3 ⟩; ⟨d 1 ⟩; ⟨d 4 ⟩; ⟨d * 5 ⟩; ⟨d * 6 ⟩. This simple pattern can theoretically lead to unlimited simultaneous alignment objectives. We expect MetaAligner to make generalizable weak-to-strong corrections under these unseen conditions via its in-context learning ability. This advancement marks a new exploration into generalizable multi-objective preference alignment.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: we release all codes, trained models, and part of the training data to facilitate the reproduction of our results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental settings: 
 Datasets. We transfer the following three alignment datasets into dynamic multi-objective datasets:(1) HH-RLHF [3] : a large-scale dataset with 160K prompts and corresponding response pairs. We follow Yang et al. [35] and use open-sourced reward models on three objectives: "Harmless", "Helpful", and "Humor" to score the responses; (2) UltraFeedback [7] : a multi-aspect alignment dataset with 64K prompts with preferences obtained from GPT-4, including "Instruction following", "Honest", "Truthful", and "Helpful" objectives; (3) IMHI: we create an alignment dataset on the IMHI dataset [34] targeting interpretable mental health analysis. We invite domain experts to label 7.2K response pairs considering 3 objectives: "Correct", "Informative", and "Professional". Figure 2 shows the objective distributions on two datasets. The objectives display balanced overall distributions across objective set sizes, training MetaAligner to adjust targets dynamically. Most objectives also cover considerable proportions in each column category, alleviating label imbalance problems. Models. We train MetaAligner-(1.1B, 7B, 13B) models based on TinyLLaMA-1.1B [37] and LLaMA2-(7B, 13B) [30] foundation models. We utilize MetaAligner to perform multi-objective alignment on the following open-source policy models:LLaMA2-Chat-(7B,13B,70B) [30] , Gemma-instruct-(2B,7B) [29] , and Vicuna-(7B, 13B, 33B) [6] . We also align two advanced close-source foundation models: GPT-3.5-Turbo [21] and Claude-3-Sonnet [2] , where model parameters are inaccessible.Evaluation Metric. On each objective, we quantify the alignment performance of model outputs by comparing their win rates against the ground-truth response provided by the benchmark datasets. Considering the large amounts of test samples, we leverage GPT-4 [1] , a widely utilized evaluation tool in previous works [10, 28, 18] , to perform the judgments. Each target response, ground-truth response, query, and evaluated objectives are provided via prompt engineering. GPT-4 is required to compare and select the response with higher alignment on the specified objective or indicate a tied performance of the two responses.More details about the training process, model cards, dataset statistics, IMHI dataset annotation, and evaluation settings are presented in Appendix D.MetaAligner-(1.1B, 7B, 13B) performance on 3 alignment datasets are shown in Table 2 . According to the results, the MetaAligner models achieve substantial improvement for most objectives and policy models. For example, on UltraFeedback, there is an average of 11.47% advantage for MetaAligner-1.1B on "Honest", 34.39% for MetaAligner-7B, and 43.79% for MetaAligner-13B. These results show the general effectiveness of MetaAligner on various upstream models and the feasibility of plugand-play multi-objective alignment. On the mental health analysis benchmark IMHI, MetaAligner models also show remarkable win rates on all objectives, proving their effectiveness in performing multi-objective alignment in domain knowledge-intense scenarios. We further evaluate MetaAligner on each IMHI sub-task and the results are shown in Appendix E.From the policy model scale perspective, MetaAligner provides successful alignments to open-source models with sizes ranging from 2B to 70B, significantly extending the size of MetaAligner itself. In the extreme case, MetaAligner-1.1B advances the win rates of LLaMA2-Chat-70B outputs, a policy model with 63× more parameters, by an average of 12.19% on HH-RLHF, 13.08% on UltraFeedback, and 13.26% on IMHI. These results prove MetaAligner as a parameter-efficient alignment strategy compared to previous multi-objective alignment methods, where the policy model weights are updated, leading to an inevitable surge of computation resources as policy model sizes grow. MetaAligner also significantly improves performance on close-source LLMs: GPT-3.5-Turbo and Claude-3-Sonnet. These results prove its potential for application in close-source scenarios and effective multi-objective alignment of state-of-the-art policy models. Within most policy model families, we observe a decreasing trend in win-rate advantage as their sizes increase. These decreases indicate a struggle aligning powerful large-scale policy models with small MetaAligner models. Fortunately, MetaAligner's capabilities also show scalability. Increasing the size of its base model leads to a higher win-rate advantage on most policy models. For example, on UltraFeedback, MetaAligner-7B outperforms MetaAligner-1.1B on all 10 policy models, and MetaAligner-13B further surpasses MetaAligner-7B by an average of 12.58%. These observations motivate further explorations in model size-performance balance for MetaAligner. We compare the performance of MetaAligner with MORLHF, MODPO, SFT-based methods, and Aligner. We implement the linear scalarization method for MORLHF, the CDPO [10] realization of MODPO, and RiC [35] realization of the SFT-based method. As Aligner is not suitable for multi-objective alignment, we train Aligner-7B on "Helpful" annotations for HH-RLHF and "IF" for UltraFeedback. We compare these methods on the LLaMA2-Chat-7B policy model. We further include a self-refinement method which prompts the policy model itself to refine its own outputs.We compare self-refinement on the LLaMA2-Chat-70B policy model as it requires strong in-context learning ability from the policy model. The results are presented in Table 3 . Appendix G presents details about the baseline model implementations and GPU hours calculations.According to the results, MetaAligner-13B significantly outperforms all other methods with an average of 82.67% win rate on HH-RLHF and 61.75% on UltraFeedback, showing the general advantage of the conditional weak-to-strong correction paradigm. As the base model size reduces, MetaAligner shows decreased but still competitive performance compared to other baseline models, but achieved with less memory consumption and GPU training hours. Impressively, the MetaAligner-1.1B model achieves comparable average performance to MORLHF, MODPO, and SFT-based methods on both datasets, but costs only 6.37%-48.71% of their GPU training hours, with a 6.36× smaller size than the LLaMA2-Chat-7B policy model. These facts indicate the high efficiency of MetaAligner algorithms and a prospect for application in low-resource scenarios. Compared to previous methods, MetaAligner models can also achieve balanced and stable performances in objective-wise evaluations, including contradictory objectives such as "Harmless" and "Helpful", without requiring explicit hyper-parameter tuning for achieving Pareto optimal solutions [10, 35, 19] .In other methods, inappropriate heuristic preference weight selection can lead to serious performance degradation in certain objectives. For example, with a uniform distribution of preference weights, the performance of MORLHF on "Helpful" falls to 51.2%, a huge gap to other methods. Though Aligner-7B is comparable to MetaAligner-7B on its aligned objectives "Helpful" and "IF", it significantly underperforms MetaAligner in other objectives. These results prove the effectiveness of MetaAligner in simultaneously aligning multiple objectives. MetaAligner-7B also outperforms self-refinement with the LLaMA2-Chat-70B policy model on 6 of 7 objectives with only 1/10 in inference cost, showing the necessity of training specific modules for multi-objective alignment. Ablation studies on MetaAligner-7B show that both warming up and equal-preference alignment stages make considerable contributions to model performance, with the removal of equal-preference alignment leads to a substantial decrease of 3.53% on HH-RLHF and 3.51% on UltraFeedback in average win rates.Figure 3 : Zero-shot alignment on 6 unseen objectives. In the x-axis, "Aligned Obj." denotes the 4 supervised objectives ("⋄" markers), and "+" denotes further addition of an unseen objective ("•" markers). "⋆" denotes the win rates for the unseen objectives before all zero-shot alignments, "-." lines identify win rate fluctuations before alignment, and solid lines identify fluctuations after alignment.In this section, we explore zero-shot preference alignment by utilizing MetaAligner to align with six unseen objectives: "Specific", "Factual", "Readable", "Fair", "Repeat", and "Length" [9] . More details about these objectives are in Appendix C. We randomly select 2,700 queries from the UltraFeedback dataset and re-align the LLaMA2-Chat-70B outputs with these unseen objectives added to the objective set O one-by-one, with 10 aligned objectives in total. Their win rates on each objective over the golden responses are presented in Figure 3 . We have the following conclusions:MetaAligner performs effective zero-shot alignment for unseen objectives. With most MetaAligner models, incorporating an unseen objective into the objective set significantly improves its corresponding win rate. For example, MetaAligner-7B improves by 25.17% on "Specific", 14.5% on "Factual", and 17.5% on "Readable" compared to each of these objectives unaligned. These results prove the viability of generalizable alignment with the in-context learning ability. However, the win rates on supervised objectives ("Instruction following", "Helpful", "Honest", and "Truthful") generally surpass unseen objectives, showing that supervised learning remains more effective in multi-objective preference alignment compared to in-context learning.Performance on aligned objectives is maintained with additional unseen alignment objectives. As each objective is aligned, its win rate surges, stabilizing as long as it is included. On simultaneously aligning 10 objectives, MetaAligner-7B outperforms LLaMA2-Chat-70B outputs by an average of 14.25% on unseen objectives. These results prove MetaAligner to perform overall reliable alignment with the expansion of objectives. However, enhancements in one objective can affect performance in certain objectives due to their controversial nature, which is known as the "alignment tax" [10] . For example, aligning on "Fair" (+Fair) with MetaAligner-(7B, 13B) benefits its win rates, but harms performance on objectives such as "Readable" and "Factual" compared to when "Fair" is unaligned.MetaAligner's generalizability shows scalability. Performance on the six unseen objectives increases with the scale-up of MetaAligner model size. MetaAligner-1.1B provides limited improvement on most unseen objectives, but MetaAligner-7B extends the win rates to an average of 48.5%, and MetaAligner-13B further reaches 61.25%. MetaAligner-13B also more effectively aligns objectives such as "Length", where smaller models perform badly. This scalability is attributed to larger foundation models' growing in-context learning ability, which enables accurate interpretations of the objective descriptions and instructions. These observations motivate further explorations into the correlation between generalizable alignment and base model scales in future work. We evaluate the objective-wise performance of MetaAligner by decoupling the target objectives.We utilize MetaAligner to perform six levels of alignments: unaligned, aligning on each objective ("Instruction following", "Helpful", "Honest", and "Truthful"), and full alignment. We leverage GPT-4 to score the responses ranging from 0 to 10. The results are shown in Figure 4 . Experimental details and more results are shown in Appendix F. We have the following observations:Objective-wise alignment improves performance on the primary target and boosts the performance on other objectives. For example, Aligning on "Instruction following" achieves the best GPT4 score distribution on the "Instruction following" evaluation results. It also significantly increases GPT4 scores on "Helpful", "Honest", and "Truthful" over the unaligned responses. This tendency holds with other policy models and alignment objectives. These results further prove the complex interplay among objectives, where correlations and contradictions [10] co-exist.Full alignment on all objectives provides balanced performance. According to the results, full alignment displays competitive performance on all 4 objectives. Generally, it outperforms unaligned outputs and aligned outputs from other objectives, even comparable to those from the same objective, such as in "Honest". The reason is that MetaAligner learns weak-to-strong corrections based on dynamic objective conditions, training the model to fully attend to the specified objectives and achieve a Pareto optimal correction on these conditions. 
related work: 
 This paper focuses on advancing multi-objective alignment of language models with human values, which is mainly related to two research areas: (1) Large Language Models, including the latest development in close-source AI agents [1, 21, 2] and open-source foundation models [30, 29, 6] . (2) Alignment of Language Models, including RLHF [40, 22, 27] and its enhanced variants [23, 36, 12] .Multi-objective alignment methods include MORLHF [26, 19, 24] , MODPO [39, 10] , and SFT-based methods [35, 10] . A detailed review of related work is in Appendix B. 
discussions: 
 Conclusion. This paper proposed MetaAligner, the first policy-agnostic and generalizable method for multi-objective preference alignment. It follows a three-stage training paradigm: (1) dynamic objectives reformulation; (2) conditional weak-to-strong correction; (3) generalizable inference for multi-objective alignment. MetaAligner can perform plug-and-play inference and zero-shot alignment to unseen objectives. Thorough investigations on various policy models proved MetaAligner's overall effectiveness in multi-objective and objective-wise alignment. Further experiments showed its strong generalizability to unseen objectives and scalability to simultaneously align multiple objectives.Limitations and Future Work. Firstly, stacking MetaAligner module on policy models inevitably leads to increased computational burdens during alignment inference [12] , which affects model deployment, especially for scenarios such as local deployment on mobile devices. Secondly, due to limited resources, we only tested the generalizability of MetaAligner on 6 unseen objectives, which does not provide a clear landscape of its alignment performance on more objectives. In future work, we aim to explore improving MetaAligner in domain-specific alignment scenarios utilizing techniques such as retrieval-augment generation [17] . We will also dive deep into the scalability of MetaAligner to evaluate its impact on alignment performance, including the model scale-performance balance. We will also provide a clearer landscape of their generalizable alignment ability by examining larger base model sizes and aligning on much more unseen objectives (we only expanded to 10 objectives). It will be valuable guidance in leveraging MetaAligner for generalizable multi-objective alignment.We leveraged 3 publicly available datasets to build our dynamic multi-objective datasets: HH-RLHF, UltraFeedback, and IMHI. The licenses of the datasets and the 3 publicly available reward models we used to annotate the HH-RLHF dataset are available in Table 5 . In Sec. 4.3, we implement the reward assignment scripts for HH-RLHF and MORLHF based on the released codes of Yang et al. [35] , which is available at Github. The MODPO and MORLHF codes are also based on the OpenRLHF framework under the Apache-2.0 license. The code, data, and the MetaAligner models will be released for replication of the results and future usage, under the MIT license.In this work, MetaAligner provides an effective and model-agnostic method for generalizable and expandable alignment of LLM outputs with multiple human expectations. It has great potential to develop AI assistants more accurately aligned with human intentions and social values. However, the prompt-based nature of the objective selection process facilitates the customization of new alignment objectives, which can be easily misused to align responses with malicious objectives (e.g. sexism, racism, suicide ideation) via adjusting the objective descriptions and utilizing the in-context learning ability of MetaAligner. These actions can lead to harmful outputs from MetaAligner. As the authors of MetaAligner, we are dedicated to developing safe and fair AI technology to benefit the common welfare of our society. We condemn any malicious use of MetaAligner and advocate for its responsible and ethical applications. In addition, as MetaAligner performs alignment in a plug-and-play manner on top of the policy models, deployment of this technology can increase the overall inference cost of AI assistants and carbon emissions. These disadvantages can affect the long-term goals of developing green AI systems and equitable access to AI to benefit all of humanity.This released codes, data, and MetaAligner models are provided for research only. None of the material constitutes actual diagnosis or advice, and help-seekers should get assistance from professional psychiatrists or clinical practitioners. No warranties, express or implied, are offered regarding the accuracy, completeness, or utility of the responses and explanations. The authors and contributors are not responsible for any errors, omissions, or any consequences arising from the use of the information herein. Users should exercise their own judgment and consult professionals before making any clinical-related decisions. The use of the software and information contained in this paper is entirely at the user's own risk.The collected queries to build our IMHI preference dataset are from the publicly available IMHI dataset [34] , and we strictly follow the privacy protocols and ethical principles to protect user privacy and guarantee that anonymity is properly applied in all the mental health-related texts. In addition, to minimize misuse, all examples provided in our paper are paraphrased and obfuscated utilizing the moderate disguising scheme.In addition, recent studies have indicated LLMs may introduce some potential bias, such as gender gaps. Meanwhile, some incorrect prediction results, inappropriate explanations, and overgeneralization also illustrate the potential risks of current LLMs. Therefore, there are still many challenges in applying the models to real scenarios.By using or accessing the information in this paper, the users agree to indemnify, defend, and hold harmless the authors, contributors, and any affiliated organizations or persons from any and all claims or damages.Large language models (LLMs) have reached approaching-human capabilities across a wide spectrum of tasks related to understanding, generating, and reasoning with natural language [1, 30, 20] . Notable examples include commercially available LLMs like ChatGPT [21] , GPT-4 [1] , and Claude-3 [2] . Due to the high inference cost of these close-source models, research trend in open-source foundation models surges, leading to cutting-edge open-source models like LLaMA2 [30] , Gemma [29] , and Vicuna [6] . Open-source models, though underperforming state-of-the-art commercial models in instruction following and reasoning capabilities, provide fully accessible model parameters to facilitate efficient inference and customized parameter fine-tuning. Despite the advancements of LLMs, recent studies found that they can exhibit problematic behaviors, including the generation of inaccurate information [38, 32] , flattery, and deception, raising concerns about their potential negative impacts and associated risks on society [4] . To address these issues, considerable research has been dedicated to refining LLMs' outputs to better align with human values and preferences [13] .Many studies have delved into enhancing the responses of LLMs in core characteristics of human values like "Helpful", "Harmless", and "Honest". Early efforts are largely centered on Reinforcement Learning from Human Feedback (RLHF) [40, 22, 27] , where the alignment of human values is manifested by maximizing a scalar value obtained from the reward model with a KL-regularization, using RL-algorithms such as PPO [25] . However, PPO faces challenges including inefficiency and instability, driving development in simplified algorithms such as DPO [23] , rank-based learning [36] , and weak-to-strong correction [12] . Nonetheless, human expectations and values include a broad spectrum of heterogeneous and multi-dimensional objectives, where a scalar reward model proves inadequate for aligning LLMs with varied human preferences. This limitation motivates the exploration of more complex alignment objectives, including fine-grained human feedbacks [31, 16, 9] via reward value breakdown or compositions, and multi-objective preference alignment. Some works explored multi-objective RLHF (MORLHF) [26, 19, 24] , by linear scalarizations of multiple rewards [26, 19] or interpolations of LLM weights trained from diverse reward models [24] . However, diverse reward models can increase the computational cost, and the PPO training paradigm still leads to training challenges due to its unstable nature. Recent studies further explore the multi-objective direct preference optimization (MODPO) [39, 10] without the RL paradigm. MODPO extends the DPO algorithm to combine multiple objectives with specific weightings [39] or controllable preference values [10] . Other methods further simplify the paradigm with SFT-based methods [35, 10] , which use customized prompting strategies to explicitly incorporate multiple reward values and optimize with supervised fine-tuning (SFT) or DPO. These methods also facilitated objective-wise controllable generation during inference. Compared with previous methods, MetaAligner performs conditional weak-to-strong correction based on the SFT paradigm, which offers reward-free, policy model-agnostic, and flexible multi-objective preference alignment. The model also effectively aligns unseen objectives, mark-ing the first step towards generalizable multi-objective preference alignment.The text descriptions for all tested objectives in this paper are included in Table 4 . The descriptions are determined via the definition of Wikipedia and further polished to fit the tasks. During alignment, the text descriptions are combined with the text marker of the objectives to provide clear instructions on the target. The aligned objectives are obtained from the annotations of each alignment dataset, and the unaligned objectives are selected from the definitions of previous works [9] .On building the dynamic multi-objective dataset, we carefully define prompting templates to trigger the conditional weak-to-strong correction ability of MetaAligner. A full list of the used templates is shown below. Specifically, for the preference subset of the HH-RLHF dataset [3] , since the model is required to improve the response considering a multi-turn dialogue history, we have:[T (q, y, O, "better")] You are an assistant to human. You will be provided with a context and an answer. Consider the context, then edit the answer to improve it considering these aspects: {O} | Context: {q} | Answer: {y} | Edit: For the equal-preference subset, we have:[T (q, y, O, "equal")] You are an assistant to human. You will be provided with a context and an answer. Consider the context, then edit the answer to make it equal considering these aspects: {O} | Context: {q} | Answer: {y} | Edit:In the UltraFeedback dataset [7] , the model is required to improve the response considering a single query. For the preference subset, we have:[T (q, y, O, "better")] You are an assistant to human. You will be provided with a query and an answer. Consider the query, then edit the answer to improve it considering these aspects: {O} | Query: {q} | Answer: {y} | Edit:For the equal-preference subset, we have:[T (q, y, O, "equal")] You are an assistant to human. You will be provided with a query and an answer. Consider the query, then edit the answer to make it equal considering these aspects: {O} | Query: {q} | Answer: {y} | Edit:We provide an example of the dynamic objectives reformulation algorithm 1 to better illustrate its working mechanism. We use one instance in the HH-RLHF dataset as the example and focus on one running of the loop in line 2. Specifically, we have q:Human: What are some cool countries to visit in Asia? Assistant: Hm, it's difficult to pick just one. Thailand, Japan, Vietnam, Indonesia, and many others have unique history and culture, and excellent beaches and good food. Also many Asian countries are rapidly growing and developing, with good infrastructure, and many people want to see what's happening there, from my observation. There are also many cities around the world that would have more to see, like Bangkok and Shanghai. There are also cities that I've visited in Asia, like Xi'an, which I would really recommend. Human: Okay, which one do you like? Assistant: y 1 : If I was a tourist in Asia, I'd love to visit Kolkata. Maybe Xi'an, because it's just a cool place to visit. y 2 : I like Thailand and Japan, because they have a strong culture that is still being preserved and a unique history.We have the following reward score assignments from the reward models:Response Helpful Harmless Humour yDetails about the training process of MetaAligner and the building process of the 3 datasets are presented in Table 5 . During the performance evaluation process, GPT-4 is leveraged to compare the responses on the corresponding objective. Specifically, we have the aligned test dataset:{q i , g i , R origin i , R aligned i } n i=1, where q denotes the query, g denotes the ground-truth response from the dataset, R origin denotes the original response from the policy model, and R aligned denotes the aligned response from MetaAligner. We use the following prompting template and probe GPT-4 to perform judgment:[E(q, r 1 , r 2 , ⟨r⟩)] You are a skilled evaluator of helpful AI assistants. You will be presented with one query and two different responses to this query. QUERY: {q} | RESPONSE 1: {r 1 } | RESPONSE 2: {r 2 }. Consider the following aspect: {⟨r⟩}, then return the number of the better response. If tied, return 0. You must only return 1, 2, or 0.where r 1 , r 2 are the compared response pairs, and ⟨r⟩ denotes the text description of the target objective. With the above information and the target objective description ⟨r t ⟩, we obtain the win rates using Algorithm 2.TinyLLaMA-1.1B [37] . A compact 1.1B language model pre-trained on around 1 trillion tokens for approximately 3 epochs. Building on the architecture and tokenizer of LLaMA2, TinyLlama leverages various advances contributed by the open-source community (e.g., Flash-Attention), achieving better computational efficiency. Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes. We use TinyLlama-1.1B-Chat-v1.0 as the base model for MetaAligner-1.1B.LLaMA2-(Chat)-(7B, 13B, 70B) [30] . A collection of pre-trained and fine-tuned large language models (LLMs) trained and released by Meta, ranging from 7 billion to 70 billion parameters. The fine-tuned LLMs, called LLaMA2-Chat, are optimized for dialogue use cases. The models outperform other open-source models on most benchmarks. Further human evaluations prove that LLaMA2-Chat also excels in helpfulness and safety. LLaMA2 models are among the most advanced open-source foundation models. We use LLaMA2-(7B, 13B) as base models for MetaAligner-(7B, 13B), and use LLaMA2-Chat-(7B, 13B, 70B) as policy models to evaluate the alignment performances.Vicuna-(7B, 13B, 33B) [6] . Vicuna is a family of open-source chatbots trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT. Preliminary evaluation using GPT-4 as a judge shows Vicuna-13B achieves more than 90% quality of OpenAI ChatGPT and Google Bard while outperforming other models like LLaMA and Stanford Alpaca in more than 90% of cases. We use Vicuna-(7B, 13B)-V1.5 and Vicuna-33B-V1.3 as policy models to evaluate the alignment performances.Gemma-instruct-(2B, 7B) [29] . A family of open-source models based on Google's Gemini models. Gemma models are pretrained on 6T tokens of text, using architectures, data, and training recipes inspired by the Gemini model family. Like Gemini, these models achieve strong generalist capabilities in text domains, alongside state-of-the-art understanding and reasoning skills at scale. Gemma-instruct models are further fine-tuned for dialogue, instruction-following, helpfulness, and safety. Gemma-instruct is developed in two sizes: a 7B version for efficient deployment and development and a 2B version for CPU and on-device applications. We select both models as policy models to evaluate the alignment performances.MentaLLaMA-(7B, 13B, 33B) [34] . MentaLLaMA is the first open-source instruction-following LLM series for interpretable mental health analysis. Based on LLaMA2-(7B, 13B) and Vicuna-33B foundation models, MentaLLaMA is trained on the Interpretable Mental Health Instruction (IMHI) dataset with 105K instruction samples, the first multi-task and multi-source instruction-tuning dataset for interpretable mental health analysis on social media. MentaLLaMA can perform mental health analysis on social media data and generate high-quality explanations for its predictions. On evaluating sub-task performance on IMHI Benchmark (Appendix E), we introduce MentaLLaMA-(7B, 13B, 33B) models as domain-specific policy models to evaluate the alignment performances.GPT-3.5-Turbo [21] . GPT-3.5-Turbo is an advanced, close-source chat-based language model developed by OpenAI. It is a sibling model to InstructGPT, which is trained to follow instructions in a prompt and provide a detailed response. The model is firstly fine-tuned with SFT with conversations in which the model played both sides-the user and an AI assistant. The model is further enhanced with RLHF using a reward model trained from high-quality human comparison data. In our experiments, we use the gpt-3.5-turbo-0125 API provided by OpenAI as a strong policy model to evaluate the alignment performances.Claude-3 [2] . Claude-3 is among the state-of-the-art foundation models for industry benchmarks across reasoning, math, coding, multi-lingual understanding, and vision quality, developed by Anthropic. The model family includes 3 models: (1) Opus, the most capable model; (2) Sonnet, which provides a combination of skills and speed; (3) Haiku, the fastest and least expensive model. All models are multi-modal and demonstrate strong performance across benchmark evaluations. Due to the budget limits, we select claude-3-sonnet-20240229 API provided by Anthropic as a strong policy model to evaluate the alignment performances.GPT-4 [1] . Developed by OpenAI, GPT-4 is a large-scale, multimodal foundation model that can accept image and text inputs and produce text outputs. GPT-4 marks the highest level of achievement in AI industry and exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. We leverage the strong capability of GPT-4 and use it as an oracle to evaluate the large-scale test samples. Considering the high cost of evaluating large-scale test data and our limited budget, we use the cheaper GPT-4-turbo model with the gpt-4-turbo-preview API provided by OpenAI in practice.MetaAligner-(1.1B, 7B, 13B). Our proposed MetaAligner is the first policy-agnostic and generalizable method for multi-objective preference alignment. The models are based on TinyLLaMA and LLaMA2 foundation models. We train MetaAligner models on all 3 model scales for each of the 3 benchmark datasets. Specifically, HH-RLHF-MetaAligner is trained to align the responses of a general daily AI assistant with specified objectives considering multi-turn dialogue contexts. UltraFeedback-MetaAligner is trained to align responses of another general AI assistant considering a single-turn query, but the queries include professional questions such as programming language and history, and the aligned responses are usually more complicated. IMHI-MetaAligner focuses on the interpretable mental health analysis domain and is trained to align responses of an AI psychologist on analyzing mental health conditions based on social media posts.We select 1,200 queries from the IMHI benchmark covering 9 mental health analysis tasks. We obtain 4 responses to each query from 4 different policy models: GPT-4-turbo [1] , GPT-3.5-Turbo [21] , MentaLLaMA-13B [34] , and LLaMA2-Chat-13B [30] , with human annotations on ranking different objectives of the responses. We utilize the above policy models to generate explanations for the same query simultaneously. The annotation protocol is developed through collaborative efforts with 2 domain experts (Ph.D. students majoring in quantitative psychology) and considerations of human evaluation criteria for previous mental health analysis tasks [34, 33] . Specifically, 3 objectives are assessed: (1) Correctness: the explanations should make correct label predictions in Figure 5 : Heatmap of the objective distributions on the dynamic multi-objective dataset built from IMHI. "Pro." and "Info." denote the "Professional" and "Informative" objectives. Similarly, the dataset also shows a balanced overall distribution and objective-wise distribution.the corresponding mental health analysis task; (2) Informativeness: the response should express clear logic and provide consistent evidence; (3) Professionalism: the response should provide evidence with high quality and reliability from the perspective of domain experts. Each aspect is divided into four standards rating from 0 to 3. Higher ratings reflect more satisfactory performance and 3 denotes approaching human performance. Each LLM-generated explanation is assigned a score by 2 domain experts for each corresponding objective, followed by the examination of 1 domain expert. All annotators are PhD students majoring in quantitative psychology.Annotators will be given generated responses from the 4 policy models and need to score and annotate the responses from the following objectives:Correctness. Correctness measures the trustworthiness of the classification results. Annotators should assess whether the classification result is based on facts, has misinformation, and wrong reasoning according to the given post.• 0: Completely unreliable information with factual hallucination (e.g. non-existent symptoms). • 1: Partly reliable information with wrong reasoning based on facts.• 2: Mostly reliable information with non-critical misinformation or wrong reasoning.• 3: Completely reliable information.Informativeness. Whether the text builds from sentence to sentence to a coherent body of information and logic about mental health and supports the classification results. Annotators should assess if the generated explanation gives consistent supporting evidence to its classifications and is well-structured.• 0: Inconsistent with the classification results.• 1: Consistent with the classification results, but with poor readability and several errors. Professionalism. Professionality measures the rationality of the generated explanations by evaluating the evidence that supports the classification results from the psychology perspective. Annotators should assess whether the explanation includes the following specified common diagnosis criteria of depression. To ensure the quality of the annotation scheme, we invite our domain experts to develop a list of common symptoms related to depression and sort these symptoms by criticality. The domain experts consult the Patient Health Questionnaire (PHQ-9) on determining the symptoms and sorting these symptoms on their knowledge. Specifically, the following symptoms are checked (sorted by criticality):• Suicide ideation: Thoughts that you would be better off dead.• Self-harm ideation: Thoughts of hurting yourself in some way.• Feeling down, depressed, or hopeless.• Self-guilt ideation: Feeling bad about yourself -or that you are a failure or have let yourself or your family down. • Symptoms above are classified as with high criticality, and symptoms below are classified as with low criticality.• Feeling tired or having little energy. Little interest or pleasure in doing things.• Poor appetite or overeating.• Trouble falling or staying asleep, or sleeping too much.• Trouble concentrating on things, such as reading the newspaper or watching television.• Moving or speaking so slowly that other people could have noticed. Or the oppositebeing so fidgety or restless that you have been moving around a lot more than usual • Uncontrollable sexual desire or sexual frigidity.• Other symptoms.Based on the above symptoms, the annotators score the professionality of each explanation with the following criteria:• 0: The explanation provides no supportive evidence or symptoms with high criticality are missing in the explanation. • 1: The explanation provides a few supportive evidence, while some symptoms with higher criticality (than provided evidence) are missing. • 2: The explanation provides several supportive evidence, while some symptoms with lower criticality (than provided evidence) are missing. • 3: The explanation provides all related supportive evidence in the post.We stack MetaAligner on different policy models to perform alignment on all 3 objectives: "Correct", "Informative", and "Professional". We include MentaLLaMA-(7B, 13B, 33B) [34] , the first opensource instruction-following LLM series for interpretable mental health analysis into the policy models. Details about the 9 sub-tasks are provided in Table 6 . The overall performance of MetaAligner on the IMHI benchmark and its separation into 9 different sub-tasks are shown in Table 7 .According to the results, the MetaAligner models achieve substantial improvement in overall performance on all 11 policy models, with an average of 26.89% advantage on win rates for MetaAligner-1.1B, 28.01% for MetaAligner-7B, and 36.6% for MetaAligner-13B. These results show the general effectiveness of one MetaAligner on various upstream models and the feasibility of plug-and-play multi-objective alignment. MetaAligner also greatly improves performance on each sub-task. For example, MetaAligner-7B outperforms the unaligned outputs by over 25% on 7 sub-tasks. These results indicate that MetaAligner alignment can be effectively adapted to tasks that require different knowledge and response formats.From the policy model scale perspective, MetaAligner provides successful alignments to models with sizes ranging from 2B to 70B, significantly extending the size of MetaAligner itself. In the extreme case, MetaAligner-1.1B advances the win-rate of LLaMA2-Chat-70B outputs by 21.18%, a P origin ← E(q i , r origin 1 , r origin 2 , ⟨r t ⟩)▷ Prompt for comparing origin and ground-truth response.7:J origin i ← Call-GPT-4(P origin )▷ Call GPT-4 API to perform judgement. P aligned ← E(q i , r aligned 1 , r aligned 2 , ⟨r t ⟩) 15:J aligned ← Call-GPT-4(P aligned ) policy model with 63× more parameters. These results prove MetaAligner as a parameter-efficient alignment strategy compared to previous multi-objective alignment methods, where the policy model weights are updated, leading to an inevitable surge of computation resources as policy model sizes grow. Besides the general-domain foundation models, MetaAligner also improves the performance by an average of 28.32% on MentaLLaMA models, which are fine-tuned on mental health analysis tasks. These results show that MetaAligner can make reasonable corrections on weak responses while maintaining their expertise from domain-specific policy models.We randomly sample 1,200 queries from the UltraFeedback test set and probe the target policy models to provide responses to all queries, which are regarded as unaligned responses. Then MetaAligner is used to align these responses under different objectives, including aligning on each objective (Instruction following, Helpfulness, Honesty, and Truthfulness), and another full alignment on all 4 objectives. During evaluation, GPT-4 is leveraged as a reward model to score these responses considering different objectives. Specifically, the following prompt is developed to obtain the scores:[S(q, a, ⟨r⟩)] You are a skilled evaluator of helpful AI assistants. You will be presented with one query and a response to this query. QUERY: {q} | RESPONSE: {a} Assign a score ranging from 0 to 10 to this response considering the following aspect: {⟨r⟩}. The rating improves as the score rises, where 0 denotes inferior performance and 10 denotes approaching-human performance. You must only return the assigned score.where q and a denotes the target query and response, and r ∈{Instruction following, Helpful, Honest, Truthful} is the target objective for evaluation.More experimental results are presented in Figure 6 . The results are the performance of MetaAligner-7B on Gemma-instruct-7B, LLaMA2-Chat-70B, and GPT-3.5-Turbo outputs from the UltraFeedback test set. According to the results, MetaAligner can perform effective objective-wise alignment on outputs of different policy models, from the small Gemma-7B model to the competitive commercial models GPT-3.5-Turbo. Unlike full alignment which shows weaker performance as the capability of the policy model increases, objective-wise alignment provides stable improvement for different policy models. For example, aligning "Instruction following" leads to over 20% of approachinghuman responses for all policy models, and aligning "Honesty" leads to over 30% of approachinghuman responses for all policy models. The reason is that single-objective alignment doesn't involve the complex interactions between multiple objectives, which facilitates the full realization of MetaAligner's potential to improve the response on the corresponding objective.Existing SFT-based methods [10, 35] for multi-objective preference alignment explicitly incorporate reward values into the query via prompt engineering, which includes a text marker for each alignment objective and their corresponding value numbers in the current response. The model is trained to predict the response given the enhanced query, which enables it to learn a mapping between the reward value and its reflection in the response. During inference, we achieve alignment by assigning a higher reward value to the target objectives. Specifically, we define the following prompting template for HH-RLHF dataset:[F(q, r 1 , r 2 , r 3 ] <Harmlessness>: {r 1 }; <Helpfulness>: {r 2 }; <Humour>: {r 3 } | {q} where q denotes the query, r 1 , r 2 , r 3 denote the corresponding reward values for the current response, ranging from 1 to 5. We define the following prompting template for UltraFeedback dataset:[F(q, r 1 , r 2 , r 3 , r 4 ] <instruction_following>: {r 1 }; <honesty>: {r 2 }; <honesty>: {r 3 }; <helpfulness>: {r 4 } | {q} where q denotes the query, r 1 , r 2 , r 3 , r 4 denote the corresponding reward values for the current response, obtained from existing reward models.During inference, we aim to simultaneously align all objectives with 1 model to enable fair comparisons to MetaAligner. For UltraFeedback, since all rewards range from 1 to 5, we set all values to 5 during inference: r 1 = r 2 = r 3 = r 4 = 5. For HH-RLHF, there are no unified ranges for each objective as all training values are obtained from reward models. Therefore, we set each objective value to its maximum in the training dataset to enable higher alignment. The values are set to: r 1 = 4.19, r 2 = 3.03, r 1 = 1.58.In calculating the GPU hours, we sum the training hours for fine-tuning the LLaMA-Chat-7B policy model for HH-RLHF and UltraFeedback datasets.We implement MODPO by implementing the Controllable Direct Preference Optimization (CDPO) [10] algorithm based on its paper introduction. We bypass the controllable preference SFT stage by utilizing the trained model from SFT-based methods as the foundation model. In the CDPO stage, each query q is accompanied by two responses c 1 and c 2 , where each response is constructed into contrastive pairs. In HH-RLHF, we have the following prompting template:[F(q, c 1 , r 1 1 , r 2 1 , r 3 1 ] <Harmlessness>: {r 1 1 }; <Helpfulness>: {r 2 1 }; <Humour>: {r [F(q, c 2 , r where r j i denotes the reward value for the i-th response on the j-th objective. For HH-RLHF, there are no unified ranges for each objective as all training values are obtained from reward models. Therefore, we set each preference value to its maximum in the training dataset to enable higher alignment. The values are set to: r 1 = 4.19, r 2 = 3.03, r 3 = 1.58. Similarly, on UltraFeedback we have:[F(q, c 1 , r 1 1 , r 2 1 , r 3 1 , r 4 1 ] <instruction_following>: {r 1 1 }; <honesty>: {r For UltraFeedback, since all rewards range from 1 to 5, we set all values to 5 during the DPO training process: r 1 = r 2 = r 3 = r 4 = 5. In determining the preference scores, we use the CDPO learning goal to transform the task into a conditional multi-objective optimization problem. Specifically, the reward value R i for response c i is calculated as follows:EQUATIONwhere ω i represents the weight of the i-th objective and g i is calculated as follows:EQUATIONwhere λ i represents the weight of the controlled objective and p j is a pre-defined preference value for the j-th objective. In our implementation, we set ω i = λ i = 1. We aim to simultaneously align all objectives with 1 model to enable fair comparisons to MetaAligner. For UltraFeedback, since all rewards range from 1 to 5, we set all values to 5 during inference: p 1 = p 2 = p 3 = p 4 = 5. For HH-RLHF, there are no unified ranges for each objective as all training values are obtained from reward models. Therefore, we set each preference value to its maximum in the training dataset to enable higher alignment. The values are set to: p 1 = 4.19, p 2 = 3.03, p 3 = 1.58. After calculation, the response with a higher reward value R i is used as the chosen response, and the other response is used as the rejected response for MODPO training. Specifically, the model is trained via the following loss function:EQUATIONwhere x denotes the query, y w , y l denote the chosen and rejected prompts, c denotes the corresponding condition, π θ and π ref denote the target policy model and the reference model. For implementation, we build our MODPO code based on the OpenRLHF library.In calculating the GPU hours, we include the training hours for tuning the SFT-based policy model using the CDPO algorithm for HH-RLHF and UltraFeedback datasets. We also include the training hours for fine-tuning the original LLaMA2-Chat-7B model for fair comparisons.We use the linear scalarization [26, 19] realization of MORLHF with the KL-divergence regularization. Specifically, the model is trained via the following objective function:EQUATIONwhere ω = [ω 1 , ..., ω N ] s.t.i=1 ω i = 1, ω i ≥ 0 is the heuristic target preference vector, q, y denote the query and the response, R denotes the reward functions for the target objectives. In implementing the MORLHF algorithm, we first train a reward model for each objective based on random samples from 50% of the training data. Following the reward models we used in building the HH-RLHF dataset, we select the GPT2-large as foundation models for the reward model, and optimize the reward models using the following pair-wise loss functions:EQUATIONwhere σ denotes the Sigmoid function, R c and R r denote the reward output of the chosen response and the rejected response, and margin denotes the margin loss for the corresponding response pairs when multiple responses are ranked, such as in UltraFeedback. Secondly, we fine-tune the LLaMA2-Chat-7B policy model with the highest-ranked responses from the HH-RLHF and UltraFeedback datasets to obtain sub-optimal starting points for RLHF. The SFT training process is formalized as follows: argmaxEQUATIONwhere q and y denote the query and its corresponding highest-ranked response. Thirdly, following most works in RLHF, we leverage the PPO algorithm [25] to enable parameterized optimization of the policy model. In linear scalarization, we set ω 1 = ω 2 = ... = ω N = 1 N . For implementation, we build our MORLHF code based on the OpenRLHF library.In calculating the GPU hours, we include the training hours for all reward models in HH-RLHF and UltraFeedback datasets, with a sum of seven reward model training processes. We also include the training hours for fine-tuning the original LLaMA2-Chat-7B model for reaching the sub-optimal starting points. Finally, the PPO training hours for HH-RLHF and UltraFeedback are included in the GPU hours.We include a prompt engineering-based self-refinement method as a baseline method to further demonstrate the effectiveness of MetaAligner. Specifically, this approach involves obtaining an initial response from the policy model and then prompting the same model to further refine its own output. For the second stage, we utilize the same prompting strategies as MetaAligner, which is detailed in Appendix C.2. However, this method often requires aligner models with strong in-context learning capabilities, leading to high inference costs due to larger model sizes or expensive commercial models. Therefore, we select LLaMA2-Chat-70B, a strong policy model as the target policy model for self-refinement. As self-refinement does not involve any training procedures, we do not report its GPU hours as other methods.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: all experimental setting are provided in Appendix D.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We mostly used GPT-4 as judges to rate the responses, a trustworthy but costly AI system. Repeating these evaluation processes or the LLM-based tuning algorithms would be too expensive and time-consuming.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: details about computational resources are provided in Appendix D.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: our research conducted in the paper conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: the broader impacts are discussed in Appendix A.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: the safeguards are discussed in Appendix A.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: all used assets (datasets, source codes) are properly cited and their licenses are listed in Appendix A.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: the proposed models, datasets, and codes are provided with documentation.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: the research didn't involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: the research didn't involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: Paper presents MetaAligner, a multi-objective preference alignment of language models. It is both policy-agnostic (working with various LLMs) and generalizable to unseen objectives.

I lean towards a borderline acceptance as a poster presentation: The paper addresses a relevant problem of multi-objective alignment in a way that could be beneficial, a policy-agnostic, retrained-free approach for diverse preferences is a desirable goal in the field.
Reviewers i5Yu, E7pS, and vbsw all see merit in the work, acknowledging its potential and highlighting aspects like novelty, clarity, and experimental effort. 

Core ideas are worth exploring further despite some weaknesses.

The authors have actively engaged with reviewers, demonstrating a willingness to clarify misunderstandings and addressed concerns.

Reviews: review: paper_summary: paper_summary: This work extends Aligner to multi-objective alignment scenarios. The main contribution is adding a textual description to each sample in the existing preference datasets to indicate the reason for the preference between chosen and rejected samples. The authors found that MetaAligner is more efficient than previously trained multi-objective alignment algorithms. However, this work is incremental, with some unclear details, insufficient evaluation benchmarks, and a lack of baselines.
strength_weakness: strength_weakness: 1. The innovation is incremental, essentially applying Aligner to a new domain and altering training data to change the alignment objectives.

2. The most critical detail, which is how the preference descriptions between each chosen and rejected pair in the existing preference datasets are obtained, is unclear.

3. The authors did not adequately present results on objective datasets such as MATH and HumanEval, nor on mainstream subjective evaluation sets like MT-Bench/AlpacaEval.

4. There is a lack of comparative analysis with previous multi-objective alignment methods, such as "SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling."
questions: questions: 1. How are the objectives in Section 3.1 obtained?
limitations: limitations: N/A
paper_summary: paper_summary: This work proposes MetaAligner, a plug-and-play multi-objective alignment method that can generalize to unseen objectives.
strength_weakness: strength_weakness: 1. In the experimental section, the aligned answers are compared to the ground truth answers in the dataset to calculate the win rate. This lacks a more direct performance comparison, making the argument for performance advantage insufficiently robust. For example, comparing different baselines by win rate against the ground truth answers in the dataset is inadequate to demonstrate the performance advantage over the baselines.
2. There is a lack of performance comparison with prompt engineering-based aligners. Specifically, it is unclear whether similar effects could be achieved by prompting the chat model to modify the answers within the prompts.
questions: questions: 1. I am interested in understanding the authors' rationale for choosing an indirect comparison with the ground truth and how they view the validity of this choice, especially considering that many models, after being aligned through RLHF, are likely to already exhibit strong performance on the objectives of test set.
2. In the scenario where columns contain exactly two of the three attributes: Harmless, Helpful, and Humorous, which corresponds to the position of the second column in Figure 2 HH-RLHF. Let $( x_1, x_2, x_3 )$ denote the probabilities of having two characteristics, specifically including $Harmless$, $Helpful$, and $Humorous$. According to the figure, $( x_1 = 0.57 )$, $( x_2 = 0.53 )$, and $( x_3 = 0.85 )$. Now, there are exactly three possible combinations of choosing two out of the three characteristics: $(Harmless, Helpful)$, $(Helpful, Humorous)$, and $(Harmless, Humorous)$, and these combinations are mutually exclusive. Let the probabilities of these three events be $( a_1, a_2, a_3 )$ respectively, such that $( a_1 + a_2 + a_3 = 1 )$. The event `contains two characteristics, with one of them being Harmless` can be described as the disjoint union of the events `only contains (Harmless, Helpful)` and `only contains (Harmless, Humorous)`. Therefore, $( x_1 = a_1 + a_3 )$. Similarly, we have $( x_2 = a_1 + a_2 )$ and $( x_3 = a_2 + a_3 )$. Summing both sides of these three equations, we obtain $( 1.95 = 0.57 + 0.53 + 0.85 = x_1 + x_2 + x_3 = 2(a_1 + a_2 + a_3) = 2 \times 1 = 2 )$. I am sincerely inquiring whether my understanding of Figure 2 is mistaken.
limitations: limitations: The article provides a thorough discussion of its limitations in terms of deployment and the number of objectives to be selected.
paper_summary: paper_summary: The author proposes Meta-Objective Aligner (MetaAligner), the first policy-agnostic and generalizable method for multi-objective preference alignment.
strength_weakness: strength_weakness: 1. I am not entirely clear about the difference between this paper and Aligner. The author mentions:
> As shown, conditional weak-to-strong correction of MetaAligner extends Aligner [12] to multi-objective alignment scenarios, which are not directly solvable by Aligner itself.

I do not fully understand the distinction between this paper and Aligner. After reading the Aligner paper, I noticed that the authors effectively expand on multi-objective alignment. However, the introduction seems to lack a description of Aligner, making it difficult for readers to understand.

2. I did not understand what the author is trying to convey with Figure 2. Can the author explain the meaning of Figure 2? The elements it aims to express are very complex. Could there be a more streamlined figure, similar to Figure 1 in the Aligner paper? Is the extension to multi-objective alignment at the methodological level or the dataset level?
3. In the experimental section, how did the authors generate an effective correction dataset based on HH-RLHF, UltraFeedback, and IMHI?
4. After reading the Aligner paper, I observed that it conducted experiments on the three dimensions of "Helpful," "Harmless," and "Honest" as described by the authors in line 42. I hope the authors can add comparative experiments with Aligner in the ablation study section. From a reviewer's perspective, the contribution of this paper to LLM alignment is undeniable, but I am particularly concerned about the effective improvement and extension of MetaAligner compared to Aligner.

My questions are all included in the Weaknesses section, with point 4 being my primary concern. I hope the authors can effectively enhance the ablation study section. I believe that algorithms like MetaAligner and Aligner are more efficient compared to RLHF and DPO. I look forward to the authors' response and will consider raising the score further.
questions: questions: see above.
limitations: limitations: N/A.
paper_summary: paper_summary: Multi-objective alignment of language models is a significant topic for LLM community, while many prior work are either costly-to-compute or policy-dependent, restricting the further depolyment. This work is aimed at providing a retraining-free and policy-agnostic approach for multi-objective alignment, namely *MetaAligner*.

The framework is composed of 3 stages:
- Data reorganization from multi-objective datasets, getting $D_e, D_p$;
- Supervised model training:
   - Warming up on a randomly downsampled dataset of $D_e$;
   - Finetune on $D_e$, as a equal-preference alignment step;
   - Contrastive training on $D_p$.
- Inference with prompting for even unseen objectives.

Experiments are conducted on three commonly-used datasets, and the improvements are reported as prominent. Furthermore, the authors claim that they can achieve generalizable alignment to unseen objectives, through prompting.
strength_weakness: strength_weakness: *Major*
- The whole framework is complicated and a bit engineering. There is no ablation study on each components. For example, it is better to show the performance if discarding the $D_e$ dataset, which is claimed as novel.
- In Table3, the comparison of *MetaAligner* with other baselines is somehow unfair, since *MetaAligner* utilizes at least double parameters of *MORLHF*, *MODPO*, and *RiC* (the 1.1B version performs worse than baselines, so the reviewer is talking about >=7B version).
- The proposed method is not able to cater to different users' taste, because it can only be aligned to multiple objectives while no weights on them are introduced. But many works[1][2][3][4][5] are able to do this, making the contribution less significant.

*Minor*
- this work is a bit marginal:
  - The reorganization process is just preparing the prompting format, which is trivial.
  - The SFT training is a common practice in the community[1][2][3]. The only difference is the design of a detached aligner.
  - The inference stage is simply prompting engineering, lacking guarantees.
- The *MetaAligner* is not a light-weight approach, as the aligner model is still large, making it hard to be deployed.

[1] Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment. arXiv preprint arXiv:2402.10207.

[2] Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment. arXiv preprint arXiv:2402.19085.

[3] Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards. arXiv preprint arXiv:2402.18571.

[4] Decoding-Time Language Model Alignment with Multiple Objectives. arXiv preprint arXiv:2406.18853

[5] Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. arXiv preprint arXiv:2310.11564.
questions: questions: - In Figure 3, why would the performance on some unseen objectives drop after adding more objectives? For example, "repeat" and "readable" for *MetaAligner-7B*.
- How much efforts has the author spent on tuning hyper-parameters of *MetaAligner*?
- How does the author set the preference weights for *MORLHF*, *MODPO*, and *RiC*? Is that fair?
limitations: limitations: The limitations are well discussed in Section A.2.

Comments: title: Response to Reviewer XHXG's Comments
reply to the author
Reminder on Rebuttal and Discussions
Reminder on Rebuttal and Discussions
Response to Reviewer i5Yu's Comments
Response to Reviewer vbsw's Comments
comment: The authors sincerely thank the reviewer for considering our response. Here we provide brief responses to the reviewer's new comments. We hope the reviewer can further consider these points:

**Response to Comment 1**: As discussed in Sec. 3.3, the authors mentioned "This simple pattern can theoretically lead to unlimited simultaneous alignment objectives". It is worth noting that according to our detailed experimental results in Sec. 4.4, training on 4 objectives already effectively expands alignment performance on up to 10 objectives, which significantly supports the above statement. We believe they are strong evidence of the generalizability of MetaAligner.

**Response to Comment 2**: Please note that though we incorporated the training method of Aligner, all modules are centered on achieving the two capabilities. 
1) instance-level alternation of the alignment objectives without re-training;
2) generalizable multi-objective alignment to align unseen objectives in a zero-shot manner.

We designed dynamic objectives reformulation (Sec. 3.1) to build training data with various combinations of target objectives which are used to train MetaAligner in a conditional weak-to-strong correction manner (Sec. 3.2). With the dynamic multi-objective capability triggered by the above modules, We further extend it to generalizable inference (Sec. 3.3) with the in-context learning ability of MetaAligner. 

**Response to Comment 3**: The authors would like to clarify that by mentioning our experiments on the mental health analysis domain, we didn't mean to list it as a novelty, but as an empirical contribution that Aligner or other alignment methods didn't consider. 

**Response to Comment 4**: The authors would like to clarify that testing on UltraFeedback and HH-RLHF also thoroughly evaluates the basic LLM capabilities on alignment. 
1) The testing data covers most categories reflected in benchmarks such as MT-Bench, including Writing, Reasoning, QA, Extraction, etc. 
2) Our testing data has much more samples than the mentioned benchmarks, which allows a thorough evaluation of the capabilities of the LLMs.
The authors thank the reviewer for reading and considering our rebuttal. We are glad that our response thoroughly addressed most of the reviewer's concerns.

We hope you will soon recover from COVID-19!
I got COVID-19 and was unable to respond to the author in time. I sincerely apologize.

Thank you for your comprehensive response. Most of my concerns have been thoroughly addressed. I choose to keep the score unchanged.
Thank you for your rebuttal, but I plan to keep my score because:

Regarding novelty:

1. The authors have not demonstrated that MetaAligner can handle unlimited simultaneous alignment objectives.
2. Generalizing Aligner to multi-objective alignment is an intuitive idea.
3. The use of different experimental datasets cannot be considered a measure of novelty.

Regarding the experimental datasets, although the authors argue that benchmarks like MATH, HumanEval, MT-Bench, etc., are not suitable for the multi-objective alignment scenario, please note that the goal of multi-objective alignment is still to achieve alignment. Therefore, ensuring that the basic LLM capabilities do not decline is a fundamental requirement.
Dear Reviewer XHXG,

This is a gentle reminder that we have provided a detailed response to your concerns. Please note that the discussion period is ending soon. We'd appreciate it if you could find time to check the response and further consider your evaluations of our paper. We look forward to our further discussions based on these new clarifications.

Best wishes,

On behalf of the authors of Paper 5185
Dear Reviewer E7pS,

This is a gentle reminder that we have provided a detailed response to your concerns. Please note that the discussion period is ending soon. We'd appreciate it if you could find time to check the response and further consider your evaluations of our paper. We look forward to our further discussions based on these new clarifications.

Best wishes,

On behalf of the authors of Paper 5185
The authors thank the reviewer for considering our response and raising the score. We are glad that our response addresses most of the reviewer's concerns. We will review and modify the figures and data in the future version of our paper. 

Regarding Figure 1, we will simplify the figure to clarify the model structure. We refer to our response to Weakness 2 of reviewer E7pS for a detailed outline of the modification plan.
The reviewer deeply appreciates the responses provided by the authors, which have addressed most of my concerns.
Regarding the issue with data points, I recommend that the authors conduct a more thorough review of the figures and data presented in the paper. As a side note, the Figure 1 is somewhat complex.
Thank you for your clarification!
The authors thank the reviewer for reading and considering our rebuttal. We are glad that our response addresses most of the reviewer's concerns. Here we provide brief responses to the reviewer's new questions. We hope the reviewer can further consider these points:

**Response to Concern  1**: We agree that weight control can be useful in scenarios such as conflicting objectives. Though without weight control, MetaAligner archives other more useful features as outlined in our response to weakness 3. It is worth noting that instead of requiring an explicit weighting action, MetaAligner handles conflicting objectives more simply by automatically considering the weight specification process during alignment.

**Response to Concern 2**: We are sorry for the inaccuracy in our statement in this sentence. As discussed in Sec. 3.3, the authors mean to say "This simple pattern can **theoretically** lead to unlimited simultaneous alignment objectives". We agree that the actual performance is limited by data and training time. But It is worth noting that according to our experimental results in Sec. 4.4, training on 4 objectives already effectively expands alignment performance on up to 10 objectives, which significantly supports the above statement.

**Response to Question 1**: We thank the reviewer for pointing out the MODPO implementation problem. Please note that based on the authors' knowledge, the implementation of MODPO wasn't fully released until their recent acceptance to ACL 2024, which was after when most of this work was finished. We tried to implement the algorithm but many details were vague in the original paper. Therefore, we selected an easier-implemented algorithm, the CDPO. We agree that comparing with the MODPO algorithm is important, and will include the new results in the future version of our paper.
The reviewer thanks the authors for the detailed responses to my questions. On reading the responses, most of my concerns have resolved, while I partially disagree with some arguments:
- "MetaAligner’s weight-free specification of desired objectives and zero-shot alignment on unseen objectives are more useful features that no other methods achieve." The reviewer thinks flexible control on weights is significant, especially when involving conflicting objectives.
- "MetaAligner achieves unlimited simultaneous alignment objectives at the instance level". The number of simultaneously aligned objectives can not be unlimited, since it should be constrained by #data and #training time.

I will keep my ratings.

*Another question:*

Why the authors implemented another version of MODPO, instead of using https://github.com/ZHZisZZ/modpo?