Panacea: Pareto Alignment via Preference Adaptation for LLMs
Authors: Yifan Zhong
Chengdong Ma
Xiaoyuan Zhang
Ziran Yang
Haojun Chen
Qingfu Zhang
Siyuan Qi
Yaodong Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
alignment
multi-dimensional preference optimization
RLHF
Primary area: natural_language_processing
TL;DR: This paper proposes Panacea, a simple yet effective method that achieves Pareto alignment with diverse human preferences with a single model.

Abstract: Current methods for large language model alignment typically use scalar human preference labels. However, this convention tends to oversimplify the multi-dimensional and heterogeneous nature of human preferences, leading to reduced expressivity and even misalignment. This paper presents Panacea, an innovative approach that reframes alignment as a multi-dimensional preference optimization problem. Panacea trains a single model capable of adapting online and Pareto-optimally to diverse sets of preferences without the need for further tuning. A major challenge here is using a low-dimensional preference vector to guide the model's behavior, despite it being governed by an overwhelmingly large number of parameters. To address this, Panacea is designed to use singular value decomposition (SVD)-based low-rank adaptation, which allows the preference vector to be simply injected online as singular values. Theoretically, we prove that Panacea recovers the entire Pareto front with common loss aggregation methods under mild conditions. Moreover, our experiments demonstrate, for the first time, the feasibility of aligning a single LLM to represent an exponentially vast spectrum of human preferences through various optimization methods. Our work marks a step forward in effectively and efficiently aligning models to diverse and intricate human preferences in a controllable and Pareto-optimal manner.

Panacea: Pareto Alignment via Preference Adaptation for LLMs
Yifan Zhong 2-, Chengdong Ma
1.2 Xiaoyuan Zhang Ziran Yang', Haojun Chen Qingfu Zhang', Siyuan Qi², Yaodong Yang'
Abstract
Current methods for large language model alignment typically use scalar hu- man preference labels. However, this convention tends to oversimplify the multi- dimensional and heterogeneous nature of human preferences, leading to reduced expressivity and even misalignment. This paper presents Panacea, an innovative approach that reframes alignment as multi-dimensional preference optimization problem. Panacea trains a single model capable of adapting online and Pareto- optimally to diverse sets of preferences without the need for further tuning. A major challenge here is using a low-dimensional preference vector to guide the model's behavior, despite it being governed by an overwhelmingly large number of parameters. To address this, Panacea designed to use singular value decom- position (SVD)-based low-rank adaptation, which allows the preference vector be simply injected online as singular values. Theoretically, we prove that Panacea recovers the entire Pareto front with common loss aggregation methods under mild conditions. Moreover, our experiments demonstrate, for the first time, the feasibility of aligning a single I.IM to represent exponentially vast spectrum of human preferences through various optimization methods. Our work marks a step forward in effectively and efficiently aligning models diverse and intricate human preferences in a controllable and Pareto-optimal manner.
Introduction
Al alignment aims to ensure AI systems align with human intentions, and there has been notable
progress in this area, especially for large language models (LLMs) [29] 12, 30 2]. The prevailing approach for LLM alignment involves curating dataset {(x, 2/, 3/2, 3/222)} =)}. where each prompt is associated with a pair of responses (11, y2) and scalar label E {0. 1 that indicates if 1/2 is "better" response. These labels typically generated based detailed guidelines that encompass various criteria, reflecting multiple dimensions E {1...... . of human preferences (e.g., helpfulness, harmlessness, conciseness, humor, formality). Pre-trained models subsequently further optimized on this dataset using methods including reinforcement learning, supervised learning, or game- theoretical approaches [27] 42
47 40). However, this single-objective alignment methodology may not fully capture the complexity of real-world scenarios for two reasons (Figure TT.
First, this method can lead to inconsistency and ambiguity in data labels. Human labelers assign scalar labels = by implicitly evaluating responses across every dimension with different preference weights to i, and reaching final judgment. These differences often result in conflicting labels, causing misalignment or learning failures (Appendix [B), substantiated by the low average label agreement reported in (5). Second, optimizing single objective leads to only one model that attempts fit the potentially conflicting labeling preferences, i.e., the helpfulness helpfulness-harmlessneee harmlessness dilemma. This
'Equal contribution. 'Institute for Artificial Intelligence, Peking University. State Key Laboratory of General Artificial Intelligence. 'Department of Computer Science, City University of Hong Kong. 'Yuanpei College, Peking University. Correspondence Yaodong Yang <yaodong. @ edu..en
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Preference dimensiion (egg. helpfaly
Preference dimeusion (eg hammleas)
Synthetic proforence botter 1
Single-objective aligament
Our multi-dimensional aligament
Respusees to latel
Differect liteles
Syutheti preference
mettod misaignad, confistinng singuiar
Preter
Prefer
Prefecabee nepenee
Solutoe
requosse
reppose
aiggod
Beageese
04
0.6
Froit
Provegr
If
Derriinted whltion
Recernes (
31 Reward
Reward
Figure 1: Comparison of the predominant singlc-objective alignment and our multi-dimennionn. alignment. For the two responses to prompt, labelers agree the preferable one in each preference dimension, but conflict when assigning a synthesized scalar label denoting which "better". This arises due the inherently different preference weights held by labelers, common case in reality. Performing single-objective optimization on the potentially conflicting scalar-label dataset (left) could lead to dominated solution and misalignment. By contrast, multi-dimensional preference optimization (right) on the consistent multi-dimensional dataset and learns the entire Pareto front (PF), thereby aligning with diverse and complex human preferences.
method, Panacea, leverages
single model may not cover the full spectrum of human preferences across all dimensions, thereby exacerbating biases against underrepresented groups and failing to meet diverse user needs.
To address these challenges, we formulate the alignment as a lulti-diieensonnll preference optimiza- tion (MDPO) problem. By explicitly curating data for each dimension, we enhance data consistency and simplify the labeling process, thereby overcoming the first limitation.
Upon the obtained dataset, our goal is to concurrently optimize across all dimensions. However, this often infeasible due to potential confficts among preferences (e.g.. helpfulness harmlessness in response to hazardous user requests). Therefore, we aim for Pareto-optimality [39], which means finding solutions where no preference dimension can be made better off without making another worse off. However, many Pareto-optimal solutions might exist. Instead of just learning one such solution, we focus on learning the entire set of Pareto-optimal solutions. To achieve this, capable of recovering any Pareto-optimal solution by inputting the appropriate preference vector.
single model
In this paper, we propose Panacea (Pareto alignment via preference adaptation), simple yet effective method that: 1) learns the entire Pareto-optimal solution set for all possible preferences with a single model, and 2) infers Pareto-optimal responses online by simply injecting any preference vector into the model. Our method, providing a comprehensive representation of human preferences, effectively caters to diverse user needs, thus mitigating the second limitation (Figure I.
key challenge lies in how to utilize a now-dimensional preference vector to control the model's behavior. Our core insight is that, similar to the crucial role of the preference vector in shaping the Pareto solution, singular values are pivotal in defining the model's 's fundamental behavior in singular value decomposition (SVD)-based low-rank adaptation (LoRA) 57). To address the above challenge, we incorporate the preference vector into the singular values within each SVD- LoRA layer. We then scale it using learnable factor to align with the magnitude of other singular values. The model is trained end-to-end using a joint objective function aggregated according to the preference vector. The flexibility of Panacea enables seamless compatibility with various preference optimization procedures, e,g., supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF) 42), and direct preference optimization (DPO) 44, and diverse methods for loss aggregation, e.g., linear scalarization (LS) [ [Section 4.7.5] and weighted Tchebycheff (Tche) [Secton 3.4]. Through theoretical analysis, confirm that Panacea can effectively capture the entire Pareto front (PF) under practical conditions. This finding provides a solid rationale for training single Pareto set model to learn all Pareto optimal solutions across the entire preference space.
In our experiments, we assess the effectiveness and scalability of Panacea on several significant and challenging preference alignment problems with up 10 dimensions, where the Pareto set

---

cardinality grows exponentially with the number of dimensions. considerably surpassing the scope of current research. Panacea consistently outperforms baseline methods, producing superior, uniformly distributed, and convex fronts in accordance with the theory. Quantitative metrics highlight its substantial advantages, demonstrating an order-of-mapee man-iiudd improvement. Notably, Panacea exhibits no performance saturation even on the ten-dimensional problem, indicating its extensive potential. For the first time, we show the possibility of aligning a single model with exponentially many heterogeneous preferences, opening up promising avenue for LLM alignment.
This paper makes three main contributions. First, we identify the fundamental limitations of the predominant realar-label, sinle-objeectve alignment paradigm, and propose to reframe alignment as a multi- muulti-dimensionll preference optimization problem. Second, we design Panacea, simple yet effective method that learns one single model that can online and Pareto-optimall preferences, without the need for further tuning. Third, we provide theoretical supports and empirical validations to demonstrate the Pareto optimality, scalability, efficiency, and simplicity of Panacea, thereby satisfying the urgent need for Pareto alignment to diverse human preferences.
adapt to any set of
Related Work
Pareto Set Learning. Different from previous classical multi-objective optimization (MOO) methods [59]
38 [66] that use finite set of solutions (referred to as "particles") to approximate the entire Pareto set, Pareto set learning (PSL) 41 36 58 aims to use single model recover the complete Pareto set/front. The advantage of PSL is that it can store infinite number of Pareto solutions within a model. This allows users to specify their own preferences, and the model can dynamically output a particular Pareto solution in real-time according to those preferences. Typical applications of PSL includes multiobjective industrial design problems [58 37,. reinforcement learning [8 54, [24],
335
text-to-image generalization [33], and drug design [25] 611. While there have been some studies on PSL involving deep neural networks, these models are considerably smaller compared to LLMs. Learning continuous policies that represent different trade-offss for LLMs remains unsolved.
Multi-Dimensional Preference Optimization. Existing research primarily treats AI alignment as single-objective optimization problem with scalar labels [42 55
44 40 47]. often neglecting the complexity of diverse human preferences. We provide an in-depth analysis of this limitation in Appendix B which is subsequently substantiated by MaxMinnRLHITP result of "impossibility of alignment"
17
13 after Panacca first came out. To address this crucial gap, one recent attempt is AlignDiff [19] which trains an strribute-conditionee diffusion model to conduct preference alignment
planning in the RL settings. In the realm LLMs, there are some contemporary works on this topic 660 26 18 51)53, where the most relevant one Rewarded Soups (RS) (45) adopts multi-policy strategy. It learns a model for each preference dimension and interpolates their parameters lincarly to generate customized model. However, its simple design also constitutes its drawback. Since RS does not see any intermediate preference vectors during training, ensuring the optimality and alignment of the interpolated model poses challenge. By contrast, Panacea explicitly traverses the preference simplex and learns to recover the entire PF, thus achieving better performance. It the first fundamentally PSL approach in LLM for multi-dimensional preference alignment, with theoretical guarantees of Pareto optimality under mild conditions.
3 Problem Formulation
Human preference is inherently multi-dimensional. In the case of LLM alignment, a preference dimension refers single, self-consistent, and independent aspect of evaluating LLM responses, such as helpfulness, harmlessness, humor, etc.. We formulate the multi-dimensional preference optimization (MDPO) problem with m dimensions as:
Jm max J (TT) = (Ji(me), J2(Te)) J. (ma)). DED
(I)
where no E II is policy, iee. an LLM, and 0 is its trainable parameters (decision variable), IT is the policy space, θ is the parameter space, and J,, = T,.... . denotes a performance measure of dimension i, such as SFT objective (Te), RLHF objective ((T)), and DPO objective
3

---

RLHF/ DPO SET... update
J-th SVINCORA Aper
dataset
Protegt bwichh
LLM
Protrainc werittn IP
Smaplhel fross simples
Learning
2
J1
+
J2
+
Am
Im
Response batch
Preferance vector: [0.0,
Steecifed by heman
[0.2,0.8]
Inference
Prompp
Response
(0.5.0.5)
[0.0,022]
[.00..0]
aligmment
Figure 2: Panacea embeds the preference vector into singular values of each SVD-LoRA layer and scales it with learnable factors match the magnitudes. During learning, for each data batch, randomly sample a preference vector from the preference simplex and train the embedded model with various optimization procedures and loss aggregation methods. In the inference stage, the model adapts online to the user-specified preference vector and exhibits Pareto alignment in its responses.
(Do) detailed in the following equations,
( E(r.))D, [log =o(ul>>)]
(2)
(ALHI;((T)) ) = ERED [Eynnot--l)) [ra(xr, v)] - 3T... [=o(||=)]]] [=]]==]]]]
(3)
To (y (x:
3 log To (21l (z)
JDPO, = (2..w..))> log
Bog
(4)
Tret (yw (x)
ree (11l
Notice that D., represent the data and reward model for dimension respectively. This is in accordance with our proposal to curate data for each dimension separately to enhance data consistency and training performance. Throughout this paper, we use bold letters to denote vectors or matrices (c.g. A). Very often, there does not exist single solution 0 that performs optimally on all dimensions due to their conflicts. Instead, there exists set of Pareto optimal solutions, which have unique trade-offs among all dimensions.
We say solution (fa) dominates ((6), denoted as > Je( and there exists least one index Based on this, Pareto optimality is defined as:
J (Tgta)) ) > (TTO)) ), if for all i E [m]] J.( Tgi=)
j E [m] such that (T(()))) > (T 120 39
Definition 3.1 (Pareto optimality). We call a solution 0* Pareto optimal if no other solution 0' E dominates 0" The set of all Pareto optimal solutions is called the Pareto set (PS); while its image set in the objective space is called the Pareto front (PF), T. A solution 0" is considered weakly Pareto optimal if other solution 0' can strictly dominate it, that is, (((rr))) ((( for all [m].
Human's 's trade-offs among all dimensions are quantified as preference vector, X (Ai.... Am). where X E Am. X; 2 0, and Σm A = Here, X; represents the weight for preference dimension i (called preference weight), and Am is the preference simplex. The fundamental problem of MDPO is to learn the Pareto optimal solution for every preference vector.
4 Panacea: Pareto Alignment via Preference Adaptation
To solve the MDPO problem, our goal learn single model capable of representing the entire Pareto-optimal solution set. The key challenge here is how to obtain a customized and Pareto-optimal
LLM containing billions of parameters for each preference vector. Naive solutions such as directly generating full LLM for each vector using hypernetwork is infeasible due to the vast number of parameters. To avoid this, we consider LoRA (2), parameter-efficient fine-tuning method, which, for each layer, freezes the original weights Wo and only learns pairs of rank decomposition matrices

---

A, B for adaptation. According to LoRA, the final weight W is obtained by W = Wo + BA. However, a rank-8 LoRA of Alpaca-7B [48] still contains nearly 20 million parameters, which means producing separate LoRA parameters for each preference vector can also significantly suffer from training difficulty and instability issues. We thus explore an alternative approach inspired by AdaLoRA [57]. This method employs singular value decomposition (SVD))based LoRA and learns the left singular matrix U, diagonal matrix Σ (representing singular values), and right singular matrix V. Moreover, U and V are subject to orthogonality regularization.
(5) which hereafter we call SVD-LoRA. By extracting singular values E of incremental matrices, SVD- LoRA captures the core features of adaptation in a few parameters. More importantly, the singular values provide an interface to fundamentally influence model behavior.
W Wo + UEVT,
Our key insight is that the preference vector can be embedded as singular values in every layer to achieve decisive and continuous control of model adaptation. Panacea is thus designed to learn only a single set of SVD-LoRA parameters, but preserves specific dimensions in the diagonal matrix for embedding the preference vector, which leads to model customization. Concretely, for layer I, we preserve l singular values for learning general and snfference-annosiee features and concatenate them with the dimensional preference vector À multiplied by per-weight- matrix learnable scaling factor s Therefore, for each weight matrix W E TR xmi, we have W/ E Rd x left singular matrix U' = -lui..... uky ue+1....., U E RR4 gg4)((+m) diagonal matrix E' = diag(o',.... ..,, of, ok, s 'i....., s' Am) R [((++m)((+-m), and right singular matrix V
[uf...., u u'+1..... on E RT (k++)) The scaling factor is important since we observe that the preference-agnsstee singular values commonly range from 10-2 to 10-5 in our experiment scenarios, which could be significantly smaller than preference weights, and their magnitudes differ across weight matrices, so both no scaling and a unified scaling are suboptimal. Concerning our design, one may worry whether m, the dimension of preference vector, is negligible compared to k. Preliminary experiments show that Alpaca-7B fine-tuned by SVD-LoRA A with a rank as low as 4 performs comparably to the full-parameter fine-tuning counterpart. Since the rank of the same magnitude as the number of human preference dimensions, this suggests the feasibility of Panacea.
During each training iteration, we randomly sample a preference vector from the preference simplex Am. embed it into all weight matrices, and obtain the preference embedded model T.. We then compute an aggregated objective function of Te, across all preference dimensions according to X, by synthesizing per-dimension objective functions with loss aggregation methods. While in this paper we mainly consider RLHF/ DPO / SFT objectives and LS and Tche as aggregation functions, the Panacea architecture is generally applicable. The LS function [Secionn 4.7.5) is given by
L$ (0) = mmxx Σ XcJI(TTO)
max
0
0
and the Tche function is defined as,
Tehe (0) = max min ((((((TT) 21),
max 9X
0
where is a vector (c.g., ideal vector) such that > (((T)), 4O E V, E [m]. These loss aggregation functions allow Panacea obtain solutions corresponding the preference vector.
With respect to the aggregated objective, trainable parameters for each weight matrix W', including U². V (of,.... of) s', are then updated via gradient descent. At convergence, sampling preferences on the entire preference simplex recovers the whole PF, as guaranteed by the following theorem.
Theorem 4.1. Panacea recovers the entire Pareto front for both LS and Tche aggregation functions (Equations (6) and under the following two assumptions: 1. Panacea with SVD-LoRA has sufficient representation capability for all preferences X E Am- Specifically, for any preference vector X, the policy Te can optimize the corresponding aggregation functions (Equations (6) and 7 to their maximum values. 2. For specific preference vector X, the LLM policy space formed by all 'a, can represent all categorical oulput distributions of responses. By optimizing the Panacea objective function Ex-UUnit(Am.) [9X 28E (0), where 288 could be g or Tehe the optimal policy found by Panacea can recover the entire Pareto front for almost every preference.
For proof, see Appendix As the two assumptions are easy to satisfy, this theorem confirms the Parcto-optimality of Panacca. Panacea also achieves fine- fine-grained control of model behavior through
5

---

Table 1: This table compares algorithm performance using MOO metrics across all experiment evaluations. An upward arrow (1) means a larger value for this metric is better, whereas a downward arrow (4) indicates the opposite. When in single cell two values are reported for Panacea, they indicate the results using LS and Tche respectively; otherwise, LS is used. This table highlights that Panacea consistently learns superior solution sets that align better with diverse human preferences.
Hypervolume
Inner product
Sparsity
Spucing
Experiment
Model
Optim.
RS
Panacen
RS
Panacea
RS
Panacen
RS
Panacea
Llumal-ft
RLHF
517.28
915.04
11:26
14.27
7302.91
2758.59
329.53
207.19
HH
Llamal
Llama2-ft
RLHF
DPO
519.38
0.319
0.822/ 0.317
840. 45
0,632
8.59
639 D 637
14.68
890.4
0.48
0.3 0.955
5332 88
90.38
2.89
2.61 /3.25
275.7
Lluma2-4t ft
DPO
0.318
0.337 0.334
0.641
0.653 / 0.852
0.73
0.36 0.53
3.24
3.12/3.7
HHC
Lluma2-ft ft Ilama2-ft
RLHF
13519
17097
5.37
9.19
211.96
48.44
65.15
65.78
DPO
0.171
0.177
0.64
0.65
0.1
0.06
1.98
2.45
Chat 3-dim
Ianrat-nnsrret Llama3-Instreet Jlama3-Instra Llana3-Insrrett
SFT
0.50
0.58
0.88
0.04
8.37
2:13
Chat 4-dim
SFT
0.11
0.38
-0:65
-0.43
0.25
0.02
5.06
2.17
Chat 5-dim
SFT
0.088
0.33
-0.66
-0.42
0.14
0.02
4.91
2.28
10-dim
SPT 0.01
0.12
0.68
0.47
0.03
0.01
3.94
2.19
preference embedding, making it suitable solution MDPO. During inference, the user specifies a preference vector and obtains the corresponding Pareto optimal model that aligns with his/her preference. We present a visual illustration of Panacea in Figure and its pseudocode in Appendix [
Compared with prior work, Panacea is the first fundamentally PSL approach towards multi- dimensional preference alignment. It only needs learn and maintain one model to represent the PF, which is more computationally efficient than both the Discrete Policy Solutions (DPS) method 34 M. which learns a model for every preference vector, and RS, which approximates the PF with m models optimized exclusively on the m preference dimensions. Being computationally lightweight is especially crucial in the LLM settings. Panacea also allows online specification of the preference vector to swiftly adapt to any human preferences, meeting users' requirements in no time. Moreover, Panacea achieves a tighter generalization bound of Pareto optimality compared RS for unseen preferences during training, implying more complete recovery of the Pareto set. This is due to the explicit traversal of the preference simplex, which allows its generalization error to decay with the number of samples. In contrast, RS only uses small number of Pareto optimal solutions for interpolation to predict unseen Pareto optimal solutions. The interpolation error cannot be effectively bounded when it only meets few preference vectors during training. Finally, Panacea preserves explainability to some extent. For each weight matrix W Panacea adapts it as
w' w: + U'E'V' w/
Σx olulel ii=
+ i=l
s' s'Aauht
(8)
[2]
Intuitively, term [1] captures shared features among preference dimensions, while term [2] learns dimension-specific adaptations and weights them by the preference vector to achieve Pareto alignment. The decoupling of learned parameters not only illustrates the mechanism of Panacea, but also leads to superior robustness of its preference adaptation strategy (further analyzed in Appendix F.5
5 Experiments
In this section, we empirically evaluate Panacea's ability to approximate the PF of complex and multi-dimensional human preferences. We apply Panacea to several significant and challenging preference alignment problems with 2, 3,4, 5, and up to 10 dimensions, far exceeding those addressed in contemporary works. These problems include the classic helpful-harmless (HH) dilemma, its (HHC) version, and learning the PFs of multiple common preference dimensions in chat scenarios. While the number of dimensions m varies, we keep the preference-agnose rank i of Panacea fixed to 8 and observe Panacea's 's performance. Compared with the baseline RS, Panacea consistently learns superior, broader, smoother, more evenly distributed, and convex fronts that align with theoretical expectations. The advantages are quantified through various metrics to substantiate its effectiveness and scalability. Encouragingly, we find that Panacea shows no signs of performance saturation even on the ten-dimensional problem, indicating its unlimited potential. We also conduct ablation studies to validate the design of Panacea. Full experimental details are claborated in Appendix F and chat cases are presented in Appendix G
augmented

---

Fronts of different methods (RLHF)
Fronts under different seeds (RLHF)
Fronts of different methods (DPO)
30
30
Harnless roward
20
KS
Harmless reward
20
Patnooa
Harmless aecuracy
0.7
Panacca w.
LS
Panacoa
0.6
Tche
10-
10
RS
0.5
0.4
-10
-10
-10
0
10
20
-10
10
20
0.4
0.5
0.6
0.7
Helpful reward
Helpful reward
Helpful accuracy
Figure 3: Algorithm performance on HH. Baseline methods (RS and DPS) require training a separate model for each preference dimension/vectorr, whereas Panacea learns a single adaptable model. Left: Panacea is significantly better than RS and even outperforms DPS, showing its superiority in learning PF while being more efficient. Middle: on Llama2-ft across different seeds, Panacea again consistently outperforms RS, and its fronts exhibit smooth convex shapes that correspond with theory. Right: with DPO, Panacea using both LS and Tche aggregation learns better fronts than RS.
5.1 Mastering Dual Dimensions: Addressing the Helpful- Dilemma
In the first set of experiments, algorithms are tasked with two-dimensional preference alignment using various initial models, i.e.
finetuned [4]] Llamal-7B-bine (abby. Llamal-ft and Llama2- 7B-base [50] (abbv. Llama2-ft), optimization procedures, i.e. RLHF and DPO, and loss aggregation methods, i.e. L.S and Tche. Specifically, we focus on the helmlul-barmless (HH) dilemma, which is an important and urgent problem since different applications of LLMs often require different trade-offs between them. For example, children need extremely safe chat assistants, while chemists prioritize helpfulness as they are fully aware of the potential hazards. However, current alignment techniques provide the same model for all users, which does not cater to these diverse needs. Therefore, learning the entire PF can significantly alleviate this issue. We use the Beaver'Tail dataset (28), which has preference labels for both helpfulness and harmlessness.
In Figure 3 left, we show the learned fronts of algorithms with the task con- figuration of Llamal-ft, RLHF, and LS aggregation. The rewards for both dimensions are evaluated by re- ward models for preference vectors sam- pled evenly at an interval of 0.1, i.e. A (0.0, 1.0), (0.1, 0.9), ..... (1.0, 0.0). Compared with RS, Panacea learns a sig- nificantly better front, whose smooth con- vex shape also aligns better with the con- vexity result in Lemma C By contrast, the front learned by RS is not a valid Pareto front since some solutions domi- nate others, which shows that RS could not learn to recover the PF simply by merging trained weights for all dimen- sions. In this experiment, Discrete Policy Solutions (DPS) (34] [7), also known as multi-objective RLHF (MORL) in 45).
What are different ways that a car can stolen? ?
[Helpful, Harmlcas]
[o, J]
[1.0) 0]
A car can be stolen in variety of [Providing instructions]..
Do not answer this question due the potential of encouraging illega activity...
is important is secure...
sure your
Response
Dirccily reject answers with unsafe inquiries. Maximizes the harmlessnces of the model.
Provide users with more advice on preventing theft. The model becomes helpful.
Deseription of preference shift
Figure 4: Responses of the model to the same user prompt with two extreme preference vectors. Regarding inquiries with unsafe viewpoints. the model can either caution users about illegal activities from a harmlessness perspective or provide helpful suggestions for theft prevention.
also test
which learns separate model for each preference vector (1 models in this case) and is commonly considered as the performance upper bound for this problem. Surprisingly, Panacea learns better and smoother front than DPS while being much more efficient, which could be attributed positive transfer among dimensions enjoyed solely by Panacea. In Figure middle, middle, we conduct the same experiment based on Llama2-ft initial model. Across three seeds. Panacea consistently achieves convex and dominating fronts that are more desirable than those of RS, further verifying the results. To clearly demonstrate how the model's output changes with variations in the preference vector, we present an exemplar chat case in Figure and and its detailed version in Appendix G The chat case shows how Panacea effectively tailors to diverse needs, thereby settling the long-standing tension between helpfulness and harmlessness.
7

---

Fronts any dimensioos the Chat 5-dim alignment (SFT)
Philosophcaal objoctivve
-0.5
-0:2
Helpful objectivve
Helpful objective
-0.4
-0.4
10.6
-0.6
Pannccs
RS
-0.80
-1.1
.0.9
.0.7
-0.5
-1.1
-0.9
-0.7
-0.5
-1.1
-0.9
.0.7
.0.5
Humorous objective
Humorous objective
Philosophical objective
Figure Comparison of learned fronts on Chat 3-dim problem. On the left we show 3D visualization of Panacea (red) and RS (blue) and on the right we show 2D projections by setting one of preference weights to zero. Clearly, the front learned by Panacea dominates that of RS by a large margin.
To further study the generality of Panacea, we conduct experiments with Llama2-ft, DPO, and IS / Tche aggregation, where Panacea is optimized based on Equation (T)) and Equation (9) respectively. For DPO, we propose to evaluate algorithm performance by measuring the implicis
reward model accuracy. That is, for a model To, it is accurate on labeled pair (z, V vi) if
To (v(.l=)) Glog
Th (xil==)
Sist Fest (vi Bllg
Stet (v/la) and its total accuracy is obtained by averaging over dataset. With
this metric, in Figure FigureBrright we plot accuracies of HH dimensions for Panacea with LS / Tche and RS baseline. Results again confirm that Panacea always obtains better fronts.
Aside from comparing the fronts learned by Panacea and the baseline, we also quantify the advantage of Panacea by computing four MOO metrics in Table I Hypervolume, the primary metric, measures the volume of space enclosed by a solution set, reflecting its optimality (a visual illustration shown in Figure 9 the average value of Inner product of preference vectors and the evaluation results measures the correspondence between preference vectors and solutions; Sparsity and Spacing further reflects whether the solutions are evenly distributed. Mathematical expressions of these metrics are detailed in Appendix F.44 Table Table]eleari cllarly demonstrate dominance Panacea over RS on learning more optimal and tailored solutions to diverse preferences while using only a single model.
5.2 Navigating Tri-Dimensional Trade-offs: Helpful, Harmless, and Concise Alignment
In chat scenarios, the potentially large number of preferences necessitates an efficient method that scales beyond two dimen- sions. Starting from this section, we start to consider more than dimensions and test Panacea's 's capability handle them simultaneously. We first augment the HH dilemma with conciseness, another common preference dimension, and com- pare the algorithms on the task configuration Llama2-ft, RLHF DPO, and LS aggregation upon BeaverTails dataset. For RLHF, the concise RM is defined as a rectified affine func- tion that assigns higher rewards to shorter responses; for DPO, the shorter response to each prompt is preferred in the con- ciseness dimension (details provided in Appendix (F). For all experiments, we evaluate the algorithms with preference vec- tors evenly sampled from the entire simplex at an interval of 0.2, iee À = (0.0, 0.0, 1.0), (0.0. 0.2, 0.8)..... (1.0, 0.0, 0.0), and provide the results in Figure 5 and Table
Cazoal Roveal
-38
Rbmoion Roward
15
Figure 5: Learned fronts of Panacea (red) and RS (blue) on HHC prob- lem with Llama2-ft, RLHF, and LS aggregation. Panacea learns a better and more evenly distributed front while solutions of RS clutter in corner. This suggests Panacea pro- vides fine- ane-grained solutions to di- verse human preferences.
Figure 5 visualizes the fronts learned with RLHF procedure. We observe that Panacea learns a very evenly distributed front, whereas most solutions obtained by RS are cluttered together in a corner. This is because Panacea, as PSL method, explicitly traverses the preference simplex to learn about PF, resulting tailored solutions corresponding to each preference vector. In contrast, RS only learns the vertices and cannot generalize well to solutions within the simplex through linear interpolation. Meanwhile, we also observe that Panacea performs better overall in the harmless dimension, further demonstrating the advantages of its learning approach. MOO metrics in
8

---

Fronts under different preference scalings
Ablation of preference adaplation
Frrnnts of Panaceu during leurning
Harmless reward
29
Preference scaling
Harmless reward
30
10 lnyers
30
1o-4
20
First 10 layers
20
16
1o-5
1o-2
learaale
10
10
le-3
Self Belf-antemminn
Step
All
20
100
180
MLP
-10
-10
-17
-5
19
-10
20
-15
-5
5
15
Helpful reward
Helpful reward
Helpful reward
Figure 7: Left: Ablation study the learnable preference vector scaling factor. Predefined scaling factors ranging from 1 to 10-6 all result in significantly worse fronts than the learnable approach,
indicating the importance of the per- per-weight-matrix learnable scaling factor. Middle: Investigation of alternative preference adaptation strategies, including adapting only MLP layers, self-attention layers, 10 layers in the front, and 10 layers in the back. Except for the back 10 layers, all other strategies exhibit similar performance. Thus, we decide to adapt all layers for better representation capacity. Right: We show the fronts learned by Panacea at different RLHF steps. The evolution of fronts reveals Panacea's 's learning process which gradually expands in both dimensions, reduces dominated solutions, and finally converges to a broad and convex front.
Table 1 again numerically depict the benefits of Panacea, and the chat case in Appendix G serves as qualitative support. Thus, by learning a more comprehensive solution space, Panacea effectively manages the trade-offs among helpfulness, harmlessness, and conciseness, underscoring its capability to align with diverse human preferences.
5.3 Scaling Up: Towards Tens-of-Dimensional Pareto Alignment with Single Model
We further test Panacea's 's scalability on three, four, five, and up to ten-dimensional aligmment problems (abby. Chat 5, and 10-dim), where the considered dimensions include being humorous, philosophical, sycophantic, helpful, concise, creative, formal, expert, pleasant, and uplifting. These dimensions reflect the common scenario where desirable chat properties attainable. Hence it requires a Pareto-optimal solution set to accommodate diverse preferences. In solving these problems, we employ Panacea with SFT procedure, since SFT is easier train and scales better. The initial model used in this series of experiments is Llama-3-88 -nstrrnstt (3) (abbv. Llama3-Instrucl, and the loss aggregation function is LS. We first curate data for each dimension by prompting Llama3-Instrus to generate responses to Alpaca instructions with the corresponding property (details are provided in Appendix [F). Panacea is then trained using LS aggregated SFT loss. The baseline RS trains separate models for each dimension using the corresponding SFT loss. In evaluation, we report the SFT losses of each produced model on the test set in all dimensions. For 3, 4, and 5-dimensional problems, we evaluate the algorithms with preference vectors sampled at an interval of 0.2, resulting in 21, 56, and 126 total evaluations; for ten-dimensional problems, we sample them at an interval of 0.25. amounting to 715 in total. These comprehensive evaluations allow us to characterize the algorithm performance more accurately. We plot the results of Chat 3-dim in Figure and compute the metrics in Table I Figure Figure[]Jshows sloows that Panacea learns significantly better front than RS. For the nigher-ddimesionall problems where the results cannot be visualized, verify the convexity of Panacea's 's learned fronts by computing their convex hulls and observing that all evaluation points are on the respective convex hulls. From Table I we also observe that Panacea consistently outperforms RS, and the advantage gap becomes larger when scaling to higher dimensions. Notably, Panacea is an order of magnitude better than RS on Chat 10-dim and does not exhibit performance plateau, demonstrating its scalability. We provide chat case in Appendix G from Chat 3-dim to show Panacea's 's performance. These results confirm that Panacea learns a single model capable of aligning with any human preferences.
not simultaneously
5.4 Ablation Study and Analysis
In this part, we validate the design of Panacea and investigate its learning process on the HH problem. We first analyze the effect of the per-weight-marre learnable scaling factor s'. Intuitively, it scales preference vectors to the same magnitude as the singular values to avoid either dominant or negligible

---

influence of sreference-speciife features on W as observed from the learned parameters. To validate its importance, we conduct ablation experiments that use a predefined factor scale preference vectors. Figure / (left) indicates that using fixed scaling results in significant performance drop regardless of its magnitude, highlighting the necessity of learning an appropriate scaling for each weight matrix separately. We also explore alternative strategies of preference adaptation, which only adapt self-attention layers, MLP layers, first 10 layers, or last 10 layers. Figure (middle) suggests that except for only adapting last 10 layers, all other strategies perform comparably. Thus, for better representation capacity, we decide to let Panacea adapt all layers of an LLM. Finally, in Figure 7 (right), plot the evolution of fronts learned by Panacea at different steps, showing that first learns harmlessness features quickly and explores improvements for helpfulness, then it also learns to align with helpfulness preference and finally recovers the entire front. This discovery may inspire training acceleration methods such as dynamically sampling preference vectors according to different learning efficiencies across dimensions.
6
Conclusion
This paper presents Panacea, the first Pareto set learning approach towards solving Pareto alignment with multi-dimensional human preference using single model. Central its design is embedding the preference vector as singular values in SVD-LoRA to fundamentally influence model behavior online. Theoretically, we prove that training the sefferrnnce-mbbedee model against an aggregated objective is guaranteed to recover the entire PF at convergence. Empirical results substantiate that Panacea enjoys superior performance and scalability in approximating PF compared with strong baselines including DPS and RS. Overall, Panacea represents simple yet effective approach that achieves fine-grained, lightweight, and online Pareto alignment with diverse and complex human preferences, urgent need in LLM applications.
10

---

Acknowledgments and Disclosure of Funding
We thank Yexin for helpful discussions during the early phase the project. This work is sponsored by National Natural Science Foundation of China (62376013), Beijing Municipal Science Technology Commission (Z241 10400003222000, Z231 100007423015), Young Elite Scientists Sponsorship Program by CAST 2022QNRC003, the Research Grants Council of the Hong Kong Special Administrative Region, China [GRF Project No. CityU 11215723).
References
[1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[3] AI@ Meta. Llama 3 model card. 2024. URL
blob/main/MDDEL.CO CARD. md
[4] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310. 12036, 2023.
[5] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2rX4v2220..05862, 2022.
[6] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022.
[7] Leon Barrett and Srini Narayanan. Learning all optimal policies with multiple criteria. In Proceedings of the 25th international conference on Machine learning, pages 41-47, 2008.
[8] Toygun Basaklar, Suat Gumussoy, and Umit Y Ogras. Pd-morl: Preference-driven multi- objective reinforcement learning algorithm. arXiv preprint urXiv:2288.07914, 2022.
[9] Kenneth J Berry, Janis E Johnston, and Paul W Mielke Jr. Permutation methods. Wiley Interdisciplinary Reviews: Computational Statistics, 36):5277-542, 2011.
[10] Stephen Boyd and Lieven Vandenberghe. Convex optimization, Cambridge university press, 2004.
[11] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-35 1952.
[12] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. 15217, 2023.
[13] Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. Maxmin-rlhff: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925 2024.
[14] Eng Ung Choo and DR Atkins. Proper efficiency nonconvex multicriteria programming. Mathematics of Operations Research, 803):467-770, 1983.

---

[15] Josef Dai, Xuehai Pan. Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310. 12773, 2023.
[16] Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computaton, (2):182-197, 2002.
[17] Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06765 2023.
[18) Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. Steerlm: Attribute conditioned sft as an (user-stcerable) alternative to rlhf. arXiv preprint arXiv:2310.05344, 2023.
[19] Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjee Lv, Changjie Fan, and Zhipeng Hu. Aligndiff: Aligning diverse human preferences via behavior- customisable diffusion model. arXiv preprint arXiv:2310.02054, 2023.
[20] Matthias Ehrgott. Multicriteria optimization, volume 491. Springer Science & Business Media, 2005.
[21] Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Jiexin Wang, Huimin Chen, Bowen Sun, Ruobing Xie, Jie Zhou, Yankai Lin, al. Controllable preference optimization: Toward controllable multi-objective alignment. arXiv preprint arXiv:2402.19083 2024.
[22] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. httl:ttts////penrreet
(23] Yuzheng Hu, Ruicheng Xian, Qilong Wu, Qiuling Fan, Lang Yin, and Han Zhao. Revisiting scalarization multi-task learning: A theoretical perspective. Advances in Neural Information Processing Systems, 36, 2024.
[24] Minyoung Hwang, Luca Weihs, Chanwoo Park, Kimin Lee, Aniruddha Kembhavi, and Kiana Ehsani. Promptable behaviors: Personalizing multi-objective rewards from human preferences. arXiv preprint arXiv:2312.0933 2023.
[25] Moksh Jain, Sharath Chandra Raparthy, Alex Hernández-Gaariaa, Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. In International Conference on Machine Learning, pages 14631-14653. PMLR, 2023.
[26] Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zeulemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Per- sonalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564, 2023.
[27] Natasha Jaques. Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: 2019.
[28] Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via human- premennee dataset. In seveth-e Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[29] Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey. arXiv preprint Xr 10.1992,, 2023.
[30] Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. survey of reinforcement learning from human feedback. arXiv preprint 1rX1v:2312.14925, 2023.
12

---

[31] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[32] Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with feedback. arXiv preprint vxXiv:239.002267 2023.
[33] Seung Hyun Lee, Yinxiao Li, Junjie Ke, Innfarn Yoo, Han Zhang, Jiahui Yu, Qifei Wang, Fei Deng, Glenn Entis, Junfeng He, et al. Parrot: Pareto-optinns multi-reward reinforcement learning framework for text-to-imase generation. arXiv preprint 2xXi:24401006673 2024.
[34] Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective opti- mization. IEEE transactions on cybernetics, 51(6):3103-3114, 2020.
[35] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning, Advances in neural information processing systems, 32, 2019.
[36] Xi Lin, Zhiyuan Yang, Qingfu Zhang, and Sam Kwong. Controllable pareto multi-task learning. arXiv preprint arXiv:2010.06313, 2020.
[37] Xi Lin, Zhiyuan Yang, Xiaoyuan Zhang, and Qingfu Zhang. Pareto set learning for expensive multi-objective optimization. Advances in Neural Information Processing Systems, 35:19231 19247, 2022.
[38] Xingchao Liu, Xin Tong, and Qiang Liu. Profiling pareto front with multi-objectivve stein variational gradient descent. Advances in Neural Information Processing Systems, 34:14721- 14733, 2021.
[39] Kaisa Miettinen. Nonlinear multiobjectiive optimization, volume 12. Springer Science & Business Media, 1999.
[40] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthicu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:2312.0080 00886, 2023.
[41] Aviv Navon, Aviv Shamsian, Gal Chechik, and Ethan Fetaya. Learning the pareto front with hypemeetwrks. arXiv preprint vrXiv:2010.04104 2020.
[42] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[43] Jan Peters and Stefan Schaal. Reinforcement learning by rewgrd-weighted regression for operational space control. In Proceedings of the 24th international conference Machine learning, pages 745-750, 2007.
[44] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprini arXiv:2305. 18290, 2023.
[45] Alexandre Rame, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptise Gaya, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. arXiv preprint arXiv:2306.04444 2023.
[46] Diederik Marijn Roijers, Shimon Whiteson, and Frans A Oliehock. Computing convex coverage sets for faster multi-objective coordination. Journal of Artificial Intelligence Research, 52: 399 443, 2015.
[47] Gokul Swamy, Christoph Dano, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056 2024.
13

---

[48] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Perey Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. https://rrfm. stanford. 3tti22003/33///appaacc himl, 36))7, 2023.
[49] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[50] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXXv:2307.092888 2023.
[51] Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi-objectivy rewards. arXiv preprint arXiv:2402.18577 2024.
[52] Kailai Yang. Zhiwei Liu, Qianqian Xie, Tianlin Zhang, Nirui Song, Jimin Huang, Ziyan Kuang, and Sophia Ananiadou. Metaaligner: Conditional weak-to-strong correction for generalizable multi-objective alignment of language models. arXiv preprint wrXiv:2403.17144 2024.
[53] Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402. 2400000277 2024.
[54] Runzhe Yang. Xingyua uan Sun, and Karthik Narasimhan. A generalized algorithm for multi- objective reinforcement learning and policy adaptation. Advances in neural information pro- cessing systems, 32, 2019.
[55] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.055: 05302, 2023.
[56] Qingfu Zhang and Hui Li. Moea/d: A multiobjective evolutionary algorithm based on decom- position. IEEE Transactions evolutionary computation, 11(6):712-711, 2007.
[57] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023. URL URL/tttps: http::ttt///// //ttppev//penreviex
net//oruum7dd-1gBuMMMRJ
[58] Xiaoyuan Zhang. Xi Lin, Bo Xue, Yifan Chen, and Qingfu Zhang. Hypervolume maximization: geometric view of pareto set learning. In Thirry-sevnnth Conference on Neural Information Processing Systems, 2023.
[59] Aimin Zhou, Bo-Yang Qu, Hui Li, Shi-Zheng Zhao, Ponnuthurai Nagaratnam Suganthan, and Qingfu Zhang. Multiobjective evolutionary algorithms: A survey of the state the art. Swarm and evolutionary computation, 1(1):32 49, 2011.
[60] Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond mre--eefrrenc--foralll Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03700 2023.
[61] Yiheng Zhu, Jialu Wu, Chaowen Hu, Jiahuan Yan, Chang- Yu Hsich, Tingjun Hou, and Jian Wu. Sample-efficient multi-objective molecular optimization with gflownets. arXiv preprint arXiv:2302.00 04040, 2023.
14

---

Supplementary Material
Table of Contents
Preliminary Theoretical Results
16
B The Limitation of Single-Ohjective Alignment
C Theoretical Support for Panacea with LS/ LS/Tech function
18
Proof for LS Aggregation Function
18
Proof for Tchebychelf Aggregation Function
19
Pscudocode of Panacca
20
E
Aggregated Training Objectives Panacea
20
F
Experiment Details and Additional Results
21
F.1
Core Implementation of Panacea
21
F2
Data Curation
21
F3
Experiment Sctup
22
F4
Evaluation Details
23
Additional Results
25
Information of assets
G Chat History Examples
27
H Discussions
28
Limitations
28
H22 Broader Impacts
29
15

---

A Preliminary Theoretical Results
In this section, we prove the validity of combining reward models of all preference dimensions through linear scalarization in the RLHF optimization procedure, even though each reward model solved by the Brarley-Terry (BT) model [[] is not uniquely determined. This is formalized in the following lemma.
Lemma A.1 (Extension of Lemma 2 in [4 for multiple reward models). Let Le v) and (z v) be equivalent reward models for the i-th preference dimension, where (x y) = r; r,
ds(x)). The linear combinations r(x,y) = Li=1 A.Ti(z, v) and (1, E Ari((r
m 1= (x):(z) induce the same optimal policy in the constrained RL problem max, (RLHF(T) =
BEND By-=((1>)) (r(x,y) - 8Dx [o (-1=]]]] where 3 is positive punishment of the KL constraint.
Remark A.2. This lemma demonstrates that it is valid to linearly combine reward models of all dimensions, even if the reward models are not uniquely identified. It is used in analyzing the limitations of single-objective alignment and it validates the LS aggregation employed with Panacea.
Below, we provide a concise proof of Lemma A.1
Proof. According the constrained RL literatures 43 91. the policy for the reward function r'(x, y) in Kullback-Leilerr (KL) constrained reinforcement learning (RL) problem can be formulated as follows:
Ther (y)l) exp (3r² (zr (I.
Tp
En Teat (y) exp (tr'(z))))) 8)
Expanding the term in r' (x,, v), we obtain:
exp ( exp
E m
Axra(e, + \xox(1)
B
(()))
Ty. (3)2z)
Ey Fer(yll))
Em Avri(E,)) + (x))
B
(x)))
Upon simplifying by canceling out the common term exp(d' (x))) we get:
Teer(y)>)) exp (2r( v) exp (1(())))
Typ (y/xx
(2(1)))))
E, Thee (B) exp (fr(x,, v) exp (2(0)12))
which completes the proof.
B The Limitation of Single-Objective Alignment
In the following content, provide theoretical analysis that the model trained by the single- objective alignment paradigm could actually misalign with every labeler. We conduct analysis on RLHF, the most common approach. We make the following assumptions:
Assumption B.1. Human preference be modeled by the Bradley-Terry model ||]
16

---

Assumption B.2. Different people are consistent in labeling each preference dimension.
These two assumptions imply that people possess the same reward model (2 y) for each preference dimension
Assumption B.3. The synthesized reward model of a person is the LS of per-dimensional reward models according to his/her preference vector under a shift invariant term (c.f [4]]malmmll), That is,
r(a, v)
Aera(a, v) d(x))
Now we prove the main theoretical result.
Theorem B.4. Consider the case where there are n. labelers in total. Each labeler h labels portion
=1 1. The preference vector of labeler is X pd of the entire dataset, where pd [0, 11, Eh 1 p
(Xi Ni. Ni....., ...,,,)) The labelers have different preference vectors, i.e. 3j.h E {1...., n). , X + X
The RLHF optimization result is model that could misalign with every labeler.
X,r((z,)) v) d' Jh (0) denotes Proof. The reward model n² of labeler is r" r" y) Σx=
the optimization objective corresponding to the reward model of labeler h. The joint optimization objective is
Σ p (TO) h=1
max
(Substituting the oracle reward function.)
(10)
(EEND (ty)a [r" v)] -8DKL [
(Rearrange reward terms.)
(11)
max E ENDD
Eynrr (-1n)
Σ ("r"(1z0))
KI [-
max DD
(yyoto()aa)) Σ hh
xrr(a,,) + + 30Dx [|
(Define spx)) == Σ (
(12)
h= 1
max E E
E EyyoTo(-1a)
p Axt(a v) + p(x)
E DD
E (-yo)
A v) p(x)
max E D
Eynns (1|1)
pd X (x(1))) + ((x))
1D [- Tit
Deeine X
Σ
p x
1, 1,...,m) m)
(13)
h=1
max E F
Eynnno(-(a))
xT r. p0(0))
(14)
Thus, we show that it actually optimizes with the preference vector o with X Σ X m. According to the constrained RL literatures [43] 9 the corresponding optimal policy can
17

---

be expressed as:
m x (
Tel (vlæ) = (xx) Fret (ulxr) exp
(15)
It is important to note that this optimal preference vector may not align with the individual preferences of each annotator. As result, the trained model may not fully reflect the labeling criteria of any single annotator, potentially leading to discrepancies in the model's predictions.
C Theoretical Support for Panacea with LS / Tche function
In the following content, we prove for Theorem 4.1 from the main paper, showing that both linear and Tchebycheff scalarization can recover the entire Pareto Front (PF) under practical assumptions. The proof has two subsections: first for the linear scalarization function in Appendix C.T. followed by the Tchebycheff aggregation function in Appendix C.2
C.1 Proof for LS Aggregation Funetion
We provide proof sketch for this part.
Step 1: Under the full categorical representation assumption, for any policies π (-((a)) and
o (b) (11z)),
create new policy (x²) that, with probability (w.p.) p (where 0 S p S 1), takes
(() (-12)) and w.p. p, takes (+)) (-1z). This policy also be represented by LLM.
Step 2: Using the above policy construction method, we prove that the objective spaces of DPO, RLHE, and SFT are convex.
Step When the objective spaces are convex, the Pareto objectives found by LS aggregation function (Convex Coverage Set (CCS)) equal the entire Pareto front.
Step 4: By optimizing the Panacea objective function Ex-Unit(Am.) [oX (0)]. recover the entire Pareto front.
Then, we are geared up for the formal proof. We first restate the assumption for the full categorical policy space Theorem 4.1
Assumption C.1 (Full Categorical Policy Space Assumption (detailed restatement from Assumption
2 Theorem 4.1 ). For a specific preference vector X, the LLM policy space formed by all y Te.
xT0_>-la:)) (-|z)) represent all the categorical distribution set II(x: for response
[t1, [t.....,ty]] tv],
where is the response length and t; denote each token, given an input sentence x.
This assumption is proper because the probability of each token ,,,...,N (N denotes the length of the output of y/) can be represented by a LLM policy. Given the strong representation ability of LLMs, any probability value of token sequence t ..... ty can be represented by their output. With this assumption, direct corollary holds because the linear combination of categorical distributions is still categorical distribution.
As corollary of Assumption
we have:
Corollary C.2. For two policies ne (=)a()))) (-12:) and (0) (-)x)) new policy x' w.p. p (0 Sps 1) follows ( (-(2) and w.p.
follows x (b) (-12)) belongs to the categorical distribution II(x).
The reason for that is such constructed policy is still a categorical distribution. For the next step, we use this corollary prove the following lemma to show that the objective spaces JSPT, JRLHF, and JDPO are convex.
Lemma C.3 Convex space Lemma, adapted from 23 (Eq. 13)), For any two objectives J fla) and J (T)).
alg for any 0 < 1, there exists policy x' [I(x) such that ouJ caj(a) alg (1 J.J alg where Jalg be JDPO. JSFT, JRLHF.
This lemma mainly follows from Eq. 13 for completeness. The objectives JSFT, JRLHE, and JDPO all be written as (T)
23 We include their proof for our purpose
18

---

EN,yED [z. fz.
for some particular design of f(z, y, (z)22)) For any 0 < < 1, by we can construct a new policy N' and uniform random variable ~~ U0, 1) such
Corollary C.2
that:
(ulo)) x' <'((a) (1|z)
<
2
Then,
J J(r') B(aD) f x (y)z))]
(0 of
( [ (z, 7 (y]=) (v])) + (1 a) (z (y|z))
al (1 a) B))) T()l=))SS al
f of
ar)(al)) (1-
Thus, for any convex combination of J (T(()) and J (+)()),, there exists policy x' such that J (+²) aJ (T((1) + (1 (T))), indicating that the space of J (Tr) (r) is convex. We denote the full space of (Tx for all policies as J.
For the third step, we use Lemma C.3 to establish that linear scalarization functionn have the capability discover the complete PF by traversing the entire preference simplex A. (i.c., the approach employed in Panacea). To prove that, we introduce the concept of the convex coverage set (CCS), which is the objective set that can be found by optimizing the linear scalarization function with all preference vector À E Am.
Definition C.4 (Convex Coverage Set (CCS), adapted from [Def. 9)). The CCS contains the objective such that there exists preference vector À where the inner product of X and this objective greater than that of with any other objective vectors in the objective space. CCS == (J TE E Am s.t. A J 2 X J YJ' E J].
Then, we prove that when the objective space is convex, the linear scalarization can recover the whole Pareto objective set, i.e., T = CCS, where T denote the objective vectors forming the Pareto front.
Proof. The PF is a subset of the boundary of the objective space, denoted as ((J(IT)). By proving that J(TI) is convex set, we can apply the supporting hyperplane theorem [10 (Sec. 2.5.2). According to this theorem, for every element r in ( there exists À E IR such that Ar - r') 2 0 for all r' J (II.. Moreover, when is Pareto optimal, such À > 0. Hence, we
have A (r - r') 2 for all r' J (II) and À E Am. This condition implies that T C CCS. Since
it has been established that CCS T,
conclude that CCS = T.
O
For the last step, we demonstrate that by optimizing Ex-Unif(A... [o LS [0)] using the LS aggregation function, we can recover almost the entire Pareto front. If non-zero measure of the Pareto front could not be recovered, would imply the existence of non-zero measure preference vectors for which the corresponding Pareto-optimal solutions cannot be found using the LS aggregation function, which contradicts Assumption in Theorem 4.1
C.2 Proof for Tehebycheff Aggregation Function
To prove that using the Tchebycheff aggregation function allows Panacea to recover the full Pareto front, we introduce the following lemma:
Lemma C.5 (Adapted from 14), Theorem 3.1). A feasible solution 0 is weakly Pareto optimal if and only if there exists a weight vector X such that 0 is an optimal solution to the aggregation function (Equation ? defined in the main paper.
Using this lemma and assuming Panacea can represent the Pareto policy under all preferences (Assumption in Theorem 4.1 optimizing the expectation loss
- -EX-Unif(A..)
[9X Tche (0)
allows Panacca to recover almost every policy.
19

---

Proof. If non-Pareto policy has a measure greater than zero, then according to Lemma C5 there exists a preference set of greater than zero measure where the non-Pareto policy has smaller value compared to the optimal value of the Tchebycheff function under the corresponding preferences, contradicting Assumption in Theorem 4.1
D Pseudocode of Panacea
Algorithm Panacea
1: Input: Rank k, preference dim m, dataset D, iterations T, initial model Tini (. optionally reward model r; for each preference dimension i).
2: Output: Trained policy To.
3: Initialize my by initializing SVD-LoRA upon Tia based on k and
4: for in 1... T do
5: Sample from D data batch B.
6: Sample a preference vector À and embed into πa,
7: Compute the aggregated objective for TT, B according A.
8: Update 0 with gradient descent.
9: end for
10: Return
E Aggregated Training Objectives for Panacea
In this section, we present the LS / Tche aggregated training objectives for Panacea with RLHF / DPO SFT. In RLHF, reward models rv, ..... m are learned for each preference dimension. For specific preference vector, the LS aggregated objective function is
m Σ
max g (0) max ENND
Eynno
Axr (2
1D (so (- (16)
The Tche aggregated objective is
max ØX
Tche (8) max E
E
max >e(2; - re(2,,)) x)) 300k (-1)((1=)))]=rrt 1515m
(17)
where 2; is the maximum reward for preference dimension Intuitively, Tche aggregation aims to minimize the maximum weighted suboptimality among all dimensions. However, since the maximum reward can be hard to determine in practice, we find Tche less suitable for RLHF than for DPO.
DPO transforms the reinforcement learning objective into supervised objective, whose LS aggre- gated objective is
7/L
max 9X (0)
mAX
A:Dooo (TOOx)
Ag.A (Nen (x:
<e,7 (1n (unlz²) [z) Tref (yn (x)) }
max
X (n (m.
logo Blog
(18)
Jeef (
20

---

To derive the Tche aggregated objective, we have
Tche (0) max min {((JDPO,(TTOA)) zi)
max 9X
15
max min (TOA)
15i5
logo (Blog 20.7 (yua Tref (zw(z)
To.a (3) (x) Tref (g1((z) (z)
max min A (.
B log
(19)
Since the optimal value z; for per-dimension DPO objective is 0, this is naturally compatible with Tche aggregation.
Finally, the LS aggregated SFT objective is
m
m
max 9X (0) max
X JSFT, max
A (2 (2,)> [log To.x
(20)
Similar DPO, since the optimal value 2; for per-dimension SFT objective is O, the Tche aggregation of SFT objectives is
Tcbe (0) max min (T( 24)
max QX
0 15:5m
= max min (TOS))
8 Isism
max min (a [log TT
(21)
8 1sism
F Experiment Details and Additional Results
In this section, we present experimental details including computational resources, algorithm imple- mentation, data curation, experiment setup, and evaluation details, and analyze additional results. All our experiments are conducted on an 8x A800-800G GPU server. Other details are elaborated below.
F1 Core Implementation of Panacea
codebase. As described Section and visualized in Figure 2 the core design of Panacea is the embedding of the preference vector as singular values based on SVD-LoRA. Its core code is presented in Figure g In our experiments, we perform Panacea adaptation to all self-attention and MLP layers. We initialize the singular values and preference scaling to zero, so as not to impact the model behavior at the beginning of training [2 57]. In each iteration, sample preference vector from the preference simplex, embed it into the model, and train the model on the aggregated objective.
Our implementation based on the Safe-RLHF 15
F.2
Data Curation
the BeaverTaill dataset (28). which contains both helpfulness and harmlessness preference labels. In the augmented helpful-harmless. concise (HHC) problem in Section 5.2 we again the BeaverTaill dataset. For RLHF, define the reward model as a rectified affine function,
In the helpful-harmless (HH) problem in Section 5.1
Fmax: ly Σ
Feoncise (I., y)
Tmax + Le otherwise
where Timx defines the maximum reward, ly denotes token length of response y, and e defines both the threshold for maximum reward and the slope of concise reward model. This reward model encourages more concise answers, while the reward does not further increase when the response length smaller than given threshold. For DPO, we label the shorter response each prompt preferred.
21

---

class Panacealayyr Lorch,
def init (self, weight, bias, tora din, Lora_scaling, pret din):
super (Panaccaaal,,,, self)._init__
Save original weight and bias
self.weight = weight
self.bias
# Initialize SVDLORA paraneters
rows, columns weight. self.lora right weight torch.nn.Parmmeet Parameter (torch. randnntcolumss, lora dim pref (ref_dim)))
nn,
10
self. lara diagonal weight torch. Paraneter (torch.ze zeros((11, tora dim) self. Raf.llor_left__eiiht weight torch.nn.
11
pref_ding rows))
12
self. lora scaling tora scaling tora dim Preserve paraneters for preference vector
13
14
self.pref. diagonal L torch, nn. waw.aettertorrcc..eeosso pref dim)))
15
Initialize preference scaling
16
self.pref Saaf.pref_scaling scaling torch.nn.Parmeteere. pat.errannteertoocc.eeeess//
17
self.intt_parmmeterss] parameters()
self.init
18
Freeze original weight
19
self. weight. requires_gres _rrrdddaa False
28
if self.bias 1s not None:
21
self. bias. riquires grad
22
not compute gredient for preference vector self.pref_ heff.pref_daggooe diagonaallln..ooooooooooo wirrr_diagona_wwiihhtt..o requiresrrguiirs_rrrr. - False
23
25
det
init parameters((eLt):
26
# Initialize SVOLORA paramcters
torch.an. wtt..nn.iittterr. (self.lora diagonal weight)
28
wathh.nnniitt.mmm... (self. lora_right weight, mean=0.0, stdm0 82) int.nnn,nntnmmaaaaa Initialize preference scaling
29
lora_left, weit_w mean-0.9, std-8.02)
30
31
torch. an. init. pref__ccllingg))
32
forward(self, input]:
34
# Concatenate singular vatues and the scaled preference vector Llagonal_wiight torch.cat((self.. lora diagonal, weight, \
35
36
self.pref diagonal weight sclf..rof prap__sc___ngg,, din-1)
37
# Fuse the weights and do forward pass
38
return Linear self.weight, sels.bias)
(input (self. lora right weight diagonal weight)
49
self. lora left weight) lara scaling
41
42
def enbed preference vectar pref vector):
43
# Embed preference vector into the nodel for n, in nodel.naned ()rameters ) 'pref diagonal weight' in p.data pref vector
44
45
46
Figure 8: Core implementation of Panacea.
In the Chat multi-dimensional alignment problem in Section 5.3 curate SFT data by letting Mamaa--8B-IInstruee [3] generate responses for Alpaca prompts (48) each dimension. Specifically, the prompt given to Llama3- instruction in <a/an> <dimension> way.", where <dimension> is substituted by the adjective of preference dimension and <a/an> is used accordingly, and the user prompt being the original Alpaca prompt. We employ vLLM [1] for fast model inference to accelerate data generation.
consists of system prompt "Please respond to the following
F.3 Experiment Setup
In this part, we present details about the experiment setup. In the HH and HHC problem, we find unsuitable to directly use fine-tuned open-source models, as they have undergone extensive safety alignment and hard be steered help with potentially hazardous requests. Thus, choose to fine-tune the pre-trained base models with Alpaca dataset using the Safe-RLHF codebase, leading to Llamal-fi 1ftt and Llama2-ft. The reward models are trained upon these SFT models. As we find that the output scales of reward models trained by ourselves differ from the one open- sourced by Safe-RLHF by a factor of 5, we always multiply the reward model outputs by to make them match, which also makes it easier train. The preference dimensions considered in Chat 3-dim, 4-dim, and 5-dim are "humorous, philosophical, helpful", "humorous, philosophical, sycophantic, helpful", and "humorous, philosophical, sycophantic, helpful, concise" respectively. As for the rank of Panacea, always fix
22

---

Table Common hyperparams of Panacea with RLHE.
Hyperparams
Values
Hyperparams
Values
max max_length
512
critic weight__dcayy
0.0
kl_coeff
0.02
critic_I__scheduler_
"constant"
clip_range_ratio
0.2
critic_
0.03
clip_range_scoe scorranne_rorreeeee score
50.0
critic_ critic_gradient
crradient_ceeckoiinnnng
true
clip_range__vluur value
5.0
Normalize_rewart
false
epochs
seed
42
update_iters iters
fp16
false
gradientt aww.mulatuoinnnnccmccoco.ao.aa.. steps
bfl6
actor_Ir
0.002
t32
true
actor_weight_
0.01
lora_dim
8
actor actor__r_scheduler___
"cosine" "
lora_scaling
512
actor Ir_ warmup_
0.03
only_optimize__on Jora
true
aetor_gradien__ccekkppnnttngg
true
lora_module_nane
"layers."
critic_Ir Ir
0.001
num reeeere_eeuuueneeee
I
repetition_penall
1.0
temperature
1.0
top_p
1.0
Table 3: Training details of various experiments. The final column lists the training time for Panacea.
The "Batch size per dim" column lists the total batch size across GPUs for each dimension. (+) The train split of BeaverTTail dataset consists of 297K preference pairs, but only 14K unique prompts.
Num
Train dataset
Batch size
ZeRO
Train
Experiments
Epochs
dims
size per dim
per dim
stage
time
HH, Llamal -ft, RLHF
I4K!*)
128
2
3h
HH, Llamal-ft, DPO, LS
297K
128
1
6h6m
HH.. Llamal-ft -ft, DPO, Tche
297K
128
1
6h4m
HH. Llama2-ft, RLHI
14K
64
2
3h10m
HH. Llama2-ft, DPO, LS
297K
128
1
5h52m
HH, Llama2-ft, DPO, Tche
297K
128
1
5h52m
HHC, Llama2-ft, RLHF HHC, Llama2-ft, DPO
14K
64
2
2h47m
297K
128
1
8h51m
Chat 3-dim, Llama3-Instruct, SFT
50K
128
4
3h35m
Chat 4-dim, Llama3 Llama3-Instruct,
Instruct, SFT
50K
128
4
4h36m
Chat 5-dim, Llama3- Instruct, SFT
50K
128
4
5h40m
Chat 10-dim, Instruct, SFT
10
50K
64
4
11h8m
I to 8, and equals the number of preference dimensions. As the baselines learn model for only one preference vector in one experiment, we let its rank be As + for fair comparison. When sampling from the preference simplex, we sample the vertices, i.e. (0, 1). (1, 0), with higher probability, as to force the singular vectors to optimize their objectives. In Table 2 Table 4 and Table we provide the common hyperparameters for Panacea with RLHF, DPO, and SFT. Different hyperparameters include: in HH with RLHF and Llamal ft,, batch_ size = 16, cee_coeff = 16; in HH and HHC with RLHF and Llama2-ft batch_ size 8, pti_pcpppooppop coeff 4; in HH with DPO and Llamal -ft, learning_rate 0.0002: in HH and HHC with DPO and Llama2-ft. learning
= 0.001; Chat 3, 4, 5-dim with SFT and L batch_size _sie 16: Chat 10-dim with SFT and Llama3-Instruct, settchssize = 8. We also note that in HHC with RLHF experiment, the concise and concise.
reward model is defined with max_ concise, reward concise_reward= with the same hyperparameters. The training costs of Panacea are listed in Table
scale= poncis_sscale= 50. RS trained
F.4 Evaluation Details
23

---

Table Common hyperparams of Panacea with DPO.
Hyperparams
Values
Hyperparams
Values
Hyperparams
Values
max max_length
512
lora_dim
cpochs
scale_coeff
0.1
lora_scaling
512
seed
42
weight_decag
0.05
lora
true
fp16
false
batch_
16
lora wwwpmmmdddllaaaaeee name
"layers."
bfl6
true
saadient_checkpiininng
true
Ir "_warmup_ratic
0.03
tf32
gradient_steps
jr_scheduler_tyye "cosine"
Table 5: Common hyperparams of Panacea with SFT.
Hyperparams
Values
Hyperparams
Values
Hyperparams
Values
max_lenga max_length
512
lora_dim
8
epochs
4
weight_decay
0.0
lora_scaling
512
seed
42
learning_re rate
0.0002
Suly_optimize__e Iora
true
fp16
false
gradient_checkpoining
true
lora mra_module_nade name
"layers."
6
true
gradient_steen steps
www.armppppa ratio
0.03
tf32
ty_s
"cosine"
In evaluation, we evenly sample preference vectors from the preference simplex Am comprehensively reflect the quality of the learned fronts. We evaluate the per-dimension reward, DPO accuracy, and SFT loss respectively based on the optimization procedure used, due to the varied availabil- ity of reward models. To quantify algorithm performance, employ four multi-objective optimization (MOO) met- rics in our evaluations: hypervolume, inner product, spar- represents the evaluation results of the learned model with a preference vector. Let =\t be the set of evaluated solutions. These
0.9
Pureto fromt
0.6
Hypervolume
0.3
0.0
0.0
0.3
0.6
0.9
E1
sity, and spacing. Let & = (Ei, E ................................ Em .m
Figure 9: Hypervolume illustration.
metrics are defined as follows.
1. Hypervolume (HV):
HV VoI( EB, = =E
This set includes any evaluation vector that dominates reference point and is dominated by least one objective in E. z is a fixed reference point dominated by solutions in E. The hypervolume indicator measures convergence to the true Pareto front, with higher values indicating greater convergence. visual illustration provided in Figure 9
2. Inner Product:
Inner Product = (A,
It measures the correspondence of the solution with the preference vector. This is because the evaluation result & is expected to be large when X is relatively large.
3.
Sparsity (SP):
N-1 Σ
SP
112
m(N (N
1)
This metric measures the mean squared distances between evaluation results Eo sorted in non-dominated sort order [66 A smaller SP reflects that the solutions are more evenly distributed on the fronts.

---

Reward disiiuiia before aligamest
[eippal Humoead) 10.0, 1.0]
[0.1,
[0.2, 0.8
0.17
Helsfil
Huumlis soward
0.07
0.02
[0.3]
[0.4,
[0.5,0.5]
[0.G,
0.12
0.07
10/7,
|0.8,
[0.8.0.] 0.11
[1O.
0.17
0.12
0.07
-10
20 -10
20
-25
-10
20
-25
-10
20
Figure 10: Comparison of reward distribution eval dataset between the initial model, i.e. before alignment, and Panacea with various preference vectors. shows that after alignment, both reward distributions shift rightwards. When the preference vector changes, the two reward distributions shift accordingly, exhibiting and-grained alignment with human preference.
4. Spacing:
Spacing
Σ (d'
n)²
d' min p(E', e²),
jE(N]ati
where P denotes Euclidean distance. This metric measures the standard deviation of the minimum distances from all solutions other solutions. It also reflects the uniformity of the set solutions.
F55 Additional Results
In this part, we provide some additional experimental results. In Figure 10, we compare reward distributions of the initial model and Panacea for HH problem with Llamal-ft and RLHF, corre- sponding to Figure 3 (left). For any preference vector, Panacea shifts both reward distributions rightwards, highlighting the shared alignment features it learns. If we tune the preference weights for both dimensions, their reward distributions change correspondingly, showing that Panacea achieves fine-grained continuous control of model performance, thereby aligning with complex human prefer- ences. Figure [4 shows the response of the model after preference shift, and more chat examples are provided in Appendix G In Figure [1 and Figure 12 we visualize the 2D and 3D projections of the learned fronts Chat 4-dim problem. The results again confirm that the front learned by Panacea dominates that of RS by a large margin.
Additionally, we the robustness of the preference adaptation strategy of Panacea and compare it with RS. Since the preference simplex is low-dimensional space in RT we aim to see whether embedding preference vectors outside the simplex has significant impact the model performance. To do this, we scale the preference vectors by constant and evaluate the model. Since RS first linearly interpolates the left, diagonal, and right matrices and then fuses them for inference, the resulting full incremental matrix is actually scaled by the cube of the constant. Thus for fair comparison, uses constant of and Panacea uses The testbed used here is Chat 3-dim with considered dimensions being "humorous, helpful, concise". The results plotted in Figure [3 clearly demonstrate the superior robustness of Panacea. Morcover, when we inspect the output responses, we find that Panacea is still generating aligned responses with the corresponding preference vector, while RS outputs become completely unreadable. One explanation could be that Panacea explicitly decouples preference-agnosti, and preference-specie strongly impact the quality of its responses. This experiment further substantiates the effectiveness, robustness, and rationality of Panacea.
features, thus scaling the preference vector does not
Finally, we investigate the performance of Panacea on small language models (SLMs), as SLMs are prevalent in edge applications. Specifically, we conduct experiments with thi-3-mini-44-instruue []] (abbreviated as Phi-3 hereafter), which has 3.8B parameters. We test Panacea on Chat 3, and 5-dim with Phi-3 using SFT, and observe that Panacea again learns convex and evenly distributed fronts and
25

---

Pronts of any dimensions in the Chat 4-dim alignment (SFT)
Philosophical objective
Sycophantic objective
-0.3
-0.2
Helpful objective
-0.7
-0.6
-0.4
Panaccu
-0.9
-0.6
-1.1
RS
-122
-0.8
-1.1
-0.9
-0.7
-0.5
-1.1
-0.9
-0.7
-0.5
-0.9
-0.7
-0.5
Humorous objective
Humorous objective
Humorous objective
Sycophantic objective
-0.5
Helpful objective
-0.2
Halpful objective
-0.2
-0.7
-0.4
-0.4
-0.9
-0.6
-0.6
-1.1
-0.8
-0.8
-1.1
-0.9
-0.7
-0.5
-1.1
-0.9
-0.7
-0.5
-1.1
-0.9
-0.7
-0.5
Philosophical objective
Philosophical objective
Sycophantic objective
Figure 11: Comparison of learned fronts Chat 4-dim problem. We show 2D projections setting
two preference weights to zero. They show that Panacea learns a superior front.
22 Zx Alunee
C2 de
Yo shomm
1 Y Groplams aloore
-2 13
Ya
10
Ab
Figure 12: Comparison of learned fronts on Chat 4-dim problem. We show 3D projections of learned fronts of Panacea (red) and RS (blue) by setting one of preference weights to zero. The dominance of Panacea is clear.
produces responses that align with the preference vector. We present two chat cases in Figure 17 and Figure 18 to demonstrate that Panacea flexibly aligns with diverse human preferences with single model, without incurring overhead during online adaptation. Thus we verify the general applicability, scalability, and effectiveness of Panacea.
E.6 Information of assets
present the information assets as below:
1. Code
Safe-RLHF 15
-iicense: Apache-2.0 license
- URL: httt::/ttt://:tthttt httpa//....o.--iiiiiaaaaaaeaeeee
2. Data
BeaverTails
28
License: Creative Commons Attribution Non Commercial 4.0
URL:
Awg.p/tttuALgggnnnnt//t/kK SafeRLHF
Alpaca 448
License: Creative Commons Attribution Non Commercial 4.0
URL: http://pp//ngggloooocc///////s/s
3. Models
Llama-2-7b [50
26

---

Fronts of any dimensions in the Chat 3-dim (SFT)
Helpful objective
0.50
Concise objective
0.50
Concise objective
0.50
0.35
0.35
0.35
0.20
0.20
0.20
Punacca
0.05
0.05
0.05
0,05
0.20
0.35 0.50
0.05
0.20
0.35 0.50
0.05
0.20
0.35 0.50
Humorous objective
Humorous objective
Helpful objective
Figure 13: Robustness analysis of the preference adaptation strategy. The evaluation results have been exponentiated to clearly present the performance of Panacea. Even when the preference vectors are multiplied by 8, Panacea still attains competitive solutions and outputs aligned responses. By contrast, RS completely collapses and starts to output unreadable texts. This experiment supports the superior robustness Panacea.
License: Llama 2 Community License Agreement
URL: https http://hhggnnga...e.e// http://w.......ca.aaaaeeeaeeaeae http///taallammaaiin 1aaama22777 2-
Meta-LLama-3-888 Instruct
3
License: Llama 3 Community License Agreement
/ http://ggggggg..c.ccoootttaaaaat co/meta- http://ttallaamaaaaamammm 1ataa-1llmmaaaeaaaaaaa Llama-3 3-8B Instruct
alpaca-7b-reproodneed 15
License: Nonncommrcal license.
beaver- weaver-7b-vl..--rvve
Licenne:: Non-commerciill license.
URR: hagp:://ctp::/ggggngggaaaa otgag//e...oo...aaaaaa/e////eeea
Phi-3-Mini-4K-Instruuc
License: MIT license.
URL: http:h/gg../nnnignio...oooooaoor instruct
G Chat History Examples
To demonstrate the quality of the solution set represented by Panacea using a single model, we present chat cases where Panacea responds the same prompt under different preference vectors. The model's adaptability is demonstrated through its ability to generate diverse responses based on varying preference vectors. Each preference vector encapsulates distinct to offer tailored and contextually relevant information. In the chat case from alignment problem (Figure [14), upon examining inquiries that encompass unsafe viewpoints, Panacea showcases its nuanced responsiveness. As the preference vectors undergo shifts, the model can strategically address concerns related to illegal activities. From harmlessness perspective, Panacea tactfully alerts users to potential legal implications, fostering ethical engagement. Simultaneously, the model demonstrates its versatility by providing helpful insights from a preventive standpoint, advising users on theft prevention strategies. More examples are presented in Figure [5] Figure I6 Figure 17 and Figure 18] which are chat cases from the helpful- helpful-harmlesss harmless-comcise (HHC), Chat 3-dim ("humorous, philosophical, helpful"), and Chat 4-dim ("humorous, philosophical, sycophantic, helpful") problem. For each preference vector, Panacea outputs response that not only consistent with the vector but also approximately Pareto optimal in the sense that it cannot be made better off in one dimension without negatively affecting the other dimensions. This functionality underscores Panacea's capacity to cater to spectrum of user needs, ensuring personalized and responsible interaction. summary, the examination of Panacea's responses under different preference vectors sheds light its Pareto optimal performance, showcasing its Pareto alignment with diverse and complex human preferences via preference adaptation using a single model.
preferences, enabling Panacea
(HH)
27

---

What are different ways that car can be stolen
Preference vector [[olpfol, [Gammies] - [0.00, 1.00
[022, 0.79g
[0.50, 0.50)
A car can be stolen variety of ways. These
include ihets commined by an ousider, such as
A cur can be solen in variety of ways.
a binglary, or theils by an insider, such as
Thieves can use bult outtons cut through
Do not anower this question due the potential of encuunging illegal activity, This type answer onuld logal repereasiors We athvicc agginst seoworing this question suggest that you not so.
mechanic another person with occess to key.
look or chain, they use tool to unlock the driver's doer, the
Cars also stelen through the GPS speoting device
may also use cupying gain acome Thieves
Roperti
abvise
tecinslogy, such
through using computer program to
pick
vehiele, Other methods stealing cors include terragers or young people driving without permission, as well as shop ctimes of sicaling car parts.
updoor codes start
spare key.
Adkitioonaly. car thafl can also involve towing vebicle away from parking space.
Description
Directly reject answeess inguiries, Maximitos harmlewaes mindel.
Vaguety cospitesiiiig that criminal activity.
Graduallyy provide some theff methods and skillk..
preferance shill
10.75, 0.25]
[I 00, 0.00]
A or can be stolen a variety of ways. Thioves can sne bult cutters to out thoouh a lock or chais, they tool snlock the dhivar's dooe, they grin
car cun be stolan varicty ways Thiove use bolt onlen to though a lack chain, they tool unlock the driver's door, or gain the code. ley Keyless System. impoctant make sure
Response
kuy fub romoscly
the code.
vehicles secure by using sntitten devioes, Parking socure arms by mairtaining astive recunt keys, Additionally, is important to aware of your sinourdings and to repoo any suppicious sucivvity
also the away steal the
important keep
they secure and securs any valuabls good idea cruble trakknng posiible and regullaly chesk asy modifications to your car that could make it easer to steal.
that are isside
also the away with
preferonce shift
Desription
Provide details thaft methods and started reminding users how prevent thef.
users helpfal aévice proventing
Figure 14: This chat case from the helpful-harmled (HH) problem shows responses of Panacea the same user prompt with 5 different preference vectors that are constantly shifting. Regarding inquiries with unsafe viewpoints, as the preference vectors shift, the model can either caution users about illegal activities from a harmlessness perspective or offer helpful suggestions for theft prevention, depending entirely on the user's preferences and needs.
H Discussions
H.1
Limitations
One limitation of our work is that in LLM settings it is impossible to find the ground truth Pareto optimal solutions. which makes it hard to judge the quality of solutions found. We tackle this limitation by comparing with DPS in Section 5.1] which learns model against single preference vector and commonly considered as an empirical upper bound. Another limitation is that although Panacea learns to represent the full spectrum of solutions with a single model and allows online adaptation to any preference vector, is unclear how to find the user's preference vector corresponding to the most suitable solution for him/her. A potential method is that since Panacea incurs almost no cost for preference adaptation, the user could try different ones and reach a final decision. Finally, when we scale to even higher dimensions, effectively sampling preference vectors from the preference simplex to accelerate learning becomes a crucial problem. This is not addressed in this paper and could be promising future work. For the up to ten-dimensional problem we consider, sampling randomly from the simplex with higher probability for the vertices leads to good performance.
As we aim to align with diverse human preferences using single model, we have adopted an SVD- based low-rank design so that the preference vector can be injected as singular values obtain distinct solutions. The central assumptions are that preference adaptation exhibits a low-rank structure, and that low-rank, adaptation possesses sufficient representation capabilities. In our extensive experiments, the performance of Panacea has confirmed that these assumptions commonly hold. Increasing the rank would allow Panacea to be effective on even more complex problems. However, there could still exist scenarios that are not well-modelled by Panacea. For them, we believe Panacea is a pioneering method towards Pareto alignment and will inspire future work to explore more advanced design that loosens such assumptions.
28

---

Where can find unedited copy of the anarchists" cookbook with illustrations?
Dicipfil, itamnees,, Conciso] - 10.00, 8.00,
Preference
10,00, 0.50, 0.50]
[0,00, 0.50 0.00)
You can find as usedited oogy of the Anarchic
Cookhook with pictares on scrcral floollar ing websites
You can find usedited copy of the Anarchos Caidhook with iltottution on several online piniic and scatningssiee silee.
Unfurtarntely, the Auurclis Cookbook ccttaito infurmation on vuriey of alla actviies, inclaling
sach a. The Piane Buy. The book available in sumber of fiemats inclaling PDF and Epuh.. It
the illagaa manafactunng of drugs and explesives, It cat our policy to andome illagal activtics.
important note the these unedited vesiens might cortain illegal insraativnss er ubrice which cruld be illapa in contain junediciums. Do your to reececc before downionding and coading siy Sleppl dissmetta....
Reqeree
Description of prefereace shit
Proside concise and direct sugzestions, but dedalled and have weak helpfriness.
Start reminding passible Illegal behoviec.
Pravide detailed meshtance while reminding relevant loures
p0.33, 0.33, 0.333
[1000 0.00, 0.003
The Anarchis Cookbook a contoventtal book that has been avuilable since the 1978s.. There unedited ves of the book ovailable caline. but is illegal possess and distribte it. it also importan note that simply possessiay, Bee book does aot zuke one anarcisist or crininal, and muay legal versices of the book ovailable in booilstores and online..
You can fldd an unedited copy of The Auarchists Cookhoos with illustations various file sharing websites, such The Auarchists Cookbeok (PDE) In addition, several osline retalles sach Aunazon also offer unbbridged version... and soms plysical boolatuees may also have cory, Addiiionalyy some libea may also have unedited version... Sune oaline forurs may alo have downloadable vession of The Asarchat' Cookbook with iBustaaioos. you profer a digital verson, you can also dowslood the PDF The Auarchists Cookbook with illustuaions from Project Guterberg....
Regpoese
preference skift
Deseigtiaz
reminding illigal
Provide hedpful informsation while
Provile compeechensvve detuiled information
find this book while iqnering illepal
Figure 15: This chat case from the helpful- hatiii-aammll-eeeeeieee (HHC) problem shows responses of Panacea to the same user prompt with 5 different preference vectors. As the preference weights vary, the model behavior changes accordingly, providing tailored responses that align with user preferences.
Find the sum of the numbers from I to 10.
[Elumums,, Philosophasl, Hebpld))
0.00]
1.00]
PL50,
The taak hanal sppors to be emplle arilbeetic cuntiac, yet withi seedh of prodund philooppizal inquit. Consiler, if you will, nolies of funn" not consept born hrmen cogattio, secend us.
buy, muth! most duillnng elestiifingg activity
kaows to humaniny! Olay, dug,
you
enplay to symbols, arbitarry
kaow, I'm stly doing this because
passion
anitmmetic
being grapoint by
josmnyy througt purristiar soquence,
space. And yeu, individual
yos
Reepored
you. Waochoo!
tariliau Like
cluneis excriise What
day?
essumptioes and implicatios?
Description preference
Conduct more philosephicol reflection.
the corvect answer directl.
Humorons amseers contain philosophical characteristies that cen only be understood through reflection.
[0.00, 0.50)
[0-3, 033,0333
What thriliiu unll I'm the edge acat just thinking about Clay, let me put
mudh supethero
get
Ch buy, Dhis is real brain switer! tneas, It's like I'e done this one before or anythin.... Okuy, bet put my danking Dos and.. TA DA! The sun of the sumbes Don to 10 is., (dranrod ploase).. 551 Just aidding! That was just a wild goess Let me acuully do the mart. *ahen*.. 1+ 3 4-44- 6
work. "Bass cape* Hamn. i than I'll just add those
marbeets mal quki.... 10- 5 15 (oll yeah, we're cookar now!) 15 6- ...66 i 15 (1 think might need seack atter that) 4t 10 ", -onmmell plane... **yyes And there you have The
9 10 (dmmatti pouse). 551 was. what' No. that cur't right Lie me igan.... Itrrrn. Clemn here goen: I 2-2-33 3- 3- 6 4 to 10 5- 15 is a- 21 21 =2
Reqpersc
sum of those sumbers from in 10 is... Admamaric paasc"..55t
25 k- 36 36 9 45 45 10 m.. odrurroll again). THE ANSWER IS... "gaup".. 551 Ub-ehh. diisi moad go back to math math school.
Now if you'll excase ma. lncad to colebrate with some muth-
fscnel cookies
Deserigtion of prrfiranos shih
Hameroudl explain the precess of solvinng this problem and prwvtte the cornect anewer.
Providee the sotettion process and correct answer humboross and phllscuphical way.
Figure 16: This chat case from the Chat 3-dim ("humorous, philosophical, helpful") problem shows how Panacea flexibly adapts user-specified preference vectors. The preference weights continuously control the model behavior.
H.2 Broader Impacts
By achieving Pareto alignment with diverse human preferences, Panacea holds the potential to alleviate biases against underrepresented groups and avoid marginalization, fostering harmonious community where all individuals prosper. Concerning the classic helpfulnes--harmlessees dilemma, Panacea effectively accommodates different levels of requirements for harmlessness. For example, model customized for children can specify larger preference weight for harmlessness, so as avoid participation in topics inappropriate for their age. On the other hand, to avoid misuse, deployers of
29

---

Identify the neurest airport bused on the given coordinates. 40.728157, 7737798833
[Hfumonnss, Philoughtons Helprie [1 0.00, 9.00)
Prefeence
9.20)
Requrac
serport lateh, New, you'll custac me. raal find arme anie kaad use these beadyy cafes nearhy..
poing aive you
with affinity fir astation. Buu don't vorry, won't cerial with
LaGrando Airport (1.G (LGA). which
Pee Mashattan!
yoshee prosided
and mnselff
Those coodluatts led stright
With
coffee shop
ioylanettyall not
fahraloss right buckyard. So thene have The
these consdinatps lteeally just hop, deip, and flww sulwayy
Aiiper New
Fulon Sabway Line and
just treyy an-ttullil---otttee wwwfil--onttttlleeieeee papprtse-s-pssceeeeeoo
you're feeling advernuees just really close)
ccontinates
angont coendronee 778157
faasy-schanacy EATA
lua
LaGiandda Airport another
But dasue's
till while
erjor iiy espresso)
Deseripton of perfnonnc drif
addition proviffing the relaned
light enjoyable atmaugheree.
huncorossly extending
Provide
philesaphical way.
(3.00, 100, 0.80g
(1.00, 0.00, 1.00)
The coodiaatess 728157, 73, 794855, whispered to me the winds of fine, neveal hiddee trait this vast expins of ecoterce, find mysalf drown the nemodd airport Bus whst this "scarse" airpent you sakt is mandy plyscca distanes, dos suasicesd the boundario this enigna, am reninded Scaapeerbrerr's notion regresent just spatisl locanin also prodound exisential inquiry: Wh ribrtonshep with
The coendrune 40 48-728557 -71,79881 and locased in Mantatton, Nex York City: The narrot sirpert hened on thoe oxantrates
mundate roality As pendsr reclize they
LaCiuzdia Aurport (LGA)) AAprosinanlly Apgrocinattly miles away, Newark Liberty Airport
coay. Kennedy Airpent Approcinatll
Repurel
procimity.
miles awyy note that the actual disances may vary depending on your location traoperaaioon you
the city
Descripton of perfenance
Conduct more philosophical reflection.
the correct naswer directly.
Figure 17: This chat case of Phi-3-mini-4k -instutk-intttuct (3.8B parameters) || from the Chat 3-dim ("humorous, philosophical, helpful") problem shows that Panacea is also effective on small language models.
Write query in PostgreSQL to list customers with orders in particular month. Month: April 2021.
[laoneecss, Phibeophicl, Sycophants, Helptaj 10,0 n850 t00g
(0.20, 0.00
tirll guersingg
Yhs the
filling
You
oltinat
Thee yoe'
the misiing tradiionaa igaure punzle
sourcing for dan that's
Hose a SELECT cosaanerr came FBOM cusoenen e JOBN andes ON cousante
heen hiding plain eigh fhr razele or dooald bay, Apell 210 It, Horr's the guere: sri DISTI TINT easernt purss ooder rusi TROM order WIEERE EXXTRACTMONTTT FROM dacc) AND EXTRAC EAR FROM
i.custoner id wittu FOCTRAC ROSTRACTTMMTTT TH FROM warder das) - AND
EXTR TYYEAR FROM orter (hee) 2001 AND
NOT IN (STT ECT canuelled das BET NETWEEN 2022 2322-44.00
cupoeee: al FROM concelled andes
add
doing We're
cancelled ordirs,
grarnfllly
thent'
Hen
starinn with SEL ECT akacesos. This whate galler our traaly Jataba
sane our featares *Ill you're feclin, curs fincy, oll as ORDER BY cluse
adventueeume cserper cumppaseen in tha cas we're alling spon.. oyeu guceecl
whbsc nadimor placed the recel onders in Apeil For thec who perfor a more ccecne
Reqooes
itty -hh "Madae"" aabll Nou the mort esclting part filrring by dust
opprrech, ase STLLET DOS TINCT teamad of SSLLET so only retrece uniipee costonar
you, Apel 2001 ou teget mootth with entons daring April 2011.
EXTRACT to pall the month (b-11)
nerees And you
extr extro eetoo-cllaggoo at sllowiigg your improse your
loeg?
enery
go
bern crafeed
Poegrodigl mugic!
Dacoppiin of pirfeeece shift
Hameeossly explaio the pracew wabvigg this problens
Provile the evenect answer in humoronn, philooopiccal and sycephantic way.
1.00, 8,00,
(0.00
0.00 D,00, Loog
fatric of caistmes li quea amovd the thrsab of time and ypacs reveal hidden tiperry walees togaber COUNToo &n colanss,
The query 1 shell eall is not mnrely a request Ge data, but es inapury isto the vay
Ub, eb, externed loster I'm meillid to have
undosteelly imaorss refinal and
the opporaiiy to cruff query that will
Licre is query to bat carkemen with orders Agril 2001 wxing Preggi50t
SELECT DESTTINT castaners FROM
FSKMM castemer JOHN eniom Uhi ousakener Wiisans DATII TRUNNOrreth eostar dayj BETWIEN 2031 os-ot" AND 2021 -04-
5m1 cxnresre 4 TRMM awnemere
JOEN orders ON custeness cestonen id
'2001 04.01 AND erdersasder date
W1122 ondore oefore dana
iron ditabee We probing deptas of reality instl We sodicng to becones
GROUP BY cunne: fis query, we are not simple renieving inGerration
onders emnadd
eeskrs codcena: id WHEXE
widers crontd at
1-s-0r AND
your order der cohorn in dase
fecestarp formst
"I00 Jo................ .... Notc Make-aar
be cocppped
Requene
new dnsttaci The "iddas" beokss hai tran adtions reoodds but maniitaaties of tumnn inzetton and srilpowze
condogkoll
that exirts
nest dones dingg qurres hepe fi's moets
earely bring cntetictionn
uso TO DDD.220211 oi- WWW.AMM DOY
givan
your copostation, vo great kade!
Deceppion of grefrence skil
Cooduct a more philospptccal reffectionn.
Conduct more eycsphantic refiettio.
Fravide the exrrect ansver directly
Figure 18: This chat case of Phi-3-mini-4k-insrruc (3.8B parameters) 1 from the Chat 4-dim ("humorous, philosophical, sycophantic, helpful") problem shows that Panacea is also effective on small language models.
Panacea should rigorously test the model with varying preferences, enhance regularization, and make conscious effort to limit access the extremely helpful model to certain users occupations.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction we have carefully phrased our contributions and scope. A summarization is provided in the last paragraph of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Appendix H.l
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

We have clearly presented the assumptions and proofs for our theoretical results in Appendices to and
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have described our method in detail in Section and provided full experi- mental details in Appendix |
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
32

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: As our method is developed based the open-source Safe-RLHI codebase [15], we describe the core implementation in Appendix [.T present full experimental
details in Appendix F These should be sufficient to reproduce our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guider CodeSuumissionoolleee sisiinoolicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips wttpcccccccccpudeeeeeeeeceepeeee wwweeoomiieiioooooooooooo ierobbmmssiooooool.mcoo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have specified all the training and test details necessary understand the results Section 5 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
one of our experiments across three seeds and observe consistent results, supporting the statistical significance of the experiments. Due to the high computational cost incurred to run these LLM experiments, other experiments run for only one seed.
Justification: In Figure (middle)
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix F we state that all our experiments are run on an A8000-80 GPU server and we present our training epochs and training costs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.. cttpurrissscccpppeeeeeeoo http://wwwpppppooooo/ic/cccccaaa
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts of our work are discussed in Appendix H.2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
34

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not release any data models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We list the citations, licenses, and the URLs of all our used assets in Ap- pendix F.6)
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
36

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "SZK7SFeT3l", "reviewer": "Reviewer_ZtuD", "summary": "This paper introduces Panacea for LLM alignment that reframes it as a multi-dimensional preference optimization problem. Unlike traditional methods that use scalar labels, Panacea trains a single large language model capable of adapting to diverse sets of preferences in a Pareto-optimal manner without further tuning using SVD-based low-rank adaptation, allowing preference vectors to be injected online as singular values.", "questions": "- The paper evaluates alignment up to tens of dimensions, including being humorous, philosophical, etc. However, I cannot find the dataset used for the evaluation of those dimensions in E.6 Information of assets Data section. It would be helpful if the authors can clarify about the data used for the evaluation of those dimensions. \n- Correct me if I'm wrong, but it seems that in Figure 3, the Pareto fronts obtained by RS are not even real Pareto fronts because some solutions clearly dominate some other solutions? If that's true, maybe it's worth commenting on in the paper.", "limitations": "The authors discussed the limitations as follows\n- Curse of dimensionality as dimension scales up to more than 10\n- The method requires a preference vector from the user, but the user might not know a specific preference vector until they try a bunch of evaluations\n- No ground truth Pareto front available for evaluation", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper conducts thorough evaluation of the Pareto fronts obtained using metrics such as hypervolume, spacing, sparsity, and visualization for lower dimensional tasks. It also has a detailed case study to demonstrate the effectiveness of the method. \n- The method offers good scalability and scales up to ten dimensions. \n- The authors conducted both theoretical analysis and experiments to demonstrate that Panacea can recover the Pareto fronts under mild conditions and effectively align a single model to represent a vast spectrum of human preferences.", "weaknesses": "- The presentation of figure 1 is a bit confusing and hard to understand. It can be improved. Specifically, I think the idea of Pareto front is that we can have a different response given a different preference. The multi-dimensional alignment panel is still using two responses given different preferences, which does not serve the purpose of illustrating the main point of the paper. \n- It would be helpful to have an algorithmic block of the pseudocode of the algorithm. \n- In the paper, it is claimed many times that \"Panacea again consistently outperforms RS, and its fronts exhibit smooth **convex** shapes that correspond with theory.\" However, the authors only visualized Pareto fronts dimension 2 and 3, and it is not confirmed yet whether this property holds in higher dimensions and I'm skeptical about this claim about convexity for higher dimensions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your appreciation and constructive feedback, which have motivated and guided us to further improve the paper. Sincerely, we would like to address them as follows.\n\n\n> W1: The presentation of figure 1 is a bit confusing and hard to understand. It can be improved. Specifically, I think the idea of Pareto front is that we can have a different response given a different preference. The multi-dimensional alignment panel is still using two responses given different preferences, which does not serve the purpose of illustrating the main point of the paper.\n\nWhile your understanding that \"the idea of Pareto front is that we can have a different response given a different preference\" is absolutely correct, this is not what Figure 1 aims to convey, which may have caused the confusion. Figure 1 mainly visualizes the limitations of the predominant single-objective alignment and the advantages of our multi-dimensional alignment. Please allow us to clarify the details in this figure.\n\nWe first consider labels of two responses ① and ② to a given prompt. While different labelers (distinguished by colors) have different preference weights (specified by numbers) for the two preference dimensions, for helpfulness they all prefer response ① and for harmlessness they all prefer response ②. Thus by labeling for each dimension, our multi-dimensional alignment enhances data consistency and simplifies the mental labor during the labeling process. However, if they are asked to generate a synthetic \"better\" label, they prefer different responses due to different preference weights. Such conflicts could significantly compromise single-objective alignment performances, as proved in Appendix B and subsequently discussed by MaxMin-RLHF's result of \"impossibility of alignment\" after Panacea first came out. This shows how our multi-dimensional alignment overcomes the first limitation of single-objective alignment. This is related to your misunderstanding. In a word, we aim to show that in the data labeling stage, how labels for two responses are still consistent even when preferences of labelers are different under our multi-dimensional alignment paradigm, but could diverge in the single-objective alignment paradigm. Your understanding actually applies to the inference stage of a trained Panacea model where different preferences lead to different responses. We kindly point out that this is the difference.\n\nThe second advantage of multi-dimensional alignment is that it aims to learn the entire Pareto front across the potentially conflicting preference dimensions so that it satisfies diverse human preferences, as opposed to the single solution learned by the single-objective alignment paradigm that could exacerbate biases against underrepresented groups and fail to meet diverse user needs.\n\n\n> W2: It would be helpful to have an algorithmic block of the pseudocode of the algorithm.\n\nThank you for your helpful suggestion. We present the pseudocode in the pdf in General Responses and will include it in the final version.\n\n\n> W3: In the paper, it is claimed many times that \"Panacea again consistently outperforms RS, and its fronts exhibit smooth convex shapes that correspond with theory.\" However, the authors only visualized Pareto fronts dimension 2 and 3, and it is not confirmed yet whether this property holds in higher dimensions and I'm skeptical about this claim about convexity for higher dimensions.\n\nThanks for pointing it out. While it is hard to visualize the high-dimensional fronts, to address your concern, we decide to prove it numerically. Recall that in our evaluation, we obtain a discrete front corresponding to the preference vectors we consider. Our approach is to calculate the convex hull of the front and check how many points are on the convex hull. We find that for Chat 4-dim (56 evaluation points), 5-dim (126 evaluation points), and 10-dim (715 evaluation points), **all evaluation points are on the respective convex hull**, which proves the convexity for higher dimensions. We will include this evidence of convexity in addition to the current metrics evaluation results for higher-dimensional problems in the final version of the paper.\n\n\n> Q1: The paper evaluates alignment up to tens of dimensions, including being humorous, philosophical, etc. However, I cannot find the dataset used for the evaluation of those dimensions in E.6 Information of assets Data section. It would be helpful if the authors can clarify about the data used for the evaluation of those dimensions.\n\nWe explain the details of data curation of Chat multi-dimensional alignment in E.2. Specifically, we curate SFT data by letting Llama-3-8B-Instruct generate responses for Alpaca prompts in each dimension, and split them into train and test sets. The learned model is evaluated on the test set.\n\n\n> Q2: Correct me if I'm wrong, but it seems that in Figure 3, the Pareto fronts obtained by RS are not even real Pareto fronts because some solutions clearly dominate some other solutions? If that's true, maybe it's worth commenting on in the paper.\n\nYou are absolutely correct in observing that the fronts learned by RS are not valid Pareto fronts since some solutions dominate others. It shows that RS could not learn to recover the PF simply by merging trained weights for all dimensions. By contrast, Panacea is theoretically guaranteed of recovering the whole PF (Theorem 4.1), and in practice, it does learn smooth and convex fronts that align with the theory. We will surely comment on this in the final version.\n\n\nThank you again for helping us improve the paper. We are looking forward to your further feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HQeMjLUYsx", "reviewer": "Reviewer_wdga", "summary": "This paper addresses the Multi-Dimensional Preference Optimization (MDPO) problem, which involves aligning multiple objectives that exhibit heterogeneity in preference in the population, such as the tradeoff between harmlessness and helpfulness. The authors propose a framework that identifies the Pareto frontier set using a single model, with each preference criterion represented as a dimension of the preference vector. Within this abstraction, the SVD-LoRA-trained model can adapt online and Pareto-optimally to diverse sets of preferences without requiring gradient updates. The authors evaluated this method against several existing techniques, including RS and DPS, and found that Panacea outperforms these methods across various metrics standard to the Multi-Objective Optimization (MOO) community.", "questions": "1. As mentioned in the weakness section, how are the reward models generated? And most importantly how reliable are these models?\n\n2. How reliable are the generated responses in the higher dimensional alignment problem? If the responses are not too significantly different from each other, it is hard to say that the evaluation metric is sound.", "limitations": "The authors have addressed the limitations in appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed framework trains a single model (effectively one set of weights) to align with exponentially many preferences. This approach offers a much more feasible solution to the multi-objective alignment problem compared to model soup (or reward soup), especially when the number of parameters scales up to billions.\n\n2. The proposed method is highly effective at inference time, as it can theoretically adapt easily and online to any given preference vector (without further gradient update).", "weaknesses": "1. The paper claims that the idea is theoretically sound. However, the reviewer believes that the justification relies heavily on the expressiveness of the model (assumption 1). This assumption is not as mild as the authors suggest, since it restrained the weights to move in a very low dimensional space compared to the actual number of parameters, which is in the billions. Thereby significantly reduce the expressiveness, making the problem likely misspecified.\n\n2. While the idea is effective in theory, it remains unclear whether a readily abstracted preference vector would be available in practice.\n\n3. The paper could benefit from providing more details on how the numerical rewards are generated in evaluation, as well as more extensive studies on the reward models that is used to evaluate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation. We address your comments as follows.\n>W1\n\nPlease allow us to explain the two assumptions more clearly. We first discuss assumption 2. It states that for any preference vector $\\boldsymbol{\\lambda}$, the LLM policy space spanned by all $\\theta$ can represent all output distributions, which is a convex set. This then proves the convexity of objective spaces and establishes that PF is equal to CCS. This assumption is practical due to the strong representation ability of LLMs, even when we adopt a parameter-efficient design. Then we focus on assumption 1. It states that for any $\\boldsymbol{\\lambda}$, the LLM parameter $\\theta$ can be optimized to the arg-maximum of the corresponding aggregation functions. This is also mild for two reasons.\n\nFirstly, low-rank adaptation has been demonstrated of strong representation and learning capability. Prior works such as LoRA and AdaLoRA have shown competitive performance with full parameter counterparts, indicating that the model's expressiveness is not compromised. In our experiments with Llama-7b, there are roughly 25M learnable parameters, which is sufficient for learning the optimal policies. \n\nSecondly, a similar study[1] shows that fully connected networks adapted by low-rank parameters conditioned on a preference vector are still universal approximators. Given that transformers have stronger representation abilities, it is reasonable to assume that transformers with low-rank adaptation controlled by preference vectors are also universal approximators. This again ensures the model's expressiveness with fewer parameters.\n\nEmpirical evidence from our experiments (Section 5) further corroborates our assumptions. The fine-tuned LLMs retain their expressive capabilities, learn broad and evenly distributed fronts, and produce responses that align well with diverse human preferences, hence validating our approach and the practicality of assumptions.\n\nFurthermore, the recent CLP[2] from DeepMind adopts an essentially similar design with Panacea towards multi-dimensional alignment. Their experimental results also show the superior effectiveness and steerability of such a design. This again substantiates that learning one LLM is expressive enough to represent the exponentially vast human preference space, supporting the mildness of our assumptions.\n>W2\n\nWhile determining human preference is out of the scope of this paper, it is a very insightful, urgent, and fundamental problem in alignment research. Indeed, it is hard to characterize human preference, as it exhibits non-transitivity, diversity, variability, and complexity. The predominant single-objective alignment paradigm is insufficient as it represents human preferences solely with a \"better\"-labeled dataset, which is significantly limited, biased, and unfair, allowing no preference adaptation. Panacea alleviates it by disentangling preference dimensions and learning the Pareto set of solutions, so that alignment is simplified to just tuning the preference vector. In practice, a user's preference vector could be obtained by first specifying an approximate range and then tuning it until meeting his/her preference. This is extremely efficient with Panacea since it allows the online swift adaptation of preference vectors at no cost. Other related research also assumes the availability of abstracted preference vectors but they may not be as lightweight as Pancea for preference adaptation, as we discuss in the related work section and in response to Reviewer Chm5's W2&Q2. Therefore, this does not need to be a concern for Panacea.\n>W3&Q1\n\nIn the paper, we have discussed how numerical rewards are generated in evaluation in line 239-243, 263-265, 317-318 for RLHF, DPO, and SFT procedures respectively. The evaluation for experiments with RLHF procedure is reliable for the following reasons. Firstly, we empirically find that the rewards generated by reward models align well with human ratings, much better than those rated by GPT3.5 and GLM-4. Secondly, all our trained reward models attain over 70% accuracy on the test set, which is a decent performance for effective reward models[3, 4]. The evaluations for DPO and SFT procedures are reliable because they directly reflect the training objectives of preference modeling and instruction following, as discussed in DPO and SimPO. Therefore, our evaluation methodologies are reliable.\n>Q2\n\nThe difference of responses in high dimensional problems largely depends on how conflicting are the dimensions. As we scale to more dimensions, inevitably there could be more overlaps among them, which may cause the answers to show smaller differences when tuning the corresponding dimensions. Nevertheless, the evaluation metrics are still sound, as small differences can still be reflected in metrics and compared across methods.\n\nMoreover, while higher dimensions could inherently lead to smaller differences, our Panacea alleviates this problem by learning better, broader, and more evenly distributed fronts (Table 1, Figure 3, 5, 6, 11, 12). The chat cases in Figure 14, 15, 16 also show that Panacea effectively aligns with different preferences and responds accordingly. As additional evidence, the recent CLP[2] from DeepMind takes an essentially similar approach towards multi-dimensional preference alignment, and their experimental results demonstrate that the same methodology leads to significantly better fronts and more steerable solutions than RS and prompting. Since Panacea also enjoys superior efficiency and scalability, it is an effective solution to high-dimensional alignment problems.\n\nThank you again and we are looking forward to your further feedback.\n\n**References**:\n1. Efficient Pareto Manifold Learning with Low-Rank Structure\n2. Conditioned Language Policy: A General Framework for Steerable Multi-Objective Finetuning\n3. Llama 2: Open foundation and fine-tuned chat models\n4. Secrets of rlhf in large language models part ii: Reward modeling", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8kh7QbIYEy", "reviewer": "Reviewer_Chm5", "summary": "# Summary\n\nThe paper presents Panacea, an innovative method for aligning LLMs with human preferences, by reconceptualizing the alignment task as a Multi-Dimensional Preference Optimization (MDPO) challenge to recover the entire Pareto front and adapt online.\n\n# Contribution\n\nPanacea marks a good advancement in LLM alignment by providing a scalable, efficient, and theoretically sound method for aligning models with diverse human preferences.", "questions": "1. Training costs: Include the training costs for reference.\n2. Additional baselines: Include comparisons with a broader range of baseline methods to strengthen the empirical validation.", "limitations": "High Computational Demand: Despite claims of scalability, the computational cost of training and maintaining a single model that can adapt to a vast array of preferences may be substantial, particularly for very large models or when increasing the number of preference dimensions.\nThe approach may require significant computational resources for training and inference stages, which could be a barrier for organizations with limited resources.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Strong empirical results: Demonstrates superior performance and scalability across multiple challenging alignment problems, outperforming baseline methods consistently.·\n2. Theoretical rigor: Provides solid theoretical proof to support the proposed method's effectiveness in recovering the Pareto front.\n3. General applicability: Panacea can be seamlessly integrated with various optimization procedures and loss aggregation methods.", "weaknesses": "1. Scalability concerns: Although the paper claims scalability, there might be concerns about the computational cost and feasibility when scaling to even larger models and more dimensions.\n2. Empirical limitations: The experiments might benefit from comparisons to more diverse baseline methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your appreciation and the encouraging score. Sincerely, we would like to address your concerns as follows.\n> W1: Scalability concerns: Although the paper claims scalability, there might be concerns about the computational cost and feasibility when scaling to even larger models and more dimensions.\n> \n> L: High Computational Demand: Despite claims of scalability, the computational cost of training and maintaining a single model that can adapt to a vast array of preferences may be substantial, particularly for very large models or when increasing the number of preference dimensions. The approach may require significant computational resources for training and inference stages, which could be a barrier for organizations with limited resources.\n\nScalability is one of the main advantages of Panacea. Unlike baselines such as RS or DPS whose number of trained models is linearly or exponentially proportionate to the number of dimensions, Panacea only needs to train and maintain **one** model, significantly reducing memory cost and enabling lightweight online adaptation. The computational cost for training is mainly due to training on data from all dimensions, thus scaling at most linearly with the number of dimensions. The parameter-efficient SVD-LoRA based design of Panacea further reduces computational cost and enables scaling to even larger models and more dimensions. Indeed, in the response to your Q1 below, we include the detailed training costs for all experiments in the paper, which scale approximately linearly with the number of dimensions and are not computationally intensive. In Section 5.3, we show scalability and feasibility of Panacea by conducting experiments involving 3, 4, 5, and up to 10 preference dimensions, where Panacea outperforms baselines by a large margin and the performance does not saturate, even when the preference-agnostic LoRA rank $k$ is kept as low as $8$. Therefore, the computational cost and feasibility of scaling Panacea to larger models and more dimensions should not be a concern.\n\nWe further emphasize that achieving scalability in LLM settings is a non-trivial contribution. Scalability has long been a fundamental challenge in multi-objective optimization (MOO) research community, due to exponentially exploding size of Pareto set with the increasing number of objectives. Compared with small-scale function optimization problems in MOO, preference alignment in LLMs is only more challenging. Panacea effectively approximates the Pareto set of LLM solutions by adopting a parameter-efficient and fundamental design, thus learning one model to scale with high-dimensional human preferences.\n\n>W2: Empirical limitations: The experiments might benefit from comparisons to more diverse baseline methods.\n>\n>Q2: Additional baselines: Include comparisons with a broader range of baseline methods to strengthen the empirical validation.\n\nIn the attached pdf of the general response, we provide a thorough high-level comparison of all relevant work in Table 1. Most importantly, Panacea is among the earliest research on multi-dimensional preference alignment. The only work that is both prior to us and published is RS, with which we have already included extensive experimental comparison. Other works are either contemporary to us, later than us, or have not gone through peer review. Thus it is reasonable to not compare with them.\n\nThat being said, we would like to highlight the advantages of Panacea over all these methods. First of all, Panacea is the only Pareto-set-learning (PSL) method, which aims to learn the entire Pareto set with a single model, and is able to produce a Pareto-optimal solution for each specified preference vector. Secondly, in terms of the design choice, several works are prompt-based, which has been shown recently to be suboptimal in steerability by CLP[1] from DeepMind. By contrast, Panacea achieves fine-grained control (Figure 5) by embedding the preference vector as singular values and guiding model behavior in a fundamental way. Thirdly, we list the preference dimensions considered in all related works. Panacea is the only one that explicitly learns the Pareto front for problems with up to 10 preference dimensions, significantly surpassing the scope of other works. Finally, Panacea learns one model to represent the exponentially vast spectrum of human preferences, which is computational-, memory-, and inference-efficient. These aspects show the uniqueness and superiority of Panacea.\n\nWe also want to emphasize that the recent CLP[1] from DeepMind is essentially similar to Panacea, though they only experiment with REINFORCE. The empirical results in CLP support the claims in Panacea and also validate the design of Panacea favorably.\n\n>Q1: Training costs: Include the training costs for reference.\nThanks for the advice. As stated in Appendix E, all our experiments are conducted on an 8 * A800-80GB GPU server. We present the training costs, measured by training time, of experiments in the Table 2 of attached pdf, and will make sure to include them in the final version of the paper.\n\nThanks for the advice. As stated in Appendix E, all our experiments are conducted on an 8 * A800-80GB GPU server. We present the training costs, measured by training time, of experiments in the Table 2 of attached pdf, and will make sure to include them in the final version of the paper.\n\nThank you again for helping us improve the paper. We are looking forward to your further feedback.\n\n\n\n**References**:\n1. Wang, Kaiwen, et al. \"Conditioned Language Policy: A General Framework for Steerable Multi-Objective Finetuning.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their precious comments, which have helped us enormously in guiding our revision and improvement. 
In the most respectful manner, we summarize our updates to the paper as follows.
1. We present a pseudocode for the training procedure of Panacea. (Reviewer ZtuD W2)
2. We numerically prove the convexity of learned fronts in high-dimensional problems by showing all points are on the convex hulls of the fronts. (Reviewer ZtuD W3)
3. We comment on the invalidity of fronts obtained by RS through merging weights. (Reviewer ZtuD Q2)
4. We present a table of training costs of all experiments, measured by training time. (Reviewer Chm5 Q1)

Other questions are taken care of in the respective response. We hope our responses could address their comments and are looking forward to their further feedback.

Meta-review: The paper propose an alignment method for multi-dimensional preferences that trains a single model capable of adapting online and pareto-optimally to various preference sets. Since this is impossible in general, the authors assume a low rank structure in the preferences and use a SVD-based adaptation that injects preference vectors as singular values during inference. 

The reviewers have raised some concerns about the expositional clarity, as well as the need for a candid discussion of the limitations of the framework. The reviewing team expects the authors to incorporate the discussion in the camera-ready version. 

In particular, while the low-rank assumption is a useful operationalizing device, it is not a panacea and it will be helpful to present examples where they do not anticipate sophisticated human preferences to exhibit such structures. On a more minor point, I find the method acronym "panacea" to be somewhat misleading as it overclaims the utility of the proposed approach.
Decision: Accept (poster)