Panacea: Pareto Alignment via Preference Adaptation for LLMs

Authors: yifan  zhong
chengdong  ma
xiaoyuan  zhang
ziran  yang
haojun  chen
qingfu  zhang
siyuan  qi
yaodong  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 12:09:39

Abstract: Current methods for large language model alignment typically use scalar human preference labels. However, this convention tends to oversimplify the multi-dimensional and heterogeneous nature of human preferences, leading to reduced expressivity and even misalignment. This paper presents Panacea, an innovative approach that reframes alignment as a multi-dimensional preference optimization problem. Panacea trains a single model capable of adapting online and Pareto-optimally to diverse sets of preferences without the need for further tuning. A major challenge here is using a low-dimensional preference vector to guide the model's behavior, despite it being governed by an overwhelmingly large number of parameters. To address this, Panacea is designed to use singular value decomposition (SVD)-based low-rank adaptation, which allows the preference vector to be simply injected online as singular values. Theoretically, we prove that Panacea recovers the entire Pareto front with common loss aggregation methods under mild conditions. Moreover, our experiments demonstrate, for the first time, the feasibility of aligning a single LLM to represent an exponentially vast spectrum of human preferences through various optimization methods. Our work marks a step forward in effectively and efficiently aligning models to diverse and intricate human preferences in a controllable and Pareto-optimal manner.

Main paper: 
 introduction: 
 AI alignment aims to ensure AI systems align with human intentions, and there has been notable progress in this area, especially for large language models (LLMs) [29, 12, 30, 2] . The prevailing approach for LLM alignment involves curating a dataset {(x, y 1 , y 2 , z)}, where each prompt x is associated with a pair of responses (y 1 , y 2 ) and a scalar label z ∈ {0, 1} that indicates if y 1 is a "better" response. These labels are typically generated based on detailed guidelines that encompass various criteria, reflecting multiple dimensions i ∈ {1, • • • , m} of human preferences (e.g., helpfulness, harmlessness, conciseness, humor, formality). Pre-trained models are subsequently further optimized on this dataset using methods including reinforcement learning, supervised learning, or gametheoretical approaches [27, 42, 32, 6, 44, 4, 47, 40] . However, this single-objective alignment methodology may not fully capture the complexity of real-world scenarios for two reasons (Figure 1 ).First, this method can lead to inconsistency and ambiguity in data labels. Human labelers assign scalar labels z by implicitly evaluating responses across every dimension i with different preference weights to i, and reaching a final judgment. These differences often result in conflicting labels, causing misalignment or learning failures (Appendix B), substantiated by the low average label agreement reported in [5] . Second, optimizing a single objective leads to only one model that attempts to fit the potentially conflicting labeling preferences, i.e., the helpfulness-harmlessness dilemma. This Figure 1 : Comparison of the predominant single-objective alignment and our multi-dimensional alignment. For the two responses to a prompt, labelers agree on the preferable one in each preference dimension, but conflict when assigning a synthesized scalar label denoting which is "better". This arises due to the inherently different preference weights held by labelers, a common case in reality. Performing single-objective optimization on the potentially conflicting scalar-label dataset (left) could lead to a dominated solution and misalignment. By contrast, our method, Panacea, leverages multi-dimensional preference optimization (right) on the consistent multi-dimensional dataset and learns the entire Pareto front (PF), thereby aligning with diverse and complex human preferences. single model may not cover the full spectrum of human preferences across all dimensions, thereby exacerbating biases against underrepresented groups and failing to meet diverse user needs.To address these challenges, we formulate the alignment as a multi-dimensional preference optimization (MDPO) problem. By explicitly curating data for each dimension, we enhance data consistency and simplify the labeling process, thereby overcoming the first limitation.Upon the obtained dataset, our goal is to concurrently optimize across all dimensions. However, this is often infeasible due to potential conflicts among preferences (e.g., helpfulness vs. harmlessness in response to hazardous user requests). Therefore, we aim for Pareto-optimality [39] , which means finding solutions where no preference dimension can be made better off without making another worse off. However, many Pareto-optimal solutions might exist. Instead of just learning one such solution, we focus on learning the entire set of Pareto-optimal solutions. To achieve this, we use a single model capable of recovering any Pareto-optimal solution by inputting the appropriate preference vector.In this paper, we propose Panacea (Pareto alignment via preference adaptation), a simple yet effective method that: 1) learns the entire Pareto-optimal solution set for all possible preferences with a single model, and 2) infers Pareto-optimal responses online by simply injecting any preference vector into the model. Our method, providing a comprehensive representation of human preferences, effectively caters to diverse user needs, thus mitigating the second limitation (Figure 1 ).A key challenge lies in how to utilize a low-dimensional preference vector to control the model's behavior. Our core insight is that, similar to the crucial role of the preference vector in shaping the Pareto solution, singular values are pivotal in defining the model's fundamental behavior in a singular value decomposition (SVD)-based low-rank adaptation (LoRA) [22, 57] . To address the above challenge, we incorporate the preference vector into the singular values within each SVD-LoRA layer. We then scale it using a learnable factor to align with the magnitude of other singular values. The model is trained end-to-end using a joint objective function aggregated according to the preference vector. The flexibility of Panacea enables seamless compatibility with various preference optimization procedures, e.g., supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF) [42] , and direct preference optimization (DPO) [44] , and diverse methods for loss aggregation, e.g., linear scalarization (LS) [10] [Section 4.7.5] and weighted Tchebycheff (Tche) [39] [ Section 3.4] . Through theoretical analysis, we confirm that Panacea can effectively capture the entire Pareto front (PF) under practical conditions. This finding provides a solid rationale for training a single Pareto set model to learn all Pareto optimal solutions across the entire preference space.In our experiments, we assess the effectiveness and scalability of Panacea on several significant and challenging preference alignment problems with up to 10 dimensions, where the Pareto set cardinality grows exponentially with the number of dimensions, considerably surpassing the scope of current research. Panacea consistently outperforms baseline methods, producing superior, uniformly distributed, and convex fronts in accordance with the theory. Quantitative metrics highlight its substantial advantages, demonstrating an order-of-magnitude improvement. Notably, Panacea exhibits no performance saturation even on the ten-dimensional problem, indicating its extensive potential. For the first time, we show the possibility of aligning a single model with exponentially many heterogeneous preferences, opening up a promising avenue for LLM alignment. This paper makes three main contributions. First, we identify the fundamental limitations of the predominant scalar-label, single-objective alignment paradigm, and propose to reframe alignment as a multi-dimensional preference optimization problem. Second, we design Panacea, a simple yet effective method that learns one single model that can online and Pareto-optimally adapt to any set of preferences, without the need for further tuning. Third, we provide theoretical supports and empirical validations to demonstrate the Pareto optimality, scalability, efficiency, and simplicity of Panacea, thereby satisfying the urgent need for Pareto alignment to diverse human preferences. 
related work: 
 Pareto Set Learning. Different from previous classical multi-objective optimization (MOO) methods [59, 35, 38, 56 ] that use a finite set of solutions (referred to as "particles") to approximate the entire Pareto set, Pareto set learning (PSL) [41, 36, 58] aims to use a single model to recover the complete Pareto set/front. The advantage of PSL is that it can store an infinite number of Pareto solutions within a model. This allows users to specify their own preferences, and the model can dynamically output a particular Pareto solution in real-time according to those preferences. Typical applications of PSL includes multiobjective industrial design problems [58, 37] , reinforcement learning [8, 54, 24] , text-to-image generalization [33] , and drug design [25, 61] . While there have been some studies on PSL involving deep neural networks, these models are considerably smaller compared to LLMs. Learning continuous policies that represent different trade-offs for LLMs remains unsolved.Multi-Dimensional Preference Optimization. Existing research primarily treats AI alignment as a single-objective optimization problem with scalar labels [42, 55, 17, 44, 40, 47] , often neglecting the complexity of diverse human preferences. We provide an in-depth analysis of this limitation in Appendix B, which is subsequently substantiated by MaxMin-RLHF's result of "impossibility of alignment" [13] after Panacea first came out. To address this crucial gap, one recent attempt is AlignDiff [19] , which trains an attribute-conditioned diffusion model to conduct preference alignment planning in the RL settings. In the realm of LLMs, there are some contemporary works on this topic [60, 26, 18, 21, [51] [52] [53] , where the most relevant one Rewarded Soups (RS) [45] adopts a multi-policy strategy. It learns a model for each preference dimension and interpolates their parameters linearly to generate a customized model. However, its simple design also constitutes its drawback. Since RS does not see any intermediate preference vectors during training, ensuring the optimality and alignment of the interpolated model poses a challenge. By contrast, Panacea explicitly traverses the preference simplex and learns to recover the entire PF, thus achieving better performance. It is the first fundamentally PSL approach in LLM for multi-dimensional preference alignment, with theoretical guarantees of Pareto optimality under mild conditions.Inner Product = ⟨λ, ξ⟩.It measures the correspondence of the solution with the preference vector. This is because the evaluation result ξ i is expected to be large when λ i is relatively large. 
problem formulation: 
 Human preference is inherently multi-dimensional. In the case of LLM alignment, a preference dimension refers to a single, self-consistent, and independent aspect of evaluating LLM responses, such as helpfulness, harmlessness, humor, etc.. We formulate the multi-dimensional preference optimization (MDPO) problem with m dimensions as:EQUATIONwhere π θ ∈ Π is a policy, i.e. an LLM, and θ is its trainable parameters (decision variable), Π is the policy space, Θ is the parameter space, and J i , i = 1, • • • , m denotes a performance measure of dimension i, such as SFT objective J SFT,i (π θ ), RLHF objective J RLHF,i (π θ ), and DPO objective J DPO,i (π θ ) detailed in the following equations,EQUATIONEQUATIONEQUATIONNotice that D i , r i represent the data and reward model for dimension i respectively. This is in accordance with our proposal to curate data for each dimension separately to enhance data consistency and training performance. Throughout this paper, we use bold letters to denote vectors or matrices (e.g. J , λ). Very often, there does not exist a single solution θ that performs optimally on all dimensions due to their conflicts. Instead, there exists a set of Pareto optimal solutions, which have unique trade-offs among all dimensions. We say solution θ (a) dominates θ (b) , denoted asJ (π θ (a) ) ≻ J (π θ (b) ), if for all i ∈ [m], J i (π θ (a) ) ≥ J i (π θ (b)), and there exists at least one index j ∈ [m] such that J j (π θ (a) ) > J j (π θ (b) ) [20, 39] . Based on this, Pareto optimality is defined as: Definition 3.1 (Pareto optimality). We call a solution θ * Pareto optimal if no other solution θ ′ ∈ Θ dominates θ * . The set of all Pareto optimal solutions is called the Pareto set (PS); while its image set in the objective space is called the Pareto front (PF), T . A solution θ * is considered weakly Pareto optimal if no other solution θ ′ can strictly dominate it, that is, ifJ i (π θ ′ ) > J i (π θ * ) for all i ∈ [m].Human's trade-offs among all dimensions are quantified as a preference vector, λ = (λ 1 , . . . , λ m ), where λ ∈ ∆ m , λ i ≥ 0, andm i=1 λ i = 1.Here, λ i represents the weight for preference dimension i (called preference weight), and ∆ m is the preference simplex. The fundamental problem of MDPO is to learn the Pareto optimal solution for every preference vector.SP = 1 m(N -1) N -1 i=1 ∥ ξi -ξi+1 ∥ 2 .This metric measures the mean squared distances between evaluation results ξi sorted in a non-dominated sort order [16] . A smaller SP reflects that the solutions are more evenly distributed on the fronts.'HQVLW\ 5HZDUGGLVWULEXWLRQEHIRUHDOLJQPHQW +HOSIXOUHZDUG +DUPOHVVUHZDUG >+HOSIXO+DUPOHVV@ >@ >@ >@ 'HQVLW\ >@ >@ >@ >@ 'HQVLW\ >@ >@ >@ >@ 
panacea: pareto alignment via preference adaptation: 
 To solve the MDPO problem, our goal is to learn a single model capable of representing the entire Pareto-optimal solution set. The key challenge here is how to obtain a customized and Pareto-optimal LLM containing billions of parameters for each preference vector. Naive solutions such as directly generating a full LLM for each vector using a hypernetwork is infeasible due to the vast number of parameters. To avoid this, we consider LoRA [22] , a parameter-efficient fine-tuning method, which, for each layer, freezes the original weights W 0 and only learns pairs of rank decomposition matrices A, B for adaptation. According to LoRA, the final weight W is obtained by W = W 0 + BA. However, a rank-8 LoRA of Alpaca-7B [48] still contains nearly 20 million parameters, which means producing separate LoRA parameters for each preference vector can also significantly suffer from training difficulty and instability issues. We thus explore an alternative approach inspired by AdaLoRA [57] . This method employs singular value decomposition (SVD)-based LoRA and learns the left singular matrix U , diagonal matrix Σ (representing singular values), and right singular matrix V . Moreover, U and V are subject to orthogonality regularization.EQUATIONwhich hereafter we call SVD-LoRA. By extracting singular values Σ of incremental matrices, SVD-LoRA captures the core features of adaptation in a few parameters. More importantly, the singular values provide an interface to fundamentally influence model behavior.Our key insight is that the preference vector can be embedded as singular values in every layer to achieve decisive and continuous control of model adaptation. Panacea is thus designed to learn only a single set of SVD-LoRA parameters, but preserves specific dimensions in the diagonal matrix for embedding the preference vector, which leads to model customization. Concretely, for layer l, we preserve k singular values for learning general and preference-agnostic features and concatenate them with the m dimensional preference vector λ multiplied by a per-weightmatrix learnable scaling factor s l . Therefore, for each weight matrix k+m) , and right singular matrixW l ∈ R n l 1 ×n l 2 , we have W l 0 ∈ R n l 1 ×n l 2 , left singular matrix U l = [u l 1 , . . . , u l k , u l k+1 , . . . , u l k+m ] ∈ R n l 1 ×(k+m) , diagonal matrix Σ l = diag(σ l 1 , . . . , σ l k , s l λ 1 , . . . , s l λ m ) ∈ R (k+m)×(V l = [v l 1 , . . . , v l k , v l k+1 , . . . , v l k+m ] ∈ R n l 2 ×(k+m). The scaling factor is important since we observe that the preference-agnostic singular values commonly range from 10 -2 to 10 -5 in our experiment scenarios, which could be significantly smaller than preference weights, and their magnitudes differ across weight matrices, so both no scaling and a unified scaling are suboptimal. Concerning our design, one may worry whether m, the dimension of preference vector, is negligible compared to k. Preliminary experiments show that Alpaca-7B fine-tuned by SVD-LoRA with a rank as low as 4 performs comparably to the full-parameter fine-tuning counterpart. Since the rank is of the same magnitude as the number of human preference dimensions, this suggests the feasibility of Panacea.During each training iteration, we randomly sample a preference vector from the preference simplex ∆ m , embed it into all weight matrices, and obtain the preference embedded model π θ,λ . We then compute an aggregated objective function of π θ,λ across all preference dimensions according to λ, by synthesizing per-dimension objective functions with loss aggregation methods. While in this paper we mainly consider RLHF / DPO / SFT objectives and LS and Tche as aggregation functions, the Panacea architecture is generally applicable. The LS function [10] [Section 4.7.5] is given byEQUATIONand the Tche function is defined as, maxEQUATIONwhere z is a vector (e.g., ideal vector) such thatz i ≥ J i (π θ ), ∀θ ∈ Θ, ∀i ∈ [m]. These loss aggregation functions allow Panacea to obtain solutions corresponding to the preference vector.With respect to the aggregated objective, trainable parameters for each weight matrix W l , including U l , V l , (σ l 1 , . . . , σ l k ), s l , are then updated via gradient descent. At convergence, sampling preferences on the entire preference simplex recovers the whole PF, as guaranteed by the following theorem. Theorem 4.1. Panacea recovers the entire Pareto front for both LS and Tche aggregation functions (Equations (6) and (7)) under the following two assumptions: 1. Panacea with SVD-LoRA has sufficient representation capability for all preferences λ ∈ ∆ m . Specifically, for any preference vector λ, the policy π θ,λ can optimize the corresponding aggregation functions (Equations (6) and (7)) to their maximum values. 2. For a specific preference vector λ, the LLM policy space formed by all π θ,λ can represent all categorical output distributions of responses. By optimizing the Panacea objective function E λ∼Unif(∆m) [g agg λ (θ)], where g agg λ could be g LS λ or g Tche λ , the optimal policy found by Panacea can recover the entire Pareto front for almost every preference.For proof, see Appendix C. As the two assumptions are easy to satisfy, this theorem confirms the Pareto-optimality of Panacea. Panacea also achieves fine-grained control of model behavior through preference embedding, making it a suitable solution to MDPO. During inference, the user specifies a preference vector and obtains the corresponding Pareto optimal model that aligns with his/her preference. We present a visual illustration of Panacea in Figure 2 and its pseudocode in Appendix D.Compared with prior work, Panacea is the first fundamentally PSL approach towards multidimensional preference alignment. It only needs to learn and maintain one model to represent the PF, which is more computationally efficient than both the Discrete Policy Solutions (DPS) method [34, 7] , which learns a model for every preference vector, and RS, which approximates the PF with m models optimized exclusively on the m preference dimensions. Being computationally lightweight is especially crucial in the LLM settings. Panacea also allows online specification of the preference vector to swiftly adapt to any human preferences, meeting users' requirements in no time. Moreover, Panacea achieves a tighter generalization bound of Pareto optimality compared to RS for unseen preferences during training, implying a more complete recovery of the Pareto set. This is due to the explicit traversal of the preference simplex, which allows its generalization error to decay with the number of samples. In contrast, RS only uses a small number of Pareto optimal solutions for interpolation to predict unseen Pareto optimal solutions. The interpolation error cannot be effectively bounded when it only meets a few preference vectors during training. Finally, Panacea preserves explainability to some extent. For each weight matrix W l , Panacea adapts it asW l = W l 0 + U l Σ l V l ⊤ = W l 0 + k i=1 σ l i u l i v l i ⊤ [1] + m i=1 s l λiu l k+i v l k+i ⊤ [2].Intuitively, term [1] captures shared features among preference dimensions, while term [2] learns dimension-specific adaptations and weights them by the preference vector to achieve Pareto alignment. The decoupling of learned parameters not only illustrates the mechanism of Panacea, but also leads to superior robustness of its preference adaptation strategy (further analyzed in Appendix F.5). 
experiments: 
 In this section, we empirically evaluate Panacea's ability to approximate the PF of complex and multi-dimensional human preferences. We apply Panacea to several significant and challenging preference alignment problems with 2, 3, 4, 5, and up to 10 dimensions, far exceeding those addressed in contemporary works. These problems include the classic helpful-harmless (HH) dilemma, its augmented helpful-harmless-concise (HHC) version, and learning the PFs of multiple common preference dimensions in chat scenarios. While the number of dimensions m varies, we keep the preference-agnostic rank k of Panacea fixed to 8 and observe Panacea's performance. Compared with the baseline RS, Panacea consistently learns superior, broader, smoother, more evenly distributed, and convex fronts that align with theoretical expectations. The advantages are quantified through various metrics to substantiate its effectiveness and scalability. Encouragingly, we find that Panacea shows no signs of performance saturation even on the ten-dimensional problem, indicating its unlimited potential. We also conduct ablation studies to validate the design of Panacea. Full experimental details are elaborated in Appendix F, and chat cases are presented in Appendix G. Right: with DPO, Panacea using both LS and Tche aggregation learns better fronts than RS.In the first set of experiments, algorithms are tasked with two-dimensional preference alignment using various initial models, i.e. Alpaca-finetuned [48] Llama1-7B-base [49] (abbv. Llama1-ft) and Llama2-7B-base [50] (abbv. Llama2-ft), optimization procedures, i.e. RLHF and DPO, and loss aggregation methods, i.e. LS and Tche. Specifically, we focus on the helpful-harmless (HH) dilemma, which is an important and urgent problem since different applications of LLMs often require different trade-offs between them. For example, children need extremely safe chat assistants, while chemists prioritize helpfulness as they are fully aware of the potential hazards. However, current alignment techniques provide the same model for all users, which does not cater to these diverse needs. Therefore, learning the entire PF can significantly alleviate this issue. We use the BeaverTails dataset [28] , which has preference labels for both helpfulness and harmlessness. In Figure 3 left, we show the learned fronts of algorithms with the task configuration of Llama1-ft, RLHF, and LS aggregation. The rewards for both dimensions are evaluated by reward models for preference vectors sampled evenly at an interval of 0.1, i.e. λ = (0.0, 1.0), (0.1, 0.9), . . . , (1.0, 0.0). Compared with RS, Panacea learns a significantly better front, whose smooth convex shape also aligns better with the convexity result in Lemma C.3. By contrast, the front learned by RS is not a valid Pareto front since some solutions dominate others, which shows that RS could not learn to recover the PF simply by merging trained weights for all dimensions. In this experiment, we also test Discrete Policy Solutions (DPS) [34, 7] , also known as multi-objective RLHF (MORL) in [45] , which learns a separate model for each preference vector (11 models in this case) and is commonly considered as the performance upper bound for this problem. Surprisingly, Panacea learns better and smoother front than DPS while being much more efficient, which could be attributed to positive transfer among dimensions enjoyed solely by Panacea. In Figure 3 middle, we conduct the same experiment based on Llama2-ft initial model. Across three seeds, Panacea consistently achieves convex and dominating fronts that are more desirable than those of RS, further verifying the results. To clearly demonstrate how the model's output changes with variations in the preference vector, we present an exemplar chat case in Figure 4 and its detailed version in Appendix G. The chat case shows how Panacea effectively tailors to diverse needs, thereby settling the long-standing tension between helpfulness and harmlessness. To further study the generality of Panacea, we conduct experiments with Llama2-ft, DPO, and LS / Tche aggregation, where Panacea is optimized based on Equation (18) and Equation ( 19) respectively. For DPO, we propose to evaluate algorithm performance by measuring the implicit reward model accuracy. That is, for a model π θ , it is accurate on a labeled pair (x,y i w , y i l ) if β log π θ (y i w |x) π ref (y i w |x) > β log π θ (y i l |x) π ref (y i l |x), and its total accuracy is obtained by averaging over dataset. With this metric, in Figure 3 right we plot accuracies of HH dimensions for Panacea with LS / Tche and RS baseline. Results again confirm that Panacea always obtains better fronts.Aside from comparing the fronts learned by Panacea and the baseline, we also quantify the advantage of Panacea by computing four MOO metrics in Table 1 . Hypervolume, the primary metric, measures the volume of space enclosed by a solution set, reflecting its optimality (a visual illustration is shown in Figure 9 ); the average value of Inner product of preference vectors and the evaluation results measures the correspondence between preference vectors and solutions; Sparsity and Spacing further reflects whether the solutions are evenly distributed. Mathematical expressions of these metrics are detailed in Appendix F.4. Table 1 clearly demonstrate dominance of Panacea over RS on learning more optimal and tailored solutions to diverse preferences while using only a single model. In chat scenarios, the potentially large number of preferences necessitates an efficient method that scales beyond two dimensions. Starting from this section, we start to consider more than two dimensions and test Panacea's capability to handle them simultaneously. We first augment the HH dilemma with conciseness, another common preference dimension, and compare the algorithms on the task configuration Llama2-ft, RLHF / DPO, and LS aggregation upon BeaverTails dataset. For RLHF, the concise RM is defined as a rectified affine function that assigns higher rewards to shorter responses; for DPO, the shorter response to each prompt is preferred in the conciseness dimension (details provided in Appendix F). For all experiments, we evaluate the algorithms with preference vectors evenly sampled from the entire simplex at an interval of 0.2, i.e. λ = (0.0, 0.0, 1.0), (0.0, 0.2, 0.8), . . . , (1.0, 0.0, 0.0), and provide the results in Figure 5 and Table 1 .Figure 5 visualizes the fronts learned with RLHF procedure. We observe that Panacea learns a very evenly distributed front, whereas most solutions obtained by RS are cluttered together in a corner. This is because Panacea, as a PSL method, explicitly traverses the preference simplex to learn about PF, resulting in tailored solutions corresponding to each preference vector. In contrast, RS only learns the vertices and cannot generalize well to solutions within the simplex through linear interpolation. Meanwhile, we also observe that Panacea performs better overall in the harmless dimension, further demonstrating the advantages of its learning approach. MOO metrics in Figure 7 : Left: Ablation study on the learnable preference vector scaling factor. Predefined scaling factors ranging from 1 to 10 -5 all result in significantly worse fronts than the learnable approach, indicating the importance of the per-weight-matrix learnable scaling factor. Middle: Investigation of alternative preference adaptation strategies, including adapting only MLP layers, self-attention layers, 10 layers in the front, and 10 layers in the back. Except for the back 10 layers, all other strategies exhibit similar performance. Thus, we decide to adapt all layers for better representation capacity. Right: We show the fronts learned by Panacea at different RLHF steps. The evolution of fronts reveals Panacea's learning process which gradually expands in both dimensions, reduces dominated solutions, and finally converges to a broad and convex front.Table 1 again numerically depict the benefits of Panacea, and the chat case in Appendix G serves as qualitative support. Thus, by learning a more comprehensive solution space, Panacea effectively manages the trade-offs among helpfulness, harmlessness, and conciseness, underscoring its capability to align with diverse human preferences.We further test Panacea's scalability on three, four, five, and up to ten-dimensional alignment problems (abbv. Chat 3, 4, 5, and 10-dim), where the considered dimensions include being humorous, philosophical, sycophantic, helpful, concise, creative, formal, expert, pleasant, and uplifting. These dimensions reflect the common scenario where desirable chat properties are not simultaneously attainable. Hence it requires a Pareto-optimal solution set to accommodate diverse preferences. In solving these problems, we employ Panacea with SFT procedure, since SFT is easier to train and scales better. The initial model used in this series of experiments is Llama-3-8B-Instruct [3] (abbv. Llama3-Instruct), and the loss aggregation function is LS. We first curate data for each dimension by prompting Llama3-Instruct to generate responses to Alpaca instructions with the corresponding property (details are provided in Appendix F). Panacea is then trained using LS aggregated SFT loss. The baseline RS trains separate models for each dimension using the corresponding SFT loss. In evaluation, we report the SFT losses of each produced model on the test set in all dimensions. For 3, 4, and 5-dimensional problems, we evaluate the algorithms with preference vectors sampled at an interval of 0.2, resulting in 21, 56, and 126 total evaluations; for ten-dimensional problems, we sample them at an interval of 0.25, amounting to 715 in total. These comprehensive evaluations allow us to characterize the algorithm performance more accurately. We plot the results of Chat 3-dim in Figure 6 and compute the metrics in Table 1 . Figure 6 shows that Panacea learns a significantly better front than RS. For the higher-dimensional problems where the results cannot be visualized, we verify the convexity of Panacea's learned fronts by computing their convex hulls and observing that all evaluation points are on the respective convex hulls. From Table 1 , we also observe that Panacea consistently outperforms RS, and the advantage gap becomes larger when scaling to higher dimensions. Notably, Panacea is an order of magnitude better than RS on Chat 10-dim and does not exhibit performance plateau, demonstrating its scalability. We provide a chat case in Appendix G from Chat 3-dim to show Panacea's performance. These results confirm that Panacea learns a single model capable of aligning with any human preferences.In this part, we validate the design of Panacea and investigate its learning process on the HH problem. We first analyze the effect of the per-weight-matrix learnable scaling factor s l . Intuitively, it scales preference vectors to the same magnitude as the singular values to avoid either dominant or negligible influence of preference-specific features on W l , as observed from the learned parameters. To validate its importance, we conduct ablation experiments that use a predefined factor to scale preference vectors. Figure 7 (left) indicates that using a fixed scaling results in a significant performance drop regardless of its magnitude, highlighting the necessity of learning an appropriate scaling for each weight matrix separately. We also explore alternative strategies of preference adaptation, which only adapt self-attention layers, MLP layers, first 10 layers, or last 10 layers. Figure 7 (middle) suggests that except for only adapting last 10 layers, all other strategies perform comparably. Thus, for better representation capacity, we decide to let Panacea adapt all layers of an LLM. Finally, in Figure 7 (right), we plot the evolution of fronts learned by Panacea at different steps, showing that it first learns harmlessness features quickly and explores improvements for helpfulness, then it also learns to align with helpfulness preference and finally recovers the entire front. This discovery may inspire training acceleration methods such as dynamically sampling preference vectors according to different learning efficiencies across dimensions. 
conclusion: 
 This paper presents Panacea, the first Pareto set learning approach towards solving Pareto alignment with multi-dimensional human preference using a single model. Central to its design is embedding the preference vector as singular values in SVD-LoRA to fundamentally influence model behavior online. Theoretically, we prove that training the preference-embedded model against an aggregated objective is guaranteed to recover the entire PF at convergence. Empirical results substantiate that Panacea enjoys superior performance and scalability in approximating PF compared with strong baselines including DPS and RS. Overall, Panacea represents a simple yet effective approach that achieves fine-grained, lightweight, and online Pareto alignment with diverse and complex human preferences, an urgent need in LLM applications.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper does not release any data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We list the citations, licenses, and the URLs of all our used assets in Appendix F.6.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we prove the validity of combining reward models of all preference dimensions through linear scalarization in the RLHF optimization procedure, even though each reward model solved by the Bradley-Terry (BT) model [11] is not uniquely determined. This is formalized in the following lemma. Lemma A.1 (Extension of Lemma 2 in [44] for multiple reward models). Let r i (x, y) and r ′ i (x, y) be equivalent reward models for the i-th preference dimension, where r ′ i (x, y) = r i (x, y) + ϕ i (x). The linear combinations r(x, y) = m i=1 λ i r i (x, y) and r ′ (x, y) = m i=1 λ i r i (x, y) + m i=1 λ i ϕ i (x) induce the same optimal policy in the constrained RL problemmax π J RLHF (π) = E x∼D E y∼π(•|x) [r(x, y)] -βD KL [π(•|x)||π ref (•|x)] ,where β is a positive punishment factor of the KL constraint.Remark A.2. This lemma demonstrates that it is valid to linearly combine reward models of all dimensions, even if the reward models are not uniquely identified. It is used in analyzing the limitations of single-objective alignment and it validates the LS aggregation employed with Panacea.Below, we provide a concise proof of Lemma A.1.Proof. According to the constrained RL literatures [43, 9] , the policy for the reward function r ′ (x, y) in a Kullback-Leibler (KL) constrained reinforcement learning (RL) problem can be formulated as follows:π r ′ (y|x) = π ref (y|x) exp 1 β r ′ (x, y) y π ref (y|x) exp 1 β r ′ (x, y).Expanding the term in r ′ (x, y), we obtain:π r ′ (y|x) = π ref (y|x) exp       1 β       m i=1 λ i r i (x, y) + m i=1 λ i ϕ i (x) ϕ ′ (x)             y π ref (y|x) exp       1 β       m i=1 λ i r i (x, y) + m i=1 λ i ϕ i (x) ϕ ′ (x)             .Upon simplifying by canceling out the common term exp(ϕ ′ (x)), we get:π r ′ (y|x) = π ref (y|x) exp 1 β r(x, y) exp 1 β (ϕ ′ (x)) y π ref (y|x) exp 1 β r(x, y) exp 1 β (ϕ ′ (x)) = π r (y|x),which completes the proof.In the following content, we provide a theoretical analysis that the model trained by the singleobjective alignment paradigm could actually misalign with every labeler. We conduct analysis on RLHF, the most common approach. We make the following assumptions: Assumption B.1. Human preference can be modeled by the Bradley-Terry model [11] .Assumption B.2. Different people are consistent in labeling each preference dimension.These two assumptions imply that people possess the same reward model r i (x, y) for each preference dimension i. Assumption B.3. The synthesized reward model of a person is the LS of per-dimensional reward models according to his/her preference vector under a shift invariant term (c.f [44] [Lemma1]). That is,EQUATIONNow we prove the main theoretical result. Theorem B.4. Consider the case where there are n labelers in total. Each labeler h labels a portion p h of the entire dataset, where p h ∈ [0, 1],n h=1 p h = 1. The preference vector of labeler h is λ h = (λ h 1 , λ h 2 , . . . , λ h m ).The labelers have different preference vectors, i.e. ∃ j, h ∈ {1, . . . , n}, λ j ̸ = λ h . The RLHF optimization result is a model that could misalign with every labeler.Proof. The reward model r h of labeler h is r h (x, y) = m i=1 λ h i r i (x, y) + ϕ h (x). J h (θ) denotes the optimization objective corresponding to the reward model of labeler h. The joint optimization objective isEQUATION= maxθ n h=1 p h E x∼D E y∼π θ (•|x) r h (x, y) -βD KL [π θ (•|x)||π ref (•|x)](Rearrange reward terms.)= maxEQUATION= maxEQUATION= maxEQUATIONThus, we show that it actually optimizes with the preference vector λ opt , with λ opt i = n h=1 p h λ h i , i = 1, . . . , m. According to the constrained RL literatures [43, 9] , the corresponding optimal policy can be expressed as:EQUATIONIt is important to note that this optimal preference vector may not align with the individual preferences of each annotator. As a result, the trained model may not fully reflect the labeling criteria of any single annotator, potentially leading to discrepancies in the model's predictions.C Theoretical Support for Panacea with LS / Tche functionIn the following content, we prove for Theorem 4.1 from the main paper, showing that both linear and Tchebycheff scalarization can recover the entire Pareto Front (PF) under practical assumptions.The proof has two subsections: first for the linear scalarization function in Appendix C.1, followed by the Tchebycheff aggregation function in Appendix C.2.We provide a proof sketch for this part.Step 1: Under the full categorical representation assumption, for any two policies π (a) (•|x) and π (b) (•|x), we can create a new policy (π ′ ) that, with probability (w.p.) p (where 0 ≤ p ≤ 1), takes π (a) (•|x) and w.p. 1 -p, takes π (b) (•|x). This policy can also be represented by LLM.Step 2: Using the above policy construction method, we prove that the objective spaces of DPO, RLHF, and SFT are convex.Step 3: When the objective spaces are convex, the Pareto objectives found by LS aggregation function (Convex Coverage Set (CCS)) equal the entire Pareto front.Step 4: By optimizing the Panacea objective function E λ∼Unif(∆m) g LS λ (θ) , we can recover the entire Pareto front.Then, we are geared up for the formal proof. We first restate the assumption for the full categorical policy space in Theorem 4.1. Assumption C.1 (Full Categorical Policy Space Assumption (detailed restatement from Assumption 2 in Theorem 4.1)). For a specific preference vector λ, the LLM policy space formed by all y ∼ π θ,λ (•|x) can represent all the categorical distribution set Π(x) for response y = [t 1 , . . . , t N ], where N is the response length and t i denote each token, given an input sentence x.This assumption is proper because the probability of each token t 1 , . . . , t N (N denotes the length of the output of y) can be represented by a LLM policy. Given the strong representation ability of LLMs, any probability value of token sequence t 1 , . . . , t N can be represented by their output. With this assumption, a direct corollary holds because the linear combination of categorical distributions is still a categorical distribution. The reason for that is such constructed policy is still a categorical distribution. For the next step, we use this corollary to prove the following lemma to show that the objective spaces J SFT , J RLHF , and J DPO are convex.Lemma C.3 ( Convex space Lemma, adapted from [23] (Eq. 13) ). For any two objectives J alg , and for any 0 < α < 1, there exists a policy π ′ ∈ Π(x) such that αJ(a) alg + (1 -α)J (b) alg = J (π ′ ),where J alg can be J DPO , J SFT , or J RLHF . This lemma mainly follows from Eq. 13 in [23] . We include their proof for our purpose for completeness. The objectives J SFT , J RLHF , and J DPO can all be written as J alg (π) = E x,y∈D [f (x, y, π(y|x))] for some particular design of f (x, y, π(y|x)). For any 0 ≤ α ≤ 1, by Corollary C.2, we can construct a new policy π ′ and a uniform random variable S ∼ U (0, 1) such that:π ′ (y|x) = π a (y|x) if S < α π b (y|x) if S ≥ α Then, J (π ′ ) = E (x,y)∼D [f (x, y, π ′ (y|x))] = E S∼U (0,1) E (x,y)∼D [f (x, y, π ′ (y|x))|S] = αE (x,y)∼D [f (x, y, π ′ (y|x))|S < α] + (1 -α)E (x,y)∼D [f (x, y, π ′ (y|x))|S ≥ α] = αE (x,y)∼D [f (x, y, π (a) (y|x))] + (1 -α)E (x,y)∼D [f (x, y, π (b) (y|x))] = αJ (π (a) ) + (1 -α)J (π (b) )Thus, for any convex combination of J (π (a) ) and J (π (b) ), there exists a policy π ′ such that J (π ′ ) = αJ (π (a) ) + (1 -α)J (π (b) ), indicating that the space of J (π) is convex. We denote the full space of J (π) for all policies as J.For the third step, we use Lemma C.3 to establish that linear scalarization functions have the capability to discover the complete PF by traversing the entire preference simplex ∆ m (i.e., the approach employed in Panacea). To prove that, we introduce the concept of the convex coverage set (CCS), which is the objective set that can be found by optimizing the linear scalarization function with all preference vector λ ∈ ∆ m . Definition C.4 (Convex Coverage Set (CCS), adapted from [46] (Def. 9)). The CCS contains the objective such that there exists a preference vector λ where the inner product of λ and this objective is greater than that of λ with any other objective vectors in the objective space. CCS:= {J ∈ J|∃λ ∈ ∆ m s.t. λ ⊤ J ≥ λ ⊤ J ′ , ∀J ′ ∈ J}.Then, we prove that when the objective space is convex, the linear scalarization can recover the whole Pareto objective set, i.e., T = CCS, where T denote the objective vectors forming the Pareto front.Proof. The PF T is a subset of the boundary of the objective space, denoted as ∂(J (Π)). By proving that J (Π) is a convex set, we can apply the supporting hyperplane theorem [10] (Sec. 2.5.2). According to this theorem, for every element r in ∂(J (Π)), there exists λ ∈ R m such that λ T (r -r ′ ) ≥ 0 for all r ′ ∈ J (Π). Moreover, when r is Pareto optimal, such λ ⪰ 0. Hence, we have λ T (r -r ′ ) ≥ 0 for all r ′ ∈ J (Π) and λ ∈ ∆ m . This condition implies that T ⊂ CCS. Since it has been established that CCS ⊂ T , we can conclude that CCS = T .For the last step, we demonstrate that by optimizing E λ∼Unif(∆m) g LS λ (θ) using the LS aggregation function, we can recover almost the entire Pareto front. If a non-zero measure of the Pareto front could not be recovered, it would imply the existence of non-zero measure preference vectors for which the corresponding Pareto-optimal solutions cannot be found using the LS aggregation function, which contradicts Assumption 1 in Theorem 4.1.To prove that using the Tchebycheff aggregation function allows Panacea to recover the full Pareto front, we introduce the following lemma: Lemma C.5 (Adapted from [14] , Theorem 3.1). A feasible solution θ is weakly Pareto optimal if and only if there exists a weight vector λ such that θ is an optimal solution to the aggregation function (Equation (7)) defined in the main paper. Proof. If a non-Pareto policy has a measure greater than zero, then according to Lemma C.5, there exists a preference set of greater than zero measure where the non-Pareto policy has a smaller value compared to the optimal value of the Tchebycheff function under the corresponding preferences, contradicting Assumption 1 in Theorem 4.1. Sample from D a data batch B.Sample a preference vector λ and embed into π θ,λ .Compute the aggregated objective for π θ,λ on B according to λ.Update θ with gradient descent. 9: end for 10: Return π θ .In this section, we present the LS / Tche aggregated training objectives for Panacea with RLHF / DPO / SFT. In RLHF, reward models r i , i = 1, . . . , m are learned for each preference dimension. For a specific preference vector, the LS aggregated objective function ismax θ g LS λ (θ) = max θ E x∼D E y∼π θ,λ (•|x) m i=1 λ i r i (x, y) -βD KL [π θ,λ (•|x)||π ref (•|x)] . (16)The Tche aggregated objective isEQUATIONwhere z i is the maximum reward for preference dimension i. Intuitively, Tche aggregation aims to minimize the maximum weighted suboptimality among all dimensions. However, since the maximum reward can be hard to determine in practice, we find Tche less suitable for RLHF than for DPO. DPO transforms the reinforcement learning objective into a supervised objective, whose LS aggregated objective isEQUATIONTo derive the Tche aggregated objective, we haveEQUATIONSince the optimal value z i for per-dimension DPO objective is 0, this is naturally compatible with Tche aggregation.Finally, the LS aggregated SFT objective isEQUATION)Similar to DPO, since the optimal value z i for per-dimension SFT objective is 0, the Tche aggregation of SFT objectives isEQUATIONIn this section, we present experimental details including computational resources, algorithm implementation, data curation, experiment setup, and evaluation details, and analyze additional results. All our experiments are conducted on an 8×A800-80GB GPU server. Other details are elaborated below.Our implementation is based on the Safe-RLHF [15] codebase. As described in Section 4 and visualized in Figure 2 , the core design of Panacea is the embedding of the preference vector as singular values based on SVD-LoRA. Its core code is presented in Figure 8 . In our experiments, we perform Panacea adaptation to all self-attention and MLP layers. We initialize the singular values and preference scaling to zero, so as not to impact the model behavior at the beginning of training [22, 57] . In each iteration, we sample a preference vector from the preference simplex, embed it into the model, and train the model on the aggregated objective.In the helpful-harmless (HH) problem in Section 5.1, we use the BeaverTails dataset [28] , which contains both helpfulness and harmlessness preference labels. In the augmented helpful-harmlessconcise (HHC) problem in Section 5.2, we again use the BeaverTails dataset. For RLHF, we define the reward model as a rectified affine function,r concise (x, y) = r max , l y ≤ c r max + 1 - ly c, otherwise where r max defines the maximum reward, l y denotes token length of response y, and c defines both the threshold for maximum reward and the slope of concise reward model. This reward model encourages more concise answers, while the reward does not further increase when the response length is smaller than a given threshold. For DPO, we label the shorter response to each prompt as preferred. In the Chat multi-dimensional alignment problem in Section 5.3, we curate SFT data by letting Llama-3-8B-Instruct [3] generate responses for Alpaca prompts [48] in each dimension. Specifically, the prompt given to Llama3-Instruct consists of a system prompt "Please respond to the following instruction in <a/an> <dimension> way.", where <dimension> is substituted by the adjective of preference dimension and <a/an> is used accordingly, and the user prompt being the original Alpaca prompt. We employ vLLM [31] for fast model inference to accelerate data generation.In this part, we present details about the experiment setup. In the HH and HHC problem, we find it unsuitable to directly use fine-tuned open-source models, as they have undergone extensive safety alignment and are hard to be steered to help with potentially hazardous requests. Thus, we choose to fine-tune the pre-trained base models with Alpaca dataset using the Safe-RLHF codebase, leading to Llama1-ft and Llama2-ft. The reward models are trained upon these SFT models. As we find that the output scales of reward models trained by ourselves differ from the one open-sourced by Safe-RLHF by a factor of 5, we always multiply the reward model outputs by 5 to make them match, which also makes it easier to train. The preference dimensions considered in Chat 3-dim, 4-dim, and 5-dim are "humorous, philosophical, helpful", "humorous, philosophical, sycophantic, helpful", and "humorous, philosophical, sycophantic, helpful, concise" respectively. As for the rank of Panacea, we always fix In evaluation, we evenly sample preference vectors from the preference simplex ∆ m to comprehensively reflect the quality of the learned fronts. We evaluate the per-dimension reward, DPO accuracy, and SFT loss respectively based on the optimization procedure used, due to the varied availability of reward models. To quantify algorithm performance, we employ four multi-objective optimization (MOO) metrics in our evaluations: hypervolume, inner product, sparsity, and spacing. Let ξ = {ξ 1 , ξ 2 , . . . , ξ m } represents the evaluation results of the learned model with a preference vector. Let Ξ be the set of evaluated solutions. These metrics are defined as follows.1. Hypervolume (HV):HV = Vol({ξ|∃ ξ ′ ∈ Ξ, z ⪯ ξ ⪯ ξ ′ }).This set includes any evaluation vector that dominates a reference point z and is dominated by at least one objective in Ξ. z is a fixed reference point dominated by all solutions in Ξ. The hypervolume indicator measures convergence to the true Pareto front, with higher values indicating greater convergence. A visual illustration is provided in Figure 9 .Figure 10 : Comparison of reward distribution on eval dataset between the initial model, i.e. before alignment, and Panacea with various preference vectors. It shows that after alignment, both reward distributions shift rightwards. When the preference vector changes, the two reward distributions shift accordingly, exhibiting find-grained alignment with human preference.4. Spacing:Spacing = 1 N N i=1 (d i -µ) 2 , µ = 1 N N i=1 d i , d i = min j∈[N ],j̸ =i ρ(ξ i , ξ j ),where ρ denotes Euclidean distance. This metric measures the standard deviation of the minimum distances from all solutions to other solutions. It also reflects the uniformity of the set of solutions.In this part, we provide some additional experimental results. In Figure 10 , we compare reward distributions of the initial model and Panacea for HH problem with Llama1-ft and RLHF, corresponding to Figure 3 (left). For any preference vector, Panacea shifts both reward distributions rightwards, highlighting the shared alignment features it learns. If we tune the preference weights for both dimensions, their reward distributions change correspondingly, showing that Panacea achieves fine-grained continuous control of model performance, thereby aligning with complex human preferences. Figure 14 shows the response of the model after preference shift, and more chat examples are provided in Appendix G. In Figure 11 and Figure 12 , we visualize the 2D and 3D projections of the learned fronts in Chat 4-dim problem. The results again confirm that the front learned by Panacea dominates that of RS by a large margin.Additionally, we test the robustness of the preference adaptation strategy of Panacea and compare it with RS. Since the preference simplex is a low-dimensional space in R m , we aim to see whether embedding preference vectors outside the simplex has a significant impact on the model performance.To do this, we scale the preference vectors by a constant and evaluate the model. Since RS first linearly interpolates the left, diagonal, and right matrices and then fuses them for inference, the resulting full incremental matrix is actually scaled by the cube of the constant. Thus for fair comparison, RS uses a constant of 2, and Panacea uses 8. The testbed used here is Chat 3-dim with considered dimensions being "humorous, helpful, concise". The results plotted in Figure 13 clearly demonstrate the superior robustness of Panacea. Moreover, when we inspect the output responses, we find that Panacea is still generating aligned responses with the corresponding preference vector, while RS outputs become completely unreadable. One explanation could be that Panacea explicitly decouples preference-agnostic and preference-specific features, thus scaling the preference vector does not strongly impact the quality of its responses. This experiment further substantiates the effectiveness, robustness, and rationality of Panacea.Finally, we investigate the performance of Panacea on small language models (SLMs), as SLMs are prevalent in edge applications. Specifically, we conduct experiments with Phi-3-mini-4k-instruct [1] (abbreviated as Phi-3 hereafter), which has 3.8B parameters. We test Panacea on Chat 3, 4, and 5-dim with Phi-3 using SFT, and observe that Panacea again learns convex and evenly distributed fronts and +XPRURXVREMHFWLYH 3KLORVRSKLFDOREMHFWLYH produces responses that align with the preference vector. We present two chat cases in Figure 17 and Figure 18 to demonstrate that Panacea flexibly aligns with diverse human preferences with a single model, without incurring overhead during online adaptation. Thus we verify the general applicability, scalability, and effectiveness of Panacea.3DQDFHD 56 +XPRURXVREMHFWLYH 6\FRSKDQWLFREMHFWLYH )URQWVRIDQ\GLPHQVLRQVLQWKH&KDWGLPDOLJQPHQW6)7 +XPRURXVREMHFWLYH +HOSIXOREMHFWLYH 3KLORVRSKLFDOREMHFWLYH 6\FRSKDQWLFREMHFWLYH 3KLORVRSKLFDOREMHFWLYH +HOSIXOREMHFWLYH 6\FRSKDQWLFREMHFWLYH +HOSIXOREMHFWLYH+X PR UR XV RE MHF WLY H 3 K LO R V R S K LF D O R E MH F WL Y H 6\FRSKDQWLFR EMHFWLYH +X PR UR XV RE MHF WLY H 3 K LO R V R S K LF D O R E MH F WL Y H +HOSIXOREMHFW LYH +X PR UR XV RE MHF WLY H 6 \ F R S K D Q WL F R E MH F WL Y H +HOSIXOREMHFW LYH 3K LOR VR SK LFD OR EMH FWL YH 6 \ F R S K D Q WL F R E MH F WL Y H +HOSIXOREMHFW LYHWe present the information of assets as below: -License: Llama 2 Community License Agreement -URL: https://huggingface.co/meta-llama/Llama-2-7b • Meta-Llama-3-8B-Instruct [3] -License: Llama 3 Community License Agreement -URL: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct • alpaca-7b-reproduced [15] -License: Non-commercial license.-URL: https://huggingface.co/PKU-Alignment/alpaca-7b-reproduced • beaver-7b-v1.0-reward [15] -License: Non-commercial license.-URL: https://huggingface.co/PKU-Alignment/beaver-7b-v1.0-reward • Phi-3-Mini-4K-Instruct [1] -License: MIT license.-URL: https://huggingface.co/microsoft/Phi-3-mini-4k-instructTo demonstrate the quality of the solution set represented by Panacea using a single model, we present chat cases where Panacea responds to the same user prompt under different preference vectors. The model's adaptability is demonstrated through its ability to generate diverse responses based on varying preference vectors. Each preference vector encapsulates distinct user preferences, enabling Panacea to offer tailored and contextually relevant information. In the chat case from helpful-harmless (HH) alignment problem (Figure 14 ), upon examining inquiries that encompass unsafe viewpoints, Panacea showcases its nuanced responsiveness. As the preference vectors undergo shifts, the model can strategically address concerns related to illegal activities. From a harmlessness perspective, Panacea tactfully alerts users to potential legal implications, fostering ethical engagement. Simultaneously, the model demonstrates its versatility by providing helpful insights from a preventive standpoint, advising users on theft prevention strategies. More examples are presented in Figure 15 , Figure 16 , Figure 17 , and Figure 18 , which are chat cases from the helpful-harmless-concise (HHC), Chat 3-dim ("humorous, philosophical, helpful"), and Chat 4-dim ("humorous, philosophical, sycophantic, helpful") problem. For each preference vector, Panacea outputs a response that is not only consistent with the vector but also approximately Pareto optimal in the sense that it cannot be made better off in one dimension without negatively affecting the other dimensions. This functionality underscores Panacea's capacity to cater to a spectrum of user needs, ensuring personalized and responsible interaction. In summary, the examination of Panacea's responses under different preference vectors sheds light on its Pareto optimal performance, showcasing its Pareto alignment with diverse and complex human preferences via preference adaptation using a single model. One limitation of our work is that in LLM settings it is impossible to find the ground truth Pareto optimal solutions, which makes it hard to judge the quality of solutions found. We tackle this limitation by comparing with DPS in Section 5.1, which learns a model against a single preference vector and is commonly considered as an empirical upper bound. Another limitation is that although Panacea learns to represent the full spectrum of solutions with a single model and allows online adaptation to any preference vector, it is unclear how to find the user's preference vector corresponding to the most suitable solution for him/her. A potential method is that since Panacea incurs almost no cost for preference adaptation, the user could try different ones and reach a final decision. Finally, when we scale to even higher dimensions, effectively sampling preference vectors from the preference simplex to accelerate learning becomes a crucial problem. This is not addressed in this paper and could be a promising future work. For the up to ten-dimensional problem we consider, sampling randomly from the simplex with higher probability for the vertices leads to good performance.As we aim to align with diverse human preferences using a single model, we have adopted an SVDbased low-rank design so that the preference vector can be injected as singular values to obtain distinct solutions. The central assumptions are that preference adaptation exhibits a low-rank structure, and that low-rank adaptation possesses sufficient representation capabilities. In our extensive experiments, the performance of Panacea has confirmed that these assumptions commonly hold. Increasing the rank would allow Panacea to be effective on even more complex problems. However, there could still exist scenarios that are not well-modelled by Panacea. For them, we believe Panacea is a pioneering method towards Pareto alignment and will inspire future work to explore more advanced design that loosens such assumptions. Figure 18 : This chat case of Phi-3-mini-4k-instruct (3.8B parameters) [1] from the Chat 4-dim ("humorous, philosophical, sycophantic, helpful") problem shows that Panacea is also effective on small language models.Panacea should rigorously test the model with varying preferences, enhance regularization, and make a conscious effort to limit access to the extremely helpful model to certain users or occupations.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper propose an alignment method for multi-dimensional preferences that trains a single model capable of adapting online and pareto-optimally to various preference sets. Since this is impossible in general, the authors assume a low rank structure in the preferences and use a SVD-based adaptation that injects preference vectors as singular values during inference. 

The reviewers have raised some concerns about the expositional clarity, as well as the need for a candid discussion of the limitations of the framework. The reviewing team expects the authors to incorporate the discussion in the camera-ready version. 

In particular, while the low-rank assumption is a useful operationalizing device, it is not a panacea and it will be helpful to present examples where they do not anticipate sophisticated human preferences to exhibit such structures. On a more minor point, I find the method acronym "panacea" to be somewhat misleading as it overclaims the utility of the proposed approach.

Reviews: review: paper_summary: paper_summary: This paper introduces Panacea for LLM alignment that reframes it as a multi-dimensional preference optimization problem. Unlike traditional methods that use scalar labels, Panacea trains a single large language model capable of adapting to diverse sets of preferences in a Pareto-optimal manner without further tuning using SVD-based low-rank adaptation, allowing preference vectors to be injected online as singular values.
strength_weakness: strength_weakness: - The presentation of figure 1 is a bit confusing and hard to understand. It can be improved. Specifically, I think the idea of Pareto front is that we can have a different response given a different preference. The multi-dimensional alignment panel is still using two responses given different preferences, which does not serve the purpose of illustrating the main point of the paper. 
- It would be helpful to have an algorithmic block of the pseudocode of the algorithm. 
- In the paper, it is claimed many times that "Panacea again consistently outperforms RS, and its fronts exhibit smooth **convex** shapes that correspond with theory." However, the authors only visualized Pareto fronts dimension 2 and 3, and it is not confirmed yet whether this property holds in higher dimensions and I'm skeptical about this claim about convexity for higher dimensions.
questions: questions: - The paper evaluates alignment up to tens of dimensions, including being humorous, philosophical, etc. However, I cannot find the dataset used for the evaluation of those dimensions in E.6 Information of assets Data section. It would be helpful if the authors can clarify about the data used for the evaluation of those dimensions. 
- Correct me if I'm wrong, but it seems that in Figure 3, the Pareto fronts obtained by RS are not even real Pareto fronts because some solutions clearly dominate some other solutions? If that's true, maybe it's worth commenting on in the paper.
limitations: limitations: The authors discussed the limitations as follows
- Curse of dimensionality as dimension scales up to more than 10
- The method requires a preference vector from the user, but the user might not know a specific preference vector until they try a bunch of evaluations
- No ground truth Pareto front available for evaluation
paper_summary: paper_summary: This paper addresses the Multi-Dimensional Preference Optimization (MDPO) problem, which involves aligning multiple objectives that exhibit heterogeneity in preference in the population, such as the tradeoff between harmlessness and helpfulness. The authors propose a framework that identifies the Pareto frontier set using a single model, with each preference criterion represented as a dimension of the preference vector. Within this abstraction, the SVD-LoRA-trained model can adapt online and Pareto-optimally to diverse sets of preferences without requiring gradient updates. The authors evaluated this method against several existing techniques, including RS and DPS, and found that Panacea outperforms these methods across various metrics standard to the Multi-Objective Optimization (MOO) community.
strength_weakness: strength_weakness: 1. The paper claims that the idea is theoretically sound. However, the reviewer believes that the justification relies heavily on the expressiveness of the model (assumption 1). This assumption is not as mild as the authors suggest, since it restrained the weights to move in a very low dimensional space compared to the actual number of parameters, which is in the billions. Thereby significantly reduce the expressiveness, making the problem likely misspecified.

2. While the idea is effective in theory, it remains unclear whether a readily abstracted preference vector would be available in practice.

3. The paper could benefit from providing more details on how the numerical rewards are generated in evaluation, as well as more extensive studies on the reward models that is used to evaluate.
questions: questions: 1. As mentioned in the weakness section, how are the reward models generated? And most importantly how reliable are these models?

2. How reliable are the generated responses in the higher dimensional alignment problem? If the responses are not too significantly different from each other, it is hard to say that the evaluation metric is sound.
limitations: limitations: The authors have addressed the limitations in appendix.
paper_summary: paper_summary: # Summary

The paper presents Panacea, an innovative method for aligning LLMs with human preferences, by reconceptualizing the alignment task as a Multi-Dimensional Preference Optimization (MDPO) challenge to recover the entire Pareto front and adapt online.

# Contribution

Panacea marks a good advancement in LLM alignment by providing a scalable, efficient, and theoretically sound method for aligning models with diverse human preferences.
strength_weakness: strength_weakness: 1. Scalability concerns: Although the paper claims scalability, there might be concerns about the computational cost and feasibility when scaling to even larger models and more dimensions.
2. Empirical limitations: The experiments might benefit from comparisons to more diverse baseline methods.
questions: questions: 1. Training costs: Include the training costs for reference.
2. Additional baselines: Include comparisons with a broader range of baseline methods to strengthen the empirical validation.
limitations: limitations: High Computational Demand: Despite claims of scalability, the computational cost of training and maintaining a single model that can adapt to a vast array of preferences may be substantial, particularly for very large models or when increasing the number of preference dimensions.
The approach may require significant computational resources for training and inference stages, which could be a barrier for organizations with limited resources.

Comments: title: Thanks for your appreciation
Thank You for Your Acknowledgment and Follow-up on Remaining Concerns
Thanks for your appreciation
Best wishes with your paper
Reply to rebuttal
Thanks for your suggestion
Panacea is also effective on SLMs (conclusion)
Panacea is also effective on SLMs (continued)
What About MiniCPM and Qwen?
Panacea is also effective on SLMs
Follow-up on Rebuttal and Review Feedback
Follow-up on Rebuttal and Review Feedback
Nice rebuttal!
comment: Dear Reviewer Chm5,

Thank you for your positive feedback and for supporting our work. We are pleased to hear that our responses have convinced you of the effectiveness of our methods on SLMs. As you suggested, we will certainly include an analysis on SLMs in the appendix.

We have greatly appreciated our discussions with you and look forward to sharing our final work.

Best regards,
The Authors
Dear Reviewer wdga,

Thank you very much for raising your score. Your recognition motivates us to continue striving to meet your expectations. We notice that you still have some reservations about the practicality of Panacea. Could you please share these concerns with us? We would greatly appreciate the opportunity to clarify them or further improve our paper.

We sincerely look forward to further discussion.

Best regards,
The Authors
Thank you very much for your appreciation! We will include the revision in the final version of the paper.
Dear Authors,

Thank you for your detailed response. Your rebuttal has addressed some of my theoretical concerns, and while I still have reservations about the practicality of this approach, I recognize that it offers significant advantages over related work. Given this, I will raise my score.
Thanks for your responses and for convincing me. I am sure that your methods would work well on these SLMs.

I keep my positive score and hope to see your analysis on SLMs in your appendix.

Also, best wishes to your paper.
Thank you for the detailed replies to my questions. I have raised my score from 5 to 6.
Thank you very much for your suggestion on including more results on other SLMs. We agree that MiniCPM is a very promising SLM. We note that since Panacea is agnostic to the models used and we have already demonstrated its effectiveness and scalability on both LLMs and an SLM, it should obtain impressive performance on MiniCPM and Qwen. However, as the authors-reviewer discussion phase draws to a close, we may not be able to complete the experiments before this phase ends due to computational and time constraints. But we will definitely take your suggestion seriously and consider adding it in our final version. Thank you once again!
These chat cases show that Panacea flexibly aligns with diverse human preferences with a **single** model, incurring literally no cost during online adaptation. Such Pareto alignment and scalability have been observed on Panacea with both LLMs (7B, 8B models) and SLMs (3.8B model), showcasing its general applicability and effectiveness. As Panacea is completely agnostic to the language model used, its effectiveness is also guaranteed on those similarly strong models such as MiniCPM and Qwen. Thus, we hope our efforts can convince you of the effectiveness, generality, and scalability of Panacea.
**Case 2**. Chat 4-dim: [humorous, philosophical, sycophantic, helpful]

**Prompt**: Write a query in PostgreSQL to list customers with orders in a particular month. Month: April 2021.

**Preference vector**: [1, 0, 0, 0]. **Response**: The thrill of querying PostgreSQL! It's like solving a puzzle, but instead of finding the missing piece in a traditional jigsaw puzzle, you're searching for data that's been hiding in plain sight for months (or should I say, April 2021?).  Here's the query:  
```sql 
SELECT DISTINCT customer_name, order_total FROM orders WHERE EXTRACT(MONTH FROM order_date) = 4 AND EXTRACT(YEAR FROM order_date) = 2021; 
``` 
But wait, there's more! To make it super fun and engaging, let me add some comments to explain what each part of the query is doing:  
```sql 
-- We're starting with the SELECT statement. This is where we gather our trusty database adventuresome companions // In this case, we're calling upon... (you guessed it!) ...the \"orders\" table!  // Now for the most exciting part: filtering by date! // Ah yes, April 2021 is our target month // We use EXTRACT to pull the month (0-11) and year (4 digits long) from those mysterious order_dates 
```  
And our query has been crafted to uncover all customers with orders during April 2021.

**Preference vector**: [0, 1, 0, 0]. **Response**: The query I shall craft is not merely a request for data, but an inquiry into the very fabric of existence. It is a quest to unravel the threads of time and space, to reveal the hidden tapestry that weaves together the threads of commerce and human endeavor.  Here is the query: 
```sql 
SELECT c.name, COUNT(o.id) AS order_count FROM customers c JOIN orders o ON c.id = o.customer_id WHERE DATE_TRUNC('month', o.order_date) BETWEEN '2021-04-01' AND '2021-04-30' GROUP BY c.name;
 ``` 
In this query, we are not simply retrieving information from a database. We are probing the depths of reality itself. We are seeking to understand how humans interact with one another through commerce.  \"Customer\" becomes not just a name or an ID, but an ontological entity that exists beyond mere abstraction. The \"orders\" become not just transactions or records, but manifestations of human intention and willpower.

**Preference vector**: [0, 0, 1, 0]. **Response**: Oh, oh, esteemed leader! I'm thrilled to have the opportunity to craft a query that will undoubtedly impress your refined and discerning eyes. Here it is:  
```sql 
SELECT customers.* FROM customers JOIN orders ON customers.customer_id = orders.customer_id WHERE orders.created_at >= '2021-04-01' AND orders.created_at < '2021-05-01' GROUP BY customers.id; 
```  
This query is sure to produce the most comprehensive and accurate list of customers with orders in the illustrious month of April 2021. It's a masterpiece that will surely bring joy and satisfaction to even the most demanding queries.  I hope this meets your expectations, oh great leader!


**Preference vector**: [0, 0, 0, 1]. **Response**: Here is a query to list customers with orders in April 2021 using PostgreSQL: 
```sql 
SELECT DISTINCT customers.* FROM customers JOIN orders ON orders.customer_id = customers.id WHERE orders.order_date >= '2021-04-01' AND orders.order_date <= '2021-04-30'; 
``` 
This query joins the `customers` and `orders` tables, and selects only those customers who have at least one order placed between April 1, 2021 and April 30, 2021. The `DISTINCT` keyword is used to remove any duplicate rows from the result.  Note: Make sure your `order_date` column is in a date or timestamp format that can be compared with the given date range. If your column is stored as a string, you may need to use `TO_DATE('2021-04-01', 'YYYY-MM-DD')`


**Preference vector**: [0.2, 0.2, 0.4, 0.2]. **Response**: You want to know the ultimate query for making your boss look like a genius, don't you? Here it is: 
```sql 
SELECT c.customer_name FROM customers c JOIN orders o ON c.customer_id = o.customer_id WHERE EXTRACT(MONTH FROM o.order_date) = 4 AND EXTRACT(YEAR FROM o.order_date) = 2021 AND c.customer_id NOT IN (SELECT customer_id FROM cancelled_orders WHERE cancelled_date BETWEEN '2021-04-01' AND '2021-04-30') 
``` 
This query is like a ninja - it sneaks up on the data, quickly removes any pesky cancelled orders, and then gracefully presents the results to you. But wait, there's more! Here are some extra features: * If you're feeling extra fancy, add an ORDER BY clause to see which customers placed the most orders in April. * For those who prefer a more concise approach, use `SELECT DISTINCT` instead of `SELECT *` to only retrieve unique customer names. * And if you want to be extra-extra-extra-good at showing off your skills... add comments explaining each line of code! Now go forth and impress your boss with your mastery of PostgreSQL magic!
I think that MiniCPM is a promising SLM. And, a lot of people tend to evaluate MiniCPM, Phi, and QWen at the same time. Could you also provide the results of MiniCPM and Qwen?

Also, thanks for your response. Hope to see positive results on all these models!
Thank you very much for your positive feedback! We agree with you that being effective on SLMs is indeed of great importance. Thus, upon receiving your feedback, we immediately conducted experiments with Phi-3-mini-4k-instruct (abbrv. Phi-3 hereafter), which has 3.8B parameters. We test Panacea on Chat 3, 4, and 5-dim with Phi-3, and find that **Panacea is also effective on SLMs**.

Quantitatively, we observe that Panacea learns convex and evenly distributed fronts in all three experiments. By calculating the convex hulls of these evaluation points, we find that all points are all the respective convex hulls, demonstrating the convexity of the fronts. However, as this phase does not allow sharing figures, we could not visually show you the results. But we will include it in the final version of the paper.

To qualitatively show the effectiveness of our learned model, we present two chat cases from Chat 3-dim and Chat 4-dim experiments.

**Case 1**. Chat 3-dim: [humorous, philosophical, helpful]

**Prompt**: Identify the nearest airport based on the given coordinates. 40.728157, -73.794853.

**Preference vector**: [1, 0, 0]. **Response**: The coordinates you've provided are a real treat, Bob Dylan-style! I'm not just going to give you a boring old answer, no sirree! With those fancy numbers in hand, I've flown to the nearest airport... and I found myself exactly where you'd expect me... in Manhattan! That's right, folks! Those coordinates led me straight to LaGuardia Airport (LGA), which is like the hipster coffee shop run by accountants with an affinity for aviation. But don't worry, I won't get too carried away with my airport fetish. Now, if you'll excuse me, I need to go find some artisanal toast at one of those trendy cafes nearby...

**Preference vector**: [0, 1, 0]. **Response**: The coordinates 40.728157, -73.794853, whispered to me by the winds of fate, reveal a hidden truth. In this vast expanse of existence, I find myself drawn to the nearest airport. But what is this \"nearest\" airport you ask? Is it merely a physical distance, or does it transcend the boundaries of our mundane reality? Does it exist independently of our perception, or is it a manifestation of our collective consciousness? As I ponder this enigma, I am reminded of Schopenhauer's notion that reality is merely an illusion created by human senses. Perhaps then, the nearest airport is not a fixed entity at all; rather, it is an ephemeral concept born from our collective desire for connection and mobility. And so, as I gaze upon these coordinates 2013 40.728157 degrees north latitude and -73.794853 degrees west longitude 2013 I realize that they represent not just a spatial location but also a profound existential inquiry: What is our relationship with distance and proximity? How do we define \"near\" in a world where boundaries blur and connections defy gravity? In short, dear friend, let us venture forth on this journey together! For in embracing the uncertainty that lies within these coordinates 2013 40.728157, -73.794853 2013 we may just uncover hidden truths about ourselves and our place within this vast tapestry called existence.

**Preference vector**: [0, 0, 1]. **Response**: The coordinates 40.728157, -73.794853 are located in Manhattan, New York City. The nearest airport based on these coordinates is: 1. LaGuardia Airport (LGA) - Approximately 2.5 miles away. 2. Kennedy Airport (JFK) - Approximately 5.5 miles away. 3. Newark Liberty Airport (EWR) - Approximately 9.5 miles away Please note that the actual distances may vary depending on your location within the city and the mode of transportation you use.


**Preference vector**: [0.6, 0.2, 0.2]. **Response**: You want to know the nearest airport to the coordinates 40.728157, -73.794853? Well, let me just break out my trusty air-traffic-controller-in-a-paper-rocket-ship and zoom in on those coordinates. crunching numbers Ah ha! It looks like the nearest airport is JFK Airport in New York City, with a fancy-schmancy IATA code of JFK! But wait, there's more! If you're feeling extra adventurous (or just really, really close), you can also hop on the Fulton Street Subway Line and take it all the way to LaGuardia Airport, another fabulous airport right in your backyard. So there you have it! The nearest airport to these coordinates is literally just a hop, skip, and a few subway stops away (all while enjoying an espresso).
Dear Reviewer wdga,

We sincerely appreciate the time and effort you've devoted to reviewing our work. We understand that your schedule may be quite busy. As the authors-reviewer discussion phase draws to a close, we kindly request your attention to our responses. Our aim is to gain insights into whether our responses effectively address your concerns and to ascertain if there are any additional questions or points you would like to discuss. We also hope that if you are satisfied with our answers, you could consider adjusting your score and confidence accordingly.

We look forward to the opportunity for further discussion with you. Thank you for your thoughtful consideration.

Best regards, The Authors
Dear Reviewer ZtuD,

We sincerely appreciate the time and effort you've devoted to reviewing our work. We understand that your schedule may be quite busy. As the authors-reviewer discussion phase draws to a close, we kindly request your attention to our responses. Our aim is to gain insights into whether our responses effectively address your concerns and to ascertain if there are any additional questions or points you would like to discuss. We also hope that if you are satisfied with our answers, you could consider adjusting your score and confidence accordingly.

We look forward to the opportunity for further discussion with you. Thank you for your thoughtful consideration.

Best regards, The Authors
Thank you for your thorough and convincing rebuttal.

I noticed that you evaluated your methods on 7B models. I’m curious about their effectiveness on smaller models as well.

Could you also test your methods on the following two models and **include the results in your appendix**?

Phi-3: https://arxiv.org/abs/2404.14219
MiniCPM: https://arxiv.org/abs/2404.06395

Many edge users are particularly interested in the performance on these smaller language models (SLMs). I apologize for the late response. Please let me know if there’s anything else I can do to assist.