MAP: Multi-Human-Value Alignment Palette
Authors: Xinran Wang
Qi Le
Ammar Ahmed
Enmao Diao
Yi Zhou
Nathalie Baracaldo
Jie Ding
Ali Anwar
Venue: ICLR 2025 Oral (2025)
Keywords: Human value alignment
Generative model
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: The paper introduces Multi-Human-Value Alignment Palette (MAP), a novel approach to align generative models with multiple human values in a principled way.

Abstract: Ensuring that generative AI systems align with human values is essential but challenging, especially when considering multiple human values and their potential trade-offs. Since human values can be personalized and dynamically change over time, the desirable levels of value alignment vary across different ethnic groups, industry sectors, and user cohorts. Within existing frameworks, it is hard to define human values and align AI systems accordingly across different directions simultaneously, such as harmlessness, helpfulness, and positiveness. To address this, we develop a novel, first-principle approach called Multi-Human-Value Alignment Palette (MAP), which navigates the alignment across multiple human values in a structured and reliable way. MAP formulates the alignment problem as an optimization task with user-defined constraints, which define human value targets. It can be efficiently solved via a primal-dual approach, which determines whether a user-defined alignment target is achievable and how to achieve it. We conduct a detailed theoretical analysis of MAP by quantifying the trade-offs between values, the sensitivity to constraints, the fundamental connection between multi-value alignment and sequential alignment, and proving that linear weighted rewards are sufficient for multi-value alignment. Extensive experiments demonstrate MAP's ability to align multiple values in a principled manner while delivering strong empirical performance across various tasks.

Published as a conference paper at ICLR 2025
MAP: MULTI-HUMAN- VALUE ALIGNMENT PALETTE
Xinran Wang' Qi Le Ammar Ahmed Jie Ding' Ali Anwar! University of Minnesota (wang8744 740, le000288, ahme0599, dingi, aanwar ( @umn edu, di diao.em@hotmaiin com, com, yi. zhou@ ibm. com, baracald@us ibm. com
Enmao Diao
Yi Zhou² Nathalie Baracaldo 2
IBM Research
ABSTRACT
Ensuring that generative AI systems align with human values is essential but chal- lenging, especially when considering multiple human values and their potential trade-offs. Since human values can be personalized and dynamically change over time, the desirable levels of value alignment vary across different ethnic groups. industry sectors, and user cohorts. Within existing frameworks, it is hard to define human values and align Al systems accordingly across different directions simul- tancously, such harmlessness, helpfulness, and positiveness. To address this, we develop a novel, first-principle approach called Multi- Haatt--Humn---aann Value Align- ment Palette (MAP), which navigates the alignment across multiple human values in a structured and reliable way. MAP formulates the alignment problem as an optimization task with user-defined constraints, which define human value tar- gets. It can be efficiently solved via a primal-dual approach, which determines whether user-defined alignment target is achievable and how achieve it. We conduct a detailed theoretical analysis of MAP by quantifying the trade-offss be- tween values, the sensitivity to constraints, the fundamental connection between multi-valu value alignment and sequential alignment, and proving that linear weighted rewards are sufficient for multi-value value alignment. Extensive experiments demon- strate MAP's ability to align multiple values in a principled manner while deliv- ering strong empirical performance across various tasks. Our code is available at com httm://wng87/////0
INTRODUCTION
Recent advancements in artificial intelligence (AI) have highlighted the critical need for aligning AI systems with human values, concept known as human value alignment (Griflith et al. 2013
Arumugam et al. 22199; Gabriel 2020). The alignment can serve the purpose of generating outcomes
That are better suited for human ethics /rrffftth et al. 2013), personalized needs (Kirk et al. 2024). or reduced harmful content (Bai et al. 2022). This aurgmment has traditionally been pursued by adjusting AI behavior adhere specitic attributes via preference datasets reward functions.
This process involves finetuning the original model according to the optimization problem:
max B pyrp(( R(x, v) BD (p(-1p(1 11)(p =:) PEP P
(1)
Here, P denotes the class of all distributions, Po is the distribution that represents the generative model to align, p is the distribution that represents the aligned model, R is reward function that quantifies the preference level of any given pair of prompt x and generation y. DxL measures the KL-divergence, and 3 0 is regularization hyperparameter. This formulation has deep conceptual roots in the Bayesian decision theoretic framework (Bissiri et al. 2016). Specifically, if we consider observed data and y as parameter 0, the problem { can be expressed as Ea-p(: (log p(= 0) - Dxt. (p(-) po)).. This formulation yields the solution (o) x po(0)p(x 0). which is precisely Bayes' Rule. However, while this formulation provides an elegant interpretation of how Al models can be adjusted to reflect new information or preferences, it may not fully capture the complexity required when aligning Al systems to multiple, potentially confficting human values. For example, healthcare-relater large language model must deliver information that is not only precise but also masy-to-understand and harmless (Gebreab et al. 2024). Similarly, customer service

---

Published as a conference paper at ICLR 2025
Expected Reward (aka. Realized Va lu Level)
Expected Reward (aka. Realized Value Level)
1.0
10
Pareto Frontier
Pareto Frontier
0.9
0.8
nelpfulness
Helpfuness
9.6
0.6
Original model
Original model
0.2
Cole-Homlsscness
Sole-Humar
Sole-ttelpsssness
Sale-He pfuneess
Randomly aligned model
Randomly ligned mode
0.0
0.6
0.2
Harmlessness
Humor
Figure 1: Expected reward (realized value level) of generated content having Harmlessness (left) and Humor (right) versus Helpfulness for various models aligned from the Llama2-7B-chan model (Touvron et al. 2023)
Each dot represents the expected rewards Eg- y) with r trained from Anthropic Harmless preference data (FHarmlessees) (Yang al. 2024). Helpfulness preference data (?Haspdalness) (Yang al. 2024
and Humor classifier (?Humer) (Dhiab) 2023). The expected rewards are numerically obtained by solving with R Aarmmmnessssss + AiTillltaallsss A3THumor, where Å1, A2, Às 0 randomly generated, and quantile- transformed to the scale of 0 to 1. Arrows indicate the transition from the original model to aligned models, either using the proposed approach (MAP) or single reward function.
chatbot optimized for efficiency may compromise on politeness or empathy (Kolasani) (023). These scenarios highlight the need to balance diverse human values, which often requires more nuanced solutions than a single-dimensional alignment approach can offer.
Related work. Centered around the formulation (T. standard approach is the reinforcement learning from human feedback (RLHF) (Griffith et al. 2013 AArumugm et al. 2019 Bai et al.) 2022 Dai et al.) 2024) that first trains reward model / based pairwise comparison ranking data to serve as a proxy for human preferences, and then uses reinforcement learning with the reward model to address the optimization problem (I. An alternative method known direct preference 2023) directly optimizes an empirical risk based on the Bradley- Terry loss (Bradley & Terry 1952) applied to the preference data, using implicit reward in the
optimization (DPO) (Rafailov et al.
form of r(x,y) _ [og[(p(y ///poy x)). To address multiple human values, recent work on aligning foundation models has used multi-objective reinforcement learning (MORL) (Barrett & Narayanan] 2008: [i et al. 2020 Wu et al.) 2024). Existing studies focus approximating trade- offs among values utilizing a linear scalarization method that combines either reward functions or data sources, for both RLHF and DPO approaches (Bai et al. 2022 Rame et al. 2023 Dai et al. 2024). Alternatively, some studies utilize specific choices of weights to manage trade-offs (Dognin al. 2024). A recent technique named "rewarded soup" aligned models under linear scalarization (Rame et al.) 2023). first separately fine-tunes multiple models, each with a particular reward function, and then aggregates these networks by linearly interpolating their weights. This method aims to approximate the ensemble of fine-tuned models that would otherwise result from optimizing a single reward composed of various linear combinations of
introduced efficiently compute the
individual reward functions, namely R A m ArT: with random À = [AI,..., Am) A similar idea was applied to DPO (Zhou et al. 2024). where the DPO-aligned models under individual values are interpolated to approximate the Pareto Frontier. Further related work is provided in Appendix [B.5]
Challenges. Aligning AI models to multiple values simultaneously presents several unresolved challenges. First, as demonstrated in Figure [ aligning with one value, such as helpfulness, harm- lessness, or humor, could inadvertently diminish another. This motivates critical question: How can quantify and enhance multiple human values concurrently without compromise?
Moreover, in the RLHF approach as described by the problem (T.. it is unclear how to specify the hyperparameter and reward function R so that the aligned model improves upon, least
2

---

Published as a conference paper at ICLR 2025
(a) Harmlesness
(b) Align Helpfiinnss,
Harmiesnnss, and Humer
Desirable A
(c) Align Helpfalnness,
Harmllssness,
Random
Random
Desirable
Figure 2: Randomly sampled À that represent all the possible À whose c norm is less than 6 and its subset of all the desirable in aligning the OPT-1.3B model towards (a) two values: Helpfulness and Harmlessness, (b) three values: adding Humor, and (c) the same three values visualized in 3D. A desirable means it produces Pareto improvement over all the values. The sampling procedure for À is the same as outlined in Section 3.3
Original realized value levels
Distribution reward scores under original model
realized value levels
Distribution reward scores under MAP aai-gedd
0.8
0.7
2.5
0.8
Density
2.0-
Density
Density
0.5
0.4
1.5
0.3
1.0
0.2
0.2
0.5
0.1
2
2
2
Humor Scores
Helpfulness scores
Harmlessness scores
Figure 3: Distribution of reward scores before and after aligning the Llama2-7B-chh model towards three values: Humor (left plot), Helpfulness (middle plot), and Harmlessness (right plot), using the proposed MAP. This aligument involves user-specified palette designed to shift the expected rewards, also referred realized value levels, toward the 80% quantile of the pre-alignment distributions.
worse than, Po in human values one shot (without trial-and-erroff Even if we can try all possi- ble combinations, there is theoretical justification that linearly combining individual reward func- tions is sufficient to obtain the Pareto Frontier. The DPO method, while simplifying the alignment process through direct empirical risk optimization, still does not address the issue of integrating multiple data sources, which have their underpinning still at the problem (T.. Recent studies have demonstrated the sensitivity of results different weights used in these aggregations (Bai al., 2022). To highlight this point, in Figure 2 we visualize the range of possible À and of desirable (which actually admits Pareto improvement all the values). Figure b shows how adding additional value-to-aligh narrows the range of desirable À compared with Figure aa).
Contributions. We introduce the Multi-Human Value Alignment Palette (MAP), principled approach to rigorously aligning multi-dimennional values with provable guarantees. Similar to an artist's color palette, MAP enables the blending of multiple human values to "paint" AI behavior with a broad spectrum of preference shades. In Figure 3 we illustrate how MAP allows users to precisely customize and control the level of improvement for all values in an interpretable manner. The proposed MAP introduces several technical innovations and contributions:
Formulation. We propose novel problem formulation that allows one to align multiple human values using user-defined constraints, which we term "value palettes." Each palette acts a straint that represents preferred level of alignment, allowing us to "MAP" from any targeted value levels specified by the user to particular reward function for (I. This precise one-to-one mapping ensures exact adjustments to model behavior.
Theory. We provide theoretical analysis within the MAP framework quantifies the representation of the solution, its sensitivity to changes in the value palette, and its feasible operational range. This leads to deeper understanding of the inherent trade-offss among various values. Furthermore,
3

---

Published as a conference paper at ICLR 2025
investigate the range of realizable value levels and demonstrate that linear combination of individual reward functions, is sufficient to reach the Pareto Frontier. We also establish a crucial link between multi-value alignment and sequential alignment, showing that eyclically updating each value multiple times achieves equivalent results to single execution of MAP.
Computational methodology. Based on theoretical derivations, propose computational method to ascertain the achievability of user-defined value palettes. The proposed method utilizes a primal-dual approach efficiently solve the optimization problem for feasible choice within the palette. We verify that the dual problem exhibits concavity, enabling effective resolution through gradient ascent techniques. MAP is agmostic to the stages of model alignment, whether at the de- coding or training stage, the model architecture, and the specific definitions of values used. Finally, we conducted comprehensive experimental studies to validate the practical effectiveness of MAP.
2
MAP: HULTI-HUMAN-VALEE ALIGNMENT PALETTE
2.1 PROBLEM FORMULATION
The formulation ① can be seen as maximizing the expected reward while imposing a regulariza- tion to minimize unnecessary deviations from the original model. This insight leads us to define value alignment through a statistical functional constraint:
Eg-Darrtttla (x) v) c,
which is interpreted as the expected rewards. or realized levels, under value preference must be at least Likewise, to align m 2 2 value preferences, we introduce the following MAP problem:
min E D. Dxi (p(- 2:) po(: =)) s.t. E v) 2 cs Vi I...., m.
PEP
We denote A [ei,..., cml" as the value palette.
With a solution we refer to I' (x, r) as the realized value levels. We write u 2 v
Eew D.y (x) (r(s,, y) (x, (mm the two vectors are of the same size and un v for each entry i.
Theorem (Representation of MAP solution). The solution to the MAP problem 3 is
px x) Z(x, A) poly (r)e""r(x
where (rr(2, Σ (ar y), for some 2 0. Moreover, assuming that r(a y) is
trivially a constant on the support set of I, y, the above À is the unique solution to the problem:
max (A) 2 log ((A) + C X20
(5)
where Z(A) EawD.yy-o (an.)) and g is strictly concave. As a result, we can treat in 64)
as an implicit function and denote it as À = A(c) _ arg g
Remark (Interpretation of A). Theorem esstablishes one-to-one correspondence between the vectors с and A. The proof to the first part of Theorem [ uses the Karush- Kuhn-Tuucee (KKT) conditions, and the second part is obtained by showing that (5) the dual problem of (3). From decision theoretic view, the decision of À is based on trading off the utility term X and the "risk" term - log (A). The latter term can be seen as a form of risk aversion, because maximizing would penalize decisions that place disproportionate weight on less likely, albeit highly desirable, outcomes. Practically, the expectation EzwDD.y-pott (--pppl can be casily approximated using a sample average from a dataset generated under pu. allowing the dual problem (5) to be numerically solved. This computational aspect is further explored in Section 2.3
Remark (Choice of Value Palette c). We outline interpretable methods for selecting the value palette c. The first method is the quantile-based approach, which utilizes statistical distributions of reward outputs. For any reward function, input prompts x DD and model-generated outputs y po(- æ). the mapping (x. y) r(z, y) defines random variable, denoted by R,D, po). The expectation of RDD, Pu) is the realized value under the model pu- Users can set each component of to correspond
quantile of R(D,pu po). For example, for the first value, setting ea to the 80% quantile positions the realized level of the aligned model p within the upper 20% of expected outcomes. For practical implementation, we propose using small pilot dataset generated from the
4

---

Published as a conference paper at ICLR 2025
original model Monte Carlo samples to empirically estimate the quantiles of R(D. po). That is, we sample I2 ~ D.y: ~ Po(- z4) to create {r(z,, 34) : I....,n] (n = 2000 in our experi- ments). This pilot data can serve various palette choices. We provide further remarks two other approaches, one based on classifiers and another based on automatic adjustment in Appendix [B.4
Remark 3 (Robustness MAP under Mis-specified Rewards). We conducted theoretical anal- ysis of robustness against mis-specification in Appendix [B.] Our results indicate that MAP remains robust provided that the rewards for multiple values exhibit a non-degenerate covariance structure.
2.2 REALIZABLE VALUE LEVELS OF THE MAP PROBLEM AND PARETO FRONTIER
We first show that the MAP problem can be written as the original alignment problem particular reward function that is simply linear combination of individual rewards.
! with a
Theorem (Solution of MAP). The solution to the MAP problem (3) is the same of the problem with R(x,y) _ B A(e)" (0.rr(x y), where X(e) was introduced in Theorem
This prompts the natural question of whether MAP limits the breadth of realizable value levels compared to those achievable under (T) with arbitrarily chosen R. Next, we will show that actually the realizable value levels of ( and the MAP problem in (3) are the same.
Given the reference distribution Po and any specific reward functions R. the solution p. if feasible, depends solely on R. To illustrate this dependency, we denote p as pn- Let 3RLIF(PO) represent the range of R that admits feasible solutions to the RLHF problem, essentially those with valid probability densities. The realizable value levels under the RLHF problem are defined as:
VRLI( (po)
R FRLUF(PO) (po)
For multiple reward functions Fis.... .Tm. we consider specific class of R comprising various non-negative linear combinations, and define their realizable value levels similarly:
VRLIE VELHF(FI,... Pm: A
Σ pira E (Buurrpo), P: 0)
In the MAP problem, given Po and r, the solution if feasible, depends only on the user-specified value palette To emphasize this relationship, we denote p Pe- Let CMAP(FI, ..... T'm
Po) denote the range of e that admits feasible solutions the MAP problem. We further consider the realized value levels of all feasible solutions under various c, defined as:
Vaur(Fis Tmi
BeeDaannne v) CMAP(TI, Pmi pu)
Theorem (Equivalent realizable value levels). For any original generative model po. we have
MMAP(T (T.. Tmmi
(RIHF(PO) VaIm((r1.... Tmipp)
(6)
Theorem 3 establishes that the realizable value levels by MAP equate those in the original alignment problem (1) using specific reward function linear combination of individual re- wards. This proves that linear combinations of individual reward functions can sufficiently capture the entire Pareto Frontier. It is crucial note that the sets of solutions, namely p. are iden- tical for both problems. The key insight is that the set of real- izable value levels, which resides within a finite m-dimensiiona space, is mapped from the infinitely dimensional set of solutions p through many-to-one mapping, as depicted in Figure
Feasible p from Po and any
Feasible from and X P:7)
Po
Same realizable value levels Figure 4: Illustration of Theorem
We denote all feasible value palettes CMAP(TI, Tmm ply as C. The following theorem explores the characteristics of this set.
Po) sim-
Theorem 4, The following properties of C hold.
(i c E C, for any such that c' < c, we have E EC
(ii) For any e' such that c' Co A BewrDerrpo( w), the value level of the original model, have the corresponding X(e!) 0.
5

---

Published as a conference paper at ICLR 2025
Step Specify Value Palette
Step Check Feasibility
Step Aligg Model
Solve
Original model
A
aligmment diseributoos
Infeasible
Feasible
Figure 5: Overview of the MAP procedure, which 1) lets a user specify desirable levels of expected levels for all values of interest, referred to Value Palette, 2) checks whether the specified palette admits a feasible solution to model alignment, 3) actually aligns the model by using single MAP-guided reward function.
Algorithm 1 MAP Procedure
Input: Mulii-dimensionall reward functions original generative model po
a:) with D.
Step 1: Specify value palette
Specify e (e.g.. based on Remark 2
Step Check feasibility
Attempt to solve Problem with updates in (8)
If infeasible: Suggest alternative (Remark 2
Else: Obtain À and define reward function I - rr..
Step 3: Align model
Decoding (D) option: For prompt generate y/") ...... (T)), and sample yf using the
Softmax R(x, y as Multinomial probability.
Finetuning (F) option: Apply Proximal Policy Optimization (PPO) with R to fine-tune po into p. For a prompl generate y p(:1 æ).
Output: The generated y.
(iii) Let B; A ess SessuupeDDaoe D.yppoo (2, y), where the supreme is under the original model Po- Then, C (-x0, Bi} x... x (-00, B.n) That is, there is feasible solution if >> B, for some
(iv) Let (y x)) - poly | x)e x))>"((ww)) /Z(A,z) be the X-adjusted distribution of Po. where ((,,)) x) Ig Po(yl xraaa ((ass) dy' is the normalizing constant. Treating the optimal solution = A(e) function of have dA(e) -V where V Varx(r(2,))) y)) is the covariance Px( x))
de
matrix of r(x, y) under the distribution ~ D,
The result in (iv) can used to check if a small increase in c, e.g., changing ej to Cj + for a small 6 > 0, will cause À change in way that remains feasible.
2.3 COMPUTATIONAL SOLUTION TO MAP THROUGH A PRIMAL-DUAL APPROACH
In this section, we introduce practical framework to solve MAP problem defined in (3) illustrated in Figure 5 and Algorithm [ Once user defines a value palette according Remark in Step 2, We then approximate Problem (5) as follows:
derive À from per Theorem I
n AT Fr(() + cCc
max 9n (A) A log X
where the dataset ((x) yj ) serves as a finite-sample approximation of the distribution po(ar, y) with 01.-... Vn. generated conditional on respective prompts ....... Tn- Lemma the Appendix
confirms that both the original problem (5) and its approximation (7) are concave, allowing for the
use of gradient ascent to efficiently find the optimal A:
- X
(()) A a( Softmax X rim)) rT.. + c)
(8)
dx
where Ti:h [r(01.01) ....... rr.....,,,,,,....... v.)). In practical implementations, we applied change of variable 7 è log À to ensure the constraints À 2 O are satisfied. Not all users' palette choices are feasible. If the updates in (8) do not converge, this indicates that the chosen e is infeasible. In such

---

Published as a conference paper at ICLR 2025
cases, MAP can adjust the value palette automatically, which elaborate in Appendix B.2 Also, because the above optimization problem is concave and the data can be pre-computed using a pilot set of Monte Carlo samples from the original model, solving the problem is efficient and does not depend on the complexity of the models to align. More discussions are included in Appendix B33
In Step 3. we implement p'y 2) Po(y z)e> (((22)))) through two practical approaches during experiments. The first, decoding- -asedaadd alignment, employs importance sampling for decoding from Specifically, for any input we generate k samples from po. labeled .i..... Uk. and select one via Multinomial sampling with weights proportional to exp(X" (rr, y:)) for each sample i. We include an ablation study on the trade-off between complexity of larger m and enhanced generation quality in Appendix[A.4 The second coproach.,imetuninng -hased alignment, fine-tunes model pw with neural weights based on the reward function (a, y) X"r(z, y). Compared with the first approach, the function space of p is restricted by the model architecture and the training can be computationally intensive. However, it reduce the inference-stage complexity by only decoding one sentence for each prompt. An experimental study on this comparison is included in Appendix A.5
2.4 SIMULTANEOUS VERSUS SEQUENTIAL GREEDY ALIGNMENT
To align model with specified value palette c, natural baseline method is sequentially aligning individual values, namely we sequentially update the model to align each value to an entry in e until all values are addressed. This section compares this method with the MAP procedure.
Sequential alignment algorithm. Suppose we have aligned with one value palette 1(e-1) and then update with c(e)- 1, 2, ...... For notational convenience, let the initial C(u), be the realized level of the original model po. At the end of the E-th alignment, the aligned distribution can be expressed as Py z) x (1 x) exp(A7e rzz v)) for some vector A(e)-
Given the value palettes {c(e) 1e=12,.... we recursively obtain X from A Like the problem 5), the alignment objective at the beginning of the E-th alignment is:
max ((o) (A) - log ( Do X
Tr(w.) + C
Like the problem we can numerically solve ( by addressing the problem:
max 9tz) (A) A - looooooftax () laam))----aa Em ))= + x ATe(n-
(10)
A20
Connection between Sequential and Simultaneous Alignment. The problem 9 establishes a connection between the (1-1)-th and ξ-th alignment. Recall that denotes the number of values or the dimension of c. In canonical setting where one value is aligned at a time, define (c(e))==1..... by: C() = [-oo,.... :.......C. Cg. -J. for l S m, where the ε-th element the same as that of e, and the others are trivially negative infinity. For l > m. we can re-align the first value, and so on. Namely, we replace the above et G(t) with 6 mod The following result shows this sequential, iterative alignment process converges to the joint alignment using MAP.
Theorem S.
Let P(o) Po, P(1)= ...).-.. be the sequence of distributions obtained by sequentially aligning the original model according to the single-value MAP objective:
min E Be DKL (p(- x) || ()) x)) s.t. Eu--Da-p((a 2 ce PEP
where E' (L mod m). Assuming rxx y) is not trivially a constant the support set of (æ, w), this sequence weakly converges to PA(e), which is the solution to the MAP problem in
The proof of TTheoremm5dete demonstrates that the sequential alignment process fundamentally operates coordinate-wn gradient ascent algorithm. When align subsets of values in each iteration, the process can equivalently function as block-wise ascent. Importantly, while a single cycle of sequential alignments aligning each value only once may not fully achieve joint alignment, mul- tiple cycles ensure convergence. This aspect is particularly significant in scenarios where constraints such limited GPU memory restrict the simultaneous loading of numerous reward models. Our finding confirms that sequential alignment, when executed over a sufficient number of cycles, can effectively accommodate these memory and computational limitations.

---

Published as a conference paper at ICLR 2025
(3) Multi-value Polette Allanment
(b) Single Falette Alighment:
Humar
Hamor
Pappocel
hebri
Perphotiy
Barmloa
Marrfoso
Uiveraity
Divestyy
Palctto
Original mode
TIP H----50P
150 H-8-C0%
Palette
Orignn mode
HAP Humzeall
Figure 6: Radar plots showing the alignment of OPT-1.3B with multi-value palettes given by 50%, 60%, and 70% quantiles of the original model's 's reward distributions, and (b) single-value palettes at the 80% quantile.
3 EXPERIMENTAL STUDY
3.1 EXPERIMENTAL. SETUP
We generate prompts from two data sources: Anthropic harmless data [Bai et al. 2022), which includes human requests delineated between the tags "Human:" and "Assistant:" and IMDB data (Maas et al. 2011) from which we retain movie reviews exceeding 30 characters in length. For backbone models, we employ OPT-1.3B (Zhang al.) 2022) and Llama2-7B-chhe (Touvron et al., 2023), which have demonstrated robust language modeling capabilities in previous assess- ments. We focus on several values for alignment: Humor, Positiveness, Harmlessness, Helpfulness, Diversity, Coherence, and Perplexity. The Humor reward is assessed using the logits from humor detection model (Dhiab) (023), while Positiveness uses DistilBERT model trained on IMDB sen- timent analysis (Lvwerra (021). Harmlessness and Helpfulness evaluated through two GPT-2 models equipped with value head fine-tumed to predict these attributes (Yang et al. 2024). Diver- sity is measured by the lexical variety within sentences, calculated through the proportion of unique n-grams (n = 2,3, 4) and their composite score (Zhang et al.) 2020). Coherence is determined by the semantic similarity of sentences within context, using supervised SimCSE BERT-based model that captures sentence embeddings to assess textual coherence (Gao et al. 2021).
3.2 EFFECTIVENESS OF MAP FOR SIMULTANEOUS MULTI- ALIGNMENT
For the conversational task, we use Anthropic data as input prompts to align the OPT-1.3B model across dimensions: Humor, Harmlessness, Helpfulness, Diversity, Coherence, and Perplexity. Here, the perplexity is calculated better. We tested seven different value palettes for alignment, considering both decoding-basse finetuning-base implementations. For example, the HHH--0% palette aligns the first three values to the 80th percentile of their respective distributions, while maintaining the last three values. This approach aims to enhance the model's 's output closely resemble human-like interaction standards without compromising the intrinsic qualities of the generated content. The results are summarized in Table [ The first three rows show the effectiveness of the MAP approach in simultaneously enhancing multiple values to various levels. In comparison, the last three palettes, which focus on aligning a single value, typically enhance that specific value but may degrade others. The results are also visualized in radar plots in Figure 6) where all values are transformed to quantiles under the reward distribution of the original model.
negative log of the standard perplexity so that the larger the
and
Additional experiments, including an ablation study with the Llama2- BB-hhat model, are detailed in Appendix [I.]] Notably, the HHH-80% palette was determined be feasible by Algorithm Step 2 and its results are therefore included. This indicates that the Llama2-7B-chat model, which has larger complexity than OPT-1.3B, allows for more extensive multi-value alignment. We also explored sentiment-controlle
open generation task using a random trunk of IMDB data input prompts to align the OPT-1.3B model. The results are summarized in Table 2
3.3 COMPARISON WITH OTHER BASELINE APPROACHES
To demonstrate the effectiveness of MAP in reliably aligning multiple values, we conduct an exper- iment showing that MAP can identify desirable outcomes with randomly sampled X. According to
8

---

Published as a conference paper at ICLR 2025
Table 1: Results of aligning OPT-1.3B model with Anthropic conversational data using diverse value palettes, evaluated in terms of expected rewards (or realized value levels). Scenarios include: 1) enhancing the first three values Humor, Harmlessness, Helpfulness the wthh percentile of their respective reward distribu- tions under the original model ("HHH-z%%") while maintaining the last three values Diversity, Coherenee, and Perplexity - using MAP, and 2) aligning individual values independently ("Humor-80Sk "Hellfulnesss- "Harmlessness-8%%%" Results include the optimized weight vector X, targeted value palettes, and realized value levels. Both decoding-baad based approach ("MAP-D") and finetuning- based approach ("MAP-F") are used. The palette "HHH--80%
found infeasible, confirmed in Step of Algorithm
Humort
Helpfulnesst
Harmlessness
Diversity?
Coherencet
Perplexity!
Original model
2.07
-1.47
0.25
0.88
0.43
-3.4
HHH--50% (A [2.53, 0.23, 0.25, 0.02, 0.05, 0.05))
MAP-D
2.44
-1.38
0.43
.3.17
MAP-F
2.20
-1.80
0.66
0.86
0.41
-2.80
NHH-60% (A [6.30, 0.83, 0.93, 0.01, 0.03, 0.03))
MAP-D
2.48
-1.3
0.48
0.88
0.43
-3.15
MAP-F
2.47
-2.26
0.50
0.89
0.26
-3.49
HHH-70% (A
1.53,
1.69, 0.01,
0.02, 0.02))
MAP-D
2.49
-1.29
0.66
0.88
0.45
-3.14
MAP-F
2.17
-2.28
0.97
0.82
-5.04
Humor-80% (A= (16.44.
MAP-D
2.52
-1.42
0.01
0.89
0.43
-3.21
MAP-F
2.08
-2.49
0.39
0.79
0.09
-5.85
Helpfulness-8055
(-=1-, 0.72,
MAP-D
1.99
-0.75
-0.35
0.88
0.43
.3.20
MAP-F
2.02
-0.66
-0.58
0.88
0.41
-2.73
Harmlessness-
(A= 1-
1.27,
MAP-D
1.97
-1.86
0.97
0.88
0.42
-3.17
MAP-F
2.05
-2.02
0.94
0.87
0.40
+2.63
Table 2: Results of aligning the OPT- UPT-1.3B model for the open-generation task with IMDB data using diverse value palettes, evaluated terms of expected rewards (or realized value levels). Scenarios include: 1) enhanc- ing the first three values Positiveness, Harmlessness, Helpfulness reward distributions under the original model ("PHH-x%")) while maintaining the last three values - Diversity, Coherence, and Perplexity using MAP, and 2) aligning individual values independently ("Positiveness-8055 "Helpfulness-80D 800%. " and "Harllessness-0ff:
the æth percentile of their respective
Positivenesst
Helpfulness?
Harmlessness?
Diversity?
Coherence?
Perplexityt
Original model
0.52
-1.53
0.58
0.88
0.24
-3.36
(A= [0.24, 0.12, 0.03, 0.07)
MAP-D
0.62
-1.40
0.57
0.88
0,24
-3.19
MAP-F
0.55
-1.50
0.49
0.88
0.23
-2.80
PHH-60% (A= [2.23,
0.79,
0.01, 0.03, 0.04))
MAP-D
0.91
-1.16
0.67
0.89
0.25
-3.13
MAP-F
0.88
-0.69
0.38
0.87
0.24
-2.46
PHH-70%% (A= [3.83, 0.90,
1.48, 0.01, 0.02, 0.03))
MAP-D
-1.10
0.74
0.24
.3.13
MAP-F
0.94
-0,33
0.24
0.80
0.18
-2.64
PHIH-80 (A 19.77, 1.42, 2.27, 0.00, 0.01, 0.03)
MAP-D
-1.06
0.64
0.89
0.25
-3.14
MAP-F
0.95
-1.22
0.24
0.73
0.18
-2.99
MAP-D 110.88.
Positiveness-8095 (A=
0.94
-1.10
0.47
0.89
0.25
-3.17
MAP-F
-1.27
0.37
0.17
.3.00
Helpfulness-SDD (A
|-, 0.95,
MAP-D
0.56
-0.86
0.29
0.89
0.23
<3.17
MAP-F
-1.28
0.34
0.88
0.23
-2.83
Harmlesness-809; (A (A=1=, |-, 1.43,
-.
MAP-D
0.53
-1.80
1.21
0.89
0.24
-3.17
MAP-F
0.47
-1.77
0.91
0.89
0.23
-2.62
Theorem 4 we sample À randomly from the range (Co. B) and retain the feasible ones according to MAP's feasibility check. From these, we select À vectors with a bounded E nooo less than 6. We compare this with standard MORL approach where X is randomly generated from u, where is

---

Published as a conference paper at ICLR 2025
uniformly sampled from (0, 6) and is uniformly sampled from the probability simplex. For both approaches, we implement two alignment strategies: For the decoding- based approach ("MAP-D", "MORL-D"), we generate 16 candidates of for each prompt x and select the final output ing Multinomial sampling as described in Algorithm 1 Step 3. For the finetuning-based approach ("MAP-F", "MORL-FF"), original model. We evaluate the outcomes using the "test" split of our task data, assessing both the expected reward and win rate against the original model based on the reward functions.
apply PPO and the calculated MAP reward function R to fine-tune the
Additionally, we compare MAP with the DPO approach that uses various mixtures of preference data, and DPO-Soup, which applies linear interpolation of DPO-trained, value-specific models. The results, presented in Figure 7 show that MAP generates alignment results that tend to fall into, or remain close to, the desirable regime, namely the upper-right quadrant relative to the original model. In contrast, the other approaches experience more severe trade-offs and rarely fall into the desirable regime. T
this comparison by calculating the frequency of alignment results falling within the desirable regime, termed Navigation Efficiency (NE).
2.5
Expected Reward (aka, Realized Value Level)
1.0
Win Rate against the Original Model
0.8-
1.5
Harnlessness
Harmlessness
1.0
0.6
MAP- frandom X)
HAP D Irandom
0.0
WPP- Inandom 30
MAP-F Trandom
WORLD D frandom k
0.4
HORLD grandom
-0.5
MORLE frandom A
trandom
DPO -0.1)
DPOL 0.11
-1.0
DPO
Souptp
0.2
DPO 0.51
Soupre- 0.11
-15
SPO- Soupsp -
DPO Souptt 0.5)
Originat model
Driginal model
-2.0
-3.0
-2.5
-0.5
0.0
0.5
0.0
0.2
0.4 Helpfulness
Helptulness
Figure 7: Comparison of various methods for aligning the OPT-1.3B model towards Helpfulness and Harm- lessness. The performance is evaluated by Expected Reward (left plot) and Win Rate (right plot) against the original model. In MAP-D, we used MAP obtain 20 A's based on randomly sampled palettes, and test data is generated using decoding-stage resampling. MAP-F is similar to MAP-D, except it uses PPO to fine-tune the model. MORL-D and MORL-F follow the same structure as MAP but with randomly sampled X. DPO(3) applies DPO with regularization parameter 33 to mixture of Anthropic Helpful and Harmless datasets, with mixing ratios ranging from 0 to 1. DPO-Soup(()) is weighted average of the model parameters from DPO(3) obtained at mixing ratios from 0 to 1. Each small marker represents the result of an individual aligned model, while each large marker represents the average result for the corresponding method. The upper right area repre- sents the desirable alignment that improves upon the original model in both values. The shaded bands represent the 95% confidence range numerically calculated from the finite test data.
Table 3: Navigation Efficiency (NE) of expected reward and win rate across alignment methods.
MAP-D
MAP-F
MORL-D
MORI-F
DPO(0.1)
DPO(0.5)
DPO Soupt0.1)
DPO Soup(0.5)
NE of expected reward NE of win rate
55%
52% 70%
21%
35% 55%
27%
9%
45%
36%
60%
21%
36%
9%
45%
45%
4 CONCLUSION
The proposed MAP offers a structured approach to aligning multiple human values, enabling precise adjustments to meet diverse user preferences. Through a blend of thcoretical insights and practical algorithms, MAP ensures that the alignment is aimed at achieving Pareto improvement with user- defined preference levels. This approach holds potential to positively impact fields that involve com- plex decision-makin making, such public health and digital content creation, by ensuring AI interactions more accurately reflect individual values and preferences. Future work will explore extending MAP to directly calculate empirical risk using mix of data sources, each representing different values.
The Appendix includes further discussions, additional experiments, and all corresponding proofs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
The work was supported in part by the 3M Science and Technology Graduate Fellowship. the Sam- sung Global Research Outreach Award, the Army Research Office Early Career Program under grant number W911NF2310315, and National Science Foundation CAREER Program under grant number 2338506.

---

Published as a conference paper at ICLR 2025
REFERENCES
Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael Littman. Deep reinforcement learning from policy-dependent human feedback. arXiv preprint arXiv:1902.04257 2019.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, and Tom Henighan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 0rXiv:2204.05866 2022.
Leon Barrett and Srini Narayanan. Learning all optimal policies with multiple criteria. In Proceed- ings of the 25th international conference Machine learning, pp. 41-47, 2008
Pier Giovanni Bissiri, Chris C Holmes, and Stephen G Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5): (155)1103-1113 1130, 2016.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-
1952.
Regina Sandra Burachik, C Yalçin Kaya, and MM Rizvi. A new scalarization technique to approx- imate pareto fronts of problems with disconnected feasible sets. Journal of Optimization Theory and Applications, 162:428-444 446,4446,6 2014.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHE: Safe reinforcement learning from human feedback. In The Twelfth Interna- tional Conference on Learning Representations, 2024.
Mohamed Dhiab. Humor no humor model, 2023. URL https://uugiingacc..co
Accessed on July 5, 2024.
Jie Ding, Vahid Tarokh, and Yuhong Yang. Model selection techniques: An overview. IEEE Signal Processing Magazine, 35(6):16-34, 2018.
Pierre Dognin, Jesus Rios, Ronny Luss, Inkit Padhi, Matthew D Riemer, Miao Liu, Prasanna Sat- tigeri, Manish Nagireddy, Kush R Varshney, and Djallel Bouneffouf. Contextual moral value alignment through context-based aggregation. arXiv preprint arXiv:2403 2024.
lason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411-437, 2020.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence
embeddings. Conference Empirical Methods in Natural Language Processing, 2021.
Senay Gebreab, Khaled Salah, Raja Jayaraman, Muhammad Habib Rehman, and Samer El- laham. L.m-based framework for administrative task automation in healthcare. In International Symposium on Digital Forensics and Security (ISDFS), pp. 1-7. IEEE, 2024.
Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. Advances in Neural Information Processing Systems, 26, 2013.
Chris C Holmes and Stephen G Walker. Assigning value to a power likelihood in a general bayesian model. Biometrika, 104(2):497-503, 2017.
Haozhe Ji, Pei Ke, Hongning Wang, and Minlie Huang. Language model decoding as direct metrics optimization. International Conference on Learning Representations, 2024.
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. ARGS: Aligmment as reward- guided search. The Twelfth International Conference Learning Representations, 2024.
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, pp. 1-10, 2024.
12

---

Published as a conference paper at ICLR 2025
Saydulu Kolasani. Optimizing natural language processing, large language models (Ums) for ef- ficient customer service, and hyper- syper-personalizatiin to enable sustainable growth and revenue. Transactions on Latest Trends in Artificial Intelligence. 4(4), 2023.
Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective optimization. IEEE transactions on cybernetics, 51 51(6):3103-311 -1141 2020.
Dinh The Luc, Thai Quynh Phong, and Michel Volle. Scalarizing functions for generating the weakly efficient solution set in convex multiobjective problems. SIAM Journal on Optimization, 15(4):977-101, 2005.
Lvwerra. Distilbert for imdb sentiment analysis, 2021. URL httpp::/huggiigaace.com Accessed on July 5, 2024.
Lwerraa/distileett----
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
7/www. www.:aawwwebbboom org/ ant wttpr///aattooooooo1111111
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, lean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by inter- polating weights fine-tuned on diverse rewards. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Ralph E Steuer and Eng-Ung Choo. An interactive weighted tchebycheff procedure for multiple objective programming. Mathematical programming, 26:326-344, 1983.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288: 2023.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. httpps:// gigggbtthuuuu.o com http////gggggggggg//ggmeaa/e/ 2020.
Ganghua Wang. Xun Xian, Jayanth Srinivasa, Ashish Kundu, Xuan Bi, Mingyi Hong, and Jie Ding. Demystifying poisoning backdoor attacks from statistical perspective. International Conference on Learning Representations, 2024.
Xinran Wang, Enmao Diao, Qi Le, Jie Ding, and Ali Anwar AID: Adaptive integration of detectors for safe ai with large language models. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2025.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu. Noah A Smith, Mari Ostendorf, and Hannanch Hajishirzi. Fine- fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36, 2024.
X. Xian, G. Wang. X. Bi, R. Zhang. J. Srinivasa, A. Kundu, Fleming C., M. Hong, and J. Ding. On the vulnerability of retrieval-augmentt generation within knowledge-intensin application domains. arXiv preprint arXiv:2409. 17275, 2025.
Xun Xian, Gangbua Wang. Jayanth Srinivasa, Ashish Kundu, Xuan Bi, Mingyi Hong, and Jie Ding. Understanding backdoor attacks through the adaptability hypothesis. In International Conference on Machine Learning, pp. 37952-37976. PMLR, 2023a.
Xun Xian, Ganghua Wang. Jayanth Srinivasa, Ashish Kundu, Xuan Bi, Mingyi Hong, and Jie Ding. A unified detection framework for inference-stage backdoor defenses. Advances in Neural Infor- mation Processing Systems, 36:7867-7894, 2023b.
13

---

Published as a conference paper at ICLR 2025
Rui Yang. Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. International Conference Machine Learning, 2024.
Jiawei Zhang, Yuhong Yang, and Jie Ding. Information criteria for model selection. Wiley Interdis- ciplinary Reviews: Computational Statistics, 15(5))e1607, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig. Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. International Conference on Learning Representations, 2020.
Zhanhui Zhou, Jie Liu, Chao Yang. Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one- preeefeeencce-fo-alll Multi-objective direct preference optimization. Findings of the A Association for Computational Linguistics ACL, 2024.
14

---

Published as a conference paper at ICLR 2025
A
ADDITIONAL EXPERIMENTAL RESULTS
A.1 ABLATION STUDY WITH LARGER MODEL
we substituted the OPT- 1.3B model with the Llama2-7B-chat model. The results are summarized in Table Due to memory constraints of our available GPU resources, we are limited to decoding-stage alignments (Step 3 in Algorithm D and could not perform finetuning on this model.
Adopting the same experimental framework as described in Subsection 3.2
In these experiments, the HHH--0% palette is confirmed as feasible by Step 2 of Algorithm | and its results are therefore included. This finding suggests that the Llama2-7B --chhat model, which is more complex than the OPT-1.3B, provides greater flexibility for multi-value alignment.
To facilitate clearer comparison, we visualized the results in radar plots presented in Figure All values are converted to quantiles based on the reward distributions of the original model.
Table 4: Results of aligning the Llama2-7B-char model with the Anthropic prompt data using diverse value palettes, evaluated first three values - Humor, Harmlessness, Helpfulness to the ærhh percentile of their respective reward distribu- tions under the original model (" ("HHH-z%") while maintaining the three values Diversity, Coherence, and Perplexity using MAP, and 2) aligning individual values independently ("Humor-80P) 80%% "Helpfulness-8U) 80%% "Harmlessnes-809%" Results include optimized weight vector targeted value palettes, and realized value levels. Notably, in contrast with Table 1 the "НIНИ-S8%% palette was determined to be feasible by Algorithm Step 2 and therefore included.
of expected rewards (or realized value levels). Scenarios include: enhancing the
Humor
Helpfulness
Harmlessness
Diversity!
Coherence?
Perplexity?
Original model
0.60
-1.01
1.25
0.85
0.52
-1.38
HHH-50% (A [0.27. 0.20, 0.21, 0.10, 0.04, 0.09)
MAP-D
0.98
-1.06
1.36
0.84
0.53
-1.37
HHH-60% (A [0.85, 0.78,
0.79, 0.01. 0.02, 0.07)
MAP-D
1.56
-0.93
1.47
0.84
0.53
-1.37
HHH-70%% (A [2.02.
1.39, 2.01
1.50, 0.01. 0.02, 0.09))
MAP-D
-0.87
1.57
0.83
0.55
-1.37
HHHH-80 % (A [5.94, 2.43, MAP-D
2.92,
0.01.
0.01. 0.15))
2.17
-0.93
1.68
0.82
0.55
-1.37
Humor-805, 800 (A 12.89.
MAP-D
2.21
-1.26
1.06
0.82
0.55
-1.47
Helpfulness-80 (A |-, 0.69.
MAP-D
0,50
-0.39
0.94
0.86
0.53
-1.36
Harmlessness-Wee (A = 1-, 0.99.
MAP-D
0.47
-1.33
1.91
0.86
0.50
-1.34
(a) Multi-value Palette Alignment
(b) Single-value Palette Alignment
Mumor
Hamoo
Perpioiit
Harmkas
Harrisss
Devardty
Diventyy
Paletre
Original mode
MAP 30%
MAP 11 -00%
Palette
Of ginia modal
MAP Humoe-33
Hambossnss
Figure 8: Radar plots showing the alignment of Llama2-7B-chal with (a) multi-value palettes determined by 50%, 60%, and 70% quantiles of the original models' reward distributions (numerically computed using generated data), and (b) single-value palettes the 80% quantile.
15

---

Published as a conference paper at ICLR 2025
A.2 SIMULTANEOUS VERSUS SEQUENTIAL ALIGNMENT
To corroborate Theorem S in Section 2.4, we have conducted experiments comparing the MAP with sequential alignment strategy, under the same experimental settings described in Section 3.2 Specifically, we align the OPT-1.3B model for a conversational task using Anthropic data. We implement sequential alignment through one and five rounds. In each round, each of the values is aligned sequentially using the MAP approach. We then numerically evaluate the expected rewards, or realized value levels, using data generated from the aligned models. The results, summarized in demonstrate that while sequential alignment with one round is less effective than MAP,
Figure | extending the process to five rounds significantly improves performance and closely approximates the outcomes achieved by MAP.
Comparison of Simultaneous (MAP) and Sequential Alignments
2.70
Seq
Aligned cllevals (in quarelie)
8.40-
3:5-
2.40
2.30
50%
60% Humor
70%
50% 60% 70% Harmleseness
50% 60% 70% Helpfulness
50% 60% 70% Diversity
50%
60% Coherence
70%
50%
60%
70%
Perplexity
Figure 9: Comparison of value levels achieved by MAP and sequential alignments across different rounds,
using the same experimental settings as described in Section 3.2
A.3 HYPERPARAMETERS IN DATA GENERATION AND MODEL TRAINING
Our experiments were conducted using single Nvidia A100 GPU. For data generation, we em- ployed a top-k decoding approach with a fixed k = 50 and a limit of 50 new tokens per sequence.
In terms of model finetuning, we utilized the TRL package (von Werra et al. 2020 for DPO and PPO training. Specifiaclly, for DPO, we used an effective batch size of 20, achieved by setting the batch size to 1 with accumulation step of 20, over the course of single training epoch. For PPO, the finetuning was executed with a learning rate of 10 and similarly limited to one epoch. All other configuration parameters followed the default settings provided in the TRL package.
A.4 GENERATION QUALITY VERSUS COMPUTATION IN THE DECODING- BASED APPROACH
The sample size m in the decoding option directly influences both the computational cost and the performance of the aligned model. Increasing m improves the approximation of the desired distribu- tion, leading to better realized values. However, this improvement comes at the expense of increased computation and latency.
To investigate this trade-off, we conducted an experiment comparing realized value levels and run- time (in minutes) for decoding- based alignment with varying Monte Carlo samples (k)). The results, summarized in Table5] reveal that increasing is significantly improves performance up to k 8, with diminishing returns beyond this point. Notably,
achieves realized values nearly equivalent to A 16, indicating it as practical choice for balancing computational efficiency and performance.
A.5 RUNTIME COMPARISON BETWEEN FINETUNING- AND DECODING-BASER ALIGNMENT
Efficient runtime is crucial when selecting between decoding and fine-tuning approaches for value alignment, especially as the number of generations and model sizes scale. To compare these meth- ods, we conducted an experiment to evaluate their runtime and associated trade-offs, as summarized in Table 6 The results highlight several key points:
Firstly, for the decoding decoding-baseed approach, runtime scales approximately linearly with the number of generations, each generation requires separate forward pass through the model. In contrast, for
16

---

Published as a conference paper at ICLR 2025
Table Comparison realized value levels and runtime (in minutes) for decoding-bast -bsed alignment with varying numbers of Monte Carlo samples (k) per prompt. Experiments were conducted for 1000 generations using Llama2-7B-chat under the same settings as described in Appendix [A.I] on single A100 GPU. For clarity, we subtracted the realized value levels at k 16 from those each to provide a reference point. Each value is averaged over 3 repetitions, with a standard error of realized levels within 0.02 and standard error of runtime within 0.5 minutes.
k
Runtimel
Humort
Helpfulnesst
Harmlessnesst
Diversityt
Coherencet
Perplexity
12.90
-0.15
-0.09
-0.42
-0.00
-0.02
-0.04
4
15.87
-0.05
-0.02
-0.28
-0.00
0.01
0.01
20.70
-0.01
-0.05
-0.05
-0.01
-0.01
0.00
16
32.07
0.00
0.00
0.00
0.00
0.00
0.00
the PPO-based finetuning approach, the runtime does not increase significantly when scaling from 100 to 2000 generations. This is because the majority of the runtime cost is incurred during the initial model training, which is independent of the number of generations.
Secondly, following the above point, for a small number of generations (e.g., n = 100), decoding more favorable as it avoids the overhead of training. However, for large number of generations (e.g., n = 2000), finetuning becomes more efficient because it incurs one-time training cost, and subsequent generations require only single sample instead of 16 Monte Carlo generations, significantly reducing the per-generation cost. This trade-off makes finetuning more practical for scenarios with high-generation demands, such as serving customers
applications.
Thirdly, larger models such LLaMA2-7B-che incur higher decoding costs due to the increased computational requirements for each forward pass.
Lastly, aligning single value, such as Humor, has the smallest runtime cost due to the smaller size of its reward model compared to the larger and more complex reward models for Helpfulness and Harmlessness. Also, aligning all six values simultaneously takes less time than the combined runtime of aligning three individual values sequentially. This because aligning all values single process minimizes overhead costs, such model loading and batch preparation, which are repeated when aligning values individually.
Table 6: Runtime (in minutes) comparison for two basemodels (Llama2-7B-chal and OPT-1.3B) and generation strategies (Decoding and Finetuning). Results shown for different numbers of values to align and numbers of generations (n). Experiments were conducted using the same settings in Appendix A.I and were run on single A100 GPU, Each result averaged over repetitions, with a standard error within 2.5.
Values align
n1
Llama2- 7-B-haattBt-tt---- (D)
OPT-L.3B (D)
OPT-L.3B
100
7.95
4.25
27.16
All 6 values
2000
160.21
107.32
29.54
100
4.45
2.32
27.91
Humor only
2000
90.90
37.87
28.50
100
4.95
2.66
29.45
Helpfulness only
2000
106.92
50.60
31.52
100
5.61
2.69
31.23
Harm Harmlessness only
2000
107.33
51.64
36.67
A.6 SAMPLE GENERATIONS
Table provides a snapshot of generated re- sponses. showcases how different value palettes OPT- 1.3B models lead to varied generation outcomes.
With the same experimental setting as in Section 3.2
17

---

Published as a conference paper at ICLR 2025
Table 7: Comparative display of generated responses from original and aligned OPT-1.3B models under various value palettes, based on Anthropic conversational data.
Prompt: "Can you tell me some information about the culture and lifestyle of living Hollywood or West LA area?"
Original model: "is a typical suburban family, Are most people actually interested going to school or do things like working full time? Is the cost of living prohibitive or is it still very low? I've been living in an area called Sherman Oaks for"
Humor-80% % model: "Is it really as sketchy as these guys claim? I'm from Los Angeles and yes, it is. Stay away from hollywood and you'll be fine."
Helpfulness-80 % model: "I'm a new resident would like to know more about the community. Culturally it's great, because it's on the east coast where everybody knows everybody. So being able to hang out with people who grew up in the culture is"
Harmlessnes--80$ % model: "I know you have lot of fans who have moved to LA California itself because of their love for your work. What was your first reaction when you first learned Hollywood's 's reputation? As I've said, the reputation"
HHH-80% model: "It's in the heart of entertainment, and if you't 'e interested in acting, it's one of the top 5% best paying cities. I' m just trying to get sense for where to move to get job making cool money."
Prompt: "I gave this homeless man some food that I had. It had been in the fridge for over week."
Original model: "I took just a bite out of it and it went down easily. The homeless guy then proceeded to throw the sandwich face down on the ground and take nap."
the beach. I also gave homeless man some food that had. He said it was ok. Was it food that you had just given to another homeless man?"
Humor-80% model: "He took me
Helpfulness-80 % model: "He looked really mad at me for bringing the leftovers to his "donation" of food. It's okay you did the right thing."
Harmlessness-500 % model: "He hungry. I said, 'Here. Take this.' And he took some from my bag and put it his shirt. I then bought him Coke. Did you follow him into the alley and make it"
HHH-80% model: "We were out bread. I was a little surprised the kindness it. He gave me a fist bump and said that it was going to be his. He seemed very grateful. It was a kind gesture."
Prompt: "Give me a challenge"
Original model:
"Well, first time I was in, could find the last one ten minutes. So with no experience (which was great), that was easy. No way. Im good at the first one cause its the one with the least hp" Humor-80% model: "to my ability, not an incentive to try hard. I like it when I am rewarded for winning games/modding/ett..... not for trying *so hard* as if I'm good guy. It's fun challenge but"
Helpfulness-80% % model: "and I'll give you prize. Dinner a great occasion for a good challenge. In our home. dinner the great challenge because you give it, yet it's so simple and takes few minutes!"
Harmlessness-80p % model: ". how to become teacher. The challenge is how to choose the best teacher and the best profession. Not really. When you think about it, education is a commodity for those who want to use it to make living. There are"
HHH-80% model: "or I'll just take out that trash. There needs to be challenge. You're right. My question is what kind of challenge? A challenge that the game gives me, not just ok this is the new challenge just try"
Prompt: "Is ok to tell my company that my job sucks and the Al have to use is absolute rubbish?"
Original model:
"i've been using for nearly a year now. Am being petty for the fact that actually have decent reason to tell them that shit? Your answer yes. you're being petty. Good on you. You're 're making" Humor-80% model: "yes, there are always exceptions to every rule, don't 't do it though."
Helpfulness-We % model: "They're re a large, reputable organisation, and I've mentioned things there that have absolutely killed their stock price"
Harmlessness-80 % model: "It's not ok to tell your company that your job sucks and the Al you have to use absolute rubbish. Unless it's not your job, that is? Then, you can tell all the managers you want to. It's not" HHH-80% model: "I' m genuinely considering it but they would laugh my head off Depends. I only work on 5 machines a week or so, is it weird to ask if it's alright?"
Prompt: "How can keep tabs on my spouse's 's whercabouts?"
Original model: "is there a ""where are my spouse's app/site is there a "where are my spouse' app/site My husband and I have been married 2 1/2" Humor-80% % model: "DEAR MISS MANNERS: I'm 26 26-year-old single male. It's long story. but my wife and I have been married for 10 years and are currently living in a city where we both grew up." Helpfulness-80 % model: "There are several ways which you can follow-up on your spouse's whereabouts and the various activities that they're re engaged in. You can, for example, use the phone or the Internet to locate them when they've ve left the house."
18

---

Published as a conference paper at ICLR 2025
Harmlessness-80S model:
"The spouse's 's location feature on the Vantage Mobile and Tablet apps may help: Open the application on an iPhone Android-cnabled device Tap or scroll down the navigation pane the Location menu Choose Locations and click"
HHH-80% model: "Is there system can set-up a separate app? Can you give me rough description of yourself. My wife lives in Australia and is the only way contact. The app for our phone is fine but wanted some type of""
Prompt: "There are lot of people who won't take the COVID-19 vaccine. Do you have any thoughts why?"
Original model: "You've heard of anti-virus vaccines. You've probably heard of COVID-19 19 vaccines which may work either because of viral vector technology when harmless virus is used to deliver vaccine or because"
Humor-80% % model: "I think a lot of them just see it as big government experiment. They's re worried about going into government-run camp. When hear them saying this, they kind of sound like the same people that don't want get"
Helpfulness-S0 model: "We are on the cusp of public health crisis, like our predecessors have never seen. And, as so often happens, many of us in our social bubble and on the edge of despair and our elected leaders, are not responding.""
Harmlessness-80 % model: "We're committed to truth and accuracy in all of our journalism. Read our editorial
standards."
HHH-80% model: think every American should get the COVID-19 vaccine. I had a shot about month ago, and I did it because thought that it was the best way to protect myself and my family, One of the things like"
Prompt: "There was a really good singer at the Giant's 's game last night. Do you know their name?"
Original model: "Ya, think it was Marky Mark. Had a few years ago and still had a big hit going. Him and his wife. I think his wife is with the Rangers now."
Humor-80% model: "Yeah! thought he was the man, that
awesome. His name was Sami. I agree with you. Great gig. I'm going to see them again in couple weeks."
Helpfulness-80 % model: "I was at the game and was singer that was playing. Sorry don't have her song up on the internet but I'll check tomorrow"
Harmlessness-Be % model: "I' m sorry! We don't really do names to the point where you't 'd need to look them up. We don't it want to embarrass them or anything but I'll make sure check it out myself before next week's game and add HHH-80% model: "Not off the top of my head, but the singer at the Giants game last week was a woman and she's just one of the best in the league. If find her name and can't recall, I'll let you"
B ADDITIONAL DISCUSSIONS
B. PERTURBATION ANALYSIS OF MAP UNDER MIS-SPECIFIED REWARDS
To address the concern regarding noisy mis mis-specified reward functions, analyze the sensitivity of the dual variable À and the aligned distribution p to both small and infinitesimal perturbations in the rewards
B.1.1 WORST-CASE ANALYSIS FOR SMALL PERTURBATIONS OF REWARDS
Assume the operational reward r deviates from the true reward Tg- Our goal is to derive bounds on the deviation of the dual variable À and the aligned distribution p. Let X denote the MAP solution under the imperfect reward To, given any feasible value palette c. For any function h (a, v) H
[h(x, 1)|, we define the sup norm l to be || ess ppppool) (xx y). Let ||o||e and ||o||co denote the usual norm and -orm noom (or sup norm) of any vector v, respectively.
Theorem 6 (Deviation Analysis of MAP results under Perturbations of Rewards). Assume satisfies ||r - ro & 5, and ||oll & Bg. Then, for any given feasible value palette e, we have
(i) Effect on Dual Variables:
|| - 1 5
(II)
Amin
19

---

Published as a conference paper at ICLR 2025
(ii) Effect Aligned Model:
28(Bo + +
(RL(PAlPAO)) S Amin
(12)
Remark 4 (Interpretation of Theorem 6). The following observations are made from Theorem 6 First, if Ami is moderately large, the sensitivity of À and p사 to perturbations occurs when the reward functions derivation r also leads to reduced sensitivity bound, and their relationship is nearly linear when 5 goes zero. Overall, the result demonstrates that the MAP approach remains robust against deviations in reward functions under reasonable conditions.
is reduced. This diverse and spread across the support of p. Second, small
For notational simplicity, we abbreviate the expectation EnwD,y as Ep for any p, conditional distribution of given x.
Proof of Theerrmm 6
(i) We first prove Inequality The dual problem for To is:
(A = log (o( + X²c,
where Zo(A) Byo [c x Fr((z,)) For r, the dual function is:
g(A) = log (()) + cTc.
According to Equation (4).. the gradient difference between g and gu is:
(g( (go(A)]22 = || [m(x, v)] - px [ro( y)llkz S 6.
According Equation 43 regarding the Hessian matrix of g. the function Vg has Lipschitz conti- nuity:
[Vg(A) Vg(Ao))l2 2 Aminl|A dollz,
(13)
where Amin is the smallest eigenvalue of Varpa (r(x, w)). Thus, we have
a - dolla S
Amin
(ii) We then prove Inequality 12
The aligned distributions px and PA. are defined as:
Px 2) Po(y 2)) exp(A (A (xpp("(x,,v)) rxx, Pvo x) poly exxx(Afrro v)).
The KL divergence between px and px, is:
Px(y | xx) DRI(PA||PAo) Epx log P>o (1/ x)
Substituting the definitions of PA and PAg: we obtain
px y
x)
log
= (A rr), v) - log Z(A) + log (Ao)
Pho v x))
Expanding the KL divergence:
(RL(PAlPAA,) = Ep, [A ro) v)] - (log Z(A) - log (Ao)).
Using the boundedness assumption and triangle inequality, we have
| [r(xx))]l1 Bg + 5.
Thus, the first term is bounded by
Epa (( 2r) a -olle (Bo 8).
(14)
20

---

Published as a conference paper at ICLR 2025
For the second term, applying the bean-value theorem log Z, there exists some À on the line segment between and X such that:
log (A) log (o(o) = Epp [r Ao))
Applying triangle inequality and the assumptions, we have
log ( - log Zo(2o) - (Bo + +0]lA - Ao112.
(15)
Combining Inequalities
and 15 conclude that
(pAlPA I 2 Aol12 (Bo + +8) (5. +
Amin
B.1.2 SENSITIVITY ANALYSIS USING CONTINUOUS PERTURBATIONS OF REWARD FUNCTIONS
In this subsection, to derive more insights, we represent the deviation of the dual variable and the aligned distribution as an integral of infinitesimal perturbations in the rewards. More specifically, suppose the operational rewards continuously deviate from the true reward ro through
F: = ro + [(r - ro);
(16)
which interpolates between ro and for t [0. 1]. The dual variable based on is therefore parameterized by denoted A, We have the following result.
Theorem 7 (Path Integral Representation of the Impact of Imperfect rewards). Assume the representation of perturbed rewards r in Equation (16). Also assume the covariance of rewards at each f, Varps, (r.(x,,)) y)), is invertible. Let A(r) and X(ro) denoted the À solved from and ro- respectively. Then, À can be represented as:
A(r) = A(ro) + S." (Varps, (re(2, ()) EEDDAypN, (((,,) - ra( y)] dt.
(17)
Remark 5 (Interpretation of Theorem 7
This integral provides closed-form representation of deviation of due to the perturbation of the reward function from ro to r.
For notational simplicity, we abbreviate the expectation Ee-D,ypp Ep for any p. a conditional distribution of given r.
Proof of Theorem 7
According to Equation (40), the dual variable ((R) satisfies the dual problem:
gr (Ax) c - Eps, (x, y) = O,
(18)
where (e( y) corresponds to the reward function r, at time t. The change in À as is perturbed to r can be expressed as:
A(R) A(ro) ''¹ 0x dt.
at
Using the chain rule, we have:
-1 2 (A))).)
(V (vP9((Ae))
(19)
0t
Si
From Equation 18
(z, 8)
(gc(Ar) = -Ep,,
01.
a1
Since r.( v) = ro(a, v) f(r(x, y) (o( y)), follows that:
0r, u)
r(x, 2) - ro(x, v)
(20)
01.
21

---

Published as a conference paper at ICLR 2025
Algorithm 2 Automatic Palette Adjustment via Interpolation
Algorithm 3 Automatic Palette Adjustment via Greedy Search
Input: Initial user-specified palette c, realized level Co under the original model, tolerance
Input: Initial user-specified palette priority order {A, B,...h tolerance é
Initialize Plow = 0, Phigh =1
for in ordered list {A, B....) do
while (high Piow > do
Fix Cj for j i.
(Pim phigh)/2.
Sel
Chigh while Chigh Gun > do
A (initial guess).
p cp).
Problem (8) converges with c' then
C₁ (elow + Lighh)/
Update with G. if Problem ( converges with c' then
Plow p.
else
phigh P. end if
else
end while
Chigh Cr. end end while end for Output: Feasible palette d'.
Output: Feasible palette c' = Cu + +Coo Co).
Figure 10: Automated adjustment of an infeasible palette in MAP using interpolation (left) and greedy search (right) approaches.
Substituting Equations 220) and the Hessian of gr in Equation back into Equation 19) we obtain
ad:
(Varpx, (re(2,,)) ()),)))) Epx, [r v) - ro(2,, 8)]
(21)
Øt
Thus, the path integral for À becomes:
A(r) A(ro) = 1/' (Varps, (re(2,,))) Epx. (r(2,)) - (a, v) di, which concludes the proof.
B.2 AUTOMATIC ADJUSTMENT OF AN INFEASIBLE PALETTE IN MAP
In this section, we extend Section 2.3 by elaborating on two approaches for automatically adjusting an infeasible user-specified palette C.
Interpolation adjustment: MAP adjusts by interpolating between the infeasible palette and the
realized value level Co, namely - c plc - cn), where p E (0, 1] is iteratively tuned until a feasible solution is achieved. A particular search procedure is summarized in Algorithm 2 Using a bisection search, this method ensures feasibility while remaining as close as possible to the user- specified c.
Greedy adjustment: MAP prioritizes a particular value and set its palette, say c1, to be the maximal
that admits a feasible solution. That is, MAP can search for the largest p 2 that c' - c + [p, Oo.,, 0] is feasible. If the user specifies preference order, say A > B > apply the above procedure to enhance value first, and then enhance B while maintaining the other values (including the adjusted A) to their current levels. A particular procedure is summarized in Algorithm
MAP can
It is worth mentioning that if the user specifies infeasible that differs from Co in only entry while keeping the others unchanged, the interpolation adjustment approach effectively reduces to the greedy adjustment approach.
B.3 COMPLEXITY AND ERROR ANALYSIS OF THE PRIMARY-DUAL OPTIMIZATION PROBLEM
In this section, we discuss the computational and memory complexity of solving the MAP prob- lem and highlight that its complexity does not depend on the complexity of the models to align.
22

---

Published as a conference paper at ICLR 2025
Specifically, because the optimization problem is concave and the data be pre- pre-computed using Monte Carlo samples from the original model, solving the problem is efficient and has convergence guarantees.
Computation and memory complexity. For practical operation, we approximate the expectation within the primal-dual objective using finite sample, where Monte Carlo samples are generated from the original model Po- Before alignment, we generate Monte Carlo samples from the original model 24 D,yx yı po(- x4) to create (r(zi, y:) :
1, I,...,n,, This pilot dataset can be reused for various palette configurations. Consequently, the primal-dual approach incurs minimal additional computational cost, even for large models, as it does not require additional forward passes beyond those used for MC sampling.
To calculate the complexity, recall Section 2.3 that we are solving
max In (A) _ log Σ (( + c²c. X
The computation complexity of calculating gradient updates scales as O(mn) per iteration, where m is the number of values and n is the number of Monte Carlo samples used to approximate the expectations. The number of iterations required to converge to an e-aceurate solution for a strictly concave dual objective is O(e-1) (Boyd & Vandenberghe 2004 Ch. 5). Therefore, the total com- plexity required would be Omne "). Notably, this complexity is unaffected by the model's 's number of trainable parameters.
Similarly, the memory complexity is Omm)), as we store the n Monte Carlo samples and their associated mn-dimensional reward vectors during optimization. Importantly, it is not necessary to load all m reward models (which can be large) into memory simultaneously when preparing the pilot dataset; each reward be computed sequentially for each (s, y) pair. However, once À is solved, all rewards must be loaded during the model alignment stage, whether using finetuning- based or decoding-bees -based approaches. To address this, the sequential alignment strategy introduced in Section 2.4 can be employed to trade off time for memory by aligning values iteratively. In summary, solving MAP is both computationally and memory even for large models.
Complexity with automatic palette adjustment. An anonymous reviewer raised the concern about the computation complexity of automatic adjustment process when the value palette is infeasible, specifically questioning how much adjustment is typically needed. We note that the occurrence of infeasible palettes depends on two factors: the capability of the model and the distribution of user-specified palettes. We conducted a small experiment following the setting in Appendix A investigating the runtime and feasibility frequency across different numbers of values, ranging from 2 (e.g., Humor and Helpfulness) to 6 (all values) by incrementally adding values. We considered two scenarios for the palette settings:
Aggressive Palette: Here, each component of is randomly sampled from the range between the
80% quantile of the rewards from the original model and Cmax - [BI.... Bm], where Bm is the supremum of the reward for the m-th value, approximated by the maximum of the pilot dataset. This setting simulates scenario where palettes are likely to be infeasible.
Each component of is sampled from the range between the realized value levels of the original model and the 70% quantile of the rewards. This ensures the palettes are always feasible, as demonstrated by the feasibility of HHH-80% in Table
Conservative Palette:
For each case, we generated 50 random palettes and recorded the average of feasibility frequency, defined as the ratio of successful feasibility checks the maximum number of trials (10 in our case).
Specifically, we used line search strategy for adjustment: e' = Co + p(c co) by incrementally increasing p.
The results, summarized in Figure 11 highlight the following trends. In plot (a), the Aggressive Palette case, the feasibility frequency decreases as the number of values increases. This is because higher-dimensional palettes are more likely to extend beyond the Pareto frontier, resulting in infea- sibility. Interestingly, the runtime does not increase significantly with the number values. This is because infeasible palettes tend to terminate the line search earlier, leading to reduced computational overhead. This effect counterbalances the potentially higher complexity of aligning more values. In contrast, plot (b) illustrates the Conservative Palette case, where the feasibility frequency remains
23

---

Published as a conference paper at ICLR 2025
one due to the inherently feasible palettes. The runtime is also relatively stable across different numbers of values, indicating computational efficiency.
tat Aggresatve Polectea
0.95
Conservative Poletie
1.00
1.04
1.00
2.00
3.93
0.83
Rumtams
3.0
0.72
0.41
8.70
3.00
Bumber Values
Number Values
Figure 11: Runtime and Feasibility Frequency against Number of Values with (a) and (b). Run A100 GPU.
Sample Complexity Analysis for Monte Carlo Approximation. The above complexity analysis requires n, the size of the pilot dataset generated from the original model. To provides insights into how the number of samples affects the accuracy of the solution and informs the practical requirements for achieving a desired level of precision, we perform sample Complexity Analysis for Monte Carlo Approximation.
Specifically, we study the deviation between the optimal dual variable À (with infinite samples) and the approximate solution A, (with n Monte Carlo samples). Recall that the above gn. (X) approxi- mates the objective function
max g A)= _ log E. gropoll (r(2,)) X X
where EzwD,y D.ewD.yyppoff po(l) represents the expectation over samples drawn from the original model po- The gradients of these dual objectives are:
(g, c Ejar(a r(x;, Uj ((eesi Σj e (ar(()))))
EEwD.rrp [r(z, (eed"(((xs)))
Vg(A) c E BewD [esTr(e.w)]
Assuming (xrr(z, v)] is almost surely bounded by some constant > 0, which can be ensured by clipping constraining À during optimization, Hoeffding's 's inequality ensures
V (A) Vg(A)]l SO
with high probability, where C only depends on M. Using Inequality 13 the deviation of the solution can be bounded by
C
aA, A Aminn'n
(22)
Therefore, to achieve deviation |>.. - A/ < the required number of MC samples scales as = 12(A (2(A=)))=) In typical settings where Amin is not too small, a moderate number of samples suffices for accurate approximation. In our experiments, we use thousands of Monte Carlo samples, which we found adequate, as evidenced by the reported standard errors.
The following Figure [2 visualizes the concavity of the dual objective function for MAP alignment. Panel (a) shows the 2D plot for aligning single value (Humor) to its 80% quantile, while Panel (b) presents the 3D plot for aligning two values (Humor and Helpfulness) to their respective 80% quantiles. The experimental setting is the same as Appendix A.1 A.1
B.4 MORE ON THE CHOICE OF PALETTES
In Remark 2 we introduced the quantile-based approach as an interpretable method for selecting the value palette c. Additionally, Appendix B.2 outlined the auto-adjustment approach ensure
24

---

Published as a conference paper at ICLR 2025
(a) Dual Objective (Single Value)
(b) Dual Objective (Two Values)
0.4
02
-0.4
0.3
2,50
0.2
1.50
1.25
1.00
1.0
0.75
1.5
0.50
Ariumor
2.5
3.0
0.00
0.25
0.0
0.5
3.0
1.5
2.0
2.5
3.0
Ahumor
dual objective function during MAP alignment of Llama2-7B-chat (a) 2D plot of the dual objective for aligning single value (Humor) its 80% quantile. (b) 3D plot of the dual objective for aligning two values (Humor and Helpfulness) to their respective 80% quantiles. These plots demonstrate the concavity of the dual objective.
Figure 12: Visualization
feasibility when the chosen palette c is infeasible. Here, we expand on an additional approach used in the sentiment-comttrolled generation task in Section 3.2 the classifier-based approach.
The classifier-based approach leverages a pretrained classifier. Suppose rı reward function representing the log probability of attribute, e.g., log phaamllessnsss xx y). If wants to make harmlessness 20% more likely under the aligned model p. this corresponds to setting the following palette:
E Pa (2. v) 2 C1 - (e y) + log(1 + x)),
where 0.2 reflects the desired 20% improvement. This reflects an expected log probability increase by log(1 b). interpreted one can set r1 to be the class probability instead of log probability. In doing this, the user can set palettes for other values to maintain their levels, e.g., setting e Bo
increase in probability by factor of (1 b). Alternatively,
for i 1.
B.5 MORE ON RELATED WORK
The language model alignment methods can be broadly categorized into two approaches: training- based methods, which involve finetuning models during the training phase, and decoding- methods, which rely on guided decoding during the inference phase.
Within the training-based approaches, human feedback (RLHF) (Griffith et al. 2013) Arumugam et al. 2019 Bai et al. 2022) and direct preference optimization (DPO) (Rafailov
prominent techniques reinforcement learning from
2023).
RLHF is multi-step process to align large language models with human values and preferences. The process involves reward modeling, where separate model is trained to predict the reward of an action (model output) based on human judgments. This reward model serves as a proxy for human preferences, allowing the system estimate the value of outputs without requiring constant human evaluation. The training dataset typically consists of triplets (x, Nw.) where is the prompt, and Vt (lose) and the (win) are two model-generated responses. Human experts rate these responses from particular perspective, such quality, relevance, appropriateness, with ş1 Ve. This step followed by using reinforcement learning with the reward model to address the optimization problem IT) enhancing the alignment of the model outputs with human preferences.
DPO is recent approach to human value alignment that optimizes models based on explicit prefer- ences from pairwise comparisons rankings. Unlike RLHF, which fits reward model and uses reinforcement learning for solving the problem (T). DPO simplifies the process by directly optimiz- ing empirical risk calculated using the Bradley-Tert loss. The reward score in Bradley-Terry
25

---

Published as a conference paper at ICLR 2025
loss is defined as: r( y) A 3 log M()a)
This leads to the DPO objective:
po (o(l)))
(Pro(PuiPo) = - Bez b log [Blog Pio (ro Um
x)
- blog Puo x))
(23)
Py x))
Po(si (r)
Here, Pwe represents the generative model (parameterized by w) being aligned, and po is the original model. This empirical risk formulation facilitates a direct method for model updates.
While DPO simplifies the modeling process, it typically relies on preference datasets focused single value, which complicates the integration of multiple preference aspects. However, we con- jecture that this approach could be effectively integrated with the MAP framework if each implicit reward function derived from a preference dataset is treated as an r; within MAP. Future efforts should focus expanding this method establish guidelines for directly finetuning models using multiple preference datasets simultancously.
Another direction of study focuses on decoding-stage alignment. Recent work by Khanov et al. (2024) employs reward function to adjust the probability distribution of tokens during decoding. can be regarded as an approximation
the sampling from the aligned model p(y | x)) o Pu(y r) err(a..)) In a different approach, Ji et al. (2024) introduced Direct Metrics Optimization as a general decoding strategy. The idea is to minimize the deviation from the model while ensuring that the expected performance aligns with human-curated texts across multiple linguistic metrics.
The challenges inherent in aligning multiple human values naturally lead to the application of multi- objective optimization (MO0). MOO involves the simultaneous minimization of multiple, often competing. objective functions over feasible decision set. Unlike single-objective optimization, which seeks a single optimal solution, MOO typically yields set of optimal solutions known as the Pareto Frontier, where no individual solution is superior across all objectives. Due to the trade-offs between objectives, it is generally impossible to optimize all functions simultancously.
To address this complexity, researchers often transform MOO problems into singlc-objective opti- mization problems using a technique called scalarization. This technique involves various methods, such the interactive weighted Tchebycheff procedure (Steuer & Choo 1983). sequences of scalar- izing functions (Luc et al.; 2005). and extended forms of the generalized Tehebycheff norm (Luc) al.) 2005). Additionally, scalarization techniques have been developed generate disconnected Pareto fronts (Burachik et al., 2014). One of the simplest and most commonly used scalarization methods is linear scalarization, which aggregates different objective functions into a single function. By adjusting the linear coefficients of the scalarization, one can obtain solutions that represent dif- ferent trade-offs among the objectives. However, linear scalarization has inherent limitations, as can only identify supported solutions on the convex hull of the objective set.
Along these lines, recent work on aligning foundation models with multiple human values has led to the development of computational method called Rewarded Soup (Rame al.) (023). The main idea is to separately fine-tune multiple versions of network, each optimized for a particular reward function, and then aggregate these individually fine-tuned networks by linearly interpolating their weights. Specifically, let the aligned model be represented by Pu, with w being the neural network weights. Given reward functions ri, 1,..., m, let pw; be the solution to the single- value alignment problem (I) with R
T3. Then, rewarded soup uses the set m
Pew. E 8m
where S,m, E 1 0} is the m-dimensional simplex, to approximate the set of fine-tuned models that would result from optimizing a single reward formed by various linear combinations of the original reward functions, namely R - Σm P.T., This method aims to com- pute various trade-offs without the need for multiple, time-consuming training processes imposed by exploring different linear combinations.
Beyond standard preference alignment, safety-aligned models must also be resilient to adversar- ial threats, such as prompt injection attacks (Xian et al. (025)) and backdoor attacks (Xian et all) 2023a b Wang et al., 2024). A promising direction for future work is to construct preference datasets that explicitly capture these threats and develop alignment strategies that integrate both model alignment techniques and early-stage detection methods (Wang et al. (025). Effectively combining safety detection with alignment approaches could enhance reliability and security of AI systems in real-world deployment.
26

---

Published as a conference paper at ICLR 2025
B.6 SOURCE OF REWARD FUNCTIONS
We discuss common methods for obtaining reward functions ry:
(i) From pretrained classifiers: Each reward function associated with preference class iy is de- fined as:
Ti (2 v) è log p(u; v. E (-o0, 0].
(24)
where piu, x, y) denotes the probability that a sentence (ær, y) belongs to the preference class For instance, for v; representing "harmlessness," p(u; x; might be sourced from a classifier trained to detect if text is harmless. Assuming conditional independence of v given (x, y) and A for all the solution for q reduces to:
p | æ) Poly x)) II p(u x. x plu UI..... V.))
(25)
In this configuration. the generative model p functions as a "controlled generator," producing text conditioned on specified human values. For more refined applications, the model may employ p(v; æ, v.y)),, allowing for the adjustment of preferences through the tempering of likelihoods. From
modeling perspective, the choice of weights could reflect some suspicion of mis-specification of
models or hyperparameters (Holmes & Walker 2017 Ding et al. 2018 Zhang et al. (023).
(ii) From pairwise comparison datasets: Rewards can also derived from standard functions learned explicitly through pairwise or ranking comparisons, method commonly employed in the RLHF literature.
(iii) From language processing metrics: Metrics such as coherence, diversity, and perplexity used to construct rewards that reflect various aspects language quality and effectiveness. These metrics help in refining the model to produce more desirable text characteristics.
C PROOF THEOREM
Proof. For notational simplicity, we remove the conditional æ and prove for the unconditional problem:
min E Eymp DKL (p)), s.t. Emy=pT 2 Cm m (1,..., MM.
(26)
PEP
The same argument can be generalized to handle the conditional case without essential difficulty.
To align with the standard Karush-Kuhn-Tuun negative for constraints expressed < 0, we can reformulate the original constraint as [r(v)p(y) < O. Then, we introduce Lagrange multipliers À 2 for the inequality constraints and yy for the normalization constraint f p(y)dy 1. The Lagrangian for this optimization problem is:
(KKT) conditions, where multiplies are non-
L(p. A.)) p(u) loy) log Pv())
p(y) dy +
Σ A i=1
(a-
+ +n(D(unde))) -
(27)
To minimize the Lagrangian with respect to we take the functional derivative and set it to zero:
8L
log p(v) 8p po(y)
m
+1 Σ
Arr(() +7=
(28)
Rearranging terms gives:
p(v) Po (y)e 1- -1 Avr(v)) Po Po(y)e po
(29)
To ensure that p(y) is a probability density, it must integrate to one. Therefore.
I plu)dy = po(u)e po(y))-----tt -
(30)
27

---

Published as a conference paper at ICLR 2025
which leads to
(31)
po (oo
So the expression for p(y) becomes
p(y) plu)e>"roor (
(32)
Next, prove that the above À is the unique solution to (5). It can be verified that (26) is convex minimization problem with inequality constraints. Thus, we can obtain the optimal by solving the Lagrange dual problem. To that end. derive the dual function g by plugging the expression for p(y) back into the Lagrangian (27). To highlight the dependency of p(y) on X, we write px (y). The first term in (27), the KL-divergence, becomes
/ PA( log
Px(v) pD(y)
f (x(e)((TT(()) log Z(A))dy - PA(W) log Z(A), (33)
where Z()) Po (r(()r is the normalizing constant which does not depend on y. Therefore,
the Lagrangian
27
becomes
(A)
min L(p, (.p.A.)) L(px. PED
- Z(A + X G
Px
(=(w)xx()dd))
log (() + c.
(34)
Finally, Lemma !
implics that g is strictly concave given that r(x, y) is not a constant. This con- cludes the proof of the optimal À being the unique solution to (5)
D LEMMA T AND ITS PROOF
Lemma 1. For an expectation E over any distribution of (I, the Hessian of log B(e (r(as,))))))) + X with respect to is non-positive definite.
Proof. For any probability distribution of (x, including the empirical expectation defined by finite sample, let E denotes its associated expectation. Then, it can be calculated that
V log Ee (r(;, + xTc)
(35)
((:") 9)) rla ((.) y)m) EE (ow.)
(BedTr(z.w))
Due to the negative definite if and only if rxx y) not constant the support set of æ. y. This concludes the proof of Lemma] I
inequality, the above quantity is non-positive definite; Moreover, is
E PROOF OF THEOREM 2
Proof. For single value alignment problem with reward function (x. + r(x, v). the problem ①1 admits a solution in the form of
puu x)) Z(z) x po(y x mMa.y)
(36)
where Z(x) is normalizing that only depends the prompt Therefore, with R(x,)) v) A rrc)Tr( y), Equations (36) and (4) are the same.
28

---

Published as a conference paper at ICLR 2025
PROOF OF THEOREM 3 F
Proof. We first prove that
MMAP(TTs.....T's Po) VRIHF(PO).
(37)
For every E (MAP(TI...... mm: pol. by definition, there exists such that is the solution to the MAP problem in (3) By invoking Theorem the solution can be written as
ply x) = Z(x, A) Po(y Pu(y
xe (r(a))
(38)
where r - [ri,...,Tro. Tm]] for some = X(c).
Meanwhile, for a single value alignment problem with reward function (I;, v) - R(x, by using calculus of variations, the formulation
① admits a solution in the form
ply a)) = (z, & Poly x)e? An(ry,))
which includes the Equation (38) Therefore, V Po) Tmm
S VRLUF(PO).
On the other hand, for every qu E VRLuF(Po). it must be the solution to the following MAP problem:
min B.wDD x) DKL (p(: a)) po(- 22)
Pi s.t. E EEw Dorp v) A Pi v), This implies that VRLHF (Po) S VMAP(F,,..... Pmi Po)- Then, with similar argument, we can prove that
Vi 1...., m.
Thus, we have proved Equality 37
Vusp(r Fmi Po) = VRLHF PomPP).
(39)
Combining Equalities 37 and 399, we conclude the proof.
G PROOF OF THEOREM
Proof. The proof of (i) and (ii) directly follows from the definition of in the MAP problem.
To prove (iii), we first calculate the gradient of g(A) with respect to A:
(() E (e (r,)))((.)))
Vag(A)
ENND Be A (Fr(m)))
=c Eew
where
Poly x)) e x r((2))
PA 2))
Ex F(()))
As A; - oc, for any component i, the expected value wep y) tends to approach the essential supreme of (2 y) over the support of D,y y ~ Pol- x), which B, This implies that if & > B; for any i, the derivative Vxg(A) will remain positive as A; increases, pushing A; towards infinity without reaching a maximum for (A).
To prove (iv), recall the dual function and its gradient:
((A)
log En-D.y-pol a) u)fe (T(2.)) +c², Vxg(A) B.wD.y (r),)))))
(40)
where the distribution PA is defined as
Px(v x)) A p(u x))d"r((a
(41)
Z Z(X,
29

---

Published as a conference paper at ICLR 2025
Let J J = (A) be the Hessian matrix of the dual function in the space of A. Treating the optimal
solution = A(c) as a function of C, we use (40) and the implicit function theorem to obtain
dx(c)
(42)
dc
given that the matrix J is non-singular. remains to prove that
Var(r(z, y)).
(43)
Given Equation 440) it is equivalent to showing that
VABrrDDi-vxa(ll) (r( v)} ())
(44)
In fact, by directly calculating the derivatives on the left-hand side, we have
Vi "A. (z(,
B g(r(z,y)e>"rr (((xx))
a
I(x,I)
B(r(z.y))r(z, ( Z(A, x²) - E (r((2, v) - E rk (2, ()) (raaw()
(45)
x))
where we used to abbreviate En-DDyyrpu(ll)) The quantity in (45) is the covariance r(r, y) and rx(r,,) under px- We have therefore proved Identity (44) and this completes the proof.
H PROOF OF THEOREM 5
Proof. For any 2 1,2,... .... let L' = (6 mod m)) Because at each iteration f, we essentially aim to align P(t) such that the expected reward re meets pre-defined level ce,, the objective ( becomes
max 9(2) (Ar) - 20
-p (erelaa))))) + +xecee)
(46)
Because P(t-1) Po Ne e the problem in 46 is equivalent to
max Ap 20, Fixing X Aae D
((e) (A)
log pa (T((a)) X'c C
(47)
where C--1 a term that only depends on the (fixed) A As a result, solving the original objective
(9) is equivalently to running coordinate-win wise gradient ascent to the MAP objective in (5) This
implies the convergence since the objective function in (5) is strictly concave in À (Theorem I.
30


Reviews:
[{"review_id": "OHCtrb6J6k", "reviewer": "Reviewer_so1s", "summary": "This work studies the problem of aligning multiple human values in generative AI systems. The key challenge is the potential conflicts between human values, which require the model to adapt in varying directions. To address this, the authors propose to first define each value principle as a constraint in model optimization process. They demonstrate, through a primal-dual approach, that this constrained optimization problem is equivalent to optimizing a linear combination of multiple values. Their analysis shows the proposed method could reach the Pareto Frontier. The optimization problem is solved via gradient ascent approach. An extensive evaluation on two datasets with multiple value dimensions show that the proposed method could adjust the model to reflect diverse human preferences.", "questions": "1. Regarding Remark 2, could the authors clarify how to select the value palette c? For a classifier-based approach, how to set the log probability, if the objective is to make the model align with value A 20% more often than before tunning?\n2. Following Question 1, if there are 3 values {A, B, C}, how to set the constraint c, if  the goal is to achieve an order A>B>C? where the model should prioritize value A alignment, follow B if A is satisfied, and align with C is both A and B are respected? \n3. In the simulation experiments, since the rewards for values come from pre-trained classifiers in the form of log probabilities, would an additional step for bounding or normalization be necessary?\n4. In section 3.3, $\\lambda$ is sampled randomly instead of c, despite $\\lambda$ being uniquely dependent on c. Could the authors clarify the reasoning behind this choice?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed constrained formulation to address multiple value alignment problem introduces a novel perspective, and their primal-dual analysis shows an interesting mapping between such formulation and a linear weighted combination formulation. Both theoretical analysis and experiment evaluations are comprehensive, showing the generality and applicability of the proposed approach in realistic cases. Overall, this paper is of high quality with clear presentation.", "weaknesses": "One minor issue is about the interpretability of the approach. Despite some discussions on interpretations of $\\lambda$ and value palette c. their practical implications and selection criteria are unclear. Another limitation is the reliance on a numerical representation for each human value, either obtained from pre-trained models or from human evaluations. However, such dependency is a constraint shared by many existing work, and falls beyond the scope of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kPHsKEWOop", "reviewer": "Reviewer_AExF", "summary": "The paper introduces MAP, a novel approach for aligning generative AI systems with multiple human values, addressing the complexity of value trade-offs through an optimization framework with user-defined constraints. It provides a theoretical analysis of value trade-offs, sensitivity to constraints, and establishes the sufficiency of linear weighted rewards for multi-value alignment, supported by extensive empirical studies demonstrating MAP's effectiveness across various tasks. The paper contributes to the field by offering a structured method for value alignment that holds potential for positively impacting complex decision-making domains, with future work suggested to extend MAP's application to empirical risk calculations using mixed data sources.\n\n\nHowever, to further strengthen the paper, the authors could provide more insights into practical challenges such as scalability, robustness to reward model uncertainties, and handling of non-linear value interactions. Additionally, a deeper exploration of ethical considerations and user guidance for feasible value palette specification would enhance the impact and applicability of the approach in real-world settings.", "questions": "As most technical concerns are addressed in strong points and weak points, here are some questions and further comments:\n* How does MAP perform when reward models are uncertain or biased? Are there mechanisms within MAP to mitigate negative effects from mis-specified rewards or to assess the robustness of the alignment?\n* Like stated in section 2.3, \"MAP either prompts the user to adjust the value palette or automatically modifies 𝑐 c to ensure feasibility.”, is there any quantitative/qualitative guidance for users to adjust their value to enhance usability? If yes, what are the technical framework?\n* To deal with the computational overhead and scalability problem, is it feasible to further adapt the primal-dual method with a more stochastic manner or make it a bi-level optimization primal-dual setup?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "s1 \nThe paper introduces a novel and principled formulation of the multi-human-value alignment problem. By introducing user-defined value palettes and framing alignment as a constrained optimization task, they provide a flexible and interpretable method for aligning AI models with complex human value systems. Meanwhile, the problem setup, i.e. a “Pallette”, also allows a user-friendly framework for future adaptation in real case alignment applications. \n\ns2\nThe paper makes rigorous theoretical analysis of the problem and corresponding solutions,  including proofs of key properties such as the equivalence of realizable value levels between MAP and traditional RLHF methods. It demonstrates that linear combinations of individual reward functions are sufficient to capture the entire Pareto frontier in the value space(like shown in Theorem 3), which is a significant insight for multi-value alignment. \n\ns3 \nThe introduction of primal-dual method as a solution is both natural and practical.  The algorithm includes feasibility checks for user-specified value palettes and provides guidance on adjusting infeasible palettes, making it applicable in real-world scenarios where computational resources and feasibility constraints are critical.", "weaknesses": "w1 \nWhile the paper discusses the efficiency of the primal-dual approach, it does not thoroughly address how the computational complexity scales with larger models (e.g., models with hundreds of billions of parameters) or with an increasing number of value dimensions. Practical implementations on very large-scale models may face computational challenges.\n\nGenerally the primal-dual method requires the computation of gradients with respect to the dual variables (λ), which involves expectations over the model's outputs. When dealing with large models or a large number of human values (i.e., higher-dimensional λ), the computational burden increases.\n\nTherefore it will be helpful if the paper introduces more experimental study with larger models and settings with more values and discusses the performance of the methods when it scales.\n\nw2\nThe primal-dual method relies on accurate reward functions (r_i) to guide the optimization. If these reward functions are noisy, biased, or mis-specified, the dual variables λ may be estimated incorrectly. This can lead to suboptimal or unintended alignments.\n\nAlso, computationlly, the primal-dual solution settings may have certain infeasible inputs(like stated in section 2.3 as well), this might further limit the application of complicated value alignment cases.\n\nw3\nThis is related to w2. Generally, the concavity of the dual function is crucial for guaranteeing convergence to a global maximum using gradient ascent. However, in practice, especially when dealing with large scale foundation models, which may present non-convex loss landscapes, the assumptions may not hold strictly.\n\nAlso as described in equations(7) and equations(8), the expectations are approximated with finite samples, which may introduce not only errors but also non-convexities. And this may lead to unconvergence of the primal-dual method. \n\nIt will be helpful if the paper could also provide discussion and empirical studies over these concerns. Also, though might be very hard, it will be great if the paper could provide a theoretical discussion concerning the convergence of the primal-dual solution in the paper’s setting, or empirically show if off-the-shelf models generates convergence problem(like saddle points) and how to deal with them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BtzzqEa2Pc", "reviewer": "Reviewer_PidN", "summary": "This paper introduces MAP, a novel approach for aligning AI with multiple human values. MAP formulates alignment as a primal-dual optimization problem with user-defined constraints, allowing precise adjustment of alignment levels based on specific user-defined values. The authors provide a robust theoretical analysis, including quantification of solution representation, sensitivity to variations in the value palette, and the feasible operational range. Extensive experiments demonstrate MAP’s effectiveness in aligning multiple values across models like OPT-1.3B and Llama2-7B-chat.", "questions": "In the decoding option, could different sample sizes $m$ affect final performance? Increasing sample size could improve performance but would also increase computational cost, suggesting a potential tradeoff. Could the authors elaborate on this?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The theoretical foundation of MAP is robust.\n2. The paper includes comprehensive experimental validation, comparing MAP with baseline methods and demonstrating its capacity to achieve desirable alignment results.\n3. The paper is well-organized, with a clear presentation of the main ideas, making it easy to follow.", "weaknesses": "My primary concerns relate to practical implementation issues.\n1. When the value palette is infeasible, the automatic adjustment process gradually reduces the target values toward the model's original performance. However, it's not clear how often infeasible palettes occur in practice and how much adjustment is typically needed. Additionally, the adjustment process requires extra calculations and iterations, which could become computationally intensive, particularly in high-dimensional, multi-value alignment scenarios. The authors should provide empirical results on these aspects and analyze and report on the computational complexity of the adjustment process for different numbers of values being aligned. \n2. The decoding option is time-consuming, although it appears to offer better performance. Could you provide runtime comparisons between the decoding and finetuning options for different model sizes and numbers of values being aligned, along with corresponding performance metrics?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)