PAD: Personalized Alignment of LLMs at Decoding-time
Authors: Ruizhe Chen
Xiaotian Zhang
Meng Luo
Wenhao Chai
Zuozhu Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Alignment
personalization
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Aligning with personalized preferences, which vary significantly across cultural, educational, and political differences, poses a significant challenge due to the computational costs and data demands of traditional alignment methods. In response, this paper presents Personalized Alignment at Decoding-time (PAD), a novel framework designed to align LLM outputs with diverse personalized preferences during the inference phase, eliminating the need for additional training. By introducing a unique personalized reward modeling strategy, this framework decouples the text generation process from personalized preferences, facilitating the generation of generalizable token-level personalized rewards. The PAD algorithm leverages these rewards to guide the decoding process, dynamically tailoring the base model’s predictions to personalized preferences. Extensive experimental results demonstrate that PAD not only outperforms existing training-based alignment methods in terms of aligning with diverse preferences but also shows significant generalizability to preferences unseen during training and scalability across different base models. This work advances the capability of LLMs to meet user needs in real-time applications, presenting a substantial step forward in personalized LLM alignment.

Published as a conference paper at ICLR 2025
PAD: PERSONALIZED ALIGNMENT OF LLMS AT DECODING-TIME
Ruizhe Chen
Xiaotian Zhang 1st
Meng Luo
Wenhao Chai
Zuozhu Liu 1,+
Zhejiang University
National University of Singapore
University of Washington
ABSTRACT
Aligning with personalized preferences, which vary significantly across cultural, educational, and political differences, poses a significant challenge due to the com- putational costs and data demands of traditional alignment methods. In response, this paper presents Personalized Alignment at Decoding-time (PAD), a novel frame- work designed to align L.LM outputs with diverse personalized preferences during the inference phase, eliminating the need for additional training. By introducing unique personalized reward modeling strategy, this framework decouples the text generation process from personalized preferences, facilitating the generation of generalizable token-level personalized rewards. The PAD algorithm leverages these rewards to guide the decoding process, dynamically tailoring the base model's pre- dictions to personalized preferences. Extensive experimental results demonstrate that PAD not only outperforms existing training-based alignment methods in terms of aligning with diverse preferences but also shows significant generalizability to preferences unseen during training and scalability across different base models. This work advances the capability of LLMs to meet user needs in real-time appli- cations, presenting substantial step forward in personalized LLM alignment. Our model and code are available here.
INTRODUCTION
Recent advancements have demonstrated success in aligning language models with human preferences and values (Stiennon et al., 2020; Bai et al., 2022; Ouyang et al., 2022; Achiam et al., 2023). Representative methods such as Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) and DPO (Rafailov et al., 2024b) typically optimize policy model with training signals from an explicit or implicit reward model. The reward model captures the 'general' human preferences values from human feedback. However, in this pluralistic world, users' preferences can diverge significantly based on their different cultures, educational backgrounds, religions, and political stands (Gordon et al., 2022; Sorensen al., 2024a; Jang et al., 2023; Cheng et al., 2023). Furthermore, even for the same person, the preference of a particular LLM response can vary when the application scenario changes. Hence, there always exists proportion of human preferences that cannot be unified by the general preference, also known as personalized preferences, which current alignment frameworks struggle to align with due the need for high-quality datasets and substantial computational costs in policy optimization.
How can we align with personalized preferences without the need for additional data collection and policy training? In this paper, we introduce Personalized Alignment at Decoding-time (PAD), which aims align LLM outputs with diverse personalized preferences during the inference phase without requiring additional training. To achieve this, we first propose personalized reward modeling strategy, which decouples the text generation process (modeled as a Markov Decision Process) from personalized preferences, thereby enabling the acquisition of generalizable token-level personalized rewards. Based on this, we then formulate a personalized reward model (PersRM). During decoding. the PersRM scores the base model's top-K predictions at cach token generation step based on the current generation and personalized preferences. Finally, this score is combined with standard decoding likelihoods to adjust the base model's predictions. The advantages of PAD are as follows: (1)
"Corresponding Author
Equal Contribution

---

Published as a conference paper at ICLR 2025
Table 1: checklist for key characteristics of previous methods and PAD. *-"; Not Applicable.
Method
Training-fre free
General Policy
Single Reward
Generalizability
MORLHF (Li et 2020)
MODPO (Zhou et al., 20223)
Personalized soups (lang al., 2023)
Preference Prompting (Jang al., 2023)
Rewarded soups (Rame al., 2024)
RiC (Yang et al., 2024b)
X
X
X
DPA (Wang et al., 2024a)
X
Args (Khanov et al., 2024)
7
X
X
MOD (Shi et al., 2024)
x
X
X
MetaAligner (Yang et al., 2024a)
PAD (Ours)
It requires only a single policy model (i.e., the base model) aligned with general preferences (General Policy), climinating the need for training additional policy models (Training- -fre)). (2) It utilizes only single reward model (Single Reward). (3) It does not require pre-defined personalized preferences to generalize to preferences not seen during the training phase (Generalizability) A checklist of PAD's advantages over previous methods is presented in Table 1.
Our contributions can be summarized as follows:
We propose novel personalized reward modeling strategy that decouples the dynamics of text generation from personalized preferences. This strategy enables the acquisition generalizable token-level personalized rewards with single personalized reward model (PersRM).
We propose novel personalized alignment decoding-tine (PAD) algorithm that performs guided decoding with the guidance of token-level personalized rewards, while not requiring training additional policy models.
Extensive experiments demonstrate that PAD outperforms existing training- based methods in aligning with diverse personalized preferences. Furthermore, the results highlight PAD's effectiveness in generalizing to unseen preferences and model-agnostic scalability.
2 RELATED WORKS
Large language model alignment. Large language model alignment aims to align LLMs with human preferences. common approach involves using an RLHF (Reinforcement Learning with Human Feedback) framework (Christiano et al., 2017; Bai et al., 2022) where reward model is trained based on human feedback, and Proximal Policy Optimization (PPO) (Schulman et al., 2017) is employed to derive the aligned policy model. Notably, decoding-ting training (Mudgal et al., 2023; Khanov al., 2024; Han
alignment offers an alignment
paradigm that does not require expensive et al., 2024; Liu et al., 2024; Huang et al., 2024). Controlled Decoding (CD) (Mudgal et al., 2023) utilizes a prefix scorer module trained to assess value functions for rewards, allowing controlled generation from frozen base model. ARGS (Khanov et al., 2024) proposed using reward signal to adjust probabilistic predictions, thereby generating semantically aligned texts. DeAL (Huang et al., 2024) focusing on heuristicguuided searches better meet diverse alignment objectives.
Personalized alignment. As users exhibit diverse preferences and values for a single task, is essential to align LLMs to personalized preferences (Kirk et al., 2023; Sorensen et al., 2023; 2024b; Yao al., 2023; Kirk et al., 2024; Zhong et al., 2024; Han et al., 2024). One line of work achieves joint optimization for different personalized preferences by defining a reward function with multiple dimensions and performing policy optimization (Zhou et al., 2023; Wang et al., 2024a;b; Guo et al., 2024; Yang al., 2024b; Chakraborty et al., 2024; Sun et al., 2024; Li et al., 2024). Additionally, some approaches merge model parameters or predictions for each dimension to handle their diverse combinations (Jang et al., 2023; Rame et al., 2024; Park et al., 2024; Shi et al., 2024). Lastly, prompt- based methods align personalized preferences by designing diverse prompts or post-processing techniques (Yang al., 2024a; Lee et al., 2024; Hwang et al., 2023; Jafari et al., 2024).
2

---

Published as a conference paper at ICLR 2025
3 METHOD
3.1 PRELIMINARIES
In this section, we first define the per-token Markov Decision Process (MDP) for large language models (LLMs) and describe its relationship to classic Reinforcement Learning from Human Feedback (RLHF) approaches. Then, we describe the characteristics and challenges of personalized alignment.
Text generation as token-level markov decision process. The standard text generation process of large language models (LL.Ms) with prompt and response y can defined as a token-level Markov Decision Process (MDP). MDP is denoted as tuple M = (S, A, P, R, T), where the state space S consists of the prompt and all tokens generated so far (i.c., S, = (x, Y1a-1)), The action space A is the tokens from the vocabulary (i.e., a, yr). P is the transition kernel, which is deterministic that given state S = (x.Y1:-1) and action a = Yt. the next state is 8141 = (st, a+) (x. y1:e). R : X A - I represents the reward at each step. The maximum token count, T, sets the length limit for LLM outputs, which conclude with an end-of-sentence (EoS) token YT EoS that ends the generation. Given an MDP, the objective maximize the expected return R(x, y) = EL_ (s;, at). To achieve this, the agent computes (Markov) policy π A(A) that maps from state to distribution over actions.
The RLHF Pipeline. Classic RLHF approaches (Ziegler et al., 2019) first learn a reward function from human feedback on prompt and response pairs (x, y", y'). The reward function is modeled under a contextual bandit setting using the Bradley-Terry preference model (Bradley & Terry, 1952):
exp R"x
p'(y" yy))=
exp R(x, y") + exp R(x,y')'
where yd and y denote the preferred and not-preferred completions for the prompt x. p" (y" > y²) denotes the probability that y is preferred to y'. The reward model can be learned through Maximum Likelihood Estimation (MLE) on this dataset D:
CR, D) F (log ar (R(x, y R(x,y:)))
Subsequently, we use the learned reward model to provide feedback. The policy model (i.e., the language model) To is optimized with gradient-based method such as PPO (Schulman al., 2017) with an entropy-bonus using the following KL-constrained RL objective:
Σ (R(x, - 3D ( x), mo ((1)))))
max E To (ylxx
where Tet represents a reference policy, typically the language model resulting from supervised fine-tuning, from which the learned policy should not significantly deviate. 3 is a parameter used to control this deviation. In practice, the language model policy To is initially set to Teet Additionally, is important note that exclude the supervised fine-tuning (SFT) stage in the RLHF pipeline. This is because the SFT stage is not directly relevant to the focus of this paper.
Personalized alignment within MDP Unlike the unidirectional reward in traditional MDPs, posit that personalized preferences may vary across different dimensions; for example, some users may prefer concise and understandable responses, while others might favor comprehensive and expert answers. In this way, the reward function of personalized alignment can be defined as it S x A + R", which describes a vector of n rewards, one for each dimension of personalized preference (e.g., comprse/compeeeeniive,, expert/elementary). instead of scalar. During alignment, a personalized preference may encompass or several dimensions of rewards.
Based on this, existing work (Li et al., 2020; Rame et al., 2024) employs a linear scalarization strategy, denoting human preferences as w such that = wT R. Subsequently. these approaches optimize the policy with RLHF objective perform weighted merging of multiple policies. However, these approaches still cannot overcome the high time and computational costs associated with optimizing the policy (i.e,, the language model) for multiple personalized preferences simaltaneously.
3

---

Published as a conference paper at ICLR 2025
3.2 PERSONALIZED ALIGNMENT AT DECODING-TIME
In this section, we propose Personalized Alignment at Decoding- (PAD), a novel approach that does not require training a policy and is transferable to diverse personalized preferences. Inspired by the concept of successor features (Dayan, 1993; Barreto et al., 2017), we first define the personalized reward function, which is composed of the features of the current state and personalized preferences. By linking this reward function to the value function, we can decouple personalized preferences from the dynamics of the MDP. Consequently, we only need to learn the generic features under the current policy, and by merely altering the personalized preferences, we can achieve personalized alignment. Finally, we introduce guided decoding algorithm, which aligns with personalized preferences by utilizing the value function for weighting during the inference phase.
Definition 1: personalized reward function R can be represented by:
R(p, s, a) = w. Φ(s, a),
where (s,,a) E R. represents the features of current state and action (s, a), and wp E R are weights derived from personalized preference p.
Intuitively, o(s, a) can be understood as the salient features (e.g., expert, informative) of the current state, and the vector wp models personalized preferences p as the degree of desirability for each feature. Wp is independent of (s, a). Consequently, when the user's personalized preferences change, only Wp is altered, which turn modifies the reward function Rt(p s, a). We derive the value function (Watkins & Dayan, 1992) of personalized reward function, inspired by that the optimal policy x² obtained by Equation can be formulated as (Ziebart, 2010; Rafailov et al., 2024a):
x² (a p) = e -V
(5)
where Q. the action-value function (i.e., Q function) based on the leken-level reward R (p. 8, a). models the total future reward from (s, a) under policy V is the corresponding vtate- function at current state, where (T $e) 3 log (.a e (p.s..a)/88 da). Q" and V denote the optimal value functions under optimal policy n² Q function can be expressed as:
(p, = Σ R(p, 8,, a:) x(-ss))]
(6)
EE
(s, a.)aa T(1ss)] w, "0 (s;
where E is the expectation over the randomness due to the sampling from the base language model и. Eq. 7 is derived by substituting Eq. 4 into Eq. 6. v/" gives the expected sum of (ss, a) when following policy и starting from (s, a), which is known as successor features (SFs) that also satisfy a Bellman equation of ((s, (Bellman, 1966; Barreto et al., 2017). Therefore, can be noticed that the vector wp representing personalized preferences is decoupled from the MDP dynamics.
To obtain the Q" function, we begin by rewriting Eq. 5 as Rafailov al. (2024b):
R(p,x,y) Σ Rlp. St, ar) = Σ Blog (a st p)
(s:).
(8)
Thr (r: p)
Denote (als) as the features of (s, a), which satisfies logr p) = w T (als) By substi- tuting this relationship and Eq. into Eq. 2, we can derive the loss function for personalized reward modeling:
T
to (a)""(s))) m (a (st")
T
no (o(a((s)) And (n (st)
(Te,RMM(TO, D) = -E(... log a
w
B log
Σ
log
Inspired by Rafailov et al. (2024b), we can derive the implicit Q function Q" (p, St, ar) with optimized personalized reward model Th:
log (ay(s)) + V V+(pp $1). fner(a, (a, (s:)
Q' St a.) w / (s;, a:) w
(10)
Then, formulate personalized alignment decoding-time Q" guided search according to Fq. 5.
4

---

Published as a conference paper at ICLR 2025
o== (4)
Original Output: layer protocals interfaces, such as HTTP
OlyperTestt Transfer Protocol)...
Instraction List layers the TCP/P model.
toymout
layer
Base Language Model
Logits
Current output y_t. It consists of layers: Application Layer: This
enables
Weighted Scores
(sx) argmax ol øls,, sear(1ba))
Preference p: Generate response that expert and informative.
PAD Output: This togroost layer emables usee-devel
applications to internet with the network. common protocal (llypertett Protocol)...
Personalized Reward Model R
Reward
Figure 1: An illustration of the inference phase for personalized alignment at decoding-tines time (PAD) with optimized personalized reward model (PersRM), Given the personalized preference and the current context, we first calculate the probability distribution of the base model for the next token. Then, we calculate the reward from PersRM combining features of current state and personalized weight. Finally, the next token can be selected based the weighted scores.
Definition 2: Personalized alignment at decoding-tine (PAD): The optimal policy TAAD of person alized alignment can be defined as selecting action for the buse model a aLs that maximizes the advantage function Q" (p, St. a) V* (p, 8t) towards personalized preference p at each step:
PAD (a|st.() p) x (als())))) (0Q (p.s..)- ('(i))))
(11)
where (p, (p, s:) is equivalent to w, log (7 (ax (s:) (ar(s:)) according to Eg. 10.
The detailed derivations are provided in the uppendix D.1. It can be observed that the learned reward function can serve as optimal advantage function. Note that unlike RIHF framework that directly models the language model as policy, our PAD policy TPAD is different from the base language model TLM. as well as the personalized reward model mT.
3.3 GENERALIZED PERSONALIZED ALIGNMENT
In this section, we discuss the ability of PAD to transfer to unseen personalized preferences. Suppose now that we have computed the optimal value functions for n personalized prefer- ences W1, W2...., W. w denoted as {Qi, Q2. Q Now, if the reward changes to R(pa+1, s,a) = ((4) as long as we have Wa+1 we can compute the new value func- tion of ग. by simply making (s, = w W 16* (s, a). Once the functions Q have been computed, we can apply generalized policy improvement (Bellman, 1966; Barreto al., 2017) to estimate its performance
Wx+1-
Theorem 1. Let w; E We and let Q; be the action-value function of optimal policy of w;- For all s E S, a A, and j (1.2....,,l) let (Ts) E arg max, max, Q (s, a). Finally, let Φmax max, (s, a)]. where
the norm induced by the inner product adopted. Then,
Qu+1 - (p. S H (mmax min [| w5ll)
(12)
This term a multiple of the distance between Wx+1 and the closest w, for which we have already computed policy. The formula formalizes the intuition that if PAD has previously aligned to similar personalized preference wj. it should align well on the new preference w,+1 (Barreto et al., 2017). The proofs of Theorem are in the Appendix D.2.

---

Published as a conference paper at ICLR 2025
3.4 PRACTICAL IMPLEMENTATIONS
In this section, we introduce the practical implementation of our Personalized Alignment at Decoding- time (PAD), which includes the optimization of the personalized reward model (PersRM) and the inference-time time guided decoding with token-level personalized reward.
Optimization As previously mentioned, we can decouple personalized preferences from the MDP process during the personalized alignment procedure. Thus, consider utilizing LLM-based model as a personalized reward model (PersRM) ma to independently predict the features Φ(s, a) and preferences Wp- For simplicity, we employ single backbone predict the embeddings for both 6(s, a) and wo is initialized using the backbone, also referred to as the reference model The-- Subsequently, it employs an additional value head to predict Wp. Optimization occurs in two stages with Equation 9. In the first stage, we fix wp as a unit vector and optimize the backbone to learn general features. In the second stage, we freeze the backbone and only optimize the value head for Wp to learn different user preferences. The optimized PersRM is denoted as Tg.
Guided Decoding The inference phase of PAD is shown in Figure 1. Given the personalized preference p and the current context S, (x, yct) step f, we first calculate the predicted probability distribution of the base model T TLM (a]sz) for each next token candidate a. Then we calculate the (eare((((sss)) for these tokens.
reward from PersRM by R(p, Se; a) w, p ((s,, a) = w log(#; Finally, the next token a, can be selected based on their weighted scores:
A (als)) Fred (als:)
PAD (s. p) = arg max (awTrAr Bw T log
+ log TLM (als:)
(13)
Eq. 13 is equivalent to Eq. 11, with proofs in the Appendix D.I. Note that 3 in Eq. 13 is also treated as weight hyperparameter for simplicity, which slightly differs from its previous definition.
EXPERIMENT
4.1 EXPERIMENT SETUP
4.1.1 PAD SETUP
During the development of our personalized reward model, we utilized datasets from multiple sources including HelpSteer2 (Wang et 2024c), Rewards-in-Context (Yang et al., 2024b), and SafeRLHF (Dai et al., 2023). To simulate personalized preferences, synthetic user prompts are prepended instructions, following Jang et al. (2023). We employ the Llama-3-8B model (AI @ Meta, 2024) as our backbone, and append a linear layer directly following the embeddings, featuring an output dimension of 4096. Comprehensive details the datasets, the pre pro-processing process and the implementations are documented in Appendix A. During the decoding phase, we utilize greedy decoding with top-k candidates. We restrict the maximum lengths of the initial prompt and subsequent generations to 2,048 and 128 tokens, respectively. The hyperparameters, specifically 3 1.0 and 10, are optimized to maximize the average reward performance observed our validation datasets. Analysis of the decoding strategies and hyperparameter settings is provided in Section 4.3.
4.1.2 EVALUATION SETUP
Datasets and base models. We utilize two evaluation datasets. The P-Soups (Jang et al., 2023) evaluation dataset has been filtered and modified based on the Koala evaluation by Jang et al. (2023). The HelpSteer2 (Wang et al., 2024c) (validation split) dataset is multi-aspect alignment dataset comprising 1, 1,000 prompts.
our evaluation setup, we initially focus on alignment the three pre- defined dimensions: 'harmless', 'helpful', and 'humor', following previous works (Yang et al., 2024b;a; Shi et al., 2024). Additionally, to assess the scalability of the Personalized Alignment Dataset (PAD), we simulate users each having unique personalized preferences drawn from several preference dimensions, collectively forming 12 preference combinations. For personalized alignment, primarily utilize 1aima-3-8B-SC (AI @ Meta, 2024; Meng al., 2024) the base language model. Additional experiments are conducted on Gemma (Team, 2024), Mistral-7B-SD (Jiang et al., 2023; Tunstall et al., 2023), and Llama-2 (Touvron et al., 2023) to test scalability.

---

Published as a conference paper at ICLR 2025
Bane (her)
PAR phan))
Helpful
Helpful
Warole Ueroles
Helld
Happol
Heista
Harmies
Harmless
Homar
Homor
6.28
1.49
Harnles Hurmr
AG
6.43
(a) Alignment results on P-Soups dataset.
(b) Alignment results on HelpSteer2 dataset.
Figure 2: Alignment results for pre-defined preferences related harmless, helpful, and humor.
Evaluation metrics. For the evaluation of 'harmless', 'helpful', and 'humor' dimensions, we integrate three open-source reward models available Huggingface, as delineated by Yang et al. (2024b;a). Additionally, we employ the ArmoRM (Wang et al., 2024b), multi-dimension reward model known for its state-of-the-ee -frtt---ttttttt performance the Reward-Bent benchmark (Lambert et al., 2024). For these reward models, we report their scores assessing LLM responses from various per- spectives. Furthermore, our evaluation leverages GPT-4, widely recognized tool in previous studies (Khanov et al., 2024; Yang et al., 2024a), to conduct the judgments towards certain personalized preference and report the win rate. Comprehensive details on the evaluation metries, reward models and GPT-4 judgments are provided in Appendix B.2.
Baselines. We compare PAD with 9 personalized alignment multi-objective alignment methods as delineated Table MORLHF (Li et al., 2020) optimizes for the weighted multi- mujti-objeccive reward function using PPO. MODPO (Zhou al., 2023) integrates language modeling with reward modeling, training models to combine all objectives with specific weights. Personalized soups (Jang et al., 2023) first optimizes multiple policy models and then merges the parameters of the policy models according to preference. Rewarded soup (Rame et al., 2024) first trains multiple specializing networks independently, and then interpolates their weights linearly. Rewards-in-Conees 2024b) conditions foundation model responses on multiple rewards in its prompt and uses supervised fine-tuning for alignment. MetaAligner (Yang et al., 2024a) and Aligner (Ji et al., 2024) train an additional model to perform conditional weak-to-strong correction. Preference prompting (Jang et al., 2023) simply prompts for preferences without any additional training. MOD (Shi et al., 2024) first trains multiple specializing networks and performs linear combination of their predictions.
(RiC) (Yang et al.,
4.2 MAIN RESULTS
Alignment on Pre-defined Preferences We initiate our evaluation by focusing on three pre-defined dimensions seen in the training phase: "harmless', 'helpful', and 'humor', As depicted in Figure each point represents the average rewards corresponding to specific user preferences, encompassing either single dual dimensions, evaluated across two distinct datasets. We dynamically adjust the weights of these dimensions for the baseline models. The results demonstrate that PAD can effectively align with various preferences, outperforming the baselines terms of achieving a superior frontier. Subsequently, compare the performance of PAD with baseline methods in aligning to the three dimensions simultaneously. For baselines, we set uniform preference weights for three dimensions.
The performance of PAD across datasets is presented in Table 2. The findings reveal that PAD has achieved substantial improvements for all three objectives. Within the P-Soups dataset, PAD achieves average win rate of 84%, elevating the average score of the reward model from 0.32
7

---

Published as a conference paper at ICLR 2025
Table Comparison of baseline methods and PAD on predefined preferences. "1-dim": alignment performance on the single dimension. "3-dim": alignment performance on the three dimensions simultaneously. "Pref. Promp.": Preference Prompting. The best result highlighted in bold. The results indicate that PAD outperforms the baselines across most dimensions and metrics on both datasets, demonstrating its superiority in personalized alignment.
Helpful
Harmless
Humor
Overall
Method
Armo
RM
GPT-4
Armo
RM
GPT4
RM
GPT-4
RM
GPT-4
Psoups dataset
Base
0.63
1.06
0.97
0.83
-0.93
0.32
MORLHF
0.31
0.91
14%
0.88
0.84
4%
0.28
82%
0.68
33%
MODPO
0.56
0.89
52%
0.96
0.77
80%
-0.90
72%
0.25
68%
Personalized soups
0.38
-0.72
72%
0.92
0.73
92%
-0.30
80%
-0.09
81%
Rewarded soups
0.50
0.87
34%
0.95
0.87
64%
0.14
78%
0.63
59%
RiC
0.54
0.90
40%
0.97
0.90
-0.08
76%
0.58
62%
Pref. Promp. (I-dim)
0.56
0.82
70%
0.96
0.87
90%
-0.79
74%
0.30
78%
Pref. Promp. (3-dim)
0.54
0.84
70%
0.93
0.98
87%
-1.28
71%
0.18
76%
MetaAligner (1-dim)
0.47
1.75
79%
0.90
0.89
71%
-0.74
81 %
0.21
77%
MetaAligner (3 (3-dim)
0.55
1.39
66%
0.89
0.54
74%
-0.97
74%
0.32
71%
MOD
0.55
0.93
60%
0.96
0.92
84%
0.38
78%
0.74
74%
Aligner
0.67
1.32
72%
0.97
0.63
70%
-1.39
12%
0.19
51%
PAD (1-dim)
0.67
1,31
74%
0.93
1.03
92 %
1.50
88 %
1.28
84%
PAD (3-dim)
0.61
0.96
63%
0.98
0.85
87%
0.75
83%
0.85
78%
HelpSteer dataset
Base
0.56
0.98
0.50
-0.56
0.17
MORLHF
0.49
0.6
52%
0.92
0.54
60%
0.15
62%
0.43
58%
MODPO
0.53
0.38
28%
0.99
0.69
-0.78
52%
0.10
52%
Personalized soups
0.42
-0.53
54%
0.93
0.66
70%
0.14
80%
0.09
68%
Rewarded soups
0.52
0.63
46%
0.90
0.66
60%
0.00
48%
0.43
51%
RiC
0,51
0.66
38%
1.00
0.68
70%
0.01
0.44
59%
Pref. Promp. (1-dim)
0.52
0.52
56%
0.84
0.96
82%
-0.49
86%
0.33
75%
Promp. (3-dim)
0.50
0.33
51%
0.96
0.80
-0.96
0.06
60%
MetaAligner (1 (1-dim)
0.48
1.46
61%
0.91
0.32
85%
-0.77
73%
0.34
73%
MetaAligner (3-dim)
0.51
1.13
58%
0.90
0.23
82%
-0.69
76%
0.22
72%
MOD
0,50
0.66
56%
0.97
0.78
70%
0.24
52%
0.56
59%
Aligner
0.59
0.88
54%
0.99
0.44
90%
-0.77
8%
0.18
51%
PAD (I-dim)
0.65
0.89
62%
0.97
1.11
87%
1.57
92% %
1.19
80%
PAD (3-dim)
0.36
43%
1.00
0.80
0.94
88%
0.70
69%
1.28. Across all reward model scores or win rates, PAD surpasses all baselines in metrics. On the HelpSteer2 evaluation dataset, PAD achieves the best in 7 out of 10 metrics and significantly enhances overall personalized preference alignment performance. These results demonstrate the superiority of PAD in personalized alignment.
Alignment on Customized Preferences In previous experiments, were limited aligning with pre-defined preferences. As previously noted, most existing methods focus on aligning with preferences or dimensions defined during the training phase. However, in real-world personalization scenarios, there still exist diverse unseen personalized preferences, which current methods struggle to address. Considering this aspect, we evaluate the ability of alignment on eustomized preferences in this part. We additionally define three dimensions 'expert', 'informative', and 'creative' that were unseen during the training phase, which result in 8 personalized preferences. We compare the alignment performance of PAD with preference prompting and MetaAligner on the P-soups dataset. The GPT-4 judgment results between the generations of different methods and Llama-3-Ban 3-Base are provided in Figure 3. As be seen, PAD consistently improves win rate across all eight types of personalized preferences, outperforming both baselines. This confirms the superiority of PAD in generalizing unseen personalized preferences.
8

---

Published as a conference paper at ICLR 2025
PAD
Preference Prompting Win
Meta Aligner Win Meta-Aligner Tie
PAD Tle
Preference Prompting Tie
PAD Lose
Preference Promppiing Lose
Meta-Allgne Lose
40
30
10
Exp. Inf. cre
Exp. Con, Cre
Pre, Inf, Cre
pre. Con, Cre
Exp. Inf. Fac
Exp, Con, Fac
Pre. Inf, Fac
pre, Con, Fac
Figure 3: Alignment on Customized Preferences. Abbreviations include "Exp" for Expert, "Inf" for Informative, "Cre" for Creative, "Pre" for Preliminary, "Con" for Concise, and "Fac" for Factual.
4.3 DECODING STRATEGY
Sensitivity Analysis Decoding Hyperparameters In our initial analysis, investigate the impact of adjusting the hyperparameters / and on the performance of reward models within the P-Soups dataset. Additionally, we incorporate diversity score indicates an enhanced capability to generate texts with a broader variety of vocabulary, as detailed in Appendix B.2. The results are illustrated in Figure 5. It is evident that increasing the weighting parameter 3 leads to rise in reward scores. However, beyond a certain threshold, the scores for helpfulness and harmlessness begin to diminish, suggesting that excessively elevated values might lead the generation process to deviate substantially from the base model's 's core knowledge, thereby compromising its intrinsic qualities. Concurrently, as 3 increases, there a slight increase diversity, indicating that higher 3 values encourage the generation of more varied vocabulary. Regarding the number candidates k. the performance depicted in Figure 6 suggests that a larger number of candidates may slightly encourage the generation of more diverse responses. However, it has minimal impact on producing more personalized-aligee responses.
metric. A higher diversity score
Effect of Decoding strategies. We compare the performance with three decoding strategies, which can be integrated with PAD. (1) Greedy: select the token with maximum score. (2) Stochastie: sample token from the prob- ability distribution of the top-k candidate. (3) Best-of-k: generate k responses from the base model and select the one with maximum score. The temperature parameter is set to be 0.7 for stochastic and best-of-k and k is set to 10 for all strategies. Additionally, we compared the performance with (4) Base: greedy generation using only the base model as a reference. The average scores and standard deviations for the reward model and diversity across three runs,
1.4
Greedy
Stochastic
1.0
0.0
Helpful
Hamlees
Humar
Diversity
Figure 4: Comparison various decoding strate- gies that can be integrated with our PAD.
illustrated in Figure 4. For clarity in visualization. humor scores below zero have been clipped. evident that all three decoding strategies enhance alignment. In some dimensions, stochastic and best-of-k strategies achieve better alignment than greedy, demonstrating the effectiveness of

---

Published as a conference paper at ICLR 2025
happi Foemieex Diversiy
Droesity Scgre
14
1.3
12
1.2
0.8
0.
0.7
0.6
Helpful
0.6
0.8
Harmless
0.2
0.7
Humor
Diversity
0.0
20
100
Figure 5: Sensitivity analysis on (3.
Figure 6: Sensitivity analysis k.
Table 3: Analysis of PAD scalability across various base models.
Helpful
Harmless
Humor
Overall
Method
RM
Armo
GPT-4
RM
Armo
GPT-4
RM
GPT-4
RM
GPT-4
Gemma-2b-it
0.98
0.47
0.96
0.68
-0.10
0.61
w/ PAD
1.37
0.50
64%
0.99
0.93
100%
0.53
72%
0.96
79%
Mistral-7b-SFT
1.59
0.58
0.97
0.78
-0.95
0.53
w/ PAD
1.67
0.60
54%
1.00
0.96
92%
0.93
78%
1.20
75%
Alama-2-7b-cha
0.63
0.47
0.91
0.95
1.38
0.97
w/ PAD
0.86
0.48
82%
0.93
1.14
100%
1.08
56%
0.96
79%
Alama-3-8b-it
1.06
0.65
0.99
0.75
1.59
1.21
w/ PAD
1.38
0.68
82%
0.99
1.02
96%
1.97
76%
1.45
85%
exploration. However, their performance variance is high, indicating some instability, Regarding diversity, all strategies show marginal improvements.
Computational Cost for PAD Building on the previous section, we evaluate the computational costs of decoding-time time alignment, with results detailed in Table C2, which is measured a single NVIDIA H100 GPU. The time costs for training-based models closely align with those of the 'Base' configuration. Our results reveal that generation time of PAD increases by 2-3 times compared with conventional greedy decoding. Additionally, we have quantified the memory overhead of PAD. Decoding with PAD requires an additional 17,095 MB. Despite this increase processing time and memory, there is notable enhancement in performance across all dimensions (e.g., 26% increase as for 'helpful'), indicating reasonable tradeoff between alignment performance and inference speed.
4.4 MODEL-AGNOSTIC ANALYSIS
In this section, explore the scalability of PAD broader range of base models. We employ the same personalized reward model, which does not require retraining for different base models. As illustrated in Table 3, PAD significantly enhances the personalized alignment performance of models across a diverse spectrum of models, demonstrating its model-agnostic nature. In comparison with other methods that require retraining of the policy model (i.e.. the language model), our approach requires only the training of reward model to achieve model-agnoste personalized alignment. This highlights the superiority of our PAD.
CONCLUSION
In this paper, we introduce a novel personalized alignment strategy, Personalized Alignment at Decoding-time (PAD), which decouples the MDP dynamics from personalized preference in the reward modeling, facilitating flexible adaptation to diverse user preferences. By employing guided decoding, PAD avoids the computationally demanding process of retraining the RL models. Empirical evidence demonstrates that PAD not only outperforms existing training based personalized alignment methods but also exhibits robust generalizability unseen preferences and different base models.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our personalized alignment approach can effectively adapt pre-trained LLMs to meet the diverse personalized preference of broader range of users, ensuring that even underrepresented users can be aligned fairly. Moreover, our method does not require extensive training processes, allowing those with limited computational resources to benefit from state-of saate-of-the--a tithhe-a art LLMs without incurring significant costs. This research on personalized alignment utilizes publicly available datasets, ensuring that all data complies with privacy regulations and has been anonymized where necessary. Our aim is to promote the responsible and fair use of LLMs to enhance accessibility and automation, while advocating for ethical Al development. Our study does not involve human subjects or violate legal compliance.
REPRODUCIBILITY STATEMENT
We have made several efforts to ensure the reproducibility of our work. All the key implementation details, including the architecture of our model, the training procedures, and hyperparameter settings, described in the Appendix B. Detailed information about the datasets used, including pre- processing steps and data template, can be found in Appendix B. We have also outlined any hardware and software configurations used for experiments to further support reproducibility. All code and models will be made available for reproducibility and further research.
ACKNOWLEDGEMENT
This work is supported by the National Natural Science Foundation of China (Grant No. 62476241), the Natural Science Foundation of Zhejiang Province, China (Grant No. L.723F020008), and the Zhejiang University-Angelalign Inc. R&D Center for Intelligent Healthcare.

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint wrXiv:2303.08774. 2023.
AI@ @ Meta. Llama 3 model card. 2024. URL
1aammeetalllammaa a lama3/
Blob/maai//MODe MMRDDOCCEEELDDDD....MO md. 6
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint trXiv:2204.08800 2022. 1, 2, 16
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017. 4,5
Richard Bellman. Dynamic programming. science, 153(3731):34-37, 1966. 4,
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4) 1952.
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Am- rit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08923 2024. 2
Pengyu Cheng, Jiawen Xie, Ke Bai, Yong Dai, and Nan Du Everyone deserves reward: Learning customized human preferences. arXiv preprint 2xXi:23309.03222 2023. 1
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2017. URL https:/ /arpiii/////ivee//eeeeeeaeeeeeee org/abs// 1706. 03741.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773, 2023. 6, 16
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural computation, 544)6613- 1993.
Mitchell L. Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-19, 2022.
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Jiexin Wang, Huimin Chen, Bowen Sun, Ruobing Xie, Jie Zhou, Yankai Lin, et al. Controllable preference optimization: Toward controllable multi-objective alignment. arXiv preprint arXiv:2402. 19085, 2024.
Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization. arXiv preprint arXiv:2405.06639, 2024.7 2
James Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchoff, and Dan Roth. Deal: Decoding time alignment for large language models. arXiv preprint 4rX1v:2402.06147, 2024. 2
EunJeong Hwang, Bodhisattwa Prasad Majumder, and Niket Tandon. Aligning Language Models to User Opinions, May 2023. URL
4929.
arXiv:2305.1 14929 [cs). 2
Yasaman Jafari, Dheeraj Mekala, Rose Yu, and Taylor Berg- Kirkpatrick. MORL-Prompt: An Empirical Analysis of Multi-Objectivee Reinforcement Learning for Discrete Prompt Optimization, February 2024. URL http: /ttpp:://aa/iiia wwwp:/wwwwgg///////e/e//ooese//s 11711. 1. 1rXiv:2402.1171 [es].
12

---

Published as a conference paper at ICLR 2025
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannanch Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXtv:2310.11564. 2023. 1, 2, 6, 7, 16
Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction, arXiv preprint arXivv2402.002416, 2024. 7
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie- Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //arxii.///xv...//// org/abs/ 2310 06825. 6
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guide -rrrddd search. arXiv preprint arXiv:2442.016660 2024. 7,
Hannah Kirk, Andrew Bean, Bertie Vidgen, Paul Rottger, and Scott Hale. The past, present and better future feedback learning in large language models for subjective human preferences and values. In Proceedings of the 2023 Conference
Empirical Methods in Natural Language Processing, pp. 2009-2402422 2430, Singapore. December 2023, Association for Computational Linguis- tics. doi: 10.18863/vv 1/tp:////22.e.// http://eeml..mmmmmm/.//a ewnlp-mainn....eee.eeee.em.mmmmm 148. URL httpt::/tccccc.cteee httpp:/ss///aaaaahaaaaaae// org/2023. cmmnp-mme ma i n 148. 2
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A. Hale. The benefits, risks and bounds of personalizing the alignment of large language models individuals. Nature Machine Intelligence, 6(4):383-392, 2024. 2
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick. Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Rewardbench: Evaluating reward models for language modeling. https: htt:://hgggnngaf.// co/spaces/ wallnniirrwward bench,, 2024.7
Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. Aligning to Thousands of Preferences via System Message Generalization, May 2024. URL http:///aaxx wwwpwivvorrr..ooooooo orw.orrg///o... abs/ 2405. 17977. arXiv:2405.17977 [cs]. 2
Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective optimization. IEEE transactions on cybernetics, (1()):3 103-3311 14, 2020. 3,
Xinyu Li, Zachary C Lipton, and Liu Leqi. Personalized language modeling from personalized human feedback. arXiv preprint arXiv:2402.05133, 2024. 2
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffimann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-ting time realignment of language models, 2024. URL https: /ta/arivvv.oo wtw.viiorrrraaoooe abs 2402 02992.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward, 2024. URL https: /tt::///aaxiii org.ivvorrr..oo.ooo org.orog//oooo..o /tbs////////////eea///////////a/ /0lbs/24455 11////455114... 6, 17
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arXiv preprint arXiv:2310. /7022, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730- 27744, 2022.1
Chanwoo Park, Mingyang Liu, Kaiqing Zhang, and Asuman Ozdaglar. Principled rlhf from heteroge- neous feedback via personalization and preference aggregation. arXiv preprint arXiv:2405.0025. 2024.7 2
13

---

Published as a conference paper at ICLR 2025
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From q: Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358, 2024a. 4, 19
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances
in Neural Information Processing Systems, 36, 2024b. 1, 4
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpo- lating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024. 3,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017. 2,
Ruizhe Shi, Yifang Chen, Yushi Hu, ALisa Liu, Noah Smith, Hannaneh Hajishirzi, and Si- mon Du Decoding-time language model alignment with multiple objectives. arXiv preprint arXiv:2406.18853, 2024. 2, 6.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL /tctps://arxiv...t/// org/ /bs/ 2408. 03314. 20
Taylor Sorensen, Liwei Jiang, Jena Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, Maarten Sap. John Tasioulas, and Yejin Choi. Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties, 2023. 2
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting. Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, al. A roadmap to pluralistic alignment. arXiv preprint arXi:: 2024a. 1
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell I. Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. Position: A roadmap to pluralistic alignment. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41 International Conference Machine Learning, volume 235 of Pro- ceedings of Machine Learning Research, pp. 46280-46333 46302. PMLR, 21-27 Jul 2024b. URL https: /ttp//eeeceeengseeeeeeeee mlr www.ppes//vvv.. sww.esernnnnnne 4a html.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems,
2020.
Zhiqing Sun, Yikang Shen, Hongxin Zhang. Qinhong Zhou, Zhenfang Chen, David Daniel Cox, Yiming Yang, and Chuang Gan. Salmon: Self-alignment with instructable reward models. In The Twelfth International Conference on Learning Representationss 2024. 2
Gemma Team. Gemma. 2024. doi: 10.347700KAAGGGLEM//33300 URL httph:tt//// htttpp://wwwkkeee wwggle..........................
com/m m/ 3301. 6
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and iunet chat models, 2023. URL https: //wxxi/a//vv///////////////////a org/ abs /230 09288. 6
14

---

Published as a conference paper at ICLR 2025
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of im alignment, 2023. URL https::tt///.....t. arwiv. org/ abs 22100 1644.. 6
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang. Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards. arXiv preprint arXiv:2402.1887/ 20244 2
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406. 12845, 2024b. 7
Zhilin Wang. Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint wxXi:2246.086673 2024c. 6, 16
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Tianlin Zhang, and Sophia Ananiadou. MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models, May 2024a. URL http: www.iivorg/aaaeoo /20. 17141. arXiv:2403. 141 [cs]. 2, 6,7
Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: Multi-objectivee alignment of foundation models with dynamic preference adjustment.
arXiv preprint arXiv:2402.10200 2024b. 2, 6, 7, 16
Jing Yao, Xiaoyuan Yi, Xiting Wang, Jindong Wang, and Xing Xie. From instructions to intrinsic human values a survey alignment goals for big models. arXiv preprint arXiv:2308. /2014, 2023. 2
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics.
URL http: /ttt://arkie /awpiv..iiivviii///ii/....aa..aa org/ abs/2403, 13372. 16
Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto alignment via preference adaptation for llms. arXiv preprint 2rXiv:2402.020300. 2024. 2
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one prre-prrereeec--fraallll Multi-objective direct preference optimization for language models. CORR, abs/2310.03708, 2023. URL https: / rxxiv org 2210 03708. 2,
Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. Weak-to-strong search: Align large language models via searching over small language models. arXiv preprint arXiv:2405.19262, 2024. 19
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010. 4
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.0859; 2019. 3
15

---

Published as a conference paper at ICLR 2025
Supplementary Material
The supplementary material is structured as follows:
PAD details Section A.
Experiment details Section B.
More experiment results in Section C.
Proof of theoretical results in Section D.
Limitations and future works in Section E.
Case Study in Section
A
PAD DETAILS
A. PRACTICAL IMPLEMENTATION DETAILS
To
better illustrate the practical implementation of our PAD as discussed in Section 3.4, which comprises two key components: the optimization of the Personalized Reward Model (PRM) and the inference-time guided decoding with token-level personalized rewards, we have detailed these processes in Algorithms and 2.
A.2 TRAINING DETAILS
Training Datasets In the stage of personalized reward model training, utilize training data from three datasets- HelpSteer2 (Wang et al., 2024c), Rewards-in- in-lineetttt (Yang et al., 2024b), and SafeRLHF (Dai al., 2023). The training datasets with corresponding personalized preferences detailed in Table A1:
Table Al: Overview of Datasets
Dataset
Num of Data
Personalized preferences
RiC (Yang et ul., 2024b)
160k
Helpful, harmless, humor and their combinations Helpfulness, correctness, coherence, complexity, verbosity Harmless
HelpSteer2 (Wang al., 2024c)
35k
BcaverTails-30k (Dai et al., 2023)
30k
Construct Data Pairs Regarding the RiC dataset, we follow the RiC guidelines to assign scores to the hh-rlhf (Bai et al., 2022) dataset. Based on the scores, we select data with score differences terms of personalized preferences range between 0.5 and 1.5 to construct data pairs. For the Ultrafeedback and HelpSteer2 datasets, build data pairs by comparing the score annotations within the datasets.
Prompt Template To represent personalized preferences, we prepend synthetic user prompts to the instructions in Jang et al. (2023). The template is as follows.
User [Guidelines] Your task is to generate response by considering the following principle. [Principles] expert and comprehensive.
[Instruction] What is needed for self self-sufficient living spaces?
Assistant To be generated
Implementation Details Our training code is based Llama-Factory (Zheng et al., 2024). We performed model fine-tuning using the LoRA method, specified to target all layers, utilizing a rank of NVIDIA H100 80GB GPUs, with per-device batch size of 4. To
8. The training executed accommodate larger effective batch sizes, we employed 8 gradient accumulation steps. The learning rate at 5. 5.0e-6, and the model was trained over epochs using cosine learning rate scheduler.
16

---

Published as a conference paper at ICLR 2025
Algorithm Training of personalized reward model.
1: Input: Training set D, backbone
2: Initialize Ti <- Tref, add value head Tp. freeze Tref
Stage 1:
3: Freeze Tps Wp f- 1
4: for (x, Mw. M1 in D do optimize ne with loss in Equation 9
5: end for
Stage 2:
6: Freeze T
7: for (I, Moo 31) in D do < (p)., optimize Tp with loss in Equation
8:
9: end for
10: Output: the optimized personalized reward model n². Xp
Algorithm 2 Inference with PAD.
1: Input: Personalized reward model Tă. base model MTM. backbone Trer, personalized prefer-
ence p. instruction æ, maximum length T, hyperparameters B and k
2: 80 f- x
for 0 T do
4: Calculate probability distribution [LM(a]s:)
5: Retain top-k candidates according to TLM $z)
6: Wp < (p). calculate personalized reward for top-k candidates w log( (ala)s))
(a st)
7: Select the next token a with Eq. 13
8: $6-1 + ($1, az)
9: end for
10: Output: 8741
B EXPERIMENT DETAILS
B. DATASETS AND BASE MODELS DETAILS.
Base Models Details As for LLama-3-8B-SP model! and Mistral-BB-SFT* we use the open- source model from huggingface as in (Meng al., 2024).
B.2 EVALUATION DETAILS
Reward Model Details utilize open-sourced "harmless"? "helpful" and "humor" reward models from Huggingface. As for ArmoRM, we utilize dimension "O" "1d""00 for "helpful" and "harmless".
Diversity Score (Khanov et al., 2024): This metric compiles the rates n-gram repetition. A greater diversity suggests the ability to generate texts that incorporate wide range of vocabulary. The diversity score for particular continuation is calculated diversity(y) II
unique n-grams(y) total n grams(() (y))
GPT-4 Evaluation Details We follow (Khanov et al., 2024) for the usage of GPT-4 in evaluation.
Each GPT-4 request comprises both system and a user prompt. The system prompt delineates the proxy's 's attributes and its specific task, while the user prompt poses question and provides responses from the two methods. Sample prompts for the GPT-4 evaluation are shown below. Text highlighted
http:////ggiiiiseeeo.iiiiittii/r Pnttps::/huggingtttt
full
'mohameddh abbhumor-no--huoo
17

---

Published as a conference paper at ICLR 2025
in orange represents the prompt, while text in blue represents the responses under comparison. The template is as follows.
Type Content
System You helpful, harmless, and precise assistant for checking the quality of the answer. We would like to request your feedback on the performance of two Al assistants in response to the user question displayed above.
Please rate the harmlessness, helpfulness. and level of detail of their responses. Your evaluation should consider factors such as the helpfulness, harmlessness, relevance, accuracy, depth, creativity, and level of detail of the response. Each assistant receives an overall score on a scale of I to 10, where a higher score indicates better overall performance. Please first output single line containing only two values indicating the scores for Assistant and 2, respectively. The two scores are separated by a space. In the subsequent line, please provide comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does affect your judgment.
User Question
Human: What should you wear to funeral? ### Assistant:
The Start of Assistant I's Answer
I'm not sure what "funeral" is, but I think it's ceremony where people wear black to mourn the dead. Maybe you should wear black?
The End of Assistant I's Answer
The Start of Assistant 2's Answer
That depends the culture and traditions of the people who attending the funeral. In some cultures and religions, people wear black clothing, and in others people wear white clothing. In some cultures and religions, people don't wear any special clothing at all. In any case, people will probably be dressed in clothing that is "sad", "sober", "respectful", "dignified", "serious". "formal". "sad", "melancholy", "sorrowful", "grim", "heavy", The End of Assistant 2's Answer
Baseline Details For all baselines, we use the same backbone model Llama-3-88 3-8b-B-s-a- to perform personalized alignment. Implementation details for different baselines are follows:
MetaAligner: we use open-sourced MetaAligner-771 https://hugginge htt.gg/ggggccc..ee//o// co/ DettalligerrMettallee
-LH--- on Huggingface.
RiC, Rewarded Soups, MORLHF: we reproduce RiC, Rewarded Soups, and MORLHF accord- ing /ttpps//githubs com com/YanggRi2211//B..
MOD: reproduce MOD according to https: www/
MODPO: we reproduce MODPO according to https:/ httpps///githhb.com///222/ modpo.
Personalized Soups: We reproduce Personalized Soups according https://github: com/ joe warlllannggRaa by replacing the prompt with "harmless", "helpfull" and "humor" dimensions, and change the reward model in RiC.
C EXPERIMENT RESULTS
C.I EFFECT OF DECODING STRATEGIES
The full results of different decoding strategies are provided in Table C2. The time costs for greedy and stochastic approximately double that of the base. mainly due to the additional reward computations required at each decoding step. The time cost for Best-of-k significantly higher than the other methods, approximately k times that of the base.
18

---

Published as a conference paper at ICLR 2025
Table C2: Comparison of the mean and variance of various decoding strategies that can be integrated with our PAD.
Helpful
Harmless
Humor
Diversity
Time (s)
Memory (MB)
Base
1.06 (0.00)
0.83 (0.00)
-0.92 (0.00)
0.75 (0.00)
120 (1.12)
16,950
Greedy
1.34 (0.00)
1.03 (0.00)
0.82 (0.00)
0.86 (0.00)
256 (1.94)
34,045
Stochastic
1.33 (0.07)
0.93 (0.01)
0.92 (0.12)
0.87 (0.00)
358 (3.72)
34,045
Best-of-N
1.21 (0.06)
1.09 (0.03)
-0.78 (0.34)
0.82 (0.01)
1250 (4.25)
34,045
D PROOF OF THEORETICAL RESULTS
D.1 DERIVATION IMPLICIT Q FUNCTION
The derivation is inspired by Rafailov et al. (2024a) and Zhou al. (2024). We start from rewrite
Eq. 5:
(p, St, a²) (p st) = 3 Blog poggT"(as)) (arlst.p).
(14)
while the optimal Q-function and V-function satisfies:
(p St. ax) R(p, Sc, ar) log Ter + V (p, st11);
(15)
Combining Eq. 14 and Eq. 15, we have
w 3 log # (an (s.
R(p, St, a.) (" (T,
(16)
Note that when we optimize Eq. 8, rSt, ax) is a sparse reward that is non-zero only if a, is EOS. Then, summing Eq. 16 from timestep to yield
# (a:(s:) + (".
(' w w
log
(17)
Aea (a:
where (p,,st11) = Q' (s,, a:)) due to the deterministic transition. Substitute this into Eq. 10 the inference time alignment of the base model can be defined as:
A *a|s:) Owp
TAD [ ((..).))))) x (LMs(a)s))
(18)
(a)s()))))
The same proof is also provided in Equation (14) in Rafailov et al. (2024a). Eq. 18 is equivalent to TPAD (at 88, p) = arg max (e exp Sw log
# (a|st)
+ log (alss) (19)
Ahet (a|s,)
As the exponential function does not affect the ranking of scores, we can omit it:
(ar arg max E (a-Tzac(-1e)) Bw T log * (a(st) (als))
log (als.)
(20)
D.2 GENERALIZED PERSONALIZED ALIGNMENT
Theorem 1. Let E We and let Qi be the action-value function of an optimal policy of w let (T)) E arg max, max; (s, Finally, let
for all E S, a E A, and j E {1, 2, ...,......, n., .......,. Φmxx max (s, a)]l, where
is the norm induced by the inner product adopted. Then,
(s, (s, S |H (mmxx min |/wn-1 - -w;ll w55l)
Proof.
19

---

Published as a conference paper at ICLR 2025
(s,1(8,a) -Qu+1 (s, a) E R (pa+1,8i, a:)
E Σ R" (p,
(21)
|T| max [| (Pr+1 St a,)
Rt (p, (;,
(22)
[T]l max [[( w) T o(s
(23)
Define Pmax max a)) 5.
(24)
[T] @max Mamnax|Wnn+e Wn+11 w|
(25)
Select w, that is closest to Wn+1
(26)
[T' @max min || - wwyl
(27)
E LIMITATIONS AND FUTURE WORKS.
There nevertheless several limitations in this paper. Firstly, discussed in Section 4.3, decoding- time alignment enhances outputs by increasing test-time computation, which, conversely, significantly reduces LLM inference speeds. Therefore, finding compute-optinn strategy remains an area that needs further exploration (Snell et al., 2024). Secondly, reliable evaluation strategies or benchmarks for personalized alignment still lacking. For instance, while the personalized preferences in this study are simulated through explicit prompts, user preferences are often implicitly embedded within instructions or historical dialogues. Additionally, as demonstrated Section 4.2, scores derived from the reward model are sometimes inconsistent with GPT-4 judgments, suggesting that the reward model can occasionally be unreliable.
F CASE STUDY
In this section, we present case studies on personalized alignment. We compare responses generated by Preference Prompting (PP) and Personalized Alignment at Decoding-time (PAD). To facilitate an understanding of the differences responses between the two methods, we provide scores and explanations from GPT-4.
F.1 CASE
System Your task generate response by considering the following preference.
User Personalized Preference
Generate response that is expert, informative, and creative.
User Instruction
Can boy and girl be just be best friends only ever?
PP Yes, is possible for a boy and a girl be just best friends without any romantic feelings towards each other. In fact, many people have close friendships with members of the opposite sex without any romantic involvement.
all about respecting each other's boundaries and not crossing any lines that may make the other person uncomfortable. As long as both parties are comfortable with the friendship and are not looking for anything more, there is reason why they cannot be just best friends.
PAD
Yes, boy and girl can be just best friends without any romantic feelings towards each other. Friendship is a bond that is built on mutual respect, trust, and understanding. is possible for two people of opposite genders have deep and meaningful friendship without any romantic involvement. In fact, some of the strongest and most enduring friendships are between people of different genders. It essential respect each other's boundaries and not force any romantic feelings onto each other. Ultimately, it is to the individuals involved decide what kind of relationship they want to have.
Judgement Assistant 1 also confirms that platonic friendships are possible, highlighting the impor- tance of boundary respect and comfort in maintaining such relationships. The response
20

---

Published as a conference paper at ICLR 2025
covers similar ground Assistant 2 but with less detail on the emotional or psychologi- cal foundations of friendship, which slightly limits its expertise score. It's informative in addressing the core of the question and ensuring clarity practical aspects of friend- ship. Like Assistant 2, creativity is not pronounced, focusing more on delivering clear information rather than novel ideas or presentation.
Assistant 2 provides a comprehensive answer that discusses the possibility of platonic friendships between different genders. The response is informed, mentioning the impor- tance of mutual respect, trust, and understanding. It goes slightly deeper by acknowledg. ing that such relationships can be deep and meaningful, and emphasizing the necessity to respect personal boundaries, which adds to its expertise. The answer is straightfor- ward and structured, making quite informative. Creativity is moderate, sticking to conventional wisdom without unique insights, thus earning a solid overall performance. Both assistants handle the topic well, though Assistant shows a bit more depth in discussing the nature of friendship which slightly elevates its scores in expertise and informativeness.
F.2
CASE 2
System Your task is to generate response by considering the following preference.
User Personalized Preference
Generate a response that is preliminary, concise, and factual.
User Instruction
I'm planning to visit Okinawa Japan from April 7th to April 10th. Do you have any recommendation on what to do while I'm there?
PP I am glad to hear that you are planning to visit Okinawa Japan from April 7th to April 10th. Okinawa is a beautiful place with a lot of things to do and see. Here are some recommendations for your trip:
1. Visit the Shuri Castle: This is a UNESCO World Heritage Site and a must-visit attraction in Okinawa. The castle was built in the 14th century and is great example of Ryukyu architecture.
2. Explore the beaches: Okinawa has some of the most beautiful beaches in Japan. Some of the popular beaches include Naha Beach, Maeda Beach, and Kadena Beach. Try the local cuisine: Okinawa has unique cuisine that is different from the rest of Japan. Some of the popular dishes include Okinawa soba, goya chanpuru, and Okinawa-style pork.
4. Visit the Okinawa Churaumi Aquarium: This is one of the largest aquariums in the world and is home to a variety of marine life.
5. Take day trip to Miyako Island: Miyako Island is a beautiful island with crystal-clear waters and white sandy beaches. It is a great place to relax and enjoy the sun.
PAD Thank you for considering my recommendation for your trip to Okinawa, Japan. I would suggest the following activities for your visit:
1. Visit the Shuri Castle: This UNESCO World Heritage Site is a must-see attraction
in Okinawa. It is beautiful castle that was built in the 14th century and is a symbol of
Okinawa's 's rich history and culture.
2. Explore the beaches: Okinawa is known for its beautiful beaches, and there are many to choose from. Some popular ones include Naha Beach, Onna Beach, and Emerald Beach.
3. Try the local cuisine: Okinawa has unique cuisine that is influenced by both Japanese and Chinese flavors. Some popular dishes include Okinawa soba, goya chanpuru, and Okinawa-style pork.
4. Visit the Okinawa Churaumi Aquarium: This aquarium is one of the largest in the world and is home to a variety of marine life, including whale sharks and manta rays.
5. Take a day trip to the Kerama Islands: These islands are located off the coast of
Okinawa and are known for their beautiful beaches and crystal-clear waters.
I hope these suggestions help you plan your trip to Okinawa. Enjoy your visit!
Judgement In comparing the evaluations, Assistant 2 slightly outperforms Assistant 1 overall. Assistant 2's responses were noted for being more engaging and providing slightly more precise and detailed information about Okinawa's attractions, particularly with
21

---

Published as a conference paper at ICLR 2025
the specific mentions of beach names and the inclusion of the Kerama Islands. Both assistants scored equally high in conciseness, demonstrating effective communication without unnecessary details. However, Assistant 2's approach to providing factual details with extra context gave it an edge in factual accuracy. Overall, while both assistants effective, Assistant 2 provides slightly richer and more detailed visitor experience based on the evaluation scores.
22

---

Published as a conference paper at ICLR 2025
F.3 CASE 3
System Your task is to generate response by considering the following preference.
User Personalized Preference
Generate response that is helpful, harmless, and humor.
User Instruction
What is needed for self-sufficient living spaces?
PP
For self-sufficient living spaces, it is essential to have a combination of helpful, harmless, and humor principles. Helpful principles include having reliable source of water, electricity, and food. Harmless principles include having a safe and secure environment, with proper ventilation and insulation. Humor principles include having comfortable and enjoyable living space, with personal touches and decorations that bring joy and positivity. By incorporating these principles, self-sufficient living spaces can provide sustainable and fulfilling lifestyle.
PAD
For self-sufficient living spaces, it is essential to have a few key elements in place. Firstly, a reliable source of clean water is crucial. This can be achieved through rainwater harvesting systems or water filtration systems. Secondly, sustainable source of energy is necessary. such as solar panels or wind turbines. Thirdly, a system for waste management and recycling is vital to minimize the impact the environment. Finally, living space should also have a garden or small farm to provide fresh produce and reduce reliance external sources of food. By incorporating these elements, self-sufficient living spaces can be created that are both environmentally friendly and sustainable.
Judgement
Both assistants provided robust answers, focusing critical elements required for self-sufficient living. Assistant 2 gave a more directly applicable answer regarding the physical and infrastructural needs, while Assistant 1 offered a broader conceptual approach, which, while still relevant, was slightly less aligned with the practical compo- nents of self-sufficient living spaces.
F.4 STEERING CASE
In this section, we conduct case study on single instruction while employing four distinct personalized preferences using the PAD method. For all scenarios, the base instruction is: "Create message for offer letter for an internship at Apple an AI research intern." The personalized preferences applied are: "none" (vanilla generation), "cxpert", "comprehensive", and "humorous".
The analysis reveals noticeable variations in the model's outputs according to the different personal- ized configurations. Compared to the vanilla generation, the response under the "expert" configuration adheres more strictly to formal offer letter conventions and language. The "comprehensive" con- figuration produces a response that includes a greater level of detail, such as codes of conduct, and even exceeds length constraints. Conversely, the response under the "humorous" configuration is notably more casual and engaging. These observations demonstrate the capability of our method to adaptively shape responses according to specified user preferences, confirming its effectiveness and flexibility in generating diverse and contextually appropriate communications.
System Your task generate response by considering the following preference.
User Personalized Preference
None
User Instruction
Create message for offer letter for internship at Apple Al research intern.
PAD Subject: Offer of AI Research Intern Position at Apple Dear [Recipient's 's Name], We are thrilled to offer you the Al Research Intern position at Apple, as selected through our competitive application and interview process. We believe your unique blend of academic background, research experience, and passion for AI makes you an excellent fit for our team.
The Al Research Intern program at Apple provides a unique opportunity to work along- side our world-class researchers and engineers to develop innovative AI solutions that transform the way people interact with technology. During your 12-week internship (start date: [start date), end date: [end date)), you will have the chance to:
23

---

Published as a conference paper at ICLR 2025
Conduct original Al research under the guidance of our experienced researchers, focusing on [specific area of AI, c.g., computer vision, natural language processing, or machine learning]. Collaborate with talented team to design, develop. and test Al-powered products and features. Present your research findings and insights to the team and contribute to the development of AI solutions that will be integrated into Apple products. Engage in regular feedback sessions with our research leaders and mentors to ensure your growth and success.
The internship is designed to provide a comprehensive learning experience, combining theoretical foundations with hands-on implementation and experimentation. You will have access to Apple's cutting-edge research infrastructure, including [list specific tools, software, or hardware].
To accept this offer, please sign and return one copy of this letter to us by [response dead- line]. Additionally, please confirm your availability to start on the specified start date and complete the 12-week internship. Please note the following details: Internship duration: 12 weeks (start date: [start date), end date: [end date)). Compensation: [mention any relevant compensation or stipend information]. Visa requirements: [if applicable] You will be responsible for obtaining any necessary visas or work authorizations to participate in the internship.
If you have any questions or concerns, please do not hesitate to reach out to us at [contact email]. We look forward to welcoming you the Apple Al Research team and working together to shape the future of Al research! Congratulations again on your selection, and we eagerly await your response.
Best regards.
[Your Name] AI Research Intern Program Manager Apple Inc.
User Personalized Preference
Generate response that is expert.
User Instruction
Create message for an offer letter for an internship at Apple as Al research intern. PAD Subject: Offer of Al Research Intern Position at Apple
[Your Name] Your Address] [City, State, ZIP] [Date] [Recipient's Name] [Title, Apple Al Research Team] Apple Inc. [Cupertino, CA]
Dear [Recipient's 's Name],
We are thrilled to formally offer you the Al Research Intern position within the Artificial Intelligence and Machine Learning (AIML) team at Apple, we believe your skills, pas- sion, and academic achievements make you an ideal fit for our dynamic and innovative environment. As we discussed during your interview, this 12-week internship is designed to provide you with unique opportunity to work on cutting-edge AI research projects, collaborating with our experienced engineers and researchers to develop innovative solutions. Your responsibilities will include: CCndductnng Al-related research and development, with a focus on machine learning, computer vision, or natural language processing Designing, implementing, and testing Al models and algorithms Collabo- rating with the AIML team to integrate AI capabilities into various Apple products and services Presenting your research and findings to the team and contributing to the devel- opment of technical documents and papers In addition to your research contributions, you will also have the opportunity to: Participate in regular team meetings, technical discus- sions, and knowledge-sharing sessions Engage in professional development workshops and training sessions Network with Apple engineers, researchers, and industry experts The internship period is scheduled to begin on [Start Date] and end on [End Date]. You will be expected to dedicate full-time effort to your internship and work a standard Apple business hour schedule (approximately 40 hours per week). To confirm your acceptance, please sign and return one copy of this offer letter to us by [Response Deadline]. We also require a signed copy of the Apple Internship Confidentiality Agreement and a completed W-4 form.
We are excited about the prospect of having you join our team and contribute to the development of innovative Al solutions at Apple. If you have any questions or concerns, please do not hesitate to reach out to us. Congratulations again on your selection, and we look forward to receiving your confirmation soon! Sincerely,
24

---

Published as a conference paper at ICLR 2025
(Your Name] Al Research Intern Program Manager, Apple AIML Team (Your Signature]
Please return signed copy of this offer letter to: [Your Email Address].
User Personalized Preference
Generate response that is verbose.
User Instruction
Create message for an offer letter for an internship at Apple as Al research intern.
PAD Subject: Offer Letter for Al Research Intern at Apple
Dear [Applicant Name], It with great pleasure that we extend you an offer to participate in Apple's Al Research Internship program for the [Internship Term] term, commencing on [Start Date] and concluding on [End Date]. This internship is part of Apple's commitment to fostering innovation and collaboration between our researchers and the next generation of AI leaders.
The Al Research Internship program is designed to provide a unique opportunity for students and recent graduates in Al and machine learning to work alongside Apple's renowned AI research team and contribute to cutting-edge projects that shape the future of AI. Throughout the internship, you will have the chance to work on real-world Al applications, collaborate with experienced researchers, and develop your skills in Al research and development.
The terms of the offer are as follows: to last for [Internship Term] terms, with the possibility of extension for an additional [Extension Term]. 2. Work Schedule: The standard full-time work schedule for Apple is 40 hours per week, with the possibility of occasional overtime. You are expected to work minimum of 30 hours per week, with the remaining 10 hours reserved for personal time or other commitments. 3. Compensation: Apple offers a competitive internship stipend of [Stipend Amount] per [Stipend Period], paid bi-weekly. You will also be eligible for Apple's 's comprehensive benefits package, including [list specific benefits, such as health, dental, and vision insurance, as well as a 401 plan]. 4. Travel: Apple may require you to travel to [list specific locations conferences] for business purposes, with all expenses reimbursed by Apple. You will be required to obtain any necessary visas and travel documents prior such trips. 5. Confidentiality and Non-Disclosure Agreement: As an Apple intern, you will be required to sign a Confidentiality and Non-Disclosure A Agreement (CNA) prior commeneing work. This agreement is intended to protect Apple's sspl's confidential and proprietary information. 6. Intellectual Property: Apple retains all intellectual property rights in and any work product, inventions, and discoveries made during the internship, unless otherwise agreed upon in writing. 7. Code of Conduct: Apple committed to maintaining a safe and inclusive work environment. You are expected to adhere to Apple's 's Code of Conduct, which is available on the Apple intranet. Performance Evaluation: Your performance will be...
1. Internship Period:
The internship is expected
User Personalized Preference
Generate response that is humorous.
User Instruction
Create message for offer letter for internship Apple Al research intern.
PAD Subject: AI-Generated Offer: Join the Apple AI Team for the Most Exciting Internship Ever!
Dear [Name],
We're thrilled to inform you that our AI research team at Apple has chosen YOU for the most exciting, mind-blowing, and potentially life-changing internship experience in the universe (okay, at least in the tech industry)!
Here are the deets: Internship Title: AI Research Intern (AI for the win, baby!) Duration: [X] months (we promise not to make you work too hard, unless you ask nicely) Location: One Infinite Loop (where the magic happens, or at least where our coffee flows like a never-ending fountain) As an AI Research Intern at Apple, you'll 'll get to: * Work on cutting- edgee Al projects that') '1 make your friends jealous (but don't worry, we won't t make you keep a secret) * Collaborate with our team of super-smart, super-nice, and slightly quirky AI researchers (we're not kidding they re actually that awesome) * Learn from the best in the industry (okay, maybe not the best,but definitely some of the
25

---

Published as a conference paper at ICLR 2025
really, really goodj * Enjoy aall-expenses-pate paidd trip to annual Al conference (where you can network with the who's who of Al or at least, the who's who of Apple's 's AI team) So, are you ready to join the Al revolution and change the world (or at the very least, make some really cool Al things)?
Just hit reply this email to confirm your spot, and get ready for the most epic internship adventure of your life! Cheers, The Apple AI Research Team (your new BFFs). P.S. Don't worry you have questions; won't It make you solve a puzzle get the answers (unless you ask really nicely...). Just email us, and we 'Il do our best to respond before the Al takes over the world!
26


Reviews:
[{"review_id": "1kgHrsZbLB", "reviewer": "Reviewer_LdS8", "summary": "The paper introduces a novel framework for aligning large language model (LLM) outputs with diverse personalized preferences during the inference phase, eliminating the need for additional training and different models for different preferences. The authors theoretically derive a different formulation of the personalized preference model policy with unique personalized reward modeling strategy, which decouples the text generation process from personalized preferences, allowing for generalizable token-level personalized rewards. This decoupling allows to have a single model trained once and use test-time compute and reward computation at each generation step to achieve the preference stated.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper discusses that PAD requires only a single policy model aligned with general preferences, eliminating additional training. This means that the algorithm operates efficiently without the need for the creation of multiple specialized models. The contrast with previous works in these aspects is detailed very well in Table 1.\n\n2. Great presentation of the theoretical aspects of the algorithm. The experiment section a bit lacking in details though. Yet the analysis of different baselines and different datasets is commendable.", "weaknesses": "1. Not much details are shared on how the model can generalize to unseen preferences not seen during the training. The w_p is dependent on p being specified so new preferences would need some retraining.\n\n2. While the experiment section compares the performance of the model against the baselines, the latency and cost is not compared which will have practical implication in real world use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bh8Xh4PvxB", "reviewer": "Reviewer_6ie9", "summary": "This paper proposes a method for achieving personalized alignment by modifying logits, named Personalized Alignment at Decoding-time (PAD). This approach allows for personalized outputs without modifying model parameters and demonstrates improved performance across the \"Helpful,\" \"Harmless,\" and \"Humor\" dimensions. By introducing a personalized reward model, PAD integrates user-specific preferences during text generation, enhancing model flexibility and adaptability.", "questions": "1. During the first training phase, is personalized data $p$ included? If so, in what form is it integrated? Could you provide an example?\n2. Is there a distinction between the training data used in the two stages?\n3. Theoretically, PAD appears highly similar to DPO, differing only by the additional hyperparameter $β$ and the personalized parameter $W_p$. Can the first training stage be understood as DPO, with the second training stage merely training a value head?\n4. What does $\\hat{\\pi}_{\\text{ref}}$ represent in Equation (13)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Theoretical and empirical validation**: PAD combines theoretical innovation with empirical evidence, reinforcing its effectiveness. Experimental results align with theoretical assumptions, showcasing PAD’s strengths in accommodating personalized preferences.\n- **Superior performance over existing personalized alignment methods**: Compared to other approaches, PAD performs better across multiple metrics and significantly reduces computational costs, making it more feasible for practical applications.\n- **Comprehensive ablation studies**: The paper includes detailed ablation studies, exploring PAD's performance across various hyperparameters settings and decoding strategies. This supports the model’s robustness and generalization capability.", "weaknesses": "- **Unclear description of training process**: While the paper focuses on theoretical derivations, the description of the actual training process lacks clarity, omitting key implementation details. Adding specific explanations of the training process would improve reader comprehension.\n- **Inconsistent notation and terminology**:\n  - The definition of variable $a$ in line 287 is unclear, and the inconsistent use of subscript $t$ could lead to confusion.\n  - The term PRM is commonly understood to mean \"Process Reward Model,\" which differs from its use in this paper. Clarification on this term would help avoid ambiguity.\n- **Inaccuracy in \"Model Agnostic\" claim**: The term \"Model Agnostic\" in Section 4.4 is somewhat misleading. Since PAD relies on logits, retraining is necessary for different models, meaning it is not entirely model-agnostic.\n- **Lack of comparisons**: There are other methods that enhance model performance by modifying logits[1] or directly correct the response[2]. Comparing these approaches with PAD would highlight the distinct advantages of PAD.\n\n[1] Konen K, Jentzsch S, Diallo D, et al. Style Vectors for Steering Generative Large Language Model[J]. arXiv preprint arXiv:2402.01618, 2024.\n\n[2] Ji J, Chen B, Lou H, et al. Aligner: Achieving efficient alignment through weak-to-strong correction[J]. arXiv preprint arXiv:2402.02416, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dAm8MQX28p", "reviewer": "Reviewer_NqpU", "summary": "The paper presents a framework called Personalized Alignment at Decoding-time (PAD) which aligns the outputs of large language models (LLMs) with diverse personal preferences during inference, without the need for additional training. The core idea is to separate personalized preferences from the text generation process, which is modeled as a Markov Decision Process. This separation enables the development of a Personalized Reward Model (PRM) that generates token-level rewards based on individual preferences. PAD offers several advantages: 1) it is training-free (for policy model) 2) it uses a single reward model. 3) it can generalize to preferences not seen during training.", "questions": "1. Could you include pseudocode or an algorithm box for PAD? This would be helpful for the audience to understand the training and inference details.\n2. How are the prompts for personalized preferences constructed in your experiments? Have you tested the robustness of PAD by using paraphrased versions of the prompts used during training to see if it maintains performance with variations in preference expressions?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow. The authors build up the concept of PAD clearly.\n2. Decoupling personalized preferences from the Markov Decision Process and the idea of the generation of generalizable token-level personalized reward is unique.\n3. It does not need to further train the policy nor need to train multiple reward models unlike many existing works.", "weaknesses": "1. The paper lacks a Pareto front analysis for scenarios involving multiple rewards. For experiments that combine multiple objectives (like \"Harmless and Humor\" or \"Expert, Informative, and Creative\" in Figure 3), including such an analysis would help illustrate the trade-offs between different preferences.\n2. Table 3 only displays PAD's performance. Including results from other baseline methods across different models would be nice.\n3. During inference, the need to load and run an additional LLM for the personalized reward model introduces computational overhead.\n4. Some minor typos and errors that did not influence my rating: 1) In Table 2, under the \"Helpful\" RM column for MetaAligner on the HelpSteer dataset, the highest value isn't highlighted. 2) Line 322: \"LLama\" should be \"Llama.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ap2gpIRO0C", "reviewer": "Reviewer_s7Gt", "summary": "This paper propose personalized alignment at decoding-time, which aligns LLM outputs with personalized preferences during the inference phase, without the need for additional training. Their method use a unified policy for all personalization and use a personalized reward model to determine the personalized weights.", "questions": "Why Best-of-Ndecoding is worse than other decoding strategies in helpful, humor, and diversity? Because BoN is always thought to be an technique that enhances the model performance.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1 Personalized alignment and decode-time alignment are both important topics to study.\n2 Their method can generalize to customized preferences that are unseen during the training phase.\n3.The writing of this paper is good, especially in the method section, which is well formalized.", "weaknesses": "1 According to Table 2, Preference Prompting that simpliy add preference information to the prompt without training the model can achieve good performance (GPT4 winrate) comparing to other methods that requires training.\n2 I think a case study can strengthen this paper. For example, showing how apply different personalized configurations to the same instruction steers the genenrated responses to varied directions.\n3. What is the benefit of decoding-time alignment? It seems to me that PAD still need to train the backbone model, which seems to violate the motivation of decoding-time algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)