A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
Authors: Zhouhang Xie
Junda Wu
Yiran Shen
Raghav Jain
Yu Xia
Xintong Li
Aaron Chang
Ryan A. Rossi
Tong Yu
Sachin Kumar
Bodhisattwa Prasad Majumder
Jingbo Shang
Prithviraj Ammanabrolu
Julian McAuley
Venue: COLM 2025 (2025)
Keywords: Preference Alignment
Personalization
Pluralistic Alignment
Large Language Models
TL;DR: A survey on personalized and pluralistic preference Alignment in Large Language Models

Abstract: Personalized preference alignment for large language models (LLMs), the process of tailoring LLMs to individual users' preferences, is an emerging research direction spanning the area of NLP and personalization. In this survey, we present an analysis of works on personalized alignment and modeling for LLMs. We introduce a taxonomy of preference alignment techniques, including training time, inference time, and heuristic-driven methods. We provide analysis and discussion on the strengths and limitations of each group of techniques and then cover evaluation, benchmarks, as well as open problems in the field.

Published as a conference paper at COLM 2025
A Survey on Personalized and Pluralistic Preference Align- ment in Large Language Models
Zhouhang Junda Wu', Yiran Shen', Raghav Jain', Yu Xia', Xintong Li', X Aaron Chang", Ryan Rossi², Tong Yu², Sachin Kumar Bodhisattwa Prasad Majumder Jingbo Shang', Prithviraj Ammanabrolu' Julian McAuley'
1 University of California, San Diego University of California, Los Angeles
9 "Adobe Research "The Ohio State University 5 Allen Institute for Al
1zhx022,jw066 jessss,, jes038, r6jain, yux078 xil240, priti4,, prittvii,jmcalleeee.. jmcauley @ucsd. edu aaronchang21@g eg acla. {ryrossi, tyu com kumar. 1450osu. edu, bodhisattwameallenni.ccoo
Abstract
Personalized preference alignment for large language models (LLMs), the process of tailoring LLMs to individual users' preferences, is an emerg- ing research direction spanning the area of NLP and personalization. In this survey, we present an analysis of works on personalized alignment and modeling for LLMs. We introduce taxonomy of preference align- ment techniques, including training time, inference time, and additionally, user-modeling based methods. We provide analysis and discussion on the strengths and limitations of each group of techniques and then cover evaluation, benchmarks, as well as open problems in the field.
1 Introduction
Recently, sigmificant progress has been made in aligning LLMs to the overall preferences of users (Zhao al., 2024; Shen et al., 2023). However, prior works show that there no one-size-fits-all solution for preference alignment (Sorensen et al., 2024; Kirk et al., 2024a). Intuitively, while there are universal preferences that are shared across users, such as "it is good to respond in the tone of a friendly and helpful assistant", user preferences are often also individualized and use-case dependent (i.e., contextual). For example, users might have different preferences towards the tone and style of responses (Jang et al., 2023), and even for the same user, the preferred style of response would change depending on the context of interaction, akin to prior works in contextualized personalization (Meng et al., 2023). For example, an expert might have different preferences from a beginner when asking the same question about concept, and even the same person might have different preferences depending on when they interact with the system.
Along with the emergence of these challenges, the notion of personalization in the context of NLP and LLM research has recently attracted increasing attention within the NLP and machine learning communities (Sorensen et al., 2024; Kirk et al., 2024a; Flek, 2020). At first glance, personalization seems to be an intuitive solution for catering LLMs to individualized and contextual preferences described above. However, practice, the notion of personaliza- tion for LLMs is convoluted, ranging from applications of LLMs to classical personalization pole-playing and simulation of individual human behaviors (Chen et al., 2024d; Mou et al., 2024). As we shall discuss later (Section 7), not all types of personalization benefit individual users in a conversation setting.
tasks (Tan & Jiang, 2023; Chen et al., 2024c)
In this work, we focus personalized and pluralistic preference alignment, specific notion of personalization that aims at adapting an LLM's behavior to dynamic user preferences across individuals, groups, and contexts to enhance user satisfaction. We start by defining the problem formulation for personalized preference alignment (Section 3), then introduce intuitive technique taxonomy covering training and test-time methods for preference

---

Published as a conference paper at COLM 2025
User feedback?
(LM
Yes
User
Fine-tuning?
modellig (97)
User preference
User-LM
No
feedbacks
Interactions
Training time Mtd. ($4)
Inference- Mtdd (s5)
User- sser-iieecific
Prompting, Guided Decoding, Logit rectification..
Parameters. Steer/controllabie
Users
(1a)
(1b)
Figure 1: (1a) Overview on personalized preference alignment for LLMs. This includes training (Section 4) and test-time (Section 5) methods, leveraging various feedbacks such as verbal feedback and choices. (1b) An over-simplified decision tree for determining the class of method to use for personalized preference alignment.
alignment (Section Section 5). We then discuss the connection between modeling and personalized preference alignment (Section 7), and discuss benchmark and evaluation (Section 6) well as open problems in the field (Section 8).
What's covered? Broadly speaking, as shown in Figure 1, personalization in the context of preference alignment can be categorized into two classes of methods: Training-time (Section 4) and inference-time (Section 5) personalization that leverages personalized user feedback at different stages of LLMs' life cycle. Additionally, when user preference feedback is absent, personalized adaption of LLMs and LLM-based systems can sometimes also be achieved by modeling the user, which we discuss in Section 7.
What's not covered. Under the theme of LLMs and personalization, there are other relevant areas such as user-behavior modeling (i.e., systems that predict the behavior of users) (Tan & Jiang, 2023; Wu et al., 2024b; Chen et al., 2024c), user-group-behavie
modeling (i.e.,
LLM role-playing) (Chen et al., 2024d;d; Tseng al., 2024), and LLMs for personalized recommender systems (e.g., LLMs & Jiang, 2023; Wu et al., 2024b; Chen et al., 2024c). However, these lines work does not involve directly catering the behavior of LLMs LLM-bbsed conversational systems to user preferences. We point interested readers to related surveys (Section 2) on these topics.
component in product recommender systems) (Tan
Contribution statement. Despite numerous recent efforts consolidating works that involve LLM and personalization (Sorensen et al., 2024; Kirk et al., 2024a; Zhang et al., 2024b; Wu et al., 2024a; Tan & Jiang, 2023; Wu et al., 2024b; Chen et al., 2024c), show personalized alignment is an emerging valuable research direction, with its own methods and evaluation paradigm. To this end, this survey provides a comprehensive overview of personalized and pluralist alignment while establishing its differences and connections to adjacent domains. Further, personalized preference alignment is an emerging domain with no universally acknowledged evaluation and benchmarks. To make it easier for future research to access existing evaluation schemes, we provide an overview the current state of evaluation methods. Overall, this survey serves as up-to-date resource for practitioners and researchers working on personalized and pluralistic alignment and, more broadly, LLM personalization and alignment.
2 Related Surveys and Key Differences
User modeling and personalizing LLMs to simulate user behavior. Recently, few position papers have discussed the concept of personalized preference alignment (Sorensen et al., 2024; Kirk et al., 2024a). However, these works focus on discussing the implications without sunmarizing current progress. There have also been surveys on personalizing LLMs (Zhang et al., 2024b; Wu et al., 2024a), but they do not differentiate between the replicating user
2

---

Published as a conference paper at COLM 2025
E-g. User-apectlie embeddings (Iá et al,, 2124)), stcering vectors (Cao et al, 2024), PEFT modules (Tan et al, 2024boa), soft prompts (Huang 20244), inference heads (Zhuang at, 2024), and mixture pret- erence experts (Zhou et al, 2024).
Useroppefic Parameters (Section 4.1)
A Survey of ofrreronniii aeieedieed Aligrment Medeling
Training-time Personelized Alignment (G4)
models that adaptable conditioned on user stated system prompts Lee al. (2024b), user persons Wang al. (2021b)), user group identifeers (Kumar et al. 202(b), as well as in- fered userspeeiifcc latent (Poddar et 2021; Chen plicit (Balepur et al., 2025) personas.
Eg. LLMs and
ecetr/ccotellllbll Models (Section 4.2)
2024b)
Eg. URIAL (Lin al., 2023), RAIN (L) al, 2023e), OPO (Xu all, 2023), (Salemi 2024), (Kim & Yang, 2024), BPO (Chang et all 2023a), Rewrite Prompt (Ii et al, 20246)
Prompting (Section 5.1)
(Mysore et 2023a), FERMI
Eg. ARGS (Khanov et al,, 2023), RAD (Deng & Rafiel, 2023), CARDS (L. et al,, 2024a), DeAL (Huing et al, 2024n), GenARM (Xu et al., 2004), PAD (Chen al. 2124e), CD (Mudgal et al. 2023), PPO-MCTS (Liu al, 202(b), (Liu et al., 2024d), VAS (Han al., 2024)
Porsonalized Allgnment (85)
Reward and Value-Guided Guided Decoding (Section 5.2)
Logit Rectificstion and Re- Aligament (Section 5.3)
Egg, EFT (Mitchell et al,, 2023), Procy-Tuning (Liu et al, 2024s), MOD (Shi 2024), DeRa
2024c), Alginerdii et al., 2024), MetaAl- giner (Yong al, 2024). Amalet (Zhang al,, 2025)
Figure 2: Technique taxonomy on personalized and pluralistic preference alignment.
behavior to catering for user preferences. Finally, another stream of recent works focuses on LLM-based role-laying (Tseng et al., 2024). However, their focus is still empowering LLMss to replicate certain user groups' behavior. In contrast, our work focuses on summarizing the progress with respect personalized preference alignment.
Application of LLMs to downstream personalization tasks. There are also few surveys on the application of LLMs to other tasks related to personalization, such as recommender systems and user modeling (Tan & Jiang, 2023; Wu et al., 2024b; Chen et al., 2024c), person- alized wearable devices (Li et al., 2024e), personalization on the web (Chen al., 2024c), narrative-drive recommendation (Mysore et al., 2023b), and LLM-based role-laying (Chen et al., 2024d), which are different from the scope covered in this work since the goal of these lines of research is not to build an optimal LL.M-based dialogue system. We provide further clarification of our scope in Section 3, and discuss in more detail the relationship between (personalized) user modeling and preference alignment in Section 7.
3 Problem Formulation and Techniques Taxonomy
3.1 Problem Statement
We begin by introducing the notion of personalization in the context of preference alignment, as illustrated in Figure 1-a. In this setting, system deployers aim to build an LLM (or LLM- based system) that caters specifically to each individual user's 's unique preferences.
For clarity, let X' denote the input space (e-g., user queries or contextual prompts), and y de- note the output space (e.g., responses generated by the LLM). We formalize the personalized reward function
r
x y x 4 R,
which measures how well an LLM's response y to an input satisfies the unique preferences of user u. Our goal is to learn a set of individualized policies { Juctu where each policy nu is tailored exclusively to user Such objective can be expressed as: Vu E U
T arg max Try
Ex-XXy>m [r y, u)]
(1)
In other words, for each user the optimal policy πミ is one that, in expectation over the distribution of inputs, generates responses that best satisfy that user's individual preferences. However, compared to personalization which emphasizes individualized prediction for each user, approaches for achieving pluralistic aligmment are more diverse, sometimes involving systems that produce sets of outputs catering for different preferences. Meanwhile, the LLM-based policies can often have shared parameters, or are built from a single model conditioned on different user information, as we shall discuss in Section 4 and Section 5.
3

---

Published as a conference paper at COLM 2025
3.2 Technique Overview
The problem formulation discussed above yields a straight-forward way to partition existing methods, as shown in Figure Specifically, in order to obtain LLM--bsed policy that cater to each individual's preferences, it is crucial to be able to adapt the LLM itself depending on the interacting user. Naturally, this can be achieved both by training models with user- specific parameters (Section 4.1) or making the model steerable with respect to user inputs (Section 4.2), as shown in Figure 1-b. On the other hand, fine-tuning or adapting LLMs is fre- quently costly. This challenge motivates another category of works that aims at influencing pre-trained LLMs' behavior at inference time, with methods such as prompting (Section 5.1), (Section 5.2), and logit manipulation (Section 5.3). Additionally, we note that there are personalization techniques that nevertheless improve base LLMs towards the goal of catering individual preferences without using user feedback, such as building user-specific memories (e.g., (Yuan et al., 2025; Zhang et al., 2022)) following the assumption that users generally likes to be remembered. We provide discussion of this complementary class of valuable techniques in Section 7.
cnntrolled-deodiingg
3.3 Granularity Personalization
While personalized preference alignment aims to cater to, ultimately, individualized prefer- ences, personalization typically suffers from the issue of sparse feedback (Li et al., 2023b). Specifically, an individual user's interactions with the system are frequently too few to allow meaningful learning to happen. To this end, similar to works in adjacent personalization areas such as recommender systems (Li al., 2023b), it is often helpful to exploit the fact that there can be groups of users that shares similar preferences, thus bypassing the feedback sparsity issue. Similar to these prior works, personalized LLM aligmment can also happen different granularity: individual users' levels and user groups based and social relationships (Sorensen et al., 2024; Kumar et al., 2024b). We note that there is also a special case of "contextual" shared preference between user groups, where a set of users momentarily shares preferences based on their purpose of interacting with LLMs. For example, a group of users that are seeking help from an LLM-based therapy chatbot may collectively wish the LLM to act as helpful therapist Stade et al. (2024). We provide discussions on this special case in Section 7.
profiles
4 Training-time Personalized Alignment
To effectively adapt LLMs and LLM-based systems to personalized user preferences, straight-forward solution is to develop specialized models via training. In this section, we introduce two popular classes of techniques: building models with user-specific parameters and training models that are sensitive to input user preferences.
4.1 Learning from Feedbacks with User-specific Parameters
Motivation. One of the most straightforward ways to build LLM that caters to individ- ualized user preference is by keeping separate parameters for each user (or user group), effectively learning set of policies whose behavior slightly deviates from each other without explicitly specified user preference under the standard RLHF setting.
Comparative Analysis. Addressing the challenge of inferring user-specific preferences without requiring explicit specification, (Li et al., 2024d) employs a lightweight user model to learn from human feedback, while (Cao et al., 2024) follows similar setting, but learn per-user -teeoong-vectors to achieve personalization. Following similar intuition, (Tan et al., 2024b) and (Tan et al., 2024a) advocates for a dedicated per-user PEFT module that stores individual behavior patterns. Other than adapter modules, (Huang et al., 2024b) explores a complimentary parameter-efficient personalization approach via soft prompts. (Zhuang et al., 2024) introduces HYDRA, factorization framework that couples shared base model with user-specific heads, yielding notable improvements over prompt- methods. (Zhou et al., 2024) introduces RL.PH,, which merges outputs from specialized
4

---

Published as a conference paper at COLM 2025
Sec 4.1 Learning separate parameters for each user
Sec 4.2
Building single steerable model based on user input
CONTEXT
Figure 3: Personalized language model alignment during training time.
expert LLMs using lightweight Preference Control Model (PCM) that dynamically adjusts token predictions based on user context; i.e, mixture of preference experts. Finally, in contrast to learning user-specific parameters in LLMs, (Park et al., 2024a) explored learning multiple reward models to handle the trade-off between bias and variance in personalized preference alignment.
Limitations. Despite these advances, existing personalized RLHF frameworks often struggle to simultaneously ensure personalized adaptation and global model performance, with many approaches relying on complex multi-stage processes or additional components that may hinder model scalability (Park et al., 2024b; Han et al., 2024; Lee et al., 2024a). Furthermore, challenges remain robustly handling heterogeneous and strategic feedback while integrating efficient privacy-preserving techniques, pointing to the need for more streamlined and resilient solutions such as federated personalized alignment (Zhang et al., 2024a; Jiang al., 2024).
4.2 Building Steerable Model that Adapts Responses Based on User Input
Motivation. Another popular choice for personalization is building single base model that's steerable (Sorensen al., 2024), where are single models' behavior changes based on the user it is interacting with, similar to prior research for controlled text generation (Hu et al., 2017). This line of research often assumes user inputs relevant individualized preferences are available, such as explicitly stated preferences or personas that imply user preferences are available at inference time, bypassing the need for user-specific parameters.
Comparative Analysis. Following the assumptions that user can provide their own pref- erences as prompts to LLM, (Lee al., 2024b) train an instruction-folloiing LLM that can adapt to user-written values in system prompt via data synthesis, (Wang et al., 2024b) builds base models geared towards ilm-as-a-judge (Zheng et al., 2023) use-cases that can adapt to explicitly sta ted adaptability can also be built into reward models. For example, (Pitis et al., 2024) builds context-conditione reward models, which are then used for personalized (i.e., context- conditioned) alignment. Similarly, (Kumar et al., 2024b) builds reward models conditioned on user group identifiers Reddit to achieve personalization. We note that there are works that attempt to infer latent user representations when users don't 't explicitly state their pref- erence, but still build steerable models based on these latent representations. For example, (Balepur et al., 2025) infer user persona from choices using LLMs, and train model to adapt to those personas. In contrast, (Chen et al., 2024b) adopts a plurality-baser approach, using ideal point and mixture modeling to learn common latent preference space that generalizes to new users, Finally, (Poddar al., 2024) also learns user latent factors, but instead opts to use user-conditional reward models to achieve personalization.
personas. Aside from building steerable LLM policies, such
Limitations. While alleviating the need for maintaining user-specific parameters, this class of methods often depends user such personas and verbal preferences, which are unavailable in standard RLHF settings. Further, collecting datasets with diverse user personas itself is poses challenge, which we discuss extensively in Section 6.
5

---

Published as a conference paper at COLM 2025
Sec 5.2
Sec 5.1
RewarddVaalue Model
Slwward//aluu-GGulloe Decoding
Sec 5.3
Prompting- based Alighment
Porsonslized Preferences
Prompts
amail model & An untured model
Loght Rectification and Re-Alignmeet
Figure 4: P'ersonalized language model alignment during test time.
5
Test-time Personalized Alignment
The growing need for language models that can adapt on the fly to diverse user prefer- ences- ences-while keeping compute costs manageable and simplifying deployment-har spurred variety of test-time alignment techniques. These methods adjust the decoding process, without altering the underlying model parameters, to steer outputs toward desired behav- iors and make debugging easier. We broadly categorize these approaches into three groups: prompting and context optimization, reward- and value-guided decoding methods, and rectification-based decoding and correction.
5.1 Prompting-basee Alignment Methods
Motivation. Prompting- based approaches modify the input context-via in-context exam- ples, retrieval, or prompt rewriting to elicit behavior that mirrors specific user preferences. This training-fres free strategy is particularly attractive for personalization, as it can quickly adapt to new or evolving user profiles (see Figure 4).
Comparative Analysis. A central challenge for prompting baased methods is achieving personalized alignment without the need for full-scale fine-tuning. (Lin et al., 2023) and (Li et al., 2023c) propose methods that harness a small number of stylistic examples or self-correction mechanisms to nudge LLMs toward outputs that align with individual user styles. Similarly, (Xu et al., 2023) dynamically incorporate external memories to retrieve rules tailored to diverse social norms, while (Salemi et al., 2024) and (Mysore et al., 2023a) further personalize responses by retrieving user-specific items to augment the prompt. In parallel, prompt optimization techniques by (Kim & Yang, 2024), (Cheng et al., 2023a), and (Li et 2024b) iteratively refine prompts based on misaligned outputs, thereby progressively incorporating user feedback.
Limitations. Despite their promise for personalization, these methods are limited by the fixed context length and increased computational overhead when handling complex, dynamic user profiles. Additionally, the reliance on relatively static representations of user preferences can hinder real-time adaptation to rapidly changing individual needs.
5.2 Reward and Value-Guided Decoding Methods
Motivation. Reward and value-guided decoding methods integrate personalized alignment objectives directly into the LLM decoding by adjusting token probabilities based on reward signals value functions. This approach supports fine-grained, token-level personalization and enables LLMs to adapt their outputs on the fly to meet individual user preferences.
Comparative Analysis. One of the major challenges is balancing high-reward personalized output with maintaining natural language fluency. (Khanov et al., 2024) and (Deng & Raffel, 2023) propose frameworks that incorporate reward signals into the decoding process, which can be tuned to reflect personalized quality metrics. Building on this, (Li et al., 2024a) and (Huang et al., 2024a) introduce strategies that break the generation process

---

Published as a conference paper at COLM 2025
into segments, allowing for iterative refinement toward personalized and fluent outputs. Complementing these reward-guided methods, (Xu et al., 2024) and (Chen et al., 2024e) focus on scalable, test-time alignment by employing autoregressive reward models and personalized reward modeling, respectively. In parallel, value-guided techniques offer another path to personalization: (Mudgal et al., 2023) and (Liu et al., 2024b) integrate auxiliary value functions during decoding, whereas (Liu et al., 2024d) and (Han et al., 2024) demonstrate that combining implicit and explicit value guidance can further enhance the model's s responsiveness with respect to users' personalized preferences and values.
Limitations. A key limitation of reward and value-guided methods is their reliance on pre- trained reward or value models, which introduces challenges such as limited transparency and sensitivity adversarial inputs. Additionally, the need for real-time evaluation of reward signals and value functions often results in significant computational overhead, making it challenging to scale these methods for high-throughput, low-latency applications.
5.3 Logit Rectification and Re-Alignment Approaches
Motivation. Logit rectification approaches modify the internal decision process of LLMs by integrating corrective signals- from smaller, fine-tuned models or auxiliary mod- direcdl into the decoding stage. This strategy enables personalized re-alignment without retraining the entire model, thereby offering a lightweight path to tailor outputs to individual user needs.
Comparative Analysis. A key challenge in this category is to harness the complementary strengths of large pretrained models and smaller, aligned models. (Mitchell al., 2023) introduce emulated fine-tuning, which decouples the contributions of pre-training and fine-tuning by combining the knowledge of large model with the behavioral adjustments of a small model; special case, LM up-scaling, demonstrates that ensembling can emulate the effects of full fine-tuning without additional training. In a similar vein, (Liu et al., 2024a) propose
lightweight algorithm that steers the output distribution of a large black-box model using the difference between the predictions of a tuned small model and its untuned counterpart. Addressing the need for multi-objective personalization, (Shi et al., 2024) develop a method that computes a linear combination of predictions from several base models, enabling flexible adjustment of competing objectives, while (Liu et al., 2024c) propose decoding- -omeing-immeeeee re-alignment to dynamically control the degree of alignment without retraining. Complementing these strategies, (Ji et al., 2024) offers a model-agnostic correction approach that learns residuals to refine outputs, and (Yang et al., 2024) extends this idea with a generalizable framework that supports multi-objective and personalized alignment by adjusting target objectives via prompt updates. Finally, (Zhang et al., 2025) frame re-alignment as an online per-token optimization problem with a closed-fone solution, achieving real-time adaptation to diverse and evolving user preferences.
approaches are highly dependent on the quality and consistency of the auxiliary correction signals. Their effectiveness can be limited if the small, tuned models do not capture the full complexity of user-specific nuances, which may lead to mismatch between the corrective adjustments and the large model's inherent output tendencies. Additionally, ensuring robust performance across a wide range of personalization scenarios without introducing extra latency remains practical challenge.
Limitations. Rectification-bass
6 Dataset & Evaluation
Datasets. Early attempts on constructing datasets for personalized preference alignment typically rely on data synthesization, with some recent attempts in collecting real-world data for preference alignment. We provide an overview of relevant benchmarks and evaluations in Table 1. For example, (Cheng et al., 2023b) introduce the Domain-Specifie Preference (DSP) dataset by augmenting the Alpaca instruction corpus (Taori et al., 2023) with answers targeting different domain (e.g., Academia or Entertainment). (Jang et al., 2023) introduce Personalized Soups (P-SOUP), where Alpaca instruction prompts (Taori et al., 2023) are answered by an LLM in contrasting styles along three dimensions (expertise level, ver-
7

---

Published as a conference paper at COLM 2025
Reference
Task/Data
Metric (Dimension)
(Cheng 20236)
Prompt selection
PPA (Domain: Acad. Bus, & Art)
Tailored prompts
P-SOUPP [ang 2025)
Pairwise feedback
PWR (Expertise, Info, Friendliness)
GPT4 simulated
MULTIFACETED (Lee al., 202
Preference alignmmen
AAS (Human pref.)
Diverse preferences
HH--LHHF (Bai et al, 2022)
RLHF datoset
CRS (Helpfulness, Harmlesseess)
Personalized dimersions
HelpSteer2 (Wang et 20244)
RLFF feedback
MPS (Helpfulness, Harmlesmnes,, Humor)
Humer-erhanced
PRISM (Kirk et 2114h)
Demographic alignment
AFI (Fairess across demogrophies)
Participalory alignment
LaMP (Salemi et 202.4)
Pensonalized generation
POQ (User-pndfe profile retrieval)
Retrieval- Sarievva---ggmmeeeee
LongLaMP (Kumar al., 2024a)
Personalized long generation
(Usee profile retnieval)
Retrseval augmented
PersonalLLM (Zolloo et 2024)
Personalization benchmark
PAS (Indavidual prefs.)
Beyond alignment
ALOF (Wu al., 2024c)
Multi-turn dialogues
PCS (Persona consistency)
Persona specilic dalaset
PersoBench (Afzoom et 2024)
Persona-aware dialogue
PAA (Personn awaremess)
Zerorshot evaluation
Table 1: Compressed overview of datasets and benchmarks for personalized alignment and generation.
bosity, and tone) and ranked based on predefined style preferences. On a much larger scale, (Lee et al., 2024b) constructed the Multifaceted Collection dataset, containing about 65k instructions, each paired with three -LM--genrated responses that vary along thousands of dimensions. Finally, (Zollo et al., 2024) present PersonalLLM, an open benchmark that generate preference rankings from a set of simulated synthetic "users". In parallel, other recent works have assembled personalized human preference datasets to study person- alization with real users. The PRISM Alignment dataset (Kirk et al., 2024b) is a notable example, recording 8,011 live chat interactions between 1,500 users across 75 countries, with each user's 's persona profile and preference feedbacks (ratings), providing a complementary setting for evaluating personalized preference alignment.
Evaluation in Personalized Alignment. Evaluating personalization in LLM alignment requires measuring how well model's output matches the particular preferences of user. When the evaluation dataset includes response variations across known dimensions (e.g., style, tone), evaluation often uses nulti-dimensional scoring or pairwise comparisons. For example, (Jang al., 2023) assess their proposed methods by scoring responses along each dimension separately, but assumes different users place different value on the set of dimensions. Another popular evaluation approach is employ LLM-as-a-judge Zheng et al. (2023), but prompt the judge LLMs with pre-defined user personas (Wu et al., 2024c; Lee et al., 2024b). This evaluation formulation leads to the creation of specialized evaluation models such as PerSE (Wang et al., 2024b), a 13B Llama-2 based evaluator fine-tuned to judge alignment with personal profiles. Overall, a unique open challenge in personalized align- ment evaluation realistically simulating personalized preference judgment (Zheng al., 2023; Wu et al., 2024c; Lee et al., 2024b; Wang et al., 2024b), calling for further investigations.
Limitations. Despite progress on model evaluations catered for personalized preference alignment, most assessments rely heavily rule-based metrics or persona-based LLM-as-a- judge Zheng et al. (2023), yielding simulations of user satisfaction. Since these heuristics and personas for evaluations are use-case specific, there are currently no unified evalu- ation across studies, posing challenge for systematic evaluation and progress tracking. As the research community moves toward widely accepted multidimensional preference benchmarks and the development of publicly available evaluators, more consistent and comparable metrics are expected to emerge.
7 On Personalized User Modeling and Personalized Preference Alignment
Why this section? As discussed previously in Section 2, personalized preference alignment is subset of personalization research for LLMs. Specifically, there is another emerging research area that focuses on LLM-based user modeling Tan & Jiang (2023), building LLM- based simulations for individual users, which most commonly manifest as directly predicting users' responses. While prior surveys do not differentiate between personalized user modeling and preference alignment (Zhang et al., 2024b; Wu et al., 2024a; Tseng et al., 2024), we note that these are two distinct and complementary research directions. To this end, this section serves two purposes. First, clearly distinguish between the two
8

---

Published as a conference paper at COLM 2025
research directions currently studied under LLMs and personalization, making it easier for future researchers to sift through relevant literature. Second, we discuss various ways user modeling can enable better-personalized preference alignment.
Personalized preference alignment research benefits from personalized user modeling. Due to the difficulty in collecting large-scale feedbacks from real users, current research for personalized preference aligmment relies heavily on user simulation. For example, various recently proposed datasets rely on persona-grounded simulation with LLMs to build benchmarks (Cheng et al., 2023b; Jang et al., 2023; Lee et al., 2024b; Zollo et al., 2024) or evaluations (Zheng et al., 2023; Wu et al., 2024c; Lee et al., 2024b; Wang et al., 2024b) for personalized preference methods. To this end, better simulation of diverse, realistic user behaviors naturally improves the development of better personalized alignment methods.
Modeling individual user (sometimes) enables preference-free personalized alignment. Meanwhile, by modeling individual users, system deployers can still increase user satis- faction, even without directly modeling user preferences. For example, in areas such as personalized review prediction (Xie et al., 2023; Ni al., 2019) and recommender system (Wu et al., 2024b), user behavior happens to strongly correlate with user preferences, and thus better prediction of user behavior directly improves preference alignment. Similarly, related research directions such as LLM-based chit-chat dialogue systems with personalized user memory (e.g., (Yuan et al., 2025)) also better caters to the preference of individual users by remembering personal facts, even when there are no explicit modeling of user preferences. Finally, simulating specific desired personas such as teachers (Wang et al., 2024c), thera- pists (Stade et al., 2024), and travel-planners (Chen al., 2024a) also enables LLMs to better cater to the corresponding user groups, such as students, patients, and travelers.
8 Future Works and Emerging Directions
Online and Continuous Personalized Alignment While existing work on personalized preference alignment primarily explores the setting of learning user preference from offline data or given explicitly stated user preference, another complementary setting is person- alized L.LM alignment in an online setting (Chen et al., 2024f). Additionally, given prior success in adjacent research (such as user modeling) on continuous personalization over multiple dialogue sessions (Li et al., 2024c; Zhang et al., 2023; Zhong et al., 2024; Qian et al., 2024), personalized preference alignment in a multi-session dialogue setting is natural extension, which typically models turn-wise or user-provided preference feedback. Nevertheless, recent work Zhao et al. (2025) show large language models frequently fail to recall user preference in continuous personalization setting, calling for more elaborate solution for personalization of LLMs over time.
Addressing long and complex user-generated value statements As discussed in prior sections, personalized preference alignment in LLMs frequently relies instruction follow- ing ability of LLMs as building blocks for alignment methods, both at training time and inference time. However, recent works show long and complex instruction-following is still an open challenge (Wu et al., 2024d; Gavin et al., 2024). Given the prevalence of per- sonalized alignment methods that rely on explicit verbal preference statements (Section 4.2 and Section 5.1), it is still unclear whether existing methods can support complex and long user value statements. To this end, developing benchmarks and methods to further research the cnatructtoo-followinn ability of LLMs on complex user preference value statements can help LLM-based dialogue systems better handle rich, multifaceted user preferences.
Conclusions
In this survey, we perform a comprehensive analysis of existing methods, datasets, and benchmarks for personalized preference alignment in LLMs and LLM-based dialogue systems. We discuss various classes of methods and their advantages and drawbacks, covering both training and inference-time, well as user-knowledge-Dol preference alignment methods. We also discuss limitations and future directions for LLMs to cater to diverse and individualistic preferences.
personalized

---

Published as a conference paper at COLM 2025
References
Saleh Afzoon, Usman Naseem, Amin Beheshti, and Zahra Jamali. Persobench: Bench- marking personalized response generation in large language models. arXiv preprint arXiv:2410.03119, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield- Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https: 05862.
org/abs/204..
Nishant Balepur, Matthew Shu, Alexander Hoyle, Alison Robey, Shi Feng, Seraphina Goldfarb-Tarre and Jordan Lee Boyd-Graber. SMART mnemonic sounds like "glue tonic": Mixing LLMs with student feedback to make mnemonic learning stick. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 14202-14225, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. (0.18853/vl emmlp-main.786. URL https::/accien https://aallanthology wttg/2224.mnnpmmmma........ emnlp-main. 786/
Nishant Balepur, Vishakh Padmakumar, Fumeng Yang, Shi Feng, Rachel Rudinger, and Jor- dan Lee Boyd-Graber. Whose boat does it float? improving personalization in preference tuning via inferred user personas, Xio preprint arXiv:2501.11549 2025.
Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. ar preprint (ix::2446600045 2024.
Aili Chen, Xuyang Ge, Ziquan Fu, Yanghua Xiao, and Jiangjie Chen. Travelagent: An ai assistant for personalized travel planning, 2024a. URL httpp:://arxi..oo///aaaaaa 08069.
Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Korlakai Vinayak. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. aiv preprint ar arXio:2406.08466 2024b.
Jin Chen, Zheng Liu, Xu Huang, Chenwang Wu, Qi Liu, Gangwei Jiang, Yuanhao Pu, Yuxuan Lei, Xiaolong Chen, Xingmei Wang, et al. When large language models meet personalization: Perspectives of challenges and opportunities. World Wide Web, 27(4)442, 2024c.
Nuo Chen, Yan Wang, Yang Deng, and Jia Li. The oscars of ai theater: A survey role-playing with language models, 2024d. URL https: htttp::/arxiv.orrr wwtp:/vv..og/bb///0/00/ 11484.
Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. Pad: Personalized
alignment at decoding-ting time. arXiv preprint arXiv:2410.04070, 2024e.
Zekai Chen, Weeden Daniel, Po yu Chen, and Francois Buet-Golfouse, Online personalizing white-box lims generation with neural bandits, 2024f. URL httpss:///xxxvv.or//aaa. /ttps//Rtttt////wwwwoo////////// 16115.
Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. Black-box prompt optimization: Aligning large language models without model training. Xiv preprint ar Xroo:2311.044155 2023a.
Pengyu Cheng, Jiawen Xie, Ke Bai, Yong Dai, and Nan Du. Everyone deserves a reward: Learning customized human preferences. ar preprint arXiv:2309.03122 2023b.
Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation with unidirectional reward model. arxiv preprint arXio:2310.09520, 2023.
10

---

Published as a conference paper at COLM 2025
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. The second conversational intelligence challenge (convai2), 2019. URL http:://arxiv.or//bbbb... 000000000000000 00098.
Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language models, 2024. URL https://arxivv org/abs 2306. 16388.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Confer- ence Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5988- 6008. PMLR, 17-23 Jul 2022. URL https:ts//// http::://proceeeee http://poooceennnn///e/ waweeeddiinnsm...d.......m. www..ppressssvviv 62/ /ttess//w2//tthhaaaaah22222 html.
Lucie Flek. Returning the N to NLP: Towards contextually personalized classification models. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7828-7838, Online, July 2020. Association for Computational Linguistics. doi: 10.16653/v1/2020. acl-main.760. URL https: https:///allnnnnhoooog org/ 2020. acl main.
Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjie Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, et al. Linear alignment: A closed-form solution for aligning human preferences without tuning and feedback. Xrviv preprint Xiiv:2401.11458 2024.
Shawn Gavin, Tuney Zheng, Jiaheng Liu, Quehry Que, Noah Wang, Jian Yang, Chenchen Zhang, Wenhao Huang, Wenhu Chen, and Ge Zhang, Longins: A challenging long-context instruction-based exam for llms, 2024. URL https://arxiv.orrr org/abs///////// 2406 17588.
Anmol Goel, Yaxi Hu, Iryna Gurevych, and Amartya Sanyal. Differentially private steering for large language model alignment. In The Thirleenth International Conference on Learning Representations, 2025. URL https: //tpnreveeeeeeeiie..... www.eewweee/foooe met//forumidd_Lkkg]EEE.
Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization. ar arxiv preprint 7XXi=2245.06633 2024.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. Con- trollable text generation. CoRR, abs/ 0tts//700.0095,, 2017. URL http://arxxxv wwwpiiiooorg/aaa org//bs//703.. 00955.
James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. Deal: Decoding-time alignment for large language models. Xiv preprint arXiv:2402.06147 2024a.
Qiushi Huang, Xubo Liu, Tom Ko, Bo Wu, Wenwu Wang, Yu Zhang, and Lilian Tang, Selective prompting tuning for personalized conversations with llms. arxio preprint arXiv:2406.18187, 2024b.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging, ar preprint ar Xix::2310.11564. 2023.
Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety aligmment of ll via numan-preference dataset, 2023. URL https::/arxine http://rrxii.oor/////////// 04657..
11

---

Published as a conference paper at COLM 2025
Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Tianyi Alex Qiu, Juntao Dai, and Yaodong Yang. Aligner: Efficient aligmment by learning to correct. Advances in Neural Information Processing Systems, 37:90853-90899, 2024.
Feibo Jiang, Li Dong, Siwei Tu, Yubo Peng, Kezhi Wang, Kun Yang, Cunhua Pan, and Dusit Niyato. Personalized wireless federated learning for large language models, 2024. URL https://arxivv https://arxv..og//bbb//444.4 13238.
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search. arXiv preprint arXio:2402.01694 2024.
Jaehyung Kim and Yiming Yang. Few-shot personalization of lims with mis-aligned re- sponses. Xiv preprint Xix:::2406. 186000618178.... 2024.
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, pp- 1-10, 2024a.
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. arXio preprint Xio::444 16019, 2024b.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023. URL https://arxiv. /rgg /rrg/bb//2// 2305.01569.
Lingkai Kong, Haorui Wang, Wenhao Mu, Yuanqi Du, Yuchen Zhuang, Yifei Zhou, Yue Song, Rongzhi Zhang, Kai Wang, and Chao Zhang. Aligning large language models with representation editing: A control perspective. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL yTTomSJsSW.
net
Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, et al. LongLaMP: A benchmark for personalized long-form text generation. arXiv preprint arXie:2407.11016, 2024a.
Sachin Kumar, Chan Young Park, Yulia Tsvetkov, Noah A Smith, and Hannaneh Hajishirzi. Compo: Community preferences for language model personalization. arXiv preprint arXie:2410.16007, 2024b.
Gihun Lee, Minchan Jeong, Yujin Kim, Hojung Jung, Jaehoon Oh, Sangmook Kim, and Se-Young Yun. Bapo: Base-anchored preference optimization for overcoming forgetting in large language models personalization, 2024a. URL
http:/wwww..g////////.////// 006933
Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. Aligning to thousands of preferences via system message generalization. arXiv preprint 4rXiv:2405.17977, 2024b.
Bolian Li, Yifan Wang, Ananth Grama, and Ruqi Zhang. Cascade reward sampling for efficient decoding time alignment. Xrviv preprint ar Xii::2406.16366, 2024a.
Cheng Li, Mingyang Zhang, Qiaozhu Mei, Yaqing Wang, Spurthi Amba Hombaiah, Yi Liang, and Michael Bendersky. Teach llms to personalize an approach inspired by writing education, 2023a. URL https: http::://axxvvoooo org/ / .0w.//2.888007..8. 07968.
Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, and Michael Bendersky. Learning to rewrite prompts for personalized text generation. In Proceedings of the ACM Web Conference 2024, pp. 3367-3378, 2024b.
Hao Li, Chenghao Yang, An Zhang, Yang Deng, Xiang Wang, and Tat-Seng Chua. Hello again! loweeed personalized agent for long-term dialogue. arXio preprint ar arXit:2406.0592. 059225, 2024c.
12

---

Published as a conference paper at COLM 2025
Xinyu Li, Zachary C Lipton, and Liu Leqi. Personalized language modeling from personal- ized human feedback. arxio preprint arXio:2402.05113, 2024d.
Yang Li, Kangbo Liu, Ranjan Satapathy, Suhang Wang, and Erik Cambria. Recent devel-
opments in recommender systems: A survey, 2023b. URL https://anxiv.org/abb//// 12680.
Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal 1lm agents: Insights and survey about the capability, efficiency and security. arviv preprint arXiv:2401.05455 2024e.
Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your
language models can align themselves without finetuning. arXiv preprint ar arXiv:2309.07124 2023c.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethink- ing alignment via in-context learning. In The Twelfth International Conference on Learning Representations, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL
abs/aass/9000 0209.07958...... 07958.
Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Isvetkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy. Xrv preprint arXiv:2401.08565 2024a.
Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Don't 't throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding. In First Conference on Language Modeling, 2024b.
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding- time realignment of language models. Xiv preprint arXiv:2402.02992, 2024c
Zhixuan Liu, Zhanhui Zhou, Yuanfu Wang, Chao Yang, and Yu Qiao. Inference-time language model aligmment via integrated value guidance. ar Xiv preprint arXiv:2409.17819 2024d.
Xiangwu Meng, Yulu Du, Yujie Zhang, and Xiaofeng Han. A survey of context-aware recommender systems: From an evaluation perspective. IEEE Transactions on Knowledge and Data Engineering, 35(7)x6575-6594 2023. doi: (0.1109/TKDE..02223184555
Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning, An emulator for fine-tuning large language models using small language models. arXiv preprint ar Xrxi::23100122 12962,, 2023.
Xinyi Mou, Xuanwen Ding, Qi He, Liang Wang, Jingcong Liang, Xinnong Zhang, Libo Sun, Jiayu Lin, Jie Zhou, Xuanjing Huang, and Zhongyu Wei. From individual to society: A survey on social simulation driven by large language model-based agents, 2024. URL https://arxiv wttp://..wr///////////// 03563.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arv preprint ar arXivv2310.1700 2023.
Sheshera Mysore, Zhuoran Lu, Mengting Wan, Longgi Yang, Steve Menezes, Tina Baghaee, Emmanuel Barajas Gonzalez, Jennifer Neville, and Tara Safavi. Pearl: Personalizing large language model writing assistants with generation-calibaatee retrievers. Xrviv preprint arX1v:2311.09180 2023a.
Sheshera Mysore, Andrew McCallum, and Hamed Zamani. Large language model aug- mented narrative driven recommendations, 2023b. URL 02250.
13

---

Published as a conference paper at COLM 2025
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- labeled reviews and fine-grained aspects. Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)) pp. 188-197, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18633vv1/D13 org/D19-1018/.
Empirical Methods in Natural
URL https: http:///ccaanhhloogyyy
Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, and Asuman E Ozdaglar. Rlhf from heterogeneous feedback via personalization and preference aggregation, In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024a.
Chanwoo Park, Mingyang Liu, Kaiqing Zhang, and Asuman Ozdaglar. Principled rlhí from heterogeneous feedback via personalization and preference aggregation. arv preprint a/Xit:245500025, 2024b.
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lu- cas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam Mc- Candlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield- Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Her- nandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discov- ering language model behaviors with model-written evaluations. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Com- putational Linguistics: ACL 2023, pp. 13387-13434, Toronto, Canada, July 2023, Asso- ciation for Computational Linguistics. doi: 10.18653. / w122022.2222// w1///23.inniingsaalll findings--ccl.887, URL https://aclantiolooy.or//////
findings-ache acl.847/
Silviu Pitis, Ziang Xiao, Nicolas Le Roux, and Alessandro Sordoni. Improving context-aware preference modeling for language models. ar Xiv preprint ar Xiv::2440 00.244077.4966, 2024.
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. arxiv preprint ar Xiw:2408.10077 2024.
Hongjin Qian, Peitian Zhang, Zheng Liu, Kelong Mao, and Zhicheng Dou. Memorag: Moving towards next-gen rag via memory-inspired knowledge discovery. ar preprint ar XXX5:2409.00591, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In seirhh Conference on Neural Information Processing Systems, 2023. URL https://openreviiet net mett/orrmmid--HuSSe
Alireza Salemi, Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. LaMP: When large language models meet personalization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 7370-7392, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. v118553/1/2224.ac0/ https://aclanthology.. wwwp:////ooy.og//22o/..g acl acl-long......... 399.
399. URL
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect?, 2023. URL https:/ //rxxv..xvvv org/abs/2303.
17545. 17548.
14

---

Published as a conference paper at COLM 2025
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: survey, 2023. URL nttts:///rxiv..r
15025.
Ruizhe Shi, Yifang Chen, Yushi Hu, ALisa Liu, Noah Smith, Hannaneh Hajishirzi, and Simon Du. Decoding-time language model alignment with multiple objectives. arviv preprint ar Xiw::400611885 2024.
Feifan Song, Yuxuan Fan, Xin Zhang, Peiyi Wang, and Houfeng Wang. Instantly learning preference alignment via in-context DPO. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the As- sociation for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa- pers), pp. 161-178, Albuquerque, New Mexico, April 2025. Association for Computa- tional Linguistics. ISBN 979-8-89176-188 https://aclanthoologg org/ maacl-long.80 8/.
doi: 10.18653/ (1186633v11//0 2t1//225.naacllloo.
URL
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. A roadmap to pluralistic alignment, 2024. URL https://arxiv. org/abs/2402. 000000000200000000 05070.
Elizabeth C. Stade, Shannon Wiltsey Stirman, Lyle H. Ungar, Cody L. Boland, H. A. Schwartz, David Bryce Yaden, João Sedoc, Robert J. DeRubeis, Robb Willer, and Johannes C. Eich- staedt. Large language models could change the future of behavioral healthcare: proposal for responsible development and evaluation. NPJ Mental Health Research, 3, 2024. URL hnttps://apt semanticscholar wrwpoaaarrrr/oouusSD/D1 26888142 423..
Zhaoxuan Tan and Meng Jiang. User modeling in the era of large language models: Current research and future directions. Xrv preprint arXiv:2312.11151 2023.
Zhaoxuan Tan, Zheyuan Liu, and Meng Jiang. Personalized pieces: Efficient personal- ized large language models through collaborative efforts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 6459-6475, Miami, Florida, USA, November 2024a. URL
Association for Computational Linguistics. doi: 10. https://aclanthol./
org/2024, emnlp-main 3
Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. Democ- ratizing large language models via personalized pfrfieter-efficient fine-tuning. ArXiv, abs/2402.04401, 2024b. URL https://app. semanticscholar. www.llaarrrrrr/rraor/eeeee wrgg/CrppssDDD... wtg//orrpsID::275222233 267523232.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://g/ thub. com comm/tatsu-lab/ tattssuulabsstaffrrrrr alpaca, 2023.
Yu-Min Tseng, Yu-Chao Huang, Teng-Yur Yun Hsiao, Yu-Ching Hsu, Jia-Yin Foo, Chao-Wed Huang, and Yun-Nung Chen. Two tales of persona in Ilms: A survey of role-playing and personalization. arXiv preprint arXiv:2406.01177 2024.
Danqing Wang, Kevin Yang, Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuan- dong Tian. Learning personalized alignment for evaluating open-ended text generation. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, Pp. 13274-13292, 2024a.
Danqing Wang, Kevin Yang, Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuan- dong Tian. Learning personalized alignment for evaluating open-ended text generation, 2024b. URL httpp:://arxiv org/abs//311 03304.
Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S. Yu, and Qingsong Wen. Large language models for education: A survey and outlook, 2024c. URL https://arxiv.. httpv/.org//bs//40.00 18800.
18105.
15

---

Published as a conference paper at COLM 2025
Yifan Wang, Weizhi Ma, Min Zhang, Yiqun Liu, and Shaoping Ma. A survey the fairness of recommender systems. ACM Trans. Inf. Syst. 41(3), February 2023. ISSN 1046-8188. doi: 10.1145 URL https://doi. http::///oooorr/100 1145/3547333.
Zekun Moore Wang, Shenzhi Wang, King Zhu, Jiaheng Liu, Ke Xu, Jie Fu, Wangchunshu Zhou, and Wenhao Huang. PopAlign: Diversifying contrasting patterns for a more comprehensive alignment. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 28893 28921,, Vienna, Austria, July 2025, Association for Computational Linguistics. ISBN 979-8-89176-251-6. doi: 10. 18653/v1 1tt///2225clllllaa lowaaclooo...000 URL https: htttt::://aclaathhologg.g
org/2025.22. attp:/222..ccl1......... 1 1403/.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint ar arXio:2406.08673, 2024d.
Junda Wu, Hanjia Lyu, Yu Xia, Zhehao Zhang, Joe Barrow, Tshita Kumar, Mehrnoosh Mirtaheri, Hongjie Chen, Ryan A. Rossi, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Namyong Park, Sungchul Kim, Huanrui Yang, Subrata Mitra, Zhengmian Hu, Nedim Lipka, Dang Nguyen, Yue Zhao, Jiebo Luo, and Julian McAuley. Personalized multimodal large language models: A survey, 2024a. URL
2222..
Junkang Wu, Kexin Huang, Xue Wang, Jinyang Gao, Bolin Ding, Jiancan Wu, Xiangnan He, and Xiang Wang. Repo: Relu-based preference optimization, 2025. URL https: //arxiv. .rgg alt.a//5...74444
Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen. A survey large language models for recommendation, 2024b. URL https: wwt....oog/bb////...... 19860.
Shujin Wu, May Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. Aligning iims with individual preferences via interaction. arXiv preprint arXiv:2410.03642. 2024c.
Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Xiangju Lu, Junmin Zhu, and Wei Zhang. Lifbench: Evaluating the instruction following performance and stability of large language models in long-context scenarios, 2024d. URL https://arxii wwwvvi.org/..oo org/ /bss/ 2411. 07037.
Zhouhang Xie, Sameer Singh, Julian McAuley, and Bodhisattwa Prasad Majumder. Factual and informative review generation for explainable recommendation. In Proceedings of the Thirly-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference Innovative Applications of Arlificial Intelligence and Thirleenth Symposium on Educational Advances in Artificial Intelligence, AAAI'23 LAAAAAAAA//////// AAAI/2/1AAA233/ AAA13//EAAA2/ EAAI'23, AAAI Press, 2023, ISBN URL https: //tp::/rg//o////////// 1609/1601 978-1-57735-88000. doi: 10.1609 haal//aaaiv33 v3711.1277111.66111
Chunpu Xu, Steffi Chern, Ethan Chern, Ge Zhang, Zekun Wang, Ruibo Liu, Jing Li, Jie Fu, and Pengfei Liu. Align on the fly: Adapting chatbot behavior to established norms. Xiv preprint ar Xiv:2312. 15907, 2023.
Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. Genarm: Reward guided generation with autoregressive reward model for test-time alignment. arv preprint ar Xii::2410.08193, 2024.
Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Tianlin Zhang, and Sophia Ananiadou. Metaaligner: Towards generalizable multi-objective alignment of language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Ruifeng Yuan, Shichao Sun, Yongqi Li, Zili Wang, Ziqiang Cao, and Wenjie Li. Personalized large language model assistant with evolving conditional memory. In Owen Rambow,
16

---

Published as a conference paper at COLM 2025
Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguistics, pp. 3764-3777, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL htl.httts///claanhollog
wwwgg/225.ccom cow.soo-annnnnnn....... main.254/.
Kai Zhang, Yangyang Kang, Fubang Zhao, and Xiaozhong Liu. LIm-based medical as- sistant personalization with short-and long-term memory coordination. arviv preprint ar arXit:2309.11696, 2023.
Tong Zhang, Yong Liu, Boyang Li, Zhiwei Zeng, Pengwei Wang, Yuan You, Chunyan Miao, and Lizhen Cui. History-aware hierarchical transformer for multi-session open-domain dialogue system. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Find- ings of the Association for Computational Linguistics: EMNLP 2022, pp. 3395-3407, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10. 10.18653/ / v /202 rindings-emnlppp.... 2477. URL https::/acclatthollgg/ org/2022 findings-emnlp. 247/
Yicheng Zhang, Zhen Qin, Zhaomin Wu, and Shuiguang Deng. Personalized federated fine-tuning for Ilms via data-driven heterogeneous model architectures. arxiv preprint a/Xit:2411.19128, 2024a.
Zhaowei Zhang, Fengshuo Bai, Qizhi Chen, Chengdong Ma, Mingzhi Wang, Haoran Sun, Zilong Zheng, and Yaodong Yang. Amulet: Realignment during test time for personalized preference adaptation of LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https: wtwppppreeeeeeeeeeooorrrrnreeffe
Zhehao Zhang, Ryan A. Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, Ruiyi Zhang, Jiuxiang Gu, Tyler Derr, Hongjee Chen, Junda Wu, Xiang Chen, Zichao Wang, Subrata Mitra, Nedim Lipka, Nesreen Ahmed, and Yu Wang. Personalization of large language models: A survey, 2024b. URL https: /wwpvv..avvvv.a////a//e..e..aa www.:/rgg///////e 2b51 00027.
Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do LLMs recognize your preferences? evaluating personalized preference following in LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview. net met:FoormmidddNunnnn.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large language models, 2024. URL https://arre http:://arxiv.org/bbbs...
18223.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Dalasets and Benchmarks Track, 2023.
URL / http::///pppereeeeeeeeef/ffueree
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhanc- ing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19724-19731, 2024.
Jin Peng Zhou, Katie Z Luo, Jingwen Gu, Jason Yuan, Kilian Q Weinberger, and Wen Sun. Orchestrating 1lms with different personalizations. arXio preprint ar axXiv:240.004188 2024.
Hanlin Zhu, Andrew Cohen, Danqing Wang, Kevin Yang, Xiaomeng Yang, Jiantao Jiao, and Yuandong Tian. End-to-end story plot generator, 2023. URL
08796..
Yuchen Zhuang, Haotian Sun, Yue Yu, Rushi Qiang, Qifan Wang, Chao Zhang, and Bo Dai. Hydra: Model factorization framework for black-box Ilm personalization. ArXiv, 0tt//240600888 2024. URL https: ptttps://api. //api. semanticscholar
17

---

Published as a conference paper at COLM 2025
Thomas P Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. Personalllm: Tailoring ilms to individual preferences. arXiv preprint wix::2449.202990 2024.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. URL http://awiii..oorbbb/////// wttg/bb/////..../// 15043.
https://arxiv.
A
On Risk of Pluralistic Preference Alignment
We note that as discussed in prior works (Kirk al., 2024a), naively personalizing LLMs to individualized preference can raise other issues such as amplifying bias or inducing polarization. To this end, similar to adjacent domain in personalization such as recommender system Wang et al. (2023), ensuring safety and fairness in pluralistic alignment remain an continuously important and open challenge. Meanwhile, pluralistic alignment generally requires collecting diverse preference signals from users, calling for privacy-aware method such differentially private LLM aligmment Goel et al. (2025).
B
On Datasets and Benchmarks for Pluralistic Alignment
Unlike recent work on general preference alignment for LLMs-whidh spans training and inference time methods, inter alia (Rafailov et al., 2023; Wang et al., 2025; Wu et al., 2025; Kong et al., 2024; Song et al., 2025), a unique ongoing challenge in the field is the lack of commonly acknowledged benchmarks and evaluation methods. To this end, we show few recent works and their evaluation datasets in Table 2, and hope this makes accessing relevant dataset easier and facilitate comparisons on shared benchmarks.
Method
Dataset
RAG/PAG (Salemi et al., 2024)
LAMP (Salemi et al., 2024)
OPPU (Tan et al., 2024b)
LAMP (Salemi et al., 2024)
Personalized-PCs (Tan al., 2024a)
LAMP (Salemi et al., 2024)
HyDRA (Zhuang et al., 2024)
LAMP (Salemi al., 2024)
P-DPO (Li et al., 2024d)
Prism Alignment Kirk et al. (2024b)
BiPO (Cao et al., 2024)
AI Persona (Perez et al., 2023), TruthfulQA (Lin et al.,
2022), ADVBench (Zou al., 2023)
SPT (Huang et al., 2024b)
ConvA12 (Dinan al., 2019)
System Message Gen. (Lee et al., 2024b)
Multi-faceted Benchmark and other helpful-
ness/ hass/hamllessness benchmarks (Lee et al., 2024b)
PerSE (Wang et al., 2024a)
Per-MPST (Wang et al., 2024a), Per-DOC (Zhu et al., 2023)
CARM (Pitis et al., 2024) PIPT (Balepur et al., 2025)
RPR Dataset (Pitis et al., 2024)
Beavertails /SHP (Ji et al., 2023), Stanford Prefer-
ences (Ethayarajh et al., 2022), HH-RLHF (Bai et al., 2022), Mnemonie (Balepur et al., 2024)
PAL (Chen al., 2024b)
Anthropic Persona Perez et al. (2023), Pick-a-
Pic (Kirstain et al., 2023)
PEARL (Mysore et al., 2023a) FERMI (Kim & Yang, 2024)
WORKSM, AITA, see PEARL paper
OpinionQA (Santurkar et al., 2023), GlobalOpin- ionQA Durmus et al. (2024), LAMP Salemi et al. (2024) FtPersLIm (Li al., 2023a)
Prompt Rewrite (Li et al., 2024b)
Linear Align. (Gao et al., 2024)
HH-RLHF Bai et al. (2022)
Table Inexhaustive List of Recent Personalized//lluraisttii Alignment Methods and Datasets Used
18


Reviews:
[{"review_id": "FRd8lvqYyJ", "reviewer": "Reviewer_ckzC", "summary": "This survey explores the topic of personalized preference alignment in LLMs, organizing techniques into training-time methods (user-specific parameters or steerable models) and inference-time approaches (prompting, reward-guided decoding, and logit rectification). The paper distinguishes preference alignment from user modeling, reviews evaluation, benchmarks, and identifies future challenges in continuous personalization and handling complex user value statements.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Provides a clear taxonomy of methods for personalized preference alignment that makes them more accessible and provides structure into studying this topic. \n- Explains the motivations behind existing methods, compares them, and lists their key limitations. \n- Unlike prior studies, it differentiates personalized preference alignment from related personalization areas which helps clarify their boundaries.", "reasons_to_reject": "- The comparative analysis lacks focus on performance and efficiency aspects of existing methods on different benchmarks.\n- There is no dedicated discussion of ethical, privacy or other potential risks and implications of personalized preference alignment.\n- Discusses limitations and a few future work directions but does not elaborate on them. It would be useful to further discuss the quality gaps and challenges that remain to be addressed in this field.", "questions_to_authors": "1. Reading through the sections 4 and 5 it is not clear to the reader which methods are better performance-wise or efficiency-wise and where are the gaps. Could you expand on that?\n2. In the future work directions, what are the anticipated challenges and potential ways to address them?\n3. What are the ethical and privacy concerns that you foresee with personalized preference alignment?", "extra_scores": {}}, {"review_id": "iIA2QAvmuL", "reviewer": "Reviewer_9asZ", "summary": "This paper presents a comprehensive survey on personalized and pluralistic preference alignment in large language models (LLMs). It categorizes existing techniques into training-time, inference-time, and user-modeling-based methods, and offers valuable insights into datasets, evaluation practices, and open challenges. I find this to be a well-structured, timely, and valuable survey. While some sections could benefit from further clarification or synthesis, the paper overall meets the standard for acceptance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Timely and relevant topic\nThe alignment of LLMs to user-specific preferences is an emerging but rapidly growing area. This paper addresses a clear gap in existing literature by focusing specifically on preference alignment, not just general personalization or user modeling.\n\n2. Comprehensive taxonomy\nThe classification into training-time, inference-time, and user-modeling techniques is intuitive and clearly explained. The paper nicely distinguishes between parameter-based personalization (e.g., PEFT modules, soft prompts) and controllable generation approaches.\n\n3. Clear problem formulation\nThe formalization of the personalization problem using a user-conditioned reward function is rigorous and helps position the surveyed methods within a consistent framework.", "reasons_to_reject": "1. Limited synthesis across methods\nWhile many methods are described in detail, the paper sometimes reads like a catalog rather than a deeply analytical synthesis. For example: What are the trade-offs between training-time and inference-time personalization in real deployment scenarios?\n\n2. Ambiguity in scope definitions.\nThe term pluralistic preference alignment is introduced early, but its distinction from personalized alignment is somewhat vague. Does “pluralistic” simply refer to accommodating diversity across users, or does it imply multi-objective or group-aware alignment?\n\n3. Minor writing issues. Some repeated citations (e.g., “Chen et al., 2024c” appears multiple times for different topics). Typos like “user-spefic” instead of “user-specific” in Section 4.1.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zkht0aEGcw", "reviewer": "Reviewer_oXrd", "summary": "The authors provided a systematic survey on preference alignment with personality settings, including many works that depend on training or customized LLM inference workflows.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is an excellent survey covering various aspects of personalized preference alignment, including training & inference techniques, and evaluation, including datasets or benchmarks.\n\nI also enjoy the authors' clear formalization of this topic, as well as distinguishing it from related areas such as LLM role playing and listing corresponding surveys, which makes the paper highly reader-friendly.", "reasons_to_reject": "1. Despite the above strengths, it is sad to see that the authors did not clarify how this topic was different from other problems in definition or experimental setting, as certain works involved in this survey can be shared in the aspect of method design.\n\n2. The authors focused on analyzing specific works and techniques. However, it would further strengthen the survey if they could provide deeper insight into this problem in Sections 1 & 3.\n\n3. The authors are recommended to add references to more related works to make this survey better, including:\n    1. Fine-tuning methods like the famous DPO[1] and recent works[2, 3].\n    2. Test-time methods like RE-CONTROL[4] and ICDPO[5].\n\n[1] Rafailov, Rafael, et al. \"Direct preference optimization: Your language model is secretly a reward model.\" Advances in Neural Information Processing Systems 36 (2023): 53728-53741.\n\n[2] Wang, Zekun Moore, et al. \"PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignment.\" arXiv preprint arXiv:2410.13785 (2024).\n\n[3] Wu, Junkang, et al. \"RePO: ReLU-based Preference Optimization.\" arXiv preprint arXiv:2503.07426 (2025).\n\n[4] Kong, Lingkai, et al. \"Aligning large language models with representation editing: A control perspective.\" Advances in Neural Information Processing Systems 37 (2024): 37356-37384.\n\n[5] Song, Feifan, et al. \"Instantly Learning Preference Alignment via In-context DPO.\" Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MBuQ499q7i", "reviewer": "Reviewer_Voas", "summary": "This paper is a survey that presents a comprehensive overview of personalized and pluralistic preference alignment in Large Language Models (LLMs). Personalized preference alignment has emerged as a key research direction, motivated by the observation that users often have diverse stylistic preferences when interacting with personal AI systems. The authors begin by categorizing existing methods into training-time and test-time approaches. Training-time methods include strategies that either fine-tune LLMs with user-specific parameters or construct steerable models capable of adapting their responses based on user input. In contrast, test-time methods modify the decoding process without altering the underlying model weights. These are further classified into three main categories: prompting and context optimization, reward- and value-guided decoding, and logit rectification and correction. Additionally, the paper explores user modeling as a complementary strategy for personalization. The survey also reviews available datasets and benchmarks, emphasizing the limited availability of real user data and the widespread reliance on synthetic or simulated user personas. It notes that evaluation practices in this area remain fragmented and lack standardized protocols.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors present a thorough and valuable survey on the timely and increasingly important topic of personalized and pluralistic preference alignment in Large Language Models (LLMs).", "reasons_to_reject": "This paper is a survey and does not include any original experimental contributions. As such, its technical novelty is limited. To offer more substantial insights to the community, it would have been beneficial to include empirical comparisons across different methods in varied settings. While the survey is well-organized and informative, its suitability for a technical conference may be debatable given the lack of new experimental findings.\n\n==================\n\nConsidering that COLM accepted survey papers last year, this paper would also be a suitable candidate for acceptance.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: There is a lingering question here whether a survey is a proper place for a conference such as COLM. Assuming it is appropriate, I believe this paper is a good fit for the conference and mostly ready for publication. The authors properly addressed the questions/comments by the reviewers and seem to have covered a large body of literature in their survey.
Decision: Accept