PersonalLLM: Tailoring LLMs to Individual Preferences
Authors: Thomas P Zollo
Andrew Wei Tung Siah
Naimeng Ye
Ang Li
Hongseok Namkoong
Venue: ICLR 2025 Poster (2025)
Keywords: Personalization
LLM
Alignment
benchmark
dataset
reinforcement learning from human feedback
language models
RLHF
preferences
Primary area: datasets and benchmarks
TL;DR: A public benchmark and dataset focused on adapting LLMs to provide maximal benefits for a particular user.

Abstract: As LLMs become capable of complex tasks, there is growing potential for personalized interactions tailored to the subtle and idiosyncratic preferences of the user. We present a public benchmark, PersonalLLM, focusing on adapting LLMs to provide maximal benefits for a particular user. Departing from existing alignment benchmarks that implicitly assume uniform preferences, we curate open-ended prompts paired with many high-quality answers over which users would be expected to display heterogeneous latent preferences. Instead of persona prompting LLMs based on high-level attributes (e.g., user race or response length), which yields homogeneous preferences relative to humans, we develop a method that can simulate a large user base with diverse preferences from a set of pre-trained reward models. Our dataset and generated personalities offer an innovative testbed for developing personalization algorithms that grapple with continual data sparsity---few relevant feedback from the particular user---by leveraging historical data from other (similar) users. We explore basic in-context learning and meta-learning baselines to illustrate the utility of PersonalLLM and highlight the need for future methodological development.

Published a conference paper ICLR 2025
PERSONALLLM: TAILORING LLMs TO INDIVIDUAL PREFERENCES
Thomas "oollo'
Andrew Wei Tung Siah" Columbia University andrew. andrew.siaheee siiah@columba edu
Naimeng Ye Columbia University ny2 wol.6///oommaa....oo edu
Columbia University tpz21058col wwt.efeoouubbiia.mmmmm edu
Ang Li
Hongseok Namkoong
Columbia University
Columbia University
a14263@ea wow.occccoliiiammmmmmmmmm a. edu
hongseok, namkoonggccllumbia edu
ABSTRACT
As LLMs become capable of complex tasks, there is growing potential for personalized interactions tailored to the subtle and idiosyncratic preferences of the user. We present public benchmark, PersonalLLM,, focusing on adapting LLMs to provide maximal benefits for a particular user. Departing from existing alignment benchmarks that implicitly assume uniform preferences, we curate open-ended prompts paired with many high-quality answers over which users would be expected to display heterogeneous latent preferences. Instead of persona- prompting LLMs based on high-level attributes (c.g., user race or response length), which yields homogeneous preferences relative to humans, develop method that can simulate a large user base with diverse preferences from a set pre-trained reward models. Our dataset and generated personalities offer innovative testbed for developing personalization algorithms that grapple with continual data sparsity few relevant feedback from the particular user-by leveraging historical data from other (similar) users. We explore basic in-context learning and meta-learning baselines to illustrate the utility of PersonalLLM and highlight the need for future methodological development. Our dataset is available lhtpps:// huggingface co datasets waw.aasttttaammmonn.g.oo 1ab
INTRODUCTION
The alignment of L.Mss with human preferences has recently received much attention, with a focus on adapting model outputs reflect universal population-level values. A typical goal is to take pre-trained model that cannot reliably follow complex user instructions (Wei et al. 2022) and can easily be made to produce dangerous and offensive responses (Perez al. 2022). and adapt the instructions of its user base (Ouyang al. 2022)
or train generally helpful and harmless assistant (Bai et al. (022). By assuming uniform preference across al. [022] Christiano al.) (2017) demonstrate the feasibility of learning and optimizing a monolithic preference ("reward model"). Alignment techniques have provided the basis for popular commercial applications like ChatGPT, as well as Instruction-tunee open-source
the population, recent successes (Ziegler al.
models (Touvron et al. 2023).
The rapid advancement in LLM capabilities opens the door to an even more refined notion human preference alignment: personalization. A personalized model should adapt to the preferences and needs of particular user, and provide maximal benefits as it accumulates interactions (see Figure IT. Given the expected data sparsity in this setting, such personalized language systems will likely also rely on historical data from other (similar) users to learn how to learn from a small set new user feedback (see Figure ( By discovering patterns across users, these systems can efficiently optimize their responses, ultimately leading more effective and beneficial conversational AI. For example, personalized learning experiences could be crafted by adapting educational chat assistants to an individual student's specific learning pace and style based on previous successful interactions with similar students. Customer support chatbots could offer more accurate and empathetic responses by drawing on wealth of previous interactions, leading to quicker resolution times and higher customer satisfaction. In

---

Published a conference paper at ICLR 2025
Interaction History
Recommend French food Manhattan.
Recommend food try Maine?
No Personalization
Maine known by oysters and lobsters!
Estela has great mussels dish!
But I'm allergic to shellfish.
Recommend food to try in Maine?
Dirty French NYC has good mushroom pasta.
Personalization
Oh, blueberry pancakes? You're allergic to oysters.
Figure 1: Standard LLMs require tedious re- re-prompting to learn a user's preferences in each session. Personal- LLM aims to learn a unique user's diverse preferences to maximize long-term satisfaction.
healthcare, personalized chatbots could provide tailored advice based on patients with similar medical histories and communication preferences.
Compared to conventional applications where prompts have a uniform notion of "ground truth" (e.g.. question answering), LLM personalization is distinguished by the need to study open-ended prompts where users exhibit heterogeneous preferences across many possible high-quality answers TFiuure[p. While personal preferences may vary according to simple features like user age Chan et al. (2024); Castricato al. (2024) and answer length and technicality/ Li et al. (2024), they also involve more abstract dimensions of culture, politics, and language Kirk al. (2024). as well aspects of personality that are difficult to explain (Hwang al. (023). A
personalized LLM should be able to adapt to subtle, idiosyncratic, and sometimes sensitive differences between user tastes as it gathers more interactions.
Inspired by the vision of future with personalized AI, we introduce PersonalLLM, public, open-source benchmark designed to adapt LLMs to provide maximal benefits for individual users. In order to explore complex differences in user tastes, our benchmark features a set of prompts with many high-quality LLM responses (from state-of-the-ard LLMs like GPT-4o, Claude Opus, and Gemini 1.5 Pro), such that humans expected to express diverse preferences over the responses. Such an approach to kataset-building stands in contrast to existing alignment datasets, where responses exhibit observable quality differences (see Figure 3).
For each prompt and set of responses, our dataset also includes scores from set of 10 reward models with heterogeneous preferences over those responses. We leverage these reward models to sample many synthetic "users" (or personal preference models) via weighted ensembles of their preferences, and in doing able to simulate an entire user base, which we argue to be a critical ingredient in a truly useful personalization benchmark. Through extensive analysis of the preferences of these users over our dataset, we show these simulated personal preference models to be diverse and non-trivial (e.g., with respect to length, formatting, or tone). We illustrate the difficulty of creating such an environment by comparing to the increasingly popular persona prompting baseline (Castricato et al. 2024: Chan et al.) 2024 (ang et al. 2023). which in our analysis produces preferences only half as diverse as a set of PersonalLLM users across multiple metrics. Taken together, the prompts, responses, and personalities present in PersonalLLM offer an innovative test for benchmarking personalization algorithms as they tailor interactions based on previous interactions with an individual user.
While fine-tuning and reinforcement learning approaches (Schulman et al. 2017: Rafailov et al. (023)) are effective for aligning to population-level preferences, personalization requires a new algorithmic toolkit, as is not practical to gather enough data or store a separate copy of the model or even low-rank adapter weights (Hu et al. 202I) for every user. PersonalLLM offers the versatility necessary to spur development across a range of new approaches to personalization: in-context learning (ICL) (Brown et al. 2020). retrieval augmented generation (RAG) (Lewis et al. 2021a), ranking agents, efficient fine-tuning, and other adaptation techniques. In our experiments, we highlight particularly salient challenge compared typical applications of personalization in recommendation systems: the space of "actions/rresonses"" is prohibitively large to be able to explore based on interactions with a single user. Since this necessitates learning across users, we model this meta-learning problem where the goal is to leverage wealth of prior interactions from historical users to tailor responses for a new user who does not have a significant interaction history (see Figure 2)
Motivated by key methodological gaps in personalizing LLMs, here we summarize our contributions: 1) We release a new open-source dataset with over 10K open-ended prompts paired with 8 high-quality responses from
2

---

Published conference paper ICLR 2025
Interaction
1. Lookup Similar Users
2. Personalized Generation
Database
New Query
History
History
Lookup Algorithm
History
Adaptation Algorithm
(e.g. Using Sparse Language Feedback)
History
History
History
New User
Short
Personalized Response
History
Figure 2: PersonalLLM enables the development of methods for learning across users, where dataset of historical users and their interactions is leveraged to personalize interactions for a new user with a limited history.
top LLMs. 2) We propose a novel method for sampling "users" (i.c., personal preference models) that, unlike existing methods, creates diverse preferences and allows for the simulation of large historical user bases. 3) We illustrate new possibilities for algorithmic development learning across users.
Our goal in creating the open-source PersonalLLM testbed is facilitate work methods to personalize the output of an LLM to the individual tastes of many diverse users. We do not claim that our simulated personal preference models provide high-fidehle depiction of human behavior. Instead, they offer challenging simula- tion environment that provides the empirical foundation for methodological innovation in capturing the complex array of human preferences that arise in practice. As analogy, while ImageNet (Russakovsky et al.) 2015) is noisy and synthetic-e.g. differentiating between 120 dog breeds is not a realistic vision task-it provides challenging enough setting that methodological progress ImageNet implies progress on real applications. Similarly. we believe PersonaILLM is reasonable initial step toward the personalization of language basee agents, building on the common reinforcement learning paradigm of benchmarking personalization algorithms with simulated rewards (Zhao et al. 2023 le et al. 2019).
PERSONALLLM
Our PersonalLLM testbed is composed of two high-level components: 1) dataset of prompts, each paired with a set high-quality responses among which humans would expected to display diverse preferences and 2) method for sampling diverse personal preference models, such that using these "personas" as our simulated users. Next, we will describe each of them in detail. Our data and code are publicly available, and full documentation for our dataset is available in Appendix
test methods for personalization
2.1 DATASET
Since our goal is to study diverse preferences, we first focus on collecting open-ended prompts, similar to chat setting. We compile 37,919 prompts from Anthropic Helpful-online, Anthropic Helpful-base (Bai et al. 2022), Nvidia Helpsteer (Wang et al. 2023), and RewardBench (Lambert et al., 2024). From this set, prompts are filtered to those with length of 2400 characters fewer as most reward models are limited to 4096 context length. We then randomly draw 10,402 prompts to form our final set.
Our next aim is to collect many high-quality responses for each prompt. Important desiderata for the generated responses are that i) they do not exhibit much variation in terms of undesirable contents (like misinformation or toxicity) or obvious dimensions helpfulness or length, as is typical in RLHF datasets, ii) they exhibit diversity across meaningful dimensions of personal preferences like political viewpoint and culture, well as difficult to describe latent features. To achieve this, we generate eight responses for each of these 10,402 prompts using selection of the top models from ChatArena and other important benchmarks: GPT-4o, Claude 3 Opus, Gemini-Pro-I Command-R-Pinn GPT-4-Turbo, Claude 3 Sonnet, Llama3-70B-Innse Instruct, and Mixtral
https: com http://wwww...toeoooooooooo/a///

---

Published as a conference paper at ICLR 2025
Existing Unimodal Preference Data
PersonalLLM Preference Data
Colosseum, Pizza, Leaning Tower of Pisa are all great.
What to try in Italy?
Pizza at Naples.
Shopping in Milan!
Great Wall of China, but you're probably too weak to climb it.
History at the Colosseum!
Figure 3: Leftt: Existing alignment datasets contain prompts paired with multiple responses, where the majority of people are expected to prefer one specific response (e.g, a harmless response). Right: Our dataset consists of prompts paired with many high-quality responses, creating testbed to build PersonalLLMs.
8x22B. We split the resulting dataset into 9,402 training examples and 1,000 test examples. See Appendix Figure for an illustration of our dataset construction pipeline.
2.2 SIMULATING PERSONAL PREFERENCE MODELS
We design our approach to creating simulated PersonalLLM users with several goals in mind. First, we aim for PersonalLLM allow for the simulation of large number of users, enabling the study of the full personalization paradigm for applications such as search engines and recommender systems (Davidson et al.) 2010: Das et al. 2007 Xu et al. 2022 Färber and Jatowt 2020) wherein historical database of user data is leveraged to personalize new interactions. Next, when applied to our dataset, our preference models should allow for the study of alignment based on diverse and complex latent preferences, as opposed simple attributes such as answer length or sensitive and reductive user characteristics, for example, race or gender. Finally, our evaluation should not rely on GPT4 which can be expensive and unsuitable for research purposes given model opacity and drift. While human annotations are the gold standard ideal for preference evaluation (Kirk et al. 2024). is often impractical obtain this feedback consistently throughout the development cycle. As result, we claim that synthetic personal preference models are needed to facilitate progress in LLM personalization.
To overcome this difficult challenge of simulating diverse preferences, we propose a solution based on a set of strong open-source RLHF reward models. While it may be the case that different reward models have fairly uniform preferences over the giggh-uality/lowwooo quallity response pairs on which they are typically trained, we hypothesize that their preferences over many high-quality responses will instead be diverse. Since the number of existing top-quality reward models is much smaller than the number of users we would like to simulate, we propose to generate users by sampling weightings over the set of reward models, such that the reward score assigned to a (prompt, response) pair by a user a weighted sum of the reward scores assigned by the pre-trained reward models. In Section 3 we validate our hypothesis regarding the diverse and non-trivial preferences created by such sampling.
More formally, for an input prompt X, an LLM produces output response y Vr where X' and y are the set of all-natural language. Then, a preference model R X' x y - R assigns a reward score to the response given to the prompt, with higher scores indicating better responses. Next, consider a set of B base reward models, denoted as RM, b = 1..... B, and set of k B-dimensional weightings, which represent set of personal preference models. The preference model corresponding to user can then be defined by a weighted average of these B base models RM1, RM2. ........ RMg, with weights W1, W2,,.... WB: R (z. y) = Σ wi RM, (z, y). For our base reward models [RM,) km)) we select 10 reward models with strong performance on RewardBench, an open-source benchmark for evaluating reward models. These reward models models such as Llama3, Mistral, and Gemma (see Appendix[A) (A). We evaluate each (prompt, response) pair in the train and test set with each model so that for any personality created in this manner, each (prompt, response) pair in the dataset can be scored via a simple weighting. Together, our dataset and personal preference models create an innovative and challenging environment for developing personalization methodology. This extends the existing paradigm of simulated rewards, commonly used in domains like recommender systems (Zhao et al. le et al. 2019) to the task of LLM personalization.
built top of popular base
2023
4

---

Published conference paper ICLR 2025
0.1 Max. vote share for response Max. vote share for any response Max. vote share for any response Max. vote share for any response
0.01
0.05
Persona Prompting
55 of Prompts
60
50%
cifferent winning responses
different winning responses
different winning responses
different winning
40
Prompts
30
20
.22
5
2:66
#2
2
=2
3
2
=2
3
% Witniing Answers
30
Commond
GPT4 Barbo
GPT40
Claude Strved
Lama3 708
Gemine Pro 1.5
Motral 8x228 host.
commandd--t
an Tarbo
G97-40
cleude opas
classe Sonret
Gemoni tho 1.5
Mosrrl ax228 lodt.
Command
Gi7-4o
Cloale Opus
Claude Sonnot
Ulma3 7088
Gomiel Pro 155
Motral 8x220
Commund
GP74 Tarbo
GPT44
Cloudo Opus
Clanoi Sunnet
Uama3 708 hot
Mocral adzee
Coudo
Uemaa
Figure Probing the heterogeneous preferences of our simulated users across the PersonaILLM dataset compared persona prompting baseline. Top: For population of simulated users, the percentage of each population's 's vote share given to the most common winning response for each prompt. Middle: A histogram showing the number of responses that receive least vote from simulated population for each prompt. Bottom: Average win rates across the population for the LLMs in our dataset.
2.2.1 SAMPLING USER WEIGHTINGS
There are many valid ways to sample the B-dimensional weighting vectors. As a simple starting point, we propose to sample from Dirichlet distribution with uniform concentration parameter across all classes (w ~ Dirichlet(a)).. As a becomes very small, the preference models converge towards the 10 base reward models; it becomes large, preferences become unimodal. Such parameter allows us to simulate user bases with different underlying preference structures, as we detail in the next section.
ANALYZING PERSONALLLM
Next, in order to validate our testbed, we explore the preferences exhibited by our simulated users over the PersonalLLM dataset.
Preference Diversity and Comparison to Persona Prompting First, examine whether populations of personal preference models sampled via the method outlined in Section 2.2 do in fact display heterogeneous preferences over the prompt/response pairs in our dataset. In Figure[4[ett 3 columns), we provide experimental results for user bases of 1,000 PersonalLLM personal preference models sampled with parameters a = [0.01, 0.05, 0.1] and applied to the PersonalLLM test set to choose winning responses among the 8 included. The top row displays the percentage of prompts in the dataset for which the most popular winning response according the population receives indicate more diversity in preferred responses. The middle row shows the percentage prompts that have a across the population: heterogeneous population
more than 50%, 75%, and 95% of the population vote; higher values
given number of responses with at least one winning preferences induce higher concentration on the right side of each plot. On the bottom, we plot the overall win rates for each LLM across all users and prompts.
5

---

Published a conference paper at ICLR 2025
User Preferences for Semantic Features
User Preferences for Syntax Features
Preference entropy based on keywords
Regression Coefiicint
Regression Coofficient
preference Entrapy
0.2
2.3
0.0
1.2
-0.3
2.1
-0.2
-0.
formaity
educational
approval
caring
excitement
joy
optimism
length
no. unique words
avg word lengrth
no. stopwords
no. purctuation
formatted list?
no. adjladvv
0.9
Imagine
opinion
pose
raste
Average
Wioo
What
When
Where
Figure 5: Simulated user preferences with respect to prompt and response contents. Left, middle: For each user, a regression model predicts winning responses based on semantic (left) or syntactic (middle) features. Right: We examine the entropy in population preferences based on keywords in prompts, comparing words we expect to inspire heterogeneity (e.g., imagine, poem) to prompts beginning with "who", "when", and "where".
In the right column, we also offer results for a persona prompting baseline. Persona prompting (Castricato et al., 2024; Chan et al. 2024 Jang et al. 2023) is an emerging method for evaluating methods for LLM personalization, wherein an LLM is prompted to decide which response would be preferred by a person of particular race, gender, age. profession, other demographic category. While one might argue that such evaluation is prima facie discriminatory and reductive, and therefore not a desirable standard for algorithmic advancement, especially in sensitive areas, we are also interested in whether persona prompting meets the technical challenge of producing a simulation environment with high degree of heterogeneity. For our baseline, we prompt the sfairXCC/Fsfirx--LLLa M-A---LLMM3---------0
reward model (Dong et al.) 2023) to score responses with respect to 1, 1,000 personas randomly sampled from PersonaHub Chan et al. (2024). a recent effort at building a database of personas that are representative of a pluralistic population.
Results are shown in Figure 4 First, we observe the left-most bar in each plot in the top row. We can see that for PersonalLLM personas the top response receives a majority user vote for only about half of the prompts, while that figure is closer to 90% for the persona prompting baseline. By summing the right-most two bars in each plot across the middle row, one can observe that for roughly 60% of prompts, at least different answers are chosen the best by at least 1 user under our set of personas. For LLM persona prompting, it is roughly 30%. Finally, our ensembled preference models have a fairly diffuse set of preferences over the response-generatinn generating LLMs, while persona prompting strongly prefers subset of 4 models. With respect to changes across the left columns, from the top row of plots we can observe that as a increases, preferences become more uniform, with more prompts featuring one highly preferred response. However, if is set too low, user preferences cluster very tightly around the base reward models (see Appendix Figure 9).
Effects of Semanties and Syntax We further analyze the effects of semantics and syntax on the preferences of simulated user base (with a 0.05 and 1,000 users) We use regression analysis understand how different features may drive the preferences of different users, including semantic response features such as the formality or educational value or the expressions of certain emotions (approval, caring, excitement, joy, optimism), as well as syntactic features like length and the use of different parts of speech and formatting, For each user, we gather their most and least preferred responses for each of the test prompts and create a binary prediction problem to predict whether a given response is a winning or losing response. Responses are embedded using crafted features (based on either syntax or semantics, which are studied separately), and a unique logistic regression model is trained for each user. Semantic features were captured using pre-trained classifiers, while syntactic features were engineered using nltk (Bird and Loper 2004). See Appendix B.T for complete details.
In Figure (left and middle), for each feature we show box plot with the resultant regression coefficient for that feature across users. A positive coefficient suggests a feature associated with winning responses, while negative coefficient suggests feature's role in losing responses. A tight box indicates homogeneous preferences, while greater spread represents heterogeneity. Here, we can see a reasonable mix of heterogeneity and homogeneity across user preferences for different features. Semantically, users tend to prefer responses with educational value and dislike highly formal responses, although the size of these preferences varies. Encouragingly. syntactic preferences do not seem to be driven by uniform preferences for simple features like length the presence of formatting such as bullets or lists. In Figure 5 (right), we compare the Shannon entropy (Shannon) 1948) in given prompt based on keywords. By comparing words we
the population preferences over the responses would expect to inspire heterogeneity (c.g., "imagine", "opinion", "poem") ") to prompts beginning with "who", "when", and "where", which evoke more objective answers. We can see that the presence of these subjective

---

Published a conference paper at ICLR 2025
Table 1: Representativeness scores in relation to real human opinions from important demographic groups for different LLMs, as well as our PersonalLLM population.
A121 Labs
OpenAl
PersonalLLM
Demographic
jl-jumbo
jl-grande-v2
ada
text-davinci-003
Ours
Black White
0.820
0.812
0.823
0.702
0.833
0.807
0.794
0.817
0.699
0.832
Less than $30,000
0.828
0.813
0.833
0.693
0.838
$100,000 or more 0.797
0.790
0.807
0.708
0.831
cues in prompts leads to a more diverse set of preferences than those seeking simple entity or date responses. Such diversity among the prompts creates setting where an algorithm must not only learn how to personalize but also when to personalize. We extend the same analysis to persona prompting in Appendix Figure g
Comparison to Human Preferences Finally, to understand how our synthetic personal preference models relate to human preferences over text responses, we surveyed population of our simulated users on set of questions with responses where a large and diverse set of humans have given their preferences in the past, the OpinionQA dataset (Santurkar et al. 2023). OpinionQA is an appropriate validation set for our personas given that its broad coverage of topies (e.g., science, economics, politics, romance, and many other topics) aligns with the open-domain nature of our prompt set. See Appendix Table for example questions and answers. Following previous work, we calculate the representativeness score of the opinion distribution given by our simulated preference models. This score is based on the Wasserstein distance of the synthetic population preferences from that of a real human population for each question| | population would have to display heterogeneous preferences over question/response sets where humans do so and produce homogeneous (and matching) preferences in cases where humans do the same. Our population of simulated users produces a representativeness score of 0.839 with respect to the overall population of the US, higher than any LLM in the original study and near as representative of the overall population as some real, large demographic groups Further, in Table we can see that our simulated users produce opinions that better represent wide range of important (and sometimes protected) groups according to demographic attributes such as race and income. In fact, this is the case for 59 of 60 demographic groups studied (see Appendix Table 4
To have a high representativeness score, our simulated user
Summary of Analysis These results show that our simulated user reward models: 1) produce heterogeneous preferences over our dataset, considerably more so than persona prompting an LLM, 2) display reasonable and diverse preferences with respect the syntactic and semantic content of prompts, and 3) simulate a user base that better represents diverse human opinions than many popular LLMs, without resorting to explicit stereotyping.
4 PERSONALIZATION EXPERIMENTS
A central challenge in personalization is the perpetual lack of data, most users will provide sparse feedback, far less than necessary to effectively adapt an LLM. Two problems emerge from such an environment: 1) how to best leverage small amounts of user-specific data for personalized adaptation and 2) how to lookup similar users based on language feedback. In order to illustrate how researchers might approach these problems, we perform experiments in two modal settings for LLM personalization research. First, we explore scenario where have access to short but relevant interaction history (i.e., previous prompts and preference feedback) for the user, and we aim to efficiently leverage that interaction history through ICL. Then, we explore more complex setting that fully leverages the advantages of PersonalLLM, where the current user possibly has no relevant interaction history, and we must instead retrieve relevant interactions from similar users in a database. Overall, our results validate the solid empirical foundations of PersonalLLM while highlighting salient algorithmic questions and the fact that there is much room for improvement in terms of personalization performance.
All experiments simulate a chatbot using in-context learning to personalize responses for a test set of new users. Our test set simulates 1,000 personal preference models (or "users") drawn with a = 0.05 (as in Section 3) and each user is associated with one test prompt from the PersonalLLM test split. For new user with an associated
'The score 1 W for distance W; a higher score indicates more representative preferences.
7

---

Published as a conference paper at ICLR 2025
test prompt, the goal is to use ICL to produce a response to maximize the reward (and win rate vs. GPT40) given by the user's personal preference model (i.e., weighted ensemble of reward models). Our underlying chatbot is Llama3-88 1--33--B-Insttuu Instruct, and all text embeddings are extracted using the top-ranking model on the MTEB
(Muennighoff' et al. 2023) leaderboard below 500M parameters Rl Further details are given in Appendix C.1
4.1 PERSONALIZED IN IN-CONTEXT LEARNING
2023), the problem may be different While ICL for broad alignment has been studied to some extent (Lin et al. when the underlying preference model is idiosyncratic and may cut against pretraining and RLHF dataset biases. In our initial set of experiments, we focus personalizing the response to given query. i.e., feedback gathered from the same user on similar prompts. By doing so, we can study key questions related to personalized inference with ICL., for example, which response(s) should be included and the importance of correct labels. Though these questions have been studied with respect more general NLP tasks (Min al., 2022 Yoo al. 2022 Pan et al. (023). it is unlikely that these findings can be extrapolated to the unique personalization task, and thus more work is needed in this area. A solid foundation of ICL techniques for personalization can then form the basis for more complex systems involving. e.g., looking up similar users.
setting wherein we have small set of useful data for the sake of
Experiment Details For each of our 1,000 test users, each with their own test prompt, we build a short but relevant interaction history by retrieving 5 other prompts based on embedding similarity. We build winning/losing response pair for each prompt based on each user's most and least preferred answers from the 8 models in our dataset. In order to establish baseline results on key questions in personalization, we include several baselines for how these interaction samples are leveraged in-context during inference: 1) Winning and Losing: Both the winning and losing responses are included. 2) Winning only: Only the winning response is included. 3) Losing only: Only the losing response is included. 4) Losing only (Mislabeled): Only the losing response included, mislabeled as a winning response. Inference is performed using 1, 3, and 5 such examples (see Appendix I.I for exact templates), and evaluated by scoring with each user's (weighted-ensemblle) preference model. We also compare to a zero-shot baseline, with no personalization.
Results Results are shown in Figure 6 left column. We can see that the best performance comes from ICL with only winning examples. This underlines the outstanding challenge of training LLMs to not only mimic winning responses in-context, but also leverage the contrast between winning and losing responses, especially when the differences may not described in the model's 's training data. Any amount of examples, even incorrectly labeled, are helpful relative to zero-shot; this may be unsurprising, as all models in our dataset are stronger than our 8B parameter chat model. An interesting result lies in the comparison between Losing Only and Losing Only (Mislabeled). While the mislabeled examples may help performance versus a zero-shot bascline (once again because they are from stronger underlying LLM), Llama-8B- Instruct gains more from having these relatively strong losing responses labeled as losing. Overall, our findings reflect that a model trained for broad alignment does have some of the necessary capabilities to do idiosyneratic personalization using only in-context examples, but that much work is left in order to fully leverage this language feedback.
4.2 LEARNING ACROSS USERS
Having established some empirical foundations for in-context personalization with PersonalLLM, we next highlight a particularly significant challenge prevalent in practice that has been under-explored in the LLM community: the cold-start problem. When a new user with limited prior interaction data arrives, or a user inquires about new topic, prior user interactions alone cannot inform a satisfactory response. We model this challenge as a meta-learning problem, where the goal is to utilize a rich reservoir of prior interactions with a diverse set of users. We are motivated by real-world scenarios where we have access to proprietary database containing extensive interaction histories from previous users. When a new user arrives, our goal is to utilize this rich, heterogeneous dataset to provide the best possible response to the new user's query despite having only limited initial interactions with them that may not be relevant to the current query. This setting resembles typical recommendation systems, but "actions" are now defined over the space of natural language outputs instead of fixed set of items. See Figure 2 for further illustration.
https::/ httttsss///uuggiggfee..ooo htt.:/gggin.........ooo/oaaanana http://eeeeleee_________ 400M v5

---

Published conference paper ICLR 2025
Personalized ICL
Meta Learning
-0.2
Reward
-0.6
-0.8
Winning Only
Losing Only
Winning Losing
Win Rate (96
25
Zero Shet
Winngly Only
Winning a Losing Losing Only (Mislebeled)
Self ICL
Losing Only
1
Winning Only
Losing Only
Winning Losing
Figure 6: Results across different personalization algorithms. (Left) Test users are accompanied by relevant interaction history with pairwise preference feedback, and we explore the LLM's s ability exploit this informa- tion in context. (Right) Test users have interaction histories that are not relevant their test prompt, and we probe methods for embedding users based on language feedback to retrieve useful examples from similar users.
Our experiment explores the open question of how best to embed (i.e., represent with some vector) users based on small amounts of natural language feedback. With effective algorithms to lookup similar users, more relevant interactions may be leveraged to improve response to a new user query. While a rich literature exists on information retrieval (i.e., RAG) for typical NL.P benchmark tasks like question answering and fact-checking (Lewis et al.)
2021b Gao et al. 2024), the personalization task necessitates new algorithms.
For each of our 1,000 test users, we build short but, in contrast to our first experiment, possibly irrelevant interaction history by retrieving 5 random prompts. Winning/losing response pairs (i.e., preference feedback) are selected as before. In order to supplement these interaction histories, we sample prompts, winning response,
Experiment Details
historical database 10,000 users (also with a = 0.05), each with a set selected randomly and the winning and losing
losing response triplets from the train set, where the prompts responses are selected as the historical user's highest and lowest scoring among the 8. We compare methods for embedding users for lookup: 1) Winning minus Losing: Average direction in embedding space between winning and losing responses for each prompt. 2) Winning only: Average direction in embedding space for winning responses. Losing only: Average direction in embedding space for losing responses. For each test user, we build a set of candidate prompt/feedback data by retrieving the 20 most similar historical users based the cosine similarity of their embeddings, and then of the pool created by those users' interaction histories, retrieving 3 examples for in-context learning based on prompt embedding similarity to the test user's 's new query. We compare Self-ICL baseline, where the test user's possibly irrelevant prompt/feedback history is used for ICL. All methods use only winning responses in-context; evaluation is done as before.
Results Our results are shown in Figure 6 We find that using the strongest user embedding method, which most fully exploits the available pairwise preference feedback, meta-learning can beat the self-ICL baseline. This positive result for meta-learning highlights the opportunity created by leveraging historical user data, and the feasibility of embedding users based on small amount of language feedback. However, the gain from our relatively naive method is small, illustrating the need for methodological innovation in building such systems.
RELATED WORK
Preference Datasets Recent developments in large language models (LLMs) emphasize the importance of
aligning LLMMs based on preference preerrencefeedback feedback rather than merely pre-training on large corpora of language in self-supervised manner. Consequently, there has been surge the creation of open-source datasets (Bai ett al., 2022) Nakano et al.) 2022 Köpf et al.) (023) Dubois et al. 2024: Lambert et al.) 2024) designed to support research on alignment methodologies. significant limitation in the existing datasets is that they mainly

---

Published a conference paper at ICLR 2025
enable fine-tuning to a single high-level notion of alignment that is uniform across the population, such as instruction-following in RLHF (Ouyang et al. 2022) and helpfulness and harmlessness (Bai et al. 2022)
Personalization Personalization has been extensively researched across different fields, with previous datasets primarily focusing on applications such search engines and recommender systems (Davidson al. 2010
Das et al. 2007; Xu et al., 2022 Färber and Jatowt, 2020). Where language model personalization has been studied, has typically been focused on learning to mimic user's style, for example in writing headlines.
crafting social media posts, or emulating historical text with dialogue models (Salemi et al. 2024 Vincent et al.
Welch et al. 2022] Ao et al. 2021; Wu et al. 2021 Mazaré et al. 2018). Recently, given the success of population--evel alignment, researchers have begun to develop testbeds and methodology wherein the goal is to achieve a more granular level of personalized alignment for LLMs (Castricato et al. [024] [ang et al. 2023
2023
Kirk et | 2024: Li et al. 2024). Much of this work has focused on alignment for real or synthetic personas based on high-level attributes like race or occupation (Castricato et al. 2024: CCan et al. 2024). or high-level notions of alignment with respect to response qualities like length, technicality, and style. For example, Jang ct al. (202) decomposes personal preferences along a handful of easily observable dimensions and performs personalized generation by merging models trained with different preference data based on these dimensions. Evaluation is often done by prompting GPT4 to select the preferred response based on preferences stated in its prompt (Jang al., 2023; Castricato et al. 2024). In an effort to highlight the need for broad participation and representation in LLM alignment, the PRISM datasel collects user-profiles and personalized preference feedback from over 1,000 diverse human participants (Kirk et al. 2024).
6 DISCUSSION
Meta-Learning for Personalization We hope to encourage more work in the meta-learning setting, as exemplified by our experiments. This setting mirrors many real-world use cases where an organization has large proprietary dataset from historical users but a very limited interaction history with this particular user. Prior work cold-start problems has focused on the task of recommending discrete content items from media (or other) library. Extending and developing these techniques for LLMs is an exciting direction for future research.
Risks and Limitations We must consider the risks and limitations associated both with the release of our original benchmark dataset, as well as the larger goal of LLM personalization. With respect to PersonalLLM, we note all prompts and responses have not been manually inspected for quality or safety by human, although prompts are sourced from existing, reputable datasets, and responses are generated from state-of-the-art language models that have (presumably in the case of black box models) undergone safety alignment. Our benchmark is also limited with respect to the realism of the personas created by weighting reward models. On broader note, the goal of LLM personalization brings particular risks, including filter bubbles, stereotypes, feedback loops, personification risks, and user manipulation. Given these and many other predictable (and unpredictable) potential risks, is important that any efforts at LLM personalization are accompanied by research in robust transparency mechanisms and safeguards for personalization algorithms. Developing an empirical foundation for such efforts is another promising avenue for future work.
Future Directions Given that LLMs have only recently reached a level of capabilities meriting their widespread adoption for industrial and personal use, the study of LLM personalization is necessarily in its earliest stages of development. follows that there are many important and exciting avenues for future research, with respect to datasets, methodology, fairness, safety, and other aspects of responsible and reliable machine learning deployment. Since PersonalLLM is the first dataset to enable the study of complex personalized preferences expressed over many high-quality responses (to our knowledge) by large, diverse user base, the benchmark can be extended in many ways. For example, one might imagine a distribution shift scenario, where over time, personal preferences shift, and the personalization algorithm must balance stability and plasticity. Also, given concerns regarding the realism of our simulated users, we hope that our testbed drives the development of even more realistic personalization datasets and evaluation methods that more closely mirror the online and non- i..dd settings and better capture the true nuance and diversity of human personal preferences. Finally, continued work in personalization algorithms must be accompanied by work in personalization safety, fairness, and reliability. Future research may consider different aspects of the deployment pipeline (c.g., model architecture, data collection) and interaction model (e.g., UI/UX) with these concerns in mind.
10

---

Published conference paper ICLR 2025
ACKNOWLEDGMENTS
We thank ONR Grant N00014-23-15 1-2466 for its generous support. This work is supported by the funds provided by the National Science Foundation and by DoD OUSD (R&E) under Cooperative Agreement PHY-2229929 (The NSF AI Institute for Artificial and Natural Intelligence).
REFERENCES
Xiang Ao, Xiting Wang. Ling Luo, Ying Qiao, Qing He, and Xing Xie. PENS: A dataset and generic framework for personalized news headline generation. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 82-92, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 acl--oog.77 URL lttps://aclanthooooe wctp://wowo..o///.....ll------
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodde Danny Hermandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei. Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
Steven Bird and Edward Loper. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214-217, Barcelona, Spain, July 2004. Association for Computational Linguistics. htt.ttttss//aallntit http:///o......oooo//o/033011111
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Louis Castricato, Nathan Lile, Rafael Rafailov. Jan- Phillpp Fränken, and Chelsea Finn. Persona: A reproducible testhed for pluralistic alignment, 2024. URL petttps:/arrxe org/ aww./b/5555 17387
Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas, 2024. URL ht.t.t://:/// //arxi//xiiiv orgg abs /200244660.00 20094
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodci. Deep reinforcement learning from human preferences, 2017.
Abhinandan S Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. Google news personalization: scalable online collaborative filtering. In Proceedings of the 16th international conference on World Wide Web, pages 271-280, 2007.
James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, al. The youtube video recommendation system. In Proceedings of the fourth ACM conference on Recommender systems, pages 293-296, 2010.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06777 2023.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2024.
11

---

Published a conference paper at ICLR 2025
Michael Färber and Adam Jatowt. Citation recommendation: approaches and datasets. International Journal on Digital Libraries, 21(4):375 August 2020. ISSN 1432-1300. doi: (0007//007999000000028888 URL
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmennte generation for large language models: survey, 2024. URL
laaegg/bs/23333 10997
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III au2, and Kate Crawford. Datasheets for datasets, 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu
Chen. Lora: Low-rank adaptation of large language models, 2021.
EunJeong Hwang, Bodhisattwa Prasad Majumder, and Niket Tandon. Aligning language models to user opinions, 2023. URL.https:/aaxxiv/ org/ abs 2305. 1..3051.192 14929
Eugene Ie, Chih wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu, and Craig Boutilier. Recsim: A configurable simulation platform for recommender systems, 2019. URL httts:
//arxiv.org/aba. abs////s/99999 04847
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging, 2023. URL https:///aa:::...coomm org abs/ 221.//210.... 11564
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He Hc, Bertie Vidgen, and Scott A. Hale. The prism aligmment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models, 2024. URL
2404
16019
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnay Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model alignment, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, IJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmentt generation for snowledge-intensing nip tasks, 2021a.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retreneal generation for knowledge-intensive nip tasks, 2021b. URL https:::p.. //tp:iaaaaivva///a org/ abs / 2005. 1400
Xinyu Li, Zachary C. Lipton, and Liu Leqi. Personalized language modeling from personalized human feedback. 2024. URL
05133
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base Imms: Rethinking alignment via in-context learning, 2023. URI.tttss: http/axxivvvrgg.o abs 2312. 01552
Pierre-Emmanne' Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. Training millions of per- sonalized dialogue agents. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun iici Tsujii, edi- tors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2775-2779, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/DI8 1298. URL http://////caaaanooooo.ooooooo
12

---

Published a conference paper ICLR 2025
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?, 2022. URL hitps://
12837
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark, 2023.
07316
Reichhrro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang. Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question- answering with human feedback, 2022.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, 2022.
Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning "leams" in-context: Disentan- gling task recognition and task learning, 2023. URL hattss//arriiv.orr/aa///////
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournaeea,, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn Machine learning in Python. Journal of Machine Learning Research, 2830,, 2011.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge, 2015. URL
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization, 2024. URL htt.tp::/aaxiii.or////aaaa/
11406
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect?, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
C. E. Shanmon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379-423, 1948. doi: 10. 110.5533-77758.....1
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Sebastian Vincent, Rowanne Sumner, Alice Dowek, Charlotte Blundell, Emily Preston, Chris Bayliss, Chris Oakley, and Carolina Scarton. Personalised language modelling of screen characters using rich metadata annotations, 03 2023.
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. Helpsteer: Multi-attribun helpfulness dataset for steerlm, 2023. URL
wttp:/iiiwrgg..////eeooooobb /211. 09528
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe Le. Finetuned language models are zero-shot learners, 2022.
13

---

Published conference paper ICLR 2025
Charles Welch, Chenxi Gu, Jonathan K. Kummerfeld, Veronica Perez-Rosas, and Rada Mihalcea. Lever- aging similar users for personalized language modeling with limited data. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors. Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1742-1752, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/w1/2022..as act.oog.. URL
https: / ac antho logg 2trg/2022.co w.l.aallllc. 20rg/2022.alllleng
122
Yuwei Wu, Xuezhe Ma, and Diyi Yang. Personalized response generation via generative split memory network. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tui Tur. Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Longuage Technologies, pages 1956-1970, Online, June 2021, Association for Computational Linguistics. doi: 10.1865331/2021. naacl-main. 157. URLhttps: htttttsss//caaattololottt/oo/ htwp:/llooyooog///oooo 2021 naacl main.157
Jiajing Xu, Andrew Zhai, and Charles Rosenberg. Rethinking personalized ranking at pinterest: An end-to- end approach. In Proceedings of the 16th ACM Conference on Recommender Systems, RecSys '22. ACM, September 2022. doi: 10. 10.1145/352322:5547799)) URL / 1ttp://11..115555222220 3347394
Sang-Woo Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang- Woo Lee, Sang goo Lee, and Taeuk Kim. Ground-truth labels matter: A deeper look into input-label demonstrations, 2022. URL
https: /ttp:///aavvvv/e www.oogggg/b/////e/e//.oe/e.ee/t 2205. 12685
Kesen Zhao, Shuchang Liu, Qingpeng Cai, Xiangyu Zhao, Ziru Liu, Dong Zheng, Peng Jiang, and Kun Gai.
Kuaisim: A comprehensive simulator for recommender systems, 2023. URI.
abs 0ttp/2399.12444
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alee Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020.
14

---

Published conference paper ICLR 2025
Prompt
Filter Prompts
Response Generation
Reward Generation
Sourcing
OpenAl/GPT4o
Anthropic
bb ab-rihf-hhee hb-rihf-helpful hase
4797
OpenAl/GPT---Turbo
Helpful-Online
bb cla helpful online
3320
10 Reward Models from RewardBench
HelpSteer
1290
Anthropic
alpacaeval
Anthropic/Claudes0Opss
caasyval-eassy
259
Helpful-Bass
alpacseval length
219
Anthropic/ClaudeSSonneet
should--ree resporeeedddd
65
Nvidia
Llmbar CPnnts
mumbar-adver-neighle
34
Cohere/Coommand-RRPlss
Prompt
Response
Helpsteer
limbar-naiural
27
Meta/Llama3-705-nsstuuct
lianbrr
Umbar-adver- GPPett
RewardBench
mi-bench mod
14
Mistral/Miixttal5k222
Reward
mi bench hard
cee--benchheee casy
Google/Gemmin-Pro-11.5
Illustration of our dataset creation pipeline. (Leftt) Prompts sourced from various open-source RLHF datasets. (Left, middle) Prompts are filtered by length. (Right, middle) Responses to each prompt are generated by 8 high-quality LLMs. (Right) For each response for each prompt, rewards scored by 10 different lop-performing reward models.
Figure 7:
A DATASET INFORMATION
This section serves as documentation for our dataset, with its organization based on the notion of datasheets (Gebru al. 2021).
Our code and dataset will be made publicly available upon the release this paper.
A.1 COMPOSITION
A.1.1 PROMPTS
In order to create a set prompts and responses over which humans (and reward models) will display diverse preferences, our first focus the collection of open-ended prompts. As source of these open-ended prompts, we collected data from Anthropic Helpful-online, Anthropic Helpful-base Nvidia Helpsteer, and RewardBench. From this set, prompts
filtered those with length of 2400 characters fewer as most reward models are limited to 4096 context length. We randomly drew 10,402 prompts from the filtered subset. The resulting distribution of prompts from different sources is shown in Table 2
Table 2: Sources of the 10,402 prompts composing train and test sets.
Source
Count
th-rlhhf-helpull
4797
hee-llh-heelful-oniie
3320
HelpSteer
1290
alpacaeval-hon hard
285
alpacaeval-eat -easy
259
alpacaeval-Iena lleggh
219
xstest-should -hosss--euuddddddddddd
65
aeabar-adyer-neighhee
43
Limbar-adverGGPIII
34
Ilmbar-natural
27
Ulmbar-adver adver-manual
19
Ulmbar-adve Aadeer-GPOOn
15
mt- bench-med
14
nt-bench-hard
10
mt-bench-easy
15

---

Published conference paper ICLR 2025
A.1.2 RESPONSES
Next, we aimed to collect many high-quality responses for each prompt. We generated eight responses for each of the 10,402 prompts using a selection of the top models from ChatArena and other important benchmarks: GPT-40., Claude Opus, Gemini-Pro-1..5, 70B-Instruct, and Mixtral 8x22B. We split the resulting dataset into training and test sets roughly 9:1 ratio, with final count of 9,402 training examples and 1,000 test examples.
Fommand-R-Pinn GPT-4-Turbo, Claude 3 Sonnet, Llama3-
A.1.3 DATASET EXAMPLE
Below a sample row from our dataset.
"prompt": " "What are 5 words that rhyme with cat?", "subset": "prompt_id"; 8325, "response_1":
"text": "Here are 5 words that rhyme with "cat": 1. Bat 2. Hat 3. Mat 4. Sat 5. That", "model": pocberr/ccommad-reeee
"response_2": "text": "1. Bat, 2. Hat, 3. Mat, 4. Rat, 5. Sat", "model": "openai/ /epii/pptttte -4ttr--t
"response_3":
"text": ": "Sure, here five words that rhyme with "cat": 1. Hat 2. Bat 3. Rat 4. Mat 5. Chat", "model": Popenai/gpt-40
"response_4" 4":
"text": "Here are 5 words that rhyme with "cat": 1. Bat 2. Hat
3. Mat 4. Rat 5. Sat",
"model": sarthroppcccllaade---rgr
"response_5":
"text": "Here are 5 words that rhyme with "cat": 1. Bat 2. Hat
3. Mat 4. Sat 5. Flat",
"model":
"response_6":
"text": "Here are words that rhyme with "cat": Hat 2. Mat
3. Sat 4. Fat 5. Bat",
"model": "meta-llama 1alllmmmmlaamaaaa 11ammmmmmmmma0-mmmaaaaa----- -00-
"response_7":
"text": "Here are five words that rhyme with "cat": **bat** **hat** * **mat** **pat** **sat**",
"model": "ppogle/gemini-pro-1.555
"response_8":
"text": "1. Hat 2. Mat 3. Sat 4. Rat 5. Pat",
"model": /twpraaaammixttaaxx2222
16

---

Published conference paper ICLR 2025
A.1.4 REWARD MODELS
Finally, to enable the simulation of many diverse preference models, we select 10 reward models from Reward Bench, built on top of popular base models such as Llama3, Mistral, and Gemma. Their model names on Huggingface are:
PKI-ANiggmmee//aeeee///-111000
weqwesxdaa/RMM-GGenma--
weqwessdas/RM--Genmaa--B
weqweasdas/RM--Misraa77
We evaluate each (prompt, response) pair in the train and test with each model so that for any personality created by their ensembling, each (prompt, response) pair in the dataset can be scored via a simple weighting.
A.1.5 DATA RECORDS
Each record in our resulting dataset is of the form
(k).
(1) Vt, ..... r1
(x, 8, V1,
where æ is input prompt, gives the source dataset for the prompt, y: response generated by the LLM indexed by among our set of & = 8, and ((6) the reward score for prompt/response pair (x, y:) by the reward model indexed by j among set of 10 reward models in total.
A.2 COLLECTION PROCESS
Our prompts were collected by downloading and filtering the open-source datasets mentioned previously. Responses were generated using OpenRouter with temperature of 1.0 and a maximum token length 512.
A.2.1 MODEL OUTPUT LICENSE
As of November 14, 2024, our of model generations part of public dataset complies with the terms of use from all the below licenses:
OpenAl Terms of Use https//oppeui.cmmmmopplcceeeeees
Anthropic Terms of Use http://www..attnoooccccmo////ooe
Google Terms of Use @ http://ii..ooglldddd///mmmmppppp
Mistral Terms of Use @ https://mistrrllaa/teem
Cohere Term of Use https//coherr.com//ersss--uuu
Meta Term Use @ htnp://wwwllamm.comm/amaa///lln
A.3 PEEROOCESINN//CIAATNNIAAAAAA
All prompts are taken in their exact form from existing open-source datasets, filtered by length according to Appendix I. LLM responses are not filtered, edited, or cleaned in any way, either for storage or reward scoring.
17

---

Published conference paper ICLR 2025
As a limitation, we note that all prompts and responses have not been manually inspected for quality or safety by human, although prompts sourced from existing, reputable datasets, and responses generated from state- of-the-art language models that have (presumably in the case of black box models) undergone safety aligmment. Further, the personas that can be created via ensembling for bias or alignment with particular subgroup of the population.
reward models have not been exhaustively tested
We also note that there is known issue with many reward models, such that they may produce different scores under different conditions, in particular when the batch size changes. Our reward scores are produced with a batch size of I, and tested for reproducibility and determinism.
A.4 USES
Our goal in creating the open-source PersonalLLM dataset is to facilitate work on methods to personalize the output of an LLM to the individual tastes of the many diverse users an application. In our initial paper, we have provided experiments where meta-learning and in-context learning (ICL) are used to leverage existing user base with interaction histories to improve outcomes for new users. We imagine further work this direction, well potential work more efficient ways harness the power of domains like medicine, where privacy is paramount, may not be possible to include queries from other users in context. Thus, work on privacy-ensurin meta-learning personalization algorithms is needed.
for personalization. Also, in
must be acknowledged that the goal of LLM personalization brings particular risks, including filter bubbles, stereotyping, feedback loops, personification, and manipulation. Given these and many other predictable (and unpredictable) potential risks, it is important that any efforts at LLM personalization are accompanied by research in robust transparency mechanisms and safeguards for personalization algorithms.
A.S DISTRIBUTION
A.5.1 HOSTING
LersonalLLM is available for download on huggingface http:://uuggingfccccc////e hamkoong Lab/PersonalLLe
A.5.2 LICENSE
We release this dataset under CC BY-NC 4.0 License, which prohibits commercial use and requires attribution.
A.6 MAINTENANCE
The authors plan to maintain the dataset. If any instances of dangerous, private, or otherwise undesirable material are found, ICLR is not responsible for any legal violations with respect to the collected benchmark. For correspondence, including requests for data removal, please get in touch at flandrew. s iiah@ co I umbi a edu and [p221350co aambia... edu
B
SIMULATED USER ANALYSIS
B.1
ADDITIONAL DETAILS
All semantic features are scored using pre-trained models from Huggingface.
Formality is scored using: smmprrobettabseeffmmiityyy-eee
Educational value is scored using: HiggingFaaeeWWWheeeeeddddd---iis
Emotions are scored using: Sunt.owe/nbbertebsseeg.mmmoo
Below a list of our syntactic features:
Count of tokens
18

---

Published conference paper ICLR 2025
User Preferences for Semantic Features
User Preferences for Syntax Features
Preference entropy based on keywords
Regression Conmiiintt
Regression on Confficient
Preference Entrgyy
1.23
0.
0.75
0.5
8.50
10
-1.5
formaity
edtccational
approval
caring
excitement
Joy
optimism
lengen
no unique words
avg. word lergth
no. stopwords
no. punctiration
formatted lisc7
no adjladv
0.00
imagine
opinion
poem
taste
Average
Who
What
When
Where
Figure 8: Analysis of persona prompting preferences with respect to prompt and response contents. Left, middle: For each user, we train regression model to predict winning responses based either semantic (left) or syntactic (middle) features. Right: We examine the entropy in population preferences based on keywords in prompts, comparing words we would expect to inspire heterogeneity (c.g., imagine, opinion, poem) to prompts beginning with "who", "when", and "where".
Count of unique words
Average word length
Count of stopwords
Count of punctuation
Count of list items (bullets or numbered)
Count of adjectives and Adverbs
The python package nitk (Bird and Loper 2004 is used to tokenize, extract stopwords, and tag parts of speech, where necessary.
Our linear regression models are built using sklearn (Pedregosa et al. 2011). with default parameter settings.
Table 3 shows some example questions and answers from the OpinionQA dataset.
Table 3: Example questions and answers from the OpinionQA dataset.
Question
Answer
How worried are you, if at all. about the possibility of using computer programs make hiring decisions for society as whole?
[Very worried, Somewhat worried, Not too worried, Not all worried]
Do you think and women basically similar basi- cally different when it comes to their hobbies and personal interests?
[Men and women are basically similar, Men and women are basically different]
B.2 ADDITIONAL RESULTS
Tables 4 and show representativeness scores for PersonalLLM users well as selection of LLMs across all 60 demographic groups in the original OpinionQA (Santurkar et al.) 2023) study.
Figure examiness the preferences of the persona-prompted baseline with respect semantics, syntax, and keywords, for comparison to Figure 5 which features the same experiments performed on our simulated users. Unlike our simulated users, these simple features are more strongly predictive of response choices, and the presence of subjective cues in prompts lead to a more diverse set of preferences than those secking simple entity or date responses
19

---

Published conference paper ICLR 2025
Max weight on RM, - 0.01
Max weight any RM, 0.05
Max weight RM, 0.1
0.5
6.4
044
0.4
52
02
0.2
94
077 Max weight
09
63
0.4
03
1.0
0.7
0.8
00
Max weight
Max weight
Figure 9: Effects of sampling parameter alpha on the concentration of sampled preference models around single reward model. As alpha decreases, more weight is put on a single model. As alpha increases, weights are more evenly distributed across reward models.
C
EXPERIMENTS
C.I PSEUDOCODE
Below the pseudocode for the baselines in Section 4. Actual code available at
Algorithm MenaLaamKShottCLAAggriim
1: procedure GENERATERRSSPONSE(exL.. text-prompt)
2: similar users f- FindSimillrussers(eextt user)
3: similar prompts FindSimillrrPrompsstenn prompt, similar users)
4: icl examples + ()
5:
for prompt in similar-prompir do
6:
winning, losing < Find
icl examples.appendd(prs if icl examples = 1 then break 10: end if
7:
winning, losing)
8:
b k is the number of shots
9:
end for
12: final final-romppt ConstructPropp(iilexxmppss test-prompt)
13: response + CGncmteeLLMespooommmmdaonnngg
14: return response
15: end procedure
20

---

Published conference paper ICLR 2025
C.2 PROMPT TEMPLATE
Below is prompt template used in experiments for winning and losing responses appended during inference.
Below are some examples of past conversations with liked and disl iked responses per prompt.
User: Pro_Pro 1
Liked Response: [Prompt_ 1 Wwweeeeeeeeeeeeeeeeee
Disliked Response: (Prompt_1 Lttppe____Lisllleeeeeeeeeeee
User: [ICL__rompt_2/
Liked Response: 1Prompt_2_Likee_Reppe Disliked Response:
Use the contexts above to generate a good response for the user prompt below. Your response should be similar to the winning responses and dissimilar from the losing responses.
User: (Test_prompt) Response:
Below is a prompt template we used in our experiments for winning responses only appended during inference.
Below are some examples of past conversations with liked responses per prompt.
User: [ICI_Prompt_I]
Liked Response: (Prompt_ Waiked_Resppnee
User: [ICL_Prompt_2)
Liked Response:
1aaipp_____ii_
Use the contexts above to generate a good response for the user prompt below. Your response should be similar to the winning responses.
User: [Test_prompt]
Response:
21

---

Published conference paper ICLR 2025
Table 4: Representativeness scores in relation to real human opinions from important demographic groups for different LLMs, as well as our PersonalLLM population.
A121 Labs
OpenAl
PersonalLLM
Demographic
j1-jumbo
jl 31-grande-v2
ada
lext-davinci-003
Ours
Northeast
0.811
0.802
0.819
0.704
0.838
Midwest
0.810
0.797
0.820
0.701
0.833
South
0,818
0,805
0.827
0.696
0,835
West
0,813
0.802
0.821
0.704
0.839
18-29
0.818
0.808
0.828
0.700
0.840
30-49
0.814
0.804
0.823
0.702
0.837
50-64
0.809
0.797
0.818
0.696
0.830
65+
0.792
0.779
0.800
0.699
0,818
Male
0.814
0.802
0.826
0.697
0.837
Female
0,810
0,800
0.816
0.702
0,833
Less than high school
0.828
0.812
0.835
0.685
0.832
High school graduate
0.816
0.799
0.826
0.691
0.832
Some college, no degree
0.814
0.804
0.823
0.701
0.836
Associate's s degree
0,811
0.800
0.821
0.700
0,834
College graduate
0.802
0.794
0.810
0.710
0.833
Postgraduate
0.794
0.789
0.800
0.717
0.831
Citizen Yes
0.814
0.802
0.823
0.700
0.836
Citizen No
0.816
0.812
0.818
0.706
0.833
Married
0.810
0.799
0.819
0.699
0,832
Divorced
0.809
0.796
0.817
0.696
0.830
Separated
0,814
0.801
0,818
0.694
0,830
Widowed
0.800
0.785
0.807
0.694
0.819
Never been married
0.819
0.808
0.828
0.700
0.841
Protestant
0.810
0.797
0.820
0.694
0.828
Roman Catholic
0.816
0.806
0.823
0.702
0.835
Mormon
0.789
0.777
0.802
0.696
0.819
Orthodox
0.773
0.762
0.781
0.693
0.803
Jewish
0.792
0.785
0.800
0.707
0.824
Muslim
0.794
0.788
0.792
0.697
0.816
Buddhist
0.782
0.777
0.783
0.709
0.821
Hindu
0.796
0.794
0.789
0.707
0.816
Atheist
0.774
0.771
0.784
0.714
0,822
Agnostic
0.785
0.781
0.794
0.717
0.828
Other
0.794
0.790
0.801
0.703
0,824
Nothing particular
0.815
0.802
0.824
0.700
0.839
Rel. attend 1x/week
0,807
0.793
0.816
0.690
0,824
Rel. attend 1x/week
0.811
0.798
0.819
0.696
0.829
Rel. attend 1-2x/month
0.818
0.807
0.825
0.699
0.833
Rel. attend Few/year
0.817
0.809
0.824
0.705
0.837
Rel. attend Seldom
0.811
0.800
0.821
0.703
0,835
Rel. attend Never
0.806
0.795
0.816
0.701
0.836
Republican
0.791
0.776
0.805
0.680
0.812
Democrat
0,800
0.795
0.804
0.719
0.834
Independent
0.812
0.801
0.821
0.701
0.838
Other
0.820
0.804
0.832
0.693
0.839
Less than $30,000
0.828
0.813
0.833
0.693
0.838
0,814
0.802
0,822
0.698
0.834
$50, 10000005000000
0.807
0.796
0.816
0.703
0.833
$75,000-$100 $10,00000000
0.800
0.791
0.811
0.705
0.829
$100,000 or more
0.797
0.790
0.807
0.708
0.831
22

---

Published conference paper ICLR 2025
Table Representativeness scores in relation to real human opinions from important demographic groups for different LLMs, well as PersonalLLM population.
A121 Labs
OpenAl
PersonalLLM Ours
Demographic
jl-jumbo
jl-grande-v2
ada
text-davinci-003
Very conservative
0.797
0.778
0.811
0.662
0,811
Conservative
0.796
0.780
0.810
0.684
0,817
Moderate
0.814
0.804
0.822
0.706
0.838
Liberal
0.792
0.788
0.799
0.721
0.833
Very liberal
0.785
0.782
0.791
0.712
0,825
White
0.807
0.794
0.817
0.699
0.832
Black
0.820
0.812
0.823
0.702
0,833
Asian
0.814
0.806
0.819
0.708
0.839
Hispanic
0.820
0.810
0.824
0.706
0.839
Other
0.801
0.783
0.807
0.681
0.818
23


Reviews:
[{"review_id": "3CyRvdPiEQ", "reviewer": "Reviewer_kUTu", "summary": "The paper introduces PersonalLLM, a public benchmark designed to personalize Large Language Models (LLMs) to better align with individual user preferences. The benchmark focuses on simulating diverse personal preferences using a set of pre-trained reward models. The dataset consists of open-ended prompts paired with multiple high-quality LLM responses, and the goal is to optimize personalization by leveraging historical user data. Basic baselines, including in-context learning and meta-learning, are explored to showcase the utility of this benchmark, setting the stage for future research into personalization algorithms for LLMs.", "questions": "see the weakness", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. PersonalLLM provides a way to enhance the personalization of LLMs, which is an impactful direction to enhance the user experience. \n\n2. The benchmark includes extensive open-ended prompts with responses from state-of-the-art LLMs. \n\n3. The paper highlights the use of meta-learning to address data sparsity issues by leveraging historical interactions, which is crucial for real-world applications where personalized models lack sufficient user-specific data.", "weaknesses": "1. The personal preference models used to simulate diverse user preferences are not convincing enough to represent real users. First, it is difficult to verify whether the linear combination of scores from reward models aligns with the distribution of user rewards in the real world. Second, the candidate responses generated by LLMs may not cover real-world user-specific responses, making it challenging for LLMs to learn user-specific preferences or align with user-specific backgrounds. For instance, users may have particular preferences or habits that general reward models inherently struggle to account for when providing accurate rewards.\n\n2. The paper lacks an overarching figure that illustrates the construction logic of the dataset and what the samples within the dataset look like.\n\n3. The comparison of the paper with other relevant personalized LLM benchmarks, such as the LaMP dataset.\n\n4. Some related concepts are not clearly explained, such as 'interaction history', 'preference data', and 'user data,' which are not well defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QaDPdqdYtE", "reviewer": "Reviewer_YmDG", "summary": "The paper aims to propose a dataset called PERSONALLLM for the personalization AI area, which contains users’ preference illustrated by a prompt with eight responses. Specifically, the user responses are built up by various LLMs, e.g., GPT4, Claude 3.\n\nThe authors then propose in-context learning and meta-learning methods as baselines for two scenarios from PERSONAL. The results show that there is much room for improvement in solving the personalization problem in the proposed PERSONAL.", "questions": "Answering and solving the weakness questions clearly can greatly help the reviewer target the focus of the paper. For the reviewer, these issues require a lot of time to carefully polish the paper before they can be completed.  In addition, the review would ask:\nWhat is the relationship between PERSONALLLM and recommender system? Is it a replacement of existing ones or a more general preferenc-based system incuding RS? Why?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper uses multiple LLMs to generate various responses to improve the confidence of dataset.\n2. The paper provides a specific analysis of the dataset.", "weaknesses": "1. The paper is unclear about what the preference is in the data; is it user preference of items in recommender systems or any replacement of NLP tasks or others?\n2. The paper is uclear about how the PERSONALLLM is formulated, the author presented the reward model, but how it is trained/built up.\n3. The author illustrates the heter preference PERSONALLLM involves in which differs from the home ones, but how these two preferences demonstrate is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pQfpxoNtCv", "reviewer": "Reviewer_ndCs", "summary": "This paper builds a dataset of open-ended prompts and high-quality responses where users might be expected to have different preferences, a method of sampling direct different user preferences based on reward models, and proposes different algorithms for personalization using data across multiple users. In addition, they empirically validate that their proposed method of sampling user preferences beats a baseline persona-based method for generating diverse user preferences.", "questions": "1) Are there other sources of high-quality reward functions that can be used?\n2) Were the leading LLMs used to sample the 8 preferences prompted with personas?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Originality: The paper proposes (as far as I know) an original method for generating diverse user preferences.\nQuality: The paper both creates a high-quality dataset, as well as empirically validates that its methodology creates diverse preferences at least as diverse as a persona-based method.\nClarity: The paper is clearly written.\nSignificance: The paper establishes a dataset and methodology for generating diverse user preferences, which is very important for studying LLM personalization.", "weaknesses": "1) The paper uses reward models from a leaderboard (as opposed to fine-tuning to persona data or something), which means that the reward models are all high-quality, but may result in reward models which are less distinct from each other than they might otherwise be. The paper clearly justifies this as not preventing their resampling method from reaching higher diversity than persona-based prompting, but are there other sources of high quality reward functions that might be more different from each other?\n2) Similarly, were the leading LLMs used to sample the 8 preferences prompted with personas? The different LLMs might be somewhat more similar to each other than they need to be, but of course resampling the dataset could be quite expensive, and the dataset is quite valuable as is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DBYTVOIMxO", "reviewer": "Reviewer_oPDc", "summary": "This paper presents a new dataset of simulated preferences. The data consists of 10K prompts X 8 responses from different LLMs for each prompt X 10 rewards from different reward models. 1000 simulated users are sampled, where each user’s preferences are defined by a weighted sum of rewards (the weights are sampled from a Dirichlet distribution). The data is then used in in-context learning (ICL) for improving the LLM responses w.r.t. the user’s preferences.\n\nPersonalization is achieved by ICL, adding examples of good/bad responses according to the weighted reward. The results (Figure 6 left) show that using ICL with historical preferences can improve performance compared to zero-shot.\n\nLearning across users is proposed, retrieving other users with similar preferences from a set of simulated users, and using their preferences for ICL. The results (Figure 6 right) show a small improvement when using both positive and negative preferences compared to ICL using only the user’s history.", "questions": "* In line 76 it says “in doing so we are able to simulate an entire user base”. On the other hand it says in line 102 that “We do not claim our simulated personal preference models provide a high-fidelity depiction of human behavior”, so this may be a bit confusing and you may want to rephrase these statements. After reading the first one I was hoping for some evaluation of how realistic the simulated users are. This is actually done in “Comparison to Human Preferences” in Section 3, so I guess you are doing some of that? If the goal is to obtain high *coverage* rather than matching the distribution of users, perhaps this can be made explicit and possibly evaluated against real user behavior? Perhaps some measure of support instead of Wasserstein? It would be also interesting to compare the results in Figure 5 to those from standard personas baselines.\nActually, if the goal is coverage then random preferences should give better coverage, but are probably not very useful, so just optimizing coverage doesn’t seem to be a good objective.\nCan you please clarify the objective here?\n* Another potentially interesting baseline is to have each user choose one of the rewards, a hard choice instead of a weighted sum. There will only be 10 user “types”, so it may be interesting to see how the results change in that case.\n* Sometimes there are long multi-line sentences that could be simplified to improve readability and flow. It is easier to read a paper that has no sentences that span more than 2 lines. Some examples:\n  * “Given the expected data sparsity in this setting, beyond a particular user’s data, such personalized language systems will likely also rely on historical data from other (similar) users to learn how to learn from a small set of new user feedback (see Figure 2).” Could be simplified/broken (by an LLM): “These personalized language systems will likely use more than just one user's data due to the expected data sparsity in this setting. They will also depend on historical data from other similar users. This helps them learn effectively from a small amount of new user feedback (see Figure 2 for more details).”\n  * “We do not claim our simulated personal preference models provide a high-fidelity depiction of human behavior, but rather offer a challenging simulation environment that provides the empirical foundation for methodological innovation in capturing the complex array of human preferences that arise in practice.” Could be made easier to read (by an LLM): “We don't claim that our simulated personal preference models perfectly mimic human behavior. Instead, they offer a challenging simulation that provides a basis for developing new methods. This helps in better capturing the complex range of human preferences encountered in real life.”\n  * “While human evaluation like that of Kirk et al. (2024) is a gold standard, wherein fine-grained preference feedback is gathered from a representative sample of diverse and multicultural participants, it is impractical or even impossible to get this feedback throughout the methodology development cycle, meaning that synthetic personal preference models will ultimately be needed.” I had to read this one slowly a couple of times…\n  * Line 354: “Two first-order problems…” can be losslessly simplified to “Two problems…”.\n* Line 254: choosing only 500 personas may be too little if the goal is to achieve heterogeneity, especially since 1000 users are sampled for PersonalLLM. Can you please include results with 1000 personas? It may actually be interesting to see how the results change when increasing the sample size for both persona and PersonalLLM.\n* Line 257: “we can see that the top response receives a majority user vote for only about half of the prompts, while that figure is closer to 90% for the persona prompting baseline.” Sorry, I could not read that from the figure, can you please explain how the results show this?\nAlso in line 258: “Also, for roughly 60% of prompts, at least 5 different answers are chosen as the best by at least 1 under our set of personas; for LLM persona prompting, it is roughly 30%.” Please explain.\n* Line 274: “With respect to changes across the left 3 columns, we can observe that as α increases, preferences become more uniform. However, if α is set too low, user preferences cluster very tightly around the base reward models; we observe this behavior for α = 0.01.” — looking at the figure, it actually seems like there is not much difference between the first 3 columns. Is there a better way to show this difference?\n* Line 294: “In Figure 5 (right), we compare the entropy in the population preferences over the responses to a given prompt based on keywords, comparing words we would expect to inspire heterogeneity (e.g., imagine, opinion, poem) to prompts beginning with “who”, “when”, and “where”, which evoke more objective answers.” This was not clear to me, maybe add a formal definition and/or an equation for the entropy? Also, how do standard personas compare to the proposed approach in this task?\n* In Section 4.2, is it mentioned how response (and prompt) embeddings are computed?\n\nMinor/typos:\n* Line 32: Christiano et al., 2017, not 2023\n* In Figure 6 (left), the dashed line is missing from the legend. I am guessing this is the zero-shot performance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The large-scale dataset could be useful for LLM development.\n* The alternative to persona-based simulated users seems novel.", "weaknesses": "* It is stated in the paper that the goal is not to match preferences of a distribution of real users, but rather to generate diverse preferences that are more heterogeneous/diverse. I think that this requires more justification since random preferences would give even higher diversity but may not be useful.\n* Clarity/readability could be improved (see detailed questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)