PersonalLLM: Tailoring LLMs to Individual Preferences

Authors: Thomas P Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, Hongseok Namkoong
Primary area: datasets and benchmarks
Keywords: Personalization, LLM, Alignment, benchmark, dataset, reinforcement learning from human feedback, language models, RLHF, preferences

Abstract:
As LLMs become capable of complex tasks, there is growing potential for personalized interactions tailored to the subtle and idiosyncratic preferences of the user. We present a public benchmark, PersonalLLM, focusing on adapting LLMs to provide maximal benefits for a particular user. Departing from existing alignment benchmarks that implicitly assume uniform preferences, we curate open-ended prompts paired with many high-quality answers over which users would be expected to display heterogeneous latent preferences. Instead of persona prompting LLMs based on high-level attributes (e.g., user race or response length), which yields homogeneous preferences relative to humans, we develop a method that can simulate a large user base with diverse preferences from a set of pre-trained reward models. Our dataset and generated personalities offer an innovative testbed for developing personalization algorithms that grapple with continual data sparsity---few relevant feedback from the particular user---by leveraging historical data from other (similar) users. We explore basic in-context learning and meta-learning baselines to illustrate the utility of PersonalLLM and highlight the need for future methodological development.

Decision: accept (poster)
Reviewer scores: 5, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1730469360895

thread_type:
official_review

<thread_object>
creator: reviewer_opdc

<content>
summary:
This paper presents a new dataset of simulated preferences. The data consists of 10K prompts X 8 responses from different LLMs for each prompt X 10 rewards from different reward models. 1000 simulated users are sampled, where each user’s preferences are defined by a weighted sum of rewards (the weights are sampled from a Dirichlet distribution). The data is then used in in-context learning (ICL) for improving the LLM responses w.r.t. the user’s preferences.

Personalization is achieved by ICL, adding examples of good/bad responses according to the weighted reward. The results (Figure 6 left) show that using ICL with historical preferences can improve performance compared to zero-shot.

Learning across users is proposed, retrieving other users with similar preferences from a set of simulated users, and using their preferences for ICL. The results (Figure 6 right) show a small improvement when using both positive and negative preferences compared to ICL using only the user’s history.

strengths:
* The large-scale dataset could be useful for LLM development.
* The alternative to persona-based simulated users seems novel.

weaknesses:
* It is stated in the paper that the goal is not to match preferences of a distribution of real users, but rather to generate diverse preferences that are more heterogeneous/diverse. I think that this requires more justification since random preferences would give even higher diversity but may not be useful.
* Clarity/readability could be improved (see detailed questions).

questions:
* In line 76 it says “in doing so we are able to simulate an entire user base”. On the other hand it says in line 102 that “We do not claim our simulated personal preference models provide a high-fidelity depiction of human behavior”, so this may be a bit confusing and you may want to rephrase these statements. After reading the first one I was hoping for some evaluation of how realistic the simulated users are. This is actually done in “Comparison to Human Preferences” in Section 3, so I guess you are doing some of that? If the goal is to obtain high *coverage* rather than matching the distribution of users, perhaps this can be made explicit and possibly evaluated against real user behavior? Perhaps some measure of support instead of Wasserstein? It would be also interesting to compare the results in Figure 5 to those from standard personas baselines.
Actually, if the goal is coverage then random preferences should give better coverage, but are probably not very useful, so just optimizing coverage doesn’t seem to be a good objective.
Can you please clarify the objective here?
* Another potentially interesting baseline is to have each user choose one of the rewards, a hard choice instead of a weighted sum. There will only be 10 user “types”, so it may be interesting to see how the results change in that case.
* Sometimes there are long multi-line sentences that could be simplified to improve readability and flow. It is easier to read a paper that has no sentences that span more than 2 lines. Some examples:
  * “Given the expected data sparsity in this setting, beyond a particular user’s data, such personalized language systems will likely also rely on historical data from other (similar) users to learn how to learn from a small set of new user feedback (see Figure 2).” Could be simplified/broken (by an LLM): “These personalized language systems will likely use more than just one user's data due to the expected data sparsity in this setting. They will also depend on historical data from other similar users. This helps them learn effectively from a small amount of new user feedback (see Figure 2 for more details).”
  * “We do not claim our simulated personal preference models provide a high-fidelity depiction of human behavior, but rather offer a challenging simulation environment that provides the empirical foundation for methodological innovation in capturing the complex array of human preferences that arise in practice.” Could be made easier to read (by an LLM): “We don't claim that our simulated personal preference models perfectly mimic human behavior. Instead, they offer a challenging simulation that provides a basis for developing new methods. This helps in better capturing the complex range of human preferences encountered in real life.”
  * “While human evaluation like that of Kirk et al. (2024) is a gold standard, wherein fine-grained preference feedback is gathered from a representative sample of diverse and multicultural participants, it is impractical or even impossible to get this feedback throughout the methodology development cycle, meaning that synthetic personal preference models will ultimately be needed.” I had to read this one slowly a couple of times…
  * Line 354: “Two first-order problems…” can be losslessly simplified to “Two problems…”.
* Line 254: choosing only 500 personas may be too little if the goal is to achieve heterogeneity, especially since 1000 users are sampled for PersonalLLM. Can you please include results with 1000 personas? It may actually be interesting to see how the results change when increasing the sample size for both persona and PersonalLLM.
* Line 257: “we can see that the top response receives a majority user vote for only about half of the prompts, while that figure is closer to 90% for the persona prompting baseline.” Sorry, I could not read that from the figure, can you please explain how the results show this?
Also in line 258: “Also, for roughly 60% of prompts, at least 5 different answers are chosen as the best by at least 1 under our set of personas; for LLM persona prompting, it is roughly 30%.” Please explain.
* Line 274: “With respect to changes across the left 3 columns, we can observe that as α increases, preferences become more uniform. However, if α is set too low, user preferences cluster very tightly around the base reward models; we observe this behavior for α = 0.01.” — looking at the figure, it actually seems like there is not much difference between the first 3 columns. Is there a better way to show this difference?
* Line 294: “In Figure 5 (right), we compare the entropy in the population preferences over the responses to a given prompt based on keywords, comparing words we would expect to inspire heterogeneity (e.g., imagine, opinion, poem) to prompts beginning with “who”, “when”, and “where”, which evoke more objective answers.” This was not clear to me, maybe add a formal definition and/or an equation for the entropy? Also, how do standard personas compare to the proposed approach in this task?
* In Section 4.2, is it mentioned how response (and prompt) embeddings are computed?

Minor/typos:
* Line 32: Christiano et al., 2017, not 2023
* In Figure 6 (left), the dashed line is missing from the legend. I am guessing this is the zero-shot performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730512941299

thread_type:
official_review

<thread_object>
creator: reviewer_ndcs

<content>
summary:
This paper builds a dataset of open-ended prompts and high-quality responses where users might be expected to have different preferences, a method of sampling direct different user preferences based on reward models, and proposes different algorithms for personalization using data across multiple users. In addition, they empirically validate that their proposed method of sampling user preferences beats a baseline persona-based method for generating diverse user preferences.

strengths:
Originality: The paper proposes (as far as I know) an original method for generating diverse user preferences.
Quality: The paper both creates a high-quality dataset, as well as empirically validates that its methodology creates diverse preferences at least as diverse as a persona-based method.
Clarity: The paper is clearly written.
Significance: The paper establishes a dataset and methodology for generating diverse user preferences, which is very important for studying LLM personalization.

weaknesses:
1) The paper uses reward models from a leaderboard (as opposed to fine-tuning to persona data or something), which means that the reward models are all high-quality, but may result in reward models which are less distinct from each other than they might otherwise be. The paper clearly justifies this as not preventing their resampling method from reaching higher diversity than persona-based prompting, but are there other sources of high quality reward functions that might be more different from each other?
2) Similarly, were the leading LLMs used to sample the 8 preferences prompted with personas? The different LLMs might be somewhat more similar to each other than they need to be, but of course resampling the dataset could be quite expensive, and the dataset is quite valuable as is.

questions:
1) Are there other sources of high-quality reward functions that can be used?
2) Were the leading LLMs used to sample the 8 preferences prompted with personas?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601507350

thread_type:
official_review

<thread_object>
creator: reviewer_ymdg

<content>
summary:
The paper aims to propose a dataset called PERSONALLLM for the personalization AI area, which contains users’ preference illustrated by a prompt with eight responses. Specifically, the user responses are built up by various LLMs, e.g., GPT4, Claude 3.

The authors then propose in-context learning and meta-learning methods as baselines for two scenarios from PERSONAL. The results show that there is much room for improvement in solving the personalization problem in the proposed PERSONAL.

strengths:
1. The paper uses multiple LLMs to generate various responses to improve the confidence of dataset.
2. The paper provides a specific analysis of the dataset.

weaknesses:
1. The paper is unclear about what the preference is in the data; is it user preference of items in recommender systems or any replacement of NLP tasks or others?
2. The paper is uclear about how the PERSONALLLM is formulated, the author presented the reward model, but how it is trained/built up.
3. The author illustrates the heter preference PERSONALLLM involves in which differs from the home ones, but how these two preferences demonstrate is not clear.

questions:
Answering and solving the weakness questions clearly can greatly help the reviewer target the focus of the paper. For the reviewer, these issues require a lot of time to carefully polish the paper before they can be completed.  In addition, the review would ask:
What is the relationship between PERSONALLLM and recommender system? Is it a replacement of existing ones or a more general preferenc-based system incuding RS? Why?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712793777

thread_type:
official_review

<thread_object>
creator: reviewer_kutu

<content>
summary:
The paper introduces PersonalLLM, a public benchmark designed to personalize Large Language Models (LLMs) to better align with individual user preferences. The benchmark focuses on simulating diverse personal preferences using a set of pre-trained reward models. The dataset consists of open-ended prompts paired with multiple high-quality LLM responses, and the goal is to optimize personalization by leveraging historical user data. Basic baselines, including in-context learning and meta-learning, are explored to showcase the utility of this benchmark, setting the stage for future research into personalization algorithms for LLMs.

strengths:
1. PersonalLLM provides a way to enhance the personalization of LLMs, which is an impactful direction to enhance the user experience. 

2. The benchmark includes extensive open-ended prompts with responses from state-of-the-art LLMs. 

3. The paper highlights the use of meta-learning to address data sparsity issues by leveraging historical interactions, which is crucial for real-world applications where personalized models lack sufficient user-specific data.

weaknesses:
1. The personal preference models used to simulate diverse user preferences are not convincing enough to represent real users. First, it is difficult to verify whether the linear combination of scores from reward models aligns with the distribution of user rewards in the real world. Second, the candidate responses generated by LLMs may not cover real-world user-specific responses, making it challenging for LLMs to learn user-specific preferences or align with user-specific backgrounds. For instance, users may have particular preferences or habits that general reward models inherently struggle to account for when providing accurate rewards.

2. The paper lacks an overarching figure that illustrates the construction logic of the dataset and what the samples within the dataset look like.

3. The comparison of the paper with other relevant personalized LLM benchmarks, such as the LaMP dataset.

4. Some related concepts are not clearly explained, such as 'interaction history', 'preference data', and 'user data,' which are not well defined.

questions:
see the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732009765597

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Response

comment:
We thank the reviewers for their time and careful feedback. We appreciate that all reviewers highlighted the significance of the personalization problem and the potential for PersonalLLM to make a novel contribution to the field of LLM personalization along several dimensions.

- Motivated by the key methodological gap in personalizing LLMs, we provide an empirical testbed that can spur algorithmic innovations. We release a new open-source dataset with over 10K open-ended prompts paired with 8 high-quality responses from top LLMs scored by 10 different SoTA reward models.
- We propose a novel method for sampling diverse "personas" via randomly weighted ensembles of reward models, to avoid the need for opaque and expensive GPT4o evaluations or unreliable (and possibly discriminatory) persona prompting. Unlike standard approaches, our novel method creates a diverse set of preferences.
- At its core, our work is guided by the belief that the value of a benchmark lies in its capacity to drive methodological progress. We do not claim our personas replicate human behavior—this is a lofty goal and outside the scope of this work. Instead, we aim to create a rigorous and reasonable simulation environment that serves as an empirical foundation for innovation in LLM personalization. 
- Our benchmark creates new possibilities for algorithmic development, by providing a challenging enough setting that methodological progress therein can imply progress on real applications. As an analogy, while ImageNet is noisy and synthetic--differentiating between 120 dog breeds is not a realistic vision task--it provides a challenging enough setting that methodological progress on ImageNet implies progress on real applications.  We thus believe PersonalLLM represents a meaningful step forward in advancing the personalization of language-based agents.

Also, we note that we attempted to follow best practices by including a dataset card, to inform users about potential concerns and how to responsibly use the data.  We also discuss the risks and ethical implications of our dataset release in Section 6. If there are any remaining concerns that we can allay here, please let us know.

Below, we respond to each reviewer's individual concerns.  We have also submitted an updated manuscript reflecting reviewers’ concerns.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces PERSONALLLM, a novel dataset for advancing research in the personalization AI domain. The dataset captures user preferences through prompts accompanied by eight responses generated by various large language models (LLMs), including GPT-4 and Claude 3. To benchmark performance on PERSONALLLM, the authors propose in-context learning and meta-learning methods as baseline approaches for two personalization scenarios. Experimental results reveal significant room for improvement in addressing personalization challenges within the dataset, highlighting the potential for further advancements in this field.

Positive points:
+ This paper introduced PersonalLLM, which is an impactful direction to enhance the user experience.
+ The dataset proposed in the paper is well-analyzed.
+ The paper is well written.

Negative point:
- The simulated dataset can be not reliable (e.g., linear combination of the reward models)
- The comparison of the paper with other relevant personalized are insufficient