PREMIUM: LLM Personalization with Individual-level Preference Feedback

Authors: Yihang Sun, Tao Feng, Ge Liu, Jiaxuan You
Primary area: foundation or frontier models, including LLMs
Keywords: Preference Ranking, Tagging System, LLM Personalization, Prompt-Based, Embedding-Based

Abstract:
With an increasing demand for LLM personalization, various methods have been developed to deliver customized LLM experiences, including in-context learning, retrieval augmentation, and parameter-efficient fine-tuning. However, most existing methods are not readily locally deployable, limited by the compute cost, privacy risks, and an inability to adapt to dynamic user preferences. Here, we propose to use a tag system to efficiently characterize user profiles, inspired from the insights from personality typology and recommendation systems. Based on the observation, we present a locally deployable LLM-agnostic framework for achieving LLM personalization: $\textbf{PREMIUM}$ ($\textbf{P}$reference $\textbf{R}$anking $\textbf{EM}$powered $\textbf{I}$ndividual $\textbf{U}$ser $\textbf{M}$odeling), which obtains individual-level feedback by having users rank responses and continuously self-iterates optimization during the interaction between the user and the LLM. Notably, a variant of PREMIUM, PREMIUM-Embed, can effectively capture user preferences while being deployable with laptop-level resources. Besides algorithmic innovation, we further prepare a novel dataset, Ranking-TAGER, which provides a valuable evaluation protocol for LLM personalization. Extensive experiments validate that PREMIUM remarkably outperforms various baselines, achieving a 15\%-50\% higher accuracy and a 2.5\%-35\% higher win rate on Ranking-TAGER, as well as a 3\%-13\% higher accuracy and a 2\%-7.5\% higher F1 Score on LaMP-2. More importantly, we further demonstrate that PREMIUM can develop an effective strategy with minimal interactive data, adapt to dynamic user preferences, and demonstrate excellent scalability in both scale and functionality.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730276342510

thread_type:
official_review

<thread_object>
creator: reviewer_cwzu

<content>
summary:
This work developes a lightweight and dynamic framework PREMIUM, which capitalizes on the tag-based approaches in recommendation, to achieve alignment between LLMs and user preferences. Tags can be selected based on prompt-driven LLM generator or embedding matching. Also, it collects a AI-annotated dataset, Ranking-TAGER, specifically designed for evaluating LLM Personalization. Experiments are conducted on LaMP-2 and Ranking-TAGER.

strengths:
1. It seems as a simple and lightweight framework, with significant improvement shown in the experiments.
2. It focuses on an interesting and important research question, Personalized LLM, with a well-surveyed understanding of the current state of the field.

weaknesses:
1. One of the contributions, the construction of the Ranking-TAGER dataset, appears to be confusing. In my view, incorporating AI annotators is not persuasive. Also, Including tags as user characteristics in datasets is uncommon in real-world scenarios. Could you explain more about the contribution of Ranking-TAGER?
2. The evaluation setting of adapting to dynamic user preference, as in one of the challenges addressed, seems simple, because the changes in user preferences have already been abstracted by different tags. Could you provide experimental results on more difficult settings?

questions:
1. Please see weakness.
2. A general question: What is the precise personalization that this work focuses on? Please provide a concise concept definition of "personalization" with specific examples.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730608143634

thread_type:
official_review

<thread_object>
creator: reviewer_j6hp

<content>
summary:
This paper proposed a new framework called PREMIUM for LLM personalization with individual-level preference. The framework adopts a tag system to represent user profiles and preferences, thereby guiding the LLM to generate more user-specific responses. The framework encompasses two main variants: PREMIUM-Prompt and PREMIUM-Embed, with the latter being particularly emphasized due to its efficiency and lightweight design, enabling deployment even on devices with limited computational resources.  

The paper also introduced a new dataset, Ranking-TAGER, explicitly designed to evaluate LLM personalization methods that employ a tag-based user profile system. The dataset adopts an AI annotator to generate preference feedback with explanations.

The results demonstrate that PREMIUM-Embed consistently outperforms a variety of baselines across different datasets and setups. Notably, it achieves higher accuracy and win rates compared to methods that do not use tag system or do not leverage preference feedback.  

Further experiments provide a deeper dive into the capabilities of PREMIUM-Embed, showcasing its adaptability to dynamic user preferences, generalizability to expanded tag libraries, and extendability to binary tags.

strengths:
The paper addresses a timely and important topic: LLM personalization.

The proposed PREMIUM framework is LLM-agnostic, broadening its applicability and increasing its relevance to the research community. It is also efficient and lightweight, enabling local deployment and making personalized LLMs more accessible.

The Ranking-TAGER dataset is a valuable contribution to the research community, providing a much-needed benchmark for evaluating LLM personalization methods that utilize tag-based user profiles.

The paper conducted extensive experiments and comprehensive analysis to showcase the proposed framework's effectiveness.

weaknesses:
The method of embedding based tag recommendation is well-studied in recommendation system community. 

The assumption of preference feedback limits its applicability in real-word use cases.

It is unclear how groundtruth user tag is generated in evaluation. 

Lack of discussion on how to determine the tags in tag library.

questions:
1. In many cases, the queries asked by users may not need any personalization at all, such as reasoning tasks, or knowledge related QA tasks. How do these extra tags affect the performance of non-personalized tasks, which ideally should not be affected?

2. Would it be possible to also add more evaluation from other LLMs instead of just Qwen1.5-72B, to see how different LLMs as autorater affect performance? 

3. What is the impact of m responses in preference ranking?

4. The paper could be even stronger in evaluation, if it can also do some evaluation with proprietary LLMs such as ChatGPT or Gemini.

5. The paper assumes the assumption of preference feedback during interaction, which might limit the applicability in real-world use cases.  It might be worth some discussion as its limitation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626231605

thread_type:
official_review

<thread_object>
creator: reviewer_62bn

<content>
summary:
This paper proposes an LLM personalization method based on tag selection. Specifically, it introduces training a tag selector that adaptively selects suitable tags for each given query. The selected tags are then combined with the query to form a new prompt, which is used to request personalized responses from the LLM. The training of the tag selector relies on fitting user rankings of responses generated with different tags.

strengths:
S1. The proposed method does not include training LLMs.
S2. The proposed method looks like it can be applied to closed-source LLMs.
S3. Most parts of the paper are generally easy to follow.

weaknesses:
W1. Relying solely on tags for personalization may result in limited effectiveness, as the ability to personalize is heavily constrained by the predefined tags and their level of granularity. To my knowledge, the tag-based method is also not dominant for recommendation.

W2. The approach to achieving personalization is not clearly defined. User tags do not seem to be directly used for prompt generation but are instead used to rank the generated outputs. If this is the case, achieving personalization would require the tag selector to be specific to each user. This implies that a separate selector would need to be learned for each user, which could lead to efficiency issues and reduced learning efficacy, especially when data is sparse.

W3. Many important baselines, e.g., [1] and [2], are not compared.

W4. When applied to real-world scenarios, the method requires users to rank numerous responses, which could hinder its practical implementation.

W5. The recommendation appears to be a suitable task for evaluating personalization capabilities. Why not directly assess the method's effectiveness with the recommendation task?

W6. How does the method perform compared to soft-prompt tuning-based methods? 

[1] Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation.
[2] User Embedding Model for Personalized Language Prompting.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717362264

thread_type:
official_review

<thread_object>
creator: reviewer_b5ce

<content>
summary:
The authors propose PREMIUM, a framework for LLM-agnostic personalization.\
PREMIUM uses a tag-based system inspired by personality typology and recommendation systems to capture user preferences.\
This approach, along with a variant called PREMIUM-Embed, can run efficiently on minimal hardware, such as a laptop.\
Extensive experiments show that PREMIUM significantly improves personalization accuracy and adaptability to changing user preferences.

strengths:
1. The proposed method provides a well-structured approach to selecting user tags, which enhances the personalization process by aligning it with user-specific characteristics.
2. The introduction of a new benchmark dataset, Ranking-TAGER, specifically designed for personalized LLMs, is a valuable contribution. \
This dataset not only allows for standardized evaluation of personalized language models but also fills a critical gap in LLM research.

weaknesses:
1. The explanation of the process after tag selection in Section 4 is unclear.\
While the paper describes selecting tags based on the embeddings of the query and tags, it lacks a clear follow-up on how these tags are subsequently used to influence the LLM's responses.\
A detailed, step-by-step breakdown of how the selected tags are integrated into the LLM's response generation would help clarify this process.\
Additionally, a flowchart or diagram showing the progression from tag selection to final response would improve comprehensibility.
2. A user study focusing on the readability and user perception of the tag selection process would enhance the paper's empirical rigor. \
Specifically, assessing the clarity of the tag-based personalization process and how users perceive its effectiveness in delivering personalized responses would provide valuable insights.

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733308667161

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Additional Experiments and Clarifications from the Discussion Phase

comment:
We sincerely thank the reviewers for their constructive feedback during the discussion phase and their recognition of the novelty, clarity, and comprehensive results of our work. We appreciate their acknowledgment of our paper’s contributions, including:  
1. **Introducing a lightweight and efficient LLM-agnostic framework for LLM personalization**  
2. **Establishing a much-needed benchmark for standardized evaluation of personalized language models**  
3. **Demonstrating substantial performance improvements over diverse baselines across multiple datasets via extensive experiments with the PREMIUM framework** 

In response to the concerns raised during the review and rebuttal process, we have conducted **extensive additional experiments** to further strengthen our submission:  
-  An experiment which increase the complexity of the experimental settings for dynamic user preferences  ***[Appendix H.4 and Figure 15]***.  
- Comparative experiments that evaluate the impact of using different LLMs as AI Annotators  ***[Appendix H.5 and Table 13]*** 
- An experiment to evaluate the impact of the number of response candidates ***[Appendix H.6 and Table 14]***
- Comparative experiments on the recommendation task using a subset of the Amazon Review Data (2018) dataset, as detailed in ***[our response to Reviewer 62bN]*** (to be added to the potential camera-ready version).  

We have also **clarified several aspects** of our submission in the rebuttal and made significant improvements to the revised version of the paper. Key clarifications include:  
- The details of how the PREMIUM framework enables the generation of personalized responses with LLMs.
- Advantages of tag-based personalization compared to alternative solutions.  
- Practicality of preference feedback in real-world applications and its benefits over other feedback types.  
- PREMIUM achieves higher data efficiency, better privacy security, and data accessibility compared to existing methods.  
- Implementation details of the Personalized Movie Tagging experiments.  
- Novelty of PREMIUM-Embed compared to existing approaches.  
- A mitigation strategy for preventing extra tags from negatively affecting personalized generation.  
- Reasons for selecting open-source LLMs as the backbone models for PREMIUM.  
- Contributions of the Ranking-TAGER dataset.  
- Definition and scope of "LLM personalization" in the context of our work.  

We believe these enhancements, grounded in the reviewers' constructive feedback, significantly improve our submission. We respectfully request that the reviewers and area chair reconsider the overall contributions of our paper and the improvements made.  

We hope these efforts demonstrate the value of our work to the research community, and we sincerely thank you for your thoughtful consideration.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a framework called PREMIUM for achieving LLM personalization through individual-level preference feedback. PREMIUM utilizes a tag-based system to represent user profiles and preferences, guiding the LLM to generate more user-specific responses. The framework is designed to be lightweight and deployable on devices with limited computational resources, which is highlighted as one of its advantages. A key component of the work is a new dataset, Ranking-TAGER, intended to evaluate LLM personalization methods using a tag-based approach. Experimental results presented in the paper demonstrate that PREMIUM outperforms several baselines across various datasets in terms of accuracy, win rates, and adaptability to dynamic user preferences.

The paper’s primary strengths lie in its focus on an important and timely topic—LLM personalization—and its effort to create a lightweight, LLM-agnostic solution. The proposed framework, PREMIUM, demonstrates efficiency and scalability, which could have practical value in real-world applications where computational resources are limited. The introduction of Ranking-TAGER as a benchmark dataset for LLM personalization is also a potentially valuable contribution to the community, given the need for standardized evaluation protocols in this space.

However, there are significant weaknesses that prevent this paper from being ready for acceptance. First, several reviewers expressed concerns about the novelty of the work, particularly the reliance on embedding-based tag recommendation, which is well-studied in the recommendation systems domain. The reviewers found the use of tags for personalization to be somewhat limited in scope and applicability, as it imposes constraints on expressiveness and granularity. Additionally, the authors did not adequately justify the real-world feasibility of the proposed approach, given that the method requires users to provide preference feedback for approximately 30 rounds, which was considered burdensome and impractical.

The evaluation methodology also raised questions, particularly around the design of the Ranking-TAGER dataset. The use of AI annotators for generating feedback and the abstraction of dynamic user preferences through pre-defined tags were seen as oversimplifications that fail to capture the complexities of real-world scenarios. Furthermore, the experimental settings and baselines chosen for comparison were deemed insufficient by some reviewers, as they did not include stronger, proprietary LLMs or state-of-the-art soft-prompt tuning methods. Finally, while the paper highlights privacy and computational efficiency as advantages of the framework, these claims were not robustly supported by experimental evidence or concrete comparisons with alternative approaches.

In summary, while the paper addresses an important problem and introduces some interesting ideas, the weaknesses in novelty, practical applicability, and experimental rigor outweigh its contributions. Therefore, I recommend that the paper not be accepted at this time.