Title: Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward
Venue: NeurIPS 2025
Accepted: yes
Keywords: Personalization in LLMs

Reinforcement Learning from Human Feedback

Conversational Agent

Intrinsic Motivation
Abstract: Effective conversational agents must personalize their interactions to adapt to user preferences, personalities, and attributes across diverse domains like education and healthcare. Current methods like Reinforcement Learning from Human Feedback (RLHF), often prioritize helpfulness and safety but fall short in fostering truly empathetic, adaptive, and personalized dialogues. Existing personalization approaches typically rely on extensive user history, limiting their effectiveness for new or context-limited users. To address these limitations, we propose leveraging a user model to incorporate a curiosity-based intrinsic reward into multi-turn RLHF. This novel reward mechanism encourages the agent to actively infer user traits by optimizing conversations to improve its user model's accuracy. Consequently, the agent delivers more personalized interactions by learning more about the user. We demonstrate our method's effectiveness in two distinct domains: significantly improving personalization performance in a conversational recommendation task, and personalizing conversations for different learning styles in an educational setting with improved generalization capabilities compared to traditional multi-turn RLHF, all while maintaining conversation quality. Our method offers a promising solution for creating more personalized, adaptive, and engaging conversational agents.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents CURIO, a framework for personalization in LLM conversations. The method adds an intrinsic curiosity-based reward that drives the LLM to learn user preferences during conversations, eliminating the need for pre-collected user data. The framework models multi-turn conversations as a POMDP with hidden user type and uses a user model to generate turn-based intrinsic rewards based on user type prediction accuracy improvements. The authors test CURIO on two tasks: Exercise Recommendation for exercise strategy recommendations and Education Dialogue for teaching style adaptation. Results show CURIO improves personalization metrics compared to multi-turn RLHF baselines while maintaining conversation quality metrics. The work connects to potential-based reward shaping theory and examines reward design considerations to prevent reward hacking. The paper contributes a method for training LLMs to personalize conversations through active user learning during interactions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- An important problem statement, personalizing to users on the fly will make LLMs very useful.\n- Novel formulation of personalized conversation as a POMDP with belief updates \n- Clear connection to theoretical foundations (POMDP, potential-based reward shaping) \n- Provides a solution that doesn't require pre-collected user data \n- Empirical validation on two distinct tasks \n- Clear ablation studies comparing different reward shaping approaches \n- Transparent discussion of limitations and failure cases\n\nWeaknesses:\n- The paper aims to build long term personalization solutions and yet is limited to a single conversation with 6 to 10 turns. \n- The evaluation setup is nowhere close to a real setting. The number of conversation turns appear to be fixed, and the system prompt is injected artificially to determine the agent's strategy. \n- There is no validation provided on why the Gemini model is an appropriate judge of winner for the experiments\n- Real world users rarely fall into discrete user types defined in the two experimental setups. \n- The algorithm relies on a very accurate user model, which in real world settings must be trained with historical data. The paper does not study how much historical data is needed, and if the proposed algorithm is feasible when historical data is not available.\n- While the proposed algorithm is domain agnostic, the policy needs to be trained for each domain. The algorithm does not adapt to different applications automatically.\n\nQUESTIONS:\n1. The paper claims that adding the potential based reward shaping makes the policy easier to learn. Are there any theoretical guarantees on the sample efficiency?\n2. The base model before policy training already appears to know to ask questions to engage the user. Is it necessary for the base model to have this capability? If so, how is the base model trained to have such capability? If it is \"SFT\", how do you collect the training data? \n3. How much compute is required to train the personalized policies? \n4. Which RL algorithm is used for training? Based on the hyper-parameters, it appears to be PPO?\n\nLIMITATIONS:\nA paper on user personalization has no users involved. There is no user data, no user judgements or qualitative analysis, no experts involved for either of the two domains.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Conventional RLHF approaches use the same reward model for all users, so they are not personalized, or they rely solely on models trained with long-term, explicitly collected user-specific dialogue data.\n- These existing personalization methods are insufficient for new users or users with limited contextual data, and they cannot leverage real-time information that emerges during conversations.\n- This paper applies a reward hacking technique to RLHF to enable real-time dialogue personalization and demonstrates its effectiveness in conversational domains such as education and recommendation dialogues.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper proposes a novel method to enhance overall dialogue performance by incorporating a turn-level curiosity-based intrinsic reward in multi-turn conversations. This reward reflects how well the agent believes it understands the user, and this estimate is integrated into each turn’s response generation.\n- The approach is validated through its application to both a conversational recommendation task and a conversational education task that adapts to personalized learning styles, demonstrating its practical utility in distinct domains.\n\n**Weaknesses**\n\n- As the authors themselves acknowledge, the idea of using intrinsic rewards is not theoretically new in applications such as POMDPs. While they argue that applying this mechanism to new dialogue domains like education and recommendation demonstrates its effectiveness, it would strengthen the paper to clarify whether there is also a novel algorithmic contribution beyond the application aspect. (See related question below.)\n\nQUESTIONS:\n- In **lines 130–132**, the authors argue that applying intrinsic rewards to LLM-based dialogue enables the system to adopt response strategies that reduce uncertainty. However, similar dialogue strategies have long been feasible in traditional dialogue systems that use POMDPs and reward hacking techniques. Therefore, some readers may feel that simply moving this idea into an LLM-based context does not, in itself, constitute a novel theoretical contribution. It would strengthen the paper to provide a clearer counterargument to this potential criticism, or to clarify that the primary contribution lies in demonstrating the effectiveness of this approach in the two specific application domains, which could be emphasized more explicitly.\n- In **lines 155–168** or **lines 215–220**, it would be helpful if the paper could elaborate more concretely on how the PBRS has been adapted and realized for LLM-based dialogue. Clarifying why this adaptation represents a novel methodological innovation would better support the claim of algorithmic contribution.\n- It would be beneficial to provide more details about the scale of the data constructed for training and testing, include concrete examples of the dialogues, and report how these aspects contributed to the observed performance improvements. Sharing the code, prompts, and datasets used would also facilitate reproducibility and strengthen the practical contribution of the work.\n- In **lines 214, 254, and 291**, references are made to an appendix, but no appendix seems to be included in the submission.\n- In **line 336**, if reward hacking is employed for dialogue personalization, it would seem natural for the extrinsic reward model and mechanisms such as PBRS to target distinct aspects of personalization. For example, the extrinsic reward could model stable user preferences, such as their general learning style, while the intrinsic reward could adapt to more situational factors, such as the user’s daily learning attitude.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CURIO, a reinforcement learning framework for personalizing multi-turn dialogue with LLMs by incorporating a curiosity-driven intrinsic reward. Rather than relying solely on sparse, end-of-conversation rewards, CURIO employs a user model to maintain beliefs over user types and rewards the agent for improving the accuracy of these beliefs during the dialogue. This mechanism encourages the agent to ask more informative questions and tailor its responses to the user in real time. Experiments in exercise recommendation and educational dialogue demonstrate that CURIO achieves better personalization than standard multi-turn RLHF approaches while preserving overall conversation quality.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: 1) The paper addresses a limitation of standard RLHF for dialogue, i.e., its failure to personalize effectively in multi-turn settings without prior user data. The proposed curiosity-driven reward is interesting and well-motivated. 2) Experiments on two tasks demonstrate improvements in personalization, with empirical evidence supporting claims.\n\nWeaknesses: 1) The approach relies on having access to a reasonably accurate user model, which even if trained separately, may be difficult to develop or generalize well for real-world settings with highly diverse user populations. 2) Both training and evaluation depend on simulated user models, raising concerns about how well the method would transfer to human-involved deployments. 3) Reward hacking remains a recognized but unresolved risk, particularly with non-potential-based intrinsic rewards, which may lead the policy to exploit unintended shortcuts. 4) Experimental validation is limited to two specific tasks, leaving its generalizability to other domains or more complex dialogue settings untested.\n\nQUESTIONS:\nQ1: How does user model misspecification or noise impact the intrinsic reward signal and policy learning outcomes for CURIO? How robust is performance if the user model is biased or less accurate for certain subgroups? Could systematic errors in user modeling lead to undesirable personalization, e.g., reinforcing stereotypes or ignoring minority preferences?\n\nQ2: How does the method handle evolving or changing user traits during long conversations? The current approach assumes static user types—how would it adapt to users whose goals or preferences shift over time? \n\nQ3: How do you monitor or control for reward hacking during training? Have you tried regularization, clipping, or auditing strategies to reduce this risk? \n\nQ4: How much additional training time, memory, or GPU usage is required to include the user model, intrinsic reward computation, and belief updates compared to standard multi-turn RLHF?\n\nLIMITATIONS:\nYes The authors clearly discussed the limitations of the work", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper presents CURIO, a framework that introduces curiosity-driven intrinsic rewards into multi-turn RLHF to improve personalization in conversational agents. Instead of relying on pre-defined user profiles or static user data, CURIO encourages the model to learn about users on-the-fly during a conversation by rewarding it for reducing uncertainty about the user’s preferences. The method is grounded in the theory of potential-based reward shaping (PBRS)  and ensures that the intrinsic reward guides learning without altering the optimal policy. CURIO is evaluated on two tasks: an Exercise Recommendation setting where personalization is central, and an Education Dialogue setting where personalization enhances performance. Both shows significant improvements over standard multi-turn RLHF in personalization quality and in generalization to unseen users.

STRENGTHS AND WEAKNESSES:
By treating personalization as an active learning process within a dialogue, CURIO enables LLMs to learn about the user during the interaction. This is especially useful in cold-start settings. The experiments are well-designed and show clear gains in both personalization and generalization. The paper also dives into why the proposed approach works, with analyses of different intrinsic reward designs, user modeling accuracy. 

However, all experiments are conducted with simulated users and automatic evaluation using LLMs (e.g., Gemini). While this is understandable at this stage, some early human evaluation would have strengthened the paper claims.

QUESTIONS:
How robust is CURIO versus common issues of RLFH such as noisy and ambiguous feedbacks?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03