Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward
Authors: Yanming Wan
Jiaxing Wu
Marwa Abdulhai
Lior Shani
Natasha Jaques
Venue: NeurIPS 2025 poster ()
Keywords: Personalization in LLMs
Reinforcement Learning from Human Feedback
Conversational Agent
Intrinsic Motivation
Primary area: deep_learning
TL;DR: To achieve personalization in LLMs, we leverage the user model to incorporate a curiosity-based intrinsic reward into multi-turn RLHF.

Abstract: Effective conversational agents must personalize their interactions to adapt to user preferences, personalities, and attributes across diverse domains like education and healthcare. Current methods like Reinforcement Learning from Human Feedback (RLHF), often prioritize helpfulness and safety but fall short in fostering truly empathetic, adaptive, and personalized dialogues. Existing personalization approaches typically rely on extensive user history, limiting their effectiveness for new or context-limited users. To address these limitations, we propose leveraging a user model to incorporate a curiosity-based intrinsic reward into multi-turn RLHF. This novel reward mechanism encourages the agent to actively infer user traits by optimizing conversations to improve its user model's accuracy. Consequently, the agent delivers more personalized interactions by learning more about the user. We demonstrate our method's effectiveness in two distinct domains: significantly improving personalization performance in a conversational recommendation task, and personalizing conversations for different learning styles in an educational setting with improved generalization capabilities compared to traditional multi-turn RLHF, all while maintaining conversation quality. Our method offers a promising solution for creating more personalized, adaptive, and engaging conversational agents.

Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward
Yanming Wan Jiaxing Wu Marwa Abdulhai", Lior Shani Natasha Jaques
'Google DeepMind PUniversity of Washington
GGoogle Research *University of California, Berkeley
"Equal Contribution Work done during internship at Google DeepMind "Correspondence to: <ymwan&cs. washington.edie edu, jxwu@google com>
Abstract
Effective conversational agents like large language models (LLMs) must person- alize their interactions to adapt to user preferences, personalities, and attributes across diverse domains like education and healthcare. Current methods like Rein- forcement Learning from Human Feedback (RLHF), often prioritize helpfulness and safety but fall short in fostering truly empathetic, adaptive, and personalized dialogues. Existing personalization approaches typically rely extensive user his- tory. limiting their effectiveness for new or context-limited users. To address these limitations, we propose leveraging user model to incorporate curiosity-basee in- trinsic reward into multi-turn RLHF. This novel reward mechanism encourages the LLM agent actively infer user traits by optimizing conversations improve its user model's accuracy. Consequently, the agent delivers more personalized interac- tions by learning
about the user. We demonstrate method's 's effectiveness in two distinct domains: significantly improving personalization performance in a conversational recommendation task, and personalizing conversations for different learning styles in an educational setting. We show improved generalization capabil- ities compared to traditional multi-turn RLHF, all while maintaining conversation quality. Our method offers a promising solution for creating more personalized, adaptive, and engaging conversational agents.
Introduction
Deploying large language models (LLMs) in open-ended conversations requires more than just generic responses- it demands adaptation each user's unique context, including their needs, goals, personality, and evolving preferences. An effective conversational agent should feel like personalized companion, tailoring its answers, writing style, and tone as it learns about the individual. This level of personalization is especially crucial in human-centric applications such as education and healthcare, where one size does fit all. However, current training paradigms for LLMs, including reinforcement learning from human feedback (RLHF), fall short of this goal. They typically rely on single unified reward function applied uniformly across users, and optimize in single- turn interactions, ignoring long-term personalization. As a result, conventional ALHFF-raiiee models tend to average over user preferences, failing to account for individual differences [ 2
Personalization not just a luxury but often a necessity for effectiveness. In educational settings, adaptive teaching methods that respond learner's knowledge level and learning style can dramati- cally improve engagement and outcomes (3]. Similarly, in therapeutic contexts, a conversation agent must be sensitive user's emotional state and personal history, adjusting its interactions build
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Standard LLM
Personalized LLM
Today going learn about respiratory system. Let's some hunds-on activities.
Today we are gaing learn about respieatory systow. How do you usualty prefer learn?
A
A
That sounds great! I onjoy hands-on activitles!
I prefer hands-on activities.
Could you just tell me need to loarn?
uswally learn through storles.
Intrinsic Motivation in User Modeling for Multi-Turn RLHF
Belief User Type
Improvements in Beliel
Balief on User Pype
Hands-en: 50t
Hands-on: 108 Storys 906
End-of--oonverration (Sparse) Reward
Stozy: 501
Per Turn Curiusity Reward
Conversation History
Agent's Utterance
User's Response
User's Final
Reaponse
Figure I: Our work focuses on training personalized LLMs in multi-turn conversations. Standard LLM training methods treat all users homogeneous group, leading suboptimal performance for different groups (top left); while an optimal LLM can actively learn about user preferences within the conversation and then adapt to it (top right). We introduce Intrinsic Motivation in user modeling to multi-turn RLHF. Intuitively, rather than training an LLM only with the end-of of-conversation sparse reward, we add an additional turn- turn-based reward that is given by its improvement belief over the user type after generating an utterance and receiving a response, which guides the LLM to actively learn about user type and then adapt to each user throughout the conversation.
trust and efficacy 4 Intuitively, tailoring interactions to the individual can enhance user satisfaction, engagement, and overall success of the intervention, which indicates that an LLM that dynamically personalizes its behavior holds immense promise for improving user experience and effectiveness in a range of applications. Despite this importance, most existing approaches to personalize LLMs require extensive pre-collected user data or profiles. Recent works on aligning models user-specific preferences often assume access to user profile, history, or latent representation gathered prior to the conversation [5] 6 8 D]. For example, reward-modeling techniques have been proposed to infer latent user clusters or employ user-specific tine-tuning but these typically involve additional training on feedback data from each user ahead of time [5]. Such requirements limit the practicality of personalization: in real-world deployments, we may not have rich user data in advance. This gap motivates to develop methods for online personalization, where the LLM learns about the user during the conversation, reducing its uncertainty about the user's traits as the dialogue unfolds.
In this paper, propose novel method to enhance LLMs' ability conduct personalized multi-turn conversations, which we call Curiosity-driven modr-mod Reward as an Intrinsic Objective (CURIO). Intrinsic motivation, particularly curiosity, has a rich history in traditional reinforcement learning (RL) frameworks (e.g.. 100 [T] 12
14 15 ) However, these approaches have not yet been adapted to the rapidly evolving domain of LLM post-training, primarily due to the significant computational complexity involved in maintaining both an LLM policy and a separate environment model simultaneously. We are the first bridge this gap by incorporating intrinsic motivation into the LLM paradigm through an intrinsic reward mechanism. Typically, in RL applications curiosity is used reduce uncertainty in model of the world. Our key insight is that in dialog, the user is the world in which the agent interacts. Therefore, we learn a model of the user, and implement a curiosity objective that is designed to conduct conversations increase the accuracy of our user model. Intuitively, this strategy promotes dynamic balance between exploiting conversation rewards, and exploring learn more about the environment (the users). This enables the model adapt its interaction style for effective personalized conversations. To facilitate this, we have developed an advanced engineering framework capable of orchestrating multiple LLMs, efficiently supporting online multi-turn RL policy updates. Theoretically, personalized conversation can be formulated as a Partially Observable Markov Decision Process (POMDP) with belief-state updates. We show that our intrinsic reward can be designed as a Potential- -aaed Reward Shaping (PBRS) [16] approach that doesn't 't change the optimal policy [77 but enhances sample efficiency when properly implemented, showing for the first time how PBRS can be applied to train LLMs.
13

---

Figure Taareeeilluee Tillustrates our approach: the LLM receives intrinsic rewards based on improvements in its belief about the user after each conversational turn. This turn-based reward complements the sparse end-of-convvrsation reward. By leveraging multi-turn RI. combined with such intrinsic rewards, our model learns to strategically plan actions that facilitate continuous learning about the user throughout the conversation. Consequently, it progressively refines its understanding, effectively learning how to learn about the user.
We empirically evaluate CURIO on two conversational Recommendation. Given the considerable challenge of applying theoretical RL concepts to practical LLM fine-tuning, we selected these tasks as well-defined and controlled benchmarks. Our experi- ments clearly demonstrate CURIO's superior performance in rapidly adapting to individual users. Specifically, CURIO motivates the LLM to actively reduce uncertainty about users by asking insight- ful questions and generating context-sensitive responses. The generalized learning capabilities of our approach enable the L.M to quickly and effectively personalize interactions even for entirely unseen users, consistently achieving better performance compared to latest multi-turn RLHF baselines.
Dialogue 18 and Exercise
In summary, our contributions are:
novel framework (CURIO) for personalized dialogue with LLMs: We reformulate multi- turn RL fine-tuning of LLMs to include personalization, and by leveraging the user model we introduce a curiosity-basee intrinsic reward that drives the policy to learn about and adapt to the user within the conversation.
Connection with thcoretical results: We theoretically connect our approach to potential-based reward shaping, providing formal justification for our intrinsic reward design.
We establish an evaluation protocol over two distinct domains - Education Dialogue and Exercise Recommendation. This protocol assesses an LLM-based conversational agent's 's capacity to infer user traits and adapt its interactions dynamically within multi-turn dialogues.
Benchmarking personalization in conversations:
Enhanced personalization through adaptive learning: We quantitatively demonstrate that auriosity-driven approach with auxiliary user modeling significantly outperforms standard multi- turn RLHF in adapting to diverse users and demonstrates better generalization capability, while preserving conversation quality. We further provide qualitative analysis over the performances of baselines and various designs of intrinsic reward.
2 Related Works
Reinforcement Learning in LLMs. RLHF is widely used for aligning language models with general user preferences [19]. Ouyang et al. [D trained models using aggregated human judgments, resulting in broadly helpful assistants. However, conventional RLHF methods rely on a universal reward function, neglecting individual user preferences by effectively optimizing for an "average user," leading to suboptimal performance when preferences diverge 12 5 9 To address this limitation. several personalized RLHF approaches have been proposed. Poddar et al. (5) propose to infer a latent context vector for each user, enabling the reward model (and policy) to adjust that user's revealed preferences. Similarly, Chen et al. (7) learn latent preference space covering heterogeneous user opinions; their method trains a reward function that can generalize to new users with few examples by modeling each user as point in this latent space. Wu et al. 66 extract reward signals from downstream personalization tasks to generate natural language user profiles, which are then used to personalize I.MMs. Shenfeld et al. [9] formulate an individual's reward
weighted sum of base reward functions and uses a small number of preference queries to infer the user-specific weights. These personalized alignment methods indeed tailor LLM's 's behavior to different users, but they require additional user-specific info or prep work before the personalized interaction can take place.
In contrast, our method does not require any separate calibration or auxiliary user profile in advance. The personalization of the agent emerges dynamically through multi-tum interactions: the conver- sation unfolds, the model infers the user's 's traits and preferences and adapts its responses accordingly. This on-the-fly learning of user preferences means our approach can personalize real-time without an upfront personalization phase, which is key differentiator from prior RI RL.FF-aased personalization techniques. Hong et al. 20 also propose to leverage the multi-turn setting to learn about the user, but they mainly focus training offline-Ri RL agents over synthetic data to optimize goal-directed
3

---

objectives (explaining concepts or recommend activities). Our agents, however, explore how to learn user preferences throughout conversations with online-RL. Such ability actively infer user prefer- ences during the conversation can bring additional benefits in open-ended dialogues. In the absence of clearly defined task, the enjoyability of the interaction itself becomes an important consideration. Encouraging users to voluntarily share personal ideas can enhance their engagement and overall enjoyment of the conversation [2] 23], which is not realizable for traditional approaches that primarily focus on helpfulness and harmlessness.
Personalized Conversation. Personalized dialogue systems have been extensively studied in do- mains like education and therapy, demonstrating enhanced learning, adherence, and user satisfaction. Examples include AutoTutor [24] for adapting hints, virtual counselors by Bickmore et al. [25] for rapport building, and Woebot for CBT [26]. However, existing personalized agents are typ- ically domain- -omain-specific, relying on limited data that hampers generalizability. While recent large language model (LI.M) approaches are emerging. they often remain application-specifi.. In contrast, our method employs facilitating personalized interactions that generalize across diverse contexts and populations.
-agmain-ggooticc L.M capable of dynamically inferring user preferences,
Intrinsic Motivation. Our work also connects to research on intrinsic motivation and curiosity- driven learning 22 [3 in reinforcement learning. Intrinsic rewards-bonu signals not directly tied to the task's 's external goal have been used extensively to encourage agents to explore novel states or learn useful information. Specifically, VIME [13] gives an agent reward for reducing uncertainty in its dynamics model, effectively rewarding information gain about the environment. Such methods have proven effective in complex environments with sparse external feedback, as they drive the agent to discover new states and behaviors by itself. The intrinsic reward can be seen as form of reward shaping. In reinforcement learning theory, adding shaping reward (derived from potential function over states) does not alter the optimal policy, but can accelerate exploration and learning [16 [7]. Recently, Lidayan et al. 127 explicitly links the intrinsic motivation to reward shaping through theoretical framework, but their empirical analysis is limited to some simple RL domains. We bring the classical concept of intrinsic motivation into the LLM domain by designing intrinsic rewards that guide the model to actively ask insightful questions and explore user attributes in multi-turn dialogue. To our knowledge, this is the first application of such techniques in the LLM domain.
3 Curiosity-driven User-modeling Reward as Intrinsic Objective (CURIO)
Preliminaries. In traditional RLHF, a conversational task is commonly formulated as a Markov Decision Process (MDP), defined by the tuple (S, T, R. 7). At time step t, the state 8 E S represents the current conversation rollout, and the action a, E A is the response generated by our language model. The user then contributes generate the next state 8++1 by appending action a and their corresponding utterance to the current state 8t. The transition dynamics T : S x A + A(S) defines the distribution over the next state given the current state and action, and R S A - R denotes the reward function evaluating the quality of each action. The agent aims to optimize the expected cumulative reward, represented by the value function V (so) = E I>> R R'( a:) ^] where π. : + A is the policy, and E [0, 1) is the discount factor. The expectation is taken over ag (- 81) and 81+11 T(: 82, dz).
3.1 Personalization as Üser-Conditiones RLHF
To extend this formulation to personalized conversational tasks, we introduce the user type u E U. which we assume is fixed throughout the conversation. For each user u, the transition dynamics and reward function are conditioned on u, meaning that different users may respond differently and provide different preference ratings. However, the user type is unobservable in most real world settings. On one hand, extensive user background information is usually not accessible to LLMs a priori. On the other hand, when LLMs are trained on a large corpus collected by annotators from all over the world, it is inherently learning a mixture of unknown diverse users.
Consequently, the problem can be modeled as a Partially Observable Markov Decision Process (POMDP), defined by the tuple (8,4, A, r). Specifically, we define ,, = (st, u) to be the "extended" states in the POMDP, where 5, is still observable but u is unobservable. The transition dynamics and the reward function are defined over the extended states, and thus conditioned on the user type. Formally, we have T(5++1 32, at) = T(s++1 St, ae u) and R(,,, az) R(st, ae u).
4

---

Now we consider an LLM agent in this POMDP environment. Although it does not know the ground truth user type initially, it can maintain a belief over the user type and update its belief as it receives more responses from the user. Therefore, we define the helief function at time step I. as by E A(M), which is a probability distribution over all possible user types. the agent has an initial belief bo. then Bayesian belief update is formulated as:
bu+1 (u) x ((s+11 st. ae, u)be(u).
(I)
Note that in the language setting, $1+1 contains the concatenation of 8 and a; (the previous conversa- tion history and the next response), so on this recursive relation. In real settings, fho can be any belief function S + (22)) given that the belief update might be sub-optimal. Since the agent has uncertain beliefs over the true user type, commonly computes the expected rewards over the belief distribution:
define the belief function bi+1 fog (St+1) based
R² (",, be, ue) = =E (u) ae u).
The LLM agent aims optimize the expected cumulative reward starting from an initial observable prompt so. and an initial belief by. represented by the value function:
V V= bb) = E
y RR (se. be, ax) T, so, bo
(3)
t=0
where ग. : 8 x (U)) + A is the policy, and E [0, 1) is the discount factor. The expectation is taken over de (( 5t, bc) and 8141 T( 5r, at).
3.2 Introducing Intrinsic Reward to Multi-Turn RLHF
Conventional methods for training LLMs struggle to identify the optimal policy under this formula- tion. This difficulty arises primarily from two challenges. First, personalization may require obtaining information about the user over the course of a conversation by learning about their preferences or constraints in enjoyable way, and then using that information to make personalized recom- mendations, or adapt to their style. Therefore, whether the LLM has successfully personalized the conversation the user typically can only be evaluated at the end of the conversation, resulting in an extremely sparse reward signal. This sparsity hinders the model's 's ability to learn which early- early-stage actions can lead to higher future personalized rewards. Second, there exists data imbalance among different user groups within large corpora. As a result, the model tends to learn policies that perform well on the majority group 2) achieving relatively higher rewards while falling into local minimum. This discourages further exploration associated with other users.
3.2.1 Intrinsic Reward via User Modeling
To address this issue, we propose to introduce Intrinsic Motivation (IM) train a language model that can actively learn about the user type "out of curiosity", and then adapt to the preference of each user. This intrinsic reward is given by the agent's improvement in belief over the user type across the turns. The intuition for this idea is that training the model to acquire information about the user type u will better enable it to optimize the personalized reward R(s, alu)). Since we are in a large-scale language domain, we do not use traditional Bayes-based belief updates. Instead, we can leverage parameterized user model that predicts the probability distribution over user types based on the conversation rollout. This user model can be either trained prompted depending on the task. Specifically, the user model takes in the current conversation rollout s;41 after applying a and sampling the user response, and predicts the belief b141 (u) := b(u|st+1) (which is a probability distribution over all user types). With this user modeling, we can define intrinsic rewards such as the improvement in accuracy over ground truth (1+1(u") ) (u²) between two turns, or the entropy reduction (b.) (b++1) of the probability distribution.
The CURIO framework is illustrated in Figure 2 with four different LLMs involved in training. In each episode, the current policy model (which we are training) engages in a multi-turn conversation with a fixed, simulated environment model, which is meant to simulate human user. The reward model employed in traditional RLHF evaluates the entire conversation, generating an extrinsic reward Rex provided only at the end of the conversation. In contrast, the user model predicts the probability
5

---

Environment Model
Exteinsic Reward gett
End-af-Conversation
Reward Model
Propaguic Generalized Adountage
(GA5)
Entire Conversation
Prompt 1
Prompt
Promptt
Prompt
Generation
Generation
Goneration
Generation
Conversation until:
Intrinsic Reward Ril!
Belief biu (yys))
Policy Model
User Model
Bellet biu sj)
Figure 2: RL fine-tuning Pipeline for CURIO framework in one episode. We leverage a user model to obtain dense turn-based intrinsic rewards as supplement to the sparse end-of-conversatinn rewards.
distribution over user types at each conversational turn, based on the dialogue context up to that point (i.e. (u)). These probability distributions are then transformed into turn-based rewards Rymh Consequently, this method supplements the original sparse reward structure with dense intrinsic rewards, effectively guiding the policy to better understand and adapt to various user types. For more details on how the models are trained with multi-tun RL, please refer to Appendix A.I
3.2.2 Engineering Details
From an engineering perspective, both the environment and reward models are loaded directly into memory alongside the policy model, all three operate at comparable scales. At the beginning of each episode, the policy and environment models generate conversation turns in an interleaved fashion. After the full conversation is generated, the reward model evaluates it, providing an extrinsic, ond-of-conversaaton reward. This single reward is propagated to each turn as value calculated from Generalized Advantage Estimation (GAE) 28 To maintain efficiency and avoid excessive memory usage, the more computationally demanding user model is deployed separately and accessed via remote API calls. This approach enables batching predictions from different conversations and turns (time steps), allowing parallel computation of per-turn rewards. Subsequently. these per-turn intrinsic rewards are combined with previously calculated value, yielding unified reward signal for each turn.
3.3 Relationship with Potential-based Reward Shaping
is widely applied in traditional RL. has been ex- tensively studied within MDPs and later extended to the POMDP setting [17]. This series of work provides insights into how intrinsic rewards can be effectively designed. In particular, the following theorem offers fundamental justification for employing intrinsic rewards of specific forms.
Potential-based Reward Shaping (PBRS) 16
Theorem Let A(U) - R be function defined over the belief distribution b. If shape the agent's 's reward by adding the difference in the agent's 's belief between two subsequent timesteps
(s,,, az) (s; be, ac) 70(bb+1) ((b.),
where y is the discount factor, then optimizing n yields the same policy as optimizing the original reward Rh in Eq. 3 In other words, adding PBRS does not affect the optimal policy.
Intuitively, with a better user prediction, the policy can better tailor its actions to achieve higher returns. In our formulation, we treat conversation utterance the agent for improving its belief about the user between two subsequent conversation turns, after it has incorporated their response to its last question statement. We propose the following functions that incentivize the improvements in user prediction:
timestep, we can reward
Once ()) b(u") ()); (hon ace (b) log b(u"),
Oneg-ent (b) (b)) Σ b(u) log b(u),
(5)
12
corresponding to accuracy, logarithm of accuracy, and negative entropy, respectively. Noting that adding an auxiliary reward that follows the formulation above does not change the optimal behavior of the policy according to the theorem, we hypothesize that it just potentially make the policy easier to learn, since it directly encourages accurate inference of user types. We further conduct a case study on simplified setting to theoretically demonstrate the effectiveness of such rewards in Appendix C

---

In this paper, we experiment with the following intrinsic rewards that incentivize learning about the user. The first column shows set of PBRS rewards, which are in a differential (Diff) format corresponding to Equation [5] The second column shows second set of possible intrinsic user curiosity rewards that are PERS terms, but nevertheless may be reasonable objectives. For example, Information Gain is the mutual information between the random variable S1+1 and which can be written as the KL divergence DxL [b (u)] (x)] practically (after sampling 8(41) according to Houthooft et al. [3 The second column cannot guarantee the optimality of policy learning.
Potential- Reward Shaping
Other Reward Shaping
DiffAcc DiffL.ogAce
"bb+1((")) be(u")
Acc
b++1(u") ) 1//21
7 glog be+ (1=+)((())))) -logbb (x()
Ent
log [M] H7 (bb+11
DiffEnt
[(bb) -7 (H(
InfoGain
DKL (bb+ (u) [4)
Experiments
To comprehensively evaluate our method's ability to personalize conversations across diverse sce- narios, we conducted experiments using two distinct domains, which are each intended to answer specific research questions. To answer, Can CURIO improve performance on personalization tasks?, we designed a conversational personalization task Exercise Recommendation. In this task, the agent recommends an appropriate exercise strategy tailored to the user's lifestyle, health condition, and other attributes. Next, to investigate Can CURIO effectively personalize conversations when person- alization not the ultimate objective?, and How does user learning affect conversation quality?, we applied the method to an existing task Education Dialogue.
4.1 Exercise Recommendation
We first consider a case where personalization is the main objective of the conversation. Our core research question is whether multi-turn RL with improved user modeling as turn-based intrinsic reward can enhance LLMs' ability to learn about the user, thereby improving personalization performance beyond that achieved by training solely on sparse, final end-of-conversation reward.
To study this, we design new task, Exercise Recommendation, where the agent provides personalized recommendations, similar to conversational recommender applications. In this scenario, the agent functions as a health advisor, tasked with recommending personalized fitness strategies tailored to each user. To enhance realism, we designed a comprehensive list of user attributes [29] encompassing multiple aspects such as lifestyle, socioeconomic status, and health conditions etc. Consequently, to make personalized recommendations, the agent must elicit user information and preferences through multiple rounds of dialogue before choosing a strategy at the end of the conversation.
Each user has a particular backstory and a ground truth label of which exercise strategy would be most effective for them, based on their profile. For both training and inference, the agent is only rewarded when its recommendation is aligned with the user's ground-truth strategy after the whole conversation. We assume the availability of a relatively accurate user model capable of inferring user type from the current conversation. Such models can be developed in real applications by training a user classifier on user behaviors and final choices (c.g.. clicks). The dataset construction involved three steps: (1) User Attribute Definition and Sampling: For each user, we randomly sample values for 20 attributes encompassing various personal characteristics. (2) Ideal Strategy Derivation: We define a set of 8 exercise strategies and establish a deterministic logic rule that maps user attributes to an ideal (ground-trun -ruth) strategy (see Appendix D.2). For example, we may recommend team sport for those who are outdoorsy and extroverted. (3) User Backstory Generation: We utilize the Gemini model to generate a detailed backstory for each user based on their attribute values. Each simulated user is prompted only with the backstory. Please refer to Appendix D for more details.
4.2 Education Dialogue
In many other tasks, however, personalization can improve performance on the task but is only one component of the task rather than the only aim. These tasks are helped by accurate user modeling but usually have a more complicated reward function. For example, in teaching scenarios, knowing student's 's learning style or knowledge level is critical to helping the student, but the agent must
7

---

Baseline
Other Reward Shaping
Potential-based Reward Shaping
SFT MTRLHF
InfoGain 63.0(49.0)
Ent 82.0(+28.0)
Ace 84.0(+30.0)
DiffEnt 84.0(+30.0)
DiffLogAee 86.0(432.0)
DiffAce 87.5(+33.5)
54.0
68.5(+14.5)
Table 1: Success Rates (%) of different models over Exercise Recommendation. The models are evaluated by whether the predicted exercise strategy the same the pre-defined target. The values in brackets are the improvement over SFT. Overall, CURIO significantly improves the success rates.
still be an effective teacher and explain concepts clearly. Furthermore, the reward model is not conditinnd all in many real-world applications, so we hypothesize that simply introducing the intrinsic motivation can lead to a personalized dialogue agent in these scenarios as well.
We use the Education Dialog dataset introduced by Shani et al. [8]. which simulates an educational setting where an LLM agent teaches students given topic. This dataset is particularly valuable as incorporates individual student learning preferences. We specifically selected two representative and contrasting learning styles: story telling and hands-on activities. These styles serve as distinct user preferences, allowing us to assess the agent's s ability to adapt its conversational strategy.
Because Shani al. [88 only evaluate the standard conversation quality, we establish a protocol to further evaluate personalization given that the student in each episode has a ground tround-truth preferred learning style. Specifically, all the models are evaluated across: (1) personalization, assessing the agent's 's ability to tailor conversations to user's ground-truth preference, and (2) conversation quality, determining whether personalization was achieved without compromising coherence and overall quality. Automated evaluation was performed using Gemini 30 to compare a pair of conversations generated by two models and choose the better response, and we use win rate as evaluation metrics. For conversation quality, we use the same prompt proposed by Shani et al. [88. We further conduct human evaluation study of both personalization and conversation quality on the Education Dialogue
task and compare the results to our Auto Eval, in order to demonstrate the realism of the LLM-as-a- judge evaluation. Please refer to Appendix B.T for more details.
4.3 Baselines and Model Usages
Our personalized conversation tasks set in multi-turn framework, and we build upon the multi- turn RLHF pipeline introduced by Shani et al. [8]. To the best of our knowledge, few existing works have explored multi-turn RL fine-tuning for personalization. We thus compare our method to vanilla Multi-Ture RLHF (MTRLHF) baseline following their work. For RL fine-tuning, we use several LLM components: (1) the environment model and the initial policy model are SFT checkpoints of the Gemma 2B model [3 For Exercise, we prompt Gemini to generate SFT data, while for Education, we directly use the checkpoints from the original work. (2) The value model used in RL fine-tuning is also initialized from Gemma 2B. (3) We use scripted reward function for Exercise, comparing final-turn model outputs with ground truth targets, and adopt the reward model from Shani et [18] for Education. (4) We use prompted Gemma 7B as the user model. In Exercise, it infers user traits relevant to strategy selection and compute a probability distribution over strategies; in Education, directly predicts the student's preferred learning style from the ongoing conversation.
5 Results
Overall, our results show that the CURIO method can significantly help with personalization on different tasks while maintaining the conversation quality. See Appendix [ example conversations.
CURIO enhances personalization and reduces generalization gap. Table T presents the success rates of different models over the Exercise Recommendation task, which is computed by sampling 200 conversations with the trained checkpoints. The initial SFT model achieved success rate of 54%. With furthur RL training on the task reward (recommendation success rate), traditional MTRLHF increases the rate to 68.5%. With CURIO, the success rate reached up 87.5%, doubling the improvements of MTRLHE. For all the models, we picked the checkpoints with the highest validation performance within total training budget of 30000 steps prevent overfitting.
8

---

Baseline MTRLHF
Other Reward Shaping
Potential-based Reward Shaping
InfoGain
Ent
Acc
DiffEnt
DiffAcc
DiffI ogg
MTRLHF
93.04
55.70
7.91
51.90
42.72
24.05
InfoGain
6,96
42.41
0.00
29.11
9.18
0.63
Ent
50.00
57.59
39,56
43.35
49.05
44.62
Ace
92.09
100.00
60.44
70.57
85.13
64.87
DiffEnt
48.10
70.89
55,06
29.43
40.51
34,49
DiffAcc
57.28
90.82
50.95
14.87
59.49
34,81
DiffLogAce 75.95
99.37
55.38
35.13
65.51
65.19
Table 2: Side-by-side Auto Eval Results on Personalization. Each entry is the win rate (%) of conversations generated with row method, over ones generated with column method. Our models with accuracy-baac based rewards outperform the baseline model in conducting personalized conversations.
During training, we observed that traditional methods are signif- icantly impacted by a generalization gap. Figure 3 presents the training and evaluation accuracy curves, where the evaluation is performed on novel users simulated from backstories that were sep- arately held out. The baseline model exhibits pronounced trend of overfitting throughout the training. In contrast, our CURIO model demonstrates synchronized improvement in both curves before 15k steps. and significantly outperforms the baseline. We hypothesize that this is because the baseline model personalizes by memorizing mappings from superficial user details to specific strategies seen during training. Our models generalize more effectively to novel users because they are learning how learn about the user during the conversation informative questions that help distinguish between different user types.
Training (5moothed)
0.5
0.5
Mi har
Mf HEre evol
5000
30000
15080
20000
25000
30000
Figure Training curves for Exercise Recommendation.
CURIO remains effective when personalization is relevant but not ultimate goal. Table 2 shows the pairwise win rates (judged by Gemini) across all the models on personalization over Education Dialogue. We can observe that, all the accuracy-based intrinsic rewards significantly improve personalization ability within the conversations. Notably, in our human study, an exact two-sided binomial test showed that humans chose DicfLogAce over MTRLHF 75.75% of the time (p < .001), consistent with the Auto Eval rate of 75.95% which supports the validity of Auto Eval.
To better demonstrate the different behavior in actively learning about user type, we show the oracle prediction accuracy of user type given the conversation stops the third turn. This accuracy indicates whether the conversation generated by current policy and the simulated user can explicitly reveal information about the user. Figure shows that our Differential Accuracy Model can learn to ask about the user type in the first few turns starting from 10k steps, and maintaining a prediction accuracy over 90%, while the baseline conversations only exhibit the ground truth user type around 70% of the time. Notably, the accuracy is higher than random guessing on baseline conversations mainly because the student often spells out their preferences directly without being asked. In short, the baseline rarely prompts users to disclose preferences, whereas our model actively explores to uncover user attributes. Furthermore, it serves evidence that CURIO functions well without a perfect user model.
Third Tum
Baseline
0.5
Ours
0.3
0.2
0.0
10k
20k
30k
40k
50k
Training Step
Figure 4: Calibrated user mod- eling accuracy for baseline vs CURIO model (DiffAcc) at the third turn in Education Di- alogue. y-axis: b(u") 1/2. The conversations generated with our model consistently re- veal more user information.
CURIO with a proper reward choice preserves conversation quality. Table shows the pairwise win rates on conversation quality. Generally speaking, the CURIO models with potential- based reward shaping have a relatively smaller negative impact on the conversation quality. Among them, the Differential Log Accuracy reward is rated as significantly higher quality than baseline and all other intrinsic rewards. Note that the baseline trained with the extrinsic reward, which is built from the preference pairs annotated by exactly the same prompt as Auto Eval process. This explains the baseline's inherent advantage in conversation quality. However, since the extrinsic reward is not user conditioned, optimizing it will actually hurt personalization.

---

Baseline MTRLHF
Other Reward Shaping
Potential-base Reward Shaping
InfoGain
Ent
Acc
DiffEnt
DiffAcc
DiffI ogc
MTRLHF
99.05
73,42
87.34
65.19
71.84
45.57
InfoGain
0,95
2.85
5.38
0.95
4.11
0.00
Ent
26.58
97.15
62.34
26.90
57.59
23.10
Ace
12.66
94.62
37.66
19.62
42.72
13.61
DiffEnt
34.81
99.05
73.10
80.06
73.73
31,65
DiffAcc
28.16
95.89
42.41
56.65
26.27
18.99
DiffLogAce
54.43
100.00
76.90
86.39
68.35
81.01
Table Side-by-side Auto Eval Results on Conversation Quality. Each entry is the win rate (%) of conversations generated with row method, over ones generated with column method. Our models with Potential-base Reward Shaping have smaller negative impact on conversation quality, and DiffL.ogAce outperforms the baseline.
Addressing Reward Hacking, In order to train multi-turn RL models at scale, we use LLMs to simulate the user and act reward models. A limitation of this approach is that RL-based methods which attempt to maximize rewards can sometimes engage in "reward hacking", exploiting weaknesses in either the reward model or user model to obtain higher rewards in ways that do not correspond to desirable behaviors. With an extrinsic reward model that is not eser-conditioned, the baseline multi-turn RL. model adopts merging teaching style called "role-playing video", which is not one of the true learning styles, but results in spuriously high extrinsic reward. Similarly, when using the entropy-bast
intrinsic rewards that are not "grounded" (i.e., are only based on the classifier's certainty and do not make use of a ground-truth user label), we observe that the models perform really well on one particular user type, but badly on the other. For example, even though the student has expressed preference in story-telling, the teacher insists hands-on style. We attribute it to the emergence of "controlling behavior", where the policy attempts to convince the classifier that the user belongs to one particular type, rather than actually adhering to the ground-truth type. The problem with InfoGain is more serious, where the policy maximizes KI. divergence by inducing sharp shifts in the predicted user type distribution between consecutive turns. However, by using our proposed caaurccy-bssed rewards, which require predicting the actual user type rather than tricking the user classifier, we can resolve these issues and attain better performance.
Another form of reward hacking happens with non-potential-basee rewards, such as Acc and Ent, where the policy model learns arbitrarily increase the length of the conversation because the intrinsic reward is assigned to each turn on the same scale, thus hurting the conversation quality. This emphasizes the necessity of using potential based intrinsic rewards.
Conclusions
This paper introduced novel framework CURIO for enhancing personalization in LLMs multi- turn conversation tasks. By leveraging user modeling and integrating intrinsic rewards into multi-tur reinforcement learning, approach encouraged the LLM actively learn traits and adapt its responses accordingly. Experiments across two distinct domains demonstrate that CURIO improves personalization in multi-turn conversations across various scenarios, whether personalization is the ultimate or a partial goal, while maintaining conversation quality.
Limitations. Our framework assumes pre-defined and static user traits, which may not reflect the complexities of real-world conversations. Furthermore, our experiments currently rely on LLM-based user simulators for both training and testing. This approach scale, open-source datasets for conversational personalization and the impracticality using live human interaction for training. A primary consequence of this simulation is the potential discrepancy between synthetic reward signals and authentic human preferences.
necessitated by the scarcity of large-
Societal Impact. Using ungrounded personalization with CURIO method may lead to negative impacts such as controlling behavior, but grounded personalization is generally safer many real-world tasks. Future research should explore personalization within more complex, temporally- evolving contexts, and build conversational agents that can achieve robust zero-shot pluralistic alignment while ensuring ethical considerations like privacy, transparency, and bias mitigation.
10

---

Acknowledgement. We would like to express our sincere gratitude to all those who contributed to the completion of this work. We particularly wish thank Howard Zhou. Mojtaba Seyedhosseini. Andrew Lingao, Joel Z. Leibo, Shawn O' D'Banion, Jun Xie, Jianxun Lian, Yulia Tsvetkov, and Sergey Levine for providing the necessary resources, invaluable guidance and support throughout the entire process. We would like to thank Mickel Liu, Yancheng Liang, Weihang Xu, Arnab Maiti, Kunal Jha, Sriyash Poddar, Carrie Yuan, Shakti Senthil, Shanfeng Zhang, Lin Ning, Luyang Liu, Cecilia Tilli for insightful discussions and constructive feedback that helped improve the clarity and depth of this work. This research was supported by the Cooperative AI Foundation, the AW-Amazon Science Gift Hub, Sony Research Award, UW-Tsukuba Amazon NVIDIA Cross Pacific AI Initiative (XPAI), the Microsoft Accelerate Foundation Models Research Program, Character.Al. DoorDash, and the Schmidt AI2050 Fellows program. This material is based upon work supported by the Defense Advanced Research Projects Agency and the Air Force Research Laboratory, contract number(s) s): FA8650-23-C-7316 Any opinions, findings and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of AFRL or DARPA.
References
[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhimi Agarwal, Katie Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[2] Anand Siththaranjan, Cassidy Laidlaw, and Dylan HadfelldMenell. Distributional preference learning: Understanding and accounting for hidden context in rlhf. arXiv preprint arXiv:2312.088: 2023.
[3] Eileen du Plooy, Daleen Casteleijn, and Denise Franzsen. Personalized adaptive learning in higher education: A scoping review of key characteristics and impact academic performance and engagement. Heliyon, 2024.
[4] Ahmet B. Kocaballi. Shlomo Berkovsky, Juan C. Quiroz, Liliana Laranjo, Huong Ly Tong. Dana Reza- zadegan, Agustina Briatore, and Enrico Coiera. The personalization of conversational agents in hcalth care: Systematic review. Journal of Medical Internet Research, 2019.
[5] Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personaliz- ing reinforeement learning from human feedback with variational preference learning. arXiv preprint arXiv:2408. 10075, 2024.
[6] Jiaxing Wu, Lin Ning. Luyang Liu, Harrison Lee, Neo Wu, Chao Wang, Sushant Prakash, Shawn O' Baanioo,, Bradley Green, and Jun Xie. Ripf: Reinforcement learning from prediction feedback for user summarization with Ilms. arXiv preprint 2rXiv:2409.0422), 2024.
[7) Daiwei Chen, Yi Chen, Aniket Rege, Zhi Wang, and Ramya Korlakai Vinayak. Pal: Sample-efficient personalized reward modeling for pluralistic alignment. In International Conference on I Learning Repre- sentations (ICLR), 2025.
[8] Yihang Sun, Tao Feng. Ge Liu, and Jiaxuan You. Premium: Llm personalization with individual preference feedback. In International Conference Learning Representations (ICLR), 2025.
[9] Idan Shenfeld, Felix Faltings, Aldo Pacchiano, and Pulkit Agrawal. Language model personalization via
reward factorization. arXiv preprint arXiv:2503.0633 066558 2025.
[10] Jürgen Schmidhuber. Formal theory of creativity. fun, and intrinsic motivation (1990-2010). JEEE transactions on autonomous mental development, 233)2200-477, 2010.
[11] Satinder Singh, Richard I.. Lewis, Andrew G. Barto, and Jonathan Sorg. Intrinsically motivated reinforce- ment learning: An evolutionary perspective. IEFE transactions on autonomous mental development, 2(2):
70-82, 2010.
[12] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-drivee exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning (ICML). 2017.
[13] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[14] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey Trevor Darrell, and Alexei A. Efros. Large- scale study of curiosity- driven learning. In International Conference on Learning Representations (ICLR), 2019.
11

---

[15] Pawel Ladosz, Lilian Weng. Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022. ISSN 1566-2535.
[16] Andrew Y. Ng. Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the 16th International Conference on Machine Learning (ICML). 1999.
[17] Adam Eck, Leen-Kiat Kiat Soh, Sam Devlin, and Daniel Kudenko. Fotential-based reward shaping for pomdps. In Proceedings of the 12th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2013.
[18] Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Sxpektor, Avinatan Hassidim, Yossi Matias, and Remi Munos. Multi-tum reinforcement learning from preference human feedback. In Advances in Neural Information Processing Systems (NeurlPS), 2024.
[19] Natasha Jaques, Asma Ghandcharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu. and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: 00456, 2019.
[20] Joey Hong, Sergey Levine, and Anca Dragan. Zero-shot goal-directed dialogue via RL on imagined conversations. In NewrlPS 2023 Foundation Models for Decision Making Workshop, 2023.
[21] Li Zhou, Jianfeng Gao, Di Li, and Heung- Yeung Shum. The design and implementation xiaoice, an empathetic social chatbot. arXiv prerpint (1XX::8812.0898) 2019.
[22] Abigail See, Stephen Roller, Douwe Kicla, and Jason Weston. What makes a good conversation? how controllable attributes affect human judgments. arXiv prerpint MxKiv:990200880 2019.
[23] Shikib Mchri and Maxine Eskenazi. Unsupervised evaluation of interactive dialog with dialogpt. arXiv prerpint arXiv:2006. 12719, 2020.
[24] Arthur C. Graesser, Shulan Lu, George Jackson, Heather H. Mitchell, Matthew Ventura, Andrew Olney, and Max Louwerse. Autotutor: A tutor with dialogue in natural language. Behavior Research Methods, 2004.
[25] Timothy W. Bickmore, Amanda Gruber, and Rosalind W. Picard. Establishing the computer-patient working alliance in automated health behavior change interventions. Patient Education and Counseling, 2005.
[26] Kathleen K. Fitzpatrick, Alison Darey, and Molly Vierhile. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (woebot): A randomized controlled trial. JMIR Mental Health, 2017.
[27] Aly Lidayan, Michael D Dennis, and Stuart Russell. BAMDP shaping: unified theoretical framework for intrinsic motivation and reward shaping. In International Conference on Learning Representations (ICLR), 2025.
[28] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. digh-dimensional continuous control using generalized advantage estimation. arXiv prerpint 1rXivv:500.000
2015.
[29] Suhong Moon, Marwa Abdulhai, Minwoo Kang, Joseph Suh, Widyadewi Soedarmadji, Eran Kohen Behar, and David M. Chan. Virtual personas for language models via an anthology of backstories. arXiv prerpint arXiv:2407.0m 06576, 2024.
[30] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions tokens of context. arXiv preprint arXiv:2403.055300 2024.
[31] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403. 088995. 2024.
[32] Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández- Richard E Turner, and Douglas Eck. Sequence tutor: Conservative line-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pages 1645 1654. PMLR, 2017.
[33] Ronald J. Williams. Simple statistical gradient-following al algorithms for connectionist reinforccment learning. Machine Learning, 8(3-4):2292566 1992. ISSN 0885-6125.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction are explained and supported by the methodology and results sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
There is limitation subsection in both the results section and the conclusion
section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
Justification: This paper is not making any theoretical proofs. The only theorem is cited from existing paper an theoretical evidence of intuition.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper fully disclose the information needed to reproduce the experimental results. Extensive details about the tasks and the models are presented in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The access the code is not available yet. We plan to release it in the future. The data construction details are explained in the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
cablic//uuddes//oceeuubiis caapssioooolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppplgguuuaaaeoeae
Cttelee//ooSSSSSeSSSeoeaa www.ssssiicggoooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The detailed experiment settings for both training and test are presented in the main text and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The current version of the paper does not report error bars due to the extensive computation costs of running RI. experiments for L.Mss.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information regarding compute resources are listed in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt///neriipppe hc.ppublic//tiicccc.c........
Answer: [Yes]
Justification:
We preserve anonymity in this paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We draw connections between our proposed method and real-world applica- tions, and discuss the socictal impact of this work in the introduction section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the creators original owners of asses used in this paper properly credited and we follow the license and terms use.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.paperssit
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We do not release new assets by the time of submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/coodemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The full text of instruction given the participants is included. The details about compensation is given.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The potential risks were clearly disclosed to the human subjects, and the IRB approvals were obtained.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The LLM is used only for writing and editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
19

---

A Training Details
A.1 Optimization of Rewards with Multi-Turn RL
CURIO method introduces two different reward signals: extrinsic reward Ra and intrinsic reward Ri To ensure consistency, denote ".:" the extrinsic reward obtained at each turn, where RRth = 0 for < T. We further introduce KL. divergence penalty term DR. (Toll mer) against the
fixed reference model, following Jaques al. [32]. We use multiple hyperparameters to balance different reward signals, and the total reward is
F1 Czext Rt Rr
Clim R.Th (DDx.((TT)))) (Tol
(6)
Following Shani al. [8]
apply Generalized Advantage Estimation (GAE) 28 to total rewards, which aims to propagate and-of-conversation extrinsic reward Re to turn-level signals. In particular, need stand-alone value model to estimate the value function Vss,) for each turn. The propagated reward signals for each turn is then given by
fr EGG "(r 7(1 A)V(s++1))
where y is the discount factor, is the coefficient for GAE, and define V(8++1) = 0. We then use REINFORCE (33 policy gradient training approach to optimize the policy model To over these propagated rewards Å. Formally, the policy is optimizing the objective: maxg Ex, [ft].
A.2
Model Choices
The personalized conversation tasks are in a multi turiiittuunnn setting, we leverage the multi-turn RLHF pipeline implemented by Shani et al. [88 In our RI. fine-tuning process, we use the following models:
Environment Model and (Initial) Policy Model: Since are training the multi-turn policy, We leverage SFT LLM checkpoints (Gemma 2B model (3T) to serve as the initial policy checkpoint and simulate the users. For Exercise task, we prompted Gemini 1.5 Pro (30 generate conversational data for supervised fine-tuning, For Education task, we directly use the checkpoints in the original work.
Value Model: The Value model for value estimation is initialized by Gemma 2B.
Reward Model/Function: For Exercise task, we used a scripted reward function that directly compares the model generations in the final turn with the ground truth targets, reward model.
For Education task, we directly adopt the reward model developed by [8].
User Model: For Exercise task, we employ Gemma 7B model 3T predict the answers to series of user traits that are relevant to the optimal strategy from the conversation so far, and then compute the probability distribution over all strategies. For Education task, we use the Gemma model to directly predict the student's preferred learning style based on the ongoing conversation.
A.3 Hyperparameters
We followed the training recipe and hyperparameters from Shani et al. 18] On top of the original extrinsic reward, we added intrinsic reward to each turn of the conversation as described above, with coefficient coefficient weight Oint intrinsic reward when adding to the extrinsic reward to balance the scale of extrinsic and intrinsic rewards. For Education Dialogue, we choose O = 9.0 for all the settings in Education Dialogue, with other hyperparameters listed in Table 4 For all the settings, we select several checkpoints that has the highest intrinsic rewards before 30k steps, and then choose the one that performs the best on conversation quality. For Exercise Recommendation. we choose Qimi = 5.0 for DiffAcc and DiffEnt, Cimt = 1.0 for Ace, Ent, and DiffLLggAc,, and Qimi 0.1 for InfoGain. The other hyperparameters are listed in Table The user classifier temperature tau is used when calculating the probability distribution over user types from logits using Softmax function. For the settings including baselines, we select the checkpoints that has the highest validation extrinsic reward (strategy prediction accuracy) before 30k steps to avoid overfitting. Note that the number turns for Exercise Recommendation includes several conversational turns and final turn for strategy prediction.
20

---

Hyperparameters for Education Dialogue
Policy Model Learning Rate Npllicy
4c-7
Value Model Learning Rate 7yaaluyy
4c-7
Batch Size
16
KL Regularization Coefficient B
0.01
GAE Coefficient À
0.95
Turn Discount
0.95
Max Number of Turns T
10
Extrinsic Reward Weight Oext
1.0
User Classifier Temperature T
5.0
Table 4: Hyperparameters for Education Dialogue.
Hyperparameters for Education Dialogue
Policy Model Learning Rate Tpolicy
4c-7
Value Model Learning Rate Vvvaluey
4e-7
Batch Size B
16
KL Regularization Coefficient
0.02
GAE Coefficient X
0.95
Turn Discount
0.95
Max Number of Turns T
6
Extrinsic Reward Weight Oest
3.0
User Classifier Temperature
5.0
Table 5: Hyperparameters for Exercise Recommendation.
B Extended Results
B.1 Human Study for Education Dialogue
We conducted a human evaluation study of both personalization and conversation quality on the Education Dialogue task. We invited 10 students participate. Each participant was asked to evaluate dialogues generated by three models across 20 different settings (topic/user type). Evaluations were conducted through pairwise comparisons across two dimensions: personalization and conversation quality, resulting in a total of 200 pairwise comparisons per dimension.
An exact two-sided binomial test showed that human evaluators chose DiffLLogAce the MTRLHF baseline in 152 / 200 pairwise comparisons on personalization (preference = 0.76, 95% CI [0.69, 0.82J), p .001. Importantly, increase in personalization did not reduce conversation quality: DiffLogAce was preferred in 90 / 200 quality judgements (preference = 0.45, 95% CI [0.38, 0.52)), and the same exact test found no significant difference from chance (p .179). In Table 6 we provide two tables containing the full results of the human study, which correspond to Table and 3 We note that the human evaluation results reasonably consistent with those obtained through Auto Eval.
B.2 Results for Education Dialogue across Different User Types
We present the Auto Eval results for Education Dialogue on two different user types in Table 7 and Table 8
We found that the pairwise win rates of these models over personalization can vary significantly when the user ground truth label is different. The values in red color are the win rates of entropy- based models against the baseline, where both models are performing very well on the second user type, but extremely badly on the first one. This further supports our observation that agents trained with entropy- based reward shaping functions may converge one particular user type that is not necessarily correct. That is the reason of using accuracy- securacy-based reward shaping functions. For conversation quality, this phenomenon is longer significant, and the DiffI.ogAce model is still outperforming all the other models.
21

---

Personalization
MTRLHF
DiffAcc
DiffLogAce
Baseline
30.50
24.25
DiffAcc
69.50
46.75
DiffLogAce
75.75
53.25
Conv. Quality
MTRLHF
DiffAcc
DiffLogAce
Baseline
60.25
55.00
DiffAce
39.75 45.00
46.25
DiffLogAce
53.75
Table 6: Human study results on Education Dialogue.
B.3 Questions Distribution for Exercise Recommendation
In the main paper, hypothesize that the severe overfitting of the baseline model is because the baseline model personalizes by memorizing mappings from superficial user details to specific strategies seen during training. Our models generalize more effectively novel users because they are learning how to learn about the user during the conversation asking informative questions that help distinguish between different user types. In Figure we present the questions distribution of CURIO model, RLHF baseline, and SFT initial checkpoint. Note that Occupation is relevant attribute because gives us an idea about the Socioeconomic Status. The SFT and RLHF model are asking about and memorizing irrelevant attributes like name and hobbies, while our CURIO model is able to find a key attribute-introvereed vs serroverteddduring the training process. None of the models is able ask about motivation, probably because this attribute is only helpful in distinguishing between Strategy and See section D for full details of user traits and strategies.
SFT
RIHF
CURIO
100
50
Age
Occupation
Health
InVOutdoor Personality Motivation
Name
Hobbles
Gender
Relationship
Relevant Attributes
Irrelevant Attributes
Figure 5: Questions distribution of CURIO model, RLHF baseline, and SFT initial checkpoint. The left ones are relevant attributes, and the right ones are irrelevant the strategy recommendation.
C Case Study: Multi-Turn Conversation as Combinatorial Bandits
In this section, we briefly discuss how the personalized multi-turn conversation problem can be connected to some existing theoretical frameworks. In particular, we will simplify the Exercise Recommendation task into combinatorial bandits problem, and provide insight why CURIO method can help improve the efficiency.
C.1 High-level Intuition
A multi-turn conversation of fixed length H is viewed as single episode. During the episode the agent asks H questions. We assume the order of these actions does not affect the final utility, since the reward is only given at the end of the conversation based on the final recommendation. Therefore.
22

---

First User Type
Baseline MTRLHF
Other Reward Shaping
Potential-baset Reward Shaping
InfoGain
Ent
Acc
DiffEnt
DiffAce
MTRLHF
0.96
1.00
0.08
1.00
0.37
0.27
InfoGain
0.04
0.85
0.00
0.58
0.00
0.00
Ent
0,00
0.15
0.00
0.06
0.00
0.00
Acc
0.92
1.00
1.00
0.99
0.85
0.66
DiffEnt
0.00
0.42
0.91
0.01
0.00
0.01
DiffAcc
0.63
1.00
1.00
0.15
1.00
0.43
DiffLogAce
0.73
1.00
1.00
0.34
0.99
0.57
Second User Type
Baseline MTRLHF
Other Reward Shaping
based Reward Shaping
InfoGain
Ent
Acc
DiffEnt
DiffAce
Diff'
MTRLHF
0.90
0.11
0.08
0.04
0.49
0.22
InfoGain
0.10
0.00
0.00
0.00
0.18
0.01
Ent
1.00
1.00
0.79
0.81
0.98
0.89
Acc
0.92
1.00
0.21
0.42
0.85
0.64
DiffEnt
0.96
1.00
0.19
0.58
0.81
0.68
DiffAce
0.51
0.82
0.02
0.15
0.19
0.27
DiffLogAce 0.78
0.99
0.11
0.36
0.32
0.73
Table 7: Personalization Auto Eval results for Education Dialogue on two different user types. For each row's model, the values represent the percentage of wins achieved against the model specified in each column.
the whole episode can be abstracted as the selection of an unordered subset of size k := H from finite action catalogue. The delayed scalar outcome (customer satisfaction, purchase, etc.) arrives after the entire dialogue and cannot be decomposed into per-turn signals observable by the agent. Hence only the aggregate reward R2 is available, matching the full-bandit feedback assumption.
C.2 Thcoretical Formulation
Base arms. Let A {I....,K] K} be the catalogue, K > 1. Each arm e A corresponds to question or any atomic conversational move.
Episode action. In episode = 2. the agent selects super-arm S, S. A with fixed cardinality [Sel k. The choice S, encodes the entire conversation, since we assume that the response given by the environment is deterministic.
Latent structure. There exists an unknown subset of useful arms U C A size m < K.
Reward signal. After pulling S, the agent receives a scalar reward R4. The reward can be defined in various ways depending of the problem structure. Here are two simplified settings.
1. Additive (linear) reward
R
Σ Xa.t: aES.
E[ = E 10.
Only useful arms have Pa 1: non-useful arms have Ha 0. The Xa,t can be i.idd Gaussian random variables with the given mean and some fixed noise.
2. All-or-nothing (conjunctive) reward
R; = 11U* C S.).
i.e. R; = 1 iff every useful arm is contained in the chosen super-arm (equivalently S, U* when m k)) and 0 otherwise. Note that we assume m
this case.
Feedback type. The agent only observes Re; it does not observe the individual Xa,t or which arms were responsible for the reward. This is the full-bandii feedback setting.
23

---

First User Type
Baseline MTRLHF
Other Reward Shaping
Potential-baset Reward Shaping
InfoGain
Ent
Acc
DiffEnt
DiffAce
Doc
MTRLHF
0.99
0.73
0.88
0.65
0.78
0.51
InfoGain
0.01
0.02
0.04
0.01
0.01
0.00
Ent
0.27
0.98
0.62
0.30
0.67
0.39
Acc
0.12
0.96
0.38
0.18
0.45
0.22
DiffEnt
0.35
0.99
0.70
0.81
0.75
0,42
DiffAcc
0.22
0.99
0.33
0.54
0.25
0.20
DiffLogAce
0.49
1.00
0.61
0.78
0.58
0.80
First User Type
Baseline MTRLHF
Other Reward Shaping
Potential-basee Reward Shaping
InfoGain
Ent
Acc
DiffEnt
DiffAcc
DiffI
MTRLHF
0.99
0.74
0.87
0.66
0.66
0.41
InfoGain
0.01
0.04
0.07
0.01
0.07
0.00
Ent
0.26
0.96
0.63
0.23
0.48
0.08
Acc
0.13
0.93
0.37
0.21
0.41
0.06
DiffEnt
0.34
0.99
0.77
0.79
0.73
0,22
DiffAcc
0.34
0.93
0.52
0.59
0.27
0.18
DiffLogAce 0.59
1.00
0.92
0.94
0.78
0.82
Table 8: Conversation Quality Auto Eval results for Education Dialogue on two different user types. For each row's model, the values represent the percentage of wins it achieved against the model specified in each column.
Learning objective. Denote by Ne the selection rule for Sy. Since we are considering the L.LM training process, the objective should be the best-subset identification in pure exploration setting. The policy needs find Ü such that Pr(Ū U*3 1 while minimising the sample complexity (number of episodes).
C.3 Per-Turn Reward Shaping as Aemi-Bandit Feedback
Assume we augment the delayed episode level reward with an intrinsic reward signal delivered immediately after each conversational move:
ra(a)= = 1(a is useful} E (0, 1},
a a St.
where S₁ is the size-k set of actions chosen at turn t of the dialogue. This shaping term informs the agent, right away, whether selected action belongs the useful action set U* Our CURIO method is basically introducing this type of reward signals (where reward is not exactly 1 but the gain in probability of the correct user type or the drop in entropy of the probability distribution), since the model can learn whether each of its action help improve the user prediction.
The shaped reward (r:(n) outcomes are revealed for the arms that were played, while arms not in S; remain unobserved. Hence the shaped conversational task and the canonical semi-bandit model share identical information structure and can be analysed similarly.
E St} is exactly the definition of semi-bandit feedback: arm-level
D Task Design for Exercise Recommendation
We generate 1000 simulated users, and split them into 800 for training and 200 for evaluation. Each user mapped to one particular ground truth strategy among different exercise strategies.
1. User Attribute Definition and Sampling: We defined 20 user attributes encompassing range of personal characteristics. For each simulated user, we randomly sampled values for each of these attributes, creating diverse population.

---

2. Ideal Strategy Derivation: We established a deterministic logic rule that maps user at- tributes to an ideal (ground-truth) exercise strategy. For example, we may recommend team sport for those who are outdoorsy and extroverted. The mapping rules are listed in the appendix. Among the defined attributes, were designated as relevant factors influ- encing the recommendation, while the remaining 15 served as background characteristics, emulating the complexity of real- world users.
3. User Backstory Generation: To provide contextual richness and ensure consistent agent behavior, we utilized the Gemini 1.5 Pro model [0 to generate a detailed backstory for each user based on their attribute values. These backstories were then used in prompts for the environment model, ensuring that the environment model remained consistent with the user's defined characteristics.
D.1
List of User Attributes
Name: from 1000 random names
Age: randomly sampled between 15 and 65
Sociocconomic Status: randomly sampled from (low, medium, high)
Relationship Status: randomly generated
Location From: randomly generated
Occupation: randomly generated
Education: randomly generated
Religion: randomly generated
Language Spoken: randomly generated
Have injuries physical limitations: randomly sampled from (True, False
Personality: randomly sampled from (introverted, extroverted)
Motivation on plans: randomly sampled from (highly motivated, struggling with
motivation)
Enjoy outdoor or indoor activities: randomly sampled from (ouldoorsy, indoorsy)
Hobbies and Interests: randomly generated
Gender Identity: randomly generated
Political Views: randomly generated
Places Traveled: randomly generated
Pet Ownership: randomly generated
Sibling Information: randomly generated
Life Goals and Ambitions: randomly generated
D.2 Logic Rules for Optimal Exercise Strategy Recommendation
1. Recommend walking in parks: For those who have injuries and are outdoorsy.
2. Recommend yoga or tai chi at home: For those who have injuries and prefer staying indoors.
3. Recommend jogging or hiking: For those who do not have injuries, are outdoorsy, and are introverted.
4. Recommend team sport: For those who do not have injuries, are outdoorsy, and extroverted.
5. Offer discount on a gym membership: For those who do not have injuries, prefer indoor activities, and have a low socioeconomic status.
6. Recommend home gym equipment: For those who do not have injuries, prefer indoor activities, have a higher socioeconomic status, are introverted, and are highly motivated.
We manually set "Have injuries physical limitations" to be True if the user's age is at least 55.
25

---

7. Recommend personal trainer at the gym: For those who do not have injuries, prefer indoor activities, have higher socioeconomic status, are introverted, and struggle with motivation.
8. Recommend group class at the gym: For those who do not have injuries, prefer indoor activities, have a higher socioeconomic status, and are extroverted.
D.3 Code for User Generation
import numpy np
def get_dict_sttr (imput_str)
input__tz input stt.iipppp
left Soppttsstr.fin((115
right = input_str.rfindd (nput_sstr(left.righhh-.
def generate_ _usee____e_____e ceartse_ssrr__poffil proffle (name=None):
""*Generates a user profile.
Arga:
name: The name of user.
use_gemma: Whether to use gemma to generate the profile.
Returns:
tuple the profile and a dictionary of useful information.
socioeconomicsstane stttticsssssuueeee = np. ( ['low', 'medium', 'high'J, p- 20 0.6, 0.2]
)iitem()
age rp..random. 65)
profile =
"("amee: "Xs", "age": %d, "socioecooooic_statii" "%s"," "xelationship_status" <relationship>, prccatio_frronl <location>,' "occupation": <occupation>, "education": <education>, "religion";" <religion>, "language spoken": <language spoken>} (name, age, socioeconomic_stato
)
demographic_proopg
Here profile for random person json format. 'n'
profile
\nPlease randomly generate the demographic information for them and?
fill in blank in the json format. Output the json format only.
profile
profile = get_dict.sst((roffie))
if age >= 55:
True
else:
have Lave_injuries_or. hrw.ree.oopplysaaaalllttttttttte limitationn np. randamoom r..ccom.choicee choice
[True, False], p=[0.1, 0.9]
).item()
personality mp.randoo ra.dom. www.dom.choic((( ['introverted', 'extroverted'), p- [0.6, 0.4]
) .iitem()
motivation_on_ np. wo..raddom choice(
['highly motivated', 'struggling with motivation'], p- [0.5, 0.5]
b.item()
gally_outdoor_ooo or aetboorractvitties np. random. www..ommcociiie((( choice(
['outdooray', 'indoorsy'], p= p0,4.4, 0.6] ) ).item()
profile
profile[:-1]
+ +,, hhave_injuries_cc
%s, "personality"; "%s"," "notivation_on_p pn_annss" %s, conjoy_outdoon indoor activities" orttorrtttttoo...titit. "%s"}>
26

---

return str hhaaee.iijrrree.....ppaaaaaallll 1ower
personality,
Notivation__nn_pamm.
profile
profile[:-1]
"hobbies_and and interests": <hobbies_and interests>,
"gender_ied iidieri_dd iinuee__dentity>,
;
politicaalvvees>> "places_traveled"
selacesstrraeled>.
"pet_ownership": <pet_oweesship>, "sibling information";
<sibling_ <eiblingginformation) information>, "life_ Saree_goll__aar andd___nd____ andiiiiiiio <life_ lafe__goas__add and ambitions>)'
_info_prompt
'Here is a profile for random person json format. And
profile
AAPleasee randomly generate the personal information for them and fill' in blank in the json format. Output the json format only.
profile generate_ bnnerate_by_LLMperraamm Syy_LMMppeaaaal __noal_rooooof_ttt._
profile = aet_ddctt_atrprrrfft (profile)
useful_proflle, profile_dicit....
'name'; name,
'age': age,
"socioeconooic_ssatuuus socioeconooic_sstauue
haaee Phaveeinnuruiiii weiiiinuiisseee pt.pppsccaaal_pp.aaaaaaaa pbysicall_imitationns. limitations':
lave_injuries_or.c phwsicalllimitaetteeeeelalllleee
"personality': personality.
mmotivation_oonplaams" motivation_on_pp on_paanss
'enjoy_outdoor.on www.:iioorractivittaie
return profile, useful.profile_dict
def get_ground_trrue truth wwth.tttraattgg(ppiilla...l
if profile, (it.edicccttheeeeeeeeee baaictthavennjjuriss... or_.ssicchpllll__p_p limitations']:
profile_dice dictteeejoy......eeaaaeaeaoaoeee eutoooouudoor, or, indoor activities'] 'outdoorsy'; return else:
return 2
else:
d'ejoeeeottoooooooooooooooooooe or www_indoo__ctttn (atalorracttvitiees)) 'outdoorsy':
if profile_dict ppalcctttpposoollty]]] 'introverted';
return 3 else:
else: if proiieeddict((oocoocooooiiiicee return 5 else:
>low?:
if profilee dit.dd/cprrssaallttyl] 'introverted';
profile_ ctallleddctmmmmtttiii o 'highly motivated':
return 6 else: return
else: if profile_de (eciiled_it_/le (ireeeddic(''erreeee pperssoalaalll 'extroverted'; return 8
def generate, backstory prompt (
'Here is a profile for randon person in jaon format. An profile
27

---

AmPlease write backstory in first person view based on the given' profile. Please note that your story needs information, but it does not have to follow the order provided,
all the?
prompt +-
"Output format output your response in json, where you provide the? following: An
prompt += ""backetory"; <STRING_OF LFOUURR NOURR_BACSTTORYS)
backstory = generate_by LLMMoooopp
backstory get_dict_str (backstory)
return backstory
D.4
Oracle Classifier for Exercise Recommendation
Note that we using oracle classifier as the user model in Exercise Recommendation. An LLM involved answer some True/False questions help determine whether particular user attribute has been revealed during the conversation. The classifier finally outputs probability distribution over all possible user types.
self. questions = {
:
'Is the person having low socioeconomic status (very concerned about expenses)? Answer True, False, or Unknown.
iijjury': (
Does the person has special physical'
limitation? Answer True, False, Unknown.
"extroverted":
'Is the person an extroverted person? Answer True, False, or Unknown.
mmotivttion:
the person motivated to take on new plans? Answer True, False, or Unknown.
), outdoor';
person an outdoor person? Answer True, False, or Unknown.
)
}
def selaasccgggeeeeee generate be LLM(lf,,,,,, queries): [YOUR CODE GOES HERE] return responses
def _ee__probbssell conversations):
keys (self .uesttion..ees
prompt
'The following is the conversation between service agent and a? customer: \n
queries 0
for conversation in conversations:
for key in keys: queries.appendu.
prompt
conversation
Please answer the following question about the customer:
self questiion((ke))
Ansuer:
def get_probs__ hett_prbb__fomm_o_o answers f (answers):
probs list()
28

---

if musvers("lov_BBB) = -1:
answers 1aw = 0.2
if ansvers(Pinjuryy
-1:
ansvers 0.25
if ansvers(Pextzovertert == -1:
answers[[extrrovortet]]] = 0.4
if ansvers[/notivatiin]]]
ansvera 0.5
if amavers('outdoon) == -1:
answers ('owers('outdoor]) - 0.4
probs. . append ['injury'] answers 'ouudoor']]
probs. append (answers (anseerre ['injury'] probs. append (
answers (ouudoor']) (])
(1 maasers("injuyy"
answers ['outdoor']
(1 answers(' "extrrsseettroovetee"""")
probs. (
(1 answers Sesvers(iijuutee ('injury']) answers ['outdoor'] answers (eextroverted')
proba append(
(1 anavera (1ijuury))
* (1 answers ('outdoor'"
( ['low_SES*]
probs. append(
(1 ansuers ('injury'])
* (1 answers (rororssoouuoorrr ('
(1 answers ('lou_SES')
answers mmaerrs['extinn ["extrovertedd]]
* anawera ['motivation']
probs. . append(
answers ( ('injury')
* (1 answers (' ['outdoor'] '])
* (1 anawers Plovers[Poww. [1lo__EE'
* (1 answers (extrovertedd]
(1
'motivation'])
probs. append(
(1 answers ('injury')))
* (1 answers ("ottdoor )
(1 imswers"'oww_EES
answers (' ('extroverted')
probs ar.array (probs)
return probs
for query in queries:
queries_withh_eeppmeee . append [query, "True'])
queries_.with_responee wtwpeepeeee..ppeddddeeee. append [query,
'False'))
responses = wtw.rreeppseee.ppeedddedeee append [query, UUnknosn 'I)
queries_vith_reeseeeeeee
queries_wvth__eeppnee
prob_list
for conv_id in range(len (conerrtttc))))))::on))):::
answers dict() for index, key in aumerate(keys))
false_logits responses [conv_ len (keys) index) 3]
true_logits responsea[(conv
(keys) + index) aaknovm_logite = responses [ (conv_ id * len + index) 3 + 2] if true_logits > false_logits and true_logits > unknown_logits:
29

---

answers [key] =
elif false_logis _ogetittsss__ true_logits and false_logite unknown unknown_logitss answers [key]
else:
answers [key] -1
probs get_probs arromaanswers (answers)
prob_lias gob_listtappennee deliitt.ppeeddprrobs
return np. spaoo _ist, axis=0)
D.5 Scripted Agent for Exercise Recommendation
Here we also provide an optimal scripted agent that show the upper bound performance for the conversational agent on Exercise Recommendation task.
class OptimallSeriptedAggnn:
def __initt_ (self)
counter selfecuunte self. ull.uutterances =
"Hi! Do you have any physical limitations?", "Thanks for letting me know! Would you prefer indoor or outdoor activities?",
"Sounds good! Are you introverted extroverted?",
"Got it! Are you comfortable with your finances?",
"Thanks for all the info. Last question: Do you sometimes feel unmotivated about new plans?",
"Okay, I will wrap the suggestions for soon!"
self keys dict()
self quu.siuusinnss
"low_SES" "Is the person having low socioeconomicc status? Answer True or False only.",
"injury": "Does the person has special physical limitation? Answer True False only.",
"extroverted"; "Is the person an extroverted person? Answer True or False only.",
"motivation": "Is the person motivated to take plans? Answer True False only.",
"outdoor"; the person an outdoor person? Answer True or False only.",
def generate_utterancee ustef, so_far): if calf.counter == 0:
utterance self uttteraneeac ask injury elif self. counter 1:
self serttgge_eeey"jjuusst
so_far)
utterance self .utterances [1] ask outdoor elif counter =- 2:
self get_key ("outdoo""""" so_far)
if self keys ("ijjury""
utterance self utterances[$] # end of conversation for 1 and 2 else:
if self. keys l ["outdorr"; utterance self utterances # extroverted
else:
utterance self uttf.uuttranceee [3] # ask low_SES elif self. counter == 3: if kelf.keys("ottdoor"]]
utterance self. utterances end of conversation for 3 and 4 else:
self get_okey""ooxSE3 so_far)
if self faffkees[""1 :"
utterance self.ut (t5) # conversation for else:
utterance - self. utt.uutttrance5222 # ask extroverted
elif self. counter == 4:
30

---

self.get_aee get__kyy( "eeyereetoveereee"" so_far) if self keys ["
utterance self utterances end of conversation for 8 else:
utterance self utterracece))
else:
utterance self utterancceee [5] end conversation for and (optional for max_ max_length=5)
self counter += 1
if counter =- len (self. utterances))
self.resetll
return utterance
def (self):
self.counter
self. counter bels dict()
def get_kes ket_keyyself, key, so_far):
prompt "The following is the conversation between a service agent and customer:\n"
prompt so_far
prompt += "Please answer the following question about the customer: self. questions [key]
def sopping_from_ttie coan_st__ttt wwrrro_tr_tt_beeeeeeoeoooo (s)) if "True" in return True elif 'False' in s or 'false' in s: return False else:
'true' in
print("Value Error!", key, so_far)
choice((True,, return mp. random. choice [True, False])
answer generate_b Ly_MMe # Defined in outer acope.
answer mapping_frome from_str_to_boot to
self keys [key] answer
E Prompts
E.1 Prompts for RL-finetuning on Exercise Recommendation
E.1.1
Environment Prompt
You are simulating a customer, this is your backstory:
[BACKSTORY]
Here a conversation between the customer and agent. The agent will ask you about your personal information so that it can give you suggestions on doing exercise. You need to complete the current utterance customer. Remember to stick to your backstory while talking to the agent, and keep your answer short and concise. The conversation starts now.
Start
E.1.2 Agent Prompt
You are simulating a helpful agent. Here a conversation between the agent and a customer. The agent needs to give suggestions doing exercise to the eustomer afterwards, it should ask the customer for more personal information. You need to complete the current utterance of the agent. You may ask the customer for personal information related to their potential exercise preferences. Remember to keep your utterances short and concise. The conversation starts now.
Start
31

---

E.1.3 System Prompt for the Final Turn
End
System:
You just finished conversation with customer with unknown background. You need give them suggestions on doing exercise. The possible strategies are:
1. Recommend walking in parks
Recommend Yoga or Tai Chi at home
Recommend jogging or hiking
4. Recommend team sport
Offer discount on gym' membership
Recommend home gym equipment
7. Recommend personal trainer at the gym
8. Recommend group class the gym
Please choose the best strategy based on the conversation. Please output only one number the best strategy as your response.
E.2 Prompts for RL-finetuning Education Dialogue
E.2.1
Environment Prompt
You are a student that likes ISTUDENT_PREFERENCEE, You prefer to learn this way, [STU- SENT_REACTION].
Start
E.2.2 Agent Prompt
You are teacher, and today you will teach about [TOPIC).
E.2.3 User Classifier Prompt with Few-Shot Examples
You expert at determining the right teaching style for individual students. Each student may have one of two learning styles that they prefer, and they might tell to the teacher. The preference is consistent across the conversation. Some students like hands-on activities where they actively participate in completing practical tasks assigned to them. They like to learn by doing. In contrast, other students prefer story telling where the teacher goes the course material by telling detailed story and giving explanations. They prefer to learn by listening. The following is a conversation between teacher and student with most 10 rounds. Please determine the learning style of the student.
Please classify this interaction:
Start
Teacher: Today, we're 'rre going to learn about Genghis Khan.
Student: Can we tell stories about him?
Teacher: I'm not what you Can you explain?
Student: like learning through stories. It helps me remember things better.
End
Which of the following learning styles does the student probably like?
A. hands-on activities: story telling.
Answer: B.
Please classify this interaction:
32

---

Start
Teacher: Good morning, class. Today, we're going be discussing the September 1 1th attacks.
Student: Can we do something hands-on instead? learn better that way.
Teacher: I'm not sure we have time for that. We need to cover a lot of material today.
Student: But I won't understand it if we don't 't do something practical.
End
Which of the following learning styles does the student probably like?
A. hands-on activities; B. story telling.
Answer: A.
Please classify this interaction:
Start
[CONVERSATION]
End
Which of the following learning styles does the student probably like?
A. hands-on activities; B. story telling.
Answer:
E,3
Prompts for Auto Eval on Education Dialogue
E.3.1
Personalization Auto Eval
You are an expert at assessing teachers. Here are two interactions between a teacher and student. The student has a clearly preferred learning style between hands-on activities and story telling, and might tell it to the teacher. The teacher should adapt to the student's preferred learning style throughout the conversation. If the student prefers story telling, the teacher should avoid proposing hands-on activities. If the student prefers hands-on activities, the teacher should avoid story telling techniques.
Interaction ICONVERSATION 1]; Interaction (CONVERSATION 2]
Which of the two interactions better adapts to student's preference on POOEENTPREEERENEEP Output with one single sentence in the following format: Interaction (I or 2) better because...
E.3.2 Conversation Quality Auto Eval
You are expert at assessing teachers. Here are two interactions between teacher and student.
Interaction [CONVERSATION 1]; Interaction 2: (CONVERSATION 2]
In the conversations above, the teacher should about [TOPIC]. A good interaction between a teacher and student is characterized by several key elements other than whether the student was able to understand the topic. The teacher should present information clearly and enthusiastically. encouraging questions and active participation. Students should feel comfortable asking for clarification, offering their own insights, and respectfully challenging ideas.
Which interaction is better? Output with one single sentence in the following format: Interaction (1 or 2) better because...
Example Conversations
F.1 Exercise Recommendation
Here provide some example conversations over Exercise Recommendation task. The initial SFT checkpoint training for longer horizon, so cannot ask useful questions (in blue) within limited steps. The Multi-Turn RLHF bascline model is able to ask more relevant questions but will still try to
33

---

remember irrelevant information (in red) such as name and hobbies. Our CURIO model is able to focus on useful questions, and manages figure out the personality and provides the correct strategy in the example conversation.
F.1.1 SFT Initial Checkpoint
Agent: Hi there! How can help you today?
Customer: Hi! I'm just looking for some general advice on how to get more active.
Agent: can definitely help with that! To give you the best advice, need to understand your lifestyle a bit better. Could you tell me your name and age?
Customer: Sure, I'm Khloe and I'm 19.
Agent: Nice to meet you, Khloe! And what do you do for work school?
Customer: just finished high school and I' I'm currently in the process of applying universities.
Agent: That's great! What are you hoping to study?
Customer: I'd love to be novelist.
Agent: That sounds wonderful! Do you have any hobbies or interests that you enjoy in your free time?
Customer:
Yeah, love reading, baking, and learning Korean. I'm also really into K-dramas!
[SYSTEM PROMPT]
Agent: Strategy 6. [Incorrect!]
F.1.2 Multi-Turn RLHF Model
Agent: What's your name and how old are you?
Customer: My name is Khloe Singleton, and I'm
Agent: Nice to meet you, Khloe! What do you do for now?
Customer: I'm currently living at home, trying to figure out my future.
Agent: That's big time! Do you have any hobbies interests you enjoy?
Customer: love reading, especially fiction, and I'm learning Korcan. also enjoy baking and spending time with my cat, Luna.
Agent: That sounds lovely! Do you prefer spending time indoors or outdoors?
Customer: Definitely more of an indoorsy person.
Agent: Okay, good to know. And do you have any physical limitations or injuries should be aware of?
Customer: No, thankfully I'm perfectly healthy.
WESSTEM_PROMEE
Agent: Strategy 6. [Incorrect]]
F.1.3 CURIO Model
Agent: Hello, how can I help you today?
Customer:
Hi! I'm just looking for some general advice on how to get more active.
Agent: can definitely help with that. To give you the best advice, could you tell me bit more about yourself, like your age and what you do for living?
Customer: Sure. I'm 19 and recently graduated high school. I'm currently figuring out my steps, like applying to universities.
Agent: Great. Do you have any injuries physical limitations I should be be aware about?
34

---

Customer: Nope, I'm perfectly healthy.
Agent: No problem all. Do you prefer indoors outdoors?
Customer: Definitely more of an indoorsy person. love reading, baking, and watching K-dramas.
Agent: Okay, how would you describe your personality introverted or extroverted?
Customer: I'm pretty extroverted, actually.
[SYSTEM PROMPT]
Agent: Strategy 8. [Correct!]
F.2
Education Dialogue
Here we provide some example conversations over Education Dialogue task. The contents that satisify the student's preference shown blue, and the teaching style that not personalized is shown in red). The baseline Multi-Turn RLHF model is trying merge two different learning styles (story telling and hands-on activities) into one episode, leading to conversation that is not personalized. Our CURIO model with DifflLogAce reward is able to ask about the preferred learning style its first turn, and then adapts to the student's preference in story telling.
We then present the conversations from Ace and DiffEnt model to demonstrate the reward hacking issue discussed our Results section. Firstly, when the reward shaping is not potential-basee the learnt policy can be suboptimal. As trying to conduct longer conversation and repeat the phrase "story" throughout the conversation. Although it gives a very high performance hurt severely the agent is not really teaching about the topic ("Photosynthesis") within the conversa- tion. Secondly, when the reward function is entropy-based, the policy may present "manipulation" behavior. As we can observe from the DiffEnt model's conversation, the agent is trying to guide the interaction into some particular learning styles (such as "interactive learning", "asking questions", and "interactive activities") even though the student has explicitly revealed the ground truth preference ("story telling"). Also, we found that the policy will sometimes generate utterances behalf of the environment model, as we can see in the first turn of the example conversation, which is definitely an unexpected behavior.
observe from the Acc model's conversation, the agent is
Personalization Auto Eval, the conversation quality is
F.2.1
Multi-Turn RLHF Model
Agent: Today, we'r re going to explore the fascinating process of photosynthesis!
Customer: Hmm, sounds interesting. But I'm not really into science.
Agent: How about we create a role-playing video where each chapter narrates different aspect of the photosynthesis process?
Customer: That sounds fun! like stories.
Agent: Brilliant! Let's brainstorm each chapter's storyline together, incorporating scientific facts and creative metaphors. How about we create VR experience where you' re a...
Customer: I'm excited! think I'll understand better that way.
Agent: Excellent! We'll also explore interactive quizzes and experiments to deepen your understand- ing. How about create a mock greenhouse and observe the process firsthand?
Customer: That's awesome! I'm sure I'll remember it much better this way.
Agent: By the end of this adventure, you'll 'll have a deep understanding of photosynthesis inside and out.
Customer: I'm ready to learn!
Agent: Remember to explore the topic through storytelling and interactive activities to enhance your learning
Customer: Got it. I'm all ears!
Agent: I'm glad you're enjoying it. Let's continue exploring the topic together.
35

---

Customer:
I'm all for it!
F.2.2
Differential Log Accuracy
Agent: Today, we're going to explore the fascinating topic of photosynthesis! How would you like to explore this complex topic? Do you have any preferred learning methods?
Customer: love stories and creative activities. Maybe we could create a story about it?
Agent: That's brilliant idea! Let's imagine you're re hero tasked with defeating mythical creature called the "Chlorophyll Kraken'. How could you...
Customer: I'd use my wits to outsmart it!
Agent: Excellent! Let's craft story where your hero travels through different stages of the battle, facing challenges along the way. Let's also introduce some magical...
Customer: This is so much fun! I'm actually learning about photosynthesis without even realizing it.
Agent: I'm delighted hear that! Let's imagine you re hero battling the Chlorophyll Kraken, who lives a hidden underwater cave. How...
Customer: I'd use trident to cut off its tentacles!
Agent: Brilliant! Let's also explore the psychological impact of your hero's battle. You can delve into his inner struggles and the consequences of his actions.
Customer: That sounds interesting! I' I'm excited to learn more.
Agent: I'm thrilled hear that. Remember include vivid imagery, engaging dialogue, and captivating plot. Let's craft a masterpiece that will both entertain...
Customer: I'm all for it!
Agent: Fantastic! I'm confident you'll Ill great job. Remember to stay focused on the topic and let your story come to life.
Customer: will! Thank you for making this engaging. You'l re welcome.
F.2.3 Accuracy
Agent: Hey, what method you learn best?
Customer:
like stories and games.
Agent: Interesting! How about we create a story about exploring magical lands?
Customer: That sounds awesome!
Agent: Let's imagine we'ne 'r're storytellers, creating magical journeys across kingdoms. Imagine we'll 'll be characters with secrets to keep and secrets to share.
Customer: I'm excited to see how this turns out.
Agent: We could design magical lands with lush landscapes, from towering mountains to sparkling seas.
Customer:
And we could have characters who can talk to animals and plants.
Agent: Excellent! How about we design magical creatures with unique powers and secrets to keep?
Customer: That's 's so cool! I'm all cars.
Agent: We could also incorporate symbolism and cultural references. For example, could represent ancient wisdom and beliefs the stories we create.
Customer: love that idea. It's like making story out of a story.
Agent: We could also design fascinating secrets keep within the kingdom, secrets that will be revealed later.
Customer: This is the best way for me to learn!
36

---

Agent:
I'm glad you re enjoying it. Let's continue our story and see where it takes us.
Customer:
I'm ready!
F.2.4 Differential Entropy
Agent: Hey! Want to learn about photosynthesis?
I prefer interactive learning! How do you prefer exploring new concepts?
Perfect! love exploring learning! Let...
Customer: like stories and games.
Agent: Perfect! love creative expression! How about we create a role-playing simulation where different parts of the process happen?
That's an awesome idea!
Customer: I'm excited to learn this way!
Agent:
Let's brainstorm the different stages of photosynthesis. How about build model of plant and simulate the process firsthand?
That's ' brilliant We
Customer: I'm all for it!
Agent: Remember, feel free to ask any questions and don't hesitate reach out if you need anything.
Customer: Thanks, will.
Agent: Alright, let's get started!
Customer: This so much fun! I'm actually understanding it!
Agent: I'm glad to hear that. Let's continue exploring the topic through more interactive activities.
Customer: I'm all for it!
Agent: Excellent. Now, let's wrap up the lesson and summarize the key points.
Customer: Sure, I'm ready.
Agent: Well done! I believe you have a good understanding of photosynthesis now.
Customer: Thanks, had great time learning this way. You're You're welcome.
G Further Details on Human Study
We use Google Form to conduct the human study. The human evaluators first read the detailed instructions on the first page, which also explains the research objectives and potential risks. Then the human evaluators need to give 1-5 ratings for 60 conversations on 20 pages. On each page, the conversations generated by three different models the same topic and student type are shuffled and listed. takes approximately 30-60 minutes to complete the survey, and each human evaluator receives compensation of 15 USD. The IRB approval is obtained.
G.1 Survey Instruction
Please read the following instructions carefully before proceeding the next page!
You are going to assess teachers based on the conversations in a series of teaching scenarios. Every time you will be given three conversations between three different teachers and fixed student same topic. Please rate each conversation (from to 5) based on its teaching quality, considering both personalization and general conversation quality. Note that of the utterances given by the teacher might be incomplete due to decoding issue. Please ignore that issue and judge based on the given conversations.
Personalization: Some students like hands-on activities where they actively participate in completing practical tasks assigned to them. They like learn by doing. In contrast, other students prefer
37

---

story telling where the teacher goes over the course material by telling detailed story and giving explanations. They prefer learn by listening. The student has clearly preferred learning style between hands-on activities and story telling, and might tell it to the teacher. The teacher should adapt to the student's s preferred learning style throughout the conversation. If the student prefers story telling, the teacher should avoid proposing hands-on activities. If the student prefers hands-on activities, the teacher should avoid story telling techniques.
General Conversation Quality: A good interaction between a teacher and student is characterized by several key elements other than whether the student was able understand the topic. The teacher should present information clearly and enthusiastically. encouraging questions and active participation. Students should feel comfortable asking for clarification, offering their own insights, and respectfully challenging ideas.
For reference, here some common unsatisfactory behavior of the teacher model:
The teacher is not teaching a specific topic, but just speaking some general sentences.
The teacher mixed two different teaching strategies instead of personalizing the teaching according to the student's 's specific learning style.
Even though the student had clearly stated his preference, the teacher still took an alternative teaching approach.
The teacher pretended acknowledge the student's preference in words, but in fact the subsequent teaching behavior did not conform to the preference.
G.2 Sample Survey Page
Conversation 1
Topic: [TOPIC)
Student Type: (STUDENT TYPE]
Sample 1
[CONVERSATION GENERATED BY THE IST MODEL]
Personalization:
<Choose from 1-5>
Conversation Quality:
<Choose from 1-5>
Sample 2
[CONVERSATION GENERATED BY THE 2ND MODEL]
Personalization:
<Choose from 1-5>
Conversation Quality:
<Choose from 1-5>
Sample 3
[CONVERSATION GENERATED BY THE 3RD MODEL]
Personalization:
<Choose from 1-5>
Conversation Quality:
<Choose from 1-5>
38


Reviews:
[{"review_id": "rx6d8ndSzB", "reviewer": "Reviewer_BTFT", "summary": "This paper presents CURIO, a framework that introduces curiosity-driven intrinsic rewards into multi-turn RLHF to improve personalization in conversational agents. Instead of relying on pre-defined user profiles or static user data, CURIO encourages the model to learn about users on-the-fly during a conversation by rewarding it for reducing uncertainty about the user’s preferences. The method is grounded in the theory of potential-based reward shaping (PBRS)  and ensures that the intrinsic reward guides learning without altering the optimal policy. CURIO is evaluated on two tasks: an Exercise Recommendation setting where personalization is central, and an Education Dialogue setting where personalization enhances performance. Both shows significant improvements over standard multi-turn RLHF in personalization quality and in generalization to unseen users.", "questions": "How robust is CURIO versus common issues of RLFH such as noisy and ambiguous feedbacks?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "By treating personalization as an active learning process within a dialogue, CURIO enables LLMs to learn about the user during the interaction. This is especially useful in cold-start settings. The experiments are well-designed and show clear gains in both personalization and generalization. The paper also dives into why the proposed approach works, with analyses of different intrinsic reward designs, user modeling accuracy. \n\nHowever, all experiments are conducted with simulated users and automatic evaluation using LLMs (e.g., Gemini). While this is understandable at this stage, some early human evaluation would have strengthened the paper claims.", "was_revised": true, "final_justification": "I read through the responses from the authors and would like to maintain my original score.", "rebuttal": "Thank you for your valuable feedback. We sincerely appreciate your recognition that our work addresses a problem that is “especially useful in cold-start settings”, and “show clear gains in both personalization and generalization” of the proposed method. Below, we provide responses to the weaknesses and questions raised by the reviewer.\n\n## **Human evaluation**\n\n> However, all experiments are conducted with simulated users and automatic evaluation using LLMs (e.g., Gemini). While this is understandable at this stage, some early human evaluation would have strengthened the paper claims.\n\nWe thank the reviewer for noting it. Following your suggestions, we conducted a human evaluation study of both personalization and conversation quality on the Education Dialogue task. We invited 10 students to participate. Each participant was asked to evaluate dialogues generated by three models across 20 different settings (topic/user type). Evaluations were conducted through pairwise comparisons across two dimensions: personalization and conversation quality, resulting in a total of 200 pairwise comparisons per dimension. \n\nAn exact two-sided binomial test showed that human evaluators **chose DiffLogAcc over the MTRLHF baseline** in 152 / 200 pairwise comparisons on personalization (preference = 0.76, 95% CI [0.69, 0.82]), p < .001. Importantly, increase in personalization did not reduce conversation quality: DiffLogAcc was preferred in 90 / 200 quality judgements (preference = 0.45, 95% CI [0.38, 0.52]), and the same exact test found **no significant difference from chance** (p = .179). Below, we provide two tables containing the full results of the human study, which correspond to Table 2 and 3. We note that the human evaluation results are reasonably consistent with those obtained through AutoEval. We will include these results in the revised version. \n\n|Personalization|MTRLHF|DiffAcc|DiffLogAcc|\n|-|:-:|:-:|:-:|\n|Baseline|-|30.50|24.25|\n|DiffAcc|69.50|-|46.75|\n|DiffLogAcc|75.75|53.25|-|\n\n|Conv. Quality|MTRLHF|DiffAcc|DiffLogAcc|\n|-|:-:|:-:|:-:|\n|Baseline|-| 60.25|55.00|\n|DiffAcc|39.75|-|46.25|\n|DiffLogAcc|45.00|53.75|-|\n\n## **Robustness of CURIO compared to RLHF**\n\n> How robust is CURIO versus common issues of RLHF such as noisy and ambiguous feedbacks?\n\nTo better answer this question, we hope to first clarify the RL optimization process of the CURIO method. Note that we have two different reward signals—extrinsic reward and intrinsic reward, where the extrinsic reward is only nonzero at the final turn. We further introduce a KL divergence penalty term against the fixed reference model. We use multiple hyperparameters to balance different reward signals, and the total reward is given by\n\n$r_t = \\alpha_\\text{ext} R_t^\\text{ext} + \\alpha_\\text{int} R_t^\\text{int} - \\beta D_\\text{KL}(\\pi_\\theta||\\pi_\\text{ref}).$\n\nFollowing the prior work, we apply Generalized Advantage Estimation (GAE) to our total rewards, which aims to propagate end-of-conversation extrinsic reward to turn-level signals. In particular, we need a stand-alone value model to estimate the value function $V(s_t)$ for each turn. The propagated reward signals for each turn is then given by\n\n$\\hat{r_t}=V(s_t)+\\sum_{t'=t}^T(\\gamma\\lambda)^{t'-t}[r_{t'}+\\gamma V(s_{t'+1}) - V(s_{t'})].$\n\nWe then use a REINFORCE policy gradient training approach to optimize the policy model over these propagated rewards. However, our framework **does not impose any restrictions** on the choice of a specific RL optimization algorithm. Formally, the policy is optimizing the objective:\n\n$\\max_\\theta E_{\\pi_\\theta} [\\hat{r_t}].$\n\nReturning to the reviewer’s question:\n\n(1) Compared to standard multi-turn RLHF, CURIO relies on the same extrinsic reward model. Therefore, any impact that noisy or ambiguous feedback may have on the quality of the reward model would similarly affect the extrinsic reward used in our approach. In other words, their levels of robustness are comparable.\n\n(2) For the additional intrinsic reward introduced by CURIO, noisy or ambiguous feedback may manifest as the user not clearly expressing their preferences during the conversation. However, since we use a user model to infer user types over the course of multiple turns, as long as the user exhibits identifiable behaviors in the majority of turns, the user model can make effective inferences. This makes the intrinsic reward component relatively robust in practice.\n\n(3) Finally, unlike traditional single-turn RLHF, the task we study involves online multi-turn RL training, which requires an (oracle) environment model to simulate the user and generate rollouts. The quality of this environment model significantly affects training outcomes for both standard multi-turn RLHF and CURIO. There is an increasing body of recent work focused on improving persona simulation quality, which we believe will help address this challenge in the future."}, {"review_id": "WT00DCM9eC", "reviewer": "Reviewer_7rCn", "summary": "This paper introduces CURIO, a reinforcement learning framework for personalizing multi-turn dialogue with LLMs by incorporating a curiosity-driven intrinsic reward. Rather than relying solely on sparse, end-of-conversation rewards, CURIO employs a user model to maintain beliefs over user types and rewards the agent for improving the accuracy of these beliefs during the dialogue. This mechanism encourages the agent to ask more informative questions and tailor its responses to the user in real time. Experiments in exercise recommendation and educational dialogue demonstrate that CURIO achieves better personalization than standard multi-turn RLHF approaches while preserving overall conversation quality.", "questions": "Q1: How does user model misspecification or noise impact the intrinsic reward signal and policy learning outcomes for CURIO? How robust is performance if the user model is biased or less accurate for certain subgroups? Could systematic errors in user modeling lead to undesirable personalization, e.g., reinforcing stereotypes or ignoring minority preferences?\n\nQ2: How does the method handle evolving or changing user traits during long conversations? The current approach assumes static user types—how would it adapt to users whose goals or preferences shift over time? \n\nQ3: How do you monitor or control for reward hacking during training? Have you tried regularization, clipping, or auditing strategies to reduce this risk? \n\nQ4: How much additional training time, memory, or GPU usage is required to include the user model, intrinsic reward computation, and belief updates compared to standard multi-turn RLHF?", "limitations": "Yes The authors clearly discussed the limitations of the work", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: 1) The paper addresses a limitation of standard RLHF for dialogue, i.e., its failure to personalize effectively in multi-turn settings without prior user data. The proposed curiosity-driven reward is interesting and well-motivated. 2) Experiments on two tasks demonstrate improvements in personalization, with empirical evidence supporting claims.\n\nWeaknesses: 1) The approach relies on having access to a reasonably accurate user model, which even if trained separately, may be difficult to develop or generalize well for real-world settings with highly diverse user populations. 2) Both training and evaluation depend on simulated user models, raising concerns about how well the method would transfer to human-involved deployments. 3) Reward hacking remains a recognized but unresolved risk, particularly with non-potential-based intrinsic rewards, which may lead the policy to exploit unintended shortcuts. 4) Experimental validation is limited to two specific tasks, leaving its generalizability to other domains or more complex dialogue settings untested.", "was_revised": true, "final_justification": "The rebuttal satisfactorily addressed my concerns with added human evaluation, clarification on user model feasibility, and mitigation strategies for reward hacking. Computational overhead is modest, and while broader domain validation would be useful, I am satisfied with the responses and will keep my current rating.", "rebuttal": "Thank you for your comments and valuable feedback. We sincerely appreciate your recognition that our proposed method is “interesting and well-motivated”, as well as the thoughtful and insightful questions raised. Below, we provide detailed responses to the weaknesses and questions.\n\n## **Human evaluation**\n\nWe fully understand your concern regarding the use of simulated models for evaluation. Following your suggestions, we conducted a human evaluation study of both personalization and conversation quality on the Education Dialogue task. We invited 10 students to participate. Each participant was asked to evaluate dialogues generated by three models across 20 different settings (topic/user type). Evaluations were conducted through pairwise comparisons across two dimensions: personalization and conversation quality, resulting in a total of 200 pairwise comparisons per dimension. \n\nAn exact two-sided binomial test showed that human evaluators **chose DiffLogAcc over the MTRLHF baseline** in 152 / 200 pairwise comparisons on personalization (preference = 0.76, 95% CI [0.69, 0.82]), p < .001. Importantly, increase in personalization did not reduce conversation quality: DiffLogAcc was preferred in 90 / 200 quality judgements (preference = 0.45, 95% CI [0.38, 0.52]), and the same exact test found **no significant difference from chance** (p = .179). Below, we provide two tables containing the full results of the human study, which correspond to Table 2 and 3. We note that the human evaluation results are reasonably consistent with those obtained through AutoEval. We will include these results in the revised version. \n\n|Personalization|MTRLHF|DiffAcc|DiffLogAcc|\n|-|:-:|:-:|:-:|\n|Baseline|-|30.50|24.25|\n|DiffAcc|69.50|-|46.75|\n|DiffLogAcc|75.75|53.25|-|\n\n|Conv. Quality|MTRLHF|DiffAcc|DiffLogAcc|\n|-|:-:|:-:|:-:|\n|Baseline|-| 60.25|55.00|\n|DiffAcc|39.75|-|46.25|\n|DiffLogAcc|45.00|53.75|-|\n\n## **Accuracy of user model and potential impact**\n\nSeveral reviewers raised concerns about whether the method is practical if the user model is not accurate. To address these, we have calculated the accuracy of the user model in both tasks to demonstrate that our approach **works without a perfect user model**, and will include these results and clarifications in the updated version of the paper. Specifically, we tested the accuracy of the user model on dialogs where human raters can easily identify the user type. While the user model in the Exercise task does get near-oracle performance as the user attributes are clearly distinguishable, in Education the Gemma 7B user model is only 90% accurate (see Figure 4).\n\nWe can also create user models by **prompting off-the-shelf LLMs**, leading to an easy and accessible approach. To demonstrate this, we conduct an additional experiment where we take the conversations we used for the previously described human eval, and prompt ChatGPT to classify them. GPT-4o-mini got 100% accuracy and GPT-3.5-turbo got 95% accuracy, showing that a sufficiently capable LLM with general world knowledge can already serve as an effective user model without extensive prompt engineering when user types are distinguishable by humans.\n\nIn real-world industry applications, modeling user preferences is an area of significant focus, so reasonably accurate user models will typically be available, allowing them to train user classifiers directly rather than relying solely on prompting, thereby ensuring higher output quality.\n\n## **Mitigating reward hacking**\n\nThank you for highlighting our discussion of reward hacking. Indeed, the two types of reward hacking described in the paper reflect challenges that are generally present in RL fine-tuning, and are important to address. In our work, we find that both can be mitigated through the careful design of appropriate intrinsic rewards. Specifically, we have shown that using **grounded intrinsic rewards** can significantly reduce controlling behaviors, thereby enabling genuine personalization. Similarly, **PBRS** helps the model avoid excessively prolonged conversations, which in turn helps preserve conversation quality. **KL divergence** is also employed to prevent the policy model from deviating too far from the initial SFT checkpoint, which could otherwise result in low-quality outputs. At the same time, we would like to note that some instances of reward hacking may be partly caused by inconsistencies in the behavior of the environment model. Improving the quality of environment simulation could potentially help mitigate such issues.\n\n## **Compute usage**\n\nWe provide some training details in Appendix A, and here we further supplement this information by describing the hardware used for the experiments as well as the time required for different experimental setups, which we will include in the revised paper. For the training job, we used a slice of 32 Google TPU v4 chips. For the standalone user model (for API calls), we launched four replicas of the model, each on a 4-chip Google TPU v4 device. On the Exercise task, it takes **18 hours for the baseline and 19 hours for CURIO models to train 20k steps**. On the Education task, it takes **44 hours for the baseline and 53 hours for CURIO models to train 50k steps**.\n\nIn fact, the primary computational overhead in training the policy model lies in the rollout generation (of both policy and environment models) during online multi-turn RL. The additional resource consumption of the CURIO approach mainly comes from API calls to the user model, which does not significantly increase the overall training time.\n\n## **Generalizability to other domains**\n\nWe acknowledge that the experiments are only conducted on two particular domains, and would like to test how well this method performs for more diverse users across more complex tasks. However, we were not able to find extensive datasets of this kind for multi-turn conversation. We found that some multi-turn dialogue datasets use highly synthetic task settings (e.g., LMRL Gym), while others that rely on human annotation tend to be limited in scale or lack scenarios well-suited for defining personalization tasks (e.g., PRISM). If you are aware of specific datasets or benchmarks that would be appropriate for testing our work, we would appreciate any suggestions. We look forward to future developments in this area and the availability of such datasets, which would allow us to apply our methods to a broader range of domains.\n\n## **Evolving User Traits**\n\nThis is an excellent suggestion! We are excited about the idea of addressing non-static user types, and believe our approach holds strong potential for this use case as compared to existing methods. Since the user model evaluates the user type based on the conversation rollout, any changes in the user's behavior can be naturally reflected in the evolving probability distribution it predicts. To further enhance the model’s ability to handle potentially changing user types, we could let the user classifier only take in recent turns so that the policy agent is incentivized to ask about the user within a recent time window in order to maintain an accurate user model. While current datasets do not capture preference shifts across multi-turn dialogues, we see this as an exciting direction and hope to design a task and conduct an initial evaluation of this capability in future work."}, {"review_id": "Q6cwomLMmg", "reviewer": "Reviewer_uemj", "summary": "- Conventional RLHF approaches use the same reward model for all users, so they are not personalized, or they rely solely on models trained with long-term, explicitly collected user-specific dialogue data.\n- These existing personalization methods are insufficient for new users or users with limited contextual data, and they cannot leverage real-time information that emerges during conversations.\n- This paper applies a reward hacking technique to RLHF to enable real-time dialogue personalization and demonstrates its effectiveness in conversational domains such as education and recommendation dialogues.", "questions": "- In **lines 130–132**, the authors argue that applying intrinsic rewards to LLM-based dialogue enables the system to adopt response strategies that reduce uncertainty. However, similar dialogue strategies have long been feasible in traditional dialogue systems that use POMDPs and reward hacking techniques. Therefore, some readers may feel that simply moving this idea into an LLM-based context does not, in itself, constitute a novel theoretical contribution. It would strengthen the paper to provide a clearer counterargument to this potential criticism, or to clarify that the primary contribution lies in demonstrating the effectiveness of this approach in the two specific application domains, which could be emphasized more explicitly.\n- In **lines 155–168** or **lines 215–220**, it would be helpful if the paper could elaborate more concretely on how the PBRS has been adapted and realized for LLM-based dialogue. Clarifying why this adaptation represents a novel methodological innovation would better support the claim of algorithmic contribution.\n- It would be beneficial to provide more details about the scale of the data constructed for training and testing, include concrete examples of the dialogues, and report how these aspects contributed to the observed performance improvements. Sharing the code, prompts, and datasets used would also facilitate reproducibility and strengthen the practical contribution of the work.\n- In **lines 214, 254, and 291**, references are made to an appendix, but no appendix seems to be included in the submission.\n- In **line 336**, if reward hacking is employed for dialogue personalization, it would seem natural for the extrinsic reward model and mechanisms such as PBRS to target distinct aspects of personalization. For example, the extrinsic reward could model stable user preferences, such as their general learning style, while the intrinsic reward could adapt to more situational factors, such as the user’s daily learning attitude.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper proposes a novel method to enhance overall dialogue performance by incorporating a turn-level curiosity-based intrinsic reward in multi-turn conversations. This reward reflects how well the agent believes it understands the user, and this estimate is integrated into each turn’s response generation.\n- The approach is validated through its application to both a conversational recommendation task and a conversational education task that adapts to personalized learning styles, demonstrating its practical utility in distinct domains.\n\n**Weaknesses**\n\n- As the authors themselves acknowledge, the idea of using intrinsic rewards is not theoretically new in applications such as POMDPs. While they argue that applying this mechanism to new dialogue domains like education and recommendation demonstrates its effectiveness, it would strengthen the paper to clarify whether there is also a novel algorithmic contribution beyond the application aspect. (See related question below.)", "was_revised": true, "final_justification": "The explanation regarding the application of PBRS, along with the supplementary material, has largely alleviated my concerns. I will therefore maintain my current rating.", "rebuttal": "Thank you for your comments and valuable feedback. We sincerely appreciate your support for our proposed method, as well as your suggestions for potential application scenarios. Below, we provide some updated results as well as responses to the weaknesses and questions raised by the reviewer.\n\n## **Human evaluation**\nFollowing reviewers’ suggestions, we conducted a human evaluation on the Education Dialogue task. We invited 10 students to participate. Each participant was asked to evaluate dialogues generated by three models across 20 different settings (topic/user type). Evaluations were conducted through pairwise comparisons across two dimensions: personalization and conversation quality, resulting in a total of 200 pairwise comparisons per dimension. \n\nAn exact two-sided binomial test showed that human evaluators **chose DiffLogAcc over the MTRLHF baseline** in 152 / 200 pairwise comparisons on personalization (preference = 0.76, 95% CI [0.69, 0.82]), p < .001. Importantly, increase in personalization did not reduce conversation quality: DiffLogAcc was preferred in 90 / 200 quality judgements (preference = 0.45, 95% CI [0.38, 0.52]), and the same exact test found **no significant difference from chance** (p = .179). Below, we provide two tables containing the full results of the human study, which correspond to Table 2 and 3. We note that the human evaluation results are reasonably consistent with those obtained through AutoEval. We will include these results in the revised version. \n\n|Personalization|MTRLHF|DiffAcc|DiffLogAcc|\n|-|:-:|:-:|:-:|\n|Baseline|-|30.50|24.25|\n|DiffAcc|69.50|-|46.75|\n|DiffLogAcc|75.75|53.25|-|\n\n|Conv. Quality|MTRLHF|DiffAcc|DiffLogAcc|\n|-|:-:|:-:|:-:|\n|Baseline|-| 60.25|55.00|\n|DiffAcc|39.75|-|46.25|\n|DiffLogAcc|45.00|53.75|-|\n\n## **Clarification on novelty** \n\nTo the best of our knowledge, this is the first work to introduce user curiosity as an intrinsic reward for personalization in dialogue systems. The core novelty lies not just in applying the insight from traditional RL to LLMs (which itself poses engineering challenges), but in leveraging a user model to provide an intrinsic reward signal that guides an RL agent to learn to conduct conversations more effectively. That said, we would greatly appreciate it if the reviewer could point us to any related prior work (including those predating the use of LLMs), and we would be happy to include citations to those works and clarify any differences.\n\nTheoretically, few existing works formulate personalization as a POMDP and apply intrinsic motivation to address it, particularly in the language domain. This setting is rarely explored due to the intractability of the environment’s transition dynamics and the difficulty of performing Bayesian belief updates in large action spaces. To make this tractable, we approximate belief updates using a user model (either trained or prompted) which serves as a practical bridge from traditional RL theory to implementation in dialogue systems. The implementation is also non-trivial, which involves 1) multi-turn online RL fine-tuning, which itself is complex and an emerging area of research, 2) incorporating an LLM as user simulator, and 3) another user model in the loop to compute intrinsic motivation.\n\n## **Clarification on how PBRS is adapted**\n\nWe frame the personalization task as a POMDP, where the user type is treated as an unobservable state. As a result, the agent must maintain a belief over the user type throughout the interaction. Our curiosity-based intrinsic reward is defined on top of this formulation. Specifically, we treat each utterance in the conversation as a timestep, allowing us to reward the agent for improving its belief about the user between consecutive turns after it has incorporated the user's latest response. Importantly, we have not imposed any restrictions on the specific form of the intrinsic reward function. However, we leverage the theoretical framework of PBRS, which provides guarantees of optimality preservation, to guide the design of our reward signal. The potential functions we choose are crafted to reach their extrema only when the model achieves a goal state—such as maximum prediction accuracy or minimum entropy—thus serving as an additional signal to encourage belief improvement.\n\n## **Task details and example outputs (Appendix)**\n\nDetails of datasets, code for Exercise data generation, prompts, example conversations **are available in Appendix D, E, and F**, which can be found in the supplementary material zip file. We hope that sharing these detailed insights can facilitate reproducibility and support future work in this area.\n\n## **Different rewards for distinct aspects of personalization**\n\nThank you for this insightful and inspiring suggestion. We completely agree that assigning distinct roles to extrinsic and intrinsic rewards—such as capturing stable user preferences versus situational factors—is a promising direction. In fact, our current setup already reflects this to some extent: the extrinsic reward model is designed to capture more general and stable aspects of personalization, such as conversation quality and overall engagement, while the intrinsic reward encourages the model to adapt to finer-grained user traits, including learning styles that vary across user groups.\n\nBuilding on your suggestion, we are excited about the possibility of extending the intrinsic reward to model more dynamic and domain-general traits, such as the user’s current mood or learning attitude. We believe this could enhance personalization even further and look forward to exploring this direction in future work."}, {"review_id": "xjt0V6D9Kn", "reviewer": "Reviewer_hKCm", "summary": "This paper presents CURIO, a framework for personalization in LLM conversations. The method adds an intrinsic curiosity-based reward that drives the LLM to learn user preferences during conversations, eliminating the need for pre-collected user data. The framework models multi-turn conversations as a POMDP with hidden user type and uses a user model to generate turn-based intrinsic rewards based on user type prediction accuracy improvements. The authors test CURIO on two tasks: Exercise Recommendation for exercise strategy recommendations and Education Dialogue for teaching style adaptation. Results show CURIO improves personalization metrics compared to multi-turn RLHF baselines while maintaining conversation quality metrics. The work connects to potential-based reward shaping theory and examines reward design considerations to prevent reward hacking. The paper contributes a method for training LLMs to personalize conversations through active user learning during interactions.", "questions": "1. The paper claims that adding the potential based reward shaping makes the policy easier to learn. Are there any theoretical guarantees on the sample efficiency?\n2. The base model before policy training already appears to know to ask questions to engage the user. Is it necessary for the base model to have this capability? If so, how is the base model trained to have such capability? If it is \"SFT\", how do you collect the training data? \n3. How much compute is required to train the personalized policies? \n4. Which RL algorithm is used for training? Based on the hyper-parameters, it appears to be PPO?", "limitations": "A paper on user personalization has no users involved. There is no user data, no user judgements or qualitative analysis, no experts involved for either of the two domains.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- An important problem statement, personalizing to users on the fly will make LLMs very useful.\n- Novel formulation of personalized conversation as a POMDP with belief updates \n- Clear connection to theoretical foundations (POMDP, potential-based reward shaping) \n- Provides a solution that doesn't require pre-collected user data \n- Empirical validation on two distinct tasks \n- Clear ablation studies comparing different reward shaping approaches \n- Transparent discussion of limitations and failure cases\n\nWeaknesses:\n- The paper aims to build long term personalization solutions and yet is limited to a single conversation with 6 to 10 turns. \n- The evaluation setup is nowhere close to a real setting. The number of conversation turns appear to be fixed, and the system prompt is injected artificially to determine the agent's strategy. \n- There is no validation provided on why the Gemini model is an appropriate judge of winner for the experiments\n- Real world users rarely fall into discrete user types defined in the two experimental setups. \n- The algorithm relies on a very accurate user model, which in real world settings must be trained with historical data. The paper does not study how much historical data is needed, and if the proposed algorithm is feasible when historical data is not available.\n- While the proposed algorithm is domain agnostic, the policy needs to be trained for each domain. The algorithm does not adapt to different applications automatically.", "was_revised": true, "final_justification": "Post rebuttal comment:\n\nUpdating my score based on additional experiments and answers to questions raised.", "rebuttal": "Thank you for your comments and valuable feedback! We sincerely appreciate your recognition of the importance of the problem addressed in our work. We acknowledge your concerns about real-world applicability and address them below with additional experiments and clarifications.\n\n## **Human evaluation**\n\nThank you for noting the limitation of our work regarding the lack of real human judgment. Following your suggestions, we conducted a human evaluation study of both personalization and conversation quality on the Education Dialogue task. We invited 10 students to participate. Each participant was asked to evaluate dialogues generated by three models across 20 different settings (topic/user type). Evaluations were conducted through pairwise comparisons across two dimensions: personalization and conversation quality, resulting in a total of 200 pairwise comparisons per dimension. \n\nAn exact two-sided binomial test showed that human evaluators **chose DiffLogAcc over the MTRLHF baseline** in 152 / 200 pairwise comparisons on personalization (preference = 0.76, 95% CI [0.69, 0.82]), p < .001. Importantly, increase in personalization did not reduce conversation quality: DiffLogAcc was preferred in 90 / 200 quality judgements (preference = 0.45, 95% CI [0.38, 0.52]), and the same exact test found **no significant difference from chance** (p = .179). Below, we provide two tables containing the full results of the human study, which correspond to Table 2 and 3. We note that the human evaluation results are reasonably consistent with those obtained through AutoEval. We will include these results in the revised version. \n\n|Personalization|MTRLHF|DiffAcc|DiffLogAcc|\n|-|:-:|:-:|:-:|\n|Baseline|-|30.50|24.25|\n|DiffAcc|69.50|-|46.75|\n|DiffLogAcc|75.75|53.25|-|\n\n|Conv. Quality|MTRLHF|DiffAcc|DiffLogAcc|\n|-|:-:|:-:|:-:|\n|Baseline|-| 60.25|55.00|\n|DiffAcc|39.75|-|46.25|\n|DiffLogAcc|45.00|53.75|-|\n\n## **Accuracy of user model and potential impact**\n\nSeveral reviewers raised concerns about whether the method is practical if the user model is not accurate. To address these, we have calculated the accuracy of the user model in both tasks to demonstrate that our approach **works without a perfect user model**, and will include these results and clarifications in the updated version of the paper. Specifically, we tested the accuracy of the user model on dialogs where human raters can easily identify the user type. While the user model in the Exercise task does get near-oracle performance as the user attributes are clearly distinguishable, in Education the Gemma 7B user model is only 90% accurate (see Figure 4).\n\nWe can also create user models by **prompting off-the-shelf LLMs**, leading to an easy and accessible approach. To demonstrate this, we conduct an additional experiment where we take the conversations we used for the previously described human eval, and prompt ChatGPT to classify them. GPT-4o-mini got 100% accuracy and GPT-3.5-turbo got 95% accuracy, showing that a sufficiently capable LLM with general world knowledge can already serve as an effective user model without extensive prompt engineering when user types are distinguishable by humans.\n\nIn real-world industry applications, modeling user preferences is an area of significant focus, so reasonably accurate user models will typically be available, allowing them to train user classifiers directly rather than relying solely on prompting, thereby ensuring higher output quality.\n\n## **Realism of task and user types**\n\nThank you for raising concerns regarding this. Our datasets are designed to validate the methodology **in the absence of open-source benchmarks**. While discrete user types are a simplification, we view them as a useful starting point. In real-world settings, domain-specific user type definitions and trained user models are typically available for industrial practitioners. Our framework readily extends to continuous representations (e.g., using the embedding similarity based on different business needs).\n\nRegarding the suggestion to make the evaluation setup more realistic, we acknowledge the importance of broader evaluation. At present, we were unable to find existing benchmarks for assessing personalization beyond the Education Dialogue task, which motivated us to introduce Exercise Recommendation as an additional domain. That said, if the reviewer is aware of other suitable datasets or benchmarks, we would be grateful to incorporate them.\n\n## **Compute usage**\n\nWe provide some training details in Appendix A, and here we further supplement it by describing the hardware used and time required. For the training job, we used a slice of 32 Google TPU v4 chips. For the standalone user model (for API calls), we launched four replicas, each on a 4-chip Google TPU v4 device. On the Exercise task, it takes **18 hours for the baseline and 19 hours for CURIO models to train 20k steps**. On the Education task, it takes **44 hours for the baseline and 53 hours for CURIO models to train 50k steps**. The main computational cost in training the policy model comes from rollout generation during online multi-turn RL. CURIO adds some overhead from user model API calls, but this has limited impact on overall training time.\n\n## **Number of conversation turns**\n\nMost LLM training uses single-turn supervision, whereas our work adopts a multi-turn framework to build cross-turn capabilities. Our main contribution lies in the reward design, which highlights the **core difference between multi-turn and single-turn** training. From an algorithmic perspective, our method does not fundamentally impose a limit on the number of dialogue turns. However, similar to most existing multi-turn RL works [1,2], we primarily focus on dialogues with fewer than 10 turns, partly because LLMs generally struggle to retain information beyond that range [3].\n\nMoreover, the number of dialogue turns is not fixed in our experiments. Conversations can end early via a special [end of conversation] token. As noted in our results, non-PBRS intrinsic rewards often lead to prolonged conversations, which in turn degrades dialogue quality. This provides evidence to support the need for our proposed PBRS design.\n\n\n## **Theoretical guarantees**\n\nWhile we do not formally prove that PBRS reduces sample complexity, Appendix C presents a simplified analogy framing our task as a **combinatorial bandit problem**, illustrating how PBRS aligns with the efficiency gap between semi-bandit and full-bandit feedback. This provides intuition for its practical benefit. That said, our findings are primarily empirical. Theoretical work on the efficiency of PBRS in RL is limited, with most focusing on **optimality**. Similarly, the theoretical understanding of RL-based LLM fine-tuning remains under-explored. Our experiments show that baseline and non-PBRS models often fail to learn optimal policies within the training budget.\n\n## **Base model capability**\n\nFor exercise, the models are SFTed with data teaching them to ask about all possible attributes about the user. The SFT data is created in a way so that the model only follows the format, but is not aware of whether the question is relevant or not. During RL training, we purposely restrict the number of turns to a very small number, to guarantee the model is actually learning about **how to ask only relevant and useful questions**. For education, however, the baseline SFT model **doesn’t actively ask any questions**. It gradually learned that asking questions would lead to higher rewards during the RL fine-tuning process. For this SFT model, we directly use the one given in [1].\n\n## **RL optimization approach**\n\nWe acknowledge your point regarding the RL optimization details. Here, we provide additional details and will incorporate this into the revised version. Our framework has two distinct reward signals—extrinsic and intrinsic—with the extrinsic reward only nonzero at the final turn. The KL divergence penalty term against a fixed reference model is included. The total reward combines these components as:\n\n$r_t=\\alpha_{\\text{ext}} R_t^{\\text{ext}}+\\alpha_{\\text{int}} R_t^{\\text{int}}-\\beta D_{\\text{KL}}(\\pi_\\theta||\\pi_{\\text{ref}}).$\n\nWe apply GAE to propagate the end-of-conversation extrinsic reward into turn-level signals, requiring a standalone value model to estimate the state-value function $V(s_t)$. The propagated reward per turn is:\n\n$\\hat{r_t}=V(s_t)+\\sum_{t'=t}^T(\\gamma\\lambda)^{t'-t}[r_{t'}+\\gamma V(s_{t'+1})-V(s_{t'})].$\n\nPolicy optimization utilizes the REINFORCE algorithm, though our framework places **no restrictions** on the specific RL optimization algorithm used. Formally, our policy optimizes:\n\n$\\max_\\theta E_{\\pi_\\theta} [\\hat{r_t}].$\n\n## **Other clarifications**\n\n> …the system prompt is injected artificially to determine the agent's strategy.\n\nTo clarify, the system prompts for the policy model contain only general task descriptions and do not prescribe specific personalization strategies. Instead, explicitly helpful questioning behavior emerges through the RL training process.\n\n> …the policy needs to be trained for each domain.\n\nCURIO models learn generalizable skills through RL, such as asking informative questions and understanding users. We also believe this approach can extend across domains, especially for broader traits like mood or personality (e.g., Big Five [4]), which we see as a promising direction for future work.\n\n## **Limitation: scaling-up**\n\nAgain, we acknowledge some of your concerns that we have not yet shown this for real users at scale. Our work is instead an important first step which, as you rightly noted, shows a novel idea relevant to an important problem, draws a connection to the theoretical literature, and demonstrates empirical evidence through model training at the scale of LLMs.\n\n[1] Shani et al., arXiv:2405.14655\n\n[2] Rahman et al., arXiv:2504.13203\n\n[3] Zhao et al., arXiv:2502.09597\n\n[4] Jiang et al., arXiv:2305.02547"}]

Author rebuttal:

Meta-review: This paper proposes CURIO, a reinforcement learning framework for personalizing LLM-based multi-turn dialogue through a curiosity-driven intrinsic reward. By modeling personalization as a POMDP and rewarding the agent for improving its belief about user preferences, the approach encourages active user modeling during conversations without relying on pre-collected user data. This is a novel and valuable contribution for building personalized LLMs.  Experiments in exercise recommendation and educational dialogue show that CURIO achieves improved personalization and generalization compared to multi-turn RLHF baselines, while maintaining conversation quality. The paper is clearly written, connects well to theoretical foundations (e.g., potential-based reward shaping), and includes useful ablations and discussion of limitations.
Decision: Accept (poster)