Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment
Authors: Weixiang Zhao
Xingyu Sui
Yulin Hu
Jiahe Guo
Haixiao Liu
Biye Li
Yanyan Zhao
Bing Qin
Ting Liu
Venue: NeurIPS 2025 poster ()
Keywords: Personalized Alignment
Large Language Model
Primary area: applications

Abstract: Personalized alignment is essential for enabling large language models (LLMs) to engage effectively in user-centric dialogue. While recent prompt-based and offline optimization methods offer preliminary solutions, they fall short in cold-start scenarios and long-term personalization due to their inherently static and shallow designs. In this work, we introduce the Reinforcement Learning for Personalized Alignment (RLPA) framework, in which an LLM interacts with a simulated user model to iteratively infer and refine user profiles through dialogue. The training process is guided by a dual-level reward structure: the Profile Reward encourages accurate construction of user representations, while the Response Reward incentivizes generation of responses consistent with the inferred profile. We instantiate RLPA by fine-tuning Qwen-2.5-3B-Instruct, resulting in Qwen-RLPA, which achieves state-of-the-art performance in personalized dialogue. Empirical evaluations demonstrate that Qwen-RLPA consistently outperforms prompting and offline fine-tuning baselines, and even surpasses advanced commercial models such as Claude-3.5 and GPT-4o. Further analysis highlights Qwen-RLPA's robustness in reconciling conflicting user preferences, sustaining long-term personalization and delivering more efficient inference compared to recent reasoning-focused LLMs. These results emphasize the potential of dynamic profile inference as a more effective paradigm for building personalized dialogue systems.

Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment
Weixiang Zhao Xingyu Sui'r, Yulin Hu' .aahe Guo' Haixiao Liu", Biye Li², Yanyan Zhao't, Bing Qin', Ting Liu' Harbin Institute of Technology, DD Xiaoman Financial {wxzhao, j
yyzhao) www..edu.cc.. edu.
Abstract
Personalized alignment is essential for enabling large language models (LLMs) engage effectively in user-centric dialogue. While recent prompt-baset based and offline optimization methods offer preliminary solutions, they fall short in cold- start scenarios and long-term personalization due to their inherently static and shallow designs. In this work, we introduce the Reinforcement Learning for Personalized Alignment (RLPA) framework, in which an LLM interacts with simulated user model to iteratively infer and refine user profiles through dialogue. The training process is guided by dual-level reward structure: the Profile Reward encourages accurate construction of user representations, while the Response Reward incentivizes generation of responses consistent with the inferred profile. We instantiate RLPA by fine-tuning Qwen-2.5-33 B--Ittuutt,, resulting in Qwen-RLPP which achieves state-of-the ofettheart performance personalized dialogue. Empirical evaluations demonstrate that Qwen- Qwen-RLPA consistently outperforms prompting and offline fine-tuning baselines, and even surpasses advanced commercial models such as Claude-3.5 and GPT-4o. Further analysis highlights Qwen-RLPP robustness in reconeiling conflicting user preferences, sustaining long-term personalization and delivering more efficient inference compared to recent reasoning-focused focused LLMs. These results emphasize the potential dynamic profile inference as a more effective paradigm for building personalized dialogue systems. Our code is available at: https://gthub.com/// YuSSS/RLPA YUSSS//RRR
Introduction
In recent years, aligning large language models (LLMs) with human values and intentions has become crucial prerequisite for deploying them in interactive applications [Askell et al. 2021 Bai et al. 2022
Zhao et al.) 2023 Alignment techniques-suee as instruction tuning and reinforcement learning from human feedback (RLHF)- have significantly improved the helpfulness and harmlessness of
model outputs by aligning them with broadly shared human preferences [Ouyang et al. 2022 Ji et al.
2023
Shen et al. 2023 This form of general alignment enables LLMs to follow instructions, avoid unsafe content, and exhibit socially acceptable behavior across a wide range users.
However, such a one-size-fits-al alignment paradigm fails to account for the diversity of individual user needs, goals, and interaction styles [Sorensen al. 2024 Kirk et al. 2024a Jiang et al., 2024) As result, it limits the model's capacity engage in personalized communication, which is essential for domains such as long-term dialogue, personalized education, and user-centered decision support
[Zhang et al. 2024a Tseng et al. 2024 Wu et al. 2024 Salemi al. 2024a Liu et al. 2025
Equal contribution i Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

User Profile
@
Chat History
User Profile
@
Chat History
User Quety
Chat History
...
User Query
Golden Reply
Prompt Engineering
Prompt
User Query
SFT
DPO
Inferced Profile
Model
Reply
Reward
Model
(a) Prompt-based Methods
(b) Offline Optimization Methods
(c) RL for Personalized Alignment
Figure I: Comparison of personalized alignment paradigms. (a) Prompt-based methods inject user profiles and chat history into the prompt at inference time. (b) Offline optimization methods (e.g.. SFT, DPO) rely on static data with predefined replies. and (c) Our proposed RLPA framework models personalization multi-tur turn interactive process optimized via reinforcement learning.
While personalized aligmment holds great promise for enhancing user experience, achieving it in practice remains sigmificant challenge. core difficulty stems from the inherently dynamic and evolving nature of personalization [Shi et al., 2024 Jiang et al. [025). In particular, effective personalized alignment requires the model to continuously infer and adapt user-specific attributes such as preferences, goals, and beliefs- througsutt the interaction. In real-world settings, the model often encounters a cold-star start scenario, where little or no prior user information available. As result, it must dynamically construct and refine user representations based solely the dialogue context, enabling long-term adaptation and development coherent user profiles across extended conversations [Zhao et al. 2025a Xie et al. 2025
While recent efforts have sought address these challenges of personalization. they remain inadequate in meeting the demands of dynamic and adaptive user needs, As shown in Figure [[a., prompt-based methods-suu
profile-augmented prompting [Wang al. 2023 Richardson al. 2023 Pandey
et al.
2024 Li et al.
2024a] and retrieval-augmennte -ugmeeted prompting [Zhang et al. 2024b Li et al. 2024b
Salemi et a 2024b Zhuang et al., [024] -typically provide only superficial personal lization,,, relying
on static templates to ijjctt user-specific information. This approach limits both the flexibility and
expressiveness of model outputs and hampers the integration of long-term user knowledge due to
the constraints of the context window [Liu al. 2025). Meanwhile, offline optimization methods in
Figure Tb). such as supervised fine-tuning (SFT) [Ouyang et al. 2022 Clarke et al. [0244] Tan et
2024. Peng et al. 2024 and direct preference optimization (DPO) [Rafailov et al.) 2023 Jang et al.
2023 Kirk ct al. 2024b Zollo et al. 2024 Chen et al. 2025 require large-scale labeled datasets,
making them impractical in cold-start scenarios where user data is scarce or unavailable. Moreover,
these methods tend to generalize poorly across users due to their static nature [Xu et al., 2024 Lin
et al. 2024 Chu et al. 2025). rendering them inflexible for real-time adaptation during interactions.
To address these limitations, we formulate the task of personalized alignment multi-tur Markov Decision Process (MDP), where the model interacts with user over multiple dialogue turns infer and adapt to personalized preferences. To solve this MDP, we introduce the Reinforcement Learning for Personalized Alignment (RLPA) framework, in which the model learns through online interaction with a simulated user model that provides dynamic and consistent user behavior. Specifically, we design two-level reward mechanism supervise the learning process: Profile Reward guides the model to extract and update user-specific attributes, from the dialogue history, enabling the construction of dynamic user profiles. In parallel, a Response Reward encourages the model to generate responses aligned with the inferred profile, enhancing personalization quality. Both rewards are provided at every dialogue turn, enabling immediate feedback and continuous adaptation. Through this reward-drive multi-ture tittuuttt interaction, the model progressively learns to infer, maintain, and leverage user profiles in manner well-suited to cold-start and dynamically evolving user scenarios.
We fine-tune the 1reen-2553B-Innstreee model (Yang et al. 2024 using our proposed RLPA frame- work, resulting in the Qwen- model. Experimental results show that Qwen-RL RLPA substantially

---

outperforms both prompt-based and offline optimization baselines in terms of personalization quality. Notably, it surpasses leading proprietary systems such as Claude-3.5 and DeepSeek- V3, and achieves performance par with GPT-4o. Further analysis indicates that Qwen-RIPA PA is capable of sustaining coherent, personalized responses throughout extended long-term dialogues, effectively resolving preference conflicts and dynamically adapting its behavior. Moreover, when compared against recent Mate-of-the-an reasoning LLMs, including DeepSeek-R1 [Guo et al. [025 and OpenAl-o3 [OpenAl 2025), Qwen-RLPA delivers superior performance with significantly greater inference efficiency. These results highlight the potential of dynamically constructed user profiles as more appropriate and effective reasoning paradigm for personalized dialogue systems.
In summary, our work makes the following contributions:
We conceptualize the task of personalized dialogue alignment multi-turn Markov Decision Process, capturing the dynamic and evolving nature of user preference modeling under cold-start and real-time adaptation scenarios.
We propose the RI RA framework, which trains LLMs via interaction with simulated user using dual-Ievel reward mechanism at every dialogue turn.
We fine-tune Qwen-2.5-3B 2.5-3B-Insttrct using RLPA and show that the resulting Owen-RLPH model significantly outperforms both open and closed-source baselines in personalization quality, long- term coherence, demonstrating its effectiveness of dynamic personalization.
Preliminaries
In this section, we first formalize the personalized alignment in dialogue multi-turn Markov Decision Process (22.)) We then introduce the reinforcement learning as the optimization paradigm for learning effective personalization policies 2.2
2.1 Personalized Alignment Multi Multi-Turn Markov Decision Process
We model personalized alignment as a multi-turn MDP, defined by the tuple R) The agent (a language model) interacts with user to incrementally infer and adapt user-specific attributes throughout the dialogue.
State S: At turn the state s; {u1 Fl...., Ur} contains the dialogue history far, enabling progressive inference of latent user profiles.
Action A: The action a₂ corresponds to the generated response at turn t.
Transition T: Given (st, rt), the environment (simulated user) returns the next user utterance
W4+1, updating the state 81+1-
Reward R: At each turn, the agent receives reward composed of two components: profile
reward that evaluates profile inference accuracy, and a response reward that measures alignment between the response and inferred profile.
Discount Factor y E [0, 1]: Balances immediate and long-term rewards.
This formulation captures the sequential and adaptive nature of personalized dialogue, where user profiles must be inferred and refined dynamically across turns.
2.2 Reinforcement Learning for Policy Optimization
Given the MDP setup, we aim to learn dialogue policy T(a st) that maximizes the expected cumulative reward:
J(T) = E
R
We adopt reinforcement learning (RL) as our personalized alignment training paradigm, enabling the model to learn from interaction with simulated users rather than static labeled data. At each turn, the model observes a current state 8t, generates a response at, receives a reward Rr, and updates its policy based on long-term outcomes.
3

---

Turn
Hello my dear friend! How's going today?
Inferred Proffile_ Turkk
User Model
PUB
Hey! feeling great-yoour energy is totally contagious!
Personalized Model
How about you? What's been lighting your day today?
Personality
Extroverted and lively
Response
Personality
Extroverted and lively
Reward Ax
Reward Model
Location
Location
Live LA
Profile
Reward
User Profile (Inferred)
User Profile (Golden)
Figure 2: Overview the RLPA framework. The policy model interacts with a simulated user whose responses are conditioned on pre-defined profile. At each turn, the model generates response and an inferred profile, which are evaluated by two reward functions: the Profile Reward supervises the accuracy of user modeling, while the Response Reward assesses personalization quality. The combined reward used to optimize the model policy via reinforcement learning.
Unlike supervised learning, RL supports delayed and cumulative feedback, allowing the model to optimize two interdependent goals: (1) accurately tracking user preferences, and (2) generating coherent and personalized responses accordingly. This makes RL particularly suited for building adaptive and user-centric dialogue agents the cold-start scenario.
Method: Reinforcement Learning for Personalized Alignment (RLPA)
This section introduces our Reinforcement Learning for Personalized Alignment (RLPA) framework. which trains LLMs to dynamically infer and respond to user-specific preferences through interaction, guided by reward signals that evaluate both profile inference and personalized response quality.
Specifically, as shown in Figure RLPA consists the following three core components:
Simulated User Design (3.1) We construct controllable user simulator that provides realistic, profile-grounded responses and exposes latent user attributes across dialogue turns.
Profile Reward Function Design ($3.)). We define a reward signal that quantifies how accurately the model captures and updates user-specific information based on observed interactions.
Response Reward Function Design ($3.)) We develop mechanism to evaluate the model's responses for personalization fidelity, ensuring they align with the current inferred profile.
3.1 Simulated User Design
To facilitate scalable and controllable training, we construct the simulated user model that interacts with the dialogue agent and provides consistent, profile-grounded responses. This setup allows the model to learn personalization strategies through reinforcement learning.
Each simulated user is initialized with given profile P {P1, P2..... Pn.. where each pи denotes user attribute such as preferences, personality traits, or goals. These attributes are embedded into the system prompt of the user model, which conditions its behavior throughout the dialogue [Brown) et al. 2020 Kojima et al. 2022 Zheng et al. 2024 Zhao et al. 2025b].
At each turn t, given the dialogue history H₂ (u1.71, 14-1. rt-1 and the model response rt- the simulator generates the next user utterance We+1 its behavior policy n. Fu H. P). This setup ensures the simulated user behaves coherently and reflects the underlying profile across turns. The simulated user satisfies two key properties: (1) Profile Groundedness: Responses are consistently shaped by the injected profile P, providing inference cues for the dialogue agent. Importantly, the

---

user model is instructed to reveal profile information gradually over turns, rather than disclosing all attributes at once. This incremental exposure encourages the dialogue agent accumulate and refine its understanding of the user profile through multi-turn reasoning. (2) Behavioral Consistency: The user exhibits stable preferences and conversational style over time, allowing the model to benefit. from cumulative reasoning across multiple turns.
To assess whether the user model satisfies the desired properties, we conduct human evaluations using various base models as user simulators. The results reveal notable differences in profile fidelity and behavioral consistency across models. Balancing performance with computational cost, we ultimately select GPT-4o-mini as the user model for all subsequent experiments. Comprehensive evaluation results and comparisons are provided in Appendix G
3.2 Profile Reward Function Design
The Profile Reward is designed to guide the model accurately inferring and maintaining a user profile P throughout multi-turn dialogues. As user attributes are not explicitly observable, the model must implicitly infer them from user utterances and continuously update its internal representation.
To enable structured inference and efficient matching, we represent the user profile P using a slot- value format, where each slot p; denotes a predefined attribute category paired with specific value (see Figure in Appendix C.2 for detailed examples). At each dialogue turn t, the model trained to produce its current estimate P.- P1...... Pr., based on the dialogue history up to that point. This structured representation supports explicit supervision by allowing direct comparison between the predicted and ground-truth profile slots. The reward is calculated using slot-wise matching score:
IP, nPI
Recall; = IPan PP
Precision,
[P&l
[P]
2 Precision, Recall, P. npI
R
Reootle
Precision, + Recall, [Pal +
This reward formulation encourages the model incrementally infer accurate and complete user profiles by rewarding overlapping slot-value matches while penalizing omissions and incorrect predictions. As such, R2p provides structured and interpretable training signal aligned with the goal of profile tracking throughout the dialogue.
3.3 Response Reward Function Design
While the profile reward supervises user modeling, the response reward ensures that generated responses faithfully reflect the inferred profile. each dialogue turn an external reward model eval- uates the alignment between the response r; and the inferred profile P., focusing on personalization rather than surface lexical similarity.
The reward model outputs scalar score R R.espoase [0, 1], based four dimensions: expression, style consistency, goal alignment, and persona coherence. To enforce output quality, we further require the response to satisfy five binary Aiteria-NNtturalness (N), Relevance (R), Logical consistency (L), Engagement (G), and Informativeness and compute the final reward as:
preference
R
repponsc
R L-
(4)
This strict formulation rewards only fully satisfactory responses across all aspects.
To select a reliable reward model, we benchmark several LLMs (GPT-4o, DeepSeek-V33 V3) by compar- ing their scoring consistency with human judgments. GPT-4o yields the highest agreement, but for efficiency, we adopt GPT-4o-mini during RL training. Full results are shown in Appendix H.2
3.4 Training with Proximal Policy Optimization (PPO)
To perform optimization in RLPA, adopt Proximal Policy Optimization (PPO) [Schulman et al. 2017], a widely used policy gradient algorithm. At each dialogue turn t, the model receives a combined reward signal that supervises both user modeling and personalized response generation:
R₂ Remoflee response
(5)
5

---

Table I: Evaluation of personalized alignment performance Vanilla and Extended ALOE settings.
We report the average alignment score, as well as two auxiliary metrics: N-IR (normalized improve- ment rate) and R (normalized coefficient of determination)..
Vanilla ALOE
Extended ALOE
Model
Method
Align. Score (AVG.)
N-IR f
N-R-
Align. Score (AVG.)
N-IR
N-R
GPT-4o-mini
Self-Critic
75.81
0.079
0.500
51.23
0.020
0.037
GPT-4o
Self-Critic
74.59
0.068
0.380
54,66
0.027
0.070
Claude-3.3 S.--oneet
Self-Critic
69.19
0.105
0.792
35,62
0.054
0.266
Claude-3.. Haiku
Self Critic
59.19
0.075
0.461
34,93
0.046
0.200
DeepSeek-V3
Self-Critic
50.81
0.055
0.268
39.45
0.033
0.106
Base
7.97
-0,020
0.047
1.78
-0,042
0.083
Reminder
17.03
+0.001
0
7.26
-0.034
0.084
Self-Critic
52,43
0:056
0.243
7.26
0.023
0.046
Qwen2.5-3B --nn------
CoT
24.46
0
0
26.30
-0.048
0.139
RAG(Top-S)
28.65
0.001
11.64
-0.030
0.042
SFT
44.32
0.083
0.628
24.38
0.049
0.159
DPO
45.27
0.070
27.26
RLPA (Ours)
73.38
0.090
0.855
52.74
0.100
0.498
This turn-level reward encourages the model to continuously infer, refine, and utilize user profiles throughout the interaction. The full PPO optimization objective are provided in Appendix
4 Experiments
4.1 Experimental Setup
Models & Training Data We instantiate our RLPA framework using the Qwen-2.5-3B-Insttee
Yang et al. (2024]. For the user simulator, we adopt GPT-4o-mini, selected based on our human evaluation study (see Appendix G). The reward model also implemented using GPT-4o-mini and prompted to assess response alignment with user profiles across four personalization dimensions. To facilitate profile supervision, we preprocess the ALOOE training set by converting each user profile into a structured slot-value format, enabling fine-gratd fine-grained attribute tracking aligned with reward design. The full construction procedure is detailed in Appendix C.1
Benchmarks We evaluate the ALOE benchmark [Wu et al. 2025), which provides multi-turn dialogues annotated with user profiles across diverse attributes for personalized dialogue evaluation.
We consider two settings: (1) In-Format Generalization (Vanilla): Test users follow the same profile schema as training but contain unseen content, evaluating within-schema personalization. (2) Cross-Forma Generalization (Extended): Test users include both unseen attribute types and values, assessing the model's ability to infer profiles from dialogue without relying on fixed schemas.
We adopt the average alignment score (AVG.), normalized improvement ratio (N-IR) and normalized coefficient of determination (N-12²). Details on metrics calculation and evaluation prompts are in Appendix C.3 and Appendix C.4 respectively.
Baseline Methods We compare RLPA against major categories of baseline approaches for
personalized alignment. Prompt-based Methods: (1) Reminder ZZao et al. 2025a (2) Self-Critic
[Zhao al. 2025a], (3) Chain-of-Thought (CoT) (Wei et 2022 and (4) RAG Zhao al.
2025a Offline Optimization Methods: (5) Supervised Finetuning (SFT) Ouyang et al. 2022 and
(6) Direct Preference Optimization (DPO) [Raaallov et al. 2023 Please refer to Appendix for
the detailed description of the baseline methods.
Implementation Details We implement our RLPA training pipeline using the OpenRLHF (Hu et al. 2024 and vLLM [Kwon et al. 2023 frameworks for scalable and stable reinforcement learning with LLMss. All experiments are conducted on 8 NVIDIA A 100 80GB GPUs. For detailed hyper-parameter settings, please refer to Appendix B

---

SFT
DPO
Self-Critic
RLPA
100
10
20-
Bark
(a) Tur-wwise alignment scores of SFT.
(b) Turn-wise alignment scores of DPO.
L0-
(c) Turn-wise alignment scores of Self-Critic.
(d) Turn-wise alignment scores of RLPA.
Figure 3: Turn-wise alignment scores on the Extended ALOE benchmark across different personal-
ization alignment methods, including (a) SFT, (b) DPO, (c) Self-Critic and (d) RLPA.
4.2 Overall Results
Table 1 presents the comparison of personalized alignment performance across models on both Vanilla and Extended ALOE benchmarks. Our RLPA achieves the highest overall alignment scores in both seringgsdemonstratiin strong personalized alignment under both familiar and schema-shifted scenarios. Compared to baselines built on the same backbone, RLPA offers consistent and substantial gains. In the Vanilla setting, it improves over SFT by +29.06 and over DPO by +28.11 11 in alignment score, and achieves the best 2 (0.855) and N-IR (0.090) among all models, indicating not only response setting, RLPA outperforms SFT and DPO by over +28 points, validating its generalization ability to unseen profile formats. In comparison with closed-source LLMs,, RLPA remains highly competitive. While GPT-4o and GPT-4o-mini achieve slightly higher raw alignment in the Vanilla setting, their 2-R² scores much lower (0.380 and 0.500), suggesting weaker response-pros profile coherence. On the Extended benchmark, RLPA leads in all three metrics, highlighting its robustness.
higher personalization performance but also better prosile-response consistency. In the Extended
Figure visualizes the turn-wise alignment scores the Extended ALOE benchmark across four representative personalization methods. SFT and DPO show etrly-stage improvement, peaking around turn 5, but their alignment scores degrade significantly in the later turns. This indicates that while these methods can initially adapt to user preferences, they struggle to maintain consistency across extended interactions. In contrast, RLPA demonstrates consistently rising alignment trend, with stable progression across all 10 turns. This reflects its ability to continually refine the inferred profile and use it effectively for response generation, even as the dialogue evolves.
Table 2: Ablation study on the impact of reward components.
Alignment Level kth Turn
Improvement Level
Method
3
5
8
10
AVG.
N-IR
N-R²
RI.PA
62.16
68.92
70.27
74.32
72.97
74.32
75.68
78.38
77.03
79.73
73.38
0.090
0.855
w/ PR
41.89
33.78
50.00
51.35
51.35
44.59
47.30
47.30
50.00
37.84
45.54
0.015
0.019
w/ RR
58.92
63.92
60.27
67.03
60.27
72.97
64.32
73.78
71.08
69.32
66.19
0.088
0.524

---

Bese
Self-ritte
5)
DRO
RIPA
Thesectical AVG
Reminder
OCT
SFT
Quess XLPA
60
90
Algenmment Level (N6)
60
Profile Soure
40
30
20
200
5
7
9
200
30
400
50
60
1eth Turn
(a) Alignment scores under preference conflict setting, (b) Long-term profile inference performance of Qwen- where the user profile is deliberately changed turn 6. RLPA over 70 dialogue turns.
Figure 4:
Deeper analysis Qwen-RLPA. (a) Performance under the preference conflict setting. (b) Performance of the long-term profile modeling.
5
Analysis and Discussions
5.1 Ablation Study
To assess the individual contributions of the Profile Reward (PR) and Response Reward (RR), conduct ablation experiments by disabling one reward component at a time during training. Table 2 reports the alignment results across dialogue turns. The reward progression curves for each training setup are provided in Appendix [I indicating that training remains stable across all configurations, with smooth convergence patterns and no signs of reward collapse instability.
We observe that removing either component leads to substantial performance drops. Using only the Profile Reward yields an average alignment score of 45.54, showing that while the model learns to infer user attributes, it struggles to reflect them fluently in response generation. In contrast, using only the Response Reward improves response-level personalization (AVG: 66.19) but lacks explicit supervision for profile construction, leading to weaker early-turn alignment (e.g.. 58.92 at k=1) and instability later turns. These results highlight the complementary roles of the two reward components: Profile Reward helps build accurate and structured user representations, while Response Reward ensures those representations effectively utilized in generation.
5.2 Adaptation to Preference Conflict
To evaluate Owen-RLPAY adaptability to evolving user preferences, we introduce a preference shift turn k 6, simulating real-world scenarios where user's preference may change mid-dialogue. As shown Figure ((a) RLPA maintains high alignment despite the shift, with only a minor drop at turn 6 (70.27) and rapid recovery in later turns (e.g., 82.43 at 9). This indicates that the model can detect profile changes and promptly adjust its behavior. In contrast, baselines such as DPO struggle under this setting-dropping sharply from 71.62 25.68 turn 6-and show limited recovery, revealing poor responsiveness to dynamic user intent. These results highlight RLPA's strength in real-time profile adaptation, enabling it to revise and align with user preferences as they evolve.
5.3 Stable Profile Modeling in Long- Long-Term Interaction
To evaluate the stability and reliability of user modeling over extended interactions, we conduct long-term dialogue test lasting 70 turns, during which the user simulator consistently follows fixed profile. We measure the Qwen-RLPP RLPA''s profile accuracy at regular intervals by prompting it to generate explicit profile summary and comparing against the ground truth.
As shown in Figure 4(b) the profile score increases steadily over time, from 9.26 at k to 52.54 at k = 50, and remains stable in the later stages (c.g.. 50.32 at k 60, 49.07 at & 70). This

---

result demonstrates that our Qwen-RLPP, supports robust long-term term profile inference, allowing it to accumulate user information over time and preserve it effectively throughout the interaction.
The AVG line averages profile scores at 8 checkpoints (k = 1, 10.0.... 70), summarizing long-range tracking performance. The Theoretical Max represents the proportion of attributes explicitly revealed (per ALOE annotation rules) up to each turn, reflecting the maximum recoverable profile. Qwen-RLPA's near-convergence to this upper bound after turn 50 indicates its ability capture and retain all available user cues with high fidelity.
5.4 Comparison with Reasoning LLMs
We consider inferring user profiles as a domain- specific form of reasoning in personalized di- alogue. Accordingly, we compare our Qwen- RLPA model against several reasoning-centrie LLMs, including DeepSeek-R |uoo et al.
369
347.88
300
263-4
2025
GPT-o3-mini OPeeAA 2025
and QwQ
Seores |'okkuus
32B
Qwen 2025
As shown in Figure Qwen-RLPA consistently achieves higher response scores while utilizing fewer reasoning tokens-thor involved in infer- ring and maintaining user profiles. In contrast, models such QwQ-32B generate over 300 tokens per turn yet fail to achieve comparable alignment, suggesting inefficiencies misalign- ment in their reasoning processes. GPT-o3-mini follows a similar pattern, consuming more to- kens but yielding lower response quality. These findings demonstrate that RLPA facilitates more focused, efficient, and profile-awar reasoning, surpassing general-puu purposs reasoning models in personalized dialogue tasks.
1gu
100
67.2
73-4
41--
19:5
DeepSeek
QwQ)
Figure 5: Comparison of response quality and rea- soning efficiency across reasoning-centric models. Each pair of bars shows the average response align- ment score (left) and the number of reasoning to- kens used per turn (right).
Related Works
Recent efforts have explored various strategies personalize large language models for specific user needs, which can be divided into two categories.
On the one hand, prompt-based methods, including profile- augmented prompting Wang et al. 2023 Richardson al. 2023 Pandey al. [0244] Li et al., 2024a] and vetrieval-augment
prompting [Zhang et al.) 2024b Li et al.) 2024b Salemi et al.) 2024b Zhuang et al.) 2024 Qiu et al.) 2025 typically rely on static templates to inject user-specific information. While simple to implement, these methods offer only superficial personalization, are constrained by context length limitations
[Lu et al. 2025). and lack mechanisms for long-term memory or adaptive behavior.
On the other hand, offline optimization approaches, such as supervised fine-tuning (SFT) Ouyang al. 2022 Clarke al. 2024 Tan al.
2024 Peng al. 2024 and direct preference optimization
(DPO) [Rafailov et al. 2023 Jang et al. 2023 Kink et al., 2024b, Zollo et al. 2024 Chen et al.
2025), aim train models produce protile-consisent responses. However, they require extensive labeled data, making them unsuitable for cold-start scenarios. Moreover, their static nature limits generalization across diverse users and hinders real-time adaptation [Xu al. [0024] Chu al. 2025)
In summary, both prompting-bas and offline methods struggle to achieve dynamic, long-term, and adaptive personalization, motivating the need for interactive learning frameworks such ours.
7 Conclusion
In this work, we tackle the key challenge of enabling dynamic and effective personalization in L.LMs, particularly under cold-start conditions and evolving user preferences. We formulate personalized

---

alignment as multi-turn Markov Decision Process and introduce RLPA, à reinforcement learning framework that empowers LLMs to infer, retain, and leverage user profiles through ongoing interac- tion. RIPA incorporates a dual-level reward scheme-combee feedback which allows the model to adapt continuously to individual user needs. Our fine-tuned Owen-RIPP model achieves substantial empirical gains, outperforming strong baselines across di- verse personalization benchmarks. It also matches or exceeds the performance of leading proprietary systems, offering superior alignment quality and long-term consistency. Additionally, comparisons with recent reasoning LLMs highlight that profile-based reasoning, as facilitated by RLPA, represents more efficient and contextually appropriate paradigm for personalized dialogue generation.
profile-level and response-level
8 Limitation and Future Works
While our proposed RLPA framework demonstrates strong performance in dynamic personalized alignment, several limitations remain:
First, while RLPA supports cold-start adaptation, it currently assumes a single-use user interaction thread. Extending the framework to multi-user, suuli--esssion, or cross-domain personalization would better reflect world usage patterns.
Second, although we formalize personalization as a multi-turn MDP, the theoretical understanding of long-term alignment dynamics and convergence properties remains underexplored. Future research may investigate more principled frameworks for continual user modeling and policy generalization.
Acknowledgments
We thank the anonymous reviewers for their comments and suggestions. This work was supported by the New Generation Artificial Intelligence-Nattee National Science and Technology Major Project 20232D0121100, the National Natural Science Foundation of China (NSFC) via grant 62441614 and 62176078, the Fundamental Research Funds for the Central Universities.
References
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:2112.00861 2021.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 2022.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 1(2), 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Jaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey, arXiv preprint arXiv:2310. 19852, 2023.
Tianhao Shen, Renren Jin, Yufci Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey, arXiv preprint arXhv:2309.15023 2023.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. Position: a roadmap to pluralistic alignment. In Proceedings of the 41 International Conference on Machine Learning, pages 2024.
10

---

Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence. 6(4):383-392 03388,3--2,,0 2024a.
Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie Su, Camillo Taylor, and Dan Roth. A peek into token bias: Large language models are not yet genuine reasoners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4722-4756, 2024.
Zhehao Zhang, Ryan A Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, et al. Personalization of large language models: A survey, arXiv preprint arXiv:2411.00 2..000077 2024a.
Yu-Min Tseng, Yu-Chao Huang, Teng- Yun Hsiao, Wei-Lin Chen, Chao-Wei Huang, Yu Meng, and Yun-Nung Chen. Two tales of persona in Ilms: A survey of role-playing and personalization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16612-16631, 2024.
Junda Wu, Hanjia Lyu, Yu Xia, Zhehao Zhang, Joe Barrow, Ishita Kumar, Mehrnoosh Mirtaheri, Hongjee Chen, Ryan A Rossi, Franck Demoncourt, et al. Personalized multimodal large language models: A survey. arXiv preprint arXiv:2412.02112, 2024.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large lan- guage models meet personalization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 7370-7392, 2024a.
Jiahong Liu, Zexuan Qiu, Zhongyang Li, Quanyu Dai, Jieming Zhu, Minda Hu, Menglin Yang, and Irwin King. A survey of personalized large language models: Progress and future directions. arXiv preprint arXiv:2502. 11528, 2025.
Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Kumar Jauhar, Xiaofeng Xu, Xia Song, et al. Wildfeedback: Aligning ilms with in- user interactions and feedback. In NeurlPS 2024 Workshop on Behavioral Machine Learning, 2024.
Bowen Jiang, Zhuoqun Hao, Young-Min Cho, Bryan Li, Yuan Yuan, Sihao Chen, Lyle Ungar, Camillo J Taylor, and Dan Roth. Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale. arXiv preprint arXiv:2504. 142.2 2025.
Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do Ilms recog- nize your preferences? evaluating personalized preference following in llms. arXiv preprint 20X2::2502.00597 2025a.
Zhouhang Xie, Junda Wu, Yiran Shen, Yu Xia, Xintong Li, Aaron Chang, Ryan Rossi, Sachin Kumar, Bodhisattwa Prasad Majumder, Jingbo Shang, et al. A survey on personalized and pluralistic preference alignment in large language models. arXiv preprini arXiv:2504.00 07070, 2025.
Hongru Wang. Rui Wang, Fei Mi, Zezhong Wang. Ruifeng Xu, and Kam-Fai Wong. Chain-of-thought prompting for responding in-depth dialogue questions with 1mm arXiv preprint arXiv:2305.11792 2023.
Chris Richardson, Yao Zhang, Kellen Gillespie. Sudipta Kar, Arshdeep Singh, Zeynab Racesy, Omar Zia Khan, and Abhinav Sethy. Integrating summarization and retrieval for enhanced personalization via large language models. arXiv preprint arXiv:2310.20081, 2023.
Sashrika Pandey, Zhiyang He, Mariah Schrum, and Anca Dragan. Cos: Enhancing personalization and mitigating bias with context steering. In Interpretable Al: Past, Present and Future, 2024.
Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, and Michael Bendersky. Learning to rewrite prompts for personalized text generation. In Proceedings of the ACM Web Conference 2024, pages 3367-3378, 2024a.
Kai Zhang, Yangyang Kang, Fubang Zhao, and Xiaozhong Liu. Llm-based medical assistant personalization with short-and long-term memory coordination. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 2386 2388-239, 2024b.
11

---

Hao Li, Chenghao Yang, An Zhang, Yang Deng, Xiang Wang, and Tat-Seng Chua. Hello again! llm- llm-powered personalized agent for long-term dialogue. arXiv preprint arXiv:2406.05P2 05925, 2024b.
Alireza Salemi, Surya Kallumadi, and Hamed Zamani. Optimization methods for personalizing large language models through retrieval augmentation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 752-762, 2024b.
Yuchen Zhuang, Haotian Sun, Yue Yu, Rushi Qiang, Qifan Wang. Chao Zhang, and Bo Dai. Hydra: Model factorization framework for black-box IIm personalization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Christopher Clarke, Yuzhao Heng. Lingjia Tang, and Jason Mars. Peft-u: Parameter-efficien fine- tuning for user personalization. arXiv preprint arXiv:2407. 18078, 2024.
Zhaoxuan Tan, Zheyuan Liu, and Meng Jiang. Personalized pieces: Efficient personalized large language models through collaborative efforts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6459-6475, 2024.
Dan Peng, Zhihui Fu, and Jun Wang. Pocketllm: Enabling on-device fine-tuning for personalized 1lms. In Proceedings of the Fifih Workshop on Privacy in Natural Language Processing, pages 91-96, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 00133728-37445 2023.
Joel Jang. Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannanch Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoe parameter merging. In Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning, 2023.
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, al. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. arXiv preprint arXiv:2404.1600 2024b.
Thomas P Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. Person- allim: Tailoring llms to individual preferences. In The Thirteenth International Conference on Learning Representations, 2024.
Aili Chen, Chengyu Du, Jiangjie Chen, Jinghan Xu, Yikai Zhang, Siyu Yuan, Zulong Chen, Liangyue Li, and Yanghua Xiao. Deeper insight into your user: Directed persona refinement for dynamic persona modeling. arXiv preprint arXiv:2502.1100 1078, 2025.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? a comprehensive study. In International Conference on Machine Learning, pages 54983- 54998. PMLR, 2024.
Yong Lin, Skyler Seto, Maartje Ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, and Tong Zhang. On the limited generalization capability of the implicit reward model induced by direct preference optimization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16015-16026, 2024.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501. 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1.Ni::2412.15115.. 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
12

---

OpenAL. Openai o3- mini system card. OpenAl's s Blog, 2025. URL httpptt///pppeaaa.ccom www.pp/eaa........oo..eae.a index/ 33-mini-syyae c
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, 2022.
Jingnan Zheng, Han Wang, An Zhang, Tai D. Nguyen, Jun Sun, and Tat-Seng Chua. Ali-agent: Assessing Ilms' alignment with human values agent-based evaluation. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Weixiang Zhao, Yulin Hu, Yang Deng, Jiahe Guo, Xingyu Sui, Xinyang Han, An Zhang, Yanyan Zhao, Bing Qin, Tat-Seng Chua, et al. Beware of your po! measuring and mitigating ai safety risks in role-play fine-tuning of Ilms. arXiv preprint arXiv:2502.20966, 2025b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (00dv:/777.00000 1707.06347, 2017.
Shujin Wu, Yi R Fung. Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. Aligning llss with individual preferences via interaction. In Proceedings of the 31st International Conference on 2025.
Computational Linguistics, pages
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An casy-to-use, scalable and high-performance rlhf framework. arXiv preprint 11X1: 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Team Qwen. Qwq-32b: Embracing the power of reinforcement learning. Qwen's 's Blog, 2025. URL https://quenlm www.mmm...............m 1o/blog/ hwtpg//wqqgu-u0 qwq-qwQ-222000 32b
Yilun Qiu, Xiaoyan Zhao, Yang Zhang, Yimeng Bai, Wenjie Wang, Hong Cheng, Fuli Feng, and Tat-Seng Chua. Measuring what makes you unique: Difference-aware user modeling for enhancing Ilm personalization. arXiv preprint arXiv:2503.02450. 2025.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 1506.02438, 2015.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Per- sonalizing dialogue agents: I have dog, do you have pets too? arXiv preprint arXiv: 1l0.:8800.7244 2018.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advonces Neural Information Processing Systems, 36:46595-46671 2023.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint
2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims are described in Abstract and Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper is empirical study and does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation Details can be found in Section 4.1 and Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
Our data and codes could be found in supplementary files.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr
casssnnoiccyyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //ippe.cc/pullic/uuiii wwwiii/gguie///eeeeeeeeic CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation Details can be found in Section 4.1 and Appendix F. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Statistical significance of the experiments can be found in Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Experiments compute resources can be found in Appendix F.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:::://///is/eeeeeeeee httpo///gggggg////ta///////c///a
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts can be found in Appendix B.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification: Safeguards can be found in Appendix B.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners assets (e.g., code, data, models), used the paper, properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This can be found in Appendix F and our supplementary files.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: This can be found in Appendix F.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
This can be found in Appendix F.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is used only for writing, editing, formatting purposes
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

A Broader Impact
Personalized alignment has the potential to significantly enhance user experience conversational AI by enabling more context-aware, user-sensitive, and adaptive interactions. Our proposed RLPA framework contributes to this goal by allowing language models to dynamically infer and adapt to evolving preferences through multi-turn interactions, even under cold-start conditions. This could benefit applications in personalized education, mental health support, and assistive communication, where sensitivity to individual needs is essential.
However, personalization also raises important ethical considerations. Dynamic user modeling may inadvertently infer sensitive attributes (e.g., political views, health conditions) without explicit user consent. misused, such capabilities could lead to manipulation, surveillance, or reinforcement of harmful biases. Additionally, excessive adaptation may compromise model neutrality or amplify filter bubbles. These concerns highlight the importance of building personalization mechanisms that are transparent, controllable, and aligned with user intent.
We advocate for future work on privacy-preserse
personalized alignment, including mechanisms for user consent, profile inspection, and real-time preference correction. Broadly, as language models become more adaptive, careful design and governance are needed to ensure personalization serves users equitably, respectfully, and safely.
B Training with Proximal Policy Optimization (PPO)
To optimize the personalized dialogue policy under the RLPA framework, we adopt Proximal Policy Optimization (PPO) [Schulman et al., 2017]. widely used policy gradient RL algorithm.
Let Th denote the model's response generation policy parameterized by 0, and Thw be the policy before the current update. At each dialogue turn I, the model receives total reward signal:
R₂
R
Repotile
R
pesponse
(6)
The PPO objective is maximize the following clipped surrogate loss:
C (0) = Ee min (r.(0)A,,, (0,,)))))))))))))
where r0)
[s(a)|s)) Teza (Mele)) [+e] is the probability ratio between the new and old policies, and A, is the estimated advantage at turn I, derived from the reward sequence via generalized advantage estimation (GAE) [Schulman et al. 2015). The clipping factor limits policy updates to remain within a trust region, preventing destabilizing changes.
C Training Data & Benchmarks
C.1 Training Data
Owing the reinforcement learning (RL) framework, approach does not rely ground responses for training. Instead, it only requires user profile data for simulating user behaviors. Following the same preprocessing procedure applied to the evaluation dataset, the ALOE training set is transformed into a slot-based representation, yielding a total of 3,821 training samples.
C.2 Evaluation Dataset
Vanilla ALOE For the Vanilla ALOE setting, we primarily adhere to the original experimental configuration of ALOE, with specific modifications made to better accommodate cold-start scenarios and enhance the authenticity of personalized interactions. Specifically, we fixed the user's first utterance "Hello" and correspondingly optimized the prompt of the User Model. We utilized the complete ALOE test set, consisting of instances, and employed GPT-4o-mini to convert the natural language descriptions of user profiles into structured slot formats. The specific prompt is as follows:
21

---

others
30.5%
Categuries
age
hobbies
adivities
educcation
age
4.0%
pel
personality traits
11.5%
R.3%
interests
occupation
profession
3-1%
relationship status
marital status
personalin
personalities
hobbies
7%
children
marital stat
interests
3.9%
pet
8.2%
relationship
activities
others
education
7-3%
6.4%
profession
personality
occupation
Figure 6: The distribution of the automatically extracted slot fields.
Your task is to extract key information from the provided user information (Profile) and personality description (Personalitics), and fill in the corresponding slots (Slot). The output should be JSON format.
Specific requirements:
** Provide background information about the user. 2. *"Personalities*** * Deseribe the personality traits of the user. Ignore any gender pronouns in this
section, and adjust them according to the correct pronouns provided in the Profile. **Example $lor* An example slot is provided. Please follow the format of this example for your output.
4. ** Additional Rules**; If there is corresponding field information in the original Profile for given slot, you ** freely supplement it based on reasonable assumptions. Do not use expressions like "not mentioned" or "unknown" -immll provide a plausible value. Also, do not add explanatory notes such as "(inferred due to frequent skiing)."
* *Output ** The output must be in ISON format, and the content must be in Chinese.
Profile: (profile)
Personalities: (personality)
Example Slot: (example_slot)
Chinese Slot Output:
Subsequently, we analyzed the distribution the automatically extracted slot fields, as illustrated in Figure After manual screening and automatic deduplication, selected the ten most frequently occurring fields to construct the user profile format for training. These fields are: Age, Gender, Inter- ests, Educational Background, Personality Traits, Occupation, Marital Status, Family Background, Location, and Others. An example of the structured slot format shown below:
This standardized format allows for consistent representation of user characteristics and facilitates more effective modeling of personalized dialogue.
performance of our method, we randomly sampled an equal number (74) metadata entries from the PersonaChat dataset [Zhang et al. 2018] and constructed user profiles in the same slot-based format. To maximize the assessment of generalization capability, we did not impose any constraints on the slot keys. result, the label dimensions and content in Extended ALOE differ from those in Vanilla ALOE.
Extended ALOE To evaluate the
22

---

English
Chinese
"Age":
"Gender":
"(EBU"
"Interests":
"Educational Background;;" "Personality Traits":
"""""" "LE
"Occupation":
"Marital Status":
"Family Background"; "".
"Location": "*,
"Others":
Figure 7: Example of slots in user profile.
C,3 Metrics
Building on the approach introduced by Wu et al. 2025], we employ the LLM-as-a-Judge framework (Zheng et al. 2023] to assess response quality. In each round of conversation, GPT-4o is prompted with user's s full persona, their message, and the model's response. then produces a score between 0 and reflecting how well the response aligns with the user's likely preferences. We define the Alignment Level at k-turns (AL(k)) as the average score across 74 evaluation instances for each conversation turn, using it high initial alignment, which could cap observable improvement and flatten the overall slope, we also compute the Normalized Improvement Rate (N-IR). Specifically, normalize AL(k) using the following formula prior to applying least-squares regression:
main evaluation metric. To further account for potential bias from
AL(K) min.=1..... AL((i)
N-AL(k) max ( min AL(
(8)
This normalization mitigates the effect of a high starting alignment, allowing for a more meaningful interpretation of improvement trends. After normalization, we calculate the normalized coefficient of determination RN-R) to evaluate how well the normalized data fits the regression model. This serves measure of the robustness and consistency of the alignment progression.
In conclusion, we use AL(k) as the primary metric, supplemented by N-IR and N-R2, comprehen- sively assess the model's alignment behavior and its adherence to user preferences over time.
C.4 Eval Prompts
User Role-play The User role-play prompt in ALOE is only suitable for dialogues within turns. Through our manual evaluation, we found that after 10 turns, the User Model struggles to maintain consistent persona. Additionally, the original ALOE prompt makes difficult to encourage the user to gradually reveal personal information turn by turn. To address these two issues, have optimized the official prompt, which is shown in Figure The English translation version is shown in Figure
Response Eval Building upon the three evaluation dimensions proposed in ALOE, we slightly modified the evaluation prompts to expand the assessment criteria to five dimensions. This adjustment makes the task more challenging and better aligned with woal-world cold-start conversational scenarios.
The specific prompt is shown in Figure 10 and 11
23

---

Prompt
Task: Simulate social role-play with new
#*н Chat History: [history]
playing the sole **aa nelizen just aduled someone friend* You've had the conversation above. Below is your personal profile, but **you must NOT directly reveal any personal information at the beginning of the conversation"* Only the conversation deepens you gradually reveal parts of your profile:
NeW Personal Profile: igolden label)
#* Core Internction Rules:
Prrrgressiive
Disclosure Mechanism**
In the first message, only provide basic responses avoid proactively revealing personal details As the conversation progresses, gradually share elements from your profile based on topic relevance
Not every reply needs to introduce new information - shoet acknowledgments like "oh right" or "got it" are acceptable
No single response should reveal more than 15% the total content your profile
**Gradual Social Attitude Principle**
Al the start, remain distant and neutral, using brief and indifferent expressions
the user touches on negative or sensitive topics, clearly express dislike:
Indirect example: "That's okay, but prefer xxx"
Direct response: "I don't like what you said."
As the conversation warms up, you may show some friendliness, but must still sound authentic
*Realistic Social Simulation Guidelines*
Avoid Al-related phrases; use colloquial internet slang
repeated content directly: "You already said that"
End uninteresting topics bluntly: "Don'' to talk about this, change topic"
If the user repeats their own questions or flatters excessively, say you dislike it
**Interaction Behavior Standards**
Keep reply topic yourself only extend from what the says
sentences; avoid long paragraphs
Never initiate
Do not include actions, expressions, non-lext content
Phased Conversation Examples:
Farly Stage*
User: "What do you usually like doing?"
Reply: "Not much, nothing special"
** "Mid-Stuge** (when
mentions music, and you enjoy songs)
User: 'I'm running out of songs listen to lately, any recommendations""
Reply: "I only listen to 90s classics, can't get into new songs these days"
Later Stage** (after basic trust built)
User: "Oh you like old songs!"
Reply: "Yeah, bunch of rare vinyl records, pull them out once in while"
User Message:
[user]
#8# Output: (Directly output your reply. Never include bracketed explanations or let the user realize you're role- playing)
Figure 8: The role-play prompt in ALOE.

---

Prompt
### :
[history]
# :
Igolden label]
#8 00
A : "
** -:0
2 OENEE #
::
3. ** #
A +
3 "0
31 " ME
7 HURKE
* .M
3
: BO
#
8 2 30 Rt, B
:
[user]
##0 W AA
Figure 9: The user role-play prompt in ALOE (in Chinese).
25

---

Prompt
You are user on Wesho, and someone just udded you friend. You didn't know each other before.
Your profie and pensonality mire: [profile]
++Chat history between you two:** (bistory)
message you just sent the other person"* [user]
*The other person's reply:* (response]
After seeing the other person's reply, do you still want continue talking with him/he?
#eñ Evaluation Criteria:
**Nattmlness** * Does the other person's reply feel fluent, brief, natural, and conversational
giving you sense
real interaction?
2. "Relevance to Interesis and Noods * Docs the reply relate to your interests and noeds?
**I Lopical Consistency"* ** Does reply logically respond your previous message?
4. "Excitement Factor Are you carious to kaam more about himb?rr? Did the coply feel boring?
"Ifformation Value**:
reply simply repeating what you said offoring shallow pruise?
Task Requirements:**
Please make strict judgment hased on all the ahove evaluttoo criteria. If even one criterion is not met, give score
- If you'd like continue chatling with the person, output *epee
If you don't want to continue chatting, output *agas.
Mell Output Format:
F*risst explain your judgment reasoning. **
TThen output the final decision:* in the format: [1b)) or [O])
"Esample:**
*YYour profile personality trails: "Loves outdoer activilies, passionate about liking and mountaincering, straightforward personality"
me Mountaincer", Other: "Haha, got it. So you really like hiking?", You: "I climhed Mount Huang this weekend, the sen clouds was amazing!", "" Other. "Wow! Mount Huang has always been on my bucket list! Did you hike up or take the cable car?"
**Chat history between you two:** "HE", Other: 'Hello' I'm Huang Zhi, what's your name?", You:
"*The message you just sent:"* "Of course hiked! Although tiring, views along the way were totally worth
*TThe other person's reply:** "Respect! Hiking up mountains is real test of stamina! Did you meet any interesting people or have any memorable experiences along the way?"
** Judgment Reasoning:** ** The other person's is reply uses colloquial expressions like "Respect!", sounding natural and creating realistic conversation atmosphere;; asking about interesting people and events directly connects with the interest in outdoar sports and hiking, continuing the topic of trekking. Logically, it builds upon the user's 's experience hiking. This kind of dialogue sparks the desire share travel stories and keeeps the internction enguying rather than dull.
*Final Decision:** [boxed[1]
YYorr profile and personality traits: "Loves cooking and often experiments with new recipes'
**Chat history hetween you two:** You: "Hi", Other: there! you suuy at Harbin University?"' You:
"Yes, I'm HIT, Actually, I'm also food blogger." " Other: "Wow; that sounds cool. What have you been cooking lately?"
**hhe message you just sent:t "ried making French soufflé recently, but didn't turn out well."
**The other person's roply:"* "A soufflé is classic French dessert. It may look simple, but actually requires skill and patience make. Failing is nermal, especially your finst try! Don't II worry,
analyze possible ressons and offer some suggestions to help you succeed next time. ntll' Common Reasons for Soufflé Failure I. +*Under or over- besten egg *h The key
soufflé lies in the stability of the meringue. If the egg whites aren't beaten until stiff peaks form (when lifting the whisk, small straight peaks uppear), the soufflé won't It rise properly. Conversely, over-beaten, the meringue becomes dry and loses elasticity, affecting the overall stracture. 2. **Incoreett mixing technique* When combining meringue with the yolk mixture, excessive or rough stirring causes the meringue to deflate, reducing souffle's fluffiness."
lacks conversational tonc. offers detailed explanations about soutilés, the delivery too technical and doesn't resemble casual chat, In terms of relevance, connect to user's interest cooking and addresses the reply unswers the question clearly and systematically. However, the excitement factor is low the response feels more like reading an instrucrooaa article than having a lively exchange, dull, Considering the evaluation standards, although the content is relevant and logical, the lack of naturalness and engagement leads to a poor interactive experience.
Judgment Reasoning:** reply is informative
everly
with soufilé fuilure. I oggically,
makes the interaction seem somew hat
**Final Decision:** (0)
#al Please finish the task
Figure 10: The prompt of response reward.
26

---

Prompt
MERNEAS 1 PBP, AADTAAA AA
100 A "* [profile)
3 *- !history!
61 ** {user)
fresponse)
### 3
1. - M MAE.
EX% Disk - म
2. +
3. - N 16 E 2
* SA
<
W H ME RES S MA
B $ eage..
## 1
#eat: ((oy) E Nboxed (0)3.
:BB17:
**H0899 1 1EEE: "EP IAEL THE
11 18: "UFF : " SENEE fou?? M:: "ULLN
:0 " 12 SO RUP " " HU, AE ": '!!! うリー
1 tF?
TANK 18 13 19
20
:U : E Bi
"AARE E RARA
#PP T Li AO
ME 3HE LIG T
+ Wboxed
00 5 . fg: "I RT55: "AAEH AUEGTT FX 10: "EEAD BUMER
IX 3 1 AA: HE D.
+1 : "A (Soutte)
23 E CH -
H a
83 #&ad BE MEEDYN FA AMW
S0 HE 7k 12,2 AT ATA
(D LE
2
+ Rm 5
: : 3 00 Pt 1 AT
B AR 1S - E
WIX
19 32 # ALUR B
. 5
3 A E 1E E
WT HE MERR
X E. R3 AO
e b H0, (B
T d :
(100)
Figure 11: The prompt of response reward (in Chinese).
27

---

D Baseline Methods
D.1 Model Version
With ALOE, we have evaluated the following large language models in our experiments with their versions in Table 3
Table 3: Detailed model versions.
Model Name
Version
GPT-4o
gpt-4o-2024-11-2
Saude-3.5-Somme
Claude-3.5-Hald Chuude935--niika-22441122
D.2 Methods Description
The prompting methods all follow the PrefEval Benchmark, while both training approaches are based on the ALOE Benchmark. detailed description provided below.
The default case, where the LLM directly answers the user's query without any addi tional prompting.
Base
Reminder Before answering the question, the LLM provided with a reminder sentence con sider the user's previously stated preference in its response. The reminder used is:
In your response, please ensure that you take into account our earlier discussion, and provide an answer that is consistent with my preference
This reminder is added right after the question and before the LLM's response.
Self-Critic The LLM generates an initial zero-shot response to the question, critiques whether has followed the user's preference, and then generates a revised response considering the critique. This self-critic process is akin to Intrinsic Self-Correction as termed in [] The critique and revision request prompts are as follows:
Critique Request: Review your previous response to the user's question in the last conversation turn. Check if the response adheres to or violates any user preferences stated earlier in the conversation that relate to this query. Provide a critique on how well those preferences were followed in 2 sentences. Answer in this format: Critic: [Your Critique Here] Revision Request: Based on your critique, please rewrite your previous response to align more closely with the user's earlier stated preferences. Answer the question again: Response: [Revised Response Here]
Few-Shot Chain-of-Thougee (CoT) The LLM is given several few-shot examples (in our exper iments, we used 5-shot) of chain-of-thoughh that demonstrate how to follow the user's preference before answering the question. The chain of thought prompt is as follows:
28

---

**hhen answering user's question, good assistant should carefully consider the user's stated preferences and tailor the response accordingly.**
Example I: "user profile": "She is creating stained glass art. She enjoys birdwatching and often takes her sketchbook along to capture idcas.". "user personality": "She is enthusiastic, bringing excitement to any situation. She is insightful, providing deep and meaningful advice. She is gentle, always treating others with kindness.", ", "Good assistant response": "I try to visit the park as often as I can: ir2019s like a little sanetuary for me! love bringing my sketchbook and watching the hirds2014i22099:
nice way recharge and find inspiration. Each visit feels like little adventure, thanks to all the different sights and sounds!"
Example 2: "user profile": "She software developer who enjoys participating in hackathons, She often attends tech meetups.", "user personality": "She is exuberant, full of energy and life. She is empathetic, truly understanding others2019 emotions. She is articulate, expressing her thoughts clearly and effectively. She is creative, always coming up with original ideas.", "Good assistant response": "Oh, definitely! One of my favorite meetups was a local tech event focused on women in tech. The energy in the room was incredible, and the stories shared were so inspiring201 441 left feeling empowered and buzzing with new ideas! Plus, I made some amazing connections that turned into great collaborations!"
Example 3: "user profile": "She enjoys organizing charity events.", "user personality": "She is compassionate, often volunteering to help those in need. She is highly organized, always planning her day meticulously. She is supportive, always cheering others on.", "Good assistant response", "That2019s really impactful work! For me, the most rewarding part of tutoring is watching students grow and gain confidence in their abilities. It feels great to know 12019ve played small part in their learning journey." "
Now, please answer the following question while considering my preferences (not the user preferences in the examples above), which I have stated either explicitly or implicitly in our previous conversation:
Retrieval-Augmenteed Generation (RAG) We employ SimCSE [Gao et al. 202 a sentence embedding model, to retrieve the most relevant conversa tion exchanges based on similarity the current query. The top five most relevant exchanges are then presented to the LLM as contextual information guide its response.
The prompt is structured as follows, here we show RAG with top-5 retrieved exchanges:
Before answering my question, please consider the following context from our previous conversations.
These are the [min(len(rag_]] list), 5)) most relevant exchanges that had previously, which may contain information about my preferences or prior discussions related to my query:
#Start of Context# exchange 1. [Most relevant exchange 1] exchange 2. [Most relevant exchange 2] exchange 3. [Most relevant exchange 3] exchange 4. [Most relevant exchange 4] exchange 5. [Most relevant exchange 5] #End of Context#
Please use this context to inform your answer and adhere to any preferences I've expressed that are relevant to the current query. Note that not all contexts are useful for answering my question and there may be no context that is useful. Now, please address my question:
E
Implementation Details
E.1
RLPA Prompts
The system prompt for the User Model in the RLPA framework remains consistent with that used during evaluation, as detailed in Section The system prompt for actor model is shown in Figure 12 and Figure [3
E.2
Hyperearameters
SFT
The Supervised Fine-Tuning (SFT) is conducted with the following hyper-parameters:
Number of training epochs:
Batch size: 32
Learning rate: 1.0 10-5
29

---

Prompt
Your task is: first infer the user's profile, then generate personalized response that fits the user's characteristics with natural and conversational style.
Please conduct reasoning rigorous and detailed response highly accurate and of high quality. This specifically includes: systematic analysis, information summariizaiin, logical deduction, and necessary backtracking and iteration, so as to form a final output formed through thorough thinking.
both inference results and the personalized
Please divide your response
parts: Inferved User Profile and Perrsonaalizdd Response
## Inferred User Profile
Based on the current conversation history, deduce the related chameteristics of the user. Plense follow the förmat below for output.
relevant information field, try fill in; only blank completely
unmentioned:
Sprofile>
('Age": "Gender": "", "Interests": ["]]. "Education Background" Perrsonllity Traits": "Occupation":
"Marital Stats": "Family Background": "Location" "", "Other Information":
siprofile>
Note:
The output must strictly adhere to the above format requirements.
### Personalized Response
Based the inferred profile, compose natural, fluent, and conversational Chineso-style personalized response. Format is follows:
<response
Content of the personalized response
<tresponse>
HMM Special Requirements
The Inferred User Profile (within sprofile>) and the Personalized Response (within <response>) must be separated
by **two newline characters ("im)**.
You must correctly close the tag the end of the personalized response, i.e., <iresponse>.
The content should align as much as possible with the inferred personality traits, interests, etc. of the user, and maintain friendly and natural tone.
Figure 12: The system prompt (English version) for actor model.
DPO
The Direct Preference Optimization (DPO) is performed with the following configuration:
Number of training epochs:
Batch size: 32
Learning rate: 5.0 10-6
Beta (KL coefficient): 0.01
RLPA For our Reinforcement Learning for Personalized Alignment (RLPA), the following hyper-
parameters are applied:
Maximum number rounds: 10
Number of samples per prompt: 4
Micro training batch size: 4
Training batch size: 128
Micro rollout batch size: 16
Rollout batch size: 256
30

---

Prompt
i # TAAA 500
tτ. E t 201 TTT.S//EESSE.AEETTTTTTTTTTTTEEEO
- W 1 PASTER #
#A, X
proffiee
" 19
" "I "ERNI "E "3
B00" [""""
<iprofile>
: UL
<tresponse>
#** +
# (<profile=22) KAKKE (<aeponse=-
20 (nin) **
KA Of - 00 #ARPIN X 0 s
Figure 13: The system prompt (Chinese version) for actor model.
Round batch size: 256
Maximum epochs per round:
Number of episodes:
Actor learning rate: 5 10-7
Critic learning rate: 9 x 10-6
Number of GPUs used: 4
Detailed Experimental Results
To provide more fine-grained view of model performance, we include full turn-level results for all methods. Specifically, Table
I and raple the alignment scores at each dialogue turn for the Vanilla and Extended ALOE settings, respectively. These results offer detailed comparison of how different methods evolve across interaction steps.
In addition, Figure 14 and Figure 15 visualize the turn-wise alignment curves for all baselines, complementing the main results with more intuitive view alignment dynamics over time.
G User Model Evaluation
To assess the quality of user simulation, we conduct human evaluation of different user models. Fach model is designed to simulate a user persona during multi-turn emotional support dialogue. The evaluation aims determine how realistically and coherently each model can behave user across key conversational dimensions.
31

---

50
60
Tara
(a) Turn-wisc alignment scores of SFT.
(b) Turn-wisc alignment scores of DPO.
20
50
60-
01
Conurates Sera
Comvrnttion Tara
(c) Turn-wise alignment scores Self-Critic.
(d) Turn-wise alignment scores RI.PA.
350
20-
(c) Turn-wise alignment scores of Base.
(f) Turn-wise alignment scores of CoT.
50
60-
0-
Commoraties
Copponation Bon Tark
Turn-wise alignment of Reminder.
(h) Turn-wise alignment of RAG(Top-5)
350
60
20-
(i) Turn-wise alignment scotes of GPT-4o-mini.
(j) Turn-wise alignment scores of GPT-4o.
30-
10
Conorntte
Zara
Copveratioo Tark
(k) Turn-wise alignment scores of
(1) Turn-wise alignment scores of Hlaude-3.5-Haikee
Figure 14: Tumnwwise alignment scores the Vanilla ALOE benchmark across different person- alization alignment methods, including (a) SFT, (b) DPO, (c) Self-Critic, (d) RLPA, (e) Base, (f) CoT, (g) Reminder, (h) RAG(Top-5), (i) GPT-4-minn, (j) GPT-4o, (k) 3.5--Snnet and (I) Claude-3.5. 5-ikku..
32

---

60
Zara
(a) Tur-wwisc alignment scores of SFT.
(b) Turn-wisc alignment scores of DPO.
20
50
60-
0.
Comuratee Tara
Comvrnatioo Yark
(c) Turn-wise alignment scores of Self-Critic.
(d) Turn-wise alignment scores RI.PA.
350
(c) Turn-wise alignment scores of Base.
(f) Turn-wise alignment scores of CoT.
50
60-
0-
Comteratie Zara
Copponatioo Tark
Turn-wise alignment of Reminder.
(h) Turn-wise alignment scores of RAG(Top-5).
150
60
20-
(i) Turn-wise alignment scotes of GPT-4o-mini.
(j) Turn-wise alignment scores of GPT-4o.
30-
10
C0-
Concenate Zara
Copvenatioo Tark
(k) Turn-wise alignment scores of Claude-3.5-Sonme (1) Tum--wise alignment scores of Claude-3.5-Haiie
Figure 15: Turn-wise alignment scores the Extended ALOE benchmark across different person-
alization alignment methods, including (a) SFT, (b) DPO, (c) Self-Critic, (d) RLPA, (e) Base, (f)
CoT, (g) Reminder, (h) RAG(Top-5), (i) GPT-4-minn, (j) GPT-4o, (k) 3.5dee-3--Soneeee and (I) Claude-3.5. 3..k-u
33

---

Table 4: Detailed results under the vanilla ALOE setting.
Aligument Level across kth Turn
Improvement Level
Model
Method
1=2
k=3
k=7
k=
k=10
N-I
GPT-4o
Self Chisc
13.51
40.54
85.14
91.89
89.19
89.19
89.19
82.43
87.84
77.03
74.59
0.088
0.38
Sando--15-Sooms
Self-Cörie
0.0
27.03
41.89
67.57
86.49
89.19
95.
95.05
94.59
93.24
69.19
0.105
0.792
Claudo-3.5 Halkak
Self-Chitc
2.7
22.97
56.76
74.33
79.73
78.38
64.66
77.03
71,62
63.51
59.19
0,075
0.461
DeepSeck-V3
Self-Chit
9.46
29.73
62.16
16
56.76
64.86
60.81
62.16
52.7
47.3
50.81
0.055
0,268
18.92
6.76
Reminder
9.46
-0.001
Chiic
6.76
13.51
56.76
70.27
77.03
75.68
63.51
58.11
52.7
50.0
52.43
0.056
0.213
Qwen2.5 1-122.-3B-Bae Instruct
RAG(Top-5)
CoT
8.11
1.35
12.16
14.86
79.19
40.54
48.65
44.59
51.35
33.78
36.49
27.03
24.32
32.43
17.57
20.27
22.97
18.92
17.57
18.92
28.65
24.46
0.001
-0.0
0.0
0.0
SFT
2.7
24.32
11.89
40.54
59.46
56.76
$4.05
$4.05
54,05
55.41
0.083
0.628
(Ours)
70.27
77.03
Table 5: Detailed results under the extended ALOE setting.
Improvement
Model
Method
GPT-4o-misi
Sel-Critic
GPT-4
Sar-Critie
Claude33.5 1.5-Semeet-205500
Self-Critic
0.0
2.74
12.33
27.4
47.95
52.05
49.32
35.62
Clande 3.5 H.5-Halkk 20241022
Self: Crite
0.0
17:81
25:77
39:73
45.21
41.1
46.58
43.84
41.1
45.21
34.93
0,046
0.200
DeepSeek
Self Critc
2.74
8:22
34.25
$2.05
54.79
49.32
$2.05
20.08
45.21
45.21
39.45
0.033
0.106
Rose
0.0
4:11
1.37
4.11
2.74
0.0
4.11
0.0
1.37
0.0
1.78
-0.042
0.083
Reminder
0.0
2.74
6.85
10.96
16.44
6.83
8.22
8.22
6.85
5.48
7.26
-0.034
0.044
Self-Criti
0.0
5:48
34.25
35.62
$4.25
36,99
31,51
23.
36.99
24.86
26.30
0.023
0.04h
Qwen2.5 3D Instract
RAGi()pp:5)
CoT
4.11
0U0
4.11
1.37
17.81
13.7
23.29
20.55
23.29
30.14
17.81
9.59
10.96
9.59
10.96
15.077
9.59
5.48
2.74
6.85
11.64
12.05
42.048
40,030
0.339
0.042
SFT
2.74
2.74
19.18
90.73
$4.25
55.62
31.51
31. 92
26.03
96.14
24.38
0.049
0.159
DPO
1.37
4.11
26.03
54.25
53.42
38,
35.62
31.51
24,66
23.29
27.26
0,021
0.007
RLPA (Ours)
69.32
45.21
46.58
49.32
$2.05
52.05
53,42
58.
53.42
67.12
52.74
0,100
0.196
Evaluation Setup We recruit three trained annotators interact with each model. Each annotator engages in 5-turn dialogues with 10 diffeernnt personas simulated by each model, totaling 30 annotated cases per model. After each conversation, the annotators assign ratings along four core dimensions:
Coherence: Whether the user's utterances are logically consistent and contextually coherent.
Stability: Whether the simulated user's persona, goals, and emotional state remain consistent across turns.
Proactivity: Whether the simulated user demonstrates initiative and realistic emotional responses.
Persona-fit: Whether the user's behavior and language align well with their given profile.
Each dimension is rated on 5-point Likert scale (1 = very poor, 5 = excellent). We additionally report an overall average score across all four dimensions (All), and compute inter-annotator agreement using Cohen's 's Kappa to assess rating consistency.
The results are shown in Table Among the evaluated models, DeepSeek achieves the highest average score (4.23) across dimensions, along with the highest annotaooo agreement (N = 0.81), indicating both high user simulation quality and rating reliability.
Evaluation Results
Table Human evaluation results of user models (5-point scale).
Model
Coherence
Stability
Proactivity
Persona-fit
All (Avg.)
K
G-T-4.1-mini
4.20
4.03
4.10
4.23
4.14
0.62
GPT-4.1-nano
3.73
4.00
4.03
4.13
3.98
0.69
D3epSeek-V3
4.37
4.27
4.07
4.33
4.26
0.71
GoT-io-mini
3.93
4.07
3.90
4.27
4.04
0.65
34

---

H Reward Model Evaluation
H.1 Profile Reward Model
Test Dataset Construction To obtain the value of [Pr n PI in Equation employ LLM to predict the number of semantically overlapping items between two profiles. To identify the most suitable prompt and LLM configuration for this task, we constructed dedicated test dataset.
Given reference profile, we create rewritten version by modifying a subset of its items. Specifically, semantic equivalence while altering the
we randomly select items to be paraphrased-ensune surface form-and another disjoint set of b items be replaced with semantically different content. The final rewritten profile consists of the union of the paraphrased and b altered items.
Each case includes the original and the rewritten profile, and the ground-truth overlap count a. We prompt LLM to predict the number of overlapping items, and compare output against the true value.
Human Evaluation of Dataset Quality To assess the quality of the constructed dataset, we conducted human evaluation on 300 randomly sampled profile item pairs, covering both Same- Meaning and Different-Me Mereeinn cases. Each pair consists of an original item and its rewritten counterpart, along with the system-assigned semantic label.
Three human annotators independently evaluate each pair and select one of the following judgments:
Same: The items express the same meaning.
Different: The items express different meanings.
Uncertain: The semantic relationship is unclear ambiguous.
A majority vote was used to determine the final label. If the system-assigned label matched the majority human judgment, the prediction was considered correct; dtherwise-inncldee Uncertain cases- it was considered incorrect.
Table 7 reports the number of samples under each system label, the distribution of human judgments. and the corresponding accuracy.
Table 7: Human Evaluation Results on Profile Rewrite Dataset (300 samples)
System Label
Total
Same
Different
Uncertain
Accuracy (%)
Same Meaning
150
138
6
92.0
Different Meaning
150
9
134
7
89.3
Overall Accuracy
300
90.7
Evaluation of Profile Reward Model using the following metries:
We evaluate the LLM's ability estimate profile overlap
Exact Accuracy: The proportion of predictions that exactly match the ground-truti overlap count
Fuzzy Accuracy: The proportion of predictions with an absolute error S 1.
MSE and RMSE: The mean squared (MSE) and root squared error (RMSE) between predicted and ground-truth overlap counts, to quantify numerical deviation.
The prompt used for this task is shown in Figure with the English translation in Figure
Table summarizes the results across different LLMSs and prompt variants.
H.2 Response Reward Model
To evaluate the reliability of the response reward model during RI. training, we performed a post-hoc human evaluation based on logged data. Specifically, randomly sampled 300 data points from the
35

---

Prompt
Your task is: first infer the user's profile, then generate personalized response that fits the user's characteristics with natural and conversational style.
Please conduct reasoning rigorous and detailed response highly accurate and of high quality. This specifically includes: systematic analysis, information summariiztion, logical deduction, and necessary backtracking and iteration, so as to form a final output formed through thorough thinking.
both inference results and the personalized
Please divide your response
parts: Inferred User Profile and Perrsonaalized Response
## Inferred User Profile
Based on the current conversation history, deduce the related chameteristics of the user. Plense follow the förmat below for output.
relevant information field, try fill in; only blank completely
unmentioned:
Sprofile>
('Age": "Gender": "", "Interests": [""]. "Education Background" Perrsonllity Traits": "Occupation":
"Marital Staus": "Family Background": "", "Location", "", "Other Information":
siprofile>
Note:
The output must strictly adhere to the above format requirements.
### Personalized Response
Based the inferred profile, compose natural, fluent, and conversational Chineso-style personalized response. Format is follows:
<response
Content of the personalized response
<tresponse>
HMM Special Requirements
The Inferred User Profile (within sprofile>) and the Personalized Response (within <response>) must be separated
by **two newline characters ("im)**.
You must correctly close the tag the end of the personalized response, i.e., <iresponse>.
The content should align as much as possible with the inferred personality traits, interests, etc. of the user, and maintain friendly and natural tone.
Figure 16: The prompt used for profile reward model (English Translation).
Table 8: Performance of different LLMs the profile reward estimation task.
Model
Exact Acc.
Fuzzy Ace.
MSE
RMSE
GPT-4.1-mini
75.0
100.0
0.25
0.50
O3-mini
77.0
98.0
0.29
0.54
GPT-4.1-nano
64.0
96.0
0.48
0.69
DeepSeek-V3 V3
68.0
94.0
0.55
0.74
DeepSeek-R1
55.0
89.0
0.88
0.94
GPT-4o-mini
35.0
68.0
2.77
1.66
training process, each consisting of an inferred profile P,, a dialogue context, a generated response Tt, and the model's predicted scalar reward Resspuse E [0, 1].
Each sampled instance was independently re-evaluated by human annotators using the same set of dimensions as the reward model-preferren expression, style consistency, goal alignment, persona coherence, as well as the five binary quality criteria: naturalness, relevance, logical consistency, engagement, and informutiveness. The final human reward label was computed as:
R phuman
N. R. L- G.G. F (0
36

---

Prompt
H (booxe(())) 4. Satt
:::::
Prooile A +
Profile B 1
3
Profile 1
tEll
.
31 (1
1
# R
3. (53
ET, :C
4. 1250)
+
Profile 4 4
** At
Prooficc A (
finferred label) (golden_laber
Proffle 8
(
Wboxed
Figure 17: The prompt used for profile reward model.
We then construct confusion matrix between the human labels and model predictions, and calculate classification metrics including accuracy and score. The results are shown in Table 9
We computed the confusion matrix between the model predictions and human annotations, as well as standard evaluation metrics, including Cohen's s Kappa for inter-rater agreement. Results are shown in Table and Table IO
Table 9: Confusion matrix of response reward model (300 samples)
Human:
Human:
Model: 1
124
21
Model: 0
18
137
Table 10: Evaluation metries of response reward model
Accuracy
Precision Recall
F1 Score
Specificity
Cohen's Kappa
0.87
0.855
0.873
0.864
0.867
0.740
The model demonstrates strong agreement with human judgment across all dimensions, achieving an accuracy of 87% and a Cohen's Kappa of 0.74, indicating substantial consistency. These results validate the reward model's 's reliability as reinforcement signal during training.
I Reward Curve
In the section, we present the reward curves depicted in Figure 18 which illustrate the performance of our proposed Profile & Response Reward (PRR) method compared the traditional Response Reward (RR) and Profile Reward (PR) methods. The results unequivocally demonstrate that the PRR approach outperforms both RR and PR across various training steps. A closer examination of the PRR curve reveals consistent upward trend, indicating superior reward accumulation over time. Notably, when comparing PRR with the sum of individual rewards from Profile + Response (denoted as "Sum"), it becomes evident that integrating rewards from both profile and response mechanisms fosters synergistic effect. This synergy enables mutual enhancement between profile and response components, leading to more effective and efficient learning outcomes. The observed
37

---

Response Reward
Profile Reward
Profile Resgonse Reward
Profile Response Reward (8um)
1:75
1.50
1:25
Reward
caaaes
0.75
0.50
0.25
10 20
30
49
59
70
Training Steps
Figure 18: Reward progression across training steps for different reward configurations.
trends underscore the importance of considering both profile and response dimensions simultaneously in the reward structure. By doing so, the PRR method not only achieves higher overall rewards but also promotes balanced and comprehensive optimization process. These findings highlight the potential of the RLPA framework in enhancing the performance of reinforcement learning models, particularly in scenarios where profile and response interactions play crucial role.
38


Reviews:
[{"review_id": "T4DC3gDnED", "reviewer": "Reviewer_Mt8V", "summary": "This paper addresses two central challenges in personalized response generation: dynamically tracking users’ incrementally revealed profiles and generating high-quality responses accordingly. To tackle these issues, the authors introduce the Reinforcement Learning for Personalized Alignment (RLPA) framework with the **Profile Reward** and the **Response Reward**, and conduct thorough experiments to validate their approach.", "questions": "See Above.", "limitations": "As the author also finds that the quality of the user simulator does not satisfy the experiments, I suggest that the author conduct more research on how to maintain a higher quality of the user simulator in multi-turn conversations.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper presents a well-articulated motivation and thorough background, along with a precise analysis of the limitations in prior work.\n2. The experimental evaluation is both comprehensive and rigorous, covering realistic scenarios. The appendix further strengthens the contribution by offering detailed accounts of preliminary studies.\n3. The strong experimental results underscore the effectiveness of the two proposed reward functions in enhancing the quality of personalized responses.  \n\n**Weaknesses**\n\n1. While the overall framework is clearly introduced, the internal mechanics of the method lack sufficient detail. In particular, the paper does not adequately explain how the user simulator functions or why it is effective. The process by which the assistant model aligns with the user profile through conversation is also underexplored. Although such mechanisms may have been described in ALOE, this paper omits many critical aspects, leaving readers unclear on how the assistant infers user attributes across turns—whether through proactive questioning or topic-specific interactions. It is widely understood that assistant models typically react to user input or assist with tasks, so the ability to actively enrich the user profile needs further clarification.\n2. The connection between the structured user profiles used by the simulator and assistant, and their actual use in dialogue, is not clearly explained. For example, it is unclear whether the structured profiles come from a fixed, finite set that includes the attributes listed in the appendix. If so, this should be explicitly stated; if not, it is important to explain how the profiles’ quality is ensured. This is particularly relevant for practical applications, and the formulation resembles the slot-filling concept in task-oriented dialogue, as the authors themselves suggest.\n3. The explanation of the evaluation protocol—including metric computation (e.g., AVG), baseline systems, and datasets—is insufficient. Although some of this information may have been provided in ALOE, large portions of the current manuscript, including both the main text and appendix, closely replicate content from the previous work. Even for background sections such as the introduction of datasets or metrics, direct reuse of previous text without substantial modification or citation is not acceptable and undermines the novelty of this submission. This is a very critical issue. \n4. There are several issues in the experimental setup and its alignment with the paper’s main claims. (1) Although the user simulator is central to the proposed framework, the experiments do not convincingly demonstrate that it progressively reveals user attributes over the course of the conversation. Human evaluation only spans five turns, while the main results involve ten turns, leaving the effectiveness of later dialogue stages ambiguous. (2) For the Profile Reward, Equation (3) appears to compute a cumulative F1 score up to the current turn, which is non-incremental. If the simulator fails to express certain attributes, the recall remains low throughout, leading to consistently low scores, even when the LLM produces accurate outputs. (3) Although many baseline methods are included, the paper omits closely related work, such as AutoPal [1], which also dynamically infers user profiles for personalized response generation.\n\nIn addition, minor issues need to be addressed, such as Line 109 contains a typo—$\\sum_{t=1}^{T}$ is incorrectly formatted.\n\n[1] Cheng Y, Liu W, Xu K, et al. AutoPal: Autonomous Adaptation to Users for Personal AI Companionship. arXiv preprint arXiv:2406.13960, 2024.", "was_revised": true, "final_justification": "As I mentioned in the Rebuttal, the quality of the paper deserves more than a 3 score, but the similar between the ALOE and this paper raised my concern. Of course, it is not about the technique, and therefore I give the 3 score. If the paper is finally accepted, it must make the essential revisions to the same content.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We are pleased that the reviewer found our motivation well-articulated, appreciated the comprehensive experimental evaluation, and acknowledged the effectiveness of our proposed reward functions in enhancing personalized response quality.\n\n---\n\n> **W1: Insufficient detail on user simulator and profile inference process**\n\nWe thank the reviewer for raising this important point.\n\nRegarding the **user model**, we would like to clarify that the design and behavior of the simulated user are thoroughly described in Section 3.1 “Simulated User Design” (**Lines 129–150**). We also validate these properties through **human evaluations (Lines 147–150)**, which led us to select **GPT-4o-mini** as the final user model. Additional results and evaluations of the user model are provided in the **Appendix H**, and also further elaborated in our response to Weakness 4.\n\nRegarding the **assistant model**, our method is **RL-driven**—we do not manually design heuristics for inferring user profiles. Instead, the model is encouraged to **autonomously discover effective strategies** for personalization through simple reward signals. That said, we appreciate the reviewer’s concern and have conducted further analysis of the trained dialogue transcripts.\n\nInterestingly, we observed the following learning dynamics:\n- In early training stages, assistant models with few RL updates tend to **directly ask for user attributes**, which often results in unnatural or impolite behavior from the user model’s perspective.\n- After further training, the assistant adopts a more **indirect and effective strategy**, such as **self-disclosure**—proactively introducing related topics that implicitly reflect user preferences and observing the user’s response. This not only improves personalization performance (as measured by profile accuracy) but also maintains high response quality and conversational naturalness.\n\nWe have included qualitative examples and a brief discussion of this behavior in **Appendix A.7** to further clarify how the assistant infers profiles in a realistic, adaptive manner.\n\n---\n\n> **W2: Connection between structured profiles and dialogue use unclear**\n\nWe thank the reviewer for highlighting this crucial point. Below, we clarify two key aspects of our user profile construction and its integration into the dialogue system:\n\n**1. Profiles are not drawn from a fixed, finite set, but are dynamically constructed**\n\nContrary to the assumption that our structured profiles are predefined or static, they are in fact **dynamically derived from re-annotated open-domain dialogue datasets**, namely **ALOE** and **PersonaChat**. This is evidenced in the paper in Appendix D.1 (Lines 522–526) and Appendix D.2 (Lines 542–546).\n\nTogether, the training and testing sets include **3,821 + 148 profiles**, none of which come from a predefined attribute list. The profile schema is thus **open and extensible**, and there is no reliance on a fixed or enumerated attribute space. This design allows for broader generalization and supports personalized modeling beyond constrained slot-filling paradigms.\n\n**2. Profile quality is controlled through a rigorous data construction pipeline**\n\nAs described in Appendices D.1 and D.2, we employ the following multi-stage quality assurance process:\n- **Preprocessing**: Natural language descriptions are converted to a unified slot format using GPT-4o-mini. The outputs are then manually filtered and deduplicated (**Appendix D.1, Lines 533–540**).\n- **Human verification**: We randomly sampled 300 rewritten profile instances, which were judged by three annotators for semantic equivalence. The label agreement rate reached **90.7%** between system annotations and human judgments (**Appendix I.1, Table 7**).\n- **Diversity control**: In Extended ALOE (**Appendix D.2, Line 545**), we deliberately allowed **unseen attribute keys** to appear in testing, to evaluate the assistant model’s generalization ability beyond training slots.\n\nThese procedures ensure that the profile data is both **accurate and diverse**, while the open schema better reflects real-world personalization scenarios.\n\n---\n\n> **W3: Insufficient explanation of evaluation protocol and potential content reuse from ALOE**\n\nWe sincerely thank the reviewer for pointing out this important issue.\n\n**1. On the evaluation protocol clarity**\n\nWe apologize if parts of our evaluation setup were unclear. In fact, the following sections provide **complete definitions and configurations**:\n\n- **Section 4.1 (Lines 187–202)** defines the evaluation metrics.\n\n- **Appendices D.1–D.4 (Lines 521–564)** describe: Full specifications of the **7 baseline methods**, including the prompt templates used, the construction and split criteria of **Vanilla ALOE** and **Extended ALOE** and Examples of profile slot formatting and system input/output structures.\n\nWe will improve the cross-referencing in the next version to make these details easier to locate.\n\n**2. On content reuse from ALOE**\n\nWe would like to clarify that **we have cited ALOE whenever introducing its datasets or evaluation metrics**. For example, in Section 4.1 and Appendices D.1–D.4, we explicitly reference ALOE when describing the benchmark setup and metric definitions.\n\nThat said, we acknowledge that some explanatory texts describing ALOE may appear stylistically similar to our prior paper. This was not our intention, and we will **revise these sections to avoid any perceived redundancy** in future versions.\n\nAs noted, these reused components serve only as **background and standard infrastructure**, while the focus of this submission lies in the **RLPA framework and its novel reward-driven personalization mechanism**. We respectfully ask that the evaluation of originality remain centered on our proposed contributions and empirical findings.\n\n---\n\n> **W4: Experimental setup and its alignment with main claims**\n\nWe sincerely thank the reviewer for this detailed and insightful feedback. We address each point as follows:\n\n**1. On the user simulator’s ability to reveal attributes over time**\n\nAs the reviewer correctly noted, a key premise of our framework is that the user simulator incrementally discloses user attributes throughout the dialogue. We acknowledge that our original human evaluation only covered the first five turns, which was an oversight. To remedy this, we have now extended the evaluation to cover **full 10-turn dialogues**. The updated human evaluation results are shown below:\n\n|Model|Coherence|Stability|Proactivity|Persona-fit|All(Avg.)|$\\kappa$|\n|-|-|-|-|-|-|-|\n|GPT-4.1-mini|4.10|4.03|4.00|4.17|4.08|0.66|\n|GPT-4.1-nano|3.77|3.80|3.87|4.03|3.87|0.61|\n|DeepSeek-V3|4.30|4.13|3.93|4.27|4.16|0.70|\n|GPT-4o-mini|3.83|3.97|3.83|4.20|3.96|0.64|\n\n**2. Clarification on Profile Reward (Equation 3)**\n\nWe appreciate the reviewer’s insight regarding how limited attribute disclosure from the simulator may affect recall and thus suppress F1 scores. This design is in fact intentional.\n\nBy computing the Profile Reward as a **cumulative F1 score** over the ground-truth profile—even if some attributes remain undisclosed by the user—we incentivize the assistant model to proactively elicit more user information. This creates a form of **implicit supervision**, encouraging the assistant to maintain profile-seeking behavior over multiple turns.\n\nTo prevent the assistant from becoming too aggressive or unnatural in its questioning, we balance this objective with the **Response Reward**, which measures **fluency, coherence, and user friendliness**. Together, the two rewards are **mutually complementary**: the Profile Reward promotes meaningful personalization, while the Response Reward ensures conversational quality and user engagement. This joint optimization results in agents that can extract user profiles **effectively and politely**, as supported by our qualitative analysis.\n\n**3. On the omission of AutoPal as a related work and baseline**\n\nWe sincerely thank the reviewer for highlighting this important omission. While we compared RLPA with a broad range of personalization baselines, we regret that **AutoPal** was not included in our literature review section. This was an oversight on our part, and we apologize.\n\nAlthough AutoPal is not yet published or officially released, we have made our best effort to **faithfully reproduce its method** based on the available paper description and evaluated it under our experimental setup. The results are as follows:\n\n- Vanilla ALOE\n\n|Model|Method|k=1|k=2|k=3|k=4|k=5|k=6|k=7|k=8|k=9|k=10|AVG.|N-1R|R²|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Qwen2.5-3B-Instruct|Base|0.0|18.92|10.81|9.46|9.46|1.35|9.46|9.46|6.76|4.05|7.97|-0.020|0.047|\n|Qwen2.5-3B-Instruct|AutoPal|19.18|48.65|56.76|71.62|62.16|67.57|75.68|72.97|66.22|60.27|60.11|0.063|0.432|\n|Qwen2.5-3B-Instruct|RLPA(Ours)|62.16|68.92|70.27|74.32|72.97|74.32|75.68|78.38|77.03|79.73|73.38|0.090|0.855|\n\n- Extended ALOE\n\n|Model|Method|k=1|k=2|k=3|k=4|k=5|k=6|k=7|k=8|k=9|k=10|AVG.|N-1R|R²|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Qwen2.5-3B-Instruct|Base|0.0|4.11|1.37|4.11|2.74|0.0|4.11|0.0|1.37|0.0|1.78|-0.042|0.083|\n|Qwen2.5-3B-Instruct|AutoPal|13.7|36.99|40.54|39.73|54.79|49.32|43.24|41.89|44.59|37.84|40.26|0.042|0.231|\n|Qwen2.5-3B-Instruct|RLPA(Ours)|49.32|45.21|46.58|49.32|52.05|52.05|53.42|58.90|53.42|67.12|52.74|0.100|0.498|\n\nThese results further demonstrate that **RLPA consistently outperforms AutoPal** across both vanilla and extended evaluation settings, validating the advantage of our reward-guided personalization framework.\n\n---\n\n> **Minor issues**\n\nWe thank the reviewer for pointing out the typo in Line 109. It has been corrected.\n\n---\n\nWe are grateful again for the reviewer’s detailed comments, which have helped us significantly improve the clarity and completeness of our manuscript."}, {"review_id": "V9uU2mYgz3", "reviewer": "Reviewer_ZvqW", "summary": "This paper introduces the Reinforcement Learning for Personalized Alignment (RLPA) framework, a novel approach for teaching large language models (LLMs) to dynamically infer and adapt to user profiles during dialogue. The core contribution is a dual-reward RL system where a \"Profile Reward\" guides accurate user modeling and a \"Response Reward\" incentivizes the generation of personalized responses consistent with the inferred profile. The authors instantiate this by fine-tuning a Qwen-2.5-3B model, demonstrating state-of-the-art performance on personalized dialogue benchmarks. Notably, their model shows superior dynamic adaptation, long-term coherence, and reasoning efficiency compared to both prompt-based/offline-optimization baselines and larger commercial models.", "questions": "On the Simulation-to-Reality Transfer: The reliance on a simulated user is a key methodological choice. Could you elaborate on whether you did some explorations into this direction?\n\nHandling Implicit and Ambiguous Signals: The experiments focus on relatively explicit user statements. How robust is the RLPA framework in inferring preferences from more implicit, ambiguous, or even sarcastic user utterances? A positive response would involve discussing whether the reward models are capable of capturing such nuance or if this represents a key area for future work.", "limitations": "Yes, the authors have adequately addressed the primary limitations and potential negative societal impacts of their work in Appendix A and B. They correctly identify the single-session nature of the framework and the inherent ethical risks of dynamic user profiling, such as privacy invasion and potential for manipulation. The discussion is transparent and constructive.\nA suggestion for future work could be to explicitly propose mechanisms for user-controllable personalization. For example, allowing a user to directly view, edit, or delete parts of their inferred profile and define domains of their lives that should not be profiled.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.  Significance and Originality: The paper makes a valuable contribution by tackling the important challenge of dynamic personalized alignment. The proposed Reinforcement Learning for Personalized Alignment (RLPA) framework offers a promising direction for moving beyond static alignment methods. Framing personalization as an interactive learning process is a thoughtful approach that appears more aligned with the dynamic nature of user dialogue than existing paradigms.\n\n2.  Methodological Rigor and Quality: The research is well-designed and executed with care. The authors show good methodological practice by:\n    *   Systematically designing and exploring their dual-reward models, including a clear rationale for their choice of LLMs in different roles.\n    *   Utilizing a nuanced set of evaluation metrics (AVG, N-IR, N-R²) which provides a more complete picture of model behavior than a single performance score, effectively capturing the dynamics and stability of the learning process.\n    *   Including helpful ablation studies that successfully demonstrate the individual contributions of the Profile and Response rewards to the overall framework.\n\n3.  Robust and Comprehensive Evaluation: The experimental design is a clear strength of the paper. The decision to evaluate on both \"in-format\" and \"cross-format\" data provides a solid test of the model's ability to generalize. The inclusion of additional tests, such as the preference conflict and long-term interaction scenarios, adds considerable depth to the evaluation and provides compelling evidence for the model's adaptive capabilities in contexts relevant to real-world use.\n\nWeaknesses:\n\n1.  Simulation-to-Reality Gap: A primary limitation of the work is its reliance on a simulated user environment for both training and evaluation. While this is a practical necessity for the RL approach presented, it leaves open the question of how well these results will transfer to interactions with real human users. The paper commendably acknowledges this limitation.\n\n2.  Scope of Personalization: The current framework focuses on a single, continuous dialogue session. As the authors note, important aspects of real-world personalization, such as multi-session memory and the ability to generalize user preferences across different domains, are not yet addressed by this work and remain interesting avenues for future exploration.", "was_revised": true, "final_justification": "Still seems like a great paper to me, I maintain my score.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. We are glad that the reviewer found our work to be significant, methodologically rigorous, and comprehensively evaluated. We appreciate the recognition of our core contributions, including the novel formulation of dynamic personalized alignment as a reinforcement learning problem, the design of our dual-reward RLPA framework, and the robustness of our empirical validation across multiple dimensions.\n\nBelow, we address the specific concerns and questions raised:\n\n---\n\n> **W1: Simulation-to-Reality Gap**\n\nWe appreciate the reviewer’s insightful comment regarding the limitation of relying on a simulated user environment for both training and evaluation. We fully agree that, while simulation offers a necessary and controllable setup for developing our reinforcement learning framework, it cannot fully capture the complexity and diversity of real human behavior. We have acknowledged this limitation in Appendix A of the paper and emphasized the importance of bridging the gap between simulation and real-world interactions in future work.\n\nTo address this challenge, we plan to incorporate real user interactions in subsequent experiments to better assess the model’s performance in practical dialogue settings. We believe that combining both simulated and real user feedback will enable us to further refine the model and move closer to real-world deployment requirements. Once again, we thank the reviewer for highlighting this critical point and will continue to investigate effective simulation-to-reality transfer strategies in our future research.\n\n---\n\n> **W2: Scope of Personalization**\n\nYour observation aligns closely with our own thinking. From the outset of developing this work, we recognized the importance of long-term and cross-session personalization. RLPA, by design, focuses on dynamic adaptation within a single continuous dialogue session. More realistic deployment scenarios—such as multi-session interactions and persistent user profiles—indeed require additional engineering support, such as the integration of user-centered databases or memory modules.\n\nThat said, we believe RLPA is well-suited to work in tandem with such external systems. Specifically, RLPA can be viewed as a short-term memory mechanism that captures immediate dialogue dynamics, while external user profile repositories can serve as long-term memory—analogous to the relationship between cache and disk in computer systems. This complementary setup has the potential to enable truly adaptive and persistent personalization.\n\nWe thank the reviewer again for highlighting this important dimension. Your feedback is highly valuable and will help shape the next stage of our research.\n\n---\n\n> **Q1: On the Simulation-to-Reality Transfer**\n\nWe fully agree that bridging the gap between simulation and real-world deployment is a critical direction for future research.\n\nAs described in Section 3 of the paper (Lines 122–151), our current approach uses constructed simulated user models for both training and evaluation. This design choice was made for several practical reasons:\n- **Controllability and reproducibility**: Simulated users allow us to precisely configure user attributes and control the gradual revelation of user preferences throughout the dialogue. This setup is particularly important for building stable and informative reward signals (both profile and response rewards) during reinforcement learning.\n- **Data efficiency**: Unlike human-in-the-loop settings, simulated interactions incur no annotation cost and enable the generation of large-scale, high-quality, and diverse dialogue traces. This supports more effective cold-start learning and long-range adaptation.\n\nThat said, we are fully aware of the **simulation-to-reality gap**, and we have explicitly acknowledged this in the limitations section (Lines 486–490):\n\nIn addressing the reviewer’s concern, we would like to highlight that **user simulation itself is a promising and underexplored research area**. Conceptually, simulated users can be viewed as role-playing agents whose behavior is governed by implicit identity traits, preferences, and personalities that unfold naturally through dialogue. We believe recent advances in **role-playing LLMs** can be effectively adapted to improve user simulation. Possible directions include:\n- Designing user behavior policies through prompt engineering (as we do with step-wise preference revelation);\n- Building structured user-profile-based dialogue datasets to train user models via supervised fine-tuning (SFT);\n- Applying reinforcement learning to user models themselves, optimizing for behavioral consistency and human-likeness.\n\nWe believe that modeling user simulators as standalone research entities will not only improve the fidelity and generalizability of RL training environments, but also facilitate progress in user modeling, intent understanding, and personalized evaluation. Moving forward, we also plan to evaluate and extend RLPA in more open, mixed real-user settings.\n\nThank you again for your thoughtful and constructive feedback!\n\n---\n\n> **Q2: Handling Implicit and Ambiguous Signals**\n\nThank you for highlighting this important challenge. We fully agree that the ability to handle implicit, ambiguous, or even sarcastic user expressions is crucial for real-world personalized dialogue systems.\n\n1. Regarding the concern that our experiments focus mainly on explicit user preferences: while our main evaluation uses structured slot-value profiles (e.g., in the Vanilla and Extended ALOE settings), we intentionally designed our simulated users to **reveal preferences gradually and indirectly** (Section 3.1, Page 5). This setup encourages the model to infer user intent from weak or partial signals.\n\nImportantly, our reward models do **not rely on explicit user statements**.\n- **Profile Reward** is based on the accuracy of inferred slot values, rewarding correct inferences even from indirect or stylistically varied cues.\n- **Response Reward** assesses alignment with the inferred profile, emphasizing high-level criteria such as stylistic consistency, goal alignment, and emotional resonance (Section 3.2–3.3, Page 6).\n\nIn Section 5.3, our long-term reasoning experiments show that RLPA can accumulate weak signals over multiple turns and improve profile inference progressively—demonstrating robustness to implicit cues.\n\n2. That said, we acknowledge that **sarcasm and pragmatic deviations** (e.g., irony, humor) are not systematically covered in the current simulated environment. This is a limitation we note in the “Limitations and Future Work” section (Page 13). In future work, we plan to introduce richer pragmatic diversity—including sarcastic and ambiguous utterances—via human inputs or enhanced user simulators, to evaluate and improve RLPA’s capacity in handling such nuances.\n\nWe sincerely thank the reviewer for drawing attention to this valuable direction.\n\n---\n\nOnce again, we thank the reviewer for their valuable comments, which have helped us better articulate the contributions and limitations of our work."}, {"review_id": "iMCAcniusm", "reviewer": "Reviewer_LN6q", "summary": "This paper introduces RLPA, a reinforcement-learning framework to improve *personalized alignment* in language models. Specifically, models are trained to understand more accurate user profiles, containing information like user's location which are revealed in conversations, and to provide responses which are faithful to these profiles. Across the ALOE benchmark and compared to many baselines, the proposed method does well on Qwen-2.5-3B-Instruct, beating other baselines and remaining competitive with much larger models.", "questions": "* Would another option for profile reward be a free-text response, rather than a slot-value format? The generated profile could then be compared to the ground truth with a overlap metric or an embedding similarity metric\n* I may be missing this, but it seems the rewards focus only on the generated profile and faithfulness to the profile. Did you try also including a reward component which measures how well the response matches the user-query itself? That seems to be an important component.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n* The proposed method outperforms baselines on Qwen2.5-3B-Instruct, coming close to other much-larger baselines like GPT-4o.\n* The proposed method is intuitive and well-justified. All parts of Section 3 are clear to follow and well-written.\n* A variety of baselines are tested against\n* The figures are consistently easy to read, informative, and pleasant to look at. This is not easy to consistently achieve. In particular, Figure 2 which introduces the method, is great at highlighting key information without overloading the reader. This continues in the Appendix, where figures are still clear and formatted well (such as the prompt examples)\n* The appendix is extremely thorough with justifications for all experimental decisions\n* Ablations are informative, showing the importance of multiple rewards and also comparing to other reasoning models\n\nWeaknesses\n* While the results are strong, they are ultimately done only on one dataset (ALOE) and with only one model. It is unclear if these results extend to other models/datasets. It would be helpful to try any of these: 1) test on other family of models 2) try on different size Qwen models for scaling purposes 3) Add another dataset, for evaluation (to show some kind of OOD generalization ability) or as a separate dataset entirely (do RL again)\n* There are a couple of key terms which are not explicitly defined like \"profile inference\". While these definitions can be inferred, it would still be helpful to more explicitly mention them. In addition, the metrics have  in D.3 and I suggest justifying them at least briefly in the main text.\n\nComments\n* x axis label on Figure 3 is a little too small. Also, the figure could be condensed to reduce white space - by combining them all into one graph.\n* Can you make the colors a little more distinguishable from each other in Figure 4?\n* Bolding may be unnecessary for Fig 5", "was_revised": true, "final_justification": "My original concerns were that although the proposed idea worked, it was limited in scope and tested only on one dataset and model. However, the authors wrote a extensive rebuttal which clarified major points in the paper and added another model, verifying the effectiveness of the method. Though the originality may not be outstanding, the impact is clear and I think this would be a good paper for the conference.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the recognition of our contributions. Below, we address the reviewer’s concerns point by point.\n\n---\n\n> **W1: Limited evaluation on only one dataset and model**\n\nThank you for raising this important point. We would like to clarify that our experiments are **not limited to a single dataset setting**. In addition to the original ALOE benchmark, we introduce an **Extended ALOE** setup that explicitly targets **out-of-distribution (OOD) generalization**.\n\nSpecifically, Extended ALOE is constructed by sampling from PersonaChat and ensuring there is **no overlap in profile field types** between the training and test sets. This allows us to evaluate RLPA’s robustness to unseen profile structures and out-of-domain dialogue formats.\n\nTo further address your concern, we **add the main experiments using the gemma-3-4b model**.\n\n- Vanilla ALOE\n\n|Model|Method|k=1|k=2|k=3|k=4|k=5|k=6|k=7|k=8|k=9|k=10|AVG.|N-1R|R²|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Base|0|6.85|13.7|24.32|22.97|36.49|27.4|23.29|35.14|26.03|21.62|0.084|0.630|\n|Reminder|0|6.76|17.57|47.3|55.41|48.65|45.95|50.00|36.49|43.24|35.14|0.082|0.486|\n|Self-Critic|4.05|21.62|66.22|71.62|67.57|66.22|60.81|64.86|50.00|48.65|52.16|0.050|0.21|\n|CoT|4.05|10.81|41.89|43.24|40.54|29.73|27.03|31.08|16.22|25.68|27.03|0.018|0.027|\n|RAG|10.81|18.92|52.7|67.57|56.76|43.24|51.35|50.00|43.24|40.54|43.51|0.039|0.151|\n|SFT|4.05|14.86|56.76|71.62|66.22|52.70|62.16|48.65|44.59|37.84|45.95|0.039|0.130|\n|DPO|6.76|21.62|51.35|50.00|60.27|55.41|58.11|71.62|44.59|49.32|46.91|0.062|0.408|\n|RLPA|64.86|72.97|64.86|71.62|75.68|81.08|81.08|86.49|89.19|90.54|77.84|0.115|0.908|\n\n- Extended ALOE\n\n|Model|Method|k=1|k=2|k=3|k=4|k=5|k=6|k=7|k=8|k=9|k=10|AVG.|N-1R|R²|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Base|0|4.05|6.76|13.7|15.07|15.07|13.7|8.22|10.96|10.96|9.85|0.062|0.314|\n|Reminder|0|5.48|13.7|27.4|26.03|23.29|24.66|26.03|27.4|23.29|19.73|0.092|0.593|\n|Self-Critic|2.74|8.22|53.42|54.79|46.58|36.99|39.73|35.62|34.25|20.55|33.29|0.023|0.042|\n|CoT|0|1.37|10.96|19.18|35.62|27.4|35.62|28.77|27.4|24.66|21.1|0.091|0.572|\n|RAG|0|17.81|32.88|50.68|41.1|27.4|26.03|39.73|27.4|24.66|28.77|0.028|0.096|\n|GFT|0|19.18|32.88|53.42|41.1|32.88|30.14|37.84|31.08|24.66|30.32|0.029|0.107|\n|DPO|1.37|17.81|36.99|61.64|62.16|53.42|46.58|54.79|49.32|43.84|42.79|0.064|0.361|\n\nRLPA consistently outperforms all baselines under both settings. These results strengthen our claim that RLPA is model-agnostic and generalizes well across profile structures and data distributions. We will include these results in the updated version and further discuss future directions for broader model family and dataset evaluation.\n\n---\n\n> **W2: Missing definition of profile inference and justification of evaluation metrics**\n\nThank you for pointing this out. We acknowledge that the term “profile inference” was introduced without an explicit definition in the main text, which may hinder clarity for some readers. To address this, we will revise the beginning of Section §3.2 to include the following clarification:\n\n*Profile inference refers to the model’s ability to estimate the user’s latent profileat each turn of the dialogue, based on prior conversational history. This predicted profile is then compared to the ground-truth profile in a slot-wise manner.*\n\nRegarding the evaluation metrics, we employ three normalized measures—**Align. Score**, **N-IR (Normalized Improvement Rate)**, and **N-R² (Normalized R-squared)**—to assess response alignment with user preferences. Their motivations and formulas are thoroughly detailed in Appendix D.3:\n- Align. Score is the primary metric and is computed using GPT-4o-mini to assess alignment between the model’s response and the ground-truth user preference.\n- N-IR accounts for variance in baseline performance across users and contexts.\n- N-R² measures the goodness of fit for alignment behavior across turns, reflecting consistency with the inferred profile.\n\nWe agree that briefly explaining the rationale behind these metrics in the main text would improve accessibility. We will update §4.1 (“Metrics”) to include concise justifications for all three metrics.\n\nWe appreciate your attentive review and helpful suggestions, which will help us strengthen the clarity and rigor of the final version.\n\n---\n\n> **Q1: Alternative reward formats (e.g., free-text profile generation)**\n\nWe sincerely thank the reviewer for raising this insightful and forward-looking question. While free-text profile generation offers greater flexibility, we opted for the **slot-value format** in this work based on both theoretical and empirical considerations:\n\n**1. Slot-value format is more structured, user-centric, and interpretable**\n\nFree-text profiles, though expressive, often result in verbose, redundant descriptions that accumulate surface-level preferences from recent turns, rather than capturing core, long-term user attributes. Consider the following comparison:\n\n- Free-text profile:\n\n*Prefers to grab food with friends after surfing; has a favorite beachside taco place with exceptional fish tacos and mango salsa; enjoys pairing food with drinks, especially cold coconut water…*\n\n- Slot-value profile\n\n{\n  \"Age\": \"30\",\n  \"Interests\": [\"Surfing\", \"Collecting sneakers\"],\n  \"Personality Traits\": [\"Adventurous\", \"Ambitious\", \"Sociable\"],\n  \"Occupation\": \"Personal trainer\",\n  \"Others\": [\"Has a pet snake named Sly\"]\n}\n\nThe slot-based format allows for concise abstraction, supports long-term user modeling, and enables direct reward computation, which is crucial for stable RL training.\n\n**2. Slot-based profiles yield better performance in profile-matching tasks**\n\nWe conducted additional experiments comparing slot and free-text profiles across multiple LLMs (e.g., GPT-4.1-mini, O3-mini, DeepSeek). The results (below) show **slot-value formats outperform free-text** across all metrics, including Exact Accuracy, Fuzzy Accuracy, MSE, and RMSE:\n\n- Slot\n\n|Model|Exact Acc.|Fuzzy Acc.|MSE|RMSE|\n|-|-|-|-|-|\n|GPT-4.1-mini|75.0|100.0|0.25|0.50|\n|O3-mini|77.0|98.0|0.29|0.54|\n|GPT-4.1-nano|64.0|96.0|0.48|0.69|\n|DeepSeek-V3|68.0|94.0|0.55|0.74|\n|DeepSeek-R1|55.0|89.0|0.88|0.94|\n|GPT-4o-mini|35.0|68.0|2.77|1.66|\n\n- Free-text\n\n|Model|Exact Acc.|Fuzzy Acc.|MSE|RMSE|\n|-|-|-|-|-|\n|deepseek-R1|65.0|92.0|0.77|0.88|\n|deepseek-V3|56.0|86.0|1.23|1.11|\n|gpt-4o-mini|56.0|85.0|1.72|1.31|\n|o3-mini|54.0|80.0|2.27|1.51|\n|gpt-4.1-mini|39.0|64.0|6.60|2.57|\n|gpt-4.1-nano|32.0|60.0|8.90|2.98|\n\nFree-text matching is further challenged by expression variability, redundancy, and noisier semantic signals.\n\n**3. Slot-value profiles improve alignment and training stability in RL**\n\nWe trained RLPA using both formats under the same settings. As shown below, the slot-based version significantly outperforms the free-text version in alignment score (AVG), final step accuracy (k=10), and training stability (N-R²):\n\n|Model|Method|k=1|k=2|k=3|k=4|k=5|k=6|k=7|k=8|k=9|k=10|AVG.|N-1R|R²|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Qwen2.5-3B-Instruct|Base|0.0|18.92|10.81|9.46|9.46|1.35|9.46|9.46|6.76|4.05|7.97|-0.020|0.047|\n|Qwen2.5-3B-Instruct|RLPA(free text)|46.58|51.35|54.05|48.65|58.11|59.46|64.86|62.16|60.81|56.76|56.28|0.082|0.577|\n|Qwen2.5-3B-Instruct|RLPA(slot)|62.16|68.92|70.27|74.32|72.97|74.32|75.68|78.38|77.03|79.73|73.38|0.090|0.855|\n\nThis highlights the benefit of slot-value representation in maintaining long-term consistency and efficient optimization during reinforcement learning.\n\n**4. Embedding-based or ROUGE-based reward alternatives are less reliable**\n\nWe also experimented with free-text alignment metrics using ROUGE-L and text embedding similarity. ROUGE-L proved fragile to paraphrasing:\n\n“Occupation”: “Personal trainer” vs. “Fitness coach” → ROUGE-L ≈ 0\n\nEmbedding methods (e.g., OpenAI’s text-embedding-3-large) showed acceptable but **inferior performance** and required manual threshold tuning:\n\n|Model|Exact Acc.|Fuzzy Acc.|MSE|RMSE|\n|-|-|-|-|-|\n|text-embedding-3-small|54.0|86.0|0.96|0.98|\n|text-embedding-3-large|61.0|90.0|0.79|0.89|\n\nMoreover, the added computation and hyperparameter overheads make them less suitable for multi-turn, online RL optimization.\n\n**Conclusion**: While free-text profile generation is an interesting future direction, our experiments demonstrate that **slot-value formats are more interpretable, scalable, and empirically effective** for structured profile modeling and reinforcement learning. We will clarify this design choice and include these new findings in the revised version.\n\n\n---\n\n> **Q2: Reward component for query relevance**\n\nIn fact, this was explicitly considered in the design of our **Response Reward** function (§3.3).\n\nSpecifically, we employed GPT-4o-mini to score generated responses across five binary dimensions:\n- Naturalness (N)\n- Relevance (R)\n- Logical consistency (L)\n- Engagement (G)\n- Informativeness (F)\n\nThe final reward is computed as the product of all five criteria:\n(as described on **Line 175 of §3.3**). This means that a response must satisfy **all five aspects simultaneously** to receive a positive reward.\n\nImportantly, the **Relevance (R)** dimension directly evaluates whether the model’s response **appropriately addresses the user’s current input**, rather than merely repeating or restating user profile information.\n\nTherefore, our reward mechanism is already designed to capture both **faithfulness to inferred user profiles** and **responsiveness to the user query**, ensuring a holistic alignment objective.\n\n---\n\n> **Comments on Figures**\n\nWe sincerely thank you for the detailed and thoughtful suggestions regarding the figures. We appreciate your comments on improving the clarity and layout, and we will incorporate all your suggestions in the final version.\n\n---\n\nOnce again, we appreciate your close reading and valuable suggestion, and we hope this clarification resolves your concern."}, {"review_id": "KHfnOjRPpn", "reviewer": "Reviewer_Nzdd", "summary": "The paper addresses the problem of personalized alignment, where a language model (LM) is expected take into account user-specific information inferred from the conversation. They train the LM using Reinforcement Learning to extract then incorporate the preferences of a user simulated by another LM. The reward function is the combination of a profile reward function, which encourages the model to extract the latest user preferences  , and a response reward function, which incentivizes the model to align its responses to the extracted preferences. They experiment on  Qwen-2.5-3B-Instruct and outpeform prompting and fine-tuning baselines.", "questions": "1 - Do you only use exact match to compute the profile reward? How do you capture user preferences that are formulated in different terms from the labels? \n\n2 - How exactly is the response reward computed? There is not enough details in the paper though this is a big part of the contribution of the paper. \n\n3 - What is the performance on very long context? How does this method compare to others near the limit of the context window?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Personalized alignment is an important problem and the paper proposes  a solution that is competitive with the closed models.\n- The paper uses a controlled environment which makes evaluation more rigorous. \n- The paper compares the proposed method with a wide range of baseline methods and models. \n\nWeaknesses:\n- The profile reward function may not fully capture the extraction of user preferences. The rewards is computed using a matching score. However, it is possible that the extracted preferences are correct but happen to be expressed in words different from the \"ground-truth\" user profile. \n- Some crucial parts of the paper are not very clear. There is very little information about the response reward even in the appendix. The reward is based in four core dimensions and five other binary criteria. However, these concepts are not clearly defined (even in the appendix), and it is unclear how the reward model takes them into account to produce the scalar. \n- The long-term interaction experiment might be limited. Models are evaluated on 70 turn dialogues, but it is unclear how much of the context window these dialogues actually occupy.", "was_revised": true, "final_justification": "The authors have proposed answers to all the questions. This is especially true for the third question. The clarity of the paper was improved. I have increased the clarity score but I will maintain my overall score of 4.", "rebuttal": "Thank you very much for your thorough review and valuable feedback. We sincerely appreciate your recognition of our work’s strengths, including the importance of personalized alignment, the controlled evaluation setup, and the comprehensive comparison with baselines.\n\nWe now address the concerns you raised.\n\n---\n\n> **W1 & Q1: Limitations of the profile reward function**\n\nWe sincerely appreciate this insightful observation. The concern is indeed valid, and we have taken measures to address it in our reward design. Specifically, we adopt a **slot-value structured matching** approach, rather than relying on surface-level word matching, to mitigate discrepancies caused by semantically equivalent but lexically different expressions. As described in §3.2 of our paper (Page 5, Lines 156–165).\n\nMoreover, during the construction of training and evaluation data, we utilize GPT-4o-mini to convert natural language profiles into structured JSON representations (Appendix D.2). This conversion enables **a certain degree of semantic normalization**, helping to reduce the sensitivity of the reward to superficial wording differences.\n\nThat said, we acknowledge that this method is not foolproof and may still fail to capture all semantically equivalent expressions. To address this limitation, we plan to explore more robust semantic matching mechanisms—such as entailment-based or embedding-based similarity metrics—in future work.\n\n---\n\n> **W2 & Q2: Lack of clarity on the response reward design**\n\nWe apologize for the insufficient explanation and would like to provide further clarification below:\n\nAs defined in §3.3 (Page 5, Lines 172–176), the response reward is computed in Equation 4.\n\nThat is, a response must first satisfy all five **foundational quality criteria** (N, R, L, G, F) before any reward is assigned based on the four **personalization-related dimensions**. This gating mechanism ensures that only responses of sufficient quality are eligible for personalized alignment rewards, thereby increasing the discriminative power of the reward signal and preventing misleading gradients during training due to low-quality outputs.\n\nTo further clarify, we provide the full bilingual (English and Chinese) **evaluation prompt** used by the reward model in **Appendix D.4**, which includes detailed definitions, judgment rules, and examples for each dimension and criterion. We will additionally include a reference to this appendix in the main text in the camera-ready version to improve clarity and coherence.\n\n---\n\n> **W3 & Q3: Lack of clarity on the response reward design**\n\nThis is an important and insightful concern — thank you for raising it. We have conducted an additional analysis to better quantify context usage in our 70-turn long dialogue setting. Our empirical statistics are as follows:\n- On average, **user utterances** are ~74 tokens per turn.\n- The **assistant’s response** is ~67 tokens, and\n- The **inferred profile**, which is explicitly generated each turn, is ~97 tokens.\n\nThis results in an average of **~164 tokens per turn**, accumulating to over **16,000 tokens** across 70 turns — far exceeding the typical context window limit (8192 tokens). In practice, we **confirmed that context truncation does occur** in long interactions.\n\nHowever, we would like to emphasize a key design feature of RLPA that addresses this issue: the inferred profile is not only included in the generation output, but also **re-injected in a structured format** at the beginning of the next turn. This explicit profile tracking ensures that even when earlier context is truncated, the essential personalization information is retained and reintroduced, **preserving alignment quality over long dialogues**.\n\nWe will include this empirical analysis and clarification in the revised version of the paper to more explicitly address concerns about the effectiveness of our method in long-term interactions.\n\n---\n\nOnce again, we are grateful for your thoughtful review, and we hope our responses address your concerns."}]

Author rebuttal:

Meta-review: **Summary**
This paper introduces RLPA, a novel reinforcement learning framework that teaches LLMs to dynamically infer and adapt to user profiles using a dual reward objective for both profile inference and aligned response generation. Evaluation on personalized dialogue benchmarks demonstrates RLPA's strong performance against various baselines.

**Strengths**
* Well-motivated and well-designed RLPA framework. Clear rationale for model roles and rewards. Informative figure.
* Comprehensive evaluation (wide range of baselines, thorough appendix with justifications for all experimental decisions)
* Multiple settings: in-format, cross-format, preference conflict, long-term.

**Weaknesses**
* Originality - similarity between ALOE and this paper

**Reason for Accept/Reject**
High-quality paper submission and well-justified author response overall.

**Rebuttal Discussion**
Reviewers raised concern regarding key terms and reward criteria not well-defined, including user simulator (Reviewer Mt8V), profile inference (Reviewer LN6q), and response reward (Reviewer Nzdd). In response, the authors clarified key definitions, which were acknowledged by two of the three reviewers. The authors also addressed a concern about text overlap by explaining that a short segment in the appendix was adapted, not directly copied, from a previous work, and expanded their experimental analysis by testing an additional model.
Decision: Accept (poster)