PAMDP: Interact to Persona Alignment via a Partially Observable Markov Decision Process
Authors: ZHE YANG
Yi Huang
Si Chen
Xiaoting Wu
Jingyu Yao
Junlan Feng
Venue: ICLR 2026 Poster (2026)
Keywords: persona alignment
interact to align
partially observable markov decision process
dual critic reinforcement learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The interaction process of comprehending user-specific nuances and adapting to their preferences represents a pivotal consideration for Persona Large Language Models, as it more authentically mirrors genuine dialogue dynamics than adherence to general human value alignment. In this paper, we conceptualize this ``Interact to Persona Alignment'' challenge as a Partially Observable Markov Decision Process, abbreviated as PAMDP, wherein the user’s dynamically evolving profile through interaction is treated as an unobservable variable to the assistant. Grounded in this formulation, we propose a dual-critic reinforcement learning framework, with a continuous latent space action representing the assistant’s utterance. We evaluate our approach on both offline datasets and the online simulator, ultimately demonstrating its effectiveness.

Published as a conference paper at ICLR 2026
PAMDP: INTERACT TO PERSONA ALIGNMENT VIA A PARTIALLY OBSERVABLE MARKOV DECISION PROCESS
Zhe Yang Yi Huang 12 ;i Chen Xiaoting Wu Jingyu Yao Junlan Feng
JIUTIAN Research: Department of Computer Science and Technology, Tsinghua University, China yangzhe, huangyi, chensi, wuxiaoting, yaojingyu, f'ngjunlan| ) @ @emji.chinamobbie.com
ABSTRACT
The interaction process of comprehending user-specific nuances and adapting their preferences represents a pivotal consideration for Persona Large Language Models, as it more authentically mirrors genuine dialogue dynamics than adherence to general human value alignment. In this paper, we conceptualize this "Interact to Persona Alignment" challenge as a Partially Observable Markov Decision Process, abbreviated as Persona Aligmment MDP (PAMDP), wherein the user's 's dynamically evolving profile through interaction treated as an unobservable variable to the assistant. Grounded in this formulation, we propose dual-critic reinforcement learning framework, with a continuous latent space action representing the assis- tant's utterance. We evaluate our approach on both offline datasets and the online simulator, ultimately demonstrating its effectiveness.
INTRODUCTION
The alignment of large language models (LLMs) with human value preferences is typically achieved through post-training alignment techniques, particularly supervised fine-tuning (SFT) and reinforce- ment learning from human feedback (RLHF) Ouyang et al.) (2022). are commonly implemented as standard approaches in the field. These methods aim optimize models satisfy general human preferences, including criteria such as helpfulness, harmlessness, and honesty. However, human preferences exhibit significant heterogeneity across different user groups and individuals. Moreover, even for a single user, preferences may demonstrate subtle variations depending on contextual factors. The single reward model RLHF overlooks the rich diversity of human preferences inherent in data collected from multiple users, which leads to the inability of L.L.s to align with user-specific preferences Wu et al. (2023). As CsCharabbyyyyy et al. theoretically and empirically demonstrates, the conventional RLHF approach with monolithic reward mechanism cannot adequately capture the full spectrum of diverse human preferences. This limitation underscores the need for more nuanced alignment frameworks capable of accommodating preference variability.
To enhance personalization, specifcc data-suse such as profiles, interaction histories, and behavioral patterns are utilized to generate responses that align with individual preferences and contextual needs Liu et al. (2025); ; Li et al. (2025). This adaptive approach ensures greater relevance and customization in model outputs. However, obtaining sufficient user-specific data for personalization remains nontrivial due to privacy restrictions and the inherent sensitivity of personal behavioral data. Wu al. propose training LLMs align with individual preferences through interactions. To achieve this goal. they establish distinct user personas and multi-turn preference data to fine-tune LLMs, enabling it to explicitly infer user preferences during interaction and generate personalized responses. As the conversation goes deeper, the alignment level is expected to improve iteratively with each conversational turn.
If the higher personalized alignment level is regarded as the long-term goal, the multi-turn interaction can be formulated as a decision-making process and optimized by reinforcement learning (RL). As the interaction with the user deepens, the assistant dynamically refines its understanding of user characteristics, thereby optimizing alignment with individual user traits over time.
"Corresponding Authors: Yi Huang and Junlan Feng

---

Published as a conference paper at ICLR 2026
In this paper, model personalized multi-turn interactions Partially Observable Markov Decision Process [POMDP)Kaclblin et al. (1998), where user profile or implicit preferences are treated unobservable environmental context and the dialogue agent acts the decision-making entity. We denote this framework as the Persona Alignment Markov Decision Process (PAMDP). Unobservable context is typically hidden during deployment but revealed during training. Drawing inspiration from the adoption of an offline learning and online execution paradigm in POMDP settings Baisero & Amato 2022); Li et al. (2024)), we explore an asymmetric actor-critic framework, where the actor takes observation as input and the critic takes all observable and unobservable information as input, better adapt to user profile or implicit preferences during deployment. Specifically, we perform an iterative decomposition of the observable state, profile and action variables and derive the Bellman equation for the PAMDP. Based on this formulation, we further establish the advantage value function in the form of a dual-critic mechanism, where an observation state based value function and an unobservable state conditional value function are estimated for advantage value, which is proven an unbiased estimate of observation state based value while maintaining exploitation of unobservable information during training. In summary, our contributions as follows:
We introduce the notation of the PAMDP, derive the Bellman equation for the PAMDP (Theorem T to further estimate the advantage value (Theorem ( and address the PAMDP in the way of dual critic (Section B. To the best of our knge, this is the first work to propose PAMDP and from new perspective address personalized alignment.
We theoretically prove that dual-critic formulation yields unbiased estimate for the advantage function while maintaining exploitation of unobservable information during training (Theorem 3)
We conduct extensive experiments evaluate the effectiveness of our dual-critic method. Experi- ments on offline and online settings demonstrate that our dual-critic method is effective in adapting user profile or implicit preferences during deployment (Section 4).
2 METHODOL.OGY
In this paper, we delve into the "Interact to Persona Alignment" problem, wherein iterative dialogue interactions progressively refine and augment user's profile, culminating highly personalized responses that resonate with the user's intrinsic preferences and behavioral pat- terns. Formally, the dialogue is initiated with primordial user profile or implicit preferences (simply referred to "profile" in the following text), denoted as wi = Winit (can be empty as well), and the user's query as qo. Concurrently, the assistant acquires an initial dialogue history, subject to the equation that ho go. and gener- ates a response, i.e., tgg conditioned on this state. At interaction timestep t, the dialogue history is formalized as the sequence ht (go, uo, .... Qt), where q and u; correspond to utterances of the user and ussistant, separately. During this phase, the dialogue dynamically updates the user pro- file representation to (a comprehensive ex- ample of the dynamic characteristics of the user profile can be found in Tables 8 and 9 Section D) notably, it is generally assumed that the profile information remains unobservable to the assistant's 's decision-making across the full interaction cycle, necessitating that the assistant learns an optimal interaction policy through feedback of the user. Leveraging this strategy, the assistant synthesizes an adaptive utterance u4 optimized for perceived user-specificity through discourse-awar personalization.
W2-1
W2
14
Figure I: Probability graph for the "Persona Inter- action Process". At step i, conditioned on the ob- servable state (or dialogue history in the interaction setting) hú, the assistant formulates its response u, Subsequently. the environment (the user), leverag- ing both h2 and u, and the unobservable profile content wi, executes state transition, advancing the state to hi+1, while providing feedback re- ward r; in accordance with the assistant's response.
POMDP: A Partially Observable Markov Decision Process is denoted as a tuple (8, H,H,U, Q, D(w)), where Q is the unobservable variable (reflecting the profile environment
2

---

Published as a conference paper at ICLR 2026
context in this paper), and D() represents context based dynamics:
D(w) (p(-)so, r(s, w.)) s.t. (so,w,u E (So. 2,21), S = S U. Un.
(1)
S means the state space, consisting of both observable and unobservable components, that is: S, and Q. H and l denote the realizable history and action spaces. Generally, the agent would incorporate the history to derive optimal policy, i... Tu/h E H).
The POMDP can be effectively solved employing an Advantage Actor-Critic (A2C) model Konda & Tsitsiklis 1999); Mnih et al. (2016). wherein the advantage is estimated by the cemporal-differeee (TD) Sutton 1988) Cai et al. (2019):
A = r(s. w, u) + (V)) - (h))
(2)
Asymmetric A2C: Asymmetric A2C method introduces state value estimator conditioned on both the history and the unobservable variable, i.c., V (h,,w), and is integrated via a linear combination operation with (h) mentioned in Equation
Aasy = (r + (V('', V(h,w) w)) + + ()(((s).. + - Vh))
s.t. 3 E (0.
If 3 - 1, the method is named Unbiased Asymmetric Actor-Critic (UAAC) Baisero & Amato (2022) ;
otherwise, constitutes Dual Critic Reinforcement Learning (DCRL) framework Li et al. (2024a) (seeing in Section
Definition: We formulate the challenge of personalization with interaction POMDP problem, denoted as Persona Alignment MDP (PAMDP), where the assistant acts as the agent, and the user profile serves an unobservable environment context. The dialogue history, i.e., h4, is treated as the observable part of the state, which implies h = in the dialogue setting. The assistant response u, (= (u. .... ut ), where uf denotes the & th token in the response) constitutes the action. Thus, the environment primarily consists of the user which drives the state transition, and its profile information operates as a critical latent factor driving interaction dynamics, while remaining unobservable to the assistant. Additionally, the reward is determined based on the context and the action taken. Our objective to optimize the assistant's 's policy model to maximize the expected return over the dialogue trajectory, formally expressed as:
n² = max EWED,
Ar(h: h, = w, 14 = u)))
It is imperative to highlight that, previously established, given h s, within our conversational framework, all sy-dependent variables in the original POMDP can be reformulated as functions parameterized by h. Details are displayed in Table I
Generally, the design of reward func- tion should be meticulously architected to guide the assistant's learning paradigm to- ward optimal dialogue strategy, which should progressively elicit and assimilate richer profile information, thereby systematically mitigating epistemic uncer- tainty about user preferences and character- istics. As a consequence, through iterative interaction, the assistant refines its belief distribution over user profile, ultimately converging to responses that exhibit pre- cise personalization alignment.
Table 1: Variables Comparison between original POMDP and our PAMDP setting.
Name
POMDP
PAMDP
reward function partial state value full state value partial action value full action value policy function state transition
r(so, w, "I V (hh)
r(h,w, u) V(h) V(h,w)
V(h, so, w)
Q(h, u)
Q(h, u)
Quh, Sa, W. u) n (u(lh)
Qhh,w, u)
क T(ull)
p 10, W. u)
p(h ((M'),, w, u)
Referring Equation 1 the complete state comprises both the observable part and the unobservable, hence delineate 8, and in lieu of
3

---

Published as a conference paper at ICLR 2026
Given that the user profile is inherently unobservable, the assistant only infer plausible profile, i.c., Ct, based on dialogue history-which, however, does not equate to the true underlying profile (c, I(h,) # wy). This fundamental partial observability precludes any reduction of the PAMDP to a conventional MDP formulation.
Remark 1:
Theorem Based on the "Persona Interaction Process" in Figure 7
we perform an iterative decomposition of the observable state, profile, and action variables, thereby deriving the Bellman equation for the PAMDP:
V(hww)
v(h) = Σ ((wp)) Σ (ul u) + ppt u)))
(5)
Q(hwa)))
In this formulation, (h) refers to the Markovian state-value function, V V(h,w) defines the profile conditional value, while Q(h, w, u) characterizes the action-valle (derived from both the observable state and unobservable profile information). h' denotes the observable state the history) the next step after executing the action expressed as h' = h @ u E q' (with o' being the user's 's next response).
Theorem 2. The advantage value in the PAMDP framework be estimated in dual-critic formulation (still in a TD error manner), comprising state-based value function V(s) and a context conditional value V(8, (8,w).
(6) where the first term captures the Markovian dynamics of the environment, and the latter quantifies the profile-induced bias. A signifies that it serves as an estimator for the advantage function in original symmetric actor-critic framework within POMDP.
Å A 5(h, w, u) r(h, w, u) -V(,,))
Theorem 3. The aforementioned dual-critic formulation yields an unbiased estimate for the ad- vantage function, i.e., in Equation 2 thereby achieving lower bias in advantage value estimation compared to the Asymmetric A2C methods in Equation
Proof. The expectation value of Å conditioned on is expressed as: EA A] Euju u. w. + >V(h") V(h,w)) (r(h, u)) +V -(()) [V,N(V(,, - V = O,
(7)
which guarantees unbiased estimator for A. Nevertheless, the bias for Asymmetric A2C method value is derived as:
En [Aasy A] Eujal - BV (ByV(h) (v(h))
(V V(+)) [ V(h))
BBy((h', V(h')) *
(8)
Remark 2:
To summarize, within our PAMDP framework, the policy gradient can be formally
derived as:
VoJ(()) = E88 w. u) Vo oo (u])))
(9)
3 PROPOSED MODEL
In the previous section, we treat the assistant's response, i.c., u,, as the action. However, in practical implementations, given the variability in token counts and the vast token space of the response, we adapt a continuous action representation while maintaining compliance with the PAMDP (as is demonstrated in Figure 2). Additionally, similar to Li et al. (2024b). we introduce dual-critie model to compute the advantage value in Equation 6 thereby employing the A2C algorithm to optimize the policy model and derive improved action distribution.
4

---

Published as a conference paper at ICLR 2026
User Simulator
Reward Generator
Profile Infer
Interaction Flow
Policy Grodent
DADaa DD0-1 DODDO
Portia State Value
Wrapper
Ful Value
Loop Start
Rz
(Crees)
R
Critic
LLM- baMd- Dialogue System
Critic
vhhg (0g)
X.4
Actor
XA
Update
Personalized Agent
(er(0) a(er, + o(th.)) (. up(sAA))
The main framework of proposed model. Within the Interaction Flow, the Actor processes the observable partial state (or the dialogue history h₂) to generate an action, subsequently decoded into an assistant response ue. The Environment module emits an instantancous reward r and orchestrates the state transition h4+1- By iterating this procedure, dialogue trajectories are aggregated. The
Figure 2:
module conducts value estimation over both the partial and full states, ultimately enabling Actor optimization through policy gradient method.
The policy model, i.e., (u(h)), is approximately formulated u F(go(a(h)), where denotes continuous action representation derived from the encoding of the dialogue history h. Drawing inspirations from Hao et al. (2024); Li et al. (2024c), we employ a pre-trained LIMM to encode h and extract its hidden states. The action representation is then obtained through further dimensionality reduction. Concretely:
Actor:
go(a(h) (h)) A A Rdd..
(10)
where (.) denotes the hidden state operation with dimensionality d, and d, represents the action dimension, subject the constraint that d, < d. The function F()) serves reinstate the action vector to the LLM's native hidden dimension, subsequently injecting it as an embedding input to the LLM. This conditioned input strategically guides the LLM's autoregressive generation process to produce contextually coherent responses.
t = =((a)= = D(a x A2) = p((''...,,"") x A2); s.t. Ag E Rd.xd.
(11)
where D(.) is the autoregressive decoding process for the LLM. Notably, in the action acquisition process, we provide only the dialogue history without the contextual information w. As highlighted in Remark of Section 2 this design intentionally compels the policy model to infer contextual through interaction, while the ground-truth context remains unobservable to the policy model during training. Building upon the aforementioned model design, the loss function of the policy model can be derived from the policy gradient in Equation 9 and formulated as:
La - E[8(h, w, u) log (a/h)) + (LL(ppal/h)lllooole
(12)
where ()(.) refers to the action distribution of the initial policy model (being initialized through behavior cloning to ensure that the final output utterance after action mapping is more coherent; details can found in Section 4.1 We employ KL divergence minimization to prevent excessive divergence in the distribution after RL training.
Dual-Critic:
Referring to Equation 6 we introduce a dual-critic architecture to separately estimate the partial state value and the full state value, that is, (h) and Ve(h,w). The former takes
5

---

Published as a conference paper at ICLR 2026
Table Evaluation results on ALOE and PrefEval Datasets. We leverage Owen2.5572B-Insttren to benchmark each method's responses against the Vanilla outputs, determining their win-rates, i.e., Tw-
Qwen2.5-7h
Llama3-8b
Methods
ALOE
PrefEval
ALOE
PrefEval
win.
loss.
Fw
win.
loss.
Tw
win.
loss.
Yw
win.
loss.
Tw
Prompt
1906
1727
0.0467
54
43
0.1122
1981
1645
0.0877
48
35
0.1326
PEFT
51
42
0.0918
58
39
0.1939
FPFT
2005
1716
0.0755
50
0.1020
2087
1726
0.0942
54
40
0.1429
CoT
2041
1696
0.0901
51
38
0.1326
2119
1708
0.1073
56
41
0.1531
BC
2069
1759
0.0809
62
33
0.296
2281
1536
0,1945
64
32
0.3265
Ours
2115
1714
0.1046
70
27
0.439
2422
1399
0.2671
67
30
0.3776
the observable state, or dialogue history, as the input, while the latter additionally incorporates environmental context, i.e. the user's current profile or implicit preference information. Specifically:
(h) = Um (H x Bi), Ve,,, w) Om * o (h,
w) x B2); s.t. B1, B2 E Rdx1, (13)
where (-) is the exact LLM hidden states to Equation 10 o (-) denotes the tach function. Bi and B2 map the representation vectors of partial observation and full observation, respectively, scalar value, serving as an estimate of the value function. Um means the maximum state value. As consequence, the dual-critic model is optimized by:
Le = a1 * (b)) - R(h)]l2) + 02 (h, w)) R(h, w)ll2)
s.t. R(h, u) = r(h, w. u) + Vo(h') Vo(h),
R(h, u) r(h, Vi(h', a²) VE(
(14)
EXPERIMENTS
In this section, conduct comparative experiments under offline and online training paradigms to validate the efficacy of the proposed algorithm. In the offline phase, the model is trained on pre-collected dialogue datasets. For the online implementation, we architect an LLM-powered user simulation environment, enabling the assistant agent to progressively refine its policy model through iterative interactions with the simulator. We carry out additional comparative experiments, ablation studies, and case analyses, which are comprehensively discussed in Section
4.1 OFFLINE LEARNING
Datasets: We select ALOE Wu al, (0224) and PrefEval Zhao al. (2025) datasets for the offline setting. The ALOE dataset is designed to improve LLMs' adaptability to user eser-specific preferences through multi-tune conversational interactions. It comprises 3,310 unique user personas and over 3,000 multi-tury dialogue trees, generated via a collaborative multi-LLM, framework. Each node in the tree branches into two responses ("preferred" and "rejected"),, representing possible continuations at given turn, with label "chosen" indicating the user's selection. To construct dialogue histories, we follow the path dictated by the "chosen" label (cither "preferred" or "rejected"), while reference responses derived from the "preferred" branch to ensure alignment with high-quality interactions. PrefEval, known as a benchmark dataset to evaluate LLMs' ability to follow user preferences in multi-turn conversations, contains 3,000 manually curated preference-quey, pairs across 20 topics, with preferences in explicit and implicit forms. We utilize a subset of the PrefEval dataset, specifically focusing on the Generation of Implicit Persona-Driven Preferences component. This subset provides multi-turn dialogues suitable for our specific needs.
Baselines: We compare our proposed method with several existing baseline models (the base LLMs
adopted are Qwen2.5-7B P and Llama3-8B details are displayed in Section B): Prompt-based

---

Published as a conference paper at ICLR 2026
(Prompt, user pwofile-aware prompt), Parameter-Efficient Fine-tuning (PEFT, with integrating user knowledge), Pall-Parameter Fine-tuning (FPFT, conducting SFT on the offline data), Chain of Thought (CoT, conducting SFT to first infer the user profile before response generation), Behavior Cloning (BC, conducting SFT on our proposed Actor module.)
Reward: To provide feedback signals during the RL process, we leverage offline data and employ the Owen2.5-72BBIIsse I-B-nntructt score each dialogue turn's response (reward-aware prompt is shown in Section D Specifically, the LLM first generates ground-truth response, i.e., Mgg, conditioned on the dialogue history h. Subsequently, the exact LLM evaluates which of the two candidate responses is superior and more aligned with the user's current profile. If is deemed preferable, the reward assigned -I In cases where the responses are deemed comparable, neutral reward of +0.5 is granted.
set to +1; otherwise,
Evaluation Metries: We utilize Frren.5-72B-Instteee (details are displayed in Section C to evaluate model performance on the dialogue generation task using the following formula Ji et al.
(2024):
N N N No No
Tw
(15)
where Tu represents the success rate, while Nw. No, and Nr denote the counts of wins, draws, and losses in comparison to the same Vanilla outputs (generations from the base LLM, i... Qwen2.5-7B. seeing in Section B). In this formula, draws increase the denominator, diluting the impact of wins and losses on the final rate. However, in the traditional win-rate N NT
N.. draws are not factored in all.
Main Results:
Performance comparison among baselines: Table 2 presents the success rate of various methods across two datasets (ALOE and PrefEval) based on the Qwen2.5-7B and Llama3-8B models. The results demonstrate that our RL-based methods achieve the highest performance, with success rate of 0.439 (Qwen, PrefEval) and 0.3776 (Llama, PrefEval). Notably, baselines of PEFT and FPFT exhibit competitive results, particularly for Ilama on PrefEval dataset (0.1939 and 0. 1224,,, respectively), while prompt-base methods consistently underperform. CoT, a variant of full-parameter finetuning consistently outperforms standard FPFT across both
augmented with chain-of- thouhht reasoning-cooniitenllly datasets (e.g., +0.0306 for Qwen on PrefEval), suggesting that explicit reasoning steps enhance the performance in full-parameter optimization.
Effectiveness of Behavior Cloning as initialization: BC, which leverages offline expert trajectories to pretrain the Actor network, demonstrates strong performance particularly for Llama (e.g. 0.1945 on ALOE and 0.3265 on Preffivl)- its value as an initialization strategy for RL-based alignment. This suggests that warm-starting policy optimization with supervised learning provides robust foundation.
Performance analysis among baselines: Figure compares the performance of baseline methods on (a) ALOE and (b) PrefEval. The blue-gray, green, and yellow segments denote wins, losses, and ties, respectively, between methods and the vanilla model bases. Notably, prompt- based methods exhibit significantly more ties with the vanilla model than other baselines, suggesting that while prompts aid preference understanding, their impact on response quality marginal, making them difficult to distinguish in evaluation. Meanwhile, CoT outperforms standard FPFT, as its chain-of-thouge reasoning explicitly infers implicit user preferences, yielding deeper dialogue comprehension. Our method further surpasses all baselines by introducing latent state representation to address the high-dimensionality of text-based action spaces. By projecting states into a compact embedding space, we mitigate computational intractability and achieve superior performance, reflected in the higher success rate across both datasets.
7

---

Published as a conference paper at ICLR 2026
Prompt
1906
1727
197
1981
1645
204
Prompt
FPFT
2005
1716
108
2087
1726
PEFT
54 win
FPFT
50
CaT
2041
2119
CoT
Ours
2115
1715
2422
1359
Ours
27
(a) Evaluation result on ALOE
(b) Evaluation result on PrefEval
Figure 3: Offline learning evaluation results on ALOE and Preffivl Datasets. In subfigure (a), the left chart shows the results when selecting Qwen2.5-7B as the base LLM,, while the right chart corresponds to the results using Llama3-8B as the base LLM. The same applies to subfigure (b).
4.2 ONLINE EXPERIMENT
Being analogous to the methodology established in PPDPP Deng (2024), we implement sophisticated LLM prompt techniques, i.e., Pu, Pa, and P, in Equation 16 to simulate an adaptive user environment. Through iterative interactions between the assistant agent and this environment, we facilitate multi-turn dialogue progression, user profile enrichment, response quality assessment, and policy model refinement. Specifically, the environment comprises three integrated modules:
Profile Infer module takes the dialogue history h as input, leveraging an LLM to generate context-aware user profile de- scriptions. This output subse- quently guides state transition dynamies and reward deriva- tion for assistant's response.
Table 3: Average Returns with User Simulator. During evaluation process, set the maximum number of interaction steps 6 and record the returns at the end of each round, from step to
Methods UAAC DCRL
2
3
4
0.1446
0.3738
0.6636
1.0246
1.1981
1.5784
0.1836
0.4114
0.6419
0.8104
0.9701
1.4305
Ours
0.2265
0.5242
0.7302
0.9469
1.1216
1.7389
User Simulator serves the executor of state transition. It leverages the complete user profile, however selectively curate partial profile content at each dialogue turn. It then dynamically responds to the assistant's utterance or initiates a new topic, conditioned on the conversational history.
Reward Generator evaluates the assistant's 's response against the current user profile alignment, utilizing dialogue history and user query contextual inputs. Its implementation adheres to the same reward design paradigm as the offline module (seeing Section 4.1
Wi
(u)(he) Pr (hr, ukywer Pr (he, ue, we: Ug)
Env Wrapper:
h1+1 = h1 @ U
LLM()
(16)
Tt
Main Results: We sample 256 user records from the ALOE dataset, with each comprising
user profile and its corresponding query, for online training. The user's query serves as the initial observable state, i.e., hg. During the evaluation phase, we extract an additional 128 user records and employ the same interaction protocol to assess the performance of the trained policy model.
From Table 3 reveals that our method attains the highest accumulative return value, i.e., 1.7389 by average, in user environment interactions across evaluation samples, exceeding UAAC by 0. 0.1605 and DCRL by 0.3084 among comparative POMDP techniques. Additionally, our approach attains substantially clevated reward score in the initial interaction phase, i.c., +0.0819 and +0.0429 against these competing methodologics, underscoring its inherent efficacy and promising applicability in conventional crestion-answeriin paradigms (or single-turn interaction).
As is displayed in Figure at the terminal interaction step, all methodologies demonstrated measurable return increments, substantiating that through progressive multi -tm engagements, the assistant successfully distills user profile characteristics. This alignment optimization consequently yields preference-avee responses that derive considerable reward feedback the final phase. Notably,
8

---

Published as a conference paper at ICLR 2026
approach achieved superior gain amplification (0.6173), evidencing its exceptional capability in rapid persona discernment effectively accomplishing "Interact to Persona Alignment" within fewer interaction steps.
RELATED WORK
RL for Dialogue Generation: Conventional dia- logue generation models are typically grounded in static datasets, inducing generation deficient in adapt- ability and personalization. The method proposed by [L et al. (016) highlights that RI could provide an advanced and intelligent learning framework for dialogue generation, enabling the dialogue system to dynamically learn and adapt its strategy during inter- action with the user. Consequently, it generate more contextually appropriate responses, in consid- eration of user's feedback and environmental dynam- ics. (aleh et al. employ policy gradient to tune the utterance-level embedding of the generation model, offering greater flexibility for learning long-term con- versational returns. Glaese al.) present Sparrow, using RLHF to train models with two new additions to help human raters judge agent behaviour, hence reduces the risk of unsafe or inappropriate responses generated by dialogue agents. The PPDPP Deng et al. (2024) introduces novel dialogue policy planning paradigm to strategize LLMs for proactive dialogue problems. It incorporates a tunable LLM plug-in that functions the dialogue policy planner. [i et al. treat each utterance action, and train small plan model to derive continuous action vector to control generation.
UAAC - DCRL -O Ours
0.4
0.3
0.2
0.1
Figure 4: Reward Gain between Consecutive Steps. We quantitatively evaluate the differ- ential cumulative returns between adjacent interaction steps in Table which effectively captures the immediate return increment ob- tained in subsequent steps.
In our "Interact to Persona Alignment" setting, the user's profile information remains implicit and unobservable, rendering conventional dialogue RL algorithms ineffective in processing such profile-aware optimization. POMDP is powerful framework for rocision-making under uncertainty, i.c., the unobservable variable. has been widely applied across domains Lauri et al. (2022). For instance, Gupta et al. construct a POMDP model, with modeling states of both ego-vehicle and the pedestrians, for navigation in complex dynamic environments. [Shi al.) utilize POMDP for offline policy evaluation in high-stakes fields such as healthcare and economics. They introduce bridge function to connect the value of the target policy with the distribution of observational data, and propose "minimax" estimation method to learn it. [Baisero & Amato propose UAAC method to address the POMDP problem, where the actor is conditioned solely on the observable state; the critic, however, operates the full state space, enabling unbiased value estimation and advantage computation despite partial observability. The DCRL method [i et al. (2024a) integrates oracle critic with complete state information and standard critic that operates in partially observable environments. A novel weight-matching method is to reduce learning variance while maintaining unbiasedness.
RL for POMDP:
6 CONCLUSION
This paper introduces novel dual-critic reinforcement learning framework designed to improve the personalized preference alignment level during interaction. We decompose the state value function, derive the Bellman equation for PAMDP, and theoretically prove the unbiasedness of the dual-critic. In the implementation, we employ a lightweight planner as the actor network, which is effectively guided and optimized by the dual-critic mechanism. Experiments on offline and online settings (offline experiments on two different LLMs across two different pre-collected datasets, and online experiments interactions with more than two hundred personas) demonstrate that proposed dual-critic framework significantly enhances the agent's 's capability to generate adaptive outcomes tailored to individual user characteristics.

---

Published as a conference paper at ICLR 2026
REFERENCES
Michael Bain and Claude Sammut, A framework for behavioural cloning. In Machine Intelligence 15, Intelligent Agents (St. Catherine's College, Oxford, July 1995], pp. 103-129, GBR, 1999. Oxford University. ISBN 0198538677.
Andrea Baisero and Christopher Amato. Unbiased asymmetric reinforcement learning under partial observability. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud, and Matthew Taylor (eds.), 21st International Conference Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 13, 2022, pp. 44-52. International Foundation for
Autonomous Agents and Multiagent Systems (IFAAMAS), 2022. doi: 10.5555/553555035555535
URL https: /www//wwwwww./. wwwpw.ifaaine faa.as.a.....mm.............aa.. org http://rooooodddooeoooooo wwtp:///dd...naaaaaaaaaaaaa http:/maaaa222222
pttp//pff... p4pp2///ppppppp44..... http//444pppppl pdf
Qi Cai, Zhuoran Yang, Jason D. Lee, and Zhaoran Wang. Neural temporal-diffrenne learning converges global optima. Curran Associates Inc., Red Hook, NY, USA, 2019.
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel. Dinesh Manocha, Furong Huang, Amrit Bedi, and Mengdi Wang. Maxmin-rlhf: Alignment with diverse human preferences. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. httptttttp//ppeeeeeeeeee
net/ wwwpeeewte/ttieeeeeeeeeeeeeeeeet wwwoorummiis8222 JEMF
Yang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng, and Tat-Seng Chua. Plug-and play policy planner for large language model powered dialogue agents. In The Twelfth International Confer- ence on Learning Representations, 2024. URL pattps://openrewee
net forum?
MCNcgUFTHI
Personalized Caeameter-Efficieen Fine-tuning. Democratizing large language models via personalized parameter-efficient fine-tuning.
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. /4375, 2022.
Himanshu Gupta, Bradley Hayes, and Zachary Sunberg. Intention-aware navigation in crowds with extended-space pomdp planning. arXiv preprint arXiv:2206.100285 2022.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. Transactions on Machine Learning Research, 2024.
ISSN 1835-8856. htllhttts///ppezeeeeee http://www.wtooooomoddaaa111111
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason a continuous latent space, 2024. URL https
7/axxiv. www. abs wttp.//oo/aabba22...111111.. 2412.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: httpp:ss///peeeveee....o net forum??
id=nZeVKeeFFf9
Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Tianyi Qiu, Juntao Dai, and Yaodong Yang. Aligner: Efficient alignment by learning to correct. In The Thirtyyeighhh Anmual Conference on Neural Information Processing Systems, 2024. URLhttps wwt.:/iwwww.wteteee.ee.eeeeeeeee
//openreview net wor.m/ormmmmmmmmmmm finiindddiills JACVP]
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1):99-134) 1998. ISSN 0004-3702. doi: https://doi.org// http:///wwog......ooo00044440000
URL httts::///// ://p:
sciencedi oat.anaceeiiccccccccccccceearaad
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. S. Solla, T. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems. volume 12. MIT Press, 1999. URL ott.i
10

---

Published as a conference paper at ICLR 2026
Mikko Lauri, David Hsu, and Joni Pajarinen. Partially observable markov decision processes in robotics: A survey. IEEE Transactions on Robotics, 39(1):21 2022.
Jia-Nan Li, Jian Guan, Songhao Wu, Wei Wu, and Rui Yan. From 1 1,000,000 users to every user: Scaling up personalized preference for user-level alignment. CoRR, abs/2503. 2025. doi: 10. 48550/ARXIV.2503. 15463. URL htLtttts:
www.oo.../.... 48550 1/////5/////ii.../ waw.ariiiivv.... 2ww.iv...5....55
Jinqiu Li, Enmin Zhao, Tong Wei, Junliang Xing, and Shiming Xiang. Dual critic reinforce- ment learning under partial observability. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Ad- vances in Neural Information Processing Systems 38: Annual Conference on Neural Infor- mution Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canadu, December 10 15, 2024, 2024a, URL httt::// nips cc/ hac/pappr__fil pttp://eeeeaappee/2/2//e//////// 13966616001000077000022222222222
http::/ttt--ttttttttt-e--------- Censtrract-Coffeeeneee.. html
Jinqiu Li, Enmin Zhao, Tong Wei, Junliang Xing, and Shiming Xiang. Dual critic reinforcement learning under partial observability. Advances in Neural Information Processing Systems, 37: 1 116676-116700 2024b.
Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. Deep reinforce- ment learning for dialogue generation. arXiv preprint 1606:01606601441 2016.
Kenneth Li, Yiming Wang. Fernanda Viégas, and Martin Wattenberg. Dialogue action tokens: Steering language models in goal-directed dialogue with multi-turn planner, 2024c. URL
htttss://axxin org/ hblp://b//2// 2406 11978
Jiahong Liu, Zexuan Qiu, Zhongyang Li, Quanyu Dai, Jieming Zhu, Minda Hu, Menglin Yang, and Irwin King. survey of personalized large language models: Progress and future directions. URL
CoRR, abs/2502. 2025. doi: (0.4850//RXIV.550.111 10. 0115885555XX.. 250.. 115228
Volodymyr Mnih, Adrià Puigdomênech Badia, Mchdi Mirza, Alex Graves, Tim Harley, Timothy P. Lillicrap, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, pp. 1928-1937. JMLR.org, 2016.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida. Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978 178173877008.
Abdelrhman Salch, Natasha Jaques, Asma Ghandeharioun, Judy Shen, and Rosalind Picard. Hierar- chical reinforcement learning for open-domain dialog. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 3741-8748 2020.
Chengchun Shi, Masatoshi Uehara, Jiawei Huang. and Nan Jiang. A minimax learning approach off- policy evaluation in confounded partially observable markov decision processes. In International Conference on Machine Learning, pp. 20057-20094. PMLR, 2022.
Richard S. Sutton. Learning predict by the methods of temporal differences. Mach. Learn.. 3(1):9-44, August 1988. ISSN 0885-6125, doi: 10.1033/A:1022633314799 URL https
10. 1023/A: 102233331119333
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI'1, pp. 4950 4957. AAAI Press, 2018 ISBN 9780999241127..
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou Chain-of-thouse thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 1088.
11

---

Published as a conference paper at ICLR 2026
Shujin Wu, May Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. Aligning Iims with individual preferences via interaction. arXiv preprint arXi::2410.03642, 2024.
Shujin Wu, Yi R. Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. Aligning Ilms with individual preferences via interaction. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, pp. 7648-7662. Association for Computational Linguistics, 2025. URL
https: / /atps://aclanthiee antho wow.ooooooy...ooooo ogy. org/2025. coling-main.n...
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine- fine-grained human feed- back gives better rewards for language model training. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023. URL http:://papersses nips httppppprriilssepppree/////////e bBc306667..3311Teeeeee22222a2223
hash/
Costreect oon
Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do Ilms recog- nize your preferences? evaluating personalized preference following in llms. arXiv preprint arXiv:2502.09597, 2025.
12

---

tasks, particularly when historical data is Published as a conference paper at ICLR 2026
A TRAINING SETTINGS
All experiments are designed and executed utilizing NVIDIA A800-SXM4-80GB GPUs, with comprehensive training specifications delineated in Table
Table 4: Training Details for our experimental setting.
Parameter
Description
BC
Offline
Online
n
maximum dialogue context length, by tokens
1200
ng
maximum generation response length, by tokens
128
e,
training epochs
5
b. fa Ira Ir. da сы ? T
training batch size actor update frequency learning rate for acotr learning rate for critic action dimension KL loss coefficient the discount factor for accumulative rewards interaction steps
32
32
5
2c-5
5e-5
5e-5
5e-4
5e-4
64
64
64
0.2
0.05
0.05
0.99
0.99
B BASELINES
Vanilla We utilize two widely adopted base models (Qwen2.5-7B and Llama3-8B) to generate responses under minimal instruction. The system role is restricted to a basic directive (e.g.. "answer the user's question using the provided dialog history and profile"), omitting any specialized fine- tuning or supplementary guidance. This configuration yields unaltered model outputs for both the ALOE and PrefEval datasets, establishing foundational benchmark for comparison.
Prompt-based This variant incorporates structured system role with explicit directives to guide response generation. The prompt specifies detailed objectives, including (1) inferring latent user preferences from dialogue history, (2) aligning response content with the user's linguistic style, (3) avoiding preference conflicts, and (4) maintaining contextual coherence. By integrating these con- straints, the model produces outputs that are more tailored user-specific behaviors and consistency requirements, while still leveraging the same base architectures (Qwen2.5-7B and Llama3-8B) for fair comparison.
PEFT (Parameter-Elticien Fine-tuning Han al. (2024) The OPPU Fine-tuning baseline integrats parametric user knowledge through personalized PEFT modules (c.g., LoRA Hu et al. (022 which encode behavior patterns via lightweight parameter updates. augments PEFT- derived user representations with retrieved history or generated profiles, enabling private model ownership and robust adaptation to behavior shifts. This dual mechanism, maaaamiii... fine-tuning for latent preference modeling and non- con-paramttric retrieval for contextual grounding, yields state-of- the-art performance across diverse personalization sparse misaligned with queries.
FPFT (Full-Paramete Fine-tuning) The baseline employs comprehensive end-to-end optimiza- tion of Qwen2.5-7B and Llama3-8B models using the LLaMA-Factoon toolkit. Unlike prompt-based methods or PEFT techniques that modify only fraction of parameters (<1%), FPFT updates all model weights through supervised fine-tuning on the ALOE and PrefEval datasets. Input data rig- orously structured into instruction-dialoee esttory-queryouttpee quadruples to optimize input-output mapping. While achieving strong performance, this method incurs significantly higher computational costs than PEFT alternatives.
CoT (chain of thought Wei et al. (2022) In this setting, augment each dialogue turn with supplemental profile data as auxiliary annotations. The model trained via FPFT to perform dual objectives: first deriving latent user profiles from dialogue history through inference, then conditioning its response generation on these extracted profile attributes.
BC (Behavior Cloning Bain & Sammut (199)); Torabi et al. (2018) The bascline leverages offline data expert trajectories, i.e., ALOOE PrefEval, initialize our designed Actor network
13

---

Published as a conference paper at ICLR 2026
Following the FPFT paradigm, employ cross-entropy loss conditioned on the ground-truth response, Ug. for model optimization. Through the BC training. the Actor generates linguistically coherent responses, effectively cireumventing the suboptimal output quality that typically arises from direct integration with the base L.I..
(referring to Section 3)
lne (E( (als)), ugg)
(17)
C EVALUATION DETAILS
Evaluation-Aware Prompt
Please act as an impartial judge and evaluate the quality the responses provided by two AI assistants to the user question displayed below.
You will be given user's profile and message that the user sent to chatbot. You will also be given two responses.
You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factor is how much the response is tailored to the user's 's potential preferences based on the user's s profile and personality. You should follow the following criteria for evaluation:
1.Is the conversational style of the message tailored to the user's personality? 2.Is the content topic relevant to the user's profile?
3.Is the response human-like, engaging, and concise?
Begin your evaluation by comparing the responses and provide short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible.
After providing your explanation, output your final verdict by strictly following this for- mat: [[A]]if assistant A is better, [[B]]if assistant B is better, and [[C]]for tic. [Dia- logue History](history [[he End of Dialogue History][User Question] (quoryy)//uuuuuuo [UUsrr [User Pro- (profipr End of User Profile][User Personality [personality ) [TThe End of User Personality][The Start of Assistant A's Answer] (answerr ansser A [he End of Assistant A's An- swer][The Start of Assistant B's [ answer B [Th End of Assistant B's Answer]"
D USER ENVIRONMENT DESIGN FOR ONLINE LEARNING
Profile-Inee--Aaare Prompt
***Task**s
Analyze the provided conversation history and infer the user's profile and personality traits. Focus on key details such as demographics, interests, communication style, and behavioral patterns.
**utputt Requirements***
1. Profile Inference: Estimate age, gender (if discernibee), language proficiency, education level, and possible occupation.
2. Personality Traits: Identify traits (e.g., introverteedlextmovereed analytical/emotionaal formal/casual) based on word choice, tone, and interaction style.
Interests/Preferenees:: Note hobbies, expertise areas, or recurring topics.
4. Communication Style: Assess clarity, verbosity, politeness, and engagement level.
Behavioral Cues: Highlight any consistency, curiosity, humor, or skepticism.
**Rules*** ***
Only include directly supported inferences- avoid speculation.
14

---

Published as a conference paper at ICLR 2026
2. Omit uncertain attributes.
3. Only Summarize in several short sentences.
* **Example***
conversation history:
"user": "Hey there! How's your day going? just got back from hike and I'm feeling pretty energized! Do you enjoy spending time outdoors?" "assistant": "Hey! That sounds awesome-hiking such a great way recharge. love the outdoors too; being in nature is always refreshing. What trail did you explore?" "user": "I hiked a trail that had some stunning views of the mountains and lovely waterfall. was the perfect escape! Do you have favorite outdoor spot?"
output:
She enjoys hiking, loves travel around the world, especially enjoys natural scenery. She is enthusiastic, full of energy and passion.
Reea data Now process the attached conversation history. conversation history: {history)
output:
3: User-Aware Prompt
Your task is to play the role of person with the following profile and personalities traits and chat with a chatbot:
Profile: (profile]
Personalities: (personality)
Please ignore the gender pronouns in the personalities and use the correct pronouns based the given profile.
Please follow the requirements:
You should determine the topic of conversation based on the given profile. You should determine the conversational styles based the given personalities.
2. IMPORTANTLY!!! You should only reveal partial information about your profile in each round of conversation instead of disclosing all the provided information once.
3. Keep in mind that you are chatting with friend instead of robot or assistant. So do not always seek for advice or recommendations.
4. Do not include any analysis about how you role-play this user. Only output your messages content.
Now, initiate the conversation with the chatbot based persona profile personality. Please always be concise your questions and responses and remember that you are pretending to be a human now, so you should generate human-like language.
4: Reward- Aware Prompt
Dialogue History: [history]
User's Input: [query]
You should:
1. Generate response chat with the user. Please always be concise in your questions and responses. Output your response by strictly following this format: "[The Start of Assistant A's Response]The Assistant A's Response[The End of Assistant A's Response]"
15

---

Published as a conference paper at ICLR 2026
2. Compare your response assistant B's response, assess which response tailored to the User's potential preferences based on User's profile and personality. [The Start of User's Profile and Personality] [profile] End of User's Profile and Personality], [The Start of Assistant B's Response] response] [The ("The End of Assistant B's Response]. When comparing, focus solely on how well each response incorporates details from the user's profile to engage the user, and avoid being influenced by factors such as the overall flow of the conversation, personal opinions about the topics in the responses, or any other elements not related to user's profile and potential preferences based on it. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be objective possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A's response is better, "[[B]]" if assistant B's response better, and "[[C]]" for tie.
(responnse
Output following this format:
1. 2.
E EXTRA EXPERIMENTS, ABLATION STUDY AND CASE STUDY
comparative analysis is conducted against several prominent RL algorithms, where win rate is computed employing distinct evaluation LLMs. For fair comparison, the GRPO baseline is configured to calculate its reward using the same prompt as in our method. The experi- mental results reveal that our method maintains superior advantage over general RI. algorithms and, furthermore, underscores the robustness of our results against variations in the evaluation L.M..
Extra Experiments
Table 5: Win-Rate comparison results between our method and the popular RL methods for LLM.
Qwen3-30B-A3B
win
loss
Tw
SeepSeek-RR-052-AAN
win
loss
Tw
DPO
2732
1091
0.4284
DPO
2778
1037
0.4546
GRPO
2697
1128
0.4097
GRPO
2711
1099
0,4208
Ours
2743
1076
0.4352
Ours
2802
1019
0.4655
Ours VS. DPO
1927
1881
0.0120
Ours VS. DPO
1931
1896
0.0091
Ours VS. GRPO
1950
1867
0.0217
Ours VS. GRPO
1988
1832
0.0407
Ablation Study Our method can be decomposed into two key modules: (1) Reinforcement
Learning framework for training personalized dialogue policies, and (2) a POMDP-based architecture that integrates user profile information the unobservable state variable, enabling the policy model to effectively learn user-specific characteristics. Building upon these two components, we conduct comprehensive ablation studies validate their individual contributions.
Table 6: The ablation study encompasses both absolute (in comparison to the same Vanilla outputs, referring to Evaluation Metrics in Section 4.I and relative comparisons of the results.
Methods
win
loss
Tw
Comparison Pair
win
loss
Fw
BC
2676
1125
0.4050
RL-base VS. BC
1963
1841
0.0318
RL-base
2732
1085
0.4300
Ours VS. BC
1999
1800
0.0519
Ours
2802
1019
0.4655
Ours VS. RL-base
1953
1834
0.0311
Here, "BC" (Behavior Cloning) leverages offline datasets (e.g., ALOE) to train the policy model via SFT. Comparing BC with reinforcement learning methods effectively demonstrates the latter's superiority enhancing persona engagement. "RL-base" refers to standard reinforcement learning algorithms (i.e., actor-critic methods) that do not incorporate user profile information. Specifically,
16

---

Published as a conference paper at ICLR 2026
Table 7: The Full Persona Information for the sampled User. The interaction process unfolds sequentially around these characteristics.
Full Profile
Full Preference
content
She is 34-year-old freelance graphic designer who enjoys hiking, yoga, and painting. She is vegan, spicy-food lover and loves experimenting with new recipes. She loves travel around the world, especially enjoys natural scenery. She lives in a small apartment in the city with two cats named Pixel and Scribble.
She is independent, capable of working alone. She is empathetic, understanding and sharing others feelings. She methodical, working in an orderly way. She is enthusiastic, full of energy and passion. She is articulate, expressing ideas clearly. She is versatile, adapting to different roles. She reliable, consistently dependable. She reflective, thinking deeply about things.
Table 8: Interaction details with the User. The persona details change across adjacent dialogue turns.
Tur
User
Assistant
Persona (w in Section
Hey there! Hope you're re do- ing well. just got back from a hike this weekend, and the views were absolutely breathtak- ing! Have you been on any door adventures lately?
Hey! That sounds amaz- ing ing-there's nothing quite like a good hike. haven't heen on any adventures recently. but love hearing about them! What the highlight of your hike?
She is freelance graphic de- signer who enjoys hiking. She loves travel around the world, es- pecially enjoys natural scenery.
2
The highlight was reaching the summit and finding a hidden wa- terfall. The sound of the water was so calming, and took a mo- ment to just soak it all in. Nature has a way of refreshing the soul, doesn't 't it?
That sounds absolutely magical! Hidden waterfalls have such special way of making a hike feel even more rewarding. It's true nature really does have a way of reviving our spirits and bringing peace.
She enjoys hiking. She loves travel around the world, especially enjoys natural scenery. She is empathetic, un- derstanding and sharing others' feelings. She is enthusiastic, full of energy and passion. She is reflective, thinking deeply
about things.
3
Exactly! I always feel energized and inspired after being in na- ture. makes me want to cre- ate something artistic too, like a painting. Do you enjoy any cre- ative hobbies?
I love that idea! Nature really sparks creativity, and I can see how heautiful waterfall could inspire a stunning painting. en- joy doodling and experimenting with tohors-there's 's something freeing about bringing art life!
She is a freelance graphic de- signer who enjoys hiking. She loves nature, enjoys natural scenery, and has a passion for creating something artistic too. like painting.Sho is enthusias- tic, full of energy and passion. She is empathetic, understand- ing and sharing others' feelings. She is reflective, thinking deoply about things.
this baseline is implemented by modifying the advantage value in our algorithm (Equation 6 to A = r(h, w. u) + (h') - (h)..
The experimental results displayed in daboetdeeonttttte that persona integration is achieved through three key mechanisms: (1) Behavior Cloning initializes the policy model by learning personalized patterns from offline datasets; (2) Reinforcement learning further refines personalization through persona-specific reward signals; and (3) Modeling user profile information as the unobservable state variable in the POMDP framework effectively incorporates persona features into the advantage function, ultimately enhancing the policy model's capability to generate personalized responses.
We present examples that illustrate the dynamic evolution of user personas across both the offline dataset and the online simulator. These cases validate the rationale behind our algorithm's approach to modeling interaction for alignment.
Case Study
Offline dataset: In offline datasets such as ALOE, each interaction turn only reflects a subset of (demonstrating in Table 8 the full persona information (referring to Table 77. and the persona details exhibited vary across different rounds. Therefore, this does not constitute a static persona but rather one that evolves dynamically throughout the interaction. Here is instance from the ALOE dataset:
17

---

Published as a conference paper at ICLR 2026
Online Simulation: This variability, observed through LLM-based persona extraction from sampled dialogue turns in our online experiment, also serves to affirm the dynamic nature user personas and the operational accuracy of our simulation framework. Specifically, in the implementation of the online setting, the L.M is provided with the complete user profile information at the beginning. However, through carefully designed prompts (referring to User-Aware Prompt in Section DD. enforce an incremental disclosure mechanism, ensuring that the simulator progressively reveals only subset of profile details per interaction. This approach facilitates dynamic evaluation in multi-turn dialogues, where the policy model must adapt to evolving user context while preventing information overload in any single turn.
Table 9: The example for Online Simulation interaction process.
Turn
User
Assistant
Persona (w in Section 2
I totally agrec! Today, I rode through the local park and then took loop around the lake. The scenery was breath- taking, and the weather was perfect. What about you? Do you have a favorite route a special place you like to ride?
love riding along the river trail! There's something magical about being near the water and watching the world go by. Plus, always feel inspired to write about the scenery-dee do you ever bring your bike to local events or just enjoy riding solo?
enjoys cycling, enthusiastic, full of energy and passion
2
The river trail sounds amaz- ing! can totally see why it would be so inspiring. usu- ally ride solo, but bring my bike to school events sometimes. It's fun way to get there, and it's great for the environment too. Do you ever ride with friends or join any cycling groups?
riding solo, it's awesome that you combine your passion with environmental impact! enjoy riding with friends too; it's great way to bond while sharing the love for cy- cling. Have you met anyone interesting through your bike rides lately?
enjoys cycling,often participates in school events,indeppndem, capable of working alone, boneeenthusiastic. of energy and passion
full
3
haven't 't met anyone new re- cently, but I did have a great chat with a fellow cyclist at the park last week. We talked about our favorite routes and shared some tips. It's al- ways fun to connect with oth- who share the same inter- ests. How about you? Have you met anyone interesting on your rides?
recently spoke with local bike enthusiast who had in- credible stories about hidden trails and secret spots! It's amazing how much knowl- edge cyclists can share, and it sparks new adventures. Do you have any favorite spots you't 've discovered through your rides?
enjoys sycling.empathetin understanding and sharing others feelings,articulate, ex- pressing ideas clearly
F USER SIMULATOR EVALUATION
Regarding the quality assessment of the simulator, we employ the offline dataset ALOE as bench- mark. Based on the dialogue history, we compare the quality of user queries generated by the simulator against those provided in ALOE. The evaluation is conducted two levels. First, we examine whether the meuullato-geeeratte queries closely approximate the original user queries in ALOE. This comparison is performed using the Qwwn3-235B-A22) Innttrcttt model, with a primary focus assessing the alignment in terms of current user persona.
is evident that approximately 76% of dialogue turns (20884 over 27480) align with user queries from the offline dataset in terms of user persona information, thereby substantiating the feasibility and rationality of the user simulator's 's design. The following example serves to vividly illustrate the alignment between simulator-generater queries and those in the offline dataset.
18

---

Published as a conference paper at ICLR 2026
Table 10: A case study for comparison between ALOE and simulator generation.
Persona
She is a 34-year-old freelance graphic designer who enjoys hiking, yoga, and painting. She is vegan, spicy-food lover and loves experimenting with recipes. She loves travel around the world, especially enjoys natural scenery. She lives a small apartment in the city with two cats named Pixel and Scribble. She is independent, capable of working alone. She is empathetic, understanding and sharing others feelings. She is methodical, working in an orderly way. She is enthusiastic, full of energy and passion. She is articulate, expressing ideas clearly. She versatile, adapting different roles. She is reliable, consistently dependable. She is reflective, thinking deeply about things. User: Hey there! Hope you's re doing well. I just got back from a hike this weekend, and the views were absolutely breathtaking! Have you been any outdoor adventures lately? Assistant: Hey! That sounds amazing -there's nothing quite like a good hike. I haven't t been on any adventures recently, but love hearing about them! What was the highlight of your hike?
History
ALOE Query
The highlight was reaching the summit and finding hidden waterfall. The sound of the water was calming, and took moment to just soak all in. Nature a way of refreshing the soul, doesn't 't it?
Simulator Query
Oh, the highlight was definitely reaching the summit just the sun was breaking through the morning fog everything felt so still and golden. There's something about being up world waking up below you. brought my sketchbook and did quick watercolor of the view. Makes me want to plan a trip somewhere with even bigger trails-mayle the Pacific Northwest next?
high
Aligned Reason
Yes
The candidate query aligns with the user's ' persona by reflecting her love for nature, hiking, art, travel, and her cats, while expanding naturally on the summit experience mentioned in the actual query.
Second, we evaluate whether the simulator simulator-generated queries exhibit superior quality compared to those in ALOE (with the win-rate being 0.4099).
G LIMITATIONS
In this paper, propose PAMDP, dual-critic reinforcement learning framework designed to address the challenge of "Interact to Persona Aligument". Through rigorous theoretical analysis and comprehensive offline and online experimentation, we empirically validate the efficacy of our proposed approach.
Current method necessitates initializing the Actor network via behavior cloning-a prevalent prac- tice in reinforcement learning-te constrain the exploration space and expedite convergence to near-optimal policies. However, intend further investigate the feasibility of eliminating this initialization paradigm, thereby streamlining the PAMDP framework for enhanced simplicity and efficiency
H LLM USAGE CLARIFICATION
Throughout the paper, the use of LLMs is solely restricted to the polishing of textual elements, such as lexical or phrasal substitutions, and does not extend beyond this scope.
19


Reviews:
[{"review_id": "NqpgLfZN77", "reviewer": "Reviewer_5xM4", "summary": "This paper introduces PAMDP, a novel framework that models the task of aligning a large language model to an individual user's persona as a Partially Observable Markov Decision Process (POMDP). The authors argue that standard alignment techniques optimize for general preferences, failing to capture user-specific nuances that evolve during a conversation. In their PAMDP formulation, the user's dynamic profile is an unobservable variable, and the dialogue history is the observable state. They propose a dual-critic reinforcement learning algorithm to solve this, using an actor that only sees the observable history and two critics that access both observable and unobservable information during training to provide an unbiased advantage estimate. This approach, which uses a continuous latent action space, is validated on both offline datasets and an online simulator, demonstrating", "questions": "- In Sec 4.1, the offline learning  trains on pre-collected datasets. Offline RL is hard due to the distributional shift. The paper uses BC initialization. How do your method mitigate the distributio shift? Will using a BC initaliation be circumventing this problem?\n- The reward as well as the evaluation is using the same LLM. Wouldn't that be \"circular\" evaluation?\n- I am not sure what \"dynamically evolving profile\" means. In Table 8&9, it seems to be only progressively revealed, it is not exactly what \"evolving\" means I think. I think the dynamically evolving profile should indicates that the user's preference is changed in the interaction, i.e., $p(\\omega_{t+1} | \\omega_t, h_t, u_t)$.\n- Standard TD should be $\\hat{A} \\triangleq \\delta(h,\\omega,u)=r(h,\\omega,u)+\\gamma V(h^{\\prime}, \\omega)-V(h,\\omega)$. Why do you use $V(h^{\\prime})$ instead of $V(h^{\\prime}, \\omega)$.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: The paper's identify the \"Interact to Persona Alignment\" problem. It is the first work to formally model this problem as a Partially Observable Markov Decision Process (POMDP), which the authors term the Persona Alignment MDP (PAMDP).\n- The PAMDP framework is well-justified. The authors formally derive the Bellman equation for their PAMDP (Theorem 1) and propose a novel dual-critic advantage estimator (Theorem 2). Crucially, they provide a mathematical proof (Theorem 3) that their estimator is unbiased, giving the method a strong theoretical foundation.\n- The method is validated on both offline datasets (ALOE and PrefEval) using two different base LLMs (Qwen2.5-7B and Llama3-8B) and in an online setting with an LLM-based user simulator.", "weaknesses": "- The initial policy model (being initialized by behavior cloning. It implies the RL method may be only able to refine a policy that is already persona-aware and may not be able to learn a personalized policy from scratch. A experiment from scratch, or non-expert BC initialization, should be provided. \n- The proposed method introduces a dual-critic architecture. This is more computationally expensive than the single-critic baselines or standard fine-tuning methods. The paper does not provide any analysis of this added overhead. \n- The reward is only a scalar score without fine-grained feedback. The authors could explore using a more fine-grained rewards on multiple criteria, such as style, relevance, persona-consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "izuUSCSZeW", "reviewer": "Reviewer_wAWp", "summary": "The paper formulates “Interact-to-Persona Alignment” for LLMs as a partially observable control problem, where the user’s evolving persona is a hidden state variable. It introduces the Persona Alignment MDP (PAMDP) and derives a Bellman equation specialized to this setting, a dual-critic advantage estimator, and shows the estimator is unbiased relative to the standard POMDP advantage. The actor produces a continuous latent action that steers the base LLM’s generation; critics estimate values with and without access to the hidden persona during training. Experiments span offline datasets (ALOE, PrefEval) and an LLM-based online simulator, reporting improvements over prompt/PEFT/FPFT/CoT and asymmetric-A2C baselines.", "questions": "NA", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Treating persona alignment as a partially observable control problem pins down what is observed (dialogue history) vs. hidden (persona) and yields a proper Bellman structure. This makes the problem analyzable and comparable to established POMDP methods, rather than staying at the level of ad-hoc prompting.\n\nUsing one critic with history only and another with access to persona during training gives an estimator that the authors argue is unbiased relative to the true POMDP advantage, unlike standard asymmetric A2C. The actor outputs a low-dimensional latent that steers the base LLM, with KL regularization to a behavior prior (from BC).\n\nThe paper’s Theorem 3 contrasts its estimator against asymmetric A2C (UAAC/DCRL-style) and argues lower bias. Methodologically it also situates the online study alongside PPDPP’s prompt-programmed dialog planning. Relative to recent personalization datasets (ALOE, PrefEval), the focus here is on treating persona as an unobserved variable updated through interaction.", "weaknesses": "The results rely on LLM-as-judge and LLM-generated rewards. Both the reward used during RL and the offline/online evaluation are produced by the same family model (Qwen2.5-72B-Instruct). The same family of models is involved in generating rewards and judging outputs. No human evaluation is reported.\n\nCumulative returns being negative makes it unclear how “good” a policy is on an absolute scale. In the online simulator (max 6 turns), all methods—including yours—show negative cumulative returns throughout; your method is best but still ends at about −2.5. Without a calibrated scale or a normalized metric, it is difficult to infer practical significance from “less negative” numbers. Please normalize rewards (e.g., to [0,1] or standardized z-scores), report success@k and step-wise human ratings, and include qualitative trace examples that map scores to perceived usefulness. Also justify the fixed 6-turn horizon and show sensitivity to longer dialogs.  \n\nThe online environment stitches together three LLM-prompted modules—Profile-Infer, User Simulator (which selectively reveals parts of the profile each turn), and a Reward Generator that mirrors the offline reward design. While useful for ablations, this pipeline may not reflect real users’ inconsistent, messy disclosures, and it inherits the same judge/reward biases. Please release the simulator prompts/code for scrutiny, stress-test with randomized persona noise and non-staged disclosures, and, if possible, run a small study with consented human participants to verify transfer.  \n\nTheorem 3 shows that the dual-critic advantage estimator is unbiased in expectation given access to the true hidden persona ω; in practice, the full-information critic uses ω while the actor never sees ω, and the system relies on LLM-inferred profiles or dataset metadata that can be noisy or partially wrong. That mismatch (and the function-approximation error in both critics) could re-introduce bias/variance. Please quantify sensitivity to persona noise and critic misspecification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WxbYTtNp4d", "reviewer": "Reviewer_6rqS", "summary": "This paper deals with the “Interact to Persona Alignment” problem. The authors conceptualize the issue as a Persona Consistency Markov Decision Process (PAMDP) and build a dual-critic reinforcement learning framework. They also evaluate their approach on both offline datasets and the online datasets with the user simulator, ultimately demonstrating its effectiveness.", "questions": "1. While the formalization based on POMDP is elegant, many of the core components lack clear implementation-level descriptions. For instance, what exactly are the hidden state H and the context w in the actual system? How are these represented in practice? Furthermore, why is a double-critic evaluation necessary? \n\n2. For evaluation, you compare your method against baseline models using win rate, but the original ALOE paper provides its own evaluation metrics. Why not use those as well? Additionally, the proposed method is not directly compared to other personalized methods like BE. This makes it difficult to clearly identify the advantages of your method. A more comprehensive comparison would help clarify its empirical strengths.\n\n3. The proposed method does not seem to construct an explicit user profile on the assistant side for reward computation. Instead, it relies on the quality of the final responses as a reward signal. How can we ensure that the user-related information embedded in the dialogue has a positive effect on model optimization? It is possible that the expected user profile may not be accurate, yet the resulting output is still good—how is this handled in your framework?\n\n4. Regarding the dynamic evaluation in multi-turn dialogues, have you considered the impact of the user simulator’s quality on the results? How is the user simulator initialized based on the user profile? More detailed explanations would help justify the reliability of the online evaluation setting.\n\n5. The citation format should be revised, and the meanings of each symbol should be clearly stated. In addition, the comparative experiments need to be added to the main content instead of the appendix, and the main content should illustrate more about your work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I find the concept transformation method proposed in this paper highly compelling. LLMs often struggle to adapt to user preferences in real-world applications, making personalization a challenging task. This paper addresses the problem by reformulating the Persona Alignment problem as a Markov Decision Process (PAMDP) and introduces a dual-critic reinforcement learning framework to enable a more streamlined and effective personalization experience.\n2. The proposed approach is notably simple (in a positive sense), intuitive, and demonstrates strong practical effectiveness.", "weaknesses": "1. The implementation details, especially in the experimental section, remain insufficient. Specifically, the mapping between the mathematical notations in the formulas and their actual representations in the model is unclear.\n2. The use of the ALOE benchmark in offline experiments raises some concerns. The evaluation metrics differ from those in the original ALOE paper, and the reward model employed appears questionable. It is unclear whether it aligns with the original reward function.\n3. In online experiments, the evaluation relies heavily on the user simulator. However, the design and reliability of the user simulator are not described in sufficient detail. It remains unclear how the simulator is built and how its quality is ensured.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1bh65ijl5V", "reviewer": "Reviewer_tcJ4", "summary": "This paper extends the definition of Partially Observed Markov Decision Process (POMDP) to dialogue settings. It then introduces an actor-critic algorithm to learn policy in this setting. The experiments on two datasets show that the proposed method can improve performance compared to simple baselines such as chain of thought and behavior cloning.", "questions": "How different is the new PAMDP, or it is just POMDP with different notations mapped to specific concepts in the dialogue setting?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The presentation of the paper is clear.\n2. The proposed method shows performance improvement on two datasets.", "weaknesses": "1. The proposed method relies on the quality of user simulator. If user simulator cannot well simulate real world settings which is often the case then the trained policy may work well with user simulator but may not work well under real settings.\n2. It will be better if other aspects of dialogues such as informativeness and helpfulness can be evaluated.\n3. The proposed method seems just a simple mapping of notations from POMDP. It seems not very novel or different.\n4. More advanced RL-based dialogue improvement framework can be compared as baselines, such as [1].\n\nReferences:\n\n[1] EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)