R4: Nested Reasoning-Retrieval for Reward Modeling in Role-Playing Agents
Authors: Renzhi Wang
Chongqiang Wei
Zhisheng Wang
Piji Li
Venue: ICLR 2026 Poster (2026)
Keywords: role-playing
knowledge augmented
Primary area: generative models

Abstract: Role-playing dialogue presents unique challenges for large language models (LLMs): beyond producing coherent text, models must sustain character persona, integrate contextual knowledge, and convey emotional nuance. Despite strong reasoning abilities, current LLMs often generate dialogue that is literal, stylistically bland, and misaligned with character-specific traits. Existing approaches such as retrieval-augmented generation (RAG) or reinforcement learning (RL) with scalar rewards are insufficient, as they cannot capture nuanced preferences or adapt reliably to diverse character contexts.
In this work, we introduce R4, a unified framework that equips both the reward model and the role-playing agent with reasoning and retrieval capabilities. Our reward model reformulates evaluation as structured reasoning: it integrates multi-step deliberation and retrieved knowledge to assess responses along multiple dimensions. This reward supervision is then used within reinforcement learning to train a dialogue agent with the same dual capabilities, enabling contextually grounded and persona-consistent generation.
Experiments demonstrate that R4 substantially improves dialogue quality, particularly in persona fidelity, narrative coherence, and emotional expressiveness. Analysis of training dynamics and case studies further shows that R4 agents employ retrieval more effectively, engage in retrieval-informed self-reflection, and achieve emergent role-playing behaviors unattainable by prior methods.

Published as a conference paper at ICLR 2026
R4: NESTED REASONING-RETRIEVA FOR REWARD MODELING IN
AGENTS
Renzhi Wang' Chongqiang Wei Zhisheng Wang', Piji Li NNanjing University of Acronautics and Astronautics Teeneett (rzhwang, pjli J@nuaa. edu.
ABSTRACT
Role-playing dialogue presents unique challenges for large language models (LLMs): beyond producing coherent text, models must sustain character persona, integrate contextual knowledge, and convey emotional nuance. Despite strong reasoning abilities, current LLMs often generate dialogue that literal, stylistically bland, and misaligned with character-speciff traits. Existing approaches such as -ugmeeetd generation (RAG) or reinforcement learning (RL) with scalar rewards are insufficient, as they cannot capture nuanced preferences or adapt reliably to diverse character contexts. this work, we introduce R4, a unified framework that equips both the reward model and the role-playing agent with reasoning and retrieval capabilities. Our reward model reformulates evaluation as structured reasoning: it integrates multi-step deliberation and retrieved knowledge to assess responses along multiple dimensions. This reward supervision is then used within reinforcement learning train dialogue agent with the same dual capabilities, enabling contextually grounded and persona-consisten generation. Ex- periments demonstrate that R4 substantially improves dialogue quality, particularly in persona fidelity, narrative coherence, and emotional expressiveness. Analysis of training dynamics and case studies further shows that R4 agents employ retrieval more effectively, engage in -erfevvlliifoormed self-reflection, and achieve emergent role-playing behaviors unattainable by prior methods.
INTRODUCTION
Large language models (LLMs), such as DeepSeek-RI (DeepSeek-Al et al. 2025) and OpenAl's series (OpenAl 2024p. have demonstrated remarkable reasoning capabilities across diverse tasks. including multi-hop question answering, code generation, and mathematical problem-solving. How- ever, this reasoning prowess does not readily translate open-domain dialogue, particularly in role-playing scenarios that require embodying established characters from rich narrative sources like novels. In this context, success is not defined by factual accuracy alone, but by nuanced fusion of character coherence, factual consistency, and emotional engagement (Feng et al. (025). Responses generated by reasoning models are often overly literal, stylistically flat, and insufficiently grounded in the speaker's persona, undermining the potential for immersive interaction.
The challenge lies in a fundamental misalignment between the objectives of conventional LLMs and the demands of authentic role-playing. Specifically: (1) aassoning-orienter models favors correctness, which leads to formulaic and rigid dialogue, sacrificing the expressiveness vital for role-playing, and (2) the absence of role-specific knowledge- such backstory, relationships, and evolving motivatioss-undrmiing narrative continuity and realism.
Retrieval Augmented Generation (RAG) has emerged as a promising approach for incorporating external knowledge into LLM outputs. While beneficial in knowledge-centric tasks, standard one- shot RAG approaches prove inadequate for role-playing dialogue due static query formulation and inability to adapt to evolving conversational contexts. Advanced retrieval systems, such as DeepResearcher (Zheng al. 2025) and WebThinker (Li al. 20256), offer more sophisticated
WWork done during internship in Tencent. Corresponning author.

---

Published as a conference paper at ICLR 2026
KL
LL.M Policy
Y/1
Reference Model
71
A
Y/2
F2
Group Computation
A2
Reasoning
Query
YG Rollout
Reward Colculation
Yg
...
Ag
...
Retrieval
E
Reward
Advantage
(a)
Query
x
<thinio </think> <search>
R4's Reward Model
Reasoning
Retrieval
</search>> <result> </result> <answer>
Condidate
323
response is \boxed()
(b)
Pair
Query
Reward
<thinko -... </think> <search> .onter
R4's Role-Playing Agent
</sesurch> <result> </result> <answer>
Do you Nono?
Reasoning
Retrievol
Love her? That's dangerous
(c)
question <hamswer>
Figure I: Architecture R4 framework. (a) Training pipeline. (b-c) Reward model performs struc- tured reasoning over response pairs and assigns comparative rewards to guide advantage estimation and policy updates. Both the agent and reward model integrate reasoning and retrieval throughout.
retrieval dynamics but rely heavily on handcrafted prompts and rule-based orchestration, making them brittle and computationally expensive. Recent iterative search frameworks like ReSearch (Chen et al. 2025a) and Search-R1 (Jin et al. 2025) integrate retrieval more seamlessly through reinforcement learning, yet still underperform in role-playing contexts where success depends not only on factual grounding but also on persona alignment and emotional authenticity.
A key challenge in applying reinforcement learning to role-playing dialogue is constructing reliable, multi-dimensional reward signals that capture the subjective and multifaceted nature of character portrayal. Unlike structured tasks such code generation mathematical reasoning- -where cor- rectness is objectively measurable- role playing dialogue quality encompasses persona consistency, emotional appropriateness, narrative coherence, and stylistic authenticity. As noted by (Tu et al.. 2024), evaluating candidate responses requires interpreting user intent, identifying appropriate evalu- ation criteria, and reasoning over implicit character motivations and narrative cues. This complexity necessitates reward models capable of structured reasoning grounded in character-specific knowledge.
However, current reward modeling approaches fall short. Our systematic analysis across diverse char- acters and evaluation contexts reveals two fundamental biases: (1) role bias: evaluation consistency varies dramatically based on character familiarity, with well-known characters receiving more stable assessments while lesser-knowe characters suffer from inconsistent scoring; and (2) reference bias: the availability of character-spe -paeciter-speccfiic contextual information significantly impacts evaluation quality, with models producing more accurate and consistent judgments when provided with relevant back- ground knowledge. These findings reveal a critical gap: existing reward models evaluate responses in isolation, without the contextual reasoning and character-specific grounding essential for reliable supervision in role-playing pole-playing scenarios. Consequently, reward models, like dialogue agents themselves, need to integrate reasoning and retrieval to generate credible supervision signals.
To address these challenges, we propose R4, a unified framework that endows both the Reward model and the Role-playing dialogue agent with the ability Reason and Retrieve. Our approach consists of three core components: (1) a character-specific knowledge construction pipeline that systematically extracts and organizes persona-relevant information from narrative sources, particularly focusing on literary characters from novels; (2) a reasoning- reasoning-augmentee reward model that performs structured multi-dimensional evaluation through reasoning chains grounded in retrieved character context; and (3) role-playing agent that integrates the same dual capabilities to generate contextually appropriate and Character-consister responses. The main contributions of this work are:
We reformulate reward modeling as structured reasoning task, introducing a novel reward model architecture that integrates lilei-idimeniioall evaluation through reasoning chains and retrieved knowledge, systematically addressing role bias and reference bias in existing approaches.
We propose end-to-end training framework that unifies reasoning and retrieval across reward modeling and response generation, enabling more reliable supervision and higher-quality dialogue tailored to literary character role-playing.
2

---

Published as a conference paper at ICLR 2026
We show across model scales that approach consistently enhances character fidelity, emotional expressiveness, and overall dialogue quality over existing methods.
METHODOLOGY: R4
Existing methods often treat reward modeling and response generation disjoint components, limiting their ability to reason contextually and adapt to character-specific knowledge. In contrast, our proposed R4 framework tightly integrates reasoning and retrieval mechanisms across both the reward model and the dialogue agent. In what follows, we describe the full design of R4.
2.1 CHARACTER-SPECIHC KNOWLEDGE CONSTRUCTION
To enable effective retrieval-augnee -aggmeted reasoning for role-playing dialogue, we develop com- prehensive knowledge construction pipeline that systematically extracts, organizes, and maintains character-specifie contextual information from narrative sources. This knowledge repository serves as foundational external memory accessed by both reward model and dialogue agent throughout training and inference, ensuring consistent character grounding across all system components.
Narrative Segmentation and Analysis. Our pipeline begins by processing narrative texts through an LLM-based segmentation strategy that partitions content into semantically coherent and plot-relevant units, inspired by (Duarte et al. 2024). For each identified segment, employ structured prompting to generate comprehensive character-centric representations that capture multiple dimensions of character information: (1) persona traits including personality characteristics, behavioral patterns, and distinctive mannerisms; (2) emotional states reflecting both explicit emotional expressions and latent psychological conditions; (3) contextual knowledge encompassing character backgrounds, relationships, and domain-specing expertise; and (4) narrative goals representing both short-term objectives and long-tenn perg-term character ares within the story context.
Knowledge Organization and Indexing. The extracted character information is structured into a hierarchical knowledge organization that supports efficient retrieval during reasoning processes. Fach knowledge entry is associated with multiple indexing keys, including character identifiers, emotional contexts, relationship dynamics, and narrative situations. This multi-faceted indexing enables precise retrieval of relevant information based conversational context and character interaction patterns. Additionally, we implement semantic clustering to group related knowledge entries, facilitating multi-hop reasoning scenarios where the model needs connect disparate pieces of character information.
Dynamic Knowledge Expansion and Validation. To enhance coverage and adapt to behaviors observed during model training, we implement a dynamic expansion mechanism that continuously enriches the knowledge base. During both reward model and dialogue agent training, retrieval queries generated by the models and use them to identify knowledge gaps. These queries then processed through automated synthesis using advanced language models (e.g., GPT-4o) and human-aun -athon- annotations for critical aspects requiring nuanced understanding. To ensure reliability, we employ automated consistency checks and periodic expert reviews to maintain knowledge quality and coherence across different narrative contexts.
collect
Implementation details are provided in Appendix B This character-specific knowledge construction pipeline forms the backhone of the R4 framework, providing both the reward model and dialogue agent with access to rich, contextually relevant information necessary for generating authentic and engaging role-playing interactions.
2.2 REWARD MODEI.
We now introduce the reasoning- and retrieval-augmented reward model in R4, designed to address fundamental limitations in conventional reward modeling approaches for role-playing dialogue.
Biases in Existing Reward Models. Current reward modeling approaches fall into two main paradigms. Scalar-based reward models (ScalarRM) (Liu et al. (2024a) formulate evaluation as binary or ordinal classification tasks, offering computational efficiency but providing opaque reward signals with no interpretable reasoning process. Generative-hased reward models (GenRM) (Zhang
3

---

Published as a conference paper at ICLR 2026
et al. 2025) leverage the full language modeling capacity to produce free-form explanations or preference judgments, enabling more flexible feedback but often lacking grounding in external context and struggling with consistency in subjective domains like character dialogue. To understand the limitations in role-playing contexts, we conduct systematic analysis of representative approaches from both paradigms: CharacterRM (TTu et al.) (024) a scalar-based method and instruction- following models as generative- based approaches (detailed in Appendix C.. Our analysis reveals two critical biases that compromise supervision reliability in role-playing tasks (Figure 2
Role Bias. Evaluation reliability is highly sensitive to character familiarity. As shown in Fig- ure a), main characters achieve both higher human consistency scores (e.g.. 0.87 vs. 0.61 under CharacterRM) and markedly lower stability variance (2.1 vs. 14.0) compared to minor charac- ters. This indicates that reward models benefit from abundant pretraining priors when judging well-known or prototypical roles, producing more confident and human-aligned evaluations. In contrast, lesser-known characters lack such priors, leading to unstable assessments where identical responses may receive conflicting scores, with evaluators often misattributing intentions or emotional tone. This instability undermines the reliability of reward signals and systematically disadvantages characters outside the model's training distribution.
Reference Bias. Evaluation outcomes are disproportionately shaped by the presence of explicit reference material. As shown in Figure bb) providing character-ppeeffic grounding (c.g.. persona descriptions, narrative summaries, role guidelines) yields consistently higher human consistency scores (0.79 vs. 0.70 under CharacterRM: 0.59 vs. 0.40 under Reasoning) and significantly lower stability variance (4.1-5.7 vs. up to 16.9). With such scaffolding, evaluators anchor judgments in concrete context, improving alignment with human expectations. Without reference, however, reward models frequently hallucinate motivations, overlook subtle traits, or fall back generic heuristics, leading to inconsistency and drift. This dependency highlights a critical limitation: conventional reward paradigms fail to robustly internalize character identity, rendering supervision brittle in open-ended or reference-scarce scenarios.
These observations reveal that existing reward models evaluate responses in isolation, without contextual reasoning over char- acter intent or narrative back- ground. Consequently, they fail to capture the multi-dimensional criteria essential for role-playing dialogue, including persona fi- delity, emotional appropriate- ness, and narrative coherence.
Performance on HalnyMinor Charactere
Performance with///thout Reference
Charsoe
wC's Nalenii
Hemsn Ceeatrerny Scors
servi lnd
neti Staaloo
Stasiles (eviaeee)
Boward Model Paradigm
Roward Model Paradign
Figure 2: Biases in conventional reward models.
Reward Modeling as Structured Reasoning. To address these limitations, reformulate reward modeling as a reasoning task that integrates multi-step deliberation with Sbaracter-specing knowledge retrieval. Given dialogue prompt and two candidate responses, our reward model must not only deter- mine preference but also generate a structured reasoning chain that systematically evaluates responses across multiple quality dimensions. The model accesses retrieved character-speciff information from our character-specific knowledge base, enabling deliberative, context-aware comparison analogous to human annotators reasoning over character intent and emotional appropriateness. Moreover, rather than relying on holistic impressions, we guide the model to consider specific aspects of role-playing dialogue quality, including conversational competence (fluency, coherence, consistency), character alignment (knowledge exposure, knowledge uccuracy, hallucination, persona filelity). and expres- sive quality (emotional authenticity, engagement, stylistic diversity). This systematic consideration ensures comprehensive evaluation while maintaining the flexibility of reasoning- -bsed approaches. The model conducts multi- turn reasoning, retrieving relevant character knowledge as needed, and synthesizes evidence across dimensions to reach well-grounded preference decisions.
Training Objective and Reward Function. Inspired by recent advances in reasoning model training (DeepSeek-AA et al. (025). we apply reinforeement learning directly optimize the reward model for high-quality comparative analysis, without requiring supervised reasoning traces. We guide learning through
reward function that integrates three complementary components:
4

---

Published as a conference paper at ICLR 2026
Format Reward. Ensures adherence structured reasoning format, validating proper of reasoning tags, retrieval operations, and conclusive preference statements.
Answer Reward. Measures the correctness of predicted preference against ground-truth label using accuracy.
Consistency Reward. Encourages alignment between reasoning and final decisions. Without such supervision, models may produce conclusions inconsistent with their reasoning chains. We train an auxiliary verifier to evaluate reasoning-decisine alignment, ensuring that preference choices are well-supported by generated analysis.
While consistency is valuable, integrating it as an independent additive reward introduces new risk: responses with incorrect answers but strong internal consistency may still receive high total reward. To prevent over-emphasis on consistency, we apply gated multiplicative formulation where consistency only enhances reward for correct answers:
T rm
Fans
Amti Fimmt
Foons) - ju(1 Tans)
where Tans E {0, denotes answer accuracy, Fint E {0, format compliance (single boxed conclusion with required tags), and Tcons [0, 1] the consistency score from an auxiliary verifier. We
set Aimi Acams = 0.1, and apply a mild penalty = 0.05 when the answer is incorrect..
Training with GRPO. We adopt the Guided Reward Policy Optimization (GRPO) framework (Shao et al. (2024) to train the reward model, combining hard preference labels with soft supervision from auxilaary reward components. Given an old policy Tow, and reference policy To.. GRPO optimizes policy To using groups of G rollouts (v:
Towa (-1z)) for each input x ~ 'D by maximizing:
I(0)
1 -
Σ min (piA,, clip(pt, 1-e, 1+c)A,) - (KL T
where p; Te (n (a) and A, Ih represents normalized advantages, with fr, Ty the mean and or standard deviation of (r,)g=] The KI. penalty TDx controls deviation from the reference model. To support retrieval-augmented reasoning, we extend the rollout format to include explicit search operations. During generation, the model issues search queries between <search> and </search> tags, with retrieved content inserted between <resul t> and </result> t> tags, enabling dynamic knowledge access throughout the reasoning process. Since retrieval results are externally generated, mask them during gradient computation, ensuring unbiased credit assignment and preventing overfitting to retrieved content.
Nolg: (2)
Training Data Construction. We construct comprehensive reward training dataset that supports reasoning-basse based evaluation across diverse role-play pole-playing scenarios. Our dataset combines multiple sources to ensure broad coverage of character types, dialogue contexts, and evaluation challenges:
Model-generated data (10K): We sample prompts from ChatHaruhi (Li et al.. 2023) and collect responses from 12 diverse dialogue models, including Qwen2.5 series, LLaMA3 series, GPT- 4o, and CharacterGI.M-O6 (Zhou al., 2024a). For each prompt, we randomly pair response and determine preference through committee voting using GPT-4o, Qwen2.5-72B --77B-Innttrct and CharacterRM, Response pairs with low inter-model agreement are submitted for human annotation to ensure reliability.
Filtered public data (4K): We augment the dataset with carefully filtered samples from Char- acterEval (Tu al., 2024), retaining only examples with well-defined prompts and reliable preference annotations matching our reasoning- bbaond evaluation framework.
Human-annotated data (2K). We manually annotate challenging response pairs exhibiting persona misalignment, knowledge hallucination, and emotional inconsistency, with 1K samples for training and IK reserved for evaluation. These examples particularly benefit from reasoning- based analysis, as they require nuanced understanding of character context and narrative coherence.
To mitigate role bias, ensure balanced character representation across the dataset, covering both well-known and lesser-known characters from diverse narrative sources. This yields a comprehensive corpus of 15K preference-labeeed instances that challenges the reward model develop robust
5

---

Published as a conference paper at ICLR 2026
reasoning capabiliities across varied role-playing contexts. On the held-out evaluation set (1K), trained reward model achieves 87 % agreement with human preferences, demonstrating reliable capture of nuanced judgments through structured reasoning processes.
2.3 ROLE-PLAYING AGENT
A central challenge in developing role-playing dialogue agents lies in acquiring high-quality supervi- sion that captures the nuanced requirements of character portrayal. Traditional approaches rely on manually curated dialogue datasets reflecting consistent character behavior, but constructing such supervised data is aabor-intensive and difficult to scale across diverse literary characters with complex narrative contexts.
Reinforcement Learning Framework. We adopt a reinforcement learning paradigm that circum- vents the need for supervised dialogue pairs by leveraging our reasoning -augmenne- reward model. Instead of learning from fixed examples, the agent receives preference-based feedback reflecting character alignment, narrative coherence, and emotional authenticity. This approach requires only character profiles and user queries, allowing the model to explore response strategies while developing the same reasoning and retrieval capabilities embedded the reward model.
Reward and Training. We adapt the trained pairwise reward model GRPO training through a scoring transformation that preserves reasoning-based evaluation quality. For each prompt, we sample candidate responses yl, WG and compute pairwise comparisons using the reward model's structured reasoning. Each response receives relative preference score:
prreter,
Σ 191
[RM(,, y1. 93) yrl
(3)
G
where RM(r, We, Mj) denotes the reward model's preferred response for prompt between yı and yj- These scores are then normalized to obtain advantages used in the GRPO objective. The final reward for response y combines preference scores with format compliance (Ammaa 0.1):
Tagent
Uprefer
Ximii
Throughout training, the role-playing agent has access to the same knowledge base as the reward model and is explicitly guided to reason over this context during response generation. The agent learns to formulate contextual queries, retrieve relevant character information, and integrate knowledge through multi-step reasoning processes. This creates a mutually reinforcing dynamic where improved reasoning enhances retrieval effectiveness, and better retrieval enables more sophisticated character understanding. As training progresses, the agent develops increasingly sophisticated reasoning patterns, conducting multi-hop reasoning across character traits, emotional states, and narrative contexts. The model learns to engage in self-reflective reasoning, adjusting response strategies based on retrieved information in ways that mirror authentic character portrayal processes.
Training Data Construction. We construct diverse prompt collection spanning multiple character types and interaction scenarios. Our dataset includes 10K high-quality queries from five novels covering 20 distinct characters in ChatHaruhi and CharacterEval, with balanced representation across character archetypes. Additionally. we manually construct IK focused prompts targeting three Dragon Raja characters to capture nuanced emotional dynamics and complex narrative situations requiring deep character understanding.
EXPERIMENTS
Datasets & Evaluation Metrics. We conduct end-to-end evaluation of the final role-playing agent using CharacterEval (Tu et al.) 2024) across 12 metrics in three categories: (1) Conversational Ability evaluates general dialogue competence through three core metrics: Fluency (Flu.) measures grammatical correctness and natural language flow; Coherency (Coh.) assesses topical relevance and logical connection to user prompts; and Consistency (Cons.) evaluates internal logical coherence across dialogue turns. (2) Character Consistency measures alignment with character identity across

---

Published as a conference paper at ICLR 2026
two complementary levels. Knowledge-baad based metries include Knowledge Exposure (KE), assessing appropriate demonstration of character-relevan information: Knowledge Accuracy (KA), measuring factual alignment with the character backgrounds; and Knowledge Hallucination (KH), evaluating the model's ability to avoid fabricating contradictory information. The persona-based metrics comprise Persona Behavior (PB), capturing alignment in actions, decision-making patterns, and behavioral consistency; and Persona Utterance (PU), assessing stylistic consistency in speech patterns and lin- guistic mannerisms. (3) Role-Playing Attractiveness reflects the expressive and emotional qualities essential for engaging character interactions, including Human-Likeness (HL), measuring natural and believable character portrayal; Communication Skills (CS), assessing interactive engagement and social competence; Expression Diversity (ED), evaluating varied and dynamic communication styles; and Empathy (Emp.), measuring appropriate emotional responses and empathetic engagement with users. To capture nuanced quality differences, we scale all ratings from the original 5-point scale to 100-point scale, enabling more precise performance differentiation across models.
Base LLMs & Baseline Methods. We compare R4 with three groups of baselines to ensure thor- ough evaluation: (1) Instruction models: representing general-purposs conversational AI, including GPT-4 Turbo, Llama-3. 1/3.3 series, and Qwen2. 5 series. (2) Reasoning models: optimized for multi-step reasoning and complex problem-solving, including OpenAI 01 mmini, DWQ-32B-Previeee DeepSeek-R1, and DeepSeek-R1-Dsse
variants across different model scales (Llama-8B/70B, Qwen-7B/32B). These models test whether general reasoning capabilities translate effectively to role- playing scenarios. (3) Specialized role-playing models; trained specifically on fooracter-focused dialogue data, including CharacterGLM--66 Xingcher[] MiniMax, and BC-NPC-Turbo (Tu et al. 2024). These models represent current best practices in dedicated role- ail-ppaying system development. These baselines establish performance expectations for general-purpose LLMs in role-playing con- texts. To ensure fair comparison, all baseline models are built on the same RAG infrastructure, retrieving from an identical curated, character-specifie knowledge base with consistent prompts, retrieval policies, and top-k settings.
Implementation Details. For the knowledge generation pipeline, we use gpt-40-2024-05-1 to perform all related steps, including narrative segmentation based on plot structure and the generation of character-specific knowledge such as persona attributes, internal states, and inferred goals. The reward model is built upon Qwen2.5-32B -nntuu-nntt and trained for 2 epochs on our constructed dataset. Similarly, the role-playing agents is based on wwwen...-B//2B-BB11 Instrruct and also trained for 2 epochs. We initialize from instruction-tuned models rather than base models to ensure more stable reinforcement learning and better final performance conoossnnneeecnniseeeen with findings reported in prior work such as ReSearch. The reinforcement learning framework is implemented using Verl [Sheng iet al. (2025) and ReSearch Chen et al. (2025a). Both the reward model and the role-playing agent share the same retrieval backend. We use multilingual-e5-lan with indexing and embedding handled by FlashRAG Jin et al. (2024a). Both models query the top-3 retrieved documents per prompt during training and inference. For baseline models comparisons, we directly adopt the implementations and services provided by FlashRAG. All training experiments are conducted on 64 NVIDIA H100 GPUs. Additional training configurations and hyperparameters provided in Appendix
Wang et al. (2024a) as the retriever,
3.1
MAIN RESULTS
Table I presents comprehensive performance comparisons between R4 and all baseline models.
Effectiveness of R4. On character consistency metrics, R4-32B- Instruct achieves the highest overall performance (64.64 64 average), significantly outperforming the best baseline (BC-NPC-Turbe at 55.28). Notable improvements include Knowledge Accuracy (+9.52 over the best baseline), Knowledge Hallucination (+2.34), and particularly striking gains in Persona Behavior (+9.8), demonstrating the framework's 's superior ability to align responses with character-specifie behavioral patterns. These substantial improvements validate the effectiveness of reasoning- augmented reward modeling in capturing both factual accuracy and nuanced character portrayal requirements. In conversational ability, R4 models achieve competitive performance with the strongest baselines while maintaining high scores across all dimensions. R4-32B-Instruct matches or exceeds top performers in Coherency
7

---

Published as a conference paper at ICLR 2026
Table 1: Detailed evaluation results. The best performances are highlighted in bold.
Model
Flu.1 Cab.
Conversational Ability
Avg-1
KAT
Character
Consistency
Avg-
Role play-ig Aftractiveness
Emp-1 Ang.T
Inotrnct Model
65.38
51.00
27.
|55.00
56. 80
Insruct
71.00
52,68
63.60
43.60
70B Instruct
37.00
Qwee2.5 5-78-lnstuuc
72.00
65.40
68.07
40.60
68
62.20
32.00
54.15
Qwen2.5 5-326-mnemut
67.80
75:00
69.30
70.73
44.20
62.00
53.80
44.00
58.00
53.40
66.00
63.00
38.60
64:20
$7.95
Reasoning Model
01-trini
72:30
57.40
QwQ-32B-PP Preview
66.00
72:20
66.20
68. 1.
41.30
61.60
57.00
12.80
55,20
49.63
63.00
58.80
30.60
62.35
$3.80
Deepweek- R1
45.30
63.30
60.40
50.67
41.04
47.60
42.20
25. 440
42.40
39.34
53.80
46.20
24.00
64:20
47.05
DanpSock-R1 I-Fisill-f Lame-88
55.40
61.60
$3.60
56.87
37.60
53.00
47.00
60
47.20
43.48
53,60
48.00
20: 50
52.3 7000
45.90
DangSeck-R1 --Fostll------------ Jame-708
62.00
60. 10.44
62.40
64.10 64
39.30 80
99.33 20
53.60
28. 60
$2.40
46.72
59.80
55.80
28.00
59 411
50.75
DeepSeek RI Distill- Qwen 7B
60.50
63.00
63.80
02.55
38.50
54.60
$1.40
20.00
53,60
43.62
49.00
54,00
25.20
63. 00
47.80
DeeySeck-RL -i---i-i-------- Quel---QeneBB 328
04.00
71,00
65.00
66.87
401.20
61.20
55.60
26.20
54.20
47.48
61.80
56,60
26.50
61.60
51.65
Speciallize model
M BB
68.28
74.34
74.74
72:45
32.50
56.78
54.76
46.02
59.38
49.67
14.76
45.30
39.32
56.24
53.91
Xingehen
67.56
76.14
75.08
72.93
32.72
55. 55.36
54.86
55. .4.44
61.10
51.90
75.14
45.44
42.00
54.64
MiniMax
72.18
78.64
76.22
75.68
36 70
58.20
58.88
55.
62.90
54.35
75.36
53.44
43.00
00.34
58.04
NPC Turbo
71.56
77.96
59 29.28
59.86
55.28
76.72
52.86
59.42
58,93
Ours
R4-70- Instintttntt
70.20
78.0
79.08
75.76
43:20
62:40
59.60 60
50,50
63.00
55.74
78.00
55.20
46.80
65 50
80,95
R4-32B
82.50
78.90
48.00
68,00
77,00
50.00
69.60
64,33
(82.30 vs. 78.64 for MiniMax) and Consistency (80.30 vs. 78.32 for BC-NPC-Turbo), while main- taining high Fluency scores (74.10). This demonstrates that reasoning augmented training enhances saraccer--pecifii capabilities without compromising fundamental conversational competence. Most remarkably, R4 shows exceptional performance in role-playing attractiveness, where 44-32B-Instruue achieves the highest overall score (64.93 vs. 58.93 for the best baseline). Particularly notable are improvements in Human-Likenede (+0.28), Communication Skills (+),10), Expression Diversity (+3,28), and Empathy (+5.40). These gains indicate R4's superior capacity for generating emotionally rich, socially engaging responses that create more immersive role-playing experiences.
Scale Sensitivity of R4. The cross-model comparison reveals fundamental limitations in existing approaches while confirming R4's framework advantages. Specialized role-playing models (BC- NPC-Turbo) achieve strong attractiveness scores but exhibit weaker character consistency, indicating supervised dialogue training alone cannot capture complex character reasoning requirements. Reason- ing models show moderate knowledge-baat based performance but consistently underperform in persona alignment (avg. 49.68 vs. R4's 64,64), confirming general reasoning capabilities do not transfer effec- tively to character portrayal. Instruction models demonstrate the most balanced performance among baselines but still fall short across all categories. Notably, model scale amplifies these architectural differences: while baseline performance improvements from scale are modest (e.g., Qwen2.5-7B to 32B: 48.80 vs. 53.40 in character consistency), R4 shows substantial scale benefits (55.74 to 64.64), indicating that the parameter counts. This scale sensitivity suggests that R4's unified approach requires sufficient model capacity to fully realize its integration benefits, distinguishing it from simpler supervised or scalar reward approaches.
framework becomes increasingly effective with larger
Human Evaluation. Our human evaluation with three annotators ranking 500 dialogue in- stances confirms that automatic metrics capture meaningful quality differences (Table 2 R4 achieves first rank in 68.2% of cases compared to GPT-4o's 21.6% and CharacterGLM-66'ss
Table 2: Human evaluation results.
Model
Rank 11
Rank 2
Rank 31,
Mean Rank?
R4-32u-Instruct
68.2 %
22.7 %
9.1 %
1.42
21.6% %
48.4 %
30.0%
1.94
ChanatteGLLM-6
10.2 %
28.9 %
60.9 %
2.64
10.2%, with statistical significance (p < 0.01). Crucially, annotator feedback indicates that R4's advantages stem primarily from superior persona fidelity and narrative dimensions our reasoning- augmentedd approach was designed to address. This alignment between automatic metrics and human judgment validates both our evaluation framework and the practical relevance of our improvements.
ppeccssey the
Training Dynamics. Figure reveall R4's learning process. The consistent reward improvement across training and validation sets demonstrates stable learning dynamics and effective alignment with multi-dimensional character portrayal objectives. The steady increase in response length without
8

---

Published as a conference paper at ICLR 2026
Training Rewerd
Vadidaaion Reward
Respense Lengght
Humber of Search
Traniig-Soaa
Traning Seup
Tranding Seap
Traiding Stap
Figure 3: Training dynamics of R4, including training/validation reward, response length, and search count over time.
corresponding quality degradation suggests that R4 develops genuine expressiveness rather than mere verbosity. Similarly, the evolution of retrieval operations (initial increase followed by efficiency gains) indicates that the model learns to formulate more effective queries over time rather than simply retrieving more information.
3.2 ABLATION STUDY
To validate each component's contribution, we conduct ablation studies examining both architectural choices and reward modeling approaches.
Q1: Which Matters More: Reward Quality or Agent Capability? Table 3 highlights fundamental asymmetry in the contributions of reasoning and retrieval to role-playing dialogue quality. Our findings show that reward model capabilitiess establish the foundation, while agent capabilities set the upper bound. Re- moving reasoning from the reward model (R4- RM w/o Reasoning) leads to a collapse in per- formance (49.87), nearly identical to the catas- trophic failure of removing reasoning entirely (48.23). In contrast, removing reasoning only from the agent (R4-Agent w/o Reasoning) re- tains substantially higher performance (56.94), despite operating under the same reward supervision (87% RM accuracy).
Table 3: Ablation Study.
Model Variant
Ability
Character Concistency
Role-Playing Attractiveness
Accuracy
Framework Components R4 (Full)
78,90
64.64
64,99
87.00
R4-ail who Reasoning
72.15
48.23
$2.84
74.20
R4-all wio Retrieval
74, 82
$1.67
55.19
76.80
R4-all GimericRatrivall
R4-RM wlo Reasoning
70.85
49.87
53.76
R4- gent win Retrieall
75.67
$5.47
57.92
87.00
R4 Ageni woo Reasoning
76.12
56.94
60.18
$7.00
Reward Fanction Components R4-RM wio Consistency w/o
Altermative Reward Model
ScalaRRM+AAgent
68.34
43.89
48.72
69.80
GenRM-Agent
69.87
45.34
50.16
71.40
CharacterRMi
46.87
This asymmetry yields key insight for role-playing system design: constrains what agents can achieve, regardless of their architectural sophistication. Even under perfect reward signals, agents without reasoning plateau 56.94; conversely, agents with reasoning but weaker supervision (76.3% RM accuracy) reach only 49.87. These results suggest multiplicative rather than additive interaction between reward quality and agent capability.
supervision quality fundamentally
Equally important, our analysis shows that simply replacing the reward model in existing systems is not sufficient. For instance, combining CharacterRM with our dialogue agent yields only 46.87-worse than even our ablated R4 variants (e.g., R4-RM w/o Reasoning: 49.87). This indicates that R4's effectiveness comes not from stronger individual components but from their deliberate co-design. In other words, reward and agent must be engineered to complement each other's reasoning and retrieval processes, rather than swapped in isolation.
Q2: Is Character-Speciffc Knowledge Necessary? The contrast between R4-all w/ GenericRe- trieval (47.95) and R4-RM w/o Retrieval (58.31) highlights the critical role knowledge specificity, Counterintuitively, removing retrieval from the reward model outperforms equipping the entire system with generic retrieval demonstrating that inaccurate or irrelevant knowledge is more damaging than having and underscores a key design principle: effective role- vole-playing systems depend on precisely curated, character-specife information, rather than broad but unfocused knowledge access.
all. This result overturns common assumptions about knowledge augmentation

---

Published as a conference paper at ICLR 2026
4 CONCLUSION
We present R4, unified framework that addresses the fundamental challenge of reliable supervision in role-playing dialogue by equipping both reward models and dialogue agents with reasoning and retrieval capabilities. By formulating reward modeling a structured reasoning task and integrating retrieval into both supervision and generation, our approach enables fine- tne-grained, cessonn--ligned evaluation and expressive, contextually grounded dialogue generation.
ACKNOWLEDGMENT
This research is supported by the National Natural Science Foundation of China NN..22761277, the Natural Science Foundation of Jiangsu Province (No. No.BKK0242039) the Basic Research Program of the Bureau of Science and Technology (ILF24001), the Scientific Research Starting Foundation of Nanjing University of Aeronautics and Astronautics (No. YQR21022) and the High Performance Computing Platform of Nanjing University of Aeronautics and Astronautics.
REFERENCES
Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. Research: Learning to reason with search for Iims via reinforcement learning. CoRR, abs/2503. 19470, 2025a. doi: (0-48550/ARXIV.250..2ooo
URL httLhhhts//ooo org 48w50//a..i.. 2503. 19470
Nuo Chen, Yan Wang. Haiyun Jiang, Deng Cai, Yuhan Li, Ziyang Chen, Longyue Wang, and Jia Li. Large language models meet harry potter: A dataset for aligning dialogue agents with characters. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6- 6-10, 2023, pp. 8506-8520. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023 HAw.://222.IIIIINSEEEEEIEE URL httph: org/10. 18653/v1/2023 Lindings-ennlp...
Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. Judgelrm: Large reasoning models as a judge, 2025b. URL https://arxivv www../o///bbb//5//// 00050
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv. Shuo Wang. Yuan Yao, Xu Han, Hao Peng, Yu Cheng. Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards. CoRR, abs/2502.01456, 2025. doi: 10448550/ARX1122550001155 URL https: org/10 48550/arXiv. www.aaaww............
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li. J. L. Cai. Jiaqi Ni. Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu. Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang. Minghua Zhang. Minghui Tang. Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. Li. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. CoRR, abs/2501. 12125501..199..... 2025. doi: 10.4850/ARXIV2500.1220 www.:/..//........2022 2501. 12948.
URL https://doi.
Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou Self-play with execution feedback: Improving instruction-follont following capabilities of large language models. In The Thirteenth International Conference Learning Representations, ICLR 2025,
10

---

Published as a conference paper at ICLR 2026
Singapore, April 24-28, 2025. OpenReview.net, 2025a. URL httpps//opeerrvieeeeeet/oooo id=cRR8oDFEBC
Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang. Ji-Rong Wen, and Zhicheng Dou. Understand what LLM needs: Dual preference alignment for retrieval-augmennte generation. In Guodong Long, Michale Blumestein, Yi Chang, Liane Lewin-Eytan, Helen Huang, and Elad Yom-Tov (eds.), Proceedings of the ACM Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 2025- 2 May 2025, pp. 4206 4225. ACM, 2025b. doi: 10.1145/3696400. 3714717. URL https://doi..... org/ 10. 11 / 3696410..... 3714717,
André V. Duarte, João Marques, Miguel Graça, Miguel Freire. Lei Li, and Arlindo L. Oliveira. Lumberchunker: Long-form narrative document segmentation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguisties: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pp. 6473-6486. Association for Computational Linguistics, 2024. URL https://aclaantho
org/ 2024. fwwningnn-niiiee -maiinggg-eenpppp emnlp. 377
Xiachong Feng, Longxu Dou, and Lingpeng Kong. Reasoning does not necessarily improve role- playing ability. CoRR, abs/2502. 16940, 2025. doi: 10448550/ARXX.25022.6444 URL https
7/doi. 100.g/10.oo 48550/8555111 http://arx.....2..1112 199...16949
Bowen Jin, Hansi Zeng. Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- rl: Training Ilms to reason and leverage search engines with reinforcement learning. CoRR, abs/2503.0pd 16, 2025, doi: 10448500ARXXV..5000000 2503.. 09516
https:////// http:////6o..01111.45555555 48.550
Jajji Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. Flashrag: A modular toolkit for efficient retrieeal-augmeened generation research. CoRR, abs/2405, 13576, 2024a. doi: 00-4850/ARXII.2405.1555
URL htttp:://dii.rrg./
48550/arXiv. 2405. 13576
Jajji Jin, Yutao Zhu, Yujia Zhou, and Zhicheng Dou. BIDER: bridging knowledge inconsistency for efficient eetrieval-augment llms via key supporting evidence. In Lei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 750-761. Association for Computational Linguistics, 2024b. doi: 1108660/11/224.F/NIINGGOO ACL.42. URL https //doi. 16533v1/2204 findings-acl.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmmentd generation for knowledge-intensiee NLP tasks. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Elsua--Tien Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, Decem- ber 6-12, 2020, virtual, 2020. URL htttps::/prroceednggss neur ips. cttpcc/pppppe...p..pp wwtpcc/pppppp//////// 6b493230285f78888 240
Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, Linkang Zhan, Yaokai Jia, Pingyu Wu, and Haozhen Sun. Chatharuhi: Reviving anime character in reality via large language model. CoRR, abs/2308 09597, 2023. doi: URL //doi. orgoorg////// 48550/arXii0 2308 09597
(0046550/ARXII2233.005555
Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yongkang Wu, Zhonghua Li, Qi Ye, and Zhicheng Dou. Retrollm: Empowering large language models to retrieve fine-grained evidence within generation. CoRR, abs/2412.11919 2024. doi: 10.48550/ARXI..4122.111
URL https://d/// 1ttp//ii..rr/11....5.513
Xiaoxi Li. Guanting Dong, Jiajie Jin, Yuyao Zhang. Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. CoRR, abs/2501 05366, 2025a. doi: 10.48550/ARXIV.5001 .0566..
arXiv 2501.250.000 05366
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability, 2025b. URL htt.l/tttsss//axi..rr/////////// 22177

---

Published as a conference paper at ICLR 2026
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang. Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in lims. CoRR. abs/2410. 18451, 2024a. doi: 1048550AARXX22410.88551111 @rriv. 2410. 18451
https:/p: http://oo.orgg//// org/ /
Jingyu Liu, Jiaen Lin, and Yong Liu. How much can RAG help the reasoning of llm? CoRR, abs/2410.02338,, 2024b. doi: 1044850/ARRXV.240000233.
https:::://d/toorr 48550/
arXiv. 221.2411..00 023388
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma. Chong Ruan, Peng Li, Yang Liu, and Yu Wu.
Inference-time scaling for generalist reward modeling, 2025. URL htttps://ariii org/oggg /bs/ 2504. 02495
Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. Uncertainty-aware reward model: Teaching reward models to know what is unknown. CoRR, abs/2410.00877, 2024. doi:
httpt::ss//ddo org/10, 48550/ 2410.09847
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: benchmark for general Al assistants. The Twelfth International Conference on Learning Representattins, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreviee,
OpenAl. Learning reason with Ilms, 2024.
URL https:///ppenai.commiii
Tearning- reraannootorrrr-wiiii -immm
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL/http //openreview.net//e
Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-LLLM: A trainable agent for role-playing. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 13153-13187, Singapore. December 2023. Association for Computational Linguistics. URL https: /taplant///llanthoooo org/ e00..emnlp-mmmm maappp-mmin...mmm 814/
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Decpscekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.033000, 2024. doi: 10.48550/ARXIIV2200222 https://doi. org/ www....5///a.ii....o. 242.5//2.....222..... 03300
URL
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 - 3 April 20255 pp. 1279-1297. ACM, 2025. doi: 10.1 00.1145/3680031 3696075. URL /tt.//oiioooooo org/ /00 11 /145/ /444//5663111.1 3696075)
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R -earrchr:::: Incentivizing the search capability in Iims via reinforcement learning. CoRR, abs/2503.05592 48550/arXiv.285 250.l....5.3..555
2025. doi: 10.48550/ARXI.22503.555
URL httpp::/dooi.org//
Jiejun Tan, Zhicheng Dou, Yutao Zhu, Peidong Guo, Kun Fang, and Ji-Rong Wen. Small models, big insights: Leveraging slim proxy models decide when and what to retrieve for llms. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand,
August 11-16, 2024, pp. 4420-4436 Association for Computational Linguistics, 2024. doi: 10. 18653/V1/2024 Aww.10224.ALLLLLLNNG... 2422. J //toi org/ 18653/v1/2024 acl - long. 242
12

---

Published as a conference paper at ICLR 2026
Quan Tu, Shilong Fan, Zihang Tian, Tianhao Shen, Shuo Shang, Xin Gao, and Rui Yan. Charactereval: A chinese benchmark for role-playing conversational agent evaluation. In Lun- Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), ACI. 2024, Bangkok, Thailand, August 11-16, 2024, pp. 11836-1 1850. Association for Computational Linguisties, 2024. doi: 10.18653 AC2-L24ACL
URL http::///oo..rr/10..888////////0
acl -ong. 638
Liang Wang. Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 9414-9423, Association for Computational Linguistics, 2023a. doi: 00.18633V1//2000 EMNLP-MAIN.55
org. 080.00.16633//111000 emnlp-main. 585
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual E5 text embeddings: A technical report. CoRR, abs/2402.05672. 2024a. doi: 10. 10.48550/ARXXI.2202..856
httpss//:///di http:////ooorr//////// 48..4855//rra /arvv. 24022056672
Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, and Furu Wei. Chain-of- retrieval augmented generation. CoRR, abs/2501.14442. 2025. doi: 10:48500ARXIV.2501.240
URL httpt:h//ooo....ooo http///......////./ 48550/ wtt.oo/g//.....//rrrak...o 2501. 14342.
Noah Wang, Z.y. Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang. Jie Fu, and Junran Peng. RoleLLM: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 14743-14777, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2024, findings-ace -al.ggs-acl.778. URL httpt:::///et //tp:::://aaaaaaa / longoologytyoyooooooooooooooyoo org 2024 wt1.:22...iiniiii findings acl..78.......... 878/
Renzhi Wang and Piji Li. Semantic are beacons: A semantic perspective for unveiling parameter- efficient fine-tune tuning in knowledge learning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August / 1-16, 2024, volume ACL 2024 of Findings of ACL, pp. 9523 9537. Association for Computational Linguisties, 2024a. doi: 10. 1/0.5//////222/AIIIIIG....
URL https::/doo org/rrgg//o 10. 18653/v112024 findings-anni acl.561 567
Renzhi Wang and Piji Li. Lemoe: Advanced mixture of experts adaptor for lifelong model editing of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 2551-2575 Associa- 1155://///222..MM..PMMMMM
tion for Computational Linguistics, 2024b. doi: https:///oi org/ 1tt.://....1115//111 18653/v1/2024 emnlp-main. main...mnn... 149
URL
Renzhi Wang and Piji Li. Memoe: Enhancing model editing with mixture of experts adaptors. CoRR. abs/2405. 19086. 2024c. doi: 104455WARXIV.2405..900 URL https://// org/10. 4
arxiv. 2405. 19086]
Renzhi Wang, Jing Li. and Piji Li. Infodiffusion: Information entropy aware diffusion process for non-autoregressive text generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6- 10, 2023, volume EMNLP 2023 of Findings of ACL., pp. 13757-13770. Association for Computational Linguistics. 2023b. doi: 10. 10. 16533v11/223.. Lindinns-eennpp 919
FINDIIGGS-E-SS EMNLP919. URL https:://oiii.....o org/
Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng. Wei Wang, Jiangjie Chen, Cheng Li, and Yanghua Xiao. InCharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties ( (Volume 1: Long Papers), pp. 1840-1873, Bangkok, Thailand, August 2024c. Association for Computational Linguistics. doi: 10. 10265///12222..... acll-long.. 1on. URL https://acllantholggy.orr//////
acl acl-long. 1027
13

---

Published as a conference paper at ICLR 2026
Zekun Xi, Wenbiao Yin, Jizhan Fang, Jialong Wu, Rumnan Fang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Omnithink: Expanding knowledge boundaries in machine writing through thinking. CoRR, abs/2501 097511 ab2.22000900. 2025. doi: 10.4850/ARXXV22501.0099 URL
https://doi. http://o..orrgg///// 10. 48550/arXiv arviv. 2501 .97511
Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and
Chunyuan Li. Llava-critic: Learning to evaluate multimodal models. CoRR, abs/2410.027122 2024. doi: 10.4550/ARXIV224100271 URL https://doi. org/10. 48558/ 2410. 02712
Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: improving retrieval-augmented Ims with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations, ICLR 2024. Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024a.
URL
Rui Xu, Xintao Wang. Jiangjie Chen, Siyu Yuan, Xinfeng Yuan, Jiaqing Liang, Zulong Chen, Xiaoqing Dong, and Yanghua Xiao. Character is destiny: Can large language models simulate persona-driven decisions in playin?? CoRR, 120p//444..11138. 2024b. doi: 10.48550/ARXIV.
2404.12138. URL httpp:://oo..i org/ 10. 48550/arXiv 2404. 12138
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The Thirleenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRe- view.net, 2025.
Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang. Wentao Zhang, and Bin Cui. Retrieval-augmenned generation for ai-generated content: A survey. CoRR, abs/2402.194733, 2024. doi: 10-4850/AAXXX122402..443.. URL https //doi.org/10 011111..555/ark/i 2402. 19473
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments, 2025. httpssstts///axii org/ abs 2504. 03160
Jialun Zhong, Wei Shen, Yanzeng Li, Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Zhou, Jinjie Gu, and Lei Zou. A comprehensive survey reward models: Taxonomy, applications, challenges, and future, 2025. URL https:///axxi grg/ abs///bs///55/ 12328
Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Pei Ke, Guanqun Bi, Libiao Peng, JiaMing Yang. Xiyao Xiao, Sahand Sabour, Xiaohan Zhang, Wenjing Hou, Yijia Zhang, Yuxiao Dong, Hongning Wang. Jie Tang, and Minlie Huang. CharacterGLMr: Customizing social characters with large language models. In Franck Dernoncourt, Daniel Preofiuc- Pietro, and Anastasia Shimorina (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 1457-1476. Miami, Florida, US, November 2024a. Association for Computational Linguistics. doi: 10.18653/v 1/2024.c2mm)))mm 21..44.mllpppai caanll-iduuttr.107 URL httph::tttaalaaeeeeeeet
wrgp:/y..rrg224444// wttp:/org222...mmp--nnddsutttete innlp-indsstr... 1077
Yujia Zhou, Zheng Liu, and Zhicheng Dou. Assistrag: Boosting the potential of large language models with an intelligent information assistant. CoRR, abs/241 .06805, 2024b. doi: 10.48550/
ARXIV.241 106605. URL/https: //do:. org/10. 48550/ aat.s0//xxiv... www.iiiii111111....... 2411.06805
14

---

Published as a conference paper at ICLR 2026
A
RELATED WORK
A.1 RETRIEVAL--UGGEENTER GENERATION
Retrieval-Augee Augmented Generation (RAG) enhances the capabilities of language models by incorporat- ing external knowledge through retrieval mechanisms, groouddnn responses in factual, domain-specific knowledge (Lewis et al. 2020 Zhao al.) 2024 Wang & Li 2024a). This approach has been widely studied across several dimensions, including determining when retrieval is necessary (Tan et al. 2024).
refining query representations (Mialon et al. 2024 Wang al. 2023a), compressing and filtering
retrieved content (Jin et al. | Xu et al. 2024a). mitigating noise from retrieved documents (Liu
2024b Dong et al. (2025b), and improving retrieval quality through instruction tuning (Dong et al.
al.
2025a; Zhou et al., 2024b). Beyond single-step retrieval, more sophisticated multi-step or
structured RAG pipelines have been proposed to support complex tasks such as multi-hop reasoning, knowledge planning, and decision-making in domain-specific contexts (Li et al., 2024 Wang et al.,
2025, Xi et al. (025). These methods often integrate search with structured task decomposition to guide model reasoning over intermediate knowledge states. Recent work also explores tighter coupling between retrieval and reasoning through prompt engineering or agentic frameworks. For instance, Search-ol
(Li et al. 2025a) introduces modular agent that dynamically interleaves retrieval and document-level reasoning Others employ reinforcement learning to jointly optimize search policies (Zheng al.
2025
al. 2025b) and reasoning procedures from scratch (Jin et al. 2025
Song et al. 2025 Chen et al. 2025a
Despite these advancements, current RAG methods remain underexplored in role-playing dialogue settings. Most prior efforts focus on fact-based or reasoning-centte benchmarks, offering limited support for persona-grounded interaction. In particular, they struggle retrieve and apply character- specific knowledge in emotionally rich or stylistically nuanced that lack consistency with the speaker's persona or fail to sustain user engagement. Our work addresses this gap by leveraging retrieval not just for factual grounding, but as a mechanism to dynamically support character fidelity and dialogue appeal in role-play scenarios.
inveersttios-eaddiie to responses
A.2 REWARD MODEL FOR REINFORCEMENT LEARNING
Reward modeling (RM) plays central role in aligning language models with human preferences in re- inforcement learning. Early reward models primarily focused on outcome-level supervision-rankinn complete outputs based on human preferences (Zhong et al. 2025). However, such models often fail to capture nuanced qualities such as reasoning faithfulness or process transparency. To address these limitations, recent work has explored process reward models (PRMs) that judge the correctness of intermediate steps in chain-of -f-thuugtt reasoning (Lightman et al. 2024 /Stlur ct al.) 2025| Cui et al. 2025). While effective, PRMs rely on manually annotated intermediate labels task-specific schemas, limiting their generalizability. To reduce dependence on hand-crafted annotations, genera- tive reward models (GRMs) have been proposed. Models such as Generative Verifiers (Zhang et al.) 2025), DeepSeek-GRN (Liu et al. 2025), ) and JudgeLM (Chen et al. (0255) frame reward modeling as a next-token prediction task, using generated reasoning chains and voting strategies to produce more interpretable and consistent judgments. Other approaches include prompt-based scoring using instruction-tune language models (Xiong et al. 2024), and scalar scoring with learned reward heads (Liu et al. [ Lou et al. 2024). While efficient, these methods tend to lack interpretability or rely heavily on prompt design.
Despite these advances, reward modeling remains underexplored in role-playing dialogue, where preferences are shaped not only by correctness, but also by persona alignment, emotional tone, and narrative coherence. Moreover, most reward models operate without access to character-specific knowledge or dynamic reasoning capabilities. In this work, we propose a reward model that integrates reasoning, retrieval, and verifiable feedback. By treating reward modeling reasoning task and grounding it in retrieved contextual knowledge, our model provides more accurate and interpretable supervision for training character-aware dialogue agents.
A.3 PLAYING DIALOGUE AGENT
Role- bole-playing dialogue agents aim to generate responses that are not only coherent but also consistent with the persona, background, and emotional state of given character. Most current approaches
15

---

Published as a conference paper at ICLR 2026
simulate character behavior through prompt engineering or instruction-tunin with character profiles, where models are conditioned on static persona descriptions (e.g., background. tone, or occupation)
to emulate specific roles during interaction (Chen al. 2023 Tu et al.) 2024 Zhou et al. 2024a).
Further improvements have been made through charaeter-specific fine-tuning, memory-basee re- trieval, and consistency-oriented generation, which enhance short-term persona fidelity by aligning generated responses with predefined style or factual constraints (Li et al. 2023 [hao et al. 2023 Wang et al., 2024b). However, these methods often focus on surface-level traits (such as tone and wording) and fall short in modeling the underlying cognitive process behind a character's decisions or expressions. Some recent works have explored deeper aspects of role-play, such as simulating internal motivations (Xu et al. 2024b) or reasoning from the character's perspective (Wang et al. 2024c). Yet, most of these rely on human-written references or evaluations, and lack a scalable method for embedding such reasoning capabilities into the generative process itself.
In contrast, our work approaches role-playing as a reasoning task grounded in character-speciff knowledge. We go beyond static prompts stylistic imitation by modeling how agents recall relevant context, reason about their own persona constraints, and produce responses that reflect both situational understanding and consistent character behavior. Our framework explicitly equips the dialogue model with retrieval and reasoning abilities, bridging the gap between surface-level persona simulation and deeper, goal-driven character modeling.
B DETAILS OF CAARACEE--SPECIEEE KNOWLEDGE CONSTRUCTION
B.1 NARRATIVE SEGMENTATION
System Prompt for Narrative Segmentation
You will receive as input an English Chinese document with paragraphs identified by 'ID XXXX: <text>'.
Task:
Find the first paragraph (not the first one) where the content clearly changes compared to the previous paragraphs.
Output: Return the ID of the paragraph with the content shift as in the exemplified format: 'Answer: ID XXXX'.
Additional Considerations: Avoid very long groups of paragraphs. Aim for good balance between identifying content shifts and keeping groups manageable.
Document:
To support retrieval-augmennted reasoning in role-playing dialogue, we design dynamic narrative segmentation procedure tailored for long-form fiction. This module decomposes narrative texts into semantically coherent and contextually self-contained segments, enabling more precise and relevant retrieval during downstream reasoning. Our approach draws inspiration from prior work on -.g.-e segmentation (e.g., LumberChunker (Duarte et al. (024)), but is adapted to better reflect the episodic, character-centric structure of fictional narratives.
We begin by preprocessing the input novel or story into paragraph-level units, each tagged with unique identifier. These paragraphs are grouped sequentially into candidate windows G,, such that the total token count of each window remains below a pre-defined threshold 8. The threshold is empirically chosen balance contextual completeness with model efficiency: large enough to preserve inter enter-paragraph coherence, yet small enough to avoid context overflow during LLM inference.
16

---

Published as a conference paper at ICLR 2026
For each group G, we query GPT-4o to determine whether semantic shift occurs within the window i.., whether paragraph introduces a new event, emotional beat, or dialogue context distinct from the preceding content. This shift point is selected as segmentation boundary. The next group G starts from the identified boundary, and the process repeats until the entire document is segmented. This iterative, LLMM-iffrmmed strategy ensures that each resulting chunk reflects a topically unified unit of narrative, improving both retrieval granularity and interpretability in downstream character-centrie knowledge synthesis. The prompt used is provided in Table B.1
B.2 CHARACTER-SPERR SPECIFIC KNOWLEDGE SYNTHESIS
Following narrative segmentation, we construct structured, character-aligned knowledge repository using the extracted narrative segments. To achieve fine-grained and persona-consistent retrieval and reasoning, we represent each narrative segment as a dynamic, structured "story-event tree", inspired by recent advances in hierarchical modeling and dataset construction Wang & Li 2024b/c). Each event tree explicitly captures key narrative dimensions, including temporal structure (time), environmental and contextual setting (scene), character states and relationships (character), and event progression (event), thus providing a comprehensive, interconnected representation of the narrative content. The synthesis process employs systematic four-agent pipeline, where each agent addresses specific aspects of character-centric knowledge construction:
Knowledge Extraction Agent. This agent leverages GPT--4 to extract essential factual information from each narrative segment through structured prompting. The extraction process targets five core categories: (I) explicit temporal markers (start and end timestamps, duration, sequence relationships), (2) spatial-environmmnttal contexts (geographical locations, physical settings, atmospheric conditions), (3) causal event chains (action-cone consequence relationships, trigger events, outcome states), (4) character presence and participation levels, and (5) narrative significance scores. To ensure consistency, we employ a standardized prompt template that constrains the output format to JSON schema with predefined fields, reducing extraction variability across segments.
Perspective Transformation Agent. This specialized agent reconstructs narrative events from individual character viewpoints through perspective-awart prompting strategies. For each character present in segment, the agent generates character-specific interpretations by conditioning on three key factors: (1) the character's established personality profile (accumulated from prior segments), (2) their emotional state trajectory, and (3) their relationship dynamics with other characters. The transformation process systematically infers internal attributes across multiple dimensions: personality traits introversionneettavvrsionn risk tolerance, emotional stability), emotional states (anxiety levels, mood valence, arousal intensity), and cognitive patterns (decision-make makinn heuristics, attention focus, memory priorities).
Mind Agent. Building upon persona-aligned event representations, this agent synthesizes deeper psychological reasoning through theory-of-mind modeling. The agent explicitly reconstructs each character's internal cognitive processes using structured psychological frameworks: motivation hierarchy (following Maslow's framework), cognitive biases (confirmation bias, attribution patterns), and emotional regulation strategies (coping mechanisms, defense patterns). We prompt GPT-4o with targeted psychological queries such "Given this character's 's established personality and current emotional state, what unconscious motivations drive their behavioral choice?" and "What cognitive dissonance or internal conflicts emerge from this situation?" The agent outputs detailed psychological annotations structured as destef-ddsirr-ieentii (BDI) triplets, enabling downstream reasoning about character behavior patterns.
Dialogue Extract Agent. This agent enhances the knowledge repository by identifying and pre- serving high-fidelity conversational exemplars from the source narrative. Rather than generating synthetic dialogues, the agent employs a two-stage selection process: (1) dialogue identification using linguistic markers (quotation detection, speaker attribution, conversational turn boundaries), and (2) relevance scoring based on psychological significance. The relevance scoring algorithm evaluates dialogues across four criteria: emotional intensity (measured through sentiment analysis and emotional lexicon matching), character revelation (degree of personality or motivation disclosure), re- lationship dynamics (power shifts, intimacy changes), and narrative pivotality (impact subsequent
17

---

Published as a conference paper at ICLR 2026
plot development). Selected dialogues annotated with contextual metadata including speaker emotional states, conversational goals, and implicit subtext interpretations. This approach ensures authentic character voice preservation while providing rich conversational anchors for downstream generation tasks.
Quality Assurance and Validation To ensure knowledge quality, implement mtlti-stage validation process: (I) inter-agent consistency checking to identify contradictions across different perspectives, (2) temporal coherence validation ensure character development follows logical progressions, and (3) psychological plausibility assessment using established personality psychology frameworks. Inconsistencies trigger automated revision cycles where conflicting interpretations are resolved through evidence- videence-bssed arbitration. This systematic approach ensures knowledge base maintains both factual accuracy and psychological authenticity.
The output from all these crents-Structuredd facts, cearacter-perspecttie interpretations, psychologi- cal annotations, and dialogue exemplars- are integrated into a unified character-speede speccffi-- knowledge base, organized into composable and indexed event-tree structure. This comprehensive knowledge representation enables precise multi-dimensional retrieval nuanced reasoning during downstream dia- logue generation and reward assessment tasks, significantly improving the realism and the consistency of the generated role- pole-playing interactions.
C DETAILS OF BIASES IN REWARD MODEL
To systematically investigate biases in reward models used for role-playing dialogue evaluation, conducted a detailed comparative study between two representative reward modeling paradigms: scalar-based reward modeling and generative-based reward modeling. Specifically, compared CharacterRM (scalar-based) with instruction tuned Breen..5-7BIInstrue and reasoning reasoning-enabled enablld generative models DeepSeek-RI. --i----DDi-iiileeeent
We constructed a controlled evaluation corpus based on two representative fictional universes: Dragon Raja and Harry Potter. We selected two main characters- Mingfei (Dragon Raja) and Harry Potter (Harry Potter) and four secondary characters: Hui Liyi, Xia Mi, Luna Lovegood, and Severus Snape. Each character was associated with 100 dialogue-response evaluation pairs, evenly covering diverse interaction contexts and emotional complexities. To establish reliable reference, the evaluation pairs were manually reviewed by the authors.
D IMPLEMENTATION DETAILS
For the knowledge generation pipeline, we use gpt-40-2024-05-13 to perform all related steps, including narrative segmentation based on plot structure and the generation of character-speciffc knowledge such persona attributes, internal states, and inferred goals. The reward model is built upon Qwen2.5-321 --ren2.5-32BInsttuucc Instruct and trained for epochs on our constructed dataset. Similarly, the role-playing agents is based on Qwen2.5-7B/32B. Insttrcctt and also trained for 2 epochs. We initialize from instruction-tuned models rather than base models to ensure more stable reinforcement learning with findings reported in prior work such ReSearch. The
and better final cerformance-cconistte reinforcement learning framework is implemented using Verl (Sheng et al. 2025) and ReSearch (Chen) al. 2025a). Both the reward model and the role- aile-pplyggg agent share the same retrieval backend. We use multilingualee5-lan
(Wang et al.) (024a) as the retriever, with indexing and embedding handled by FlashRAG [Tin et al. 2024a). Both models query the top-3 retrieved documents per prompt during training and inference. For baseline models comparisons, we directly adopt the implementations and services provided by FlashRAG. All experiments were conducted on 64 NVIDIA H100 GPUs, with full parameter optimization. For reinforcement learning of reward model and dialogue agent, used a learning rate x 10 and a training batch size 64. The model was trained for epochs, with each prompt generating 5 rollouts during the sampling phase. We set the rollout temperature to 1.0 to balance exploration and response diversity. Additionally, the KL loss coefficient is 0.001 and clipping ratio is 0.2 constrain policy updates and ensure training stability Wang al. (003b))
18

---

Published as a conference paper at ICLR 2026
Case study of R4's role-plin playing agent: Case (Romantic Ambiguity and Persona Adherence)
Character: Lu Mingfei
User Question: Do you love Nono?
Agent Response:
<think> This is direct question about Lu Mingfei's romantic feelings- core emotional are the story. Lu Mingfei doesn't openly confess. He often avoids the topic or hides behind humor and self-deprecation. Still, he can't outright deny it doing so would break character consistency. A proper response needs to balance emotional tension: denial on the surface, vulnerability underneath, wrapped in sarcasm or jokes. </think>
<search>> Lu Mingfei's feelings toward Nono </search>>
<result> Lu Mingfei secretly likes Nono but never confesses directly... </result>
<think> The answer should reflect that: He won't say "yes" outright; He'll joke, then slip in wistful, subtle admission; Tone should shift between light sarcasm and buried longing; End on ambiguity, leaving the user to guess the real feeling. </think>
<answer>> Like her? She's the kind of girl who walks in sunlight with her own background music. Me standing shoot-as do you think could like someone like that?
her? It's like the local ramen guy photobombing a global fashion
Sure, fantasized-so much I could write 25 25-episode drama. No ads included. But in real life... just sneak glances and pretend I'm looking at the vending machine. So... don't know. </answer>
Case study of R4: Case (Self-Reflection and Character Growth)
Character: Lu Mingfei
User Question: Why did you go on that mission involving the Sheqi Bajia? thought you afraid of dying?
Agent Response:
<think> User's question touches a core contradiction Mingfei: his fear of death versus his pattern of self-sacrifice. On the surface, he's someone who avoids danger. My first instinct to say he didn't 't volunteer-he was pushed into it. </think>
<search>> Lu Mingfei motivation Sheqi Bajia mission </search>>
<result>> Initially hesitant and fearful; attempted avoidance... Later learned of the mission's 's link to Chen Motong, his brother </result>
<search>> Lu Mingfei emotional reaction to Chen Motong </search>
<result>> Shift from avoidance to personal responsibility... Expressed guilt and emotional paralysis upon learning of Chen's 's death... Quote: "I'm I' not a brave man, but sometimes even cowards get pushed forward." </result>
<think> Wait-thar that contradicts my initial assumption. He did go willingly in the end, out of guilt and unresolved family ties. Saying he only went because he was forced would ignore his emotional growth and deeper motivations. The final answer should reflect this progression: from fear to acceptance, driven by emotional weight. </think>
<answer>> I'm afraid dying- course am. You think didn't consider faking fever skip it? But that time was different. I learned it involved my brother, Chen Motong and he was already gone. just... froze. Sometimes you do things you're 're ready for, because not doing them would be worse. didn't brave-1 just didn't 't want to regret it later. </answer>
19

---

Published as a conference paper at ICLR 2026
E
CASE STUDY
We present two cases where the agent role-plays Lu Mingfei from Dragon Raja.
In the first case (Table[),, when asked "Do you love Nono?", the agent avoids direct confession, instead responding with humor, self-deprecation, and emotional ambiguiy--faitfully reflecting Lu Mingfei's canonical behavior. This illustrates the model's ability maintain nuanced character consistency and emotional appeal.
In the second case (Table D). the user challenges the agent's motivations: "Why did you go on that mission? thought you were afraid of dying?" Initially, the model assumes fear-based reluctance, but after retrieving relevant context (e-g.. family ties), it revises its reasoning and generates a more accurate, emotionally grounded response. This self-correction process highlights an emergent self- reflection ability, where retrieval informs dynamic reasoning adjustment.
These examples showcase R4's capacity to blend character grounding, retrieval-augmennte reasoning. and high-level behavioral coherence- key to effective role-playing dialogue.
20


Reviews:
[{"review_id": "DVVUcaD0ZS", "reviewer": "Reviewer_S5sd", "summary": "This research describes R4 (Reward, Role-play, Reason and Retrieve), a unified framework and system designed and developed to improve the performance of LLMs in role-playing dialogue. The authors identify a key problem: standard LLMs produce dialogue that is often bland and inconsistent with a character's persona. They argue and show that existing methods like simple Retrieval-Augmented Generation (RAG) or Reinforcement Learning (RL) with scalar rewards are insufficient and not optimal for maintaining the fidelity of the role playing dialogue agents.\n\nThe main contribution of this research is to equip both the reward model (RM) and the dialogue agent with interleaved strong reasoning and SOTA retrieval capabilities. The framework has three main parts:\n- A character-specific knowledge base is constructed by segmenting narrative sources (similar to lumbarchunking pipeline) and extracting persona traits, emotional states, and goals.\n- A reasoning-augmented reward model is trained using reinforcement learning (GRPO) to evaluate dialogue responses. Instead of just giving a score, this RM generates a structured reasoning chain, justifying its preference by retrieving and analyzing character knowledge. This process is designed to mitigate observed \"role bias\" (poor performance on lesser-known characters via use of balanced training datasets) and “(scarce-)reference bias\" (by providing relevant context).\n- A role-playing agent is then trained using the output of this RM as its reward signal. Because the agent shares the same reasoning-retrieval architecture, it learns to generate responses that are persona-consistent, emotionally expressive, and grounded in the narrative context.\n\nExperiments show that R4 significantly outperforms a wide range of baseline models—including instruction-tuned, reasoning-specialized, and dedicated role-playing models—particularly on metrics of character consistency and role-playing attractiveness.", "questions": "- Can you provide more details about multi-faceted hierarchical indexing scheme? How is the hierarchical index built (data structures, algorithms, etc) \n- Role-play can easily drift into unsafe behaviors; any studies on this topic would have made this paper more insightful and interesting from ML system’s safety perspective (have the authors thought of using safety aware rewards). \n- Beyond role-play and dialog systems implications, what broader ML insight should readers take from R4? Why is this role-play a good testbed for RLHF research? \n- There is a brief mention of dynamic expansion of the KB during construction - how are knowledge gaps detected and what if there are contradictions between synthesized content and original text?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Making the reward model and the agent symmetrical—both using the same reasoning and retrieval mechanisms—is compelling. It addresses the problem of misaligned objectives by ensuring the agent is optimized by a reward signal that understands and values the tasks (reasoning and retrieval) the agent must perform.\n- The experiments are comprehensive (showing results across many SOTA baselines) leveraging multiple SOTA metrics and methods from the role playing dialog literature (CharacterEval, FlashRAG, LumbarChunker)\n- The ablations in Table 3 provide insightful findings. The finding that reward model and supervision signal quality (reward formulation via structured reasoning over semantically retrieved content) is very critical to achieve best performance. Furthermore, demonstrating that generic retrieval is worse than no retrieval for the RM highlights the importance of knowledge specificity. -> this is the most critical result most suitable for this conference -> trying to understand this better will add a lot of value to this work (how well does this finding generalized across other domains?, more insights here)", "weaknesses": "- There’s no evidence that R4 transfers to other domains where reward + agent reasoning & retrieval might matter (e.g., tool-using agents, search-based assistants, task-oriented dialogue, or multimodal agents). Any reinforcement and validation of these findings from other existing literature will be very helpful here to make this work more relevant for ICLR venue (imo, this work seems more relevant for ACL/EMNLP venues or more focused tracks/workshops at ML conferences) \n- The paper makes a claim about using a \"multi-faceted hierarchical knowledge organization\" to support efficient and multi-hop reasoning. However, the description in Section 2.1 is vague and lacks implementation details. It is unclear how this structure is technically different from a standard vector database with metadata filtering. The paper does not explain the data structure, the clustering method, or provide a concrete example of how this hierarchy enables multi-hop reasoning in a way that a flat document index cannot. This part feels over-claimed and under-explained.\n- The paper does not discuss the practical, real-world applications for such a complex/speacialized  system.\n- The system is quite complex comprising of advanced (and costly, how costly?) segmentation, extraction, transformation, etc - these complex interaction may result in cascading errors, trying to understand what types of errors are more critical along with their relative impact will be insightful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c1KSLQr400", "reviewer": "Reviewer_qcjU", "summary": "The paper presents R4, a unified reasoning–retrieval framework for reliable reward modeling in role-playing dialogue. It identifies two key sources of bias—role bias and reference bias—and proposes a reasoning-augmented reward model combined with a retrieval-enhanced dialogue agent. The work is timely and addresses a genuine weakness in current LLM-based dialogue systems.", "questions": "- In Equation (1), the reward function includes rfmt1 with coefficient λfmt1, but it is unclear how this differs from λfmt2 used in Equation (4). Are these two separate formatting rewards, or do they share the same objective?\n\n- What is the architecture and training procedure of the consistency verifier used in the consistency reward? How is it supervised and integrated into the optimization pipeline?\n\n- How are retrieval queries generated during training? Are they manually templated, LLM-generated, or learned through gradient feedback?\n\n- How many characters are included in each category (main / minor)? Is the dataset balanced in terms of narrative diversity?\n\n- How sensitive is the overall system to errors or inconsistencies in the extracted knowledge base? Has any ablation been performed to measure this impact?\n\n- What is the inference latency relative to baseline models? How does computational cost scale with the number of retrieval operations?\nIs the requirement of 64 × H100 GPUs essential, or can the framework be trained with smaller resource budgets?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "- The paper tackles an important and underexplored problem — reliable reward modeling for role-playing dialogue — and proposes a reasonable unified reasoning-retrieval framework (R4) as a solution.\n\n- The identification of role bias and reference bias is insightful, and the experimental results demonstrate consistent and interpretable improvements across multiple dimensions.", "weaknesses": "- Methodological clarity is limited. The description of the structured / character-aligned knowledge base synthesis is vague. Specifically, the four agents involved in this process (knowledge extraction, perspective transformation, mind modeling, dialogue extraction) are not clearly defined in terms of their inputs, outputs, and organizational relationships. Furthermore, the final hierarchical tree structure of the character-aligned knowledge base is not explicitly illustrated or formalized.\n\n- Figure 2 lacks essential experimental details. It is unclear how “main” versus “minor” characters were defined and selected for evaluation. The selection criteria may directly affect the observed role bias.\n\n- Quality of narrative segmentation is not analyzed. Since the segmentation quality likely influences downstream character-specific knowledge construction, some form of segmentation-level evaluation (e.g., coherence metrics or human judgment) would strengthen the paper.\n\n- Missing comparison with related reasoning-augmented RL methods. The paper does not discuss or compare against recent works such as Process Reinforcement through Implicit Rewards (Cui et al., 2025), which is already cited. This omission weakens the claim of methodological novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hh090HRUwe", "reviewer": "Reviewer_2saS", "summary": "This paper proposes R4, a unified framework designed to enhance reward modeling and response generation for role-playing dialogue agents by integrating reasoning and retrieval capabilities.  The authors address the key limitations of existing methods (such as literal, stylistically bland responses, and poor alignment with character personas) by equipping both the reward model and the role-playing agent with structured reasoning and access to character-specific knowledge.\n\nThe key contribution of this work includes:\n\n- It proposes a novel reward model that integrates multi-dimensional evaluation and refines reward modeling as structured reasoning to mitigate role/reference biases.\n\n- Unifies reasoning/retrieval across reward modeling and response generation for role-playing.\n\n- Extensive experiments are conducted to show the advantages of the proposed R4.", "questions": "1. Did you calculate inter-annotator agreement for the 2K human data? \n\n2. How about the performance of R4 on long multi-turn dialogues (5+) ?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Role-playing dialogue is an interesting topic.\n2. This work refines reward modeling for role-playing dialogue as a structured reasoning task,  which can effectively address  role and reference biases in existing scalar/generative reward models.\n3. Unifies reasoning and retrieval across both reward models and dialogue agents, creating a mutually reinforcing framework for more authentic role-playing.\n4. Enough empirical study to show the shortages of prior reward models.\n5. The experiments are detailed and comprehensive. And the corresponding experimental results are good.", "weaknesses": "1. It is not very clear that how R4 performs with non-literary role-playing characters instead of only novel-derived ones.\n2. It would be better to compare more specialized baselines.\n3. The computational efficiency detail of this work is not very clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tcOMKYCWPt", "reviewer": "Reviewer_vFCm", "summary": "This paper considers training methods to improve role-playing agents. For role-playing agents, the authors show that it is insufficient to just return factually correct responses, as users also look for dimensions such as character coherence, factual consistency, and emotional engagement. To this end, the authors proposes R4, a method that first trains a generative reward model that performs both reasoning and retrieval from a character database (also constructed by the authors) using GRPO. Then, R4 uses the trained reward model to train the role-playing agent which also performs both reasoning and retrieval using GRPO. Results on the CharacterEval benchmark shows that the proposed method outperforms both current strongest reasoning models as well as some existing role-playing finetuned models such as CharacterGLM.", "questions": "- how many training and test runs were performed to obtain results in Table 1? Can the authors also report standard deviations, as the performance gap on some metrics are small.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- I believe the proposed method for constructing a character-specific knowledge database is sound. The database contains both diverse information (e.g., personal traits, emotional states, contextual knowledge, and narrative goals) and good quality checks (using both LLM and expert reviews).\n\n- R4-trained models showed better performance than all existing reasoning models/role-playing models on CharacterEval.\n\n- I believe some of the analysis provided in this work is sound and insightful. For example, findings such as \"reward model capabilities establish the foundation, while agent capabilities set the upper bound\" (L451-453) could generalize beyond role-playing tasks.", "weaknesses": "I believe the some of the contribution mentioned in the introduction is over-claimed and does not align well with the novelties I perceived in the method section. I detail them below.\n\n1. In L100-102 the paper claims \"we reformulate reward modeling as a structured reasoning task..\" and  \"novel reward model architecture that integrates ...reasoning and retrieval\". However, equipping generative reward models with retrieval (or more generally, tools) is not new, and has already been proposed in existing work such as [1-2].\n\n2. in L103-105 the paper claims \"we propose end-to-end training that unifies reasoning and retrieval...\", but the general method of performing retrieval (e.g., as a tool) during RL is has already been proposed by prior work such as Search-R1 and ReSearch. The main training difference is the reward model used during training (labeled reward for RLVR or a trained reward model), which I believe is too small of a change to be claimed as a novelty.\n\nHowever, I do believe the paper is novel in its focus on character-specific knowledge base construction and the importance of good reward models for role-playing (e.g., via RL training). I suggest the authors to perhaps adjust the focus in introduction towards section 2.1 and 2.2, instead of section 2.3.\n\n\n---\n\nReferences\n\n[1] Zhu, Jiachen, et al. \"Retrieval-Augmented Process Reward Model for Generalizable Mathematical Reasoning.\" arXiv preprint arXiv:2502.14361 (2025).\n\n[2] Gou, Zhibin, et al. \"Critic: Large language models can self-correct with tool-interactive critiquing.\" arXiv preprint arXiv:2305.11738 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)