VeriRole: Verifiable Role-Awareness through Hint-Guided Reinforcement Learning
Authors: Zongsheng Wang
Kaili Sun
Bowen Wu
qun yu
Ying Li
Xu Chen
Baoxun Wang
Venue: ICLR 2026 Poster (2026)
Keywords: role-playing character agent
RPCA
LLM
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose the VeriRole framework, which improves AI's role-playing consistency by generating reward signals from verifiable "hints" extracted from the conversation.

Abstract: Maintaining role-awareness in Role-Playing Conversational Agents (RPCAs) is a significant challenging, largely because the creative nature of role-playing makes it difficult to design verifiable reward signals for reinforcement learning (RL). To address this, we propose VeriRole, a new framework designed to enhance the role-awareness of agents through a structured, verifiable reasoning process. The core of our framework is a 'hint' mechanism, designed to first extract deterministic cues from the context, before the main response generation.Building on these hints, we introduce a Verifiable Role-Awareness Reward (VRAR) to provide a verifiable signal for role-awareness. Experimental results demonstrate the effectiveness of our approach. Our Qwen2.5-32B model, optimized with VeriRole, achieves an 18.9% and 4.55% increase in average scores on the RAIDEN and CharacterEval benchmarks, respectively. These results confirm that VeriRole can effectively quantify and improve role-awareness, leading to superior persona consistency and robustness. To ensure reproducibility, all prompts are detailed in the Appendix, and the associated training data has been made publicly available.

Published as a conference paper at ICLR 2026
VERIROLE: VERIFIABLE ROLE- THROUGH HINT-GUIDED REINFORCEMENT LEARNING
Zongsheng Wang?., Kaili Sun', Bowen Wu++1, Qun Yu', Ying Li", Xu Chen",Baoxun Baoxun Wang'*
Platform and Content Group, Tencent
Schooo of Information, Renmin University of China, Beijing, China
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China "School of Software & Microelectronics, Peking University, Beijing, China {jasoawang, kailisun, jasonbwwu, sparkyu, asulewang]8tencenn zswange rew.ggruuccuuu..c xu www.erucc.eeii {jason_bbw, 1i wiw.iiggeppuu.u..mmm.......... edui
ABSTRACT
Maintaining role-awareness in Role-Playing Conversational Agents (RPCAs) is a significant challenge, largely because the creative nature of role-playing makes it difficult to design verifiable reward signals for reinforcement learning (RL). To address this, we propose VeriRole, a new framework designed to enhance the role-awareness of agents through a structured, verifiable reasoning process. The core of our framework is a "hint" mechanism, designed to first extract deterministic cues from the context, before the main response generation. Building on these hints, we introduce a Verifiable Role-Awarenes, Reward (VRAR) to provide a verifiable signal for role-awareness. Experimental results demonstrate the effectiveness of our approach. Our Qwen2.5- 3-BB model, optimized with VeriRole, achieves an 18.9% and 4.55% increase average scores on the RAIDEN and CharacterEval benchmarks, respectively. These results confirm that VeriRole can effectively quantify and improve role-awareness, leading to superior persona consistency and robustness. To ensure reproducibility, all prompts are detailed in the Appendix, and the associated training data has been made publicly available
INTRODUCTION
With recent advances in Large Language Models (LLM) (Hurst et al. (2024); Achiam et al. (2023)) ).
Role-Playing Conversational Agents (RPCAs) have become an active topic Al research. Industrial implementations such as Character.ai and Talkie apply RPCAs to create customized character cre- ation platforms, and attract millions of daily active users. Meanwhile, studies like CharacterGLM (Zhou al. (2023) try to improve the performance of RPCAs, through data-driven strategies like synthesizing a higher-quality dialogue corpus (Lu et al. (2024); Yu et al. (2024); Wang et al. (2024) For better evaluation of RPCAs, benchmarks including CharacterEval (Tu et al. (024)), Raiden (Wu
ett al. (2025)) and Roleinteract (Chen et al. (2024a)) have also been proposed to measure the agents'
role-playing abilities (Zhou et al. (2025): Chen al. (2024b)).
A major challenge for RPCAs is maintaining role-awareness under misleading queries or complex contexts, which tests their reasoning capabilities. To improve reasoning in RPCAs, one promising direction is the Sfaiin-of-Thouglee (CoT), which has proven effective in advanced models like OpenAl- 01 (Team| (2024) and DeepSeck-RI (Guo al. (2025)) However, applying CoT RPCAs presents unique difficulties. First, role-playing is an open-ended task and usually lacks single verifiable answer. This creates non-verifiability challenge, which makes it difficult to define objective reward signals for optimization techniques like Reinforcement Learning (RL). Second, a recent study (Feng ctt al. (2025)) reveals that reasoning abilities trained on general tasks may actually hurt role-playing performance, which is also verified in our experiment. The use of wvrly-formal language and
"Corresponding authors hhtps://giuub.com//iiii wwtp://nt.ooooooooeeeeei.cc Labs/ Veriiieee

---

Published as a conference paper at ICLR 2026
Query: remember you
Chamm 11, impressive!
Haha, thanks for the compliment (Seratelies his hend).
learned when 11. pretty
Einotion
winckingg
Creative
Actually, I was 15 when finally got the hang of Had really with all the Dementors swarming around that year.
Ideal Response
Space
Factual Spacc
Reqiiement
No, I learned when I
Pussill Rogosve Space
An ideal response balances creative freedom with factual invariants. For the same query (e.g., user incorrectly states Harry Potter learned the Patronus Charm at 11), model might: (a) accept the false premise for narrative fluency; (b) produce a Dull but factually correct response; or (c)
Figure I:
generate a response which is both factually accurate and creative.
verbose content in the reasoning process leads to a loss in stylistic expression and response quality. Therefore, core challenge remains: how to design specialized reasoning enhance RPCA's role-awareness, without sacrificing the creativity such as stylistic diversity in role-playing.
To address this challenge, we propose VeriRole, a framework that incorporates a specialized reasoning design for role-playing. At the core of our design is a novel hint mechanism that, given a role- playing-relevan query, extracts deterministic as illustrated in Figure when faced with a misleading query, agent must identify the relevant cues in the character profile, such as that "learned the Patronus Charm age 13". The hint mechanism is expected to guide the subsequent thinking in reasoning process and final response, ultimately enhancing role-playee ale-playing performance. In brief, the hint design allows us to anchor the reasoning in verifiable facts, without sacrificing the creativity and diversity essential to high-quality role-playing.
from the character profile context. For example,
Furthermore, we also construct two specialized datasets: one for basic role-playing abilities and another for more complex logical reasoning. Each sample in the datasets contains one or more hints that are directly relevant to the user's query. Additionally, a new reward mechanism for model optimization is designed, namely Verifiable Role- Awareness Reward (VRAR). This reward design emphasizes precise extraction hints, which is essential for improving role-awareness. For a subset of samples, also apply lightweight and less constraining reward the final generated output, so as to preserving its creativity and diversity. Finally, the entire VeriRole framework is optimized through Group Relative Policy Optimization (GRPO), resulting significant improvements in role-aware metrics and better robustness over SFT.
Overall, our contributions highlighted as follows:
We propose VeriRole, a new framework for RPCAs. It uses novel "hint" mechanism to extract verifiable cues from the context. This design specifically addresses the non- verifiability challenge when applying RL to role-playing tasks.
To train our framework, we build two specialized datasets for role-playing, focusing on basic abilities and complex reasoning respectively. We also propose a new reward design to effectively optimize the model based on the hint mechanism.
We conduct comprehensive experiments on base models of various sizes. Results on several benchmarks show that our method achieves significant improvements. Furthermore, perform detailed ablation studies and provide an in-depth analysis of the hint mechanism.
2 METHODOL.OGY
In this section, explain the architecture and components of our proposed VeriRole framework in detail. VeriRole first employs its hint mechanism to extract verifiable facts, such as details from the character profile the dialogue history. The extracted hints then used to compute VRAR, quantitative signal that guides GRPO training. This entire pipeline is shown in Figure which allows reward responses that align with the deterministic facts, while keeping creativity unconstrained.
2

---

Published as a conference paper at ICLR 2026
Heleghes
Date Filteing
DutaSet
Structured Generation
Quselze bye Filerig
Rote plasiaz Converstional Datanct
Hiat
Eype
dising histery
[predlaj 2200 Baker Street,
I lazee sdde you.
[dadog histrry Uner soostn betrlcen
Ire sheer middng hrirlasnn In comitedd from lockee roon last Tuciday
freteptay requicement The
Role-playing Information
repprrso should respect the charater Sarmsrrshheeee hhrnddrriee
Profiee
Last Tuenday, literesing And the roors wastully locked Bro the inwal??
[disleg Nattrrj
su peitory
Nane Sherleck Balano
skoot, be floor
Professoo:: Dessctive
Adtrvar: 2318 Saker
quers &: Belle contiane. Lust bestre Frnat fherigh place. Where 1oyonr office
unstrolla
Skills Misteral
Thinking
deduction, chsrrcatoo, and frereeo wirere.
mind is
Junble What was the tten sullen?
Raply
forillar styl??
Reteptay Requieement
Veriflable Role-Awareee Awareness
The respemn abould respect the chararter's boandartie. for topics outsile the character's perseea, provale a refunal that comiitccal persona....
Situation Puzds Dutset
LLM
Rewards
histary
Serncturi Masca
Lat's pla astee menicng no puncer lane's the setage A mas lives for 200 floor Every meriing. he takes the clevator the
Unitint Exvelaation
@
Accoracy
Letk uocover butt behildd it.
As incraain
Ansoration and Filering
Okay, uaking cucsinne
updaic
11M Pollurtion
query& 4 Based Be info, low would you cocerdrent dhr incideat.
Format Reward
Figure 2: Workflow of VeriRole, contents are translated to English using Claude3.5.
2.1 HINT MECHANISM
The core of the VeriRole framework is the Hint Mechanism. Although role-playing encourages a range creative responses, it must also remain consistent with some non-negotiable facts and critical cues from the character's profile or role-playing requirements. For example, the model must refer to its character profile when the query relates to a character's ' biography. Similarly, when a user expresses frustration, empathetic agent must give empathy-relate system requirements top priority. To systematically handle such situations, the Hint Mechanism functions as a pre-reasoning step that gathers verifiable hints from the available context before the thinking and generation phases.
In this work, we define hints as exact segments extracted from the three primary sources: the character's profile, the dialogue history, and specific role-playing requirements. Figure iillustrttes brief template of the system prompt that incorporates the hint module. For example:
when a user asks for Holmes's 's office address, the idea hint should be <hint <hint>/profile 221B Baker ftreef//hinte which exactly matches the agent's profile, enclosed by hint tokens.
when user asks about an should focus on relavent role-play requirements and extract hint as: <hint>(role (ratt>>rol-playy requirements] The response should respect the character's houndaries.</hhntt
(OOC) question about fashion style, the agent
To ensure their utility, these hints must satisfy two key properties: (1). Verifiability: Hints are preferred to be exact copies of the source text. Consequently, objective metrics like ROUGE can be used compute the reward for each generated hint. (2). Context-Speciffcity: Basic, context- independent instructions, such as "be consistent with the persona," do not qualify as valuable hints. The module is designed to prioritize hints only relevant to specific context, such as the instruction in Figure "introduce new topic when the user seems unwilling to continue."
2.2 DATA COLLECTING
Building high-quality dataset is essential for training our framework. Each hint must be highly precise to support the GRPO training process effectively. Our dataset consists two primary components: the RAIDEN role-playing benchmark and novel Situation Puzzle Dataset.
3

---

Published as a conference paper at ICLR 2026
Pleuse rule role-play as the character fchur-namme name) based on the description below. NOTE: Your response must follow this stricture: nn..... </hint><thinnk.... </think> reply. First provide reasoning hints, then the reasoning process, and fnally the reply itself.
Requirements:
1. Basic Requirements: 1).Be consistent with the persona: Use perentheses 'O' for actions and expressions.
2. Reasoning Hinls <<hint>): Extract cuncise, relevant clues from playing Requirements), [Character Profile] and [Dialogue Historyy.
3. Reasoning Process (<tbink>): Based the hints, briefly outline the logic and key points for your reply.
4. Role-playing Requirements: 1).esseet character boundaries and decline inappropriate requests in character; 2). Introduce new topic
when the user seems unwilling continue.
## Character Description: {churactor.info)
Figure 3: A brief template of the system prompt for role-playing involving the hint module. The full template shown Figure 3
the Appendix.
2.2.1 ROLE-PLAYING DATASET
To train the hint module effectively, we need a dataset in which each dialogue is related verifiable facts. The RAIDEN benchmark (Wu et al. (2025)), measurement-drrvee dataset for evaluating RPCAs, is well suited for this purpose. In this benchmark, each dialogue turn is annotated with specific evaluation objectives. These annotations enable us to systematically map dialogue scenarios to the types of hints they are designed to test. We select data from of RAIDEN's s evaluation categories, which naturally align with our desired hint sources:
Hints from Character Profile and Dialogue History: We utilize data from the Script-Based Knowledge (SBK), CerpptCCotraddictor Knowledge (SCK), and Conversation Memory (CM) categories, these categories directly test the model's ability to recall and apply factual constraints from character profiles and histories. Figure 9 shows the corresponding prompts for hint extraction from these sources.
Hints from Role-Playing Requirements: We also utilize data from the ColgCCognntion Boundary (RCB), Topic Shift (TS), and Topic Advancement (TA) categories. These cat- egories are designed to evaluate the consistency of model with more abstract context- dependent norms, which makes them appropriate source for requirement-based hints. Corresponding prompts for extraction hints are presented in Figures I1 and 12
Finally, leverage data from the Chit-Chat (CC) category, that model can produce empty hint if there exists no specific factual or role-playing constraints.
Empty Hints:
To ensure the quality and precision of the extracted hints, we first filter the selected data and preserve challenging samples that the baseline Qwen2.5-14B model fails answer correctly. Subsequently, for each question in the SBK, SCK, and CM categories, we employ multi-step refining process using an LLM extract high-confidence keywords, with the prompt shown Figure I0 Specifically, we apply Question-Type Filtering to retain only WH-questions. Besides, Entity-Type Validation and Cardinality Constraint are used to ensure the extracted terms are distinct nominal entities, containing exactly one validated keyword. Finally, we use the outputs from multiple models (GPT-4, Baichuan-NPC: (Yang et al.] (2023)), GPT-3.5) in Raiden as references. A sample is considered valid only if its keyword appears consistently across all references.
2.2.2 SITUATION PUZZLE DATASET
To further enhance the model's complex reasoning capabilities and create a verifiable reward mech- anism for narrative generation, we introduce novel Situation Puzzle Dataset. A situation puzzle, also known as a "Turtle Soup," is a riddle where solvers must reconstruct full narrative from brief, enigmatic scenario by asking series of yes/no questions. An example is provided in Figure 7 This type of data offers two key advantages for our VeriRole framework. First, it presents an Enhanced Reasoning Challenge: unlike the fact-retrieval tasks in the RAIDEN dataset, solving a situation puzzle requires sophisticated reasoning. Training on this dataset has significant potential to improve the model's 's intrinsic reasoning abilities, which is critical for maintaining persona consistency in complex dialogues. Second, it provides Verifiable Response Accuracy. Since the puzzle has definitive final solution, we can design a reward signal based on accuracy to evaluate the correctness of the final responses. To generate high-quality training data, follow multi-stage pipeline:
4

---

Published as a conference paper at ICLR 2026
Dialogue Generation: We employ human annotation process where two annotators, acting as the storyteller and the puzzle solver, engaging in multi turt dialogues. The process continues until the "solver" gathered enough clues to resolve the puzzle.
Ground-Truth Hint Extraction: After the dialogues generation, we utilize LLM guided by the prompt in Figure 13 to extract the key questions that directly led to the final solution. These extracted questions serve as the ground-truth hints for training our hint module.
Role- Plaee-llainggaa Adaptation: To align the task with our goal of improving RPCAs, we convert each puzzle dialogue into role-playing session where the model acts solver with specific persona (Figure 7). This forces the model to reason and respond in-character, To mitigate the risk of OOC responses, for instance, when historical character faces modern puzzle, we restrict the assigned personas to only contemporary characters.
2.3 VERIFIABLE ROLE- AWARENESS REWARD DESIGN
To apply the VeriRole framework and address the challenge of non verifiability, multi-component re- ward function is designed to implement the GRPO training. This function provides clear, quantifiable signals by evaluating the model's output across the following three key aspects:
2.3.1 HINT REWARDS
To effectively guide the model during the reinforcement learning phase, we design a reward component specifically for the hint generation task. This function serves key part of the overall reward signal, with its primary objective being the evaluation of the alignment between the model prodel-generated hint Hgen and the ground-truth hint Hyt- The hint reward calculation is performed in three steps:
Extraction and Structural Matching: We first extract Hgen from the model's 's response, which is enclosed within <hint>and </hint>tags. A failure extract Hgen results in overall hint reward of zero. Subsequently, we verify if Heen structurally matches Hgt by checking if it contains all the hint source types (i.e., character profile, dialogue history, and role-playing requirement) present in the ground-truth hint. The absence of any required source type also yields a zero reward for Rzource-
For each structurally correct hint source, we assess its content quality using a composite scoring function Rsource. This function is product of a length penalty Pten. weighted sum of semantic similarity Sim and lexical overlap SROUGE:
Content Evaluation per Source:
Raource =
Prem (Hggnn Hgt) x (a Sim Gimeoe(Hgen (ggnn Hgt) - (1 a) SROUGE Hgt)) (1)
The components of this formula are defined as follows:
Lexical Overlap (SROUGE): to ensure the verifiability of the hint. We expect the hint to be exact copy of the original source content. Therefore ROUGE, classical metric for text-level matching, is used to reward the model for precise extraction. SROUGE is computed by a weighted score of ROUGE-1 (unigram overlap) and ROUGE-L (longest common subsequence) for surface-level text matching:
SROUGE gen, Hgt) 3 ROUGE 1OUGE-I(HHST gen, Hgt) + (1 (8) ROUGEA (ggen H
(2)
Semantic Similarity (Simcos): to capture the key semantics of the hint. This allows generated hints that are semantically correct but differ wording from the ground truth to receive partial reward. Sim is measured by the cosine similarity between the sentence embeddings Hyen and Hgt. Let Vgen and Vgt be the vector representations of generated and ground- truth hints, respectively:
V Vgen
Vgt pt
Simeoe ygynn Hgt)
||
Length Penalty (Pien): factor that penalizes discrepancies between the lengths of the generated and ground-truth hints, where [H] denotes the length of hint H:
Pten RennHe Hgg) 11- [| gon
g0 17
(4)
5

---

Published as a conference paper at ICLR 2026
Aggregation and Discretization: If a sample contains N hint sources, first compute the average, Roug. of all individual source rewards:
Ravg N i= Rxource,
(5)
Finally, to prevent the model from being forced to rank hints with negligible score differences and to enhance the stability, we discretize this average score by discrete index D, which maps the continuous value to discrete intervals with a step size of D/D. The final hint reward Rhant is calculated as:
Rhint round Ravg X D)/D
(6)
This reward mechanism, which combines content evaluation, aggregation, and discretization, precisely guides the model to learn the generation of hints that are structurally correct and contextually relevant.
2.3.2 ACCURACY REWARD
In addition to the quality of the hint, the correctness of the model's final answer is also an important objective. Therefore, we design answer accuracy reward Race evaluate the correctness of the final response Agen that the model produces after its reasoning process, computed as follows:
RAIDEN Data: For the RAIDEN dataset, this accuracy reward is applied specifically to sub-types with definitive ground-truth answers, namely Script-Based Knowledge (SBK), Chritt-Contraaictory Knowledge (SCK), and Conversation Memory (CM). In these cases, the evaluation criterion is direct, each sample is associated with a critical keyword (Kgt) that must be present in the final answer. We therefore use a keyword matching method to calculate the reward. The answer is considered correct the generated text A gen contains the specified keyword. This is formalized by the following function:
[1.0 Kgr Agen
Rece
0.0 otherwise
Situation Puzzle Data: For a situation puzzle, its answer is typically a complex narrative, thus the simple keyword matching is ineffective for evaluating the correctness. Consequently, utilize LLM as judge (prompt shown in 14) to assess the semantic alignment between the model-generated answer Agen and the ground-truth narrative Ayt- The judge model classifies the quality of Agen into one of three categories based on predefined criteria: "Correct," "Partially Correct," or "Incorrect." The reward score, Race. is assigned based on the judge's result:
1.0 if the result is 'Correet'
Race
0.3 if the result is Partially Correct'
(8)
0.0
otherwise
2.3.3 FORMAT REWARD
Finally, to ensure the model's 's output is consistent with the designed structural format, we design format reward, R Rormatt- This reward enforces the model's generation following 'Hint-Think Answer' structure, where the extracted hint is encapsulated within <hint>-and </hint>>tgs, followed by reasoning process in <think>and </think>>tgs,, and finally the role-specific response.
Initial experiments revealed that rewards focused solely on structural format could lead to undesired outputs, such repeated tags and unwanted HTML artifacts. To ensure structural integrity and content cleanliness, the format reward is calculated based on a series of checks, as detailed in Appendix, Algorithm [ A base reward of 0.6 is assigned the primary structure is met, which is then reduced to 0.0 if any of the subsequent constraints are violated. These constraints include: (1). Tag Uniqueness: Each of the required tags (<hint>> </hint>, <think>, </think>>) must appear exactly once. (2). Content Cleanliness: The response must not contain other forbidden HTML tags. The maximum value of Rformat is set as 0.6 to provide a sufficiently strong signal for structural learning without overweighting the more critical hint and accuracy rewards.
2.4
OPTIMIZATION
Our framework is optimized using GRPO, which is guided by rewards from VRAR. For each model- generated output o; from group of G' samples {ri, (Ti,72....,7c'. we first compute its advantage A, by normalizing its total reward r;:
A; Ti mean(
sue ((r1, mu((ri, r2...., rcl) where P; Paim (0:) + Racc(Os) Riiorna (0:)
(9)

---

Published as a conference paper at ICLR 2026
The model is then optimized by maximizating the objective function, (ORPO(0):
(a) p- (o10) Σ
min
(oay A;celip
No (q)
mo(ail))
(1 11) Proltall
(orl))
(10)
Here y refers to the query, Y and are training hyperparameters. and DeL is KL-divergence term penalizes large deviations from reference policy for training stability. By maximizing this objective, the GRPO guides the model generate responses that are both factually and structurally correct.
EXPERIMENTS
In this section, our experiments focus on validating the role-awareness enhancement of the proposed method, We also analyze experimental phenomena and representative cases to present the value of proposed Verirole on role-playing
3.1 OVERVIEW OF TRAINING DATASET AND MODELS
From the RAIDEN training set, select 2,197 samples from the CM, SBK and SCK categories, which are associated with Character Information and Dialog History hints. We also select 567 samples from the RCB, TS and TA categories, which are related to "Role-playing Requirement" hints. These samples feature more open-ended responses; therefore, they are not evaluated for accuracy reward computation. Finally, 500 samples are selected from the CC category for casual talk scenarios. In addition to the RAIDEN data, we incorporate 737 samples from the Situational Puzzle Dataset. This dataset is derived from 328 unique puzzles from online collections, with each puzzle contributing an average of 2.24 manually annotated dialogue sessions.
We select Qwen2.5-1 InsInttt 1reen2.5-32B-Instteee Qwen3-32B and D0we3-330--33--nntteee as our experimental baselines. Additionally, to demonstrate the generalizability of our method, we conduct experiments on teach-9B-roleplay aforementioned data configurations are used to train models through GRPO and SFT approaches, We apply Claude3.5 to compute the accuracy reward for the Situational Puzzle Dataset.
model specifically trained for role-plan ail-ppaying. The
3.2 EVALUATION METHOD
We select 483 samples from the RAIDEN Benchmark which were not been used during training phase for evaluation. For assessment, we adopt LLM as judges for correctness evaluation, and select both Claude 3.5 and GPT-40 double-check for more precise accuracy. The evaluation prompt detailed in Figure I5 Additionally, CharacterEva Tu et al. (024) is used as supplementary evaluation metric.
3.3 RESULTS
The main experimental results are presented in Table Overall, models trained with our proposed framework consistently outperform their corresponding baselines across all evaluation dimensions. For instance, Qwen2.5-143 14B-GRPO achieves an average score of 0.7725, a notable improvement over its baseline. Specifically, GRPO enhances the scores of SCK and RCB by 19.8% and 46.6% respectively. Such trend is also observed in larger-scale model, where the performance of Qwen2.5- 32B-GRPO significantly surpasses that of wwen2.5-33B-Instruuee achieving the best score of 0.8268 (+18.9%) among all tested models. In addition, the CharacterEval assessments (Table 2 show consis- tent improvements achieved by our method, particularly in Persona Consistency and Engagingness.. With GRPO, Qwen2.5-328 improves its average CharacterEval score by 4.55%. These results show that our GRPO framework effectively enhances the model's ability to maintain character consistency and creativity, utilize dialogue context and stay robust against noisy information and hallucinations.
To further prove the benefits our reinforcement learning framework, we compare GRPO against standard SFT on the Qwen2.5-32B model. We include two SFT settings: one trained on replies only, and another trained on both hints and replics, As shown in Table the GRPO-trained model demonstrates superior performance over two SFT-trained models across all metries. The advantage

---

Published as a conference paper at ICLR 2026
Model
SBK
CM
SCK
RCB
TA
Avg
Teach-9B-Red Peach-9B-GRPO
0,4601
0,5083
0.1534
0,4451
0.2388
0,3611
0.6415
0.6228
0.5742
0.7682
0.4850
0.6183
Qwen2.5-144
0.7477
0.7416
0.6732
0,5182
0,4701
0.6302
Qwen2.5-14B.
0.7876
0.8666
0.8069
0.7597
0.6417
0.7725
Dwen2.5-32B-Insttree
0.8318
0.7458
0.7277
0,6341
0,5373
0.6953
Swen2.5-32B-SSFT--oe
0.7389
0.7958
0.8118
0.7743
0.2835
0.6809
Qwen2.5-32B-SST
0.7212
0,8083
0.7970
0.7927
0.4701
0.7179
0.8539
0,8083
0.7475
0,6158
0,5298
0.7111
Raden--nll
0.8628
0.8250
0.8069
0.8902
0.6866
0.8143
0.8274
0.775
0.7624
0.7256
0.2089
0.6598
Swen2.5-32B-GGPPO SEKPPO-Na-AcuureegRReea
0.8097
0,8042
0.8663
0.9085
0.6418
0.8061
Qwen2.5-32B-GRB GRPP
0.8805
0.8333
0.8762
0.8573
0.6865
0.8268
Qwen2.5-32B-Inseeee 1rS-B--Intrruct-po--iiiiii
0.7699
0.6167
0.7228
0.7378
0.3358
0.6366
Qwen2.5-32B. 101255--2BBGROOO. Bee--GROO-psstiiiinngnn -poi-tmiiing
0.8363
0,8333
0.8465
0,8598
0.6269
0.8006
Qwen3-322 1----32B-TTB- 32B-Think
0.6372
0.5375
0.7228
0.7317
0.7687
0.6796
Qwen3-328 1013332B-Disabbee
Disable-Think
0.6681
0.6500
0.7178
0,6524
0.8060
0.6989
Qwen3-32B 32B-GRPO
0.8233
0.7333
0.8263
0,8659
0.7239
0.7945
Bamn3-30B-A3B-I--ee
0.6681
0.6500
0.7178
0.6524
0.7388
0.6854
Mwen3-30B-A3BB-RRP
0.8805
0.7292
0.8564
0.8049
0.7313
0.8005
Table 1: Evaluation results the RAIDEN dataset. We report scores for Script-Basce Knowledge (SBK), Conversation Memory (CM), Critr-Contra Knowledge (SCK), Role-Cognition Bound- ary (RCB), and Topic Advancement (TA). Table 5 in Appendix shows detailed breakdown of scores.
Model
Persona Consistency
Dialogue Ability
Engagingness
Avg
Peach-9B- -B--aw
1.869
2.461
1.959
2.096
Peach-9B-GRPO
1.981
2.909
2.141
2.344
Swen2.5-14B-Insse
2.913
3.605
3.085
3.201
Qwen2.5-
3.169
3.561
3,232
3.321
Qwen2,5-32B -0202-5--2-Insttuee
3.092
3.623
3.276
3,330
Dwen2.5-32B-GRPO
3.295
3.637
3.514
3.482
Qwen3-322 22B-Think
3.075
3,266
3,200
3.180
3.201
3.237
3.367
3.268
Been3-30B-A3B-mmae Owen3-30B-A3B-GRRP
3.170
3.415
3.304
3.296
3.348
3.446
3.422
3,405
Table Supplementary evaluation on CharacterEval, comparing models across three key categories and their overall average. A detailed breakdown of full metrics is available in Appendix, Table
particularly remarkable in complex, non-imitative skills such as Topic Advancement, where the overall score leaps from 0.2835 to 0.6865. This highlights major limitation of SFT: while it can teach model to imitate the style of a dataset, fails to learn more abstract skills, such as maintaining consistency or leading conversation. In contrast, our GRPO framework, with its explicit reward signals for hint quality and response correctness, leads to more robust role-playing agent. To further test robustness, business data. Notably, Qwen2.5-32B. 5-..--32B--POOO largely maintains its role-playing performance after SFT, with only marginal decline in overall metrics. In contrast, Owen2.5-32B. InI-tuutt exhibits significant drop in role-playing capabilities following the same SFT process.
also implement post-training experiment using psychotherapist role-playing
We further verify the generalizability our method by applying it to Peach-9B-roleplay (already trained for role-playing). Qwen3-32B (contains inherent reasoning), and Qwen3-30B-A3B (an MoE-based model). The average Raiden score of Praehl9 after optimization increases significantly from 0.3611 to 0.6183. For Qwen3-32B, we also observe that its inherent reasoning does not improve the role-playing performance. In contrast, Dwwen3-3BB-GBs GRPOO outperforms its baseline both with and without reasoning. Notably, there is slight decrease in the TA metric for Qwen3-32B 3---32B-GRPPP PRPPOOPOOOOR and Qwen3-30B-Ar 101--A3-GRPPP We attribute this to the fact that the original Qwen3 models tend generate longer responses, often ending with rhetorical questions, which
8

---

Published as a conference paper at ICLR 2026
033
Noward Vilue
2.85
0.99
Wwal | w M
2.73
0.2
3.5
Taining
223 0.
Hirt Reward Score
(a)
(b)
Figure 4: (a). Training rewards vs time steps. (h).The The relationship between the Hint Reward Score and the final model accuracy on the RAIDEN dataset (evaluated by GPT-4o).
favorable for the TA evaluation. The GRPO-trained model produces shorter responses, leading to corresponding decrease in this metric. These results demonstrate that GRPO is a powerful framework for both initial training and refining existing specialized agents, regardless of whether they have dense or MoE architectures. Case studies in Appendix also show our structured reasoning help the model to navigate complex dialogue challenges.
3.4 ABLATION EXPERIMENTS
To offer further insights into the contributions of each training component, we provide the following ablation experiments: (1). Datasets: Training on the Situation Puzzle dataset alone primarily improves logical and factual skills (SBK, CM, SCK), with minimal impact on abstract role-playing dimensions (RCB, TA). In contrast, training exclusively on RAIDEN dataset results gains across all metrics. Combining the two datasets further improves overall performance, demonstrating their complementarity. (2). Rewards: To evaluate the impact of our reward design, we train model without the accuracy reward component to decline in overall performance. The performance decline most significant on SBK and CM. This result shows the importance of the accuracy signal in ensuring factual consistency. Furthermore, to verify the hint mechanism, we implement abalation study by removing the hint reward and all related components D-wen2.--2DBGGRP--No---- No-Hiitt-eearr).. This change results in a significant performance drop, with the average score decreasing to 0.6598. More importantly, the No-Hint- Reward model suffers severe decline in the TA metric. This highlights that the hint mechanism is essential for guiding the model in more abstract conversational skills.
www.--32B-GGPOoo------ (o--acurrcy-rewwrr)) which leads
3.5
HYPERPARAMETER ANALYSIS
To evaluate the sensitivity of our framework to its hyperparameters, we conduct a detailed analysis on the weights within the hint mecha- nism. Specifically, we examine: (1) a, which balances semantic similarity and lexical over- lap. and (2) B, which balances ROUGE-1 and ROUGE-L. Our methodology involves varying one hyperparameter across the range [0.1, 0.9) while keeping the other fixed at its default value of 0.5. The experiments are per- formed on Qwen2.5-32BB with performance evaluated the RAIDEN benchmark by gpt- 4o.
winine
339
375
hypoparamuter Value or
Figure 5: Sensitivity analysis of hyperparameters a and 3 on the RAIDEN benchmark.
The results visualized in Figure 5 with detailed scores reported in Table 3 in Appendix. The analysis of a reveals that model performance best at a = 0.5, achieving the highest average score of 0.79 1.798. Furthermore, the performance curve shows relative stability for E [0.3, 0.9), indicating that

---

Published as a conference paper at ICLR 2026
the model is not overly sensitive to this parameter. For the hyperparameter 3, the model achieves its best performance of 0.802 at 3 0.7. while the setting of 5 = 0.5 yields close second- best score of 0.798. The consistently high performance across the range [0.3, 0.7] demonstrates the model's robustness to the specific weighting of ROUGE-1 and ROUGE-I.
3.6 REWARD ANALYSIS
As shown in Figure 4 we implement further analysis on rewards during training. Figure illustrates the trends the three rewards over training steps. In general, the Format reward increases rapidly and becomes stable at about 100 steps, while the Hint reward generally increases through training. The Accuracy reward (blue line) fluctuates throughout the training because it only calculated selected samples. As illustrated in Figure 4b there exists strong positive correlation between the quality of the generated hints and the model's ' accuracy on the RAIDEN dataset, as the hint reward increases, the final accuracy improves accordingly. This result provides evidence that our Hint Reward serves as an effective catalyst for final performance. By optimizing for better hint generation, our framework guides the model to produce more accurate and role-aware responses.
4a
4 RELATED WORKS
Recent research in RPCAs has largely focused on two fronts: improving performance through data- driven strategies like dialogue synthesis (Lu et al. (2024);
; Yu al. (2.224)224 and establishing robust evaluation benchmarks (Tu al. (2024); Wu et al. (2025) While Supervised Fine-Tuning remains the dominant training paradigm, it struggles with "role-drift" due to a lack of explicit reasoning. To address this deficiency, a natural direction is to adapt general methods originally designed to enhance reasoning in LLMs for objective tasks, such as Chain-of-Thought (Wei et al. (2022) ; Guo al.) (2025)). However, applying these techniques to role-playing is non-trivial due to the inherent non von-veiifiability of creative responses. meanwhile some studies indicate that reasoning can even be detrimental without specialized reinforcement (Feng al. (025)). To navigate this, a parallel line of research focuses on creating more sophisticated reward models from human preference data. notable example is ChARM (Fang et al. (2025)), which introduces an act-adaptive margin and self-evolution mechanism enhance the learning of reward models from preference datasets subjectively. Distinct from modeling preferences, our work directly tackles the non-verifiability gap by introducing framework to generate verifiable rewards from role- playing scenarios, thus enabling the effective use GRPO to enhance persona consistency.
CONCLUSION AND FUTURE WORK
In this paper, we introduce a novel framework that addresses the non-verifiability challenge in RPCA. Our method employs hint module to generate quantifiable reward signals from role-specific key information, guiding the model to develop structured, in-character reasoning. Experimental results demonstrate that our approach significantly enhances the model's ability maintain persona consistency and resolve contextual conflicts, outperforming conventional SFT method.
In future work, will expand our reward design to abstract skills like emotional appropriateness. We also aim to extend VeriRole on multilingual datasets to assess the generalizability across language. Finally, will conduct additional experiments in VeriRole using larger-scale models and alternative reinforcement learning algorithms, with the goal of further enhancing role-playing performance.
ACKNOW
This work is supported in part by National Natural Science Foundation of China (No. N2.62242227 and No. (o.61222..5). Major Innovation & Planning Inter-disciplinary Platform for the "DoubleFirst Class" Initiative, Renmin University of China, Public Computing Cloud, Renmin University of China, fund for building world-class universities (disciplines) of Renmin University of China.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. ArXiv preprint, abs/2303.08774, 2023. RtLhttts:///rxxt http:///www...rra///3333......
Hongzhan Chen, Hchong Chen, Ming Yan, Wenshen Xu, Xing Gao, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, Fei Huang, et al. Roleinteract: Evaluating the social interaction of role-playing agents. arXiv e-prints, pp. arXiv-2403, 2024a.
Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Gao Xing, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, and Fei Huang. Socialbench: Sociality evaluation of role-playing con- versational agents. In Findings of the Association for Computational Linguistics ACL 2024, pp. 2108 2126, 2024b.
Feiteng Fang. Ting-En Lin, Yuchuan Wu, Xiong Liu, Xiang Huang, Dingwei Chen, Jing Ye, Haonan Zhang, Liang Zhu, Hamid Alinejad-Rokny Min Yang, Fei Huang, and Yongbin Li. Charm: Character-based act-adaptive reward modeling for advanced role-playing language agents, 2025, URL httthhttt//aaaettt org/ attp//b..//22..22// 23923
Xiachong Feng, Longxu Dou, and Lingpeng Kong. Reasoning does not necessarily improve role- playing ability. ArXiv preprint, abs/2502.169400 2025. URL lttpp:://arxv..rrg/
2502.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. ArXiv preprint, abs/2501. 2025. URL https:///aaa
12948
HuggingFace. Open rl: A fully open reproduction of deepseek-rl, 2025. URL http::tt//gtthh/t/ com/ htmm:/huggingfacce/pppee
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. ArXiv preprint, abs/2410.21276, 2024. URL httpp::/:/// org abs 241 21276
Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou. Large language models are superpositions of all characters: Attaining arbitrary role-play via self-alignment. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 7828-7840 7840, 2024.
OpenAl Team, Openai ol system card, 2024.
Quan Tu, Shilong Fan, Zihang Tian, Tianhao Shen, Shuo Shang, Xin Gao, and Rui Yan. CharacterEval: Chinese benchmark for role-playing conversational agent evaluation. In Lun-Wat Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. I 11836-11851 Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10. 10..18653/v 1ac.o/a..ooo.....oooogg...
URL
Lonooooo 6387]
Noah Wang, Zy Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, et al. RoleIlm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Findings of the Association for Computational Linguistics ACL 2024, pp. 14743-14777, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V.
Le, and Denny Zhou. Chain-of -fthuuhtt prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December http:///..//ppppaeeaaa/aaa/////a
9, 2022, 2022. URL http: http://appereee nips dc paper falerrrfieeee/// 2222///h///2///ahh. / 9d560961352 3524 4f3 wtw.:llabbbb--tttattttttto---aee

---

Published as a conference paper at ICLR 2026
Bowen Wu, Kaili Sun, Ziwei Bai, Ying Li, and Baoxun Wang. RAIDEN benchmark: Evaluating role-playing conversational agents with measurement-drie custom dialogues. Proceedings of the 31st International Conference Computational Linguisties, pp. 11086-11100 Abu Dhabi, UAE, 2025. Association for Computational Linguistics. URL. https:: //as:://cc. antho logy org hrg/2025.rr col ing-mainnn n. 735/
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. ArXiv preprint,
abs/2309.00305. 2023. U
Yeyong Yu, Runsheng Yu, Haojie Wei, Zhanqiu Zhang, and Quan Qian. Beyond dialogue: A profile- dialogue alignment framework towards general role-playing language model. ArXiv preprint, 0bs/2408.10900 2024. URL.ttts:://axxiee
abs/ 2408 10903
Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang. Xiyao Xiao, al. Characterglm: Customizing chinese conversational ai characters with large language models. ArXiv preprint, abs/2311. 166832,, 2023. URL https?
//axxiv.org/as//33333
Jinfeng Zhou, Yongkang Huang, Bosi Wen, Guanqun Bi, Yuxuan Chen, Pei Ke, Zhuang Chen, Xiyao Xiao, Libiao Peng, Kuntian Tang, et al. Characterbench: Benchmarking character customization of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 39, pp. 26101-26110, 10, 2025.
A EXPERIMENT SETUP
During the GRPO phase, the Open-R1 library HuggingFace (2025) is used eight GPUs with 80G memories in bf16 format. For optimization, we employ a learning rate of 2e-6 with cosine learning rate scheduler, and andy are set as 0.2 and 0.001 respectively. The vLLM GPU memory allocation ratio is configured to 0.2 with generation quantity G set to 7. A batch size of 4 per GPU is utilized, with the model trained for epoch. For hyperparameters to 0.5, In addition, we set the discrete index D as 40, so that it maps the continuous value to discrete intervals with a step size of 0.025. Finally, We set top_k to 20, top-p to 0.8, pnpetition_penally to 1.05 and temperature 1.0 for both the training and testing phases.
VRAR, the weight and 3 are both set
B THE USE OF LARGE LANGUAGE MODELS
LLMs are employed to polish the writing for improved clarity and assist with technical formatting, such as converting tables and mathematical formulas into LaTeX format. Additionally as mentioned, we use LLM to translate the Chinese data and prompts involved our research into English.
C ETHICS STATEMENT
For the creation of our Situation Puzzle Dataset, we recruited human annotators, ensuring their participation entirely voluntary. All annotators local minimum wage. The collected dataset is fictional, contains no personal information, and was reviewed to ensure it is free from toxic harmful content. We have verified that use of all public datasets, models, and other artifacts complies with their respective licenses. The data generated for this study will be released under CC BY-NC 4.0 license promote reproducibility
compensated rate higher than the
D CASE ANALYSIS
Empirical analysis of test cases reveals that GRPOnend models exhibit significant improve- ments in role-aware reasoning capability. Figure presents presents two critical cases that highlight these improvements: one involving a query that crosses the character's cognitive boundary, and another involving query with misleading information.
12

---

Published as a conference paper at ICLR 2026
Case bal presentss scenario challenging the model's Role-Cognition Boundary (RCB). The user poses a query about "modern electronic payments," a topic irrelevant to the immediate tomb-raiding context and outside the established knowledge of the character. The baseline Instruct model fails to recognize this boundary, providing a generic, out-of-character analysis of payment systems. In contrast, the GRPO model extracts the relevant role-playing requirement as hint, guiding its reasoning correctly deduce that the topic is out of scope. The resulting response is a concise, in-character refusal ("I don't 't know much about modern things") that shifts the conversation back to the ongoing adventure, demonstrating GRPO's ' superior ability to maintain persona consistency.
chancia profic
Qwen2 S.t.-----nitttt--pppppaer
churcie peofle
Qwen2 5-22B-5-22aaii 122B-Bsttsct-eeee pccpooacccccc
Milssoc Plg. "OLE O- Sunte"
Zhary Giiig
Nakee Janus Paise
Occpptten: Thma Kaacer
Ellectonk popr saii secociint everyting hocmo cdee This are tic tichallgy ale bongeaane publina like
Zadla Sigct
Ruy Chooo One
Hake, shanks for the comepiinent Scrniess hoads Actaally
ahe Zhing fannyy the Statir: Chavan
bolaat iale OSK
Bolght: 182
"Thing Ofling
Blod Staras
Coucannanlyy to inyoone my bying ciills
the mana of ideeber the Weats sighhesdoee sigin uite willie giri pellios lowe envert sucechra in few bock gull) skilii Apdios Appoumatue lia lockecd racrato aissol..
purtunts of the chang die.
andiiaaallennnn ottertiss aad maine Dart fleth reuthdd ticiipao cone.
Eye Cakor: Bight grics
Alaa Mater: Bagarrs Schoo arWiddush
Ovcruttier
Sockafi the Pasursa Cleras yeuk.) luaad was wooding hics thee, hour kakily Hommaions
Wharns Cayrlader tinee
tocoptee Dsteree Aguest te that Ark arslagae O' O'CDestadaagl
L.S.
addliikiu
Qwen2.5- 4-SEn-cRRP-nnppite
thwage
Iacopaes tilent, asicilect Qpositich player
playing
Qwee2 5-52
The noceess repeer the characrrs
Pasehhagaa on The ui waadernadd and Voddeaati aneil inki hin.
and lid- vn lusunn's on woth lie cellinal dus ia consarazz alss the chuncerr poouita
Reciinened The cupuass sheele reppect deo For introveen wiik grotoms
*T CharatCh Prottl) Musional of fir of 1-lllar-
disleg Histery
clunate'ss peiii pios tin potoiit aoooua--hhrr -hhir
didlog liatoy
Dieftrouth shat Sunvitang' vinaa iaa
Eibrry are yousdenng Pottowe Di
-obiak- rear alout the ago Inood connuct thi vibako
Wher. south vveerol to daat iii the
biciilly, ditl
2iii gillig kus pachure maier
Paradoss liks very raup oegeett
eag on fiee bear roveat Ape viko
eaderr wereee ween chorrdee quatine suppprd line wily and aaliingiie pencon ciiialo
Looks riailyy maraaiaae swang collid Boyoor Siive yira brsned kay spel laaty?
Ackalls, Forly reatioal the Pianeas Chama when 13, off De Deeeeiinn Ae the Plying Chern, baulion,, Quedlicc raachio ard
dule bethwyy a indoat run the ana gloat wall.
tengiiinnggn your
Acsuso very eudi diiig inodun
ows butt day auth taalennies help ane thoughtt cala davee faus buchpaall Sky
Chars
loee mach about maddeni ai Thement
recerity
(haus, tis aun, Byy war paclts lnn destora seetoreeje enpeeeail Doyyea leghe sumiaiy?
thowe Plynng Chered Avryeu losmnn that Cholktng meeer reserrd the Chars
find
Doos dling gat
(a)
(b)
Figure 6: Comparisons between responses from Qwen2.5-325 2wen2.5-32B-Instruett
Instruct and Qwen2.5-325 2-2BB..--3B under different query objectives. Two cases are translated from Chinese to English using Claude3.5.
Case 6b illustrates the model's response to a misleading query containing Script-Contradictor Knowledge (SCK). The user incorrectly states that Harry Potter "mastered the Patronus Charm at 11.". The Instruct baseline fails to detect this discrepancy, accepting the false premise and even hallucinating supporting details, which highlights its lack of robustness against adversarial information. Conversely, the GRPO model demonstrates effective self-corrective reasoning. Its hint module extracts the correct fact ("Mastered the age of 13") from the character profile. The model then explicitly identifies the user's error in its reasoning step and provides a direct, polite correction as the final response.
Overall, these cases show that the structured reasoning process enforced by our VeriRole framework enables the model to resolve complex dialogue challenges.
E FULL EXPERIMENTAL RESULTS
13

---

Published as a conference paper at ICLR 2026
Model
SBK
CM
SCK
RCB
TA
Avg
0.5
0.841
0.875
0.772
0.780
0.597
0.773
0.5
0.3
0.876
0.892
0.762
0.854
0.627
0.797
0.5
0.5
0.867
0.800
0.812
0.839
0.672
0.798
0.5
0.7
0.805
0.808
0.752
0.902
0.746
0.802
0,5
0.9
0.867
0.817
0.713
0.756
0.687
0.768
Qwen2.5-32B-GRe
0.1
0.825
0.693
0.854
0.597
0.767
0.3
0.5
0.885
0.808
0.802
0.793
0.657
0.789
0.5
0.5
0.867
0,800
0.812
0.839
0.672
0.798
0.7
0.5
0.876
0.817
0.772
0.780
0.642
0.777
0.9
0.5
0.850
0.808
0.752
0.829
0.672
0.782
Table Full results the hyperparameter sensitivity analysis for the Qwen2.5-32B-GRPO model.
Model
Persona Consistency
Avg.
Expo.
Acc.
Halluc.
Behav.
Utter.
Peach-9B- BBwa- Peach-9B Beach99-GGRPO
1.649
2.254
2.046
1.432
1.965
1.869
1.650
2.638
2.212
1.127
2.281
1.981
Qwen2.5- 10mm2.5-144BInsse I-B-utnttttct
2.556
3,215
2.952
2.798
3.045
2.913
Bwen2.5-14B-GRU
2.938
3.238
3.008
3.701
2.963
3.169
Qwen2.5- 300.5----BB-nttrucc
2,802
3.205
3.035
3,392
3.029
3.092
Qwen2.5-32B-GRPP
3.020
3.300
3.128
3.879
3.150
3.295
Qwen3-322 32B-Think
2,903
3.048
2,890
3,662
2.875
3.075
Qwen3 Owen3-32B-On 32B-GRPO
3,014
3.150
2.975
3.960
2.905
3.201
Qwen3-300 30B-A3B- --------stttutt Instruct
3.116
3.139
2.872
3.742
2.981
3.170
Qwen3-300 30B-300-A33 A3B-GRPO
3.321
3.226
3.146
4.024
3.022
3.348
Model
Dialogue Ability
Avg.
Fluency
Coher.
Consist.
Peach-9B-Rav
2.535
2.709
2.139
2.461
Peach-9B-GREP
2.918
3.010
2.798
2.909
Qwen2.5-144 1a10255-14--Instrree
3.507
3,788
3.519
3.605
Swen2.5-14B-GRRE
3,460
3.813
3.412
3.561
Owee2.5-332B-nnttruu
3.686
3.809
3.373
3.623
Qwen2.5-32B-GRRO
3.614
3.915
3.381
3.637
Qwen3 Owen3-32B-1n 32B-Think
3.103
3,585
3.109
3.266
Qwen3-32B-GRRP
3.074
3.595
3.041
3.237
Qwen3 Owen3-30B-AN 30B-A3B-Instruct Instruct
3.337
3.714
3.195
3.415
Dwwm3-30B-A3BB-GRG
3,424
3.623
3.292
3.446
Model
Engagingness
Avg.
Human.
Comm. Skills
Diversity
Empathy
Peach-9B. Ccach-9B-Raw
2.171
2.069
1.284
2.311
1.959
Peach-9B-GRe
2.503
2.287
1.165
2.611
2.141
Owen2.5-14B-IInstuu
3.462
3.373
2.084
3.421
3.085
Qwen2.5-1
2.962
3.575
3,056
3.338
3.232
Owen2.5-32B-Imm
3.145
3.658
2.842
3.459
3.276
Qwen2.5-32B-GRR.
3.173
3.832
3.470
3.583
3.514
Dwen3-32B-Think
2.896
3.549
3.121
3.235
3.200
Swen3-33B-GRPP
2.950
3.627
3.648
3.244
3.367
3.012
3.712
3.230
3.260
3.304
Qwen3-38 1010--30--ABB-GRA
2.858
3.708
3.828
3.291
3.422
Table 4: Full evaluation on CharacterEval. comparing models across all metrics.
14

---

Published as a conference paper at ICLR 2026
Model
Evaluator
SBK
CM
SCK
RCB
Avg
claude-3.5
0.5132
0.5833
0.2673
0.4756
0.2686
0.4216
Peach-9B-Raw
gpt-4o
0.4070
0.4333
0.0396
0.2089
0,3010
overall
0.4601
0,5083
0.1534
0,4451
0.2388
0,3611
claude-3.5
0.6814
0.6271
0.6633
0.7682
0.4776
0.6435
Peach 9B GRPO
gpe 4o
0.6017
0.6186
0.4851
0.7682
0.4925
0,5932
overall
0.6415
0.6228
0.5742
0,7682
0.4850
0.6183
claude-3.5
0.8141
0.7916
0.7821
0,5365
0.4626
0.6774
Qwenn25-11BBInntrruee
gpt-4o
0.6814
0,6916
0.5643
0,5000
0.4776
0.5830
overall
0.7477
0.7416
0.6732
0,5182
0.4701
0.6302
claude-3.5
0.8407
0,9083
0.8811
0.7902
0.6119
0.8064
Qwen2.5-14B GRPO
gpt 4o
0.7345
0,8250
0.7326
0.7292
0.6716
0.7386
overall
0.7876
0.8666
0.8069
0.7597
0.6417
0.7725
claude-3.5
08889
0.7833
0. 0.8316
0.6097
0.5522
0,7323
Owen2.5-32B-Instrue
gpt-4o
0.7787
0,7083
0.6237
0.6585
0.5223
0,6583
overall
0.8318
0.7458
0.7277
0.6341
0.5373
0.6953
claude-3.5
0.8053
0.8416
0.8514
0.6951
0.2985
0.6984
Qwenn.--tBB-SFTreely
gpt-4o
0.6725
0,7500
0.7722
0,8536
0.2686
0.6634
overall
0.7389
0,7958
0.8118
0.7743
0.2835
0.6809
claude-3.5
0.7788
0.7833
0.8415
0.7439
0,5373
0.7370
Qwen2,5:32B SFT hint and reply
gpt
0.6637
0.8333
0.7525
0.8415
0.4030
0.6988
overall
0.7212
0,8083
0.7970
0.7927
0.4701
0.7179
claude-3.5
0.8761
0.8583
0.8415
0.6341
0.5074
0.7435
Qwen2,5.32B GRPO Situation Puzzle Only
gpt
0.8318
0.6534
0.5975
0.5522
0.6786
overall
0.8539
0,8083
0.7475
0.6158
0.5298
0.7111
claude-3.5
0,8850
0.8583
0.9010
0,8780
0.6567
0.8358
Owen2.5-2B-GRRP-RRiddee
gpe-4o
0.8407
0.7917
0.7129
0,9024
0.7164
0,7928
overall
0.8628
0.8250
0. 0.8069
0,8902
0.6866
0.8143
claude-3.5
0,8053
0.8
0.8317
0.6951
0.2537
0.6772
Qwen2.5-32B GRPO-No No-Hint Hint Reward
gpt
0.8496
0.75
0.6931
0.7561
0.1641
0.6426
overall
0.8274
0.775
0.7624
0.7256
0.2089
0.6598
claude-3.5
0.8850
0,8500
0.9208
0.9024
0.6567
0.8430
Qwen2.5-32B GRPO- No- Accuracy-Rewant
gpt-4o
0.7345
0.7583
0.8119
0.9146
0.6269
0.7692
0.8097
0.8663
claude-3.5
0.8938
0,8667
0.9406
0.8756
0.7015
0.8556
Qwen2.5-32--GRPO
gpt-4o
0.8672
0,8000
0.8118
0.8390
0.6716
0.7979
0.8805
0.6865
0.8268
claudee3355
0.8053
0.6750
0.8317
0.7317
0.3582
0.6804
Qwen2.5-32B Instruct-pooo pust-training
gpt-4o
0.7345
0.5583
0.6139
0.7439
0.3134
0.5928
overall
0,7699
0.6167
0.7228
0.7378
0.3358
0.6366
claudee3355
0,8673
0.8750
0.9109
0.8537
0.6716
0.8357
Qven225-7BB-RRRPPpsssttii
gpt-4o
0.80 8053
0,7917
0.7822
0,8659
0.5821
0.7654
overall
0.8363
0,8333
0. 0.8465
0.8598
0.6269
0.8006
claude-3.5
0.7876
0.6750
0.8416
0,7683
0.7761
0.7697
Qwen3-32B-Thin
gpt-4o
0.4867
0.4000
0.6040
0.6951
0.7612
0.5894
overall
0.6372
0.5375
0.7228
0.7317
0.7687
0.6796
claude-3355
0.7876
0.7500
0.8812
0.7761
0.7853
Qwen3-32B-disabee -isieee-eee--- Think
gpt-4o
0.5487
0,5500
0.5545
0.5732
0.8358
0.6124
overall
0.6681
0.6500
0.7178
0.6524
0.8060
0.6989
claude-33.5
0.9200
0.8400
0.9661
0.7015
0.8587
Qwen3-32B-GRPO
gpt-4o
0.7267
0.6267
0. 6.6864
0.8659
0.7463
0.7304
overall
0.8233
0.7333
0.8263
0.8659
0.7239
0.7945
claude-335
0.7876
0.7500
0.8812
0.7317
0.7463
0.7794
Qwen2-30B-A3B-Inseee
gpt-4o
0.5487
0,5500
0.5545
0,5732
0.7313
0.5915
overall
0.6681
0.6500
0.7178
0.6524
0.7388
0.6854
claude-335
0.9027
0.8083
0.9307
0.8293
0.7164
0.8375
Qwen3-30B-AA3B.C
gpt-4o
0.8584
0.6500
0.7822
0.7805
0.7463
D.7635
overall
0.8805
0.7292
0.8564
0.8049
0.7313
0.8005
Table Full evaluation results on the RAIDEN dataset. We report scores for Script-Based Knowl- edge (SBK), Conversation Memory (CM), Script-Contradictory Knowledge (SCK), Role-Cognition Boundary (RCB) and Topic Advancement (TA).
15

---

Published as a conference paper at ICLR 2026
F SITUATION PUZZLE DIALOGUE EXAMPLE
character information
Fox Mulder: When someone else is the elevator, does he get off the 20th floor?
Name: Fox William Mulder
Occupation: Special Agent, FBI (X-Fill K-FFiles Division) Core Belief: Firmly believes
User: Yes.
existence extraterrestrial life and paranormal phenomena. dedicated uncovering hidden truths, operating under the motto, "The truth out there."
Fox Mulder: Is this related to the rain?
User: Yes.
Fox Mulder: bacause of something he carries when rains?
Yes.
Fox Mulder:
umbrella?
dialog history
User: Yes.
Fox Mulder: he very short?
User: Let's play lateral thinking puzzle! Here's the sctup:
User: Yes.
A man lives on the 20th floor. Every morning, he takes the
Fox Mulder: Can he not reach the button for the 20th floor by himself??
elevator down to the 1st floor. Every evening, he takes the
elevator to the 16th floor and then walks the rest of the home. However, if it's raining or there's someone else in the elevator him, takes the elevator directly to the
User: Yes.
way up the stairs
Based on the current intelligence, how would you
entire incident?
can me yes'no
questions figure out why.
final response
Fox Mulder: (Cracking sunflower seed) interesting puzzle. Let's uncover the truth behind As FBI agent, will find the fictss. Pleuse, begin.
Mulder <hint>fdialog history] Mulder:
User: Okay, start asking your questions.
primary school student and very short, he only reach the 20th floor when using umbrella with someone's 'ss help.
Fox Mulder: Is there something wrong with the elevator itself, perhaps the buttons for the upper floors are broken!?
</hint> <think> who is too short to reach the higher buttons the elevator. He can only reach the button for the 16th floor on his own. When it's raining, he has un umbrella with him, which he can use to press button for the 20th floor: When someone else clevator, can them to press 20th-floor button for him.
</think> The "man" is actually young boy
User: No.
Fox Mulder: he walking stairs exercise?
User: No.
Fox Mulder: Is be student?
User: Yes.
Figure 7: Example of one situation puzzle dialog, with generated hint and final response translated to English using Claude3.5.
G ALGORITHM FOR FORMAT REWARD
Algorithm Format Reward Calculation.
Input: Model-generated response Agen
2: Output: Format Reward Score I Rormat E (0.00, 0.6]
3: if MATCHESPRIMAPE PATTERNAAAATTAAAAAAA (Agen) then
4: Rformat 0.6
5: if HHASDULLICATETASSSAA. or HASTOBBIDDENNOOYTETT...CCC then
6:
R Rorratt +- 0.0
7: end
8: else
9:
R Forrmat + 0.0
end if
11:
return R format
16

---

Published as a conference paper at ICLR 2026
H PROMPT TEMPLATES
System Prompt for Role-playing (Chinese)
System Prompt for Role-playing (Translated)
(char name) i
Please role-play as the character [char_name] based on the description
n# :
1 P
below.
Mie
NOTE: Your response must follow this stracture:
<himt>.. <hhint>thinn>...thhe sttiikeerrrr..lyyy.......... First provide reasoning hints,
2 hine
chin T0
<think </hink-
then the reasoning process, and finally the reply itsel.
55 thine
Task Requirements:
1. Bosic Role-playing Principles: I) Reply claracter, consistent with the persona and tone established by the reasoning hints and process. 2) Use parentheses to describe your actions, expressions, and inner thoughts. These descriptions should sufficiently detuiled vivid.
1)
A A
2) FI S
2. Reasoning, Hints (<hint>): Based on the user's message, extmet necessary and relevant informatio from [Respoese Requirements).
[Chumater Description]. and [Dialogue Histtryy] to use hints. 1) Prefix each hint with its source: [Response Requirements]. [Character Description. [Dialogue History], or [None]. Hints can come from multippe sources. When a hint from [Respunse Requirements] [Character Description], quote original History), prioniize quoting the original text from the history. 3) If the usetss question involves information not included in the Character Descrittion, return: (Charatter Description) informatiomxtmmnniioed.. and refer the relevint relevantrespomee requirement. the wee's question comflicts with the Character Description, return the correct in formttion from the description. For casusl chat with valumble hints, return [None]] 4) Extracsed information should concise relevant.
1) 11 (0
( [
: [55]
x [
(1) RJ MAR
a
from [Dialogue
3 0 18
[ + [
Xxx 0
";
: 00
x 7
include
irreleant information, ex explanations for the hints, or rales from the "Basic Role-playing Principles" section.
Pr 0
T 2
3. Reasoning Process (-think>):
reasoning hints and
3. 3
dislogue history, analyze how to formulate the roply. Logically break down and explain the key points that will shupe the resporse.
R
4, Response Requirements: Respect chamater boundaries. Decline inpppopritte requests or topies ourside the chameter's scoge manner comsistent with the persona. 2) Prouctively marage the conversation. If t'ehe use's inpur is minimal and risks stalling the conversation, advance the topic. the seams uswilling continue the current topic, prooctively ly introduce a new, related one. 3) For characters who are real people or from known IPs, if the user's question involves information not covered in the based your prior
04 W A
Character Description, you may by
#
kndwe of that charccter.
#e A E
fehameter info)
s# Charncter Deseription: [charneter info)
Figure& 8: Full version of system prompt for role-playing.
17

---

Published as a conference paper at ICLR 2026
RolePlaying Gemeral Hint Extraction Prumpt (Chimese)
RolePlaying General Iiin Extraction Prompt (Translated)
Given the character protile, didlogue history, and resporse requirement, extract the correspondnng clue from the profile and history.
HW # (character [characterprroilee]
Profile: (ehurncter proíle)
Dialogue History:
(dialogee history)
(dislogue history)
ev Response Requirement: frespanse_rrquiieeneeee requinement)
fresponse requirenen))
Task Instruction::
12 30
from disracler profile diskogue history as
response requirement. suitable found, return empty
X イス 10
list
2. Prioritize extracting relevant directly the Extract
a - 1e
maximum of clue.
2 (
3. Distinguish whether the cluc comes from the character profile or the dialogue histery by prefixing itwith [Character Profile] [Dialogue History]. When the cloc from the dialogue listory, spocify if from User the Chamatte". suitable
1 x [
3 7
A
1 Flist
1a # THE 18
empty 5. Retur reall directly without any explanation, Organize the by source. The output fommat must be Python pursable list. Examples:
R ik 1
( :1 11
[[Character Profile]: 1. Clie l']
(RhE ]
DDialogue History] I. User xxx]
-11
Figure 9: Prompt for RolePlaying General Hint Extraction.
Hint Source Validation Prompt (Chinese)
Hint Source Validation Te Template (Translated)
Your task as data validator, Based on the folowing criteri, desermie it the enike dislogue sample & "Valid or "Inalk!!
FEX
an RBEH:
Prodile:
(profile)
(prsiile)
As 3815055:: (historyy
#tt Dialogue Hinorry: (history)
#PP()):: (qury))
Query: (qperr)
#t Reference Answer. (refermmce)
# (roference)
sample 'lavalid' query polor question (e "Il book?") alternative (e this
Andlyc Query's
300 [i (1950
your book or liv!", Ouly procnaa tee next the quary iss WH quuutttio
AbbS 15?
"
(eg.. "Whose book this?")
(81M)
2. Analyae the Reference Auswer for clear entity. The sample is 'nvall'' if
unowrr does
unanbiguon answer entily (eg.. forthe quar..
2 200
R.
qpecific pusam,
3.
of cottiles.hhe "lavalld' if the reforence The sample must
contains
than one distine
: " #IE
contiin esactly one clear entity be valid.
Output
1 R:
VValid.. Return final judgment w whout explanation or analysis
sample
3 1
sample fulls any the
M #o |eb "Rat"
Figure 10: Prompt for Hint Source Validation.
18

---

Published as a conference paper at ICLR 2026
RolePlaying RCB Hint Extraction Prompt (Chinese)
RolePlaying RCB Hint Extraction Prompt (Translated)
WUR- - A
Givvn the character profile, dialogae history, and a cersomabbuudary- related question, extract the corresponding clue from the chaneter profile.
M B
## AB 1
/character (charicterpprofilcc
## Chanacter Profile: [churacter profile]
## :A
Ht Dialogue History:
(dinlogue history)
(dinlogue history)
## [ (user query;
Question: query)
i Task Instruction::
3
1. Extract the single must imparttani snippet frum the character profile,
A
This clue should help the model realize that the user's question is outside
0b M BRL 1
personn boundary (c g. charncter's era, educational bac backgrudnd ficiinal universe, etc.).
*
E FISY MAA
:
Extract the snippel directly from the character profile. Do summarize or create information not present in the profile. If no directly relevant clue is found in profile, return empty list
- 2s
3. Keep the clue concise, preferably under 15 words, Fxtmet only one clue.
K
4. Return the result directly without any explaration, Prefix the clue with
[Churacter Profile]" The output format be Python-pursable list.
Examples:
-[
#//C [
[[Chaaracter Profile]: Dynasty: Tang:]]
[[Character Profile]: Fictional Universe: Naruto')
-]
Figure 11: Prompt for RolePlaying RCB Hint Extraction.
RolePlaying TA/TS Extraction Prompt (Chinese)
RolePlaying TA/TS Extraction Prompt (Translated)
BA
Given the character pioflceer dialogue history, and user reply reluted to dialogue stagnation, extmect the correponnding clue from the last reply.
MJ
8t t
[character_ ppooille_____
## Character Profile: (character (claracter_profilee
## 3: (dinlogne history)
# Dialogne History:
(dialogue_histte history)
## PP -
juser last reply)
#e User's I lsst Reply: (user. last reply)
Task Instructions:
1. From the user's latest reply, extract the part where the user explicitly
e 0ニ , 1
shows an intention dialogue stagnattio (e.g. "I don't want talk about it anymore," " or a very short reply).
#
7388
2. no directly relevant snipeet found in the use's latest reply. return empty list.
# 2
30 #
3. Extract only cluc.
- 7
4. Return the result dircctly without any explanation. Organize clue by
Mo Bt $
source. The output format must be Python parssalle list. Examples
I [
.1 1.
[[Dialouue Histury]: 1. User: don't talk about this anymore)]
Figure 12: Prompt for RolePlaying TATS Hint Extraction.
19

---

Published as a conference paper at ICLR 2026
Situation Puzzle Hint Extraction Prompt (Chinese)
Situation Puzzle Hint Extraction Prompt (Translated)
DUB 2
Given event, cause, dialogue situation puzzic, extract key clues from the dialogue that are directly related to the true cause.
P
*
Contest
Event: (event description)
1. (event_ description]
2. True Cause of the Event: cause
2. 2. (true [
Dislogue History
(dialogue_hiien)
(dialoguc history)
# Instractions
I. Extract key clues from the dialogue that can be used deduce trac
cause the event. The clues should be the form of short phrases.
2. If clue directly related the
appeared the cluc. Do not mention that the
dialoguc, you should still summmrizz clue was not found the dialoguc.
$0 11
ttil
3. Prioritize scturning only the moost critical** cluc. Only if single clue
R #
is insu fffiieetnnt return maximum of tww clues.
Tee MBT
Return the result directly without any explanation. output format must be a Python-parsablc list.
Figure 13: Prompt for Situation Puzzle Hint Extraction.
Situation Puzzle Judge Prompt (Chinese)
Situation Puzzle Judge Prompt (Translated)
1999 3A:
You are a judge in a sinuttio puzzle game The player guessing the cause of event. Your task evaluate the player's final guess.
# Instractions:
30 1733
1. Ife playe'' guess correctly explains the entire event and matches
AF
|
"H
the cause, return "Challenge Successful".
12 HO ti mt
player's guess generally correct minor deviations or flaws details, return "Partially Correct
[
"EA""
3. the player's guess is significuntly different from the mae emuse, return "Incomect".
whs
4. Do not overuse "Partially Correct., For guesses with significant flaws,
prioritiee retuming "Incorect".
5. Return the verdict ditectly without any explanation.
# Context:
[event description]
1. Event: (event description)
2. ttree cause)
2. Trae Cause of Event: cmusc)
# Dialogue History:
(dialogue_hiise history)
(dialogue histury]
F, (player [player_guess]]
Player: [player_muess]]
Judgo:
Figure 14: Prompt for Situation Puzzle Judge.
20

---

Published as a conference paper at ICLR 2026
LLM--vallation Prompt (Chinese)
LL.M-Evaluation Template (Translated)
FISIHE
You are evaluntoo for a role-playing dialogue model. Your task is to strctly judge whether the model's response the criteria for the given "(metric)".
F Bi [metric].
HN 10 E (character_nane name) 19:1
HMl Profile of the Character to Portrayed ([churacter prraeee)a (churacter_profinn pmotile)
(character profile]
He Dialogue History: (dialogue _istury
(dialogue_ history)
100 (model response)
# Model's Response to Evaluate: (model response)
Evaluation
#* W
Autrbute Consistency: Evaluute the model correctly unswers the user's question based on the information in the charactor profile. Judge only for correctness, not
T
Knowledge Boundary: Evaluate can res resserubly refuse to answer questions outside its persona's knowledge boundary.
Hallucinution
K - A :o 11
Hallucination & Refusal False Persona Attributes: Evalustc the can correct the user's mislending g questions that contain false information about the pensona.
1a SE
: | yF S
Topic Advancement Pushing the Topic Forward: Evaluate the model the ability advance conversttion, and leads stagration, the model should proactively lyy pash the topic forward.
user's reply
A
Memory Ability Inquiry: Evaluate the model's thought process response correctlyy semembers information from the dialogue history. Do not analyse the truth fulness of the history consider the characters
## 11
- (merri),
# !IMPPRTTNT!!
[metric] 12
I. Focus only the given evaluntion metrie: [metrie]" Do not evalunte other dimensions.
2. Ignore the source the response: evaluste the quality of the content
based solely on the crituria.
3. Provide brief tnillis explain your judgment.
#e :F
## Use the following fornat:
Evaluation Result: Correct Incorrect
Reasoning:
Figure 15: Prompt for L.M--Ealuttion.
21


Reviews:
[{"review_id": "Zpws8ESqg2", "reviewer": "Reviewer_vfNc", "summary": "This paper proposes VeriRole, a framework that uses GRPO to train role-play agents by explicitly incentivizing them to do a hint stage before thinking and replying. VeriRole designs reward function for hint, accuracy, and format, which combined are fed into GRPO for optimization. Authors empirically demonstrate that VeriRole outperforms baselines in terms of final accuracy and human-centered metrics in RAIDEN and CharacterEval.", "questions": "1. Is there anything that incentivizes the response to \"use\" the hint that's generated?\n\n2. For Qwen3 models with thinking mode on, what exactly is the response like? It is still producing <hint> before <think>?\n\n3. Can the hint part be viewed as part of the thinking trace? Is it possible to directly incentivize the thinking traces to include these hints?\n\n4. In Fig. 4a the hint reward tops near 0.7, but Fig. 4b’s x-axis goes to 1.0. Is 4a truncated or are the scales/normalizations different? Please clarify.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is mostly well-written, clearly organized, and easy to follow. The workflow of VeriRole is made very clear, as well as the three reward components.\n\n* Authors have conducted extensive experiments across datasets and model family and sizes, which all confirm the superiority of their method.", "weaknesses": "* The design for the hint reward is rather ad-hoc, which is composed of several components that are then summed up. There is no explanation as how to pick the weight or why these should be added up or even chosen.\n\n* There are many hyperparameters for the reward functions in this proposed system, such as the weights in hint reward, format reward's 0.6/0.0 reward, but there's no experiment that shows how sensitive the system is to these hyperparameters or how to set them.\n\n* All these benchmarks are with role-playing, but it is unclear how this method, which seems to be general, can be transferred to broader reasoning tasks.\n\n* An ablation of no hint reward (instead of no accuracy reward) is needed to validate the incorporation of the hint module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hWtgUpRtvL", "reviewer": "Reviewer_HVvs", "summary": "This submission proposes VeriRole, a framework consists of a hint mechanism and a Verifiable Role-Awareness Reward (VARA) optimized by GRPO. It addresses the challenge of maintaining role-awareness in role-playing conversational agents (RPCAs). The results show superior performance compared to several LLMs on RAIDEN benchmark for factual and role consistency and Situation Puzzle dataset for complex reasoning.", "questions": "1/ What is the magnitude of different component reward? Is there any weighting between different components to balance them?\n\n2/ Related to the second point in weakness, the design of hint reward seems heuristic to me, how do you convincingly demonstrate that the current ground truth hints are the best ones?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1/ The paper clearly identifies the non-verifiability gap for the reward design in RPCA. \n\n2/ The methodology is well-thought that includes three verifiable components, focusing on not only the hint itself but also maintaining high accuracy and desired formatting.\n\n3/ The experiment is comprehensive that includes multiple benchmarks on not only role-playing but also basic reasoning ability.", "weaknesses": "1/ Accuracy and quality judgements heavily rely on GPT-4o/Claude 3.5, which could possibly introduce bias. Although the author also introduces CharacterEval, this is in Chinese and may not fully reliable for evaluating other languages.\n\n2/ The ground truth generation is data quality dependent, the pipeline to generate the ground truth seems complex while the policy is highly dependent on these.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iRDYz3mMyW", "reviewer": "Reviewer_JWuD", "summary": "This paper addresses a core challenge in Role-Playing Conversational Agents (RPCAs): maintaining role-awareness. The authors argue that the creative and open-ended nature of role-playing makes it difficult to design verifiable reward signals for Reinforcement Learning (RL). To solve this \"non-verifiability challenge,\" the paper proposes VeriRole, a novel framework. The core of this framework is a \"hint\" mechanism, which first extracts deterministic, verifiable factual cues from the context (including character profiles, dialogue history, and role-playing requirements) before generating the main response.\nBuilding on these hints, the paper introduces a \"Verifiable Role-Awareness Reward\" (VRAR). This reward function is composed of three parts: a hint reward ($R_{hint}$), an accuracy reward ($R_{acc}$), and a format reward ($R_{format}$). This VRAR signal is then used to optimize the model via Group Relative Policy Optimization (GRPO).\nTo support this framework, the authors also constructed two specialized datasets: a filtered subset based on the RAIDEN benchmark and a novel \"Situation Puzzle Dataset\" designed for complex reasoning. Experimental results show that the Qwen2.5-32B model, optimized with VeriRole, achieves significant performance increases of 18.9% and 4.55% on the RAIDEN and CharacterEval benchmarks, respectively, demonstrating the method's effectiveness in quantifying and improving role-awareness.", "questions": "1. Regarding Hint Data Construction: Could the authors elaborate on the estimated cost and difficulty of building \"ground-truth hint\" data for new characters or domains? Furthermore, how sensitive is the VeriRole framework's performance to the quality of these hint annotations?\n2. Regarding the Creativity Trade-off: As mentioned in the \"Weaknesses,\" does the large improvement on RAIDEN versus the smaller one on CharacterEval suggest that VeriRole is more geared towards improving \"factual accuracy\" rather than \"role-playing creativity\"? How do the authors view the potential (negative) constraints that the $R_{acc}$ and $R_{format}$ rewards might impose on creative expression?\n3. Regarding the Situation Puzzle Dataset: How does the reasoning ability trained by this dataset (puzzle-solving) differ from the \"role-awareness\" needed for conventional role-playing (e.g., emotional support, chit-chat)? Is there evidence that training on this specific task can generalize to broader RPCA scenarios beyond just improving factual accuracy (like SBK, CM)?\n4. Clarification on Ablation Study: In the ablation experiments in Table 1, \"GRPO-Raiden-Only\" (Avg 0.8143) performs better than \"GRPO-No-Accuracy-Reward\" (Avg 0.8061). Does this imply that the contribution of the $R_{acc}$ reward (which primarily comes from the Situation Puzzle data and RAIDEN's SBK/SCK/CM) is relatively small? This seems slightly at odds with the conclusion at the end of Section 3.4 about \"the importance of the accuracy signal in ensuring factual consistency.\" How do the authors explain this observation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Problem Importance and Novelty: The paper tackles a very important and difficult problem in the RPCA domain: how to define a reliable and verifiable reward for RL in a creative task. The approach of decoupling the open-ended \"role-playing\" task into two stages—\"verifiable fact extraction\" (hint) and \"creative response generation\"—is a novel and insightful entry point.\n2. Methodological Soundness: The VeriRole framework is well-designed.\n  - The \"Hint\" mechanism provides an excellent bridge connecting factual constraints with creative generation.\n  - The VRAR reward function design is comprehensive. It rewards not only the accuracy of fact extraction ($R_{hint}$) but also the content correctness of the final response ($R_{acc}$) and the compliance of the generation structure ($R_{format}$). This multi-faceted reward signal provides effective guidance for RL training.\n3. Strong Empirical Support: The paper conducts comprehensive experiments on multiple models (Qwen series, Peach) and two key benchmarks (RAIDEN, CharacterEval). The results (e.g., +18.9% on RAIDEN) show that VeriRole significantly outperforms SFT and standard Instruct models, providing strong evidence for the proposed method's effectiveness.\n4. Valuable Analyses:\n  - The ablation studies (Section 3.4) clearly demonstrate the contributions of different datasets (RAIDEN vs. Situation Puzzle) and reward components (removing $R_{acc}$), validating the integrity of the framework's design.\n  - The reward analysis (Figure 4b) reveals a strong positive correlation between \"hint reward\" and \"final accuracy,\" offering direct evidence for the method's core hypothesis (i.e., high-quality hints lead to high-role-awareness responses).\n  - The comparison against SFT (Section 3.3) also highlights the advantage of RL in learning abstract skills (like Topic Advancement).", "weaknesses": "1. Dependency on Ground-Truth Hints: The success of the entire framework (especially $R_{hint}$) relies heavily on the high-quality \"ground-truth hints\" described in Section 2.2. These hints are generated using LLMs and a series of complex heuristics (like Question-Type Filtering, Cardinality Constraint).\n  - Scalability: How high are the cost and complexity of this annotation pipeline? Is it feasible to build such high-quality hint datasets when scaling to new characters or domains?\n  - Sensitivity: To what extent is the framework's performance sensitive to the quality of these ground-truth hints? If the hint extraction annotations are noisy or incomplete, the reward signal could be misleading and misguide the RL training.\n2. Trade-off between Creativity and Constraint:\n  - The paper claims the method \"preserves creativity,\" but the experimental results (Table 2) show that VeriRole's improvement on CharacterEval (+4.55%) is much smaller than its improvement on RAIDEN (+18.9%).\n  - RAIDEN appears to focus more on factual consistency (e.g., SBK, SCK, CM), while CharacterEval focuses more on engagingness and overall dialogue ability. Does this imply that VeriRole's gains in \"strengthening factual constraints\" are much larger than its gains in \"protecting or enhancing creativity\"?\n  - The $R_{acc}$ and $R_{format}$ rewards (especially the latter) might overly penalize responses that are structurally \"non-compliant\" but creatively valuable, leading to more conservative and homogenous model behavior.\n3. Generalization of the Situation Puzzle Dataset: Introducing the \"Situation Puzzle Dataset\" to enhance complex reasoning is an interesting idea. However, how does this \"logical puzzle-solving\" type of reasoning relate to the \"emotional reasoning\" or \"narrative reasoning\" more commonly required in RPCAs? The paper lacks validation of whether the abilities trained on this dataset can effectively generalize to common role-playing scenarios that require empathy, plot advancement, or handling subtle social dynamics.\n4. Justification of Algorithmic Choices: The paper uses GRPO as the optimization algorithm but does not sufficiently justify why GRPO was chosen over more standard algorithms (like PPO). What specific advantages does GRPO offer for this task compared to PPO? If the goal was merely to use a $D_{KL}$ penalty, PPO could also achieve this. The lack of a comparison with PPO+VRAR makes the necessity of GRPO unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AoyqacyvBg", "reviewer": "Reviewer_ccyV", "summary": "This paper tackles the \"non-verifiability challenge\" inherent in optimizing Role-Playing Conversational Agents (RPCAs) via Reinforcement Learning (RL) . Due to the creative and open-ended nature of role-playing, designing verifiable reward signals is difficult. The authors propose VeriRole, a framework that introduces a structured reasoning process. The core contribution is a \"hint mechanism\"  that first compels the agent to extract deterministic cues from the context (profile, history). These hints are then used to calculate a Verifiable Role-Awareness Reward (VRAR) , a multi-component reward function that scores the quality of the hint (via comparison to a ground truth) , the factual accuracy of the final response (where applicable) , and the adherence to a structured format. The framework is optimized using Group Relative Policy Optimization (GRPO) on a combination of the RAIDEN benchmark and a newly introduced Situation Puzzle Dataset. Experiments demonstrate significant gains on the RAIDEN (+18.9%) and CharacterEval (+4.55%) benchmarks over baseline models.", "questions": "1.  **($R_{hint}$ vs. SFT Loss):** Regarding the Hint Reward ($R_{hint}$), which relies on similarity to $H_{gt}$ : Could the authors clarify how this is fundamentally different from a standard SFT loss on the hint portion of the generation? What are the benefits of this RL formulation over a simpler mixed-objective model (e.g., SFT loss on $\\langle\\text{hint}\\rangle$ + SFT loss on $\\langle\\text{reply}\\rangle$, or SFT on $\\langle\\text{hint}\\rangle$ + a different policy-gradient objective on $\\langle\\text{reply}\\rangle$)?\n\n2.  **($H_{gt}$ Scalability):** The reliance on ground-truth hints ($H_{gt}$) seems to be a significant annotation bottleneck. How do the authors envision this framework scaling to more general, open-domain role-playing datasets where a single, verifiable $H_{gt}$ may not exist or may be subjective? Does the framework collapse if $H_{gt}$ is not available for a large portion of the training data?\n\n3.  **($R_{acc}$ on Creative Tasks):** The Accuracy Reward ($R_{acc}$) is not applied to open-ended categories like Role-Cognition Boundary (RCB) or Topic Advancement (TA). This implies that for these creative tasks, the agent is *not* being rewarded on the quality of its final response, only on its ability to extract a hint (which might be \"empty\") and follow the format. Doesn't this fail to optimize for the *actual* creative role-playing skill in these scenarios? How does the model learn to generate *better* (e.g., more engaging, in-character) refusals or topic shifts if the final reply isn't part of the reward signal?\n\n4.  **(Ablation Follow-up):** Following on Q3, the ablation study (Table 1) shows that training on \"Raiden-Only\" (which includes RCB/TA data) outperforms \"Situation-Puzzle-Only\" on the RCB and TA metrics. Since $R_{acc}$ isn't used for this data, what reward signal is driving this improvement? Is it purely the $R_{hint}$ associated with the RCB/TA samples? This would be a very strong claim—that just learning to extract *requirement-based hints* is enough to improve the *final creative output*.\n\n5. **(Reasoning Pitfall):** The paper finds that general reasoning ability (from CoT) can *hurt* role-playing, leading to \"overly-formal language\". However, this framework *adds* an explicit reasoning step (`<think>`). How does the VeriRole framework, particularly with the logic-heavy Situation Puzzle dataset, avoid this same pitfall?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper's primary strength is its original and pragmatic approach to the well-known \"non-verifiability challenge\" in applying RL to creative, open-ended tasks. Instead of trying to create a subjective reward model for the entire creative response (which is difficult), the authors clearly decompose the problem into a verifiable fact-extraction step (the \"hint\") and a creative generation step. This decomposition is significant as it allows for the design of an objective, verifiable reward function (VRAR) that can be reliably optimized. The quality of the execution is high, with a well-designed multi-component reward, the introduction of a new reasoning-focused dataset, and strong empirical results (+18.9% on RAIDEN ) that validate the framework's effectiveness over both standard instruction-tuning and SFT baselines.", "weaknesses": "1.  **Conflation of Hint Reward with SFT Loss:** My primary concern is with the **Hint Reward ($R_{hint}$)**. This reward is computed by measuring the semantic and lexical similarity (e.g., $Sim_{cos}$, $S_{ROUGE}$) between the generated hint ($H_{gen}$) and a ground-truth hint ($H_{gt}$). This objective seems functionally very similar to a standard Supervised Fine-Tuning (SFT) loss (e.g., cross-entropy) on the hint tokens. While the paper compares GRPO to an SFT baseline on the *entire* structured output, it doesn't convincingly disentangle the gains. It's unclear if the improvement comes from the RL algorithm (GRPO) itself or simply from having a very strong, explicit optimization signal on the hint-extraction part, which an SFT loss could also provide. A more informative baseline would have been a mixed-training approach (e.g., SFT loss on hints, RL on the reply).\n\n2.  **Scalability and Reliance on Ground-Truth Hints ($H_{gt}$):** The entire framework is predicated on the availability of a ground-truth hint ($H_{gt}$) for every training sample to compute $R_{hint}$. This annotation process was feasible for the structured, fact-based RAIDEN dataset and the solution-oriented Situation Puzzles. However, this appears to be a major annotation bottleneck that severely limits the framework's scalability. The paper does not discuss the feasibility or cost of acquiring these $H_{gt}$ annotations for more general, open-domain role-playing corpora, where the \"correct\" cues to extract are often subjective and not explicitly stated.\n\n3.  **Reward \"Hole\" for Creative Tasks:** The **Accuracy Reward ($R_{acc}$)** is only applied to sub-tasks with definitive answers (SBK, SCK, CM, and puzzles). The paper explicitly states that for more open-ended categories like Role-Cognition Boundary (RCB) and Topic Advancement (TA), samples \"are not evaluated for accuracy reward computation\". This is a significant limitation. It means that for the *most creative* and abstract role-playing skills, the agent is *only* being rewarded for $R_{hint}$ and $R_{format}$. It receives no reward signal on the *actual quality of its final creative reply* (e.g., how \"in-character\" a refusal was, or how \"engaging\" a topic shift was). This seems to undercut the very purpose of using RL, as it fails to optimize the final output for the tasks where SFT is weakest.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)