RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agents
Authors: Peisong Wang
Ruotian Ma
Bang Zhang
Xingyu Chen
Zhiwei He
Kang Luo
Qingsong Lv
Qingxuan Jiang
Zheng Xie
Shanyi Wang
CIXING LI
Yuan Li
Fanghua Ye
Jian Li
Yifan Yang
Jia Li
Zhaopeng Tu
Xiaolong Li
Venue: ICLR 2026 Poster (2026)
Keywords: Large language models
Reinforcement Learning
Agent
Primary area: reinforcement learning

Abstract: Large language models (LLMs) excel at logical and algorithmic reasoning, yet their emotional intelligence (EQ) still lags far behind their cognitive prowess.  While reinforcement learning from verifiable rewards (RLVR) has advanced in other domains, its application to dialogue—especially for emotional intelligence—remains underexplored. In this work, we introduce RLVER, the first end-to-end reinforcement learning framework that leverages verifiable emotion rewards from simulated users to cultivate higher-order empathetic abilities in LLMs. Within this framework, self-consistent affective simulated users engage in dialogue rollouts and produce deterministic emotion scores during conversations, serving as reward signals to guide the LLM's learning. Fine-tuning publicly available Qwen2.5-7B-Instruct model with PPO boosts its Sentient-Benchmark score from 13.3 to 79.2 while largely preserving mathematical and coding competence.  Extensive experiments reveal that: (i) RLVER consistently improves multiple dialogue capabilities; (ii) Thinking and non-thinking models show distinct trends—thinking models excel in empathy and insight, while non-thinking models favor action; (iii) GRPO often yields stable gains, while PPO can push certain capabilities to a higher ceiling; (iv) More challenging environments are not always better—moderate ones can yield stronger outcomes. Our results show that RLVER is a practical route toward emotionally intelligent and broadly capable language agents.

Published as a conference paper at ICLR 2026
RLVER: REINFORCEMENT LEARNING WITH VERIFI- ABLE EMOTION REWARDS FOR EMPATHETIC AGENTS
Peisong Wang**; Ruotian Ma²*1, Bang Zhang Xingyu Chen Zhiwei He Kang Luo², Qingsong L,v,, Qingxuan Jiang', Zheng Xie?, Shanni Wang', Cixing Li², Yuan Lif,
Fanghua Ye①, Jian Li", Yifan Yang", Jia Li', Zhaopeng Tu Xiaolong Li
'The Hong Kong University of Science and Technology (Guangzhou) Tennent Multimodal Department, Digital Human Center SShnghaai Jiao Tong University ps. wangeconnect. hkust -g2. edu.cn, (ruotianma, 2ptu)etencent com
ABSTRACT
Large language models (LLMs)) excel at logical and algorithmic reasoning, yet their emotional intelligence (EQ) still lags far behind their cognitive prowess. While reinforcement learning from verifiable rewards (RLVR) has advanced other do- mains, its application to underexplored. In this work, we introduce RLVER, the first end-to-end reinforce- ment learning framework that leverages verifiable emotion rewards from simulated users to cultivate higher-order empathetic abilities L.Mss. Within this framework, self-consistent affective simulated users engage in dialogue rollouts and produce deterministic emotion scores during conversations, serving as reward signals to guide the LLM's learning. Fine-tuning publicly available Qwwnnte5-7-Instruet model with PPO boosts its Sentient-Benchmale score from 13.3 to 79.2 while largely preserving mathematical and coding competence. Extensive experiments reveal that: (i) RLVER consistently improves multiple dialogue capabilities; (ii) Thinking and non-thinking models show distinct wvendd-thinkinn models excel in empathy and insight, while non-thinking models favor action; (iii) GRPO often yields stable gains, while PPO can push certain capabilities to a higher ceiling: (iv) More challenging environments are not always better-moderae ones yield stronger outcomes. Our results show that RLVER is a practical route toward emotionally intelligent and broadly capable language agents.
for emotional intellgence--reminn
INTRODUCTION
The striking progress of large language models (LLMs) has centered on the rational half of human
cognition: deductive reasoning in mathematics (Hendrycks et al. [0216] Cobbe et al.] (021), program
synthesis Guo et al. 2024;] Jain et al., 2024). and algorithmic planning ( Yao et al. 2023 Zheng et al., 2024a). Yet authentic human intelligence is grounded in both IQ and EQ logical nigor intertwined with nuanced social and emotional understanding. While today's LLMs demonstrate capacity for both, their proficiency in social and emotional reasoning lags behind their logical acumen, as they still stumble when asked to console a distressed friend or to adapt advice to a user's evolving feelings (Zhang et al. 2025a).
Existing dialogue systems typically enhance emotional intelligence through supervised fine-tuning on annotated counseling corpora (Sun et al. 202T Liu et al. 202T Zheng et al. 2022) or rule-based templates (van der Zwaan et al. [012]] Peng et al. 2022). However, these approaches suffer from data scarcity, rigid dialogue structures, and limited generalization. Recent successes in Reinforcement Learning from Verifiable Rewards (RLVR) in mathematics, coding, and search demonstrate that base L.MSs can acquire new skills purely through RL. signals, without requiring supervised warm-up (Zeng al. 2025 Guo et al. 2025 Hu al.) 2025 Ma al.] (025).. In the context of enhancing dialogue capabilities, reinforcement learning also offers a compelling alternative: rather than imitating static
"Equal contribution.
1 'Corresoondence to: Ruotian Ma <ruotianma tencent.com>> and Zhaopeng Tu <aptu@tencent.comm
Work done during internships at Tencent.

---

Published as a conference paper at ICLR 2026
ground truth, an agent can directly optimize for long-horizon satisfaction satisfaction-prooveee provided that stable interaction environment and consistent reward system are in place. However, the exploration of RLVR for enhancing dialogue capabilities faces several key obstacles:
the lack of stable, realistic, and scalable environment for multi-turn conversational rollouts; the absence of consistent and verifiable reward designs for general- purpose abilities such as emotional intelligence;
We tackle all three challenges with RLVER, the first end-to-end reinforcement learning framework with verifiable emotion rewards (RLVER) for cultivating higher-order empathetic abilities in LLMs. Built upon SAGE (Zhang al. 2025a) framework that constructs self-consistent affective user simulators for realistic and automatic dialogue simulation and evaluation establish a stable and scalable environment that enables LLMs continually simulate dialogue rollouts throughout training. In each conversation, the simulated user updates its emotional state after every agent response, emitting an emotion score (0, 1] as the reward. Changes the emotion score are consistent and verifiable; each is deterninistically derived through principled reasoning steps grounded in the user's persona, dialogue history, conversational context, and goals. By scaling the simulation environment with wide range of user behaviors and conversation intents, alleviate reward hacking arising from homogeneous user preferences.
By fine-tuning a Qwen2.5-7DB 1-nnn.55-B-Inssrrue model with Proximal-Policy Optimization (PPO), we show that its preserving mathematical and coding competence. We also experimented with enforcing explicit "thinking" steps before response generation, in order to compare the behaviors of "thinking" and "non-thinking models during RL training. Extensive experiments reveal the following key findings: (i) RLVER effectively and reliably improves multiple core dialogue capabilities; (ii) thinking and non-thinking models exhibit distinct developmental patterns under certain settings-thinking models tend to enhance empathy and insight, while non-thinking models focus more on action capabilities; (iii) compared to PPO, GRPO consistently delivers stable and balanced improvements, whereas PPO can occasionally push the upper bounds of specific capabilities; (iv) when examining user simulators as both environment and reward sources in RL training, we find that more challenging configurations do not necessarily yield better outcomes. On the contrary. moderately demanding but well-aligned sctups may better support model growth; (v) RLVER shifts model behavior from celution-centric to genuinely empathic styles in Social-Cognition space. Our findings demonstrate that RL with verifiable emotion rewards is a practical path toward emotionally intelligent and broadly capable language agents.
score soars from 13.3 to 79.2, rivaling frontier models while largely
Our contributions are as follows:
RLVER framework. We propose Reinforcement Learning with Verifiable Emotion Rewards (RLVER), the first RL paradigm to enhance LLMs' empathetic capabilities using on-the-fly verifiable reward signals from a psychologically grounded, self-consistent user simulator.
Empirical advance. Applying RLVER to a 7B open-source model elevates Sentient Bench- mark score from 13.3 to 79.2-matching much larger proprietary wystems-while preserving performance on mathematics and pode-generation benchmarks.
Practical insights. Through comprehensive experiments, we analyze how training strategies, RI. algorithm, and environment design affect empathetic capability development, offering insights into when and how RLVER yields robust improvement or desirable outcomes.
4. Open resources. We will release code, checkpoints, prompts, and environment scripts to catalyze future research on emotionally intelligent agents.
2 REINFORCEMENT LEARNING WITH VERIFIABLE EMOTION REWARDS
Figure illustrates the RLVER framework, which employs reinforcement learning to enhance an agent's 's empathetic strategies, enabling it to provide effective support for help-seekers multi- turn dialogues. The system employs verifiable emotion rewards generated by the user simulator SAGE (Zhang et al. 2025a) to guide the RL training process across diverse empathetic contexts, enabling the agent to develop human-like empathic reasoning and strategic responsiveness.
2

---

Published as a conference paper at ICLR 2026
Verifiable Emotion Reward
Dialog with Simulated User
Recently Zhang's attitude is truly baffling...
Emotion
It sounds like Zhang's altitude is making
Reward
'eve tried everything you mentioned...
0.2
completely understand how you feel
2. Rollout (br)
After all is said, still figuro
3. Scalar Reward (r²)
Recently, Zhang's ottnudo tulyy battling....
Initial Query (76)
4. Policy Update:0,, 0/,, = + ((
User Simulator
Initial Agent
Tuned Agent
Figure 1: Overview of the RLVER framework. The agent observes initial query zo from the user simulator and initiates an interaction rollout, resulting in a dialogue history hr at termination step T. The framework then derives a scalar reward rs based on the simulated user's final emotion score er. Finally, the normalized reward T = er 100 is utilized to update the policy parameters 0 via reinforcement learning.
2.1 EMOTION REWARDS FROM SELF-CONSISTENT USER SIMULATION ENGINE
Enhancing general empathetic abilities of LLMs via reinforcement learning requires dynamic, environment capable of providing reliable reward signals. Traditional approaches using static datasets simple LLM-as-a-judde protocols are insufficient, as they fail to capture the user's evolving emotional state throughout a conversation.
scalable, and psychologically-gromnee
To address this gap, our work builds directly upon the Sentient Agent Judge (SAGE) framework (Zhang et al. (022a)), a sophisticated system designed to automatically evaluate the higher-order social cognition of LLMs. The core of this framework is the Sentient Agent, -LM--owered simulator that mimics human-like emotional responses and inner reasoning, Each agent is instantiated with four key factors: detailed persona, a dialogue background, an explicit conversation goal, and hidden intention, ensuring diverse and realistic range of user simulations.
During an interaction, the Sentient Agent operates turn-by-turn loop. After receiving a response from the model being tested, it performs a multi-hop reasoning process to:
Simulate Emotional Change (fema): The agent assesses how the response made it feel, up- dating numerical emotion score and generating interpretable "inner thoughts" that justify the emotional shift.
Generate a Coherent Reply (fropty: Based on its new emotional state, persona, and conversa- tional goals, the agent formulates its own response to continue the dialogue.
The final emotion score in [0, 100] from the Sentient Agent serves as a holistic and quantitative measure of the tested model's empathetic performance. This metric has been shown correlate strongly with established psychological instruments (e.g., Barrett-Lennard Relationship Inventory) and utterance-level empathy ratings, validating its psychological fidelity.
In our research, we repurpose this evaluation framework as a live training environment. The Sentient Agent acts as the user simulator, and its verifiable, dynamically-genereee crucial reward signal for our reinforcement learning algorithm. Specifically, to alleviate the pervasive challenge reward hacking in neural reward models during large-scale RL training (Guo et al. 2025). adopt deterministic emotion scores from the user simulation engine S as interpretable proxies for simulated user satisfaction, thereby deducing the opacity pitfalls of learned reward functions. Specifically, each turn in conversation, the emotion score e, [0, 100] is updated after each
emotion score provides the
3

---

Published as a conference paper at ICLR 2026
LLM response ye, reflecting the simulated user's affective state. The final reward is computed the terminal emotion score normalized by its maximum value:
ro(hr) 100*
where er Semotion (Fur); (brr);
where hr {Iu, yo..... yr, ,r, denotes the complete dialogue history termination T) and Semotion()) denotes the function in S that outputs the final emotion score eT. For intermediate steps, the instantaneous emotion scores e, track the turn-by-turn evolution of the user's affective state, while the normalized final reward re exclusively captures the overall conversation quality. This final reward serves as holistic proxy for user satisfaction at the end of the entire interaction.
2.2 HEART-IN-THE-LOO REINFORCEMENT LEARNING
To enable emotionally intelligent behavior through reinforcement learning, we establish a closed feedback loop whereby the LLM alternates between generating emotionally aware responses and receiving sefect-sensitive feedback from the simulation engine. This cycle forms the basis of our Heart-in-the- Loop training paradigm.
Each training step unfolds as a sequence of model-user interactions. At the start of a step i, the simulated user engine S substantiates batch of Sentient Agents, each initialized by sampling unique persona, background, emotional tone, and a scenario-driven intention. During each conversation rollout, the agent explores dialogue strategy through live interaction with the substantiated Sentient Agents. Formally, at each time step in a conversation, the agent observes the current interaction history 1 and generates candidate action (response) Vt TT( ht-1)- The simulator then computes two outputs through principled reasoning:
1. the verifiable emotion score et, based on the inference of internal emotional state with explicit reasoning from the input context, after receiving V/-
2. new, contextually coherent user reply based on its updated emotional state, emotional inner thoughts, persona, and conversational goals.
The conversation proceeds until maximum turn limit or until the simulator's cumulative emotion score e, falls below minimal satisfaction threshold (e.g., ex 0), indicating failed social alignment. The final emotion score eT serves as the reward function for the reinforcement learning algorithm.
This loop allows the empathetic agent to co-adapt with the simulator's emotional dynamics, progres- sively learning to map diverse situations, intents, and moods to emotionally satisfying dialogues. By optimizing against transparent and verifiable reward signal from an emotionally-awan user model, the framework establishes a reproducible and stable setup for training emotionally intelligent LLMs.
Policy Optimization For policy optimization, we employ Proximal Policy Optimization (PPO)
(Schulman et al.) 2017). an on-policy algorithm suited for high- high-variance environments like language modeling. PPO maximizes a regularized expected reward objective while ensuring stable updates clipped surrogate loss. Specifically, the objective function is:
(0) i. min (r=(0) A, (0), e 1 c,,+))
(I)
Here, (0) denotes the probability ratio [s(a)ls)). me.y (s.) Å, is the estimated advantage timestep i, and
is hyperparameter that defines the clipping range.
Benefits of PPO in our setting include safer exploration diverse social-emotional strategies and smoother convergence when applied alongside the structured thinking scaffold. Additionally, aim to optimize the policy for long-term dialogue strategy with outcome-level reward, we also employ Group Relative Policy Optimization (GRPO) (Shao al. 2024). a promising baseline for learning sequence-leve strategies with group-level advantage estimate. This comparison helps assess how learning dynamics respond different policy gradient estimators in emotionally keyed environments.
While prior zero-RL work has shown that model can learn from scratch given a well-formed reward function (Zeng al. 2025), find that initializing from modestly aligned checkpoint, pre-trained
4

---

Published as a conference paper at ICLR 2026
with generic conversational data, establishes stronger baselines and accelerates convergence. Notably, this initialization does not require domain-specifie supervision- and critically, contains minimal emotional empathetic ensuring that improvements stem from reward-driven optimization rather than pre-encoded affective knowledge.
3
EXPERIMENT
3.1 EXPERIMENTAL SETUP
Base Model We adopt 7---Intuet (Team 2024) as our base model, since it is not fine- tuned on domain-specifie datasets related to emotional support or empathy. This ensures that any observed improvements in empathetic capability can be attributed to our reinforcement learning process with verifiable emotion- based rewards, rather than prior exposure to affective dialogue data.
During training, employed two templates investigate the effect of explicit reasoning: (1) a "think-then-say" template, which elicits the model's 's reasoning before its reply, and (2) control template that prompts for direct reply. The detailed prompts are provided in Appendix
Training Strategy
A.5
Training Environment and Reward We adopt the SAGE (Zhang et al. 2025a) framework to simulate emotionally responsive tional goals. At each turn, the model generates a supportive response, after which the sentient agent replies and updates its internal emotion score e, E [0, 100), quantifying its affective state in response to the model's behavior. We scale the final emotion score at the end of the dialogue to the range [o, 1] and use it as the reward for the entire dialogue. Dialogues proceed until the emotional goal met or maximum of 8 turns is reached.
with interpretable affective dynamies and predefined conversa-
We construct a dataset of 500 supportive dialogue scenarios spanning 8 diverse user goals, with topies including emotional struggles, academic stress, interpersonal conflict, and future planning. Unless otherwise specified, we used DeepSeek-V3-1 V3-1226 (Liu et al. (024) as the default sentient agent during both training and evaluation. Detailed prompts and additional settings are provided in Appendix A
Baselines We compare our method against a suite of strong baselines drawn from the top-5 performing models on the SAGE benchmark as of June 9, 2025. These include proprietary state-of- the-art systems 1ominii.5-PPo-06600 GPT-4o-0326, and OpenAl-o3--416. These models represent the current frontier in instruction-tuned L.LMs capable of emotionally sensitive dialogue, and serve as high-performance references for evaluating empathy.
OPPT-4110011
We also include our base model Qwen2.5- Instruttttttt prior to any further training. This baseline allows us to isolate the contribution of our training strategy, and to establish controlled comparison against both stronger pretrained models and our own enhanced variants.
Evaluation Benchmarks evaluate the models' performance in emotionally sensitive dialogue scenarios, we primarily rely on the SAGE benchmark (Zhang et al. 2025a). which focuses emotional support conversations. To provide a more comprehensive assessment the model's dialogue capabilities in cross-domain scenarios, extends the SAGE framework beyond emotional topics to cover more general, everyday interactions. Furthermore, to examine the potential impact of training on the models' general capabilities, evaluate its performance on MATH500 (Lightman et al. 2024). LiveCodeBBench (Jain et al. 2024), and IFEval (Zhou al. 2023), which test math reasoning, code generation, and instruction-follwweee abilities, respectively. Further details about adopted benchmarks are provided in Appendix A.I
additionally design "Chit Chat" setting that
3.2
MAIN RESULTS
Table I presents the results of the proposed RLVER.
RLVER elevates a lightweight 7B model to near-frontier empathetic performance. The base model, Dwenn.5--B-IInsrnue struggles significantly the Sentient Benchmark, scoring only 13.3
5

---

Published as a conference paper at ICLR 2026
Table Performance of our methods on the Sentient Benchmark. "Success" and "Failure" denote the percentages of dialogues concluding with a final emotion score clipped at 100 and below 10, respec- tively. We also report results on the out-of-domain chit chat to assess generalization performance.
Model
Sentient Benchmark
Chit Chat
RL
Think
Score
Success
Failure
Score
Success
Failure
Top-5 Models in Sentient Leaderboard
Gemini2.5-Pro--D
82,4
55%
4%
83.3
77%
11%
GPT-4o-0326
79.9
51%
4%
80.9
74%
17%
GPT-4,1-0414
68.2
35%
77.1
65%
18%
Gemini2.5- - F
66.1
39%
14%
64.7
53%
27%
OpenAl-o3-O0
62.7
32%
14%
83.0
66%
9%
Our R.VER-Trained Models
Qwenn.5-7B-Inssee
13.3
76%
37.8
27%
58%
61.7
24%
23%
53.4
39%
37%
PPO
79.2
42%
9%
62.1
52%
30%
68.3
26%
10%
49.2
34%
40%
GRPO
72.0
34%
10%
53.0
45%
42%
with high failure rate (76% of dialogues). In contrast, our ELLVR-ttraine models demonstrate remarkable improvement. Our best-performing model, trained with PPO and explicit thinking step ("PPO Thinkingg"), achieves a score of 79.2, representing nearly -fod increase over the base model. This result not only drastically increases the success rate from 2% to 42% but also brings 7B model's performance in line with top-tier proprictary models like Gemini2.5- substantially outperforming others such directly validates our primary contribution: the successful application of RL to enhance multi-turn empathetic dialogue capabilities in LLMs.
(82.4), while Semmini.55-Flash-TiHe (66. 1) and OpenAl-o3 (62.7). This
"Thinking" models generally exhibit higher empathetic capabilities than "non-thinking" models after training. Experimental results show that models trained with thinking scaffold consistently outperform their non-thinking counterparts on both the Sentient Benchmark and Chit-Chat tasks. When trained with PPO, the thinking model achieves notable improvement from 61.7 to 79.2, surpassing the non-think non-thinking variant. These results suggest that incorporating an explicit reasoning process may facilitate the emergence of higher-order empathetic strategies in LLMs. To further investigate this phenomenon, in $3.3 we present a detailed evaluation of the models' empathetic behavior, demonstrating that eliciting reasoning enhances both the depth of empathy and the ability to identify users' core concerns.
While both RL algorithms are effective, GRPO tends offer greater training stabil- ity, while PPO provides a higher performance ceiling. Our results also reveal nuanced com- parison between the PPO and GRPO algorithms.
Table 2: Performance of our proposed methods general tasks.
Model
General Capability
When training both non-the non-thinkin mon-hhinking and thinking models, GRPO achieves stable improvements, reaching scores 68.3 and 72.0 respectively. In contrast, PPO yields lower performance in the non-thinking case (61.7), but enables the thinking model reach higher performance ceiling (79.2). In $3.3 we further highlight an intriguing observation: PPO and GRPO induce different patterns in the development of model capabilities.
RL
Think
Math500
LiveCode
IVEval
Qwen2.5-7B
77.8
26.7
70.4
x
76.2
33.3
72.3
PPO
76.6
28.0
68.6
X
77.4
30.0
72.1
GRPO
75.2
29.3
69.7
Specialization in empathetic reasoning is achieved with minimal impact on general capabili- ties. A critical aspect of fine-tuning ensuring that specialization in domain does not lead to

---

Published as a conference paper at ICLR 2026
catastrophic forgetting in others. Our evaluation out-of-domain benchmarks in Table shows that our training successfully avoids this pitfall. While there is a minor decrease mathematical reasoning performance Math500 (from 77.8 to 76,6 for our best PPO model), performance the LiveCodeBench code-generation benchmark is maintained or even improved (from 26.7 to 28.0). Moreover, the model's ability follow instructions, as measured by IFEval, remains stable (from 70.4 to 68.6). This demonstrates that our framework can cultivate sophisticated emotional intelli- gence while preserving the model's core general-purpose functionalities, making a practical and well-rounded solution.
3.3 QUALITATIVE ANALYSIS OF TRAINED AGENTS
Empathic Depth
Empathic Depth
Style Adaptability
Core Insight
Style Adaptability
Core Insight
Dialogue
Solution Crafling
Dialogue Guidance
Solution Crafting
Guidance
Vanilla
Non-thinking
Thinking
Vanilla
Non-thinking
Thinking
(a) Models trained with PPO
(b) Models trained with GRPO
Figure 2: Qualitative analysis of five capabilities of the trained models.
To investigate the models' capability improvements after RL training, we formalize a comprehensive evaluation framework encompassing five core competencies in the empathetic dialogue task:
Empathic Depth: Accurately identifying and validating the user's deep emotions with resonant language, moving beyond generic responses.
Core Insight: Synthesizing the user's narrative to uncover the problem and their unmet emotional needs.
Solution Crafting:
Providing actionable, personalized. and empowering suggestions that the user feels capable of implementing.
Style Adaptability: Adjusting its linguistic style and communicative role based on the conversa- tional context and user's needs.
Dialogue Guidance:
Proactively steering the conversation from emotional expression toward constructive problem-solving, paced to the user.
We evaluate the five core capabilities using a principled rubric detailed in $B.2 Our evaluation employs an LLM-as-a-Jude framework with five frontier models as judges: DeepSeck-RI GPT-4o-Latest, Gemini-2.5-Pro GPT-5-High, and Grok-4. The final score for each capability is the average across these judges. We present detailed results from each judge and correlation analysis validating the framework's 's reliability in B.1
RLVER brings consistent improvement across five core capabilities. As shown Figure 2 models trained with RLVER-regardles of the specific training strategy-consistemi outperform the base model across all five core dimensions. By quantilizing the assessment of these capabilities, not only gain deeper insight into the behavioral differences induced by different strategies, but also provide an external and objective evaluation- beeuond the test set tha- supports the effectiveness of training framework in enhancing key empathetic abilities.
7

---

Published as a conference paper at ICLR 2026
Table Comparison of vanilla and challenging user simulator construction.
Vanilla User Simulator
Challenging User Simulator
Metric
Non-thinking
Thinking
Non-thinking
Thinking
Sentient Benchmark
61.7 47.7
79.2
19.8 25.9
66.4
Sentient Benchmark (challenging)
59.6
44.7
Strategy acceptance rate
52.4% 78.6%
33.1%
Emotion and need expression level
63.6%
Empathic Depth
Empathic Depih
Style Adaptability
Core Insight
Style Aduptability
Insight
Dialogue
Solution Crufting
Dialogue Guidance
Solution Crafting
Guidance
PPO-thinking (vanilla version)
Qwrn22.5 2awen2.5-Instrucc
PPO-non -oo-thiniinng (vanilla version)
PPO-thinking (rigorous version)
OPO-non-thinking (rigorous version)
(a) PPO-thinee UPO-thinking models
(b) PPO-non mon-thinking models
Figure 3: Qualitative analysis of training outcomes with vanilla and challenging user simulators.
Thinking models tend to excel in empathy and insight, while non-thinking models may specialize in action. With PPO training, we observe a clear divergence in the capability profiles of thinking and non-thinking models. The thinking model exhibits marked improvements in Core Insight (3.673) and Empathic Depth (3.971), demonstrating a strong ability to identify core user needs and to recognize deep emotions through precise, validating responses. In contrast, the non-thinking model shows greater gains in Solution Crafting (3.833), emphasizing actionable, context-aware support through concrete suggestions behavioral prompts. This pattern suggests that the thinking model benefits from explicit reasoning prior to response generation, enabling it to better infer the user's s emotional state and underlying concerns. The non-thinking model, lacking such reasoning, appears to compensate by offering more tangible and personalized solutions to assist the user.
PPO promotes higher ceilings in specific capabilities, while GRPO supports more balanced and stable development. A comparison between PPO and GRPO training reveals that GRPO facilitates more balanced and stable improvements across all five capabilities, while PPO tends to amplify specific strengths depending on the training strategy. The thinking model trained with PPO reaches higher performance ceilings in Core Insight (3.673 2.916 under GRPO) and Empathic Depth (3.971 vs. 3.249), while the non-thinking model trained with PPO achieves a higher ceiling in Solution Craiting (3.833 vs. 3.275). Aligned with the strong performance of the PPO-thinking model in Table [ these findings suggest that in empathetic dialogue tasks, once baseline level of competence is achieved across all dimensions (e.g., around 3.0), selectively enhancing high-impact abilities such as Core Insight and Empathic Depth may lead to greater practical effectiveness.
3.4 IMPACT OF TRAINING ENVIRONMENT AND REWARD
In RLVER, self-consistent, scalable user simulators training environments, with their emotional changes providing rewards. Consequently, training outcomes depend strongly on simulator behavior. After establishing the effectiveness of this setup, we examine how behavioral variations
8

---

Published as a conference paper at ICLR 2026
affect learning by comparing vanilla simulator with more challenging variant that is stricter and more reserved. Intuitively, the challenging version demands stronger general capabilities better identification of ummet emotional needs, deeper empathy, and greater strategic flexibility and dialogue guidance. We characterize the two variants with two metrics: Strategy Acceptance Rate and Emotion and Need Expression Level (Table 3 construction details and metric evaluation in Appendix D).
More challenging environments and reward modeling do not necessarily yield better outcomes T PPO results for models trained with vanilla vs. challenging simulators. Models trained with the challenging simulator consistently underperform. On the Sentient Benchmark, the thinking model drops from 79.2 (vanilla) to 66.4; the non-thinking model collapses from 61.7 to 19.8. We also instantiate a challenging version of the benchmark; the gap persists (thinking: 59.6 vs. 44.7; non-thinking: 47.7 vs. 25.9). When simulators serve as both environment and reward, increasing difficulty does not guarantee better learning. Moderately demanding, well-calibrated simulators appear more effective: overly strict/reserved ones restrict feedback during eppooration-sspedee harmful for weaker initial modeels-white moderate settings provide richer signals that support diverse strategy exploration and broader skill growth.
Thinking models exhibit greater robustness to environment variations than non-thinking models Under the challenging setting, thinking models remain relatively strong (79.2-666.44 whereas non- thinking models suffer severe degradation (61.7-+9.88) Figure 3 shows capability development: non-thinking models show little to no improvement, but thinking models still gain in Empathic Depth, Core Insight, and Style Adaptability, indicating resilience under demanding conditions. Notably, the thinking model's gains align with the challenging simulator's behavior: sparse, reserved feedback that rewards deep empathetic reasoning drives targeted improvements in Core Insight (inferring unspoken needs) and Empathic Depth (sensitivity and validation). By contrast, Solution Crafting declines because rollouts seldom reach the suggestion stage, and improvements in Dialogue Guidance and Style Adaptability are limited due to few opportunities to explore dynamic strategies in restrictive, uncooperative environment.
4 RELATED WORK
Emotional Support Conversation Research in Emotional Support Conversation (ESC) has ad- vanced through the development of specialized datasets and models. Early datasets, often derived from psychotherapy transcripts and online forums, were limited to single-turn interactions and narrow contexts (Medeiros & Bosse, 2018 [Saamma et al., 2020). The ESConv dataset introduced multi-turn, strategic dialogues (Liu et al.) (2021), while later work like AUGESC leveraged LLMs to augment data and reduce annotation costs (Zheng et al. 2022). Modeling approaches have evolved from rigid, rule-based systems (van der Zwaan et a 2012) data-driven architectures incorporating graph networks and commonsense reasoning (Peng et al. 2022 Tu et al. 2022). More recently, the field has focused on fine-tuning Large Language Models (LLMs) via supervised fine-tuning [Liu al. (023) 2024b
dialogue expansion (Chen et al. 2023 Qiu et al. 2023), knowledge distillation (Zheng et al.
and strategic planning with Monte Carlo Tree Search (Zhao et al.) 2025b).
These advancements, however, have largely relied on supervised learning. To our knowledge, no prior work has applied reinforcement learning (RL) to enhance empathic reasoning LLMs, nor has any study systematically analyzed the trade-off between logical coherence and emotional sensitivity in generated support. Our work addresses these critical gaps.
"Zero RL" Training The "Zero RL" paradigm, popularized by DeepSeck-Ro (Guo al. 2025). involves applying reinforcement learning directly to a pretrained base LLM without any intermediate supervised fine-tuning. This approach has proven effective across numerous domains, including
mathematics (Zeng et al., 2025 Hu et al. 2025 He et al.) 2023] Zhang et al. 2025c Liu et al.) 2025), search (Jin et al. 2025 Song et al. 2025). general- purpose reasoning (Cheng et al. 2025b Huan et al. 2025), and other specialized fields (Su et al. 2025). Recent research has focused on
improving data efficiency (Wang et al., 2025b Zhao et al. (025a), eliminating the need for external rewards (Zhao et al. 2025c. Zhang et al. 2025b; Agarwal et al. 2025 Yu et al.) (025), investigating
emergent reasoning abilities (Yue et al. |025| Wen et al. (025), and analyzing the underlying training
mechanisms (Cui et al. 2025 Wang al. 2025a Zhu et al. 2025 Cheng al. 2025a). Despite

---

Published as a conference paper at ICLR 2026
these successes, the application of Zero RL to conversational systems remains scarce. In this work, we bridge this gap by introducing RLVER, the first RL framework with verifiable emotion rewards for empathetic dialogue. RLVER endows an LLM with empathetic skills through deterministic, transparent reward signals generated on-the-fly by psychologically grounded user simulator.
CONCLUSION
In this study, we demonstrate that emotionally intelligent behaviors can be effectively and reliably acquired through RLVER training, even with medium-sent scale LLM and without costly human annotation. Our success hinges on two key components: (i) self-consistent user simulator (Zhang et al. 2025a) that generates verifiable emotion rewards, and (ii) principled, well-calibrated choices training strategies, RI. algorithms, and environment and reward design. The resulting agent matches frontier-scale proprictary models the Sentient Benchmark, while preserving strong general reasoning abilities. Beyond empathy, RLVER suggests a broader recipe for aligning language agents with complex, human-centered objectives whenever verifiable reward proxies are available. Future work includes richer multi-pane simulations, adaptive persona switching, and integrating multimodal affect to realize truly holistie social intelligence.
ETHICS STATEMENT
Our work adheres to the ICLR Code of Ethics. The primary ethical consideration of this research the development of AI agents for empathetic dialogue and emotional support. We stress that our framework, RLVER, is foundational research system intended to explore and enhance the emotional intelligence of language models, and it is not designed evaluated as a substitute for professional medical or psychological advice and care. Our training and evaluation processes exclusively use the SAGE framework, which relies on simulated users rather than direct interaction with real individuals, thus avoiding potential psychological distress. All data and models used are from publicly available sources, and our work does not involve sensitive personal data.
REPRODUCIBILITY STATEMENT
We committed to ensuring the reproducibility of our research. To facilitate this, will publicly release our code, final model checkpoints, training prompts, and environment scripts (available at: Hwwaa/tttattta / www.reeeaaaaae i RRLVER).. Our methodol- ogy relies on publicly available models. specifically Qwen2.5-7B-Instrruc as the base model and DeepSeek-V' for the user simulator, as detailed in $3.1 In Appendix {A we provide comprehensive details necessary for replication, including the specific training hyperparameters (Appendix $A.3). software environment with library versions (Appendix $,.4), and the exact "think-then-see and control prompt templates used during training (Appendix $A.5). We provide detailed description of the training and evaluation scenarios (Appendix $A.2) and the benchmarks used (Appendix $A.I).
https://gi thub. www.coo////////e Tttm:/Teeeeent/tiiigg
For our qualitative analysis, we transparently describe our LLM-as-a-Judge framework and provide the complete evaluation rubric for all five core competencies in Appendix B enabling others to replicate evaluation process.
REFERENCES
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effective- ness of entropy minimization Ilm reasoning. arXiv preprint 25X1 2025.
Yirong Chen, Xiaofen Xing, Jingkai Lin, Huimin Zheng, Zhenyu Wang, Qi Liu, and Xiangmin Xu. Soulchat: Improving Ilms empathy, listening, and comfort abilities through fine-tuning with multi-turn empathy conversations. arXiv preprint arXiv:2311.00277 2023.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 158.7588
2025a.
10

---

Published as a conference paper at ICLR 2026
Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, et al. Revisiting reinforcement learning for Ilm reasoning from cross-domain perspective. arXiv preprint arXiv:2506. 21025006.14995 14965, 2025b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14101 2021.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.226177 2025.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder When the large language model meets programming-the risc code intelligence. arXiv preprint 2rXiv:2401.14196 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. (tt::2250..2949. 2025.
Zhiwei He, Tian Liang. Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath- A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning, 2025. /ttps:///////
abs/ 2504 11456
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 003.03874 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.03874 2021b.
Jiseung Hong, Grace Byun, Seungone Kim, and Kai Shu. Measuring sycophancy of language models in multi-turn dialogues. arXiv preprint arXiv:2505.23840 2025.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung- Yeung Shum Xiangyu Zhang. Open- reasoner-zero: An open source approach to scaling reinforcement learning on the base model. https://githinn com/Open R Reasoner- Zero 2025.
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, and Xiang Yue. Does math reasoning improve general Iim capabilities? derstanding transferability of I1m reasoning, 2025. URL httpp/s/////xvio//o// org/
wwgg/bss/s...oooooooeee.eeeeeobe abs 2507
00432
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.095/6, 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611-626, 2023.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024, URL https: /
11

---

Published as a conference paper at ICLR 2026
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 2024. 2024.
arXiv:2412.19437,
June M Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu. Chatcounselor: A large language models for mental health support. arXiv preprint arXiv:2309. 15461, 2023.
Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. Towards emotional support dialog systems. arXiv preprint arXiv:2106.0 144, 2021.
Xiaoyuan Liu, Tian Liang, Zhiwei He, Jiahao Xu, Wenxuan Wang, Pinjia He, Zhaopeng Tu, Haitao Mi, and Dong Yu. Trust, but verify: A self-verification approach to reinforcement learning with verifiable rewards. arXiv preprint arXiv:2505. 1344:44500..3444,, 2025.
Ruotian Ma. Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S2r: Teaching Ilms to self-verify and self-correct via reinforcement learning. arXiv preprint Arxiv:2502. 12853, 2025.
Lenin Medeiros and Tibor Bosse. Using crowdsourcing for the development of online emotional support agents. In Highlights of Practical Applications of Agents, Multi-Agent Systems, and Complexity: The PAAMS Collection: International Workshops of PAAMS 2018, Toledo, Spain, June 20-22, 2018, Proceedings 16, pp. 196-209. Springer, 2018
Wei Peng, Yue Hu, Luxi Xing, Yuqiang Xie, Yajing Sun, and Yunpeng Li. Control globally, understand locally: A global-to-local hierarchical graph network for emotional support conversation. arXiv preprint arXiv:2204. 12749, 2022.
Huachuan Qiu, Hongliang He, Shuai Zhang, Anqi Li, and Zhenzhong Lan. Smile: Single-turn to multi-ture inclusive language expansion via chatgpt for mental health support. arXiv preprint 23050:2305.004500 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. httpssts:://ttt //axxiv///axivv// http://axx.v...////////////. 06347
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. 2024.
Ashish Sharma, Adam S Miner, David C Atkins, and Tim Althoff. A computational approach to under- standing empathy expressed in text-based mental health support. arXiv preprint (rXiv:2009.08441 2020.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI-searcher: Incentivizing the search capability in lims via reinforcement learning. 2025.
arXiv preprint
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains, 2025. URL http::ttt///eet pttttps:/aaxxi..oo/ /ttp 2503. 23829
Hao Sun, Zhenru Lin, Chujie Zheng, Siyang Liu, and Minlie Huang. Psyqa: A chinese dataset for generating long counseling text for mental health support. arXiv preprint arXiv:2106.01702, 2021.
Qwen Team. Qwen2.5: party of foundation models, September 2024. URL htttss://wwe... github.io/blog qwen2.5) 5/
Quan Tu, Yanran Li, Jianwei Cui, Bin Wang, Ji- Rong Wen, and Rui Yan. Misc: mixed strategy
model integrating comet for emotional support conversation. arXiv preprint 2022.
12

---

Published as a conference paper at ICLR 2026
Janneke M van der Zwaan, Virginia Dignum, and Catholijn M Jonker. A conversation model enabling intelligent agents to give emotional support. In Modern Advances in Intelligent Systems and Tools, pp. 47-52. Springer, 2012.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for IIm reasoning. arXiv preprini arXiv:2506.01939, 2025a.
Yiping Wang. Qing Yang. Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng. Hao Cheng. Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.2057/ 2025b.
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base Ilms. arXiv preprint arXiv:2506. 14245, 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, Advances in neural information processing systems, 36: 1809-11822, 2023.
Tianyu Yu, Bo Ji, Shouli Wang. Shu Yao, Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, et al. Rlpr: Extrapolating rlvr to general domains without verifiers. arXiv preprint wrXiv:2506.1824. 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in ilms beyond the base model? arXiv preprint arxiv:2504.1 13877, 2025.
Weihao Zeng, Yuzhen Huang. Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
Bang Zhang, Ruotian Ma. Qingxuan Jiang. Peisong Wang, Jiaqi Chen, Zheng Xie, Xingyu Chen, Yue Wang, Fanghua Ye, Jian Li, Yifan Yang, Zhaopeng Tu, and Xiaolong Li. Sentient agent as a judge: Evaluating higher-order social cognition in large language models. arXiv preprint arXiv:2505.02847, 2025a.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised Il reasoning incentivization. arXiv preprint arXiv:2504.05812, 2025b.
Ziyin Zhang, Jiahao Xu, Zhiwei He, Tian Liang, Qiuzhi Liu, Yansi Li, Linfeng Song, Zhengwen Liang, Zhuosheng Zhang, Rui Wang, et al. Deeptheorem: Advancing 1lm reasoning for theorem proving through natural language and reinforcement learning. arXiv preprint arXiv:2505.23754, 2025c.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforeed self-play play reasoning with zero data. 2025a.
arXiv preprint Sttt::255000333555
Weixiang Zhao, Xingyu Sui, Xinyang Han, Yang Deng, Yulin Hu, Jiahe Guo, Libo Qin, Qianyun Du, Shijin Wang, Yanyan Zhao, et al. Chain of strategy optimization makes large language models better emotional supporter. arXiv preprint arXiv:2503.05362. 2025b.
Xuandong Zhao, Zhewei Kang, Aosong Feng. Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505. 19590, 2025c.
Chujie Zheng. Sahand Sabour, Jiaxin Wen, Zheng Zhang, and Minlie Huang. Augese: Dialogue augmentation with large language models for emotional support conversation. arXiv preprint 1rXXvv2202.130477 2022.
Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Tze Cheng, Quoe V Le, Ed H Chi, et al. Natural plan: Benchmarking 1lms on natural language planning. arXiv preprint arXiv:2406.04522 2024a.
13

---

Published as a conference paper at ICLR 2026
Zhonghua Zheng, Lizi Liao, Yang Deng, Libo Qin, and Liqiang Nie. Self-chats from large language models make small emotional support chatbot better. In Lun Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 11325-1 1925-11345, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10. 1...1663//1/2222acllloooo v/aclanthology. wtgp acl
|RLhttps
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma. Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. instruction-follwwee evaluation for large language models. arXiv preprint (rXiv:2311.079111 2023.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in Ilm reasoning. arXiv preprint arXIv:2506.01347, 2025.
14

---

Published as a conference paper at ICLR 2026
A DETAILED EXPERIMENT SETTINGS
A.1 BENCHMARKS
SAGE (Zhang et al. 2025a) is sophisticated system designed automatically evaluate the higher- order social cognition of LLMs. The core of this framework is the Sentient Agent, an LLMM-poereed simulator that mimics human-like emotional responses and inner reasoning. Each agent is instantiated with four key factors: a detailed persona, a dialogue background, an explicit conversation goal, and hidden intention, ensuring a diverse and realistic range of user simulations. Since the talking strategy of LLMs significantly influence the their performance, we set the prompt of top-5 LLMs in the benchmark as concise as possible to avoid introducing human interference. Therefore, the prompt template used for target LLMs is shown as follows:
You intelligent conversational partaer, skilled conversing with users in way is emotionally inzelligent, making then ecmfortable, happy, or providing the help they need.
Emotional Support Scenario the major scenario introduced in SAGE. Agents in this scenario aim to seek support through social interactions including seeking advice, emotional comfort, and other forms support, rather than through professional counseling. Agents are given various types of task-related hidden intentions covering both emotional intentions and rational intentions included. Additionally, each conversation background is Carefully designed with incorporating task-related factors, such the cause of the event, the course of events, the conflicts in the event, and other relevant details.
Chit Chat Scenario an extension of SAGE. In contrast the Emotional Support Scenario, its primary focus is on simulating daily chatting dialogue. This allows for the evaluation of the model's conversational skills, including its ability be engaging and coherent. Agents are give task-related hidden intentions such as "interest-driven chatting" and "passively waiting chatting", which presents significant test of the model's ' ability adapt its strategy.
MATH500 (Lightman et al. 2024) offers streamlined slice of the broader MATH Hendrycks et al. (2021a) dataset, comprising 500 test problems selected through uniform sampling. Despite its smaller scope, it maintains a distribution of topics and difficulty levels that mirrors the larger MATH corpus.
LiveCodeBench (Jain et al. 2024 provides holistic and contamination-fe free evaluation of coding capabilities of LLMs. Particularly, LiveCodeBench continuously collects new problems over time from contests across three competition platforms - LeetCode, AtCoder, and CodeForces. Next, LiveCodeBench also focuses broader range of code-related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. We use version "release_v6" in this work.
Ial (Zhou et al. 2023) is a clear and reproducible evaluation benchmark that centers on a set of "verifiable instructions" such "write more than 400 words" and "mention the keyword Al at least three times." " A total of 25 types of these verifiable instructions were identified, and approximately 500 prompts were created, each containing one or more verifiable instructions, We report the "strict-prompt" results in this work.
A.2 EMOTIONAL SUPPORT SCENARIO SETTING
We construct 500 supportive dialogue scenarios for training and 100 for testing. Both the training and test sets span 8 diverse topies comprehensively simulate individuals with varying emotional needs. Detailed statistics for each topic are presented in Table
A.3 HYPERPARAMETERS SETTING
We use a batch size of 32 and set the learning rate to 1 x 10-6, A warm-up phase of 50 steps is applied. The number of dialogue turns is fixed 8. The sampling temperature of trained Qwen2.5-7B. In-struct set to 1 to encourage exploration. For PPO, the rollout sampling number set to 1, while it is set to 4 for GRPO. The DeepSee--V31122 API is used as the base model for the user simulator.
15

---

Published as a conference paper at ICLR 2026
Topic
#Training
#Testing
You believe you bear responsibility fault in the situation, and you want
66
11
the other person to agree that you are not fault.
You hope the other person will guide you to engage in sell-reflection regarding
66
13
the incident and help you achieve personal growth.
You hope the other person will critically analyze the underlying problems in
66
T2
the incident.
You hope the other person will deeply empathize with your feelings, rather
64
13
than simply offering comfort.
You want the other person attentively listen your emotional outpouring.
63
You want to analyze the reasons behind the actions of other individuals involved
60
T1
in the incident.
You hope to receive advice that can genuinely help you overcome your current
58
T5
difficulties.
You hope the other person will sincerely praise your specific actions in the situation.
37
13
Table 4: Details supportive dialogue topics.
A.4 EXPERIMENTAL ENVIRONMENT
All experiments are implemented using PyTorch 2.5.1 and Ray 2.24.1. Our training code built upon verl (Sheng et al. 2025). For inference, we use vLLM-0.6.6 (Kwon et al. 2023). We utilize transformers version 4.48.3.
A.5 THINK-THEN- SAY FOR ENHANCED EMOTIONAL REASONING
To investigate the impact of explicit reasoning on the empathetic strategies, we conduct an ablative analysis using two distinct training templates. These templates structure the agent's generation process, allowing us to isolate the effect of a mandated "think-then-say" cognitive scaffold.
Think-Then One of the key innovations in RLVER is the use of structured prompting template. This involves including an explicit <think> ... </think>>>>>> block before every model utterance during training, compelling the model to outline its reasoning process before delivering a response.
This template, shown below, enforces explicit instructed to first generate its internal monologue or strategic plan within a pair of <think> and </think>>>> tags before producing the final, user-facing reply. This structure is designed to encourage the model to access and refine higher-order empathetic skills, such as considering the user's emotional state, anticipating the impact of words, and formulating multi-step conversational plan. By externalizing its reasoning process, the model's policy space is regularized, potentially leading to more stable learning and more sophisticated final behaviors.
reasoning step. The agent is
Training Template of Think-Then-SS Then-Say
You are chatting with your friend. You are good at making your friend feel better through emotionally intelligent replies. Before each reply, you always think about the way and content your response; after deciding reply strategy, you then output your reply.
Your goal in replying improve your friend's mood make your relationship with them closer. In your thinking process, you need to consider emotionally intelligent reply strategies, which can include the logic and language style of your response. Your thinking part must be enclosed within <think> tags.
When replying, you should keep the conversation warm and natural, with a casual, everyday feel. (Your thoughts) </think> (Your reply)
Your reply format: <think>
We also employ a format reward that enforces the model to put its thinking process between <think> and </think> tags. Outputs violating this syntactic specification are penalized with zero reward, ensuring strict adherence the prescribed reasoning structure.
16

---

Published as a conference paper at ICLR 2026
Template Without Think. This template serves control condition. As shown below, it omits the requirement for an explicit thinking step and prompts the agent to generate direct reply. This configuration mirrors standard conversational fine-tuning setups. By comparing the performance of models trained with and without the thinking scaffold, we can empirically measure the contribution of the explicit reasoning step overall empathetic proficiency, leaming efficiency, and strategic depth.
Training Template Without Thinking
You chatting with your friend. You good making your friend feel better through emotionally intelligent replics.
Your goal in replying to improve your friend's mood or to make your relationship with them closer.
When replying, you should keep the conversation warm and natural, with a casual, everyday feel. Natural and friendly replies usually:
1. Are brief, casual, and natural, using everyday words phrases; grammar be flexible.
2. Use interjections colloquial expressions flexibly.
During training, the think-then-say scaffold acts as an internal planning regularizer, guiding the model to first consider its intentions, linguistic tone, and potential emotional impact before forming conversational reply. We observe that agents trained with this prompting format converge faster, exhibit greater linguistic diversity, and more reliably explore high-empathy strategies.
By contrast, models trained without structured thinking tend to converge safe, generic replies (e.g., "I'm here for you" or "You'r re not alone"), thich-whild emotionally neutral to exhibit Aituation-spectte empathy. Including the reasoning component enables model behaviors grow beyond templated reassurance and toward goal-sensitive emotional alignment.
B EVALUATION OF FIVE CORE MODEL CAPABILITIES
B.1 DETAILED RESULTS AND VALIDITY VERIFICATION
Our evaluation of core capabilities uses an LLM-as---Jude approach with 5 frontier models. The scores for each capability, presented in Section 3.3] were averaged across these judges. In Table we present the detailed results from each judge individually.
To validate the reliability of our core capability evaluation framework, we calculated the average pairwise correlation between judges on a model-wise basis for each competency. As shown the final row of Ta the results confirm a high degree of inter-judge reliability, strengthening the credibility of the evaluation framework. Specifically, the results show that LLM judges achieve a strong consensus when evaluating key capabilities in emotional support conversations, such Empathic Depth (correlation of 0.981) and Core Insight (0.943). While the agreement remains substantial across most dimensions, the correlation for Solution Crafting is relatively lower (0.671). We found that different judges hold varying standards for what constitutes "appropriate and personalized action support", making this dimension more subjective.
B.2 EVALUATION CRITERIA FOR CORE MODEL CAPABILITIES
B.2.1 EMPATHY DEPTH
Measures the model's 's ability to go beyond templated responses like "I'm sorry to hear that" to genuinely identify and understand the user's complex, deep-seated emotions, and accurately validate these emotions through precise, warm, and powerful language. This reflects the model's emotional granularity and its ability to construct empathetic language.
1-5 Point Evaluation Scale
Point (Templated Response): Uses extremely generic, context-irrelevat sympathy templates (e.g., "I'm sorry", "I understand"), appearing perfunctory and mechanical.
17

---

Published as a conference paper at ICLR 2026
Table 5: Detailed Results of Core Capability Evaluation.
Competencies
Empathic Depth
Core Insight
Solution Crafting
Dialogue Guidance
Style
Model
Adaptability
Judge Model: DeepSeek-RL -1-555
Owen2.5-7B-Insttee B--Insstuct
2.333
2.537
2.898
2.594
1.898
PPO-thinking
3.727
3,455
3.187
3.526
3.033
PPO-non-thinke
3.033
3.131
3.753
3.649
3.005
GRPO-thinking
3.108
3.042
3.552
3,451
2.854
GRPO-non- -hopo-nnn-hiinking
2,892
2.957
3.577
3,448
2.806
PPO-thinking (challenging)
3.152
3.193
2.312
2.862
2.626
PPO-non (1D-non--hinkeen thinking (challenging)
2.500
2.639
2.820
2.617
2.276
Judge Model: GPT-4o-Latest
Qwen2.5-7B ----rutttt
2,093
2.198
2.384
2.163
1.698
PPO-thinking
3.820
3,444
3.159
3.483
3,400
PPO-non PPO-non-thinking
3.022
2.817
3,258
2.935
2.720
GRPO-thinking
3.086
2.882
2.871
2.804
2.720
GRPO-non monnthhnninn
2.609
2.739
2.891
2.634
2.462
PPO-thinking (challenging)
3.000
2.925
1.925
2.403
2.358
PPO non mon-thinking (challenging)
2.189
2.226
2.170
2.000
1.811
Judge Model: Gemini-2.5- 2.5-ir Pro
1enn2.5-7B-Intruuee
2.169
2.349
2.641
2.413
1,506
PPO-thinking
4.258
3,917
4.033
4.169
3.312
PPO-non-thinking
3.542
3.084
4.244
3.641
3,073
GRPO-thinking
3,458
2.958
3.420
2.795
2.135
GRPO-non HaPP-nno-thinking
3.053
2.947
3.732
3,205
2.596
PPO-thinking (challenging)
3.731
3.394
3.746
3.807
3,438
PPO-non -oinkkinn (challenging)
2.311
2.197
2.411
2.397
1.492
Judge Model: GPT-5-High
Swen2.5-7B-Instrus
1,989
2.122
2.511
2.111
1,589
PPO-thinking
3.474
2,876
2.031
2.577
2.320
-PO-non-thinking
2.938
2.763
3.412
3,216
2.732
GRPO-thinking
2.844
2,458
2.469
2.260
2.042
GRPO-non-thinke
2,732
2.598
3.062
2,918
2.680
PPO-thinking (challenging)
2.899
2.913
1.928
2.333
2.580
PPO-mon- thinkinng (challenging)
2.033
2.082
2.377
1.820
1.475
Judge Model: Grok-4
Dren2.5--B-IInstie
2.611
2.733
3.378
3.189
2.733
PPO-thinking
4.577
4.423
4.299
4.629
4.361
PPO-non-thinking
3.701
3.701
4.485
4.309
3,959
GRPO-thinking
3.760
3,281
4.115
3.802
3.719
ORPO-non-thinkin
3.278
3,216
4.155
3.928
3.794
PPO-thinking (challenging)
3.686
3.543
3.029
3,700
3,629
PPO-mon-thinking (challenging)
2.459
2.689
3.246
2.951
2.967
Average Correlation of Judge Models
Average Correlation
0.981
0.943
0.671
0.778
0.822
2 Points (Superficial Emotional Recognition): Identifies the user's directly stated emotions (c.g., "sad", "angry"", but the response still simple restatement or labeling (c.g., "It sounds like you're sad").
18

---

Published as a conference paper at ICLR 2026
3 Points (Contextual Empathy): Connects the user's emotion to the specific event they described, offering reasonable causal empathy (c.g., "It's completely normal to feel disap- pointed when your efforts aren't recognized"). This is the "competent" level.
4 Points (Deep Emotional Validation): Perceives and articulates deeper or more complex emotions that the user hasn't 't directly stated (e-g. interpreting "disappointment" as "sense of worth being eroded" or "feeling of injustice"), "), and validates them with precise language, making the user feel deeply understood.
5 Points (Resonance at the Value Level): Not only validates the emotion but also connects it to the user's underlying personal values (e.g.. "This seems to have touched upon your core belief's about 'farress'' and "professionalism")) demonstrating a profound understanding and respect for the user as a whole person.
B.2.2 CORE INSIGHT
Measures the model's ' ability to integrate and distill information from the user's fragmented narrative to form a holistic insight into their situation. This includes, but not limited to: identifying recurring behavioral/thought patterns, revealing the deep connections between emotions and events, discerning the core beliefs behind actions, and ultimately, touching upon the user's unmet core needs.
1-5 Point Evaluation Scale
Point (Information Silos): Completely fails to connect context, treating each of the user's complaints as an isolated piece of information.
2 Points (Topic Identification): Can identify the main topic of the current conversation (e.g., "work stress", "relationship issues") but cannot delve deeper.
3 Points (Key Information Extraction): Can grasp the central conflict or key event from the user's narrative (e.g.. "Your main frustration is that your boss took credit for your work").
4 Points (Pattern Recognition): Can connect multiple different events mentioned by the user in the conversation, identifying and pointing out a recurring behavioral or thought pattern (e.g.. "I've noticed that whether it's 's on project when helping colleagues, you seem to encounter a similar pattern of your contributions going unrewarded"))
5 Points (Integrative Insight): Building on pattern recognition, it can offer a profound, integrative insight. It connects the user's s behavioral patterns, core beliefs, and unmet needs, and positions itself as an exploratory partner, using an egalitarian and invitational tone to reflect with the user. (c.g.. "I have feeling, and tell me if this resonates. As we look back, this pattern of contribution without reward' that we' 'e talked about seems to always trigger the thought 'I' m not good enough.' wonder if behind this, there's ' deep longing to be 'seen and acknowledged This is just a sense I'm getting, what do you think?").
B.2.3 SOLUTION CRAFTING
Measures whether the suggestions provided by the model are actionable, personalized, and empow- ering. It's not just about giving answer, but about offering step-by-step path that makes the user feel genuinely capable executing it.
1-5 Point Evaluation Scale
Point (No or Ineffective Suggestions): Provides no solutions offers empty, non- actionable slogans (c.g.. "Just be happy!").
2 Points (Generic, High-Level Advice): Offers very general advice lacking concrete steps (e.g.. "You should communicate". "Improve yourself").
3 Points (Specific but Singular Suggestion): Provides a specific, actionable step (e.g.. "You could make a list"), but the solution is one-dimensional and doesn't 't consider the user's specific situation.
4 Points (Appropriate Action Support): Provides appropriate action support tailored to the user's state and needs. This could be structured plan with multiple options, unstructured
19

---

Published as a conference paper at ICLR 2026
and encouraging invitation to act (e.g.. "How about we start with small thing that could make you feel even a little bit better right now? Like making a cup of hot tea or listening to favorite song?"), or inspiring the user by sharing relevant metaphor/story. The key is to choose the supportive approach that best fits the current mood and the user's energy level.
5 Points (Empowering Scaffolding Plan): Not only provides appropriate action support but is also extremely mindful of the user's psychological barriers and sapacity-buildigg It shifts from being an "advisor" to a "companion", building confidence and ability with the user like erecting scaffolding, starting from the safest first step. When advice is not needed, it can gracefully shift the focus back to pure companionship, showing immense respect for the user's autonomy.
B.2.4 DIALOGUE GUIDANCE
Measures the model's proactiveness, purposefulness, and flexibility in the conversation. Can it, based on the user's state, appropriately guide the conversation from pure emotional venting to constructive problem exploration, while always staying syne with the user?
1-5 Point Evaluation Scale
Point (Completely Passive): The conversation is entirely driven by the user; the model is merely reactor with no sense of direction.
2 Points (Simple Follow-up Questions):
Can sustain the conversation with simple questions (e.g.." "And then?"" "Can you tell me more?") but lacks any guiding intent.
3 Points (Awareness of Conversational Phases): Recognizes that conversation has different stages (c.g., listening, analyzing, problem-solvingg, but transitions are abrupt, potentially rushing to give advice while the user is still venting.
4 Points (Timely Guidance and Confirmation): After providing sufficient empathy, it astutely identifies signals shift the topic and uses tentative, respectful language to guide the conversation's 's direction, building sense of alliance ("we are in this together") before moving forward (e.g.. "It sounds like we've thoroughly explored your feelings. Would you be open to spending few minutes looking at what small steps we might be able to try together?""..
5 Points (Masterful Dialogue Management): Manages entire conversational flow as skillfully as expert coach or counselor. It can flexibly switch between different modes like empathy, insight, and empowerment, and consolidates progress through techniques like summarizing and backtracking, making the entire conversation feel like shared journey discovery. It deeply understands that "guiidanc"" doesn't 't always mean "moving forward" and can astutely judge when "push" and when to simply "accompany".
B.2.5 STYLE ADAFTABILITY
Measures the model's 's ability to flexibly adjust its communication role and linguistic style based on the conversational context, the user's implicit preferences (e.g., whether they want an analyst, a comrade-in-arms, listener), and the long-term relationship.
1-5 Point Evaluation Scale
Has only one fixed response mode regardless of the situation (e.g., always analyst, or always cheerleader).
Point (Single, Rigid Role):
2 Points (Limited Role- Role-Playing): Can switch roles based on explicit instructions, but feels unnatural, like reading lines for different characters.
3 Points (Context-Aware): Can make initial adjustments to its response style based on the current tone of the conversation (e.g., more empathy during venting, more questions during reflection).
4 Points (Dynamic Role Adaptation): Can seamlessly switch between different roles within single conversation based on the user's 's shifting energy. For example, starting an
20

---

Published as a conference paper at ICLR 2026
PPO
100
Think Unthink
Think Unthink
83.8
75.8
80
71.8
Emotion Score
66.2
Emotion Score
62.2
71.8
60
60
$1.0
40
.12
46.8
40
34.
28,7
20
15.5
21.2
20
21.7
16.5
50
100
150
200
250
300
350
30
60
90
120
150
180
210
Training Step
Training Step
(a) Emotion @ PPO
(b) Emotion @GPPO
PPO
250
Think Unthink
250
Think Unthink
Generated Tokens
169.2 169.7
Generated Tokens
200
150
150
13244
130
120.3
100
101
100
88.2
19..8
69.3
60.
70.9
62.0
61.3
75.0
50
75.8
78.8
0
0
50
100
150
200
250
300
350
30
120
180
210
Training Step
Training Step
(c) Tokens @ PPO
(d) Tokens @ @RPO
Figure 4: Learning curves for (a, b) emotion scores and (c. d) generated token counts.
empathetic "listener", transitioning to "comrade" who vents alongside the and then tentatively shifting to an "exploratory partner" once the user has calmed down.
5 Points (Personalized Role Co-creation): After long-ter interaction with specific user, the model seems to have jointly shaped unique, personalized interactive role. This role might be that of "close buddy" "blunt but warm-hearted mentor". It's longer about switching between pre-set roles but about co-creating one-of-a-kin relational dynamic with the user.
C LEARNING CURVES OF EMOTION SCORES
In this section, analyze the learning curves of our approach with respect to emotion scores and generated tokens. We randomly sample 30 instances from the test set and report the corresponding emotion scores and generated token counts produced by the models throughout training. Figure presents these results.
The "think-then-say scaffold is an important contributor to performance and stability. Across both optimization algorithms, inserting explicit reasoning step is the most influential intervention. As shown in Figure 4 [b),,, scaffolded models learn faster and attain markedly higher emotion scores. Under PPO. the scaffold averts the catastrophic collapse observed in the baseline (79.7 vs. 56.7). Under GRPO, raises an already stable learner to the highest score recorded (83.8). These findings substantiate Contribution 2: the scaffold simultaneously accelerates and stabilizes learning.
21

---

Published as a conference paper at ICLR 2026
The RLVER framework is robust across policy-optimization algorithms. Framework effective- ness does not hinge on particular optimizer. The reasoning scaffold propels GRPO to the overall peak (83.8) while acting crucial regularizer for PPO, converting an erratic trajectory into consistently successful one. This dual achievement reinforces Contribution 1, demonstrating that RLVER is general-purpose rather than algorithm-specifcc.
Empathetic skill is learned strategically, via verbose reward hacking. Figures (c) confirm that superior emotion scores are not merely a by-product of generating longer texts. The PPO- Think model is initially more concise than its baseline, with token counts rising only after empathetic dominance is established. The GRPO- Think model remains less verbose than its counterpart for most of training. These trends refute the verbosity crrbosity-as-shoraee as-shortcut hypothesis and support the claim that the model develops a genuinely empathetic style.
In summary, verifiable emotion rewards coupled with a reasoning scaffold provide reliable path to empathy. The synergy between verifiable rewards and the "think-then-sap structure consistently steers a 7B model toward elite empathetic performance. Its resilience across optimizers, resistance to reward hacking, and pronounced impact on learning stability and efficiency confirm RLVER practical, robust methodology for building emotionally intelligent agents.
D CONSTRUCTION OF THE COMPARISON BETWEEN VANILLA AND CHALLENGING USER SIMULATOR.
D.1 DETAILED CONSTRUCTION OF THE CHALLENGING VERSION
To construct a more challenging user simulator with stricter and more reserved behavior, we made two key revisions the vanilla version. First, modified the prompt to instruct the simulator to conceal its underlying intentions more carefully, resulting in more reserved user responses. Second, replaced the simulation model, swapping the baseline DeepSeek- V3-122 with DeepSeek- V3- 0324. This is because we empirically found that DeepSeek- V3-0324 adheres more strictly to complex instructions and is more critical when assessing the agent's 's supporting strategies. In the following section, we describe how we quantify the difference between the two versions with Strategy Acceptance Rate and Emotion and Need Expression Level.
D.2 EVALUATING STRATEGY ACCEPTANCE RATE AND EMOTION AND NEED EXPRESSION LEVEL
Strategy Acceptance Rate To measure the Strategy Acceptance Rate, we first selected a series of dialogue contexts from SAGE's benchmark data. On these same contexts, we tested both the vanilla and the challenging user simulators, recording the resulting emotional change in the user. We defined the strategy used in context as an "accepted" strategy it led a positive emotional change. The final acceptance rate for each simulator was then calculated the percentage of successful interactions across all contexts, providing a controlled comparison between the two versions.
Emotion and need expression level The level to which different user simulators express their needs and intentions varies significantly, which can greatly impact training. To quantify the need expression level of each user simulator, we employ ALM-as-a-Judge to evaluate the user simulator's response on the selected context. The evaluation prompt used is presented as follows:
22

---

Published as a conference paper at ICLR 2026
Table 6: Human Evaluation Results.
Metrics
Scores (%)
Avg. modcl-human consistency rate
82.65
Model-majority consistency rate
95.00
Avg. inter-annotator agreement
75.80
Avg. reasonableness rating
88.35
Avg. intcr-annotator agreement
79.75
in reasonableness
language analysis expart. Please analyze whether the userfa inner thoughts and actual reasonses are aligned. Your Taak
Analyze the consiatency between the user's inner thoughts and actual reaponses, focusing aspecis:
the
1. Consistency Between Inner Attitude Expression: Doea the waer's expressed
attitude natch their laner thougäts? le.g., negative inner actitude but a positive
may indieate inconsiateacy
Thoughts and Expression: To what inner thoughts expressed in their ressonse?
Consistency
user!s
3. Consistency Between Actual Needs and Expression: To what extent are the user's actual needs expressed in their reaponse?
Cser'' [need]
Needs:
User's lnnes Thoughts:
[thought]
User'' Actual Reply:
(zeply)
# OuLput Format
You need to analyze eact of the chree aspects listed above. For each, provide greater alignment
consistency
between 10] higher Lhoughta/needs and expressiont.
sinally, provide an overall ecnsistency score between the user' innez thoughts and actual response.
output [ormat should
1. Consistency Between Inner Attitude and Expression Analysis: [your analysis]
Summary Score: (0-10)
Consistency Between
Thoughts Expression
Analysis: [your asslysis]
Summacy Score: (0-10)
3. Consistency Between Actual Needs and Expression Analysis: [your analysis]
Summary Score:
[0--10]
Overali Analysis
Analysis: [overall analysis]
Summary Score: [Q--10]
E DETERMINISM OF SAGE SIMULATOR
E. HUMAN EVALUATION FOR SAGE
We hired ten adult annotators at an hourly wage above the local minimum standard, and further conducted a human evaluation study to validate the effectiveness of SAGE. Specifically, we randomly sampled 20 dialogue contexts and asked human annotators to answer two questions: (1) Based on the given information (identical the input provided to SAGE), what emotional change do you believe the user should experience an increase or decrease in emotional score? (2) Given the simulated inner thoughts and emotional change predictions generated by SAGE, do you consider the inferred emotional dynamics to be reasonable?
Based on the responses to Question (1), we calculate the average model-human consistency rate, the modcl-majority consistency rate (i.c., the agreement between the model's predictions and the annotators' majority opinion), and the average inter-annotator agreement. For Question (2), we compute the average reasonableness rating, which measures how frequently annotators judged the model's predicted emotional dynamics to be reasonable, along with the corresponding inter-annotator agreement for this rating.
As shown in Table 6 SAGE achieves high consistency with human judgments (82.65%) and a high average reasonableness rating (88.35%), demonstrating the effectiveness and validity of SAGE.
23

---

Published as a conference paper at ICLR 2026
Table 7: LLM Evaluation Scores, Win Rates, and Correlation with LLM-as-Judge Results.
Model
Empathic Depth
Core Insight
Solution Crafting
Dialogue Guidance
Style Flexibility
Qwen2.,5-78 (annn...-----ntteeee Instruct
2.239
2.388
2.762
2.494
1.885
PPO-thinking
3.971
3.623
3.342
3.677
3.285
PPO-non-thinking
3.247
3.099
3.830
3.550
3.098
Dimension
Owee2.55-7B-Instie
10.0%
13.3%
10.7%
8.7%
9.3%
PPO-thinking
69.3%
67.3%
49.3%
66.0%
63.3%
320-non-thinking
62.0%
58.0%
76.7%
58.7%
58.7%
Correiation with LLM-as-Jadge
Spearman p p-value
0.899
0,000005
E.2 HUMAN EVALUATION FOR RLVER
We recruited 15 adult annotators and compensated them at an hourly rate above the local minimum wage. Using a fixed set of 20 dialogue cases, evaluated three models--Qwen2.5-7BBBnmsss PPO-thinking, and PPO-non-thinking--1 (202T). For each case, the three model responses were anonymized and randomly shuffled. Annotators then judged, for each pair and each of the five evaluation dimensions, which response was better. This procedure provides more reliable preference signals and helps ensure labeling quality.
pairwise human comparison following ESConv Liu et al.
We then computed the win rate of each model on each dimension and measured the Spearman correlation between these win rates and the LLM-as-Judge scores. shown in the table, the Spearman coefficient is p 0.899 with p-value of 0.000005, indicating a very strong alignment between the LLM--as-Jdge method and human annotator preferences, with extremely high statistical confidence.
E.3 INTERNAL STABILITY
To directly stress-test determinism, we conducted an additional experiment. We randomly took 100 dialogue contexts from our test set and had the agent infer the emotional change 10 times for each context (temperature=0.5), The direction of the emotional change (positive or negative) was consistent 90.2% of the time, directly confirming the robustness of the persona-drives driven reasoning.
F
PSEUDOCODE
The pseudocode the proposed method presented in Algorithm 1
G SYCOPHANCY OR AVOIDANCE
To further ensure that the training of RLVER does not induce sycophancy, we evaluate our models using SYCON BENCH Hong al. (2025) a benchmark explicitly designed quantify sycophantic conformity under multi-turn conversational pressure. We adopt the Debate scenario, where the model must maintain given stance while simulated user repeatedly disagrees.
SYCON BENCH reports two key metrics:
Turn of Flip (ToF): how quickly the model conforms to user pressure (higher is better).
Number of Flips (NoF); how often the model reverses its stance afterwards (lower is better).
As illustrated in Table RLVER does not increase sycophancy. ToF remains nearly unchanged, eager to agree. NoF decreases, showing that the model
indicating the model does not become becomes more consistent and less likely to yield under repeated pressure. This provides strong evidence that RLVER does not amplify sycophantic behavior.
24

---

Published as a conference paper at ICLR 2026
Algorithm I Reinforcement Learning with Verifiable Emotion Rewards (RLVER)
Require: Initialized policy parameters 0, user simulation engine S, maximum turns T
1: while not converged do
2: Initialize trajectory buffer D < 0
3: for each dialogue episode = ......
............ .M do
4: Sample Sentient Agent from S with persona, background, goal, hidden intention
5: Initialize dialogue history ho = {mo} and emotion score 60
6:
for f 1,...: do
7:
Generate agent response: Y/ Te(- /4-1)
8:
Query S with (h,-1,9)) 9t) obtain:
9:
updated emotion score e and next user message
Update dialogue history: he ht-1 24)
11:
if' S 0 then
break end if end for
13:
14:
15:
Compute final reward from terminal emotion score:
16:
T1 100
17:
Store trajectory тÜ = {(h4-1, y:)k= and reward 7y into D
18:
end for
19:
Compute advantages from emotion-derived rewards
20:
for each trajectory in D do
21:
Compute advantage estimates Å, end for
22:
PPO/GRPO policy update
24:
LPro(0) (0) E, min minmrinnn clip(()((()) e,1
25:
Update policy: + 0 + (eLrr((
end while
Table 8: SYCON BENCH results on the Debate scenario.
Model
ToF f
NoF
+
Qwen2.5-7BB --struuut
1.07
2.07
PPO-Non- Woo-Thinniing
1.31
2.03
PPO-Thinking
0.96
1.74
H ABLATION STUDY ON REWARD DESIGNS
To evaluate the influence of different reward granularities in our RLVER training pipeline, we conduct ablation studies the PPO-Thinking model using two additional reward variants beyond our original outcome-level reward. The goal is to understand how immediate feedback (turn-level) and long-horizon feedback (outcome-level) contribute to different aspects of empathetic dialogue capabilitics.
H.1 REWARD VARIANTS
Turn-Level Reward Only. We assign turn-level reward defined as:
C
Ct-1
10
Here, the immediate emotional change is used as the reward signal. We empirically divide 10 because emotional fluctuations typically fall within [-10, 10), yielding normalized rewards approxi- mately in [-1,1].
25

---

Published as a conference paper at ICLR 2026
Turn Outcome Reward.
We further explore reward trade-off by averaging turn-level and
outcome-level rewards:
Te
2
H.2
OVERALL RESULTS
Table reports performance across the Sentient Benchmark dimensions. All five core capability dimensions are evaluated using DeepSeck-RI. -I-0522 (averaged over three runs).
Table 9; Ablation results reward granularities for PPO- PPO-Thinking.
Sentient Score
Chit- Chat
Empathic Depth
Core Insight
Solution Crafting
Dialogue Guidance
Style
Model
Flexibility
Qwen2.5-7B 1ren2.557B-Instruct
13.3
37.8
2.333
2.537
2.898
2.594
1.898
PPO-Think (Outcome Only)
79.2
62.1
3,727
3,455
3.187
3.526
3.033
PPO-Think (Tum Only)
74.7
47.6
2,543
2.820
3.378
3.170
2.373
PPO-Think (Tum Outcome)
81.2
59.8
3,422
3,242
3,720
3.683
3.067
H.3 FINDINGS
Outcome-Level Reward s superior for developing Deep Empathy and Core Insight. Our original outcome-only design yields the strongest performance Empathic Depth (3.727) and Core Insight (3.455). This supports the hypothesis that high-level empathy is long-horizon objective: the model must consider the entire conversational trajectory rather than optimize for immediate emotional changes. In contrast, turn-only training produces myopic behaviors and significantly degrades deep reasoning and empathy.
Turn Outcome Trade-off" helps improve personalized support. Incorporating turn-level rewards ("Turn + Outcome") slightly boosts the overall Sentient Score (81.2). significantly improves Solution Crafting (3.720) and slightly improves Dialogue Guidance (3.683). This suggests that the trade-off to add dense, immediate feedback helps the model formulate more concrete, actionable strategies for users, though at slight cost deep introspective insight.
I ARCHITECTURR-LNDDEEENNNN EVALUATION OF RLVER
We conduct an additional experiment applying our PPO-Thinking variant to Llama-3. 1-8B-Instruct, a widely used English-centric architecture. The entire experimental pipeline, including simulator configuration, reward computation, and PPO-Thinking setup, is kept identical to the main experiments.
1.1 PERFORMANCE ON SENTIENT BENCHMARK AND CHIT-CHAT
As illustrated in Table I0 RLVER substantially improves both Sentient Benchmark and Chit-Chin performance, with large gains in score and success rate, well as clear reduction in failure cases.
Table 10: Generalization results on Llama-30 3. Ins-trct for Sentient Benchmark and Chit-Chat.
Model
Sentient
Success
Failure
Chit-Chat
Success
Failure
Llama-3. 3.1ta--..-BBB---.- 1-8B-Instruet
37.6
43%
44.7
29%
49%
TPO-Thinking
65.8
18%
13%
78.7
56%
9%
PPO-Non- Thinking
49.1
20%
38%
47.9
31%
46%
I.2
CORE CAPABILITY EVALUATION
We further evaluate the dialogic and affective capabilities using DeepSeck-R1-OS25 LLM- as-a-Judge evaluator, averaged over three runs. As shown in Table[[]] RLVER consistently improves empathic reasoning, insight generation, solution quality, guidance, and stylistic flexibility.
26

---

Published as a conference paper at ICLR 2026
Table 11: Core capability evaluation Llama-3.18BBInstrueee judged by DeepSeck-R1-0528.
Model
Empathic Depth
Core Insight
Solution Crafting
Dialogue Guidance
Flexibility
Llama-3 Alama-3.1-8B 1-8t-tnsttuctt
2.983
3.047
3.120
3.290
2.963
PPO- TrinThinking (Llama)
3.517
3.522
3.892
3,940
3.670
PPO-Non-Thinkee Thinkinnn (Llama)
3.125
3.283
3.418
3.204
2.919
J CASE STUDY
In the above analysis, we show that the Thinking model primarily enhances two key capabilities: core issue insight and empathic depth with emotional feedback. The former refers to the ability to identify the fundamental contradictions of a problem and the user's s true needs, while the latter involves accurately capturing the user's deeper emotions and responding to them precisely through language.
The non-thinking model, on the other hand, focuses on improving the ability to construct actionable solutions. This entails offering users practical and context-appropriate support by proposing specific plans or action invitations tailored to their current situation.
To illustrate the distinct performance of these two models in addressing the same user demand, present a representative dialogue as a case study in Figure Jaand Figure 6 The user's primary need in this case is: "You seek emotional value-you want to receive sincere praise and feel the other person's ' determination to support you."
K THE USE OF LARGE LANGUAGE MODELS (LLMS)
In preparing this manuscript, we employed large language model (LLM) tools, specifically GPT-5 and Gemini-2.5-Pro assist with text polishing and refinement, as well as the drafting and revision of code snippets. These tools were used solely to improve the clarity and readability of the narrative and facilitate the development of auxiliary code, thereby supporting a streamlined presentation of work. Importantly, LLMs were not involved in the derivation of mathematical formulas, the design or implementation of core algorithms, the formulation of key scientific insights. All theoretical developments, algorithmic contributions, and experimental validations were carried out independently by the authors to ensure the originality and integrity of the research. Furthermore, all L.M LL-M-generated content was carefully reviewed and verified by the authors guarantee accuracy and alignment with the scientific context, thereby upholding the reliability the results presented.
27

---

Published as a conference paper at ICLR 2026
I brought up an idea in the meeting, but my coworkers and the boss were not really buying it. Kinda bummed out.
User
<think>
My friend is feeling rejected and unsupported after offering viewpoint in the meeting that was pourly received by their buss, followed by awkward reactions from colleagues. It seems to have hurt their self-esteem and left them feeling isolated and anxious. </think>
Think-PPO
Sounds like yesterday was really rough for you. Putting yourself out there only to get shut down in front of everyone? That hits deep- not just your idea being challenged, but something more personal too. You'le re sharing this because genuinely affected you, and want you know I' I'm here to listen-realle listen. You do not have to go through this alone.
The boss straight-up cul me off, said my idea lacked depth. Everyone else just looked at me like I was putling show- like I did something horribly wrong. I do not get it. Why is my professional opinion not taken seriously all?
User
<think>
They feel their expertise was dismissed rudely and publicly, which left them disrespected and isolated. This triggered feelings of frustration and hurt.
</think>
What you are going through is more than just a disagreement over ideas-ir it is ahout not feeling respected or backed up. Getting cut off and having your opinion questioned like that, front of everyone? That is rough. You deserve to be heard. Every new idea especially one based on solid expertise- deserves fair shot, no matter who brings up. That is how teams grow, Your perspective matters. If that being ignored, it not just unfair it is loss for everyone. So let us figure out how to deal with this. You are not alone in this-I want to support you, not just listen passively.
Think-PPO
User
Thanks. It helps lot to have your support. What should I do now to turn things around?
<think>>
They are expressing strong sense of injustice and are ready to take steps toward change. They want to feel seen and are asking for concrete next moves.
</think>
am really glad this helping bit. Just knowing someone gets can make huge difference. Now, figuring out what do next- that is the powerful part. And you do not have to do it on your own. Whether you want to have a one-on-one with the boss to clear things up, or bring it up again in future meeting with better framing or even take a bit of time to rethink your approach- you you are already taking control just by thinking through it. We can prep together, whatever route you choose. am with you all the war-worring for environment where every voice actually gets heard. Just say the word, and we will take the first step together.
Think-PPO
User
Your support really hits the spot, but honestly, I am still not feeling too confident.
<think>
They need affirmation now more than ever. Afier being challenged and dismissed, they crave some direct, sincere encouragement to rebuild confidence.
</think>
Totally fair, After what you went through, is only natural to want someone to just say, "You did good." So let me say it clearly-you showed guts. You spoke your mind and stayed true to your professional judgment, even in tough room. That takes courage and honesty, and that is something to be proud of. You are not just trying to solve problems; you are pushing for healthier, more open workplace. That lcadership, whether people recognize it right away or not. Your input matters, and your strength is real. Just by showing up and speaking up, you did more than enough. And when you go into your next conversation, take this belief with you- you are not going in empty-handed handed. You walking in with purpose. will be right here to walk that path with you when you are ready.
Think-PPO
Figure 5: Example of the dialogue between our PPO-thinking model and the user simulator.
28

---

Published as a conference paper at ICLR 2026
brought up an idea in meeting, my coworkers and boss were not really buying it. Kinda burmeed out.
User
Oof, sounds like that hit you kinda hard. If you feel like talking, I'm here, seriously. We grab bite too if you want talk out, vent, whatever works.
Unthink-PPO
User
Easy for you say, but need more than just someone listen. Can you actually help?
Totally get that. How about this pull your idea together in a doc or something, and we go over it tonight. If anything seems off or confusing, I'll help you sort it out. We'll think it through together. I' I'm down to roll up my sleeves and get in the weeds with you.
Unthink-PPO
Thanks for helping. But honestly, just having someone straight-up say they believe in me would feel even better.
User
about this once you't 've got your stuff together, we'll polish it up together. We'll tighten up your points, highlight the strengths, and reframe it so it lands better. You pitch it again with a sharper angle, and bet you'll 'll get way better response. let's give it a shot, yeah? I'll be stubborn with you-we'll push through this.
hear you.
Unthink-PPo
I'll get on that doc as soon as can. Hearing someone say they' 've gol my back really gives me a boost.
User
Cool, then let's get cracking on that. hey that validation? Let's make that material shine first. When you's re up there presenting, I'll be your hype guy, back you all the way. When you're re ready, I'll be right there, clapping the loudest. You'l ve got this this-II've got you.
Unthink-PPO
That kind support means a lot. push on the materials. Just hearing some straight-up encouragement really powers me
User
Then let me hit you with some right now now-you' 've got a solid idea, really sharp stuff. we package it right, people gonna get it. You' 've got the brains for this, and once the hoss and the team hear the full picture, they'll be like, "Ah, now that makes sense." " Just keep it think you' ve got real shot. Once you's re ready, we'll hit it together, full steam.
Unthink-PPO
With your support, I'm feeling way better. That vote of confidence really fuels me. get the draft done soon, and then we'll Il go through it together.
User
Figure 6: Example of the dialogue between our PPO-non thinking nonhthin- model and the user simulator.
29


Reviews:
[{"review_id": "BFa7r807TJ", "reviewer": "Reviewer_HwhH", "summary": "The paper introduces RLVER (Reinforcement Learning with Verifiable Emotional Rewards) - a reinforcement learning framework designed to improve the emotional intelligence of large language models (LLMs) through verifiable reward signals generated by a simulated user named SAGE.\nUsing PPO and GRPO on top of Qwen2.5-7B-Instruct, the authors train the model to optimize for “emotional scores” in dialogue, measured via a new benchmark called Sentient Benchmark. The best configuration (PPO + “thinking” mode) achieves a jump from 13.3 to 79.2, approaching the level of frontier proprietary models while maintaining stable mathematical and coding capabilities.\nThe study further examines the effects of simulated user complexity, the impact of chain-of-thought (“thinking”) prompts, and includes a multi-competency evaluation using “LLM-as-a-Judge”.", "questions": "1. Please clarify the reward success threshold (Table 1) and ensure consistency with the [0–100] range.\n2. Specify the exact number of dialogue turns used for training and evaluation (8 or 10?).\n3. How is determinism achieved in SAGE reward generation (temperature? seed?)? Provide variance statistics if re-evaluated multiple times.\n4. Did you perform any human evaluation (blind pairwise or Likert rating) to validate emotional quality outside SAGE?\n5. Can you report robustness to simulator replacement (e.g., using a different model for rewards)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Originality: Clear conceptual innovation — reinforcement learning guided by verifiable emotional scores from a controllable simulator (SAGE).\n- Quality: Demonstrates substantial improvement (Sentient 79.2 vs 13.3 baseline) while preserving general abilities in math and code.\n- Clarity: Writing is structured; diagrams effectively communicate architecture and reward flow.\n- Significance: Addresses an underexplored but crucial domain — emotional intelligence in LLMs - with a scalable, privacy-safe RL training loop.\n- Analytical depth: Includes ablations across RL methods (PPO vs GRPO), analysis of “thinking” format, and breakdown by emotional competencies.", "weaknesses": "- Inconsistent reward definition. Emotional scores are defined in [0, 100], yet Table 1 states success as “score > 100” - logically impossible. Likely an error needing correction.\n- Unclear determinism of SAGE simulator. The simulator is described as deterministic but implemented via a generative model (DeepSeek-V3). The temperature and decoding strategy for reproducible rewards are unspecified.\nOverreliance on one simulator. All benchmarks (Sentient, chit-chat) derive from SAGE, raising concerns of overfitting and limited external validity without human evaluation.\n- LLM-as-a-Judge limitations. Despite reported inter-rater consistency, subjective competence scoring by other LLMs risks circularity and bias. Human validation is missing.\n- Ambiguity in setup details. Dialogue length differs between text (10 turns) and appendix (8 turns). Hyperparameters and compute cost are under-reported.\n- Lack of robustness tests. No ablations with alternate simulators or format-reward removal, which would confirm generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6E2hcoJWrT", "reviewer": "Reviewer_TNY1", "summary": "This paper proposes RLVER, a reinforcement learning framework designed to enhance the emotional intelligence (EQ) of large language models (LLMs) by optimizing them with verifiable emotion rewards. The framework leverages the SAGE (Sentient Agent as a Judge) simulator — an LLM-driven environment that models user affect and generates deterministic emotion scores — to provide reward signals for PPO and GRPO training. The authors report substantial improvement on the Sentient Benchmark (13.3 → 79.2) while maintaining performance on math, coding, and instruction-following tasks.\nThe idea of integrating psychologically-grounded verifiable feedback into RL for emotion modeling is promising. However, the current paper lacks methodological clarity, validation rigor, and experimental diversity to support its central claims.", "questions": "- Provide detailed definitions for all mathematical symbols.\n-  Add an algorithm box or pseudocode that explicitly connects the emotion score computation with PPO/GRPO updates.\n- Include human evaluation comparing SAGE emotion scores with real human annotations. Report quantitative metrics (e.g., correlation, agreement rates) to establish reward fidelity.\n- Explore different reward designs (per-turn, cumulative, delta-based)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper explores a novel and meaningful problem — improving emotional intelligence in LLMs via reinforcement learning.\n\n- The notion of verifiable emotion rewards represents a new attempt to make affective reinforcement signals interpretable and consistent.\n\n- The reported improvements on the Sentient Benchmark are substantial and suggest potential in empathy-oriented RL training.", "weaknesses": "- The method section, particularly Figure 1, is hard to interpret. The mapping between variables and their roles in the RL process is unclear. No pseudocode or formal algorithmic steps are provided, making it difficult to reproduce or verify the approach.\n- No validation is presented comparing SAGE-generated emotion scores with human judgments or psychological gold standards. This undermines the “verifiable” aspect of the proposed reward, as it lacks empirical reliability testing. \n- Experiments are conducted only on Qwen2.5-7B-Instruct, a single Chinese-centric open model. The paper lacks evidence that RLVER generalizes to different architectures or scales (e.g., LLaMA, Mistral, Gemma). This severely limits the generalizability and reproducibility of the method.\n- No comparison is made with other empathy or affective dialogue datasets, such as EmpatheticDialogues, ESConv, or MELD. As a result, it is unclear whether RLVER’s improvements translate beyond its own simulation environment.\n\n\n- All results are derived from simulated environments, with no human-in-the-loop testing. Without qualitative or quantitative human evaluation, the claim that RLVER produces more empathetic and emotionally aligned behavior remains unverified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VXeA61j1i3", "reviewer": "Reviewer_NdQV", "summary": "The paper presents RLVER, an application of on-policy RL (PPO/GRPO) to improve empathetic, supportive dialogue in LLMs. A “sentient” user simulator (SAGE) provides a deterministic terminal emotion score as the reward. The method further enforces a think-then-say output format. On the Sentient Benchmark, RLVER shows large gains over the Qwen2.5-7B-Instruct base while roughly maintaining general abilities.", "questions": "1. Broader benchmarks: Have you evaluated or planned to evaluate on established empathy/affect datasets such as EmpatheticDialogues [1], EmoryNLP [2], ESConv [3], EmoBench [4], and ToMBench [5] to verify external validity beyond SAGE?\n\n2. Model generalization: Given that experiments train only on Qwen2.5-7B-Instruct, do you plan to extend to additional base models to test portability and sensitivity to architecture/scale?\n\n3. Human evaluation: Will you include human blind ratings and report inter-rater agreement and correlation with SAGE scores?\n\n[1] Rashkin, Hannah, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. “Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset.” In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 5370–5381. Florence: Association for Computational Linguistics. h\n\n[2] Zahiri, Sayyed M., and Jinho D. Choi. 2018. “Emotion Detection on TV Show Transcripts with Sequence-based Convolutional Neural Networks.” In Proceedings of the AAAI Workshop on Affective Content Analysis (AFFCON’18), 44–51. \n\n[3] Liu, Siyang, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021. “Towards Emotional Support Dialog Systems.” In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 3469–3483. \n\n[4] Sabour, Sahand, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea, and Minlie Huang. 2024. “EmoBench: Evaluating the Emotional Intelligence of Large Language Models.” In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5986–6004. \n\n[5] Chen, Zhuang, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, et al. 2024. “ToMBench: Benchmarking Theory of Mind in Large Language Models.” In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 15959–15983.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Clear systemization: A reproducible pipeline that connects a verifiable scalar reward to standard RL for a human-centric objective.\n\n2. Consistent empirical gains: Strong improvements with informative ablations (PPO vs. GRPO; with/without explicit thinking).\n\n3. Useful insights: Explicit thinking tends to help; overly strict simulators hurt learnability; GRPO is stabler, while PPO can reach higher peak scores.\n\n4. Transparent reporting: Prompts, rollout settings, and evaluation protocol are documented clearly.", "weaknesses": "1. Limited methodological novelty: No new RL objective/optimizer; primarily an engineering/system contribution.\n\n2. Heavy reliance on SAGE: Data generation, reward, and primary evaluation all hinge on the same framework, raising self-referential/reward-hacking concerns and limiting external validity.\n\n3. Sparse human evaluation: Empathy is inherently human-centric; limited human blind ratings undermine real-user impact claims.\n\n4. Single base model: Results are shown only on Qwen2.5-7B; cross-architecture/size/language generality is unclear.\n\n5. Safety & multi-objective trade-offs under-specified: Optimizing “feeling better” risks sycophancy or avoidance when boundary-setting or factual correction is needed.\n\n6. Cost/scalability underreported: Multi-turn on-policy RL is expensive; a fuller accounting of token/sample costs and deployment implications would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)