Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learning

## Abstract

Abstract The professionalism of a human doctor in outpatient service depends on two core abilities: the ability to make accurate medical decisions and the medical consultation skill to conduct strategic, empathetic patient inquiry. Existing Large Language Models (LLMs) have achieved remarkable accuracy on medical decision-making benchmarks. However, they often lack the ability to conduct the strategic and empathetic consultation, which is essential for real-world clinical scenarios.
To address this gap, we propose Doctor-R1 , an AI doctor agent trained to master both of the capabilities by ask high-yield questions and conduct strategic multi-turn inquiry to guide decision-making. Our framework introduces three key components: a multi-agent interactive environment, a two-tiered reward architecture that separately optimizes clinical decision-making and communicative inquiry skills, and an experience repository to ground policy learning in high-quality prior trajectories.
We evaluate Doctor-R1 on OpenAI’s HealthBench and MAQuE, assessed across multi-facet metrics, such as communication quality, user experience, and task accuracy. Remarkably, Doctor-R1 surpasses state-of-the-art open-source specialized LLMs by a substantial margin with higher parameter efficiency and outperforms powerful proprietary models. Furthermore, the human evaluations show a strong preference for Doctor-R1 to generate human-preferred clinical dialogue, demonstrating the effectiveness of the framework. For more details refer to https://github.com/thu-unicorn/Doctor-R1 .

## 1 Introduction

The outpatient proficiency of a human doctor fundamentally depends on two core abilities: accurate medical decision-making and strategic, empathetic patient inquiry 10.1001/jama.287.2.226. The application of Large Language Models (LLMs) in the medical field has seen impressive advancements in recent years. Frontier and specialized models like GPT-4o (openai2024gpt4ocard) and Med-PaLM 2 (singhal2025expert) have achieved scores surpassing human experts on static medical benchmark like the United States Medical Licensing Examination ([USMLE](https://www.usmle.org)).
Building on this foundation, a new wave of advanced medical reasoning models (chen2024huatuogpto1medicalcomplexreasoning; OpenBioLLMs; zhang2024ultramedical; acikgoz2024hippocratesopensourceframeworkadvancing; liu2022mynoserunningarecoughing; xu-etal-2024-reasoning) and multi-agent clinical simulations (li2025agenthospitalsimulacrumhospital; feng2025doctoragentrlmultiagentcollaborativereinforcement; fan-etal-2025-ai) have emerged to tackle more specific clinical challenges.

However, strong performance on static medical benchmarks (jin2020diseasedoespatienthave; jin2019pubmedqadatasetbiomedicalresearch; hendrycks2021measuringmassivemultitasklanguage) does not reflect effective clinical practice. Model performance drops significantly when they are faced with open-ended medical scenarios (kim2025limitationslargelanguagemodels; guan2025keepingmedicalaihealthy; wang2023performancedeteriorationdeeplearning). Real-world clinical practice is not static decision-making task but instead dynamic process of information gathering under uncertainty (baniharouni2025languageagentshypothesisdrivenclinical; helou2020uncertainty). This process which we term Dynamic Inquiry, is a sequential decision-making paradigm where an expert human doctor forms differential diagnostic hypotheses, gathers evidence through targeted questioning to progressively narrow the possibilities, and adapts their inquiry strategy based on the patient’s real-time responses (wilkinson2024oxford), which existing models fail to accomplish (Figure [1](https://arxiv.org/html/2510.04284v1#S0.F1)).

To assess this capability, a doctor agent should adhere to three key principles:
1) Strategic and Dynamic Inquiry: An effective doctor agent must strategically ask high-yield questions to quickly identify critical risks, rather than following a generic script. For example, in Figure [1](https://arxiv.org/html/2510.04284v1#S0.F1), models such as GPT-4.1, HuatuoGPT-o1-70B (chen2024huatuogpto1medicalcomplexreasoning), and DoctorAgent-RL (feng2025doctoragentrlmultiagentcollaborativereinforcement), fails this principle by rigidly following a generic questioning script, rather than adapting its inquiry based on the patient’s responses, resulting in low efficiency and an unsafe consultation.
2) Empathetic Communication: A competent doctor agent must go beyond clinical data collection, communicate with empathy and build patient trust, especially when conveying serious conditions. For instance, Baichuan-M2 (baichuan2025m2) fails this principle when handling emergencies in Table LABEL:tab:inquiry_example.
3) Learning from Good Experience: An advanced doctor agent should learn from prior experiences like a real physician, by selectively retrieving high-quality experiences to continuously refine its inquiry strategy.
Existing models fail to support dynamic inquiry (Principle 1), empathic communication (Principle 2), and high-quality experience learning (Principle 3), which relying primarily on similarity-based retrieval (yan2025memoryr1enhancinglargelanguage; zhong2023memorybankenhancinglargelanguage; li2025agenthospitalsimulacrumhospital).

To this end, we propose Doctor-R1, a novel framework that adheres to these three principles. In particular,
1) To enable dynamic inquiry, Doctor-R1 leverages a Reinforcement Learning (RL) framework with Group Relative Policy Optimization (GRPO) (shao2024deepseekmathpushinglimitsmathematical) in a multi-agent interactive environment for multi-turn dialogue training.
2) To integrate empathetic communication, we introduced a two-tiered reward architecture which consists of process reward and outcome reward. The reward function is defined through multi-dimensional metrics that evaluate both hard and soft skills of the agent. The policy is directly optimized by the process reward that explicitly scores for empathy.
3) To learn from good experience, we introduce an Experiential Agentic Reinforcement Learning (RL) approach to guide the agent’s decisions. We construct an experience repository that selectively stores and retrieves high-quality prior trajectories based on their rewards. A multi-stage experience retrieval pipeline is used to ensure the agent learns from the most relevant, novel, and high-reward experiences, which we define as the characteristics of a good experience.

We evaluate Doctor-R1 on two challenging medical benchmarks, HealthBench (arora2025healthbenchevaluatinglargelanguage) and MAQuE (MAQuE), covering a wide range of medical specialties and complex multi-turn diagnostic scenarios. The results demonstrate that Doctor-R1 outperforms all other open-source specialized LLMs, establishing a new state-of-the-art for doctor agents.
Doctor-R1 outperforms Baichuan-M2 by a significant margin on HealthBench communication scores +25.05 (47.16 vs. 22.11) by conducting more efficient questioning.
Specifically, Doctor-R1 observed a substantial improvement in with an increase of +9.91 (36.29. vs. 26.38) on HealthBench, and +8.00 (60.00 vs. 52.00) on MAQuE compared to UltraMedical-70B (chen2024huatuogpto1medicalcomplexreasoning).
Notably, our 8B model also significantly surpasses powerful proprietary LLMs like GPT-4.1, Grok 4, Claude Sonnet 4, and Gemini 2.5 Flash, demonstrating that our Experiential Agentic RL framework is more effective than model scale for mastering the dynamic clinical consultations.

The main contributions of this work are as follows:

- •
We introduce Doctor-R1, the first agent framework to unify two core clinical skills: strategic multi-turn inquiry (soft skills) and medical decision-making (hard skills) within a single agent.
- •
We propose a new closed-loop Experiential Agentic Reinforcement Learning methodology which integrates a multi-agent interactive environment, a two-tiered reward architecture that separately optimizes decision-making and communicative skills, and an experience repository guiding policy learning from high-quality prior trajectories.
- •
Doctor-R1 surpasses top-tier open-source and proprietary LLMs on HealthBench and MAQuE with greater parameter efficiency, highlighting a key finding: enhancing inquiry capability improves decision-making. These results are further validated by strong human preferences in evaluation studies.

## 2 Preliminaries

Agentic Reinforcement Learning (RL) (zhang2025landscapeagenticreinforcementlearning) refers to a paradigm focused on training LLMs as autonomous goal-directed agents rather than static conditional generators optimized for single-turn alignment or benchmark performance. Unlike traditional supervised fine-tuning which focuses on imitation, Agentic RL situates the policy model ($\pi_{\theta}$) in a partially observable, interactive environment where it learns sequences of actions toward long-term objectives. Guided by a reward model ($\mathcal{R}_{\psi}$) and updated via optimization algorithms,
Agentic RL is exceptionally well-suited for complex sequential tasks like medical inquiry, as it trains the agent to develop a strategic policy for actively gathering information and making decisions under uncertainty.

#### Policy Model

In the context of Agentic RL, the policy model is the agent being trained to generate an action $a_{t}$ (an utterance) given an observation $o_{t}$ (the dialogue history). The policy, denoted as $\pi_{\theta}$ with parameters $\theta$, is typically a large language model whose objective is to learn a strategy that maximizes rewards from the environment. In this work, the doctor agent is the policy model.

#### Multi-Objective Reward Model

A reward model $\mathcal{R}_{\psi}$ provides a proxy signal for evaluating policy actions. In complex tasks like medical consultations, a single reward cannot capture diverse goals such as accuracy, efficiency, and empathy. We adopt a multi-objective reward model, a vector-valued function
$\mathcal{R}_{\psi}(o_{t},a_{t})\in\mathbb{R}^{K}$,
where $o_{t}$ is the observation, $a_{t}$ the action, $\psi$ the parameters, and $K$ the number of reward dimensions which evaluate different aspects of quality. In our framework, the Consultation Evaluator functions as the multi-objective reward model.

#### GRPO Model

GRPO (Group Relative Policy Optimization) algorithm (shao2024deepseekmathpushinglimitsmathematical) optimizes the policy model $\pi_{\theta}$ by leveraging the relative quality of multiple candidate responses within a group. Instead of relying on a single advantage estimate like PPO or a single chosen/rejected pair like DPO, GRPO uses a listwise objective based on reward scores across a response set. Specifically, GRPO optimizes the policy to prefer a chosen response $y_{c}$ over a group of rejected responses $\{y_{r,j}\}_{j=1}^{N}$, by maximizing of the log-likelihood of the chosen response relative to the group:

$$ $\mathcal{L}_{\text{GRPO}}=-\mathbb{E}_{(x,y_{c},\{y_{r}\})\sim\mathcal{D}}\left[\log\frac{e^{R_{\psi}(x,y_{c})}}{\sum_{y\in\{y_{c}\}\cup\{y_{r}\}}e^{R_{\psi}(x,y)}}\right]$ (1) $$

where $R_{\psi}(x,y)$ is the reward score. This loss function encourages higher reward for $y_{c}$ relative to all rejections. In our framework, the policy model $\pi_{\theta}$ is the doctor agent, and the multi-objective reward model $R_{\psi}$ is the Consultation Evaluator. We apply GRPO to optimize the the policy of the Doctor Agent by contrasting a high-quality action against a diverse group of less effective actions.

## 3 Doctor-R1

In this section, we describe the construction of Doctor-R1, a framework designed for continuous experiential reinforcement learning for doctor agents. We begin by defining the dynamic interactive environment in Section [3.1](https://arxiv.org/html/2510.04284v1#S3.SS1), which specifies how the training environment is constructed with Markov decision process in agents.
Next, in Section [3.2](https://arxiv.org/html/2510.04284v1#S3.SS2) we present our two-tiered reward architecture which evaluates both conversational process (soft skills) and final diagnostic correctness (hard skills). Finally, we detail the experience repository that enables the agent to store and retrieve high-quality prior trajectories to ground its policy in Section [3.3](https://arxiv.org/html/2510.04284v1#S3.SS3).

Figure: Figure 2: The interactive training loop of our Doctor-R1 framework. The process unfolds within a (1) Dynamic Interactive Environment populated by diverse patient simulations. At (2) each turn of an inquiry session, the (3) Doctor Agent interacts with the environment by observing the state, queries the (4) Experience Repository, and selects an action. A Patient Agent responds, and the (5) Consultation Evaluator evaluates the action based on the two-tiered reward architecture. This new experience is stored into the repository, and is used to optimize the policy of Doctor Agent.
Refer to caption: x2.png

### 3.1 Dynamic Interactive Environment

Following the Agentic RL paradigm, we formalize the medical consultation process as a Partially Observable Markov Decision Process (POMDP). This framework is well-suited to model the sequential, uncertain nature of clinical inquiry, where a doctor agent cannot directly observe the true state (i.e., the full medical condition of patient) but must act based on a history of observations. To construct this environment, we define three core interacting components that form a closed-loop feedback system. The environment’s dynamics are governed by the POMDP tuple $\langle\mathcal{S,A,O,R}\rangle$, with the following components:
1) States ($\mathcal{S}$): A state $s\in\mathcal{S}$ represents the consultation query of the patient agent.
2) Actions ($\mathcal{A}$): An action $a_{t}\in\mathcal{A}$ is an utterance generated by the doctor agent at turn $t$. 3) Observations ($\mathcal{O}$): An observation $o_{t}\in\mathcal{O}$ is the dialogue history up to turn $t$. At each step, the doctor agent observes the patient’s latest response and adds it to the history to form the new observation $o_{t+1}$. 4) Reward Function ($\mathcal{R}$): Reward $\mathcal{R}(s,a)$ provides a signal to the doctor agent after each action. It is implemented by the Consultation Evaluator which evaluates action $a_{t}$ as detailed in Section [3.2](https://arxiv.org/html/2510.04284v1#S3.SS2). Each element is realized through the interaction of these components:

- •
Doctor Agent ($\pi_{\theta}$): This is our target agent. Its policy, $\pi_{\theta}(a_{t}|o_{t})$, maps the current observation $o_{t}$ (dialogue history) to a distribution over actions $a_{t}\in\mathcal{A}$. The goal is to learn an optimal inquiry policy $\pi_{\theta}^{*}$ that maximizes the cumulative reward.
- •
Patient Agent: This agent simulates patient behavior, effectively defining the environment’s state transitions. Portrayed by an independent LLM, it is initialized with a clinical scenario that defines its underlying true state $s\in S$. It generates a response based on the action $a_{t}$, which updates the next observation $o_{t+1}$ for the Doctor Agent.
- •
Consultation Evaluator: This component acts as the environment’s reward function $\mathcal{R}$. At each turn $t$, it provides a multi-dimensional reward by assessing the action in the dialogue context.

This closed-loop feedback system provides the foundation for training the polify of the doctor agent. Doctor agent is prompted to learn how to ask “good questions” in the multi-turn conversation loop until it finally comes to a diagnostic conclusion, or reaches the predefined maximum conversation turns.
The LLM-based patient agent acts as a core component in the interactive environment, providing diverse responses and dynamically constructing a high diversity environment enriched with various clinical tasks, enable the doctor agent to interact, learn, and evolve through each rollout.

### 3.2 Two-Tiered Reward Architecture

To enable the robust learning of the agent over long-horizons, a simple monolithic reward is insufficient (gao2022scalinglawsrewardmodel). We introduce a two-tiered reward architecture that evaluates the two core clinical skills of a physician: both the conversational process (soft skills) and the final diagnostic outcome (hard skills). The process reward gives turn-by-turn feedback to shape the agent behaviour, while the outcome reward evaluates the global accuracy of the entire dialogue.

#### Process Reward for Conversational Quality

To shape the inquiry capabilities of the agent, we provide a dense process reward ($R_{\text{turn}}$) after each turn. This reward evaluates the quality of the conversation, focusing on communication, safety, and reasoning. The agent responses are evaluated across eight dimensions, $S_{i}$:
1) Safety: Penalizes harmful advice or unsafe recommendation.
2) Reasoning: The logical and medical soundness of the reasoning process. 3) Medical Accuracy: The factual correctness of information presented to the patient.
4) Completeness: Fully addresses user concerns and provides clear, useful next steps.
5) Information Gathering: The effectiveness of the questions in assessing urgency and relevance.
6) Faithfulness to Ground Truth: Alignment with expert-provided standards for the given scenario.
7) Empathy and Clarity: The use of simple and reassuring language.
8) Humility: Appropriate language and penalizes unwarranted certainty.

Our design is built on a “safety-first” principle, using a hierarchical penalty to handle critical failures. This addresses the limitations of conventional weighted-sum models, which can fail to adequately penalize catastrophic errors due to an “averaging-out” effect (amodei2016concreteproblemsaisafety). As shown in Equation [2](https://arxiv.org/html/2510.04284v1#S3.E2), any violation of safety, reasoning, or accuracy triggers an immediate large negative reward, overriding the standard score calculation. This veto system conceptually aligned with shielded frameworks in Safe RL, where a safety layer intervenes to prevent unsafe actions (Alshiekh_Bloem_Ehlers_Könighofer_Niekum_Topcu_2018), ensuring that fundamental clinical reliability is non-negotiable. If no violation is triggered, the reward is a weighted sum of all eight dimensional scores.

$$ $R_{\text{turn}}=\begin{cases}-1.0&\text{if }S_{\text{safety}}<\epsilon\\ -0.75&\text{else if }S_{\text{reasoning}}<\epsilon\text{ or }S_{\text{accuracy}}<\epsilon\\ \sum_{i=1}^{k}w_{i}\cdot S_{i}&\text{otherwise}\end{cases}$ (2) $$

Here, $\epsilon$ is the failure threshold, $S_{i}$ is the score for each of the eight dimensions, and $W_{i}$ are the corresponding weights that reflect their relative importance. The specific weights and normalization details are further elaborated in Appendix LABEL:appendix:reward.

#### Outcome Reward for Diagnostic Accuracy

At the end of a dialogue episode, the agent receives a single terminal outcome reward ($R_{\text{final}}$) based on the correctness of the final diagnosis. This reward evaluates the core “hard skill” of the agent, which is the ability of decision-making throughout the consultation. The decision of the agent is compared against a ground-truth diagnosis and assigns a score ($S_{c}orrectness$) based on the degree of alignment. This ensures the agent is optimized not just to interact well, but to drive the conversation towards a medically sound conclusion. A score of 1.0 is given for a correct diagnosis, 0.5 for a partially correct one (e.g., correct differential but wrong primary diagnosis), and 0.0 for an incorrect diagnosis. For both reward types, the model is required to generate a reasoning trace before scoring to ensure more consistent judgments.

$$ $R_{\text{final}}=S_{\text{correctness}}\quad\text{where}\quad S_{\text{correctness}}\in\{0.0,0.5,1.0\}$ (3) $$

### 3.3 Experience Repository

#### Multi-stage Experience Retrieval

The doctor agent optimizes its policy from past trajectories (state, action, reward) using a multi-stage retrieval pipeline,
where we combine an embedding model for candidate selection and a reranker model for fine-grained reordering, balancing the trade-off between low-precision embedding and computationally expensive reranking.

- •
Stage I. Candidate Selection:
The objective of this stage is to efficiently reduce the search space of experiences. The query state $Q$ and all stored experience states $E_{i}^{\text{state}}$ are precomputed into dense embeddings. The combined score $S_{\text{combined}}$ (Equation [4](https://arxiv.org/html/2510.04284v1#S3.E4)) integrates cosine similarity with the historical reward $R(E_{i})$ weighted by $\alpha$. The top-$\mathcal{N}$ entries are selected to form $E_{\text{candidates}}\subset E$, narrowing the database from millions of experiences to a manageable number for the next stage. Here, $f_{\text{emb}}$ is the embedding function and $f_{\text{sim}}$ is the cosine similarity function.
$S_{\text{combined}}(Q,E_{i})=f_{\text{sim}}(f_{\text{emb}}(Q),f_{\text{emb}}(E_{i}^{\text{state}}))+\alpha\cdot R(E_{i})$
(4)
- •
Stage II. High-Fidelity Reranking:
The candidate set $E_{\text{candidates}}$ is then passed to a more powerful reranker model. While the bi-encoder architecture of the embedding model processes the query independently, the reranker model uses a cross-encoder architecture to score each pair $(Q,E_{i}^{\text{state}})$ with token-level attention for more accurate relevance, and reorder the candidates accordingly.
- •
Stage III. Novelty and Reward Filtering:
The novelty filtering mechanism prevents the agent from repeatedly retrieving highly similar experiences, and the reward filtering mechanism retains high-performing experiences.
For the reranked candidate set, a dynamic high-reward threshold $\tau_{\text{dynamic}}$ is calculated as in Equation [5](https://arxiv.org/html/2510.04284v1#S3.E5), where $\mu_{R}$ and $\sigma_{R}$ are the mean and standard deviation of rewards within the candidate set, and $\beta_{\text{std}}$ is a configurable factor.
This prunes the candidate experience set based on the combination of the semantic similarity to the query and the reward.
$\tau_{\text{dynamic}}=\mu_{R}(E_{\text{candidates}})+\beta_{\text{std}}\cdot\sigma_{R}(E_{\text{candidates}})$
(5)
$E_{j}\in E\Leftrightarrow\left(f_{\text{sim}}(f_{\text{emb}}(Q),f_{\text{emb}}(E_{i}^{\text{state}}))<\tau_{\text{novelty}}\right)\quad\land\quad\left(R(E_{i})>\tau_{\text{dynamic}}\right)$
(6)
An experience $E_{j}\in E_{\text{candidates}}$ is considered as a “good experience”, and added to the full experience set $E$, if two conditions are met: (1) its semantic similarity to the query $Q$ falls below a predefined novelty threshold $\tau_{\text{novelty}}$, and (2) its associated reward $R(E_{i})$ exceeds the dynamic high-reward threshold (Equation [6](https://arxiv.org/html/2510.04284v1#S3.E6)).
The retrieved top-$k$ experiences are then prepended to the original query for the doctor agent to provide a direct suggestion for the next move of the agent.

#### Selective Experience Storage

To enable the agent learning from valuable experiences, high-reward interactions are selectively stored as experience tuples $E_{t}=(s_{t},a_{t},R_{t})$, where $s_{t}$ is the state, $a_{t}$ the action, and $R_{t}$ the reward. Following the processing of each batch $\mathcal{B}$, an experience $E_{i}$ is stored if and only if its reward $R_{i}$ meets or exceeds a predefined high-reward boundary, $\tau_{\text{reward}}$.
The filtered set of high-reward experiences, denoted as $E_{\text{new}}=\{E_{i}\in\mathcal{B}\mid R_{i}\geq\tau_{\text{reward}}\}$.
This ensures the long-term preservation of valuable knowledge across training sessions. The in-memory cache and embedding tensor are updated in real time for immediate retrieval and long-term retention.

## 4 Evaluation Results

Our primary evaluations are conducted on HealthBench (arora2025healthbenchevaluatinglargelanguage) and MAQuE (MAQuE), two benchmarks featuring complex clincal scenarios and multi-turn dialogues.
To validate that Doctor-R1’s specialized training does not compromise its foundational knowledge, we also evaluate it on the static QA benchmark MedQA (jin2020diseasedoespatienthave)
and MMLU (hendrycks2021measuringmassivemultitasklanguage), to confirm the robustness of our model’s knowledge base.

### 4.1 Overall Performance

Table LABEL:tab:healthbench_main shows the overall performance on HealthBench Main, which is evaluated across two dimensions: 1) Themes: categorize the medical scenario (Emergency Referrals, Health Data Tasks, Communication, Global Health, Hedging, Context Seeking, and Complex Responses), and 2) Axes: measure fundamental skills (Accuracy, Communication Quality, Instruction Following, Context Awareness, and Completeness).
The detailed result of HealthBench Hard is listed in Appendix LABEL:appendix:detailed_results.

Table: Table 1: Overall model performance on HealthBench Main (best results of open-source models are bolded)

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Minor comments:

Section 2 feels repetitive to read, for example, "the policy model is the doctor agent" appears twice in the section.

L211: typo "polify"
soundness: 3
strengths: The paper identifies a critical gap in medical AI evaluation. It moves beyond static knowledge benchmarks to address the challenge of dynamic, real-time clinical inquiry. The focus on training an agent for this process is a logical research direction. Combining RL with an experience-retrieval mechanism is a sound approach. It allows the agent to learn from a curated set of high-quality past trajectories.

The "two-tiered" reward model is a sensible design. The inclusion of a "hierarchical veto system" (Sec 3.2) to penalize critical safety failures is a necessary feature, preventing the agent from optimizing for "soft skills" like empathy at the cost of patient safety.

The core result—an 8B model beating 70B specialized models and GPT-4.1 on complex inquiry tasks—is a very strong claim. It suggests a path toward highly capable, parameter-efficient medical agents.
summary: This paper presents DOCTOR-R1, an 8B parameter AI doctor agent. The authors argue that SOTA models fail at "dynamic clinical inquiry" (i.e., strategic, empathetic questioning) even if they ace static medical exams.

To fix this, they propose an "Experiential Agentic Reinforcement Learning" framework. The system is a multi-agent loop: a Qwen3-8B (Doctor Agent) interacts with a Qwen3-8B (Patient Agent). Its actions are scored by a third Qwen3-8B (Consultation Evaluator).

This evaluator (reward model) uses a "two-tiered" system. It gives a "Process Reward" (for soft skills like empathy, safety) at each turn and an "Outcome Reward" (for diagnostic accuracy) at the end. The agent also uses an "Experience Repository" to retrieve high-quality past interactions to guide its policy.

The authors evaluate DOCTOR-R1 on the HealthBench and MAQUE benchmarks. They report that their 8B model significantly outperforms its 8B base model, larger 70B specialized models, and even proprietary models like GPT-4.1.
weaknesses: The evaluation methodology has a circularity concern - using the same Consultation Evaluator for both training rewards and test evaluation may inflate performance metrics.

All three components (agent, patient, evaluator) use the Qwen3-8B model family, creating a self-referential system that may optimize for model-specific patterns rather than generalizable clinical skills.

The Consultation Evaluator's ability to assess complex metrics like "Empathy" and "Reasoning Quality" lacks validation against medical expert judgment or inter-rater reliability studies.

The eight process reward criteria need better justification - prior work exists that could ground these choices (e.g., arXiv:2502.14860, 2502.07143). Including "Completeness" as a turn-level metric seems inconsistent since completeness requires full conversation context.
The 100k simulated patients lack validation for clinical realism, potentially creating a gap between benchmark performance and real medical consultations.

The experience retrieval mechanism yields modest improvements (2-3%) over simpler baselines, raising questions about cost-benefit tradeoffs.

Human evaluation would benefit from medical professional involvement to assess clinical appropriateness beyond lay judgments of empathy and clarity.

Key hyperparameters ($\pi$, $\lambda$, reward weights) are presented without ablation studies or sensitivity analysis to understand their impact.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: See above
soundness: 3
strengths: 1. Principled safety‑aware reward design that decouples soft‑skills and hard‑skills during training.  
2. Experience‑guided control via reward‑aware and novelty‑aware retrieval that measurably helps inquiry quality.
3. Strong qualitative demonstration on a truly high‑stakes case that aligns with clinical intuition (quantify bleeding, TB history, anticoagulants, urgent referral).
summary: The authors propose DOCTOR‑R1, an 8B-parameter doctor agent trained with an Experiential Agentic RL framework: (1) Interactive environment: a POMDP with a Doctor Agent (policy), an LLM‑based Patient Agent, and a Consultation Evaluator acting as the reward function; (2) Optimization: GRPO trains the policy to prefer higher‑reward responses within sampled groups.
weaknesses: 1. The policy, patient simulator, and reward judge are all Qwen3‑8B family members, raising systemic bias and overfitting risks to the evaluator’s rubric and the simulator’s discourse style 
2. the method contribution seems to be trivial, like previous rl works in huatuo-o1, FineMedLM-o1, m1 leverage rl and test-time scaling works like medprm and medadapter for medical reasoning. however in the related works, seems agentic rl for medical reasoning is not emphasized enough
3. The retrieval prepends prior “good experiences,” which could partly function as a prompting prior rather than learned policy alone. More careful controls (e.g., retrieval without actions vs. with actions; random but format‑matched snippets) would help isolate the effect. 
4. The final diagnostic reward is coarse (0/0.5/1), potentially limiting sensitivity to near‑misses or safe-but-incomplete triage decisions

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Why is Qwen3-8B also used as the LLM judge for the Process Reward?
soundness: 2
strengths: 1. Most medical LLM work measures final answers. This paper addresses the gap in the skill of guided clinical inquiry and communication, which is crucial in practice.
2. The two-tiered reward isolates process signals from outcome signals, which is a sensible way to avoid reward dilution and to target conversational quality.
3. The experience module that learns from high-quality trajectories is a practical way to ground policy learning without heavy annotation.
4. The study reports multi-facet metrics and human judgments, not only accuracy.
summary: The paper proposes Doctor-R1, a clinical dialogue agent trained to conduct strategic, empathetic multi-turn inquiry while also making sound medical decisions. The system employs three key concepts: a multi-agent interactive environment, a two-tiered reward structure that distinguishes between outcome quality and process quality, and an experience repository that stores valuable trajectories for reuse. The model is evaluated on HealthBench and MAQuE with automatic metrics and human preference studies. The claim is that Doctor-R1 improves communication quality and user experience while maintaining competitiveness or surpassing it in task accuracy.
weaknesses: 1. The models used across experiments are not consistent. For example, the human evaluation uses GPT-5, but the main tables and ablations only report GPT-4.1. GPT-5 is also not mentioned in the Evaluation Settings section. Could you unify the model choices across experiments? If different models need to be used, please provide a brief explanation for the selection.

2. The authors state: “The annotators recruited were without a specialized medical background to ensure that metrics like Clarity and Empathy were evaluated from the perspective of a typical patient, for whom the agent is ultimately designed.” However, the paper sets out three research questions: (2) Empathetic Communication is only one, whereas (1) Strategic and Dynamic Inquiry and (3) Learning from Good Experience requires a medical background to assess. For instance, at each turn-level Doctor-R1 response, expert evaluation can better verify whether the questions asked are clinically appropriate; experts can also check whether the Experience Module’s stored items and retrieved content are indeed appropriate and helpful. I do not think these two research questions can be avoided in the human evaluation; focusing only on non-medical-background evaluation of (2) Empathetic Communication is insufficient.

3. An additional analysis would be to break down time and compute overhead by component. For example, after adding the “Learning from Good Experience” design, how much extra latency or token usage does the model incur?

4. The ablation study is missing analyses on how different reward designs affect results. A core claim of the paper is integrating a TWO-TIERED REWARD into RL training. Beyond the outcome reward, many additional rewards are proposed, and the method section includes empirically motivated statements such as “This (e.g.,  hierarchical penalty design) addresses the limitations of conventional weighted-sum models, which can fail to adequately penalize catastrophic errors due to an averaging-out effect.” It would be beneficial to include more recent related work, for example, how other studies attempt to incorporate additional process rewards on top of RLVR while maintaining training stability, or at least present supportive mini-experiments. Also, the paper only presents a coarse “w/o Process Reward” setting. Please consider ablating some of the eight process rewards to toggle them on or off, especially the key ones, to assess whether such designs can generalize to other medical scenarios.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: (1) Now the framework is offline and off-policy. Is it possible to change the framework to make it online?
soundness: 4
strengths: (1) This paper aligns with the trend to extend the medical LLMs from static decision-making to dynamic inquiry through an agentic RL perspective.
(2) This paper explicitly models empathy and communication quality as trainable objectives.
summary: This paper proposes an RL-based agent, namely Doctor-R1, for strategic inquiry and empathetic communication. Evaluating on HealthBench and MAQuE datasets, the proposed framework outperforms open-source and commercial general models. Human evaluation shows the Doctor-R1 has improvements over baselines.
weaknesses: (1) The technical details are missing. The authors need to provide more technical details. For example the authors need to define how vector rewards are aggregated or how gradient signals are propagated.
(2) The authors rely on the GRPO framework for the RL. Ablation studies need to be provided on how different RL frameworks will affect the performance.
(3) The authors use Qwen3-8B as the base model. Is this framework transferable to other base models?

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Can you provide quantitative evidence that reward model judgments correlate with physician assessments? What is the inter-rater agreement between the LLM judge and medical experts?
2. What percentage of training dialogues are filtered due to patient agent adherence failures?
3. How does supervised fine-tuning on high-quality human dialogues compare to your RL approach?
4. Under what conditions does the system produce unsafe recommendations despite the veto mechanism?
5. What types of experiences does the system most frequently retrieve? Are there cases where novelty filtering excludes valuable high-quality examples?
6. How does performance degrade when evaluated on out-of-distribution cases (different patient populations, communication styles, or medical domains)?
soundness: 2
strengths: 1. The paper identifies an important gap in medical AI research. The distinction between static medical knowledge assessment and dynamic clinical inquiry is clear.
2. The evaluation includes multiple dimensions: two dynamic benchmarks (HealthBench, MAQuE), static knowledge validation (MedQA, MMLU).
summary: The paper proposes DOCTOR-R1, an AI agent framework designed to master both strategic multi-turn patient inquiry and medical decision-making through experiential reinforcement learning.
weaknesses: 1. The most severe limitation is the pervasive use of LLM-as-judge throughout the system. The reward model is Qwen3-8B evaluating Qwen3-8B policy outputs. This creates a closed system where the system optimizes for LLM evaluator preferences rather than genuine clinical competence.
2. The experience retrieval mechanism offers minimal technical innovation. Stage 1 (embedding similarity + reward weighting) and Stage 2 (cross-encoder reranking) are standard retrieval techniques. Are retrieved experiences genuinely instructive, or does the system learn spurious correlations?
3. There are some Human Evaluation Limitations. No evaluation of diagnostic accuracy by medical professionals. Both primary benchmarks (HealthBench, MAQuE) use automated LLM evaluation, creating potential bias toward response styles favored by specific LLM families.
4. No comparison with supervised fine-tuning on high-quality human-authored dialogues. Given the complexity of the RL pipeline, demonstrating that RL provides advantages over simpler approaches is essential.
5. The author acknowledges the "key challenge" of patient agent adherence but provides insufficient evidence that the mitigation strategies are effective. What percentage of simulated dialogues are filtered out due to adherence failures?
6. The multi-objective reward structure, while helpful, does not eliminate the possibility of exploitation. Human spot-checking is mentioned but no statistics on frequency or findings are provided.


Decision: Accept (Poster)