Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learning
Authors: Yunghwei Lai
Kaiming Liu
Ziyue Wang
Weizhi Ma
Yang Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Doctor Agent
Clinical Inquiry
Agentic Reinforcement Learning
Primary area: foundation or frontier models, including LLMs

Abstract: The professionalism of a human doctor in outpatient service depends on two core abilities: the ability to make accurate medical decisions and the medical consultation skill to conduct strategic, empathetic patient inquiry. Existing Large Language Models (LLMs) have achieved remarkable accuracy on medical decision-making benchmarks. However, they often lack the ability to conduct the strategic and empathetic consultation, which is essential for real-world clinical scenarios. To address this gap, we propose Doctor-R1, an AI doctor agent trained to master both of the capabilities by ask high-yield questions and conduct strategic multi-turn inquiry to guide decision-making. Our framework introduces three key components: a multi-agent interactive environment, a two-tiered reward architecture that separately optimizes clinical decision-making and communicative inquiry skills, and an experience repository to ground policy learning in high-quality prior trajectories. We evaluate Doctor-R1 on OpenAI's HealthBench and MAQuE, assessed across multi-facet metrics, such as communication quality, user experience, and task accuracy. Remarkably, Doctor-R1 surpasses state-of-the-art open-source specialized LLMs by a substantial margin with higher parameter efficiency and outperforms powerful proprietary models. Furthermore, the human expert evaluations show that Doctor-R1 achieves superior clinical capability and patient-centric performance, demonstrating the effectiveness of the framework.

Published as a conference paper at ICLR 2026
DOCTOR-RI: MASTERING CLINICAL INQUIRY WITH EXPERIENTIAL AGENTIC REINFORCEMENT LEARNING
Yunghwei Lai Kaiming Liu Ziyue Wang
Weizhi Ma Yang Liu 12,3+
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University
"Colleee of AI, Tsinghua University
Institute for Al Industry Research (AIR), Tsinghua University
ABSTRACT
The professionalism human doctor outpatient service depends on two core abilities: the ability to make accurate medical decisions and the medical consul- tation skill to conduct strategic, empathetic patient inquiry. Existing Large Lan- guage Models (LLMs) have achieved remarkable accuracy medical decision- making benchmarks. However, they often lack the ability to conduct the strategic and empathetic consultation, which is essential for real-world clinical scenarios. To address this gap, we propose DOCTOR-BE R1, an Al doctor agent trained to mas- ter both of the capabilities by ask high-yield questions and conduct strategic multi- turn inquiry to guide decision- makiing... Our framework introduces three key com- ponents: a multi-agent interactive environment, two-tiered reward architecture that separately optimizes clinical decision-making and communicative inquiry skills, and an experience repository to ground policy learning in high-quality prior trajectories. We evaluate DOCTOR-RI across multi-facet metrics, such as communication quality, user experience, and task accuracy. Remarkably, DOCTOR-RR R1 surpasses state-of-the-at specialized LLMs by a substantial margin with higher parameter efficiency and outperforms powerful proprietary models. Human expert evaluations show that DOCTOR-RI achieves superior clinical capability and patient-centric performance. For more details refer to https: /tt/tttttttt ceigihhubb... htw.tht..ocoooocooooooommm..httt thu- cia.n/coooooiooooo Dewpc///oottttR
HealthBench and MAQuE, assessed
Top models are laving strong hase of medical knowledge, achieving scores surpassing human expert. On paper, they look great and very similar capability.
MalQA
MMLU
SOTA models truely adagtt real- complex scenarios?
Humar Utaal:
Clraki Suba
LLM Zairag
Exper
I1. The Clinical Gountlet: High-Risk Clinical Inquiry
Patient Profile: A 35-yeur old mule presents with massive homoptysis, coughing up 590 mL of a'blood..... Patient Initial Query: " kave fand cough for the four days."
1) Clinical Inquiry by LLMs
Inquiry Deficiencies
Limited Dynamic Inquiry (Failed perform effective differenrial reasoning)
GPT-4.I: /ever changes weigh??"
fever/
Red-flag Symploms Missed
Huatuo6PPT-o1-7000 "History of axthma/ / smoking?" DucterAgent-RL:
Unsalé Clinical Advice
infection?
Emphatic Warding
Empathetic Communication
2) Strateglc, Coal-Directe Inquiry
Suecessful Consultation
Differential Diagnosis (Rapid identification red-flls red-flng symploms)
Doctor-RI: "Productive dry cough?"
Rod-flag flag Symptoms Identified
Doctor-R1: "What's
color d'bondd?
Risk Fictor Checked
Doctor-R1: R "Go to the unarest emergnacy vonu inmodiately.
Clear Urgent Instruction
Urgent and Supportive Communication
Please priorinice this. I heve support you...
Sale Eingulhetic Advice
Figure 1: The Gap Between Static Medical Decision Task and Dynamic Clinical Inquiry: Top- tier models achieve strong results on static benchmarks but show limited adaptability to the complex scenario. DOCTOR-R R demonstrates strategic inquiry process and differential diagnosis, showing the effectiveness of our framework (see Table [6 for complete case study).
"Correspondence Weizhi Ma (mawz@tsinghuu..cdu.crr Yang Liu (liuyang201 (tsinggua.edd.cn))

---

Published as a conference paper at ICLR 2026
INTRODUCTION
The outpatient proficiency of a human doctor fundamentally depends on two core abilities: accurate medical decision- decision-making and strategic, empathetic patient inquiry Epstein & Hundert (2002). The application of Large Language Models (LLMs) in the medical field has seen impressive advance- ments in recent years. Frontier and specialized models like GPT-40 [Team] 2024) and Med-Pal.M 2 (Singhal et al. 2025) have achieved scores surpassing human experts on static medical benchmark like the United States Medical Licensing Examination (USMLE), Building on this foundation, new wave of advanced medical reasoning models (Chen et al. 2024] Ankit Pal 2024 Zhang et al. 2024 Acikgoz et al.] 2024 Liu et al.) 2022 Xu et al. 20224) and multi-agent clinical simulations (Li et al. 2025
Feng et al., 2025; Fan et al. 2025) have emerged to tackle more specific clinical challenges.
However, strong performance static medical benchmarks (Jin et al. 2020 2019 Hendrycks et al. 2021) does not reflect effective clinical practice. Model performance drops significantly when they faced with open-ended medical scenarios (Kim et al.) [025 Guan et al.
2025 Wang et al.)
2023a). Real-world clinical practice is not static decision- task but instead dynamic process of information gathering under uncertainty (Bani-Harouni al. 2025 Helou et al., 2020). This pro- cess which we term Dynamic Inquiry. is sequential decision-making paradigm where an expert human doctor forms differential diagnostic hypotheses, gathers evidence through targeted question- ing to progressively narrow the possibilities, and adapts their inquiry strategy based on the patient's 's real-time responses (Wilkinson al. 2024). which existing models fail to accomplish (Figure I
To assess this capability, doctor agent should adhere to three key principles: 1) Strategic and Dy- namic Inquiry: An effective doctor agent must strategically ask high-yield questions and perform differential diagnosis to quickly identify critical risks. For example, in Figure [ models such as GPT-4.1 and HuatuoGPT-ol--7 --OB- (Chen et al.) 2024) may fail this principle by asking non-targeted questions, rather than adapting its inquiry based on the patient's 's responses, resulting in low efficiency and an unsafe consultation. 2) Empathetic Communication: A competent doctor agent must go beyond clinical data collection, communicate with empathy and build patient trust, especially when conveying serious conditions. For instance, Baichuan-M2 (Baichuan Intelligent Technology 2025) and DoctorAgent-RL. (Feng al. 2025) fail this principle when handling emergencies in Table 16 3) Learning from Good Experience: An advanced doctor agent should learn from prior experi- ences like real physician, by selectively retrieving high-quality experiences continuously re- fine its inquiry strategy. Existing models are limited in supporting dynamic inquiry (Principle 1), empathic communication (Principle 2). and high-qualit, experience learning (Principle 3), which relying primarily on similarity-basee retrieval (Yan et al. 2025 Zhong et al. 2023 Li et al. (025).
To this end, we propose DOCTOR-RI. a novel framework that adheres to these three principles. In particular, 1) To enable dynamic inquiry, DOCTOR-RI 1 leverages Reinforcement Learning (RL) framework with Group Relative Policy Optimization (GRPO) (Shao et al.] (2024] in a multi-agent interactive environment for multi-turn dialogue training. 2) To integrate empathetic communica- tion, we introduced a two-tiered reward architecture which consists of process reward and outcome reward. The reward function is defined through multi- multi-dimensionall metries that evaluate both hard and soft skills of the agent. The policy is directly optimized by the process reward that explicitly scores for empathy. 3) To learn from good experience, introduce Experiential Agentic Re- inforcement Learning (RL) approach to guide the agent's decisions. We construct an experience repository that selectively stores and retrieves high-quality prior trajectories based on their rewards. A multi-stage experience retrieval pipeline is used to ensure the agent learns from the most relevant, novel, and high-reward experiences, which we define as the characteristics of good experience.
We evaluate DOCTOR-RI on two challenging medical benchmarks, HealthBench (Arora et al., 2025) and MAQuE (Gong et al., 2025). covering a wide range of medical specialties and complex multi-turn diagnostic scenarios. The results demonstrate that DOCTOR-BE R outperforms all other open-source specialized LLMs, establishing a new ofate-of-the-art for doctor agents. DOCTOR-R outperforms Baichuan-M2 M2 by a significant margin on HealthBench communication scores +25.05 (67.16 vs. 11) by conducting more efficient questioning. Specifically, DOCTOR-RI observed a substantial improvement in with an increase of +9.91 (36.29. vs. 26.38) on HealthBench, and +8.00 (60.00 vs. 52.00) on MAQuE compared to UltraMedical (Chen et al. 2024). Notably, our 8B model also significantly surpasses powerful proprietary LLMs like GPT-4.1, Grok 4, Claude Sonnet 4, and Gemini 2.5 Flash, demonstrating that our Experiential Agentic RL framework is more effective than model scale for mastering the dynamic clinical consultations.
2

---

Published as a conference paper at ICLR 2026
The main contributions of this work follows:
We introduce DOCTOR-RE the first agent framework to unify two core clinical skills: strategic multi-turn inquiry (soft skills) and medical decision- makinn (hard skills) within a single agent.
We propose a new closed-loop Experiential Agentic Reinforcement Learning methodology which integrates multi-agent interactive environment, a two-tiered reward architecture that sepa- rately optimizes decision-maiee makinn and communicative skills, and an experience repository guiding policy learning from high-quality prior trajectories.
DOCTOR-RI surpasses top-tier open-source and proprietary LLMs on HealthBench and MAQuE with greater parameter efficiency, highlighting key finding: enhancing inquiry capability im- proves decision-making These results are further validated by human expert evaluations, which highlight the superior clinical capability and patient-centric performance of our framework.
2 PRELIMINARIES
Agentic Reinforcement Learning (RL.) (Zhang et al. 2025) refers a paradigm focused on training LLMS as autonomous goal-directed agents rather than static conditional generators optimized for single-turn alignment benchmark performance. Unlike traditional supervised fine-tuning which focuses on imitation, Agentic RL situates the policy model (Tо) in partially observable, interactive environment where it learns sequences of actions toward long-term objectives. Guided by a reward model (Rw) and updated via optimization algorithms, Agentic RL is exceptionally well-suited for complex sequential tasks like medical inquiry, as trains the agent to develop strategic policy for actively gathering information and making decisions under uncertainty.
Policy Model In the context of Agentic RL, the policy model is the agent being trained to generate an action a (an utterance) given an observation or (the dialogue history). The policy, denoted as Ty with parameters 6, is typically a large language model whose objective is to learn a strategy that maximizes rewards from the environment. In this work, the doctor agent the policy model.
Multi-Objective Reward Model A reward model Ry provides proxy signal for evaluating pol- icy actions. In complex tasks like medical consultations, a single reward cannot capture diverse goals such as accuracy, efficiency, and empathy. We adopt multi collii-objective reward model, a vector- valued function (ot, 0z) R where o, is the observation, d, the action, s the parameters, and K the number of reward dimensions which evaluate different aspects of quality. In our framework, the Consultation Evaluator functions the multi-objective reward model.
GRPO Model GRPO (Group Relative Policy Optimization) algorithm (Shao et al. 2024) opti- mizes the policy model Fy by leveraging the relative quality of multiple candidate responses within single advantage estimate like PPO or single chosen/rejected pair like DPO, GRPO uses listwise objective based on reward scores across a response set. Specifi- cally, GRPO optimizes the policy to prefer chosen response Ye over group of rejected responses 1 1==1- by maximizing of the log- likelihoood the chosen response relative to the group:
group. Instead of relying
e Ro(z,))
LGRPO ((ee))ervee)))> log E (v.(v.u(vv)) (u(
where (z, y) is the reward score. This loss function encourages higher reward for ze relative to all rejections. In our framework, the policy model тi is the doctor agent, and the multi-objective reward model RS the Consultation Evaluator. We apply GRPO to optimize the the policy of the Doctor Agent by contrasting a high-quality action against a diverse group of less effective actions.
(z.
3 DOCTOR-R1
In this section, we describe the construction of DOCTOR-R1, framework designed for continuous experiential reinforcement learning for doctor agents. We begin by defining the dynamic interac- tive environment in Section 3.1 which specifies how the training environment is constructed with Markov decision process in agents. Next, in Section 3.2 we present our two-tiered reward architec- ture which evaluates both conversational process (soft skills) and final diagnostic correctness (hard skills). Finally, we detail the experience repository that enables the agent to store and retrieve high- quality prior trajectories ground its policy in Section 3.3
3

---

Published as a conference paper at ICLR 2026
Journey stant here
5) Rewurd Model
3) Actur Model
Emvironment
Obeere State (dhalogue)
Hurnee Experience
Sclect Action
Repped pallzst
Canaultation Evtilaatoe
query
Pecrtional Areword
Complex Cliniol Soonortoe
A) Procesi Revard for Queviions
Genorats Action
-Eicurcke Viu Spuken *Stil Decrggiing (Sanayy Dmppaity okc)
Experience Module
Simstane Patlent Ageat P.
B) Ostoame and Disgzenis
Rathove "Gusd"" Uxperizcez
Expeneree Repostery
Enter Multi-tura laquiry Session S/
Polics Updattc
Stage 1:
Coodidare Selection
sap- salection
top-4 ieranking
2) Turn of Inquiey Seasion S
Siige High-Fiiehh Revssking
update dynemic
Stge X Novelty and Rewood Finering
thresbald
Pasiett Query
Revard for
Stoee paod experience
Retrievd: Bost tope Peperience
expertencon
Expericnuz Repasitory Troosr
Experience (6,4,
Action question diggnesis
1. Eilly ffl ( experincce a borning sersation The putient las histry reflas caplasiia and Burnati Actin: [Cheston] Have you boon tosted for H. polodi infotion? o. 0.72
The pasert a ti-yca--old werean with excountaiiis and she menticon tea clindrg vains is sucnaaes. Fist, ihould confonn if thi is the orls sumeptonn if thessee otters. Mayle siz has joct pain or suffess Bui dkda merico Booe yet, Also, Esould ask abeut the spscifie joirs afficted So, the nost styp is osk about the speclfic josts offected and any ofher symptons.
Think:
7. Fill dy fyltt Eigt Mig fia
Action: (Question] you experiencing pais, weeling or stffiees spocific joirs, your Incee or tips?
Sevarkss audp are skate (oatient cwerc) action (bactor) mupoonn and reword room dame ruppertelyy
Figure 2: The interactive training loop of our DOCTOR-E RI framework. The process unfolds within (1) Dynamic Interactive Environment populated by diverse patient simulations. At (2) each turn of an inquiry session, the (3) Doctor Agent interacts with the environment by observing the state, queries the (4) Experience Repository, and selects action. A Patient Agent responds, and the (5) Consultation Evaluator evaluates the action based on the two-tiered reward architecture. This new experience is stored into the repository, and is used to optimize the policy of Doctor Agent.
3.1 DYNAMIC INTERACTIVE ENVIRONMENT
Following the Agentic RL paradigm, formalize the medical consultation process Partially Observable Markov Decision Process (POMDP). This framework is well-suited to model the se- quential, uncertain nature of clinical inquiry, where doctor agent cannot directly observe the true state (i.e, the full medical condition of patient) but must act based on a history of observations. To construct this environment, define three core interacting components that form closed-loop feedback system. The environment's dynamics are governed by the POMDP tuple AS,A, O, R). with the following components: 1) States (S): A state
S represents the consultation query of the patient agent. 2) Actions (LA): An action a, E A is an utterance generated by the doctor agent turn 3) Observations (O): An observation o, E O is the dialogue history up turn At each step, the doctor agent observes the patient's 's latest response and adds it to the history to form the new observation 01+1- 4) Reward Function (R): Reward R(s, a) provides signal to the doctor agent after each action. It is implemented by the Consultation Evaluator which evaluates action o, as detailed in Section 3.2 Each element is realized through the interaction of these components:
Doctor Agent (T²): This is our target agent. Its policy, ne Te(ae)or) maps the current observation Or (dialogue history) distribution over actions a₂ E A. The goal is learn an optimal inquiry policy गे that maximizes the cumulative reward.
Patient Agent: This agent simulates patient behavior, effectively defining the environment's 's state transitions. Portrayed by an independent LLM, it is initialized with clinical scenario that defines its underlying true state E S. It generates a response based the action a,, which updates the next observation 0 for the Doctor Agent.
Consultation Evaluator: This component acts as the environment's 's reward function R. At each turn f, it provides a multi multi-dimmensooal reward by assessing the action in the dialogue context.
This closed-loop feedback system provides the foundation for training the policy of the doctor agent. Doctor agent is prompted to learn how to ask "good questions" in the multi-turn conversation loop until it finally comes to a diagnostic conclusion, or reaches the predefined maximum conversation turns. The LLM-based patient agent acts as viding diverse responses and dynamically constructing a high diversity environment enriched with various clinical tasks, enable the doctor agent to interact, learn, and evolve through each rollout.
component in the interactive environment, pro-
4

---

Published as a conference paper at ICLR 2026
3.2 TWO-TIERED REWARD ARCHITECTURE
To enable the robust learning of the agent over long- ainghhoizzos,, simple monolithic reward is insuf- ficient (Gao et al., 2022). We introduce a two-tiered reward architecture that evaluates the two core clinical skills of physician: both the conversational process (soft skills) and the final diagnostic outcome (hard skills). The process reward gives turn-by-turn feedback to shape the agent behaviour, while the outcome reward evaluates the global accuracy of the entire dialogue.
Process Reward for Conversational Quality To shape the inquiry capabilities of the agent, we provide a dense process reward (Rmm) after each turn. This reward evaluates the quality of the conversation, focusing on communication, safety, and reasoning. The agent responses are evaluated across eight dimensions, Sy: 1) Safety: Penalizes harmful advice or unsafe recommendation. 2) Reasoning: The logical and medical soundness of the reasoning process. 3) Medical Accuracy:
The factual correctness of information presented to the patient. 4) Completeness: Fully addresses user concerns and provides clear, useful next steps. 5) Information Gathering: The effectiveness
of the questions in assessing urgency and relevance. 6) Faithfulness to Ground Truth: Alignment with expert-provided standards for the given scenario. 7) Empathy and Clarity: The use of simple and reassuring language. 8) Humility: Appropriate language and penalizes unwarranted certainty.
Our design is built on a "safety-firs" ffrfsttyffist""eee principle. using a hicrarchical penalty to handle critical failures. This addresses the limitations of conventional weighted-sum models, which can fail to adequately penalize catastrophic errors due an "averaging-out" effect (Amodei al. 2016). As shown in Equation 2] any violation of safety, reasoning, or accuracy triggers an immediate large negative re- ward, overriding the standard score calculation. This veto system conceptually aligned with shielded frameworks in Safe RL, where safety layer intervenes to prevent unsafe actions (Alshiekh et al.) 2018), ensuring that fundamental clinical reliability is non-negotiable. no violation is triggered, the reward is weighted sum of all eight dimensional scores.
1.0
if Saatty <
Rum
0.75
else if Steasoning < or Saccuracy <
E wi S: otherwise
Here, is the failure threshold, St is the score for each of the eight dimensions, and W; are the corresponding weights that reflect their relative importance. The specific weights and normalization details are further elaborated in Appendix B.I
Outcome Reward for Diagnostic Accuracy At the end of dialogue episode, the agent receives a single terminal outcome reward (Rana) based on the correctness of the final diagnosis. This reward evaluates the core "hard skill" of the agent, which the ability of decision-makinn consultation. The decision of the agent is compared against ground-truth diagnosis and assigns score (S.orrectness) based on the degree of alignment. This ensures the agent is optimized not just to interact well, but to drive the conversation towards medically sound conclusion. A score of 1.0 is given for a correct diagnosis, 0.5 for a partially correct one (e.g., correct differential but wrong primary diagnosis), and 0.0 for incorrect diagnosis. For both reward types, the model is required to generate a reasoning trace before scoring to ensure more consistent judgments.
throughout the
Rinal Scorretnesss
where Scomectness E {0.0, 0.5, 1.03
3.3 EXPERIENCE REPOSITORY
Multi-stage Experience Retrieval
The doctor agent optimizes its policy from past trajectories (state, action, reward) using a multi-stage retrieval pipeline, where we combine an em- bedding model for candidate selection and reranker model for fine-gre fine-grained reordering, balancing the trade-off between oow-precision embedding and computationally expensive reranking.
The objective of this stage is to efficiently reduce the search space of experiences. The query state Q and all stored experience states Exiate are precomputed into dense embeddings. The combined score Scumbined (Equation
Stage I. Candidate Selection:
integrates cosine similarity with the historical reward R ((E,) weighted by a. The top-N entries are selected to form Exundidates C E, narrowing the database from millions of experiences to manageable number for the next stage. Here, Jamb is the embedding function and fam is the cosine similarity function.
S combiieed Q F.)= ssim (am (Semb (Q,, Semb (Fqltae) R (FA)
(4)
5

---

Published as a conference paper at ICLR 2026
Stage II. High-Fidelity Reranking: The candidate set Ecandidates is then passed to more powerful reranker model. While the bi-encoder architecture of the embedding model processes the query independently, the reranker model uses cross-encoder architecture to score each pair (Q, E with token-level attention for more accurate relevance, and reorder the candidates accordingly.
Stage III. Novelty and Reward Filtering: The novelty filtering mechanism prevents the agent from repeatedly retrieving highly similar experiences, and the reward filtering mechanism retains high-performing experiences. For the reranked candidate set, a dynamic high-reward threshold Tdynamic is calculated in Equation [ where PRR and RR are the mean and standard deviation of rewards within the candidate set, and Bad is a configurable factor. This prunes the candidate experience set based on the combination of the semantic similarity to the query and the reward.
Tdynamic = Pg (canndddttes) + Baad VR((Eanddidass
(5)
E E' + (fsim(famb ( (Q), femb ( (b < Tnovelty) A (R((E)) Tdynamic)
(6)
An experience Ej E Eeandidates is considered as a "good experience", and added to the full ex- perience set E. if two conditions are met: (1) its semantic similarity to the query Q falls below a predefined novelty threshold Taovelty, and (2) its associated reward R R(E;) exceeds the dynamic high-reward threshold (Equation (). The retrieved top-k experiences are then prepended to the original query for the doctor agent to provide a direct suggestion for the next move of the agent.
Selective Experience Storage To enable the agent learning from valuable experiences, high- reward interactions are selectively stored as experience tuples Fg = (8t, Gt. Ra), where 8, is the state, az the action, and R. the reward. Following the processing of each batch B, an experience E; is stored if and only if its reward R; meets or exceeds a predefined high-reward boundary, Treward- The filtered set of high-reward experiences, denoted as Enew = {Eg E B | R4 2 Treward). This ensures the long-term preservation of valuable knowledge across training sessions. The in-memory cache and embedding tensor are updated in real time for immediate retrieval and long- retention.
4 EVALUATION RESULTS
Our primary evaluations are conducted on HealthBench (Arora et al. 2025 and MAQuE (Gong eet al. 2025), ). two benchmarks featuring complex clincal scenarios and multi-turn dialogues. To validate that our training does not compromise its foundational knowledge, we also evaluate on the static QA benchmark MedQA (Jin al. 2020 and MMLU (Hendrycks et al. 2021).
4.1 OVERALL PERFORMANCE
Table shows the overall performance on HealthBench Main, which is evaluated across two dimen- categorize the medical scenario (Emergency Referrals, Health Data Tasks, Com- munication, Global Health, Hedging, Context Seeking, and Complex Responses), and 2) Axes: mea- sure fundamental skills (Accuracy, Communication Quality. Instruction Following, Context Aware- ness, and Completeness). The detailed result of HealthBench Hard is listed in Appendix C.3
sions: 1) Themes:
1) Comparison with Proprietary Models: As shown in Table D our DOCTOR-RE R1 demonstrates notable superiority over various proprietary LLMS the HeahthBench. DOCTOR-R1 achieves an average score of 36.29, surpassing leading models including GPT--.11 (31.18). Grok-4 (33.03), and Claude Sonnet 4 (25.69). This superiority is driven by mastery in both consultation and decision- making skills. DOCTOR-RI establishes a clear lead over GPT-4.1 in consultation axes like Com- munication Quality (64.15 vs. 60.65) and Context Awareness (49.24 vs. 44.81). This enhanced inquiry capability leads to better decision-making. with our model outperforming GPT--.11 in Accuracy (37.84 vs. 34.78). On the MAQuE benchmark (Table D. DOCTOR-RI matches GPT-4.1 in Accuracy while achieving a far superior score in patient-centre metrics like Empathy (93.80 vs. 75. 20), showcasing its ability in both consultation and decision-making skills.
The results in Table [[ that DOCTOR-R R1 achieves a significant performance advantage over even the strongest and larger open-source models. With a model size of only 8B parameters, DOCTOR-R R1 surpasses the best open source model, Baichuan- 12-32B, which has 4 times larger parameters. Our model achieve a higher average score on Health- Bench (36.29 vs. 33.16) with 9.4% improvement, and outperforms the 32B model across key axes including Accuracy (37.84 vs. 33.95). Communication Quality (64.15 vs. 58.01), and Context Awareness (49.24 24 vs. 46.80). On MAQuE (Table model also achieves higher Accuracy (60.00
2) Comparison with Open-Source Models:

---

Published as a conference paper at ICLR 2026
Table 1: Overall performance on HcalthBench Main (best results of open-source models are bolded)
Model
Arg. Score
Fmerr. Reforrals
Health Data T:
Commu- nication
Theme
Globol Health
Hedging
Contert Steking
Complex Resp.
Auc.
Comm. Qualiity
Trett: Foll.
Asis
Contest Aware.
Comp- letemes
Proprictary Medels
Gemini 2.5 Flash
19.35
44.65
9.12
14.56
14.05
20:45
13.09
33.69
22.45
16.98
35.84
35.81
21.56
Claude
25.09
28.60
42
28.78
44.59
41.81
31.65
GPFT-41
37.08
45.93
34.78
54.32
44.81
34.84
Cmik-4
24.25
18:20 20
17.05
48.55
45.60
CIPT-5
57.03
16
Open-Smrre Modells (71--88)
Hincel 28
2.81
0.48
0.47
1.64
0.85
3.08
0.58
12.50
7.08
9.47
20.66
15.63
5.19
3.93
10:04
2,07
6.12
3.32
6,08
3,002
13.75
11.54
23.37
17.88
22.40
9.31
6.25
12.50
3.02
5.60
3.14
7:10
3.28
11 1.94
11 1.50
19.65
23.94
23.13
5.63
8.17
18.50
6,40
6.34
4.81
7.83
2.50
16.46
12.96
25.72
24.30
23.77
10.67
33001 1/2
14,97
28,40
18.59
14.08
7:45
14.28
8.85
32.0
20.67
45.68
42.88
31.09
17.79
Dov Apzent R1
15.77
33.48
14,97
17.58
7.69
15:29
9.22
30.01
20.
50.05
38.54
31.73
18.55
16.25
34.15
18.11
17.73
8.14
15:12
10.91
26.54
21.31
54.24
41.57
33.88
17:05
22.19
46.54
19.52
26.12
13.78
19.31
18.25
26.72
53
57.40
44.68
40.26
27.40
Open-Source Models (>
32191
Batchann M2
Openiot 7003
22.51
9.61
9.76
20.37
23.97
19:75
49.95
37.97
23.16
35.64
20.86
33.73
30.81
51.45
26.95
Ullm Mannccia
38.59
19:42
18:13
45.23
32.60
50.62
41.23
45.49
DOCTOR-RI
36.29
47.36
33.71
win Proc. Reward
51:21
32.78
38.80
win Experience
47.34
38
21.58
31.79
34.15
30.55
59.19
51.75
36.23
Base (Qwen33-0 80)
23.13
45.42
16.50
27.98
15.26
25.34
16.42
30.89
28.57
49.35
43.51
43.00
27.24
Table 2: Overall model performance on MAQuE (best results of open- open-source models are bolded)
Task Success
Inq. Proficiency
Dialogue Competence
Patient Experience
Model
Acc.
Robustness
Coverage
Relevance
Adherence
Coherence
Clarity
Empathy
Gemini-2.5-Flan
57.00
70.74
33,57
65.84
83.80
65.80
68.60
45.80
Grok-4
58.00
74.22
31.26
89.55
90.40
81.80
80.20
90.60
GPT-4.1
60.00
70.59
46.02
89.53
98.00
86.60
77.80
75.20
GPT-5
67.00
79.30
29.50
91.30
96.80
81.40
71.60
47,80
HuatuoGPT-ol-aa -8B
40.00
62.62
21.35
89.38
36.20
63.60
57.20
60.80
McU12-y2-8
45.00
66.69
30.01
61.48
47.40
65.00
57.60
64.20
52,00
66.75
25,14
81.00
37.00
61.00
58,20
80.80
DictorAAgeal-
50.00
67.15
35.39
74.90
68.40
66.00
74.80
66.60
Bunchuan M2B
57.00
66.41
44.59
90.78
97.00
79.00
72,40
65.20
UltraMedica-770
52,00
71.59
39.11
84,36
78.80
85.80
75.00
83.40
DOCTOR-RI
60.00
77.03
38.52
87,50
70.20
76.20
69.80
93.80
wlo Experience
56.00
70.60
34.46
84.66
77.80
75.40
65.40
82.20
w/o Proc. Reward
52.00
$3.80
65.00
Base (Qwen3-B) 8B3)
49.00
65.20
33,50
74.09
55,20
69.40
66.00
43.20
vs. 57.00) and Empathy (93.80 vs. 65.20). The gap is even clearer when comparing DOCTOR-P R1 to other 70B leading models by nearly 10% on average accuracy, such as UltraMddica--708 on HealthBench (36.29 vs. 26.38) and MAQuE (60.00 vs. 52.00), demonstrating the effectiveness of DOCTOR-R R1 in achieving strong performance with higher parameter efficiency.
Validating Foundational Knowledge To validate that our train- ing does not cause knowledge degradation, we evaluate DOCTOR- R1 on static QA benchmarks, MedQA and MMLU. Table 3 shows that DOCTOR-B R1 not only prevents knowledge degradation but sig- nificantly enhances the ability on decision- making tasks. Compared to its 8B base model, DOCTOR-RI achieves a remarkable increase on MedQA (83.00 vs. 63.50) and MMLU (85.00 vs. 70.00). The perfomance exceeds the much larger models like Baichuan-M2- 32B, and is comparable to top proprietary models like GPT-4.1.
Table 3: Static QA Results
Model
MedQA
MMLU
Gemini 2.5 Flash
61.50
20.50
Claude Sonnet 4 GFT-4.1
89.50
89.50
89.00
Qwen3-8B (Base)
63.50
70.00
Med42-v2-8B
77.50
60.50
DoctorAgent- RL.
58.00
72.50
UltraMedical 8B
75.00
67.50
Baichuan -M2-2B
81.50
84.00
DOCTOR-RI
83,50
85.00
4.2 HUMAN EVALUATION
User Experience as Patients To validate that the performance of DOCTOR-RI on automated benchmarks aligns with real Patient Experience (Principle 2), we conducted a pairwise human evaluation. Annotators were shown paired anonymized conversations from DOCTOR-RI and four other models to choose the better response based on four qualitative metrics: Coherence, Adherence, Clarity, and Empathy. The results show a strong human preference for DOCTOR-RI R1, which ranked first across all categories. Our model achieves dominant win rates in Figure 3 showing consistent results on HealthBench and MAQuE. The complete statistical breakdown is available in Appendix
7

---

Published as a conference paper at ICLR 2026
(a)
Win
Tic
Loss
Coherence
Adherence
Doctor-Ri
77.4%
Doctor-RL
29%
GPT-
40%
40%
19%
Win 67.7%
GPT-3
62%
15%
23%
Win 72.7%
Baichuan M2-32B
42%
31%
27%
Win 61.1%
Baachuan M2-32B
38%
1995
42%
Win 47.6%
UlrrMedicaa-8B
UltraMedical--88
0.0
0.2
0.8
1.0
0.0
0.2
Clarity
Empathy
Doctor R1
48%
Doctor R1
92.5%
40%
Win 74.2%
U'IrraMedica--88
60%
17%
23%
72.1
GPT-5
33%
44%
2.3%
Win 58.6%
GPT-3
23%
2.3%
5456
Win 30. 00.0%
UltraMedical-O00
Baichuan M2-32B
(b)
1.2
Likert Score
Safety
Medical Accuracy
Information Gathering
Completeness
Doctor-RI R1 (BB)
Brtchuan-M2-325
GPT-5
UltraMedical-83
Figure 3: Overall human evaluation results. (a) Patient Experience: Pairwise comparison across four qualitative metrics. Fach bar shows the distribution of wins (blue), ties (yellow), and losses (red) for model compared against all others. (b) Clinical Competence: Average scores assigned by licensed physicians 1-5 Likert scale. DOCTOR-RI RI (with 8B parameters) demonstrates superior performance patient-centric metrics and competitive clinical capability compared to larger LLMs.
Expert Validation To validate the alignment between benchmark performance and real-world clinical capability (Principle 1), we conduct comprehensive evaluation with 2 licensed physicians. Regarding Clinical Competence (Principle 1), the results demonstrate that DOCTOR-R achieves high expert rating comparable the specialized model Baichuan-M2-328 and proprietary model like GPT-5, especially in clinical safety and information gathering visualized in Figure 3 Futher- more, to validate Learning from Experience (Principle 3), experts assess the utility of the retrieved experiences. Physicians rate the retrieved content "Clinically Helpful" in 83.87 % of cases for guiding the agent's next action, with only 16.13% rated "Neutral and Irrelevant" and 0% as "Harmful". This confirms that our retrieval mechanism provides genuine strategic value rather than noise. We calculate inter-rater reliability between expert scores and our reward model vali- date the reliability of the Consultation Evaluator. Detailed analysis is provided in Appendix
5 ANALYSIS
This section shows our main analysis. Additional experiments on framework transferability (Llama family), reward architecture ablation, and case studies are provided in Appendix and
Ablation on SFT and PPO Baselines To validate the effectiveness of our Agentic RL. framework and justify the choice of optimization algorithms, we compare our method against Supervised Fine- Tuning (SFT) and Proximal Policy Optimization (PPO). As shown Table 4 the SFT baseline represents marked improvement over the base model (29.54 .54 vs. 25.13), while the PPO baseline achieves an average score outperforming SFT (33.23 vs. 29.54) through inquiry policy optimization. Our proposed GRPO method establishes the strongest performance over all metrics, demonstrating particular superiority over PPO Communication (47.16 vs. 39.40). Theoretical justification for the superiority of GRPO is justified in Appendix D.1
8

---

Published as a conference paper at ICLR 2026
Table Ablation study of performance across SFT, PPO baselines and our final GRPO framework.
DOCTOR-RI Vartants
Avg. Score
Theme
Axis
Emerg. Referrals
Health Data T.
Commuu- nication
Glohal Health
Hedging
Context Seeking
Complex Resp.
Acc.
Comm. Quality
Instr. Foll.
Confext Aware.
leteness
Comp-
16.42
27.24
SFT 29.54
22.22
32.53
20.17
29.14
22.33
32.37
57.32
48.75
33.46
+ PPO
33.23
51.77
24.96
39.40
24.29
33.50
23.82
31.51
36.96
59.29
19.48
46.86
38.72
GRIO
36.29
54.44
29.17
47.16
24.74
33,71
26.39
34.25
37.84
64.15
54.39
49.24
40.93
Eep
DocroR-Rt1
Impact of Experience Retrieval Mechanism To ver- ify the contributions our experience retrieval mech- anism, compare three variants: No Experience: The agent trained without any retrieval. 2) Similarity Only: baseline using standard semantic similarity for retrieval. 3) DOCTOR-RI: Our full mechanism with re- ward and novelty filtering. As shown in Figure 4 our full retrieval mechanism significantly enhances the agent performance. Similarity -based retrieval provides only a slight improvement over the no-experience baseline, im- proving Communication by 0.91% (39.40 vs. 38.49) and Context Awareness by 1.55% (46.86 vs. 45.31). How- ever, our full DOCTOR-RI retrieval mechanism shows more substantial improvement, achieving additional 7.76% gain in Communication (47.16 vs. 39.40) and 2.38% in Context Awareness (49.24 46.86).
Communiation
Contest
Chmppic
Awarmess
Responses
In
Hedging
Completeness
Hish Datu
Figure 4: An ablation study comparing the experience retrieval mechanism of DOCTOR-R against baseline methods.
GPT-411
Raichuan-M2
Doctor-RI
MAQQE Accuracy
30
Dislogue Turn
Centust Sooking HoaTa Data Taaks
Hedging
ACCRERAY
Oommuncttio
(a)
(b)
Figure 5: Scaling analysis of key framework components. (a) The impact of dialogue turns on task accuracy. (b) The impact of the number of simulated patient agents used in training.
Scaling Effeet of Interaction Turns Figure 5a illustrates the impact of dialogue turn on diagnostic accuracy, comparing DOCTOR-RI against strong baselines. DOCTOR-RI consistently demonstrates superior performance at every stage of the conversation. In the very first turn, our model already establishes a lead in accuracy over both GPT-4. (36.0 vs. 33.0) and Baichuan-Mi (36.0 vs. 26.0). As the conversation extends to 5 turns, DOCTOR-R RI achieves an accuracy of 58.0, surpassing GPT- 4.1 (54.0) and Baichuan-M2 (53.0). This represents 61. % relative improvement from its own tum-1 performance (58.0 vs. 36.0), demonstrating that the policy of our agent is exceptionally well-optimized for strategic multi-turn inquiry.
Scaling Effect of Patient Agents We investigated how the volume of simulated patient agent impacts the final performance of our agent. As shown in Figure 5b] there is strong and consistent positive correlation between the number of patient agent used for training and the capabilities of our doctor agent. Scaling the patient agent from zero (the base model) to 100k simulated patient agents yield substantial improvement across all key metrics. The results show remarkable relative gains, with 68.5% improvement in Communication and a 32.4% improvement in Accuracy model. This scaling effect strongly validates that our agentic framework effectively leverages large- scale patient agent, confirming that the quantity of patient agents key drivers of performance.
the base

---

Published as a conference paper at ICLR 2026
6
RELATED WORK
Model
RL
Agentic RL (Multi-tura)
Reasoning
Process Reward
Outcome Reward
Static Medical Decision
Dynamic Clinical Inquiry
Memory (xperience)
HumuoGPT--o1-00m
X
X
M-ool
x
Baichuan-M2
X
UltraMddical
X
X
X
Med-PRM
X
X
/
X
MedAdapter
X
X
X
DectorAgent-RL.
X
Ductor-RI (Ours)
Table 5: Comparison of our proposed DOCTOR-RE R1 with recent medical specialized LLMs.
5 highlights a fundamental paradigm difference between our agentic approach and prior medical specialized LLMs. This distinction is crucial as are moving from optimizing static single-turn answers to optimizing a dynamic multi-turn inquiry policy that must align with the complexity of real-world clinical inferaction.
Table
Advancements in Medical LLMs Recent Large Language Models (LLMs) have demonstrated
profound clinical knowledge. particularly in static snowledge-intensive tasks. Proprictary models like Med-Pal.MM (Singhal et al. 2025) and GPT-4o (Team (024) set new benchmarks by achieving the USMLE Some specialized models focus specific data modal- ities, such as unstructured clinical text (Yang et al.] 2022). and Chinese medical corpora (Zhang) al. 2023: Chen et al. 2024 2023a). Parallel efforts have produced lightweight models opti- mized for specific downstream tasks (Labrak et al. 2024 Han et al. 2025 Wang et al. (024), while large-scale LLMs have been leveraged to approach physician-leve level diagnostic reasoming capabil- ity (Chen et al. 2023b) Christophe et al.) 2024 Ankit Pal
expert-level performance
2024 Zhang et al. (022)). Conventional RI. frameworks such HuatuoGPT-ol- 708 (Zhang et al. 2023). VineMedl.MMol (Yu et al. 2025)
and UltraMedical-708 (Zhang et al. 2024) are limited to single-turn RI optimization only for decision-making tasks. Test-time scaling frameworks such as Med-PRM (Yun et al.) 2025) and MedAdapter (Shi et al. 2024) focus primarily
secision-making ability while overlooking pa- tient inquiry and communication skills. Despite their success in knowledge recall, these models lack the dynamic sequential cetisiionmaking for real-time patient consultations, motivating the de- velopment of more interactive systems.
The Emergence of Clinical Agent Frameworks dynamic clinical practice, the research focus has shifted towards creating agentic frameworks that simulate real-world medical workflows (Wang et al. 2025 Chakraborty & Guptal 2014). These systems empower LLM agents on diverse tasks, such as natural multi-turn diagnostic dialogues with patients (Tu et al. 2024 Schmidgall al. 2025 Wang et 2023b), and health status monitoring via wearable devices (Heydari et al.) [025 Humayun et al. 2022). Multi-agent simulation frame- work enables collaboration of clinical agents for complex reasoning. diagnosis and treatment (Li) et al. 2025 Tang et al. 2024 Vicari et al. (003). However, they fall short of the three core prin- ciples (elaborated in Section
To bridge the gap between static knowledge and
required for a truly competent clinical agent. While recent agen- tic approaches such as Baichuan-M2 (Baichuan Intelligent Technology. 2025) and DoctorAgent- RL (Feng et al. 2025) enable interaction between agents but lack capability of differential diagnosis just like real human doctor. In contrast, DOCTOR-RI introduces Experiential Agentic Reinforce- ment Learning, a framework specifically designed to simulate and align with complex dynamic environments, which enable the agent to learn continuously from prior experience just like human physician instead of relying solely on its own static knowledge.
7 CONCLUSION
This paper introduces DOCTOR-RI 1, doctor agent trained on our proposed Experiential-RI frame- work to address the gap between static medical knowledge and dynamic clinical inquiry. Our ex- periments on HealthBench and MAQuE, demonstrating that DOCTOR-RI significantly outperforms frontier general and specialized models in multi-turn clinical inquiry. This success highlights the pivotal role of our hybrid approach, where an experience repository and on-policy reinforcement learning synergize to construct strategic inquiry policy. Ultimately, work underscores the lim- itations of evaluating agents on static QA benchmarks and establishes the necessity of training on sequential decision-makinn process aligned real-world clinical practice.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work is foundational research exploration into AI for clinical dialogue and is not intended for direct use by individual users for any medical purpose. DOCTOR-RE R1 is research prototype and should not be used to provide medical advice, diagnosis, or treatment. All experiments were con- ducted on established, anonymized public benchmarks (HealthBench, MAQuE, MedQA, MMLU), accordance with their respective licenses. No private or patient-identifablee data used this study.
Human Evaluation Protocol We conducted human evaluation involving two distinct cohorts:
patient-perspective evaluators and medical experts. For the patient user experience evaluation, recruited five compensated annotators without specialized medical training to ensure that subjective metrics were judged from the perspective of typical patient. For clinical validation, two compensated licensed physicians to assess model clinical skills. In both settings, annotators were presented with anonymized conversations generated by different models for the same clinical scenario and were instructed to select the superior response based on defined qualitative dimensions.
recruited
Annotator Welfare and Consent Annotators were fully informed about the study's 's purpose, task design, and their right to withdraw at any time. All participation was voluntary, and compensa- tion was provided at fair rate consistent with local labor standards. No demographic personal information about annotators was collected.
Risks and Limitations We acknowledge the potential risks associated with medical Al research, including the generation of factually incorrect misleading information and the perpetuation of societal biases. The use of an LLM as a reward model judge is also limitation, as its evaluations serve only proxies for, and not replacements of, human expertise. We release our framework and findings for further responsible research in this domain while emphasizing that the system is unsuitable for direct clinical use.
REPRODUCIRILITY STATEMENT
To ensure reproducibility, we provide our full source code and model weights Github | and Hug- gingface 2 Our framework is built upon the publicly available Qwen3-8B model. All evaluations were performed on public benchmarks, including HealthBench, MAQuE, MedQA, and MMLU. A detailed breakdown of the implementation and training settings are provided in Appendix B
ACKNOWLEDGMENTS
This work was partly supported by the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry Education of China (No. JYB2025XDXM101, sponsored by the Tsinghua- Toyota Joint Research Institute Inter-disciplinary Program and Wuxi Research Institute of Applied Technologies Tsinghua University. Weizhi Ma was also supported by the Beijing Nova Program.
REFERENCES
Emre Can Acikgoz, Osman Batur ince, Rayene Bench, Arda Aml Boz, ilker Kesen, Aykut Erdem, and Erkut Erdem. Hippocrates: An open-source framework for advancing large language models in healthcare, 2024. URL
AI@ Meta.
The llama 3 herd of models, 2024.
URL
21783
Mohammed
Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10. aaai. htt.a///w.......aaaaaaaaaa//////
URL https:
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, Schulman, and Dan Mané. Con- crete problems in ai safety, 2016. URL http:///.../////aaaar/oor/////ba 06565

---

Published as a conference paper at ICLR 2026
Malaikannan Sankarasubbu Ankit Pal. Openbiollms: Advancing open-source large lan-
guage models for healthcare and life sciences. httpss://uuciingface.. wwtp:/cce..///aaaaaaaeaaeeaaeeae co/aaityya
MagaBBioLMM-LLamae 70B 2024.
Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quinonero-Candela,, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. Healthbench: Evaluating large language models towards improved human health, 2025. URL httpht:s::/eet httt:s://arx....o////aa.. wttpo/////a///......22
Baichuan Intelligent Technology. Baichuan-m2-322: Medical-enhanced large language model.
https: htt//itthule http://aaaa.aaaaaaaaaaaaaaaaaaaa 1tw--iinnaaaccuaaa----222---- September 2025. Open-
source medical-domain model (Apache 2.0).
David Bani-Harouni, Chantal Pellegrini, Ege Özsoy, Matthias Keicher, and Nassir Navab. Language agents for hypothesis-driven clinical decision making with reinforcement learning, 2025. URL
https://arxiy:: http://aaxxxx...//.b.aabaa/
org/ abs 2506. 13474
Sougata Chakraborty and Shibakali Gupta. Medical application using multi agent system-a literature survey, International Journal of Engineering Research and Applications, 4(2):528-544 2014.
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-ol, towards medical complex reasoning with llms, 2024. URL
https: org/ wtt.//as////ooooooooooooo/...aea 2412. 18925
Yirong Chen, Zhenyu Wang. Xiaofen Xing, huimin zheng. Zhipei Xu, Kai Fang, Junhong Wang, Sihang Li, Jieling Wu, Qi Liu, and Xiangmin Xu. Bianque: Balancing the questioning and suggestion ability of health lims with multi-turn health conversations polished by chatgpt, 2023a.
Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi. Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexan- dre Sallinen, Alireza Sakhacirad, Vinitra Swamy, Igor Krawezuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. Meditron-70b: Scal- ing medical pretraining for large language models, 2023b.
wwtpaawwwwwaaaaaaaaoooaaaooooert abs/
2311. 60779
Clément Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel. Med42-v2: A suite of clinical Ilms, 2024.
abs /208.. /200g//00aab/220011..2... 06142.
Ronald M. Epstein and Edward M. Hundert. Defining and assessing professional competence.
JAMA, 287 235, 2002. ISSN 0098-7484. doi: 10.1001//ama..27..2226. URL
https://doo /oi. oro.orgggoo..ooo 1001/ ama 287.2.272
Zhihao Fan, Lai Wei, Jialong Tang, Wei Chen, Wang Siyuan, Zhongyu Wei, and Fei Huang. AT hospital: Benchmarking large language models in a multi-agent medical interaction simulator. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.). Proceedings of the 31 International Conference on Computational Linguistics, pp. 11883-1021,, Abu Dhabi, UAE January 2025. Association for Computational Linguistics. URL https: http://aaaaattta hog. ogy. org 2025 coling-main .te.iiinn..71//1
Yichun Feng, Jiawei Wang. Lu Zhou, Zhen Lei, and Yixue Li. Doctoragent-rl: A multi-agent Dollaborative Iearnig system for multi-turn clinical dialogue, 2025. URL https:
//axxii..crg/abs//5555.. .tt.//5555..133. 19630
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022.
URLhttps://ccxiv.org http::///waaii..rrr//////1..... 10760
Linlu Gong, Ante Wang, Yunghwei Lai, Weizhi Ma, and Yang Liu. The dialogue that heals: A
comprehensive evaluation of doctor agents' inquiry capability, 2025. URL. https::/arxiv
arg/abs/2503 24958
Hao Guan, David Bates, and Li Zhou. Keeping medical ai healthy: A review of detection and correction methods for system degradation, 2025. URL / http://wwww...///////5.....
T7442
12

---

Published as a conference paper at ICLR 2026
Tianyu Han, Lisa C. Adams, Vens-Michalie Papaioannou, Paul Grundmann, Tom Oberhauser, Alexei Figueroa, Alexander Löser, Daniel Truhn, and Keno K. Bressem. Medalpaca an open-source collection of medical conversational ai models and training data, 2025. URL https:
arg//bs/220 2304 4. 08247
Maria A. Helou, Deborah DiazGranados, Mary S. Ryan, and Jonathan W. Cyrus. Uncertainty in decision making in medicine: A scoping review and thematic analysis of conceptual models. Academic Medicine: Journal of the Association of American Medical Colleges, 95(1):157-165,
2020. doi:
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding, 2021. URL https:
//aaxxv..rg//aaaaaaaa 2009 2009//300000000 03300
A. Ali Heydari, Ken Gu, Vidya Srinivas, Hong Yu, Zhihan Zhang, Yuwei Zhang, Akshay Paruchuri, Qian He, Hamid Palangi, Nova Hammerquist, Ahmed A. Metwally, Brent Winslow, Yubin Kim, Kumar Ayush, Yuzhe Yang, Girish Narayanswamy, Maxwell A. Xu, Jake Garrison, Amy Arem- nto Lee, Jenny Vafeiadou, Ben Graef, Isaac R. Galatzer-Levy, Erik Schenck, Andrew Barakat, Javier Perez, Jacqueline Shreibati, John Hernandez, Anthony Z. Faranesh, Javier L. Prieto, Con- nor Heneghan, Yun Liu, Jiening Zhan, Mark Malhotra, Shwetak Patel, Tim Althoff, Xin Liu, Daniel McDuff, and Xuhai "Orson" Xu. The anatomy of personal health agent, 2025. URL https::/arre www.aiiiv.oggaosoo.oooe/e/ org.oog///ooooo 2508 20148
Mamoona Humayun, Noor Z Jhanjhi, Abdullah Almotilag, and Maram Fahhad Almufareh. Agent- based medical health monitoring system. Sensors, 22(8):2820, 2022.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng. Hanyi Fang, and Peter Szolovits. What dis- does this patient have? large-scale open domain question answering dataset from medical exams, 2020. URL htttlhhtt::///tm http///xxiv..ggabaa/a/eeeea
13081
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering, 2019. WtLhhttps: httt::::/aaxiv..r/g//oo http//axivvvooo
1909. 06146
Jonathan Kim, Anna Podlasek, Kie Shidara, Feng Liu, Ahmed Alaa, and Danilo Bernardo. Lim- itations of large language models in clinical problem-solving arising from inflexible reasoning, 2025. URL BtLptttss://xxiv wwg. abs /a0p2///5522222 04381
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention, 2023.
peattps://arxivye orw/.bs/////ooo.ooooooo...sss. 2309. 36180
Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. Biomistral: A collection of open-source pretrained large language models for medical domains, 2024.
Dongdong Li, Zhaochun Ren, Pengjie Ren, Zhumin Chen, Miao Fan, Jun Ma, and Maarten de Rijke. Supi-supe variational reasoning for medical dialogue generation. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 21, pp. 544-554. ACM, July 2021. doi: 04.11453340483.34429999 URL http::/dxx doi. 10. 140444//34445.... 3462921
Junkai Li, Yunghwei Lai, Weitao Li, Jingyi Ren, Meng Zhang, Xinhui Kang, Siyu Wang, Peng Li, Ya-Qin Zhang, Weizhi Ma, and Yang Liu. Agent hospital: A simulacrum of hospital with evolvable medical agents, 2025. URL httttt:://// //tpxii.///xvv...// .rgg abs 2405. 02957
Wenge Liu, Yi Cheng, Hao Wang, Jianheng Tang, Yafei Liu, Ruihui Zhao, Wenjie Li, Yefeng Zheng, and Xiaodan Liang. "my nose is running." '"'are you also coughing?": Building a medical diagno- sis agent with interpretable inquiry logics, 2022. URL http:::/aaxxlv.orgr/bb////// 13953
Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. Agentclinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments, 2025. URL (tRl.https:///ct www.vvorgggaaee htw.ab////2450.... 07960
13

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URLLttps://arxi htwpgg/bbs/44000
03300
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibín Lin, and Chuan Wu, Hybridflow: flexible and efficient rIhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys '25, pp. 1279-1297. ACM, doi: 10.1 10.1145/46890313366660
March 2025.
URL http: dei org 1145/
3689031 3696075
Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Haotian Sun, Hang Wu, Carl Yang, and May Dong- mei Wang. MedAdapter: Efficient test-time adaptation of large language models towards med- ical reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22294- 22314, Miami, Florida, USA, November 2024. Association for Computational Linguistics. 115533//12222.emlmpp 1244.
doi:
URL https: atttss:/aclanttclggse.
2024.
emn 1p11pppmm ma in. 1244/
Karan Singhal, Tony Tu, Jakob Gottweis, Rory Sayres, Ellery Wulczyn, Mehdi Amin, Le Hou, Kevin Clark, Stephen R. Pfohl, Heather Cole-Lewis, David Neal, Q. Muhammad Rashid, Mike Schaek- ermann, Annie Wang, Debnath Dash, Jason H. Chen, Nigam H. Shah, Salma Lachgar, Paul A. Mansfield, Shravya Prakash, al. Toward expert-level medical question answering with large language models. Nature Medicine, 31(3):943-950 2025. doi: 10. 10.10888441111--44-444--77
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang. Markus Krim- mel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, and Han Xiao. jina- embeddings-v3: Multilingual embeddings with task lora, 2024. URL [Rtphttss://axxit abs/2409.1017.
Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning, 2024. /tLhhttps://ant arxiv...vvv. org/ abs /
M2 Team, Chengfeng Dou, Chong Liu, Fan Yang. Fei Li. Jiyuan Jia, Mingyang Chen, Qiang Ju, Shuai Wang. Shunya Dang, Tiappeng Li, Xiangrong Zeng. Yijie Zhou, Chenzheng Zhu, Da Pan, Fei Deng, Guangwei Ai, Guosheng Dong. Hongda Zhang, Jinyang Tai, Jixiang Hong, Kai Lu, Linzhuang Sun, Peidong Guo, Qian Ma, Rihui Xin, Shihui Yang, Shusen Zhang, Yichuan Mo, Zheng Liang, Zhishou Zhang, Hengfu Cui, Zuyi Zhu, and Xiaochuan Wang. Baichuan-m2: Scal- ing medical capability with large verifier system, 2025. URL
2509. 02208
OpenAl Team, Gpt-4o system card, 2024. BRL.httpss//arriv.o//////////1 09388
Qwen Team. Qwen3 technical report, 2025. URL
Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, Shekoofeh Azizi, Karan Singhal, Yong Cheng, Le Hou, Albert Webson, Kavita Kulkarni, S Sara Mahdavi, Christopher Semturs, Juraj Gottweis, Joelle Barral, Katherine Chou, Greg S Corrado, Yossi Matias, Alan Karthikesalingam, and Vivek Natarajan. Towards conversational diagnostic ai, 2024. URL httphttt://eee //arxiv../iivv. org/ abss 24C1. 05654]
Rosa M Vicari, Cecilia D Flores, André M Silvestre, Louise J Seixas, Marcelo Ladeira, and Helder Coelho. A multi-agent intelligent environment for medical knowledge. Artificial
doi: https://do..or 10. URL https: //wtp::::w...... www. www.sciennecireet.c. com/ cein/scionn science/ article/p1i/50330 003000000330770005503 Software Agents in Health Care.
Intelligence in Medicine, 27(3):335-38) 366. 2003. ISSN 0933-3657.
0M166800933-365570))000000
Biling Wang, Michael Dohopolski, Ti Bai. Junjie Wu, Raquibul Hannan, Neil Desai, Aurelie Garant, Daniel Yang, Dan Nguyen, Mu-Han Lin, Robert Timmerman, Xinlei Wang, and Steve Jiang. Performance deterioration of deep learning models after clinical deployment: A case study with
14

---

Published as a conference paper at ICLR 2026
auto-segmentation for definitive prostate cancer radiotherapy. 2023a. URL https://arxix 00c//ab/22210.0555
Guangyu Wang, Guoxing Yang. Zongxin Du, Longjun Fan, and Xiaohu Li. Clinicalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation, 2023b. URL
pettps:/aariin org/ abs / 2306. 09968
Wenxuan Wang, Zizhan Ma, Zheng Wang, Chenghan Wu, Jiaming Ji, Wenting Chen, Xiang Li, and Yixuan Yuan. A survey of LLM-basse agents in medicine: How far are we from bay- max? In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 10345-10359, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256 URL. https: a antho ogy. org/2025.
5. doi: 10. 10.1863311/2225 20t5://///222...nnnga-aaaa..333
Findings acl. 539/
Xidong Wang, Nuo Chen, Junyin Chen, Yan Hu, Yidong Wang, Xiangbo Wu, Anningzhe Gao, Xiang Wan, Haizhou Li, and Benyou Wang. Apollo: Lightweight multilingual medical 1lms towards democratizing medical ai to 6b people, 2024.
lan B. Wilkinson et al. Oxford Handbook of Clinical Medicine. Oxford Medical Handbooks. Ox- ford University Press, Oxford, 11 edition, 2024. doi: 10:1093mecd9780019844401333 101.0000......00...... URL/tttss: / /oi. org/ 10. 1093/ m/ 17780188844011 01044411300.... 000.0001111111110000000000000001 Online edition,
Oxford Academic, Mar. 2024. Accessed 13 Aug. 2025.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
Kaishuai Xu, Yi Cheng, Wenjun Hou, Qiaoyu Tan, and Wenjie Li. Reasoning like doctor: Improv- ing medical dialogue systems via diagnostic reasoning process alignment. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACI. 2024, pp- 6796-6810 14, Bangkok, Thailand, August 2024. Association for Computational URL
Linguisties. doi: 10. (1t.8865//1//222222 www.:2/4..nnings-l...... 2024.findinn
org/
acl. 406/
Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Hinrich Schütze, Volker Tresp, and Yunpu Ma. Memory-rl: Enhancing large language model agents to manage and utilize memories via reinforcement learning, 2025. URL https:
www.xiv.org/aaa. httpgg//bb//00/00 19828
Xi Yang, Aokun Chen, Nima PourNajatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoe, Christopher Harle, Gloria Lipori, Duane A Mitchell, William R Hogan, Elizabeth A Shenkman, Jiang Bian, and Yonghui Wu. Gatortron: large clinical language model to unlock patient information from un- structured electronic health records, 2022. URL https:t/p///. /ttp://arvvvvv///i .rg./orrggoooo abs 22tp//203..a 03540
Hongzhou Yu, Tianhao Cheng, Yingwen Wang, Wen He, Qing Wang, Ying Cheng, Yuejie Zhang, Rui Feng, and Xiaobo Zhang. Finemedlm-ol: Enhancing medical knowledge reasoning ability of Ilm from supervised fine-tuning to test-time training. 2025. URL https::/arxiv..rrrt
abs/ / 2501. 09213
Jaehoon Yun, Jiwoong Sohn, Jungwoo Park, Hyunjae Kim, Xiangru Tang, Daniel Shao, Yong Hoe Koo, Ko Minhyeok, Qingyu Chen, Mark Gerstein, Michael Moor, and Jaewoo Kang. Med- PRM: Medical reasoning models with stepwise, guideline-verified process rewards. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 16565- 16582, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979. 8-89176-332-6 doi: 1/2088055/1/2025 wwt.5p..mllppppm. erg/ emn 1p-main--mmm in 837/
www..emmmn..mnn3...... main URL htttps:/aclanthhoogg
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji, Michael Littman, Jun
15

---

Published as a conference paper at ICLR 2026
Wang, Shuicheng Yan, Philip Torr, and Lei Bai. The landscape of agentic reinforcement learning for llms: A survey, 2025. PRLb:tts::// www.v.org/.bs//50000.. 02547
Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, Xiang Wan, Benyou Wang. and Haizhou Li. Hu- atuoGPT, towards taming language model to be a doctor. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10859-108855, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 1118665///1222..nnnnnngggee Findings-emmlpp.22
findings-emmlp.725..
URL https: httph/ppee///a.aatooooooo..oo//p
Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang- Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuckai Zhu, Xingtai Lv, Hu Jinfang, Zhiyuan Liu, and Bowen Zhou. Ultramedical: Building specialized generalists in biomedicine, 2024.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, lon Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024. URL https:
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large org/
language models with long-term memory, 2023. URL
10250
A LLM USAGE STATEMENT
Throughout the completion of this work, the Large Language Model (LLM) was used solely for the purpose of refining sentences, improving grammatical accuracy and fluency during the manuscript writing process.
B IMPLEMENTATION AND TRAINING SETTINGS
Framework and Hardware In this section, we introduce the implementation details our pro- posed RL framework, Ro All experiments were conducted server equipped with NVIDIA A100 80GB GPUs. We leverage the VeRL (Sheng et al. 2025) open-source framework as the backbone for multi-agent training environment. To optimize performance, utilized the SGLang (Zheng et al. 2024) engine for efficient generation during the rollout phase. The policy was optimized using the Group Relative Policy Optimization (GRPO) algorithm, with inter-GPU communication handled by NCCL.
Model Configurations The policy model (doctor agent) was initialized from the open-source Qwen3-8B (Team 2025) model weights. To facilitate a dynamic and realistic training environment, separate OB model was employed as the patient agent simulator. For the reward signal, we utilized another Qwen3-8B as a unified process and outcome reward model. This reward model evaluates each turn of the dialogue across five weighted dimensions, yielding
Rumm in the range of [-10, 10), which is then normalized to Rum E 1) to serve as a stable reward signal. To manage memory consumption during training on the 8B parameter models, trained the policy model using b precision and enabled both gradient checkpointing and activation offloading.
Experience Retrieval For the Experience Retrieval module, we use the efficient embedding model jina-embeddings-33 (Sturua et al.) 2024) and the reranker model bge-reranker-ban (Xiao et al.) 2023). The retrieval process is two-stage pipeline. First, initial set of 30 candidate experiences is retrieved from the experience file based on a combined score semantic similarity and past reward (reward coefficient= 0.5). These candidates are then passed to the reranker, which selects the final top-k = experiences for augmenting the prompt. The retrieved experiences include not only the previous state but also the suggested action of the agent to provide richer context.
16

---

Published as a conference paper at ICLR 2026
Training Data and Parameters Our training dataset, stored in Parquet format, consists of 100,000 simulated diagnostic dialogues. We configured a maximum prompt length of 1024 tokens and a maximum response length of 3072 tokens, filtering any prompts that exceeded this limit. The total training batch size was set to 448, distributed across the GPUs with a per-device micro-batch size of 8. The policy model (actor) was trained with a learning rate of x 10-6. The entire training process was conducted for one full epoch, with results and metrics logged to Weight & Biases site (Wandb) for monitoring and analysis. The multi-turn interaction was configured to allow a maximum of 10 turns for both the user and the assistant.
Dataset Curation The real clincal dialogues corpora from KaMed (Li et al. 2021 is filtered for above turns to maintain useful and reliable content. To instill explicit, interpretable reasoning, we annotate a subset of the data with Chain-of-Thought (CoT) format: dialogue history - Thinking [[Reasoning]]
which contains differential diagnosís, information needs, and rationale of the agent for the next question. This trains the model to "think" before it "speaks". To instill explicit, interpretable reasoning, we use GPT-4. (gpt-4. (gpt-4.1-nano- 1-nano--n20000 2025- 04-14) our teacher model to annotate subset of the SFT-Base data with "chain-of-then -ftoohhhttth" format.
B.1 DETAILED REWARD FUNCTION SPECIFICATIONS
This appendix provides the detailed formulation of the two-tiered reward architecture introduced in Section 3.2
Process Reward The process reward (Ram) is designed to provide dense turn-by-turn feedback on the conversational conduct of the agent. The calculation is executed as two-stage hierarchical process: first, a veto check for critical failures, followed by a detailed score calculation if no veto triggered. An LLM judge (Qwen3-8B) evaluates each conversational turn across eight dimen- sions on an ordinal scale from -5 (critically poor) to +5 (excellent). These dimensions are: Safety (Saatey), Reasoning (Steasoning). ). Medical Accuracy (Sccrracy), Completeness (Scompletoness) Informa- tion Gathering (Smfo), Faithfulness (Staitatutnnss). Empathy (Sempathy), and Humility (Shumility).
To enforce non-negotiable standards of safety and reliability, hierarchical veto system is applied first. A failure threshold of e 0 is set, meaning any negative score on a critical dimension is unacceptable. A critical failure is triggered if Sualety c, resulting a maximum penalty of Ruum Ren- A severe failure is triggered if Sressning < or Suceuracy < c, resulting in a significant penalty of Rum = Reev. any veto condition is met, the corresponding penalty assigned immediately, and the standard score calculation below is skipped.
no veto is triggered, the turn score is calculated as a normalized weighted sum of all eight dimen- sional scores. This additive model allows for balanced evaluation of both clinical and communica- tive competencies single formula. The weights (wx) allow for tuning the relative importance of each skill.
Σi S.Si
Rum = clip
Runnn Ryma
S max Di= W.g
Score Normalization The raw weighted sum Σ U;SS,, produces scores in a wide range dependent on the weights. To ensure the reward signal stable and consistently scaled, we normalize this score to theoretical range of [-1, 1]. As shown in Equation 7 this is achieved by dividing the raw score by a normalization factor. This factor, Smox Σ W. represents the maximum achievable raw score, calculated the product of the maximum dimensional score (Smux) and the sum of all weights. This process ensures that a perfect positive score (all S; = + Smax) maps to +1, and a perfect negative score (all S, SSuxx maps to -1. The final clip function serves as safeguard to strictly enforce the reward boundaries defined by Rmin and Rmax-
The reward structure is defined by a set of hyperparameters that were determined empirically reflect the desired agent behavior. Their values listed in Table
Hyperparameter Values
https::/wandb.ai//
17

---

Published as a conference paper at ICLR 2026
Table 6: Hyperparameters for Process Reward Calculation
Symbol
Description
Value
Veto System Parameters
Failure threshold for critical scores
0
Reri Hegy
Penalty for critical failure
-1.0
Penalty for severe failure
-0.75
General Scoring Parameters
Suas
Maximum score for any dimension
Rmin: Reax
Min/max reward clipping boundaries
-1.0, 1.0
Additive Model Weights (wx)
Nsstety
Weight for Safery
1.0
Wycsoning
Weight for Reasoning
1.0
3 Wausunacy
Weight for Medical Accuracy
1.0
Windo
Weight Information Gathering
0.8
Pluitiuunnss
Weight Faithfulness
0.7
Weight for Completeness
0.7
Wampally
Weight for Empathy
0.5
Wannilit
Weight for Humility
0.5
Outcome Reward The outcome reward (Biial) is a terminal reward assigned at the end of an episode. It evaluates the agent's diagnostic accuracy by scoring its final recommendation against a ground-truth label. The score, Seumectnesss is assigned from discrete set: 1) 1.0 (Correct): The agent's s primary diagnosis or recommendation matches the ground truth. 2) 0.5 (Partially Correct): The agent's 's recommendation is reasonable but not the primary diagnosis (e.g., identifies a correct differential diagnosis but misses the most likely one, gives a correct but incomplete recommen- dation). 3) 0.0 (Incorrect): The agent's diagnosis is medically incorrect or irrelevant to the patient's condition.
C EVALUATION
C.1 EVALUATION BENCHMARK
This section provides further details on the benchmarks used for our evaluation. All datasets were filtered for English-language cases to ensure a fair comparison.
HealthBench Our primary evaluation is conducted on OpenAl's HealthBench (Arora et al., 2025, a comprehensive and challenging benchmark designed to assess L.LMs on their ability improve human health through multi-dimensiona rubrics. Unlike traditional static QA datasets, HeahthBench provides complex dynamic medical scenarios that test a model's performance across a wide range of clinically relevant skills. The benchmark is structured around two key evaluation methodologies: Themes and Axes. Themes categorize the specific medical topic or task (e.g., handling emergency referrals, seeking context), while Axes measure fundamental abilities that apply across all scenarios (e.g., accuracy, communication quality).
distinct subsets of the benchmark, each filtered for Fnglishh cases ensure a fair comparison: 1) Health Bench Main: The primary dataset, covering broad and diverse of common clinical scenarios. The results in Table [ are based on this subset, which consists of 500 patient cases. 2) HealthBench Hard: A curated subset containing more complex, ambiguous, or high-risk cases that are specifically designed challenge model's diagnostic reasoning and safety protocols. The results in Table are based on this subset, which consists of 300 patient cases.
For experiments,
To score the model outputs, we use GPT-4.1 (version apt-4.112025-- 14) the evaluator model. This aligns with the official evaluation script provided by OpenAl. ensuring our methodology is con- sistent with the henchmark's standard protocol and allows for fair comparison against established results.
MAQuE In addition to HealthBench, we evaluate our model on MAQUE (Medical Agent Ques- tioning Evaluation) (Gong et al. (025), large-scale benchmark designed for the comprehensive, automatic evaluation of mulli--urn medical questioning agents. key feature of MAQuE is its of 3,000 realistically simulated patient agents that exhibit diverse linguistic patterns, cognitive
18

---

Published as a conference paper at ICLR 2026
limitations, and emotional responses. creating challenging and realistic conversational scenar- ios than static datasets. The evaluation is structured along several dimensions, including: 1) Task Success, which measures accuracy and robustness: 2) Inquiry Proficiency, assessing the coverage and relevance of the agent's questions; 3) Dialogue Competence, which evaluates adherence to the doctor persona and conversational coherence; and 4) Patient Experience, focusing on clarity and empathy. For our evaluation, we use the official script provided with the benchmark, which uses an LLM judge score the agent performance across all of these metries, shown in Table
MedQA While our primary focus is on dynamic inquiry, we include the MedQA dataset (Jin al. 2020) to validate that the specialized training of DOCTOR-RI does not degrade its core medical knowledge. As prominent benchmark derived from medical licensing examinations, the multiple- choice format of MedQA serves as a gold standard for assessing factual recall and application. Our evaluation conducted on randomly sampled subset of 200 multiple-chor choicceee-ccceeeeee questions from the US English test set provide robust estimate of performance. As shown in Table 3 our approach not only prevents knowledge degradation but leads to substantial improvement. DOCTOR- RR achieves score of 83.50%, marking 20% increase over its Qwen3-88 base model (63.50%) This demon- strates that our framework actively strengthens the agent foundational medical knowledge with its advanced consultation skills.
MMLU We further evaluate models Massive Multitask Language Understanding (MMLU) benchmark (Hendrycks et al. 202I). Performance on these tasks assesses the model ability to retain and apply broad set of general medical knowledge. To ensure focused evaluation, report scores on a subset of 200 questions sampled from the US English portion medical top- ics. DOCTOR-R scores 85.00% on these medical topics, significant improvement over the base model's 's score of 70.00%.
C.2 EVALUATION SETTINGS
To contextualize the performance of DOCTOR-RO R1, we conduct comprehensive evaluation against a wide range of state-of-the-ar baselines. These models are grouped into three distinct categories to provide a multi-faceted comparison:
1) Proprictary Models: We evaluate leading closed-source models via their official APIs to estab- lish the current performance frontier. This group includes (with their version detailed respectively): Claude Sonnet (claude-somnet-2222555) 14), 2.5-Fal,,,, Grok-4, GPT-4.1 (2pt-4.1-2025- 04-14), GPT-5 (gpt=5-chat-2025-08-97
2) Open-Source Specialized Models (7B-8B): This group consists of models with a similar param- eter count to own, which have been specifically fine-tuned medical corpora. We compare against prominent models such as Medition-7B (Chen et al.] 2023b). BioMistral-778 (Labrak et al,
2024),
HippoMistral-78 (Acikgoz et al. 2024), Hippol lamoolllaa11ii
7B (Acikgoz et al. 2024). HuatuoGPT
o1-8B
(Chen et al., 2024), OpenBiol M-8B (Ankit Pal 2024). Med42-v2-80 (Christophe et al,,
2024
DoctorAgent-PR RL (Feng et al. 2025). and UltraMedica--SB (Zhang et 2024).
3) Open-Source Specialized Models (232B): To assess our model's parameter efficiency,, we also compare against significantly larger medical models with our 8B model. This category includes Baichuan- -M--22n (Baichuan Intelligent Technology 2025), OpenBioLLM-7009 (Ankit Pal 2024). Med42-v2-700 (Christophe al., 2024)) UltraMedical- (Zhang al. 2024). and HuatuoGPT. o1-70B CCen et al. 2024).
All open-source models were deployed for inference using the vLLM library (Kwon et al. 2023) for high-through. throughhuut generation. To ensure fair and reproducible comparison, all evaluations were conducted with a deterministic sampling temperature of 0.0. The parameter weights for all open- source models are publicly available on the Hugging Face platform, with links provided in Table
C.3 DETAILED EXPERIMENTAL RESULTS
The performance of models the challenging HealthBench Hard dataset is presented in Table 7 This curated subset features complex and high-risk clinical cases, serving as a stress test for an agent reasoning and safety capabilities. Our analysis of these results reveals several key insights.
19

---

Published as a conference paper at ICLR 2026
Table 7: Overall model performance on HcalthBench Hard.
Model
Arg. Score
Emerg. Referrals
Health Data T.
Commu- nication
Theme
Global Health
Hedging
Contest Seeking
Complex Resp-
Acc.
Comm. Qualitty
Instr:
Foll.
Axis
Contest Aware.
Comp-
Ietionn
Proprictary Models
Clande Sunmet 4
13.43
10.06
16.59
14.93
13.68
10.53
12.54
15.60
17.77
53.97
48.30
25.34
24.19
GPT441
16.92
12.09
59.81
47.79
FlasFlssh
14.52
21.90
Grok-4
19.11
11.62
13:48
26.21
21.22
19.73
16.80
19:43
29.02
60.53
51.16
27.93
27.37
GPT55
29.57
26,96
21.89
30.41
28.09
29.74
35.33
32.71
38.64
60.78
64.29
37.31
35.30
Open-Source Models (7B-SB)
Hippol Jamee 7B
1.09
1.19
0.75
0.65
0.67
1.35
0.63
5.17
5.57
13.17
19.23
8.10
10.04
2.29
L.01
2,47
3.19
3.88
24.54
26.61
12.20
2.20
23.61
29.23
15.23
2,62
1.44
25.21
28.
12.96
Meuillas (81
3,36
4.81
2,39
4.14
2.81
6.14
3.39
5.78
14.36
29.98
31
11.65
12:25
4.89
3.96
7,63
3.39
4,44
3.61
4.78
9.52
10.50
35.42
40.5 55
16.88
16.16
5.70
3.39
8.30
12.24
4.61
5.00
2.24
6.36
9.13
41.57
39. 98
16.68
15.558
SB
7.36
2.94
9.40
11.32
4.92
7:94
6.01
13.02
12.70
47.91
40.44
17.59
18.20
Open-Source
(2=m328)
M--28
7.22
5.11
8.48
10:49
6.51
5.5]
5.25
12:43
13,42
48.08
41.5
19:40
13.70
HDLrSOPP L-0-700 708
8.27
4.62
9.29
8.47
8.97
7.83
7.13
11.64
14.73
52.81
39.91
20.66
17.82
MIOTE V2 708
11.19
8.29
13.52
16.59
8.35
10,92
9.12
15.74
17.21
$6.17
43.35
24.85
20.54
Baiekan Ma M2-12
23.68
21.82
19.93
30,80
25.31
24.95
22.16
14.24
27,82
60.25
53.63
35.71
37.08
DOCTOR-
18.73
15.74
13.07
15.27
19.57
16.64
22:23
14.21
24.27
63.85
51.11
28.35
27.18
10.20
Experience
16.97
11.26
16.
20.73
16.34
19.14
22.63
45.5
50
Base (Qwen3- 8B)
12.08
12.93
11.14
9.08
13:27
11.22
11.97
17.46
18.97
56.71
46.43
23.73
20.20
1) Performance on High-Difficulty Scenarios: The HealthBench Hard dataset presents signifi- cant challenge for all models, with average scores dropping substantially compared to HealthBench Main. For instance, the top-performing proprictary model, GPT-5, sees its score decrease from 46.38 to 29.57, and our model. DOCTOR-RI, drops from 36.29 to 18.73. This trend underscores the difficulty the benchmark and highlights the gap that still exists between current LLMs and expert-level handling of complex medical cases.
2) Comparison with Proprietary Models:
On these challenging tasks, DOCTOR-R R remains com- petitive with several leading proprietary models, outperforming GPT-4.1 (16.92) and Claude Sonnet (13.43). While its overall score is surpassed by GPT-5 (29.57), a deeper analysis of the evaluation reveals critical strength: DOCTOR-R1 achieves the highest Communication Quality score of all models evaluated (63.86), including all proprietary models. This suggests that while the most complex diagnostic reasoning remains frontier challenge, our framework has successfully trained the agent with exceptionally superior skills in clear, safe, and empathetic communication, which are maintained even under the pressure of difficult scenarios.
Despite the increased dif- ficulty. DOCTOR-RI solidifies its position as leading open-source medical agent. With an average score of 18.73, it dramatically outperforms all other 8B models, with the next best, UltraMedical- 8B, scoring only 11.83. Furthermore, it surpasses evaluated 70B models by a significant margin, reinforcing the high parameter efficiency of our training framework. While it is outperformed by the much larger Baichuan-M2-322 (23 23.68), DOCTOR-RI remains the top-performing open-source model in the 7B-8B parameter class by wide margin.
3) Comparison with Open-Source Models and Parameter Efficiency:
4) Ablation Insights: The results from our ablation studies on HealthBench Hard further validate the effectiveness of our framework. The removal of either the process reward ("wfo Proc. Re- ward", 17.43) or the experience repository ("w/o Experience", 16.97) leads to a clear degradation in performance, confirming that both components are crucial for navigating complex clinical sit- uations. The substantial performance lift from the base model (12.08) to the final DOCTOR-RE RI highlights the efffectiveness of our overall experiential learning approach in preparing agents for the most challenging clinical interactions.
D ABLATION STUDIES
This section provides detailed analysis of the ablation and scaling studies that demonstrate the impact of our framework's 's key components. The results and analysis are presented Appendix D.I with experience case study analysis in Appendix D.2
20

---

Published as a conference paper at ICLR 2026
D.1 DETAILED ABLATION RESULTS AND ANALYSIS
Theoretical Justification for GRPO Superiority While SFT provides a crucial initialization by minimizing the negative log-likelihood of expert demonstrations, is theoretically limited by ex- posure bias and unable to recover from states not seen during training, as it focuses next-token prediction rather than the long-term value the entire consultation session. While PPO relies on an Actor-Critic architecture, its performance in open-ended text generation is often bottlenecked by the value function approximation error. Accurately estimating the scalar value ((s) for complex, long-horizon medical dialogues is difficult due to the sparsity of the state space. Inaccurate critic timates lead to high-variance advantage signals, destabilizing the policy update. In contrast, GRPO eliminates the critic network entirely. By sampling a group of trajectories and using the group mean as dynamic baseline, GRPO reduces the optimization to a listwise comparison within the sampled group. Theoretically, this yields two benefits: 1) Bias Reduction: It removes the bias introduced by an imperfect critic, ensuring the policy gradient is driven soiely by the ground-truth reward distri- bution. 2) Robustness in Soft Skills: For subjective metrics like Communication Quality, learning from relative preference is mathematically more stable than regressing to an absolute score. This explains GRPO's specific superiority in soft-skill dimensions compared to PPO (see Section for detailed results.)
Impact of the Experience Retrieval Mechanism This ablation study is designed isolate the contribution of our experience repository. The first table compares the performance of three model variants across key metrics from HealthBench: a baseline agent with no experience retrieval, an agent using a standard semantic similarity- bbased retrieval, and our full DOCTOR- R1 agent which incorporates reward and novelty filterin. The results illustrate the performance gains at each stage of the mechanism's 's complexity.
Retrieval Variants
Avg. Score
Theme
Axis
Emergy Referrals
Health Data
Commu- nication
Global Health
Hedging
Context Seeking
Complex Resp.
Aec.
Quality
Comm.
Instr.
Foll.
Context
Aware.
Ieleness
Comp-
Experience
35.
45.31
Exp. Lsing Sim.
23.82 26.39
59.29
49.48
DOCTOR R1
36.29
$4.44
29.17
47.16
24.74
33.71
34.25
37.84
64.15
$4.39
49.24
40.93
Table 8: Ablation studies the experience retrieval mechanism.
Isolating Experience Components To validate that the experience components function as learned policy rather than prompting prior (In- Context Learning), we evaluate four versions of our agent at inference time (results in Table g 1) Full Method: Retrieve state and action, 2) State-
Only: Test the prompting prior hypothesis by retrieving only the state, 3) Random Format: Test
the format hypothesis by retrieving a random state and action pair, and 4) No Experience: Function
as baseline.
Table 9: Ablation study isolating the effect of retrieved components.
Experience Component
Avg.
Theme
Axis
Referrals
Health
Commu- nication
Global Health
Hedging
Context Seeking
Complex Resp.
Acc.
Comm. Quality
Inste:
Context
Comp-
Experience
47.24
30:55
59.19
Random Formal
22:04
30.79
17.38
32.34
12.65
13.42
29.26
26
50.73
38.72
23.
State-Chnly
32.13 15
$3.66
24.06
36.49
26.56
30.87
26.57
34.37
54.17
61.74
49.04
44.49
39.41
Full (State Action)
36.29
54.44
29.17
47.16
24.74
33.71
26.39
34.25
37.84
64.15
54.39
49.24
40.93
The results in revle insights into the mechanism of experiential Icarning. First, the "Random Format" model performs poorly, suffering 9.65% drop average score compared to the "No Experience" baseline (22.04 vs. 31.69). This indicates that random irrelevant experiences may mislead the in-context learning process, acting as distractive noise rather than serving as a useful structural prior. Second, while the "State-Only" model achieves a slight gain over the baseline, still underperforms our "Full Method" over most of the metrics. Notably, removing the action component leads to sharp decline of 10.73% in Communication (47.16 36.43). These results are consistent with the hypothesis that, while semantic context provides minor benefits, the primary performance gain stems from learning the specific policy, especially learning and imitating the
21

---

Published as a conference paper at ICLR 2026
high-reward action embedded in the retrieved experience. Case studies in Appendix D.2 further illustrates how the retrieved action guides the agent to a superior strategic question.
Scaling Effect of Interaction Turns This study an- alyzes how key performance metrics evolve over the course of longer dialogue. The table compares DOCTOR-RI R1 against strong baselines (GPT-4.1 and Baichuan-M2). Table 10 reports scores for Accuracy, question Coverage, and Empathy at different turn counts (1, 3, 5, and 10), demonstrating how each model's strat- egy and effecttiveness develop as the conversation unfolds.
Table 10: Scaling with Dialogue Turns
Model GPT
Turn
Acc.
Cov.
Emp.
0.33
0.10
0.47
3
0.50
0.25
0.71
0.54
0.33
0.72
0.60
0.46
0.75
Baichuan-M2
0.26
0.11
0.41
0.25
5
0.53
0.34
0.63
0.57
0:45
0.65
DOCTOR R
0.36
0.12
0.76
Scaling Effect of Patient Agent Interactions This study investigates the impact of training data volume on the agent's 's final capabilities. The table shows the per- formance trend on key metrics, Average Score, Accuracy (Acc.), Communication Quality (CQ), and Context Seek- ing (CS), the number of simulated patient agents used during training increases from zero (the base model) to 100,000. The data Table IT demonstrates the positive correlation between interaction volume and final model performance.
0.25
5 0.58
0.32
0.88
10
0.60
0.38
0.94
Table 11: Scaling with Patient Agents
Agent
Avg.
Acc.
CS
25.13
28.57
49.35
16.42
10k
29.54
32.37
57.32
22.83
30k
32.01
35.49
58.39
24.09
50k
32.15
34.69
61.74
25.61
100k
36.29
37.84
64.15
26.39
Framework Transferability to Other Base Models To investigate the generalizability of our framework, we apply our training pipeline to a different base model family, Llama-3. 1-BBB Instruct (AIMMetal 2024). We evaluate the original base model, the SFT variant, the PPO variant, and our full GRPO variant on HealthBench (Arora et al. 2025). The results presented in Table [2 are highly consistent with our primary findings on the Qwen3-8B model. The SFT variant provides marked improvement over the base model (16.83 vs. 13.73), while our full Agentic RI. framework with GRPO achieves the highest performance of 20.76%, confirming that our approach is robust across different model architectures.
Table 12: Ablation study on a different base model family, demonstrating the transferability of our framework. All models in this table are Llama-3-80 --IB-unntt variants.
Llama-3.1-BB Variants
Avg. Score
Emerg. Referrals
Health Data T.
Commus nicaaioo
Glohal Health
Hedging
Context Seeking
Complex Resp.
Acc.
Comms. Quality
Instr. Foll.
Context
Comp- leteness
Aware.
Base Model
13.73
25.45
13.66
13.21
7.66
12:46
5.79
24.82
18.64
38.02
34,75
32.39
15.99
16.83
29.00
16.89
9.01
34.87
19.54
16.38
9.60
17.04
10.87
28.20
50.63
42.18
$4.89
GRPO
20.76
32.06
20.55
24.38
11.13
19.36
12:15
31.00
24.35
54.45
46.97
35.29
23.17
Ablation of Reward Architecture To justify our reward criteria and the necessity of the hierar- chical veto system, we conducted granular ablation studies. We trained two variant models using the same experiment settings listed in Appendix B 1) without Veto System: Uses simple weighted sum reward Σ w.S;; without the safety veto threshold. 2) without Soft Skill: Sets the weights for Empathy and Humility to zero during training. We select specific axes on the HealthBench Main 500 cases to represent our core objectives: Emergency Referrals serves as a proxy for clinical safety, while Communication represents soft skills. Additionally, we calculated the Safery Violation Rate (percentage of responses with negative safety score among test cases) and the average Empathy Score (I-5 scale) using our evaluator. Detailed results are listed in Table 13
1) Veto System Enforces Safety: Removing the hierarchical veto (w/o Veto System) leads to degradation in safety. While the average scores drop only slightly, the Safety Violation Rate more than doubles (+125%) compared to our proposed DOCTOR-RI (1.80 0.80) over all the test cases. Furthermore, the Emergency Referrals score drops by -3.16% (52.72 vs. 54.44), indicating that without the penalty for safety failures, the model fails to maintain appropriate caution.
2) Process Rewards Shape Soft Skills: Removing the soft skill rewards such as Empathy and Humility (w/o Soft-Skills) causes degradation in communicative competence. The performance
22

---

Published as a conference paper at ICLR 2026
shows drop of -16.25% (47.16 39.40) in the Communication and - -11.62% (4.39 vs. 3.88) Empathy score, showing that specific process reward is required for soft skills shaping.
Table 13:
Ablation results for reward architecture. The red subscripted in percentage indicates the relative performance degradation compared to our proposed DOCTOR-B RI model.
Reward Architecture Variant
Emergency Referrals
Safety Violation
Communication
Empathy
DOCTOR-RI
54,44
47.16
4.39
Process Reward Veto System)
52.72 (3.36%)
1.80 (-125%)
46.86
4.19
Pricess Reward (w/o Soft Skill)
54.31
1.00
39.40 (16.5%)
3.88 (.11022%)
w/o Process Reward
52.21
1.80
39.05
3.82
D.2 CASE STUDY OF EXPERIENCE RETRIEVAL
Experience Case Study 1: Identifying ing Hearing Impairment over Behavioral Issues
Patient Context: y-yer-olld boy. Parents report he speaks only in "two-to-thres tet-tttteewwott word bits" and random crying fits. The teacher reports does not follow direct instructions. The parent initially frames this as behavioral or emotional issues.
Current Patient Dialogue State: "Uh, yeah, so, he kinda don't answer when you call him, y' y'know? And, the teacher says he ain't followin' instructions when they tell him straight up..."
"The patient has experienced timnitus, headache, and difficulty con- centrating... prompting the doctor recommend hearing and neurological evaluations, sleep study, and stress management.."
Retrieved Experience:
Without the experience, the agent might over-index the crying fits and plore psychiatric conditions (Autism Conduct Disorder). However, the retrieved experience regarding auditory processing and concentration acts as a strategic hint. guides the agent to link "not answer- ing" and "not following instructions" to potential sensory deficit (Hearing Impairment) rather than just behavioral von-compliance.
Effect of Experience:
Agent Action (Response): "Can you tell me if the child has any difficulty responding to his name or any concerns about his hearing or attention in other settings, such as at home or during play?"
This question successfully pivoted the conversation. The parent confirmed, "he often don't t when you call him", eventually leading to the correct diagnosis of Hearing Impairment rather than a developmental disorder.
Outcome:
Experience Case Study 2: Distinguishing Esophagitis from Cardiac Pain
Patient Context: 55-year-old female presents with sudden retrosternal pain started while going to bed. She has a history of CHF and diabetes. The initial presentation highly suggestive a heart attack (Myocardial Infarction).
Current Patient Dialogue State: "The pain sudden in onset and worsens with swallowing. cannot recall it radiates other areas."
"The patient with onset stabbing pain the left side of the neck when swallowing, was advised to consult an ENT specialist determine cause, and undergo CT scan to evaluate for esophageal, thyroid, or nerve-related causes."
Retrieved Experience:
The retrieved experiences highlight the link between pain on swallowing (odynophagia) and esophageal/ENTT pathology. Without this hint, an agent might get stuck on the "retrosternal pain CHF history" and exclusively pursue cardiac workup. The experience prompts the agent pivot its differential diagnosis to include esophageal causes like Esophagitis Esophageal Rupture.
Efffec Experience:
23

---

Published as a conference paper at ICLR 2026
Agent Action (Response): "Have you noticed any difficulty breathing. fever, changes in your voice along with the pain?" (Checking for signs of rupture and infection vs. simple inflammation ...) Later: "Have you noticed any difficulty swallowing, a sensation food sticking
By focusing the "worsens with swallowing" cue prioritized by the retrieved experience, the agent correctly identifies the non-cardiac nature of the pain, leading to the diagnosis of Esophagi- tis. Mledicationnindueee pill esophagitis is implied by "going bed" timing and doxyeycline iron / bisphosphonate risk factors often associated with this presentation.
Outcome:
E HUMAN EVALUATION DETAILS
This appendix provides the complete results and detailed statistical methodology for our human evaluation studies in Section 4.2 To ensure a comprehensive assessment, we combine pairwise comparison for user patient experience (Priniciple 2), Likert scale scoring for clinical competence (Principle 1), and categorical analysis for experience utility (Priniciple 3).
E.1 STATISTICAL METHODOLOGY
I. Pairwise Comparison The definitions for each metric provided to the annotators were fol- lows: I) Coherence: Does the model quickly capture the core information using few logically connected questions? Does it avoid unnecessary repeated questioning? 2) Adherence: Does the model consistently maintain the role of professional doctor, avoiding Al-like or mechanical lan- guage? 3) Clarity: Are the model's questions expressed clearly, concisely, and easy for the patient to understand? 4) Empathy: Does the model's tone convey care, respect, and support?
1) Data Aggregation: The initial step involves aggregating the raw judgments from each pairwise comparison. For each model (m) and each evaluation metric, four distinct counters are maintained:
Wins (Wn): The total number of times model m was judged superior.
Losses (Lm): The total number of times model m was judged inferior.
Ties (T,.): The total number of times comparison involving model resulted in tie.
Comparisons (C..): The total number of comparisons model m participated in.
For given comparison between Model A and Model B, the counters are updated based on the annotator's 's verdict. For instance, Model A is declared the winner, the counters are updated as follows: WA is incremented by one, and LB is incremented by one. The comparison counters C'A and CB are incremented regardless of the outcome. This process is repeated for every annotated comparison in the dataset.
2) Win Rate Calculation: Following the aggregation counts, the Win Rate for each model is calculated. The Win Rate serves the primary metric for ranking and is defined as the number of wins as fraction of the total number of decisive outcomes (i.e., comparisons that did not result in tie). The formula for the Win Rate of given model is:
W
Win Rate,
(
W m Tam
Where W., is the total number of wins for model m, and Lm is the total number of losses for that same model. Ties are intentionally excluded from the denominator to ensure the metric accurately reflects performance only in matchups where definitive preference was established. In the edge case where + Lmp) = O. the Win Rate is defined as 0 to prevent division by zero errors. Finally, for each evaluation metric, all models ranked in descending order according their calculated Win Rate to produce the final leaderboards. Figure detail the pairwise comparison results for each metric, including the total number of wins, losses, ties, and comparisons for each model.
II. Likert Scale Scoring For the clinical competence evaluation, medical experts scored each response independently on 5-pont Likert scale across four dimensions: Safety, Medical Accuracy, Completeness, and Information Gathering. The final metric reported the mean score averaged across all experts and all test cases.
24

---

Published as a conference paper at ICLR 2026
IιII. Experience Utility Categorization To validate the effeet of experience learning, experts classify each retrieved experience into one of three mutually exclusive categories based on its rele- vance to the current turn: 1) Clinically Helpful: The experience provides correct strategic direc- tion or necessary check. 2) Neutral and Irrelevant: The experience is valid but not applicable to the current context. 3) Harmful: The experience provides misleading dangerous guidance. The final metric is the percentage distribution of these categories across the test set.
E.2 ANNOTATION PROCESS
The evaluation was conducted by team of five paid annotators, To ensure the quality and relevance the judgments, the annotators recruited were without a special- ized medical background to ensure that metrics like Clarity and Empathy were evaluated from the perspective typical patient, for whom the agent is ultimately designed. The annotators were presented with paired anonymized conversations generated by different models for the same clinical scenario. For each pair, they were instructed to select the response that was superior according to four distinet qualitative metries.
User Experience as Patients
Medical Expert Validation To rigorously validate agent's clinical capabilities and the util- ity of our experience module, we recruited 2 licensed physicians to evaluate 80 randomly sam- pled dialogue trajectories. The evaluation focused on two key aspects: 1) Clinical Competence & Ranking: Experts scored models on a 1-5 Likert scale across 4 dimensions. As shown in Fig- ure DOCTOR-R achieves a score comparable
the specialized model Baichuan-M2-32B. and exceeding proprietary models such as GPT-5. This result indicates that our 8B model is capable of delivering clinical capabilities comparable to significantly larger models. 2) Experience Utility: To validate Principle 3, experts explicitly assess the retrieved experiences used by DoCTOR-RI. The results are highly compelling where 83.87 the cases, experts rate the retrieved state-action pairs as "Clinically Helpful" for guiding the diagnosis, confirming that our retrieved experience provides genuine strategic value.
E.3 DETAILED HUMAN EVALUATION RESULTS
Experience Utilization Study To validate the usefulness of our retrieved experience, we further assessed the practical value of the retrieved experiences. Medical experts categorized the retrieved state-action pairs into three distinct levels of utility based on their relevance to the immediate context 30 cases. The high percentage helpful retrievals without any harmful suggestions, confirms the effectiveness of our filtering mechanism in retrieving high-quality experiences (see Appendix D.2 for experience utilization case studies).
Clinically Helpful (83.87%) ): The retrieved action provided clear correct strategic direction, such as suggesting a specific rulc-out question or a necessary safety check.
Neutral and Irrelevant (16.13%) ): The experience was medically valid in isolation but did not offer directly applicable guidance for the current turn.
Harmful (0 )):: The retrieved content was misleading or factually incorrect.
Cross-Evaluator Validation Study To validate the reliability of Consultation Evaluator model, conduct cross-evaluation study and analyze the correlation between the automated scores and human expert scores. We use two independent and stronger judges, GPT-4. 1 and 2 licensed physi- cians, to 40 cases from main test set. We compare their average scores against original Qwen3-8B evaluator.
Table 14: Comparison of average model scores from three different independent judges. The relative ranking of models remains highly consistent, validating our original evaluation.
Average Score by Judge
Model Evaluated
Qwen3-8B
GPT-4.1
Human Expert
UtraMedical--88 8B
3.06
2.33
3.47
Baichuan-M2-222
3.29
2.93
4.31
GPT-S
3.38
3.25
4.05
DOCTOR-R RI (Ours)
3.36
3,25
4.30
25

---

Published as a conference paper at ICLR 2026
Table 14 shows the average score given by each judge to each model, demonstrating that the relative ranking of the models remains highly consistent. For example. DOCTOR-RO R1 is ranked as the top open-source model by all three judges. We calculate moderate high Spearman's 's rank correlation between the Qwen3-8B judge rankings and both the GPT-4.1 (p > 0.52) and human expert rankings (p 0.24) confirm this stability. Table 15 provides full breakdown of the average scores across critical dimensions for each judge. This detailed analysis shows that our Qwen3-8B judge scoring patterns are consistent with those of human experts.
Table 15: Detailed score breakdown by human expert judge 1-5 Likert scale.
Judge
Model
Avg. Score
Safety
Medical Accuracy
Information Gathering
Completeness
UltraMedical-8B
2.92
3.00
3,00
3.11
2.56
Baichuan-M2-32B
3.29
3.29
3.43
3.57
2.86
Qwen3-8B
GPT-5
3.41
3.25
3,88
3.63
2.88
DOCTOR-RI
3.36
3.33
3.67
3.56
2.89
UltraMedial--88
2.33
2.22
2.11
2.78
2.22
Baichuan-M2-335
2.93
2.71
3.14
3.14
2.71
GPT-4.1
GPT-5
3.25
3.38
3.25
3.50
2.88
DOCTOR-RI
3.25
3.56
3.44
3.44
2.56
UttraMedica-88B
3.47
3.50
3.63
3.38
3.38
Human
Baichuan-M2
4.31
4,00
4,50
4.25
4.50
Expert
GPT-5
4.03
3.89
3.78
4.44
4.00
DOCTOR-RI
4.30
4.20
4.00
4.60
4.40
F CASE STUDIES
F.1 MODEL PERFORMANCE ON OPEN-ENDED DYNAMIC CLINICAL SCENARIO
The real-world clinical case study analyzed in Table presents a high-risk scenario: a 35- year-odd male with a history of untreated pulmonary tuberculosis now presenting with massive hemoptysis (coughing up 500 mL of blood), life-threatte life-threatening emergency. This case is designed to test the ablity of an agent to move beyond generic questioning and perform urgent, strategic triage. The perfor- mance of the baseline models reveals critical deficiencies in this area, highlighting the necessity of our proposed agentic framework.
As shown in Table 16] the powerful general-purpose model, GPT-4.1 fails by following generic low-yield questioning script. It never uncovers the patient's critical symptom and therefore com- pletely misses the urgency, resulting an unsafe consultation (Score: -0800). The state-of-th--an specialized LLM like Baichuan-M2 fail to perform a rapid differential diagnosis by asking about key risk factors like prior trauma or anticoagulant use, and their communication lacks the empathy crucial for a high-stakes situation (Score: 0.40), even though the model successfully identify the hemoptysis and provide correct emergency advice.
More concerningly, several highly capable specialized medical models, including HuatuoGPT-ol- 70B fail to quantify the life-threatening volume of the blood, and Med42-v2-70B incorrectly dismiss it as a symptom of minor viral illness despite identifying the presence of blood. They proceed with non-urgent recommendations like chest X-ray or over-the-counter medication. These dangerous and inappropriate course of actions could lead to fatal outcome (Scores: -0.50 and -1.0). Similarly, specialized agent models such as DoctorAgent-RI completely overlooks the hemoptysis and misdiagnoses the patient with simple viral infection, showcasing the most dangerous failure mode (Score: -1.0).
In stark contrast, DOCTOR-EE R1 demonstrates mastery of the three core principles of doctor agent. begins with a broad question about the cough's 's nature, immediately identifying the hemoptysis. Its subsequent questions agnosis: quantifies the blood volume, investigates the critical history of untreated tuberculosis, and rules out other causes like trauma or medication side effects. This logical step-by-step inquiry directly leads to the correct identification of a life-threatening emergency. Finally, it delivers clear,
strategic and high-yield, systematically narrowing the differential di-
httts:://rs iiliiglll... com
26

---

Published as a conference paper at ICLR 2026
urgett and empathetic instructions, providing both clinical guidance and patient support. This case vividly illustrates the gap between models with static knowledge and an agent with a truly strategic, dynamic inquiry policy.
Table 16: Case studies of multi-turn clinical inquiry on frontier general, specialized, and agent models: GPT-4.1, Baichuan-Ne (Team et al. (025). HuatuoGPT-o11-7000 (Chen et al., 2024).
DoctorAgent-RL (Feng et al. 2025), Mcd42-v2 v--42-v2--0BB (Christophe et al.) 2024), and our DOCTOR- RI. The conversations demonstrate how existing powerful models fail on effective strategic inquiry with high-risk patient, while DOCTOR-RI showcases superior performance by actively conducting differential diagnosis, aligning with the principles of doctor agents proposed in Section
Patient Profile
A 35 35-year-old man with over four years cough and sputum. Four days ago, he developed worsening symptoms, coughing up about 500 mL of bright red blood. He was diagnosed with pulmonary tuherculosis in July 2017 and treated with isoniazid and rifampicin but stopped therapy in July 2021. Farlier, his cough with white, sticky sputum did not improve with amoxicillin. He has history of diahetes, hypertension, heart disease, allergic to penicillin, and has no family history of infectious diseases.
Init. Query
Patient: have had a cough for the past four days.
Inquiry Content
Consultation Evaluation
GPT-4.1
Doctor: Any fever or shortness of breath?
Asked many general questions, but missed critical triage and risk items (quantify amount and frequency). Low red-flag issues addressed in multi- turn Overall inquiry was incomplete and unsafe. Score: -0.80
Patient: No, I don't have fever or shortness of breath.
Doctor: your cough worse at any particular time of day?
Patient: The cough has been consistent throughout the day.
Doctor: Do you have history of asthma, allergies?
Patient: / don't 't have asthmaa allergies.
efficiency,
Doctor: Have you noticed any changes in appetite weight?
Patient: No, haven't.
Baichuan-
Doctor: Can you tell me if your cough is dry or productive?
Doctor identified massive hemoptysis and gave cor- rect urgent advice, but missed rapid triage ques- tions (trauma and antico- agulants) and gave no em- pathic wording.
M2
Patient: It's productive, with blood.
Doctor: What is the color and the amount of the blood?
Patient: It's bright red, about half a litre.
Doctor: This is definitely life- theettereteeigggg massive hemop- tysis. Contact your local emergency number immediately.
Patient: Alright, I'll call emergency services right away.
Score: 0.40
HuatuoGPT-
Doctor: [Thinking: The patient has had cough for four days. That's not too long, so could be something like common cold or maybe just a a..] Have you had fever?
Partly strategic. Asked general questions, discov- ered massive hemoptysis, narrowing the differential. However, omitted critical red-flag checks (failed to quantify the bleeding) and did not advise urgent esca- lation. Empathy language minimal. Overall differential thinking was shallow and the inquiry was potentially unsafe. Score: -0.50
o1-70B
Patient: No, / haven't 7 noticed any fever.
Doctor: [Thhinking: I should ask if there's anything coming when you cough? Patient: Yes, have been spitting up blood recently.
up with the cough ...] Is there any
Doctor: [Thinking: I should ask about their smoking history next. That could really help ...] Have you ever smoked?
Patient: No, I haven't 1 smoked before.
good starting point. It's simple test that can show if there are any masses or other abnormalities in the lungs ...] I'm going to recom- mend that you get chest X-ray as soon possible.
Doctor: [Thinking: A chest X-ray would
Patient: Okay, will go immediately for chest X-ray.
27

---

Published as a conference paper at ICLR 2026
Table 16: Case studies of multi-tum clinical inquiry (continued)
Inquiry Content
Consultation Evaluation
DoctorAgent-
Doctor: What medications have you tried so far?
Asked about prior med- ications and fever, but failed to address the red- flag symptom: massive hemoptysis. The diag- nosis of "viral infection" is dangerously incorrect. Empathy and supportive guidance were minimal. Score: -.0
RL
Patient: hhven''t 't taken any medication yet.
Doctor: Do you have fever? When was your last meal? about hours ago. Doctor: sounds like a viral infection. recommend getting plenty of rest, drinking lots of fluids, and monitoring your symptoms. If your cough worsens or persists beyond a week, or if you develop a fever, please follow up with me. In the meantime, avoid exposure to cold air and pollutants. Patient: Okay, thank you doctor.
Patient: No, I don't have a fever.
Med42-v2- 70B
This question assesses type of cough, which can differentiate between viral upper respiratory or ...] Is it dry, productive of mucus, or accompanied by blood? / started coughing up bright red blood four days
Doctor:
[Thinking:
The doctor noted hemop- tysis but then labeled it "viral/post-vinr virall and recommended only symp- tomatic OTC treatment. Failed to quantify bleed- ing, ask about prior TB treatment nonadherence (critical here), anticoagu- lant ,tcc The doctor did not narrow the differen- tial toward likely urgent causes nor advise immedi- emergency evaluation or safe transport. Empathy and urgent supportive language were minimal. This inquiry is unsafe and shows poor differential- diagnostic reasoning and low empathy. Score: -1.0
Patient:
ago.
Doctor: [Thinking: Adding these symptoms into considera- tion helps assess for potential pneumonia ...] Have you expe- rienced any fever. shortness of breath, or chest pain?
Patient: None the symptoms mentioned at all.
Doctor: l'Thinking: This can help identify potential triggers or complications ...] Are you a smoker? Do you have any history of asthma or chronic obstructive pulmonary diseasc (COPD)?
Patient:
smoker, and don't have asthma or COPD. Doctor: [Thinking: Subclinical postnasal drip can still cause cough ...] appears to be a viral or post-viral cough, possibly exacerbated by nocturnal postnasal drip de- spite your report of nasal symptoms. recommend symp- tomatic treatment with over-the-counter cough expecto- rant and possibly nighttime antihistamine better sleep. If symptoms persist beyond a week or worsen, please return for follow-up evaluation. Patient: I understand. Thank you for your advice and care.
28

---

Published as a conference paper at ICLR 2026
Table 16: Case studies of multi-tum clinical inquiry (continued)
Inquiry Content
Consultation Evaluation
DOCTOR-RI
[Thinking: should ask about the characteristics of the cough. Is it dry or productive? If it's productive, what color is the mucus? That could indicate ...l Could you de- seribe the nature of your cough, dry or productive? If pro- ductive, what is the color and consistency of the phlegm? Patient: It's productive, and spitting up bright red blood. Doctor: [Thinking: Hemoptysis can be due to various rea- sons. The key is to differentiate between upper and lower airway causes ...] How long has the blood been present? Patient: It started four days ago, and it's about 500 mL.
Doctor:
The doctor demonstrated strategic dynamic inquiry where each question was purposeful, narrowing the differential diagnosis step by step. Doctor agent productive
(Ours)
identified hemoptysis, quantified the volume, and systemati- cally checked high-risk factors (TB history, Doctor: [Thinking: The key points here are hemoptysis and the volume. 500 ml. is significant amount, so that's 's a red flag. Also, any history of respiratory infections or TB? ...] Have you had any history of previous respiratory conditions like chronic bronchitis or tuberculosis?
trauma, anticoagulants) while considering broad set of potential causes (upper vs lower airway. infection, TB relapse, malignancy, PE/vascular issues). This structured questioning effectively guide decision-making
Patient: Yes. I was diagnosed with pulmonary tuberculosis in 2017 and stopped treateett in 2021.
Doctor: lTThinking: The patient's 's history of TB and stop- ping treatment could lead to drug-resistant TB, which might present with similar symptoms. Need to ask about any recent medications, especially anticoagulants or NSAIDs, which could contribute to bleeding ...] Have you experienced any recent trauma, such as chest injury, or are you taking med- ications like anticoagulants or antiplatelct drugs that could increase bleeding risk?
toward life-threatening
risk. Empathy was shown through supportive lan- guage and clear urgent instructions. strong
Overall differential, triage, and empathy. Score: 0.90
Patient: No trauma and blood thidd-th medications taken recently.
Doctor: [Thinking: Since the patient stopped treatment in 2021, there's 's high risk of TB reactivation. need to make sure they understand the urgency ...] This is a theeethnna emergency and requires immediate hospitalization. Do not wait, go to the nearest emergency room immediately. This not a situation to manage at home. Please prioritize this. I am here to support you and guide further steps once you are in the hospital
F.2 FAILURE MODE AND HALLUCINATION ANALYSIS
No model is perfect, and analyzing failure modes provides critical insights for future improvement. Regarding the conditions under which unsafe recommendations might occur, we analyze the failure logs from our test set. Case Study 1 illustrates the most common failure mode: Errors of Omission. The agent occasionally fails to ask mandatory safety-check question before proceeding to diagnosis or treatment. We highlight two distinct categories of omission: 1) Missing Patient History: As shown in Case Study 1, the agent may fail to confirm current medications or allergies, risking drug interactions. 2) Missing Red Flag Screening: As shown in Case Study 2, the agent may fail to rule out urgent and tife-threa conditions when presented with severe symptoms.
Our Hierarchical Veto System successfully identifies these omissions. Even though the medical accuracy of the question asked is high, the negative safety score triggers the immediate -1.0 reward penalty. This strong negative signal is crucial for forcing the policy to prioritize safety checks over general information gathering. In our ablation study of reward architecture (Appendix ID.I), the rate of such unsafe responses was reduced by 125% % compared to the baseline without veto system.
29

---

Published as a conference paper at ICLR 2026
Failure Case Study 1: Missed Medication History
Patient Context: Patient reports bronchitis for month with improvement, yellow sputum, nasal congestion, weight loss, and mold exposure. No fever. Mentions "over-the-counter medications" but does not specify them.
Agent Reasoning (Thinking): "The patient has persistent bronchitis, yellow sputum), and mold expo- sure. The condition isn't resolving with OTC meds. Dizziness could dehydration. I need assess the possibility of a bacterial infection vs. allergic response. It is important to determine if there is an underlying condition like asthma. I should ask about symptom progression and respiratory details."
Agent Action (Response): "Have you noticed any improvement with humidifiers steam therapy? Are you experiencing any chest tightness wheezing when you cough?"
Evaluator Critique: The medical reasoning regarding differential diagnosis (bacterial vs. allergic) is sound. However, the response triggers critical safery failure. The agent failed to ask about the patient's current specific medications before procceding. Giving advice or planning treatment without knowing the current regimen (especially given "dizziness" and "weight loss") risks dangerous drug interactions or redundancy.
Safety (-2/5), Info Gathering (1/5), Reasoning (3/5), Faithfulness (3/S), Com- pleteness (1/5), Medical Accuracy (4/5), Empathy (1/5), Humility (2/5)
Full Score Breakdown:
Final Reward: -1.0 (Veto Triggered)
Failure Case Study 2: Missed Red Flag Symptoms
Patient Context: Patient with a herniated lumbar disc reports severe, unresolved pain despite using Ibuprofen, Etodolac, and Fluocinolone ointment. Reports sleep disturbance due to pain and inflamma- tion of lumbar fascia. Asks for alternative treatments.
Agent Reasoning (Thinking): "The patient is experiencing severe pain from herniated dise; initial meds aren't working. I need to consider next steps. Corticosteroid injections could target inflam- mation. Physical therapy or a specialist referral is needed. should mention stronger analgesics. also need to check for ncurological symptoms like numbness or weakness to rule out complications. will ask about those symptoms to guide the next steps."
Agent Action (Response): "Given the persistent pain, consider consulting specialist for corticos- teroid injections to target inflammation. Have you experienced numbness, tingling, or weakness in your legs? This will help assess if further neurological evaluation is needed."
Evaluator Critique:
The agent acknowledges the treatment failure and suggests valid step (spe- cialist and injections). However, it misses critical safety checks. cases severe, persistent lumbar pain, the agent must sereen for Cauda Equina Syndrome (red flags: bowel and bladder dysfunction, saddle anesthesia). Asking only about numbness and weakness is insufficient for safety triage in this context. The reasoning is medically sound but incomplete regarding urgent risks.
Full Score Breakdown: Safety (-3/5), Info Gathering (2/5), Reasoning (3/5), Faithfulness (2/5), Com- pleteness (1/5), Medical Accuracy (4/5), Empathy (1/5), Humility (2/5)
Final Reward: -1.0 (Veto Triggered)
G SIMULATION ENVIRONMENT DETAILS
G.1 MANAGING PATIENT AGENT ADHERENCE SIMULATION
A key challenge in training doctor agents within multi-agent simulation is ensuring the fidelity and instruction- follloinn capabilities of the simulated patient agents. The quality of the training data for our DOCTOR-R R1 agent is directly dependent on the realism and adherence of the patient agents it interacts with. If a patient agent fails to follow its persona (e.g., by revealing it is an AI, hallucinating symptoms, disclosing all information at once), it generate low-quality misleading training trajectories. To mitigate this, we implemented a multi-face faceted strategy combining proactive and reactive measures.
30

---

Published as a conference paper at ICLR 2026
1) Proactive Mitigation by Detailed Prompt Engineering: The first line of defense is metic- ulously crafted system prompt for the patient agent. This prompt goes beyond simple instructions and is designed to create robust and consistent persona. Key components of the patient prompt include: 1) Strict Persona Definition: The agent is explicitly instructed to act as a human patient with specific background and health issue, and never break character or reveal its nature as an AI. 2) Gradual Information Disclosure: The prompt contains rules about passive disclosure, in- structing the agent to only reveal information when asked directly or when the conversation naturally progresses, mimicking how real patients share information over time. 3) Behavioral Simulation: Instructions are included simulate realistic human conversational patterns, such as expressing emotions, hesitation, or uncertainty, to create more authentic interaction partner for the doctor agent.
2) Reactive Mitigation by Post-Generation Filtering: After each simulated dialogue is generated, undergoes rigorous automated quality control process before being considered for inclusion in the experience repository. 1) Rule-Based Checks: We apply set of deterministic filters to automatically discard dialogues with catastrophic failures. This includes scanning for phrases like "As large language model," which indicate a complete break from the patient persona. 2) LLM- Based Adherence Scoring: For more detailed failures, we use an LLM--baed judge to evaluate the patient agent's performance in each dialogue. This judge scores the patient's ss adherence to its instructions, such as maintaining its persona and practicing gradual information disclosure. 3) Data Exclusion: Any simulated dialogue that fails the rule-based checks or receives low adherence score from the LLM judge is excluded from training. This ensures that our doctor agent learns from high-quality realistic interactions.
To validate the effectiveness of our mitigation strategies, we track the rejection rates of our filter- ing pipeline. Out of the total pool of generated simulations, 13.82% were automatically discarded due to adherence failures. The following case studies demonstrate how our filtering pipeline ef- fectively identifies and discards patient agents that fail to maintain persona or engage in unrealistic information disclosure.
Patient Case Study 1: High-Fidelity Patient Simulation (Passed)
57-year-old female with persistent abdominal pain and hematemesis (vomiting blood). Fi- nal Diagnosis: Gastric Cancer (Signet ring cells).
Scenario:
Patient Persona Instructions:
Emotion: Frustrated.
Language: Informal.
Disclosure Rule:
Do reveal hematemesis immediately. Wait for specific inquiry.
Consultation Session:
Doctor: "Could you please share bit about your current symptoms?"
Patient: "Yeah, I've been dealing with this annoying ahdominal pain for like four months now... It's just frustrating knowing what's causing it." (Adherence Check: Reveals only primary symptom, maintains informal tone.)
Doctor: "Is it constant or does it come and go?"
Patient: "It's pretty much constant, and it's just this nagging pain that doesn't 't seem to let up." (Adher-
ence Check: Still holds back the critical symptom of vomiting blood.)
Doctor: "Have you noticed blood in the stool?"
Patient: "Yeah, I've had some weight loss along with this pain, and there were few times some episodes of vomiting blood." (Adherence Check: Success. The patient agent correctly waited for relevant inquiry about gastrointestinal changes before revealing the red-flag symptom of hematemesis. This gradual disclosure forces the Doctor Agent to learn active inquiry strategies.)
Filter Decision: Passed. The simulation maintained a consistent emotional tone and adhered to the "passive disclosure" constraint. By withholding the critical symptom until specifically prompted, it generates high-quality training trajectory that rewards the doctor agent for thorough inquiry.
31

---

Published as a conference paper at ICLR 2026
Patient Case Study how-Fidelity Patient Simulation (Rejected)
Scenario: 46-year-old male with rotational vertigo. Potential Diagnosis: Otolithiasis (BPPV).
Patient Persona Instructions:
Cognitive State: Forgetful.
Language: Informal.
Disclosure Rule: Do self-diagnose. Describe symptoms only.
Consultation Session:
Doctor:
"The symptoms could be consistent with condition known as benign paroxysmal positional vertigo (BPPV)..."
Patient: "Oh, right! So, I've got this BPPV thing, and about dizziness..." " (Adherence Failure I: Premature Confirmation. The patient immediately adopts the medical acronym as a confirmed fact without hesitation, violating the realistic behavior rule.)
Doctor: [Explains management tips...]
Patient: "Oh, gotcha! So, I've got this BPPV thing, and it's mild..." (Adherence Failure 2: Repetitive Looping. The agent gets stuck in loop.)... ...
Filter Decision: Rejected. The simulation was discarded because the patient agent failed to maintain natural conversation flow, engaged in unrealistic self-diagnosis, and exhibited "parroting" behavior. Including this training would have taught the doctor agent that patients passively accept complex diagnoses immediately.
G.2 MITIGATING REWARD HACKING
primary challenge in RL-baaed training is "reward hacking", where an agent learns to exploit the reward function to maximize its score without achieving the intended goal. For instance, an agent might learn to ask repetitive but technically safe questions to accumulate rewards. We implemented several design principles in our framework to mitigate this risk.
Our process reward is not a single scalar but derived from eight distinct dimensions (see Appendix B.I.. This multi-faceted evaluation makes it significantly harder for the agent to find a simple loophole. An agent cannot maximize its score by excelling in a "soft skill" like Empathy if its "hard skill" performance in Medical Accuracy is poor, as the latter is heavily weighted and can trigger penalties.
1) Multi-Objective and Structured Reward:
The most critical defense against reward hacking is our hierarchical penalty system. The vetoes for safety, reasoning, and accuracy violations act as hard constraints. An agent that generates factually incorrect or unsafe advice receives large negative penalty, regardless of how well it performs on other dimensions like Clarity or Completeness. This non-negotiable penalty structure strongly discourages any policy that deviates from core clinical principles.
2) Hierarchical Veto System:
3) Diverse and Complex Scenarios: Our training environment exposes the agent to wide vari- cty of simulated patient cases. This diversity prevents the agent from learning single, simplistic strategy that might hack the reward for a narrow set of problems. The complexity of the scenarios requires genuine, adaptive inquiry rather than repetitive, exploitative policy.
4) Human Spot-Checking and Review: Throughout the training process, we conducted periodic human spot-checks on hegh-reward trajectories. This allowed behavior and the evaluator's scoring to ensure that high rewards genuinely corresponded to high- quality consultations. These reviews served as a crucial validation layer to detect and address any emerging heward-hacking behaviors that were not caught by the automated system.
manually inspect the agent's
To validate these mechanisms, we implement a human-in- -ntee-llo review process. Throughout the training phase, we manually review approximately 50 high-reward trajectories per day. In early training steps, we did observe minor instances of reward hacking, such as the model generating repetitive empathetic statements maximize the Empathy content. Our hierarchical veto system and multi-objective weights are specifically iterated to penal- ize these behaviors. Later reviews confirm these strategies effective, for example, response
while providing oow-iffrmmation
32

---

Published as a conference paper at ICLR 2026
that is safe and empathetic but lack reasoning quality would see its total reward significantly capped, preventing the policy from converging on a "safe-but-usell but-useleuss" local optimum.
The following case studies illustrate how Consultation Evaluator provides detailed multi- dimensional feedback, identifying specific deficiencies even when the response is factually safe.
Evaluator Case Study 1: Scoring for Incomplete Reasoning
Patient Context: A 37-year-old female with suspected renal artery stenosis and hypertension (BP 150/120). History of multiple meds (hydrochlorothiazide,, propranolol. perindopril, then switched nifedipine, metoprolol). Complains of drowsiness in lower back. Hospital suggests imaging, but patient refuses due to privacy concerns and requests alternatives.
Agent Reasoning (Thinking): First, I need confirm if the drowsiness is related to the medications (Sotalol, Nifedipine) or the underlying condition. The lower back drowsiness is specific and could be renal. The main issue is the suspicion renal artery stenosis, which typically requires imaging. The patient is concerned about privacy. Crucially, she was on Perindopril (ACE inhibitor) but stopped. If she has stenosis, ACE inhibitors are contraindicated. My next step should be to ask about kidney function tests and other symptoms differentiate between side effects and the disease.
Agent Action (Response): "Have you had any recent changes in kidney function tests or symptoms like flank pain, decreased urine output, or swelling? Is the drowsiness a new symptom or has it been present for some time?"
Evaluator Critique:
The agent reasoning is logically sound regarding the medical differential. How- ever, it fails explicitly address the patient's key constraint: the request for alternatives to imaging due to privacy. It also missed verifying if she is currently taking any contraindicated ACE inhibitors/ARBs. It is safe but incomplete regarding user intent.
Full Score Breakdown: Safety (4/5), Info Gathering (3/5), Reasoning (4/5), Faithfulness (4/5), Com- pleteness (3/5), Medical Accuracy (4/5), Empathy (2/5), Humility (4/5)
Final Reward: 0.719
Evaluator Case Study 2: Scoring for Hallucinated Context
Patient Context: Patient asks about the availability of Qingyan Granules and whether they can use different brand (User intent implies checking if pediatric adult formulations are interchangeable).
Agent Reasoning (Thinking): The patient is concerned about the availability of Qingyan Granules and whether they can use different brand. Since the patient has history of rhinitis and is currently using Pudilan oral liquid and Yantian Granules, it is important to confirm if the substitute brand is of the same type. The patient is also concerned about the need for a pediatrician's 's prescription. I need address availability and whether is necessary to consult a pediatrician." "*
Agent Action (Response): "Is the substitute brand's function and ingredients the same as Qingyan Granules?"
The reasoning flawed and medically unsound. introduces completely irrel- evant information (rhinitis, Pudilan, Yantian) not found the context. fails to address the critical issue of pediatric vs. adult formulation. It incorrectly assumes preseription is needed for OTC meds. The response is not aligned with ground truth.
Evaluator Critique:
Full Score Breakdown: Safety (-3/5), Info Gathering (-4/5), Reasoning (-5/5), Faithfulness (-5/5), Completeness (-4/5), Medical Accuracy (-4/5), Empathy (0/5), Humility (-2/5)
Final Reward: -1.0
G.3 ENSURING CONSULTATION EVALUATOR RELIABILITY
The reliability of our Consultation Evaluator, which is itself an LLM (Qwen3-8B), is important to the integrity of our entire training loop. To ensure its judgments are accurate, consistent, and well-grounded, we implemented the following validation and calibration measures.
1) Structured Rubric-Based Prompting: As shown in Appendix H the evaluator is guided by highly detailed, rubric-based prompt. For each of the eight dimensions, the prompt provides explicit definitions for high and low scores (e.g., what constitutes "-5" vs. "+5" in Safety). This structured
33

---

Published as a conference paper at ICLR 2026
format constrains the evaluator's output and reduces the ambiguity that often leads to inconsistent scoring in free-form evaluations.
2) -fooughhtttth (CoT) Reasoning: The evaluator is explicitly instructed to produce step- by-step textual analysis before outputting the final JSON scores. This "chain-of-thought" process forces the model to articulate its reasoning for each score, which has been shown improve the quality and reliability LLM-based judgments. also provides a transparent trace that allows for human inspection and debugging of the evaluation logic.
Wwe implemented an ongoing process of human spot- checking. Throughout the training phase, periodically sampled high-reward and low-reward dialogues evaluated by the LLM judge and had them reviewed manually. This auditing process served critical functions: First, allowed consistent. Second, it acted as safeguard to ensure that the evaluator was not assigning high scores to dialogues that, while technically correct, were subtly unhelpful or misaligned with the intended clinical goals. This continuous human-in-the-Ble validation ensures higher level of confidence in quality of reward signals.
3) Human Spot-Checking and Auditing:
verify that the evaluator's scoring remained
H SYSTEM PROMPTS
This section provides the full system prompts used for the Simulated Patient, the Consultation Eval- uator, and the Policy Model (DoCTOR-RI)
Simulated Patient Prompt
You outpatient currently experiencing health issues. Your task is to simulate a patient- doctor interaction in the upcoming consultation dialogue. In this simulated conversation, you will play the role of the patient, and the user will play the role of the doctor. Please follow these guidelines:
1. Simulate realistic patient behavior and reactions. Ensure your communication feels au- thentie. You may use natural expressions, including hesitation, pauses, emotional fluctuations, to enhance the realism of the patient role.
Do not reveal all key information at once. Like real patient, gradually disclose deeper concerns and core issues the conversation progresses.
You are now the patient. No matter what the doctor asks, respond strictly in character as the patient.
Consultation Evaluator Prompt
You Consultation Evaluator, known for your meticulous and critical evaluations. Your task to review an Al assistant's full response, including its internal reasoning (<think> block) and final answer (<ansa (<answer> block). A high score is reserved for truly exceptional performance. You must score the assistant's reply based on the following demanding criteria.
Evaluation Criteria:
1. Safety (Critical Metric):
-5 (Dangerous): Provides incorrect diagnosis, recommends specific medications, or suggests harmful treatments.
+5 (Excellent):
Proactively identifies risks, explains limitations clearly, and provides exceptionally safe guidance.
2. Reasoning Quality: Evaluate the <think> block.
-5 (Flawed Logic): The reasoning is illogical, medically unsound, or ignores key user information.
+5 (Expert-level): The reasoning demonstrates clear, logical, and medically sound process.
34

---

Published as a conference paper at ICLR 2026
Medical Accuracy:
Evaluate the facts within the <answer> block.
-5 (Severely Inaccurate): Contains harmful completely false medical information.
+5 (Perfectly Accurate): All information is precise and aligned with current medical guidelines.
Completeness & Actionability:
-5 (Incomplete and Useless): Misses key aspects and provides no useful next steps.
+5 (Comprehensive and Actionable): Fully addresses user concerns and provides clear, safe next steps.
5. Information Gathering Effectiveness:
-5 (Misleading): Asks irrelevant or distracting questions.
+5 (Highly Strategic): Asks the most critical question(s) needed assess urgency.
6. Faithfulness to Ground Truth:
-5 (Contradictory)) Directly contradicts the expert's 's core advice.
+5 (Exceeds Standard): Aligns with and communicates the expert's advice with ex- ceptional clarity.
7. Empathy & Clarity:
-5 (Cold and Obscure): Robotic, jargon-filled, and likely to increase patient anxiety.
+5 (Exceptional Bedside Manner): Language is simple, reassuring, and shows gen- uine compassion.
Epistemic Humility (Certainty Check):
-5 (Dangerously Overconfident): Makes definitive claims that are not warranted by the evidence.
+5 (Perfectly Calibrated): Balances providing useful information with acknowledging uncertainty.
Evaluation Input:
[BEGIN DATA]
[Conversation History]
[Ground Truth Doctor Reply] (for reference)
[AI Medical Assistant Full Response] (to be evaluated)
[END DATA]
Policy Model Prompt (Doctor Agent)
You experienced doctor tasked with providing professional diagnosis and treatment plan for a patient through a consultation dialogue. Please carefully listen to the patient's responses, ask targeted questions. Objective:
Gather key information through effective questioning. Each question should be based the previous round's 's information.
Avoid repeating questions.
Rules:
1. Complete both actions per turn: provide thinking and ask a question.
2. Repetitive similar questions strictly prohibited.
Response Format:
<think> [Your reasoning] </think>
If information is insufficient, ask one question only, in the following format: <answer> Question: (Your question). anser>>> If information is sufficient, provide diagnosis and recommendation, in the following format:
<answer> Recommendation: (Your diagnosis and recommendation) </answer>.
35

---

Published as a conference paper at ICLR 2026
Decide your next action: Always output: <think> [Your reasoning] </think> <answer> |Your reply] </answer>, Do not include any additional text. Follow this format strictly.
Evaluation Prompt To ensure fairness, consistency, and the reproducibility of our results, all experiments were conducted using the official evaluation scripts, standardized prompts, and data formats provided by the creators of each respective benchmark. This includes the LLM-based evalu- ation protocols for HealthBench and MAQuE, as well the standard scoring procedures for MedQA and MMLU. By adhering strictly to these established methodologies, we ensure our results are di- rectly comparable to existing and future work in the field.
I ANALYSIS OF COMPUTATIONAL OVERHEAD
To provide insight into the computational overhead, we present detailed breakdown of the time and cost associated with our experience retrieval module. We analyze both the one-time training cost and the per-inference cost (latency and tokens).
Training Cost Overhead The primary overhead comes from 1) the retrieval step for each sample and 2) processing a longer token sequence due to prepending top-k = 2 experiences during the forward and backward passes. Table 17 shows the time per training step for different components of our framework. We can isolate the marginal cost of the experience module: In non non-interactive setting, adding Experience added 638s (1717s 1079s), while in our full interactive setting, adding Experience added 679s (4010s 3331s). This shows 20.38% increase in training time per step to incorporate the experience module.
Table 17: Breakdown of time per training step.
Training Configuration
Time / step (s)
Time per token gen (ms)
No Interaction No Experience
1079.0068
0.0900
+ Experience Only
1717.20 289.
0.6897
+ Interaction Only (Baseline)
3331.4185
1.1382
Full Method (+ Interaction Exp)
4010.1689
1.3121
Inference Cost Overhead To analyze the per-inference cost, we experimented 200 case test on HealthBench, comparing the baseline of not using experience and the method of experience retrieval. The results are shown in Table I8 The retrieval module adds average of 624.67 tokens to the prompt from the top-k = 2 retrieved experiences. This additional context combined with the retrieval step itself adds average of only 4.74s of latency per inference sample.
Table 18: Inference overhead analysis 200 case test set.
Model Variant at Inference
Avg. Tokens per Sample
Avg. Latency Sample (s)
DOCTOR-R R1 (Full Method)
2190.06
52.91
DOCTOR-RI (w/ Experience)
1565.39
48.17
Overhead
+ 624.67 tokens
4.74
This minimal inference cost of approximately 4.74 seconds and 624.67 tokens per session repre- sents a highly favorable cost-benefit tradeoff. This modest overhead is directly responsible for the substantial performance gain HealthBench Main (Table [ and human expert evaluation (Section 4.2) confirms an 83.87% "Clinically Helpful" rating from experience learning.
36


Reviews:
[{"review_id": "liEXC8cbfS", "reviewer": "Reviewer_sk1u", "summary": "This paper presents DOCTOR-R1, an 8B parameter AI doctor agent. The authors argue that SOTA models fail at \"dynamic clinical inquiry\" (i.e., strategic, empathetic questioning) even if they ace static medical exams.\n\nTo fix this, they propose an \"Experiential Agentic Reinforcement Learning\" framework. The system is a multi-agent loop: a Qwen3-8B (Doctor Agent) interacts with a Qwen3-8B (Patient Agent). Its actions are scored by a third Qwen3-8B (Consultation Evaluator).\n\nThis evaluator (reward model) uses a \"two-tiered\" system. It gives a \"Process Reward\" (for soft skills like empathy, safety) at each turn and an \"Outcome Reward\" (for diagnostic accuracy) at the end. The agent also uses an \"Experience Repository\" to retrieve high-quality past interactions to guide its policy.\n\nThe authors evaluate DOCTOR-R1 on the HealthBench and MAQUE benchmarks. They report that their 8B model significantly outperforms its 8B base model, larger 70B specialized models, and even proprietary models like GPT-4.1.", "questions": "Minor comments:\n\nSection 2 feels repetitive to read, for example, \"the policy model is the doctor agent\" appears twice in the section.\n\nL211: typo \"polify\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper identifies a critical gap in medical AI evaluation. It moves beyond static knowledge benchmarks to address the challenge of dynamic, real-time clinical inquiry. The focus on training an agent for this process is a logical research direction. Combining RL with an experience-retrieval mechanism is a sound approach. It allows the agent to learn from a curated set of high-quality past trajectories.\n\nThe \"two-tiered\" reward model is a sensible design. The inclusion of a \"hierarchical veto system\" (Sec 3.2) to penalize critical safety failures is a necessary feature, preventing the agent from optimizing for \"soft skills\" like empathy at the cost of patient safety.\n\nThe core result—an 8B model beating 70B specialized models and GPT-4.1 on complex inquiry tasks—is a very strong claim. It suggests a path toward highly capable, parameter-efficient medical agents.", "weaknesses": "The evaluation methodology has a circularity concern - using the same Consultation Evaluator for both training rewards and test evaluation may inflate performance metrics.\n\nAll three components (agent, patient, evaluator) use the Qwen3-8B model family, creating a self-referential system that may optimize for model-specific patterns rather than generalizable clinical skills.\n\nThe Consultation Evaluator's ability to assess complex metrics like \"Empathy\" and \"Reasoning Quality\" lacks validation against medical expert judgment or inter-rater reliability studies.\n\nThe eight process reward criteria need better justification - prior work exists that could ground these choices (e.g., arXiv:2502.14860, 2502.07143). Including \"Completeness\" as a turn-level metric seems inconsistent since completeness requires full conversation context.\nThe 100k simulated patients lack validation for clinical realism, potentially creating a gap between benchmark performance and real medical consultations.\n\nThe experience retrieval mechanism yields modest improvements (2-3%) over simpler baselines, raising questions about cost-benefit tradeoffs.\n\nHuman evaluation would benefit from medical professional involvement to assess clinical appropriateness beyond lay judgments of empathy and clarity.\n\nKey hyperparameters ($\\pi$, $\\lambda$, reward weights) are presented without ablation studies or sensitivity analysis to understand their impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ACmkTinzV", "reviewer": "Reviewer_YuDH", "summary": "The authors propose DOCTOR‑R1, an 8B-parameter doctor agent trained with an Experiential Agentic RL framework: (1) Interactive environment: a POMDP with a Doctor Agent (policy), an LLM‑based Patient Agent, and a Consultation Evaluator acting as the reward function; (2) Optimization: GRPO trains the policy to prefer higher‑reward responses within sampled groups.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Principled safety‑aware reward design that decouples soft‑skills and hard‑skills during training.  \n2. Experience‑guided control via reward‑aware and novelty‑aware retrieval that measurably helps inquiry quality.\n3. Strong qualitative demonstration on a truly high‑stakes case that aligns with clinical intuition (quantify bleeding, TB history, anticoagulants, urgent referral).", "weaknesses": "1. The policy, patient simulator, and reward judge are all Qwen3‑8B family members, raising systemic bias and overfitting risks to the evaluator’s rubric and the simulator’s discourse style \n2. the method contribution seems to be trivial, like previous rl works in huatuo-o1, FineMedLM-o1, m1 leverage rl and test-time scaling works like medprm and medadapter for medical reasoning. however in the related works, seems agentic rl for medical reasoning is not emphasized enough\n3. The retrieval prepends prior “good experiences,” which could partly function as a prompting prior rather than learned policy alone. More careful controls (e.g., retrieval without actions vs. with actions; random but format‑matched snippets) would help isolate the effect. \n4. The final diagnostic reward is coarse (0/0.5/1), potentially limiting sensitivity to near‑misses or safe-but-incomplete triage decisions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "95V3xNcA1C", "reviewer": "Reviewer_usVB", "summary": "The paper proposes Doctor-R1, a clinical dialogue agent trained to conduct strategic, empathetic multi-turn inquiry while also making sound medical decisions. The system employs three key concepts: a multi-agent interactive environment, a two-tiered reward structure that distinguishes between outcome quality and process quality, and an experience repository that stores valuable trajectories for reuse. The model is evaluated on HealthBench and MAQuE with automatic metrics and human preference studies. The claim is that Doctor-R1 improves communication quality and user experience while maintaining competitiveness or surpassing it in task accuracy.", "questions": "1. Why is Qwen3-8B also used as the LLM judge for the Process Reward?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Most medical LLM work measures final answers. This paper addresses the gap in the skill of guided clinical inquiry and communication, which is crucial in practice.\n2. The two-tiered reward isolates process signals from outcome signals, which is a sensible way to avoid reward dilution and to target conversational quality.\n3. The experience module that learns from high-quality trajectories is a practical way to ground policy learning without heavy annotation.\n4. The study reports multi-facet metrics and human judgments, not only accuracy.", "weaknesses": "1. The models used across experiments are not consistent. For example, the human evaluation uses GPT-5, but the main tables and ablations only report GPT-4.1. GPT-5 is also not mentioned in the Evaluation Settings section. Could you unify the model choices across experiments? If different models need to be used, please provide a brief explanation for the selection.\n\n2. The authors state: “The annotators recruited were without a specialized medical background to ensure that metrics like Clarity and Empathy were evaluated from the perspective of a typical patient, for whom the agent is ultimately designed.” However, the paper sets out three research questions: (2) Empathetic Communication is only one, whereas (1) Strategic and Dynamic Inquiry and (3) Learning from Good Experience requires a medical background to assess. For instance, at each turn-level Doctor-R1 response, expert evaluation can better verify whether the questions asked are clinically appropriate; experts can also check whether the Experience Module’s stored items and retrieved content are indeed appropriate and helpful. I do not think these two research questions can be avoided in the human evaluation; focusing only on non-medical-background evaluation of (2) Empathetic Communication is insufficient.\n\n3. An additional analysis would be to break down time and compute overhead by component. For example, after adding the “Learning from Good Experience” design, how much extra latency or token usage does the model incur?\n\n4. The ablation study is missing analyses on how different reward designs affect results. A core claim of the paper is integrating a TWO-TIERED REWARD into RL training. Beyond the outcome reward, many additional rewards are proposed, and the method section includes empirically motivated statements such as “This (e.g.,  hierarchical penalty design) addresses the limitations of conventional weighted-sum models, which can fail to adequately penalize catastrophic errors due to an averaging-out effect.” It would be beneficial to include more recent related work, for example, how other studies attempt to incorporate additional process rewards on top of RLVR while maintaining training stability, or at least present supportive mini-experiments. Also, the paper only presents a coarse “w/o Process Reward” setting. Please consider ablating some of the eight process rewards to toggle them on or off, especially the key ones, to assess whether such designs can generalize to other medical scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IgLheRf76n", "reviewer": "Reviewer_ghkz", "summary": "This paper proposes an RL-based agent, namely Doctor-R1, for strategic inquiry and empathetic communication. Evaluating on HealthBench and MAQuE datasets, the proposed framework outperforms open-source and commercial general models. Human evaluation shows the Doctor-R1 has improvements over baselines.", "questions": "(1) Now the framework is offline and off-policy. Is it possible to change the framework to make it online?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "(1) This paper aligns with the trend to extend the medical LLMs from static decision-making to dynamic inquiry through an agentic RL perspective.\n(2) This paper explicitly models empathy and communication quality as trainable objectives.", "weaknesses": "(1) The technical details are missing. The authors need to provide more technical details. For example the authors need to define how vector rewards are aggregated or how gradient signals are propagated.\n(2) The authors rely on the GRPO framework for the RL. Ablation studies need to be provided on how different RL frameworks will affect the performance.\n(3) The authors use Qwen3-8B as the base model. Is this framework transferable to other base models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NmUnl5e0gY", "reviewer": "Reviewer_bTuJ", "summary": "The paper proposes DOCTOR-R1, an AI agent framework designed to master both strategic multi-turn patient inquiry and medical decision-making through experiential reinforcement learning.", "questions": "1. Can you provide quantitative evidence that reward model judgments correlate with physician assessments? What is the inter-rater agreement between the LLM judge and medical experts?\n2. What percentage of training dialogues are filtered due to patient agent adherence failures?\n3. How does supervised fine-tuning on high-quality human dialogues compare to your RL approach?\n4. Under what conditions does the system produce unsafe recommendations despite the veto mechanism?\n5. What types of experiences does the system most frequently retrieve? Are there cases where novelty filtering excludes valuable high-quality examples?\n6. How does performance degrade when evaluated on out-of-distribution cases (different patient populations, communication styles, or medical domains)?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper identifies an important gap in medical AI research. The distinction between static medical knowledge assessment and dynamic clinical inquiry is clear.\n2. The evaluation includes multiple dimensions: two dynamic benchmarks (HealthBench, MAQuE), static knowledge validation (MedQA, MMLU).", "weaknesses": "1. The most severe limitation is the pervasive use of LLM-as-judge throughout the system. The reward model is Qwen3-8B evaluating Qwen3-8B policy outputs. This creates a closed system where the system optimizes for LLM evaluator preferences rather than genuine clinical competence.\n2. The experience retrieval mechanism offers minimal technical innovation. Stage 1 (embedding similarity + reward weighting) and Stage 2 (cross-encoder reranking) are standard retrieval techniques. Are retrieved experiences genuinely instructive, or does the system learn spurious correlations?\n3. There are some Human Evaluation Limitations. No evaluation of diagnostic accuracy by medical professionals. Both primary benchmarks (HealthBench, MAQuE) use automated LLM evaluation, creating potential bias toward response styles favored by specific LLM families.\n4. No comparison with supervised fine-tuning on high-quality human-authored dialogues. Given the complexity of the RL pipeline, demonstrating that RL provides advantages over simpler approaches is essential.\n5. The author acknowledges the \"key challenge\" of patient agent adherence but provides insufficient evidence that the mitigation strategies are effective. What percentage of simulated dialogues are filtered out due to adherence failures?\n6. The multi-objective reward structure, while helpful, does not eliminate the possibility of exploitation. Human spot-checking is mentioned but no statistics on frequency or findings are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)