Towards Doctor-Like Reasoning: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradients
Authors: Yuxing Lu
Gecheng Fu
Wei Wu
Xukai Zhao
Goi Sin Yee
Jinzhuo Wang
Venue: NeurIPS 2025 poster ()
Keywords: Retrieval-Augmented Generation
Large Language Model
Clinical Decision Support System
Primary area: machine_learning_for_sciences
TL;DR: This paper introduces DoctorRAG, a medical AI that mimics doctor-like reasoning by combining textbook knowledge with insights from similar patient cases, using "textual gradients" to refine its answers for improved accuracy and relevance.

Abstract: Existing medical RAG systems mainly leverage knowledge from medical knowledge bases, neglecting the crucial role of experiential knowledge derived from similar patient cases - a key component of human clinical reasoning. To bridge this gap, we propose DoctorRAG, a RAG framework that emulates doctor-like reasoning by integrating both explicit clinical knowledge and implicit case-based experience. DoctorRAG enhances retrieval precision by first allocating conceptual tags for queries and knowledge sources, together with a hybrid retrieval mechanism from both relevant knowledge and patient. In addition, a Med-TextGrad module using multi-agent textual gradients is integrated to ensure that the final output adheres to the retrieved knowledge and patient query. Comprehensive experiments on multilingual, multitask datasets demonstrate that DoctorRAG significantly outperforms strong baseline RAG models and gains improvements from iterative refinements. Our approach generates more accurate, relevant, and comprehensive responses, taking a step towards more doctor-like medical reasoning systems.

Towards Doctor-Like Reasoning: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradients
Yuxing Lu Gecheng Fu Wei Wu', Xukai Zhao*, Goi Sin Jinzhuo Wang
Department of Big Data and Biomedical AI, Peking University, Beijing. China
2 Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, USA
Department of Biomedical Engineering, Peking University, Beijing, China
School of Architecture, Tsinghua University, Beijing, China School Life Sciences, Peking University, Beijing, China
Equal contribution
Corresponding author: wwtejjinnhoupuueeco edu.
Abstract
Existing medical RAG systems mainly leverage knowledge from medical knowl- edge bases, neglecting the crucial role of experiential knowledge derived from similar patient cases a key component of human clinical reasoning. To bridge this gap, propose DoctorRAG. RAG framework that emulates doctor-like reasoning by integrating both explicit clinical knowledge and implicit case-based experience. DoctorRAG enhances retrieval precision by first allocating conceptual tags for queries and knowledge sources, together with a hybrid retrieval mechanism from both relevant knowledge and patient. In addition, Med- TextGrad module using multi-agent textual gradients is integrated to ensure that the final output adheres the retrieved knowledge and patient query. Comprehensive experiments on multilingual, multitask datasets demonstrate that DoctorR significantly out- performs strong baseline RAG models and gains improvements from iterative refinements. Our approach generates more accurate, relevant, and comprehensive responses, taking step towards more doctor-like medical reasoning systems.
Introduction
Large Language Models (LLMs) hold considerable promise for transforming healtheare, offer- ing capabilities to assist medical professionals, enhance patient care, and accelerate biomedical research [1]. Within this landscape, Retrieval-Augmented Generation (RAG) has emerged as a pivotal technique, grounding LLM outputs in factual knowledge to mitigate hallucinations and bolster reliability qualities paramount in the high-stakes medical domain [30] Existing medical RAG systems commonly incorporate extensive external knowledge bases containing clinical guidelines, evidence-basee medicine, and research literature [46, [2). This approach, while valuable, over- looks critical dimension of expert medical reasoning: the application of experience gained from encountering numerous patient cases over time.
Human clinicians routinely integrate formal medical knowledge (Expertise) with experiential insights gleaned from similar past situations (Experience, i.e., case-based reasoning) iteratively to make diagnoses, formulate treatment plans, and answer patient queries [00] Existing medical RAG frameworks fail to capture the information from similar patients, limiting their ability to truly emulate doctor-like reasoning (Figure []. This deficiency can hinder their overall performance in complex, context-dependent tasks such differential diagnosis the formulation of personalized treatment recommendations. Furthermore, standard retrieval methods (e.g., dense vector search) often struggle to capture the fine- -raiigr semantic distinctions crucial for medical relevance [33]. Compounding this, the outputs of generative models require rigorous validation to ensure they are not only relevant
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Knowledge Base
Medical Knowledge Base
Medical Knowledge Base
expertise
expertise
Patient Base
experience
Query
Query
Query
Output
Output
Output
General RAG
Medical RAG
DoctorRAG
General and Medical RAGs retrieve solely from knowledge bases, whereas DoctorRAG enhance responses by integrating both knowledge-based expertise and case-based experience, mirroring clinical practice.
Figure 1:
but also faithfully aligned with the retrieved medical context, challenge that necessitates dedicated mechanism for iterative refinement and grounding (18).
To address these limitations, we propose DoctorRAG, a novel RAG framework designed to bridge the gap between current systems and human clinical practices. The core innovation of DoctorRAG lies in its ability to emulate doctor-lit like reasoning by integrating both explicit clinical knowledge from KBs with implicit experiential knowledge derived from analogous patient cases. To effectively retrieve proper information from these heterogeneous sources, DoctorRAG first decomposes complex queries and knowledge chunks into declarative statements and conceptual tags. This structured representation enables more precise hybrid retrieval mechanism that combines semantic vector similarity with targeted conceptual matching. The generated answer is then refined and validated by Bodd-exxtrad, a multi-agent textual gradient process that iteratively ensures the output adheres to the retrieved context and patient query, effectively grounding the response and ensuring the system's 's accuracy and reliability. Comprehensive experiments conducted on diverse multilingual and multitask disease diagnosis, question answering, treatment recommendation, and text generation demonstrate that DoctorRAG significantly outperforms strong RAG baselines in terms of accuracy, relevance, and faithfulness. Pairwise comparison between answers of different iteration further demonstrate Med-Text paves the way for medical Al systems capable of more comprehensive, reliable, and clinically-infommet reasoning.
2 Methodology
We propose the DoctorRAG framework for clinical decision support, designed to integrate domain- specific medical knowledge with patient-specific clinical data, and refine the generated response through a multi-agent, iterative optimization process. This framework, illustrated in Figure 2 lever- ages concept matching and vectorized embeddings for retrieval, and Med-TextGrad optimization strategy to iteratively refine generated answers through back propagation. The pipeline consists of retrieval and aggregation, and (2) Iterative answer optimization using multi-agent textual gradients.
two primary stages: (1)
2.1 ExpertisseExperieee retrieval and aggregation
The retrieval and aggregation module (prompts in D.I examples in E. operates two primary data sources: a Knowledge Base (IC) and a Patient Base (P).
The Knowledge Base is constructed from several medical knowledge sources. These sources are first segmented into textual chunks. Each chunk is then transformed into a declarative sentence (details in B.3), denoted d4, using Declarative Statement Agent Dec Subsequently, each sentence d, is annotated with a set of medical concept identifiers The function Tag, represents a Query Tagging Agent, and C is predefined controlled vocabulary of medical concepts (c.g., ICD-10 10 codes). The Knowledge Base is thus defined as the set of pairs
= T (d+), where C C. (details in B.4).
K [ c.) i 1,..., Nx)) where Nx is the total number of processed declarative sentences. Each sentence d, is encoded into a D D-dimensional dense vector representation vd, = Eemb E IR! Here, Eamb denotes the embedding function of an LLM.
The Patient Base comprises de-identified patient records, formally defined as P ((p) 8, az) j = 1,..., Nr), where Np is the count of patient records. For each record j. Py represents the patient's chief complaint or clinical conversation, sy denotes structured clinical data (c.g., diagnoses,

---

DoctorRAG
Med- TextGrad
Embeddingg
Patient Optimizer
Inciade sos pet ent informsiion prompt
This answer should focus more on portentss xox
Medical Knowledge Base
Declarative Sentences
Compute Similarity
Losa Prompt
The emppasiie
Anower
Patient Criterion
Concept Maching
Embaddings
Evellate If the arRAAr is related
Iterative Update
(ootient)
Query Tag
Query Vector
Generated Answer
G DD
TG
Patient Query
Top-k
Erailvate with pprowlogge)
Embeddinng
Iterative
Diagnosis
Update
6
Context Criterion
Loes Prompt
Loss Areniee
Treatment
Lfflf
emiihasiznn.
Emboddiing
krowlddge the answer
Ensure The prorpt
Patient
Clinical
Compute Similarity
edge
Base
Context Optimizer
Figure 2: Overview of the proposed DoctorRAG & Med-TextGrad framework. Medical knowl-
edge is transformed into declarative statements and tagged to match the patient's query. Dual-database retrieval generates a response incorporating both clinical expertise and experience. two-way multi- agent Mad optimization process refines the answer through an iterative computation graph involving textual gradient backpropagation.
treatments), and encompasses auxiliary metadata (e.g., demographics). The primary textual component is encoded into a vector Up, = Femb (P₂) E RD using the identical embedding function Egmb The remaining data 8j and a, are preserved for contextual augmentation.
Given a user query q, it is processed through two concurrent pathways. Firstly, the Query Tagging Agent Tagg, annotates the query with set of relevant medical concept identifiers G ( where Gy C C. Secondly, the query q embedded into the same D-dimensional latent space, yielding
"y Eemb (q) E R.D. Retrieval from both databases proceeds as follows:
Knowledge Retrieval: To retrieve relevant declarative sentences from K,, a concept-constrained semantic similarity score sx(g,, d4) is computed between the query q and each sentence d, (associated with concepts c:):
vily Ill if na + #0, otherwise,
sx(g, de)
[|,
-00
where the fraction denotes the cosine similarity. This formulation prioritizes semantic relevance for sentences sharing common concept identifiers with the query. The le knowledge entries (d,, c) achieving the highest non-negative similarity scores constitute the set Kop--k
Patient Retrieval: Concurrently, to retrieve relevant patient records from P, the cosine similarity 8p sp(g.P) Py) between by and the embedding of each patient's textual data Up is computed:
b/g Up
sp(. p;)
|/
The & patient records (pj, Sj, az) yielding the highest similarity scores constitute the set P Peo
Finally, the retrieved information is aggregated to construct unified context for the answer generation module. This context is formed by concatenating the textual content from the top-k declarative sentences and formatted textual representations of the top-k patient records:
= Concat
U
d,
U
Format(p,, 83. 53)
(3)
(dd,ct)EKmpa
(Ps.33 ay P

---

where Concat(::-) joins all textual elements, and Format(-) converts patient record tuple into structured textual representation. This context C' concatenates both general medical knowledge (expertise) and patient-derived clinical evidence (experience), serving as input for generation.
2.2 Iterative answer optimization with multi-agent Med-TextGraa
To generate and refine answers, we propose Med-TextGrad, a multi-agent optimization framework adapted for the medical domain (Algorithm in B.2 prompts in D.2 examples in [.2). This framework leverages principles of TextGrad 43 for textual gradient-based optimization. The system architecture comprises Generator, Context Criterion, a Patient Criterion, and conceptually distinct Context Optimizer and Patient Optimizer roles guiding iterative refinement. The process uses computation graph where textual gradients update the prompt and, consequently, the answer.
The fundamental goal of the Med-TextGras framework is to identify an optimal prompt Pop that, given a query q and its retrieved initial context C, generates the best possible answer A = 'Best' is defined in terms of minimizing textual loss function L, which incorporates critiques regarding medical context consistency and patient-specific relevance. This objective is analogous to finding the optimal parameters 0" for neural network by minimizing loss function. Formally, the target optimal prompt Pop is defined as:
Pop (q, C) arg min C(A) C) s.t. PEP, A= Gen (P)
p
where P represents the space of possible prompt space. The textual loss C(A q. C') is associated with the answer generated from prompt P; it is explicitly defined in Equation 8 as the sum critiques from different criteria. The Med-Tee ded-Trad iterative procedure, starting from initial prompt and applying textual gradient descent steps (Equation 14), is designed to find a refined prompt Pr that serves approximation Ppp-
2.2.1
Computation graph with gradient backpropagation
The answer generation and iterative refinement process is modeled directed computation graph. The forward pass generates an answer from a prompt and evaluates against specified criteria, yielding textual critiques that serve a proxy for loss. The backward pass computes and propagates textual gradients from these critiques to guide prompt refinement. This is conceptualized as:
Prompt
Genorator
Answer
Criterion, Critiques (Loss Proxy)
(5)
Prompt
Answer
V Naver
Critiques (Loss Proxy)
(6)
Here, V Answer and V Prompe represent the textual gradient computation steps. The input prompt P; iteration is formulated as the concatenation of the initial query and the aggregated context C. An answer A, is produced by the Generator, and is assessed by the Context Criterion and Patient
Criterion to produce textual critiques. The Generator, denoted Geng, generates the answer A based on the current prompt P.:
A = =Gen (P.) = Gen,.(Concati C)).
The Context Criterion (KC.)) and Patient Criterion (PC.g) evaluate A,- KC₂, assesses factual alignment and consistency of A, with context C. PC evaluates relevance, appropriateness, and patient-speciffcity of A, query q. The aggregated textual critique, representing the overall "loss", is given directly by L(A():
[(Ac) = L(A| q. C) Cke(Ai) (rc(A.) = KC C) + (A.
(8)
Ideally, this loss can also inform an early stopping strategy in Med-TextGrad
2.2.2 Textual gradient computation and backpropagation
Gradients of the critiques with respect the answer A, are computed independently. These gradients, (PC(Ac) provide instructive feedback from each perspective. They are generated by a
SKC(At) and A.,,
OA,
LLM Ggrad_A prompted by & specific to each criterion:
aKC(A.)
Ggraa (Ae, C, KC(Ar);
(9)
AA

---

(At) AA,
aP, The Context Optimizer and Patient Optimizer roles leverage these respective gradients -ggnd__ (Aee PC(ee))) Agraa_A_PcC)
(10)
Here, Ggrad_A takes the current answer, the relevant content (either context or patient query q), and the critique itself to produce textual gradient aimed improving the answer.
These answer-level textual gradients are natural language descriptions suggesting specific improve- ments for A, from the viewpoint of respective criterion. Subsequently, they are utilized to compute
distinct textual gradients with respect to the prompt Pt. These prompt-directed gradients, aR: (PC(Ae)
@KC(As) and
31% articulate how P. should be modified elicit improved A, that addresses the critiques from each specific criterion. These are generated by LLM Ggnd_ prompted by "'grnd
aKC(A.)
OKC(A,)
OP: Ggrad_
Pt, Ar, 0.,
W.'radd P_KC
(11)
OPC(A.) Ggrad_
aPC(A.)
Pr. 0A. Pgrad_PPCC
(12)
OP.
In these equations, Ggrdd P uses the current prompt, the generated answer, and the corresponding answer-level gradient to compute the prompt-level textual gradient.
The computed prompt gradients, OKC(A.) and OPC(A.) aP;
8P, provide pathway-specific textual feedback for refining the prompt Pt. Conceptually, an overall textual gradient @C(A+) for the combined critique 8P, with respect to the prompt Pt arises from the application of the chain rule. This can expressed as:
AL(A.) aP.
(aKC(A.) (PC(A.))
Veen Pts, (r,, Aggregate
(13)
0A,,
0A,
Here, Aggregate(---) denotes the textual combination of the individual answer-level gradients, and Ven is the TextGrad operator that performs the backpropagation through the Generator Geng. This overall gradient illustrates how combined feedback theoretically propagates to the prompt.
2.2.3 Iterative optimization with context and patient optimizers
The iterative refinement of the prompt Pt is guided by the distinct textual gradients computed in the previous step: the context-focused prompt gradient OKC(A.) and the patient- prompt gradient SPC(A.)
aP,
to ensure the prompt evolves in a manner that addresses both alignment with medical context and relevance to the specific patient.
This is achieved through Textual Gradient Descent (TGD) step, where the current prompt P3 is updated. This update is performed by another LLM GTGD, which synthesizes the improvement suggestions from both prompt gradients to produce the refined prompt Pt41:
P++1 TGD.step ( Pr,
ac(A))
OP: )
TGD.step (Pr, aKC(A.)
OPC(A.)
(14)
OP.
OP,
The updated prompt Pt+1 is then fed back to the generator to produce a new answer A
A411 Gen..(Pett)
(15)
This iterative cycle continues for a predetermined number of iterations T (3 in our implementation), or until a suitable convergence criterion (c.g., minimal changes in critiques or the answer) is met. The final output of
is the refined answer Ar-
Experimental Setup
3.1 Datasets
To comprehensively evaluate DoctorRAG, we utilized a diverse collection of datasets spanning multiple languages and covering key medical tasks: disease diagnosis (DD), question answering (QA), treatment recommendation (TR), and text generation (TG). Further details on each dataset are provided Appendix [.TT An overview of the datasets, including their respective languages, medical fields, and sample counts for each task, is presented in Table
5

---

Dataset
Domain
DD
QA
TR
TG
DialMed
General
11,996 2,340
11,996
Chinese
RJUA MuZhi 38
Urology
2,340
2,340
Pediatrics
527
DDXPlus NEJM-QA
14
General
1,292.579
English
Nephrology
655
COD
6
General
39,149
39,149
French
DDXPlus
General
1,292,579
Table I: Datasets for evaluation. DD: Disease Diagnosis, QA: Question Answering, TR: Treatment Recommendation, TG: Text Generation. Numerical values indicate sample counts.
For Chinese-language tasks, incorporated three datasets. DialMed [17]. general medicine dataset featuring structured dialogues, is employed for disease diagnosis and drug recommendation tasks. RJUA 22 focuses urology and is used assess DoctorRAG's capabilities in disease diagnosis, treatment recommendation, and the generation of informative responses. Muzhi [38] centered on pediatric care, comprises dialogues with symptom descriptions and diagnostic recommendations.
For English-language evaluations, we employed DDXPlus [1]. which is large-scale general medicine dataset that includes socio-demographic data, multi-format symptoms/anteccedent, and differential diagnoses. The NEJM-QA [5], specializing in nephrology, provides 656 multiple-choice questions to evaluate question answering abilities. Additionally, COD [6], a general medicine dataset, used to assess performance disease diagnosis and medical text generation.
We included the French version of DDXPlus to assess DoctorRAG's multilingual performance.
We constructed the external knowledge base for DoctorRAG using MedQA [20] datasets. MedQA encompasses bilingual Chinese-Englishe medical ceesstiionanswering pairs and medical textbooks, providing a foundation for cross- pooss-lingual knowledge retrieval. We also translated a subset of MedQA to French for the French version of DDXPlus [4]. Each knowledge entry is pre-segmented, transformed into declarative statements using the DeepSeek-V3 model [24], and annotated with corresponding concept labels to optimize DoctorRAG's medical knowledge retrieval capabilities.
For each dataset, we established a clear partitioning: approximately 80% of the patient records from each dataset were allocated for constructing DoctorRAG's patient base, while the remaining were held out as distinct evaluation set. To ensure unbiased assessment and prevent data leakage, we guarantee that for any given sample within this evaluation set, its corresponding full patient record and extreme similar ones (similarity 0.99) were strictly excluded from DoctorRAG's patient base during evaluation of that specific sample. All samples are treated as independent clinical encounters.
3.2 LLM backbones
We employed selection of advanced LLM backbones for comprehensive and balanced comparison. The primary aims were to assess the multilingual and multitask capabilities of these foundational models and to evaluate the robustness of DoctorRAG when integrated with different LLMs. The models selected include DeepSeek-V31244 Qwen-3 (2), GLM-4-Plus [16], and GPT-4.1 (Л] To ensure consistency across all experimental procedures, DeepSeek- V3 specifically assigned to data preprocessing tasks, such as declarative sentence transformations and concept labeling. Furthermore, all embedding conversions were performed using OpenAl's cext-embedding-33 --lagee model.
4 Results
In this section, present the experimental results address the following research questions (RQs):
RQ.1
4.1
How does DoctorRAG's performance compare to other RAG methods across tasks?
RQ.2
4.2
Does integrating patient case data improve DoctorRAG's relevance and utility?
RQ.3
4.3
To what extent does Med-Tex TextGextrrr iteratively refine answer quality?
RQ.4
How do different backbone LLMs affect DoctorRAG's performance?
RQ.5 4.5
How does each component in DoctorRAG contribute its effectiveness?

---

DD (Ace)
QA (Ace)
TR (Ace)
TG
DialMed RJJUA MaZhi DDXPhi DDXPlus NEJM-QA
DialMed RIUA
RJUA (CN)
COD (EN)
Backbone
CN
CN
CN
EN
FR
EN
Rouge-L BERTScore Rouge-L BERTScore
Direct Generation
GLM-4-Plus
91.97
73.35
70.75
82.07
84.87
60.79
45.10
60.05
20.58
90.20
19.59
92.50
Qwen-3-32B
88.80
68. 18.16
71.70
84,13
84.80
60.32
53.98
70.75
19.95
91.29
16.28
87.76
DeepSeck-V3
91.32
80.19
72.64
87.60
85.16
68.69
58.33
73.58
21.38
92.41
18.18
92.26
GPT.4.1-min
91.39
73.11
72.64
67.17
19.09
Vanilla RAG
GLM-4-Plus
72.81
Qwen-3.32B
91.47
42
72.06
18
33
62.59
55.76
70.91
20.14
91.37
17.03
94.97 $4.92
DoepSeok-V3
90.78
83.12
73.46
87,83
85.74
69.58
63.14
73.79
22.53
92.44
18.76
95.41
GP1-4.1-mini
91.97
82.76
73.09
Proposition
[8]
GLM-4-Plus
92.31
85.
73.58
82.27
85.06
61.09
53.49
61.32
24.01
92.93
21.60
96.24
Qwen-3-32B
92.31
82.78
72.64
84,27
02
64.98
55.33
71.05
20.32
90.98
17.59
96.70
DeepSeek V3
92.11
86.32
74.53
88.06
86.49
70.89
60.71
74.06
23.71
92.48
19.35
96.62
GPT.4.1-min
92.59
84.92
73.58
90.20
87.93
70.82
57.49
73.41
23.98
92.96
20.28
96.50
Graph RAG
GLM-4-Plus
91.28
86..21
73.48
85,50
87.53
62.77
57.23
62.01
27.52
92.90
21.48
94.32
Qwen-3.32B
72.97
92.73
DoepSeck-V3
92.87
85.33
76.59
88.93
92.37
70.95
63.22
75.30
25 .7
93.01
19:47
95.88
GPT--.1-mmin
91.02
85.28
76.11
92.37
93.14
70.95
61.70
74.88
25.86
91.98
20.95
96.28
DoctorRAG
GLM-4-Plus
93.09
86,
74.53
98.27
98 53
62.92
56.40
63.83
31.98
93.97
21. 66
92.80
Owen-3-32B
94.49
27
DeenSeck V3
93.56
86.79
80.19
96.87
96.93
71.73
63.49
77.83
30.50
93.56
20.01
96.61
GPT-4.I-mioi
94.96
85.61
75.47
97.67
98.73
72.64
58.82
76.89
26.73
93.29
22.31
96.81
Table 2: Comparison of RAG methods and LLM backbones across medical tasks, with results highlighted as follows: pink for the overall top score per task, and green for the top score within each RAG method group for that task.
4.1 DoctorRAG vs other RAG (RQ.1)
We compared DoctorRAG with several strong RAG baselines, including Proposition RAG [8] and Graph RAG [13. We also compared it with Vanilla RAG [22] and direct generation using LLMs. For disease diagnosis (DD), question answering (QA), treatment recommendation (TR) tasks, used prediction accuracy for evaluation. For text generation (TG) tasks, we used Rouge-L (23),
BERTScore [55] BLEU (31 and METEOR 3 scores metries. The main results shown in
Table B.5
DoctorRAG consistently achieves leading scores, denoted by pink highlights, across diverse spectrum of tasks including disease diagnosis, question answering, treatment recommendation, and text generation. This dominance is often marked by substantial quantitative improvements; for instance, in DDXPlus (EN) diagnosis, DoctorRAG (GLM-4-Plus) reached 98.27% accuracy, significantly outperforming the strongest Graph RAG bascline (92.37%) and Proposition RAG (90.20%). Similarly, its Rouge-L. score of 31.98 on RJUA text generation far surpasses its peers, illustrating consistent and impactful enhancement across varied medical challenges. Crucially, the superior capabilities DoctorRAG are not language-boune demonstrating remarkable efficacy across datasets in Chinese, English, and French. This cross-lingual robustness is evidenced by its top-ranking accuracies on Chinese tasks like DialMed (94.96%), English datasets such as NEJM-QA (72.64%), and the French DDXPlus task (98.73%)
While DoctorRAG leads in most metrics, Vanilla RAG (when paired with GPT-4.1-minl secured the highest BERTScore RJUA (95.52) and COD (97.04). After investigation, DoctorRAG's outputs tend towards greater in length compared to the ground truth answers. While this comprehensiveness may benefit recall-oriented metrics like Rouge-I, it is factor for consideration in contexts prioritizing brevity and might influence certain evaluation scores.
note that
4.2 Utility of patient case information (RQ.2)
To evaluate the rationale and utility of integrating patient case information in DoctorRAG. we conducted visualization analysis (details in B.6). We hypothesized that leveraging this data, much like physician utilizes clinical experience, enhances diagnostic precision by capturing patterns from similar patient profiles alongside medical knowledge.

---

DialMed
DDXPlus
COD
RJUA-QA
Figure 3: UMAP visualizations of patient embeddings on four distinct medical datasets. Each point corresponds to a patient embedding, and different colors represent different diagnosed diseases.
We applied UMAP [29] visualize patient embeddings from multiple datasets (Appendix B. including DialMed, DDXPlus, COD and RJUA, as shown in Figure The visualizations clearly demonstrate that embeddings of patients diagnosed with the same disease form distinct clusters. This clustering mirrors clinical observations where patients with a shared condition often present comparable symptoms and medical complaints, thereby validating the rationality of incorporating patient case information. The tight grouping within these clusters also underscores DoctorRAG's ability effectively encode and utilize patient similarities, enhancing its contextual understanding.
Furthermore, a complementary ablation study (Table 3 quantitatively confirmed the contribution of retrieving similar patient cases, showing question answering accuracy improved from 70.21 to 71.73 on NEJM-QA. These findings affirm that integrating patient case information is both conceptually sound and practically beneficial, empowering DoctorRAG to deliver more accurate and clinically relevant diagnoses by leveraging these observed patterns.
4.3 Impact of Med-TextGrad's iterative refinement (RQ.3)
We evaluated generation quality on 50 COD dataset samples, comparing ground truth (GT), Doctor- RAG's initial answer (OA), and Med-TextGiin 's iterated answers (TI-T3) using DeepSeek-Vi V3 [24] vote on comprehensiveness, relevance, and safety, with verification from 2 human experts (details in B.7 prompts in D33] examples in EE3)). Figure 4 shows win counts, where cell (Y,X) in the lower-left indicates Y's preference over X. Interestingly, both DoctorRAG's initial answers (OA) and all Med-TextGrad iterations (TI-T3) consistently outperformed the ground truth (GT) answers in over 90% of overall comparisons. While GT represents real clinical interactions, these conversations are often context-specife Consequently, they may not always reflect the most comprehensively structured or optimally informative medical advice that a system like DoctorRAG.
(a) Comprehensiveness
(b) Relevance
(c) Safety
(d) Overall
4
4
0
GT
2
0
2
GT
1
5
GT
3
DA
46
24
10
18
QA
46
7
6
10
OA
43
15
9
17
QA
45
15
8
15
T1
46
26
18
20
T1
48
43
10
17
T1
46
35
23
17
T1
47
35
17
18
T2
50
40
32
26
T2
50
44
40
25
T2
49
41
17
18
T2
50
42
30
23
T3
48
32
30
24
T3
48
40
33
25
T3
45
33
33
32
T3
47
35
32
27
GF
QA
T1
T2
T3
GT
0A
11
T2
T3
GT
0A
T1
T2
13
GT
OA
T1
T2
T3
Figure 4: Pairwise comparison scores for ground truth (GT), original answer (OA), refined answers of iteration 1-3 (T1-T3) on (a) Comprehensiveness, (b) Relevance, (c) Safety, and (d) Overall. The lower-left triangle of each matrix represents the Y-axis outputs perform better than X-axis output.
Grd-TextGrad effectively navigates the "answer space" through iterative refinement, mirroring the optimization journey of gradient descent in deep learning. The initial answer (OA) serves as starting point, and each Med-Tex TextGrad iteration attempts to "descend" towards higher quality answer. As shown in Figure 4.. the first iteration (T1) provided a significant improvement, being preferred over OA in 70% (35/50) of cases. The second iteration (T2) often approached an optimal state, much like reaching favorable point in a loss landscape, by winning against OA in 84% (42/50) of comparisons and against TI in 60% (30/50) cases. However, similar to gradient descent nearing convergence overfitting, the gains from T2 to T3 were less distinct. Overall, T2 was preferred

---

over T3 in 23/50 cases. This plateau slight decline was also observed in specific dimensions: for comprehensiveness (Figure 4a). T2 maintained a slight edge over T3 (26 vs. 24 wins in their mutual comparison). This suggests that, like over-training a model, excessive iterations might not always yield better results across all facets.
4.4 Different LLM backbone (RQ.4)
The selection of LLM backbone is critical factor in shaping DoctorRAG's performance capabilities across diverse medical tasks (Table ( Performance exhibited distinct patterns on language and task within the DoctorRAG framework.
On Chinese datasets, DeepSeek-V2 V3 demonstrated particular strengths in DD (RJUA 86.79, MuZhi 80.19) and showed a prominent lead in TR (DialMed 63.49, RJUA 77.83). GPT-4. 1-mimi secured the leading position on the DialMed DD task (94.96), while GLM-4- achieved the most favorable results Chinese TG (RJUA 93.97 BS). In English/French contexts, GPT-4. 1 I-mini frequently emerged as the top performer, especially in QA (NEJM-QA 72.64), French DD (DDXPlus 98. 73), and English TG (COD 96.81 BS). GLM---Plus attained the highest accuracy in English DD (DDXPlus 98.27). Overall, DeepSeek-VV V3 demonstrated strong and consistent performance across all evaluated tasks, A closer examination task-specific capabilities within DoctorRAC highlighted specialized model aptitudes. In Disease Diagnosis, performance leadership was multifaceted: GPT-4.1-min demonstrated an edge on DialMed and DDXPlus (FR), DeepSeek-V V3 on RJUA and MuZhi, and GLM-4-Plus DDXPlus (EN), For Question Answering (NEJM-QA), GPT-4. 1-mini stood out as the leading model. DeepSeek- V3 was the frontrunner in Treatment Recommendation across both datasets. Regarding Text Generation, model efficacy closely tied to linguistic context, with GLM-4-Plus excelling in Chinese (RJUA) and GPT-4.1-ming in English (COD).
While these performance characteristics can be attributed differences in model training corpora and strategies, the collective results underscore the resilience and broad applicability of the DoctorRAG methodology across varied linguistic and task environments.
4.5 Ablation Study and Token Analysis (RQ.5)
We assessed the individual contributions of DoctorR AG's components and its RAG computational cost-performance profile through ablation and token analysis (Table
3
Figure details B.8).
Table 3: Ablation study of DoctorRAG.
3143
80.2
80.4
Tokens Sample
2856
79.6
Ablation Configuration
DialMed
NEJM-QA
Accuracy
76.9
2584
DoctorRAG (Full Model)
93.56
71.73
2313
Patient Base Retrieval
92.20
70.21
Knowledge Base Retrieval
91.34
70.52
74.5
1817
-Concept Tagging
91.65
70.52
Declarative Statement
92.23
71.12
Retrieve Top-K
Retrieve Top-K
Vanilla RAG
90.78
69.58
Figure 5: Performance v.s. Token consumption.
The ablation study (Table demonstrates the value of DoctorRAG's architecture. Removing any key component- Patient Base (experience), Knowledge Base (expertise), Concept Tagging, or Declarative Statement transformation degraded performance on both two tasks, confirming the crucial role of fusing structured knowledge with case- based experience via precise retrieval.
The token- peeen-performncce analysis (Figure 5) highlights a distinct trade-off. While increasing the contextual information (which includes both clinical knowledge and patient cases, controlled by the variable k) enhances DoctorRAG's performance on the Muzhi dataset, it also leads to linear increase of tokens processed per sample. Notably, this performance improvement appears to converge after & exceeds 4, indicating potentially diminishing returns for context added beyond this threshold.
Conclusion
This paper introduces Doc a medical RAG framework that emulates doctor-like reasoning by fusing explicit medical knowledge with experiential insights from analogous patient cases. Doctor- RAG employs a dual-retrieval mechanism enhanced by conceptual tagging and declarative statement

---

transformation, and uniquely integrates tion process, iteratively refine outputs for accuracy, relevance, and faithfulness to retrieved context and patient queries. Comprehensive multilingual, multitask experiments demonstrate DoctorR AG's outperformance of strong RAG baselines, generating more accurate, relevant, comprehensive and safe responses, thereby marking crucial step towards more robust and clinically astute medical reasoning systems.
multi-agent ageet textual gradient- based optimiza-
Acknowledge
This research was supported by National Key Research and Development Program of China (2044YF00007400) and National Natural Science Foundation of China (6220071694).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087774, 2023.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. (et.:,99,,,6, 2023.
[3] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72, 2005.
[4] Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural networks: Tricks of the trude: Second edition, pages 437-478. Springer, 2012.
[5] Carol H Cain, Anna C Davis, Benjamin Broder, Eugene Chu, Amanda Hauser DeHaven, Anthony Domenigoni, Nancy Gin, Anuj Kapoor, Vincent Liu, Ainsley MacLean, et al. Quality assurance during the rapid implementation of an ai-assisted clinical documentation support tool. NEJM AI, Alc2:4742400097,, 2025.
[6] Junying Chen, Chi Gui, Anningzhe Gao, Ke Ji, Xidong Wang, Xiang Wan, and Benyou Wang. Cod, towards an interpretable medical agent using chain of diagnosis, 2024.
[7] Minghui Chen, Ruinan Jin, Wenlong Deng, Yuanyuan Chen, Zhi Huang, Han Yu, and Xiaoxiao Li. Can textual gradient work in federated learning? arXiv preprini arXiv:2502. 19980, 2025.
[8] Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. Dense x retrieval: What retrieval granularity should we use?, 2024.
[9] Yixiang Chen, Penglei Sun, Xiang Li, and Xiaowen Chu. Mrd-rag: Enhancing medical diagnosis with multi-round retrieval-augmmented generation. arXiv preprint urXiv:2504.07724, 2025.
[10] Nabanita Choudhury and Shahin Ara Begum. A survey on case-based reasoning in medicine. International Journal of Advanced Computer Science and Applications, 136)1136-144, 2016.
[11] Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys I Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löffler, Sophie- Sophie-Caroline Caroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, al. The future landscape of large language models in medicine. Communications medicine, 3(1) 3(1):141, 2023.
[12] Sudeshna Das, Yao Ge, Yuting Guo, Swati Rajwal, JaMor Hairston, Jeanne Powell, Drew Walker, Snigdha Peddireddy, Sahithi Lakamana, Selen Bozkurt, et al. Two-layer retrieval- augmented generation framework for low-resource medical question answering using reddit data: Proof-of-concept study. Journal of Medical Internet Research, 27:e66220, 2025.
[13] Darren Edge, Ha Trinh, Newman Cheng. Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph rag approach to qucry-focused summarization, 2025.
10

---

[14] Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. Ddxplus: new dataset for automatic medical diagnosis. Advances in neural information processing systems, 35:31306-31311 2022.
[15] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmenter generation for large language models: A survey, arXiv preprint arXiv:2312. 10997, 2:1, 2023.
[16] Team GLM, Aohan Zeng. Bin Xu, Bowen Wang. Chenhui Zhang. Da Yin, Dan Zhang. Diego Rojas, Guanyu Feng, Hanlin Zhao, al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
[17] Zhenfeng He, Yuqiang Han, Zhenqiu Ouyang, Wei Gao, Hongxu Chen, Guandong Xu, and Jian Wu. Dialmed: A dataset for dialogue-based medication recommendation. arXiv preprint arXiv:2203.07099 2022.
[18] Yucheng Hu and Yuxing Lu. Rag and rau: survey on retrieval -uggmentd language model in natural language processing. arXiv preprint arXiv: 19543, 2024.
[19] Xinke Jiang, Yue Fang. Rihong Qiu, Haoyu Zhang, Yongxin Xu, Hao Chen, Wentao Zhang, Ruizhe Zhang, Yuchen Fang, Xu Chu, et al. Te-rag: Turing-complete rag's case study on medical Ilm systems, arXiv preprint arXiv:2408.09) 09199, 2024.
[20] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14))6421, 2021.
[21] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop. Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling rein- forcement learning from human feedback with feedback. arXiv preprint wxXiv2300.00227. 2023.
[22] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmenned generation for Knowledge-intensiv nip tasks. Advances in neural information processing systems, 33:9459-9474, 2020.
[23] Chin-Yew Lin, Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[24] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprini arXiv:2412. 19437, 2024.
[25] Yuxing Lu, Sin Yee Goi, Xukai Zhao, and Jinzhuo Wang. Biomedical knowledge graph: A survey of domains, tasks, and real-works applications. arXiv preprint arXiv:2501. 11632, 2025.
[26] Yuxing Lu, Xukai Zhao, and Jinzhuo Wang. Clinicalrag: Enhancing clinical decision support through heterogeneous knowledge retrieval. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 64-68, 2024.
[27] Shiwei Lyu, Chenfei Chi, Hongbo Cai, Lei Shi, Xiaoyan Yang. Lei Liu, Xiang Chen, Deng Zhao, Zhiqiang Zhang, Xianguo Lyu, Ming Zhang, Fangzhou Li, Xiaowei Ma, Yue Shen, Jinjie Gu, Wei Xue, and Yiran Huang. Rjua-qa: A comprehensive qa dataset for urology, 2023.
[28] Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08889 2023.
[29] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint
2018.
[30] Karen Ka Yan Ng, Izuki Matsuba, and Peter Chengming Zhang. Rag in health care: novel framework for improving communication and dacision-making by addressing IIm limitations. NEJM AI, A1r:AAra240000,,, 2025.
11

---

[31] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 31 311-318, 2002.
[32] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 01553728-53744 2023.
[33] Mahimai Raja. E Yuvaraajan, et al. A rag-based medical assistant especially for infectious diseases. In 2024 International Conference on Inventive Computation Technologies (ICICT), pages 1128 1133. IEEE, 2024.
[34] Mohamed Rashad, Ahmed Zahran, Abanoub Amin, Amr Abdelaal, and Mohamed AlTantawy. Factalign: Fact-level hallucination detection and classification through knowledge graph align- ment. In Proceedings of the 4th Workshop Trustworthy Natural Language Processing (TrustNLP 2024), pages 79-84, 2024.
[35] Xiaoshuai Song, Zhengyang Wang, Keqing He, Guanting Dong, Yutao Mou, Jinxu Zhao, and Weiran Xu. Knowledge editing on black-box large language models. arXiv preprint arXiv:2402.08631, 2024.
[36] Chengrui Wang, Qingging Long, Meng Xiao, Xunxin Cai, Chengjun Wu, Zhen Meng, Xuezhi Wang, and Yuanchun Zhou. Biorag: A rag-Ilm framework for biological question reasoning. arXiv preprint arXiv:2408.01107 2024.
[37] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[38] Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuan-Jing Huang. Kam-Fai Wong, and Xiang Dai. Task-oriented dialogue system for automatic diagnosis. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201-207, 2018.
[39] Junde Wu, Jiayuan Zhu, Yunli Qi, Jingkun Chen, Min Xu, Filippo Menolascina, and Vicente Grau. Medical graph rag: Towards safe medical large language model via graph retrieval- augmented generation. arXiv preprini arXiv:2408.06 1877 2024.
[40] Sean Wu, Michael Koo, Fabien Scalzo, and Ira Kurtz. Automedprompt: A new framework for optimizing Ilm medical prompts using textual gradients. arXiv preprint arXiv:2502. 15944 2025.
[41] Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval- augmenel- generation for medicine. In Findings of the Association for Computational Linguistics ACL 2024, pages 6233-6251, 2024.
[42] Guangzhi Xiong, Qiao Jin, Xiao Wang, Minjia Zhang, Zhiyong Lu, and Aidong Zhang. Im- proving
generation in medicine with iterative follow-up questions. In Biocomputing 2025: Proceedings of the Pacific Symposium, pages 199-214, World Scientific, 2024.
[43] Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative ai by backpropagating language model feedback. Nature, 63968055)6009-66 616. 2025.
[44] Cyril Zakka, Rohan Shad, Akash Chaurasia, Alex R Dalal, Jennifer L Kim, Michael Moor, Robyn Fong, Curran Phillips, Kevin Alexander, Euan Ashley, et al. Almanac Almanac-rettrivaall augmented language models for clinical medicine. Nejm ai, 1(2):Alo2200068, 2024.
[45] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 2019.
12

---

[46] Xuejiao Zhao, Siyan Liu, Su-Yin Yang, and Chunyan Miao. Medrag: Enhancing retrieval- augmented generation with knowledge graph-elicited reasoning for healtheare copilot. In Proceedings of the ACM on Web Conference 2025, pages 4442 4457,44572222 2025.
[47] Yinghao Zhu, Changyu Ren, Shiyun Xie, Shukai Liu, Hangyuan Ji, Zixiang Wang, Tao Sun, Long He, Zhoujun Li, Xi Zhu, et al. Realm: Rag-driven enhancement of multimodal electronic health records analysis via large language models. arXiv preprint arXiv:2402.0700 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction claim that DoctorRAG integrates clinical knowl- edge and case-based experience, enhances retrieval precision, and uses Med-TextGrrad for output adherence, outperforming baselines. These claims supported by the methodology (Section 2) and results (Section 4 presented.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
"Limitations" explicitly discusses limitations such as dependency on knowledge base quality, LLM backbone capabilities, and the need for human oversight.
Justification: Section C.3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
14

---

detailed judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
The paper proposes framework and empirically evaluates it; it does not primarily present theoretical results with mathematical proofs. While there are equations describing the methodology (e-g., similarity scores, loss function), these are definitions within the framework rather than theorems requiring formal proofs.
Justification:
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper details the datasets used (Section 3.1 Appendix B.I. LLM back- bones (Section (.22, evaluation metrics (Section 4.1 and provides detailed prompts and algorithm descriptions in the appendices (Appendix D.I. Algorithm [T]. This should be sufficient for others to understand and attempt to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
15

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Appendix B. states. "An overview of these datasets, along with direct links to their corresponding research papers and data repositories, is presented in Table 47 Appendix [B.T also mentions, "The source code for our proposed models, DoctorRAG, Med- TextGrad, along with pairwise comparison evaluations is available in the Supplementary Materials."
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (kttps://nips.co Sogpll//guuiiisss/dddbbbbbssssoo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https htappp/ccc/plll.eaiiiiee//e/eees
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
16

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Section 3.1 and Appendix B.1 provide details on data splits (80% for patient base, rest for evaluation), LLM backbones used for specific tasks like preprocessing and embedding, and the iterative process of Hed-Tex(Grad (3 iterations or convergence). The setup for the RAG framework and Med-TextGrad process is described.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We reported the results with mean +/- 95% CI in Figure
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper use different L.M backbones for evaluation, and provided the token analysis in Figur(5] All experiments use APIs.

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g,, preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurips. http:/pp..pppppp.cteccccccccaaa.
Answer: [Yes]
The research aims to improve medical reasoning systems, which is beneficial application. The paper discusses limitations and ethical considerations (Section C.3 and C.I such as the tool being decision support and not replacement for human professionals. The datasets used are cited and appear to be from public sources or de-identified patient records.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The paper primarily focuses on the positive societal impact of creating more robust and doctor-like medical reasoning systems assist medical professionals and enhance patient care.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
18

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper utilizes existing LLMs and datasets (some involving de-identified patient records). The paper does not detail specific safeguards for the release of new models datasets it might generate that have high risk for misuse, beyond using de-identified data and citing established datasets. The focus is on the framework itself.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper cites the sources for the datasets used (Table [ Table Section 3.1 and the LLM backbones (Section 3.2). Table [ provides links to dataset repositories, where license information would typically be found.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcooeee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
The primary contribution is the DoctorRAG framework and Med-TextGrad process rather than new datasets or standalone models released as new assets. The framework itself and its components are described in the paper and appendices. The source code for the models is mentioned as available in Supplementary Materials.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cone/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We don't use crowdsourcing experiments in our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We don't perform research with human subjects in our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: LLMs are a core component of the DoctorRAG framework. Section 3.2 details the LLMs used (DeepSeek- V3, Qwen-3-32B. GLM-4-Plus, GPT-4.1) and their roles (e.g.. data preprocessing, embedding, main model backbone). The entire methodology revolves around using LLMs for aurieval-augmentie generation and refinement through textual gradients.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy [tteee/r//errrppaccc///offrrrers for what should or should not be described.
21

---

A Related Work
A.1 Medical RAG systems
Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external knowledge sources, which is essential in the medical domain where precision and timeliness critical [88] [3]. By combining retrieval and generative AI, RAG mitigates LLM limitations such as hallucinations and outdated data, supporting tasks such as clinical decision making, medical question answering, and electronic health record (EHR) summarization using sources such as clinical guidelines, knowledge graph, academic literature, and EHRs 39
9 19 25
Several RAG frameworks have been tailored for medical applications. For example, Almanac integrates a curated medical database and advanced reasoning to improve factuality in clinical question answering [4]. Das et al. proposed two-layer RAG framework leveraging social media data for low-resource medical question-ansserrngg effective in constrained settings [12]. MedRAG improves diagnostic accuracy and personalized prrisionnmaking using RAG model augmented by knowledge graphs [46]. Clinical RAG accesses heterogeneous medical knowledge to perform multi-agent medical reasoning [26). i-MedRAG refines medical answers through iterative retrieval and follow-up queries for more accurate, personalized care [42). BioRAG [36] and REALM [47 also use biomedical domain knowledge to enhance biological question reasoning and personalized decision making.
Despite these advances, existing RAG frameworks rely static knowledge bases, such guidelines or the literature, and under-utilize historical patient data. Our DoctorRAG framework addresses this by integrating external expert knowledge databases with patient outcome repositories, enabling personalized, contextually grounded clinical recommendations, distinct from prior approaches and better suited for real-world medical practice.
A.2 Post-generation processes
While RAG significantly improves the quality of LLM outputs by grounding them in retrieved documents, the generated text may still exhibit issues such as factual inaccuracies, incoherence, or misalignment with user intent. To address these limitations, post-generation processes are applied to refine and optimize the outputs produced by RAG systems.
Hallucination detection and mitigation are critical, with methods like SelfCheckGPT (28 and Chain- of- Verification (28] enabling LLMs to self-evaluate and verify claims against retrieved documents, while FACTALIGN 34) improves factuality in long-form responses. Coherence and style are enhanced through techniques like Chain of Thought prompting [77) while alignment methods, including Direct Preference Optimization (DPO) (32) and Reinforcement Learning with AI Feedback (RLAIF) (1), ensure outputs align with user preferences and ethical standards. Knowledge editing frameworks like postEdit [35 enable targeted factual updates, maintaining consistency with evolving information.
DoctorRAG introduces an advanced post-generation framework by leveraging multi-agent, iterative optimization approach tailored for medical reasoning. It addresses limitations in factual accuracy and contextual relevance through its Med- -ed-Grr optimization, inspired by TextGrad [33 40 [7]. This method refines outputs via dual feedback loops incorporating clinical knowledge and patient-specific criteria, ensuring alignment with medical evidence and enabling doctor-like like reasoning for complex tasks such as differential diagnosis and treatment planning.
B
Technical Appendices and Supplementary Material
B.1 Datasets and source code
In our experiments, we utilize seven diverse datasets to ensure a comprehensive evaluation of our methodologies across multiple languages and medical tasks (Table DD. These datasets, spanning Chinese, English, and French, encompass range of applications critical medical informatics, including disease diagnosis (DD), medical question answering (QA), treatment recommendation (TR), and text generation (TG). An overview of these datasets, along with direct links to their corresponding research papers and data repositories, is presented in Table
These selected datasets provide a robust foundation for assessing the multilingual and multitask capabilities of DoC They represent a variety of real-world and benchmark scenarios in the medical domain.
22

---

Table Overview of Datasets Utilized in Experiments. All linked resources were last accessed on May 16, 2025.
Language
Dataset
Paper
Dataset
DialMed
httpa///arttv.org/aaa///// 07094
Chinese
RJUA
http://arttv..rr/aa///22...
httpa://itthub.com
Muzhi
atpp:///arttv.o00////////90 10623
DDXPlus
http:://arrtx..rrr/////222..0.00
NEJM QA
httpa:
English
1056/ 0966/A1c22409977
//hgginngacc. //miggiagfaco.co//aaa htgp:///c....coaaaaaaaaa/ao...aa
COD
ht ht nttpaa//arxxv.org/aab//////
htipa://huggingtt wtt.c//.c/occeaaaaaaaaaaaaaa
FreadonIntol Siw.mlnnntliggoo/CCCoooo Pttpoooaaaaittmmmaaaao.aa.
French
DDXPlus
attpe://azzti.ogg/bba/220...
httpe://github.coo http://thhb...o/nllliieie paailllliiiddpllll
The source code for proposed models, DoctorRAG, Med-TextGrad, along with pairwise compari- son evaluations is available the Supplementary Materials.
B.2 Med-TextGrad algorithm
Algorithm 1 outlines the Med-TextGraa iterative answer refinement process. The primary goal is to enhance an initial answer, Astgel (generated by the DoctorRAG), by leveraging both domain-specific knowledge and patient-specific data. The process takes the user's query q. an aggregated context (comprising general medical knowledge from and relevant patient case experiences from P), the initial answer Astgett, and predefined number of iterations 7 as input. This iterative cycle of answer refinement, multi-faceted critique, and gradient-guided prompt optimization continues for T iterations. The specific instructions and templates used for each LLM call (e.g.. Generator KnowledgeCriterion.. LLMans_gun gradd_ggad___a ke. ke. LLMgd. etc.) are detailed Appendix D.2
Algorithm Med-TextGrad Iterative Answer Refinement
Require: Initial User Query y
Require: Aggregated Context (derived from Knowledge Base K. and Patient Base P)
Require: Original Answer Asuget
Require: Number of iterations T
Ensure: Final Refined Answer V/nal
1: function MED-TEXTGRADDC.CC Aguget T)
2: Pourrent + Construclnttalleefeeeeeetttooooo C, Astgel)
3: A' Agurrent null
4: Agiital refined by MedTextGrad f- null
5:
for 1=
do
6:
Step (a): Answer Generation/Refinement
7:
Averrent < Generator (P
8:
Step (b): Critique Generation
9:
Okc + KnowledgeCriterion, (A C)
10:
Opc + PatientCriterion (AA 9)
Step (c): Answer-Level Textual Gradient Computation
12:
C + LLM ans grad ke (.uumrens ( C, Okc)
V V.. t LLMans grad ppl
Aquitet q, Cpc)
14:
Step (d): Prompt-Level Textual Gradient Computation
V PCCKC LLM LLMpoopp
prompt grad grad Pourrents Aquurent: V.CC
16:
V PpC & LLM prompt Bacp___rad____ee
( Poument: A'urents A.
17:
ifl <T then
18:
> Step (e): Prompt Update (Textual Gradient Descent)
Poext <- LLM Ca.MigaPaarmennt (Paureett PCCK, PCCP)
20:
Poutrett Pnext
21:
Al f- A
22:
return Agual
23

---

B.3 Declarative Sentence Transformation
A key step in our experimental setup involved transforming the bilingual (Chinese and English) knowledge contained within the MedQA dataset. The original multiple-choice question format was converted into declarative sentences. This process aimed to distill the essential information by integrating the question with its correct answer, while explicitly excluding the incorrect options. To achieve this transformation, we utilized the following specific prompt:
have medical multiple choice question. Please convert into a statement that includes only the question and the correct answer, without the incorrect options.
Question: [question]
Options: [formatted_options)
Correct answer: {answer_ key). [answer_jen
Please oulput only the converted statement without any additional explanation.
This declarative sentence transformation offers several benefits for our experimental objectives. Firstly, standardizes the knowledge representation, converting varied question structures into uniform, factual format that is more amenable to computational processing. By directly asserting the relationship between the question's premise and the correct answer, this method provides clear and unambiguous factual input to our models, free from the potential noise or misdirection of incorrect options. Such direct factual representation including knowledge retrieval and model training, as it simplifies the interpretation and utilization of the core information. Furthermore, transforming diverse knowledge types into declarative statements facilitates more effective integration and reasoning across different sources.
advantageous for downstream tasks,
Beyond the multiple-choice questions, we also leveraged knowledge from medical textbooks in MedQA. This textbook-derived information was segmented into appropriately sized chunks and, similar to the MedQA questions, transformed into declarative statements to ensure consistency. This formed additional distinct knowledge source for our experiments, benefiting from the same advantages of a standardized and direct factual representation.
B.4 Concept tagging
To enhance the precision of knowledge matching between our information corpus and user queries, we implemented concept tagging system. This system assigns relevant concept tags to each piece knowledge and every incoming user query. In our approach, we selected the International Classification of Diseases, Tenth Revision (ICD-10) codes as the foundational basis for these concept tags.
ICD-10 is a globally recognized medical classification system published by the World Health Orga- nization (WHO). It provides standardized nomenclature for diseases, signs, symptoms, abnormal findings, complaints, social cireumstances, and external causes of injury or diseases, assigning unique alphanumeric codes to each. Its hierarchical structure and comprehensive coverage make it an invaluable tool for organizing and retrieving health-related information.
To maintain manageable and effective tagging framework, and simplify the matching process. we chose to utilize only the first-level categories of ICD-10 10 (c.g.. "A00-B99 Certain infectious and parasitic diseases." " "C00-D48 Neoplasms"), This decision prevents the tagging system from becoming overly granular and complex, which could hinder efficient matching.
The process of assigning these first-level ICD-10 categories was automated using the following prompt directed at LLM:
As medical expert, determine the most likely ICD-10 category (first level only) for a patient with the following symptoms:
Symptoms: (all call_symptomss
Please select the appropriate ICD-10 category from the following list: [ICDIO CATEGORIES JSON STRING]
Return only the category code (e.g. 'A00-R99") without any additional text or explanation.

---

This concept tagging approach offers several key benefits, significantly enhancing our system's s capabilities. It improves matching aceuracy by focusing on underlying medical concepts rather than mere keyword overlap, thereby increasing semantic relevance and reducing ambiguity. The use of ICD- 10 ensures a standardized representation, providing a consistent and universally understood framework for categorizing medical information. Consequently, this leads to more efficient information retrieval, as tags enable structured and effective access, connecting user queries to conceptually related knowledge.
B.5 Rest metrics generative tasks
Here present another two metrics (BLEU-4 and METEOR) for evaluating DoctorRAG's capability in text generation tasks. The full result table can refer to Table 2
Metric
Dataset
Backbone
Direct Gen.
Vanilla RAG
Prop. RAG
Graph RAG
DoctorRAG
GLM-4-Plus
5.56
6.45
13.24
3-32B
5.01
5.05
RJUA (CN)
DeepSeek-Ve V3
5.52
5.81
6.16
8.92
11.00
GPT-4,I-mini
6.28
6.37
6.45
7,48
8.53
BLEU4
GLM-4-Plus
17.39
18.21
20.44
Qwen-3-32B
15.78
16.62
COD (EN)
DeepSeek-V2 V3
15.78
16.18
16.62
16.87
17.09
GPT-4.Immini
18.16
17.98
17.77
18.54
19.33
GLM-4-Plus
24.42
24.12
23.86
28.37
33.56
Qwen-3-32B
R.UUA (CN)
DeepSeek-V V3
20.91
21.19
21.52
24.08
29.23
GPT-4.1-mini
24.98
25.83
26.77
27.95
29.19
METEOR
Gl.M-4-Plus
31.35
Qwen-3-32B
32.70
32.81
32.91
32.97
33.02
COD (EN)
DeepSeek V3
30.76
31.07
31.40
31.49
31.58
GPT-4.1-mini 29.75
30.52
31.33
30.81
30.32
Table 5: BLEU and METEOR metries for generative tasks across different datasets, with results highlighted as follows: pink for the overall top score per task, and green for the top score within each RAG method group for that task.
B.6 Umap visualization
To visualize the distribution and clustering of disease-related embeddings across multiple datasets, employed UMAP (29 for dimensionality reduction. Vector representations were extracted from the FAISS index constructed from patient databases of various datasets, with labels sourced from the original task files of each dataset. We focused the five most frequent diseases per dataset, sampling up to 1000 instances per disease to ensure balanced representation. The high-dimensional embeddings were projected into dimensions using the Euclidean metric, with UMAP parameters was to n m_meighbors = 20 and min min_diss fiitt = 0.1. Disease labels were integer-encoded for visualization.
B.7 Pairwise comparison of RAG methods
To evaluate DoctorRAC AG against other RAG methods comprehensiveness, relevance, and safety, we performed pairwise comparisons of their generated responses. We used DeepSeck- V3 [24] as the evaluator determine which method's response was computed as the average of the three dimension scores. This prompt was applied uniformly, with the query and responses tailored to each evaluation instance. The structured format ensured clear, reasoned judgments, enabling robust comparisons between DoctorRAG and other RAG methods. The overall score, derived from the average of the three dimensions, provided a comprehensive measure of performance.
superior in each dimension. The overall score
B.8 Ablation study
Our ablation studies were conducted from three perspectives.
Firstly, we performed an ablation study on the doctorRAG by systematically removing individual modules and evaluating the model's performance two datasets: Muzhi and NEJM-QA. The results, presented Table 33, demonstrate that each component doctorRAG significantly contributes the overall performance.
25

---

Secondly, we conducted an ablation study targeting Med-TextGrad This involved pairwise compar- isons of answers generated by Section 4.3 the findings indicate that Med-Texttina enhances the comprehensiveness, relevance, and safety of the generated answers. However, it also exhibited tendency to converge after a few iterations.
across varying numbers of iterations. As detailed in
Thirdly, our ablation experiments examined the impact of the volume of indexed knowledge and the quantity of patient information on the outcomes. These results, illustrated in Figure 5 show that as the amount of knowledge and patient information increased, token consumption rose linearly, accompanied by an improvement in the model's s output quality. Nevertheless, a similar convergence phenomenon was observed in this aspect as well.
C Discussion
Our work introduces DoctorRAG, a framework designed to emulate doctor-like reasoning by fusing explicit medical knowledge with implicit experiential knowledge from patient cases, and refining outputs via the Med-TextGrad process. The promising results across diverse multilingual and multitask datasets demonstrate the potential of this approach. However, it is important to discuss several aspects, including the nature of textual gradients, inference strategies of TextGr,,,,,,, and the inherent limitations of our current work.
C.1 Generalizability and nature of "Textual Gradients"
The Med-TextGrad process introduces "textual gradients" a mechanism for iterative refinement 43 It's important to clarify that this term is used conceptually to describe LLM--genrrted textual feedback that guides the optimization of prompts and answers, akin to how numerical gradients guide parameter updates in neural networks [4]. These are not mathematical gradients in the traditional sense but rather structured, actionable critiques and suggestions generated by LLM agents.
The generalizability of this textual gradient approach hinges on several factors. Firstly, LLM capabilities are paramount; the effectiveness of the Context Criterion, Patient Criterion, and the Optimizer agents generating high-quality, actionable "gradients" directly tied to the underlying reasoning and language understanding capabilities of the LLMs used. As LLMs continue to advance, the precision and utility of these textual gradients are expected to improve. Secondly, prompt engineering plays critical role; the design of the prompts (detailed in D.2 for each agent is critical, and while we provide robust prompts, their optimal formulation might vary across different LLM backbones specific medical sub-domains. Thirdly, domain adaptability is consideration; while developed for the medical domain, the core concept of multi-agent iterative refinement using textual feedback could be generalizable to other domains requiring faghffaithfulnes,, context-adherent, and nuanced text generation, such as legal document drafting or technical writing.
C.2 Automated metrics for iterative refinement
Our current evaluation of the Med-TextGrad iterative refinement process (Section 2.2 relies on pairwise comparisons using LLM-based voting with human expert verification. While insightful, this method can be resource-intensive for large-scale experimentation. Future work will focus on developing and incorporating a more diverse set of automated metrics to rigorously evaluate the iterative refinement process and its impact on answer quality. Potential directions include developing faithfulness metrics to quantify the factual consistency of the generated answer against retrieved knowledge, which could involve natural language inference (NLJ) models fine-tuned on medical text or knowledge graph alignment techniques.
Designing convergence and improvement trajectory metrics that can track the answer's evolution across
iterations will help determine optimal stopping points automatically. Also, developing aspect-specific evaluation metrics for qualities such as empathy, clarity for patients, and comprehensiveness in addressing all parts of patient's 's query will be beneficial. These automated metrics would allow for more scalable and nuanced analysis of the Med-Textrrad process and facilitate faster experimentation with different configurations and LLM agents.
C.3 Limitations
Despite its promising performance, DoctorRAG has several limitations that warrant discussion. The performance is heavily dependent on the quality, eurrency, and comprehensiveness of the medical knowledge in the KB and the Patient Base; maintaining these sources is significant undertaking
26

---

Also, the entire framework relies fundamentally on the capabilitiees of the chosen LLM backbones, and any inherent limitations or biases in these LLMs can directly impact performance, especially the quality of generated "textual gradients."
must be emphasized that DoctorRAG is intended as a clinical decision support tool, not a replace- ment for human medical professionals; ethical considerations and human oversight are paramount, and the risk of over-reliance on Al-generated advice must actively mitigated. Addressing these limitations will be central to future research efforts work towards developing more robust, reliable, and ethically sound Al systems for healthcare.
D Prompts of different tasks
D.1 Prompts for DoctorRAG
This section provides the detailed prompt used for the DoctorRAG framework (use the DialMed disease diagnosis task as an example). This prompt designed guide the LLM in its role as medical diagnostic expert, leveraging patient information, knowledge context, and similar patient cases to predict the most likely disease. The overall behavior of the LLM is first set by a system message.
System Message
Role Description: This system message prepended the user prompt define the LLM''s role. Message Content: You medical diagnostic expert. Answer with just the disease name.
The prompt for DoctorRAG combines patient-specion information with contextual knowledge and examples to guide the diagnostic prediction.
Prompt: Disease Diagnosis for Patient (DialMed)
The LLM acts as a medical diagnostic expert. Its task is to predict the most likely disease for patient based their symptoms, provided medical knowledge, and information from similar patient cases. The "System Message" described above used.
Role Description:
Prompt Template Structure:
Input:
Patient Symptoms:
(evidences) (List of symptoms) Knowledge Context: (enowledge_text) (Text containing relevant medical
knowledge)
Similar Patients Information: Saamilar_patieens_teeee (Text describing
similar patient cases)
Valid Disease Options: (VALID DIEEDDDISEAESS (List of possible diseases to choose from)
Instructions within the prompt to the LLM: As a medical diagnostic expert,
predict most likely disease this patient.
Patient Information:
Symptoms: (evidences)
Relevant Information:
(knowledge_ text)
(similar aalaarrppttt _ttppaiiiaa ents ants_text]
Based on the above information, determine the most likely disease from the following
options only:
BALLIDDDSEASES
Return only the name of the disease without any additional text.
Output format: Only the name of the predicted disease.
D.2 Prompts for Med-TextGrald
This section provides the detailed prompts used for the different L.M agents within the Med-TextGrad framework. Each prompt is designed to guide the LLM in its specific role, from answer generation
27

---

and critique to textual gradient computation and prompt refinement. The overall behavior of the LLM first set by system message, which varies depending on the task.
The "Default" system message, detailed below, is prepended for most critique and gradient computa- tion tasks, instructing the LLM to be a specialized medical text refiner.
System Message (Default)
This system message prepended to the user prompt for most critique and gradient computation tasks.
Role Description:
You sophisticated Al assistant specializing in refining medical text based on critiques and context, ensuring accuracy and relevance while preserving core meaning. Output only the requested text without preamble or explanation unless otherwise specified.
Message Content:
For tasks involving the generation or direct update of answers and prompts, slightly different system message is employed to focus the LLM on generation and adherence to instructions.
System Message (Answer Generation Prompt Update)
Role Description:
This system message prepended to the user prompt for answer genera- tion/refinement and prompt update tasks.
Message Content: You are an AI assistant that generates and refines prompts or answers for a medical question-answent system. Focus on clarity, accuracy, and adherence to instructions. Output only the requested text without preamble or explanation unless otherwise specified.
The Med-TextGrad process begins with an initial prompt designed for the Generator agent (Gen.)). This prompt, Po, combines the patient's query, supporting context, and the current answer to be refined.
Prompt: Initial Refinement Prompt for Generator
The LLM acts as helpful medical consultation AI. task is to refine a given 'current answer (A)' based on the patient's query and supporting context. This prompt is combined with the "current answer (A)' to form the full input the Generator. The "Answer Generation / Prompt Update" system message is used.
Role Descríption:
Prompt Template Structure (combined with current answer A):
Input:
Patient Query (q): {query_q)
Supporting Context (C): (context_c)
Current Answer (A) to Refine: (answer_dd _a]
Instructions within the prompt to the LLM: You are helpful medical consultation AI. Your task is to REFINE the 'current answer (A)' (which will be provided at the end of this prompt) by critically evaluating it against the patient's query and the supporting context. The goal is to improve the 'current answer (A)'s accuracy, completeness, and relevance to the patient's query, while preserving its valid core information and avoiding unnecessary deviations from its original intent.
Patient Query (q): (query_q)
Supporting Context (C): [context_c)
Instructions for refinement: 1. Carefully read the 'Patient Query (q)' and the 'Sup- porting Context (C)', 2. Critically evaluate the 'current answer (A)' (provided below) against this information. Generate improved and refined version of the 'current answer (A)'. 4. Focus on addressing any shortcomings in the 'current answer (A)' regarding accuracy, completeness, clarity, and direct relevance to the patient's ss query. 5. Ensure your refined answer is factually sound based on the context, empathetic, and easy for a patient understand. IMPORTANT: Your output must be ONLY
28

---

the refined medical answer itself. Do not include any preamble, conversational phrases, meta-commentary. or any text other than the refined answer.
Current Answer (A) to Refine: {answer _a)
Output format: Only the refined medical answer.
Once an answer A, is generated, it is evaluated by two criterion agents. The first is the Context Criterion Agent p). which assesses the answer's alignment with the provided medical knowledge.
Prompt: Context Criterion
Role Description:
The LLM critiques a generated answer based its factual alignment, consistency, and completeness with respect to provided medical context. The "Default" system message is used.
Prompt:
Input:
Medical Context (C): [context_e ct
GGneeatedd Answer (A): (answer_al
Instructions to the LLM: Given the following medical context (C): {contexte))
And the following generated answer (A): {answer
Critique the answer (A) focusing ONLY its factual alignment, consistency, and completeness with respect to the provided medical context (C), Provide specific, con- cise points of criticism or confirmation. These are CONTEXT-FOCUSSE critiques. Identify areas where the answer might misrepresent or omit crucial information from the context. Output ONLY the critique. Output format: Only the critique.
Concurrently, the Patient Criterion Agent (PC+) evaluates the same answer A, for its relevance and appropriateness concerning the original patient query.
Prompt: Patient Criterion
Role Description: The LLM critiques generated answer based on its relevance, appro- priateness, and specificity to the original patient query. The "Default" system message is used.
Prompt:
Input:
Original Patient Query (q): (query_g
Generated Answer (A): cinswer_a]
Instructions the LLM: Given the original patient query (q): (query_q)
And the following generated answer (A): (answer_a)
Critique the answer (A) focusing ONLY on its relevance, appropriateness, and specificity to the patient's query (q). Provide specific, concise points of criticism or confirmation. These are PATIENT-FOCUSED critiques. Does the answer fully address the patient's 's concerns as expressed in the query? Output ONLY the critique. Output format: Only the critique.
The critiques from the Context Criterion Agent then used to compute textual gradients for the answer (Gyad A kc). This gradient provides actionable instructions on how to revise the answer to address knowledge-relate shortcomings.
Prompt: Textual Gradient for Answer (Knowledge)
The LLM acts as expert medical editor, providing actionable instructions refine medical answer based on knowledge-focuse focused critiques and context. The "Default" system message is used.
Role Description:
Prompt:
29

---

Input:
Original Answer (A): [answer
Relevant Medical Knowledge Context (C): [context_co c)
Context-Focused Critiques (Cxc): (critiques __ccs]__c_____cc_
Instructions to the LLM: You are an expert medical editor. Your task is to provide actionable instructions to refine a given medical answer based on specific critiques related to context.
Original Answer (A): {answer_at
Relevant Medical Knowledge Context (C): {contexi_c)
CONTEXT-FOCUSEE Critiques on the Original Answer: {critiques_cke
Based ONLY the provided critiques and referring the knowledge context, explain step-by-step how to revise the Original Answer to address these critiques. The explanation should focus on specific, targeted revisions that address the critiques while maintaining the overall structure and valid information in the original answer much possible. Your output should be actionable instructions for improving the answer. Do not write the revised answer itself. Output ONLY the instructions. Output format: Only the actionable instructions.
Similarly, textual gradients for the answer are computed based on the Patient Criterion Agent's critiques (ggnd A_PC), focusing on improving patient-specific relevance.
Prompt: Textual Gradient for Answer (Patient)
The LLM acts as an expert medical editor, providing actionable instructions to refine medical answer based on patient-query -uery-focusdd critiques. The "Default" system message is used.
Role Description:
Prompt:
Input:
Original Answer (A): (answer_om a}
Original Patient Query (q): {query_q] Patient-Focused Critiques (Cpc): {critiques_cpp cpc)
Instructions the LLM: You are an expert medical editor. Your task is to provide actionable instructions to refine a given medical answer based on specific critiques related to the patient's query.
Original Answer (A): (answer_a)
Original Patient Query (q): {query_4}
PATIENT-FOCUSED Critiques on the Original Answer: {critiques_ _ccc)
Based ONLY on the provided critiques and referring to the patient's s query, explain step-by-step how to revise the Original Answer to address these critiques. The explanation should focus on specific, targeted revisions that address the critiques while maintaining the overall structure and valid information in the original answer as much as possible. Your output should be actionable instructions for improving the answer. Do not write the revised answer itself. Output ONLY the instructions. Output format: Only the actionable instructions.
These answer-level gradients are then used to compute gradients for the prompt itself. The (Ggnd (Ggnd_P_KC)) agent refines the original prompt P. based on the knowledge-focusee answer gradient.
Prompt: Textual Gradient for Prompt (Knowledge)
Role Description:
The LLM acts as an Al assistant that refines prompts for medical question-answering system, focusing on knowledge-based feedback. The "Answer Generation Prompt Update" system message is used.
Prompt:
Input:
Original Prompt (P): (prompt_p)
30

---

Generated Answer (A) from P: {answer_at
(Knowledge-focusee answer gradient [grad_answer aww.aaawwaaaa..... [ewer__c) [grad_ Instructions the LLM: You are an Al assistant that refines prompts for medical
Feedback on
V.
guestion-answering system.
The Original Prompt (P) used was: (prompt_p)
The answer A generated using P was: {answer a)
The Answer (A) requires revisions from KNOWLEDGE perspective, as suggested by the following feedback on A: (grad_an crrud_anwwer_kt
Your task: Explain precisely how to modify or rewrite the ORIGINAL PROMPT (P) to create improved P. This P should better guide the L.M to refine answer like A, addressing the KNOWLEDGE- DEWLLEEEE-FCUSED issues identified in the feedback. Provide clear, actionable advice on prompt improvement. Output ONLY the advice.
Output format:
Only the advice for prompt improvement.
In parallel, the (ggrad Pc) agent refines the prompt Pt based on the patient- answer gradient, aiming improve the prompt's ability to elicit matient-relevan answers.
Prompt: Textual Gradient for Prompt (Patient)
The LLM acts as an Al assistant that refines prompts for medical question- answering system, focusing patient query-based feedback. The "Answer Generation Prompt Update" system message is used.
Role Description:
Prompt:
Input:
Original Prompt (P): (prompt_p)
-Gnerated Answer (A) from P: {answer_a)
Feedback on (Patient-focusse answer gradient Crc): fgrad answer pc)
Instructions to the LLM: You are an Al assistant that refines prompts for medical
nuestion-answwring system.
The Original Prompt (P) used was: (prompt_p)
The answer generated using was: (answer_dd a)
The Answer (A) requires revisions from a PATIENT QUERY perspective, as sug- gested by the following feedback on A: ggrad_answer.a
Your task: Explain precisely how to modify or rewrite the ORIGINAL PROMPT (P) to create an improved P. This should better guide the LLM to refine an answer issues identified in grad_an aad__aswer_pee Provide clear, actionable advice on prompt improvement. Output ONLY the advice. Only the advice for prompt improvement.
like A, addressing the PATIENT-FOCUSE
Output format:
Finally, the distinct prompt gradients (context and agent to perform a textual gradient descent step. This step generates an updated prompt Ptt1 for the next iteration.
synthesized by the (GTGD)
Prompt: Prompt Update (Textual Gradient Descent Step)
The LLM acts as an Al assistant tasked with refining a medical consultation prompt based on aggregated feedback (gradients). The "Answer Generation / Prompt Update" system message is used.
Role Description:
Prompt:
Input:
Original Prompt (P:): {current_prompt_p)
Original Answer (A₂) generated by Pt: [answer_d]
Feedback from Context Critiques (VpCkc): (graa_ prompt_kkt kc)
Feedback from Patient Query Critiques pVcCr [grad hardd_proppppppee
Original Patient Query (q): (query_g)
31

---

Example Answer that P. was used to refine: [initial_answer wwwwwer_exampeer
Instructions to the LLM: You are an AI assistant tasked with refining medical consultation prompt.
The Original Prompt (P_t) I) to be improved is: {current (curren_prompt__pp
The Original Answer generated by P_t is: {answer_at
We need to create an Updated Prompt (P) based on the following feedback, which
aims to make P more effective for refining answers like A. Feedback derived from Context critiques (grad_prompp _eppmpttoppeeeee
poruddprompl__g
Feedback derived from PATIENT QUERY critiques (grad prppptppppp_p
(graddprompl_""
Original Patient Query (q) for context: (query_g) Example of an answer (A) that P_t was used to refine: [initial answer pnawereexmppee
Your task: Generate the Updated Prompt (P). This P must be complete set of instructions and context. When P is later combined with new 'current answer (A)', the LLM receiving (P actual text of A) should be guided to: 1. Produce refined version of that curreet answer (A)'. 2. Implicitly incorporate the improvements suggested by the feedback (grad prompt_ kc, pwtelrrroppppppeeee _e)pppe))) through your new instructions in P. 3. Ensure the refined answer directly addresses the patient's query, is factually sound, clear, and empathetic. Output ONLY the refined medical answer itself, without any conversational preamble, meta-commentary, self-correction notes, any text other than the refined answer.
Focus on making P a robust set of instructions for the refinement task.
Output ONLY the Updated Prompt (P).
Output format: Only the Updated Prompt (P).
This iterative process of generation, critique, gradient computation, and prompt update allows Med-TextGrad to progressively refine the medical consultation provided.
D.3 Prompts for pairwise comparison
This section provides the detailed prompts used for pairwise comparison of different answers.
Prompt: Pairwise L Comparison
You are clinical expert. Your responsibility is to: Compare responses from two Retrieval-Augmented Generation (RAG) methods for the same medical query. 2. Compare the responses across three dimensions: comprehensiveness, relevance, and safety. 3. Provide structured evaluation with justifications for each dimension.
Role Description:
Prompt:
Input: A medical query: {query) and two responses (Response A: (Response A) and Response B: (Response B)) from different RAG methods.
Output format: A structured evaluation in the format:
Comprehensiveness: [Response A/B] [Justification]
Relevance: [Response A/B] [Justification]
Safety: [Response A/B] - [Justification]
Key Requirements:
Comprehensiveness: Evaluate whether the response is thorough, covering all relevant aspects of the query.
Relevance: Determine if the response directly addresses the query without extraneous information.
Safety: Ensure the response is medically accurate, avoiding harmful or mislead- ing advice and adhering to clinical best practices.
Error Handling:
If a response is incomplete or unclear, note the issue in the justification and evaluate based on available content.
If the query is ambiguous, base the evaluation on the most likely interpretation.
32

---

E Examples of outputs
E.1 Examples for DoctorRAG
This section presents an example of DoctorRAG's output using the COD dataset. We include the initial medical query, the relevant knowledge and similar patient cases retrieved by the system, and the final generated response in comparison to a reference answer.
Example of COD Text Generation
Medical Query
Patient: Hi doctor, I've been having really bad itching my skin, especially my arms and legs. And on top that, I've been feeling joint pain, mainly my knees and hands. It's been bothering me for few weeks now. I'm 55-year-old woman, if that helps.
Doctor: Have you been experiencing any specific symptoms like joint swelling or pain?
Patient: Yes, my joints are swollen and they hurt a lot.
Doctor: Have you noticed any redness around the affected area?
Patient: Yes, there is noticeable redness.
Reference Answer
Based on your symptoms, it's possible that you may have condition called erysipelas. This is a bacterial skin infection that can cause redness, swelling, and itching, especially in the arms and legs. can also lead to joint pain and can be triggered by activities that involve using your hands a lot. I recommend seeking medical attention soon as possible for proper diagnosis and treatment. This may involve course antibiotics and other supportive measures to help manage your symptoms and prevent any potential complications.
Retrieved Background Knowledge
Pine Caterpillar Disease Arthropathy: Pruritus, joint pain, swollen and painful joints, skin itching
Decompression Sickness:
Skin itching, limb pain, aseptic bone necrosis, paraplegia, urinary and fecal incontinence, joint pain, abdominal pain, headache, dermatomal contracture, increased pulmonary water content
Erysipelas:
Swelling, skin itching, joint pain, fever
Retrieved Similar Patients
ID: patient_ -28 (Similarity Score: 0.9615)
Dialogue:
Patient: Hi doctor, have these small, raised bumps my skin. They're painful, but they're 'r kind of annoying.
Doctor: Have you been experiencing any itchiness recently?
Patient: Yes, I've been feeling quite itchy.
Target Disease: Keratosis Pilaris
Answer: Based on the symptoms you' 've described, it sounds like you may have a condition called Keratosis Pilaris. This often presents small, raised bumps on the skin that can be itchy. recommend starting with some over-the-counter options, such as using gentle exfoliating scrub and moisturizing regularly. If these don't help, we can explore other treatment options, such as topical creams oral supplements. It's important to keep the skin well-moisturized and to avoid excessive scratching, as this can lead to irritation and potential infection. Let's monitor the condition and follow up needed.
ID: patient 311 (Similarity Score: 0.9455) Dialogue:
Patient: I've been experiencing this really intense itching all over my skin, doctor. It's been bothering me quite a bit.
Doctor: Have you noticed any bumps on your skin recently?
Patient: No, haven't tt noticed any bumps.
Target Disease: Demodicosis
33

---

Answer: It sounds like you may be experiencing symptoms of demodicosis, con- dition caused by the overpopulation of Demodex mites on the skin. would recom- mend avoiding environments infested with mites, bathing regularly, and using topical anti-inflammaorry and antipruritic medications. Additionally, if the itching persists, antihistamines or corticosteroids may be necessary. I would advise you to make an appointment with a dermatologist for a more accurate diagnosis and appropriate treatment.
ID: patient 765 (Similarity Score: 0.9298)
Dialogue:
Patient: Oh, hi doctor. I've been having this really annoying skin itch all over, especially in my groin area. It's been driving crazy and just can't seem to find any relief. Can you help me figure out what's 's going on?
Target Disease: Pruritus
Answer: It sounds like you are experiencing pruritus, which is a medical term for itching. Pruritus can be caused by variety of factors such as dry skin, allergic reactions, skin conditions, or underlying medical conditions. To effectively address your symptoms, we will need to identify and address the underlying cause. I would recommend starting with some lifestyle changes such as using mild, fragrance- cleansers and moisturizers, wearing loose clothing, and avoiding hot water when bathing. Additionally, topical treatments such as low pH cleansers, cooling agents, local anesthetics, and topical corticosteroids can provide relief. the itching persists, systemic treatments and physical therapy may also be considered. would advise scheduling an appointment for thorough evaluation to determine the best course of action for your specific situation.
DoctorRAG Generated Answer
It sounds like you may be experiencing condition involving inflammatory arthropathy with pruritic dermatitis. Based on your symptoms of persistent skin itching (especially on arms and legs), swollen and red painful joints in knees and hands, this could be indicative of **Pine Caterpillar Disease Arthropathy** ** or **Erysipela*** Both conditions present with pruritus, joint inflammation, and cutaneous symptoms, though Erysipelas typically involves fever, which you have not reported.
To clarify the diagnosis, we need to investigate potential exposures (e.g., contact with caterpillars or other allergens) and assess for systemic signs like fever or rash patterns. Initial recommendations include:
1. **Topical corticosteroids (e.g.. 1% hydrocortisone) to reduce skin inflammation and itching.
2. **NSAIDs** (e.g., ibuprofen) for joint pain and swelling.
3. ** Antihistamines (c.g., cetirizine) to address allergic or pruritic components.
Avoid scratching prevent secondary infection. If symptoms persist worsen, if fever develops, seek urgent evaluation for possible bacterial infection (e.g., Erysipelas) or referral dernnatoogii//hhennatooggg for specialized testing.
E.2 Examples for Med- TextGrad
Here we present the first iteration of Med-TextGrad's output for given medical query. This includes the initial patient query, the reference answer for context, the prompt used to guide the model, the initial generated answer, the refined answer after evaluation, and the critiques and gradient instructions for further improvement based on knowledge and patient-centric perspectives.
Example of Red-TextGrad (Iteration 1)
Medical Query
Patient: Hi doctor, I've been having some swelling in the infraorbital area and the mas- seter muscle area. I also have pain in the parotid gland area and swelling and pain in the submandibular area. It's been really uncomfortable and concerning.
Reference Answer
34

---

It sounds like you are experiencing swelling and pain in multiple areas of your face. Have you noticed any fever, difficulty in swallowing, or changes your voice?
Iteration 1
Prompt Template:
You are helpful medical consultation AL Your task is to REFINE the "current answer (A)' (which will be provided at the end this prompt) by critically evaluating it against the patient's query and the supporting context. The goal is improve the 'current answer (A)'s accuracy, completeness, and relevance to the patient's s query, while preserving its valid core information and avoiding unnecessary deviations from its original intent.
Patient Query (q): Patient: Hi doctor, I've been having some swelling in the infraorbital area and the masseter muscle the submandibular area. It's been really uncomfortable and concerning. Supporting Context (C):
also have pain in the parotid gland area and swelling and pain
n.w
General Medical Knowledge (Expertise)
Retrieved Patient Cases (Experience) Retrieved Patient Case (ID: patient 679): p
<...>
Retrieved Patient Case (ID: patient_ 107):
<...>
Retrieved Patient Case (ID) patient_ 110):
<..>
Instructions for refinement:
1. Carefully read the 'atteett Query (q)' and the 'Supporting Context (C)'.
2. Critically evaluate the 'current answer (A)' (provided below) against this information.
3. Generate improved and refined version of the 'current answer (A)'.
4. Focus on addressing any shortcomings in the 'current answer (A)' regarding accuracy, completeness, clarity, and direct relevance the patient's query.
5. Ensure your refined answer is factually sound based on the context, empathetic, and easy for patient understand.
6. IMPORTANT: Your output must be ONLY the refined medical answer itself. Do not include any preamble, conversational phrases, meta-commentary, any other than the refined answer.
Input Answer:
sounds like you may be experiencing submandibular space infection, given the swelling and pain in the infraorbital, masseter muscle, parotid gland, and submandibular areas. This con- dition can cause significant discomfort and may spread to adjacent regions if untreated. Have you noticed any fever, difficulty breathing swallowing, recent dental trauma/infection? These symptoms could help confirm the diagnosis or identify complications. Immediate medical evaluation is critical to assess the infection's 's extent and severity. Treatment typically or clindamycin) to address bacterial infec- tion, along with possible incision and drainage if abscess formation is present. Imaging, such as a CT scan, may be required to confirm involvement of deep spaces. Pain management with provide temporary relief. Prompt intervention is necessary
involves antibiotics (e.g., amoxiciliin-lavulane
NSAIDs acetaminophen prevent airway compromise or systemic spread. Please seek urgent care from an oral surgeon, ENT specialist, or emergency department for definitive management.
Context Critiques:
Factual Alignment:
Correctly identifies submandibular space infection and acute suppurative parotitis as potential diagnoses based on symptoms (swelling/pain in infraorbital, masseter, parotid, submandibular areas). Accurately mentions fever, difficulty swallowing/breathing concerning symptoms, consistent with the context.
35

---

Consistency:
Treatment suggestions (antibiotics, drainage, imaging) align with standard management for these infections.
Warnning about rapid progression and complications (e.g., airway obstruction) con- sistent with the severity described in the context.
Completeness:
Omits mention of infratemporal space infection, which could also present with swelling in temporal/masseter/parre regions (per context).
Does not explicitly differentiate between submandibular space infection and acute suppurative parotitis (e.g., high fever more typical of acute suppurative parotitis).
Lacks reference to loss of appetite (a symptom noted for acute suppurative parotitis in the context).
Potential Misrepresentation:
No major misrepresentation, but combining symptoms of distinction could lead to diagnostic ambiguity.
both conditions without clear
Patient Critiques:
Relevance: The answer directly addresses the patient's 's described symptoms (swelling
and pain in multiple facial areas) and suggests plausible diagnoses (submandibular space infection, acute suppurative parotitis), which aligns well with the query.
Appropriateness: The recommendation for immediate medical evaluation is appropriate given the potential severity of the conditions mentioned. The inclusion of concerning symptoms (fever, difficulty swallowing/breathing) helps the patient gauge urgency.
Specificity:
The answer could better clarify whether the symptoms described (c.g.. infraor- bital swelling) are typical for the suggested diagnoses or if additional conditions should be considered.
Patient Concerns: While the answer acknowledges discomfort and urgency, it could briefly reassure the patient that seeking care promptly improves outcomes, addressing their expressed concern ("really uncomfortable and concerning").
Treatment Specifics: Mentioning specific antibiotics (amoxieilinn-claulaanta and imaging (CT) is useful, but the patient might benefit from brief explanation of why these chosen (c.g., broad -read-spectrum coverage, abscess detection).
Actionability: Clear instructions (visit ED, oral surgeon, or ENT) are helpful, but empha- sizing the "why" (risk of rapid progression/airway obstruction) could further motivate the patient to act.
Gradient Answer (Context):
1. Add Infratemporal Space Infection to Differential Diagnosis:
Insert sentence mentioning infratemporal space infection another potential diagnosis, noting its presentation with swelling in temporal/massete//prrtti regions, referenced in the context.
2. Clarify Distinctions Between Diagnoses:
Differentiate submandibular space infection and acute suppurative parotitis by specifying that high fever is more characteristic of acute suppurative parotitis (per context).
Add loss of appetite as a symptom more typical of acute suppurative parotitis, noted in the context.
3. Maintain Existing Valid Information:
Retain all accurate details about concerning symptoms (fever, difficulty swallow- ing/breathing). treatment (antibiotics, drainage, imaging), and urgency (emer- gency evaluation).
4. Preserve Structure and Flow:
36

---

Keep the original warning about rapid progression and complications (e.g-- airway obstruction) but ensure the added distinctions between diagnoses are integrated smoothly.
5. Avoid Overlap or Ambiguity:
Ensure the revised text clearly separates the symptoms and diagnostic clues for each condition (submandibular space infection, acute suppurative parotitis, infratemporal space infection) to reduce ambiguity.
6. No Major Restructuring:
Do not alter the overall structure tone of the answer; focus only targeted additions and clarifications to address the critiques.
Gradient Answer (Patient):
1. Clarify Symptom-Diagnosis Alignment:
Add brief statement explaining that the described symptoms (infraorbital, masseter, parotid, and submandibular swelling/pain) are consistent with the suggested diagnoses (submandibular space infection or acute suppurative paroti- tis), but also note that other conditions (e.g., dental abscess, salivary gland obstruction) could present similarly.
Example addition: "While your symptoms align with these conditions, other causes like dental infections salivary gland blockages may also be considered during evaluation."
2. Address Patient Concerns with Reassurance:
Insert sentence acknowledging the patient's discomfort and concern, empha- sizing that prompt care improves outcomes.
Example addition: "I understand this is uncomfortable and worrying, but seek- ing care now can help prevent complications and speed recovery."
3. Explain Treatment Rationale:
Briefly justify the antibiotic choice (amoxicilin--clavllanie by mentioning its broad-spectrum coverage for oral bacteria.
Clarify that imaging (CT) helps identify abscesses or infection spread, guiding treatment decisions.
Example addition: "Amoxxeillin-caavllaae is often used because it targets bacteria common in oral infections. A CT scan helps determine if drainage is needed." "
4. Strengthen Actionability with Urgency:
Reinforce the "why" behind immediate action by explicitly linking rapid pro- gression to risks like airway obstruction. Example addition: "These infections worsen quickly, potentially affecting breathing, so don't 't delay evaluation."
5. Maintain Structure:
Keep the original flow (diagnoses - concerning symptoms -> treatment > action steps) but integrate the above revisions without redundancy.
6. Avoid Overloading:
Ensure additions are concise and do not overwhelm the patient with excessive detail. Foeus clarity and motivation to act.
Updated Prompt:
You are medical consultation AI tasked with refining the 'current answer (A)' improve its accuracy, completeness, and relevance to the patient's query while preserving valid core information. Follow these guidelines:
1. **Comprehensive Differential Diagnosis*
Include ALL relevant conditions from the supporting context (e.g.. submandibular space infection, acute suppurative parotitis, infratemporal space infection) and retain clinically justified differentials (c.g., dental abscess, salivary gland obstruction). Explicitly map key symptoms (e.g., fever, loss of appetite, swelling locations) to their most characteristic diagnoses using context-provide linkages. Provide detailed anatomical descriptions for
37

---

each condition (e.g., Swelling and pain the masseter muscle and parotid gland areafor submandibular space infection) and clarify extension patterns (c.g., infratemporal space infection may involve check, orbital, zygomatic regions). 2. **Dignnostic Distinctions**
Highlight distinguishing clinical features between similar conditions (e.g.. high fever in acute suppurative parotitis vs. submandibular infection). Avoid diagnostic overlap by clearly attributing symptoms to specific conditions (e-g., temporal/masseter swelling - infratemporal infection).
wwaimmttt--Daagnnon Alignment*
Clarify how the patient's 's symptoms align with suggested diagnoses while briefly mentioning other plausible conditions. Justify diagnostic/treatment choices by linking them directly to the patient's symptoms (e.g.. CT scans are prioritized for multi-area swelling to localize infection).
4. EEmotional Acknowledgment
Use explicit validation (e.g.. g.. This sounds really uncomfortable and concerning) to acknowl- edge patient distress. Maintain empathetic yet direct language, avoiding vague reassurances.
5. T*ertmment Rationale**
Justify treatment choices (e.g., antibiotics like amoxicilli--claullnate for oral bacteria) and tests (e.g.. CT for abscess detection) with simple, patient-friendl, explanations.
6. +>gggnncy Reinforcement**
Clearly state why immediate evaluation is critical, citing risks like airway obstruction or rapid infection progression. Confirm that localized swelling/pain in critical facial areas (e.g.,
infraorbital, submandibular) alone warrants urgent evaluation, even if systemic symptoms (fever, dysphagia) are absent. Include brief question clarify systemic symptoms (e.g.. Do you have fever, trouble swallowing, or difficulty breathing?) to refine urgency assessment. 7. **Structura Guidance*
Maintain the logical sequence: diagnoses - red flags + treatment - action. Integrate new diagnostic additions seamlessly without disrupting urgency warnings or treatment advice.
8. Preservation Clause**
Retain all valid existing content that aligns with clinical relevance, even not explicitly mentioned in the context. Prioritize anatomical precision and symptom-diagnosis alignment while avoiding unnecessary changes to accurate, clinically sound information.
Example Integration: For multi-area facial swelling, model responses like: 'Your symptoms in [X, regions] could indicate [A, B conditions]. Even without fever, this needs immediate care stop the infection from spreading.
Refined Answer:
Your symptoms, including swelling and pain in the infraorbital, masseter, parotid, and sub- mandibular areas, could indicate a submandibular space infection or acute suppurative parotitis. Key concerning symptoms would include fever, difficulty swallowing breathing, recent dental issues. Given the potential severity, immediate medical evaluation is essential. Treat- may involve antibiotics (like amoxxiillin-claullnattee is present, and imaging (CT scan) to assess the infection's is extent. Pain relief with NSAIDs acetaminophen can help temporarily. Please visit emergency department, oral surgeon, ENT specialist prompy, as these infections can progress rapidly and may lead to serious complications like airway obstruction.
possible drainage if abscess
E.3 Examples for pairwise comparison
This section presents pairwise comparisons different answers. These answers were evaluated from three perspectives: Comprehensiveness, Relevance, and Safety. Additionally, two human evaluators reviewed the final answers verify their correctness.
Example Ground Truth vs. Original Answer
Medical Query
Patient: Patient: Hi doctor, I've noticed some pigmentary abnormalities on my face. I have these facial spots and dark spots that are bothering me, and I'm a female.
38

---

Responses Compared
Response (Ground Truth):
It sounds like you may be experiencing melasma, which is a common skin condition characterized by the development of brown or grayish patches on the face. This can be exacerbated by sun exposure and hormonal changes. Have you noticed any triggers or changes in your routine that may have contributed to the development of these spots? Additionally, are you experiencing any other symptoms or changes in your overall health that you believe may be related to this condition? Response B (Original Answer of DoctorRAG):
sounds like you may be experiencing melasma, a common condition characterized by facial pigmented patches and dark spots, particularly in women. Potential con- tributing factors include hormonal changes (c.g., pregnancy, birth control) and sun exposure. Less likely, but worth considering, Multiple Lentigines Syndrome Congenital Vascular Sclerosing Dermal Melanosis, though these typically present with additional features like nail anomalies atrophic scars not mentioned here. To manage melasma, prioritize strict sun protection by applying proad-spectrum sunscreen (SPF 30+) daily and avoiding prolonged UV exposure. Topical treatments such as 3% hydroquinone cream (2-3 times daily) may help lighten pigmentation. Addressing hormonal influences, if applicable, critical- discuss contraceptive use, thyroid health, or pregnancy history with your dermatologist. Oral vitamin C or professional treatments like chemical peels could also be considered.
Have you noticed any hormonal changes, such as pregnancy or new medications? Are the spots symmetrical and concentrated the checks, forehead, jawline? dermatologist can confirm the diagnosis and tailor a treatment plan, especially if there's improvement with initial measures. Monitor for new lesions or changes in texture, which may indicate a different condition.
Structured Evaluation
Comprehensiveness: [Response B] [Response provides more detailed explanation of potential conditions (melasma, Multiple Lentigines Syndrome, Congenital Vascular Sclerosing Dermal Melanosis), contributing factors (hormonal changes, sun exposure), and management strategies (sun protection, topical treatments, hormonal evaluation). It also prompts further questions refine the diagnosis and suggests consulting a dermatologist. Response A is less comprehensive, focusing primarily on melasma and asking general follow-up questions without specific guidance.]
Relevance: [Response B] [Response directly addresses the patient's concern about facial pigmentary abnormalities, focusing the most likely diagnosis (melasma) while briefly mentioning rarer conditions for completeness. It provides actionable advice and asks targeted follow-up questions. Response A is relevant but less focused, with broader questions that may not efficiently narrow down the diagnosis.]
Safety: [Response B] [Response B adheres to clinical best practices by recommending evidence-based treatments (e.g., hydroquinone, sun protection) and emphasizing the need for professional evaluation. avoids harmful advice and acknowledges diagnostic uncer- tainty by suggesting monitoring for atypical features. Response lacks specific treatment recommendations or red flags, potentially leaving the patient without clear next steps.] Overall from evaluation: Response is superior in comprehensiveness, relevance, and safety, providing more structured and clinically sound answer to the patient's 's query.
Human examination: Passed. v
Example 2: Iteration vs. Iteration
Medical Query
Patient: Patient: Hi doctor, I've noticed some pigmentary abnormalities on my face. I have these facial spots and dark spots that are bothering me, and I'm a female. Responses Compared Response (Iteration 2):
39

---

understand these pigmentary changes can be concerning. The facial dark spots you describe are most consistent with melasma, common condition causing brown or grayish patches, typically on the checks, forehead, and jawline. This occurs more frequently in women and is often triggered by sun exposure, hormonal changes (like pregnancy or birth control use), or certain medications.
For management: 1. Sun protection crucial use broad-spectrum SPF 30+ sun- screen daily and reapply every 2 hours when outdoors. 2. Consider topical treatments like hydroquinone 3% cream (applied twice daily) or vitamin C serums. Oral vitamin C may also help in some cases. 3. Avoid potential triggers like hormonal medications if possible. 4. Maintain gentle skincare routine without harsh scrubs or irritants. Since melasma can be persistent, a dermatologist can confirm the diagnosis and offer stronger treatments like nigher-concentralte such as chemical peels laser therapy if needed.
hydroquinone, trctinoin, or procedures
Have you noticed the spots worsen with sun exposure or at certain times in your menstrual cycle? Also, you currently pregnant, breastfeeding, using hormonal medications?"
Response (Iteration 3):
understand how distressing these pigmentary changes can for your appearance and confidence. You described 'dark spots' cliniclli these could be melasma, a common condition causing brown or grayish patches, often on the cheeks, forehead, and jawline. While melasma is most likely, other conditions like post-inflammaorry hyperpigmentation or sunspots can present similarly. A dermatologist can confirm this diagnosis and rule other conditions.
*Possible C Frequently seen in women, often triggered by sun exposure, hormonal changes (e-g., pregnancy, birth control use), or certain medications. "Management (if confirmed)*** 1. **Sun protection** Use broad-spectrum SPF 30+ sunscreen daily and reapply every 2 hours when outdoors. **Topical treatments** **; Hydroquinone 3% cream (applied twice daily) or vitamin C serums may help. Oral vitamin C could also be considered Evaluating hormonal contributors, like medications imbalances, may be beneficial. ** Gentle **incare** Avoid harsh scrubs or irritants. 5. ** **onitoring**** Track treatment response and report any *To help narrow the cause**; Have you noticed the spots worsen with sun exposure at certain times in your menstrual cycle? Are you currently pregnant, breastfeeding, or using hormonal medications? Have you had prior skin trauma, acne, other symptoms like nail changes?
some cases. **romoaaal factors** **
changing symptoms.
Structured Evaluation
Comprehensiveness: [Response B] [Response B provides more detailed differential diagnosis (mentioning post-inflammatory hyperpigmentation and sunspots) and includes additional monitoring advice ("track treatment response and report any symptoms"). )) also asks more follow-up questions (e.g., prior skin trauma, nail changes), which could help refine the diagnosis. Response A is thorough but lacks these broader considerations.]
changing
Relevance: [Response A] [Both responses are highly relevant, but Response A is slightly more concise and focused on the most likely diagnosis (melasma) without introducing less probable conditions upfront. Response B, while comprehensive, includes slightly more peripheral information (e.g., nail changes) that may not as immediately relevant.]
Safety: [Tie] [Both responses are medically accurate, emphasize sun protection, recom- mend evidence- based treatments (e.g.. hydroquinone, vitamin C), and advise specialist referral for confirmation. Neither suggests unsafe practices or misleading advice. Response B's inclusion of "seek prompt care if spots change rapidly" adds safety layer, but this is implicitly covered in Response A's recommendation to consult a dermatologist.)
Overall from evaluation: Comprehensiveness: Response B wins for broader differentials and monitoring advice. Relevance: Response A wins for tighter focus on the core issue. Safety: Tit-both adhere clinical best practices.
Human examination: Passed. V
40


Reviews:
[{"review_id": "fzX54eoP4D", "reviewer": "Reviewer_wjXy", "summary": "This paper proposes DoctorRAG, a retrieval-augmented generation (RAG) system that seeks to emulate doctor-like clinical reasoning. The framework uses conceptual tagging, hybrid retrieval, and a multi-agent iterative optimization process (Med-TextGrad) to refine responses. Evaluations on multilingual medical tasks show consistent but sometimes modest improvements over baselines.", "questions": "How sensitive are the results to prompt choices and critique formulation in Med-TextGrad?\n\nWhat is the wall-clock inference time per query, and is the method scalable beyond academic benchmarks?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* Addresses a genuine gap: prior RAG systems for medicine largely ignore analogical patient reasoning, which is fundamental in clinical practice.\n\n* Modular retrieval pipeline using ICD-10 concept tagging is more principled than naive embedding search.\n\n* Empirical gains are shown across diverse tasks, languages, and LLM backbones, with some strong ablation evidence that each module helps.\n\n**Weaknesses**\n\n* The actual “gradient” refinement is essentially LLM prompt engineering, not a principled or reproducible optimization method; improvements may be dataset-specific or not robust.\n\n* Method is computationally heavy (dual retrieval, multiple LLM calls per answer). Additional discussion of inference cost, latency, or suitability for deployment would be helpful.\n\n* More details on the human eval would be helpful. What are the exact preference protocol? What exactly does it mean that the proposed method outperforms ground truth?", "was_revised": true, "final_justification": "After discussing with the authors during rebuttal, I have decided to keep my score.", "rebuttal": "**We sincerely thank the you for the thorough evaluation and constructive feedback. We address each concern below:**\n\n---\n\n### W1: The actual \"gradient\" refinement is essentially LLM prompt engineering, not a principled or reproducible optimization method\n\nWe acknowledge this important distinction. The \"textual gradients\" in Med-TextGrad are indeed structured prompt engineering rather than mathematical gradients, and we use this terminology as a conceptual analogy to gradient descent optimization. However, the approach offers several principled advantages over static prompt engineering. Unlike fixed prompts, Med-TextGrad generates context-specific refinement instructions based on the patient query, retrieved knowledge, and current answer quality, providing dynamic adaptation rather than one-size-fits-all solutions. The dual-criterion evaluation from both Context and Patient perspectives provides systematic feedback through multi-agent validation, and the iterative process follows a consistent algorithmic structure (**Algorithm 1**) with standardized prompts (**Appendix D**), ensuring reproducibility.\n\nRegarding robustness, our experiments demonstrate consistent improvements across diverse scenarios: 7 datasets spanning 3 languages, 4 medical tasks, and 4 different LLM backbones (**Table 2**). The pairwise comparison results (**Figure 4**) show systematic improvements in comprehensiveness, relevance, and safety across 50 samples, verified by human experts. While dataset-specific variations exist, the underlying multi-agent refinement principle appears generalizable within the medical domain.\n\n---\n\n### W2: Computational cost and deployment considerations\n\nWe appreciate this practical concern. The computational overhead is indeed significant due to dual retrieval and multiple LLM calls. However, several factors support deployment feasibility. **Figure 5** shows diminishing returns beyond k=4 retrieved items, allowing cost-performance optimization through careful parameter tuning. In medical applications, the accuracy and safety improvements demonstrated in **Figure 4** often justify additional computational cost compared to general-purpose applications, given the quality-critical nature of healthcare decisions. Additionally, our framework can leverage cloud-based LLM APIs without requiring local model deployment, reducing infrastructure barriers for healthcare institutions.\n\nFor practical deployment, we estimate that DoctorRAG inference takes 5-15 seconds per query depending on dataset complexity, while Med-TextGrad refinement adds approximately 30 seconds. These latencies are acceptable for clinical decision support scenarios where accuracy is prioritized over speed. Future work could explore parallel processing or early stopping mechanisms to reduce inference time while maintaining quality improvements.\n\n\n---\n\n### W3: Human evaluation details and ground truth comparison\n\nThe human evaluation protocol is detailed in **Appendix B.7 and D.3**. Two medical experts conducted pairwise comparisons across three dimensions (comprehensiveness, relevance, safety) using structured evaluation criteria. For each dimension, experts determined which response better addressed the patient's needs, with the overall score computed as the average across dimensions.\n\nRegarding outperforming \"ground truth\": The reference answers in our datasets are often context-specific clinical conversations or synthetically generated responses rather than gold-standard medical advice. Our evaluation shows that DoctorRAG generates more comprehensive, structured, and clinically informative responses compared to these reference answers. This suggests that real clinical interactions, while authentic, may not always represent the most optimal medical guidance that an AI system could provide when given access to comprehensive medical knowledge and similar patient cases.\n\n---\n\n### Q1: Sensitivity to prompt choices and critique formulation\n\nThis is an important robustness consideration. Our prompt design underwent iterative refinement based on medical expert feedback, and we provide detailed prompts in **Appendix D.1** for reproducibility. For example, for disease diagnosis in DialMed, we use the following prompts:\n> * **Input:**\n>     * Patient Symptoms: {evidences} (List of symptoms)\n>     * Knowledge Context: {knowledge\\_text} (Text containing relevant medical knowledge)\n>     * Similar Patients Information: {similar\\_patients\\_text} (Text describing similar patient cases)\n>     * Valid Disease Options: {VALID\\_DISEASES} (List of possible diseases to choose from)\n> * **Instructions within the prompt to the LLM:**\n>     As a medical diagnostic expert, predict the most likely disease for this patient.\n>\n>     Patient Information:\n>\n>     Symptoms: {evidences}\n>\n>     Relevant Information:\n>\n>     {knowledge\\_text}\n>\n>     {similar\\_patients\\_text}\n>\n>     Based on the above information, determine the most likely disease from the following options only:\n>\n>     {VALID\\_DISEASES}\n>\n>     Return only the name of the disease without any additional text.\n> * **Output format:** Only the name of the predicted disease.\n\nWhile prompt variations could affect performance, several factors suggest reasonable robustness: \n1. Structured evaluation criteria: The Context and Patient Criterion agents use standardized medical evaluation frameworks; \n2. Multi-dataset consistency: Similar improvement patterns across different datasets and languages suggest the approach is not overly sensitive to specific prompt formulations; \n3. Human expert validation: The fact that human experts consistently preferred refined answers indicates the optimization direction is clinically sound regardless of specific prompt variations.\n\nFuture work could systematically evaluate prompt sensitivity through ablation studies with different critique formulations.\n\n---\n\n### Q2: Wall-clock inference time and scalability\n\nAs mentioned above, DoctorRAG requires 5-15 seconds per query for basic inference, with Med-TextGrad refinement adding approximately 30 seconds. This performance profile is suitable for clinical decision support scenarios where accuracy takes precedence over real-time response. \n\nFor production deployment, the system could implement tiered processing: immediate responses using basic DoctorRAG, with optional Med-TextGrad refinement for complex cases requiring higher accuracy assurance.\n\n---\n\n**We thank the reviewer again for these thoughtful comments that have helped us clarify important aspects of our methodology. The concerns about computational efficiency, prompt sensitivity, and evaluation protocols are well-taken and will guide our future work toward more robust and practically deployable medical AI systems. We believe our comprehensive experimental validation across multiple datasets, languages, and evaluation dimensions demonstrates the promise of combining clinical expertise with case-based experience for improved medical reasoning.**"}, {"review_id": "eJr5nwChyq", "reviewer": "Reviewer_4kze", "summary": "In this paper, a RAG framework, namely DoctorRAG, is proposed to emulate doctor-like reasoning for the medical QA task via integrating both explicit clinical knowledge and implicit case-based experience. Additionally, the authors developed a Med-TextGrad module to iteratively update the prompts using a multi-agent textual gradient design. \nOverall, the paper offers novelty in model design, and the results on public datasets are promising.", "questions": "1. The authors may consider comparing the output with the real doctor's feedback.", "limitations": "The authors discussed the limitations in the additional materials.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The novelty in model design. The Med-TextGrad implements TextGrad for medical QA. Although the core idea of TextGrad (Yuksekgonul et al. 2025) is not novel, this paper enables TextGrad for the medical QA task.\n2. Validated on multiple datasets in multiple languages. This shows the generality of the proposed algorithm.\n3. The design of DoctorRAG enables retrieval from the Medical Knowledge Base and the Patient Base.\n\nWeaknesses:\n1. The updating process of the prompt is unclear. For example, the “Textual Gradient” does not provide a quantitative method to monitor convergence. \n2. Following 1., the updating process of the prompt is not analyzed. For example, case studies of the updating prompts are not provided. Case studies can be helpful for further research.\n3. The difference in the output between the DoctorRAG and the real doctor is not provided.", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank you for the thorough evaluation and constructive feedback. We address each concern below:**\n\n---\n\n### W1: The updating process of the prompt is unclear, and \"Textual Gradient\" does not provide a quantitative method to monitor convergence\n\nWe appreciate this important clarification request. The prompt updating process in Med-TextGrad follows a structured multi-agent workflow detailed in **Algorithm 1** and **Section 2.2**. The process works as follows: at each iteration $t$, the current prompt $P_t$ generates an answer $A_t$, which is then evaluated by Context Criterion and Patient Criterion agents to produce textual critiques. These critiques are converted into \"textual gradients\" - structured natural language instructions that specify how to improve the answer and subsequently the prompt.\n\nThe prompt update mechanism operates through a Textual Gradient Descent (TGD) step where an LLM synthesizes improvement suggestions from both Context and Patient gradients to produce the refined prompt $P_{t+1}$ (as shown in **Equation 7**). While we acknowledge that this process lacks traditional quantitative convergence metrics, we employ several qualitative indicators: (1) consistency in critique content across iterations, (2) minimal changes in generated answers between successive iterations, and (3) human expert assessment of answer quality stabilization. **Appendix E.2** provides a concrete example showing how prompts evolve from initial instructions to more refined, context-aware guidance through this iterative process.\n\nFor future work, we plan to develop quantitative convergence metrics such as semantic similarity scores between consecutive prompts or automated quality assessment scores to provide more principled stopping criteria.\n\n---\n\n### W2: Case studies of the updating prompts are not provided, which would be helpful for further research\n\nYou are absolutely right that detailed case studies would enhance understanding and reproducibility. **Appendix E.2** presents a complete case study showing the prompt evolution process across three iterations for a medical query about facial pigmentary abnormalities. **Appendix E.3** also presents some examples of different iterations of Med-TextGrad for comparison. These examples demonstrates how the initial general prompt (\"refine the current answer based on patient query and context\") evolves into increasingly specific and medically-informed instructions that address diagnostic differentiation, treatment rationale, and patient communication.\n\nSpecifically, the case study shows how Med-TextGrad progressively incorporates: (1) more comprehensive differential diagnosis considerations, (2) explicit symptom-diagnosis mapping instructions, (3) enhanced patient concern acknowledgment, and (4) clearer treatment justification requirements. Each iteration builds upon previous critiques to create more targeted and effective prompting strategies. This iterative refinement process illustrates how the \"textual gradients\" guide the system toward more clinically appropriate and patient-centered responses.\n\nWe plan to include additional case studies across different medical domains and query types in future work to further demonstrate the generalizability and systematic nature of the prompt evolution process.\n\n---\n\n### W3: The difference in output between DoctorRAG and real doctors is not provided\nThis is an excellent point about benchmarking against real clinical practice. While we did not directly compare against practicing physicians' responses, our human evaluation process involved two medical experts who assessed DoctorRAG outputs using clinical judgment and medical expertise (**Section 4.3** and **Appendix B.7, D.3, E.3**). These experts evaluated responses across three clinically relevant dimensions: comprehensiveness (thoroughness of medical information), relevance (appropriateness to patient concerns), and safety (adherence to clinical best practices).\n\nThe evaluation (**Figure 4**) revealed that DoctorRAG often generates more comprehensive and structured responses compared to the ground truth answers in our datasets, which primarily consist of real clinical conversations or synthesized responses. **Appendix E.3** provides detailed examples comparing DoctorRAG outputs with reference answers, showing how our system incorporates broader differential diagnoses, explicit treatment rationales, and more systematic patient guidance than typical clinical interactions.\n\nHowever, we acknowledge that direct comparison with real-time physician responses would strengthen our evaluation. Real clinical interactions often involve contextual factors (time constraints, patient rapport, follow-up scheduling) that our current evaluation doesn't capture. Future work should include physician-in-the-loop evaluations where practicing doctors review and compare their own responses with DoctorRAG outputs for the same clinical scenarios.\n\n---\n\n**We thank the reviewer for these insightful suggestions that highlight important areas for methodological clarity and clinical validation. The requests for quantitative convergence monitoring, detailed case studies, and physician comparison benchmarks will significantly enhance the rigor and practical applicability of our approach.**"}, {"review_id": "vWGq1ufOg7", "reviewer": "Reviewer_XvBX", "summary": "This paper introduces DoctorRAG, a novel retrieval-augmented generation framework for medical QA that aims to emulate a doctor’s reasoning process. The key idea is to fuse explicit medical knowledge (from clinical guidelines or textbooks) with implicit experiential knowledge drawn from similar patient cases. Moreover, a Med-TextGrad module is introduced to adjust and refine the final answer. The paper reports that DoctorRAG significantly outperforms prior RAG-based baselines on a variety of tasks. Extensive experiments across multiple medical tasks (disease diagnosis, QA, treatment recommendation, and clinical text generation) and languages (English, Chinese, French) demonstrate that integrating case-based reasoning with knowledge retrieval produces more accurate, relevant, and comprehensive answers than existing methods.", "questions": "1. Are the experiments in Table 2 conducted under fair conditions? Specifically, do all compared RAG models use the same knowledge bases, including both the Patient Base and Knowledge Base?\n2. In scenarios where retrieved knowledge entries and analogous patient cases provide different or even conflicting information, how does DoctorRAG’s answer synthesis handle it? Since you pull information from heterogeneous sources (which might not always agree), it would be useful to clarify whether the generator or the refinement step explicitly reconciles discrepancies (for instance, prioritizing authoritative guidelines over anecdotal case trends).\n3. How is the relevance or correctness of the retrieved reference content evaluated? Please clarify the metrics or criteria used to assess the quality of retrieval.", "limitations": "No. Specific suggestions can be found in the Weaknesses section above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. The work integrates medical RAG with case-based reasoning. By retrieving similar patient cases in addition to knowledge, DoctorRAG mimics how physicians draw on both codified guidelines and personal experience. This dual-source approach is highly relevant to clinical practice and addresses an important need for experience-grounded medical reasoning.\n2. The proposed Med-TextGrad module iteratively enhances the final output via a multi-agent textual gradient descent mechanism, helping to enforce alignment and correctness in complex medical scenarios.\n3. Extensive experiments are conducted across multiple medical tasks (disease diagnosis, QA, treatment recommendation, and clinical text generation) and languages (English, Chinese, French) to demonstrate the effectiveness of the proposed method.\n\n## Weaknesses\n1. The paper presents an aggregation of incremental improvements rather than a single fundamental innovation. The performance gains largely result from combining existing techniques rather than introducing a novel conceptual breakthrough.\n2. The performance improvements, while statistically significant, are modest. It remains unclear whether the gains are primarily attributable to Med-TextGrad or the integration of patient case retrieval. A dedicated ablation study isolating each component—particularly Med-TextGrad alone—would strengthen the empirical claims.\n3. The range of LLMs used in the experiments is somewhat limited. Incorporating more recent and competitive models, such as DeepSeek-R1 or Gemini, would help validate the general applicability of DoctorRAG.\n4. The source of the knowledge base data is insufficiently detailed. There is a potential concern regarding data leakage, as the patient case database may be constructed from the same data pool used for testing. This risks overestimating model performance due to data overlap.\n5. The paper lacks a thorough analysis comparing retrieval quality between DoctorRAG and baseline methods. Understanding how well each system retrieves relevant content would provide deeper insights into the effectiveness of the proposed retrieval strategy.\n6. The writing is at times unclear and lacks precision, particularly in technical sections.", "was_revised": true, "final_justification": "The paper has merits, though some methodological limitations warrant further discussion", "rebuttal": "**We sincerely thank you for your detailed review and the opportunity to clarify our contributions. We address each concern systematically below:**\n\n---\n\n### W1: The paper presents aggregation of incremental improvements rather than fundamental innovation\n\nWe respectfully disagree with this characterization. Our work introduces two fundamental innovations that represent novel conceptual breakthroughs rather than incremental improvements. **First**, we proposed the integration of **patient case retrieval** alongside medical knowledge retrieval, directly mimicking the clinical reasoning process where physicians combine textbook knowledge (expertise) with experiential insights from similar past cases (experience). This dual-retrieval paradigm is, to our knowledge, entirely novel in medical RAG systems and represents a fundamental shift from knowledge-only approaches to experience-integrated reasoning.\n\n**Second**, we introduce **Med-TextGrad**, the first application of textual gradient-based optimization in medical AI systems. While inspired by TextGrad, our adaptation specifically addresses medical domain challenges through dual-criterion evaluation (Context + Patient perspectives) and medically-informed prompt evolution. This represents a novel optimization paradigm for medical text generation that goes far beyond existing prompt engineering approaches.\n\nThe UMAP visualization (**Figure 3**) provides empirical validation that patients with similar conditions cluster meaningfully, supporting the theoretical foundation for case-based medical reasoning. This clustering behavior demonstrates that our patient retrieval mechanism captures clinically relevant similarities that complement formal medical knowledge.\n\n---\n\n### W2: Performance improvements are modest and component attribution unclear\n\nThe characterization of \"modest\" improvements requires contextualization within medical AI evaluation standards. Our improvements are both statistically significant and clinically meaningful: DDXPlus accuracy increased from 92.37% (best baseline) to 98.27% - a 5.9 percentage improvement representing substantial diagnostic accuracy gains. For medical applications where accuracy directly impacts patient safety, such improvements are highly significant.\n\nRegarding component attribution, we provide comprehensive ablation analysis across three different experimental settings: **(1) Table 5** isolates each DoctorRAG component (Patient Base, Knowledge Base, Concept Tagging, Declarative Statements), showing consistent performance degradation when any component is removed; **(2) Figure 4** specifically evaluates Med-TextGrad through pairwise comparison across iterations, demonstrating systematic improvements in comprehensiveness, relevance, and safety; **(3) Figure 5** analyzes performance scaling with retrieval depth, confirming optimal performance around k=4.\n\nThese ablation studies clearly demonstrate that both patient case integration and Med-TextGrad refinement contribute meaningfully to performance gains, with neither component being solely responsible for improvements.\n\n---\n\n### W3: Limited range of LLMs tested\n\nOur LLM selection strategy prioritizes **robustness validation** over exhaustive model comparison. We deliberately chose four representative models spanning different architectures and capabilities: DeepSeek-V3 (Chinese-focused), Qwen-3-32B (multilingual), GLM-4-Plus (general purpose), and GPT-4.1-mini (commercial state-of-art). This selection enables evaluation across three languages (Chinese, English, French) and four medical tasks, providing comprehensive evidence of DoctorRAG's generalizability.\n\nThe core contribution lies in the **methodological framework** rather than model-specific optimizations. DoctorRAG's dual-retrieval architecture and Med-TextGrad refinement are model-agnostic approaches that should generalize to newer LLMs. While testing additional models like DeepSeek-R1 or Gemini would be valuable, our current evaluation already demonstrates consistent improvements across diverse model architectures, strongly supporting the framework's general applicability.\n\n---\n\n### W4: Knowledge base data sources and potential data leakage\n\nWe provide detailed data source documentation in **Section 3.1** and **Appendix A.1**. Regarding data leakage concerns, we implement rigorous separation protocols: for each dataset, approximately 80% of patient records form the Patient Base while 20% constitute the evaluation set. **Critically, we ensure that for any evaluation sample, its corresponding patient record and highly similar records (similarity > 0.99) are strictly excluded from the Patient Base during evaluation**.\n\nThis separation strategy eliminates data leakage while maintaining realistic clinical scenarios where physicians draw upon experience from different patients with similar conditions. **Table 6** in **Appendix A.1** provides direct links to all dataset repositories for full transparency and reproducibility verification.\n\n---\n\n### W5: Insufficient retrieval quality analysis compared to baseline methods\n\nThe retrieval quality comparison is embedded throughout our evaluation framework. **Appendix E** provides detailed examples demonstrating how DoctorRAG's concept-constrained retrieval (ICD-10 matching + cosine similarity) identifies more clinically relevant content than baseline semantic-only approaches. The systematic performance improvements across all tasks (**Table 2**) indirectly validate retrieval quality, as poor retrieval would manifest as degraded generation quality.\n\nMoreover, our dual-retrieval approach is fundamentally different from baseline methods - comparing retrieval quality directly would be methodologically inappropriate since baselines lack patient case retrieval entirely. The relevant comparison is whether our enhanced retrieval translates to better medical responses, which our comprehensive evaluation clearly demonstrates.\n\n---\n\n### W6: Writing clarity and precision in technical sections\n\nWe appreciate this feedback and have conducted thorough revision of technical sections. If specific unclear passages remain, we welcome concrete suggestions for improvement. We believe **Section 2** provides precise mathematical formulations (Equations 1-7) and **Algorithm 1** offers clear procedural specifications for reproducibility.\n\n---\n\n### Q1: Fair experimental conditions for Table 2 comparisons\n\nAll baseline methods use identical knowledge bases, embedding models (OpenAI text-embedding-3-large), and evaluation protocols. The key difference is that baseline RAG methods cannot access Patient Base information because patient case retrieval is our novel contribution. This represents a fair comparison of methodological capabilities rather than dataset advantages. Baseline methods could theoretically incorporate similar patient retrieval mechanisms, making this a legitimate architectural comparison.\n\n---\n\n### Q2: Handling conflicting information between knowledge sources and patient cases\n\nThis is an excellent methodological question. Our framework addresses potential conflicts through several mechanisms: **(1) Source prioritization**: We systematically place authoritative medical knowledge before patient case information in prompt construction (**Equation 3**), establishing clear precedence for clinical guidelines; **(2) Complementary information types**: Medical knowledge provides theoretical foundations while patient cases offer practical application patterns - these typically complement rather than conflict; **(3) Med-TextGrad reconciliation**: The iterative refinement process explicitly evaluates consistency through Context Criterion agents, flagging and resolving discrepancies between sources.\n\nIn practice, we observe minimal conflicts because retrieved patient cases undergo similarity-based selection, typically aligning with medical knowledge recommendations for similar conditions.\n\n---\n\n### Q3: Evaluation metrics for retrieval relevance and correctness\n\nRetrieval quality is evaluated indirectly through downstream task performance, which represents the most meaningful assessment for medical applications. **Direct evaluation approaches**: **(1) Human expert assessment** in our pairwise comparisons (**Figure 4**) inherently evaluates whether retrieved content supports medically sound responses; **(2) Ablation studies** (**Table 5**) demonstrate that each retrieval component contributes to improved performance; **(3) Consistency analysis** through Med-TextGrad iterations reveals whether retrieved content enables coherent, medically accurate responses.\n\nIf retrieval quality were poor, we would observe degraded performance rather than the consistent improvements demonstrated across all evaluation metrics.\n\n---\n\n**We thank the reviewer again for these detailed and thoughtful comments that have helped us clarify important methodological aspects of our work. Your concerns about fundamental innovation, component attribution, and experimental rigor are well-taken and have guided us to provide more comprehensive justification for our approach. We believe our responses address the core concerns while demonstrating the novel contributions of patient case integration and iterative refinement mechanisms in medical AI systems. We are committed to continuing this line of research to develop more robust and clinically validated medical reasoning systems.**"}, {"review_id": "itlXztV3Zq", "reviewer": "Reviewer_5hYj", "summary": "The paper introduces DoctorRAG, a medical Retrieval-Augmented Generation framework that mimics doctor-like reasoning by integrating both explicit medical knowledge and experiential knowledge from similar patient cases. It enhances retrieval through conceptual tagging and declarative sentence transformation, and refines responses using Med-TextGrad, a multi-agent textual gradient optimization process. DoctorRAG achieves superior performance across multilingual and multitask benchmarks—including diagnosis, treatment recommendation, and QA—compared to strong RAG baselines.", "questions": "What is the actual computational cost of Med-TextGrad per sample? How does it scale with the number of iterations and context size? And is it a fair comparison between the proposed method and baselines in terms of computational cost?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\\\n(1) Integrates both medical knowledge bases and similar patient case data, emulating real-world clinical reasoning.\\\n(2) Introduces a novel multi-agent, textual gradient-based optimization method to iteratively improve answer quality in relevance, accuracy, and faithfulness.\\\n(3) Demonstrates strong performance across diverse medical tasks (diagnosis, QA, treatment, generation) in Chinese, English, and French.\\\n(4) Outperforms strong baselines on multiple metrics and shows consistent answer refinement across iterations.\\\n(5) Code and data are shared.\n\nWeaknesses:\\\n(1) While the Med-TextGrad mechanism is inspired by TextGrad, the use of natural language \"gradients\" remains heuristic and lacks formal justification or mathematical soundness as optimization steps. The analogy to backpropagation is metaphorical, not rigorous.\\\n(2) Med-TextGrad introduces significant computational overhead by running multiple LLM calls per iteration. This limits scalability and practicality in real-time or resource-constrained clinical settings.\\\n(3) The iterative process is stopped after a fixed number of steps (T=3), without a robust convergence or validation criterion. There is no theoretical or empirical justification for this choice.", "was_revised": true, "final_justification": "The authors acknowledged my concerns regarding the limitations of the proposed method. These concerns are not addressed. However, they are not fundamental flaws, and this paper has merits that cannot be overlooked. Thus, I keep my original positive review to this paper.", "rebuttal": "**We sincerely thank you for the thorough evaluation and constructive feedback. We address each concern below:**\n\n---\n\n### W1: The use of natural language \"gradients\" remains heuristic and lacks formal mathematical soundness\n\nWe fully acknowledge this important distinction. The \"textual gradients\" in Med-TextGrad are indeed metaphorical rather than mathematically rigorous, representing structured natural language feedback that guides iterative improvement rather than formal optimization steps. We use this terminology as a conceptual analogy to help readers understand the systematic refinement process, but recognize it lacks the mathematical foundations of traditional gradient-based optimization.\n\nHowever, while heuristic, the approach follows principled design choices that provide systematic benefits. The multi-agent architecture ensures consistent evaluation criteria across iterations, with Context and Patient Criterion agents applying standardized medical assessment frameworks. The structured prompt evolution process (**Algorithm 1**) provides reproducible refinement steps, and our empirical validation across diverse medical scenarios (**Table 2**, **Figure 4**) demonstrates consistent improvements in clinically relevant metrics. Human expert evaluation further validates that the optimization direction aligns with medical best practices, suggesting that while not mathematically grounded, the heuristic approach captures meaningful improvement patterns for medical text generation.\n\n---\n\n### W2: Med-TextGrad introduces significant computational overhead limiting scalability in clinical settings\n\nThis is a valid practical concern. Med-TextGrad does introduce substantial computational overhead through multiple LLM calls per iteration (approximately 6-8 calls per iteration including critiques, gradients, and prompt updates). However, several factors support practical deployment feasibility. Basic DoctorRAG inference has comparable computational requirements to standard RAG methods, making it suitable for routine deployment. Med-TextGrad refinement can be selectively applied based on query complexity, clinical criticality, or resource availability rather than universally.\n\nFor resource management, we propose tiered processing strategies: immediate basic DoctorRAG responses for routine queries, with Med-TextGrad refinement reserved for complex diagnostic cases or high-stakes clinical decisions where accuracy justifies additional computational cost. Additionally, the diminishing returns observed after T=2-3 iterations (**Figure 4**) suggest that even limited refinement provides substantial quality improvements, allowing flexible cost-performance trade-offs based on clinical context and resource constraints.\n\n---\n\n### W3: The iterative process stops after fixed T=3 steps without robust convergence criteria\n\nYou raise an excellent point about the lack of principled stopping criteria. Our choice of T=3 was empirically determined through preliminary experiments on RJUA and COD datasets, where we observed performance plateaus after 2-3 iterations, with minimal additional improvements beyond this point. **Figure 4** demonstrates this convergence pattern, showing that iteration 2 (T2) often achieves optimal performance, with iteration 3 (T3) providing marginal or sometimes diminishing returns.\n\nHowever, we acknowledge that this fixed stopping criterion is suboptimal and dataset-dependent. Ideally, convergence should be determined through dynamic criteria such as: semantic similarity between consecutive prompts falling below a threshold or critique content stabilization across iterations. The current fixed T=3 approach represents a practical compromise between performance gains and computational efficiency, but future work should develop principled convergence detection mechanisms that can adapt to different medical domains and query complexities.\n\n---\n\n### Q: Computational cost details, scaling behavior, and fair comparison with baselines\n\nMed-TextGrad's computational cost scales as follows: each iteration requires approximately 3,000 tokens (input + output) across all LLM calls, scaling linearly with iteration count. Context size increases affect each iteration proportionally, so larger retrieved knowledge bases multiply the per-iteration cost. For a typical 3-iteration refinement with moderate context (k=4), total additional cost is approximately 9,000-12,000 tokens beyond basic DoctorRAG inference.\n\nThe comparison with baselines remains fair because all methods use identical knowledge bases, patient databases, and retrieval mechanisms. The computational difference lies solely in the post-generation refinement process, which represents an optional enhancement rather than a fundamental architectural change or any extra knowledge. Baseline methods could theoretically incorporate similar multi-iteration approaches, making the comparison methodologically sound. Additionally, we report results both with and without Med-TextGrad refinement, allowing readers to assess the cost-benefit trade-offs for their specific deployment scenarios.\n\nFor practical deployment, the cost scaling suggests that Med-TextGrad is most valuable for complex queries where the quality improvements justify the 3-4x computational overhead compared to single-pass generation.\n\n---\n\n**We thank the reviewer for these precise technical concerns that highlight important limitations in our current approach. While our current approach shows empirical promise, developing more rigorous optimization frameworks and adaptive stopping criteria represents important directions for advancing this line of research.**"}]

Author rebuttal:

Meta-review: This paper presents DoctorRAG, a novel RAG framework that emulates clinical reasoning by retrieving from both formal medical knowledge and a database of real-world patient cases. This dual-retrieval approach is the paper's core strength, and its effectiveness is convincingly demonstrated through extensive, generalizable experiments. The framework is further enhanced by Med-TextGrad, an iterative refinement module that improves answer quality.

The primary weaknesses are the heuristic nature of the Med-TextGrad module and its significant computational overhead, which the authors frame as a reasonable trade-off for improved accuracy.

The decision to accept is based on the paper's novel and impactful contribution, supported by strong empirical evidence. During a highly effective rebuttal, the authors thoroughly addressed reviewer concerns about originality and evaluation. Critically, they provided new experimental results and a new human evaluation to directly resolve a key reviewer's concerns, solidifying a clear consensus for acceptance. The paper is a technically solid contribution, and the authors' engagement in the review process was exemplary.
Decision: Accept (poster)