MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making
Authors: Yubin Kim
Chanwoo Park
Hyewon Jeong
Yik Siu Chan
Xuhai Xu
Daniel McDuff
Hyeonhoon Lee
Marzyeh Ghassemi
Cynthia Breazeal
Hae Won Park
Venue: NeurIPS 2024 oral (2024)
Keywords: Medical Decision Making
Multi-Agent Collaboration
Primary area: machine_learning_for_healthcare
TL;DR: MDAgents, a framework that adapts the collaboration of LLMs for complex medical decision-making, improving performance on major medical benchmarks

Abstract: Foundation models are becoming valuable tools in medicine. Yet despite their promise, the best way to leverage Large Language Models (LLMs) in complex medical tasks remains an open question. We introduce a novel multi-agent framework, named **M**edical **D**ecision-making **Agents** (**MDAgents**) that helps to address this gap by automatically assigning a collaboration structure to a team of LLMs. The assigned solo or group collaboration structure is tailored to the medical task at hand, a simple emulation inspired by the way real-world medical decision-making processes are adapted to tasks of different complexities. We evaluate our framework and baseline methods using state-of-the-art LLMs across a suite of real-world medical knowledge and clinical diagnosis benchmarks, including a comparison of
LLMs’ medical complexity classification against human physicians. MDAgents achieved the **best performance in seven out of ten** benchmarks on tasks requiring an understanding of medical knowledge and multi-modal reasoning, showing a significant **improvement of up to 4.2\%** ($p$ < 0.05) compared to previous methods' best performances. Ablation studies reveal that MDAgents effectively determines medical complexity to optimize for efficiency and accuracy across diverse medical tasks. Notably, the combination of moderator review and external medical knowledge in group collaboration resulted in an average accuracy **improvement of 11.8\%**. Our code can be found at https://github.com/mitmedialab/MDAgents.

gDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making
Yubin Kim
Chanwoo Park
Hyewon Jeong
Yik Siu Chan
Xuhai Xu² Daniel McDuff
Hyeonhoon Lee
Marzyeh Ghassemi
Cynthia Breazeal- Hae Won Park2
Massachusetts Institute of Technology
Google Research
"Sooul National University Hospital
{ybkim95, cpark97, hyewonj
ppyro9,, y yiksiuc, xoxu, mghassem, cynthiab, haewon)@mit......... edu dmcduff com com
hhoon@snu. ac. ac.
Abstract
Foundation models are becoming valuable tools in medicine. Yet despite their promise, the best way leverage Large Language Models (LLLMs) in complex medical tasks remains an open question. We introduce novel multi-agent frame- work, named Medical this gap by automatically assigning a collaboration structure to a team of LLMs. The assigned solo or group collaboration structure tailored to the medical task hand, simple emulation inspired by the way real-world medical decision- processes are adapted tasks of different complexities. We evaluate our frame- work and baseline methods using state-of-the-as LLMs across a suite of real-world medical knowledge and medical diagnosis benchmarks, including a comparison of LLMs' medical complexity classification against human physicians MDAgents achieved the best performance in seven out of ten benchmarks on tasks requiring understanding of medical knowledge and multi-modal reasoning, showing sig- nificant improvement of up to 4.2% % (p < 0.05) compared to previous methods' best performances. Ablation studies reveal that MDAgents effectively determines medi- cal complexity to optimize for efficiency and accuracy across diverse medical tasks. Notably, the combination of moderator review and external medical knowledge in group collaboration resulted in an average accuracy improvement of 11.8% %. Our code can be found /https://ggiiie com http://ttiiiiilllaa/aaattttt
crtision-making Agents (MDAgents) that helps address
Introduction
Medical Decision-Maken Makinng (MDM) a multifaceted and intricate process in which clinicians collabo- ratively navigate diverse sources of information to reach precise and specific conclusion (97]. For instance, a primary care physician (PCP) may refer a patient to specialist when faced with a complex case, or patient visiting the emergency department urgent care might be triaged and then directed to specialist for further evaluation [ (4). MDM involves interpreting complex and multi-modal data, such imaging, electronic health records (EHR), physiological signals, and genetic informa- tion, while rapidly integrating new medical research into clinical practice
68 78) Recently, Large
Language Models (LLMs) have shown potential for Al support in MDM 22
37 48
64
72 [0] It
known that they are able to process and synthesize large volumes of medical literature 74 and
clinical information 11. well support probabilistic [94] and causal (3) reasoning, makes L.MMs
promising tools. However, there is no silver bullet in medical applications that require careful design.
Hyewon Jeong received her MD degree from Yonsei University College of Medicine, South Korca. AAppendi F contains detailed comparison results between human physicians and L.Mss.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Primary Clinkcaan (PCC)
Knowledge
A fi-yean-s Caussaan main premarti to your officn with hypcsigmented undergoes
Low
the town arsonca scccytee in exitemm, Milcwing meat diagnesta? Three versicator Di Abbitun C. Willige Solar go Melarom
Ans
Promptting
Difiernniaa Diagnosis (DOx))
Multidiaciplinary Team (MDT)
Sex M, Agu: Ovoggaa apricaa riglon North Amnriiaa Pottiology: PSVT
Moderate
Symppoom
patn. pain hugging Ruuming.....
Ans
ElPeeettil daggooee:
log
0.11, Anaptotads 0.11, Cluster hesclctfr 6o Crages: D.7, Sountmidd food palkniigg 0.07, HV intect
PSVT 222, Anarta: S. 10 Phanic sttad:: D. Arta aort
LLM
Collaborative Discussion
M-rounds
Checker
Integrated Care Team
Wher does the didde in image D sumound?
Abnormal mitolic Rggrrs
High
Ans
Central bemiiiaattonn
Haport
Report
Frank
Team
Team 2
Team
Areas mecrodx
Report Generation
Medical Query
Complexity Check
Recruitment
Analysts and Synthesis
4. Final Decision
Figure 1: Medical Decision-Making Agents (MDAgents) framework. Given a medical query from different medical datasets, the framework performs 1) medical complexity check, 2) recruitment, 3) analysis and synthesis, and 4) decision-making steps.
86 have shown promise in non- 62 65), their evaluation in health applications has been limited. To date, their "generalist" design has not effectively integrated the real-world systematic MDM process [57] which requires an adaptive, collaborative, and tiered approach. Clinicians consider the current and past history of the patient, available evidence from medical literature, and their domain expertise and experience [20] for MDM. One example of MDM is triage patients in emergency room based on the severity and complexity of their medical conditions [22 26 87]. Patients with pathognomonic, single uncomplicated acute conditions, or stable chronic conditions that their PCP could manage [85] could be low complexity cases. On the other hand, patients with injuries that involve multiple organs, chronic conditions with side effects, or superimposed diseases who often require multiple collaborative discussions (MDT) or sequential consultations (ICT) among specialty physicians [27 61) are considered high complexity cases Fl
While decision-making tools including multi-agent LLMs
medical domains [31 32 44
46
Inspired by the way that clinicians make decisions in practice, we propose Medical ppeissonnmaking framework that leverages LLMs to emulate the hierarchical diagnosis procedures ranging from individual clinicians to collaborative clinician teams (Figure []. MDAgents work in three steps: 1) Medical complexity check; 2) Recruitment based medical complexity; 3) Analysis and synthesis and 4) Final decision-making to return the answer. Our contributions are threefold:
Agents (MDAgents), an adaptive medical decision-makinn
1. We introduce MDAgents, the first adaptive decision-make framework for LLMs that mirrors real-world MDM processes via dynamic collaboration among Al agents based on task complexity. 2. MDAgents demonstrate superior performance in accuracy over previous solo and group methods on 7 out of 10 medical benchmarks, and we show an effective trade-off between performance and efficiency (i.e. the number of API calls) by varying the number of agents.
3. We provide rigorous testing under various hyperparameters (e.g. temperatures), demonstrating better robustness of MDAgents compared to solo and group methods. Furthermore, our ablations evidence MDAggntts' ability to find the appropriate complexity level for each MDM instance.
2 Related Works
LLMs have shown promise in a range of ap- 76 90 96). They can answer questions from medical exams [43 [2]. perform biomedical research 36), clinical risk prediction [37]. and clinical diagnosis (55 67 Medical L.MSs also evaluated on generative tasks, including creating
Language Models in Medical Decision-Making
plications within the medical field [14 37
40
48
63
75
medical reports (7)), describing medical images [81] constructing differentials (53), performing
diagnostic dialogue with patients [77], and generating psychiatric evaluations of interviews 24]. To
advance the capabilities of medical LLMs, two main approaches have been explored: (1) training with
Detailed examples of low-, moderate- and high-complexity cases provided in Appendix E.1

---

Table Comparison between our framework and previous methods (Solo and Group), Among these works, MDAgents is the only one to perform all key dimensions of LLM decision Decision-making.
MDAgents (Ours)
Method
Single
Voting
3
Debate
17
MedAgents
7
ReConcile To
Interaction Type
Multiple Roles
Early Stopping
Refinement
Complexity Check
X
Multi-party Chat
x
Conversation Pattern
Flexible
Static
Static
Static
Static
Static
domain-specific data (28), and (2) applying inference-time techniques such as prompt engineering [67] and Retrieval Augmented Generation (RAG) 192]. While initial research has been concentrated on pre-training and fine-tunie tinettniig with medical knowledge, the rise of large general general-purpose purpose LLMs has enabled training- frree methods where models leverage their latent medical knowledge. For example, GPT-4 66). with richer prompt crafting, surpasses the passing score on USMLE by over 20 points and with prompt tuning can outperform fine-tuned models including Med-Pal.M (58 59 The promise of general-purposs models has thus inspired various techniques such as Medprompt and ensemble refinement to improve LLM reasoning [67]. well as RAG tools that use external resources to improve the factuality and completeness of LLM responses 38 92]. Frameworks like MEDIQ 3 advance LLM reliability in clinical settings by enhancing information-sseeknng and uncertainty reduction, supporting more realistic diagnostic processes. Our approach leverages these techniques and the capabilities of ceneral-purposs models while acknowledging that a solitary LLM [37 [8] (90] may not fully encapsulate the collaborative and multidisciplinaty nature of real-world MDM. We thus emphasize joinging multiple expert LLMs for effective collaboration in order to solve complicated medical tasks with greater accuracy.
[49] and UoT
through adaptive
Multi-Agent Collaboration An array of studies have explored effective collaboration frameworks between multiple LLM agents (47 86 to enhance capability above and beyond an individual LLM [0]. A common framework is role-playing, where each agent adopts a specific role (e.g. Assistant Agent or a Manager Agent), a task is then broken down into sub-steps and solved collaboratively 47 86). While role-playing focuses collaboration and multi-step mooleen-solving (88), another framework, "multi-agent debate", prompts each agent to solve the task independently [17.. Then, they reason through other agents' answers to converge on shared response, this approach can improve the factuality, mathematical ability and reasoning capabilities of the multi-agent solution [7 [5]. Similar frameworks include voting (22). multi-disciplinn
collaboration (72), group discussions (ReConcile [10), and negotiating 23 Table [ compares existing setups across key dimensions
in multi-agen interaction. Although these frameworks have shown improvement in the respective tasks, they rely on a pre-determined number of agents and interaction settings. When applied on a wider variety of tasks, this static architecture may lead suboptimal multi-agent configurations, negatively impacting performance [5T. Furthermore, multi-agent approaches run the risk of being computationally inefficient or expensive to employ and need to justify these costs with noticable performance gains [17). Given that different models and frameworks could generalize better to different tasks (93, we propose framework that dynamically assigns the optimal collaboration strategy at inference time based on the complexity of the query. We apply our strategy to MDM, a task that requires teamwork and should benefit from multi aggnt collaboration 72]
3 MDAgentss Medical Decision-making Agents
The design of MDAgents (Figures I and 2 incorporates four stages: 1) Medical Complexity Check The system evaluates the medical query, categorizing it as low, moderate, or high complexity based on clinical dicisiion-mking techniques 7
71 84 2) Expert Recruitment Based on complexity, the framework activates single Primary Care Clinician (PCC) for low complexity issues, or Multi-disciplinary Team (MDT) or Integrated Care Team (ICT) for moderate or high complexities
21
7 18 34 45 71). 3) Analysis and Synthesis Solo queries use prompting techniques like Chain-of- Thought (CoT)) and Self-Consistence (SC). MDTs involve multiple L.LM agents forming consensus, while ICTs synthesize information for the most complex cases. 4) Decision-making The final stage synthesizes all inputs to provide a well-informed answer to the medical query.
3

---

Step Complexity Check
Step 2. Expert Recraltment
Step Assessment
Question: What the of Groes Sectionad Myakolpoma domonstrate?
Moderute
High
[1] Answer: (A) fibrosts
(Al florosis
Reason
Pathologga (4)
(2) Answer (A) fibrosts
(B) information
[C) hamorrhage
pathology might
While the fientiiiation of gross
Radiologist (2)
Endocrinstoist (5)
Areower hamonhage
(D) caloticction
zraighttloword to specialist pathology, the distinction betweon firosis, hemorhhage and other conditios sometimes be and subject intertraatiio....
Moderstor
Answer. hemorthage
(5) Answer: (A) fibroats
Oncologat
(4)
Step 4, Collaborative Discussion
Step S. Reviow and Final Decision
GP Moderator
expertio Agert |Pathologler & jin
their orguncnt for (A) fltrosis is noted. the gross pothologial festures desorloed by Agent 3 (Burgeon 8 and Agent (Onoologist R seer to align more dionely with tysccal presentation twmorttage
henontaege
Terminate
documonted compliction mysiclipmmss partouuaryy larger onee or those subjected
trsuma.
Therofore, based analysls conversetion history, the final docsion skgret with oraepintaaion (C) hemontage (ned smow) srgued by Agann. (Burgeon Round Tumn and Agent (Oncologist @ Round 2. condiusion supperied by the grose pathological feotu consistent hamanhagg neher than loreass deople the nessomed oprnicrs, for firrass by Agaents (Patooogge S). (Radivingee and (fndosttooogii
Communcction channel
Round E (1. 8)
Figure 2: Illustrative example of MDAgents in a moderate complexity case from the PMC-VQA
dataset. More detailed case studies can be found in Figure 11 and 12 in the Appendix.
3.1 Agent Roles
Moderator. The moderator agent functions general practitioner (GP) emergency department doctor who first triages the medical query. This agent assesses the complexity of the problem and determines whether should be handled by single agent, MDT, or an ICT. The moderator ensures the appropriate pathway be selected based on the query's 's complexity and oversees the entire decision-making process.
Recruiter. The recruiter agent is responsible for assembling the appropriate team of specialist agents based on the complexity assessment of the moderator. The recruiter may assign a single PCP agent for low-complexity cases, while MDT ICT with relevant expertise will be formed for moderate and high-complexity cases.
General Doctor/Speciale
These agents are domain-specifi or general physicians recruited by the recruiter agent. Depending on the complexity of the case, they may work independently or as part of team. General physicians handle less complex, routine cases, whereas specialists are recruited for their specific expertise in more complex scenarios. These agents engage in the collaborative decision-makin process, contributing their specialized knowledge to reach consensus provide detailed reports for high-complexity cases.
3.2 Medical Complexity Classification (Line of Algorithm Appendix C.2
The first step in the MDAgents framework is to determine the complexity of a given medical query by the moderator LLM which functions generalist practitioner (GP). The moderator aims to act as a classifier to return the complexity level of the given medical query, is provided with the information on how medical complexity should be defined and is instructed to classify the query into one of three different complexity levels:
1. Low-Simme -imple,, well-defined medical issues that can be resolved by a single PCP agent. These typically include common, acute illnesses or stable chronic conditions where the medical needs are predictable and require minimal interdisciplinary coordination.
2. Moderate - The medical issues involve multiple interacting factors, necessitating a collaborative approach among an MDT. These scenarios require integration of diverse medical knowledge areas and coordination between specialists through consultation to develop effective care strategies. 3. High Complex medical scenarios that demand extensive coordination and combined expertise from an ICT. These cases often involve multiple chronic conditions, complicated surgical or trauma cases, and decision-makinn makings that integrates specialists from different healthcare departments.
3.3 Expert Recruitment (Line 3, 7, 17 of Algorithm I
Given a medical query, the goal of the recruiter LLM is enlist domain experts individuals, in groups, or as multiple teams, based on the complexity levels determined by the moderator LLM. Specifically, we assign medical expertise and roles to multiple LLMs, instructing them either act independently as solo medical agents or collaborate with other medical experts in a team. In Figure D in the Appendix, we also provide frequently recruited agents for each benchmark as a reference.

---

3,4 Medical Collaboration and Refinement
The initial assessment protocol of our decision-making framework categorizes query complexity into low, moderate, and high. This categorization grounded in established medical constructs such as acuity [25] for straightforward cases, comorbidity [69] and case management complexity [13] for intermediate and multi-disciplinary care requirements, and severity of illness [16] for high complexity cases requiring comprehensive management. We outlines the specific refinement approach:
Low Straightforward cases (Line 2-4 of Algorithm||] I For queries classified under Low com- plexity, characterized by straightforward clinical decision pathways, a single PCP agent (Figure TO (a)) is deployed by the definition in Section 3.2] The domain expert who is recruited by recuriter LLM, applies few-shot prompting to the problem. The output answer, denoted as ans, directly obtained from the agent's response to Q without the need for iterative refinement, formalized as ans = AQ) with Agent representing the engaged PCP agent.
Moderate Intermediate complexity cases (Line 6-14 of Algorithm []. In addressing more complex queries, the utilization of an MDT (Figure (b) and (c)) approach has been increasingly recognized for its effectiveness in producing comprehensive and nuanced solutions [45]. The MDT framework leverages the collective expertise of professionals from diverse disciplines, facilitating a holistic examination of the query at hand. This collaborative method is particularly advanta- geous in scenarios where the complexity of problem transcends the scope of single domain, necessitating a fusion of insights from various specialties 77] 71) The MDT approach not only enhances decision-making quality through the integration of multidimensional perspectives but also significantly improves the adaptability and efficiency of the mroblem-solving process [21].
Building upon this foundation, our framework specifically addresses queries of moderate complexity through a structured, multi-tiered collaborative approach. An MDT recruited by recruiter LLM (see Figure 10 in Appendíx) starts an iterative discussion process aimed reaching a consensus with at most Rt rounds (Line 10-12). For each round r E R, agents A,, i E 1,..., NN indicate participation and preferred interlocutors. The system facilitates message exchanges for I turns. If consensus not reached and agents agree to continue, a new round begins with access to previous conversations. For every round, consensus within the MDT is determined by parsing and comparing their opinions. In the event of a disagreement, the moderator agent, consistent with the one described in Section 3.2 reviews the MDT's discourse and formulates feedback for each agent.
High- Complex care cases (Line 17-24 of Algorithm[]. In contrast to the MDT approach, the ICT (Figure 10 (d)) paradigm is essential for addressing the highest tier of query complexity in healthcare. This structured progression through the ICT ensures depth of analysis that is specialized and focused at each stage of the decision-making process. Beginning with the Initial Assessment Team, moving through various diagnostic teams, and culminating with the Final Review & Decision Team, our ICT model aligns specialist insights into a cohesive narrative that informs the ultimate decision (Appendix Algorithms [ Lines 19-21). key component of this process is the report generation process described in Appendix with the prompt, where each team, led by a lead clinician, collaboratively produces a comprehensive report synthesizing their findings. This phased approach, supported by evidence from recent healthcare studies, has been shown to enhance the precision of clinical decision decision-making, making, as each team builds upon the foundation laid by the previous, ensuring meticulous and refined examination of complex medical cases [34] The resultant reports, accumulating throughout the ICT process, are not only reflective of comprehensive medical evaluations but also of systematic and layered analysis that is critical in the management of intricate health scenarios [18]
3.5 Decision-making
In the final stage of our framework, the decision-maker maker agent synthesizes the diverse inputs generated throughout the decision-makin process to arrive at well-informed final answer to the medical query
q. This synthesis involves several components depending on the complexity level of the query:
1. Low:
Directly utilizes the initial response from the primary decision decision-making making agent.
Moderate: Incorporates the conversation history (Interaction) between the recruited agents to understand the nuances and disagreements in their responses.
3. High: Considers detailed reports (Reports) generated by the agents, which include comprchensive
analyses and justifications for their diagnostic suggestions.
The decision-making process is formulated as anss Agent(-) where the final answer, ans deter- mined by integrating the outputs from analysis and synthesis step based on its medical complexities. This integration employs ensemble techniques such as temperature ensembles to ensure the decision is robust and reflects consensus among the models when applicable (see Appendix C.2 for details).
5

---

Table 2: Accuracy (%) on Medical benchmarks with Solo/Group/Adaptin setting. Bold represents the best and Underlined represents the second best performance for each benchmark and model. All benchmarks except for MedVidQA were evaluated with GPT-4(V) and MedVidQA was evaluated with Gemini-Pro(Vision Full experimental results with other models are listed in Table 12 in Appendix.
Medical Knowledge Retricval Datassess
Category
Method
MedQA
PubMalQA
Path-VQA
PMC-VQA
MedViaQA
O
Zero-shot Few-shot
75.0 1.3
61.5
+
$7.9 L6
49.0 =: 3.7
37,9 =: 8.4
72.9 11:4
63.1
11.7
57.5 4.
52.2
3.0
47.1
Single agent
83 CoT-SC
51.3
18.5
55
83.9 127
58.7
1.50
61.2 1:21
50.5
522
49.2
EX 67
81.9 2.1
36.0
1.0
61. 1:4.
52.7 19
48.5
4.
Medprompt
82,4
Majority Voting
90.5
72.2
569
36,8
50.8
Multi-apent
Woighed Varing
78.8 1:111
72.2 1.64
62.1 1114
25,4
57.8
(Single model)
Barda Count
70.3 1:5
66.9 1:50
61.9 1:811
27.9 1:53
54.5 1:40
MedAgims b2
79.1
69.7 4.7
45.4
39.6
$1.6 a
Meta Promping
10
80.5
73.3
55.3
42.6 4.40
Multi agent
Reconeile [6
8T.3 3 3.
79.7 3.50
57.5
31,4 12
(Multi-medel)
AutoCien Sc
60.5 1:10
27.3 2:23
43.0
37.3 6.61
DyLAN
73.6
Adaptive
MDAgeets (Ours)
88.7 4/
75.0 + 100
65.3 +:30
9644 4:45
56,2 +
Clinical Reasoning and Diagnostic Dutasets
Category
Method
DDXPlus
SynCCa
JAMA
ModRullets
MIMIC-CXR
Zeru-sbot
70.3
88.7
62.0
67.0
40.0
69.4
72.0
Single agent
Col
B3
72.7 2.
78.0
2.0
56.0 57
70.0 + 0.0
36.2
Col-SC C2
52.1 6.4
K3.3 1:31
68.0 23
76.0 1.5
51.7
82.7
71.
76.0
Modgcamp
59.5
87.3
1:45
71.0
Majority Voting
91.9
70.0
49.5
(Single-mode))
Multi-agent
Hoeda Count
Weighted Vating
68.9 1.15
07.1 1.677
78.0 1:1
90.5 1.79
66.1 1:41
61.0 5.
66.0 1 17
66.0
45.3 1.4
$3.5 1.7
ModAgents
90.0
77.0
Meta Prompeing
52.6
73.3+223
61.7
49. +11
42.0 4.0
(Mulli-model)
Multi
Reconcile 10
Antotien 3m
67.3
08.4 1.41:
73.3
50.6
G0. 1:3
64.6111
59.3 1777
55.3
353 1.1.4"
43.3
DyLAN [1)
56.4
75.3 4:6
60.1 3.1
57.3 4.1
38.7 1.2
Adaptive
MDAgents (Ours)
77.9 21
931 1.0
2099 0.
80.8 1.7
55.9
CoT: Chain-uf-Trough. SC: Self-Cansistenyy Ensemble Refinement
① tex only. ① imagettex. Q vidcovicst
All experiments were iested with random seads
Experiments and Results
In this section, evaluate our framework and baseline methods across different medical benchmarks in Solo, Group, and Aaptive settings. Our experiments and ablation studies highlight the framework's 's performance, demonstrating robustness and efficiency by modulating agent numbers and temperatures. Results also show a beneficial convergence of agent opinions in collaborative settings.
4.1 Setup
To verify the effectiiveness of our framework, we conduct comprehensive experiments with baseline methods on ten datasets including MedQA, PubMedQA, DDXPlus, SymCat, JAMA, MedBullets, PQA, PMC-VQA, MIMIC-CXR and MedVidQA. A detailed explanation and statistics for each dataset are deferred to Appendix AAand Figure 8 inference time for each complexity was: low 14.7s, moderate 95.5s, and high 226s in average. For all the quantitative experiments this section, we compare three settings: (1) Solo: Using single LLM agent in the decision-making state. (2) Group: Implementing multi-agents to collaborate during the decision-making process. (3) Adaptive: Our proposed method MDAgents, adaptively constructs the inference structure from PCP to MDT and ICT. We use 3-shot prompting for low-complexity cases and zero-shot prompting for moderate and high-complexity cases across all settings.
We use 50 samples per dataset for testing, and the
Medical Question Answering With MedQA (35). PubMedQA (36), MedBullets [9 and JAMA 19. we focus on question answering through text, involving both literature- based and conceptual medical knowledge questions. Specifically, PubMedQA tasks models to answer questions using abstracts from PubMed, requiring synthesis of biomedical information. MedQA tests the model's 's ability to understand and respond to multiple-choice questions derived from medical educational materials and examinations. MedBullets provides USMLE Step 2/3 type questions that demand

---

the application of medical knowledge and clinical reasoning. JAMA Clinical Challenge presents challenging real-world clinical cases with diagnosis or treatment decision-making making questions, testing the model's clinical reasoning (Figure 8 in Appendix shows complexity distribution for each dataset) Diagnostic Reasoning DDXPlus 73 and SymCat [2] involve clinical vignettes that require differential diagnosis, closely mimicking the diagnostic process of physicians. These tasks test the model's 's ability to reason through symptoms and clinical data to suggest possible medical conditions, evaluating the AI's diagnostic reasoning abilities similar a clinical setting. SymCat [2] uses synthetic patient records constructed from a public disease-symptom data source and is enhanced with additional contextual information through the NLICE method.
Medical Visual Interpretation Path-VQA 30), PMC-VQA (95), MedVidQA (29], and MIMIC- CXR 33 challenge models to interpret medical images and videos, requiring integration of visual data with clinical knowledge. PatVVQA focuses on answering questions based on pathology images, testing Al's capability to interpret complex visual information from medical images. PMC-VQA evaluates AI's proficiency in deriving answers from both text and images found in scientific publi- cations. MedVidQA extends to video-based content, where Al models need to process information from medical procedure videos. MIMIC-CXR VQA specifically targets chest radiographs, utilizing diverse and large-scale dataset designed for visual question -attiionnnswwiinggg tasks in the medical domain. Baseline Methods
Solo: The baseline methods considered for the Solo setting include the following: Zero-shot
41
directly incorporates a prompt to facilitate inference, while Few-shot [8] involves small number of examples. Few-shot CoT 33 integrates rationales before deducing the answer. Few-shot CoT-SC
[82] builds upon Few-shot COT by sampling multiple chains to yield the majority answer. Ensemble Refinement (ER) 677 is a prompting strategy that conditions model responses on multiple reasoning
paths to bolster the reasoning capabilities of LLMs. Medprompt [59] is composition of several prompting strategies that enhances the performance of LLMs and achieves state-of-ther of-tth-artttttt results on multiple benchmark datasets, including medical and non- non-medical domains.
Group: We tested five group decision-makinn methods: Voting (82). MedAgents 722 Reconcile [10,, AutoGen [86], and DyLAN [51]. Autogen was based on four agents, with one User, one Clinician, one Medical Expert, and one Moderator, with response per agent [86]. DyLAN setup followed the base implementations of four agents with no specific roles and four maximum rounds of interaction (51] While the methods support multiple models, GPT-4 was used for all agents.
4.2 Results
In Table report the classification accuracy on MedQA, PubMedQA, DDXPlus, SymCat, JAMA, MedBullets, VQA, PMC-VQA and MedVidQA dataset. We compare our method (Adaptive) with several baselines in both Solo and Group settings.
Adaptive method outperforms Solo and Group settings. As depicted in Figure and Table 2 MDAgents significantly outperforms (p < 0.05) both Solo and Group setting methods, showing best performance in 7 out of 10 medical benchmarks tested. This reveals the effectiveness of adaptive strategies integrated within our system, particularly when navigating through the text-only (e.g., DDXPlus where it outperformed the best performance of single-agent by 5.2% and multi-agent by 9.5%) and text-image datasets (e.g., Path- VQA, PMC-VQA and MIMIC-CRR). Our approach not only comprehends textual information with high precision but also adeptly synthesizes visual data, pivotal capability in medical diagnostic evaluations.
1.0
Low
Moderate
High
Average Ralio
0.8
0.6
Counts
Counts
Counts
0.4
0.2
2
0.0
orgmax
argnid
argmin
0.0
0.5 Accuracy (b)
1.0
0.0
0.5 Accuracy (c)
1.0
0.0
0.5 Accuracy
1.0
(d)
Figure 3: Experiment with the MedQA dataset (N=25 randomly sampled questions). (a) LLM's capability to classify complexity. (b-d) Evaluating 25 medical problems by solving each one 10 times at various complexity levels. The x-axis represents the accuracy achieved for each problem, while the y-axis shows the number of problems that reached that level of accuracy.
7

---

Why Do Adaptive Decision- Decision-making Framework Work Well? is important to accurately assign difficulty levels to medical questions. For instance, a medical question is obviously easy, utilizzing a team of specialists (such as an IDT) might be excessive and potentially lead to overly pessimistic approaches. Conversely, if a difficult medical question only tackled by a PCP, the problem might not be adequately addressed. The core issue here is the LLM's capability to classify the difficulty of medical questions appropriately. If an LLM inaccurately classifies the difficulty level, the chosen medical solution may not be suitable, potentially leading to the wrong decision making. Therefore, understanding what constitutes an appropriate difficulty level is essential.
We hypothesize that an LLM, functioning as a classifier, will select the optimal complexity level for each MDM problem. This hypothesis is supported by Figure 33] which illustrates that the model appropriately matches the complexity levels; low, moderate, and high of the given problem. To determine this, we assessed the accuracy of solutions across various difficulty levels. Specifically, we evaluated 25 medical problems by repeating each problem for 10 times at each difficulty level. By measuring the success rate, we aimed to identify the difficulty level that yielded the highest accuracy. This approach ensures that the LLM's complexity classification aligns with the most effective and accurate medical solutions, thereby optimizing the application of medical expertise to each question. Formally. for any given problem P. we denote the probability that the correct answer can be solved at specific complexity level as Caumplexittyleeel (P), where complexity-Ievel E {low, moderate, high). arg max P) E (low, moderate, high} refers to the complexity level that has the highest proba- bility among Prew (P), Pmodeerte (P), and Phigh (P). Similarly, arg min(P) is the complexity level with the lowest probability, and arg mid( is the one with the middle probability. We denote a, b, and e as the probabilities that the LLM selects the complexity levels corresponding to arg max, arg mid, and arg mini respectively. Thus, the accuracy of our system for problem P can be described by a Parg max (P) + Parg mia mia + c -Pag min (P), and the overall accuracy is given by Ep (a- Parg max (P) + b. Parg mid (P) + c. Parg min (P)]. The estimated values of b, e a = 0.81 + 0.29, 6 = 0.11 = 0.28, and c = 0.08 + 0.16, which indicates that LLM can provide an optimal complexity level with probability at least 80%.
These findings suggest that a classifier LLM can implicitly simulate various complexity levels and optimally adapt to the complexity required for each medical problem, as shown in Figure [ This ability to adjust complexity dynamically proves to be crucial for applying LLMs effectively in MDM contexts as shown by the competitiveness of our Adaptive approach.
Solo vs. Group Setting in MDM. The ex- perimental results reveal distinet performance patterns between Solo and Group settings across various medical benchmarks. In simpler datasets like MedQA, solo methods, leveraging Few-shot CoT and CoT-SC, achieved up to 83.9% accu- racy compared to the group's 's best of 81 81.3% Conversely, for more complex datasets like Sym- Cat, group settings perform better, with SymCat showing 91.9% accuracy the group settings versus 88. 7% in solo settings. Notably, group settings (e.g. Weighted Voting, Reconcile) per- formed better in multi-modal datasets such as Path-VQA (image + text), MedVidQA (video + text), and MIMIC-XRR (image text), high- lighting the advantage collaborative process in complex tasks. This result aligns with findings from 4. which showed that pooled diagnoses from groups of physicians significantly outperformed those of individual physicians, with accuracy increasing as group size increased. Óverall, solo settings outperformed group settings in four benchmarks, while group settings outperformed solo in six benchmarks. These results reveals that while solo methods excel in straightforward tasks, group settings provide better performance in complex, multi-faceted tasks requiring diverse expertise.
PPAA
PLEMACCAA
00.4%
VasSukea
30%
DUP
Solo
Group
Arzuneyy
Figure 4: Our method outperforms Solo and Group settings across different medical benchmarks.
4.3 Ablation Studies
Impact of Complexity Selection. We evaluate the importance of the complexity assessment and adaptive process through ablation studies (Figure Our adaptive method significantly outperforms static complexity assignments across different modality benchmarks. For text-only queries, the Adaptive method achieves an accuracy of 81.2% significantly higher than low (64.2%) moderate (71.6%), and high (65.8%) settings. Interestingly, 64% of the text-only queries were classified as
8

---

high complexity, indicating that many text-based queries required in-depth analysis with different expertise. In the image text modality, the Adaptive method classified 55% of the queries as low complexity, suggesting that the visual information often provides clear and straightforward cues that simplify the srccision-making process. Finally, for video text queries, 87% of these queries were classified as low complexity, reflecting that the dynamic visual data in conjunction with text can often be straightforwardly interpreted. However, further evaluation on more challenging video medical datasets is needed, MedVidQA contains relatively less complex medical knowledge.
Impact of Moderator's Re-
Method
Avg. Accuracy (%)
view and RAG Table 3 ex-
MDAgents (Ours)
71.8
amines the impact of incorpo-
+ MedRAG
75.2 (f 4.7 %)
rating external medical knowl-
+ Moderator's Review
77.6 (1 8.1 %)
edge and moderator reviews
+ Moderator's Review & MedRAG 80.3 (1 11.8 %)
into the MDAgents framework on accuracy. MedR AG [89] a systematic toolkit for Retrieval-Augmented Genera- tion (RAG) that leverages various corpora; biomedical, clinical and general medicine, to provide comprehensive knowledge. The baseline accuracy of MDAgents is 71.8%. Integrating MedRAG increases accuracy to 75.2% (up 4.7%), while the moderator's s review alone raises to 77.6% (up 8.1%). The combined use of both methods achieves the highest accuracy at 80.3% (up 1 11.8%).
Table 3: Ablations for the impact of moderator's review and
MedRAG. The Accuracy were averaged accuracy across all datasets.
The results indicate that MedRAG and moderator review both enhance performance, with their combined effect being synergistic. This highlights that leveraging recent external knowledge and structured feedback mechanisms is crucial for refining and converging on accurate medical decisions. This improvement underscores the importance of hybrid strategy, aligning with real- world practices of continuous learning and expert consultation to optimize performance in medical applications.
4.4 Impact of Number of Agents in Group Setting.
Our experiment with varying the number of agents in a collaborative Group setting (Appendix Figure 6 (a-b)) shows that higher number of agents does not lead to better performance. Rather, our Adaptive method achieves optimal performance with fewer agents (N=3, peak accuracy of 83.5% by intelligently calibrating the number of collaborating agents. This not only indicates efficiency in decision-making but also computational and economic benefits, considering the reduced number of API calls needed, especially when contrasted with the Solo and Group settings.
With regards to computational efficiency, the Solo setting (5-shot CoT-SC) resulted in a 6.0 and Group setting (MedAgents with N=5) resulted in a 20.3 API calls, suggesting a high computational cost without a corresponding increase in accuracy. On the other hand, our Adaptive method exhibits a more economical use of resources, demonstrated by fewer API calls (9.3 with N=3) while maintaining high accuracy, critical factor in deploying scalable and cost-effective medical AI solutions.
4.5 Robustness of MDAgents with different parameters.
Our Adaptive approach shows resilience to changes in temperature (Appendix Figure 6 (c), low (T=0.3) and high (T=1.2)) with performance improving under higher temperatures. This suggests that our model can utilize the creative and diverse outputs generated at higher temperatures to enhance Docision-making
property that is not pronounced in the Solo and Group conditions. This robustness is particularly valuable in real-world medical domains with high uncertainty and ambiguity in datasets [3]. Additionally, studies have shown that creative diagnostic approaches
Aa
text-only
AF
Aa
image + text
D
Aa
video text
80-
81.2
m
65.8
Accurary
Accirocy
50.2
58.
51.1
51.7
40-
41.5
Adaptive
Low
Moderate
High
Adaptive
Low
Moderate
High
Adaptive
Moderate
High
Figure 5: Impact of complexity selection of the query. Accuracy of each ablation on text-only (left), text+image (center) and texi+video (right) benchmarks are reported.

---

Group IMudAganta WT agenca)
Tempura snwee
Sebo IDords Count)
81.5
Our
75-
88
Pccuracy
-2-
66,9
65
PC6
6.
58-
Group
50-
40
Ours
Sala
Group
Number Aqents
Numbber of Agents
(a)
(b)
Figure 6: Impact of the number of agents on (a) Accuracy, (b) Number of API Calls on medical benchmarks with GPT-4 (V) and (c) Performance of three different settings under low (T=0.3) and high (7=1.2) temperatures on medical benchmarks. Our Adaptive setting shows better robustness to different temperatures and even takes advantage higher temperatures.
text-only
nixt video e teot
Entropy
0.6
0.
Step 0
Step
Step
Step 3
Step
Step 5
Figure 7: An illustration of consensus entropy in group collaboration process of MDAgents (w/ Gemini-Pro (Vision), N=30 for each dataset) on medical benchmarks with different modality inputs.
can mitigate cognitive biases and improve diagnostic accuracy [66]. while fostering flexibility and adaptability in decision-making [19]. These insights support the enhanced performance observed under higher temperatures in our framework. However, the future work should explore a wider range of temperatures fully understand the robustness and adaptability of our approach.
4.6 Convergence Trends in Consensus Dynamies
There is clear trend towards consensus among MDAgents cross various data modalities (Figure 7 The text+video modality demonstrates a rapid convergence, reflecting the agents' efficient processing of combined textual and visual cues. On the other hand, the text+image and texi-only modalities display a more gradual decline in entropy, indicating a progressive narrowing of interpretative diversity among the agents. Despite the differing rates and initial conditions, all modalities exhibit convergence of agent opinions over time. This uniformity in reaching consensus highlights the MDAgents' capability integrate and reconcile information. Please refer Appendix B for detailed explanation of the entropy calculation.
5
Conclusion
This paper introduces MDAgents, framework designed enhance the utility of LLMs in complex medical decision-making by dynamically structuring effective collaboration models. To reflect the nuanced consultation aspects in clinical settings, MDAgents adaptively assigns LLMs either roles independently or within groups, depending on the task's complexity. This emulation of real-world medical decision processes has been comprehensively evaluated, with MDAgents outperforming previous solo and group methods in 7 out of 10 medical benchmarks. The case study illustrates the practical efficacy and collaborative dynamics of our proposed framework, providing insights into how differing expert opinions are synthesized to reach more accurate diagnosis. This is evidenced by our agents' ability to converge on the correct diagnosis despite initially divergent perspectives. Ablation studies further elucidate the individual contributions of agents and strategies within the system, revealing the critical components and interactions that drive the framework's success. By harnessing the strength of multi-modal reasoning and fostering a collaborative process among LLM agents, our framework opens up new possibilities for enhancing M.L--sssisted medical diagnosis systems, pushing the boundaries of automated clinical reasoning.
10

---

Acknowledgments and Disclosure of Funding
We thank Yoon Kim at MIT, Vivek Natarajan at Google, WonJin Yoon and Tim Miller at Harvard Medical School, Seonghwan Bae at Sacheon Public Health Center, Hui Dong Lim at Scoul National University Hospital for their revisions, feedback, and support. C.P. acknowledged support from the Takeda Fellowship, the Korea Foundation for Advanced Studies, and the Siebel Scholarship.
References
[1] Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. Large language models are few-shot clinical information extractors. arXiv preprint arXiv:2205. 12689, 2022.
[2] Zaid Al-Ars, Obinna Agba, Zhuoran Guo, Christiaan Boerkamp, Ziyaad Jaber, and Tareq Jaber. Nlice: Synthetic medical record generation for effective primary healthcare differential diagnosis, 2024.
[3] Seongsu Bae, Daeun Kyung, Jaehee Ryu, Eunbyeol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric Chang. Tackeun Kim, et al. Ehrxqa: A multi-modal question answering dataset for electronic health records with chest x-ray images. Advances in Neural Information Processing Systems, 36, 2024.
[4] Michael L Barnett, Dhruv Boddupalli, Shantanu Nundy, and David W Bates. Comparative accuracy of diagnosis by collective intelligence of multiple physicians vs individual physicians JAMA network open, 2e ((3)9990096--0 (00096-e190090 2019.
[5] Michael L Barnett, Nancy L Keating, Nicholas A Christakis, A James O' O'mallcy, and Bruce E Landon. Reasons for choice of referral physician among primary care and specialist physicians. Journal of general internal medicine, 77:506-512, 2012.
[6] Ofir Ben-Assuli, Nanda Kumar, Ofer Arazy, and Itamar Shabtai. The use of analytic hierarchy process for measuring the complexity of medical diagnosis. Health Informatics Journal, 26(1):218-22) 232, 2020.
[7] Justin Bitter, Elizabeth van Veen-Berky, Hein G Gooszen, and Pierre van Amelsvoort. Mul- tidisciplinary teamwork is an important issue to healthcare professionals. Team Performance Management: An International Journal, 19(5/6):263 278, 2013.
[8] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
[9] Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions, 2024.
[10] Justin Chih Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse Ilms, 2024.
[11] Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors, 2023.
[12] Michael Christ, Florian Grossmann, Daniela Winter, Roland Bingisser, and Elke Platz. Modern triage in the emergency department. Deutsches Ärzteblatt International, 107(50)882,, 2010.
[13] Jane Cioffi and Roslyn Markham. Clinical decision by midwives: managing case complexity. Journal of advanced nursing, 25(2):265-211 272, 1997.

---

[14] Jan Clusmann, Fiona R Kolbinger. Hannah Sophie Muti, Zunamys I Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Lalch, Chiara Maria Lavinia Löffler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, et al. The future landscape of large language models in medicine. Communications medicine, 3(1):141, 2023.
[15] Nick Craddock and Laurence Mynors- Wallis. Psychiatric diagnosis: impersonal, imperfect and important. The British Journal of Psychiatry, 204(():93-95, 2014.
[16] Lesley F Degner and Jeffrey A Sloan. Decision making during serious illness: what role do patients really want to play? Journal of clinical epidemiology, 45(9):941-950, 1992.
[17] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate, 2023.
[18] Carolyn Ee, James Lake, Joseph Firth, Fiona Hargraves, M De Manincor, Tanya Meade, Wolfgang Marx, and Jerome Sarris. An integrative collaborative care model for people with mental illness and physical comorbidities. International Journal of Mental Health Systems, 14:1-16, 2020.
[19] Arthur S Elstein and Alan Schwarz. Clinical problem solving and diagnostic decision making: selective review of the cognitive literature. BMJ, SAA,32227339)7729-72222 2002.
[20] Glyn Elwyn, Dominick Frosch, Richard Thomson, Natalie Kinnersley, Emma Cording, Dave Tomson, Carole Dodd, Stephen Rollnick, al. Shared decision making: a model for clinical practice. Journal of general internal medicine, 27:1361- 1367, 2012.
Amy Lloyd, Paul
[21] Nancy E Epstein. Multidisciplinary in in-hospital teams improve patient outcomes: A review. Surgical neurology international, 5(Suppl 7):5295, 2014.
[22] Zhihao Fan, Jialong Tang, Wei Chen, Siyuan Wang. Zhongyu Wei, Jun Xi, Fei Huang, and Jingren Zhou. Ai hospital: Interactive evaluation and collaboration of llms as intern doctors for clinical diagnosis, 2024.
[23] Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from ai feedback, 2023.
[24] Isaac R Galatzer-Levy, Daniel McDuff, Vivek Natarajan, Alan Karthikesalingam, and Matteo Malgaroli. The capability of large language models to measure psychiatric functioning, arXiv preprint arXiv:2308. 01834, 2023.
[25] Amy L Garcia. Variability in acuity in acute care. JONA: The Journal of Nursing Administration, 47(10):476-43,, 2017.
[26] Nicki Gilboy, Paula Tanabe, Debbie Travers, Alexander M Rosenau, et al. Emergency severity index (esi): A triage tool for emergency department care, version 4. implementation handbook 2012 edition. AHRQ publication, 12, 2011.
[27] David Grembowski, Judith Schaefer, Karin E Johnson, Henry Fischer, Susan I. Moore, Ming Tai-Seale, Richard Ricciardi, James R Fraser. Donald Miller, Lisa LeRoy, et al. A conceptual model of the role of complexity in the care of patients with multiple chronic conditions. Medical care, 52:57-S14, 2014.
[28] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specitf language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare, 3(1):1223, October 2021.
[29] Deepak Gupta, Kush Attal, and Dina Demner-Fushhan. A dataset for medical instructional video classification and question answering. 2022.
[30] Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering, 2020.
12

---

[31] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprini arXiv:
2021.
[32] Sirui Hong. Mingehen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang. Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative framework, 2023.
[33] Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng. Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty-awat planning enhances information seeking in large language models. arXiv preprint arXiv:2402.03271, 2024.
[34] M. Jimenez-Lara. Reaping the benefits of integrated health care. stanford social innovation review. Stanford Social Innovation Review,, 2016.
[35] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams, 2020.
[36] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567-2577, 2019.
[37] Qiao Jin, Zhizheng Wang, Yifan Yang, Qingqing Zhu, Donald Wright, Thomas Huang, W John Wilbur, Zhe He, Andrew Taylor, Qingyu Chen, et al. Agentmd: Empowering language agents for risk prediction with large-scale clinical tool learning. arXiv preprini TrXiv:2402.13255 2024.
[38] Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. Genegpt: augmenting large language models with domain tools for improved access biomedical information. Bioinformatics, 40(2), February 2024.
[39] Emre Kıeuman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint arXiv:2305.0000 000500 2023.
[40] Yubin Kim, Xuhai Xu, Daniel McDuff, Cynthia Breazeal, and Hae Won Park. Health-Ilm: Large language models for health prediction via wearable sensor data, 2024.
[41] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-225 22213,, 2022.
[42] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.
2024.
[43] Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and Victor Tseng. Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLOS Digital Health, 2(2):1-12, 02 2023.
[44] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453 466, 2019.
[45] Danielle I. LaFrance, Mary Jane Weiss, Ellie Kazemi, Joanne Gerenser, and Jacqueline Dobres. Multidisciplinary teaming: Enhancing collaboration through increased understanding. Behavior analysis in practice, 12(3):709-726. 2019.
13

---

[46] Beibin Li, Konstantina Mellou, Bo Zhang, Jeevan Pathuri, and Ishai Menache, Large language models for supply chain optimization. arXiv preprint arXiv:2307.0387. 2023.
[47] Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society, 2023.
[48] Junkai Li, Siyu Wang, Meng Zhang, Weitao Li, Yunghwei Lai, Xinhui Kang, Weizhi Ma, and Yang Liu. Agent hospital: A simulacrum of hospital with evolvable medical agents. arXiv preprint 2XXi:2245500255, 2024.
[49] Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. Mediq: -askiig Ilms for adaptive and reliable medical reasoning. arXiv preprint arXiv:2406.0000 00922, 2024.
[50] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang, Rui Wang, Yujuu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate, 2023.
[51] Zijum Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic Ilm-agent network: An Ilm-agent collaboration framework with agent team optimization, 2023.
[52] Valentin Liévin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. Can large language models reason about medical questions?, 2023.
[53] Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang. Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, et al. Towards accurate differential diagnosis with large language models. arXiv preprint arXiv:2312.00164, 2023.
[54] Ateev Mehrotra, Christopher B Forrest, and Caroline Y Lin. Dropping the baton: specialty referrals in the united states. The Milbank Quarterly, 89(1):39-68 2011.
[55] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M. Krumholz, Jure Leskovec, Eric J. Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):299-65,, 2023.
[56] Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety, Advances in Neural Information Processing Systems, 2024.
[57] Mark A Musen, Blackford Middleton, and Robert A Greenes. Clinical decision-support systems. In Biomedical informatics: computer applications in health care and biomedicine, pages 795-840. Springer, 2021.
[58] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capa- bilities of gpt-4 on medical challenge problems, 2023.
[59] Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, and Eric Horvitz. Can generalist foundation models outcompete special- tuning? case study in medicine. November 2023.
[60] OpenAl. Gpt-4 technical report, 2024.
[61] Anand K Parekh, Richard Goodman, Catherine Gordon, Howard K Koh, and HHS Intera- gency Workgroup on Multiple Chronic Conditions. Managing multiple chronic conditions: strategic framework for improving health outcomes and quality of life. Public health reports, 1 126(4):460-471, 10
[62] Chanwoo Park, Xiangyu Liu, Asuman Ozdaglar, and Kaiqing Zhang. Do llm agents have regret? case study in online learning and games. arXiv preprint S.03::2403.16444 2024.
14

---

[63] Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz. Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilitiess of gemini models in medicine. arXiv preprint urXiv:2404. 18416, 2024.
[64] Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce Ho, Carl Yang. and May D. Wang. Ehragent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records, 2024.
[65] Mohit Shridhar, Xingdi Yuan, Mare-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworid: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020.
[66] Hardeep Singh, Denise M Connor, and Gurpreet Dhaliwal. Five strategies for clinicians to advance diagnostic excellence. BMJ, 376, 2022.
[67] Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Senevi- ratne. Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman Blaise Agüera Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajko- mar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. Nature, 620(7972): 172-1880 2023.
[68] Harold C Sox, Michael C Higgins, Douglas K Owens, and Gillian Sanders Schmidler. Medical decision making. John Wiley & Sons, 2024.
[69] J Stairmand, Louise Signal, D Sarfati, Jackson, I. Batten, M Holdaway, and C Cunningham. Consideration of comorbidity in treatment decision making in multidisciplinary cancer team meetings: systematic review, Annals of Oncology, 26(7):1325- 2015.
[70] Mirac Suzgun and Adam Tauman Kalai. Meta-prompting: Enhancing language models with task-agnostic scaffolding, 2024.
[71] Miren Taberna, Francisco Gil Moncayo, Enric Jané-Salas, Maite Antonio, Lorena Arribas, Esther Vilajosana, Elisabet Peralvez Torres, and Ricard Mesía. The multidisciplinary team (mdt) approach and quality of care. Frontiers in oncology, 10:85, 2020.
[72] Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning, 2024.
[73] Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. Ddxplus: A new dataset for automatic medical diagnosis, 2022.
[74] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine. 1930-1993, 2023.
[75] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature Medicine, 1930-199,,, 2023.
[76] Daniel Truhn, Jan-Niklas Eckardt, Dyke Ferber, and Jakob Nikolas Kather. Large language models and multimodal foundation models for precision oncology. NPJ Precision Oncology, 8(1):72, 2024.
[77] Tao Tu, Anil Palepu. Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, Shekoofeh Azizi, Karan Singhal, Yong Cheng, Le Hou, Albert Webson, Kavita Kulkarni, S Sara Mahdavi, Christopher Semturs, Juraj Gottweis, Joelle Barral, Katherine Chou, Greg Corrado, Yossi Matias, Alan Karthikesalingam, and Vivek Natarajan. Towards conversational diagnostic ai, 2024.
15

---

[78] Sean R Tunis, Daniel B Stryer, and Carolyn M Clancy. Practical clinical trials: increasing the value of clinical research for decision making clinical and health policy. Jama, 290( 1632, 2003.
[79] Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean- Delbrouck, Asad Aali, Chris- tian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Sechofnerová, Nidhi Rohatgi, Poonam Hosamani, William Collins, Neera Ahuja, Curtis Langlotz, Jason Hom, Sergios Gatidis, John Pauly, and Akshay S. Chaudhari. Adapted large language models can outperform medical experts clinical text summarization. Nature Medicine, 30(4): 1142, February 2024.
[80] Lei Wang. Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang. Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey on large language model based autonomous agents, 2024.
[81] Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. Chatcad: Interactive computer-aidee aided diagnosis on medical image using large language models, 2023.
[82] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, arXiv preprint arXiv:2203. 2022.
[83] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thouge prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-22837, 2022.
[84] Thomas Weida and Jane Weida. Outpatient e/m coding simplified. Family Practice Management, 29(1):6631,, 2022.
[85] Robin M Weinick, Rachel M Burns, and Ateev Mehrotra. Many emergency department visits could be managed urgent care centers and retail clinics. Health affairs, 29(9):1630-1638 1636, 2010.
[86] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang. Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation, 2023.
[87] Richard C Wuerz, Leslie W Milne, David R Eitel, Debbie Travers, and Nicki Gilboy. Reliability and validity of new five-level triage instrument. Academic emergency medicine, 7(3):236-22 242, 2000.
[88] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang. Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, and Tao Gui. The rise and potential of large language model based agents: A survey, 2023.
[89] Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking generation for medicine. arXiv preprint arXiv:2402.13177, 2024.
augmeetted
[90] Weixiang Yan, Haitian Liu, Tengxiao Wu, Qian Chen, Wen Wang, Haoyuan Chai, Jiayi Wang, Weishan Zhao, Yixin Zhang, Renjun Zhang, et al. Clinicallab: Aligning agents for multi- departmental clinical diagnostics in the real world. arXiv preprint 47819 2024.
[91] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physies of language models: Part 2.2, how to learn from mistakes on grade-school math problems, 2024.
[92] Cyril Zakka, Akash Chaurasia, Rohan Shad, Alex R. Dalal, Jennifer L. Kim, Michael Moor, Kevin Alexander. Euan Ashley, Jack Boyd, Kathleen Boyd, Karen Hirsch, Curt Langlotz, Joanna Nelson, and William Hiesinger. Almanac: Retrieval-augmented language models for clinical medicine, 2023.
16

---

[93] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language, 2022.
[94] Haodi Zhang. Jiahong Li, Yichi Wang, and Yuanfeng Songi. Integrating automated knowledge extraction with large language models for explainable medical decision-making In 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 1710-1717. IEEE, 2023.
[95] Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering, 2023.
[96] Hongjian Zhou, Fenglin Liu, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam S. Chen, Peilin Zhou, Junling Liu, Yining Hua, Chengfeng Mao, Chenyu You, Xian Wu, Yefeng Zheng, Lei Clifton, Zheng Li, Jiebo Luo, and David A. Clifton. A survey of large language models in medicine: Progress, application, and challenge, 2024.
[97] Junbin Zhou and Xiao Xu. The difficulty of medical maaiii: should patients be involved? Hepatobiliary Surgery and Nutrition, 12(3):407, 2023.
17

---

Limitations and Future Works
Despite the successes of our framework in showing promising performance in medical decision- making tasks, we recognize several limitations that open pathways for future research.
Medical Focused Foundation Models. An essential enhancement would be to incorporate the foundation models and systems specifically trained on medical data, like Med-Gemini 63], AMIE [77], and Med-PaLM [67]. These models excel in generating professional medical terminologies, which can facilitate more effective and accurate communication between multiple agents involved in the decision -aaiing process. By leveraging these specialized models, the agents can interact using shared, precise medical vocabulary. enhancing the system's 's overall performance and reliability. This approach not only ensures more medically accurate content generation but also supports better collaboration and understanding among the agents, which is essential for complex medical decision- making tasks.
Patient-Centen Diagnosis. A primary limitation lies in the fact that our current framework operates within the confines of multiple-choice question answering and does not account for the interactive, patient-centered nature of real-world diagnostics. Effective diagnosis often relies on continuous exchanges that include the patient's 's narrative, the physician's 's expertise, and input from caregivers. To bridge this gap, future iterations of our framework will aim to incorporate a more interactive system that not only assists physicians but also directly engages with both patients and caregivers in multi-stakeholder approach. Moreover, by incorporating regret-aware [91] decision- making, the system can learn to minimize diagnostic regret over time, refining its responses based on the outcomes of prior interactions. This regret-aware framework will help guide the L.M to seek additional information when uncertainties arise, thereby supporting more informed decisions scenarios. Embedding these real-world interactions within the
across complex, feedback loop will enable the system to provide more nuanced and patient-centric support, enhancing the quality and personalization of medical decision-making across all involved parties.
Potential Risks and Mitigations. While our framework shows promise, potential risks include medical hallucinations and the generation of inaccurate or misleading information. To address these risks, integrating self-correction mechanisms, such as those proposed by (42), could enable the model to autonomously identify and rectify its own errors via reinforcement learning-basee -basen self-correction. Additionally, implementing rule-based reward structures, as suggested in (56), would allow the model to adhere to specific safety and accuracy guidelines during training. These methods can support safer, more reliable diagnostic support tool by introducing corrective feedback loops and standardized behavior guidelines. Furthermore, integrating confidence scores and uncertainty estimates with the model's recommendations could enhance the decision-making process by enabling end-users to weigh various diagnostic options, ultimately increasing the system's 's trustworthiness and safety.
A Dataset Information
We evaluate multi-agent collaboration frameworks across seven common medical question-ans -ann-nniswwnng datasets, which vary in question complexity. Generally, questions are deemed more complex if they involve multiple modalities or entail lengthy, detailed diagnostic task. Below, we detail each dataset and provide a sample entry:
1. MedQA. The MedQA dataset consists of professional medical board exams from the US, Mannlln China, and Taiwan (35]. Our study focuses on the English test set, comprising 1,273 questions sourced from the United States Medical Licensing Examination (USMLE). These questions are formatted as multiple-choice text queries with five options. Due their textual nature and brevity, we categorize these questions as low.
Sample Question: "A 47-year-old female undergoes a thyroidectony for treatment of Graves' disease. Post-operatively, she reports hoarse voice and difficulty speaking. You suspect that this is likely a complication of her recent surgery. What is the embryologic origin of the damaged nerve that is most likely causing this patient's hourseness?"
Options: A: 1st pharyngeal arch, B: 2nd pharyngeal arch, C: 3rd pharyngeal arch, D: 4th pharyngeal arch, E: 6th pharyngeal arch
18

---

Table 4: Summary of the Datasets. Text, Image, detailed sample information for each benchmark.
Video. Appendix
we provide
Tattssr
Modality
Format
Choice
Testing Size
Domain
MedQA
Question Answer
MBBDD
1273
US Medical Licensing Examination
PubMadQA
Quostion Contest Answer
YouNoMMayhe
500
PahMed abstracts
DDxPlus
Question Answer
A/R/C///
134
Pathologics. Symptoms and Anizoddents from Pationss
SymCat
Question Answer
MBOD
369
Disease-yympg symase--mmpiommm records from public datanett
JAMA
Cose Question Ansuer
MBCD
1534
Chullenging real-world Clinical Cases from divense Medical
Domains
MedBullets
Question
M/CD
Online Platfomn
for Medical Study
MIMIC-CXR
Question
Closed Answer
Chest images Free-test Reports
PMC-VQA
Question
MBOD
pairs scross Images, spunning Modalities Divesses
Put--YQA
Question Answer
YeuNoo
Open-ended Questions Pathologg Images
MedVidQA
Question Answer
ABBDD
Aids, Medical Emteggency. and Medical Education Questions
2. PubMedQA. PubMedQA is a QA dataset based on biomedical research [36. It requires yes/no/maybe answers questions grounded in PubMed abstract. The dataset comprises entries each containing question, a relevant abstract minus the conclusion, and ground truth label. We used 50 samples for testing. Given its binary choice format, consider the complexity of this dataset to be low.
Sample Question: "Can predilatation in transcatheter aortic valve implantation be omit- ted?"
Context: "The use of balloon expandable stent valve includes balloon predilatation of the aortic stenosis before valve deployment. The aim of the study was to see whether or not balloon predilatation necessary transcatheter aortic valve replacement (TAVI). Sixty consecutive TAVI patients were randomized to the standard procedure or to a protocol where balloon predilatation was omitted. There were no significunt differences between the groups regarding early hemodynamic results or complication rates."
3. DDXPlus. DDXPlus is medical diagnosis dataset using synthetic patient information and symptoms 773. Each instance represents a patient, with attributes including age, sex, initial evidences, evidence, multiple options of possible pathologies, and a ground truth diagnosis. Due its text-only and multiple-choie nature, we consider the complexity to be low. Sample Patient Information: Age: 96, Sex: F
Evidences:
| /'666', 'insp_siffla' 'j45', "posttus_emesic" 'trav] @_',, 'vaccination')
Initial evidence: ponsttus_emssti
Options: (A) Bronchite (B) Coqueluche
4. PMC-VQA. PMC-VQA is a large-scale medical visual question -anwwerin dataset that contains 227K Visual Question Answering (VQA) pairs of 149K images (95]. It is structured as a multiple-choice QA task with one image input accompanying each question. Since the query requires model to consider both text and image inputs, while maintaining medical expertise, we consider the complexity to be moderate.
Sample Question: What is the appearance of the hyperintense foci in the basal ganglia TI-weighted MRI image? ? Image: PMC8415802 21t.158222FFG.ipp FIG1.jpg
Options: A: Hypodense, B: Hyperdense, C: Isointense, D: Hypointense
5. Path-VQA. PathVQA is a VQA dataset specifically on pathology images 30 Differ- ent from PMC-VQA which consists of multiple choice questions, Path-VQA includes open-ended questions and binary "yes/no" questions. For the purpose of maintaining a standardized accuracy evaluation, we use only the yes/no questions. Similar to PMC-VQA, we consider the complexity to be moderate.
Sample Question: Was gravid uterus removed for postpartum bleeding? Image: test 2er__2223jpp
6. MedVidQA. MedVidQA dataset consists of 3,010 health-related questions with visual answers from validated video sources (e.g. medical school, health institutions, etc). We
19

---

enhanced the dataset by using GPT-4 to generate multiple-chore choice answers, including one correct "golden answer' and several false options. expanding its use for training and evaluat- ing automated medical guestion-answering systems.
Sample Question: How to perform corner stretches to treat neck pain?
Video: 155Mw550TTLMMmm
Options: A: By bending your knees and touching your toes, B: By performing jumping jacks,
C: By leaning into corner with your elbows up at shoulder level, D: By doing push-ups
7. SymCat. SymCat is a synthetic dataset which includes 5 million symptom- condiioonnn samples, covering 801 distinct conditions each with 376 potential symptoms dataset 22.
Sample Patient Information: Age: 42, Gender: F. Race: White, Ethnicity: Nonhispanic
abdomen!!:: Worsening after Sample Patient Symptoms: Pain:cramminng:Abddmnnn...
meals::2; Altered stop::Lump:::..::: Oftenn:44; Flatulence::::::::: Bloating:::::::4
Options: "A"; "Asthma", "B": "IBS (Constipation type)", "C": "Viral meningitis ( (Varicella zoster virus)", "D": "Bacterial (Gastro)enteritis (Yersinia infection most likely)"
8. JAMA. JAMA includes 1524 clinical cases collected from the JAMA Network Clinical Challenge archive, which are summaries of actual challenging clinical cases. Each sample is framed as a question, with a long case description and four options [)]
Sample Case: 62-year-old woman undergoing peritoneal dialysis (PD) for kidney failure due to IgA nephropathy presented to the PD clinic with 1-day history of severe abdominal pain and cloudy PD fluid. Seven days prior, she inadvertently broke aseptic technique when tightening leaking connection of her PD catheter tubing. On presentation, she was afebrile and had normal vital signs. Physical examination revealed diffuse abdominal tenderness. Cloudy fluid that was drained from her PD catheter was sent for laboratory analysis (Table 1 J.Aaiit peritoneal dialysis fluid culture results before starting intruperitoneal antibiotics. Sumple Question: What Would You Do Next?
"A": "Administer empirical broad cpec--ppetruum intraperitoneal antibiotics", "B": "Administer empirical broad-spectren intravenous antibiotics", "C": "Await peritoneal dialysis fluid culture results before starting intraperitoneal antibiotics", "D": "Send blood cultures"
9. Medbullets. Medbullets comprises 308 USMLE Step 2/3 style questions collected from open- tweets on X (formerly Twitter) since April 2022. The difficulty is comparable that of Step 2/3 exams, which emulate common clinical scenarios (9].
Sample Question: A 2-week-old boy is evaluated by his pediatrician for abnormal feet. The patient was born at 39 weeks via vaginal delivery to a GIPI 29-year-old woman. The patient has been breastfeeding and producing 5 stools/day. He is otherwise healthy. His lemperature is 99. 99.500b0F (37. 507,0,0 blood pressure is 60/38 mmllg, pulse 150/min, respirations are 24/min, and oxygen saturation is 98% on room air. A cardiopulmonary exam is notable for a benign flow murmur. A musculoskeletal exam reveals the findings shown in Figure A. Which of the following is the most appropriate next step in management? Options: "A": "Botulinum toxin injections", "B": "Reassurance and reassessment in 1 month", "C": "Serial casting", "D": "Surgical pinning".
10. ALMIC-CXR-VOL MIMIC-CXR is a large-scale visual question-answer -nswering dataset of 377.110 chest radiographs. It was obtained from 227,827 imaging studies sourced from the BIDMC between 201 1-2016. It includes patient identifiers which can be linked to MIMIC-IV B.
Sample Question: Are there any abnormalities in the upper mediastinum? Image:
B Entropy Calculation for Consensus Dynamics
The entropy H serves as an indicator of consensus progression among the agents. It is quantified as:
M H -= Σ p(x:) loga p(ara)
(B.1)
where M is the total number of unique answers, 3; represents unique answer, and p(r;) is the probability of occurring among all answers. This calculation helps to measure the degree of agreement among the agents over time, with lower entropy indicating higher consensus. The trends in entropy
20

---

across different data modalities provide insights into how quickly and effectively MDAgents can reach a unified decision.
C Prompt Templates
C.1 A single agent setting
Few-shot multiple choice questions
{(instruction}}
The following multiple choice questions (with answers) about medical knowledge.
www__shotexxmpppese [[context]} **Question:** [(questionn)) f[anaver_choices)) ** **nnwer:**(
Chain-of- Thain-of-Thought multiple choice questions
{[instruction}}
The following multiple choice questions (with answers) about medical knowledge.
www__shot_examplee w/ CoT Solutions)}
{[context]} ** *Question:** [[question]} (Kansver_choicea) **Anwwer***
Ensemble Refinement multiple choice questions
([insrruction}}
The following multiple choice questions (with answers) about medical knowledge. ((feu_shot_exxl) cthho__eampless w/ CoT Solutions)}
([context]} ** *Question:** [[question]] f[ansver_choicee ((reasoning-pathll patha}} **Answer***
Medprompt multiple choice questions
((instruction))
The following are multiple choice questions (with answers) about medical knowledge. ((few_ (Cfeu_shot_exxe chot_examples w/ CoT Solutions from similarity calculation)) for N times do [(context]} **Question ([question)} **Answer:**
C.2 Multi-agent setting
21

---

Complexity check prompt
You are a medical expert who conducts initial assessment and your job is to decide the difficult//ompllxity of the medical query.
Now, given the medical query as below, you need to decide the difficulty/complexity of it: {(question}>
Please indicate difficulty/complexity the medical query among below options:
1) low: a PCP or general physician can answer this simple medical knowledge checking question without relying heavily on consulting other specialists.
2) moderate: PCP or general physician can answer this question in consultation with other specialist in team.
3) high: Team of multi-departmental specialists can answer the question which requires specialists consulting another department (requires lot team effort to treat case).
#
Recruiter prompt
You experienced medical expert who recruits group of experts with diverse identity and ask them discuss and solve the given medical query.
Now, given the medical query as below, you need to decide the difficulty/complexity of { {qquetion)>>
You can recruit up N experts in different medical expertise. Considering the medical question and the options for the answer, what kind of experts will you recruit to better make accurate answer?
Also, you need to specify communication structure between experts (e.g., Pulmonologist Nconatologist = Medical Geneticist == Pediatrician Cardiologist)
example, you want recruit five experts, you answer can like:
{ {[eammpler eeppeers)) s}}
Please answer in above format, and do not include your reason.
#
D Additional Results
This section presents additional experimental results and analyses that provide further insights into the performance and characteristics of our MDAgents framework.
Agent initialization prompt
You are [(role}} who {(description)), Your job is to collaborate with other medical experts in team.
Agent interaction prompt
Given the opinions from other medical agents in your team, please indicate whether you want to talk to any expert (yes/no). not, provide your opinion. {(opinions}]
Next, indicate the agent you want to talk to: ([agen__list)
Remind your medical expertise and leave your opinion to an expert chose. Deliver your opinion once you confident enough and in way to convince other expert with short reason.
22

---

Final decision prompt
You are a final medical decision maker who reviews all opinions from different medical experts and makes the final decision.
Given the {(inputs)), please review the ((imputs)) and make the final answer to the question by (tdecision_methodd/) (e.g., majority voting, ensemble refinement). *#nswer:**
Report Generation
Given the {fagent answers)}, please complete the following steps:
Take careful and comprehensive consideration of the provided reports.
2. Extract key knowledge from the reports.
3. Derive comprehensive and summarized analysis based on the extracted knowledge.
Generate refined and synthesized report based your analysis.
D.1 Accuracy on entire MedQA 5-options Dataset
To provide comprehensive evaluation of our approach, we conducted experiments on the entire MedQA 5-options dataset using GPT-4o mini. This expands upon the subsampled experiments presented in the main experiments Table methods.
Below Table shows the accuracy results for various
Table 5: Accuracy (%) entire MedQA 5-options dataset with GPT-4o mini
Category
Accuracy
71.5
Single-sgent
3ochot
72.3
CoT -CTTSC
76.6
77.2
Malti (Multi-medel)
Weighted
Mijeriy Voring
76.3
Bunda Count
Multi-agnnt (Singlo-model)
Adaptive
MDAgents
These results demonstrate that our MDAgents approach outperforms both single-agent and other multi-agent methods across the full dataset, achieving an accuracy of 83.6%. This underscores the effectiveness of our framework in handling diverse medical questions at scale.
D.2 Estimated Costs for Full Test Set Experiments
To provide transparency and aid in reproducibility, we estimated the costs associated with running experiments on the entire test sets using GPT-4 (Vision). Below Table presents these cost estimates in USD for various datasets and methods.
Table 6: Estimated costs for experimenting with entire test sets with GPT-4 (Vision) (in USD)
Method
MedQA
PobMedQA
Path-VQA
PMC-VQA
DDXPls
SomCal
JAMA
MedBullees
MIMIC-CXR
Total Cost
CoT
55.34
13.16
3,028.54
27.134.00
16.461.90 10
10,503.09
134.55
61.23
1.388.00
58,871 29
Ours
17249
41.36
9,369.45
82,194.34
44,11497
31,17605
367.13
161.20
4.406.90
172,704.33
While our approach incurs higher costs due to its multi-agent nature, the significant performance improvements justify this increased computational expense for critical medical decision-makiin tasks.
23

---

D.3 Impact of Knowledge Enhancement with RAG
Method
Accuracy (%)
MDAgents (baseline)
71.8
MedRAG
75.2
+ Medical Knowledge Initialization
76,0
+ Moderator's Review
77.6
+ Moderator's 's Review & MedRAG
80.3
Table 7: Impact knowledge enhancement on MDAgents performance
We investigated whether simply assigning roles to agents is sufficient for expert-like performance, and explored the impact of equipping agents with different knowledge using Retrieval-Augmenté Generation (RAG). Table 7 presents the results of these experiments.
These results indicate that while role assignment provides a foundation, augmenting agents with specific knowledge (using MedRAG) and structured reviews (Moderator's Review) significantly enhances their ability to simulate domain expertise. The combination of Moderator's Review and MedRAG yielded the best performance, highlighting the synergy between structured collaboration and domain-specift knowledge retrieval.
D.4 Complexity Assignment and Collaborative Settings
To address the impact of complexity assignment on accuracy and API costs, we conducted additional experiments focusing on high-complexity cases, particularly in image+text scenarios. Table 8 shows the results for various collaborative settings.
Collaboration Setting
Accuracy (%)
Sequential & No Discussion
39.0
Sequential & Discussion
45.0
Parallel & No Discussion
56.0
Parallel & Discussion
59.0
Table 8: Impact collaboration settings on high-complexity image+text tasks
These results underscore the importance of multi-turn discussions, particularly in complex cases. The parallel collaboration with discussion yielded the highest accuracy (59.0%), suggesting that enabling agents to work simultancously and engage dialogue is crucial for handling intricate medical queries. The significant performance gap between discussion and no-discussion scenarios (45,0% 39.0% for sequential, and 59.0% vs. 56.0% for parallel) highlights the value of interactive deliberation in medical makinngn processes.
24

---

Table 9: Accuracy (%) on Medical benchmarks with Solo setting. Bold represents the best and Underlined represents the second best performance for each benchmark and model.
Knowledge Retrieval
Category
Method
McdQA
PuhMuiQA
Pat--QQA
PMC-VQA
MedViiQQ
Zeno-sho
48.5 2:33
56.8
13.0
Few-shot
47.8 1: 16.4
59.0
1
1.0
G.T-3.5
CuT
54,2
69.7
ER
60.2 44
$2.9
Medprompt
00.1 108
59.8 1.83
Zero-shor shor
75.0
61.5
57.9
49.0 8:30
GPT-KV)
82.5
57.6
COT-SC
83.9 1:3.7
58.7 1:50
61.2 1:11
$0.5 17
ER
81,9
$6.0
61.4 4:41
$2.7
Medprompt
Few-shot
34,0 7:2
$3.0 0.0
64.3
48.2 1.
47.1 #:
Gamini-Paan Piot Viskos)
Col
50.0 bd
00.2
9.0
66.4 11.7
47.1 4:42
48.5
4:53
SC
52.7
55.8
63.6
46.3
49.2
52,0
57.6
Medprompi
45.3 1:11
50.6 1 5.4
$5.0 1: 7.0
41.8 2
44.5 1.70
Clinical Reasining Diagnostic Datasets
Category
DDXPlus
SymCat
JAMA
MedBallets
MIMIC-CXR
Few-shot
86.0 2:2
4:42
59.0
GPT:3.5
Col
528 5.4
82.0 ::00
34.0 24
56.0 5.2
63.0
1.
58.0
Modgrampt
41.2 2.62
860 1:77
43.0 14
$4.0 1: 57
Zern-sbot
70.3 1.20
88.7 1.35
62.0 1.00
67.0 1.14
400 13
Few-shot
69.4
86.7 2:11
69.0 4:42
72.0
35.3
GFE-(V)
SC
CaT [3
SM
R3.3 3:1
28
70.0
76.0 2:28
4.0
BR
07
61.3
82.7 1:23
71,0 1:1.
76.0 1:37
500
Medgromp:
20
39.5
$7.3
71.0
Zeru-sbot
49.9 6:65
83.9 2:64
42.7
40.0
40.0 2:
89.2
Gemini-Proo(isinn
CaT
B3
65.5 1.49
91.9 1.14
38.0 1.
$2.7 1.71
45.2 1.4
Col-SC
82
60.3 1:24
92.0 1:15
46.0
$1.0
$49 1:1A
Medpcomp:
58.2 4:55
92.5 4:45
44.4
$4.0 -
Chain-of- SC: Self-Canistmey. Fasemble Refinement
I012 only. ① image+ten, O videoticar
All experiments rested randor scads
25

---

Table 10: Accuracy (%) Medical benchmarks with Group setting. Bold represents the best performance for each benchmark and model.
Medical Knowkegge Retrieval Dutasets
Category
Methed
McdOA
PubMedQA
Path-VQA
PMC-VQA
MedViaQA
Majority Veting
60.4
68.5 MB
GPT-3.5
Weighed Varing
55.3
70.2
MedAAgmis
56.0 5:53
$5.0 d: 1.4
Majority Voting
80.5 1:34
72.2 1.64
56.9 191
36.8 6.67
GPT-4VV
Weighted Voking
78.8
66.9
72.2
62.1
27.9
25.4 =99
MedAgents 14
79.1 1-4
69.7 4.3
45.4
39.5 300
Majority Noting
$1.6 2:2.2
65.3 12.9
58.2 6:73
27.1 =:54
50.8 7.4
Gamini-Poot Prot Vision)
Bonda Count
Weighted Voting
19.4
52.3 +:333
57.7
63.7 130
66.4 11.
68.
25,3
20.9 8:38
54.5
57.8 #:2
MedAgents
48.4
Reconcile
57.5
31.4
Multi-ggent
AutoGion
60.6 1.50
77.3 1:23
43.0 1.89
37.3 1.
DyLAN 51
64.2 2:3
73.6 1:42
41.3 1:12
34.0
Meti- ProtiPro
Clinical Reasoning and Diagnoslic Datasets
Calegory
Method
DDXPlus
SynCat
JAMA
MedBallets
MIMIC CXR
Majority Vating
$3.6 1:6-+:20 22
83.7 +:33
47.0 +:00=10
66.0 + 4.1
GPT-3.5
Weiglted Votiag
85.9
43.0
ModAgeans 22
47.3 1. 11.0
87.0 1:42
41.0 1:
56.0 1:42
Majority Voting
67.8 1:49
91.9 1:22
70.0 5:52
70.0 t:00
49.5 II1
GPT-KV)
Hoeda Count
Weighted Vating
65.9 1.1%
90.5 1.79
66.1 1:41
66.0
45.3.1
53. 1.
ModAgents b2
62.8
90.0
37
77.0
2.70
Majority Vating
$2.3
73.5 1:49
44.0 1:42
1.4
Gemini- Prod Vision)
Weighted Noting
54,3 +: 163
64.6 6:65
42.0 32
43.6 : 3.9
43.2 2.0
Borda Count
67.0
77.3
37.0
46.1
44.6
MedAgents
Reconcile
90.6
$9.5
Multi-agent
AutoGen 86
67.3 11.3
73.3 +:3.1
64.6
55. 3.1
433 442
DyLAN 51
56.4 1.29
75.3 1:46
6011 1.
57.3 1.6.1
38.7 1:2
Meta-
1V
77.3
493
CoTl: Chain-of-Thoughh, SC: Self-CCnniisteeee Ensemble Refinement
teu--only. ① Imugestex, videosical
All experiments were tested with random scads
Table 11: Accuracy (%) on Medical benchmarks with Our method. Bold represents the best performance for each benchmark and model.
Medical Knowledge Retrieval Datasets
MedQA
PuhMadQA
Path-QQA 00
PMC-VQA
McdViaQA
GPT-33.5
64.0
66.0
88.7
56-4
Gemini-Prot Pisiiin)
57.
72.0
62.2
56.2
Clinical Keasoning and Diagnostic Datasets
Method
DDXPlss
SymCli
JAMA
MedlBullets
MIMIC-CXR
GPT-335
62.5 =z 6.7
85.7 30
48.6 6.5
$5.3 4:4
GPT-4()
77,9 3.1
93.1 1/0
70.9 0.
80.8 3 17
55.9 9.1
Geminn-ProoVision
39.2 1.3
65.0 kl
47.0 1.3.
42.9 1. MM
48.1 1.5
CoT: Chain-dfTTroghh. SC: Sclf-Canisstenny ER: Fisemble Refinement
tetz-only. image+ten, vidcotiosr
experiments randor
26

---

Table 12: Accuracy (%) on Medical benchmarks with Solo/Group/Adaptive settings with increased number of samples (N=100). All benchmarks except for MedVidQA (Gemini Flash) were
evaluated with 40 mini.
Category
Methed
McdQA
PuliMoQQA
Path-VQA
0
PMC-VQA
MedVidQA
O
$0.0
Few-shut
77.0
58.0
50.0
51.0
Single agent
+ CoT
78.0
$0.0
59.0
$1.0
$3.0
-CoT-CC
79.0
53.0
5300
52.0
Medpcompt
79.0
58.0
60.3
540
330
Majority Voring
79.0
68.0
63.9
$1.0
$4.0
Multi-agent
Weighted Voring
90.0
68.0
64,0
51.0
15.0
(Single-mode)
Bunla
50.0
$20
MedAgents
52.0
Meta Prompting
82.0
69.0
56.0
49.0
Reconcile
R3D
2000
558.
35.17
(Mulii model)
DyLAN
68.0
67.0
42.0
40.0
48.0
Adaptive
MiAgennss (Ours)
DDXPlus
SymCCr
MedRRullets
MIMIC-CXR
Category
Zenrshol Few shot CoT
84.0
57.0
49.0
38.0
60.0
$2.0
58.0
$2.0
33.0
Single-agent
66.0
$1.0
55.0
61.0
33.0
COT--S
FR
76.01
80.0
56.0
59.0
43,0
Madprompe
70.0
84.0
62.0
60.0
43.0
Majority Vosing
53.0
82.0
56.0
59.0
$4.0
Weighted
(Single model)
agent
Med-Agents
Count
56.0
80.9
86.0
$1.0
59.0
58.0
51.0
40.9
Meta Promping
53.0
79.0
56.0
51.0
48.0
Reconcile
87.0
59.0
60.0
Multi-agent (Multi-mode)
AutGGen
47.0
$2.0
$3.0
55.0
47.0
DyLAN
51.0
81.0
55.0
57.0
42.0
Adaptive
MDAgents (Ours)
75.0
89.0
59.0
67.0
56.0
CoT: Chain of Thought SC: Self Consistency, ER: Eusemble Refinement
inagetteu, videoeicat
27

---

MedQA
PubMedOA
DDXPlus
20.0
27.5
15.0
25.0
20:
12.3
12.5
Count
10.0
26.0-
50-
5.-
low
moderate
high
low
moderate
high
low
moderate
high
MedBullets
SymCat
JAMA
15.
Count
Count
10
25-
20
10W
moderate
high
low
moderate
high
10W
moderate
high
PMC-VOA
Path-VOA
MIMIC-CXR
38
27.5
15.0
25.0
Counte
12.5
22.5
CourtE
10.0
20.0
1.5
28-
5.0
25
0.
low
moderatt
high
low
moderate
high
low
moderate
high
MedVidOA
Count
high
Figure 8: Complexity Distribution for each dataset classified by GPT-4(V) and Gemini-Pro (Vision) (for MedVidQA). The plot illustrates the varying levels of medical complexity across datasets, reflecting the diverse nature of medical question answering, diagnostic reasoning, and medical visual interpretation tasks. For instance, MedQA is categorized under Medical Knowledge Retrieval due to their focus text-based questions and literature synthesis, while MIMIC-CXR categorized under Clinical Reasoning and Diagnostic tasks, shows high complexity distribution due to the need for interpreting detailed radiographic images (See Section Section 4.1 for the task categorization)
28

---

MedQA
PubMedQA
DOXPlus
5-
4-
Pedistriciin
Endoctinologia
Peychiatrstt
Radiologist
Cardiovasculat Surgeon
Medical Oncologist
Pulmonslogitt
Cardiologist
Neurologid
MedDullets
SymCat
52
Cardiologist
Neurologist
Pulmorologist
ID Specialist
Neurolegist
Gastroenteroloiist
Dermatologist
Ophthallmologist
Radiologist
PMC-QQA
Path-VQA
MIMIC-CXR
17.5
15.
25.0
10.0
75
10
75
25
0.
Radiotogist
Puthelogist
Heunllopec
Pathologist
Radlologist
Orthopedic Surgeon
Blomed Informatics
0.0
Radiologist
Pulmanologist
MedVidOA
text-only
image + text
video text
Neurologict
Orthopodic Surpeon
Physical Therapist
Figure 9; Top-3 most recruited medical experts in each benchmark. The alignment between the dataset characteristics and the recruited experts is evident in several cases. For instance, MIMIC-CXR, which features chest x-ray images, predominantly recruits Radiologists, Pulmonologists, and experts Biomedical Informaties due their expertise interpreting medical imaging.
Moderator
Morterator
Teams
Moderator
Chincian
Clridon
Clincian
Chrison
Clrician
Chricon
(a)
(c)
(d)
Figure 10: Simplified agent structure examples assigned during the expert recruitment process ranging from (a) Primary Care Clinician (PCC), (b) Multi-disciplinary Team (MDT), (C) MDT w/ hierarchy to (d) Integrated Care Team (ICT).
29

---

Algorithm Adaptive Medical Decision- making Framework
Require: Problem
I: Complexity COMPLEXITYCHECK(()
Determine the complexity of the medical query
2: if Complexity = low then
3: Agent + RECRUIT(Q, Complexity))
Recruit a Primary Care Clinician agent
4: ans +- (Q)
5: else Complerity moderate then
MDT < RECRUIT(Q, Complezity)
Recruit a Multi multi-disciplinary Team
7: Agent + RECRUIT(Q, Complexity, MDT)
8: r
9:
Consensus False
10: Interaction +
11:
while r S R not Consensus do
12:
Conserssus, Log t- COLLIBORATVVDDSCCSIIONQQQ..
Iterative discussions
13:
if not Consensus then
14:
for all Agent M DT do
15:
Feedback t Moderator (Interatot Agent) Agent. UPDATE eedback)
Review and provide feedback b Update the feedback
16:
17:
end for Interaction Interaction + [Log] + [Feedbock] end
18:
19:
20:
r +- 1+
21:
end while
22:
ans + Agent(Q, Interaction)
Moderator agent makes the final decision
23:
else
24:
ICT RECRur((
Recruit an Integrated Care Team
25:
Reports + []
26:
for Team E ICT do
27:
Report GENERATEREPORTIO. Team)
Each Team curates a report
28:
Reports +- Reports [Report]
29: end for
30:
ama + Agent(Q, Reports)
Final decision made
31:
end if
32:
return
30

---

E
Case Study
E.1
Medical Decision Making Case Studies
MDM requires efforts of both individual expertise and collaboration to navigate the complexities of patient care. Clinicians often face challenging scenarios that necessitate comprehensive approach. integrating insights from various specialties to arrive at the best possible outcomes.
E.1.1 Real- World Medical Cases
Below are the real-world example cases that could be classified as low, moderate, to high complexity cases.
Case 1: Adjusting Medication Dosage for Chronic Disease (Low Complexity) A 55-year-old female patient with type 2 diabetes visits her PCP for routine check-up. The patient has been taking 500 mg of metformin orally twice a day and has been adhering to a low-carbohydrate diet. Upon testing with fasting glucose level, the glucose level is above normal. PCP reviewed the current medical dosage and increased the dosage to manage the blood glucose level of the patient.
Case 2: Differential Diagnosis in the Emergency Department (Moderate Complexity) A 40- year-old male patient arrives at the emergency department (ED) with a high fever, severe headache, and muscle pain, raising concerns about potential infectious disease. The ED physician conducts an initial examination but recognizes the need for a more detailed evaluation to identify the underlying cause. The patient is referred to the infectious disease department for further assessment. An infectious disease specialist, along with the ED physician, reviews the patient's 's symptoms, travel history, and recent exposures. They collaborate on ordering specific diagnostic tests, including blood cultures and imaging studies. Through this teamed decision-making process, they diagnose the patient with dengue fever and promptly initiate appropriate antiviral treatment.
Case 3: Managing Adverse Responses to Medication in Chronic Disease (High Complexity) 60-year-old female patient with chronic heart failure has been experiencing new symptoms of shortness of breath and mild fever, suggesting either complication due to her chronic heart failure or new infection. The urgent care doctor identifies the severity of the situation and promptly refers the patient to the emergency department of large hospital, where the patient has triaged to see cardiologist and an infectious disease doctor for specialized care. The team conducts a detailed review of the patient's medication history and current symptoms, does a physical exam to listen to lung sounds, and orders few exams including labs, a chest x-ray, echocardiogram, and electrocardiogram. The team identifies that the patient has pulmonary effusion and upper respiratory viral infection.
E.1.2 Medical Cases from MedQA Dataset
Now, let us look at the cases from the MedQA 35 dataset that illustrate either individual PCP or teamed decision-making crucial in managing medical conditions, ranging from low to high complexity levels of potential cases. These examples highlight the importance of checking the complexity of the case for proper management.
Case 4: Diagnosis by PCP The case below with the "Low Complexity" header is classified as low complexity by medical doctor. In this case, PCP can answer this question without consulting gastroenterologist. The diagnosis of gastric cancer and management based on the manifestation of the disease, that has been described in this question and beyond should from gastroenterologist. However, PCPs are expected to have the basic scientific and pathophysiologica knowledge that is related to gastric cancer and use that knowledge to solve this problem.
Case 5: Diagnosis and management by single Pediatric Endocrinologist The case below with the "Moderate Complexity" header classified as moderate complexity by the medical doctor. In this case, a pediatric endocrinologist (specialist) alone can diagnose patient and have treatment plan. Note that this patient could have been referred to this pediatric endocrinologist by a PCP who is regularly seeing this patient.
31

---

Low Complexity
Question: A ye-yeor-old man comes to the physician because of 4-mouth history of epigastric pain, nausea, and weakness. He has smoked one pack of cigarettes daily for 50 years and drinks one alcoholic beverage daily. He appears emaciated. He is 175 em (5 ft in) tall and weighs 47 kg (103 lb); BMI is 15 kg/m². He is diagnosed with gastric cancer. Which of the following cytokines is the most likely direct cause of this patient's 's examination findings?
Answer:
A) TGF-8
B) IL-6
C) IL-2
D) TNF-8
Moderate Complexity
Question: 5 S-yerr-old girl is brought the clinic by her mother for excessive hair growth. Her mother reports that for the past 2 months she has noticed hair at the axillary and pubic areas. She denics any family history of precocious puberty and reports that her daughter has been relatively healthy with an uncomplicated birth history. She denies recent illnesses, weight change. fever, vaginal bleeding, pain, medication Physical examination demonstrates Tanner stage 4 development. pelvic ultrasound shows an ovarian mass. Laboratory studies demonstrate an elevated level of estrogen. What the most likely diagnosis?
Answer:
A) Granulosa cell tumor
B) Idiopathic precocious puberty
C) McCune Albrightt syndrome
D) Sertoli-Leydig tumor
Case 6: Diagnosis and management by multidisciplinary team The case below of "High Complexity" patient primarily having neurological symptoms but with problems with vision, which requires neurologist consult to ophthalmology department for further evaluation.
High Complexity
Question: A 63-ycar-old woman presents her primary-care doctor for 2-month history of vision
changes, specifically citing the gradual onset of double vision. Her double vision is present all the time and does not get better or worse throughout day. She also noticed that she hard keeping her right eye open, and her right eyelid looks 'droopy' in the mirror. Physical exam findings during primary gaze are shown in the photo. Her right pupil is 6 mm and poorly reactive to light. The rest of her neurologic exam is unremarkable. Laboratory studies show an Hb Alc of 5.0%. Which of the following the next best test for this patient?
Answer:
A) Direct fundoscopy
B) Intraocular pressures
C) MR angiography of the head
D) Temporal artery biopsy
E.2
Cases Studies with MDAgents
In this section, we provide two examples of our framework with moderate (Figure 1T and high (Figure 12) complexity in PMC-VQA (inage+text) and DDXPlus (text-only) respectively. These
32

---

Complexity Check
Step
Initial
Mysic pome sumorahuta??
Insernadiate
IR) infarmation
PC Arower Shooss
tansehaage
Of
Sagann
HQ Artwer hamuefagee
harantage
Collsborative Discussion
Round
Turn
Turn
14 DI
Turn
for vatoc codioallooiiiii conditons. i hawe edeenve
are surgee ioooonooo scedaaove surgical
kurgical canetona bolading canar: dounm.. uour
facton. roull mautt, Seedin diaccars cartan
Hoplow aga nove niler cuttage
snled vesea aiu sciidy
numourde ene ahet fesuee this come stent spper
hamantaage
blaeding
bloo wark com cau du ca vanady facars. of cane rage ras speen R.
Lar confoart toccuue save ecanaiv ssperiane actoiin thearig
mosemaen or sigo.
ridtlogia se toques serpebsiiool
ciam an cncalegen whe elve the o eeghcd and testmer of carar
ultess sciarann siparance dlgg
Law wadiie splice our san
arows ni dicips skood dicce seaied.
ceraratake ritraate
tudl pou the bo ancure
repert,
Lur anciaal slacrec in mage shees 1nd lice in tue
hamemmppe
viiic ans benign suncia corppced of bara boiolla
off peommee eagereerg...n Buoel
care cuuc specaly sopaars faank
other me expert, nen bellor
unte luteer but fellvee that modeen
youree
dispnest dieene through
dboen mon iisepriati megin fastoon
the mestre
coeunes of
suppor my caliion
Step Review and Final Decision
duction chaling "g inliciiniiieeetasii mags, rrcuear
Interaction Leg
anow preciin, ottn dencred oxeentoos viich dhed whet maps veems dipllyy
anhormonn, hemoore informe rodicaaioon vaste tomen crtociere
sudeeded
Thenlor
neleone und doe titeeehhe
Aumet cherge oi Hovea
qudkitte the pooe gl fask an wemortage waher than ca werched Agent
and Aggal yourch ge woeenn wi Trucc
Figure II: Illustaaioon of our proposed framework in moderate complexity setting. Given a medical query (image + text) the framework performs reasoning in five steps: (i) complexity check, (ii) expert recruitment, (iii) initial assessment, (iv) collaborative discussion, and (v) review and final decision-making. Green text represents the correct answer and the Red text represents the incorrect answer.
case studies reveals how our framework provides an environment for agents to collaborate, gather information, moderate and make final decisions in complex medical scenarios.
33

---

Quenterr the brloving informal piaar indicate most deggina
Cympleiity Check
Expart Racruitment
Mysthere gome
oo complettt the
(31 Meocarte
(5. sut deral tther surcalere
CA Chapae
UT: www.:r/aeaeeee....o.o
Stop S. Informattin Gathering and Report Generation
Herrstuge busk dppit seecoges mheaest
Nersa yat
seltoonss suue eaco Jedaplc coed disartens attualyy
ow creado by adleit
canvia poi der Spartelu Joae sabed mustcoute exllading
bload lajue eaiyal--i andl taal daginiss
bo prean cole opect tuney,
The
lepaber caat modieii
dublniing by soodud
Compre offecve
The tocing hase pallso aed only kupper Musat
bend mude
lubie
Assesament (AT)
Summariaed Repprr
eente Care altu Lseed Cleaa Can Coveciatle Futioall Ancroutte
Clmbul Cana Cannicater
dl dhens an metate felypm
ronsplyy resumes
tinyy steut paeemua
condees
encuu hos conern degenoo cin nolule ank de
bey sepiciriie can
paleed sug resee pose Loadd
nes doe solenial acters basccl
teyliaatoos the A.kowah og seternuu siliy somes
Repurd cabeed our ultonaphoryy deer La
the regining
Review (FROT)
4. Final Decition
Dombrned Raport
the mushee olbudd snalorr Hover pas aibudd ope (deat coude
Illustration of our proposed framework in high complexity setting. Given medical query (text-only) the framework performs reasoning in four steps: (i) complexity check, (ii) expert recruitment, (iii) information gather and report generation, (iv) final decision. Green text represents the correct answer
Figure 12:
F Medical Complexity Comparison with Human Physicians
The core premise of our MDAgent framework is its ability to adapt to the complexity of medical tasks. To validate this approach and gain insights into how LLMs perceive medical complexity compared to human experts, we conducted an annotation study. This study aimed to explore the alignment between LLMS and physicians in assessing medical question complexity, a critical factor in the effectiveness of our MDAgent framework.
Study Design We selected 50 representative questions from the MedQA dataset, ensuring a balanced representation across USMLE steps 1, 2, and 3. This selection process aimed to cover wide range of medical topics and complexity levels, mirroring the diverse challenges that our MDAgent framework is designed to address.
Three physicians participated in our study: two with two years of Internal Medicine training (Post Graduate Year 2, PGY-2) and one general physician. This composition allowed to capture range
34

---

of clinical perspectives. The physicians rated each question on scale of -1 (low complexity), (moderate complexity), and (high complexity).
Inter-rater Reliability To quantify the agreement among our physician raters, employed Intraclass Correlation Coefficients (ICC). ICC is a widely used statistical measure in medical research for assessing the consistency of ratings among multiple raters. We specifically chose two ICC variants:
ICC2k (Two-way random effects, average measures): 0.269 [-0.14, 0.55]
ICC3k (Two-way mixed effects, average measures): 0.280 [-0.15, 0.57)
ICC2k was selected because assumes our raters are randomly selected from larger population of similar raters, allowing for generalization of our findings. ICC3k, on the other hand, treats the raters as fixed, focusing on the consistency among our specific set of physicians.
Both ICC values indicate moderate agreement among the raters. This level of agreement reflects the inherent complexity and subjectivity in evaluating medical questions, even among trained pro- fessionals. It also highlights the challenging nature of the task our MDAgent framework aims to address.
Annotation Interface To facilitate the annotation process for both physicians and LLMs, developed a specialized interface. This interface was designed to present medical questions in clear and consistent manner, allowing for efficient and standardized complexity ratings. Figure 13 shows screenshot of the annotation interface used in our study.
MDAgents
Home
Please your
questions
Linqqu ID:
Inscou.
Instructions:
Given the medical questions telow, return the medical floulyylo mplesity each queston, please select among the followingg optinn:
1. Low: PCP general ultysicion can anower this cuestion without oor corsulingg a spocialot
Moderate: general ptysician answer guestion lution with specialist, single spocialist questin.
High: mut--departmmttal specialists
requines
the (Itequires
casa).
Quesdion 47 fomale thyroddectoyy for traatmnnt Graves"
speaking You
kuryy compliaation her moiit surgery. What the ambryningic origin demaged nere most liwyyy cassing this palient's hrommeeeee?
pharyngeal
phoryngeat arch
(C) ard pharyngeal arch
(D) 4th phe ehrryngoal arch
fih paweyngen anch
complesitt questtoo::
Moderato
High
Figure 13: Annotation interface used for medical complexity assessment. The latest version can be found at atttps:/dxxagnts github..h..... httpabb..i//22//////////mmgggg//
LLM Annotations and Comparison To compare LLM performance with human expert judgments, employed several state-of-the-art models to annotate the set of questions. We then compared these assessments with the majority opinion of the physicians, determined by the mode of their ratings (or the mean in cases of complete disagreement).
Table [3 presents the Pearson correlation between each L.MM's complexity ratings and the physicians' majority opinions:
35

---

Model
Correlation with Physician Majority
got-4o-mini
-0.090
gpt-4o
0.022
gpt-4
0.070
gemini-1.5-faan
0.110
Table 13: Correlation between LLM complexity ratings and physician majority opinions
The results of our study provide valuable insights into the current state of LLM capabilities in medical complexity assessment and underscore the importance our MDAgent framework:
Subjectivity in medical complexity: The moderate ICC values among physicians highlight the inherent subjectivity in assessing medical question complexity. This finding validates our approach in MDAgent, which doesn' 't rely on a single, fixed assessment of complexity but rather adapts its collaboration structure dynamically.
2. Current LLM limitations:
The low correlations between LLM and human assessments indicate that current LLMs may not fully capture the nuances that human experts consider when evaluating medical complexity. This observation reinforces the need for our MDAgent framework, which can compensate for individual LLM limitations through collaborative decision-making.
3. Potential for improvement: The variation in correlation across different LLM models (from -0.090 0. 0.110) suggests there room for improvement in LLM performance. This aligns with our MDAgent approach, which can leverage the strengths of multiple models and adapt future improvements in LLM capabilities.
4. Value of human expertise: The discrepancy between LLM and physician assessments derscores the continued importance of human medical expertise. Our MDAgent framework acknowledges this by incorporating human-like collaboration strategies and the potential for human oversight in critical decisions.
5. Adaptability of MDAgent:
The challenges revealed in this study highlight the wisdom of our MDAgent's 's adaptive approach. By dynamically adjusting its collaboration structure based on perceived task complexity, MDAgent can mitigate the limitations of individual LLMs and approach the nuanced understanding demonstrated by human experts.
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction state the primary contributions the paper, including the introduction of the MDAgents framework, its adaptive decision-maker structure, and the significant performance improvements demonstrated through experimental results.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes section on limitations, discussing the scope of the framework, and areas for future improvement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
37

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
The paper does not include theoretical results that require assumptions or proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The paper provides comprehensive details on datasets, experimental setups, and methodologies used, ensuring that the results be reproduced accurately.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
38

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The paper includes link to the code repository and provides sufficient instructions in the supplemental material to reproduce the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co.
www.bmmmsssiooooccc...... for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. www.cccpwccccccccceeceoeooooo gttc://wiissseeeee wwwweoewwisssoonnooooooooooooeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper specifies all relevant experimental details, including data splits, number of samples, and number of seeds, ensuring transparency and reproducibility of the results.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports eгтоr bars and includes information on the statistical sig- nificance of the experimental results, ensuring clear understanding of the variability and reliability of the findings.
Guidelines:
The answer NA means that the paper does not include experiments.
39

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The experiments primarily involved inference using API calls to GPT-3.5. GPT-4 (V), and Gemini-Pro (Vision). The type of compute workers, memory, and time of execution are managed by the API providers (OpenAl and Gemini). Details about the number of API calls and the specific configurations used for each experiment are provided, ensuring reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research adheres to the NeurlPS Code of Ethics, ensuring responsible conduct throughout the study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
40

---

Answer: [Yes]
Justification: The paper discusses potential societal impacts, both positive and negative, including the improvement of medical decision-man -aaiin and the risks associated with misuse or biases in the Al system.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The paper discusses potential limitations, including the risks of medical hallucinations and false knowledge. It includes guidelines within the main text to ensure responsible use of the MDAgeetss framework and publicly available datasets, emphasizing ethical usage and adherence to best practices.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper properly eredits the creators of existing assets used and clearly states the licenses and terms of use.
41

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
42

---

Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
43


Reviews:
[{"review_id": "rDBKzq13Op", "reviewer": "Reviewer_g9qW", "summary": "The paper presents MDAgents, a multi-agent LLM system for answering medical questions, ranging from medical question answering and diagnostic reasoning to medical visual interpretation. The main novelty is a crafted collaboration scheme of multiple agents with designated roles, where a medical question is categorized into being of low, medim or high complexity. The question is then delegated to either individual specialized LLMs or teams of LMMs, and finally a decision is taken by a last LLM. The complete system does not focus on training or fine-tuning specialized medical agents, but rather use an appropriate foundational model and designated prompts to design the individual agents.\n\nThe approach is evaluated on ten medical diagnosis datasets, where 50 samples are used for testing. The authors compare their adaptive multi-agent system with a solo agent as well as a fixed group of agents, each with respective SoTA approaches. The results are based on GPT-4(V) or Gemini-Pro(Vision). The results show that MDAgents is either on-par or better than its competitors for all datasets.", "questions": "- Please explain why, for the evaluation, you use 50 samples for inference and why it suffices / not have an impact on the relative results to the competitors.\n\n- Please better explain, also in the paper, what a multi-party chat (mentioned in table 1) would look like for specific tasks. The case studies with MDAgents in the appending is insightful, but a better formalization/presentation of the topic in the main paper would be helpful.\n\n- Are the expert discussion always one-to-one? If so, will there always be exhaustive one-to-ones?\n\n- How dynamic is the system with respect to generating teams of experts of high severity cases? The meta analyses show that more agents are not necessarily better and that the system calibrates the number of collaborators, but how is it done in detail?\n\n- Are the individual expert LLMs also variated (in terms of prompts)?\n\n- Why was MedVidQA evaluated with Gemini-Pro(Vision)?\n\n- To be sure my understanding is correct: are the individual LLMs mostly zero-shot learners? For few-shot learners (maybe only in the low complexity setting): how many dataset-specific examples, if any, are used for prompting in the evaluation?\n\n### After author response ###\nI appreciate the detailed answers. After additionally taking into account the answers and other reviews, I increase my score to accept, as the claims are now clearer backed up.", "limitations": "The authors discuss limitations to sufficient degree, including the lack of comparison to human clinicians or, most importantly, the danger of using systems that possibly hallucinate in critical situations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Important use case: LLMs for medical decision making have large potential to add value, as they might support physicians or directly patients.\n- Novel idea: The paper makes a good case in comparing the main idea to relevant related works, emphasizing how the combination of used concepts and their grouding in the respective agent roles is new. The related work section is sufficiently structured and broad and shows how the current work extends prior work.\n- Good evaluation setup in terms of used datasets and competitors: The chosen dataset range is sensible and all relevant competitors, as discussed in the related work section, seem to be evaluated against. \n- Strong results: The approach beats the other SoTA approaches in 70% of used datasets, while being on par for the others. Since the chosen datasets not only use medical question answering, but also vision problem settings, this is a strong result.\n- Sufficiently deep discussion of results: The authors discuss the results in sufficient depth, highlighting reasons for the added value of using the novel multi-agent setup, especially categorizing the severity of cases by an initial agent.", "weaknesses": "- The paper could me more specific on method descriptions, e.g., multi-party chat / collaboration / report generation: The description is quite generic in the main paper and one only finds examples in the appendix. It is left open for me what implications the unknown design decisions have on the performance of the system. The same holds for, e.g., how the optimal number of collaborating agents is chosen on a case basis.\n- The paper only uses 50 samples for inference without argumentation: It might be fine if other works to the same, but it should be clearly stated why 50 samples where chosen. Maybe the close competitors, e.g., MedAgents, do the same? What percentage is this for the respective datasets?\n- Possible missing details on the evaluation setup: It might be that it is obvious or simply not used, but I am unsure when and where zero-shot or few-shot prompting is used. I see it for the low complexity cases for the recuriter LLM if the approach and also for solo competitors in the evaluation. Is it not used at other places? Extending on this point, maybe it would be helpful to emphasize such a point in the paper or point to it in the appendix for all used, established prompting strategies. If none is used in additition to the role descriptions and collaboratin schemes, it might be also worth noting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful review and the recognition of our paper’s potential contributions to the field. Your insights are invaluable in guiding us to enhance our work and clarify the findings.\n\n**W1. & Q2. Lack of detailed descriptions and examples**\n\nWe recognize the importance of providing more detailed descriptions to clarify our methods, particularly in the areas of multi-party chat, collaboration, and report generation. In the revised version, we will enhance our explanations and include visual aids for better understanding.\n\n* **Number of Agents:** We will explain in Section 3.4 how the recruiter LLM determines the optimal number of agents, considering task complexity and expertise, as outlined in Appendix C.2.\n\n* **Report Generation:** We will expand on the report generation process in Section 3.4 by including the specific prompt used and detailing how information is synthesized from multiple agents, building on the example in Figure 12.\n\n* **Multi-Party Chat and Collaboration:** We will provide a more comprehensive description of agent interactions and collaboration dynamics in the main text, focusing on their role in fostering effective teamwork, as illustrated in Figure 11.\n\n**W2. & Q1. Why use only 50 samples in the experiments?**\n\nWe selected 50 samples to maintain consistency with similar studies, such as [1], which employ a sample size 50 for evaluation. This decision was made to ensure a manageable yet statistically significant analysis with three random seeds, aligning with our experiment budget constraints (experiments cost listed in Table 3 in attached pdf file).\n\nIn our experiments, we utilized the gpt-4 (vision) model, which is approximately up to 200 times more expensive than the gpt-4o-mini model. Given this consideration, we increased the number of samples to 100 with gpt-4o-mini and reproduced our experimental results to provide a more comprehensive evaluation (Table 1 in attached pdf file).\n\nIn particular, for MedQA, we evaluated the entire test set to demonstrate that performance trends remain consistent across different models, thus reinforcing the reliability of our findings (Table 2 in attached pdf file). We believe this approach balances practical constraints with the need for rigorous evaluation and hope this addresses your concerns.\n\nAdditionally, it is important to note that our use of multiple datasets reflects our intention to test the multi-agent system across various medical scenarios, acknowledging the diversity and complexity of medical diagnostics. By selecting a smaller number of samples from each of the ten datasets, we aim to capture a broad spectrum of medical conditions and scenarios, making our study comparative and comprehensive relative to others..\n\nLastly, the percentage that these 100 samples represent of the total test set sizes varies, which can be calculated based on Table 5 in our Appendix:\n\n* MedQA: 0.0078%\n* PubMedQA: 20%\n* DDXPlus: 0.074%\n* SymCat: 0.027%\n* JAMA: 6.56%\n* MedBullets: 32.47%\n* MIMIC-CXR: 6.53%\n* PMC-VQA: 200%\n* Path-VQA: 0.00295%\n* MedVidQA: 64.52%\n\n**W3. & Q7. Details missing in evaluation setup and prompting strategies**\n\nIn our framework, we employed a 3-shot setting for low-complexity cases where PCP agents make decisions. For moderate and high-complexity cases involving multi-LLM agents, we used a zero-shot setting. We will update our experimental setup in the revised paper to clearly indicate where zero-shot and few-shot prompting are used, and we will add this notation to the relevant figures. \n\n**Q3. Is expert discussion one-to-one?**\n\nIn our implementation, we allowed the LLMs to decide if they wanted to engage with other agents. Rather than limiting interactions to one-to-one, we facilitated many-to-many discussions, where multiple agents could interact simultaneously. We will add more detailed descriptions in Section 3.4 and clarify this aspect in Figure 2.\n\n**Q4. How does the system dynamically calibrate expert teams for high severity cases?**\n\nThe system dynamically calibrates expert teams based on the complexity of the medical query. Initially, the recruiter LLM determines the number of agents allocated to each team. For the meta-analysis, we fixed the number of agents to assess the impact on performance, rather than allowing the recruiter LLM to decide dynamically. In our implementation, instead of removing the agents during discussions,  we ask each LLM agent to participate actively by contributing when they have relevant insights or corrections. \n\nFor moderate complexity case, agents could engage in discussions to clarify or emphasize points in each round / turn. In high complexity case, a lead clinician agent guided the process, asking assistant LLMs for specific investigations and deciding which agents to consult based on their expertise.\n\n**Q5. Are Individual Expert LLMs Variated by Prompts?**\n\nYes, we assigned specific roles to the LLMs with detailed descriptions during the initialization step. These roles were determined by the recruiter LLM, to give LLM a clear function, which allowed for varied interactions based on the prompts. The initialization prompt is detailed in Appendix C.2, where we show the Agent initialization prompt.\n\n**Q6. Why was MedVidQA evaluated with Gemini Pro Vision?**\n\nWe evaluated MedVidQA with Gemini-Pro Vision because it was the only model capable of effectively handling both text and video inputs with reasonable performance. Alternatively, we can sample frames from the videos and transcribe (e.g. whisper) the spoken text to provide images and text to vision LLMs (e.g. gpt-4v), but Gemini-Pro Vision offered a more integrated solution for video input.\n\nWe hope our responses have addressed your concerns. Please do not hesitate to let us know if any further explanations or updates are needed to assist in the re-evaluation of our paper.\n\n**Reference**\n\n[1] Nori et al. (2023). Can generalist foundation models outcompete special-purpose tuning? case study in medicine.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QKxFGLI8yC", "reviewer": "Reviewer_mREC", "summary": "In this paper, the authors propose MDAgent, a multi-agent framework for medical decision making. In this framework, the complexity of the problem is initially assessed by an agent. Based on this assessment, either a single agent or a group of agents is assigned to solve the problem. The authors evaluate their framework on 10 medical benchmarks, including both text-only and multi-modal datasets. Experimental results show that MDAgent outperforms existing baselines on 7 datasets and achieves high efficiency compared to other multi-agent methods.", "questions": "- The authors could discuss more about the motivation behind the design of PCC, MDT, and ICT, considering that readers may not be familiar with the medical decision-making process. For example, why does MDT contain a multi-turn discussion but not ICT? Shouldn't the most complex questions require more discussion between agents?\n- In section 4.3, the authors discuss Figure 5 in the first paragraph and Table 3 in the second paragraph. However, Table 3 appears on page 8 before Figure 5 appears on page 9, which could confuse readers.\n- The formatting instructions for NeurIPS 2024 state that \"All tables must be centered, neat, clean, and legible. The table number and title always appear before the table.\" However, Table 3 violates this rule.", "limitations": "- Most of the datasets used for evaluation are in multiple-choice question or true/false question format. In real-world scenarios, there are no options for doctors to choose from. Therefore, the authors could include some open-ended questions to simulate realistic applications.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors propose a adaptive multi-agent framework for medical decision making, which dynamically assesses the complexity of each problem and assigns na appropriate group of agents to solve it.\n- The authors conduct experiments on 10 datasets, including both text-only and multi-modal ones, and compare their method with various baselines. Experimental results show that the proposed method outperforms existing baselines on 7 datasets.\n- The authors focus not only on performance, but also on efficiency and robustness, which are crucial for realistic application.", "weaknesses": "- The description of MDT and ICT is unclear. For example, how are the prompts for each specialist prepared (the {{description}} in Agent initialization prompt, Appendix C.2), are they handcrafted or generated from LLM? How does the hierarchy shown in Figure 10 (c) and (d) work?\n- The experiment shown in Figure 3 uses only 10 questions from a single dataset, which is not convincing. Additionally, subfigures (b), (c), and (d) are not mentioned in the analysis. What conclusions can be drawn from these results?\n- The results shown in Figure 5 seem counterintuitive. For image + text and video + text, the score for low is higher than for moderate and high. This suggests that some questions that can be correctly solved with a single agent result in incorrect answers when multiple agents are involved. More analysis is needed to uncover the reason.\n- The experiment incorporating the moderator's review and RAG shown in Table 3 has little relation to other parts of the paper. Considering that the moderator's review and RAG can also be combined with other multi-agent methods, the authors should compare the performance gain when attaching the moderator's review and RAG with different multi-agent frameworks if they want to demonstrate that their method is more suitable for the moderator's review and RAG.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review and the opportunity to address your concerns. We have conducted numerous additional experiments to validate our approach and enhance the robustness of our findings.\n\n**W1. The description of MDT and ICT is unclear.**\n \n* **Agent Initialization:** In our framework, the roles and descriptions for specialists within the MDT and ICT are dynamically determined by a specialized recruiter LLM. This process automates the generation of role-specific prompts, ensuring each agent is precisely tailored to the demands of the case. \n\n* **Hierarchy Explanation:** The hierarchical configuration in MDT and ICT, as depicted in Figure 10 (c-d), is inspired by traditional clinical reporting structures. This protocol ensures structured communication flow and oversight, preventing information discrepancies and fostering coherent team collaboration.\n\n**W2. Experiment shown in Figure 3 with only 10 questions not convincing**\n\nWe acknowledge the limitation of using only 10 questions in Figure 3. Initially, we had 10 questions because this already required generating 300 question-answer pairs (10 solutions * 3 difficulty levels * 10 questions). However, with the current significant reduction in price offered by gpt-4o-mini, we added 15 more questions which results in 750 question-answer pairs.\n\nWe will expand our experiments to include more questions from multiple datasets to provide a more robust evaluation.\n\n**W3. The results in Figure 5 seem counterintuitive**\n\nIn response to your concerns, we have conducted additional experiments and analyses to clarify these outcomes.\n\nFor the image+text scenario, further experiments with different settings has revealed the following results:\n\n* Sequential & No Discussion: 39.0%\n* Sequential & Discussion: 45.0%\n* Parallel & No Discussion: 56.0%\n* Parallel & Discussion: 59.0%\n\nThese results suggest that the integration of multi-turn discussions substantially benefits the decision-making process, particularly in complex cases. The initial absence of this feature in our methodology likely contributed to the earlier lower performance figures.\n\nFor the video+text scenario, the use of the deprecated gemini-pro vision model initially restricted multi-turn chats (multi-modal cases). By summarizing the video content with one agent and then re-initializing another for further multi-turn discussions, we attempted to overcome this limitation. We assume that this approach might have influenced the accuracy in the moderate and high complexity scenarios.\n\nTo thoroughly address these issues and refine our understanding, we plan to further conduct detailed experiments focused on:\n\n* Investigating the impact of different agent initialization strategies on the consistency and accuracy of outcomes (with gemini-1.5 flash).\n\nWe will ensure that these additional analyses are included in the revised manuscript for the camera-ready version, aiming to provide a more comprehensive understanding of the interplay between model capabilities and task complexities.\n\n**W4. Need to Compare Moderator's Review and RAG Integration with Other Multi-Agent Frameworks to Demonstrate Suitability**\n \nOur primary focus was on demonstrating how the MDAgents framework can enhance medical decision-making through adaptive collaboration structures with initial complexity classification. The inclusion of the moderator's review and Retrieval-Augmented Generation (RAG) in Table 3 was intended to show the potential improvements in accuracy when integrating external knowledge sources and structured review processes.\n\nWhile we recognize that the moderator's review and RAG could be applied to other multi-agent frameworks, our aim was to illustrate their effectiveness specifically within MDAgents. However, it's important to note that RAG itself is not a medical-specific technique. We focused on demonstrating how the integration of RAG and structured reviews can be particularly effective within a medical-aware structured adaptive multi-agent system compared to a naive multi-agent system.\n\n**Q1. Discuss more about the motivation behind the design of PCP, MDT, and ICT, and why doesn't ICT include multi-turn discussions?**\n\nThe design of PCP, MDT and ICT represents the real-world clinical decision making processes which is mostly dependent on the complexity of the medical cases (refer to Appendix Section D.1.1. for real-world examples). If the case or task is with low complexity where PCP could solve it without consulting specialists (Case 1, Appendix D.1.1), if it is moderate complexity PCP might have to consult to a specialist agents (Case 2, Appendix D.1.1.), and lastly if the case is complex so that it involves multi-disciplinary consult (Case 3, Appendix D.1.1) we let multidisciplinary agents interact with one another.\n\n**Additional experimental results with ICT setting (Accuracy):**\n\n* Sequential report generation w/ discussion: 84%\n* Sequential report generation w/o discussion: 78% (Our previous approach)\n* Parallel report generation w/ discussion: 82%\n* Parallel report generation w/o discussion: 80%\n\nThe results indicate that incorporating discussions among lead clinicians in ICT enhances decision-making accuracy, particularly in sequential report generation. This evidence supports your point that complex cases benefit from more extensive deliberation.\n\nIn the updated manuscript, we will adjust the ICT model to include more robust discussion protocols.\n\n**Q2 & Q3. The order of Table 3 Appearing Before Figure 5 confuses readers and Table 3 violates NeurIPS 2024 formatting rules**\n\nIn the final version, we will ensure that the figures and tables are presented in the same order as they are discussed in the text to enhance clarity for the readers and revise it to comply with the guidelines in the updated manuscript.\n\nWe believe our extensive additional experiments and clarifications have addressed your queries. Please let us know if further details are required to support the re-evaluation of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ow5vp3MMZA", "reviewer": "Reviewer_mJeV", "summary": "This paper introduces a framework called MDAgents, which optimizes collaboration between multiple large language models (LLMs) for medical decision-making tasks. The main technical contribution of MDAgents is the deployment of a moderator agent to assess the complexity of incoming queries, categorizing them into low, moderate, and high difficulty. To improve efficiency, MDAgents either call single agents to solve low-complexity problems or use several agents to work together using real health studies-inspired collaboration schemes.", "questions": "- MedAgents seems to use the same LLMs for all the agents, with the differences among the agents being the Agent Initialization Prompts. When the Multi-disciplinary Teams were recruited, is assigning an agent a role enough to let the LLM act like an expert in that specific discipline, which requires a lot of domain knowledge? How about equipping different agents with different knowledge for RAG?\n- The observation that questions labeled as \"Low\" complexity have lower accuracy rates than \"Moderate\" ones in Figure 3 casts doubt on the reliability of the moderator's assessment. Also, the reasoning claim that the complexity assessment can increase accuracy by at least 80% is way more optimistic and seems like a really bold statement to me.\n- In the ablation study, assigning all queries the highest complexity level results in reduced accuracy compared to the adaptive approach. Why is that? Does it mean calling multiple agents to collaborate in a complicated might not always lead to better outcomes, and incur higher API calls?", "limitations": "Addressed", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Comprehensive experiments and benchmarking.\n- Performance appears promising.", "weaknesses": "- The complexity assessment lacks details. More explanation on \"low, moderate, high\" is required.\n- Additionally, the assessment is entirely determined by an LLM, raising concerns about whether poor judgment by the moderator agent may propagate.\n- The experiments employing just 50 samples per dataset, may not adequately represent the performance of the proposed methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed review and the opportunity to refine our work based on your feedback. Your suggestions are crucial in guiding our efforts to provide a more comprehensive analysis and evaluation.\n\n**W1 & W2. The complexity assessment lacks details and concerns about judgment made by the moderator agent may propagate.**\n\nWe address this issue in Section 4.3 through an ablation study, the results of which are presented in Figure 5. This study evaluates the effectiveness of our adaptive complexity selection mechanism against static assignments across different modalities.\n\nOur findings indicate that the adaptive method significantly outperforms static settings, demonstrating robustness in our approach to complexity assessment and reducing the potential for error propagation. Additionally, a detailed comparison of human doctor annotations with the LLM’s assessments is provided in the pdf file attached.\n\n**W3. 50 Samples per dataset may not show true method performance**\n\nPlease refer to the general response and the pdf file attached for extra experiments with N=100 samples for all datasets and with entire test samples for the MedQA dataset.\n\n**Q1. Is assigning an agent a role enough for LLMs to act like experts? What if equipping different knowledge with RAG?**\n\nThank you for your idea about the knowledge initialization for agents using RAG in our MDAgents framework. To address your point on domain expertise, we conducted extra experiments on top of Table 3:\n\nMDAgents: 71.8%\n \n \\+ MedRAG: 75.2% \n\n \\+ Medical Knowledge Initialization: 76.0% (**your suggestion**)\n\n \\+ Moderator’s Review: 77.6%\n\n \\+ Moderator’s Review & MedRAG: 80.3%\n\nThese results indicate that augmenting agents with specific knowledge and structured reviews have potential to improve their ability to simulate domain expertise. We will detail these findings in our revised manuscript.\n\n**Q2. Doubts on Complexity Labels and Optimistic Accuracy Claims in Figure 3**\n\nTo re-explain Figure 3, we need to explain why this experiment is needed, which will help to understand the meaning of Figure 3. \n\nIt is important to accurately assign difficulty levels to medical questions. For instance, if a medical question is obviously easy, utilizing a team of specialists (such as an Interdisciplinary Doctor Team, IDT) might be excessive and potentially lead to overly pessimistic approaches. Conversely, if a difficult medical question is only tackled by a PCP, the problem might not be adequately addressed.\n\nThe core issue here is the LLM's capability to classify the difficulty of medical questions appropriately. If an LLM inaccurately classifies the difficulty level, the chosen medical solution may not be suitable, potentially leading to the wrong decision making. Therefore, understanding what constitutes an appropriate difficulty level is essential.\n\nWe hypothesize that the appropriate difficulty for each question corresponds to the difficulty level at which the probability of correctly solving the question is highest, as we cannot incorporate doctor in the difficulty decision making (while we also have ablation studies with human doctors as well) \n\nTo determine this, we assessed the accuracy of solutions across various difficulty levels. Specifically, we evaluated 10 medical problems (increased to 25 after rebuttals) by solving each problem 10 times at each difficulty level. By measuring the success rate, we aimed to identify the difficulty level that yielded the highest accuracy.\n\nThis rigorous approach ensures that the LLM's classification of problem difficulty aligns with the most effective and accurate medical solutions, thereby optimizing the application of medical expertise to each question.\n\nGoing back to Reviewer mJeV’s question, \n>The observation that questions labeled as \"Low\" complexity have lower accuracy rates than \"Moderate\" ones in Figure 3 casts doubt on the reliability of the moderator's assessment. \n\nIt is important to clarify that Figure 3(b) does not indicate specific questions labeled as \"Low\" complexity. Instead, it shows the probability that the LLM can correctly answer questions when c our “Low Complexity” solution is applied for **all questions**. This explanation extends to Figures 3(c) and 3(d) as well. Figure 3(a), on the other hand, illustrates whether the LLM is choosing the difficulty level that provides the highest accuracy.\n\n> Also, the reasoning claim that the complexity assessment can increase accuracy by at least 80% is way more optimistic and seems like a really bold statement to me.\n\nWe did not make such a claim. Our assertion is that the LLM is automatically selecting the appropriate difficulty level with an accuracy rate close to 80%.\n\n**Q3. Why does assigning high complexity to all queries reduce accuracy and increase API costs?**\n\nTo address this issue, we conducted additional experiments to see if we missed the benefit to improve the performance in high complexity cases.\n\nFor the image+text scenario, we explored various collaborative settings and found these outcomes:\n\n* Sequential & No Discussion: 39.0%\n\n* Sequential & Discussion: 45.0%\n\n* Parallel & No Discussion: 56.0%\n\n* Parallel & Discussion: 59.0%\n\nThis indicates the importance of multi-turn discussions, particularly in complex cases and the exclusion of this feature likely contributed to lower performance.\n\nWe hope our detailed responses have addressed your concerns effectively. Please feel free to add follow-up questions for further clarifications or updates needed for your re-evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GUWzRq8Ysd", "reviewer": "Reviewer_QC4Q", "summary": "This paper proposes MDAgents (Medical Decision-making Agents), a LLM collaboration framework for medical question answering. Given a single-modal or multi-modal medical question, MDAgents first classifies its complexity into low, moderate, and high. Based on the complexity checking result, MDAgents assigns a single primary care clinician LLM agent (for low complexity), a team of multidisciplinary LLM agents (for moderate complexity), or a team of integrated care LLM agent (for high complexity). The agent collaboration adopts multi-turn discussion and iterative report refinement. Evaluated on multiple medical QA datasets, MDAgents show better performance than a variety of baseline models, including other prompting strategies as well as other agent framework. Overall, this is an interesting paper.", "questions": "1. What's the relative cost of MDAgents v.s. GPT-4 zero-shot CoT on each of the dataset?\n2. If you remove the LLM complexity checker, and use the ICT method for all questions, will it achieve the highest result?\n3. Can you aggregate the GPT-3.5 results like the main results table? They are currently scattered in different tables.", "limitations": "1. The evaluations, while comprehensive, are mostly on multi-choice question answering tasks. This is not a realistic setting in medicine.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is generally clear and the displays are informative.\n2. A new medical domain-specific agent collaboration method has been proposed for decision making.\n3. Comprehensive experiments have been conducted to show the superior performance of MDAgents.\n4. Interesting additional analysis.", "weaknesses": "1. The main issue of this article is that the reported scores are not consistent with the literature, so it is unclear whether MDAgents is really state-of-the-art. For example, the original Medprompt paper reported their performance on MedQA as 90.2 and PubMedQA as 82.0. However, this paper reports Medprompt scores of 82.4 and 51.8 for these two datasets. The authors need to explain such discrepancy.\n2. The complexity checker is something novel but is not well evaluated. The authors might need to sample a set of questions and ask human physicians to score its complexity (e.g., from 1-5), and report the correlation between LLM complexity and human complexity. \n3. Some figures in the results sections are confusing. I am not exactly sure what Figure 3 means, and it contains additional lines that should be removed. Additionally, for figure 5, does the \"Low\" mean the subset performance of questions classified as \"Low\", or the performance if all questions are classified as \"Low\"?\n4. Studies on other medical agents (e.g., https://arxiv.org/abs/2402.13225) should also be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful review and the valuable insights you provided. Your feedback helps us clarify key aspects of our research and improve the overall quality of our submission.\n\n**W1. The reported scores not consistent with the literature**\n \nThank you for pointing out the discrepancies in the reported scores. We appreciate your attention to this detail and would like to clarify the differences between our experimental setup and the original paper [1] that introduced Medprompt:\n\n**1) Dataset Differences:** The high accuracy reported by [1] for Medprompt was achieved using the MedQA-US dataset with 4-option questions. Our evaluation, however, was conducted on the MedQA-US dataset with 5-option questions, which is inherently more challenging and likely contributes to the lower accuracy.\n\n**2) Implementation Variations:** In the original Medprompt implementation, they utilized five kNN-selected few-shot exemplars with a 5x ensemble. For our experiments, we used three exemplars to ensure fair comparisons with other methods in our main experiments. We initially considered using a different number of exemplars to calibrate our implementations further. However, we chose a smaller number due to the increased cost and time required for comprehensive testing. Under similar conditions, such as using more exemplars like [1], we anticipate that our MDAgents and other baseline methods might also demonstrate improved performance.\nWe will ensure these distinctions in the implementations are clearly outlined in our paper to provide proper context for the reported results.\n\n**W2. What is the correlation between LLM complexity scores and human Physician's judgments?**\n\nWe have addressed this in the general response by detailing a study where three human physicians annotated question complexity and conducted a correlation analysis with LLM assessments.\n\n**W3 & Q2. Figure 3 and 5 confusing and what if we remove LLM complexity checker in the ablation?**\n\nFor Figure 5, \"Low,\" \"Moderate,\" and \"High\" denote the performance outcomes when all questions in the dataset are manually set into each respective complexity category, rather than relying on the complexity obtained by the LLM complexity checker. This means:\n\n* **\"Low\"** shows the accuracy when all questions are set to low complexity.\n* **\"Moderate\"** indicates the accuracy when all questions are set to moderate complexity.\n* **\"High\"** reflects the accuracy when all questions are set to high complexity.\n\nThis setup was designed to evaluate how the model's performance varies when operating under uniform complexity assumptions across the dataset. We will ensure that the figure description and ablation setup in our paper (Section 4.3) are updated to clearly explain this methodology. \n\n**W4. Studies on other medical agents should also be discussed**\n\nWe will include a discussion of the study [1] suggested and others [2,3,4,5] to better contextualize our work and clarify how MDAgents compare with existing approaches in medical decision-making.\n\n**Q1. What is the relative cost of MDAgents vs. gpt-4 zero-shot CoT on each dataset**\n\nAs detailed in Table 3 of the attached pdf file, MDAgents require higher costs across the datasets compared to Zero-shot CoT. Our methods use a 3-shot setting across different medical complexities and recruit multiple agents, which are needed to effectively handle the complexity of medical datasets which contributes to the enhanced performance.\n\n**Q3. Need to aggregate GPT-3.5 results in Table 3**\n\nWe will aggregate the GPT-3.5 results into the main results table for clarity and easier comparison with other models.\n\n**L1. Evaluations limited to multi-choice question tasks which is not a realistic medical setting**\n\nTo address the issue, we conducted additional experiments using the MedQA dataset without predefined options, aiming to better mirror the open-ended nature of clinical decision-making.\n\nRecognizing that real-world medical scenarios often involve complex, multi-turn interactions, we are committed to refining our evaluation methods to more accurately simulate actual clinical conditions. In our latest experiments with the gpt-4o-mini model, we evaluated our method alongside 3-shot CoT-SC and Reconcile, using 100 samples from the MedQA dataset in an open-ended format. The results were as follows:\n\n* **3-shot CoT-SC:** 52 % accuracy\n* **Reconcile:** 40 % accuracy\n* **Ours:** 56 % accuracy\n\nThese results indicate that our approach performs competitively even in more realistic settings, underscoring its potential in clinical applications. We are open to further suggestions and welcome any recommendations for additional datasets or evaluation frameworks that could enhance the realism and robustness of our assessments.\n\nWe believe our responses have addressed your concerns and provided clarity. Please let us know if you require any additional information or further clarifications for your re-evaluation.\n\n**References**\n\n[1] Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., ... & Horvitz, E. (2023). Can generalist foundation models outcompete special-purpose tuning? case study in medicine. \n\n[2] Jin, Q., Wang, Z., Yang, Y., Zhu, Q., Wright, D., Huang, T., … & Lu, Z. (2024). AgentMD: Empowering Language Agents for Risk Prediction with Large-Scale Clinical Tool Learning. \n\n[3] Li, J., Wang, S., Zhang, M., Li, W., Lai, Y., Kang, X., ... & Liu, Y. (2024). Agent hospital: A simulacrum of hospital with evolvable medical agents. \n\n[4] Fan, Z., Tang, J., Chen, W., Wang, S., Wei, Z., Xi, J., ... & Zhou, J. (2024). Ai hospital: Interactive evaluation and collaboration of llms as intern doctors for clinical diagnosis. \n\n[5] Yan, W., Liu, H., Wu, T., Chen, Q., Wang, W., Chai, H., ... & Zhu, L. (2024). ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real World.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their valuable and constructive feedback on our submission. We are encouraged that they found the work to be important and novel with a comprehensive evaluation and strong results. 

Based on the reviews we have made significant updates to our paper and would like to highlight these changes:

**1. Additional experiments with an increased number of samples.**

We have conducted additional experiments by increasing the sample size from 50 to 100 for all benchmarks. We were initially limited by the computational and economic cost of running a larger sample, we do believe that 100 samples provide a robust performance number. To support this, we have performed an evaluation on the entire test set for the MedQA dataset. The results from these expanded experiments, detailed in the attached pdf file, demonstrate that the performance trends are consistent across all benchmarks. This consistency suggests that our initial selection of 50 samples and the new results with 100 samples are representative of the performance on the whole dataset. 

To reiterate, the decision to use 50 samples initially was based on our intention to align with previous studies, such as [1], which employed an identical sample size in their initial version of the paper. Moreover, our choice was influenced by the constraints of our experimental budget (refer to Table 3 in attached pdf file), particularly the computational costs associated with using the gpt-4-turbo model.

We also plan to further add experimental results incorporating 100 samples with 3 random seeds in the camera-ready version to ensure meaningful comparisons and enhance the robustness of our findings.

**2. Medical complexity annotations obtained from human physicians**

We have conducted an annotation study with three physicians to evaluate the complexity of 50 representative questions from the MedQA dataset. The questions were carefully selected to ensure they required equivalent medical expertise across different USMLE steps (1, 2, and 3), providing a comprehensive assessment of complexity.

Two among the three physicians had two years of medical training Internal Medicine (Post graduate year 2 (PGY-2) and one among them is a general physician. They rated the questions on a scale of -1 (low), 0 (moderate), and 1 (high). To assess inter-rater reliability, we calculated the Intraclass Correlation Coefficients (ICC), focusing on the most informative types:

* ICC2k (Two-way random effects, average measures): 0.269 [-0.14, 0.55]

* ICC3k (Two-way mixed effects, average measures): 0.280 [-0.15, 0.57]

These ICC values indicate moderate agreement among the raters, highlighting the inherent complexity and subjectivity in evaluating medical questions. The variability in ratings could be attributed to differences in individual experience, interpretation of the question's context, and the nuances of medical knowledge.

Additionally, we used several LLMs to annotate the same questions and compared their assessments with the majority opinion of the physicians. To determine the majority opinion among the physicians, we calculated the mode of their ratings. If the ratings were entirely different (e.g., -1, 0, 1), we used the mean value as the final complexity scale, ensuring a balanced representation when no consensus was reached. 

* gpt-4o-mini: -0.090 correlation

* gpt-4o: 0.022 correlation

* gpt-4: 0.070 correlation

* gemini-1.5-flash: 0.110 correlation

The LLMs showed low agreement with human complexity judgments, reflecting the challenges in automating nuanced medical assessments. Differences in physician ratings show subjectivity, suggesting that clear guidelines could improve consistency. We believe enhancing LLMs with better context understanding, medical knowledge, and diverse training datasets may help improve alignment with human physicians.

**Reference**

[1] Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., ... & Horvitz, E. (2023). Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452.

Meta-review: In this manuscript, the authors propose MDAgent, a multi-agent framework based on LLMs for medical decision making. The main novelty is a crafted collaboration scheme of multiple agents with designated roles, where a medical question is categorized into being of low, medium or high complexity. Based on the complexity checking result, MDAgents assigns a single primary care clinician LLM agent (for low complexity), a team of multidisciplinary LLM agents (for moderate complexity), or a team of integrated care LLM agent (for high complexity). The agent collaboration adopts multi-turn discussion and iterative report refinement. As much the question is complex, greater will be the benefit of multi-agent approach. 

The authors during the rebuttal provided satisfactory answers concerning the design of the empirical evaluation. The critical points were the extension of test sample (50 questions versus 100) and how realistic is the simulation of the clinical decision-making scenario (single-turn versus multi-turn interactions). They shared additional results, according to the request of the reviewer, as a proof of robustness for the property of the proposed model. The Reviewer recognized and acknowledged the additional effort of Authors in answering the questions.

This work indicates an interesting further research directions as follow-up of the empirical investigation: how to modulate the expertize of the team of agents in relation to the heterogeneity of expertise of physicians. The issue is how to balance the agreement/disagreement among physicians and the agreement/disagreement among the agents.

Despite this work presents ethical issues related to the critical health domain, e.g. hallucinations in complex situations, I believe that this contribution should be evaluated from the computational point of view and not from the perspective of deployment or provisioning as a service.
Decision: Accept (oral)