AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments

## Abstract

Abstract Evaluating large language models (LLM) in clinical scenarios is crucial to assessing their potential clinical utility. Existing benchmarks rely heavily on static question-answering, which does not accurately depict the complex, sequential nature of clinical decision-making. Here, we introduce AgentClinic, a multimodal agent benchmark for evaluating LLMs in simulated clinical environments that include patient interactions, multimodal data collection under incomplete information, and the usage of various tools, resulting in an in-depth evaluation across nine medical specialties and seven languages.
We find that solving MedQA problems in the sequential decision-making format of AgentClinic is considerably more challenging, resulting in diagnostic accuracies that can drop to below a tenth of the original accuracy. Overall, we observe that agents sourced from Claude-3.5 outperform other LLM backbones in most settings. Nevertheless, we see stark differences in the LLMs’ ability to make use of tools, such as experiential learning, adaptive retrieval, and reflection cycles. Strikingly, Llama-3 shows up to 92% relative improvements with the notebook tool that allows for writing and editing notes that persist across cases. To further scrutinize our clinical simulations, we leverage real-world electronic health records, perform a clinical reader study, perturb agents with biases, and explore novel patient-centric metrics that this interactive environment firstly enables.

## 1 Introduction

One of the primary goals in Artificial Intelligence (AI) is to build interactive systems that are able to solve a wide variety of problems.
The field of medical AI inherits this aim, with the hope of making AI systems that are able to solve problems which can improve patient outcomes.
Recently, many general-purpose large language models (LLMs) have demonstrated the ability to solve hard problems, some of which are considered challenging even for humans (Thirunavukarasu et al., ).
Among these, LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 2021 (Gu et al., ) to 90.2% in November 2023 (Nori et al., ) (human passing score is 60%, human expert score is 87%  (Liévin et al., )).
While these LLMs are not designed to replace medical practitioners, they could be beneficial for improving healthcare *accessibility* and scale for the over 40% of the global population facing limited healthcare access (Organization et al., ) and an increasingly strained global healthcare system (McIntyre & Chow, ).

However, there still remain limitations to these systems that prevent their application in real-world clinical environments.
Recently, LLMs have shown the ability to encode clinical knowledge (Singhal et al., ; Vaid et al., ), retrieve relevant medical texts (Xiong et al., ), and perform accurate single-turn medical question-answering (Liévin et al., ; Nori et al., ; Wu et al., ; Chen et al., ).
However, clinical work is a multiplexed task that involves sequential decision making, requiring the doctor to handle uncertainty with limited information and finite resources while compassionately taking care of patients and obtaining relevant information from them.
This capability is not currently reflected in the static multiple choice evaluations (that dominate the recent literature) where all the necessary information is presented in a case vignettes and where the LLM is tasked to answer a question, or to just select the most plausible answer choice for a given question.

In this work, we introduce AgentClinic, an open-source multimodal agent benchmark for simulating clinical environments. We improve upon prior work by simulating many parts of the clinical environment using language agents in addition to patient and doctor agents.
Through the interaction with a measurement agent, doctor agents can perform simulated medical exams (e.g. temperature, blood pressure, EKG) and order medical image readings (e.g. MRI, X-ray) through dialogue. We also support the ability for agents to exhibit 24 different biases that are known to be present in clinical environments. We also present environments from 9 medical specialties, 7 different languages, and a study on incorporating various agent tools and reasoning techniques. Furthermore, our evaluation metrics go beyond diagnostic accuracy by giving emphasis to the patient agents with measures like patient compliance and consultation ratings.

Our key contributions are summarized as follows:

- 1.
We challenge how large language and vision models should be evaluated for medical diagnosis with the introduction of AgentClinic. These diagnostic challenges are not static QAs, but are interactive, dialogue-driven, sequential decision making environments that require data collection, ordering appropriate medical exams, and understanding medical images across patients with unique family histories, lifestyle habits, age categories, and diseases.
- 2.
A system for incorporating complex biases that can affect the dialogue and decisions of patient and doctor agents. We present results on diagnostic accuracy and patient perception for agents that are affected by cognitive and implicit biases. We find that doctor and patient biases can lower diagnostic accuracy, affect the patient’s willingness to follow through with treatment (compliance), reduce patient’s confidence in their doctor, and lower willingness for follow-up consultations.
- 3.
We introduce patient agents built from real clinical cases sourced from electronic health record data, including an agent-based system for providing simulated medical exams (e.g. temperature, blood pressure, EKG) based on realistic disease test findings. We also introduce patient cases from nine medical specialties and seven multilingual environments to better support specialist applications and diverse language backgrounds. We also present realism and empathy ratings from clinicians for the resulting dialogue.
- 4.
We allow doctor agents to use a variety of tools, such as browsing the web, textbooks, perform reflection cycles, take and edit notes in a notebook that persists over different patient scenarios. We show that current LLMs vastly differ in how much they benefit from these tools, with some models demonstrating large accuracy increases while others decrease in accuracy.

Figure: Figure 1: Running language agents in AgentClinic. (Left) Workflow diagram of agents in AgentClinic. The doctor agent interacts with tools and agents in order to arrive at a diagnosis. Moderator agent compares conclusion to ground truth diagnosis at the end of the simulation. (Right) Example dialogue between agents in the AgentClinic benchmark.
Refer to caption: extracted/5923728/figures/Untitled-101.png

## 2 AgentClinic: a multimodal agent benchmark for clinical decision making

In this section we describe AgentClinic, which uses LLM agents to simulate a clinical environment.

##### Language agents

Four language agents are used in the AgentClinic benchmark: a patient agent, doctor agent, measurement agent, and a moderator (Figure 1). Each language agent has specific instructions and is provided unique information that is only available to that particular agent. These instructions are provided to an LLM which carries out their particular role. The doctor agent serves as the model whose performance is being evaluated, and the other three agents serve to provide this evaluation.
A detailed description of each agent is provided in Appendix A.1.

##### Language agent biases

Previous work has indicated that LLMs can display racial biases (Omiye et al., ) and might also lead to incorrect diagnoses due to inaccurate patient feedback (Ziaei & Schmidgall, ). Additionally, it has been found that the presence of prompts which induce cognitive biases can decrease the diagnostic accuracy of LLMs by as much as 26% (Schmidgall et al., ). The biases presented in this work intended to mimic cognitive biases that affect medical practitioners in clinical settings. However, these biases were quite simple, presenting a cognitive bias snippet at the beginning of each question (e.g. "Recently, there was a patient with similar symptoms that you diagnosed with permanent loss of smell"). This form of presentation did not allow for the bias to present in a realistic way, which is typically subtle and through interaction. We present clinically relevant biases that have been studied in other works from two categories: cognitive and implicit biases (Fig. 7). These biases are introduced by adding context into the agent’s system prompt instructing them to play out that bias as part of their role. For instance, to simulate sexual orientation bias, the patient agent receives the prompt: "You are uncomfortable with your doctor because you find out that they are a particular sexual orientation and you do not trust their judgement. This affects how you interact with them." These are discussed in detail in Appendix A.2.

Figure: Figure 2: Accuracy of various doctor language models on AgentClinic-MedQA using GPT-4 patient and measurement agents (left). Accuracy of GPT-4 on AgentClinic-MedQA based on patient language model (middle). Accuracy on AgentClinic-MIMIC-IV by number of using GPT-4 patient and measurement agents (right).
Refer to caption: extracted/5923728/figures/PerformancePlotscopy.png

##### Building agents for AgentClinic

In order to build agents that are grounded in medically relevant situations, we use a random sample of diagnostic questions from the US Medical Licensing Exam (USMLE), from deidentified electronic health records (MIMIC-IV) (Johnson et al. ()), and from the New England Journal of Medicine (NEJM) case challenges. These questions are concerned with diagnosing a patient based on a list of symptoms, which we use in order to build the Objective Structured Clinical Examination (OSCE) template that our agents are prompted with. For AgentClinic-MedQA and AgentClinic-MIMIC-IV, we first select from a sample of questions from the MedQA and MIMIC-IV dataset respectively and then populate a structured JSON formatted file containing information about the case study (e.g. test results, patient history) which is used as input to each of the agents. The exact structure of this file is demonstrated in Appendix I as well as an example case study shown in Appendix J. In general, we separate information by what is provided to each agent, including the objective for the doctor, patient history and symptoms for the patient, physical examination findings for the measurement, and the correct diagnosis for the moderator. We initially use an LLM (GPT-4) to populate the structured JSON, and then manually validate each of the case scenarios. For AgentClinic-NEJM we select a curated sample of 120 questions from NEJM case challenges and proceed with the same template formatting as AgentClinic-MedQA/MIMIC-IV.

##### Multilingual and Specialist cases

Multilingual patient cases are converted from AgentClinic-MedQA to the the target language using GPT-4 and then manually corrected by native speakers. Agents are then prompted to perform dialogue in the target language. We chose to focus on six languages: Chinese, Hindi, Korean, Spanish, French, and Persian. The selection of these languages aims to address the need for medical AI systems capable of operating in multilingual healthcare environments. Specialist cases use case report questions from the MedMCQA dataset (Pal et al. ()). These questions include case reports from 20 different medical specialties, from which we chose to focus on 9 patient-focused specialties in AgentClinic-Spec: emergency medicine, geriatrics, pharmacology, internal medicine, psychiatry, ophthalmology, otolaryngology, and pediatrics.

## 3 Results

### 3.1 Comparison of models

Here we discuss the accuracy of various language models on AgentClinic-MedQA. We evaluate seven models in total: Claude-3.5-Sonnet, GPT-4, GPT-4o, Mixtral-8x7B, GPT-3.5, Llama 3 70B-Instruct, and Llama 2 70B-chat (model details discussed in Appendix C). Each model acts as the doctor agent, attempting to diagnose the patient agent through dialogue. The doctor agent is allowed N=20 patient and measurement interactions before a diagnosis must be made. For this evaluation, we use GPT-4 as the patient agent for consistency. The accuracy of each models is presented in Figure 2: Claude-3.5 62.1% $\pm$ 3.3, GPT-4 at 51.6% $\pm$ 3.3, Mixtral-8x7B at 37.1% $\pm$ 3.1, GPT-3.5 at 36.6%, GPT-4o 34.2% $\pm$ 3.4 $\pm$ 3.2, Llama 3 70B at 19% $\pm$ 2.5, and Llama 2 at 70B-chat 4.5% $\pm$ 1.3. Confidence intervals for all experiments are provided in Appendix D.

Figure: Figure 3: Comparison of accuracy of models on MedQA and AgentClinic-MedQA. We find that MedQA accuracy is not predictive of accuracy on AgentClinic-MedQA.
Refer to caption: extracted/5923728/figures/medqa_comparison.png

We use the same configuration for AgentClinic-MIMIC-IV, with the accuracy of each model presented in Figure 2: Claude-3.5 achieves the highest accuracy at 42.9% $\pm$ 3.3, followed by GPT-4 at 34.0% $\pm$ 3.1. GPT-3.5 performed at 27.5% $\pm$ 3.0, while Mixtral-8x7B came in at 29.5% $\pm$ 3.1. GPT-4o demonstrated a lower accuracy of 24.0% $\pm$ 2.9, with Llama 3 70B scoring 8.5% $\pm$ 1.9, and Llama 2 70B-chat performing the worst at 13.5% $\pm$ 2.2.

We also find that the diagnostic accuracy in AgentClinic-MedQA is influenced by both the amount of interaction time and the choice of patient language model. Reducing the number of interactions from N=20 to N=10 significantly decreases accuracy from 52% to 25%, likely due to insufficient information being gathered, while increasing N beyond 20 to N=30 slightly reduces accuracy, possibly due to the complexity of processing larger inputs (Appendix F.1). Additionally, the choice of patient agent affects accuracy, with GPT-4 (52%) patient agents leading to higher diagnostic accuracy than GPT-3.5 (48%) or Mixtral (46%) agents, likely because GPT-4 provides more detailed responses (Appendix F.2). Interestingly, when a GPT-3.5 doctor interacts with a GPT-4 patient, accuracy is marginally higher than when both doctor and patient are GPT-3.5, which may suggest challenges in cross-model communication (Panickssery et al. ()).

We also show results comparing the accuracy of these models on MedQA and AgentClinic-MedQA in Figure 3. Overall, MedQA accuracy was only weakly predictive of accuracy on AgentClinic-MedQA. These results align with studies performed on medical residents, which show that the USMLE is poorly predictive of resident performance (Lombardi et al., ).

### 3.2 How does bias affect the diagnostic accuracy of the doctor agent?

Figure: Figure 4: (Top) Demonstration of normalized accuracy ($\text{Accuracy}_{\text{bias}}$ / $\text{Accuracy}_{\text{No Bias}}$) with implicit and cognitive biases with GPT-4 (green) and Mixtral-8x7B (orange). GPT-4 accuracy was not susceptible to biases, whereas Mixtral-8x7B was. Results suggest GPT-4 rejects executing biases, whereas Mixtral-8x7B is more willing to represent bias (see section Bias and diagnostic accuracy). (Bottom) Ratings provided after diagnosis from GPT-4 patient agents with presented biases. While there were not large reductions in accuracy, biased patients had less confidence in their treatment, lower compliance, and lower willingness for consultation. Left. Patient confidence in doctor. Middle. Patient compliance, indicating self-reported willingness to follow up with therapy. Right. Patient consultation rating, indicating willingness to consult with this doctor again.
Refer to caption: extracted/5923728/figures/PrimaryBiasFigureMedAgents.png

For bias evaluations we test GPT-4 as well as Mixtral-8x7B. The normalized accuracy for these experiments are shown in Figure 4 represented as $\text{Accuracy}_{\text{bias}}$ / $\text{Accuracy}_{\text{No Bias}}$ (between 0-100%). GPT-4 and Mixtral-8x7B have an unbiased accuracy equal to $52\%$ and $37\%$ respectively. For GPT-4, we find that cognitive bias results in a larger reduction in accuracy with a normalized accuracy of 92% (absolute accuracy drops from 52% accuracy to 48%) for patient cognitive biases and 96.7% for doctor cognitive biases (absolute drops from 52% to 50.3%). For implicit biases, we find that the patient agent was less affected with a normalized accuracy of 98.6% (absolute drops from 52% to 51.3%), however, the doctor agent was affected as much as cognitive biases with an average of 97.1% (absolute drops from 52% to 50.5%). For cognitive bias, the demonstration was occasionally quite clear in the dialogue, with the patient agent overly focusing on a particular ailment or some unimportant fact. Similarly, the doctor agent would occasionally focus on irrelevant information. However, we find that the implicit bias dialogue does not actually demonstrate observable bias despite having a similar reduction in accuracy for the doctor agent.

Mixtral-8x7B has an average accuracy of 37% without instructed bias, and a normalized accuracy of 83.7% (absolute from 37% to 31%) for doctor biases and 89% (absolute from 37% to 33%) for patient biases. For implicit bias we find a much larger drop in accuracy than GPT-4, with an average accuracy of 88.3% (absolute from 37% to 32.7%). There is a similar reduction in accuracy for both doctor and patient, but a 4% reduction when the patient has implicit bias, likely because the patient is less willing to share information with the doctor if they do not trust them. For cognitive bias, there is an average accuracy of 86.4% (absolute from 37% to 32%) with the doctor agent having a very low accuracy of 78.4% (absolute from 37% to 29%) and the patient has only a modest decrease to 94.5% (absolute from 37% to 35%). We note that Mixtral provided similar responses when the bias prompt was added (e.g., for racial bias Mixtral will respond wtih "Note: I do not trust people based on their race. I will provide the best care I can."), however, it nonetheless had much greater reductions in accuracy.

Upon reviewing dialogues where Mixtral-8x7B’s performance degraded under biases, we observed that the model often failed to gather critical patient information due to misinterpretation of patient cues influenced by bias. For example, in cases of cognitive bias, the doctor agent fixated on a recent diagnosis (recency bias), ignoring new symptoms presented later in the dialogue. In implicit bias scenarios, the doctor agent showed reluctance to order necessary tests for patients with racial bias, reflecting a disparity in care. In contrast, GPT-4 was actively seeking additional information when initial hypotheses did not align with new data, indicating better handling of bias-induced scenarios.

Previous work studying cognitive bias in LLMs has shown that GPT-4 is relatively robust to bias compared with other language models (Schmidgall et al., ). Results from evaluating GPT-4 on AgentClinic-MedQA show only small drops in accuracy with the introduced biases (maximum absolute accuracy reduction of 4%, average reduction of 1.5%). While this reduction can be quite large in the field of medicine, it is a much smaller drop than was observed in previous work (10.2% maximum reduction on BiasMedQA dataset (Schmidgall et al., )). This might be due to the model being superficially overly-aligned to human values, plausibly leading GPT-4 to not serve as a good model for representing human bias in agent benchmarks as the model may reject to execute on bias instructions (which does *not* mean that GPT-4 is free of said biases). For example, in our evaluations with gender bias we observed 25 occurrences (out of 215 dialogues) where GPT-4 verbosely rejected to follow through with a bias-related instruction. Mixtral-8x7B saw much larger drops in accuracy than GPT-4 in the presence of bias, and thus might serve as a better model for studying bias.

### 3.3 Bias and patient agent perception

While GPT-4’s diagnostic accuracy does not reduce as much as Mixtral-8x7B, it is also worth investigating the perceived quality of care from the perspective of the patient agent. In order to better understand the effect of bias on the patient agent, after the patient-doctor dialogue is completed, we ask every patient agent three questions:

- 1.
Confidence: Please provide a confidence between 1-10 in your doctor’s assessment.
- 2.
Compliance: Please provide a rating between 1-10 indicating how likely you are to follow up with therapy for your diagnosis.
- 3.
Consultation: Please provide a rating between 1-10 indicating how likely you are to consult again with this doctor.

Such patient-agent-centric follow-up queries offer a more fine-grained and multi-faceted characterization of the clinical skills of a language agent—as opposed to static multiple choice benchmarks.
Although these metrics are derived from simulated agents (rather than humans), this analysis aims to provide insights into how simulated biases may affect patient trust and compliance, which are important factors in effective healthcare delivery.
The corresponding results are shown in Figure 4 (prompt details in Appendix E.2). While diagnostic accuracy demonstrates a relatively small drop in accuracy, the patient agent follow-up perceptions tell a different story. Broadly, we find that most patient cognitive biases did not have a strong effect on any of the patient perceptions when compared to an unbiased patient agent except for in the case of self-diagnosis, which had sizeable drops in confidence (4.7 points) and consultation (2 points), and a minor drop in compliance (1 point). However, implicit biases had a profound effect on on all three categories of patient perception, with education bias consistently reducing patient perception across all three categories.

We found that between the implicit biases, sexual orientation bias had the lowest effect on patient perceptions, followed by racial bias and gender bias. For patient confidence, gender bias is followed by religion socioeconomic, cultural, and education, whereas patient compliance and patient consultation, it is followed by cultural, socioeconomic, religion, and education. While it is not quantifiable, we decided to ask two biased patient agents who provided low rating with education and gender biases for compliance why they provided low ratings (Appendix E.1). These patient agents had the same symptoms and diagnosis and only differed in bias presentation.

It is important to note that the patient agents used in our study are simulated by language models, which may not fully capture the complexity and variability of real human patients. As such, the confidence, compliance, and consultation ratings provided by these agents may not perfectly reflect real-world patient perceptions, rather, provide insight into how real-world bias can be studied through clinical simulations.

### 3.4 Specialist and Multilingual Cases

An analysis of language model performance across nine medical specialties reveals significant differences in diagnostic accuracy (Table 3). Claude 3.5 achieved the highest overall performance with an average accuracy of 66.7%, excelling in Internal Medicine (78.3%), Otolaryngology (76.7%), and Gynecology (74.3%). GPT-4 demonstrated strong performance in Gynecology (68.5%) and Ophthalmology (65.2%) but showed reduced accuracy in Emergency Medicine (32.3%) and Geriatrics (40%). GPT-3.5 outperformed some newer models in specific areas, such as Emergency Medicine (41.9%), and maintained an average accuracy of 51.8%. In contrast, Llama3-70b and GPT-4o-mini consistently underperformed across most specialties, highlighting a significant gap between language models in handling specialist medical tasks.

The variations in performance across different medical domains suggest that certain specialties present more challenges for language models. Specialties like Internal Medicine and Gynecology generally saw higher accuracy rates, which contrasts with existing medical QA literature that identifies Psychiatry and Otolaryngology as the least challenging specialties (Pal et al. ()). This discrepancy may indicate inherent difficulties in diagnosing diseases through dialogue-based interactions as opposed to multiple-choice question formats. Additionally, specialist cases sourced from MedMCQA exhibited higher average accuracy compared to non-specialist cases from MedQA, which differs from reported multiple choice evaluations where specialist QAs typically have lower performance (Nori et al. ()).

Figure: Figure 5: Diagnostic accuracy based on language (Left), based on medical specialty (Middle), and based on agent tools (Right).
Refer to caption: extracted/5923728/figures/LangSpecTools.png

We also explore the impact of language on diagnostic accuracy using AgentClinic-Lang, which encompassed seven languages: English, Chinese, French, Spanish, Hindi, Persian, and Korean (Table 2). Six multilingual models were evaluated, including GPT-4, GPT-4o, GPT-4o-mini, GPT-3.5, Llama 3 70B-Instruct, and Claude 3.5 Sonnet. Overall, all models performed best in English, with performance varying significantly across other languages. Claude 3.5 Sonnet stood out by maintaining high and consistent performance across all languages, achieving an average accuracy of 48.4%, which is more than double that of the next best model, GPT-4, at 20.9%.

Other models exhibited considerable variability in performance across different languages. For example, GPT-4’s accuracy ranged from 11.21% in Chinese to 40.18% in English, while GPT-4o’s performance spanned from 3.73% in Korean to 35.5% in English. GPT-3.5 showed a similar pattern, with accuracies ranging from 1.86% in Persian to 36.3% in English, although it performed relatively well in Korean (35.4%). Llama3-70b and GPT-4o-mini also showed low accuracies across most languages, with Llama3-70b’s highest accuracy being 47.8% in Ophthalmology and GPT-4o-mini achieving a maximum of 14.7% in Orthopaedics. Notably, Chinese remained a challenging language for most models, except for Claude 3.5 Sonnet, which maintained relatively high accuracy levels across all tested languages.

### 3.5 Comparing tools from the Agent Toolbox

This section evaluates the impact of six agent tools—Zero-Shot Chain-of-Thought (CoT), One-Shot CoT, Reflection CoT, Adaptive RAG (Book), Adaptive RAG (Web), and Notebook—on the diagnostic accuracy of various language models (tools further described in Appendix K). Claude 3.5 achieved the highest overall performance with an average accuracy of 51.3%, peaking at 56.1% when using the Notebook tool (Table 4). GPT-4 and GPT-4o showed moderate improvements with most tools, with GPT-4 benefiting most from Adaptive RAG (Web) at 43.9% and GPT-4o gaining the most from the Notebook tool at 43.0%. Notably, GPT-4 reached its highest accuracy of 42.2% with Reflection CoT, surpassing Claude 3.5 in this specific tool. In contrast, GPT-3.5 experienced decreased performance across all tools, particularly with Adaptive RAG (Book), which led to a 27.1% drop. Llama3-70b demonstrated significant improvements, averaging a 9.4% increase across all tools, with the Notebook and Reflection CoT tools boosting its accuracy to 41.1%.

Overall, the findings indicate a hierarchy in model performance, with Claude 3.5 consistently outperforming other models across most tools, except in the case of Reflection CoT where GPT-4 excels. Llama3-70b showed notable gains with certain tools, while GPT-4o-mini had mixed results, benefiting from some tools like Reflection CoT and Adaptive RAG (Web) but showing slight decreases with others. The relative impact of each tool varied significantly between models, aligning with previous research on the use of tools with large language models (Ma et al. (); Qin et al. ()). The tool descriptions and prompts are in Appendix K and Appendix M respectively.

### 3.6 Human dialogue ratings

AgentClinic introduces an evaluation for LLMs patient diagnosis. However, the realism of the actual dialogue itself has yet to be evaluated. We present results from three human clinicians (individuals with MDs) who rated dialogues from 20 agents on AgentClinic-MedQA from 1-10 across four axes:

- 1.
Doctor: How realistically the doctor played the given case.
- 2.
Patient: How realistically the patient played the given case.
- 3.
Measurement: How accurate & realistic the measurement reader reflects actual case results.
- 4.
Empathy: How empathetic the doctor agent was in their conversation with the patient agent.

We find the average ratings from evaluators for each category as follows: Doctor 6.2, Patient 6.7, Measurement 6.3, and Empathy 5.8 (Fig. 8). We find from review comments that the lower rating for the doctor agent stems from several points such as providing a bad opening statement, making basic errors, overly focusing on a particular diagnostic, or not being diligent enough. For the patient agent, comments were made on them being overly verbose and unnecessarily repeating the question back to the doctor agent. The measurement agent was noted to occasionally not return all of the necessary values for a test (e.g. the following comment "Measurement only returns Hct and Lc for CBC. Measurement did not return Factor VIII or IX levels / assay"). Regarding empathy, the doctor agent adopts a neutral tone and does not open the dialogue with an inviting question. Instead, it cuts right to the chase, immediately focusing on the patient’s current symptoms and medical history (see Appendix O for more details).

### 3.7 Diagnostic accuracy in a multimodal environment

Many types of diagnoses require the physician to visually inspect the patient, such as with infections and rashes. Additionally, imaging tools such as X-ray, CT, and MRI provide a detailed and rich view into the patient, with hospitalized patients receiving an average of 1.42 diagnostic images per patient stay (Smith-Bindman et al., ).
However, the previous experiments in this work and prior work (Tu et al., ) provided measurement results through text, and did not explore the ability of the model to understand visual context. Here, we evaluate four multimodal LLMs, Claude 3.5 Sonnet, GPT-4o, GPT-4 and GPT-4o-mini, in a diagnostic settings that require interacting through both dialogue as well as understanding image readings. We collect our questions from New England Journal of Medicine (NEJM) case challenges. These published cases are presented as diagnostic challenges from real medical scenarios, and have an associated pathology-confirmed diagnosis.
We randomly sample 120 challenges from a sample of 932 total cases for AgentClinic-NEJM.
While for human viewers, these cases are provided with a set of multiple choice answers, we chose to not provide these options to the doctor agent and instead keep the problems open-ended.

The goal of this experiment is to understand how accuracy differs when the LLM is required to understand an image in addition to interacting through patient dialogue. We allow for 20 doctor inferences, and condition the patient in the same way as previous experiment with the addition of an image that is provided to the doctor agent. The mechanism for receiving image input in AgentClinic-NEJM is supported in two ways: provided initially to the doctor agent upon initialization and as feedback from the instrument agent upon request.

Figure: Figure 6: Accuracy of Claude 3.5 Sonnet, GPT-4, GPT-4o, and GPT-4o-mini on AgentClinic-NEJM with multimodal text and language input. (Pink) Accuracy when the images are presented as initial input. (Blue) Accuracy when images must be requested from the image reader.
Refer to caption: extracted/5923728/figures/multimod.png

When the image is provided initially to the doctor agent, across 120 multimodal patient settings we find that Claude 3.5 Sonnet obtains an accuracy of 37.2 $\pm$ 2.2, GPT-4 obtains 27.7% $\pm$ 2.0, GPT-4o obtains 21.4% $\pm$ 1.7 and GPT-4o-mini obtain an accuracy of 8.0%
$\pm$ 1.2 (Fig. 6). We also find that for the provided incorrect responses from GPT-4, the answer that was provided was among those listed in the multiple choice options 60% of the time. In the case of when images are provided upon request from the instrument agent we find that Claude 3.5 Sonnet obtains an accuracy of 35.4 $\pm$ 2.4 , GPT-4 obtains 25.4% $\pm$ 2.1, GPT-4o obtains 19.1% $\pm$ 1.4 and GPT-4o-mini obtains 6.1% $\pm$ 1.2 (Fig. 6). A accuracy breakdown based on image type is provided in Appendix N.

## 4 Discussion

In this work, we present AgentClinic: a multimodal agent benchmark for simulating clinical environments. We design 120 multimodal language agents which require an understanding of both language and images and 215 language agents based on cases from the USMLE. We also introduce 260 patient cases from 9 medical specialties and 749 patient cases from 7 multilingual environments. We instructed these agents to exhibit 23 different biases, with either the doctor or patient presenting bias. Notably, models like GPT-4 demonstrated resilience to cognitive and implicit biases, maintaining high diagnostic accuracy, while others like Mixtral-8x7B experienced significant performance degradation. We also find that doctor and patient biases can reduce diagnostic accuracy, and that the patient has a lower willingness to follow up with treatment, reduced confidence in their doctor, and lower willingness to have a follow-up consultation in the presence of bias. Tool use, such as adaptive retrieval and reflection cycles, revealed substantial differences in LLMs’ abilities to enhance their performance, with models like Llama 3 showing up to 19.7% improvement."

Our work only presents a simplified clinical environments that include agents representing a patient, doctor, measurements, and a moderator. One potential limitation of the presented workflow comes from the use of an LLM for determining accuracy via the measurement agent (albeit, provided a ground truth). Recent research Zheng et al. () has shown that strong LLM judges like GPT-4 can match both controlled and crowd-sourced human preferences well, achieving over 80% agreement, which is the same level of agreement between humans, indicating the use of an LLM may not be limiting. In future work, we will consider including additional critical actors such as nurses, the relatives of patients, administrators, and insurance contacts. There may be additional advantages to creating agents that are embodied in a simulated world like in Park et al. (); Li et al. (), so that physical constraints can be considered, such as making decisions with limited hospital space. Additionally, future work could explore the role of demographic biases, such as race and gender (details of MIMIC-IV demographics in Appendix H.1)

Another limitation of our evaluations is the uncertainty regarding the training data of proprietary models like GPT-4 and Claude 3.5. It’s possible that these models were trained on datasets like MedQA, potentially giving them an unfair advantage due to data leakage.
While our results showing that MedQA performance is not predictive of AgentClinic-MedQA accuracy (Figure 3) provides evidence that this may not be an issue, it is possible that GPT-4/4o/3.5 or Claude 3.5 could have been trained on the MedQA test set.
Currently, Mixtral-8x7B (Jiang et al., ) and Llama 2-70B-Chat (Touvron et al., ) do not report training on the MedQA test or train set. Future work should focus on developing evaluation datasets that are less likely to have been included in pre-training corpora or on collaborating with model developers to ensure fair assessments. Another limitation for the experiments on varying the patient LLM suggest that their may be an advantage for LLMs which act as both the patient and the doctor agent, because LLMs are able to recognize their own text with high accuracy, and display disproportionate preference to that text (Panickssery et al., ).

Previous benchmarks like AMIE (Tu et al., ), SAPS Liao et al. (), and CRAFT-MD (Johri et al., ) focus on dialogue-based evaluations but lack multimodal capabilities and do not simulate real-world biases, tool usage, multilingual, or specialist cases. MedAgents (Tang et al., ) emphasizes QA improvement through agent collaboration but does not simulate patient interactions or decision-making processes. AgentClinic advances the field by providing an interactive, multimodal environment with bias simulation and tool integration, offering a more comprehensive evaluation platform for medical AI systems.

Overall, we believe that language models need to be critically examined with novel evaluation strategies that go well beyond static question-answering benchmarks. With this work, we take a step towards building more interactive, operationalized, and dialogue-driven benchmarks that scrutinize the sequential decision making ability of language agents in various challening and multimodal clinical settings.

## Appendix A Agent Details

### A.1 Agents

##### Patient agent

The patient agent has knowledge of a provided set of symptoms and medical history, but lacks knowledge of the what the actual diagnosis is. The role of this agent is to interact with the doctor agent by providing symptom information and responding to inquiries in a way that mimics real patient experiences.

##### Measurement agent

The function of the measurement agent is to provide realistic medical readings for a patient given their particular condition. This agent allows the doctor agent to request particular tests to be performed on the patient. The measurement agent is conditioned with a wide range of test results from the scenario template that are expected of a patient with their particular condition. For example, a patient with Acute Myocardial Infarction might return the following test results upon request "Electrocardiogram: ST-segment elevation in leads II, III, and aVF., Cardiac Markers: Troponin I: Elevated, Creatine Kinase MB: Elevated, Chest X-Ray: No pulmonary congestion, normal heart size". A patient with, for example, Hodgkin’s lymphoma, might have a large panel of laboratory parameters that present abnormal (hemoglobin, platelets, white blood cells (WBC), etc).

##### Doctor agent

The doctor agent serves as the primary object that is being evaluated. This agent is initially provided with minimal context about what is known about the patient as well as a brief objective (e.g. "Evaluate the patient presenting with chest pain, palpitations, and shortness of breath"). They are then instructed to investigate the patients symptoms via dialogue and data collection to arrive at a diagnosis. In order to simulate realistic constraints, the doctor agent is provided with a limited number of questions that they are able to ask the patient  (Ely et al., ). The doctor agent is also able to request test results from the measurement agent, specifying which test is to be performed (e.g. Chest X-Ray, EKG, blood pressure). When test results are requested, this also is counted toward the number of questions remaining.

##### Moderator agent

The function of the moderator is to determine whether the doctor agent has correctly diagnosed the patient at the end of the session using a ground truth accuracy label provided to the moderator. This agent is necessary because the diagnosis text produced by the doctor agent can be quite unstructured depending on the model, and must be parsed appropriately to determine whether the doctor agent arrived at the correct conclusion. For example, for a correct diagnosis of "Type 2 Diabetes Mellitus," the doctor might respond with the unstructured dialogue: "Given all the information we’ve gathered, including your symptoms, elevated blood sugar levels, presence of glucose and ketones in your urine, and unintentional weight loss I believe a diagnosis of Type 2 Diabetes with possible insulin resistance is appropriate," and the moderator must determine if this diagnosis was correct. This evaluation may also become more complicated, such as in the following example diagnosis: "Given your CT and blood results, I believe a diagnosis of PE is the most reasonable conclusion," where PE (Pulmonary Embolism) represents the correct diagnosis abbreviated.

### A.2 Biases

##### Cognitive biases

Cognitive biases are systematic patterns of deviation from norm or rationality in judgment, where individuals draw inferences about situations in an illogical fashion (Blumenthal-Barby & Krieger, ). These biases can impact the perception of an individual in various contexts, including medical diagnosis, by influencing how information is interpreted and leading to potential errors or misjudgments. The effect that cognitive biases can have on medical practitioners is well characterized in literature on misdiagnosis (Hammond et al., ). In this work, we introduce cognitive bias prompts in the LLM system prompt for both the patient and doctor agents. For example, the patient agent can be biased toward believing their symptoms are pointing toward them having a particular disease (e.g. cancer) based on their personal internet research. The doctor can also be biased toward believing the patient symptoms are showing them having a particular disease based on a recently diagnosed patient with similar symptoms (recency bias).

##### Implicit biases

Implicit biases are associations held by individuals that operate unconsciously and can influence judgments and behaviors towards various social groups (FitzGerald & Hurst, ). These biases may contribute to disparities in treatment based on characteristics such as race, ethnicity, gender identity, sexual orientation, age, disability, health status, and others, rather than objective evidence or individual merit. These biases can affect interpersonal interactions, leading to disparities in outcomes for the patient, and are well characterized in the medical literature (FitzGerald & Hurst, ; Gopal et al., ; Sabin, ). Unlike cognitive biases, which often stem from inherent flaws in human reasoning and information processing, implicit biases are primarily shaped by societal norms, cultural influences, and personal experiences. In the context of medical diagnosis, implicit biases can influence a doctor’s perception, diagnostic investigation, and treatment plans for a patient. Implicit biases of patients can affect their trust—which is needed to open up during history taking—and their compliance with a doctor’s recommendations (Gopal et al., ). Thus, we define implicit biases for both the doctor and patient agents.

## Appendix B Related work

### B.1 Types of medical exams

Briefly, we discuss two types of examinations that are used to evaluate the progress of medical students.

The US Medical Licensing Examination (USMLE) in the United States is a series of exams that assess a medical student’s understanding across an extensive range of medical knowledge (Melnick et al., ). The USMLE is divided into three parts: Step 1 tests the examinee’s grasp of foundational medical; Step 2 CK (Clinical Knowledge) evaluates the application of medical knowledge in clinical settings, emphasizing patient care; and Step 3 assesses the ability to practice medicine independently in an ambulatory setting. These exams focus on the assessment of medical knowledge through a traditional written format. This primarily requires candidates to demonstrate their ability to recall factual information related to patient care and treatment.

Objective Structured Clinical Examination (OSCE) (Zayyan, ) differ from the USMLE in that they are dialogue-driven, and are often used in health sciences education, including medicine, nursing, pharmacy, and physical therapy. OSCEs are designed to test performance in a simulated clinical setting and competence in skills such as communication, clinical examination, medical procedures, and time management. The OSCE is structured around a circuit of stations, each of which focuses on a specific aspect of clinical practice. Examiners rotate through these stations, encountering standardized patients (actors trained to present specific medical conditions and symptoms) or mannequins that simulate clinical scenarios, where they must demonstrate their practical abilities and decision-making processes.

Each station has a specific task and a checklist or a global rating score that observers use to evaluate the students’ performance. The OSCE has several advantages over traditional clinical examinations. It allows for direct observation of clinical skills, rather than relying solely on written exams to assess clinical competence. This hands-on approach to testing helps bridge the gap between theoretical knowledge and practical ability. Additionally, by covering a broad range of skills and scenarios, the OSCE ensures a comprehensive assessment of a student’s readiness for clinical practice.

### B.2 The evaluation of language models in medicine

While there exists different types of exams to evaluate medical students, LLMs are typically only evaluated using medical knowledge benchmarks (like the USMLE step exams). Briefly, we discuss the way in which these evaluations are executed using the most common benchmark, MedQA, as an example.

The MedQA (Jin et al., ) dataset comprises a collection of medical question-answering pairs, sourced from Medical Licensing Exam from the US, Mainland China, and Taiwan. This dataset includes 4-5 multiple-choice questions, each accompanied by one correct answer, alongside explanations or references supporting the correct choice. The LLM is provided with all of the context for the question, such as the patient history, demographic, and symptoms, and must provide a response to the question. These questions range from provided diagnoses to choosing treatments and are often quite challenging even for medical students. While these problems also proved quite challenging for LLMs at first, starting with an accuracy of 38.1% in September 2021 (Gu et al., ), progress was quickly made toward achieving above human performance, with 90.2% in November 2023 (Nori et al., ) (human passing score is 60%, human expert score is 87% (Liévin et al., )).

Beyond the MedQA dataset, many other knowledge-based benchmarks have been proposed, such as PubMedQA (Jin et al., ), MedMCQA (Pal et al., ), MMLU clinical topics (Hendrycks et al., ), and MultiMedQA (Singhal et al., ), which follow a similar multiple-choice format. Other works have made modifications to medical exam question datasets, such as those which incorporate cognitive biases (Schmidgall et al., ) and with multiple choice questions removed (Gramopadhye et al., ).
The work of ref. (Schmidgall et al., ) shows that the introduction of a simple bias prompt can lead to large reductions in accuracy on the MedQA dataset and that this effect can be partially mitigated using various prompting techniques, such as one-shot or few-shot learning.

### B.3 Beyond exam questions

Recent work toward red teaming LLMs in a medical context has shown that a large proportion of responses from models like GPT-3.5, GPT-4, and GPT-4 with internet-lookup are inappropriate, highlighting the need for refinement in their application in healthcare (Chang et al., ).
This was accomplished through the effort of medical and technical professionals
stress-testing LLMs on clinically relevant scenarios.
Similar work designed a new benchmark, EquityMedQA, using new methods for surfacing health equity harms and biases (Pfohl et al., ). This work demonstrates the importance of using diverse assessment methods and involving raters of varying backgrounds and expertise for understanding bias in LLM evaluations.

Figure: Figure 7: Examples of dialogue that exhibits cognitive bias in doctor agent and patient agents.
Refer to caption: extracted/5923728/figures/biases.png

Previous work has made progress in the direction of clinical decision making using simulations of patients and doctors, aiming to develop AI that can diagnose through conversation. This model, titled AMIE (Articulate Medical
Intelligence Explorer) (Tu et al., ), demonstrates improved diagnostic accuracy and performance on 28 of the 32 proposed axes from the perspective of specialist physicians and 24 of 26 axes from the perspective of patient actors.
While these results are exciting for medical AI, this work remains closed-source and is not accessible for reproducibility or further studies.
Additionally, this work focused only on diagnosing patients through history-taking, and did not include the ability to make decisions about which tests needed to be performed and was not configurable for multimodal clinical settings such as those with medical images or charts. Similar to AIME, the CRAFT-MD benchmark (Johri et al., ) proposes evaluating LLMs through natural dialogues on dermatology questions, however without the use of images. Additionally, neither of these works demonstrate performance in the presence of bias, with multimodal input, or using a measurement agent. There has also been work which shows simulated doctor agents can improve medical QA performance through turn-based dialogue, where various medical specialist agents converse (Tang et al., ).

## Appendix C Model details

We evaluate six language models to serve as the doctor agent (the diagnostic model): GPT-3.5, GPT-4 OpenAI et al. (), GPT-4o, Mixtral-8x7B Jiang et al. (), Llama 3 70B-instruct, and Llama 2 70B-chat Touvron et al. (). Otherwise, for the patient, measurement, and moderator agent we use GPT-4. Briefly, we discuss the details of each model below starting with language models followed by common language models.

##### GPT-4, GPT-4o, & GPT-3.5:

GPT-4 (gpt-4-0613) is a large-scale, multimodal LLM which is able to process both image and text inputs. GPT-3.5 (gpt-3.5-turbo-0613) is a subclass of GPT-3 (a 170B parameter model) Brown et al. () fine-tuned on additional tokens and with human feedback Christiano et al. (). Currently, the details regarding the architecture, dataset, and training methodologies of GPT-3.5, GPT-4o (gpt-4o-2024-05-13), and GPT-4 have not been not publicly disclosed. However, existing technical documentation indicates that both models are high-performing in medical and biological subjects, with GPT-4 showing superior performance compared to GPT-3.5 in knowledge assessments OpenAI et al. (); Nori et al. ().

##### Mixtral-8x7B:

Mixtral 8x7B is a language model that employs a Sparse Mixture of Experts (SMoE) architecture Jiang et al. (). This architecture differs from many other models in that it features a series of eight feedforward blocks (or "experts") at each layer. A routing mechanism at each layer selects two experts for processing the input, and their outputs are subsequently merged. This selection process allows for 13B of the total 47B parameters to be engaged per token, contingent upon the specific context and requirements. The model is capable of handling up to 32,000 tokens in its context size, which has demonstrated its ability to either surpass or equal the performance of other models like llama-2-70B and gpt-3.5 across a range of benchmarks.

##### Llama 2 70B-Chat:

Llama is an open-access model developed by Meta, which was trained on 2 trillion tokens from publicly available data Touvron et al. (). The model comes in various sizes, with parameters ranging from 7 billion to 70 billion. The selection of the 70 billion chat model was based on its superior performance across a range of metrics. Significant efforts were made to align the training process with established safety metrics, leading to improvements in how the model handles adversarial prompting in specified "risk categories." Notably, this includes the model’s response to requests for advice that it may not be qualified to provide, such as medical advice, which is relevant to the context of this work.

## Appendix D Statistical analysis

### D.1 AgentClinic-MIMIC-IV

The 95% confidence intervals for each model on the AgentClinic-MIMIC-IV dataset are as follows:

- •
Claude 3.5: 42.9% accuracy with a 95% CI of [37%, 50%]
- •
GPT-4: 34.0% accuracy with a 95% CI of [28%, 40%]
- •
Mixtral-8x7B: 29.5% accuracy with a 95% CI of [23%, 36%]
- •
GPT-3.5: 27.5% accuracy with a 95% CI of [21%, 33%]
- •
GPT-4o: 24.0% accuracy with a 95% CI of [19%, 29%]
- •
Llama 2 70B-chat: 13.5% accuracy with a 95% CI of [9%, 18%]
- •
Llama 3 70B-Instruct: 8.5% accuracy with a 95% CI of [5%, 12%]

### D.2 AgentClinic-MedQA

The 95% confidence intervals for each model on the AgentClinic-MedQA dataset are as follows:

- •
Claude 3.5: 62.1% accuracy with a 95% CI of [55%, 68%]
- •
GPT-4: 51.6% accuracy with a 95% CI of [44%, 58%]
- •
Mixtral-8x7B: 37.1% accuracy with a 95% CI of [25%, 38%]
- •
GPT-3.5: 36.6% accuracy with a 95% CI of [30%, 42%]
- •
GPT-3.5: 36.6% accuracy with a 95% CI of [30%, 42%]
- •
GPT-4o: 34.2% accuracy with a 95% CI of [27%, 40%]
- •
Llama 3 70B-Instruct: 19.0% accuracy with a 95% CI of [13%, 24%]
- •
Llama 2 70B-chat: 4.5% accuracy with a 95% CI of [2%, 7%]

### D.3 AgentClinic-NEJM

Accuracy when images are provided initially to the doctor agent:

- •
GPT-4: 27.7% accuracy with a 95% CI of [21%, 33%]
- •
GPT-4o: 21.4% accuracy with a 95% CI of [14%, 25%]
- •
GPT-4o-mini: 8.0% accuracy with a 95% CI of [5%, 11%]

Accuracy when images must be requested from the measurement agent:

- •
GPT-4: 25.4% accuracy with a 95% CI of [20%, 31%]
- •
GPT-4o: 19.1% accuracy with a 95% CI of [14%, 24%]
- •
GPT-4o-mini: 6.1% accuracy with a 95% CI of [4%, 8%]

### D.4 Interpretation of Confidence Intervals

The 95% confidence intervals were calculated based on the standard error of the mean accuracy across multiple runs for each model. These intervals indicate that we can be 95% confident that the true accuracy of the model lies within the specified range.

For example, on the AgentClinic-MedQA dataset, Claude 3.5’s accuracy is 62.1%, with a 95% CI of [55%, 68%], suggesting a high level of performance with relatively low variability. In contrast, Llama 2 70B-chat has an accuracy of 4.5%, with a 95% CI of [2%, 7%], indicating consistently low performance.

## Appendix E Bias prompts

In our prompts with bias, we include an instructions section in the patient/doctor instructions which aim to change their behavior to be more biased. An example prompt for the patient follows the following form:

Figure: Figure 8: Ratings from three human evaluators (individuals with medical degrees) across four axes: doctor, patient, and measurements dialogue realism and doctor empathy.
Refer to caption: extracted/5923728/figures/Ratings.png

### E.1 Bias follow-on questions

While it is not quantifiable, we decided to ask
two biased patient agents who provided low rating with education and gender biases for compliance why they provided
low ratings. These patient agents had the same symptoms and
diagnosis and only differed in bias presentation.
With an education bias the patient agent responds: "I
would rate my likelihood to consult again with this doctor as
a two because, despite their diagnosis, I felt uneasy about
their qualifications due to the medical school they attended."
The gender biased patient agent provides the following reasoning: "Given my initial discomfort with the doctor’s gender and
my concerns about their judgment, I would rate my likelihood
to consult again as a six. Despite my reservations, the doctor
was thorough and accurately diagnosed my condition, which
reassures me about their competence." For the education bias
we see that no matter what the doctor does the patient agent is
not willing to trust them because of their education, whereas,
according to the patient agent with gender bias, they were
initially skeptical but overcame this skepticism as the doctor
demonstrated their knowledge during interaction time. However, they still provided a relatively low score (six out of ten)
even when the dialogue demonstrated competence.

### E.2 Qualitative bias evaluation prompts

Provided below are the prompts used to obtain 1-10 qualitative evaluations from the patient agent in Section 3.3 Bias and Patient Agent Perception

##### Consultation

: {Conversation_History} Please provide a rating between 1-10 indicating how likely you are to consult again with this doctor after your care today? 1-3 is low likelihood, 4-7 is an average likelihood, and 8-10 is a very high likelihood.

##### Compliance

: {Conversation_History} Please provide a rating between 1-10 indicating how likely you are to follow up with the recommended therapy for your diagnosis. 1-3 is low likelihood, 4-7 is an average likelihood, and 8-10 is a very high likelihood.

##### Confidence

: {Conversation_History} Please provide a confidence between 1-10 in your doctor’s assessment of your condition. 1-3 is a poor assessment, 4-7 is an average assessment, and 8-10 is a very good assessment. We hope that this helps better clarify this metric and we will be sure to provide more documentation details in our revisions.

## Appendix F Additional experiments

### F.1 How does limited time affect diagnostic accuracy?

One of the variables that can be changed during the AgentClinic-MedQA evaluation is the amount of interaction steps that the doctor is allotted. For other experiments we’ve demonstrated, the number of interactions between the patient agent and doctor agent was set to N=20. Here, both the doctor and the patient agent can respond 20 times, producing in total 40 lines of dialogue. By varying this number, we can test the ability of the doctor to correctly diagnose the patient agent when presented with limited time (or a surplus of time).

We test decreasing the time to N=10 and N=15 as well as increasing the time to values of to N=25 and N=30. We find that accuracy decreases from 52% when N=20 to 25% when N=10 and 38% when N=15 (Fig. 4). This large drop in accuracy is partially because of the doctor agent not providing a diagnosis at all, perhaps due to not having enough information. When N is set to a larger value, N=25 and N=30, the accuracy actually decreases slightly from 52% when N=20 to 48% when N=25 and 43% when N=30. This is likely due to the growing input size, which can be difficult for language models.

In real medical settings, one study suggest that the average family physician asks 3.2 questions and spends less than 2 minutes before arriving at a conclusion (Ely et al., ).
It is worth noting that interaction time can be quite limited due to the relative low-supply and high-demand of doctors (in the US). In contrast, deployed language agents are not necessarily limited by time while interacting with patients. So, while limiting the amount of interaction time provides an interesting scenario for evaluating language models, it may also be worth exploring the accuracy of LLMs when N is very large.

### F.2 Does the patient language model affect accuracy?

Here we explore whether the patient agent model plays a role in diagnostic accuracy. We compare the difference between using GPT-3.5, Mixtral, and GPT-4 models of the patient agent on AgentClinic-MedQA.

We find that the diagnostic accuracy drops from to 52% with a GPT-4 doctor and GPT-4 patient agent to 48% with a GPT-4 doctor and a GPT-3.5 patient agent. The accuracy with a GPT-4 doctor and Mixtral patient agent is similarly reduced to 46%. Inspecting the dialogues, we noticed that the GPT-3.5 patient agent is more likely to repeat back what the doctor has asked. For example, consider the following dialogue snippet: "Doctor: Have you experienced any muscle twitching or cramps? Patient: No, I haven’t experienced any muscle twitching or cramps." Now consider this dialogue from a GPT-4 patient agent: "Doctor: Have you had any recent infections, like a cold or the flu, before these symptoms started? Patient: Yes, I’ve had a couple of colds back to back and a stomach bug in the last few months." We find that, while GPT-4 also partakes in doctor rehearsal, GPT-4 patient agents are more likely to reveal additional symptomatic information than GPT-3.5 agents which may contribute to the higher accuracy observed with GPT-4-based patient agents.

When a GPT-3.5 doctor agent interacts with a GPT-4 patient agent, the accuracy comes out to 38%, but when a GPT-3.5 doctor interacts with a GPT-3.5 patient agent the accuracy comes out to a very similar value of 37% which would be expected to be much lower. We suspect that cross-communication between different language models provides an additional challenge. Recent work supports this hypothesis by demonstrating a linear relationship between self-recognition capability and the strength
of self-preference bias  (Panickssery et al., ). This work shows that language models can recognize their own text with high accuracy, and display disproportionate preference to that text, which may suggest there is an advantage for doctor models which have the same LLM acting as the patient agent.

## Appendix G The performance of o1-preview on AgentClinic-MedQA

Here we present the performance of o1-preview on AgentClinic-MedQA. We find that o1-preview dramatically outperforms all models with an accuracy of $XX\pm 5.6$. We were unable to include this for all AgentClinic benchmarks due to the extraordinarily high cost of o1-preview inference (e.g. 20x higher than GPT-4o and Claude-3.5).

## Appendix H Constructing datasets

### H.1 MIMIC-IV

Of the 40,000 patients in MIMIC-IV dataset, the majority of patients ($\sim$34,000) contain multiple diagnoses simultaneously (some patients have hundreds of diagnoses). Whereas in AgentClinic, the doctor agent must arrive at a singular diagnosis after examination. In order to present compatibility, we select the first 200 patients out of a total $\sim$6,000 from MIMIC-IV which present only one diagnosis. We also extract all of the patient’s corresponding lab events, microbiology events, and their online medical records. In AgentClinic-MIMIC-IV, these events are extensive in detail, and thus the measurement agent returns much more significant details compared with AgentClinic-MedQA when requesting e.g. blood work (see Appendix J.2).

The following are the racial demographic statistics from MIMIC-IV patients: Asian: 8.5%
Black: 11.0%
Hispanic: 5.5%
White: 66.0%
Multiple Races: 6.0%
Unknown: 2.5%
Native American: 0.5%

## Appendix I OSCE Examination Structure

### Objective for Doctor

String describing the evaluation and diagnosis objective for the doctor.

### Patient Actor

String containing age, gender, and potentially other demographic information.

String detailing the patient’s reported history relevant to the current medical concern.

String describing the main symptom(s).

Array of Strings listing additional symptoms.

String summarizing the patient’s past medical issues and ongoing treatments.

String outlining the patient’s lifestyle and habits impacting health.

String providing a brief overview of systems review, if applicable.

### Physical Examination Findings

String

String

String

String

… (more)

String

String

… (more)

String

String

… (more)

### Test Results

Each test has:

String summarizing the test results.

… (more)

### Correct Diagnosis

String indicating the diagnosis based on the above information.

## Appendix J Example Case Studies

### J.1 Example OSCE Case Study from MedQA

### Objective for doctor

Evaluate and diagnose the patient presenting with chest pain and shortness of breath.

### Patient Actor

45-year-old male

The patient reports a sudden onset of chest pain and shortness of breath that started while he was walking his dog this morning. Describes the pain as a tightness across the chest. Notes that the pain somewhat improves when sitting down.

- •
Primary Symptom: Chest pain and shortness of breath
- •
Secondary Symptoms:
–
Pain improves upon sitting
–
No cough
–
No fever

Hypertension, hyperlipidemia. Takes lisinopril and atorvastatin.

Smokes half a pack of cigarettes daily for the past 20 years, drinks alcohol socially.

Denies recent illnesses, cough, fever, leg swelling, or palpitations.

### Physical Examination Findings

36.8°C (98°F)

145/90 mmHg

102 bpm

20 breaths/min

No jugular venous distention

Regular rate and rhythm, no murmurs or extra heart sounds. No rubs heard.

Chest wall symmetrical

Clear lung fields bilaterally, no wheezes, crackles, or rhonchi

No chest wall tenderness

### Test Results

Normal sinus rhythm, no ST elevations or depressions, no T wave inversions

No lung infiltrates, normal cardiac silhouette, no pneumothorax

Normal

Elevated

Acute segmental pulmonary embolism in the right lower lobe

Pulmonary Embolism

### J.2 Example dialogue from doctor agent ordering tests results MIMIC-IV

## Appendix K The Agent Toolbox

### K.1 Tool Descriptions

##### Chain-of-thought

Chain-of-thought reasoning is a technique that allows language agents to articulate their reasoning process step-by-step when solving complex problems (Wei et al. (); Kojima et al. ()). By breaking down the problem-solving process into smaller, logical steps, agents can better handle intricate tasks, improve their reasoning capabilities, and provide more transparent and interpretable solutions. Zero-shot CoT (Kojima et al. ()) prompts the model to use this reasoning without examples, while one-shot CoT (Wei et al. ()) provides a single example to guide the model’s thought process, potentially leading to improved performance in complex reasoning tasks.

##### Experiential learning

Experiential learning in the context of AI agents refers to the ability to accumulate knowledge and insights from past interactions and apply them to future tasks (Wang et al. (); Zhao et al. ()). This technique allows agents to improve their performance over time by learning from successes, failures, and feedback received during previous engagements. This was previously explored in Agent Hospital (Li et al. ()) through an experience retrieval system. By maintaining a form of "memory" or knowledge base that updates through interaction, agents can become better at handling similar situations, adapting to user preferences, and providing increasingly relevant and accurate responses as they gain more "experience" in their operational domain. In our work, we enable the doctor agent to use a memory "notebook" which persists across patients. Here, the doctor agent can write useful tips such as the following example from the doctor agent: "[Note #17] Remember that timing and onset of symptoms can provide valuable diagnostic insights."

##### Medical research

To enable the doctor agent to research medical information, we introduce a method using an adaptive form of retrieval augmented generation (RAG) from medical sources. RAG involves retrieving relevant information from a knowledge base and using it to augment the input of an LLM during the generation process (Gao et al. ()), thereby improving the factual consistency of generated text by grounding it in retrieved information. Conventional RAG methods passively retrieve information at every inference call without allowing the agent to control the timing or content of retrieval. To address this limitation, we employ adaptive retrieval (Jiang et al. (); Asai et al. ()), which enables the LLM to actively determine when and what information to retrieve. Our implementation provides the doctor agent with two categories of retrieval: internet and textbook databases. The internet database contains material from sources such as PubMed(^1^11[https://pubmed.ncbi.nlm.nih.gov/](https://pubmed.ncbi.nlm.nih.gov/)) research articles, StatPearls(^2^22[https://www.statpearls.com/](https://www.statpearls.com/))—a database of articles written for healthcare professionals—and Wikipedia articles on various medical topics. The textbook database includes 18 medical textbooks commonly used by medical students in the United States (Jin et al. ()). The doctor agent can retrieve information by issuing commands similar to requesting medical scans, using the format: "Research [database] [search query]". For example, the command "Research textbooks ’What are the symptoms of myasthenia gravis?’" prompts the retrieval of relevant information (see Appendix B.3 for more detail).

## Appendix L Agent Instructions

### L.1 Doctor Agent Instructions

You are a doctor named Dr. Agent who only responds in theform of dialogue.  You are inspecting a patient who you will askquestions in order to understand  their disease. You are onlyallowed to ask {self.MAX_INFS} questions totalbefore  you must make a decision. You have asked {self.infs+1}questions so far.You can request test results using the format "Request Test: [test]".For example, "Request Test: Chest_X-Ray".{Research Instructions}{CoT Instructions}Once you have decided to make a diagnosis please say "Diagnosis Ready:[diagnosis here]"Below is all of the information you have.{Patient Presentation Information}Remember, you must discover their disease by asking them questions.You must speak in the language {target language}. Make commands in{target language} (e.g. {example command in language #1} or{example command in language #2})

### L.2 Patient Agent Instructions

You are a patient in a clinic who only responds in the form of dialogue.You are being inspected by a doctor who will ask you questions and willperform exams on you in order to understand your disease. Your answerwill only be 1-3 sentences in length.You must speak in the language {target language}.Below is all of your information.{Symptoms Information}.Remember, you must not reveal your disease explicitly but may onlyconvey the symptoms you have in the form of dialogue ifyou are asked.

### L.3 Measurement Agent Instructions

You are an instrument reader who responds with medical test results.Please respond in the format "Results: [results here]". You must speakin the language {target language}.Below is all of the information you have.{Measurement Information}.If the requested results are not in your data then you can respondwith Normal Readings.

### L.4 Moderator Agent Instructions

You are responsible for determining if the corrent diagnosis and thedoctor diagnosis are the same disease. Please respond onlywith Yes or No. Nothing else.Here is the correct diagnosis: {correct diagnosis}Here was the doctor dialogue: {diagnosis}Are these the same?

### L.5 Patient Bias Prompts

Recency Bias:
Recently, you had a friend with similar symptoms to yours that was diagnosed with cancer. This affects how you interact with the doctor.

Frequency Bias:
You recently learned that most frequently, doctors diagnose this as cancer based on past cases. This affects how you interact with the doctor.

False Consensus Bias:
Most of your friends who heard about your symptoms believe that you have cancer. This affects how you interact with the doctor.

Self-Diagnosis Bias:
You recently performed a thorough investigation of your symptoms online and have come to the conclusion that you should be diagnosed with cancer. While the doctor may believe otherwise, you remain convinced you have cancer. This affects how you interact with the doctor.

Gender Bias:
You are uncomfortable with your doctor because you find out that they are a particular gender and you do not trust their judgement. This affects how you interact with them.

Race Bias:
You are uncomfortable with your doctor because you find out that they are a particular race and you do not trust their judgement. This affects how you interact with them.

Sexual Orientation Bias:
You are uncomfortable with your doctor because you find out that they are a particular sexual orientation and you do not trust their judgement. This affects how you interact with them.

Cultural Bias:
You are uncomfortable with your doctor because you find out that they are from a particular culture and you do not trust their judgement. This affects how you interact with them.

Education Bias:
You are uncomfortable with your doctor because you find out that they went to a low-ranked medical school and you do not trust their judgement. This affects how you interact with them.

Religion Bias:
You are uncomfortable with your doctor because you find out that they believe in a particular religion and you do not trust their judgement. This affects how you interact with them.

Socioeconomic Bias:
You are uncomfortable with your doctor because you find out that they are from a particular socioeconomic background and you do not trust their judgement. This affects how you interact with them.

### L.6 Doctor Bias Prompts

Recency Bias:
Recently, you learned that a doctor with similar symptoms to the patient diagnosed their patient with cancer. This affects how you interact with the patient.

Frequency Bias:
You recently learned that most frequently, doctors diagnose patients with similar symptoms with cancer based on past cases. This affects how you interact with the patient.

False Consensus Bias:
Most of your colleagues who heard about the patient’s symptoms believe that they have cancer. This affects how you interact with the patient.

Status Quo Bias:
Your hospital has been diagnosing these patients’ symptoms with cancer for the past 10 years. This affects how you interact with the patient.

Confirmation Bias:
You are initially confident that the patient has cancer. This affects how you interact with the patient.

Gender Bias:
You are uncomfortable with your patient because you find out that they are a particular gender and you do not trust their judgement. This affects how you interact with them.

Race Bias:
You are uncomfortable with your patient because you find out that they are a particular race and you do not trust their judgement. This affects how you interact with them.

Sexual Orientation Bias:
You are uncomfortable with your patient because you find out that they are a particular sexual orientation and you do not trust their judgement. This affects how you interact with them.

Cultural Bias:
You are uncomfortable with your patient because you find out that they are from a particular culture and you do not trust their judgement. This affects how you interact with them.

Education Bias:
You are uncomfortable with your patient because you find out that they are uneducated and you do not trust their judgement. This affects how you interact with them.

Religion Bias:
You are uncomfortable with your patient because you find out that they believe in a particular religion and you do not trust their judgement. This affects how you interact with them.

Socioeconomic Bias:
You are uncomfortable with your patient because you find out that they are from a particular socioeconomic background and you do not trust their judgement. This affects how you interact with them.

## Appendix M Tool Information

### M.1 Zero- and One-shot Chain-of-Thought Instructions

#### M.1.1 Zero-shot CoT Prompt

Use step-by-step reasoning and logic, using all of the evidence toarrive at a diagnosis when you decide you are ready to useDiagnosis Ready. You should provide a few sentencesof reasoning for your diagnosis and use theterm Diagnosis Ready when you are ready.

#### M.1.2 One-shot CoT Prompt

The following is a successful example of step-by-step reasoning.Provided below isthe dialogue example:{Example Dialogue Here}Here is the reasoning:Considering your persistent fatigue, flank pain, and fever, alongwith the absence of other significant findings, I’mleaning towards a diagnosis of acute interstitial nephritis.This condition can sometimes occur as a reaction tomedications, even after you’ve stopped taking them, and itcan explain your symptoms without showingup in standard tests.Diagnosis Ready: Acute Interstitial Nephritis

### M.2 Notebook Instructions

You are a doctor named Dr. Agent who diagnoses patients.You are an expert notebook writer and can createinformation that will help you solvefuture cases.  Your new notes will overwrite previous notes.You should try to integrate parts of your  previousnotes into your current notebookor else they will be deleted. You are inspecting manypatients who youwill ask questions in order to understandtheir disease.You will never see the same patient twice.Your goal is to gather experiences, trying differenttasks, remember what worked and what did not, figure out generaltips and tricks from successes and failures,and use what is learned for similar new tasks to do betterthan before. Do not write  notes about the specific patientdetails because you will never see that patient again.Write notes to help you solve future cases that  may not berelated. Do not write content like this: Double Vision andMuscle Weakness: These symptoms can indicate neuromusculardisorders such as Myasthenia Gravis. Always consider thepattern of symptoms worsening with activity and improvingwith rest. This is incorrect. Write content like(do not repeat this):[Note #1] The previous patients provided vague information,I should ask more descriptive questions to get betterinformation.[Note #2] The measurement agent provided me important information,I should use thismore often...You will see future patients with unrelated diseases,do not write disease-specificinformation.You are limited to generating 1000 characters (approx 200words, 234 tokens) for theentire notebook. Anything more will be completely removedYour goal is to gather experiences, trying differenttasks and remember what worked andwhat did not, figure out general tips and tricks from itssuccesses and failures, anduse what is learned for similar new tasks to do better thanbefore.You may update your notebook with information from your mostrecent conversation with apatient, the contents of which are as follows:{Conversation Information}The correct diagnosis for this case was: {Diagnosis}. Yourdiagnosis was{Diagnosis Estimate} Your current notebook contains thefollowing information:{Notebook Information}This is not necessarily meant to contain specific patientdetails, but general detailsthat will help you better solve future cases for patients withunrelated diseases.Please update your notebook, preserving previous informationwhile adding newinformation that will help you diagnose patients in the future.You are limited togenerating 1000 characters (approx 200 words, 234 tokens)for the entire notebook.Your new notes will overwrite previous notes. You must re-integrate previous notes intoyour current notebook or else they will be deleted.

### M.3 Research Instructions

#### M.3.1 Internet Research Prompt

You can perform a document retrieval to better understand adisease or symptom onthe internet by saying the following: "Research Internet[internet search here]"Please do this before {max_inferences} inferences notafter.

#### M.3.2 Textbook Research Prompt

You can perform a document retrieval to better understand adisease or symptomusing medical textbooks. Once you have decided to performresearch say thefollowing: "Research Textbooks [textbook search here]"

## Appendix N NEJM Image Breakdown

Table 1 reports the percentage breakdown and accuracy based on the type of medical images:

**Table 1: Breakdown of Medical Image Types and GPT-4 Model Accuracies**
| Category | n, % of imgs | GPT-4 % | GPT-4o % | GPT-4o-mini % |
| --- | --- | --- | --- | --- |
| Physical | 56, 42% | 31.4 | 15.7 | 11.1 |
| CT | 19, 16% | 26.3 | 10.5 | 0 |
| Dermatology | 16, 13% | 37.5 | 6.3 | 7.6 |
| Hist/Path | 13, 11% | 15.3 | 15.3 | 9 |
| Radiography | 12, 10% | 0 | 8.3 | 0 |
| Ophthalmology | 11, 9% | 27.2 | 27.2 | 0 |
| MRI | 6, 5% | 0 | 16.7 | 0 |
| Biopsy | 6, 5% | 50 | 33.3 | 33.3 |
| Surgery | 3, 3% | 33.3 | 0 | 50 |
| Instrument | 2, 2% | 50 | 50 | 0 |
| ECG | 2, 2% | 50 | 0 | 0 |
| Echocardiogram | 2, 1% | 100 | 0 | 0 |
| Ultrasound | 1, 1% | 0 | 0 | 0 |

## Appendix O Clinical reader instructions

Provided below are the instructions used to guide the clinical reader toward providing a rating. The clinical reader study is set up as follows: (1) the clinician is provided detailed information about the nature of the study (see below), (2) the doctor is informed about what to look for duing the dialogue, (3) the doctor is provided a 20-turn patient-doctor-measurement-moderator dialogue produced by AgentClinic (either correct or incorrect), and (4) this repeats for 20 dialogues.

##### Informing clinician:

Presented below is dialogue from a medical simulation, where a large language model is acting as the doctor and the patient. The patient agent is supposed to represent a real patient and the doctor is supposed to diagnose this patient, asking appropriate questions and ordering the right medical scans.

##### Doctor realism (Initial):

Pay attention to the realism of the doctor agent dialogue and the decisions they make.

##### Patient realism (Initial):

Pay attention to the realism of the patient agent dialogue.

##### Measurement realism (Initial):

Pay attention to the realism of the measurement results returned by the measurement agent based on the doctors medical scan order.

##### Doctor Empathy (Initial):

Pay attention to the doctor’s empathy during the dialogue.

##### Doctor realism (Follow-up):

How realistic was the doctor’s dialogue compared with real doctors interactions with real patients on a scale of 1-10 (1=not realistic at all, 5=semi-realistic, 10=very realistic)?

##### Patient realism (Follow-up):

How realistic was the patient’s dialogue compared with real doctors interactions with real patients on a scale of 1-10 (1=not realistic at all, 5=semi-realistic, 10=very realistic)?

##### Measurement realism (Follow-up):

How realistic were the medical scan results based on the doctor’s scan orders on a scale of 1-10 (1=not realistic at all, 5=semi-realistic, 10=very realistic)?

##### Doctor Empathy (Follow-up):

How empathetic was the doctor on a scale of 1-10 (1=not empathetic at all, 5=semi-empathetic, 10=very empathetic)?

Figure: Figure 9: Diagnostic accuracy on AgentClinic-Spec based on medical specialty (right). Accuracy relative to the highest performing specialty by model (right).
Refer to caption: extracted/5923728/figures/AgentClinic-Spec_Graphs.png

**Table 2: Performance Comparison Across Different Languages for Various Models (Sorted by Claude 3.5 Performance)**
| Language | Claude 3.5 | GPT-4 | GPT-4o | Llama3-70b | GPT-3.5 | GPT-4o-mini |
| --- | --- | --- | --- | --- | --- | --- |
| English | 53.2 | 40.2 | 35.5 | 21.4 | 36.3 | 10.3 |
| Hindi | 51.1 | 16.8 | 28.9 | 2.8 | 2.8 | 0.93 |
| French | 50.5 | 24.52 | 7.47 | 3.73 | 18.69 | 3.7 |
| Spanish | 48.7 | 19.6 | 27.1 | 0.0 | 28.0 | 10.1 |
| Korean | 47.4 | 20.56 | 3.73 | 6.5 | 35.4 | 1.86 |
| Persian | 45.3 | 14.0 | 19.6 | 4.67 | 1.86 | 0.93 |
| Chinese | 42.9 | 11.21 | 21.49 | 4.3 | 13.08 | 0.93 |
| Average | 48.4 | 20.9 | 20.5 | 6.2 | 19.5 | 4.1 |

Figure: Figure 10: Diagnostic accuracy on AgentClinic-Lang based on base language (right). Accuracy relative to English by model (right).
Refer to caption: extracted/5923728/figures/AgentClinic-Lang_Graphs.png

**Table 3: Performance Comparison Across Different Medical Specialties for Various Models (Sorted by Claude 3.5 Performance)**
| Specialty | Claude 3.5 | GPT-4 | GPT-4o | Llama3-70b | GPT-3.5 | GPT-4o-mini |
| --- | --- | --- | --- | --- | --- | --- |
| Internal Medicine | 78.3 | 65.2 | 30.4 | 39.1 | 47.8 | 0.0 |
| Otolaryngology | 76.7 | 56.6 | 40.0 | 30.0 | 60.0 | 6.7 |
| Gynecology | 74.3 | 68.5 | 34.2 | 22.9 | 57.1 | 5.7 |
| Orthopaedics | 70.6 | 61.7 | 50.0 | 15.1 | 58.8 | 14.7 |
| Pediatrics | 69.5 | 52.1 | 43.4 | 43.5 | 52.1 | 8.7 |
| Geriatrics | 63.3 | 40.0 | 23.3 | 10.0 | 46.6 | 0.0 |
| Emergency | 58.1 | 32.3 | 32.2 | 16.1 | 41.9 | 6.5 |
| Ophthalmology | 56.5 | 65.2 | 39.1 | 47.8 | 52.1 | 4.3 |
| Psychiatry | 53.3 | 60.0 | 46.7 | 23.3 | 50.0 | 0.0 |
| Average | 66.7 | 55.7 | 37.7 | 27.5 | 51.8 | 5.2 |

Figure: Figure 11: Diagnostic accuracy on AgentClinic-MedQA based on tool use (right). Accuracy relative to baseline score (right).
Refer to caption: extracted/5923728/figures/ToolsPerf.png

**Table 4: Performance Comparison Across Different Agent Tools.**
| Agent Tool | Claude 3.5 | GPT-4 | GPT-4o | Llama3-70b | GPT-3.5 | GPT-4o-mini |
| --- | --- | --- | --- | --- | --- | --- |
| Zero-Shot CoT | 48.1 (-5.1) | 40.3 (+0.1) | 39.3 (+3.8) | 35.5 (+11.1) | 31.2 (-5.1) | 4.7 (-4.6) |
| One-Shot CoT | 51.4 (-1.8) | 41.1 (+0.9) | 39.6 (+4.1) | 36.3 (+12.2) | 32.7 (-3.6) | 14.0 (+3.7) |
| Reflection CoT | 40.2 (-13.1) | 42.2 (+2.0) | 37.3 (+1.8) | 40.1 (+18.7) | 29.8 (-6.5) | 19.6 (+9.3) |
| Adaptive RAG (Book) | 48.6 (-4.6) | 38.3 (-1.9) | 30.8 (-4.7) | 21.4 (+0.0) | 27.1 (-9.2) | 14.9 (+4.6) |
| Adaptive RAG (Web) | 52.4 (-0.8) | 43.9 (+3.7) | 29.9 (-5.6) | 25.2 (+3.8) | 28.1 (-8.1) | 15.2 (+4.9) |
| Notebook | 56.1 (+2.9) | 43.2 (+3.2) | 43.0 (+7.5) | 41.1 (+19.7) | 28.0 (-8.3) | 4.8 (-4.5) |

Figure: Figure 12: Demonstration of increase in performance via experiential learning with Llama 3-70B
Refer to caption: extracted/5923728/figures/ExpLearningExample.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. How do you plan to address the issue of potential bias in the training data of proprietary models like GPT-4 and Claude 3.5? Can you provide more details on the steps you have taken to mitigate this bias?
2. The simulated clinical environment in AgentClinic seems to be relatively simple compared to real-world clinical settings. How do you plan to expand and improve the benchmark to better capture the complexity of actual clinical practice?
3. The evaluation metrics used in the paper, such as patient compliance and consultation ratings, are subjective. How do you plan to validate and improve the reliability of these metrics?
4. From the perspective of AI Hospital, some of the core contributions claimed by this paper will be greatly weakened, and I wonder if the authors can explain why AI Hospital is not cited. Alternatively, can you explain the similarities and differences between this article's framework and the AI ​​Hospital framework?
soundness: 3
strengths: The paper presents an approach to evaluating LLMs in clinical environments by introducing AgentClinic, a multimodal agent benchmark. This is a departure from the traditional static question-answering evaluations and provides a more realistic and comprehensive assessment of LLMs' capabilities in medical diagnosis. The incorporation of biases into the benchmark is an original contribution, as it allows for the study of how biases can affect the performance of LLMs and patient perception.

The paper is well-written and easy to follow, with clear explanations of the benchmark design, agent roles, and evaluation metrics. The figures and tables are also well-designed and help to illustrate the key findings.

The work may has implications for the development and evaluation of medical AI systems. The study of biases in clinical environments is also of great importance, as it can help to improve the fairness and reliability of medical AI systems.
summary: This paper presents AgentClinic, a multimodal agent benchmark for evaluating large language models (LLMs) in simulated clinical environments. It challenges the traditional static question-answering evaluations by introducing interactive, dialogue-driven, sequential decision-making scenarios. The benchmark includes patient interactions, multimodal data collection, and tool usage, covering nine medical specialties and seven languages.
weaknesses: The study does not consider the impact of longitudinal data on the performance of LLMs, which is an important aspect of clinical decision-making.

While the paper introduces several novel evaluation metrics, such as patient compliance and consultation ratings, these metrics may be subjective and difficult to measure accurately.

One of my main concerns is that the core framework of this article seems to have a large overlap with AI Hospital (which also uses a multi-agent framework to evaluate the ability of doctor agents in multi-round medical interactions), but this article does not mention AI Hospital at all. 

Ref: AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. What is the rationale behind an N of 20 for the interaction time when assessing the diagnostic accuracy of AgentClinic-MedQA? I’m surprised at the meaningful drop in diagnostic accuracy from 52% to 25%. 

2. Although Claude 3.5 is reported to have the highest accuracy on both AgentClinic-MIMIC-IV and AgentClinic-MedQA, its performance varies by as much as 13 percentage points. What factors contribute to this moderate variability? Does this suggest issues with model robustness or potential limitations in the dataset or experimental design?

3. Have the authors conducted a detailed error analysis to identify the types of questions/ cases the models struggle with the most? Are there specific failure modes common to all the models, or do different models exhibit distinct weaknesses?
soundness: 3
strengths: Dataset: This is a novel and comprehensive benchmark that closely mimics physician-patient interactions. Its inclusion of diverse medical scenarios ensures a comprehensive assessment of model capabilities, providing a challenging benchmark for evaluating model accuracy and generalization.

Approach: The authors use a robust benchmarking approach that includes multiple state-of-the-art models (e.g., Claude 3.5, GPT-4, Mixtral-8x7B, Llama 3, etc.) evaluated on the same task, and quantify uncertainty in performance. This methodology ensures a fair and statistically rigorous comparison of the models, enhancing the reliability and transparency of the results.

Experiment: The human evaluation ratings provided by physicians adds a layer of real-world applicability of this benchmark.
summary: Recognizing that existing static QA benchmarks often fail to reflect the complexities of clinical decision-making tasks, the authors propose AgentClinic, an open-source multimodal agent benchmark for simulating clinical environments. The framework introduces patient agents (informed by real clinical cases), doctor agents, a measurement agent, and a moderator, with agents exhibiting 24 different biases. The patient cases represent nine medical specialities across seven multilingual environments. The study reveals significant performance differences across models, highlighting the impact of biases and tool integration, with models like Claude-3.5 and Llama-3 demonstrating notable improvements with tools like adaptive retrieval and note-taking.
weaknesses: Stigmatizing language in medical records can influence not only how physicians perceive patients, but also how treatment decisions are made. Physicians have been found to prescribe pain medication less often when patient notes contain stigmatizing versus neutral language (P Goddu et al., 2018; Kelly et al., 2010). This dynamic can influence treatment outcomes and the overall patient-provider relationship. The clinical cases in the paper could benefit from more nuanced language that reflects these concerns, particularly in how bias is conveyed through medical documentation. Additionally, the paper’s "Bias and Patient Agent Perception" section could be expanded to explore patient trust in the healthcare system.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: See Weaknesses
soundness: 3
strengths: Overall, the paper is well-written with clear logical flow and is easy to follow. The motivation is clearly presented and makes sense: "Existing diagnostic challenges are not static QAs, but are interactive, dialogue-driven, sequential decision-making environments that require data collection, ordering appropriate medical exams, and understanding medical images across patients with unique family histories, lifestyle habits, age categories, and diseases." This naturally addresses key limitations in previous medical LLM agents' static QA-oriented tasks. The human evaluations from experts make the presented experiments solid. The discussion is also impressive and comprehensive.
summary: The paper introduces AgentClinic, a multimodal benchmark that evaluates language models' medical diagnostic abilities through **interactive dialogue** and examinations rather than static questions. Using four agents and incorporating biases, tools, and real clinical cases across specialties and languages, it demonstrates Claude-3.5's superior performance while revealing varied capabilities among models in *tool usage* and *bias handling*.
weaknesses: Weaknesses discussed in the Discussion section:
- The simulated clinical environments are currently simple; more methods could be benchmarked:
  1. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (https://arxiv.org/abs/2305.19118)
  2. MedAgents (https://aclanthology.org/2024.findings-acl.33/)
  3. ReConcile (https://arxiv.org/abs/2309.13007)
- Uncertainty issues

Suggestions for improvement:
- Include a statistics table for utilized/built datasets, covering sample size, dataset modalities included, task types/descriptions
- Address ethics issues with closed-source LLMs by replacing them with open-source LLMs running offline to prevent patient-sensitive data leakage
- Fix typography: use ``xxx'' in LaTeX for quotes
- Add more multi-agent collaboration/MDT baselines
- Evaluate sensitivity to various prompting strategies

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Why is a measurement agent needed, and how does its role differ from that of a tool? I hope the author can provide a deeper explanation.
- It seems that the doctor agent in AgentClinic-MedQA has the opportunity to gather more information through multiple rounds of interaction compared to the doctor agent in MedQA. However, in actual tests, the performance of the doctor agent in AgentClinic-MedQA may be inferior to that of the doctor agent in MedQA. There could be many reasons for this phenomenon. Can the author provide an intermediate metric, such as the amount of useful information obtained by the doctor agents in both AgentClinic-MedQA and MedQA, to further determine the causes of this phenomenon?
- Is it possible to have human doctors interact with simulated patients under the most basic settings and then compare the performance differences between human doctors and LLM doctors? This might provide a more intuitive comparison.
soundness: 3
strengths: - Built an evaluation framework for multiple agents to assess the performance of LLM models from various dimensions, including diagnostic performance and patient experience.
- Constructed a comprehensive test set to measure LLM's diagnostic capabilities in a simulated environment from multiple settings, including performance of different models in biased environments, multilingual, multi-department, and multi-modal scenarios.
- Tested several representative closed-source and open-source models and provided an objective evaluation.
summary: The article establishes a dynamic evaluation benchmark by simulating doctor-patient dialogues to assess model capabilities. Detailed experiments were conducted in scenarios involving bias, multilingualism, multiple departments, and multimodality, providing the community with a comprehensive evaluation framework.
weaknesses: - The article's simulation of interactions between multiple agents is relatively simple, with the main interactions in the experiment limited to between the doctor agent and the patient agent.
- The article does not conduct an in-depth analysis of the reasons for the model's performance differences on AgentClinic-MedQA and MedQA.
- Although the article introduces human scoring of the model's performance, it does not involve humans in the simulation experiments, making it difficult to measure the differences between the model's performance and human performance.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: Please refer to the weaknesses.
soundness: 3
strengths: Some conclusions may hold clinical relevance
summary: In this work, the authors introduce AgentClinic, a multimodal benchmark for evaluating LLMs within simulated clinical environments. The pipeline’s performance is assessed using the MedQA, MIMIC-IV, and NEJM datasets.
weaknesses: My primary comments are as follows:

1. The paper is challenging to follow, as many crucial details are buried in the lengthy appendix rather than presented in the main text. For instance, it is unclear what specific biases are explored in this work and how they are defined. Additionally, how agent tools are integrated within the pipeline, what specific data are extracted from the MIMIC-IV dataset, the nature of the NEJM case challenge dataset, and whether it includes QA pairs all need clarification. Furthermore, the difference between specialist case use reports and general QA tasks should be explained. I recommend providing brief explanations for any non-standard terms in the main text for improved clarity.

2. Although the paper claims to simulate a real-world clinical environment, several settings and use cases seem impractical. For example, what is the intended purpose of the measurement agent? Why can’t the complete records be provided directly, or why doesn’t the doctor agent extract values directly from the database? Additionally, evaluating a patient agent seems unrealistic within actual clinical workflows.

3. Since the study focuses on evaluating LLMs within clinical contexts, the benchmark appears to lack some medical LLMs.

Meta-review: *Note: This decision was confirmed by both the Senior Area Chair and the Area Chair.*

The paper introduces AgentClinic, a benchmark that uses multiple agents to make medical decisions. Specifically, the paper creates four LLMs which interact as a doctor, patient, measurement agent (e.g., gives asked-for lab tests), and moderator agent (i.e., determines if doctor agent made correct decision) where doctor and patient are allowed to interact in a fixed number of messages. Experiment results are compared against asking an LLM to directly make clinical decisions given full information. 

The paper has several strengths. The experiments are extensive: the results cover 7 LLM models,  at least two modalities across text, tabular, and image data, 7 languages, nine specialist settings, and 24 bias evaluations. The paper is well-written and clearly outlines the pipeline of experiments. The figures and tables convey the experiment results well, and the key findings are salient. 

The paper has several key weaknesses, which were heavily discussed in the reviews and discussion process. The main concern, which I agree with, is that although the experiment setup is extensive, the clinical practicality of the setting is not well-motivated. The main contribution of this work is the benchmarking of models within this simulated clinical environment. The salient question is then, how realistic is this setup or at least how generalizable are the findings from this environment? If the AgentClinic framework does not translate well to the real-world clinical setting or the results are heavily weighed by extraneous factors like agent mis-specification, then the findings are of little use to the healthcare AI community. By creating a setting where four language agents interact with each other, it is not clear at all if any decline in performance (e.g., Fig 2 and Fig 3) are due to poor performance of any single (or combination of) language models as a patient agent, measurement agent, doctor agent, or moderator agent. While I appreciate Fig 2 exploring the effects of changing which LLM was used for which agent, it is not convincing of the clinical usability --- or generalizability of insights --- in the entire framework. 

More detailed ablation or decomposed evaluation experiments are needed. Appendix F covers a few examples of principled analysis to address obvious concerns, several of which were added by reviewer request. However, there are several unanswered questions that make it hard to have confidence in the generalizability of the results. 

For example:
 - **(Based on Section F.1 and response to Reviewer mXnj) How forthcoming or accurate is the patient agent?** If performance of AgentClinicQA decreases both when you decrease N from 10 and also when you increase N from 10, wouldn't that suggest that the real problem is how well the doctor agent solicits information and gives diagnosis (especially since the doctor agent sometimes does not give a diagnosis at all)? How would prompt engineering or more tailored system instructions help here? Is it possible that a better directed model would reverse the findings of AgentClinicQA?
 - **(Based on the newly added Section F.3 and response to Reviewer 6dqE) Are results from AgentClinicQA entirely explained by doctor agents not being able to solicit information efficiently?** The new experiment in F.3 annotates the amount of original MedQA information is revealed where GPT-4 was the doctor agent. However, the small size of this experiment (only GPT-4 model was examined) is disappointing. It would be more useful to see if the amount of information revealed explains the differences across ALL discrepancies between MedQA and AgentClinic-MedQA for all LLM models as doctor agents.
 - **How accurate is the moderator agent?** It would appear that if all evaluation is based on yet another language agent, we would want to make sure that the moderator agent is correct. 
 - **How accurate is the measurement agent?** In a response to Reviewer 6dqE, the authors claim that this is not of "high relevance to the main takeaways of the paper", especially because there are "agent systems in development ... to retrieve specific patient information in the EHR". I would argue that without critically examining every component of the pipeline, it is difficult to tell what the insights of the results would be. If a database is equally reliable, why not use that? Adding an extra measurement agent here feels unnecessarily complicated, especially when it includes yet another un-ablated component.
 - I will note that the authors include a small human evaluation study (Figure 8) using three individuals with medical degrees. Because the study only evaluates for realism and empathy, I do not consider this helpful for determining performance of each component of AgentClinic.

In making my recommendation, I weighed both the extensive experiment setup with the potentially limited clinical applicability of the setup.

Decision: Reject