MedRAX: Medical Reasoning Agent for Chest X-ray
Authors: Adibvafa Fallahpour
Jun Ma
Alif Munim
Hongwei Lyu
BO WANG
Venue: ICML 2025 poster (2025)
Keywords: healthcare
agent
multimodal
chest X-ray
benchmark
Primary area: applications->health_medicine
TL;DR: We introduce MedRAX, a state-of-the-art AI agent for chest X-ray interpretation.

Abstract: Chest X-rays (CXRs) play an integral role in driving critical decisions in disease management and patient care. While recent innovations have led to specialized models for various CXR interpretation tasks, these solutions often operate in isolation, limiting their practical utility in clinical practice. We present MedRAX, the first versatile AI agent that seamlessly integrates state-of-the-art  CXR analysis tools and multimodal large language models into a unified framework. MedRAX dynamically leverages these models to address complex medical queries without requiring additional training. To rigorously evaluate its capabilities, we introduce ChestAgentBench, a comprehensive benchmark containing 2,500 complex medical queries across 7 diverse categories. Our experiments demonstrate that MedRAX achieves state-of-the-art performance compared to both open-source and proprietary models, representing a significant step toward the practical deployment of automated CXR interpretation systems. Data and code have been publicly available at https://github.com/bowang-lab/MedRAX

MedRAX: Medical Reasoning Agent for Chest X-ray
Adibvafa Fallahpour 1234
Jun Ma *23 Alif Munim
*3 Hongwei Lyu Bo Wang" 1236
Abstract
significant time burden radiologists, requiring hours of careful analysis (Bahl et al., 2020; Adibi et al., 2025).
Chest X-rays (CXRs) play an integral role in driv- ing critical decisions in disease management and patient care. While recent innovations have led to specialized models for various CXR interpre- tation tasks, these solutions often operate in iso- lation, limiting their practical utility clinical practice. We present MedRAX, the first versatile Al agent that seamlessly integrates state-of-the- art CXR analysis tools and multimodal large lan- guage models into a unified framework. MedRAX dynamically leverages these models to address complex medical queries without requiring addi- tional training. To rigorously evaluate its capa- bilities, we introduce ChestAgentBench, a com- prehensive benchmark containing 2,500 complex medical queries across 7 diverse categories. Our experiments demonstrate that MedRAX achieves state-of-the-art performance compared to both open-source and proprictary models, representing significant step toward the practical deployment of automated CXR interpretation systems. Data and code have been publicly available at https: //git hubb. cow/bowarg....oooooooooo.cw..... Lab /RAX.
The gradual introduction of AT into clinical practice has demonstrated promising potential to alleviate this burden. Task-specific AI models have shown success in automating various aspects of CXR interpretation. from classification and segmentation to automated report generation (Yang et al., 2017; Huang al., 2023; Tanno et al., 2024; Ouis & Akhloufi, 2024). When integrated into clinical workflows, these tools have improved report turnaround times and in- terobserver agreement (Baltruschat et al., 2021; Ahn et al., 2022; Pham et al., 2022; Shin et al., 2023). However, the fragmented nature of these solutions- each operating in iso- lation-hai has hindered their widespread adoption in practical clinical settings (Erdal et al., 2023; Fallahpour et al., 2024).
Foundation models (FMs), including large language mod- els (LLMs) and large multimodal models, are promising solutions to this fragmentation, enabling unified, scalable AI-driven medical image-text reasoning. OpenAl's GPT-4 established dominance of this approach with unprecedented scale. Trained on enormous multimodal data, it demon- strated exceptional medical understanding and reasoning without explicit training (Nori et al., 2023; Yan et al., 2023; Javan et al., 2024; Eriksen et al., 2024; Baghbanzadeh et al., 2025). LLaVA-Mes (Li et al., 2024b), trained on 15 mil- lion biomedical figure-caption pairs, established new bench- marks in medical visual question answering (VQA) with strong zero-shot image interpretation. CheXagent (Chen et al., 2024a) focused CXR analysis, achieving GPT-4- level performance with significantly fewer parameters.
1. Introduction
Chest X-rays have been widely used to make critical decisions in disease detection, diagnosis, and monitoring, comprising the largest proportion of over 4.2 billion diagnostic radiology procedures performed annually worldwide (United Nations Scientific Committee on the Effects of Atomic Radiation, 2022). However, the systematic evaluation of key anatomical structures places
While FMs have advanced the field, they face critical lim- itations hindering direct clinical application. L.Ms expe- rience hallucinations and inconsistencies reasoning, par- ticularly concerning where medical accuracy paramount, They struggle with complex multi-step reasoning required for diagnostic tasks, failing to systematically evaluate all relevant anatomical structures integrate findings across different image regions. Their end-to-end architecture lacks the transparency and specialization of purpose-built medi- cal Al tools. These limitations suggest more structured, tool-based approach combining foundation model flexibility with clinical AI system reliability.
"qqual contribution 'Department of Computer Seience, Uni- versity of Toronto, Toronto, Canada 'Vector Institute, Toronto, Canada UUnivesstyy Health Network, Toronto, Canada *Cohere, Toronto, Canada CCohere Labs, Toronto, Canada 'Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Canada. Correspondence to: Adibvafa Fallahpour <adib- fallahpor @ wtt.lluttoooo.cc...
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MedRAX: Medical Reasoning Agent for Chest X-ray
To bridge this gap, we present MedRAX, the first special- ized Al agent framework for CXR interpretation. Our key contributions include:
gent's broad focus across imaging modalities may dilute its domain-specifie expertise. Additionally, MMedAgent requires retraining integrate new tools, reducing its flexi- bility for adapting to evolving clinical workflows.
MedRAX, a specialized Al agent framework that seam- lessly integrates multiple CXR analysis tools without additional training, dynamically orchestrating specialized components for complex medical queries.
More recently, ol ol-powered Al agents (Jaech et al., 2024) have been proposed as an alternative to traditional model- based approaches, demonstrating strong multi-step reason- ing and improved diagnostic consistency. However, these systems also face critical challenges: (1) high computational demands, making them impractical for real-time applica- tions, (2) closed-source and proprietary nature, limiting customization and adaptation to specific medical require- ments, and (3) redundant reasoning in simpler tasks, leading to inefficiencies in tool selection and execution.
ChestAgentBench, a comprehensive evaluation frame- work with 2,500 complex medical queries across 7 cat- egories, built from 675 expert-curated clinical cases to assess multi-step reasoning in CXR interpretation.
Experiments show that MedRAX outperforms both general-purposs and biomedical specialist models, demonstrating substantial improvements in complex rea- soning tasks while maintaining transparent workflows.
Alongside these agentic frameworks, specific large vision- language models like RaDialog (Pellegrini et al., 2025) have been developed for radiology report generation and conver- sational assistance. Furthermore, models such as M4CXR (Park et al., 2024) are exploring the multi-task potentials of multi-modal LLMs for chest X-ray interpretation.
Development of a user-friendly interface, enabling flexi- ble deployment options from local to cloud-based solu- tions that address healthcare privacy requirements.
2. Related Work
2.3. Evaluation Frameworks
2.1. LLM-based Agent Architectures
To systematically evaluate LLM--bssd agents, several benchmarks have been introduced. AgentBench (Liu et al., 2023) assesses multi-step reasoning, memory retention, tool use, task decomposition, and interactive problem-solving, revealing that even perpefforming models like GPT-4o and -laude-3.5-Sonnee struggle with long-term context reten- tion and autonomous decision-making Expanding on these limitations, MMAU (Yin et al., 2024) evaluates agent capa- bilities across five domains tooil use, graph- reason- ing, data science, programming, and mathematics. Results highlight persistent weaknesses in structured reasoning and iterative refinement.
The emergence of Al agents built upon LLMS has funda- mentally changed how we approach autonomous reason- ing, planning. and tool use. Recent surveys on ALM--ased agents (Xi et al., 2025; Zhao et al., 2023: Masterman et al., 2024) have outlined generalizable agent framework com- prising three core components: (1) a reasoning engine driven by LLMs, (2) perceptual modules that process multimodal inputs, and (3) action mechanisms that execute API calls, retrieve information, or interact with external tools.
This paradigm shift has enabled AI agents to surpass tradi- tional task-specific models by dynamically adapting to di- verse applications without additional training. However, de- spite these advances, there are very few LLM-based agents that have been evaluated for Nomain-specift robustness, particularly in high-stakes medical applications where hal- lucinations, lack of systematic reasoning, and specialized tool integration remain significant challenges.
In software engineering, SWE-bench (Jimenez et al., 2023) presents 2,294 real-world GitHub issues to evaluate LLMs' ability to modify large codebases. By January 2025, the best-performing agent has solved less than 65% of issues, underscoring the challenges of multi-file reasoning and iter- ative debugging. These benchmarks collectively highlight LLM agents' deficiencies in contextual understanding, struc- tured planning, and domain-specine tool use, reinforcing the need for specialized, clinically validated AI frameworks in high-stakes applications such as medical imaging.
2.2. Medical Agents
By enabling LMMs to operate in a collaborative, agentic set- ting, frameworks such as MDAgents (Kim et al., 2024) have demonstrated enhanced clinical reasoning through multi- agent interaction. Similarly, MMedAgent (Li et al., 2024a) explores tool integration across multiple medical imaging modalities, allowing LMMs to leverage external machine learning models for more robust decision-making.
Beyond general-purpose benchmarks, MedAgentBench (Jiang et al., 2025) assesses LLMs' ability to retrieve pa- tient data, interact with clinical tools, and execute structured decision decision-making in interactive healthcare environments. Re- sults indicate that even the best-performing model, GPT-40, achieves only 72% accuracy, with substantial performance variability across different medical tasks.
However, MDAgents introduces significant computational overhead due to multi-agent coordination, while MMedA-
2

---

MedRAX: Medical Reasoning Agent for Chest X-ray
Algorithm MedRAX ReAct Framework
User Query
Input:
Memory LangChain
Sate
Q: User query
Σ: Set of input CXR images (can be empty)
Visual QA CheXagent. LLaVA-Med
T: Available medical AI tools
M: Memory buffer
Report Generation CheXpert Plus
bmaz Maximum allowed time
Output:
Segmentation MedSAM, ChestX- Det
R: Final response to query
MedRAX
Tools
Initialize:
Grounding Maira-2
start GetCurrent(Tmme()
ReAct Loop
state = Observe M)
Classification TorchXRayVision
while GetCurren'Timel tstart < tmax do
thoughts = Reason (sttee, M)
Image Generation
Agent Response
RoentGen
RequireseUserlnputtthuuuuu then return GeneraaeeUseerPPoop
M)
end if
Figure Architecture of MedRAX. framework implements ReAct loop that processes user queries by integrating short- term memory (LangChain) and specialized medical tools for vi- sual QA (CheXagent (Chen al., 2024b), LLaVA-Med (Li et al., 2024b)), segmentation (MedSAM (Ma et al., 2024; Kirillov et al., 2023; Ma et al., 2025), ChestX-Det (Lian et al., 2021; Zhao et al., 2017)), grounding (Maira-2 (Bannur et al., 2024)), report genera- tion (model trained CheXpert Plus (Irvin et al., 2019; Chambon et al., 2024)), classification (TarchXRay Vision (Cohen et al., 2022; 2020)), and image generation (RoentGen (Chambon al., 2022)).
if Chnnenorackssspnee(hhuunitttt then
return GenerateRessponsettoouuuu M)
end if
tools = SelctTToobsthhouggtts.::
results = ExccuteePrralll((tool, state)
M = MM {(hooughss lools, resulls))
state Observe(state, M)
end while
return GmneraeTTmmmoreeepaassssstdddr
AgentClinic (Schmidgall et al., 2024) offers comprehen- sive multimodal agent benchmark to evaluate AI in sim- ulated clinical environments, emphasizing interactive dia- logue and active data collection processes. These findings reinforce the critical need for domain-specific benchmarks and highlight that while progress has been made, current medical agents still face significant challenges in tool inte- gration, efficiency, and flexibility.
3.1. LLM Driven Agent
MedRAX employs a LLM the core to drive ReAct (Reasoning and Acting) loop, which breaks down complex medical queries into sequential analytical steps (Yao et al., 2023). The system processes user query through iterative cycles of (1) observation analyzing the current state and query, (2) thought determining required actions, and (3) ac- tion executing relevant tools and integrating findings from previous steps to inform subsequent reasoning. Through- out this process, the system maintains short-term memory of user interactions, tool outputs, and images to support multi-turn turn interactions. The reasoning loop continues until the system either generates a response or asks the user for additional input (Algorithm 1). Further details of the core methodology are provided in Appendix A.
3. MedRAX
We present MedRAX, an open-source agent- based frame- work that dynamically reason, plan, and execute multi- step CXR workflows. Compared to previous approaches (Chen et al., 2024b; Bansal et al., 2024), MedRAX integrates multimodal reasoming abilities with structured tool-based decision-man making, allowing real-time CXR interpretation without unnecessary computational overhead. By balanc- ing computational efficiency with domain specialization and eliminating the need for retraining when incorporating new tools, MedRAX offers greater adaptability to evolving clinical needs. Our framework integrates heterogeneous machine learning models-from lightweight classifiers to large LMMs L.MMs-speciallne specialized for diverse downstream tasks, al- lowing it to decompose and solve complex medical queries by reasoning across multiple analytical skills (Figure 1).
3.2. Flexible Tool Integration
MedRAX integrates state-of-the-at models for various downstream CXR interpretation tasks:
Visual Question Answering (VQA).
Answering free-form questions about CXR images by combining visual understanding with medical knowledge.
Models: CheXagent, a vision-language foundation model trained on CheXinstruct, with over 8.5M samples across
3

---

McdRAX: Medical Reasoning Agent for Chest X-ray
35 tasks, capable of fine-grained visual reasoning and CXR interpretation (Chen et al., 2024b).
User Query
Ground the main disease this CXR.
LLava-Med, a biomedical 7B VLM, trained on 600K biomedical image-caption pairs from PMC-15M and 60K cntructtiontuninng data (Li et al., 2024b).
MadRAX
MedRAX
<thought> First, need to diagnose the disease.
<action> ("toal": "chest_xereeee _rayyrryy.... classifier", "image_path"" pacte___ah....]
Segmentation.
Partitioning CXR images into semantically meaningful regions by assigning each region to anatomical structures.
<toob ('Effusion":85"" 85%, "Atelectasis" 66%.
<thought> The most probably disease is effusion. I will ground it.
oftte-of-theaarr biomedical segmen- tation model trained on 1,570,263 medical image-mask pairs, covering 10 imaging modalities and over 30 cancer types (Ma et al., 2024; 2025).
Models: MedSAM,
<action> ("tool": pphraae"
"pleurl effusion")
<tool> ("bounding_ boxes": [(0.05, 0.39, 0.42, 0.63)). ...
<tool> ("tool": "image visualizer", "image_ _atee___a_h____
PSPNet model trained on ChestX- dataset, consisting of 3,578 images from NIH ChestX-14, annotated with 13 common categories of diseases or abnormalities (Lian et al., 2021; Zhao et al., 2017).
<response> CXR shows clear signs of pleural effusion.
Here a visuelizetion of grounded result:
User Query Segment the major affected lung.
Grounding.
Localizing specific visual regions in medical images that correspond to given textual descriptions findings.
MedRAX
<observation> Based on boundings, right lung is mostly affected.
Model: Maira-2, a 7B VLM trained on MIMIC- CXR, PadChest, and USMix datasets, excellent in grounding specific phrases generating findings of radiology report with or without grounding (Bannur et al., 2024).
<thought> will segment the right lung.
<action> ("tool": "segmentation" "organs": :: ("right lu.g"]....)
<response> e is mostly in the right lung.
Report Generation.
Here is the segmented region:
Writing radiology reports with findings and impressions.
Model: A V2inV2 Transformer with a two-layer BERT decoder trained on 223K sspert-annotated reports from CheXpert Plus dataset to generate findings and impres- sions (Irvin et al., 2019; Chambon et al., 2024).
Figure 2. MedRAX Interaction Flow. An example of how MedRAX handles multi-turn conversation through its ReAct loop (<thought>, <action>>, <observation>> along with tool out- puts and final response. For clarity, the production interface shows only tool outputs and agent responses.
Disease Classification. Detecting and classifying pathologies and abnormalities. Model: A DenseNet-121 model from the TorchXRay Vi- sion library, trained on NIH ChestX-ray, CheXpert, MIMIC-CXR, and PadChest datasets. can predict 18 pathology classes including Pneumonia, Pneumothorax, Edema, Effusion, and Nodule (Cohen et al., 2022; 2020).
subsequent tool selection. Through its memory, MedRAX caches tool outputs to prevent redundant computations, op- timizing performance in multi-step analyses that might ref- erence the same intermediate results.
The framework supports parallel execution of independent tools and flexible deployment configurations tools can be quantized for efficiency and distributed across CPU or GPU, Figure 2 shows an example user interaction with MedRAX.
Chest X-ray Generation.
Synthesizing realistic CXR images from text descriptions of anatomical features and pathologies.
Model: RoentGen, medical vision- language model adapted from Stable Diffusion, trained on the MIMIC- CXR dataset, generates diverse, high-fidelity chest X-rays given text prompts (Chambon et al., 2022).
3.3. Modularity
MedRAX is built on the LangChain and LangGraph frame- works. The reasoning engine can be any LLM, accommo- dating both text-only and multimodal models, from open- source to proprietary. This flexibility enables deployments ranging from local installations to cloud- solutions, addressing diverse healthcare privacy requirements. Our reference implementation uses GPT-4o with vision, while supporting integration of alternative models.
Utilities.
Processing DICOM images, generating custom plots, and visualizing figures to user.
The agent continuously monitors tool outputs and errors, incorporating these results into its reasoning loop to inform

---

McdRAX: Medical Reasoning Agent for Chest X-ray
Each tool operates an independent module with defined loading and inference. Tools can be modified, replaced, or repurposed for multiple tasks without affecting other components. Integration of new tools requires only a class definition specifying the tool's input/output formats and capabilities, with the LLM learning its usage without any training. The framework decouples tool creation from agent instantiation, enabling multiple agents share tools and allowing each to access its own customized set of tools.
From its chest imaging section, curated 675 patient cases with associated chest X-rays and complete clinical docu- mentation. These cases cover 53 unique areas of interest including lung, thorax, and mediastinum. Figure 3 provides an overview of the benchmark, showing (a) the creation pipeline, (b) patient gender distribution, (c) age distribution, and (d) most frequent anatomical areas of interest. The
dataset encompasses diverse clinical settings and broad spectrum of pathologies, with detailed breakdowns provided in Appendix B.
3.4. User-friendly Interface
4.2. Benchmark Creation
MedRAX includes a production-ready. interface built with Gradio that facilitates seamless deployment in clinical set- tings. The interface supports uploading of radiological im- ages in all standard formats, including DICOM, and main- tains an interactive chat session for natural multi-turn inter- actions. The interface further provides transparency into tool execution by tracking and displaying intermediate outputs. This end-to-end implementation enables quick integration of MedRAX into existing clinical workflows.
ChestAgentBench comprises six-choice questions, each de- signed to evaluate complex CXR interpretation capabilities.
We first established seven core competencies alongside rea- soning that are essential for CXR interpretation:
Detection: Identifying specific findings. (e.g., "Is there a nodule present in the right upper lobe?""
Classification: Classifying specific findings. (e.g.. "Is this mass benign or malignant in appearance?"))
4. ChestA genttench
Localization: Precise positioning of findings. (e.g.. "In which bronchopulmonary segment is the mass located?""
While MedRAX offers a powerful framework for complex CXR interpretation, existing medical VQA benchmarks typ- ically focus on simple, single-step reasoning tasks, mak- ing them insufficient for evaluating its full capabilities. In contrast, ChestAgentBench offers several distinctive advan- tages:
Comparison: Analyzing relative sizes and positions. (e.g., "How has the pleural effusion volume changed com- pared to prior imaging?")
Relationship: Understanding relationship of findings. (e.g., "Does the mediastinal lymphadenopathy correlate with the lung mass?")
It represents one of the largest medical VQA benchmarks, with 2,500 questions derived from expert-validated clini- cal cases, each with comprehensive radiological findings, detailed discussions, and multi- modal imaging data.
Diagnosis: Interpreting findings for clinical decisions. (e.g., "Given the CXR, what is the likely diagnosis?")
The benchmark combines complex multi-step reasoning assessment with a structured six-choice format, enabling both rigorous evaluation of advanced reasoning capabili- ties and straightforward, reproducible evaluation.
Describing specific finding attributes. (e.g., "What are the margins of the nodule- smooth, spic- ulated, or irregular?")
Characterization:
The benchmark features diverse questions across seven core competencies in CXR interpretation, requiring inte- gration of multiple visual findings and reasoning to nirror the complexity of real-world clinical decision-maie makiin...
Explaining medical rationale and thought. (e.g., "Why do these findings suggest infectious rather than malignant etiology?""
Reasoning:
These competencies are combined into five question types, each designed to evaluate specific combinations of core competencies while requiring medical reasoning:
4.1. Dataset
We utilize Eurorad, the largest peer-reviewed radiological case report database maintained by the European Society of Radiology (ESR). This database contains detailed clinical cases consisting of patient histories, clinical presentations, and multi-modal imaging findings. Each case includes de- tailed radiological interpretations across different modalities, complemented by in-depth discussions that connect findings with clinical context, and concludes with reasoned interpre- tations, differential diagnosis list and final diagnoses.
Detailed Finding Analysis: detection, localization, and characterization
Pattern Recognition & Relations: detection, classifica- tion, and relationships
Spatial Understanding: localization, comparison, and relationships
5

---

MedRAX: Medical Reasoning Agent for Chest X-ray
Eurorad
ChestAgentBench
b
Hampton hump Palla sign: Radiographic signs of pulmonary embolism
40-yeer-old
presents with right-
Gender Distribution
sided pleuriiic pain. He recently underwent Achilles tendon surgery. He has oxymal soturation, and abnormal infarmatorry parameters. Besed on figure 1, which of the following the most Iely diagnesis?
CASE 16302
Chast imaging
GPT-40
Right lobe preumonia B) Right lower lobs stelectesis
Pulmonsry emboliss
HISTORY
Pulmonary hemorrhage
Pleural effusion
Preumothoras
unkrown
675 Clinical Cases
2,500 Visual QAs
Age Distribution
d
Percentage of Cases by Area of Interest
Cempfrug
Pequerty Disniistoo
0.02
brergeto
0.2
6.055
Resp
30
50
Parient Age (Years)
Poncomaaee Cases
Figure 3. Overview of ChestAgget (a) Benchmark creation pipeline that uses GPT-4o to generate 2,500 six-choice questions from 675 Eurorad clinical cases. (b) Gender distribution, showing 55.4% male, 44. % female, and 0.45% unknown. (c) Age distribution, bimodal with a mean age of 46.0 years (SD=20.4, median=47.0 7.0 years). (d) Distribution of anatomical areas of interest across cases, with lung (51.2%) thorax (42.8%) and mediastinum (15.8%) representing the most frequently examined regions from 53 unique areas.
Clinical Decision Making: classification, comparison, and diagnosis
5. Experiments
5.1. Implementations
Diagnostic Characterization:
classification, characteri-
zation, and diagnosis
MedRAX uses GPT-4o as its backbone LLM, and we deploy on a single NVIDIA RTX 6000 GPU using the same con- figuration as described in Section integrates CheXagent (Chen al., 2024b) and LLaVA-Med (Li et al., 2024b) for visual QA, Maira-2 for grounding (Bannur et al., 2024), model trained on ChestX-Det for segmentation (Lian et al., 2021), TorchXRay Vision for classification (Cohen et al., 2022), and a model trained on CheXpert Plus for report generation (Chambon et al., 2024).
For each clinical case and question type, we first prompted GPT-4o to analyze the case and generate a six-choice ques- tion that would best assess the target analytical skills of that question type. We then instructed it to ensure the ques- tion has the necessary context from the clinical case and its correct answer could be explicitly verified from the case's s radiological findings and discussion.
We performed automated quality verification using GPT- 4o to evaluate each question for: (1) structural consistency (six-choice format with one correct answer), (2) explicit grounding in the clinical and radiological context, and (3) clear verifiability from the original Eurorad case material. Questions failing these criteria were automatically excluded.
MedRAX implements tool execution with structured ISON APT calls, where the agent formulates precise requests with required arguments (c.g., image paths, text prompts) to call target tools. We evaluate all baseline models using their official implementations and recommended configurations.
We process model responses using regex to extract letter choices. For unclear responses, errors, or timeouts, we retry up to three times. Responses that remain invalid or do not choose single choice marked incorrect.
The benchmark uses straightforward accuracy metric (per- centage of correct answers) to enable easy evaluation across different agent architectures. All questions underwent qual- ity check, during which we removed questions that exhibited issues such as ungrounded answers or missing information.
6

---

MedRAX: Medical Reasoning Agent for Chest X-ray
Table Model Performance ChestAgentRench. Accuracy (%) of five vision-language models (I.aa (Li et al., 2024b),
CheXagent (Chen et al., 2024a), Llama-3.2-900 GPT-4o, and MedRAX) compared across seven categories of our 2,500-question benchmark. MedRAX significantly outperforms both ceneral-purpose models and specialized biomedical models across all categories.
Categories
LLaVA-Med
CheXagent
Llama-3.2-90B
GP'T-4o
MedRAX
Detection
32.4
38.7
58.1
58.7
64.1
Classification
30.8
34.7
56.5
54.6
62.9
Localization
30.2
42.5
59.9
59.0
63.6
Comparison
30.6
38.5
57.5
55.5
61.8
Relationship
31.8
39.8
59.3
59.0
63.1
Diagnosis
29.3
33.5
55,9
52.6
62.5
Characterization
28.8
34.2
58.0
56.1
64.0
Overall
28.7
39.5
57.9
56.4
63.1
Table Model Performance CheXbench. Accuracy (%) five vision-language models (L-aaV-Meed (Li al., 2024h), CheXagent (Chen et al., 2024a), Llama-3.2-90R, GPT-4o, and MedRAX) compared on 238 Visual QA (Rad-Restrus and SLAKE) and 380
Image-Tex Text Reasoning questions (OpenI). MedRAX exeels in VQA while achieving overall performance.
Categories
LLaVA-Med
CheXagent
Llama-3.2-90B
GPT-4o
MedRAX
Visual QA Rad-Restruct
34.9
57.1
62.6
53.9
68.7
SLAKE
55.5
78.1
74.0
85.4
82.9
Fine-Grained Reasoning
45.8
59.0
49.2
51.1
52.6
Overall
45.4
64.7
61.9
63.5
68.1
5.2. Experimental Setup
(2) CheXbench, popular benchmark that evaluates seven clinically-relevant CXR interpretation tasks. We specifically focus on the visual question answering (238 questions from Rad-Restruch (Pellegrini et al., 2023) and SLAKE (Liu et al., 2021) datasets) and fine-graide mne-grai image-text reasoning (380 questions from Openl dataset) subsets, as they most closely mirror complex clinical workflows that require precise dif- ferentiation between similar findings.
We evaluate MedRAX AX against four models: LLaVA- Med, finetuned LLaVA-13B model for biomedical vi- sual question answering (Li et al., 2024b), CheXagent, Vicuna- 13B VLM trained for CXR interpretation (Chen et al., 2024b), along with GPT-4o and 1.2ma-3.2-90B Vi- sion as popular closed and open-source multimodal LLMs respectively. Additionally, we compare against baseline models from recent literature including RadFM (Wu al., 2023), a generalist foundation model for radiology, MAIRA- 1 (Hyland et al., 2024), specialized multimodal model for radiology report generation, LLaVA-Raa (Zambrano Chaves et al., 2024), a radiology-adaptet version LLaVA, and Med-PaLM M 84B (Tu et al., 2023), large-scale biomed- ical foundation model, with performance values obtained from M4CXR paper (Park al., 2024).
(3) MIMIC-CXR Radiology Report Generation, which evaluates single-image chest X-ray findings generation of 3,858 images from the MIMIC-CXR test set. This bench- mark assesses clinical accuracy of generated reports using micro-averaged F1 scores (mF1-14, mF1-5) and macro- averaged Fl scores (MF1-14, MF1-5) for 14 medical ob- servation labels and key findings (cardiomegaly, edema, consolidation. atclectasis, plcural effusion), respectively.
We evaluate models four complementary benchmarks:
(4) SLAKE VQA, which evaluates medical visual ques- tion answering using 114 chest X-ray test samples with close-ended questions in English, filtered from the original SLAKE test set of 2,094 samples. Performance is mea- sured by accuracy (exact matches) and recall (proportion of ground truth words present in generated responses).
(1) ChestAgentBench. our proposed benchmark described in Section 4, which assesses comprehensive CXR reasoning through 2,500 six-choice questions across seven categories: detection, classification, localization, comparison, relation- ship, characterization, and diagnosis. Model performance is measured by accuracy across all questions.
For more information on preparation and evaluation on benchmarks 3 and 4, see M4CXR paper (Park et al., 2024).
7

---

MedRAX: Medical Reasoning Agent for Chest X-ray
Table Single-image performance on MIMIC-CXR
Clinical accuracy (%) of generated radiology findings, evaluated using aveera- Fl scores for all 14/5 key labels (cardiomegaly, edema.
CheXbert Fl scores for 14 medical observations. IFF1-14/mF/p consolidation, atelectasis, pleural effusion); 1FF-14/MFI-5- 5: macro-aae
14/5 key labels. MedRAX is compared with CheXagent (Chen et al., 2024b), MAIRA-1 (Hyland et al., 2024), LLaVA-Rad (Zambrano Chaves et al., 2024), Med-PaLM M 84B (Tu et al., 2023), M4CXR (Park et al., 2024). Performance values for baseline models obtained from Park al. (Park et al., 2024).
macro-averaged averaged scores
Model
m 111-14
mF1-5
MF1-14
MF1-5
Med-PaLM M 84B
53.6
57.9
39.8
51.6
CheXagent
39.3
41.2
24.7
34.5
MAIRA-1
55.7
56.0
38.6
47.7
LLaVA-Rai
57.3
57.4
39.5
47.7
M4CXR
60.6
61.8
40.0
49.5
MedRAX
79.1
64.9
34.2
48.2
significant performance drop across all models, with the pest-performin CheXagent achieving only 59.0% accu- racy, almost equal to random performance (50% baseline).
Table Medical VQA performance on ST.AKE benchmark. Accuracy (%) and recall (%) of vision-language models on 14 chest X-ray visual question answering samples with close-ended English questions. MedRAX achieves state-of-the-att performance compared to RadFM (Wu et al., 2023), CheXagent (Chen et al., 2024b). and M4CXR (Park et al., 2024). Performance values for baseline models obtained from Park et al. (Park al., 2024).
MIMIC-CXR Radiology Report Generation. Shown in Table 3, MedRAX achieves the highest micro-averaged F1 scores with mF1-14 of 79.1% and mF1-5 of 64.9%, substantially outperforming the next-best M4CXR (60.6%, 61.8%) and other baselines. However, MedRAX shows lower macro-averaged performance, with M4CXR achiev- ing the highest MF1-14 of 40.0% and Med-PaLM M 84B leading MF1-5 at 51.6%. This suggests MedR stronger performance on more prevalent conditions.
Model
Accuracy
Recall
RadFM
68.4
69.7
CheXagent
71.1
73.2
M4CXR
85.1
86.0
MedRAX
90.35
91.23
SLAKE VQA. Shown in Table 4, MedRAX demonstrates clear superiority in medical visual question answering, achieving 90.35% accuracy and 91.23% recall. This repre- sents significant improvements over M4CXR (85.1% accu- racy, 86.0% recall) and CheXagent (71. 11.1% accuracy. 73.2% recall). The consistent performance gains across both met- rics indicate MedRAX's tool- bal-based approach effectively handles diverse medical VQA tasks while maintaining high precision in response generation.
5.3. Quantitative Analysis
ChestAgentBench. Shown in Table 1, McdRAX achieves consistently saate-of-the-a performance (63%) across all seven categories, a significant improvement over the base- line models. There is a clear performance hierarchy among models, with GPT-4o (56.4%) and Llama-3.2-90P (57.9%) performing notably better than specialized medical models like CheXaa (39.5%) (Chen al., 2024a) and LLaVA- Med (28.7%) (Li al., 2024b). Interestingly, general- purpose VLMs outperform domain-specific ones across all categories, with particularly large gaps in characterization and diagnosis tasks.
5.4. Case Studies
We present two representative cases that compare MedRAX to GPT-4o (Figure 5.3).
Medical Device Identification (Eurorad Case 17576).
CheXbench. Shown Table 2, we observe distinct per- formance patterns across different task types. On visual QA tasks, MedRAX demonstrates strong performance on Rad-Restruct (68. 7%) and SLAKE (82.9%). This notably surpasses both domain-specing CheXagent (57.1 %, 78. 78.1%) (Chen et al., 2024a) and larger purepoe models like GPT-4o (53.9%, 85.4% 85.4%) suggesting that our tool-based ap- proach particularly excels at fine- grained visual understand- ing. However, on image-text reasoning tasks, we observe
This question asks the model to determine the type of tube present in the CXR. GPT-4o incorrectly suggests an endotra- cheal tube based on the central positiong of the tube alone. MedRAX, integrated findings from multiple tools like re- port generation and visual QA, and correctly identifies chest tube despite one tool (LLaVA-Med (Li et al., 2024b)) suggesting otherwise. This demonstrates MedRAX's abil- ity to resolve conflicting tool outputs through systematic reasoning.
8

---

MedRAX: Medical Reasoning Agent for Chest X-ray
Query
2 GPT-4o
MedRAX
The tollowing chest raray is provided.
Based appearance and cantral positioning along trachea, the tube is Ikaly an endotrachoal (ET) tube, used for airway management and mechenical ventilation. However, precise identification requires professionel medical eveluesion to confirm its type and plecement.
Chest Report: identifeed a left chest tube place.
Case #17576
What tube present?
CheXagent: Confirmed presence chest tube. > LLaVA-Med GA: incorrectly identified a nasogastric tube.
A] Endotracheal (ET) tube BI Chest tube
The majorily of tooks, inckuding the detailed report and expert analysis. Indicate presence chest tube. LLaWA-Medd QA toots response seems to be an outller in this case.
idientifyy the main disease visible X-ray image and determine which lung shows more severe symptorss the condition.
The X-ray suggests potential lung condition, possibly pneumonis pulmonary edems cheracterized increased opacity in the right lung compared to the left, indicaling that disease dominant the right lung. For accurate dlagnasis and trestment, consult a healthcore professional.
-hhet X-ray Report: Moderste- slzed left pneumothorax
Case #16703
Clessifier: High probebility of mass and preumothorax. Segmentation: Left lung has a larger area and higher Intensity compared the right lung.
Highighhts presence left eneumothoras as findiigg with the right lung being less affected in this context.
Figure 4. MedRAX and GPT-4o Case Study, (Case 17576) Correct answer is chest tube. GPT-4o incorrectly identifies as endotracheal tube based on position, while MedRAX correcll identifies chest tube by integrating multiple tool outputs, even resolving conflicting tool suggestions. (Case 16703) Correct answer is left pneumothorax. GPT-4o misdiagnoses as right-sided pneumonia/edema, while MedRAX correctly identifies left pneumothorax through sequential tool application for disease detection and comparative lung analysis.
Multi-step Disease Diagnosis (Eurorad Case 16703). This questions asks about diagnosing and comparing dis- ease severity across lungs. GPT-4o misinterprets the CXR as showing pneumonia with right lung predominance. MedRAX, through sequential tool application of report gen- eration for disease identification and segmentation for lung opacity analysis, correctly determines left pneumothorax as the main finding. This demonstrates MedRAX's ability to break down complex queries into targeted analytical steps.
Limitations. While MedRAX excels in structured reason- ing, sometimes struggles with resolving contradictory tool outputs, particularly in fine fine-grained visual tasks when classi- fication and segmentation tools provide confficting interpre- tations. Additionally, the system's 's computational overhead from running multiple specialized tools can impact response times compared to end-to-end models. The framework also lacks robust uncertainty quantification mechanisms.
Future Work, Key areas for advancement include formally analyzing optimal tool utilization balance, as neither full reliance nor complete absence yielded best results. This involves exploring trade-offs between accuracy gains and computational costs, speed, error propagation, and spurious correlations in pretrained models. Systematic investigation of prompting techniques for critical tool evaluation is needed to optimize LLM-tool interactions. Reinforcement learning approaches could enhance reasoning capabilities and reduce hallucinations, following advances in models like DeepSeek- RI (Guo al., 2025). Comprehensive clinical validation remains essential for establishing real-world utility.
6. Discussion
MedRAX achieves state-of-the-ar performance in com- plex CXR interpretation tasks, outperforming both general- purpose and specialized medical models. We discover valu- able insights about structured tool use in medical AI, sug- gesting that a hybrid approach-leevrrging both large-scale reasoning capabilities and semaii- expertise offers superior performance over purely end-to-end models.
Task Decomposition. MedRAX shows that the ReAct loop dynamically composes complex reasoming chains while maintaining computational efficiency. The performance gap suggests that explicit decomposition provides advantages that scale alone cannot achieve. The process produces clear decision traces, enhancing transparency with implications beyond medical imaging for model-tool integration.
7. Conclusion
MedRAX establishes a new benchmark in AI-driven CXR interpretation by integrating structured tool orchestration with large-scale reasoning. Our evaluation demonstrates its superiority over both general-purpont and domain-specinn models, reinforcing the advantages of explicit stepwise rea- soning in medical AI. These findings highlight the potential of combining foundation models with specialized tools, a principle applicable to broader healthcare domains. Future work should focus on optimizing tool selection, uncertainty- aware reasoning, and expanding capabilities multimodal medical imaging for greater clinical impact.
Generalists Versus Specialists. A key insight is the superior performance of general-purpose models (GPT- 4o, 3l2m0-)))--00B over specialized medical models (LLaVA-Med (Li et al., 2024b), CheXagent (Chen et al., 2024a)). This suggests medical specialization may sacrifice broader reasoning capabilities from large-scale pretraining. MedRAX bridges this gap by integrating dopain-specific tools while maintaining generalist reasoning.

---

MedRAX: Medical Reasoning Agent for Chest X-ray
Impact Statement
reader performance and efficiency. JAMA Network Open, 5(8)=2222288-22229)) 2022.
MedRAX aims to improve chest X-ray interpretation, en- hancing diagnostic accuracy and efficiency as an Al co-pilot designed augment, not replace, clinical expertise. We acknowledge AI risks, including model bias, hallucinations, and data privacy, and emphasize the need for robust val- idation before any clinical deployment to ensure patient safety.
Baghbanzadeh, N., Fallahpour, A., Parhizkar, Y., Ogidi, F., Roy, S., Ashkezari, S., Khazaie, V. R., Colacci, M., Etemad, A., Afkanpour, A., and Dolatabadi, E. Advanc- ing medical representation learning through high-quality data, 2025.
Bahl, S., Ramzan, T., and Maraj, R. Interpretation and documentation of chest x-rays in the acute medical unit. Clinical Medicine, 20(2);73,, 2020.
Our development utilized the de-identified Eurorad database, and its use, along with ChestAgentBench creation, strietly adheres to the CC BY-NC-S 4.0 license; ChestAgentBench will be released under the same terms. To address data privacy, especially for datasets like MIMIC-XXR requiring stricter handling, we employed the Azure OpenAI Service configured to opt-out of data logging, following Physionet recommendations.
Baltruschat, I., Steinmeister, L.,, Nickisch, H., Saalbach, A., Grass, M., Adam, G., Knopp, T., and Itrrch, H. Smart chest x-ray worklist prioritization using artificial intelli- gence: a clinical workflow simulation. European radiol- ogy, 31:3837-3845, 2021.
MedRAX also supports local LLM deployment for max- imum data control. Our public code release will include guidelines for these derscoring our commitment to responsible Al integration in healthcare.
Bannur, S., Bouzid, K., Castro, D. C., Schwaighofer, A., Thieme, A., Bond-Taylor, S., Ilse, M., Pérez-García, F., Salvatelli, V., Sharma, H., Meissen, F., Ranjit, M., Sri- vastav, S., Gong, J., Codella, N. C. F,, Falck, F,, Oktay, O., Lungren, M. P., Wetscherek, M. T., Alvarez-Valle, J., and Hyland, S. L. Maira-2: Grounded radiology report generation, 2024.
configurations, un-
Acknowledgements
We thank Mohammed Baharoon for their assistance in pre- processing the Eurorad dataset. We are particularly grateful to Karan Singhal and Shekoofeh Azizi for their thorough feedback on ChestAgentBenchh 's design, which significantly improved the benchmark's ' clinical relevance and evaluation methodology. We also thank Yubin Kim for their insight- ful perspectives on medical Al agent architectures. This work was supported by the University of Toronto, Univer- sity Health Network (UHN) and Vector Institute.
Bansal, H., Israel, D., Zhao, S., Li, S., Nguyen, T., and Grover, A. Medmax: Mixed-modal instruction tuning for training biomedical assistants, 2024.
Chambon, P., Bluethgen, C., Delbrouck, J.-B., der Sluijs, R. V., Polacin, M., Chaves, J. M. Z., Abraham, T. M., Purohit, S., Langlotz, C. P., and Chaudhari, A. Roent- gen: Vision-language foundation model for chest x-ray generation, 2022.
Chambon, P., Delbrouck, J.-B., Sounack, T., Huang, S.- C., Chen, Z., Varma, M., Truong, S. Q., Chuong, C. T., and Langlotz, C. P. Chexpert plus: Augmenting large chest x-ray dataset with text radiology reports, patient demographics and additional image formats, 2024.
References
Adibi, A., Cao, X., Ji, Z., Kaur, J. N., Chen, W., Healey,
E., Nuwagira, B., Ye, W., Woollard, G., Xu, M. A., Cui, H., Xi, J., Chang, T., Bikia, V., Zhang, N., Noori, A., Xia, Y., Hossain, M. B., Frank, H. A., Peluso, A., Pu, Y., Shen, S. Z., Wu, J., Fallahpour, A., Mahbub, S., Duncan, R., Zhang, Y., Cao, Y., Xu, Z., Craig, M., Krishnan, R. G., Beheshti, R., Rehg, J. M., Karim, M. E., Coffee, M., Celi, L. A., Fries, A., Sadatsafavi, M.. Shung, D., Mc McWeeney, S., Daffllo, J., and Jabbour, S. Recent advances, applications and open challenges in machine learning for health: Reflections from research roundtables at ml4h 2024 symposium, 2025.
Chen, Z. Varma, M., Delbrouck, B., Paschali, M., Blanke- meier, I.., Van Veen, D.,, Valanarasu, J. M. J., Youssef, A., Cohen, J. P., Reis, E. P., et al. Chexagent: Towards a foundation model for chest x-ray interpretation. arXiv preprint arXiv:2401. 12208, 2024a.
Pashall
Chen, Z., Varma, M., Xu, J., Paschali, M.. Veen, D. V., Johnston, A., Youssef, A., Blankemeier, L., Bluethgen, C., Altmayer, S., Valanarasu, J. M. J., Muneer, M. S. E., Reis, E. P., Cohen, J. P., Olsen, C., Abraham, T. M., Tsai, E. B., Beaulieu, C. F., Jitsev, J., Gatidis, S., Delbrouck, J.-B., Chaudhari, A. S., and Langlotz, C. P. A vision- language foundation model to enhance efficiency of chest x-ray interpretation, 2024b.
Ahn, J. S., Ebrahimian, S., McDermott, S., Lee, S., Nac- carato, L., Di Capua, J. F., Wu, M. Y., Zhang, E. W., Muse, V., Miller, B., et al. Association of artificial intelligence-aide chest radiograph interpretation with
10

---

McdRAX: Medical Reasoning Agent for Chest X-ray
Cohen, J. P., Hashir, M., Brooks, R., and Bertrand, H. On the limits of cross-domain generalization in automated x- ray prediction. In Medical Imaging with Deep Learning, 2020.
Jiang, Y., Black, K. C., Geng, G., Park, D., Ng. A. Y., and
Chen, J. H. Medagenthench: Dataset for benchmarking Ilms as agents in medical applications, 2025.
Jimenez, E. Yang, J.. Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- els resolve real-world github issues? arXiv preprint arXiv:2310.067: 2023.
Cohen, J. P., Viviano, J. D., Bertin, P., Morrison, P., Tora- bian, P., Guarrera, M., Lungren, M. P., Chaudhari, A., Brooks, R., Hashir, M., and Bertrand, H. TorchXRay Vi- sion: library of chest X-ray datasets and models. In Medical Imaging with Deep Learning, 2022.
Kim, Y., Park, C., Jeong, H., Chan, Y. S., Xu, X., McDuff, D., Lee, H., Ghassemi, M., Breazeal, C., and Park, H. W. Mdagents: An adaptive collaboration of Ilms for medical gecision-mmking In The Thirty-eighth Annual Confer- ence Neural Information Processing Systems, 2024.
Erdal, B. S., Gupta, V., Demirer, M., Fair, K. H., White, R. D., Blair, J., Deichert, B., Lafleur, L., Qin, M. M., Bericat, D., and Genereaux, B. Integration and imple- mentation strategies for ai algorithm deployment with smart routing rules and workflow management, 2023.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L.,, Xiao, T., Whitehead, S., Berg, A. A.C Lo, W.-Y., Dollár, P., and Girshick, R. Segment anything, 2023.
Eriksen, A. V., Möller, S., and Ryg. J. Use of gpt-4 to diagnose complex clinical cases, 2024.
Fallahpour, A., Alinoori, M., Ye, W., Cao, X., Afkanpour, A., and Krishnan, A. Ehrmamba: Towards generaliz- able and scalable foundation models for electronic health records, 2024.
Li, B., Yan, T., Pan, Y., Luo. J., Ji, R., Ding. J., Xu, Z., Liu, S., Dong, H., Lin, Z., et al. Mmedagent: Learning to use medical tools with multi-modal agent. arXiv preprint Sttv:2407.0248.. 2024a.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., et al. Deepseck-rl: In- centivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang. J., Naumann, T., Poon, H., and Gao, J. Llava-med Training a large language- and-visoon assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024b.
Huang, J., Neill, L., Wittbrodt, M., Melnick, D., Klug, M., Thompson, M., Bailitz, J., Loftus, T., Malik, S., Phull, A., al. Generative artificial intelligence for chest radio- graph interpretation in the emergency department. JAMA network open, 6(10):22336100-22333000 2023.
Lian, J., Liu, J., Zhang, S., Gao, K., Liu, X., Zhang, D., and Yu, Y. A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentation. IEEE Transactions on Medical Imaging, 2021. doi: 10.48550/ arxiv.2104.1033 10326.
Hyland, S. L., Bannur, S., Bouzid, K., Castro, D. C., Ranjit, M., Schwaighofer, A., Pérez-García, F., Salvatelli, V., Srivastav, S., Thieme, A., Codella, N., Lungren, M. P., Wetscherek, M. T., Oktay, O., and Alvarez- Valle, J. Maira- 1: specialised large multimodal model for radiology report generation, 2024.
Liu, B., Zhan, L.-M., Xu, L., Ma, L., Yang, Y., and Wu, X.- M. Slake: A semantically- labelee knowledge-enhanced dataset for medical visual question answering, 2021.
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. Agentbench: Evalu- ating llms as agents. arXiv preprint arXiv:2308.088 033688,8... 2023.
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-lleus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Sh- panskaya, K., Seekins, J., Mong, D. A., Halabi, S. S., Sandberg, J. K., Jones, R., Larson, D. B., Langlotz, C. P., Patel, B. N., Lungren, M. P., and Ng, A. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison, 2019.
Ma, J., He, Y., Li, F., Han, L., You, C., and Wang, B. Seg- ment anything in medical images. Nature Communi- cations, 15(1), January 2024. ISSN 2041-1723. doi: 10.1038/s41466 467-024-44824--
Jaech, A., Kalai, A., Lever, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai ol system card. arXiv preprint arXiv:2412. 16720, 2024.
Ma, J., Yang. 7., Kim, S., Chen, B., Baharoon, M., Fallah- pour, A., Asakereh, R., Lyu, H., and Wang, B. Medsam2: Segment anything in 3d medical images and videos, 2025.
Masterman, T., Besen, S., Sawtell, M., and Chao, A. The landscape of emerging ai agent architectures for reason- ing. planning, and tool calling: A survey. arXiv preprint arXiv:2404. / 2024.
Javan, R., Kim, T., and Mostaghni, N. Gpt-4 vision: Multi- modal evolution of chatgpt and potential role in radiology. Cureus, 1668)8,6829, 2024.

---

McdRAX: Medical Reasoning Agent for Chest X-ray
Nori, H., King, N., McKinney, S. M., Carignan, D., and Horvitz, E. Capabillities of gpt-4 medical challenge problems. arXiv preprint arXiv: 2023.
Wu, C., Zhang, X., Zhang, Y., Wang, Y., and Xie, W. To- wards generalist foundation model for radiology by lever- aging web-scale 2d and 3d medical data, 2023.
Ouis, M. Y. and Akhloufi, M. A. Chestbiox-gen: contextual biomedical report generation from chest X-ray images using biogpt and co-attention mechanism. Frontiers in Imaging, 3:1373420, 2024.
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2): 2025,
Park, J., Kim, S., Yoon, B., Hyun, J., and Choi, K. M4cxr: Exploring mulli-task potentials of multi- modal large lan- guage models for chest x-ray interpretation, 2024.
Yan, Z., Zhang, K., Zhou, R., He, I.., Li, X., and Sun, Multimodal chatgpt for medical applications: an experi- mental study of gpt-4v, arXiv preprini arXiv:2310. 11062210.19061. 2023.
Pellegrini, C., Keicher, M., Özsoy, E., and Navab, N. Rad- restruct: A novel vqa benchmark and method for struc- tured radiology reporting, 2023.
Yang, H. M., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., al. Chexnet: radiologist-level pneumo- nia detection chest x-rays with deep learning. arXiv preprint
Pellegrini, C., Özsoy, E., Busam, B., Navab, N., and Ke- icher, M. Radialog: A large vision-language model for radiology report generation and conversational assistance, 2025.
2017.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models, 2023.
Pham, H. H., Nguyen, H. Q.. Nguyen, H. T., Le, L. T., and Khanh, L. An accurate and explainable deep learning system improves interobserver agreement in the inter- pretation of chest radiograph. IEEE Access, 10:104512- 104531, 2022.
Yin, G., Bai, H., Ma, S., Nan, F., Sun, Y., Xu, Z., Ma, S., Lu, J., Kong, X., Zhang, A., et al. Mmau: holistic benchmark of agent capabilities across diverse domains. arXiv preprint 1xXx::2407.11960 2024.
Schmidgall, S., Ziaei, R., Harris, C., Reis, E., Jopling, J., and Moor, M. Agentclinic: a multimodal agent bench- mark evaluate ai in simulated clinical environments, 2024.
Zambrano Chaves, J., Huang. S.-C., Xu, Y., Xu, H., Usuyama, N., and Zhang, S, e. a. Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation. arXiv preprint M0Xiv:240300000 2024.
Shin, H. J., Han, K., Ryu, L., and Kim, E.-K. The impact of artificial intelligence the reading times of radiologists for chest radiographs. NPJ Digital Medicine, 6(1) 6(1):82, 2023.
Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. Pyramid scene parsing network, 2017.
Tanno, R., Barrett, D. G., Sellergren, A., Ghaisas, S., Dathathri, S., See, A., Welbl, J., Lau, C., Tu, T., Azizi, S., et al. Collaboration between clinicians and vision- language models in radiology report generation. Nature Medicine, pp. 1- 2024.
Zhao, P., Jin, Z., and Cheng, N. An in-depth survey of large language model-based artificial intelligence agents. arXiv preprint arXiv:2309. 2023.
Tu, T., Azizi, S., Driess, D., Schackermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., Mustafa, B., Chowdhery, A., Liu, Y., Kornblith, S., Fleet, D., Mansfield, P., Prakash, S., Wong, R., Virmani, S., Semturs, C., Mahdavi, S. S., Green, B., Dominowska, E., y Arcas, B. A., Barral, J., Webster, D., Corrado, G. S., Matias, Y. Singhal, K., Florence, P., Karthikesalingam, A. and Natarajan, V. Towards generalist biomedical ai, 2023.
United Nations Scientific Committee on the Effects of Atomic Radiation. Sources, Effects and Risks of lon- izing Radiation: UNSCEAR 2020/2021 Report, Volume United Nations, New York, 2022. ISBN 978-92-1- 139206-7.
12

---

MedRAX: Medical Reasoning Agent for Chest X-ray
A. MedRAX Core Methodology
This appendix provides detailed exposition Algorithm 1, which forms the core operational framework of MedRAX. The algorithm implements a ReAct (Reasoning and Acting) loop that enables dynamic tool orchestration for complex CXR interpretation tasks through iterative cycles of reasoning, acting, and observing.
A.1. The ReAct Cycle: Foundational Concept
MedRAX operates on a cyclical Reason-Act (ReAct) principle, distinguishing from traditional one-shot input-output models. The agent iterates through three core phases:
Reason: Analyze the current situation (user query, conversation history, previous results) and decide on the next step
Act: necessary, perform an action, typically using specialized tool to gather more information or perform a specific task
Observe:
Incorporate the results of the action back into its understanding of the situation and repeat the cycle
This iterative process enables the agent to tackle complex problems requiring multiple steps or external information sources, such as analyzing CXRs using various diagnostic tools sequentially or in parallel.
A.2. Essential Components and Architecture
MedRAX relies on four interconnected components to execute the ReAct cycle:
1. CORE REASONING ENGINE (LLM)
powerful multimodal Large Language Model (e.g., GPT-4o) capable of understanding text, images, and erucially, using tools. The LLM performs the "Reason" step by analyzing the situation and deciding whether to answer directly or request tool execution. The chosen LLM must demonstrate proven multimodal understanding and reliable tool-calling adherence.
MedRAX employs the following system prompt to guide the reasoning engine:
You are expert medical Al assistant who can answer any medical questions and analyze medical images similar to doctor. Solve using your own vision and reasoning and use tools to complement your reasoning. Make multiple tool calls in parallel or sequence as needed for comprehensive answers. Critically think about and criticize the tool outputs. If you need to look up some information before asking a follow up question, you are allowed to do that.
A.2.2. SPECIALIZED TOOLBOX
collection pre-trained Al models for specific CXR tasks including classification, segmentation, VQA, grounding, and report generation. The framework encapsulates each specialized model in software interface that enables LLM interaction through standardized tool wrappers.
Tool Interface (Wrappers): Each tool wrapper defines four critical components:
name: A unique identifier (e.g.. "cxr "lassssaas)f)er")
description: natural language explanation detailing what the tool does, its inputs, and outputs (e.g.. "Takes a CXR image reference and returns likely pathologies and their probabilities).. This description acts implicit prompting, guiding the LLM on when to use the tool
schemas structured definition of required inputs (e.g., image identifier)
execution logic: Code to run the tool with given inputs and return results
The description field within each wrapper is paramount- it serves as the primary mechanism for the LLM to understand tool capabilities and appropriate usage contexts. Detailed tool descriptions used by MedRAX to understand utility and calling procedures are available in our public GitHub repository.
13

---

MedRAX: Medical Reasoning Agent for Chest X-ray
A.2.3. WORKFLOW ORCHESTRATOR
Manages the overall ReAct cycle, directing information flow between the LLM, Toolbox, and Memory. Key functions include:
Calling the LLM for the "Reason" step
Parsing LLM responses to check for tool requests
Calling appropriate tools from the Toolbox for the "Act" step
Updating Memory with LLM responses and tool results
Deciding when the cycle complete
A.2.4. AGENT MEMORY (STATE)
Stores the complete interaction history as sequence of structured messages including user input, LLM thoughts/responnss, and tool results. The memory provides necessary context for LLM reasoning at each step, ensuring the agent "remembers" previous findings and interactions throughout multi-turn conversations.
A.3. Detailed Algorithm Execution Flow
A.3.1. STEP O: INITIALIZATION AND STATE PREPARATION
The Observe (Q, I, M) function consolidates three critical information sources: the user query (Q), input CXR images (7), and the agent's memory buffer (M). The predefined system prompt loaded to establish agent's role and reasoning approach.
Image data is represented within Memory/Messages using URIs, IDs, embedded data like base64. The chosen LLM must process these image representations alongside text, while tool wrappers operating on images must access image data based on references passed in their arguments.
A.3.2. STEP LLM-DRIVER REASONING PROCESS
Prompt Construction: The Orchestrator prepares comprehensive input for the LLM including:
The system prompt defining the agent's role and approach
The entire message history from Memory, formatted chronologically
Tool definitions: name, description, and input for all available tools, formatted according to the LLM provider's API requirements
The Reason (state, M) function generates structured thoughts through comprehensive analysis. The LLM's decision to respond directly or use tools stems from training on vast datasets including instruction following and tool-use examples, combined with specific context. The LLM analyzes:
LLM Decision Process:
Whether current message history and internal knowledge sufficiently answer the query according to system prompt instructions
Available tools (via descriptions) to determine if any tool can provide more accurate or efficient information than internal knowledge (e.g., precise classification probabilities, segmenting specific regions)
Whether to ask clarifying questions if the query is ambiguous or lacks information
Response Generation: Based on its decision, the LLM can:
Answer/Clarify Directly: Generate only text response (final answer or clarifying question back user)
Use Tools: Generate structured Tool Call Request(s), potentially requesting multiple different tools or the same tool multiple times with different arguments for parallel information gathering
14

---

MedRAX: Medical Reasoning Agent for Chest X-ray
A.3.3. STEP 2: CONDITIONAL DECISION MAKING
The Orchestrator examines the LLM's response to determine the next action:
User Input Requirements: The RequiresUserInptt (thoughts) condition is implicitly evaluated during soning. When the LLM's thoughts indicate ambiguity or insufficient information that cannot be resolved by tools, GenerateUserPromgg (thoughts, M) formulates a natural language question based on its thoughts and context in memory to elicit needed information.
Tool Execution Path: If Tool Call Requests are present, the system proceeds to tool execution. If abseen, the system routes to response generation where the LLM synthesizes the final answer.
A.3.4. STEP 3: DYNAMIC TooL SELECTION AND EXECUTION
Tool Processing: The SolectTools (thoughts, т, M) and ExecuteParal lel (tools, state) functions handle each Tool Call Request by processing them potentially concurrently if designed for parallelism:
Identify Tool: Look up requested tool name in Toolbox to find corresponding Tool Wrapper object/class
Validate & Extract Arguments: Parse arguments dictionary provided by LLM for that specific call, checking arguments match the tool's defined input_schema. If validation fails, an error generated
Invoke Tool: arguments are valid, call the specific execution funetion/method within the Tool Wrapper, which contains
actual code to interact with the specialized model (load model, preprocess input, run inference, postprocess output)
Receive Result: Capture return value from tool's 's execution logic (typically string or structured JSON data representing
findings or error message if execution failed)
Format Result: Package result into structured Tool Result Message including result content and unique ID corresponding to the Tool Call Request
The system supports parallel execution of independent tools while providing flexible deployment configurations- can be quantized for efficiency and distributed across CPU or GPU resources.
A.3.5. STEP 4: MEMORY MANAGEMENT AND LooP CONTROL
Memory Update: The operation M = M U { (thougghs, tools, results)) maintains comprehensive records of the agent's reasoning process. This critical step logs the LLM's thoughts that led to tool calls, the identity of tools used, and results obtained. This persistent memory serves multiple functions:
Provides context for subsequent reasoning cycles
Enables reference to previous findings when synthesizing complex analyses
Supports multi-turn conversations by preserving interaction history
Caches tool outputs to prevent redundant computations in multi-step analyses that might reference the same intermediate results
Loop Control: If coming from tool execution, the system loops back to Step (Reasoning) where the LLM receives updated Memory including Tool Result Messages. The LLM can now observe and rethink, processing tool outputs by:
Synthesizing tool results with previous context
Correcting understanding tools returned unexpected information
Deciding if more reasoning or different tool calls are needed
Formulating final answers based on gathered evidence
When SunCenerateRespone (thoughts) evaluates to true, the LLM synthesizes the final response drawing upon concluding thoughts and information in memory.
15

---

MedRAX: Medical Reasoning Agent for Chest X-ray
A.4. Error Handling and Robustness Mechanisms
MedRAX incorporates several critical mechanisms ensure robust operation:
Timeout Management: The Emax parameter enforces maximum execution time, preventing indefinite loops while allowing sufficient time for complex multi-step analyses via CeneeaatTimmoon Response (state, M)
Tool Failure Recovery: When tool execution fails, error messages are formatted as Tool Result Messages clearly indicating the error. When the flow loops back to reasoning, the LLM sees this error message and can decide how to proceed based on its instructions (e.g., try different tool, ask user for clarification, inform user of limitations).
A.5. Implementation Architecture
Node and Edge Structure: The workflow orchestrator is implemented using libraries like LangGraph that provide structures for state machine implementation:
Nodes (Processing Steps): Python functions class methods representing processing steps (ReasoningStep, ActionStep). Each function accepts current agent Memory/State object input, performs specific logic, and returns dictionary containing updates to Memory/State
Standard Edges: Unconditional transitions configured by stating flow between nodes
Conditional Edges: Decision points implemented with routing functions that take current Memory/State as input, apply logic, and return string indicating next node name to execute
This comprehensive methodology establishes MedRAX as a flexible, robust, and transparent framework for Al-assisted CXR interpretation, capable of dynamic adaptation to diverse clinical scenarios while maintaining clear decision traces throughout the reasoning process.
B. ChestAgentBench: Comprehensive Benchmark Statistics
B.1. Clinical Setting Distribution
ChestAgentBench's 675 cases represent diverse clinical environments:
Table 5. Complete Distribution various Chest X-ray Findings in ChestAgentBench (Frequency % across 675 clinical cases)
Emergency Room (ER): 19.7% (133 cases)
Intensive Care Unit (ICU): 4.9% (33 cases)
Finding
(%)
Finding
(%)
Other hospital settings: 75.4% (509 cases)
Mass
26.3
Interstitial findings
6.9
Effusion
24.6
Bronchiectasis
5.9
Pleural Effusion
21.5
Atelectasis
4.9
B.2. Pathology Distribution
Consolidation
21.3
Fibrosis
4.1
The benchmark encompasses 20 distinct pathological find- ings with frequencies ranging from 26.3% (Mass) to 1.5% (Cardiomegaly), as detailed in Table This distribution reflects real-world clinical prevalence patterns, with struc- tural abnormalities (masses, nodules) and fluid collections (effusions) representing the most common findings, while cardiac and infectious conditions appear less frequently.
Nodule
17.2
Edema
3.9
Calcification
10.7
Cavitation
3.9
Pneumothorax
7.6
Fracture
3.0
Lymphadenopathy
7.6
Tuberculosis
2.6
Pneumonia
7.2
Metastasis
2.6
Emphysema
7.1
Cardiomegaly
1.5
16


Reviews:
[{"review_id": "WEy6urrVVk", "reviewer": "Reviewer_KTkS", "summary": "The key contributions of this work are:\n- MedRAX: An AI framework integrating multiple chest X-ray (CXR) analysis tools without extra training, dynamically orchestrating components for complex medical queries.\n- ChestAgentBench: An evaluation framework featuring 2,500 queries across seven categories, built from 675 expert-curated clinical cases, to assess multi-step reasoning in CXR interpretation.\n- Performance: MedRAX significantly surpasses general-purpose and biomedical-specific models in complex reasoning tasks, offering transparent workflows.\n- Interface: A user-friendly interface enabling flexible deployment from local to cloud solutions, addressing healthcare privacy needs.\n\nThis paper emphasizes that structured orchestration of medical AI tools, combining large-scale reasoning and domain-specific knowledge, outperforms purely end-to-end models.\n\n\n**update after rebuttal**\n\nI’m satisfied with the authors’ response to provide more details about the algorithm and dataset. In my initial review, I had some concerns about the fit of this paper for ICML, as I felt the topic might not attract broad interest. However, after reading the other reviews, I’ve reconsidered my stance. Therefore, I am increasing my score from 2 to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback and appreciate their recognition of the importance of this problem in the healthcare domain.\n\n**Algorithm 1 Core Methodology.** Algorithm 1 outlines the iterative reasoning process of the MedRAX agent in the ReAct loop, as follows:\n\n1.  **`Observe(Q, I, M)` / State Preparation:** This step initializes the reasoning cycle by gathering the necessary context. It aggregates the user's query (`Q`), any associated input images (`I`), and the entire history maintained in the agent's memory (`M`), which includes previous interactions, tool outputs, and reasoning steps. This consolidated state is fed into the LLM.\n\n2.  **`Reason(state, M)` -> `thoughts` (LLM Reasoning):** The LLM analyzes the current state (query, images, memory) and available tools (`T`) to generate internal 'thoughts'. These thoughts form a plan deciding the next action: generate a final response, ask the user for input, or select one or more tools.\n\n3.  **`RequiresUserInput(thoughts)` (Implicit Decision):** This condition is implicitly evaluated by the LLM during the `Reason` step. If the LLM's 'thoughts' indicate ambiguity or insufficient information that cannot be resolved by tools, it will opt to generate a clarifying question for the user instead of proceeding with a tool call or final answer.\n\n4.  **`GenerateUserPrompt(thoughts, M)` (Implicit Action):** If `RequiresUserInput` is effectively true, the LLM's generated output *is* the prompt for the user. It formulates a natural language question based on its 'thoughts' and the context in memory (`M`) to elicit the needed information.\n\n5.  **`SelectTool(thoughts, T, M)` -> `tool(s)` (LLM Tool Selection):** When the LLM's 'thoughts' indicate a need for specific capabilities (e.g., classification, segmentation), it selects the most appropriate tool or *multiple tools* from the available set (`T`). This selection is based on the LLM matching its reasoning to the predefined descriptions and capabilities of each tool. For each selected tool, the LLM also formulates the necessary input arguments in a structured format (e.g., JSON).\n\n6.  **`Execute(tool(s))` -> `result(s)` (Tool Execution):** The agent executes the selected tool function(s). This involves invoking each tool with the structured arguments prepared by the LLM in the `SelectTool` step. The output(s) of these execution(s) are captured as `result(s)`.\n\n7.  **`M ← M ∪ (thoughts, tool(s), result(s))` (Memory Update):** Following successful tool execution(s), the agent's memory (`M`) is updated. This crucial step logs the LLM's 'thoughts' that led to the tool call(s), the identity of the `tool(s)` used, and the `result(s)` obtained. This ensures that outputs from tools become part of the context for subsequent reasoning cycles.\n\n8.  **`CanGenerateResponse(thoughts)` / `GenerateResponse(thoughts, M)` (Response Generation):** Following `Reason`, this checks if the LLM's 'thoughts' plan a tool call. If not (`CanGenerateResponse` effectively true), the LLM synthesizes the final natural language answer (`GenerateResponse`), drawing upon its concluding 'thoughts' and information in memory (`M`). If 'thoughts' indicate a tool is needed, this step is bypassed, and the agent proceeds to `SelectTool` and `Execute`.\n\n**ChestAgentBench Statistics.** Thank you for the thoughtful comment. The distribution of common findings and department origins of cases are provided in response to `Reviewer DGwp`. We have revised Figure 3d to incorporate these detailed statistics.\n\n**ICML Relevance.** We submitted MedRAX under the \"Application-Driven Machine Learning\" track highlighting areas like healthcare. Our work directly addresses significant challenges within this critical domain, and we chose ICML because we believe high-quality, impactful healthcare AI research, while perhaps historically under-represented, is important for the ML community.\n\n**CheXbench Image-Text Reasoning.** This benchmark assesses fine-grained visual reasoning, requiring models to differentiate between options with subtle but critical distinctions (e.g., 'left' vs. 'right' pleural effusion). We observed poor performance across all evaluated models on this task. This suggests the challenge lies in the inherent difficulty of fine-grained radiological interpretation rather than the benchmark dataset itself (derived from the widely-used OpenI). \n\n**Clinical Relevance.** Thank you for raising this important question. MedRAX integrates into clinical workflows as an interactive radiologist copilot. Clinicians can ask questions about CXRs (e.g., for disease classification, localization, VQA) via the user-friendly interface to assist in diagnosis. Its flexible deployment options (local or cloud) and modular design address practical IT and privacy barriers to adoption.\n\n**Figure 2.** We have replaced Figure 2 with a clearer case showing distinct unilateral findings.", "claims_and_evidence": "This work proposes an agent-based system for chest X-ray interpretation, integrating domain-specific models as tools with general-purpose LLMs for reasoning. Unlike purely end-to-end approaches, this method leverages existing specialized tools, albeit at the cost of increased inference time. The paper persuasively argues that this combined strategy outperforms E2E models, as structured reasoning tailored to the relatively closed-domain nature of chest X-ray interpretation effectively introduces beneficial inductive biases. Experimental results support this claim.", "methods_and_evaluation_criteria": "This work focuses on integrating several existing tools; however, it lacks technical details on the implementation of core modules in Algorithm 1, such as Reason, RequiresUserInput, and SelectTool, among others. Providing a more in-depth explanation of these components would strengthen the paper by clarifying how these modules function and contribute to the overall system.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "To compare existing works, the authors have utilized two established benchmarks and introduced ChestAgentBench. The design of ChestAgentBench is generally logical and reasonable, and the statistical overview of the benchmark dataset is appropriate. However, providing a more detailed distribution of findings across different body regions would enhance the dataset's transparency.\n\nFor example, it would be beneficial to present the frequency of the most common chest X-ray findings, such as pleural effusion, cardiomegaly, pulmonary nodules/masses, and others, within the dataset. This would offer a clearer understanding of its composition.\n\nAdditionally, the category levels in Figure 4d should be standardized, as the current categorization appears somewhat inconsistent. A more informative approach would be to stratify the dataset as follows:\n- Statistics by department: ER, ICU, and general ward\n- Findings in chest X-ray: Pleural effusion, cardiomegaly, tuberculosis, nodule/mass, pneumothorax, consolidation, etc.\n\nThis refinement would provide a more structured and comprehensive breakdown, improving the interpretability of the dataset.", "relation_to_broader_scientific_literature": "This work is highly relevant to foundation models in the medical domain, as it explores an alternative approach to integrating sophisticated existing models with reasoning modules, rather than relying solely on an end-to-end foundation model.", "essential_references_not_discussed": "I think the Related Work section does a great job of discussing the limitations of previous studies and clearly highlighting how this work differs from them.", "other_strengths_and_weaknesses": "The presentation of this work is good making it easy to follow, and I enjoyed reading the manuscript. This paper serves as a strong application of machine learning in healthcare, making it a good fit for the Machine Learning for Healthcare track (or similar tracks, if exists). However, for the regular track at ICML, it is unclear whether it would attract significant interest from the broader audience. Since it somewhat lacks novel or particularly compelling machine learning techniques or methodologies that would appeal to the ML research community, the paper's positioning may not align with ICML’s focus.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "#1. Could you elaborate on why all methods, including MedRAX, perform poorly on image-text reasoning questions? Is it due to the inherent difficulty of this specific task, or because the image datasets originate from external institutions not included during training—essentially representing an external validation scenario common in clinical studies?\n\n#2. The authors argue that MedRAX could be integrated into existing clinical workflows. However, in my understanding, the practical utility and deployment potential of the proposed agent remain unclear. Could you specify how MedRAX could be integrated in daily clinical practice? \n\n#3. Given the low resolution of Figure 2, it's difficult to be certain, but there appears to be an abnormal pattern in both the left and right lungs, possibly indicating bilateral effusions. Therefore, it may be preferable to select a different image for visualization.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vzVkp7DLAv", "reviewer": "Reviewer_b5BD", "summary": "This paper proposes MedRAX, a modular AI agent that integrates specialized chest X-ray (CXR) analysis tools with large language models to perform complex, multi-step medical reasoning. It introduces ChestAgentBench, a large benchmark of 2,500 expert-curated CXR reasoning tasks, to evaluate its performance. Experiments show MedRAX outperforms both general-purpose and specialized models in CXR interpretation, offering improved accuracy and transparency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their thorough and thoughtful comments. \n\n**Statistical Significance.** We appreciate the reviewer's point about statistical measures. All experiments were run deterministically (LLMs with temperature 0 and deterministic tools), thus eliminating variability in model outputs for identical inputs. Additionally, since MedRAX integrates pre-trained models without any further training or fine-tuning, there is no randomness from training procedures. This approach ensures full reproducibility and makes traditional confidence intervals less applicable in this context.\n\n**Failure Cases.** For MedRAX failure cases, we have observed instances where the LLM becomes overconfident in its own reasoning and neglects to effectively utilize available tools, leading to incorrect conclusions that could have been corrected with proper tool integration. Regarding baseline failures, models like LLaVA-MED often struggle due to limited training data, resulting in poor generalization to diverse clinical scenarios. MedRAX mitigates this limitation by combining specialized tools with a general-purpose LLM that has broader knowledge and reasoning capabilities. We have included specific examples demonstrating these failure patterns in the revised manuscript.\n\n**Experiment Probing Spurious Correlations.** We appreciate the reviewer's thoughtful suggestion on investigating spurious correlations within the pretrained models used by MedRAX. The concern about biased predictions is addressed through diverse tools that provide cross-validation capabilities. Specifically, MedRAX integrates grounding models that highlight disease regions, allowing the LLM to validate classification predictions against visual evidence from multiple sources. This multi-tool approach helps mitigate reliance on any single potentially biased component. We agree that the reviewer's suggested approach represents a valuable direction for our future work.\n\n**Related Work.** We have added \"RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance\" to the related work section.\n\n**Strengths and Weaknesses.**\n\n1. Thank you for raising this important question. MedRAX integrates into clinical workflows as an interactive radiologist copilot. Clinicians can ask questions about CXRs (e.g., for disease classification, localization, VQA) via the user-friendly interface to assist in diagnosis. Its flexible deployment options (local or cloud) and modular design address practical IT and privacy barriers to adoption.\n\n2. Discussed above.\n\n3. Table 1 serves as a partial ablation, demonstrating the performance differential between standalone VQA tools (e.g., CheXagent), general-purpose models (GPT-4o), and our integrated MedRAX framework. However, we acknowledge that a more granular ablation study selectively disabling specific tools would provide deeper insights into which tools are most critical for different categories. We will include this analysis in the revised manuscript.\n\n4. The final response is processed through GPT-4o. Our framework is designed to minimize hallucinations by providing comprehensive context from multiple specialized tools, creating an information-rich environment for the LLM. This multi-tool approach helps ground the model's responses in concrete findings from validated medical AI systems rather than relying solely on the LLM's internal knowledge.\n\n5. MedRAX utilizes LangChain to manage the reasoning process dynamically. The Reason(state,M) function relies on: (1) an initial system prompt defining the agent's role, available tools with descriptions, and required output structure, and (2) the continuously updated conversation history (memory M) containing past thoughts, actions, and tool results. The agent framework routes execution based on the LLM's output: structured tool calls trigger tool execution, while a final response concludes the process. We have included the agent system prompt and tool descriptions in the revised manuscript.\n\n6.  MedRAX is flexible: while our evaluation used GPT-4o vision for performance, it can operate without vision LLMs by relying on specialized visual tools. The benchmark evaluation cost using GPT-4o vision involved 2,500 questions with on average 1.85 images per question (~512x512px, ~255 tokens per image) at GPT-4o's input rate of $3.75/1M tokens.\n\n7. A breakdown of core methodology is provided in response to `Reviewer KTkS`\n\n8.  Regarding patient data privacy, MedRAX's modular architecture supports locally deployed LLMs to prevent sensitive medical data transmission to external servers. Acknowledging reviewer guidelines, it also supports Azure OpenAI service with appropriate opt-out configurations. This flexibility allows institutions to implement MedRAX compliantly via on-premises deployment or properly configured cloud services, meeting their specific privacy requirements.", "claims_and_evidence": "Overall, the technical claims are plausible and generally supported by the experiments. However, i would like to see the following:\n1. Statistical significance or confidence intervals around the reported accuracy differences.\n2. The failure cases of MedRAX (cases where it fails and the potential reason). Also, for the failure cases of the baselines like LLaVA-MED, is MedRAX provide correct predictions. The authors should discuss why MedRAX is correct in those senses.", "methods_and_evaluation_criteria": "1. The authors’ chosen method—combining a large language model (LLM) “agent” (GPT‑4⁰ in their reference implementation) with specialized CXR analysis tools in a ReAct loop—makes sense for the complex, multi-step nature of real radiological queries. The evaluation criteria are primarily classification accuracy (six-choice questions in ChestAgentBench, plus standard VQA accuracy on CheXbench), which is a straightforward metric for correctness.\n\n2. While the choice of multi-choice questions (instead of free-form answers) is somewhat simplifying, it does allow for a consistent, reproducible metric. It would be helpful if the paper described the approach for verifying the “best” single correct answer in each question—especially when complex real-world findings can have nuanced interpretations. In general, the methods and metrics are appropriate and sensible for the problem.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "I would also encourage an experiment explicitly probing spurious correlations within the pretrained models that MedRAX uses. For instance, it is well documented that pneumothorax classifiers may erroneously learn to rely on chest tubes as a proxy signal, leading to biased predictions. A practical approach would be:\n\n1. Identify Influential Concepts. Use a post-hoc interpretability technique—either a concept-bottleneck model or saliency/heatmap analysis—to pinpoint the concepts or image regions that most strongly influence the model’s predictions.\n\n2. Provide These Influential Concepts to GPT‑4o. Along with the model’s prediction, feed the extracted concepts or saliency annotations into GPT‑4⁰ to ask whether those features are causally relevant or potentially spurious.\n\n3. Assess Biases and Explore Corrective Measures. If GPT‑4o flags a concept (e.g., a chest tube) as non-causal or suspicious, that insight can guide further investigation, such as retraining or debiasing strategies to mitigate reliance on that feature.\n\nSuch an experiment could not only uncover hidden biases in the classification tools but also enhance the interpretability of MedRAX’s decision-making process, strengthening confidence in its clinical applicability.", "relation_to_broader_scientific_literature": "The authors situate their work in the broader context of:\n- LLM-based agent architectures (ReAct, tool orchestration, etc.).\n- Medical VQA and radiology-centric models (CheXagent, LLaVA‑Med, MMedAgent, etc.).\n- General multimodal LLM frameworks (GPT‑4⁰ with vision, Segment Anything approaches for image segmentation).\n\nThey do a good job contrasting MedRAX with prior domain-specific solutions like CheXagent (which is specialized but not agent-based) and broad frameworks like GPT‑4o (strong reasoning but lacking targeted medical tools). I would also rather add \"RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance\" to the related work.", "essential_references_not_discussed": "See Relation To Broader Scientific Literature", "other_strengths_and_weaknesses": "1. Lack of prospective or real-world clinical validation beyond curated datasets.\n2. No statistical significance or error bars reported.\n3. Ablation studies would be helpful: for instance, measuring how MedRAX performs if certain tools are disabled (segmentation vs. classification vs. report-generation, etc.).\n4. Does the final response also goes through the LLM (GPT 4o). If so, then there is a potential chance of halucination. Does the author have any idea to reduce it.\n5. Certain details is not clear. For ex, i believe the interaction with the LLM needs sepcific prompt to generate thought (Reason(state,M) function in Algo 1) and action. However no mention of them in the paper \n6. Does the author provide the CXRs to LLM. Then this method will be expensive. So, i would like to see the cost breakdown of the method consuming the LLMs.\n7. Is algorithm 1 is an API? then detailed description needed for the API.\n8. Sending CXR and other details, like report to GPT4o can be problematic as they can use these data for training and storing them in their server. The authors shall use these guidelines (https://physionet.org/news/post/gpt-responsible-use) to make sure that these patient data wont be shared to commercial LLMs like GPT4o. So details is necessary on how they send the data through LLM.", "other_comments_or_suggestions": "Please answer my points in Other Strengths And Weaknesses and i will raise the score.", "questions_for_authors": "See Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Privacy and Security"], "ethical_review_concerns": "See #8 in Other Strengths And Weaknesses"}}, {"review_id": "1FPW4GRvgS", "reviewer": "Reviewer_DGwp", "summary": "The paper introduces MedRAX, an AI-driven agent for the interpretation of chest X-rays (CXRs). MedRAX integrates various specialized state-of-the-art chest X-ray analysis tools and multimodal large language models into a single unified framework. Unlike existing solutions that often operate independently, MedRAX dynamically utilizes these specialized components to handle complex medical queries without additional training.\n\nThe authors propose a novel evaluation framework, ChestAgentBench, featuring 2,500 expert-validated questions across seven essential CXR interpretation categories. In comparison experiments, MedRAX significantly outperformed other general-purpose and specialized biomedical models across all assessed tasks, including detection, classification, localization, comparison, relationship understanding, characterization, and diagnosis.\n\nOverall, the study demonstrates that integrating structured reasoning with multimodal specialized tools enhances both accuracy and interpretability in medical imaging tasks, presenting MedRAX as a practical step towards clinical deployment of automated CXR interpretation systems.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the insightful feedback.\n\n**ChestAgentBench Representativeness.**\nWe appreciate the reviewer's question on benchmark representativeness. We designed ChestAgentBench to be both representative and capable of assessing advanced reasoning, addressing limitations in prior benchmarks:\n* **Addresses Evaluation Gaps in Complex Reasoning:** Existing benchmarks often focus on simpler, single-step VQA or isolated tasks, which are insufficient for evaluating sophisticated AI agents designed for clinical practice. These simpler tasks do not capture the multi-step diagnostic reasoning, evidence integration, and tool use inherent in real-world radiology workflows. ChestAgentBench was explicitly created to fill this critical evaluation gap by presenting complex challenges that require agents to demonstrate deeper, sequential reasoning and integrated analytical capabilities, truly assessing their readiness for clinical application support.\n* **Ensures Broad Clinical Scope and Realistic Data Distribution:** The benchmark's representativeness is supported by its diverse foundation and statistically verified distributions. Derived from 675 real clinical cases, it includes scenarios from various hospital settings and covers a wide spectrum of common and important chest X-ray findings.\n\n**Distribution by Department:**\n* Emergency Room (ER): 19.7%\n* Intensive Care Unit (ICU): 4.9%\n*(Note: The remaining cases originate from other hospital settings, including general wards.)*\n\n**Frequency of Common Chest X-ray Findings:**\n* Mass: 26.3%\n* Effusion: 24.6%\n* Pleural Effusion: 21.5%\n* Consolidation: 21.3%\n* Nodule: 17.2%\n* Calcification: 10.7%\n* Pneumothorax: 7.6%\n* Lymphadenopathy: 7.6%\n* Pneumonia: 7.2%\n* Emphysema: 7.1%\n* Interstitial findings: 6.9%\n* Bronchiectasis: 5.9%\n* Atelectasis: 4.9%\n* Fibrosis: 4.1%\n* Edema: 3.9%\n* Cavitation: 3.9%\n* Fracture: 3.0%\n* Tuberculosis: 2.6%\n* Metastasis: 2.6%\n* Cardiomegaly: 1.5%\n\nThis broad distribution across 53 anatomical areas, varied patient demographics, and numerous pathologies ensures agents are tested on a realistic range of clinical challenges.\n\n* **Grounded in Expert Cases and Assesses Integrated Clinical Competencies:** The benchmark's questions are directly derived from and verifiable against detailed findings and expert discussions within 675 authentic, expert-curated clinical cases, ensuring clinical validity and grounding in real-world knowledge. Furthermore, ChestAgentBench systematically evaluates the integration of seven core clinical competencies (such as detection, localization, comparison, relationship analysis, and diagnosis) through complex question types. This design forces agents to demonstrate multifaceted reasoning akin to a clinician synthesizing diverse information, rather than just performing isolated tasks, thereby providing a comprehensive assessment of their true diagnostic reasoning abilities grounded in expert practice.\n\n\n**Benchmark Quality Check.**\nWe thank the reviewer for highlighting the need for clarification on our quality check procedure. Following the initial generation of questions by GPT-4o based on Eurorad cases, we performed an automated quality verification, also utilizing GPT-4o. Specifically, the automated check evaluated each generated question and answer set for structural consistency (e.g., six-choice question with one correct answer), explicit grounding in the provided clinical and radiological context, and clear verifiability of the correct answer from the original Eurorad source case material. Any questions failing these criteria were automatically identified and excluded. We have provided more details on this verification procedure in the revised manuscript.", "claims_and_evidence": "The paper clearly presents the MedRAX framework and thoroughly supports its claims with experimental results, demonstrating substantial advantages over existing methods. The key ideas—structured tool-based reasoning, modular tool integration, and comprehensive benchmarking—are explicitly defined and clearly validated by experimental outcomes.", "methods_and_evaluation_criteria": "The methods consist of integrating various specialized medical AI tools (such as visual question answering, segmentation, grounding, report generation, disease classification, and chest X-ray generation) into a modular, structured reasoning framework known as MedRAX. The authors utilize a Reasoning and Acting (ReAct) loop that iteratively breaks down complex medical queries into sequential analytical steps. This systematic approach aligns well with practical clinical workflows in radiology, where interpretation often involves multiple interdependent steps and reasoning based on various specialized analyses.", "theoretical_claims": "The paper provided does not include theoretical proofs or formal algorithmic claims that require rigorous mathematical validation.", "experimental_designs_or_analyses": "The experimental evaluation uses straightforward accuracy metrics (percentage correct answers), which is suitable given the benchmark’s multiple-choice format. The comparisons with existing state-of-the-art general-purpose and biomedical models are clearly defined and implemented using official code and recommended configurations. The evaluation procedure (including handling retries for invalid responses and using regex-based response parsing) appears clearly described and methodologically sound, avoiding ambiguity in how outcomes are measured or interpreted.\n\nI have a minor concern for the benchmark creation: ChestAgentBench and CheXbench:\n- The authors describe the ChestAgentBench as containing 2,500 expert-curated questions across seven essential clinical competencies for chest X-ray interpretation. These categories (Detection, Classification, Localization, Comparison, Relationship, Diagnosis, Characterization) comprehensively represent clinically relevant tasks. I understand this is a comprehensive question pool, but how representative are these questions? \n- Questions were generated from expert-curated clinical cases (from Eurorad) using GPT-4o, with a clear methodology ensuring that answers are grounded explicitly in the original case descriptions. Any further verification by human experts for benchmark purposes?", "relation_to_broader_scientific_literature": "The paper positions itself clearly within broader scientific literature related to artificial intelligence (AI) in medical imaging, specifically chest X-ray (CXR) interpretation, and contributes by building upon several established ideas and addressing previously identified limitations.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "see above", "other_comments_or_suggestions": "see above", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BqIbRfFJIy", "reviewer": "Reviewer_4LiS", "summary": "This paper introduces an AI agent multimodal, multi-task chest X-ray (CXR) interpretation and analysis called MedRAX. This method leverages an LLM-controlled reasoning and acting (ReAct) loop, dynamically reasoning through multi-step queries and selecting pretrained, task-specific tools to complete each step. The authors also introduce a new benchmark of 2,500 medical queries, on which MedRAX outperforms existing generalist and domain-specific multimodal language models on a wide variety of tasks.\n\n## Update after rebuttal\nI thank the authors for their detailed rebuttal, particularly the inclusion of quantitative results on existing benchmarks. While I appreciate the additional descriptions of methodology, this still does not meet the standard of reproducibility. I understand that the open-source implementation will facilitate reproducibility in practice, but it is critical for a scientific paper to communicate the methodology clearly so that reviewers and readers can understand and evaluate its validity. As much as I appreciate the ambition and forward-thinking nature of this paper, I feel that the technical description of methodology remains a major shortcoming -- I am left wondering what is happening \"under the hood\" at each stage beyond the high-level descriptions provided. I will maintain my original recommendation of Weak Reject.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the insightful comments. We have worked hard to answer their concerns and think the suggestions have helped improve the clarity of our work.\n\n**Key Methodoloy.** The full details of Algorithm 1 are provided in response to `Reviewer KTkS`. Additionally, an anonymous GitHub repo is prepared at https://github.com/syaro1383/medrax\n\n**Prior Work.** We have added M4CXR and AgentClinic to the prior work section of the revised manuscript.\n\n**Need for ChestAgentBench.** The justification for making a new benchmark is provided in response to `Reviewer DGwp`.\n\n**Further Evals.** Thank you for suggesting an expanded evaluation with more benchmarks. We've conducted additional benchmarking to compare MedRAX with state-of-the-art models on two benchmarks. Model performances were obtained from Park et al. (2024)\n\n1. MIMIC-CXR Radiology Report Generation\n\nThis benchmark evaluates single-image chest X-ray radiology report generation on the MIMIC-CXR test set, which includes 3,858 images. It assesses the clinical accuracy of generated reports by analyzing the presence or absence of 14 observations of medical conditions using CheXbert.\n\n- mF1-14: Micro-averaged F1 score for all 14 CheXbert observation labels\n- mF1-5: Micro-averaged F1 score for 5 key findings (cardiomegaly, edema, consolidation, atelectasis, pleural effusion)\n- MF1-14: Macro-averaged F1 score for all 14 labels\n- MF1-5: Macro-averaged F1 score for 5 key findings\n\n**Table A: Single-image performance on MIMIC-CXR test set**\n\n| Model | mF1-14 | mF1-5 | MF1-14 | MF1-5 |\n|-------|--------|-------|--------|-------|\n| LLM-CXR† | 36.0 | - | 21.1 | - |\n| RaDialog | - | - | 39.4 | - |\n| METransformer† | - | - | - | - |\n| DCL† | - | - | - | - |\n| PromptMRG | - | - | 38.1 | - |\n| LM-RRG | - | - | - | - |\n| Med-PaLM M 84B | 53.6 | 57.9 | 39.8 | 51.6 |\n| CheXagent* | 39.3 | 41.2 | 24.7 | 34.5 |\n| MAIRA-1 | 55.7 | 56.0 | 38.6 | 47.7 |\n| LLaVA-Rad | 57.3 | 57.4 | 39.5 | 47.7 |\n| M4CXR | 60.6 | 61.8 | 40.0 | 49.5 |\n| MedRAX | 79.1 | 64.9 | 34.2 | 48.2 |\n\n2. SLAKE VQA Benchmark\n\nThe SLAKE benchmark evaluates medical visual question answering using 114 chest X-ray test samples with close-ended questions in English. These questions typically focus on the presence or absence of abnormalities, anatomical identifications, and medical condition assessments.\n\n- Accuracy: Percentage of exact matches between model predictions and ground truth answers\n- Recall: Proportion of ground truth words present in the generated responses\n\n**Table B: Medical VQA performance**\n\n| Model | Accuracy | Recall |\n|-------|----------|--------|\n| RaDialog | 0.0 | 45.6 |\n| RadFM | 68.4 | 69.7 |\n| CheXagent | 71.1 | 73.2 |\n| M4CXR | 85.1 | 86.0 |\n| MedRAX | 90.35 | 91.23 |\n\n**Other Comments.** We have corrected \"Llava-Med\" to \"LLaVA-Med\" and added metric descriptions to all table captions.\n\n**Questions.**\n1-9. Discussed above.\n\n10. Following the initial generation of questions by GPT-4o based on Eurorad cases, we performed an automated quality verification, also utilizing GPT-4o. Specifically, the automated check evaluated each generated question and answer set for structural consistency (e.g., six-choice question with one correct answer), explicit grounding in the provided clinical and radiological context, and clear verifiability of the correct answer from the original Eurorad source case material. Any questions failing these criteria were automatically identified and excluded. We have provided more details on this verification procedure in the revised manuscript.\n\n11. During development, we observed that heavily favouring only tool use could lead to rigid or incorrect outputs if a tool failed or misinterpreted, while discouraging tool use missed opportunities for leveraging specialized analysis. Finding a balance, where the agent reasons first but uses tools judiciously to complement and tune its reasoning, appeared to yield better results empirically. A formal quantitative analysis exploring this balance was outside the scope of this paper's experiments but is noted as valuable future work.\n\n12. Yes, the prompt for MedRAX is as follows:\n> \"Answer this question correctly using the chain of thought reasoning and carefully evaluating choices. Solve using your own vision and reasoning and then use tools to complement your reasoning.\"\n\nThis prompt encourages the agent to combine its own reasoning ability and external tools to complement or refine initial assessments. The prompt worked better empirically during development. A formal quantitative comparison of prompting strategies is planned for future work.", "claims_and_evidence": "If the goal is to claim superiority over existing state-of-the-art, then experimental validation appears sound but could be strengthened. The justification for creating a new benchmark was not entirely made clear – what specifically is lacking in existing medical (or CXR-specific) reasoning and analysis benchmarks?\n\nFurther, for a model with such diverse capabilities, it is surprising to see performance reported on two benchmark datasets against just four baseline models. I imagine this is due to the limited number of models capable of all tasks, but analysis could be performed on subsets of tasks or benchmarks could be chosen to facilitate comparison with more existing methods.", "methods_and_evaluation_criteria": "The benchmark datasets used in this paper are appropriate for this unique and diverse multi-task setting. The proposed method is uniquely tailored to handling the suite of CXR analysis tasks considered.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experimental design appears sound, but it is difficult to evaluate such complex multi-task evaluation without more granular methods description or access to source code. Key methodological and implementation details range from completely missing to insufficiently described.", "relation_to_broader_scientific_literature": "This is a unique and ambitious paper that may merit acceptance based on novelty alone. I imagine that this will be among the first of many future works to consider task-agnostic “agentic” approaches to healthcare data analysis, and I commend the authors for their foresight and quick experimentation.", "essential_references_not_discussed": "This is a nascent space, and the authors sufficiently capture the few relevant prior studies. Perhaps AgentClinic [1] is worth mentioning, even though fully text-based, as an example of “agentic” AI for clinical use cases.\n\nWhile M4CXR [2] does not adopt an agentic approach, it is a relevant example of a multimodal foundation model for CXR analysis capable of a wide variety of tasks like MedRAX.\n\n[1] Schmidgall, Samuel, et al. \"AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments.\" arXiv preprint arXiv:2405.07960 (2024).\n\n[2] Park, Jonggwon, et al. \"M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation.\" arXiv preprint arXiv:2408.16213 (2024).", "other_strengths_and_weaknesses": "*Strengths*:\n- Highly unique and novel idea that is likely to garner significant attention and even be a cornerstone for future work. It is a forward-thinking approach to CXR interpretation\n- The paper is well-written and clearly organized with simple, effective visuals. \n\n\n*Weaknesses*:\n- This reads more like a tech report than a scientific paper – there is virtually no description of the core methodology. Nearly every line in Algorithm 1 requires concrete definition and detailed explanation. Certain key components of this should appear in the main text, but most implementation details should minimally appear in the Supplement (currently there is none!).\n- While I appreciate the work that went into curating a new benchmark, the justification for this needs to be clarified – what specifically is lacking about current benchmarks?\n- Similarly, while I am aware that only a select few models are even capable of all tasks considered in this setting, evaluation can be expanded to more benchmarks and more baseline methods. E.g., using established baselines could facilitate direct comparison with existing state-of-the-art on individual tasks or subsets of tasks.", "other_comments_or_suggestions": "This is a difficult paper to evaluate as a reviewer. I cannot in good conscience recommend acceptance for a paper that leaves so many core methodological details undescribed. However, if these details are clarified in the rebuttal (and, ideally, if evaluation is strengthened), then this is a very straightforward acceptance of what I could imagine becoming an influential paper.\n\nMinor comments:\n-\tLine 157 on RHS: Change “Llava-Med” -> “LLava-Med”\n-\tI would clarify the metric used in each table caption, even if it is mentioned elsewhere in the text.", "questions_for_authors": "Algorithm 1 poses many questions about the method that go unanswered:\n\n1.\tWhat does “Observe()” mean? \n2.\tWhat is “Reason()”? Define this and explain how it is implemented.\n3.\tWhat is “RequiresUserInput()”? How is this determined?\n4.\tWhat is “GenerateUserPrompt()”?\n5.\tHow is it determined whether the agent can generate a response?\n6.\tHow is tool selection performed?\n\nMajor questions:\n\n7.\tWhat specifically is lacking about existing benchmarks that required the creation of a new one?\n8.\tCan the authors provide additional evaluation on individual tasks or subsets of tasks in order to facilitate comparison with more relevant baselines? E.g., might it be possible to form comparisons on select tasks with M4CXR [2], MedVersa [3], or even Google’s MedPalm M [4] if using the same evaluation benchmarks as them? For a method with such diverse capabilities, the extent of evaluation feels underwhelming.\n9.\tHow does MedRAX perform compared to “specialized” models on individual tasks? This could be included as a gray row in each table to provide context for a reasonable upper bound on task performance.\n\nMinor questions:\n\n10.\tWhat does it mean that “all questions underwent a quality check”? Was this performed by anyone with medical expertise?\n11.\tThis is mentioned in the Discussion: “Our initial observations suggest the importance of balanced tool utilization, where neither complete reliance on tools nor their complete absence produced optimal results.” This sounds fascinating but cannot be evaluated since no evidence was provided for this – what observations? Can the authors provide some quantitative (or other) analysis of this in the results? \n12.\tWere any handcrafted prompts used in evaluation? What were they and how were they decided?\n\n[3] Zhou, Hong-Yu, et al. \"A generalist learner for multifaceted medical image interpretation.\" arXiv preprint arXiv:2405.07988 (2024).\n\n[4] Tu, Tao, et al. \"Towards generalist biomedical AI.\" Nejm Ai 1.3 (2024): AIoa2300138.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces MedRAX, a modular medical reasoning agent that integrates multimodal tools and LLMs for chest X-ray interpretation. It also presents ChestAgentBench, a benchmark for evaluating complex clinical queries. The system performs well in comparison to prior generalist and domain-specific models across multiple tasks, and the authors provide a compelling case for its clinical relevance. Reviewers appreciated the novelty and utility of the agent-based architecture, the quality of the benchmark, and the thoughtful design of the evaluation. The rebuttal effectively addressed concerns around implementation detail and evaluation breadth including new results on MIMIC-CXR and SLAKE. Some reviewers increased their scores after these clarifications while one reviewer maintains their weak reject score. AC believes the paper will be valuable for the community and the pros overall outweight the limitations. Therefore the paper is recommended for acceptance. Authors are reminded to improve the implementation details in revision and open-source the code as promised.
Decision: Accept (poster)