MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
Authors: Yuxin Zuo
Shang Qu
Yifei Li
Zhang-Ren Chen
Xuekai Zhu
Ermo Hua
Kaiyan Zhang
Ning Ding
Bowen Zhou
Venue: ICML 2025 poster (2025)
Keywords: Medicine
Benchmark
Multimodal
Primary area: applications->health_medicine
TL;DR: We introduce MedXpertQA, a highly challenging and comprehensive benchmark to evaluate expert-level medical knowledge and advanced reasoning.

Abstract: We introduce MedXpertQA, a highly challenging and comprehensive benchmark to evaluate expert-level medical knowledge and advanced reasoning. MedXpertQA includes 4,460 questions spanning 17 specialties and 11 body systems. It includes two subsets, Text for text evaluation and MM for multimodal evaluation. Notably, MM introduces expert-level exam questions with diverse images and rich clinical information, including patient records and examination results, setting it apart from traditional medical multimodal benchmarks with simple QA pairs generated from image captions. MedXpertQA applies rigorous filtering and augmentation to address the insufficient difficulty of existing benchmarks like MedQA, and incorporates specialty board questions to improve clinical relevance and comprehensiveness. We perform data synthesis to mitigate data leakage risk and conduct multiple rounds of expert reviews to ensure accuracy and reliability. We evaluate 18 leading models on MedXpertQA. Moreover, medicine is deeply connected to real-world decision-making, providing a rich and representative setting for assessing reasoning abilities beyond mathematics and code. To this end, we develop a reasoning-oriented subset to facilitate the assessment of o1-like models.

MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
Yuxin Zuo *1
Shang Qu
Yifei Li
Zhangren Chen
Xuckai Zhu' Ermo Hua"
Kaiyan Zhang'
Ning Ding R
Bowen Zhou 12
Leaderboard:
https:
o Code: http::/wwww.........../a//////ee
Abstract
MMLU (Medical)
MedQA
MedMCQA Dev
MedXpertQA Text
100
We introduce MedXpertQA, a highly challeng- ing and comprehensive benchmark to evaluate expert-level medical knowledge and advanced rea- soning. MedXpertQA includes 4, 460 questions spanning 17 specialties and 11 body systems. It includes two subsets, Text for text evaluation and MM for multimodal evaluation. Notably, MM in- troduces expert-level cxam questions with diverse images and rich clinical information, including patient records and examination results, setting apart from traditional medical multimodal bench- marks with simple QA pairs generated from im- age captions. MedXpertQA applies rigorous fil- tering and augmentation to address the insufficient difficulty existing benchmarks like MedQA, and incorporates specialty board questions to im- prove clinical relevance and comprehensiveness. We perform data synthesis mitigate data leak- age risk and conduct multiple rounds of expert reviews to ensure accuracy and reliability. We evaluate 18 leading models on MedXpertQA. Moreover, medicine is deeply connected to real- world decision- making, providing a rich and rep- resentative setting for assessing reasoning abilities beyond mathematics and code. To this end, we develop reasoning-orienter subset to facilitate the assessment of ol-like models.
75
51.3
56.3
25
1y Qwen2.5-728
GPT-40
o1
DeepSeek R1
Performance of different models on MedX- perlQA Text and other benchmarks. For other benchmarks, report results for ol ol-preview in place of ol. Ap- pendix F shows full results.
Figure 1:
1. Introduction
Large Multimodal Models (LMMs) have demonstrated promising potential in advancing general medical Al sys- tems for applications in clinical scenarios (Achiam et al., 2023; Liu et al., 2024b; Saab et al., 2024). However, cur- rent text and multimodal benchmarks for evaluating general medical Al capabilities have numerous limitations.
First, existing text medical benchmarks, such Pub- MedQA (Jin et al., 2019), MedQA (Jin et al., 2021), MedM- CQA (Pal et al., 2022), and MMLU (Medical) (Wang et al., 2024b), lack comprehensive coverage of fine-grained and diverse real-world diagnostic scenarios, including highly specialized fields such as family and addiction medieine. This lack essential breadth limits the applicability of med- ical AI in thoroughly addressing realistic medical scenarios. Moreover, these benchmarks fall short of sufficiently chal- lenging current advanced AI, hindering progress toward reli- able medical AI. For instance, o1 ol-preview iew has achieved 96% and 99% accuracy on MedQA-USMLE and MMLU Medical Geneties, respectively (Nori et al., 2024).
'Equal contribution 'Tsinghua University. Beijing, China "Shanghai Artificial Intelligence Laboratory, Shanghai, China. Correspondence to: Ning Ding <dn97 tai..aiaaaaaua.cu.uuuo cdu.cn.. Bowen Zhou <zhoubowen @ mail. wwn.llssinghu..uu.....
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
Second, traditional multimodal medical benchmarks, such as VQA-Rad (Lau al., 2018), Path-VQA (He et al., 2020), Slake (Liu et al., 2021), PMC-VQA (Zhang et al., 2023), and OMn (Hu et al., 2024), are critically incon- sistent with real-world clinical scenarios due to: 1) Lim- ited Scope and Insufficient Diffiicllt. These benchmarks solely evaluate basic visual perception and medical knowl- edge, neglecting the complexity of real-world medical tasks across different stages of the diagnosis process. They fail to assess the expert-level knowledge and reasoning ability required for diagnostic decision-making and treatment plan- ning. 2) Lack of Authenticity and Clinical Relevance. Current benchmarks lack detailed clinical information and rely on automatically generated simple QAs paired with isolated medical images, diverging considerably from re- alistic clinical scenarios. Medical exam questions used in existing text benchmarks present a promising solution, and Med-Gemi n (Saab et al., 2024) also demonstrates the sig- nificance of such evaluations. However, the field still lacks such a systematic and high-quality benchmark.
thousands of human responses, calibrated to the difficulty ratings annotated by human experts. We then filter questions based on 14 sampling votes from 8 Al experts for each ques- tion. Additionally, we use probabilistic semantic similarity and exact matching precision to identify and remove highly similar questions, thereby enhancing robustness. To ensure MedXpertQA is unseen for existing Al models, we use two proprictary models to rewrite questions and augment options, enhancing diversity and minimizing data leakage risk. To increase difficulty and robustness, we also elimi- nate low-quality distractors (i.e. easy and incorrect answer choices) and expand the number of options to 10 and 5 for Text and MM, respectively. Finally, to mitigate potential errors such as factual inaccuracies, we engage medical ex- perts with physician's 's licenses to review MedXpertQA, annotating and correcting any errors.
We evaluate 18 proprietary and open-source LMMs and LLMs on MedXperIQA, including cutting-edge inference- time scaled models. Results demonstrate that current lead- ing models still exhibit limited performance on MedX- pertQA, partieularly in complex medical reasoning tasks. In conclusion, our contributions are as follows:
To address these challenges, we present MedXperIQA, a highly challenging and comprehensive medical multiple- choice benchmark. It encompasses MedXperIQA Text for text-only evaluations and MedXpertQA MM for mul- timodal assessments, making it suitable for a wide range of Al models. Both subsets are currently the most challeng- ing benchmarks in their respective fields. Figure 1 presents model performance comparisons of MedXpertQA Text and other benchmarks. MedXpertQA MM includes di- verse image types to simulate the wide range of visual in- formation encountered in real-world diagnosis. Figure 2 shows an overview. Overall, MedXpertQA covers wide range of medical specialties and systems and includes chal- lenging real-world clinical tasks, enabling comprehensive evaluation of expert-level medical abilities.
We introduce MedXpertQA, a highly challenging and comprehensive medical multiple-choice benchmark. MedXperIQA integrates specialty-specifte assessments into medical benchmarking and challenging exam ques- tions with real-world clinical information into multimodal medical benchmarking.
After rigorous filtering and data augmentation, MedX- pertQA demonstrates exceptional difficulty and robust- ness. We further implement data synthesis to mitigate data leakage risk and conduct multiple rounds of expert reviews to ensure accuracy and validity.
MedXpertQA follows a rigorous and systematic construc- tion process, including data collection, filtering, synthesis, and expert review. We collaborate closely with medical prac- titioners throughout the design and review of Med MedXperIQA to ensure that its development reflects real-world medical practice and clinical reasoning. Specifically, we first curate a large-scale question bank from professional exams and textbooks, ensuring difficulty and diversity. Sources include the United States Medical Licensing Examination (USMLE) and the Comprchensive Osteopathic Medical Licensing Ex- amination of the United States (COMLEX-USA) for general medical evaluation, 17 American specialty board exams for specialized scenarios, and 3 image-rich sources, such as the NEJM Image Challenges. Thus, it primarily aligns with medical practices and standards in the United States. We subsequently perform extensive and multi-dimensional ques- tion filtering. First, we conduct hierarchical filtering using an adaptive Brier score (Zhu et al., 2024) threshold based on
We evaluate 18 leading models and analyze their limita- tions in medical capabilities. Moreover, we obtain Rea- soning subset specifically designed for assessing medical reasoning abilities, which is well-suited for evaluating ol reasoning models.
2. Related Work
Multimodal Medical Benchmarks Traditional multi- modal medical benchmarks can be broadly categorized into two types: specialized and general purpose (Chen al., 2024b). Specialized benchmarks focus on a specific modal- ity medical domain. VQA-Rad (Lau et al., 2018), VQA- Med (Ben Abacha et al., 2019), and Slake (Liu et al., 2021) are primarily centered on radiology, while Path-VQA (He et al., 2020) focuses on pathology. These benchmarks provide extensive evaluation for their intended specialties, yet have a highly constrained scope and limited general-
2

---

MedXperiQA: Benchmarking Expert-Level Medical Reasoning and Understanding
Data Sources
Previous Benchmarks
Medical Exams
+ USMLE
NBOME
COMI EXTUSA
colon more prominent on the patient's right side or left side?
Simple reasoning process Basic medical knowledge
Image- Rich Data
EDiR
ACR
FOURSAAL of MESTICCNE
(A) Right (B) Left
Final answer: (B) left
BB
Medical Boards
Medican
Portinty
Neuraloyy
Surgery
Medicn
Medkore
Image Types
MedXpertQA MM
272r woman prosents amergency department with a migraine headacho. complains photophobia aura, and severe nausea She reports vomitting several times, past medical history shows hypothyrsidismm and depression Her medications include levothyrooiie she lying with her hand over her eyes and appears be in moderate distress. medication several doses an antemmttc hour later, migraine has broken ciplopia. temperature
Complex, multi- step reasoning Expert- level decision- maniin Aligned with realistic scenarios
Radiology: X-Ray, CT, MRL PET, Ultrasound,
Vital Signs: ECG, Blood Pressure,
Pathology
sertraline On physical axam,
given an abortive
Possible cause! metodopramide
Treatment:
anticolenergic
pressure 152/3t mmilg pulse 107/min, respirations 20/man. repeat exam, extended and her jaw clenched. eyes upwards conwergence testing. Har pefieses are throughout. Laborstory testing performed and reveals: mEy/
patient complains
(37)))) blood
symptoms:
Additional
Benztropine
drug. eg (E)
Retinal Images
Photos
Other: Diagrams, Statistical Plots, Pamphiet Pages, Tables,
stress responses
Final
Key symptoms:
diagnosis:
Skills
Medical Categories
oculogyric crisis
acule
Understanding
Understanding 554
Systems
Specialties
HC03- 25 mEa'l
09 mg/dl.
Serum:
nitrogen: mg/dl. Creatinines best step?
mEa/
Known side effect: prolonged QT interval
dystonia
589
(A) Dantrolene (B) Progranclal Lorarepam
Reasoning
1861
Reasoning
1446
Diagnosis: Primary,
Tasks
(D) Magnesium sulfate Benztropine
460 ms
Enological Functional...
Treatment: Examinations Trestments.
Underlined text are information relevant to the incorrect options.
Basic Medicine: Anatomy,
Text
MM
Discases...
Figure 2: Overview of MedXpertQA. The left side illustrates its diverse data sources, image types, and question attributes. The right side compares typical examples from MedXpertQA MM and a traditional benchmark (VQA-Rad)
izability. With advancements in LMMs, recent general- purpose benchmarks, such as OmniMedVQA (Hu et al., 2024) and GMAI-MMBench (Chen et al., 2024b), have provided more comprehensive evaluations. However, these benchmarks cover limited medical image types, while real- world diagnostic scenarios encompass a wider variety of medical and even general images. Furthermore, all tradi- tional multimodal medical benchmarks mentioned above are constructed automatically based on image captions, de- viating from real-world medical tasks. MMMU (H& & M) Series (Yue al., 2024a;b) alleviate this issue to some tent by introducing exam questions. However, they are not specifically designed for the medical domain, and the difficulty is restricted to the university level. Moreover, these benchmarks still lack detailed clinical information commonly encountered in real-world scenarios.
3. MedXpertQA
3.1. Overview
We introduce MedXpertQA, a universal medical bench- mark consisting of challenging text and multimodal subsets Text and MM, which are each divided into a few-shot devel- opment set with 5 questions and test set. Figure shows an overview. MedXpertQA is designed to assess expert-level medical knowledge and reasoning capabilities across 17 medical specialties, 11 human body systems, and 3 task cat- egories. It includes a total of 4, 460 questions sourced from examinations at the medical licensing level or higher, of which 2, 005 are multimodal questions with total 2, 839 images. It is the first to introduce medical specialty evalua- tions better simulate realistic medical scenarios. More- over, MedXpertQA MM is the first multimodal medical benchmark to incorporate challenging medical examination questions and real-world medical scenarios. Meanwhile, due to medicine's 's rigorous requirements for highly reliable outcomes across a comprehensive scope, we refrain from solely prioritizing difficulty and instead aim for question diversity and range as well. As a result, MedXpertQA both substantially challenges current models and showcases remarkable diversity across multiple dimensions. Figure 3 illustrates its wide coverage.
Text Medical Benchmarks Existing text medical bench- marks such MedQA (Jin et al., 2021), MedMCQA (Pal et al., 2022), MMLU (Medical) Series (Hendrycks et al., 2020; Wang et al., 2024b) are derived from medical entrance and licensing exams, which primarily emphasize general medical knowledge and evaluation. PubMedQA (Jin et al., 2019) is closed-domain benchmark, with questions gener- ated from PubMed articles. However, these benchmarks do not provide thorough evaluation of highly specialized diag- nostic scenarios to align sufficiently with real-world clinical practice. Their difficulty has also become inadequate.
Medical Coverage MedXpertQA collects questions from 17/25 member board exams (specialties) of the American
3

---

MedXperiQA: Benchmarking Expert-Level Medical Reasoning and Understanding
Body Systems
Question Topics
Endocrine
Muscular
5.5%
5.0%
Inlegumentary
Urinary
6.3%
3.9%
Reproductive
Other
Respiratory
8.4%
Skelettl
19.8%
Desgliatory Dystunatiinn
Digestive
9.8%
Cardiovasculer
Nervous
Diuduuale DESIAAlEES SUST
Womoot HOORD
13.7%
13.7%
Tasks & Subtasks
Diagnosis (50.54%)
Treatment (26.83%)
Basic Medicine (22.63%)
Urrnory TSQJASO
Etiologic (32.81%).
Medicines (45.98%),
Anatomy (39.23%),
Nedhhal
Differential (18.99%).
Surgical Procedures (28.98%), Other
Basic Biology (16.29)
Syndromic (18.23%),
Diseases (13.31%), Medical Genetics (11.72%), Statistics
Primary (20.85%),
Therapies (11.89%),
Predictive (4.14%),
Preventive Measures
Prognostic (3.69%)
(6.95%), Rehabilitation...
(6.85%), Medical
Selmntan Shentte
Retrospective...
Procedures...
Kenndl Ewalts
Figure 3: Attribute distributions of MedXpertQA showcase its diversity and comprchensiveness.
Board of Medical Specialties to enable evaluation of highly specialized medical scenarios. Moreover, applying the cate- gorization in Liachovitzky (2015), we instruct an LLM to annotate each question with its most relevant human body system. MedXpertQA covers all 11 systems.
Phase
MM
Text
Sum
Data Collection
868
26, 675
5413
Difficulty-Baan -Based Data Filtering
Human Expert Filtering
5. 133
5,013
10, 146
AT Expert Filtering
2, 125
2,612
4, 737
Modal Diversity The data in MedXpertQA covers broad spectrum of modalities. Its questions and answer choices both incorporate structured data, such as tables, and semi- structured documents. MedXpertQA MM's images simi- larly demonstrate high diversity and wide coverage. It not only encompasses medical imaging results obtained from various techniques diverse formats, but also other im- age types such as diagrams, charts, and documents, fully covering the spectrum of visual information that human doc- tors are expected to analyze. Specifically, MedXpertQA encompasses the following image categories: Radiology, Pathology, Medical Optical Imaging. Photos. Vital Signs, Diagrams, Documents, Charts, Tables, and Others.
Quality-Based Filtering and Augmentation
Edit Distance Filtering
2, 121
2,592
7.3
Semantic Similarity Filtering
2, 105
2,578
683
Question and Option Augmentation
2, 105
2,578
683
Multi-Rounds Expert Review Delete
2, 005
2,455
4, 460
Table :: Data statistics after different construction phases.
cedure involving data filtering, synthesis, and expert review. Table presents dataset statistics across different construc- tion phases. After initial data collection and filtering based question difficulty, the reductions in question count occur during the Similarity Filtering and Expert Review stages, which remove 54 and 223 questions, respectively. While the early stages primarily optimize for difficulty, these later stages focus on improving data quality. Given that MedX- pertQA constructed from authoritative and high-quality sources, filtering based on difficulty naturally has a more substantial impact on dataset size compared quality-based filtering. Additionally, during the expert review phase, re- viewers directly corrected the errors most of the flagged questions instead of deleting them.
Task Diversity To enable in-depth analysis, we use 4pt-40-2024-11-2 annotate each question's core medical task (Diagnosis, Treatment Planning, or Basic Medicine) (Huang et al., 2025) and fine fine-grained subtask. Appendix I.I presents relevant prompts.
3.2. Benchmark Construction
To construct MedXpertQA, we begin with data collection, followed by a carefully designed and resource-intensive pro-
We begin by constructing a large-scale question bank, sourcing difficult multiple-choice questions from authoritative medical examinations and textbooks. Pre-
Data Collection
hntps//wwww.abms.org

---

MedXper(QA: Benchmarking Expert-Level Medical Reasoning and Understanding
vious work primarily relied on USMLE questions for train- ing and evaluation (Jin et al., 2021). We expand the scope by including questions from COMLEX, another major med- ical licensing examination in the U.S., to capture the unique challenges of medical image interpretation in orthopedic practice. To further evaluate multimodal medical capabili- ties, we incorporate questions from the American College of Radiology (ACR) DXIT and TXIT exams, the European Board of Radiology (EDiR) exams, and the New England Journal of Medicine (NEJM) Image Challenge. In addition, we collect questions from 17 American medical specialty board exams, spanning a wide array of common medical specialties. Ultimately, we collect 37, 543 questions. We additionally obtain human response distributions and expert annotations, including explanations and difficulty ratings.
the maximum threshold set at the 25th and the minimum set at the 3rd percentile of all scores. Approximately 16.78% of questions lack the annotations above and thus do not undergo human expert filtering.
Step 3: Similarity Filtering. A key factor in achiev- ing robust evaluation is ensuring high diversity and avoid- ing repetitive assessments. Therefore, we filter data by identifying question pairs with extremely high semantic or edit distance similarities. For semantic similarity, we use MedCPT-Que -Encoderr (Jin et al., 2023), medical embedding model developed from P PBERT (Gu et al., 2020), to obtain sentence representations of all questions, then compute pairwise cosine similarities. Since similar- ities are one-dimensional and do not follow Gaussian distribution, we use the Interquartile Range (IQR) to iden- tify outliers with abnormally high similarity. For detected outlier question pairs, we compare their difficulty annota- tions and remove the easier question. Similarity Filtering plays a critical role in ensuring benchmark robustness. Tra- ditional visual medical benchmarks, such as Slake (Liu et al., 2021), often rely on rigid, template-based question generation, which limits question diversity. This is problem- atic since model performance can vary greatly depending question type, and repetitive questions allow models to exploit complate-speciiii shortcut patterns, leading to unexpected performance gains. In contrast, since MedX- pertQA was not generated from fixed templates, it exhibits high question diversity from the start, resulting in relatively few questions being removed this stage. Nonetheless, even minimal filtering is indispensable, as it removes near- duplicate questions that could otherwise cause benchmark overfitting and distort evaluation results.
Data Filtering We conduct Al & Human Expert Filtering to identify questions that challenge both humans and Al. Subsequent Similarity Filtering further enhances robustness.
Step 1: AI Expert Filtering. We employ 8 models, catego- rized as basic or advanced, as Al experts to vote on and filter questions. First, each basic Al expert performs 4 sampling attempts for each question. If any expert answers a question correctly in all attempts, the question is deemed too simple and removed. Second, questions that are answered incor- rectly by all AT experts are retained. This method effectively differentiates between questions that leading AI models can solve and those that remain challenging. Appendix C.1 lists models used in this phase.
Step 2: Human Expert Filtering. We use prior and pos- terior human expert annotations to identify questions that challenge humans. Using human response distributions, we first assess each question's o posterior difficulty by calculat- ing its Brier score, a widely applied metric of prediction accuracy (Zhu et al., 2024). We consider the question's 's response distribution over all answer choices as the answer prediction. Given the prediction vector y and label vector y, the Brier score B is:
Question and Option Augmentation Since questions in MedXpertQA are from publicly available sources, we per- form question and option augmentation to mitigate possible data leakage issues. Option augmentation also increases both the difficulty and robustness of MedXpertQA. For question augmentation, we instruct the LLM to rephrase the question through alternative expressions structural adjust- ments while preserving all original information. We empha- size professional language style, factual accuracy, and objec- tiveness. For option augmentation, we first identify and re- move low-quality distractors with low human response rates while retaining the correct option and at least two distrac- tors. Most multimodal questions consist of 5 options which are often image-dependent and therefore cannot be feasibly expanded. Therefore, we set the total number of options for MedXpertQA Text and MedXpertQA MM to 10 and 5, respectively. The generated distractors should be reasonable in the question's 's context, challenge medical experts, and remain consistent with the original options in both language and content style. We use gpt-40-200 2024--2--2 4-11 1-20 and
N U
1 N
B
)
where N' is the number of options, is the label for option i (0 or 1), and ÿJ is the proportion of responses selecting option i. A lower Brier score indicates a more accurate over- all prediction distribution, suggesting the question is easier. Compared to accuracy, the Brier score accounts for the response rates of all options, providing more precise diffi- culty measurement. Subsequently, we normalize the prior difficulty ratings annotated by medical experts and catego- rize questions into 3 levels, each associated with an adaptive Brier score threshold for stratified sampling. Higher-rated questions are assigned higher Brier score thresholds, with
5

---

MedXper(QA: Benchmarking Expert-Level Medical Reasoning and Understanding
Avg Lens
# Image Types
Clinical Scenarios
Benchmark
# Size
# Images
# Rate
Annotation
Specialties
VQA-Rad (Lau al., 2018)
451
204
0.45
14.61
Automatic Automatic
VQA-Med (Ben Abacha et al., 2019)
500
500
1.00
8.96
1
X
Path-VQA (He et al., 2020)
6, 6,719
858
0.13
15.38
1
Automatic
X
X
Slake-En (Liu et al., 2021)
1,061
96
0.09
13.97
1
Automatic
X
X
PMC-VQA (Zhang et al., 2023)
33, 430
29, 021
0.87
61.84
2
Automatic
X
X
OmniMcdVQA (Hu et al., 2024)
127,995
118, 010
0.92
42.40
4
Automatic
X
X
GMAI-MMBench (Chen et al., 2024b)
211
21, 180
1.00
49.85
4
Automatic
X
v
MMMU (H & M) (Yue et al., 2024a)
7,2
1,994
1.14
83.56
8
Expert
X
X
MMMU-Pro (H & M) (Yue et al., 2024h)
346
431
1.25
107.08
7
Expert
X
X
MedXpertQA MM
2, 0000
2, 2,2
1.43
149.35
10
Expert
Table 2: Comparisons with existing multimodal medical benchmarks. MedXpertQA MM demonstrates high complexity through its high average question length, a large number of images, and diverse image categories. The first category includes commonly used traditional benchmarks. The second category includes recently emerging and more comprehensive benchmarks. The third category includes general multimodal benchmarks that include medicine-related subsets but are not specifically designed for medical fields.
Benchmark
# Size
Avg Lens
Clinical Scenarios
Specialties
standing) in Appendix G.1), hence are suboptimal for isolat- ing the model's 's medical reasoning ability. Therefore, while leading o/-like models show promise in medicine (Xie et al., 2024; Nori et al., 2024), systematic evaluations specifically focused on their reasoning- masseein--based medical proficiency re- main lacking.
PubMacQQA (Jin et al., 2019)
1, 0.000
328.
X
MedMOQA-Dev (Pal et el., 2022)
4, 1888
53.84
MedQA-USMLE (Jin al., 2021)
1,273
215.46
MMLJ (Med.) (Hendrycks et al., 2020)
1,009
100.07
MMLU-Pro (Med.) J (Wang al., 2024h)
586
166.63
MedXpertQA Text
2,450
257.37
Table 3: Comparisons with text medical benchmarks.
Moreover, medicine tightly interweaves professional knowl- edge with complex reasoning. Clinical reasoning is multi- factorial (Yazdani & Hoseini Abardeh, 2019), requiring syn- thesizing diverse information, navigating uncertainty (Patel et al., 2024), and engaging in heterogeneous cognitive pro- cesses (Shin, 2019). These characteristics are represented in MedXpertQA through complex, information-rict tasks such multiple-choit style differential diagnosis (Seller & Symons, 201 1) (see Appendix G.I and G.2). Therefore, we underscore the importance of identifying reasoning- -rientee tasks to facilitate fine-grate ine-grained performance feedback and ac- curate assessment. On MedXpertQA, we use got-4o to annotate questions according to whether they require com- plex reasoning. As seen in Appendix I.1, we instruct the LLM to categorize complicated, reasoning -havy questions as Reasoning. In contrast, other questions that involve little to no reasoning and instead assess skills such as medical knowledge and image perception are categorized as Under- standing. As shown in Figure 2, within both Text and MM, a majority of questions focus on Reasoning.
claude 1-55-sonnet 20241022 in this stage, and Appendix I.2 shows all prompts. Section 5.1 compares the leakage risk before and after data augmentation to verify the effectiveness of this process.
Expert Review Finally, medical experts with physician's 's licenses review each question in MedXpertQA, assessing the general quality of the original and augmented questions and options based comprehensive guidelines. Objectives of the question review include identifying missing informa- tion, detecting factual inaccuracies, and removing extra con- tent. Options review primarily emphasizes evaluating the options' validity and context-appropriateen Appendix E provides a detailed description of this procedure.
3.3. Medical Reasoning Benchmarking
OpenAl's ol models advance reasoning capabilities, which have been extensively validated in math and coding (Wu et al., 2024). However, evaluation in specialized domains like medical reasoning has been insufficient, as relevant benchmarks remain underdeveloped. Recent works de- veloping -liee medical reasoning models with reinforce- ment learning (Chen et al., 2024a) or inference-time scal- ing (Huang et al., 2025) similarly face limitations. Bench- marks chosen, such as MedQA, contain questions assessing medical knowledge only (see Example of MedQA (Under-
We initially considered having human experts annotate Rea- soning and Understanding labels. However, we found that LLMs can perform this task reliably when provided with clear and precise prompt instructions. Specifically, we pro- vide expert-written answers and explanations collected from data sources and explicitly emphasize the distinction be- tween reasoning complexity and general difficulty in our prompt design, as shown in Table 15). This extensive guid-
6

---

MedXper(QA: Benchmarking Expert-Level Medical Reasoning and Understanding
ance enables a simplified form of annotation under expert supervision. Following automatic labeling, engage hu- man experts to conduct review. For randomly selected 10% subset of the questions, 490 from Text and 400 from MM, the reviewers identified 28 and 11 questions, respec- tively, that were incorrectly labeled. This corresponds to an error rate of approximately only 4.3%.
latest progress in medical reasoning capabilities.
Large Multimodal Models: 2--22--2--171 (Jaech et al., 2024), DVQ-72B-Previeww (Team, 2024a), GPT-40-2024-
11-20 (Hurst et al., 2024), 109T-4--mini-2024-07--1111 (Hurst et al., 2024),
(Anthropic, 2024), Gemini-1.5-Pro-0022 (Team et al., 2024), Gemini-
2.0-Flash-Expp (Google, 2024), Qwen2.5-VL-72B (Team,
2025), Qwen2-VL-72B (Wang et al., 2024a).
3.4. Comparisons with Existing Benchmarks
Large Language Models: 03-mini! (OpenAI, 2025), DeepSeek-R1 (Guo et al., 2025), QwQ-32B- Preview! (Team, 2024b), DeepSeck-V3 (Liu et al., 2024a),
Tables 2 and 3 compare two subsets of MedXpertQA with existing benchmarks. Traditional multimodal bench- marks deviate from real-world clinical tasks, reflected in the limited number of image types, low Amage-to-question ra- tios, and automatically generated questions and annotations. Meanwhile, MMMU (H& & M) Series primarily based on university subject exams, falls short in scope, difficulty, and specificity to the medical domain.
(Anthropic, 2024), LLaMAA333-70B-Insttuer (Dubey et al., 2024), LLaMA- B-70B-UIrrMeddcal (Zhang al., 2024), Qwen2.5-72B- Instruct (Yang et al., 2024), Qwen2.5-32BBInstrun (Yang et 2024).
MedXpertQA MM demonstrates advantages in length and image richness. It is the first multimodal medical benchmark to incorporate real-world clinical scenarios, resulting in a substantial increase in question length. Mcanwhile, the rich image information in MedXpertQA MM can be seen from its broadest range of image types, highest number of images within scme-scale benchmarks, and highest image-to-question ratio.
4.3. Main Results
Tables 4 and 5 show the main results. Overall, the low ac- curacies of evaluated models demonstrate MedXpertQA's ability to pose challenges to state-of-the-art models.
Human Performance Evaluation We establish a base- line denoted as Expert (Pre-Licensed), referring to upper-year medical students or trainees preparing for licensing examinations such as USMLE, as shown in Ta- ble 4. This baseline is derived from aggregated response distributions based on answers provided by these examinees, collected from the original data sources. Since Question and Option Augmentation preserves all core clinical content, its impact on human accuracy is minimal. Accordingly, perfor- mance on the final questions closely aligns with that on the original versions. Most questions have substantial response counts, with large portion of the dataset containing at least 500 responses per question and some reaching up to 238k. Fewer than 200 final questions lack response data. These characteristics ensure that the resulting statistics provide stable and representative estimates of human performance.
MedXpertQA Text is the first text medical benchmark to purposefully account for medical specialty assessment, sup- porting evaluations of highly specialized medical scenarios.
4. Experiments
4.1. Implementation Details
We evaluate all models using zero-shot CoT prompting (Ko- jima et al., 2022) unless otherwise specified. For answer cleansing, we follow the script provided by Kojima et al. (2022). Appendix G.3 shows case of evaluation. We employ greedy decoding for output generation if ensuring result stability. For reasoning models with specific evaluation requirements, we follow their respective instruc- tions. Appendix C.2 presents additional implementation details. We could not evaluate o1 and o3-mini on the full MedXperiQA due to costs. Instead, for both MedX- pertQA Text and MedXpertQA MM, we sample 10% of questions from the Reasoning and Understanding subsets respectively. The seed is set to 42.
available,
Comparison of LMMs Among vanilla LMMs, GPT-40 consistently performs best across all sub- sets. Gemi n LMM after GPT-4o 40 with an impressive performance on MedXpertQA MM, highlighting its advantage in multi- modal tasks. We also observe that the highly representative open-source LMM, Qwen2.5-vL-77. -5VVL-72B, outperforms yet still falls behind GPT-4o.
G 12. 0-Fl ash is the highest-scoring vanilla
4.2. Models
Comparison of LLMs DeepSeek-RI shows the strongest performance among LLMs, most notably on the Reasoning set, where it substantially outperforms other mod- els. The performance of Qwen2. .e0ann55333 5-32B, particularly on the Reasoning set, is nearly equivalent to random selection, so we limit the evaluation to models of this scale and above.
We evaluate leading LMMs and LLMS on full MedX- pertQA and MedXpertQA Text, respectively. We include both proprietary and open-source models, and also evaluate advanced inference-time scaled models with a focus on the
7

---

MedXper(QA: Benchmarking Expert-Level Medical Reasoning and Understanding
MedXperIQA Text
MedXpertQA MM
Model
Avga
Avgu
Avg
Reasoning
Understanding
Avg
Reasoning
Understanding
Avg
Expert Licensed)
41.74
45.44
42.60
45.76
44.97
45.53
43.48
45.20
43.92
Inference- Time Sculed Large Multimodal Models
off
46.24
39.66
44.67
52.78
65.45
56.28
49.09
52.21
49.89
QVQ-72B-Previce
22.08
20.71
21.76
33.54
33.57
33.55
27.09
26.95
27.06
Vanilla Large Multimodal Models
GPT-40
30.63
29.54
30.37
40.73
48.19
42.80 80
35.05
38.58
35.96
Cludee3.5-SSone.
19.88
25.81
21.31
33.33
32.85
33.20
25.76
29.22
26.65
Gemini-1 1-5mimi-1.5-Prry Pr5-Pro
19.18
21.22
19.67
32.85
37.36
34.10
25.16
29.05
26.16
GPT-4o-mini
17.09
20.20
17.84
28.22
27.62
28.05
21.95
23.80
22.43
Gemini-2.00 F.shFah
20.53
20.71
20.57
35.48
41.70
37.20
27.06
30.88
28.04
Qwen2.5-VL
17.89
18.17
17.96
29.53
31.05
29.95
22.98
24.41
23.35
Qwen2-VT. VI.-2BB
16.39
18.68
16.94
25.86
34.84
28.35
20.53
26.51
22.07
Table 4: Performance of LMMs MedXpertQA. & indicates the result is evaluated on a sampled subset. Avgr and Avg indicate the average performance on the Reasoning and Understanding subsets, respectively.
Model
Reasoning
Understanding
Avg
models generally score higher on MedXpertQA MM, this can be attributed to the smaller number answer choices for MedXpertQA MM and more rigorous filtering param- eters used for constructing MedXpertQA Text. Therefore,
Expert (Pre Licensed)
41.74
42.60
ol
16.2
39.66
11.67
QVQ-72B-Preee
22.08
20.71
21.76
Inference-Tinne Time Scaled LLMs
direct performance comparisons between the two subsets are not informative.
03-mini
37 63
36.21
37.30
DecpScck-R1
37.55 88
37.35
37.76
QwQQ-3B-PPeee
18.
15.79
18.00
Vanilla LLMss
5. Analysis
DeepSeek-V3.
23.91
24.96
24.16
Claude 3,5 Haiku
16.71
21.05
17.76
5.1. Data Leakage
23.86
26.49
24.49
LLAMA-3-70B-UlraMeer
20.
21.05
21.80
To evaluate whether Question and Option Augmentation reduces leakage risk, we follow Xu al. (2024) use perplexity (PPL) and N-gram accuracy (ROUGE-L and edit distance similarity) as metrics. We make certain adaptations and simplifications to assess leakage risk at the instance level before and after data synthesis. After concatenating the original question with a specific prompt such as "Answer:" to form the input, we calculate the PPI. of the model's output. Moreover, to assess whether the evaluated model's is rationale is similar to the corresponding explanations we collected, we compute the ROUGE-L and edit distance similarity between the output and the explanation. We analyze the outputs of GP T 4o, since it is the most effective among all vanilla LMMs, possibly reflecting a higher risk of data leakage.
Owen2.5-72B
18. 18.
20.03
18.90
Owen2.5-32B
14.02
18.34
15.06
Table 5: Performance of LLMs on MedXpertQA Text.
Phase Perplexity 1
Rouge-L.4
Edit Distance Similarity J
Before 1.03E 218
0.1893
0.2691
After 1.35E
247
0.1664
0.2416
Table 6: Data leakage analysis results.
Medical Reasoning Performance Table 4 indicates that the textual reasoning performance of vanilla LMMs other than GPT-4o is relatively close, while their visual reason- ing capabilities show marked differences, suggesting that visual perception plays a crucial role in approaching MedX- pertQA questions. Moreover, vanilla LLMs and LMMs consistently perform worse on the Reasoning subset than on the Understanding subset. However, this gap noticeably narrows for olie inference-tine scaled models. This un- derscores the inherent challenges associated with medical reasoning, as well as the high quality of our Reasoning sub- set. In other words, our annotations effectively produce a challenging wosooning-orientte subset that is well-suited for evaluating ol-like reasoning models. Additionally, while
Table 6 shows results. Based on the threshold set by Xu et al. (2024), the data leakage risk before synthesis al- ready relatively low, which be attributed to two factors: I) Questions in MedXpertQA are derived from difficult medical exams that may have not yet been considered for training; 2) Even questions already used for training may be difficult for the model to learn due to their complexity (Lin et al., 2024). Leakage risk is further reduced following data synthesis, particularly indicated by notable increase in per- plexity. Results demonstrate that MedXpert has a low risk of data leakage and can objectively assess model ability.
8

---

MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
Deepdeek V3
Qwen2. 323 Instruct
Qwen2 VL 72B
DeepSeek-Ri R1
QwD-32B -2e-eeiee
40.0
20.0
31.0
Accurncy
35.0
18.0
32.0
25.0
14.0
28.0
26.0
20.0
12.0
Understanding Reasoning
Understanding Reasoning
Understanding Reasoning
Figure 4: Performance of three groups of models on MedXpertQA.
5.3. Medical Insights
Cardiovascular
Accuracy
Digestive
Pct. Change
We analyze GPT-4o's performance based on the annotated system labels to derive tne-ggrined medical insights. Fig- ure shows that the proportion of Integumentary questions in the correctly answered set markedly increases compared with the full question set. We also observe much higher accuracy these questions than questions other systems. This suggests that GPT-4o has deeper understanding and stronger performance
Endocrine
Integumentary
lymphatic
Muscular
Nervous
Reproductive
Respiratory
Skeletal
Urinary
Other / NA
Integumentary-relatee questions. In contrast, it exhibits lower accuracy on the Cardiovascular subset, with a noticeable proportion decline in the correct set, indicating suboptimal capability this regard.
10
10 20
30
40
Accuracy (%) Percentage Change (.1%)
Figure 5: System analysis results. "Accuracy" bars show accuracies on each system's 's questions. "Percentage Change" bars show the gap between the percentages of each system's questions the correctly answered set and the set.
6. Conclusion
We introduce MedXpertQA, highly challenging and com- prehensive medical multiple-choice benchmark evaluating expert-level knowledge and reasoning in real-world clinical scenarios. MedXpertQA encompasses diverse medical spe- cialties, body systems, and clinical tasks. It addresses criti- cal gaps in current benchmarks, including inadequate cover- age of medical specialties, insufficient difficulty, and limited clinical relevance. By incorporating expert-level medical examination questions rooted in comprehensive clinical in- formation, MedXpertQA MM marks crucial advance- ment in multimodal medical benchmarking. We mitigate data leakage risk through data synthesis and perform expert review to ensure accuracy and validity. We benchmark 18 leading models and analyze their system-specifie medical capabilities. Additionally, we construct reasoning-orient subset, which demonstrates that current Al systems still face challenges when tackling complex medical reasoning tasks.
5.2. Impact of Inference-Time Scaling
We employ two groups of Qwen-Series models for compar- ative analysis to investigate the impact of inference-ting scaling on the models' performance on challenging medi- cal tasks. Specifically, they represent two text and multi- modal ol-like models, along with their corresponding back- bones. We similarly assess and compare DeepSeek-R1 and DeepSeek-V3.
DeepSeek R1 vs DeepSeek-VV Evaluation on MedXpertQA Text.
QwQ-32 32B Preview vs Qwen2. 3-2.-----------trttttt Evaluation on MedXpertQA Text.
QVQ 72B Preview ws Dwen2-Vi. VV-72B: Evaluation on MedXpertQA MM.
Figure 4 illustrates the performance the three groups across MedXpertQA Text and MedXpertQA MM. The accuracy variation observed each subplot is consistently more pronounced in the Reasoning set. Furthermore, curacy the Reasoning set exhibits clear upward trend, while performance on the Understanding set fluctuates, with occasional declines. This suggests that, even without ad- ditional medical training data, inference-time -ime scaling can improve complex medical reasoning skills.
A Acknowledgements
We thank anonymous reviewers for their insightful com- ments and suggestions. This work is supported by National Science and Technology Major Project (2023ZD0121400 Young Elite Scientists Sponsorship Program by CAST (2023QNRC001), and National Natural Science Foundation of China (No. 62406165).

---

MedXper(QA: Benchmarking Expert-Level Medical Reasoning and Understanding
Impact Statement
Chen, J., Cai, Z., Ji, K., Wang, X., Liu, W. Wang, R., Hou,
J., and Wang, B. Huatuogpt-ol, towards medical complex reasoning with Ilms. arXiv preprint arXiv:2412.1882 2024a.
This paper introduces MedXpertQA, a comprehensive medical benchmark to advance the field of healthcare application-Oriven machine learning.
Chen, P., Ye, J., Wang, G., Li, Y., Deng, Z., Li. W., Li, T., Duan, H., Huang, 7.,, Su, Y., et al. Gmai- mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. arXiv preprint 2024b.
The deployment of AI in healthcare raises ethical concerns, encompassing issues such as data privacy, algorithmic bi- ases, and the potential for excessive dependence on auto- mated systems. To mitigate these risks, it is crucial to prior- itize transparency, explainability, and continuous validation by healtheare professionals. Furthermore, Al systems must undergo ongoing assessment prevent the perpetuation of existing healthcare disparities.
Chen, Z., Wu, J.. Wang, W., Su, W., Chen, G., Xing, S., Zhong, M.. Zhang, Q.. Zhu, X., Lu, L.,, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024c.
While MedXpertQA demonstrates considerable promise for advancing medical AI, it also underscores the importance of responsible development and oversight to ensure that these technologies are applied ethically and equitably within healthcare settings.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Google. Gemini 2.0 flash. https: geminigmnnnnnn/ google.. com, 2024. Accessed: 12/2024.
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., and Poon, H. Domain-specific language model pretraining for biomedical natural lan- guage processing. 2020.
Anthropic. The claude model family: Opus, sonnet, haiku. 2024. URL https: /appi. saappissemme semant icscholac.. org/CorpusID :::.II2222292999
Guo, D.,, Yang, D.,, Zhang, H.,, Song. J., Zhang, R., Xu. R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501. 2501:25501.1249 2025.
Ben Abacha, A., Hasan, S. A., Datla, V. V., Demner- Fushman, D., and Müller, H. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. In Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 Septem- ber 2019.
He, X., Zhang, Y., Mou, L., Xing. E., and Xie, P. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. 10286, 2020.
Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., Dong, X., Duan, H., Fan, Fei, Z., Gao, Y., Ge, J., Gu, C., Gu, Y., Gui, T., Guo, A., Guo, Q., He, C.,, Hu, Y., Huang, T., Jiang, T., Jiao, P., Jin, Z., Lei, Z., Li, J., Li, J., Li, L., Li, S., Li, W., Li, Y., Liu, H., Liu, J., Hong, J., Liu, K., Liu, K., Liu, X., Lv, C., Lv, H., Lv, K., Ma, L., Ma. R., Ma, Z., Ning. W., Ouyang, L., Qiu, J., Qu, Y., Shang, F,, Shao, Y., Song, D., Song, Z., Sui, Z. Sun, P. Sun, Y., Tang, H., Wang, B., Wang, G., Wang, J.. Wang, J., Wang. R. Wang, Y.. Wang, Z., Wei, X., Weng, Q. Wu, F., Xiong, Y., Xu, C., Xu, R., Yan, H., Yan, Y., Yang, X., Ye, H., Ying, H., Yu, J., Yu, Zang, Y., Zhang, C., Zhang, I.., Zhang, P., Zhang, P.. Zhang, R., Zhang, S., Zhang, S., Zhang, W., Zhang, W., Zhang, X., Zhang, X., Zhao, H., Zhao, Q., Zhao, X., Zhou, F., Zhou, Z., Zhuo, J., Zou, Y., Qiu, X., Qiao, Y., and Lin, D. Internim2 technical report, 2024.
Hendrycks, D.,, Burns, C., Basart S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint 2009.:2009.03330 2020.
Hu, Y., Li, T., Lu, Q. Shao, W., He, J., Qiao, Y., and Luo, P. Omnimedvqa: new large- comprehensive evaluation benchmark for medical Ivlm. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22170-22183, 2024.
Huang, Z.., Geng, G., Hua, S., Huang, 7.,, Zou, H., Zhang, S., Liu, P., and Zhang, X. 01 replication journey- part 3: Inference-time scaling for medical reasoning. arXiv preprint arXiv:2501.0645 2025.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A.,
10

---

MedXper(QA: Benchmarking Expert-Level Medical Reasoning and Understanding
Radford, A., al. Gpt-40 system card. arXiv preprini arXiv:2410.21276 2024.
medical challenge problems and beyond. arXiv preprint arXiv:241 2024.
Jaech, A., Kalai, A., Lerer, A., Richardson, A., Ei-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., al. Openai ol system card. arXiv preprint arXiv:2412. 167200 2024.
OpenAL. Gpt-o3-mini.
https://openai.ccm
2025. 2025-01-31.
Pal, A., Umapathi, L. K., and Sankarasubbu, M. Medmeqa: A large-scale multi-subject multi-choice dataset for medi- cal domain question answering. In Conference on health, inference, and learning, pp. 248-260. PMLR, 2022.
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., and Szolovits, P. What disease does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
Patel, B., Gheihman, G., Katz, J. T., Begin, A. S., and Solomon, S. R. Navigating uncertainty in clinical prac- tice: A structured approach. Journal of General Internal Medicine, pp. 1-8, 2024.
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W. and Lu, X. Pubmedqa: A dataset for biomedical research question answering. arXiv preprini arXiv: 00009006666 2019.
Phan, L., Gatti, A., Han, Z., Li, N., Hu. J., Zhang. H., Zhang, C. B. C.,, Shaaban, M., Ling, J., Shi, S., et al. Humanity's last exam. arXiv preprint arXiv:2501.142249 2025.
Jin, Q. Kim, W., Chen, Q., Comeau, D. C., Yeganova, L., Wilbur, W. J., and Lu, Z. Medept: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformat- ics, 39(1 1,tad 2023.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlla github. io hoollg/gwwe....mm
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213, 2022.
Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wul- czyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18411 2024.
Lau, J. J., Gayen, S., Ben Abacha, A., and Demner- Fushman, D. dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Seller, R. H. and Symons, A. B. Differential Diagnosis of Common Complaints E-Book. Elsevier Health Sciences, 2011.
Shin, H. S. Reasoning processes in clinical reasoning: from the perspective of cognitive psychology. Korean journal of medical education, 31 (1(4):299, 2019.
Liachovitzky, C. Human Anatomy and Physiology Prepara- tory Course. 2015.
Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., al. Rho-1: Not all tokens what you need. arXiv preprint mrXiv:2404.07965, 2024.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint NKT::2403.053300 2024.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint ar arXiv: 19437, 2024a.
Team, Q. Qvq: To see the world with wisdom, Decem- ber 2024a. URL https:: //wwen/m github. io/ blog/ qvq 72b-ppbb-r.
Liu, B.. Zhan, L.-M., Xu, L., Ma. L., Yang. Y., and Wu, X.- M. Slake: A semantically- knowledge-enhanced dataset for medical visual question answering. In 2021
Team, Q. Qwq: Reflect deeply on the boundaries of the 202embber0024b. unknown, November 2024b. URL https: http::::://wwwnaae qwwnnmm
wtrthub.oobbogg/// htt.huu.../blloog//e-------e-p/
IEEE 18th International Symposium on Biomedical Imag- ing (ISBI), pp. 1650-1654. IEBE, 2021.
Team, Q. Qwen2.5-v1, Jamuary 2025. URL https://
qwenlm htt.gg........ooggoo/ggooooooote
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
Wang. P., Bai, S., Tan, S., Wang. S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing lanion-l model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12/9/, 2024a.
Nori, H., Usuyama, N., King, N., McKinney, S. M., Fer- nandes, X., Zhang, S., and Horvitz, E. From med- prompt to ol: Exploration of nun-time strategies for

---

MedXper(QA: Benchmarking Expert-Level Medical Reasoning and Understanding
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language under- standing benchmark. arXiv preprint arXiv:2406.01574, 2024b.
Zhu, K., Zheng, Y., and Chan, K. C. G. Weighted brier score-an overall summary measure for risk prediction models with clinical utility consideration. arXiv preprint arXiv:2408.0 01626, 2024.
Wu, J., Deng, W., Li, X., Liu, S., Mi, T., Peng. Y., Xu, Z., Liu, Y., Cho, H., Choi, C.-I,, et al. Medreason: Eliciting factual medical reasoning steps in Ilms via knowledge graphs. arXiv preprint arXiv:2504. 00993, 2025.
Wu, S., Peng, Z., Du, X., Zheng. T., Liu, M., Wu, J., Ma, J., Li, Y. Yang, J., Zhou, W., et comparative study on reasoning patterns of openai's ol model. arXiv preprint arXiv:2410. 13639, 2024.
Xie, Y. Wu, J. Tu, H., Yang, S., Zhao, B., Zong, Y., Jin, Q. Xie, C., and Zhou, Y. A preliminary study of ol in medicine: Are we closer to an ai doctor? arXiv preprint vrXiv:2409.15777 2024.
Xu, R., Wang. Z., Fan, R.-Z., and Liu, P. Benchmark- ing benchmark leakage in large language models. arXiv preprint ar 24v:22404. 18824, 2024.
Yang. A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint 11xiv22412.15115. 2024.
Yazdani, and Hoseini Abardeh, M. Five decades of research and theorization on clinical reasoning: critical review. Advances in medical education and practice, pp. 703-716, 2019.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W.. Sun, Y., et al. Mmmu: A massive multi musti-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024a.
Yue, X., Zheng, T., Ni, Y., Wang. Y., Zhang, K., Tong, S., Sun, Y., Yu, B., Zhang, G., Sun, H., et Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.028//1 2024b.
Zhang, K., Zeng, S., Hua, E., Ding, N., Chen, Z.R.., Ma, Z., Li, H., Cui, G., Qi, B., Zhu, X., al. Ultramedi- cal: Building specialized generalists in biomedicine. Ad- vances in Neural Information Processing Systems, 37: 26045-26081, 2024.
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., and Xie, W. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305. 10415, 2023.
12

---

MedXperiQA: Benchmarking Expert-Level Medical Reasoning and Understanding
A. Leakage Prevention Statement
The data of MedXpertQA is strietly intended for model or human medical evaluation purposes only. Please DO NOT use for any form of training, including training on synthesized or rephrased versions of the data.
To mitigate the potential data leakage risks, we refrain from releasing the data sources and request that you do not share any example of MedXpertQA online, whether in plain text, image, or any other format.
We obtained all data from freely and publicly accessible sources. We only retained a small percentage of source data, and all questions were subject to question rephrasing and option shuffling, fully ensuring that MedXpertQA complies with U.S. fair use laws.
B. Discussions
B.1. Ethical Considerations
To evaluate the reliability of MedXpertQA under real-world clinical variability, we consider the impact of input incomplete- ness on model performance. Specifically, we propose future assessments using semantically equivalent clinical scenarios with systematically omitted salient information to quantify the sensitivity of model predictions to such variations. This is motivated by the stochastic nature of L.LMs and their reliance on prompt phrasing. Furthermore, we highlight that benchmark performance does not imply clinical readiness. MedXpertQA, while constructed through rigorous expert filtering and augmentation, remains a synthetic evaluation framework. Safe use of medically oriented language models requires external safeguards beyond benchmark scores, including uncertainty estimation, structured oversight, and integration with clinical workflows. These considerations are essential for aligning benchmark utility with responsible use in healthcare contexts.
B.2. Comparison with HLE (Med)
To illustrate the difficulty of MedXpertQA, we compare it against HLE (Phan et al., 2025). HLE is derived from original questions contributed by nearly 1, 000 experts representing over 500 institutions across 50 countries. It has attracted considerable attention and is now regarded as one of the most challenging benchmarks for evaluating current state-of-the-art models. MedReason (Wu et al., 2025) specifically evaluated fine fine-graing grained medical subset HLE, referred to as HLE (Med), which is distinet from the B/M subset introduced in Phan et al. (2025). This medical subset provides new opportunities for detailed comparisons in terms of dataset statistics and evaluation difficulty.
Notably, MedXpertQA demonstrates higher level of challenge compared HLE (Med) as shown in Wu al. (2025). MedXpertQA is not only more comprehensive but also presents significantly greater difficulty. Despite the extensive human effort involved in the construction of HLE (Med), MedXpertQA is approximately 24x larger and imposes more demanding evaluation. These characteristics position MedXpertQA as the most comprehensive and challenging medical question-answering benchmark currently available.
C. Additional Implementation Details
C.1. Al Expert Models
Specifically, the basic AI experts include Owen2. (Qwen Team, 2024). InternLM.. (Cai et al., 2024), and LLaMA-33 1-88 (Dubey et al., 2024) for text-only medical questions filtration, Qwen2-VL-7B (Wang et al., 2024a), InternVL2-B 8B (Chen et al., 2024c), and LLaMA-3 2 2-11B 1e-11B-Viiiii -V-isii (Dubey et al., 2024) for multi- modal medical questions filtration. The advanced Al experts include proprietary models 2ott-4o-2024005 and claude-3 ---ee------o-ett-.-. -onnett-0221022.
C.2. Evaluation Settings
For both ol and DeepSeek-R1, we follow the corresponding recommended prompting guidelines to remove the system prompt. For DeepSeek-Be R1, we also include the prompt: Put your final answer within {bozed[]. Additionally, the API parameter semperature for these two models unsupported. For SVQ-72B-PrevA ew, since it only supports
13

---

MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
single-round conversation, remove the second round of zero-shot CoT and adjust the answer extraction script to handle choices within }
D. Error Analysis
In this section, we analyze the reasons behind model errors by classifying different models' incorrect answers into several error types. We consider the following error types:
Reasoning Process Error: The model's prediction rationale indicates errors in key reasoning steps, which led the incorrect answer.
Perceptual Error (for MedXpertQA MM only): The incorrect answer stems from a misunderstanding or misinterpretation of the image or images provided in the question.
Question Understanding Error: The answer shows an incorrect understanding of the original question.
Lack of Medical Knowledge: The model's prediction shows lack of medical knowledge necessary for arriving at the correct answer.
Formatting Error: The answer includes the correct content but is formatted improperly and causes the answer extraction process to fail.
For each model analyzed below, we sample 200 incorrectly answered questions from the MedXpertQA Text and MedX- pertQA MM subsets, respectively. We use gpt-4o- 200-2024-1121 to label each error type based on the question, correct answer, correct explanation, incorrect answer, and incorrect prediction rationale produced by the model.
Reasoning Process
Perceptual
Question Understanding
Medical Knowledge
Formatting
GPT-4o
GPT-40
Claude-3.5- Sonnet
Claude-3.5- Sonnet
Gemini-1.5
Gemini-1.5-PP
MedXpertQA Text
MedXpertQA MM
Figure 6: Error type distribution of three models on MedXpertQA.
Figure compares error type distributions across the following models: gemini-1.5 5-pro. Across all models, the high percentage of Reason- ing Process Errors on both MedXperIQA Text and MedXpertQA MM reflects the difficulty of our benchmark in terms of medical reasoning. Perceptual Errors are also prominent in MedXpertQA MM, demonstrating not only room for improvement in medical image perception for leading models, but also that image interpretation is core part of answering MedXpertQA MM questions, as expected.
In Appendix G.2, we provided examples error types with detailed analysis.
14

---

MedXperiQA: Benchmarking Expert-Level Medical Reasoning and Understanding
E. Expert Review
summarize the error types found and corrected during the expert review of augmented questions and options. Table provides a statistical overview.
Subset
Error Type
# Questions
Question Formatting Error
15
Narrative Inconsistency
5
Information Inconsistency
73
MedXpertQA Text
Option Formatting Error
21
Inapplicable Options
399
Unreasonable Options
15
Question Formatting Error
100
Contained Question Source Information
53
MedXpertQA MM
Narrative Inconsistency
70
Information Inconsistency
217
Table 7: Statistics of errors identified during the human expert review process.
Appendix H lists reviewer guidelines for each phase of expert review in our benchmark construction and annotation.
F. Full Results of Performance Comparison
Table 8 presents the results of different models on MedXpertQA Text and other text benchmarks.
Benchmark
Qwen2.5-72B
GPT-4o
ol
DeepSeek-Ri
MMLU (Medical)
89.62
91.90
95.30
94.12
MedQA-USMLE
72.701
84.40=
96.00F
92.38
MedMCQA-Dev
66.20t
76.40=
83.90F
79.12
MedXpertQA Text
18.90
30.37
44.67
37.76
Table 8: Performance of different models on MedXper Text and other benchmarks. t indicates the result is derived from Chen et al. (2024a). indicates the result is derived from Nori et al. (2024).
15

---

MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
G. Cases
G.I. Question Cases
G.1.1 Example of MedQA
Reasoning
3-ycar-oldd boy presents to the emergency department with a inspiratory noise that is audible without stethoscope. His mother reports that his cough has worsened over the last few hours. The patient's blood pressure is 118/78 mm Hg, pulse is 90/min, respiratory rate 35/min, and temperature is 38.39C (101.1 On physical examination, the boy is sitting and leaning forward in apparent respiratory distress with suprasternal and intercostal retractions. Auscultation reveals inspiratory stridor without wheezing. He has frequent barking cough and hoarse voice when he speaks. What is a chest X-ray likely to show?
Question:
bark' and a high-pitched
Answer Choices: (A) Increased interstitial markings (B) Lobar consolidation in the lingual (C) Thumbprint sign on lateral image (D) Steeple sign
Understanding
Question: A 3-month-old boy is brought the emergency department by his parents after episode of cyanosis and muscle hypotonia that resolved after 2 minutes. Diagnostic evaluation fails to discover an exact etiology of the boy's 's symptoms and the episode is classified as a brief resolved unexplained event (BRUE). The risk profile for BRUE in infants remains largely unknown. The pediatrician who saw the boy identify risk factors for BRUE. She is aware of several confounders, including age, socioeconomic background, and family history of medical illness. She recruits 75 infants under year of age with BRUE and 75 infants without BRUE of the two groups with regard to history of feeding problems and history of recent upper respiratory infection. Which of the following methods was conducted to control confounding bias in the study? Answer Choices: (A) Blinding (B) Restriction (C) Randomization (D) Matching
emergency department trying to
age, socioeconomic background, and family history of medical illness. She then compares the
G.1.2 Example MedXperIQA MM
Reasoning
Question: 30-year-old male presents to primary care with complaints of gradually worsening fatigue and shortness of breath over several months. His med- ical history reveals no smoking, alcohol use, illicit drug use, and his family history is noncontributory. He reports travel to South America one year prior but cannot recall any illness afterward. His vital signs show a temperature of 37.0%C (98.69F), heart rate of 75/min, and blood pressure 1//0 mmHg. Clin- ical examination reveals visible respiratory distress, systolic murmur the cardiac apex, and bilateral diffuse rales lung auscultation. A cardiac biopsy is performed. Based on the biopsy findings shown, what is the most probable cause of the patient's 's condition?
Answer Choices: (A) Autoimmune granulomatous disease (B) Viral infection (C) Bacterial infection (D) Fungal infection (E) Parasitic infection
16

---

MedXper(QA:
Benchmarking Expert-Level Medical Reasoning and Understanding
Understanding
Question: A pathologist views cross-sectional slides of the brainstem and identifies the following structure (denoted by an asterisk).
Which of the following best characterizes the function of the fibers passing through this structure?
Answer Choices: (A) Transmission of pain signals to the brain (B) Transmission of unconscious pro- prioceptive sensation (C) Conduction of temperature sensation (D) Initiation voluntary arm movements
(E) Regulation muscles of mastication
G.1.3 Example of MedXpertQA Text
Reasoning
Question: A 68- -e8-yea-old right-handed male is brought to the emergency department for evaluation sudden-onset weakness. He woke up with clumsiness in his left hand, which quickly progressed to weakness on his left side, more pronounced in the arm than the leg. His medical history includes hypertension, type 2 diabetes mellitus, and hyperlipidemia, but he has refilled his medications in year. He has 40-year history of smoking one pack of cigarettes daily. Vital signs reveal temperature of C (99.19F), pulse of 98/min, respirations of 20/min, and blood pressure of 200/1 10 mm Hg. Physical examination shows slurred speech, left-sided facial weakness sparing the forehead, 2/5 strength in the left upper limb, and 4/5 strength in the left lower limb, with normal strength and sensation on the right side. MRI reveals a right-sided infarct in the paramedian pontine base. Which additional finding most likely in this patient?
Answer Choices: (A) Ipsilateral tongue deviation (B) Contralateral loss of proprioception (C) Contralateral hemiparesis sparing the face (D) Eye deviation the left (E) Vertical gaze palsy (F) Ipsilateral facial weakness
involving the forehead (G) Eye deviation to the right (H) Loss of pupillary light reflex the right (I) Bitemporal
hemianopia (J) Contralateral hemianopia with macular sparing
Understanding
Question: 25-yyar-old previously healthy man presents to the clinic with cervical pain and stiffness that developed after long-duration flight from California New York the previous day. His symptoms have progressively worsened since onset. He reports no trauma the area and denies other symptoms. Physical examination reveals patient in mild discomfort with normal vital signs and fever. The paraspinal region from C5-C7 demonstrates hypertonicity. Which motion represents the primary movement of the affected cervical segments?
Answer Choices: (A) Coupled rotation (B) Extension (C) Flexion (D) Sidebending (E) Lateral flexion (F) Axial compression (G) Rotation (H) Retraction (I) Translation (J) Protraction
17

---

MedXper(QA: Benchmarking Expert-Level Medical Reasoning and Understanding
G.2. Error Cases
Reasoning Process Error
Question: A 64-year-old man is brought to the emergency room following a fall at his farm, presenting with a shoulder injury shown in Figure A. After successful reduc- tion, post-reduction images are obtained (Figures B and C). During his follow- up appointment 10 days later, he demon- strates an inability to perform shoulder ab- duction. Which diagnostic test would be most appropriate to confirm the suspected diagnosis?
Answer Choices: (A) CT-angiogram of the affected extremity (B) EMG (C) MRI of the brachial plexus (D) MRI of the shoulder (E) Repeat shoulder x-rays
Correct Answer: (D)
Model Answer: (B)
Prediction Rationale:
The presented scenario involves a 64-year-old man who sustained a shoulder injury after a fall and subsequently shows an inability to perform shoulder abduction following the incident. This could suggest nerve injury, specifically to the axillary nerve, which responsible for innervating the deltoid musele, crucial for shoulder abduction. Here's the step-by-step breakdown:
1. Injury and Reduction: The man had shoulder injury, likely dislocation, based on initial x-rays and description. The reduction was successful, as indicated by follow-up x-rays.
2. Symptoms Post-Reduction: After 10 days, the patient demonstrates an inability to abduct the shoulder.
3. Common Cause of Inability to Abduct: Injury to the axillary nerve can cause an inability to abduct the shoulder,
as it innervates the deltoid muscle.
Most Appropriate Diagnostic Test: Electromyography (EMG) often used assess the function of muscles and nerves. If there is suspicion of nerve injury, such the axillary nerve in this case, EMG would be helpful in confirming nerve dysfumction damage.
Considering the options: (A) CT-angiogram of the affected extremity would assess vascular issues, not nerve. (B) EMG would assess nerve injury. (C) MRI of the brachial plexus could be used for structural visualization of the nerve, but isn't typically the first choice for function assessment.
(D) MRI the shoulder focuses on soft tissue and structural injury, not specific nerve function. (E) Repeat shoulder x-rays assess bone placement, not nerve or muscle function.
Based on this analysis, the most appropriate diagnostic test to confirm the suspected axillary nerve injury would be (B) EMG.
Analysis:
Upon approaching this question, the model correctly analyzed the provided X-ray images to determine that reduction for the patient had been successful and that X-rays showed no bone injuries. However, it proceeded to conclude that injury to the axillary nerve must have caused the patient's 's inability to abduct the shoulder, and followed this reasoning process to select the corresponding test, (B) EMG. Although the model was correct stating that axillary nerve injuries can impact the deltoid muscle, demonstrating correct medical knowledge, it does not sufficiently compare the likelihoods of several candidate diagnoses. Axillary nerve injury is relatively unlikely to fully account for the severe shoulder injury reported, and in comparison, a rotator cutoff tear is likely to occur in this demographic, and also better explains the patient's 's symptoms. This conclusion leady to the correct diagnostic test, (D) MRI of the shoulder. This case reveals that the model followed the first plausible explanation found and could not perform the nuanced reasoning necessary for weighing several possibilities. Multi-step reasoning is notable challenge in this question- the answerer needs first determine the most likely hypothetical diagnosis, then select the medical examination accordingly. The candidate diagnoses not being explicitly listed likely contributed to the model's oversimplifidd reasoning process.
18

---

MedXper(QA: Benchmarking Expert-Level Medical Reasoning and Understanding
Pereeptual Error
Question: A 57-year-old man is being monitored in the cardiothoracic intensive care unit following an uncomplicated coronary artery bypass graft surgery. During overnight monitoring, the patient's telemetry alerts staff tachyarrhythmia, prompting rapid re- sponse call. His vital signs show: temperature 37.0%C (98.69F), heart rate 140/min, respiratory rate 14/min, and blood pressure 134/72 mmHg. The patient re- ports only post-surgical chest pain at the incision site. Given the electrocardiogram findings shown below and the patient's clinical presentation, what is the most probable diagnosis?
Answer Choices: (A) Atrial flutter (B) Multifocal atrial tachycardia (C) Atrial fibrillation with rapid
ventricular response (D) Paroxysmal supraventricular
tachycardia (E) Sinus tachycardia
Correct Answer: (A)
Model Answer: (C)
Prediction Rationale:
To determine the most probable diagnosis, need assess the clinical information alongside the ECG findings.
1. Heart Rate:
The patient has heart rate of 140 beats minute, which indicates tachycardia.
2. ECG Analysis: The ECG shows an irregularly irregular rhythm. There are no distinct waves visible before each QRS complex; instead, there a disorganized electrical activity, which is typical of fibrillatory waves.
3. Differential Diagnosis: (A) Atrial flutter: Typically gives a "saw-tooth" pattern and usually has a more regular rhythm. (B) Multifocal atrial tachycardia (MAT): Characterized by an irregular rhythm with least three different wave morphologies which isn't I apparent here. (C) Atrial fibrillation with rapid ventricular response: Characterized by an irregularly irregular rhythm without distinct P waves, which fits the ECG findings.
(D) Paroxysmal supraven- tricular tachycardia (PSVT): Usually presents with regular rhythm and wouldn't have the irregular characteristies seen in this case. (E) Sinus tachycardia: Would have regular rhythm and identifiable P waves.
4. Post-Surgical Context: Atrial fibrillation (AF) is relatively common after cardiac surgery, such as coronary artery bypass grafting. due to inflammation, electrolyte imbalances, and other stressors on the heart.
Considering the irregularly irregular rhythm and absence of distinct waves, the most probable diagnosis is: (C)
Atrial fibrillation with rapid ventricular response.
Analysis:
The provided ECG demonstrates characteristic flutter waves with a 3:1 and 4:1 AV nodal block, indicative of (A) atrial flutter. The model correctly states that atrial flutter typically gives a "saw-tooth" pattern, but fails to determine that such patterns are present in the image. Instead, it incorrectly interprets the ECG as having an irregularly irregular rhythm indicative of (C) atrial fibrillation. The main distinction between these two diagnoses the regularity of atrial activations seen the ECG, detailed piece of visual information that challenging for models accurately interpret.
19

---

MedXperiQA: Benchmarking Expert-Level Medical Reasoning and Understanding
Question Understanding Error
Question: A 60-year-old male with multiple comorbidities including coronary artery disease (status post stent placement months ago), hypertension, diabetes, and mild to moderate asthma presents with a two-day history of cough and wheeze. His medication regimen includes aspirin, atorvastatin, atenolol, metformin, and hydrochloroth- iazide. He recently added salbutamol inhaler and prednisolone for his respiratory symptoms. The patient reports experiencing occasional low blood glucose readings the past few weeks. Physical examination reveals stable vital signs with no respiratory distress. Which medication in his current regimen is the most likely cause of his hypoglycemic episodes?
Answer Choices: (A) Aspirin (B) lbuprofen (C) Enalapril (D) Naproxen (E) Atorvastatin (F) Prednisolone (G)
Atenolol (H) Hydrochlorothiazide (I) Phenformin (J) Metformin
Correct Answer: (A)
Model Answer: (G)
Prediction Rationale:
When analyzing this case, need to identify which medication in the patient's regimen is the most likely cause of his hypoglycemic episodes. Here is step-by-step approach:
Step 1: Review the patient's medications: 1. Aspirin Antiplatelet therapy. 2. Atorvastatin - Lipidd-owering agent (statin). 3. Atenolol- A beta-blocker used for hypertension. 4. Metformin - A first-line diabetes medication. 5. Hydrochlorothiazide - diuretic used for blood pressure control. 6. Salbutamol (recently added) - A short-acting beta-agonist for asthma. Prednisolone (recently added) A corticosteroid for respiratory inflammatory issues. Step Focus on the patient's problem hypoglycemia: The patient reports low blood glucose readings. Hypo- glycemia can be caused by certain medications, especially those that: 1. Lower blood glucose directly (e.g., diabetes medications). 2. Affect glucose metabolism indirectly.
Step 3: Evaluate each medication for its relationship to hypoglycemia
Aspirin: At high doses, can rarely cause hypoglycemia. However, at low doses (commonly used for antiplatelet therapy), this is unlikely. Not the most likely cause.
Ibuprofen and Naproxen (if taken, not explicitly listed in the regimen): NSAIDs may not directly relevant to hypoglycemia unless used at high doses affecting insulin function. Likely not relevant here. Enalapril: ACE inhibitors can rarely cause hypoglycemia by increasing insulin sensitivity. However, this medication not listed in his current regimen. Irrelevant.
Metformin: Unlikely to cause hypoglycemia as it does not stimulate insulin secretion. Its primary effect is reducing hepatic gluconeogenesis. Not the cause.
Atorvastatin: Statins are not associated with significant risks of hypoglycemia. Not the cause.
Prednisolone: Corticosteroids typically increase blood glucose (hyperglycemia), not decrease it. Not the cause. Atenolol: Beta-blockers particularly con-cardioeeeecttve ones, can mask the symptoms of hypoglycemia (e.g.. tachycardia) and impair the response hypoglycemia (e-g.. by reducing glycogenolysis). Cardioselective beta- blockers, such atenolol, are less likely to do this but can still blunt the counterregulatory response during hypoglycemia. Possible contributor.
Hydnechlooothiziide: Thiazides are more typically associated with hyperglycemia. Unlikely to cause hypoglycemia. Step 4: Consider the most likely cause: The most likely cause the hypoglycemia is Atenolol because: 1. Beta- blockers impair the body's natural response to a drop in glucose by reducing glycogenolysis and gluconeogenesis. They also mask typical warning signs of hypoglycemia (e.g., palpitations or tremors), which may lead to unrecognized or prolonged hypoglycemic episodes.
Analysis:
The above response fails to fully grasp the question's 's implications in several ways. For the incorrect answer (G) Atenolol, the model generally states that some beta-blockers reduce glycogenolysis and gluconeogenesis, but does not consider the specificity of atenolol - as a selective beta-1 blocker, has less effect on glucose metabolism. While the claim that it masks warning signs of hypoglycemii more accurate, this does not align with the question' emphasis on finding cause of hypoglycemia. For the correct answer (A) Aspirin, the model's 'ss judgment is only accurate regarding non-diabetic individuals, while ample evidence supports aspirin' role in decreasing glucose levels in diabetic patients. The remaining parts of the model's 's analysis similarly disregard the context of the pulient's is diabetes. The response is therefore inconsistent with the patient's 's general background presented in the question.
20

---

MedXper(QA: Benchmarking Expert-Level Medical Reasoning and Understanding
Lack of Medical Knowledge
Question: A 69-year-old male visits his primary care physician for a routine checkup. He maintains an active lifestyle since retirement, engaging in daily exercise and home projects, and follows a balanced diet with regular evening time with his spouse. However, he reports feeling less refreshed upon waking compared to his younger years, when he slept hours and felt excellent, versus his current 7 hours of sleep. His wife reports that his sleep appears peaceful. His medical conditions include hypertension and diabetes, managed with lisinopril and metformin. Physical examination reveals: temperature 98.6 (379C), blood pressure 13 1/85 mmHg, pulse 7 71/min, respirations 12/min, normal neurological findings, good muscle tone, healthy weight, and pleasant affect. He denies current fatigue or tiredness. Which physiological alteration would most likely be observed in this patient?
Answer Choices: (A) Reduced acetylcholine (B) Increased serotonin (C) Decreased orexin (D) Altered circadian
rhythm proteins (E) Elevated histamine levels (F) Decreased melatonin (G) Elevated cortisol levels (H) Increased dopamine (I) Decreased growth hormone (J) Increased adenosine
Correct Answer:
Model Answer: (1)
Prediction Rationale:
This scenario describes an older adult experiencing reduced restorative sleep, which is a common part of the aging process. To determine the most likely physiological alteration, is important to consider how aging impacts sleep. Key details from the case:
Age-related sleep changes: The patient 69 years old. Aging is associated with alterations in sleep architecture, which may lead reductions in total sleep duration, slow-wave (deep) sleep, and sleep efficiency. Older adults often when sleep appears otherwise normal.
feel less refreshed upon waking,
2. Patient's is sleep specifics: He now sleeps 7 hours and wakes feeling less refreshed compared to his younger years when sleeping 9 hours. His sleep appears peaceful, and he experiences no disruptive symptoms such as snoring or awakenings.
3. Ruling out other causes: Good management of hypertension and diabetes, normal vitals, a well-rounded active lifestyle, and no reported fatigue or significant tiredness suggest no acute medical condition contributing to his symptoms.
4. Physiological changes associated with aging: Aging often leads to decrease in growth hormone, which is secreted primarily during slow-wave sleep and stimulates restorative processes during sleep. This can result in less restorative sleep and less refreshed feeling upon waking.
5. Other options to consider: Decreased melatonin (option F): Melatonin levels decline with age, but melatonin primarily regulates sleep initiation and circadian rhythm rather than sleep quality or restoration. Altered circadian rhythm proteins (option D): Circadian changes occur with aging. but his symptoms are more closely linked to sleep restoration rather than timing. Other neurotransmitters like acetylcholine, orexin, serotonin, histamine, cortisol, and adenosine not directly relevant the described problem.
Correct Answer: (I) Decreased growth hormone. This is the most likely physiological change contributing the patient's symptoms, as reduced growth hormone secretion in older adults is linked to decreased restorative properties of sleep.
Analysis:
The question generally describes a patient experiencing reduced sleep quality, commonly associated with aging. The model's response reflects its insufficient medical knowledge on the different effects of (D) decreased growth hormone and (F) decreased melatonin sleep. While both are indeed correlated with aging, growth hormones primarily promotes deep sleep, or slow-wave sleep (SWS), while decreased melatonin more directly accounts for the subjective lack of restfulness upon waking referred to in the question.
21

---

MedXper(QA:
Benchmarking Expert-Level Medical Reasoning and Understanding
G.3. Evaluation Case
System: You are helpful medical assistant.
User: Q: A 57-year-old male presents to the emergency department with a one-week history of weakness and subjective fever. His medical history includes IV drug use, hepatitis C, atrial fibrillation, cirrhosis, alcohol dependence, obesity, and depression. On examination, his temperature 102 (38.9"C) blood pressure 157/98 mmHg, pulse is 100min, respirations are 17/min, and oxygen saturation 98% on room air. The patient appears fatigued with diffuse jaundice. Cardiopulmonary examination reveals bibasilar crackles, and abdominal examination shows distension, dullness to percussion, fluid wave, and generalized tenderness. The patient is admitted to the hospital and started on piperacillin-tazobactan. By day 4, he is afebrile, with a pulse of 92/min, reduced abdominal tenderness, and persistent jaundice. Nurses report bleeding from two peripheral IV sites requiring pressure to control, along with new bruises on the patient's ss arms and legs. What is the most likely explanation for this patient's condition?
Answer Choices: (A) Fibrinolysis activation due to liver failure (B) Vitamin K deficiency due liver dysfunction (C) Diffuse activation of the coagulation cascade (D) Disseminated intravascular coagulation (DIC) (E) Platelet dysfunction secondary sepsis (F) Decreased production of coagulation factors (G)
platelet
destruction (H) Hypersplenism causing thrombocytopenia (I) Decreased metabolism of an anticoagulant (J) Bacterial destruction
A: Let's ' think step by step.
Assistant: Analysis:
This patient has advanced liver disease (cirrhosis) and hospitalized on broad-spectrum antibiotics for four days. Although his infection improving (afebrile, lower heart rate, less abdominal tenderness). he has developed new signs of a bleeding diathesis (bleeding from IV sites, new bruises). In patient with chronic liver disease, an "acute on chronic" coagulopathy like this most often points to additional impairment in clotting-facte synthesis, above and beyond what is already chronically reduced by cirrhosis.
Two common acute drivers of coagulopathy in cirrhotic patients are:
Vitamin K deficiency (especially when on broad- seedd-ppettrum antibiotics that eradicate gut flora responsible for synthesizing vitamin K).
Disseminated intravascular coagulation (DIC), usually in the setting of sepsis or shock, but here the patient's sepsis improving, making DIC less likely.
Thus, the best explanation cirrhotic patient, newly on rroad-spectrum antibioties, who suddenly develops worse bleeding tendencies, vitamin deficiency superimposed his baseline liver dysfunction. Answer: (B) Vitamin K deficiency due liver dysfunction.
User: Therefore, among A through J, the answer is
Assistant: The correct answer is (B) Vitamin K deficiency due to liver dysfunction.
22

---

MedXperiQA: Benchmarking Expert-Level Medical Reasoning and Understanding
H. Expert Review Guideline
GUIDELINES FOR EXPERT REVIEW OF AUGMENTED QUESTIONS.
*Intended Audience:** Human medical expert reviewers
#Objective: Conduct a comprehensive review and proofreading of medical questions rewritten by large language models to ensure that the questions are accurate, clear, scientific, practical, and free from hallucinations factual errors.
## 1. Review of Original Question
= Verify whether the original question contains any logical factual errers, is incomplete, or unanswerable. such issues exist, no further steps should be taken, and the specific situation should be recorded.
#eqquirements: Clearly indicate the prob' ems in the original question. Mark the questicn invalid.
2. Review of Rewritten Question
### 2.1 Content Accuracy Check
**Objective:** Ensure the rewritten question maintains medical knowledge accuracy and is free from factual errors.
**Key Points:** Check whether any key information from the original question is missing.
Verify that no inaccurate or deviated content has been introduced the rewrite. Ensure that no content contradicts medical facts or deviates from the original context.
### 2.2 Language Clarity and Expression Quality Check
Objective: ** The language of the rewritten question should be precise, clear, and professionsl.
**o Points:** Ensure the language is professional and avoids overly casual or colloquial expressions. Confirm that medical terms are used accurately and scientifically,
clcarly conveying the meaning of the original question. Verify that the expression is concise and clear, avoiding arbiguity or unclear phrasing.
#*# 2.3 Logical Consistency Check
* Ob jective:: ** Ensure the question follows a Logical flow, has reasonable structure, and is answerable.
** Points:** Check whether the question fits into real-worln clinical or research context. Look for logical errors such as contradictions, lack of causality, reasoning mistakes, illogical phrasing. Ensure the question structure is clear, complete, coherent, and has clear focus.
### 2.4 Other Issues
Tabbluaa Data:** If the original question includes tabular data, verify that the rewritten question retains the original data's content and format. The rewritten table should not omit any data or alter the format.
Medicaa Terminology Abbreviations: If the rewritten question replaces the full medical
term with abbreviation, verify whether the abbreviation is standard and in line with medical conventions. If professional medical practitioners can understand the abbreviation in the context of the question, it is considered acceptable.
Table 9: Guidelines for expert review of augmented questions (Part 1).
23

---

MedXperiQA: Benchmarking Expert-Level Medical Reasoning and Understanding
GUIDELINES FOR EXPERT REVIEW OF AUGMENTED QUESTIONS.
## 3. Summary Requirements
**Strict Adherence to Standards:** All medical experta must strictly follow these guidelines for evaluation and proofreading to the accuracy, scientific integrity, and practicality the questions.
Quality of Evaluation Results:** The content must be accurate, with reliable facts. The language expression should be clear and meet professional medical standards. The logic should be sound, and the questions should answerable.
Maintain Objectivity:** Maintain objectivity and fairness throughout the evaluation and procfreading process, aveiding subjective judgment or misleading rewrites.
Table 10: Guidelines for expert review of augmented questions (Part 2).
24

---

MedXperiQA: Benchmarking Expert-Level Medical Reasoning and Understanding
GUIDELINES FOR EXPERT REVIEW OF AUGMENTED OPTIONS.
**Intended Audience:** Human medical expert reviewers
= ** Conduct comprehensive review and proofreading of the expanded options generated by Large language models for medical miltiple-choice questions, ensuring the optiens are accurate, clear, reasonable, and challenging.
## 1. Option Inspection Standards
### 1.1 Content Rationality Check
#b Ensure the expanded options are reasonable and acceptable within the context of the original question.
**Y Points:** Avoid adding options that do not align with the question's context. For example, if there are images in the question, generating an option for VImage 5" would be unacceptable. Avoid adding options that do not correspond to the topic of the questicn. For example, if the question asks to choose from several time periods, do not generate options with unrelated content.
## 1.2 Language Expression and Professionalism Check
+bjective::* Ensure the language of the options is accurate, clear, and professional.
**y Points:** The language style of the expanded options should match that of the original options. The language should be professional, avoiding overly casual or
collequial expressions. Medical terminology should be used accurately and in line with current medical standards. The expression should be clear, concise, and avoid ambiguity or overly complex phrasing.
**# 1.3 Hallucination Check
Obeettive: Ensure that the original opti errers.
contain obvi hal Sallucination
Key Points:** ** The original options should not be changed.
The generated options should not contain hallucinated information, i.e., content that is
disccnnected from the original question context lacks scientific basis. There should be no repetition of content between options, and no redundant information in the options.
4. Summary and Requirements
S*strcct Adherence to Standards:** All medical experts must strictly follow these guidelines for evaluatien and proofreading to ensure the accuracy, scientific integrity, and practicality of the options.
Quallity Evaluation Results:** Content must be accurate and reliable. Language expression should be clear and meet professional medical standards. Option design should be logically sound and have a high degree of differentiation.
Maintain Objectivity:** Maintain objectivity and fairness throughout the evaluation and procfreading process, avoiding subjective judgment or mislcading expansions.
Table 11: Guidelines for expert review of augmented options.
25

---

MedXper(QA: Benchmarking Expert-Level Medical Reasoning and Understanding
I. Prompts
The following lists all prompts used this work.
L1. Attribute Annotation Prompts
PROMPT FOR TASK ANNOTATION.
You are an experienced medical doctor and independent practitioner. You will be asked to label medical question according to its task.
You will be given list of tasks and their corresponding explanations and descriptions. You will then be given a medical examination question.
Please determine which task the question best pertains
If the question is related to multiple tasks, only select the most zelevant one. Directly output the task name you have selected. Do not provide any additional explanations. Ensure that it is valid task within the list of available tasks provided.
Tasks:**
Diagnosis:
Tasks that oceur in the diagnosis stage of medical examination. These include identifying diseases based on symptoms, deducing the causes symptoms or diseases, predicting disease prognosis or additional symptoms, and proposing internediate steps (such as further examinations) to facilitate diagnosis.
Treatment: Tasks that occur in the treatment stage of medical examination. These include selecting treatments for diseases, proposing preventive measures for diseases, and suggesting lifestyle changes for the patient.
Basic Medicine: Tasks that focus basic scientific principles that may be applied in medical practice. These include understanding anatomy, diseases and medical procedures, and general STEM principles.
*uesstion**
(questicn_text:
**Output ***
Table 12: Prompt for Task Annotation.
26

---

MedXperiQA: Benchmarking Expert-Level Medical Reasoning and Understanding
PROMPT FOR SUBTASK ANNOTATION.
You are an experienced medical doctor and independent practitioner. You will be asked to **ask:** Diagnosis
label a medical question's subtask according to its task.
You will be given medical examination question and its task type. You will then given list of subtypes within the predefined type. Please deternin which subtask question best pertains to.
If the question is related to multiple subtasks, only select the most relevant one.
Directly output the subtask name you have selected. Do not provide additional explanations. Ensure that it is valid task within the list of available tasks provided.
**Subtasks:**
Primary, Differential, Eticlogic, Prognostic, Retrospective, Syndromic, Predictive, Intermediate
**Question***
(questicn_text)
**Output:**
Table 13: Prompt for Subtask Annotation. We the prompt for diagnosis tasks example. Each question's prompt will contain the specific subtasks within the question's 's labeled task.
PROMPT FOR SYSTEM ANNOTATION.
You are an experienced medical doctor and independent practitioner. Your task will be to label medical question according to the numan body system it corresponds to.
You will be given determine which system the question best pertains to. If the question is related to multiple systems, only select the most relevant one.
human body systems, followed by medical question. Please
Directly cutput the name of the final system you selected from th 1ist of available systems.
+Systems:** ***
Integunentary, Skeletal, Muscular, Nervous, Endocrine, Cardiovascular, Lymphatic, Respiratory, Digestive, Urinary, Reproductive
*uesttion:**
{question_texx
**Output:** ***
Table 14: Prompt for System Annotation.
27

---

MedXperiQA: Benchmarking Expert-Level Medical Reasoning and Understanding
PROMPT FOR REASONING AND UNDERSTANDING ANNOTATION.
You are an experienced medical doctor and independent practitioner. Your task will be to determine whether a medical question primarily challenges the answerer's a medical knowledge understanding or medical reasoning skills.
You will be given challenging ng redical cadical exan question designed for professi onal medical practitioners, followed by its correct answer and explanation to help you understand the question.
Please determine whether the question primarily challenges the answerer's medical knowledge understanding medical reasoning ability.
Consider the amount of information provided in the question, as well as the number and complexity of reasoning steps required to answer it.
You will be given the questien you will label with its answer and explanation, Directly output either "Understanding" or "Reasoning" your answer without any additional infermation or explanations.
*uesstion:**
!questien_text
#Explanation:
(explanaticn_text)
**Output:**
Table 15: Prompt for annotating questions based on their focus on either reasoning or understanding skills, used for constructing reasoning subset of MedXpertQA.
28

---

MedXperiQA: Benchmarking Expert-Level Medical Reasoning and Understanding
1.2. Data Augmentation Prompts
PROMPT FOR QUESTION REPHRASE.
### Task
Rephrase given medical multiple-choice question according to the following guidel nes::
### Suggested Approaches
Consider the following strategies to rephrase the question:
*angguae Rewriting Use different expressions and sentence structures that are equivalent in meaning to the original.
2. **Information Restructuringet:
Alter the order that information is presented in the original question without disrupting the logical coherence of the question.
### General Requirements
1. Maintain the content:** Ensure that the rephrased question retains the content, reasoning logic, and correct
of the original question.
2. **Ensure a professional language style:** Maintain a professional, formal, and clear language style similar to the original question. Rigorously ensure clarity and avoid ambiguity. Feel free copy parts of the original question if alternative appropriate phrasing is not possible.
3. *Maintain factual consistency:** Ensure that the rewritten question retains every piece of information in the original. Do not change, add, or delete any factual information.
4. Immtaate original formatting:** Keep any special formatting in the original question unchanged, especially regarding structured data presentation. Pay special attention to keep any tabular data in completely the format the original.
**Final output format:** Knaure that the options section of the question remains
unchanged and the format remains as "Answer Choices: (A) [Option A] (8) [Option B]
Only output the rephrased question. Do not include any additional information or explanations.
(dmmnsstratonn]
### Input
*Original Question:**
(question)
**Correct Answer:**
(label)
#xxplanation:**
The follewing an explanation to help with understanding the question: (explanaticn)
### Output
**eephraeed Question:**
Table 16: Prompt for Question Rephrase.
29

---

MedXperiQA: Benchmarking Expert-Level Medical Reasoning and Understanding
PROMPT FOR OPTION GENERATION.
###
Task
Given a medical multiple-choice question designed to evaluate the capabilities of medical experts and its correct answer, please design additional incorrect options to challenge experts when they answer the question.
#*# Suggested Approaches
1. **Consider Errorneous Perspectives:** Include distractors that interpret key information in the question incorrectly.
2. *#Levergee Common Misconceptions:** Consider designing distractors leveraging common errers or medical concepts that are frequently confused.
3. **Logical :isdirection:** Introduce distractors grounded logical reasoning that seemingly plausible but incorrect.
### General Requirements
1. **Mantain Consistency:** Ensure that the generated new options match the original options in terms of length, structure, word count, and granmatical form. They should be clear, concise, and professionally worded.
*voii oversimplified Distractors:** not include options that can be easily dismissed based on intuition or surface-level analysis.
3. **Ensure High Plausibility:** Maintain the plausibility of each generated option. Avoid opticns that overtly illogical unsupported.
4. **Final Format: ** Present the original question and options, followed by the (ggeeraate_nmm additional options. Ensure that the generated options fol the same format as the original: "Answer Choi ces: (A). [Option A] (B) [Option B] .. Do not cutput anything after the options.
[demesstaatons)]
## Input
#rigginal Question:**
(questien)
**Correct Answer:**
(label)
### Output
(generetee__um options appended after the original question:
Table 17: Prompt for Option Generation.
30


Reviews:
[{"review_id": "sX1o1g0jcT", "reviewer": "Reviewer_Rrtx", "summary": "In this work, the authors contribute a new synthetic dataset for the evaluation of medical reasoning of large language models (LLM), and the newest models in this class, also called large reasoning models (LRM). The creation of the dataset follows several steps that are well described, to ensure the benchmarking tasks are varied, of sufficient difficulty and do not suffer from data leakage with the training data of the models. As the authors outline, it is very important to shape the benchmarks properly as they, in turn, shape the model development.\n\n## update after rebuttal\nI have slightly increased my grade after the discussion period, but some important points have not been addressed by the authors, in particular a comment about the relatively low human performances on their benchmark, a comment about the need of the different steps based on the number of questions filtered out at each step, and the measure of any performance bias for different genders for the models present in the benchmark. The question of the question difficulty is key to the relevance of this benchmark, as the authors claim that one of the advantages of this benchmark is its difficulty, but it may be related to impossible questions, which undermines the relevance of the difficulty.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We hope that our clarifications fully address your concerns!\n# Response 1 - Claims And Evidence\n### **1.1 Target Audience**\n- We agree with the value of expert insight. We worked closely with medical practitioners when designing and reviewing MedXpertQA.\n- Audience from an AI background is irreplaceable. Since the goal of MedXpertQA is to help researchers understand and improve model limitations, our most important target audience is medical AI researchers.\n### **1.2 Real-World Representation**\nPlease see Response 2.2 to Reviewer gh7D.\n### **1.3 Concern on Patient Data Collection**\n- This is only one type of question in MedXpertQA, whose subtasks span different stages of medical decision-making (Figure 3). It also contains questions requiring data processing and organizing: https://postimg.cc/ZCVYzGXs.\n- We acknowledge that MedXpertQA doesn't explicitly model the multi-step process of single patients. This is an inherent limitation of MCQA, and relevant works, e.g. MedChain [1], tackle the issue. Nevertheless, MedXpertQA's diverse task coverage ensures that it adequately tests wide-ranging capabilities.\n- Finally, patient data collection may not be most challenging. Poor results of sota models reflect the difficulty of downstream clinical decision-making, the focus of MedXpertQA.\n# Response 2 - Methods And Evaluation Criteria\n### **2.1 Benchmark Construction Details**\nWe've attempted to present the construction process concretely and in detail. We provide metric formulas, hyperparameters, etc. This level of detail exceeds many prominent benchmarks, e.g. MMLU-Pro, MMMU. If any aspect remains unclear, we can provide further clarification.\n- Unfortunately, we are unable to provide the initial dataset distribution, as labeling 37k questions for multiple attributes would be too costly. We appreciate your understanding.\n- Statistics on questions remaining after each step: [Dataset Filtering Stats](https://postimg.cc/zVggJc4r).\n- Our dataset construction is tailored to the medical domain, thus not intended for direct application to general AI. Our main contribution is the benchmark itself, especially MM, which fills crucial gaps in current medical multimodal evals.\n### **2.2 Human Performance Evaluation**\nWe already have some preliminary human performance results: https://postimg.cc/fkkJCjy0. The details:\n- **Expert (Junior) Score**: Response distributions collected from question sources mostly come from medical students and reflect human performance. Rewriting has little impact on human performance, since it retains question information. Thus, human performance on original questions can be compared with final model results. The large response number of each question (up to 238k) makes these stats highly representative. Fewer than 200 final questions lack response data, and we'll hire humans of similar caliber to answer them and complete the human performance evaluation. We'll incorporate this information before 4.8 AoE.\n- **Expert (Senior) Score**: We will assess human experts with medical licenses/MDs for a separate expert performance measurement. These experiments will take longer and will be added in future versions.\n### **2.3 Model Biases**\n- MedXpertQA primarily aligns with practices from the US. We'll cover this in the revised paper.\n- Model bias analysis, while important, exceeds the scope of this work. Bias mitigation should be conducted by model developers, and works such as [2,3,4] would be better references for practitioners interested in these issues.\n- We provide MedXpertQA's coverage of patient demographics. Gender (from keyword matching):\n    - **Text**: Male 1025, Female 903\n    - **MM**: Male 874, Female 550\n    - **Total**: Male 1899, Female 1453\n\nFor ethnicity, we re-examined 100 questions and found no mentions.\n# Response 3 - Experimental Designs Or Analyses\nSee Response 2 (R2) for stats.\n\nTable D was from expert evaluations. **Experts not only reviewed each question, but also did multiple rounds of editing.**\n\nFigure 5 results for all models: https://www.hostize.com/zh/v/TH3ddN2Bz3.\n# Response 4 - Essential References Not Discussed\nPlease see R2 to Reviewer vM2L.\n\nFor MedXpertQA specifically, an interesting result is the noticeable gap between R1 and o1, which would be unexpected if we directly extrapolated from other reasoning benchmarks.\n# Response 5 - Questions For Authors\n1. -> R2.1\n2. -> R2.2\n3. Figure 4 requires data on paired backbone LLMs and LRMs, thus can't be done for all models. Please refer to the main results table for results on more models.\n4. -> R3\n5. -> R2.3\n6. -> R4\n7. -> R1.1-1.3 MedXpertQA's main goal is evaluating models' fundamental medical abilities. These abilities provide crucial support for clinical use, but downstream applications are not our focus.\n# References:\n\n[1] https://arxiv.org/abs/2412.01605\n\n[2] https://www.nature.com/articles/s41591-024-03113-4\n\n[3] https://www.nature.com/articles/s41746-025-01503-7\n\n[4] https://www.nature.com/articles/s43856-024-00601-z", "claims_and_evidence": "The authors claim that their benchmark is of adequate difficulty and robust, and provides a satisfying evaluation of the models, which seems well supported by evidence. However, the claim that the benchmark reflects real-world clinical data and scenarios is more difficult to prove (as a writer) and to verify (as a reviewer). My main concerns is that the audience of this conference has an extremely limited medical background, and the existence of a benchmark has the power to drift the orientation of future research for medical informatics, so the audience of this paper, to further discuss the content of the benchmark and the relevance of the medical tasks should include medical doctors. A journal of medical informatics might be better suited for this task. That being said, providing benchmarks with tasks from other domains is valuable for the improvement of future developed models, but the claims of relevance and usefulness for the domain of origin should be attenuated in the absence of a thorough multi-disciplinary discussion.\n\nOne concern that I have regarding relevance would be that the state of the patient is provided as a clean summary narrative to the model, which already incorporates the doctor's work to gather all relevant information. It's possible that this preliminary data selection and organization is the most difficult task for the doctor.", "methods_and_evaluation_criteria": "The main evaluation criterion for the relevance of the benchmark is the performances of the different tested models on this benchmark, which seems to exhibit a good discrimination ability between the different models. There is also a series of selection criteria involved through the different steps of the benchmark construction, however there are simply described, and it would be very informative to provide the distribution of metrics across all the 37k initial questions, the number of filtered questions at each step, and the filtering threshold with respect to the whole distribution, to better grasp which filtering steps have been the most crucial, as it could inform the construction of future similar benchmark, in the medical field of other fields. \n\nDetailing more the article as a good methodology to create a benchmark would be a way to make it more adapted to this conference's audience rather than to a medical informatics journal if that is the authors' wish. The authors have actually relied on similar information from previous works to assess the data leakage, illustrating the relevance of a more thorough reporting of all the steps of their work in details.\n\nIt would be also extremely insightful as at some of those steps, there is a human performance assessment. This performance assessment is not on the complete final benchmark, but can be a very good proxy of human performance. Maybe it would be worth reporting the 16 models' performances at this intermediate step to provide a comparison, even if this intermediate performance should be taken with cautious as it occurs before the mitigation of data leakage step.\n\nIn the impact statement, the authors mention the importance of ethical concerns, mostly with respect to medical data privacy, however there are well-known issues with models biases. It could be relevant to assess the performances of the different models for different categories of patients. Gender seems to be mentioned in most medical questions. How about ethnicity? Also in this line of thought, the mention that this benchmark is in English language and might be tailored to US (or English-speaking country) medicine could be discussed.", "theoretical_claims": "no theoretical claims.", "experimental_designs_or_analyses": "As mentioned before, a lot of relevant \"experimental\" results of evaluation of the benchmark through all the steps are not reported, which is a major issue of this article, except for the table D in Supplementary that does not mention the number of questions evaluated at this step, only the number of questions flagged by the experts (unless I missed the info in the text, but it should be mentioned again with the table). Figures like figure 4 for all models should be reported in the supplementary as well.\n\nFigure 5 reports interesting results of performance variation with respect to the medical specialty. Do the other models have the same pattern? how about human performances assessed in step 2?", "relation_to_broader_scientific_literature": "This article brings a new evaluation benchmark for LLMs and LRMs. The main novelty of this paper seems to be the discrimination power for method evaluation, however, it is not clear that this line of tasks is relevant for actual clinical practice, though it might be interesting to evaluate a different aspect of model reasoning in general.", "essential_references_not_discussed": "It would be interesting to discuss and cite relevant literature comparing the reasoning ability of those models in different domains to see if there is a specificity to the medical domain, or said otherwise, does the evaluation of this medical reasoning task provide a different ranking of the models compared to other reasoning benchmarks?", "other_strengths_and_weaknesses": "The provided dataset seems more diverse and thorough than previously existing benchmarks", "other_comments_or_suggestions": "there is a typo MMMU p. 5", "questions_for_authors": "I have disseminated questions throughout the review. I attempt a summary of the main ones here, but more details and context is given in the previous sections\n1) provide the distribution of metrics across all the 37k initial questions, the number of filtered questions at each step, and the filtering threshold with respect to the whole distribution, to better grasp which filtering steps have been the most crucial, as it could inform the construction of future similar benchmark, in the medical field of other fields.  (addressed in the rebuttals, but no discussion about which steps are important and why)\n2) report human performances (and model performances on the benchmark at this step) (done for a junior doctor, not yet for a senior doctor, the overall performance would need further discussion as human performances are low. Are the questions unsolvable because of a lack of information???? that would importantly affect the relevance of the benchmark if half of the questions are actually not sovable)\n3) figure 4 for all models - addressed when possible (with the table)\n4) figure 5 for all models + human performances - addressed\n5) variations of performances for gender, and ethnicity if/when available (not addressed so far in the rebuttal, the authors have reported the number of questions for male and female patients, but not the associated performances)\n6) change of model ranking with respect to other reasoning tasks in the literature (to show the relevance of yet another LLM benchmark)\n7) discussion of the actual relevance for example in routine clinical use", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N56iS7aPNK", "reviewer": "Reviewer_y9fU", "summary": "The paper introduces MedXpertQA, a novel and challenging benchmark for evaluating expert-level medical knowledge and reasoning. MedXpertQA consists of 4,460 questions covering 17 medical specialties and 11 body systems, divided into text-based (Text) and multimodal (MM) subsets. The authors employed a rigorous methodology for benchmark construction, including extensive filtering based on both AI and human expert evaluations, data synthesis to mitigate leakage risks, and multiple rounds of expert review to ensure accuracy. Additionally, they developed a reasoning-oriented subset to facilitate the assessment of advanced reasoning capabilities in medical AI models. The authors evaluate 16 leading language and multimodal models on MedXpertQA, demonstrating that current state-of-the-art models still face significant challenges in expert-level medical reasoning tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful comments!\n# Response 1 - Claims And Evidence\n### **1.1 Leakage Prevention**\nFirst, we note that data leakage prevention is an extra precaution we took on top of our main contribution, a challenging, clinically relevant benchmark. Models' subpar performance on MedXpertQA already reflects that its questions haven't been well learned during pretraining.\n\nOur literature review found no effective method for reducing leakage risk of benchmarks, so we used the intuitive LLM rewriting strategy. We detail our extensive efforts here: https://postimg.cc/D4w1hw7g.\n\nChanging questions drastically tends to lower their quality and relevance, and our method balances general quality and low leakage risk. Our method exceeds \"simple paraphrasing\" - it combines meticulously designed instructions with strict multi-round human reviews and error correction (AppendixD).\n### **1.2 Leakage Risk Evaluation**\n- We intend to solidly show MedXpertQA's low leakage risk, not devise a new method.\n- We highlight the validity of the metrics we used (Please reply for further clarification. Thanks!).\n- No previous publication in medical AI benchmarking covered this.\n- A recent work compared leakage risks of different benchmarks, showing the superiority of MedXpertQA: https://postimg.cc/yDrvtTB3.\n### **1.3 Reasoning-Oriented Evaluation**\n- We initially considered having experts label Reasoning (*R*)/Understanding (*U*), but we realized that this task is quite straightforward for LLMs given clear prompt guidelines. We also provide expert-written answers and explanations collected from sources (Table13). This dense guidance enables a simplified form of annotation under expert supervision.\n- We sufficiently considered the distinction between reasoning complexity and general difficulty when designing our labeling prompt (Table13).\n- Human reviewing: For 10% sampled questions (Text-490, MM-400), reviewers found 28 and 11 questions incorrectly labeled as *R*. We think this error rate (~4.3%) is acceptable.\n- Empirical results (Section 5.2) reflect the validity of annotations. LRMs perform much better than their backbones on *R*, and this does not hold on *U*. We even note the opposite trend for Qwen-series models on *U*, which would not hold if *U* questions were merely easier. For single models, *R* scores aren't consistently lower than *U*.\n# Response 2 - Methods And Evaluation Criteria\n### **2.1 Sequential Reasoning**\nTwo of our question sources included many sequential questions. We found that each question had sufficient context and wasn't dependent on answering previous ones, so we used individual questions instead of preserving the sequence. Though MedXpertQA dissects the multistep process into separate questions, its coverage of all stages of clinical decision-making (Figure3) ensures that it tests wide-ranging abilities needed for realistic multistep tasks. Recent works, e.g. MedChain[1], have focused on this, and we'll add discussions of these works in the next version of the paper.\n### **2.2 Statistics on Individual Filtering Steps**\nPlease see Response 2.1 to Reviewer Rrtx\n### **2.3 Difficulty Quantification and Calibration**\nWe combine two metrics: https://postimg.cc/RNrP4tWF.\n# Response 3 - Experimental Designs Or Analyses\nPlease see Response 1\n# Response 4 - Essential References Not Discussed\nWe'll add these works in the next version, omitting it here due to space limitations.\n# Response 5 - Other Strengths And Weaknesses\n**1 & 2:** Please see Response 1\n\n**3:** We show examples in F.1, F.3 (with human analyses of model errors). Models handle complex clinical information and make nuanced decisions between different possibilities. In addition, fine-grained analysis of model reasoning will be more informative once models achieve better performance.\n# Response 6 - Other Comments Or Suggestions\n- Please see Response 2.2 to Reviewer Rrtx\n- Valuable suggestion! We will consider adding it in the next version.\n- In Page 4 paragraph 2, we used GPT-4o-2024-11-20. We'll clarify this in revisions.\n# Response 7 - Questions For Authors\n1. Ground truth answers were collected from question sources, whose QA pairs were designed by medical experts.\n2. Difficulty: -> Response 2.3\n3. Leakage Prevention: -> Response 1.1\n4. Risk Evaluation: -> Response 1.2\n5. Sequential decision-making: -> Response 2.1\n6. While MCQA is not tailored for interactive chat models, MedXpertQA covers diverse tasks, some of which touch on clinical questioning: https://postimg.cc/ZCVYzGXs.\n7. Reasoning: -> Response 1.3\n---\nWhile responding to your comments, we noticed aspects where our current paper did not sufficiently reflect our efforts. That being said, we believe the drastic improvement in benchmark difficulty is a convincing indication of our extensive efforts. We hope our response addresses your concerns and look forward to incorporating relevant information into the paper.\n### **References:**\n[1] https://arxiv.org/abs/2412.01605", "claims_and_evidence": "The authors clearly state the limitations of existing medical benchmarks and provide detailed justification for the development of MedXpertQA: \n- Statistics on the coverage of the benchmark across specialties, body systems, and task types\n- Explanation of data collection, filtering, and quality assurance processes\n- Evaluation of 16 models and quantitative analysis demonstrating the benchmark's difficulty level compared to existing benchmarks \n\nThe clinical relevance claim is supported by including questions from 17 medical specialty board exams, though additional validation with practicing clinicians would further strengthen this claim.\n\nTwo claims that could benefit from stronger evidence:\n- Data leakage prevention: The authors' approach to preventing data leakage primarily on having LLMs \"rephrase the question through alternative expressions or structural adjustments while preserving all original information.\" This simple paraphrasing strategy is unlikely to be effective against modern LLMs that can recognize semantic equivalence despite surface-level rewording. The metrics in Table 5 (perplexity and n-gram similarity) may not adequately capture whether models have seen semantically equivalent content during training. More rigorous methods and analysis would be needed to substantiate this claim.\n- Reasoning-oriented evaluation: The distinction between reasoning and understanding questions was made using GPT-4o, but the paper would benefit from clearer operational definitions and validation of these categorizations by medical experts. The paper does not sufficiently demonstrate that the \"reasoning\" subset genuinely requires medical reasoning rather than just being more difficult questions.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate and well-motivated for the problem at hand. The authors' approach to benchmark creation is methodical, involving (1) data collection from authoritative medical sources, including USMLE, COMLEX-USA, specialty board exams, and image-rich sources; (2) filtering using both AI expert filtering and human expert filtering (3) similarity filtering to ensure diversity and remove redundant questions; (4) question and option augmentation to mitigate data leakage and increase difficulty and (5) expert review. \nThe evaluation metrics (accuracy) and zero-shot CoT prompting method are standard and appropriate for this task. The creation of distinct reasoning and understanding subsets enables a more nuanced evaluation of models' capabilities.\nSome areas that can be improved: \n- no explicit discussion is provided on realistic patient contexts—would MedXpertQA be applicable to realistic scenarios requiring sequential diagnostic reasoning in medical records?\n- no ablation study is presented to measure the impact of individual filtering steps. Understanding how much each filtering stage contributes to difficulty would be beneficial.\n- The paper would benefit from more detail on how humans quantified the difficulty, and finer-grained analysis of human annotations across different medical specialties would add real value for diverse specialties.", "theoretical_claims": "The paper makes limited theoretical claims, focusing primarily on empirical evaluation. The authors' claim about the relationship between inference-time scaling and medical reasoning capabilities is supported by their experimental results.", "experimental_designs_or_analyses": "The experimental designs and analyses are well-executed. The authors evaluate a diverse set of 16 models ranging from proprietary to open-source, including both vanilla models and inference-time scaled versions. The zero-shot CoT prompting approach is appropriate for the evaluation. The breakdown of performance by task type (reasoning vs. understanding) and by the medical system provides valuable insights into model capabilities and limitations. Below are some points that can be improved: \n- The data leakage analysis methodology using perplexity and n-gram-based metrics to assess potential memorization can potentially be a main issue for the realistic usage of the constructed benchmark. \n- Further analysis is needed to determine whether MedXpertQA’s reasoning subset accurately captures distinct clinical reasoning.", "relation_to_broader_scientific_literature": "The authors position MedXpertQA effectively within the broader landscape of medical benchmarks. They provide a comprehensive comparison with existing text-based benchmarks (PubMedQA, MedQA, MedMCQA, MMLU) and multimodal medical benchmarks (VQA-RAD, VQA-Med, Path-VQA, SLAKE-En, PMC-VQA, OmniMedVQA, GMAI-MMBench, MMMU), highlighting key differences in terms of complexity, clinical relevance, and diversity.", "essential_references_not_discussed": "The paper has a good coverage of relevant literature. However, a few potentially relevant works not discussed include:\n- Relevant research on medical reasoning benchmarks\n- Recent work on medical AI evaluation and how such benchmarks might eventually be to understand LLMs' practical real-life usage.", "other_strengths_and_weaknesses": "Strengths:\n- The benchmark focuses on expert-level questions with filtering and augmentation, addressing the insufficient difficulty of existing benchmarks like MedQA and providing new evaluation datasets for recent models. \n- Developing a reasoning-oriented subset demonstrates the recognition that medicine provides a rich context for assessing complex reasoning. \n- By incorporating questions from 17 medical specialty board exams, MedXpertQA achieves clinical relevance and specialization diversity compared with previous benchmarks.\n\nWeaknesses:\n- Inadequate Data Leakage Prevention: The paper's approach to preventing data leakage relies primarily on simple paraphrasing (\"rephrase the question through alternative expressions or structural adjustments\"). This strategy is likely insufficient against modern LLMs that can recognize semantically equivalent content despite surface-level rewording. More sophisticated techniques would be needed to genuinely prevent the leakage of medical exam questions that may already be in training data.\n- While the reasoning vs. understanding categorization is valuable, the paper lacks clear operational definitions of different reasoning types in medicine and relies on GPT-4o for these annotations, which could be a concern on the validity. \n- Unclear validation of reasoning - do models exhibit clinically meaningful reasoning or just perform well on structured multiple-choice formats?", "other_comments_or_suggestions": "- The benchmark would benefit from establishing human performance baselines on the released benchmark, which would provide valuable context for evaluating model performance and validating the difficulty levels.\n- Consider including confidence calibration analysis for the evaluated models, as this is particularly important in high-stakes medical domains.\n- Minor typos: Page 4, paragraph 2: \"we instruct an LLM to annotate each question with its most relevant human body system\" - It's unclear which LLM was used", "questions_for_authors": "- The paper mentions that MedXpertQA includes questions from 17 American specialty board exams. Could you clarify if the ground truth responses for the questions are also collected by rich sourcing? Are there expert annotations involved for quality control on responses?\n- What criteria did you use to quantify \"difficulty\" in medical questions, and how did you calibrate these metrics across different humans to ensure the measurement of challenge levels is consistent?\n- The approach to preventing data leakage relies primarily on paraphrasing (\"rephrase the question through alternative expressions or structural adjustments\"). Given that modern LLMs can recognize semantically equivalent content despite rewording, why do you believe this strategy is sufficient? Did you consider a verification or test with more extensive modifications that might more effectively prevent recognition?\n- The metrics in Table 5 (perplexity and n-gram similarity) may not fully capture whether models have seen semantically equivalent content during training. Did you explore alternative methods to evaluate potential leakage?\n- Do any MedXpertQA questions require decision-making over time (e.g., follow-up management after an initial diagnosis)? Would adding a longitudinal subset improve clinical realism?\n- Since MedXpertQA is largely based on medical exams, have you considered incorporating non-medical exams to improve generalizability? Could MedXpertQA be used to evaluate interactive AI models that engage in back-and-forth questioning, mimicking real physician-patient interactions?\n- How do you determine whether a question requires deep multi-step reasoning versus factual recall?\n\nHappy to adjust scores if those major concerns can be solved.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7QkgI9fp1i", "reviewer": "Reviewer_gh7D", "summary": "Authors present a new expert-level knowledge and reasoning benchmark for real-world clinical scenarios. It seems to be the largest Multi-modal dataset in this category (with human annotations). It also is the second largest in the text only category. There seems to be a new barrier in this particular task with this new dataset for the most common LMM models.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you very much for your recognition of our work and your valuable suggestions!**\n\n# Response 1 - Other Strengths And Weaknesses\n**You make a great point. Please refer to Response 1.2 (Specialist Model Results) to Reviewer vM2L. Thank you for your understanding!**\n\n# Response 2 - Other Comments Or Suggestions\n### **2.1 o1 Performance on USMLE**\n\n- We want to clarify that the difficulty of MedXpertQA is not due to additional question sources or countries. During dataset construction, we found USMLE questions to be the most difficult, even compared to specialist board exams. Specialty assessments were included to enhance clinical relevance, not to increase difficulty.\n\n- To explain your observation of o1's high scores: USMLE questions in our benchmark and those in MedQA [1] are markedly different. USMLE does not publicize exam questions, so \"USMLE questions\" usually refer to mock questions devised by third-party experts, with varying difficulties. For MedXpertQA, we collected questions from high-quality, diverse sources and performed rigorous dataset construction steps, e.g. question filtering. This ensured the difficulty of MedXpertQA, accounting for o1’s lower score compared to other benchmarks.\n\nIn summary, the difficulty of MedXpertQA is primarily due to the dataset construction process.\n\nNote: EDiR questions only make up a small percentage of MedXpertQA, and the large majority of questions still represent US exams.\n\n### **2.2 Real-World Representative**\n\n**Point 1:** First, we would like to emphasize that even if exam questions do not completely simulate real-world clinical scenarios, this is **a common issue faced by all existing medical benchmarks** (every baseline in Table 1, 2), not a new challenge introduced by our work. However, these benchmarks still represent the most important and impactful way to evaluate medical AI. For example, MedQA has already been cited over 100 times this year alone and has been used to evaluate frontier medical AI models such as MedPaLM 2 [2] and Med-Gemini [3]. We believe MedXpertQA can greatly contribute to medical AI progress, and that its impact is amplified by the widespread use of similar, yet less clinically relevant alternatives.\n\nMoreover, your concerns are totally valid - we are aware of some works calling into question whether exam questions are real-world representatives. However, full representation of clinical relevance seems to be an elusive and distant topic that no existing benchmark can achieve, nor is it a claim of our work. We intend to convey in good faith that new work should be recognized primarily for the significant improvements it introduces over existing works, rather than for pursuing overarching goals.\n\n**Point 2:** Secondly, we need to clarify that we do NOT claim that MedXpertQA **completely** simulates real-world scenarios. Our claim has always been that it **significantly improve** (in Abstract) clinical relevance compared with previous, widely adopted benchmarks, which we believe is sufficiently proved and represents an important contribution in itself. Compared with previous benchmarks, MedXpertQA improves clinical relevance through fundamental improvements for both subsets:\n\n- For Text, our addition of medical specialist evaluations is certain to improve relevance, since realistic medical tasks are highly specialized and assigned to different departments. A single, general evaluation suite is evidently inadequate for evaluating the full spectrum of clinical tasks. GMAI-MMBench [4] raised a similar claim. Our discussions with medical expert collaborators further verified this point.\n\n- For MM, current benchmarks commonly design surface-level questions without realistic patient information, leading to extremely limited relevance. MedXpertQA's questions were constructed by human experts and intended to demonstrate how well a medical student will likely perform facing real patients. These expert-designed questions far surpass those constructed through fixed templates or automatic LLM generation of existing benchmarks in quality and relevance. The corresponding images are also realistic and diverse.\n\n# Response 3 - Questions for Authors\nAgain, your concerns are reasonable, and we were also worried about this during our work. This is the main reason we did not publicize the sources of our questions. We also note that this issue is ubiquitous for benchmarks across different domains, such as the AIME datasets in mathematics. While omitting sources may not solve the problem completely, there's not more we can feasibly do for now, and we hope you understand the difficulty behind this issue.\n\n### **References:**\n\n[1] [https://arxiv.org/abs/2009.13081](https://arxiv.org/abs/2009.13081)\n\n[2] [https://arxiv.org/abs/2305.09617](https://arxiv.org/abs/2305.09617)\n\n[3] [https://arxiv.org/abs/2404.18416](https://arxiv.org/abs/2404.18416)\n\n[4] [https://arxiv.org/abs/2408.03361](https://arxiv.org/abs/2408.03361)", "claims_and_evidence": "Yes. (Minor question on this asked at the end)", "methods_and_evaluation_criteria": "Yes. (Not aware of the best practices in general)", "theoretical_claims": "None.", "experimental_designs_or_analyses": "Sound.", "relation_to_broader_scientific_literature": "Extremely important. \n\n1. While there are larger datasets, this seems to be the most varied and largest expert annotated dataset.\n2. There is a clear need for a tougher benchmark. The results here depict a nice \"barrier\".\n3. Overall evaluation framework seems to be well within the accepted norms across various previous publications in the domain.", "essential_references_not_discussed": "None missing to my limited knowledge.", "other_strengths_and_weaknesses": "While not a weakness per say, why not evaluate a domain specific model such as LLaVA-Med (https://github.com/microsoft/LLaVA-Med)?", "other_comments_or_suggestions": "1. If o1 has such a high score in USMLE (stand-alone), does that simply imply that other exams are harder? Why is there such a drastic reduction in performance when exams of other countries are considered.  It would be interesting to have a comment on that whether this stems from the dataset structuring or is representative of the exams themselves.\n\n2. While I am not a clinician, I am not sure there is succinct literature proving that exam questions (which seem to be the sole data source) are completely \"real-world\" representative. (Note: In my short search, I could not find a definite citation for the agreement or disagreement of this.) Maybe having a comment on this can provide stronger confidence to a non-clinical user/reader?", "questions_for_authors": "1. It seems to me that, this is a \"simple (only in terms of comparison, not effort)\" extension of previous datasets by collecting questions across more exams. Would it then not be easy to fine-tune on these exams (which I imagine are available to all, even for a price) and get a boosted score knowing that all the questions are limited to these exams?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Legal Compliance (e.g., GDPR, copyright, terms of use)"], "ethical_review_concerns": "Massive medical dataset involving human expert annotation."}}, {"review_id": "YfdGroVl4q", "reviewer": "Reviewer_vM2L", "summary": "This paper introduces MedXpertQA, a challenging benchmark comprising 4,460 clinical medical questions. Comparing to previous simple Question-Answering (QA) datasets, MedXpertQA conducts difficulty and diversity filtering to ensure the data quality. \nAdditionally, the authors evaluate frontier LLMs on the proposed benchmark.\nEvaluation results are also thoroughly discussed.\n\n## update after rebuttal\nThe authors' rebuttal addressed most of my concerns. I decided to keep my original score as accept. This paper is a valuable contribution for the medical LLMs community.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and hope to further address your concerns.\n\n# Response 1 - Experimental Designs Or Analyses\n\n### **1.1 Evaluation on Different TTS Methods**\n\nThank you for the valuable suggestion. We will include these comparisons in the next versions of the paper. At the same time, the main motivation of our work is the high-quality benchmark itself, and dedicated works that focus on comparing inference-time methods (MedPrompt [1]) are better references for researchers interested in this direction. We also note that the inference scheme we use, *i.e.* zero-shot CoT, aligns with mainstream consensus for evaluating foundation models. We hope the above points address your concerns.\n\n### **1.2 Specialist Model Results**\n\nYour suggestions are very valuable. We agree with the need to include medical specialist models in our evaluation. We did consider it, but only left it out due to time constraints. As preliminary experiments, we evaluated a specialist text model UltraMedical-70B [2]: [UltraMedical-70B Results](https://postimg.cc/RJRkLgVN).\n\nWe provide the results of o1 for comparison, showing that the specialist model still falls behind the most advanced general reasoning model.\n\nWe plan to more thoroughly survey domain-specific models and present the full results in the next version of our paper. We will also systematically compare the performances of generalist models, specialist models, and humans to gain further insights.\n\n# Response 2 - Essential References Not Discussed\n\nWe totally agree that comparisons with other demanding benchmarks will be informative. We provide the dataset statistics of Humanity’s Last Exam (Biology / Medicine): [Humanity’s Last Exam Dataset](https://postimg.cc/1nZRWYPH).\n\nWe note obvious discrepancies in relative model performances between the two benchmarks, which demonstrate the informative value of diverse benchmarks.\n\nRegarding benchmark difficulty, we do recognize that the absolute scores for the Biology / Medicine subset of HLE are lower than MedXpertQA. However, this does not take away from the quality and value of MedXpertQA for two reasons:\n\n- First, most of HLE's biology / medical questions focus on biology instead of clinically relevant medical tasks, for instance: [Biology Question Example](https://postimg.cc/t1xwzcP4).\n    \n- Second, questions in HLE covering comprehensive patient information, thus supporting realistic clinical reasoning, as do most questions in MedXpertQA, are scarce (~35 in total). Many questions related to medical tasks only cover single steps within the complex reasoning process required to form a clinical decision for a patient. For example, the following question represents the step of interpreting a single statistic (R-R interval) from a single piece of patient information (ECG results):  \n\n    > What is the longest R-R interval in seconds on this rhythm strip ECG?\n\n    In contrast, questions in MedXpertQA that involve ECGs typically include the image as one piece of information within a multifaceted, realistic patient profile. The answerer not only interprets the ECG but also needs to consider the role of this information within a more complex decision, *e.g.* proposing a diagnosis or treatment. The example illustrated in Figure 2 in our paper precisely shows this scenario. Therefore, while medical questions in HLE effectively pinpoint challenging individual tasks, they are less informative of models' holistic abilities and less clinically relevant. As Reviewer y9fU happened to mention, \"reasoning-heavy\" and \"difficult\" are two different concepts, and we believe MedXpertQA has an advantage over HLE in terms of the former.\n\n- Third, MedXpertQA and HLE are fundamentally different types of benchmarks. The construction of HLE was extremely labor-intensive, requiring experts to manually design individual questions, whereas the construction process of MedXpertQA is scalable and systematic, enabling large-scale evaluations over questions that are clinically relevant and diverse. Moreover, the difference in benchmark scale is crucial. It is in fact possible for us to achieve a similar level of difficulty by further stringent filtering, reducing the dataset size — we simply need to retain questions that stump current models. This will, however, contradict the need for systematic and comprehensive coverage in medical evaluation, as we mentioned in Section 3.1. We believe MedXpertQA strikes a good balance.\n\n# Response 3 - Other Comments Or Suggestions\n\nYes, thanks for your suggestion! We will clarify this in the caption later.\n\n### **References:**\n\n[1] https://arxiv.org/abs/2311.16452\n\n[2] https://arxiv.org/abs/2406.03949", "claims_and_evidence": "This paper claims the increased difficulty and diversity of the proposed MedXpertQA dataset. These claims are supported by the observed decline in performance of evaluated LLMs and the analysis of data distribution.", "methods_and_evaluation_criteria": "The data collection and filtering in this paper is reasonable and solid. Table 1 offers a comparison among current medical QA datasets, which supports validating the quality of the proposed benchmark.", "theoretical_claims": "This paper employs the Brier score to evaluate the posterior difficulty of each question, which is a logical approach.", "experimental_designs_or_analyses": "Tables 3 and 4 benchmark leading LLMs on MedXpertQA, suggesting the effectiveness of reasoning LLMs (such as o1 and QVQ-72B) on addressing challenging medical questions.\nHowever, several weaknesses still exist in the experiment setting:\n1. Lack of evaluation on different test-time scaling methods (such as RAG) and prompting strategies (few-shot, CoT) on the proposed benchmark. Will the retrieval-based test-time scaling methods facilitate the solution of challenging medical reasoning problems?\n2. Lack of benchmarking of specialized medical LLMs. Will these medical LLMs perform better on knowledge questions?", "relation_to_broader_scientific_literature": "This paper provides a more challenging benchmark to further evaluate the performance of LLMs in the medical domain. Comparing to previous works such as medqa[1], medmcqa[2], pubmedqa[3], proposed MedXpertQA can better evaluate the capability of solving challenging clinical problems.\n\n[1] Jin, Di, et al. \"What disease does this patient have? a large-scale open domain question answering dataset from medical exams.\" Applied Sciences 11.14 (2021): 6421.\n\n[2] Pal, Ankit, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. \"Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.\" Conference on health, inference, and learning. PMLR, 2022.\n\n[3] Jin, Qiao, et al. \"Pubmedqa: A dataset for biomedical research question answering.\" arXiv preprint arXiv:1909.06146 (2019).", "essential_references_not_discussed": "The main contribution of this paper lies in presenting a varied and demanding medical benchmark. It would be beneficial if the author could offer a comparison regarding difficulty and data distribution with the medical part of Humanity's Last Exam[4].\n\n[4] Phan, Long, et al. \"Humanity's Last Exam.\" arXiv preprint arXiv:2501.14249 (2025).", "other_strengths_and_weaknesses": "For weaknesses, please refer to Experimental Designs and Essential References part.", "other_comments_or_suggestions": "Does AvgR and AvgK in Table 1 refer to the performance on reasoning and understanding subsets? The author should clarify this.", "questions_for_authors": "Please refer to weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The overall reviews of this paper are positive. The contribution to provide a new and challenging test bed for medical reasoning is significant, with more diverse and challenging questions, and the authors answered many of the reviewers' concerns during the rebuttal. The only remaining concerns are relative to data leakage, which seems bound to happen for any open benchmark, and there was no further discussion after the rebuttal, and the feasibility evaluation of the questions, which seems to be addressed by the expert filtering. I think that introducing such a challenging benchmark, with a large number of questions validated by experts and which shows that current LLMs are not able to solve them, is a significant contribution. Moreover, the sound methodology to create the benchmark could be used to create follow-up datasets to evaluate benchmark leakage/overfitting.  
For these reasons, I recommend accepting the paper.
Decision: Accept (poster)