MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports

## Abstract

Abstract Doctors and patients alike increasingly use Large Language Models (LLMs) to diagnose clinical cases. However, unlike domains such as math or coding, where correctness can be objectively defined by the final answer, medical diagnosis requires both the outcome and the reasoning process to be accurate. Currently, widely used medical benchmarks like MedQA and MMLU assess only accuracy in the final answer, overlooking the quality and faithfulness of the clinical reasoning process. To address this limitation, we introduce MedCaseReasoning , the first open-access dataset for evaluating LLMs on their ability to align with clinician-authored diagnostic reasoning. The dataset includes 14,489 diagnostic question-and-answer cases, each paired with detailed reasoning statements derived from open-access medical case reports. We evaluate state-of-the-art reasoning LLMs on MedCaseReasoning and find significant shortcomings in their diagnoses and reasoning: for instance, the top-performing open-source model, DeepSeek-R1, achieves only 48% 10-shot diagnostic accuracy and mentions only 64% of the clinician reasoning statements (recall). However, we demonstrate that fine-tuning LLMs on the reasoning traces derived from MedCaseReasoning significantly improves diagnostic accuracy and clinical reasoning recall by an average relative gain of 29% and 41%, respectively. The open-source dataset, code, and models are available at https://github.com/kevinwu23/Stanford-MedCaseReasoning .

## 1 Introduction

Large language models (LLMs) are increasingly deployed in medicine (AMA, 2025), where they show promise in performing complex tasks like clinical reasoning (Goh et al., 2024) and disease diagnosis (McDuff et al., 2025). While there are a number of benchmark datasets designed to assess the diagnostic capabilities of medical LLMs (e.g. MedQA (Jin et al., 2020), MMLU (Hendrycks et al., 2020), MultiMedQA (Singhal et al., 2023), and MedXpertQA (Zuo et al., 2025)), these datasets all share the same limitation in that they only assess the correctness of the model’s final answer. Unlike domains like mathematics (Ahn et al., 2024) or coding (Ding et al., 2024), where the reasoning process is secondary to the correct answer, medical diagnoses require both the final diagnosis and the thought process to be defensible. In clinical practice, physicians need to articulate coherent rationales, which is important for ensuring informed consent, meeting the standard of care, facilitating documentation and billing, and enabling clinical collaboration (Patel et al., 2005). Mistakes or deficiencies in reasoning, even if the correct diagnosis is given, can risk case mismanagement. Highlighting this gap, a recent study found that even frontier models like GPT-4 could produce the correct diagnosis for incorrect reasons in up to a third of clinical scenarios (Jin et al., 2024). Thus, evaluating and improving model capabilities to reason and answer correctly is crucial for LLMs to gain clinical viability and trustworthiness.

Figure: Figure 1: A schematic of the MedCaseReasoning case processing pipeline. A: From the initial set of 98,994 PMC Open Subset Case Reports, we select 28,313 appropriate candidate cases, which are converted to QA format and then filtered for quality. The resulting 14,489 cases form the MedCaseReasoning dataset, with 897 test cases and 13,092 training cases. A full example is available in Table [8](https://arxiv.org/html/2505.11733v2#S5.T8). B: For each test case, the case presentation is posed to an LLM to reason and then answer with a diagnosis. The clinician-authored diagnostic reasoning is compared with the LLM-generated reasoning to produce the reasoning recall score, and the case diagnosis is compared with the predicted diagnosis to produce diagnostic accuracy. C: The original case report text is converted into three sections: the case presentation that contains the relevant and sufficient patient information for making a diagnosis; the diagnostic reasoning that includes the diagnostic decision-making by the case author (in enumerated statements); and the final diagnosis.
Refer to caption: x1.png

In this work, we propose MedCaseReasoning, a diagnostic reasoning dataset with two complementary goals: (1) an evaluation benchmark to assess how well LLM reasoning aligns with clinician-authored diagnostic reasoning, and (2) a training dataset for improving diagnostic reasoning in LLMs from reasoning traces. This clinician-validated dataset is created from publicly available case reports published in PubMedCentral (PubMed Central, 2025). We perform filtering and clinician validation from an initial set of 98,994 case reports to produce a corpus of 14,489 diagnostic QA, with a high-quality test subset of 897 cases.
First, we observe that MedCaseReasoning is a meaningfully difficult and unsaturated benchmark, with the top commercial model, OpenAI o3, achieving only 65% on 10-shot diagnostic accuracy. Second, we find that even top-performing reasoning models are substantially limited in their ability to reason like clinicians, where the best open-source model, DeepSeek-R1, only recalls around half of the reasons provided in the case report. Third, we find that training on MedCaseReasoning can significantly improve the diagnostic and reasoning capabilities of LLMs: both
diagnostic accuracy and clinician reasoning alignment increased after fine-tuning three top medical open-source LLMs, with an average relative gain of 29% and 41%. respectively. Importantly, after training on MedCaseReasoning, their performance on NEJM CPC, a held-out test dataset, also improved, showcasing the value of MedCaseReasoning in improving diagnostic generalizability.

Main Contributions:

- •
We release MedCaseReasoning, an open-access dataset of 14K+ clinical diagnostic cases from 800+ medical journals and 30+ specialties with accompanying clinician-authored reasoning.
- •
To produce MedCaseReasoning, we describe a scalable, multi-step, clinician-validated pipeline for converting raw case reports into high-quality QA-format diagnostic cases.
- •
We evaluate state-of-the-art reasoning LLMs models on MedCaseReasoning, which reveals limitations in medical diagnostic reasoning.
- •
We show training on dense reasoning traces from MedCaseReasoning improves the diagnostic accuracy and reasoning recall of open-source LLMs.

Our work is the first medical benchmark dataset that explicitly evaluates the reasoning accompanying a diagnosis by comparing it to real-world case reports written by clinicians. These cases contain relevant differential diagnoses along with case presentations that present sufficient detail for an accurate diagnosis. Our findings underscore the significant challenges of aligning the reasoning of current top-performing LLMs with established clinical practice and provide a dataset for improving this alignment.

Related Works
While commonly used medical benchmark datasets like MedQA (Jin et al., 2020), MMLU (Hendrycks et al., 2020), and MedXpertQA (Zuo et al., 2025) include diagnostic reasoning questions, the clinical cases presented are typically shorter, textbook-style cases intended to evaluate students and test medical knowledge, not real-world clinical cases that physicians may encounter. Several datasets construct clinical vignettes from real-world sources, such as PubMedQA (Jin et al., 2019), which generated QA examples based on PubMed articles, and MedAlign (Fleming et al., 2023), which generated clinical vignettes from electronic health records. By comparison, MedCaseReasoning is derived directly from clinician-authored case reports, which are valuable tools for advancing clinical practice (Nayak, 2010).

The New England Journal of Medicine Clinicopathologic Conferences (NEJM CPC) is a collection of case reports sourced from Massachusetts General Hospital in Boston, MA (McDuff et al., 2025). MedCaseReasoning improves upon this dataset in several ways. First, the collection of case reports from PubMedCentral represents a globally diverse set of clinician and patient backgrounds, whereas NEJM CPC represents the medical practices of physicians from a single hospital system on a specific population. Second, the NEJM CPC dataset consists of only 302 test cases, while MedCaseReasoning is comprised of over 14K cases, allowing for more in-depth analysis as well as model fine-tuning. Third, MedCaseReasoning is derived from open-access articles on PubMedCentral, whereas NEJM CPC is available only under license.

Several studies (Goh et al. (2025) and Strong et al. (2023)) have employed grading rubrics applied to curated clinical vignettes, but are restricted to a handful of cases, not openly accessible, and focus on human-AI comparison rather than establishing benchmarks for standalone LLM reasoning performance. Studies by Kanjee et al. (2023), McDuff et al. (2025), and Gemini (2023) leveraged cases from NEJM CPC to evaluate model capabilities, but primarily focusing on generating differential diagnosis (DDx) lists, not full reasoning traces.

Recent models like HuaTuoGPT-o1 (Chen et al., 2024) and MedReason (Wu et al., 2025) involve supervised fine-tuning (SFT) on diagnostic reasoning traces, but the traces are distilled from larger proprietary models (e.g., GPT-4) rather than being grounded in reasoning authored by clinicians based on real patient cases. One study by (Savage et al., 2024) conducted manual clinician evaluations of reasoning traces from GPT-3.5 and GPT-4 to identify logical inconsistencies, but such methods are labor-intensive and inherently limited in scale, making them infeasible for large-scale benchmarking. Collectively, prior work highlights a gap in large-scale, open-access benchmarks grounded in real-world, clinician-authored reasoning that evaluate the complete diagnostic thought process beyond just final answer accuracy.

Figure: Figure 2: (Left) Evaluation of LLMs on the MedCaseReasoning test dataset (N = 897), along with diagnostic accuracy and reasoning recall percentage. (Right) Model performance on the MedCaseReasoning test set and NEJM case studies (N = 302), showing a strong correlation between the two benchmarks. In both plots, accuracy is computed with 10-shot accuracy; circle size encodes the average length of the model-generated reasoning trace. LLaMA-3.1-8B and Qwen-2.5-7B are both Instruct variants (omitted for brevity).
Refer to caption: x2.png

**Table 1: Comparison of clinical diagnosis benchmark datasets. MedCaseReasoning is the only and largest open-access dataset with clinician reasoning. Existing datasets do not contain expert reasoning.**
| Dataset | $N$ | Open Access | Reasoning | Question Type | Question Source |
| --- | --- | --- | --- | --- | --- |
| MedQA |  |  |  |  |  |
| (Jin et al., 2020) | 12,723 | ✓ | $\times$ | Clinical vignettes | USMLE |
| MedXpertQA |  |  |  |  |  |
| (Zuo et al., 2025) | 1,050 | ✓ | $\times$ | Clinical vignettes | Licensing exams |
| NEJM CPC |  |  |  |  |  |
| (McDuff et al., 2025) | 302 | $\times$ | $\times$ | Real diagnostic cases | NEJM CPC |
| MedCaseReasoning | 14,489 | ✓ | ✓ | Real diagnostic cases | PMC Case Reports |

## 2 Methods

**Table 2: An example of comparing reasoning traces against the case report reasoning. The reasoning from the case report is extracted using an LLM as an enumerated list. Then, each element of the list is referenced against the model’s reasoning trace, which gives us the “reasoning recall” for a given trace.**
| Reasoning | LLaMA-3.1-8B-Instruct (Base) | LLaMA-3.1-8B-Instruct (SFT on MedCaseReasoning) | Groundtruth (from MedCaseReasoning) |
| --- | --- | --- | --- |
| Trace | The patient has a history of Wilson disease, a genetic disorder that leads to copper accumulation in the body. However, the brain MRI shows no evidence of pathological copper deposition, and laboratory tests show normal copper metabolism – suggesting the current condition is not related to Wilson disease.<br>The symptoms … are consistent with a psychotic disorder. The history of Wilson disease and orthotopic liver transplantation may have led to post-liver-transplant psychosis …<br>Given the patient’s symptoms and history, post-liver-transplant psychosis is the most likely diagnosis. | We first considered acute graft rejection due to non-adherence with tacrolimus and mycophenolate, but this was ruled out because tacrolimus trough levels were low.<br>Next, tacrolimus neurotoxicity was excluded for the same reason.<br>We also considered copper overload from Wilson<br>disease, but MRI showed no pathological copper deposition.<br>Delirium was excluded; finally, we evaluated substance-induced psychosis but found “no other medical condition or substance use disorder.”<br>With alternatives excluded, schizophrenia was suspected. | 1. Wilson-disease recurrence — “no abnormalities, including pathological copper accumulation … no evidence of abnormal copper metabolism … psychosis due to recurrence of Wilson disease was ruled out.”<br>2. Tacrolimus-induced psychosis — “Tacrolimus-induced psychosis was ruled out because the trough tacrolimus concentration had not increased before onset of psychosis, and psychosis persisted after the trough value decreased.”<br>3. Substance-related or other medical causes — “No evidence of other medical conditions or a substance use disorder that could cause psychotic symptoms was found.” |
| Answer | Post-liver-transplant psychosis | Schizophrenia | Schizophrenia |
| Coverage | 1/3 reasons recalled | 3/3 reasons recalled | Groundtruth |
|  |  |  |  |

### 2.1 Data curation

Case reports are typically written to highlight a rare or complex disease. A core tradeoff exists between the novelty and usefulness of a case report – models should learn from difficult cases, but should be able to practically deduce the diagnosis from the presented information. To this end, we produced a LLM-based pipeline with the goal of maximizing case novelty and usefulness. As expert verification on each step is prohibitively expensive, we perform our clinician validation only on the outputs of the last step.

- 1.
Data Provenance: We started with the PMC Open Subset pulled from Jan 1st, 2005-April 27th, 2025. The query was filtered for free-access articles with article type ”case report”, which yielded 98,994 case reports. The article’s text was extracted from the main body in the XML file, along with the PMCID, publication date, title, journal name, and article link.
- 2.
Candidate Selection: Next, we excluded case reports without the word differential (ie. a discussion on differential diagnoses), which reduced the set to 28,313 case reports. These initial candidate case reports were converted into diagnostic QA with o4-mini, a strong reasoning model that can be run at scale. The full prompt is available in the Appendix (Prompt [5](https://arxiv.org/html/2505.11733v2#S5)) and instructs models to identify the core conflict within the case report and provide an information cutoff that would ensure the presented information does not give away the final diagnosis. In parallel, each candidate case report was scored according to the following criteria: 1) thoroughness of case presentation, 2) presence of explicit differential diagnosis, 3) dependence on integrative clinical reasoning, 4) transparency of diagnostic reasoning process, and 5) presence of stated final diagnosis. These criteria were developed by clinicians, and o4-mini was used to produce the actual scores. Criteria 1, 3, and 4 are given out of 1-5 points, whereas criteria 2 and 5 are yes/no responses.
We filtered out cases where 1) thoroughness of case presentation was either 1 or 2 (seriously deficient or major gaps), and also removed cases that either did not discuss 2+ plausible alternatives or did not state a final diagnosis. The full prompt is available in the Appendix (Prompt [5](https://arxiv.org/html/2505.11733v2#S5)). After filtering, there remained 19,428 total case reports.
- 3.
Quality Filter: Next, to avoid model blind spots, we checked each of the generated case reports using a separate LLM (gemini-2.5-pro), which evaluated the generated case report’s faithfulness to the source article and the plausibility of each case report. The prompt is also available in the Appendix (Prompt [5](https://arxiv.org/html/2505.11733v2#S5)). We removed cases that had any flags, leaving a final total of 14,489 cases. We created an initial testing subset of 897 cases where the transparency and integrative diagnostic reasoning scores were at least 4 or 5.

The case reports used in MedCaseReasoning span over 800 different medical journals, with the top 10 shown in Table [5](https://arxiv.org/html/2505.11733v2#S5.T5). Additionally, the diagnostic case prompts from MedCaseReasoning are significantly longer and more detailed than the ones from MedQA (displayed in Figure [3](https://arxiv.org/html/2505.11733v2#S5.F3)). Furthermore, case report publication dates are heavily enriched towards years after 2020, with over 16% after Jan 1 2024 (Figure [4](https://arxiv.org/html/2505.11733v2#S5.F4)). Additionally, our pipeline can be regularly updated, incorporating new case reports with low marginal cost.

To validate the case prompts, diagnostic reasoning, and final diagnosis extracted from each case report, a team of four board-certified physicians reviewed a total of 100 randomly selected cases. For each case, they responded to three questions related to the presence of hallucinations, faithfulness, and reasonableness of each case. The full questions can be found in the Appendix (Table [4](https://arxiv.org/html/2505.11733v2#S5.T4)).

### 2.2 Evaluating Diagnostic Accuracy in Models

We evaluate each model’s diagnostic correctness using LLM-as-a-judge, in accordance to previous literature (e.g., McDuff et al. (2025); Wu et al. (2025)). We adopt the same prompt used in McDuff et al. (2025), (Prompt [5](https://arxiv.org/html/2505.11733v2#S5)), which has been validated to have high concordance with human raters, and use gpt-4o-mini as the judge model for its speed and accuracy.

Differential diagnoses typically contain between five to ten candidate diagnoses which are followed-up in a clinical setting. Also following evaluation schema from McDuff et al. (2025), each model is evaluated a total of 10 times (temperature of 0.8 and top-p of 0.95), and the N-shot performance is recorded.

As an external validation to the MedCaseReasoning Test Set, we also evaluate models on case reports from NEJM Clinicopathologic Conferences. In particular, we use a subset of 302 cases from previous works (McDuff et al., 2025; Kanjee et al., 2023; Gemini, 2023) as a gold-standard dataset of complex diagnostic cases. We manually extracted each case report with a personal license and fed the case presentation portion as the case prompt without any additional modifications. As NEJM CPC is not an open-access publication, we are not able to open-source this validation set. Additionally, no NEJM case reports were included in MedCaseReasoning or used in training any models.

### 2.3 Evaluating Reasoning Traces in Models

While some case studies include comprehensive reasoning steps, it is not guaranteed that case reports include an exhaustive set of reasoning steps. As such, we focus our study on evaluating how well models recall clinician-produced reasons (as opposed to evaluating precision). We define this metric as the “Reasoning Recall”.
(Note: We were not able to evaluate reasoning recall for OpenAI o3 and other frontier models where the full reasoning traces are not available via API call. While they can produce reasoning tokens when prompted (i.e. with ¡think¿ tags), the comprehensive reasoning that is used to inform a diagnosis is not accessible.)

###### Definition 1 (Reasoning Recall) .

Let $N$ be the total number of cases. For each case $i$, let

$$ $R_{i}=\{\text{groundtruth reasons from case report}\},\quad T_{i}=\{\text{ reasons from model reasoning trace}\}.$ $$

For case $i$, the recall rate is

$$ $c_{i}=\frac{\lvert R_{i}\cap T_{i}\rvert}{\lvert R_{i}\rvert}.$ $$

The *Reasoning Recall* is the average of these rates:

$$ $\mathrm{RR}=\frac{1}{N}\sum_{i=1}^{N}c_{i}=\frac{1}{N}\sum_{i=1}^{N}\frac{ \lvert R_{i}\cap T_{i}\rvert}{\lvert R_{i}\rvert}.$ $$

For example, a reasoning recall of 50% indicates that on average, a model’s reasoning trace will cover half of the reasons provided by the case report. We evaluate the reasoning trace associated with its best-of-10 response, where we use the trace from a randomly selected correct response (or incorrect if there are no correct responses). The evaluation is implemented using an LLM-as-a-judge, where o4-mini is instructed to return a JSON with a decision on whether each of the groundtruth reasons are found in the reasoning trace. We validate this step with annotations from a board-certified physician. The prompt used is found in the Appendix (Prompt [5](https://arxiv.org/html/2505.11733v2#S5)).

###### Definition 1 (Reasoning Recall) .

### 2.4 Supervised Fine-Tuning on Case Report Reasoning

In addition to evaluating model reasoning traces, we explore whether fine-tuning on extracted reasons directly can improve model performance on diagnostic accuracy and reasoning recall. One technical challenge is that the extracted diagnostic reasoning is formatted as an enumerated list of summary points and quotations, rather than a cohesive reasoning trace. Naturally, we can use LLMs to ”stitch” the list of points into reasoning traces without adding new information. We provide the prompt used to perform this step in the Appendix (Prompt [5](https://arxiv.org/html/2505.11733v2#S5)). The stitching process may introduce biases if a stronger model introduces its own priors into the reasoning traces. To control for this, we ensure that the model that is being fine-tuned is also the model that stitches the reasoning traces together. An example of a stitched reasoning trace can be found in the Appendix (Prompt [7](https://arxiv.org/html/2505.11733v2#S5.T7)).

We perform supervised fine-tuning (SFT) on two popular open-sourced models: Qwen-2.5-7B-Instruct and LLaMA-3.1-8B-Instruct. Additionally, to test the marginal value of our reasoning dataset, we include MedReason-8B, a model based on LLaMA-3.1-8B-Instruct that was previously fine-tuned on a synthetic medical reasoning dataset. We updated each model with full-weight fine-tuning for three epochs on 8 NVIDIA H100 GPUs with a learning rate of 2e-5 and batch size of 256. All other hyperparameters are default according to the verl Python package implementation of SFT (version v0.3.0.rc0).

## 3 Results

**Table 3: Performance of models on MedCaseReasoning’s test set (N=897). Each model is evaluated on reasoning coverage and diagnostic accuracy (broken down into 1, 5, and 10-shot). Reasoning coverage is not available for OpenAI o3 as the reasoning traces are not provided via API. We additionally perform supervised fine-tuning (SFT) on three open-source models on the training split of MedCaseReasoning and find significant improvements in both reasoning coverage and diagnostic accuracy.**
| MedCaseReasoning Test Set |  |  |  |  |
| --- | --- | --- | --- | --- |
| Base Models | Reasoning Recall | 1-shot Acc. | 5-shot Acc. | 10-shot Acc. |
| OpenAI o3 | N/A | 0.470${}_{\scriptstyle 0.440\text{-}0.500}$ | 0.609${}_{\scriptstyle 0.579\text{-}0.639}$ | 0.645${}_{\scriptstyle 0.618\text{-}0.675}$ |
| DeepSeek R1 | 0.642${}_{\scriptstyle 0.616\text{-}0.667}$ | 0.320${}_{\scriptstyle 0.291\text{-}0.349}$ | 0.447${}_{\scriptstyle 0.417\text{-}0.478}$ | 0.480${}_{\scriptstyle 0.450\text{-}0.510}$ |
| QwQ-32B | 0.590${}_{\scriptstyle 0.560\text{-}0.619}$ | 0.272${}_{\scriptstyle 0.245\text{-}0.302}$ | 0.371${}_{\scriptstyle 0.341\text{-}0.400}$ | 0.398${}_{\scriptstyle 0.368\text{-}0.428}$ |
| MedReason-8B | 0.407${}_{\scriptstyle 0.383\text{-}0.431}$ | 0.248${}_{\scriptstyle 0.224\text{-}0.275}$ | 0.331${}_{\scriptstyle 0.303\text{-}0.363}$ | 0.382${}_{\scriptstyle 0.353\text{-}0.412}$ |
| LLaMA-3.1-8B-Instruct | 0.451${}_{\scriptstyle 0.428\text{-}0.475}$ | 0.161${}_{\scriptstyle 0.138\text{-}0.184}$ | 0.281${}_{\scriptstyle 0.252\text{-}0.311}$ | 0.332${}_{\scriptstyle 0.304\text{-}0.360}$ |
| m1-7b-23k | 0.495${}_{\scriptstyle 0.440\text{-}0.551}$ | 0.155${}_{\scriptstyle 0.133\text{-}0.177}$ | 0.238${}_{\scriptstyle 0.211\text{-}0.264}$ | 0.291${}_{\scriptstyle 0.262\text{-}0.321}$ |
| Qwen-2.5-7B | 0.324${}_{\scriptstyle 0.301\text{-}0.347}$ | 0.174${}_{\scriptstyle 0.152\text{-}0.197}$ | 0.252${}_{\scriptstyle 0.223\text{-}0.279}$ | 0.287${}_{\scriptstyle 0.259\text{-}0.316}$ |
| Fine-Tuned Models | Reasoning Recall | 1-shot Acc. | 5-shot Acc. | 10-shot Acc. |
| MedReason-8B (SFT) | 0.522${}_{\scriptstyle 0.498\text{-}0.545}$ | 0.303${}_{\scriptstyle 0.276\text{-}0.331}$ | 0.430${}_{\scriptstyle 0.400\text{-}0.459}$ | 0.501${}_{\scriptstyle 0.470\text{-}0.532}$ |
| LLaMA-3.1-8B-Instruct |  |  |  |  |
| (SFT) | 0.485${}_{\scriptstyle 0.460\text{-}0.510}$ | 0.278${}_{\scriptstyle 0.249\text{-}0.307}$ | 0.411${}_{\scriptstyle 0.378\text{-}0.443}$ | 0.479${}_{\scriptstyle 0.448\text{-}0.509}$ |
| Qwen-2.5-7B (SFT) | 0.486${}_{\scriptstyle 0.461\text{-}0.510}$ | 0.249${}_{\scriptstyle 0.221\text{-}0.278}$ | 0.363${}_{\scriptstyle 0.335\text{-}0.394}$ | 0.425${}_{\scriptstyle 0.394\text{-}0.455}$ |

### 3.1 Validity of Generated Case Prompts, Diagnostic Reasonings, and Final Diagnoses

We performed a human validation of the model-generated case prompts, diagnostic reasons, and final diagnoses on the MedCaseReasoning dataset. Our study included four U.S. board-certified physicians reviewing a total of 100 cases. Each physician reported spending an average of 10-20 minutes per case. We found a high degree of agreement that the generated cases were free of hallucinations in the case prompt or diagnostic reasoning (98%). We also report that 92% of final diagnoses were reported to have been faithful to the article and reasonably inferrable from the details of the case prompt. Finally, 93% of diagnostic reasoning steps were faithful to the case report and clinically relevant. The results are found in the Appendix (Table [4](https://arxiv.org/html/2505.11733v2#S5.T4)).

### 3.2 Diagnostic Accuracy

We evaluate the following combination of frontier reasoning models and popular open-sourced models: OpenAI o3, DeepSeek R1, QwQ-32B, MedReason-8B, m1-7b-23k, LLaMA-3.1-8B-Instruct, and Qwen-2.5-7B-Instruct.

The models are each evaluated on 1-shot, 5-shot, and 10-shot accuracy, listed in Table [3](https://arxiv.org/html/2505.11733v2#S3.T3) and Table [6](https://arxiv.org/html/2505.11733v2#S5.T6). Overall, we found that diagnostic reasoning remains a difficult task for top-performing LLMs. OpenAI o3 performs significantly better than the rest of the models, with 64.5% 10-shot accuracy on the MedCaseReasoning test set, whereas DeepSeek R1 achieves 48.0%. In comparison, OpenAI o3 and DeepSeek R1 achieve similar scores of 62.3% and 43.7% 10-shot accuracy on NEJM CPC, signaling the validity of MedCaseReasoning as an open-access alternative evaluation set for diagnostic cases.

We find that supervised fine-tuning significantly improves model performance across both test sets. For example, on the MedCaseReasoning test set, MedReason-8B improves by 31% 10-shot accuracy, outperforming DeepSeek R1. Of note, this model also improves by 18% on NEJM CPC and outperforms QwQ-32B. This provides evidence of the generalizability of MedCaseReasoning training data, as NEJM CPC consists of out-of-distribution, hand-crafted diagnostic cases that are often much longer in length.

### 3.3 Validity of Reasoning Recall Metric

We validate our LLM-as-a-judge for determining reasoning recall with verification from a board-certified physician. The physician was given N=33 cases and was asked to verify the LLM judge’s decisions on a total of 89 pairs of groundtruth reasons and model thinking traces. The cases were randomly sampled across all evaluated models. For example, for a given case report that contained three reasoning steps, the physician was asked to cross-check each step against the entire model reasoning trace to see if it was considered. The physician found 84/89 (94.4, 95% CI of 84.8%-100%) of pairs of reasons and thinking traces to be correctly assessed by the model, and a total of 31/33 (93.9, 95% CI of 84.8%-100%) cases to be completely accurately assessed by the model.

### 3.4 Evaluation of Reasoning LLMs

We evaluate six of seven models that provide reasoning traces on reasoning recall and find that the top model, DeepSeek R1, covers 64.2% of the reasoning steps provided in case reports. We also found that MedReason-8B, a model explicitly trained on synthetically generated medical reasoning traces, did not have substantially higher reasoning recall, and even had lower recall than LLaMA-3.1-8B-Instruct’s base model. However, after SFT on MedCaseReasoning, each of the base models improved significantly on reasoning recall. For example, MedReason-8B improved by 28% and Qwen-2.5-7B-Instruct improved by 50%.
Common types of case report reasoning not found within the base model reasoning traces were either missing candidate diagnoses or exclusionary symptoms for common diagnoses. For example, one case report focused on a case of Adult-onset Still’s disease (AOSD), which presented very similarly to Rocky Mountain Spotted Fever (RMSF). The model (m1-7b-23k) made a diagnosis for RMSF, missing reasoning about the patient presentation from the case report: “We feel that our patient’s symptoms were not due to RMSF, because the rash was salmon colored and worsened with the fever spikes.”. We found a significant correlation between model performance and reasoning recall (Pearson r=0.710, p=0.0485), indicating the value of measuring reasoning steps as a proxy for model performance itself. Furthermore, we also observe a significant correlation between the length of the model’s reasoning trace and the reasoning recall (r=0.790, p=0.0196).

## 4 Discussion

Our work introduces MedCaseReasoning, the first open-access dataset designed to evaluate the alignment of LLMs with clinician-authored diagnostic reasoning. MedCaseReasoning addresses a critical gap in current medical benchmarks: the assessment of diagnostic accuracy without scrutinizing the underlying reasoning process. We report two key findings: first, even top-performing LLMs exhibit deficiencies in aligning with clinician reasoning, achieving a maximum recall of 64.2%. Second, fine-tuning models on the reasoning traces derived from MedCaseReasoning significantly improves both their recall with clinician reasoning and their final diagnostic accuracy. These findings underscore the value of MedCaseReasoning in both evaluating and enhancing the clinical diagnostic capabilities of LLMs.

SFT with the MedCaseReasoning training dataset demonstrates that smaller models, such as Llama 3.1 8B and Qwen 2.5 7B, can achieve diagnostic accuracy comparable to or exceeding that of larger models like Qwen1.5-32B and DeepSeek-R1 after SFT. While prior studies (Chen et al. (2024); Wu et al. (2025)) have explored learning reasoning from synthetic traces generated by more powerful models, our research is the first to demonstrate the efficacy of training directly on clinician-written diagnostic reasoning.

Compared to established benchmarks like MedQA, where leading models such as GPT-4o have already achieved over 90% accuracy, diagnostic performance on MedCaseReasoning currently peaks at 64.5%. This suggests MedCaseReasoning presents a more challenging task, focusing on the nuanced alignment with expert reasoning. This characteristic is shared with complex diagnostic case report datasets like the NEJM CPC; indeed, we observe a strong correlation in diagnostic performance between MedCaseReasoning and NEJM CPC (Figure [2](https://arxiv.org/html/2505.11733v2#S1.F2)). However, MedCaseReasoning offers distinct advantages: it is open-access, unlike the license-restricted NEJM CPC, and provides a substantially larger corpus, with nearly 14,489 examples (including a 13,092-example training set), compared to NEJM CPC’s 302 test cases. Additionally, the MedCaseReasoning case curation pipeline is also extensible to other case reports, so the dataset is able to be extended as more reports become available, allowing the dataset to be updated to reflect current medical guidelines.

Our study presents several limitations. First, some case reports may lack sufficient detail for a definitive diagnosis or present trivial cases. The QA conversion process can also introduce variability, where case details may be inadvertently left out or hallucinated. While we implemented a clinician-validated filtering pipeline for the test set to ensure reasoning statements are grounded in the case presentation, some intractable cases (eg. ones where the diagnosis cannot be made without certain information) or trivial cases (eg. ones where the diagnosis is given away in the prompt) still persist. Second, MedCaseReasoning captures the case presentation at a single timestep before asking for a final diagnosis. It does not reflect the iterative, multi-stage nature of real-world clinical diagnosis, which involves refining differential diagnoses based on evolving information from tests, imaging, and treatment responses. Third, our reasoning recall metric only captures clinical reasoning provided within case reports. Diagnostic reasoning is inherently subjective, and while our extensive training corpus aims to encompass diverse diagnostic standards, the alignment metric should be interpreted as adherence to observed clinical reasoning patterns across a diverse range of clinicians rather than an absolute, single gold standard. Diagnosing rare and complex diseases has broad societal implications on patient health, patient-doctor interactions, and the trustworthiness of LLMs. Our study aims to shed light on a key factor mediating all three of these factors, namely diagnostic reasoning.

## References

- Ahn et al. (2024)
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., and Yin, W.
Large language models for mathematical reasoning: Progresses and challenges.
*arXiv*, 2024.
doi: 10.48550/arxiv.2402.00157.
- AMA (2025)
AMA.
AMA: Physician enthusiasm grows for health care AI undefined american medical association, 2025.
URL [https://www.ama-assn.org/press-center/ama-press-releases/ama-physician-enthusiasm-grows-health-care-ai](https://www.ama-assn.org/press-center/ama-press-releases/ama-physician-enthusiasm-grows-health-care-ai).
- Chen et al. (2024)
Chen, J., Cai, Z., Ji, K., Wang, X., Liu, W., Wang, R., Hou, J., and Wang, B.
HuatuoGPT-o1, towards medical complex reasoning with LLMs.
*arXiv [cs.CL]*, December 2024.
- Ding et al. (2024)
Ding, H., Fan, Z., Guehring, I., Gupta, G., Ha, W., Huan, J., Liu, L., Omidvar-Tehrani, B., Wang, S., and Zhou, H.
Reasoning and planning with large language models in code development.
In *Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, volume 35, pp.  6480–6490, New York, NY, USA, August 2024. ACM.
- Fleming et al. (2023)
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E. P., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., Patel, B. S., Chiang, C.-C., Callahan, A., Huo, Z., Gatidis, S., Adams, S. J., Fayanju, O., Shah, S. J., Savage, T., Goh, E., Chaudhari, A. S., Aghaeepour, N., Sharp, C., Pfeffer, M. A., Liang, P., Chen, J. H., Morse, K. E., Brunskill, E. P., Fries, J. A., and Shah, N. H.
MedAlign: A clinician-generated dataset for instruction following with electronic medical records.
*arXiv [cs.CL]*, August 2023.
- Gemini (2023)
Gemini.
Gemini: A family of highly capable multimodal models.
*arXiv [cs.CL]*, December 2023.
- Goh et al. (2024)
Goh, E., Gallo, R., Hom, J., Strong, E., Weng, Y., Kerman, H., Cool, J. A., Kanjee, Z., Parsons, A. S., Ahuja, N., Horvitz, E., Yang, D., Milstein, A., Olson, A. P. J., Rodman, A., and Chen, J. H.
Large language model influence on diagnostic reasoning.
*JAMA Network Open*, 7(10):e2440969, 2024.
doi: 10.1001/jamanetworkopen.2024.40969.
- Goh et al. (2025)
Goh, E., Gallo, R. J., Strong, E., Weng, Y., Kerman, H., Freed, J. A., Cool, J. A., Kanjee, Z., Lane, K. P., Parsons, A. S., Ahuja, N., Horvitz, E., Yang, D., Milstein, A., Olson, A. P. J., Hom, J., Chen, J. H., and Rodman, A.
GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial.
*Nature Medicine*, 31(4):1233–1238, 2025.
ISSN 1078-8956.
doi: 10.1038/s41591-024-03456-y.
- Hendrycks et al. (2020)
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J.
Measuring massive multitask language understanding.
*arXiv*, 2020.
doi: 10.48550/arxiv.2009.03300.
- Jin et al. (2020)
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., and Szolovits, P.
What disease does this patient have? a large-scale open domain question answering dataset from medical exams.
*arXiv [cs.CL]*, 2020.
- Jin et al. (2019)
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., and Lu, X.
PubMedQA: A dataset for biomedical research question answering.
*arXiv [cs.CL]*, September 2019.
- Jin et al. (2024)
Jin, Q., Chen, F., Zhou, Y., Xu, Z., Cheung, J. M., Chen, R., Summers, R. M., Rousseau, J. F., Ni, P., Landsman, M. J., Baxter, S. L., Al’Aref, S. J., Li, Y., Chen, A., Brejt, J. A., Chiang, M. F., Peng, Y., and Lu, Z.
Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine.
*NPJ Digit. Med.*, 7(1):190, July 2024.
- Kanjee et al. (2023)
Kanjee, Z., Crowe, B., and Rodman, A.
Accuracy of a generative artificial intelligence model in a complex diagnostic challenge.
*Jama*, 330(1):78–80, 2023.
- McDuff et al. (2025)
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al.
Towards accurate differential diagnosis with large language models.
*Nature*, pp.  1–7, 2025.
- Nayak (2010)
Nayak, B. K.
The significance of case reports in biomedical publication.
*Indian J. Ophthalmol.*, 58(5):363–364, September 2010.
- Patel et al. (2005)
Patel, V. L., Arocha, J. F., and Zhang, J.
Thinking and reasoning in medicine.
*The Cambridge handbook of thinking and reasoning*, 14:727–750, 2005.
- PubMed Central (2025)
PubMed Central.
PMC open access subset.
[https://pmc.ncbi.nlm.nih.gov/tools/openftlist/](https://pmc.ncbi.nlm.nih.gov/tools/openftlist/), 2025.
Accessed: 2025-5-15.
- Savage et al. (2024)
Savage, T., Nayak, A., Gallo, R., Rangan, E., and Chen, J. H.
Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine.
*npj Digital Medicine*, 7(1):20, 2024.
doi: 10.1038/s41746-024-01010-1.
- Singhal et al. (2023)
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., Payne, P., Seneviratne, M., Gamble, P., Kelly, C., Babiker, A., Schärli, N., Chowdhery, A., Mansfield, P., Demner-Fushman, D., Agüera Y Arcas, B., Webster, D., Corrado, G. S., Matias, Y., Chou, K., Gottweis, J., Tomasev, N., Liu, Y., Rajkomar, A., Barral, J., Semturs, C., Karthikesalingam, A., and Natarajan, V.
Large language models encode clinical knowledge.
*Nature*, 620(7972):172–180, August 2023.
- Strong et al. (2023)
Strong, E., DiGiammarino, A., Weng, Y., Kumar, A., Hosamani, P., Hom, J., and Chen, J. H.
Chatbot vs medical student performance on free-response clinical reasoning examinations.
*JAMA Intern. Med.*, 183(9):1028–1030, 2023.
- Wu et al. (2025)
Wu, J., Deng, W., Li, X., Liu, S., Mi, T., Peng, Y., Xu, Z., Liu, Y., Cho, H., Choi, C.-I., Cao, Y., Ren, H., Li, X., Li, X., and Zhou, Y.
MedReason: Eliciting factual medical reasoning steps in LLMs via knowledge graphs.
*arXiv [cs.CL]*, April 2025.
- Zuo et al. (2025)
Zuo, Y., Qu, S., Li, Y., Chen, Z., Zhu, X., Hua, E., Zhang, K., Ding, N., and Zhou, B.
MedXpertQA: Benchmarking expert-level medical reasoning and understanding.
*arXiv [cs.AI]*, January 2025.

## 5 Appendix

**Table 4: Physician agreement rates with 95% confidence intervals for diagnostic prompt evaluation, with $N=100$ total case reports reviewed. We find that diagnostic cases generated from case reports contain low rates of hallucinations and are faithful to the original reports.**
| Question | Physician Agreement |
| --- | --- |
| Q1: There are no hallucinated details in the case prompt or diagnostic reasoning. | 98.0%${}_{95.0\text{–}100.0\%}$ |
| Q2: The final diagnosis is faithful to the article and reasonably inferrable from the details in the case prompt. | 92.0%${}_{86.0\text{–}97.0\%}$ |
| Q3: The diagnostic reasoning steps are faithful to the case report and are clinically relevant. | 93.0%${}_{88.0\text{–}98.0\%}$ |

**Table 5: Top 10 medical case report journals by prevalence in the PMC Open Access Subset dataset. In total, there are 813 unique journals in MedCaseReasoning, spanning 30+ different medical specialties across multiple countries.**
| Journal Name | Prevalence (%) |
| --- | --- |
| Journal of Medical Case Reports | 8.20% |
| Clinical Case Reports | 7.57% |
| International Journal of Surgery Case Reports | 6.81% |
| Radiology Case Reports | 6.36% |
| Journal of Surgical Case Reports | 2.88% |
| JAAD Case Reports | 2.47% |
| SAGE Open Medical Case Reports | 2.16% |
| Journal of Orthopaedic Case Reports | 2.10% |
| Case Reports in Medicine | 1.87% |
| European Heart Journal: Case Reports | 1.60% |

Figure: Figure 3: Comparison of length of questions from MedQA vs. MedCaseReasoning. Diagnostic case prompts are, on average, 2.5x longer in MedCaseReasoning and contain real patient information vs. synthetic case vignettes.
Refer to caption: x4.png

Figure: Figure 4: Distribution of dates of publication for PMC case reports used in MedCaseReasoning. Cases are largely from recent dates (after 2020), and over 500 cases are after Jan 1 2025.
Refer to caption: x5.png

**Table 6: To validate the generalization of our dataset, we evaluate models on NEJM Clinicopathologic Conference (CPC) cases (N=302), previously used in McDuff et al. (2025). We find that models fine-tuned on MedCaseReasoning also significantly improve on NEJM cases, which are considered a gold-standard in diagnostic case reports.**
| NEJM Clinicopathologic Conferences (CPC) - Generalization |  |  |  |
| --- | --- | --- | --- |
| Base Models | 1-shot Acc. | 5-shot Acc. | 10-shot Acc. |
| OpenAI o3 | 0.430${}_{\scriptstyle 0.377\text{-}0.490}$ | 0.579${}_{\scriptstyle 0.526\text{-}0.639}$ | 0.623${}_{\scriptstyle 0.569\text{-}0.675}$ |
| DeepSeek R1 | 0.272${}_{\scriptstyle 0.225\text{-}0.318}$ | 0.407${}_{\scriptstyle 0.354\text{-}0.467}$ | 0.437${}_{\scriptstyle 0.384\text{-}0.497}$ |
| QwQ-32B | 0.182${}_{\scriptstyle 0.136\text{-}0.225}$ | 0.281${}_{\scriptstyle 0.232\text{-}0.334}$ | 0.341${}_{\scriptstyle 0.288\text{-}0.397}$ |
| MedReason-8B | 0.169${}_{\scriptstyle 0.126\text{-}0.215}$ | 0.262${}_{\scriptstyle 0.212\text{-}0.308}$ | 0.295${}_{\scriptstyle 0.245\text{-}0.348}$ |
| LLaMA-3.1-8B-Instruct | 0.070${}_{\scriptstyle 0.043\text{-}0.099}$ | 0.142${}_{\scriptstyle 0.103\text{-}0.182}$ | 0.212${}_{\scriptstyle 0.165\text{-}0.255}$ |
| m1-7b-23k | 0.099${}_{\scriptstyle 0.066\text{-}0.136}$ | 0.172${}_{\scriptstyle 0.136\text{-}0.215}$ | 0.209${}_{\scriptstyle 0.162\text{-}0.252}$ |
| Qwen-2.5-7B | 0.070${}_{\scriptstyle 0.043\text{-}0.099}$ | 0.109${}_{\scriptstyle 0.076\text{-}0.142}$ | 0.123${}_{\scriptstyle 0.086\text{-}0.159}$ |
| Fine-Tuned Models | 1-shot Acc. | 5-shot Acc. | 10-shot Acc. |
| MedReason-8B (SFT) | 0.179${}_{\scriptstyle 0.136\text{-}0.225}$ | 0.278${}_{\scriptstyle 0.228\text{-}0.328}$ | 0.348${}_{\scriptstyle 0.295\text{-}0.404}$ |
| LLaMA-3.1-8B-Instruct (SFT) | 0.169${}_{\scriptstyle 0.126\text{-}0.212}$ | 0.255${}_{\scriptstyle 0.205\text{-}0.305}$ | 0.308${}_{\scriptstyle 0.255\text{-}0.358}$ |
| Qwen-2.5-7B (SFT) | 0.129${}_{\scriptstyle 0.093\text{-}0.166}$ | 0.202${}_{\scriptstyle 0.159\text{-}0.248}$ | 0.252${}_{\scriptstyle 0.205\text{-}0.305}$ |

**Table 7: The model being fine-tuned (LLaMA-3.1-8B-Instruct) is used to convert the enumerated diagnostic reasoning into a cohesive reasoning trace, as shown above. The model is asked to stitch the reasoning without introducing any additional information.**
| Diagnostic Reasoning | Stitched Reasoning |
| --- | --- |
| 1.<br>Benign soft-tissue tumors (lipoma or fibroma) were considered but deemed unlikely because “Lipomas and fibromas typically present as solid masses and are rarely associated with infection.”<br>2.<br>A canal of Nuck cyst was considered but excluded because “The canal of Nuck represents a cystic lesion extending into the inguinal region, which was ruled out based on imaging.”<br>3.<br>Bartholin’s gland abscess was supported by pelvic imaging showing a contrast-enhancing fluid collection at the gland sites; therefore, a diagnosis of bilateral Bartholin’s gland abscesses was made. | Lipomas and fibromas typically present as solid masses and are rarely associated with infection, making benign soft-tissue tumors such as lipomas or fibromas unlikely.<br>The canal of Nuck represents a cystic lesion extending into the inguinal region, which was ruled out based on imaging.<br>Pelvic imaging showed a contrast-enhancing fluid collection at the gland sites, supporting the diagnosis; therefore, a diagnosis of bilateral Bartholin’s gland abscesses was made. |

**Table 8: A single example from MedCaseReasoning of a case prompt, extract diagnostic reasoning, and final diagnosis.**
| Type | Content |
| --- | --- |
| Article Title | Sebaceous carcinoma of the breast predominantly characterized by intraductal growth: a case report |
| Journal | Surgical Case Reports |
| Article Link | [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7040145/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7040145/) |
| Case Prompt | A 47-year-old Japanese woman with no breast symptoms was referred after screening mammography revealed clustered pleomorphic calcifications in the left breast. Her mother had bilateral breast and ovarian cancer. On examination, there was no palpable mass or nipple discharge. Serum tumor markers (CEA, CA15-3, NCC-ST-439, BCA225) were within normal limits. Breast ultrasound showed a 13×12×7 mm irregular, hypoechoic mass with clear margins and internal high-echogenic foci in the left breast. MRI demonstrated a 14×11×12 mm lesion with early arterial enhancement in the same region. PET-CT revealed focal uptake (SUVmax 3.54) in the left breast without abnormal uptake elsewhere. Core-needle biopsy of the mass showed ductal carcinoma in situ, nuclear grade 2, negative for ER, PgR, and HER2, with a Ki-67 labeling index of 32.4%. Given her family history, hereditary breast and ovarian cancer syndrome was considered, but the patient declined genetic testing. The patient elected to undergo total left mastectomy with sentinel lymph node biopsy. |
| Diagnostic Reasoning | 1. Consideration of invasive ductal carcinoma — “SC is characterized by lobular forms or nests of tumor cells that exhibit sebaceous differentiation, which distinguishes SC from invasive ductal carcinoma (IDC).”<br>2. Consideration of glycogen-rich clear cell carcinoma — “Glycogen-rich clear cell carcinomas typically become periodic acid-Schiff (PAS)-positive and therefore can be distinguished from SC cells because they do not produce glycogen.” |
| Final Diagnosis | Sebaceous carcinoma |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 2
presentation: 1
questions: I want to see some discussions on the weaknesses.
soundness: 1
strengths: 1. The dataset may be potentially useful.
summary: The paper presents a dataset for evaluating LLMs’ medical reasoning capability, together with reasoning traces. The dataset originates from the PMC Open Subset, and case reports are filtered so that all selected case reports come with differential diagnosis, come with final diagnosis, etc. The paper seems to claim that the evaluation of the reasoning process. The dataset can be used for fine-tuning LLMs.
weaknesses: 1. I don’t see what the diagnostic reasoning (or reasoning traces) is. This idea should be fully explained/defined, as it seems important for the paper. 
2. I’m not sure about this, but it looks like the paper uses differential diagnosis as reasoning traces. From the terms “diagnostic reasoning” or “reasoning traces,” it sounds like the deduction process. Meanwhile, differential diagnosis looks like a comparison between possible diseases. It would be nice if the paper could provide if they can be seen as the same. Otherwise, the paper should explain why differential diagnosis can be used to evaluate the “diagnostic reasoning.”

Because the core part of the paper is not fully explained, I have to rate the paper negatively.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: Please address the weaknesses.
soundness: 2
strengths: This paper’s main contribution is the creation of a large, open-access benchmark that explicitly ties diagnostic answers to clinician-written reasoning, moving beyond accuracy-only evaluations that dominate MedQA/MMLU-style datasets. The authors assemble 14489 QA cases from PMC case reports via a multi-stage, clinician-validated pipeline and release a curated 897-case test split, which makes the task both realistic and unsaturated. The suite of results is also compelling: models perform far from ceiling (e.g., OpenAI o3 at 65% 10-shot on the test set) and the proposed training use of the dataset yields sizable, documented gains in both diagnosis and alignment with clinician rationales, with performance correlating across MedCaseReasoning and an external NEJM CPC set.
summary: This paper introduces an open-access benchmark termed MedCaseReasoning and also a corpus to evaluate diagnostic accuracy and alignment with clinician reasoning. The paper defines a metric to check if a model's explanation covers the enumerated reasons extracted from the source. Based on this benchmark, the paper evaluated the models underperforming on both diagnosis and reasoning alignment.
weaknesses: The data curation and evaluation loops lean heavily on LLMs (o4-mini for QA/trace extraction and scoring; Gemini-2.5-pro for quality checks; GPT-4-class models for judging), which raises concerns about hidden biases, propagation of subtle hallucinations, and circularity, especially given that only 100 cases received clinician spot-checks and that "pass/fail" filtering depends on model judgments. 

The central metric (reasoning recall against clinician bullets) captures coverage but not precision (spurious or incorrect steps), internal consistency, or causal structure. As a result, models could game the score by producing longer traces, a risk the paper acknowledges only indirectly via correlation with trace length. 

The reliance on LLM-as-a-judge to score diagnostic correctness also introduces measurement uncertainty, and comparisons are asymmetric because some frontier models do not expose full traces, preventing apples-to-apples reasoning evaluation. 

Finally, the benchmark focuses on single-snapshot diagnostic vignettes. It is unclear how gains from SFT on these traces transfer to multi-stage clinical workflows or to adjacent tasks (e.g., MedQA/MedXpertQA) beyond the NEJM CPC correlation.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. The dataset construction heavily relies on LLM-generated outputs. How do the authors ensure that these generated diagnostic QA pairs are free from bias and hallucination? Was the performance of o4-mini evaluated on the final MedCaseReasoning test set?
2. The reasoning metric currently measures only recall. Have the authors considered incorporating precision (or F1) to penalize spurious or incorrect reasoning steps that might inflate recall?
3. Reasoning recall is reported using the best of 10 generated responses. Could this setup overestimate true model performance? Would the authors provide the average recall across all generations, and discuss whether reasoning recall correlates with diagnostic accuracy?
4. While MedQA and MedXpertQA are marked as datasets without "expert reasoning," their questions still require reasoning from LLMs. How well do models fine-tuned on MedCaseReasoning generalize to such medical reasoning benchmarks that lack explicit rationales?
soundness: 2
strengths: - This paper focuses on a new task about reasoning alignment grounded in clinician-authored case reports. It provides large, open-access corpus spanning 800+ journals and 30+ specialties.
- A "reasoning recall" metric is proposed to evaluate the reasoning quality of the model outputs compared to constructed references.
- Experiments include a broad range of LLMs, showing that baseline models struggle on this benchmark.
- SFT on reasoning traces brings consistent improvements in both diagnostic accuracy and reasoning recall, and these improvements transfer to NEJM CPC.
summary: The authors introduce MedCaseReasoning, an open-access benchmark and training corpus built from 14489 clinician-authored case reports (with 897 test cases) to evaluate both diagnostic accuracy and alignment with clinician reasoning. They define a "reasoning recall" metric that checks whether a model's explanation covers the enumerated reasons extracted from the source case. Using this benchmark, frontier and open models underperform on both diagnosis and reasoning alignment. Supervised fine-tuning on MedCaseReasoning's reasoning traces boosts both accuracy and recall, and gains transfer to NEJM CPC cases.
weaknesses: - The data construction heavily relies on LLMs, raising concerns about bias and hallucination in generated content. Although manual evaluations were conducted by physicians on a small subset, the performance of o4-mini, which generated the diagnostic QA pairs, is not reported on the final MedCaseReasoning test data.
- The reasoning metric measures only recall, not precision. As a result, a model may "over-explain" by adding spurious or incorrect reasons without affecting the evaluation scores.
- The reasoning recall is computed by selecting the best response (if any) among 10 generated samples, which may overestimate true recall. Reporting the average recall across samples and discussing whether recall correlates with diagnostic accuracy would strengthen the analysis.
- While MedQA and MedXpertQA are labeled as having "no expert reasoning," their questions still require reasoning by LLMs. It remains unclear whether LLMs fine-tuned on MedCaseReasoning can generalize to other medical reasoning tasks such as MedQA or MedXpertQA.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: None
soundness: 2
strengths: This is a highly meaningful research direction, as it promotes a more comprehensive evaluation of LLM capabilities in the medical domain. The use of real clinical cases enhances the practical relevance and potential impact of the dataset.
summary: This study addresses a significant gap in the evaluation of medical LLMs by focusing not only on diagnostic accuracy but also on the clinical reasoning process. The introduction of the MedCaseReasoning dataset, built from real-world cases, is a valuable contribution that aligns well with practical clinical scenarios.
weaknesses: However, as a dataset construction paper, the study has several notable shortcomings:

(1) Unclear and Confusing Description of Dataset Construction:

+ In Section 2.1 (Data Curation), the "Candidate Selection" process is poorly explained and lacks clarity. Additionally, the 1–5 scoring scale mentioned in this section is not clearly defined. The prompts included in the appendix are also confusing and difficult to interpret.

+ The overall description of the dataset is insufficient. Key details—such as the number of diagnostic types included, the distribution of reasoning steps, and other structural characteristics—are missing. These are critical for understanding the dataset's scope and for enabling its broader adoption.

(2) Poorly Explained Evaluation Methods:

+ In Section 2.1, the authors state: "As expert verification on each step is prohibitively expensive, we perform our clinician validation only on the outputs of the last step." It is unclear how this approach aligns with later sections that refer to full reasoning trace evaluation, creating confusion about which steps were actually validated.

+ In Section 2.3 (Evaluating Reasoning Traces in Models), the authors use an LLM-as-a-judge approach, where o4-mini is prompted to return a JSON indicating whether ground-truth reasons are present in the model's reasoning trace. The evaluation criteria are vague: Is it based on exact match, keyword presence, or semantic similarity? This needs clarification.

(3) Insufficient Experimental Analysis:

+ The evaluation of the dataset is limited to few-shot and supervised fine-tuning (SFT) settings. A more thorough analysis—such as performance across different reasoning step lengths, case complexities, or diagnostic categories—would better demonstrate the dataset's characteristics and utility.

+ In Section 2.4, the authors mention using an LLM to "stitch" extracted reasoning points into a coherent trace. The justification for this method is unclear, and no ablation study is provided to compare this approach with alternative formatting strategies.

(4) Other Issues:

+ There is an inconsistency in the data split numbers: Figure 1 mentions "897 test cases and 13,092 training cases," which does not match the total of 14,489 cases mentioned earlier.

+ In Section 3.3 (Validity of Reasoning Recall Metric), the origin of the "89 pairs of groundtruth reasons" is not explained.