AnesSuite: A Comprehensive Benchmark and Dataset Suite for Anesthesiology Reasoning in LLMs

## Abstract

Abstract The application of large language models (LLMs) in the medical field has garnered significant attention, yet their reasoning capabilities in more specialized domains like anesthesiology remain underexplored. To bridge this gap, we introduce AnesSuite, the first comprehensive dataset suite specifically designed for anesthesiology reasoning in LLMs. The suite features AnesBench, an evaluation benchmark tailored to assess anesthesiology-related reasoning across three levels: factual retrieval (System 1), hybrid reasoning (System 1.x), and complex decision-making (System 2). Alongside this benchmark, the suite includes three training datasets that provide an infrastructure for continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning with verifiable rewards (RLVR). Leveraging this suite, we develop Morpheus, the first baseline model collection for anesthesiology reasoning. Despite undergoing limited training with SFT and group relative policy optimization (GRPO), Morpheus demonstrates substantial performance improvements, rivaling the performance of larger-scale models. Furthermore, through comprehensive evaluations and experiments, we analyze the key factors influencing anesthesiology reasoning performance, including model characteristics, training strategies and training data. Both AnesSuite and Morpheus will be open-sourced at https://github.com/MiliLab/AnesSuite .

## 1 Introduction

The success of LLMs has fundamentally reshaped the domain of medical artificial intelligence and catalyzed the creation of specialized medical models (Liu et al., 2024b). However, the reasoning ability of LLMs within medicine and its subdisciplines still need significant improvement, particularly when compared with general reasoning tasks such as mathematics and programming (Weng et al., 2023).

Anesthesiology, as a highly specialized discipline, requires extensive medical expertise and high-stakes clinical decision-making. In clinical practice, anesthesiology encompasses the simultaneous management of diverse and interdependent physiological systems—including airway and respiratory function, cardiovascular stability, electrolyte balance and sedation levels, as well as perioperative physiological fluctuations. This multifaceted responsibility means that anesthetic decision-making requires not only the factual recall of System 1 thinking but also the complex, context-aware judgment and higher-order reasoning of System 2 (Kahneman, 2011). Existing medical benchmarks and models often overlook the unique characteristics of anesthesiology (Panagoulias et al., 2024; Olatunji et al., 2024; Abacha et al., 2019; Liu et al., 2024c).
Others focus mainly on assessing factual recall (Li et al., 2023; Grzybowski et al., 2024; Wang et al., 2024a; Zhou et al., 2025).

To address this gap, we introduce the first comprehensive suite of benchmark and datasets, AnesSuite, dedicated to anesthesiology reasoning. This suite includes AnesBench, a bilingual and structured benchmark comprising 7,972 questions designed to evaluate both factual retrieval and clinical decision-making in anesthesiology, which accompanied by a full range of training resources: AnesCorpus, an extensive collection of over 2.4 million documents for continued pre-training; AnesQA, a SFT (Ouyang et al., 2022) dataset with over 20,713 question-answer (QA) pairs; AnesR1, a dataset with over 10,287 instances of verifiable anesthesiology questions with chain-of-thought annotations, which can be used for SFT (Ouyang et al., 2022) or RLVR (Lambert et al., 2024).

Leveraging AnesSuite, we propose Morpheus, the first collection of reasoning LLMs specifically designed for anesthesiology. Initialized from the Qwen2.5-7B and Qwen2.5-14B (Yang et al., 2024), it was trained on AnesR1 using SFT and GRPO (Shao et al., 2024). Despite undergoing limited training, experimental results demonstrate that Morpheus achieves substantial performance improvements on AnesBench, rivaling the performance of larger-scale models.

Moreover, using AnesBench, we evaluate a range of leading LLMs, including proprietary models such as GPT-4o (OpenAI, 2024a), Claude-3.7-Sonnet (Anthropic, 2025), Gemini-2.5-Flash and Gemini-2.5-Pro (Comanici et al., 2025). We also conducted extensive ablation and analytical experiments using the training datasets in AnesSuite. Our analysis yields several key insights: (1) Model performance benefits from larger model scales, whereas System 2 tasks exhibit comparatively smaller gains. Both systems experience diminishing marginal returns, with performance improvements decreasing as model scale increases. (2) The length of the Chain-of-Thought (Wei et al., 2022, CoT) is a critical factor in enhancing performance, particularly for System 2 tasks, as it enables more structured, step-by-step reasoning. (3) Language transferability remains a significant factor limiting the performance of multilingual models. (4) Model’s multilingual anesthesiology reasoning capabilities are highly sensitive to the training data of CPT phase, necessitating careful corpus management.
(5) Data from the general medical domain serves as a valuable complement to anesthesiology-specific data, enhancing overall reasoning performance.

Our contributions are threefold: (1) Dataset Suite Construction: We introduce AnesSuite, the first comprehensive dataset suite for anesthesiology reasoning. It includes AnesBench, a structured, cross-lingual benchmark for evaluating factual retrieval and clinical decision-making, along with three training datasets supporting CPT, SFT, and RLVR. (2) Baseline Model: We present Morpheus, the first collection of reasoning LLMs for anesthesiology. With SFT and GRPO training on the AnesR1 dataset, it achieves substantial performance improvements, underscoring the value of specialized, domain-specific training. (3) Comprehensive Evaluation and Analysis: Through systematic evaluations and extensive ablation studies, we provide an in-depth analysis of factors influencing anesthesiology reasoning, such as model characteristics, training strategies and training data. Overall, we are committed to leveraging our dataset suite, baseline models, and in-depth analysis collectively provide valuable insights for developing LLMs with more enhanced anesthesiology reasoning capabilities. We will publicly release AnesSuite, Morpheus, and all associated resources.

## 2 Related Work

### 2.1 Reasoning LLMs

Reasoning refers to solving problems involve complex, multi-step processes with intermediate steps (Li et al., 2025b). The emergence of reasoning LLMs such as OpenAI o1/o3 (OpenAI, 2024c; 2025) and Deepseek R1 (Guo et al., 2025) has spurred interest in enhancing LLM’s reasoning ability.
Recent research indicates that LLMs tend to generate longer outputs when solving mathematical or reasoning problems (Xiang et al., 2025). Model performance benefits from scaling the length of the reasoning process (Tang et al., 2025; Huang et al., 2025).
Models that exhibit strong exploratory tendencies or incorporate intrinsic verification mechanisms in their outputs often demonstrate enhanced reasoning capabilities (Xiang et al., 2025; Guo et al., 2025). Moreover, different model scales have been demonstrated to exhibit significant differences in training dynamics and reasoning characteristics (OpenAI, 2024c; 2025; Zhao et al., 2024). On the other hand, training and inference paradigms, such as reinforcement learning fine-tuning (Luong et al., 2024; Guo et al., 2025; Team et al., 2025b, RFT), self-improvement methods (Zelikman et al., 2022; Gulcehre et al., 2023; Singh et al., 2024), and structured search techniques exemplified by MCTS (Bi et al., 2025; Qiu et al., 2024; Li et al., 2025a), have shown significant development potential.

### 2.2 Anesthesia-related Benchmarks and LLMs

Anesthesiology, due to its unique interdisciplinary nature, is often implicitly classified under surgery, dentistry, or similar categories (Panagoulias et al., 2024; Olatunji et al., 2024; Abacha et al., 2019), even in benchmarks that categorize based on different departments. Some benchmarks explicitly treat anesthesiology as a separate category (Grzybowski et al., 2024; Wang et al., 2024a), while some studies employ existing anesthesiology-related questions (Khan et al., 2024; Angel et al., 2022; Shay et al., 2023; Kinoshita et al., 2024; Patnaik & Hoffmann, 2023) (e.g., exam questions from American Society of Anesthesiologists, ASA and Japanese Society of Anesthesiologists, JSA) to evaluate LLMs. However, the overall scale of these evaluations for anesthesiology remains very limited. Chinese Anesthesiology Benchmark (Zhou et al., 2025, CAB) is the first to focus primarily on anesthesiology. Nevertheless, CAB’s primary emphasis on factual retrieval questions results in insufficient exploration of the unique challenges in anesthetic reasoning and decision-making. Additionally, its restriction to the Chinese language further limits its applicability and impact. Moreover, Hypnos and some works (Wang et al., 2024b; Guthrie et al., 2024) attempts to use anesthesiology-related corpora for supervised fine-tuning to improve question-answering abilities, but they have not significantly enhanced the reasoning capabilities.

## 3 AnesSuite Dataset

We propose AnesSuite, the first comprehensive dataset suite specifically designed for reasoning and decision-making in anesthesiology. This suite includes AnesBench, a cross-lingual structured benchmark. In addition, AnesSuite includes three training datasets, AnesCorpus, AnesQA, and AnesR1, which collectively support a range of fine-tuning and application in anesthesiology.

### 3.1 Datasets Overview

Figure: Table 1: Overview of datasets in AnesSuite.
[Uncaptioned image]: x2.png

AnesBench is a bilingual benchmark featuring parallel English and Chinese subsets, containing 4,418 and 3,554 multiple-choice questions (MCQs), respectively. Its core design is a three-level of cognitive demand: System 1, focused on factual recall; System 2, involving complex reasoning; and System 1.x, a hybrid category merging recall with elementary reasoning (Saha et al., 2025). Critically, shown in Fig. [2](https://arxiv.org/html/2504.02404v2#S3.F2), questions demanding higher-order cognitive skills (System 1.x and System 2) constitute a substantial 20-30% of the total items. This hierarchical structure and emphasis on complex tasks, can provide a nuanced methodology for evaluating the advanced anesthesiological reasoning capabilities of LLMs.

Complementing the benchmark are three training datasets designed to support model development. AnesCorpus is a large-scale text collection for continued pre-training, comprising over 1.8 million English and 0.6 million Chinese documents pertinent to anesthesiology. AnesQA contains 20,713 English QA pairs, where each question is annotated with one of five question type labels to enable targeted fine-tuning, as presented in Fig. [4](https://arxiv.org/html/2504.02404v2#S3.F4). AnesR1 comprises a total of 10,287 items, distributed across Chinese and English subsets. Each item consists of a verifiable MCQ accompanied by a detailed CoT reasoning process. Like AnesBench, it also employs the three-level cognitive demand framework. The inclusion of detailed CoT reasoning is a defining feature of AnesR1, which is reflected in its data structure. As illustrated in Fig. [4](https://arxiv.org/html/2504.02404v2#S3.F4), this leads to substantially longer answers in AnesR1 compared to the concise answers in AnesQA, even though question lengths remain comparable across AnesBench, AnesQA, and AnesR1. Collectively, these training datasets provide a foundational infrastructure that supports the entire workflow, from CPT and SFT to RLVR.

Figure: Table 2: Number of choices in AnesBench and AnesR1.
[Uncaptioned image]: x3.png

### 3.2 Datasets Curation

AnesBench was constructed from authoritative sources, including American Board of Anesthesiology (ABA) examination materials, standardized textbooks, and validated online assessment tools. After collection, each item was annotated for its cognitive demand level using the DeepSeek-R1 (Guo et al., 2025). This annotation was performed via a prompting strategy that incorporated comprehensive guidelines and few-shot examples (the complete prompt is included in the Appendix [A.2](https://arxiv.org/html/2504.02404v2#A1.SS2)). For quality assurance, 60% of the items were randomly chosen for manual review. Subsequently, we conducted a data contamination (whether the questions have been seen in each model training) analysis using a specialized algorithm (Ni et al., 2025) for MCQs to verify that data contamination was minimal. The detailed results of the manual verification and the data leakage analysis are presented in the Appendix [A.1](https://arxiv.org/html/2504.02404v2#A1.SS1).

To create AnesCorpus, we sourced data from two large-scale web datasets, namely Fineweb(^1^11[Fineweb: https://huggingface.co/datasets/HuggingFaceFW/fineweb](https://huggingface.co/datasets/HuggingFaceFW/fineweb)) and Chinese Fineweb(^2^22[Chinese Fineweb: https://huggingface.co/datasets/opencsg/chinese-fineweb-edu](https://huggingface.co/datasets/opencsg/chinese-fineweb-edu)). We employed a frequency-based approach to filter this extensive collection. Specifically, we curated a list of keywords pertinent to anesthesia and pain management and categorized them into two groups (the complete list is available in the Appendix [A.4](https://arxiv.org/html/2504.02404v2#A1.SS4)). A document was included in AnesCorpus only if it met two criteria: (1) at least one keyword from the first group appeared once per 4,000 characters, and (2) the document contained at least one keyword from the second group.
To prevent contamination from this pre-training corpus to the AnesBench benchmark, we implemented a two-stage decontamination process. The initial stage used a rapid n-gram-based screening to identify documents with potential content overlap. In the second stage, these flagged documents were subjected to a more granular analysis using Longest Common Substring (LCS). In line with previous studies, such as HuatuoGPT-o1 (Chen et al., 2024) and Med-PaLM 2 (Singhal et al., 2025), documents with an LCS exceeding 64 characters when compared to a question in AnesBench were removed. A detailed description of this decontamination protocol is provided in the Appendix [A.4](https://arxiv.org/html/2504.02404v2#A1.SS4).

The construction of AnesQA began with sourcing anesthesiology-related research papers from PubMed (^3^33[PubMed: https://pubmed.ncbi.nlm.nih.gov/](https://pubmed.ncbi.nlm.nih.gov/)). These papers underwent an extensive cleaning process—including content removal, format standardization, and relevance filtering—to yield high-quality, domain-specific text. We then employed an LLM-based automated pipeline to generate question-answer pairs. This pipeline involved segmenting the cleaned papers into text segments based on length and section structure. For each segment, LLaMA3.3-70B-Instruct (Grattafiori et al., 2024) was utilized to generate a self-contained question. Subsequently, Qwen2.5-72B-Instruct (Yang et al., 2024) was employed to select the most informative and domain-relevant questions and to generate their corresponding answers. The complete prompts used in this process are detailed in the Appendix [A.5](https://arxiv.org/html/2504.02404v2#A1.SS5). While this dual-model approach can partially mitigate the inherent biases of LLMs, the generated content may still contain formatting errors or hallucinations. To address this, we conducted a further manual verification stage: we inspected a sample of the LLM outputs to identify common errors, then applied filters to remove all QA pairs exhibiting these issue.

The construction of AnesR1 followed a procedure similar to that of AnesBench, starting with the collection of anesthesiology multiple-choice questions from a distinct set of authoritative sources. During data deduplication, we implemented a process mirroring the second stage of the AnesCorpus decontamination: any question with an LCS greater than 64 characters when compared to an item in AnesBench was flagged for manual inspection. Questions deemed highly similar were subsequently eliminated. For each validated question, reasoning trajectories were generated using DeepSeek-R1 (Guo et al., 2025). A rejection sampling protocol was then enforced, accepting only those trajectories that culminated in the correct answer. If three sampling attempts failed to produce a correct solution, the source question was excluded from the final AnesR1 dataset.

## 4 Evaluation and Baseline Model

### 4.1 Evaluation

#### Benchmarked LLMs.

We conducted a comprehensive zero-shot evaluation on over 50 LLMs from diverse sets. The evaluation encompassed four leading proprietary models, namely GPT-4o (OpenAI, 2024a), Gemini-2.5-Flash, Gemini-2.5-Pro (Comanici et al., 2025), and Claude-3.7-Sonnet (Anthropic, 2025). The selection of open-source models included both domain-specific LLMs, such as HuatuoGPT-o1 (Chen et al., 2024) and BioMistral (Labrak et al., 2024), as well as prominent general-purpose LLMs like DeepSeek-R1/V3 (Guo et al., 2025; Liu et al., 2024a), Qwen3 (Team, 2025), and Llama4 (Meta, 2025). To facilitate a more granular analysis, we included models of varying parameter scales from each open-source family. While the main text presents a selection of results, the complete list of models and their corresponding evaluation outcomes are detailed in the Appendix [B.1](https://arxiv.org/html/2504.02404v2#A2.SS1).

**Table 3: Main Evaluation Results on AnesBench. The highest and second-highest accuracies in each column are highlighted in bold and underlined, respectively. Complete results are in Appendix [B.3](https://arxiv.org/html/2504.02404v2#A2.SS3).**
| Model |  | \cellcolorblue!10AnesBench-English |  | \cellcolorgreen!10AnesBench-Chinese |  | Avg. |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Sys1 | Sys1.x | Sys2 | Total |  | Sys1 | Sys1.x | Sys2 | Total |  |  |  |
| \rowcoloryellow!10    Medical Specific LLMs |  |  |  |  |  |  |  |  |  |  |  |  |
| BioMistral 7B Chat |  | 0.43 | 0.30 | 0.32 | 0.39 |  | 0.24 | 0.25 | 0.16 | 0.24 |  | 0.31 |
| HuatuoGPT-o1 7B |  | 0.56 | 0.45 | 0.38 | 0.52 |  | 0.73 | 0.59 | 0.55 | 0.71 |  | 0.61 |
| HuatuoGPT-o1 8B |  | 0.58 | 0.46 | 0.39 | 0.53 |  | 0.58 | 0.46 | 0.46 | 0.56 |  | 0.54 |
| HuatuoGPT-o1 70B |  | 0.70 | 0.58 | 0.48 | 0.65 |  | 0.71 | 0.57 | 0.49 | 0.68 |  | 0.66 |
| HuatuoGPT-o1 72B |  | 0.71 | 0.61 | 0.48 | 0.67 |  | 0.79 | 0.67 | 0.61 | 0.76 |  | 0.71 |
| \rowcoloryellow!10    Open-Source LLMs |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen3-0.6B |  | 0.39 | 0.31 | 0.26 | 0.36 |  | 0.34 | 0.29 | 0.22 | 0.35 |  | 0.31 |
| Qwen3-1.7B |  | 0.48 | 0.37 | 0.30 | 0.44 |  | 0.53 | 0.39 | 0.34 | 0.50 |  | 0.50 |
| Qwen3-4B |  | 0.60 | 0.46 | 0.34 | 0.54 |  | 0.64 | 0.66 | 0.50 | 0.37 |  | 0.58 |
| Qwen3-8B |  | 0.65 | 0.50 | 0.40 | 0.60 |  | 0.72 | 0.44 | 0.39 | 0.70 |  | 0.65 |
| Qwen3-14B |  | 0.70 | 0.57 | 0.45 | 0.65 |  | 0.79 | 0.63 | 0.52 | 0.76 |  | 0.70 |
| Qwen3-32B |  | 0.72 | 0.64 | 0.48 | 0.68 |  | 0.81 | 0.64 | 0.57 | 0.78 |  | 0.70 |
| Qwen3-30B-A3B |  | 0.73 | 0.60 | 0.48 | 0.68 |  | 0.77 | 0.62 | 0.50 | 0.74 |  | 0.61 |
| Qwen3-235B-A22B |  | 0.78 | 0.67 | 0.57 | 0.74 |  | 0.85 | 0.72 | 0.58 | 0.81 |  | 0.77 |
| Llama-4-Scout-17B-16E |  | 0.77 | 0.66 | 0.55 | 0.72 |  | 0.80 | 0.66 | 0.55 | 0.77 |  | 0.74 |
| Llama-4-Maverick-17B-128E |  | 0.83 | 0.73 | 0.64 | 0.79 |  | 0.86 | 0.72 | 0.59 | 0.83 |  | 0.81 |
| GPT-OSS-20B |  | 0.70 | 0.60 | 0.52 | 0.66 |  | 0.71 | 0.58 | 0.49 | 0.68 |  | 0.67 |
| GPT-OSS-120B |  | 0.80 | 0.72 | 0.62 | 0.76 |  | 0.80 | 0.64 | 0.54 | 0.76 |  | 0.76 |
| DeepSeek-V3 |  | 0.77 | 0.69 | 0.55 | 0.73 |  | 0.87 | 0.75 | 0.60 | 0.84 |  | 0.78 |
| DeepSeek-R1 |  | 0.85 | 0.78 | 0.70 | 0.82 |  | 0.86 | 0.77 | 0.61 | 0.83 |  | 0.82 |
| \rowcoloryellow!10    Proprietary LLMs |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o |  | 0.81 | 0.72 | 0.59 | 0.77 |  | 0.79 | 0.64 | 0.52 | 0.76 |  | 0.76 |
| Gemini-2.5-Flash |  | 0.84 | 0.76 | 0.68 | 0.81 |  | 0.84 | 0.72 | 0.59 | 0.81 |  | 0.81 |
| Gemini-2.5-Pro |  | 0.89 | 0.82 | 0.77 | 0.86 |  | 0.88 | 0.75 | 0.60 | 0.85 |  | 0.85 |
| Claude-3.7-Sonnet |  | 0.80 | 0.73 | 0.63 | 0.77 |  | 0.82 | 0.65 | 0.55 | 0.78 |  | 0.77 |

#### Supplementary Evaluation

To further refine our assessment, we introduced a supplementary open-ended subset, which focused on open-ended questions in anesthesiology, to capture and analyze the more nuanced reasoning capabilities of LLMs within the anesthesiology domain. Additionally, we conducted a comparative analysis using the MCQ subset from CAB (Zhou et al., 2025) against AnesBench. The detailed methodologies and results for these supplementary evaluation are available in the Appendix [G](https://arxiv.org/html/2504.02404v2#A7).

#### Implementation details.

The metric reported is Accuracy on multiple-choice questions. To ensure reproducibility, all models were evaluated with the temperature set to 0 and maximum output length set to 2048 tokens. We employed a Zero-Shot CoT prompting strategy to encourage structured, step-by-step reasoning (Wei et al., 2022). A detailed description of the experimental setting and the prompt is provided in Appendix [B.2](https://arxiv.org/html/2504.02404v2#A2.SS2).

#### Main Results.

Tab. [4.1](https://arxiv.org/html/2504.02404v2#S4.SS1.SSS0.Px1) presents the main evaluation results on the AnesBench. We observe that complex anesthesiology reasoning remains a significant challenge for contemporary LLMs. On the AnesBench, most open-source models achieve scores below 0.5 on System 2 questions. Even proprietary models, such as GPT-4o and Claude-3.7-Sonnet, only achieve accuracies of around 0.5 to 0.6. Furthermore, a consistent trend across most models is that performance on System 1.x and System 2 tasks is markedly lower than on System 1 tasks. This discrepancy suggests that the principal challenge for current LLMs is not a simple deficit in anesthesiological knowledge, but rather an underdeveloped ability to apply this knowledge to complex reasoning problems. Notably, medical specific LLMs do not demonstrate a significant advantage over recent general-purpose reasoning models in our evaluation. This finding underscores the specialized nature of anesthesiological reasoning, distinguishing it from general medical domain.

### 4.2 Baseline Model

We developed baseline models by fine-tuning Qwen2.5-7B and Qwen2.5-14B (Yang et al., 2024) on the AnesR1. Both models underwent SFT followed by GRPO and are designated Morpheus-7B and Morpheus-14B, respectively. For comparison, we also present counterpart models trained for an equivalent number of steps using SFT alone. Complete training details are provided in Appendix [C.2](https://arxiv.org/html/2504.02404v2#A3.SS2).

The evaluation results for Morpheus are presented in Tab. [4](https://arxiv.org/html/2504.02404v2#S4.T4). A key observation is that, after GRPO training, the model achieves consistent improvements across various systems, demonstrating the effectiveness of the GRPO algorithm in enhancing reasoning capabilities. Notably, following SFT and GRPO training on AnesR1, Morpheus-7B and Morpheus-14B reach performance levels comparable to Qwen2.5-14B-Instruct and Qwen2.5-32B-Instruct, respectively. This further indicates that AnesR1 is effective in advancing complex anesthesiology reasoning abilities.

**Table 4: Evaluation Results of Morpheus on AnesBench.**
| Model | Stage | AnesBench-English | AnesBench-Chinese | Avg. |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SFT | GRPO | Sys1 | Sys1.x | Sys2 | Total | Sys1 | Sys1.x | Sys2 | Total |  |  |
| Qwen2.5-7B-Instruct | - | - | 0.56 | 0.44 | 0.36 | 0.51 | 0.69 | 0.55 | 0.55 | 0.66 | 0.59 |
| Morpheus-7B (SFT only) | ✓ | ✗ | 0.59 | 0.45 | 0.35 | 0.54 | 0.58 | 0.42 | 0.47 | 0.56 | 0.54 |
| Morpheus-7B | ✓ | ✓ | 0.61 | 0.48 | 0.40 | 0.56 | 0.73 | 0.55 | 0.50 | 0.70 | 0.63 |
| Qwen2.5-14B-Instruct | - | - | 0.61 | 0.52 | 0.41 | 0.57 | 0.75 | 0.63 | 0.53 | 0.72 | 0.64 |
| Morpheus-14B (SFT only) | ✓ | ✗ | 0.65 | 0.51 | 0.38 | 0.60 | 0.57 | 0.51 | 0.45 | 0.55 | 0.57 |
| Morpheus-14B | ✓ | ✓ | 0.68 | 0.55 | 0.47 | 0.63 | 0.78 | 0.62 | 0.53 | 0.75 | 0.69 |
| Qwen2.5-32B-Instruct | - | - | 0.65 | 0.55 | 0.44 | 0.61 | 0.79 | 0.65 | 0.54 | 0.76 | 0.68 |

## 5 Ablation and Analysis

In this section, we present a series of ablation studies and analyses to provide insights for developing advanced LLMs for anesthesiology reasoning. Our investigation examines two key aspects: model characteristics, training methodologies and data. Furthermore, the majority of the insights and claims presented in this section have undergone statistical significance testing. The full results are provided in Appendix [F](https://arxiv.org/html/2504.02404v2#A6).

### 5.1 Model Characteristic

#### Performance Across Model Scale.

Models are arranged in increasing order of their accuracy on the AnesBench-English subset. As depicted in Fig. [6](https://arxiv.org/html/2504.02404v2#S5.F6), there is a strong positive correlation between model performance and scale, which exhibits diminishing marginal returns; larger increases in model scale correspond to progressively smaller performance gains. Furthermore, the fitted-line slopes for System1 and System1.x are relatively close. In contrast, the slope for System2 is significantly lower than System1.x, indicating that performance gains from increasing model size are markedly smaller for System2.

Figure: Figure 5: Impact of Model Scale. Colors denote model scale (model index) and problem type (scatter points). Models are sorted by overall score in ascending order.
Refer to caption: x5.png

Figure: Figure 7: Impact of Output Length Shapes and colors denote model families and scales.
Refer to caption: x7.png

#### Performance Across Language.

To evaluate model performance across different linguistic settings, we performed a cross-translation of the AnesBench-English and AnesBench-Chinese subsets using GPT-4o (OpenAI, 2024a). Specifically, the English subset was translated into Chinese, and the Chinese subset into English. This process resulted in each subset having versions in both languages, enabling direct cross-lingual comparisons. The prompts and details are detailed in Appendix [C.1](https://arxiv.org/html/2504.02404v2#A3.SS1).

As shown in Fig. [6](https://arxiv.org/html/2504.02404v2#S5.F6), Qwen2.5-7B-based and other models exhibit minimal disparity between English and Chinese performance. In contrast, Llama-3.1-8B-based models, despite claiming multilingual support (Grattafiori et al., 2024), perform notably worse in Chinese. These results indicate that language transferability remains a key factor influencing multilingual model performance. According to the theory proposed in (Chen et al., 2025), once an LLM reaches a stable language learning stage, different languages form independent knowledge systems rather than relying on translation. At this stage, a lack of domain-specific knowledge within the language-specific knowledge system can lead to substantial performance disparities across languages. Therefore, we recommend supplementing bilingual domain-specific knowledge during the different training stage to mitigate cross-linguistic performance gaps.

#### Performance Across Output Length

By comparing output length and accuracy across three levels of questions in AnesBench-English, we discovered: (1) Models with longer CoT reasoning processes tend to exhibit superior response performance. As illustrated in the right subplot of Fig. [7](https://arxiv.org/html/2504.02404v2#S5.F7), for System2 questions, models achieve higher scores with longer outputs. (2) However, this trend is not pronounced in System1 and System1.x, which do not require extensive reasoning processes. As shown in the left and middle subplots of Fig. [7](https://arxiv.org/html/2504.02404v2#S5.F7), the scores of models are almost solely correlated with the size of the model.

### 5.2 Training Strategy and Data

This section focuses on two key research questions: (1) the effect of CPT training strategies on model performance, and (2) the impact of training data from different domains on the model’s performance.

To investigate these questions, we subjected models to SFT for an equivalent number of steps using two distinct datasets: AnesQA, representing the anesthesiology-specific domain, and the bilingual Medical-o1 (Chen et al., 2024), representing the general medical domain. To isolate the impact of CPT, this process was applied to models both with and without prior continued pre-training on AnesCorpus. Notably, since AnesQA lacks native Chinese data, we translated the dataset using GPT-4o-mini (OpenAI, 2024b). This step was crucial to align its format with the setting of Medical-o1 and to ensure the validity of evaluations on AnesBench-Chinese. Further details on the training and translation are provided in Appendix [C.3](https://arxiv.org/html/2504.02404v2#A3.SS3) and Appendix [C.1](https://arxiv.org/html/2504.02404v2#A3.SS1). The results, presented in Tab. [5](https://arxiv.org/html/2504.02404v2#S5.T5), yield two key insights.

**Table 5: Effectiveness of Training Strategies and Data The Qwen2.5-7B-Base-CPT model is trained on our AnesCorpus, with Qwen2.5-7B-Base serving as the foundation model.**
| Model | SFT Data | AnesBench |  |  |
| --- | --- | --- | --- | --- |
| AnesQA | Medical-o1 | English | Chinese |  |
| Qwen2.5-7B-Base | ✓ | ✗ | 49.3 | 64.9 |
| ✗ | ✓ | 49.1 | 63.0 |  |
| ✓ | ✓ | 49.7 | 65.9 |  |
| Qwen2.5-7B-Base-CPT | ✓ | ✗ | 49.7 | 50.7 |
| ✗ | ✓ | 50.7 | 59.4 |  |
| ✓ | ✓ | 51.2 | 60.0 |  |

#### Complementarity of AnesQA and Medical-o1.

The results underscore the complementary nature of AnesQA and Medical-o1 for fine-tuning. As demonstrated, integrating AnesQA with Medical-o1 leads to varying degrees of performance improvement across models. This finding suggests that, although Medical-o1 is not exclusively focused on anesthesiology, its inclusion of general medical knowledge is advantageous for enhancing the model’s anesthesiological capabilities, thereby complementing the specialized content provided by AnesQA. This synergy highlights that, even within highly specialized domains such as anesthesiology, the incorporation of broader medical knowledge remains essential.

#### Impact of Continued Pre-training.

It was found that continued pre-training with AnesCorpus has a dichotomous effect on multilingual performance. While it enhances performance on the AnesBench-English, raising the maximum accuracy from 49.7% on the base model to 51.2% after CPT, it concurrently degrades performance on the AnesBench-Chinese. This demonstrates that domain-specific pre-training does not uniformly guarantee improvements across all languages. We hypothesize that this disparity stems from the model’s inability to effectively transfer knowledge across languages; the gains in English proficiency are not successfully propagated to the Chinese counterpart, potentially inducing catastrophic forgetting. Consequently, we recommend that the linguistic distribution within CPT corpora be carefully managed to avoid compromising the model’s multilingual versatility.

## 6 Limitations

While AnesSuite contributes to anesthesiology reasoning with LLMs, several limitations remain. First, the System 2 questions are constructed from abstract scenarios rather than real clinical cases. Second, future iterations of AnesSuite should consider incorporating additional modalities, to better reflect real-world clinical environments.

## 7 Conclusion

This paper introduces AnesSuite, the first comprehensive suite dedicated to anesthesiology reasoning. AnesSuite features a structured evaluation benchmark and three training datasets, establishing a complete infrastructure for key development stages: CPT, SFT, and RLVR. We also developed Morpheus, the first strong baseline model collection for this domain. Furthermore, extensive ablation studies and analyses yield several key insights for improving these specialized reasoning capabilities. We hope this work will serve as a cornerstone for developing LLMs with enhanced anesthesiology reasoning capabilities.

Appendix

## References

- Abacha et al. (2019)
Asma Ben Abacha, Yassine Mrabet, Mark Sharp, Travis R Goodwin, Sonya E Shooshan, and Dina Demner-Fushman.
Bridging the gap between consumers’ medication questions and trusted answers.
In *MEDINFO*, 2019.
- Abdin et al. (2024)
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al.
Phi-4 technical report.
*arXiv preprint arXiv:2412.08905*, 2024.
- Angel et al. (2022)
Mirana C Angel, Joseph B Rinehart, Maxime P Canneson, and Pierre Baldi.
Clinical knowledge and reasoning abilities of ai large language models in anesthesiology: A comparative study on the american board of anesthesiology examination.
*Anesthesia & Analgesia*, 139(2):10–1213, 2022.
- Ankit Pal (2024)
Malaikannan Sankarasubbu Ankit Pal.
Openbiollms: Advancing open-source large language models for healthcare and life sciences.
[https://huggingface.co/aaditya/OpenBioLLM-Llama3-70B](https://huggingface.co/aaditya/OpenBioLLM-Llama3-70B), 2024.
- Anthropic (2025)
Anthropic.
Claude 3.7 sonnet: Frontier reasoning made practical, September 2025.
URL [https://www.anthropic.com/news/claude-3-7-sonnet](https://www.anthropic.com/news/claude-3-7-sonnet).
- Baichuan (2023)
Baichuan.
Baichuan 2: Open large-scale language models.
*arXiv preprint arXiv:2309.10305*, 2023.
URL [https://arxiv.org/abs/2309.10305](https://arxiv.org/abs/2309.10305).
- Bi et al. (2025)
Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang.
Forest-of-thought: Scaling test-time compute for enhancing llm reasoning.
In *ICML*, 2025.
- Cai et al. (2024)
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al.
Internlm2 technical report.
*arXiv preprint arXiv:2403.17297*, 2024.
- Chen et al. (2025)
Jiawei Chen, Wentao Chen, Jing Su, Jingjing Xu, Hongyu Lin, Mengjie Ren, Yaojie Lu, Xianpei Han, and Le Sun.
The rise and down of babel tower: Investigating the evolution process of multilingual code large language model.
In *ICLR*, 2025.
- Chen et al. (2024)
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang.
Huatuogpt-o1, towards medical complex reasoning with llms.
*arXiv preprint arXiv:2412.18925*, 2024.
- Comanici et al. (2025)
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025.
- ContactDoctor (2024)
ContactDoctor.
Bio-medical: A high-performance biomedical language model.
https://huggingface.co/ContactDoctor/Bio-Medical-Llama-3-8B, 2024.
- GLM et al. (2024)
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al.
Chatglm: A family of large language models from glm-130b to glm-4 all tools.
*arXiv preprint arXiv:2406.12793*, 2024.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Grzybowski et al. (2024)
Łukasz Grzybowski, Jakub Pokrywka, Michał Ciesiółka, Jeremi I Kaczmarek, and Marek Kubis.
Polish medical exams: A new dataset for cross-lingual medical knowledge transfer assessment.
*arXiv preprint arXiv:2412.00559*, 2024.
- Gulcehre et al. (2023)
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al.
Reinforced self-training (rest) for language modeling.
*arXiv preprint arXiv:2308.08998*, 2023.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al.
Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.
*Nature*, 645(8081):633–638, 2025.
- Guthrie et al. (2024)
Estefania Guthrie, Dominique Levy, and Gabriel Del Carmen.
The operating and anesthetic reference assistant (oara): A fine-tuned large language model for resident teaching.
*The American Journal of Surgery*, 234:28–34, 2024.
- Huang et al. (2023)
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, jiayi lei, Yao Fu, Maosong Sun, and Junxian He.
C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.
In *NeurIPS*, 2023.
- Huang et al. (2025)
Zhongzhen Huang, Gui Geng, Shengyi Hua, Zhen Huang, Haoyang Zou, Shaoting Zhang, Pengfei Liu, and Xiaofan Zhang.
O1 replication journey–part 3: Inference-time scaling for medical reasoning.
*arXiv preprint arXiv:2501.06458*, 2025.
- Kahneman (2011)
Daniel Kahneman.
Fast and slow thinking.
*Allen Lane and Penguin Books, New York*, 2011.
- Khan et al. (2024)
Adnan A Khan, Rayaan Yunus, Mahad Sohail, Taha A Rehman, Shirin Saeed, Yifan Bu, Cullen D Jackson, Aidan Sharkey, Feroze Mahmood, and Robina Matyal.
Artificial intelligence for anesthesiology board–style examination questions: Role of large language models.
*Journal of Cardiothoracic and Vascular Anesthesia*, 38(5):1251–1259, 2024.
- Kinoshita et al. (2024)
Michiko Kinoshita, Mizuki Komasaka, and Katsuya Tanaka.
Chatgpt’s performance on jsa-certified anesthesiologist exam.
*Journal of Anesthesia*, 38(2):282–283, 2024.
- Labrak et al. (2024)
Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour.
Biomistral: A collection of open-source pretrained large language models for medical domains.
*arXiv preprint arXiv:2402.10373*, 2024.
- Lambert et al. (2024)
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al.
Tulu 3: Pushing frontiers in open language model post-training.
*arXiv preprint arXiv:2411.15124*, 2024.
- Li et al. (2023)
Binbin Li, Tianxin Meng, Xiaoming Shi, Jie Zhai, and Tong Ruan.
Meddm: Llm-executable clinical guidance tree for clinical decision-making.
*arXiv preprint arXiv:2312.02441*, 2023.
- Li et al. (2024)
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin.
Cmmlu: Measuring massive multitask language understanding in chinese.
In *ACL*, 2024.
- Li et al. (2025a)
Jierui Li, Hung Le, Yingbo Zhou, Caiming Xiong, Silvio Savarese, and Doyen Sahoo.
Codetree: Agent-guided tree search for code generation with large language models.
In *NAACL*, 2025a.
- Li et al. (2025b)
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al.
From system 1 to system 2: A survey of reasoning large language models.
*arXiv preprint arXiv:2502.17419*, 2025b.
- Liu et al. (2024a)
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
Deepseek-v3 technical report.
*arXiv preprint arXiv:2412.19437*, 2024a.
- Liu et al. (2024b)
Lei Liu, Xiaoyan Yang, Junchi Lei, Xiaoyang Liu, Yue Shen, Zhiqiang Zhang, Peng Wei, Jinjie Gu, Zhixuan Chu, Zhan Qin, et al.
A survey on medical large language models: Technology, application, trustworthiness, and future directions.
*arXiv preprint arXiv:2406.03712*, 2024b.
- Liu et al. (2024c)
Mianxin Liu, Weiguo Hu, Jinru Ding, Jie Xu, Xiaoyang Li, Lifeng Zhu, Zhian Bai, Xiaoming Shi, Benyou Wang, Haitao Song, et al.
Medbench: A comprehensive, standardized, and reliable benchmarking system for evaluating chinese medical large language models.
*Big Data Mining and Analytics*, 7(4):1116–1128, 2024c.
- Luong et al. (2024)
Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li.
Reft: Reasoning with reinforced fine-tuning.
In *ACL*, 2024.
- Meta (2025)
Meta.
The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, April 2025.
URL [https://ai.meta.com/blog/llama-4-multimodal-intelligence/](https://ai.meta.com/blog/llama-4-multimodal-intelligence/).
- Ni et al. (2025)
Shiwen Ni, Xiangtao Kong, Chengming Li, Xiping Hu, Ruifeng Xu, Jia Zhu, and Min Yang.
Training on the benchmark is not all you need.
In *AAAI*, 2025.
- Olatunji et al. (2024)
Tobi Olatunji, Charles Nimo, Abraham Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, Chinemelu Aka, Folafunmi Omofoye, Foutse Yuehgoh, Timothy Faniran, et al.
Afrimed-qa: A pan-african, multi-specialty, medical question-answering benchmark dataset.
*arXiv preprint arXiv:2411.15640*, 2024.
- OpenAI (2024a)
OpenAI.
Hello gpt-4o, May 2024a.
URL [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/).
- OpenAI (2024b)
OpenAI.
Gpt-4o mini: advancing cost-efficient intelligence, July 2024b.
URL [https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/).
- OpenAI (2024c)
OpenAI.
Learning to reason with llms, September 2024c.
URL [https://openai.com/index/learning-to-reason-with-llms/](https://openai.com/index/learning-to-reason-with-llms/).
- OpenAI (2025)
OpenAI.
Openai o3-mini, January 2025.
URL [https://openai.com/index/openai-o3-mini/](https://openai.com/index/openai-o3-mini/).
- Ouyang et al. (2022)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe.
Training language models to follow instructions with human feedback.
In *NeurIPS*, 2022.
- Panagoulias et al. (2024)
Dimitrios P Panagoulias, Persephone Papatheodosiou, Anastasios P Palamidas, Mattheos Sanoudos, Evridiki Tsoureli-Nikita, Maria Virvou, and George A Tsihrintzis.
Cognet-md, an evaluation framework and dataset for large language model benchmarks in the medical domain.
*arXiv preprint arXiv:2405.10893*, 2024.
- Patnaik & Hoffmann (2023)
Sourav S Patnaik and Ulrike Hoffmann.
Comparison of chatgpt vs. bard to anesthesia-related queries.
*medRxiv*, pp. 2023–06, 2023.
- Qiu et al. (2024)
Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Huazheng Wang, Kaixuan Huang, Yue Wu, and Mengdi Wang.
Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling.
*arXiv preprint arXiv:2410.16033*, 2024.
- Qwen (2025)
Qwen.
Qwq-32b: Embracing the power of reinforcement learning, March 2025.
URL [https://qwenlm.github.io/zh/blog/qwq-32b/](https://qwenlm.github.io/zh/blog/qwq-32b/).
- Saha et al. (2025)
Swarnadeep Saha, Archiki Prasad, Justin Chih-Yao Chen, Peter Hase, Elias Stengel-Eskin, and Mohit Bansal.
System-1. x: Learning to balance fast and slow planning with language models.
In *ICLR*, 2025.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Shay et al. (2023)
Denys Shay, Bhawesh Kumar, David Bellamy, Anil Palepu, Mark Dershwitz, Jens M Walz, Maximilian S Schaefer, and Andrew Beam.
Assessment of chatgpt success with specialty medical knowledge using anaesthesiology board examination practice questions.
*BJA: British Journal of Anaesthesia*, 131(2):e31, 2023.
- Sheng et al. (2025)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
In *EuroSys*, 2025.
- Singh et al. (2024)
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al.
Beyond human data: Scaling self-training for problem-solving with language models.
*Transactions on Machine Learning Research*, 2024.
ISSN 2835-8856.
- Singhal et al. (2025)
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al.
Toward expert-level medical question answering with large language models.
*Nature Medicine*, 31(3):943–950, 2025.
- Tang et al. (2025)
Zeyu Tang, Zhenhao Chen, Loka Li, Xiangchen Song, Yunlong Deng, Yifan Shen, Guangyi Chen, Peter Spirtes, and Kun Zhang.
Reflection-window decoding: Text generation with selective refinement.
In *ICML*, 2025.
- Team et al. (2024)
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al.
Gemma 2: Improving open language models at a practical size.
*arXiv preprint arXiv:2408.00118*, 2024.
- Team et al. (2025a)
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al.
Gemma 3 technical report.
*arXiv preprint arXiv:2503.19786*, 2025a.
- Team et al. (2025b)
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al.
Kimi k1. 5: Scaling reinforcement learning with llms.
*arXiv preprint arXiv:2501.12599*, 2025b.
- Team (2025)
Qwen Team.
Qwen3, April 2025.
URL [https://qwenlm.github.io/blog/qwen3/](https://qwenlm.github.io/blog/qwen3/).
- Wang et al. (2025)
Guoxin Wang, Minyu Gao, Shuai Yang, Ya Zhang, Lizhi He, Liang Huang, Hanlin Xiao, Yexuan Zhang, Wanyue Li, Lu Chen, et al.
Citrus: Leveraging expert cognitive pathways in a medical language model for advanced medical decision support.
*arXiv preprint arXiv:2502.18274*, 2025.
- Wang et al. (2024a)
Xidong Wang, Guiming Chen, Song Dingjie, Zhang Zhiyi, Zhihong Chen, Qingying Xiao, Junying Chen, Feng Jiang, Jianquan Li, Xiang Wan, et al.
Cmb: A comprehensive medical benchmark in chinese.
In *NAACL*, 2024a.
- Wang et al. (2023a)
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.
Self-consistency improves chain of thought reasoning in language models.
In *ICLR*, 2023a.
- Wang et al. (2023b)
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.
Self-consistency improves chain of thought reasoning in language models.
In *ICLR*, 2023b.
- Wang et al. (2024b)
Zhonghai Wang, Jie Jiang, Yibing Zhan, Bohao Zhou, Yanhong Li, Chong Zhang, Liang Ding, Hua Jin, Jun Peng, Xu Lin, et al.
Towards training a chinese large language model for anesthesiology.
*arXiv preprint arXiv:2403.02742*, 2024b.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou.
Chain-of-thought prompting elicits reasoning in large language models.
In *NeurIPS*, 2022.
- Weng et al. (2023)
Yixuan Weng, Bin Li, Fei Xia, Minjun Zhu, Bin Sun, Shizhu He, Kang Liu, and Jun Zhao.
Large language models need holistically thought in medical conversational qa.
*arXiv preprint arXiv:2305.05410*, 2023.
- Xiang et al. (2025)
Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al.
Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though.
*arXiv preprint arXiv:2501.04682*, 2025.
- Yang et al. (2024)
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report.
*arXiv preprint arXiv:2412.15115*, 2024.
- Young et al. (2024)
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al.
Yi: Open foundation models by 01. ai.
*arXiv preprint arXiv:2403.04652*, 2024.
- Yu et al. (2025)
Hongzhou Yu, Tianhao Cheng, Ying Cheng, and Rui Feng.
Finemedlm-o1: Enhancing the medical reasoning ability of llm from supervised fine-tuning to test-time training.
*arXiv preprint arXiv:2501.09213*, 2025.
- Zelikman et al. (2022)
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman.
Star: Bootstrapping reasoning with reasoning.
In *NeurIPS*, 2022.
- Zhang et al. (2024)
Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, Xingtai Lv, Jin-Fang Hu, Zhiyuan Liu, and Bowen Zhou.
Ultramedical: Building specialized generalists in biomedicine.
In *NeurIPS*, 2024.
- Zhao et al. (2024)
Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang.
Marco-o1: Towards open reasoning models for open-ended solutions.
*arXiv preprint arXiv:2411.14405*, 2024.
- Zheng et al. (2024a)
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al.
Sglang: Efficient execution of structured language model programs.
In *NeurIPS*, 2024a.
- Zheng et al. (2024b)
Yaowei Zheng, Richong Zhang, Junhao Zhang, YeYanhan YeYanhan, and Zheyan Luo.
Llamafactory: Unified efficient fine-tuning of 100+ language models.
In *ACL*, 2024b.
- Zhou et al. (2025)
Bohao Zhou, Yibing Zhan, Zhonghai Wang, Yanhong Li, Chong Zhang, Baosheng Yu, Liang Ding, Hua Jin, Weifeng Liu, Xiongbin Wang, et al.
Benchmarking medical llms on anesthesiology: A comprehensive dataset in chinese.
In *TETCI*. IEEE, 2025.

## Contents

## Appendix A Datasets Curation Details

### A.1 Data Leakage Analysis

Given the increasing volume of pre-training data utilized by large language models, coupled with the fact that the sources and specific acquisition methods are often regarded as core secrets and rarely disclosed, a pertinent issue arises: whether the questions in our benchmark have been inadvertently leaked into the pre-training data of various models. We have employed a method specifically designed for multiple-choice questions to detect potential data leakage (Ni et al., 2025). For a given multiple-choice instance $x=(x_{\text{stem}},o_{1},o_{2},\dots,o_{n})$, where $x_{\text{stem}}$ represents the question stem and $o_{1},o_{2},\dots,o_{n}$ are the options, we generate all possible permutations of the options, resulting in $\{x_{1},x_{2},\dots,x_{n!}\}$. For each permutation, the sequence is decomposed into tokens as $x=(t_{1},t_{2},\dots,t_{m})$. We then compute the confidence score for each sequence using the following method:

$$ $\displaystyle\text{Conf}_{\text{LLM}}(x)=\left(\prod_{i=\text{1}}^{m-1}p_{\text{LLM}}(t_{i}|x)\right)^{\frac{1}{m-1}}.$ (1) $$

If the original order $x_{0}$ has the highest confidence score among all permutations, we consider that the multiple-choice instance may have experienced data leakage. We randomly select 500 questions from AnesBench, excluding those with excessive options. Under the assumption that the LLM has no prior knowledge and calculates probabilities randomly, this method yields a potential leakage proportion of 0.04. As shown in Fig. [8](https://arxiv.org/html/2504.02404v2#A1.F8), most models exhibit a potential data leakage proportion around 0.10, with the exception of the Qwen and Yi model families, which are slightly higher (consistent with the evaluation results of MMLU(Wang et al., 2023a), CMMLU(Li et al., 2024), C-Eval(Huang et al., 2023), and CMB(Wang et al., 2024a) in the original text (Ni et al., 2025)). Overall, the low potential leakage proportions of most models on AnesBench validate the quality of our work.

Figure: Figure 8: Proportion of potential data leakage in the model. Each $\blacktriangle$ below the x-axis denotes a model family.
Refer to caption: x8.png

### A.2 Details of AnesBench and AnesR1 Annotation

We annotated AnesBench and AnesR1 using detailed few-shot prompts with the inference model DeepSeek-R1. The specific prompts are as follows.

### A.3 Details of Manual Inspection in AnesBench

#### General Review

To ensure the data quality of AnesBench, we selected approximately 60% of the questions for manual inspection. Specifically, the types of errors we identified and their proportions among the inspected questions are shown in Tab [6](https://arxiv.org/html/2504.02404v2#A1.T6). The proportions of the above three types of errors are all very low, which can basically guarantee the data quality of AnesBench. All the problematic questions we have checked above has been excluded from AnesBench.

**Table 6: Summary of Manual Inspection on AnesBench**
| Issue Type | Count | Percentage |
| --- | --- | --- |
| Format Error | 1 | 0.02% |
| Content Mistake | 5 | 0.1% |
| Missing Key Information | 11 | 0.22% |

### A.4 Details of AnesCorpus Curation

To construct AnesCorpus, we first retrieved anesthesia-related documents from FineWeb and Chinese FineWeb based on manually selected prompts. Then, through a two-stage data cleaning process on AnesBench, we finally obtained AnesCorpus. The two sets of keywords we used in the first stage are as follows. The first set of keywords is highly relevant to anesthesiology. The second set is closely related to perioperative management in anesthesiology, which helps us not only focus on theoretical knowledge of anesthesiology but also obtain more content related to clinical perioperative management.

**Table 7: Keywords used for AnesCorpus Construction**
| Group | Keyword |
| --- | --- |
| Keywords1 | 麻醉,神经阻滞,镇静 ,镇痛 ,anesthe,analge ,sedation ,nerve block |
| Keywords2 | 手术 , 围术期 ,surgery,surgical,operation, operative |

#### Decontamination protocol

We employed a two-stage decontamination protocol to eliminate potential data contamination. The first stage consisted of a preliminary screening based on n-gram overlap. Specifically, we constructed an inverted index of n-grams from the AnesBench evaluation dataset, mapping each n-gram fragment to the set of questions in which it appears. We then processed the initial corpus by querying each document’s n-grams against this index to compute the number of shared n-gram fragments with AnesBench. Documents exceeding a predefined threshold were flagged as potentially contaminated.

The second stage implemented a more stringent filtering procedure, following the methodology of HuatuoGPT-o1 (Chen et al., 2024). In this phase, any document exhibiting a LCS greater than 64 characters with any item in AnesBench was definitively excluded from the corpus. In accordance with this protocol, we set the character-level n-gram size to 35 and the overlap threshold to 9. During the first stage, we identified 359,598 suspicious candidates. In the second stage, we further screened and detected a total of 3,240 contaminated documents that exhibited an overlap with AnesBench greater than 64 characters.

### A.5 Details of AnesQA Curation

AnesQA was constructed using a multi-stage, LLM-based pipeline to transform existing anesthesiology literature into question-answer pairs. The generation process began by segmenting source documents into smaller chunks based on length and section structure. For each chunk, LLaMA3.3-70B-Instruct (Grattafiori et al., 2024) was prompted to generate self-contained questions. Subsequently, a distinct model, Qwen2.5-72B-Instruct (Yang et al., 2024), was employed to filter these questions, selecting only the most informative and domain-relevant ones to answer. This two-model approach was implemented to mitigate potential self-preference bias.

To address the risk of model-induced hallucinations, we instituted several quality assurance protocols, including model cross-validation and manual quality assessments on a sampled subset. The value of AnesQA is further substantiated by its demonstrated ability to enhance the reasoning performance of LLMs, as detailed in the main text. This finding serves as an indirect validation of the dataset’s quality and its utility for domain adaptation. The specific prompts used for the generation and answering stages are provided below.

To ensure data quality and integrity, we performed a comprehensive technical review of the AnesQA dataset. Initially, a manual inspection of 100 randomly sampled QA pairs revealed a specific formatting error. Based on this finding, we developed a regular expression to systematically scan the entire dataset for all instances of this and similar errors. This automated process identified 119 problematic QA pairs, which were then expunged from the dataset. The resulting refined AnesQA dataset, consisting of over 20,000 QA pairs, is well-suited for its designated purpose.

## Appendix B Evaluation Details

### B.1 List of Evaluation Models

In addition to the results in Tab [4.1](https://arxiv.org/html/2504.02404v2#S4.SS1.SSS0.Px1) in the main text, we have evaluated the anesthesiology reasoning performance of more than 50 LLMs using AnesBench-English in total. The complete list of evaluated models is shown in Tab. [8](https://arxiv.org/html/2504.02404v2#A2.T8).

**Table 8: List of Evaluation Models**
| Model Group | Models |
| --- | --- |
| Qwen | Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3-14B, Qwen3-30B-A3B, Qwen3-32B, Qwen3-235B-A22B (Team, 2025),QwQ-32B-Preview (Qwen, 2025)<br>Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Qwen2.5-32B-Instruct, Qwen2.5-72B-Instruct (Yang et al., 2024) |
| DeepSeek | DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Qwen-14B, DeepSeek-R1-Distill-Qwen-32B, DeepSeek-R1-Distill-Llama-70B, Deepseek-R1 (Guo et al., 2025), DeepsSeek-V3 (Liu et al., 2024a) |
| Gemma | Gemma-3-1b-it, Gemma-3-4b-it, Gemma-3-12b-it, Gemma-3-27b-it (Team et al., 2025a)<br>Gemma-2-9b-it, Gemma-2-27b-it (Team et al., 2024) |
| Llama | Meta-Llama-3-8B-Instruct, Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct (Grattafiori et al., 2024), Llama-4-Scout-17B-16E ,Llama-4-Maverick-17B-128E (Meta, 2025) |
| HuatuoGPT | HuatuoGPT-o1-7B, HuatuoGPT-o1-8B, HuatuoGPT-o1-70B, HuatuoGPT-o1-72B (Chen et al., 2024) |
| Other Chinese Models | chatglm3-6b, glm-4-9b-chat (GLM et al., 2024), internlm3-8b-instruct (Cai et al., 2024), Baichuan2-7B-Chat, Baichuan2-13B-Chat (Baichuan, 2023), Yi-1.5-34B-Chat (Young et al., 2024) |
| Other Medical Models | BioMistral-7B (Labrak et al., 2024), FineMedLM, FineMedLM-o1 (Yu et al., 2025), Citrus1.0-llama-70B (Wang et al., 2025), Llama3-OpenBioLLM-8B, Llama3-OpenBioLLM-70B (Ankit Pal, 2024), Llama-3-70B-UltraMedical, Llama-3.1-8B-UltraMedical (Zhang et al., 2024), Bio-Medical-Llama-3-8B (ContactDoctor, 2024) |
| Proprietary Models | GPT-4o (OpenAI, 2024a), Gemini-2.5-Flash, Gemini-2.5-Pro (Comanici et al., 2025), Claude-3.7-Sonnet (Anthropic, 2025) |
| Other General Models | Phi-4 (Abdin et al., 2024) |

### B.2 Experimental Environment for Evaluation

The evaluation experiments were primarily conducted on a server equipped with eight NVIDIA A800 GPUs. The sglang framework (Zheng et al., 2024a) was employed for inference. To ensure deterministic and reproducible outputs, the sampling temperature was set to 0, with the maximum number of output tokens limited to 2048. We adopted the Zero-Shot COT methodology for prompting (Wei et al., 2022). For proprietary and larger models (e.g., Llama-4-Maverick-17B-128E), inference was performed via their respective APIs.

### B.3 Complete Evaluation Results

The complete evaluation results on AnesBench-English are shown in Tab. [9](https://arxiv.org/html/2504.02404v2#A2.T9).

**Table 9: Evaluation Results on AnesBench-English**
| Model | Sys1 | Sys1.x | Sys2 | Total |
| --- | --- | --- | --- | --- |
| Qwen3-0.6B | 0.39 | 0.31 | 0.26 | 0.36 |
| Qwen3-1.7B | 0.48 | 0.37 | 0.30 | 0.44 |
| Qwen3-4B | 0.60 | 0.46 | 0.34 | 0.54 |
| Qwen2.5-7B-Instruct | 0.56 | 0.44 | 0.36 | 0.52 |
| Qwen3-8B | 0.65 | 0.50 | 0.40 | 0.60 |
| Qwen2.5-14B-Instruct | 0.61 | 0.52 | 0.41 | 0.57 |
| Qwen3-14B | 0.70 | 0.57 | 0.45 | 0.65 |
| Qwen3-30B-A3B | 0.73 | 0.60 | 0.48 | 0.68 |
| Qwen3-32B | 0.72 | 0.64 | 0.48 | 0.68 |
| QwQ-32B-Preview | 0.69 | 0.58 | 0.44 | 0.64 |
| Qwen2.5-32B-Instruct | 0.65 | 0.55 | 0.44 | 0.61 |
| Qwen2.5-72B-Instruct | 0.72 | 0.60 | 0.48 | 0.67 |
| Qwen3-235B-A22B | 0.78 | 0.67 | 0.57 | 0.74 |
| DeepSeek-R1-Distill-Qwen-1.5B | 0.32 | 0.28 | 0.27 | 0.31 |
| DeepSeek-R1-Distill-Qwen-7B | 0.40 | 0.34 | 0.28 | 0.38 |
| DeepSeek-R1-Distill-Qwen-14B | 0.64 | 0.51 | 0.40 | 0.59 |
| DeepSeek-R1-Distill-Qwen-32B | 0.67 | 0.56 | 0.45 | 0.63 |
| DeepSeek-R1-Distill-Llama-70B | 0.77 | 0.68 | 0.56 | 0.73 |
| DeepSeek-V3 | 0.77 | 0.69 | 0.55 | 0.73 |
| DeepSeek-R1 | 0.85 | 0.78 | 0.70 | 0.82 |
| Gemma-3-1b-it | 0.33 | 0.26 | 0.21 | 0.30 |
| Gemma-3-4b-it | 0.46 | 0.36 | 0.35 | 0.42 |
| Gemma-2-9b-it | 0.53 | 0.40 | 0.36 | 0.49 |
| Gemma-3-12b-it | 0.56 | 0.46 | 0.36 | 0.52 |
| Gemma-2-27b-it | 0.60 | 0.43 | 0.36 | 0.54 |
| Gemma-3-27b-it | 0.63 | 0.52 | 0.40 | 0.58 |
| Meta-Llama-3-8B-Instruct | 0.54 | 0.42 | 0.39 | 0.50 |
| Llama-3.1-8B-Instruct | 0.58 | 0.45 | 0.36 | 0.53 |
| Llama-3.1-8B-UltraMedical | 0.63 | 0.47 | 0.41 | 0.57 |
| Llama-3.3-70B-Instruct | 0.74 | 0.63 | 0.51 | 0.70 |
| Llama-3-70B-UltraMedical | 0.73 | 0.60 | 0.47 | 0.68 |
| Llama3-OpenBioLLM-70B | 0.68 | 0.55 | 0.44 | 0.63 |
| Llama-4-Scout-17B-16E | 0.77 | 0.66 | 0.55 | 0.72 |
| Llama-4-Maverick-17B-128E | 0.83 | 0.73 | 0.64 | 0.79 |
| HuatuoGPT-o1-7B | 0.56 | 0.45 | 0.38 | 0.52 |
| HuatuoGPT-o1-8B | 0.58 | 0.46 | 0.39 | 0.53 |
| HuatuoGPT-o1-70B | 0.70 | 0.58 | 0.48 | 0.65 |
| HuatuoGPT-o1-72B | 0.71 | 0.61 | 0.48 | 0.67 |
| chatglm3-6b | 0.37 | 0.28 | 0.25 | 0.34 |
| glm-4-9b-chat | 0.48 | 0.36 | 0.36 | 0.44 |
| internlm3-8b-instruct | 0.60 | 0.43 | 0.40 | 0.54 |
| Baichuan2-7B-Chat | 0.39 | 0.31 | 0.30 | 0.37 |
| Baichuan2-13B-Chat | 0.42 | 0.31 | 0.34 | 0.39 |
| Yi-1.5-34B-Chat | 0.54 | 0.44 | 0.35 | 0.50 |
| BioMistral-7B | 0.43 | 0.30 | 0.32 | 0.39 |
| FineMedLM | 0.40 | 0.35 | 0.27 | 0.38 |
| FineMedLM-o1 | 0.43 | 0.34 | 0.26 | 0.39 |
| Citrus1.0-llama-70B | 0.71 | 0.60 | 0.52 | 0.67 |
| Llama3-OpenBioLLM-8B | 0.44 | 0.35 | 0.30 | 0.41 |
| Bio-Medical-Llama-3-8B | 0.53 | 0.41 | 0.38 | 0.49 |
| GPT-4o | 0.81 | 0.72 | 0.59 | 0.77 |
| Gemini-2.5-Flash | 0.84 | 0.76 | 0.68 | 0.81 |
| Gemini-2.5-Pro | 0.89 | 0.82 | 0.77 | 0.86 |
| Claude-3.7-Sonnet | 0.80 | 0.73 | 0.63 | 0.77 |
| phi-4 | 0.69 | 0.57 | 0.41 | 0.64 |

## Appendix C Training Experiment Details

### C.1 Experiment Details of Using AnesQA for SFT

In the ablation study presented in Section [5.2](https://arxiv.org/html/2504.02404v2#S5.SS2), we utilized the AnesQA dataset as domain-specific data to evaluate its effectiveness in enhancing the anesthesiology reasoning capabilities of LLMs. As AnesQA consists exclusively of English question-answer pairs, we translated it into Chinese using GPT-4o-mini (OpenAI, 2024b). This step was taken to ensure the reliability of our results on the AnesBench-Chinese by mitigating the influence of language distribution and to align with the bilingual configuration of the Medical-o1 dataset. The specific prompt used for this translation process is provided below.

### C.2 Experiment Details for Baseline Model

The training of the baseline model comprises two primary stages: SFT and GRPO. All training experiments were conducted on a server with 8 NVIDIA A800 (80G) GPUs. For the SFT stage, we employed the Llamafactory framework (Zheng et al., 2024b). The key training hyperparameters are detailed in Tab. [10](https://arxiv.org/html/2504.02404v2#A3.T10).

**Table 10: Key Training Hyperparameters of SFT**
| Category | Hyperparameter | Value |
| --- | --- | --- |
| Training Setup | Cutoff Length | 4096 |
| Num Train Steps | 100 |  |
| Optimization | Learning Rate | $1.0\times 10^{-5}$ |
| Warmup Ratio | 0.175 |  |
| LR Scheduler | cosine |  |
| Batch | Per-device Train Batch Size | 16 |
| Gradient Accumulation Steps | 4 |  |

In the second stage, we employed the Verl framework (Sheng et al., 2025) for model training using GRPO. The primary training parameters were configured as follows: a training batch size of 512 and a mini-batch size of 256 for each gradient update. The maximum response length was constrained to 2048 tokens. For each prompt, the group size for rollout was set to 5. The training process was conducted for a maximum of 4 epochs, with a learning rate of $1\times 10^{-6}$ and a learning rate warmup ratio of 0.3. Under this configuration, the Morpheus-7B and Morpheus-14B models were trained for 50 and 40 steps, respectively.

### C.3 Experiment Details for Ablation Study

#### Hardware and Software

All experiments were conducted on 8 NVIDIA A800 (80G) GPUs, interconnected via NVLink. We utilized Llama-Factory as the training infrastructure. Performance was further enhanced by leveraging BF16 mixed-precision, FlashAttention-2, and the LIGER kernel.

#### Continued Pre-training (CPT)

The CPT stage involved full-parameter fine-tuning of the Qwen2.5-7B base model. We set the maximum sequence length to 2048 tokens. The model was trained for 3 epochs using DeepSpeed with a ZeRO Stage 1 configuration. Key hyperparameters included an effective batch size of 64 (a per-device batch size of 8 with 8 gradient accumulation steps), a peak learning rate of $1.0\times 10^{-4}$ decayed with a cosine annealing schedule, and a linear warmup phase over the initial 5% of training steps.

#### Supervised Fine-tuning (SFT)

Starting from the CPT model, we performed full-parameter supervised fine-tuning. The maximum sequence length was increased to 4096 tokens. This stage ran for 5 epochs, employing a more memory-efficient DeepSpeed ZeRO Stage 3 configuration. The hyperparameters were adjusted to an effective batch size of 32 (a per-device batch size of 8 with 4 accumulation steps) and a reduced peak learning rate of $1.0\times 10^{-5}$, also with a cosine schedule. The warmup period was extended to 10% of the total steps, and gradient clipping with a maximum norm of 5.0 was applied to ensure stability.

## Appendix D AnesCorpus Visualization

As illustrated in Fig. [9](https://arxiv.org/html/2504.02404v2#A4.F9), our constructed anesthesia-related corpus encompasses a diverse range of medical terminologies, both in English and Chinese. The English word cloud highlights key terms such as “general anesthesia”, “local anesthetic”, “surgical procedure”, and “pain management”, reflecting a strong focus on clinical applications, patient care, and perioperative considerations. Additionally, frequent mentions of terms like “nerve block”, “blood pressure”, and “side effect” indicate a broad coverage of anesthesia techniques and associated physiological impacts.
Similarly, the Chinese word cloud presents a comparable distribution, with dominant terms such as “手术(surgery)”, “治疗(treatment)”, “麻醉(anesthesia)”, indicating a strong alignment with procedural and treatment-related aspects. The presence of terms related to medical interventions, such as “气管插管(intubation)” and “药物治疗(medication)”, exhibits emphasis on clinical procedures and pharmacological considerations.

Figure: Figure 9: Word clouds of CPT corpus.
Refer to caption: x9.png

## Appendix E Case Study

To highlight the challenges inherent in specialized medical reasoning, we conducted a comparative case study featuring Gemini-2.5-Pro, Qwen2.5-14B-Instruct and Morpheus-14B. The cases presented are those in which Morpheus-14B provided the correct answer while other two models did not. The objective of this analysis is not to assert the overall superiority of Morpheus-14B. Rather, it is to illustrate that for a highly specialized discipline such as anesthesiology, professional knowledge and general reasoning capabilities are insufficient. Subtle distinctions in clinical scenarios can lead to fundamentally different outcomes, underscoring the need for more advanced and specialized reasoning.

For example, in Case 1, Gemini-2.5-Pro demonstrates clear reasoning and accurately recognizes that the $\mu$-opioid receptor is blocked. However, it fails to acknowledge that naltrexone exhibits affinity not only for the $\mu$ receptor but also for the $\kappa$ and $\delta$ opioid receptors. As a result, the combined use of naltrexone and nalbuphine cannot achieve an analgesic effect. Consequently, option C-“Continue naltrexone with small doses of nalbuphine every 4 hours as needed”, is incorrect.

In Case 2, the reasoning of Gemini-2.5-Pro was almost correct; however, it failed to recognize that its rationale is valid only in the context of severe hypotension. According to the case description, the clinician’s primary concern should be pump failure and hypoperfusion rather than hypotension. As a result, the model’s recommendation was premature in the specific clinical scenario.

The following are the specific contents of the two cases.

## Appendix F Significance Test

To rigorously validate the claims presented in Sec. [5.1](https://arxiv.org/html/2504.02404v2#S5.SS1), we conducted a series of statistical tests to confirm the validity and significance of our insights. This section presents the results of these tests and discusses additional findings uncovered during the analysis.

#### Methodology

By default, all comparisons were evaluated using bootstrap resampling. We report the difference in means ($\Delta$), the 90% confidence interval (CI, corresponding to $\alpha=0.1$), and the one- or two-sided p-value.

### F.1 Significance Test on Model Characteristic

#### Performance Across Model Scale

Our analysis confirmed the relationship between model scale and performance as described in Sec. [5.1](https://arxiv.org/html/2504.02404v2#S5.SS1). The claim of a “strong positive correlation between model performance and scale” was strongly supported by Spearman correlation tests across all systems. As detailed in Tab. [12](https://arxiv.org/html/2504.02404v2#A6.T12), the Spearman’s rank correlation coefficient ($\rho$) exceeds 0.8 for all three systems, while the corresponding p-values approach zero. These results indicate a strong and statistically significant positive association.

**Table 11: Statistical Test for Model Scale**
| System | $\rho_{s}$ | p-value |
| --- | --- | --- |
| System1 | $0.859$ | 1.49$e^{-15}$ |
| System1.x | $0.861$ | 1.07$e^{-15}$ |
| System2 | $0.818$ | 3.31$e^{-16}$ |

The hypothesis of diminishing marginal returns was also supported. We employed a quadratic ordinary least squares (OLS) model to examine the relationship between model size and performance score. For this analysis, the alternative hypothesis posits that the coefficient of the quadratic term, $\beta_{2}$, is less than zero. The results of one-sided tests confirmed a statistically significant negative coefficient for this quadratic term across all evaluated systems (all $p<0.025$), indicating the presence of diminishing returns.

To evaluate the claim that ”the slope for System2 is significantly lower than System1.x, indicating that performance gains from increasing model size are markedly smaller for System2,” we employed a linear model. The model is defined as:

$$ $\displaystyle y=\beta_{0}+\beta_{1}x+\beta_{2}\cdot\text{group}+\beta_{3}(x\times\text{group})+\varepsilon.$ (2) $$

In this model, the group variable is a binary indicator, specifying whether the data point originated from System1.x or System2. We conducted a one-sided bootstrap hypothesis test with the alternative hypothesis that $\beta_{3}$ is greater than zero.

Our tests revealed this holds true when comparing System 2 to System 1.x (p=0.0009). In contrast, the difference between System 2 and System 1 (factual recall) was not statistically significant (p=0.1538), suggesting that the scaling advantage diminishes most sharply when moving from hybrid reasoning to complex decision-making.

#### Performance Across Language

Statistical analysis corroborates the cross-lingual observations detailed in Section [5.1](https://arxiv.org/html/2504.02404v2#S5.SS1). To test the hypothesis that English accuracy exceeds Chinese accuracy ($H_{1}:Acc_{\text{EN}}>Acc_{\text{CH}}$), we performed bootstrap tests on the AnesBench Chinese-English mutual translation benchmark. The results strongly indicate that models based on Llama-3.1-8B perform notably worse in Chinese. For all evaluated models in this series, this performance disparity was statistically significant ($p<0.05$), highlighting that language transferability remains a key challenge.

**Table 13: Statistical Test for Performance Across Language**
| Model | p-value |
| --- | --- |
| Llama-3.1-8B-Instruct | 9$e^{-5}$ |
| Llama-3.1-8B-UltraMedical | 9$e^{-5}$ |
| FineMedLM-o1 | 9$e^{-5}$ |

#### Performance Across Output Length

Regarding the impact of reasoning length, we tested the claim that “Models with longer CoT reasoning processes tend to exhibit superior response performance.” in Section [5.1](https://arxiv.org/html/2504.02404v2#S5.SS1). A Spearman correlation test provided weak evidence for this relationship in the context of System 2 tasks ($\rho$=0.4917, p=0.0741). No significant correlation was found for System 1 or System 1.x tasks, reinforcing the idea that extended reasoning is primarily beneficial for problems that demand complex, multi-step cognitive processes.

### F.2 Significance Test on Training Methodology

#### Impact of Continuous Pre-Training

We commenced our investigation by examining the claim from Sec. [5.2](https://arxiv.org/html/2504.02404v2#S5.SS2) concerning the dichotomous effect of CPT. This involved a comparative analysis of the top-performing models trained with and without the CPT methodology. Specifically, we formulated distinct alternative hypotheses ($H_{1}$) for the English and Chinese benchmarks. For AnesBench-English, the hypothesis was $H_{1}:\text{AnesQA + Medical-o1 (with CPT)}>\text{AnesQA + Medical-o1 (without CPT)}$, whereas for the AnesBench-Chinese, it was $H_{1}:\text{AnesQA + Medical-o1 (without CPT)}>\text{AnesQA + Medical-o1 (with CPT)}$.

To test these hypotheses, we employed a one-sided bootstrap test. The results, as presented in Table [14](https://arxiv.org/html/2504.02404v2#A6.T14), indicate that the p-values for both tests are below the 0.05 significance level. Consequently, both alternative hypotheses are accepted. This finding substantiates the initial claim: CPT improves the upper bound of the model’s performance on AnesBench-English, while it degrades performance on the AnesBench-Chinese.

**Table 14: Statistical Test for the Impact of Continuous Pre-Training**
| Subset | $\bm{\Delta}$ | p-value | 90% CI |
| --- | --- | --- | --- |
| AnesBench-English | $0.0149$ | $0.0413$ | (0.0009, 0.0287) |
| AnesBench-Chinese | $0.0592$ | $0.000\,09$ | (0.0438, 0.0746) |

#### Complementarity of AnesQA and Medical-o1

To assess the complementarity between AnesQA and Medical-o1, we conducted four sets of tests under both CPT-trained and non-CPT-trained conditions. These experiments evaluated whether combining AnesQA and Medical-o1 could yield better performance than using either individually. As shown in Tab. [15](https://arxiv.org/html/2504.02404v2#A6.T15), a statistically significant improvement was observed only when both models were combined under the CPT-trained setting and evaluated on AnesBench-English. Accordingly, we used the phrase “leads to varying degrees of performance improvement” in the main text to reflect this outcome. Although the performance gains from the combination are not universally significant, consistent improvements are observed regardless of CPT training and across both AnesBench-English and AnesBench-Chinese benchmarks. This consistency suggests that combining the two models generally yields incremental benefits.

**Table 15: Overall Complementarity of Supervised Fine-Tuning Datasets**
| Comparison | $\bm{\Delta}$ | p-value | 90% CI |
| --- | --- | --- | --- |
| Without Continuous Pre-Training |  |  |  |
| AnesQA + Medical-o1 $>$ AnesQA | $0.0041$ | $0.2794$ | (-0.0079, 0.0149) |
| AnesQA + Medical-o1 $>$ Medical-o1 | $0.0059$ | $0.2216$ | (-0.0070, 0.0185) |
| With Continuous Pre-Training |  |  |  |
| AnesQA + Medical-o1 $>$ AnesQA | $0.0154$ | $0.0237$ | (0.0029, 0.0278) |
| AnesQA + Medical-o1 $>$ Medical-o1 | $0.0056$ | $0.2284$ | (-0.0068, 0.0181) |

## Appendix G Supplementary Evaluation

### G.1 Self-Consistency in Test Time Scaling

Figure: Figure 10: Test time scaling of Self-Consistency.
Refer to caption: x11.png

We utilized AnesBench to evaluate the efficacy of test-time scaling techniques, such as Self-Consistency (Wang et al., 2023b), for enhancing reasoning in anesthesiology. Specifically, we employed the Morpheus-7B as the policy model for sampling. As illustrated in Fig. [10](https://arxiv.org/html/2504.02404v2#A7.F10), the application of Self-Consistency markedly surpasses the performance of direct inference. Furthermore, the model’s performance consistently improves with an increased number of samples. These findings underscore that test-time scaling is an effective method for unlocking the full potential of LLMs in complex anesthesiology reasoning tasks.

### G.2 Evaluation on the Open-ended Subset

To address the limitations of MCQs and further validate our findings, we augmented our evaluation with open-ended questions. A subset of System 2 benchmark items was converted into an open-ended format.

To construct our experimental dataset, we manually curated a specific subset from System2 questions of AnesBench-English. We identified and selected 64 items that were “option-independent,” meaning they could be answered based solely on the information provided in the question stem, without reliance on the accompanying options. Subsequently, we employed the Gemini-2.5-Pro (Comanici et al., 2025) to transform this curated subset of questions into an open-ended format. This conversion was guided by a specific prompt designed to minimize unnecessary alterations to the original questions.

To ensure robust and objective scoring, we use automated metrics such as BLEU and F1 along with LLM-as-judge to evaluate performance on open-ended questions. To implement LLM-as-judge, we engineered a detailed prompt that instructed the evaluator models (GPT-4.1 and Gemini-2.5-Pro) to adopt the persona of an expert medical evaluator. To standardize the assessment, the prompt included a detailed 10-point scoring rubric, mapping score ranges to specific quality levels. We repeated each evaluation three times. The final scores for each response were then averaged across both evaluator models to ensure robust and reliable assessment. The prompts used for LLM-as-Judge and question rewriting are provided as below.

**Table 16: Evaluation Results of Open-Ended QA vs. AnesBench (MCQ)**
| Model | LLM-based Scores | Automated Metrics | AnesBench |  |  |
| --- | --- | --- | --- | --- | --- |
|  | GPT Score | Gemini Score | BLEU | F1 | Acc (%) |
| DeepSeek-R1 | $8.22$ | $8.76$ | $0.037$ | $0.230$ | $82$ |
| Gemini-2.5-flash | $7.88$ | $8.53$ | $0.026$ | $0.196$ | $81$ |
| DeepSeek-V3 | $7.85$ | $8.34$ | $0.037$ | $0.230$ | $73$ |
| GPT-4o | $7.77$ | $7.65$ | $0.029$ | $0.205$ | $77$ |
| Gemma-3-27b-it | $6.55$ | $6.09$ | $0.000$ | $0.000$ | $58$ |
| Qwen3-30B-A3B | $6.44$ | $6.02$ | $0.000$ | $0.000$ | $68$ |
| DeepSeek-R1-Distill | $6.26$ | $5.79$ | $0.000$ | $0.020$ | $63$ |
| Qwen2.5-72B-Instruct | $6.17$ | $5.51$ | $0.000$ | $0.000$ | $67$ |
| Qwen3-14B | $6.10$ | $5.45$ | $0.000$ | $0.068$ | $65$ |
| Claude-3.7 | $5.11$ | $5.35$ | $0.020$ | $0.165$ | $77$ |

The results presented in Table [16](https://arxiv.org/html/2504.02404v2#A7.T16) reveal a strong correlation between the LLM-based scores (GPT and Gemini) and the original MCQ accuracy on AnesBench, which validates the use of MCQs as a reliable proxy for complex reasoning. In contrast, automated metrics like BLEU and F1 scores were consistently low (mostly below 0.25), underscoring their inadequacy for assessing the nuanced, multi-step logic required in medical decision-making. Therefore, we prioritize MCQs in our benchmark as a more robust and standardized method for evaluating reasoning performance.

### G.3 Evaluation on CAB

In addition to AnesBench, we evaluated the models on a supplementary resource of over 8,000 MCQs from the CAB (Zhou et al., 2025). The complete evaluation results are presented in Tab. [17](https://arxiv.org/html/2504.02404v2#A7.T17). Although this benchmark effectively highlights performance discrepancies among different models, its lack of a structured design precludes a detailed analysis of model capabilities across various systems. Furthermore, the exclusive inclusion of Chinese questions restricts its applicability for multilingual assessment.

**Table 17: Evaluation Results on MCQs from CAB**
| Model | Acc. | Model | Acc. |
| --- | --- | --- | --- |
| Baichuan2-7B-Chat | 0.43 | Baichuan2-13B-Chat | 0.48 |
| Bio-Medical-Llama-3-8B | 0.48 | BioMistral-7B | 0.26 |
| chatglm3-6b | 0.36 | Citrus1.0-llama-70B | 0.71 |
| deepseek-r1 | 0.87 | DeepSeek-R1-Distill-Llama-70B | 0.64 |
| DeepSeek-R1-Distill-Qwen-1.5B | 0.26 | DeepSeek-R1-Distill-Qwen-7B | 0.33 |
| DeepSeek-R1-Distill-Qwen-14B | 0.63 | DeepSeek-R1-Distill-Qwen-32B | 0.67 |
| deepseek-v3 | 0.78 | FineMedLM | 0.31 |
| FineMedLM-o1 | 0.35 | gemma-2-9b-it | 0.52 |
| gemma-2-27b-it | 0.56 | gemma-3-1b-it | 0.26 |
| gemma-3-4b-it | 0.43 | gemma-3-12b-it | 0.58 |
| gemma-3-27b-it | 0.65 | glm-4-9b-chat | 0.61 |
| gpt-4o | 0.78 | HuatuoGPT-o1-7B | 0.70 |
| HuatuoGPT-o1-8B | 0.56 | HuatuoGPT-o1-70B | 0.70 |
| HuatuoGPT-o1-72B | 0.81 | internlm3-8b-instruct | 0.84 |
| Llama-3-70B-UltraMedical | 0.71 | Llama-3.1-8B-Instruct | 0.53 |
| Llama-3.1-8B-UltraMedical | 0.54 | Llama-3.3-70B-Instruct | 0.68 |
| Llama-4-Scout-17B-16E-Instruct | 0.78 | Llama3-OpenBioLLM-8B | 0.25 |
| Llama3-OpenBioLLM-70B | 0.64 | Meta-Llama-3-8B-Instruct | 0.49 |
| phi-4 | 0.57 | Qwen2.5-7B-Instruct | 0.67 |
| Qwen2.5-14B-Instruct | 0.73 | Qwen2.5-32B-Instruct | 0.76 |
| Qwen2.5-72B-Instruct | 0.81 | Qwen3-0.6B | 0.37 |
| Qwen3-1.7B | 0.53 | Qwen3-4B | 0.53 |
| Qwen3-8B | 0.65 | Qwen3-14B | 0.76 |
| Qwen3-30B-A3B | 0.73 | Qwen3-32B | 0.80 |
| Qwen3-235B-A22B | 0.75 | QwQ-32B-Preview | 0.73 |
| Yi-1.5-34B-Chat | 0.65 |  |  |

## Appendix H Datasheet for AnesSuite

### H.1 Motivation

1. For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.

A1: The AnesSuite dataset was created to address a critical gap in artificial intelligence research: the limited exploration of reasoning abilities in LLMs within the highly specialized field of anesthesiology. Its primary goal is to serve as the first comprehensive resource specifically designed to train and evaluate LLMs on tasks relevant to anesthesiology. AnesSuite includes the AnesBench benchmark, which assesses model performance across a wide range of reasoning challenges—from straightforward factual recall to complex clinical decision-making. Additionally, it offers dedicated training datasets that support various strategies for model improvement, including continuous pretraining, supervised fine-tuning, and reinforcement learning.

2. Who created this dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?

A2: This dataset is created by the authors of this paper.

3. Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.

A3: N/A.

### H.2 Composition

1. What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances(e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.

A1: The dataset suite’s instances include four types: (a) MCQs that assess anesthesiology reasoning at three cognitive levels (System 1, 1.x, 2) in AnesBench; (b) plain-text anesthesiology documents for continued pre-training in AnesCorpus; (c) QA pairs in AnesQA; and (d) verifiable MCQs paired with detailed CoT reasoning in AnesR1.

2. How many instances are there in total (of each type, if appropriate)?

A2: Instance counts: AnesBench has 4k English MCQs and 3k Chinese MCQs; AnesCorpus contains  1.8 million English and  0.6 million Chinese documents; AnesQA has 20k English QA pairs; AnesR1 totals 10k items (3.2k English + 7k Chinese) of MCQs with CoT.

3. Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).

A3: AnesSuite is a sample drawn from a larger set of instances. AnesBench and AnesR1 are MCQs related to anesthesiology, but they do not imply full representativeness of all anesthesiology questions. AnesCorpus is a keyword-filtered sample from large web corpora; it is therefore a targeted subset, not a census of all anesthesiology texts. AnesQA questions were generated from cleaned PubMed papers via an LLM pipeline with further filtering, which can’t imply full representativeness of all anesthesiology QAs.

4. What data does each instance consist of? “Raw” data (e.g., unprocessed text or images)or features? In either case, please provide a description.

A4: Per-instance data: AnesBench items are MCQs annotated with a cognitive-demand level (System 1/1.x/2). AnesCorpus instances are plain-text documents relevant to anesthesia and pain management. AnesQA instances consist of a question and its answer, with each question typed into one of five categories. AnesR1 instances pair a verifiable MCQ with a full CoT reasoning trace under the same three-level cognitive framework.

5. Is there a label or target associated with each instance? If so, please provide a description.

A5: Yes, MCQ datasets (AnesBench, AnesR1) include the correct choice as the target.

6. Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.

A6: No.

7. Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? If so, please describe how these relationships are made explicit.

A7: No.

8. Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.

A8: Yes, as mentioned in the main text, we recommend using AnesBench for testing, and AnesCorpus, AnesQA, and AnesR1 for training. There is no explicit validation set, and users can split it from the training set by themselves.

9. Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.

A9: Yes, since AnesSuite contains datasets from different sources, for example, AnesCorpus is derived from cleaned web data, but it may still contain noise. The construction of AnesQA relies on an LLM pipeline, which may also contain hallucinations and noise. However, each part has corresponding dataset validation methods, as detailed in the relevant sections of the main text and appendices.

10. Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.

A10: Yes, AnesSuite is self-contained, and users can obtain it directly from the project homepage, HuggingFace, and other channels once the dataset is made public.

11. Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctorpatient confidentiality, data that includes the content of individuals non-public communications)? If so, please provide a description.

A11: No, although AnesSuite contains abstract clinical scenarios, they are all virtualized and do not include any personal information.

12. Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.

A12: No.

### H.3 Collection Process

1. How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.

A1: Yes, in AnesSuite, except for AnesQA, all other data instances can be acquired directly. AnesQA is derived from the LLM pipeline, and its verification process is detailed in the relevant sections of the main text and appendices.

2. What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?

A2: The collection of AnesSuite mainly uses Python scripts. After obtaining the raw data, we adopted both manual and automated verification methods, as detailed in the relevant sections of the main text and appendices.

3. If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?

A3: The sampling of AnesSuite is a deterministic process, not based on specific sampling probabilities. For example, due to the keyword filtering method, AnesCorpus is only a sample of a deterministic subset related to these keywords among all anesthesia-related texts.

4. Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?

A4: The authors of this paper.

5. Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.

A5: The collection period of AnesSuite is from 2024 to 2025, which does not exactly match the creation time of the data.

### H.4 Preprocessing/cleaning/labeling

1. Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section.

A1: Yes, labeling and cleaning we performed on the data can refer to the relevant sections in the main text and appendices.

2. Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data.

A2: No.

3. Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.

A3: All of our preprocessing/cleaning/labeling are done using Python scripts, not a specific software.

### H.5 Uses

1. Has the dataset been used for any tasks already? If so, please provide a description.

A1: Yes, we have used AnesBench from AnesSuite to evaluate the reasoning abilities of current state-of-the-art LLMs. Additionally, using its training set, we have developed Moupheus, the first collection of reasoning LLMs in anesthesiology.

2. Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.

A2: N/A.

3. What (other) tasks could the dataset be used for?

A3: AnesSuit contains three training sets, among which AnesR1 and AnesQA also include instance labels. This design allows users to freely combine and train models according to their needs. AnesBench can also be used to test other LLMs.

4. Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?

A4: No.

5. Are there tasks for which the dataset should not be used? If so, please provide a description.

A5: N/A

### H.6 Distribution

1. Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.

A1: Yes, the dataset will be made available to the public after the paper is published.

2. How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?

A2: AnesSuite will be distributed via GitHub and HuggingFace.

3. When will the dataset be distributed?

A3: The dataset will be distributed once the paper is accepted after peer review.

4. Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.

A4: For the specific license, please refer to the project homepage and Huggingface after we open-source the AnesSuite.

5. Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.

A5: No.

6. Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.

A6: No.

### H.7 Maintenance

1. Who will be supporting/hosting/maintaining the dataset?

A1: The authors.

2. How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

A2: The public contact email will be posted post-acceptance.

3. Is there an erratum? If so, please provide a link or other access point.

A3: After the paper is published, we will make public the link pointing to the erratum.

4. Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?

A4: Yes, dataset will be updated. We welcome feedback from AnesSuite users. The authors of this paper will update the dataset on the project homepage and GitHub from time to time based on the feedback received, including removing any errors identified.

5. Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to users.

A5: No, each time we update the dataset, we will mark the update time simultaneously, which will help users understand when each version of the dataset becomes obsolete.

6. If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description.

A6: N/A.

## Appendix I Usage of LLM

#### Writing Assistance

We use LLMs to assist with the proofreading and refinement of the manuscript, primarily for correcting grammar, optimizing sentence structure, and improving the clarity of the language. All content generated with AI assistance was strictly reviewed and revised by the authors to ensure its accuracy and academic integrity. We take full responsibility for the final content of the manuscript.

#### Research Applications

Beyond assisting writing, LLMs are integral to our research. Specifically, we utilized LLMs in the process of dataset construction, evaluation, training and so on. The specific methodologies, implementation details, and the function of these applications in our research are described in detail within the corresponding sections of this paper.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. For AnesR1, did clinicians evaluate a subset of DeepSeek‑R1 chains for medical correctness beyond the final answer? Any evidence that RL on these chains avoids entrenching subtle misconceptions?  

2. What exact reward(s) were used in GRPO—answer correctness only, or also intermediate checks (e.g., vital‑sign logic)? How robust are results to reward misspecification?
soundness: 2
strengths: - First-of-it-kind domain‑specific reasoning suite for anesthesiology that separates fast recall from complex decision‑making (System 1→2) and offers training corpora aligned with CPT/SFT/RLVR stages. Prior work either subsumed anesthesiology under broad medical categories or focused on factual recall; this suite targets decision‑making explicitly;

- Bilingual design with explicit experiments on language transfer. The ablation shows CPT on English‑heavy corpora can help EN while hurting ZH, highlighting data‑language balance as a first‑order concern in medical domains;

- Reasoning‑centric baselines (Morpheus) showing that limited RL with verifiable rewards (GRPO) plus CoT data can close the gap to much larger models on this domain.
summary: The paper introduces AnesSuite, a suite aimed at assessing and improving anesthesiology reasoning in LLMs. It comprises:
(1) AnesBench — a bilingual (EN/ZH) multiple‑choice benchmark (7,972 items) stratified by cognitive demand into System 1 (factual), System 1.x (hybrid), and System 2 (complex decision making). 
(2) AnesCorpus — ~2.4M anesthesia‑related documents (1.8M EN + 0.6M ZH) for continued pretraining (CPT), filtered from FineWeb/Chinese‑FineWeb with keyword rules and a two‑stage decontamination (n‑gram screen + LCS>64 removal) to reduce overlap with the benchmark.
(3) AnesQA — 20,713 English QA pairs produced via a two‑model LLM pipeline (Llama‑3.3‑70B to generate questions; Qwen‑2.5‑72B to filter/answer), with manual spot‑checks and regex filtering to excise 119 flawed items.
(4) AnesR1 — 10,287 verifiable MCQs (EN+ZH) with chain‑of‑thought (CoT) traces generated by DeepSeek‑R1 and kept via rejection sampling only when the final choice is correct. Using these resources, the authors train Morpheus‑7B/14B (Qwen‑2.5 inits) with SFT + GRPO; they report improvements over the instruction‑tuned bases and parity with larger baselines on AnesBench.
weaknesses: 1. External validity of System‑2 constructed from exam/textbook‑style abstractions rather than real perioperative trajectories; this may under‑represent key anesthetic complexities (e.g., dynamic vitals, ventilator settings, lab time‑series). 

2. Leakage analysis scope: the permutation‑confidence heuristic over 500 items is informative but indirect; it cannot rule out contamination in proprietary models, nor leakage via near‑duplicates beyond LCS>64.

3. GRPO is described at a high level; the exact verifiable reward signal (e.g., answer‑only vs. additional structure) and stability/variance aren’t deeply analyzed.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: Please see the question about weakness above.
soundness: 3
strengths: 1.  The study makes a clear and timely observation that evaluation should reflect realistic clinical settings, and it identifies contextual complexity as a common bottleneck for reasoning across tasks.

2. The three-tier benchmark design and the linked training resources form a coherent pipeline, and the Morpheus baselines with ablations provide useful evidence on which choices lead to gains.
summary: The paper introduces AnesSuite, a dataset and benchmark collection designed to test and improve LLM reasoning in anesthesiology. AnesBench provides evaluation tasks that cover factual knowledge and complex decision making in English and Chinese. The authors also release three training datasets that target continued pretraining, supervised fine tuning, and reinforcement learning with verifiable rewards. Using these resources, they build a baseline model named Morpheus and run a broad set of experiments against strong public models. The analysis examines how language transfer, training data choices, and training methods affect performance on anesthesiology reasoning.
weaknesses: 1. I appreciate the careful build of a bilingual, cognitively tiered benchmark and the broad empirical sweep, yet the main novelty lies in the dataset suite rather than a method or algorithm that is central to ICLR. The focus on anesthesiology is narrow for a broad ML venue and is also narrow even within medical or clinical tracks, which limits general use.

2. The dataset and benchmark draw mainly on public exams and textbooks and are text only. They do not include images, multimodal clinical data, or real clinical or PHI data. Existing resources already support assessment of medical knowledge, such as MedQA, MedMCQA, and PubMedQA. Clinically grounded datasets that capture real decision-making and multimodal signals are more directly relevant to patient care. The contribution is limited to that, not a direct link to clinically actionable use.


3. I am concerned about data quality. On AnesBench, SFT on the proposed dataset does not beat the Qwen2.5-Instruct baselines: Morpheus 7B with SFT only averages 54 percent compared with 59 percent for the 7B baseline, and Morpheus 14B with SFT only averages 57 percent compared with 64 percent for the 14B baseline. Most of the gains appear after the GRPO stage, to 63 percent and 69 percent, and that is modest. SFT slightly improves the English subset but drops the Chinese subset by a large margin, which points to issues with data quality, data balance, or objective alignment. Train and test sit in the same domain due to vertical fine-tuning. One would usually expect larger SFT gains in that case, so the results are underwhelming.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: Please refer to the Weaknesses section above.
soundness: 3
strengths: - The authors have conducted a substantial amount of work, delivering all the necessary sources and artifacts for the medical domain of anesthesiology, a field that is currently underserved by existing resources.

- The paper presents a solid methodology for assessing both contamination and originality across all proposed resources, ensuring the integrity of their contributions.
summary: The paper introduces a comprehensive suite of artifacts for the medical domain of anesthesiology, including benchmarks, a specialized corpus, and a anesthesiology-aligned model. The authors detail their end-to-end pipeline, covering pre-training data collection, benchmark creation, and the continuous training of a base model on the resources. The proposed benchmark is used to benchmark current SoTA medical LLMs, as well as open-weight and frontier models.
weaknesses: **[W1]** While the authors acknowledge that anesthesiology is underrepresented in current medical NLP resources and provide a setup that uses both reasoning and factual knowedge, they do not offer a clear, tangible motivation for how these resources will benefit medical-aligned LLMs overall (i.e. examples illustrating how these resources could be applied in real-world scenarios).


**[W2]** The methodology for benchmark creation and human verification is insufficiently detailed. Although their hybrid approach (LLMs + human verification) is reasonable, the paper does not specify:
- The number of human annotators involved per sample
- The instructions or rubric used for verification
- Annotator agreement rates
- How quality is ensured for data not verified by humans
These missing details make it difficult to assess the reliability and validity of the benchmarks.

**[W3]** The paper does not discuss how alignment in the anesthesiology domain affects performance on other downstream medical tasks (e.g., MedQA) or general tasks (e.g., MMLU). 

**[W4]** The pre-training and SFT data are likely to be used in continuous pretraining of medical-aligned models that need to perform well across multiple medical domains. However, the authors do not provide a realistic CPT scenario demonstrating how their data would be used in such a context. Even a hypothetical example would help clarify the potential impact and application of their resources.

**[W5]** The use of anesthesiology-related documents from Fineweb for ontinuous pretraining involves replaying existing data rather than introducing new resources. As a result, this aspect of the contribution is less substantial compared to the creation of the main benchmark.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: How would the model trained on the proposed dedicated data perform on other medical tasks or even general tasks?

Overall, I appreciate the effort of constructing this suite for such a specific domain, it would potentially be a better paper with further clarifications and improvement. I would consider increasing my score once most of my concerns are addressed.
soundness: 4
strengths: 1. The first suite for benchmarking, training, and analyzing failure patterns in the particular domain of anesthesiology
2. Comprehensive analysis and vivid figures demonstrating model performance gaps and some insights on the benchmark
summary: This paper introduces a comprehensive suite for analyzing, training, and evaluating LLMs’ capabilities in the domain of anesthesiology. The benchmark results demonstrate that current models, especially open-weight solutions may not perform perfectly on the benchmark. And both Qwen2.5 models and Morpheus would improve after being trained on the provided datasets.
weaknesses: 1. The benchmark is not challenging enough. Overall the best open-weight model could solve 82% of the problems (DeepSeek-R1) and even a smaller Qwen3-14B could solve most of them (70%). These observations indicate that these problems in the benchmark are not challenging enough, especially considering the form of MCQA. One solution may be to refactor the MCQA questions into the free-end generation task that asks the model to answer the question without providing options.
2. Training data effectiveness is not properly demonstrated: table 4 presents models trained on the proposed training set. However, this type of training-testing paradigm falls back to the in-distribution evaluation. A better way to validate the training data would be mixing them with other post-processing data and test the model or test the trained model on other OOD medical tasks.


Decision: Accept (Poster)