A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?

## Abstract

Abstract Large language models (LLMs) have exhibited remarkable capabilities across various domains and tasks, pushing the boundaries of our knowledge in learning and cognition. The latest model, OpenAI’s o1 , stands out as the first LLM with an internalized chain-of-thought technique using reinforcement learning strategies. While it has demonstrated surprisingly strong capabilities on various general language tasks, its performance in specialized fields such as medicine remains unknown. To this end, this report provides a preliminary exploration of o1 on different medical scenarios, comprehensively examining 3 key aspects: understanding , reasoning , and multilinguality . Specifically, our evaluation encompasses 6 tasks using data from 37 medical datasets, including two newly constructed and more challenging question-answering (QA) tasks based on professional medical quizzes from the New England Journal of Medicine and The Lancet. These datasets offer greater clinical relevance compared to standard medical QA benchmarks such as MedQA, translating more effectively into real-world clinical utility. Our analysis of o1 suggests that the enhanced reasoning ability of LLMs may (significantly) benefit their capability to understand various medical instructions and reason through complex clinical scenarios. Notably, o1 surpasses the previous GPT-4 in accuracy by an average of 6.2% and 6.6% across 19 datasets and two newly created complex QA scenarios. But meanwhile, we also identify several weaknesses in both the model capability and the existing evaluation protocols, including hallucination, inconsistent multilingual ability, and discrepant metrics for evaluation. We release our raw data and model outputs at https://ucsc-vlaa.github.io/o1_medicine/ for future research.

## 1 Introduction

Intelligence, a complex and elusive concept, has puzzled psychologists, philosophers, and computer scientists for years (Bubeck et al., ).
While there is no single agreed-upon definition of intelligence, it is widely accepted that it spans a broad range of cognitive skills, rather than being confined to a specific task (McCarthy et al., ). Creating artificial systems with such general intelligence has been a long-standing and ambitious goal of AI research.
The most exciting progresses in AI are achieved by language models in these years, from the initial start of ChatGPT to its evolution and other open-source projects (Touvron et al., ; ; Jiang et al., ; Bai et al., ; Peng et al., ).

Early LLM pioneers set out goals to understand and interact with human by exploring generalizable reasoning mechanisms and building knowledge bases with vast amounts of commonsense information. With parameters and data volume in place, the question of how to effectively prompt the model from the user end and train it from the developer end has become a trending topic of exploration (Wei et al., ; Ouyang et al., ). On the user side, varying prompting techniques can significantly impact model performance. Chain-of-thought (CoT) prompting (Wei et al., ; Dong et al., ; Saunders et al., ), one of the most popular strategies, leverages the model’s internal reasoning patterns to enhance its ability to solve complex tasks. OpenAI capitalized on this by embedding the CoT process into model training, integrating reinforcement learning, and finally introduced the o1 model (OpenAI, ).
While the o1 model demonstrates strong performance in general domains, its effectiveness in specialized fields like medicine—where domain-specific training may be lacking—remains uncertain. Moreover, current benchmarks for LLMs in the medical domain often evaluate models only on a limited set of factors, often focusing on isolated aspects such as knowledge and reasoning (Nori et al., ; Liévin et al., ), safety (Han et al., ), or multilinguality (Wang et al., ).
These factors make a comprehensive assessment of LLMs’ capabilities—especially for advanced models like o1—in medical challenging tasks (Figure LABEL:fig:radar).

This paper aims to provide an initiative to close this gap, focusing on o1.
We identify three fundamental aspects of LLMs in medicine: *understanding*, *reasoning*, and *multilinguality*.
To evaluate these capabilities, we assembled 35 existing medical datasets and developed two novel, challenging QA datasets that include instructions and expected outputs, ensuring comprehensive assessment. With evaluation on this extensive suite, our key findings include:

- •
o1 demonstrates improved transfer of clinical understanding and reasoning abilities, validating its competence in real-world diagnostic scenarios compared with both close- and open-source models as presented in Figure LABEL:fig:radar and Figure LABEL:fig:bar;
- •
No single model excels across all tasks on our medical leaderboard, though o1 comes close to dominating most evaluations;
- •
o1 still suffers from the long-standing issue of hallucination and complex multilingual medical cases;
- •
Inconsistencies in metrics for medical NLP can significantly affect models’ standings, which calls for a re-evaluation of reliable metrics for future LLMs;
- •
CoT prompting can further enhance o1 in medicine, despite its training having already integrated CoT data.

In addition to these findings, we also elevate the discussion section as an initial attempt to address the issues identified during our benchmarking in Section 5.
Particularly, we highlight the potential negative effects of o1, emphasize the urgent need for consistent and unified evaluation metrics for future LLMs, and advocate for improved instruction templates that can be applied to models with embedded prompting strategies.

## 2 Related works

#### Large Language Models with Enhanced Reasoning Ability.

Large Language models (LLMs) based on next token prediction pre-training (Touvron et al., ; ; Achiam et al., ) have demonstrated promising capabilities on various language undersanding tasks.
Instruction fine-tuning further improved the abilites of these LLMs for following user instructions.
However, recent studies suggest that LLMs struggle with complex tasks involving logical reasoning.
To address this issue, some researches propose to instruct LLMs to mimic human thinking processes by producing a chain-of-thought (CoT) (Feng et al., ; Wei et al., ) before generating a final answer.
Reinforcement learning from human feedback (Ouyang et al., ) has also been employed to enhance reasoning while make sure the models align with human values (Tu et al., ; ).
Recently, OpenAI introduced o1, which was trained on a vast amount of CoT data, further enhancing the capability of LLMs in solving scientific problems.
In this paper, we aim to investigate whether enhanced abilities of o1 effectively transfer to the clinical medical domain.

#### Medical Large Language Models.

Benefiting from the generalization capabilities of LLMs, general-purpose models such as GPT-4 have demonstrated impressive performance on challenging medical problems (Nori et al., ; Wu et al., ).
Some researchers have attempted to further equip LLMs with biomedical knowledge by fine-tuning them using domain-specific corpora (Chen et al., ; Wang et al., ; Wu et al., ; Li et al., ).
However, for clinical applications, LLMs are not only required to understand medical domain-specific knowledge but also to produce reliable responses by performing logical reasoning.
In this paper, we aim to explore the potential of o1 as a clinical viable model.
Our experimental findings reveal that with enhanced *understanding*, *reasoning*, and *multilinguality* medical capabilities, o1 makes a step closer to reliable clinical AI-system.

## 3 Evaluation Pipeline

### 3.1 Overall Taxonomy of Evaluations

Figure: Figure 1: Our evaluation pipeline has different (a) *aspects* with various (b) *prompting strategies* using the latest (c) *language models*. We leverage a comprehensive set of (d) *evaluations* to present a holistic view of model progress in the medical domain.
Refer to caption: x1.png

**Table 1: Six datasets across three fundamental aspects employed in our evaluation suite. Asterisks (*) denotes the newly constructed datasets from public sources.**
| Aspect | Task | Dataset | Description | Metrics |
| --- | --- | --- | --- | --- |
| Understanding | Concept<br>Recognition | BC5-disease (Li et al., ) | Entity extraction for disease. | F1-score |
| BC5Chem (Li et al., ) | Entity extraction for chemical. |  |  |  |
| BC4Chem (Savery et al., ) | Entity extraction for chemical names from PubMed article abstracts. |  |  |  |
| Species800 (Pafilis et al., ) | Extraction of organism names from PubMed article abstracts. |  |  |  |
| HoC (Baker et al., ) | Classification of the hallmarks of cancer given biomedical article abstracts. |  |  |  |
| HumanDiseaseOntology (Schriml et al., ) | Disease ontology-based entity extraction. | BLEU, ROUGE,<br>AlignScore, Mauve |  |  |
| BioLORD (Remy et al., ) | Elaboration of biomedical concepts. |  |  |  |
| PMC-Patient (Zhao et al., ) | Patient-related entity (gender and age for example) extraction from PubMed Central articles. | Accuracy |  |  |
| PICO-Participant (Nye et al., ) | Information extraction of outcome, interven-<br>tion, and participant from article abstracts. |  |  |  |
| PICO-Intervention (Nye et al., ) |  |  |  |  |
| PICO-Outcome (Nye et al., ) |  |  |  |  |
| ADE Corpus (Gurulingappa et al., ) | Drug dose extraction given the drug information. |  |  |  |
| Text<br>Summary | MIMIC-IV-Ultrasound (Johnson et al., ) | Summarization of patient reports from emer-<br>gency departments. | BLEU, ROUGE,<br>AlignScore, Mauve |  |
| MIMIC-IV-CT (Wallace et al., ) |  |  |  |  |
| RCT-Text (Wallace et al., ) | Summarization of medical evidence from clinical studies in literature reviews. |  |  |  |
| MedQSum (Lee et al., ) | Summarization of patient notes, reports, and health records. |  |  |  |
| Reasoning | Knowledge<br>QA | PubMedQA (Jin et al., ) | QA data built on PubMed abstracts. | Accuracy |
| MedQA (Jin et al., ) | QA data for medical knowledge assessment. |  |  |  |
| MedMCQA (Pal et al., ) | QA data from AIIMS & NEET PG entrance exams. |  |  |  |
| LancetQA (^1^11[https://www.thelancet.com/](https://www.thelancet.com/)) | QA data crawled from Lancet picture quiz gallery. |  |  |  |
| NEJMQA (^2^22[https://www.nejm.org/](https://www.nejm.org/)) | QA and diagnostic challenge requests from NEJM quiz. |  |  |  |
| Medbullets (Chen et al., ) | QA data from Medbullets online medical study platform. |  |  |  |
| Clinical<br>Decision Support | DDXPlus (Fansi Tchango et al., ) | Diagnostic decision making of synthesized patient data. | Accuracy |  |
| SEER (Dubey et al., ) | Treatment planning for breast cancer cases. |  |  |  |
| MIMIC4ED-Hospitalization (Xie et al., ) | Prediction of clinical outcomes in emergency<br>medicine from MIMIC-IV-ED. |  |  |  |
| MIMIC4ED-72h ED Revisit (Xie et al., ) |  |  |  |  |
| MIMIC4ED-Critical Triage (Xie et al., ) |  |  |  |  |
| MedNLI-Dis. (Romanov & Shivade, ) | Discriminative entailment task for clinical hypotheses. |  |  |  |
| PUBHEALTH Ver. (Kotonya & Toni, ) | Verification of health-related information from the public. |  |  |  |
| EBMS (Mollá & Santiago-Martinez, ) | Justification verification using the EBMS corpus. | BLEU, ROUGE,<br>AlignScore, Mauve |  |  |
| PUBHEALTH Exp. (Kotonya & Toni, ) | Explanation of health-related information from the public. |  |  |  |
| ChatDoctor (Li et al., ) | Patient-doctor dialogues from online medical consultations. |  |  |  |
| MedNLI-Gen. (Romanov & Shivade, ) | Generative entailment task for clinical hypotheses. |  |  |  |
| Agent | AI Hospital (Fan et al., ) | Multi-agent task simulating dynamic medical interactions in Chinese. | Accuracy |  |
| AgentClinic (Schmidgall et al., ) | Agent benchmark in simulated clinical environments from MedQA and NEJMQA scenarios. |  |  |  |
| Medical<br>Calculation | MedCalc-Bench (Khandekar et al., ) | Medicine dose level calculation from ADE corpus. | Accuracy |  |
| Multi-<br>linguality | Knowledge<br>QA | XMedBench (Wang et al., ) | Multilingual benchmark for medical understanding and interaction. | Accuracy |
| Agent | AI Hospital (Fan et al., ) | Multi-agent task simulating dynamic medical interactions in Chinese. | Accuracy |  |

First, we present the taxonomy of our evaluation, along with an overview of the evaluation pipeline as shown in Figure 1.
Firstly, we specify three aspects of the model capabilities, namely *understanding*, *reasoning*, and *multilinguality*, that correspond to the real-world needs of clinical physicians.
To ensure a comprehensive evaluation, we collect a diverse range of medical tasks and datasets that fall under these three aspects.
Moreover, we explore three prompting strategies in our pipeline, including (1) direct prompting, which instructs LLMs to solve specific problems directly, (2) chain-of-thought, which requires models to think step-by-step before generating the final answer, (3) few-shot prompting, which providing models with several examples to learn the input-output mapping on the fly.
Lastly, appropriate metrics are utilized to measure the discrepancy between generated responses and ground-truth answers. Details about metrics utilized in each dataset are provided in Table 1.

### 3.2 Aspects and Tasks

In Table 1, our evaluation efforts are structured into three main parts: aspect, task, and dataset.
Specifically, a dataset refers to the data itself along with the metrics used in the current context. We utilize 35 existing datasets and create 2 additional challenging datasets for evaluation.
A task is a collection of multiple datasets that share a common goal or evaluate similar capabilities within the model. We categorize all 37 datasets into 6 tasks for clearer evaluation and analysis.
An aspect describes a specific capability or property to understand how well the model performs in a particular area.
In our evaluation pipeline, we focus on three key aspects.

Formally, we illustrate these three evaluation aspects with their corresponding tasks as follows:

- •
Understanding refers to the model’s ability to utilize its internal medical knowledge to comprehend medical concepts. For example, in concept recognition task, the model is required to extract or elaborate medical concepts from article (Savery et al., ; Pafilis et al., ; Nye et al., ) or diagnosis report (Zhao et al., ). And in text summarization, the model need to understand concepts in complex texts to generate a concise summary (Lee et al., ; Wallace et al., ; Johnson et al., ; ).
- •
Reasoning is the ability to conduct multiple steps of logical thinking to arrive at the conclusion. In question answering tasks, the model is prompted to select correct option from multi-choices based on reasoning derived from the medical information provided in the question. In addition to common question-answering datasets (Jin et al., ; Pal et al., ; Jin et al., ), we collect real-world clinical questions from The Lancet, the New England Journal of Medicine (NEJM), and Medbullets (Chen et al., ) to better assess the clinical utility of LLMs. In the clinical suggestion task, the model is required to provide treatment suggestions (Dubey et al., ; Li et al., ) or diagnostic decisions (Xie et al., ; Fansi Tchango et al., ) based on patients’ information. In the AI Hospital (Fan et al., ) and AgentClinic (Schmidgall et al., ) datasets, we task the model with serving as a medical agent. Furthermore, in the MedCalc-Bench (Khandekar et al., ) dataset, the model is required to perform mathematical reasoning and calculate answers.
- •
Multilinguality is the ability to complete a task when the languages of input instruction and/or output answers are changed to different languages. For example, XMedBench (Wang et al., ) dataset requires LLMs to answer medical questions in six languages, including Chinese, Arabic, Hindi, Spanish, Chinese and English. In AI Hospital dataset (Fan et al., ), the model is required to serve as an agent using Chinese.

### 3.3 Metrics

In this section, we elaborate on metrics employed in our evaluation pipeline.

- •
Accuracy is used to directly measure the percentage of models’ generated answer which exactly match with the ground-truth.
We use accuracy for multi-choice question datasets, MedCalc-Bench (Khandekar et al., ) dataset, and portions of clinical suggestion and concept recognition datasets where the ground-truth answer is a single word or phrase.
- •
F1-score (Pedregosa et al., ) is the harmonic mean of precision and recall. It is employed in datasets where the model is required to select multiple correct answers.
- •
BLEU (Papineni et al., ) and ROUGE (Lin & Hovy, ) are NLP metrics measuring the similarity between the generated respond and the ground-truth. Specifically, we utilize BLEU-1 and ROUGE-1 for all free-form generation tasks in our evaluation.
- •
AlignScore (Zha et al., ) is a metric to measure the factual consistency of generated text. In this paper, we use AlignScore for all free-form generation tasks to evaluate the extent of model’s hallucination.
- •
Mauve (Pillutla et al., ) is a measure of gap between distribution of generated and human-written text. It is employed for all free-form generation tasks.

All metrics range from 0 to 100, and a higher number indicates better quality output from the model.

## 4 Experiments

### 4.1 Experiment Details

#### Prompting strategies.

For most datasets, we employ the same prompting strategy as described in previous literature (Wu et al., ; Nori et al., ; ): For knowledge QA tasks, agent tasks, medical calculation tasks, and multilingual-related tasks, we use the direct prompting evaluation method, which is consistent with the settings of these benchmarks.
For other tasks derived from MedS-Bench (Wu et al., ), we follow their benchmark settings, leveraging a few-shot (3-shot) prompt strategy with its template shown in Section A.1.
As officially suggested by OpenAI, common prompting techniques such as Chain-of-Thought (CoT) (Wei et al., ) and in-context examples may not boost o1’s performance as it has implicit CoT built in.
To further validate this claim, we also investigate the effect of several advanced promptings in our evaluation (*e.g*., CoT, Self-Consistency (Wang et al., ), and Reflex (Shinn et al., )), the detailed input instruction formats are in Section A.1

#### Models for evaluation.

We choose the following models to evaluate: GPT-3.5 (gpt-3.5-turbo-0125)(^3^33[https://platform.openai.com/docs/models/gpt-3-5-turbo/](https://platform.openai.com/docs/models/gpt-3-5-turbo/)), an advanced language model by OpenAI known for its enhanced contextual understanding; GPT-4 (gpt-4-0125-preview) (Achiam et al., ), the successor to GPT-3.5 with significant improvements in reasoning and language comprehension; o1 (o1-preview-2024-09-12) (OpenAI, ), the lastest LLM model that is capable of performing highly complex reasoning by employing chain-of-thought reasoning.
Apart from these close-source models, we have also incorporated two open-source ones in our experiments: MEDITRON-70B (Chen et al., ), an LLM trained with medical-centric data and Llama3-8B (Meta, ), the latest and strongest open LLM right now.

**Table 2: Accuracy (Acc.) or F1 results on 4 tasks across 2 aspects. Model performances with * are taken from Wu et al. () as the reference. We use the gray background to highlight o1 results. And we present the average score (Average) of each metric in the table**
| Aspect | Task | Datasets | Metric | o1 | GPT-4 | GPT-3.5 | MEDITRON*<br>(70B) | Llama3*<br>(8B) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Understanding | Concept<br>Recognition | PMC-Patient (Zhao et al., ) | Acc. | 76.4 | 75.7 | 74.4 | 72.2 | 96.0 |
| PICO-Participant (Nye et al., ) | Acc. | 75.0 | 75.0 | 52.5 | 72.1 | 58.2 |  |  |
| PICO-Intervention (Nye et al., ) | Acc. | 77.5 | 75.0 | 75.0 | 46.6 | 79.1 |  |  |
| PICO-Outcome (Nye et al., ) | Acc. | 67.5 | 65.0 | 60.0 | 51.2 | 58.2 |  |  |
| ADE Corpus (Gurulingappa et al., ) | Acc. | 78.3 | 78.3 | 71.6 | 95.7 | 69.6 |  |  |
|  | Average | Acc. | 74.9 | 73.8 | 66.7 | 67.6 | 72.2 |  |
|  | BC5-disease (Li et al., ) | F1 | 69.5 | 63.0 | 38.9 | 1.4 | 25.3 |  |
|  | BC5Chem (Li et al., ) | F1 | 72.2 | 71.2 | 43.1 | 4.2 | 37.9 |  |
|  | BC4Chem (Savery et al., ) | F1 | 73.4 | 65.1 | 32.7 | 2.0 | 19.5 |  |
|  | Species800 (Pafilis et al., ) | F1 | 71.6 | 66.8 | 55.4 | 0.4 | 11.9 |  |
|  |  | HoC (Pafilis et al., ) | F1 | 76.3 | 59.0 | 59.8 | 23.7 | 38.3 |
|  |  | Average | F1 | 72.6 | 65.0 | 46.0 | 6.3 | 26.6 |
| Reasoning | Clinical<br>Decision Support | DDXPlus (Fansi Tchango et al., ) | Acc. | 64.0 | 56.0 | 41.0 | 29.6 | 33.8 |
| SEER (Dubey et al., ) | Acc. | 80.0 | 69.6 | 5.0 | 68.3 | 56.1 |  |  |
| MIMIC4ED<br>-Hospitalization<br>(Xie et al., ) | Acc. | 64.0 | 61.0 | 62.0 | 56.3 | 39.1 |  |  |
| MIMIC4ED<br>-72h ED Revisit<br>(Xie et al., ) | Acc. | 59.7 | 58.0 | 53.6 | 48.5 | 9.3 |  |  |
| MIMIC4ED<br>-Critical Triage<br>(Xie et al., ) | Acc. | 61.7 | 66.7 | 58.7 | 45.7 | 8.8 |  |  |
| MedNLI-Dis. (Romanov & Shivade, ) | Acc. | 88.0 | 84.0 | 57.0 | 60.9 | 63.9 |  |  |
| PUBHEALTH Ver. (Kotonya & Toni, ) | Acc. | 76.4 | 75.7 | 74.4 | 32.7 | 63.9 |  |  |
|  | Average | Acc. | 70.5 | 67.3 | 50.2 | 48.9 | 39.3 |  |
| Knowledge<br>QA | PubMedQA (Jin et al., ) | Acc. | 75.0 | 52.8 | 25.4 | 74.4 | 73.0 |  |
| MedQA (Jin et al., ) | Acc. | 75.5 | 69.7 | 53.8 | 47.9 | 60.9 |  |  |
| MedMCQA (Pal et al., ) | Acc. | 95.0 | 79.5 | 58.8 | 59.2 | 50.7 |  |  |
| Medbullets (Chen et al., ) | Acc. | 90.6 | 66.9 | 50.7 | - | - |  |  |
| LancetQA | Acc. | 81.5 | 76.0 | 61.0 | - | - |  |  |
| NEJMQA | Acc. | 91.2 | 83.5 | 65.0 | - | - |  |  |
|  |  | Average | Acc. | 84.8 | 71.4 | 52.5 | 60.5 | 61.5 |
|  | Medical Calculation | MedCalc-Bench (Khandekar et al., ) | Acc. | 34.9 | 25.5 | 10.8 | - | - |

**Table 3: BLEU-1 (B-1) and ROUGE-1 (R-1) results on 3 tasks across 2 aspects. We use the gray background to highlight o1 results. We also present the average score (Average) of each metric**
| Aspect | Task | Datasets | o1 | GPT-4 | GPT-3.5 | MEDITRON*<br>(70B) | Llama3*<br>(8B) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| B-1 $\uparrow$ | R-1 $\uparrow$ | B-1 $\uparrow$ | R-1 $\uparrow$ | B-1 $\uparrow$ | R-1 $\uparrow$ | B-1 $\uparrow$ | R-1 $\uparrow$ | B-1 $\uparrow$ | R-1 $\uparrow$ |  |  |  |
| Understanding | Text<br>Summary | MIMIC-IV-Ultrasound (Johnson et al., ) | 22.2 | 28.8 | 15.9 | 27.0 | 11.0 | 21.1 | 3.8 | 6.1 | 18.1 | 20.0 |
| MIMIC-IV-CT (Johnson et al., ) | 19.0 | 26.4 | 15.7 | 22.7 | 18.7 | 25.9 | 16.3 | 23.9 | 24.5 | 29.4 |  |  |
| RCT-Text (Wallace et al., ) | 19.5 | 23.4 | 19.5 | 23.4 | 20.6 | 24.2 | 4.0 | 16.4 | 15.4 | 14.6 |  |  |
| MedQSum (Lee et al., ) | 39.2 | 46.8 | 36.3 | 43.0 | 26.5 | 39.6 | 15.6 | 23.1 | 22.5 | 25.1 |  |  |
|  | Average | 25.0 | 31.4 | 21.8 | 29.0 | 19.2 | 27.7 | 9.9 | 17.4 | 20.1 | 22.3 |  |
| Concept<br>Recognition | HumanDO (Schriml et al., ) | 24.9 | 33.1 | 9.7 | 16.2 | 12.2 | 19.4 | 7.7 | 25.4 | 14.9 | 18.8 |  |
|  | BioLORD (Remy et al., ) | 23.0 | 31.8 | 14.7 | 21.8 | 12.8 | 19.1 | 11.8 | 22.7 | 8.9 | 14.6 |  |
|  |  | Average | 24.0 | 32.5 | 12.2 | 19.0 | 12.5 | 19.3 | 9.8 | 24.1 | 11.9 | 16.7 |
| Reasoning | Clinical<br>Decision Support | EBMS (Mollá & Santiago-Martinez, ) | 16.2 | 20.4 | 12.0 | 16.3 | 15.4 | 19.4 | 11.6 | 15.8 | 16.5 | 16.5 |
| PUBHEALTH Exp. (Kotonya & Toni, ) | 15.8 | 23.6 | 15.1 | 22.0 | 16.6 | 23.6 | 6.1 | 8.7 | 16.8 | 20.3 |  |  |
| ChatDoctor (Li et al., ) | 12.2 | 27.6 | 20.9 | 4.7 | 14.0 | 27.0 | - | - | - | - |  |  |
| MedNLI-Gen. (Romanov & Shivade, ) | 17.0 | 26.0 | 16.9 | 25.8 | 10.0 | 18.3 | 4.4 | 14.1 | 21.3 | 22.8 |  |  |
|  |  | Average | 15.3 | 24.4 | 16.2 | 17.2 | 14.0 | 22.1 | 7.4 | 12.9 | 18.2 | 19.9 |

### 4.2 Main Result: Yes! We are one Step Closer to an AI Doctor

#### Enhanced ability of o1 transfers to its clinical understanding.

Given the established results from o1, which underscore its remarkable effectiveness in knowledge and reasoning abilities such as mathematical problem-solving and code generation (OpenAI, ), we observe that this superior capability can also be transferred to the specific clinical knowledge understanding.
Results presented in Table 2 demonstrate that o1 outperforms other models on the *understanding* aspect in most clinical tasks.
We also present these statistics in Figure LABEL:fig:radar, where we observe that o1 has a larger cover radius across various medical datasets.
For instance, on 5 concept recognition datasets that use F1 as the metric, o1 outperforms both GPT-4 and GPT-3.5 by an average of 7.6% and 26.6%, respectively (i.e., 72.6% *vs*. 65.0% *vs*. 46.0%), with a notable 24.5% average improvement on the widely used BC4Chem dataset.

Additionally, on the summarization task in Table 3, o1 achieves a 2.4% and 3.7% increase in ROUGE-1 score over GPT-4 and GPT-3.5 (*i.e*., 31.4% *vs*. 29.0% *vs*. 27.7%), demonstrating its enhanced capacity for real-world clinical understanding.
This improved performance confirms that advancements in general NLP capabilities for LLMs can effectively translate to enhanced model understanding in the medical domain.

**Table 4: AlignScore and Mauve results on 3 tasks across 2 aspects**
| Aspect | Task | Datasets | AlignScore $\uparrow$ | Mauve $\uparrow$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| o1 | GPT-4 | GPT-3.5 | o1 | GPT-4 | GPT-3.5 |  |  |  |
| Understanding | Text<br>Summary | MIMIC-IV-Ultrasound (Johnson et al., ) | 27.5 | 30.9 | 23.6 | 6.1 | 7.4 | 7.3 |
| MIMIC-IV-CT (Johnson et al., ) | 14.4 | 13.3 | 13.8 | 0.4 | 0.5 | 0.5 |  |  |
| RCT-Text (Wallace et al., ) | 4.9 | 4.9 | 5.7 | 3.1 | 2.7 | 11.9 |  |  |
| MedQSum (Lee et al., ) | 34.5 | 37.1 | 13.6 | 42.1 | 52.7 | 0.6 |  |  |
|  | Average | 20.3 | 21.6 | 14.2 | 12.9 | 15.8 | 5.1 |  |
| Concept<br>Recognition | HumanDO (Schriml et al., ) | 17.5 | 5.5 | 5.2 | 8.2 | 0.4 | 0.4 |  |
|  | BioLORD (Remy et al., ) | 13.0 | 19.0 | 17.9 | 51.6 | 4.2 | 1.1 |  |
|  |  | Average | 15.3 | 12.3 | 11.6 | 29.9 | 2.3 | 0.8 |
| Reasoning | Clinical<br>Decision Support | EBMS (Mollá & Santiago-Martinez, ) | 9.0 | 6.6 | 5.7 | 19.5 | 1.9 | 2.3 |
| PUBHEALTH Exp. (Kotonya & Toni, ) | 14.8 | 19.0 | 17.9 | 2.1 | 0.8 | 1.1 |  |  |
| ChatDoctor (Li et al., ) | 26.5 | 20.4 | 16.6 | 0.7 | 0.5 | 0.6 |  |  |
| MedNLI-Gen. (Romanov & Shivade, ) | 6.8 | 9.7 | 2.5 | 5.3 | 4.5 | 0.9 |  |  |
|  |  | Average | 14.3 | 13.9 | 10.7 | 6.9 | 1.9 | 1.2 |

#### The o1 model demonstrates strong reasoning in clinical diagnosis scenarios.

On the reasoning aspect, o1 takes a significant step forward in demonstrating its advantages in real-world diagnostic situations.
In our newly constructed challenging QA tasks, NEJMQA and LacentQA, o1 showcases an average accuracy improvement of 8.9% and 27.1% over the performance of GPT-4 (79.6%) and GPT-3.5 (61.5%) on the respective datasets (Table 2).
Another noteworthy improvement in o1 is its capacity for mathematical reasoning, elevating the baseline of MedCalc-Bench to 34.9%, which surpasses GPT-4 by a significant 9.4%.
In more complex reasoning scenarios that involve multi-turn conversations and environmental simulations, o1 outperforms both GPT-4 and GPT-3.5 on the AgentClinic benchmark, achieving accuracy gains of at least 15.5% and 10% with scores of 45.5% and 20.0% on its MedQA and NEJM subsets, respectively.
These observations serve as compelling evidence of o1’s competence in complex real-world diagnosis and clinical utility scenarios.

Figure: Figure 2: Answers from o1 and GPT-4 on a question from LancetQA. o1 provides a more concise and accurate reasoning process compared to GPT-4.
Refer to caption: x2.png

In addition to delivering higher accuracy, o1 provides more concise and straightforward answers. In the example illustrated in Figure 2, o1 generates shorter interpretations while offering the correct answer.
In contrast, GPT-4 tends to generate hallucinated explanations alongside incorrect answers.
We believe o1’s improvement in both knowledge and reasoning is primarily attributed to the enhanced data and infrastructure employed during the training process (*e.g*., CoT data and the reinforcement learning technique).

These results together provide a positive answer to the question we raised in this paper: *Yes!* We are getting closer to an automatic AI doctor with the latest o1 model.

**Table 5: Accuracy of LLMs on two agentic benchmarks**
| Tasks | AI Hospital<br>(Fan et al., ) | AgentClinic<br>(Schmidgall et al., ) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Symp. | Medical<br>Exam. | Diagnostic<br>Results | Diagnostic<br>Rationales | Treatment<br>Plan | MedQA | NEJM |  |
| o1 | 67.0 | 43.4 | 45.1 | 45.1 | 39.9 | 45.5 | 20.0 |
| GPT-4 | 66.7 | 45.0 | 44.2 | 45.8 | 38.2 | 30.4 | 10.0 |
| GPT-3.5 | 62.0 | 40.7 | 35.8 | 36.3 | 24.7 | 25.2 | 7.5 |

### 4.3 Further Analysis

#### No model excels across all tasks in the medical domain.

Table 2 and Table 3 indicate that, for now, there are always trade-offs (even under the same metric) to be made when selecting a model to use in the medical domain.
One example is the clinical decision support task in Table 2, o1 outperforms both GPT-4 and GPT-3.5 on most datasets, but lags far behind GPT-4 on the MIMIC4ED-Critical Triage dataset by 5% in accuracy.
Interestingly, we also found the recent released open LLM—Llama3 takes a lead in PMC-Patient and PICO-Intervention datasets with an unexpected 19.6% accuracy gap between o1 and Llama3 on PMC-Patient (76.4% *vs*. 96.0%).
Nevertheless, o1 comes close to being the best in most situations, it boasts a leading position across datasets in clinical decision support, knowledge QA, and medical calculation.
This claim is supported by the average result over 19 dataset accuracy in Table 2 and Figure LABEL:fig:bar:
o1 (74.3%) $>$ GPT-4 (68.1%) $>$ GPT-3.5 (53.2%)

#### Advanced prompting can partially help models trained with CoT data.

o1 was released using chain-of-thought (CoT) data embedding in the training process; however, we found that applying the CoT prompting still enhances o1’s performance on knowledge QA tasks in medicine, as shown in Table 6. The table reveals an average boost of 3.18% over the original 83.6% accuracy of o1. While this improvement is not as significant as with GPT-4, CoT proves to be a promising way for guiding o1 in medical tasks.
However, when it comes to other fancy promptings, such as self-consistency (SC) (Wang et al., ) and reflex (Shinn et al., ), this conclusion may not stand still.
We witness an average performance decline of 12.8% using these two strategies compared to only CoT on LancetQA (Table 7).

**Table 6: Accuracy results of model results with/without CoT prompting on 5 knowledge QA datasets**
| Datasets | o1 | o1 (CoT) | GPT-4 | GPT-4 (CoT) |
| --- | --- | --- | --- | --- |
| PubMedQA (Jin et al., ) | 75.0 | 75.2 | 52.8 | 62.2 |
| MedQA (Jin et al., ) | 95.0 | 95.2 | 79.5 | 86.1 |
| MedMCQA (Pal et al., ) | 75.5 | 81.9 | 69.7 | 72.6 |
| LancetQA | 81.5 | 85.5 | 76.0 | 81.5 |
| NEJMQA | 91.2 | 96.3 | 83.5 | 86.4 |

**Table 7: Accuracy ablation results of using different promptings using o1 on our LancetQA**
| CoT | SC | Reflex | Accuracy |
| --- | --- | --- | --- |
|  |  |  | 81.5 |
| $\checkmark$ |  |  | 85.5 |
| $\checkmark$ | $\checkmark$ |  | 84.5 |
| $\checkmark$ |  | $\checkmark$ | 61.0 |

**Table 8: Accuracy of models on the multilingual task, XmedBench (Wang et al., )**
| Models | English | Chinese | French | Spanish | Arabic | Hindi | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| o1 | 76.4 | 80.2 | 95.4 | 95.0 | 74.9 | 89.3 | 85.2 |
| GPT-4 | 75.7 | 61.0 | 89.4 | 91.2 | 60.8 | 76.3 | 75.7 |
| GPT-3.5 | 72.0 | 47.4 | 58.9 | 74.2 | 39.7 | 32.5 | 54.1 |
| Meditron-70B* | 58.7 | 44.3 | 53.3 | 59.7 | 19.3 | 31.3 | 44.4 |

#### Hallucination remains a significant challenge.

We use AlignScore (Zha et al., ) to evaluate hallucination in LLMs. In Table 4, the o1 model demonstrates a 1.3% decrease in AlignScore compared to GPT-4 across five text summarization datasets.
Moreover, the overall improvements of o1 across three tasks (Table 4) in AlignScore significantly lag behind those of other evaluation metrics—averaging 0.7 in AlignScore compared to 9.9 in Mauve relative to GPT-4. This indicates that o1 is still susceptible to language hallucination, highlighting that such problem remains a persistent challenge in LLMs.

#### o1 struggles in reasoning over complex multilingual tasks.

Advanced LLMs are expected to demonstrate equivalent reasoning abilities to languages other than English.
However, as o1 consistently outperforms other models in multilingual QA tasks: o1 (85.2%) $>$ GPT-4 (75.7%) $>$ GPT-3.5 (54.1%) on average (Table 8), it falls short in a much more complex Chinese agent benchmark in Table 5—showing a 1.6% accuracy drop in the medical examinations scenario over GPT-4 (43.4% *vs*. 45.0%), leaving its multilingual reasoning in complex situations to be desired.
This interesting outcome might be attributed to the lack of multilingual CoT data during o1’s training, as learning complex reasoning routes generally requires more efforts than plain instructions in the few-shot paradigm (Kim et al., ; Singh et al., ).
We present a failure example of o1 on AI Hospital in Figure 3. We identified instances of mixed language output in the generation from the doctor, which contribute to the suboptimal performance of o1 in this context.

#### LLMs are facing biased judgement using different metrics.

Choosing different metrics can lead to varied results of LLM evaluation (Liang et al., ), in our experiments, we observe a similar unaligned trend even leveraging traditional NLP metrics such as BLEU-1, ROUGE-1, and Mauve.
In most cases from Table 3, o1 surpasses GPT-4 in both two traditional reference-based measurements (*i.e*., BLEU-1, ROUGE-1) on average.
One exception arises in the BLEU-1 comparison for clinical suggestion tasks. While o1 significantly triumph over GPT-4 in ROUGE-L (24.4% *vs*. 17.2%), it surprisingly underperforms in BLEU-1: o1 (15.3) $<$ GPT-4 (16.2).
When considering Mauve scores, although o1 consistently surpasses GPT-4 in both averaged BLEU-1 and ROUGE-1 for text summarization tasks, it still falls short by 2.9 points in Mauve, even when evaluated on the same output texts.
A similar anomaly can also be observed in the comparison between accuracy and F1 score. While Llama3 significantly outperforms o1 in accuracy on two concept recognition datasets, it consistently falls behind o1 in F1 on the same cases.
These findings underscore the urgent need to identify or devise more reliable metrics for modern LLMs.

## 5 Discussion

#### What adverse impacts does o1 bring?

The model o1 has made significant strides in both general NLP and the medical domain—as demonstrated in this paper. But what adverse impacts does o1 have on users compared to the previous generations of LLMs?
While embedding the Chain of Thought (CoT) process during generation by default requires more time (OpenAI, ), what exactly distinguishes o1 from other OpenAI models?
In Table 10, we see that o1 has more than 2$\times$ and 9$\times$ longer decoding time cost on four medical tasks compared to GPT-4 and GPT-3.5, respectively (13.18s *vs*. 6.89s *vs*. 1.41s).
This increased decoding time can lead to significant waiting periods when handling complex tasks.

Additionally, o1 does not always outperform other models, with inconsistent performance across different tasks. For instance, in the concept recognition task detailed in Table 2, o1 underperforms compared to other LLMs on half of the datasets.
This discrepancy may relate to recent findings suggesting that CoT data is most advantageous in more complex reasoning tasks (Sprague et al., ). However, in tasks that do not require complex reasoning, such as concept recognition, o1 does not have significant advantages over them.

#### Rethinking evaluation metrics for stronger LLMs.

Traditional evaluation metrics like BLEU and ROUGE, which rely on n-gram overlap, have long been criticized for their limitations in capturing the quality of generated text, particularly for LLMs. As a result, using models like GPT-4 as evaluators, *i.e*., “LLM-as-a-judge”, has gained popularity for assessing the outputs of other models. However, this approach may not be valid when applied to the most advanced models such as o1, as GPT-4 is even less capable and thus may produce less reliable evaluation. This is especially true for specialized domain like medicine. Therefore, there is a growing need to develop more robust and nuanced evaluation metrics that can better assess the performance of state-of-the-art LLMs in complex scenarios.

#### Call for reliable prompting techniques for future LLMs.

As noted in Section 4.3, not all advanced prompting techniques positively impact o1’s performance.
As future LLMs like o1 may continue to evolve with internal prompts for efficient user instruction, new prompting methods should consider their adaptability to existing strategies.
One potential exploration could be the integration of two prompting strategies (Wang et al., ; Zheng et al., ).

#### Limitations.

While we conduct comprehensive evaluations in the medical domain on understanding, reasoning, and multilingual capabilities, there are many other dimensions to consider such as safety (Han et al., ) and we leave them for future work.
Additionally, we leave more advanced prompting techniques such as retrieval augmented generation (RAG) (Lewis et al., ) for future work, which may enhance the factuality and mitigate hallucination.
It is worth noting that current GPT-like models may still underperform BERT-based specialists in classification tasks (Nori et al., ). However, we focus on GPT-like generalists in this paper due to their greater flexibility as zero-shot learners.

## 6 Conclusion

This preliminary study assesses 3 important aspects across 35 existing and 2 novel medical datasets using the latest o1 model. It marks the first step towards a holistic evaluation of o1 in medicine, and we present our initial results, analysis, and discussion over the benchmark. The findings provide convincing evidence that o1 is narrowing the gap between AI and human doctors, shaping the vision of an ideal AI doctor closer to reality.

## Acknowledgement

This work is partially supported by the OpenAI Researcher Access Program and Microsoft Accelerate Foundation Models Research Program. Q.J. is supported by the NIH Intramural Research Program, National Library of Medicine. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.

## Appendix A Supplemental Material

### A.1 Prompting Strategies

### A.2 Details about Datasets

In this paper, we present a summary of 36 medical-related datasets spanning 6 distinct tasks, as outlined in Table 1. Notably, the inclusion of commercial models, particularly o1, leads to significant costs and response latency. To address this, for some tasks we randomly sampled a subset of test cases, which are detailed below.

#### Concept Recognition

- •
BC4Chem (Savery et al., ) is a dataset comprising 10,000 PubMed abstracts with 84,355 chemical entity mentions, manually annotated by expert chemistry literature curators. The task is to extract chemical names from the given abstracts. For evaluation, we randomly sample 300 instances from the test set.
- •
BC5Chem and BC5Disease are from BC5CDR (Li et al., ), a widely-used resource in biomedical natural language processing, annotated for chemical and disease entities and their relationships. Following MedS-Bench (Wu et al., ), BC5CDR is split into 2 datasets: chemical name extraction and disease name extraction. For evaluation, we randomly sample 300 instances from each task’s test set.
- •
Species800 (Pafilis et al., ) comprises 800 PubMed abstracts with annotated organism mentions. The task is to extract organism names from the given abstracts. For evaluation, we randomly sample 300 instances from the test set.
- •
HoC (Baker et al., ) is a specialized dataset containing 1,852 PubMed publication abstracts, expertly annotated according to a taxonomy of cancer hallmarks. The task is to classify the hallmarks of cancer based on the given biomedical publication abstracts. For evaluation, we use the entire test set consisting of 158 instances.
- •
HumanDiseaseOntology (Schriml et al., ) is a database providing consistent, reusable, and sustainable descriptions of human disease terms, phenotype characteristics, and related medical vocabularies. The task is to explain specified medical professional entities, with the database descriptions serving as ground truth. For evaluation, we randomly sample 300 instances.
- •
BioLORD (Remy et al., ) comprises pairs of biomedical concept names and descriptions. The task is to elaborate on concise concepts by generating long, detailed definitions. For evaluation, we randomly sample 300 instances.
- •
PMC-Patient (Zhao et al., ) is a collection of 167,000 patient summaries extracted from case reports in PubMed Central (PMC), annotated with basic patient information. The task is to extract patient gender and age information from given clinical texts. For evaluation, we randomly sample 300 instances.
- •
PICO-Participant, PICO-Intervention and PICO-Outcome are three datasets derived from PICO (Nye et al., ), consisting of 5,000 abstracts from medical articles on randomized controlled clinical trials. The tasks involve extracting information about study participants, interventions, and outcomes from given sentences. For evaluation, we use the entire test set of 43 instances for each task.
- •
ADE Corpus (Gurulingappa et al., ) provides information on drugs and their corresponding adequate doses within sentences. The task is to extract the dosage levels of specified drugs from given sentences and drug names. We use the dataset prompted by Super-Instruction with a 9:1 ratio for instruction tuning and evaluation. The test set consists of 23 instances.

#### Text Summary

- •
MIMIC-IV-CT and MIMIC-IV-Ultrasound (Johnson et al., ; Wallace et al., ) are subsets of MIMIC-IV Report, a large deidentified medical dataset of patients admitted to the Beth Israel Deaconess Medical Center. The task is to summarize radiology reports, treating the impression part as a general summary of the findings. Following (Wu et al., ), we randomly sampled 500 cases from body region part of Chest CT and 100 cases from ultrasound modality for evaluation.
- •
RCT-Text (Wallace et al., ) is a dataset for summarizing medical evidence from clinical studies in literature reviews. The task is to output the primary conclusions of each study given the titles and abstracts. For evaluation, we randomly sample 100 instances.
- •
MedQSum (Lee et al., ) is derived from a large database of de-identified health-related data. The task is to generate a summary of detailed findings from imaging diagnostic reports, with the conclusion of the note serving as ground truth. For evaluation, we randomly sample 100 instances.

#### Knowledge QA

- •
MedQA (Jin et al., ) is a collection of medical multiple-choice questions in English. We use the 4-option English version with the official split. The test set contains 1273 samples.
- •
PubMedQA (Jin et al., ) is an English question-answering dataset based on PubMed abstracts. The task is to answer research questions with yes/no/maybe. We use the PQA-L subset as the test set, containing 1000 samples.
- •
MedMCQA (Pal et al., ) is a large-scale English multiple-choice question-answering dataset from AIIMS & NEET PG entrance exams. We use the official test split containing 4183 questions, each with 4 choices.
- •
LancetQA and NEJMQA are datasets curated from The Lancet and the New England Journal of Medicine case challenges, focusing on patient diagnosis based on symptoms. We use 200 samples for LancetQA and 100 samples for NEJMQA.
- •
Medbullets (Chen et al., ) is a dataset curated from the Medbullets online platform, comprising 308 USMLE Step 2&3 style questions. Each question includes a case description, four answer choices, and an explanation.

#### Clinical Decision Support

- •
DDXPlus (Fansi Tchango et al., ) is a dataset for Automatic Symptom Detection and Automatic Diagnosis systems, featuring synthesized patient data. The task is to make diagnostic decisions based on dialogues. For evaluation, we randomly sample 300 instances.
- •
SEER (Dubey et al., ) is a treatment planning dataset based on the Surveillance, Epidemiology, and End Results breast cancer databases. The task is to recommend treatment plans from five types. For evaluation, we randomly sample 300 instances.
- •
MIMIC4ED-Hospitalization, MIMIC4ED-72h ED Revisit, and MIMIC4ED-Critical Triage are datasets from the MIMIC4ED Benchmark (Xie et al., ) for predicting clinical outcomes in emergency medicine. For each dataset, we randomly sample 300 instances for evaluation.
- •
MedNLI-Dis. (Discriminative) and MedNLI-Gen. (Generative) are derived from MedNLI (Romanov & Shivade, ), a natural language inference dataset for the clinical domain. The dataset involve discriminative and generative entailment based on clinical premises. For each task, we randomly sample 300 instances for evaluation.
- •
EBMS (Mollá & Santiago-Martinez, ) is a justification verification dataset. We use the entire test set of 304 instances for evaluation.
- •
PUBHEALTH Exp. (Explanation) (Kotonya & Toni, ) requires models to provide explanations for specified claims using supporting material from given paragraphs. For evaluation, we randomly sample 300 instances.
- •
PUBHEALTH Ver. (Verification) (Kotonya & Toni, ) is a fact verification task where models determine if a claim contradicts evidence in a given paragraph. For evaluation, we randomly sample 300 instances.
- •
Chatdoctor (Li et al., ) is based on 100K patient-physician conversations from an online medical consultation website(^4^44www.healthcaremagic.com). The task involves engaging in medical consultations based on this data.

#### Agent

- •
AI Hospital (Fan et al., ) is a multi-agent framework simulating medical interactions in Chinese. It includes Patient, Examiner, Chief Physician, and Doctor agents, with 506 cases from diverse departments. The task involves simulating clinical scenarios through dialogue. Evaluation uses Chief Physician’s 1-4 scale scoring across five dimensions: symptoms, examinations, diagnostic results, rationales, and treatment plan. 200 cases are sampled for evaluation.
- •
AgentClinic (Schmidgall et al., ) is a clinical environment benchmark with 107 patient agents from MedQA and 15 multimodal agents from NEJM challenges. The task is patient diagnosis through dialogue and data collection. Evaluation considers diagnostic accuracy and patient perception metrics in biased scenarios.

#### Medical Calculation

- •
MedCalc-Bench (Khandekar et al., ) evaluates LLMs’ medical calculation abilities using 1,047 instances across 55 tasks. It requires computing medical values from patient notes and questions. Evaluation compares LLM outputs to ground truth, with exact matches for rule-based and 5% tolerance for equation-based calculators.

#### Multilinguality

- •
XMedBench (Wang et al., ) is a multilingual medical benchmark in six languages: English, Chinese, Hindi, Spanish, French, and Arabic. It uses multiple-choice questions from various sources, including translated versions for Arabic and Hindi. The task evaluates LLMs’ medical knowledge across languages, using accuracy as the primary metric.
- •
AI Hospital (Fan et al., ) is a multi-agent framework simulating medical interactions in Chinese. We also include this dataset into the multilinguality aspect because it is in Chinese.

### A.3 Model-based Evaluation

As discussed in Section 5, Rethinking evaluation metrics for stronger LLMs, we also explore using techniques such as "LLM-as-a-judge" to assess the quality of generated outputs. Table  9 shows that o1 achieves nearly the same score as GPT-4 and outperforms GPT-3.5 (i.e., 3.3% vs. 3.3% vs. 3.0%), which contrasts with the traditional evaluation metrics in Table 3. This indicates that the “LLM-as-a-judge” method may be unreliable when applied to advanced models like o1, as GPT-4, being less capable, may provide less accurate evaluations. This limitation is particularly evident in specialized domains such as medicine. The prompt used for "LLM-as-a-judge" is shown in Figure A.3.

**Table 9: GPT Evaluation Score Comparison**
| Task | Datasets | GPT Score $\uparrow$ |  |  |
| --- | --- | --- | --- | --- |
| o1 | GPT-4 | GPT-3.5 |  |  |
| Text Summarization | medqsum | 4.1 | 3.8 | 4.1 |
| RCT-Text | 3.2 | 3.2 | 3.1 |  |
| MIMIC-IV-Ultrasound | 3.8 | 3.8 | 3.4 |  |
| MIMIC-IV-CT | 3.8 | 3.8 | 3.7 |  |
| Clinical Suggestion | MedNLI-Generative | 2.3 | 2.4 | 2.5 |
| EMBS Justification Ver. | 3.1 | 3.0 | 3.0 |  |
| PUBHEALTH Exp. | 3.0 | 3.3 | 3.2 |  |
| Do Entity Exp. | 3.7 | 3.6 | 3.3 |  |
| BioLORD Concept Exp. | 3.3 | 3.3 | 3.0 |  |
| ChatDoctor | 2.5 | 2.6 | – |  |
| Average | 3.3 | 3.3 | 3.3 |  |

### A.4 Decoding Time

**Table 10: Model time cost and averaged number of decoding tokens for 4 datasets across 4 tasks**
| Task | Dataset | Model | Time (s) | Prompt<br>Tokens | Completion<br>Tokens | Reasoning<br>Tokens | Total<br>Tokens |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Knowledge<br>QA | MedQA | o1 | 11.13 | 247.78 | 953.42 | 924.16 | 1201.20 |
| GPT-4 | 0.83 | 236.20 | 9.26 | 0 | 245.46 |  |  |
| GPT-3.5 | 0.52 | 236.20 | 10.02 | 0 | 246.22 |  |  |
| Clinical<br>Decision Support | ChatDoctor | o1 | 11.40 | 122.64 | 1127.44 | 83.84 | 1250.08 |
| GPT-4 | 18.88 | 124.24 | 509.28 | 0 | 633.52 |  |  |
| GPT-3.5 | 2.40 | 124.24 | 150.10 | 0 | 274.34 |  |  |
| Text<br>Summary | MIMIC-IV | o1 | 20.56 | 1305.54 | 1080.54 | 1057.28 | 1373.32 |
| GPT-4 | 6.26 | 1254.84 | 162.68 | 0 | 1417.52 |  |  |
| GPT-3.5 | 2.02 | 1254.84 | 159.94 | 0 | 1414.78 |  |  |
| Concept<br>Recognition | BC5Chem | o1 | 9.62 | 292.78 | 1080.54 | 1057.28 | 1373.32 |
| GPT-4 | 1.60 | 297.24 | 19.64 | 0 | 316.88 |  |  |
| GPT-3.5 | 0.68 | 297.24 | 12.80 | 0 | 310.04 |  |  |

We evaluated the model’s time cost and the average number of decoding tokens across various tasks, including Knowledge QA, Clinical Decision Support, Text Summary, and Concept Recognition. For each task, we select a representative dataset and perform inference on 50 samples. The time and decoded tokens are then averaged to obtain the results for each response, as illustrated in Table.  10. The decoding time for o1 is significantly higher than both GPT-4 and GPT-3.5, taking more than double the time of GPT-4 and over nine times that of GPT-3.5 across four medical tasks (13.18s compared to 6.89s and 1.41s, respectively).

### A.5 Case Study

To demonstrate how the advanced o1 model outperforms previous methods in potential clinical applications, we present comparative case studies. As shown in Figure  4 and Figure  2, when responding to questions from NEJMQA and LancetQA, o1 provides the correct answer with a more concise and precise reasoning process. In contrast, GPT-4 generates a longer reasoning path while leading to an incorrect answer. in Figure  5, when offering diagnosis and treatment based on patient records, o1 delivers a more accurate diagnosis and practical treatment plan, closely aligning with human experts and outperforming GPT-4. These cases demonstrate that o1 more closely resembles an AI Doctor.

Figure: Figure 3: Failure case of o1 on AI Hospital. The model struggles with generating the right diagnosis and outputs mixed-language, resulting to its suboptimal performance in this context.
Refer to caption: x3.png

Figure: Figure 4: Comparison of the answers from GPT-o1 and GPT-4 for a question from NEJMQA. o1 provides a more concise and accurate reasoning process compared to GPT-4.
Refer to caption: x4.png

Figure: Figure 5: Comparison of the answers from GPT-o1 and GPT-4 for a case from the Chinese dataset AI Hospital, along with its English translation. o1 offers a more precise diagnosis and practical treatment suggestions compared to GPT-4.
Refer to caption: x5.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: Q1: Can the authors provide an accurate and detailed description of the data scraping, processing, proofreading, total number of items, and whether the two datasets will be open-sourced?

Q2: Are there any additional examples or detailed explanations that could further explore the reasons and extent of the phenomena "o1 still benefits from CoT" and "o1's hallucination phenomenon remains serious"?

Q3: Regarding the hallucination issue with o1, can it be improved through prompt or encoding parameters adjustments? Does this issue occur randomly, or is it consistently present in certain types of questions?
soundness: 3
strengths: - This is the first systematic evaluation of o1 medical capabilities, providing the community with a glimpse into o1's medical reasoning abilities and offering first-hand data insights.

- Provide a detailed analysis of o1's experimental results, highlighting the inconsistency in performance rankings across different computational metrics for open-ended question-answering tasks.

- The authors found that o1 can still benefit from Chain-of-Thought (CoT) techniques and remains constrained by the issue of hallucinations.

- Introduce two novel and challenging QA datasets.
summary: This paper represents the first systematic evaluation of o1 medical capabilities within the community, providing a reference framework for the medical LLM community through comprehensive dataset collection. Through thoughtful analysis of the experimental process and results, the authors identify existing shortcomings in o1 and raise practical issues, such as the lack of evaluation metrics, for the community's consideration.
weaknesses: > Limited Contribution to the Community in Terms of Data and Technology:

- In terms of data, the article introduces two novel and challenging datasets, but it does not clearly state whether they will be open-sourced. Additionally, the processes of data scraping, processing, proofreading, and the total number of items for both datasets are not described in the main text. For other collected datasets, the article does not design cross-experiments or conduct secondary processing, resulting in limited insights derived from data collection and experimental evaluation design, aside from those provided by the o1 model itself.

- Technologically, the paper raises issues related to text similarity computation metrics, prompt design, and hallucination through experimental analysis, yet it does not attempt to mitigate even one of these problems.

> Minor Flaws in Presentation and Result Analysis:

The results presented in the paper, particularly the surprising findings that "o1 still benefits from CoT" and "o1's hallucination phenomenon remains serious," lack examples to help readers grasp these concepts intuitively. Furthermore, the authors should provide more analysis, possibly incorporating examples or statistical data, to explain the reasons behind these phenomena.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Major Comments

1. In line 142, the authors state: "Our experimental findings reveal that with enhanced understanding, reasoning, and multilingual medical capabilities, o1 makes a step closer to a reliable clinical AI system."
What specific experiments demonstrate that o1 has moved closer to a reliable clinical AI system due to these "enhanced understanding, reasoning, and multilingual medical capabilities"?

2. Related to the first point, in line 151, the authors mention: "understanding, reasoning, and multilinguality, that correspond to the real-world needs of clinical physicians."
This claim requires supporting evidence. How does multilinguality specifically relate to the real-world needs of clinical physicians?

3. I am curious why MedQA, MedMCQA, and PubMedQA are categorized under "Reasoning" rather than "Understanding." According to the paper (line 226), "Understand refers to the model’s ability to utilize its internal medical knowledge," and in line 183, MedQA is described as "QA data for medical knowledge assessment."
Are all datasets classified under KnowledgeQA tasks truly indicators of reasoning ability? In the paper "Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine," LLMs' reasoning capabilities are assessed by having physicians evaluate the reasoning process beyond just the answers. Simply measuring accuracy on static questions seems more related to assessing internal knowledge rather than reasoning. Could the authors clarify this categorization?

4. In Tables 2 and 3, the authors compare Meditron and Llama using results from other papers rather than conducting direct experiments.
Why didn't the authors directly experiment with Llama and Meditron? For instance, on the PMC-Patient dataset, o1 scores 76.4 while Llama3-8b scores 96.0, indicating a mismatch in evaluation schemes. The current comparisons mainly involve o1, GPT-4, and GPT-3.5, effectively excluding Llama and Meditron. The comparison models seem insufficiently robust. Would it not strengthen the study to include models like Claude or Gemini?

5. The authors introduce LancetQA and NEJMQA datasets but provide limited details about them.
Why were LancetQA and NEJMQA created, and how were they constructed? The paper lacks detailed explanations beyond a brief mention in the appendix (line 967): "LancetQA and NEJMQA are datasets curated from The Lancet and the New England Journal of Medicine case challenges, focusing on patient diagnosis based on symptoms. We use 200 samples for LancetQA and 100 samples for NEJMQA." More comprehensive information about the purpose and construction of these datasets is necessary.

Minor Comments

6. The Abstract and Introduction mention benchmarking on 37 medical datasets.
Why does Figure 2 report averages based on only 19 datasets?

7. At line 162, it's stated: "Asterisks (*) denote the newly constructed datasets from public sources."
There are no asterisks in the table.

8. The experiments include Meditron-70b but not Llama3-70b.
Why was Llama3-70b excluded from the experiments? Considering that Meditron-70b is based on further medical pretraining of Llama2-70b, comparing Meditron-70b with Llama2-70b could offer valuable insights.

9. The authors claim (line 303): "We investigate the effect of several advanced promptings in our evaluation," and (line 431): "When it comes to other fancy promptings, such as self-consistency and reflexion, this conclusion may not stand."
Is it sufficient to make these conclusions based on a single dataset (LancetQA)? It seems additional data would strengthen these claims.
soundness: 2
strengths: - Comprehensive Evaluation: The paper attempts a broad assessment of o1's capabilities in the medical domain, covering understanding, reasoning, and multilinguality.

-  Introduction of New Datasets: By creating LancetQA and NEJMQA, the authors provide more clinically relevant and challenging benchmarks that may better reflect real-world medical scenarios.
summary: The paper evaluates OpenAI's latest large language model, "o1," in the context of medical applications. Focusing on understanding, reasoning, and multilingual capabilities, the authors assess o1's performance across various medical scenarios using six tasks derived from 37 medical datasets. Notably, they introduce two new and more challenging QA tasks—LancetQA and NEJMQA—based on professional medical quizzes from The Lancet and the New England Journal of Medicine. The findings suggest that o1 surpasses the previous GPT-4 model by an average of 6.2% and 6.6% in accuracy across the evaluated datasets. Despite these improvements, the paper also highlights weaknesses such as hallucinations, inconsistent multilingual abilities, and discrepancies in evaluation metrics.
weaknesses: - Datasets like MedQA, MedMCQA, and PubMedQA are categorized under "Reasoning" despite being primarily assessments of medical knowledge. This raises concerns about whether the tasks truly evaluate reasoning or merely recall of information. The methodology for evaluating reasoning appears limited, relying on static questions and accuracy metrics rather than assessing the reasoning process.

- Results for models like Meditron and Llama are taken from other papers, leading to discrepancies in evaluation methods. For instance, the significant performance difference on the PMC-Patient dataset suggests that evaluation schemas may not be consistent. The paper also does not include experiments with competitive models like Claude or Gemini, limiting the robustness of the comparative analysis.

- The creation and justification for LancetQA and NEJMQA are inadequately described. The paper lacks details on how these datasets were constructed, their content, and why they were necessary.

- Statements about advanced prompting techniques affecting o1's performance are based on limited data from a single dataset (LancetQA). This is insufficient to generalize conclusions about the efficacy of such techniques.

- Overall, the experiments and explanations are too insufficient concerning the various points claimed in the paper. It feels like this paper is 
 just simply testing GPT o1 on several benchmarks and reporting the scores.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: While the paper in its current form is not suitable for publication, it contains potentially valuable insights that could significantly contribute to the field. The most interesting finding appears to be the inadequacy of current testing methodologies, particularly ROUGE/BLEU, in evaluating free-form text medical capabilities. For critical domains such as medicine, it is paramount that the BioNLP community uses appropriate methodologies to ensure patient safety.

## Reframing the paper

The paper could be substantially improved by reframing its focus on the limitations of current free-form text evaluations in medical AI. This reframing should include a comprehensive review of existing evaluation methods, a detailed analysis of why ROUGE/BLEU metrics are insufficient for medical text evaluation, and proposals for more appropriate evaluation techniques. By shifting the focus, the paper could make a significant contribution to the ongoing discussion about responsible AI development in healthcare.

## Methodology

1. The statistical rigor needs enhancement. This includes conducting and reporting appropriate statistical analyses to compare model results, including confidence intervals or p-values for reported metrics.

2. The benchmark selection and reporting require justification and improvement. The inclusion of benchmarks with small sample sizes, such as NEJMQA with only 100 samples, raises concerns about result reliability. The authors should either exclude these or discuss their limitations extensively.

3. For custom benchmarks like LancetQA and NEJMQA, a detailed methodology is needed. This should explain how questions with images or non-textual media were handled, the process for removing or substituting images with textual findings, and the potential impacts of these modifications on benchmark validity.

4. Incorporating human evaluators for at least a portion of the free-form text outputs would significantly enhance the paper's credibility. This should include a description of the human evaluation methodology, rater qualifications, and inter-rater reliability measures.

5. The results could be strengthened by evaluating more appropriate models such as Claude 3.5 Sonnet and Llama3.1 405b in addition to the existing evaluations.

6. A thorough review of the benchmarking code is necessary to address discrepancies with previous work. Making the evaluation code publicly available would allow independent verification of results, enhancing transparency and reproducibility.

## Results

7. The authors should revise their interpretation of results, avoiding direct links between BLEU/ROUGE metrics and "real-world clinical understanding" without supporting evidence. A more nuanced discussion of what the results might indicate, acknowledging the limitations of the metrics used, would be more appropriate.
soundness: 1
strengths: The paper addresses a timely and important topic: the evaluation of the medical capabilities of OpenAI's latest o1 model. This focus is particularly relevant given the increasing deployment of these models in clinical settings. The paper is generally well-written and structured, which aids in navigating its content.

The paper uses a wide range of benchmarks, uses different evaluation metrics and introduces two new benchmarks which contributes to demonstrating that the improvements of o1 over previous models is not limited to a few benchmarks.

The analysis in section 4.3 and the discussion provide valuable insights. These sections accurately describe the limitations of the metrics used and highlight the need for new evaluation methodologies that more accurately assess medical capabilities. This critical perspective on current evaluation methods is the paper's most significant contribution.

The appendix contains useful additional data, including the prompts used and decoding times, which adds a degree of transparency to the study. The inclusion of case studies, while limited, provides concrete examples that complement the main text.
summary: The work presented aims to evaluate the medical capabilities of OpenAI's latest `o1` model using 37 medical benchmarks, including tasks in concept recognition, clinical decision support and knowledge QA. The authors use a different set of metrics depending on the benchmarks, such as accuracy, f-score, BLEU/ROUGE and AlignScore. The authors compare the results with previous models `GPT-4`, `GPT-3.5` and open-source models `Meditron-70b` and `Llama3 8b`. The results demonstrate overall higher raw scores compared to previous models. Additional results demonstrate that using CoT prompting techniques on the older models improve their performance. The authors briefly discuss the limitations of the metrics used and the need for new evaluation methods. Based on these results, the authors conclude that these scores demonstrate that the `o1` model is closing the gap with human physicians.
weaknesses: While the paper addresses a gap in literature, the methodology used, the reported results and especially their interpretation raise major concerns.    
    
## Evaluation pipeline

1. The use of BLEU and ROUGE metrics for assessing tasks such as summarization is questionable, given that these metrics primarily measure n-gram overlap. This concern is further amplified by the results in Tables 3 and 4, which demonstrate no correlation between AlignScore and the BLEU/ROUGE results. This discrepancy raises doubts about the interpretation of results for these tasks.

2. The model selection appears limited in scope. A more comprehensive comparison would include comparable models from competitors, such as Claude 3.5 Sonnet. Additionally, the decision to compare OpenAI's latest and most powerful model to the 8b parameter version of Llama 3 is questionable. A more appropriate comparison would involve evaluating Llama 3 with 405b parameters.

## Experiments

3. There are significant inconsistencies in the reported results, raising concerns about their validity. For instance:
    - GPT-4 and GPT-3.5 are reported to achieve 52.8% and 25.4% on PubMedQA, respectively. However, previously reported results by Microsoft and OpenAI were 75.2% and 71.6% in zero-shot settings [1].
    - Inconsistencies exist between tables, such as the apparent inversion of MedQA and MedMCQA results in Table 6 compared to Table 2.
    - The NEJMQA results are not whole numbers, which is puzzling given that only 100 questions were used for evaluation.
The lack of reproducible code exacerbates these concerns.

4. The results presented in Table 2 are incomplete, with Meditron and Llama 3 lacking results for MedBullets, LancetQA, NEJMQA, and MedCalc-Bench.

5. The reported results, while adequately formatted, lack statistical significance or confidence intervals. This omission is particularly problematic for benchmarks that were subsampled due to budget constraints, such as NEJMQA (evaluated with only 100 questions). A t-test comparing the results of o1 and GPT-4 on this benchmark yields a p-value > 0.05, indicating no statistically significant difference.

6. The main conclusion (Section 4.2) stating "Yes! WE ARE ONE STEP CLOSER TO AN AI DOCTOR" is not sufficiently supported by the evidence presented. The paper does not demonstrate that higher performance on these benchmarks correlates with medical proficiency. Modern models of clinical reasoning emphasize that medical decision-making does not rely solely on internal cognition [2]. Even with proper evaluations of internal cognitive processes, additional assessments addressing the various cognitive processes involved in clinical reasoning would be necessary to support such a conclusion.

## Case study

7. The case study presented in Figure 6 is incomplete and potentially misleading. The NEJM question referenced was originally an image challenge [3]. It appears that the models were not provided with the necessary images, as evidenced by GPT-4's response mentioning "radiographic findings without seeing the images." This suggests that 'o1' may have hallucinated the radiographic findings ("isolated ulnar fracture") and arrived at the correct answer for incorrect reasons.

## References

[1] Nori, H., King, N., McKinney, S.M., Carignan, D., & Horvitz, E. (2023). Capabilities of GPT-4 on Medical Challenge Problems. ArXiv, abs/2303.13375.

[2] Parsons, A. S., Wijesekera, T. P., Olson, A. P. J., Torre, D., Durning, S. J., & Daniel, M. (2024). Beyond thinking fast and slow: Implications of a transtheoretical model of clinical reasoning and error on teaching, assessment, and research. Medical Teacher, 1–12. https://doi.org/10.1080/0142159X.2024.2359963

[3] NEJM Challenge 2023/04/20, https://www.nejm.org/image-challenge?ci=20230420