Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving
Authors: Yuxuan Zhou
Xien Liu
Chenwei Yan
Chen Ning
Xiao Zhang
Boxun Li
Xiangling Fu
Shijin Wang
Guoping Hu
Yu Wang
Ji Wu
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model
Medical Evaluation
Bloom's Taxonomy
Primary area: deep_learning->large_language_models
TL;DR: We construct a multi-cognitive-level medical evaluation framework for LLMs, and conduct a systematic evaluation across several well-known LLM families.

Abstract: Large language models (LLMs) have demonstrated remarkable performance on various medical benchmarks, but their capabilities across different cognitive levels remain underexplored. Inspired by Bloom's Taxonomy, we propose a multi-cognitive-level evaluation framework for assessing LLMs in the medical domain in this study. The framework integrates existing medical datasets and introduces tasks targeting three cognitive levels: preliminary knowledge grasp, comprehensive knowledge application, and scenario-based problem solving. Using this framework, we systematically evaluate state-of-the-art general and medical LLMs from six prominent families: Llama, Qwen, Gemma, Phi, GPT, and DeepSeek. Our findings reveal a significant performance decline as cognitive complexity increases across evaluated models, with model size playing a more critical role in performance at higher cognitive levels. Our study highlights the need to enhance LLMs' medical capabilities at higher cognitive levels and provides insights for developing LLMs suited to real-world medical applications.

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Basee Problem Solving
Yuxuan Zhou'
Xien Liu
Chenwei Yan
Chen Ning' Xiao Zhang' ! Boxun Li Xiangling Fu Shijin Wang* +
Guoping Hu"
Yu Wang' 1 Ji Wu 156 1
Abstract
2023a;b; Nori et al., 2023a; Qiu et al., 2024), demonstrating their promise in real-world medical applications. However, studies (Hager et al., 2024; Yan et al., 2025) demonstrate that these models still face challenges in handling tasks that more closely resemble real-world medical scenarios, such as clinical diagnosis and treatment. This disparity raises critical question: how far are current LLMs from being truly applicable in real-world medical scenarios?
Large language models (LLMs) have demon- strated remarkable performance on various med- ical benchmarks, but their capabilities across different cognitive levels remain underexplored. Inspired by Bloom's Taxonomy, we propose a multi-cognitivv--evel evaluation framework for assessing LLMs in the medical domain in this study. The framework integrates existing medical datasets and introduces tasks targeting three cog- nitive levels: preliminary knowledge grasp, com- prehensive knowledge application, and scenario- based problem solving. Using this framework, we systematically evaluate sfte-of--heear general and medical LLMs from six prominent families: Llama, Qwen, Gemma, Phi, GPT, and DeepSeek. Our findings reveal a significant performance de- cline as cognitive complexity increases across evaluated models, with model size playing a more critical role in performance at higher cognitive levels. Our study highlights the need to enhance LLMs' medical capabilities at higher cognitive levels and provides insights for developing LLMs suited to real-world medical applications.
Instead of directly answering this question, let's first con- sider a related question: what steps are required for a med- ical student to become a qualified physician? ? To become qualified physician, medical student typically under- goes series of training stages, as shown in Figure 1 (a). First, the student acquires basic medical knowledge (e.g.. anatomy, physiology, and pathology) through textbooks and lectures during the study in medical school. Then, the stu- dent learns to apply this knowledge in analyzing clinical cases during their clinical internship. Finally, the student practices diagnosing and treating patients under the guid- ance of experienced physicians during their residency. Such training process is designed to align with the human cog- nitive process, as outlined in Bloom's 's Taxonomy (Anderson & Krathwohl, 2001): first memorizing and understanding knowledge, then applying it comprehensively. and finally using it to plan and solve problems in real-world scenarios.
1. Introduction
Similar to the training process, the evaluation process in the human education system also follows the human cognitive nature. For example, typical exam sheet includes variety of question types (e.g.. multiple-choice short- and long-response questions) that are intentionally designed to assess students' abilities across different cognitive levels. In contrast, though existing medical benchmarks provide valuable insights into LLMs' medical capabilities, they pri- marily evaluate LLMs capabilities at specific cognitive levels: most medical benchmarks (Jin et al., 2021; Pal et al., 2022; Wang et al., 2024; Cai et al., 2024; Qiu et al., 2024) evaluate LLMs' capabilities through question-answeriin answerins (QA) tasks, which mainly focus on assessing LLMs' pre- liminary knowledge grasp; some other benchmarks (Hager et al., 2024; Ouyang et al., 2024) adopt tasks such as clinical diagnosis and treatment to evaluate LLMs' capabilities of
Large language model (LLM) technology has witnessed rapid advancement (Ouyang et al., 2022; Achiam et al., 2023; Anil et al., 2023; Touvron et al., 2023a) and shown po- tential in various fields, including medicine. Recently, start- of-the-art LLMs (e.g., GPT-4) have achieved expert-level performance across medical benchmarks (Singhal et al.,
'Department of Electronic Engineering, Tsinghua University, Beijing, China 'School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China 'Infinigence-Al Beijing, China "iFLYTEK Research, Hefei, China "College of AI, Tsinghua University, Beijing, China BBRist Beijing, China. Correspondence to: Xien Liu <xeliuu mail wwwstssiinnnuu....c
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Evaluating LLMs Across Multi-Cogniiiie Levels: From Medical Knowledge Mastery to Scenario- Based Problem Solving
(a) Cognitive Development of Human Doctors
(b) Cognitive Evaluation of Large Language Models in Medical Tasks
Quns: Cognttivedavee Evalustion
Exiating Evaluations
Scenarin to lasse Probtem
Solve
by Scosario losed Problem Soluing
Pros: Understanding which models perfore well and which not.
Residert Treince
Plan
High-
Compreharaive Application
Analyze
by Comprehersive Krowledge
Ciinical intern
Apply
Applictton
Corn: No detailed Information difierent
Understand
Ly Preliiinaryy Knoweledge Group
Knowledge Grasp
Remember
Clinical Physician Training Process
Cognitive Levels Knowledge
Inspired by Bloom's Taxonomy
LLMs
GPT
Owen
Figure 1. (a): The cognitive development process of human doctors; (b): Comparison of the existing evaluations and our proposed multi-cognitive-lveell evaluation framework (inspired by Bloom's 's Taxonomy) regarding cognitive levels.
scenario-based problem solving.
Based on the proposed framework, we systematically evaluate state-of-the general and medical LLMs across six prominent families.
In this paper, we argue that the evaluation of LLMs should also follow the cognitive development process, i.e., evalu- ating LLMs' medical capabilities across multiple cogmitive levels. To this end, we propose a munti--oonn evaluation framework EMultiCogEvall) to provide com- prehensive evaluation of LLMs' medical capabilities from cognitive perspective. The schema of this framework is depicted in Figure 1 (b). Specifically, we consider three cog- nitive levels corresponding to the training process of human clinicians: preliminary knowledge grasp, comprehensive knowledge application, and scenario-based problem solving. Built on that, we design tasks that target each cognitive level and integrate three existing medical datasets to generate test samples for each task. To make the performance of LLMs comparable across different cognitive levels, we align the medical knowledge coverage across tasks at different levels as much as possible and normalize the performance metrics for each task.
We reveal significant performance decline cog- nitive complexity increases across evaluated models, offering insights for developing LLMs suited to real- world medical applications.
2. Related Work
2.1. LLM Medical Evaluation Benchmarks
Recently, several medical evaluation benchmarks have been proposed to assess LLMs' medical capabilities. Most of existing medical benchmarks typically utilize the Question- answering (QA) form, where the questions are sourced from medical exams (Jin et al., 2021; Pal et al., 2022; Cai et al., 2024; Qiu et al., 2024), medical literature (Jin et al., 2019), 1),, and consumer health questions (Ben Abacha et al., 2017: Singhal et al., 2023a). Recent studies (Nori et al., 2023a;b; Singhal et al., 2023a) indicate that several LLMs perform no- tably these henchmarks. For example, GPT-4 achieves an accuracy of 90.2 (with complex chain-of-thoughts prompt- ing strategy) on the medical exam benchmark MedQA, ap- proaching expert-level performance. Other benchmarks, such as MIMIC-IV-Ext (Hager et al., 2024) and CLIMED- Bench (Ouyang et al., 2024), adopt scenario-based tasks such as clinical diagnosis evaluate LLMs' capabilities in solving medical problems in real-world scenarios. However, these benchmarks primarily focus on evaluating LLMs at specific cognitive levels and lack a holistic view of LLMs' medical capabilities across multiple cognitive levels.
Using this framework, we systematically evaluate existing general and medical LLMs across 2B 70B parameters from six prominent families: I Llama, Qwen, Gemma, Phi, GPT, and DeepSeek. Our findings reveal that while cur- rent SOTA LLMs generally perform well on the prelim- inary knowledge grasp level, their performance declines significantly as the cognitive level increases. Moreover, we find that model size plays more crucial role in perfor- mance higher cognitive levels. Our study provides clear landscape of LLMs' medical capabilities across different cognitive levels and highlights the need to enhance LLMs' medical capabilities at higher cognitive levels. The codes and datasets are available at https: / thub. t Our contributions are:
THUMLP
2.2. Bloom's Taxonomy
We propose a novel evaluation framework for assessing LLMs' medical capabilities across multiple cognitive levels inspired by the human cognitive process.
Bloom's 's Taxonomy (Bloom et al., 1956) is a widely used framework for categorizing learning objectives. For learn- ing objectives in the cognitive domain, Bloom's Taxonomy
2

---

Evaluating LLMS Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Sienario-Basci Problem Solving
initially proposed six levels: Knowledge, Comprehension, Application, Analysis, Synthesis, and Evaluation. In 2001, revision of Bloom's Taxonomy (Anderson & Krathwohl, 2001) was proposed. where the six levels were renamed and reordered as follows: Remembering, Understanding, Applying, Analyzing. Evaluating, and Creating. Inspired by the Revised Bloom's Taxonomy and the training process of human clinicians, this paper introduces a multi-cognitive- level evaluation framework to assess whether L.MMs have achieved the learning objectives in the medical domain across varying cognitive levels.
medical capabilities by aligning with the training process of human clinicians and referring Bloom's Taxonomy. The three cognitive levels are as follows!:
Preliminary Knowledge Grasp (L1): The first level corresponds the education of medical students in medical school, where they learn and grasp basic med- ical knowledge through textbooks and lectures. At this level, we focus on evaluating LLMs' capabilities in memorizing and initially understanding medical knowl- edge that is essential for medical practice.
Comprehensive Knowledge Application (L2): The second level relates to the clinical internship stage, where students learn to apply basic medical knowledge to handle simple clinical cases. This level aims to evalu- ate LLMs' capabilities in applying medical knowledge to analyze and solve complex medical problems.
3. Methodology
3.1. Multi--Conittive-Leeeell Evaluation Priciples
We first illustrate the idea of the proposed multi-cognitive- level evaluation. Given a large language model M and series of cognitive levels L, L.,..., In,, the goal of the multi-cognitive-lewe evaluation is to assess the capabilities of M at each cognitive level:
Scenario-based Problem Solving (Lz): This level cor- responds to the residency training stage, where students practice advanced clinical skills (e.g., diagnosis, treat- ment) under the guidance of experienced physicians. At this level, we aim to evaluate LLMs' capabilities in planning and solving problems in real-world medical scenarios using medical knowledge.
f(M)= = faM), (M))
(L)
where f refers to the whole multi-cognin cognitive-leve evalua- tion process, S/ represents the evaluation process at the i-th cognitive level, and fiM denotes the corresponding eval- uation results.
3.3. Julti-Cognitivv--Leeel Evaluation Framework
A qualified multi-cognitive-level evaluation should also ad- here to the following principles:
Based on the defined cognitive levels, we develop multi- cognitive- evaluation framework that integrates existing medical datasets and designs tasks tailored to each cognitive level. The overview of the proposed framework is illustrated in Figure 2. In the following sections, we will introduce the detailed design of the evaluation tasks, dataset construction, and performance metrics that follow the multi-cognitive- level evaluation principles proposed in Section 3.1.
Task Relevance: The evaluation tasks should target different cognitive levels, aligning with the knowledge- learning process.
Knowledge Consistency: In order to isolate the effects of cognitive levels, it is important to ensure that the evaluation tasks have consistent knowledge coverage across different cognitive levels, minimizing the influ- ence of knowledge coverage on the evaluation results.
Task Design For Low-Level tasks (f1), we adopt multiple- choice questions (MCQs) as simple knowledge recalling tasks for Low-Level, where the model is required to recog- nize the correct answer from multiple candidate options (see the left part of Figure 2). Such task format commonly used in existing medical benchmarks and human examina- tions to evaluate students' basic knowledge understanding.
Metric Alignment:
The performance metrics should be aligned (normalized) across different cognitive lev- els to ensure the comparability of the evaluation results.
In our study, we adopt these principles to develop a multi- cognitive-level evaluation framework for assessing LLMs' medical capabilities. Though we primarily focus on the med- ical domain, the proposed multi multt--connitiv--eveel evaluation principles can also be generalized to other domains.
For Mid- tasks (f2), given that existing medical bench- marks fail to adequately address this cognitive level-either being too simplistic, as in QA-based benchmarks, or overly complex, as in scenario-basee benchmarks we reformu- late the original MCQs into set of "Complex Knowledge Application" tasks for this level. These tasks are designed to assess the cognitive skills required for real-world medical
3.2. Cognitive Levels in Medical Domain
Following the multi--oonnitiv--leeell evaluation principles, we first define three cognitive levels for assessing L.MMs'
In the following sections, we refer to these levels Low-Level, Mid-Level, and High-Level for simplicity.
3

---

Evaluating LLMs Across Multi-Cogniting Levels: From Medical Knowledge Mastery to Scenario-Bass Problem Solving
Simple Knowledge Recalling Tasks
Complex Knowledge Application Tasks
Full-ppath Clinical Diagnosis Tasks
Reducing
Statement:
'Anti-HBsAg lgG only elevated during scule Hepaliis infection."
yo w hopothyrodem and muliipe ED admissions for sale bad requests due to cio skin and eye yallowing.....
Question:
a pationt with Hopatitis infection. which one of the following markers only elevated during acule infection?
Additional Clues
True
False
Increasing
Q1: Alice choose C, is her answer correct?
Incorrect.
Pationt history suggesi potantial hepatoblllary related diseases. Requie laboratory test of
Reasoning Steps
Q2: (Tincorrec, what the correct answer?
Scenarios
A2: D. Anti HbcAg lgM.
Options:
Hepallii Virus IgM Antbody Nep Hepalllie Core Antibody ight Pos"
A. HBskg
Options: A: Hbssg B: Hbccg Antr
D. Anti-HBcAg igM
B. HBckg
Anti-HBeAg IgG
Decision Space
Expanding
Hbssg lgG Anti-HBcAg lgG
the correct answer provided the options above? Yes
Based on ... diagnosis should be: Acute hepatitie B.
No
Preliminary Knowledge Grasp
Comprehensive Knowledge Application
Scenarlo-based Problem Solving
Figure 2. An overview of the proposed multi culti-coognttive-evwe medical evaluation framework.
tasks from different perspectives without relying on specific scenarios, thereby effectively evaluating the model's ability in the Mid-Level. Specifically, we consider the following task types (see the middle part of Figure 2):
XN Steps
MOM
Consult
Diagnose
Present limasce
History of
Examination Selection
Results Analysis
Final Diegnosis
Statement Validation Questions: We transform the question and one of the candidate options from the original MCQs into statement, requiring the model to determine whether the statement is true or false. While this task may intuitively seem simpler than MCQs, the answer choices in MCQs can provide additional clues, allowing the model to choose the most plausible an- swer without fully understanding why it correct. In contrast, statement validation emphasizes the model's ability to precisely discriminate knowledge, closely mirroring wealdworld clinical scenarios where options are typically not provided. Furthermore, given that some MCQ distractors significantly increase task diffi- culty, we construct two statements per MCQ: one re- flecting the correct answer and the other derived from distractor, maintaining their difficulty.
Siaple Diagnoste
Physical Laboratory
Microbioiogy
Imaging Exam
Pulk-athh Blagnoot
miniinting the meakaroti clegnostis
Figure 3. An overview of the full- full-path clinical diagnosis task ap- plied in the proposed evaluation framework.
(e.g.. predefined answer choices), whereas real-world tasks involve significantly broader decision spaces. For instance, there may be hundreds of potential candidate diseases in clinical diagnosis, requiring more complex decision-making to reach the correct diagnosis. To sim- ulate this, we design task where the model is asked to determine whether the correct answer exists in a set of candidate options. For each original MCQ, we gener- ate pair of questions with opposite labels to eliminate the influence of random guessing: one question retains the original options, while the correct option in the other question replaced with a distractor.
Multi-Step Rectification Questions: Handling QA tasks typically requires single decision- decision-making step, whereas problem-solving in real-world scenarios often involves multiple steps. For instance, when managing patient's care, a doctor first establishes a diagnosis and then prescribes treatment based on the diagnosis made in the previous step. Inspired by this, we reformulate the original MCQ into a two-step task: we first ask the model to verify whether provided option is the correct answer of the MCQ, and then, if the answer is incorrect, we ask the model to give the correct answer from the rest of options. We generate two questions for each MCQ: one provides the correct answer, while the other provides a randomly sampled wrong option.
For High-Level tasks (fz), we follow Hager et al. (2024) and assess LLMs' capabilities of solving scenario-based problems using full full-path clinical diagnosis task, as illus- trated in Figure 3. Compared the tasks in the Low-Level and Mid-Level, this task simulates real-world diagnostic process by requiring the model to sequentially order exami- nations and integrate information from the obtained exam- ination results to arrive at a final diagnosis. Solving this task requires models to perform multi-step makisinnmmakinggg within larger decision space, mirroring the complexity of
Answer Existence Judgment Questions: QA tasks typically operate within a constrained decision space
4

---

Evaluating LLMs Across Multi-Cogniiie Levels: From Medical Knowledge Mastery to Srenario-Based Problem Solving
Recognize & Retrieve Procedure for High-leval Dataset Construction
Table 1. The basic statisties of the constructed multi-cognitive
level medical evaluation benchmark.
EHR Datlatosse
Boanaric-besed Disgnosis Tasks
Levels
Tasks
Sources
# Samples
Previous Levels
NER
Retrieve
Filter
Gensrate
Full--path
Low
MCQs
MedQA
795
MedMCQA
2,816
Reformulated
MedQA
4,770
Admissioo Records
High-Leval Evaluation
Middle
Tasks
MedMCQA
16,896
Scenario- based
High
MIMIC-IV
2,176
Figure 4. The recognize- aengnize-and--aee metrretrieve procedure for constructing the Hige-Level dataset, ensuring alignment with previous levels in terms of knowledge coverage.
Diagnosis
diagnosis task. Table summarizes the key statistics of the constructed nulti-cognitive-leve evaluation benchmark, while additional details, including task examples, prompts, and answer parsing, provided in Appendix A.
real-world medical scenarios.
Dataset Construction For Low-Level tasks, we directly use the questions from the MedQA (Jin et al., 2021) and MedMCQA (Pal et al., 2022) datasets, which cover wide range of medical knowledge and are widely used to evaluate LLMs' medical capabilities. For Mid-Level tasks, as intro- duced above, we reformulate each original MCQ into two statement validation questions, two multi-step rectification questions, and two answer existence judgment questions. We apply the advanced GPT-4o to assist in data generation. Specifically, GPT-4o employed to (1) generate statements based on the questions and the provided options for the statement validation tasks and (2) generate the distractors for the answer existence judgment tasks based on the origi- nal question and the correct answer. We arrange for three doctors with over three years of experience to verify the statements and distractors generated by GPT-4o and find that the error rate of GPT-4o is less than 5%.
Performance Metrics We adopt the accuracy for the Low- Level and Mid-Level tasks. For the High-Level task, we propose full-path diagnosis accuracy, new metric that assesses LLMs' diagnostic performance by considering both procedural correctness and outcome accuracy:
T(Diag (
Diag (0) x Recall
Accuracy proc
exam
(2)
(1) and Diag denote the predicted and ground- truth diagnoses of the i-th admission record, respectively. I(-) is the indicator function, and N is the number of admis- sion records. Recall exam is the examination recall of the i-th admission record, which is calculated as:
where Diag
For High-Level tasks, we leverage the MIMIC-IV dataset (Johnson et al., 2023), which contains detailed elec- tronic health records (EHRs) of patients, including the his- tory of present illness, physical examination, laboratory tests, microbiology tests, and imaging tests. We employ a recognize-and-reericee procedure to construct the dataset, ensuring its knowledge coverage remains consistent with previous levels (see Figure 4). Specifically, medical NER tool MedCAT (Kraljevic et al., 2021) to iden- tify diseases in previous-level tasks and construct disease pool. We then filter the admission records associated with these diseases using ICD-10-CM codes, which are widely employed for disease classification. To ensure alignment between primary diagnoses and the disease pool, we extract admission records where the primary discharge diagnosis matches disease in the pool. Records with more than two diseases in the discharge diagnosis are excluded to maintain a one-to-one mapping between records and diseases. After processing, 2,176 admission records remain, covering 42 diseases across eight human body systems. Finally, we con- vert these records into test samples for the full full-path clinical
Recall exam
# Correct Ordered Exam Items Total Exam Items in the Record
In this study, we prioritize examination recall, as missing critical examination information can have far more severe consequences than ordering non-essential tests during diag- nosis. To ensure fairness, we compute the macro-average of both examination recall (across examination types) and diagnosis accuracy (across diseases), assuming equal impor- tance for each examination type and disease in this task.
first use the
Considering that the random guessing performance may vary across different tasks, we also calculate normalized accuracy of each task by reducing the effect of random guessing to ensure the comparability of the evaluation re- sults across different cognitive levels:
Accuracy model Accuracy rand
Accuracy norm
(4)
Accuracy rand
More details about the metries and normalization process are provided in Appendix B.
5

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario- Based Problem Solving
Table Performance (mean and standard of the normalized accuracy (%) M. different cognitive levels evaluated the proposed benchmark. Llamal's performance on the High-Leve task is unavailable due to the absence of instruction tuned versions. Asterisks: due to high cost GPT-4o and DeepSeek- API, we evaluated on ~00% the original dataset.
Low-Leve
Mid-Level
High-Leww MIMIC-IV
Model
MedQA
MedMCQA
MedQA
MedMCQA
Llama- 7B
3.36 = 0.82
10.40 + 0.24
2,47 + 0.76
4.38 = 0.62
Llama- 13B
13.18 0.57
18.98 + 0.22
2.62 - 0.65
3.15 0.71
Llama- B
24.09 0.26
26.88
12.85
10.44 0.19
Llama-655
26.64 = 0.49
29.59 + 0.29
16.82 0.73
13.48 0.44
Llama2-7B
15.88 = 0.28
18.32 + 0.22
4.21 1.01
5.40 = 0.55
2.65 + 0.20
Llama2-133 13B
21.10 = 0.30
20.94 + 0.33
6.18 + 1.25
7
4.92 + 0.15
Llama2-70B
37.99 0.34
35.60 1 0.26
20.92 0.80
1
5.16 0.18
Llama3-8B
37.30 0.53
39.11
11.47 0.43
11.40 1 0.28
10.50 0.16
Llama3-70B
63.68 1 0.34
60.82 1 0.28
41.79 0.85
36,25 1 0.46
17.81 0.41
Qwen-77 7B
19,40 = 0.31
24.85 1 0.21
7.34 1 0.39
7.08 1 0.32
2.31 0.31
Qwen- 14B
38.05 0.38
39.25 + 0.13
17.98 +: 0.63
15.98 + 0.32
4.14 +: 0.29
Qwen-72B
50.60 = 0.39
50.95 +
28.39 0.70
26.28 + 0.31
5.89 0.35
Qwen2-
36.73 = 0.44
39.85 = 0.15
21.22 0.51
20,93 = 0.18
6.99 + 0.25
Qwen2-72h
65.91 = 0.24
60.41 + 0.19
47.32 + 0.43
37.19 0.18
15.10 1.10:: 0.09
Qwen2.5-7B
42.86 0.49
45.39 0.17
27.01 0.21
24.75
9.50 0.04
Qwen2.5-14B
52.23 0.30
51.74
35.62 0.37
30.73 0.31
12.07 + 0.21
Qwen2.5-322
59.21 + 0.21
57.17 0.09
38.45 0.56
3 0.24
11.87 0.22
Qwen2.5-72B
67.83 + 0.20
61.82 +
49.47 + 0.69
40.57 + 0.39
16.05 + 0.24
Gemma-22 2B
6.16 = 0.63
15.04 0.24
0.52 1 0.31
2.82:0.33
0.59 + 0.09
Gemma-7B
30.35 0.42
27.41 0.38
15.09 0.48
13.64
0.33 0.09
Gemma2-28
14.87 0.34
24.18 + 0.31
1.78 0.71
3.85 +.55
5.02 0.33
Gemma2-9B
44.21 = 0.18
46.20 + 0.20
0.20
24.52 + 0.28
23.05 0.24
9.04 + 0.28
Gemma2-27B
53.65 1 0.31
50.45 1 0.18
33.40 1 0.88
30.33 0.26
11.19 1. 0.64
Phi3-3.8B
37.52 0.28
40.96 0.20
23.10 1.15
22.53 2
2.21 0.28
Phi3-7B
49.25 = 0.30
46.95 + 0.30
28.87 + 0.31
21.75
7.74 0.16
Phi3-14B
54.06 + 0.49
49.70 1 0.27
31.67 0.55
28.46 1 0.24
3.89 + 0.01
Phi4-14B
56.19 + 0.40
52.62 + 0.30
39.30 + 0.42
33.46 0.23 0.23
12.18 0.37
GPT--4omiini
62.67 0.20
55.33 0.06
42.98 0.59
32.28 0.66
13.81 0.84
GPT-4o*
78.33 = 0.75
64.00 + 1.02
56.96 1.84
41.65 = 1.59
19.33 0.30
DeepSeek-V3.
78.33 = 0.75
65.78 + 0.22
44.79 + 0.30
40.07 = 1.49
19.42 = 0.17
4. Experiments
diagnosis), it involves multi-turn interaction, evalu- ate the instruction fine-tuned versions of the corresponding models and leverage the zero-shot learning setting to eval- uate the models' performance. Specifically, for each test sample, we provide the model with the instructions for the task and ask it to order the specific exam items sequentially and make the final diagnosis. We utilize the UMLS Metathe- saurus (Bodenreider, 2004) along with handcrafted syn- onym set identify the model's predictions for ordered exam items and the final diagnosis (the detection rate across all categories reached ~95%) We conduct repeated eval- uations 3-5 times and report the average performance and standard error. More details about the evaluation setting (e.g.. hyperparameters) are provided Appendix D.
4.1. Experimental Setup
conduct systematic evaluation on current LLMs, we select total of 32 general LLMs across six LLM families, including Llama (Touvron et al., 2023a;b; Dubey al., 2024), Qwen (Bai al., 2023; Yang et al., 2024; Hui et al., 2024), Gemma (Mesnard et al., 2024; Team et al., 2024), Phi (Abdin et al., 2024b;a), GPT-4o (OpenAI, 2024), and DeepSeek (Liu et al., 2024). We also evalu- ate 8 medical LLMs from series: ClinicalCamel (Toma al., 2023), Med42 (v1, v2) (Christophe et al., 2024a;b), Meditron (Chen et al., 2023), and MMMe-LLlama (Qiu al., 2024). The detailed statistics provided in Appendix C.
Evaluated Models
4.2. Cognitive-Level Analysis
Evaluation Setting For tasks in the Low-Level and Mid- Level, we leverage the five-shot in-context learning (Brown et al., 2020) for evaluation, where five input-output pairs are sampled as the demonstrative examples to guide the model to generate the correct answer for the test sample (see Figure in Appendix A). For the High-Level task (full-path
Higher the Cognitive Level, Worse the Model Perfor- mance We first compare the performance of LLMs across different cognitive levels on the proposed benchmark and present the results in Table 2. For the Mid-Level, we use the
6

---

Evaluating LLMs Across Multi-Cognitie Levels: From Medical Knowledge Mastery to Scenario-Base Based Problem Solving
100
Log Normalizer lizealizeddd Accuracy
0.15
& Accuracy v.s. Backbone
0.10
0.05
10-1
-0.05
-0.10
Low-Level
Mid-Level
High-Level
Low-Level
Mid-Level
High-Level
Llama2
Qwenl
Qwen2.5
ClinicalCamel-70B
Med42-v2-708
Llama3
Qwen2
GPT
Med42-70B
Meditron3-88
78/GPT-4o-mini
70B/GPT-40
Meditron-708
Meditron3-700
Med42-v2-88
MMed-Llama-3-88
Figure 5. Performance of I.LMs across different parameters sizes on the proposed benchmark. The performance Low Mid- Level tasks is the macro average across MedQA and MedMCQA.
Figure 6. Performance gains of medical LLMss over their backbone models across different cognitive levels.
average performance of the three task types to compare with the other levels (Detailed performance is provided in Table 8 of Appendix E). We observe that several state-of-the- art LLMs, such GPT-4o, DeepSeck-Ve V3, and Llama3-70B, achieve remarkable performance (>x (0%% on the Low-Level (preliminary knowledge grasp) tasks. However, all of these models experience a significant performance drop (~20%) when evaluated on the Mid-Level tasks (comprehensive knowledge application). Moreover, on High-Level tasks (scenario-based problem solving), the performance of all models further declines, with the best-performing model, DeepSeek-V3. V3, achieving only 19.4 full-path diagnosis ac- curacy. This indicates that while current LLMs grasp basic medical knowledge well, they still face significant chal- lenges at higher cognitive levels, particularly when solving complex problems weal-world medical scenarios.
Table 3. Performance (%) LLMs fine-tuned with/without inference-tine scaling. The performance of Low and Mid-Level tasks the macro average across MedQA and MedMCQA.
Model
Low-Level
Mid-Level
High-Level
DeepSeck-V. V3
72.1
42.4
19.4
DeepSeek-RI
81.9
65.5
26.5
GPT-4o-mimi
59.0
37.7
13.8
03-mini
81.3
64.5
15.1
Medical-Domain Finetuning Primarily Benefits Low- and Mid-Level Tasks To investigate the impact of medical-domain finetuning on LLMs across different cogni- tive levels, compare the performance of medical LLMs with their backbone models. As shown in Figure 6, these specialized models generally outperform their backbone counterparts on Low- and Mid-Level tasks, with perfor- mance gains reaching up to 15%. However, they fail to achieve significant improvements on High-Level tasks and even underperform their backbone models. This phe- nomenon indicates that while medical-domain finetuning strengthens LLMs' grasp of basic medical knowledge and its comprehensive application, it remains less effective in addressing complex real-world medical scenarios.
Parameter Sizes Matter, Especially at Higher Cogni- tive Levels We further investigate the impact of model parameter sizes across different cognitive levels. Specifi- cally, we examine 7B and 70B models from the Llama and Qwen families, along with GPT-4o and GPT-4o-mini2. As shown in Figure 5, larger models consistently outperform their smaller counterparts within the same model series. Additionally, the performance gap between model sizes in- creases significantly from the Low-Level to the Mid-Level on logarithmic scale, suggesting that model size plays a more critical role enabling LLMs to tackle complex problem-solvin tasks. However, this gap narrows slightly the High-Level, implying that even the 70B-level models struggle to effectively solve scenario-based problems.
Inference-time Scaling Works Well, Especially for Higher Levels We further investigate the effectiveness of inference-time scaling on LLMs' medical capabilities across different cognitive levels. We select two represen- tative models with varied parameter scales (DeepSeek-V) and GPT-4o-mini and their corresponding inference-time scaling versions (DeepSeek-R) and o3-mini) for study. As presented in Table 3, the inference-time scaling models con- sistently outperform their backbone models across all cog-
DeepSeek-VV V3 not involved in this analysis since it does have a smaller version for comparison.

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario- Based Problem Solving
70B Model
7B Model
Diagnosis Accuracy
0.4
0.3
0.2
0.1
0.0
Adhesive caprulets
Aspendictis
Peivie infiammatory disedra D Deficiancy oremia
Chuecystai
Oriecorteena
Hyporatcemua Nomaa preseure hydrocaphi trocappttus Heart falire
Pancrestels
Hurpa conter
aattable arriina
Anno Ra pentaaionn
hemortiig
Hupotito Hyppertent Penntt rrtt
Hypathyree prriaurs
Lime dieats
Sepiis
Homeuria
Eosinopaai
Hemebyik premia
Carpa tunnel
Myatthensa
Necrotzznn fauciitss
Ventncuutr pehycartla
Acthma
Pertcardee
hypdemseen
prosuaitis
Bacteremi
Inftablo bowel syndromt
Polyrmyln resuratic Tabattar necrott Sidt aino
Ciasdication
Cardicc comport
Esaphageas
Subaracheond
Infectious move
Orthoraticc
Figure Average diagnosis accuracy LLMs (Llama2 and Qwenl, 2, and 2.5) across diseases in the full-path clinical diagnosis task.
Table Detailed performance (%) of LLMs on tasks in the Mid-
Table S. Detailed performance (%) of LLMs on the full- full-path clini- cal diagnosis task. Exam Recall: the macro average of examination recall across examination types. End-point: the accuracy of the final diagnosis without considering the examination recall.
Level. StmtVal: Statement Validation; AnsExist: Answer Exis-
lence Judgment; MultiRect: Multi-Step Rectification.
Mid-Level AnsExist
Model
Low
Stmt Val
MultiRect
Diagnosis Accuracy
Model
Exam Recall
Llama3-70B
62.3
47.2
9.9
60.0
End-point
Full-path
Qwen2.5-72B
64.8
50.2
23.8
61.0
Llama3-70B
38.8
38.1
18.1
Gemma2-27B
52.1
38.5
10.4
46.7
Qwen2.5-72B
34.4
39.6
15.8
Phi4-14B
54.4
40.1
17.5
51.6
Gemma2-27B
27.5
33.4
10.6
GPT-4o
71.2
58.8
22.4
66.3
Phi4-14B
27.1
39,2
12.2
DeepSeck-VV V3
72.1
53.7
6.3
67.4
GPT-4o
31.8
49.2
19.3
DeepSeek-VV V3
30.0
53.6
19.4
nitive levels, while they achieve more significant improve- ment on Mid-Level tasks compared to Low-Level tasks (e.g., +23.1 vs. +9.8 for DeepSeek-RI), Note that the performance gap narrows on high-level tasks due to their significantly in- creased difficulty. This suggests that inference-time scaling is a promising approach to enhance LLMs' medical capabil- ities, particularly for complex problem-solving tasks.
world medical scenarios.
Low Procedural Correctness Causes Bad Performance in Scenario-based Tasks We further analyze the perfor- mance of large language models (LLMs) on the full-path clinical diagnosis task. As shown in Table 5, LLMs exhibit moderate performance in terms of end-point accuracy (i.e., diagnosis accuracy without considering examination recall), with DeepSeek- V3 achieving the highest accuracy at 53.6%. However, when taking examination recall into consideration (full-patt performance), the accuracy of all models declines significantly, with the best-performing model, DeepSeek- V3, achieving only 19.4% diagnosis accuracy. This decline can primarily be attributed to low examination recall (less than 40%) across models, suggesting that while current LLMs can occasionally reach the correct diagnosis, they fail to fully consider all relevant possibilities and request the necessary information throughout the diagnostic process.
4.3. Fine-Grained Analysis
LLMs Perform Constantly Worse Across Tasks in the Mid-Level We further provide a fine- imn-grrined analysis of LLMs' performance across tasks within the Mid-Level and compare with the Low-Level tasks. We illustrate results of the best-performed model from each LLM family in Ta- ble 4, with the full results presented in Table 8. We observe that LLMs consistently perform worse on all Mid-Level tasks compared to Low-Level tasks. Specifically, the per- formance drop on the answer existence judgment tasks is more significant, where almost all models experience a per- formance decrease of over 40%. This suggests that current LLMs struggle with medical problems involving signifi- cantly larger decision spaces, which is a key requirement in
LLM Performance Across Diseases Presents Long- Long-Tail Distribution Finally, we analyze the diagnosis perfor- mance of LLMSs across diseases the full-path clinical
8

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Base Problem Solving
Low- and Mid-Level tasks but has limited impact on High- Level tasks. This suggests that current medical-specific finetuning strategies are insufficient for enhancing LLMs' reasoning abilities in complex real-world medical scenarios; (3) Inference-time scaling shows promise in boosting L.LMs medical capabilities, especially in tasks requiring complex knowledge application. However, further research is re- quired to enhance LLMs' High-Level capabilities, such as planning, requesting key information, and reasoning based on the acquired information.
Table 6. Clinician validation results on the proposed benchmark. Clinician Acc.: the percentage of correct answers provided by the clinicians. Clinician Subj. Diff.: the subjective difficulty by clinicians on scale of 1-10 (I: very easy, 10: very difficult).
Clinician Subj. Diff.
Level
Clinician Acc. (%)
Mean
Median
Low
68.8
5.0
5.5
Mid
54.2
6.0
5.8
High 23,5
7.5
7.5
Based on these findings, we offer the following insights for applying and developing large language models to address real-world clinical challenges: (1) Model Selection: For low-level medical tasks such as knowledge-basd based question answering, LLMs with around 10B parameters are generally sufficient. However, for more complex clinical tasks such as diagnosis and treatment recommendation L.LMs are necessary; (2) Medical LLM Development: Future medical-domain finetuning efforts should focus more on enhancing LLMs' High-Level cognitive abilities, including clinical planning, proactively acquiring key diagnostic and treatment information, and conducting multi-step reasoning. Inference-tine scaling emerges as promising direction to support these high-level capabilities.
diagnosis task and present the results in Figure 7. We ob- serve that the performance of LLMs varies significantly across different diseases, forming long-tail distribution, while 70B models generally perform better than 7B models across all diseases. This suggests that current LLMS may struggle with diagnosing rare or complex diseases, which are critical in roal-world medical scenarios.
4.4. Clinician Validation
Finally, to ensure that the tasks at different levels of the proposed benchmark genuinely correspond to distinct lev- els of cognitive difficulty, we further conduct a small-scale clinician validation study. Specifically, we randomly select 100 samples from the constructed benchmark and recruit four licensed clincians with 3 to 8 years of experience to assess the benchmark's 's difficulty from two perspectives: (1) their accuracy in answering the questions and (2) their sub- jective difficulty ratings to the questions on a scale from (Easy) to 10 (Hard). The evaluation results are provided in Table We found that clinicians' accuracy also decreases as the cognitive level increases across three levels, indicat- ing that tasks at higher levels are indeed more challenging. Moreover, their subjective difficulty ratings align with this trend, further demonstrating the validity of the proposed benchmark. More details about the clinician validation study, including the data selection and evaluation process, are provided in Appendix F.
To the best of our knowledge, this work is the first to evaluate LLMs' medical capabilities across multiple cognitive levels. While promising, the evaluation may be constrained by the scope of medical knowledge coverage and task diversity. Future research could further explore these areas, expanding the range of medical domains and tasks to offer a more holistic view of LLMs' medical abilities.
A Acknowledgements
This work was supported by the Noncommunicable Chronic Discases-National Science and Technology Major Project (Grant No. 20232D0506501) and Beijing Natural Science Foundation (NO. 4252046). We would like to thank the clin- icians who participated in the validation and the anonymous reviewers for their valuable feedback.
5. Conclusion
In this work, we propose multi-cognittve-leeell evalua- tion framework that assesses LLMs' medical capabilities at three cognitive levels. Using the proposed framework, we construct a new medical benchmark and systematically evaluate existing LLMs on the benchmark. Our study leads to the following key findings: (I) While the performance of smaller LLMs (~10B) gradually approaches that of larger LLMS 70B) on Low-Level tasks, the performance gap remains significant on Mid-Level and High-Level tasks, in- dicating that model size plays a crucial role in enabling LLMs to tackle complex medical problems; (2) Medical- domain finetuning significantly improves performance on
Impact Statement
This work mainly explores a multi-coognttiv--leeel evalua- tion framework for assessing L.Ms'' medical capabilities. The results indicate that current LLMs generally perform well on the preliminary knowledge grasp but face challenges at higher cognitive levels, providing insights for develop- ing LLMs suited to real-world medical applications. The datasets and codes used in this study are built based on pub- lic medical datasets and are publicly available to facilitate further research. We have not identified any potential nega- tive ethical consequences requiring further consideration.

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Bass Based Problem Solving
References
M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-1 2020. virtual, 2020.
Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gu- nasekar, S., Harrison, M.. Hewett, R. J., Javaheripi, M., Kauffmaan, P., et al. Phi-4 technical report. ArXiv preprint, abs/2412.089000 arxiv. org/ abs / 24/2/4122222 08905.
2024a. URL https://
Cai, Y., Wang, L., Wang, Y. de Melo, G., Zhang, Y., Wang, Y., and He, L. Medbench: A large-scale chi- nese benchmark for evaluating medical large language models. In Wooldridge, M. J., Dy, J. G., and Natara- jan, S. (eds.), Thirty-Eighth AAAI Conference on Artifi- cial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, FAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 17709-17717 AAAI Press,
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A. Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. ArXiv preprint, 142199 2024b. URL httptttsss//axxi.... /ttpxv..ivv///////////..//awaa org/ abs/2404, 14219.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. I.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. ArXiv preprint,
2024. doi: 10. (0.1699/AAAAA1338166222227 URL https: / //doi.orr//// 100g///1.000 1 wtt.:aaa.vv.i.1.........
abs/2303.08774, 2023. URL https: /ttp/a.axvvv.o/////ee.... 07774.
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Ma- toba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. Meditron-70b: Scaling medical pretraining for large language models. ArXiv preprint, abs/2311.16079, 2023. URL. httpst/t: /ttp:://rxxvvvrooe abs/2311.16077
Anderson, L. W. and Krathwohl, D. R. A laxonomy for learning, teaching, and assessing: revision of Bloom's ss taxonomy of educational objectives: complete edition. Addison Wesley Longman, Inc., 2001.
Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: family of highly capable multimodal models. ArXiv preprint, abs/2312. 1805, 2023. URL https: /wt.xviivv...////////aa///aaavva wwgg/oggg//... /tt.///....11111
Christophe, C., Kanithi, P., Munjal, P., Raha, T., Hayat, N., Rajan, R., Al Mahrooqi, A., Gupta, A., Salman, M. U., Pimentel, M. A., et al. Med42-evaluating fine-tuning strategies for medical llms: Full-parameter vs. parameter- efficient approaches. In AAAI 2024 Spring Symposium on Clinical Foundation Models, 2024a.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang. F., et al. Qwen technical report. ArXiv preprint, abs/2309. 16609, 2023. URL https: /atp://aaxvv..eii wwaz/iiv.g///...... wtwp//gg/////oooooeooooooeeeeere / 2209. 10.99.660000 16609
Christophe, C., Kanithi, P. K., Raha, T., Khan, S., and Pi- mentel, M. A. Med42-v2: A suite of clinical llms, 2024b. URL https::tp/// aaaairiiivvv////////aaaaa//aaavv orgg hatp//bb/24000
06142.
Ben Abacha, A., Agichtein, E., Pinter, Y., and Demner- Fushman, D. Overview of the medical question answering task at trec 2017 liveqa. In TREC 2017, 2017.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. ArXiv preprint, abs/2407.21783, 2024. URL https: http///axxvvv..// org/ abs 2407
Bloom, B. S., Englehart, M. D.. Furst, E. J., Hill, W. H., Krathwohl, D. R., et al. Taxonomy of educational objec- tives, handbook i: the cognitive domain. new york: David mckay co, 1956.
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. Evaluation and mitigation of the lim- itations of large language models in clinical decision- making. Nature medicine, 30(9):2613 222226223222222 2024.
Bodenreider, O.
The unified medical language system (umls): integrating biomedical terminoloyy. Nucleic acids research, 32(suppl.1):D267-12222 2004.
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., et al. Qwen2. 5-coder technical report. ArXiv preprint, abs/2409. 12186, 2024. URL httptttss///ae /ttppps:////avvi./e/ org/ abs 249.. 12186.
Brown, T. B., Mann. B., Ryder, N., Subbiah, M.. Kaplan, J. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato,
Jin, D., Pan, E.,, Oufattole, N., Weng, W.-H., Fang, H., and Szolovits, What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 1 11(14):6421, 2021.
10

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Base Problem Solving
Jin, Q., Dhingra, B., Liu, Z.., Cohen, W., and Lu, X. Pub- MedQA: A dataset for biomedical research question answering. In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Prucessing (EMNLP-IJCNLP), pp. 2567-2577,
Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Ouyang, Z., Qiu, Y., Wang, L., De Melo, G., Zhang, Y., Wang, Y., and He, L. Climedbench: large-scale chi- nese benchmark for evaluating medical large language models in clinical scenarios. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 1428-8435 2024.
Hong Kong, China, 2019. Association for Computa- tional Linguistics. doi: 10.18653/w1/119-13355 URL ht https: //////aaaatahhot hww.aanthoooooyoooooooo (tt.looyyyrr//19-1133330
Pal, A., Umapathi, L. K., and Sankarasubbu, M. Medm- cqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Flores, G., Chen, G. H., Pollard, T., Ho, J. C., and Naumann, T. (eds.), Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pp. 248-260. PMLR, 2022. URL URL https: /ttptttttt/////eeeeaaeeie /ttpr/eeeeeeeng..eeeeeeepee/..se mlr press/ v1 14 / pa 22a htmml.
Johnson, A. E., Bulgarelli, L., Shen, L., Gayles, A., Sham- mout, A., Horng S., Pollard, T. J., Hao, S., Moody, B., Gow, B., al. Mimic-iv, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
Kraljevic, Z., Searle, T., Shek, A., Roguski, L., Noor, K., Bean, D., Mascio, A., Zhu, L.,, Folarin, A. A., Roberts, A., al. Multi-domain clinical natural language process- ing with medcat: the medical concept annotation toolkit. Artificial intelligence in medicine, 117:102083, 2021.
Qiu, P., Wu, C., Zhang, X., Lin, W., Wang, H., Zhang, Y., Wang, Y., and Xie, W. Towards building multilingual language model for medicine. Nature Communications, 15(1)88384, 2024.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C.,, Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S.,
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, W., Scales, N., Tanwani, A., Cewe-Lewis, H., Pfohl, S., Large language models encode elinical knowledge. Nature, 620(7972) 2023a.
Pathak, S., Sifre, L.., Rívière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. ArXiv preprint, abs/2403.08295, 2024. URL https: http://axxvvvv..
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulezyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. Towards expert-level medical question an- swering with large language models. ArXiv preprint, abs/2305.09617, 2023b.
Nori, H., King, N., McKinney, S. M., Carignan, D., and Horvitz, E. Capabilities of gpt-4 on medical challenge problems. ArXiv preprint, abs/2303. 13375, 2023a. URL https: / /arxivivvv/////////v//a//////aia org/ hblgg/bbb abs 22//2333...1 13375.
Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al. Can generalist foundation models outcompete special- purpose tuning? case study in medicine. ArXiv preprint, abs/2311. 16452, 2023b. htw.httss///rxii.ortr
Team, G., Riviere, M.. Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. ArXiv preprint, abs/2408.001 18, 2024. http::/:://axii abs/2408.00118
OpenAl. Gpt-40 mini: advancing cost-efficient intelli- gence, 2024. URL htttss: com index/ hello-gpt-40.
Toma, A.. Lawler, P. R., Ba, J., Krishnan, R. G., Rubin, B. B., and Wang, B. Clinical camel: An open expert-level medical language model with dialogue-based knowledge encoding. ArXiv preprint, abs/2305. 1031,, 2023. URL https: /tt///aaxivi wwtp://///.pbbb....aba/ 12031.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. ArXiv preprint, ab22000..200000.1191..22222 2023a. URL wwt.tttpps//// / /aww/vi.vvv/../////v/////////.a. wtw.vv//o/b/////...155.

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Base Based Problem Solving
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A.,
Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.09288. 2023b. URL https:ttp///o / /ta.iaivv.... wtt.://///../.....//// 09288.
Wang, X., Chen, G., Dingjie, S., Zhiyi, Z., Chen, Z.,, Xiao, Q. Chen, J., Jiang. F., Li, J., Wan, X., Wang, B., and Li, H. CMB: A comprehensive medical benchmark in Chi- nese. In Duh, K., Gomez, H., and Bethard, S. (eds.). Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 6184 6205, Mexico City, Mexico, 2024. Associa- tion for Computational Linguistics. URL https:// // ac aclantte sclanthology. org/ 2trg/200 2024 maacl-long 343.
Yan, C., Fu, X., Xiong. Y., Wang. T., Hui, C., Wu, J., and Liu, X. Llm sensitivity evaluation framework for clinical diagnosis. In Proceedings of the 31st International Con- ference on Computational Linguistics, pp. 3083-3094, 2025.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. ArXiv preprint, abs/2407. 10671, 2024. URL https:/ //ts:///rxvv... wtggabs//4aaseooo 10671.
12

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Basee Based Problem Solving
A. Details of Datasets
Question: 25-year-old man comes to the office because of pain in his left shoulder. He says that this started years ago Which the following enzymes is most likely deficient in this patint?
Antimalarial drug used for causal prophylaxis act which stage developmental cycle?
Question:
Options:
Options:
A: Branched-chain alpha-ketoscid dehydrogenase
A. Gametogony
Cystathionine synthase deficiency
Erythrocylic schizogany
C: Hamogemitsic acid oxidase
Pre-erythrocytc schizogony
Phenylalanine hydroxylase
Exo-erythrocylic schizogony
E: Propionyl-CoA carboxylase
Answer: C
Answer: C
Figure Examples of Low-Level tasks. Left: a MedQA question. Right: a MedMCQA question.
A.1. Low-Level Tasks
As introduced in Section 3.3, we adopt the original multiple-choice questions the MedQA and MedMCQA datasets for Low-Level tasks. MedQA a large-scale medical exam dataset, containing medical exam questions sourced from three different regions. In this study, use the questions in the US subset, which contains five-option multiple-choice questions collected from the United States Medical Licensing Examination (USMLE). MedMCQA is another large-scale medical exam dataset that contains multiple-choice questions sourced from All India Institute of Medical Sciences Entrance Examination (AIIMS) and National Eligibility cum Entrance Test for Postgraduate (NEET-PG) in India. The questions in MedMCQA are four-option MCQs. The language used in both datasets is English. For MedQA, we use the test set for evaluation; for MedMCQA, we use the dev set, because the ground truth of MedMCQA's test set not available to the public. We filtered out the MedMCQA questions that are marked multiple correct answers. As result, we have 800 questions from MedQA and 2,816 questions from MedMCQA for Low-Level tasks. Examples of Low-Level tasks shown in Figure 8.
(Example 1)
Question: Antimalarial drug used for causal prophylaxis act which stage of developmental cycle?
Options:
Gametogony Erythrocytic schizogony Pre-erythrocylic schizogony Exo-erythrocylic schizogony
Answer:
(Example 2)
Question: Which the following vilamin K dependent procosgulant?
Options: Factor II Factor C: Factor Factor XI
Answer: D
(The other three examples)
Question: The anteroposterior curve is
Options:
A: Curve of Spee B: Curve of Wilson C: Curve of Monson D: Bonwill's curve
Answer:
Figure 9. Input prompt format the five-shot in-context learning setting.
We use five-shot in-context learning to evaluate the performance of LLMs on Low-Level tasks. Figure 9 shows the input prompt format the five-shot in-context learning setting. The input prompt consists five examples, each of which is a question-answe answer pair. After the five examples, we append the test sample, which is question without the correct answer. The model required to predict the correct answer for the test sample. For answer parsing, we find that current LLMss
13

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Basee Problem Solving
always follow the correct format in the five-shot in-context learning setting. Therefore, we directly compare the generated answer with the ground truth answer.
A.2. Mid-Level Tasks
comes the office because of pain his left shoulder. becausec of pain in his left shoulder. He that this pain started years ago and has progressively worsened. denies joint trauma, fever, dysuria, or morning stiffness. Stalement: "Homagentisic is most likely deficient in this patient', is the statement above
Question: 25-year-ol
Question: 25-year-old comes
Question: A 25-yeer-dld man ... There is 50% chance that the correct answer is not included in the options. Please
because of Which of the following enzymes is most likely
patient?
answer among the given options and respond with yes or no.
Options:
A: Branched-chain alpha keloacid dehydrogenase
B: Cystathionine synthase deficiency
Options:
C: Homogentisic acid oxidase
A: Branched-chain alpha ketoació dehydrogenase
incorrect?
Phenylalanine hydroxylase
Cystathionine synthese
E: Propionyl-Cok Con carborylase
C: Tyrosine aminotransferase (distractor)
Alice chose and if not, provide the correct answer.
Please verify this correct,
Phenylalarine hydroxylase
Propionyl-CoA carboxylase
Answer: correct
Answer: incorrect, answer the correct answer
Answer: no
Figure 10. Examples of Mid-Level tasks. Left: Statement Validation tasks; Middle: Multi-step Rectification tasks; Right: Answer
Existence Judgment tasks.
We construct Mid-Level tasks by reformulating the original MCQs in MedQA and MedMCQA into statement verification tasks, multi-step rectification tasks, and answer existence judgment tasks. For the Statement Validation task, we prompt GPT-4o to generate a statement based on the question and given option, using the following prompt format:
You medical expert and are given multiple choice question. Please change the question into statement verification based given option.
Example 1:
**Inpu***
Original Question: Gene for Dentin mineralization
Options: A: MAPIB B: PHEX C: DEN D: PHIX
Answer: B
Given Option: A
**Outpu***
Statement: MAPIB is the gene for Dentin mineralization.
Label: False
Example 2:
**Inpu***
Original Question: Child of Vasanthi was weaned from breast milk on the day and was given sugarcane juice child developed hypoglycemia and hepatomegaly biochemical examination showed hypophosphatemia and enzyme deficiencies- reducing substances in urine. The child is probably suffering from which of the following enzyme deficiencies
Options: A: Fructokinase Aldolase C: Glucose Phosphatase D: Beta galactosidase
Answer: B
Given Option: B
**Output**
Statement: Child of Vasanthi was weaned from breast milk on the 5th day and was given sugarcane juice the child developed hypoglycemia and hepatomegaly biochemical examination showed hypophosphatemia and enzyme defcciencis-rrduccng substances in urine. The child is probably suffering from the deficiency of enzyme Aldolase B.
Label: True
Test:
**Inpu***
Original Question: [Original MCQ ]uestion]
Options: [Options]
Answer: [Correct Answer]
Given Option: [Give Option]
**Outpur**
14

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario- Based Problem Solving
We found that providing both the correct answer and the given option in the prompt can help the model generate more accurate statements. We also provide examples of the generation process in the prompt help the model understand the task. For the Answer Existence Judgment task, we prompt GPT-4o to generate distractor based on the question and the correct answer:
The following a medical question along with its correct option. To increase the difficulty of the question, please generate a misleading distractor based on the correct option. This distractor should be an incorrect answer to the question. Directly generate the distractor without extra content.
Question: [Original MCQ Question]
Options: [Options]
Correct Option: [Correct Answer]
Distractor:
To ensure that GPT-4o can correctly generate the desire statements and distractors, we have three experienced medical experts manually verify a batch of 100 generated statements and distractors. We found that GPT-4o can generate correct statements and distractors with an accuracy of around 95%.
For generating questions of Mid-Level, we leverage question templates illustrated in Figure 10 to generate corresponding questions, using options and correct answers from the original MCQs, the generated statements, and the generated distractors. For evaluation, use the same five-shot in-context learning setting as in Low-Level tasks.
A.3. High-Level Tasks
For the High-Level task, following Hager et al. (2024), construct full-path cinical-diagnosis evaluation dataset based the MIMIC-IV dataset. MIMIC-IV is large-scale, de-identified, and publicly available dataset that contains electronic health records (EHRs) of patients admitted in Beth Israel Deaconess Medical Center in Boston, MA. To make the generated dataset align with tasks in the previous levels, we first identify and extract diseases involved in the MedQA and McdMCQA datasets to form a disease pool, and then retrieve the corresponding admission records from the MIMIC-IV dataset. Specifically, we first use the ICD-10-CM codes to filter the relevant admission records for each disease, and extract the discharge diagnosis section of these admission records. To ensure that the admission records are highly relevant the diseases, we only keep the records that the corresponding disease is in the first place of the discharge diagnosis, We further exclude the records that the discharge diagnosis contains multiple diseases in the discase pool. To ensure the statistic significance of the evaluation and balance across selected diseases, we only keep the diseases that have more than 10 admission records, and randomly sample 100 admission records for disease that have more than 100 records. As a result, we obtain 2,17 1177 76 admission records for 42 diseases. The distribution of the number of admission records for each disease and their affected human body systems are shown in Figure 11.
Cardiovascular Respiratory
Digestive Nervous
Genitourinary Endocrine
Musculoskeletal Hematologic & Immune
Others
Number Records
100
50
Ortiontatie oporessien
Aanilatton
Weat fallure
Cluuticaaoo focation
iintricaloon tachy prycariiaa
Periceentts
Hyportanve tarstan
Unsosii ongaina
Centaa repoooadee
Bick siovp
Servettiy
Autiony
Penc eecreeatis
Goola
Esophoggen cortict
Arsaaii Pisatii lpawll syndr ontre
dulbat Retharactione aeterrthage
Harmal pressuree copiiii
Herpee, coster
Corpal tamres
Herratoria
Privic Infarmmtton drcasi
Photattils
Padymonggie coccraati
Deficencc arennia
infectiovs incronutiion is
Hemulgg areenio
Seonis
dectereria
Lyme Rssers
Nemratzngg feccons Apootteenia
Tabsiar
Adiesivv
Figure 11. Number of admission records for each disease in the constructed full-path clinical- evaluation dataset.
We then construct the full-path clinical-diagnoiis evaluation dataset by extracting the history of present illness, physical examination sections from the clinical notes of the admission records, and the corresponding laboratory tests, microbiology
15

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario- Based Problem Solving
tests, and imaging tests from the clinical data tables. Then, we construct an agent-based evaluation setting, where the model is required to make decisions (e.g., request specific tests, make the final diagnosis) based on the information it obtain through the interaction process. Specifically, for each admission record, we first provide the history of present illness to the model, using the following prompt format:
You are an experienced medical Al assistant. Your ultimate goal is to help the doctor diagnose the patient's 's condition. You will be provided with the patient's 's history and the results of any tests that the doctor has already performed. You can also order additional tests for more information, including physical examinations, laboratory tests, microbiology tests, and imaging.
The action you can chooxe are:
1. PE: Perform physical examination of patient and receive the observations.
LAB: Run laboratory lests and receive their values. will get all the lab tests results once.
3. MICRO: Run microbiology tests and receive their values. You will get all the microbiology tests results at once.
4. IMAGE: Do specific imaging scans and receive the radiologist report. You will get all the imaging results at once.
5. OUTPUT: Output the final diagnosis.
Note: To improve diagnostic efficiency, please perform physical examinations (PE), laboratory tests (LAB), microbiological tests (MICRO), and imaging scans (IMAGE) only when necessary for diagnosis. When you are confident, choose the "OUTPUT" action and you will be asked to output the corresponding diagnosis.
Your output format should be:
Rationale: (your reasoning process for choosing the next action)
Action: (one of the actions [PE, LAB, IMAGE, MICRO, OUTPUT))
Now patient comes to see the doctor.
Patient History: [History of Present Illness].
Please choose your action from [PE, LAB, IMAGE, MICRO, OUTPUT).
If the model chooses the "PE" action, we provide the physical examination section of the clinical notes to the model using the following prompt format:
Physical Examination of this patient: (Physical Examination).
Please choose your action from [Lefted Actions].
Otherwise, if the model chooses one of the "LAB", "MICRO", or "IMAGE" actions, we will further ask the model to provide the detailed list of examination items of the corresponding test type:
You choose (Model's 's Action] as the next action. Please provide the specific list of [Model's 's Action] you want to run: please output your choice the following format:
Rationale: (your reasoning process for choosing the next action)
Lab Tests: (the specific laboratory tests you want run, separated by commas)
To parse the model's requested examination items, leverage the name of examination items appeared in MIMIC-IV to form an examination pool. Then, we use UMLS to map the examination items in the pool to the corresponding CUIs, and retrieve the corresponding synonyms of the CUIs. Finally, we extract and match the synonyms of the CUIs with the model's s requested examination items using fuzzy matching. If the similarity between the synonyms and the model's requested examination items is above a certain threshold (0.9), we consider the examination items are correctly parsed. We further conduct manual verification over the examination items that are not identified by the fuzzy matching, and construct a mapping table between the missing examination items and their standard names in MIMIC-IV, Finally, we verify the recall of the examination items parsing process by randomly sampled 100 recognition results randomly sampled from the evaluated models' outputs, and have three experienced doctors manually verify the results. We calculate the recall of the examination items parsing process by comparing the manual verification results with the recognition results of the evaluated models. We find that the recall is around 95% across all three types of tests.
After the parsing process, we provide the model with the results of the requested tests, and ask the model to choose the next action:
16

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario- Based Problem Solving
Here are the [Model's Action] results of this patient:
[Results of the requested tests]
Please choose your next action from [Lefted Actions].
If the model fails to output the valid action, we will warn the model and ask the model to choose the next action again:
You have chosen an action that is not available or used the wrong format.
Your output format should be:
Rationale: (your reaxoning process for choosing the next action)
Action: (one of the actions in [Lefted Actions]]
Please choose your next action from [Lefted Actions].
Once the model chooses the "OUTPUT" action, we will further ask the model to output the final diagnosis:
You choose "OUTPUT" as the next action. Please output the final diagnosis for this patient. Your output format should be:
Diagnosis: (the final diagnosis, specific disease name)
We use the zero-shot learning setting to evaluate the performance of LLMs on this full- full-path clinical diiggossss task. When the chat history length exceed the maximum token length of the model, we ask the model to summarize the chat history and provide the summary to the model as the input prompt:
Summarize our chat history, condense the content as much as possible while preserving all essential information related to the diagnosis. Eliminate redandant or irrelevant information, and ensure the summary muintains coherence and clarity.
Your output format should be:
Summary: (your summary of the dialogue)
For answer parsing, we construct disease synonyms mapping based the UMLS matching and manual verification process, and use the mapping to match the model's output with the ground truth. We also record models' requested examination items to calculate the full-path clinical-diage diiggosss accuracy.
B. Details of Evaluation Metrics
As introduced in Section 3.3, we use accuracy the evaluation metric for Low-Level and Mid-Level tasks. For High- tasks, we use the full-path dinical-diagnose accuracy as the evaluation metric. To make the evaluation metrics across different levels comparable, we normalize the accuracy of Low-Level and Mid-Level tasks using Equation (4), Specifically, for the Low-Level tasks (MCQs), we set the random accuracy as N. where N, is the number of options in the MCQs. For the Statement Validation tasks and Answer Existence Judgment tasks, we set the random accuracy as 0.5, as there are only two possible labels. For the Multi-step Rectification tasks, we find that the random accuracy of questions with the correct answer provided is much higher than that of questions with the wrong answer provided, because for the latter case, the model need to further choose the correct answer from the rest of N. options. Therefore, consider weight the accuracy of the questions with the correct answer provided by and the accuracy of the questions with the wrong answer provided by 1 -- a to make the random accuracy independent of the random strategy. Considering random strategy that deciding the given option is correct with a probability of p. the random accuracy of the Multi-step Rectification tasks is calculated as:
Accuracy rand = a x p + (1-a) x (1-p)
N. -
Since we want the random accuracy be invariant to the random strategy, the derivative of the random accuracy with respect p should be zero, which leads to a N. Therefore, we set N for the Multi-step Rectification tasks, and the random accuracy is N. as well.
For the full futh- clinical-dian -dinios-s accuracy, since the task format is open- open-ended, we set the random accuracy as zero and directly report the accuracy of the evaluated models.
17

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Baser Problem Solving
C. Details of Evaluated Models
list the basic information LLMs evaluated in our study in Table 7. The models are divided into general and medical- domain specific models. We denote Phi3-mini- wimmmiin-nttutt-l-l instruct-128k Phi3-small maan--smll--nstruct-- Phi4 Phi3-3.8B, Phi3-7B, Phi3- 14B, and Phi4-14B, for simplicity.
1ii3-medium-Iieg Inst-nnsstttta 122k, and
Table The basic information of large language models evaluated in study, including the model type, family, backbone model. parameter size, and training data size.
Model Type
Family
Model
Backbone Model
Parameter Size (B)
Training Data Size (T)
Llama-7B
1.4
Llama- 13B
Llama-33B
33
1.4
Llama-65B
65
1.4
Llama
Llama2-7B
2
Llama2-13B
Llama2-70B
Llama3-8B
15
Llama3-70B
70
15
Qwen-7B
Qwen-14BB
Qwen-72B
72
3
Qwen2-7B
7
7
Qwen
Qwen2-72B
72
Owen2.5-7B
Qwen2.5-
18
General Domain
Qwen2.5-32BB
32
18
Swen2.5-72B
72
18
Gemma-2B
Gemma-
Gemma
Gemma2-2B
2
2
Gemma2-9B
8
Gemma2-271 2-27B
27
13
Phi3-3.8B
3.8
3.3
Phi3-7B
4.8
Phi
Phi3-14B
3.8
Phi4-14B
14
9.8
GPI-4o-mimi
Not Available
Not Available
GPT
GPT-4o
Available
Available
03-mini
Not Available
Not Available
DeepSeek-V V3
671 (37 activated)
14,8
DeepSeek
DeepSeck-RI
671 (37 activated)
14.8
ClinicaCCamel-700
Llama2-708
Med42-70B
Llama2-700 70B
70
2
Mcditron-70B
Llama2-700 70B
70
2
Mcd42-v2-8B
Llama3-88 8B
8
15
Medical Domain
Others
Med42-v2-7000
JJama3-70B
Meditron
Llama3-8B
Meditron3--0B
Llama3-70B
70
15
MMed-LJama3-8B
Llama3-8B
8
15
D. Details of Evaluation Settings
For the Low-Level and Mid-Level tasks, we set the model temperature as 0 to ensure the model outputs the most confident answer. We conduct five repeated experiments by randomly selecting five demonstrative samples from the rest of dataset. then calculate the average accuracy of the five experiments, and report the mean and standard error of the accuracy across all test samples. For the High-Level tasks, considering that low temperature may lead to the model being conservative, set the model temperature 0.8 to encourage the model to explore more possibilities. We also verified through preliminary experiments that LLMs produce stable outputs with this parameter setting. We conduct repeated experiments three times and report the mean and standard error of the accuracy.
18

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Seenario-Basch Problem Solving
E. Details of Evaluation Results
We list in Table 8 the detailed results of the evaluated LLMs on tasks across all cognitive levels. The results are reported in the format of mean and standard error of the accuracy. We notice that some medical models (Meditron- LLMed-Llama3-ae achieve near-to-zero performance on the High-Level tasks. After carefully checking the outputs of these models, we find that these models fail to follow the instruction and tend to repeat the given instruction output irrelevant information.
Table 8. Full performance (mean and standard error of normalized performance (%)) of Large Language Models on tasks across different cognitive levels. Considering the high cost of GPT-4o, 03-mini, DeepSeek- -V. and DeepSeek-RI. we only evaluate these models with three repeated experiments.
Mid-Level
Low-Level
High-Level
Model
StmiVal
AnsExist
Mulrikess
McdQA
McdMCQA
McdQA
MedMCQA
McdQA
McdMCQA
McdQA
McdMCQA
MIMIC-IV
Llama-7B
3,36 : 0.82
10:40
0.24
4,88 0.84
4.52 0.45
0.86 i 0.70
0.55 1 0.43
1.60 0.63
8.06 0.26
Llama- 13B
13.18 1 0.57
18.98
0.22
6.47
0.74
6,80 0.57
0.05 1. 0.63
0.49
0.54
1.45 1 0.26
2.17 0.16
Llama-33B
0.26
0.34
16.70
0.24
12.29
0,48
186 0.26
0.27
19.91
0.84
17.99
0.34
Llama 65B
26.64 1: 0.49
29.59 1 0.29
22.26
0.65
17.03
1 0.51
5:53 1 1.01
2.59 0.20
22.60
1: 0.31
20.83
0.18
Llama2-7B
18.32
0.70
0.59
0.25
0.22
0.43
0.31
2.65
Llama2-133 13B
21.10 0.30
20.94
0.33
9.58
1.77
12.44
0.32
1.08 1: 0.92
1.25 1: 0.35
7.84 1: 0.32
8.65 1
4.92 1 0.15
Llama2. 70B
37.99 1: 0.34
35.60 4: 0.26
23.77
0.43
17.14 0.17
5.51 1 0,76
2.58 1: 0.68
33.47
0.52
28.65
1 0.26
5.16 1: 0.18
Llama3-8B
37.30
22.77
0.15
Llama3: 70m
63.68 1 0.34
60.82 1 0.28
53,16
0.46
41.29 1 0.34
9.36 1 1.01
10.37 0.64
62.87
1. 0.29
$7.07 1 0.16
17.81 1: 0.41
Qwen-7R
19:40 + 0.31
24.85 + 0.21
14.19
0.49
13.00
0.69
2.74 0.93
1.00 0.25
5.09
0.43
7.24 + 0.21
2.51 + 0.31
Qwen-148
38.05 + 0.38
39.25
21,13
18:56
4.93 +
3.06 0.27
27,89
0.09
26.31
0.22
4.14 + 0.29
Qwen 72B
50.60 1: 0.39
50.95 + 0.18
36.43
0.66
30.99
1: 0.36
10.42 1: 0.55
7.09 0.18
38.26
1: 0.41
40.74
1: 0.49
5.89 1: 0.35
Qwen2-78
36.73 +
39.85
28.08
0.20
24.66
0.30
4.00
4.68 0.25
31.60
0.43
33.46
0.24
6.99 0.25
QWB
65.91 1: 0.24
60.41 1 0.19
55,09
0.23
42.23
0,23
24.00 1 0.58
11.67 1 0.14
62.86
0.27
57.68
0.26
15.10 1 0.09
Qwen2.5-78
42,86 + 0.49
45.39 0.17
30.29
0.34
26.43
0.38
13.66 + 0.23
7.49 0.32
37.08 + 0.22
40.34
+ 0.19
9.50 + 0.04
Qwen2.5.14B
33,98
0.29
11.82
46.40
0.15
Qwen2.5.32P 32R
59.21 + 0.21
57.17 + 0.09
48.05
0.41
37.92
0.06
14:31 + 0.60
11.36 0.36
$3.01 + 0.33
$1.04
+ 0.11
11.87 + 0.22
Qwen2.5-728
67,83 1: 0,20
61,82 1 0.12
55,70
0.42
44.79
0.53
28.98 1: 0.65
0.33
63,70 1: 0.36
58.39 1: 0.13
16.05 1 0.24
GBmma-2B
6.66:.066
1.21
2
-1.28 1.01
00.44
0.49
4.72
0.59 = 0.09
Gemma- 7B
30,35 1: 0.42
27.41
- 0.38
17.03
0.49
18.90
1:00::000 0.10
5.16 1 0.34
2.89 0.31
23.05 1: 0.32
19.14
1: 0.43
0.33 : 0.09
Gemma2-28
14.87 0.34
24.18
0.31
4,06 0.62
10.23
4 0.45
-044 + 0.58
0.68 0.69
0.86 = 0.30
0.63 0.05
5.02 - 0.33
Gemma2-9p
44.21 1: 0.18
46.20 = 0.20
32.86
= 0.57
27.27
0.31
6.97 1 0.74
6.76 0.40
33.78 0.32
35.11:0.3
9.04 -: 0.28
Gemmna2-27B
53.65 0.31
50.45 = 0.18
41,36
0.68
66
0.22
10:59 + 1.17
10.14:00.0
48.23 = 0.16
45. 0.25
11.19 = 0.64
Phi3-3.8BB
37.52
29.01
0.72
24.40
0.04
+
7.50 0.29
30.31
35.70
0.40
2.21
Phi33-B
49.25 1: 0.30
46.95 0.30
38,92
0.29
27.63
11 0,25
4.83 1 0.45
2.13 1 0.23
42.61
0.24
35.58
0.30
7.74 1: 0.16
Phi3-14B 14B
54.06 1: 0.49
49.70 : 0.27
40.45
0.59 44.88
31.22
0.26
5.79 4: 0.52
8.24 0.48
48.75
0.46
45.21
1 0.20
3.89 1 0.01
Phi4--44B
16.19
52.62
15.13
53.16
0.26
49.98
0.15
GPI-to-mini
0.17
55.40 0.10
47.9
0.44
36,46
24.9% = 0.74
12.32
56.11 = 0.51
42
0.19
13.81 = 0.84
GPT-4o
78.33 0.75
64.00
1 1.02
67,83
0.60
49.67
0.33
33.33 1: 2.20
11.83 2.91
70.92 = 1.23
63.39 1 0.39
19.33 - 0.30
03-mini
87.71 0.55
74.44
76.67
0.17
57.50
0.76
62.83 +
30.50
1.89
86.17
11.12
15.05 0.45
DeepSeek-V3 V3
78.33 +: 0.75
65.78
: 0.22
56.50
0.63
50.83
+ 1.17
5.17 + 0.17
7.33
1.69
72.71
0.40
62.06
+ 0.11
19.42 + 0.17
DeepSeck-R1 R1
89.79 1: 0.75
74.44 1 0.80
77,33
1.33
62,00
0.29
54.17 + 1,33
36.33
1.09
87.67 1: 0.65
75.44
1: 0.11
26.54 1 0.31
Clinics[Camcl-708
0.52
0..14
21.68
0.22
7.09 E 1.02
0.332
36.82
0.63
52.80
:
2.19 0.08
Mcd42-70B
19.18
48.11
0.21
33.41
0.72
21.74.0.0
12.65 + 0.54
0.556
40.01
1 0.78
36.77
1: 0.24
4.25 - 0.28
Meditron-700 70B
4.55
36.18 0.31
28.15
0.65
20.89 0.47
8.18 1.05
4.23
37.64
0.33
29.80
0.32 0.01
Llamg3-Med42-88
0.64
+7.13
32.10 0.65
2.12
9.31 + 0.60
7.08 0.35
40.47 = 0.29
41.81 0.25
11.22 2 0.21
Llama3- Mad42-
65.88
62,74 0.19
0.223
44 0.24
20.86 1.13
+
62.68 + 0.28
58.83 0.17
12.37 0.18
Meditron3-8B
4..60
0.23
4.48 0336
5.31
25.28
Meditro3-700 70B
0.266+0.27
62.27+0.11
548220029
0.29
6.52 1.22
0.55
61.70
+ 0.52
$4.76
10.36
MMcd-Llama3-98
3..96
0.33 021
0.55
5.31 - 0.27
0.33
24.64 0.24
0.32 0.16
19

---

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Cemarroo-assee Problem Solving
F. Details of Clinician Validation
To validate that the cognitive difficulties of tasks in our benchmark are consistent with cognitive levels defined in Section 3.2, we conducted clinician validation. Specifically, we first randomly sampled 20 questions from each task form subset of 100 questions. For Low-Level and Mid-Level tasks, we sample questions from the MedQA benchmark. Then, recruited four licensed clinicians with to years of experience evaluate this subset from perspectives: (1) their accuracy in answering the questions and (2) their subjective difficulty ratings to the questions (I=Easy, 10=Hard), The detailed labeling instructions are as follows:
Please answer the following questions based on your medical knowledge. For each question, please provide your answer and rate the difficulty of the question on scale from 1 to 10, where means "very easy" and 10 means "very hard".
Low-Level Tasks: These tasks require basic medical knowledge and understanding of medical concepts. Please choose the most appropriate answer from the given options.
complex application of medical knowledge. For statement validation tasks, please determine whether the statement is true or false. For answer existence judgment tasks, please determine whether the answer exists in the given options. For multi-step rectification tasks, please first determine whether the provided answer is correct, and if not, provide the correct answer.
Mid-Level Tasks: These tasks require
High-Level Tasks: Given the history of present illness section from inpatient admission note, the task is provide the patient's ' primary diagnosis.
At each step, the following operation types are allowed:
(1) Physical Examination: Request a physical examination for the patient;
(2) Laboratory Tests: Request laboratory tests for the patient:
(3) Microbiological Culture: Request microbiological cultures (c.g., blood, urine);
(4) Imaging:
Request imaging tests (e.g., CT, X-ray, ultrasound);
(5) Diagnosis: A diagnosis should be provided when (a) the test results sufficient support a diagnosis,
or (b) all possible tests have been completed. Output the patient's 's primary diagnosis (can include multiple diagnoses), including any conditions already mentioned in the HPI.
Note: Please record the sequence in which the test types were requested for diagnosis.
20


Reviews:
[{"review_id": "QuC9iEvjvk", "reviewer": "Reviewer_z2s4", "summary": "The paper proposed a novel medical LLM evaluation benchmark inspired by Bloom's taxonomy. Different from existing benchmarks that only evaluate the LLM on one single style of QA tasks, the proposed benchmark constructs a multi-cognitive-level evaluation framework and provides more informative results. The proposed method revealed the fact that the existing LLM is only performing well on low-cognitive-level tasks that mainly involve knowledge memorization. However, even the most state-of-the-art LLM can fail in high-level clinical diagnosis tasks. It also reveals the important relationship between model size and its ability in high-level complex scenarios.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your kind feedback as well as your recognition of our work. Below are our responses to each of the concerns you raised.\n\n1. **Discussion with other LLM benchmark**: We sincerely appreciate your kind suggestions. Zhou et al. [1] proposed a medical evaluation framework to generate reliable and diverse test samples based on knowledge bases, addressing the issues of low reliability and diversity in automatic test sample generation. Meanwhile, our work aims to explore the limitations of LLMs in solving real-world medical problems by constructing a benchmark with medical tasks of varying cognitive levels and systematically evaluating existing LLMs on this benchmark, offering insights for developing LLMs suited to real-world medical applications. Once again, thank you for your kind suggestions. We will strengthen discussions with other medical LLM benchmarks in the revised paper.\n\n   [1] Zhou, Yuxuan, et al. \"Reliable and diverse evaluation of LLM medical knowledge mastery.\" arXiv preprint arXiv:2409.14302 (2024).\n\n2. **Name of proposed benchmark**: Thank you for your kind suggestions. Indeed, a good benchmark name could facilitate reference and discussion. We plan to name the benchmark MulCogEval (Multiple-Cognitive-Level Evaluation) and will include relevant annotations in the paper.", "claims_and_evidence": "The proposed medical LLM benchmark is inspired by Bloom's taxonomy, which is very intuitive and convincing. The proposed task construction protocol is reasonable and thoughtfully evaluated by human experts. The evaluation results provide intuitive but still inspiring results. Also, the QA example provided in the supplementary helps to better understand each task.", "methods_and_evaluation_criteria": "The proposed novel benchmark dataset seems to be very promising and significant to the reviewer. The concerns about the LLM's performance on medical application is a long-existing problem. However, current benchmarks mainly focus on simple knowledge-based QA tasks, ignoring the complex situation in the real-world diagnosis procedure. The proposed benchmark dataset will be helpful to better evaluate the capability of existing LLMs and provide more informative results to guide the development of the field.", "theoretical_claims": "N/A. There is no new theoretical claims proposed in the paper.", "experimental_designs_or_analyses": "The experiment in the paper is very convincing and thorough. It covers more than 20 different general LLMs and multiple medical-specific LLMs fine-tuned from these baseline LLMs. The baselines have also covered different parameter sizes from 2 B to 70 B. The task-specific results in Tables 3 and 4 further help to understand the behavior of the medical LLMs. Overall, the experiment is convincing and reasonable. The evaluation of the medical-specific LLMs is very interesting since it helps illustrate the potential problem within these LLMs, where the post-training potentially ruined the reasoning capability of the original model, resulting a worse high-level task performance.", "relation_to_broader_scientific_literature": "The paper has properly discussed related literature and the status quo of current medical LLM evaluation. It can potentially serve as the standard evaluation protocol for future medical LLM evaluation, providing a more reasonable and realistic evaluation.", "essential_references_not_discussed": "While the paper has a detailed discussion of existing works. It will be great if it can provide some more discussion on the other medical LLM benchmark released recently, such as [a]. But this will not harm the contribution of the paper.\n\n[a] Zhou, Yuxuan, et al. \"Reliable and diverse evaluation of LLM medical knowledge mastery.\" arXiv preprint arXiv:2409.14302 (2024).", "other_strengths_and_weaknesses": "Overall, the paper looks pretty solid and convincing. The proposed benchmark provides a more detailed understanding of existing LLMs. It can serve as an important step towards LLM's application in real-world diagnosis.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "One small thing: I am wondering if the proposed benchmark has a name or not. It will be much easier for others to refer to in the future.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UYdBZgGRjD", "reviewer": "Reviewer_kVjJ", "summary": "This paper assesses large language models (LLMs) across multiple cognitive levels, based on Bloom's taxonomy that proposes six cognitive objectives/levels in ascending order of complexity. In particular, tasks pertaining to three cognitive levels (preliminary knowledge grasp, comprehensive knowledge application, scenario-based problem solving) were defined and attempted with five state-of-the-art LLMs. It was found that LLM performance declined with increased task cognitive complexity, and that larger LLMs performed better when higher cognitive complexity was required.\n\n## update after rebuttal\nThe authors' response is appreciated and improved our assessment of the study.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful and constructive feedback. Below, we address each of the concerns you raised.\n\n1. **Comparability between different cognitive levels**: Thank you for your thoughtful concern. Indeed, comparing LLMs’ performance across different cognitive levels is crucial for analyzing their medical capabilities. Considering this, we have implemented a metric alignment strategy in the \"Performance Metrics\" section (see Equation 4 on page 5) to eliminate interference caused by random guessing and ensure the comparability of tasks across different cognitive levels. \n\n   In fact, we have carefully considered the cognitive difficulties when designing the tasks in our benchmark. Compared to MCQs, mid-level tasks increase difficulty by removing clues, increasing reasoning steps, and expanding the decision space. High-level tasks go further by requiring LLMs to request necessary information instead of receiving it directly from the question. To ensure that the constructed benchmark meets our expectations, we followed your kind suggestions and conducted a clinician evaluation. Specifically, we randomly sampled 20 questions from each task, resulting in a clinician evaluation subset of 100 questions. Four licensed clinicians with 3 to 8 years of experience were recruited to assess the benchmark’s difficulty from two perspectives: (1) their accuracy in answering the questions and (2) their subjective difficulty ratings to the questions on a scale from 1 (Easy) to 10 (Hard). The evaluation results are as follows:\n\n   |Cognitive Level|Clinician Accuracy|Clinician Subjective Difficulty|\n   |-|:-:|:-:|\n   |Low|68.8|Mean:5.0 Median: 5.5|\n   |Mid|54.2|Mean:6.0 Median: 5.8|\n   |High|23.5|Mean:7.5 Median: 7.5|\n\n   We found that clinicians’ accuracy also decreases as the cognitive level increases across three levels. Moreover, their subjective difficulty ratings align with this trend, further demonstrating the validity of our benchmark. Once again, we appreciate your kind suggestions and will incorporate this clinician evaluation into the revised paper.\n\n2. **LLM scale effect issue**: Thank you for your thoughtful comments. Indeed, the scaling law indicates that larger models generally perform better, but it lacks a detailed analysis of the impact of parameter size on tasks across different cognitive levels (difficulties). Considering this, we conducted a systematic evaluation to investigate parameter size effects at varying cognitive levels and found that larger LLMs significantly outperform smaller ones on harder medical tasks, demonstrating the critical role of parameter scale in real-world clinical problem-solving. Moreover, our analysis of medical-specific LLMs reveals that existing post-training methods fail to enhance high-level task performance, offering key insights for the development of medical LLMs.\n\n3. **Evaluation setting & parameter issues**: We sincerely appreciate your thoughtful comments. \n\n   (1) Evaluation setting: For low- and mid-level tasks, we adopted the few-shot learning (widely used in LLM benchmarks), as it introduces minimal subjective bias by only guiding LLMs with examples, while prompt engineering and CoT techniques may introduce subjective biases through prompt design, potentially affecting the fairness across LLMs. For high-level tasks, regarding the reasoning difficulty, we adopted an agent-based setting [1], guiding the model to generate a rationale before producing the corresponding action.\n\n   (2) Decoding parameters: We chose the decoding parameters to align with task characteristics. For low- and mid-level tasks, we set temperature = 0 (greedy decoding) as these tasks have well-defined answer formats, and greedy decoding selects the most confident answer. For high-level tasks, we set the temperature=0.8 to balance the reasoning diversity and output stability of LLM responses. We also verified through preliminary experiments that LLMs produce stable outputs with this parameter setting. \n\n   [1] Hager P et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature medicine, 2024.\n\n4. **Examples of LLM answer**: Thank you for your constructive advice. We will include more examples of LLM answers in the appendix to better illustrate our evaluation process.\n\n5. **Clarification of the Ambiguous Statement**: Thank you for your careful reading. We sincerely apologize for the ambiguity—here, \"training samples\" refer to the demonstrative examples used in few-shot in-context learning. Regarding few-shot learning, we have briefly introduced the process in Section 4.1 (\"Evaluation Setting\") and illustrated the corresponding input format in Figure 9. Notably, our evaluation does not involve any model training. Once again, we appreciate your careful reading. We will conduct thorough proofreading to avoid similar ambiguities in the revised manuscript.", "claims_and_evidence": "The claims are based on comprehensive empirical evaluation on 29 separate general-domain LLMs from five main families (Llama, Qwen, Gemma, Phi3, GPT), as well as eight medical-domain-specific LLMs (Tables 2 & 6). The main claims on performance declining with increasing task complexity as well as increasing with model size appear generally true, within each LLM model family.", "methods_and_evaluation_criteria": "While the proposed evaluation framework (Figure 2) is reasonable, whether the (accuracy) results are directly comparable can be contested, especially between low/mid-level and high-level tasks. For example, the low-level \"preliminary knowledge grasp\" tasks involve MCQs with four options, the mid-level tasks involves multiple steps but fewer options (equivalence justified in Section B), and the high-level tasks appear to be free-form (Section A.3). Empirical evaluation of task difficulty by clinicians would help to calibrate/justify their appropriateness.", "theoretical_claims": "No theoretical claims are presented.", "experimental_designs_or_analyses": "The experimental design appears largely sound.", "relation_to_broader_scientific_literature": "While the experiments are comprehensive, the main findings that higher-level cognitive tasks are more challenging, and that larger models fare better on such tasks, are largely unsurprising.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "A potential weakness would be the lack of prompt engineering and chain-of-thought query techniques, that can significantly affect LLM performance.", "other_comments_or_suggestions": "1. Some preliminary empirical justification for the selected model temperature (and other) parameters (as discussed in Section D) would be appropriate.\n\n2. Some examples of LLM answers (especially for a high-level case) would be welcome, in the appendix.", "questions_for_authors": "1. From Section D, it is stated that for low and mid-level tasks, the variance in reported performance metrics is due to \"conduct[ing] five repeated experiments by randomly selecting five training samples from the rest of dataset\". However, it is not immediately clear as to whether any additional \"training\" is performed for the LLMs, for purposes of this study. This statement might thus be clarified.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "tn2OJDeXyx", "reviewer": "Reviewer_VJ98", "summary": "This paper presents an evaluation framework inspired by Bloom’s Taxonomy, integrating multiple tasks to reflect diverse cognitive levels. The author evaluates popular general and medical LLMs, observing a significant performance decline in performance with the increasing of cognitive complexity.\nTheir findings highlight the importance of scaling up models' parameter sizes to more effectively tackle clinical challenges.\n\n## update after rebuttal\nIn the rebuttal stage, the authors provide details about the human evaluation of the cognitive levels. These evidence addressed my concerns regarding the experiment design. The paper proposed a valuabale evaluation pipeline to analysis the clinical ultility of LLMs.\nTherefore, I update my score to 3 (weak accept). \nHowever, I still think the findings within this paper, larger LLMs performs better in more difficult tasks, is not supervising. I hope the author could provide more in-depth analysis regrading these results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful and constructive feedback. Below, we address each of the concerns you raised.\n\n1. **Benchmark design issue** : We highly appreciate your kind comments. Indeed, the cognitive difficulty of a question is affected by multiple factors, including task type and question complexity. However, the difficulty of evaluation questions is typically constrained by the task format. For example, while the MedQA dataset is designed to evaluate LLMs’ ability to analyze clinical cases, its multiple-choice format somewhat limits its ability to fully reflect how well LLMs request and integrate diagnostic information in real clinical settings. In fact, the task formats in our benchmark inherently reflects variations in cognitive difficulty: Mid-level tasks reflect the impact of increasing reasoning steps and expanding decision space on performance, while high-level tasks further evaluating LLMs in requesting and integrating critical information for decision-making in clinical scenarios. While there is indeed some variation in difficulty among questions within the same task format, the constructed benchmark generally meets our design expectations. \n\n   We also conducted a clinician evaluation to validate the proposed benchmark. Specifically, we first randomly sampled 20 questions from each task to form a subset of 100 questions. Then, we recruited four licensed clinicians with 3 to 8 years of experience to evaluate this subset from two perspectives: (1) their accuracy in answering the questions and (2) their subjective difficulty ratings to the questions (1=Easy, 10=Hard). The evaluation results are as follows:\n\n   |Cognitive Level|Clinician Accuracy|Clinician Subjective Difficulty|\n   |-|:-:|:-:|\n   |Low|68.8|Mean:5.0 Median: 5.5|\n   |Mid|54.2|Mean:6.0 Median: 5.8|\n   |High|23.5|Mean:7.5 Median: 7.5|\n\n   The experimental results show that clinicians’ accuracy decreases as the cognitive level increases, aligning with their subjective difficulty rating. This indicates that the designed tasks successfully achieved the intended difficulty levels and can effectively reflect how far LLMs are from effectively solving real clinical problems. Thank you again for your kind suggestions, and we will add this clinician evaluation into the revised paper.\n\n2. **Evaluation of reasoning LLMs**: Thank you for your constructive suggestions. Indeed, since our paper submission, reasoning LLMs have been receiving increasing attention. Following your kind suggestions, we primarily evaluated two typical reasoning LLMs, DeepSeek-R1 and o3-mini, and compare them with SOTA chat LLMs in corresponding parameter scales. The evaluation results are as follows:\n|Model|Low-Level|Mid-Level|High-Level|\n   |-|:-:|:-:|:-:|\n   |DeepSeek-V3|78.3|44.8|19.4|\n   |**DeepSeek-R1**|**89.3**|**73.1**|**25.9**|\n   |GPT-4o-mini|62.7|43.0|13.8|\n   |**o3-mini**|**88.1**|**75.2**|**15.5**|\n\n   Results show that reasoning LLMs outperform chat LLMs across all cognitive levels, though the performance gap narrows on high-level tasks due to their significantly increased difficulty. Note that we have not evaluated other reasoning LLMs due to time and cost constraints (as the price of o1 is quite high). We plan to add this evaluation and include additional evaluations in our revised paper.\n\n3. **Insights of medical LLM development**: Thank you for your thoughtful comments. This work provides two key insights into the development of medical LLMs: \n\n   (1) Although the performance of smaller LLMs (<10B) on medical benchmarks is gradually approaching that of larger LLMs, our study indicates that increasing parameter size remains crucial for tackling tasks in higher cognitive levels. \n\n   (2) While medical post-training (used in current medical LLMs) and inference-time scaling (applied in reasoning LLMs) strategies work on low- and mid-level tasks, effectively solving high-level tasks demand further advancements in clinical reasoning abilities—particularly in the retrieval and integration of key information for decision-making in real-world scenarios. \n\n   Again, we appreciate your comments and will further highlight our insights in the revised paper.\n\n4. **Discussion with more benchmarks**: We are sincerely grateful for your kind suggestions. Wu et al. investigate LLMs’ ability to handle diverse medical tasks by constructing MedS-Bench, a large-scale benchmark covering 11 types of clinical tasks. Johri et al. study LLMs' ability to obtain the necessary patient information for diagnosis through diaglogue by proposing CRAFT-MD, an evaluation framework that simulates multi-turn doctor-patient interactions using a multi-agent approach. Meanwhile, our work focuses on evaluating how close LLMs are to effectively solving real-world clinical tasks by designing evaluations with progressively increasing cognitive difficulty and systematically analyzing existing LLMs. We will further enhance the discussion of other medical LLM benchmarks in our revised paper.", "claims_and_evidence": "Problematic claim:\n1. The author claims that question-answering (QA) is a task with low cognitive complexity, as large language models (LLMs) primarily need to memorize the medical information. However, some clinical questions in MedQA[1] require that LLMs analyze symptoms and engage in intricate reasoning to arrive at the correct answer.\n\n2. The cognitive complexity of Mid-Level tasks (such as statement validation questions) is influenced by the question's complexity. An example from the MedMCQA[2] dataset illustrates this:\n\"Q: Which vitamin is provided solely by animal sources?\" represents a straightforward knowledge question. Even if reformulated into a statement validation question such as: \"Does Vitamin C come exclusively from animal sources?\", it remains a simple task, requiring the LLM simply to memorize relevant information, without applying it to clinical scenarios.\n\nIn summary, the author classifies different cognitive levels based on the tasks with different input and output formats. However, in my opinion, a more suitable way to classify cognitive levels is based on the complexity of questions.\n\n[1] Jin, Di, et al. \"What disease does this patient have? a large-scale open domain question answering dataset from medical exams.\" Applied Sciences 11.14 (2021): 6421.\n\n[2] Pal, Ankit, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. \"Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.\" Conference on health, inference, and learning. PMLR, 2022.", "methods_and_evaluation_criteria": "This paper adopts accuracy and full-path diagnosis accuracy as evaluation metrics. I think this part is reasonable.", "theoretical_claims": "This paper does not contain theoretical claims.", "experimental_designs_or_analyses": "1. The evaluation of different LLMs on the proposed benchmark ignores state-of-the-art reasoning LLMs (such as openai o1 and o3-mini, Deepseek-R1).\n\n2. This paper ignores the evaluation of LLMs from (1) post-training (2) inference-time scaling dimensions. How different post-training and inference-time strategies can enhance the capability of LLMs at different cognitive levels is not explored.", "relation_to_broader_scientific_literature": "The findings within this paper are aligned with the scaling law[3].\n\n[3] Kaplan, Jared, et al. \"Scaling laws for neural language models.\" arXiv preprint arXiv:2001.08361 (2020).", "essential_references_not_discussed": "The key contribution of this paper is evaluating LLMs from different cognitive levels. However, previous works [4][5] have thoroughly evaluated LLMs across a variety of tasks and dimensions. It is important for the authors to discuss these studies.\n\n[4] Wu, Chaoyi, et al. \"Towards evaluating and building versatile large language models for medicine.\" npj Digital Medicine 8.1 (2025): 58.\n\n[5] Johri, Shreya, et al. \"An evaluation framework for clinical use of large language models in patient interaction tasks.\" Nature Medicine (2025): 1-10.", "other_strengths_and_weaknesses": "Strength: this paper comprehensively evaluates a wide range of popular LLMs across different tasks, which is valuable for the community to better understand the clinical capability of LLMs.\n\nWeakness: the classification of different cognitive levels is not convincing enough for me. Furthermore, this paper offers limited insights regarding how to enhance the performance of medical LLMs on high cognitive level tasks beyond merely increasing parameter sizes.", "other_comments_or_suggestions": "I will increase my overall score if the authors can address my concerns mentioned above.", "questions_for_authors": "Please refer to previous 'Claims and Evidence', 'Method and Evaluation Criteria', 'Experimental Designs or Analyses', and 'Other Strengths and Weaknesses' parts.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a multi-cognitive-level evaluation framework for medical LLMs, drawing inspiration from Bloom’s Taxonomy. The core contribution is the systematic design of tasks with increasing cognitive complexity—from basic knowledge recall to scenario-based clinical reasoning—and the comprehensive evaluation of over 30 general and medical-domain LLMs. The framework is further supported by clinician-based validation, which reinforces the credibility of the cognitive-level distinctions. However, the methodology largely repurposes existing QA datasets such as MedQA, MedMCQA, and MIMIC-IV, without introducing new benchmarks or fundamentally novel evaluation paradigms. While the cognitive framing adds structure and interpretability, recent works [1,2] have already explored broader and more diverse evaluation strategies for medical LLMs. Overall, although the conceptual novelty is limited, the framework is well-executed and the empirical analysis is thorough. Given the reviewer scores and practical relevance of the proposed benchmark, I lean toward a very weak accept, but would not object if the paper is ultimately rejected.
Decision: Accept (poster)