Benchmarking Large Language Models on CMExam - A comprehensive Chinese Medical Exam Dataset
Authors: Junling Liu
Peilin Zhou
Yining Hua
Dading Chong
Zhongyu Tian
Andrew Liu
Helin Wang
Chenyu You
Zhenhua Guo
Zhu Lei
Michael Lingzhi Li
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: large language model
chinese medical licensing exam
multi-choice
question answering
TL;DR: Evaluate the capabilities of large language models on the proposed Chinese Medical Exam Dataset (CMExam) from multiple dimensions.

Abstract: Recent advancements in large language models (LLMs) have transformed the field of question answering (QA). However, evaluating LLMs in the medical field is challenging due to the lack of standardized and comprehensive datasets. To address this gap, we introduce CMExam, sourced from the Chinese National Medical Licensing Examination. CMExam consists of 60K+ multiple-choice questions for standardized and objective evaluations, as well as solution explanations for model reasoning evaluation in an open-ended manner. For in-depth analyses of LLMs, we invited medical professionals to label five additional question-wise annotations, including disease groups, clinical departments, medical disciplines, areas of competency, and question difficulty levels. Alongside the dataset, we further conducted thorough experiments with representative LLMs and QA algorithms on CMExam. The results show that GPT-4 had the best accuracy of 61.6% and a weighted F1 score of 0.617. These results highlight a great disparity when compared to human accuracy, which stood at 71.6%. For explanation tasks, while LLMs could generate relevant reasoning and demonstrate improved performance after finetuning, they fall short of a desired standard, indicating ample room for improvement. To the best of our knowledge, CMExam is the first Chinese medical exam dataset to provide comprehensive medical annotations. The experiments and findings of LLM evaluation also provide valuable insights into the challenges and potential solutions in developing Chinese medical QA systems and LLM evaluation pipelines.

Benchmarking Large Language Models on CMExam A Comprehensive Chinese Medical Exam Dataset
Junling Liu Peilin Zhou
Yining Hua
Dading Chong
Zhongyu Tian
Andrew Liu
Helin Wang"
Chenyu You
Zhenhua Guo
Lei Zhu Michael Lingzhi Li
Alibaba Group Hong Kong University of Science and Technology (Guangzhou)
Harvard University Boston Children's Hospital PPeking University
6 "econd Affiliated Hospital of Zhejiang University School of Medicine 'Johns Hopkins University
Yale University Tianyi Traffic Technology 10 Ant Group 11 Harvard Business School
{william 1iuj, zhoupalin, andrew.promedd cszguo, zhulei0305) @gmail com 1601213984@pku edu. cn, zhongyutian@2ju. edu en, hwang2580jhn edu yininghuaßg. harvard. edu, chenyu.you@yale edu, mili@hbs edu
Abstract
Recent advancements in large language models (LLMs) have transformed the field question answering (QA). However, evaluating LLMs in the medical field is challenging due to the lack of standardized and comprehensive datasets. To address this gap, we introduce CMExam, sourced from the Chinese National Medical Licensing Examination. CMExam consists of 60K+ multiple-choice questions for standardized and objective evaluations, as well as solution explanations for model reasoning evaluation in an open-ended manner. For in-depth analyses of LLMs, we invited medical professionals label five additional question wise annotations, including disease groups, clinical departments, medical disciplines, areas of com- petency, and question difficulty levels. Alongside the dataset, we further conducted thorough experiments with representative LLMSs and QA algorithms on CMExxa. The results show that GPT-4 had the best accuracy of 61.6% and a weighted Fl score of 0.617. These results highlight great disparity when compared to human accuracy, which stood at 71.6%. For explanation tasks, while LLMs could generate relevant reasoning and demonstrate improved performance after finetuning, they fall short of a desired standard, indicating ample room for improvement. To the best of our knowledge, CMExam is the first Chinese medical exam dataset to pro- vide comprehensive medical annotations. The experiments and findings of LLM evaluation also provide valuable insights into the challenges and potential solutions in developing Chinese medical QA systems and LLM evaluation pipelines
1 Introduction
Recent advancements brought by large language models (LLMs) such as (Raffel et al. 2020) and
GPT-4 OpenAl 2023) have revolutionized natural language processing (NLP). However, evaluating LLMs in the medical field poses significant challenges due to the paucity of standardized and
comprehensive datasets compiled from reliable and unbiased sources (Li et al. 2023 Zhou et al.
2023b
Hua et al. 2024 Ye al. 2023 Liu et al. 2023d). Most existing medical datasets (Hendrycks
al. 2020 Abacha al.) 20019) Li al. 2023 Zhou et al. 2022) for language model evaluation
"Coresponniig Author. "Co-first authors
'The datasct and relevant code are available at
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

ID
Question
Candidate anoers
Asswer
Explanation
Aditiion annutations
WAREE CIRA CO & BE SOIE M 1
CD-1 Groupe Circ
Clisiel Depattent: IM
Thetrigg
Aivillaiun arbyyhma
Discppine: ClinMed
ctuasibulion heat
fille
D < Caniiianniii Dras
6-910 M.oM///MMMoeiiiill Infertion
commun antotthmiia vegani discee. and an importent facter inducing heart fallmry incrowed blood volume...
Competaey:: Medhund
Difficulty
Eacy
Figure I: An example question of CMExam. Abbreviations: Circulatory System Diseases (Circ), Internal Medicine (IM), Clinical Medicine (ClinMed), Medical Fundamentals (MedFund).
have limitations that hinder comprehensive assessment of LLM performance (Nori et al. 2023 Many datasets are insufficient in terms of size and diversity, preventing a thorough evaluation of LLM capabilities. Furthermore, most datasets primarily focus text generation tasks rather than utilizing clear choice evaluations, impeding objective and quantitative measurement LLM performance. Additionally, majority of these datasets (Li et al. [0223] Pal et al. 2022 Zhu al. 2020) are sourced from online forums and consumer feedback, which could suffer from significant bias and error. These challenges particularly amplified in English languages, such Chinese, due to the pervasive inequality in language resources that exists in the NLP field (Bird) 2020 Zeng et al. 2022 Fang
al. 2023). ). Overall, due to the lack of qualified evaluation datasets, the strengths and weaknesses of LLMMs in the medical field have not been fully studied.
In response, present novel dataset called CMExam to overcome these challenges and benchmark LLM performance. CMExam is sourced from authentic medical licensing exams. It contains more than 60K questions and utilizes the multiple-choice question format to allow standardized and objective evaluations. Questions in CMExam have corresponding solution explanations that can be used to test LLM's 's reasoning ability in open-ended manner. To offer diverse perspectives for measuring LLM performance in the medical field, we created five additional question -iie annotation dimensions based on authenticated resources and objective metries. To reduce the substantial time and labor costs associated with annotating large-scale datasets, we propose an innovative strategy called GPT- Ass-stetteed Annotation. This approach harnessed the power of GPT-4 to automate the initial annotation process. Subsequently, the annotated data underwent meticulous review and manual verification conducted by two medical professionals. Figure shows an example question from CMExam and the annotation process.
Furthermore, benchmark the performance of general domain LLMs and medical domain LLMs on answer prediction (multiple-choice) and answer reasoning (open-ended) tasks of CMExam. This comprehensive assessment aims to highlight the strengths and weaknesses of various approaches in Chinese medical QA, with a focus on LLMs. The main findings of this benchmark are as follows:
GPT-4 [OpenAl 2023) demonstrates impressive zero-shot performance on the answer prediction task compared other models, though still significantly lagging behind human performance. GPT-3.5 (Brown et al.) 2020) and GPT-4 generated reasonable answers on the answer reasoning task despite low BLEU and ROUGE scores. This is because they tended to generate short answers with reasonable quality.
Existing medical domain LLMs, such as Huatuo Li et al. 2023) and DoctorGLM (Xiong et al. 2023), exhibit poor zero-shot performance on both tasks, indicating their limited coverage of medical knowledge and substantial room for improvement.
Lightweight LLMs (c.g., ChatGLM (Du et al. 2022) fine-tuned CMExam with supervision chieve performance close to GPT-3.5 on the answer prediction task. They also significantly outperform GPT-3.5 and GPT-4 on the reasoning task while having only 3% of the parameters of GPT-3.5.
In summary, this study provides valuable insights into the performance of LLMs in medical contexts from multiple perspectives, benefiting both the artificial intelligence research community and the medical research community. Our findings contribute to a deeper understanding of the capabilities and limitations of L.LMs in the medical domain. Additionally, the CMExam dataset and benchmark introduced in this study serve valuable resources to inspire researchers to explore more effective

---

ways of integrating medical knowledge into LLMs, ultimately enhancing their performance in medical applications.
Table 1: A review of medical QA datasets. indicates availability of additional annotations with authoritative references, indicates availability of benchmarks, and t indicates datasets with more than 50K questions
Language
Source Type
Question Type
Multiple Choice
Open-ended
LiveQA-Med (Abacha 2017
CliCR' (Suster and Dac lemans
2018
HealthQA [ZNU er 2019
MEDIQA (sbaccia er Mr 20 9b
Consumer Questions
MedMOQA Pal
emrQA (Pampun et al 2018
MedQuaD [Bea
Demner- Fushrra
2019
English
Medication [Alacaa et di
2015
MEDIQA AADIQAAAASS.. (Savery
2020
MASH-QA (Zh ot 2020
MEDQA Jin et al. 2021
MMLU11 Hondryeks of 2020
BioASQ (Krithm el al. 2023
Research, Books, Exams
MedMOQA et 2022
MultMMddQA (Singhal
2022
MultiMedQA (Singhal cr
wehMedQQ" He al. 2019
eMedQA-v1100 of
Zhann
Consumer Questions
cMcdQA-12.0f 0f Zhane
Chinese
ChiMed TTan ar 2719
Huntup 26M
2123
MLEC-QA Zeng al. 2025a
MLEC-QA ZZng ct al. 2025g
Research, Books, Exams
CMExam "aaamm"ittuss ((ous)
CMExam ((our))
2
Related Work
Medical Question-Answering Datasets Table presents summary of medical QA datasets published after 2017. In particular, we focus on categorizing the data source and question types of the different datasets. Most existing medical QA datasets adopt open-ended format, primarily because they were constructed directly from consumer questions and answers from doctors. However, multiple-choice and fill-intthe-blnnk questions provide
standardized and objective evaluation, and only a small portion of medical QA datasets have adopted these formats. Notable examples
include CliCR (Suster and Daelemans, 2018), MEDQA (Jin et al. 2021), MMLU (Hendrycks et al. 2020), MLEC-QA (Zeng al. (023a), and MedMCQA (Pal 2022)) Note that the multiple- choice questions in MultiMedQA (Singhal et al. 2022 MMLU.
come from MEDQA, MedMCQA, and
Data source types generally determine the reliability of dataset. Consumer questions collected from web sources require human review ensure the correctness of the answers. As datasets grow in size, quality control becomes increasingly challenging (Li et al., (202).. In contrast, datasets built from case reports (c.g., CliCR), research literature (e.g.. BioAsq (Krithara et al.) 2023 )) medical books, exams, and related practices (e.g., MMLU and MedMCQA) are often more reliable.
From Table I we observe that there are few datasets based on multiple- questions from authoritative sources. This characteristic distinguishes CMExam from the MLEC-QA dataset, which is also derived from the Chinese National Medical Licensing Examination. In essence, CMExam has been meticulously crafted as foundational benchmark dataset. introduces question explanations for reasoning ability inspection, incorporates expansive annotation facets with authoritative references, and includes question- medical competencies and difficulty ratings calculated from human performance. These features make CMExam indispensable resource for authoritative LLM performance assessment and meaningful human-machine comparisons. Table 2 presents a list of innovations and characteristics of CMExam, which are discussed in detail in the following sections.
Other Benchmark Datasets of Large Language Models The assessment of LLMs has witnessed significant progress, with the introduction of diverse benchmarks that evaluate different dimensions across multiple languages, models and tasks (Liu et al. 2023b Zhou et al. 2023a). Many datasets focus on assessing natural language understanding and reasoning capabilities of L.MSs. RACE (Lai al. 2017) includes English exams for Chinese middle and high school students. TriviaQA (Joshi al. 2017) consists of question-answer pairs authored by trivia enthusiasts. DROP (Dua et al. 2019)
3

---

Table 2: Additional annotations of CMExam.
Annolation Content
References
Unique values
Disease Groups
The 1Ith rovision of ICD-11
27
Clinical Departments
The Directory of Medical Institution Diagnostic and Therapeutic Calegories (DMIDTC)
36
Medical Disciplines
List Graduate Education Disciplinary Majors (2022)
Medical Competencies
Medical Professionals
4
Difficulty Level
Human Performance
5
evaluates reading comprehension with discrete reasoning and arithmetic components. GLUE (Wag et a 2018) encompasses four existing NLU tasks, while SuperGLUE (Wang et al., 2019 extends it with a more challenging benchmark of eight language understanding tasks. Other datasets, such as HellaSwag (Zellers et al. (019) and WinoGrande (Sakaguchi et al.) (02T), focus on commonsense reasoning. TruthfulQA (Lin et al. 2 includes health, law, finance, and politics, to assess L.MS' ability to mimic human falsehoods, while MMCU (Zeng) 2023) covers medical, legal, psychology, and education to evaluate multitask Chinese understanding. In addition to language understanding and reasoning, several datasets focus specific subjects and topics, such as Python coding tasks (Chen et al., (221), middle school mathematics questions (Cobbe et al., 202I) and defending against attacks (Yi et al. [2023 Xie et al.] [023]] Pi et al. 2024). Notably, both C-Eval (Huang et al. (023) and M3KE (Liu et al. (0233) serve as multi-level multi-subject evaluation benchmarks, making them particularly suitable for assessing the capabilities of LLMs across multiple domains.
3 The CMExam Dataset
Data Collection and Pre-processing CMExam comprises authentic past licensed physician exams in the Chinese National Medical Licensing Examination (CNMLE) collected from the Internet. The CNMLE, also known as the Physician Qualification Examination, is a standardized exam that assesses applicants' medical knowledge and skills in China. It includes a written test with multiple-choice questions covering various medical subjects and a clinical skills assessment simulating patient diag- nosis and treatment. We excluded questions that rely on non-textual information, including questions with external information such as images and tables, and questions with keywords "graph" and "table". Duplicate questions were removed from the dataset. In total, 96, 161 questions, 68, 119 of which were retained after pre- processiigg. The dataset was then randomly split into training/developmentee with a ratio of 8:1:1. Each question in the dataset is associated with an ID, five candidate answers, and correct answer. 85.24% of questions have brief solution explanations and questions in the test set contain additional annotations.
sets
Data Annotation CMExam provides a comprehensive analysis of LLM performance through five additional annotation dimensions. The first dimension involves disease groups based on the 1 11th revision of the International Classification of Diseases (ICD-11) (World Health Organization (WHO), 202T). ICD-11 is a globally recognized standard classification system for documenting and categorizing health conditions, consisting of 27 major disease groups. The second dimension comprises 36 clinical departments derived from the Directory of Medical Institution Diagnostic and Therapeutic Categories (DMIDTC) published by the National Health Commission of China. DMIDTC an authoritative guide used for categorizing and naming diagnostic and therapeutic subjects within healthcare institutes. In cases where the question cannot be successfully classified by ICD-11 or DMIDTC, the annotation marked as "N/A". The third dimension refers to medical disciplines, which are categorized based on the List of Graduate Education Disciplinary Majors (2022) published by the Ministry of Education of the People's s Republic of China" This dimension encompasses seven categories representing study majors used in universities. The fourth dimension was created by two medical professionals within the team to assess the primary medical competency tested by each associated question. It consists of four categories. The fifth dimension represents five potential difficulty levels of each question. determined by analyzing the correctness rate observed in human performance data collected alongside the questions. For detailed information on these additional annotations including their potential values, please refer to Table 9 12 proposed GPT-Assisted Annotation strategy is shown in supplementary materials.
3
And our
shtml
http: http://www.me..ovvcclte http:///ww....eeoooooooooooooooo http://wwww.....ooooooooooooottp

---

Dataset Characteristies The CMExam dataset has several advantages over previous medical QA datasets regarding: l)Reliability and Authenticity: CMExam is sourced exclusively from the CNMLE that undergoes rigorous review and validation processes, ensuring its accuracy and adherence to established medical standards. 2) Standardization and Comprehensiveness: CMExam includes both multiple-choice questions that ensure fair and objective evaluations of models' performance and question-wiz open-ended reasoning that allows in-depth analysis and assessment model reasoning abilities and comprehension. Despite the inherent absence of explanations within the CNMLE, we cross-referenced exam questions with solutions offered by diverse online medical examination preparation platforms, effectively enhancing the dataset's s informational depth. CMExam reflects the comprehensive coverage of medical knowledge and reasoning required clinical practice, as is sourced from carefully designed national medical exams. The inclusion of five additional annotation dimensions enhances the dataset's rigor and offers valuable insights for in-depth evaluation and analysis. 3) Scale: CMExam consists of over 60K high-quality questions, providing large and reliable dataset.
Data Statistics The dataset has total of 68,119 questions, with 65,950 answers being single-choice and 2, 169 being multiple-choice, with a maximum of five answer choices. Among all questions, 85.24% have associated solution explanations Figure shows additional statistics visualization and more basic statistics CMExam can be seen in supplementary materials. Within the test set, 4,493 questions (65.97%) have corresponding disease group annotations. The most prevalent disease group Traditional Medicine Disease Patterns (TMDP), followed by Digestive System Diseases, Certain Infectious (Digest) and Parasitic Discases (InfDis), Endocrine, Nutritional, Metabolic Diseases (Endo), and Circulatory System Diseases (Circ). For the associated clinical department annotations, 4,965 questions (72.90%) have been assigned values. The most frequently represented clinical departments are Internal Medicine (IM) and Traditional Chinese Medicine (TCM), with Dentistry (Dent) and Surgery (Surg) following closely. Every question in the discipline, where Clinical Medicine (ClinMed) comprises the largest proportion. Additionally, each question has been categorized into competency area, with Medical Fundamentals (MedFund) being the predominant category. The difficulty levels of the questions align with common exam patterns, with greater number of easy questions and smaller number hard questions.
has been labeled with
ICD-11 Group
Clinical department
1000
750
500
TMPP
IrDs
Enso
Crn
Neo
GU
Farbor
Relp
Nours
MOK-
Pay
Blood
CBST
Inmuno
Blin
Puri
Dac
Voarn
Sym-
Est
SpeePurp
Bax-
Donk
0B0
drinig
Dorn
Sponabe
Discipline
Competency
Difficulty level
TCM
Ex. difficult
ClinMed
MedFund
PHL
2 2%
Difficul:
Essy
25 9% (17643)
(5518) BB)
TCPham
13 8% (934)
10.3% (7016)
Moderale
11.2% (7829)
Treat
(23092)
22. 22.4% (15259)
(2896)
(1306)
17.7%
14.95 14.9% (10150)
(12067)
16.4% (11172)
Pharm
31. ...2% (21253)
(1631)
РН&PM
Diag
Managable
Figure Additional CMExam statisties, For the question length distribution subplot, only the portion within IQR is shown.
cn/
http://www.jsnyinggiie
https: htttps///wvre ://ww///wwww/ http:/wwww.j.iii....
com. cn/
5

---

4 Benchmarks
4.1 Baselines, Settings, and Metries
Model Selection The LLMS we henchmarked on the CMExam can be divided into two groups based
on domains: 1) General Domain LLMs: This group comprises GPT3.5/4 (Brown al. 2020 OpenAI
)023), ChatGLM (Du et al. 2022; Zeng et al. 2023b), LLaMA (Tourron et al., 2023)) ) Alpaca (Taori
al. 2023), and Vieuna (Chiang al. 2023). D. These models are general-purpose language models trained on a massive amount of general - -purpose corpora; 2) Medical Domain LLMs: This group can
be further divided into two subgroups. The first subgroup consists of representative LLMss specifically
designed for the medical domain, including DoctorGLM (Xiong et al. (023) and Huatuo (Wang et al.) 2023). DoctorGLM is a healthcare-specifie language model initiallized with ChatGLM-6B parameters and further fine-tuned on Chinese medical dialogues extracted from ChatGPT. Huatuo, on the other hand, is a snooledge-enhaneee model, which builds upon the ILLMA architecture and is additionally supervised-fing sordddfiettunnee with knowledge- baaeeee--adddd--e instruction data harvested from the Chinese medical knowledge graph (CMeKG). The second subgroup comprises medical LLMs that were constructed through supervised fine-tuning of LLMs using the CMExam training set. This subgroup includes models fine-tuned on BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019), PrompiCLUE (Zhang and Xu 2022 (T5-based), BART (Shao et al., 2021). Huatuo, ChatGLM, LLaMA, Alpaca, and Vicuna.
Human Performance To effectively gauge the medical proficiency of LLMs, incorporating a measure of human performance into the benchmarking process of paramount importance. Therefore, during data collection, we preserved the accuracy of human responses for each question. Human performance estimated by computing a weighted average of response accuracy within each dimension, with weights determined by the number of respondents. This design ensures robust comparison of LLMs' performance relative to human capabilities, particularly when larger respondent samples contribute to question's 's accuracy.
Experimental Setting For GPT models, we leveraged OPENAI's API to access the GPT-3.5-ture and GPT--4033 14 models, given that their open- open-source variants are currently unavailable. The LLaMA, Alpaca, and Vicuna models were used in their respective 7B versions, while ChatGLM was evaluated using its publicly accessible 6B version. Additionally, we performed fine-tuning on open-source models using the CMExam dataset. We used P-tuning V2 (Liu et al. 202T) for ChattGLM-66, with the length of prefix tokens set to 128, and the learning rate set to 2e-2, 2, LoRA (Hu et al.) (021] for LLaMÅ, Alpaca, Vicuna, and Huatuo models, with the rank set to 8, alpha set to 16, and dropout at 0.05. For BERT models, we followed the fine-tuning methods outlined in (Devlin et al. (211)), with batch size set to 16, learning rate set to 2e-4, hidden dropout probability set to 0.4, and maximum input length set to 192. The fine-tuning processes for all models except BERT involved a batch size of 64, a maximum input length, and a target length of 256. All fine-tuning was performed using NVIDIA V100 GPUs for 10 epochs.
Metries We assess model performance on multiple choice questions using accuracy and weighted F1 score. These metrics are commonly employed in information retrieval and question-answering tasks to evaluate model performance. For the open-ended solution explanations of CMExam, BLEU (Papineni et al.) (002) and ROUGE (Lin and Hovy, (003) were used to evaluate the discrepancy between model-generated explanations and ground truth.
4.2 Results and Analysis
Overall Comparison We first assessed the performance of general domain L.MSs and medical domain LLMs for answer prediction and reasoning tasks. The results are displayed Table For the answer prediction task, GPT-4 significantly outperforms other methods, demonstrating a zero-shot performance with an accuracy of 61.6% and an Fl score of 0.617. While a performance gap still exists when compared to human performance (which stands at 71.6% accuracy), it's noteworthy that this gap has been greatly reduced from what was observed with GPT-3.5. Among lightweight, general domain LLMs, ChatGLM outperforms LLaMA, Alpaca, and Vicuna, likely attributable to their limited coverage of the Chinese corpus. This restriction seemingly hampers their ability to provide accurate responses to CMExam queries. Furthermore, noticeable deficiency in zero- shot performance evident in lightweight medical domain L.MMs such as Huatuo, owing to their restricted medical corpus diversity, which hampers the acquisition of broad medical knowledge and

---

Table Overall comparison on CMExam dataset. We bold the best result and underline the second
best result.
Model type
Models
size
Prodiction
Reasoning
Acc (16)
F1 (98)
BLEC-1
BLEU-4
KOUGE-1
ROUGE-2
ROUGE-L
G-r--.S-turbo
175B
46.4=0.5
46.1402
3.56#3577
14950.51
33.80sa0.9 19
16.39e0.1
14.83+0.13
GPT-
616:0.1
61.750.1
0.17:0.00
0.0600.00
29.74a0.09
14.852004
11.51+0.00
Domain
CharGLM
LLaMA
36.--000
0.4400
25.7+0.1
0.3:0.
16.51-0.88
11,99=0.03
5.000006
5,70=00
35.18-0.11
27.33+0006
15,75+005
11,88+003
10,78+0004
17,0940.13
Vicuna
7B
5.060.0
4.8.0.1
20.15:0.01
9.26:0001
38.436002
16.90+001
15.3340.01
Alpuca
7B
8.510.0
8.420.0
4.7540.00
2,5020.00
22.52:0.00
9,5410000
8:4010.00
Huatuo
7B
12.900.0
7.00.00
0.21+0.00
0.12+0000
25,11+008
11,56+0.04
9.7540.02
MedAlpaca
200000.0
10.7+00
0.00+0.00
0.000.000
1,90=0.00
0.04+0.00
0.52+0.03
DoctorGLM
9,43=0.09
2,650000
21.115000
6.8660011
PrompiCCLUE-buse UEEaase
0.1B
18.75.0.08
6.6500.5
40.8810.11
21,9010111
18,110011
Medical Domain
Bart-hise- hase--hinnee-CMaaa
0.18
23 00:0000..0 40
10.35+0.16
44.33+0
24,240009
20
Bart large chinese- C CMExam
0.18
26.372:0.18
11.65:0.0
44.9220.12
24.3420.12
21.7550003
BERT-CMExaa
0.16
31.8:0.2
31.240.2
SUBERT&-CMEEsn
17.1
36.7004
MedAlpaca CMExam
30.5g0.1
30.420.1
9.7820.77
44.31
24.5540-43 27.05a0.50
Huatuo-CMENa CMExam
7B
28.6.0..5
29.3.0.2
29.0110.01
16.72.000
43.8510.24
25.3610022
21.:210.21
6B
45.2a1 4
31.10.0.23
18.94.6.12
43,94400.75 78
31.48
29.39.0.14
11 aMMA-MMFum
7R
18.3+0.5
20.600.5
29, 25-0 23
16.4640 10
45.88=0004
26.57+0.04
23, 03.31+0.02
Alpuca- CMExam
7B
21.1:0.6
21.91044
29. 19.57.0.10 10
16.10.0.1
45,1810
25.5310.15 18
22.9710.006
CMExam
27.3.0.5
2..3.
09.82.0
17.30s001
44,9810.16
26.2510.13
22.4410.09
Rundom
Random
3.1.0.2
3.1.0.3
Homan Perforance
Human volunteers
71.6
accurate interpretation of CMExam questions. Our findings suggest that finetuning models with CMExam enhance their performance. For instance, with an accuracy of 45.3%, C is comparable to GPT-3.5's performance, despite utilizzig only about 3% of the parameters employed by GPT-3.5. It is noteworthy that encoder-only LLMs, such as BERT and RoBERTa, remain a robust baseline for answer prediction tasks. Their performance can par with, or even exceed, that of certain decoder-only LLMs, such as LLaMA-CMExam and Alpaca-CMExam, despite having fewer parameters.
For the solution explanation task, we observe that GPT models performed poorly on the BLEU metric, likely due to their tendency to generate short explanations. However, they exhibited an advantage on the ROUGE metric. As DoctorGL.M is unable to return answer options according to the prompt, we only report its performance in the solution explanation task. Through finetuning, LLM was able to generate more reasonable explanations. For instance, ChatGLM-CMExam achieved scores of 31.10 and 18.94 on BLEU-1 and BLEU-4, respectively, and scores of 43.94, 31.48, and 29.39 on the ROUGE metrics.
Results by Disease Groups Drawing upon ICD-11 annotations (26 categories), we conducted an analysis of the performance several LLMS across various categories. To mitigate the potential impact of random variability resulting from the number of questions,
limited our analysis to
categories containing more than 100 questions. According to Table LLMs have uneven performance and significant gaps in knowledge. GPT-4's accuracy ranges from 74.4% for Neo to 44.3% for TCMDP, GPT-3.5's accuracy ranges from 63.9% for Neo to 31.0% for TCMDP and ChatGLMM- CMExam's accuracy ranges from 54.7% for Psy to 42.9% for RESP.
Results by Clinical Departments To compare model performance regarding the clinical department dimension (36 categories), we only analyzed categories with more than 50 questions to ensure result representativeness. Results presented in Table 5) highlight that the models show relatively high accuracy on questions associated with commonly encountered departments, such as Emergency Medicine (EM), Internal Medicine (IM) and Surgery (Surg). Their accuracy on questions associated with rarer departments, such Traditional Chinese Medicine (TCM). There is marked discrepancy in the average accuracy among different departments, with the highest being 50,9% and the lowest being only 13.9%. This observation suggests there are notable variations in medical knowledge and reasoning approaches among different departments. Consequently, it may be necessary to examine specific optimization strategies for different departments.
Results by Medical Disciplines Then, we evaluated LLM performance across seven medical disciplines. As depicted in T the performance of LLMs across disciplines such as Traditional Chinese Medicine (TCM), Traditional Chinese Pharmacy (TCPharm), and Pharmacy (Pharm) was notably subpar, with all accuracy rates falling below 42%. This pattern suggests potential deficiency
7

---

Table Comparing disease classifications.
Categories
GPT-4
GPT-3.5
ChatGLM
ChatGL.--CMExam
Average
Neo
74,4+2.2
63.9+1.
32.4+1.6
$1.940.2
55.640.8
Psy
74.0+0.7
62.0+1.7
33.3#1.3
54.720.8
56.040.9
Factors
70.0=1.0
57.5=14
28.0+1.1
51.1=1.4
51.6+0.5
MSK
65.920.8
53.8=0.8
29.2#0.4
53.5+0.0
50.6#0.4
GU
69.2#0.4
52.1=1.1
30.0#0.2
49.5+0,9
50.2+0.3
Inj
65.9+2.3
45.7=1.3
37.2+2.9
49.1=1.8
19.
Cire
68.810.3
49.320.7
30.9+0.7
47.020.3
49.0+0.2
Endo
70.6+1.1
49.4
25.520.8
46.120.4
47.940.2
Digest
67.0=1.0
48.8=1.4
26.2+0.7
49.4+1.1
47.8+0.4
InfDis
66,0+0.5
49.2+0.8
27.5+0.6
48.240.8
47.740.4
Neuro
64,4+1.2
48.7=3.1
28.6+0.4
1..3+1.3
46.7+1.1
OBST
63.5+0.3
45.0#2.4
25.7+0.9
49.420.3
45.9+0.5
BLOOD
69.420.3
45.3#1.4
18.9#1.6
43.3+0.7
44.2+0.4
Resp
62.7+0.8
44.3=1.4
24.5+0.3
42.9+0.0
43,620.7
N/A
60.0=0.1
46.8+0.3
24.9#0.2
42.5+0.1
43.5+0.1
TCMDP
44.320.9
31.020.6
24.240.4
47.920.0
36.940.6
Table 5: Comparing clinical department.
Categories
GPT-4
GPT-3.5
ChatGLM ChatGL.MCMMExam
Average
EM
67. 07.420.2
49.840.7
36.320.4
50.2+0,5
50.9
OBGYN
66.421.0
51.721.5
28.620.5
52.0=0.0
49.7+0.3
IM
70.2+0.6
51.8+0.8
26.0=1.1
47.9=0.9
49.0+1.0
ID
67.4+1.9
49.5+3.3
26.1
49.6st3.8
48.2+1.2
Surg
63.640.8
49.5
28.8
47.7+0.9
1.
ClinNutr
68.3+2.4
48.3+2.9
23.9=1.1
47.8+0.5
47.1
MedL_ahSel
69.2+0.6
48.3+2.0
29.0=1.5
40.8+0.6
46,8+0.2
Ped
64.520.0
47.241.4
26.7+2.1
41.925.5
45.1
N/A 62.640.2
48.6+1.1
24.620.4
44.320.9
45.0+1.0
Ophth
60.920.5
39.120.8
21.80088
54.020.2
44.020.8
OceMed
61.5+4.3
38.5&1.6
31.324.3
41.5+3.3
43.2+2.5
DENT
54,942.0
41.241.6
27.9=0.8
43.520,9
41.9+100
TCM
43.121.3
31.41.3
24.5+1.9
45.8+4.4
36.240.6
ENT
41.3+0.8
28.0#0.6
29.3+0.1
26.7+101
31.3+0.5
ICM
33.3+0.0
11.1#15.7
0.0+0.0
11.1+15.7
13.9+4.8
in the exposure of these models to data within these categories. Conversely, disciplines such as ClinMed and Ph&PM demonstrated higher accuracy rates, likely due the abundance of relevant data. The observed variability in performance across different disciplines underscores the distinctiveness of data characteristics and complexities inherent to each field, thereby advocating for discipline -speciff model optimizations and enhancements.
Results by Competencies Evaluations based medical competency
aimed at higher-level
understanding of model capability in solving medical problems. As indicated in Table 7 the lowest average accuracy across LLMs was observed within the domain of mastering Medical Fundamentals (MedFund), with a meager average score of 42. %. This result demonstrates that these models, predominantly trained on general textual data, have inadequate exposure to medical-specific data. While tie-ttning did provide some improvement, these models could benefit from additional medical scenario data to further augment their performance. is worth highlighting that the average accuracy in the domain of Public Health Laws and Ethics (PHL) was reasonably high, notably achieving an average of 47.6% In addition, the LLMs showcased their proficiency in accurate disease diagnosis.
Results by Question Difficulty To evaluate model performance tackling questions of varying levels of difficulty, we conducted experiments regarding the question difficutty dimension, which was calculated based on human exam-taker performance. As shown in Table there's 's an evident trend where model accuracies decrease as question complexity rises. This pattern suggests that more sophisticated questions demand an extensive knowledge base and complex reasoning, which are challenging for the LLMs, thus reflecting patterns observed in human performance.

---

Table 6: Comparing medical discipline.
Categories
GPT-4
GPT-3.5
ChatGLM
ChatGL.MCCMExmm
Average
ClinMed
67.920.1
51.420.4
27.340.3
48.940.4
48.840.7
PH&PM
68,2+0.4
52.7+1.7
26.2+0.3
47.3+1.0
48.640.5
ICWM
56.1+0.1
40.0=2.3
29.4+0.8
53.6+0.7
44.8+0.9
Dent
59.5+0.7
43.9=1.9
28.5+1.1
45.3+0.6
44.3+0.3
Pharm
61.1#0.4
46.3+0.5
23.2+0.2
37.0#0.1
41,9+0.3
TCM
53.5+0.4
35.9=0.2
24.1#0.3
49.1=0.0
40.6+1.1
TCPharm
45.4+1.2
35.620.1
24.1+1.0
43.120.4
37.1
Table 7: Comparing LLMs' competencies.
Categories
GPT-4
GPT-3.5
ChatGLM ChatGLM-CMExxm
Average
Diag
70.1+5.5
50.9+2.1
30,9+2.8
51.6+1.0
50.9+1.4
PHL
64.2.0.7
50,0#0.5
26.8+0.3
49.6+0,1
47.6+0.3
Treat
56,5+0.5
43.0=1.1
25.740.2
47.4+0.6
43,240.8
MeFund
58.3+0.3
44.6=0.7
23.9+0.5
41.6+0.4
42.1+0.9
N/A
54.820.2
30.420.4
23.720.1
38.540.2
36.940.3
Table 8: Results by question difficulty.
Categories
GPT-4
GPT-3.5
ChatGLM ChatGIMMCMMExm
Average
Easy
74.6#0.1
58.5=0.6
31.4=0.2
61.5+0.3
56.5+0.4
Manageable
63.9=0.2
47.4+0.7
25.9+0.5
46.1+0.3
45.8
Moderate
51.3+0.6
36.8+0.8
23.0=0.4
34.5+0.6
36.4+0.7
Difficult
36.4:0.9
26.2+0.7
18.9+0.5
24.3+0.9
26.5+0.6
Extremely difficult
27.2+1.0
21.4=2.2
15.8+1.0
12.2+1.1
19.1+1.1
Results by Question Length Finally, to investigate if model performance is associated with input lengths, we compared their performance regarding question lengths. Figure illusttrate that Large Language Models (LLLMs) generally show higher accuracy with problem lengths be- tween 60 and 90. However, their performance seems to falter with problems that are either too short overly long. Additionally, we noticed that the effect of ques- tion length varies across different LLMs. For instance, GPT models tend to incrementally improve as the problem length expands, performing optimally within the 50 to 90 range. Conversely, ChatGLM CMExam's fluctuates noticeably with varying lengths, and it tends to fall short compared to GPT models when addressing longer problems.
20%
697-
GBT 0.D
ChatGLM-CMEXam' performance
Quation Langt
Figure 3: Results stratified by question length.
5 Conclusion and Discussions
In this work, developed CMExam, dataset sourced from the stringent Chinese National Medical Licensing Examination, featuring 60,000+ multiple-choice questions, with detailed explanations. CMExam ensures reliability, validity, and adherence medical standards. also demonstrates the practicality of employing GPT-4 to automate the annotation process, which strikes a harmonious balance between efficiency and cost-effectiveness while maintaining the desired level of accuracy and reliability of the annotation. Utilizing this large and reliable corpus, we tested several LLMs for answer selection and reasoning tasks. A performance gap was observed between LLMs and human experts, signaling the need for additional LLM research. CMExam's standardization and comprehensiveness also ensure objective evaluations of models while enabling in-depth analysis of their reasoning capabilities. The questions cover wide spectrum of medical knowledge, augmented with five additional annotation dimensions for rigorous evaluation. This study aims to spur further exploration of LLMs in medicine by providing comprehensive benchmark for their evaluation.

---

We anticipate CMExam to contribute significantly to future advancements of LLMs, particularly in handling medical question-ansee answering tasks.
Limitations Firstly, while CMExam derived from meticulously designed medical examinations, our process of excluding questions requiring non-textual information may inadvertently affect the balance of the remaining questions, potentially introducing unexpected biases. It is critical to acknowledge this aspect while interpreting any findings analyses conducted using this dataset. Furthermore, the current BLEU and ROUGE metrics primarily evaluate the explanation task, but these measures are insufficient for assessing the reasonableness of the answer. In future work, we will incorporate human evaluation to provide a more comprehensive assessment of the models.
Ethics CMExam is dataset derived from the Chinese National Medical Licensing Examination, which aligns with numerous datasets containing similar National Medical Licensing Examinations
(Zeng al. 2023a Hendryeks al., [0200] Jin al.) 2021 Pal et al.. 2022 Singhal al.. (022).
We have ensured adherence to applicable legal and ethical guidelines during data collection and
use. The authenticity and accuracy of the exam questions have been thoroughly verified, providing reliable basis for evaluating LLMs. Please note that the CMExam dataset is intended for academic and research purposes only. Any commercial use or other misuse that deviates from this purpose is expressly prohibited. We urge all users to respect this stipulation in the interest of maintaining the integrity and ethical use of this valuable resource.
Societal Impacts While CMExam aims to enhance LLM evaluations in the medical field, it should not be misused for assessing individual medical competence for patient diagnosis. Conclusions drawn from models trained on this dataset should acknowledge its limitations, especially given its single source and the specific context of the CNMLE. The use of this dataset should strictly be limited to research purposes to avoid potential misuse.
References
Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner-Fushman 2017. Overview of the medical question answering task at TREC 2017 LiveQA.. In TREC. 1-12.
Asma Ben Abacha, Yassine Mrabet, Mark Sharp. Travis R Goodwin, Sonya E Shooshan, and Dina Demner-Fushman 2019a. Bridging the Gap Between Consumers' Medication Questions and Trusted Answers.. In MedInfo. 25-29.
Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. 2019b. Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering, In Proceedings of the 18th BioNLP Workshop and Shared Task. 370-379.
Asma Ben Abacha and Dina Bemner-Fushman 2019. A question-entailmen approach to question answering. BMC bioinformatics 20, (2019), 1-23.
Steven Bird. 2020. Decolonising speech and language technology. In Proceedings of the 28th International Conference on Computational Linguistics. 3504-3519.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint 20X.::::107.033: (2021).
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* * ChatGPT Quality. https //1msys. hrtpg///.rrr/booog/22-/-/- 03- 3iivviina/
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1416 (2021).
10

---

Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- tre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv fbs//180000405 (2019).
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang. and Jie Tang. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). 320-335.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint 1903v /016// (2019).
Changchang Fang, Jitao Ling, Jing Zhou, Yue Wang, Xiaolin Liu, Yuan Jiang, Yifan Wu, Yixuan Chen, Zhichen Zhu, Jianyong Ma, et al. 2023. How does ChatGPT4 preform on Non-English National Medical Licensing Examination? An Evaluation in Chinese Language. medRxiv (2023), 2023-05.
Junqing He, Mingming Fu, and Manshu Tu. 2019. Applying deep matching networks Chinese medical question answering: study and a dataset. BMC medical informatics and decision making 19, 2 (2019), 91-100.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardr. 2020. Measuring massive multitask language understanding. arXiv preprint 0:Xx::2009.03300 (2020).
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. ArXiv 0bs/2100.0966. (2021).
Yining Hua, Fenglin Liu, Kailai Yang. Zehan Li, Yi-han Sheu, Peilin Zhou, Lauren V Moran, Sophia Ananiadou, and Andrew Beam. 2024. Large Language Models in Mental Health Care: Scoping Review. arXiv preprint arXiv:2401.0223 02984 (2024).
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023. C-eval: A multi-level nulti-discipline chinese evaluation suite for foundation models. arXiv preprint a305v: 003222 (2023).
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits 2021. What disease does this patient have? a scage open domain question answering dataset from medical exams. Applied Sciences 11, 14 (2021), 6421.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.0335 005503551 (2017).
Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. BioASQ-QA: A manually curated corpus for Biomedical Question Answering. Scientific Data 10. (2023), 170.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint 1004::1704.0400 (2017).
Jianquan Li, Xidong Wang, Xiangbo Wu, Zhiyi Zhang, Xiaolong Xu, Jie Fu, Prayag Tiwari, Xiang Wan, and Benyou Wang. 2023. Huatuo-26M, a Large-scale Chinese Medical QA Dataset. arXiv preprint arXiv:2305.0152 (2023).
Chin-Yew Lin and Eduard H. Hovy. 2003. Automatic Evaluation of Summaries Using N-gram Co- occurrence Statistics. In North American Chapter of the Association for Computaaiion Linguistics.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falschoods. arXiv preprint arXiv:2109.07558 (2021).

---

Chuang Liu, Renren Jin, Yuqi Ren, Linhao Yu, Tianyu Dong, Xiaohan Peng, Shuting Zhang, Jianxiang Peng, Peiyi Zhang, Qingqing Lyu. al. 2023a. M3KE: A Massive Multi-Leve Multi- Subject Knowledge Evaluation Benchmark for Chinese Large Language Models. arXiv preprint arXiv:2305. 10263 (2023).
Junling Liu, Chao Liu, Peilin Zhou, Renjie Lv, Kang Zhou, and Yan Zhang. 2023b. Is chatgpt good recommender? preliminary study. arXiv preprint arXiv:2304. 20t.:3304400/4992 (2023).
Junling Liu, Chao Liu, Peilin Zhou, Qichen Ye, Dading Chong, Kang Zhou, Yueqi Xie, Yuwei Cao, Shoujin Wang, Chenyu You, et al. 2023c. Limrec: Benchmarking large language models on recommendation task. arXiv preprint
12241 (2023).
Junling Liu, Ziming Wang, Qichen Ye, Dading Chong, Peilin Zhou, and Yining Hua. 2023d. Qilin- med-vl: Towards chinese large vision-language model for general healthcare. arXiv preprint MvXiv:2310.17990 (2023).
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021. P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks, ArXiv abs/2110.07660 (2021).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11699 (2019). arXiv:1907.11692 attp://arxia. org/
abs abs/1907 11692
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 medical challenge problems. arXiv preprint arX1v:2303. (2023).
OpenAL. 2023. GPT-4 Technical Report. ArXiv abs/2303.08774 (2023).
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedMCQA: A large- scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning. PMLR, 248-260.
Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrqa: A large corpus for question answering on electronic medical records. arXiv preprint MXXv:1809.0075: (2018).
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Annual Meeting of the Association for Computational
Linguistics.
Renjie Pi, Tianyang Han, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, and Tong Zhang. 2024. ALLMM-Protector Ensuring MLLM's Safety without Hurting Performance, arXiv preprint arXiv:2401.000 0229000226 (2024).
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485-5551.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Commun. ACM 64, 9 (2021), 90-1
Max Savery, Asma Ben Abacha, Soumya Gayen, and Dina Demner-Fushman, 2020. Question-driver summarization of answers to consumer health questions. Scientific Data 7, (2020), 322.
Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Fei Yang, Li Zhe, Hujun Bao, and Xipeng Qiu. 2021. CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation. arXiv preprint 100.005299 (2021).
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022. Large Language Models Encode Clinical Knowledge. arXiv preprint arXiv:2212. (2022).
12

---

Simon Suster and Walter Daelemans. 2018. CliCR: a dataset of clinical case reports for machine reading comprehension. arXiv preprint arXiv.1003.0972) (2018).
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An instruction-folloiin LLaMA model. https: //gtttubbbbbbt... cow/tatsu sab/staaford. alpaca
Yuanhe Tian, Weicheng Ma, Fei Xia, and Yan Song. 2019. ChiMed: A Chinese medical corpus for question answering. In Proceedings of the 18th BioNLP Workshop and Shared Task. 250-260.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. ArXiv abs/2302.1397 (2023).
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances neural information processing systems 32 (2019).
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint (eXXvv:804.004444 (2018).
Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. 2023.
HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge. 2XXi:2334.06675 [cs.CL]
World Health Organization (WHO). 2019/2021. International Classification of Diseases, Eleventh Revision (ICD-11). https://ict who. Int/browse11 Licensed under Creative Commons 3.0 IGO licence (CC BY-ND 3.0 IGO).
AttribtiionNooDeriveee
Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending ChatGPT against jailbreak attack via self-reminders. Nature Machine Intelligence (2023), 1-11.
Honglin Xiong. Sheng Wang, Yitao Zhu, Zihao Zhao Yuxiao Liu, Linlin Huang, Qian Wang, and Dinggang Shen. 2023. DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task. ArXiv abs/2304.01097 (2023).
Qichen Ye, Junling Liu, Dading Chong, Peilin Zhou, Yining Hua, and Andrew Liu. 2023. Qilin- med: Multi-stage knowledge injection advanced medical large language model. arXiv preprint arXiv:2310.0P 09089009999000 (2023).
Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2023. Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models. arXiv preprint wxXiv:2312.14199 (2023).
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can machine really finish your sentence? arXiv preprint arXiv:1905. 07830 (2019).
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang. Yuxiao Dong, and Jie Tang. 2023b. GLM-130B: An Open Bilingual Pre-trained Model. The Eleventh International Conference on Learning Representations (ICLR).
htttps://openreviss wtt.lewww.eeeeeeeeeeeeee
Hui Zeng. 2023. Measuring Massive Multitask Chinese Understanding. arXiv preprint arXiv:2304.1290 12986 (2023).
Qingcheng Zeng, Lucas Garay, Peilin Zhou, Dading Chong, Yining Hua, Jiageng Wu, Yikang Pan, Han Zhou, Rob Voigtand, and Jie Yang. 2022. GreenPLM: Cross-Linguss Transfer of Monolingual Preined Language Models at Almost No Cost. the 32nd International Joint Conference on Artificial Intelligence (2022).
13

---

Qingcheng Zeng, Lucas Garay, Peilin Zhou, Dading Chong, Yining Hua, Jiageng Wu, Yikang Pan, Han Zhou, and Jie Yang. 2023a. greenPLM: Cross-Lingual Transfer of Monolingual Pre- Tr-TTraiee Language Models at Almost No Cost. In Proceedings of the 2023 Conference International Joint Conference on Artificial Intelligence (Al and Social Good Track).
Sheng Zhang, Xin Zhang, Hui Wang. Jiajun Cheng, Pei Li, and Zhaoyun Ding. 2017. Chinese medical question answer matching using end-to-end character-level multi-scale CNNs. Applied Sciences 7, (2017), 767.
Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. 2018. Multi-scale attentive interaction networks for chinese medical question answer selection. JEEE Access 6 (2018), 74061-74071.
Xuanwei Zhang and Liang Xu. 2022. PrompiCLUE: A zero-shot learning model that supports full Chinese tasks. https://gitthbb com/ clue ai/PromptCLUE
Hongjian Zhou, Boyang Gu, Xinyu Zou, Yiru Li, Sam S Chen, Peilin Zhou, Junling Liu, Yining Hua, Chengfeng Mao, Xian Wu, et al. 2023b. A survey of large language models in medicine: Progress, application, and challenge. arXiv preprint arXiv:2311.055
(2023).
Peilin Zhou, Meng Cao. You-Liang Huang. Qichen Ye, Peiyan Zhang, Junling Liu, Yueqi Xie, Yining Hua, and Jaeboum Kim. 2023a. Exploring recommendation capabilities of gpt-4v (ision): A preliminary case study. arXiv preprint arXiv:2311.04199 (2023).
Peilin Zhou, Zeqiang Wang, Dading Chong, Zhijiang Guo, Yining Hua, Zichang Su, Zhiyang Teng, Jiageng Wu, and Jie Yang. 2022. METS-CoV Dataset of Medical Entity and Targeted Sentiment on COVID-19 Related Tweets. In Advances in Neural Information Processing Systems, Vol. 35. 21916-21932.
Ming Zhu, Aman Ahuja, Da-Cheng Juan, Wei Wei, and Chandan K Reddy, 2020. Question answering with long multiple-span answers. In Findings of the Association for Computational Linguistics: EMNLP 2020. 3840-3849.
Ming Zhu, Aman Ahuja, Wei Wei, and Chandan Reddy. 2019. A hierarchical attention retrieval model for healthcare question answering. In The World Wide Web Conference.
14

---

A Appendix
A.1 Abbreviations, Full Names, and Translations of Additional Annotations
This section presents four tables of additional annotations that contain translation. It showcases abbreviations, full English names, and Chinese names for each group in each annotation dimension. Table showeass all disease groups included in the 1 11th revision of the International Classification of Diseases (ICD-11). We present the disease group in the same order found on the official website. Table 12 offers a classification of 36 clinical departments derived from the Directory of Medical Institution Diagnostic and Therapeutic Categories. Table T0] presents a breakdown of medical disciplines based on the List of Graduate Education Disciplinary Majors published by the Ministry of Education of the People's 's Republic of China. This categorization comprises seven study majors used in universities. Table IT provides all groups of areas of medical competency assessed in Chinese medical licensing exams.
Table 9: ICD-11 Groups
Code
Abbreviation
Full English Name
Chinese Name
latDis
Cestain infectious or parasitic discases Neoplasms
Neo
Blood
Diseases of the blood or blood forming oegans
Irmmune
Discases of immune system
05
Endo
Endocrine, nutritional or metabolic discases
Psy
Montal, behavioural aeuredovcopmental disorders
07
Sleep
Sleep-wake disorders
RESIF, T2 R
Neuro
Discases of the nervous system
Vision
Discases of the visual system
21, OZ K6
10
Ear
Diseases of the car or mastoid process
H.t: 9259E
Cire
Diseases of circulatory system
12
Resp
Discascs of the respiratory system
05-0 #6 1K
Digest
Diseases digestive system
(P. 36 EE
14
Skin
Diseases of the skin
Hig DRcolEsP
MSK
Diseases musculoskeletal system connective tissue
30L 5% 2
16
GU
Discases of the genitourinary system
Sex
Conditions related to sexual bealth
18
OBST
Pregnuncy. childbirth the puerperium
19
Peri
Certain conditions originsting in the perinatal period
SE T
Developmental anomalies
21
Sym
Symptoms, signs or clinical findings, not eleewhere classified
SIE BR FE THARELA
Inj
Injury, poisoning or certain other consequences external causes External causes of mor morbidity or mortality
SIN
Est
SEE ST &R DE A94 E
Factors
Fuctors influencing health status or contact with health services
SpecPurp
Coles for special purposes
EE +4 BEE E AA
26
TCMDP
Sapplementary Chapter Traditional Medicine Conditions - Module Sapplementary section functioning assessment Extension Codes
TE 1312 EE #M
PuncAssess
JEM
X
ExcCodes
1dd F4
Applicable
Table 10: Medical Disciplines
Ahhreviation
Full English Name
Chinese Name
ClinMed
Clinical Medicine
IA 7
Dent
Dentistry
1 B
ICWM
Integrated Chinese and Western Medicine
# PE
PH&PPM
Public Health and Preventive Medicine
12
Pharm
Phurmacy
$5
TCM
Traditional Chinese Medicine
# 4
TCPharm
Traditional Chinese Pharmacy
Table 11: Areas of competencies
Abbreviation
Full English Name
Chinese Name म
Diag
Disease Diagnosis and Differential Diagnosis
MedFund
Medical Fundamentals
bx 3 T
N/A
Not Applicable
PHL
Public Health Law and Ethics
2 #
Treat
Disease Treatment
15

---

Table 12: Clinical Departments
Abbreviation
Full English Name
Chinese Name
AesthMed
Aesthetic Medicine
237 3
Anesth
Anesthesiology
ClinNutr
Clinical Nutrition
Dent
Dentistry
n M
Derm
Dermatology
EM
Emergency Medicine
n #$
EndemicD
Endemic Disease
ENT
Otolaryngology
H #4
EthnoMed
Ethnic Medicine
GenMed
General Medicine
ICM
Intensive Care Medicine
#
ID
Infectious Discases
#
TM
Internal Medicine
14
ITCWM
Integrated Traditional Chinese and Western Medicine
MedLabSci
Medical Laboratory Science
N/A
Not Applicable
T
OBGYN
Obstetries and Gynecology
OceMed
Occupational Medicine
Onc
Oncology
Ophth
Ophthalmology
#
PainMed
Pain Medicine
* #4
PallCare
Palliative Care
Path
Pathology
#
Ped
Podiatrics
L
PedHC
Pediatrie Health Care
J 6A $
PedSurg
Pediatric Surgery
PrevMed
Preventive Medicine
Psych
Psychiatry
#
PT
Physical Therapy
#4
Radiol
Radiology
RehabMed
Rehabilitation Medicine
MiddMed&MilMed
Special Medical and Military Medicine
SportsMed
Sports Medicine
Surg
Surgery
サ/
TB
Tuhereulosis
TCM
Traditional Chinese Medicine
#
WH
Women's 's Health
16

---

A.2 Instructions for Pre-annotation
In this section, we present instructions used to pre-annotate CMExam test set data using GPT4. As shown in F 7 we first constrained the output from GPT4 to return only specific categories. We then annotated each of the five additional annotation dimensions relevant to this study with all the category information for each dimension. Next, we provided specific prompt information and finally, we performed filtering on the GPT4 output to improve the effectiveness of pre-annotation. During the actual annotation process, specifie categories and prompt information should be filled in the grey background areas.
EN.Tie Tinited
calgaay.
21 1-185
BEMERE B0 Mill INReORR te BE
1 M 21.......
12 WU22
M4 MI H- He RiffEEE "HEE LE MA
EN Theve beeate seci celogaies Cartais inDetioss purssiic dieases, Nogplanis Disases bood Dkop Scrning organs, Diases of intuaae spitees Endociie
mazntoral mesihlli stasicc Mirral, Inderrama momeicedlsgeenne diredess Meep-ande daenkm; Uisasso namosa symam: Dracees staae onlem: Uiscasc
car proceic Dasisss the dnculitery suasans Disancs reqpesaory sysiem Dacazs dissatve opieas, Diceass skis; Dissao
system
pretonsteary
Tregrency.
parperten:
seganating
Sepplmoniary frnctioning
Uneertam unienoore of meeestity morality
2i:: Wil Wid 9 9
PUB Fitkk- (- F SEDEE M. 37 1E Bh. REEE 2 11116"01
FN) Asorming and oepert the medical indhary, plesse dearmin which MDDll eauggrry this guection billngs Tracco of catogodas apply mtam the label "NNA"
ZH.B H TEEE setert baade NLBdc 11 RUS H.RM A RAA IKHOGWIN APPLLMI taa Witiiit
queston Segcorald
experieseed popposive reduction mestrrl Fox been ancooneee
licketing roods)
the Sollowinng reckenod loo parforel Pronseemmnn
Blood
hormone Dhaznonic carcingi"
ENNote that F5
L- 1-88 wlich of H:
Besnyy-s00 catggrros viliiioral
Figure 4: Pre-annotation Instructions for Disease Groups.
ZIEi H 10 Fe 1 32 PAAA
ENNeThee retum format linited the name a specific category.
6238 113
2
There as six categories: Preventive Medcciee
Surgeny, Obsesiss Gynecology: Women's Health, Pedianies;
Pediaii: Surpesy: Podiatic Health Oggntalnelooyy. Oholarynppopy, Deneatry, Dennatoloyy: Paychiszy, Infectious Disease;; Oneology: Emergency Modkcne:
Traiitional Chince Medkeins; Tubuncalsiis, Pain Mediiin: Asshotic Malkinn:: Enikenis Disease; Rehubiiltuiin Mistciin; Maysical Thanay:: Sparis Malisings
Docuptional Medicine Special Medical and Millaay Medicin;; Pallative Cas; Clinical Nuztion, Integratdd Tradiioral Clisese and Wassem Molicine, Edini Modicine, Asssieiitooyyy Modical Labonsory Seience; Patusiogy, Radallgy, Intessive Care Medicina
ZE:: B:B 1-88---ee fr ic. HAP vill Ai ri &E i's BMaDTEH &.),, 2t 2t- 4. the medical indrutry, plase determine which (MIDII cregory this quostion belongs If none the categories apply, morr the
EN: Asteming label "NVA"
EN-The question Skeycrrald wurman has experimnnd progressive moliction manersal
burn ammnorchiie momils
betsiing for hrmmuns Diagaoniic cunsiange"
monibe Which Gillesingg preiirmal ted parfeen A. Progsatemm leai Blood
dt. 4:37 Misl WA/ ft. Ft te HiX A M4 Ag 1:136 3 4a00 1-3 A Mi 28 19 36 t #:0 4. A 2
EN-Note that you do not neal to moowrr the quedion itelff just retum which the liinesit categonie liaal above moet releori to suection Retar only une the fhlity 58 conegontes, additional words necessary.
Figure 5: Pre-annotation Instructions for Clinical Departments.
17

---

(hepee The return format limited the name of speeific catogory.
1198E & 2 ESRE), 75 PB 11 EN:There are seven calegoriea: Clinkcal Medicine, Demistry, Integrated Chinese and Western Medicine, Public Health and Preveative Medicine, Pharnacy, Traditional Chinese Medicinee Traditional Chinese Pharmacy.
ZH: 13 M
ZH: ff EN.Assuming you are an expert in the modical indusity, please determine which calegory this question belongs to. If none of the categories upply, return "NIA"
25 1 "I /
ZH: 1t 18 2334
ICHCGR
C I ME BI
EN:The question is 34-yaa-olld woman bas experienced progressive reduction mensirual flow and has been amenorrheic for monthi. She has been lactating for 3 monihs. Which of the following the preferred perform? A. Progesterane test B. Blood BCG test C. Blond PRI test Sex hormone test Diagnostic curetiage"
Z
7 B 18
T
EN:Note that you do not need answer the question itself, just return which of the seven categories listed above is most relevant to question, Retum only one of seveni categorics, additional words necessary.
Figure 6: Pre-annotation Instructions for Medical Disciplines.
ENe return format limited the name of spocific category.
Zu::3 Bl:
Ee ut Sn Ma 23: 16
EN:There categories: Basic Medical Knowledge, Diseuse Diagnosis and Differential Diagnosis, Disease Treatment, Public Health Law Bilnes.
2K:: -- KF 1. WOL F F AM # # ENtAsuuming you are an expert categories apply, return the label "N'A"
industry, please determine category question belangs none
ZII EI H. 1334 FESEUEETTEEE HIE RE 2 IPPRLAEE HEMAMM EI
HCHGGEE
EN: question 34-ycar-old has months. Sbo has been lactating for 3 mooths. Which of the following the preferred test to perform? A. Progesterone test B. Blood
progressive reduction
flow and amenorrheie for
Diagnostic curettage".
7H:
2
4-W0
EN.Note that you not need to answer the question itself, just return which of the jour categories listed above most relevant Return only the flour categories, additional words necessary.
Figure 7: Pre-annotation Instructions for Areas of Competencies.
A.3 Analysis of Model Generation Ability
In Figure we present partial explanations generated by various models for medical question from the CMExam dataset. Notably, GPT-4 and 5 produce concise and sensible explanations, which may account for the lower BLUE scores. Conversely, models like Vicuna, LLaMA, and Huotuo demonstrate
prominent repetition phenomenon, while Alpaca simply duplicates the provided options without providing an explanation.
Fine-tuning models on the CMExam dataset significantly reduces the repetition phenomenon and improves the overall reasonableness of the explanations. For instance, the ChM model analyzes each option in a similar manner to the solution explanation. However, some models still generate unreasonable explanations, as observed in LLLMA-CMEXan Alpaca-CMExam, and Vicuna-CMEExnn This could be attributed to their training generic data and lack of specific knowledge in the medical domain. This underscores the significance training large language models with focus the medical domain.
18

---

Chinese
Translated English
Question
T M. ALAU & 1S R. k
silition joint svelling and pain, the most most significnt munifataaioon for the diggneis of
heemmsedd antrriks is:
A J. W aiAiON
A. Pain in the hoel and sole the foot
Question
B 30 19 fn (2 F W15 1
B. Painlaa redules under the skin at the siesit joint peominenec and prosuee
C S0: +9.M RU
Diffusee pulmenary interstriial disease glucose conten))
D (6 P7 (M MM
Plural effesion
E 4 1::50 H-1771 TI
Printul subcuraneous nodules it lower
Ground Truth
9:19
The most commonn sites of theumatoid arthritis are the wrists, fingers, and proxinal
R:11 CAMl
12.5.9
1-38 1.
intrphalangeel joinss. Pain in the beel and sole of the foot (A faise) is generally not seen in thoumaned artrrtis, is aut the most sigutficar, clitical munilcstation Estra-aticular
ff,
mandfesatioss include reeumattdd sodules, which commontty found joist prominences and typical
Solution Explonation
proseure They
nodule (8
pathological changes can help with diagnesis theumatoid antriiss Palmonary involvenent is ottnn charactetized by interstiiaal changes in the lungs. Single or miltple nodules may appear in the lungs, but they are not the most significstt clinical manifestation (C fabec. Whas plauricy (P505) occurs thermatoid artheitis presents a small amount of pleural effuson on ond or both siee of the chest. The pleural fluid exudative with a low glaooe content falee) lowe: lega false)
BR.K DM
4-BEXt6 B2 1- 17 (EI) (8:38
Prinful subcuraneous nodules
theumatoid artrrts.
Medel Generation
GPT-4
at sites yoint grominesee and preceuee are in impoctent manifesttion the dingnosis of thermatoid arthritis
Painless mudales under
ANG A de X.00 MUN M. 10 Rul 18 15
The significant manifestation rhasmattedd antreiis preseees painlees modules under the skin lhe sites of juint preeninenee and presamee Those modales ispically apperr sarr the joints and range is size from fice grain to fist size. These nodules are callod rheumatodd
GFT:3.5
F FI
A)
nedhiles and are associsted with the severiyy and progrosis of rhermatosd ariteiis
R 53 E
stild Pi. #1 1.
The most valrable manifestation
the diagneris theumatoid painless nodules under the skis at the sites joint prominence and presuuee These modiles
presence
RE Li 1plt
F-ll (4 S... 46
13.441 '510
AET X 13 [IF 1t it. E.1..220
-f12--611 3:13 14 (119)
tysically appear on the skin amund the joists and may be producaal by infurmattry cells
McCE li E: (1)
(hitit 4.,
10:55 HIS:
surrounding the joinss While pain in the heel and sole of the foot and diffuse insersiiiall ling discase are other common manifestation
ChatGeM
IE.AL 52,
22.. 3t 92.11 16-1118 1L VEN
theumatoid arthntis, they not valuable for
E. JI Xf di 12..4
1g SC 52.0 19-00 75 3: P1 8:
presence puitless nodales the swallen and presured joinss. Pleural efftsion (with normal glucose levels) painfal subcutaneous nodiless in the logs are also monifestaiion of theumatoid artheitis, but they are typically not the most valuable manitestaions.
diagnosing theumatoid artheilis
In RE de T 21 A. t1 451
84.2 (1 at (E:R)
areas
RE BE Feb "9 Mt. 18 at 96.4.. St 99 1:00 82
IE 11:8 in
10 11t BER8.
A. JE 2 9 B1
Pain the buel sole Soot.
Painless nodiles under discese.
joint procinence pressure.
Diffine palmonury
Vicuna
Painful subcuranecus nodules it lower leg, inswer is pain
Plaural
glucns content).
sole
foot,
question pain The znswer this question A., pain the heel and
The znswer to this question A. pain the heel and
ELaMA
I 2.
6.F
A 2 C.C 4 S.F. 6.A 7.
A J. E. 3t ta
A. Pain in lhe hadl and sole of the foot
B 2 11 do to II (ER F
B. Puinless nodiles under the skm the sites joint peocnnnencs and pressure
Alpaca
C So N P0
Diffuse
intersiiiial disease gluctsc content)
D 06 PAI MEIEE
Plaual eifision
E 1 11 5t 1-47
Painful schcuranecus nodules lower
IH Mi. MEAT S0. N BiR MEN. 18 #
HAT.. 26 R.
#
47
Tumor, pain, diffioe molules, efficiion, dishcts, infection, puin, nedullss pain, nodiles, pain,
Huotuo
fo
15.
nednles, nodnles, nodules, nodnies, nodules, nodules, nodules, nodules, nodules, nodules. nedullss nodnles, nodules, nodiies, nodules, nodules, nodules, nodules, nodules, nodules
96 B.
Ah
No 19,
Ri
40
31% REMPT
The key io disgncing theunstedd artrri is the pecsence puinlos todiles If lfiles is puin, ofth may be diagrosed as meumation or similar discase. there is thoerstoid arthnits with poin, may he disgroued as theumanodd arbrrtic Have a good understanding the knowllgge related 'houmsttdd anthrilie
10 4n. ft sä th
M.
Hentse- CMFxam
Sunk 'innegireeoodd
2222222322222
21229212/2222
Rheumstand
commonly presents
recllng pain. Hesseses acditiin
14 22
At W th13 tk dul do "1 B0 (:
to joint swelling and pain, the most signifienal maniiestaioon diagoosing choumstoid as tleitis
VT 2 N6-
48 28 82 19-b5110 41 2.8 22:3.88 B is the presence of painless modules under the skin at the sines of prominence and presure (B
XL
W
B 4:
1 50 11::::1: (k(B.91). E. (mic) Pain in the heel and sele the fort (A falee) on be woon in thermatoid bethriks, but is
ChaiGEM- CMExam
AI
A2 ff 2
22
2 R2
ST
$3:22"
"1 9c
X 17 [. 7-
of R. 99 false) can be diagrosing theumeroid antrits. Planrsl offesion false) be seen infections coused Myobactesium tubecculoois but
bi not the most significant mazifesttion for diagtosis. Difluse pulmonnry interstrial disense (C
concitions such in pulmonary toberculoeis but is the significant
R8 25. 1-V6
22 2a 15
2.15 SE3y.
glatos content)
(NL E SCHR P5
N. 9----
45 fi 12 1593,
most significant manifetzziion for dagnosing meumstod antirris Painful subeutaneous nodules the lower ley (E false) can be seen in dicumatic andules, but they are not the most signilieare manifesiatio for dingrosing thoumoroid arhriss.
7 DE 5 0 AL
6i X. 1. de.t. 11G0:
tEU2 Fet 1h 11
36 10 3t dh 2: Bay 30 2030 MJB S: st 80 11
The main pathological maniivattion of thoumatoid uttletii the presence of puitles nolules under the siin at the sites of joint pooeninence and pressure (8 true). In the subeutateous cevity
HE Au 92 5 52 (608 F 16 T. (B8))
LLaMA- CMExam
Hi AS 181 at 2 BacE 10
It MiE Eig of the joint prominence, there is prolifecation of gramilstion tisuu cells, After proliferatin, these
(2 (4 (21
cels colarge from the inside the madle udtward, faming clasters of grauuatiion livus cells within the muscle cavily, which eventually
$ Kandiinaa WI. Fiseet 1h.. 12 T1
rodales The fonnation these aodules moscl celle by mednler cells
Pul 59
dependent
AURRE PU 4. 2E 1808,2
Rhoumatoid artlaiii'' clinical munifetationn include systemic symptoms and signs. Rheumatood LE paliens often expesience systemic symphoms such fever, ablegm abdominal chess may also expesience anoresia, dianthes. abdominal pain, limb pain, Reel pain. The conditon can wecses rapidly, and severe casca, these might shdeminal pain, chest pain, foot pain, limb pain, beadache, fever, and phlepm-hens.
Alpsea-MEEsam
E.W. (1-W
pain. lime pain, foot pann.
jall 10
BWW.
Min..
16%).
MA 15 R Sy M4 7558 M08 NC 92 R
X 32
38 1hst thi4 18 9eft. 1:32 PAR6.
The uain pallological clanges in theomatedd athrrii ure the types and severity of themmntedd arthettis. puthological changes thcumancid antriti maiibly inclade: Topes and severity theumasold anhnles: @ Acuse type of t'thumaatod arthettis 2 Late type theumateod 4 (f) srililis @ Clounic type of shoumatoid setheiti
Vicuna-CEExmm
AUR pt JE
1.30 KL
E1 32 12
Figure 8: A case study of LLMs' generated explanations.
19

---

A.4 Analysis of Model Generation Correctness
To assess the accuracy of model-generated explanations, we conducted study using randomly selected sample of 50 cases in which the Language Models (LLMs) correctly predicted the answers. Medical experts were then invited to manually verify the correctness of the explanations, focusing not only on the accuracy of the answer predictions but also on the quality of the accompanying explanations.
Our investigation revealed that despite the correct answer predictions by the models, certain samples exhibited errors in their corresponding explanations. These errors were categorized by the experts into three groups: explanations that were irrelevant, repeated, or inaccurate. The statistics presented demonstrate that the number of samples with accurate explanations generated by the GPT models exceeded 45, accounting for over 90% of the total. However, important note that both the ChatGLMM and C-CMFaa models may produce some erroneous explanations, primarily consisting of inaccuracies and irrelevance. We have included examples of these incorrect explanations in Figure 10
in Figure 9
59
15
10
Reportal
CChalLM
Chail M-Miiaan
Figure 9: Correctness analysis.
A.5 Analysis of Few-Shot and Chain-of-Thoughee Prompts
In our research, designed few-shot and chain-of-though prompts for the answer prediction and reasoning tasks and conducted experiments on the GPT models. As shown in Table [3] our results demonstrate that while the use of few-shot improvements in the prediction task, but there was a notable enhancement in the reasoning task.
mhain-of-thougee prompts did not yield significant
Specifically, for the GPT-4 model, the utilization of few-shot prompts increased the BLUE-1 from 0.17 to 5.95, and the BLUE-4 from 0.06 to 2.25. Furthermore, incorporating chain-of-thought prompts further inereased the BLUE-1 to 7.29. Similarly, positive effects were observed the GPT-3.5 model, where few-shot prompts improved the BLUE-1 and BLUE-4 to 14.62 and 4.80, respectively. Additionally, the ROUGE-1, ROUGE-2, and ROUGE-L increased to 38.08, 18.35, and 18.37.
These improvements can be attributed to the fact that few-shot prompts provide examples that GPT models can reference when generating detailed explanations for each option during the reasoning process. Similarly, tf-thugghtt prompts can achieve similar effects, aiding in the enhancement of model performance.
Table 13: Few-shot and chain-of-thought prompting experiment results of GPT models
Models
Prediction
Reasoning
ACC
BLUB-1
BLUE-4
ROUGE-1
ROUGE-2
ROUGE-L
GPT-4
61.6% 31.6%%...1
61.7%a..1
0.17:0.00
0.06:0.000
29.7420.09
14.84:0.04
11.51
opT--_eew-shot
62.0%
61.4%:005
5.9520.12
2.2500007
37.2420.35
19.2340.26
17.24.0.07
GPT-4ccot
61.6%a.0.
61.49009
7.29:0.71
2.20:0.25
35.8520.78
16.7920.83
17.1810.30
GPT-1.5
46.4%-20.0
46 16.28
1.56=00.88
14990.06
33. 1..00e0.11 11
16.39e0.05
14.83+0.13
GPE335 few-shot
45.356-00.6
4
14.6240.1 16
4 8.0000066
38 38.08+0.44 44
18.3540
18.3740.29
GPT-3.5.cot
47.9%s...
47.7%s..7
13/4740.52
3.6920.18
36/4720.42
16.41.0.4
17.8240.51
20

---

E. Tracttal oevirstiinn sntionniooon by fiecign body
13 Motoellle caoiiaiiaa
C Lange amound of plan eitizion
R. Cranial bcainn tnueyy
The coccooecco of mew.bbliic alalleis mr
E 11- 197
C 10L.. K.W
E. Frak dlaoureddepoooe propunations
C. Tralitien Plirss parcst medidiie
it Clinss Nurbal lou pilss
A Chituss berhal mallciee
listaases
1 1R As (5
C 19844
ls # #
250000 36 A-B yagee /1 PAP0000 A 49
E. Cogathi eoneeeetteee toktigg
D Comierrpea faad faailansigg ansiisi
C diggran
regions with recasional galle wave dischanger The A Boin
peffrrddd derrdddagpp dippootic meeteedss is
ikis right sile. Elecmmons liecimnncglu mimencelllll bargun
musle
Hall a ago, vemting mice boagrais fone incapsiiienn innie to sitttiinng She sund himily streed it bed wiftoai getine un and led two quiodd of conval novalssons on Blodd presue
nicompttiall hondactes
copetteee 40
2 CH
Mi M. HaM00mat F RILEE 2 410 DE
Be MAKE IHLA. 11N 1%%
3 WF SERL.. Rid. HBILOK 1-15
options that nor covered under metical
products bo Sopocial infoortio))
SR ISS oe BO #2
odema
prokun mndkmiiinn the (iiij Diflnse
Quaciin
matty incroved
and cniaal
Anwer
A
brain LA. olees likee and indiaal ohe mytion a lemion livly (E fik-) am all sine decresse estnechill dfinoe (R liv)) lassural olora (ike)) and inciaal abo sottintiion by Rocgimerry aclins due to impaiee vanliion and the eonouoni dismetioe of foeiin boly IE filel C02 elimintcen
cliiiig lntulclluie pultasium to mave out the cells. At the samee tns, Charli MLLMMEE san: entined hydingm fikes move move
Hopelolems (0 mii lods to decease in otracdlldar pesssoam concontation
1000 1ER01 OL ih 22E iisiea Ne ti s..
at 1981)) 88 Bo k. (Ci)) SE the the (Es W- 1 7 (A.%) A S00 A0A. J Haat " Whee IR Sr 1E
15/ ELEE (DB)) Huuie witk Cutmaaa
fand is eiuuattlly the Mirertyy of ahor Seartyy
mediction, and mossell (6) Utter posein nnd control
hathll pilis (4) Fulk-lloweed propeniion and Ribroossr uides is vurion
(3) Vaniou wine resocniiee made from Chinee ietal moncralt and Chinae
seimral and intmal orgi paales pralads the on by used is fini unal (waec fruit. un clamcillyy uxcesaay diis d Cialamy
insuuute: (1) hug poinudly usd
Arick
This guadin osamiin the mode madertiss cariisss that ae noe comma under busk mafrall Chanch NL4 CM mm som invimate The opiion tuat is nin coverd under bitk medical innente is frite This quesittn commine furoned repundtsss (Optoo Ei Acceding
(-) H dHe anp of
Hi de in 3 12 So DL A Fo AA T (K) Wol
16 Ut Milh tiet We ACC Wa on PR. ( 180 19R 17 (050
A A (ab IG Aat We Wolk
The patient bad binory of boddshos amacss and voniing two wedls apo pocsiganed by ginde lovrr and untsponivenn crsesseees (mundieand as redlved sposhhand and pyydemssor ( sumiiaaion mesidd nerteed modee difiue deiu and postine Bibasli ypil wen dische Thorsfonn: lls enceptaains The proferes dupootic mathed is boit Mil (A mue), as Mil can deteminn the inderfiig caues of headscer and masso, which is emana fxr Frammen and di II hypolitnne and 12 Inporistene lran pestrdilnnal lsonns Comparad to cehatiitatki parposes more accumtt alentfyn byng only changs iB Gibe) and oin goik appormate trommes
CT.
AE POTTER IEVE FILL ALAI Aoseee (att))
With 20R HHRE (8
1tat M PHL 20T RIN HES
B0B 0-1611111 11 SU 30-C- 2501
thet fallowing dais diniiond auplaaenn aerertatoo (i Scote Sourty A Mihitnamn and ints R-tay his na coneoree clicn meceun drus ml
Figure 10: Examples of incorreet model -generateddd explanation.
de ot dily 10 HI HE ae te
smoun oilh losiing tidoes, alkokoii coes inpolcloon, the extroollider petsosium poncomentt niin decomoe, moutting menibdil alaioot
to for sam (il FinE CharG CLMM sim: ur Walg wot IM meseee th
Rass
ii. oy it (1)) te ee Went
Explanaioo
made tose in CHATCLM fivena and right siák.
Tinetim Mexurs the fnanncee Liinn Emplopis unter buik Emplogess is semaiiionn mediar medical mommos in tmmnnd tmmuraaonss www.swoooooooooooooooooooo
poy didinin mite thonkk the sitil prisileet usiial wikn inteppeting BEC rouh,, Vilikonely bosn MR/ can holp
oi APs RIME 15 R) CHATGLAE
medial aa
special Indiatioon
with socsiioal Boool fix pilccts comptions comizsiio roults prottred
Mill AI: F REA REAN CAIRR
(: 6x de d: 5 5 4: dr
01 03 07 9 0 0 0 3
() 9t 5 th St 9 SL 9h 51 5t (X 13
Salmwing end coddinne notheon XX
0: 5 5t
(: 3t 3t St ON C3 11- AE 31
3:55 11 2E 23E N10 et M 1 500 30 MA PRM se a M RE
MRd
10. AL C1 1i 5-2 7700 SMMRL BF MRU
C2
under
91 1
cm tlselcomml
nen the catio antio o the mooire ins pens dnn digg
5
2EAL FIRE UR S 1- 2.
01
E.404 A.B8 IE A (7. 3 1I IL Mi A RoroR
pointally and to ropinius woleis
31
3
3
31
5X O 1x 02 5: 55 3%
Mosih-gancnnal explariiion
(y ty ty 2
91
01
dessuppp $10 san 25 anse chr mest onc of the
3
12
31
11
3
36
51
5x 0 i7 1: Sy St 9 5y
36
30
31
58
9X
9
0 37 1r 3: 53 17 0 5
11 91 91 0r 9 9x 9
01 11 32 5t
02 0e 00 9 0 de
ms Numoo Heattee
me (Oron The
and dagnosis mothed
de 0z
ian. optisn that
30
50
51
51 5x dx
Farre ciucofication
Reposee
Reposed
Imvlevare
Inoleus
The coineelluuar pid Tncrsaes
Repicttd platoearena can be malll obeervel
4E AL SE
clfiei of MRN in irrdewen
The ant.aiiiie the
Eagart lendysii
21

---

A.6 Data statistics
Questions in CMExam have median length 17 (Q1: 12, Q3: 32). Regarding solution explanations, the median length is 146 tokens (Q1: 69, Q3: 247). Table 14sowss more basic statistics of CMExam,
Table 14: Basic statistics of CMExam. Q: questions; E: explanations; Q1/3: the first/ third quantile.
Irain
Dev
Test
Total
Question
54,497
6,811
68.119
Vocab
4,545
3,620
3,599
4,629
Q tokens
Max E tokens
2,999
2,678
2,680
2,999
Avg tokens
29.78
30.07
32.63
30.83
Avg tokens
186.24
188.95
201.44
192.21
Median (Q1. Q3) Q tokens
17 (12, 32)
(12, 32)
18 (12. 37)
18 (12. 32)
Median (Q1. Q3) tokens
146 (69, 246)
143 (65, 247)
158 (80, 263)
146 (69, 247)
A.7 Guidelines for Expert-Annotation
During the annotation phase, we invited one expert physician from the Second Affiliated Hospital of Zhejiang University and one senior doctoral student from Zhejiang University School of Medicine to carry out the annotations. The expert physician has over two years of clinical experience. The annotation guidelines have the following sections:
1. Comprehensive Question Understanding: Prior to initiating the annotation process, metic- ulously comprehend the medical question, ensuring a holistic grasp of its context and significance.
2. Subject Categorization: Identify the precise subject or medical field that the question pertains to, such as cardiology, pediatrics, or pathology.
3. Principal Symptoms or Medical Conditions:
Ascertain and pinpoint the primary symptoms or medical conditions expounded in the question.
4. Examination of Pertinent Factors: Scrutinize the question for any associated factors that might be present, including the severity of the ailment, its etiology, and patient history given in the question.
5. Examination of Pertinent Factors: Scrutinize the question for any associated factors that might be present, including the severity of the ailment, its etiology, and patient history given in the question.
6. Appropriate Classification System Usage:
Use the accurate classification system for an- notation in alignment with the determined subject and symptoms. Suitable systems could encompass the 11th revision of the International Classification of Diseases (CCD-11), the Directory of Medical Institution Diagnostic and Therapeutic Categories (DMIDTC), and others.
7. Addressing Multiple Annotations: In scenarios where the question encompasses multiple symptoms medical conditions, opt for the most related classification for annotation.
8. Ensuring Segh-Quaiity Annotations: Adhere to the guidelines and definitions within the chosen classification system. This diligence helps avert subjectivity and ambiguity, fostering precision in the annotations.
9. Navigating Queries and Uncertainties: Should any doubts uncertainties emerge during the annotation process, consult the official documents and glossaries of the chosen classification system. Engaging in discussions with professionals is also advised to achieve clarity.
10. Resolving Discrepancies: When disagreements emerge between annotators, collaborative discussion shall initiated. The objective is reach a consensus and unify the annotation decision.
A.8 Prompt strategies for Pre-Annotation
During the experimental process, we indeed tried different prompts to enable GPT to better understand and complete the annotation task. The specific strategies were follows:
22

---

1. Without ICD-11 Category Instructions: We did not provide detailed ICD-11 category information as instruction but directly supplied the question information and asked GPT to respond. Under this setup, a significant portion of the categories returned by GPT did not strictly belong ICD-11 classifications, yielding unsatisfactory results.
2. Batch Processing for Cost Efficiency: Initially, we concatenated multiple questions and, through single dialogue, had GPT return annotations for multiple questions. Under this setup, expert validation showed that the accuracy of GPT's annotations was relatively low.
3. Consistency in Formatting: When no format guidance given, GPT's return format was inconsistent, resulting in a higher parsing cost. Hence, after multiple trials, we eventually opted for more rigorous format guidance.
Our annotation process was carried out two stages: First, GPT conducted initial round of annotaintnion. Subsequently, we invited an expert physician from the Second Affiliated Hospital of Zhejiang University and doctoral student from Zhejiang University School of Medicine to annotate. The expert physician had over two years of clinical experience. In instances where there were disagreements in annotations, both annotators would discuss and eventually arrive at a consensus.
23


Reviews:
[{"review_id": "xAvfgSlsBJ", "reviewer": "Reviewer_989i", "summary": "", "questions": "", "limitations": "Yes, they discussed the possible potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The creation of CMExam presents a novel contribution to the Chinese medical LLMs community, addressing the current scarcity of evaluation resources. The paper offers valuable insights into LLM performance, highlighting the limitations and potential areas of improvement.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "AAR2HjZm5U", "reviewer": "Reviewer_V544", "summary": "", "questions": "", "limitations": "1. The authors have adequately addressed the limitations and potential societal impact of their work in the conclusion section.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The proposed dataset incorporates over 60,000 Chinese medical questions, annotated across multiple dimensions using both GPT-4 and medical experts. This comprehensive annotation process provides valuable insights into the wide-ranging medical proficiencies of different models.\n\n2. This work can serve as a benchmark for recently released large-scale models, conducting quantitative and qualitative evaluations of their performance. \n\n3. In addition to the main analyses, supplementary experiments and assessments on few-shot learning and chain-of-thought reasoning were performed, which offer a more holistic evaluation of the models.\n\nIn conclusion, this work provides a carefully annotated medical QA dataset and a thorough evaluation of the latest models. The insights and opportunities it presents will be valuable for researchers in the fields of healthcare and NLP.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "vPY3MAdent", "reviewer": "Reviewer_Vnjt", "summary": "", "questions": "", "limitations": "The authors appropriately addressed limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "An important contribution of the paper is the introduction of furhter classification for each question in the dataset. \nThis allows to stratify results along multiple axis, e.g. according to medical competency.\nSecond major contribution is the extensive benchmarking performed. The authors assessed the performance of 11 LLMs,\nfine-tuning 5 on their dataset.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Reviewers are generally positive in recommending the acceptance of this manuscript but also raise concerns such as the comprehensiveness of the benchmark and the use of larger LLMs.
Decision: Accept (Poster)