Venue: NAACL 2024

# CMB: A Comprehensive Medical Benchmark in Chinese

Xidong Wang1, Guiming Hardy Chen1, Dingjie Song1, Zhiyi Zhang,

Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li,

**Xiang Wan, Benyou Wang2, Haizhou Li The Chinese University of Hong Kong, Shenzhen

Shenzhen Research Institute of Big Data wangbenyou@cuhk.edu.cn**

Equal Contribution.Corresponding author.

Footnote 1: Equal Contribution.

###### Abstract

Large Language Models (LLMs) provide a possibility to make a great breakthrough in medicine. The establishment of a standardized medical benchmark becomes a fundamental cornerstone to measure progression. However, medical environments in different regions have their local characteristics, e.g., the ubiquity and significance of traditional Chinese medicine within China. Therefore, merely translating English-based medical evaluation may result in _contextual incongruities_ to a local region. To solve the issue, we propose a localized medical benchmark called CMB, a Comprehensive Medical Benchmark in Chinese, designed and rooted entirely within the native Chinese linguistic and cultural framework. While traditional Chinese medicine is integral to this evaluation, it does not constitute its entirety. Using this benchmark, we have evaluated several prominent large-scale LLMs, including ChatGPT, GPT-4, dedicated Chinese LLMs, and LLMs specialized in the medical domain. We hope this benchmark provide first-hand experience in existing LLMs for medicine and also facilitate the widespread adoption and enhancement of medical LLMs within China. Our data and code are publicly available at [https://github.com/FreedomIntelligence/CMB](https://github.com/FreedomIntelligence/CMB).

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6184–6205

June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

Over the past two centuries, medical advancements have substantially increased human life expectancy. Medicine's effectiveness often hinges on experience, with veteran physicians typically outperforming novices. In parallel, large language models like ChatGPT are shaped by their vast data experiences. This mutual reliance on experiential learning between physicians and LLMs suggests a promising frontier for LLMs in the medical domain.

**Medical evaluation is highly professional.** Although the future of _LLMs for medicine_ is promising, their evaluation is a challenging topic. Deploying LLMs in hospitals raises significant ethical concerns that real-world feedback becomes difficult. Existing works on LLMs tend to leverage subjective evaluation Zheng et al. (2023) where none of references is used during the assessment. However, the evaluation in medicine is much more professional than that of the general domain. For instance, assessing _radiology_-related issues poses a challenge for the public, a senior professor in medicine, or even a _general practitioner_. Subjective evaluation would be difficult to be scaled up since professional manual judging is expensive.

**Benchmark for medical knowledge.** Another school of evaluation protocol is objective evaluation, where the expected output has a clear reference. Certain protocols emphasize natural language understanding tasks that are not knowledge-intensive, as seen in studies Zhang et al. (2022); Peng et al. (2019). In the era of Large Language Models (LLM), modern NLP evaluations underscore the significance of knowledge Huang et al. (2023); Hendrycks et al. (2021b). In biomedicine, a typical example to probe knowledge is BioLAMA Sung et al. (2021); however, it is tailored to evaluate masked language models instead of auto-regressive ones. Another benchmark is MultiMedBench Tu et al. (2023), covering question answer, report summarization, visual question answering, report generation, and medical image classification. Note that MultiMedBench is only in English.

**The necessity to localize medical benchmark.** During economic globalization, a unified medical standard may overlook the unique medical needs and practices of different regions and ethnic groups, indicating the necessity to localize medical benchmarks. For example, in Asia, Traditional Chinese Medicine (TCM) not only offers profound insights and localized medical solutions in the prevention, treatment, and rehabilitation of diseases but also has formed a medical paradigm closely associated

with regional, climatic, dietary, and lifestyle characteristics, over its long historical evolution. In TCM, a disease has two aspects: "bing" and "zheng". The former is often translated as "disease entity". The latter, and more important one, is usually translated as "pattern". For example, the disease entity of a common cold might present with a pattern of wind-cold in one person, and with the pattern of wind-heat in another1.

Footnote 1: [https://en.wikipedia.org/wiki/Traditional_Chinese_medicine#Six_Excesses](https://en.wikipedia.org/wiki/Traditional_Chinese_medicine#Six_Excesses)

Simultaneously, it poses significant challenges when applying the Western medical framework to a local environment, which needs cross-cultural communication and understanding. In terms of disease diagnosis, examination of the tongue and the pulse are among the principal diagnostic methods in TCM. For example, redness on the tip of the tongue might indicate heat in the Heart, while redness on the sides of the tongue might indicate heat in the Liver. For drugs, there are roughly 13,000 compounds used in China and over 100,000 TCM recipes recorded in the ancient literature. Therefore, we should adopt a _native_ medical benchmark instead of a _translated_ medical benchmark for a local environment. Note that the precise translation of medical terminologies necessitates both medical professionals and the cultural context in the target language.

**CMB's Philosophy.** The CMB dataset comprises two parts: **CMB-Exam**, featuring multiple-choice questions from qualification exams, and **CMB-Clin**, including complex clinical diagnostic questions derived from real case studies. The dataset spans 6 major categories and 28 subcategories, totaling 280,839 multiple-choice questions. For **CMB-Exam**, we selected 400 questions from each subcategory to create an evaluation set. Additionally, **CMB-Clin** is formed from 74 expert-curated medical record consultations, drawn from clinical diagnostic teaching materials. Each multiple-choice question in the dataset offers four to six options, with one or more correct answers. The clinical diagnostic questions are based on real, intricate cases, with correct answers determined by a consensus of teaching experts.

**Take-away messages from CMB.** After benchmarking various LLMs in CMB, we get the following observations that might be insightful. **I)** GPT-4 and recent open-sourced LLMs such as Qwen-72B-Chat and Yi-34B-Chat, have achieved an accuracy rate exceeding 60%, surpassing the threshold required for obtaining license; **II)** Accuracy exhibits significant disparities across professional levels and knowledge areas, notably between **traditional Chinese medicine** and Western medicine; **III)** The effectiveness of the **CoT and few-shot prompts** varies among models with different accuracy levels, especially presenting potential risks in knowledge-intensive tasks; and **IV)** Results of automatic evaluation using GPT-4 highly agree with **expert evaluation** results.

## 2 The Philosophy of CMB

### The Overall Philosophy

We surveyed different medical professionals (physicians, nurses, technicians, and pharmacists) about the exams they encountered in their career development. Our research focused on common assessment types, leading us to select two key tasks for further study: multiple-choice questions and iterative questioning based on complex medical records. The former evaluates the model's knowledge grasp, while the latter assesses its practical problem-solving skills. Both tasks, having standard answers, provide reliable and stable performance indicators.

### Philosophy of CMB-Exam

Existing medical benchmarks, sourced from the internet Li et al. (2023b), hospitals, etc., face privacy and accuracy challenges. We opted for qualification exams as our data source, creating the **CMB-Exam** subset. This choice is due to two key advantages: (I) qualification exams offer objective and typically accurate ground truths; (II) they provide a clear benchmark, namely a 60% accuracy rate, which corresponds to the expertise level in specific domains. The multiple-choice questions in **CMB-Exam** encompass four clinical medical professions: _physicians_, _nurses_, _medical technicians_, and _pharmacists_. These exams span the entire professional journey, from undergraduate basics, graduate selections, standardized tests, professional qualifications, to intermediate and advanced professional title exams.

In the Chinese medical field, significant work has been done on multiple-choice tasks. MLEC-QA Li et al. (2021) compiled 21,700 manually annotated questions from the Chinese National Licensed Pharmacist Examination. Similarly, CMExam Liu et al. (2023) gathered 68,119 tagged questions from 

the same exam. However, it's important to note that the potential assistance provided by LLMs in medical professions is not confined to just pharmacy. It also encompasses a broad range of other health-related occupations, such as nursing and medical technology, among others. Given that the Licensed Pharmacist Examination represents only a fraction of the career growth spectrum, its limited knowledge scope and occupational coverage do not provide detailed feedback. To address this, we compiled **CMB-Exam**, encompassing all medical-related occupations and the full range of exams encountered throughout their professional development.

### Philosophy of CMB-Clin

Besides the theoretical exam content in **CMB-Exam**, the second subset, **CMB-Clin**, focuses on practical skills. This subset comprises complex clinical diagnostic problems to test the model's synthesis of knowledge and reasoning. It requires the model to utilize its medical knowledge for answering questions and to analyze case reports for informed responses. **CMB-Exam** and **CMB-Clin** together offer a comprehensive evaluation framework, applicable to both the career development of medical professionals and the learning trajectory of medical LLMs. To our knowledge, **CMB-Clin** is the inaugural multi-round question-answering dataset based on real, complex medical records.

## 3 Dataset Creation

### Taxonomy of CMB-Exam

To obtain a precise taxonomy of medical evaluation, we aligned it with the disciplinary and examination systems of the medical field. First, we chose four main medical professions: physicians, pharmacists, medical technicians, and nurses, covering various occupational difficulty levels of examinations. Considering the learning trajectories and professional growth paths, we additionally include _discipline examinations_ and _graduate entrance examinations_ for these four professions, ultimately resulting in six categories: Physician, Nurse, Technician, Pharmacist, Undergraduate Disciplines, and Graduate Entrance Exam. One could refer to Table 1 for the detailed taxonomy. Moreover, we carried out a more detailed subject division within each subcategory, resulting in a total of 174 categories, the detailed directory list of which can be found in Appendix B. Through this structured arrangement, our directory structure reflects characteristics closely connected to the actual medical field, providing a solid foundation for further analysis and research.

### Creation of CMB-Exam

Data SourcesThe data is derived from publicly available examination questions and coursework exercises with clear solutions provided by experts. A significant portion of these materials comes from the Medtiku2, from which we obtain explicit permission to share the data.

Footnote 2: [https://www.medtiku.com/](https://www.medtiku.com/)

Data PreprocessingQuestions undergo a standardized data preprocessing procedure, including de-duplication and cleansing. For character recognition errors caused by OCR, we conduct a large amount of manual calibration to ensure the consistency with the original document. For possi

\begin{table}
\begin{tabular}{l l c c c} \hline \multirow{2}{*}{**Category**} & **Subcategory** & \multicolumn{1}{c}{\# **Subject**} & \multicolumn{1}{c}{\# **Question**} \\ \cline{2-5}  & Resident Physica (\(\hat{\pm}\)\%) & Licesal Assistant Physician (\(\hat{\pm}\)\%\#\%); Licensed Physician (\(\hat{\pm}\)\%\#); Associate & 81 & 124,926 \\  & Professional Physician (\(\hat{+}\)\%\#\%); Advanced Confessional Physicians (\(\hat{\pm}\)\%\#\%) & \multicolumn{1}{c}{} & \\ \hline Nurse (\(\Phi^{\prime}\) ==) & Practicing Name (\(\Phi^{\prime}\) ==), Licensed Practical Nurse (\(\Phi^{\prime}\) ==), Nature Name (\(\hat{\pm}\)\%); Advanced Practice News (\(\hat{\pm}\)\%\#) & 8 & 16,919 \\ \hline Technicians (\(\hat{\pm}\) \#) & Medical Technique (\(\hat{\pm}\)\#); Medicaid Technologies (\(\hat{\pm}\)\#); Suggressing Handbook (\(\hat{\pm}\)\#) & 21 & 27,004 \\ \hline Pharmacist (\(\hat{\pm}\)\#) & Licensed Pharmaceutical (\(\hat{\pm}\)\#) & \(\hat{\pm}\)\#); Licensed TCM Pramramatic (\(\hat{\pm}\)\#) & \(\hat{\pm}\)\#); Junior Pharmacist (\#) (\#) (\#) (\#) (\#); Junior & 8 & 33,354 \\  & Pharmaceuticals Assistant (\(\hat{\pm}\)\#); Junior TEM Parmausiak (\(\hat{\mp}\)\#) \#); Junior TEM Pirnamatisiak Assistant (\(\hat{\pm}\)\#) \#); Junior TEM Pirnamatisiak Assistant (\(\hat{\pm}\)\#) (\#) \#); & 8 & 33,354 \\  & Chief Pharmacistics (\(\hat{\pm}\)\#); Direct TEM Pharmacists (\(\hat{\pm}\)\#) \#) \#) & \(\hat{\pm}\)\#) (\#) \\ \hline Undergraphulate Dis. Fundamental Medicine (\(\hat{\pm}\)\#) \#) \#) \#) \#) \#) \#) \#) \#) \#); Preventive Medicine (\(\(\(\hat{\pm}\)\#)\#) \#) \#) \#) \#) \#) \#) \#) \\ \# \# \# \# \# \# \# \# \# \# \# \# \# \# \# \# \#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\ \#\#\#\ \#\#\#\ \#\#\#\ \#\#\#\#\ \# 

ble grammatical or transcription errors, we use the comment system of Medtiku to delete data items with "problematic", "wrong question" and "incorrect" comments. Comparison of different directories before and after deletion is shown in Table 5.

Data StatisticsFinally, we obtained a total of 280,839 multiple-choice questions. To assess the model's comprehension of medical knowledge, we randomly selected 400 questions from each subcategory as a test set. Additionally, to facilitate experiments with few-shot learning strategies, we randomly selected 10 questions from each subcategory as a development set. We then enlisted the help of three medical specialists to generate explanations for each of these questions, specifically for the purpose of conducting chain-of-thought experiments (example shown in Figure 4). The remaining 269,359 questions were used as the train set.

### Creation of CMB-Clin

CMB-Clin is designed to investigate models' proficiency in knowledge application amidst real-life diagnosis and treatment circumstances.

Data PreprocessingIn order to obtain a high-quality dataset, we initially collected 108 cases with questions and answers from a variety of official medical textbooks. These problems covered a wide range of disease types. Subsequently, we performed quality screening based on the following criteria: Firstly, we eliminated problems that required reliance on image information, such as those that needed CT, MRI, and other imaging data for resolution. Secondly, we selected problems that contain sufficient diagnostic information to answer the questions. Lastly, we removed cases with similar diseases to ensure the diversity of the problems. After such screening, we finally obtained 74 high-quality, complex, and real cases with 208 questions, as exemplified in Figure 1, for the construction of the CMB-Clin subset.

Task FormulationWe transform the question answering task into the multi-turn dialogue task. Specifically, for each case with questions, we simulate dialogue interactions between an **examiner** and a **candidate**, focusing on assessing the model's diagnostic and therapeutic capabilities.

Figure 1 illustrates the structure of each case, which includes three main parts: **I) Description**\(D\): patient information, including medical history summaries and chief complaints, physical examinations (e.g., visual and tactile inspection), and ancillary examinations (e.g., biopsy and CT scans); **II) Questions**\(Q\): questions related to diagnosis and treatment based on descriptions. Some questions might be interrelated; and **III) References**\(R\): corresponding reference answers to questions.

Formally, to simulate dialogue interactions, we concatenate the patient's description with the previous question-answer pairs and the current question in each conversation round (e.g., the \(k\)-th round). This concatenated input, denoted as \(x\), is represented as follows: \(x=D_{i}+Q_{i}+R_{i}+\ldots Q_{i+k}\). The reference answer for this input is \(R_{i+k}\). For the response \(\hat{R}_{i+k}\), we will evaluate it from four dimensions, including: Fluency, Relevance, Completeness, and Proficiency. These dimensions are adopted as per suggestions from experts.

Data DistributionWe commissioned a medical professional to examine the distribution of questions within the CMB-Clin from two perspectives: the department associated with the medical record, and the point in the consultation process where the question arises. Table 3 displays the distribution of medical records across various departments. It reveals a wide-ranging coverage, with internal medicine and surgery representing the most substantial segments (15.57% and 14.87% respectively). Table 4 illustrates where in the consultation process the questions are situated. It indicates that questions permeate the entire diagnostic procedure, with a significant portion arising during the treatment principles stage (30.09%).

\begin{table}
\begin{tabular}{l|c c c c c c c c} \hline \hline
**Diagnosis process** & **Treatment** & **Diagnosis** & **Differential** & **Medical test** & **Medical study analysis** & **Operational** & **Indications** \\  & **principles** & & **diagnosis** & **recommation** & **history analysis** & **knowledge** & **for surgery** \\ \hline
**Percentage (\%)** & 30.09 & 18.44 & 14.32 & 12.71 & 11.65 & 9.7 & 3.09 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Distribution of questions located within the consultation process.

 



## 4 Experiments on CMB-Exam

### Experimental Setup

ModelsWe evaluate the following Chinese medical LLMs to compare their performance on CMB-Exam: HuatuoGPT-II Chen et al. (2023), ChatMed-Consult Zhu and Wang (2023), MedicalGPT Xu (2023), ChatGLM-Med Meng et al. (2023), DoctorGLM Xiong et al. (2023), BianQuea (Chen et al., 2023), Bentsao Wang et al. (2023), IvyGPT Wang et al. (2023), Sunsimiao Xin Yan (2023) and DISC-MedLLM Bao et al. (2023). In addition to these specialized models, we also evaluate some publicly-available general-domain instruction-following model series, namely ChatGLM3-6B Du et al. (2022), Baichuan2 Baiichuan (2023), Qwen Bai et al. (2023), Yi3, Deepseek-llm AI (2023), Mistral Jiang et al. (2023) and Internlm Team (2023). For closed source commercial models, we evaluate ChatGPT4 GPT-45, ShunkuMPT, AntGLM-Med JianpeiGPT. We also test the performance of DISC-MedLLM trained on CMB-Exam-Train. All experiments are conducted in August, 2023. Please refer to Appendix C for more details about models and training.

Footnote 3: [https://github.com/@1-ai/Yi](https://github.com/@1-ai/Yi)

Footnote 4: We use the version gpt-3.5-turbo-16k-0613.

Footnote 5: We use the version gpt-4.

Decoding HyperparametersFor all the aforementioned open source models, we adopt their default hyper-parameters. Besides, to reduce the variance in generation, we adopt greedy decoding for all models on both CMB-Exam and CMB-Clin. And the min_new_tokens and max_new_tokens are set to 1 and 512, respectively, to avoid empty or lengthy answers on CMB-Exam.

Evaluation DetailsWe evaluate the models in both answer-only and chain-of-thought (CoT) settings. We extract answers from model outputs using an empirically designed regular expression. Each extracted answer is compared to the solution and is deemed correct if and only if they are exactly matched. We adopt accuracy as our metric. All evaluation experiments and training experiments take around 1000 GPU-hours on 8 NVIDIA A800 80GB GPUs.

### Benchmarking Results

We report the results in Table 6 and Table 7. There are several observations drawn from different aspects.

Figure 1: Case of Short Bowel Syndrome from CMB-Clin. English translations are shown for better readability.

\begin{table}
\begin{tabular}{l|c c c c c} Category & Physician & Nurse & Technicians & Pharmacist & Undergraduate Disciplines & Graduate Entrance Exam \\ Before Processing & 125,102 & 16,923 & 27,008 & 33,362 & 62,435 & 16,367 \\ After Processing & 124,926 & 16,919 & 27,004 & 33,354 & 62,271 & 16,365 \\ \hline \end{tabular}
\end{table}
Table 5: Sample Numbers of different directories before and after Preprocessing

On General LLMs.As shown in Table 6, there is no general model that is particularly lacking in medical knowledge. Taking GPT-4 and ChatGPT as the watershed, Qwen-72B, Yi-34B, and Yi-6B have achieved accuracy exceeding GPT-4, and most models have achieved performance exceeding ChatGPT. Yi-6B achieves such good results with a smaller number of parameters is impressive, but it also reminds us of the possibility of data leakage. At the same time, we also noticed that Qwen-1.8B showed strong in-context learning capabilities. Considering its number of parameters, this is also an exciting discovery. Domestic general models have completed catching up with OpenAI in medical knowledge in chinese.

On Medical LLMs.As shown in Table 4, the gap between medical models is obvious. HuatuoGPT-II surpasses all commercial models and ranks first, demonstrating its outstanding capabilities. At the same time, the commercial model is also significantly ahead of other open source medical LLMs except HuatuoGPT-II. Considering that it has been a long time since most medical LLMs were open sourced, we believe that the new generation of open source medical LLMs will quickly close the gap. After trained on CMB-Exam-Train, DISC-MedLLM ranked second among open source medical models, indicating that the large amount of deterministic medical knowledge contained in multiple-choice questions in CMB-Exam-Train is

\begin{table}
\begin{tabular}{l|c|c c c c c c} \hline \hline
**Model** & **Open** & **Physician** & **Nurse** & **Pharmatch** & **Technicala** & **Dicyclines** & **Graduate Extance Exam** & **Average** \\ \hline  & & \multicolumn{5}{c}{_Geansal Models_} \\ \hline Question-72B-Chatt & ✓ & **78.55 (08.00)** & 83.56 (88.69) & 79.78 (08.31) & **77.92 (79.50)** & **68.26 (67.58)** & 58.19 (**57.44**) & **74.38 (74.81)** \\ + CGT & ✓ & 78.04 (0.15) & 83.31 (81.69) & **80.13 (76.73)** & 77.78 (73.03) & 67.69 (65.38) & **58.81 (55.50)** & 74.24 (71.42) \\ \hline Vi-3dB-Chatt & ✓ & 11.00 (72.95) & 75.66 (04.04) & 73.16 (74.03) & 73.76 (75.92) & 66.56 (67.31) & 52.94 (55.63) & 69.17 (71.22) \\ + CoT & ✓ & 69.05 (58.45) & 74.75 (63.13) & 70.25 (60.06) & 68.00 (57.08) & 63.00 (56.31) & 51.09 (45.44) & 66.12 (56.75) \\ \hline Yi-6B-Chatt & ✓ & 67.25 (68.70) & 76.38 (77.06) & 68.50 (69.38) & 67.33 (68.00) & 61.75 (62.44) & 53.50 (55.88) & 65.87 (66.91) \\ + CoT & 64.04 (59.95) & 73.50 (67.38) & 64.44 (61.03) & 65.33 (56.67) & 69.53 (55.25) & 50.50 (48.75) & 62.95 (58.17) \\ \hline GPT-4 & ✗ & 89.90 (0.16) & 69.31 (01.04) & 51.29 (61.03) & 51.60 (63.28) & 39.69 (60.25) & 54.19 (58.12) & 59.46 (60.22) \\ + CoT & ✗ & 85.15 (96.93) & 70.31 (01.02) & 53.09 (65.21) & 52.44 (61.38) & 68.09 (60.25) & 52.30 (54.21) & 59.45 (60.11) \\ \hline Deepweb-llm-6fB-Chut & ✓ & 52.90 (59.15) & 61.80 (68.19) & 54.28 (59.22) & 51.42 (55.52) & 51.19 (55.63) & 40.63 (38.88) & 51.99 (55.55) \\ + CoT & ✓ & 56.20 (51.80) & 61.90 (60.26) & 54.44 (53.69) & 50.75 (47.58) & 51.38 (51.63) & 41.90 (38.50) & 52.33 (50.58) \\ \hline Batchman-13B-Chatt & ✓ & 49.55 (50.06) & 56.75 (57.75) & 49.41 (50.50) & 50.08 (48.50) & 42.58 (49.06) & 39.13 (40.63) & 48.37 (49.58) \\ + CoT & ✓ & 48.90 (45.85) & 57.25 (54.95) & 49.88 (49.16) & 49.33 (47.08) & 46.68 (44.37) & 48.38 (46.37) \\ \hline Queen-7B-Chatt & ✓ & 48.00 (49.45) & 54.25 (55.13) & 48.34 (47.49) & 48.08 (49.25) & 44.87 (45.00) & 35.94 (36.56) & 46.58 (47.22) \\ + CoT & ✓ & 42.58 (43.30) & 51.75 (56.16) & 44.59 (44.59) & 45.50 (33.00) & 43.00 (40.44) & 32.56 (43.02) \\ + CoT & ✓ & 43.55 (38.25) & 41.66 (41.73) & 44.22 (43.91) & 43.70 (39.50) & 42.69 (39.63) & 32.44 (20.56) & 42.94 (38.16) \\ \hline ChuGLMLM-6B & ✓ & 42.55 (43.03) & 47.31 (44.81) & 39.56 (38.22) & 41.08 (39.33) & 37.44 (37.63) & 32.05 (32.13) & 40.00 (38.74) \\ + CoT & ✓ & 38.90 (38.15) & 45.38 (43.25) & 38.19 (34.53) & 38.08 (37.08) & 36.12 (35.25) & 30.13 (26.75) & 37.80 (35.84) \\ \hline ChuGLPTP & ✗ & 40.75 (40.75) & 45.69 (46.96) & 36.95 (46.99) & 40.08 (46.08) & 37.94 (37.94) & 28.81 (28.81) & 38.31 (38.33) \\ + CoT & ✓ & 17.75 (17.75) & 19.49 (19.60) & 16.00 (25.025) & 20.25 (20.25) & 19.16 (19.19) & 18.23 (28.38) \\ \hline Interlink-Clot2b & ✓ & 39.35 (39.55) & 45.44 (43.00) & 38.53 (36.25) & 37.92 (38.25) & 38.12 (38.06) & 29.63 (29.63) & 38.17 (37.46) \\ + CoT & ✓ & 34.48 (32.58) & 42.13 (36.11) & 33.69 (32.41) & 43.75 (30.00) & 33.35 (31.06) & 27.94 (26.94) & 34.91 (23.46) \\ + CoT & ✓ & 35.55 (47.40) & 41.38 (38.31) & 33.88 (32.41) & 35.83 (35.42) & 34.88 (32.19) & 27.88 (25.50) & 34.73 (33.09) \\ \hline Mixtral-8rB-32xueglen & ✓ & 34.50 (39.00) & 42.00 (41.85) & 25.06 (33.13) & 30.83 (39.50) & 31.81 (36.44) & 22.25 (28.25) & 31.07 (36.37) \\ + CoT & ✓ & 34.50 (28.00) & 42.00 (34.06) & 25.06 (24.69) & 30.83 (34.75) & 31.81 (27.50) & 22.25 (17.56) & 31.07 (27.76) \\ \hline Queen-18B-Chatt & ✓ & 26.20 (41.15) & 34.06 (60.53) & 28.03 (39.78) & 27.08 (39.25) & 25.69 (56.56) & 23.50 (33.7 

very helpful for improving performance. How to efficiently inject the knowledge of multiple-choice questions into LLM is a promising task.

On Different CategoriesLLMs show varied performance across clinical specialties. Specifically, scores for pharmacist-related questions tend to be lower, while those concerning nursing staff are typically higher. This difference might arise from the foundational knowledge nurses require, which is straightforward, compared to the intricate distinctions in drug names and indications pharmacists deal with. Despite these performance variations among specialties, the models exhibit a consistent trend, suggesting no inherent bias towards any particular domain.

On Prompt StrategiesFor the vast majority of domestic General LLM and Medical LLM, both the Few-shot and CoT strategies have little effect on improving model accuracy. The few-shot strategy has improved significantly for models such as Deepseek-llm, Mistral-7B, Mixtral-8x7B, which originally had limited support for Chinese, and smaller models such as Qwen-1.8B and Yi-6B. The CoT strategy even has negative effects on models such as Mistral and ChatGLM-Med, which have very low original accuracy. In CMB-Exam, for problems that do not require reasoning, the CoT strategy may confuses correct information with irrelevant context, thereby reducing accuracy.

On the Perceived DifficultyAs shown in Figure 2, the professional level continues to improve from bottom to top. Only the Nurse category meets expectations with accuracy decreases from bottom to top. For the Physician, Advanced Professional subcategory have the lowest accuracy and Resident Physician have the second lowest accuracy. After sample analysis, we found that the questions covered in the Resident Physician subcategory involve many uncommon details and knowledge, which increases the probability of hallucinations. For Technologist, it's interesting that the accuracy rate is completely opposite to expectations. We found that there are many questions focus on personnel management and communication in Supervising Technologist subcategory, which is indeed what LLMs are good at. For the Pharmacists, there is no obvious trending. But subcategories involving

\begin{table}
\begin{tabular}{l|c|c c c c c c} \hline \hline
**Model** & **Open** & **Physiclin** & **Name** & **Pharmacion** & **Technitain** & **Disclimbies** & **Graduate Extreme** & **Average** \\ \hline  & \multicolumn{9}{c}{_Commercial Model Models_} \\ \hline JianPoGPT & ✗ & 73.60* & 77.00* & 72.84* & 65.00* & 70.13* & 78.40* & 72.84* \\ \hline ShuKimQPT & ✗ & 68.65* & 71.44* & 70.78* & 61.92* & 62.81* & 51.06* & 64.44* \\ \hline AnGLM-Med & ✗ & 62.85* & 66.81* & 60.06* & 48.50* & 54.69* & 51.06* & 55.00* \\ \hline  & \multicolumn{9}{c}{_Open source Method Methods_} \\ \hline  & \multicolumn{9}{c}{_Compressed Multitask_} \\ \hline  & \multicolumn{9}

 

traditional Chinese medicine always have relative low accuracy, indicating that additional data on traditional Chinese medicine still needs to be supplemented.

## 5 Experiments on CMB-Clin

### Experimental Setup

Prompt constructionEvery prompt comprises two components: a description that may (or may not) encompass conversation history \(D_{i}\), and the question \(Q_{i}\). To integrate the conversation history into the description, we prepend the appropriate roles to each question and reference.

Expert and Automatic EvaluationTo prove the validity of our evaluation, we engage three annotators with professional medical knowledge to evaluate on a randomly selected subset of 320 responses of 11 models from different tiers. Equipped with a reference solution, they score each response across four aspects -- Fluency, Relevance, Completeness, and Medical Proficiency -- using a grading scale from 1 to 5. The user interface is shown in Appendix D.1. To enhance efficiency and reduce expert evaluation costs, we leverage GPT-4 to assess the responses of all models, adhering to the same guidelines as those used in expert evaluations. The prompt template for the automatic evaluation is detailed in Appendix D.2.

### Benchmarking Results

On General LLMsAs shown in Table 8, except for Deepseekllm-67B and Miktral-8x7B, which have insufficient support for Chinese models, the other General LLMs have shown strong dialogue capabilities based on complex medical records. Taking GPT-4 and ChatGPT as the dividing line, Yi-34B has achieved the same medical dialogue capability as GPT-4. Qwen-72B is weaker than GPT-4 but stronger than ChatGPT, and the remaining models are all weaker than ChatGPT. Compared with their strong performance in CMB-Exam, domestic General LLMs still lag behind OpenAI in CMB-Clin, which is closer to real scenarios. Except for the Yi LLMs, the ability of other domestic LLMs to solve real problems does not match their ability to answer multiple-choice questions, suggesting that they may have been specially strengthened for multiple-choice questions. Such model iteration direction actually deviates from actual needs. During the iteration process, we recommend using both CMB-Exam and CMB-Clin for model capability awareness.

On Medical LLMsAs shown in Table 9, the overall dialogue ability of Medical LLMs is lower than that of General LLMs. Although the three models of DoctorGLM, Bentsao, and ChatGLMMed all claim to be optimized for consultation, the actual results show that their conversational capabilities have not been enhanced. It is worth noting that although BianQue-2 performed poorly in CMB-Exam, it performed well in CMB-Clin, indicating that it just lacks the ability to do multiple-choice questions and follow instructions. Although HuatuoGPT-II-34B surpasses GPT-4 in CMB-Exam, it still lags behind GPT-4 and is even lower than its base model Yi-34B in CMB-Clin, indicating that multiple rounds of dialogue data need to be added during its training process. It is noted that performance of DISC-MedLLM trained on CMB-Exam-Train drops significantly on CMB-Clin, indicating the need to add other data or re

\begin{table}
\begin{tabular}{l c c c c c c c c c} Aspects & GPT-4 & Yi-3dB & Busen-72B & CharGPT & Baichuan-213B & ChatGLM3-GB & Intermlm-20B & Deepseekllm-67B & Mistral-8x7B \\ \hline Fluency & 4.95 & 4.99 & 4.96 & 4.97 & 4.93 & 4.92 & 4.9 & 4.78 & 2.53 \\ Relevance & 4.71 & 4.69 & 4.58 & 4.49 & 4.41 & 4.11 & 3.91 & 4.04 & 2.28 \\ Completeness & 4.35 & 4.34 & 4.12 & 4.12 & 4.03 & 3.74 & 3.25 & 2.62 & 1.54 \\ Proficiency & 4.66 & 4.64 & 4.55 & 4.53 & 4.36 & 4.23 & 4.14 & 4.16 & 3.04 \\ \hline Average & 4.67 & 4.67 & 4.55 & 4.53 & 4.43 & 4.25 & 4.05 & 3.90 & 2.35 \\ \hline \end{tabular}
\end{table}
Table 8: Results of CMB-Clin on Automatic Evaluation using GPT-4 for General Models.

\begin{table}
\begin{tabular}{l c c c c c c c c c} Aspects & HutatoGPT-II-34B & BianQue-2 & DISC-MedLLM & ChuMed-Consult & MedicalGPT & DISC-MedLLM-Train & DoctorGLM & Bentsuo & ChatGLM-Med \\ \hline Fluency & 4.96 & 4.86 & 4.82 & 4.88 & 4.48 & 4.57 & 4.74 & 3.88 & 3.55 \\ Relevance & 4.61 & 3.52 & 3.24 & 3.08 & 2.64 & 2.52 & 2.00 & 2.05 & 1.97 \\ Completeness & 4.31 & 3.02 & 2.75 & 2.67 & 2.19 & 1.89 & 1.65 & 1.71 & 1.61 \\ Predicency & 4.53 & 3.60 & 3.51 & 3.30 & 2.89 & 3.19 & 2.30 & 2.58 & 2.37 \\ \hline Average & 4.60 & 3.75 & 3.58 & 3.48 & 3.05 & 3.04 & 2.67 & 2.55 & 2.38 \\ \hline \end{tabular}
\end{table}
Table 9: Results of CMB-Clin on Automatic Evaluation using GPT-4 for Medical Models.

 

construct multiple-choice questions in the form of dialogues. To enhance the robustness of our findings, we have included supplementary evaluation results in Appendix E.2.

On Different MetricsRegarding the Fluency indicator, there is not much difference between General LLMs with most LLM above 4.9, but there are still many Medical LLMs models below 4.5, indicating a lack of basic dialogue capabilities. Relevance, Completeness and Proficiency are all highly differentiated indicators, among which Completeness has the lowest average value, indicating that for medical record consultation scenarios, the completeness of the dialogue and obtaining complete information are the most difficult task.

### Agreements between Automatic and Expert Evaluation

Figure 3 demonstrates a strong agreement of resulted rankings between GPT-4 and expert evaluation, with the spearman correlation of rankings being 0.93. The rankings agree with each other except for a flip for GPT-4 and ChatGPT (dashed and solid brown lines are parallel, except for a flip at GPT-4 and ChatGPT). We also provide a more fine-grained alignment analysis in Appendix E.3. The results indicate that the automatic evaluation is highly aligned with expert evaluation.

## 6 Conclusion

Although LLMs have potential in the realm of medicine, their accurate evaluation remains pivotal for real-world applications. The introduction of the CMB benchmark, tailored to the local cultural environment in China, gives a more contextualized and comprehensive evaluation benchmark. Although not framed as a competitive leaderboard, it serves as a crucial tool for tracking LLM progress in medical domains. This might pave the way for a broader and more effective utilization of LLMs in China's medical landscape.

## Ethical Statement

In terms of Data Anonymity, all data utilized in this study primarily originate from publicly accessible examination questions and coursework exercises that have been processed by experts, and there is no personal information.

In terms of Data Quality, we fully ensure data quality from two aspects: data source and data cleaning. Through strict screening of data sources, the accuracy and authority of the exercises are ensured; through strict data processing, character errors caused by OCR and low-level errors generated during the transcription process are minimized.

In terms of Exaggerating the Abilities of the LLM, we follow the reviewer's suggestions and conducted multiple experiments to ensure the stability of the experimental results. We provide an explanation based on experimental results that provides a possible option for solving the evaluation dilemma of open-ended questions.

## Acknowledgement

This work was supported by the Shenzhen Science and Technology Program (JCYJ20220818103001002), Shenzhen Doctoral Startup Funding (RCBS20221008093330065), Tianyuan Fund for Mathematics of National Natural Science Foundation of China (NSFC) (12326608).

## Limitations

The limitations of our study are twofold. Firstly, while our benchmark encompasses various subjects in the Chinese medical domain, there remain numerous subjects that necessitate multi-modal capabilities for addressing real-world issues. Secondly, within the CMB-Clin section, we standardized the model evaluation method. However, in the real world, diverse medical conditions may require distinct evaluation criteria. Therefore, we advocate the adoption of disease-specific criteria for assessing model performance.

Figure 3: Rankings by perspective and model. Dashed lines and solid lines are the resulted rankings from expert and ChatGPT evaluation, respectively. For visual clarity, each line is shifted vertically for a small value. A model is better if it has a smaller ranking (a higher position) on the vertical axis.

 



## References

* B. Abacha, E. Agichtein, Y. Pinter, and D. Demner-Fushman (2017)Overview of the medical question answering task at TREC 2017 liveqa. In Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017, Gaithersburg, Maryland, USA, November 15-17, 2017, Vol. 500-324 of NIST Special Publication, External Links: ISBN 978-1-6953-3 Cited by: SS1.
* Proceedings of the 17th World Congress on Medical and Health Informatics, Lyon, France, 25-30 August 2019, pp. 264 of Studies in Health Technology and Informatics, pp. 25-29. Cited by: SS1.
* DeepSeek AI (2023)Deepseek llm: Let there be answers. Note: [https://github.com/deepseek-ai/DeepSeek-LIM](https://github.com/deepseek-ai/DeepSeek-LIM) Cited by: SS1.
* J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wu, B. Wu, B. Wu, B. Wu, B. Xu, J. Yang, H. Yan, J. Zhang, X. Yang, Y. Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu (2023)Qwen technical report. arXiv preprint arXiv:2309.16609. Cited by: SS1.
* B. Baichuan (2023)Baichuan 2: open large-scale language models. arXiv preprint arXiv:2309.10305. Cited by: SS1.
* Z. Bao, W. Chen, S. Xiao, K. Ren, J. Wu, C. Zhong, J. Peng, X. Huang, and Z. Wei (2023)Disc-medllm: bridging general large language models and real-world medical consultation. Cited by: SS1.
* Y. Cai, L. Wang, Y. Wang, G. de Melo, Y. Zhang, Y. Wang, and L. He (2024)Medbench: a large-scale chinese benchmark for evaluating medical large language models. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, CAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 17709-17717. Cited by: SS1.
* J. Chen, X. Wang, A. Gao, F. Jiang, S. Chen, H. Zhang, D. Song, W. Xie, C. Kong, J. Li, X. Wan, H. Li, and B. Wang (2023)Huatuogpt-ii, one-stage training for medical adaption of llms.. Cited by: SS1.
* Y. Chen, Z. Wang, X. Xing, Z. Xu, K. Fang, S. Li, J. Wang, and X. Xu (2023b)Bioanque-1.0: improving the "question" ability of medical chat model through finetuning with hybrid instructions and multi-turn doctor qa datasets. github. Cited by: SS1.
* Y. Cui, Z. Yang, and X. Yao (2023)Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177. Cited by: SS1.
* Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang (2022)GIm: general language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335. Cited by: SS1.
* D. Hendrycks, C. Burns, S. Basart, A. Zou, and C. Mazeika (2021)Mearing coding challenge competence with apps. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, Cited by: SS1.
* D. Hendrycks, C. Burns, S. Basart, A. Zou, and C. Mazeika (2021)DAN: measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, OpenReview.net, Cited by: SS1.
* D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021)Measuring mathematical problem solving with the mati dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, Cited by: SS1.
* Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. (2023)C-eval: a multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322. Cited by: SS1.
* A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. Singh Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. (2023)MIST ?d. arXiv preprint arXiv:2310.06825. Cited by: SS1.
* D. Jin, E. Pan, N. Oufatole, W. Weng, H. Fang, and P. Szolovits (2020)What disease does this patient have?: a large-scale open domain question answering dataset from medical exams. CoRRabs/2009.13081. Cited by: SS1.
* Dongfang Li, J. Yu, B. Hu, Z. Xu, and M. Zhang (2023)Explaincpe: a free-text explanation benchmark of chinese pharmacist examination. Cited by: SS1.



In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 1922-1940. Association for Computational Linguistics.
* Li et al. (2023) Jianquan Li, Xidong Wang, Xiangbo Wu, Zhiyi Zhang, Xiaolong Xu, Jie Fu, Prayag Tiwari, Xiang Wan, and Benyou Wang. 2023b. Huatuo-26m, a large-scale chinese medical qa dataset. _arXiv preprint arXiv:2305.01526_.
* Li et al. (2021) Jing Li, Shangping Zhong, and Kaizhi Chen. 2021. MLEC-QA: A Chinese Multi-Choice Biomedical Question Answering Dataset. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 8862-8874, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* A comprehensive chinese medical exam dataset. _CoRR_, abs/2306.03030.
* Pal et al. (2022) Ankit Pal, Logesh Kumar Umapathi, and Malaikann Sanarkasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In _Conference on Health, Inference, and Learning, CHIL 2022, 7-8 April 2022, Virtual Event_, volume 174 of _Proceedings of Machine Learning Research_, pages 248-260. PMLR.
* November 4, 2018_, pages 2357-2368. Association for Computational Linguistics.
* Peng et al. (2019) Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In _Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019)_.
* Sawada et al. (2023) Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kashitij Gupta, and Aran Komatsuzaki. 2023. ARB: advanced reasoning benchmark for large language models. _CoRR_, abs/2307.13692.
* Singhal et al. (2021) Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Senvirante, Paul Gamble, Chris Kelly, Nathaneal Scharli, Akakansha Chowdhery, Philip Andrew Mansfield, Blaise Aguera y Arcas, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nonad Tomasev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. 2022. Large language models encode clinical knowledge. _CoRR_, abs/2212.13138.
* Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abbabkar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, Agniesza Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kourek, Ali Safaya, Ali Tzaxur, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Desouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmuller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angelia Jiang, Angelica Chen, Ann Vuong, Aniness Gupta, Anna Gottoradi, Antonio Norelli, Annu Venkatesh, Arash Gholamiudavoidodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykt Erdem, Ayla Karakas, and et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _CoRR_, abs/2206.04615.
* Sung et al. (2021) Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang. 2021. Can language models be biomedical knowledge bases? _arXiv preprint arXiv:2109.07154_.
* Suster and Delemans (2018) Simon Suster and Walter Daelemans. 2018. Cliccr: a dataset of clinical case reports for machine reading comprehension. In _Proceeding of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_, pages 1551-1563. Association for Computational Linguistics.
* Team (2023) InternLM Team. 2023. Intermlm: A multilingual language model with progressively enhanced capabilities. [https://github.com/IntermLM/InternLM](https://github.com/IntermLM/InternLM).
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gauttier Izacard, Xavier Martinet, Marie-Anne Lacchaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Tu et al. (2023) Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, et al. 2023. Towards generalist biomedical ai. _arXiv preprint arXiv:2307.14334_.
* Wang et al. (2023a) Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. 2023a. Houtauo: Tuning llama model with chinese medical knowledge.
* Wang et al. (2023b) Haochun Wang, Chi Liu, Sendong Zhao Zhao, Bing Qin, and Ting Liu. 2023b. ChatglIm-med: \(\&\)T \(\&\)T \(\&\)X \(\&\) \(\&\) \(\&\) \(\&\) chatglIm \(\&\) \(\&\) \(\#\). [https://github.com/SCIR-HI/Med-ChatGLM](https://github.com/SCIR-HI/Med-ChatGLM).

 

Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun Dong, Junqing He, Jianheng Zhuo, Qi Yang, Yongfeng Huang, Xiayu Li, Yainghan Wu, Junyu Lu, Xinwu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao Wang, Xiaojun Wu, Zhongshan Zeng, Chongpei Chen, Ruyi Gan, and Jiaxing Zhang. 2022. Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence. _CoRR_, abs/2209.02970.
* Wang et al. (2023c) Rongsheng Wang, Yaofei Duan, ChanTong Lam, Jiesri Chen, Jiangsheng Xu, Haoming Chen, Xiaohong Liu, Patrick Cheong-Iao Pang, and Tao Tan. 2023c. Ivygpt: Interactive chinese pathway language model in medical domain. _arXiv preprint arXiv:2307.10512_.
* Welbl et al. (2018) Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. _Trans. Assoc. Comput. Linguistics_, 6:287-302.
* Yan (2023) Dong Xue* Xin Yan. 2023. Sunsimiao: Chinese medicine llm. [https://github.com/thomas-yanxin/Sunsimiao](https://github.com/thomas-yanxin/Sunsimiao).
* Xiong et al. (2023) Honglin Xiong, Sheng Wang, Yitao Zhu, Zihao Zhao, Yukiao Liu, Qian Wang, and Dinggang Shen. 2023. Doctorglm: Fine-tuning your chinese doctor is not a herculean task. _arXiv preprint arXiv:2304.01097_.
* Xu (2023) Ming Xu. 2023. Medicalgpt: Training medical gpt model. [https://github.com/shibing624/MedicalGPT](https://github.com/shibing624/MedicalGPT).
* Yang et al. (2023) Yuchen Yang, Yusheng Liao, Yu Wang, LinLin Wang, Liang He, Ya Zhang, and Yanfeng Wang. 2023. Genmedicaleval: A unified medical evaluation benchmark for chinese llms.
* Zhang et al. (2022) Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, Luo Si, Yuan Ni, Guotong Xie, Zhifang Sui, Baobao Chang, Hui Zong, Zheng Yuan, Linfeng Li, Jun Yan, Hongying Zan, Kunli Zhang, Buzhou Tang, and Qingcai Chen. 2022. CBLUE: A Chinese biomedical language understanding evaluation benchmark. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7888-7915, Dublin, Ireland. Association for Computational Linguistics.
* Zhang et al. (2023a) Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023a. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. _arXiv preprint arXiv:2306.05179_.
* Zhang et al. (2023b) Xiaotiian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023b. Evaluating the performance of large language models on goakao benchmark. _arXiv preprint arXiv:2305.12474_.
* Zheng et al. (2023) Linamin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhouhan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-jadge with mt-bench and chatbot arena. _arXiv preprint arXiv:2306.05685_.
* Zhong et al. (2023) Wanjun Zhong, Ruiziang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models.
* Zhu et al. (2020) Ming Zhu, Aman Ahuja, Da-Cheng Juan, Wei Wei, and Chandan K. Reddy. 2020. Question answering with long multiple-span answers. In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_, volume EMNLP 2020 of _Findings of ACL_, pages 3840-3849. Association for Computational Linguistics.
* Zhu et al. (2019) Ming Zhu, Aman Ahuja, Wei Wei, and Chandan K. Reddy. 2019. A hierarchical attention retrieval model for healthcare question answering. In _The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019_, pages 2472-2482. ACM.
* Zhu and Wang (2023) Wei Zhu and Xiaoding Wang. 2023. Chatmed: A chinese medical large language model. [https://github.com/micheal-wzhu/ChatMed](https://github.com/micheal-wzhu/ChatMed).
* Zhu et al. (2023) Wei Zhu, Xiaoling Wang, Huanan Zheng, Mosha Chen, and Buzhou Tang. 2023. Promptcblue: A chinese prompt tuning benchmark for the medical domain. _arXiv preprint arXiv:2310.14151_.

 

Related work

### Medical Benchmark

Medical benchmarks have evolved to broadly encompass two types of tasks based on the capabilities of the models they seek to probe: objective tasks and subjective tasks. The former typically assumes the form of multiple-choice questions Welbl et al. (2018); Pal et al. (2022); Hendrycks et al. (2021); Singhal et al. (2022), information retrieval Abacha et al. (2017); Zhu et al. (2019); Abacha et al. (2019), and cloze-style reading comprehension Suster and Daelemans (2018); Pampari et al. (2018); Zhu et al. (2020), which serve to evaluate a model's medical knowledge with unbiased accuracy. Sources for these tasks range from medical textbooks and exams to case reports Suster and Daelemans (2018), Wikipedia Welbl et al. (2018), and medical practices Hendrycks et al. (2021); Pal et al. (2022). In contrast, subjective tasks involve open-ended text generation constructed directly from consumer queries and doctor responses, often sourced from online medical forums. The task typically demands models to generate consumer-oriented replies Singhal et al. (2022); Li et al. (2023) or explanations for multiple-choice questions Liu et al. (2023). As of now, there are relatively few open-ended text generation question-answering tasks that specifically center around providing consultation based on diagnostic reports.

Few existing benchmark datasets encapsulate both task types, with PromptCBLUE Zhu et al. (2023), MultiMedQA Singhal et al. (2022) and CMExam Liu et al. (2023) sharing the closest resemblance to our work. Our dataset exceeds in size and includes questions not only from the Chinese National Medical Licensing Examination but also from various authoritative medical textbooks.

Several other datasets have been developed that contribute to the ongoing advancement of medical NLP, albeit with a narrower career focus. For instance, the ExplainCPE Li et al. (2023) is solely career-specific to pharmacists, while GenMedicalEval Yang et al. (2023), Medbench Cai et al. (2024) and Medbench_Opencompass6 are exclusive to physician careers. Our dataset includes questions not only pertaining to physicians but also to nurses, technicians, and pharmacists. These questions are derived from a mix of sources, including the Chinese National Medical Licensing Examination and various authoritative medical textbooks, thereby offering a larger and more comprehensive resource than previously available datasets.

Footnote 6: [https://medbench.opencompass.org.cn/home](https://medbench.opencompass.org.cn/home)

### Other Benchmarks of Large Language Models

The explosive growth in the number and capability of LLMs has led to a multitude of works aiming to discern their true capacity, evaluating both their general and specific abilities. General ability benchmarks include comprehensive test suites, each targeting different aspects of LLM's proficiency, ranging from handling multi-turn dialogues Zheng et al. (2023) to gauging language comprehension and reasoning abilities Srivastava et al. (2022); Zhang et al. (2023); Zhong et al. (2023).

In terms of specific abilities, several benchmarks, apart from those related to medicine, aim to evaluate different capabilities of models. ARB Sawada et al. (2023) was introduced to assess LLMs' performance in high-level reasoning tasks across multiple domains. C-Eval Huang et al. (2023) serves as the first comprehensive benchmark to evaluate the advanced knowledge and reasoning abilities of Chinese-based models. Gaokao Zhang et al. (2023), MATH Hendrycks et al. (2021), and APPS Hendrycks et al. (2021) focus on assessing LLM proficiency in complex, context-specific tasks, and code generation, respectively.

## Appendix B Dataset

Table 15, 16, 17 present a detailed directory structure of CMB-Exami. Initially, the organization is based on clinical professions and the exams commonly undertaken by these professionals, divided into six primary sections. Upon this foundation, each section is further categorized based on career progression and examination subjects. Within each sub-category, we have meticulously classified according to specific departments or courses.

### Keywords in the TCM substcalog and Medqa-USMLE

We used jieba7 and NLTK8 to perform 2-gram word segmentation on the CMB TCM subcatalog and Medqa-USMLE Jin et al. (2020) respectively, and compared the top 5 keywords (excluding words with no medical meaning). The results are shown 

in the Table 10. It can be seen that the expressions in TCM and English medicine are very different.

## Appendix C Details of Evaluated Models

In this section, we introduce and detail the models utilized in our evaluation. These models fall under three primary categories: 12 Chinese medical LLMs, 5 proprietary LLMs, and 13 publicly-available general-domain LLMs.

**Chinese medical LLMs**:

* HuatuoGPT-II: HuatuoGPT2 employs an innovative domain adaptation method to significantly boost its medical knowledge and dialogue proficiency. It showcases state-of-the-art performance in several medical benchmarks, especially surpassing GPT-4 in expert evaluations and the fresh medical licensing exams. The number of parameters of the model is 7B, 13B, and 34B.
* BianQue: It enhances its questioning ability by asking patients for more information to solve the issue that patients may not reveal all information in a single-turn conversation.
* ChatMed-Consult: It is built upon Chinese LLaMA Cui et al. (2023) using real-world questions and synthetic responses from ChatGPT.
* MedicalGPT: It is based on ZiyadLLaMa Wang et al. (2022) and adopts a four-stage training recipe, including continued pre-training, supervised fine-tuning, reward modeling, reinforcement learning.
* ChatGLM-Med: It is finetuned on ChatGLM-6B Du et al. (2022) using instruction tuning data, which are built upon CMeKG9. Footnote 9: [https://github.com/king-yyf/CMeKG_tools](https://github.com/king-yyf/CMeKG_tools)
* Bentsao: It is finetuned on LLaMa-7B Touvron et al. (2023) using the same data as ChatGLM-Med. It's the first Medical LLM trained on LLaMa.
* DISC-Med: DISC-MedLLM is a large model in the medical field specially designed for medical and health conversational scenarios.
* DISC-MedLLM-13B (CMB-Exam-Train): The model after fine-tuning DISC-MedLLM on CMB-Exam-Train. ALL of above large language models are fine-tuned for 2 epoch on the full training set with a batch size of 32, with a learning rate of 105 using Adam. The warm-up rate of cosine scheduling is set to 0.03.
* IvyGPT: An LLM based on LLaMA that is trained and fine-tuned with high-quality medical question-answer (QA) instances and Reinforcement Learning from Human Feedback (RLHF).
* Sunsimiao: Sunsimiao is fine-tuned from Baichuan-7B and ChatGLM-6B series on 100,000-level high-quality Chinese medical data.
* Doctor GLM: Based on the Chinese consultation model of ChatGLM-6B, it is fine-tuned on a single A100 80G in 13 hours. It's the first Medical LLM trained on ChatGLM.

**Proprietary models**:

* ChatGPT: Developed by OpenAI, ChatGPT, rooted in the GPT-3.5 architecture, excels in both understanding and generating natural language.
* GPT-4: Another offering from OpenAI, GPT-4employs deep learning techniques to elevate natural language processing capabilities, showcasing remarkable advancements across diverse tasks.
* JianPeiGPT:A Medical LLM developed by the Pecking Doctor team of Hangzhou Jianpei Technology Co., Ltd 10. The evaluation results were submitted and made public on December 15, 2023 after the opening of CMB. Footnote 10: [http://www.jiampecion.com/](http://www.jiampecion.com/)
* ShukunGPT: A Medical LLM developed by Shukun Technology11. The evaluation results were submitted and made public on October 23, 2023 after the opening of CMB. Footnote 11: [https://www.shukun.net/](https://www.shukun.net/)
* AntGLM-Med: A Medical LLM developed by the algorithm research group from AntGroup12. The evaluation results were submitted and made public on December 23, 2023 after the opening of CMB. Footnote 12: [https://www.antgroup.com/en](https://www.antgroup.com/en) 



**Publicly-available general-domain LLMs**:

* ChatGLM-3: The third version of ChatGLM, which is an open source, bilingual dialogue language model.
* Baichuan2-chat: An advanced variant of Baichuan-13B model, focuses on dialogue tasks, boasting 13 billion parameters for efficient and effective conversation generation. The number of parameters of the model is 7B and 13B.
* Qwen-Chat: Tongyi Qianwen large model series developed by Alibaba Cloud based on Transformer, which is trained on extremely large-scale pre-training data. The number of parameters of the model is 1.8B, 7B and 72B.
* Yi: Large language models trained from scratch by developers at 01.AI13. The number of parameters of the model is 6B and 34B.

Footnote 13: [https://www.lingyivanwu.com/](https://www.lingyivanwu.com/)

* Deepseek-llm-67B-Chat: An advanced language model comprising 67 billion parameters. It has been trained from scratch on a vast dataset of 2 trillion tokens in both English and Chinese.
* Internlm-Chat: It's pre-trained on over 2.3T Tokens containing high-quality English, Chinese, and code data. Chat version has undergone SFT and RLHF training, enabling it to better and more securely meet users' needs. The number of parameters of the model is 7B and 20B.
* Mistral14: A 7B dense Transformer, fast-deployed and easily customisable. Small, yet powerful for a variety of use cases. Supports English and code, and a 8k context window. Footnote 14: [https://mistral.ai/](https://mistral.ai/)
* Mistral-8x7B-32kseqlen: A 7B sparse Mixture-of-Experts model with stronger capabilities than MISTral 7B. Uses 12B active parameters out of 45B total. Supports multiple languages, code and 32k context window.

\begin{table}
\begin{tabular}{l|c} Dataset & Top 5 Medicine-Related Keywords \\ \hline CMB-TCM & ⚺¿

Experiment Details of CMB-Clin

### Screenshot of Human Evaluation UI

We show the screenshot of human evaluation UI in Figure 6 and Figure 7. We split the screenshot into two figures for better visual clarity.

### Prompts for Automatic Evaluation

The prompt for automatic evaluation contains task instructions, metrics, criteria, and placeholders for information to be evaluated. It is designed based on the suggestion of experts and used by both ChatGPT and GPT-4.

```
YouareanAIevaluatorspecializingins assessingthequalityofanswersprovidedbyotherlanguagemodelsYourprimarygoalistorentheanswersbasedontheirfluency,relevance,completeness,proficiencyimedicine.Usethefollowingscalestoevaluateeachcriterion:Fluency:1:Completelybrokenandunreadablesentencepieces2:Mostlybrokenwithfewreadabletokens3:Moderatelyfluentbutwithlimited vocabulary4:Mostlycoherentinexpressingcomplexsubjects5:Human-levelfluency Relevance:1:Completelyunrelatedtothesquestion2:Somerelationothequestion,butmostlyoff-topic3:Relevant,butlackingfocusorkeydetails4:Highylevent,addressingthemainaspectsofthequestion5:Directlyrelevantandpreciselytargetedothequestion Completeness:1:Extremelyincomplete2:Almostincompleetwithmitedinformation3:Moderatecompletenesswithsomeinformation4:Mostlycompletewithmostoftheinformationdisplayed5:Fullcompletewithallinformationpresented Proficiencyintemoherentec:1:Usingplainanguageswithnomedicalterminology.2:Equippedwithsommedicalknowledge butlackingin-depthdetails3:Conveyingmoderatelycomplexmedicalinformationwithclarity4:Showingsolidgraspofmedicalterminologybuthavingsomeminormistakesindetail5:Fullycorrectinallpresented medicalknowledge Youwillbeprovidedwiththefollowinginformation:-adescription-acoversationbasedonthedescription(optional)-aquestionbasedonthedescriptionandconversation-thesolutionothequestion-amodel'answertothequestion[description]{description} [endofdescription]{conversion} [history]{endofconversation} [Question]{question} [endofquestion]{solution} [endofsolution]{answer} endofanswer] Makeurstoprovidgevourevaluation resultsinJSONformatandONLYthe JSON,withseparateratingsfforeachofthementiomedcriteriaasinthefollowingexample:{fluency':3,'relevance':3,'completeness':3,'proificiency':3}
```

## Appendix E Additional Results on CMB-Clin

### Results of Expert evaluation

320 model responses are randomly sampled for this experiment due to a large number of answers to be evaluated and limited expert resources. We present the detailed results of expert evaluation in Table 11.

To show the robustness of GPT-4-as-a-judge, we also present the results of 3 independent runs for each general and medical LLMs in Table 13 and Table 14, respectively. All experiments are conducted between January 20th and January 26th, 2024.

For Medical LLMs, as shown in Table 14, although the absolute scores deviate from the results in Table 9, their rankings remain the same. For General LLMs, as shown in Table 13, the rankings remain the same except for Yi-34B. The standard deviations shown in each cell indicate the robustness of evaluating CMB-Clin with GPT-4.

 



### Agreement of Expert and GPT-4 Evaluation

Figure 5 shows the agreement between expert and GPT-4 evaluation on each perspective. The pearson correlations are all above 0.71, indicating a strong linear correlation between the two evaluation approaches.

### Pairwise Correlation of Rankings under Different Temperatures

We evaluate the results generated under each setting (_i.e.,_ under different temperatures) using ChatGPT. Then for each setting, we obtain a ranking for all models. We then calculate the pairwise spearman correlation between all sets of rankings. The results are summarized in Table 12.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Settings & Original & T-0.2 & T-0.6 & T-1.0 & T-1.5 \\ \hline Original & 1.00 & 0.95 & 0.90 & 0.87 & 0.87 \\ T-0.2 & 0.95 & 1.00 & 0.98 & 0.88 & 0.88 \\ T-0.6 & 0.90 & 0.98 & 1.00 & 0.90 & 0.90 \\ T-1.0 & 0.87 & 0.88 & 0.90 & 1.00 & 1.00 \\ \hline \hline \end{tabular}
\end{table}
Table 12: Pairwise Spearman correlations between results under different decoding temperatures. Original: results of greedy decoding (temperature 0). T-\(x\): results of using nucleus sampling under temperature \(x\).

Figure 5: Correlation of expert and automatic evaluation on CMB-Clin of each perspective with pearson correlation. The four plots show correlations in fluency, relevance, completeness and proficiency in medicine, respectively. Each plot consists of 320 data points with many overlapped. The darker a point is, the more overlapped data there are at that position. Each expert score is averaged over the three expert annotators.





\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline Aspects & GPT-4 & Yi-3dB & Owen-72B & ChatGPT & Bachuan-13B & ChaGLM3-6B & Interlin-20B & DecosekIm-67B & Mistral-8x7B \\ \hline Fluency & 4.94 \(\pm\) 0.06 & 5.00 \(\pm\) 0.00 & 5.00 \(\pm\) 0.00 & 5.00 \(\pm\) 0.00 & 4.96 \(\pm\) 0.01 & 4.96 \(\pm\) 0.01 & 4.92 \(\pm\) 0.01 & 4.82 \(\pm\) 0.07 & 2.38 \(\pm\) 0.20 \\ Relevance & 4.54 \(\pm\) 0.17 & 4.77 \(\pm\) 0.04 & 4.51 \(\pm\) 0.05 & 4.43 \(\pm\) 0.06 & 4.13 \(\pm\) 0.05 & 4.07 \(\pm\) 0.05 & 3.98 \(\pm\) 0.08 & 4.12 \(\pm\) 0.08 & 2.25 \(\pm\) 0.13 \\ Completeness & 4.10 \(\pm\) 0.24 & 4.65 \(\pm\) 0.04 & 4.06 \(\pm\) 0.06 & 4.15 \(\pm\) 0.07 & 3.85 \(\pm\) 0.04 & 3.74 \(\pm\) 0.07 & 3.48 \(\pm\) 0.06 & 2.73 \(\pm\) 0.07 & 1.56 \(\pm\) 0.08 \\ Prodicency & 4.45 \(\pm\) 0.29 & 4.76 \(\pm\) 0.06 & 4.47 \(\pm\) 0.08 & 4.45 \(\pm\) 0.07 & 4.17 \(\pm\) 0.05 & 4.12 \(\pm\) 0.05 & 4.15 \(\pm\) 0.07 & 4.18 \(\pm\) 0.11 & 2.73 \(\pm\) 0.16 \\ \hline Average & 4.51 & 4.79 & 4.51 & 4.51 & 4.28 & 4.22 & 4.13 & 3.96 & 2.23 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Results of multiple runs on CMB-Clin using GPT-4 for **general models**. Each cell in the first four rows show the mean and standard deviation of repeating the experiments 3 times. The last row shows the average values of means in their corresponding column.

Figure 6: The guideline for human evaluation and the introduction to components of user interface (in Chinese). Note that Figure 6 precedes Figure 7 in the same webpage.





\begin{table}
\begin{tabular}{|c|c|r|r|} \hline Category & Subcategory & Subject & \# Questions \\ \hline  & & Clinical Pathology & 1124 \\  & & Oral & 1074 \\  & & Otolaryngology & 952 \\  & & Rehabilitation Medicine & 461 \\  & & Ophthalrology & 951 \\  & & Neurology & 791 \\  & & Orthopedics & 939 \\  & Resident Physician & Anesthesiology & 907 \\  & & Pediatrics & 749 \\  & & Dermatology & 977 \\  & & Psychiatry & 903 \\  & & General Practice & 712 \\  & & Medical Imaging & 964 \\  & & Internal Medicine & 752 \\  & & Ultrasound & 430 \\  & & Surgery & 829 \\  & & Obstetrics and Gynecology & 800 \\  & & Pediatric Surgery & 296 \\ \hline  & & Integrated Chinese and Western Medicine & 3441 \\  & & Clinical & 5364 \\  & & Chinese Medicine & 3454 \\  & & Public Health & 2067 \\  & & Oral & 1090 \\ \hline  & & Chinese Medicine & 4490 \\  & & Public Health & 4085 \\  & & Clinical & 10241 \\  & & Oral & 1505 \\  & & Integrated Chinese and Western Medicine & 5320 \\ \hline  & & General Medicine & 3492 \\  & & Internal Oral & 858 \\  & & Orthopedics & 894 \\  & & Chinese Internal Medicine & 2896 \\ Physician & & Surgery & 5071 \\  & & Ultrasound Medicine & 2218 \\  & & Dermatology and Vennerology & 1158 \\  & & Odulatory Physiology & 983 \\  & & Internal Medicine & 5671 \\  & & Infectious Diseases & 600 \\  & & Obstetrics and Gynecology & 2864I \\  & & Cardiovascular Internal Medicine and Respiratory Internal Medicine & 617 \\  & & Oncology & 942 \\  & & acupuncture Attending in TCM & 1169 \\  & & Pathology & 1642 \\  & & Preventive Medicine & 2817 \\  & & Pediatrics & 3773 \\  & & Psychotherapy & 1393 \\ Associate Professional Physician & Radiology & 2001 \\  & & Psychiatry & 754 \\  & & Oral Restoration & 1183 \\  & & Dermatology & 909 \\  & Digestive Interval Medicine & 160 \\  & & Rehabilitation Medicine & 630 \\  & Infections Disease & 861 \\  & & Nuclear Medicine & 1250 \\  & Oral Medicine & 862 \\  & & Integrated Chinese and Western Internal Medicine & 1101 \\  & & Ophthalrology & 988 \\  & & Assemblyology & 923 \\  & Hospital Infection & 827 \\  & & Nutrition & 1009 \\  & Tuberculosisosis & 58 \\  & Critical Care Medicine & 579 \\  & Psychological Counsef & 495 \\  & Pain Medicine & 884 \\  & Neurotory & 126 \\  & Orthodontics & 578 \\  & Oral and Musculoskeletal Surgery & 367 \\  & Plastic Surgery & 187 \\  & Spethology & 81 \\  & Rheumatology and Clinseal Immunology & 37 \\  & Occupational Disease & 54 \\ \hline  & & Respiratory Internal Medicine & 1522 \\  & & Orthopedics & 1245 \\  & Endiculogy & 1326 \\  & Cardiology & 1604 \\ Advanced Professional Physicians & digestive Internal Medicine & 1577 \\  & General Surgery Senior & 1850 \\  & Gyneology and Obstetrics & 3249 \\  & General Internal Medicine & 607 \\  & General Practice & 74 \\  & Pediatrics & 65 \\ \hline \end{tabular}
\end{table}
Table 15: Catalog Structure of Physician

Figure 7: The user interface for scoring an answer (in Chinese). Note that Figure 7 follows Figure 6 in the same webpage.

 



\begin{table}
\begin{tabular}{c|c|c|c} \hline Category & Subcategory & Subject & \# Questions \\ \hline  & & Pathophysiologyology & 1455 \\  & & Medical Psychology & 932 \\  & & Biochemistry and MolecularBiology & 2402 \\  & & Cell Biology & 1399 \\  & & Medical Immunology & 2485 \\  & & Pathology & 2786 \\  & & Medical Genetics & 1369 \\  & & Parastiology & 806 \\  & & Systematic Anatomy & 1967 \\  & & Bioinformatics & 185 \\  & & Physiology & 2306 \\  & & Pharmacology & 2424 \\  & & Medical Microbiology & 1342 \\  & & Local Anatomy & 489 \\  & & Histology and Embryology & 1398 \\  & & Human Parasatology & 766 \\  & & Medical Statistics & 198 \\ \hline  & & Medical Imaging & 1858 \\  & & Radiology & 541 \\  & & Experimental Diagnostic Medicine & 548 \\  & & Neurology & 1163 \\  & & Surgery & 2164 \\  & & Dermatology and Venereology & 2168 \\  & & Pediatrics & 3760 \\  & & Nuclear Medicine & 1383 \\  & & Physical Diagnosis & 621 \\  & & Dental Pulp Disease & 346 \\  & & Basic Nursing & 978 \\  & & Diagnostics & 103 \\  & & Ultrasonic Medicine & 192 \\  & & Oral Care & 263 \\  & & Evidence-Based Medicine & 95 \\  & & Fundamental Nursing & 393 \\  & & Epidemiology & 864 \\  & & Oral Tissue Pathology & 387 \\  & & Infectious Diseases & 287 \\  & & Oral Anatomy and Physiology & 362 \\  & & Anesthesiology & 606 \\  & & Interventional Radiology & 81 \\ \hline  & & Preventive Medicine & 1926 \\ TCM and Chinese Herbal Medicine & Hygiene & 1316 \\  & & Medical Ethics & 500 \\ \hline  & & TCM Ophthalmology & 915 \\  & Essential Prescriptions Worth a Thousand Gold & 1051 \\  & Basic Theories of TCM & 2706 \\  & & TCM Diagnosis & 2036 \\ Preventive Medicine and Public Health & TCM & 1921 \\  & Warn Disease Theory & 1088 \\  & History of Chinese Medicine & 662 \\  & TCM Internal Medicine & 1738 \\  & & TCR Pediatrics & 694 \\  & Treatise on Cold Pathogenic Diseases & 1390 \\  & Lecture on Inner Canon & 456 \\ \hline \end{tabular}
\end{table}
Table 16: Catalog Structure of Undergraduate Disciplines



\begin{table}
\begin{tabular}{c|c|c|c} Category & Subcategory & Subject & \# Questions \\ \hline  & \multicolumn{1}{l|}{Practicing Nurse} & Practicing Nurse & 3303 \\ \cline{2-4}  & Licensed Practical Nurse & Licensed Practical Nurse & 4223 \\ \multirow{2}{*}{Nurse} & \multirow{5}{*}{Charge Nurse} & Pedatrici & 905 \\  & & Internal Medicine & 958 \\  & & Charge Nurse & 4558 \\  & & Surgery & 341 \\  & & Obstetrics and Gynecology & 755 \\ \cline{2-4}  & Advanced Practice Nurse & Advanced Practice Nurse & 1876 \\ \hline  & \multicolumn{1}{l|}{Rehabilitation Medicine Therapy} & 1752 \\ \multirow{5}{*}{Technician} & Medical Technician & Radiology & 1033 \\  & & Inspection & 1166 \\ \cline{2-4}  & & Oncology & 1086 \\ \cline{2-4}  & \multicolumn{1}{l|}{} & Rehabilitation Medicine Therapy & 1739 \\ \multirow{5}{*}{Medical Technologist} & Oncology & oncology & 1538 \\  & & Radiology & 1337 \\ \cline{2-4}  & & Inspection & 1458 \\ \hline \multirow{5}{*}{Technician} & \multirow{5}{*}{Supervising Technologist} & Radiation Therapy for Oncology & 1701 \\  & & Ultrasonic Medicine & 145 \\ \cline{2-4}  & & Blood Transfusion Technology & 2199 \\ \cline{2-4}  & & Microbiological Inspection & 704 \\ \cline{2-4}  & & Radiology & 1428 \\ \cline{2-4}  & & Pathology & 2407 \\ \multirow{5}{*}{Technician} & Physical Chemical Inspection & 783 \\  & & Clinical Medicine Inspection & 1378 \\ \cline{2-4}  & & Medical Record Information & 1331 \\ \cline{2-4}  & & Nuclear Medicine & 1275 \\ \cline{2-4}  & & Electrocardiology & 1021 \\ \cline{2-4}  & & Disinfection Technology & 575 \\ \cline{2-4}  & & Rehabilitation Medicine and Treatment & 948 \\ \hline \multirow{5}{*}{Graduate Entrance Exam} & Nursing & Surgical Nursing & 1112 \\  & & Basic Nursing & 902 \\ \cline{2-4}  & Political Science & Political Science & 1514 \\ \cline{2-4}  & Integrated Western Medicine & Integrated Western Medicine & 8913 \\ \cline{2-4}  & Integrated TCM & Integrated TCM & 3924 \\ \hline \multirow{5}{*}{Pharmacist} & Licensed Pharmacist & Licensed Pharmacist & 8248 \\ \cline{2-4}  & Licensed TCM Pharmacist & Licensed TCM Pharmacist & 4460 \\ \cline{2-4}  & Junior Pharmacist & Junior Pharmacist & 2720 \\ \cline{2-4}  & Junior Pharmacist Assistant & Junior Pharmacist Assistant & 3705 \\ \cline{2-4}  & Junior TCM Pharmacist & Junior TCM pharmacist Assistant & 3502 \\ \cline{2-4}  & Junior TCM Pharmacist & Junior TCM Pharmacist Assistant & 4017 \\ \cline{2-4}  & Chief Pharmacist & Chief Pharmacist & 3403 \\ \cline{2-4}  & Chief TCM Pharmacist & Chief TCM Pharmacist & 3299 \\ \hline \end{tabular}
\end{table}
Table 17: Catalog Structure of Nurse, Technician, Graduate Entrance Exam and Pharmacist