SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Overall comparison on CMExam dataset. We bold the best result and underline the second best result.
Model type	Models	size	Prediction	Reasoning
Acc (%)	F1 (%)	BLEU-1	BLEU-4	ROUGE-1	ROUGE-2	ROUGE-L
General Domain	GPT-3.5-turbo	175B	46.4±0.6	46.1±0.7	3.56±0.67	1.49±0.51	33.80±0.19	16.39±0.18	14.83±0.13
GPT-4	-	61.6±0.1	61.7±0.1	0.17±0.00	0.06±0.00	29.74±0.09	14.84±0.04	11.51±0.03
ChatGLM	6B	26.3±0.0	25.7±0.1	16.51±0.08	5.00±0.06	35.18±0.11	15.73±0.05	17.09±0.13
LLaMA	7B	0.4±0.0	0.3±0.0	11.99±0.03	5.70±0.0	27.33±0.06	11.88±0.03	10.78±0.04
Vicuna	7B	5.0±0.0	4.8±0.1	20.15±0.01	9.26±0.01	38.43±0.02	16.90±0.01	16.33±0.01
Alpaca	7B	8.5±0.0	8.4±0.0	4.75±0.00	2.50±0.00	22.52±0.00	9.54±0.00	8.40±0.00
Medical Domain	Huatuo	7B	12.9±0.0	7.0±0.0	0.21±0.00	0.12±0.00	25.11±0.08	11.56±0.04	9.73±0.02
MedAlpaca	7B	20.0±0.0	10.7±0.0	0.00±0.00	0.00±0.00	1.90±0.00	0.04±0.00	0.52±0.03
DoctorGLM	6B	-	-	9.43±0.09	2.65±0.03	21.11±0.03	6.86±0.01	9.99±0.06
PromptCLUE-base-CMExam	0.1B	-	-	18.75±0.08	6.65±0.05	40.88±0.11	21.90±0.11	18.31±0.11
Bart-base-chinese-CMExam	0.1B	-	-	23.00±0.40	10.35±0.16	44.33±0.09	24.29±0.09	20.80±0.09
Bart-large-chinese-CMExam	0.1B	-	-	26.37±0.18	11.65±0.08	44.92±0.12	24.34±0.12	21.75±0.03
BERT-CMExam	0.1B	31.8±0.2	31.2±0.2	-	-	-	-	-
RoBERTa-CMExam	0.3B	37.1±0.1	36.7±0.4	-	-	-	-	-
MedAlpaca-CMExam	7B	30.5±0.1	30.4±0.1	16.35±0.80	9.78±0.47	44.31±0.85	27.05±0.50	24.55±0.43
Huatuo-CMExam	7B	28.6±0.5	29.3±0.2	29.04±0.01	16.72±0.03	43.85±0.24	25.36±0.22	21.72±0.24
ChatGLM-CMExam	6B	45.3±1.4	45.2±1.4	31.10±0.23	18.94±0.12	43.94±0.28	31.48±0.14	29.39±0.14
LLaMA-CMExam	7B	18.3±0.5	20.6±0.5	29.25±0.23	16.46±0.10	45.88±0.04	26.57±0.04	23.31±0.02
Alpaca-CMExam	7B	21.1±0.6	24.9±0.4	29.57±0.10	16.40±0.12	45.48±0.12	25.53±0.18	22.97±0.06
Vicuna-CMExam	7B	27.3±0.5	28.2±0.3	29.82±0.03	17.30±0.01	44.98±0.16	26.25±0.13	22.44±0.09
Random	Random	-	3.1±0.2	5.1±0.3	-	-	-	-	-
Human Performance	Human volunteers	-	71.6	-	-	-	-	-	-
accurate interpretation of CMExam questions. Our findings suggest that finetuning models with CMExam enhance their performance. For instance, with an accuracy of  $45.3\%$ , ChatGLM-CMExam is comparable to GPT-3.5's performance, despite utilizing only about  $3\%$  of the parameters employed by GPT-3.5. It is noteworthy that encoder-only LLMs, such as BERT and RoBERTa, remain a robust baseline for answer prediction tasks. Their performance can par with, or even exceed, that of certain decoder-only LLMs, such as LLaMA-CMExam and Alpaca-CMExam, despite having fewer parameters.
For the solution explanation task, we observe that GPT models performed poorly on the BLEU metric, likely due to their tendency to generate short explanations. However, they exhibited an advantage on the ROUGE metric. As DoctorGLM is unable to return answer options according to the prompt, we only report its performance in the solution explanation task. Through finetuning, LLM was able to generate more reasonable explanations. For instance, ChatGLM-CMExam achieved scores of 31.10 and 18.94 on BLEU-1 and BLEU-4, respectively, and scores of 43.94, 31.48, and 29.39 on the ROUGE metrics.
Results by Disease Groups Drawing upon ICD-11 annotations (26 categories), we conducted an analysis of the performance of several LLMs across various categories. To mitigate the potential impact of random variability resulting from the number of questions, we limited our analysis to categories containing more than 100 questions. According to Table 4, LLMs have uneven performance and significant gaps in knowledge. GPT-4's accuracy ranges from  $74.4\%$  for Neo to  $44.3\%$  for TCMDP, GPT-3.5's accuracy ranges from  $63.9\%$  for Neo to  $31.0\%$  for TCMDP and ChatGLM-CMExam's accuracy ranges from  $54.7\%$  for Psy to  $42.9\%$  for RESP.
Results by Clinical Departments To compare model performance regarding the clinical department dimension (36 categories), we only analyzed categories with more than 50 questions to ensure result representativeness. Results presented in Table 5 highlight that the models show relatively high accuracy on questions associated with commonly encountered departments, such as Emergency Medicine (EM), Internal Medicine (IM) and Surgery (Surg). Their accuracy on questions associated with rarer departments, such as Traditional Chinese Medicine (TCM). There is a marked discrepancy in the average accuracy among different departments, with the highest being  $50.9\%$  and the lowest being only  $13.9\%$ . This observation suggests there are notable variations in medical knowledge and reasoning approaches among different departments. Consequently, it may be necessary to examine specific optimization strategies for different departments.
Results by Medical Disciplines Then, we evaluated LLM performance across seven medical disciplines. As depicted in Table 6, the performance of LLMs across disciplines such as Traditional Chinese Medicine (TCM), Traditional Chinese Pharmacy (TCPPharm), and Pharmacy (Pharm) was notably subpar, with all accuracy rates falling below  $42\%$ . This pattern suggests a potential deficiency
Table 4: Comparing disease classifications.
Categories	GPT-4	GPT-3.5	ChatGLM	ChatGLM-CMExam	Average
Neo	74.4±2.2	63.9±1.4	32.4±1.6	51.9±0.2	55.6±0.8
Psy	74.0±0.7	62.0±1.7	33.3±1.3	54.7±0.8	56.0±0.9
Factors	70.0±1.0	57.5±1.4	28.0±1.1	51.1±1.4	51.6±0.5
MSK	65.9±0.8	53.8±0.8	29.2±0.4	53.5±0.0	50.6±0.4
GU	69.2±0.4	52.1±1.1	30.0±0.2	49.5±0.9	50.2±0.3
Inj	65.9±2.3	45.7±1.3	37.2±2.9	49.1±1.8	49.5±1.4
Circ	68.8±0.3	49.3±0.7	30.9±0.7	47.0±0.3	49.0±0.2
Endo	70.6±1.1	49.4±1.1	25.5±0.8	46.1±0.4	47.9±0.2
Digest	67.0±1.0	48.8±1.4	26.2±0.7	49.4±1.1	47.8±0.4
InfDis	66.0±0.5	49.2±0.8	27.5±0.6	48.2±0.8	47.7±0.4
Neuro	64.4±1.2	48.7±3.1	28.6±0.4	45.3±1.3	46.7±1.1
OBST	63.5±0.3	45.0±2.4	25.7±0.9	49.4±0.3	45.9±0.5
BLOOD	69.4±0.3	45.3±1.4	18.9±1.6	43.3±0.7	44.2±0.4
Resp	62.7±0.8	44.3±1.4	24.5±0.3	42.9±0.0	43.6±0.7
N/A	60.0±0.1	46.8±0.3	24.9±0.2	42.5±0.1	43.5±0.1
TCMDP	44.3±0.9	31.0±0.6	24.2±0.4	47.9±0.0	36.9±0.6
Table 5: Comparing clinical department.
Categories	GPT-4	GPT-3.5	ChatGLM	ChatGLM-CMExam	Average
EM	67.4±0.2	49.8±0.7	36.3±0.4	50.2±0.5	50.9±0.1
OBGYN	66.4±1.0	51.7±1.5	28.6±0.5	52.0±0.0	49.7±0.3
IM	70.2±0.6	51.8±0.8	26.0±1.1	47.9±0.9	49.0±1.0
ID	67.4±1.9	49.5±3.3	26.1±1.9	49.6±3.8	48.2±1.2
Surg	63.6±0.8	49.5±1.5	28.8±0.5	47.7±0.9	47.4±1.5
ClinNutr	68.3±2.4	48.3±2.9	23.9±1.1	47.8±0.5	47.1±0.7
MedLabSci	69.2±0.6	48.3±2.0	29.0±1.5	40.8±0.6	46.8±0.2
Ped	64.5±0.0	47.2±1.4	26.7±2.1	41.9±5.5	45.1±1.7
N/A	62.6±0.2	48.6±1.1	24.6±0.4	44.3±0.9	45.0±1.0
Ophth	60.9±0.5	39.1±0.8	21.8±0.8	54.0±0.2	44.0±0.8
OccMed	61.5±4.3	38.5±1.6	31.3±4.3	41.5±3.3	43.2±2.5
DENT	54.9±2.0	41.2±1.6	27.9±0.8	43.5±0.9	41.9±1.0
TCM	43.1±1.3	31.4±1.3	24.5±1.9	45.8±4.4	36.2±0.6
ENT	41.3±0.8	28.0±0.6	29.3±0.1	26.7±0.1	31.3±0.5
ICM	33.3±0.0	11.1±15.7	0.0±0.0	11.1±15.7	13.9±4.8
in the exposure of these models to data within these categories. Conversely, disciplines such as ClinMed and Ph&PM demonstrated higher accuracy rates, likely due to the abundance of relevant data. The observed variability in performance across different disciplines underscores the distinctiveness of data characteristics and complexities inherent to each field, thereby advocating for discipline-specific model optimizations and enhancements.
Results by Competencies Evaluations based on medical competency areas aimed at a higher-level understanding of model capability in solving medical problems. As indicated in Table 7, the lowest average accuracy across LLMs was observed within the domain of mastering Medical Fundamentals (MedFund), with a meager average score of  $42.1\%$ . This result demonstrates that these models, predominantly trained on general textual data, have inadequate exposure to medical-specific data. While fine-tuning did provide some improvement, these models could benefit from additional medical scenario data to further augment their performance. It is worth highlighting that the average accuracy in the domain of Public Health Laws and Ethics (PHL) was reasonably high, notably achieving an average of  $47.6\%$ . In addition, the LLMs showcased their proficiency in accurate disease diagnosis.
Results by Question Difficulty To evaluate model performance in tackling questions of varying levels of difficulty, we conducted experiments regarding the question difficulty dimension, which was calculated based on human exam-taker performance. As shown in Table 8, there's an evident trend where model accuracies decrease as question complexity rises. This pattern suggests that more sophisticated questions demand an extensive knowledge base and complex reasoning, which are challenging for the LLMs, thus reflecting patterns observed in human performance.
Table 6: Comparing medical discipline.
Categories	GPT-4	GPT-3.5	ChatGLM	ChatGLM-CMExam	Average
ClinMed	67.9±0.1	51.4±0.4	27.3±0.3	48.9±0.4	48.8±0.7
PH&PM	68.2±0.4	52.7±1.7	26.2±0.3	47.3±1.0	48.6±0.5
ICWM	56.1±0.1	40.0±2.3	29.4±0.8	53.6±0.7	44.8±0.9
Dent	59.5±0.7	43.9±1.9	28.5±1.1	45.3±0.6	44.3±0.3
Pharm	61.1±0.4	46.3±0.5	23.2±0.2	37.0±0.1	41.9±0.3
TCM	53.5±0.4	35.9±0.2	24.1±0.3	49.1±0.0	40.6±1.1
TCPharm	45.4±1.2	35.6±0.1	24.1±1.0	43.1±0.4	37.1±0.5
Table 7: Comparing LLMs' competencies.
Categories	GPT-4	GPT-3.5	ChatGLM	ChatGLM-CMExam	Average
Diag	70.1±5.5	50.9±2.1	30.9±2.8	51.6±1.0	50.9±1.4
PHL	64.2±0.7	50.0±0.5	26.8±0.3	49.6±0.1	47.6±0.3
Treat	56.5±0.5	43.0±1.1	25.7±0.2	47.4±0.6	43.2±0.8
MeFund	58.3±0.3	44.6±0.7	23.9±0.5	41.6±0.4	42.1±0.9
N/A	54.8±0.2	30.4±0.4	23.7±0.1	38.5±0.2	36.9±0.3
Table 8: Results by question difficulty.
Categories	GPT-4	GPT-3.5	ChatGLM	ChatGLM-CMExam	Average
Easy	74.6±0.1	58.5±0.6	31.4±0.2	61.5±0.3	56.5±0.4
Manageable	63.9±0.2	47.4±0.7	25.9±0.5	46.1±0.3	45.8±0.6
Moderate	51.3±0.6	36.8±0.8	23.0±0.4	34.5±0.6	36.4±0.7
Difficult	36.4±0.9	26.2±0.7	18.9±0.5	24.3±0.9	26.5±0.6
Extremely difficult	27.2±1.0	21.4±2.2	15.8±1.0	12.2±1.1	19.1±1.1
Results by Question Length Finally, to investigate if model performance is associated with input lengths, we compared their performance regarding question lengths. Figure 3 illustrates that Large Language Models (LLMs) generally show higher accuracy with problem lengths between 60 and 90. However, their performance seems to falter with problems that are either too short or overly long. Additionally, we noticed that the effect of question length varies across different LLMs. For instance, GPT models tend to incrementally improve as the problem length expands, performing optimally within the 50 to 90 range. Conversely, ChatGLM-CMExam's performance fluctuates noticeably with varying lengths, and it tends to fall short compared to GPT models when addressing longer problems.
Figure 3: Results stratified by question length.
5 Conclusion and Discussions
In this work, we developed CMExam, a dataset sourced from the stringent Chinese National Medical Licensing Examination, featuring  $60,000+$  multiple-choice questions, with detailed explanations. CMExam ensures reliability, validity, and adherence to medical standards. It also demonstrates the practicality of employing GPT-4 to automate the annotation process, which strikes a harmonious balance between efficiency and cost-effectiveness while maintaining the desired level of accuracy and reliability of the annotation. Utilizing this large and reliable corpus, we tested several LLMs for answer selection and reasoning tasks. A performance gap was observed between LLMs and human experts, signaling the need for additional LLM research. CMExam's standardization and comprehensiveness also ensure objective evaluations of models while enabling in-depth analysis of their reasoning capabilities. The questions cover a wide spectrum of medical knowledge, augmented with five additional annotation dimensions for rigorous evaluation. This study aims to spur further exploration of LLMs in medicine by providing a comprehensive benchmark for their evaluation.
We anticipate CMExam to contribute significantly to future advancements of LLMs, particularly in handling medical question-answering tasks.
**Limitations** Firstly, while CMExam is derived from meticulously designed medical examinations, our process of excluding questions requiring non-textual information may inadvertently affect the balance of the remaining questions, potentially introducing unexpected biases. It is critical to acknowledge this aspect while interpreting any findings or analyses conducted using this dataset. Furthermore, the current BLEU and ROUGE metrics primarily evaluate the explanation task, but these measures are insufficient for assessing the reasonableness of the answer. In future work, we will incorporate human evaluation to provide a more comprehensive assessment of the models.
Ethics CMExam is a dataset derived from the Chinese National Medical Licensing Examination, which aligns with numerous datasets containing similar National Medical Licensing Examinations (Zeng et al., 2023a; Hendrycks et al., 2020; Jin et al., 2021; Pal et al., 2022; Singhal et al., 2022). We have ensured adherence to applicable legal and ethical guidelines during data collection and use. The authenticity and accuracy of the exam questions have been thoroughly verified, providing a reliable basis for evaluating LLMs. Please note that the CMExam dataset is intended for academic and research purposes only. Any commercial use or other misuse that deviates from this purpose is expressly prohibited. We urge all users to respect this stipulation in the interest of maintaining the integrity and ethical use of this valuable resource.
Societal Impacts While CMExam aims to enhance LLM evaluations in the medical field, it should not be misused for assessing individual medical competence or for patient diagnosis. Conclusions drawn from models trained on this dataset should acknowledge its limitations, especially given its single source and the specific context of the CNMLE. The use of this dataset should strictly be limited to research purposes to avoid potential misuse.
Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner-Fushman. 2017. Overview of the medical question answering task at TREC 2017 LiveQA.. In TREC. 1-12.
Asma Ben Abacha, Yassine Mrabet, Mark Sharp, Travis R Goodwin, Sonya E Shooshan, and Dina Demner-Fushman. 2019a. Bridging the Gap Between Consumers' Medication Questions and Trusted Answers.. In MedInfo. 25-29.
Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. 2019b. Overview of the medija 2019 shared task on textual inference, question entailment and question answering. In Proceedings of the 18th BioNLP Workshop and Shared Task. 370-379.
Asma Ben Abacha and Dina Demner-Fushman. 2019. A question-entailment approach to question answering. BMC bioinformatics 20, 1 (2019), 1-23.
Steven Bird. 2020. Decolonising speech and language technology. In Proceedings of the 28th International Conference on Computational Linguistics. 3504-3519.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Improving GPT-4 with  $90\%$  * ChatGPT Quality. https://lmsys.org/blog/2023-03-30-vicuna/
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021).
Qingcheng Zeng, Lucas Garay, Peilin Zhou, Dading Chong, Yining Hua, Jiageng Wu, Yikang Pan, Han Zhou, and Jie Yang. 2023a. greenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost. In Proceedings of the 2023 Conference on International Joint Conference on Artificial Intelligence (AI and Social Good Track).
Sheng Zhang, Xin Zhang, Hui Wang, Jiajun Cheng, Pei Li, and Zhaoyun Ding. 2017. Chinese medical question answer matching using end-to-end character-level multi-scale CNNs. Applied Sciences 7, 8 (2017), 767.
Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. 2018. Multi-scale attentive interaction networks for chinese medical question answer selection. IEEE Access 6 (2018), 74061-74071.
Xuanwei Zhang and Liang Xu. 2022. PromptCLUE: A zero-shot learning model that supports full Chinese tasks. https://github.com/clue-ai/PromptCLUE
Hongjian Zhou, Boyang Gu, Xinyu Zou, Yiru Li, Sam S Chen, Peilin Zhou, Junling Liu, Yining Hua, Chengfeng Mao, Xian Wu, et al. 2023b. A survey of large language models in medicine: Progress, application, and challenge. arXiv preprint arXiv:2311.05112 (2023).
Peilin Zhou, Meng Cao, You-Liang Huang, Qichen Ye, Peiyan Zhang, Junling Liu, Yueqi Xie, Yining Hua, and Jaeboum Kim. 2023a. Exploring recommendation capabilities of gpt-4v (ision): A preliminary case study. arXiv preprint arXiv:2311.04199 (2023).
Peilin Zhou, Zeqiang Wang, Dading Chong, Zhijiang Guo, Yining Hua, Zichang Su, Zhiyang Teng, Jiageng Wu, and Jie Yang. 2022. METS-CoV: A Dataset of Medical Entity and Targeted Sentiment on COVID-19 Related Tweets. In Advances in Neural Information Processing Systems, Vol. 35. 21916-21932.
Ming Zhu, Aman Ahuja, Da-Cheng Juan, Wei Wei, and Chandan K Reddy. 2020. Question answering with long multiple-span answers. In *Findings of the Association for Computational Linguistics: EMNLP* 2020. 3840-3849.
Ming Zhu, Aman Ahuja, Wei Wei, and Chandan K Reddy. 2019. A hierarchical attention retrieval model for healthcare question answering. In The World Wide Web Conference. 2472-2482.
A.1 Abbreviations, Full Names, and Translations of Additional Annotations
This section presents four tables of additional annotations that contain translation. It showcases abbreviations, full English names, and Chinese names for each group in each annotation dimension. Table 9 showcases all disease groups included in the 11th revision of the International Classification of Diseases (ICD-11). We present the disease group in the same order found on the official website. Table 12 offers a classification of 36 clinical departments derived from the Directory of Medical Institution Diagnostic and Therapeutic Categories. Table 10 presents a breakdown of medical disciplines based on the List of Graduate Education Disciplinary Majors published by the Ministry of Education of the People's Republic of China. This categorization comprises seven study majors used in universities. Table 11 provides all groups of areas of medical competency assessed in Chinese medical licensing exams.
Table 9: ICD-11 Groups
Code	Abbreviation	Full English Name	Chinese Name
01	InfDis	Certain infectious or parasitic diseases	某些感染性疾病或寄生虫病
02	Neo	Neoplasms	肿瘤
03	Blood	Diseases of the blood or blood-forming organs	血液或造血器官疾病
04	Immune	Diseases of the immune system	免疫系统疾病
05	Endo	Endocrine, nutritional or metabolic diseases	内分泌、营养或代谢疾病
06	Psy	Mental, behavioural or neurodevelopmental disorders	精神、行为或神经发育障碍
07	Sleep	Sleep-wake disorders	睡眠-觉醒障碍
08	Neuro	Diseases of the nervous system	神经系统疾病
09	Vision	Diseases of the visual system	视觉系统疾病
10	Ear	Diseases of the ear or mastoid process	耳或乳突疾病
11	Circ	Diseases of the circulatory system	循环系统疾病
12	Resp	Diseases of the respiratory system	呼吸系统疾病
13	Digest	Diseases of the digestive system	消化系统疾病
14	Skin	Diseases of the skin	皮肤疾病
15	MSK	Diseases of the musculoskeletal system or connective tissue	肌肉骨骼系统或结缔组织疾病
16	GU	Diseases of the genitourinary system	泌尿生殖系统疾病
17	Sex	Conditions related to sexual health	性健康相关情况
18	OBST	Pregnancy, childbirth or the puerperium	妊娠、分娩或产褥期
19	Peri	Certain conditions originating in the perinatal period	起源于围生期的某些情况
20	Dev	Developmental anomalies	发育异常
21	Sym	Symptoms, signs or clinical findings, not elsewhere classified	症状、体征或临床所见，不可归类在他处者
22	Inj	Injury, poisoning or certain other consequences of external causes	损伤、中毒或外因的某些其他后果
23	Ext	External causes of morbidity or mortality	疾病或死亡的外因
24	Factors	Factors influencing health status or contact with health services	影响健康状态或与
25	SpecPurp	Codes for special purposes	用于特殊目的的编码
26	TCMDP	Supplementary Chapter Traditional Medicine Conditions - Module I	补充章传统医学病证-模块1
V	FuncAssess	Supplementary section for functioning assessment	功能评定补充部分
X	ExtCodes	Extension Codes	扩展码
-	N/A	Not Applicable	不符合
Table 10: Medical Disciplines
Abbreviation	Full English Name	Chinese Name
ClinMed	Clinical Medicine	临床医学
Dent	Dentistry	口腔医学
ICWM	Integrated Chinese and Western Medicine	中西医结合
PH&PM	Public Health and Preventive Medicine	公卫预防
Pharm	Pharmacy	药学
TCM	Traditional Chinese Medicine	中医学
TCPharm	Traditional Chinese Pharmacy	中药学
Table 11: Areas of competencies
Abbreviation	Full English Name	Chinese Name
Diag	Disease Diagnosis and Differential Diagnosis	疾病诊断和鉴别诊断
MedFund	Medical Fundamentals	医学基础知识
N/A	Not Applicable	不符合
PHL	Public Health Law and Ethics	公共卫生法律伦理
Treat	Disease Treatment	疾病治疗
Table 12: Clinical Departments
Abbreviation	Full English Name	Chinese Name
AesthMed	Aesthetic Medicine	医疗美容科
Anesth	Anesthesiology	麻醉科
ClinNutr	Clinical Nutrition	临床营养科
Dent	Dentistry	口腔科
Derm	Dermatology	皮肤科
EM	Emergency Medicine	急诊医学科
EndemicD	Endemic Disease	地方病科
ENT	Otolaryngology	耳鼻咽喉科
EthnoMed	Ethnic Medicine	民族医学科
GenMed	General Medicine	全科医疗
ICM	Intensive Care Medicine	重症医学科
ID	Infectious Diseases	传染科
IM	Internal Medicine	内科
ITCWM	Integrated Traditional Chinese and Western Medicine	中西医结合科
MedLabSci	Medical Laboratory Science	医学检验科
N/A	Not Applicable	不符合
OBGYN	Obstetrics and Gynecology	妇产科
OccMed	Occupational Medicine	职业病科
Onc	Oncology	肿瘤科
Ophth	Ophthalmology	眼科
PainMed	Pain Medicine	疼痛科
PallCare	Palliative Care	临终关怀科
Path	Pathology	病理科
Ped	Pediatrics	儿科
PedHC	Pediatric Health Care	儿童保健科
PedSurg	Pediatric Surgery	儿童外科
PrevMed	Preventive Medicine	预防保健科
Psych	Psychiatry	精神科
PT	Physical Therapy	理疗科
Radiol	Radiology	医学影像科
RehabMed	Rehabilitation Medicine	康复医学科
SpecMed&MilMed	Special Medical and Military Medicine	特种医学与军事医学科
SportsMed	Sports Medicine	运动医学科
Surg	Surgery	外科
TB	Tuberculosis	结核病科
TCM	Traditional Chinese Medicine	中医科
WH	Women's Health	妇女保健
A.2 Instructions for Pre-annotation
In this section, we present instructions used to pre- annotate CMExam test set data using GPT4. As shown in Figure 4,5,6,7, we first constrained the output from GPT4 to return only specific categories. We then annotated each of the five additional annotation dimensions relevant to this study with all the category information for each dimension. Next, we provided specific prompt information and finally, we performed filtering on the GPT4 output to improve the effectiveness of pre-annotation. During the actual annotation process, specific categories and prompt information should be filled in the grey background areas.
ZH:返回格式限制为某个具体类目的名称即可。
EN:The return format is limited to the name of a specific category.
ZH共有27个类别：
某些传染病或寄生虫病；肿瘤；血液或造血器官的疾病；免疫系统疾病；内分泌、营养或代谢疾病，精神、行为或神经发育障碍；睡眠、清醒障碍；神经系统疾病；视觉系统疾病；耳或乳突疾病；循环系统疾病；呼吸系统疾病；消化系统疾病；皮肤疾病；肌肉骨骼系统或结缔组织疾病；泌尿生殖系统疾病；与性健康有关的情况；妊娠、分娩或产期；围生期某些疾患；发育异常；其他未分类的症状、体征或临床表现；损伤、中毒或外部原因引起的其他后果；影响健康状况的因素或与卫生服务的接触；传统医学疾病；功能评估补充部分；扩展代码：疾病或死亡的不确定或未知原因。