SlowGuess/NeurIPS_2023_OCR huggingface dataset:
EN: There are twenty-seven categories: Certain infectious or parasitic diseases; Neoplasms; Diseases of the blood or blood-forming organs; Diseases of the immune system; Endocrine, nutritional or metabolic diseases; Mental, behavioral or neurodevelopmental disorders; Sleep-wake disorders; Diseases of the nervous system; Diseases of the visual system; Diseases of the ear or mastoid process; Diseases of the circulatory system; Diseases of the respiratory system; Diseases of the digestive system; Diseases of the skin; Diseases of the musculoskeletal system or connective tissue; Diseases of the genitourinary system; Conditions related to sexual health; Pregnancy, childbirth or the puerperium; Certain conditions originating in the perinatal period; Developmental anomalies; Symptoms, signs or clinical findings, not elsewhere classified; Injury, poisoning or certain other consequences of external causes; Factors influencing health status or contact with health services; Traditional Medicine conditions; Supplementary section for functioning assessment; Extension codes; Uncertain or unknown cause of morbidity or mortality.
ZH:假设你是一位医疗行业专家，请判断下面这个题目属于哪个ICD-11的类别，若都不符合，则只返回"不符合"这个标签
EN: Assuming you are an expert in the medical industry, please determine which ICD-11 category this question belongs to. If none of the categories apply, return the label "N/A".
ZH:题目信息为“女34岁。月经量进行性减少，现闭经半年，泌乳3个月，首选检查项目应是：A孕激素试验，B血HCG测定，C血PRL测定，D性激素测定，E诊断性刮囊”。
EN: The question is "A 34-year-old woman has experienced progressive reduction in menstrual flow and has been amenorrheic for 6 months. She has been lactating for 3 months. Which of the following is the preferred test to perform? A. Progesterone test B. Blood HCG test C. Blood PRL test D. Sex hormone test E. Diagnostic curettage".
ZH:注意，不需要回答问题本身，只需要返回这个题目与上述27个类目中的哪个类目最相关，返回27个类目中的一个，不需要其他文字。
EN:Note that you do not need to answer the question itself, just return which of the twenty-seven categories listed above is most relevant to this question. Return only one of the twenty-seven categories, no additional words necessary.
Figure 4: Pre-annotation Instructions for Disease Groups.
ZH:共有36个类别：
预防保健科：全科医疗；内科；外科；妇产科；妇女保健；儿科；儿童外科；儿童保健科；眼科；耳鼻咽喉科；口腔科；皮肤科；精神科；传染科；肿瘤科；急诊医学科；中医科；结核病科；疼痛科；医疗美容科；地方病科；康复医学科；理疗科；运动医学科；职业病科；特种医学与军事医学科；临终关怀科；临床营养科；中西医结合科；民族医学科；麻醉科；医学检验科；病理科；医学影像科；重症医学科。
EN: There are thirty-six categories: Preventive Medicine; General Medicine; Internal Medicine; Surgery; Obstetrics and Gynecology; Women's Health; Pediatrics; Pediatric Surgery; Pediatric Health Care; Ophthalmology; Otolaryngology; Dentistry; Dermatology; Psychiatry; Infectious Diseases; Oncology; Emergency Medicine; Traditional Chinese Medicine; Tuberculosis; Pain Medicine; Aesthetic Medicine; Endemic Disease; Rehabilitation Medicine; Physical Therapy; Sports Medicine; Occupational Medicine; Special Medical and Military Medicine; Palliative Care; Clinical Nutrition; Integrated Traditional Chinese and Western Medicine; Ethnic Medicine; Anesthesiology; Medical Laboratory Science; Pathology; Radiology; Intensive Care Medicine.
ZH:假设你是一位医疗行业专家，请判断下面这个题目属于哪个DMIDTC的类别，若都不符合，则只返回"不符合"这个标签.
EN: Assuming you are an expert in the medical industry, please determine which DMIDTC category this question belongs to. If none of the categories apply, return the label "N/A"
ZH:题目信息为“女34岁。月经量进行性减少，现闭经半年，泌乳3个月，首选检查项目应是：A孕激素试验，B血HCG测定，C血PRL测定，D性激素测定，E诊断性刮宫”。
EN:The question is "A 34-year-old woman has experienced progressive reduction in menstrual flow and has been amenorrheic for 6 months. She has been lactating for 3 months. Which of the following is the preferred test to perform? A. Progesterone test B. Blood HCG test C. Blood PRL test D. Sex hormone test E. Diagnostic curettage"
ZH:注意，不需要回答问题本身，只需要返回这个题目与上述36个类目中的哪个类目最相关，返回36个类目中的一个，不需要其他文字。
EN:Note that you do not need to answer the question itself, just return which of the thirty-six categories listed above is most relevant to this question. Return only one of the thirty-six categories, no additional words necessary.
Figure 5: Pre-annotation Instructions for Clinical Departments.
ZH:共有7个类别：临床医学、口腔医学、中西医结合、公卫预防、药学、中医学、中药学。
EN: There are seven categories: Clinical Medicine, Dentistry, Integrated Chinese and Western Medicine, Public Health and Preventive Medicine, Pharmacy, Traditional Chinese Medicine, Traditional Chinese Pharmacy.
ZH:假设你是一位医疗行业专家，请判断下面这个题目属于哪个类别，若都不符合，则只返回"不符合"这个标签.
EN: Assuming you are an expert in the medical industry, please determine which category this question belongs to. If none of the categories apply, return the label "N/A"
ZH:题目信息为"女34岁。月经量进行性减少，现闭经半年，泌乳3个月，首选检查项目应是：A孕激素试验，B血HCG测定，C血PRL测定，D性激素测定，E诊断性刮宫"。
EN:The question is "A 34-year-old woman has experienced progressive reduction in menstrual flow and has been amenorrheic for 6 months. She has been lactating for 3 months. Which of the following is the preferred test to perform? A. Progesterone test B. Blood HCG test C. Blood PRL test D. Sex hormone test E. Diagnostic curettage".
ZH:注意，不需要回答问题本身，只需要返回这个题目与上述7个类目中的哪个类目最相关，返回7个类目中的一个，不需要其他文字。
EN:Note that you do not need to answer the question itself, just return which of the seven categories listed above is most relevant to this question. Return only one of the seven categories, no additional words necessary.
Figure 6: Pre-annotation Instructions for Medical Disciplines.
ZH:共有4个类别：医学基础知识、疾病诊断和鉴别诊断、疾病治疗、公共卫生法律伦理。
EN: There are four categories: Basic Medical Knowledge, Disease Diagnosis and Differential Diagnosis, Disease Treatment, and Public Health Law and Ethics.
EN:Assuming you are an expert in the medical industry, please determine which category this question belongs to. If none of the categories apply, return the label "N/A"
ZH:注意，不需要回答问题本身，只需要返回这个题目与上述4个类目中的哪个类目最相关，返回4个类目中的一个，不需要其他文字。
EN:Note that you do not need to answer the question itself, just return which of the four categories listed above is most relevant to this question. Return only one of the four categories, no additional words necessary.
Figure 7: Pre-annotation Instructions for Areas of Competencies.
A.3 Analysis of Model Generation Ability
In Figure 8, we present partial explanations generated by various models for a medical question from the CMExam dataset. Notably, GPT-4 and GPT-3.5 produce concise and sensible explanations, which may account for the lower BLUE scores. Conversely, models like Vicuna, LLaMA, and Huotuo demonstrate a more prominent repetition phenomenon, while Alpaca simply duplicates the provided options without providing an explanation.
Fine-tuning models on the CMExam dataset significantly reduces the repetition phenomenon and improves the overall reasonableness of the explanations. For instance, the ChatGLM-CMExam model analyzes each option in a similar manner to the solution explanation. However, some models still generate unreasonable explanations, as observed in LLaMA-CMExam, Alpaca-CMExam, and Vicuna-CMExam. This could be attributed to their training on generic data and lack of specific knowledge in the medical domain. This underscores the significance of training large language models with a focus on the medical domain.
Figure 8: A case study of LLMs' generated explanations.
A.4 Analysis of Model Generation Correctness
To assess the accuracy of model-generated explanations, we conducted a study using a randomly selected sample of 50 cases in which the Language Models (LLMs) correctly predicted the answers. Medical experts were then invited to manually verify the correctness of the explanations, focusing not only on the accuracy of the answer predictions but also on the quality of the accompanying explanations.
Our investigation revealed that despite the correct answer predictions by the models, certain samples exhibited errors in their corresponding explanations. These errors were categorized by the experts into three groups: explanations that were irrelevant, repeated, or inaccurate. The statistics presented in Figure 9 demonstrate that the number of samples with accurate explanations generated by the GPT models exceeded 45, accounting for over  $90\%$  of the total. However, it is important to note that both the ChatGLM and ChatGLM-CMExam models may produce some erroneous explanations, primarily consisting of inaccuracies and irrelevance. We have included examples of these incorrect explanations in Figure 10.
Figure 9: Correctness analysis.
A.5 Analysis of Few-Shot and Chain-of-Thought Prompts
In our research, we designed few-shot and chain-of-thought prompts for the answer prediction and reasoning tasks and conducted experiments on the GPT models. As shown in Table 13, our results demonstrate that while the use of few-shot or chain-of-thought prompts did not yield significant improvements in the prediction task, but there was a notable enhancement in the reasoning task.
Specifically, for the GPT-4 model, the utilization of few-shot prompts increased the BLUE-1 from 0.17 to 5.95, and the BLUE-4 from 0.06 to 2.25. Furthermore, incorporating chain-of-thought prompts further increased the BLUE-1 to 7.29. Similarly, positive effects were observed on the GPT-3.5 model, where few-shot prompts improved the BLUE-1 and BLUE-4 to 14.62 and 4.80, respectively. Additionally, the ROUGE-1, ROUGE-2, and ROUGE-L increased to 38.08, 18.35, and 18.37.
These improvements can be attributed to the fact that few-shot prompts provide examples that GPT models can reference when generating detailed explanations for each option during the reasoning process. Similarly, chain-of-thought prompts can achieve similar effects, aiding in the enhancement of model performance.
Table 13: Few-shot and chain-of-thought prompting experiment results of GPT models
Models	Prediction	Reasoning
ACC	f1	BLUE-1	BLUE-4	ROUGE-1	ROUGE-2	ROUGE-L
GPT-4	61.6%±0.1	61.7%±0.1	0.17±0.00	0.06±0.00	29.74±0.09	14.84±0.04	11.51±0.03
GPT-4_few-shot	62.0%±0.4	61.4%±0.5	5.95±0.12	2.25±0.07	37.24±0.35	19.23±0.26	17.24±0.07
GPT-4_cot	61.6%±0.9	61.4%±0.9	7.29±0.71	2.20±0.25	35.85±0.78	16.79±0.83	17.18±0.30
GPT-3.5	46.4%±0.0	46.2%±0.1	3.56±0.08	1.49±0.06	33.80±0.11	16.39±0.05	14.83±0.13
GPT-3.5_few-shot	45.3%±0.6	44.9%±0.6	14.62±0.16	4.80±0.06	38.08±0.44	18.35±0.16	18.37±0.29
GPT-3.5_cot	47.9%±0.7	47.7%±0.7	13.47±0.52	3.69±0.18	36.47±0.42	16.41±0.24	17.82±0.31
e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e
A.6 Data statistics
Questions in CMExam have a median length of 17 (Q1: 12, Q3: 32). Regarding solution explanations, the median length is 146 tokens (Q1: 69, Q3: 247). Table 14 shows more basic statistics of CMExam,
Table 14: Basic statistics of CMExam. Q: questions; E: explanations; Q1/3: the first/ third quantile.
Train	Dev	Test	Total
Question #	54,497	6,811	6,811	68,119
Vocab	4,545	3,620	3,599	4,629
Max Q tokens	676	500	585	676
Max E tokens	2,999	2,678	2,680	2,999
Avg Q tokens	29.78	30.07	32.63	30.83
Avg E tokens	186.24	188.95	201.44	192.21
Median (Q1, Q3) Q tokens	17 (12, 32)	18 (12, 32)	18 (12, 37)	18 (12, 32)
Median (Q1, Q3) E tokens	146 (69, 246)	143 (65, 247)	158 (80, 263)	146 (69, 247)
A.7 Guidelines for Expert-Annotation
During the annotation phase, we invited one expert physician from the Second Affiliated Hospital of Zhejiang University and one senior doctoral student from Zhejiang University School of Medicine to carry out the annotations. The expert physician has over two years of clinical experience. The annotation guidelines have the following sections:
1. Comprehensive Question Understanding: Prior to initiating the annotation process, meticulously comprehend the medical question, ensuring a holistic grasp of its context and significance.
2. Subject Categorization: Identify the precise subject or medical field that the question pertains to, such as cardiology, pediatrics, or pathology.
3. Principal Symptoms or Medical Conditions: Ascertain and pinpoint the primary symptoms or medical conditions expounded in the question.
4. Examination of Pertinent Factors: Scrutinize the question for any associated factors that might be present, including the severity of the ailment, its etiology, and patient history given in the question.
5. Examination of Pertinent Factors: Scrutinize the question for any associated factors that might be present, including the severity of the ailment, its etiology, and patient history given in the question.
6. Appropriate Classification System Usage: Use the accurate classification system for annotation in alignment with the determined subject and symptoms. Suitable systems could encompass the 11th revision of the International Classification of Diseases (ICD-11), the Directory of Medical Institution Diagnostic and Therapeutic Categories (DMIDTC), and others.
7. Addressing Multiple Annotations: In scenarios where the question encompasses multiple symptoms or medical conditions, opt for the most related classification for annotation.
8. Ensuring High-Quality Annotations: Adhere to the guidelines and definitions within the chosen classification system. This diligence helps avert subjectivity and ambiguity, fostering precision in the annotations.
9. Navigating Queries and Uncertainties: Should any doubts or uncertainties emerge during the annotation process, consult the official documents and glossaries of the chosen classification system. Engaging in discussions with professionals is also advised to achieve clarity.
10. Resolving Discrepancies: When disagreements emerge between annotators, a collaborative discussion shall be initiated. The objective is to reach a consensus and unify the annotation decision.
A.8 Prompt strategies for Pre-Annotation
During the experimental process, we indeed tried different prompts to enable GPT to better understand and complete the annotation task. The specific strategies were as follows:
1. Without ICD-11 Category Instructions: We did not provide detailed ICD-11 category information as instruction but directly supplied the question information and asked GPT to respond. Under this setup, a significant portion of the categories returned by GPT did not strictly belong to ICD-11 classifications, yielding unsatisfactory results.
2. Batch Processing for Cost Efficiency: Initially, we concatenated multiple questions and, through a single dialogue, had GPT return annotations for multiple questions. Under this setup, expert validation showed that the accuracy of GPT's annotations was relatively low.
3. Consistency in Formatting: When no format guidance was given, GPT's return format was inconsistent, resulting in a higher parsing cost. Hence, after multiple trials, we eventually opted for more rigorous format guidance.
Our annotation process was carried out in two stages: First, GPT conducted an initial round of pre-annotation. Subsequently, we invited an expert physician from the Second Affiliated Hospital of Zhejiang University and a doctoral student from Zhejiang University School of Medicine to annotate. The expert physician had over two years of clinical experience. In instances where there were disagreements in annotations, both annotators would discuss and eventually arrive at a consensus.