EMPEC: A Comprehensive Benchmark for Evaluating Large Language Models Across Diverse Healthcare Professions
Authors: Zheheng Luo, Chenhan Yuan, Qianqian Xie, Sophia Ananiadou
Accepted at: acl25

EMPEC: A Comprehensive Benchmark for Evaluating Large Language Models Across Diverse Healthcare Professions

Zheheng Luo◊*, Chenhan Yuan◊*, Qianqian Xie†♣
Sophia Ananiadou◊
◊The University of Manchester †School of Artificial Intelligence, Wuhan University

Abstract
Recent advancements in Large Language Models (LLMs) show their potential in accurately answering biomedical questions, yet current healthcare benchmarks primarily assess knowledge mastered by medical doctors, neglecting other essential professions. To address this gap, we introduce the Examinations for Medical Personnel in Chinese (EMPEC), a comprehensive healthcare knowledge benchmark featuring 157,803 exam questions across 124 subjects and 20 healthcare professions, including underrepresented roles like Optometrists and Audiologists. Each question is tagged for release time and source authenticity. We evaluated 17 LLMs, including proprietary and open-source models, finding that while models like GPT-4 achieved over 75% accuracy, they struggled with specialised fields and alternative medicine. Notably, we find that most medical-specific LLMs underperform their general-purpose counterparts in EMPEC, and incorporating EMPEC’s data in fine-tuning improves performance. In addition, we tested LLMs on questions released after the completion of their training to examine their ability in unseen queries. We also translated the test set into English and simplified Chinese and analyse the impact on different models. Our findings emphasise the need for broader benchmarks to assess LLM applicability in real-world healthcare, and we will provide the dataset and evaluation toolkit for future research. Our data and code are in https://github.com/zhehengluoK/eval_empec.

1 Introduction
Recent advancements in Large Language Models (LLMs) have demonstrated the potential of LLM-based Artificial Intelligence (AI) in providing accurate answers to questions about world knowledge. These advancements are reflected in a series of studies and models, including but not limited to GPT-4, Gemini, Mistral, and Llama series (OpenAI, 2023; Google, 2023; Jiang et al., 2023; Touvron et al., 2023). To benchmark the internal knowledge of LLMs, multiple datasets (Hendrycks et al., 2020; Clark et al., 2018; Lin et al., 2021) have been introduced, focusing on their ability to measure and respond with accurate information. More recently, there has been a significant push towards adapting LLMs for use in the biomedicine domain (Singhal et al., 2023; Chen et al., 2023a; Xie et al., 2024a), exploring the possibility of deploying these models in real healthcare scenarios. To assess the effectiveness of LLMs in healthcare, considerable effort (Cai et al., 2023; Wang et al., 2023b; Jin et al., 2021; Kasai et al., 2023; Pal et al., 2022) has been invested in benchmarking their capabilities. Among these efforts, several studies (Cai et al., 2023; Wang et al., 2023b) have incorporated long-form diagnostic questions into their benchmarks to more evaluate the models’ capacities to function akin to real physicians. In another line of work, multiple-choice questions have emerged as a straightforward and objective means of evaluation (Jin et al., 2021; Kasai et al., 2023; Pal et al., 2022). These studies mostly utilize questions from medical licensing exams, research papers, and textbooks to assess LLMs’ knowledge and suitability to serve in a physician-like role.

Existing medical benchmarks are limited in scope and authenticity In real-world healthcare environments, medical doctors, while integral, represent only a fraction of the entire healthcare system. The International Standard Classification of Occupations (ISCO) by the International Labour Organization categorises health professionals into two Sub-major Groups, with medical doctors listed alongside ten other minor occupational groups. These include Nursing and Midwifery Pro-
Figure 1: Distribution of professions in the EMPEC dataset. The left panel illustrates the total number of questions attributed to each healthcare profession within the dataset. The right panel provides a visual representation of the proportionate distribution of questions across the various professions.

fessionals and Paramedical Practitioners, among others. Each minor group comprises at least two unit groups, which in turn consist of multiple professions. This diversity in healthcare roles highlights that the expertise of medical doctors is just a subset of the broader healthcare knowledge spectrum. Nevertheless, as shown in Table 1, existing medicine-related benchmarks face challenges in diversity. For instance, MedQA (Jin et al., 2021) and MedBench (Cai et al., 2023) contain questions solely for physicians. MedMCQA (Pal et al., 2022) gathers questions for postgraduate medical students. CMB (Wang et al., 2023b) and CMExam (Liu et al., 2024) have made limited attempts to expand coverage beyond physicians to a maximum of five professions, leaving many health professionals unrepresented. Consequently, there remains a significant gap in our ability to assess the performance of LLMs across the broader spectrum of medical contexts. As a result, there remains a significant gap in our ability to assess the performance of LLMs across the broader spectrum of medical contexts. To more comprehensively gauge the effectiveness and applicability of LLMs in healthcare, it is crucial to expand the scope of benchmarks to include a wider range of professions within the healthcare system. Furthermore, despite being collected from credible sources, Table 1 reveals that most existing works cannot reference an authoritative source for their data, casting doubt on the authenticity of the collected problems. Additionally, essential metadata, such as release dates, are often missing. Therefore, the results from existing benchmarks fail to accurately reflect LLMs’ performance across the entire healthcare system.

In response to the identified research gap, we introduce EMPEC, a comprehensive large-scale healthcare knowledge benchmark. EMPEC comprises 157,803 exam questions spanning 124 subjects across 20 healthcare professions. This comprehensive benchmark goes beyond the commonly assessed professions such as Physicians and Nurses, to include occupations like Optometrist and Audiologist often overlooked in previous assessments, thereby filling a critical void in existing benchmarks. The benchmark is originally in traditional Chinese; to broaden the applicability of our work beyond Chinese-speaking regions, we add an English translation of the test set generated by GPT-4. EMPEC stands out not only for its substantial size and extensive coverage but also for its authoritative and time-sensitive features. Each question within EMPEC is tagged with its release time, ensuring that the benchmark continuously integrates the latest questions from the source as soon as they are made available. This feature effectively mitigates the risk of data contamination (Nori et al., 2023) during evaluation processes, thereby maintaining the benchmark’s relevance and accuracy over time. Furthermore, the provenance of every question in EMPEC is documented, with each item linked to its original release. This verification ensures the benchmark’s reliability and authenticity, positioning it as a critical tool for the evaluation of LLMs within the healthcare sector.

We conducted extensive experiments on 17
Table 1: Review of existing healthcare-related benchmarks, \( \checkmark \) represents the dataset that has the feature and \( \times \) represents it does not. Source-verifiable means the dataset can be verified from its acclaimed source. Trad and simp are short for traditional and simplified respectively.

<table>
  <tr>
    <th>Dataset</th>
    <th>Resources</th>
    <th>Language</th>
    <th>#Professions</th>
    <th>#Question</th>
    <th>Source-verifiable</th>
  </tr>
  <tr>
    <td>CMExam</td>
    <td>Gov Publication</td>
    <td>Simp Chinese</td>
    <td>5</td>
    <td>68,119</td>
    <td>\( \times \)</td>
  </tr>
  <tr>
    <td>CMB</td>
    <td>Open database</td>
    <td>Simp Chinese</td>
    <td>4</td>
    <td>280,839</td>
    <td>\( \times \)</td>
  </tr>
  <tr>
    <td>MedQA</td>
    <td>Text Books</td>
    <td>English, Chinese</td>
    <td>1</td>
    <td>61,097</td>
    <td>\( \times \)</td>
  </tr>
  <tr>
    <td>MedMCQA</td>
    <td>Open website&Books</td>
    <td>English</td>
    <td>-</td>
    <td>193,155</td>
    <td>\( \checkmark \)</td>
  </tr>
  <tr>
    <td>MedBench</td>
    <td>Gov Publication</td>
    <td>Simp Chinese</td>
    <td>1</td>
    <td>40,041</td>
    <td>\( \times \)</td>
  </tr>
  <tr>
    <td>EMPEC</td>
    <td>Gov Publication</td>
    <td>Trad Chinese</td>
    <td><b>20</b></td>
    <td>157,803</td>
    <td>\( \checkmark \)</td>
  </tr>
</table>

LLMs, including two proprietary models and 14 open-source models, evaluating their zero-shot performance. Additionally, we include one fine-tuned model using the training data from the EMPEC dataset. Our analysis contrasted the performance of medical domain LLMs with general LLMs, as well as models primarily trained on Chinese versus English data. Furthermore, we evaluated the models on both the full test set and a subset comprising the most recent questions, which could be reliably assumed to be absent from the models’ training data. Finally, we compared the models’ performance on questions in traditional Chinese with their performance on simplified Chinese and English versions of the same questions. Our findings on the experimental results are as follows:

I) GPT-4 leads the evaluation by achieving more than 75% accuracy while open-source LLMs are catching up with the frontier. II) While leading LLMs perform well in frequently encountered professions, EMPEC shows that they struggle with more specialized fields knowledge such as Dentists and Optometrists, and alternative medicine like Traditional Chinese Medicine Practitioners. III) Previous LLMs trained for the medical domain unexpectedly show inferior performance compared to their general-purpose counterparts. Moreover, incorporating EMPEC data into training significantly boosts model performance. IV) The results on questions released later than the time cutoff of several models reveal consistency with the overall performance trends observed in the EMPEC test set, suggesting that the models’ performance on the test set can be extrapolated to predict their effectiveness in addressing unseen healthcare-related queries. V) The conversion from traditional to simplified Chinese characters appears to have a negligible impact on the performance of the models. While on the translated English version, models primarily trained in English enjoy an improvement. Despite the change in language, the accuracy trends by profession remain consistent.

2 Related work

Healthcare Knowledge Benchmark  Multiple endeavours have been made to evaluate the medical knowledge in LLMs to propose question-answering(QA) datasets. Long-form QA datasets such as MedicationQA (Abacha et al., 2019), BioASQ (Krithara et al., 2023) are usually drafted and annotated by domain experts to assess LLMs via comparing their free-format response to the human written answers. Medical exams have become an ideal source for collecting medical QA materials due to their endorsement by national institutions and under rigorous scrutiny. Several benchmarks, including MedQA (Jin et al., 2021), CMExam (Liu et al., 2024), IgakuQA (Kasai et al., 2023), and Polish MFE (Rosol et al., 2023) leverage medical exams from different nations to test knowledge, focussing primarily on physicians for the respective languages. MedBench (Cai et al., 2023) went beyond Chinese MLE to advanced physician exams such as the Resident Standardisation Training Examination, the Doctor In-Charge Qualification Examination, and real-world clinic cases encompassing examinations. CMB (Wang et al., 2023b) expands the assessment of medical knowledge by introducing exams for nurses, technicians, and pharmacists. Though of high quality, these existing benchmarks mostly focus on assessing a small proportion of health professions represented by physicians, leaving the knowledge mastered by other healthcare personnel unchecked, limiting these datasets’ capability to comprehensively assess LLMs’ knowledge in the entire healthcare system.

Knowledge-related Benchmarks for LLM  Given the advance of LLMs, there was a multitude
Figure 2: An example of questions in EMPEC, texts in blue are English translations of the original Chinese question and answers.

of works focussing on benchmarking knowledge within the models. MMLU (Hendrycks et al., 2020) and ARC (Clark et al., 2018) collected various questions from a diverse set of subjects aiming to comprehensively assess the world knowledge of models. GSM8K (Cobbe et al., 2021) collected grade school math word problems to benchmark models’ mathematics reasoning ability. FinBen (Xie et al., 2024b) aggregated 35 datasets across 23 financial tasks aiming to fully assess LLMs’ knowledge and capacity in finance. LawBench (Fei et al., 2023) compiled datasets to evaluate LLM’s legal capacities from three cognitive levels. However, in healthcare, most existing benchmarks concentrate on examining physicians’ expertise, being unable to assess models’ knowledge in the whole healthcare system.

3 The EMPEC Dataset

Data Collection and Pre-processing EMPEC compiles officially published past exams for healthcare professionals in the Professional and Technical Examinations of Taiwan, Republic of China1. Each profession’s exams consist of multiple related subjects designed to comprehensively assess the candidates medical knowledge and clinical skills for the profession. Tests for some professions are conducted biannually, and we have gathered exams held from 2011 to 2024 of issuance to streamline training and evaluation processes. To ensure the quality of EMPEC, we conducted the following pre-processing steps: 1) Excluding exams for questions requiring non-textual information such as images or tables; 2) For questions that shared the same premise, we added the premise for the all following questions. 3) We remove questions belonging to subjects like "Traditional Chinese literature" and "Pharmaceutical Administration and Regulations" to rule out problems that are irrelevant or only applied in the local region to calibrate EMPEC’s concentration in healthcare knowledge. 4) We use Data-Juicer (Chen et al., 2024) to deduplicate questions that are highly similar to each other. Detailed process configuration is given in Appendix E. As a result, we have curated a collection of 157,803 multiple choice questions. In addition, to investigate whether the model’s performance would be affected by language, we employed GPT-4 2 to translate the test set to English using a simple prompt as "Translate the following question into English". We set the temperature at 0 to ensure reproducibility and asked domain experts to carefully review a subset of the English translations to assess their quality (see C for further details on the evaluation process). We also adopted zhconv3 to convert the questions into simplified Chinese. This extensive collection covers 20 medical professions across 124 subjects. An example of questions in EMPEC is shown in Figure 2. Lastly, we will follow this pipeline to collect the latest questions issued from the officials.

Dataset Statistics In EMPEC, each question presents four options, with only one being the correct choice. Figure 1 illustrates the distribution of EMPEC questions in various professions. There are 11 professions taking more than 6% per cent, 7 professions taking about 3%. The questions for Optometrists and Public Health Specialists take less than 1% since the exams for the two professions were recently introduced into the national exams. In Appendix D, we explicitly show the subjects examined and the number of questions in each profession. The average subject of examination for each profession is more than 6, showing the diverse assessment of the expertise of each profession. Thus, EMPEC offers a more comprehensive coverage compared to prior studies (Wang et al., 2023b; Jin et al., 2021; Cai et al., 2023) which primarily focused on physicians. We split the dataset into 3 subsets, train, validation, and test. We first split the test set from the full dataset via stratified sampling by profession to ensure the same distribution. Then, we sampled 10 questions per profession

1https://wwwq.moex.gov.tw/exam/wFrmExamQandASearch.aspx
2https://platform.openai.com/docs/models/gpt-4o
3https://github.com/gumblex/zhconv
from the rest data to form the validation set to facilitate a few-shot evaluation. The remaining data serves as the training subset, and the statistics are shown in Table. 2.

Table 2: Statistics of each split of EMPEC.

<table>
  <tr>
    <th>Split</th>
    <th>#Profession</th>
    <th>#Subject</th>
    <th>#Year</th>
    <th>#Question</th>
  </tr>
  <tr>
    <td>Train</td>
    <td>20</td>
    <td>124</td>
    <td>14</td>
    <td>149,603</td>
  </tr>
  <tr>
    <td>Validation</td>
    <td>20</td>
    <td>100</td>
    <td>14</td>
    <td>200</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>20</td>
    <td>124</td>
    <td>14</td>
    <td>8,000</td>
  </tr>
</table>

Dataset Characteristics EMPEC has several advantages over existing medical QA benchmarks: 1) Comprehensive Healthcare Knowledge: Unlike MedBench (Cai et al., 2023) and CMExam (Liu et al., 2024), which mainly focus on physicians, EMPEC evaluates 20 healthcare professions, offering broader coverage. Existing benchmarks cover fewer than five occupations. 2) Extensive Question Pool: EMPEC includes over 157K questions from 124 subjects, surpassing CMExam, MedQA, and MedBench. 3) High-Quality Assurance: The questions are crafted by experts and vetted by Taiwan’s Ministry of Examination. 4) Timestamp and Self-Growth: EMPEC tracks question release years and integrates new questions, allowing for assessment of data contamination in LLM training.

4 Benchmark

4.1 Tested Models

General LLMs We select leading proprietary LLMs GPT-4-turbo and GPT-3.5-turbo aiming to set an upper bound for LLMs’ performance on EMPEC. For open-source models, We choose leading English-majored LLMs including Llama-3 (AI@Meta, 2024), Mistral (Jiang et al., 2023) as well as five Chinese-majored LLMs Yi (Young et al., 2024), Qwen1.5 (Bai et al., 2023), Baichuan2 (Yang et al., 2023), InternLM2 (Cai et al., 2024), and Ziya-Llama(IDEA-CCNL, 2021). Both Yi and Qwen1.5 have shown performance on par with or better than GPT-4 on multiple Chinese benchmarks.

Medical Domain LLMs As further training LLMs on in-domain data usually brings improvement in domain-specific tasks (Wang et al., 2023a; Chen et al., 2023b), we further examined the LLMs that have been trained or fine-tuned on biomedical texts. BioMistral-7B (Labrak et al., 2024) is based on Mistral 7B Instruct v0.1 and further pre-trained on PubMed Central data. HuatuoGPT2 (Chen et al., 2023a). The 13B and 34B versions of HuatuoGPT2 are pre-trained on Baichuan2-13B and Yi-34B respectively using more than 5 million synthetic medical instruction tuning data. HuatuoGPT2-13B is reported to outperform GPT3.5 by more than 20% on two Chinese medical benchmarks CMB (Wang et al., 2023b) and CMExam (Liu et al., 2024). MMedLM2 (Qiu et al., 2024), based on InternLM2, is further pre-trained on 22.5 billion tokens of health-related text across 6 languages including Chinese. MedGPT (Xu, 2023) is based on Ziya-Llama-13B (IDEA-CCNL, 2021), which is a Llama variant with an expanded vocabulary and further pre-trained on Chinese and English texts. MedGPT, on top of Ziya-Llama-13B, is fine-tuned on 240 million Chinese and English medical instruction-tuning data. Qwen1.5-7B-SFT, we fine-tuned a Qwen1.5-7B using the training data of EMPEC.

Specifically, we contrast the medical LLMs with their general counterparts like Baichuan2 and Yi for HuatuoGPT2, ZiyaLlama for MedGPT, InternLM2 for MMedLM2, Qwen1.5-Chat for Qwen1.5-SFT to explore the effect of the domain-adaption training of these models.

4.2 Evaluation settings

We use the 0125 version of GPT-turbo models. To facilitate zero-shot prompting, we choose the "Chat" or "Instruct" versions of Llama-3, Mistral(v0.2), Baichuan2, Qwen1.5, and Yi. We fine-tuned Qwen1.5-7B on the EMPEC training subset for 3 epochs using the learning rate of 1e-4 on 2 A100 GPUs. We use vLLM 0.4.1 (Kwon et al., 2023) and enable greedy decoding to ensure the stability and reproducibility of the results. The prompt used in zero-shot prompting and fine-tuning is in Appendix A. The evaluation procedure follows common practice in previous work (Bai et al., 2023; Huang et al., 2023) to extract the correct option from the model’s response. We allow the model to generate up to 256 tokens for each question. Moreover, as existing benchmarks suffer the problem of data leakage (Nori et al., 2023; Zhang et al., 2024), we explicitly conduct the evaluation of the question from the exams held in 2024 on models of which the pre-training data cut-off is before 2024. During our evaluation, questions that are filtered by the API provider or models fail to respond with the correct choice are deemed incorrect. A baseline is set by randomly choosing a choice from A to D. We did not examine few-shot as most
Table 3: Accuracy of each model on the test set of EMPEC, split by profession. C stands for Chat version and I stands for Instruction version. Bch2 is short for Baichuan2.

<table>
  <tr>
    <th rowspan="2">Split</th>
    <th colspan="14">Accuracy</th>
  </tr>
  <tr>
    <th>GPT4</th><th>GPT3.5</th><th>Qwen1.5</th><th>Llama3-I</th><th>Yi-C</th><th>HuatuoGPT2</th><th>Bch2-C</th><th>Mistral-I</th><th>InternLM2</th><th>MMedLM2</th><th>MedGPT</th><th>Ziya</th><th>BioMistral</th>
  </tr>
  <tr>
    <td>Nurse</td><td>82.75</td><td>63.53</td><td>77.65</td><td>61.18</td><td>69.22</td><td>70.20</td><td>52.75</td><td>73.53</td><td>48.04</td><td>41.96</td><td>17.25</td><td>55.10</td><td>40.20</td><td>62.16</td><td>58.43</td><td>24.31</td><td>29.41</td><td>18.82</td>
  </tr>
  <tr>
    <td>Dentist</td><td>69.27</td><td>49.83</td><td>53.30</td><td>47.22</td><td>33.99</td><td>57.47</td><td>49.31</td><td>56.60</td><td>34.38</td><td>33.85</td><td>18.40</td><td>42.71</td><td>35.24</td><td>46.18</td><td>47.22</td><td>24.65</td><td>23.06</td><td>18.75</td>
  </tr>
  <tr>
    <td>Midwife</td><td>85.43</td><td>67.42</td><td>72.30</td><td>58.01</td><td>68.47</td><td>83.62</td><td>68.29</td><td>73.87</td><td>46.34</td><td>41.81</td><td>18.48</td><td>49.07</td><td>35.11</td><td>53.16</td><td>52.89</td><td>20.91</td><td>20.43</td><td>16.08</td>
  </tr>
  <tr>
    <td>Physician</td><td>89.90</td><td>67.42</td><td>72.30</td><td>58.01</td><td>68.47</td><td>83.62</td><td>68.29</td><td>73.87</td><td>46.34</td><td>41.81</td><td>18.48</td><td>49.07</td><td>35.11</td><td>53.16</td><td>52.89</td><td>20.91</td><td>20.43</td><td>16.08</td>
  </tr>
  <tr>
    <td>Dietitian</td><td>74.76</td><td>53.40</td><td>70.55</td><td>52.75</td><td>61.49</td><td>65.37</td><td>47.93</td><td>63.43</td><td>39.16</td><td>44.66</td><td>20.71</td><td>50.49</td><td>37.54</td><td>56.96</td><td>52.43</td><td>29.77</td><td>20.71</td><td>18.77</td>
  </tr>
  <tr>
    <td>Pharmacist</td><td>75.86</td><td>58.15</td><td>60.36</td><td>49.09</td><td>57.14</td><td>72.03</td><td>53.32</td><td>64.59</td><td>42.05</td><td>45.88</td><td>15.90</td><td>45.07</td><td>39.44</td><td>56.74</td><td>51.71</td><td>30.38</td><td>26.76</td><td>21.73</td>
  </tr>
  <tr>
    <td>Audiologist</td><td>72.18</td><td>53.11</td><td>63.05</td><td>44.37</td><td>54.21</td><td>59.51</td><td>44.71</td><td>57.39</td><td>34.31</td><td>34.13</td><td>19.37</td><td>40.14</td><td>33.45</td><td>49.30</td><td>44.37</td><td>23.59</td><td>20.77</td><td>13.03</td>
  </tr>
  <tr>
    <td>Orthodontist</td><td>66.18</td><td>52.38</td><td>61.57</td><td>47.35</td><td>54.15</td><td>61.15</td><td>47.38</td><td>54.18</td><td>36.02</td><td>31.62</td><td>17.62</td><td>43.87</td><td>38.12</td><td>53.07</td><td>52.49</td><td>21.26</td><td>25.29</td><td>14.10</td>
  </tr>
  <tr>
    <td>Veterinarian</td><td>78.35</td><td>55.56</td><td>63.22</td><td>49.81</td><td>63.41</td><td>63.22</td><td>50.38</td><td>60.73</td><td>38.70</td><td>36.02</td><td>17.62</td><td>43.87</td><td>38.12</td><td>53.07</td><td>52.49</td><td>21.26</td><td>25.29</td><td>14.10</td>
  </tr>
  <tr>
    <td>Speech Therap</td><td>69.05</td><td>46.43</td><td>54.76</td><td>40.87</td><td>59.13</td><td>52.38</td><td>36.90</td><td>51.19</td><td>23.81</td><td>33.33</td><td>19.05</td><td>38.49</td><td>29.37</td><td>40.87</td><td>41.67</td><td>27.78</td><td>22.22</td><td>15.87</td>
  </tr>
  <tr>
    <td>Dental Tech</td><td>59.39</td><td>41.12</td><td>58.88</td><td>50.25</td><td>53.81</td><td>46.70</td><td>44.16</td><td>52.28</td><td>36.55</td><td>35.03</td><td>13.71</td><td>38.07</td><td>23.86</td><td>44.67</td><td>43.65</td><td>21.37</td><td>26.90</td><td>17.77</td>
  </tr>
  <tr>
    <td>Phys Therap</td><td>70.25</td><td>51.43</td><td>52.61</td><td>52.27</td><td>58.48</td><td>60.01</td><td>52.10</td><td>59.33</td><td>37.98</td><td>36.13</td><td>19.50</td><td>46.89</td><td>36.47</td><td>53.28</td><td>50.76</td><td>28.24</td><td>24.54</td><td>19.66</td>
  </tr>
  <tr>
    <td>Pharm Tech</td><td>67.43</td><td>51.43</td><td>52.61</td><td>52.27</td><td>58.48</td><td>60.01</td><td>52.10</td><td>59.33</td><td>37.98</td><td>36.13</td><td>19.50</td><td>46.89</td><td>36.47</td><td>53.28</td><td>50.76</td><td>28.24</td><td>24.54</td><td>19.66</td>
  </tr>
  <tr>
    <td>Clin Psych</td><td>87.65</td><td>67.73</td><td>77.29</td><td>59.76</td><td>65.74</td><td>72.11</td><td>59.36</td><td>72.11</td><td>46.22</td><td>43.03</td><td>19.92</td><td>54.98</td><td>45.82</td><td>67.33</td><td>64.54</td><td>31.47</td><td>31.87</td><td>23.11</td>
  </tr>
  <tr>
    <td>Occup Therap</td><td>76.85</td><td>58.89</td><td>60.93</td><td>53.52</td><td>65.56</td><td>62.41</td><td>51.85</td><td>61.85</td><td>39.44</td><td>36.48</td><td>20.56</td><td>50.37</td><td>40.56</td><td>56.11</td><td>54.63</td><td>30.37</td><td>26.48</td><td>19.81</td>
  </tr>
  <tr>
    <td>Rad Tech</td><td>76.97</td><td>54.32</td><td>58.73</td><td>43.57</td><td>59.12</td><td>64.68</td><td>54.14</td><td>57.30</td><td>30.33</td><td>33.40</td><td>17.66</td><td>38.96</td><td>34.36</td><td>51.06</td><td>48.37</td><td>25.33</td><td>20.92</td><td>15.55</td>
  </tr>
  <tr>
    <td>Legal Psych</td><td>82.53</td><td>61.59</td><td>61.59</td><td>51.61</td><td>58.24</td><td>61.59</td><td>51.61</td><td>58.24</td><td>37.06</td><td>32.49</td><td>14.18</td><td>42.94</td><td>34.81</td><td>52.94</td><td>51.18</td><td>17.65</td><td>13.65</td><td>8.41</td>
  </tr>
  <tr>
    <td>PH Spec</td><td>82.35</td><td>70.59</td><td>82.82</td><td>64.27</td><td>52.30</td><td>58.82</td><td>61.71</td><td>52.06</td><td>42.19</td><td>41.18</td><td>14.24</td><td>52.94</td><td>41.18</td><td>41.88</td><td>17.65</td><td>13.65</td><td>8.41</td><td>5.29</td>
  </tr>
  <tr>
    <td>Med Lab Sci</td><td>85.17</td><td>67.71</td><td>68.86</td><td>55.68</td><td>63.26</td><td>76.67</td><td>58.81</td><td>66.72</td><td>40.53</td><td>43.00</td><td>18.12</td><td>54.04</td><td>44.81</td><td>60.30</td><td>58.32</td><td>30.21</td><td>26.19</td><td>22.57</td>
  </tr>
  <tr>
    <td>TCM Prac</td><td>50.08</td><td>36.24</td><td>63.26</td><td>39.57</td><td>54.33</td><td>44.81</td><td>34.27</td><td>55.85</td><td>32.13</td><td>21.91</td><td>13.84</td><td>43.16</td><td>24.71</td><td>49.59</td><td>44.98</td><td>16.31</td><td>17.30</td><td>8.40</td>
  </tr>
  <tr>
    <td>Micro Avg.</td><td>75.35</td><td>55.86</td><td>63.24</td><td>50.79</td><td>60.84</td><td>64.46</td><td>51.41</td><td>62.29</td><td>38.79</td><td>37.45</td><td>17.81</td><td>47.20</td><td>37.44</td><td>54.50</td><td>52.08</td><td>26.08</td><td>24.51</td><td>18.25</td>
  </tr>
  <tr>
    <td>Marco Avg.</td><td>75.25</td><td>55.96</td><td>63.17</td><td>50.77</td><td>61.22</td><td>63.02</td><td>51.23</td><td>61.90</td><td>39.71</td><td>38.06</td><td>18.86</td><td>47.17</td><td>37.09</td><td>53.94</td><td>51.02</td><td>27.00</td><td>24.26</td><td>18.74</td>
  </tr>
</table>

* A random guess baseline gets 24.96% micro average accuracy

of the tested models are specifically fine-tuned for following instructions.

5 Analysis

In general, all the tested models show unsatisfying performance. The average accuracy of 12 out of the 17 tested models is under 60% on EMPEC where random guessing can reach about 25%. The results suggest the difficulty of our benchmark and the healthcare knowledge gap in existing LLMs.

5.1 Results of general LLMs

From the accuracy of each profession and the average overall profession, GPT-4 shows an evident lead among the tested models, with only a second Qwen1.5-70B-Chat in Traditional Chinese Medicine Practitioner. Another proprietary model, GPT-3.5 shows a nearly 20% performance drop from GPT 4. Open-source models are far behind GPT-4, the accuracy of the best-performing models, Qwen1.5-70B-Chat and Llama3-70B-Instruct, is around 63% — 12% less than GPT-4. However, gained from training on a large proportion of Chinese data, Yi-34B and Qwen1.5-70B beat GPT 3.5. Observing the results of Yi, Qwen1.5, and Llama-3, we see that increasing the model size can benefit models’ performance as expected. Among models of around 7B size, InternLM2, Qwen1.5-Chat, and Llama3-8B-Instruct achieve over 50% accuracy while Mistral and Yi are slightly lower than 40%.

5.2 Results of medical domain LLMs

HuatuoGPT2, MMedLM2, and MedGPT, despite being based on Chinese-focused LLMs and specifically trained on medical data, show poor performance in EMPEC. The results of MedGPT are slightly higher than random guess while HuatuoGPT2-13B even underperforms the random baseline. MMedLM2 performs best among medical LLMs, achieves 45% accuracy with 7B parameters, but still lags behind Yi-6B and Qwen1.5-7B. We noticed that part of the reason for poor performance is the loss of instruction-following ability. For instance, HuatuoGPT2, despite being able to generate plausible response, sometimes deviates from the questions and often fails to conclude an answer. HuatuoGPT2 underperforms Baichuan-2-13B-Chat and Yi-34B-Chat which shares the same base model but fine-tuning on general instruction data. The findings are different from the results reported in Wang et al. (2023b) where HuatuoGPT2 outperforms Baichuan2 by a large margin in Chinese medical questions. In addition, InternLM2 leads MMedLM2 by more than 2% accuracy while fine-tuned MMedLM2 outperforms InternLM2 on MedQA (Jin et al., 2021). Only MedicalGPT outperforms Ziya-Llama by around 2%. In conclusion, we do not observe evident enhancement on EMPEC brought by fine-tuning or pre-training in the three works, contrasting the findings on existing benchmarks like CMB (Wang et al., 2023b). Therefore, we argue that, compared to the existing Chinese healthcare benchmark, our new EMPEC provides a more robust platform for the eval-
uation of domain-adaptation LLMs. This finding further suggests that the current improvement in domain adaptation could result in an over-fit of the tested distribution, which might not hold once the distribution switches. Moreover, our fine-tuned Qwen1.5-7B model achieves 61% accuracy, nearly 11% better than the Chat counterpart and close to Qwen1.5-70B-Chat, suggesting that the training data of EMPEC can be an effective supplement of current medical domain adaptation endeavours.

5.3 Results by profession
We further examined the performance of the models by profession. While GPT-4 prevails on 19 of the 20 professions, Qwen1.5-70B-Chat achieves the highest accuracy in Traditional Chinese Medicine Practitioner. Then we rank the accuracy of each profession within every model and then aggregate rankings across models. The best three performed professions are Clinical Psychologist, Physician, and Nurse while the models mostly struggle to answer questions for Dental technician, Speech Therapist, Optometrist, Dentist, and TCM Practitioner, which seem connect more to specific medical subject. We assume this difference is rooted in the frequency of knowledge in the training data of these models and the rareness of knowledge of these bad-performing professions as Kandpal et al. (2023) has suggested. Take GPT-4 as an example, it achieves nearly 90% accuracy for physicians of which related-documents are rich on the Internet and books, while its performance on rarely touched professions like Dental technician and Traditional Chinese Medicine Practitioner is lower than 60%. Interestingly, all tested medical domain LLMs except Qwen1.5-7B-SFT do not show a better performance compared to their general counterparts even on physicians which is the focus of their finetuning data. However they generally unsatisfying performance in each profession indicating the importance of data diversity in building an LLM for healthcare. By breaking the overall results into professions, EMPEC is able to precisely detect LLM’s knowledge gap in the healthcare domain.

5.4 Results on the unseen questions
During the collection of EMPEC, we found that some of the past questions are freely available on the Internet, the possibility of the data having been seen by LLMs cannot be ruled out. Thus, we explicitly composed a dataset containing 3497 questions issued in the exams in 2024, which should not be seen by the models of which cut-offs are before 2024. The statistics of this data set are given in Table 6. We tested GPT-4-turbo(cutoff on Dec. 2023), GPT-3.5-turbo(cutoff on Sept. 2021), Yi (cutoff on Dec. 2023), Baichuan-2 (released on Sept. 2023), and Mistral (released at Dec. 2023) on the questions released in 2024. It is important to note that this split might have overlapped with the test set of EMPEC and the included professions are limited, as the exams for the rest professions have not been held yet. The results are in Table 4. In general, we observed that the results of questions 2024 are very close to the results obtained from the test set across assessed models. In other words, the performance rankings are held as in the test set. Moreover, the professions where models perform well, such as Physician and Nurse, perform poorly, such as Dentist and Traditional Chinese Medicine Practitioner also keep in the latest questions set. The findings strongly suggest the robustness of the EMPEC test set, as the performance of the models is consistent with that of the unseen data. In addition, it also indicates that the tested models do not gain their performance by reciting seen examples.

![Bar chart comparing micro average accuracy of models on traditional Chinese and simplified Chinese. C stands for Chat version and I stands for Instruction version](page_1012_1342_388_246.png)

Figure 3: Micro average accuracy of models on traditional Chinese and simplified Chinese. C stands for Chat version and I stands for Instruction version

5.5 Results by language
As many of the tested models are trained primarily in English, we tested them to examine the impact brought by transferring to English. Figure 4 shows the performance of models in the English-translated test set, more results are shown in Table 5, we observed that GPT-4, Llama 3, Yi, MMedLM 2, and Mistral demonstrated improved performance following translation. However, InternLM 2 and Qwen 1.5-Chat exhibited a decline. It
Table 4: Performance of models on questions from 2024. Colored numbers are the results of questions from exams in 2024 while the black ones are the results of the test set. Red indicates higher while blue suggests degrade.

<table>
  <tr>
    <th rowspan="2">Profession</th>
    <th colspan="5">Accuracy</th>
  </tr>
  <tr>
    <th>GPT 4</th>
    <th>GPT 3.5</th>
    <th>Yi-Chat-34B</th>
    <th>Baichuan2-Chat-13B</th>
    <th>Mistral-Instruct-7B-v0.2</th>
  </tr>
  <tr>
    <td>Phys Therap</td>
    <td>73.05 (70.25)</td>
    <td>50.78 (51.43)</td>
    <td>54.34 (59.33)</td>
    <td>49.00 (46.89)</td>
    <td>36.97 (36.47)</td>
  </tr>
  <tr>
    <td>Rad Tech</td>
    <td>72.58 (76.97)</td>
    <td>51.52 (54.32)</td>
    <td>58.68 (54.70)</td>
    <td>44.63 (38.96)</td>
    <td>34.41 (34.36)</td>
  </tr>
  <tr>
    <td>Physician</td>
    <td>85.74 (89.90)</td>
    <td>63.39 (67.42)</td>
    <td>72.18 (73.87)</td>
    <td>48.74 (54.70)</td>
    <td>44.77 (45.12)</td>
  </tr>
  <tr>
    <td>TCM Prac</td>
    <td>57.05 (50.08)</td>
    <td>36.58 (36.24)</td>
    <td>50.65 (55.85)</td>
    <td>46.97 (43.16)</td>
    <td>24.03 (24.71)</td>
  </tr>
  <tr>
    <td>Dentist</td>
    <td>67.22 (69.27)</td>
    <td>47.29 (49.83)</td>
    <td>50.82 (56.60)</td>
    <td>39.76 (42.71)</td>
    <td>38.82 (35.24)</td>
  </tr>
  <tr>
    <td>Pharmacist</td>
    <td>76.24 (75.86)</td>
    <td>58.01 (58.15)</td>
    <td>63.54 (64.59)</td>
    <td>47.51 (45.07)</td>
    <td>38.40 (39.44)</td>
  </tr>
  <tr>
    <td>Med Lab Sci.</td>
    <td>85.97 (85.17)</td>
    <td>63.92 (67.71)</td>
    <td>67.93 (66.72)</td>
    <td>47.88 (54.04)</td>
    <td>43.21 (44.81)</td>
  </tr>
  <tr>
    <td>Dietitian</td>
    <td>75.93 (74.76)</td>
    <td>55.51 (53.40)</td>
    <td>60.29 (63.43)</td>
    <td>45.59 (50.49)</td>
    <td>40.44 (37.54)</td>
  </tr>
  <tr>
    <td>Nurse</td>
    <td>87.76 (82.75)</td>
    <td>71.31 (63.53)</td>
    <td>78.48 (73.53)</td>
    <td>56.54 (55.10)</td>
    <td>43.46 (40.20)</td>
  </tr>
  <tr>
    <td>Micro Avg.</td>
    <td>75.12 (74.54)</td>
    <td>54.48 (55.47)</td>
    <td>61.11 (62.82)</td>
    <td>47.07 (47.70)</td>
    <td>37.95 (37.44)</td>
  </tr>
</table>

![Bar chart showing micro average accuracy of models on traditional Chinese and English. C stands for Chat version and I stands for Instruction version](page_184_668_495_312.png)

Figure 4: Micro average accuracy of models on traditional Chinese and English. C stands for Chat version and I stands for Instruction version

is evident that models primarily trained in English or continual-pretrained in English like MMedLM2 benefit from this transition. However, it is crucial to note that, in English, accuracy trends by profession remain consistent. Models continue to struggle with specialities such as Dietitian, Audiologist, Optometrist, and Veterinarian, further underscoring the unique challenges posed by the EMPEC dataset. Furthermore, the slightly lower performance of BioMistral compared to Mistral reinforces our assertion that current medical LLMs do not consistently outperform their general-purpose counterparts in comprehensive healthcare knowledge.

The difference between traditional Chinese and simplified Chinese can affect the readers’ analytic skills (Liu and wen Hsiao, 2012). We further investigated the performance of a subset of tested models on the test set in traditional Chinese and simplified Chinese. The results are shown in Figure 3. From traditional to simplified, contrary effects are observed among the models with GPT-4, Yi-Chat, and Qwen1.5-Chat suffering slight performance drops while HuatuoGPT2, MMedLM2, Llama3-Instruct, Qwen1.5-SFT enjoying slight improvement. However, the performance across these models between the two kinds of Chinese does not vary greatly though the traditional and simplified characters do not share the same tokens. We assume that the character difference in traditional and simplified Chinese does not affect the model’s performance as the co-occurrence relations of these characters are similar in the two kinds of Chinese.

6 Conclusion and Discussions

We introduced EMPEC, the most comprehensive healthcare knowledge benchmark to date, encompassing 157,803 questions across 124 subjects and 20 healthcare occupations. EMPEC goes beyond the physician-centric focus of previous medical benchmarks to provide a holistic assessment of the knowledge of LLM in a wide spectrum of health disciplines. Our extensive experiments, which tested both proprietary and open-source models, revealed several important findings. Although the best general LLMs performed reasonably well in common professions such as Physicians and Nurses, they struggled with rarer specialities like Dental Technicians, Optometrists, and TCM Practitioners. This highlights the need for more work to improve the knowledge of LLMs in healthcare beyond the expertise of physicians that has been the primary focus so far. Interestingly, we found that existing medical domain-specific LLMs did not perform better on EMPEC compared to their general counterparts not specialised in healthcare. Furthermore, our experiments indicate that the models’ performance on the EMPEC test set can predict their effectiveness in addressing unseen healthcare-related queries. We also found that the transition from traditional to simplified Chinese characters
<table>
  <tr>
    <th></th>
    <th>InterLM2-7B-Chat</th>
    <th>BioMistral-7B</th>
    <th>MMeLM2-7B-7B</th>
    <th>Yi-Chat-6B</th>
    <th>Qwen1.5-7B-Chat</th>
    <th>Mistral-7B-Instruct v0.2</th>
    <th>GPT4</th>
    <th>Llama3-8B-Instruct</th>
    <th>GPT3.5</th>
  </tr>
  <tr>
    <td>Nurse</td>
    <td>57.45</td>
    <td>56.86</td>
    <td>63.33</td>
    <td>60.20</td>
    <td>44.71</td>
    <td>62.35</td>
    <td>82.45</td>
    <td>81.37</td>
    <td>77.76</td>
  </tr>
  <tr>
    <td>Dentist</td>
    <td>40.28</td>
    <td>45.49</td>
    <td>45.49</td>
    <td>42.38</td>
    <td>38.54</td>
    <td>46.40</td>
    <td>74.13</td>
    <td>53.65</td>
    <td>53.30</td>
  </tr>
  <tr>
    <td>Midwife</td>
    <td>51.05</td>
    <td>54.55</td>
    <td>58.39</td>
    <td>56.29</td>
    <td>45.45</td>
    <td>56.64</td>
    <td>84.27</td>
    <td>61.54</td>
    <td>66.43</td>
  </tr>
  <tr>
    <td>Physician</td>
    <td>49.48</td>
    <td>55.05</td>
    <td>65.16</td>
    <td>60.80</td>
    <td>49.83</td>
    <td>55.40</td>
    <td>90.77</td>
    <td>75.09</td>
    <td>69.86</td>
  </tr>
  <tr>
    <td>Dietitian</td>
    <td>48.87</td>
    <td>50.16</td>
    <td>55.66</td>
    <td>51.13</td>
    <td>42.39</td>
    <td>51.46</td>
    <td>77.35</td>
    <td>58.90</td>
    <td>64.08</td>
  </tr>
  <tr>
    <td>Pharmacist</td>
    <td>46.98</td>
    <td>48.79</td>
    <td>54.44</td>
    <td>50.00</td>
    <td>41.73</td>
    <td>53.63</td>
    <td>76.61</td>
    <td>57.46</td>
    <td>60.89</td>
  </tr>
  <tr>
    <td>Audiologist</td>
    <td>42.95</td>
    <td>44.17</td>
    <td>50.48</td>
    <td>42.48</td>
    <td>34.28</td>
    <td>42.05</td>
    <td>72.04</td>
    <td>49.47</td>
    <td>56.89</td>
  </tr>
  <tr>
    <td>Podiatrist</td>
    <td>34.62</td>
    <td>38.46</td>
    <td>43.59</td>
    <td>39.00</td>
    <td>53.85</td>
    <td>52.56</td>
    <td>65.38</td>
    <td>51.28</td>
    <td>60.26</td>
  </tr>
  <tr>
    <td>Veterinarian</td>
    <td>40.23</td>
    <td>47.89</td>
    <td>52.11</td>
    <td>50.00</td>
    <td>43.68</td>
    <td>50.38</td>
    <td>81.23</td>
    <td>60.34</td>
    <td>65.90</td>
  </tr>
  <tr>
    <td>Speech Therap</td>
    <td>40.87</td>
    <td>38.10</td>
    <td>45.24</td>
    <td>36.51</td>
    <td>30.16</td>
    <td>40.08</td>
    <td>75.40</td>
    <td>50.79</td>
    <td>54.37</td>
  </tr>
  <tr>
    <td>Dental Tech</td>
    <td>34.52</td>
    <td>42.64</td>
    <td>47.21</td>
    <td>46.19</td>
    <td>40.10</td>
    <td>41.62</td>
    <td>68.02</td>
    <td>46.19</td>
    <td>47.72</td>
  </tr>
  <tr>
    <td>Phys Therap</td>
    <td>41.32</td>
    <td>50.79</td>
    <td>54.97</td>
    <td>46.37</td>
    <td>39.63</td>
    <td>45.70</td>
    <td>75.01</td>
    <td>57.17</td>
    <td>57.70</td>
  </tr>
  <tr>
    <td>Occup Therap</td>
    <td>42.12</td>
    <td>44.81</td>
    <td>47.88</td>
    <td>45.96</td>
    <td>39.04</td>
    <td>45.77</td>
    <td>74.51</td>
    <td>51.54</td>
    <td>56.92</td>
  </tr>
  <tr>
    <td>Clin Psych</td>
    <td>54.40</td>
    <td>63.60</td>
    <td>68.80</td>
    <td>58.80</td>
    <td>54.00</td>
    <td>64.00</td>
    <td>85.20</td>
    <td>72.00</td>
    <td>76.00</td>
  </tr>
  <tr>
    <td>Occup Therap</td>
    <td>48.15</td>
    <td>50.19</td>
    <td>55.56</td>
    <td>52.78</td>
    <td>40.19</td>
    <td>51.48</td>
    <td>78.70</td>
    <td>58.89</td>
    <td>65.93</td>
  </tr>
  <tr>
    <td>Rad Tech</td>
    <td>44.51</td>
    <td>51.25</td>
    <td>54.72</td>
    <td>47.98</td>
    <td>39.50</td>
    <td>47.21</td>
    <td>75.72</td>
    <td>55.49</td>
    <td>57.23</td>
  </tr>
  <tr>
    <td>Consel Psych</td>
    <td>54.69</td>
    <td>57.81</td>
    <td>62.11</td>
    <td>59.38</td>
    <td>43.75</td>
    <td>56.25</td>
    <td>85.55</td>
    <td>67.19</td>
    <td>71.09</td>
  </tr>
  <tr>
    <td>PH Spec</td>
    <td>70.39</td>
    <td>52.94</td>
    <td>70.94</td>
    <td>52.94</td>
    <td>62.94</td>
    <td>64.71</td>
    <td>82.35</td>
    <td>82.35</td>
    <td>82.35</td>
  </tr>
  <tr>
    <td>Med Lab Sci</td>
    <td>50.74</td>
    <td>57.33</td>
    <td>63.10</td>
    <td>58.32</td>
    <td>45.80</td>
    <td>56.84</td>
    <td>86.00</td>
    <td>69.03</td>
    <td>71.00</td>
  </tr>
  <tr>
    <td>TCM Prac</td>
    <td>30.31</td>
    <td>28.34</td>
    <td>36.24</td>
    <td>32.78</td>
    <td>26.85</td>
    <td>29.98</td>
    <td>51.57</td>
    <td>32.45</td>
    <td>34.76</td>
  </tr>
  <tr>
    <td>Marco Avg.</td>
    <td>46.16</td>
    <td>48.96</td>
    <td>54.74</td>
    <td>50.06</td>
    <td>42.32</td>
    <td>50.74</td>
    <td>76.25</td>
    <td>59.11</td>
    <td>62.21</td>
  </tr>
  <tr>
    <td>Micro Avg.</td>
    <td>45.06</td>
    <td>48.96</td>
    <td>54.15</td>
    <td>49.79</td>
    <td>41.08</td>
    <td>49.70</td>
    <td>77.10</td>
    <td>58.26</td>
    <td>60.87</td>
  </tr>
</table>

Table 5: Performance of each model on the English translation of the test set of EMPEC, split by profession.

had negligible impact on model performance. EMPEC lays the foundation for the development of advanced LLMs for the healthcare domain, providing a more comprehensive and nuanced evaluation of their capabilities in a wide range of health professions.

Limitations EMPEC has several limitations: Some professions included in EMPEC have a relatively smaller representation compared to others, which may restrict EMPEC’s effectiveness in evaluating knowledge in those specific professions. The English translation is generated by LLMs without further examination of experts.

Societal Impacts Although EMPEC is designed to improve the evaluation of LLM in the medical field, it should not be used to assess individual medical competence or to diagnose any patient. Any conclusions drawn from models trained on this dataset should take into account its limitations. The dataset’s use should be confined to research purposes to prevent potential misuse. In addition, as the simplified Chinese version and English version are obtained including conversion and translation, we cannot guarantee the accuracy are fully aligned with the original questions.

The datasets are built using publicly available open-source data from Taiwan’s Ministry of Examination. Their use is encouraged, provided the source is clearly cited. Furthermore, the questions in the datasets do not contain any private information.

References

Asma Ben Abacha, Yassine Mrabet, Mark Sharp, Travis R Goodwin, Sonya E Shooshan, and Dina Demner-Fushman. 2019. Bridging the gap between consumers’ medication questions and trusted answers. In MedInfo, pages 25–29.

AI@Meta. 2024. Llama 3 model card.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xinxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Owen technical report. arXiv preprint arXiv:2309.16609.

Yan Cai, Linlin Wang, Ye Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, and Liang He. 2023. Medbench: A large-scale chinese benchmark for evaluating medical large language models. ArXiv, abs/2312.12806.

Zheng Cai, Maosong Cao, Haojing Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Interlm2 technical report. arXiv preprint arXiv:2403.17297.

Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, Jinyang Gao, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024. Data-juicer: A one-stop data processing system for large language models. In International Conference on Management of Data.

Junying Chen, Xidong Wang, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, Dingjie Song, Wenya Xie, Chuyi Kong, Jianquan Li, Xiang Wan,
Haizhou Li, and Benyou Wang. 2023a. Huatuogpt-ii, one-stage training for medical adaption of llms. Preprint, arXiv:2311.09774.

Zeming Chen, Alejandro Hern’andez Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Kopf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. 2023b. Meditron-70b: Scaling medical pretraining for large language models. ArXiv, abs/2311.16079.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. ArXiv, abs/2110.14168.

Zhiwei Fei, Xiaoyu Shen, D. Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. Lawbench: Benchmarking legal knowledge of large language models. ArXiv, abs/2309.16289.

Gemini Team Google. 2023. Gemini: A family of highly capable multimodal models. ArXiv, abs/2312.11805.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjin Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Fanchao Qi, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. ArXiv, abs/2305.08322.

IDEA-CCNL. 2021. Fengshenbang-lm. https://github.com/IDEA-CCNL/Fengshenbang-LM.

Albert Qiaocheu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. ArXiv, abs/2310.06825.

Di Jin, Eileen Pan, Nassim Ouafatole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696–15707. PMLR.

Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir Radev. 2023. Evaluating gpt-4 and chatgpt on japanese medical licensing examinations. arXiv preprint arXiv:2303.18027.

Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. Bioasqqa: A manually curated corpus for biomedical question answering. Scientific Data, 10(1):170.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.

Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. Biomistral: A collection of open-source pretrained large language models for medical domains. arXiv preprint arXiv:2402.10373.

Stephanie C. Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. In Annual Meeting of the Association for Computational Linguistics.

Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, et al. 2024. Benchmarking large language models on cmexam-a comprehensive chinese medical exam dataset. Advances in Neural Information Processing Systems, 36.

Tianyin Liu and Janet Hui wen Hsiao. 2012. The perception of simplified and traditional chinese characters in the eye of simplified and traditional chinese readers. Journal of Vision, 12:533–533.

Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems. ArXiv, abs/2303.13375.

OpenAI. 2023. Gpt-4 technical report.

Ankit Pal, Logesh Kumar Umapathi, and Malaikanan Sankarasubbu. 2022. Medmcqa : A large-scale multi-subject multi-choice dataset for medical domain question answering. In ACM Conference on Health, Inference, and Learning.

Pengcheng Qiu, Chaoyu Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024. Towards building multilingual language model for medicine. arXiv preprint arXiv:2402.13963.
Maciej Rosol, Jakub S Gasior, Jonasz Laba, Kacper Korzeniewski, and Marcel Młyńczak. 2023. Evaluation of the performance of gpt-3.5 and gpt-4 on the polish medical final examination. Scientific Reports, 13(1):20512.

Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172–180.

Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esioibu, Jude Fernandes, Jeremy Fu, Wenying Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaev, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binhh Tlang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.09288.

Hao Wang, Chi-Liang Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. 2023a. Huatuo: Tuning llama model with chinese medical knowledge. ArXiv, abs/2304.06975.

Xidong Wang, Guimin Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, and Haizhou Li. 2023b. Cmb: A comprehensive medical benchmark in chinese. ArXiv, abs/2308.08833.

Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, et al. 2024a. Me llama: Foundation large language models for medical applications. arXiv preprint arXiv:2402.12749.

Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yuerv He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Zi-Zhou Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhiwei Liu, Guojun Xiong, Zhiyang Deng, Yuechen Jiang, Zhiyuan Yao, Haohang Li, Yangyang Yu, Gang Hu, Jiajia Huang, Xiao-Yang Liu, Alejandro Lopez-Lira, Benyou Wang, Yanzhao Lai, Hao Wang, Min Peng, Sophia Ananiadou, and Jimin Huang. 2024b. The finben: An holistic financial benchmark for large language models. ArXiv, abs/2402.12659.

Ming Xu. 2023. Medicalgpt: Training medical gpt model. https://github.com/shibing624/MedicalGPT.

Ai Ming Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Hai Zhao, Hang Xu, Hao-Lun Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, Juntao Dai, Kuncheng Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Pei Guo, Ruiyang Sun, Zhang Tao, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yan-Bin Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. 2023. Baichuan 2: Open large-scale language models. ArXiv, abs/2309.10305.

01.AI Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024. Yi: Open foundation models by 01.ai.

Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. 2024. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332.
A  Prompt

The prompt used in the zero-shot evaluation and supervised fine-tuning is shown in Fig. 5.

[Question]
答案: (Answer:)

Figure 5: The prompt used in the zero-shot evaluation and supervised fine-tuning. The texts in blue are the English translations of the Chinese content.

B  Statistics of EMPEC Questions Issued in 2024

The detailed statistics of EMPEC

<table>
  <tr>
    <th>Professors</th>
    <th>Number of questions</th>
  </tr>
  <tr>
    <td>Physical Therapist</td>
    <td>449</td>
  </tr>
  <tr>
    <td>Radiologic Technologist</td>
    <td>363</td>
  </tr>
  <tr>
    <td>Physician</td>
    <td>478</td>
  </tr>
  <tr>
    <td>Traditional Chinese Medicine Practitioner</td>
    <td>462</td>
  </tr>
  <tr>
    <td>Dentist</td>
    <td>425</td>
  </tr>
  <tr>
    <td>Pharmacist</td>
    <td>362</td>
  </tr>
  <tr>
    <td>Medical Laboratory Scientist</td>
    <td>449</td>
  </tr>
  <tr>
    <td>Dietitian</td>
    <td>272</td>
  </tr>
  <tr>
    <td>Nurse</td>
    <td>237</td>
  </tr>
</table>

Table 6: Number of questions for each profession in questions issued in 2024

C  Human assessment of English translation

We conducted a manual evaluation of the English translations generated by GPT-4 using 10%(800) of the EMPEC test set. The evaluation was performed by a PhD-level medical student who is a native Chinese speaker and fluent in professional biomedical English. No errors were identified in the translations reviewed.

D  Subject distributions in EMPEC

We show the distribution of subjects in EMPEC in Table 7.
Table 7: Subjects examined for each profession

<table>
  <tr>
    <th>Profession</th>
    <th>Subjects</th>
    <th>#Subjects</th>
    <th>#Questions</th>
  </tr>
  <tr>
    <td>Traditional Chinese Medicine Practitioner</td>
    <td>Basic Chinese Medicine (1-2); Clinical Chinese Medicine (1-4); Pharmacy and Biopharmaceutics</td>
    <td>7</td>
    <td>11954</td>
  </tr>
  <tr>
    <td>Medical Laboratory Scientist</td>
    <td>Clinical Hematology and Blood Bank; Biochemistry and Clinical Biochemistry; Microbiology and Clinical Microbiology; Clinical Serum Immunology and Clinical; Medical Molecular Testing and Clinical; Clinical Physiology and Pathology; Clinical Mirror Examination</td>
    <td>7</td>
    <td>11938</td>
  </tr>
  <tr>
    <td>Physical Therapist</td>
    <td>Basic Physical Therapy; Cardiopulmonary and Pediatric Disease Therapy; Orthopedic Disease Physical Therapy; Neurological Disease Physical Therapy; Introduction to Physical Therapy; Physical Therapy Techniques</td>
    <td>6</td>
    <td>11698</td>
  </tr>
  <tr>
    <td>Dentist</td>
    <td>Dentistry (1-6)</td>
    <td>6</td>
    <td>11468</td>
  </tr>
  <tr>
    <td>Physician</td>
    <td>Medicine(1-6); Clinical Psychology Special Topics</td>
    <td>7</td>
    <td>11366</td>
  </tr>
  <tr>
    <td>Occupational Therapist</td>
    <td>Anatomy and Physiology; Psychological Disability Occupational Therapy; Occupational Therapy Techniques; Introduction to Occupational Therapy; Pediatric Occupational Therapy; Physiological Disability Occupational Therapy</td>
    <td>6</td>
    <td>10646</td>
  </tr>
  <tr>
    <td>Radiologic Technologist</td>
    <td>Basic Medical Science; Radiation Therapy Principles and Techniques; Nuclear Medicine Diagnosis Principles and Techniques; Medical Physics and Radiation Safety; Radiological Equipment; Radiological Diagnosis Principles and Techniques</td>
    <td>6</td>
    <td>10307</td>
  </tr>
  <tr>
    <td>Respiratory Therapist</td>
    <td>Cardiopulmonary Basic Medical Science; Respiratory Principles and Applications; Intensive Respiratory Therapy; Respiratory Therapy Equipment; Respiratory Diseases; Basic Respiratory Therapy</td>
    <td>6</td>
    <td>10301</td>
  </tr>
  <tr>
    <td>Veterinarian</td>
    <td>Veterinary Laboratory Diagnosis; Veterinary Pharmacology; Veterinary General Diseases; Veterinary Infectious Diseases; Veterinary Pathology; Veterinary Public Health</td>
    <td>6</td>
    <td>10292</td>
  </tr>
  <tr>
    <td>Nurse</td>
    <td>Internal and Surgical Nursing; Basic Nursing; Basic Medical Science; Mental Health and Community Health Nursing; Obstetric and Pediatric Nursing; Overview of Basic Medical Science; Obstetrics, Psychiatry and Community; Overview of Basic Nursing; Overview of Internal and Surgical Nursing</td>
    <td>9</td>
    <td>10066</td>
  </tr>
  <tr>
    <td>Pharmacist</td>
    <td>Pharmacochemistry; Pharmacy; Dispensing and Clinical Pharmacy; Pharmacology and Pharmaceutical Chemistry; Pharmacy and Biopharmaceutics</td>
    <td>5</td>
    <td>9767</td>
  </tr>
  <tr>
    <td>Dietitian</td>
    <td>Group Meal Design and Management; Diet Therapy; Nutrition; Food Hygiene and Safety; Physiology and Biochemistry; Public Health Nutrition</td>
    <td>6</td>
    <td>6088</td>
  </tr>
  <tr>
    <td>Midwife</td>
    <td>Midwifery (1-2); Nursing for All Specialties; Basic Medical Science; Basic Nursing</td>
    <td>5</td>
    <td>5642</td>
  </tr>
  <tr>
    <td>Audiologist</td>
    <td>Basic Audiology; Hearing and Language Communication Disorders; Health of Auditory and Balance Systems; Electrophysiological Audiology; Behavioral Audiology; Principles and Practice of Hearing Aids</td>
    <td>6</td>
    <td>5600</td>
  </tr>
  <tr>
    <td>Counseling Psychologist</td>
    <td>Counseling and Psychotherapy Theories; Counseling and Psychotherapy Practice and Human Behavior and Development; Group Counseling and Psychotherapy; Case Assessment and Psychological Evaluation; Psychological Foundations of Counseling; Mental Health and Abnormal Psychology; Mental Health; Psychological Testing and Assessment; Counseling and Psychotherapy Practice</td>
    <td>10</td>
    <td>5014</td>
  </tr>
  <tr>
    <td>Speech Therapist</td>
    <td>Articulation and Fluency Disorders; Basic Linguistics; Communication Disorders Overview; Neurological Communication Disorders; Child Language Disorders; Voice and Swallowing Disorders</td>
    <td>6</td>
    <td>4973</td>
  </tr>
  <tr>
    <td>Clinical Psychologist</td>
    <td>Special Topics in Clinical Psychology (1-2); Clinical Psychology Special Topics (1-2); Basic Clinical Psychology</td>
    <td>5</td>
    <td>4923</td>
  </tr>
  <tr>
    <td>Dental Technician</td>
    <td>Dental Technology (1-4)</td>
    <td>4</td>
    <td>3885</td>
  </tr>
  <tr>
    <td>Optometrist</td>
    <td>Optometry; Low Vision; Eye Anatomy, Physiology and Ethics; Contact Lens and Dispensing; Visual Optics</td>
    <td>5</td>
    <td>1538</td>
  </tr>
  <tr>
    <td>Public Health Specialist</td>
    <td>Health Administration and Management; Epidemiology; Environmental and Occupational Health; Biostatistics; Health Social Behavior</td>
    <td>5</td>
    <td>337</td>
  </tr>
  <tr>
    <th>Total</th>
    <th></th>
    <th>124</th>
    <th>157803</th>
  </tr>
</table>
E Example of Deduplicate Configuration

# global parameters
project_name: 'all' # project name for distinguish your configs
dataset_path: '/path/to/your/dataset' # path to your dataset directory or file
export_path: '/path/to/result/dataset.jsonl' # path to processed result dataset.
np: 4 # number of subprocess to process your dataset
open_tracer: true # whether to open the tracer to trace the changes
# It might take more time when opening tracer

# process schedule: a list of several process operators with their arguments
process:
  # Mapper ops. Most of these ops need no arguments.
  - clean_email_mapper: # remove emails from text.
  - clean_links_mapper: # remove web links from text.
  - clean_copyright_mapper: # remove copyright comments.
  - expand_macro_mapper: # expand macro definitions in Latex text.
  - fix_unicode_mapper: # fix unicode errors in text.

  # Filter ops
  - average_line_length_filter: # filter text with the average length of lines
    max_len: 1500
  - language_id_score_filter: # filter text in specific language with language scores
    lang: zh # keep text in what language
    min_score: 0.95 # the min language scores to filter text
  - maximum_line_length_filter: # filter text with the maximum length of lines
    #min_len: 10 # the min length of filter range
    max_len: 7328 # the max length of filter range
  - special_characters_filter: # filter text with special-char ratio out of specific range
    #min_ratio: 0.0 # the min ratio of filter range
    max_ratio: 0.3 # the max ratio of filter range
  - text_length_filter: # filter text with length out of specific range
    min_len: 10 # the min length of filter range
    max_len: 10000 # the max length of filter range
  - document_simhash_deduplicator: # deduplicate text samples using SimHash-LSH method
    tokenization: space # tokenization method for text.
    window_size: 6 # window size of shingling
    num_blocks: 6 # number of blocks in SimHash computing
    hamming_distance: 4 # the max hamming distance to regard 2 samples
    ignore_pattern: null # whether to ignore sub-strings with specific pattern

Reviewer paper summaries: This paper introduces EMPEC (Examinations for Medical PErsonnel in Chinese), a comprehensive healthcare knowledge benchmark that addresses a need in current medical LLM evaluation. The benchmark contains 157,803 exam questions across 124 subjects and 20 healthcare professions, including underrepresented roles like Optometrists and Audiologists. Each question is tagged with release time and source authenticity. The authors evaluated 17 LLMs, including proprietary and open-source models, finding that while models like GPT-4 achieved over 75% accuracy, they struggled with specialized fields and alternative medicine. The paper also investigates the impact of language variations (traditional vs simplified Chinese and English translations) on model performance. The dataset and evaluation toolkit will be made available for future research.

This paper introduces a benchmark of 157k medical exam questions in Traditional Chinese from Professional and Technical Examinations of Taiwan. This benchmark encompasses 124 medical subjects and 20 healthcare professions. The authors evaluated 17 LLMs on this benchmark and analyzed their performance according to healthcare profession in questions and LLMs domains specialty (general/medical). In all evaluated models, GPT-4-turbo yielded the best performance, no matter the healthcare profession. The authors also evaluated LLMs contamination on questions released in 2024 after pretraining data cut-off of LLMs and found that the performance was similar to that of the original test set. An automatic traduction of the benchmark in Simplified Chinese and English has been conducted to analyze the performance difference.

The paper introduced EMPEC which was designed to benchmark large language models across a diverse range of healthcare professions.
Summary of strengths: - The benchmark covers 20 healthcare professions, significantly expanding beyond the physician-centric focus of previous medical benchmarks. This provides a more holistic evaluation of LLMs' healthcare knowledge.

- The investigation of performance across traditional Chinese, simplified Chinese, and English translations provides important insights into language-specific challenges and model capabilities.

- The questions are sourced from official Taiwanese medical examinations, ensuring authenticity and professional relevance. Each question is tagged with release time, enabling evaluation of models on unseen questions.

- The paper conducts extensive experiments across 17 different LLMs, including both proprietary and open-source models, providing valuable insights into current capabilities and limitations.

- The findings reveal important gaps in current medical LLMs, particularly in specialized fields, which can guide future development of healthcare AI systems.

(A) New artifact for evaluating LLMs in Traditional Chinese that include medical subjects and healthcare professions that are not present in other Chinese medical benchmarks. 

(B) LLMs performance analysis from multiple perspectives: models, healthcare professions, language with automatic translation.

Evaluated 17 LLMs. EMPEC included 157,803 multiple-choice questions, covering 124 subjects and 20 healthcare professions.
Summary of weaknesses: - While the paper includes English translations, the primary focus is on Chinese medical examinations. This may limit the generalizability of findings to other healthcare systems and languages.

- The paper could benefit from a deeper analysis of why medical-specific LLMs underperform their general-purpose counterparts, including potential architectural or training data explanations.

- The paper acknowledges that some professions have relatively smaller representation (e.g., Optometrists and Public Health Specialists with less than 1% of questions). This imbalance could affect the reliability of evaluations for underrepresented professions and bias the understanding of the results.

- The English translation is generated by GPT-4 without expert verification, which could introduce biases or inaccuracies in the English evaluation results. It's also possible to face concepts that are inexistant in other languages or proper to the Chinese healthcare system.

- While the paper includes questions from 2011-2024, the analysis might not provide a complete picture of temporal generalization and could be more detailed.

(C) Limited novelty in task and data: The multiple-choice question answering (MCQA) format with four-choice datasets is well-established in LLM evaluation, with several existing Chinese benchmarks. Moreover, the authors demonstrate no significant performance difference between Traditional and Simplified Chinese questions. This suggests that it may be interesting to examine the overlap between EMPAC and existing Chinese benchmarks to only include in EMPEC unique questions that are not in other benchmarks, particularly for the common healthcare professions across those datasets. Evaluating models on questions that are not in existing Simplified Chinese benchmarks would provide more originality and insight. Additionally, such an approach could help assess potential data contamination in LLM training, as some EMPAC questions might already be present in other Simplified Chinese datasets.

(D) The models included in this study are relatively outdated, with some being over a year old. Given the rapid pace of LLM advancements, more recent versions should be considered. While it is understandable that running new experiments to evaluate newer models can be costly, the study should discuss the benchmark difficulty and its long-term relevance. The best-performing model, GPT-4 Turbo, achieves 75% accuracy. Given the rapid progress in LLM capabilities, recently released models such as o1, o3-mini, or DeepSeek-R1 are likely to give 90%+ accuracy. This calls into question the dataset's usefulness over time and its lifespan as a benchmark, at least for closed-source models.

(E) The writing could benefit from enhancements in certain sections. Ideally, the final manuscript should undergo additional editing for clarity and fluency. For instance, there are several redundant pieces of information in the introduction.

Experiment design. Please check Comments Suggestions And Typos.
Comments, suggestions and typos: - Consider adding a section discussing potential biases in the source examinations and how they might affect the benchmark's generalizability.

- The paper would benefit from a more detailed discussion of the implications of the findings for healthcare AI development and deployment.

- Consider adding error analysis to understand common failure patterns across different professions.

- The paper could be strengthened by including a comparison with existing medical benchmarks in terms of question difficulty and coverage.

- Consider adding a discussion of potential ethical implications and guidelines for using this benchmark in healthcare AI development.
Minor typos:
> Line 100-101: Repetition of "there remains a significant gap in our ability to assess the performance of LLMs across the broader spectrum of medical contexts"
> Line 261: "Regulations" should be in quotes
> Line 270: "GPT-4 2" should be "GPT-4"

(F) The dataset was split according to healthcare professions. What is the distribution of questions per year for each healthcare profession?

(G) l381-382: "The evaluation procedure follows common practice in previous work". What are those practices ?

Typo:
l535: "exmaine" -> "examine"

The authors translated the test data into English and translated it to simplified Chinese in the second step. The healthcare professional vocabulary is highly sensitive and the GPT-based translations are not that reliable.

All questions were only from Taiwan’s national healthcare exams.

The official exam questions are always knowledge-based rather than generated from real clinical situations. It focused on knowledge recall.

The large models sometimes use a lot of anonymous crawled data. We can't verify the data of these data clearly. Older exam questions may have already been leaked into LLM training sets. 

HuatuoGPT2 performed worse than LLMs on EMPEC. I think there is a risk of overfitting. Please add more details.
Overall assessment scores: 3

2

1.5
Metareview score: 2
Metareview: This paper introduces EMPEC (Examinations for Medical PErsonnel in Chinese), a benchmark designed to evaluate LLMs across healthcare professions. EMPEC consists of 157k exam questions across 124 subjects and 20 healthcare roles, including underrepresented professions like Optometrists and Audiologists. The authors evaluated 17 LLMs, revealing that general-purpose models (e.g., GPT-4) outperformed domain-specific ones and identified challenges in specialized fields and alternative medicine. They further explored performance variations across languages (Traditional Chinese, Simplified Chinese, and English translations) and over time, offering insights for future development of LLMs in healthcare.
Summary of reasons to publish: - EMPEC extends beyond physician-centric benchmarks to include 20 healthcare professions and 124 subjects, making it a broader and more inclusive evaluation resource.

- The paper conducts an in-depth analysis across multiple dimensions: healthcare professions, language variations, and model types (general-purpose vs. medical-specific LLMs), providing valuable insights into LLM capabilities and limitations.