Should I Believe in What Medical AI Says? A Chinese Benchmark for Medication Based on Knowledge and Reasoning
Authors: Yue Wu, Yangmin Huang, Qianyun Du, Lixian Lai, Zhiyang He, Jiaxue Hu, Xiaodong Tao
Accepted at: acl25

Should I Believe in What Medical AI Says? A Chinese Benchmark for Medication Based on Knowledge and Reasoning

Yue Wu, Yangmin Huang*, Qianyun Du*, Lixian Lai,
Zhiyang He, Jiaxue Hu, Xiaodong Tao
Xunfei Healthcare Technology Co., Ltd.
{yuewu21, ymhuang9, qydu, lxla1i2, zyhe, jxhu2, xdtao}@iflytek.com

Abstract
Large language models (LLMs) show potential in healthcare but often generate hallucinations, especially when handling unfamiliar information. In medication, a systematic benchmark to evaluate model capabilities is lacking, which is critical given the high-risk nature of medical information. This paper introduces a Chinese benchmark aimed at assessing models in medication tasks, focusing on knowledge and reasoning across six datasets: indication, dosage and administration, contraindicated population, mechanisms of action, drug recommendation, and drug interaction. We evaluate eight closed-source and five open-source models to identify knowledge boundaries, providing the first systematic analysis of limitations and risks in proprietary medical models.

1 Introduction
Large language models (LLMs) have made significant strides in various domains, including medication, where they provide information and recommendations related to medical treatments (Singhal et al., 2022; Nori et al., 2023). However, a significant challenge remains: these models are prone to generating hallucinations and confidently providing incorrect or incomplete information, especially in cases where they lack adequate knowledge (Stefansson and Johansson, 2021; Shukla et al., 2022). In the context of medication and drug usage, such hallucinations can lead to critical errors, particularly in high-risk situations like identifying contraindicated populations or recommending unsafe drug combinations. Despite the progress made in medical AI, a notable gap remains in the development of systematic benchmarks to evaluate the full range of a model’s capabilities in medication applications.

Considering that many existing public benchmarks are constructed by collecting data from online sources and may be susceptible to data leakage risks, we construct a Chinese benchmark, ChiDrug, from scratch, based on authoritative drug brochures. This benchmark is specifically designed to evaluate LLMs’ knowledge and reasoning capabilities in the medication domain. As shown in Figure 1, our benchmark is structured into two key subdimensions: parametric knowledge and reasoning capability. We construct six diverse datasets that cover crucial aspects of drug information, including dosage and administration, indication, contraindicated population, mechanisms of action, medication recommendation, and drug interaction.

To evaluate the capabilities of existing models, we apply our benchmark to eight closed-source and five open-source models. Our work also explores various methods for expressing knowledge boundaries, providing insights into the potential risks of overconfident but inaccurate AI-generated responses.

Our contributions include: (1) This benchmark

![Our benchmark involves four datasets that directly examine model parametric knowledge and two datasets that examine model reasoning ability.](page_682_370_482_312.png)

Figure 1: Our benchmark involves four datasets that directly examine model parametric knowledge and two datasets that examine model reasoning ability.
serves as the first systematic tool for analyzing the capabilities of LLMs in the field of medicine across various dimensions. (2) To the best of our knowledge, this is the first work to systematically conduct knowledge boundary analysis on medical models in the medication domain, providing a comprehensive overview of their performance in real-world medical applications.

2 Related Work

2.1 Chinese Benchmark in Medication

Evaluating the capabilities of Large Language Models (LLMs) in the medical field necessitates specialised benchmarks, particularly when addressing Chinese medical texts. Recent efforts have led to the development of several Chinese-specific medical benchmarks, focusing on various domains such as clinical question answering, knowledge recall, and medication recommendations (Singhal et al., 2023; Liu et al., 2024; Wang et al., 2024; Yue et al., 2024).

MedExpQA (Alonso et al., 2024) proposes a multilingual benchmark evaluating models on medical question-answering tasks, including drug-related and clinical guideline questions. DialMed (He et al., 2022) focuses on dialogue-based medication recommendations, testing models on handling patient symptom queries and drug interactions. However, existing datasets do not have a dedicated benchmark built in the field of medication in Chinese to evaluate the model’s ability in this area.

2.2 Abstention in LLMs

The ability of Large Language Models (LLMs) to refrain from providing answers when uncertain—is crucial for enhancing model reliability and safety. Studies have explored various methods to improve this capability (Wen et al., 2024):

Currently, methods to guide models in refusing to answer include: Calibration-Based Methods: After the model provides an answer, continue by asking, “Are you sure about your answer?” to verify its confidence (Tian et al., 2023). Training-Based Methods: Construct a training set containing both questions the model can answer and those it cannot, training the model to refuse to answer questions with unfamiliar knowledge (Slobodkin et al., 2023; Zhang et al., 2023; Stengel-Eskin et al., 2024). Consistency-Based Methods: Perform multiple samplings and calculate the consistency score of the model’s responses to assess reliability (Kuhn et al., 2022; Feng et al., 2024a). Token Probability Methods: Ensemble the probability of each token generated by the model to determine the uncertainty of the response (Liang et al., 2024; Malinin and Gales, 2021).

3 Dataset

ChiDrug is designed to assess models’ parametric knowledge and reasoning ability in handling critical medication-related tasks. Below, we outline the dataset construction process and the verification procedures used to ensure the quality and reliability of the data. The entire benchmark construction process is shown in Figure 2.

3.1 Dataset Construction

We began by collecting official drug brochures for existing medications from the internet1. We organized this information into a table that includes details on 8,000 drugs, encompassing their generic names, ingredients, specifications, indications, dosages, contraindications, drug interactions, adverse reactions, and mechanisms of action. This structured dataset served as the foundation for developing questions that evaluate the model’s parametric knowledge in four areas: Indication, Dosage and Administration, Contraindicated Population, and Mechanism of Action. We extracted the relevant sections from each drug brochure and utilized Spark2 to generate multiple-choice question stems and answer options. In constructing these questions, we ensured that the incorrect options did not overlap with the correct ones (the left part of Figure 2).

The second step involved constructing questions for Medication Recommendation (the middle part of Figure 2). We collected doctor-patient dialogues from the existing DIALMED dataset (He et al., 2022), where Spark transformed these dialogues into question formats, using the doctor’s recommended medication as the correct option. To generate distractor options that could confuse the model, we first used Spark to extract the patient’s symptoms and demographic information, then searched the drug brochures for medications that treat the same symptoms but are not suitable for the patient’s demographic group, thereby creating incorrect options (e.g., “symptom in indication and de-

1https://drugs.dxy.cn
2https://xinghuo.xfyun.cn
<table>
  <tr>
    <th>ID</th>
    <th>Medication Name</th>
    <th>Indications</th>
    <th>Dosage and Administration</th>
    <th>Contraindicated Population</th>
    <th>Mechanisms of Action</th>
  </tr>
  <tr>
    <td>1</td>
    <td>Azithromycin Dispersible Tablets 阿奇霉素分散片</td>
    <td>It is indicated for infections caused by susceptible bacteria, such as those caused by Chlamydia trachomatis in adults.</td>
    <td>For sexually transmitted diseases caused by Chlamydia trachomatis, the dose of 1.5 g is sufficient... 对沙眼衣原体所致性传播疾病，成人单次口服本品1.5g。</td>
    <td>This drug is contraindicated in patients with a history of hypersensitivity to azithromycin or other macrolide antibiotics. 对本品或其它大环内酯类抗生素过敏者禁用。</td>
    <td>Its mechanism of action is to bind to the 50S subunit of the ribosome in sensitive microorganisms. 阿奇霉素作用机制为与敏感微生物的核糖体50S亚基结合。</td>
  </tr>
  <tr>
    <td>2</td>
    <td>...</td>
    <td>...</td>
    <td>...</td>
    <td>...</td>
    <td>...</td>
  </tr>
</table>

Indications
Azithromycin dispersible tablets are mainly used to treat which of the following infections?
(A) Bronchitis
(B) Fatty liver

Contraindicated Population
Which of the following groups should avoid using azithromycin dispersible tablets?
(A) Patients allergic to penicillin
(B) Patients allergic to macrolide antibiotics
(C) Patients allergic to acetylsalicylic acid

Mechanism of action
What is the main mechanism by which azithromycin inhibits the growth of sensitive microorganisms?
(A) Inhibits the 30S subunit in ribosomes
(B) Inhibiting the 50S subunit in ribosomes

Usage and dosage
What is the recommended single oral dose of azithromycin dispersible tablets for treating sexually transmitted diseases caused by Chlamydia trachomatis in adult patients?
(A) 1.5 g...
(B) 1.5 g...

Parametric Knowledge Requests

Reasoning Skill Requests

Figure 2: Overview of our benchmark construction process

mographic in contraindicated population").

In the third step, we constructed a dataset for Drug Interaction (the right part of Figure 2). First, doctors defined three risk levels for drug interaction (high, medium, and low). We then randomly selected a drug from the brochures and identified its combination guidelines. From there, we extracted the ingredients involved in drug interactions and further searched for medications that contained the same ingredients. Finally, we input the two drugs and the interaction documentation into Spark to generate the appropriate risk level as the correct answer.

3.2 Verification
Since we automatically generated the questions for the dataset, we implemented a double-check process to ensure the questions were reasonable. Each question was tested by three large models (GPT-43, Qwen-max4, ERNIE bot5). We gave these models the question, options, and document sources and asked them to check the following: (1) If the question is clear, well-phrased, and free of ambiguity. (2) If the answer is correct. (3) If the answer is unique. A question was considered valid only if all three models agreed it was correct. During dataset construction, we initially generated 7,100 questions. In the LLM verification stage, Spark-generated questions had an acceptance rate of 79.32%, resulting in the removal of 1,468 questions.

Additionally, we hire doctors with licensed qualifications to examine all the datasets we construct (Appendix C).

Ultimately, we construct a benchmark dataset comprising a total of 5,243 samples, encompassing the following categories: Indication (705), Dosage and Administration (651), Contraindicated Population (659), Mechanism of Action (773), Medication Recommendation (838), and Drug Interaction (1,617). More details are provided in Appendix B.

4 Experiment
In this section, we evaluate the performance of large language models (LLMs) on our benchmark. We assess both closed-source and open-source models, using our benchmark to examine their capabilities in handling medication-related queries and their ability to identify knowledge gaps and overconfidence. Table 1 presents the results for the model ability, while the second table focuses on the methods to express the knowledge boundaries in seven different methods.

4.1 Model Performance Evaluation
We selected models with strong Chinese language capabilities, including GPT40 (Hurst et al., 2024), Claude3.5-Sonnet6, Qwen-max7, Doubao8, GLM4

3https://chatgpt.com
4https://tongyi.aliyun.com/qianwen
5https://yiyan.baidu.com
6https://claude.ai
7https://tongyi.aliyun.com/qianwen
8https://www.doubao.com/chat
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">Close-source Models</th>
    <th colspan="3">Medication Recommendation</th>
    <th colspan="3">Drug Interaction</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Dosage and Administration</th>
    <th>Indication</th>
    <th>Contraindicated Population</th>
    <th>Mechanism of Action</th>
    <th>Avg.</th>
    <th>Drug Interaction</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>XiaoYi</td>
    <td><b>81.1</b></td>
    <td><u>77.87</u></td>
    <td>66.51</td>
    <td>92.35</td>
    <td><b>68.31</b></td>
    <td><b>63.37</b></td>
    <td>73.52</td>
  </tr>
  <tr>
    <td>GPT4o</td>
    <td>66.41</td>
    <td>73.65</td>
    <td>69.35</td>
    <td>92.13</td>
    <td>59.79</td>
    <td>59.93</td>
    <td>70.21</td>
  </tr>
  <tr>
    <td>ERNIE Bot</td>
    <td>67.64</td>
    <td>65.3</td>
    <td>57.97</td>
    <td>92.76</td>
    <td>51.43</td>
    <td>38.59</td>
    <td>62.28</td>
  </tr>
  <tr>
    <td>Qwen-max</td>
    <td>69.02</td>
    <td>72.13</td>
    <td>68.19</td>
    <td>93.28</td>
    <td>61.22</td>
    <td>54.73</td>
    <td>69.76</td>
  </tr>
  <tr>
    <td>Doubaio</td>
    <td>71.32</td>
    <td>71.24</td>
    <td>54.17</td>
    <td>92.77</td>
    <td>63.25</td>
    <td>55.35</td>
    <td>68.02</td>
  </tr>
  <tr>
    <td>GLM4</td>
    <td><u>71.32</u></td>
    <td><u>72.71</u></td>
    <td><u>71.02</u></td>
    <td><b>94.16</b></td>
    <td>59.79</td>
    <td>54.92</td>
    <td>71.15</td>
  </tr>
  <tr>
    <td>Claude3.5</td>
    <td>54.59</td>
    <td>74.53</td>
    <td>70.29</td>
    <td>89.92</td>
    <td>54.06</td>
    <td>60.23</td>
    <td>67.24</td>
  </tr>
  <tr>
    <td>Baichuan4</td>
    <td>62.14</td>
    <td>69.97</td>
    <td>69.24</td>
    <td>90.35</td>
    <td>52.98</td>
    <td>52.81</td>
    <td>66.25</td>
  </tr>
  <tr>
    <th colspan="9">Open-source Models</th>
  </tr>
  <tr>
    <td>Bencao</td>
    <td>28.92</td>
    <td>19.88</td>
    <td>12.2</td>
    <td>40.71</td>
    <td>16.23</td>
    <td><b>38.28</b></td>
    <td>26.04</td>
  </tr>
  <tr>
    <td>MedGLM</td>
    <td>38.92</td>
    <td>17.21</td>
    <td>8.75</td>
    <td>44.86</td>
    <td>20.17</td>
    <td>34.59</td>
    <td>26.75</td>
  </tr>
  <tr>
    <td>MedicalGPT</td>
    <td>33.51</td>
    <td>10.14</td>
    <td>3.18</td>
    <td>49.41</td>
    <td>13.84</td>
    <td>30.98</td>
    <td>23.51</td>
  </tr>
  <tr>
    <td>ChiMedical</td>
    <td>33.51</td>
    <td>16.04</td>
    <td>14.32</td>
    <td>38.54</td>
    <td>24.71</td>
    <td>36.05</td>
    <td>27.20</td>
  </tr>
  <tr>
    <td>HuatuoGPT2</td>
    <td><b>55.83</b></td>
    <td><u>47.03</u></td>
    <td><u>18.66</u></td>
    <td><u>77.16</u></td>
    <td><u>25.18</u></td>
    <td><u>25.60</u></td>
    <td><u>41.58</u></td>
  </tr>
</table>

Table 1: This table presents the performance of 8 closed-source models and 5 open-source models across various medication-related tasks. Bold indicates the best performance, while underlining denotes the second-best.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">Precision</th>
    <th colspan="3">A-Acc</th>
    <th colspan="3">Precision</th>
    <th colspan="3">A-Acc</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Dosage and Administration</th>
    <th>Indication</th>
    <th>Contraindicated Population</th>
    <th>Mechanism of Action</th>
    <th>Avg.</th>
    <th>Drug Interaction</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>77.41</td>
    <td>77.41</td>
    <td>77.41</td>
    <td>18.66</td>
    <td>77.16</td>
    <td>77.16</td>
    <td>25.18</td>
    <td>25.18</td>
    <td>41.58</td>
    <td>41.58</td>
  </tr>
  <tr>
    <td>Post-calibration</td>
    <td><b>55.94</b></td>
    <td><u>57.91</u></td>
    <td>57.01</td>
    <td>18.57</td>
    <td>77.41</td>
    <td>77.41</td>
    <td>25.18</td>
    <td>25.18</td>
    <td>41.58</td>
    <td>41.58</td>
  </tr>
  <tr>
    <td>IDK</td>
    <td>54.26</td>
    <td>54.88</td>
    <td>47.18</td>
    <td>17.90</td>
    <td>78.68</td>
    <td>80.28</td>
    <td>24.47</td>
    <td>24.66</td>
    <td>41.79</td>
    <td>41.79</td>
  </tr>
  <tr>
    <td>LNS</td>
    <td>53.99</td>
    <td>50.46</td>
    <td>50.78</td>
    <td>18.18</td>
    <td>23.03</td>
    <td>79.97</td>
    <td>71.07</td>
    <td>28.22</td>
    <td>26.21</td>
    <td>40.87</td>
    <td>40.87</td>
  </tr>
  <tr>
    <td>Probing</td>
    <td>66.11</td>
    <td>43.90</td>
    <td>66.11</td>
    <td>51.86</td>
    <td>22.83</td>
    <td>27.43</td>
    <td>85.24</td>
    <td>80.01</td>
    <td>19.80</td>
    <td>31.31</td>
    <td>50.24</td>
    <td>42.26</td>
  </tr>
  <tr>
    <td>Running</td>
    <td>57.79</td>
    <td>53.79</td>
    <td>58.06</td>
    <td>22.37</td>
    <td>31.90</td>
    <td>81.90</td>
    <td>70.08</td>
    <td>12.55</td>
    <td>14.25</td>
    <td>28.40</td>
    <td>29.59</td>
    <td>29.59</td>
  </tr>
  <tr>
    <td>Semantic Entropy</td>
    <td><b>66.85</b></td>
    <td>46.20</td>
    <td>50.03</td>
    <td>52.54</td>
    <td>11.67</td>
    <td><b>89.37</b></td>
    <td><b>78.60</b></td>
    <td><b>30.15</b></td>
    <td><b>20.86</b></td>
    <td><b>20.90</b></td>
    <td><b>30.01</b></td>
    <td><b>39.13</b></td>
  </tr>
  <tr>
    <td>Semantic Entropy</td>
    <td>64.79</td>
    <td>55.32</td>
    <td><u>70.28</u></td>
    <td>52.71</td>
    <td>26.95</td>
    <td>86.24</td>
    <td><u>83.09</u></td>
    <td>29.22</td>
    <td>24.95</td>
    <td><u>38.56</u></td>
    <td><u>31.76</u></td>
    <td><u>52.87</u></td>
    <td><u>45.79</u></td>
  </tr>
</table>

Table 2: This table displays the performance of 7 different methods on the models’ ability to detect knowledge boundaries and manage uncertainty.

(GLM et al., 2024), Baichuan49, XiaoYi10, and ERNIE Bot11, for evaluation of closed-source models. For open-source models, we chose Bencao (Wang et al., 2023), MedGLM (Haochun Wang, 2023), MedicalGPT (Xu, 2023), ChiMedical (Tian et al., 2024), and HuatuoGPT2 (Chen et al., 2024) for evaluation.

The results summarised in Table 1 show that the closed-source models generally outperformed the open-source models across all dimensions, with XiaoYi leading in overall performance, followed closely by GPT4o and ERNIE Bot. In Open-source models, Bencao and MedicalGPT demonstrated lower performance, particularly in complex tasks like Contraindicated Populations and Drug Interactions, while HuatuoGPT2 generally outperformed other models. We will provide a more detailed analysis of each model in the Appendix A.

4.2 Methods for Knowledge Boundary Detection

4.2.1 Task Definition

A formal definition of knowledge boundary detection can be briefly stated as follows:
Input: A model \( M \) and a query \( q \).
Output: A response \( r \) where

\[
r = \begin{cases}
M(q), & \text{within parametric knowledge} \\
U, & \text{beyond parametric knowledge}
\end{cases}
\]

where \( M(q) \) is the model’s generated answer, and \( U \) is an explicit uncertainty expression or abstenion (e.g., “I don’t know”, a confidence score, or an alternative uncertainty marker).

4.2.2 Methods

In this subsection, we apply seven methods to explore their impact on expressing uncertainty or abstention, using HuatuoGPT2 as the backbone.

Post-Calibration (Tian et al., 2023): Enhances model confidence by prompting it to verbalize its certainty after providing an answer.

IDK (I Don’t Know): Similar to non-of-the-above (NOTA) in (Feng et al., 2024b), we incorporate an additional “I don’t know” option and instruct the model to abstain from answering.

LNS (Malinin and Gales, 2021): Utilizes probabilistic ensemble-based techniques to assess uncertainty in structured prediction tasks, aiding in more reliable outputs.

Probing (Slobodkin et al., 2023): Analyzes internal model representations to understand how they encode information about answerability, helping detect overconfidence and hallucinations.
R-tuning (Zhang et al., 2023): Instructs models to explicitly state when they lack knowledge, reducing the generation of hallucinated information.

Self-Consistency (Kuhn et al., 2023): Enhances reasoning by generating multiple reasoning paths and selecting the most consistent answer, improving response reliability.

Semantic Entropy (Feng et al., 2024a): Estimates uncertainty in natural language generation by considering linguistic invariances, allowing models to assess the reliability of their outputs better.

4.2.3 Evaluation Metrics
Given the definition as follows:
\( C \) = the number of correct answers.
\( A \) = the number of total answered questions (excluding abstentions).
\( A_{\text{correct}} \) = the number of correct abstentions, i.e., questions the model correctly refused to answer because the answer was unknown or uncertain.
\( N \) = the total number of questions.
In this experiment, two evaluation metrics are used:

Precision: Measure the proportion of correct answers out of the total predictions made, without abstaining.

\[
\text{Precision} = \frac{C}{A}
\]

Abstain Accuracy (Feng et al., 2024a): Evaluates the proportion of correct answers and correct abstention due to uncertainty.

\[
\text{Abstain-Acc} = \frac{C + A_{\text{correct}}}{N}
\]

Results are shown in Table 2. Post-calibration and IDK cannot achieve good results in Huatuogpt2 due to its weak instruction capabilities. Self-Consistency improved accuracy in complex tasks like Medication Recommendations. Probing refined uncertainty estimations with varying effectiveness. R-tuning reduced hallucinations but sometimes sacrificed performance on complex tasks, while LNS showed mixed results, improving Medication Recommendations but hindering performance on Drug Interaction. Overall, Semantic Entropy has achieved good results in both metrics, and we further analyze the effectiveness of this method on multiple models in Appendix D.

5 Conclusion
We present ChiDrug, a benchmark designed to evaluate LLMs (Large Language Models) in medication-related tasks, with an emphasis on their knowledge and reasoning abilities. Both GLM4 and XiaoYi performed exceptionally well; however, even these advanced models exhibited gaps in drug knowledge. Our work highlights the need for effective methods to align the knowledge boundaries of LLMs, particularly for high-risk tasks.

6 Limitations
This study primarily focuses on Chinese medical texts, which may limit its generalizability. The benchmark doesn’t fully capture the complexities of real-world medical decision-making. Additionally, model generalization to new knowledge, handling uncertainty, and reliance on high-quality, up-to-date data are ongoing challenges for AI in healthcare.

7 Ethical considerations
The medication dictionary we constructed is entirely sourced from DingXiangYuan12, a public medical website. The data on DingXiangYuan was also collected from the China National Medical Products Administration, and the website has a statement allowing non-commercial citations. Documents within our dictionary do not contain private information, so there is no risk of privacy leakage. All the drug information we collect has obtained the national drug approval certificate and is free from copyright issues, in accordance with the regulations of the Chinese government.

References
Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. 2024. Medexpqa: Multilingual benchmarking of large language models for medical question answering. Artificial intelligence in medicine, 155:102938.

Junying Chen, Xidong Wang, Ke Ji, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, Dingjie Song, Wenya Xie, Chuyi Kong, Jianquan Li, Xi-ang Wan, Haizhou Li, and Benyou Wang. 2024. Huatuogpt-ii, one-stage training for medical adaptation of llms. Preprint, arXiv:2311.09774.

Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. 2024a. Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration. arXiv preprint arXiv:2402.00367.

Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. 2024b.

12https://drugs.dxy.cn
Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration. Preprint, arXiv:2402.00367.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793.

Sendong Zhao Bing Qin Ting Liu Haochun Wang, Chi Liu. 2023. Chatglm-med: chatglm. https://github.com/SCIR-HI/Med-ChatGLM.

Zhenfeng He, Yuqiang Han, Zhenqiu Ouyang, Wei Gao, Hongxu Chen, Guandong Xu, and Jian Wu. 2022. Dialmed: A dataset for dialogue-based medication recommendation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 721–733.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os-trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2022. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations.

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664.

Xun Liang, Shichao Song, Zifan Zheng, Hanyu Wang, Qingchen Yu, Xunkai Li, Rong-Hua Li, Yi Wang, Zhonghao Wang, Feiyu Xiong, et al. 2024. Internal consistency and self-consistency in large language models: A survey. arXiv preprint arXiv:2407.14507.

Mianxin Liu, Jinru Ding, Jie Xu, Weigou Hu, Xiaoyang Li, Lifeng Zhu, Zhan Bai, Xiaoming Shi, Benyou Wang, Haitao Song, Pengfei Liu, Xiaofan Zhang, Shanshan Wang, Kang Li, Haofen Wang, Tong Ruan, Xuanjing Huang, Xin Sun, and Shaoting Zhang. 2024. Medbench: A comprehensive, standardized, and reliable benchmarking system for evaluating chinese medical large language models. Preprint, arXiv:2407.10990.

Andrey Malinin and Mark Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations.

Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems. Preprint, arXiv:2303.13375.

Abhay Shukla, Paheli Bhattacharya, Soham Poddar, Raddeep Mukherjee, Kripabandhu Ghosh, Pawan Goyal, and Saptarshi Ghosh. 2022. Legal case document summarization: Extractive and abstractive methods and their evaluation. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1048–1064.

Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, Philip Mansfield, Blaise Aguera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Senturs, Alan Karthikesalingam, and Vivek Natarajan. 2022. Large language models encode clinical knowledge. Preprint, arXiv:2212.13138.

Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaeckermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Senturs, S. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. 2023. Towards expert-level medical question answering with large language models. Preprint, arXiv:2305.09617.

Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. 2023. The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3607–3625.

Elis Stefansson and Karl H. Johansson. 2021. Computing complexity-aware plans using kolmogorov complexity. Preprint, arXiv:2109.10303.

Elias Stengel-Eskin, Peter Hase, and Mohit Bansal. 2024. Lacie: Listener-aware finetuning for confidence calibration in large language models. arXiv preprint arXiv:2405.21028.

Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafaílov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433–5442.

Yuanhe Tian, Ruyi Gan, Yan Song, Jiaxing Zhang, and Yongdong Zhang. 2024. Chined-gpt: A chinese medical large language model with full training regime
and better alignment to human preferences. Preprint, arXiv:2311.06025.

Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. 2023. Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975.

Xidong Wang, Guiming Chen, Song Dingjie, Zhang Zhiyi, Zhihong Chen, Qingying Xiao, Junying Chen, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, and Haizhou Li. 2024. CMB: A comprehensive medical benchmark in Chinese. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6184–6205, Mexico City, Mexico. Association for Computational Linguistics.

Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang. 2024. Know your limits: A survey of abstention in large language models. arXiv preprint arXiv:2407.18418.

Ming Xu. 2023. Medicalgpt: Training medical gpt model. https://github.com/shibing624/MedicalGPT.

Wenjing Yue, Xiaoling Wang, Wei Zhu, Ming Guan, Huanran Zheng, Pengfei Wang, Changzhi Sun, and Xin Ma. 2024. Tcmbench: A comprehensive benchmark for evaluating large language models in traditional chinese medicine. Preprint, arXiv:2406.01126.

Hanning Zhang, Shizhe Diao, Yong Lin, Yi R Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2023. R-tuning: Teaching large language models to refuse unknown questions. arXiv preprint arXiv:2311.09677.

A Model Performance Analysis

A.1 Visualization of Model Performance

In this section, we present radar chart visualizations to highlight the performance of both closed-source and open-source models across different medication-related tasks. As shown in Figure 3 and Figure 4, the radar charts provide a clear, comparative view of how various models handle tasks such as Indication, Dosage and Administration, Contraindicated Population, and Mechanisms of Action. Notably, models such as GLM4 and XiaoYi stand out for their excellent performance, with XiaoYi leading the closed-source models and GLM4 demonstrating remarkable consistency. On the other hand, HuatuoGPT2 significantly outperforms the other open-source models. These findings underscore the importance of model selection in high-stakes domains like healthcare, where the quality of responses directly impacts patient safety.

![Radar Chart Representation of Open-Source Models Performance.](page_1012_120_388_388.png)

Figure 3: Radar Chart Representation of Open-Source Models Performance.

![Radar Chart Representation of Close-Source Models Performance.](page_1012_540_388_388.png)

Figure 4: Radar Chart Representation of Close-Source Models Performance.

A.2 Knowledge Mastery Assessment of Common Drugs

To further evaluate model capabilities, we focus on a subset of 282 commonly used drugs. For each drug, we constructed questions about Indication, Dosage and Administration, Contraindicated Population, and Mechanism of Action, drawing from the benchmark dataset. The knowledge boundary of the models was then assessed by visualizing their performance on these tasks, as shown in the radar charts for GLM4, XiaoYi, and GPT4o, and the results are presented in Table 3.

Figure 5 illustrates the knowledge boundary performance of three models across 282 drugs, where each drug has 4 different sub-task questions. The radial score axis (0-4) represents the total number of correct answers per drug: Since each drug has 4 questions, a score of 4.0 means the model answered all 4 questions correctly.

Orange regions indicate that the model had only one chance to answer all 4 questions.

Yellow regions indicate that the model had up to 5 attempts per drug to answer correctly. If it answered any question correctly in any of these attempts, it was awarded 1 point.
The yellow regions that are not covered by orange represent cases where the model initially gave incorrect answers but later recovered and answered correctly within the 5 attempts. These visualizations emphasize that while some models show robustness in their knowledge, significant gaps remain in certain drug-related tasks.

The results indicate that GLM4 and XiaoYi exhibit stronger consistency in answering these questions correctly across the four tasks compared to GPT4o. However, there were cases where even the most advanced models struggled to demonstrate comprehensive knowledge across all aspects of these drugs. This highlights a key issue—despite their advanced capabilities, large models still fall short in areas of medication-related knowledge.

<table>
  <tr>
    <th>Model</th>
    <th>Dosage and Administration</th>
    <th>Indication</th>
    <th>Contraindicated Population</th>
    <th>Mechanisms of Action</th>
  </tr>
  <tr>
    <td>NanoYi</td>
    <td>82.27</td>
    <td>78.61</td>
    <td>63.12</td>
    <td>72.25</td>
    <td>78.98</td>
  </tr>
  <tr>
    <td>GPT4o</td>
    <td>67.02</td>
    <td>74.11</td>
    <td>70.2</td>
    <td>92.91</td>
    <td>76.24</td>
  </tr>
  <tr>
    <td>ERNIE</td>
    <td>68.35</td>
    <td>64.89</td>
    <td>58.51</td>
    <td>81.11</td>
    <td>70.97</td>
  </tr>
  <tr>
    <td>Qwen-max</td>
    <td>67.73</td>
    <td>74.11</td>
    <td>71.46</td>
    <td>82.82</td>
    <td>76.31</td>
  </tr>
  <tr>
    <td>Qwen-mini</td>
    <td>70.49</td>
    <td>75.49</td>
    <td>66.42</td>
    <td>82.82</td>
    <td>76.31</td>
  </tr>
  <tr>
    <td>GLM4</td>
    <td>74.11</td>
    <td>75.89</td>
    <td>71.99</td>
    <td>83.97</td>
    <td>78.99</td>
  </tr>
  <tr>
    <td>Chuan1.5</td>
    <td>67.09</td>
    <td>67.73</td>
    <td>54.26</td>
    <td>81.46</td>
    <td>70.04</td>
  </tr>
  <tr>
    <td>Baichuan</td>
    <td>59.93</td>
    <td>70.21</td>
    <td>51.08</td>
    <td>81.49</td>
    <td>68.17</td>
  </tr>
</table>

Table 3: Performance of various models on Common Drugs

A.3 Performance on Reasoning Models

While the GPT family is known for its strong reasoning capabilities, our results reveal nuanced performance differences, as shown in Table 4. In particular, OpenAI’s o1 performs worse than GPT-4o, suggesting that strong reasoning ability alone does not guarantee superior performance, especially in knowledge-intensive domains like medicine. We argue that a model’s performance ceiling in such domains is also closely tied to its parametric knowledge.

For instance, DeepSeek-R1 excels in tasks such as Drug Indication and Contraindicated Population, and this strength naturally extends to better performance in Medication Recommendation, which relies heavily on knowledge of drug usage constraints. In contrast, o1 performs worse in Indication, which correlates with its weaker performance in the Medication Recommendation task.

Furthermore, although GPT-4o is not explicitly trained on complex chain-of-thought (CoT) reasoning datasets, it demonstrates competitive reasoning ability. However, models like o1 and o3-mini, which are optimised for reasoning in code and mathematics, do not show a clear advantage in our medical reasoning benchmark, highlighting the limits imposed by insufficient medical knowledge.

To further validate that reasoning ability becomes the primary bottleneck when knowledge is sufficient, we constructed knowledge-complete prompts for Medication Recommendation and Drug Interaction tasks. These prompts explicitly included all necessary domain knowledge required to answer the questions.

As the results illustrated in Table 5 confirm, our hypothesis is supported: when provided with sufficient knowledge, models with strong reasoning capabilities, such as o1 and o3-mini, outperform GPT-4o, which lacks comparable reasoning-specific training. This highlights the importance of not only enriching models with knowledge but also enhancing their reasoning mechanisms—especially in professional domains like medicine.

In summary, our benchmark suggests that advancing domain-specific reasoning is a critical frontier for LLM development, and we hope our work offers meaningful insight for future research in this direction.

B Dataset Statistics

This section presents key statistics of our benchmark dataset across six task categories. As shown in Table 6, each task varies in terms of average input length, the number of unique drugs, and the number of associated diseases. Notably, the Drug Interaction task contains the longest samples and the largest set of drugs, reflecting its complexity. These statistics highlight the diversity and richness of our dataset, which is crucial for evaluating both the parametric knowledge and reasoning capabilities of LLMs in the medication domain.

C Expert Review of Datasets

During the manual verification phase, we hire 10 doctors with licensed qualifications to examine all the datasets we construct. We divided the 10 doctors into two groups, each consisting of five doctors. One group was responsible for reviewing the questions related to Drug Indication, Dosage and Administration, Contraindicated Population, and Mechanisms of Action, while the other group reviewed the questions for Medication Recommendation and Drug Interaction. After the initial review, the groups conducted a cross-check to ensure accuracy.

For data points that the doctors identified as problematic, we directly archived them and excluded
Figure 5: Knowledge boundary chart for GLM4, XiaoYi, and GPT4o across 282 common drugs. The orange area indicates that the model answered correctly once, while the yellow area indicates 5 times opportunities to answer correctly. The orange areas in the yellow-covered region represent cases where a model made an error in a single attempt but was able to recover after multiple tries.

<table>
  <tr>
    <th>Task</th>
    <th>Drug Indication</th>
    <th>Drug Dosage and Administration</th>
    <th>Drug Contraindicated Population</th>
    <th>Drug Mechanism of Action</th>
    <th>Medication Recommendation</th>
    <th>Drug Interaction</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>DeepSeek-R1</td>
    <td>70.81</td>
    <td>77.06</td>
    <td>75.69</td>
    <td>91.98</td>
    <td>61.59</td>
    <td>62.75</td>
    <td>73.31</td>
  </tr>
  <tr>
    <td>OpenAI o1</td>
    <td>56.69</td>
    <td>74.80</td>
    <td>58.31</td>
    <td>92.93</td>
    <td>61.53</td>
    <td>45.10</td>
    <td>64.89</td>
  </tr>
  <tr>
    <td>OpenAI o3-mini</td>
    <td>68.52</td>
    <td>70.69</td>
    <td>72.57</td>
    <td>94.15</td>
    <td>51.77</td>
    <td>64.79</td>
    <td>70.42</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>68.41</td>
    <td>73.65</td>
    <td>69.35</td>
    <td>92.13</td>
    <td>59.79</td>
    <td>59.93</td>
    <td>70.21</td>
  </tr>
</table>

Table 4: Zero-shot accuracy of reasoning models across medical knowledge tasks.

<table>
  <tr>
    <th>Model</th>
    <th>Medication Recommendation</th>
    <th>Drug Interaction</th>
  </tr>
  <tr>
    <td>OpenAI o1</td>
    <td>84.73</td>
    <td>92.02</td>
  </tr>
  <tr>
    <td>OpenAI o3-mini</td>
    <td>80.79</td>
    <td>91.34</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>79.83</td>
    <td>88.62</td>
  </tr>
</table>

Table 5: Accuracy on reasoning tasks with knowledge-complete prompts.

<table>
  <tr>
    <th>Task</th>
    <th>Average Length</th>
    <th>Drugs Involved</th>
    <th>Diseases Involved</th>
  </tr>
  <tr>
    <td>Drug Indication</td>
    <td>86.21</td>
    <td>605</td>
    <td>1061</td>
  </tr>
  <tr>
    <td>Drug Dosage and Administration</td>
    <td>108.36</td>
    <td>581</td>
    <td>188</td>
  </tr>
  <tr>
    <td>Drug Contraindicated Population</td>
    <td>94.45</td>
    <td>607</td>
    <td>636</td>
  </tr>
  <tr>
    <td>Drug Mechanism of Action</td>
    <td>108.25</td>
    <td>791</td>
    <td>224</td>
  </tr>
  <tr>
    <td>Medication Recommendation</td>
    <td>114.61</td>
    <td>540</td>
    <td>830</td>
  </tr>
  <tr>
    <td>Drug Interaction</td>
    <td>479.97</td>
    <td>1293</td>
    <td>None</td>
  </tr>
</table>

Table 6: Task-wise statistics of the ChiDrug benchmark.

them from the final dataset. As a result, the accept rate of question is 93.09%. The cost of hiring an doctor to label a single sample was 2 RMB (approximately 0.26 USD).

D Semantic Entropy (SE) Method for Knowledge Boundary Expression

In this section, we explore the Semantic Entropy (SE) method used to detect knowledge boundaries, as introduced in Section 4.2. The SE method is particularly noteworthy for its effectiveness in expressing model uncertainty and improving response reliability, as demonstrated in our experiments. We applied this method to HuatuoGPT2 and XiaoYi, observing that it significantly enhanced the models’ performance on challenging tasks, such as Medication Recommendations and Drug Interactions.

As shown in Table 7, the SE method proved to be robust and consistent across different model architectures and sizes. It improved both Precision and Abstain Accuracy, regardless of the model’s scale. This reinforces the notion that SE is an effective tool for managing uncertainty, making it an essential method for enhancing the reliability of models in real-world medical applications.

E Case Study

In Figure 6, we present a case to illustrate the practical section of the ChiDrug.
<table>
  <tr>
    <th>Model</th>
    <th>Method</th>
    <th>Dosage and Administration</th>
    <th>Indication</th>
    <th>Contraindicated Population</th>
    <th>Mechanisms of Action</th>
    <th>Medication Recommendation</th>
    <th>Drug Interaction</th>
  </tr>
  <tr>
    <td rowspan="2">HuatuoGPT2</td>
    <td>w/o SE</td>
    <td>Precision: 35.83<br>A-Acc: 35.83</td>
    <td>Precision: 47.03<br>A-Acc: 47.03</td>
    <td>Precision: 18.66<br>A-Acc: 18.66</td>
    <td>Precision: 77.16<br>A-Acc: 77.16</td>
    <td>Precision: 25.18<br>A-Acc: 25.18</td>
    <td>Precision: 25.60<br>A-Acc: 25.60</td>
  </tr>
  <tr>
    <td>SE</td>
    <td>Precision: 79.47<br>A-Acc: 79.47</td>
    <td>Precision: 82.11<br>A-Acc: 82.11</td>
    <td>Precision: 28.71<br>A-Acc: 28.71</td>
    <td>Precision: 83.71<br>A-Acc: 83.71</td>
    <td>Precision: 86.57<br>A-Acc: 86.57</td>
    <td>Precision: 87.27<br>A-Acc: 87.27</td>
  </tr>
  <tr>
    <td rowspan="2">XiaoYi</td>
    <td>w/o SE</td>
    <td>Precision: 81.1<br>A-Acc: 81.1</td>
    <td>Precision: 77.87<br>A-Acc: 77.87</td>
    <td>Precision: 26.71<br>A-Acc: 26.71</td>
    <td>Precision: 92.85<br>A-Acc: 92.85</td>
    <td>Precision: 59.31<br>A-Acc: 59.31</td>
    <td>Precision: 63.27<br>A-Acc: 63.27</td>
  </tr>
  <tr>
    <td>SE</td>
    <td>Precision: 83.42<br>A-Acc: 84.46</td>
    <td>Precision: 94.01<br>A-Acc: 94.15</td>
    <td>Precision: 80.14<br>A-Acc: 81.28</td>
    <td>Precision: 91.71<br>A-Acc: 92.1</td>
    <td>Precision: 67.71<br>A-Acc: 75.71</td>
    <td>Precision: 63.85<br>A-Acc: 69.02</td>
  </tr>
</table>

Table 7: Application of the SE method on HuatuoGPT2 and XiaoYi models, showcasing the performance improvements achieved through the SE method. This method enhances precision and uncertainty handling, effectively reducing hallucinations.

<table>
  <tr>
    <th>Tasks</th>
    <th>Question</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td>Dosage and Administration</td>
    <td>复方锌布颗粒剂的推荐服用方式为多少包，多少次一天？<br>(A)儿童半岁一次2包，一天3次<br>(B)成人一次2包，一天3次<br>(C)成人一次1包，一天3次<br>(D)6～14岁儿童一次1包，一天2次<br>(E)6～14岁儿童一次1包，一天3次</td>
    <td>B<br>用法用量：<br>口服。3～5岁儿童，一次半包；6～14岁儿童，一次1包；成人，一次2包，一日3次。</td>
  </tr>
  <tr>
    <td>Indication</td>
    <td>复方锌布颗粒剂主要用于缓解以下哪些症状？<br>(A)普通感冒引起的发热<br>(B)急性胃肠炎<br>(C)普通感冒引起的四肢酸痛<br>(D)普通感冒引起的打喷嚏</td>
    <td>ACD<br>适应症：<br>用于缓解普通感冒或流行性感冒引起的发热、头痛、四肢酸痛、鼻塞、流涕、打喷嚏等症状。</td>
  </tr>
  <tr>
    <td>Contraindicated Population</td>
    <td>复方锌布颗粒剂不适用于以下哪些人群？<br>(A)心肌病患者<br>(B)哺乳期妇女<br>(C)对阿司匹林过敏的哮喘患者<br>(D)高血压患者</td>
    <td>BC<br>禁忌：<br>1.对其他非甾体抗炎药过敏者禁用。2.孕妇及哺乳期妇女禁用。3.对阿司匹林过敏的哮喘患者禁用。</td>
  </tr>
  <tr>
    <td>Mechanism of Action</td>
    <td>关于复方锌布颗粒剂各组分的主要药理作用是：<br>(A)布洛芬具有抗炎作用，葡萄糖酸锌促进蛋白质合成，马来酸氯苯那敏D有镇咳作用。<br>(B)布洛芬具有解热镇痛作用，葡萄糖酸锌能增强免疫功能，马来酸氯苯那敏能抑制前列腺素合成，具有解热镇痛作用。<br>(C)布洛芬为抗炎解热药，葡萄糖酸锌具有解热功能，马来酸氯苯那敏具有镇咳作用。<br>(D)布洛芬为解热镇痛药，葡萄糖酸锌参与多种酶的合成与激活，马来酸氯苯那敏为抗组胺药</td>
    <td>作用机制：<br>布洛芬能抑制前列腺素合成，具有解热镇痛作用；葡萄糖酸锌中锌离子能参与多种酶的合成与激活，有增强吞噬细胞的吞噬能力的作用。马来酸氯苯那敏为抗组胺药，能减轻由感冒或流感引起的咳嗽、流涕、打喷嚏等症状。</td>
  </tr>
  <tr>
    <td>Drug Recommendation</td>
    <td>回答以下不定项选择题（可能包含1个或多个正确选项）：一位晚期肺癌患者因为咳嗽出现咳嗽、喉咙里有异物感以及扁桃体发炎，可以考虑推荐的药物是：<br>(A)双黄连口服液<br>(B)热毒宁注射液<br>(C)贝美前列素滴眼液<br>(D)银杏胶囊</td>
    <td>A<br>双黄连口服液的适应症：<br>疏风解表，清热解毒。用于外感风热所致的感冒，症见发热、咳嗽、咽痛。</td>
  </tr>
  <tr>
    <td>Drug Interaction</td>
    <td>注射用氨菲酶与抗纤溶药联用的风险等级是？</td>
    <td>高风险<br>注射用氨菲酶<br>使用本品应避免与水杨酸类药物（如：阿司匹林）合用。抗凝血药可加强本品作用，引起意外出血；抗纤溶药可抵消本品作用，禁止联用。</td>
  </tr>
</table>

Figure 6: Partial cases of ChiDrug on 6 sub datasets.

Reviewer paper summaries: This paper introduces ChiDrug, a Chinese benchmark aimed at evaluating LLMs in medication-related QA, including: indication, dosage and administration, contraindicated population,
mechanisms of action, drug recommendation, and drug interaction. Several mainstream advanced LLMs as well as medical LLMs are evaluated. and the knowledge boundaries are explored through experiments.

This work introduces a Chinese benchmark containing six datasets in medication domain and conducts experiments on eight closed-source models and five open-source models.
Summary of strengths: The benchmark contains multiple dimensions of medicine knowledge, providing a valuable dataset as well as evaluation results. And it uses multiple LLM abstention methods to explore the capabilities of LLM from a knowledge boundary perspective.

* This work introduces a Chinese benchmark containing six datasets in medication domain.

* This work conducts experiments on eight closed-source models and five open-source models.

Summary Of Weaknesses:

* The data is sourced from the internet and may have been pre-trained as part of existing corpora. This may lead a data leakage problem, author should discuss it.
Summary of weaknesses: 1. The data in this paper were obtained from the Internet, and the author may need to discuss copyright issues.

2. In the construction process of the dataset, the authors use the Spark model to automatically extract fields and generate QA pairs. My confusion as a reader is that the reasons for using Spark are not explained. Is there any potential bias in relying on Spark (especially for non-multiple-choice questions)? If additional checks and modifications are made manually, then Spark should also be included in the evaluation baseline.

3. Some expressions are informal, e.g., the criteria for "make sense" are not clear(lines 169-170).

4. The task definition for “Knowledge Boundary Detection” is missing in Section 4.1. It seems that the authors wanted to use some methods such as self-consistency and abstention to validate the knowledge of medicines within the LLMs, but a clear task definition may help better understanding.

5. As a benchmark paper, the composition of the annotators and more of the annotation process needs to be represented. More analyses need to be shown, such as error case analysis, from which more intuition can be provided for subsequent research.

* The data is sourced from the internet and may have been pre-trained as part of existing corpora. This may lead a data leakage problem, author should discuss it.
Comments, suggestions and typos: The captions in Figures 3 and 4 are reversed.

I have tried to understand what Figure 5 expresses, but it is still unclear, especially the meaning of the numbers in the figure.

"Additionally, We" in line 173.

Questions:

* I'm not sure the details about the verification. Could author clarify the annotation evaluation in lines 169-171? In particular, how can large models determine if a question "makes sense"? Please provide examples. Additionally, are doctors required to have the same annotation guideline as LLM?

* Please clarify the details about IDK method and cite a reference since I'm not sure of the mechanism.

Comments:

* Since the knowledge boundary analysis is one of the contributions in this work, author should give a formal definition about the knowledge boundary detection, at least containing the input and output.

* More detailed data statistics are needed, such as the average word/token length.

* It is best to outline the dataset construction steps in Figure 2, such as labeling them as Step 1, Step 2, etc., to enhance readability and comprehension.

* Author should discuss the data privacy and copyright in a "Ethical Consideration" section.

typos:

* citations lack the year in line 96, line 139, line 219, line 236.

* "Additionally, We" shoud be "Additionally, we" in line173.
Overall assessment scores: 3.5

3.5
Metareview score: 3.5
Metareview: This paper introduces a new benchmark in Chinese for evaluating LLMs for medication tasks. Authors experiment with various LLMs and provide an analysis of various models in the medication domain.
Summary of reasons to publish: 1. The paper introduces a Chines medication benchmark to evaluate LLMs. 
2. Authors experiments with variety of open weights and close source LLMs and provide in-depth analysis.