Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models

## Abstract

Abstract Large-scale language models (LLMs) have achieved remarkable success across various language tasks but suffer from hallucinations and temporal misalignment. To mitigate these shortcomings, Retrieval-augmented generation (RAG) has been utilized to provide external knowledge to facilitate the answer generation. However, applying such models to the medical domain faces several challenges due to the lack of domain-specific knowledge and the intricacy of real-world scenarios.
In this study, we explore LLMs with RAG framework for knowledge-intensive tasks in the medical field. To evaluate the capabilities of LLMs, we introduce MedicineQA, a multi-round dialogue benchmark that simulates the real-world medication consultation scenario and requires LLMs to answer with retrieved evidence from the medicine database. MedicineQA contains 300 multi-round question-answering pairs, each embedded within a detailed dialogue history, highlighting the challenge posed by this knowledge-intensive task to current LLMs. We further propose a new Distill-Retrieve-Read framework instead of the previous Retrieve-then-Read . Specifically, the distillation and retrieval process utilizes a tool calling mechanism to formulate search queries that emulate the keyword-based inquiries used by search engines. With experimental results, we show that our framework brings notable performance improvements and surpasses the previous counterparts in the evidence retrieval process in terms of evidence retrieval accuracy. This advancement sheds light on applying RAG to the medical domain.

## 1 Introduction

Large language models (LLMs) (Achiam et al., ; Touvron et al., ; Team et al., ) have revolutionized the field of natural language processing, showing remarkable impacts with the well-documented emergence of zero-shot capabilities in a variety of downstream tasks, like machine translation (Zhang et al., ), text generation (Kojima et al., ) and machine reading comprehension (Samuel et al., ). Such impressive abilities stem from the ever-increasing number of parameters and large-scale training corpus.

Despite the massive knowledge, LLMs still struggle with considering issues of hallucination (i.e., prone to generate factually incorrect statements) (Bang et al., ; Ji et al., ) and temporal misalignment (i.e., unable to capture the changing world) (Kandpal et al., ) in a set of tasks (Yin et al., ; Lewis et al., ). Such knowledge-intensive tasks require access to a vast amount of knowledge beyond the training data.
Towards this issue, existing methods (Li et al., ; Jiang et al., ; Xu et al., ; Wang et al., ; Cheng et al., ) incorporated external knowledge with LLMs by retrieval augmentation, dubbed as Retrieval Augmented Generation (RAG). In detail, LLMs retrieve the relevant information for the input query and utilize the retrieved evidence as additional context to generate the response. Such Retrieve-then-Read framework cleverly combines flexible knowledge sources in a non-parameterized form for knowledge-intensive tasks and has become one of the hottest paradigms to alleviate the drawbacks in naive LLM generations.

Figure: Figure 1: The medication consultation: a detailed discussion between healthcare professionals and users about prescribed medications, including their names, indications, usage, side effects, etc. Professionals utilize the knowledge in the medicine database to provide a more robust response.

Beneath the advancements, we find a notable gap in applying LLMs to medical fields, especially for knowledge-intensive tasks, like medication consultation. As shown in Figure 1, medication consultation aims at providing real-time accessibility for medication-related inquiries and enhancing medication safety through searching from the database, requiring depth in domain-specific areas. In real-world scenarios, the dialogs are usually ambiguous and verbose, e.g., users tend to use layman’s terms instead of standard terms and provide much more information than what might be medically relevant.
We ask: Is the LLM with vanilla RAG enough for the medication consultation?

In this work, we introduce a new benchmark, MedicineQA, to evaluate the proficiency of LLMs in medication consultation scenarios. We recruited a panel of 5 board-certified physicians to create the benchmark as follows: sourcing and rephrasing questions from an online medical consultation website; simulating multiple rounds of dialogue scenarios via GPT-4 (Achiam et al., ). Our research reveals that vanilla RAG methods suffer from serious challenges in retrieving relevant information with intricate dialogue history.

Based on PULSE Zhang et al. (), we propose RagPULSE via the search engine tool. Instead of the Retrieve-then-Read framework adopted by previous retrieval-augmented work, RagPULSE utilizes a novel Distill-Retrieve-Read framework to access the external knowledge. Specifically, RagPULSE processes a medication inquiry by summarizing the dialogue history to keywords for searching API calls and integrating the retrieved evidence from the medicine database to formulate a comprehensive response.

Our main contributions can be summarized as follows:

- •
We present MedicineQA, a new benchmark derived from real-world medication consultation, aimed at evaluating LLMs’ ability in the medical domain.
- •
We propose a pioneering retrieval augmentation framework, Distill-Retrieve-Read, via the “tool calling” mechanism.
- •
Incorporated with the framework, our proposed RagPUSLE outperforms all publicly available models in performance and is competitive with state-of-the-art commercial products with a smaller parameter size.

## 2 Related Work

Large Language Model in Medical Domain.
The impressive abilities of large language models (LLMs) across various applications have catalyzed extensive investigation into employing them in healthcare and medical domains. This surge in attention is documented through a growing body of research (Thirunavukarasu et al., ; Clusmann et al., ). Some recent works have studied to augment LMMs with real-world data. ChatDoctor (Li et al., ), trained by fine-tuning LLaMA (Touvron et al., ) on a large dataset of patient-doctor dialogues, achieves high accuracy and reliability in medical scenarios with an external information retrieval module. From the other line, some adopt the synthetic data for fine-tuning. Zhang et al. () utilized real-world data from medical professionals alongside distilled data from ChatGPT to fine-tune the model. To enhance the capability in the multi-round conversation, BianQue (Chen et al., ) trained the model on a self-constructed dataset containing multi-round inquiries and health suggestions. Despite the remarkable performance, there is still a gap in applying LLMs in real-world scenarios due to the lack of domain-specific knowledge. To further evaluate the proficiency of LLMs in medical domains, we introduce MedicineQA, a benchmark derived from real-world medication consultation scenarios.

Retrieval-Augmented Generation.
LLMs require external knowledge to alleviate the factuality drawbacks.
Retrieval-augmented generation (RAG) has been regarded as an effective solution to mitigate the aforementioned hallucinations and temporal misalignment issues inherent in large language models, especially for knowledge-intensive tasks.
Generally, studies of RAG can be categorized into three types (Gao et al., ), namely Naive RAG, Advanced RAG, and Modular RAG. Naive RAG means a straightforward Retrieve-then-Read framework (Lewis et al., ; Karpukhin et al., ; Izacard et al., ). To enhance retrieval quality, the Advanced RAG builds upon the foundation of Naive RAG by incorporating pre-retrieval (Li et al., ) and post-retrieval (Jiang et al., ; Xu et al., ) strategies. Modular RAG improves the overall performance by decomposing the Retrieve-then-Read framework into fine-grained modules with distinct functionalities, such as a search module(Wang et al., ), memory module(Cheng et al., ).

## 3 Method

In Section (3.1), we propose MedicinceQA, a novel benchmark to evaluate LLMs’ capabilities toward knowledge-intensive tasks in medical fields. We curate the benchmark from various real-world medication consultation scenarios and unified them into multi-round dialogue. Then, we present RagPULSE in Section (3.2), a dedicated pipeline that adopts Distill-Retrieve-Read framework for multi-round medication consultation. The fundamental operations of RagPULSE comprise three main steps: (1) the LLM calls the search engine tool and distills the dialogue history into a new query to gather evidence from the external medicine database; (2) the generated search query is executed to retrieve related evidence following a hierarchical form; (3) the retrieved evidence is provided to the LLM, and the LLM respond the user’s question by the retrieved evidence.

### 3.1 Benchmark Creation

Existing benchmarks for evaluating the capabilities of LLMs in medical fields primarily focus on widely known or widely available tasks given a specific context (e.g., Automatic Structuring of medical reports and Named Entity Recognition). However, these benchmarks are insufficient for assessing LLMs’ proficiency in knowledge-intensive tasks. Therefore, we introduce MedicineQA, a novel benchmark designed for evaluating LLMs within the context of medication consultation.

Data Collection. In an effort to align the benchmark with real-world scenarios, we crawled data from websites for medical consultation, which comprise numerous online consultation records between users and medical experts. Each record contains multiple rounds of dialogue, we categorized each record into three categories: 1) Diagnostic Process, where the expert diagnoses based on symptoms provided by the user; 2) Medication Consultation, where the expert addresses queries regarding medications for certain conditions; 3) Other, which includes the patient’s medical history and some trivial communication. In total, we amassed 1,028,090 records comprising 6.24M pairs.

Data Refinement. Given the crawled data, we first conducted an initial statistical analysis and identified the 200 most commonly mentioned medicines as the scope for further processing.
To ensure the correctness, we recruited a panel of 5 board-certified physicians to curate the content.
The physicians filtered out irrelevant dialogues of each selected record and summarized it into one question about a specific medicine.
For each summarized question, we utilized GPT-4 Achiam et al. () to expand them into multi-round dialogue. Subsequently, physicians manually revised the dialogues to ensure a logical progression of questions, with each answer building on the information provided in the preceding dialogues and without repeating information. This process yielded 300 multi-round dialogue questions focused on medication consultation.

Medicine Database.
To provide precise and structured information, we introduce an entity-oriented medicine database with 42764 medicines, where each medicine is represented in three forms: brand name, generic name, and detailed attributes like usage, contraindications, adverse reactions, etc. Formally, for each medicine $M_{i}$ in our database $D$, we first concatenated its generic name with each attribute $a_{j}$ to obtain the entity-attribute items $E_{ij}$, respectively. Then, each item is embedded into vectors and stored in a tree form according to the entity, i.e., the information of the medicine $M_{i}$ is stored in the form of $E_{i}=\{E_{i1},E_{i2},E_{i3},\dots\}$, accompanied by its corresponding keys $K^{n}_{i}$ and $\{K_{i1}^{a},K_{i2}^{a},K_{i3}^{a},\dots\}$. In our database $D$, $E_{i}$ and $E_{ij}$ can be obtained via $D[K^{n}_{i}]$ and $D[K_{ij}^{a}]$, respectively.

Figure: Figure 2: (a) The distribution of our proposed MedicineQA. MedicineQA involves ten specific scenarios of the medication consultation. The distribution of the benchmark is similar to that of the real scenario. (b) Samples of the benchmark: Interaction, Adverse reactions, and Contraindications. Our benchmark is available in both English and Chinese.

Annotation.
In our benchmark, each question is associated with the corresponding medicine descriptions extracted from the medicine database, to serve as the retrieved evidence. To evaluate the retrieval process, we further labeled two types of retrieval ground truths: one is the document-level for coarse-grained evaluation $K_{c}$, and the other is the specific sections in the relevant documents for fine-grained attribute-level assessment $K_{f}$. One sample of our MedicineQA can be formulated as $\mathbf{S}=<{H,Q_{T+1},K_{c},K_{f}}>$, where $H=\{(Q_{i},A_{i})\},i=1,2,\dots,T$ is the dialogue history, $(Q_{i},A_{i})$ denotes a round of conversation between the user and the agent, and $T$ is the number of dialogue rounds. $Q_{T+1}$ represents a question about one specific medicine. $K_{c},K_{f}$ are the coarse-grained and fine-grained ground truth for evaluating the retrieval process, respectively. In detail, $K_{c}$ is the $K^{n}_{i}$ in $D$, and $K_{f}$ is a subset of $\{K_{i1}^{a},K_{i2}^{a},K_{i3}^{a},\dots\}$. We display the relative distribution of our proposed benchmark and present samples of the created data in Figure 2.

### 3.2 RagPULSE

Figure: Figure 3: The overall workflow of our RagPULSE in the medication consultation scenario, consists of three steps: (1) Distilling the key information and forming the searching query from the dialogue history; (2) Retrieving the corresponding medicine evidence from the medicine database; (3) Generating the response according to the retrieved evidence.

We choose PULSE (Zhang et al., ) as the LLM, which demonstrates impressive performance in the medical field, and augment it with the Distill-Retrieve-Read framework. As shown in Figure 3, the process can be formulated into three steps. The LLM is first tasked to call the search engine tool and summarize the search query supported by the combination $[H,Q_{T+1}]$. Subsequently, the search engine retrieves relevant keys $\hat{K}$ from the medicine database $D$ and obtains the evidence $\hat{E}$ from the medicine database $D$. Finally, the LLM generates the answer $A_{T+1}$ according to $[H,Q_{T+1},\hat{E}]$.

Tool Calling. A simple but robust retrieval query is vital to clarify the search need from the context and eliminate irrelevant information in the external knowledge base. Recent studies either directly adopt the query from the dataset (Liu et al., ) or rewrite it by the black-box generation (Ma et al., ). However, there is inevitably a gap between the query and the evidence that needs to be obtained, especially for such a task with a long context. Only relying on the original capability of the LLM and human-written prompt lines makes it difficult to summarize correct inquiries from the intricate context while preserving key information. Inspired by the program of thought (PoT) (Chen et al., ), where the LLM generates Python code for retrieving, we integrate “tool calling” with the LLM. This approach prompts the LLM to generate search keywords for search tools, mimicking the use of search engines. With the above paradigm, the LLM is able to call the search tool and generate the retrieval query according to the current dialogue.

Synthetic Dataset. To endow the LLM with the distillation ability, we construct a synthetic dataset for the dialogue distilling task following previous works (Ma et al., ; Hsieh et al., ; Ho et al., ). First, we collect a large-scale question set (including but not limited to dialogue questions and search engine questions) from several websites (e.g., Google and Baidu). Then, the selected questions are distilled and summarized as pseudo labels by prompting GPT-4 (Achiam et al., ) to utilize function call. After fine-tuning, the LLM shows remarkable performance in distilling the context into simple inquiries containing key information. The samples of synthetic data and the instructions for “tool calling” are shown in Table 1.

Table: Table 1: The instructions and samples of the synthetic dataset for fine-tuning the LLM.

## 4 Experiments

In this section, we measure the performance of RagPULSE on MedicineQA and compare it to existing LLMs and commercial products (4.2). We ablate the Distill-Retrieve-Read on the MedicineQA dataset, showing their importance (4.3). Finally, we present some cases to investigate the hallucinations of LLMs towards medication consultation.

### 4.1 Experimental Settings

Implementation Details
We develop RagPULSE with Distill-Retrieve-Read framework in Pytorch (Paszke et al., ) and fine-tune it by the proposed synthetic dataset. It is worth noting that a single
machine with eight NVIDIA A100 GPUs proved sufficient for the memory requirements of PULSE (Zhang et al., ). Our training framework integrates tensor parallelism (Wang et al., ) and ZeRO-powered data parallelism (Rajbhandari et al., ).
To further accelerate training without sacrificing accuracy, we implement mixed-precision training, where we execute forward and backward computations in BFloat16 and conduct optimizer updating in Float32. For the compared models, we adopt the pre-trained weights and settings provided on the official website.

Baselines. Given the variety of current LLMs and the fact that MedicineQA is the medical domain, we choose open-sourced models and commercial products with notable performance in the medical domain to fully explore the current proficiency of LLMs in medication consultation scenarios. For a fair comparison, we utilize models that the results can be reproduced as follows: DoctorGLM (Xiong et al., ), ChatGLM3 (Du et al., ), BianQue2 (Chen et al., ), MING (Liao et al., ), QWen2 (Bai et al., ), Baichuan2 (Baichuan, ) and ChatGPT3.5 (^1^11https://chat.openai.com)

Metrics. To evaluate the accuracy of the evidence retrieval stage, we employ the Hit Rate (HR@num), which represents the proportion of instances where the retrieval candidates contain the corresponding knowledge, with “num” indicating the number of candidates to be retrieved. We respectively calculate the hit rate of coarse-grained and fine-grained retrieval through the retrieved database key and the search ground truth. Given the answer of the medication consultation is in the form of free text, which is a challenge for evaluating the correctness, we utilize the Elo rating system (Elo, ; Chiang et al., ; Dettmers et al., ) to gauge the performance of LLMs on MedicineQA. It adjusts a player’s rating based on the outcome of their games, taking into account the expected score versus the actual score. In our settings, each model is one competitor, and the powerful GPT-4 (Achiam et al., ) serves as the referee to determine which model performs better. More details can be seen in the Appendix.

**Table 2: Evaluation on MedicineQA. Our study employs the PULSE model with varying parameter sizes, augmented by the Distill-Retrieve-Read framework. We compare them with other LLMs and commercial products. “Retrieved Doc.” refers to the process of only searching the generic name of the medicine (coarse-grained), while “Retrieved Attr.” denotes calculating the results via the combination of the generic name and the specific attribute (fine-grained).**
| Model Name | Param.<br>Size | Ins. follow<br>rate (%) | Retrieved Doc. (%) | Retrieved Attr. (%) | Generation |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HR@1 | HR@5 | HR@10 | HR@1 | HR@5 | HR@10 | Elo Rating | Elo Rank |  |  |  |
| BianQue2 | 6B | 3.33 | 7.33 | 9.00 | 10.00 | 1.67 | 2.00 | 2.00 | 883 | 10 |
| DoctorGLM | 6B | 47.00 | 12.67 | 15.00 | 16.00 | 2.33 | 2.67 | 3.00 | 920 | 8 |
| ChatGLM3 | 6B | 92.33 | 27.33 | 32.00 | 34.00 | 8.00 | 9.33 | 9.67 | 999 | 7 |
| MING | 7B | 8.00 | 20.00 | 28.33 | 30.67 | 5.67 | 7.67 | 8.00 | 1017 | 6 |
| BenTsao | 7B | 16.67 | 33.33 | 45.33 | 48.00 | 12.67 | 17.33 | 18.33 | 913 | 9 |
| Baichuan2 | 14B | 98.33 | 52.67 | 66.67 | 71.33 | 26.67 | 35.33 | 38.00 | 1045 | 4 |
| QWen2 | 14B | 100.00 | 57.67 | 68.33 | 76.67 | 25.33 | 28.33 | 30.33 | 1018 | 5 |
| ChatGPT3.5 | - | 100.00 | 63.67 | 72.33 | 78.67 | 27.00 | 31.33 | 32.67 | 1072 | 2 |
| RagPULSE | 7B | 100.00 | 63.67 | 73.00 | 78.33 | 28.33 | 32.00 | 33.33 | 1058 | 3 |
| RagPULSE | 20B | 100.00 | 65.67 | 75.33 | 78.33 | 27.33 | 31.67 | 32.33 | 1074 | 1 |

### 4.2 Results

Here we thoroughly evaluate models using the MedicineQA benchmark. To assess the performance of evidence retrieval, we prompt those baseline models to formulate search queries by summarizing preceding dialogues and then calculate their accuracy in retrieving relevant evidence. Due to the limitations of some baseline models in retrieving evidence from the medicine database, we immediately adopt the attached corresponding medicine information as the context to guide the generation of the final responses. It is worth noting that our RagPULSE leverages the retrieved evidence to generate the answer. Experimental results are reported in Table 2.

From Table 2, we can see that some open-sourced models with smaller model sizes suffer from following the instructions for summarizing key information in specific format from complex dialogue histories, highlighting the inherent difficulties in medication consultation tasks. Finetuned on the synthetic dataset, our RagPUSLE (7B) presents a surprising performance in the instruction following rate. This outcome validates the effectiveness of adopting the code form of “tool calling,” underscoring the potential benefits of integrating programming paradigms into LLMs to bolster their understanding and execution of complex tasks.
As shown in Table 2, the Distill-Retrieve-Read framework brings performance gains for the evidence retrieval process. Incorporated with the ability to distill dialogue history, RagPULSE is capable of summarizing the retrieval query. Compared with models whose number of parameters is less than 7 billion, RagPUSLE (7B) demonstrates a notable performance enhancement in the context of retrieval accuracy, achieving at least a 30% improvement in document retrieval and a 15% increase in attribute retrieval according to HR@1 metrics. This shows that some of the current open-sourced LLMs still struggle with distilling key information from the long context for searching relevant evidence. Regarding the models with more parameters, RagPUSLE (7B) still maintains a substantial lead, as evidenced by a 5% improvement in HR@1. Surprisingly, RagPUSLE (7B) surpasses all models in attribute retrieval and RagPUSLE (20B) performs better than ChatGPT (65.67 vs. 63.67 in document retrieval ). These results indicate that using “tool calling” to distill context benefits the query generation. Moreover, we can see that RagPULSE outperforms all competing models and products in terms of responding to medication consultation even with the retrieved evidence. Depending on the remarkable capabilities of PULSE in the medical field, RagPULSE achieves a higher score than other open-sourced models.

Additionally, RagPULSE distinguishes itself across all metrics in the domain of medication consultation responses, even when utilizing retrieved evidence. Attributable to the specialized proficiency of PULSE in medical contexts, RagPULSE attains higher performance metrics than other publicly available models. This evidence highlights the superior capability of the Distill-Retrieve-Read framework in processing and responding to complex medical inquiries, reinforcing its value in enhancing the accuracy and relevance of evidence retrieval in this specialized field.

**Table 3: Ablation of the Distill-Retrieve-Read framework. The “History” setting implements the retrieval process by using dialogue history as the query and the “Last Question” setting conducts searching via the last question. We also prompt RagPULSE by the instruction used for baseline models, which are denoted as ^†.**
| Model Name | Param.<br>Size | Retrieved Doc. (%) | Retrieved Attr. (%) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HR@1 | HR@5 | HR@10 | HR@50 | HR@1 | HR@5 | HR@10 | HR@50 |  |  |
| History | - | 18.33 | 27.00 | 31.00 | 40.33 | 5.33 | 6.67 | 7.67 | 9.00 |
| Last Question | - | 28.33 | 35.00 | 37.67 | 40.00 | 12.33 | 15.67 | 16.33 | 17.67 |
| PULSE | 7B | 53.00 | 62.67 | 66.00 | 70.33 | 18.00 | 21.00 | 22.00 | 23.33 |
| RagPULSE^† | 7B | 58.67 | 69.67 | 75.67 | 78.67 | 19.67 | 22.67 | 23.67 | 25.00 |
| RagPULSE | 7B | 63.67 | 73.00 | 78.33 | 82.00 | 28.33 | 32.00 | 33.33 | 35.00 |
| PULSE | 20B | 56.33 | 66.33 | 69.67 | 74.00 | 22.00 | 26.33 | 26.67 | 28.00 |
| RagPULSE^† | 20B | 60.33 | 70.67 | 75.00 | 81.00 | 29.33 | 34.00 | 34.67 | 38.67 |
| RagPULSE | 20B | 65.67 | 75.33 | 78.33 | 82.33 | 27.33 | 31.67 | 32.33 | 35.33 |

### 4.3 Ablation Studies

To fully investigate the contribution of our proposed Distill-Retrieve-Read framework, we conduct a quantitative analysis and report performances on MedicineQA when toggling the distillation part. The first two rows of Table 3 underscore the importance of distilling key information from dialogue history, which otherwise includes extraneous details detrimental to effective evidence retrieval. In addition, relying solely on the most recent query for information search proves inadequate due to the critical context embedded within the dialogue. Notably, RagPULSE (7B) exhibits more pronounced improvements, which outperforms PULSE (7B) with a notable 10% improvement.

Furthermore, as in the previous experiments, we also prompt our models to summarize the keywords without calling the tool. Compared with the PULSE without fine-tuning, RagPULSE^† are observed to have significant performance gains in the two retrieval results. The results validate the effectiveness of our proposed synthetic dataset for summarizing the history and confirm that fine-tuning models on our synthetic dataset can endow models with distillation abilities.

Figure: Figure 4: Case studies of LLMs’ retrieval process and generated responses. LLMs first summarize the dialogue history and then generate search queries. The responses are formulated via the retrieved document. Key information is marked by red text.

### 4.4 Case Study

To intuitively show how the Distill-Retrieve-Read framework makes a difference in the evidence retrieval process, we present examples (i.e., ChatGLM3, Baichuan2, ChatGPT3.5, and RagPULSE-7B) in Figure 4 to compare the generated searching queries and the retrieved evidence. As can be seen in the upper part, in scenarios involving lengthy history, extraneous information often leads to the generation of redundant and ineffective search queries. It is evident that, despite LLMs’ ability to generate queries encapsulating all necessary information, the complexity of such queries frequently results in retrieval failures. In the lower part, although the query contains the corresponding medicine, the LLMs fail to understand the question, resulting in the omission of crucial keywords. Additionally, we can observe that ChatGPT3.5 still fails despite generating the correct keywords since the query does not contain key information about the question. These examples clearly indicate the state of current LLMs in the medication scenarios. With supplemented knowledge, RagPULSE shows hopeful performance in generating responses for medication consultation.

## 5 Conclusion

In this paper, we introduce MedicineQA, a new benchmark derived from real-world medication consultations, which aims at evaluating the capabilities of LLMs towards knowledge-intensive tasks in the medical domain. Our study shows that the LLM with vanilla RAG is not enough for the medication consultation. To address this, we propose RagPULSE with a novel framework, Distill-Retrieve-Read, which revolutionizes the conventional Retrieve-then-Read through the innovative use of the “tool calling” mechanism. Extensive experiments demonstrate that our model gains superior performance compared to existing models in two evidence retrieval processes. Furthermore, integrated with an entity-oriented medicine database, our RagPULSE presents impressive results in responding to inquiries in medication consultation. We hope our work can motivate further innovation in applying LLMs in the medical domain.

## Appendix A Appendix

### A.1 Details of Elo

The Elo rating system, devised by Arpad Elo, is a methodical framework used to calculate the relative skill levels of players in competitor-versus-competitor games. Initially conceived for chess, the Elo system has found widespread application across various sports and games to gauge individual or team performance. The fundamental principle of the Elo system is to assign a numerical rating to each player, which adjusts based on match outcomes against other rated players. The adjustment in ratings is predicated on the difference between the actual and expected match outcomes, allowing for a dynamic representation of a player’s skill level over time.

The core of the Elo rating system is encapsulated by the formula used to update player ratings post-match. The expected score for a player, $E_{A}$, against an opponent, is calculated as:

$$ $E_{A}=\frac{1}{1+10^{\left(R_{B}-R_{A}\right)/400}}$ $$

where $R_{A}$ and $R_{B}$ are the current ratings of the player and the opponent, respectively. Following the completion of a match, the actual score $\left(S_{A}\right)-1$ for a win, 0.5 for a draw, and 0 for a loss -is compared against the expected score to update the player’s rating:

$$ $R_{A}^{\prime}=R_{A}+K\left(S_{A}-E_{A}\right)$ $$

In this formula, $R_{A}^{\prime}$ represents the new rating of the player, and $K$ is a factor that determines the maximum possible adjustment per game. This factor can vary depending on the level of competition and the governing body’s regulations, allowing for flexibility in the sensitivity of rating adjustments to match outcomes. The Elo system’s adaptability and simplicity have contributed to its enduring popularity and applicability across different competitive disciplines.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Typo? The manuscript mentioned "Table 3.2," but this is not Table 3.2; I guess it should be Table 1.
Question: Will the medicine database be released to the public?
soundness: 2
strengths: This paper proposed a new dataset for medication consultation.
summary: This paper built a multiple-round dialogue benchmark MedicineQA and proposed a distill-retrieve-read framework for RAG improved medication consultation capability of LLM. The dataset size is limited, and the multiple-round dialogues were generated by GPT-4 but not from real-world dialogue; this may affect the value of the application for the proposed framework. More details about the dataset should be introduced, such as the number of dialogue rounds.  It seems the dialogue distilling tool was trained on the general domain synthetic dataset but not for the medical domain, and the details of the dialogue distill tool training are not clear (which base model? data size? training settings? list of data sources). The proposed method lacks novelty; it applies the dialogue distilling module in the RAG setting and tests on the small MedicineQA dataset. Experiments are only based on a single dataset.
Typo? The manuscript mentioned "Table 3.2," but this is not Table 3.2; I guess it should be Table 1.
Question: Will the medicine database be released to the public?
weaknesses: The dataset size is limited, and the multiple-round dialogues were generated by GPT-4 but not from real-world dialogue; this may affect the value of the application for the proposed framework. More details about the dataset should be introduced, such as the number of dialogue rounds.  It seems the dialogue distilling tool was trained on the general domain synthetic dataset but not for the medical domain, and the details of the dialogue distill tool training are not clear (which base model? data size? training settings? list of data sources). The proposed method lacks novelty; it applies the dialogue distilling module in the RAG setting and tests on the small MedicineQA dataset.  Experiments are only based on a single dataset.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1) What is the use of the question annotations in MedicineQA? If I understand correctly, it is the explicit reformulation of the questions from the conversation history (which might be ambiguous and contain superfluous information, as shown in "Dialogue History" in Figure 3). I understand that the synthetic dataset collected and described in lines 257-264 is used to convert questions into keywords, but during inference, what is actually used as input? Is it the dialogue history, of the extracted questions. If the latter, how are they extracted from dialogue history?

2) Could you please indicate how you measure the H@K for "baseline models". From reading lines 300-306, It is not clear to me what the baselines are actually doing. Are they generating the medication names from their parameters? Are they generating queries that are run against the database? 

3) Furthermore, could you please elaborate on what you mean by "we immediately adopt the attached corresponding medicine information as the context to guide the generation [...]" in line 323. Is it the ground-truth medicine annotations from MedicineQA? If so, I find this result highly counter-intuitive and contradictive, as the paper states later, in lines 389 ff, that the improvements of RagPULSE are due to the proposed RAG framework rather than due to domain fine-tuning of the PULSE LLM. But since RagPULSE's retrieval is far from optimal (e.g. H@k rates in Tables 2 and 3), it would suggest that imperfect retrieval somehow leads to better generations. Similarly, could you please indicate what "instruction used for baseline models" in the description of Table 3 means: "We also prompt RagPULSE by the instruction used for baseline models, which are denoted as †".
soundness: 2
strengths: The collected resources seem helpful and, even if collected automatically, are validated by domain experts. The proposed approach seems to improve over LLM-based baselines.

The experiments seem sound in what they report/investigate.

The overall writing quality is good.
summary: This paper introduces a benchmark, database and method for describing medications in dialogues with LLMs. The benchmark is collected automatically and reviewed by medical experts and the database is curated to support retrieval of medication information. The paper evaluates the proposed approach of converting conversations with their histories into a set of keywords to query the medical database and the results indicate favourable outcomes, based on GPT-4- and human-based evaluations of the generations.
weaknesses: My main concerns is the lack of meaningful comparison and under-specification of some of the details.

- While the paper suggests that the proposed distill-then-retrieve framework is better than RAG methods, I did not find meaningful comparisons with either general RAG frameworks as e.g. discussed in lines 117-126, or with domain-specific RAG approaches, such as e.g. BioReader (https://aclanthology.org/2022.emnlp-main.390.pdf) RAFT (https://arxiv.org/abs/2403.10131).

- The experimental setting is somewhat under-specified, see my questions to the authors. In summary, I don't understand the exact settings for the comparisons in Tables 2 and 3.

- There is very little detail on the construction of the medical database, which seems to be the key contribution enabling the retrieve-then-distill framework. Particularly, data sources, data quality measures etc are missing.

I gravitate towards rejection but if my queries are answered with sufficient detail, I am open to re-evaluating my decision.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. How ragpulse deals with retrieval errors (if any)? Also, did you see any failure modes commonly found and can you discuss them, please?

2. The benchmark is available in English and Chinese as per the paper. but how well do we know if MedicineQA can be adapted to other languages (maybe low-resource languages)?
soundness: 4
strengths: This paper shows an approach to enhancing LLMs in the medical domain by reformulating rag for multi-round medication consultations. Also, using a tool-calling mechanism for search query formulation is a novel method of tailoring retrieval processes in dialogue-based tasks in healthcare. The paper shows good quality in methodology. Also, the authors did around 600 experiments to validate the effectiveness of ragpulse and provided comparisons with existing open-source and commercial models. The paper has clear objectives, methods, and findings sections written. The study addresses a research gap in applying LLMs to the medical field and introduced MedicineQA, ragpulse. Their work shows good retrieval and response generation capabilities.
summary: The paper has introduced a benchmark called 'MedicineQA' to evaluate llms in medical consultation. Their suggested benchmark is focusing on rag for multi-turn dialogues in medication consultation. The authors also propose RagPULSE framework to deal with limitations of standard RAG frameworks. This framework utilizes a tool-calling mechanism to enhance retrieval accuracy by summarizing dialogue history into search queries and retrieving evidence from a medicine database.
weaknesses: I feel that the number of multi-round dialogues (300) are very less and that makes it difficult in terms of generalizing the results. So, maybe increase the dataset size by expanding the set of topics like for various specific conditions like breast cancer, diabetes, chronic pain.

Another thing is that the study should discuss retrieval errors for the cases where the model fails to retrieve accurate responses. The authors can try to include 'error analysis' section (maybe) that show various retrieval failures types or examples. This analysis will certainly go a long way in understanding if any adjustments should be made in distillation process.