PlugMed: Improving Specificity in Patient-Centered Medical Dialogue Generation using In-Context Learning
Authors: Chengfeng Dou
Zhi Jin
Wenpin Jiao
Haiyan Zhao
Yongqiang Zhao
Zhengwei Tao
Venue: EMNLP 2023 Findings (2023)
Keywords: Medical
Dialogue Generation
In-context Learning
TL;DR: Using ICL to Enhance Large Language Models for Medical Conversation

Abstract: The patient-centered medical dialogue systems strive to offer diagnostic interpretation services to users who are less knowledgeable about medical knowledge, through emphasizing the importance of providing responses specific to the patients. It is difficult for the large language models (LLMs) to guarantee the specificity of responses in spite of its promising performance even in some tasks in medical field.
Inspired by in-context learning, we propose PlugMed, a Plug-and-Play Medical Dialogue System, for addressing this challenge. PlugMed is equipped with two modules, the prompt generation (PG) module and the response ranking (RR) module, to enhances LLMs' dialogue strategies for improving the specificity of the dialogue. The PG module is designed to stimulate the imitative ability of LLMs by providing them with real dialogues from similar patients as prompts.  The RR module incorporates fine-tuned small model as response filter to enable the selection of appropriate responses generated by LLMs. Furthermore, we introduce a new evaluation method based on matching both user's intent and high-frequency medical term to effectively assess the specificity of the responses. We conduct experimental evaluations on three medical dialogue datasets, and the results, including both automatic and human evaluation, demonstrate the effectiveness of our approach.

PlugMed: Improving Specificity in Patient-Centered Medical Dialogue Generation using In-Context Learning
Chengfeng Dou, Zhi Jines, Wenpin Jiaos,, Haiyan Zhao, Yongqiang Zhao, Zhenwei Tao
School of Computer Science, Peking University; Key Laboratory of High Confidence Software Technologies(PKUU) MOE, China zhi zhhy sei Jēpku edu.cn cn (tttzw, www.gyoggggggggnooooooooooooooo pku. edu.cm cn
Abstract
dh d'akk cxpty
my uomuch caiilisw.
The patient-centered medical dialogue systems strive to offer diagnostic interpretation services to users who are less knowledgeable about med- ical knowledge, through emphasizing the im- portance of providing responses specific to the patients. It is difficult for the large language models (LLMs) guarantee the specificity of responses in spite of its promising performance even in some tasks in medical field. Inspired by in-context learning, we propose PlugMed, Houg-and-Play Medical Dialogue System, for addressing the challenge. PlugMed equipped with prompt generation (PG) module and a response ranking (RR) module to enhances LLMs' dialogue strategies for improving the specificity of the responses. The PG module is used to stimulate the imitative ability of L.IMs by providing them with real dialogues from similar patients as prompts. The RR module in- corporates fine-tuned small model as response filter to enable the selection of appropriate re- sponses generated by LLMs. Furthermore, we introduce new evaluation method based on matching both user's intent and high-frequency medical term to effectively assess the speci- ficity of the responses. We conduct experi- mental evaluations on three medical dialogue datasets, and the results, including both auto- matic and human evaluation, demonstrate the effectiveness of our approach.
Porver
It Wacke S SI S
Jas. 12 is (A R2 Wa Drinking looey engry skoreds
ofter hulf en hov, whschore redoce the intirtion to streach
LIM
6M7, 22 TI Héb BUE 10 WAME
Fulls lung leve you bore aisutionn Vooo you over bad courads
Figure 1:
An example of the medical dialogue. The LLM and the doctor adopt different response strategies for the same question.
many of which integrate general medical knowl- edge from knowledge bases into the models by utilizing knowledge injection mechanism. As the scale of large language models (LLMs) continues to expand, the lack of knowledge being alleviated. This is evident in the performance of LLMs such as Instruct GPT (Ouyang et al., 2022), which outper- form small models based on fine-tuning (Singhal et al., 2022) on medical question answering tasks without any additional training.
However, LLMs have been criticized for lacking medical diagnostic logic to the patient, which can lead to untargeted and even risky response sugges- tions (Howard et al., 2023; Zhang et al., 2023). For instance, as illustrated in Figure (1), when patient claims have a particular ailment and seeks med- ication, the doctor's priority is to first investigate the underlying disease to make targeted sugges- tions. In contrast, LLMs are more inclined to give straightforward medical advice, rather than further gathering patient information to give accurate ad- vice. Once LLMs give overconfident advice, it is difficult for patients to discern the effectiveness and safety of such advice.
1 Introduction
As key lask in health conversational assistants, Medical Dialogue Generation aims to automati- cally generate informative responses to the users. It's better for such dialogues to be medical knowl- edgeable, patient-specific and context-aware, as the patients may be less knowledgeable about medical knowledge and their dialogue contexts may vary. Prior studies (Li et al., 2021a; Varshney al., 2023b,a) have emphasized the critical role of do- main knowledge in medical dialogue generation,
Researches have demonstrated that in-context

---

learning (ICL) possesses the ability to impact the LLMs' conversational style and mitigate prejudice and toxicity concerns (Roy et al., 2023; Meade et al., 2023) by demonstrating a few examples. This phenomenon is due to the powerful imitation learn- ing and few-shot capabilities of LLMs, by learning patterns from a small number of samples and ap- plying them in generation (Dong al., 2022).
An ICL-based approach that enhances LLMs generale responses that conform the di- agnostic strategy.
The comprehensive evaluation metrics for medical dialogue automation that take both the intent accuracy and the high-frequency medical term accuracy into account.
Taking inspiration from these studies, we pro- pose leveraging ICL to shape the LLMs' dialogue strategies and accordingly design a Plug-and and-Play Medical Dialogue System, named PlugMed, which embodies two crucial components: a Prompt Gen- eration (PG) Module and a Response Ranking (RR) Module. Specifically, PlugMed uses the PG mod- ule identify examples by considering informa- tion from both global and local views. From global view, the PG module choose relevant examples for ensuring that the model acquires a comprehensive understanding of the entire dialogue process by exploiting the similarity with the entire dialogue history. Conversely, from local view, the PG mod- ule priorities recent utterances to capture the most relevant information for generating responses. To further maximize advantages of both the global and local views, PlugMed uses the RR module to au- tonomously select the most appropriate response for the ongoing dialogue through utilizing a fine- tuned small model.
Experiments demonstrating the key elements of automated medical diagnosis.
Methodology
2.1 Overview
We propose a framework as shown in Figure (2), which employs ICL to guide the LLM towards generating high-quality replies, where the Prompt Generation (PG) Module and the Response Rank- ing (RR) Module are two key components. The PG Module accepts the dialogue history as input and outputs multiple In-context prompts, along with single Instruct prompt. Based on these prompts, the LLM generate multiple system sponses. Then, the RR Module utilizes small language model (SLM) select the best response. We will elaborate these two components in the fol- lowing subscctions.
2.2 Prompt Generation Module
The PG module uses a multi-strategy retrieval framework to retrieve examples similar to the input sample and then employ them to generate prompts.
Another critical consideration is about appropri- ate automatic evaluation metrics for medical dia- logue systems. Previous studies (Varshney al., 2023b; Zhang et al., 2023) only relied on open- domain dialogue evaluation methods. However, as indicated (Ji et al., 2023), these evaluation methods may be unreliable in task-oriented scenar- ios. To gain a comprehensive understanding of the system's 's real-world performance, we undertake a thorough evaluation that is twofold: the intent accu- racy and the high-frequency medical term accuracy. Here, the intent accuracy is used to evaluate the rea- sonableness of the dialogue actions adopted by the system, and the high-frequency medical term accu- racy focuses on measuring the presence of essential medical information in the system's 's responses.
2.2.1 Basic Ideas
As shown in 'Dialogue History' at the left part of Figure (2), the main idea is as follows. Firstly, from the global view, we considers the entire his- tory of the dialogue, and the global retriever re- trieves the similar dialogues from the the training set of the dataset. Secondly, considering the global view is susceptible to distractions caused by the abundance of irrelevant information, which may lead to inappropriate retrieved examples, we in- clude the local view for enhancing the relevance of the retrieval. Concretely, the local retriever first extracts the patient's 's symptom information from past conversations, serving the initial filter for the samples. Considering the recent rounds of con- versations hold the utmost relevance the system- generated responses, we then utilize these conversa- tions as query to select examples. This effectively mitigate the interference of irrelevant information.
We evaluate approach three widely used large medical datasets. i.e., Meddg, MedDialogue and Kamed datasets. Both automatic and human evaluations show that our approach can substan- tially improve the specificity of LLMs. Our contri- butions can be summarized as follows:

---

Dedaaaal Ehaor
Aysiom: here eocette for taa dacas, but ca conrstied, you tike cas Eir soree ITS.
Preqt Gonaraes Modal
Glubal Firr rFudl cuotesd
Potlerr It tat pastove dhe ati-a0c asibodic does that mein Astuakty tereetrrrnneee zmaic yeur selluolies
Pronept Uimnnat:
prokiee
boondedl pauns
Hoppae Santing Nocali
sealr?
Kanker
Bester No, doart affer paral.
Senvil
Diandrr
Glbba Senove
Lane live offe intrrnedd Bilkeger phegs fin dordeee dky Riigenh syndenee
Fill Geseli
Pasivrr Da,, ase tase rigrnaa courcs belore a biood ton affict the rera??
Genertal
fallen Of, bot Doc snaa flenc tie divavve
LIM
Claik
(PanDKIO Bead
Saxple
The overview of MlugMed. Our system consists of two core components, i... Prompt Generation (PG) Module, which retrieves similar examples in the dialogue history from both global and local views to generate prompts, and Response Ranking (RR) Module, which ranks the outputs LLM corresponding to these prompts and selects the best responses.
Figure 2:
Thirdly, we employ the retrieved examples from both views to take advantage of their respective strengths.
'Instruct prompt' and 'In-context prompts'. Each In-context prompt corresponds to a distinct cxam- ple retrieval strategy.
It is needed to compress examples include more demonstrations, given the input length con- straint imposed by the LLM, when generating the in-context prompts. Drawing inspiration from Hu et al. (2022), for each example, we keep only the most recent rounds of conversations, and we restrict the maximum conversation length to no more than n. Moreover, employ the previous mentioned chief complaints (up m characters long) the dialogue abstract, replacing the excluded history achieve dialogue compression.
2.2.2 Implementation
Global Retriever. The global retriever utilizes the full context of dialogue history as a query for searching samples. It employs Sentence- -ertt (SBERT) (Reimers and Gurevych, 2019) to encode the query and examples, and then utilizes cosine similarity to identify the closest examples.
Local Retriever. The local retriever retrieves samples in terms of symptoms and recent utter- ances. For getting symptoms, we develop a medical dialogue summary model that utilizes the ICMS- MRG (Chen et al., 2022) dataset in conjunction with BART (Lewis et al., 2019) as the backbone model. This model enables the extraction of the chief complaint, containing the patient symptom information, and we use SBERT as the encoder of chief complaints to provide embeddings for the fol- lowing operations. We first encode the chief com- plaints of examples and use the K-Means algorithm to divide them into IC groups. Then, we extract the embeddings each cluster centers to serve as the symptom index for querying. When performing the search, we first retrieve the candidate exam- ples by computing the cosine similarity between the embeddings of the sample's chief complaint SBERT to
The Instruct prompt includes the full context, and we include the instruction before the history to prompt LLM to act as doctor. We this prompt to activate the zero-shot capability of LLMs. Appendix E gives some examples of both prompts.
2.3 Response Ranking Module
Through the experimentation, observe that LLMs tends to generate significant number of medical terms during dialogue, resulting in more comprehensive responses. However, LLMs often exhibit overconfidence, as they question patients less frequently, resulting in a decrease in the overall quality of responses. On the contrary, small lan- guage models (SLMs) that are fine-tuned on dia- logue corpus behave cautiously and tend to include limited number of medical terms in their output. This phenomenon is illustrated in Figure (3). Ac- knowledging the complementary nature of LLMSs and SLMs, we propose using SLM to evaluate the responses of the LLM.
and the symptom index. Then, retrieve examples from the candidates based on recent dialogue utterances.
The prompt generator gener- ates two types of prompts, as depicted in Figure (2):
Prompt Generator.

---

3.1 Intent Evaluation
The intent accuracy (Int) is used to assess the con- formity of the system's response to the ground-truth in terms of dialogue actions. We train medical dialogue intent classifier for calculating Int, and Int is calculated by the following formula:
Int E f(Pred,, Golden;)
(2)
QUARY
SLOOM
BAHT
Here, N represents the total number of samples. Pred, and Golden; denote the model's predicted response and the corresponding actual response, re- spectively. The function f(---) evaluates the inten- tions, which are extracted by the aforementioned intent classifier, assigning value of 1 if the in- tentions of the two responses are the same, and value of otherwise. Appendix A.1 presents the implementation details of the intent classifier.
Recuutt
Figure 3: We compared the behavior patterns of LLMs and SIMM, using Bloom (Scao et al., 2022) as a rep- resentative of LLMs and Bart (Lewis et al., 2019) as representative of SL.Ss. We counted the distribution dialogue actions taken by them on the validation set of KaMed (Li al., 2021b), where the human action is denoted as Golden. The Request action indicates the collection patient information, while the Inform ac- tion indicates the provision of advice the patient.
3.2 Medical Term Evaluation
We evaluate the completeness and correctness of the responses using the micro-fl score, which mea- sures the overlap of high-frequency medical terms between the ground need to avoid relying solely on exact matching when measuring the term overlap. For instance, dif- ferent doctors may prescribe different medications for some diseases that may have same effect the patient. Employing exact matching alone may result an underestimation of the performance exhibited by models that possess the ability to gen- erate diverse treatment options.
Specifically, for a given sample with dialogue history h, we use perplexity as the score of the response r generated by the LLM. We compute the score using the following equation:
and the prediction. We
s(r) = - Σ logp(r; res, 0)
(1)
Here, / represents the length of response r, and denotes the model parameter of the SLM. This eval- uation model uses an encoder-decoder architecture, where h is input to the encoder side of the model, and rei is input to the decoder side for calculating the generation probabilities. We select the response with the lowest score as the system output.
Hence, we introduce a novel approach called Top-n Match (TnM) to address the problem. Con- cretely, let T = {f, f2, .... (TT be set consist- ing of [T'l terms, and let be similarity score function that satisfies 0 < s(4,, 4)) I for all 4, T. For given term 4 T. define ST (s, 42) C as the set of n terms that are clos- est to 4 based on the similarity function s. We say that ty and tj are Top-N Match if and only if ST (s, 4) n ST (s, 44) + 0. TnM with different n can represent the term-mateen matching scores for differ- ent similarity levels. We present the fl fl-score results using T3M configurations. Appendix A.2 and A.3 will give more details about the term extraction and matching.
3 Automatic Evaluation Metrics
We observe that previous studies (Varshney al.. 2023b,a: Zeng et al., 2020) employ the metries for open domain dialogue tasks that often cannot effec- tively measure system performance task-oriented settings (Ji et al., 2023; Risch et al., 2021). From the results given in Figure (3), we observe that the LLM exhibits overconfidence. To judge whether approach alleviates the problem, becomes crucial evaluate the dialogue actions taken by the LLM. is equally important evaluate the dialogue content generated by L.M.. Hence, we introduce two metrics that consider both intent and the usage of high-frreuency medical terms.
Experiments
This section delincates the evaluation setup and the results of the proposed approach in the context

---

Dataset
Train / Valid / Test
Turn
complaint to retrieve examples with similar symp- toms. From these examples, a number of samples are randomly selected to generate the prompt. This strategy corresponds to the initial step of the local retriever's ' two-step search. 4) Local Secondary: This strategy searches for examples through a local retriever and utilizes full two-step search. All examples are from the training set of the dataset.
Meddg
14,864 / 2,000 / 1,000
9.92
MedDialog
32,723 / 3,000 / 3,000
4.76
KaMed
57,754 / 3,000 / 3,000
11.62
Table 1: The statistics of datasets, where Turn indicates the average number of rounds contained in each dia- logue session contains.
Limited by input length, the In-context prompt contains 4 examples, while each example is limited to the last 140 (m = 20, n = 120) tokens. When constructing the symptom index using the K-Means algorithm, we set the number of cluster centers to 100 and iterate 20 times. During response ranking, BART serves as the scoring model. We conduct the experiments using PyTorch² and Huggingface Inference API
medical dialogue generation.
4.1 Datasets and Evaluation Metrics
We conduct experiments on three large datasets for our evaluation. 1) The Meddg dataset (Liu et al., 2022) which is collected from Doctor Chunyu! and consists of 17,864 dialogue sessions. 2) The MedDialogue-Ch dataset (Zeng ct al., 2020) which is collected from HaoDaifiu and comprises 38,723 dialogues without any provided annotations. 3) The KaMed dataset (Li et al., 2021b) which is also collected from Doctor Chunyu, but larger scale, containing 63,754 dialogue sessions. Statis- tics of three datasets are presented in Table (1).
4.3 Baselines
Fine-tuning Based Baselines. These baselines utilize small language models as the backbone, trained on the aforementioned datasets for the med- ical dialogue generation task, which include the following models: 1) Bart (Lewis et al., 2019), well-known encoder-decoder model that is rec- ognized for its text generation capabilities. 2) Mars (Sun et al., 2022), an advanced model ex- plicitly crafted for the MultiWoZ (Budzianowski et al., 2018) dataset, renowned for its exceptional performance addressing the lass-Oriented Dia- logue (TOD) task. We migrated Mars into our tasks and trained it to focus on the generation of medical terms. Appendix B gives the further information on the migration process.
To evaluate the quality of generation, we employ the Rouge-L (Lin, 2004) and Bert-Score (Zhang et al., 2019) to measure the overall similarity be- tween the generated text and ground-truth We also utilize micro-Fl scores for entity matching in T3M settings to assess medical term correctness based on the aforementioned definitions. Additionally, we employ the INT metric to measure the accuracy of the intended responses. The validity analysis of each metric can be found in the Section 4.8.
4.2 Implementations
These baselines employ large language models to generate dialogue responses, Our comparison targets include: 1) Bloom (Scao et al., 2022), widely used open-source multilin- gual language model with 176 billion parameters. 2) Bloomz (Scao et al., 2022), instruction-tuned model derived from Bloom and specializes in zero- shot tasks. In addition to these models that use Instruct-prone prompt as input, we also include two base- lines that utilize In-context prompts as input: 1) ICL Rand, which selects a set of dialogue exam- ples randomly to construct the prompt. 2) ICL Sbert, which utilizes Sbert (Reimers and Gurevych, 2019) to encode the dialogue history and find the
LLM Baselines.
Our approach employs BLOOM as the founda- tion model, and ensure reproducibility, we avoid any form of sampling and instead utilize a greedy
decoding strategy. We generate a set of four prompts for a given sample, including an Instruc- tion prompt (referred to as "Vanilla") and three In-context prompts. These prompts are as follows: 1) Vanilla: This strategy instructs the model to act doctor by prefixing the samples with in- struction. 2) Global View: This strategy involves looking for examples through a global retriever, us- ing full context as query. 3) Local Primary: In this strategy, we first consider the patient's 's chief
hhtps:/wwww.chunyyysseeee.. hhtps:/hhoodf.com
'httss//ppytorch.org/ hhtn:///wugnnfaascccooocconnnmme

---

Meddg
MedDialogue
KaMed
Model
R-L
B-S
T3M
INT
R-L
B-S
T3M
INT
R-L
B-S
T3M
INT
Fine-tume Fine-tuning based
Bart
24.1
64.8
9.9
42.0
11.8
59.0
16.2
42.0
15.9
60.1
15.5
45.0
Mars
20.9
63.1
10.3
39.9
10.5
58.4
16.8
37.7
12.7
59.3
16.4
41.4
LLM based
Bloom
14.2
60.3
10.9
26.3
10.7
57.2
16.2
33.0
11.3
58.7
16.9
34.1
Bloomz
14.5
58.9
8.5
24.8
10.0
57.9
12.4
31.9
9.7
57.8
15.0
32.0
ICL Rand
16.4
61.2
10.6
29.6
9.9
57.3
14.8
36.6
9.3
56.9
15.0
36.1
ICL Sbert
18.7
62.7
11.3
36.3
11.9
60.0
17.7
37.3
12.0
59.5
16.8
37.9
(lug
21.1
64.1
12.1
41.3
12.8
60.4
18.4
37.9
14.1
60.1
18.1
41.7
Table Automatic evaluation on the Meddg, MedDialog and KaMed datasets. R-L B-S denote Rouge-L and
Bert-Score, respectively. Boldface scores indicate best results. The performance improvement of PlugMed over the baselines is significant with p < 0.05.
closest examples to the given sample. Both base- lines Bloom the fundamental model.
hancement in the quality of system responses due the integration of the RR module. This observa- tion underscores the efficacy of the fine-tuned SLM in effectively evaluating the LLM's output.
4.4 Overall Performance
Table (2) present the automatic evaluation results on Meddg, MedDialog, and KaMed. Remarkably, PlugMed consistently attains the top-ranking posi- tions across majority of the metries and achieves best performance in T3M, outperforming even the strongest baseline. Meanwhile, among all baselines leveraging the LLM, PlugMed generates responses with the most reasonable intent.
Furthermore, observe that Global View demonstrates comparable performance to PlugMed when assessed using the MedDialogue dataset. However, PlugMed significantly surpasses Global View on the Meddg and KaMed datasets. We attribute this discrepancy to the MedDialogue dataset's 's characteristic of having shorter dialog his- tories, averaging only 4.76 rounds. Conversely, Global View's retrieval effectiveness diminishes the samples' length increases, elaborated in Sec- tion 2.2.1, making it less effective than PlugMed the other two datasets. This underscores the syner- gistic relationship between Global View and Local View, highlighting their complementary strengths.
Our analysis uncovers interesting observations. Firstly, discovered that fine-tuned small mod- els tend have higher INT scores but lower term matching-related scores. This suggests that while these models excel in emulating the dialogue ac- tions of doctors, they often struggle to generate ap- propriate medical terminology due to their limited medical knowledge. Secondly, we observed that Bloomz performed worse than Bloom, indicating that the instruct-tuning process may compromise the diagnostic capability of the model. More details can be found in Section 4.6.
To delve deeper into the contributions of these strategies to the outcomes, we utilize the RR mod- ule to rank the responses corresponding to each strategy, documenting the percentage of times each strategy achieves the top rank. The results are illus- trated in Figure (4). Based on the findings depicted in the figure, we can infer that the responses gen- crated by the Vanilla strategy exhibit significantly inferior quality compared to those produced by the other three strategies. Furthermore, the probability of being selected by the remaining three strategies is approximately equal, indicating their comple- mentary nature.
4.5 Ablation Study
To investigate the influence of different prompt generation strategies system responses and the efficacy of the RR module, we conduct ablation experiments. The experimental results, shown in Table (3), indicate that both the global and local view effectively enhance the quality of the LLM output. Moreover, we observe a substantial en-

---

Bloom
ChatGPT
Dataset
Settings
R-L
B-S
T3M
INT
R-L
B-S
T3M
INT
Vanilla
14.2
60.3
10.9
26.3
6.7
56.1
10.5
23.4
Global View
18.7
62.7
11.3
36.3
12.3
60.0
12.7
33.8
Meddg
Local Primary
18.7
62.5
10.7
34.6
11.6
56.1
12.3
30.7
Local Secondary
18.0
62.3
11.2
34.9
11.7
59.7
12.1
30.3
+Ranking (Ours)
21.1
64.1
12.1
41.3
12.9
60.4
12.9
34.4
Vanilla
10.7
57.2
16.2
33.0
7.5
56.5
16.5
40.5
Global View
11.9
60.0
17.7
37.3
10.1
58.6
17.8
38.8
MedDialogue
Local Primary
11.0
59.0
15.5
36.6
10.0
58.5
18.3
38.2
Local Secondary
12.0
60.0
16.7
36.1
9.9
58.4
18.0
39.1
+Ranking (Ours)
12.8
60.4
18.4
37.9
10.1
58.6
18.5
40.2
Vanilla
11.3
58.7
16.9
34.1
6.1
54.9
14.9
36.2
Global View
12.0
59.5
16.8
37.9
8.9
57.5
17.1
39.8
KaMed
Local Primary
12.0
59.5
17.1
39.4
8.6
57.2
17.0
38.9
Local Secondary
12.1
59.4
17.7
36.7
8.4
57.2
17.1
38.7
+Ranking (Ours)
14.1
60.1
18.1
41.7
9.1
57.6
17.1
40.5
Table Ablation studies on Meddg, MedDialog and KaMed datasets. Vanilla, Global view, Local primary and Local secondary represent the four prompt generation strategies respectively, and +Ranking represents the results of the selection based on the four strategies using the RR module.
Bloom. This discrepancy can be attributed the safety protocols integrated into ChatGPT by Ope- nAI. Specifically, ChatGPT often advises patients to seek professional assistance instead of providing diagnoses, which affects its multi-round conversa- tion potential. Conversely, we noticed that our ap- proach has minimal impact on Bloomz, as detailed in Appendix C. This disparity be attributed to Bloomz's 's training dataset, bigscience/xP3 5, which primarily comprises single-round Q&A tasks, mak- ing less adaptable to multi-round conversations. To summarize, the generalization ability of our method is influenced by the model's pre-training task, and improving our method's 's effectiveness en- tails pre-training models based on multi-round dia- logues and reducing safety interventions.
22
10
Middd
VedDelggu
Gooal
Figure 4: The evaluation results of the RR module for four strategies. The vertical axis represents the propor- tion of the strategy get the best score. Local P and Local S correspond to Local Primary and Local Secondary, respectively.
4.7 Human Evaluation
4.6 Analysis of Generalization Capacity
We manually evaluate seven selected dialogue mod- els conduct comprehensive comparison. We ensure a thorough analysis by randomly selecting 100 samples from cach dataset. Each sample is examined by three physicians who assign scores based on the criteria outlined in Table (4). The eval- uators check the responses in the following order: Role Consistency, Empathy, Correctness, Neces-
We conducted experiments using ChatGPT and Bloomz to assess the effectiveness of our approach in terms of generalization. The results, as presented in Table (3) for ChatGPT, revealed that our ap- proach successfully enhances ChatGPT's perfor- mance in completing medical conversations. How- ever, it was observed that ChatGPT's capability for multi-round conversations lags behind that of

---

Check Item
Content
Role Consistency
The response should exhibits doctor-like response style and utilizes natural language.
Empathy
The response should demonstrate understanding of the user's intents and employ friendly tone. adhere
Correctness
The responses
medical and be evidence-bused.
Necessity
The response should provide assistance advancing diagnosis satisfying the patient's curiosity.
Richness
The responses should incorporate additional relevant medical information to significantly enhance the patient experience.
Table 4; Check items for human evaluation.
Model
Human
BLEU
R-L
B-S
TIM
T3M
TSM
INT
Golden
90.4
Bart
58.9
4.1
22.1
63.0
8.3
11.6
12.3
40.9
Mars
44.6
5.4
19.2
62.3
9.1
12.8
13.9
33.2
Bloom
46.7
3.8
13.8
59.6
7.4
8.8
10.9
24.0
Bloomz
49.8
3.3
13.7
59.5
3.6
5.8
7.3
28.3
Bloom-Rand
52.9
3.9
15.4
61.0
7.3
13.2
14.8
30.1
Bloom-Sber
57.5
4.3
16.5
61.9
9.9
14.0
16.2
33.1
PlugMed
61.0
4.1
19.3
62.8
8.2
13.0
14.6
37.3
Pearson (6)
-22.7
48.9
59.7
27.1
42.8
41.2
71.3
Table We evaluated three datasets human ratings, For evaluation purposes, we randomly selected 100 samples from each dataset. The average score for each sample was calculated based on the assessments of three physicians.
sity, and Richness. This order also corresponds to the importance of the check items, and subse- quent items only had test significance the model satisfied the previous check item. Therefore, we apply the following rules for scoring. If a target response successfully pass a particular test, it re- ceives score of 20. However, if the target response does not pass specific test, it assigned score of for all subsequent check items.
coefficient quantify their alignment with human evaluations. A score approaching 1 indicates a stronger metric performance. The corresponding outcomes can be found in Table (5).
Our examination indicates that the INT met- ric surpasses all others in performance, with Bert- Score coming in close second. These two metrics exhibit stronger correlations with human assess- ment, indicating the superiority of semantically based evaluation. Simultaneously. this outcome implies that the model's intent should align closely with the corpus. Lower scores for BLEU (Chen and Cherry, 2014) and higher scores for Rouge-L suggest preference among individuals for more comprehensive model responses over precision- oriented ones. Lastly, based on the outcomes of TIM (exact match), T3M, and T5M, it becomes ident that factoring term similarity is imperative when calculating term matching scores.
We compute the average score of each model on the dataset and summarize the evaluation results in Table (5). Our analysis reveals that PlugMed exhibits the highest performance based on human evaluations. However, there still exists signifi- cant gap between the responses generated by the models and the human responses, indicating that the models have not fully grasped the diagnostic capabilities. Additionally, we observe that BART performs well in the manual evaluation, primarily because we give very low priority to richness. Appendix D provides case study for illustrating this finding.
Related Work
5.1 Medical Dialogue Systems
According to system architecture, medical dialogue systems can be of two types: the pipeline and the end-to-end systems (Valizadeh and Parde, 2022). The pipeline systems usually involve four steps: the natural language understanding, the dialogue state
4.8 Analysis of Evaluation Metrics
Within this section, we have included an evaluation the metric's 's reliability. To gauge the quality of these metrics, we employed the Pearson correlation

---

tracking, the dialogue action generation, and the natural language generation. Wei al. (2018); Xia et al. (2020) propose learn the dialogue policies for automated diagnosis using reinforcement learn- ing. other studies, Lin et al. (2019) proposes to model the associations between symptoms by constructing a symptom graph, aiming to enhance symptom diagnosis performance. Li et al. (2021a) proposes to symptoms and diseases keys to generate responses instead of dialogue states and actions based on knowledge graph.
on either example retrieval or dialogue compres- sion. In contrast, our work combines the two tech- niques for integrating multiple retrieval strategies.
Conclusion
In this paper, we use in-context learning to develop a patient-centered medical dialogue model. To this end, we introduce a Prompt Generation module capable of generating LLM input from both global and local views. Additionally, we construct a Re- sponse Ranking module using supervised trained small model to filter the LLM output. Experimen- tal results indicate that the responses generated by PlugMed exhibit greater inclusion of compre- hensive medical terms and Plug.edd yields more accurate dialogue intents than other large language model based methods.
The end-to-end models, usually a sequence-to- sequence architecture (Sutskever et al., 2014), first attracted attention. The fine-tuning pre-trained models, such as GPT-2 (Radford al., 2019), have been proven effective task-based dialogue scenar- ios, as demonstrated by Su et al. (2021); Yang et al. (2021); Wang et al. (2022). BioBERT (Lee et al., 2020) and BioGPT (Luo et al., 2022) try to improve the performance by employing pre-training on med- ical corpora. Varshney et al. (2023b,a); Tang et al. (2023) propose explicitly using the Unified Medical Language System (UMLS) to incorporate knowl- edge the dialogue generation process.
Limitations
Based on human evaluation, we have identified shortcomings in PlugMed's diagnostic efficiency. This suggests that Plug has difficulty rapidly identifying the patient's 's disease, which may lead to increase in average conversational discourse and harm the patient's experience. Our future work will prioritize addressing this issue.
In summary, existing studies mainly emphasize the role of knowledge enhancement based on small models. In contrast, our work is conducted on knowledge-rich LLMs, emphasizing the enhance- ment of dialogue strategies.
7 Acknowledgement
Our work is supported by the National Key Re- search and Development Program of China (Project Number: 2020AAA0109400), We kindly appre- ciate all the researchers who provide valuable in- sights, discussions, and comments on this work.
5.2
ICL for Dialogue
As LLMs continue to advance, ICL has emerged paradigm in natural language processing. The exploration of ICL's ability to evaluate and infer LLMs has become a prominent trend (Dong et al., 2022). Some studies have tried to apply ICL in dialogue generation. Roy et al. (2023) proposes a two-stage style transfer framework to leverage ICL for dialogue style transfer. Meade et al. (2023); Lee et al. (2022) employ satrieval-based framework to mitigate bias and toxicity in chatbot-generated re- sponses, guiding the model towards safer and more responsible dialogue. Hu et al. (2022); Chen et al. (2023) propose method for long dialogue com- pression, enabling each prompt to contain more examples and improving dialogue state tracking performance. ICL has also been utilized for un- supervised generation of dialogue data in certain contexts (Li et al., 2022), expanding the potential applications of this approach.
References
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašié. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint 181.007880.00278.
Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence- level BLEU. Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 362-367, Baltimore. Maryland, USA. Association for Compu- tational Linguistics.
Derek Chen, Kun Qian, and Zhou Yu. 2023. Stabilized in-context learning with pre-trained language models for few shot dialogue state tracking. arXiv preprint arXiv:2302.059233
Overall, the aforementioned works concentrate

---

Wei Chen, Zhiwei Li, Hongyi Fang. Qianyuan Yao. Cheng Zhong, Jianye Hao, Qi Zhang. Xuanjing Huang, Jiajie Peng, and Zhongyu Wei, 2022. A Benchmark for Automatic Medical Consultation Sys- tem: Frameworks, Tasks and Datasets. Bioinformat- icy. Btac817.
Dongdong Li, Zhaochun Ren, Pengjie Ren, Zhumin Chen, Miao Fan, Jun Ma, and Maarten de Rijke. 202 Semi-supervised variational reasoning for medical dialogue generation. In Proceedings of the 44th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, SIGIR 22,, page 544-554, New York, NY, USA. As- sociation for Computing Machinery.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun. Jingjing Xu, and Zhifang Sui. 2022. A survey for in- in-context learning. arXiv preprint arX.022301.0023.
Dongdong Li. Zhaochun Ren, Pengjie Ren, Zhumin Chen, Miao Fan, Jun Ma, and Maarten de Rijke. 2021b. cemi-supervised variational reasoning for medical dialogue generation. In Proceedings of the 44th International ACM SIGIR Conference Re- search and Development in Information Retrieval, pages 344-
Tongfeng Guan, Hongying 7an, Xiahing Zhou, Hongfei Xu, and Kunli Zhang. 2020. Cmeic: Construction and evaluation of chinese medical information extrac- tion dataset. Natural Language Processing and Chinese Computing: 9th CCF International Con- ference, NLPCC 2020, Zhengzhou, China, October 14-18, 2020, Proceedings, Part 9, pages 270-282. Springer.
Zekun Li, Wenhu Chen, Shiyang Li, Hong Wang. Jing Qian, and Xifeng Yan. 2022. Controllable dialogue simulation with in-context learning. arXiv preprint arXiv:2210.04188
Zan Hongying, Li Wenxin, Zhang Kunli, Ye Yajuan, Chang Baobao, and Sui Zhifang. 2021. Building a pediatric medical corpus: Word segmentation and named entity annotation. In Chinese Lexical Seman- ties: 21st Workshop, CLSW 2020, Hong Kong. China, May 28-30, 2020, Revised Selected Pupers 21, pages 652.664.35 Springer.
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona. Spain, Association for Computational Linguistics.
Xinzhu Lin, Xiahui He, Qin Chen, Huaixiao Tou, Zhongyu Wei, and Ting Chen. 2019. Enhancing dia- logue symptom diagnosis with global attention and symptom graph. In Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5033-5042, Hong Kong, China, Association for Computational Linguistics.
Alex Howard, William Hope, and Alessandro Gerada. 2023. Chatgpt and antimicrobial advice: the end of the consulting infection doctor? The Lancet Infec- tious Diseases, 2344):405-406
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A Smith, and Mari Ostendorf. 2022. In-context learning for few-shot dialogue state tracking, arXiv prepriat xrXiv22203.0855
Wenge Liu, Jianheng Tang. Yi Cheng, Wenjie Li. Yefeng Zheng, and Xiaodan Liang. 2022. Meddg: entity-centric medical consultation dataset for entity- aware medical dialogue generation. Nat- ural Language Processing and Chinese Computing: 11th CCF International Conference, NLPCC 2022, Guilin, China, September 24-25, 2022, Proceedings, Part I. pages 447-459 Springer.
Ziwei Ji, Nayeon Lee, Rita Frieske Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung, 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 25(12):1-38.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-12)
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs//197. 11692.
Young-Jun Lee, Chae-Gyun Lim, and Ho-Jin Choi. 2022. Does GPT-3 generate empathetic dialogues? a novel in-context example selection method and au- tomatic evaluation metric for empathetic dialogue generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 669- 683. Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
empathe
Renqian Luo, Liai Sun, Yingee Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6). Bbac409.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising
Nicholas Meade, Spandana Gella, Devamanyu Haz- arika, Prakhar Gupta, Di Jin, Siva Reddy, Yang Liu, and Dilek Hakkani-Tür. 2023. Using in-context learning to improve dialogue safety. arXiv preprint arXiv:2302.00877
pre-training for natural language generation, translation, and comprehension.

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Nearal Information Processing Systems, 35:27730-27744,
Deeksha Varshney, Aizan Zafar, Niranshu Kumar Be- hera, and Asif Ekbal. 2023a. Knowledge graph as- sisted end-to-end medical dialog generation. Artifi- cial Intelligence in Medicine, page 102535.
Deeksha Varshney, Aizan Zafar, Niranshu Kumar Be- hera, and Asif Ekbal. 2023b. Knowledge grounded medical dialogue generation using augmented graphs. Scientific Reports, 13(1333310.
Alec Radford. Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 18):9
Weizhi Wang, Zhirui Zhang. Junliang Guo, Yinpei Dai, Boxing Chen, and Weihua Luo. 2022. Task-oriented dialogue system as natural language generation. In Proceedings of the 4Sth International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks,
Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. 2021. Semantic answer similarity for eval- uating question answering models. arXiv preprint 008.
Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing Huang, Kam-fai Wong, and Xiangying Dai. 2018. Task-oriented dialogue system for automatic diagnosis. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pages 201-207, Melbourne. Australia. Association for Computational Linguistics.
Shamik Roy, Raphael Shu, Nikolaos Pappas, Elman Mansimov, Yi Zhang. Saab Mansour, and Dan Roth. 2023. Conversation style transfer using few-shot learning. arXiv preprint arXiv:2302.08366
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint 2.X5::221..05000
Yixuan Weng. 2022. A large chinese medical eqa. https://github ctm COmN/ENGSYX/CMCOA
Yuan Xia, Jingbo Zhou, Zhenhui Shi, Chao Lu, and Haifeng Huang. 2020. Generative adversarial reg- ularized mutual information policy gradient frame- work for automatic diagnosis. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1062-1069.
Karan Singhal, Shekoofch Azizi, Tao Tu, S Sara Mah- davi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022. Large language models encode clinical knowledge. arXiv preprint 13X11:2212..3338.
Yunyi Yang. Yunhao Li, and Xiaojun Quan. 2021. Ubar: Towards fully end-to-end task-oriented dialog system with gpt-2. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 14230- 14238.
Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2021. Multi-task pre-training for plug-and-play task-oriented dialogue system. arXiv preprint arXiv:2109. 149799.999.4739.
Haipeng Sun, Junwei Bao, Youzheng Wu, and Xiaodong He. 2022. Mars: Semantic-aware contrastive learn- ing for end-to-end task-oriented dialog.
Guangtao Zeng. Wenmian Yang. Zeqian Ju, Yue Yang. Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng. Xiangyu Dong, Ruoyu Zhang. Hongchao Fang. Penghui Zhu, Shu Chen, and Pengtao Xie. 2020. MedDialog: Large-scale medical dialogue datasets. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP). pages 9241 Online. Association for Computa- tional Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27.
Chen Tang, Hongbo Zhang, Tyler Loakman, Chenghua Lin, and Frank Guerin. 2023. Terminology-awe aware medical dialogue generation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE.
Hongbo Zhang, Junying Chen, Feng Jiang. Fei Yu. Zhihong Chen, Jianquan Li, Guiming Chen, Xi- angbo Wu, Zhiyi Zhang, Qingying Xiao, Xiang Wan, Benyou Wang, and Haizhou Li. 2023, Huatuogpt. to- wards taming language models to doctor. arXiv preprint arXiv 15075.
Mina Valizadeh and Natalie Parde. 2022. The Al doctor is in; A survey of task-orienter dialogue systems for healthcare applications. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6638- 6660, Dublin, Ireland. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint 1rXiv:1904.09675

---

A Details of Automatic Evaluation
Our approach involves using the skip-gram algo- rithm discover synonyms of the initial vocab- ulary and incorporate them into the vocabulary. This method helps us identify aliases and common names of the terms and facilitates the calculation of the TnM fl-score by providing term similarity information. To accomplish this, we train skip- gram word embeddings using the CMCQA (Weng, 2022) corpus, which consists of 1.3 million com- plete conversations, 19.83 million sentences, and 650 million tokens. Our skip-gram algorithm em- ploys a word vector dimension of 300, a window size of 5, a sub-sample ratio of 3, and undergoes 8 rounds of training. Using the obtained word vec- tors, we enrich the high-frequency terminology list by adding the 10 synonyms with the closest se- mantic meanings for each term. Subsequently, we save the term vectors obtained from the training facilitate the calculation of similarity scores.
A.1 Intent Classifier
To compute Int, we employ the IMCS-IR dataset (Chen et al., 2022) and utilize Roberta- large (Liu et al., 2019) as the backbone model for training the classifier. To process a response, we concatenate it with the dialogue history and input the combined sequence into the model for classifi- cation. The input format is structured as follows:
<>> [dialogue history] 
 [response]
We extract the hidden layer vector of the '
' to- ken as the embedding representation for the re- sponse, which is subsequently classified using a two-layer neural network with a hidden dimension of 768. The intentions considered are presented in Table (6), and the model achieves an accuracy of 0.86 on the validation set.
Discussion. It is worth to note that Meddg (Liu et al., 2022) and a few other datasets propose sim- ilar evaluation metrics based on medical entity matching. This raises the question of why we are considering medical terminology matching instead of medical entity matching. We chose terminology matching instead of entity matching because we observe several issues.
Action Type
Target
Request
Symptom
Request
Etiology
Request
Basic Information
Request
Existing Examination and Treatment
Inform
Drug Recommendation
Inform
Medical Advice
Inform
Precautions
Inform
Diagnose
Firstly, most medical named entity recognition (NER) datasets (Hongying et al., 2021; Guan et al., 2020) focus on recognizing a limited range of entity types, which often excludes entities crucial to the consultation process, such as etiology related eating habits. Secondly, since our target audience patients without medical expertise, the dialogue content incorporates numerous colloquial words. This poses challenges for NER models that are primarily designed for professional terminology. Consequently, the traditional NER model can only identify a limited number of entities within the dataset. For example, in the report of Meddg, average sentence contains only 0.56 entities, which hampers the evaluation of the dialogue content.
Other
Other
Table 6: The intents of the doctor.
A.2 Term Extraction
Following the implementation outlined in (Li et al., 2021a), we employ word matching to extract high- frequency terms from responses. To accomplish this, we initially construct glossary of medical high-frequency words. We then appropriately ex- pand the glossary to ensure a satisfactory hit rate for term matching. Our approach is detailed as follows.
Initialization of the glossary. We utilized the Chinese medical high-frequency vocabulary ac- quired from THUOCL6 as the initial glossary, comprising 18,749 frequently employed terms in doctor-patient communication.
Therefore, we believe that evaluating high- frequency medical terms would be a more logi- cal approach. The glossary we selected offering an 81.9% coverage of the labeled entity classes in Meddg and featuring an average of 2.13 entities per utterance. Table (7) illustrates the comparison between Meddg's 's original entity extraction and our improved term extraction, clearly showing that our extracted terms better cover the dialogue content.
Glossary expansion. We utilize the skip-gram al- gorithm to expand the list of high-frequency terms.

---

Role
Dialogue History
Meddg
Ours
Patient
How colitis, the doctor prescribed reha- bilitation liquid and mesalachin granules, please read the be sheet (Female, 35)
Doctor
What are your current symptoms? Have you had gastroscopy?
Gastroscopy
Gastroscopy
Patient
feel like I have diarrhea, but can't get out, and my stomach doesn't good. don't know if it's serious, don't have a gastroscopy.
Doctor
medication prescribed by the doctor symptomatic. just keep taking it. There may ulcers the intestines. Usually, you should also pay attention to your diet, avoid spicy stim- ulation, quit smoking and limit alcohol.
Medicine, Intestinal, Uleer, Diet
Patient Doctor
necessary
a gastroscopy?
Because there is an uleer the intestine, the stomach should also be checked. But it is okay to take medicine first.
Intestine, Ulcer, Stomach, Check
Patient Doctor
How long does this mesalachin need to take? The duration of medication is determined by the symptoms. When you feel well, you can stop the medication. Pay attention to your diet during the day.
Symptoms, Duration of medication, Diet
Table 7: A comparison of the entities provided by Meddg with the terms extracted. It can be observed that our extracted terms can better cover the content of the conversation.
A.3 Term Matching
B Implementation of Mars
Let A; = {at...,,,, ./,} represent the set of terms in the system response i, and B, = -b. b',} rep- resent the set of m terms in the standard response i. We use the cosine similarity of the skip-gram vector corresponding to the term the required similarity score for Top-n Match. To calculate the fl score, we define the following term types:
The overview of the migration process are shown in Figure (5). Mars (Sun et al., 2022) incorporates two decoders, namely the Dialogue State Decoder and the Action State Decoder, which both utilize Shared Encoder. The original workflow of Mars is as follows: 1) Utilize the Shared Encoder encode the context C' and user input U, and employ the Dialogue State Decoder decode the dialogue state D. 2) Employ the dialogue state to retrieve the corresponding entity DB from the database. 3) Utilize Shared Encoder to encode C, U, and DB, and employ the Action State Decoder to decode the dialogue action A and dialogue response R.
Truth-Positive (TP): For any at E As. if there exists by, E B, Top-n Matches with it, then classify at TP term.
False-Positive (FFP): If a is not TP term, it is classified as an FP term.
Since our task does not involve interaction with the database, we omit the second step but employ D instead of DB. Considering that the dataset lacks labeled dialogue states and actions, we adopt the approach of Li et al. (2021a) and utilize high- frequency medical vocabulary found in sentences as the states and actions. Our subsequent experi- ments show that this approach facilitates the model to generate more medical terms while improving the fluency of the output. We keep the other set- (3) tings the same as the original configuration.
False-Negative (FN); If by does not top-n match with any o; E Ag, we classify bi, as an FN term.
Then the calculation formula for micro-fl is as follows, where N denotes the number of samples, and i denotes the index of the samples.
P TP Σ TP, + + FP. ΣN TP:
R E,TP, FN 2PR
(4) C ICL on Bloomz
This section explores the adaptability of Bloomz to (5) In In-Context Learning. We observed that the impact
F P+R

---

queey
DB
A.
D
A.
Dialogue Decoder
Dialogue Decuder
Decoder
Decoder
migration
Shared Encoder
Shared Encedee
Shared Encoder
Shared Encoder
Originl
Figure Migration methods for Mars models.
ICL on BLoomz was minimal. Adjusting the order of the examples or changing the number of examples did not result in sigmificant changes in the model's performance. To illustrate this phe- nomenon, provide the details of the following experiment.
C.2 Experimental Result
The test results are presented in Figure (6). The horizontal axis of the figure represents the number of unique outputs obtained from the 10 prompts per sample, while the vertical axis represents the percentage of samples in the dataset that have that specific number of unique outputs. From the fig- ure, can observe that Bloom generates almost different answers for ICL prompts containing dif- ferent examples. In contrast, Bloomz generates more duplicate answers, indicating that Bloomz is less influenced by the sample. The observation that Bloomz's dialogue strategy is less mallcable, sug- gests that the Instruct-tuning process may compro- mise the model's In-Context Learning capabilities on the medical dialogue generation task.
C.1 Experimental Setup
In order to evaluate the impact of In- In-Context Learning (ICL.) on the performance of Bloom and Bloomz, conducted experiments the valida- tion set of the KaMed dataset. For each sample in the validation set, we generated 10 prompts, where each prompt consisted of random number ran- domly selected examples. The maximum number examples included
prompt was limited to less than 4. Using these 10 prompts, we made pre- dictions using Bloom and Bloomz models respec- tively. We then count the number of non-repeats of the system's responses.
D
Case Study
We selected a representative sample from the KaMed dataset to facilitate an interpretable com- parison of different models. The corresponding results presented in Table (8). In the scenario depicted within this sample, the patient's mother provided limited information to the doctor, which is insufficient to support an accurate diagnosis. Con- sequently, the doctor made the decision to gather more necessary symptom information.
Our observations reveal that the Bart model ex- hibit awareness of the information gap. How- ever, Bart inquire about known symptom, indi- cating that the small model lacks certain medical common sense and doesn't 't know what information to collect. Mars places greater emphasis on generat- ing pertinent medical terms. In the absence of other
12
Figure 6: Diversity comparison of Bloom output and
Bloomz output.

---

baby has cold sixth day, bronchitis, phlegm, cough, runny nose. I went to hospital traditional Chinese medicine and prescribed Chinese medicine without decoction.
History
After taking it for three days, didn't seem work. What should (male, 13 months)
Golden
Hello! Does your baby have other symptoms? Such fever?
BART
Hello, is your baby coughing badly?
Mars
The baby is bronchial sinusitis, bronchitis, cough, runny nose.
BLOOM
Your haby needs timely medical treatment. The child's current symptoms obviously suppurative tonsillitis. In addition taking anti-inflammatrry treatment for this discase, it is recommended to perform tonsil removal surgery at the same time.
BLOOMZ
The baby is still young. it is recommended take western medicine. Chinese medicine has too many side effects on the baby, and the effect is not good.
ICL RAND
If your child coughs and has phlegm, you take phlegm-reducing medicines, such Mucosolvan and keep
avoid catching cold.
ICL SBERT
The baby
coughing and has phlegm. It is considered bronchitis. You continue take traditional Chinese medicine, or you take cephalosporins, combined with atomization treatment. Hello, does your hahy have fever now?
PlugMed
Table One case extracted from KaMed. Note that these dialogues are originally in Chinese, the English version may not get the same responses.
information, Mars choose to reiterate the patient's words in an attempt to enhance the likelihood of a terminological hit. Unfortunately, this approach re- sult in the generation of responses of lower quality, The LLM produce lengthier responses, while ICL Rand and ICL Shert attempt to provide diagnosis directly. On the other hand, Bloom and Bloomz generate false and biased utterances, respectively, indicating a dearth diagnostic strategies within these larger models. contrast, PlugMed generate responses similar to the ground-turth, thus suggest- ing the effectiveness of our proposed method.
E Prompt Format
Our experiments employ two prompts: the Instruct prompt, shown in Figure (7), and the In-context prompt, shown in Figure (8). The Instruct prompt consists of a concise statement of the task goal, fol- lowed by direct input of all conversation histories into the model, serving test for the model's zero-shot capability. On the other hand, the In- context prompt involves presenting examples prior to the samples activate the model's imi- tation ability.

---

: 30 BE4, F
Task: Play as doctor and continue wriling the following medical dialogue.
#E B 9H, B (5) 233)
Patient: Gastritis, chest tightness, shortness of breath, indigestion (Male,
E:: 23, 29 7, A
Doctor: Hello, how long has this been going on and there any other discomfort?
Be $1 77. $J St a0 3 f 8ig th 10 ft Patient: Over two months, hiccups, stomach numbling is this functional indi with superficial gustritis?
B:: F-E. / m A R H 7 ft R #b fr. M - (6) 222
Doctor: No, superficial gustritis superficial gustritis indigestion functional dingnosis. symptoms you have both, are you taking medicution?
#: "b4g
Patient: I have takem some Chinese medicine.
E::: (53512
Doctor: Have taken western medicine?
Z MiN, ME -77-BAE 7 SEEFIE AA FATTEE FA 1
*??
Patient: I have tooridee rabeprazole,, and Daxil home. I've taking them for while and feel better when cut them, but don't feel better when I stop, and I have very had hiccups. How can I get my weight back?
tt:: 66166 # 2 11 # Fath TARAETF x $0 11 peft,
15 pylari,
Look, you have three pylori infection positive, you also need :e ABl [ 2 i (90)224 tHR 2 73 7of 2 E.k
drugs
eat, recommended medicine
weeks row. And you
Patient: the H. pylori infection is negative, can this stomach discase cause wasting and why can't regain my weight when my appetite back?
Doctor:
Figure An example of the Instruct prompt.

---

Dialoguc abetract: The parient developed inflammation of the ona tonsils three months and her symptoms have not improved significantly.
##: 1 R8 2 3 2213?
Patient: How often do usually need to be tested for gastroscopy?
E#: E1 f5 fi fid HAAEE M #
Doctor: Three months think. Have you checked for pylori, if then three months.
m#: 11 7 19
used
medicine now have
:E R 19 B8l TAME t0? [EOF]
Doctor: Have you laken enough anti Helicobacter pylori medication for 14 days? [EOF)
33 11 mh B Hi 80 5i is 86f -
Dialogue abstract: Patient presented gastroscopic superficial gustritis with erosion for week.
BE: Mi (TMM 2218 1 S #2
Doctor: Have you been checked for pylori, bacterium that generally couses your condition.
#::: # #127 / PPP 12B 183881888111 2
Patsent: just want to simply ask if discase is serious and how to treat
BE EI A 10 ADB HH WEBDIE II MENA AE M JAKKEM d [RO]]
Doctor: Since you have symploms gemerally need sterilization resistance letter the morning piece ocal month. gustric triple oral one week
Then all drugs stop a month after the review of H. pylon [BOF]
3d ST M: a# H5 101 ar MER 2A FR)
Dialogue abstract: The patient developed gastritis with crosion three weeks ago and was treated with Rabeprazole for three weeks.
:8:2: 30 145t -2222,, REEEE
Patient: The pharnacy told take capsales day, and then take them every other day.
:E::: 1]
Doctor: Thar's fine.
#: ::: 16,0, We ME B BWNR
Patient: been taking rabepnzzlle
feel fine, just foreign body feeling in my throat.
Doctor: Well, will slowly climinated
88: HH B 15 5 60M IE 7- Magig
Patient: Do you mean
need esophagoscope for a foreign body foeling throat?
SE::: 7V [FOF)
Doctor: Not nccossary [FOF]
RI 9: a # 8i th HI E E
Dialugue abstract: The patient developed bloating two months ago and currently has no significant improvement in symptoms.
B#: 10. #t MIE 7 HaY 1M #6 # #AMM X 75. H 8 Heg Patseal: No. am currently taking pyloric quadruple modication followed by probiotics and letracyoline, Currently the stools are not formed and am particularly g 12i2 R 26, (99 2 3115
diarthen
two drugs, and It : # 78 #07777
time take some probioties.
Patient: there any psychistric medication?
: HT A (B t
Doctor: You can cat some flpentiool but need to go the hospical specialist to prescribe [EOF)
31 : A M. MYA 407, B BH M. R
Dialogue abstracn: patient developed hiceups, chest rightness, and shortness of two months ago, current symptoms have not improved
significantly.
::::: (, =
4M. 3 -
Doctor: Look, you have three kinds drugs still have to cat, it recommended to take 4 woeks row, And need to check the H. pylori, if the pyfori infoction positive you also need take medicine to cure.
B# HMl JF # 39 ty # $339 N ft 25 RA
Patseal: If the H. pylori infoction is negative, can this stomach discase cause wasting and why I can't regain my weight when my appetite back?
Doctor:
Figure 8: An example of In-context prompt. Blue text indicates examples and red text indicates the sample.


Reviews:
[{"review_id": "u6j1Zwa7uE", "reviewer": "Reviewer_nsEF", "summary": "", "questions": "> The authors did not report standard evaluation metrics for dialogue generation, such as ROUGE, METEOR, and PPL. Additionally, there is a lack of semantic evaluation metrics, such as BERT Score, which could have provided further insights.\n\n> The paper lacks a discussion on the scale of models used. Failing to demonstrate significant efficacy with only a small gain in performance, especially considering the increased complexity of the models, raises concerns.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The paper lacks technical novelty as most of its components are well-known, simply combined into a system. Moreover, it fails to address the technical challenges relevant to the research.**\n\nWe appreciate your feedback and concerns. We think that the concern \"lack of technological innovation\" in our paper may raise from a paradigm shift in the research approach, which we'd like to elucidate. Owing to the large size of LLMs and the high cost of their training, the application of Prompts Engineering techniques, including ICL and CoT, are frequently employed for the adaption of downstream tasks. Prompts Engineering is a training-free methodology that strives to formulate prompt generation strategies, and it is an important means of realizing Artificial General Intelligence (AGI). This kind of research does not emphasis on the model design but treats the off-the-shelf AI models as callable services. **The challenge with prompt engineering lies in the design of a task-oriented prompt generation strategy**. We have experimentally demonstrated that our proposed approach is effective in improving the dialog strategy of LLM and solve the problems of example retrieval as well as example compression in ICL tasks. Therefore, we believe we have addressed the technical challenges in our research. \n\nYour feedback highlights the need for a clearer distinction between ICL and traditional methods in the relevant work section. We will remedy this deficiency in the final version of the paper.\n\n\n**Detailed illustrations or elaborate descriptions of the internal structure of the prompt generator, SLM Encoder, and SLM Decoder are missing in the paper. Although the mentioned frameworks are noted, the specific architectures remain inadequately explained.**\n\nThank you for the advice. At present, the detailed information about encoders and decoders have not been included, as we feel that the module implementation is mainly related to the training data, not its internal structure. And training with a conventional encoder-decoder pre-trained model can get basically the same effect. The encoder-decoder pre-trained model we adopted in this paper is BART model. We will add this discussion in the revised version.\n\n**The authors did not report standard evaluation metrics for dialogue generation, such as ROUGE, METEOR, and PPL. Additionally, there is a lack of semantic evaluation metrics, such as BERT Score, which could have provided further insights.**\n\nWe understand the reviewer's concerns about missing some standard evaluation metrics. The reason why we did not include those metrics stems from the specific nature of our task, which is to complete the Task-Oriented Dialogue (TOD). In this case for evaluating the dialog quality, Success Rate, Inform Rate, and BLEU are widely used. These metrics can measure the accuracy of dialogue actions and entities mentioned in dialogs, which is more in line with the essence of our evaluation criteria. Of course, we can include some of the experiment results for those missed metrics in the revised version if it is necessary.\n\nAnother pertinent aspect is that LLM typically generates a more comprehensive response. However, in real-world situations, medical practitioners usually choose concise answers. This tendency puts LLM at a relative disadvantage in these standard evaluation metrics. Nevertheless, it is worth emphasizing that a more comprehensive response always helps to enrich the patient experience. We believe that this is an advantage of LLM, but these standard evaluation metrics may unintentionally underestimate this advantage.\n\n**The paper lacks a discussion on the scale of models used. Failing to demonstrate significant efficacy with only a small gain in performance, especially considering the increased complexity of the models, raises concerns.**\n\nThank you for your thoughtful comment. Indeed, we should report the scale of the model used here. We will include the information in the revised version. Of course, in the work, our research goal is to explore how to use non training methods to enable LLM to perform medical dialogue tasks. Because we do not need to train the LLM, the number of model parameters is not the focus of our consideration, and we do not affect the generalization ability of LLM. Meanwhile, we have experimentally demonstrated that the generalized model outperforms the specialized model in specific fields. Our other contribution is to demonstrate the effectiveness of the research path of “large model + strategy injection”, which is different from the traditional method of “small model + knowledge injection”.\n\n**The paper overlooks an essential aspect of ethical considerations, especially in the context of medical terminology. An ethical discussion is crucial when dealing with medical data and applications.**\n\nThank you for your comment. We appreciate your emphasis on the importance of ethical considerations, particularly within the realm of medical terminology and data. We will enhance the paper by incorporating a comprehensive discussion on ethical implications, underscoring the significance of responsible handling and applications of medical data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors take into consideration both the global context and the local context for generating responses. The global context captures similar dialogues from the dataset, and the local context is used to extract symptom information for generating medically appropriate responses.\n\n2. The authors test their model on three datasets containing a substantial number of dialogues which gives assurance to the credibility of the proposed method.\n\n3. The intent accuracy metric guarantees reasonable response generation, and the high-frequency medical term accuracy makes sure appropriate medical terms and information is present in the generated responses.", "reasons_to_reject": "1. The paper lacks technical novelty as most of its components are well-known, simply combined into a system. Moreover, it fails to address the technical challenges relevant to the research.\n\n2. Detailed illustrations or elaborate descriptions of the internal structure of the prompt generator, SLM Encoder, and SLM Decoder are missing in the paper. Although the mentioned frameworks are noted, the specific architectures remain inadequately explained.\n\n3. The authors did not report standard evaluation metrics for dialogue generation, such as ROUGE, METEOR, and PPL. Additionally, there is a lack of semantic evaluation metrics, such as BERT Score, which could have provided further insights.\n\n4. The paper lacks a discussion on the scale of models used. Failing to demonstrate significant efficacy with only a small gain in performance, especially considering the increased complexity of the models, raises concerns.\n\n5. The paper overlooks an essential aspect of ethical considerations, especially in the context of medical terminology. An ethical discussion is crucial when dealing with medical data and applications.", "questions_to_authors": "> The authors did not report standard evaluation metrics for dialogue generation, such as ROUGE, METEOR, and PPL. Additionally, there is a lack of semantic evaluation metrics, such as BERT Score, which could have provided further insights.\n\n> The paper lacks a discussion on the scale of models used. Failing to demonstrate significant efficacy with only a small gain in performance, especially considering the increased complexity of the models, raises concerns.", "paper_topic_and_main_contributions": "The paper introduces a two-module framework for generating responses and employs a two-step evaluation metric. The primary objective is to achieve automatic Medical Dialogue response generation using Language Models (LLMs). This is accomplished by harnessing In-context learning and a two-component system comprising a prompt generation module and a response ranking module.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "The paper overlooks an essential aspect of ethical considerations, especially in the context of medical terminology. An ethical discussion is crucial when dealing with medical data and applications."}, "extra_text": {}}, {"review_id": "6if0475BSJ", "reviewer": "Reviewer_V8yn", "summary": "", "questions": "1. What data is the SLM trained on? The dialogue corpus in the selected datasets? If so, it makes sense that it will select the response more similar to the datasets.\n\n2. The baseline model which is simply using SBERT and history to perform retrieval. What is the different with Global retrieval? And the performance simply with SBERT is very competitive and even outperforms all other methods in some cases. Then, what is the necessity for global retrieval and local retrieval?\n\n3. The new evaluation metric requires a set of medical terms. How the terms are collected? The quality of the terms. Does it has a wide coverage of different medical domains?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**In this paper, only BLOOM model is used as the backbone. The paper claims the proposed method is a plug-and-play module. Therefore, the effect on multiple backbone LLMs should be shown to prove its generalizability and the effect also applies to other models. Also, it is shown ICL does not help with the BLOOMZ model, which also demands further investigation on other LLMs.**\n\nThank you for your suggestion. We primarily utilize Bloom due to its dual merits of being open source and interpretable. These qualities empower us to formulate the ensuing conjectures. We think that our approach possesses theoretical applicability to the majority of the LLM's foundational models—those that have exclusively undergone unsupervised training. However, problems may arise if our approach is applied to models that have undergone instruct tuning or RLHF. \n\nWe posit that the root cause for this phenomenon could be attributed to the nature of the training tasks to which the model has been exposed. For instance, Bloomz's instruction fine-tuning dataset, designated as bigscience/xP3, predominantly comprises single-round QA tasks, lacking the inclusion of multi-round dialog tasks. It is conceivable that this pre-training emphasis on single-round QA potentially compromised the model's adeptness in partaking in multi-round dialogues. Simultaneously, the dataset primarily orients itself towards the general domain, potentially hampering the model's prowess in specialized domains. Our experimental findings demonstrate that such issues were not encountered in the case of ChatGPT.\nWe randomly selected 1000 samples from the Meddg dataset to test the ChatGPT, and the results are as follows:\n\n| **Model**        | **BLEU** | **T1M@P** | **T1M@R** | **T1M@F** | **T3M@P** | **T3M@R** | **T3M@F** | **T5M@P** | **T5M@R** | **T5M@F** | **INT** |\n| ---------------- | -------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | ------- |\n| Vanilla          | 4E-05 | 3.5       | 22.8      | 6.1       | 6.4       | 39.1      | 11        | 8.3       | 48.7      | 14.2      | 23      |\n| Global View      | 0.4      | 5.0       | 15.9      | 7.6       | 8.3       | 26.0      | 12.6      | 10.2      | 32.0      | 15.5      | 32.2    |\n| Local Primary    | 0.4      | 4.8       | 15        | 7.3       | 8.4       | 25.6      | 12.6      | 10.1      | 31.1      | 15.3      | 29.1    |\n| Local Secondary  | 0.4      | 4.7       | 15.1      | 7.2       | 8.1       | 25.3      | 12.3      | 10.1      | 31.3      | 15.3      | 31.5    |\n| \\+Ranking (Ours) | 0.4      | 5.1       | 16.4      | 7.7       | 8.7       | 27.4      | 13.2      | 10.8      | 33.3      | 16.3      | 33.6    |\n\n\nWe observed that, ICL played a pivotal role in enhancing ChatGPT's diagnostic capabilities. At the same time, our rerank strategy is also valid for the model. Meanwhile, a noticeable discrepancy arises in the comparison of ChatGPT's BLEU and INT scores with those of Bloom's. Analysis of the generated output reveals a discernible pattern wherein ChatGPT frequently refrains from directly addressing the patient's inquiries. Instead, it frequently opts to conclude the diagnostic procedure by advising the patient to seek medical consultation. This strategy avoids making mistakes but results in a much less useful model. Consequently, one could reasonably posit that the integration of security protocols into ChatGPT by OpenAI might have compromised the model's diagnostic capabilities.\n\nDue to the additional time required to supplement the human evaluations, we will add the full experimental content of ChatGPT in the revised version.\n\n**The re-ranking approach is somehow heuristic to me. The BART model is producing more chitchat outputs with lower information density. Using the perplexity of BART to rank LLM outputs will result in choosing output with less information, which is claimed to be more human-like. All validations and analyses are with BLOOM(Z) model which is shown to easily give conclusions. However, it is not clear how another model performs. Besides, from my experience, changing the prompts can also guide the model to be more careful before making medical conclusions/suggestions.**\n\nThanks for your comments. The rerank model functions mainly based on whether the dialog actions are reasonable or not, but not only on the information density. That is why the INT and TnM metrics be improved after reordering. There is no necessary causal relationship between information density and whether the dialog action is correct or not. At the same time a decrease in information density is likely to lead to a decrease in TnM@R, i.e., the rerank model selects replies that contain fewer medical entities. From the results of the ablation experiments, PlugMed's TnM@R is not the lowest among the four strategies, and on some datasets, instead, it has a high ranking. \n\nWe observed that Bloom and Bart diverge the most in their choice of whether to take the \"inform\" action. Therefore, instead of interpreting this phenomenon in terms of information density, we propose a more plausible explanation: the rerank model, with its conservative nature (tends to request), would penalize overconfident responses (tends to inform), thereby acting as a counterbalance. Due to the dialogue action are corrected, the hit rate for medical entities is similarly improved.\n\nFor the last question, we believe that enhancing the model's self-awareness by only imposing instructions may be overly simplistic. The model must not only act with caution, but also understand the relevant data collection priorities that depend on different diseases. The dynamic nature of collected information on various diseases makes the formulation of precise instructions complex. This is the role of ICL in alleviating complex instruction challenges.\n\n**Overhead computation and grounding knowledge. The plug-in-and-play model introduces additional overhead for retrieval and reranking. A discussion on this is needed. Meantime, it requires a medical conversation pool to retrieve the samples. Currently, the setting is to retrieve samples from the same dataset. It may lead to bias because similar dialogue styles and domains are covered. The generalizability to other data sources is not very clear.**\n\nThank you for your valuable feedback and insightful questions. The primary overhead lies in the generation of multiple responses, a process that frequently exceeds ten seconds (LLMs are slow). This is in contrast to the minimal time required for retrieval and reranking. We will add the description of this in the revised version. Moreover, the dataset we have selected originates from China's widely used online medical consultation platform. The platform has registered doctors from all over the country covering the most common types of diseases. As a result, we hold the conviction that this dataset adequately encompasses a diverse array of situations within medical consultation scenarios.\n\n**What data is the SLM trained on? The dialogue corpus in the selected datasets? If so, it makes sense that it will select the response more similar to the datasets.**\n\nThe SLM undergoes training using the dialog dataset employed by ICL. Subsequently, it possesses the capability to rectify the diagnostic strategy of the LLM. It is necessary to highlight that the SLM doesn't directly interfere with the LLM's response generation process. As a result, the ultimately chosen responses retain their comprehensiveness and depth of medical knowledge.\n\n**The baseline model which is simply using SBERT and history to perform retrieval. What is the different with Global retrieval? And the performance simply with SBERT is very competitive and even outperforms all other methods in some cases. Then, what is the necessity for global retrieval and local retrieval?**\n\nSBERT and Global retrieval are indeed same method. Our research revealed that Global retrieval's efficacy is influenced by the length of conversation history, leading to a decline in retrieval quality as the history lengthens. In response to this challenge, we introduced Local retrieval. This variant remains unaffected by conversation length, bolstering its performance with lengthy conversations. Substantiation for this lies in the comparable performance of Global retrieval and PlugMed on MedDialog, characterized by an average of 4.76 conversation rounds. Conversely, on MedDG (averaging 9.92 rounds) and KaMed (averaging 11.62 rounds), PlugMed notably outperforms Global retrieval. \n\nMeanwhile, we believe that the effective performance of Global Retrieval originates from the long-tailed distribution of sample lengths. In particular, during testing, the dialog session is commonly divided into multiple samples based on the dialog rounds. Clearly, a majority of these samples encompass only brief conversational histories. Such a testing approach significantly in favor of Global Retrieval strategy. Nonetheless, due to the frequent occurrence of lengthy conversations in real-world situations, integrating Local Retrieval becomes essential.\n\nWe will add the discussion of this issue in the revised version.\n\n**The new evaluation metric requires a set of medical terms. How the terms are collected? The quality of the terms. Does it has a wide coverage of different medical domains?**\n\nWe appreciate your insightful questions regarding our evaluation metric. The collection of medical terms for our metric is facilitated through THUOCL (THU Open Chinese Lexicon), a high-quality Chinese thesaurus compiled by Tsinghua University's Laboratory of Natural Language Processing and Social and Humanities Computing. This compilation incorporates terms from diverse sources, including mainstream websites, hot keywords, and input methods thesaurus. Rigorous manual screening has been undertaken in multiple iterations to ensure the precision of included terms. Upon examination, we found that the thesaurus demonstrates a commendable breadth by encompassing various medical domains, including diseases, symptoms, drugs, and food. Its contents adhere to higher standards of terminological uniformity and reliability. We will add the explanation of this issue in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is in general well-written and easy to follow.\n\n2. The proposed method is logically sound. Some evidence is given to validate the proposed ideas.\n\n3. The paper proposes a couple of things including 1) a retrieval method, 2) a reranking method 3) two evaluation metrics specifically for medical dialogue generation.", "reasons_to_reject": "1. In this paper, only BLOOM model is used as the backbone. The paper claims the proposed method is a plug-and-play module. Therefore, the effect on multiple backbone LLMs should be shown to prove its generalizability and the effect also applies to other models. Also, it is shown ICL does not help with the BLOOMZ model, which also demands further investigation on other LLMs.\n\n2. The re-ranking approach is somehow heuristic to me. The BART model is producing more chitchat outputs with lower information density. Using the perplexity of BART to rank LLM outputs will result in choosing output with less information, which is claimed to be more human-like. All validations and analyses are with BLOOM(Z) model which is shown to easily give conclusions. However, it is not clear how another model performs. Besides, from my experience, changing the prompts can also guide the model to be more careful before making medical conclusions/suggestions.\n\n3. Overhead computation and grounding knowledge. The plug-in-and-play model introduces additional overhead for retrieval and reranking. A discussion on this is needed. Meantime, it requires a medical conversation pool to retrieve the samples. Currently, the setting is to retrieve samples from the same dataset. It may lead to bias because similar dialogue styles and domains are covered. The generalizability to other data sources is not very clear.\n\n4. I have some questions which are not clearly explained in the current manuscript (or I missed them) and are elaborated on 'Question for the authors'.", "questions_to_authors": "1. What data is the SLM trained on? The dialogue corpus in the selected datasets? If so, it makes sense that it will select the response more similar to the datasets.\n\n2. The baseline model which is simply using SBERT and history to perform retrieval. What is the different with Global retrieval? And the performance simply with SBERT is very competitive and even outperforms all other methods in some cases. Then, what is the necessity for global retrieval and local retrieval?\n\n3. The new evaluation metric requires a set of medical terms. How the terms are collected? The quality of the terms. Does it has a wide coverage of different medical domains?", "paper_topic_and_main_contributions": "This paper presents a new approach to dialogue response generation in medical consulting scenarios.\n\nThe primary purpose is to enhance the pre-trained general-purpose Chat-LLM with in-context learning and response re-ranking methods. \nMoreover, two new metric scores for medical dialogue generation are proposed by measuring 1) dialogue intent and 2) specific medical terms.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "cyZKNuTD6R", "reviewer": "Reviewer_X2S1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The author only compares their method with 4 LLMs, and these LLMS are not state-of-the-art.**\n\nWe sincerely appreciate the reviewer's insightful feedback. We opted to utilize the Bloom family primarily due to its complete open-source nature. This choice stems from our conviction that conducting experiments on a fully open-source LLM yields enhanced interpretability. To assess the efficacy of our approach, we will complement the final version with experiments involving Chatgpt, thus confirming its generalizability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors a patient-centred medical dialogue model called PlugMed, which enhances LLMs to generate responses that conform to the diagnostic strategy.\n2. The authors propose two evaluation metrics, i.e., the intent accuracy and the high-frequency medical term accuracy, where the intent accuracy is used to evaluate the reasonableness of the dialogue actions adopted by the system, and the high-frequency medical term accuracy focuses on measuring the presence of essential medical information in the system’s responses.\n3. Experimental results indicate that the responses generated by PlugMed exhibit a greater inclusion of comprehensive medical terms and PlugMed yields more accurate dialogue intents than other large language model-based methods.", "reasons_to_reject": "The author only compares their method with 4 LLMs, and these LLMS are not state-of-the-art.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors propose a patient-centred medical dialogue model based on prompt learning from large language model (LLM) called PlugMed, which is equipped with a prompt generation (PG) module and a response ranking (RR) module to enhance LLMs’ dialogue strategies for improving the specificity of the responses.\nExperimental results indicate that the responses generated by PlugMed exhibit a greater inclusion of comprehensive medical terms and PlugMed yields more accurate dialogue intents than other large language model-based methods.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a patient-centered medical dialogue model named PlugMed, which leverages prompt learning from large language models (LLM). PlugMed is equipped with two key modules: a prompt generation (PG) module and a response ranking (RR) module. These components work in tandem to enhance LLMs' dialogue strategies, with the primary goal of improving response specificity.
The experimental results presented in the paper demonstrate that PlugMed outperforms other large language model-based methods by achieving more accurate dialogue intents.

However, I agree with one of the reviewers that this paper primarily combines established components into a unified system and lacks novelty. Another shortcoming of the paper is the absence of standard evaluation metrics commonly used for dialogue generation tasks, such as ROUGE or BLEU. These metrics are essential for assessing response quality and should be included to strengthen the paper's empirical findings.
Decision: Accept-Findings