HuatuoGPT, Towards Taming Language Model to Be a Doctor
Authors: Hongbo Zhang
Junying Chen
Feng Jiang
Fei Yu
Zhihong Chen
Guiming Hardy Chen
Jianquan Li
Xiangbo Wu
Zhang Zhiyi
Qingying Xiao
Xiang Wan
Benyou Wang
Haizhou Li
Venue: EMNLP 2023 Findings (2023)
Keywords: large language models
medical application
ChatGPT
TL;DR: A medical ChatGPT mainly for Chinese language

Abstract: In this paper, we present HuatuoGPT, a Large Language Model (LLM) for medical consultation. The core recipe of HuatuoGPT is to leverage both distilled data from **ChatGPT** and real-world data from **doctors**  in the supervised fine-tuning stage. This is not only because purely using **ChatGPT**-distilled data might cause 'model collapse', but also because real-world data from **doctors** would be complementary to **ChatGPT**-distilled data. The responses from ChatGPT are usually detailed, well-presented, fluent, and instruction-followed, but it cannot perform like a doctor in many aspects, e.g. for interactive diagnosis. Therefore, the extra doctors' data could tame a distilled language model to perform like doctors. To synergize the strengths of both data sources, we introduce RLMF (Reinforcement Learning from Mixed Feedback) where a reward model is trained to align the language model with the merits that both sources (ChatGPT and doctors) bring. Experimental results (in GPT-4 evaluation, human evaluation, and medical benchmark datasets) demonstrate that HuatuoGPT achieves state-of-the-art results in performing medical consultation among open-source LLMs. It is worth noting that by using additional real-world data and RLMF, the distilled language model (i.e., HuatuoGPT) outperforms its teacher model (i.e., ChatGPT) in most cases.

HuatuoGPT, Towards Taming Language Models
To Be a Doctor
Hongbo Zhang Junying Chen Feng Jiangl.2.3t Fei Yu!.2 Zhihong Chen
1,2.3
Jianquan Li Guiming Chen 1.2 Xiangbo Wu?, Zhiyi Zhang" Qingying Xiao',
Xiang Wan 12 Benyou Wang Haizhou Li
Shenzhen Research Institue of Big Data, 'The Chinese University of Hong Kong, Shenzhen UUiversity of Science and Technology of China
hongboz 1omoooz188mmmmalll com, junying. csêgmail, 1. com, Jeffreyiang@euhk.dde wangbenyou@cuhk, edu.cn
Abstract
output. Considering the confluence of experience- driven foundations in both LLMs and medicine, we believe that LLMs hold significant potential to im- prove medical consultation and diagnostic support.
In this paper, we present HuatuoGPT, Large Language Model (LLM) for medical consulta- tion. The recipe of HuatuoGPT is to lever- age both distilled data from ChatGPT and real- world data from doctors the supervised fine- tuning stage. This is not only because purely using ChatGPT-distilled data might cause 'model collapse", but also because real- worldworld data from doctors would he complementary to ChatGPT-distiin data. The responses from ChatGPT usually detailed, pellepresented, fluent, and nnstucctionfolllwwer
Learning from ChatGPT? Recent stud- ies (Taori al., 2023; Chiang al., 2023; Chen et al., 2023) show distilling from ChatGPT could equip language model with fluent chat capability in some common scenarios. However, ChatGPT, and even GPT-4, exhibit relatively poorer perfor- mance in vertical domains (such as medicine) compared to domain experts. One contributing factor to this phenomenon is that annotators for ChatGPT did not reach the expert level and lack of proficiency in medical knowledge that medical doctors have. Therefore purely distill a medical LLM from a poor teacher (i.c., ChatGPT), as popular general democratized LLMs did (Chiang et al., 2023; Chen et al., 2023). Moreover, purely using ChatGPT-distille data might cause 'model collapse' (Shumailov et al., 2023) and therefore limits the generalization of medical LLMs.
but it cannot perform like a doctor many aspects, e.g. for interactive diagnosis. Therefore, the extra doctors' data could tame a distilled language model to perform like doctors. To synergize the strengths of both data sources, introduce RLMF (Reinforcement Learning from Mixed Feedback) where reward model is trained to align the language model with the merits that both sources (ChatGPT and doctors) bring, Ex- perimental results (in GPT-4 evaluation, human evaluation, and medical benchmark datasets) demonstrate that HuatuoGPT achieves state-of- the-art results in performing medical consulta- tion among open-source LLMs. It worth not- ing that by using additional real-worke data and RLMF, the distilled language model (i.e., Hu- atuoGPT) outperforms its teacher model (i.e., ChatGPT) in most
problematic
Learning from Doctors? On the other hand, when a model only learns from real-world med- ical diagnosis dialogues (Wei et al., 2018; Xu et al., 2019; Xia et al., 2020: Zeng et al., 2020), it pro- duces overly concise responses and lacks the neces- sary level of detail due to the time constraints doc- tors face during the diagnostic process, leading condensed presentation of information. Compared to the responses from ChatGPT that are detailed and well-presented (e.g., with well-organized para- graphs, tables and bullets), doctors' responses are not friendly enough in terms of patience, level of detail, and logical organization.
1 Introduction
Medicine stands a paramount pillar in human existence, with its effectiveness heavily relying on the expertise and experience of professionals. Yet, the advent Large Language Models (LLMs) like ChatGPT heralds a transformative era for such experience-driven domains (Wang et al., 2023a). LLMs learn and mimic human language by lever- aging retrospective data and generating prospective
Philosophy of HuatuoGPT To overcome the above issucs, the core recipe of this work is to leverage both real-world data from doctors and
"Corresponding author. "The first three authors con- tributed to this work equally.

---

CharGIT vs. Doctor
User: 6, 7-ft (1 have a stamachacch.)
CharGiP:
8
Doctor: 1 12 51.0
2. (Where is your abdionuiaal patn? severe or mild? How lung has bern going on?)
A if 8t. have stomochachc. Stomcchaches can have verrous cunses, includinn digestive issucs,
1238 T
F
morry to hear you
infectionn, gostreintcttnil probleuss canset procide precise medical diognoris, the discomffrt of stomoc T & --g is help allevistc siomachuche)
frrith conditions. As language alleviate
"I M. A A. #3 72 # ahdommn, painfol, about half day.
User:
4 night,
give you somt
3
W.it 4-8 J. (Here some common methods that
Doctor:
12 (Are
# (Rest): is 2. #it 12. St 2 P. f2
(Gire yourself
there any accompanyyng comptons such as nauses, vomniing. or fever?)
rent, relox bodfy, and avoid physical activits)
J8 & (Wamn compeeee)) TT R. 5. 4 M& & A. $ 11 a (You placing water bottle lowel the stomuchcche.
User: Ri Bt vomited))
alleviate discomioon.)
Doctor: + #b. A 7 (Whar did night??
(Modical concutatioon: & AARRAFEE R. (2 f. de de 1. 2E
#5 12 7
(Eaning
come comzlishh night, rould canse
# Of your sssnachache persins worsens,
accompanied other Levere
sympttoss persistont vomiting, fewee, bloady soots, soon ai possible for professional
and treatment.
Doctor: $3 &
86 AR.) K.2 G 01
-2,tt R 1 &
& 1373885
*e highly paccible you have acate postrueniettil.
(Please remommber obow ruggestions for reforence only. If your synptoms persist or worsen, is best comnalt doctor for professional dinguoor and traimmnt advice.)
1::00
which often cansed by food potsoning. Crzyfih, in particulerr can contain bacteria and pararites that casily lead to gasteuu-- restinal infections.)
Example of ChatGPT responses (left) and doctor responses (right), where texts translated from Chinese to English. Questions raised by doctors are blue, and medical diagnoses are underlined. Note that ChatGPT usually does not raise questions in response to patients or provide medical diagnoses like doctors.
Figure 1:
distilled data from ChatGPT. We name our model HuatuoGPT' mainly designed for Chinese. Techni- cally, we propose two-stage training framework that fully leverages the strengths of ChatGPT and doctors. The complementarity between real-world medical data from doctors and distilled data from ChatGPT is further discussed in Sec. 2.
RLAIF (Bai et al., 2022). is used to reward gen- erated responses that are not only patient-friende (learned from ChatGPT with better presentation quality, detailed, dustruction-folloviii abilities, and fluent chat), but also doctor-like (learned from doctors with professional and interactive diagno- sis).
In the Supervised Fine-Tuning (SFT) stage, we leverage both real-world data from doctors and distilled data from CharGPT. The distilled data from ChatGPT used tame language models to follow medical instructions and talk fluently. The additional real-world medical data not only inject medical knowledge into language models but also tame the language models to perform medical diag- noses like a doctor. Moreover, the two sources of data are aligned with each other by role-enhanced promoting and ChatGPT polishing respectively. After the SFT stage, to further leverage the strengths of ChatGPT and doctors and meanwhile mitigate their weaknesses, we propose Reinforce- ment Learning with Mixed Feedback (RLMF), inspired by RLHF (Ouyang et al., 2022) and
Evaluation We propose a comprehensive eval- uation method that includes both manual and automatic evaluations for medical LLMs, cover- ing single-turn and multi-turn medical consulta- tions. The experimental results show that our Hu- atuoGPT outperforms existing open-source med- ical LLMs and ChatGPT automatic and man- ual evaluation. More impressively, our model sur- passes the performance of GPT-3.5-turbo terms of automatic evaluation and doctors' evaluation. Moreover, HuatuoGPT achieves state-of-the (SOTA) zero-shot performance in several medi- cal benchmarks such as CmedQA (Zhang et al., 2018), webmedQA (He et al., 2019), and Hu- atuo26M (Li al., 2023a) datasets. We will open- source our training data, code, HuatuoGPT model, and the reward model at https: //gi com/ FreedomIntell hwwpee/nnnnennnneeeeeeeeeeeeeeee
To commemorate the renowned Chinese physician Hua Tuo thttps://en wik www.pedia.aoooo wwt.pippd.....iiiiiieeeee www./oogg/////oooooo /tt///iii/aaauuuo Tuo).

---

Instruction Data
Conversation Data Real-world
Model
Language
Training Method
Distilled
Real-world
Distilled
ChatDuctor
English
SFT SFT
MEDALPACA
English
/
Visual Med-Alpacca
English
/
SFT
MedicalGPT-zih
Chinese
/
SFT
BenTsao
Chinese
/
SFT
DoctorGLM
Chinese
SFT
HuatuoGPT (Ours)
Chinese
SFT+RLMF
Table
Comparison of Data Sources and Training Method Across Popular Medical Models.
2 Motivation
Methodology
Following the above motivation, propose combine the strengths of both distilled data (from ChatGPT) and real-world data (from Doctors) tame the medical LIM to perform like a doctor, as illustrated in Table For example, is ex- pected to not only provide detailed, informative, and well-presented content but also conduct ac- curate and interactive diagnostics (usually posing clarifying questions) like doctors. To this end, our approach focuses integrating the charac- teristics of both doctor and ChatGPT enhance the quality of responses in medical consultations through a two-stage training strategy: our approach first hybrid distilled and real-world data in the su- pervised fine-tuning stage (SFT) and then employ the Reinforcement Learning from Mixed Feedback (RLMF) to further leverage the strengths of both data and meanwhile mitigate their weaknesses. The schematic of HuatuoGPT is shown in Figure 2.
As shown in Figure 1, the responses from ChatGPT and doctors are different but complementary.
ChatGPT Responses Although ChatGPT usu- ally generates informative, well-presented and log- ical responses, ChatGPT does not perform like doctors that conduet interactive diagnosis. For example, when a user provides too little informa- tion to make a diagnosis decision, it usually enu- merates multiple possibilities and provides general yet unprofessional advice while doctors usually ask clarifying questions before giving advice.
Doctors' Responses Doctors are adept at inquir- ing about the symptoms and providing accurate diagnoses. Their responses typically exhibit pro- fessionalism that meets the personalized consulta- tion. However, due to limited time inquiry, their replies often informal and concise in nature, and sometimes incoherent. Our preliminary study shows that training from purely patient-doctor in- teraction data is not satirized: (1) it cannot fluently follow diverse instructions; (2) the responses are short, poorly presented, and sometimes uninforma- tive, which are not patient-friendly.
3.1 SFT with Hybrid Data
In the first stage, we employ a blend distilled data and real-world data, capitalizing on both strengths to endow the model with Doctor-like and Patient- friendly characteristics. Within each data category, we collect instruction data and conversation data to imbuc the model with the capacity for instruction- following and interactive diagnosis. It notewor- thy that all of the real-world data use is pub- licly available. The single-turn real-world data is sourced from Huatuo26M (Li al., 2023a), and the multi-tum turn data is sourced from Med-dialog (Zeng et al., 2020).
Complementarity between ChatGPT and Doc- tors By distilling from ChatGPT, the model could generate informative, well-presented, and logical responses. Conversely, real-world data, harvested from authentic doctor- pattent interactions, provide an indispensable perspective into the complexities of actual medical scenarios. The primary strength of real-world data lies in their high accuracy and professionalism. Therefore, sponses from ChatGPT and Doctors could be com- plementary; we expect a model to not only chat fluently like ChatGPT but also behave like doctors.
Distilled Data from ChatGPT We collect syn- thetic instructions and conversations from Chat- GPT using different manners. Following the work of self-instruct (Wang et al., 2022; Taori et al., 2023; Chen et al., 2023), we construct a set
believe that re-

---

Real-world Data (from Doctors)
Question Answers
Response
Diagnostic Conversations
ChatGPT
Doctor
Feedback
LLM-polished
Hybrid Data
HuatuoGPT
1. SFT
2. RLMF
Doctor-like
Role-enhanced
SFTed model
Patient-friendly
Medical Instructions
PO0
Instructionffolloiing
Medical Conversations
Interactive diagnostic ?
Distilled Data (from ChatGPT)
Figure 2: Schematic of HuatuoGPT.
medical instruction data to enable the model to fol- low the user's medical instructions. Especially, we have employed top-down manner to create more natural and comprehensive responses: design a taxonomy to collect or manually create seed in- structions based on the roles and use cases, and generate instructions separately using self-instruct. It provides a wide range of instructions and, mean- while, keeps enough instructions for each role and use case. More details are shown in App. A.1.
soning and raising questions to guide patients in describing their symptoms. However, raw data is unsuitable to be fed directly to LLMs because they usually very colloquial and contain noisy infor- mation such as typos, slang, and irrelevant informa- tion. To mitigate this, we utilized language models to polish the original content. Our goal with this polishing process is to make the doctors' responses well-presented detailed, and more patient-friendly. For additional specifics, please see App.A.3 and App.A.4.
For the distilled conversation dataset, we em- ploy two ChatGPTs, each assigned specific role-either as a doctor or patient. Initially, we source real-world data as foundation for medical knowledge, using it to inspire synthetic dialogue. The conversation synthesis involves alternating di- alogue utterances generated by the two ChatGPTs. The patient ChatGPT derives its responses from background patient information, while the doctor ChatGPT only references final diagnosis. It's worth noting that the doctor ChatGPT doesn't 't have access to the patient's data, and vice versa. The dialogues produced by these large language mod- els (LLMs) tend to be informative, detailed, well- structured, and stylistically consistent. Additional information be referenced in App. A.2.
3.2 RL with Mixed Feedback
In the Supervised Fine-Tuning (SFT) phase, we introduced diverse datasct to enable HuatuoGPT to emulate the inquiry and diagnosis strategy doctors while maintaining the rich, logical, coher- ent characteristics of LLMs' responses. In order to further align the model's generation preferences to needs, propose reinforcement learning with mixed feedback improve the quality of the mod- els' responses. Previously, OpenAI introduced rein- forcement learning with human feedback (Ouyang et al., 2022) to align LLMs with human preference but at a significant cost of time and labor. Bai et al. (2022) demonstrated that with a carefully de- signed prompt, AI could imitate human preferences and give relatively consistent scores generated responses. Inspired by these alignment methods, design new pipeline by considering different sources of feedback to force the model generate informative and logical responses without deviat- ing from the doctor's 's diagnosis.
Real-world Data from Doctors Real-world data consists of crestion-answering and conversations between doctors and patients from real scenarios, where doctors' responses are professional, with high conciseness and relevance the patient's questions. Especially in multi-turn medical con- sulting conversations, doctors' responses often de- mand diverse abilities, including long-range rea-

---

Reward Model We employ reward model trained on mixed feedback to align with the traits of both doctors and ChatGPT. Our training data com- prises authentic instructions and dialogues, from which we extract multiple responses using our re- fined model. In the case of multi-turn interactions, the dialogue history is incorporated to streamline our model's response generation. Subsequently, we prompt ChatGPT to evaluate different re- sponses. The scoring isn't solely based on fac- tors like informativeness, coherence, and alignment with human preferences; also weighs the fac- tual accuracy by comparing them against responses from real doctors. The paired responses receive scores from ChatGPT, which are then incorporated into the reward model. The details for ChatGPT preference scoring
4.1
Evaluation Protocol
4.2
Baselines
We select the following three groups of models as baselines: (1) Two popular general models: ChatGPT (GPT-3.5-turbo) and GPT-4 2, which are the strongest language models in the gen- eral domain; (2) Two open-source general mod- els specialized in Chinese: ChatGLM-6B (Zeng el al., 2023) and Ziya-LLaMA-I3B", (3) Two most representative operesource Chinese medical mod- els: BenTsao (Wang et al., 2023b) and Doctor- GLM (Xiong et al., 2023).
4.3 Pairwise Evaluation by GPT-4/Doctors
We comprehensively evaluate HuatuoGPT perfor- mance by pairwise evaluation with it and other baselines single-turn questions and multi-tum conversations.
be found in App. B. To account for potential positional bias in ChatGPT, each evaluation is conducted twice, with data po- sitions swapped. Only selections that consistently achieve high scores both positions are taken into consideration.
For the single-turn questions, we extract 100 questions representing 10 intents (shown in Ta- ble 4 in App. G) from the validation set of the Knowledge-basee Universal Automated Knowl- edge Extraction for Query Intent Classification (K (KUAKE-IIC) in Chinese Biomedical Language Understanding Evaluation (CBLUE (Zhang et al., 2021))². KUAKE-QUE is collected from search engine queries, which makes it suitable for single- turn questions. To filter noisy data, these questions are initially scored by ChatGPT, and manual fil- tering process is conducted to select higher-quality candidate questions for the test set.
Reinforcement Learning In the RL process, we sample & different input samples (x1...., } and generate responses (.i,..., yk3 } by current policy Each sample y/ is fed to our reward model to provide a reward score rnas- To ensure that the model does not deviate too far from the initial state no. add the empirically-estimated KL penalty term(Schulman et al., 2017), and the final reward function is as follows:
TRM DKLDDL
(1)
For the multi-turn conversations, we utilize the patient cases from Med-dialog (Zeng et al., 2020), select 20 departments (shown in Table 5 in App. G) and randomly sample 5 patient cases from each de- partment, resulting a total of 100 real patient cases. These cases are provided to ChatGPT, which plays the role of the patient, interacting with each doctor model to obtain the diagnosis results. When eval- uated by doctors, we randomly sample 50 patient cases from 100 test cases.
where ÀKL is a hyperparameter for KL penalty, DKL is the KL penalty function. Input queries de- de-duplicated and sampled from the remaining SFT hybrid data. This ensures a diverse range of inputs while retaining the model's response preferences both the single-turn instruction and the multi- turn conversation scenarios. The training details of HuatuoGPT can be seen in App. F.
In both single-turn question and multi-turn con- versation scenarios, uation as metrics compare responses generated from different models.
4 Experiments
GPT-4 and human eval-
In this section, we first introduce the baselines and then present the evaluation protocol and results in- cluding pairwise evaluation by GPT-4/Doctors on single-turn questions and multi-turn conversations. Considering the small scale of pairwise evaluation, we also evaluate our model on the lcrge med- ical QA benchmark to validate HuatuoGPT.
We use the gpt-3.5-ttt t-rrbo and gpt-4-8314 APL. httpp:://uuggingface Ctt.://cc.../DACCCCLLLLL Ziya- LLya-LLaMM-13B-11
hhttt::://uuggnggface thinksoso/
lora- Kora-11ama-mn 1lama-med
hhttps: / thub. http.////BLEEEBeecaarr///UL

---

Ours Win
Tic
Ours Loss
Ours Win
Tic
Ours Loss
DoctorGLM
DoctorGLM
Ziya LLaMA 13B
Zäya LLaMA
CharGLM-6B
43
18
ChatGLM-68
27
34
ChatGPT
47
32
ChatGPT
GPT-4
31
52
GPT-4
22
24
0%
20%
60%
80%
100%
20%
40%
80%
100%
(a) GPT-4 Evaluation
(b) Doctor Evaluation
Figure 3: Response Comparison of HuatuoGPT with Other Baselines on the Single-tum Question.
Ours Win
Tic
Ours Loss
Ours Win
Tic
Ours Loss
DoctorGLM
97
21
DoctorGLM
96
ZYy-LLaMA-131
75
22
Ziy--LLaMA-138
-42
28
M-66
ChatGLM-68
CharGPT
62
ChatGPT
16
0%
40%
80%
100%
60%
100%
(a) GPT-4 Evaluation
(b) Doctor Evaluation
Figure 4: Response Comparison HuatuoGPT with Other Baselines on the Multi-tur turn Conversations.
GPT-4 Evaluation We prompt GPT-4 con- sider the following aspects of responses: doctor-
4.4 Medical QA Benchmarks
We select three existing Chinese medical QA datasets, namely cMedQA2 (Zhang et al., 2018), webMedQA (He et al., 2019) and Huatuo- 26M (Li et al., 2023a). cMedQA2 is publicly available dataset based on Chinese medical ques- tions and answers consisting of 108, 000 questions and 203, 569 answers. webMedQA is real-world Chinese medical QA dataset collected from online health consultancy websites consisting of 63, 284 questions. Huatuo-26M (Li et al., 2023a) the largest Chinese medical QA dataset which has 26M QA pairs from online medical consultation, knowl- edge bases and encyclopedias. Following the pre- vious works (Li et al., 2023a), we utilize metrics such as BLEU, ROUGE, GLEU, and Distinct.
like language, symptom inquiry capability, the ef- feet and reliability of the treatment recommenda- tions and prescriptions, and helpfulness. Given the question and the two corresponding responses from different models, GPT-4 is asked to first compare the advantages of each response and then select the better response. Recent studies show that GPT-4 has high agreement with human judgment (Zheng al., 2023; Chen et al., 2023; Li et al., 2023b), but there no study on the agreement between GPT-4 and expert doctors. Concerned about the impact the response position (Wang et al., 2023c), we randomly switched the position of two responses.
4.5 Experimental Results
Doctor Evaluation We invite three doctors to compare responses from pairwise models accord- ing to the guidelines: (1) diagnosis accuracy, (2) treatment recommendation accuracy, and (3) med- ication knowledge and prescription accuracy. In evaluation, doctors asked provide assess- ments of different responses generated by two mod- els for the same context. We ensure that each re- sponse data is scrambled and anonymized with the utmost strictness. See App. F for details.
Results on Single-turn Questions For the evalu- ation of the single-turn questions, all the model per- formance results are shown in Figure 3 and the com- parison among models for cach category is shown in Table 4 in App. G. According to the evaluation of GPT-4 and doctors, HuatuoGPT outperforms all baselines except GPT-4. From Table 4, HuatuoGPT is much better than BenTsao and DoctorGLM all categories. Compared with Ziya-LLaMA- -133B

---

ChatGLM and ChatGPT, HuatuoGPT stands out in terms of efficacy, medical expense and indica- tors interpretation. However, HuatuoGPT is still worse than GPT-4 in many categories, where attains similar performance to GPT-4 two cat- egories (Efficacy and Medical Expenses). Such results confirm the validity of our approach, and HuatuoGPT can receive more preference compared other models.
0.8
Questions Mulli-um Conversatonn
0.7
0.66
0.645
0.64
rate
0.501
"1:47
0.45
0.4
0.585
0.3
0.2
HuatuoGPT
(w/o RLMF) (w/ dist)
(w/ rea))
Model
Results on Multi-turn Conversations As shown in Figure 4, HuatuoGPT performs exceptionally well against all models in multi-turn conversations, which reveals that HuatuoGPT excels in extended dialogue contexts, evidenced by a 97% win rate against DoctorGLM and 62% against ChatGPT, This result is made possible by our conversation data, which comprise informative synthetic conver- sation data, and real conversation data that contains the diagnostic capability of doctors. From Table 5 in App. G, it shows that HuatuoGPT outperforms other models in almost all departments. This again confirms the effectiveness of approach: Hu- atuoGPT has a more prominent interactive diag- nostic capability in patient consultation scenarios and its performance preferred by both GPT-4 and doctors.
Figure 5: The win-rate of our ablation models com- pared with ChatGPT, with ties evenly divided between wins and losses. (w/o RLMF) represents HuatuoGPT without RLMF, while (w/ dist) and (w/ real) denote Hu- atuoGPT (w/ distilled data) and HuatuoGPT (w/ real data), respectively.
evaluation methodology used previously, we com- pare all four models ChatGPT in both single and multi-turn situations and employ GPT-4 score the responses of a pair of models. The results of the evaluation are shown in Figure 5. The results show that HuatuoGPT (w/ real data) performs better in multi-turn conversations while HuatuoGPT (w/ distilled data) performs better in single-turn ques- tion answering. It confirms that distilled data from ChatGPT bring more of boost the model in single-turn situations, allowing the model bet- ter follow instructions and provide more complete answers to a single question. In contrast, real data from doctors can bring more of a boost to the model in multi-turn conversations, allowing the model to better learn from doctors to perform the interactive diagnosis, obtain patient information and provide diagnosis in multi-turn conversations. Compar- ing HuatuoGPT (w/o RLMF) with HuatuoGPT, can be seen that the RLMF process further im- proves the performance of the model in both single- turn and multi-turn cases, widens the gap in per- formance comparing to ChatGPT. This suggests that the RLMF process can pull the model further toward being both
Results on Public QA Benchmark HuatuoGPT demonstrates impressive performance across vari- ous Chinese medical benchmarks, achieves consis- tently high scores across all metrics, and demon- strates a high level of accuracy, fluency. and di- versity in its generated responses. In cMedQA2, HuatuoGPT even outperforms fine-tuned T5, sug- gesting that it has robust generalization capability and is able effectively handle wide range of medical question question-answering answering tasks.
4.6 Ablation Study
In this section, we delve into an exploration of how two distinct types of data, real-world and distilled, along with the RLMF process, influence model. To do so, trained three distinet models. The first, HuatuoGPT (w/ real data), exclusively utilizes real-world data. The second, HuatuoGPT (w/ dis- tilled data), solely relies on distilled data. Lastly, we have HuatuoGPT (w/o RLMF), a model that does not incorporate the RLMF process.
and doctor-like
Case Study We compare the variations in re- sponses between four models for the same set of questions shown in Table 9 in App.G. HuatuoGPT (w/ real data) has tendency to ask clarifying ques- tions to patients. performing as expected, similar doctor. However, minor flaw is that the question is superficial and appears less well-organized for reading, On the other hand, HuatuoGPT (w/ dis- tilled data) generates well-organized, detailed, and
GPT-4 Evaluation In order investigate the in- fluence of different types of data, we perform au- tomated evaluation experiments. Following the

---

Dataset
Model
BLEU-1 BLEU-2
BLEU-3
BLEU-4
GLEU
ROUGE-I ROUGE-2
ROUGE-L Distinct-1 Distinct-2
T5 (fine-tuned)
20.88
11.87
7.69
5.09
7.62
27.16
9.30
20.11
0.41
0.52
DoctorGI.M
13.51
7.10
3.72
2.00
5.11
22.78
5.68
12.22
0.85
0.96
ChatGPT
19.21
7.43
3.14
1.24
5.06
20.13
3.10
12.57
0.69
0.99
cMedQA2
ChatGLM-6B
24.90
12.74
6.99
3.87
8.49
28.52
7.19
18.21
0.68
0.99
Ziya-LLaMA-138
27.03
13.87
7.48
4.09
7.77
7.10
14.81
HuatuoGPT
27.39
14.38
8.06
4.55
8.52
29.26
8.02
15.46
0.74
0.93
15 (fine-tuned)
21.42
13.79
10.06
7.38
8.94
31.00
13.85
25.78
0.37
0.46
DoctorGI.M M
9.91
5.20
2.78
1.54
4.67
23.01
5.68
11.96
0.84
0.95
ChatGPT
18.06
6.74
2.73
1.09
4.71
20.01
2.81
12.58
0.65
0.87
webMedQA
ChatGLM-6B
23.42
12.10
6.73
3.83
8.04
28.30
6.87
18.49
0.63
0.87
Ziya-LLaMA-I33B
22.16
11.70
6.53
3.74
6.91
27.41
6.80
13.52
0.76
0.93
HuatuoGPT
24.85
13.42
7.72
4.51
7.50
28.30
7.72
14.50
0.73
0.93
15 (fine-tuned)
26.63
16.74
11.77
8.46
11,38
33.21
13.26
24.85
0,51
0.68
DoctorGLM
11.50
6.00
3.14
1.69
4.65
22.39
5.47
12.14
0.85
0,96
Huatuo-26M ChatGPT
18.44
6.95
2.87
1.13
4.87
19.60
2.82
12.46
0.69
0.89
ChatGLM-6B
24.46
12.75
7.20
4.13
8.50
28.44
7.31
18.58
0.67
0.89
Ziya-LLaMA-II3.
25.58
13.39
7.46
4.24
7.30
28.14
7.18
14.78
0.77
0.93
HuatuoGPT 27.42
14.84
8.54
4.96
8.01
29.16
8.29
15.84
0.74
0.93
Table Benchmark on Chinese medical QA dataset (Li et al., 2023a). and HuatuoGPT are zero-shot setting while T5 is fine-tuned detailed from the original paper.
ChatGPT, Ziya-LLaMA-I33 133, ChatGLM-6B.
informative content. Nevertheless, its responses tend to provide suggestions rather than making a diagnostic decision. To assess the impact of RLMF, we also compare HuatuoGPT and HuatuoGPT (w/o RLMF). It is worth noting that HuatuoGPT (w/o RLMF) does not ask additional questions to patients. This might be attributed to the fact that its training data could be biased towards the Chat- GPT data, while real-world data may have been overlooked. In contrast, our default model, Hu- atuoGPT, can function like a doctor by asking follow-up questions to patients to get more accu- rate diagnoses.
data from ChatGPT (GenMedGPT-5k and disease database). Based on the dataset of ChatDoctor, MEDALPACA (Han et al., 2023) added more data points across a diverse range of tasks, including openly curated medical data transformed into Q/A pairs with ChatGPT and collection of established NLP tasks in the medical domain. Different from the previous work, Visual Med-Alpaca' is only trained on the distilled instruction dataset while the 54K samples are filtered and edited by human ex- perts. Recently, Med-PaLMM2 (Singhal et al., 2023) was published, which is based on PaLM2 and fine- tuned in MultiMedQA for expert-Level medical question answering.
5
Related Work
In Chinese, BenTsao (Wang et al., 2023b) is knowledge-ehhanced Chinese Medical LLM trained on over 8K instructions generated from CMeKG by ChatGPT without real-world data. MedicalGPT-zh (Liu et al., 2023) is a Chinese medical model based on ChatGLM-6B LoRA with 16-bit instruction fine-tuning. The training dataset was obtained from Chinese medical knowl- edge peestion-and-answee pairs and clinical guide- line texts from 28 medical departments. Doctor- GLM (Xiong al., 2023) is Chinese medical ChatGLM- LLM trained on multiple medical datasets. In addition to the distilled instruction data from ChatDoctor through translation, they incorpo- rate some Chinese medical conversation data into the training dataset. See App. H for more details
The medical language model has always been a con- cern for researchers. The early models were mainly based on the GPT-2 series models to continue pre- training in the domain, such as BioMedLM' and BioGPT (Luo et al., 2022). Recently, the success of distilled from ChatGPT (Taori et al., 2023; Chiang et al., 2023) has stimulated some efforts to fine-tune large-scale language models using medical--elated instruction data.
ChatDoctor (Li et al., 2023c) is medi- cal model based on LLaMA (Touvron al.,
2023) trained on the real-world instruction data (HealthCareMagii- 100k and icliniq 10k) between patient and physician and the distilled instruction
"https: //www mosai cml con/blog/ introducing settroducin--uumeed-ee pubmed-gua
Ptttps: //gi thub. wttpccaoooiiiecceee com/canbridgeetl/ visual-mest med- alpaca

---

on medical LLMs.
scholarly community's diligence in upholding these stringent standards.
6 Conclusion
Compensation and Ethical Collaboration with Medical Professionals We collaborated with three esteemed doctors, including a chief physi- cian, whose expertise and insights were invaluable to our work. Each of these doctors was compen- sated at a rate of 200 RMB per hour for evaluations, culminating in a total of 800 RMB. This compen- sation rate aligns with the prevailing local salary standards and has been determined to ensure that their specialized knowledge and time are fairly munerated.
This paper presents HuatuoGPT, medical large language model, which leverages hybrid data and incorporates reinforcement learning from mixed feedback with ChatGPT and doctors. Gathered merits distilled data from ChatGPT and real-world data from doctors, HuatuoGPT is able to gener- ate patient-friendly and doctor-like responses, By refining its responses based mixed feedback, the model can improve its conversational abilities while maintaining the reliability necessary for med- ical applications. The experimental results of au- tomatic and manual evaluations indicate that our model surpasses the existing medical open-source LLMs and even outperforms ChatGPT in most cases. As we continue to explore and develop this approach, we believe that further research in this area holds significant potential for advancing the field of Al in medieine.
Concerning Accurateness of Medical Advice We emphasize the potential risks associated with generation- medical consultation. The main concern lies the challenge of verifying the accu- racy and correctness of the generated content. the medical domain, the dissemination of mislead- ing information can have severe ethical implica- tions. Although generative QA has shown promise, especially with the success of models like Chat- GPT, they are not yet prepared for real-world deployment in the biomedical domain.
Limitations
Knowledge-intensive Benchmarking Recent studies tend to evaluate large language models us- ing knowledge-intenes
While generation methods currently hold great potential, is important to exercise caution and prudence before deploying them in real-world ap- plications. Further research and development are necessary to refine these models, enhance their accuracy, and establish robust mechanisms for accurateness-checking and error correction. Only through scrutiny and continual improvement can we minimize the risks and ethical concerns associ- ated with generation based medical QA.
tasks e.g. multiple-choice questions (Hendryeks et al.. 2021; Huang et al.. 2023). We argue that such benchmarking perfor- mance highly depends on the foundation models, which is unaffordable for us in terms of computing resources. Therefore, we leave it for future work.
HuatuoGPT for more Languages Our models are specific to Chinese, where healthcare inequality in China is a significant issue. We wish it could leverage high-quality medical resources in first-tier cities and then distribute via online HuatuoGPT rural areas. We will release new versions in other languages following the same philosophy of HuatuoGPT.
Autonomy We respect the principle of auton- omy for AI. The decisions made by our LLMs are not intended to replace doctor decision decision-making pro- cesses but rather to augment doctor capabilities and offer additional insights.
To demonstrate that the effectiveness of our pro- posed method is independent of the scale of the backbone, we also used Bloom-7BI 1-mt as the back- bone for the internal evaluation for comparison, shown in Table 10 in App. J. We acknowledge its license and won't 't release the Bloom Bloom-based model.
Ethic Statement
Data Collection In our study, we relied solely on publicly accessible datasets and did not utilize any in-hospital data. Details can be found in the Methodology section. To address potential privacy concerns, meticulously reviewed the dataset ensure the absence of any private or sensitive information. Our commitment data privacy and safety is unwavering. We greatly value the ethical handling of information and are grateful for the
A Acknowledgements
This work is supported by Chinese Key- Area Research and Development Program Guangdong Province (202080101350001), the

---

Shenzhen Science and Technology Program UCCJ)202081810300001
Tatsunori B. Hashimoto. 2023b. Alpacaeval: An automatic evaluator of wow.utttioo-oooooinngg models. https://g //gi wwwb.com/tee..... tatsu- plpac_eeall.
the Guangdong Provincial Key Laboratory of Big Data Com- puting, The Chinese University of Hong Kong, Shenzhen, Shenzhen Key Research Project (C10120230151) and Shenzhen Doctoral Startup Funding (RCBS20021008003300000
Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, and You Zhang. 2023c. Chatdoctor: A medical chat model fine-tuned on Ilama model using medical do- main knowledge.
Hongcheng Liu, Yusheng Liso, Yutong Meng, Yu Wang, and Yanfeng Wang. 2023. Medicalgpt-zh: $ * E $ ot ih is ş H E. https::/githuue com/
References
Medi SeBrain-STTMMMddcallC.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint
Renqian Luo, Liai Sun. Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie- Yan Liu. 2022. BioGPT: generative pre-trained transformer for biomedical text generation and mining.. 23(6).
arXiv:2212.08073.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Zhihong Chen, Feng Jiang, Junying Chen, Tiannan Wang, Fei Yu, Guiming Chen, Hongbo Zhang, Juhao Liang, Chen Zhang, Zhiyi Zhang, et al. 2023. Phoenix: Democratizing chatgpt across languages. arXiv preprint arXiv:2304. 10453.
Wei-Lin Chiang. Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.032
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Com- puting. Networking, Storage and Analysis, pages 1- 16. LEEE
Tianyu Han, Lisa C. Adams, Jens-Michalis Papaioan- nou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K. Bressem. 2023, Medalpaca- open-source collection of medical conversational ai models and training data.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint arXiv: 1707.06047
Junqing He, Mingming Fu, and Manshu Tu. 2019. Ap- plying deep matching networks to chinese medical question answering: study and dataset. BMC med- ical informatics and decision making, 19(2)911100..
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Paperot, and Ross Anderson. 2023. The curse of recursion: Training on generated data makes models forget.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Con- ference on Learning Representations (ICLR).
Karan Singhal, Tao Tu, Juraj Goutweis Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl. Heather Cole-Lewis, Darlene Neal. Mike Schaeker- mann, Amy Wang. Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong. Christopher Sem- turs, S. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. 2023. Towards expert-level medical question answering with large language models.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C- eval: A multi-level muult-disccpllin chinese evalu- ation suite for foundation models. arXiv preprint arXiv:2305.0832..
Jianquan Li, Xidong Wang. Xiangbo Wu, Zhiyi Zhang, Xiaolong Xu, Jie Fu, Prayag Tiwari, Xiang Wan, and Benyou Wang. 2023a. Huatuo-26m, a large- scale chinese medical qa dataset. arXiv preprint arXiv:2305.01526.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction followwng Ilama model. https:// // gi thub. ctw/tatsu tatsu-lab Lanford_alpaca
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang. Sicheng Wang, Ruisi Zhang. Meng Zhou, Jiaqi Zeng. Xiangyu Dong, Ruoyu Zhang, et al. 2020. Med- dialog: Large-scale medical dialogue datasets. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9241-9250.
Benyou Wang. Qianqian Xie, Jiahuan Pei, Zhihong Chen, Prayag Tiwari, Zhao Li, and Jie Fu. 2023a. Pre- trained language models in biomedical domain: A systematic survey, ACM Computing Surveys, 56(3):1- 52.
Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang. Lei Li. Xin Shang. Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, al. 2021. Cblue: A chinese biomed- ical language understanding evaluation benchmark. arXiv preprint urXiv:2166.08087
Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao. Bing Qin, and Ting Liu. 2023b, Hu- atuo: Tuning Nlama model with chinese medical knowledge.
S. Zhang, X. 7hang. H. Wang. L. Guo, and S. Liu. 2018. Multi-scale attentive interaction networks for chinese medical question answer selection. IEEE Access. 6:74061-74077
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023c. Large language models are not fair evaluators. arXiv preprint 179..
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing. Hao Zhang. Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi. and Han- naneh Hajishirzi. 2022. Self-instrue Aligning lan- guage model with self generated instructions. arXiv preprint arxiv 22 02. 10560.
Zhongyu Wei, Qianlong Liu, Baolin Peng. Huaixiao Tou, Ting Chen, Xuan-Jing Huang, Kam-Fai Wong, and Xiang Dai. 2018. Task-oriented dialogue system for automatic diagnosis. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), pages 201- 207.
Yuan Xia, Jingbo Zhou, Zhenhui Shi, Chao Lu, and Haifeng Huang. 2020, Generative adversarial reg- ularized mutual information policy gradient frame- work for automatic diagnosis. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1062-1069.
Honglin Xiong, Sheng Wang, Yitao Zhu, Zihao Zhao, Yuxiao Liu, Qian Wang. and Dinggang Shen. 2023. Doctorgim: Fine-tuning your chinese doctor is not herculean task. arXiv preprint arXiv:2304.01097
Lin Xu, Qixian Zhou, Ke Gong, Xiaodan Liang, Jian- heng Tang, and Liang Lin. 2019. End-to-end knowledge-routed relational dialogue system for au- tomatic diagnosis. In Proceedings the AAAI con- ference on urtificial intelligence, volume 33, pages 7346-7353.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma. Yufei Xue, Jidong Zhai, Wenguang Chen. Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations (ICLR).

---

A Methodology details
be something that the GPT language model can accomplish, unable to generale and draw images, unable to read audio and webpage links; Instruc- tions should be 1-2 sentences in length, which can be either command sentences or interrogative sentences; Instructions usually have placeholder, placeholder, such as "this below" or "some", and the "input" field will be specified.
A.1 Distilled Instructions from ChatGPT
Deriving from previous work, we use self- instruction to generate the instructions from ChatGPT with the medical seed instructions we manually build and well-designed roles or use cases. The prompt is shown below:
3. The input should be specific example of the in- struction, providing real substantive content, as the instruction may be empty and need to be qualified with a specific input. The input should not be just a link or file name, or an unspecified 'paper', but rather specific content. It is recommended to input no more than 200 words.
2 # メ>Σオル - = A # (8 $
* 8 t
4. The roles, instructions, and inputs are mostly in Chinese, and the roles, instructions, and inputs should not be repeated. Instructions are mandatory, please provide roles and inputs as much as possible. Input can be empty
(8]
2
The list of 20 triples is as follows:
Different from the original self-instruction, we generated role-enhanced instructions and they will be used to generate the output with the following prompt.
3. 4 69
2 #
Role]. . # #
B m
14 A 6288 d *
220
4. 1
# Ag. #a
{Question) # 2, $
1 $ /
Translation:
Assuming you are an experienced [Medical Role], you will provide very patient and comprehensive answers to patients, and your tone will be gentle and friendly, which is very popular with patients. If the patient does not provide you with enough information to make a judgment, you will ask them relevant questions. And at the end of the diagnosis, you will also give him some additional advice. If the patient asks: Question you will answer:
Translation:
You are asked to design 20 different triplcts of<chaaacters. instructions, input>. The first line is the character, the second line is the instruction that the character wants GPT to help him improve his work efficiency, and the third line is the corre- sponding input for the instruction. requirement: 1. The role can be very specific and needs to be related to the medical scene. If it is a doctor, it can even be refined to the medical department, such as "respiratory physician".
A.2 Distilled Conversations from ChatGPT
We show prompts used for patient LLM and doctor LLM. Prompt for patient LLM:
2. The description of each instruction should be diverse, and the types of instructions should be diverse. Verbs should be avoided as much as possi- ble to maximize diversity. Each instruction should
- Th HuatuoGPP # &
#

---

A.3 Real-world Instructions from Doctors
- -
In the experiment, we collect real-world question answering data from web and sample set of high quality question-answerinn pairs used for training. Every pair is refined by LLMs. The prompt is shown below:
1
Translation:
You are a patient, and here is your condition, you are consulting the HuatuoGPT AI doctor about the relevant conditions of your illness. Please remem- ber that this is multi-round consultation process, each inquiry should be more refined, and the first inquiry should be as simple and as little content as possible.
+ # p} : $(Patient_Question)
< X $ (DocttrrRRsooossee
f Aे & 87
_ . / $
((medical_case }
$ ビ
When you think the whole consultation should be over, please say: Goodbye.
<HuatuoGPT ;>:
Translation:
Prompt for doctor LLM:
<Patient Question>:$ IilPaaient_uussttiiiee
<Response Reference>:$ [DDccoo__esponntttt
- 4 J 4
You are HuatuoGPT Al model, based on the pa- tient's question, please refer to the response and then give a reply to the patient's question, speak like doctor and have gentle and kind tone, re- ply to the patient's 's query in more detailed and helpful way, and ask the patient for more informa- tion if necessary if you cannot clearly diagnose the patient's disease.
 # * 4
#.
<HuatwoGPT Response>:
t0
$
A.4 Real-world Conversation from Doctors
[ddottrrddiagnosi..
The same as real-world instructions. Real-world conversations have to be refined by LLMs. The prompt is shown below:
Translation:
You are an experienced doctor who gives patient and comprehensive answers to patients. You speak like a doctor, and your tone is gentle and kind, which is popular with patients. Your responses to patient inquiries should be more detailed and helpful. If the patient does not provide enough information for diagnosis, you should ask them related questions to get more information for di- agnosis. After making diagnosis, you will also give them some additional detailed advice. Please note, you can only receive the patient's 's description and cannot view materials attachments such as images.
B - A f
, A4 & =
17 3
2
-
$[dialogue}
#
*
If you cannot make clear diagnosis, please ask the patient for more information about their condition. The final diagnosis results can be:
Translation:
You are now a medical language model and will provide you with a dialogue between a patient and a doctor. Please modify the following multi-round

---

dialogue in the same way doctor into dialogue between a patient and medical language model. The refined dialogue needs to meet the following requirements:
D Prompt for Auto Evaluation
The prompt for auto evaluation shwon as below:
[Assistant 1]
1. Be more informative, but do not change the diagnostic information.
$ [Response_A]
[End of Assistant 1]
2. Preserve the logic and order of the dialogue.
[Assistant 2]
3. Modify the doctor's information that appears in the dialogue. The doctor now medical language model, which does not have affiliated hospital and personal information.
(Response_B: _B]
[End of Assistant 2]
[System]
We would like to request your feedback on two multi-turn conversations between the Al assistant and the user displayed above.
The content of the dialogue is shown below: $ ((dialogue)
Please give the modified dialogue directly, in strict accordance with the original format and order:
Requirements: Focus the Al's response in the conversation. The Al assistant should act like the doctor using the tone, manner, and vocabulary the human doctor would use. It should be to the point, without unnecessary elaboration or extraneous in- formation. The Al assistant should respond ap- propriately to the user in manner that helps to progress the conversation. The description of symp- toms should be comprehensive and accurate, and the provided diagnosis should be the reason- able inference based on all relevant factors and pos- sibilities. The treatment recommendations should be effective and reliable, taking into account the severity or stages of the illness. The prescriptions should be effective and reliable, considering indica- tions, contraindications, and dosages. Please com- pare the performance of the Al assistant in each conversation. You should tell me whether Assistant is 'better than', "worse than', sistant 2. Please first compare their responses and analyze which one is more in line with the given requirements.
B Prompt for Mixed feedback
The prompt for Mixed feedback shown as below:
Here is a conversation history: [History]
$(History)
[End of History]
Here is the final question and the
standard answer:
[Question]
$[Query)
[End of question]
[Standard answer]
$(Doctor_response)
'equal to" As-
[End of standard answer] Based on the conversation history, user question, and standard answer, please rate the following two AI responses scale of 1 to 10,
In the last line, please output single line contain- ing only single label selecting from 'Assistant 1 is better than Assistant 2', 'Assistant is worse than Assistant 2', and 'Assistant is equal Assistant 2'.
considering accuracy, conciseness, and similarity the standard answer. Please provide the ratings in the following format: "Rating A: [score]; Rating B: [score]".
E Training Details
[Assistant A]
s(Response_A)
Our model is implemented in celerate and trlx packages with LLaMA (Touvron et al., 2023) as the base architecture. We initial- ize the model parameters using Ziya-LLaMA-13B Pretrain. It was a LLaMA- model continue pre- trained on massive Chinese corpus, which enables better follow Chinese instruction and store more
P PyTorch using the Ac-
[End of Assistant A]
[Assistant B]
$(Response_B) [End of Assistant B]
C
The Details of Hybrid SFT Data
d hhttps://hugiiggface http://hhggggaaeeeecccc///eeeeee httpe///cc/a/aaeeeereeeeiinead htttp:: thub. httt//occcc///rrrrr///ttti
The details of hybrid SFT data is shown in Table 3.

---

Chinese knowledge. We leverage ZeRO (Rajbhan- dari al., 2020) strategy to distribute the model across 8 100 GPUs for training. the supervised fine-tuning process, we set the learning rate, batch size, and maximum context length to 5e 5, 128, and 2048, respectively. All models are trained for 3 epochs and weights performed the best the validation set saved. During the reinforcement learning process, we only update the parameters of the last two layers. The entire process encompasses 16, 000 steps, undertaken a learning rate of l We establish a rollout size of 64, chunk size of 8, and 4 PPO epochs. The initial divergence co- efficient is set at 0.1. In addition, to enhance the model's conversational and Instruction-folloviii capabilities in the general domain, we have incorpo- rated Chinese instruction data (the Chinese Alpaca dataset (Peng al., 2023) and conversation data ) This enhances the model's ability to effectively understand and generate responses in various conversational scenarios and accurately follow instructions across different domains.
tion recommendations. Evaluators are provided a of medical cases symptom descriptions and assess the accuracy and reliability of the medication recommendations based on medical knowledge and guidelines.
G Details of Evaluation
We present detailed evaluation scores correspond- ing to each category Table and Table 5. Sam- ple responses from four models for ablation study are shown in Table
H Details of Popular Medical Language Models
The details of popular medical language models are shown in Table
I Cases
I.1 Single-turn Question Cases
Single-turn question cases are shown in Table 7.
1.2 Multi-turn conversation cases are shown in Table 8.
Multi-turn Question Cases
F Evaluation Guidelines for doctors
In the manual evaluation of the HuatuoGPT, we think that the following three aspects should be considered, particularly in medical consultation and medication prescription, and take them as the guidelines for evaluation:
1.3 Ablation cases
Ablation cases are shown in Table 9.
J Evaluation on Other Backbones
Diagnosis accuracy. This aspect evaluates the model's 's accuracy and comprehensiveness in diag- nosing patient symptoms. Evaluators are provided set of medical cases symptom descriptions and assess the correctness, relevance, and reason- ableness of the model's diagmosis. Comparisons can made with assessments made by medical profcssionals to ensure the model's s accuracy.
Using the same method and data described in this paper, we have also conducted a simple study on Bloom-7B1-mt and compared the performance with ChatGPT on human evaluation, which is shown in Table 10. This shows that even with a much-smalles backbone model, HuatuoGPT could also achieve competitive performance with Chat- GPT. The philosophy of HuatuoGPT is effective regardless of which backbone and how large the backbone is.
Treatment recommendation accuracy. This aspect assesses the accuracy and appropriateness of the model's treatment recommendations for pa- of medi- cal cases or symptom descriptions and evaluate whether the model's 's treatment recommendations align with medical knowledge and real-world ap- plications that are effective and reliable for the patient's s main condition and problem.
tients. Evaluators
provided
Medication knowledge and prescription accu- racy. This aspect evaluates the model's 's understand- ing of medications and the accuracy of its prescrip-
10 "https://uugiiggf ace. ctteee.o/ddaaeeee htteee//aaattttt///hssccc.m sharegpt-raw

---

Data
Sources ChatGPT
Scale 61,400
Description
Example
Medical instructions generated with self-instruction. Seed in- structions are manually written.
User: A2 2
Instruction
$
Doctor: D
*
1. :
T
$
2. 0
26 2 44.
Doctors 69,768
Single-turn medical consultants
User: 1Brr,
real-world scenarios, which the doctur's 's response is more spe-
cific and can address the user's medical concerns.
Doctor: SG
* 0.100000
A. $.
$. *
ChalGPT 68,888 Medical multi turn conversations are created using two ChalGPTs:
User: -
Conversation
$
one acting as the patient and the other the doctor, The conversa- based real-world pa- tient cases and their correspond- ing treatment.
Doctor: Oo000
T
1. 1
3. 3
User:
T.
Doctor: 2 23445
* "
Doctors 25,986
Medical multi-turn conversations in real-world scenarios. Conver- sations include multiple rounds of questioning doctors and continuous questioning by pa- tients.
User: #
Ductor: 1 77
User: #
Doctor: : 
R P
Lo b
2
ix k
44 26 da 5
Table 3: The Details Hybrid SFT Data.

---

Category
HustaoGPT BeaTiso
HeatnsGPT YA.
HuatnoGPT
HoatooPP ChaGLLM66
HustoGPT VA. CharGPT
HaatoGPT ss.
DectorGLM
Zyy-LLLMMA-138
GPE4
10/0/0
10/0/0
6/1/3
$13/2
4/1/5
7/2/1
Efficacy
(9/1/0)
(9/1/0)
(5/312)
(3/3/4)
(3/4/3)
(4/1/5)
10/0/0
9/0/1
7/1/2
4/4/2
8/1/1
7/2/1
Medical Expenses
(10/0/0)
(8/2/0)
(7/2/1)
(5/4/1)
(3/6/1)
(4/175)
10/0/0
10/0/0
4/0/6
4/2/4
3/3/4
1/2/7
Consequences Description
(8/0/2)
(9/1/0)
(6/0/4)
(4/3/3)
(3/413)
(2/1/7)
10/0/0
10/0/0
4/3/3
410/6
21414
3/0/7
Medical Advice
(8/0/2)
(10/0/0)
(8/0/2)
(3/4/3)
(11415)
(3/1/6)
10/0/0
8/1/1
7/0/3
512/3
9/0/1
4/1/5
Indicators Interpretation
(9/1/0)
(8/0/2)
(4/0/6)
(6/0/4)
(5/2/3)
(0/1/9)
10/0/0
10/0/0
3/3/4
2/1/7
314/3
3/1/6
Treatment Plan
(10/0/0)
(9/1/0)
(315/2)
(51213)
(51015)
(3/3/4)
10/0/0
10/0/0
5/1/4
5/1/4
3/3/4
2/0/8
Precautions
(10/0/0)
(10/0/0)
(612/2)
(3/215)
(6/0/4)
(1/5/4)
9/1/0
9/0/1
2/414
51213
5/1/4
2/3/5
Disease Description
(8/1/1)
(7/1/2)
(2/2/6)
(2/315)
(1/6/3)
(3/2/5)
10/0/0
10/0/0
6/0/4
4/0/6
5/1/4
0/3/7
Etiological Analysis
(10/0/0)
(9/0/1)
(4/3/3)
(4/4/2)
(2/14)
(1/4/5)
9/1/0
9/1/0
6/0/4
51312
5/3/2
2/3/5
Condition Diagnosis
(9/0/1)
(8/2/0)
(5/2/3)
(4/2/4)
(6/2/2)
(1/5/4)
98 /
95/2/3
50 13
43/ 28/18/38 /9
/2/21/32 (35 (35/33/33)
31 17 52 (22/24/ 54)
Overall
(91 /
(87 (87/8/5)
(50 39)31)
(39 /
Table 4: The detailed results of the single-turn question evaluation. The top value is #votes from GPT-4, and the bottom blue value #votes from doctors. The value indicates the outcome as "Win/Tie/Loss".

---

HuatuoGPT v.s. DoctorGLM
HuatuoGPT v.S.
HuatuoGPT v.s. ChatGLM-6b
HuatuoGPT v.s.
Category
Ziya-LLaMA-138
ChatGPT
4/1/0
4/1/0
5/0/0
3/1/1
Traditional Chinese Medicine
(2/0/0)
(1/0/1)
(1/0/1)
(1/0/1)
5/0/0
3/1/1
4/0/1
3/0/2
Obstetrics
(2/0/0)
(1/0/1)
(1/1/0)
(0/0/2)
5/0/0
4/0/1
4/0/1
3/0/2
Pediatrics
(3/0/0)
(2/0/1)
(1/1/1)
(2/0/1)
5/0/0
4/0/1
4/1/0
3/0/2
Internal Medicine
(4/0/0)
(2/0/2)
(0/1/3)
(3/170)
5/0/0
3/0/2
2/1/2
1/2/2
Stomatology
(/0//)
(1/2/0)
(2/0/1)
(2/1/0)
5/0/0
3/0/2
3/0/2
3/0/2
Surgery
2/0/0)
(0/1/1)
(0/0/2)
(0/1/1)
5/0/0
4/0/1
3/0/2
3/0/2
Obstetrics and Gynecology
(2/0/0)
(1/0/1)
(0/1/1)
(L/1/0)
4/1/0
3/1/1
3/1/1
2/1/2
Gynecology
(2/0/0)
(1/1/0)
(1/1/0)
(1/1/0)
4/0/1
3/0/2
3/0/2
3/0/2
Cardiovascular Medicine
2/0/0)
(0/2/0)
(2/0/0)
(L/1/0)
5/0/0
4/0/1
4/0/1
3/0/2
General Surgery
(/0//)
(0/0/3)
(1/0/2)
(L/0/2)
5/0/0
5/0/0
3/0/2
4/1/0
Urology
(3///0)
(1/2/0)
(2/0/1)
(2/1/0)
5/0/0
3/0/2
2/0/3
4/0/1
Gastroenterology
(1/0/))
(1/0/0)
(1/0/0)
(1/0/0)
5/0/0
4/0/1
4/0/1
4/0/1
Andrology
(11/0)
(0/1/1)
(1/0/1)
(1/0/1)
$/0/0
5/0/0
4/0/1
4/0/1
Dermatology and Venereology
3/0/0)
(1/2/0)
(1/1/1)
(1/11)
5/0/0
4/0/1
4/1/0
5/0/0
Dermatology
(1/0/)
(0/1/0)
(1/0/0)
(1/0/0)
S/0/0
3/0/2
2/2/1
4/0/1
Ophthalmology
(2/0/1)
(1/0/2)
(3/0/0)
(1/0/2)
5/0/0
4/0/1
1/2/2
2/1/2
Neurology
(1/0/0)
(0/0/1)
(1/0/0)
(0/0/1)
5/0/0
3/0/2
3/0/2
2/0/3
Neurosurgery
(S/0/0)
(4/0/1)
(3/1/D)
(3/0/2)
S/0/0
4/0/1
4/0/1
3/0/2
Psychiatry
(2/0/0)
(1/1/0)
(1/0/1)
(2/0/0)
5/0/0
5/0/0
4/0/1
3/0/2
Orthopedics
(4/0/0)
(3/170)
(3/0/1)
(3/0/1)
97 /
75/3/22 (21 15)
66/8/26
62/6/32
Overall
(11)
(26/7/17) 17)
(27/8/15 15)
Table 5: The detailed results of the multi-turn question evaluation. The top value is #votes from GPT-4, and the bottom blue value is #votes from doctors. The value indicates the outcome "Win/Tie/Loss".
Language
Training
Size
Mcd-PaLM2-
English
PaML2 (340B)
MultMedQA
193k
ChaiDrotor
English
LMM (7B)
MaaullaCCMMMaiiOOOO0 100K, :Cliniq 10K
110K
Visual Med-Alpuca
English
LLaMA (7B)
MEDIQA RQE, MedQA, MedDialog, MEDIQA QA, PubMedQA
54K
MEDALPACA
English
LLaMA (7B/13B)
Medical Mcadow
160K
DoctorGLM
Chinese
CharGLM (6B)
CMD., MedDialog, ChatDuctor, HearithcareMngic
4.4M
BenTsao
Chinese
LLaMA (7B)
CMcKG-8K
SK
Table 6: The Popular Medical Language Models Training Data Sources and Model Specifications.

---

User:
(Why is there blood on the baby's white eyeball?)
BenTsao:
E.. i 9 12 43.
(Your baby may have simple supraocular hematuria and a visit the ophthalmologist recommended for examina- tion and treatment.)
Do
(Hello, according your account, consider the inllammation caused by the general and local pathogenic infections related. you can take oral Amorin, metronidazole treatment try, pay attention rest, do stimulate food, drink more water, slowly will improve.)
Ziva-LLaMA-13B:
I. 1:
- AL T
2. H|
3. 12
33
0) A 6 65
R # #
(Hello, the appearance of blood your baby's white eyeballs may be due variety of causes. Common causes
include:
Conjunctivitis: When the conjunetiva infected or irritated, it can lead to conjunctival congestion and edema, which can manifest itself as red or yellow blood the white eychall.
2. Overuse of eyes: prolonged use of electronic devices, reading books and other activities can easily cause visual fatigue and dryness. leading to bloodshot eyes on the white eyeball.
3. Overall health condition: your baby suffers from other diseases, such as anemia and high blood pressure, it may also affect the eye health and lead to bloodshot eyes on the white eyeballs.
We recommend that you take your baby to the hospital so that the doctor can treat the symptoms, At the same time, pay attention to maintaining good eye habits, avoiding excessive eye use, ensuring sufficient sleep, and enhancing physical fitness.
ChatGLM-6B: I é B WA
Pi: 2 3
2. 53 2
3. 1t 1
$,
4. 4 5. WW
# $
(Blood in the white of your baby's 's eyes can be caused by a variety of reasons, the following are some common
causes:
1. Blocked tear ducts: When the tear ducts are blocked, tears cannot drain smoothly, which can lead to increased
eye discharge, including blood. This phenomenon usually occurs in borno or infants.
2. Conjunctivitis: Conjunctivitis is common eye discase that usually leads to increased eye discharge and blood in the white of the eyc.
3. Allergic conjunctivitis: If your baby's 's eyes are allergic to pollen, dust, animal hair and other substances in the air,
it can lead to increased eye discharge and blood in the white eyeball.
4. Eyelashes backwards: Eyelashes may be inserted inside the baby's eyeballs, causing irritation and friction.
resulting in increased eye discharge and blood on the white eyeballs.
5. Congenital blockage of the tear duct: Congenital blockage of the tear duct is common neonatal condition that causes tears to not drain smoothly. resulting increased eye discharge and blood in the white of the eye. If your baby has bloodshot white eyes, it is advisable to seek medical attention, so that the ophthalmologist can examine and diagnose the cause, and take appropriate treatment measures.

---

ChatGPT: 9 A T
2. HWW /
3,
4.40 K7 8AA
70 #
-
44
F
F $, 12 A B] 0B. I x 2
(Blood on haby's 's white eychalls may be due to the following reasons:
1. Intense crying or coughing causes the eye pressure to rise and blood vessels to dilate causing blood to appear on the eyeballs.
2. Infections, such as conjunctivitis, keratitis, trachoma, etc. These diseases can cause symptoms such as redness and swelling of the eyes, increased secretion and pain.
3. Stasis of blond duc to eye injury.
4. Dry eye disease. Prolonged use of electronic devices. long periods of time without closing or blinking the eyes, etc. may cause dry eyes and lead to dilated hlond vessels.
If you find blood on your baby's eyes, it is recommended to observe for one or two days. If the situation persists or worsens, or if it is accompanied by other symptoms, such as loss of vision and eye pain, you should seek medical attention promptly It is also important to maintain eye hygiene and avoid eye injuries or overexertion.
GPT-4:
1. 1
N. AO BR 68 4E
AE de
(6)
2. . N 24 24.
;
3.  341.:
$ #
4. h
5. :
$ 71 BR A 78
$$420 XA
4-Hu
(Bloodshot eyes in babies can be caused by variety of reasons. Here are some common possible factors: 1. Eye fatigue: If your baby watches TV. plays on the phone or reads for long time, it may lead to excessive eye fatigue, which in turn causes the blood vessels on the surface of the eye dilate, manifesting as bloodshot eyes.
2. Eye irritation: Smoke, dust, and allergic reactions may cause eye irritation, which may lead to eye congestion and manifest as bloodshot eyes.
3. Ocular dryness: dry indoor air, prolonged computer and other factors may lead to eye dryness, which may trigger eye congestion and manifest bloodshot eyes.
4. Infection: If your baby's eyes show symptoms such redness, pain and increased discharge, it may be due to bacterial viral infection.
5. Injury: If your buby's eyes have suffered physical injuries such as bumps and scratches, they may also cause the blood vessels the surface of the eye rupture and manifest as blood streaks.
Whatever the case may be, if your baby's eyes appear abnormal, it recommended that you seek professional medical advice as soon possible. particular, symptoms such as redness, swelling, pain, and increased discharge may require prompt medical treatment.
HuatuoGPT: & #
# 2 x hh 3 + 3
AL A HN DE 63 I AL. SH & &
T ME A 47
x 2
-
#
* $
16 #6 M.
$$
24 $ 3
$. (Hello, dear parents! Thank you asking me this question. According to your deseription, the appearance of blond inside your baby's 's eyes may be caused by a variety of reasons. One of the most common causes is eye diseases such as conjunctivitis
x 1 ff 32 32 15
4
keratitis, which may also be related to respiratory diseases such as allergic rhinitis.
In order to help you better, would like to ask some related questions. First of all, has the baby been exposed to dust, pollen, animal hair and other allergens frequently since birth? Secondly. how is the baby's sleep quality? Are the eating habits healthy and balanced? Are there any other uncomfortable symptoms, such as cough, runny nose, etc.? After getting more information, will try my best to provide more specific suggestions and treatment plans. In the meantime, suggest you take your baby to regular hospital in time so that the doctor can conduct a comprehensive examination and diagnosis and make reasonable treatment plan. hope your baby gets well soon!
Table 7: Sample Responses of Single-turn Questions.

---

Description:
$ #
# #3 3444::/w88.2HHnn3333333333333333
Æ (1467558800400000333333333353331
$ 32220018177771100001133333333333
$ - "
#ル 23
16 7 12t 2. 1
1522 22-
3 1
(Disease: Skin Allergy
Description of the condition: Initially, took the child to the hospital because he kept scratching his lower legs and arms, causing them to bleed. The doctor said it was due to dry skin, and prescribed Er Fu Kang lotion. After using it, the condition improved. However, recently the child started scratching again. We initially thought was still due to dryness and continued to apply Er Fu Kang lotion, but it didn't itdn'tien help. Later, we found something similar to goosehumps on the child's skin, which wasn't I red but itchy. Whenever he took off his clothes, he started scratching severely, often causing bleeding. This happened would form, similar to ringworm. The doctor at the hospital said it from Peking Union Medical College Hospital (we started applying it yesterday aftemoon) and Desloratadine syrup. But today it seems have worsened, with patches of red appearing on his legs. When he was young, he ate shrimp, but after his skin condition worsened, dared feed him shrimp any more.re After eating eggs, bumps would appear around his mouth but they would disappear after while. He usually eats very little, but four days ago when visited friend's house, he grabbed egg. and trighteningly. his mouth was filled with bumps, and wherever his hand touched, bumps would form. However, they disappeared after woke up from sleep.
his arms, back, waist, buttocks, legs, and the back of his neck. After scratching, thick layer
allergy and prescribed Fu Le skincare cream
stopped for while. When tried again, he developed bumps all over his body.
Help needed: Is this allergy? Do need Duration of illness: Within month
other medicines?
Summary of condition:
Urticaria, Eczema
Advice: Take Desloratadine syrup orally as per the instructions. Apply Fu cream Hydrocortisone butyrate cream twice a day. necessary, take Yu Ping Feng granules orally twice day, half pack each time, one course of treatment lasts for one month.)

---

DoctorGLM:
66 AAA: & WK. HARTHER 41113 124
(Patient: have recently developed itchy skin and my doctor might be an allergy. was prescribed Skin Care Cream Desloratadine Syrup from Peking Union Medical College, but today I found that my symptoms don't seem to be getting better. Does have caused by allergies?)
DoctorGL.M: 12.88 65 &t
A E.E at d5 BK # 13 17 32 it. $ 4 B8 #
IR at SE
/) 9P M
pp
T
(DoctorGLM: According to your description, it is a case of skin allergy. It is recommended to take oral anti-allergy medication such as paracetamol tablets or vitamin C, calcium and topical glyburide lotion to restore health. The condition can be relieved by preventing the reoccurencce of spicy and irritating foods and seafood. The above is suggestion for "the recent itchy skin problem', hope you have help. wish you good health!)
&i A 18 t6. 4 ty Hi to dj st iR it [i] 4x H. &. CC $ # 3 2 PJ BT
3 ME
"E? A 3 R F I. RL 25
(Patient: Yes, thank you very much for your advice. How long do the paracctamol tahlcts or vitamin C need to last? Also, what do I need to do with the stove glycolic lotion? Do I need to go to the hospital for another visit?)
DoctorGI.M 18 # H8 it & x 2 16 M AE $ St b4 $ 1845 65 A A it 12 i if db & 99 4 i5 Bf tt i ft
(DoctorGLM: According to your description you should belong to the allergic reaction caused by the performance of hives, we recommend that you go the hospital for allergen testing to clarify the diagnosis and then actively carry out treatment.)
& 49
A $ d. &C #
# ill i
#L T F 11 fo d: 4
30 #0
(Patient: Yes, I will go the hospital promptly for allergen testing. Other than that, I would like to know the advice on the of Furamyl lotion and the dose and duration of paracetamol tablets, vitamin C and calcium. Thank you!) DoctorGI.M: 1B 43: 8f W. 65 ti 22. al ix t 3) H AB E. & il - F E. E Ht Ht f - &
(DoctorGLM: According your situation, is recommended that you consult doctor your local hospital for some better)
$ A: 15 #3
# AQ t $5 No 3E # 12 K8 68 t A fo A| +
1) (Patient: Yes, thank you for your advice. will the medication as prescribed my doctor and read the instructions carefully to ensure the correct usage and dosage. Still, thank you very much for your help and advice. Bye!)
by 4 AR!

---

ChatGLM-6B:
# do *. A t5 20
(Patient: I recently had itchy and goosebump-like skin. My doctor suid it might be an allergy and prescribed Skin Care Cream and Desloratadine Syrup. but the symploms don't t seem to be completely relieved. want to know if this is an allergy or not, and I need to use any other medicine?)
ChatGLM-6B. : to X to 2%e 45 SE OF it 8 IE # AK
&8 I BR Is iE th
B 1R, RE
$ th.
te *
T 2,
* RN 86. *
2. x 18 H 2
3. 3.
3
C4E
15 SMTRSERAE.
4,
& 1 #
5. #2 B
13-04
(ChatGL M-66: If your doctor has given a treatment plan for Skin Care Cream and Destoratadine Syrup, it is recommended that these medications be used as recommended by your doctor. If symptoms are not relieved, consult your doctor or pharmacist for more information and advice to determine if you need to change to other medications or other treatments. In addition to medication, the following measures can be taken to relieve symptoms such as itching and chicken skin: 1. Keep your skin clean and dry. Use lukewarm water for bathing and avoid using excessively hot water, while not rubbing the skin excessively.
2. Wear loose, breathable clothes to promote air circulation and reduce irritation to the skin.
3. Pay attention to your diet and cat more foods rich in vitamins A, C and E, such as carrots, spinach, peanuts, etc., which help improve the quality of your skin.
4, Avoid using cosmetics and skin care products that contain irritating ingredients to avoid aggravating the symptoms.
5. If any adverse reaction occurs during the treatment, inform your doctor immediately and make adjustments.)
& A: 39-65 ( f is at it. & a tie aR $ ist ş it dt do X 2 & 95 A 4j RE
A . & & F a & 14 65
(Patient: Yes, thank you for your suggestion, I will follow your advice and if I still have questions afterwards, I will consult again.)
CharGLM-6B T R. 15 El the R!! for R. & fs H6-fT RE [+] M Bf # of X t & 1 [+]
(ChatGLM-6B: You're re welcome, and wish you a speedy recovery! If you have any questions, feel free ask me.)
46A $5-)) # to in J. # % 5 ilt
(Patient: Okay, I got thanks lot!) ChatGLM-6B: : * & ". [.
BY Bb.
AL AE **
(ChatGLMM-a M-68: You'de welcome, hope my advice helps, and I wish you gond health! you have any other questions, please feel free to ask me.)
A 19,6, # f & 62 4 L
(Patient: Yes, thank you for your good wishes! Bye.)
Ziya-LLaMA-13B:
AA. t A
T AL 1 22 2 2 X.?
# #)
DE R # 65 "5
(Patient: Recently, my child's body is red like goose humps, but it is itchy. I used Beijing Union's 's Skin Care Cream started applying yesterday aftcrnoon) and Desloratadine Syrup, but today found that it seems he little hit stronger again, and my legs are starting get red patches. Is an allergy? Do I need to use any other medicine?)
Ziya-LLaMA-I #o 3.
#t 505 2h H8 t8 66 di aR. of NE DE R.
B E ff @ L 86 B 45
i 2 & B2
1 tk
At
12
IR dL t0 for of AP JP di
52 RR 10 R
t 24
Wy Ha l DJ
B 2b.
#
(Ziya-LLaMa 13B: According to the symptoms you described, may be hives. Urticaria is a common skin condition that mainly manifests as wind clumps, swelling and itching. Although symptoms can be relieved by using BJCC's Skin Care Cream and Desloratadine Syrup, if symptoms worsen or persist for a longer period of time, it is recommended to consult doctor and undergo further diagnosis and treatment. Also, taking care to keep the skin clean and dry and avoiding exposure to irritants can help reduce symptoms.)
$6 A. 66. , # $ i MN A by & & &
RR 3. #
. 38 x W A8 (Patient: Yes, thank you very much for your answer. I will pay attention and observe the child's s symptoms again. If the symptoms do not improve, I will take my child to the hospital to consult the doctor's advice. Thank you again for your help!)
ihy $ 6h
36
Ziya-LLaMA-13B: : A 24 49 M # 2 7 & - t H [9)
A6 th
JR $
(Zya-LLLMMA-13B: I'm glad could help you and hope your child recovers soon. If any other problems arise, feel free to contact me and I will he happy to help you.) &6 A: $5 65 at # & M6 4 R
(Patient: Yes, thank you very much. Bye!)

---

ChatGPT:
.6 A H TRR EA - SA P
- $1
(Patient: I recently developed an itchy skin condition, and my doctor said was an allergy and prescribed some medication. But recently seems to have gotten little worse, so would like ask need use any other medicine.)
ChatGPT: IR R
E.
a 12
I + I dir if #5 It T
B 7 2
3i fL (ChaIGPT: If the itching is severe, is recommended that you see your doctor again so that he or she can reassess your condition. In addition to oral medications, your doctor may also recommend topical creams or baths to relieve symptoms such as itching and redness. You also need to take care to keep your body clean, avoid contact with allergens, get enough sleep and diet, and avoid mental stress, all of which will help improve your symploms.) $6 A é) 3 in 4i & A A 1 E. f & A 12 # /6 2 66 X &
# -i PL to f
&
(Patient: Yes, thank you for your advice, will sec my doctor again and pay attention to personal hygiene and lifestyle improvements.)
ChatGPT: : T f. A # 12
Pi f 4o # P 15
$ 1R
(ChaiGPT: You're welcome, and hope you get well soon. Remember to follow the doctor's instructions, take your medication on time, pay attention to self protection and regulation at the same time, and consult your doctor whenever have questions. wish AA; # # & Hl fo an fo Y 12, & 2 (Patient: Thank you very much for your blessing and concern. will pay attention protect and regulate my body and recover my health as soon possible. Thank you again!)
good health and happiness!)
2 & 58 4- fo in $ E 2 2 # &
#
ChatGPT: T. %, 1t - by X 41 to R 28 H A fe 19 2 i
R (t 12 #:
(ChatGPT: You're welcome, and I wish you all the best! If you have any other questions you need to ask, feel free to ask
me.)
SA. 39 #
1hy 66
(Patient: Yes, I will definitely ask you again if need to. Bye!)

---

HuatuoGPT:
A:A: B it,
(Patient: I have recently developed skin allergies and would like ask this caused by allergies and I need to use any other medication?)
HuatuoGPT: 2 - B 65
ETELANNRR 12 M 10
R. 8
2. x2
# $1 #5
4, $ (2 M Ro
# X 1
M.. #
(HuatuoGPT: Hello, thank you for consulting with me. Based on the information you have provided. you may be experiencing an allergic reaction. However, in order to determine your situation more precisely, need to know more about your symptoms and physical condition.
1. Have you been exposed to substances such us new cosmetics or skin care products?
2. Have you taken any new medications or caten certain specific foods?
3. Where is your skin area? Does it occur with other symptoms such as itching, redness, swelling, pain, etc.?
4. What is the treatment you are currently using? What is the effeet?
In response to the above questions, please describe your situation in much detail as possible, so that I can make more
accurate diagnosis and provide appropriate treatment recommeedaations))
A 1. & 28 AT de At (ff-f #5 ét f2. tk So Hi 1
$ Hy 18
2. X के 19 H M 2i ft thg de 2 (R (E 4i é) 65
3. 1 1k
31
16
AL
A AE
4. A (t 6)
2 # dt lb fo 65 1: I
Se 42.
& # 1 # x R&44
(Patient: have not been exposed to any new substances such as cosmetics skin products.
2. I have not consumed any foods taken any medications.
3. Skin areas mainly my arms, back, waist, buttocks, legs and the back of my neck, appearing small bumps like goose bumps that trigger itching and thick layer of bumps similar ringworm when seratched, and recently my legs have started to glow red in patches. Other than that, I have other significant physical symptoms.
4. The treatments currently used Skin Care Cream and Desloratadine Syrup from Peking Union Medical College, which do not seem be very effective.)
HuatuoGPT: # t Al ai E R 6 #) it dur 12 13
44 dn
A (urticaria)
B th it & (hives). E &. T
#
NEL 8f 12
$ # H R.
E
12
Bt
38
ME 2 (HuatuoGPT: Thank you very much the detailed information. With your symptoms, is initially considered that you may be suffering from urticaria, also known as hives, which is an allergic disease. Since urticaria has variety of manifestations and the symptoms are usually recurrent, treatment plan will need to be developed on a case-by-case basis, Based on the fact that you have tried Skin Care Cream and Desloratadine Syrup without getting good relief. recommend that you visit the dermatology clinic of your nearest hospital to have your symptoms examined by a professional doctor and individualize your treatment according to the actual situation. At the same time, avoid exposure to sunlight, wear loose and breathable clothes, keep your indoor environment clean and hygienic, and avoid contact with substances that may cause allergies, Hope you will recover soon!) a) A 64 it. & 4 & Ie
t XL
$6 A.A. 66 1 18 :
W
(Patient: Yes, thank you very much for your advice, I will visit the dermatologist as soon as possible, follow the doctor's instructions for treatment and pay attention to daily prevention, thank you for your help. Bye!)
Table 8: Sample Responses of Multi-turn Conversations.

---

User:
& 4. # # 11- in 1 85 - 2
(I've been experiencing diarthea recenty, with my stool appearing frothy yellow liquid. suspect might due to the milk I've been drinking. Could this be the reason? Thank you.)
HuatuoGFT (w/ real data);:
@ #f 88.8 fi #0 TRA X1 th ME ( 2f # (Hello! Based on the symptoms you described, there might be potential issue with your gastrointestinal system. Moreover, dietary habits can also affect occurrence of such symptoms. From traditional Chinese medicine perspective. this primarily related disorder of spleen and stomach functions, since they closely linked. Therefore, treatment would involve methods strengthen the spleen and stomach. Additionally, you should consider light and easily digestible dies in your daily eating habits.)
Hu (w/ distilled data):
6)
18.38 18
R
25
02
ILA:
K.
18
2.
%
3.32 #l
4.52
# 4
2M
# M #
$****
H $2
& 1 18
(Hello, thank you for reaching out. Based on the symptoms you' 've described, there could be inflammation in the digestive tract or some form of gastrointestinale dysfunction.
Firstly, to alleviate your diarrhea symptoms, please pay attention the following points:
1. Drink plenty of clear fluids to prevent dehydration.
Eat small, frequent meals maintain balanced diet and avoid overeating or hunger.
3. Avoid foods that can irritate the digestive system, such chili peppers and ginger.
4. Take adequare rest and avoid strenuous exercise and fatigue.
Socondly, regarding the possibility you mentioned about milk cansing diarrhea, further information about your personal circumstances would be aceded to make judgement. Typically, milk contains rich leading lactose intolerance protein allergies that can cause diarchaa. However, many people can consume milk normally without such reactions. Therefore. suggest you continue to monitor your symptomss some time to see if there is any improvement. If symptoms persist, recommend you medical attention. other discases have been ruled if necessary, your doctor can provide you more specific treatment plan.)
and proteins. Certain individuals lack the cuzyıne digest these substances.
HustuoGPT (w/o RLMF):
2
A AL MUS de 16
d6 18
(ib
(5)
11
1.
fe
R19
2.
$
16
#
5.
x
#1 ??
(Hello, according
description of the symptoms you provided. likely that the diarricca been caused The frothy yellow fluid usually indicates excessive bile excretion, which is also associated with the dictary habits you meationed (such as drinking milk).
To better help you make the correct diagnosis and get the proper treatment plan. please tell me the following information:
your age and gender?
2. how long has this condition lasted?
3. Besides diarrhea, are there any other symptoms, such as vomiting or fever?
4. What are your current cating habits and lifestyle?
Have you ever received similar treatment?
HuatuoGPT:
43
TE
12 R
$1
1.
23
2.
R.y 1 MEAL IR
5.
*
symptoms you provided, there may be conditions such gastrocnteritis indigestion. and the foamy yellow fluid also indicates that there may be some dogree of diarbea. Regarding diet, some people may be allergic milk or prone reactions such as bloating and diarrhea, but an individualized assessment is needed to reach a conclusion. In order to determine your condition more precisely, please ask you following questions:
(Hello, based on description
1. how often you pooping now?
2. Are there any changes in your daily water intake and food composition?
3. Besides diartcaa, do you have any other discomfort. such as nausea, vomiting, abdominal pain, etc.?
Do you usually fatty spicy food?
5. Are you currently taking any medications?
Afler obtaiiin this information, will make more necurate diagnosis and give you appropriate treatment advice.
Table 9: Sample Responses for Ablation Study.

---

Comparison
Win
Lose
Single-turn
HuatuoGPT-7B vs ChatGPT
52%
48%
Multi-turn HuatuoGPT-7B vs ChatGPT
58%
42%
Table 10: Manual Evaluation the Model Performance
of 7B Scale.


Reviews:
[{"review_id": "XwhQJIw0zq", "reviewer": "Reviewer_rXzS", "summary": "", "questions": "1. In line 335, \"When evaluated by doctors, we randomly sample 50 patient cases from 100 test cases\", but in Figure 4(b), the results still contain 100 cases.\n2. For medical QA, how are the results of fine-tuning the HuatuoGPT by using training data? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their insightful comments and observations.\n\n> **Q1.The contribution of this paper mostly lies in engineering**\n\n**Reply**: We acknowledge the reviewer's point about the engineering aspect of our work. Our research philosophy is to first investigate a straightforward and simple approach. Such an approach, especially in the era of large language models, often leans toward partially-engineering solutions. We believe that engineering-assisted research, as a first step, is beneficial for contextualizing the potential of large language models in medical scenarios. Specifically, this may provide valuable insights for the audience from the EMNLP (**Empirical** Methods in Natural Language Processing) community .\n\n> **Q2. Since the experiments of single-turn/multi-turn dialogue generation are conducted on filtered small test sets, and the experimental results of medical QA benchmarks are mostly worse than T5 (finetuned) except for cMedQA2, the value of the experimental results doesn't seem to be significant.**\n\n**Reply**: We appreciate the feedback on our evaluation approach and recognize its limitations due to GPT-4 access quota constraints and the cost of doctor evaluations. We believe that the evaluation is reliable, as seen from the following aspects:\n\n- **Blind Evaluation**: Our evaluation approach, involving medical professionals, was blind. This means that the evaluating doctors weren't aware of the answer's source, ensuring unbiased feedback.\n\n- **Consistency between GPT-4 and doctors**: Despite the smaller test set, the doctors' evaluations aligned closely with our GPT-4 evaluations. This consistency suggests that our evaluations, although limited in scale, can be considered reliable.\n\n\n**A slightly larger test set**\nWe understand the concern about the limited test set. Our GPT-4 quota constraints restricted us to 100 questions for testing before the submission. With an increased quota after that, we expanded our tests to 500 questions from KUAKE-QIC instead of the original 100 questions. The extended results show that HuatuoGPT still wins all baselines,  confirming the validity of our model as the dataset size grows. We appreciate this suggestion and will incorporate these extended results in the revised version.\n\n| HuatuoGPT Win rate                            | 100 Questions (win/ tie/ fail) | 500 Questions (win/ tie/ fail) |\n|-------------------------------------|--------------------------------|--------------------------------|\n| HuatuoGPT vs ChatGLM-6b             | **43** / 18 / 39                   | **251** / 59 / 190                 |\n| HuatuoGPT vs Ziya-LLaMA-13B         | **50** / 13 / 37                   | **260** / 66 / 174                 |\n| HuatuoGPT vs ChatGPT                | **47** / 21 / 32                   | **234** / 55 / 211                 |\n\n**Performance of QA benchmark**: While HuatuoGPT might lag behind the fine-tuned T5 on certain benchmarks, this comparison isn't strictly apples-to-apples. T5 is trained using a supervised dataset, a.k.a **full-shot** setting. While HuatuoGPT is evaluated in a zero-shot setting. Note that HuatuoGPT still outperforms the fine-tuned T5 in some metrics in all datasets (e.g., BLEU-1). Especially, in the zero-shot setting, HuatuoGPT outshines other chat models, demonstrating its versatility and commendable ability to perform out-of-distribution tasks. This distinction underscores the unique strengths of HuatuoGPT.\n\n\n> **Q3: In line 335, \"When evaluated by doctors, we randomly sample 50 patient cases from 100 test cases\", but in Figure 4(b), the results still contain 100 cases.**\n\n**Reply**: Apologies for the oversight. The discrepancy arises from how we presented the data. While we sampled 50 patient cases for doctor evaluation, Figure 4(b) displays results in **percentages** based on those 50 cases, making it appear as if it's out of 100. For exact numbers, please refer to Table 5. We'll ensure clearer representation in the revised version.\n\n> **Q4: For medical QA, how are the results of fine-tuning the HuatuoGPT by using training data?**\n\n**Reply**: \nDue to time limitations, we sampled 50,000 QA (20% training data) from the webMedQA training set to fine-tune HuatuoGPT. The following table shows fine-tuning HuatuoGPT with the webMedQA training set leads to an improvement in all metrics, indicating the transferability and generalization of HuatuoGPT. In addition, the table shows that even with 20% training data, HuatuoGPT achieves comparable performance or surpasses the T5 model with 100% training data in most indicators, demonstrating our model's potential. We will give more detailed experiments and analysis in the revised version.\n\n| Model                  | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | GLEU | ROUGE-1 | ROUGE-2 | ROUGE-L | \n|------------------------|--------|--------|--------|--------|------|---------|---------|---------|\n| Fine-tuned T5              | 21.42  | 13.79  | 10.06   | 7.38  | 8.94 | 31.00  | 13.85   |  25.78   | \n| HuatuoGPT              | 24.85  | 13.42  | 7.72   | 4.51   | 7.50 | 28.30   | 7.72    | 14.50   | \n| Fine-tuned HuatuoGPT（20% training data）| 25.05  | 14.15  | 9.23   | 6.32   | 7.56 | 30.89   | 9.55    | 20.95   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper develops a Chinese medical LLM, using data from both ChatGPT and doctor and adopting RLMF training stategy.\n2. Compared with other open-source LLMs, HuatuoGPT performs better on medical dialogue and QA.", "reasons_to_reject": "1. The contribution of this paper mostly lies in engineering.\n2. Since the experiements of single-turn/multi-turn dialogue generation are conducted on a filtered small test sets, and the experimental results of medical QA benchmarks are mostly worse than T5 (finetuned) except for cMedQA2, so the value of the experimental results doesn't seem to be significant. ", "questions_to_authors": "1. In line 335, \"When evaluated by doctors, we randomly sample 50 patient cases from 100 test cases\", but in Figure 4(b), the results still contain 100 cases.\n2. For medical QA, how are the results of fine-tuning the HuatuoGPT by using training data? ", "paper_topic_and_main_contributions": "This paper presents a large language model for Chinese medical consultation, HuatuoGPT, which leverages both distilled data from ChatGPT and real-world data from doctors in the SFT stage. To further synergize the strengths of both data sources,  they also adopt RLMF strategy for RL stage. Experiments are conducted on single-turn/multi-turn medical dialogue datasets and medical QA benchmarks, and achieve SOTA compared with open-source LLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PfAclACbk6", "reviewer": "Reviewer_Dcp8", "summary": "", "questions": "Although the win/loss ratios between the models are shown in Figure 4, I'm more curious about the diagnostic accuracy of each model.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1.Test set is limited, automated evaluation using GPT-4 may be able to use a larger test set.**\n\n**Reply**: We understand the concern about the limited test set. Our GPT-4 quota constraints restricted us to 100 questions for testing before the submission. With an increased quota after that, we expanded our tests to 500 questions from KUAKE-QIC instead of the original 100 questions. The extended results show that HuatuoGPT still wins ChatGLM-6b, Ziya-LLaMA-13B and ChatGPT; this confirms the validity of our model as the dataset size grows. We appreciate this suggestion and will incorporate these extended results in the revised version.\n\n| HuatuoGPT Win rate                            | 100 Questions (win/ tie/ fail) | 500 Questions (win/ tie/ fail) |\n|-------------------------------------|--------------------------------|--------------------------------|\n| HuatuoGPT vs ChatGLM-6b             | **43** / 18 / 39                   | **251** / 59 / 190                 |\n| HuatuoGPT vs Ziya-LLaMA-13B         | **50** / 13 / 37                   | **260** / 66 / 174                 |\n| HuatuoGPT vs ChatGPT                | **47** / 21 / 32                   | **234** / 55 / 211                 |\n\n> **Q2: Even though the data from different sources were processed by means of role-play and LLM-polished, their overall distributions are still very different. So how to ensure that a user-friendly response format and professional guidance are integrated after training, instead of adopting appropriate response formats for different data distributions.**\n\n**Reply**: We acknowledge the difference in distributions between the two groups data. The two means during SFT (i.e., role-play and LLM-polishing) could help it enjoy the both merits; but it is not guaranteed. To address this,  \nwe employed the RLMF to ensure the effective integration of a user-friendly response format and professional guidance after STF training. Technically, RLMF adopts the prompt of ChatGPT to emphasize a user-friendly response format while using doctors' references to reward consistency with professional doctors. Therefore, only the response that satisfies both feedback preferences can receive higher rewards.\n\n> **Q3. There is too little description of the data, such as the exact amount of data, length, etc.**\n\n**Reply**: We apologize for the oversight, but we incorporate the description in the appendix. For comprehensive data details, please refer to Table 3 in Appendix C. \n\n| Data Type       | Source                        | Number of Samples |\n|-----------------|-------------------------------|-------------------|\n| Instruction     | ChatGPT-generated                      | 61,400            |\n| Instruction     | Real-world | 69,768            |\n| Conversation    | ChatGPT-generated                       | 69,888            |\n| Conversation    | Real-world | 25,986  |\n\nIt provides a breakdown of the roughly 230,000 data points used in the SFT process, indicating the exact amount from each source. In the Instruction data, 61400 samples are distilled from ChatGPT and 69768 samples from real-world doctor-patient QAs. In Conversation data, 69888 samples are distilled from ChatGPT and 25986 samples from real-world doctor-patient conversations. We opted to place this table in the appendix due to limited space. We will incorporate more data specifics, as you've suggested, in our revised revision.\n\n> **Q4. This paper appears to be incremental, with a self-instruct method used for data construction, and RL with Mixed Feedback actually just includes real responses from doctors in the scoring, and the evaluation method is just based on GPT-4 comparisons.**\n\n**Reply**: Thank you for the feedback. Our main contribution lies in introducing an approach to effectively enjoy both merits of self-instruction data with real-world data, in a common two-stage training paradigm, which makes the HuatuoGPT model inherit the advantages of ChatGPT and the doctor to generate user-friendly and professional responses. \n\n**Data level**:  During the SFT stage, we propose a role-playing-based self-instruction and LLM-Polished method to narrow the gap between the distilled data and real-world data, ensuring a smoother integration of diverse data.\n\n**Reward level**: In the RL stage, we propose a mixed feedback RL method that aims for responses to be both patient-friendly and doctor-like through two sources of feedback: the prompt of ChatGPT and doctor references. It enables our model to integrate the two groups' data better because only the generated response that satisfies both feedback preferences can receive higher rewards.\n\nThe experimental results show that our model significantly outperforms the existing models in both automatic and manual evaluations, demonstrating the effectiveness of our method.\n \nFor evaluation, we also incorporate human evaluations and medical QA other than GPT-4 evolutions. This makes our assessment one of the most comprehensive evaluation protocols compared to concurrent works [e.g., Ziya, DoctorGLM, Bencao]. We acknowledge the challenges in evaluating medical language models.\n\n> **Q5. The presentation of Figure 5 is not very clear, does w/real mean SFT with real data only? Also, maybe add an experiment comparing HuatuoGPT and w/o RLMF, which would better show how much of a boost RLMF is.**\n\n**Reply**: We apologize for the confusion. \"w/real\" in Figure 5 refers to SFT conducted using only real data, which is a HuatuoGPT variant without distilled data and RLMF. Regarding your suggestion for a comparison between HuatuoGPT and w/o RLMF, we have conducted an additional experiment comparing the responses of GPT-4, HuatuoGPT, and w/o RLMF. The results in the following table show HuatuoGPT outperforms w/o RLMF.\n\n| HuatuoGPT Win rate                                        | Single turn              | Multi turn               |\n|-------------------------------------------------|--------------------------|--------------------------|\n| HuatuoGPT vs HuatuoGPT w/o RLMF| **51** / 26/ 23              | **49**/ 12/ 39               |\n\n**Q6: Suggesting to include a comparison between RLMF and RL (without reference when comparing) in the ablation experiments, I'm curious how much difference there is between the two?**\n\n**Reply**: Thank you for your constructive suggestion. We have added an experiment comparing HuatuoGPT between the RLMF and the RL without doctor's references. The results show using RLMF is better than using RL without doctor's reference, demonstrating the effectiveness of our method:\n\n| RLMF Win rate                          | Single turn  (win/ tie/ faile)                | Multi turn  (win/ tie/ faile)               \n|-----------------------------------|-----------------------------|----------------------------|\n|  RLMF vs RL w/o doctor reference | **46** / 14/ 40                 | **60**/ 7/ 33                  |\n\n**Q7: Although the win/loss ratios between the models are shown in Figure 4, I'm more curious about the diagnostic accuracy of each model.** \n\n**Reply**: To address your curiosity about diagnostic accuracy, we had doctors compare the diagnostic accuracy of the model's output on the Multi-turn Conversations of the experiments. In our experiments, we score the accuracy of the final advice given by the model and follow the following guidelines:\n1. Since it is a chatgpt simulation of a patient, we do not include in our calculations when the dialogue deviates from the original diagnosis and treatment context. We also do not include data in the calculation if the doctor did not give a clear conclusion in the real diagnosis.\n2. For the patient's diagnosis, we considered it accurate if the model mentioned the same diagnosis as the real doctor.\n3. For the treatment recommendation of a known condition, we consider it accurate if the model mentions the same treatment recommendation as the real doctor.\n\nThe diagnostic accuracy rates for different models are as follows:\n\n| Models                          | diagnostic accuracy                |      \n|-----------------------------------|-----------------------------|\n|  ChatGLM   |    0.23              |\n|  Ziya-LLaMA-13B   |   0.26           |\n|  ChatGPT   |   0.40               |\n|  HuatuoGPT   | 0.47             |\n\nAs can be seen, HuatuoGPT has significantly higher diagnostic accuracy than other models, though there is still room for improvement. With ChatGLM and Ziya, they can usually give very informative answers, but they are always too broad to get to the point. For ChatGPT, it has strong instruction following ability, but it is usually hard to give a clear diagnosis. We think these are the strengths of HuatuoGPT.\nFurthermore, it should be noticed that doctors will be very concerned about the accuracy of the diagnosis during the evaluation, which is the guideline we mentioned.\n\n##  Ethical Concerns:\n> **Q8: Concerns about the use of ChatGPT data**\n\n**Reply**: Thank you for raising concerns about the use of ChatGPT data.\n\nWe recognize the potential limitations of using ChatGPT as a data source: We primarily utilized ChatGPT to capture the format and tone suitable for structured and comprehensive responses, not as a source of medical accuracy.\n\nTo ensure medical correctness, we complemented this with training data from real doctors in the SFT with the Hybrid Data stage. In addition, real doctor responses also served as benchmarks during feedback collection from ChatGPT in the RL with Mixed Feedback stage.\n\nWe'd like to emphasize that our research is exploratory and for academic insights. We have no intention of deploying the model for medical practice due to its current diagnostic limitations.\n\n\n> **Q9: Concerns about potential misuse and associated risks**\n\n**Reply**: We share your concerns about potential misuse and risks.\n\nWe fully recognize the importance of ethical considerations in deploying and distributing models like HuatuoGPT.\n\nTo safeguard against misuse, we are contemplating restricted access, allowing only the scientific and professional communities to use our model.\n\nAdditionally, our research has been reviewed and approved by our institute's ethics board, ensuring it aligns with established ethical guidelines.\n\nWe deeply appreciate your concerns and are committed to taking all necessary precautions. Your feedback is invaluable, and we sincerely thank you for helping us enhance the integrity and safety of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- With supervised fine-tuning and reinforcement learning, the model's performance seems good, validating that this paradigm can be implemented well in the medical field.\n- The medical SFT data and pair-wise comparison data are valuable.", "reasons_to_reject": "- Test set is limited, automated evaluation using GPT-4 may be able to use a larger test set\n- Even though the data from different sources were processed by means of role-play and LLM-polished, their overall distributions are still very different. So how to ensure that a user-friendly response format and professional guidance are integrated after training, instead of adopting appropriate response formats for different data distributions.\n- There is too little description of the data, such as the exact amount of data, length, etc.\n- This paper appears to be incremental, with a self-instruct method used for data construction, and RL with Mixed Feedback actually just includes real responses from doctors in the scoring, and the evaluation method is just based on GPT-4 comparisons.\n- The presentation of Figure 5 is not very clear, does w/real mean SFT with real data only? Also, maybe add an experiment comparing HuatuoGPT and w/o RLMF, which would better show how much of a boost RLMF is.\n- Suggesting to include a comparison between RLMF and RL (without reference when comparing) in the ablation experiments, I'm curious how much difference there is between the two?", "questions_to_authors": "Although the win/loss ratios between the models are shown in Figure 4, I'm more curious about the diagnostic accuracy of each model.", "paper_topic_and_main_contributions": "This paper proposes HuatuoGPT, a highly effective medical model. It conducts supervised fine-tuning using both ChatGPT-distilled data and real-world medical data. Additionally, a mixed-feedback reward modeling approach is designed. The superiority of this model is demonstrated through automated evaluation with GPT4 and evaluation by expert doctors, outperforming numerous open-source models and ChatGPT. At the same time, great results are also achieved across various benchmarks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Justification_for_Ethical_Concerns": "This paper proposes a doctor-like language model to give a diagnosis. However, the data used for fine-tuning is partially extracted from ChatGPT, without any manual filtering or verification. While ChatGPT is a strong model, its efficacy as a medical diagnostic tool remains untested. This raises the question of how we can validate the accuracy of the derivative model, Huatuo. It's crucial to exercise extreme caution in this particular field.\n\nWhile I concur with the ethical statements put forth by the authors, I harbor concerns about potential misuse and serious risks associated with this model. Perhaps obtaining an Institutional Review Board (IRB) license could mitigate these risks to a certain extent."}}, {"review_id": "ygtykytyCU", "reviewer": "Reviewer_WjkR", "summary": "", "questions": "E What is the [computation budget](https://aclanthology.org/D19-1224/) to train HuatuoGPT? \n\nF Why HuatuoGPT performs similarly on Multi-turn Conversations compared to HuatuoGPT (w/ real) 0.66 vs 0.64, but so differently in Single-turn Conversations (0.575 vs 0.385) ? Could the reason be that your human data is multiround format only (no single round)?\n\nG Under multi-turn evaluation, you evaluate on the results from doctor model given patient input. But could patient-chatgpt hallucinate and produce irrelevant answer/question that may lead to a bad question/answer generated by the doctor model?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for your time and effort in reviewing our manuscript. Your insights are invaluable, and we appreciate the opportunity to clarify any ambiguities. Please find our responses to your comments below.\n\n> **Q1: Missing definition on multi-turn conversations task around line327. What is the input and expected output of this task. From figure 1, it seems you only have conversation history and patient-chatgpt’s utterance as input for doctor-model, but from the example in page 21, it seems like doctor-model also have access to clinical note (or patient case description) as doctor model’s input. Please specify the exact input and output for patient-chatgpt and doctor-model to help increase reproducibility.**\n\n**Reply**: Thank you for pointing out the lack of clarity in our description of the multi-turn conversations task. We understand the importance of clearly specifying each step to ensure reproducibility. Here's a more detailed breakdown:\n\n**Input for Patient-ChatGPT**:\nPatient-ChatGPT is provided with the \"patient case description,\" which serves as a background to help it simulate the role of a patient effectively, as mentioned in L327. After processing the patient case description, Patient-ChatGPT produces a simulated patient utterance.\n\n**Input for Doctor-Model**:\nThe Doctor-Model receives the conversation history and the simulated patient utterance from Patient-ChatGPT as its input. Contrary to what might have been implied on page 21, the Doctor-Model does not have access to the clinical note or the patient case description.\n\nThe example on page 21 was intended to showcase the background against which ChatGPT simulated the patient role, but it wasn't the direct input for the Doctor-Model. We'll work on clarifying this distinction in our revised version to prevent any confusion.\n\n\n> **Q2: You claim that Doctors’ Responses are not patient-friendly (short and etc) as your motivation in line145, and thus to train using gpt distilled data. While this may intuitively be true, but does training with gpt distilled data help improve patient-friendliness, any experiment result to show this?**\n\n**Reply**: We completely agree with the reviewer that it is essential to provide evidence for our claims. Here's a more detailed breakdown of our evidence:\n\n**Guidelines for Review**: As mentioned in L341, when we conducted evaluations using GPT4, one of our key metrics was the assessment of patient-friendliness. This is to ensure that the responses generated by the model met the patient's comfort and understanding needs.\n\n**Empirical Evidence**: We have presented a comprehensive case study in Table 9 in the appendix. A clear observation from this table is that the responses created using ChatGPT distillation data are more expansive and explanatory, making them more patient-friendly compared to the responses from doctors. To illustrate, consider the following examples in Table 9:\n| Content |\n|-------|\n| **User**: 我最近拉肚子，有泡沫黄水，怀疑是喝牛奶喝的，是不是喝牛奶的原因。谢谢。 (I’ve been experiencing diarrhea recently, with my stool appearing as frothy yellow liquid. I suspect it might be due to the milk I’ve been drinking. Could this be the reason? Thank you.)  |\n| **HuatuoGPT (w/ real data)**: 您好！根据你描述的症状，可能存在肠胃疾病的情况。此外，饮食习惯也会影响到这种情况发生。从中医角度来看，这主要与脾 胃功能失调有关。因为脾胃相表里，所以需要通过健脾胃的方法进行治疗。同时，平常饮食也需注意清淡易消化。 (Hello! Based on the symptoms you described, there might be a potential issue with your gastrointestinal system. Moreover, dietary habits can also affect the occurrence of such symptoms. From a traditional Chinese medicine perspective, this is primarily related to a disorder of spleen and stomach functions, since they are closely linked. Therefore, treatment would involve methods to strengthen the spleen and stomach. Additionally, you should consider a light and easily digestible diet in your daily eating habits.)  |\n| **HuatuoGPT (w/ distilled data)**: 您好，感谢您向我咨询。根据您所提供的症状描述，可能存在消化道炎症或者肠胃功能紊乱等情况。 首先，为了缓解腹泻症状，请注意以下几点：\n1. 多饮清水，避免脱水；\n2. 少量多餐，保持营养均衡，避免过饱或饿肚子；\n3. 避免摄入刺激性食物，例如辣椒、生姜等；\n4. 适当休息，避免剧烈运动和疲劳。\n\n其次，关于您目前所说的牛奶可能引起腹泻的问题，需要进一步了解您的个人情况才能做出判断。通常来讲，牛奶中含有丰富的 乳糖和蛋白质，而某些人群由于缺乏相应的酶类会产生乳糖不耐受或者蛋白质过敏等现象，从而引发腹泻。但也有很多人可以正 常地饮用牛奶并且没有这种反应。因此，建议您再观察一段时间，看看症状是否有改善，如果仍然持续，建议到医院就诊，排除 其他疾病后，如果需要，医生会给您更具体的治疗方案。\n (Hello, thank you for reaching out. Based on the symptoms you’ve described, there could be inflammation in the digestive tract or some form of gastrointestinal dysfunction.\nFirstly, to alleviate your diarrhea symptoms, please pay attention to the following points:\n1. Drink plenty of clear fluids to prevent dehydration.\n2. Eat small, frequent meals to maintain a balanced diet and avoid overeating or hunger.\n3. Avoid foods that can irritate the digestive system, such as chili peppers and raw ginger.\n4. Take adequate rest and avoid strenuous exercise and fatigue.\nSecondly, regarding the possibility you mentioned about milk causing diarrhea, further information about your personal circumstances would be needed to make a judgement. Typically, milk contains rich lactose and proteins. Certain individuals lack the enzymes to digest these substances, leading to lactose intolerance or protein allergies that can cause diarrhea. However, many people can consume milk normally without such reactions. Therefore, I suggest you continue to monitor your symptoms for some time to see if there is any improvement. If symptoms persist, I recommend you seek medical attention. Once other diseases have been ruled out, if necessary, your doctor can provide you with a more specific treatment plan.) \n\n> **Q3: Is gpt distilled data really needed? You hypothesis that GPT-4 exhibits poor performance in medicine compared to domain experts in Line048, which inspired you to use gpt distilled data. Any citation to support your hypothesis? From here, It seems like GPT-4 scored 92% on the USMLE Step 1 exam, surpassing the passing score of 75% by a large margin, which contradicts with your claim. Also your ablation study in figure 5, HuatuoGPT without gpt distilled data (w/ real only) did not differ much (only 0.02) from your HuatuoGPT on Multi-turn Conversations win-rate metric, especially when compared to huge gap between HuatuoGPT (w/ dist only) and HuatuoGPT (0.23).**\n\n**Reply**:  We firmly believe that ChatGPT distilled data is necessary as it enables models to provide detailed, well-organized, and rich-format content. As can be seen in Table 9 or provided examples in Question B, HuatuoGPT with only real data provides shorter, less organized answers compared to HuatuoGPT (w/ distilled data). Despite the latter providing more general answers, it exhibits doctor-like behavior by offering professional suggestions and follow-up questions.\n\nAlthough GPT-4 has performed well on the USMLE Step 1 exam, it does not perform as well as expert doctors in practice, as shared by doctors we have consulted. Additionally, there are very few people who consult GPT instead of an expert doctor, highlighting its limitations. Moreover, our HuatuoGPT model is trained in the Chinese language, and GPT-4 does not pass most examinations in Chinese, as indicated in [the study](https://arxiv.org/pdf/2308.08833.pdf).\n\n\n> **Q4:  Even say if the 0.02 improvement is indeed statistically significant in previous point C, would it be cost-effective, as the training data (w/ real) is almost the same as (w/ dist)? train twice the amount of time to just improve 0.02?**\n\n**Reply**:  We acknowledge the reviewer's concern about cost-effectiveness. During the instruction fine-tuning stage, the number of training examples or steps does not significantly affect the model's quality. For instance, [LIMA](https://arxiv.org/pdf/2305.11206.pdf) only uses 1000 examples for instruction fine-tuning. The behavior differences shown in the table could not be attributed to scaling up the instruction. Importantly, using distilled data (w/ dist) and real-world data (w/ real) exhibit distinct advantages in Single-turn Questions and Multi-turn Conversations, respectively. Combining both improves performance in both areas, indicating their complementarity.\n\nWe will clarify this point in the revised version.\n\n\n\n> **Q5: What is the computation budget to train HuatuoGPT?**\n\n**Reply**: Training HuatuoGPT requires approximately 200 A100-GPU hours, with 120 A100-GPU hours consumed during the SFT stage and about 80 A100-GPU hours consumed during the RLMF stage.\n\n\n> **Q6: Why HuatuoGPT performs similarly on Multi-turn Conversations compared to HuatuoGPT (w/ real) 0.66 vs 0.64, but so differently in Single-turn Conversations (0.575 vs 0.385) ? Could the reason be that your human data is multiround format only (no single round)?**\n\n**Reply**: Our real-world data includes both single and multi-round formats. However, as discussed in the paper, real-world data produces overly concise responses and lacks the necessary level of detail due to the time constraints faced by doctors during the diagnostic process. On the other hand, distilled data and ChatGPT generate detailed and well-presented responses. Therefore, HuatuoGPT (w/ real) learns the diagnostic ability of doctors to some extent, but its responses are neither detailed nor well-orgainzed, leading to its subpar performance in single-turn conversations. This underscores the need to use both real-world and distilled data.\n\n\n> **Q7: Under multi-turn evaluation, you evaluate on the results from doctor model given patient input. But could patient-chatgpt hallucinate and produce irrelevant answer/question that may lead to a bad question/answer generated by the doctor model?**\n\n**Reply**:  Indeed, ChatGPT, as a patient-chatgpt, may produce hallucinated or irrelevant answers/questions. However, in our experiments, we found that ChatGPT effectively generated questions with our well-designed prompt based on the provided patient information. It also responded accurately to the doctor's questions, with very few instances of asking irrelevant questions that could mislead the doctor model. We will include examples and an intriguing research direction for the future, and we leave this issue for further studies.\n\nThank you again for your insightful comments and questions. We believe that addressing these points will significantly improve our work, and we are grateful for the opportunity to clarify these aspects.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A The author supervised fine-tuned (SFT) a model which combined the strengths of both distilled data from ChatGPT and real-world data from doctors to train a medical large language model.\n\nB Other than previous hasty doctorbot models which only include SFT, the author applied a modified RLHF from the InstructGPT paper after SFT. They used mixed feedback from ChatGPT and doctors to reward the model for generating patient-friendly and doctor-like responses.\n\nC The author presented a comprehensive evaluation protocol that covered single-turn and multi-turn medical consultations, with both automatic (GPT-4 as evaluator) and human evaluation.\n\nD Ablation study shows that HuatuoGPT (w/real doctor data) performs better in multi-turn conversations while HuatuoGPT(w/ gpt distilled data) performs better in single-turn qa.", "reasons_to_reject": "A Missing definition on multi-turn conversations *task* around line327. What is the input and expected output of this task. From figure 1, it seems you only have conversation history and patient-chatgpt’s utterance as input for doctor-model, but from the example in page 21, it seems like doctor-model also have access to clinical note (or patient case description) as doctor model’s input. Please specify the exact input and output for patient-chatgpt and doctor-model to help increase reproducibility.\n\nB You claim that Doctors’ Responses are not patient-friendly (short and etc) as your motivation in line145, and thus to train using gpt distilled data. While this may intuitively be true, but does training with gpt distilled data help improve patient-friendliness, any experiment result to show this?\n\nC Is gpt distilled data really needed? You hypothesis that GPT-4 exhibits poor performance in medicine compared to domain experts in Line048, which inspired you to use gpt distilled data. Any citation to support your hypothesis? From [here](https://arxiv.org/pdf/2303.13375.pdf), It seems like GPT-4 scored 92% on the USMLE Step 1 exam, surpassing the passing score of 75% by a large margin, which contradicts with your claim. Also your ablation study in figure 5, HuatuoGPT without gpt distilled data (w/ real only) did not differ much (only 0.02) from your HuatuoGPT on Multi-turn Conversations win-rate metric, especially when compared to huge gap between HuatuoGPT (w/ dist only) and HuatuoGPT (0.23). \n\nD Even say if the 0.02 improvement is indeed statistically significant in previous point C, would it be cost-effective, as the training data (w/ real) is almost the same as (w/ dist)? train twice the amount of time to just improve 0.02?", "questions_to_authors": "E What is the [computation budget](https://aclanthology.org/D19-1224/) to train HuatuoGPT? \n\nF Why HuatuoGPT performs similarly on Multi-turn Conversations compared to HuatuoGPT (w/ real) 0.66 vs 0.64, but so differently in Single-turn Conversations (0.575 vs 0.385) ? Could the reason be that your human data is multiround format only (no single round)?\n\nG Under multi-turn evaluation, you evaluate on the results from doctor model given patient input. But could patient-chatgpt hallucinate and produce irrelevant answer/question that may lead to a bad question/answer generated by the doctor model?", "paper_topic_and_main_contributions": "This paper is about HuatuoGPT, a large language model for medical consultation in Chinese. The paper claims that HuatuoGPT can leverage both distilled data from ChatGPT and real-world data from doctors to generate patient-friendly and doctor-like responses. The paper also proposes a reinforcement learning method to further improve the model’s performance based on mixed feedback from ChatGPT and doctors. The paper evaluates HuatuoGPT on several Chinese medical benchmarks to show its effectiveness in roleplaying internet doctors and medical QA dataset, Experiment results show that HuatuoGPT outperforms existing open-source medical models and even ChatGPT in most cases.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Med-PaLM2 instead of Med-PaML2 in Table 6.\n\nTense error in “we *will* randomly switch the position of two responses.” in line354"}, "extra_text": {"Missing_References": "The paper is about taming Language Models To Be a *Doctor*. But the eval is only on doctors during *online* consultations. You should also consider a recent [competition](https://github.com/abachaa/MEDIQA-Chat-2023) that is *real-world* face-to-face consultation."}}]

Author rebuttal:

Meta-review: The paper introduces HuatuoGPT, a large language model trained for Chinese medical consultation tasks. The model leverages distilled data from ChatGPT and real-world medical data, along with a mixed-feedback reinforcement learning strategy (RLMF). The paper shows that the model performs well in both single-turn and multi-turn medical dialogue as well as medical QA tasks, often outperforming existing open-source large language models (LLMs). The reviewers generally agree that the paper is technically sound and reports state-of-the-art results. However, they express concerns about the paper's incremental contributions and lack of significant experimental validation. All reviewers highlighted that the work appears to be an engineering-focused extension of existing techniques rather than a novel scientific contribution. There are concerns about the limited size and representativeness of the test sets used for validation, which raises questions about the generalizability of the results. Considering the paper's technical soundness and its ability to contribute to the field in a meaningful, if incremental, manner, the paper is thus recommended to be accepted to the "Findings" track. To further enhance the paper's value, the authors may wish to consider addressing concerns raised by the reviewers.
Decision: Accept-Findings