NoteChat: A Dataset of Synthetic Patient-Physician Conversations Conditioned on Clinical Notes (2024)

Abstract: We introduce NoteChat, a novel cooperative 001 multi-agent framework leveraging Large Lan002 guage Models (LLMs) to generate patient003 physician dialogues. NoteChat embodies the 004 principle that an ensemble of role-specific 005 LLMs, through structured role-play and strate006 gic prompting, can perform their assigned 007 roles more effectively. The synergy among 008 these role-playing LLMs results in a cohe009 sive and efficient dialogue generation. Eval010 uation on MTS-dialogue (Abacha et al., 2023; 011 Ben Abacha et al., 2023), a benchmark 012 dataset for patient-physician dialogues-note 013 pairs, shows that models trained with the aug014 mented synthetic patient-physician dialogues 015 by NoteChat 1 outperforms other state-of-the016 art models for generating clinical notes. Our 017 comprehensive automatic and human evalua018 tion demonstrates that NoteChat substantially 019 surpasses state-of-the-art models like ChatGPT 020 and GPT-4 up to 22.78% by domain experts in 021 generating superior synthetic patient-physician 022 dialogues based on clinical notes. NoteChat 023 has the potential to engage patients directly and 024 help clinical documentation, a leading cause of 025 physician burnout (Budd, 2023). 026

text: Findings of the Association for Computational Linguistics ACL 2024, pages 15183–15201 August 11-16, 2024 ©2024 Association for Computational Linguistics
We introduce NoteChat, a novel cooperative001 multi-agent framework leveraging Large Lan-002 guage Models (LLMs) to generate patient-003 physician dialogues. NoteChat embodies the004 principle that an ensemble of role-specific005 LLMs, through structured role-play and strate-006 gic prompting, can perform their assigned007 roles more effectively. The synergy among008 these role-playing LLMs results in a cohe-009 sive and efficient dialogue generation. Eval-010 uation on MTS-dialogue (Abacha et al., 2023;011 Ben Abacha et al., 2023), a benchmark012 dataset for patient-physician dialogues-note013 pairs, shows that models trained with the aug-014 mented synthetic patient-physician dialogues015 by NoteChat 1 outperforms other state-of-the-016 art models for generating clinical notes. Our017 comprehensive automatic and human evalua-018 tion demonstrates that NoteChat substantially019 surpasses state-of-the-art models like ChatGPT020 and GPT-4 up to 22.78% by domain experts in021 generating superior synthetic patient-physician022 dialogues based on clinical notes. NoteChat023 has the potential to engage patients directly and024 help clinical documentation, a leading cause of025 physician burnout (Budd, 2023).026
1 Introduction027
Clinical dialogue is an essential part of clinical028 workflow. Clinical documentation is a two-step pro-029 cess. It first engages patients through conversation030 to collect patient-specific information such as de-031 mographic information, family history of diseases,032 and signs and symptoms and then generates elec-033 tronic health records (EHRs) from the dialogues.034 Currently clinical documentation is mainly done035 by physicians at both steps, a labor intensive pro-036 cess that contributes to physician burnout, defined037 as a state of emotional, physical, and mental ex-038
1Our synthetic patient-physician dialogue data is in supplementary material and will be publicly available together with all codes and prompts upon acceptance.
haustion caused by prolonged stress in the work- 039 place (Ortega et al., 2023; Budd, 2023). In this 040 paper, we introduce NoteChat, a novel cooperative 041 multi-agent framework leveraging Large Language 042 Models (LLMs) to generate patient-physician con- 043 versations conditioned on clinical notes. NoteChat 044 has the potential to help clinical documentation at 045 both steps. 046
NoteChat leverages LLMs, powerful artificial 047 intelligence (AI) systems extensively trained on 048 a large amount of textual data which represent a 049 significant breakthrough in AI (Brown et al., 2020; 050 Longpre et al., 2023). The GPT series by Ope- 051 nAI (OpenAI, 2023) have demonstrated impres- 052 sive outcomes and hold significant potential in rev- 053 olutionizing a broad range of sectors, including 054 marketing, education, and customer service. How- 055 ever, recent work (Ben Abacha et al., 2023) found 056 ChatGPT does not perform well enough in gener- 057 ating either patient-physician encounter conversa- 058 tion or its corresponding EHR notes. The explo- 059 ration of open-source LLMs (e.g., LLaMA2) (Tou- 060 vron et al., 2023; Taori et al., 2023; Chiang et al., 061 2023) in the medical field remains relatively un- 062 tapped (Gilson et al., 2023), despite their immense 063 potential for transforming healthcare communica- 064
1 15183
tion and decision-making (Abacha and Zweigen-065 baum, 2015). We suspect that one main reason is066 the lack of high-quality medical datasets that meet067 various needs.068
Although efforts have been made to create bench-069 mark datasets, the datasets relevant to clinical doc-070 umentation are small scale (Abacha et al., 2023;071 Ben Abacha et al., 2023; Yim et al., 2023). Yunxi-072 ang et al. (2023) collected 100k real-world patient-073 physician conversations from online medical con-074 sultation websites as ChatDoctor dataset. Xiong075 et al. (2023) converted the ChatDoctor data into076 Chinese and additionally added relevant Chinese di-077 alogue (Zeng et al., 2020) and question-answering.078 However, none of the aforementioned datasets in-079 clude dialogue-note pairs. Moreover, as indicated080 in Table 1, the maximum average number of utter-081 ances in the existing datasets (Zeng et al., 2020)082 is 3.3, which is a typical representation of online083 medical consultation websites but markedly less084 than face-to-face communication between patient085 and physician encounters (Drew et al., 2001).086
The primary challenge of creating benchmark087 datasets in the clinical domain is HIPAA regulation088 (Rindfleisch, 1997; Annas, 2003). This impediment089 prevents the use of state-of-the-art LLMs, such090 as GPTs, on real patient data. NoteChat circum-091 vents it by generating high-quality synthetic patient-092 physician conversations conditioned on clinical093
notes. This synthetic dialogue data can then be 094 used to help train downstream tasks such as clinical 095 note generation conditioned on patient-physician 096 dialogues. Therefore, NoteChat helps both steps 097 of clinical documentation, this is in contrast to the 098 existing models, which mainly focused on clinical 099 dialogue generation only (Yunxiang et al., 2023; 100 Zeng et al., 2020). 101
In this study, we introduce NoteChat, which is 102 built upon a novel cooperative multi-agent frame- 103 work to generate synthetic patient-physician con- 104 versations conditioned on clinical documents (e.g., 105 HIPAA-compliant clinical notes 2 and case re- 106 ports 3). NoteChat comprises three modules: Plan- 107 ning, Roleplay, and Polish. The planning module is 108 responsible for knowledge organization, aiming to 109 decrease hallucination and enhance the consistency 110 of medical logic. The Roleplay module includes 111 two ChatGPT agents 4 take on the roles of physi- 112 cian and patient, respectively. This setup facilitates 113 the generation of interactive dialogues in a looped 114 format. The Polish module is then utilized to re- 115 fine these dialogues, ensuring they are more closely 116 aligned with the expectations and preferences of 117 medical professionals, following the feedback and 118 suggestions obtained from physicians and medical 119
2https://github.com/abachaa/MTS-Dialog 3https://github.com/zhao-zy15/PMC-Patients 4We use OpenAI’s GPT-3.5 model gpt-3.5-turbo-0613.
2
students. Extensive automatic and human evalua-120 tions demonstrate the efficacy of our cooperative121 multi-agent framework and show that NoteChat122 holds great promise for promoting high-quality syn-123 thetic patient-physician conversations.124 In summary, our contributions are as follows:125 • We created a novel multiple roleplay LLMs co-126 operating framework and successfully deployed127 the framework for the task of generating patient-128 physician conversations conditioning on clinical129 notes. Although synthetic data generation is an130 active field in the clinical domain especially to131 overcome privacy concerns (Pereira et al., 2022;132 Shafquat et al., 2022; Mishra et al., 2023), to133 our knowledge, this is the first work to present134 an instance of multiple LLMs cooperating (Li135 et al., 2023a) to complete a patient-physician136 conversation conditioned on clinical notes.137
• We evaluated the quality of the synthetic patient-138 physician conversations generated by NoteChat139 with the state-of-the-art OpenAI’s ChatGPT140 and GPT-4 using extensive intrinsic and ex-141 trinsic evaluation methods. Through compre-142 hensive human evaluations, we demonstrate143 that NoteChat holds promise to generate high-144 quality synthetic patient-physician dialogues.145
• In this study, we released the first large and high-146 quality synthetic dialogue data conditioned on147 167k case reports that can be used to train both148 dialogue systems and EHR note-generation sys-149 tems using dialogues.150
2 Methods151
2.1 Data Resource and Preprocessing152
PMC-Patients is a comprehensive dataset com-153 prising 167K patient case reports and relations154 extracted from a diverse range of case reports155 available in the PubMed Central (PMC) reposi-156 tory (Zhao et al., 2023). PMC-Patient dataset en-157 compasses a vast array of case reports, many of158 which pertain to rare conditions. To maintain the159 quality of the generated dialogue in our study, we160 instruct ChatGPT to exclude exceptionally rare161 cases. Furthermore, we also instruct ChatGPT to162 omit case reports related to animal diseases, as they163 typically bear less relevance to our objective of164 focusing on human clinical dialogues.165
MTS-Dialog is a new collection (Abacha et al.,166 2023; Ben Abacha et al., 2023) of 1.7k short167 patient-physician conversations and corresponding168
summaries with section headers and contents fol- 169 lowing SOAP format (Podder et al., 2021) to foster 170 advancements in the field of automatic clinical note 171 generation from patient-physician conversations. 172 This 1.7k short version dataset has a correspond- 173 ing long version (Yim et al., 2023) of 87 complete 174 dialogues and clinical notes, all of which we use 175 for our evaluation. However, due to the API’s strin- 176 gent maximum token restriction, incorporating the 177 complete dialogue into a single prompt proved im- 178 practicable. Consequently, we implemented a strat- 179 egy that involved segmenting a clinical note into 180 several sections according to the traditional SOAP 181 format 5. We used each section header to construct 182 a distinct prompt with the corresponding content 183 in the note, thereby aiding the model in generating 184 individual chats for every section. We added a cor- 185 responding postprocessing step for MTS-Dialog 186 with the Combine Prompt in Appendix Table 14, 187 where we concatenated all the small chats from 188 different sections to create a complete dialogue. 189
2.2 NoteChat: Generating patient-physician 190 dialogues from notes in the GPT Era 191
To ensure that our synthetic datasets closely resem- 192 ble authentic dialogues, we first use the prompts 193 in Appendix A.2 to guide the roleplay of Chat- 194 GPT and GPT4 in generating high-quality data 195 as our baselines. In this section, we introduce 196 our NoteChat Framework for this task. All our 197 NoteChat experiments in this paper are based on 198 ChatGPT API (gpt-3.5-turbo), but NoteChat can be 199 used in any model that can handle the instructions. 200
2.2.1 Main dialogue generation loop 201 Planning module Typically, a physician’s diag- 202 nostic process adheres to a logical sequence, which 203 may be outlined as follows (First et al., 2013; John- 204 son, 2003; Tsichlis et al., 2021): 1) Eliciting symp- 205 toms, such as chest pain, 2) Inquiring about the 206 duration of these symptoms, 3) Obtaining medical 207 history, including personal and familial records, 4) 208 Conducting diagnostic tests, 5) Reaching a conclu- 209 sion and prescribing appropriate medication. Thus, 210 an effective dialogue dataset should accurately re- 211 flect the logical sequence of real-world interactions 212 between physicians and patients. Therefore, before 213 generating dialogues, it is crucial to ensure that 214 the model follows such logic. However, we found 215 models often tend to overlook crucial information, 216
5SOAP structure details can be found in the Appendix A.1.
3
create hallucination information, or messily skip217 content that should logically be in the first half of218 the dialogue and go to generating first with con-219 tent that should logically appear later. This is of-220 ten caused by the LLMs lacking sufficient medical221 knowledge (Dave et al., 2023) or low-level plan-222 ning abilities (Valmeekam et al., 2023).223
To circumvent these issues, we first extract clini-224 cal domain-specific keywords using CUI (Clinical225 Uniform Identifier) from MedSpaCy (Eyre et al.,226 2021) with QuickUMLS (Soldaini, 2016) and re-227 quire the LLM to build dialogues around these key-228 words exclusively, where we design the prompt in229 Appendix Table 11 with the list of keywords to help230 the LLM generates the dialogue draft. With this,231 we inject external clinical knowledge resources for232 semantic grounding to reduce hallucination. The233 Planning module is responsible only for high-level234 planning, which pertains to the general distribu-235 tion of different pieces of information within the236 dialogue. However, the control of each specific237 utterance at a low level is delegated to the Roleplay238 module (2.2.1). Therefore, the output of the Plan-239 ning module is not this draft, but a checklist. Each240 CUI in the checklist is extracted in sequence from241 the generated draft. Then, the Planning module will242 accompany the entire Roleplay module. That is,243 every time the Roleplay module completes a new244 round of dialogue generation, the planning module245 will count the newly added CUIs in the dialogue246 and remove them from the checklist. Therefore,247 the Planning module not only assumes the respon-248 sibility for the correct correlation of the facts but249 also helps the entire conversation narrow in a more250 definite direction until the end.251
Roleplay module The dialogue draft we gener-252 ated in the Planning module is not high-quality dia-253 logue data. Previous work (Yunxiang et al., 2023)254 shows that dialogues generated by a single LLM255 often have issues in language diversity and role ho-256 mogeneity. These are manifestations of the short-257 comings of LLMs in handling low-level planning258 for each utterance in an entire dialogue. There-259 fore, in order to generate better quality dialogues,260 we use the checklist in the Planning module to261 generate multiple rounds of dialogues using two262 LLMs to play the roles of patients and physicians,263 respectively. This strategy enables us to use distinct264 prompts based on different requirements of the cor-265 responding role so that the physician’s responses266 appear more professional and the patient’s dialogue267
sounds more normal. Furthermore, we can control 268 the direction of each dialogue round by modify- 269 ing the prompts. More specifically, we determine 270 the keywords covered in each round based on the 271 current checklist, allowing two roleplay LLMs to 272 advance the dialogue further and maximize the cov- 273 erage of the keywords. We then let the Planning 274 module update the checklist. Subsequently, we let 275 the patient-LLM respond to the physician in as col- 276 loquial a manner as possible, ensuring the patient’s 277 utterance lay language style. All prompts can be 278 found in Table 12. 279
Polish module Although the two modules of 280 Planning and Roleplay bring NoteChat more 281 fine-grained control over LLM, restoring patient- 282 physician dialogue from clinical notes requires 283 LLM to balance several challenging requirements, 284 including the planning of key information in the 285 clinical note, reasonable information not occurring 286 in the note but would appear in the dialogues, the 287 language style characteristics of different roles, and 288 the authenticity after combining everything into 289 one complete dialogue. In the previous Planning 290 and Roleplay modules, LLMs will promote new di- 291 alogues based on historical dialogues. Inspired by 292 recent work of rethinking and reranking (Gabriel 293 et al., 2021; Cobbe et al., 2021; Ravaut et al., 2022; 294 Jiang et al., 2022; Shinn et al., 2023), we added 295 the Polish module to give LLM another chance for 296
4
self-reflection and correction post-Roleplay mod-297 ule. To do this, we invited human experts who298 summarized the rules based on the preliminary re-299 sults of NoteChat to help our synthetic data align300 with experts’ preferences, and they came up with301 10 special rules: 1) Make the conversation as collo-302 quial as possible, 2) Increase the number of rounds303 of interaction, 3) Professional terms and vocab-304 ulary should come from the physicians, and pa-305 tients should be more colloquial, 4) Basic symp-306 toms and medical history should come from the307 patient, not the physician, 5) The patients’ self-308 reported signs and symptoms should be around the309 inputs, 6) Physician inquiries should be logical, 7)310 If there are multiple consultation records, you can311 split a conversation into multiple ones and then link312 them with transfer words (e.g., a few days later),313 8) Range of rounds of interaction, 9) Must contain314 the given keywords, 10) Do not generate duplicate315 information. Specifically, we added these require-316 ments to the Polish Prompt in Appendix Table 13317 and asked the LLM to polish the existing dialogue318 accordingly. We found that multiple iterations of319 the Polish step can improve the quality of the final320 synthetic dialogue 7.321
3 Automatic Evaluation322
MTS-Dialog provides the human-annotated ground323 truth conversation data for every clinical note, but324 the PMC-Patient dataset only has case reports. So,325 we use intrinsic evaluation for MTS-Dialog syn-326 thetic data but extrinsic and human evaluation for327 PMC-Patient synthetic data.328
3.1 Intrinsic Evaluation329
We measure this task of note-to-conversation from330 four aspects of the MTS-Dialog dataset.331 Similarity We use ROUGE-F1 scores (Lin, 2004)332 to measure the similarity of the generated conver-333 sation and the references.334 Factuality We follow recent work (Adams et al.,335 2023; Ramprasad et al., 2023) using medical con-336 cepts to evaluate factuality and make some im-337 provements. Specifically, we use QuickUMLS338 (Soldaini, 2016) to extract medical concepts from339 model-generated dialogues and ground truth di-340 alogues to get two corresponding concept lists.341 Then, we calculate the overlap of medical concept342 lists between two documents, offering insight into343
7After balancing the time, cost, and final performance, we set the number of iterations to 2 in our experiments
the model’s grasp of medical knowledge and termi- 344 nology. In Table 3, we report the Concept-P/R/F1 345 as the Factuality metric. 346 Extractiveness We calculate the ROUGE-F1 of src- 347 >hypo (clinical note to model-generated dialogue) 348 as our extractiveness metrics to demonstrate how 349 much information in dialogue is extracted from 350 the clinical note. For AI, a shortcut to improve 351 Factuality is to improve Extractiveness. However, 352 recent work shows increasing the factuality by this 353 way might not be ideal in many scenarios (Ladhak 354 et al., 2022; Goyal et al., 2022). 355 Diversity We use Self-BLEU (SBLEU) (Zhu et al., 356 2018) to evaluate the diversity of the generated 357 conversation for the patient utterances, physician 358 utterances, and overall. 359
3.2 Extrinsic Evaluation 360 Medical Chat Assistant: We used the PMC- 361 Patient synthetic dialogues generated by ChatGPT, 362 GPT4, and NoteChat to fine-tune the LLaMA2- 363 7B 8, where we only used physician utterances as 364 the training labels. Then, we evaluated these fine- 365 tuned LLaMA2 chatbots on the ground truth dia- 366 logues from MTS-Dialog. For evaluation, recent 367 work shows a higher human evaluation correlation 368 for GPT-4 eval than traditional metrics (Liu et al., 369 2023b; Gao et al., 2023; Fu et al., 2023; Zheng 370 et al., 2023), so we also used the GPT4 preference 371 as measurements to evaluate chatbots’ response 372 quality. Specifically, we instruct GPT4 to give pref- 373 erence ranking 9 based on the conversation history 374 and the real response. We follow Yao et al. (2023) 375 to report the Mean Reciprocal Rank (MRR) (Radev 376 et al., 2002) of each model’s final ranking in Fig- 377 ure 2. Generally, a higher MRR implies that evalu- 378 ators have a better alignment with the evaluators’ 379 preferences. 380 Conversation2Note and Note2Conversation: We 381 also used the NoteChat dataset as data augmen- 382 tation for two MTS-dialog tasks. We used the 383 same evaluation metrics (ROUGE) following Ben 384 Abacha et al. (2023). 385
3.3 Automatic Evaluation Results 386 The intrinsic evaluation results, as illustrated 387 in Table 3, show that the overall similarity of 388 the conversations generated by NoteChat and Hu- 389 man (MTS-dialog ground truth) is higher than that 390
8https://huggingface.co/meta-llama/Llama-2-7b-chat-hf 9Prompts can be found in Appendix 8.
10All experiments are done under the zero-shot setting.
5
of GPT4 and ChatGPT baselines. GPT4 outper-391 formed NoteChat and ChatGPT in both factuality392 and extractiveness metrics. NoteChat outperformed393 ChatGPT in factuality but had a lower and closer394 to human extractiveness score. In Section 4.4, we395 will discuss the impact of the different factuality396 and extractiveness scores of the three methods on397 human expert preferences on our task. Finally, we398 found that the diversity of NoteChat, especially399 for patient utterances, is significantly better than400 the baselines. The extrinsic evaluation Medical401 Chat Assistant results are illustrated in Figure 2.402 In this experiment, LLaMA2-7B is first fine-tuned403 on different PMC-Patient synthetic conversations.404 Then we use MTS-dialog as the evaluation dataset.405 NoteChat-based LLaMA2 has the highest score,406 indicating the most preferred by GPT4 when gen-407 erating real physician utterances. It is worth not-408 ing that this evaluation is also a kind of transfer409
learning because the model is only trained on dif- 410 ferent versions of PMC-Patient synthetic dialogue 411 (NoteChat, ChatGPT, GPT4) and then tested its 412 zero-shot performance on human-labeled dialogue 413 in MTS-dialog. The extrinsic evaluation Con- 414 versation2Note and Note2Conversation results 415 are illustrated in Table 4. We found that train- 416 ing on NoteChat-only can observe significant im- 417 provements in MTS-dialogue test results. The 418 best results can be obtained if NoteChat is used 419 as data augmentation of the original MTS-dialogue 420 training data. Therefore, the results of this ex- 421 trinsic evaluation show that the models trained on 422 the NoteChat dataset are generalizable to the real 423 human-annotated dataset. 424
4 Human Evaluation 425
To assess the quality of synthetic conversations 426 generated by different methods (ChatGPT, GPT-4, 427 NoteChat), we conducted a human evaluation using 428 crowd-sourcing and domain experts. 429
4.1 Human Evaluation Settings 430
The goal of expert evaluation is to have human 431 domain experts evaluate whether these machine- 432 generated conversations are comparable to real 433 patient-physician encounter conversations from a 434 professional perspective (e.g. medical common- 435 sense, knowledge, logic). To do so, we recruited 436 5 medical practitioners11, and their tasks are to 437 read clinical notes and provide qualitative feedback 438 on whether the machine-generated dialogues can 439 be defined as high-quality patient-physician inter- 440 actions in terms of factual accuracy and logical 441 coherence; if not, how should they be improved? 442
The goal of crowd evaluation is to allow the 443 general public to provide ratings for different syn- 444 thetic conversations based on their lived experi- 445 ence. Since the crowds do not have professional 446 medical knowledge, participants will first read the 447 clinical notes and medical expert annotated conver- 448 sations as references for high-quality data and then 449 rank different machine-generated conversations for 450 quantitative measurement of their preference. We 451 recruited 10 human evaluators to participate in our 452 crowd evaluation. 12 453
11Four licensed physicians and one medical student with hospital internship experience. These experts were not involved in the research, only the human evaluation.
12All the evaluators have bachelor’s degrees but do not have any medical education background.
6
4.2 Human Evaluation Measurements454 We mainly use human preference as measurements455 to evaluate synthetic conversation quality. Specif-456 ically, the participants are provided with the fol-457 lowing instructions “The following three conver-458 sations are generated by AI based on this clinical459 note. Please rank them according to the quality460 you think, from high to low.”. We collect the prefer-461 ence ranking from experts, crowds, and GPT4. We462 report the Mean Reciprocal Rank (MRR) of each463 model’s final ranking in Figure 3.464
4.3 Human Evaluation Outcome465 All the preference feedback from experts, crowds,466 and AI are shown in Figure 3. In the most crucial467 results concerning expert preferences, NoteChat’s468 MRR score significantly outperforms that of GPT4,469 indicating that from an expert’s perspective, the470 quality of dialogue data from NoteChat is higher.471 In terms of preferences among the crowds and AI,472 NoteChat also clearly surpasses GPT4, demonstrat-473 ing consistency with expert preferences. Finally,474 in all three human evaluations, both NoteChat and475 GPT4 perform better than ChatGPT.476
4.4 Heuristic Evaluation with Experts477 We interviewed 5 medical practitioners:478 Q1) What are the shortcomings of AI syn-479 thetic conversation compared with real-world480 patient-physician encounter conversation? Ex-481 perts think that synthetic conversations cover too482 much information from the clinical note compared483 to real-world conversations, because some factual484 information is not provided to note through conver-485 sation (such as lab test results). For example, in486 Table 5 Example 1, the detailed dosage information487 will be not in the conversation. In Example 2, the488 patient acts too professionally. In the answer, a lot489 of medical knowledge that physicians will know is490 described by the patient.491 Q2) What is the difference between Chat-492 GPT, GPT4, and NoteChat synthetic conversa-493 tions? All medical practitioners believe that GPT4494
and NoteChat lead ChatGPT in terms of factual- 495 ity. Since our NoteChat is based upon ChatGPT, 496 this human observation shows that our modules 497 successfully inject medical concept knowledge to 498 improve the factuality level from ChatGPT to the 499 level of GPT4. So, as shown in Figure 3, ChatGPT 500 is ranked last in all cases. 501
Regarding the comparison between NoteChat 502 and GPT4, medical practitioners actually believe 503 that the data quality of NoteChat-synthetic conver- 504 sations is generally better than the GPT4 synthetic 505 dataset, which aligns with their expert preference 506 in Figure 3. We further conducted a heuristic eval- 507 uation to explore the reason here as well as the 508 deficiency of NoteChat and GPT4 synthetic con- 509 versations and potential improvement. We further 510 conducted a heuristic evaluation to explore the rea- 511 son here as well as the deficiency of NoteChat 512 and GPT4 synthetic conversations and potential 513 improvement. First of all, GPT4 prefers to copy 514 the information directly in the note to meet the 515 requirements of factuality, but this will make the 516 conversation unreal. In Table 5 Example 2, the 517 information is highly summarized and put together 518 on the note, but it is unnatural for the same content 519 to appear directly in the dialogue. Compared with 520 the utterance generated by GPT4, a better way is to 521 use multiple conversation rounds to obtain informa- 522 tion one by one. This is a problem common to all 523 AIs in this paper, but GPT4’s problem is most ob- 524 vious. Second, in reality, physicians are expected 525 to not only answer questions but also advance the 526 discussion by asking professional questions. We 527 observe that the physician in NoteChat is more 528 likely to advance the conversation compared to the 529 physician in GPT4 due to our Roleplay module. 530
To better control language models, it’s impor- 531 tant to specify which information is spoken by the 532 physician and which by the patient. In the Table 5 533 Example 3, GPT-4 let the patient speculate about 534 their symptoms and dismiss physical activities as a 535 cause. Using a specific prompt, the NoteChat Role- 536 play module was adjusted to ensure both the physi- 537 cian and patient roles are accurately portrayed and 538 cooperate logically. Finally, The dialogue should 539 start like a real conversation, with the patient shar- 540 ing symptoms and medical history. Usually, doc- 541 tors don’t know a patient’s history, so patients need 542 to express or be asked about their symptoms and 543 history. This approach sets the direction for tests 544 and treatment plans. In GPT-4 generated dialogues, 545
7
this format should be followed, but often, the physi-546 cian character incorrectly presents this information547 first, which is not typical in real clinical settings548 (Example 5 of Table 5).549
5 Related Work550
Clinical note and conversations generation: A551 task closely related to our work, but with an in-552 verse direction, is the automatic generation of clini-553 cal notes from patient-physician conversations (Kr-554 ishna et al., 2020; Song et al., 2020; Yim and555 Yetisgen-Yildiz, 2021; Su et al., 2022; Yao et al.,556 2023). Recently, the MEDIQA-Chat 2023 14 in-557 troduced tasks in both directions (Dialogue2Note558 Summarization and Note2Dialogue Generation).559 However, their dataset is either private or lim-560 ited to less than 2k examples. One of the main561 themes of recent data-centric AI is the synthetic562 data to overcome privacy concerns (Pereira et al.,563 2022; Shafquat et al., 2022; Mishra et al., 2023).564
13Due to the obvious gap in factuality of ChatGPT, our cases focus on the difference between NoteChat and GPT4.
14https://sites.google.com/view/mediqa2023
To the best of our knowledge, we are the first to 565 introduce a large-scale publicly available patient- 566 physician conversation dataset in English, each ac- 567 companied by corresponding medical documents, 568 with an average number of utterances exceeding 569 20 rounds. In addition, our extrinsic eval shows 570 that the NoteChat can be used as auxiliary data 571 for Conversation2Note or Note2Conversation tasks 572 and can also be used as a synthetic medical dia- 573 logue dataset alone to engage patients directly and 574 help clinical documentation (Zhang et al., 2023; Li 575 et al., 2023b; Wang et al., 2023; Liu et al., 2023a; 576 Xiong et al., 2023; Zeng et al., 2020). 577
Multiple LLMs cooperation: Our work builds 578 upon the recent advances in deploying two LLMs 579 as cooperative agents (Panait and Luke, 2005) for 580 multi-round conversation generation. In particular, 581 NoteChat is inspired by CAMEL (Li et al., 2023a), 582 which assigns roles to two LLMs (e.g. student and 583 teacher) in order to facilitate conversation between 584 the two agents for a particular task (e.g. teach- 585 ing). Similar to CAMEL’s findings, we found that 586 roleplay by itself may hallucinate or generate fake 587 replies that repeat most of the previous utterances. 588 To solve this issue, we proposed a novel Planning 589 module to ground agents to certain keywords. Cho 590 et al. (2023) also addresses the challenges of us- 591 ing LLM to craft a dialogue dataset with speci- 592 fied personas. They emphasize the importance of 593 grounding and context in conversation generation. 594 Similarly, NoteChat relies on structured clinical 595 notes segmented using the SOAP format to provide 596 context for our dialogue synthesis to diagnose a 597 patient. However, their work is limited to gener- 598 ating open-domain dialogue, while we focus on 599 task-oriented dialogue. 600
6 Conclusion 601
In this study, we present NoteChat, a cooperative 602 multi-agent framework leveraging LLMs for gen- 603 erating synthetic patient-physician conversations 604 conditioned on clinical notes. NoteChat consists of 605 Planning, Roleplay, and Polish modules. Extensive 606 evaluations demonstrate that NoteChat facilitates 607 high-quality synthetic patient-physician conversa- 608 tions, underscoring the untapped potential of LLMs 609 in healthcare and offering promising avenues for 610 the intersection of AI and healthcare. 611
8
7 Limitations and Ethical Considerations612
This study offers valuable insights, but with a few613 limitations, we would like to note.614
Due to cost and time constraints, we could not615 try out many possibilities and alternatives in this616 paper. First of all, the current amount of data for617 human evaluation is not particularly sufficient. We618 are conducting more human evaluations. Secondly,619 due to cost issues, we currently do not use GPT-4620 extensively to try the NoteChat pipeline. When621 OpenAI updates the Stateful API 15, we will use622 this version to generate NoteChat-GPT4. Third,623 we extracted relevant UMLS-CUI codes for our624 Planning module, aiming to guide subsequent con-625 versations around these critical terms. Such a626 checklist can help our pipeline improve factual-627 ity (Asai et al., 2023; Huang et al., 2023), and can628 be very flexibly combined with other tools to meet629 different purposes, like information retrieval (Khat-630 tab et al., 2022), entity&relation extraction (Cai631 et al., 2023), medical jargon extraction (Kwon et al.,632 2022), causal inference (Yuan et al., 2023), ev-633 idence and reasoning path retrieval (Asai et al.,634 2019, 2021), and many other knowledge injection635 ideas (Fei et al., 2021; Yao and Yu, 2021).636
Consider Privacy Implications, LLMs can637 present privacy concerns in using clinical notes to638 generate patient-physician conversation, potentially639 violating HIPAA regulations. However, in this640 study, all experiments were sourced from publicly641 available real patient data collected from research642 articles with at least CC BY-NC-SA license. We643 also present an approach for generating synthetic644 conversations from case reports in the PubMed645 Central repository.646
Consider Biases, LLMs trained on vast amounts647 of text data may inadvertently capture and repro-648 duce biases present in the data. For example, they649 may prefer certain questions related to Metformin650 or link particular health conditions to specific pop-651 ulations. Thus the physician bot trained from our652 synthetic data may perpetuate incorrect informa-653 tion or provide inaccurate answers. Moreover, the654 case reports used to generate synthetic conversa-655 tions usually focus on unusual observations and656 rare conditions. Thus the physician bot may hallu-657 cinate or overtreat patients with common diseases.658
Considering Broader Impacts, we have per-659
15https://www.reuters.com/technology/openai-plansmajor-updates-lure-developers-with-lower-costs-sources2023-10-11/
formed a preliminary study to generate synthetic 660 conversation from case reports within research ar- 661 ticles indexed from January 2002 to July 2022 by 662 PubMed Central. The credibility of these case re- 663 ports is ensured as they are peer-reviewed and pub- 664 lished in academic journals. Moreover, the type of 665 disease is diverse as they are sourced from various 666 hospital departments and are not limited to inten- 667 sive care units (such as MIMIC). Thus, models 668 trained using our synthetic data may benefit from 669 these characteristics. 670
References 671 Asma Ben Abacha, Wen-wai Yim, Yadan Fan, and 672
Thomas Lin. 2023. An empirical study of clinical 673 note generation from doctor-patient encounters. In 674 Proceedings of the 17th Conference of the European 675 Chapter of the Association for Computational Lin- 676 guistics, pages 2283–2294. 677
Asma Ben Abacha and Pierre Zweigenbaum. 2015. 678 Means: A medical question-answering system com- 679 bining nlp techniques and semantic web technologies. 680 Information processing & management, 51(5):570– 681 594. 682
Griffin Adams, Jason Zucker, and Noémie Elhadad. 683 2023. A meta-evaluation of faithfulness metrics 684 for long-form hospital-course summarization. arXiv 685 preprint arXiv:2303.03948. 686
George J Annas. 2003. Hipaa regulations: a new era 687 of medical-record privacy? New England Journal of 688 Medicine, 348:1486. 689
Akari Asai, Matt Gardner, and Hannaneh Ha- 690 jishirzi. 2021. Evidentiality-guided generation for 691 knowledge-intensive nlp tasks. arXiv preprint 692 arXiv:2112.08688. 693
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, 694 Richard Socher, and Caiming Xiong. 2019. Learn- 695 ing to retrieve reasoning paths over wikipedia 696 graph for question answering. arXiv preprint 697 arXiv:1911.10470. 698
Akari Asai, Sewon Min, Zexuan Zhong, and Danqi 699 Chen. 2023. Retrieval-based language models and 700 applications. In Proceedings of the 61st Annual Meet- 701 ing of the Association for Computational Linguistics 702 (Volume 6: Tutorial Abstracts), pages 41–46. 703
Asma Ben Abacha, Wen-wai Yim, Griffin Adams, Neal 704 Snider, and Meliha Yetisgen. 2023. Overview of the 705 mediqa-chat 2023 shared tasks on the summarization 706 and generation of doctor-patient conversations. In 707 ACL-ClinicalNLP 2023. 708
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie 709 Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind 710 Neelakantan, Pranav Shyam, Girish Sastry, Amanda 711
9
Askell, Sandhini Agarwal, Ariel Herbert-Voss,712 Gretchen Krueger, Tom Henighan, Rewon Child,713 Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu,714 Clemens Winter, Christopher Hesse, Mark Chen, Eric715 Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,716 Jack Clark, Christopher Berner, Sam McCandlish,717 Alec Radford, Ilya Sutskever, and Dario Amodei.718 2020. Language models are few-shot learners. In719 Advances in Neural Information Processing Systems,720 volume 33, pages 1877–1901.721
Jeffrey Budd. 2023. Burnout related to electronic health722 record use in primary care. Journal of Primary Care723 & Community Health, 14:21501319231166921.724
Pengshan Cai, Zonghai Yao, Fei Liu, Dakuo Wang,725 Meghan Reilly, Huixue Zhou, Lingxi Li, Yi Cao,726 Alok Kapoor, Adarsha Bajracharya, et al. 2023.727 Paniniqa: Enhancing patient education through728 interactive question answering. arXiv preprint729 arXiv:2308.03253.730
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng,731 Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan732 Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion733 Stoica, and Eric P. Xing. 2023. Vicuna: An open-734 source chatbot impressing gpt-4 with 90%* chatgpt735 quality.736
Won Ik Cho, Yoon Kyung Lee, Seoyeon Bae, Ji-Hwan737 Kim, Sangah Nancy Park, Moosung Kim, Sowon738 Hahn, and Nam Soo Kim. 2023. When crowd meets739 persona: Creating a large-scale open-domain persona740 dialogue corpus. ArXiv, abs/2304.00350.741
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian,742 Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias743 Plappert, Jerry Tworek, Jacob Hilton, Reiichiro744 Nakano, et al. 2021. Training verifiers to solve math745 word problems. ArXiv preprint, abs/2110.14168.746
Tirth Dave, Sai Anirudh Athaluri, and Satyam Singh.747 2023. Chatgpt in medicine: an overview of its ap-748 plications, advantages, limitations, future prospects,749 and ethical considerations. Frontiers in Artificial750 Intelligence, 6:1169595.751
Paul Drew, John Chatwin, and Sarah Collins. 2001.752 Conversation analysis: a method for research into753 interactions between patients and health-care profes-754 sionals. Health Expectations, 4(1):58–70.755
H. Eyre, A. B. Chapman, K. S. Peterson, J. Shi, P. R.756 Alba, M. M. Jones, T. L. Box, S. L. DuVall, and O. V.757 Patterson. 2021. Launching into clinical space with758 medspaCy: a new clinical text processing toolkit in759 Python. AMIA Annu Symp Proc, 2021:438–447.760
Hao Fei, Yafeng Ren, Yue Zhang, Donghong Ji, and761 Xiaohui Liang. 2021. Enriching contextualized lan-762 guage model from knowledge graph for biomedical763 information extraction. Briefings in bioinformatics,764 22(3):bbaa110.765
Lewis R First, Humayun J Chaudhry, and Donald E 766 Melnick. 2013. Quality, cost, and value of clinical 767 skills assessment. New England Journal of Medicine, 768 368(10):963–964. 769
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei 770 Liu. 2023. Gptscore: Evaluate as you desire. arXiv 771 preprint arXiv:2302.04166. 772
Saadia Gabriel, Antoine Bosselut, Jeff Da, Ari Holtz- 773 man, Jan Buys, Kyle Lo, Asli Celikyilmaz, and Yejin 774 Choi. 2021. Discourse understanding and factual 775 consistency in abstractive summarization. In Pro- 776 ceedings of the 16th Conference of the European 777 Chapter of the Association for Computational Lin- 778 guistics: Main Volume, pages 435–447, Online. As- 779 sociation for Computational Linguistics. 780
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Ship- 781 ing Yang, and Xiaojun Wan. 2023. Human-like sum- 782 marization evaluation with chatgpt. arXiv preprint 783 arXiv:2304.02554. 784
Aidan Gilson, Conrad W Safranek, Thomas Huang, 785 Vimig Socrates, Ling Chi, Richard Andrew Taylor, 786 David Chartash, et al. 2023. How does chatgpt per- 787 form on the united states medical licensing examina- 788 tion? the implications of large language models for 789 medical education and knowledge assessment. JMIR 790 Medical Education, 9(1):e45312. 791
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. 792 News summarization and evaluation in the era of 793 gpt-3. arXiv preprint arXiv:2209.12356. 794
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, 795 Zhangyin Feng, Haotian Wang, Qianglong Chen, 796 Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting 797 Liu. 2023. A survey on hallucination in large lan- 798 guage models: Principles, taxonomy, challenges, and 799 open questions. 800
Dongfu Jiang, Bill Yuchen Lin, and Xiang Ren. 2022. 801 Pairreranker: Pairwise reranking for natural language 802 generation. arXiv preprint arXiv:2212.10555. 803
Hillary Johnson. 2003. A critical review of standard- 804 ized patient examinations as part of the usmle. AMA 805 Journal of Ethics, 5(12):426–429. 806
Omar Khattab, Keshav Santhanam, Xiang Lisa 807 Li, David Hall, Percy Liang, Christopher Potts, 808 and Matei Zaharia. 2022. Demonstrate-search- 809 predict: Composing retrieval and language mod- 810 els for knowledge-intensive nlp. arXiv preprint 811 arXiv:2212.14024. 812
Kundan Krishna, Sopan Khosla, Jeffrey P Bigham, 813 and Zachary C Lipton. 2020. Generating soap 814 notes from doctor-patient conversations using mod- 815 ular summarization techniques. arXiv preprint 816 arXiv:2005.01795. 817
Sunjae Kwon, Zonghai Yao, Harmon S Jordan, David A 818 Levy, Brian Corner, and Hong Yu. 2022. Medjex: A 819
10
medical jargon extraction model with wiki’s hyper-820 link span and contextualized masked language model821 score. arXiv preprint arXiv:2210.05875.822
Faisal Ladhak, Esin Durmus, He He, Claire Cardie, and823 Kathleen McKeown. 2022. Faithful or extractive?824 on mitigating the faithfulness-abstractiveness trade-825 off in abstractive summarization. In Proceedings826 of the 60th Annual Meeting of the Association for827 Computational Linguistics (Volume 1: Long Papers),828 pages 1410–1421, Dublin, Ireland. Association for829 Computational Linguistics.830
G. Li, Hasan Abed Al Kader Hammoud, Hani Itani,831 Dmitrii Khizbullin, and Bernard Ghanem. 2023a.832 Camel: Communicative agents for "mind" explo-833 ration of large scale language model society. ArXiv,834 abs/2303.17760.835
Jianquan Li, Xidong Wang, Xiangbo Wu, Zhiyi Zhang,836 Xiaolong Xu, Jie Fu, Prayag Tiwari, Xiang Wan, and837 Benyou Wang. 2023b. Huatuo-26m, a large-scale838 chinese medical qa dataset.839
Chin-Yew Lin. 2004. ROUGE: A package for auto-840 matic evaluation of summaries. In Text Summariza-841 tion Branches Out, pages 74–81, Barcelona, Spain.842 Association for Computational Linguistics.843
Hongcheng Liu, Yusheng Liao, Yutong Meng, Yu Wang,844 and Yanfeng Wang. 2023a. Medicalgpt-zh. https://845 github.com/MediaBrain-SJTU/MedicalGPT-zh.846
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang,847 Ruochen Xu, and Chenguang Zhu. 2023b. Gpte-848 val: Nlg evaluation using gpt-4 with better human849 alignment. arXiv preprint arXiv:2303.16634.850
Shayne Longpre, Le Hou, Tu Vu, Albert Webson,851 Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le,852 Barret Zoph, Jason Wei, and Adam Roberts. 2023.853 The flan collection: Designing data and methods for854 effective instruction tuning.855
Prakamya Mishra, Zonghai Yao, Shuwei Chen, Bein-856 ing Wang, Rohan Mittal, and Hong Yu. 2023. Syn-857 thetic imitation edit feedback for factual align-858 ment in clinical summarization. arXiv preprint859 arXiv:2310.20033.860
OpenAI. 2023. Gpt-4 technical report. arXiv preprint861 arXiv:2303.08774.862
Marcus V Ortega, Michael K Hidrue, Sara R Lehrhoff,863 Dan B Ellis, Rachel C Sisodia, William T Curry,864 Marcela G Del Carmen, and Jason H Wasfy. 2023.865 Patterns in physician burnout in a stable-linked866 cohort. JAMA Network Open, 6(10):e2336745–867 e2336745.868
Liviu Panait and Sean Luke. 2005. Cooperative multi-869 agent learning: The state of the art. Autonomous870 Agents and Multi-Agent Systems, 11:387–434.871
Mayana Pereira, Sikha Pentyala, Anderson Nascimento, 872 Rafael T de Sousa Jr, and Martine De Cock. 2022. 873 Secure multiparty computation for synthetic data 874 generation from distributed data. arXiv preprint 875 arXiv:2210.07332. 876
V Podder, V Lew, and S Ghassemzadeh. 2021. Soap 877 notes.[updated 2021 sep 2]. StatPearls [Internet]. 878 StatPearls Publishing. Available from: https://www. 879 ncbi. nlm. nih. gov/books/NBK482263. 880
Dragomir R Radev, Hong Qi, Harris Wu, and Weiguo 881 Fan. 2002. Evaluating web-based question answering 882 systems. In LREC. Citeseer. 883
Sanjana Ramprasad, Elisa Ferracane, and Sai P Selvaraj. 884 2023. Generating more faithful and consistent soap 885 notes using attribute-specific parameters. 886
Mathieu Ravaut, Shafiq Joty, and Nancy Chen. 2022. 887 SummaReranker: A multi-task mixture-of-experts 888 re-ranking framework for abstractive summarization. 889 In Proceedings of the 60th Annual Meeting of the 890 Association for Computational Linguistics (Volume 891 1: Long Papers), pages 4504–4524, Dublin, Ireland. 892 Association for Computational Linguistics. 893
Thomas C Rindfleisch. 1997. Privacy, information tech- 894 nology, and health care. Communications of the 895 ACM, 40(8):92–100. 896
Afrah Shafquat, Jason Mezey, Mandis Beigi, Jimeng 897 Sun, and Jacob W Aptekar. 2022. A source data 898 privacy framework for synthetic clinical trial data. 899 In NeurIPS 2022 Workshop on Synthetic Data for 900 Empowering ML Research. 901
Noah Shinn, Federico Cassano, Beck Labash, Ashwin 902 Gopinath, Karthik Narasimhan, and Shunyu Yao. 903 2023. Reflexion: Language agents with verbal rein- 904 forcement learning. 905
Luca Soldaini. 2016. Quickumls: a fast, unsupervised 906 approach for medical concept extraction. 907
Yan Song, Yuanhe Tian, Nan Wang, and Fei Xia. 2020. 908 Summarizing medical conversations via identifying 909 important utterances. In Proceedings of the 28th 910 International Conference on Computational Linguis- 911 tics, pages 717–729, Barcelona, Spain (Online). 912
Jing Su, Longxiang Zhang, Hamidreza Hassanzadeh, 913 and Thomas Schaaf. 2022. Extract and abstract with 914 bart for clinical notes from doctor-patient conversa- 915 tions. In Interspeech. 916
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann 917 Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, 918 and Tatsunori B. Hashimoto. 2023. Stanford al- 919 paca: An instruction-following llama model. https: 920 //github.com/tatsu-lab/stanford_alpaca. 921
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier 922 Martinet, Marie-Anne Lachaux, Timothée Lacroix, 923 Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal 924 Azhar, Aurelien Rodriguez, Armand Joulin, Edouard 925
11
Grave, and Guillaume Lample. 2023. Llama: Open926 and efficient foundation language models. arXiv927 preprint arXiv:2302.13971.928
Jason T Tsichlis, Andrew M Del Re, and J Bryan Car-929 mody. 2021. The past, present, and future of the930 united states medical licensing examination step 2931 clinical skills examination. Cureus, 13(8).932
Karthik Valmeekam, Matthew Marquez, Sarath Sreed-933 haran, and Subbarao Kambhampati. 2023. On the934 planning abilities of large language models–a critical935 investigation. arXiv preprint arXiv:2305.15771.936
Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang,937 Sendong Zhao, Bing Qin, and Ting Liu. 2023. Hu-938 atuo: Tuning llama model with chinese medical939 knowledge. arXiv preprint arXiv:2304.06975.940
Honglin Xiong, Sheng Wang, Yitao Zhu, Zihao Zhao,941 Yuxiao Liu, Qian Wang, and Dinggang Shen. 2023.942 Doctorglm: Fine-tuning your chinese doctor is not a943 herculean task. arXiv preprint arXiv:2304.01097.944
Zonghai Yao, Benjamin J Schloss, and Sai P Selvaraj.945 2023. Improving summarization with human edits.946 arXiv preprint arXiv:2310.05857.947
Zonghai Yao and Hong Yu. 2021. Improving formality948 style transfer with context-aware rule injection. arXiv949 preprint arXiv:2106.00210.950
Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal951 Snider, Thomas Lin, and Meliha Yetisgen. 2023. Aci-952 bench: a novel ambient clinical intelligence dataset953 for benchmarking automatic visit note generation.954 Submitted to Nature Scientific Data.955
Wen-wai Yim and Meliha Yetisgen-Yildiz. 2021. To-956 wards automating medical scribing: Clinic visit dia-957 logue2note sentence alignment and snippet summa-958 rization. In Proceedings of the Second Workshop on959 Natural Language Processing for Medical Conversa-960 tions, pages 10–20.961
Siyu Yuan, Deqing Yang, Jinxi Liu, Shuyu Tian, Jiaqing962 Liang, Yanghua Xiao, and Rui Xie. 2023. Causality-963 aware concept extraction based on knowledge-guided964 prompting. arXiv preprint arXiv:2305.01876.965
Li Yunxiang, Li Zihan, Zhang Kai, Dan Ruilong, and966 Zhang You. 2023. Chatdoctor: A medical chat model967 fine-tuned on llama model using medical domain968 knowledge. arXiv preprint arXiv:2303.14070.969
Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang,970 Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng,971 Xiangyu Dong, Ruoyu Zhang, et al. 2020. Med-972 dialog: Large-scale medical dialogue datasets. In973 Proceedings of the 2020 Conference on Empirical974 Methods in Natural Language Processing (EMNLP),975 pages 9241–9250.976
Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu,977 Zhihong Chen, Jianquan Li, Guiming Chen, Xi-978 angbo Wu, Zhiyi Zhang, Qingying Xiao, Xiang Wan,979
Benyou Wang, and Haizhou Li. 2023. Huatuogpt, to- 980 wards taming language models to be a doctor. arXiv 981 preprint arXiv:2305.15075. 982
Zhengyun Zhao, Qiao Jin, Fangyuan Chen, Tuorui Peng, 983 and Sheng Yu. 2023. Pmc-patients: A large-scale 984 dataset of patient summaries and relations for bench- 985 marking retrieval-based clinical decision support sys- 986 tems. 987
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan 988 Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, 989 Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, 990 Joseph E. Gonzalez, and Ion Stoica. 2023. Judging 991 llm-as-a-judge with mt-bench and chatbot arena. 992
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan 993 Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A 994 benchmarking platform for text generation models. 995 SIGIR. 996
12
A Appendix997
A.1 SOAP Structure998
The SOAP (Subjective, Objective, Assessment, and999 Plan) structure is commonly used by providers1000 (Podder et al., 2021).1001
1. The Subjective section is a detailed report1002 of the patient’s current conditions, such as1003 source, onset, and duration of symptoms,1004 mainly based on the patient’s self-report. This1005 section usually includes chief ccomplaint, his-1006 tory of present illness and symptoms, current1007 medications, and allergies.1008
2. The Objective section documents the results of1009 physical exam findings, laboratory data, vital1010 signs, and descriptions of imaging results.1011
3. The Assessment section typically contains1012 medical diagnoses and reasons that lead to1013 medical diagnoses. The assessment is typ-1014 ically based on the content from the chief1015 complaint, and the subjective and objective1016 sections.1017
4. The Plan section addresses treatment plans1018 based on the assessment.1019
A.2 Prompts for ChatGPT&GPT41020
We use the following prompts to instruct Chat-1021 GPT and GPT4 to generate the synthetic patient-1022 physician dialogue based on the provided clinical1023 note.1024
Generate the conversation between physician1025 and patient. But for some cases, if the patient even-1026 tually dies (according to the clinical note), you can1027 add the patient’s family at the end of the conversa-1028 tion to make it more reasonable. The conversation1029 should include all the information in the following1030 note, especially paying attention to those numbers1031 and medical concepts. The conversation can be1032 more colloquial. When the physician is speaking,1033 the patient can have many modal particles (e.g.1034 hmm, yes, okay) to increase interaction. All the1035 numbers and medical concepts that appear in the1036 note should be mentioned by the physician. Pro-1037 fessional medical terms and numbers should more1038 likely occur in the physician’s utterances but not in1039 the patient’s answer. The physician may describe1040 and explain professional judgment to the patient1041 and instruct the patient on follow-up requirements1042 but not ask questions that require professional med-1043 ical knowledge to answer. The patient’s answer1044 should be succinct and accurate in a colloquial lay1045
language style. 1046
A.3 Experimental Settings 1047
In our study on generating conversation datasets 1048 using ChatGPT and GPT-4, we adopted a temper- 1049 ature setting of 0.7. This setting was consistently 1050 applied across our methodologies. For each round 1051 of dialogue, we set the max tokens for physician 1052 role-play as 200 tokens and the patient role-play as 1053 100 tokens. For the intrinsic evaluation phase, we 1054 selected a subset of 20 data points from the MT- 1055 Dialog dataset and randomly chose 100 datasets 1056 from the pmc dataset for testing. In terms of ex- 1057 ternal evaluation, we selected three random data 1058 points from each model’s output on the pmc dataset 1059 to use as few-shot examples. These were inputted 1060 into GPT-4, which then generated dialogues from 1061 clinical notes or clincal notes from conversations 1062 based 20 data sets from the MT-Dialog dataset. 1063 During the external chatbot evaluation, we used 1064 10k datasets generated by ChatGPT, GPT-4, and 1065 NoteChat-ChatGPT to fine-tune LLaMA2-7b on 1066 two A100-40g gpus. During the fine-tuning pro- 1067 cess, we used DeepSpeed Zero-2 for training, with 1068 a learning rate of 2e-5, a batch size of 16, max 1069 tokens of 4048 and 1 training epochs. We employ 1070 the same settings to train LLaMA2-7b for the gen- 1071 eration of clinical tasks from dialogues and the 1072 dialogues from clinical notes. 1073
A.4 Color for Polish Promopt 1074
We have used consistently different colors to indi- 1075 cate in the polish prompt, as shown in Table 13, 1076 which parts of our prompt have achieved these ten 1077 different functions. 1078
1. Yellow: Make the conversation as colloquial 1079 as possible 1080
2. Orchid: Increase the number of rounds of in- 1081 teraction 1082
3. Pink: Professional terms and vocabulary 1083 should come from the physicians, and patients 1084 should be more colloquial 1085
4. Gray: Basic symptoms and medical history 1086 should come from the patient, not the physi- 1087 cian 1088
5. BrickRed: The questions asked by the physi- 1089 cian should be around the case (to avoid hal- 1090 lucination) 1091
6. SkyBlue: Physician inquiries should be logi- 1092 cal 1093
13
7. Emerald: If there are multiple consultation1094 records, you can split a conversation into mul-1095 tiple ones and then link them with transfer1096 words (e.g., a few days later)1097
8. BurntOrange: Range of rounds of interaction1098
9. Thistle: Must contain the given keywords1099
10. Periwinkle: Do not generate duplicate infor-1100 mation1101
Note that there are some similar and repeated1102 parts in the prompt, which are because we found1103 that mentioning a certain point multiple times in1104 different places in the prompt is more helpful for1105 LLM to avoid certain problems.1106
Group Our Score GPT-4 Score ChatGPT Score physicians 0.78 0.80 0.93
Crowd 0.70 0.75 0.90
A.5 Ablation Study for Planning Module1107 To demonstrate the importance of the planning1108 module, we designed the following experiment:1109 We conducted evaluations using both GPT-4 and1110 human assessments. In the absence of the checklist1111 and planning module, relying solely on role play1112 and polishing for dialogue generation, the results1113 were as follows:1114
• GPT-4 Evaluation Win Rate:1115
– Our model (without checklist & plan- 1116 ning): 32% 1117
– GPT-4: 68% 1118
• Human Evaluation Win Rate: 1119
– Our model (without checklist & plan- 1120 ning): 38% 1121
– GPT-4: 62% 1122
The absence of the checklist and planning mod- 1123 ule resulted in the model’s inability to ensure 1124 comprehensive coverage of necessary information. 1125 While the generated dialogues were logically co- 1126 herent, they significantly lacked informational con- 1127 tent. This deficiency is primarily attributable to our 1128 model being based on GPT-3.5, which has a sub- 1129 stantially lower capacity for information coverage 1130 compared to GPT-4. 1131
Furthermore, when relying solely on a randomly 1132 ordered checklist, the results were as follows: 1133
• GPT-4 Evaluation Win Rate: 1134
– Our model (without planning module): 1135 54% 1136
– GPT-4: 46% 1137
• Human Evaluation Win Rate: 1138
14
– Our model (without planning module):1139 40%1140
– GPT-4: 60%1141
These results indicate slight differences. When1142 evaluated by GPT-4, our model without the plan-1143 ning module appeared superior due to providing1144 more information in shorter dialogue turns and ex-1145 tended conversations. However, human evaluators1146 found the generated dialogues logically disorga-1147 nized, primarily due to the absence of the planning1148 module. The randomly ordered checklist led to1149 each conversational turn lacking logical progres-1150 sion, making it seem less like a real dialogue. This1151 highlights the critical importance of the planning1152 module.1153
15
16
17
18
19

heading: Future Work
text: Chatgpt in medicine: an overview of its ap-748 plications, advantages, limitations, future prospects,749 and ethical considerations. Frontiers in Artificial750 Intelligence, 6:1169595. 751
Paul Drew, John Chatwin, and Sarah Collins. 2001. 752 Conversation analysis: a method for research into753 interactions between patients and health-care profes-754 sionals. Health Expectations, 4(1):58–70. 755
H. Eyre, A. B. Chapman, K. S. Peterson, J. Shi, P. R. 756 Alba, M. M. Jones, T. L. Box, S. L. DuVall, and O. V. 757 Patterson. 2021. Launching into clinical space with758 medspaCy: a new clinical text processing toolkit in759 Python. AMIA Annu Symp Proc, 2021:438–447. 760
Hao Fei, Yafeng Ren, Yue Zhang, Donghong Ji, and761 Xiaohui Liang. 2021. Enriching contextualized lan-762 guage model from knowledge graph for biomedical763 information extraction. Briefings in bioinformatics,764 22(3):bbaa110. 765
Lewis R First, Humayun J Chaudhry, and Donald E 766 Melnick. 2013. Quality, cost, and value of clinical 767 skills assessment. New England Journal of Medicine, 768 368(10):963–964. 769
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei 770 Liu. 2023. Gptscore: Evaluate as you desire. arXiv 771 preprint arXiv:2302. 04166. 772
Saadia Gabriel, Antoine Bosselut, Jeff Da, Ari Holtz- 773 man, Jan Buys, Kyle Lo, Asli Celikyilmaz, and Yejin 774 Choi. 2021. Discourse understanding and factual 775 consistency in abstractive summarization. In Pro- 776 ceedings of the 16th Conference of the European 777 Chapter of the Association for Computational Lin- 778 guistics: Main Volume, pages 435–447, Online. As- 779 sociation for Computational Linguistics. 780
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Ship- 781 ing Yang, and Xiaojun Wan. 2023. Human-like sum- 782 marization evaluation with chatgpt. arXiv preprint 783 arXiv:2304. 02554. 784
Aidan Gilson, Conrad W Safranek, Thomas Huang, 785 Vimig Socrates, Ling Chi, Richard Andrew Taylor, 786 David Chartash, et al. 2023. How does chatgpt per- 787 form on the united states medical licensing examina- 788 tion? the implications of large language models for 789 medical education and knowledge assessment. JMIR 790 Medical Education, 9(1):e45312. 791
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. 792 News summarization and evaluation in the era of 793 gpt-3. arXiv preprint arXiv:2209. 12356. 794
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, 795 Zhangyin Feng, Haotian Wang, Qianglong Chen, 796 Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting 797 Liu. 2023. A survey on hallucination in large lan- 798 guage models: Principles, taxonomy, challenges, and 799 open questions. 800
Dongfu Jiang, Bill Yuchen Lin, and Xiang Ren. 2022. 801 Pairreranker: Pairwise reranking for natural language 802 generation. arXiv preprint arXiv:2212. 10555. 803
Hillary Johnson. 2003. A critical review of standard- 804 ized patient examinations as part of the usmle