WangLab at MEDIQA-Chat 2023: Clinical Note Generation from Doctor-Patient Conversations using Large Language Models
Year: 2023

Abstract:
This paper describes our submission to the MEDIQA-Chat 2023 shared task for automatic clinical note generation from doctor-patient conversations. We report results for two approaches: the first fine-tunes a pre-trained language model (PLM) on the shared task data, and the second uses few-shot in-context learning (ICL) with a large language model (LLM). Both achieve high performance as measured by automatic metrics (e.g. ROUGE, BERTScore) and ranked second and first, respectively, of all submissions to the shared task. Expert human scrutiny indicates that notes generated via the ICL-based approach with GPT-4 are preferred about as often as human-written notes, making it a promising path toward automated note generation from doctor-patient conversations.1

heading: 1 Introduction
text: The growing burden of clinical documentation has emerged as a critical issue in healthcare, increasing job dissatisfaction and burnout rates among clinicians and negatively impacting patient experiences (Friedberg et al., 2013; Babbott et al., 2014; Arndt et al., 2017). On the other hand, timely and accurate documentation of patient encounters is critical for safe, effective care and communication between specialists. Therefore, interest in assisting clinicians by automatically generating consultation notes is mounting (Finley et al., 2018; Enarvi et al., 2020; Molenaar et al., 2020; Knoll et al., 2022).
To further encourage research on automatic clinical note generation from doctor-patient conversations, the MEDIQA-Chat Dialogue2Note shared task was proposed (Ben Abacha et al., 2023). Here, we describe our submission to subtask B: the generation of full clinical notes from doctor-patient dialogues. We explored two approaches; the first finetunes a pre-trained language model (PLM, §3.1),
∗Core contributors. See author contributions 1https://github.com/bowang-lab/
MEDIQA-Chat-2023
while the second uses few-shot in-context learning (ICL, §3.2). Both achieve high performance as measured by automatic natural language generation metrics (§4) and ranked second and first, respectively, of all submissions to the shared task. In a human evaluation with three expert physicians, notes generated via the ICL-based approach with GPT-4 were preferred about as often as humanwritten notes (§4.3).

heading: 2 Shared Task and Dataset
text: MEDIQA-Chat 2023 proposed two shared tasks:
1. Dialogue2Note Summarization: Given a conversation between a doctor and patient, the task is to produce a clinical note summarizing the conversation with one or more note sec-
323
tions (e.g. Assessment, Past Medical History).
2. Note2Dialogue Generation: Given a clinical note, the task is to generate a synthetic doctorpatient conversation related to the information described in the note.
We focused on Dialogue2Note, which is divided into two subtasks. In subtask ‘A’ (Ben Abacha et al., 2023), the goal is to generate specific sections of a note given partial doctor-patient dialogues. In subtask ‘B’ (Yim et al., 2023), the goal is full note generation from complete dialogues. The remainder of the paper focuses on subtask B; see Appendix A for our approach to subtask A, which also ranks first of all submissions to the shared task.

heading: 2.1 Task definition
text: Each of the k examples consist of a doctor-patient dialogue, D = d1, . . . , dk and a corresponding clinical note, N = n1, . . . , nk. The aim is to automatically generate a note ni given a dialogue di. Each note comprises one or more sections, such as “Chief Complaint”, and “Family history”. During evaluation, sections are grouped under one of four categories: “Subjective”, “Objective Exam”, “Objective Results”, and “Assessment and Plan”.2 See Figure 2 for an example doctor-patient conversation and clinical note pair.
2See here for the mapping

heading: 2.2 Dataset
text: The dataset comprises 67 train and 20 validation examples, featuring transcribed dialogues from doctor-patient encounters and the resulting clinician-written notes. Each example is labelled with the ‘dataset source’, indicating the dialogue transcription system used to produce the note.

heading: 3 Approach
text: We take two high-performant approaches to the shared task. In the first, we fine-tune a pre-trained language model (PLM) on the provided training set (§3.1). In the second, we use in-context learning (ICL) with a large language model (LLM, §3.2).

heading: 3.1 Fine-tuning pre-trained language models
text: As a first approach, we fine-tune a PLM on the training set following a canonical, sequence-tosequence training process (Figure 1 A; see Appendix C for details). Given the length of input dialogues (Figure 3), we elected to use LongformerEncoder-Decoder (LED, Beltagy et al. 2020), which has a maximum input size of 16,384 tokens. We begin fine-tuning from a LEDLARGE checkpoint tuned on the PubMed summarization dataset (Cohan et al., 2018), which performed best in preliminary experiments.3 The model was fine-tuned using HuggingFace Transformers (Wolf et al., 2020) on a single NVIDIA A100-40GB GPU. Hyperparameters were lightly tuned on the validation set.4
3https://huggingface.co/patrickvonplaten/ led-large-16384-pubmed
4See Appendix B.1 for details

heading: 3.2 In-context learning with LLMs
text: As a second approach, we attempt subtask B with ICL. We chose GPT-4 (OpenAI, 2023)5 as the LLM and designed a simple prompt, which included natural language instructions and in-context examples (Figure 4). We limited the prompt size to 6192 tokens — allowing for 2000 output tokens, as the model’s maximum token size is 8,192 — and used as many in-context examples as would fit within this token limit, up to a maximum of 3. We set the temperature parameter to 0.2 and left all other hyperparmeters of the OpenAI API at their defaults.
Natural language instructions During preliminary experiments, we found that GPT-4 was not overly sensitive to the exact phrasing of the natural language instructions in the prompt. We, therefore, elected to use short, simple instructions (Figure 4).
In-context example selection Each in-context example is a note from the train set. To select the notes, we first embed the dialogues of each training example and the input dialogue. Train dialogues are then ranked based on cosine similarity to the input dialogue; notes of the resulting top-k training examples are selected as the in-context examples (see Figure 1, B). Dialogues were embedded using Instructor (Su et al., 2022a), a text encoder that
5Specifically, the 03/14/2023 snapshot, “gpt-4-0314”
supports natural language instructions.6 Lastly, we restricted in-context examples to be of the same ‘dataset source’ (see §2.2) as the input dialogue, hypothesizing that this may improve performance.7

heading: 3.3 Evaluation
text: Models are evaluated with the official evaluation script8 on the validation set (as test notes are not provided). Generated notes are evaluated against the provided ground truth notes with ROUGE (Lin, 2004), BERTScore (Zhang et al., 2020) and BLEURT (Sellam et al., 2020). We report performance as the arithmetic mean of ROUGE-1 F1, BERTScore F1 and BLEURT-20 (Pu et al., 2021).

heading: 4 Results

heading: 4.1 Fine-tuning pre-trained language models
text: We present the results of fine-tuning LED in Table 1. Due to the non-determinism of the LED implementation,9 we report the mean results of three training runs. Unsurprisingly, we find that scaling the model size from LEDBASE (12 layers, ∼162M parameters) to LEDLARGE (24 layers, ∼460M parameters) leads to sizable gains in performance. Performance further improves by initializing the model with a checkpoint fine-tuned on the PubMed summarization dataset (LEDLARGE-PubMed). This is likely because (1) Dialouge2Note resembles a summarization task, and (2) text from PubMed is more similar to clinical text than is the general domain text used to pre-train LED.10 Our submission to the shared task using this approach ranked second overall, outperforming the next-best submission by 2.7 average score; a difference comparable to the improvement in performance we see by doubling model size (see LEDBASE vs. LEDLARGE, Table 1).

heading: 4.2 In-context learning with LLMs
text: We present the results of ICL with GPT-4 in Table 2. We note several interesting trends in order of magnitude of impact. First, selecting in-context
6We used the following instructions: “Represent the Medicine dialogue for clustering: {dialogue}”
7Manual review revealed that dataset source was predictive of note structure & style; likely because it indicates which clinician or electronic health record system produced the note
8https://github.com/abachaa/MEDIQA-Chat-2023/ blob/main/scripts/evaluate_summarization.py
9https://github.com/huggingface/transformers/ issues/12482
10LED is initialized from BART (Lewis et al., 2020), which was pre-trained on a combination of text from Wikipedia and BooksCorpus (Zhu et al., 2015)
examples based on the similarity of dialogues has a strong positive impact, typically improving average score by 4 or more. Using only notes as in-context examples, as opposed to dialogue-note pairs, also has a positive impact, typically improving average score by ∼1. Surprisingly, increasing the number of in-context examples had a marginal effect on performance. Together these results suggest that the in-context examples’ primary benefit is providing guidance with regard to the expected note structure, style and length. Finally, filtering in-context examples to be of the same ‘dataset source’ as the input dialogue has a negligible impact on performance.
The best strategy out-performs LED by almost 3 average score (60.8 vs. 57.9, see Table 1 & Table 2) and achieves first place of all submissions to the shared task, out-performing the runner up by > 9 average score. We conclude that (1) few-shot ICL with GPT-4, using as little as one example, is a performant approach for note generation from doctorpatient conversations, and (2) using the notes of semantically similar dialogue-note pairs is a strong strategy for selecting the in-context examples.

heading: 4.3 Human evaluation
text: Automatic evaluation metrics like ROUGE, BERTScore and BLEURT are imperfect and may not correlate with aspects of human judgment.11 Therefore, we conducted an expert human evaluation to validate our results. To make annotation feasible, we conducted it on the validation set (20 examples) using the best performing fine-tuned model: LEDLARGE-PubMed (Table 1), and best performing ICL-based approach: 3-shot, similar, noteonly examples filtered by dataset type (Table 2).
Three senior resident physicians12 were shown a ground truth note, a note generated by the fine-tuned model, and a note generated by the ICL-based approach for each example (presented in random order as clinical note ‘A’, ‘B’ and ‘C’)
11See §6 for an extended discussion 12The three annotators are a subset of the authors who did not interact with the model or model outputs before annotation
and asked to select which note(s) they preferred, given a dialogue and some simple instructions:
Instructions: Please asses the clinical notes A, B and C relative to the provided doctor-patient dialogue. For each set of notes, you should select which note you prefer (‘A’, ‘B’, or ‘C’). If you have approximately equal preference for two notes, select (‘A/B’, ‘B/C’, or ‘C/A’). If you have no preference, select ‘A/B/C’. A ‘good’ note should contain all critical, most non-critical and very little irrelevant information mentioned in a dialogue:
• Critical: Items medico-legally required to document the diagnosis and treatment decisions whose absence or incorrectness may lead to wrong diagnosis and treatment later on, e.g. the symptom "cough" in a suspected chest infection consultation. This is the key information a note needs to capture correctly in order to not mislead clinicians.
• Non-critical: Items that should be documented in a complete note but whose absence will not affect future treatment or diagnosis, e.g. "who the patient lives with" in a consultation about chest infection.
• Irrelevant: Medically irrelevant information covered in the consultation, e.g. the pet of a patient with a suspected chest infection just died.
The definitions of critical, non-critical and irrelevant information are taken from previous work on human evaluation of generated clinical notes (Moramarco et al., 2022; Savkov et al., 2022).
In short, notes generated by ICL are strongly preferred over notes generated by the fine-tuned model and, on average, slightly preferred over the human-written notes (Table 3), validating the high performance reported by the automatic metrics. We note, however, that inter-annotator agreement is low and speculate why this might be in §6.

heading: 5 Related Work
text: Automated note generation from doctor-patient conversations has received increasing attention in recent years (Finley et al., 2018; Enarvi et al., 2020; Molenaar et al., 2020; Knoll et al., 2022). Different methods have been proposed, such as extractiveabstractive approaches (Joshi et al., 2020; Krishna et al., 2021; Su et al., 2022b) and fine-tuning PLMs (Zhang et al. 2021, similar to our approach in §3.1). Others have focused on curating data for training and benchmarking (Papadopoulos Korfiatis et al., 2022), including the use of LLMs to produce synthetic data (Chintagunta et al., 2021). Lastly, there have been efforts to improve the evaluation of generated clinical notes, both with automatic metrics (Moramarco et al., 2022) and human evaluation (Savkov et al., 2022). While recent literature has commented on the potential of ICL for note generation (Lee et al., 2023), our work is among the first to evaluate this approach rigorously.

heading: 6 Conclusion
text: We present our submission to the MEDIQA-Chat shared task for clinical note generation from doctorpatient dialogues. We evaluated a fine-tuning-based approach with LED and an ICL-based approach with GPT-4, ranking second and first, respectively, among all submissions. Human evaluation with three physicians revealed that notes produced by GPT-4 via ICL were strongly preferred over notes produced by LED and, on average, slightly preferred over human-written notes. We conclude that ICL is a promising path toward clinical note generation from doctor-patient conversations.
Limitations
Evaluation of generated text is difficult Evaluating automatically generated text, including clinical notes, is generally hard due to the inherently subjective nature of many aspects of output quality. Automatic evaluation metrics such as ROUGE and BERTScore are imperfect (Deutsch et al., 2022) and may not correlate with aspects of expert judgment. However, they are frequently used to evaluate model-generated clinical notes and do correlate with certain aspects of quality (Moramarco et al., 2022). To further validate our findings, we also conducted a human evaluation with three expert physicians (§4.3). As noted previously (Savkov et al., 2022), even human evaluation of clinical notes is far from perfect; inter-annotator agreement is generally low, likely because physicians have differing opinions on the importance of each patient statement and whether it should be included in a consultation note. We also found low interannotator agreement in our human evaluation and speculate this is partially due to differences in specialties among the physicians. Physicians 1 and 3, both from family medicine, had high agreement with each other but low agreement with physician 2 (cardiac surgery, see Table 3). Investigating better automatic metrics and best practices for evaluating clinical notes (and generated text more broadly) is an active field of research. We hope to integrate novel and performant metrics in future work.
Data privacy While our GPT-4 based solution achieves the best performance, it is not compliant with data protection regulations such as HIPAA; although Azure does advertise a HIPAA-compliant
option.13 From a privacy perspective, locally deploying a model such as LED may be preferred; however, our results suggest that more work is needed for this approach to reach acceptable performance (see Table 3). In either case, when implementing automated clinical note-generation systems, healthcare providers and developers should ensure that the whole system — including text-tospeech, data transmission & storage, and model inference — adheres to privacy and security requirements to maintain trust and prevent privacy violations in the clinical setting.
Ethics Statement
Developing an automated system for clinical note generation from doctor-patient conversations raises several ethical considerations. First, informed consent is crucial: patients must be made aware of their recording, and data ownership must be prioritized. Equitable access is also important; the system must be usable for patients from diverse backgrounds, including those with disabilities, limited technical literacy, or language barriers. Addressing issues of data bias and fairness are necessary to avoid unfair treatment or misdiagnosis for certain patient groups. The system must implement robust security measures to protect patient data from unauthorized access or breaches. Establishing clear lines of accountability for errors or harms arising from using an automated system for note generation is paramount. Disclosure of known limitations or potential risks associated with using the system is essential to maintain trust in the patient-physician relationship. Finally, ongoing evaluations are necessary to ensure that system performance does not degrade and negatively impact the quality of care.

heading: Acknowledgements
text: This research was enabled in part by support provided by the Digital Research Alliance of Canada (alliancecan.ca) and Compute Ontario (www.computeontario.ca). We thank all internal and external reviewers for their thoughtful feedback, which improved earlier drafts of this manuscript.
Author Contributions
John Giorgi (JG), Augustin Toma (AT) and Ronald Xie (RX) led the project in general, including
13https://azure.microsoft.com/en-us/products/ cognitive-services/openai-service#security
data cleaning and processing, model implementation, and running experiments. JG wrote the initial manuscript and designed the human evaluation with feedback from AT and RX. AT and RX recruited Sondra S. Chen, Kevin R. An and Grace X. Zheng, who served as expert physicians in the human evaluation. Bo Wang provided highlevel feedback and advice.

heading: A Subtask A
text: In subtask A of the Dialogue2Note Summarization shared task, given a partial doctor-patient dialogue, the goals are to: (1) predict the appropriate section header, e.g. “PASTMEDICALHX” and (2) generate that specific section of a note. We approached this task by fine-tuning a PLM on the provided training set, following a canonical, sequence-tosequence training process (see Appendix C for details). In preliminary experiments, we found that the instruction-tuned FLAN-T5 (Chung et al., 2022) performed particularly well at this task.
We hypothesized that jointly learning to predict the section header and generate the section text would improve overall performance. To do this, we preprocessed the training set so the targets were of the form: “Section header: {section_header} Section text: {section_text}”. After decoding, the section header and text were parsed using regular expressions and evaluated separately (Figure 5). Section header prediction was evaluated as the fraction of predicted headers that match the ground truth (accuracy), and section text was evaluated similarly to subtask B (see §3.3). In cases where the model output an invalid section header,14 we replaced it with “GENHX” (general history), which tends to summarize the contents of the other sections. The model was fine-tuned on a single NVIDIA A100-40GB GPU. Hyperparameters were lightly tuned on the validation set (Table 4).
We present the results of our approach on the 14In practice, we found that the fine-tuned model rarely, if
ever, generates invalid section headers
validation set in Table 5. Similar to subtask B (see §4.1), we find, perhaps unsurprisingly, that scaling the model size from FLAN-T5BASE (24 layers, ∼250M parameters) to FLAN-T5LARGE (48 layers, ∼780M parameters) leads to large improvements in performance. Performance is further improved by jointly learning to predict section headers and generate note sections. Our submission to the shared task based on this approach tied for first on section header prediction (78% accuracy), and ranked first for note section generation (average ROUGE-1, BERTScore and BLEURT F1-score of 57.9).

heading: B Subtask B
text: B.1 Hyperparameter tuning of LED
We lightly tuned the hyperparameters of LEDLARGE-PubMed on the subtask B validation set against the average ROUGE-1 F1, BERTScore F1 and BLEURT-20 scores. The best hyperparameters obtained are given in Table 6. We used the same hyperparameters when fine-tuning LEDBASE and LEDLARGE in §4.1.
B.2 Post processing LEDs outputs
In practice, we found that the fine-tuned LED model sometimes produces invalid section headers; notably, this problem did not occur with the ICL-based approach using GPT-4. Therefore, we lightly post-processed LEDs outputs using a simple script that identifies section headers produced by the model not in the ground truth set and uses fuzzy
string matching15 to replace them with the closest valid header. For example, in one run, this process converted the (incorrect) predicted section header “HISTORY OF PRESENT” to the nearest valid header “HISTORY OF PRESENT ILLNESS”.

heading: C Fine-tuning Seq2Seq Models
text: When training the sequence-to-sequence (seq2seq) models for both subtask A (Appendix A) and B (§3.1), we followed a canonical supervised fine-tuning (SFT) process. We start with a pretrained, encoder-decoder transformer-based language model (Vaswani et al., 2017). First, the encoder maps each token in the input to a contextual embedding. Then, the autoregressive decoder generates an output, token-by-token, attending to the
15We used https://github.com/seatgeek/thefuzz
outputs of the encoder at each timestep. Decoding proceeds until a special “end-of-sequence” token (e.g. 
) is generated, or a maximum number of tokens have been generated. Formally, X is the input sequence, which in our case is a doctor-patient dialogue, and Y is the corresponding output sequence of length T , in our case a clinical note. We model the conditional probability:
p(Y |X) = T∏
t=1
p(yt|X, y<t) (1)
During training, we optimize over the model parameters θ the sequence cross-entropy loss:
ℓ(θ) = − T∑
t=1
log p(yt|X, y<t; θ) (2)
maximizing the log-likelihood of the training data. As is common, we use teacher forcing during training, feeding previous ground truth inputs to the decoder when predicting the next token in the sequence. During inference, we generate the output using beam search (Graves, 2012). Beams are ranked by mean token log probability after applying a length penalty. Models are fine-tuned using the HuggingFace Transformers library.16
16https://github.com/huggingface/transformers/ blob/main/examples/pytorch/summarization/run_ summarization.py

heading: Future Work
text: We hope to integrate novel and performant metrics in future work. Data privacy While our GPT-4 based solution achieves the best performance, it is not compliant with data protection regulations such as HIPAA; although Azure does advertise a HIPAA-compliant
option. 13 From a privacy perspective, locally deploying a model such as LED may be preferred; however, our results suggest that more work is needed for this approach to reach acceptable performance (see Table 3). In either case, when implementing automated clinical note-generation systems, healthcare providers and developers should ensure that the whole system — including text-tospeech, data transmission & storage, and model inference — adheres to privacy and security requirements to maintain trust and prevent privacy violations in the clinical setting