Team Cadence at MEDIQA-Chat 2023: Generating, augmenting and summarizing clinical dialogue with large language models
Year: 2023

Abstract:
This paper describes Team Cadence’s winning submission to Task C of the MEDIQA-Chat 2023 shared tasks. We also present the set of methods, including a novel N-pass strategy to summarize a mix of clinical dialogue and an incomplete summarized note, used to complete Task A and Task B, ranking highly on the leaderboard amongst stable and reproducible code submissions. The shared tasks invited participants to summarize, classify and generate patient-doctor conversations. Considering the small volume of training data available, we took a data-augmentation-first approach to the three tasks by focusing on the dialogue generation task, i.e., Task C. It proved effective in improving our models’ performance on Task A and Task B. We also found the BART architecture to be highly versatile, as it formed the base for all our submissions. Finally, based on the results shared by the organizers, we note that Team Cadence was the only team to submit stable and reproducible runs to all three tasks.

heading: 1 Introduction
text: MEDIQA-Chat 2023 Shared Tasks included three tasks on the summarization and generation of doctor-patient conversations to promote research on these topics (Ben Abacha et al., 2023). Task A (Short Dialogue2Note Summarization) expected a section summary (section header and content) given a short input conversation. We recognized
that generating the summary content was an abstractive summarization (Chopra et al., 2016) task while predicting the section header was a multiclass (twenty normalized section labels) classification task. Task B (Full Dialogue2Note Summarization) was another abstractive summarization task that required submissions to generate a complete clinical note from a whole dialogue between a patient and a doctor. The complete clinical note was expected to have the following first-level section headers: "HISTORY OF PRESENT ILLNESS", "PHYSICAL EXAM", "RESULTS", and "ASSESSMENT AND PLAN". Finally, Task C (Note2Dialogue Generation), a data augmentation (Shorten et al., 2021) task, was about generating patient-doctor conversations for complete input notes.
Aside from predicting section headers for Task A, all other tasks could be formulated as sequenceto-sequence (Sutskever et al., 2014) learning tasks. Various model architectures based on transformers (Vaswani et al., 2017) have proved to be successful at tackling such tasks. Therefore, leveraging pre-trained model checkpoints from public repositories was considered the right choice. Encouraged by the leaderboard for SAMSum (Gliwa et al., 2019) on HuggingFace (Wolf et al., 2020), a dialogue summarization dataset, we chose BART (Lewis et al., 2019) as the base model for our experiments. Specifically, we picked the facebook/bart-
228
large 1 model checkpoint (referenced as bart-large in this text from hereon) for its effectiveness on text-generation tasks.
The SAMSum (Gliwa et al., 2019) dataset is intended to train dialogue summarization models. However, we recognized that the input and target labels could be inverted to train a dialogue generation model. We trained/validated bart-large on the inverse of SAMSum (Gliwa et al., 2019) dataset followed by the Task C training dataset provided by the task organizers, achieving ROUGE-1 and ROUGE-2 scores of 59.11 and 23.69, respectively, on the validation set. This model was then used to augment datasets for Task A and Task B summarization tasks. In order to generate synthetic patientdoctor conversations, we chose to sample a thousand discharge summary notes from the MIMIC-IVNote (Johnson et al., 2023; Goldberger et al., 2000) dataset. We then added these dialogue-note pairs to the Task A and Task B training datasets provided by the organizers. The impact of this augmentation technique is noted in Section 5 below.
For Task A summarization, bart-large was fine-tuned on the SAMSum (Gliwa et al., 2019) dataset followed by fine-tuning on the augmented dataset for Task A, which achieved ROUGE-1 and ROUGE-2 scores of 50.7 and 21.4, respectively, on the validation set. Our methods yielded an overall improvement (over the baseline) of 13.1% and 14% in ROUGE-1 and ROUGE-2 scores, respectively. Results from fine-tuning bart-large on the unaugmented (original) Task A dataset were considered the baseline in this comparison.
Inspired by the significant gains exhibited by the Task A model, we decided to use it as the base model for Task B. Fine-tuning this base model on the augmented Task B dataset yielded ROUGE-1 and ROUGE-2 scores of 54.16 and 26.04, respectively - a 13.7% gain in ROUGE-2 score over the baseline. Results from fine-tuning the base model on the unaugmented (original) Task B dataset were considered the baseline in this comparison. The final submission(run1) achieved ROUGE-1 and ROUGE-2 scores of 49.5 and 23.4 on the test set. Unfortunately, the Task B dataset comprised input conversations almost twice as long as the maximum number of tokens accepted by bart-large, which naturally prohibits the model’s ability to summarize the entire conversation. To solve this prob-
1https://huggingface.co/facebook/ bart-large
lem, we developed an N-pass strategy in which the model attempts to summarize the conversation in multiple steps. Each step (or pass) involves the model taking as input the summary note of the dialogue processed till that step, concatenated with the rest of the dialogue. In other words, we trained the model to summarize a partial mix of an incomplete clinical note and an incomplete patientdoctor conversation. This strategy led to a gain of 6.6% and 8.1% in ROUGE-1 (57.76) and ROUGE2 (28.15) scores, respectively, on the validation set. We submitted the N-pass model as run2, which outperformed the run1 submission by 6.8%, both for ROUGE-1 (52.9) and ROUGE-2 (25) scores, on the test set. It also improved the division-based aggregate score by 16.75%. Overall, our methods improved the baseline ROUGE-2 score by 22.9% on the validation set, while the baseline ROUGE-1 score was found to be slightly better by 0.45%.
Given the promising performance of bart-large on the summarization tasks, we also decided to use it for Task A classification. We leveraged the BartForSequenceClassification wrapper offered by HuggingFace (Wolf et al., 2020), a BART model with a sequence classification head on top (a linear layer on top of the pooled output). Using this approach, we achieved an accuracy of 78% and an F1 score of 78.37%. The final submission was reported to have an accuracy of 73.5% on the test set.

heading: 2 Background and Related Work
text: Studies like the ones from Alkureishi MA et al. (Alkureishi et al., 2016) and Rathert et al. (Rathert et al., 2017) have presented evidence on EHRs (Electronic Health Records) impacting the quality of patient-doctor conversations. Digital scribes (van Buchem et al., 2021) and summarization tools (Shanafelt et al., 2016) can mitigate some of these problems. However, many challenges are associated with clinical dialogue summarization (Zhu and Penn, 2006). Some significant challenges include omitting key medical concepts (Knoll et al., 2022) and hallucinating unsubstantiated information.
Several attempts have been made to address said inherent challenges and automatically generate high-quality summaries of clinical encounters. Approaches like the ones used by Enarvi et al. (2020) have utilized a transformer (Vaswani et al., 2017) model to summarize doctor-patient conversations. Joshi et al. (2020) and Michalopoulos et al.
(2022) have also incorporated medical knowledge into these models. On the data generation front, Chintagunta et al. (2021) showed that large language models can be used for augmenting medical summarization datasets.
To the best of our knowledge, the N-pass strategy used to address long input sequences of Task B is novel. However, multiple multi-stage summarization approaches have been proposed so far. For example, Krishna et al. (2020) used modular summarization techniques to produce notes from patientdoctor conversations. Zhang et al. (2021) used multi-stage summarization for long inputs, whereas Gidiotis and Tsoumakas (2020) split a long document and its summary into multiple source-target pairs using sentence similarity. Recursive summarization incorporating human feedback (Wu et al., 2021) even achieved state-of-the-art results in book summarization.

heading: 3 Datasets

heading: 3.1 MEDIQA-Chat-2023
text: Task A training (validation) dataset (Ben Abacha et al., 2023) provided by the organizers consists of 1,201 (100) pairs of conversations and associated section headers and summaries. There were 20 unique normalized section headers overall. The Task B and Task C training (validation) set consists
of 67 (20) pairs of conversations and full clinical notes (Yim et al., 2023).

heading: 3.2 SAMSum
text: The SAMSum dataset contains 16369 conversations and their summaries (Gliwa et al., 2019), with a train/val/test split of 14732/818/819. Several dialogue summarization models have leveraged this dataset (Ni et al., 2022) and achieved promising results on the task. We note the impact of this dataset in the ablation study (Section 5).

heading: 3.3 MIMIC-IV-Note
text: MIMIC-IV-Note contains 331,794 deidentified free-text clinical notes for patients included in the MIMIC-IV clinical database (Johnson et al., 2023; Goldberger et al., 2000). We sampled a thousand notes from this dataset and used the Task C (dialogue generation) model for downstream data augmentation of Task A and Task B. Ablation study (Section 5) highlights significant contributions of this dataset to improving the results.

heading: 4 Methods

heading: 4.1 Dialogue Generation
text: We discovered that by flipping input and target labels, the SAMSum (Gliwa et al., 2019) dataset could also train a dialogue generation model. Our
recipe included fine-tuning bart-large on the inverted SAMSum (Gliwa et al., 2019) dataset for 10 epochs, followed by fine-tuning on a dataset that combined training and validation datasets from Task A and Task C for another 10 epochs. Finetuning was performed using the Trainer API offered by HuggingFace (Wolf et al., 2020), and the hyperparameters used are described in (Table 1). We did not perform a comprehensive sweep and recognize that a more optimal set of hyperparameters could yield better results. The model yielded by this recipe was also used for generating synthetic data for Task A and Task B summarization. Specifically, patient-doctor conversations were generated for 1000 discharge summary notes sampled from the MIMIC-IV-Note (Johnson et al., 2023; Goldberger et al., 2000) dataset. We used ROUGE-1 and ROUGE-2 scores for evaluating the model’s performance on the validation set (Lin, 2004).

heading: 4.2 Dialogue Summarization
text: Summarization models for Task A and Task B leveraged bart-large fine-tuned on the SAMSum (Gliwa et al., 2019) dataset for 10 epochs as the base model. The base model was then fine-tuned on the augmented version of the Task A training dataset for 30 epochs. Like dialogue generation, fine-tuning was performed using the Trainer API offered by HuggingFace (Wolf et al., 2020), and the hyperparameters used are described in Table 1. We did not perform a comprehensive sweep and recognize that a more optimal set of hyperparameters could yield better results. With a working hypothesis that the
Task A model can capture local themes in conversations with fewer turns, we used the model yielded by the above recipe as the base model for Task B.
Before augmenting the Task B dataset with the dialogue generation model, we sanitized the 1000 notes sampled from the MIMIC-IV-Note (Johnson et al., 2023; Goldberger et al., 2000) dataset. The sanitization process mainly included removing firstlevel section headers not accepted for evaluation by the organizers, as laid out in (Section 1). The base model was then fine-tuned on the sanitized-andaugmented dataset (named Augmented(Sections) in result tables) using the same process as Task A. This fine-tuned version was submitted as run1 and suffered from a significant drawback - the inability to handle input sequences longer than 1024 tokens. To address the shortcoming, we developed a novel N-pass approach by training a model that can generate summaries given a partial mix of incomplete summaries and incomplete dialogue. Specifically, a 2-pass version, named run2, was submitted to the shared task.
The N-pass approach is illustrated in Figure 1. The idea is to summarize long conversations in multiple passes, where each pass accepts as input the next block of the unsummarized dialogue concatenated with the summary output by the previous pass. The intuition behind this approach is to accommodate the limit on the number of input tokens accepted by the model by feeding it the dialogue in blocks but still propagating the context by incorporating the summary generated till that point. For run2, the model used for run1 was fine-tuned for
30 epochs on a dataset that concatenated the first block summary with the second block of the dialogue. The first block summaries were generated by the run1 model. A block size of 512 tokens was used for both the input and the output (except the final pass where output is 1024 tokens). We used a combination of ROUGE-1 and ROUGE-2 scores for evaluating the model’s ability to summarize the conversations in the validation set (Lin, 2004).

heading: 4.3 Classification
text: We used a simple yet effective classification approach to producing section headers for Task A. Given the promising results from using bart-large on the summarization and dialogue generation tasks, we chose to stick with the same for classification. To be exact, we fine-tuned the model used for Task A submission on the classification task by leveraging the BartForSequenceClassification wrapper offered by HuggingFace (Wolf et al., 2020), a BART model with a sequence classification head on top (a linear layer on top of the pooled output). Again, the Trainer API was used with no hyperparameter sweep. Table 1 lists the hyperparameters used for fine-tuning the classifier.

heading: 5 Experiments and Ablation Study

heading: 5.1 Task A
text: In Table 2, we compare the results obtained on the Task A validation set by using three different training datasets - original Task A training data, augmented Task A training data, and sanitizedand-augmented (defined in Section 4.2) training
data. The augmented version outperforms the original Task A training data by 6.7% (ROUGE-1) and 8% (ROUGE-2). As expected, the sanitized-andaugmented training data yields smaller gains because the summary notes for Task A are shorter and do not include first-level section headers in Task B training data.
An ablation study (Table 3) was also conducted on the impact of fine-tuning bart-large on the SAMSum(Gliwa et al., 2019) dataset. It was found that fine-tuning on the SAMSum (Gliwa et al., 2019) dataset improved performance on the validation set by 6% (ROUGE-1) and 5.4% (ROUGE-2).
Task A summarization model fine-tuned on classification achieved an accuracy of 78% and an f1 score of 78.37% on the validation set.

heading: 5.2 Task B
text: Table 4 shows that the 2-pass summarization strategy leads to a gain of 6.6% (ROUGE-1) and 8.1% (ROUGE-2). Furthermore, training on the sanitized-and-augmented dataset yields improvements of 12.5% (ROUGE-1) and 37% (ROUGE-2), driving home the value of data augmentation by clinical dialogue generation. Interestingly, simply fine-tuning on the SAMSum(Gliwa et al., 2019) dataset led to worse results (Table 5) on the Task B validation set, which could be explained by the discrepancy in the length of the conversations and the summaries between the two datasets.

heading: 5.3 Task C
text: The ablation study (Table 6) for Task C highlights two significant ideas. First, adding the training data from Task A contributed a hike of 4.5% (9%) in the ROUGE-1 score and 14.7% (31.7%) in the ROUGE-2 score for the model (not) fine-tuned on the inverse of the SAMSum (Gliwa et al., 2019) dataset. Second, fine-tuning on the inverse of SAMSum (Gliwa et al., 2019) led to a gain of 5.5% (1.1%) in ROUGE-1 scores and 19.5% (4.1%) in ROUGE-2 scores when training data from Task C (Task A + Task C) was used. It shows that the additional data from Task A is more critical when fine-tuning on the inverse of SAMSum (Gliwa et al., 2019) is skipped.

heading: 6 Results
text: Team Cadence’s submission for Task C earned rank-1 amongst all participants, beating the nextbest submission by 28.3% (ROUGE-1) and 99% (ROUGE-2).
The organizers shared test set results (Ben Abacha et al., 2023) along with a code status description where a code status of 1 meant that the organizers were able to run the submitted code and reproduce the results, and a code status of 2 meant that they were able to run the code and found minor differences with no changes in rankings. Code statuses 3,4, and 5 meant that the organizers found the submitted code to be unstable or not runnable under their configurations. Amongst code statuses 1 and 2, Team Cadence achieved the following ranks: rank-2 on TaskBsummarization, rank-3 on TaskA-summarization, rank-3 on TaskB-summarization(note-divisions), and rank-5 on TaskA-classification. The code for generating the submitted runs is being shared publicly2.
2https://github.com/ashwyn/ MEDIQA-Chat-2023-Cadence

heading: 7 System Specification
text: In the spirit of reproducibility, we share details of the systems used to run these experiments. The models were fine-tuned on g4dn.12xlarge AWS Sagemaker notebook instances 3. HuggingFace’s Python package transformers (Wolf et al., 2020) version 4.27.1 was used in a Python3.8 environment. Reported results were aggregated from 4 different runs using 4 different random seeds.

heading: 8 Limitations and Future Work
text: The methods described in this paper do not leverage any external medical knowledge, a technique that has been shown to be effective by other studies (Joshi et al., 2020; Michalopoulos et al., 2022). And like other methods based on large language models, in theory, our models are also prone to hallucinations and omission of key-clinical concepts. We plan to explore constrained beam search4 as a mitigation strategy for addressing these challenges in the future.
Although the impact of the Task C model as a data augmentation tool is undoubtedly positive (Section 5), qualitative error analysis of patientdoctor conversations produced by the model showed that the output contained a small number of dialogue turns, and each individual turn was too long, packed with information. Producing conversations with a more natural flow should yield an even better boost on downstream tasks, and we leave exploring such methods to future experimentation. We also recognize that N-pass summarization for Task B with higher values of N should be able to cover the entirety of the input conversations in the Task B datasets, albeit with diminishing returns as N increases. We hope to evaluate them in future iterations of similar shared tasks.

heading: 9 Conclusion
text: The two key takeaways from the experiments and results in this paper are significant improvements in summarization results driven by data augmentation and the N-pass summarization technique for handling long input patient-doctor conversations. Furthermore, the fact that our submissions to all three tasks share the same base (bart-large) model
3https://docs.aws.amazon.com/ sagemaker/latest/dg/notebooks.html
4https://huggingface.co/blog/ constrained-beam-search
speaks volumes of its versatility. Finally, the results demonstrate the effectiveness of fine-tuning on custom datasets for specialized domains like medicine.

heading: Future Work
text: We plan to explore constrained beam search4 as a mitigation strategy for addressing these challenges in the future. Although the impact of the Task C model as a data augmentation tool is undoubtedly positive (Section 5), qualitative error analysis of patientdoctor conversations produced by the model showed that the output contained a small number of dialogue turns, and each individual turn was too long, packed with information. Producing conversations with a more natural flow should yield an even better boost on downstream tasks, and we leave exploring such methods to future experimentation. We also recognize that N-pass summarization for Task B with higher values of N should be able to cover the entirety of the input conversations in the Task B datasets, albeit with diminishing returns as N increases. We hope to evaluate them in future iterations of similar shared tasks