NewAgeHealthWarriors at MEDIQA-Chat 2023 Task A: Summarizing Short Medical Conversation with Transformers
Year: 2023

Abstract:
This paper presents the MEDIQA-Chat 2023 shared task organized at the ACL-Clinical NLP workshop. The shared task is motivated by the need to develop methods to automatically generate clinical notes from doctor-patient conversations. In this paper, we present our submission for MEDIQA-Chat 2023 Task A: Short Dialogue2Note Summarization. Manual creation of these clinical notes requires extensive human efforts, thus making it a time-consuming and expensive process. To address this, we propose an ensemble-based method over GPT-3, BART, BERT variants, and Rule-based systems to automatically generate clinical notes from these conversations. The proposed system achieves a score of 0.730 and 0.544 for both the sub-tasks on the test set (ranking 8th on the leaderboard for both tasks) and shows better performance compared to a baseline system using BART variants.

heading: 1 Introduction
text: Telecare has experienced an exponential increase in utilization since the onset of the COVID-19 pandemic, leading to the emergence of a vast network of healthcare providers and patients Garfan et al. (2021). We consider this to be a significant use case within the Telecare domain, medical personnel often need to provide a concise summary of the conversation they had with their patient in order to ensure that a colleague is able to follow up on the next consultation. Both patients and medical professionals can use these summaries to refer back to their interactions in the future. Unfortunately, manually creating conversation notes after each encounter consumes a significant amount of time, and energy, and also poses challenges when done at scale.
Recent advancements in Natural Language Processing (NLP) with large language models (LLMs) like GPT-3 have shown promising results in their ability to generate convincing natural language and
successfully solve tasks including classification, answering questions, and summarization even in zero-shot and few-shot environments Brown et al. (2020). This makes them a popular choice as opposed to a pre-trained model, which needs to be adjusted separately for each downstream task. In this paper, we propose an ensemble of rule-based methods, traditional sequence models, large language models, and BERT-based models to develop an automated system for generating these notes from doctor-patient conversations. We also show that few-shot large language models outperform traditional sequence-to-sequence models in the setting of limited data.

heading: 2 Related Work
text: Summarization is a crucial task in NLP, particularly for extracting key information from multispeaker conversations. Various approaches have been proposed for meeting summarization, such as DialogLM Zhong et al. (2022), a pre-trained neural encoder-decoder model. In the context of medical dialogues between doctors and patients, identifying symptoms, diagnoses, and treatments is essential for deriving a medical solution. Song et al. (2020) introduced the hierarchical encoder-tagger model (HET) to specifically identify important utterances in medical conversations for summarizing medical conversations. Krishna et al. (2021) introduced pointer generator networks for deep summarization of physician-patient dialogues. Joshi et al. (2020) introduced a variant of the pointer generator network that handles negations and imposes a penalty on the generator distribution and Zhang et al. (2021) fine-tuned BART models for summarizing doctor-patient interactions.

heading: 3 Task and Dataset Details
text: The MEDIQA-Chat 2023 Ben Abacha et al. (2023a) shared task has been developed to foster research in the field of automatic clinical note
414
generation derived from doctor-patient conversations. It comprises three tasks1, namely, Short Dialogue2Note Summarization (Task A) Ben Abacha et al. (2023b), Full Dialogue2Note Summarization (Task B) wai Yim et al. (2023), and Note2Dialogue Generation (Task C) wai Yim et al. (2023). Our work primarily focuses on Task A. This task requires us to create a section summary, encompassing both the section header and text, based on a short input of a doctor-patient conversation.
The dataset consists of 20 distinct section headers for each conversation, such as Medications, Review of Systems, Past Surgical History, Chief Complaint, etc. The training set contains 1,201 pairs of conversations, each accompanied by their relevant section headers and text, while the validation set is composed of 100 pairs of conversations and their respective summaries. Table 1 shows statistics around the train/val/test data splits. Table 3 shows a few snippets of actual training data containing Section Header, Section Text, and doctor-patient conversation.

heading: 4 System Description
text: In this section, we give a detailed explanation of our proposed system2. We propose two separate pipelines - one for section header identification and the other for section text generation. We also discuss our ensemble strategy and related intuition.
1Task Page: https://github.com/abachaa/MEDIQA-Chat2023
2Code: https://github.com/prakhar21/MEDIQA-CHAT2023-NewAgeHealthWarriors
Finally, we output results from both pipelines to generate final summaries.
Section Header Identification: The task of Section Header Identification involves categorizing a given doctor-patient conversation to the relevant header from a list of pre-defined headers. Table 4 lists down all the available headers along with their expanded form which we received as a part of the task description.
We developed a 2-step strategy for detecting the accurate section header for a given doctor-patient conversation. Figure 1 shows the inference flow
of the Section Header Identification pipeline. In step-1, we categorize a given conversation to its section header using our Keyword lookup list. We refer to this as ‘Keyword based Categorizer’ in the diagram. We manually curated this list by going through many examples of section texts for every section header from the training data. If no section header is identified in this step, we pass the same conversation to the Bio_ClinicalBERT Alsentzer et al. (2019) model, variant of BERT Devlin et al. (2018), which we had fine-tuned on our dataset of conversation and section header pairs. Please refer to Section 5 and 6 for more details on model description, implementation, and results.
Section Text Generation: The task of Section Text Generation involves generating a summary of the given doctor-patient conversation. We propose an ensemble of 3 transformer-based models, i.e, BART-large, BioBART-large, and few-shot GPT-3 for the same. Here, we fine-tune BARTlarge Lewis et al. (2019) and BioBART-large Yuan et al. (2022) transformer models in a sequenceto-sequence paradigm on our training dataset. The
models were fine-tuned with input as doctor-patient conversations and output as associated section text with the training objective of maximizing the likelihood of the generated summary.
For GPT-3, we adopt a few-shot prompt engineering-based Liu et al. (2023) approach for generating our section text. Few-shot prompting helps enable in-context learning for large language models like GPT-3. Figure 2 shows a detailed annotation of the GPT-3 prompt that we use for our purpose. In the figure, <Dialogue Example> is an example dialogue from the training dataset that we sample randomly based on the predicted section header(<Section Header>) on the <Test Dialogue> sequence. The intuition behind adding this extra knowledge to our prompt was to help our model learn the writing style of actual section text. We also experimented by giving multiple examples of dialogue, section header, and section text as a part of our prompt. Please refer to Section 6 for more details on experiments.
During our initial analysis of the training dataset, we observed that there were certain specific writing
patterns present in the section text. For example - in numerous cases, the section text for ‘History of Present Illness’ starts with the patient’s age and gender. There were also cases where section text had words like ‘Noncontributory’, ‘None’, etc. To accommodate such a writing style in the model’s output, we added an additional text called <Coverage Text> for some of the selected section headers. We have shown some examples of coverage text in Table 2.
Figure 1 shows the inference flow of the Section Text Generation pipeline. Once the section text from each of the models is generated, we score all of them and pick the one with the maximum score as our choice of final generated section text. For each section text, we calculate a final score based on a weighted scoring scheme that combines both central and syntactic similarity scores with weights w1 and w2 respectively. We found w1 = w2 = 0.5 to work best for our use case. For calculating central similarity for each of the generated section text, we implement the work done in Kobayashi (2018) and for syntactic similarity, we calculate the tokenlevel Jaccard similarity between the section text and input dialogue. Jaccard similarity is defined as the ratio between the intersection of two sets and the union of two sets, and it is often used as a metric of similarity. Intuitively, section text that centrally captures the majority theme across all the generated section texts will have a high central similarity score, whereas, the syntactic similarity would help ensure faithfulness. Goel et al. (2021)
defines a faithful summary to be one that contains minimal information outside the source text. Finally, output from both pipelines is used to report the final generated clinical note.

heading: 5 Model Background
text: In this section, we discuss in brief the background of various machine learning models that we have used in our implementation.
Bio_ClinicalBERT: The Bio_ClinicalBERT3 model is initialized from BioBERT Lee et al. (2020) and trained on all notes from MIMIC III Johnson et al. (2016), a database containing electronic health records from ICU patients. The model was pre-trained on a GeForce GTX TITAN X 12 GB GPU, on a batch size of 32, a maximum sequence length of 128, and a learning rate of 5x10-5.
BART & BioBART: BART Lewis et al. (2019) 4 is a transformer encoder-decoder (seq2seq) model with a bidirectional (BERT-like) encoder and an autoregressive decoder to perform complex NLG tasks like summarization, translation, etc. BART is pre-trained to reconstruct the original text from the noisy text. BioBART Yuan et al. (2022) 5, a BART variant was pre-trained on PubMed abstracts to achieve biomedical domain adaption.
GPT-3: GPT-3 Brown et al. (2020) is an autoregressive language model with 175 billion parameters. It achieves strong performance on many NLP datasets, including translation, and questionanswering, as well as several tasks that require on-the-fly reasoning or domain adaptation.

heading: 6 Experiments and Results
text: In this section, we discuss experiments, implementation details, and results obtained for both our tasks.
Section Header Identification: We trained Fasttext Joulin et al. (2016) along with multiple BERT variants like BERT-base Devlin et al. (2018), RoBERTa Liu et al. (2019), and Bio_ClinicalBERT Alsentzer et al. (2019) for identifying section headers from doctor-patient dialogues. We use simpletransformers Rajapakse
3Bio_ClinicalBERT Model:https://huggingface. co/emilyalsentzer/Bio_ClinicalBERT
4BART Model: https://huggingface.co/ facebook/bart-large
5BioBART Model: https://huggingface.co/ GanjinZero/biobart-large
(2019) python library for fine-tuning all our transformer models. Amongst all of them, Bio_ClinicalBERT gave us the best score on the validation dataset. Our final model also incorporates a Keyword-based categorizer in the pipeline giving us the best accuracy of 77% on the validation set. We use weighted cross-entropy as our loss function because of the skewed distribution of headers in the training data. The corresponding weights per header category were calculated using sklearn’s compute_class_weight function on the training dataset and we train our best model for 10 epochs.
Section Text Generation: We fine-tuned BART, BioBART architectures and, also inferred GPT-3 model in a few-shot setting. Interestingly, GPT-3 in the few-shot setting outperforms all our fully supervised models by 1.2+ Rouge-Avg points (Refer Table 7). Rouge-Avg(RA) is the average score of Rouge-1 (R1), Rouge-2 (R2), Rouge-L (RL), and RougeLSum (RLS). We train both our BARTlarge and BioBART-large models for 5 epochs, set a beam size of 5 while decoding the sequence, and use the cross-entropy loss as our objective function. We tuned all the hyperparameters based on the performance of our model on the validation set. Before coming up with the final prompt structure for few-shot GPT-3, we experimented with a couple of things. We tested by keeping <Section Header> in their original form (as received in training data - acronymized) and also by replacing them to their full form, as received in the tasks description. Please refer to table 4 for a list of all headers and their expanded forms.
We experimented with the number of ground truth examples for our prompt. We tested with a maximum of 3 examples, because, for values higher than 3, the rate of getting the maximum token limit error from the GPT-3 API had increased significantly. Across 0, 1, 2, and 3, we found 3 to be giving the best results on the validation set. Finally, our ensemble of BART, BioBART, and GPT3 outperforms all our individual models by 0.9+ Rouge-Avg points (Refer Table 7) on the validation dataset. Some sample dialogues and generated section text are shown in Table 5.
Table 6 and 7 show the result of our evaluation for the Section Header Identification and Section Text Generation pipeline on the validation and test datasets respectively. The evaluators report a few more metrics such as BERTScore Zhang
et al. (2019), a metric that focuses on computing semantic similarity between tokens of reference and hypothesis, and BLEURT Sellam et al. (2020), a learned evaluation metric based on BERT for evaluating the generated summaries. The default models used for calculating BERTScore and BLEURT were ‘microsoft/deberta-xlarge-mnli’ 6 and ‘BLEURT-20’7 respectively. We report the score for these metrics in Table 7 for the test datasets due to computing constraints.

heading: 7 Observations
text: Here we discuss some observations that we made on results as shown in Table 5.
• With reference to examples 1 and 2, our model was able to correctly capture the year, duration, and other diagnostic details.
• With reference to example 3, our model was able to capture more details and also attempted to categorize diagnosis under relevant categories, which was not originally present in the ground truth summary.
• With reference to examples 4 and 5, our model generated some made-up facts such as the duration of the day, and the chest examination being an x-ray.
• With reference to examples 6 and 7, our model was accurately able to generate text with the
6DeBERTa Model: https://huggingface.co/ microsoft/deberta-xlarge-mnli
7BLEURT-20 Model: https://huggingface.co/ lucadiliello/BLEURT-20
same findings. However, it wrote it in an elaborate manner pushing R1 to 0.0.

heading: 8 Conclusion and Future Work
text: We have presented a novel ensemble-based approach for the task of automatic short medical dialogue to note summarization. Our method effectively combines fully supervised transformer models, few-shot GPT-3, and rule-based systems, generating accurate and coherent summaries of doctor-patient conversations. The proposed system demonstrates competitive performance on the MEDIQA-Chat 2023 Task A, highlighting its potential to enhance telecare and healthcare services.
As part of future work, we plan to explore advanced pre-trained models and techniques to further improve our system’s performance in the medical context. Additionally, we aim to investigate the applicability of our approach in handling more complex dialogues. We also plan to conduct an in-depth analysis of the generated summaries to identify areas for further fine-tuning.

heading: Future Work
text: As part of future work, we plan to explore advanced pre-trained models and techniques to further improve our system’s performance in the medical context. Additionally, we aim to investigate the applicability of our approach in handling more complex dialogues. We also plan to conduct an in-depth analysis of the generated summaries to identify areas for further fine-tuning