Overview of the MEDIQA-Chat 2023 Shared Tasks on the Summarization & Generation of Doctor-Patient Conversations
Year: 2023

Abstract:
Automatic generation of clinical notes from doctor-patient conversations can play a key role in reducing daily doctors’ workload and improving their interactions with the patients. MEDIQA-Chat 2023 aims to advance and promote research on effective solutions through shared tasks on the automatic summarization of doctor-patient conversations and on the generation of synthetic dialogues from clinical notes for data augmentation. Seventeen teams participated in the challenge and experimented with a broad range of approaches and models. In this paper, we describe the three MEDIQA-Chat 2023 tasks, the datasets, and the participants’ results and methods. We hope that these shared tasks will lead to additional research efforts and insights on the automatic generation and evaluation of clinical notes.

heading: 1 Introduction
text: Recent progress in text summarization and generative AI can greatly benefit the healthcare system by automatically generating clinical notes from doctor-patient conversations. This can contribute to effective clinical care by reducing the doctors’ workload to editing and validating the generated summaries/notes instead of writing the full notes during the consultations at the expense of their time or focus when talking and interacting with the patients.
Clinical note generation has seen an increased research interest in the recent years. For instance, (Yim and Yetisgen, 2021) tackled automatic medical scribing with Dialogue2Note sentence alignment and snippet summarization. (Michalopoulos et al., 2022) introduced MedicalSum, a guided clinical abstractive summarization model for generating medical reports from doctor-patient conversations. (Grambow et al., 2022) showed that in-domain pre-training improves clinical note generation from doctor-patient conversations. (Knoll
et al., 2022) presented three user studies, on medical note generation systems and analyzed the clinicians’ views of how the system could be adapted and improved. Other efforts focused on the evaluation of medical note generation manually through consultation checklists (Savkov et al., 2022) or automatically using evaluation metrics that correlate with human judgments (Moramarco et al., 2022; Adams et al., 2023; Ben Abacha et al., 2023b). (Papadopoulos Korfiatis et al., 2022) introduced the primock57 collection of 57 mocked primary care consultations, one of the rare datasets dedicated to this task.
The previous editions of the MEDIQA shared tasks focused on medical NLP tasks such as textual inference and question answering (Ben Abacha et al., 2019) as well as the summarization of patient questions/answers and radiology reports (Ben Abacha et al., 2021). This third edition, MEDIQAChat 20231, addresses the generation of clinical notes based on the summarization of doctor-patient conversations. All of the datasets and code created for this challenge are publicly available2.
In this paper, we present the tasks and datasets in section 2 and section 3. In section 4, we present the evaluation methods and metrics used for the shared tasks. Section 5 describes and discusses the participating teams’ approaches and draws insights from the official challenge results.

heading: 2 Tasks

heading: 2.1 Task A - Short Dialogue2Note Summarization
text: The first task focuses on summarizing short doctorpatient conversations to generate a summary for only one section of a clinical note, including a section header, as described in Figure 1.
1https://sites.google.com/view/mediqa2023/ clinicalnlp-mediqa-chat-2023
2https://github.com/abachaa/MEDIQA-Chat-2023
503
The section header is one of the following 20 headers: Family History/Social History (fam/sochx), History of Present Illness (genhx), Past Medical History (pastmedicalhx), Chief Complaint [cc], Past Surgical History (pastsurgical), allergy, Review of Systems (ros), medications, assessment, exam, diagnosis, disposition, plan, Emergency Department Course (edcourse), immunizations, imaging, Gynecologic History (gynhx), procedures, other_history, and labs.

heading: 2.2 Task B - Full Dialogue2Note Summarization
text: The goal of task B is to generate a complete note for each doctor-patient encounter, as described in Figure 2. The note must include all relevant sections. As the same section can have different correct expressions for its header, we defined four main section/division categories, each associated with several correct labels/expressions for its header. The section category-header mappings are presented in table 1.
Full-encounter notes are expected to have at most one section from each category. If a generated note contains multiple sections from the same category, only the first occurring section of that category is used for evaluation. Also, depending on the encounter, Objective_Exam and Objective_Results
may not be relevant.

heading: 2.3 Task C - Note2Dialogue Generation
text: This task addresses data augmentation through the generation of synthetic doctor-patient conversations from full clinical notes. We encouraged the participants to apply the models developed for this task to generate additional data for tasks A and B.

heading: 3 Datasets
text: Table 3 describes the training, validation, and test sets created from the MTS-Dialog (Ben Abacha et al., 2023a) and ACI-Bench (Yim et al., 2023) collections.
The MTS-Dialog dataset, used in Task A, consists of 1.7k pairs of conversations and associated summaries. Table 2 presents examples from MTSDialog conversations and summaries.
The ACI-Bench dataset, used Tasks B & C consists of 207 pairs of full doctor-patient conversations and associated clinical notes.

heading: 4 Evaluation
text: In this challenge, we evaluated both the submitted runs and the submitted codes as described below.

heading: 4.1 Evaluation Metrics
text: We selected three automatic metrics that highly correlate with human judgments for the task of clinical note generation based on recent studies (Ben Abacha et al., 2023a,b) on the evaluation methods for the summarization of doctor-patient conversations. These metrics are: ROUGE-1 (Lin, 2004), BLEURT (Sellam et al., 2020), and BERTScore (Zhang et al., 2020).
We used the average score from ROUGE-1, BLEURT-20, and BERTScore (microsoft/debertaxlarge-mnli) as the main score to rank the participating systems in short note generation (AggregateScore).
For full note generation, we relied on ROUGE-1 for the evaluation of full notes as BLEURT and BERTScore have a maximum sequence length of 512 tokens. For these notes, we also performed a more fine-grained sub-note section-level evaluation using the average score of the three metrics.
In summary, we used the following evaluation metrics for each task:
• Task A - Evaluating the section header classification using Accuracy.
• Task A - Evaluating the short summaries using the average score of ROUGE-1, BERTScore, and BLEURT.
• Task B - Evaluating the long summaries/notes with two different methods: (i) Full-note evaluation using ROUGE-1 and (ii) a finegrained evaluation taking the mean of the section-based combined score of ROUGE-1, BERTscore, BLEURT, equally weighed.
• Task C - Evaluating the generated dialogues using ROUGE-1.

heading: 4.2 Code Verification
text: The participants shared their private codes with the organizers on GitHub following the provided code preparation instructions 3.
3Evaluation instructions and scripts available at https: //github.com/abachaa/MEDIQA-Chat-2023
We defined five code statuses to label each team’s code (cf. Results Section):
1. Code runs and exactly reproduces
2. Code runs with minor differences
3. Results unstable due to non-deterministic components (e.g., generative API calls)
4. Results unstable
5. Code does not run under our configurations
We provided feedback on the shared codes and their outputs/errors to the participants.

heading: 4.3 Baseline Models
text: We used the latest OpenAI models to prepare baseline models using ChatGPT (gpt-3.5-turbo) and GPT-4. We used a temperature of 1 for tasks A
Section Header Conversation Summary MEDICATIONS Doctor: Are you still taking the Trizivir?
Patient: Yes. Doctor: How much are you taking? Patient: I take one pill two times a day. Doctor: Are you taking any other medications? Patient: I take Ibuprofen for body aches from time to time but that’s it.
1. She is on Trizivir 1 tablet p.o. b.i.d. 2. Ibuprofen over-the-counter p.r.n.
ROS Doctor: Have you had any anxiety attacks lately? Patient: No. Doctor: Have you felt depressed or had any mood swing problems? Patient: No. Doctor: Any phobias? Patient: No, not really. Doctor: Okay.
PSYCHIATRIC: Normal; Negative for anxiety, depression, or phobias.
and B. For task C, we experimented with two temperatures for more variety in the generated conversations with deterministic (temperature=0) and creative (temperature=1) outputs. ChatGPT has a limit of 4,097 tokens, shared between the prompt and the output/summary, whereas GPT-4 allows 32k tokens.
We ran the baseline models on an Nvidia Tesla K80 GPU.
We used the following prompt for tasks A, B, and C:
• Prompt for Task A: "Classify the conversation into one of these 20 classes: FAMILY HISTORY/SOCIAL HISTORY, HISTORY of PRESENT ILLNESS, PAST MEDICAL HISTORY, CHIEF COMPLAINT, PAST SURGICAL HISTORY, Allergy, REVIEW OF SYSTEMS, Medications, Assessment, Exam, Di-
agnosis, Disposition, Plan, EMERGENCY DEPARTMENT COURSE, Immunizations, Imaging, GYNECOLOGIC HISTORY, Procedures, Other history, Labs. The response should start with the selected class, followed by # then the summary of the conversation in a clinical note style. The conversation is: "
• We then extracted the section headers and summaries from the outputs.
• Prompt for Task B: "Summarize the conversation to generate a clinical note with four sections: HISTORY OF PRESENT ILLNESS, PHYSICAL EXAM, RESULTS, ASSESSMENT AND PLAN. The conversation is: "
• To allow adequate division detection, we added some light rule-based post-processing for Task B outputs.
• Prompt for Task C: "write a full conversation between a doctor and a patient during a medical visit. The dialogue should cover all the medical information provided in this note: "
Team Affiliation Tasks Paper Code 1 WangLab University of Toronto, Canada A, B (Giorgi et al., 2023) 1 2 SummQA Carnegie Mellon University, USA A, B (Mathur et al., 2023) 2 3 Cadence Cadence Solutions, USA A, B, C (Sharma et al., 2023) 3 4 GersteinLab Yale University, USA A, B (Tang et al., 2023) 4 5 NewAgeHealthWarriors IIITB, India A (Mishra and Desetty, 2023) 5 6 NUS-IDS NUS, Singapore A, C - 6 7 HuskyScribe University of Washington, USA A, B - 7 8 Calvados Université de Caen Normandie, France A, B (Milintsevich and Agarwal, 2023) 8 9 DS4DH University of Geneva, Switzerland A (Zhang et al., 2023) 9 10 UMASS_BioNLP University of Massachusetts, USA A, B, C (Wang et al., 2023) 10 11 HealthMavericks University of Mumbai, India A, B (Suri et al., 2023) 11 12 Care4lang George Washington University, USA A (Alqahtani et al., 2023) 12 13 clulab University of Arizona, USA A (Ozler and Bethard, 2023) 13 14 DFKI-MedIML German Research Center for AI, Germany A, B - 14 15 iuteam1 Indiana University, USA B (Srivastava, 2023) 15 16 SZU_Clinical Shenzhen University, China B - 16 17 Teddysum Kyungpook University, South Korea B (Jeong et al., 2023) 17
1 github.com/bowang-lab/MEDIQA-Chat-2023-WangLab 2 github.com/Raghav1606/SummQA 3 github.com/ashwyn/MEDIQA-Chat-2023-Cadence 4 github.com/28andrew/MEDIQA-Chat-2023-GersteinLab 5 github.com/prakhar21/MEDIQA-CHAT-2023-NewAgeHealthWarriors 6 github.com/Elfsong/MEDIQA-Chat-2023-NUS-IDS 7 github.com/BeanHam/MEDIQA-Chat-2023-HuskyScribe 8 github.com/501Good/MEDIQA-Chat-2023-Calvados 9 github.com/tinaboya/MEDIQA-Chat-2023-ds4dh 10 github.com/believewhat/MEDIQA-Chat-2023-UMASS_BioNLP 11 github.com/suri-kunal/MEDIQA-Chat-2023-HealthMavericks 12 github.com/amalqahtani/Clinical-NLP-Models 13 github.com/kbulutozler/MEDIQA-Chat-2023-clulab 14 github.com/sitingGZ/MEDIQA-Chat-2023-DFKI-MedIML 15 github.com/dhananjay-srivastava/MEDIQA-Chat-2023-iuteam1 16 github.com/SunnyLee216/MEDIQA-Chat-2023-SZU_Clinical 17 github.com/teddysum/MEDIQA-Chat-2023-Teddysum
Table 4: MEDIQA-Chat 2023: Participating teams, number of runs (with a limit of three runs/task), submitted codes, and working notes papers.

heading: 5 Official Results

heading: 5.1 Participating Teams
text: The MEDIQA-Chat shared tasks attracted 120 registered teams from academy and industry. Among them, 17 teams submitted their codes and runs following the challenge rules. Table 4 presents the teams that participated in the three shared tasks. We limited the number of submitted runs to three runs per task.

heading: 5.2 Task A: Approaches & Results
text: Task A includes two subtasks on (i) generating the summary of a short medical conversation and (ii) classifying the sections/summaries using a predefined list of section headers. Fourteen teams participated in Task A. Table 5 presents the results of the section classification subtask and Table 6 presents the results of the summarization subtask.
In task A, most teams used fine-tuned models (e.g., BART, T5) and/or OpenAI-based solutions in the summarization subtask and leveraged finetuned BERT or RoBERTa-based models for section classification. The WangLab team (Giorgi et al., 2023) achieved the best results in the summarization subtask with 0.5789 Aggregate-Score and the best Accuracy of 0.78 in the header classification subtask using a Flan-T5 model that jointly generates the section header and content. The NUS-IDS team also achieved the best Accuracy of 0.78 in header classification and 0.5204 Aggregate-Score in summarization using a T5 model fine-tuned on data augmented by GPT-3. The HuskyScribe team also used a T5-based model (T5-Large and ClinicalT5-Large) trained in a question-answering format for section header classification. Their summarizer consisted of a BART-large-xsum model finetuned on task A’s training data, the Samsum dataset (Gliwa et al., 2019), and the Dialogue-sum dataset (Chen et al., 2021). Care4Lang (Alqahtani et al., 2023) used a Flan-T5 model fine-tuned on the training data with a pre-processed input combining the header and the dialogue for implicit header learning and conditional summary generation. Clinical-T5-
Sci models were used by the clulab team (Ozler and Bethard, 2023) to generate three different summaries for each dialogue to augment the header classification training data, and then used a Robertabased model trained on the augmented dataset to predict the header based on the summary of the dialogue instead of the dialogue itself. The Calvados team (Milintsevich and Agarwal, 2023) used a LongT5 model for summarization and clinical NER model to extract disease and treatment mentions that are then tagged in the input conversation and the output summary. They combined the classification label and the summary note into a single output, and considered the classification as a subtask within summary generation.
The SummQA team (Mathur et al., 2023) utilized an ensemble of BioClinicalBERT and GPT-4 for section header classification. GPT-4 was used
as a zero-shot classifier and BioClinicalBERT was fine-tuned on the task A training data. Their summarization method relied on GPT-4 with prompt selection based on semantic similarity to retrieve topk (k=7) examples for in-context learning and was ranked third in TaskA-Summarization with 0.5739 Aggregate-Score. The DS4DH team (Zhang et al., 2023) used a classification model (tf-idf-svm) in combination with ChatGPT (run1) or GPT-3 Curie (run2) for summarization. The UMASS-BioNLP team (Wang et al., 2023) also used ChatGPT to jointly generate the section header and note.
The Cadence team (Sharma et al., 2023) adapted a BART-large model for classification and summarization. The summarizer was a BART-large model fine-tuned first on the Samsum dataset and second on Task A data augmented with 1k note samples extracted from MIMIC-IV and their dialogues
generated by their Task C model. The NewAgeHealthWarriors team (Mishra and Desetty, 2023) also used a fine-tuned BART-large, BioBART-large and calls to GPT-3 API with custom prompt design, followed by an ensemble module to choose the best summary from the previous summarization models. A fine-tuned Bio-ClinicalBERT followed by a Keyword-based categorizer were used for section header classification. The DFKI-MedIML team used a fine-tuned microsoft/biogpt model for generating the section header and section summary. They modified the original BioGptForCausalLM model to encode a list of context input sequences for generating one target output. The HealthMavericks team (Suri et al., 2023) used an ensemble of BioBart-V2, DialogLM-LED-Base, DialogLED-Large, Flan-T5 fine-tuned on the training data (runs 1&2) and GPT-3 with the input dialogue and three randomly sampled dialogue-section-headersummary triplets as prompt.

heading: 5.3 Task B: Approaches & Results
text: Nine teams participated in Task B. We present the results of the full-note evaluation in Table 7 and the section-level evaluation in Table 8.
The WangLab team (Giorgi et al., 2023) used GPT-4 with in-context examples retrieved from the training set based on their similarity to the test dialogues and included their summaries/notes as in-context examples and obtained the best ROUGE-
1 score of 0.6141 in full-note evaluation and an Aggregate-Score of 0.6483 in section-based evaluation. SummQA (Mathur et al., 2023) used one-shot GPT-4 with dynamic prompts that include selected examples for in-context learning. The examples consist of dialogue-summary pairs selected from the Task B training data based on semantic similarity and obtained 0.5541 Aggregate-Score. Several teams also used OpenAI-based solutions: GersteinLab (Tang et al., 2023) used the Davinci model, UMASS_BioNLP (Wang et al., 2023) used GPT-4, ad healthmavericks (Suri et al., 2023) used GPT-3 to generate the summaries/clinical notes with static prompts.
The iuteam1 team (Srivastava, 2023) used three different LSG BART models to summarize long conversations using Local, Sparse, and Global Attention mechanisms and evaluated the use of multi-layer structures where multiple summarization model outputs are recombined in a single model to produce more coherent texts. The Cadence team (Sharma et al., 2023) adapted their task A method to task B data, and developed a twopass summarization approach to manage longer inputs. They fine-tuned BART on the Samsum dataset, Task A and Task B training data, and on additional examples generated from MIMIC-IV notes using their Task C solution.
The GersteinLab team (Tang et al., 2023) used a fine-tuned GPT-3 model for summarization trained
with a dynamic maximum length and a RoBERTabased model for classification. Similarly to their method for task A, the Calvados team (Milintsevich and Agarwal, 2023) used a LongT5 model finetuned on a combined data from Task A and Task B with different prompts. They split the note into four divisions; the input dialogue is copied for each division and prepended with a task-specific prompt.
The healthmavericks team used a BioClinicalBERT multi-label model with focal loss to classify an utterance into all possible sections using Task A data. The grouped utterances of each section are then passed through the summarizer to generate a summary. For summarization, they fine-tuned two transformer-based models: DialogLED-Base and DialogLED-Large and used the same ensemble techniques as in task A to select the final summary. The Teddysum team (Jeong et al., 2023) generated separate summaries for each section using the DialogLED model and experimented with contrastive learning to avoid the repetition of the same content in different sections and obtained 0.5332 ROUGE-1 in full-note evaluation.

heading: 5.4 Task C: Approaches & Results
text: Table 9 presents the results of Task C on the generation of doctor-patient conversations from clinical notes. The Cadence team (Sharma et al., 2023) achieved the best ROUGE-1 score of 0.5435 using a BART-large model, fine-tuned on an inverse version of the Samsum dataset, and then on a combination of Task A, Task B, and Task C datasets. This model was also utilized to augment the training data of the Task A and Task B summarization systems. The NUS-IDS team used T5 models finetuned on Task C’s training data. UMASS_BioNLP (Wang et al., 2023) applied ChatGPT and GPT-4 to generate conversations from the notes. In order to reduce the prompt length, they applied the models iteratively, feeding them with only the prompt for the next conversation segment at each step, and restricting the prompt content to the conversation segment generated for the previous section/topic. This allowed the generation of longer conversations within the maximum token limit.

heading: 6 Conclusion
text: With the recent progress in Large Language Models (LLMs), the MEDIQA-Chat 2023 shared tasks provided an opportunity to evaluate the recently released LLMs (e.g., GPT-4, ChatGPT) vs. older models (e.g., T5, BART) in order to develop SOTA models and approaches for the summarization and generation of doctor-patient conversations. The variety of runs submitted by the participating teams and the explored augmentation, fine-tuning, and prompting methods provided new insights on the best approaches and techniques for future research directions in domain-specific text generation. The best results in the summarization of short dialogues were obtained using a Flan-T5 model that jointly predicts the section header and generates the section text (WangLab team). The team’s approach on long dialogues also yielded the best challenge results using GPT-4 with in-context examples se-
lected from task B training data. In task C, the best results were from the Cadence team which leveraged a BART-large model fine-tuned on different datasets to generate conversations from clinical notes to augment tasks A and B training data.
The newly introduced benchmarks allowed the organization of these shared tasks and the evaluation of the participating systems on unseen test sets. Automatic evaluation remains an important and challenging task. In this edition, we relied on an ensemble of evaluation metrics and we added a new requirement to submit the code for a second evaluation of the outputs. We hope that these shared tasks will encourage further efforts towards automatic clinical note generation using recent AI advances to reduce the workload for medical professionals and to improve the quality and outcomes of doctor-patient encounters.
Limitations
The paper does not cover all types of possible methods and models for the generation of clinical notes. The challenge datasets are also limited in terms of size and medical specialities. Further experiments and evaluations are needed to validate the best performing methods on other datasets and scenarios.

heading: Acknowledgements
text: We would like to thank Thomas Lin from Microsoft Health AI and the ClinicalNLP organizers for their feedback and support for the MEDIQA-Chat 2023 shared tasks. We also thank our annotation team for preparing the data in time for the challenge and all the participating teams who contributed to the success of these shared tasks through their interesting approaches and experiments and strong engagement.

heading: Future Work
text: The variety of runs submitted by the participating teams and the explored augmentation, fine-tuning, and prompting methods provided new insights on the best approaches and techniques for future research directions in domain-specific text generation. The best results in the summarization of short dialogues were obtained using a Flan-T5 model that jointly predicts the section header and generates the section text (WangLab team). The team’s approach on long dialogues also yielded the best challenge results using GPT-4 with in-context examples se-
lected from task B training data. In task C, the best results were from the Cadence team which leveraged a BART-large model fine-tuned on different datasets to generate conversations from clinical notes to augment tasks A and B training data. The newly introduced benchmarks allowed the organization of these shared tasks and the evaluation of the participating systems on unseen test sets. Automatic evaluation remains an important and challenging task. In this edition, we relied on an ensemble of evaluation metrics and we added a new requirement to submit the code for a second evaluation of the outputs. We hope that these shared tasks will encourage further efforts towards automatic clinical note generation using recent AI advances to reduce the workload for medical professionals and to improve the quality and outcomes of doctor-patient encounters. Limitations
The paper does not cover all types of possible methods and models for the generation of clinical notes. The challenge datasets are also limited in terms of size and medical specialities. Further experiments and evaluations are needed to validate the best performing methods on other datasets and scenarios