Overview of the First Shared Task on Clinical Text Generation: RRG24 and "Discharge Me!" (2024)

Abstract: Recent developments in natural language generation have tremendous implications for healthcare. For instance, state-of-the-art systems could automate the generation of sections in clinical reports to alleviate physician workload and streamline hospital documentation. To explore these applications, we present a shared task consisting of two subtasks: (1) Radiology Report Generation (RRG24) and (2) Discharge Summary Generation (“Discharge Me!”). RRG24 involves generating the ‘Findings’ and ‘Impression’ sections of radiology reports given chest X-rays. “Discharge Me!” involves generating the ‘Brief Hospital Course’ and ‘Discharge Instructions’ sections of discharge summaries for patients admitted through the emergency department. “Discharge Me!” submissions were subsequently reviewed by a team of clinicians. Both tasks emphasize the goal of reducing clinician burnout and repetitive workloads by generating documentation. We received 201 submissions from across 8 teams for RRG24, and 211 submissions from across 16 teams for “Discharge Me!”.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 85–98 August 16, 2024. ©2024 Association for Computational Linguistics
85
Recent developments in natural language generation have tremendous implications for healthcare. For instance, state-of-the-art systems could automate the generation of sections in clinical reports to alleviate physician workload and streamline hospital documentation. To explore these applications, we present a shared task consisting of two subtasks: (1) Radiology Report Generation (RRG24) and (2) Discharge Summary Generation (“Discharge Me!”). RRG24 involves generating the ‘Findings’ and ‘Impression’ sections of radiology reports given chest X-rays. “Discharge Me!” involves generating the ‘Brief Hospital Course’ and ‘Discharge Instructions’ sections of discharge summaries for patients admitted through the emergency department. “Discharge Me!” submissions were subsequently reviewed by a team of clinicians. Both tasks emphasize the goal of reducing clinician burnout and repetitive workloads by generating documentation. We received 201 submissions from across 8 teams for RRG24, and 211 submissions from across 16 teams for “Discharge Me!”.

heading: 1 Introduction
text: An important application of natural language generation (NLG) in medical artificial intelligence (AI) is radiology report generation (RRG). Specifically, an RRG system can be designed to accept radiology images (e.g.,
*Equal contribution
chest X-rays) of a patient and generate a textual report describing the clinical observations in the images. This is a clinically important task, and offers the potential to reduce the repetitive work of radiologists and generally improve clinical communication (Pang et al., 2023). Existing studies have been conducted using a single dataset, which limits the scale and diversity of the data and results. Therefore, we introduce our first subtask, RRG24, where we curate Interpret-CXR, a large-scale collection of RRG datasets from a variety of different sources (i.e., MIMIC-CXR (Johnson et al., 2019), CheXpert (Irvin et al., 2019), PadChest (Bustos et al., 2020), BIMCV-COVID19 (Vayá et al., 2020), and OpenI (Demner-Fushman et al., 2016)). In RRG24, participants generate the Findings and Impression sections from chest X-rays. We then evaluate the generations on common leaderboards with standard and recently proposed metrics. Ultimately, this task aims to benchmark recent progress using common data splits and evaluation implementations.
NLG can also impact discharge documentation by playing a role in generating discharge summaries. Hence, we introduce our second subtask, “Discharge Me!”, with the primary objective of encouraging NLG systems that alleviate clinician burden when writing detailed discharge summaries. Clinicians play a crucial role in documenting patient progress after a hospital stay, but the creation of concise yet comprehensive Brief Hospital Course (BHC) sections and Discharge Instructions often demands a significant investment of time (Do et al., 2020; Alissa et al., 2021). These two sections in particular cannot be readily copied from prior notes, and thus must be written from scratch by clinicians who synthesize information from across the patient record (Weetman et al., 2021). This process
contributes to clinician burnout and poses operational inefficiencies within hospital workflows (Haycock et al., 2014). We hypothesize that computer-generated clinical documentation has the potential to more accurately and completely capture a patient’s hospital course while reducing the administrative burden on clinicians, which, in turn, mitigates burnout, streamlines hospital operations, and ultimately improves the quality of care. Thus, in “Discharge Me!”, participants submit generations of both target sections (BHC & Discharge Instructions). We evaluate submissions on a common leaderboard and conduct a subsequent manual clinician review to measure clinical alignment of the outputs.

heading: 2 Related Work

heading: 2.1 Radiology Report Generation
text: Recent advances in computer vision (CV) and NLG have shown great potential for the automatic generation of radiology reports. This progress can be summarized from three perspectives:
• (1) Data: Most relevant studies focus on chest Xrays, mainly owing to the current number of publicly available image-report datasets for this modality (e.g., MIMIC-CXR, PadChest, and OpenI, etc.). Recently, there have also been studies expanding the scope of radiology report generation to other modalities (e.g., computed tomography (CT) (Loveymi et al., 2021; Hamamci et al., 2024) and ultrasound (Zeng et al., 2020; Yang et al., 2021; Huh et al., 2023)). • (2) Methodology: The methods for radiology report generation have evolved from task-specific modeling to pre-training-based approaches. For the former, researchers have incorporated the task priors into the designs of the model architectures (Shin et al., 2016; Zhang et al., 2017; Jing et al., 2018; Chen et al., 2020; Zhang et al., 2020; Liu et al., 2021; Delbrouck et al., 2022a; Hou et al., 2023), whereas for the latter, researchers have performed domain-specific representation learning using vision encoders or have adopted large pretrained language decoders (Thawkar et al., 2023; Hyland et al., 2023; Tu et al., 2024). • (3) Evaluation: One of the largest factors hampering radiology report generation progress is the selection of evaluation metrics. Due to its domainspecific characteristics, simple n-gram matching metrics (e.g., BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), and CIDEr (Vedantam et al., 2015)) are sub-optimal choices for this task. However, researchers have proposed various model-based metrics for evaluating the quality of generated reports, such as BERTScore (Zhang et al., 2019), F1CheXbert (Smit et al., 2020), F1-RadGraph (Delbrouck et al., 2022a), and GREEN (Ostmeier et al., 2024).

heading: 2.2 Discharge Summary Generation
text: Previous research has also examined AI technologies for the generation of discharge summaries to alleviate clerical burden for clinicians. For instance, several studies investigated GPT-3.5’s and GPT-4’s capability to generate discharge notes in tandem with various prompting strategies. In a UK pilot feasibility study, it was observed that a set of 25 AI-generated summaries were all deemed acceptable by general practitioners, compared to 23/25 (92%) of summaries written by junior doctors (Clough et al., 2024). Other studies similarly concluded that these proprietary models exhibit great potential and are able to generate acceptable discharge summaries with minimal misinformation (Kim et al., 2024; Waisberg et al., 2023). However, despite being able to increase efficiency and reduce the time required for documentation as compared to writing or dictating notes, instances of hallucination or omission of clinically significant facts were observed for certain discharge summaries involving complex surgeries. As such, the factual correctness of these large language models (LLMs) for specific generation tasks could be improved (Williams et al., 2024; Dubinski et al., 2024).
Based on this, some studies have focused on generating a particular section common to most discharge summaries – BHC – optimizing for correctness and faithfulness. The BHC is a succinct summary of a patient’s entire journey through the hospital and are embedded within complex discharge summaries. Efforts in compiling large-scale datasets for the generation of these BHC sections (Adams et al., 2021), including those with synthetic data (Adams et al., 2022), have led to subsequent contrastive learning methods for aligning generation models (Adams et al., 2023). Finally, methods leveraging heuristics to increase factuality (e.g., retrieval and ontology referencing) have also been developed (Adams et al., 2024; Hartman et al., 2023).
Some research has similarly centered on the Discharge Instructions section, sometimes known as the Patient Instructions section. This section is patient-facing and details instructions for the patient to continue their care at home, such as information on diet, therapies, and medications, as well as any details for follow-up appointments. Patient readability of this section is critical, and LLMs could be used to reformulate them into a more patient-friendly language (Zaretsky et al., 2024). Similar to the BHC, previous work also explored frameworks for the generation of faithful Patient Instructions (Liu et al., 2022).

heading: 3 RRG24: Radiology Report Generation
text: RRG24 was hosted on ViLMedic (Delbrouck et al., 2022b), a modular framework for vision-language multimodal research in medicine. The library contains reference implementations for state-of-the-art visionlanguage architectures for medicine and also hosts shared challenges in AI. A total of 201 submissions were received from across 8 teams.

heading: 3.1 Data
text: We curated Interpret-CXR, a large-scale collection of RRG datasets from the following five sources: MIMIC-CXR (Johnson et al., 2019), CheXpert (Irvin et al., 2019), PadChest (Bustos et al., 2020), BIMCVCOVID19 (Vayá et al., 2020), and OpenI (DemnerFushman et al., 2016). The breakdown of InterpretCXR, including details of the four splits used in RRG24 (Training, Validation, Public Test, and Hidden Test) are reported in Table 1.

heading: 3.2 Evaluation
text: We applied two types of metrics to evaluate different systems: n-gram-based and model-based metrics. For the former, we adopted BLEU-4 (Papineni et al., 2002) and ROUGE-L (Lin, 2004), whereas for the latter, we adopted BERTScore (Zhang et al., 2019), F1CheXbert (Smit et al., 2020), and F1-RadGraph (Delbrouck et al., 2022a). To standardize the evaluation process, we used the same script from ViLMedic to evaluate all systems. By doing so, we avoid different teams using different versions or hyperparameters for a given metric – for example, some existing studies use differing versions of BERTScore, leading to inconsistent score reporting.

heading: 3.3 Results
text: The automatic results for the Findings and Impression sections are shown in Tables 2 and 3, respectively (Note: iHealth-Chile-1 did not submit scores for Impression generation, and thus is not included in Table 3). We congratulate e-Health CSIRO, MAIRA, and AIRI for their outstanding performance on both Findings and Impression generation. It is also worth highlighting that the other teams (Gla-AI4BioMed, SICAR, CID, iHealthChile-3&2, and iHealth-Chile-1) designed novel solutions as well, providing insights for future research in this field beyond the competition. We also ran an evaluation using GREEN for the top 2 best-scoring systems (e-Health CSIRO and MAIRA) and recorded scores of 36.9 and 35.2, respectively, aligning with the leaderboard rankings1.

heading: 3.4 Descriptions of Systems

heading: 3.4.1 e-Health CSIRO
text: e-Health CSIRO (Nicolson et al., 2024) integrated entropy regularization into self-critical sequence training to help maintain a higher entropy in the token distribution, preventing overfitting to common phrases and ensuring a broader exploration of the vocabulary during training. They applied this to a multimodal language model with RadGraph as the reward. Additionally, their model incorporated several other features: (i) the use of type embeddings to differentiate between Findings and Impression section tokens; and (ii) the use of a
1We adopted GREEN instead of the naive GPT-4 pairwise comparison since Ostmeier et al. (2024) found GPT-4 to have low correlation with expert preference.
non-causal attention mask for image embeddings and a causal mask for report token embeddings.

heading: 3.4.2 MAIRA
text: MAIRA (Srivastav et al., 2024) combined a CXRspecific image encoder with a pre-trained LLM (Vicuna7B-v1.5) via a multi-layer perceptron (MLP) adapter of 4 layers. The image encoder is a ViT-B model that leverages DINOv2, a state-of-the-art self-supervised learning method. Both the LLM and the adapter are fine-tuned in a single stage training setup for RRG. Their results indicated that joint training for Findings and Impression prediction improves the metrics for Findings generation. Additionally, incorporating lateral images alongside frontal images further enhances all metrics. They showed that scaling the model size from Vicuna-7B to Vicuna-13B also improves metrics. To handle multiple predictions for a study (as each study can have multiple frontal and/or lateral images), they utilized GPT-4 to select the best report.

heading: 3.4.3 AIRI
text: AIRI (Samokhin et al., 2024) utilized the LLaVA framework, where the vision encoder is a DINOv2 trained on medical data and the language decoder is a specialized biomedical LLM. They used the same model to generate both Impressions and Findings with different prompts: “Write findings for this X-ray.” or “Write impression for this X-ray.”. The system prompt from LLaVA-Med (Li et al., 2024) was also used.

heading: 3.4.4 Gla-AI4BioMed
text: Gla-AI4BioMed (Zhang et al., 2024) leveraged the Vicuna-7B architecture and integrated a CLIP image encoder with a fine-tuned LLM. The model underwent a two-stage training process, whereby chest X-ray features are initially aligned with the language model, and said model is subsequently fine-tuned for report generation. The model processed multiple images simultaneously by stitching them together, mimicking the workflow of radiology professionals.

heading: 3.4.5 SICAR
text: SICAR (Udomlapsakul et al., 2024) incorporated the SigLIP vision encoder and the Phi-2-2.7B language model to train an efficient RRG model. They also implemented a novel two-stage post-processing pipeline. They first enhanced the readability and clarity of the reports, then cross-verified the model outputs by integrating X-Raydar, an advanced X-ray classification model, addressing false negatives.

heading: 3.4.6 CID
text: CID (Liao et al., 2024) proposed a novel paradigm for incorporating graph structural data into the RRG model. Their approach involved predicting graph labels based on visual features and subsequently initiating the decoding process through a template injection conditioned on the predicted labels. These results provided preliminary

heading: 3.5 iHealth-Chile-3&2
text: iHealth-Chile-3&2 (Loch et al., 2024) focused on exploring various template-based strategies using predictions from multi-label image classifiers as input, which was inspired by prior work on template-based report generation. Two approaches were explored: (i) a straightforward implementation from Pino et al. (2021) directly; and (ii) replacing the fully connected layer with an attention-based pooling mechanism conditioned on a fact embedding.

heading: 3.6 iHealth-Chile-1
text: iHealth-Chile-1 (Campanini et al., 2024) developed a new strategy for in-context learning. Their system is formed using a vision-encoder, a vision-language connector or adapter, and a LLM able to process text and visual embeddings. They also designed an enriched prompt by combining a standard instruction (“Write the finding section of a chest x-ray radiology report”) with reports generated by a multi-label classifier and a group of template sentences.

heading: 3.7 Limitations & Challenges
text: The evaluation for medical text generation is challenging due to its domain-specific characteristics, making it difficult to measure performance as it relates to clinical utility. This challenge leveraged common metrics that are used by existing RRG studies. Unfortunately, these evaluations may be limited when considering the real-world clinical impact of the submitted systems.

heading: 4 “Discharge Me!”: Discharge Summary Generation
text: “Discharge Me!” was hosted on Codabench (Xu et al., 2022), an open source platform used to organize various tasks and benchmarks. A total of 211 submissions was received from across 16 teams.

heading: 4.1 Data
text: Participants were provided a dataset derived from the MIMIC-IV-Note module (Johnson et al., 2023). The modified and filtered dataset included 109,168 hospital admissions from the Emergency Department (ED), split into four sets (Training, Validation, Phase I Test, and Phase II Test) (Xu, 2024). Each visit includes chief complaints and diagnosis codes (either ICD-9 or ICD-10) documented by the ED2, at least one radiology report, and a discharge summary with both BHC and Discharge Instructions sections.
The generation targets for the BHC were extracted from the full discharge notes using a complex regular expression strategy that searched for relevant section headers and new-line formatting characters. A similar strategy was used for Discharge Instructions; however, given that this section is usually located at the end of a discharge note as its very last section, extraction was more trivial. Samples where the extracted length of either section was shorter than 10 words were removed
2We assume ED diagnosis codes are available to the discharging clinician as ED documentation is likely to be complete at the time of discharge in most cases. However, we acknowledge that ICD codes may not necessarily be finalized, so they will be removed in future iterations of the shared task.
and deemed invalid. The complete breakdown of the dataset is available in Table 4.
Participants were allowed to incorporate external datasets, either publicly available or proprietary, as well as link additional patient data from other MIMIC-IV modules. Additionally, with the exception of the test dataset, participants were given the flexibility of using all or part of the provided dataset in any combination as they see fit.

heading: 4.2 Evaluation

heading: 4.2.1 Automatic Scoring
text: Automatic scoring took place on Codabench with a Python 3.9 environment. A hidden subset of 250 samples from the test datasets of the respective phases was used to evaluate the submissions. The metrics for this task were based on a combination of textual similarity (n-gram-based lexical metrics) and factual correctness of the generated text. Specifically, we considered the following metrics to automatically score submissions: BLEU-4 (Papineni et al., 2002), ROUGE-1/-2/L (Lin, 2004), BERTScore (Zhang et al., 2019), METEOR (Banerjee and Lavie, 2005), AlignScore (Zha et al., 2023), and MEDCON (Van Veen et al., 2024).
Initially, submissions were scored on both target sections separately (BHC & Discharge Instructions). The mean across all test samples were computed for each metric, resulting in several performance scores for each of the two target sections (not reported on the leaderboard). Then, for each metric, we took the mean of the scores for each of the two target sections (reported under the metric name on the leaderboard). Finally, we computed the mean once again over all the metrics to arrive at a final overall system score (reported as Overall Score on the leaderboard).
For instance, given N samples, suppose s is defined as the score for a given sample for a given metric, then the mean across all samples for a particular target section, S, would be calculated by:
S = N∑ 1 (si)/N (1)
We then calculated β, the mean of a given metric over both target sections, for each of the 8 metrics using:
β = (SBHC + SDischargeInstructions)/2 (2)
Finally, the overall system score was calculated by taking the mean of the 8 β values:
Overall = 8∑ 1 (βi)/8 (3)

heading: 4.2.2 Clinician Scoring
text: At the end of the competition, the submissions from the top 6 best-scoring teams were reviewed by a group of six clinicians with diverse experiences in a broad range of specialties (two adult hospitalists, two clinical informatics fellows trained in pediatrics, a neurosurgeon, and a radiologist). Generated sections were evaluated for their completeness, correctness, and readability, as well as in a holistic comparison against the reference target sections (ground truth). In particular, completeness evaluates whether the generated text captures the clinically important information available in the reference text. In cases where there is inaccurate information, correctness specifies whether and how likely this mistake would lead to unintended impacts in future care. Readability was only evaluated by the clinicians for the BHC section as the intended audience of the Discharge Instructions section is the patient. Finally, the holistic comparison aimed to capture overall clinician preference.
Clinicians were presented with the reference target sections and the generated target sections side-by-side on a web-based survey dashboard hosted via Streamlit. Additionally, the full discharge summary was available in case reviewers required further context. They were then presented with a series of multiple-choice questions capturing each of the above criteria in a scale from 1 to 5, where 1 was the most negative option, and 5 was the most positive option.
Each clinician was provided with generated samples from three teams for evaluation. To minimize recall bias, we presented the generated submissions from all three teams consecutively in a randomized order for one particular sample, before moving onto the next.
Each team’s submission was evaluated by three separate clinician reviewers. Scores were averaged and several agreement and reliability scores were calculated, including Cohen’s Kappa and Fleiss Kappa for interobserver agreement (McHugh, 2012; Landis and Koch, 1977), as well as the intraclass correlation coefficient (ICC) (Liljequist et al., 2019).
Table 4: Dataset Breakdown for “Discharge Me!”
Item Total Count Training Validation Phase I Test Phase II Test
Hospital Visits 109,168 68,785 14,719 14,702 10,962 Discharge Summaries 109,168 68,785 14,719 14,702 10,962 Radiology Reports 409,359 259,304 54,650 54,797 40,608 ED Stays & Chief Complaints 109,403 68,936 14,751 14,731 10,985 ED Diagnoses 218,376 138,112 29,086 29,414 21,764

heading: 4.3 Results

heading: 4.3.1 Automatic Evaluation
text: Automatic scoring of the submissions took place on Codabench’s platform using queues connected to independent compute workers hosted on GCP. The final leaderboard on the Phase II Test set is available in Table 5.
A baseline performance was available for participants to benchmark their submissions. The baseline outputs were generated by a LLaMA-2-7B model finetuned on radiology reports from MIMIC-III (Johnson et al., 2016). While the system exhibited some clinical domain knowledge, it struggled due to the diverse formatting of discharge summaries, which greatly differed from that of the radiology reports in the training set. All submissions exceeded the baseline performance.

heading: 4.3.2 Clinician Evaluation
text: Overall clinician review scores are available in Table 6, and the specific rankings for the BHC and Discharge Instructions sections are shown in Tables 7 and 8, respectively (mean clinician scores are provided, along with their constituent scores in brackets). Interestingly, the rankings for the overall clinician review exactly reflected that of the automatic evaluation using the reported metrics.
Figure 4.3.2 illustrates the interobserver agreement between pairwise clinicians based on the Cohen’s Kappa statistic calculated for common submissions reviewed. As not all clinicians reviewed the same subset of submissions, a statistic could not be calculated for all reviewers (i.e., reviewer #5 and #6 did not have any submissions in common). There was rather poor agreement between most clinicians, likely due to subjective aspects of the evaluation and varying clinician preference during the holistic comparison.
However, the Fleiss Kappa value indicated that the reviews for the top 6 best-scoring submissions, where each submission was reviewed by 3 individual clinicians, exhibited substantial to almost perfect agreement (Table 6). Moderate reliability was also observed for the review methodology, as inferred from the presented range of ICC values.

heading: 4.3.3 Readability of Discharge Instructions
text: As the Discharge Instructions section is intended for patients who many not have medical training and knowledge of clinical acronyms, we decided to skip the clinician review and opted for an evaluation using com-
mon readability scores: the Flesch Reading Ease score and the Flesch–Kincaid Grade Level (Friedman and Hoffman-Goetz, 2006).
The writing of patient-targeting notes at an appropriate readability level is crucial as it directly relates to patient comprehension, engagement, and adherence to treatment plans post-discharge. Several healthcare institutes have placed recommendations on the readability of patient-facing material. Specifically, the National Institutes of Health (NIH) and American Medical Association (AMA) encourage a reading grade level of not higher than sixth-grade, while the Centers for Disease Control and Prevention (CDC) suggests a reading grade level of lower than eighth-grade (Johnston et al., 2018; Cotugna et al., 2005; McCray, 2005; Burns et al., 2022).
A summary of the average readability metrics for the generated Discharge Instructions section is shown in Table 8. The readability of most submissions hovered around a reading grade level of seventh-grade, with the exception of one team at around the ninth-grade. The reference sections had a Flesch Reading Ease score of 61.81 (± 11.92) and a Flesch–Kincaid Grade Level of 8.16 (± 2.12). As such, all evaluated systems were able to reasonably re-create the readability of the reference sections, with several able to generate Discharge Instructions that are more understandable and in-line with established guidelines.

heading: 4.4 Descriptions of Top Systems
text: A total of 12 system papers were received (Damm et al., 2024; Socrates et al., 2024; Wu et al., 2024; Lyu et al., 2024; He et al., 2024; Koontz et al., 2024; Guo et al., 2024; Liu et al., 2024; Frayling et al., 2024; Wendelken et al., 2024; Tang et al., 2024; Naskar et al., 2024). The top 6 best-scoring systems are detailed in this subsection.

heading: 4.4.1 WisPerMed
text: WisPerMed (Damm et al., 2024) investigated Dynamic Expert Selection (DES) consisting of a collection of LLMs fine-tuned and prompted for the task. They demonstrated that a DES system that chooses texts based on a specific length criteria performed the best on the given dataset. Thus, their objective with this strategy was to initially rank LLMs based on their archived overall scores. Subsequently, for each discharge summary, the generated sections (BHC & Discharge Instructions) from the best model that had a word count within the range of 100 to 180 words was selected. If no model generated a block of text with a word count within this range, the text with the minimum word count greater than 70 words was selected. In cases where no piece of text met these criteria (i.e., shorter than 70 words), the text from the highest-ranked model was chosen. This approach emerged from the finding that longer pieces of medical text often led to hallucinations or repetitiveness.

heading: 4.4.2 HarmonAI Lab at Yale
text: The pipeline for HarmonAI Lab at Yale (Socrates et al., 2024) consisted of two BioBART-Large models.
The one generating BHC sections was trained on all the preceding text prior to the BHC, while the Discharge Instructions model was trained on the BHC. The BHC model had an increased training dataset size due to shuffling and recombining the provided datasets. Default hyperparameter settings were largely used for training, with the exception of a lower learning rate. Models were trained for 2 epochs. For generation, a 4-beam search and limited repeats with an n-gram size of 3 was employed. The minimum output length was set to 200 tokens based on the word count summary statistics and, and the maximum output token length was restricted to 1024 tokens due to the model specifications.

heading: 4.4.3 aehrc
text: aehrc (Liu et al., 2024) used the content in the discharge summary note prior to the target sections as input context for both training and inference. To better handle the distinctions between the two sections, the team trained two separate models to generate the BHC and the Discharge Instructions. Their best model was based on PRIMERA, which is an encoder-decoder language model that is capable of handling extended input contexts and generating longer outputs. This model offered a slight edge over fine-tuning popular decoder-based LLMs at the 7/8B parameter-level with LoRA, and was also significantly faster at inference. Beam search with a size of 4 was used for decoding.

heading: 4.4.4 EPFL-MAKE
text: EPFL-MAKE (Wu et al., 2024) mainly focused on the full-text available in the dataset as they believed that most of the useful information is hidden within. The text was used as an input into their system, which first extracted all sections that contained clinically useful information. The system then combined them into a new input. Some sections may have been removed if the new input was deemed too lengthy. The pre-processed input was then put into the medical LLM Meditron-7B, which is currently one of the top open-source medically pre-trained LLMs at the 7B level, to generate the BHC and Discharge Instructions sections.

heading: 4.4.5 UF-HOBI
text: In their system, UF-HOBI (Lyu et al., 2024) employed two clinical LLMs that they have developed in their previous works, including an encoder-based model GatorTron (Yang et al., 2022) and a decoder-based model GatorTronGPT (Peng et al., 2023). The team adopted GatorTron to extract clinical concepts from the discharge summary notes, and utilized GatorTronGPT to generate the BHC and Discharge Instructions sections. GatorTron, which was fine-tuned on the 2010 i2b2 Challenge Named Entity Recognition (NER) dataset, was used to extract three categories of concepts (“TEST”, “PROBLEM”, and “TREATMENT”) from the discharge summary and radiology reports for each visit. The extracted concepts were then used to form the generation model input. Two GatorTronGPT models were then trained using the P-tuning strategy for the generation of the two respective target sections. The model inputs were thus the concepts extracted from the various other sections.

heading: 4.4.6 de ehren
text: de ehren utilized Meerkat-7B-v1.0, a compact, instruction-tuned medical AI system renowned for its advanced medical reasoning capabilities. Meerkat excelled in various medical Question Answering (QA) benchmarks, notably achieving a score of 74.3 on MedQA. To further scrutinize its performance in longform text generation and summarization tasks within the clinical domain, the team selectively extracted key sections from discharge summaries to fine-tune the model with regards to the model’s attention window size.

heading: 4.5 Limitations & Challenges
text: A primary concern was the risk of data leakage due to the release of the test sets with ground truth sections. To
mitigate this, two test sets were released in two phases (one released at the start and one released much closer to the submission deadline), and the final evaluation was conducted on a hidden subset of 250 samples selected from the test datasets of the respective phases. This approach aimed to discourage participants from using the ground truth for model inference, or from optimizing systems for the tasks metrics throughout the entire duration of the competition. However, this method ultimately relies on the adherence of the participants to task guidelines.
The task also faced the challenge of dealing with inconsistently formatted free-text where ground truth generation targets are embedded within. The nature of clinical free-text can vary greatly, making it difficult to standardize inputs.
Furthermore, certain sections of the discharge summary appearing after the generation targets may not be reasonably available to the clinician at the time of discharge and the writing of the discharge summary. This presents a dilemma, as using such information would not accurately reflect the clinician’s workflow. Although teams were reminded to justify any decisions made regarding the use of discharge summary sections, it was challenging to moderate this aspect.
Another limitation was the need to select discharge summaries of a reasonable length to make clinician review feasible. This selection process may introduce a bias, as longer or more complex summaries that could benefit from automated generation might be excluded. There was also plausible comparison bias during clinician review as clinicians were asked to review submissions that could have varied greatly in quality. However, we aimed to reduce this by randomizing the order in which submissions were presented to the clinicians.

heading: 5 Conclusion
text: As seen from the scores of the participating models for both tasks, there is great complexity in generating coherent, accurate, and clinically relevant free-text reports. Several factors contribute to this, including the inherent variability and nuance of natural language used in clinical settings.
It may be worthwhile to consider alternative approaches for fully automated report generation, such as by pre-processing reports into structured formats prior to AI generation. By breaking down the report generation process into more manageable tasks, generation systems may be able to achieve higher accuracy and coherence in their outputs (Lederman et al., 2022). However, the standardization of formatting for these reports poses a significant challenge due to the diversity of writing styles and training among clinicians.
A previous study also explored the feasibility of generating hospital discharge summaries by tracing the source origin of medical expressions that make up the report (Ando et al., 2022). Interestingly, the analysis found that a significant portion of the discharge summary originates from external sources rather than inpatient records, such as past clinical records, referral notes, and the expertise of the writing clinician. This suggests that an end-to-end generation pipeline would depend on advanced data retrieval and may ultimately require some form of manual clinician oversight.
Ultimately, we hope that this challenge will bolster the efforts of the biomedical natural language processing community in developing effective solutions for clinical text generation. We believe this task could form a solid foundation for future work on generating entire radiology reports or discharge summaries, which would help significantly reduce the time clinicians spend on administrative tasks and improve patient care quality.

heading: Future Work
text: We believe this task could form a solid foundation for future work on generating entire radiology reports or discharge summaries, which would help significantly reduce the time clinicians spend on administrative tasks and improve patient care quality