IgnitionInnovators at "Discharge Me!": Chain-of-Thought Instruction Finetuning Large Language Models for Discharge Summaries (2024)

Abstract: This paper presents our proposed approach to the Discharge Me! shared task, collocated with the 23th Workshop on Biomedical Natural Language Processing (BioNLP). In this work, we develop an LLM-based framework for solving the Discharge Summary Documentation (DSD) task, i.e., generating the two critical target sections ‘Brief Hospital Course’ and ‘Discharge Instructions’ in the discharge summary. By streamlining the recent instruction-finetuning process on LLMs, we explore several prompting strategies for optimally adapting LLMs to specific generation task of DSD. Experimental results show that providing a clear output structure, complimented by a set of comprehensive Chain-of-Thoughts (CoT) questions, effectively improves the model’s reasoning capability, and thereby, enhancing the structural correctness and faithfulness of clinical information in the generated text. Source code is available at: https://anonymous.4open.science/r/

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 731–739 August 16, 2024. ©2024 Association for Computational Linguistics
731
1 Introduction
Discharge summaries encapsulate key details of a patient’s hospitalization, from admission to discharge. These documents, however, can contain excessive amount of medical notes, making it difficult for subsequent caregivers or patients to quickly understand essential past medical information. Brief Hospital Course and Discharge Instructions then become two critical sections in discharge summaries to address this issue. The former outlines critical hospital events for healthcare providers, while the latter offers post-discharge care instructions to patients and their caregivers. The Discharge Me! shared task 1 (Xu et al., 2024) at the BioNLP Workshop, known as Discharge Summary Documentation (DSD), focuses on efficiently generating these two critical sections, a task that is both challenging and time-consuming for clinicians.
1https://www.codabench.org/competitions/2008/
In this paper, we introduce a novel LLM-based framework, namely Discharge-LLM, for the DSD task (Xu et al., 2024). Discharge-LLM employs modern prompting strategies (e.g., Chainof-Thought (CoT)) into instruction-finetuning a Mistral Large Language Model (LLM), which enhances structural correctness and faithfulness of clinical information to generate the Brief Hospital Course and Discharge Instructions sections of discharge summaries.
2 Related Work
In recent years, Large Language Models (LLMs) like GPT-2 or GPT-3 have excelled in NLP tasks such as language generation, question answering, due to their vast number of paramters and extensive training on diverse datasets. These models can be adapted to new domains and tasks through methods like prompting, which uses natural language instructions (Liu et al., 2023) or few-shot examples (Lampinen et al., 2022). However, considering DSD problem, the length and excessive information in discharge summaries hinders their use as examples for few-shot prompting. Alternatively, parameter-efficient fine-tuning, which freezes an LLM weights and inserts a small number of tunable parameters (Lin et al., 2020), has proven effective in specialized clinical tasks like radiology report generation (Van Veen et al., 2023).
From the clinical summarization perspective, research towards th DSD task was very limited. But there has been a growing focus on many clinical text generation tasks, encompassing radiology reports (Ben Abacha et al., 2021), clinical notes (Grambow et al., 2022), and summary of diagnoses and patient problems (Gao et al., 2022).
3 Problem description
A discharge summary can contain several free-text sections and medical notes compiled from EHR.
The task is to generate the Brief Hospital Course (BHC) and Discharge Instructions (DI) sections, leveraging readily available data in other sections of discharge summaries and additional information about a patient’s admission (e.g.,radiology, stays) from the dataset. Brief Hospital Course outlines critical hospital events for healthcare providers, while Discharge Instructions offers post-discharge care instructions to patients and their caregivers.
4 Methodology
We propose the Discharge-LLM framework, which adapt LLM to the each generation task of DSD, illustrated in Figure 1. Discharge-LLM applies three steps, namely Section Extraction, Radiology Report Selection and Target Section Generation to generate the two critical target sections given discharge summary and radiology report information of a patient’s hospital visit. Note that we utilize the generated BHC for the subsequent generation of DI. Table 4 and 5 (Appendix A) show the output of the two target sections generated by our framework.
4.1 Section Extraction To generate the two target sections BHC and DI, the most straightforward approach is to leverage the other readily available free-text sections in the discharge summary as the input for the generation stage. But using them all for one-stage generation is overwhelming and prone to hallucination
because some sections are irrelevant or contain thousand words of nonessential information, making key aspects of the patient’s record often be omitted. We thereby design sets of heuristics (e.g., regular expressions) to selectively extract clinical notes information from 13 relevant sections of the discharge summaries (e.g, History of Present Illness, Pertinent Results, . . . ), with definition of each section described in Figure 1. We report data distribution of these sections in Table 6 (Appendix B)
4.2 Radiology Report Selection
Through exploring the format of different sections of the discharge summary, we notice a great complications in the structure and content of the Pertinent Results section, likely due to note bloat and information overload. This section, intended to highlight key findings of radiologist to the patient’s treatment, is often cluttered with excessive laboratory and imaging data (e.g., blood tests, CT scans). These extraneous details can lead to challenges such as hallucination and high resource demands in generative tasks. Consequently, we explored using radiology reports as a viable alternative. These reports, often duplicated partially or entirely in the Pertinent Results section, succinctly convey diagnoses corresponding to specific lab results. We selected radiology reports with similar Impressions to those in the Pertinent Results and used these as a substitute, streamlining the content effectively.
4.3 Target Section Generation
In this framework, we performed instructionfinetuning on LLM to adapt the model to DSD. For computational feasiblity, we employed Low-Rank Adaptation (LoRA) (Hu et al., 2021), a parameterefficient fine-tuning method that adds a small number of trainable parameters to the model while freezing the model’s original weight, resulting in standalone adapters. The adapters, specifically finetuned for each generation task in DSD, adjust important weight of LLMs to capture and generate clinical information in the corresponding form.
Prompting Strategies Following OpenAI’s prompt engineering guidelines 2, we structured our prompts into five parts, detailed in Table 7 (Appendix D): 1) Context of the discharge summary input to be summarized 2) Definition of the generation task and the specific section for documenting the discharge summary 3) Structure of the expected output of the generating section, infused with 4) Set of Chain-of-Thought (CoT) questions expected to be answered by the LLMs to capture and generate the information in each subsection of the output. Of those, our primary strategy is Part 5, which involved curating effective and generalizable CoT questions based on analysis of numerous samples. This manual effort helped in designing templates and questions that effectively guide the LLMs to focus on critical information amidst the extensive data and noise in the discharge summaries. We analyzed the medical questionnaire essential for each section, based on hundreds of samples, in Appendix C , which underpins our CoT questions and prompt design.
5 Experiments
5.1 Baseline and Implementation Details
To showcase the utility of prompt designing for adaptation to the DSD task, we developed three baselines, corresponding to three prompt variants for instruction-finetuning LLMs. Discharge_LLMBase was fine-tuned with no instruction, but only the discharge summaries as input and the respective target section as output. Discharge_LLMContext was fine-tuned with additional natural langauge instructions as prefix to the discharge summary to provide the context and definition of the task’s input/output. Fi-
2https://platform.openai.com/docs/guides/ prompt-engineering
nally, Discharge_LLMCoT was fine-tuned using prompts outlining the structure of the respective generating target section. Along the structure, we embed some CoT questions to elicit LLMs to generate output aligned with the questions.
We choose Mistral 3 (Jiang et al., 2023) as our LM. The LLM was fine-tuned on a NVIDIA RTX 4090 GPU, and took 10 hours for fine-tuning each generation task. The following hyperparameters were used: 1 sample per device, a LoRA rank and alpha of 128 and 64 for parameter-efficient finetuning, a learning rate of 2e10− 4. We keep other hyperparameters to their default values.
Metrics We followed the organizers to measure textual similarity and factual correctness of the generated text based on several metrics, including BLEU-4 (Papineni et al., 2002), ROUGE (Lin, 2004), BERTScore (Zhang et al., 2019), Meteor (Banerjee and Lavie, 2005), AlignScore (Zha et al., 2023), and MEDCON (Yim et al., 2023).
Dataset The dataset for this task was sourced from the MIMIC-IV (Johnson et al., 2023) dataset, including 109,168 emergency department (ED) admissions and were split into a training (68,785), a validation (14,719), a phase I testing (14,702), and a phase II testing (10,962) subsets.
Data Preprocessing To address the variation in discharge summary length, we select data within the interquartile range (Q1-Q3) for training and validation. We further ensure consistency by selecting only samples with discharge summaries containing all 13 common sections and their target sections follow the most common format, as outlined in Figure 1. Overall, 11.1k and 8.7k samples were selected for training of BHC and DI generation, respectively. For experiment, due to runtime and computational limitations, we sample 250 hidden entries from each phase’s testing data, totaling 500 samples for evaluation of each generation task.
5.2 Results
Table 1 presents the performance of models finetuned by different prompt variants. Overall, in both generation tasks, natural language instructions plays a critical role in guiding the LLM with comprehensive knowledge to understand the task. Providing well-described context of the generation task already helps the model achieves up to 14% of
3https://mistral.ai/
performance gain across the metrics and tasks. Further, infusing CoT questions into the instructions effectively elicit LLM to think better, providing an additional 9% performance increase. Notably, a reasonable improvement on MEDCON score also indicates better accuracy and consistency of clinical concepts in the generated text.
5.3 Shared Task’s Evaluation Results Table 2 summarizes our framework’s overall performance on the phase 2 test set of the shared task, alongside the best-ranked system 4. We notice there
4https://www.codabench.org/competitions/2008/
is still a gap between our Discharge_LLMCoT framework and the best ranked system, of which the Overall score is 0.332. This performance dip is common across submissions, likely due to prevalent data quality issues in the Discharge Summary Documentation (DSD) task. DSD, a real-world summarization challenge, involves processing actual discharge summary information with significant variability in formatting and length. Figure 2 shows word distribution variances in the target sections. It is noticeable our models are trying to set a common length for the target sections, and are struggling to converge to the wide range of lengths of the reference text, highlighted by Table 3. We note slight performance variation of Discharge_LLMCoT in terms of Meteor, AlignScore and MEDCON, in between Table 1 and 2. A possible reason for such variation may be that hidden test data in the shared task has distributions significantly deviate from the publicly released data for model development. Furthermore, due to computational constraints and a focus on high-quality data, we utilized only a subset of the available training data. With more comprehensive training, we anticipate improved model convergence.
6 Conclusion
In this paper, we present a LLM-based framework for Discharge Summary Documentation that adopts several prompting strategies into instructionfinetuning an LLM, which enhances structural correctness and faithfulness of clinical information in generated target sections. Using small and opensource LLMs, our work also shows the feasiblity of developing and deploying future lightweight NLP systems locally for confidential clinical tasks.
References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
Asma Ben Abacha, Yassine Mrabet, Yuhao Zhang, Chaitanya Shivade, Curtis Langlotz, and Dina DemnerFushman. 2021. Overview of the MEDIQA 2021 shared task on summarization in the medical domain. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 74–85, Online. Association for Computational Linguistics.
Yanjun Gao, Dmitriy Dligach, Timothy Miller, Dongfang Xu, Matthew MM Churpek, and Majid Afshar. 2022. Summarizing patients’ problems from hospital progress notes using pre-trained sequence-tosequence models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2979–2991.
Colin Grambow, Longxiang Zhang, and Thomas Schaaf. 2022. In-domain pre-training improves clinical note generation from doctor-patient conversations. In Proceedings of the First Workshop on Natural Language Generation in Healthcare, pages 9–22, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics.
Edward Hu, Yelong Shen, Phil Wallis, Zeyuan AllenZhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.
Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. 2023. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1.
Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. 2022. Can language models learn from explanations in context? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 537–563, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2020. Exploring versatile generative language model via parameter-efficient transfer learning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 441–459, Online. Association for Computational Linguistics.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9).
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Dave Van Veen, Cara Van Uden, Maayane Attias, Anuj Pareek, Christian Bluethgen, Malgorzata Polacin, Wah Chiu, Jean-Benoit Delbrouck, Juan Zambrano Chaves, Curtis Langlotz, Akshay Chaudhari, and John Pauly. 2023. RadAdapt: Radiology report summarization via lightweight domain adaptation of large language models. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 449–460, Toronto, Canada. Association for Computational Linguistics.
Justin Xu, Zhihong Chen, Andrew Johnston, Louis Blankemeier, Maya Varma, Jason Hom, William J. Collins, Ankit Modi, Robert Lloyd, Benjamin Hopkins, Curtis Langlotz, and Jean-Benoit Delbrouck. 2024. Overview of the first shared task on clinical text generation: Rrg24 and “discharge me!”. In The 23rd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, Bangkok, Thailand. Association for Computational Linguistics.
Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, and Meliha Yetisgen. 2023. Acibench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Scientific Data, 10(1):586.
Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348, Toronto, Canada. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
A Generated Output of Brief Hospital Course and Discharge Instructions
This section presents details of Table 4 and 5, which show the output of the two “Brief Hospital Course” and “Discharge Instructions” target sections generated by our framework, taken from medical information of patient with hadm_id = 21720538 from the phase 2’s test set.
B Data Distribution of Discharge Summary Sections
Table 6 presents the percentage distribution of common sections in the discharge summary text of the training, validation and testing subsets.
C Questionnaire for Discharge Summary Documentation
C.1 Brief Hospital Course • Patient Background and Presenting Com-
plaint: "What is the patient’s background including pre-existing medical conditions, and what symptoms or events led to their current hospital admission?"
• Key Diagnoses and Evaluations: "What are the key diagnoses identified during the hospital stay? For each, how was the diagnosis reached, including any significant tests or evaluations conducted?"
• Treatment and Management Strategies: "What were the main treatment strategies employed for the patient’s conditions during their stay? Include medications adjusted, procedures performed, and any therapeutic interventions."
• Complications and Additional Diagnoses: "Were there any complications or additional diagnoses during the hospital stay? How were these addressed and managed?"
• Progress and Monitoring: "How did the patient’s condition progress throughout the hospital stay, including any monitoring of symptoms, response to treatments, and adjustments made to the treatment plan?"
• Support and Consultation Services: "Which specialist services or support consultations were involved in the patient’s care? How did these consultations impact the patient’s treatment plan and recovery?"
• Discharge Planning and Instructions: "What were the conditions and considerations for the patient’s discharge? Include the discharge medications, any changes from previous medication regimens, and follow-up care or lifestyle recommendations."
• Follow-Up and Post-Discharge Care: "What are the specific follow-up care instructions and any scheduled tests or consultations? Highlight the importance of follow-up for managing ongoing conditions or monitoring recovery."
C.2 Discharge Instructions
• Initial Assessment and Diagnosis: What led to the patient’s admission to the hospital, and what were the initial symptoms? Based on the patient’s symptoms, what diagnoses were considered and which was confirmed?
• Treatment and Hospital Stay: What treatments were provided to address the patient’s symptoms or condition during the hospital stay? Were any surgeries recommended or performed? If a surgery was recommended but not performed, what were the reasons? What were the outcomes of the treatments or interventions provided?
• Patient’s Decisions and Care Preferences: Did the patient make any specific requests regarding their care, such as refusing a treatment or requesting a transfer? How were these handled? How did the patient’s decisions affect their treatment plan and discharge process?
• Comprehensive Post-Discharge Instructions: What are the general care instructions for the patient after discharge, including diet, activity level, and medication management? Are there any specific symptoms or signs that the patient should monitor for which would require immediate medical attention? How should the patient manage their regular home medications in addition to any new medications prescribed at discharge?
• Activity and Lifestyle Recommendations: What specific activity restrictions or recommendations are given to ensure a smooth recovery? (e.g., weight lifting limits, mobility advice) Are there any restrictions on driving or
operating machinery, especially if the patient is taking new or continued pain medication?
• Follow-up Care and Monitoring: What followup appointments or tests are recommended for the patient? With whom should these ap-
pointments be made? How should the patient approach symptom management, especially if they experience pain, dehydration, or other concerning symptoms?
• Communication with Healthcare Providers:
Under what circumstances should the patient immediately contact their healthcare provider or seek emergency care? What is the recommended way for the patient to communicate with their healthcare team (e.g., phone call, hospital return)?
• Encouragement and Support: How can we encourage the patient to adhere to their discharge instructions and reassure them about their recovery process? What resources or support systems can we recommend to the patient for additional help or information post-discharge?
D Prompts for Discharge Summary Documentation
We present the prompts for generation of the two critical “Brief Hospital Course” and “Discharge Instructions” target sections in Table 7

heading: Future Work
text: Using small and opensource LLMs, our work also shows the feasiblity of developing and deploying future lightweight NLP systems locally for confidential clinical tasks. References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. Asma Ben Abacha, Yassine Mrabet, Yuhao Zhang, Chaitanya Shivade, Curtis Langlotz, and Dina DemnerFushman. 2021. Overview of the MEDIQA 2021 shared task on summarization in the medical domain. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 74–85, Online. Association for Computational Linguistics. Yanjun Gao, Dmitriy Dligach, Timothy Miller, Dongfang Xu, Matthew MM Churpek, and Majid Afshar. 2022. Summarizing patients’ problems from hospital progress notes using pre-trained sequence-tosequence models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2979–2991. Colin Grambow, Longxiang Zhang, and Thomas Schaaf. 2022. In-domain pre-training improves clinical note generation from doctor-patient conversations. In Proceedings of the First Workshop on Natural Language Generation in Healthcare, pages 9–22, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics. Edward Hu, Yelong Shen, Phil Wallis, Zeyuan AllenZhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106. 09685. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310. 06825. Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. 2023. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1. Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. 2022. Can language models learn from explanations in context? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 537–563, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2020. Exploring versatile generative language model via parameter-efficient transfer learning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 441–459, Online. Association for Computational Linguistics. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv. , 55(9). Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. Dave Van Veen, Cara Van Uden, Maayane Attias, Anuj Pareek, Christian Bluethgen, Malgorzata Polacin, Wah Chiu, Jean-Benoit Delbrouck, Juan Zambrano Chaves, Curtis Langlotz, Akshay Chaudhari, and John Pauly. 2023. RadAdapt: Radiology report summarization via lightweight domain adaptation of large language models. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 449–460, Toronto, Canada. Association for Computational Linguistics. Justin Xu, Zhihong Chen, Andrew Johnston, Louis Blankemeier, Maya Varma, Jason Hom, William J. Collins, Ankit Modi, Robert Lloyd, Benjamin Hopkins, Curtis Langlotz, and Jean-Benoit Delbrouck. 2024. Overview of the first shared task on clinical text generation: Rrg24 and “discharge me!”. In The 23rd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, Bangkok, Thailand. Association for Computational Linguistics. Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, and Meliha Yetisgen. 2023. Acibench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Scientific Data, 10(1):586. Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348, Toronto, Canada. Association for Computational Linguistics. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904. 09675. A Generated Output of Brief Hospital Course and Discharge Instructions
This section presents details of Table 4 and 5, which show the output of the two “Brief Hospital Course” and “Discharge Instructions” target sections generated by our framework, taken from medical information of patient with hadm_id = 21720538 from the phase 2’s test set. B Data Distribution of Discharge Summary Sections
Table 6 presents the percentage distribution of common sections in the discharge summary text of the training, validation and testing subsets. C Questionnaire for Discharge Summary Documentation
C. 1 Brief Hospital Course • Patient Background and Presenting Com-
plaint: "What is the patient’s background including pre-existing medical conditions, and what symptoms or events led to their current hospital admission?"
• Key Diagnoses and Evaluations: "What are the key diagnoses identified during the hospital stay? For each, how was the diagnosis reached, including any significant tests or evaluations conducted?"
• Treatment and Management Strategies: "What were the main treatment strategies employed for the patient’s conditions during their stay? Include medications adjusted, procedures performed, and any therapeutic interventions. "
• Complications and Additional Diagnoses: "Were there any complications or additional diagnoses during the hospital stay? How were these addressed and managed?"
• Progress and Monitoring: "How did the patient’s condition progress throughout the hospital stay, including any monitoring of symptoms, response to treatments, and adjustments made to the treatment plan?"
• Support and Consultation Services: "Which specialist services or support consultations were involved in the patient’s care? How did these consultations impact the patient’s treatment plan and recovery?"
• Discharge Planning and Instructions: "What were the conditions and considerations for the patient’s discharge? Include the discharge medications, any changes from previous medication regimens, and follow-up care or lifestyle recommendations. "
• Follow-Up and Post-Discharge Care: "What are the specific follow-up care instructions and any scheduled tests or consultations? Highlight the importance of follow-up for managing ongoing conditions or monitoring recovery. "
C. 2 Discharge Instructions
• Initial Assessment and Diagnosis: What led to the patient’s admission to the hospital, and what were the initial symptoms? Based on the patient’s symptoms, what diagnoses were considered and which was confirmed?
• Treatment and Hospital Stay: What treatments were provided to address the patient’s symptoms or condition during the hospital stay? Were any surgeries recommended or performed? If a surgery was recommended but not performed, what were the reasons? What were the outcomes of the treatments or interventions provided?
• Patient’s Decisions and Care Preferences: Did the patient make any specific requests regarding their care, such as refusing a treatment or requesting a transfer? How were these handled? How did the patient’s decisions affect their treatment plan and discharge process?
• Comprehensive Post-Discharge Instructions: What are the general care instructions for the patient after discharge, including diet, activity level, and medication management? Are there any specific symptoms or signs that the patient should monitor for which would require immediate medical attention? How should the patient manage their regular home medications in addition to any new medications prescribed at discharge?
• Activity and Lifestyle Recommendations: What specific activity restrictions or recommendations are given to ensure a smooth recovery? (e. g. , weight lifting limits, mobility advice) Are there any restrictions on driving or
operating machinery, especially if the patient is taking new or continued pain medication?
• Follow-up Care and Monitoring: What followup appointments or tests are recommended for the patient? With whom should these ap-
pointments be made? How should the patient approach symptom management, especially if they experience pain, dehydration, or other concerning symptoms?
• Communication with Healthcare Providers:
Under what circumstances should the patient immediately contact their healthcare provider or seek emergency care? What is the recommended way for the patient to communicate with their healthcare team (e. g. , phone call, hospital return)?
• Encouragement and Support: How can we encourage the patient to adhere to their discharge instructions and reassure them about their recovery process? What resources or support systems can we recommend to the patient for additional help or information post-discharge?
D Prompts for Discharge Summary Documentation
We present the prompts for generation of the two critical “Brief Hospital Course” and “Discharge Instructions” target sections in Table 7