SPEER: Sentence-Level Planning of Long Clinical Summaries via Embedded Entity Retrieval

Authors: griffin  adams
jason  zucker
noémie  elhadad
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 16:43:23

Abstract: Clinician must write a lengthy summary each time a patient is discharged from the hospital. This task is time-consuming due to the sheer number of unique clinical concepts covered in the admission. Identifying and covering salient entities is vital for the summary to be clinically useful. We fine-tune open-source LLMs (Mistral-7B-Instruct and Zephyr-7B-$\beta$) on the task and find that they generate incomplete and unfaithful summaries. To increase entity coverage, we train a smaller, encoder-only model to predict salient entities, which are treated as content-plans to guide the LLM. To encourage the LLM to focus on specific mentions in the source notes, we propose SPEER: Sentence-level Planning via Embedded Entity Retrieval. Specifically, we mark each salient entity span with special "{{ }}" boundary tags and instruct the LLM to retrieve marked spans before generating each sentence. Sentence-level planning acts as a form of state tracking in that the model is explicitly recording the entities it uses. We fine-tune Mistral and Zephyr variants on a large-scale, diverse dataset of ~167k in-patient hospital admissions and evaluate on 3 datasets. SPEER shows gains in both coverage and faithfulness metrics over non-guided and guided baselines.

Main paper: 
 introduction: 
 Clinical professionals are experiencing physical and emotional burnout at unprecedented rates (Maslach & Leiter, 2016; National Academies of Sciences, 2019; Kroth et al., 2019) . A significant factor driving clinician burnout is the Electronic Health Record (EHR): the information overload it produces, and the documentation burden it requires (Shanafelt et al., 2016; Moy et al., 2021) . Increased time spent at the desk means less face-to-face interaction (only 27% of working hours spent with patients) and the resulting burnout has been associated with an increased risk of errors (Salvagioni et al., 2017; Panagioti et al., 2018) .Large Language Models (LLMs) have the potential to reduce the documentation burden by either directly replacing clinicians or acting as a co-pilot to reduce the time it takes clinicians to complete certain tasks (Clusmann et al., 2023; Perlis & Fihn, 2023) . To date, much of the focus of LLMs in healthcare has been on applying closed models, such as GPT-4, to well-defined granular tasks, such as closed-book question answering (Nori et al., 2023a) and single-document radiology report summarization (Sun et al., 2023) , using small publicly available benchmarks. In this paper, we adapt open-source LLMs (Zephyr and Mistral) to an onerous, longitudinal documentation task: hospital-course summarization, by training on nearly 170k patient records from a large metropolitan hospital. Hospital-course summarization (Adams et al., 2021) involves recounting in a narrative form the events occurred during the patient stay, and why they happened. Clinicians write hospital-course summaries each time a patient is discharged from a hospital in the form of the mandatory "Brief Hospital Course" section of the Discharge Summary.Among others, the task is cognitively difficult and time consuming for two principle reasons (Adams et al., 2021) . The first relates to the identification of salience. When a patient is admitted to a hospital, every test, diagnosis, procedure, and medication, consequential or not, is typically entered into at least one clinical note. Due to frequent copy-and-paste (Hirschtick, 2006; Adams et al., 2020) , moreover, these are typically entered multiple times across notes. This leads to a severe case of note bloat (Shoolin et al., 2013) in which finding relevant information is commensurate to finding a needle in a haystack. The second major challenge involves coverage of salient information. Despite the high presence of redundant and irrelevant information, there is still a great deal of relevant information, especially for lengthy admissions, which can last up to a month or longer. In these cases, it is very easy to omit critical information, which could potentially render the summary clinically harmful.Treating the notes from admission to discharge as inputs, we construct a large scale finetuning dataset from the full patient records for all inpatient hospital admissions at Columbia University Irving Medical Center (CUIMC) from 2020-2023. Clinician-authored Brief Hospital Course summaries are extracted from the corresponding discharge summary for the admission and serve as ground-truth references. We perform full parameter fine-tuning on Mistral and Zephyr 7B parameter models and demonstrate that they frequently hallucinate and fail to cover salient entities.To better ground the LLMs on salient source entities, we train a specialized, smaller classification model to perform explicit content selection. Given the entity-dense nature of task, we select groups of synonymous entities-medical concepts-as the appropriate granularity for content selection. To help LLMs to adhere to these content plans, we propose SPEER: Sentence-Level Planning via Embedded Entity Retrieval. Specifically, we mark each salient entity span with special {{ }} boundary tags and instruct the LLM to retrieve marked spans before generating each sentence. Sentence-level planning acts as a form of state tracking in that the model explicitly records the entities it uses.Our primary contributions are to:• Fine-tune state-of-the-art open source LLMs (Mistral-7B-Instruct and Zephyr-7B-β) on a large-scale dataset for long-form clinical summarization, and test on three diverse datasets from different EHRs.• Demonstrate that content selection should be thought of as its own classification task, even in the world of LLMs. Dedicated content selection, performed by a small encoder-only classifier, outperforms implicit content selection from auto-regressive LLM decoding.• Introduce an easy-to-implement method-SPEER-which improves the coverage of salience entities and faithfulness over both non-guided and guided LLM baselines. 
related work: 
 LLM Summarization. Recent work has found that API-based closed source models, such as Claude, GPT-3, and GPT-4, can generate high quality summaries of news articles in the zero-shot setting (Goyal et al., 2022) . Humans prefer LLM-generated summaries from GPT-3 over summaries generated from the previous generation of smaller, fine-tuned models (e.g., BART, PEGASUS) (Zhang et al., 2023) . Human evaluation is critical to revealing the superiority of LLM-generated summaries given the limitations of reference-based metrics.In fact, Zhang et al. (2023) find that annotators judge LLM-generated summaries on par with expert-level summaries carefully crafted by freelance writers. By iteratively fusing new entities into an existing summary draft, Chain-of-Density (CoD) (Adams et al., 2023b) enables LLMs, e.g., GPT-4, to generate entity-dense summaries, which are often favored by human annotators over earlier, less dense, drafts. Explicit planning can also be performed with Summary Chain-of-Thought (SumCoT), a technique that guides LLMs to generate summaries by focusing on core news elements in a step-by-step manner, leading to more coherent and comprehensive summaries (Wang et al., 2023) . On the evaluation side, Chen et al. (2023) demonstrate that zero-shot prompted LLMs can outperform existing specialized classifiers on factuality detection.Guided Summarization. Abstractive summarization requires three sequential tasks: content selection (extraction), content planning (organization), and surface realization (abstraction). With simple auto-regressive generation, the first two steps are generally performed implicitly with the last. Yet, prior work suggests that making content selection an explicit step, which is handled by a separate, dedicated model, can outperform the all-in-one approach (Sharma et al., 2019) . For instance, an extractive model can be used to enhance the performance of an abstractive model by treating the extract as an auxiliary input with its own encoder (GSum, (Dou et al., 2021) ). Other work simply prepends salient content to the source text as a form of control (CTRLsum (He et al., 2022) ). Relevant to our work, CTRLsum explores using entities as a form of control. The FROST model separates content selection and planning from realization with a single model, by having the model first generate an entity-based plan before generating the full abstractive summary (Narayan et al., 2021) . SPEER, on the other hand, interleaves planning and realization and relies on a separately trained classifier for content selection. Adams et al. (2023a) demonstrate that offloading content selection can be used to directly control the diversity of downstream summaries.Their PGA model can be used to enhance the performance of both small encoder-decoder (BART and PEGASUS) and large decoder-only (GPT-3.5) abstractors. Hospital-Course Summarization. The multi-document summarization task of synthesizing the course of events during a patient's admission to a hospital is an area of active research (Adams et al., 2021) . Using a publicly available source of data (MIMIC-III), Adams et al. (2022) tackle faithfulness by re-writing, or revising, reference summaries before training models on synthetic, grounded hospital-course reference summaries. Similarly to our work, Searle et al. (2023) guide an abstractive model with clinical concepts to improve ROUGE scores of hospital-course summaries. They guide the model with all source concepts, while we perform content selection, or filtering, first. We embed the selected content back onto the source notes, whereas they guide BART with a separate encoder stream (as in GSum (Dou et al., 2021) ). Preliminary studies of LLMs into biomedical applications have largely focused on well-defined medical reasoning tasks (Nori et al., 2023b; a) , such as providing a differential diagnosis based on a patient record (McDuff et al., 2023) . Those that have explored clinical summarization have largely focused on single-document tasks of a finer temporal granularity (Veen et al., 2023) , such as generating the impressions section of a radiology report (Liu et al., 2023b; a; Chuang et al., 2023; Van Veen et al., 2023 ), ICD coding (Boyle et al., 2023) , constructing a patient's problem list construction from a progress note (Veen et al., 2023) , and generating a note from a doctor-patient conversation (Abacha et al., 2023; Ionescu et al., 2023) . 
selecting salient entities: 
 The process of extracting entities, identifying synonym pairs, and then forming Entity Synonym Groups (ESGs) is graphically depicted in Figure 1 . In Extracting Entities. We use Stanza (Qi et al., 2020) for entity extraction. In particular, we use the clinical NER model (Zhang et al., 2021) which was trained on MIMIC-III notes (Johnson et al., 2016) for the i2b2-2010 clinical NER shared task (Uzuner et al., 2011) . The model extracts entity spans from three disjoint semantic types: PROBLEM, TEST, TREATMENT.Problems are diagnoses and symptoms, Tests cover lab tests and imaging, while Treatments span medications and procedures.Identifying Entity Synonym Pairs. Clinicians frequently rely on acronyms and shorthand when documenting, which leads to large variance in how concepts are mentioned across notes (Demner-Fushman & Elhadad, 2016; Adams et al., 2020) . We follow (Adams et al., 2023d) and use similarity in embedding space to identify synonymous clusters of entity spans. Specifically, we embed all mentions with SapBERT (Liu et al., 2021) -which is trained to align synonymous clinical concepts-and use cosine similarity to identify all synonymous pairs. We manually assign binary labels (unrelated, synonymous) to 1,000 mention-pairs. We then select the threshold (0.75) for cosine similarity classification which maximizes the F1-score overlap with human labels. Figure 1 illustrates the need for semantic over lexical matching. Acronyms (WBC → White Blood Cell) have identical meanings and no lexical overlap. This also holds true for synonyms: Cytoreductive surgery and NEOPLASM DEBULKING, which both describe the resection of a tumorous growth.For each hospital admission, we collect all entity mentions across the source notes and form a graph with one node for each unique entity mention. We assign an edge between two mentions iff they are exact-match duplicates or have a pairwise SapBERT similarity of ≥ 0.75. We then treat all fully-connected sub-graphs as Entity Synonym Groups (ESG). Entity content selection is then performed over ESGs, e.g., which represent a model a single medical concept, rather than over specific span mentions.As shown in Figure 1 , the process of forming ESGs by computing fully connected sub-graphs based on the pairwise similarity graph greatly reduces entity sparsity. Eight unique entity spans form two ESGs. The entities in the ESG all relate to the same precise topic-if not the exact same concept. For instance, leukocytosis is a condition characterized by a high white blood cell (WBC) count and thus is connected directly to WBC COUNT and, via the graph, to WBC and White Blood Cell.Defining ESG salience. For each example (hospital admission), we extract the ESG from across the source notes. Based on the embedding similarity method for synonymity, given embedding similarity scores, we consider an ESG as "salient" if ≥ 1 spans in the ESG is a synonym of ≥ 1 entity span(s) extracted from the reference summary. Only 5.7% of the source ESGs are "salient" by this definition, which underscores the difficulty of content selection for this task and dataset.Learning ESG salience. We build a hierarchical token-to-ESG encoder model to perform binary classification over ESGs. This approach is inspired by previous hierarchical extractive (2023a) . First, we demarcate each entity span with newly initialized <e> and </e> tokens. Then, we concatenate all source notes and encode tokens with a long-range encoder (LongFormer (Beltagy et al., 2020) ), which can fit up to 16,384 tokens. We construct hidden-state representations of each entity span by mean-pooling the hidden states of each word-piece associated with the span (inclusive of <e> and </e> tokens). Next, we mean-pool the hidden states of all entity spans associated with the same ESG (e.g., "Diabetes type 2", "DM II", "DM2"). Then, we add an ESG modeling layer as a newly initialized, fully-connected BERT encoder layer.To exploit the fact that frequently mentioned concepts tend to be salient, we assign each ESG to a numerical range according to inverse frequency (most mentions first). We learn an embedding for relative frequency and add it to the ESG hidden state before passing through the modeling layer. A linear classification head is added to produce a single logit for each modeled ESG representation. We compute a logistic loss over each ESG logit.  
esg-guided summarization: 
 In this section, we explore methods for ESG-guided summarization: by which summary generation is conditioned on both the source documents and a pre-selected set of ESGs. We first present a simple prompt-based baseline before presenting our proposed approach, SPEER. As in (Dou et al., 2021; Adams et al., 2023a) , We train models on oracle provided ESGs as guidance while performing inference with the set of model-predicted ESGs ( §3).Prompt Guidance. A logical approach is to convert the set of salient ESGs into a natural language prompt and instruct the model to incorporate them into its summary. To compute the oracle prompt for training, we follow the same procedure to define the labels as in §3:"Defining ESG Salience". Then, we list out the salient ESGs by semantic type: "PROBLEMS", "TREATMENTS", and "TESTS". Each line contains each unique mention of an ESG across the source notes (delimited by ";").Issues with Prompt Guidance. The above approach is simple, intuitive, and helps ground the summaries onto a set of salient entities. Yet, two issues may arise. Firstly, the model may learn to focus more on the entities themselves and not their actual usage in the source notes. The source notes are lengthy and, as such, the ratio of relevant to irrelevant content is very high. The prompt guidance, however, is precise and only includes the entities which make it into the reference summary. Relatively speaking, the model may learn to over-rely on the list of entities themselves at the expense of their usage in the source notes, which can be difficult to identify. A consequence of this would be high coverage of salient entities on the surface, yet irrelevant, inconsequential, or erroneous context supplied for these entities. To show the model that it must only use embedded entities to form its plan, the model is taught to generate the entities with their boundary tags {{ }}. As shown in Figure 2 , this can be described as R 3 : Retrieve-Realize-Repeat. R 3 aims to address both the grounding and tracking concerns. Firstly, by retrieving the bracketed entities, we are encouraging the model to attend to-or focus on-a specific usage of the salient ESG in context. Secondly, the act of explicitly generating the entities to include in the next sentence makes it easier to keep track of which entities have already been included in the summary. The output template for this proposed R 3 method of summarization is:### Entities 1: {{span}} {{span}} ### Sentence 1: <sentence 1> ### Entities 2: {{span}} {{span}} ### Sentence 2: <sentence 2>The final summary is simply the concatenation of each line which begins with ### Sentence.We use oracle ESGs during training and those predicted by the ESG classifier at inference. For planning, during training, we extract the in-order entity mentions from each reference sentence and add them as spans to the corresponding ### Entities line.Pseudo Code for SPEER. To make it more concrete, we include pseudo code (Figure 3 ) for creating the training data for ESG selection ( §3) and ESG-guided summarization, e.g. SPEER ( §4). For ESG content selection, the model inputs are the set of input notes and the set of source ESGs. The ESG classifier is a hierarchical token-to-ESG encoder which encodes each token, before merging tokens into ESGs, performing ESG-ESG attention, and outputting a binary classification logit. For summarization, decorated inputs and reference summaries are concatenated into a single prompt and the decoder loss on the inputs is ignored.The process for inference is similar yet relies on source ESGs predicted as salient by the content selector, rather than the oracle rel src esgs in Figure 3 . The output at inference is parsed to remove planning lines, e.g., those starting with "### Entities".Figure 3 : Pseudo-code to generate training data for SPEER, which includes a step for learning to select salient entities ( §3), followed by entity plan-guided summarization ( §4). 
experimental setup: 
 The experimental setup is detailed in Appendix §A. In summary, we train on a single dataset of ˜167k admissions and test on three, diverse held out sets with ˜1,000 admissions each. Two of the test sets come from the same institution as the training set, yet from different time periods and EHRs, while the third is derived from publicly available MIMIC-III notes (Johnson et al., 2016) . To ensure fairness in comparisons, each model is trained for exactly the same number of steps (˜14,000) in batches of 16 with the same learning rate of 5e -6.Evaluation Metrics. We rely on two entity-based overlap metrics: Source-Grounded Recall (SGR) for salience and the Hallucination Rate (HR) for faithfulness. To avoid penalizing models for unsupported entities in references, we separately align summary entities to entities in the sources notes, and measure overlap between source-aligned entities. Specifically, we align reference and model-generated entities to a subset of ESGs from the source notes: {ESG re f →src } and {ESG model→src }. Then, we computeSGR as SGR = |{ESG re f ⇒src }∩{ESG model⇒src }| |{ESG re f ⇒src }|. We compute the hallucination rate (HR) asHR = |{ENTITY model⇏src } |{ENTITY model }| .|{ENTITY model⇏src }| denotes the number of predicted entity mentions which do not have a corresponding source synonym. HR uses entity mentions and not ESGs (as in SGR) in order to penalize multiple hallucinations of the same synonym group. For faithfulness, we also report BERTScore-Precision (BSP) (Zhang et al., 2019) and ClinDistill (Adams et al., 2023d )-a SOTA metric for hospital-course summarization which outputs a real-value number whose mean is centered at 0. We report ROUGE-1 / 2 despite its inverse relationship with faithfulness for clinical summarization (Adams et al., 2023c) .  
results: 
 We Models which rely on entity guidance achieve higher coverage of salient entities than those that do not. As shown in We suspect that Guided may learn to over-rely on the list of entities themselves at the expense of their usage in the source notes. The appended entity guidance might be stealing attention away from the source notes themselves. SPEER is more robust to unseen EHRs. The model was trained on CUIMC: 2020-2023 data, so it is unsuprising that performance is best on a held-out set of admissions from the same date range. When switching datasets and EHRs, there is a noticeable performance drop across models, especially for MIMIC. As discussed in Adams et al. (2022) , MIMIC-III notes are highly incomplete. As such, much of the reference content is not supported by the available source notes and reference-free metrics are understandably poor. Yet, it is notable that the largest advantage (coverage and recall) for SPEER comes from MIMIC, for which the data is the noisiest and the notes come from an unseen institution. SPEER might be more robust to this "zero-shot" setting because it requires the least effort on the part of the abstractive component. The LLM only needs to locate {{ }} tags, rather than needing to implicitly perform salience modeling (Non-Guided) or to link prompted guidance back onto specific parts of the source (Guided).Table 3 demonstrates incremental improvements in faithfulness and coverage of salient entities as we transition from the baseline model (Non-Guided) to the fully loaded SPEER model. As discussed earlier, going from non-guided Non-Guided) to prompt Guided) increases coverage dramatically (SGR goes from .516 → .582 while sentence-level faithfulness (ClinDistill) declines: .570 → .554. If we replace prompt guidance with embedded guidance: Embedded, we achieve a slight decline in SGR: .582 → .574 yet a decrease in hallucinations (HR): .152 → .147 and an increase in sentence-level faithfulness: .554 → .673. Embedded is SPEER without the sentence-level planning. The input is the same (notes with embedded salient ESGs) yet the target output is the summary without planning.Adding in planning, we arrive at SPEER, which leads to an increase in coverage of salient entities: .574 → .588 for SGR and a further decrease in hallucinations: .147 → .122.We do note that ROUGE scores decline: ROUGE-1 from 50.5 → 48.3 but we believe that this is a necessary side effect of sentence-level planning, which encourages the model to stick to the entities in the source text and not hallucinate plausible, yet unsupported, content. Qualitatively, planning seems to cause a reduction in the number of sentences with no entities which occur in many reference summaries, yet do not contain important details. A paraphrased example is: "Patient verbalized understanding of instructions and plans to follow up with his primary doctor in two weeks." These types of sentences often achieve high ROUGE scores as they are true for many patients, but more often than not, are never stated in the source notes and cannot be assumed to be true. Including common, yet unsupported, content can artificially boost ROUGE at the expense of faithfulness. Table 4 : Model adherence to provided entity guidance. Embedded is an ablation of SPEER without sentence-level planning as described in the Ablations paragraph.SPEER follows the instructions better than Guided Prompt. We compute the adherence to the instructions-which are to write a summary with a given set of ESGs-in a similar way as we measure entity based overlap between model-generated and reference entities. Specifically, we extract entities from each generated summary and align them to a subset of the source ESGs. Then, we measure the overlap (recall, precision, F1) scores vis-a-vis the guidance itself (the set of ESGs predicted as salient by the ESG classifier from §3). Table 4 demonstrates that for both Mistral and Zephyr, SPEER adheres better to the provided guidance. SPEER Mistral and Zephyr F1 score is .666/.691 versus .621/.629 for Guided. Even though no guidance is given, we include Non-Guided to illustrate how different the entities explicitly selected by the classifier is from the entities implicitly chosen during summary generation. In other terms, auto-regressive implicit content selection diverges from explicit content selection. Taken together with the results in Table 2 , we believe that content selection for long-form clinical summarization is best viewed as a separate task from realization, with its own set of models, architectures, and objectives. 
conclusion: 
 We are the first to explore fine-tuning LLMs (Mistral-7B-Instruct and Zephyr-7B-β) on the highly difficult, yet highly important, task of hospital-course summarization. We find that content selection is best performed by a dedicated salience classifier, which then guides the LLM in summary generation. We observe that simply appending the guidance to the prompt improves the coverage of salient entities yet harms faithfulness. To improve coverage and faithfulness, we introduce SPEER: Sentence-Level Planning via Embedded Entity Retrieval.By directly retrieving the entity guidance from the source notes, metrics suggest that SPEER summaries are more grounded and complete. 
 
 Appendix: Coarse Filtering. Source notes typically exceed the 8, 192 context window on which Mistral and Zephyr were trained. Additionally, clinical notes include many irrelevant sections, including administrative text, minutely detailed descriptions of surgical procedures, and patient disclosures. Such sections are typically easy to identify. To filter out irrelevant content and fit the maximum context window, we learn a coarse section filter. The inputs to the extraction model are individual sections (header and body), extracted from source notes with a custom toolkit based on Clarity NLP. A RoBERTA-classifier is trained with a logistic loss to predict the salience. The labels are continuous between 0 and 1 and represent the average of ROUGE-1 and ROUGE-2 F-1 scores between the reference summary and the section text. During inference, we preserve the original order of the text but remove sections-starting with the lowest scoring-until the total tokenizer token count is no greater than 8,192. We also report scores on MIMIC-III, despite MIMIC having a great deal of unsupported content in reference summaries (Adams et al., 2022) .Instruction Templates. As shown in Figures 6 and 7 , for each model, source notes are simply concatenated chronologically. For each note, we generate a set of header lines which include the title of the note and the date of the note. We also explicitly specify where the note lands in relation to the rest of the admission, e.g., "Day 1 of 4 (On Admission)".Although we separately fine-tune each model, we still include custom instructions. The baseline non-guided instruction is: "Generate the BRIEF HOSPITAL COURSE summary."The prompt guidance instruction is: "Generate the BRIEF HOSPITAL COURSE summary using only the medical entities (PROBLEMS, TREATMENTS, and TESTS) provided." The guidance (list of ESGs grouped by semantic type) is appended to the source notes. SPEER's instruction is: "Retrieve a subset of the medical entities in double brackets {{ }} and use them to generate the next sentence of the BRIEF HOSPITAL COURSE summary." The line: "### BRIEF HOSPITAL COURSE:\n" is appended to the end of the input and is an indicator to the model to start generating the summary.Datasets. We train on a single dataset and evaluate on three diverse held-out sets. Training. We train on ˜167k in-patient hospital admissions from a large metropolitan hospital from 2020-2023. It is highly diverse in terms of patient population and care setting: emergency, surgery, obstetrics, pediatrics, etc. Testing. We evaluate on a held out portion of 1,000 admissions in the same time-frame as the training set: CUIMC:2020-2023, as well as admissions from an earlier time period: CUIMC:2010-2014, in which the Electronic Health Record (EHR) system was different. As a result, note templates and titles may differ. By training on notes with one set of EHR templates and testing on both seen and unseen styles, we can test robustness of methods to subtle shifts in style and content organization. MIMIC. We also evaluate on a held-out set of 900 examples (source notes plus extracted reference summaries for evaluation) from publicly available MIMIC-III clinical notes with pre-processing from Adams et al. (2022) . Table 5 shows high-level statistics for the train-test splits. In contrast to other commonly used clinical NLP benchmarks (Chen et al., 2022; Gao et al., 2023) , the hospital-course summarization stands out as longitudinal, multi-document, and lengthy. On average, the inputs contain from 25.5 (CUIMC) to 162.7 (MIMIC) source notes, which are synthesized into lengthy (11-37 sentence) summaries. Identifying salient, non-redundant content amounts to finding many needles in many haystacks, and then de-duplicating to ensure that each needle is unique. MIMIC admissions, on average, contain a substantially higher number of source notes than CUIMC, while also having very long reference summaries (37 sentences). Despite the long inputs, Adams et al. (2022) reveal much of the content in MIMIC reference summaries is not mentioned anywhere in the source notes-due to incompleteness-which reduces scores on reference-based metrics. We initialize the token-level encoder described in §3 from a 279 million parameter encoder-only model: xlm-roberta-longformer-base-16384, which is a Longformer initialized with weights from XLM-RoBERTa (Conneau et al., 2019) without any additional tuning. For modeling ESGs, we use a randomly initialize BERT encoder layer with the same configuration as XLM-RoBERTa. As described in §3, ESGs are sorted by inverse frequency and frequency rank embeddings are added to the representations of each ESG before passing to the modeling layer. We learn 1, 024 unique ranks and, if the source notes include more than 1, 024 ESGs, we truncate to 1, 024. We train on batches of 16 with AdamW optimizer for 100k using a scheduled learning rate (maximum 3e -5 with linear warmup of 1000 steps, followed by linear decay). Weight decay of 5e -5 is used. The entity boundary tags (<e>, </e>) are added as special tokens to the tokenizer and the embeddings are updated during fine-tuning. Abstractive Summarization. We fine-tune in two stages. The first stage involves training the baseline non-guided model for 1 epoch. For the second stage, we further-finetune the baseline and entity-guided models for an extra 4,000 steps. We break up training into two stages to reduce total training time since the weights from first stage are re-used multiple times during the second stage. Initial Fine-Tune. We fully fine-tune Mistral-7B-Instruct and Zephyr-7B-β using the baseline instructions for 1 epoch with a batch size of 16 and a learning rate of 5e -6. We used the AdamW 8-bit optimizer with a cosine learning rate scheduler. To fit the model onto two Nvidia A6000 48GB GPUs, we use DeepSpeed Stage 2 (Rasley et al., 2020) , FlashAttention-2 (Dao, 2023) , bfloat16 (BF16) precision, gradient checkpointing, and train on per device batches of size 1 with gradient accumulation. We used oracle section filtering to ensure that no training examples exceeded 8,192 tokens. 1 epoch took 10 days to complete. We computed a validation loss after every 500 steps of training and, in Figure 5 , show a smooth loss curve. Further Fine-Tune. We further fine-tune all variants: baseline and entity-guided models for an additional 4,000 steps from the initial fine-tuned weights. Performance plateaus after 2500 additional fine-tuning steps. As such, we save checkpoints every 500 steps between steps 2500 and Evaluation Metrics. We rely on two entity-based overlap metrics: Source-Grounded Recall (SGR) and the Hallucination Rate (HR). Some concepts in clinical reference summaries are not present in source notes, as noted by Shing et al. (2021) and Adams et al. (2022) . Unsupported-or hallucinated-reference content should not be included when computing entity overlap. As such, instead of directly computing overlap between reference and model-generated entities, we separately align summary entities to entities in the sources notes, and measure overlap between source-aligned entities. Specifically, we align reference and model-generated entities to a subset of ESGs from the source notes: {ESG re f →src } and {ESG model→src }. in order to penalize multiple hallucinations of the same synonym group. We report the number of tokens (# of Tokens) to account for length biases in the metrics.More broadly, we capture faithfulness at both the summary-level with BERTScore-Precision (BSP) (Zhang et al., 2019) and, at the sentence level, with ClinDistill (Adams et al., 2023d )a state of the art sentence-level faithfulness metric for hospital-course summarization.ClinDistill is a regression model which is distilled from an ensemble of several pre-existing state of the art faithfulness metrics. It predicts a raw, unnormalized score for each sentence, whose mean is roughly zero. We use BERTScore-Precision (BSP) (Zhang et al., 2019) , which measures the degree to which summary tokens are well-aligned to at least one token in the source notes, rather than BERTScore-F1 because it was shown to correlate better to finegrained expert annotations for the faithfulness of hospital course summaries (Adams et al., 2023d) . Specifically, we report the BSP between the model-generated summary and the source notes. We compute contextualized embeddings for each token with the encoder from allenai/led-large-16384 and follow Adams et al. (2023d) in using just the final hidden state as the representation for each token.We also report ROUGE-1 and ROUGE-2 scores despite a known inverse relationship between ROUGE score and faithfulness for clinical summarization (Adams et al., 2023c) . This negative correlation has to do with unsupported content in references (Adams et al., 2022) . Unfaithful models will mimic unsupported content in references, while faithful models tend to stick to what is explicitly stated and, as such, be penalized by ROUGE.The concatenation of encoder hidden states is similar to Fusion-In-Decoder(Izacard & Grave,2020). 


Decision: This paper investigates the task of clinical summarization and how well fine-tuned LLMs can do on it.  They do not select the right content, so this paper augments them with a separate classification module to decide what entities to include in the summary. These entities are incorporated via a sentence-level grounding, which is a novel (to my knowledge) method of entity guidance/summary planning. Results show similar ROUGE values, but better entity coverage and reduced hallucination, which are two important metrics in this domain.

Reviewers found the problem to be worthwhile and timely, and the method interesting.  Reviewers raised concerns about aspects including the novelty, situatedness with respect to related work, evaluation, and presentation of the work. However, in the discussion, these points have largely been addressed.  The paper would benefit from related work, the human evaluation, and a more formal description of the method being added in the next version, as the authors have promised to do.

Reviews: review: paper_summary: paper_summary: The authors address the challenge of generating comprehensive and accurate hospital discharge summaries using LLMs. Fine-tuning open-source LLMs (Mistral-7B-Instruct and Zephyr-7B-β) on nearly 170k patient records revealed issues with incomplete and unfaithful summaries. To improve performance, the authors developed SPEER (Sentence-level Planning via Embedded Entity Retrieval), which uses a smaller, encoder-only model to predict salient entities and guide the LLMs. This method marks entity spans with special tags, enhancing content coverage and faithfulness. The proposed method demonstrated improved performance over non-guided and guided baselines on three diverse datasets.
strength_weakness: strength_weakness: 1. Since the final summary is the concatenation of generated summary sentences, the author provided method might affect the coherence of the final summary. However, the authors do not evaluate the coherence of the summary or mention the importance of coherence in this specific application.

2. Given that the reference summary may contain information that cannot be supported by the source document (as mentioned by the author, especially on MIMIC), it is not clear that an increase in ROUGE score by author's proposed methods mean better performance.

3. For the remaining auto-evaluation metrics, it looks like a simple baseline may also achieve a high performance. For example, suppose a model simply extracts a sentence (can extract short sentences on purpose to make the final summary short or close to the length of other model-generated summaries) that has entity mentions from each ESG based on the source document and then concatenates these extracted sentences together. This final summary would get a Source-Grounded Recall (SGR) of 1 and a Hallucination Rate (HR) of 0. Similarly, this summary would get a BertScore and ClinDistill close to 1 since all sentences are directly extracted from the source document. Would this final summary be a good one? (assuming ROUGE do not reflect the quality of LLM-generated summaries (https://arxiv.org/pdf/2209.12356, https://arxiv.org/pdf/2301.13848)).

4. The authors claim that BSP and ClinDistill are SOTA metrics for faithfulness evaluation. However, BSP do not correlate well with human judgements and it is not clear whether ClinDistill is suitable to evaluate summaries generated by LLMs as the model is initially being evaluated over non-LLM generated summaries and errors from LLM-generated summaries might be harder to detect and hence lower correlation.

5. It is not clear regarding the accuracy of forming entity synonym groups (ESG). This makes the evaluation results related to ESG not convincing enough.

6. There is a lack of significant tests in the main tables. It is not clear whether some improved results are significant.

7. There is no human evaluation of the quality of model generated summaries. Are summaries generated by the author's proposed method more coherent, readable, and factual? Do humans prefer summaries generated by author proposed methods?

8. There are no example outputs of generated summaries from different models so the reviewer cannot develop an overall feeling about the quality of those generated summaries.
questions: questions: 1. The reviewer does not find the average total length of source documents for each example.

2. The mathematical formula for HR misses “|” on the right side.
paper_summary: paper_summary: This paper presents an approach for generating clinical note summaries (in particular, discharge reports). To overcome challenges in the lengthy source text, the authors first identify important entities and then retrieve the relevance context during the generation process on the basis of these entities. Automated assessments show this process is stronger than non-guided generation using existing LLMs.
strength_weakness: strength_weakness: - The main algorithm was described imprecisely (only an English-language description and a figure). I am not certain that I would be able to reproduce the process without a clearer description.
 - The evaluation was limited to automatic assessment of the reports. It would be stronger if (at least a sample) was assessed by professionals to ensure the suitability of the reports (even if the human assessment focused on the recall and hallucination, as the automated process did)
 - The relationship between this work and prior art was not sufficiently described, especially WRT other approaches that focus on entity guidance (more below).
questions: questions: - There's prior work in guiding clinical note generation with salient entities (e.g., https://www.semanticscholar.org/paper/Attend-to-Medical-Ontologies%3A-Content-Selection-for-Sotudeh-Goharian/4c3b5f8db4f44ed4d24e15227a0da30f7c20a665?utm_source=direct_link). This work seems substantially different than those due to the focus on planning, but they probably ought to be discussed as prior related work.
 - Consider using an algorithm block to describe the algorithm more clearly and precisely.
paper_summary: paper_summary: This paper introduces SPEER (Sentence-Level Planning via Embedded Entity Retrieval), a method for improving the coverage and faithfulness of long clinical summaries generated by fine-tuned large language models (LLMs). I view this work as a unique and important contribution to real-world clinical research and practice. The method and insights can potentially be generalized to other long-form summarization tasks.
strength_weakness: strength_weakness: 1. It would benefit from adding a section for related works, including state-of-the-art LLM summarization methods.
2. It would benefit from an efficiency analysis (e.g., time efficiency, scale-up analysis).
questions: questions: 1. The current Hospital-Course Summarization seems to belong to a related work section.
2. A human/expert evaluation should be considered, especially regarding faithfulness analysis.
paper_summary: paper_summary: This paper presents a LLM-based method, named SPEER (Sentence-level Planning via Entity Retrieval), to generate hospital-course summaries from clinical notes (admission to discharge notes). The main innovation is to identify entity synonym groups from from the notes, identify the salient entities using classification among them to perform sentence-level guiding of the LLM to generate sentences that include the salient entities. They fine-tune open-source LLMs (Mistral-7B-Instruct and Zephyr-7B) for the task. They show that SPEER improves the coverage of salient entities and faithfulness over non-guided and guided LLM baselines.
strength_weakness: strength_weakness: - The main contribution is the sentence-level planning aspect, which may be considered to be of limited novelty to this community.
- It would have been interesting to assess the effect of the method on several biomedical summarization tasks to show generalizability.
questions: questions: - The ground truth references are extracted from discharge summaries. How noisy are these references in terms of content?
- They cluster related terms using embeddings. What would be the effect of using simple entity linking instead of identifying entity synonym groups? 
-The number of entity spans in reference summaries seems too low (30; Table 1). Is this correct?

Comments: title: Follow up on the rebuttal
Positive
Thank you for your rebuttal
comment: Thanks for the detailed and reassuring response. However, given the substantial changes that you'll make to the camera-ready version of the paper, I hope you understand my hesitation in changing my assessment before reading the revision. I really look forward to the next version of the paper, though!
Hi - Thank you for taking the time to review SPEER.  We wanted to follow up and see if you had any lingering doubts after reading our rebuttal.  Thanks again!
Thanks for the responses. I have increased my rating from 5 to 6.
The authors' responses make sense, and I keep my overall positive score.
I acknowledge that I have read the responses by the authors and I will keep my positive score.