SPEER: Sentence-Level Planning of Long Clinical Summaries via Embedded Entity Retrieval
Authors: Griffin Thomas Adams
Jason Zucker
Noémie Elhadad
Venue: COLM (2024)
Keywords: guided summarization, clinical text, retrieval-augmented generation, faithfulness
TL;DR: We introduce an easy-to-implement RAG-style method---SPEER---which improves the coverage of salience entities and faithfulness over both non-guided and guided LLM baselines for long-form clinical summarization.

Abstract: Clinician must write a lengthy summary each time a patient is discharged from the hospital. This task is time-consuming due to the sheer number of unique clinical concepts covered in the admission. Identifying and covering salient entities is vital for the summary to be clinically useful. We fine-tune open-source LLMs (Mistral-7B-Instruct and Zephyr-7B-$\beta$) on the task and find that they generate incomplete and unfaithful summaries. To increase entity coverage, we train a smaller, encoder-only model to predict salient entities, which are treated as content-plans to guide the LLM. To encourage the LLM to focus on specific mentions in the source notes, we propose SPEER: Sentence-level Planning via Embedded Entity Retrieval. Specifically, we mark each salient entity span with special "{{ }}" boundary tags and instruct the LLM to retrieve marked spans before generating each sentence. Sentence-level planning acts as a form of state tracking in that the model is explicitly recording the entities it uses. We fine-tune Mistral and Zephyr variants on a large-scale, diverse dataset of ~167k in-patient hospital admissions and evaluate on 3 datasets. SPEER shows gains in both coverage and faithfulness metrics over non-guided and guided baselines.

Published as a conference paper at COLM 2024
SPEER: Sentence-Level Planning of Long Clinical Summaries via Embedded Entity Retrieval
Griffin Adams
Jason Zucker & Noémie Elhadad
Department of Computer Science Columbia University
Department of Biomedical Informatics
Columbia University
New York, NY, USA
New York, NY {jz2700, ne60) (0660)ecumc..m c edu
griffin. edams@ccouubia.. edu
Abstract
Clinician must write lengthy summary each time patient is discharged from the hospital. This task is time-consuming due to the sheer number of unique clinical concepts covered in the admission. Identifying and covering salient entities is vital for the summary to be clinically useful. We fine-tune open-source LLMs (Mistral-7B) -a-aall--I-nnuuuuee Instruct and Zephyr-7B-B) on the task and find that they generate incomplete and unfaithful summaries. To increase entity coverage, we train smaller, encoder-only model to predict salient entities, which are treated as content-plans to guide the LLM. To encourage the LLM to focus on specific mentions in the source notes, we propose SPEER: Sentence-level Planning via Embedded Entity Retrieval. Specifically, we mark each salient entity span with special "{{ }}" boundary tags and instruct the LLM to retrieve marked spans before generating each sentence. Sentence-level planning acts as a form of state tracking in that the model is explicitly recording the entities it uses. We fine-tune Mistral and Zephyr variants on a large-scale, diverse dataset of 116k in-patient hospital admissions and evaluate on 3 datasets. SPEER shows gains in both coverage and faithfulness metrics over non-guided and guided baselines.
1 Introduction
Clinical professionals are experiencing physical and emotional burnout at unprecedented rates (Maslach & Leiter, 2016; National Academies of Sciences, 2019; Kroth et al., 2019). significant factor driving clinician burnout is the Electronic Health Record (EHR): the information overload it produces, and the documentation burden it requires (Shanafelt et al., 2016; Moy et al., 2021). Increased time spent at the desk means less face-to-face interaction (only 27% of working hours spent with patients) and the resulting burnout has been associated with an increased risk of errors (Salvagioni et al., 2017; Panagioti et al., 2018).
Large Language Models (LLMs) have the potential to reduce the documentation burden by either directly replacing clinicians or acting as co-pilot to reduce the time it takes clinicians to complete certain tasks (Clusmann et al., 2023; Perlis & Fihn, 2023). To date, much of the focus of LLMs in healthcare has been on applying closed models, such as GPT-4, to well-defined granular tasks, such as closed-book question answering (Nori et al., 2023a) and single-document radiology report summarization (Sun et al., 2023), using small publicly available benchmarks. In this paper, we adapt open-source LLMs (Zephyr and Mistral) to an onerous, longitudinal documentation task: hospital-course summarization, by training on nearly 170k patient records from a large metropolitan hospital. Hospital-course summarization (Adams et al., 2021) involves recounting in a narrative form the events occurred during the patient stay, and why they happened. Clinicians write hospital-course summaries each time a patient is discharged from a hospital in the form of the mandatory "Brief Hospital Course" section of the Discharge Summary.

---

Published as a conference paper at COLM 2024
Among others, the task is cognitively difficult and time consuming for two principle reasons (Adams et al., 2021). The relates to the ilentification of salience. When a patient is admitted to hospital, every test, diagnosis, procedure, and medication, consequential or not, is typically entered into at least one clinical note. Due to frequent copy-and (Hirschtick, 2006; Adams et al., 2020), moreover, these are typically entered multiple times across notes. This leads to a severe case of note bloat (Shoolin et al., 2013) in which finding relevant information is commensurate to finding a needle in haystack. The second major challenge involves coverage of salient information. Despite the high presence of redundant and irrelevant information, there is still a great deal of relevant information, especially for lengthy admissions, which can last up to a month or longer. In these cases, it is very easy to omit critical information, which could potentially render the summary clinically harmful.
Treating the notes from admission to discharge as inputs, we construct large scale fine- tuning dataset from the full patient records for all inpatient hospital admissions Columbia University Irving Medical Center (CUIMC) from 2020-2023, Clinician-authored Brief Hos- pital Course summaries are extracted from the corresponding discharge summary for the admission and serve as ground-truie truth references. We perform full parameter fine-tuning Mistral and Zephyr 7B parameter models and demonstrate that they frequently hallucinate and fail to cover salient entities.
To better ground the LLMs on salient source entities, we train a specialized, smaller classifi- cation model to perform explicit content selection. Given the entity-dense nature of task, we select groups of synonymous entities-medine for content selection. To help LLMs to adhere these content plans, we propose SPEER: Sentence-Leve Planning via Embedded Entity Retrieval. Specifically, we mark each salient entity span with special {f }} boundary tags and instruct the LLM retrieve marked spans before generating each sentence. Sentence-level planning acts as a form of state tracking in that the model explicitly records the entities uses.
concepts-as the appropriate granularity
Our primary contributions are to:
Fine-tune state-of-the-art open source LLMs (Mistral-7B-Instruee and Zephyr-78-B) on a large-scale dataset for long-form clinical summarization, and test on three diverse datasets from different EHRs.
Demonstrate that content selection should be thought of as its own classification task, even in the world of LLMs. Dedicated content selection, performed by a small encoder-only classifier, outperforms implicit content selection from auto-regressive LLM decoding.
Introduce an easy-to-implement Sethhod-SPEERRRE which improves the coverage of salience entities and faithfulness over both non-guided and guided LLM baselines.
2 Related Work
LLM Summarization. Recent work has found that API-based closed source models, such as Claude, GPT-3, and GPT-4, can generate high quality summaries of news articles in the zero-shot setting (Goyal et al., 2022). Humans prefer L.M-ggenrrted summaries from GPT-3 over summaries generated from the previous generation of smaller, fine-tuned models (e.g., BART, PEGASUS) (Zhang et al., 2023). Human evaluation is critical to revealing the superiority of LLM-generated summaries given the limitations of reference-basee metrics. In fact, Zhang et al. (2023) find that annotators judge LLM-generated summaries on par with expert-level summaries carefully crafted by freelance writers. By iteratively fusing new entities into an existing summary draft, Chain-of-Density (CoD) (Adams et al., 2023b) enables LLMs, e.g., GPT-4, to generate entity-dense summaries, which are often favored by human annotators over earlier, less dense, drafts. Explicit planning can also be performed with Summary Chain-of-Thouge
(SumCoT)), a technique that guides LLMs to generate summaries by focusing on core news elements in a step-by-step manner, leading to more coherent and comprehensive summaries (Wang et al., 2023). On the evaluation side, Chen et al. (2023) demonstrate that zero-shot prompted LLMs can outperform existing specialized classifiers on factuality detection.
2

---

area of active Published as a conference paper at COLM 2024
Guided Summarization. Abstractive summarization requires three sequential tasks: con- tent selection (extraction), content planning (organization), and surface realization (abstrac- tion). With simple auto-regressive generation, the first two steps are generally performed implicitly with the last. Yet, prior work suggests that making content selection an explicit step, which is handled by separate, dedicated model, can outperform the all-in-one ap- proach (Sharma et al., 2019). For instance, an extractive model can be used to enhance the performance of an abstractive model by treating the extract as an auxiliary input with its own encoder (GSum, (Dou et al., 2021)). Other work simply prepends salient content to the source text form of control (CTRLsum (He et al., 2022)). Relevant to our work, CTRLsum explores using entities as a form of control. The FROST model separates content selection and planning from realization with single model, by having the model first generate an entity-basee plan before generating the full abstractive summary (Narayan et al., 2021). SPEER, on the other hand, interleaves planning and realization and relies on a separately trained classifier for content selection. Adams et al. (2023a) demonstrate that offloading content selection can be used to directly control the diversity of downstream summaries. Their PGA model can be used to enhance the performance of both small encoder-decoder (BART and PEGASUS) and large decoder-only (GPT-3.5) abstractors.
Stanza entity mentions
SapBERT embedding pairwise similar concepts
Connected Subgraphs Entity Synonym Groups
leukocytosis
cyto reduction
leukocytosis
cyta
reduction
White Blaod Gell
NEOPLASM DEBULKING
White Blood Gell
NEOPLASM DEBULKING
Extract Spans
Source Notes
WBC
WBC COUNT
WBC
WBC COUNT
debulking
Cytoreductive
debulking
Cytoreductivv surgery
surgery
Figure 1: Extracting entities and forming groups of synonymous entities (ESGs). For each admission, we form a set of ESGs from the source notes and content selection is performed by classifying each ESG as salient or not.
Hospital-Course Summarization. The multi-document summarization task of synthe- sizing the course of events during patient's admission to a hospital research (Adams et al., 2021). Using publicly available source of data (MIMIC-III), Adams et al. (2022) tackle faithfulness by re-writing, or revising, reference summaries before training models on synthetic, grounded hospital-course reference summaries. Similarly to our work, Searle et al. (2023) guide an abstractive model with clinical concepts to improve ROUGE scores of hospital-course summaries. They guide the model with all source concepts, while we perform content selection, or filtering, first. We embed the selected content back onto the source notes, whereas they guide BART with separate encoder stream (as in GSum (Dou et al., 20 Preliminary studies of LLMs into biomedical applications have largely focused well-defined medical reasoning tasks (Nori et al., 2023b;a), such providing a differential diagnosis based on a patient record (McDuff et al., 2023). Those that have explored clinical summarization have largely focused on single-document tasks of a finer temporal granularity (Veen et al., 2023), such as generating the impressions section of radiology report (Liu et al., 2023b;a; Chuang al., 2023; Van Veen et al., 2023), ICD coding (Boyle et al., 2023), constructing patient's problem list construction from progress note (Veen al., 2023), and generating note from a doctor-patient conversation (Abacha al., 2023; lonescu et al., 2023).
3 Selecting Salient Entities
The process of extracting entities, identifying synonym pairs, and then forming Entity Synonym Groups (ESGs) is graphically depicted in Figure 1. In Table 1, we plot entity-based
3

---

Published as a conference paper at COLM 2024
Statistic
Source
Reference
Entity Spans
1666
36
Unique ESGs
473
% Problems
0.364
0.492
Treatments
0.268
0.339
% Tests
0.368
0.169
Table 1: Entity and ESG statistics across source notes and reference summaries. "Entity Spans" refers the total number raw entity mentions, "Unique ESGs" to the number of synonym groups formed from the raw mentions across the source notes. We also report the fractional breakdown of entities by semantic type.
statistics for source notes and reference summaries. We refer to both the Figure and Table in subsequent paragraphs.
Extracting Entities. We use Stanza (Qi et al., 2020) for entity extraction. In particular, we use the clinical NER model (Zhang al., 2021) which was trained on MIMIC-III notes (Johnson et al., 2016) for the i2b2-2010 clinical NER shared task (Uzuner et al., 2011). The model extracts entity spans from three disjoint semantic types: PROBLEM, TEST, TREATMENT. Problems are diagnoses and symptoms, Tests cover lab tests and imaging, while Treatments span medications and procedures.
Identifying Entity Synonym Pairs. Clinicians frequently rely on acronyms and shorthand when documenting, which leads to large variance in how concepts are mentioned across & Elhadad, 2016; Adams et al., 2020). We follow (Adams al., 2023d) and use similarity in embedding space to identify synonymous clusters of entity spans. Specifically, we embed all mentions with SapBERT (Liu et al., 2021)-which is trained to align synonymous clinical concepts-and use cosine similarity to identify all synonymous pairs. We manually assign binary labels (unrelated, synonymous) to 1,000 mention-paire We then select the threshold (0.75) for cosine similarity classification which maximizes the Fi-score overlap with human labels. Figure 1 illustrates the need for semantic over lexical matching. Acronyms (WBC -> White Blood Cell) have identical meanings and lexical overlap. This also holds true for synonyms: Cytoreductio surgery and NEOPLASM DEBULKING, which both describe the resection of tumorous growth.
notes [Demner-Fushmen
Forming Entity Synonym Groups (ESG). For each hospital admission, we collect all entity mentions across the source notes and form a graph with one node for each unique entity mention. We assign an edge between two mentions iff they are exact-match duplicates or have pairwise SapBERT similarity of 2 0.75. We then treat all fully-connected sub-graphs as Entity Synonym Groups (ESG). Entity content selection is then performed over ESGs, e.g., which represent model a single medical concept, rather than over specific span mentions. As shown in Figure 1, the process of forming ESGs by computing fully connected sub-graphs based on the pairwise similarity graph greatly reduces entity sparsity. Eight unique entity spans form two ESGs. The entities the ESG all relate the same precise topic-if not the exact same concept. For instance, leukocytosis is condition characterized by high white blood cell (WBC) count and thus is connected directly to WBC COUNT and, via the graph, to WBC and White Blood Cell.
Defining ESG salience. For each example (hospital admission), we extract the ESG from across the source notes. Based on the embedding similarity method for synonymity, given embedding similarity scores, consider ESG as "salient" if 2 spans in the ESG is a synonym of 2 entity ssan(s) extracted from the reference summary. Only 5.7% of the source ESGs "salient" by this definition, which underscores the difficulty of content selection for this task and dataset.
Learning ESG salience. We build a hierarchical token-to-ESG encoder model to perform binary classification over ESGs. This approach is inspired by previous hierarchical extractive
4

---

Published as a conference paper at COLM 2024
Farm Entity Synonym Groups (ESG)
ESG Modeling & Salience Classification
Embed Salient ESGs in Source
Sentence-Levt Planning R Retrieve Realize Repeat
Notes
Admission
Progress
Note
Note
([CAD)) (10M22) ((enn)
yo atmonn
(NADD)
Source
(JCAD)).
gooiiasemss dript)-
Realize
yla voman NAD hx of CAD,..
Notes
(DANZ))
(pppettensoon)
allaco
Hanglooddenaa)
HOLD alacemnne mecs.
Besly caass
(HTM)) (joinszee drip))
Realice
Figure 2: SPEER: Sentence-Level Planning via Embedded Entity Retrieval. The entire process of generating a hospital-course summary from concatenated set of clinical notes is shown above. The first two steps relate to the formation and classification of Entity Synonym Groups (ESGs) from $1. The next two steps visually describe the SPEER approach in $4. First, salient entity mentions are marked with special {{ }} boundary tags, which indicate that they are allowed to be retrieved during generation. Then, during generation, each summary sentence is generated on its own line. Above each sentence line, the model is instructed to first retrieve the entities it plans to use in the following sentence simply by generating entities within the {{ }} tags. This single-pass decoding can be explained with the acronym R Sereeve-Reelize-eeeee
because each sentence is a realization of a plan.
methods from Liu & Lapata (2019); Bi et al. (2021); Adams et al. (2023a). First, we demarcate each entity span with newly initialized <e> and </e> tokens. Then, we concatenate all source notes and encode tokens with long-range encoder (LongFormer (Beltagy et al., 2020)), which can fit up to 16,384 tokens. We construct hidden-state representations of each entity span by mean-pooling the hidden states of each word-piece associated with the span (inclusive of <e> and </e> tokens). Next, we mean-pool the hidden states of all entity spans associated with the same ESG (e.g., "Diabetes type 2", "DM II",, "DM2"). Then, we add an ESG modeling layer as a newly initialized, fully-connected BERT encoder layer. To exploit the fact that frequently mentioned concepts tend to be salient, we assign each ESG to a numerical range according to inverse frequency (most mentions first). We learn an embedding for relative frequency and add it to the ESG hidden state before passing through the modeling layer. A linear classification head is added to produce a single logit for each modeled ESG representation. We compute logistic loss over each ESG logit.
ESG classification inference. Concatenated source notes can sometimes exceed the Long- Former context window of 16,384. During training, we simply truncate to 16,38% 384. Yet, during inference, to avoid information loss, we chunk notes into disjoint windows of at most 16, 384 and perform separate token-level encoding before concatenating hidden states!. The ESG-modeling layer has maximum length of 1024 ESGs. For the rare case with > 1024 ESGs, we drop the FŠGs whose term frequency of mentions across the source is lowest.
4 ESG-Guided Summarization
In this section, we explore methods for ESG-guided summarization: by which summary generation is conditioned on both the source documents and a pre-selected set of ESGs. We first present a simple prompt-based baseline before presenting our proposed approach, SPEER. As in (Dou et al., 2021; Adams et al., 2023a), We train models on oracle provided ESGs as guidance while performing inference with the set of -odee-preddcced ESGs (63).
Prompt Guidance. A logical approach is to convert the set of salient ESGs into natural language prompt and instruct the model to incorporate them into its summary. To compute the oracle prompt for training, we follow the same procedure to define the labels as in $3:
The concatenation of encoder hidden states is similar to Fusion- In- Decoder (Izacard & Grave, 2020).
5

---

Published as a conference paper at COLM 2024
"Defining ESG Salience". Then, we list out the salient ESGs by semantic type: "PROBLEMS", "TREATMENTS" and "TESTS". Each line contains each unique mention of an ESG across the source notes (delimited by ",").
Issues with Prompt Guidance. The above approach is simple, intuitive, and helps ground the summaries onto a set of salient entities. Yet, two issues may arise. Firstly, the model may learn to focus more on the entities themselves and not their actual usage in the source notes. The source notes are lengthy and, as such, the ratio of relevant to irrelevant content is very high. The prompt guidance, however, is precise and only includes the entities which make it into the reference summary. Relatively speaking, the model may learn to over-rely on the list of entities themselves at the expense of their usage in the source notes, which can be difficult to identify. A consequence of this would be high coverage of salient entities the surface, yet irrelevant, inconsequential, or erroneous context supplied for these entities. Secondly, the entity guidance is extensive-Some reference summaries have 100+ unique ESGs. It would be difficult even for a clinician to keep track of which ESGs have been covered so far without discrete state tracking mechanism. State tracking is necessary for the model to both determine which of the salient ESGs have yet to be covered, and, consequently, if all ESGs are covered, break out.
To address both concerns: the lack of source note grounding and the lack of discrete tracking mechanism, we propose SPEER: Sentence-Level Planning via Embedded Entity Retrieval. The SPEER process is shown in the last two steps of 2. To address the grounding concern, we first Embed the salient Entities in the source notes. To do this, we demarcate each entity span from a salient ESG with {{ }} boundary tags. Before generating each summary sentence, the model generates a list of the entities it plans use, the order in which they should appear. We refer to this Sentence-Leve because the model is performing generative retrieval over a fixed set of embedded entities. To show the model that it must only use embedded entities to form its plan, the model is taught to generate the entities with their boundary tags {{ }}. As shown in Figure 2, this R aims to address both the grounding
SPEER.
planning step as "Retrieval"
can be described as R²: and tracking concerns. Firstly, by retrieving the bracketed entities, we are encouraging the model to attend to-or focus on-a specific usage of the salient ESG in context. Secondly, the act of explicitly generating the entities to include in the next sentence makes it easier to keep track of which entities have already been included in the summary. The output template for this proposed R method of summarization is:
### Entities 1: {[span)} {[span)}
### Sentence 1: <sentence 1>
### Entities 2: ([span)) {[span))
### Sentence 2: <sentence 2>
The final summary is simply the concatenation of each line which begins with ### Sentence. We use oracle ESGs during training and those predicted by the ESG classifier at inference. For planning, during training, we extract the in-order entity mentions from each reference sentence and add them spans to the corresponding ### Entities line.
Pseudo Code for SPEER. To make more concrete, we include pseudo code (Figure 3) for creating the training data for ESG selection ($3) and ESG-guided summarization, e.g. SPEER (64). For FSG content selection, the model inputs are the set of input notes and the set of source ESGs. The ESG classifier is hierarchical token-to-ESG encoder which encodes each token, before merging tokens into ESGs, performing ESG-ESG attention, and outputting a binary classification logit. For summarization, decorated inputs and reference summaries are concatenated into a single prompt and the decoder loss on the inputs is ignored.
The process for inference is similar yet relies source ESGs predicted as salient by the content selector, rather than the oracle rel src esgs in Figure 3. The output at inference is parsed to remove planning lines, egg., those starting with "### Entities".

---

Published as a conference paper at COLM 2024
Pseudo Code for generating SPEER training data.
Helpers
Main Script
Creatr on Undiractes 2am HOME ertity spane
geserate lpeer Erate ootet, reforensel:
Croieie 1iai weleee vise but Natten Ca tivebaal
Eniity Groap (05e))
ancty Syroua Srags (854)
sggs prick Tally, caweectee (
irsicion fram ketereece
relevanee
thrashate-
ret Jett Antirylaraaaeet (ntterraa
srt erbedt snouts Debuctor are ena), Desotertre ants
Jetarncn Filtert
Eits Overtpp vids Braa
Copuur Costnette cracts, embers)
spat
ei relevacstors exy, artal,
aettte GELLes sta retuer, callest
teatinn wase
Ter eo cel, (iL.or_.
rel Jivtryy jorcunns Lngg nel WiLorr enc.e9 src.ees)
bicarots tigats with the most saeri
iopats Vists, rwglacaelpae. '(C" 13P)
reference v. sent decorrate, interencaaeeeeeesss ants)
sests Sertencedg Litter reference)
Form Trainine Crampie for selection Srtm Symer
Create sentence lanel plans baed estittes
sumayr ex Trateibeen be t a slon. tereforese a woseet Hooo)
sent, alets
Emerrate Srece Trainine silanets for Sumeer tratten exttte telection
Ioarriae raseriaae plass surtances serities 12, (pbvresr Sestice (1 SPP for staiser
inlectoon [1 selectis,, sanaary genrrate, Spar, tratecneter. reforance)
II
Figure 3: Pseudo-code generate training data for SPEER, which includes a step for
learning to select salient entities ($3), followed by entity plan-guided summarization (64).
Experimental Setup
The experimental setup is detailed in Appendix $A. In summary, we train single dataset of "167k admissions and test on three, diverse held out sets with "1,000 admissions each. Two of the test sets come from the same institution time periods and EHRs, while the third is derived from publicly available MIMIC-III notes (Johnson al., 2016). To ensure fairness in comparisons, each model is trained for exactly the same number of steps ("14,000) in batches of 16 with the same learning rate of 5e 6.
the training set, yet from different
Evaluation Metrics. We rely two entity-based overlap metrics: Source-Grounded Recall (SGR) for salience and the Hallucination Rate (HR) for faithfullness To avoid pe- nalizing models for unsupported entities in references, we separately align summary entities to entities in the sources notes, and measure overlap between source-aligned entities. Specifically, we align reference and model-generated entities to subset of ESGs from the source notes: [ESGref-tsre] and (ESGmode!-srre) Then, we compute
SGR as SGR H ( We compute the hallucination rate (HR) as [[SGGm]-m)]
HR = KENTITY | KENTITY srrc denotes the number of predicted entity men-
HENTITY Ymuderl!
tions which do not have a corresponding source synonym. HR uses entity mentions and not ESGs (as in SGR) in order to penalize multiple hallucinations of the same synonym group. For faithfulness, we also report BERTSccor-PPrecssonn (BSP) (Zhang et al., 2019) and ClinDistill (Adams et al., 2023d) SOTA metric for hospital-course summarization which outputs a real-value number whose mean centered at Ó. We report ROUGE-1/ 2 despite its inverse relationship with faithfulness for clinical summarization (Adams et al., 2023c).
7

---

Published as a conference paper at COLM 2024
MIMIC
Average of Datasets
Model
Entity
BSP Clin t ROUGE
# of Tokens
Entity
BSP Clin f ROUGE
# of
SGR HHR
Distill R1
RR2
SGR
HH
Distill
R2
Tokens
Non-Guided
.200
.116
.664
.971
24.3
279
.339
.126
.683
.886
31.9 16.0
197
Mistral Guided
.266
.171
.648
.541
23.5
6.2
352
401
.151
.678
.683
33.9
18.1
251
SPEER
.302
.040
.667
1.240
25.0
7.0
324
.40
.078
.686
.947
33.9
16.6
234
Non-Guided
245
.121
.653
.899
25.0
335
.366
.138
.673
.789
33.7
16.4
257
Zephyr Guided
247
.136
.651
.593
24.0
6.3
337
.455
.132
.673
.633
34.0
16.4
267
SPEER
.306
.046
.662
1.271
25.9
7.1
364
.399
.084
.682
.907
34.1
16.2
267
CUIMC: 0020-2023
CUIMC: 2010-2014
Model
Entity
BSP Clin 1 ROUGE of
Entity
BSP Clin
ROUGE
# of
SGR HR 4
Distill R1
Tokens
SGR
HR 4
Distill R1 R2 t
Tokens
Non-Guided
447
.161 .692
.670
44.7 31.3
117
.341
.099
.695
1.02
27.0
9.9
195
Mistral Guided
.568
.193 .690
.613
49.5 33.5
180
.399
.091 .696
.903
28.9
14.8
220
SPEER
.572
.117
.696
.741
32.7 48.4
163
.417
.075
.696
.872
28.4
10.0
214
Non-Guided
516
.176
.682
.70
48.1 32.8
168
.399
.116
.684
.901
27.9
9.7
269
Zephyr Guided
.582
.152
.684
.554
49.3
33.1
203
.417
.107
.685
.758
28.7
9.8
260
SPEER
.588
.122
.692
.666
48.3
31.9
188
.424
.084
.692
.791
28.2
9.7
249
Table 2: Results from fine-tuning and Entity-Guided (Guided and our proposed SPEER) hospital-course summarization. Met- rics are defined in $5.
and Zephyr-7B-B Non-Guided
Results
We denote the proprietary test sets as CUIMC:2010-2014 and CUIMC:2020-2023. CUIMC stands for Columbia University Irving Medical Center.
0.8
LongFormer Encoder
Mistral 7B
0.6
Zephyr 7B
SGP
0.4
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
SGR
Figure 4: Comparing the entity-level performance (source-guided recall (SGR) and source- guided precision (SGP)) of explicit content selection (classifying entities with a LongFormer Encoder) versus implicit (auto-regressive decoding) on CUIMC:2010-2014.
Implicit versus Explicit Content Selection. In Figure 4, we vary the threshold for salience classification with the trained Longformer Encoder (from $3) and create a precision-recall curve, where recall is computed with SGR and precision with a similarly computed SGP(recision). On the same plot, we mark the SGP and SGR for fine-tuned Non-Guided Zephyr and Mistral models. Figure 4 demonstrates that Zephyr and Mistral point values fall well below the precision-recall curves of the classifier. The figure demonstrates that classification models can outperform nuto-regressive models at hospital-course summary content selection, even when orders of magnitude smaller (279 million versus 7 billion parameters).
Models which rely on entity guidance achieve higher coverage of salient entities than those that do not. As shown in Table 2, Guided and SPEER) have higher SGR (source-
8

---

Published as a conference paper at COLM 2024
guided entity recall) fractions than Non-Guided across all dataset and base models. Looking at the average across datasets, SGR for models with guidance is ,401/ /40 for Mistral and 415/.43 for Zephyr. The Non-Guided model covers fewer salient entities: .339 and .386 SGR for Mistral and Zephyr, respectively. Summary length (# of tokens) plays a role for Mistral (197 < 251/ /244) but less of role for Zephyr (257 < 267). For Zephyr models CUIMC:2010-2014 Non-Guided produces the longest summaries (269 > 260/ 249) while also covering fewer salient ESGs: SGR of .399 < ..17/ 424. While length can be controlled, the proclivity toward longer, more complete summaries may stem from the fact that guided models are provided clear stopping criteria: break out only when all pre-selected entities are covered.
CUIMC:
2020-2023
Model Name
Change to Model
Entity
BSP
Clin f
ROUGE
# of
SGR
HR L
1
Distill
R11
Iokens
Non-Guided
516
.176
.682
.570
48.1
32.8
168
Guided
Prompt Guidance
.582
.152
.684
.554
49.3
33.1
203
Zephyr
Embedded
Prompt - Embedded
.574
.147
.688
.673
50.5
34.7
191
SPEER
Planning with Retrieval
.588
.122
.692
.666
48.3
31.9
188
Table 3:
From Gon-Guided to SPEER: a step-by-step transition with incremental improve- ments in faithfulness.
Prompt Guided is surprisingly less faithful than Non-Guided. Across test sets and base models, prompt Guided summaries hallucinate more (higher HR) and score lower faithfulness (BSP, ClinDistill) than Non-Guided. Looking at the average of datasets for Mistral, for example, Guided summaries have worse scores for HR / BSP / ClinDistill than Non-Guided: / 16.8/ /.88 versus /26/.683//........ /886, respectively. One would expect that instructing model to stick to entities present in the source text would increase faithfulness. We suspect that Guided may learn to over-rely on the list of entities themselves at the expense of their usage in the source notes. The appended entity guidance might be stealing attention away from the source notes themselves.
SPEER improves both coverage and faithfulness. While adding the guidance to the prompt (Guided) creates a faithfulness-covrrage tradeoff, SPEER consistently improves on both fronts. When looking at the average across datasets, the coverage of salient entities (SGR) is the highest for SPEER for both Mistral and Zephyr: 430//439 versus .339/ /.36 for Non-Guided and .400 /415 for Guided. On faithfulness, SPEER hallucinates less: the average HR is www.00000000 .084 for Mistral / Zephyr versus / .122 for Non-Guided and .151 1151/133 /.122 for Guided. Additionally, (ClinDistill) are highest for SPEER. For BSP, SPEER Mistral and Zephyr score .686/ ,682, more than 1683/6677 /.63 for Non-Guided and .678/ /673 for Guided. For ClinDistill, SPEER Mistral and Zephyr score 947 907, more than .886/ .789 for Non-Guided and /683/ /633 for Guided.
SERTSScore-Preisiie
(BSP) and sentence-level average faithfulness
SPEER is more robust to unseen EHRs. The model was trained on CUIMC: 2020-2023 data, so it is unsuprising that performance is best on held-out set of admissions from the same date range. When switching datasets and EHRs, there is a noticeable performance drop across models, especially for MIMIC. As discussed in Adams et al. (2022), MIMIC-III notes are highly incomplete. As such, much of the reference content is not supported by the available source notes and reference-free metrics are understandably poor. Yet, it is notable that the largest advantage (coverage and recall) for SPEER comes from MIMIC, for which the data the noisiest and the notes come from an unseen institution. SPEER might be more robust to this "zero-shot" setting because it requires the least effort on the part of the abstractive component. The LLM only needs to locate ({ }} tags, rather than needing to implicitly perform salience modeling (Non-Guided) or to link prompted guidance back onto specific parts of the source (Guided).
SPEER Ablations. Table 3 demonstrates incremental improvements in faithfulness and coverage of salient entities as we transition from loaded SPEER model. As discussed earlier, going from non-guided Non-Guided) to prompt
the baseline model (Non-Guided) to the fully

---

Published as a conference paper at COLM 2024
Guided) increases coverage dramatically (SGR goes from .516 + .582 while sentence-level
faithfulness (ClinDistill) declines: .500 + .554. If we replace prompt guidance with embedded guidance: Embedded, we achieve slight decline in SGR: .582 - .574 yet decrease in hallucinations (HR): .152 - .14 and an increase in sentence-level faithfulness: .544 + .673. Embedded is SPEER without the sentence-level planning. The input is the same (notes with embedded salient ESGs) yet the target output is the sunmary without planning. Adding in planning, we arrive at SPEER, which leads to increase in coverage of salient entities: 5544 - .588 for SGR and further decrease in hallucinations: .147 + .122.
We do note that ROUGE scores decline: ROUGE-1 from 50.5 + 48.3 but we believe that this is a necessary side effect of sentence-level planning, which encourages the model to stick to the entities in the source text and not hallucinate plausible, yet unsupported, content. Qualitatively, planning seems to cause a reduction in the number of sentences with no entities which occur in many reference summaries, yet do not contain important details. A paraphrased example is: "Patient verbalized understanding of instructions and plans to follow up with his primary doctor in two weeks." " These types of sentences often achieve high ROUGE scores as they true for many patients, but more often than not, are never stated in the source notes and cannot be assumed to be true. Including common, yet unsupported, content can artificially boost ROUGE at the expense of faithfulness.
CUINC: 2020-2023
Model
Overlap w/ Guidance
Recall Precision
F1
Non-Guided
.376
.621
.266
Mistral Guided
.596
.695
.621
SPEER
.633
.749
.666
Non-Guided
.433
.564
462
Guided
.620
.681
.629
Zephyr
Embedded
.580
.678
.602
SPEER
.678
.745
.691
Table 4: Model adherence to provided entity guidance. Embedded is an ablation of SPEER without sentence-level planning as described in the Ablations paragraph.
SPEER follows the instructions better than Guided Prompt. We compute the adherence to the instructions-which are to write a summary with a given set of ESGs-in similar way as we measure entity based overlap between model-generated and reference entities. Specifically, we extract entities from each generated summary and align them to a subset of the source ESGs. Then, we measure the overlap (recall, precision, F1) scores vis-a-vis the guidance itself (the set of ESGs predicted as salient by the ESG classifier from $3). Table demonstrates that for both Mistral and Zephyr, SPEER adheres better to the provided guidance. SPEER Mistral and Zephyr Fl score is .666 .669 versus .621/669 for Guided. Even though no guidance is given, we include Non-Guided to illustrate how different the entities explicitly selected by the classifier is from the entities implicitly chosen during summary generation. In other terms, auto-regressive implicit content selection diverges from explicit content selection. Taken together with the results in Table for long-form clinical summarization is best viewed as separate task from realization, with its own set of models, architectures, and objectives.
believe that content selection
Conclusion
We are the first to explore fine-tuning LLMs (Mistral-78-Instruct and Zephyr-7B-f) on the highly difficult, yet highly important, task of hospital-course summarization. We find that content selection is best performed by a dedicated salience classifier, which then guides the LLM in summary generation. We observe that simply appending the guidance to the prompt improves the coverage of salient entities yet harms faithfulness. To improve coverage and faithfulness, we introduce SPEER: Sentence-Level Planning via Embedded Entity Retrieval. By directly retrieving the entity guidance from the source notes, metrics suggest that SPEER summaries are more grounded and complete.
10

---

Published as a conference paper at COLM 2024
8 Acknowledgments
We'd like to thank Alex Fabbri and Faisal Ladhak for providing feedback the SPEER algorithm. We'd also like to thank the anonymous COLM reviewers who carefully reviewed draft of this paper.
This research was supported by the National Library of Medicine (NLM) and National Institute of Allergy and Infectious Diseases (NIAID) of the National Institutes of Health (NIH) under Award Number 115LL007079. The content is solely the responsibility of the authors and does not represent the official views of the NIH.
Ethics Statement
There are severe risks associated with releasing any generative Al software for deployment in clinical setting. We do not recommend using SPEER in either fully automated or co-pilot settings without a thorough manual review of its biases and mistakes.
The CUIMC datasets contain sensitive personal health information (PHI), whose safekeeping is protected by HIPAA regulations. We adhered to best practices for safe handling of all patient materials.
References
Asma Ben Abacha, Wen-wai Yim, Griffin Adams, Neal Snider, and Meliha Yetisgen-Yildiz Overview of the mediga-chat 2023 shared tasks on the summarization & generation of doctor-patient conversations. Proceedings of the 5th Clinical Natural Language Processing Workshop, pp. 503-513, 2023.
Griffin Adams, Mert Ketenci, Shreyas Bhave, Adler Perotte, and Noémie Elhadad. Zero-shot clinical acronym expansion via latent meaning cells. In Machine Learning for Health, pp. 12-40. PMLR, 2020.
Griffin Adams, Emily Alsentzer, Mert Ketenci, Jason Zucker, and Noémie Elhadad. What's in a summary? laying the groundwork for advances in hospital-course summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, pp. 4794-4811, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1 /ttp///222.naaalllem/aaaaaaaaaaa maaacl-main.355 URL https: / httpss//aclanthologgy org/20211 naacl monecl-man.33 382.
Griffin Adams, Han-Chin Shing, Qing Sun, Christopher Winestock, Kathleen McKeown, and Noémie Elhadad. Learning to revise references for faithful summarization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.),, Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 4009-4027, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653 /v1 /2022. findings-emnlp.229 URL httph: httt::://aclathhloog// orgpolllooooooog//g//goo org/2022. findings-emmlp 296.
Griffin Adams, Alex Fabbri, Faisal Ladhak, Noémie Elhadad, and Kathleen McKeown. Generating EDU extracts for plan-guided summary re-ranking. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2680-2697, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10.18653/v1/209/ acl-long, URL https: htttps://aclannthologg. org/2023. acl acl-long. 151.
Griffin Adams, Alex Fabbri, Faisal Ladhak, Eric Lehman, and Noémie Elhadad. From sparse to dense: GPT-4 summarization with chain density prompting. In Yue Dong, Wen Xiao, Lu Wang, Fei Liu, and Giuseppe Carenini (eds.), Proceedings of the 4th New Frontiers Summarization Workshop, pp. 68-74, Hybrid, December 2023b. Association for Computational Linguistics. URL
org/2020 newsum-1.7
11

---

Published as a conference paper at COLM 2024
Griffin Adams, Bichlien Nguyen, Jake Smith, Yingce Xia, Shufang Xie, Anna Ostropo- lets, Budhaditya Deb, Yuan-lyue Chen, Tristan Naumann, and Noémie Elhadad. What are the desired characteristics of calibration sets? identifying correlates on long form scientific summarization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), PP. 10520-10542, Toronto, Canada, July 2023c. As- sociation for Computational Linguistics. doi: 10.18653/v1/200 /tt////111222..oooog.555555 URL https://aclanthoology org/2023 acl - loo...on..... 587.
Griffin Adams, Jason Zucker, and Noémie Elhadad. A meta-evaluation of faithfulness metrics for long-form hospital-course summarization. Machine Learning for Healthcare, pP. 1-36, 2023d.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document trans- former. Xiv preprint ar Xrii:2004.001500, 2020.
Keping Bi, Rahul Jha, Bruce Croft, and Asli Celikyilmaz. AREDSUM:
Adaptive redundancy- aware iterative sentence ranking for extractive document summarization. In Proceedings of the 16th Conference of the European Chapler of the Association for Computational Linguistics: Main Volume, pp. 281 291, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v www.:///eellmmaaaaa.aaa eacl-main.222 URL https: hwtp:://llaaaaaoooooooooooo org 2021 eacl-main. 22.
Joseph Boyle, Antanas Kascenas, Pat Lok, Maria Liakata, and Alison QO'eell Automated clinical coding using off-the-shelf large language models. arXiv preprint ar a/Xiv:2310.06552, 2023.
Shiqi Chen, Siyang Gao, and Junxian He. Evaluating factual consistency of summaries with large language models, 2023.
Zhihong Chen, Maya Varma, Xiang Wan, Curtis Langlotz, and Jean-Benoit Delbrouck. Toward expanding the scope of radiology report summarization to multiple anatomies and modalities. arxiv preprint ar drXiv:221..08884 2022.
Yu-Neng Chuang, Ruixiang Tang, Xiaoqian Jiang, and Xia Hu. Spec: A soft prompt- based calibration on performance variability of large language model in clinical notes summarization, 2023.
Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys I Carrero, Jan- Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löffler, Sophie-Carolin Schwarzkopf, Michaela Unger, Gregory Veldhuizen, et al. The future landscape of large language models in medicine. Communications Medicine, 3(1):141, 2023.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. ar preprint arXin:1911.00116, 2019.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. ar Xiv preprint arXiv:2307008691, 2023.
Dina Demner-Fushman and Noemie Elhadad. Aspiring to unintended consequences of natural language processing: review of recent developments in clinical and consumer- generated text processing. Yearbook of medical informatics, 25(01):224 2333, 2016.
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. GSum:
A general framework for guided neural abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4830-4842, Online, June 2021. Associ- ation for Computational Linguistics. doi: 10.18653/vil http://11111.1aaaamaaa https://aclanthologgy org/2021 naacl-main 384.
URL
12

---

Published as a conference paper at COLM 2024
Yanjun Gao, Dmitriy Dligach, Timothy Miller, John Caskey, Brihat Sharma, Matthew M
Churpek, and Majid Atshar. Dr. bench: Diagnostic reasoning benchmark for clinical natural language processing. Journal of Biomedical Informatics, 138:104286, 2023.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the era of gpt-3. arx preprint arXiv:2209.11 12356, 2022.
Junxian He, Wojciech Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. CTRLsum: Towards generic controllable text summarization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5879-5915, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 101.1 /20//22eemmmmmmainna.. URL
org/ 2022. www.e-mainnmmem 396.
Robert E Hirschtick. Copy-and-paste Jama, 25520)22252 2006.
Bogdan Ionescu, Henning Müller, A Drágulinescu, Wen-wai Yim, A Ben Abacha, Neal Snider, Griffin Adams, Meliha Yetisgen, Johannes Rückert, A García Seco de Herrera, et al. Overview of imageclef 2023: Multimedia retrieval in medical, socialmedia and recommender systems applications. In Experimental IR Meets Mullilingguality, Multimodaliity and Interaction, Proceedings of the 14th International Conference of the CLEF Association (CLEF 2023), Springer Lecture Notes in Computer Science LNCS, Thessaloniki, Greece, 2023,
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXit preprint arx::20007 01222, 2020.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic- iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. ar preprint ar "Xii:1909.05853 2019.
Philip J Kroth, Nancy Morioka-Douglar Sharry Veres, Stewart Babbott, Sara Poplau, Fares Qeadan, Carolyn Parshall, Kathryne Corrigan, and Mark Linzer. Association of electronic health record design and use factors with clinician stress and burnout. JAMA network open, 1etpee999999--99969 2019.
Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. Self- alignment pretraining for biomedical entity representations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pp. 4228 4238, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/ http/2221.naaalmmmi //aclanthology.org//// .wa.llaaaialmamamcaaaa.mm 334.
URL https:
Qianchu Liu, Stephanie Hyland, Shruthi Bannur, Kenza Bouzid, Daniel C Castro, Maria Teodora Wetscherek, Robert Tinn, Harshita Sharma, Fernando Pérez-García, An- ton Schwaighofer, et al. Exploring the boundaries of gpt-4 in radiology. arXiv preprint arXiv:2310.1457., 2023a.
Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 3730- 3740, Hong Kong, China, November 2019. Association for Computational Linguistics. doi:
URL https: http::::s///ccaaahoooooooo
1tg//D19-138)
Zhengliang Liu, Aoxiao Zhong, Yiwei Li, Longtao Yang, Chao Ju, Zihao Wu, Chong Ma, Peng Shu, Cheng Chen, Sekeun Kim, et al. Tailoring large language models to radiology: A preliminary approach to Ilm adaptation for a highly specialized domain. In International Workshop on Machine Learning in Medical Imaging, pp. 464 473. Springer, 2023b.
13

---

Published as a conference paper at COLM 2024
Christina Maslach and Michael P Leiter. Understanding the burnout experience: recent research and its implications for psychiatry. World psychiatry, 15(2):003111, 2016.
Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, et al. Towards accurate differential diagnosis with large language models. arviv preprint ar arXiv:2312.00164, 2023,
Amanda Moy, Jessica M Schwartz, Ruijun Chen, Shirin Sadri, Eugene Lucas, Kenrick D Cato, and Sarah Collins Rossetti. Measurement of clinical documentation burden among physicians and nurses using electronic health records: scoping review. Journal of the American Medical Informatics Association, 28(5):998-1008, 2021.
Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simöes, Vitaly Nikolaev, and Ryan Mc- Donald. Planning with learned entity prompts for abstractive summarization. Transactions of the Association for Compulational Linguistics, 9:1475- 1492, 2021. doi: 2011162/tacla_00443
URL httpp:://claatthhhhhht
org/2021. tacl-1.88.
& Medicine & others National Academies of Sciences, Engineering. Taking action against clinician burnout: a systems approach to professional well-being. National Academies Press, 2019.
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint ar 133: 2023a.
Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist founda- tion models outcompete special-purpose tuning? case study in medicine. arviv preprint arXiv:2311.1642,, 2023b.
Maria Panagioti, Keith Geraghty, Judith Johnson, Anli Zhou, Efharis Panagopoulou, Carolyn Chew-Graham, David Peters, Alexander Hodkinson, Ruth Riley, and Aneez Esmail. Association between physician burnout and patient safety, professionalism, and patient satisfaction: a systematic review and meta-analysis. JAMA internal medicine, 178(10): 1317-1331, 2018.
Roy H Perlis and Stephan D Fihn. Evaluating the application of large language models in clinical research contexts. JAMA Network Open, +23))=2233592444235922,,, 2023.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demon- strations, pp. 101-108, Online, July 2020. Association for Computational Linguistics. doi: 10.18653 /v1/ v 20l.//0....dd 14. URL https:://claanthol.gg. wwwp://0.....lddeeeee.... acl-demos. 14.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Denise Albieri Jodas Salvagioni, Francine Nesello Melanda, Arthur Eumann Mesas, Al- berto Durán González, Flávia Lopes Gabani, and Selma Maffei de Andrade. Physical, psychological and occupational consequences of job burnout: A systematic review of prospective studies. Plo5 one, 12(10))e0185781, 2017.
Thomas Searle, Zina Ibrahim, James Teo, and Richard JB Dobson. Discharge summary hospital course summarisation of in patient electronic health record text with clinical concept guided deep pre-trained transformer models. Journal of Biomedical Informatics, 141:104358, 2023.
Tait D Shanafelt, Lotte N Dyrbye, Christine Sinsky, Omar Hasan, Daniel Satele, Jeff Sloan, and Colin P West. Relationship between clerical burden and characteristics of the elec- tronic environment with physician burnout and professional satisfaction. In Mayo Clinic Proceedings, volume 91, pp. 836 848. Flsevier, 2016
14

---

Published as a conference paper at COLM 2024
Eva Sharma, Luyang Huang, Zhe Hu, and Lu Wang. An entity-driver framework for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International joint Conference on Natural Language Processing (EMNLP-IJCNNLP), pp. 3280-3291, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D0 v1/119- URL hctps://aclanthinn http:::/acclaattooooo.o////1/112
Han-Chin Shing, Chaitanya Shivade, Nima Pourdamghani, Feng Nan, Philip Resnik, Dou- glas Oard, and Parminder Bhatia. Towards clinical encounter summarization: Learning to compose discharge summaries from prior notes. Xiv preprint arXiv:2I04.13490 2021.
J Shoolin, I. Ozeran, C Hamann, and W Bria Ti. Association of medical directors of infor- mation systems consensus on inpatient electronic health record documentation. Applied clinical informatics, 4(02))293-3333 2013.
Zhaoyi Sun, Hanley Ong, Patrick Kennedy, Liyan Tang, Shirley Chen, Jonathan Elias, Eugene Lucas, George Shih, and Yifan Peng. Evaluating gpt-4 on impressions generation in radiology reports. Radiology, 307(5);231159,, 2023.
Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552-556, 2011.
Dave Van Veen, Cara Van Uden, Maayane Attias, Anuj Pareek, Christian Bluethgen, Malgo- rzata Polacin, Wah Chiu, Jean-Benoit Delbrouck, Juan Zambrano Chaves, Curtis Langlotz, Akshay Chaudhari, and John Pauly. RadAdapt: Radiology report summarization via lightweight domain adaptation of large language models. In Dina Demner-fushman, Sophia Ananiadou, and Kevin Cohen (eds.), The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pp. 449-460, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 ht/////2223..ool/////// https::/aalanntoology.. org////2 bionlp-1.42
URL
Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, William Collins, Neera Ahuja, Curtis P. Langlotz, Jason Hom, Sergios Gatidis, John Pauly, and Akshay S. Chaudhari. Clinical text summarization: Adapting large language models can outperform human experts, 2023.
Yiming Wang, Zhuosheng Zhang, and Rui Wang. Flement-aware summarization with large language models: Expert-aligned evaluation and -fthuuhf-thought method. In Anna Rogers, Jordan Graberr, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8640-8665, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 /v1 v1 a2023.ac--oong.485 482. URL https: /ates:///cllaain httpps://cc.aanhooooyy.o/g////// http:/hhhoolgyy../222233323 acl -lng 482.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. ar preprint ar Xixe:1904.09675, 2019.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tat- sunori B Hashimoto. Benchmarking large language models for news summarization. arXiv preprint Xrv::22301.13 Stt::2311.1348,, 2023.
Yuhao Zhang, Yuhui Zhang, Peng Qi, Christopher D Manning, and Curtis P Langlotz. Biomedical and clinical English model packages for the Stanza Python NLP library. Journal of the American Medical Informatics Association, 1899)1189-1 06 2021. ISSN 1527-974X. doi: 10.1093/jaaa /a1100///aaaa///a htt///jaaa//aaaaa ocab090. URL https: //tp::////ooiioo /ttp:::///oioorg////// 10g/10. http://.....rrr.o...///////////a 1093/jamia/ccab099.
15

---

Published as a conference paper at COLM 2024
A Experimental Setup
Coarse Filtering. Source notes typically exceed the 192 context window which Mistral and Zephyr were trained. Additionally, clinical notes include many irrelevant sections, including administrative text, minutely detailed descriptions of surgical procedures, and patient disclosures. Such sections are typically easy to identify. To filter out irrelevant content and fit the maximum context window, we learn coarse section filter. The inpuls to the extraction model are individual sections (header and body), extracted from source notes with a custom toolkit based on Clarity NLP. A RoBssfA-classifeer is trained with logistic loss to predict the salience. The labels are continuous between 0 and and represent the average of ROUGE-1 and ROUGE-2 F-1 scores between the reference summary and the section text. During inference, we preserve the original order of the text but remove pections-starting with the lowest scoring- untill the total tokenizer token count is no greater than 8,192.
Example-Level Stats
Source Stats
Reference Stats
Dataset
Split
# Admissions Avg Length of Stay
# Notes # Tokens
Sentences # Tokens
2020-:2020-0 Train
167k
6.3 days
27.8
11k
12.4
207.5
CUIMC: 2020-2023 Test
1k
5.6 days
25.5
13k
11.4
173.9
2U1NC:2010-201 Test
1k
5.2 days
41.4
12k
12.2
201.5
MIMIC
Test
900
30.8 days
162.7
44k
37,0
542.9
Table 5: Statistics for data used for training and evaluating hospital-course summarization models. we use datasets from large Metropolitan Hospital (CUIMC) at two different points of time. We also report scores on MIMIC-III, despite MIMIC having a great deal of unsupported content in reference summaries (Adams et al., 2022).
Instruction Templates. As shown in Figures 6 and 7, for each model, source notes are simply concatenated chronologically. For each note, we generate a set of header lines which include the title of the note and the date of the note. We also explicitly specify where the note lands in relation to the rest of the admission, e.g., "Day 1 of (On Admission)". Although we separately fine-tune each model, we still include custom instructions. The baseline non-guided instruction is: "Generate the BRIEF HOSPITAL COURSE summary." The prompt guidance instruction is: "Generate the BRIEF HOSPITAL COURSE summary using only the medical entities (PROBLEMS, TREATMENTS, and TESTS) provided." " Thee guidance (list of ESGs grouped by semantic type) is appended to the source notes. SPEER's instruction is: "Retrieve subset of the medical entities in double brackets ({ and use them to generate the next sentence of the BRIEF HOSPITAL COURSE summary." The line: "#u# BRIEF HOSPITAL COURSE: nr is appended to the end of the input and is an indicator to the model to start generating the summary.
Datasets. We train on a single dataset and evaluate on three diverse held-out sets. Train- ing. We train on "167k in-patient hospital admissions from a large metropolitan hospital from 2020-2023. It is highly diverse in terms of patient population and care setting: emer- gency, surgery, obstetrics, pediatrics, etc. Testing. We evaluate on held out portion of 1,000 admissions in the same lime-frame& as the training set: CUIMC:2020-2023,, as well as admissions from earlier time period: CUIMC:2010-2014, in which the Electronic Health Record (EHR) system was different. As result, note templates and titles may differ. By training on notes with one set of EHR templates and testing on both seen and unseen styles, we can test robustness of methods to subtle shifts in style and content organization. MIMIC We also evaluate on a held-out set of 900 examples (source notes plus extracted reference summaries for evaluation) from publicly available MIMIC-III clinical notes with pre-processing from Adams al. (2022). Table 5 shows high-level statistics for the train-test splits. In contrast to other commonly used clinical NLP benchmarks (Chen et al., 2022; Gao et al., 2023), the hospital-course summarization stands out as longitudinal, multi-document, and lengthy. On average, the inputs contain from 25.5 (CUIMC) to 162.7 (MIMIC) source notes, which are synthesized into lengthy (11-37 sentence) summaries. Identifying salient, non-redundant content amounts to finding many needles in many haystacks, and then
16

---

Published as a conference paper at COLM 2024
de-duplicating to ensure that each needle is unique. MIMIC admissions, average, contain a substantially higher number of source notes than CUIMC, while also having very long reference summaries (37 sentences). Despite the long inputs, Adams et al. (2022) reveal much of the content MIMIC reference summaries is not mentioned anywhere in the source notes due to aecompleteness-whiee reduces scores on reference-based metrics.
0.95
0.90
Mistral-7B-vo.1
zephyr-7b-beta
Validation Loss
0.85
0.80
0.75
0.70
0.65
2000
4000
6000
8000
Training Step
Figure 5: Validation Loss for Mistral-7B-vo.1 and zephyr-7b-beta as a function of training
steps across 1 epuch (covering "167k hospital admissions).
Training Details. ESG Content Selection. We initialize the token-level en- 279 million parameter encoder-only model:
coder described in $3 from wrrrobrrta-oonnooomr----eeee -a00----------44,,,,,, which is a Longformer initialized with weights from XERM (Conneau et al., 2019) without any additional tuning. For modeling ESGs, we use a randomly initialize BERT encoder layer with the same configuration as XLM-RoBERTa. As described in $3, ESGs are sorted by inverse frequency and frequency rank embeddings are added to the representations of each ESG before passing to the modeling layer. We learn 020 unique ranks and, the source notes include more than 1,02 ESGs, we truncate to 024. We train on batches of 16 with AdamW optimizer for 100k using scheduled learning rate (maximum 3e with linear warmup of 1000 steps, followed by linear decay). Weight decay of 5e 5 is used. The entity boundary tags (<e>, </e>) are added as special tokens to the tokenizer and the embeddings are updated during fine-tuning. Abstractive Summarization. We fine-tune in two stages. The first stage involves training the baseline non non-guided model for 1 epoch. For the second stage, we narther-finetun the baseline and
models for extra 4,000 steps. We break training into two stages to reduce total training time since the weights from first stage re-used multiple times during the second stage. Initial Fine-Time. We fully fine-tune and Rephyr-7B-P using the baseline instructions for 1 epoch with batch size of 16 and learning rate of 5e 6. We used the Adam 8-bit optimizer with cosine learning rate scheduler. To fit the model onto two Nvidia A6000 48GB GPUs, we use DeepSpeed Stage 2 (Rasley al., 2020), FlashAttentions (Dao, 2023), bfloat16 (BF16) precision, gradient checkpointing, and train on per device batches of size 1 with gradient accumulation. We used oracle section filtering to ensure that no training examples exceeded 8,192 tokens. epoch took 10 days to complete. We computed a validation loss after every 500 steps of training and, in Figure 5, show a smooth loss curve. Further Fine-Time Tume. We further fine-tune all variants: baseline and entity-guided models for additional 4,000 steps from the initial fine-tuned weights. Performance plateaus after 2500 additional fine-tuning steps. As such,
Mistral-7B-Instrune
checkpoints every 500 steps between steps 2500 and
17

---

Published as a conference paper at COLM 2024
Mistral Instruction Templates
Non-Guided
Guided
SPEER

 Generate the BRIEF




Generate the BRIEF
Retrieve a subset of the
HOSPITAL COURSE summary.
HOSPITAL COURSE
medical entities double brackets (( J) and use them
summary using only the medical entities
generate the next sentence of the BRIEF HOSPITAL COURSE summary.
Title: Admission Note
(PROBLEMS,
TREATMENTS, and TESTS) provided.
DATE: 1/1/2024 NOTE ORDER: of DAY: 1 of
### Title: Admission Note
### Title: Admission Note
ON DAY OF ADMISSION
DATE: 1/1/2024
DATE: 1/1/2024
NOTE ORDER: of 2 DAY: 1 of
NOTE ORDER: of 2
HPI:
DAY: 1 of
pt is a 90yr old w HTN
ON DAY OF ADMISSION
ON DAY OF ADMISSION
### Title: Progress Note
HPI:
HPI:
is a 90yr old w HTN
is a 90yr old w (TN }}
DATE: 1/2/2024
NOTE ORDER: 2 of DAY: 2 of
### Title: Progress Note
### Title: Progress Note
ON DAY OF DISCHARGE
DATE: 1/2/2024
DATE: 1/2/2024
NOTE ORDER: 2 of 2 DAY: 2 of
NOTE ORDER: of 2
Plan:
DAY: 2 of
pt deemed stable for discharge ACE
ON DAY OF DISCHARGE
ON DAY OF DISCHARGE


Plan:
Plan:
### BRIEF HOSPITAL
deemed stable for discharge on ACE
deemed stable for discharge on {{ ACE }}
COURSE:


### ENTITIES
### BRIEF HOSPITAL
PROBLEMS:
COURSE:
HTN; Hypertension
TREATMENTS:
ACE; ACE inhibitors
TESTS:
[/NNS]]
### BRIEF HOSPITAL
COURSE:
Figure 6: Instruction Template for Mistral, which follows the syntax used during the original instruction tuning for Mistral-7B-ve.1.
4000 (inclusive), and report the average metric scores across this range of 4 checkpoints. We do this for robustness as there is considerable random variance for metrics across checkpoints.
Generation Config. We use greedy decoding and to mitigate the problem of repetition, set a repetition penalty tyree-parameter of 1.1 (Keskar et al., 2019). We set the number of minimum tokens to be 4 and maximum new tokens to be 1, 024 for the non-guided and prompted guided models. Since SPEER must generate sentence-level plans, we double the maximum new tokens from 024 to 2, 0,848
18

---

Published as a conference paper at COLM 2024
Zephyr Instruction Templates
Non-Guided
Guided
SPEER






Generate the BRIEF
Generate the BRIEF
Retrieve subset of the medical entities in double brackets {I } and use them generate next
HOSPITAL COURSE summary.<ls>
HOSPITAL COURSE
summary using only the medical entities


### Title: Admission Note
(PROBLEMS,
sentence of the BRIEF
TREATMENTS, and TESTS)
HOSPITAL COURSE
DATE: 1/1/2024
provided.<88 <|s>
summary.<ls>
NOTE ORDER: of DAY: 1 of



 ### Title: Admission Note
### Title: Admission Note
ON DAY ADMISSION
DATE: 1/1/2024
DATE: 1/1/2024
HPI:
NOTE ORDER: of
NOTE ORDER: of 2
is a 90yr old HTN
DAY: 1 of
DAY: 1 of
DAY OF ADMISSION
DAY OF ADMISSION
### Title: Progress Note
HPI:
HPI:
DATE: 1/2/2024
pt is a 90yr old w HTN
pt is a 90yr old w {{ [TN }
NOTE ORDER: of DAY: 2 of
### Title: Progress Note
### Title: Progress Note
ON DAY OF DISCHARGE
DATE: 1/2/2024
DATE: 1/2/2024
Plan:
NOTE ORDER: of
NOTE ORDER: of
deemed stable for discharge on ACE
 

DAY: 2 of
DAY: 2 of
ON DAY OF DISCHARGE
DAY OF DISCHARGE
BRIEF HOSPITAL COURSE:
Plan:
Plan:
pt deemed stable for discharge ACE ### ENTITIES
pt deemed stable for discharge on ACE })
 <jassistant|>
PROBLEMS:
### BRIEF HOSPITAL
HTN; Hypertension
COURSE:
TREATMENTS:
ACE; ACE inhibitors
TESTS:

<lassistant]>
### BRIEF HOSPITAL
COURSE:
Figure 7: Instruction Template for Zephyr, which follows the syntax used during the original instruction tuning for zephyr-7b-beta.
Evaluation Metrics. We rely two entity-based overlap metrics: (SGR) and the Hallucination Rate (HR). Some concepts in clinical reference summaries are not present in source notes, as noted by Shing al. (2021) and Adams al. (2022). Unsupported-or hallucinated-reffrencc content should not be included when computing entity overlap. As such, instead of directly computing overlap between reference and entities, we separately align summary entities to entities in the sources notes, and measure overlap between source-aligned entities. Specifically, we align reference and model-generated entities to subset of ESGs from the source notes: (ESGref-src) and (E.SGmodel-scck Then compute source-grounded Recall (SGR)
Recall
19

---

Published as a conference paper at COLM 2024
(ESGre r } n [ESG
SGR
| seeGr
SGR does not explicitly capture entity-based faithfulness. For this, we define the halluci- nation rate (HR) as the fraction of source entity synonym:
entity mentions which do not have
HR HENTITY, MANTIIYYmdddl-ssee mord {ENTITY mode!
where [ENTITYY morel H| denotes the number of predicted entity mentions which do not have corresponding source synonym. HR uses entity mentions and not ESGs (as in SGR) in order to penalize multiple hallucinations of the same synonym group. We report the number of tokens (# of Tokens) account for length biases in the metrics.
More broadly, we capture faithfulness at both the summary-level with BERTScore-Precision (BSP) (Zhang al., 2019) and, at the sentence level, with ClinDistill (Adams et al., 2023d)- a state of the art sentence-level faithfulness metric for hospital-course summarization. ClinDistill is regression model which is distilled from an ensemble of several pre-existing state of the art faithfulness metrics. It predicts raw, unnormalized score for each sentence, whose mean is roughly zero. We use BERTScore-Preeisson (BSP) (Zhang al., 2019), which measures the degree to which summary tokens are well-aligned to at least one token in the source notes, rather than BFRTSccore-Fi because was shown to correlate better to fine- grained expert annotations for the faithfulness of hospital course summaries (Adams et al., 2023d). Specifically, we report the BSP between the model-generated summary and the source notes. We compute contextualized embeddings for each token with the encoder from Lellenai/led-aagee 1a38g--668844 and follow Adams et al. (2023d) in using just the final hidden state the representation for each token.
We also report ROUGE-1 and ROUGE-2 scores despite a known inverse relationship between ROUGE score and faithfulness for clinical summarization (Adams et al., 2023c). This negative correlation has to do with unsupported content in references (Adams et al., 2022). Unfaithful models will mimic unsupported content in references, while faithful models tend to stick to what is explicitly stated and, as such, be penalized by ROUGE.
20


Reviews:
[{"review_id": "xJnk0lIvre", "reviewer": "Reviewer_dDBd", "summary": "The authors address the challenge of generating comprehensive and accurate hospital discharge summaries using LLMs. Fine-tuning open-source LLMs (Mistral-7B-Instruct and Zephyr-7B-β) on nearly 170k patient records revealed issues with incomplete and unfaithful summaries. To improve performance, the authors developed SPEER (Sentence-level Planning via Embedded Entity Retrieval), which uses a smaller, encoder-only model to predict salient entities and guide the LLMs. This method marks entity spans with special tags, enhancing content coverage and faithfulness. The proposed method demonstrated improved performance over non-guided and guided baselines on three diverse datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your very thorough review!  We believe your concerns are all addressable and have itemized our responses below.\n\n**1. Coherence not evaluated**\n\nSPEER summaries are generated in-order in a single pass. We return the summary as-is without the plan tokens. We will remove the word concatenation since it falsely suggests independence. We did not evaluate coherence because ordering is relatively unimportant for the task. Despite high abstraction, summaries are written in bullet style without a meaningful order. Consequently, coherence metrics don’t work well (arxiv.org/abs/2105.00816).\n\n\n**2. ROUGE is inadequate**\n\nWe include ROUGE given its historic popularity, yet downplay it in S4 for “its inverse relationship with faithfulness for clinical summarization” (arxiv.org/abs/2305.07615). We emphasize faithfulness/coverage, by which SPEER is a clear winner.\n\n**3. Simple Extraction is Enough**\n\n- Extraction does not work, unfortunately. **1) Messy Inputs.** Most notes are not written in coherent sentences, yet the summaries are. Splitting creates many one word and full note sentences. Fusion, compression, re-writing are essential. Extractive doesn't always mean  faithful (arxiv.org/abs/2209.03549). **2) Redundancy.** A salient ESG may appear 1000s of times in source: it is not enough to extract a sentence at random. Picking the right context to include requires long-context reasoning. We will add an example with an oracle extract.\n\n**4. BSP / ClinDistill Not Suitable**\n\n- BSP’s correlation to experts is similar to supervised methods on this task (Table 12 in arxiv.org/abs/2303.03948). Re ClinDistill’s generalizability, we find LLM errors are more egregious than smaller E-D models. We found these models make more non-patient specific claims (obvious extrinsic hallucinations).\n\n**5. ESG Precision**\n- We implemented ESGs because simpler methods were bad and not robust to misspellings, acronyms, etc. We will show that embedding clustering is more reliable than exact-match or ontology-based alignment (arxiv.org/abs/2208.08408).\n\n**6. Significance**\n\n- We will include CI.\n\n**7. Human evaluation**\n- We are finishing a large-scale human evaluation this week. Results show meaningful LIKERT improvement of SPEER over baseline: **+0.38 for completeness** and **+0.19 for faithfulness**, which agrees with our metrics.\n\n**8. No examples**\n\n- We cannot legally publish raw summaries. Our clinicians are working on de-identification and annotation for the camera ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper focuses on a critical topic in the medical domain and the proposed method is helpful in guiding future work in generating more faithful and human-quality level long-document summaries.\n\n2. The paper clearly pointed out the challenge of selecting salient information from long clinical documents.", "reasons_to_reject": "1. Since the final summary is the concatenation of generated summary sentences, the author provided method might affect the coherence of the final summary. However, the authors do not evaluate the coherence of the summary or mention the importance of coherence in this specific application.\n\n2. Given that the reference summary may contain information that cannot be supported by the source document (as mentioned by the author, especially on MIMIC), it is not clear that an increase in ROUGE score by author's proposed methods mean better performance.\n\n3. For the remaining auto-evaluation metrics, it looks like a simple baseline may also achieve a high performance. For example, suppose a model simply extracts a sentence (can extract short sentences on purpose to make the final summary short or close to the length of other model-generated summaries) that has entity mentions from each ESG based on the source document and then concatenates these extracted sentences together. This final summary would get a Source-Grounded Recall (SGR) of 1 and a Hallucination Rate (HR) of 0. Similarly, this summary would get a BertScore and ClinDistill close to 1 since all sentences are directly extracted from the source document. Would this final summary be a good one? (assuming ROUGE do not reflect the quality of LLM-generated summaries (https://arxiv.org/pdf/2209.12356, https://arxiv.org/pdf/2301.13848)).\n\n4. The authors claim that BSP and ClinDistill are SOTA metrics for faithfulness evaluation. However, BSP do not correlate well with human judgements and it is not clear whether ClinDistill is suitable to evaluate summaries generated by LLMs as the model is initially being evaluated over non-LLM generated summaries and errors from LLM-generated summaries might be harder to detect and hence lower correlation.\n\n5. It is not clear regarding the accuracy of forming entity synonym groups (ESG). This makes the evaluation results related to ESG not convincing enough.\n\n6. There is a lack of significant tests in the main tables. It is not clear whether some improved results are significant.\n\n7. There is no human evaluation of the quality of model generated summaries. Are summaries generated by the author's proposed method more coherent, readable, and factual? Do humans prefer summaries generated by author proposed methods?\n\n8. There are no example outputs of generated summaries from different models so the reviewer cannot develop an overall feeling about the quality of those generated summaries.", "questions_to_authors": "1. The reviewer does not find the average total length of source documents for each example.\n\n2. The mathematical formula for HR misses “|” on the right side.", "extra_scores": {}}, {"review_id": "oGSPZeHfG0", "reviewer": "Reviewer_HPd5", "summary": "This paper presents an approach for generating clinical note summaries (in particular, discharge reports). To overcome challenges in the lengthy source text, the authors first identify important entities and then retrieve the relevance context during the generation process on the basis of these entities. Automated assessments show this process is stronger than non-guided generation using existing LLMs.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank your for the thoughtful and positive review!\n\nWe appreciate that you found the SPEER method to be “new”, “interesting”, generalizable (“beneficial in other domains”), and relevant (“clearly fits the remit of COLM”). You also agree with the focus on faithfulness and completeness, and found the empirical results to be “strong”.  Given enthusiasm for our work’s novelty, relevance, and strong results, it appears that your score (5) does not align. We responded to your concerns in an itemized list (see below), which are all easily addressable for the camera ready. We hope that your minor concerns are resolved and that you reconsider your score in light of the overall positivity of your review.\n\n**1. Reproducibility**\n\n- We agree with your suggestion to present SPEER more formally.  We will re-write the methods section such that it contains the precise notation necessary to reproduce it step-by-step.\n\n**2. No human eval**\n\n- We are finishing a large-scale human evaluation this week, which will be included in the camera ready. Preliminary results show meaningful LIKERT improvement of SPEER over the baseline: **+0.38 for completeness** and **+0.19 for faithfulness**, which agrees with our metrics. We will dedicate the extra page for camera ready to human eval and qualitative analysis of SPEER versus baseline summaries.\n\n**3. Limited related work**\n\n- We wrote a long related work section but it was *mistakenly* commented out during submission. We will add it back for camera ready. While entity-guided summarization is not new (GSum - https://arxiv.org/abs/2010.08014, FROST - https://arxiv.org/abs/2104.07606), our method is novel along several dimensions: 1) forming connected subgraphs of entities to reduce sparsity; 2) embedding the guidance in the source documents to mitigate hallucinations; 3) adding retrieval-based planning.\n- “Attend to Medical Ontologies: Content Selection for Clinical Abstractive Summarization” is a great paper and we will add it. Their work differs in that they use entities to perform ontology-aware word representations, whereas we use entities as a guide for sentence-level planning, which is more suitable for the LLM era. Similarly to GSum, Searle et al (https://arxiv.org/abs/2211.07126) guide an abstractive model with clinical concepts by adding a separate, ontology-specific encoder stream. They include all identified concepts. SPEER adds entity merging, filtering, embedding, and retrieval.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The healthcare domain is important and an area that could benefit a lot from LLMs\n- From what I can tell, the method is new and somewhat interesting. A similar approach could potentially be beneficial in other domains.\n- The evaluation focused on achieving high recall and a low hallucination rate -- both of which are critical to get right in this domain\n- The empirical results are strong\n- The work clearly fits the remit of COLM", "reasons_to_reject": "- The main algorithm was described imprecisely (only an English-language description and a figure). I am not certain that I would be able to reproduce the process without a clearer description.\n - The evaluation was limited to automatic assessment of the reports. It would be stronger if (at least a sample) was assessed by professionals to ensure the suitability of the reports (even if the human assessment focused on the recall and hallucination, as the automated process did)\n - The relationship between this work and prior art was not sufficiently described, especially WRT other approaches that focus on entity guidance (more below).", "questions_to_authors": "- There's prior work in guiding clinical note generation with salient entities (e.g., https://www.semanticscholar.org/paper/Attend-to-Medical-Ontologies%3A-Content-Selection-for-Sotudeh-Goharian/4c3b5f8db4f44ed4d24e15227a0da30f7c20a665?utm_source=direct_link). This work seems substantially different than those due to the focus on planning, but they probably ought to be discussed as prior related work.\n - Consider using an algorithm block to describe the algorithm more clearly and precisely.", "extra_scores": {}}, {"review_id": "Mzpm6Hqf4w", "reviewer": "Reviewer_ZNgw", "summary": "This paper introduces SPEER (Sentence-Level Planning via Embedded Entity Retrieval), a method for improving the coverage and faithfulness of long clinical summaries generated by fine-tuned large language models (LLMs). I view this work as a unique and important contribution to real-world clinical research and practice. The method and insights can potentially be generalized to other long-form summarization tasks.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for you thoughtful and positive review! We are glad that you find SPEER to be a **unique and important contribution to real-world clinical research and practice**.  We hope that you will help champion the paper during the discussion period.\n\nWe'll address your comments below:\n\n**1. Limited related work**\n\n- We wrote a long related work section but it was *mistakenly* commented out during submission. We will add it back for camera ready. While entity-guided summarization is not new (GSum - https://arxiv.org/abs/2010.08014, FROST - https://arxiv.org/abs/2104.07606), our method is novel along several dimensions: 1) forming connected subgraphs of entities to reduce sparsity; 2) embedding the guidance in the source documents to mitigate hallucinations; 3) adding retrieval-based planning. We also discuss other more recent work on summarization with LLMs, e.g., https://arxiv.org/abs/2301.13848.\n\n**2. No efficiency analysis**\n\n- This is an important consideration, especially for real-world deployment.  Given that content to selection is performed with a very small encoder-only model, we expect it to have very minimal overhead compared to auto-regressive generation with LLMs.  We can also add scaling experiments on model size and compute (training steps).\n\n**3. No human eval**\n\n- We are finishing a large-scale human evaluation this week, which will be included in the camera ready. Preliminary results show meaningful LIKERT improvement of SPEER over the baseline: **+0.38 for completeness** and **+0.19 for faithfulness**, which agrees with our metrics. We will dedicate the extra page for camera ready to human eval and qualitative analysis of SPEER versus baseline summaries.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Fine-tuning open-source LLMs (Mistral-7B and Zephyr-7B) on a large-scale dataset for long-form clinical summarization and evaluating on three diverse test sets. This demonstrates the challenges LLMs face in generating complete and faithful summaries.\n2. Showing that content selection for this task should be treated separately from content realization. A dedicated, smaller content selection classifier outperforms the implicit content selection of auto-regressive LLM decoding.\n3. Proposing SPEER, a method that marks salient entities with special tags in the source text and instructs the LLM to explicitly retrieve these entities before generating each summary sentence. This sentence-level planning improves coverage of salient content and faithfulness compared to non-guided and prompt-guided LLM baselines.\n4. The code and data release will be a valuable contribution to the entire community.", "reasons_to_reject": "1. It would benefit from adding a section for related works, including state-of-the-art LLM summarization methods.\n2. It would benefit from an efficiency analysis (e.g., time efficiency, scale-up analysis).", "questions_to_authors": "1. The current Hospital-Course Summarization seems to belong to a related work section.\n2. A human/expert evaluation should be considered, especially regarding faithfulness analysis.", "extra_scores": {"ethics_concerns_details": "N/A"}}, {"review_id": "e8DBw0fV0X", "reviewer": "Reviewer_3yvc", "summary": "This paper presents a LLM-based method, named SPEER (Sentence-level Planning via Entity Retrieval), to generate hospital-course summaries from clinical notes (admission to discharge notes). The main innovation is to identify entity synonym groups from from the notes, identify the salient entities using classification among them to perform sentence-level guiding of the LLM to generate sentences that include the salient entities. They fine-tune open-source LLMs (Mistral-7B-Instruct and Zephyr-7B) for the task. They show that SPEER improves the coverage of salient entities and faithfulness over non-guided and guided LLM baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful and positive review!  We are glad that you found SPEER to be a \"well-reasoned approach\" showing \"good performance\" based on a \"well-executed study for a difficult clinical NLP task\".  We also appreciate your comments on our \"extensive\" evaluation and ablations.\n\nWe believe your concerns are addressable and will address them below.\n\n## Concerns\n\n**1. Novelty**\n\n- While planning, especially in the LLM age is a well-studied topic, our approach (SPEER) is novel along several dimensions. The first is that we perform planning by embedding-then-retrieving salient concepts into the source notes. While simple, the effect of explicitly grounding the plan in the source notes notably improves faithfulness (as shown in our Ablations).  We believe simplicity is a virtue and not a sign of limited novelty. We did not find any related literature which incorporates embedding-then-retrieval into planning.  We also would argue that forming extractive units from fully connected entity subgraphs might not be exciting to all, but is novel and, in practice, highly effective.\n\n\n**2. Other Biomedical Tasks**\n\n- We agree that adding additional tasks helps.  Given limited compute, we were unable to perform fine-tuning for more than one task.  In total, across Zephyr and Mistral, the training runs took approximately 30-40 GPU days across 2 NVIDIA RTX A6000 GPUs.  We kindly ask the reviewer to not hold this against us.  We believe that testing on 3 diverse EHR datasets demonstrates SPEER's robustness on this critical task.\n\n## Questions\n\n**1. Noisy References**\n\n- The references are highly noisy and frequently contain content unsupported by the source notes.  Please see this paper for more quantitative analysis of the issue of noisy references for datasets extracted from real-world Electronic Health Records. Noise in references explains why we focus our findings on faithfulness and **source-grounded** recall (SGR) over ROUGE, which we admit in Section 4 is poorly suited for the task.\n\n**2. Clustering**\n\n- Clustering is necessary because ontologies are highly sparse and ontology linkers (to the UMLS) make frequent errors. The reliability of open source tools on real-world EHR datasets is a big issue for clinical NLP in general.  As such, we found that embedding based clustering with a specialized encoder substantially outperforms any method which involves brittle ontology linking. We will explain this in the camera ready and show an example!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A well-reasoned approach and a well-executed study for a difficult clinical NLP task, demonstrating good performance.\n- Extensive evaluation and ablations.", "reasons_to_reject": "- The main contribution is the sentence-level planning aspect, which may be considered to be of limited novelty to this community.\n- It would have been interesting to assess the effect of the method on several biomedical summarization tasks to show generalizability.", "questions_to_authors": "- The ground truth references are extracted from discharge summaries. How noisy are these references in terms of content?\n- They cluster related terms using embeddings. What would be the effect of using simple entity linking instead of identifying entity synonym groups? \n-The number of entity spans in reference summaries seems too low (30; Table 1). Is this correct?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the task of clinical summarization and how well fine-tuned LLMs can do on it.  They do not select the right content, so this paper augments them with a separate classification module to decide what entities to include in the summary. These entities are incorporated via a sentence-level grounding, which is a novel (to my knowledge) method of entity guidance/summary planning. Results show similar ROUGE values, but better entity coverage and reduced hallucination, which are two important metrics in this domain.

Reviewers found the problem to be worthwhile and timely, and the method interesting.  Reviewers raised concerns about aspects including the novelty, situatedness with respect to related work, evaluation, and presentation of the work. However, in the discussion, these points have largely been addressed.  The paper would benefit from related work, the human evaluation, and a more formal description of the method being added in the next version, as the authors have promised to do.
Decision: Accept