SlowGuess/NAACL_2021_OCR huggingface dataset:
Griffin Adams<sup>1</sup>, Emily Alsentzer<sup>2</sup>, Mert Ketenci<sup>1</sup>, Jason Zucker<sup>1</sup>, Noémie Elhadad<sup>1</sup>
<sup>1</sup>Columbia University, New York, NY
$^{2}$ Harvard-MIT's Health Science and Technology, Cambridge, MA
{griffin.adams, mert.ketenci, noemie.elhadad}@columbia.edu emilya@mit.edu, jz2700@cumc.columbia.edu
Summarization of clinical narratives is a long-standing research problem. Here, we introduce the task of hospital-course summarization. Given the documentation authored throughout a patient's hospitalization, generate a paragraph that tells the story of the patient admission. We construct an English, text-to-text dataset of 109,000 hospitalizations (2M source notes) and their corresponding summary proxy: the clinician-authored "Brief Hospital Course" paragraph written as part of a discharge note. Exploratory analyses reveal that the BHC paragraphs are highly abstractive with some long extracted fragments; are concise yet comprehensive; differ in style and content organization from the source notes; exhibit minimal lexical cohesion; and represent silver-standard references. Our analysis identifies multiple implications for modeling this complex, multi-document summarization task.
The electronic health record (EHR) contains critical information for clinicians to assess a patient's medical history (e.g., conditions, laboratory tests, procedures, treatments) and healthcare interactions (e.g., primary care and specialist visits, emergency department visits, and hospitalizations). While medications, labs, and diagnoses are documented through structured data elements and flowsheets, clinical notes contain rich narratives describing the patient's medical condition and interventions. A single hospital visit for a patient with a lengthy hospital stay, or complex illness, can consist of hundreds of notes. At the point of care, clinicians already pressed for time, face a steep challenge of making sense of their patient's documentation and synthesizing it either for their own decision making process or to ensure coordination of care (Hall and Walton, 2004; Ash et al., 2004).
Automatic summarization has been proposed to support clinicians in multiple scenarios, from
making sense of a patient's longitudinal record over long periods of time and multiple interactions with the healthcare system, to synthesizing a specific visit's documentation. Here, we focus on hospital-course summarization: faithfully and concisely summarizing the EHR documentation for a patient's specific inpatient visit, from admission to discharge. Crucial for continuity of care and patient safety after discharge (Kripalani et al., 2007; Van Walraven et al., 2002), hospital-course summarization also represents an incredibly challenging multi-document summarization task with diverse knowledge requirements. To properly synthesize an admission, one must not only identify relevant problems, but link them to symptoms, procedures, medications, and observations while adhering to temporal, problem-specific constraints.
Our main contributions are as follows: (1) We introduce the task of hospital-course summarization; (2) we collect a dataset of inpatient documentation and corresponding "Brief Hospital Course" paragraphs extracted from discharge notes; and (3) we assess the characteristics of these summary paragraphs as a proxy for target summaries and discuss implications for the design and evaluation of a hospital-course summarization tool.
Summarization of clinical data and documentation has been explored in a variety of use cases (Pivovarov and Elhadad, 2015). For longitudinal records, graphical representations of structured EHR data elements (i.e., diagnosis codes, laboratory test measurements, and medications) have been proposed (Powsner and Tufte, 1997; Plaisant et al., 1996). Interactive visualizations of clinical problems' salience, whether extracted from notes (Hirsch et al., 2015) or inferred from clinical documentation (Levy-Fix et al., 2020) have shown promise (Pivovarov et al., 2016; Levy-Fix, 2020).
Most work in this area, however, has focused on
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4794-4811 June 6-11, 2021. ©2021 Association for Computational Linguistics
clinical documentation of a fine temporal resolution. Traditional text generation techniques have been proposed to synthesize structured data like ICU physiological data streams (Hunter et al., 2008; Goldstein and Shahar, 2016). Liu (2018) use a transformer model to write EHR notes from the prior 24 hours, while Liang et al. (2019) perform disease-specific summarization from individual progress notes. McInerney et al. (2020) develop a distant supervision approach to generate extractive summaries to aid radiologists when interpreting images. Zhang et al. (2018, 2020); MacAvaney et al. (2019); Sotudeh Gharebagh et al. (2020) generate the "Impression" section of the Radiology report from the more detailed "Findings" section. Finally, several recent works aim to generate EHR notes from doctor-patient conversations (Krishna et al., 2020; Joshi et al., 2020; Research, 2020). Recent work on summarizing hospital admissions focuses on extractive methods (Moen et al., 2014, 2016; Liu et al., 2018b; Alsentzer and Kim, 2018).
3 Hospital-Course Summarization Task
Given the clinical documentation available for a patient hospitalization, our task of interest is to generate a text that synthesizes the hospital course in a faithful and concise fashion. For our analysis, we rely on the "Brief Hospital Course" (BHC), a mandatory section of the discharge note, as a proxy reference. The BHC tells the story of the patient's admission: what was done to the patient during the hospital admission and why, as well as the follow up steps needed to occur post discharge, whenever needed. Nevertheless, it is recognized as a challenging and time consuming task for clinicians to write (Dodd, 2007; UC Irvine Residency, 2020).
To carry out our analysis, we construct a large-scale, multi-document summarization dataset, CLINSUM. Materials come from all hospitalizations between 2010 and 2014 at Columbia University Irving Medical Center. Table 1 shows summary statistics for the corpus. There are a wide range of reasons for hospitalizations, from life-threatening situations (e.g., heart attack) to when management of a specific problem cannot be carried out effectively outside of the hospital (e.g., uncontrolled diabetes). This contributes to the high variance in documentation. For reference, Table 7 provides a comparison of basic statistics to widely used summarization
Global	Variable	Value	STD
# Patients	68,936
# Admissions	109,726	N/A
# Source Notes	2,054,828
Per
Adm.	Length of Stay	5.8 days	9.0
# Source Notes	18.7	30.1
# Source Sentences	1,061.2	1,853.6
# Source Tokens	11,838.7	21,506.5
# Summary Sentences	17.8	16.9
# Summary Tokens	261.9	233.8
Per
Sent.	# Source Tokens	10.9	12.4
# Summary Tokens	14.5	11.5
Ratio	Word Compression	42.5	164.6
Table 1: Basic Statistics for CLINSUM. Value is the total for Global, and average for 'Per Admission' and 'Per Sentence'. STD is standard deviation.
datasets. Relatively speaking, CLINSUM is remarkable for having a very high compression ratio despite having long reference summaries. Additionally, it appears highly extractive with respect to fragment density (we qualify this in Section 4.1).
Based on advice from clinicians, we rely on the following subset of note types as source documents: "Admission", "Progress", and "Consult" notes. The dataset does not contain any structured data, documentation from past encounters, or other note types (e.g., nursing notes, social work, radiology reports) (Reichert et al., 2010). Please refer to Appendix A for more details and rationale.
3.2 Tools for Analysis
Entity Extraction & Linking. We use the MedCAT toolkit (Kraljevic et al., 2020) to extract medical entity mentions and normalize to concepts from the UMLS (Unified Medical Language System) terminology (Bodenreider, 2004). To exclude less relevant entities, we only keep entities from the Disorders, Chemicals & Drugs, and Procedures semantic groups, or the Lab Results semantic type.
Local Coherence. We examine inter-sentential coherence in two ways. Next-Sentence Prediction (NSP). Since we compare across a few datasets representing different domains, we use domain-specific pre-trained BERT models via HuggingFace (Wolf et al., 2019): "bert-base-cased" for CNN/DM and Arxiv, "monologg/biobert_v1.1_pubmed" for Pubmed, and "emilysentzer/Bio_ClinicalBERT" for CLINSUM. Entity-grids. Entity-grids model local coherence by considering the distribution of discourse entities (Barzilay and Lapata, 2005). An entity grid is a 2-D representation of a text whose entries represent the presence or absence of a
discourse entity in a sentence. For our analyses, we treat UMLS concepts as entities and train a neural model, similar to Tien Nguyen and Joty (2017); Joty et al. (2018), which learns to rank the entity grid of a text more highly than the same entity grid whose rows (sentences) have been randomly shuffled. Please see Appendix B for more details.
Lexical Overlap Metric. We use ROUGE-1 (R1) & ROUGE-2 (R2) F-1 (Lin, 2004) to measure lexical overlap, while ignoring higher order variants based on analysis from other work (Krishna et al., 2021). We denote the average of R1 & R2 scores as  $R_{12}$ .
Extractive Summarization Baselines. We rely on a diverse set of sentence extraction methods, whose performance on a held-out portion of CLINSUM is reported in Table 2. Oracle models have access to the ground-truth reference and represent upper bounds for extraction. Here, we define the sentence selection criteria for each oracle variant, leaving more in-depth discussion to the subsequent analysis. ORACLE TOP-K: Take sentences with highest  $R_{12}$  vis-a-vis the reference until a target token count is reached; ORACLE GAIN: Greedily take source sentence with highest relative  $R_{12}$  gain conditioned on existing summary<sup>1</sup>. Extract sentences until the change in  $R_{12}$  is negative; ORACLE SENT-ALIGN: For each sentence in reference, take source sentence with highest  $R_{12}$  score; ORACLE RETRIEVAL: For each sentence in reference, take reference sentence from train set with largest BM25 score (Robertson and Walker, 1994); and ORACLE SENT-ALIGN + RETRIEVAL: For each sentence in reference, take sentence with highest  $R_{12}$  between ORACLE SENT-ALIGN and ORACLE RETRIEVAL. We provide two unsupervised methods as well. RANDOM: extracts random sentences until summary reaches target word count (average summary length); LEXRANK: selects the top-k sentences with largest LexRank (Erkan and Radev, 2004) score until target word count is reached. For a supervised baseline, we present CLINNEUSUM: a variant of the Neusum model adapted to the clinical genre (Zhou et al., 2018). CLINNEUSUM is a hierarchical LSTM network trained on ground-truth labels derived from ORACLE GAIN, which we detail in Appendix C.
4 Dataset Analysis & Implications
To motivate future research in multiple, self-contained directions, we distill task-specific characteristics to a few salient, standalone takeaways. For each takeaway, we provide evidence in the data and/or literature, before proposing implications of findings on model development and evaluation.
4.1 Summaries are mostly abstractive with a few long segments of copy-pasted text
tl;dr. CLINSUM summaries appear extractive according to widely used metrics. Yet, there is large variance within summaries. This directly affects the performance of a supervised extractive model, whose selection capability degrades as summary content transitions from copy-paste to abstractive. In turn, we need models which can handle abrupt transitions between extractive and abstractive text.
Background. Clinicians copy forward information from previous notes to save time and ensure that each note includes sufficient evidence for billing and insurance purposes (Wrenn et al., 2010). Copy-paste is both widely used (66-90% of clinicians according to a recent literature review (Tsou et al., 2017)) and widely applied (a recent study concluded that in a typical note,  $18\%$  of the text was manually entered;  $46\%$ , copied; and  $36\%$  imported $^{2}$  (Wang et al., 2017)). Please see Appendix D for more information on the issue of copy-paste.
Analysis - extractiveness. CLINSUM appears very extractive: a high coverage (0.83 avg / 0.13 std) and a very high density (13.1 avg / 38.0 std) (See Grusky et al. (2018) for a description of the statistics). However, we find that  $64\%$  of the extractive fragments are unigrams, and  $25\%$  are bigrams, which indicate a high level of re-writing. The density measure is large because the remaining  $11\%$  of extractive fragments are very long.
Yet, there is a strong positional bias within summaries for long fragments. Figure 1, groups fragments according to their relative order within each summary. The longest fragments are usually first. Qualitative analysis confirms that the beginning of the BHC is typically copied from a previous note and conveys the "one-liner" (e.g., pt is a 50yo male with history of CHF who presents with edema.)
This abrupt shift in extractiveness should affect content selection. In particular, when look
<sup>2</sup>Imported refers to text typically pulled in from structured data, such as a medication or problem list.
This is the Neusum model's objective (Zhou et al., 2018)
Extractive Baseline	ROUGE-1	ROUGE-2
Recall	Precision	F1	Recall	Precision	F1
RANDOM	0.16	0.24	0.17	0.04	0.03	0.03
LEXRANK	0.18	0.21	0.18	0.05	0.05	0.05
CLINNEUSUM	0.36	0.25	0.27	0.14	0.1	0.11
ORACLE TOP-K	0.28	0.52	0.32	0.16	0.32	0.19
ORACLE GAIN	0.43	0.63	0.5	0.26	0.42	0.3
ORACLE SENT-ALIGN (SA)	0.48	0.61	0.52	0.3	0.33	0.31
ORACLE RETRIEVAL	0.51	0.70	0.58	0.25	0.28	0.29
ORACLE SA + RETRIEVAL	0.6	0.76	0.66	0.4	0.49	0.43
Table 2: Performance of different sentence selection strategies on CLINSUM.
Figure 1: Average extractive fragment lengths according to their relative order within the summary.
ing at oracle extractive strategies, we should see clear-cut evidence of (1) 1-2 sentences which are easy to identify as salient (i.e., high lexical overlap with source due to copy-paste), (2) a murkier signal thereafter. To confirm this, we analyze the sentences selected by the ORACLE GAIN method, which builds a summary by iteratively maximizing the  $R_{12}$  score of the existing summary vis-a-vis the reference.
In Figure 2, two supporting trends emerge. (1) On average, one sentence accounts for roughly  $50\%$  of the overall  $R_{12}$  score. (2) Afterwards, the marginal contribution of the next shrinks, as well as the  $R_{12}$  gap between the best sentence and the minimum / average, according to the oracle.
There should also be evidence of the copy-paste positional bias impacting content selection. Table 3 reveals that the order in which the ORACLE GAIN summary is built-by maximal lexical overlap with the partially built summary-roughly corresponds to the true ordering of the summary. More simply, the summary transitions from extractive to abstractive.
Figure 2: We plot average ROUGE score as summaries are greedily built by adding the sentence with the highest relative ROUGE gain vis-a-vis the current summary, until the gain is no longer positive (ORACLE GAIN). We also include the difference between the highest scoring sentence and the average / minimum to demonstrate a weakening sentence selection signal after the top 1-2.
Unsurprisingly, a model (CLINNEUSUM) trained on ORACLE GAIN extractions gets progressively worse at mimicking it. Specifically, for each extractive step, there exists a ground-truth ranking of candidate sentences by relative  $R_{12}$  gain. As the relevance gap between source sentences shrinks (from Figure 2), CLINNEUSUM's predictions deviate further from the oracle rank (Table 4).
Analysis - Redundancy. Even though we prevent all baseline methods from generating duplicate sentences (23% of source sentences have exact match antecedents), there is still a great deal of redundancy in the source notes (i.e., modifications to copy-pasted text). This causes two issues related to content selection. The first is fairly intuitive - that local sentence extraction propagates severe redundancy from the source notes into the summary and, as a result, produces summaries with low lexical coverage. We confirm this by examining the performance between the ORACLE TOP-K and ORACLE GAIN, which represent summary-unaware and summary-aware variants of the same selection
3From Table 2, the average  $R_{12}$  score is 0.39 for ORACLE GAIN. To reconcile this number with respect to Figure 2, we note that the average oracle summary is far less than the 20 sentence upper bound shown in the chart.
Extractive
Step	Average Rank of Closest
Reference Sentence
1	4.7
2	6.0
3	6.3
4	6.7
5	7.3
>5	10.1
Table 3: ORACLE GAIN greedily builds summaries by repeatedly selecting the sentence which maximizes the  $R_{12}$  score of the partially built summary. By linking each extracted sentence to its closest in the reference, we show that this oracle order is very similar to the true ordering of the summary.
Extractive
Step	Ground Truth Rank
Average	Median
1	28	7
2	69	22
3	74	31
4	79	39
5	76	42
>5	80	60
Table 4: Rank of selected sentence vis-a-vis oracle rank at each extraction step. A perfectly trained system would have a ground-truth of 1 at each step.
method. While both extract sentences with the highest  $R_{12}$  score, ORACLE GAIN outperforms because it incorporates redundancy by considering the relative  $R_{12}$  gain from an additional sentence.
The second side effect is perhaps more surprising, and divergent from findings in summarization literature. For most corpora, repetition is indicative of salience. In fact, methods based on lexical centrality, i.e., TextRank (Mihalcea and Tarau, 2004) and LexRank (Erkan and Radev, 2004), still perform very competitively for most datasets. Yet, for CLINSUM, LexRank barely outperforms a random baseline. Poor performance is not only due to redundancy, but also a weak link between lexical centrality and salience. The Pearson correlation coefficient between a sentence's LexRank score and its  $R_{12}$  overlap with the reference is statistically significant  $(p = 0)$  yet weak  $(r = 0.29)$ .
Qualitative analysis reveals two principal reasons, both related to copy-paste and/or imported data. The first relates to the propagation of frequently repeated text which may not be useful for summaries: administrative (names, dates), imported structured data, etc. The second relates to sentence segmentation. Even though we use a cus
Figure 3: Relationship between source entity mentions and probability of inclusion in the summary.
tom sentence splitter, our notes still contain some very long sentences due to imported lists and semi-structured text-a well-documented issue in clinical NLP (Leaman et al., 2015). LexRank summaries have a bias toward these long sentences (26.2 tokens versus source average of 10.9), which have a greater chance of containing lexical centroid(s).
To bypass some of these issues, however, one can examine the link between centrality and salience at the more granular level of entities. Figure 3 shows a clear-cut positive correlation between source note mention frequency of UMLS concepts and the probability of being included in the summary.
Implications. Regarding within-summary variation in extractiveness, we argue for a hybrid approach to balance extraction and abstraction. One of the most widely-used hybrid approaches to generation is the Pointer-Generator (PG) model (See et al., 2017), an abstractive method which allows for copying (i.e., extraction) of source tokens. Another research avenue explicitly decouples the two. These extract-then-abstract approaches come in different flavors: sentence-level re-writing (Chen and Bansal, 2018; Bae et al., 2019), multi-sentence fusion (Lebanoff et al., 2019), and two-step disjoint extractive-abstractive steps (Mendes et al., 2019).
While highly effective in many domains, these approaches do not consider systematic differences in extractiveness within a single summary. To incorporate this variance, one could extend the PG model to copy pre-selected long snippets of text. This would mitigate the problem of copy mechanisms learning to copy very long pieces of text (Gehrmann et al., 2018) - undesirable for the highly abstractive segments of CLINSUM. Span-level extraction is not a new idea (Xu et al., 2020), but, to our knowledge, it has not been studied much in otherwise abstractive settings. For instance, Joshi et al.
(2020) explore patient-doctor conversation summarization and add a penalty to the PG network for over-use of the generator, yet this does not account for intra-summary extractiveness variance.
Regarding redundancy, it is clear that, in contrast to some summarization tasks (Kedzie et al., 2018), summary-aware content selection is essential for hospital course summarization. Given so much noise, massive EHR and cite-specific preprocessing is necessary to better understand the signal between lexical centrality and salience.
4.2 Summaries are concise yet comprehensive
tl;dr. BHC summaries are packed with medical entities, which are well-distributed across the source notes. As such, relations are often not explicit. Collectively, this difficult task calls for a domain-specific approach to assessing faithfulness.
Analysis - concise We find that summaries are extremely dense with medical entities:  $20.9\%$  of summary words are medical UMLS entities, compared to  $14.1\%$  in the source notes. On average, summaries contain 26 unique entities whereas the source notes contain 265 — an entity compression ratio of 10 (versus token-level compression of 43).
Analysis - comprehensive. Many summarization corpora exhibit systematic biases regarding where summary content can be found within source document(s) (Dey et al., 2020). On CLINSUM, we examine the distribution of entities along two dimensions: macro considers the differences in entity share across notes, and micro considers the differences within each note (i.e., lead bias). (1) Macro Ordering. When looking at the source notes one by one, how much additional relevant information (as measured by entities present in the summary) do you get from each new note? We explore three different orderings: (1) FORWARD orders the notes chronologically, (2) BACKWARD the reverse, and (3) GREEDY ORACLE examines notes in order of decreasing entity entity overlap with the target. Given the large variation in number of notes per admission, we normalize by binning notes into deciles. Figure 4 shows that it is necessary to read the entire set of notes despite diminishing marginal returns. One might expect the most recent notes to have the most information, considering present as well as copy-forwarded text. Surprisingly, FORWARD and BACKWARD distributions are very similar. GREEDY ORACLE gets at the level
of information concentration. On average, the top  $10\%$  of most informative notes cover just over half of the entities found in the summary. We include absolute and percentage counts in Table 5. (2) Micro Ordering. We plot a normalized histogram of summary entities by relative position within the source documents. Figure 5 reveals a slight lead bias, followed by an uptick toward the end. Clinical notes are organized by section: often starting with the past medical history and present illness, and typically ending with the plan for future care. All are needed to write a complete BHC.
Avg Notes to Read
Ordering	Number	Percent
FORWARD	8.5	0.80
BACKWARD	7.8	0.73
GREEDY ORACLE	5.0	0.50
Table 5: Number of documents necessary to cover all relevant UMLS entities—present in the summary—according to three different ordering strategies. FORWARD orders the notes chronologically, BACKWARD the reverse, and GREEDY ORACLE examines notes in order of decreasing entity overlap with the target.