Hospital Discharge Summarization Data Provenance
Year: 2023

Abstract:
Summarization of medical notes has been studied for decades with hospital discharge summaries garnering recent interest in the research community. While methods for summarizing these notes have been the focus, there has been little work in understanding the feasibility of this task. We believe this effort is warranted given the notes’ length and complexity, and that they are often riddled with poorly formatted structured data and redundancy in copy and pasted text. In this work, we investigate the feasibility of the summarization task by finding the origin, or data provenance, of the discharge summary’s source text. As a motivation to understanding the data challenges of the summarization task, we present DSProv, a new dataset of 51 hospital admissions annotated by clinical informatics physicians. The dataset is analyzed for semantics and the extent of copied text from human authored electronic health record (EHR) notes. We also present a novel unsupervised method of matching notes used in discharge summaries, and release our annotation dataset1 and source code to the community.

text: and source code to the community.

heading: 1 Introduction
text: A discharge summary is written by physicians when a patient is discharged from the hospital and provides clinicians with an overview of the patient’s hospital stay. Physicians often reference or copy EHR notes to the discharge summary. These copied notes include progress notes, consult notes, and test results. We call these previously written medical documents note antecedents since they are written prior to the discharge summary and can be used as source text as input to an automatic summarization (AS) system. The flow of information from the note antecedent to the discharge summary is traced by annotating semantically similar content and copied text across notes and sections (such as Brief Hospital Course), which can be copied or
1https://github.com/uic-nlp-lab/dsprov
paraphrased text. These annotations, called note matches, “tie” the notes’ discrete lexical spans of text. Each annotation is a text span in the discharge summary that carries a similar or identical meaning to its matching note antecedent as a span, which is a clinical medical note containing the original information that contributed to the discharge summary. In many cases, the text annotated by the note antecedents is later paraphrased or copied into the discharge summary.
Figure 1a shows a note match annotation of a physician note and the linked text in the discharge summary given in Figure 1b. In this example, the patient’s encounter is documented in the History of Present Illness section of the note antecedent and later paraphrased in the summary at the time of the patient’s discharge. The red arrow represents the connection between these two lexical spans within an admission. This connection always starts from
the original text in any clinical note and terminates in the discharge summary for the respective admission. For this work, both ends of the link are text spans written by physicians (see Section 3 for more detail on the annotation process).
Understanding the extent of meaningful data as opposed to low quality copied structured data is key in determining the feasibility and choice of methods needed to generate discharge summaries. In many instances, the copied text comes from high quality sources such as the History of Present Illness section of the physician note. However, redundancy, errors and data incoherence is pervasive in the vast amounts of medical data taken from patients during a hospital stay (Cohen et al., 2013). As much as 46% of the discharge summary is copied and pasted, 36% imported from structured data sources, leaving only 18% manually entered (Adams et al., 2021).
The extent of copied text in notes is well known, but the origins of discharge summary text is not. Since it is unclear how concepts arrive into the summary, extrapolating anything more than the measure of copied text and their semantics would necessitate understanding the decisions made by the physician while writing the summary. However, we can infer what is missing from a summarization by subtracting the portion represented by the note antecedents. Knowing what is missing from the summary and unique to the physician’s direct individual experience is fundamental for gauging the summarization performance upper bound.
For this reason, our primary goal for this work is to find and analyze the flow of data from notes written prior to the discharge summary, as shown in Figure 1. It is our position that a better understanding of the provenance of data, by note and section, is a crucial first step before AS can be successfully applied to discharge summarization future work. A secondary goal is to produce an unsupervised baseline and model for the research community.
To accomplish these goals three clinical physicians annotated admission records from the freely available Medical Information Mart for Intensive Care III (MIMIC-III) Version 1.4 (Johnson et al., 2016) corpus2. These annotations uncovered where notes overlap and offer high quality human examples for supervised learning methods. Our contributions are the annotated dataset (see Section 3), its
2Access to the MIMIC-III corpus requires creating a PhysioNet account and finishing a training course.
analysis (see Section 3.2), and a novel unsupervised method using the word mover algorithm (Kusner et al., 2015) with clustering to assist in the annotation process (see Section 4).

heading: 2 Related Work
text: Most clinical text data analysis has been in the medical field (rather than NLP) with early work in the discovery of data flow using visual summarization methods for clinical psychiatric data (Powsner and Tufte, 1997). Redundant text detection in samecategory EHR notes using latent Dirichlet allocation (Blei et al., 2003) has been explored (Cohen et al., 2013). Duplication in medical EHR notes were studied in recent work to find root causes of copied text using 10-gram token spans, which found that 58% of a physician note is copied from previous notes (Steinkamp et al., 2022).
Recent examples of cross discipline collaboration include exploring language and terminology differences between nurses and physicians (Boyd et al., 2018) and patient centric summarization (Acharya et al., 2018). SOAP (Subjective, Objective, Assessment and Plan) note generation (a type of physician note) is another example of an area of mutual interest between the NLP and medical domains (Krishna et al., 2021).
Provenance of data as it relates to the task of summarization has included multi-level recurrent neural networks (RNNs) used for extractive summarization (Zhou et al., 2018). Soon after, BART (Lewis et al., 2020) was used to abstractively condense and smooth the summaries. A large corpus was analyzed and a hybrid extractive/abstractive neural method was used to summarize the Brief Hospital Course section (Adams et al., 2021).
The summarization of MIMIC-III physician notes is perhaps the most interesting comparison and potentially most impactful (Gao et al., 2022). Clinical notes were summarized by fine-tuning the T5 (Raffel et al., 2020) and BART (Lewis et al., 2020) state of the art seq2seq models and evaluated using the BERTSCORE (Zhang et al., 2020) and ROUGE (Lin, 2004) scoring methods. In addition, co-occurring extracted Unified Medical Language System (UMLS) medical concepts (Bodenreider, 2004) were also reported. Bodenreider concludes with their intention of providing a foundation for future summarization work and acknowledgment of the challenges facing the clinical note summarization, which echoes the motivation of this work.

heading: 3 Dataset
text: Three attending physicians annotated 51 admissions from the MIMIC-III Version 1.4 (Johnson et al., 2016) corpus for overlap with the EHR note antecedents. The overlap was annotated by selecting the semantically similar portions of text from the discharge summary as shown in Figure 1. This provides statistics of overlap both at a note and a section level. To find the section overlap, we use the annotated MedSecId (Landes et al., 2022) corpus for most notes. In the few cases where note section annotations were not available, we used the pretrained MedSecId model to automatically section notes. Such sections are helpful to medical clinicians when finding topical information and useful for billing insurance companies. We direct the reader to Landes et al. for further reading on the function and utility of sections in medical notes.
The annotation selection criteria by admission was the note category and the highest note count from the MedSecId corpus. More specifically, selection SC for admission A was chosen as SC = argmaxA ∑ A∈C |A|, where C is the MIMIC-III corpus and |A| are the number of notes in the admission. Our early findings showed very little overlap of data with the exception of Consult (documentation by consulting physicians across departments) and Physician (typically written daily by the physician) notes. For this reason, admissions with these notes took priority in our annotation process to maximize note match overlap. While it could be argued that these notes’ statistics may be combined given their likeness in purpose, it was decided to keep them separate for summarization future work.
Admissions were annotated with note matches using the following process:
1. Extract SC admissions from MIMIC-III. 2. For each admission A ∈ SC :
(a) Read A’s discharge summary.
(b) For each note antecedent of the admission A, semantically similar or verbatim copied text was identified and each annotated note match annotated as:
i. A single span as character offsets tuples in the note antecedent (call it n).
ii. A single span as character offsets tuples in the discharge summary (call it d).
iii. A link between n and d.
Each note antecedent may have several note matches (as can a discharge summary), but each has a single link across both note documents.
An annotation guide was created by the lead physician annotator, who then trained the other two physician annotators. Agreement on the criteria for note matches was decided on after each annotator completed one admission. The first admission annotations were then updated per the consensus agreed upon by all annotators. The same process of annotation, discovery, and agreement repeated for three additional admissions; one for each annotator. The remaining 39 were then split among the three annotators.
The 51 admission count might give the mistaken impression of a small dataset. However, as shown in Table 1, the extent of the annotation set is comprehensive with a total of 569 note matches from 291 notes that encompassed a little over 3 million tokens and 11.7 million characters (11.65MB). The number of admissions annotated was an aspect of the time consuming nature of the task. Each admission took an average 20 minutes for the physician to review and annotate as some admissions contained up to 494 notes. However, admissions had an average of 11.16 (σ = 7.3) notes and one admission had a single note antecedent. Statistics across discharge summaries and note antecedents are separate and independent; for example the 240 note antecedent count in Table 1 does not include counts or other information reflected from discharge summaries.

heading: 3.1 Limitations
text: The MIMIC-III corpus includes a discharge summary for each admission. However, it is limited
to patient’s time in the intensive care unit (ICU), meaning that the patient’s history for any time after transfer from the department is lost. Given most patients progress to lower severity departments as they recover from intensive care, a large cross section of the patient’s notes are missing from our analysis. In Section 3.2 we discuss the statistics that justify this conclusion.

heading: 3.2 Data Analysis
text: The DSProv annotation dataset was created with the task of summarization in mind for future work. However, the dataset also provides insight into how EHR note antecedents are used by physicians to write discharge summaries and for the practicality of automatically summarizing them. This analysis provides a quantitative justification for qualitative hypotheses based on clinician’s experience writing notes with data observed during annotation.
The human annotated note match text spans were tokenized to compute overlap, ROUGEL (Lin and Hovy, 2003; Lin, 2004), BLEU (Papineni et al., 2002), and BERTSCORE (Zhang et al., 2020). Each note match annotation includes the unique MIMIC-III note antecedent and discharge summary note identifier, absolute character offset in both notes, and the section they span. An additional set of annotations were automatically generated that break spans that overlap sections.
A normalized Levenshtein edit distance (Levenshtein, 1966) was used to measure the extent of copied and pasted text. Since the distance counts the minimum number of edits, rather than a relative measure to the note match span character length, it was normalized with:
levsim(w1,w2) = 1− lev(w1,w2)
max |w1|, |w2| (1)
where lev is the Levenshtein edit distance, and |w1| and |w2| are lengths of the words in characters.

heading: 3.2.1 Note Category
text: Table 2 provides statistics and similarity measures on the DSProv annotation set. The portion columns show the token overlap between each category of note antecedent (“Ant Portion”) with the discharge summary (“DS Portion”). The “LS” is the Levenshtein edit similarity as computed with Equation 1. All similarity scores (“LS”, “BERTSCORE”, “ROUGE” and “BLEU”) are computed between the note antecedent and discharge summary span for each match.
The highest discharge summary token overlap is with physician notes (28%), which we consider surprisingly high considering the MIMIC-III corpus only includes ICU notes as mentioned in Section 3.1. We expect the other discharge summary overlap statistics to be underrepresented for the same reason. This high token overlap supports the conjecture that daily progress notes are highly summarized with little copied text. The relatively low edit distance with a high BERTSCORE further supports this conclusion as surface similarity of copied text is low but the semantic similarity is high.
ECG (electrocardiogram notes), Radiology and Echo (echocardiogram data and analysis) notes show a higher similarity (80.94, 70.1, and 65.72 respectively) with the discharge summary since they are frequently copied and pasted. However, these statistics should be higher than presented since some spans include page breaks that result in counting superfluous header and footer tokens. Also note that ECG has the highest note antecedent portion, implying that most of these short reports make it into the discharge summary.

heading: 3.2.2 Section by Discharge Summary
text: Statistics by discharge summary section are given in Table 3. Labs and Imaging (71.55%) is a highly copied and pasted section from Radiology and Echo notes. This section takes into account cultures, blood results, and lab tests, and is copied and pasted as structured data.
Hospital Course has the highest discharge summary (14%) section representation. While more analysis is needed, we believe this section has a high overlap due to transfer notes that describe patients moving between departments. These notes have an impact on summarization as they describe what happened to the patient during the hospital visit, including time of death. Generally speaking, good sections for summarization are those that have a low edit score (minimal copying and pasting) but a high similarity. In this case the comparatively low Levenshtein edit similarity (41.13) and somewhat high BERTSCORE (63.39) implies this section is a good target for AS as previously investigated in prior work (Adams et al., 2021).

heading: 3.2.3 Section by Note Antecedent
text: Table 4 shows the overlap from the perspective of the note antecedent. The Assessment and Plan section (the overall impression of the patient and how to treat them) has a high (15%) antecedent overlap. We found that the majority of the discharge summary content comes from the Brief Hospital Course section. The Echo note’s Conclusions section has a high portion of text that is summarized
from note antecedents. This section’s echocardiogram information only consists of 8% on average of the notes annotated.

heading: 4 Methods
text: Automatic methods to assist in bootstrapping the corpus were considered given the large amount of data the physician needs to sift through to find note matches3. However, the task is challenging given the pages long document lengths, and precludes transformer pre-training methods. While the task has much in common with paraphrase matching and information retrieval tasks, it is fundamentally different in the way sections of text are matched. For example in question/answer systems, a query matches to an answer in the source text. However, our task requires the correct text span in both note documents. Both the discharge summary and the note antecedent predicted text spans for the respective linked note matches are used when evaluating.

heading: 4.1 Evaluation
text: The human annotations were used for comparison since there is no previous baseline for this task. The unsupervised methods were then used to estimate the overlap portion to trace the origins of the discharge summary to the note antecedent, and then compared against the human annotations. The task is framed as a token classification task (without a label) since spans are token boundary demarcated.
3The highest note count for an admission in MIMIC-III corpus is 1,233 notes.
Our methods were evaluated using the SemEvel 2013 Task 9.1 (Segura-Bedmar et al., 2013) entity extraction scoring method since it is flexible in its strictness as a score. Given the novelty and difficulty of the task, we used the partial boundary matching metric with ROUGE as an additional reference point. However, even though the SemEval measure is flexible for partial token matching, it is not ideal since this task aims to classify single token spans on a match-by-match basis.

heading: 4.2 Word Mover
text: The word mover algorithm (Kusner et al., 2015) was used in the first step of our method. Our method frames the task as a transportation problem by using the earth mover algorithm (Pele and Werman, 2009). The intuition follows from modeling probability distributions as piles of dirt that are moved from one location to another. The algorithm treats high dimensional word embeddings of a source document as the probability distribution that “moves” words to the target document embedded distribution. The optimization algorithm minimizes the objective on words wt, . . . ,wT :
1
T T∑ t=1 ∑ j∈N (t) log p(wj |wt) (2)
where N (n) are the neighboring words, and p(wj |wt) is the word vector’s hierarchical softmax values. Word embeddings are made unit vectors and distance measures are euclidean. Our
experiments include non-contextual word embeddings (Mikolov et al., 2013a,b) and static contextual embeddings from transformer architectures (Devlin et al., 2019). Because document word frequencies are used as the histogram weight to the earth mover algorithm, surface word forms were associated with wordpiece groups (Wu et al., 2016). The centroid of each wordpiece group was used for the embeddings.

heading: 4.3 Hybrid Semantic Positional Token Clustering
text: The word mover algorithm maps words from the discharge summary onto the note antecedent efficiently, but it does not help us link the note matches. A naïve approach would be to chunk tokens based on a metric such as cosine similarity. However, words with little similarity would frequently result in too many span breaks. We still need to cluster the word embeddings computed in Section 4.3 to group concepts in each document separately. However, this still does not address the “Swiss cheese” problem of note matches with too many “holes” (span breaks). We propose to simply add a component with a normalized scaled value to the word embedding. More specifically, we defined the concatenated position vector with:
posemb(w) , [ emb(w); ( αt · i |T | )] (3)
where embi(w)s the word embedding as an application of the language model; αt is the token
position component scaler; i is the index the ith word wi, and |T | is the document token length. The higher the token position component scaler (αt) value is set, the more the word position is prioritized. This means that high values of the hyperparameter will create longer contiguous token spans, but at the cost of semantic similarity. This effect can be visually explained as a simple 2D word embedding with an additional token position axis. Figure 2 shows such a coordinate system with an example of an embedded span. On the positional axis, each token is spaced at even intervals. Because the positional components are proportionally scaled up for higher values of αt, their relative distances shrink. On the other hand, if this value is lowered, their positional components diminish, effectively reverting the word points (word location in the embedded space) to their pre-trained vectors.
Once clustering of each document’s word points is complete, each cluster’s points are assigned to note matches. For each iteration over the Cartesian product, each document’s points are added to matches by associated cluster (see Algorithm 1). The source and target documents are swapped and Algorithm 1 is run again to create flows from the target to the source. Lexical overlapping matches are combined and their flows added together and
sorted to create a ranking of matches.
Since each note match is assigned a flow (as a function of work to transport the word embedding) in both directions, they are combined as a single flow, which represents the highest similarities having the most information between the notes. Finally, matches are sorted in descending order by their flow values, so those with the maximum amount of information gain are ranked first.
Algorithm 1: Matching Algorithm Input: Documents source A and target B Output: N note match spans
1 Function MatchNoteSpans(A, B) 2 // assign word vectors and
normalize to unit
3 Va ← emb(w)||emb(w)||2 ;∀w ∈ A ; 4 Vb ← emb(w)||emb(w)||2 ; ∀w ∈ B ; 5 // assign position embeddings 6 Pa ← posemb(Va) ; 7 Pb ← posemb(Vb) ; 8 // assign word flows 9 (Fa,Fb)← WordMover(Va, Vb) ;
10 // cluster word points 11 Ca ← Cluster(Pa) ; 12 Cb ← Cluster(Pb) ; 13 // add matches 14 M← {∅} ; 15 for fa ∈ Fa do 16 for fb ∈ Fb do 17 // get cluster from flow 18 ca ← Ca[fa] ; 19 cb ← Cb[fb] ; 20 // add the match and

heading: 21 // token points
text: 22 if {(ca, cb)}not ∈M then 23 M←M∪ {(ca, cb)} ; 24 end

heading: 25 end
text: 26 end 27 returnM ; 28 end 29 Function BiMatchNoteSpans(A, B) 30 Ma→b ← MatchNoteSpans(A, B) ; 31 Mb→a ← MatchNoteSpans(B, A) ; 32 Mbi ← Sort(Ma→b,Mb→a) ; 33 end

heading: 5 Results and Discussion
text: The DSProv dataset was provided to the unsupervised algorithm described in Section 4 and evaluated against the human annotated note matches. The match spans with the highest flow (see Section 4.3) were compared and scored using the measures listed in Section 4.1. Before the evaluation, Bayesian hyperparameter optimization (Bergstra et al., 2013) was used on the human annotated dataset on a subset of the data. The model’s hyperparameters were set to the Bayesian optimized values and evaluated. Of the 569 note matches annotated, an additional 359 were optimized on 500 iterations. This process was repeated on each word embedding for each note match.
While the hybrid method explained in Section 4.3 had a relatively high SemEval partial recall of 69.06 for matching discharge summaries spans, it suffered a low precision score. This implies finding spans is not an issue, but finding correct span boundaries as more difficult. We report both the good recall but poor precision to help explain the kinds of challenges in matching spans between discharge summaries with note antecedents.
The performance for the note antecedents matching as shown Table 5 tells a better story. We see a similar pattern with a low precision, but a high recall with both netting a higher SemEval partial match harmonic mean of 15.85. Surprisingly the SapBERT (Liu et al., 2021), which models semantic relationships of biomedical domain entities, performed worse than Sentence-BERT (Reimers and Gurevych, 2019). This suggests models trained for embedding and clustering provide better embeddings for our task. The non-contextual word embeddings do not perform as well with the exception of GloVE (Pennington et al., 2014) having the best ROUGE1 for discharge summaries.
As mentioned in Section 3.1, only ICU notes and discharge summaries are provided in MIMIC-III.
This has the effect of decreasing available information to potentially summarize and also has a negative impact on results as there are fewer examples to match between note documents. However, it has an even greater impact on summarization since the machine cannot generate what is not available in the EHR. While much of this data is available as structured data that can be textually formatted, additional data provenance based methods are needed to use these information sources. For example, it is feasible to summarize sections from structured data such as tabular prescribed medicine to generate the Medication History section, which we leave as future work.

heading: 6 Conclusions and Future Work
text: We have presented DSProv, a new freely available dataset of 569 textual span matches between discharge summaries and note antecedents annotated by clinical informatics physicians. The analysis of our dataset presents new qualitative and quantitative findings of EHR notes. We have also presented a novel unsupervised method for annotating note matches using models tuned on human examples from our dataset. We believe our baseline results and our dataset analysis provide insights necessary for assessing the feasibility of traceable and faithful automatic discharge summarization in future work.
While the purpose of this work was to investigate the provenance of discharge summaries from EHR note antecedents, our secondary goal was to augment our dataset with semi-supervised annotations. Given the unsupervised baseline models have room for improvement, we leave supervised methods (such as pretrained biomedical language models fine-tuned span tagging) as future work.

heading: 7 Acknowledgments
text: This work was supported by award R01 CA225446 from the National Institutes of Health.

heading: Future Work
text: We believe our baseline results and our dataset analysis provide insights necessary for assessing the feasibility of traceable and faithful automatic discharge summarization in future work. While the purpose of this work was to investigate the provenance of discharge summaries from EHR note antecedents, our secondary goal was to augment our dataset with semi-supervised annotations. Given the unsupervised baseline models have room for improvement, we leave supervised methods (such as pretrained biomedical language models fine-tuned span tagging) as future work