A Corpus for Detecting High-Context Medical Conditions in Intensive Care Patient Notes Focusing on Frequently Readmitted Patients

A crucial step within secondary analysis of electronic health records (EHRs) is to identify the patient cohort under investigation. While EHRs contain medical billing codes that aim to represent the conditions and treatments patients may have, much of the information is only present in the patient notes. Therefore, it is critical to develop robust algorithms to infer patients' conditions and treatments from their written notes. In this paper, we introduce a dataset for patient phenotyping, a task that is defined as the identification of whether a patient has a given medical condition (also referred to as clinical indication or phenotype) based on their patient note. Nursing Progress Notes and Discharge Summaries from the Intensive Care Unit of a large tertiary care hospital were manually annotated for the presence of several high-context phenotypes relevant to treatment and risk of re-hospitalization. This dataset contains 1102 Discharge Summaries and 1000 Nursing Progress Notes. Each Discharge Summary and Progress Note has been annotated by at least two expert human annotators (one clinical researcher and one resident physician). Annotated phenotypes include treatment non-adherence, chronic pain, advanced/metastatic cancer, as well as 10 other phenotypes. This dataset can be utilized for academic and industrial research in medicine and computer science, particularly within the field of medical natural language processing.

section_name: Introduction and Related Work
paragraphs: With the widespread adoption of electronic health records (EHRs), medical data are being generated and stored digitally in vast quantities BIBREF0. While much EHR data are structured and amenable to analysis, there appears to be limited homogeneity in data completeness and quality BIBREF1, and it is estimated that the majority of healthcare data are being generated in unstructured, text-based format BIBREF2. The generation and storage of these unstructured data come concurrently with policy initiatives that seek to utilize preventative measures to reduce hospital admission and readmission BIBREF3.
Chronic illnesses, behavioral factors, and social determinants of health are known to be associated with higher risks of hospital readmission, BIBREF4 and though behavioral factors and social determinants of health are often determined at the point of care, their identification may not always be curated in structured format within the EHR in the same manner that other factors associated with routine patient history taking and physical examination are BIBREF5. Identifying these patient attributes within EHRs in a reliable manner has the potential to reveal actionable associations which otherwise may remain poorly defined.
As EHRs act to streamline the healthcare administration process, much of the data collected and stored in structured format may be those data most relevant to reimbursement and billing, and may not necessarily be those data which were most relevant during the clinical encounter. For example, a diabetic patient who does not adhere to an insulin treatment regimen and who thereafter presents to the hospital with symptoms indicating diabetic ketoacidosis (DKA) will be treated and considered administratively as an individual presenting with DKA, though that medical emergency may have been secondary to non-adherence to the initial treatment regimen in the setting of diabetes. In this instance, any retrospective study analyzing only the structured data from many similarly selected clinical encounters will necessarily then underestimate the effect of treatment non-adherence with respect to hospital admissions.
While this form of high context information may not be found in the structured EHR data, it may be accessible in patient notes, including nursing progress notes and discharge summaries, particularly through the utilization of natural language processing (NLP) technologies. BIBREF6, BIBREF7 Given progress in NLP methods, we sought to address the issue of unstructured clinical text by defining and annotating clinical phenotypes in text which may otherwise be prohibitively difficult to discern in the structured data associated with the text entry. For this task, we chose the notes present in the publicly available MIMIC database BIBREF8.
Given the MIMIC database as substrate and the aforementioned policy initiatives to reduce unnecessary hospital readmissions, as well as the goal of providing structure to text, we elected to focus on patients who were frequently readmitted to the ICU BIBREF9. In particular, a patient who is admitted to the ICU more than three times in a single year. By defining our cohort in this way we sought to ensure we were able to capture those characteristics unique to the cohort in a manner which may yield actionable intelligence on interventions to assist this patient population.
section_name: Data Characteristics
paragraphs: We have created a dataset of discharge summaries and nursing notes, all in the English language, with a focus on frequently readmitted patients, labeled with 15 clinical patient phenotypes believed to be associated with risk of recurrent Intensive Care Unit (ICU) readmission per our domain experts (co-authors LAC, PAT, DAG) as well as the literature. BIBREF10 BIBREF11 BIBREF12
Each entry in this database of consists of a Subject Identifier (integer), a Hospital Admission Identifier (integer), Category (string), Text (string), 15 Phenotypes (binary) including “None” and “Unsure”, Batch Date (string), and Operators (string). These variables are sufficient to use the data set alone, or to join it to the MIMIC-III database by Subject Identifier or Hospital Admission Identifier for additional patient-level or admission-level data, respectively. The MIMIC database BIBREF8 was utilized to extract Subject Identifiers, Hospital Admission Identifiers, and Note Text.
Annotated discharge summaries had a median token count of 1417.50 (Q1-Q3: 1046.75 - 1926.00) with a vocabulary of 26454 unique tokens, while nursing notes had a median count of 208 (Q1-Q3: 120 - 312) with a vocabulary of 12865 unique tokens.
Table defines each of the considered clinical patient phenotypes. Table counts the occurrences of these phenotypes across patient notes and Figure contains the corresponding correlation matrix. Lastly, Table presents an overview of some descriptive statistics on the patient notes' lengths.
section_name: Methods
paragraphs: Clinical researchers teamed with junior medical residents in collaboration with more senior intensive care physicians to carry out text annotation over the period of one year BIBREF13. Operators were grouped to facilitate the annotation of notes in duplicate, allowing for cases of disagreement between operators. The operators within each team were instructed to work independently on note annotation. Clinical texts were annotated in batches which were time-stamped on their day of creation, when both operators in a team completed annotation of a batch, a new batch was created and transferred to them.
Two groups (group 1: co-authors ETM & JTW; group 2: co-authors JW & JF) of two operator pairs of one clinical researcher and one resident physician (who had previously taken the MCAT®) first annotated nursing notes and then discharge summaries. Everyone was first trained on the high-context phenotypes to look for as well as their definitions by going through a number of notes in a group. A total of 13 phenotypes were considered for annotation, and the label “unsure” was used to indicate that an operator would like to seek assistance determining the presence of an phenotype from a more senior physician. Annotations for phenotypes required explicit text in the note indicating the phenotype, but as a result of the complexity of certain phenotypes there was no specific dictionary of terms, or order in which the terms appeared, required for a phenotype to be considered present.
section_name: Limitations
paragraphs: There exist a few limitations to this database. These data are unique to Beth Israel Deaconess Medical Center (BIDMC), and models resulting from these data may not generalize to notes generated at other hospitals. Admissions to hospitals not associated with BIDMC will not have been captured, and generalizability is limited due to the limited geographic distribution of patients which present to the hospital.
We welcome opportunities to continue to expand this dataset with additional phenotypes sought in the unstructured text, patient subsets, and text originating from different sources, with the goal of expanding the utility of NLP methods to further structure patient note text for retrospective analyses.
section_name: Technical Validation
paragraphs: All statistics and tabulations were generated and performed with R Statistical Software version 3.5.2. BIBREF14 Cohen's Kappa BIBREF15 was calculated for each phenotype and pair of annotators for which precisely two note annotations were recorded. Table summarizes the calculated Cohen's Kappa coefficients.
section_name: Usage Notes
paragraphs: As this corpus of annotated patient notes comprises original healthcare data which contains protected health information (PHI) per The Health Information Portability and Accountability Act of 1996 (HIPAA) BIBREF16 and can be joined to the MIMIC-III database, individuals who wish to access to the data must satisfy all requirements to access the data contained within MIMIC-III. To satisfy these conditions, an individual who wishes to access the database must take a “Data or Specimens Management” course, as well as sign a user agreement, as outlined on the MIMIC-III database webpage, where the latest version of this database will be hosted as “Annotated Clinical Texts from MIMIC” BIBREF17. This corpus can also be accessed on GitHub after completing all of the above requirements.
section_name: Baselines
paragraphs: In the section, we present the performance of two well-established baseline models to automatically infer the phenotype based on the patient note, which we approach as a multi-label, multi-class text classification task BIBREF18. Each of the baseline model is a binary classifier indicating whether a given phenotype is present in the input patient note. As a result, we train a separate model for each phenotype.
section_name: Baselines ::: Bag of Words + Logistic Regression
paragraphs: We convert each patient note into a bag of words, and give as input to a logistic regression.
section_name: Baselines ::: Convolutional Neural Network (CNN)
paragraphs: We follow the CNN architecture proposed by collobert2011natural and kim2014convolutional. We use the convolution widths from 1 to 4, and for each convolution width we set the number of filters to 100. We use dropout with a probability of $0.5$ to reduce overfitting BIBREF19. The trainable parameters were initialized using a uniform distribution from $-0.05$ to $0.05$. The model was optimized with adadelta BIBREF20. We use word2vec BIBREF21 as the word embeddings, which we pretrain on all the notes of MIMIC III v3.
Table presents the performance of the two baseline models (F1-score).
section_name: Conclusion
paragraphs: In this paper we have presented a new dataset containing discharge summaries and nursing progress notes, focusing on frequently readmitted patients and high-context social determinants of health, and originating from a large tertiary care hospital. Each patient note was annotated by at least one clinical researcher and one resident physician for 13 high-context patient phenotypes.
Phenotype definitions, dataset distribution, patient note statistics, inter-operator error, and the results of baseline models were reported to demonstrate that the dataset is well-suited for the development of both rule-based and statistical models for patient phenotyping. We hope that the release of this dataset will accelerate the development of algorithms for patient phenotyping, which in turn would significantly help medical research progress faster.
section_name: Acknowledgements
paragraphs: The authors would like to acknowledge Kai-ou Tang and William Labadie-Moseley for assistance in the development of a graphical user interface for text annotation. We would also like to thank Philips Healthcare, The Laboratory of Computational Physiology at The Massachusetts Institute of Technology, and staff at the Beth Israel Deaconess Medical Center, Boston, for supporting the MIMIC-III database, from which these data were derived.

Questions and Answers:
question: Is this dataset publicly available for commercial use?
answers: answer: unanswerable: no
yes_no: no
evidence: As this corpus of annotated patient notes comprises original healthcare data which contains protected health information (PHI) per The Health Information Portability and Accountability Act of 1996 (HIPAA) BIBREF16 and can be joined to the MIMIC-III database, individuals who wish to access to the data must satisfy all requirements to access the data contained within MIMIC-III. To satisfy these conditions, an individual who wishes to access the database must take a “Data or Specimens Management” course, as well as sign a user agreement, as outlined on the MIMIC-III database webpage, where the latest version of this database will be hosted as “Annotated Clinical Texts from MIMIC” BIBREF17. This corpus can also be accessed on GitHub after completing all of the above requirements.
answer: unanswerable: no
yes_no: yes
evidence: Phenotype definitions, dataset distribution, patient note statistics, inter-operator error, and the results of baseline models were reported to demonstrate that the dataset is well-suited for the development of both rule-based and statistical models for patient phenotyping. We hope that the release of this dataset will accelerate the development of algorithms for patient phenotyping, which in turn would significantly help medical research progress faster.
question: How many different phenotypes are present in the dataset?
answers: answer: unanswerable: no
extractive_spans: 15 clinical patient phenotypes
evidence: We have created a dataset of discharge summaries and nursing notes, all in the English language, with a focus on frequently readmitted patients, labeled with 15 clinical patient phenotypes believed to be associated with risk of recurrent Intensive Care Unit (ICU) readmission per our domain experts (co-authors LAC, PAT, DAG) as well as the literature. BIBREF10 BIBREF11 BIBREF12
answer: unanswerable: no
free_form_answer: Thirteen different phenotypes are present in the dataset.
evidence: FLOAT SELECTED: Table 1: The thirteen different phenotypes used for our dataset, as well the definition for each phenotype that was used to identify and annotate the phenotype.
question: What are 10 other phenotypes that are annotated?
answers: answer: unanswerable: no
free_form_answer: Adv. Heart Disease, Adv. Lung Disease, Alcohol Abuse, Chronic Neurologic Dystrophies, Dementia, Depression, Developmental Delay, Obesity, Psychiatric disorders and Substance Abuse
evidence: Table defines each of the considered clinical patient phenotypes. Table counts the occurrences of these phenotypes across patient notes and Figure contains the corresponding correlation matrix. Lastly, Table presents an overview of some descriptive statistics on the patient notes' lengths.
FLOAT SELECTED: Table 1: The thirteen different phenotypes used for our dataset, as well the definition for each phenotype that was used to identify and annotate the phenotype.

Figure and Table Captions:
caption: Table 1: The thirteen different phenotypes used for our dataset, as well the definition for each phenotype that was used to identify and annotate the phenotype.
caption: Table 2: Data set distribution. The second and fourth columns indicate how many discharge notes and nursing notes contain a given phenotype. The third and fifth columns indicate the corresponding percentages (i.e., the percentage of discharge notes and nursing notes that contain a given phenotype).
caption: Table 3: Patient note statistics. IQR stands for interquartile range.
caption: Figure 1: Correlation matrices of phenotypes for Nursing Notes and Discharge Summaries.
caption: Table 4: Tabulated Cohen’s Kappa values for each on the two annotator pairs: the first pair is ETM & JTW, and the second pair is JF & JW.
caption: Table 5: Results of the two baseline models (F1-score in percentage). BoW stands for Bag of Words.