Auto-generating ground truth on clinical text by leveraging structured electronic health record data

A method improves performance of natural language processing by automatically generating ground truth from electronic health records comprising unstructured clinical notes and structured data comprising entries each having respective values for fields. The method includes: linking a given one of the notes to a given one of the entries responsive to determining that a specified field within the given entry matches an item of metadata for the given note; determining an initial set of the notes which satisfy criteria selected such that the criteria are a proxy for the ground truth, wherein the given note is determined to satisfy the criteria based at least in part on the given entry linked thereto; and designating at least a portion of the initial set of notes which satisfy the criteria, and the entries linked to the portion of the initial set of notes which satisfy the criteria, as the ground truth.

BACKGROUND

The present invention relates to the electrical, electronic and computer arts, and, more particularly, to improvements in natural language processing of electronic health records.

The patient electronic health record (EHR) usually includes both structured and unstructured data. Structured EHR data typically uses a controlled vocabulary and have an organized format allowing for easier processing, and includes information such as problem list, medication list, allergies, vital signs, lab results, ordered procedures, etc. Unstructured EHR data, or clinical text, do not follow a particular format and are therefore more difficult for a system to understand, but have the benefit of allowing clinicians to document more nuanced or contextual information to paint a more complete picture of the patient's health. Some examples of clinical notes within EHR data are physician progress notes, nursing notes, discharge summaries, diagnostic test reports, correspondence, patient emails, etc.

Clinical notes within patient electronic health records (EHRs) are traditionally a rich source of data where detailed information about the patient's medical history and clinical care process is documented. Some of this information is not captured anywhere else within the EHR. but could potentially have great impact on the patient care process. Some examples include: reasoning behind medical decision-making (e.g. why did the doctor decide on treatment A vs treatment B?), medication changes (e.g. instructions to hold BP meds for a few days), adverse drug reactions (e.g. patient experiencing diarrhea on higher dose of metformin), social determinants of health (e.g. patient has limited access to a car which has resulted in multiple missed appointments), patient adherence (e.g. is patient taking medication as prescribed?), patient preferences (e.g. patient does not like needles).

However, physicians at the point of care are mostly unable to review much of this unstructured information due to the abundance of notes within the patient EHR and the time constraint inherent in the clinical setting. Advances in natural language processing (NLP) and machine learning techniques in recent years within the medical/clinical domain have shown promise in effectively analyzing EHR data with potential applications in patient care, clinical research, hospital operations management, etc.

A significant barrier in effectively utilizing machine learning techniques on clinical data is the need for a sufficiently large “ground truth” data set to train and test the models. This is because manual annotation to generate labeled data on clinical notes is a time-consuming and tedious process requiring human annotators with domain expertise. The current “gold standard” process for ground truth generation on clinical text is a manual process that requires medical experts to manually review hundreds or thousands of notes. This process is time-consuming, requires trained individuals, and is subject to human error. Current techniques require employing subject matter experts (SME), which are expensive annotation resources, who must abide by access restrictions regarding protected health information (PHI), such as the Health Insurance Portability and Accountability Act (HIPAA). Thus, automated generation of annotations for information extraction on clinical text remains a long-felt but unmet need.

SUMMARY

An illustrative embodiment includes a method for improving performance of a natural language processing task by automating generation of ground truth from electronic health records. The electronic health records comprising unstructured clinical notes and at least one table of structured data comprising entries each having respective values for one or more fields. The method includes linking at least a given one of the unstructured clinical notes to at least a given one of the structured data entries responsive to determining that a value for a specified field within the given one of the structured data entries matches an item of metadata for the given one of the unstructured clinical notes, and determining an initial set of the unstructured clinical notes which satisfy one or more criteria. The one or more criteria selected such that the one or more criteria are a proxy for the ground truth, and the given one of the unstructured clinical notes is determined to satisfy the criteria based at least in part on the given one of the structured data entries linked thereto. The method also includes designating at least a portion of the initial set of unstructured clinical notes which satisfy the criteria, and the structured data entries linked to the portion of the initial set of unstructured clinical notes which satisfy the criteria, as the ground truth.

Illustrative embodiments of the present invention have practical applications and provide technological improvements. For example, an illustrative embodiment of the proposed invention can automatically generate “silver standard” ground truth for a variety of clinical NLP tasks without requiring any preexisting ground truth, such as initial manually created “gold standard” ground truth. Illustrative embodiments of the present invention provide a much faster and scalable process for auto-generating “silver standard” ground truth adaptable to a multitude of information extraction tasks with much less expert input than conventional approaches, thus permitting faster, less labor-intensive creation of ground truth. An illustrative embodiment of the present invention provides a solution for reducing expert input needed to scale to annotate a larger dataset. These and other features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.

DETAILED DESCRIPTION

Illustrative embodiments of the present invention address the problematic lack of annotations for information extraction on clinical text. As discussed above, the current process for ground truth generation on clinical text (“gold standard”) is a manual process that is time-consuming, requires trained individuals, and is subject to human error. Conversely, illustrative embodiments of the present invention can auto-generate ground truth (“silver standard”) for information extraction from clinical text using structured EHR data. Illustrative embodiments of the proposed invention can potentially generate ground truth for a variety of clinical NLP tasks without requiring any preexisting ground truth. Illustrative embodiments of the present invention provide a much faster and scalable process for auto-generating “silver standard” ground truth adaptable to a multitude of information extraction tasks with much less expert input needed. As used herein, “gold standard” ground truth refers to manually labeling data by a human expert, while “silver standard” ground truth refers to labels automatically generated from the data itself.

FIG.1is a combined block/flow diagram depicting one or more aspects of an illustrative embodiment of the present invention. Electronic health records (EHR)105typically includes structured data115and unstructured notes125. Structured EHR data115is organized into various tables that each have a set of fields. An illustrative embodiment100may begin with step110involves extracting derived insights from structured EHR entries115. Depending on the type of structured data, the data format, and the availability of various fields, different processing steps may be implemented to extract various kinds of derived insights.

FIG.2shows exemplary structured medication order data in accordance with an illustrative embodiment of the present invention. Medication order data200may represent a portion of the structured EHR data115following execution of step110inFIG.1. Structured medication order data200includes native structured data elements280existing in115prior to step110, as well as derived insights290created from native elements280in step110. Thus, each entry of medication order data200may include native structured data elements280representing a patient identifier281, an encounter (e.g., outpatient appointment or inpatient hospitalization) identifier282, medication name283, strength284, form or formulation285, instructions (sig)286, start date287, and end date288.

From these native structured data elements280, additional derived insights290can be obtained in step110. The derived insights290for each entry within medication order data200may include daily dosage291(dosage of medication prescribed over a single day) or medication change292: e.g., if this entry represents a new medication, a refill of an existing medication with no change in daily dose (even with a different strength—e.g., two 50 mg tablets rather than one 100 mg tablet), or a change in daily dose: increase or decrease. In the example shown inFIG.2, the processing to derive daily dosage291from the native elements280may include parsing, for example, the strength field284(e.g., “50 mg”) and the instruction field286(e.g., “take two tablets twice daily”). Processing to extract medication change status292as a derived insight290may require combining the medication name283, the start date287, and the end date288originally provided 280 in the structured data115, and the previously-derived daily dosage291. If there are inconsistencies and/or unreliable data (e.g., variations in the medication name283because of different brand names or generic names), step110may include retrieving and/or normalizing the necessary information by linking the structured medication order200to an external resource such as RxNorm using either the medication name field281or a code, such as National Drug Code (NDC), if provided.

FIG.3shows exemplary structured lab test result data300in accordance with an illustrative embodiment of the present invention. Lab test result data300may represent a portion of the structured EHR data115following execution of step110inFIG.1. Structured lab test result data300includes native structured data elements380existing in115prior to step110, as well as derived insights390created from native elements380in step110. Thus, each entry of lab test results data300may include native structured data elements380representing a patient identifier381, an encounter (e.g., outpatient appointment or inpatient hospitalization) identifier382, date and time the test was ordered383, the name of the test analyte384, the measured value of the analyte for the test385, and the corresponding unit of measurement (UOM)386.

Derived insights390may include a flag indicating whether the value is within a normal range, the patient's baseline for a particular lab test (i.e. the patient's “normal” or “usual” measurement of a lab analyte), or change from previous lab result (i.e. the delta between a specific lab result and the previous result for the same lab). The baseline value for a specific lab can be estimated in a number of ways. Some proposed estimations are: (1) the mean outpatient value, (2) the most recent outpatient value, (3) the nadir outpatient value, and (4) the most recent inpatient or outpatient value. Also, depending on the type of lab, additional baseline estimations can be applied. For example, the Modification of Diet in Renal Disease (MORD) equation (back-estimation) can be used for estimating baseline for serum creatinine. All of these methods for baseline estimation can be used as part of the processing of structured lab data, each becoming a unique derived insight. InFIG.3, the derived insights include the aforementioned flag391indicating whether the value is within a normal range, the change from the immediately preceding value (delta)392, the running average (baseline: mean)393, the lowest value (baseline: nadir)394, and the immediately preceding value (baseline: prior)395.

Returning toFIG.1, step120involves linking the structured EHR entries115(including both the native fields280/380and the derived insights290/390from step110) to unstructured clinical notes125. For example, it may be desirable to link structured EHR entries115and clinical notes125generated from the same encounter (e.g., outpatient office visit or inpatient facility stay).

This linkage can occur in multiple ways depending on the availability of various fields within the EHR data105(including115and125). Usually there is an ID that groups together data elements originating from the same patient encounter (e.g.,282and382inFIGS.2and3, respectively). The existence and naming of this ID differs depending on the specific EHR system and format. For outpatient visits, there may be an ENCOUNTER ID that represents everything that happened for a single patient visit. In the case of inpatient stays, you may have an ADMISSION_ID which represents a single patient admission to the hospital, and/or an ICUSTAY_ID to specifically identify patient stays in the intensive care unit (ICU). In the absence of an ID to link together structured and unstructured EHR data elements, secondary methods of linkage can be leveraged such as: (a) the dates when events were generated (e.g. a medication order date in structured medications data115/280may be linked to note date in note metadata125), (b) the authorizing provider who initiated the events (e.g. provider who ordered lab in structured lab results115/380may be linked to note author in note metadata), or (c) the department where events originated from, etc.

The resulting output is an expanded representation of the patient EHR105where each structured entry115(from step110) contains both native data elements (280/380) and derived insights (290/390) and also (from step120) is linked to unstructured clinical notes125generated from the same encounter along with the associated note metadata (e.g., note type, note data, note author, etc.) Accordingly, the unstructured notes125(with associated metadata) for a given patient may be linked to the structured data115, including both native elements280/380as well as derived insights290/390.

Note that steps110(extracting derived insights) and120(linking to unstructured notes) can be a pre-processing step run on all EHR data105prior to deciding on a specific use case or information extraction task135. Thus, steps110and120may be specific to EHR105(e.g., dependent on the particular content and/or format of EHR105) as indicated by160, while steps130-150may be specific to task135(e.g., dependent on the particular use case or information extraction task) as indicated by170. Similar to the manner in which steps110and120were described with reference to examples of structured data115shown inFIGS.2and3, the remaining steps130-150will be discussed with reference to two sample use cases135: Sample Task “A” is to extract medication change events from clinical notes, while Sample Task “B” is to identify decline in heart failure from clinical notes.

Step130defines a proxy for silver standard ground truth based on the particular use case or task definition135. This involves using the task definition135to determine appropriate values for EHR structured elements (115/280/380), derived insights (290/390), and note metadata (125) to serve as a proxy for the silver standard ground truth145. Step130is the only step inFIG.1which may require from a domain expert to guide what structured data sources and types of notes are most pertinent.

Where task135is the aforementioned “A” (to extract medication change events from clinical notes), step130may determine criteria suitable as a proxy for ground truth to be clinical notes125linked to structured medication entries200where derived medication change insight292is START, STOP, INCREASE, or DECREASE. Where task135is the aforementioned “B” (to identify decline in heart failure from clinical notes), step130may determine criteria suitable as a proxy for ground truth to be clinical notes135with Note Author Type=Physician and Encounter Diagnosis Code=428. * that are linked to either (1) a structured lab300of type384BNP (brain natriuretic peptide) determined to be abnormal in derived insight391or (2) a structured medication200with generic name283FUROSEMIDE that has derived medication change insight292START or INCREASE.

Step140involves application of the proxy (e.g., criteria) defined in step130to the output of step120. Depending on the specific use case135and ground truth needed (e.g., medication change, disease status), the patient records (e.g., clinical notes125) are filtered using a combination of note metadata125, native attributes280/380and derived insights290/390from linked structured entries115. The resulting output145is a smaller set of clinical notes with associated metadata125and linked structural data115that satisfies the proxy definition130specific to the task at hand135. This output145may be directly used as “silver standard” ground truth for the specified task135, such as in analytic training and development.

Step150includes optional post-processing (e.g., further filtering of clinical notes) to achieve improved quality (e.g. “cleaner”) ground truth145prior to use, e.g., in analytic training and development. Step150may include applying natural language processing (NLP) to clinical text to extract information which can be leveraged to discard selected clinical notes of lower quality. Examples of processes which can be used, in isolation or in any combination, to provide further filtering mechanisms to improve the quality of selected clinical notes for silver ground truth145include but are not limited to:

1) Named Entity Recognition (NER): A pretrained model on NER task can be employed to identify various entities present in the natural language of clinical text (e.g. medication names, disease names). These identified entities can then be used to further filter selected notes.

2) Note Section Classification: A pretrained model (learned or rule-based or hybrid) can be employed to identify whether a sentence belongs to a specific note section (e.g. Assessment & Plan (AP), Medication Section, Past Medical History (PMH)) in the clinical notes. Given the certain attributes of a note section (e.g. PMH is usually about past events etc.), the selected notes can be filtered to further improve the quality of the silver ground truth.

3) Clinical Semantic Textual Similarity (STS): Sentences from various notes can be compared to derive semantic similarity between them. For example, sentences “Lisinopril was dispensed” & “The patient is on lisinopril” have high semantic similarity. This measure can be used to identify repetitive information within a note and between notes. This can help identify high veracity sentences which have a higher probability of recording the required insight.

By way of example, for the aforementioned Sample Task “A” (extract medication change events from clinical notes), one can first employ a NER extraction-based mechanism to identify medication names in the clinical text. This is based on the fact that clinical notes explicitly mentioning the medication have a higher chance of containing the ground truth for the particular medication change insight. In other words, if the medication in the medication change derived insight is also present in the clinical text of the linked note, retain this note as ground truth, and discard all other ground truth values not meeting this criterion.

One could then restrict these medication mentions in clinical notes to only those appearing in certain sections of the note using a pretrained model for note section classification. For example, the History of Present Illness (HPI) and Assessment/Plan (AP) sections of the note typically contain information most relevant to the current visit, while the Past Medical History (PMH) section is usually about past events. Therefore for this task, further restrict the silver ground truth to be only those notes where the mention of the medication name (from NER task) is present either in HPI or AP sections.

As another example, for the aforementioned Sample Task “B” (identify decline in heart failure disease status from clinical notes) one can first employ a NER extraction-based mechanism to identify mentions (and variants) of the disease “heart failure”, the medication “furosemide” or the lab “BNP” in the clinical text. Absence of these specific concepts in the clinical note can be used to discard notes which are less likely to contain information relevant to this specific task. Restricting to only notes with mentions of “heart failure”, “furosemide” or “BNP” ensures that the disease of interest was actually discussed during the specific encounter and note. One could also further restrict the silver ground truth to be only those notes where the mention of interest (from NER task) is present in the AP section (from Note Section Classification model).

Illustrative embodiments of the present invention provide a novel methodology of auto-generating silver standard ground truth for various information extraction tasks on unstructured clinical notes. Illustrative embodiments of the present invention provide a much faster and scalable process for auto-generating “silver standard” ground truth adaptable to a multitude of information extraction tasks with much less expert input needed than current approaches, thus permitting faster, less labor-intensive creation of ground truth. Illustrative embodiments include a novel methodology for identifying and automatically extracting ground truth leveraging the unique way EHR systems organize their structured and unstructured information, with particular utility in clinical NLP. Illustrative embodiments can leverage derived insights from structured EHR entries and linkages between structured and unstructured elements within the EHR to identify clinical notes of interest. Illustrative embodiments may also filter unstructured data based on the combined metadata obtained from structured and unstructured elements within the EHR to improve the quality of silver standard ground truth.

An illustrative embodiment of the present invention processes structured EHR data to derive insights, defines a silver ground truth proxy using both native and derived structured EHR elements, leverages linkages between structured and unstructured data to identify notes of interest, and autogenerates a set of notes as silver ground truth. Instead of focusing on a single task like question answering, an illustrative embodiment can adapt to a multitude of information extraction tasks in clinical notes. Thus, an illustrative embodiment can automatically generate “silver standard” ground truth for a variety of clinical NLP tasks without requiring any preexisting ground truth, e.g., initial manually created “gold standard” ground truth. An illustrative embodiment of the present invention provides a solution for reducing expert input needed to scale to annotate a larger dataset. Thus, an illustrative embodiment provides an automated way to generate a large silver standard ground truth by leveraging structured EHR data.

To recapitulate, an illustrative embodiment of the present invention may include a method for auto-generating silver standard ground truth for information extraction from clinical text using structured EHR data. This method may include extracting derived insights from structured EHR entries, and linking structured EHR entries to unstructured clinical notes. The method may also include defining criteria (e.g., values of EHR structured data, derived insights, and note metadata) as a proxy for silver ground truth, and filtering clinical notes based on these criteria. The method may optionally include further filtering of the clinical notes to achieve “cleaner” ground truth.

An illustrative embodiment of the present invention may additionally or alternatively include a method for identifying and extracting ground truth, which includes: deriving one or more insights from one or more structured data within an electronic health record; linking (associating) the one or more structured data and the one or more insights with one or more unstructured clinical notes within the electronic health record; receiving one or more filters to the associated one or more structured data, the one or more insights, and the one or more unstructured clinical notes; and filtering the one or more structured data, the one or more insights, and the one or more unstructured clinical notes. The method may also include processing the filtered one or more structured clinical notes, the one or more insights, and the one or more unstructured clinical notes based on applying at least one of named entity recognition, note section classification, and clinical semantic textual similarity. Linking the structured data with the one or more unstructured clinical notes may be based on one or more of: an ID that groups data elements originating from a same patient encounter, a date on which an event within the electronic health record is generated, an authorizing provider who initialed an event within the electronic health record, or a department from which an event within the electronic health record originates.

One or more embodiments of the invention, or elements thereof, can be implemented, at least in part, in the form of an apparatus including a memory and at least one processor that is coupled to the memory and operative to perform exemplary method steps.

One or more embodiments can make use of software running on a general purpose computer or workstation. With reference toFIG.4, such an implementation might employ, for example, a processor902, a memory904, and an input/output interface formed, for example, by a display906and a keyboard908. The term “processor” as used herein is intended to include any processing device, such as, for example, one that includes a CPU (central processing unit) and/or other forms of processing circuitry. Further, the term “processor” may refer to more than one individual processor. The term “memory” is intended to include memory associated with a processor or CPU, such as, for example, RAM (random access memory), ROM (read only memory), a fixed memory device (for example, hard drive), a removable memory device (for example, diskette), a flash memory and the like. In addition, the phrase “input/output interface” as used herein, is intended to include, for example, one or more mechanisms for inputting data to the processing unit (for example, mouse), and one or more mechanisms for providing results associated with the processing unit (for example, printer). The processor902, memory904, and input/output interface such as display906and keyboard908can be interconnected, for example, via bus910as part of a data processing unit912. Suitable interconnections, for example via bus910, can also be provided to a network interface914, such as a network card, which can be provided to interface with a computer network, and to a media interface916, such as a diskette or CD-ROM drive, which can be provided to interface with media918.

A data processing system suitable for storing and/or executing program code will include at least one processor902coupled directly or indirectly to memory elements904through a system bus910. The memory elements can include local memory employed during actual implementation of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during implementation.

Input/output or I/O devices (including but not limited to keyboards908, displays906, pointing devices, and the like) can be coupled to the system either directly (such as via bus910) or through intervening I/O controllers (omitted for clarity).

It should be noted that any of the methods described herein can include an additional step of providing a system comprising distinct software modules embodied on a computer readable storage medium; the modules can include, for example, any or all of the elements depicted in the block diagrams or other figures and/or described herein. The method steps can then be carried out using the distinct software modules and/or sub-modules of the system, as described above, executing on one or more hardware processors902. Further, a computer program product can include a computer-readable storage medium with code adapted to be implemented to carry out one or more method steps described herein, including the provision of the system with the distinct software modules.

Exemplary System and Article of Manufacture Details