Ensuring quality in electronic health data

A method, system and computer program product for ensuring quality in electronic health data. Data processing pipelines among different tenants are monitored for data artifacts generated from electronic health data, where a “data artifact” refers to the tangible by-products produced during the processing of the electronic health record datasets generated by the electronic health record systems as well as the datasets themselves. Out of the generated data artifacts, those data artifacts that are produced from the same stage of the data processing pipelines among the different tenants are aggregated. A set of metrics at various levels of a dataset hierarchy are produced from the aggregated data artifacts. Such metrics provide information about a quality of the electronic health data. A quality report, which may include flagged issues with the datasets, is then generated and presented to a user based on an analysis of the produced set of metrics.

TECHNICAL FIELD

The present invention relates generally to electronic health records, and more particularly to ensuring quality in electronic health data.

BACKGROUND

An electronic health record (EHR) is the systematized collection of patient and population electronically-stored health information in a digital format. These records can be shared across different health care settings. Records are shared through network-connected, enterprise-wide information systems or other information networks and exchanges. EHRs may include a range of data, including demographics, medical history, medication and allergies, immunization status, laboratory test results, radiology images, vital signs, personal statistics, such as age and weight, and billing information.

Ensuring good data quality in such records is essential due to the types of information stored in such records which are used to ensure the wellbeing of patients. However, defining and measuring quality in such records is challenging, especially in a multitenant analytical platform (e.g., multiple healthcare systems), due to several factors, such as the large data volume. There are hundreds of millions of EHRs being generated every day from healthcare systems.

Furthermore, defining and measuring quality in such records is challenging because of the heterogeneous data sources. Different hospitals often use distinctive EHR systems that adopt various formats. Even within the same hospital, different departments may generate duplicate or contradicting records. Thus, the standard of quality of EHR varies based on the data sources.

Additionally, defining and measuring quality in such records is challenging because of complex relations among variables. For example, there are millions of codes from standard ontologies (International Classification of Diseases (ICD), Systematized Nomenclature of Medicine (SNOMED®), RxNorm®, Common Procedure Terminology (CPT®), Logical Observation Identifiers Names and Codes (LOINC®), etc.) that describe diagnoses, medical procedures, lab tests, and drugs, not to mention countless custom codes.

Further, defining and measuring quality in such records is challenging because of operational factors, such as utilizing multiple data processing pipelines that are each configured per tenant, regularly receiving new data and utilizing a continuous delivery development model. These complications may introduce quality issues unrelated to the source data quality.

Hence, there is not currently a means for ensuring quality in the electronic health data.

SUMMARY

In one embodiment of the present invention, a method for ensuring quality in electronic health data comprises monitoring data processing pipelines among different tenants for data artifacts generated from electronic health data, where the data processing pipelines comprise a set of data processing elements connected in series where an output of one element is an input of a next element, where the data processing pipelines comprise stages of different processing, and where the generated data artifacts comprise tangible by-products produced during processing of electronic health record datasets as well as the datasets themselves. The method further comprises aggregating data artifacts of the generated data artifacts produced from a same stage of the data processing pipelines among the different tenants. The method additionally comprises producing a set of metrics at various levels of a dataset hierarchy from the aggregated data artifacts, where the produced set of metrics provides information about a quality of the electronic health data. Furthermore, the method comprises analyzing the produced set of metrics to generate aggregate outputs, where the aggregate outputs comprise one or more of the following: classified trends based on computing rollup values along several dimensions for different time intervals resulting in a superset containing both original single-metric trends and computed aggregate trends and then classifying each trend using a set of mathematical tests, grouped datasets by aggregating trend classifications identified in an output for each group and identified trends for the group as a whole, and configured checklists based on platform input limitations at each stage of the data processing pipelines among the different tenants. Furthermore, the method comprises generating a quality report based on the aggregate outputs. Additionally, the method comprises presenting the quality report to a user.

DETAILED DESCRIPTION

The present invention comprises a method, system and computer program product for ensuring quality in electronic health data. In one embodiment of the present invention, data processing pipelines among different tenants are monitored for data artifacts generated from electronic health data. A “data artifact,” as used herein, refers to the tangible by-products produced during the processing of the electronic health record datasets generated by the electronic health record systems as well as the datasets themselves. Out of the generated data artifacts, those data artifacts that are produced from the same stage of the data processing pipelines among the different tenants are aggregated. A set of metrics at various levels of a dataset hierarchy are produced from the aggregated data artifacts. In one embodiment, such metrics provide information about a quality of the electronic health data. The set of metrics are then analyzed to generate aggregate outputs, which may include: (1) the classified trends based on computing rollup values along several dimensions for different time intervals resulting in a superset containing both original single-metric trends and computed aggregate trends and then classifying each trend using a set of mathematical tests, (2) grouped datasets by aggregating trend classifications identified in an output for each group and identified trends for the group as a whole and/or (3) configured checklists based on platform input limitations at each stage of the data processing pipelines among the different tenants. A quality report is generated based on the aggregate outputs. Such a quality report may include flagged issues with the datasets. The quality report is then presented to the user. In this manner, the present invention ensures quality in the electronic health data.

Referring now to the Figures in detail,FIG. 1illustrates an embodiment of the present invention of a communication system100for practicing the principles of the present invention in accordance with an embodiment of the present invention. Communication system100includes electronic health record systems101A-101C (identified as “Electronic Health Record System A,” “Electronic Health Record System B,” and “Electronic Health Record System C,” respectively, inFIG. 1) connected to a data analyzer102via a network103. Electronic health record systems101A-101C may collectively or individually be referred to as electronic health record systems101or electronic health record system101, respectively.

Healthcare systems generate a large volume of electronic health records (EHRs) via electronic health record systems101, which may be shared across different health care settings. A “healthcare system,” as used herein, refers to the organization of people, institutions and resources that deliver health care services to meet the health needs of target populations. Such systems utilize electronic health record systems101configured to generate EHRs directed to the health services provided by the healthcare systems. These records are shared through network-connected, enterprise-wide information systems or other information networks and exchanges, such as via network103. EHRs may include a range of data, including demographics, medical history, medication and allergies, immunization status, laboratory test results, radiology images, vital signs, personal statistics, such as age and weight, and billing information.

Different healthcare systems may use distinctive EHR systems101that adopt various formats. Even within the same healthcare care system, different departments may generate duplicate or contradicting records. As a result, defining and measuring quality in such records is challenging because of heterogeneous data sources.

Furthermore, as discussed in the Background section, defining and measuring quality in such records is also challenging because of the large data volume, the complex relations among variables and the operational factors.

Hence, there is not currently a means for ensuring quality in the electronic health data.

Data analyzer102, as shown inFIG. 1, is configured to ensure quality in the electronic health data generated by electronic health record systems101by tracking and analyzing many quality metrics measured against the EHR datasets generated by electronic health record systems101at multiple stages of the data processing pipelines for the multiple tenants (e.g., multiple healthcare systems) as discussed below in connection withFIGS. 3-7 and 8A-8B. A description of the hardware configuration of data analyzer102is provided below in connection withFIG. 2.

Network103may be, for example, a local area network, a wide area network, a wireless wide area network, a circuit-switched telephone network, a Global System for Mobile Communications (GSM) network, a Wireless Application Protocol (WAP) network, a WiFi network, an IEEE 802.11 standards network, various combinations thereof, etc. Other networks, whose descriptions are omitted here for brevity, may also be used in conjunction with system100ofFIG. 1without departing from the scope of the present invention.

Furthermore, as shown inFIG. 1, data analyzer102is connected to a database104. In one embodiment, database104is configured to store the historical values for the metrics (also referred to herein as the “standard set of metrics”) obtained from previously generated data artifacts. In one embodiment, the standard set of metrics is provided by a user. A “metric,” as used herein, refers to the measurement of a particular characteristic of the processing of electronic health data in a data processing pipeline for a tenant. A “data artifact,” as used herein, refers to the tangible by-products produced during the processing of the electronic health record datasets generated by electronic health record systems101as well as the datasets themselves. A further discussion regarding metric history and data artifacts are provided further below.

System100is not to be limited in scope to any one particular network architecture. System100may include any number of electronic health record systems101, data analyzers102, networks103and databases104. Furthermore, whileFIG. 1illustrates data analyzer102as being a separate physical device, some or all of the functionality of data analyzer102may reside within electronic health record system101.

Referring now toFIG. 2,FIG. 2illustrates a hardware configuration of data analyzer102(FIG. 1) which is representative of a hardware environment for practicing the present invention. Referring toFIG. 2, data analyzer102has a processor201coupled to various other components by system bus202. An operating system203runs on processor201and provides control and coordinates the functions of the various components ofFIG. 2. An application204in accordance with the principles of the present invention runs in conjunction with operating system203and provides calls to operating system203where the calls implement the various functions or services to be performed by application204. Application204may include, for example, a program for ensuring quality in electronic health data as discussed below in association withFIGS. 3-7 and 8A-8B.

Referring again toFIG. 2, read-only memory (“ROM”)205is coupled to system bus202and includes a basic input/output system (“BIOS”) that controls certain basic functions of data analyzer102. Random access memory (“RAM”)206and disk adapter207are also coupled to system bus202. It should be noted that software components including operating system203and application204may be loaded into RAM206, which may be data analyzer's102main memory for execution. Disk adapter207may be an integrated drive electronics (“IDE”) adapter that communicates with a disk unit208, e.g., disk drive. It is noted that the program for ensuring quality in electronic health data, as discussed below in association withFIGS. 3-7 and 8A-8B, may reside in disk unit208or in application204.

Data analyzer102may further include a communications adapter209coupled to bus202. Communications adapter209interconnects bus202with an outside network (e.g., network103ofFIG. 1) thereby allowing data analyzer102to communicate with electronic health record systems101.

As stated in the Background section, ensuring good data quality in such records is essential due to the types of information stored in such records which are used to ensure the wellbeing of patients. However, defining and measuring quality in such records is challenging, especially in a multitenant analytical platform (e.g., multiple healthcare systems), due to several factors, such as the large data volume. There are hundreds of millions of EHRs being generated every day from healthcare systems. Furthermore, defining and measuring quality in such records is challenging because of the heterogeneous data sources. Different hospitals often use distinctive EHR systems that adopt various formats. Even within the same hospital, different departments may generate duplicate or contradicting records. Thus, the standard of quality of EHR varies based on the data sources. Additionally, defining and measuring quality in such records is challenging because of complex relations among variables. For example, there are millions of codes from standard ontologies (International Classification of Diseases (ICD), Systematized Nomenclature of Medicine (SNOMED®), RxNorm®, Common Procedure Terminology (CPT®), Logical Observation Identifiers Names and Codes (LOINC®), etc.) that describe diagnoses, medical procedures, lab tests, and drugs, not to mention countless custom codes. Further, defining and measuring quality in such records is challenging because of operational factors, such as utilizing multiple data processing pipelines that are each configured per tenant, regularly receiving new data and utilizing a continuous delivery development model. These complications may introduce quality issues unrelated to the source data quality. Hence, there is not currently a means for ensuring quality in the electronic health data.

The embodiments of the present invention provide a means for ensuring quality in the electronic health data by tracking and analyzing many quality metrics measured against the electronic health record datasets generated by electronic health record systems101(FIG. 1) at multiple stages of the data processing pipeline as discussed below in connection withFIGS. 3-7 and 8A-8B.FIG. 3is a flowchart of a method for ensuring quality in electronic health data.FIG. 4illustrates a data processing pipeline for a single tenant.FIG. 5illustrates a platform for a multi-tenancy model, where each tenant is associated with a separate processing pipeline that generates a separate set of data artifacts.FIG. 6is a flowchart of a method for generating the aggregate outputs.FIG. 7illustrates applying a quality measurement process to each data artifact.FIGS. 8A-8Bshow a screen capture of a tenant-level data change report targeted toward internal users monitoring data quality changes across tenants.

As stated above,FIG. 3is a flowchart of a method300for ensuring quality in electronic health data in accordance with an embodiment of the present invention.

Referring toFIG. 3, in conjunction withFIGS. 1-2, in step301, data analyzer102monitors the data processing pipelines among different tenants (e.g., healthcare systems) for data artifacts generated from electronic health data. Each data processing pipeline may generate data artifacts from the same or different electronic health data, including electronic health data that includes different types of electronic health records (e.g., radiology images, laboratory test results) for the same patient.

A “data processing pipeline,” as used herein, refers to a set of data processing elements connected in series where an output of one element is an input of a next element as discussed below in connection withFIG. 4. Furthermore, the data processing pipelines include stages of different processing as discussed below in connection withFIG. 4.

Referring toFIG. 4,FIG. 4illustrates a data processing pipeline400for a single tenant (e.g., healthcare system) in accordance with an embodiment of the present invention. As shown inFIG. 4, data processing pipeline400includes various data processing elements connected in series where an output of one element is an input of a next element. For example, data processing pipeline400includes multiple source systems401. That is, each tenant's (e.g., a particular healthcare system) dataset consists of multiple source systems401. An example of a source system is the Epic Clarity EHR system at the Cleveland Clinic Main Campus for the Cleveland Clinic tenant.

Furthermore, as shown inFIG. 4, the source system data401is extracted402, where in its unstructured form, it is stored in unstructured data lake403(data lake is a system or repository of data stored in its natural format, usually object blobs or files). In one embodiment, unstructured data lake403is maintained in a storage unit (e.g., memory205, disk unit208) in data analyzer102. Alternatively, such information may be stored in database104.

Additionally, as shown inFIG. 4, the unstructured data403is published404, where in its structured form, it is stored in structured reservoir405. In one embodiment, structured reservoir405is maintained in a storage unit (e.g., memory205, disk unit208) in data analyzer102. Alternatively, such information may be stored in database104.

After pre-processing406the structured data405, an intermediate form407of the health data is formed. In one embodiment, the intermediate form407of the health data is stored in a storage unit (e.g., memory205, disk unit208) of data analyzer102or in database104.

Standardized ontologies408are applied to the intermediate form407of the health data thereby producing data store applications409, such as search-optimized artifact applications410and data warehouse applications411.

As shown inFIG. 4, data processing pipeline400begins with the extraction402of source system data401and terminates with data store applications409. Elements401,403,405,407and409(includes elements410,411) represent a data artifact which persists beyond process execution. A “data artifact,” as used herein, refers to the tangible by-products produced during the processing of the electronic health record datasets generated by electronic health record systems101as well as the datasets themselves. Furthermore, as shown inFIG. 4, data processing pipeline400includes stages of different processing, such as extraction402, publication404, pre-processing406and applying standard ontologies408.

As discussed above, in one embodiment, data analyzer102monitors the data processing pipelines among different tenants (e.g., healthcare systems) for data artifacts generated from electronic health data as shown inFIG. 5.

FIG. 5illustrates a platform for a multi-tenancy model500, where each tenant is associated with a separate processing pipeline that generates a separate set of data artifacts, in accordance with an embodiment of the present invention.

Referring toFIG. 5, multi-tenancy model500includes multiple data processing pipelines400′,400″,400′″ corresponding to data processing pipeline400ofFIG. 4. Data processing pipeline400′ corresponds to the data processing pipeline for Tenant A (e.g., Cleveland Clinic) as shown inFIG. 5. Furthermore, data processing pipeline400″ corresponds to the data processing pipeline for Tenant B (e.g., Mayo Clinic) as shown inFIG. 5. Additionally, data processing pipeline400′″ corresponds to the data processing pipeline for Tenant C (e.g., UNC Healthcare) as shown inFIG. 5. WhileFIG. 5illustrates monitoring three data processing pipelines for three tenants, the principles of the present invention are not to be limited in scope to monitoring only three data processing pipelines for three tenants. Instead, the principles of the present invention are to include the aspect of monitoring any number of data processing pipelines for any number of tenants.

As shown inFIG. 5, data analyzer102processes electronic health data from each tenant in a separate pipeline on the platform. The processing modules within each pipeline are configured and customized for each tenant. Each tenant pipeline generates a separate set of data artifacts as discussed above in connection withFIG. 4. In one embodiment, the data artifacts generated by different tenant pipelines are from the same or different electronic health data, including electronic health data that includes different types of electronic health records (e.g., radiology images, laboratory test results) for the same patient.

In one embodiment, data analyzer102efficiently monitors the processing pipelines400′,400″ and400′″ for both source data quality and data processing issues. In one embodiment, data analyzer102efficiently triages data quality issues to produce consumable quality reports which lead to actionable investigations and resolutions at a system level as discussed below. In order to produce such quality reports, a quality measurement process is applied to each data artifact in the pipeline400′,400″ and400′″ as discussed further below.

Returning toFIG. 3, in conjunction withFIGS. 1-2 and 4-5, in step302, data analyzer102aggregates the data artifacts (e.g., element407) produced from the same stage (e.g., pre-processing406) of the data processing pipelines (e.g., processing pipelines400′,400″ and400′″) among different tenants (e.g., tenants A, B and C). In this manner, the behavior across datasets can be compared to flag unexpected changes. For example, inaccuracies in personal statistics about a patient may be identified among different electronic health records processed by different tenants.

In step303, data analyzer102produces a set of metrics at various levels of a dataset hierarchy from the aggregated data artifacts. A “metric,” as used herein, refers to the measurement of a particular characteristic of the processing of electronic health data in a data processing pipeline for a tenant. For example, data analyzer102may produce a set of clinical metrics pertaining to the unstructured data403of the electronic health data. For instance, metrics, such as record type, date of record, data values, data models, data presentation, and conformance with governance policies, may be calculated at various levels of the dataset hierarchy (e.g., database, file, record, field). In one embodiment, the set of metrics is continuously calculated as new data is ingested and as existing data is reprocessed. In one embodiment, such metrics provide information about a quality of the electronic health data, such as identifying inconsistent data values (e.g., social security numbers) among different health records for the same patient.

In step304, data analyzer102analyzes the produced set of metrics to generate aggregate outputs. For example, the outputs may include: (1) the classified trends based on computing rollup values along several dimensions for different time intervals resulting in a superset containing both original single-metric trends and computed aggregate trends and then classifying each trend using a set of mathematical tests, (2) grouped datasets by aggregating trend classifications identified in an output for each group and identified trends for the group as a whole and/or (3) configured checklists based on platform input limitations at each stage of the data processing pipelines among the different tenants as discussed further below in connection withFIGS. 6 and 7.

FIG. 6is a flowchart of a method600for generating the aggregate outputs.FIG. 7illustrates applying a quality measurement process to each data artifact in accordance with an embodiment of the present invention.

Referring toFIG. 6, in conjunction withFIGS. 4-5 and 7, in step601, data analyzer102produces or calculates the metrics at various levels of the dataset hierarchy (e.g., organization of data in a hierarchy with multiple levels, such as database, file, record and field) from the aggregated data artifacts as discussed above in connection with step303. For example, data artifacts are aggregated from the same stage of the processing pipelines400′,400″ and400′″ of multi-tenancy model500by data analyzer102to perform data summarization/metric calculation701. For instance, data summarization/metric calculation701may include metrics (e.g., clinical metrics) pertaining to the unstructured data403of the electronic health data. For example, metrics, such as record type, date of record, data values, data models, data presentation, and conformance with governance policies, may be calculated at various levels of the dataset hierarchy (e.g., database, file, record, field).

In step602, data analyzer102obtains the standard set of metrics at various levels of the dataset hierarchy, such as from database104storing the historical values for the metrics obtained from previously generated data artifacts. In one embodiment, the standard set of metrics is provided by a user. In one embodiment, the standard set of metrics corresponds to expected values for such metrics (e.g., value for a birthdate of person A), which may be based on previously recorded values for that same metric (e.g., previously recorded the value for the birthdate of person A from a document previously analyzed by data analyzer102). In another embodiment, such expected values are obtained from previously aggregated data artifacts, such as obtaining the metric value of a social security number for person A. Such information is stored in database104to be later used by data analyzer102to determine the accuracy, consistency, completeness and relevancy of the health data.

In one embodiment, a hierarchical dataset includes data where observations fall into groups or clusters, where such hierarchies can have multiple levels.

In step603, data analyzer102compares the calculated metrics with the standard set of metrics at various levels of the dataset hierarchy. For example, data analyzer102may perform an analysis, such as a trend analysis702, to compare the calculated metrics601with the standard set of metrics (stored in database104) at various levels of the dataset hierarchy. In such a manner, data analyzer102assesses the accuracy, consistency, completeness and relevancy of the health data. For example, the calculated metric of the birthdate of person A may be compared with the standard metric of the birthdate of person A to determine whether the calculated metric is within a threshold degree, which may be user-specified, of the expected value of the standard metric. If the calculated metric is not within a threshold degree of the expected value of the standard metric, then such health data may not be accurate, consistent, complete or relevant. Otherwise, such health data may be deemed to be accurate, consistent, complete and/or relevant.

In one embodiment, some parts of the monitoring process compare data extracts from similar source systems across different tenants. For example, Allscripts® is a common EHR platform among various tenants. Comparing statistical extracts of clinically relevant data from Allscripts® installations across tenants allows one to measure data quality differences and detect problems with the source data or data processing pipeline over time. In addition, comparing trend analysis at multiple stages in the pipeline across tenants allows automated troubleshooting of per-tenant configuration issues.

In step604, data analyzer102produces the aggregate outputs based on the comparison of step603. Such outputs may include classified trends703, grouped datasets704and configured checklists705.

With respect to classified trends703, data analyzer102may first compute rollup trend values along several dimensions, such as EHR system, record type, etc., and for different time intervals (e.g., year/month/week to date). This results in a superset containing both the original single metric trends and the computed aggregate trends. Data analyzer102then classifies each trend using a set of mathematical tests. For example, trends may be classified as linear increasing, linear decreasing, bimodal, unstable, etc. In another example, one important classification is whether the trend changed suddenly with the most recent processing cycle.

With respect to grouped datasets704, the source data can be grouped many ways, such as by record type (admission, diagnosis, observation, etc.), source system, or source system type. Embodiments of the present invention aggregate the trend classifications that were identified for each group (identified in classified trends703) and identifies the trends for the group as a whole. For example, if several source systems exhibit increasing numbers of bad dates (e.g., dates in future or distant past), the user can identify a problem with the corresponding tenants' pipeline processing stages. However, if the trend only presents in a single data source, the user can recommend corrective action for that source (perhaps a default value is being saved).

With respect to configured checklists705, a set of predefined checks is performed. For example, patient records which are not linked to a demographic record cannot be used by data analyzer102. Patients with too many records linked to them (e.g., because they are test patients in the source system) may unbalance the distributed processing components, and as a result, they should be flagged.

By producing such aggregate outputs, the present invention ensures quality in the electronic health data.

Returning toFIG. 3, in conjunction withFIGS. 1-2 and 4-7, in step304, data analyzer102generates a quality report706based on the aggregate outputs as shown inFIG. 7. Such a quality report706may include flagged issues with the datasets. For example, when a difference between a metric in the calculated set of metrics (see step601) and a corresponding metric in the standard set of metrics (see step602) exceeds a threshold variance, then such a metric is flagged in the quality report.

In one embodiment, the analysis output is combined into a series of reports706for end users across the organization. Some reports are specifically targeted to tenant users and focus on metrics pertaining to source data quality over various timeframes. Other reports are for internal use and focus on engineering metrics and detecting platform issues as they occur.FIGS. 8A-8Bshow a screen capture of a tenant-level data change report targeted toward internal users monitoring data quality changes across tenants in accordance with an embodiment of the present invention.

Returning toFIG. 3, in conjunction withFIGS. 1-2, 4-7 and 8A-8B, in step305, data analyzer102presents the quality report to a user, such as the user of data analyzer102.

Leveraging multitenancy for aggregate analytics from a scalable system-level framework enables the present invention to efficiently monitor large heterogeneous healthcare datasets from multiple hospital organizations. Furthermore, it also enables the present invention to monitor the multi-stage distributed platform which regularly processes new and existing source data.

Furthermore, the present invention improves the technology or technical field involving electronic health record systems. As discussed above, electronic health record systems generate electronic health records (EHRs). EHRs may include a range of data, including demographics, medical history, medication and allergies, immunization status, laboratory test results, radiology images, vital signs, personal statistics, such as age and weight, and billing information. Ensuring good data quality in such records is essential due to the types of information stored in such records which are used to ensure the wellbeing of patients. However, defining and measuring quality in such records is challenging, especially in a multitenant analytical platform (e.g., multiple healthcare systems), due to several factors, such as the large data volume. There are hundreds of millions of EHRs being generated every day from healthcare systems. Furthermore, defining and measuring quality in such records is challenging because of the heterogeneous data sources. Different hospitals often use distinctive EHR systems that adopt various formats. Even within the same hospital, different departments may generate duplicate or contradicting records. Thus, the standard of quality of EHR varies based on the data sources. Additionally, defining and measuring quality in such records is challenging because of complex relations among variables. For example, there are millions of codes from standard ontologies (International Classification of Diseases (ICD), Systematized Nomenclature of Medicine (SNOMED®), RxNorm®, Common Procedure Terminology (CPT®), Logical Observation Identifiers Names and Codes (LOINC®), etc.) that describe diagnoses, medical procedures, lab tests, and drugs, not to mention countless custom codes. Further, defining and measuring quality in such records is challenging because of operational factors, such as utilizing multiple data processing pipelines that are each configured per tenant, regularly receiving new data and utilizing a continuous delivery development model. These complications may introduce quality issues unrelated to the source data quality. Hence, there is not currently a means for ensuring quality in the electronic health data.

The present invention improves such technology by ensuring quality in the electronic health data by tracking and analyzing many quality metrics measured against the electronic health record datasets generated by electronic health record systems at multiple stages of the data processing pipelines. As a result, there is an improvement in the technical field of electronic health record systems.

The technical solution provided by the present invention cannot be performed in the human mind or by a human using a pen and paper. That is, the technical solution provided by the present invention could not be accomplished in the human mind or by a human using a pen and paper in any reasonable amount of time and with any reasonable expectation of accuracy without the use of a computer.