Processing data records in a multi-tenant environment to ensure data quality

A computer system processes data records in a multi-tenant environment to ensure data quality. A plurality of records from a plurality of data sources are processed to provide a data quality metric for each field of the plurality of records based on record values in the field. A threshold range satisfying a specificity level of a data source is selected for each data quality metric. The data quality metric is compared to the threshold range to determine whether the data quality metric violates the threshold. A data quality report is provided for the plurality of records, wherein the data quality report indicates whether the data quality metric of each field violates the selected threshold range. Embodiments of the present invention further include a method and program product for processing data records in a multi-tenant environment to ensure data quality in substantially the same manner described above.

BACKGROUND

1. Technical Field

Present invention embodiments relate to processing data records, and more specifically, to processing data records in a multi-tenant environment to ensure data quality.

2. Discussion of the Related Art

A multi-tenant environment refers to a software architecture in which a single instance of software runs on a server and serves multiple tenants. Each tenant may include a group of clients who are provided with a dedicated share of the instance, including its data, configuration, user management, and the like. Ensuring that data records meet certain quality standards may be essential for downstream processes. However, ensuring the quality of data records may be difficult in a multi-tenant environment due to the large volume of data that can be generated by numerous tenants. Furthermore, each tenant may employ different data storage schemas and formats.

SUMMARY

According to one embodiment of the present invention, a computer system processes data records in a multi-tenant environment to ensure data quality. A plurality of records from a plurality of data sources are processed to provide a data quality metric for each field of the plurality of records based on record values in the field. A threshold range is selected for each data quality metric, wherein the threshold range is selected from a plurality of threshold ranges and satisfies a specificity level corresponding to the data source. The data quality metric is compared to the threshold range to determine whether the data quality metric violates the threshold. A data quality report is provided for the plurality of records, wherein the data quality report indicates whether the data quality metric of each field violates the selected threshold range. Embodiments of the present invention further include a method and program product for processing data records in a multi-tenant environment to ensure data quality in substantially the same manner described above.

DETAILED DESCRIPTION

Present invention embodiments relate generally to processing data records, and more specifically, to processing data records in a multi-tenant environment to ensure data quality. Ensuring record quality in a multi-tenant environment may be difficult since the unique practices of each tenant may preclude application of a standardized approach to ensuring data quality. For example, in a multi-tenant environment relating to electronic healthcare records, each tenant may represent a different health care system, each of which may have multiple physical locations, electronic record systems, and data stores. In a large healthcare system, hundreds of millions of electronic healthcare records may be generated each day. Furthermore, different facilities may use distinct data storage schemas and formats, and duplicate or contradicting records may be generated by various departments of a facility. Records of each patient may be longitudinal by nature, and with various ontologies to describe the diagnoses, procedures, and tests associated with treatment of a patient, even simple treatments may require the creation of multiple records, such as demographic records, observational records, diagnoses, billing records, and the like.

Present invention embodiments ensure the quality of data records in a multi-tenant environment by incorporating a flexible thresholding scheme in which general thresholds can be set to apply to common variables across all population levels, and specific thresholds can be added for specific sub-populations. Raw data records may be analyzed in order to generate data quality metrics for each field of the records, and each data quality metric may be compared to a particular threshold range that is selected based on a level of specificity relating to the source of the data records. Instead of using data quality reporting for healthcare data that depends on explicit thresholding for every measurement, present invention embodiments provide intelligent defaults which continue to improve as thresholds and weights are tuned for various combinations of data source properties, thereby providing a more efficient mechanism for the detection of poor data quality.

It should be noted that references throughout this specification to features, advantages, or similar language herein do not imply that all of the features and advantages that may be realized with the embodiments disclosed herein should be, or are in, any single embodiment of the invention. Rather, language referring to the features and advantages is understood to mean that a specific feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, discussion of the features, advantages, and similar language, throughout this specification may, but do not necessarily, refer to the same embodiment.

These features and advantages will become more fully apparent from the following drawings, description and appended claims, or may be learned by the practice of embodiments of the invention as set forth hereinafter.

Present invention embodiments will now be described in detail with reference to the Figures.FIG. 1is a block diagram depicting a computing environment100for ensuring the quality of data records in accordance with an embodiment of the present invention. As depicted, computing environment100includes a plurality of tenants105A-105N, a network120, a user device125, and a server150. It is to be understood that the functional division among components of computing environment100have been chosen for purposes of explaining present invention embodiments and is not to be construed as a limiting example.

Each tenant105A-105N may represent an organization and may include one or more sources of data within the organization, such as tenant servers110A-110N. For example, tenant105A may represent a first healthcare organization, and tenant105B may represent a second healthcare organization. In this example, tenant server110A may represent a particular department or sub-level of the first organization, such as an ambulatory department, and tenant server110N may represent a billing department of the first organization. Similarly, tenant server110X may represent a laboratory department of the second healthcare organization. It is to be understood that the depiction of the number of tenants and tenant servers of each tenant has been chosen for purposes of explaining present invention embodiments; various invention embodiments of computing environment100may include any number of tenants, each having any number of tenant servers from which data records may be collected. It is also to be understood that references to the healthcare industry and/or electronic healthcare records have been chosen for purposes of explaining present invention embodiments and should not be construed as limiting examples, as invention embodiments may ensure quality of data records pertaining to any field or domain of study.

Each tenant server, such as tenant servers110A-110N of tenant105A and tenant server110X of tenant105N, includes a database115, a network interface130, and a processor135. Each tenant server110may include a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, a thin client, or any programmable electronic device capable of executing computer readable program instructions. Network interface130enables components of each tenant server110to send and receive data over a network, such as network120. Each tenant server110may include internal and external hardware components, as depicted and described in further detail with respect toFIG. 6.

Each tenant server110A-110N and110X may include one or more databases, such as database115. Database115may include any non-volatile storage media known in the art. For example, database115can be implemented with a tape library, optical library, one or more independent hard disk drives, or multiple hard disk drives in a redundant array of independent disks (RAID). Similarly, data on database115may conform to any suitable storage architecture known in the art, such as a file, a relational database, an object-oriented database, and/or one or more tables. Database115may store data corresponding to data records of a tenant. Records may be organized according to a hierarchy, with each database115corresponding to a particular hierarchical level of an organization.

Network120may include a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the two, and includes wired, wireless, or fiber optic connections. In general, network120can be any combination of connections and protocols known in the art that will support communications between tenants105A-105N, user device125, and server150via their respective network interfaces130in accordance with embodiments of the present invention.

User device125includes memory140, a network interface130, and at least one processor135. Memory140includes an administration module145. In various embodiments of the present invention, each user device125may include a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, a thin client, or any programmable electronic device capable of executing computer readable program instructions. Network interface130enables components of each user device125to send and receive data over a network, such as network120. Each client device105may include internal and external hardware components, as depicted and described in further detail with respect toFIG. 6.

Administration module145may include one or more modules or units to perform various functions of present invention embodiments described below. Administration module145may be implemented by any combination of any quantity of software and/or hardware modules or units, and may reside within memory140of user device125for execution by a processor, such as processor135.

Administration module145may enable a user of user device125to monitor the quality of data records of tenants105A-105N in order to ensure that the data records meet a certain level of quality. Administration module145may generate reports for an administrator that indicate whether data records satisfy predefined quality thresholds. Administration module145may enable a user of user device125to specify or modify acceptable thresholds ranges for data records. In some embodiments, administration module145provides threshold ranges to threshold module165of server150.

Server150includes memory155, a network interface130, at least one processor135, and a database180. Memory155may include a data analysis module160, a threshold module165, an anomaly detection module170, and a report module175. In general, server150monitors data records and ensures that data records meet quality metrics. Server150may include internal and external hardware components, as depicted and described in further detail with respect toFIG. 6.

Data analysis module160, threshold module165, anomaly detection module170, and report module175may include one or more modules or units to perform various functions of present invention embodiments described below. Data analysis module160, threshold module165, anomaly detection module170, and report module175may be implemented by any combination of any quantity of software and/or hardware modules or units, and may reside within memory155of server150for execution by a processor, such as processor135.

Data analysis module160may analyze raw data records that are received from one or more databases115of tenants105A-105N. In general, data analysis module160may analyze raw data records to generate data quality metrics. Each record may have one or more fields with a separate record value for each field. In some embodiments, data analysis module160generates a data quality metric for each field of a group of records received from the same data source. Data analysis module160may analyze data records periodically, or may analyze records in an ad hoc fashion as the records are made available by a tenant server. In some embodiments, data analysis module160is implemented using Spark®, and data records are stored in the Hadoop® Distributed File System (HDFS). Data analysis module160may analyze records from many data sources (e.g., databases115of tenant servers110) in parallel. While data sources may contain nested objects, such as in Apache Avro™, data analysis module160may flatten each object into a table row containing a key column corresponding to the data source (e.g., database115), and value columns containing raw data of the records.

Threshold module165may manage threshold ranges that are applied to data quality metrics. Each threshold range may include a numerical interval that provides boundaries for favorable values of a data quality metric. In some embodiments, a user of user device125may provide threshold ranges to threshold module165via administration module145. Threshold module165may organize threshold ranges according to the data quality metrics to which each threshold range applies, and by data source. For example, there may be one set of threshold ranges that are applied to records originating from a first location, and another set of threshold ranges that are applied to records originating from a second location. The difference in threshold ranges may reflect difference requirements of the first and second location. For example, a billing department may not populate laboratory result fields, but a laboratory department would populate that field, so the billing department may select a threshold range reflecting the expectation that 0% of the billing records should have laboratory result fields that are populated, whereas the laboratory department might select a threshold range of 50%-100%, since at least half of the laboratory records should have laboratory result fields that are populated.

Threshold module165may support a rollup threshold schema in which general threshold ranges are provided for fields and metrics that apply to all population levels of a tenant, and in which explicit threshold ranges can be provided for fields and metrics at any sub-population level. For example, a threshold range may be universally applied to the birth date field of all records, a broad threshold range might be applied to a particular description code field of records in a hospital, and a narrower threshold range might be applied to a particular department (e.g., sub-population) of the hospital that has stricter data quality requirements. Threshold module165may store threshold ranges in a table organized according to data source and record field, and may check the integrity of the threshold table in order to find errors such as duplicate entries, record values falling outside of an acceptable range, or conflicting rows. In some embodiments, threshold module165stores threshold ranges in a relational database.

Anomaly detection module170may apply threshold ranges to data quality metrics in order to identify any records that may be indicative of poor data quality. Anomaly detection module170may use threshold ranges that are matched with the most specific-matching population keys of records. A population key indicates the source of the data record field with the greatest amount of specificity. For example, a population key may specify that a field of a data record originated from a particular sub-department of an organization. Once a matching threshold range is selected, anomaly detection module170compares the corresponding data quality metric to the threshold range. If a data quality metric satisfies the threshold range, the data quality metric is labeled as good, and if not, then the data quality metric is labeled as poor.

Report module175may generate a report that visualizes the quality of data records. In some embodiments, report module175generates a report in a spreadsheet format, such as Microsoft® Excel®. Report module175may share the report with one or more members of an organization, such as a user of user device125. In some embodiments, report module175may notify particular members of an organization when a relevant quality metric does not satisfy the selected threshold range. For example, if a particular department of an organization has low data quality, report module175may notify a designated user who is a member of that department.

Database180may include any non-volatile storage media known in the art. For example, database180can be implemented with a tape library, optical library, one or more independent hard disk drives, or multiple hard disk drives in a redundant array of independent disks (RAID). Similarly, data on database180may conform to any suitable storage architecture known in the art, such as a file, a relational database, an object-oriented database, and/or one or more tables. Database180may store data corresponding to data records, data quality metrics, threshold ranges, and data quality reports.

FIG. 2is a flow chart depicting a method of ensuring the quality of data records in a multi-tenant environment in accordance with an embodiment of the present invention.

Data records from data sources of one or more tenants are received at operation210. Data records may be received from data sources spanning one or more tenants in a multi-tenant environment, such as database115of tenant server110A, database115of tenant server110N, database115of tenant server110X, and the like. Each data record may include multiple fields each having separate record values. For example, a data record may have a name field, a gender field, and a date field, with each field containing a separate record value. Data records may be received by server150periodically, or a record or set of records may be sent to server150when the record or set of records becomes available.

The data records are processed to generate data quality metrics for each record field at operation220. The raw data values of data records may be processed by counting the record values of each field of a set of records according to a particular quality. Counts may be stored in a histogram, and data quality metrics may then be generated. For example, record values that are null or empty may be counted and compared to record values that are non-null, and a data quality metric may be determined that reflects the percentage of null record values in a given field of a set of records. Various invention embodiments may compare the number of record values that contain any error, such as null entries, typographical errors, values falling outside of an acceptable range, dates in the future for historical data records (e.g., future birthdates), and the like. Data quality metrics may be generated by data analysis module160. The generation of data quality metrics will be described in further detail below with reference toFIGS. 3A-3D.

A threshold range is selected for each data quality metric at operation230. Threshold ranges may be selected based on a combination of data field and level of specificity of the data source of a set of records. According to a rollup threshold schema, a data quality metric may be matched with a threshold range that is general to the field and metric and applies to all population levels, unless there is a more specific threshold range available for the data quality metric. In some embodiments, threshold ranges are selected by threshold module165and/or anomaly detection module170of server150.

The data quality metric is compared to the selected threshold range at operation240. Each data quality metric may be compared to a selected threshold range by anomaly detection module170, which determines whether the data quality metric falls within the selected threshold range. A data quality report is generated at operation250that indicates whether data quality metrics fall within the selected threshold ranges. The data quality report may be generated by report module175of server150, and may be sent to one or more users of an organization, such as a user of user device125. Data quality reports may be stored in database180. In some embodiments, a user of user device125can access past and present data quality reports using administration module145.

FIGS. 3A-3Dare tables depicting stages of processing data records in accordance with an embodiment of the present invention.FIG. 3Adepicts a table300of data records that are organized according to the data source, as represented by customer field305and system field310. Table300may be received by data analysis module160from a single tenant server, such as tenant server110X. As depicted, there is one customer K1, and there are two sub-populations, K2A and K2B, that are sources of data records. Customer K1may represent a single tenant in a multi-tenant environment, and systems K2A and K2B may represent different facilities of the organization associated with the tenant. ID field315, gender field320, and birth year field325are fields that contain record values for each record of table300.

FIG. 3Bdepicts a table330of key-value combinations, and may be generated by data analysis module160. Key field335may represent each unique combination of data sources and fields, value field340may represent a indicate value, and count field345may indicate the count of that value for the combination of key and field. ValueA may correspond to an ID field value, valueB may correspond to a gender field value, and valueC may correspond to a birth year value. Each unique key-value combination is given its own row in table330with the count value345indicating the number of occurrences. For example, since key K1, K2A, valueA has three distinct values in table300(1142,1143, and1144), then table330contains three corresponding key-value-count rows. Furthermore, since the key K1, K2B, valueA combination has two instances of null values, the value of null is indicated in value field340, and the count of two is indicated in count field345.

FIG. 3Cdepicts a table350of histograms by key. In some embodiments, data analysis module160generates table350based on table330. Table350thus includes each unique key combination in key field355, and the corresponding histogram information in histogram field360. Using the histogram information, data analysis module160may then generate table365(FIG. 3D) by evaluating each row of table350. Table365includes key field370, metric field375, and result field380. In this example, number of null values and number of distinct values are computed as separate counts. In various embodiments, application-specific logic can determine which metrics should be calculated for a given key based on factors such as the data type, originating column, or data source. Thus, result field380may contain data quality metrics that are specific to a data source and a record field. The data quality metrics may then be compared to a threshold range in order to evaluate the data quality. For example, key-value combination K1, K2B, valueA may indicate poor quality records if the data threshold dictates that there should be no more than one null entry, since the null entry count of K1, K2B, valueA is two.

FIG. 4Ais a table400of data quality thresholds in accordance with an embodiment of the present invention. As depicted, table400includes a row number field405, three population level fields410,415, and420, a field indication field425, a metric field430, and a threshold range field435. Table400may be an example of a rollup threshold schema in which threshold ranges are defined according to various levels of specificity of a data source and field, with asterisks representing wildcards.

Row number field405may indicate an ID of each combination of population specificity, record field, and data quality metric. Population level fields410,415, and420correspond to levels of increasing specificity of data sources of a tenant. For example, rows1and2may apply to any tenant, since all population level entries are wildcards. Similarly, row3may apply to any data source at Hospital1of Tenant1, and row4may apply to a data source associated with the billing department of Hospital1of Tenant1. Threshold ranges may thus be provided at any level of specificity by defining data threshold ranges for every combination of data source and field.

FIG. 4Bis a table depicting a data quality report440in accordance with an embodiment of the present invention. Data quality report440includes an indication of data sources via population level fields410,415, and420, a field indication field425indicating the field of data records, a metric field430indicating the data quality metric associated with the field, and the value of the data metric in metric value field445. Row field405indicates the rows that have been matched to the combinations of data source and fields, and matched thresholds field435indicates the threshold ranges that correspond to those combinations. Data quality report440further includes quality field450, which indicates whether the value of each entry in metric value field445falls within the corresponding threshold ranges of matched threshold field435. As depicted, data quality report440indicates that row2has poor data quality, as the ICD % data quality metric is 30%, which is outside of the threshold range of 50%-100%.

FIGS. 5A-5Care tables depicting a weighted thresholding schema in accordance with an embodiment of the present invention. Table500depicts threshold settings for various population level fields505-525. Table530depicts weight fields535-555, with each weight corresponding to a population level in table500.

Table560depicts resulting metric thresholding decisions. Score field565indicates scores calculated for a given row of table560against each row of table500, and threshold row570indicates the thresholds that are selected as a result of the calculated scores. For example, the first row of table560has a score of zero as evaluated against the first row of table500, since none of the population level fields505-525match. The first row of table560has a score of twelve as evaluated against the second row of table500, since “Hospital,” and “Labor and Delivery” match. As those correspond to weight fields540and545, weight field540has a weight of four, and weight field545has a weight of eight, the total for score is twelve. Wildcard values (e.g., as indicated by asterisks) may be ignored, and fields that do not match may be treated as negative values and therefore deducted. Once a row of table560is evaluated against all four rows of table500, the highest value may selected, and the corresponding threshold may be applied. The highest value in score field565of first row of table560is sixteen, which is the score when evaluated against the fourth row of table500(as “E4112” and “LD043” both match, and their weights are eight each). Therefore, the threshold range of the fourth row, “48-52%,” is applied to the first row of table560.

In some embodiments, the weights and/or threshold ranges are dynamically tunable based on analysis of prior record quality history. Conventional or other machine learning techniques may be utilized to modify population level values and threshold range values of entries in table500, and/or to add new entries or to remove entries. Similarly, conventional or other machine learning techniques may be employed to add, remove, or modify weights of table530. For example, machine learning may be used to provide newly-added data sources with threshold ranges and/or weights based on the similarity of the newly-added data sources to pre-existing data sources.

FIG. 6is a block diagram depicting components of a computer10suitable for executing the methods disclosed herein. Computer10may implement tenant servers110A-110N, tenant server110X, user device125, and/or server150in accordance with embodiments of the present invention. It should be appreciated thatFIG. 6provides only an illustration of one embodiment and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

As depicted, the computer10includes communications fabric12, which provides communications between computer processor(s)14, memory16, persistent storage18, communications unit20, and input/output (I/O) interface(s)22. Communications fabric12can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system. For example, communications fabric12can be implemented with one or more buses.

Memory16and persistent storage18are computer readable storage media. In the depicted embodiment, memory16includes random access memory (RAM)24and cache memory26. In general, memory16can include any suitable volatile or non-volatile computer readable storage media.

One or more programs may be stored in persistent storage18for execution by one or more of the respective computer processors14via one or more memories of memory16. The persistent storage18may be a magnetic hard disk drive, a solid state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer readable storage media that is capable of storing program instructions or digital information.

The media used by persistent storage18may also be removable. For example, a removable hard drive may be used for persistent storage18. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer readable storage medium that is also part of persistent storage18.

Communications unit20, in these examples, provides for communications with other data processing systems or devices. In these examples, communications unit20includes one or more network interface cards. Communications unit20may provide communications through the use of either or both physical and wireless communications links.

I/O interface(s)22allows for input and output of data with other devices that may be connected to computer10. For example, I/O interface22may provide a connection to external devices28such as a keyboard, keypad, a touch screen, and/or some other suitable input device. External devices28can also include portable computer readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards.

Software and data used to practice embodiments of the present invention can be stored on such portable computer readable storage media and can be loaded onto persistent storage18via I/O interface(s)22. I/O interface(s)22may also connect to a display30. Display30provides a mechanism to display data to a user and may be, for example, a computer monitor.

Data relating to ensuring quality of data records in a multi-tenant environment (e.g., data records, data quality metrics, threshold ranges, organizational hierarchical data, etc.) may be stored within any conventional or other data structures (e.g., files, arrays, lists, stacks, queues, records, etc.) and may be stored in any desired storage unit (e.g., database, data or other repositories, queue, etc.). The data transmitted between tenant servers110, user device125, and/or server150may include any desired format and arrangement, and may include any quantity of any types of fields of any size to store the data. The definition and data model for any datasets may indicate the overall structure in any desired fashion (e.g., computer-related languages, graphical representation, listing, etc.).

Data relating to ensuring quality of data records in a multi-tenant environment (e.g., data records, data quality metrics, threshold ranges, organizational hierarchical data, etc.) may include any information provided to, or generated by, tenant servers110, user device125, and/or server150. Data relating to ensuring quality of data records in a multi-tenant environment may include any desired format and arrangement, and may include any quantity of any types of fields of any size to store any desired data. The data relating to ensuring quality of data records in a multi-tenant environment may include any data collected about entities by any collection mechanism, any combination of collected information, and any information derived from analyzing collected information.

It will be appreciated that the embodiments described above and illustrated in the drawings represent only a few of the many ways of processing data records in a multi-tenant environment to ensure data quality.

It is to be understood that the software (e.g., server software, networking software, administration module145, data analysis module160, threshold module165, anomaly detection module170, report module175, etc.) of the present invention embodiments may be implemented in any desired computer language and could be developed by one of ordinary skill in the computer arts based on the functional descriptions contained in the specification and flow charts illustrated in the drawings. Further, any references herein of software performing various functions generally refer to computer systems or processors performing those functions under software control. The computer systems of the present invention embodiments may alternatively be implemented by any type of hardware and/or other processing circuitry.

The software of the present invention embodiments (e.g., server software, networking software, administration module145, data analysis module160, threshold module165, anomaly detection module170, report module175, etc.) may be available on a non-transitory computer useable medium (e.g., magnetic or optical mediums, magneto-optic mediums, floppy diskettes, CD-ROM, DVD, memory devices, etc.) of a stationary or portable program product apparatus or device for use with stand-alone systems or systems connected by a network or other communications medium.

The system may employ any number of any conventional or other databases, data stores or storage structures (e.g., files, databases, data structures, data or other repositories, etc.) to store information (e.g., data relating to ensuring quality of data records in a multi-tenant environment). The database system may be implemented by any number of any conventional or other databases, data stores or storage structures (e.g., files, databases, data structures, data or other repositories, etc.) to store information (e.g., data relating to ensuring quality of data records in a multi-tenant environment). The database system may be included within or coupled to the server and/or client systems. The database systems and/or storage structures may be remote from or local to the computer or other processing systems, and may store any desired data (e.g., data relating to ensuring quality of data records in a multi-tenant environment).