Limited data enricher

Embodiments of the present invention disclose a method, a computer program product, and a computer system for enriching data. A computer receives a limited dataset and a target insight. In addition, the computer identifies an applicable richer dataset and determines a population relatedness between the limited dataset and the applicable richer dataset. Moreover, the computer calculates estimators of features using the richer dataset as well as calculates estimators for features using the limited dataset. The computer then updates the estimators of the limited dataset using estimators of the richer dataset and evaluates the updated estimators. Lastly, the computer provides a data collection recommendation as it relates to the limited dataset based on the evaluation.

BACKGROUND

The present invention relates generally to data analysis, and more particularly to enriching limited data sets using richer data sets.

Healthcare data is typically collected to serve a certain purpose, and thus only features relevant to the purpose are collected. For instance, Electronic Medical Record (EMR) data typically collects features related to clinical activity while claim data collects features relevant to administrative or financial transactions. As time goes on, one can find additional features relevant to the purpose, however collecting these additional features can be expensive. One may need new machine and/or employee training to collect new information, or need to additionally change the data collection process. For example, a data collection protocol, database architecture, storage space, etc. may need be changed or updated in order to collect the new features. Thus to collect additional features, one needs to assure that the benefit is greater than the cost.

SUMMARY

Embodiments of the present invention disclose a method, a computer program product, and a computer system for enriching limited data using richer data.

According to some embodiments, the method includes a computer receiving a limited dataset and a rich dataset, wherein the limited dataset details one or more limited features, and wherein the rich dataset details the one or more limited features and at least one additional feature. In such embodiments, the method may further include the computer determining one or more first feature estimators corresponding to the one or more limited features based on the limited dataset and the computer determining one or more second feature estimators corresponding to the one or more limited features and the at least one additional feature based on the richer dataset.

In some embodiments, the method further comprises the computer updating the one or more first feature estimators based on the one or more second feature estimators, the computer evaluating the updated one or more first feature estimators with regard to a target insight, and the computer providing a feature collection recommendation based on the evaluation.

In other embodiments of the method, evaluating the updated one or more first feature estimators further comprises determining a first effect on the target insight based on applying a model having the one or more first feature estimators to the limited dataset, determining a second effect on the target insight based on applying a model having the one or more second feature estimators to the limited dataset, and comparing the first effect on the target insight to the second effect on the target insight.

According to some embodiments, the method may further comprise determining that the limited dataset lacks a corresponding value to apply the model having the one or more second feature estimators, imputing the corresponding value, and applying the model having the one or more second feature estimators to the limited dataset.

Moreover, in some embodiments, the feature collection recommendation includes at least one of one or more of the at least one additional feature for inclusion in the limited dataset and an amount of data to be collected corresponding to the one or more of the at least one additional feature. In further embodiments, the feature collection recommendation includes the second effect on the target insight.

In addition, and according to one embodiment, receiving the limited dataset and the rich dataset further comprises determining a population relatedness between the limited dataset and one or more candidate richer datasets, then receiving a selection of the rich dataset of the one or more candidate richer datasets.

According to other embodiments, updating the one or more first feature estimators based on the one or more second feature estimators further comprises at least one of replacing the one or more first feature estimators with the one or more second feature estimators, replacing the one or more first feature estimators based on an average of the one or more first feature estimators and the one or more second feature estimators, and replacing the one or more first feature estimators based on a weighted average of the one or more first feature estimators and the one or more second feature estimators.

According to another embodiment of the present invention, a computer program product is disclosed. The computer program product includes one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices.

In addition, the computer program product includes program instructions comprising program instructions to receive a limited dataset and a rich dataset, wherein the limited dataset details one or more limited features, and wherein the rich dataset details the one or more limited features and at least one additional feature. In addition, the computer program product further comprises program instructions to determine one or more first feature estimators corresponding to the one or more limited features based on the limited dataset and one or more second feature estimators corresponding to the one or more limited features and the at least one additional feature based on the richer dataset.

In embodiments, the computer program product further includes program instructions to update the one or more first feature estimators based on the one or more second feature estimators, program instructions to evaluate the one or more updated first feature estimators with regard to a target insight, and program instructions to provide a feature collection recommendation based on the evaluation.

According to some embodiments, the program instructions to evaluate the updated one or more first feature estimators further comprise program instructions to determine a first effect on the target insight based on applying the one or more first feature estimators to the limited dataset, program instructions to determine a second effect on the target insight based on applying the one or more second feature estimators to the limited dataset, and program instructions to compare the first effect on the target insight to the second effect on the target insight.

In other embodiments, the computer program product further includes program instructions to determine that the limited dataset lacks a corresponding value to apply the one or more second feature estimators, program instructions to impute the corresponding value; and program instructions to apply the model having the one or more second feature estimators to the limited dataset.

In some embodiments, the feature collection recommendation includes at least one of one or more of the at least one additional feature for inclusion in the limited dataset, and an amount of data to be collected corresponding to the one or more of the at least one additional feature. In further embodiments, the feature collection recommendation further includes the second effect on the target insight.

According to another embodiment, the program instructions to receive the limited dataset and the rich dataset further comprise program instructions to determine a population relatedness between the limited dataset and one or more candidate richer datasets, and program instructions to receive a selection of the rich dataset of the one or more candidate richer datasets.

Moreover, in other embodiments, the program instructions to update the one or more first estimators based on the one or more second estimators further comprise at least one of program instructions to replace the one or more first estimators with the one or more second estimators, program instructions to replace the one or more first estimators based on an average of the one or more first estimators and the one or more second estimators, and program instructions to replace the one or more first estimators based on a weighted average of the one or more first estimators and the one or more second estimators.

Additionally disclosed is a computer system for enriching data, wherein the computer system comprises one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories.

In embodiments of the computer system, the program instructions comprise program instructions to receive a limited dataset and a rich dataset, wherein the limited dataset details one or more limited features, and wherein the rich dataset details the one or more limited features and at least one additional feature. In addition, the computer system further comprises program instructions to determine one or more first feature estimators corresponding to the one or more limited features based on the limited dataset and one or more second feature estimators corresponding to the one or more limited features and the at least one additional feature based on the richer dataset.

In embodiments, the computer system may further includes program instructions to update the one or more first feature estimators based on the one or more second feature estimators, program instructions to evaluate the one or more updated first feature estimators with regard to a target insight, and program instructions to provide a feature collection recommendation based on the evaluation.

According to some embodiments, the program instructions to evaluate the updated one or more first feature estimators further comprise program instructions to determine a first effect on the target insight based on applying the one or more first feature estimators to the limited dataset, program instructions to determine a second effect on the target insight based on applying the one or more second feature estimators to the limited dataset, and program instructions to compare the first effect on the target insight to the second effect on the target insight.

In other embodiments, the computer system further includes program instructions to determine that the limited dataset lacks a corresponding value to apply the one or more second feature estimators, program instructions to impute the corresponding value; and program instructions to apply the model having the one or more second feature estimators to the limited dataset.

In some embodiments, the feature collection recommendation includes at least one of one or more of the at least one additional feature for inclusion in the limited dataset, and an amount of data to be collected corresponding to the one or more of the at least one additional feature. In further embodiments, the feature collection recommendation further includes the second effect on the target insight.

According to another embodiment, the program instructions to receive the limited dataset and the rich dataset further comprise program instructions to determine a population relatedness between the limited dataset and one or more candidate richer datasets, and program instructions to receive a selection of the rich dataset of the one or more candidate richer datasets.

DETAILED DESCRIPTION

As previously mentioned, healthcare data is typically collected to serve certain purpose, and thus only features relevant to the purpose are collected. Moreover, collecting these additional features can be expensive and require new machine and/or employee training to collect new information, or changes to the data collection process, for example changing data collection protocol, database architecture, storage space, etc. Thus, to collect features in additional to those already in a data set, one needs to be sure that the benefit is greater than the cost.

Suppose in addition to the data set having limited features, known as the limited data set, there is another data set having more features, known as a richer data set. The richer data can aid in the analysis of the limited data in multiple aspects as a gold standard. The richer data can help one to evaluate which additional features should be collected and the amount of additional data to collect to better serve the purpose. Also, based on many transfer learning methods, richer data can calibrate analytic outputs from the limited data to improve accuracy/confidence.

Many statistical methods that transfer statistical learning from a richer dataset to a limited dataset have been developed. The details vary greatly based on analysis goal, population heterogeneity, data pooling availability, etc.; and among methods for similar conditions, many different ideas have been proposed. Considering there is no universal method, one needs to test many methods to find one that performs the best for the limited data and the problem-at-hand.

The proposed invention is an analytics system that utilizes information from richer data to analyse limited data. Specifically, it focuses on using the richer data to calibrate analytical outputs from limited data and guide future data collection for the limited data.

A data enriching system100in accordance with an embodiment of the invention is illustrated byFIG. 1. In the example embodiment, data enriching system100includes server110, server120, and server130. While, in the example embodiment, data and programming of the present invention are stored remotely across multiple servers within a network, in other embodiments, data and programming of the present invention may be contained on a single computing device locally.

In the example embodiment, network108is a communication channel capable of transferring data between connected devices. In the example embodiment, network108may be the Internet, representing a worldwide collection of networks and gateways to support communications between devices connected to the Internet. Moreover, network108may include, for example, wired, wireless, or fiber optic connections which may be implemented as an intranet network, a local area network (LAN), a wide area network (WAN), or any combination thereof. In further embodiments, network108may be a Bluetooth network, a WiFi network, or a combination thereof. In yet further embodiments, network108may be a telecommunications network used to facilitate telephone calls between two or more parties comprising a landline network, a wireless network, a closed network, a satellite network, or any combination thereof. In general, network108can be any combination of connections and protocols that will support communications between server110, server120, and server130.

Server110includes richer data112, analytics engine114, model repository116, and model outputs118. In the example embodiment, server110is a computing device that may be a server, a laptop computer, a notebook, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a touchtone phone, a smart phone, a mobile phone, a virtual device, a thin client, or any other electronic device or computing system capable of receiving and sending data to and from other computing devices. While server110is shown as a single device, in other embodiments, server110may be comprised of a cluster or plurality of computing devices, working together or working separately. Server110is described in more detail with reference toFIG. 5.

Richer data112is a collection of information contained in files, folders, and other document types that have many features, or discernible characteristics. In the example embodiment, richer data112may be a corpora of documents which detail bodies of categorized and subject specific data, such as medical, legal, and financial data. In other embodiments, richer data112may include uncategorized data of miscellaneous topics. In the example embodiment, however, richer data112details patient medical information such as patient gender, age, location, health conditions, etc. in the form of electronic medical records (EMRs). Such information may be structured (i.e. have associated metadata), partially structured, or unstructured. Moreover, data within richer data112may be written in programming languages of common file formats such as .docx, .doc, .pdf, .rtf, .jpg, .csv, .txt, etc. In further embodiments, richer data112may include handwritten and other documents scanned or otherwise converted into electronic form. Richer data112is described in greater detail with respect toFIGS. 2-4.

Analytics engines114are computer software and/or hardware modules that analyse the data stored in richer data112. In the example embodiment, analytics engines114are utilized to perform statistical operations, such as determining a percentage, mean, standard deviation, range distribution, etc., as well as perform more complex operations such as predictive modelling and effectiveness evaluation. Analytics engines114are described in greater detail with reference toFIGS. 2-4.

Model repository116is a collection of data detailing one or more models that are used to analyse the data stored in richer data112. In the example embodiment, model repository116contains models that use the features and estimators of richer data112. Moreover, such models may output values such as predictions of future events and effectiveness of medications. In the example embodiment, models in model repository116, and more specifically features and estimators thereof, are used to calibrate and enrich limited data. Model repository116is described in greater detail with respect toFIGS. 2-4.

Model outputs118are a collection of data detailing outputs from the models contained in model repository116. In the example embodiment, data in model outputs118includes the output values that result from applying the models contained in model repository116to the data in richer data112. Moreover, such values may relate to predictions of future events and effectiveness of medications. Model outputs118are described in greater detail with respect toFIGS. 2-4.

Server120includes limited data122, analytics engine124, model repository126, and model outputs128. In the example embodiment, server120is a computing device that may be a server, a laptop computer, a notebook, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a touchtone phone, a smart phone, a mobile phone, a virtual device, a thin client, or any other electronic device or computing system capable of receiving and sending data to and from other computing devices. While server120is shown as a single device, in other embodiments, server120may be comprised of a cluster or plurality of computing devices, working together or working separately. Server120is described in more detail with reference toFIG. 5.

Limited data122is a collection of information contained in files, folders, and other document types that has less features, or discernible characteristics, than that of richer data112. In the example embodiment, limited data122may be a corpora of documents which detail bodies of categorized and subject specific data, such as medical, legal, and financial data. In other embodiments, limited data122may include uncategorized data of miscellaneous topics. In the example embodiment, limited data122details patient medical information such as patient gender, age, location, health conditions, etc. in the form of electronic medical records (EMRs). Such information may be structured (i.e. have associated metadata), partially structured, or unstructured. Moreover, data within limited data122may be written in programming languages of common file formats such as .docx, .doc, .pdf, .rtf, .jpg, .csv, .txt, etc. In further embodiments, limited data122may include handwritten and other documents scanned or otherwise converted into electronic form. Limited data122is described in greater detail with respect toFIGS. 2-4.

Analytics engines124are computer software and/or hardware modules that analyse the data stored in limited data122. In the example embodiment, analytics engines124are utilized to perform statistical operations, such as determining a percentage, mean, standard deviation, range distribution, etc., as well as perform more complex operations such as predictive modelling and effectiveness evaluation. Analytics engines124are described in greater detail with reference toFIGS. 2-4.

Model repository126is a collection of data detailing one or more models that are used to analyse the data stored in limited data122. In the example embodiment, model repository126contains models that use the features and estimators of limited data122. Moreover, such models may output values such as predictions of future events and effectiveness of medications. In the example embodiment, models in model repository126, and more specifically features and estimators thereof, are calibrated and enriched by those of richer data112, as is described in greater detail with respect toFIGS. 2-4.

Model outputs128are a collection of data detailing outputs from the models contained in model repository126. In the example embodiment, data in model outputs128includes the output values that result from applying the models contained in model repository126to the data in limited data122. Moreover, such values may relate to predictions of future events and effectiveness of medications. In the example embodiment, outputs contained in model outputs128are utilized in determining whether the data enrichment process described herein improves limited data122by using the data as a baseline pre-enrichment value from which post-enrichment values may be compared. Model outputs128are described in greater detail with respect toFIGS. 2-4.

Server130includes data enricher132, analytics engines134, transfer learning meta data136, updated model repository137, updated outputs138, and model performance results139. Server130may be a laptop computer, a notebook, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a rotary phone, a touchtone phone, a smart phone, a mobile phone, a virtual device, a thin client, or any other electronic device or computing system capable of receiving and sending data to and from other computing devices. While server130is shown as a single device, in other embodiments, server130may be comprised of a cluster or plurality of computing devices, working together or working separately. Server130is described in more detail with reference toFIG. 5.

In some embodiments, server130may further comprise a user interface which allows a user of server130to interact with server130as well as other connected devices via network108. In addition, the user interface may be connectively coupled to hardware components, such as those depicted byFIG. 5, for receiving user input, including mice, keyboards, touchscreens, microphones, cameras, and the like. In the example embodiment, the user interface may implemented via a web browsing application that contains a graphical user interface (GUI) and display that is capable of transferring data files, folders, audio, video, hyperlinks, compressed data, and other forms of data transfer individually or in bulk. In other embodiments, the user interface may be implemented via other integrated or standalone software applications and hardware capable of receiving user interaction and communicating with other electronic devices.

Data enricher132is a software program capable of enriching a limited data set based on a richer data set. More specifically, data enricher132is capable of receiving limited data122and a target insight. In addition, data enricher132is capable of identifying an applicable richer data112and determining a population relatedness between limited data122and the applicable richer data112. Moreover, data enricher132is capable of calculating estimators of features using richer data112as well as calculating estimators for features using limited data122. Data enricher132is additionally capable of updating estimators of limited data122using estimators of richer data112and evaluating the updated estimators. Lastly, data enricher132is capable of providing a data collection recommendation for limited data122based on the evaluation. The operations of data enricher132are described in greater detail with respect toFIGS. 2-4.

Analytics engines134are computer software and/or hardware modules that analyse data. In the example embodiment, analytics engines134are utilized to perform statistical operations, such as determining a percentage, mean, standard deviation, range distribution, etc., as well as perform more complex operations such as predictive modelling and effectiveness evaluation. Analytics engines134are described in greater detail with reference toFIGS. 2-4.

Transfer learning meta data136is a collection of data detailing transfer learning methods and techniques. In the example embodiment, transfer learning methods are used to enrich limited data122based on richer data112. Meta data stored in transfer learning meta data136may vary by the goal of the analysis, but for example may include adjustment factors for effectiveness evaluation (shown inFIG. 4) or imputation rules for predictive models (shown inFIG. 3). In further embodiments, transfer learning meta data136may also include any hyper-parameter and transferring factor or techniques learned from richer data112.

Updated model repository137is a collection of data detailing updated versions of the models contained in model repository126. In the example embodiment, for instance, updated model repository137includes models contained in model repository126that have been updated in accordance with models contained in model repository116. Moreover, updates to the models stored in updated model repository137may describe updates performed at a raw, coefficient, or output level based on analysis of limited data122in light of richer data112.

Updated outputs138is a collection of data detailing the updated outputs of the updated models contained in updated model repository137. In the example embodiment, data in updated outputs138is compared to model outputs128in order to determine whether updating the model(s) in model repository126has improved data analysis. Effectively, updated outputs138serves as a post-enrichment values from which to compare the pre-enrichment values of model outputs128.

Model performance results139is a collection of data detailing a resultant change in performance between data outputs of model outputs128and data outputs of updated models in updated model repository137. In the example embodiment, model performance results may relate to a particular result, what is known as a target insight, and may include, for example, model prediction accuracy as it relates to a particular event or medication effectiveness as it relates to a particular medication.

FIG. 2illustrates the operations of data enricher132in enriching limited data122. More specifically, data enricher132enriches limited data122by determining which features that are lacking from limited data122would have a greatest impact on a particular output, referred to herein as a target insight, were the features included (or included in greater amounts) in limited data122. For example, a target insight may be a feature having a greatest impact on the accuracy of a prediction model, or a feature having a greatest impact on medication effectiveness. In the example embodiment, data enricher132enriches limited data122based on analysis of richer data112and provides a data collection recommendation. In some embodiments, the data collection recommendation may include recommended feature(s) to collect that would result a greatest impact on the target insight, an amount of data corresponding to the recommended feature(s) needed for meaningful analysis of the target insight, and a predicted effect on the target insight if the recommended feature in the recommended amount were included in limited data122. In the example embodiment, data enricher132operates in three modules (illustrated byFIGS. 2-4): population relatedness210, data analysis and update output220, and data collection recommendation230.FIGS. 2-4describe this process in greater detail.

Data enricher132determines a population relatedness between limited data122and richer data112at population relatedness210. In the example embodiment, population relatedness210includes receiving a limited data set and target insight212, identifying an applicable richer data set214, and determining a population relatedness216.

In the example embodiment, the process begins by data enricher132first identifying or receiving limited data122which lacks one or more features relevant to a target insight (step212). The target insight, as used herein, may be any desired insight or outcome as it relates to analysis of limited data122, for example an accuracy of predictions or effectiveness of medications that may deduced from limited data122. To that point, if the target insight were, for instance improved model accuracy, data enricher132may determine that inclusion of data for a particular feature not currently represented or underrepresented in limited data122may result in a better accuracy of a corresponding model. Alternatively, for instance if the target insight were improving medication effectiveness, data enricher132may identify a feature not represented or underrepresented in limited data122that has a significant impact on the medication's effectiveness.

For example, data enricher132receives limited data122which details data of patients who have acquired type II diabetes. More specifically, limited data122includes data corresponding to the four features of gender, age, body mass index (BMI), and cholesterol. In addition, data enricher132receives a target insight as it relates to the analysis of limited data122, for example improving accuracy of a model predicting readmission of the patients detailed by limited data122, or increasing medication effectiveness of a medication administered to the patients detailed by limited data122.

Data enricher132identifies one or more applicable richer data112(step214). In order to identify additional features that limited data122lacks, data enricher132may be configured to identify other data sets in a similar field, subfield, or domain to compare features included in the identified data set to those within limited data122. If data enricher132determines that one or more data sets within the similar field or subfield contain additional features that limited data122lacks, then data enricher132identifies the additional features as potential features for inclusion in analysis of limited data122. In other embodiments, additional features or richer data112may simply be user input. For instance, if new research suggests a particular feature is relevant to a target insight, data enricher132may receive the particular feature and search for richer data112that contains the particular feature in addition to the features of limited data122. Alternatively, data enricher132may be linked to a particular richer data112from which additional features are to be identified. While in the example embodiment, richer data112must contain all features detailed by limited data122, in some embodiments, only a certain percentage of the limited features must be present within richer data112.

Consider the earlier-introduced example regarding limited data122detailing type II diabetes patient information that includes the features gender, age, BMI, and cholesterol. In this example, data enricher132identifies richer data112by searching data sets within the type II diabetes domain for data sets including the features gender, age, BMI, and cholesterol, as well as additional features, for example the features creatine level and blood pressure. Alternatively, data enricher132may receive creatine level as a feature via user input and search for data sets detailing type II diabetes patients which include the features gender, age, BMI, cholesterol, and creatine levels. In yet further embodiments, data enricher132may simply receive richer data112via user input which includes the desired features.

Having identified one or more applicable richer data112, data enricher132determines a population relatedness for each feature in common between limited data122and the one or more richer data112. In the example embodiment, the population relatedness serves to aid in determining which richer data112is most applicable to limited data122, and thus which should be used to enrich it. Accordingly, the population relatedness is based on statistical measures such as a mean, standard deviation, range, etc. and, in some embodiments, composite scores thereof, such as a Mahalanobis distance. When needed, determining a population relatedness may further involve constructing a subpopulation from one or more richer data112such that a subpopulations of richer data112is more applicable to limited data122. Such subpopulations may aggregate groups of individuals from one or more richer data sets so long as the data details the features required by limited data122as well as any additional features relevant to the target insight. Thus, in some embodiments, richer data112may comprise extracted data from one or more richer data112.

In the example above, for instance, where limited data122details patients who have acquired type II diabetes and includes data for the features gender, age, BMI, and cholesterol, data enricher132computes summary statistics comparing each individual feature of gender, age, BMI, and cholesterol between limited data122and richer data112. For example, data enricher132may determine that 60% of the data in limited data122may comprise of individuals between the ages of 50 and 60 while 30% of the data in richer data112comprise individuals between the ages of 50 and 60. This process is repeated for each feature in common of the respective data sets, and thus data enricher132determines a population relatedness for gender, BMI, and cholesterol as well.

In embodiments in which several richer data112are applicable to limited data122, data enricher132may be configured to prompt selection of a particular richer data112or, alternatively, prompt selection of subpopulations corresponding to the one or more richer data112. In such embodiments, each of richer data112may be compared to limited data122based on the computed summary statistics individually or in bulk. In further embodiments, data enricher132may be configured to rank the two or more richer data112based on a population relatedness prior to prompting a user selection. Such ranking may be based on an overall highest relatedness (i.e. an average of all features), or based on individual feature relatedness, for instance a feature most relevant to the target insight. In yet further embodiments, data enricher132may be configured to automatically select a most related richer data112, for example if a feature deemed most relevant to the target insight exceeds a relatedness threshold.

In the example embodiment, data enricher132next proceeds to data analysis and update output220, which further comprises calculating estimators using richer data222, calculating estimators using limited data224, updating estimators from the limited data using the richer data226, and evaluating the updated estimators228.

Having identified richer data112, data enricher132calculates estimators for richer data112that denote an impact of a particular feature on the target insight (step222). In the example embodiment, estimators relate to the models contained in model repository116for analysing richer data112and are applied in the form of a coefficient. Here, data enricher132determines the estimators for the features that are contained only in limited data122(limited features) as well as all of the features contained in richer data112(all features). More specifically, in this particular computation, data enricher132determines estimators for the limited and full feature sets using the data of richer data112.

In the example above, for instance, data enricher132calculates estimators for the limited feature set of gender, age, BMI, and cholesterol using the data contained in richer data112. For example, data enricher132determines estimators for gender, age, BMI, and cholesterol as 0.1, 0.2, 0.3, and 0.4, respectively. Next, data enricher132determines estimators for features of the full feature set detailed by richer data112in addition to the limited feature set of limited data122. In the example above, for instance, in addition to determining the feature estimators of 0.1, 0.2, 0.3, and 0.4 for each of the above respective features, data enricher132determines an estimator of creatine level to be 0.5. If, in the example above, there were additional features related to a target insight, for example blood pressure, then data enricher132determines estimators for gender, age, BMI, and cholesterol, as well as blood pressure. Again, data enricher calculates the estimators above using the data included in richer data112.

Data enricher132determines estimators for limited data122(step224). Like the operations of the previous step in which data enricher132calculates estimators for richer data112, data enricher132similarly calculates estimators for limited data122corresponding to models in model repository126. In this iteration, however, data enricher132determines estimators for the features of limited data122, or the limited feature set, using the data of limited data122.

Continuing the example introduced earlier, data enricher132calculates estimators for the features gender, age, BMI, and cholesterol based on the data of limited data122. For example, data enricher132determines estimators for the features gender, age, BMI, and cholesterol based on limited data122to be 0.05, 0.15, 0.25, and 0.35.

Data enricher132updates estimators corresponding to limited data122based on the estimators corresponding to richer data112(step226). In the example embodiment, data enricher132updates estimators for each individual feature of models corresponding to limited data122one at a time to determine whether inclusion/calibration of the feature estimator results in an improved target insight (evaluation thereof described below). While in the example embodiment, updating the estimators comprises replacing the limited data122estimator for a particular feature with the richer data112estimator for the particular feature, in other embodiments, the estimators may be weighted, averaged, or manipulated otherwise. For example, in embodiments where data corresponding to a feature is already included in limited data122but the estimator for limited data122differs from that of richer data112, data enricher132may substitute the limited data122estimator with the richer data112estimator, average the two, or calibrate the limited data122estimator otherwise. In other embodiments where the feature is not included in limited data122, i.e. an estimator of 0, then data enricher132may create a feature estimator based on the estimator of richer data112. In the example embodiment, data enricher132may update the estimator of a limited data122model for any feature in the full or limited feature sets, depending on configuration. In order to determine which features should have a corresponding estimator updated, data enricher132updates estimators for each feature, both limited and full, using the updating techniques above until a greatest impact on the target insight is achieved. In some embodiments, achieving a greatest impact on the target insight may require the updating of only one feature estimator, while in other embodiments, may require the updating of multiple feature estimators. Thus, in addition to determining estimator sets for richer data112and limited data122in steps222and224, data enricher132performs the process described herein to identify one or more sets of estimators, for example combinations of the richer data112estimators and limited data122estimators, in order to fine tune models corresponding to limited data122. Such estimator sets are then evaluated in the next step to compute outputs that are stored in updated outputs138. Based on the model outputs stored in updated outputs138, data enricher132then determines model performance results, as will be described with greater detail below, and if the results significantly impact the target insight, data enricher132stores the estimator sets in updated model repository137.

In the example above, for instance, data enricher132replaces the estimators of gender, age, BMI, cholesterol, and creatine of limited data122with the corresponding estimators of richer data112. Thus, data enricher132replaces the estimator 0.05 with 0.1 for gender, 0.15 with 0.2 for age, 0.25 with 0.3 for BMI, and 0.35 with 0.4 for cholesterol. Alternatively, data enricher132may update feature estimators corresponding to limited data122models by averaging or weighting the estimators for each of richer data112and limited data122. In some embodiments of the example, data enricher132may only update estimators for age and gender in a manner similar to that above. In addition, data enricher132updates a limited data122creatine level estimator of 0 with 0.5. If, for example, blood pressure were also included as an additional feature, then data enricher132may also update an estimator for blood pressure in a similar manner. In general, data enricher132may update any combination of feature estimators included in models corresponding to limited data122such that a target insight is impacted.

Features having a greatest impact on the target insight are noted as being a most suitable candidate for inclusion in future data collection, as will be described below (228continued). In the example embodiment, evaluating the updated estimators may be performed through cross-validation by, for example, breaking down historical data into training data and test data, then training the model based on the training data and verifying the model based on the test data. For example, data enricher132may build a model based on ninety percent of historical data and test the model on the remaining ten percent of the historical data for accuracy. This process is repeated for evaluation of each feature estimator updated in the limited feature set, one by one, until the most impactful features as they relate to the target insight are identified. Notably, the results of the evaluation herein may serve as an indicator as to what target insights to expect should the data of a particular and lacking feature be included in limited data122. Therefore, in some embodiments, data enricher132may perform predictive modelling and output predicted target insight values for limited data122based on richer data112.

In the example above, for instance, data enricher132may determine that based on the data in limited data122, a model contained in model repository126using non-updated estimators can predict readmission of the type II diabetes patient with 62% accuracy. By incorporating one or more estimators for the features age and creatine (and in some embodiments deducing normal creatine levels), however, data enricher132may improve predictions of death by 68%. Similarly, updating an estimator for blood pressure may further increase model prediction accuracy to 73%. This process may be performed for every updated feature and updated feature set until a best combination and value of feature estimators are identified. The combinations and values of the feature estimators resulting in a greatest target insight are then stored in updated model repository137and the outputs are stored in updated outputs138.

Data enricher132provides a data collection recommendation (step232). Depending on the analysis requested, data enricher132may make one of many recommendations as it relates to a target insight, including one or more features to include in future data collection, an amount of data corresponding to the one or more recommended features needed to obtain meaningful results, and predictions as to what the result may be were the data/features included in limited data122. If, for example, collecting data for additional features results in a greatest impact on the target insight, then data enricher132recommends collecting data for that feature. If, for example, collecting more data for a particular feature results in a greatest impact on the target insight, then data enricher132recommends collecting more data for that feature. Moreover, data enricher132predicts an impact on the target insight based on the evaluation of the updated estimators.

In the example above, for instance, data enricher132may first recommend collecting data as it relates to the feature blood pressure, as it resulted in a highest accuracy of predicting readmission. Moreover, data enricher132may suggest collecting age data for 100 more patients in order to improve the target insight, in this case prediction accuracy. In addition, data enricher132may determine that incorporating the blood pressure data for 100 patients into limited data122would result in a readmission prediction rate increase up to 73%.

FIG. 3depicts a schematic diagram illustrating of the operation of data enricher132in adjusting effectiveness evaluation of a limited data set based on a richer data set, in accordance with an embodiment of the present invention. In addition,FIG. 3depicts the modules comprising data enriching system100, namely population relatedness210, data analysis and update output220, and data collection recommendation230, as well as the operation of data enricher132.

FIG. 4depicts a schematic diagram illustrating the operation of data enricher132in updating a prediction based on the richer data, in accordance with an embodiment of the present invention. In addition,FIG. 4depicts the modules comprising data enriching system100, namely population relatedness210, data analysis and update output220, and data collection recommendation230, as well as the operation of data enricher132.

FIG. 5depicts a block diagram of computing device110, server120, and/or server130of the data enriching system100ofFIG. 1, in accordance with an embodiment of the present invention. It should be appreciated thatFIG. 5provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

Computing device110may include one or more processors02, one or more computer-readable RAMs04, one or more computer-readable ROMs06, one or more computer readable storage media08, device drivers12, read/write drive or interface14, network adapter or interface16, all interconnected over a communications fabric18. Communications fabric18may be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system.

One or more operating systems10, and one or more application programs11, for example data enricher132, are stored on one or more of the computer readable storage media08for execution by one or more of the processors02via one or more of the respective RAMs04(which typically include cache memory). In the illustrated embodiment, each of the computer readable storage media08may be a magnetic disk storage device of an internal hard drive, CD-ROM, DVD, memory stick, magnetic tape, magnetic disk, optical disk, a semiconductor storage device such as RAM, ROM, EPROM, flash memory or any other computer-readable tangible storage device that can store a computer program and digital information.

Computing device110may also include a R/W drive or interface14to read from and write to one or more portable computer readable storage media26. Application programs11on said devices may be stored on one or more of the portable computer readable storage media26, read via the respective R/W drive or interface14and loaded into the respective computer readable storage media08.

Computing device110may also include a display screen20, a keyboard or keypad22, and a computer mouse or touchpad24. Device drivers12interface to display screen20for imaging, to keyboard or keypad22, to computer mouse or touchpad24, and/or to display screen20for pressure sensing of alphanumeric character entry and user selections. The device drivers12, R/W drive or interface14and network adapter or interface16may comprise hardware and software (stored on computer readable storage media08and/or ROM06).

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows: