DATA MINIMIZATION USING GLOBAL MODEL EXPLAINABILITY

An embodiment analyzes a predictive model and its input data for the predictive model using an explainability algorithm resulting in a feature importance value of a feature. The embodiment analyzes feature values of the feature using a generalization function resulting in a set of candidate feature values. The embodiment determines an alternative feature based on the set of candidate feature values, wherein the alternative feature is a generalization of the feature. The embodiment compares an accuracy of the predictive model to a threshold performance value and, responsive to the accuracy being above the threshold performance value, maps feature values in the input data that are in the set of candidate feature values to a generalized representative value in the generalized domain.

BACKGROUND

The present invention relates generally to generalizing data. More particularly, the present invention relates to a method, system, and computer program for generalizing data for predictive models.

Data minimization may refer to the practice of limiting the collection of personal information to that which is directly relevant and necessary to accomplish a specified purpose. As companies and organizations began to understand the power of data, and as data becomes more ubiquitous and easier to collect, analysts are faced with an overwhelming amount of data. For a time, the impulse was to save all of it—indefinitely. With the fast adoption of smartphones, Internet of Things (IoT) devices, or the like, organizations are faced with more and more ways to collect more and more kinds of data, including and especially private, personally identifiable data. However, information protection requirements imposed by various statutes and regulations, such as the General Data Protection Regulation (GDPR), provide for data collection and retention limits. As a result, data managers must consider such compliance obligations and their strict limitations when implementing data collection and retention policies.

SUMMARY

The illustrative embodiments provide for data minimization using global model explainability. An embodiment includes analyzing a predictive model and input data for the predictive model using an explainability algorithm, the analyzing resulting in a feature importance value of a feature. The embodiment also includes analyzing feature values of the feature using a generalization function, where the analyzing of the feature values results in generation of a set of candidate feature values. The embodiment also includes determining, based on the set of candidate feature values, an alternative feature, where the alternative feature is a generalization of the feature, where the alternative feature is a generalized feature having a generalized domain, where each feature value in the generalized domain corresponds to one or more feature values in a domain of the feature, where a number of feature values in the domain is greater than a number of feature values in the generalized domain, whereby the generalized feature is a generalization of the feature. The embodiment also includes comparing an accuracy of the predictive model to a threshold performance value, where the accuracy is based on outputs of the predictive model using the alternative feature. The embodiment also includes mapping, responsive to the accuracy being above the threshold performance value, feature values in the input data that are in the set of candidate feature values to a generalized representative value in the generalized domain. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the embodiment.

An embodiment includes a computer usable program product. The computer usable program product includes a computer-readable storage medium, and program instructions stored on the storage medium.

An embodiment includes a computer system. The computer system includes a processor, a computer-readable memory, and a computer-readable storage medium, and program instructions stored on the storage medium for execution by the processor via the memory.

DETAILED DESCRIPTION

Predictive modeling refers to a form of predictive analytics that typically uses a machine learning (ML) algorithm to build a predictive model (also referred to herein as a ML model or ML predictive model). There are countless applications for such models; however, for the sake of clarity of explanation, examples of applications are described herein that are only provided as illustrative context and are in no way intended to limit the scope of the present disclosure. There are also various performance metrics for predictive models (e.g., precision, specificity, error type, F1 score, etc.); however, for the sake of simplicity, predictive model performance will simply be referred to herein as model accuracy. The quality of a predictive model is typically based on the model's accuracy, with more accurate models generally being more desirable (subject to reasonable constraints, such as processing costs, etc.).

In many cases, the data that is collected and used as input for predictive models is subject to data collection and retention requirements. For example, the GDPR requires that only data that is strictly necessary for fulfilling a certain purpose should be collected. However, there are situations in which it is difficult to determine exactly what data is strictly necessary for a predictive model. For example, it is difficult to determine exactly what data, and what granularity (level of detail) of each feature, is strictly necessary for an ML predictive model to maintain a threshold level of accuracy.

Thus, one technical problem addressed herein involves determining what data is necessary for an ML model to make accurate predictions (i.e., predictions having a threshold level of accuracy). In some cases, it may be desired to minimize the data, e.g., the number of features that are being collected and their respective granularities, while still being able to utilize the predicative model to provide quality predictions.

Exemplary embodiments disclosed herein provide for a determination of a generalized version of the feature data that is input to an ML model and still enables the model to make predictions above a threshold level of accuracy. In some embodiments, a generalized version of numerical feature data includes a list of ranges, and a generalized version of categorical feature data is one or more groups of values.

As an example, data records that provide for feature data that is representative of people's ages may be used by a predictive ML model to predict heart attacks with about 90% accuracy. Data minimization that involves feature generalization may include replacing certain ages with an age range that still allows the ML model to maintain an accuracy of about 90%. In some embodiments, such generalizations may vary for different ranges of values. For example, an ML model may be able to maintain a threshold level of accuracy when ages from 20 to 30 are generalized to that ten-year range. However, the accuracy of that same ML model may drop below the threshold level when ages from 50 to 60 are generalized to a ten-year range, but the accuracy rises above the threshold when ages from 50 to 60 are generalized to five-year range. Thus, in such embodiments, some feature data values are generalized to a ten-year range, whereas other feature data values are generalized to a five-year range. In other words, in some embodiments, the generalized version of the feature data includes a plurality of different sizes of numerical ranges. As an example, in some embodiments, data generalization may allow for data minimization by allowing data indicative of people's exact ages to be replaced with a generalized version of the data that includes age ranges such as [1-20], [21-30], [31-40], [41-50], [51-55], [56-60], and so on, thereby reducing the granularity of the generalized feature data relative to the non-generalized version of the feature data.

In some exemplary embodiments, the alternative set of features may comprise a generalized feature corresponding to each feature of the set of features or a portion thereof. In some cases, the generalized feature may correspond to the feature itself, and have a generalized domain that comprises a smaller number of possible values than in the domain of the corresponding feature. In some cases, each value in the generalized domain may correspond one or more values in the non-generalized domain. Additionally, or alternatively, the alternative set of features may omit one or more redundant features from the set of features. The redundant features may not necessarily be redundant per se; however, their value may be relatively insignificant and may not be collected while maintaining a relative quality of prediction.

In some exemplary embodiments, a generalized instance may be obtained. The generalized instance may be a valuation to the alternative set of features. In some exemplary embodiments, a predicted label may be determined for the generalized instance. The predicted label may be determined by utilizing the predictive model. In some exemplary embodiments, the generalized instance may be used to generate an instance based thereon. In some cases, the instance may be generated using representative values to the features based on the actual values of generalized features. The predictive model may be used to predict a label for the generated instance and the predicted label may be used as the true label of the generalized instance.

Yet another technical problem dealt with by the disclosed subject matter is to minimize the data provided to a predictive model without affecting the performance measurement of the predictive model. It may be desired to minimize the data without causing the performance measurement of the predictive model to decrease below a threshold. The threshold may be an absolute threshold such as 90%, 92%, or the like. Additionally, or alternatively, the threshold may be a relative threshold, relative to the performance measurement of the predictive model before the data minimization. As an example, it may be desired that the after minimizing the data, the performance measurement may not decrease more than 5%.

Yet another technical problem dealt with by the disclosed subject matter is to minimize the amount of data that is requested from a user for a predictive model without affecting the performance of the predictive model. As an example, the predictive model may be configured to predict a label based on an income of a user. Generalization techniques disclosed herein may reveal that the predictive model is still just as accurate when the user instead only selects from certain income ranges.

One technical solution is to leverage global explainability analysis methods (such as SHAP and LIME) to help guide the process of data minimization for ML models. A global explainability analysis outputs a level of importance for each feature, where the “importance” of a feature refers to how significant or influential that feature was for the model in making its predictions. Such global explainability analysis techniques are used herein to provide global feature importance values. In some embodiments, it is desirable to perform generalizations that do not have (or have a minimal) effect on the model's accuracy. In some such embodiments, the global feature importance values are used to guide the selection of features and values to generalize by revealing the effect of each feature on the model's outcome.

In some embodiments, global explainability analysis techniques are used to reveal detailed information from a prediction model that is significantly informative for the generalization process. For example, in some embodiments, global explainability analysis techniques are used to identify features and values that “behave” similarly in the model, thereby revealing areas in the domain that, if generalized, will have a minimal effect on the model's accuracy.

In exemplary embodiments, the generalizing of one or more of a set of features yields data minimization. In some such embodiments, a predictive model is configured to predict a label for a valuation of a certain set of features, also referred to as an instance. In some exemplary embodiments, a global explainability analysis technique is used to identify features and values that “behave” similarly in the model, thereby revealing areas in the domain that, if generalized, will have a minimal effect on the model's accuracy. In some such embodiments, the overall feature importance of each feature is computed. In some such embodiments, the overall feature importance is determined by computing the normalized sums of SHAP values along each feature on a complete dataset. For categorical features that are one-hot encoded, the feature importance is accumulated across all encodings.

In an exemplary embodiment, once the relative importance of each of the features is revealed, features that have the least amount of influence on the predictive model can be identified as good candidates for generalization. If a feature has little or almost no influence on the classification predictions made by a predictive model, then there is a good possibility that the feature values can be at least somewhat altered without lowering the accuracy of the predictive model. Where feature values can be altered, a range of feature values can replace specific values, thereby allowing for generalization. Thus, in some embodiments, once the importance of each of the features is known, the generalization process selects a subset of features having relatively low feature-importance values i.e., the absolute SHAP mean value is close to zero.

In an exemplary embodiment, the generalization process evaluates feature values of the features in the subset of low-importance features. Since these features have been revealed to be good candidates for generalization, the feature values of these features are evaluated to identify actual feature values that can be generalized without degrading the quality of the predictive model. In some embodiments, this evaluation identifies the candidate feature values using a generalization function (e.g., k-means, decision tree, agglomerative clustering), meaning the generalization function identifies generalizations that minimize the reduction in accuracy of the predictive model.

In some exemplary embodiments, once a feature is identified as having one or more groups of feature values as generalization candidates, the feature is replaced with an alternative feature that is a generalization of the feature it replaces. As an example, instead of having a feature representing the precise age being any number between 1-120, the generalized feature may be a selection between a set of ranges, e.g., [1-20], [21-25], [26-50], [51-120], reducing the granularity of the non-generalized feature. In such a case, the domain of the feature comprised 120 separate values, while the domain of the generalized feature only comprises 4 separate values.

In some exemplary embodiments, a generalized feature has a generalized domain that comprises a smaller number of possible values than the domain of the corresponding original feature. In some cases, each value in the generalized domain may correspond with one or more values in the non-generalized domain.

In an exemplary embodiment, a validation process checks the generalized alternative feature to determine what effect, if any, the generalization has on the accuracy of the predictive model. In some embodiments, the validation compares an accuracy of the predictive model on the generalized data to a threshold performance value. In some embodiments, the threshold performance value is a baseline accuracy value of the predictive model using the original (pre-generalization) features. In some such embodiments, the threshold performance value and/or one or more prior accuracy values are stored as model history in the memory.

In some embodiments, the threshold performance value is an accuracy of the model prior to generalization of any of the features. In some embodiments, the performance value is based on an accuracy of the model prior to generalization of any of the features and other factors, such as a user-configurable tolerance value (e.g., to allow for reduction of accuracy within a specified tolerance percentage). In some embodiments, the performance value is a user-configurable value that allows a user to specify a value for a performance metric, for example a user-specified accuracy value.

In some embodiments, a data generalization process includes an iterative process that gradually increases the generalization of feature values while checking the accuracy of the predictive model such that the feature values are increasingly generalized until the accuracy of the model fails to satisfy the threshold performance value. This allows the data generalization process to generalize feature values to values that are generalized to the extent possible while still allowing the predictive model to maintain a threshold level of accuracy.

In some exemplary embodiments, the process may continue with additional features. After receiving a valuation for another generalized feature, the next feature may be generalized even further, or even omitted. After receiving the value of the next feature, the feature that follows may be generalized further, and so forth. In some cases, the determination of the generalized domain for a feature may be based on a value of a single previous feature, valuation of several previous features, or the like, in combination with information extracted from the auxiliary model.

In some exemplary embodiments, a User Interface (UI), such as Graphical User Interface (GUI), Voice User Interface (VUI), or the like, may be utilized to obtain from a user values for the alternative set of features, thereby obtaining the generalized instance. For each alternative feature comprised by the alternative set of features the UI may comprise a UI element corresponding thereto. In some exemplary embodiments, the user interface may be a dynamic user interface. The dynamic user interface may be configured to be updated based on dynamic display data provided thereto. In some embodiments, the dynamic display data provides instructions to update the user interface to replace a user input field configured to request or receive the original feature values with an alternative input field configured to request or receive the alternative generalized feature values. For example, an age feature, which includes any integer from 1 to 120 as a feature value, may be generalized to an alternative feature representative of age ranges, for example [1-20], [21-25], [26-50], [51-120]. In this example, the dynamic display data instructs the user interface to replace a user input field that requests an age with a user input field that requests selection of one of the age ranges. The user interface will then output a representative value that will be used as input to the ML runtime module for any value in the selected range.

In some exemplary embodiments, if a data generalization process determines that the accuracy of the predictive model using the alternative generalized feature values is acceptable, the data generalization process generates value mapping data. The value mapping data provides data mapping instructions to a value mapping module. In some embodiments, data mapping instructions include instructions to map feature values in a generalized range or group of values to a representative value for the generalized range or group. For example, if the generalization process has replaced the age feature, which includes any integer from 1 to 120 as a feature value, with an alternative feature representative of age ranges, for example [1-20], [21-25], [26-50], [51-120], the value mapping data will instruct the value mapping module to map any age from 1 to 20 to a first representative value for the first range, map any age from 21 to 25 to a second representative value for the second range, map any age from 26 to 50 to a third representative value for the third range, and map any age from 51 to 120 to a fourth representative value for the fourth range.

Furthermore, simplified diagrams of the data processing environments are used in the figures and the illustrative embodiments. In an actual computing environment, additional structures or components that are not shown or described herein, or structures or components different from those shown but for a similar function as described herein may be present without departing the scope of the illustrative embodiments.

Furthermore, the illustrative embodiments are described with respect to specific actual or hypothetical components only as examples. Any specific manifestations of these and other similar artifacts are not intended to be limiting to the invention. Any suitable manifestation of these and other similar artifacts can be selected within the scope of the illustrative embodiments.

With reference toFIG.1, this figure depicts a block diagram of a computing environment100. Computing environment100contains an example of an environment for the execution of at least some of the computer code involved in performing the inventive methods, such as an improved data generalization module200that uses global explainability analysis to identify features that are good candidates for generalizing the input data needed by a predictive model without sacrificing the quality of the model's predictions. In addition to data generalization module200, computing environment100includes, for example, computer101, wide area network (WAN)102, end user device (EUD)103, remote server104, public cloud105, and private cloud106. In this embodiment, computer101includes processor set110(including processing circuitry120and cache121), communication fabric111, volatile memory112, persistent storage113(including operating system122and data generalization module200, as identified above), peripheral device set114(including user interface (UI) device set123, storage124, and Internet of Things (IoT) sensor set125), and network module115. Remote server104includes remote database130. Public cloud105includes gateway140, cloud orchestration module141, host physical machine set142, virtual machine set143, and container set144.

Measured service: cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, reported, and invoiced, providing transparency for both the provider and consumer of the utilized service.

With reference toFIG.2, this figure depicts a block diagram of an example data processing environment201that includes a data generalization module200in accordance with an illustrative embodiment. In the illustrated embodiment, the data processing environment201includes the data generalization module200, a ML training module202, and a memory204that stores dataset206and model history208. In alternative embodiments, the data processing environment201can include some or all of the functionality described herein but grouped differently into one or more modules. In some embodiments, the functionality described herein is distributed among a plurality of systems, which can include combinations of software and/or hardware-based systems, for example Application-Specific Integrated Circuits (ASICs), computer programs, or smart phone applications.

In the illustrated embodiment, the ML training module202generates a trained machine learning (ML) model, also referred to herein as a predictive model. For example, in some embodiments, the ML model is a classification model trained to make class predictions from tabular input data, which may include features having numerical, categorical, and/or continuous data domains. In the illustrated embodiment, the training performed by the ML training module202may include training a new ML model and may include re-training a previously-trained ML model. For example, in some embodiments, the ML training module202initially trains the ML model using a non-generalized version of the dataset206. In some such embodiments, the data generalization module200then generates a generalized version of the training data, and stores the generalized training dataset in the memory204. In some such embodiments, the ML training module202then re-trains the ML model using the generalized training dataset. In the illustrated embodiment, the dataset206is stored in memory204, where the memory204is representative of any non-volatile computer readable storage medium, local or remote, from which such training data may be retrieved. In some embodiments, the dataset206includes training data that the ML training module202uses during training epochs, and may also include testing data that the ML training module202uses for subsequent model evaluation.

In some exemplary embodiments, the ML model may be an ANN model such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Deep Neural Network (DNN) model, or the like. Additionally, or alternatively, the ML model may be a classification model such as a Support Vector Machine (SVM), a decision tree, Naïve Bayes, or the like. In some exemplary embodiments, the ML model may be configured to predict a classification for an instance based on a set of features thereof.

In the illustrated embodiment, the data generalization module200determines a generalized version of feature data that is input to an ML model and still enables the model to make predictions above a threshold level of accuracy. In some embodiments, a generalized version of numerical feature data includes a list of ranges, and a generalized version of categorical feature data is one or more groups of values.

In some embodiments, the data generalization module200receives the trained ML predictive model from the ML training module202, as well as one or more instances of the training data and/or testing data used to train the ML predictive model or another dataset with similar distribution. In some embodiments, the data generalization module200generates global explainability data for each of the features. The data generalization module200then analyzes feature values of a selected feature from among the set of features using a generalization function, wherein the analyzing of the feature values results in generation of a set of candidate feature values. The data generalization module200then determines an alternative feature as a generalization of the selected feature based on the set of candidate feature values.

In some embodiments, the data generalization module200compares an accuracy of the predictive model on the generalized data to a threshold performance value. In some embodiments, the threshold performance value is a baseline accuracy value of the predictive model using the original (pre-generalization) features. In some such embodiments, the threshold performance value and/or one or more prior accuracy values are stored as model history208in the memory204.

The data generalization module200is thus able to compare an accuracy of the predictive model using generalized features to a threshold performance value, such as an accuracy of the model using the original features. This allows the data generalization module200to determine whether the generalization reduced the accuracy of the predictive model.

In some embodiments, the threshold performance value is an accuracy of the model prior to generalization of any of the features. In some embodiments, the performance value is based on an accuracy of the model prior to generalization of any of the features and other factors, such as a user-configurable tolerance value (e.g., to allow for reduction of accuracy within a specified tolerance percentage). In some embodiments, the performance value is a user-configurable value that allows a user to specify a value for a performance metric, for example a user-specified accuracy value.

In some embodiments, the data generalization module200performs an iterative process that gradually increases the generalization of feature values while checking the accuracy of the predictive model such that the feature values are increasingly generalized until the accuracy of the model fails to satisfy the threshold performance value. This allows the data generalization module200to generalize feature values to values that are generalized to the extent possible while still allowing the predictive model to maintain a threshold level of accuracy.

With reference toFIG.3, this figure depicts a block diagram of an example data processing environment300in accordance with an illustrative embodiment. In a particular embodiment, the data processing environment300is a more detailed example of the data processing environment201ofFIG.2.

In the illustrated embodiment, the ML training module202includes a testing data module302, a training data module304, an algorithm306, a training module308, and a testing module310. The data generalization module200includes a global explainability module312, a feature selection module314, a generalization module316, and a training data generation module318. In alternative embodiments, the data processing environment300can include some or all of the functionality described herein but grouped differently into one or more modules. In some embodiments, the functionality described herein is distributed among a plurality of systems, which can include combinations of software and/or hardware-based systems, for example Application-Specific Integrated Circuits (ASICs), computer programs, or smart phone applications.

In the illustrated embodiment, the ML training module202receives dataset206from memory204. In some embodiments, the ML model is a supervised model, meaning the model is trained using a supervised training process. In such embodiments, the dataset206comprises a heterogeneous matrix in which each row is an instance, and each column is a feature, except for an extra label column that is included in training data for supervised learning. Thus, each row is a valuation of the set of features for a given instance, plus a label used during the training process to check the output of the model for accuracy.

The ML training module202divides the dataset206into a training data set that is routed to the training data module304and a testing data set that is routed to the testing data module302. The training module308receives the training data from the training data module304and the algorithm306and generates a trained ML predictive model. The algorithm306may be any of a variety of known algorithms having tunable parameters that are adjusted during the training phase to create a trained ML model and improve the accuracy of the model's predicted outputs for new inputs.

In some embodiments, the ML training module202includes a testing module310that monitors the model's ability to make predictions for the testing data set. For example, in some embodiments, the testing data set includes data that has not been processed by the ML model in order to allow the testing module310to evaluate the ML model's ability to generalize and accurately make predictions about the new data of the dataset206.

In the illustrated embodiment, the data generalization module200determines a generalized version of feature data that is input to an ML model and still enables the model to make predictions above a threshold level of accuracy. In some embodiments, a generalized version of numerical feature data includes a list of ranges, and a generalized version of categorical feature data is one or more groups of values.

In the illustrated embodiment, the global explainability module312receives the trained ML predictive model from the training module308as well as one or more instances of the training/testing data and associated predictions made by the predictive model, and/or any other dataset having a distribution that is acceptably similar according to user preferences. The global explainability module312performs a global explainability analysis and outputs global explainability data that includes feature importance values that are indicative of a level of importance for each feature. The “importance” of a feature refers to how significant or influential that feature was for the model in making its predictions. In some such embodiments, the global explainability analysis includes determining a normalized sum of feature importance values for each feature. In some such embodiments, the global explainability module312sums each feature's importance values across all of the training instances. In some embodiments, the global explainability module312then normalizes each sum of feature importance values, resulting in global explainability or global feature importance values for each of the features. In some embodiments, the feature importance values are Shapely additive explanation (SHAP) scores.

In the illustrated embodiment, the global explainability module312provides the global explainability data to the feature selection module314. In some embodiments, the feature selection module314sorts the features in ascending order according to the global feature importance values. In some embodiments, additional scaling can be implemented to adjust the global feature importance values prior to sorting. For example, in some embodiments, the feature importance values are weighted for features involving sensitive data, meaning data that one would reasonably prefer to retain as private or confidential information is scaled to make it a higher priority for generalization in order to generalize such sensitive data. As another example, in some embodiments, feature importance values are scaled according to the feature's domain size, meaning features with a larger domain will be prioritized to be generalized ahead of features with smaller domains. For example, a categorical feature with four categories will be scaled by 1/4 while categorical feature with ten categories will be scaled by 1/10. This scaling works equivalently for continuous features. The assumption is that the feature with a higher initial domain can achieve better generalization. The feature selection module314will then select a feature as a generalization target based on the importance values subject to any scaling that has been done.

The feature selection module314then provides the selected feature to the generalization module316. The generalization module316evaluates feature values of the selected feature to identify candidate feature values for generalization. Since these features have been revealed to be good candidates for generalization, the feature values of these features are evaluated to identify actual feature values that can be generalized without degrading the quality of the predictive model.

In some embodiments, the generalization module316attempts to identify a set of candidate feature values of the selected feature. In some such embodiments, the generalization module316identifies the candidate feature values using a generalization function (e.g., k-means, decision tree, agglomerative clustering), meaning the generalization function identifies a generalization that minimizes the reduction in accuracy of the predictive model.

In some embodiments, the generalization function uses agglomerative hierarchical clustering. Using a bottom-up approach, the agglomerative hierarchical clustering starts without any generalizations and tries to generalize as much as possible. The algorithm will generalize/merge different values that minimize model's accuracy reduction. Each sample starts as an independent cluster. Aggregation attempts between different clusters are performed based on some distance metric (e.g., the distance between centroids), and the decision whether to aggregate or not is taken based on the criteria function (e.g., model's accuracy).

In some embodiments, the generalization function uses a decision tree. In such embodiments, the process trains a decision tree on the selected feature and prunes the decision tree accordingly. Thus, instead of aggregation that is done using the agglomerative hierarchical clustering, the decision tree embodiment prunes the decision tree.

In some embodiments, the generalization function uses a binary Search on K in a K-means algorithm. In such embodiments, the process tries to cluster the feature space into a minimal number of clusters as much as possible. For K_i (K in iteration i), the process applies the proposed generalization. Generalization is defined by the min and max values of each cluster. If the accuracy reduction is below the defined threshold, the process will try a smaller number of clusters (better generalization) K_j<K_i. Otherwise, the process will try to increase the number of clusters, with the next K being chosen based on a binary search algorithm.

In some exemplary embodiments, once the generalization module316identifies a feature as having one or more groups of feature values as generalization candidates, the generalization module316selects that feature for generalization. The generalization module316replaces the selected feature with an alternative feature that is a generalization of the selected feature. As an example, the selected feature may be representative of age based on the feature selection module314determining that the age feature has a relatively low feature importance value and the generalization module316identifying groups of ages that are good generalization candidates. The generalization module316then replaces the age feature, which includes any integer from 1 to 120 as a feature value, with an alternative feature representative of age ranges, for example [1-20], [21-25], [26-50], [51-120]. Thus, the granularity of the selected feature is reduced with the alternative feature, from a domain of 120 separate values, to a generalized domain of four separate values.

In the illustrated embodiment, the training data generation module318updates the dataset206of training/testing data by replacing the selected feature with the alternative generalized feature. In some embodiments, the ML training module202then retrains the ML model using the updated dataset206with the generalized feature. In an exemplary embodiment, a validation process checks the generalized alternative feature to determine what effect, if any, the generalization has on the accuracy of the predictive model. In some such embodiments, the testing module310evaluates the ML model's accuracy on the generalized training data and outputs an accuracy value to the generalization module316, which records the value in the model history208. The generalization module316compares an accuracy of the predictive model to a threshold performance value. In some embodiments, the threshold performance value is a baseline accuracy value of the predictive model using the original (pre-generalization) features. In some such embodiments, the threshold performance value and/or one or more prior accuracy values are stored as model history208in the memory204.

The generalization module316is thus able to compare an accuracy of the predictive model using generalized features to a threshold performance value, such as an accuracy of the model using the original features. This allows the generalization module316to determine whether the generalization reduced the accuracy of the predictive model.

In some embodiments, the threshold performance value is an accuracy of the model prior to generalization of any of the features. In some embodiments, the performance value is based on an accuracy of the model prior to generalization of any of the features and other factors, such as a user-configurable tolerance value (e.g., to allow for reduction of accuracy within a specified tolerance percentage). In some embodiments, the performance value is a user-configurable value that allows a user to specify a value for a performance metric, for example a user-specified accuracy value.

In some embodiments, the data generalization module200performs an iterative process that gradually increases the generalization of feature values while checking the accuracy of the predictive model such that the feature values are increasingly generalized until the accuracy of the model fails to satisfy the threshold performance value. This allows the data generalization module200to generalize feature values to values that are generalized to the extent possible while still allowing the predictive model to maintain a threshold level of accuracy.

With reference toFIG.4, this figure depicts a block diagram of an example data processing environment400in accordance with an illustrative embodiment. In some embodiments, the data generalization module404is an example of the data generalization module200ofFIG.1. As shown inFIG.4, in some embodiments, the data generalization module404generalizes a trained predictive model operating in a runtime environment in which the model makes predictions for new data.

In the illustrated embodiment, the data processing environment400includes the data generalization module404, a ML runtime module402, a ML model evaluation module406, and a memory408that stores model history410. In alternative embodiments, the data processing environment400can include some or all of the functionality described herein but grouped differently into one or more modules. In some embodiments, the functionality described herein is distributed among a plurality of systems, which can include combinations of software and/or hardware-based systems, for example Application-Specific Integrated Circuits (ASICs), computer programs, or smart phone applications.

In the illustrated embodiment, the ML runtime module402includes a trained machine learning (ML) model, also referred to herein as a predictive model. For example, in some embodiments, the ML model is a classification model trained to make class predictions from tabular input data, which may include features having numerical, categorical, and/or continuous data domains. In the illustrated embodiment, the ML runtime module402stores the accuracy of the ML model as model history410in memory408, where the memory408is representative of any non-volatile computer readable storage medium, local or remote, from which such training data may be retrieved.

In some exemplary embodiments, the ML model may be an ANN model such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Deep Neural Network (DNN) model, or the like. Additionally, or alternatively, the ML model may be a classification model such as a Support Vector Machine (SVM), a decision tree, Naïve Bayes, or the like. In some exemplary embodiments, the ML model may be configured to predict a classification for an instance based on a set of features thereof.

In the illustrated embodiment, the data generalization module404determines a generalized version of feature data that is input to an ML model and still enables the model to make predictions above a threshold level of accuracy. In some embodiments, a generalized version of numerical feature data includes a list of ranges, and a generalized version of categorical feature data is one or more groups of values.

In some embodiments, the data generalization module404receives the trained ML predictive model from the ML runtime module402, as well as one or more instances of the new data that is inputted to the ML predictive model. In some embodiments, the data generalization module404generates a subset of features that are identified, using global explainability data, as being among the least important features. The data generalization module404then generates of a set of candidate feature values based on an analysis of a selected feature from the subset of features. The data generalization module404then determines an alternative feature as a generalization of the selected feature based on the set of candidate feature values.

In some embodiments, the ML model evaluation module406determines an accuracy of the ML model using the generalized alternative feature. The ML model evaluation module406then provides an accuracy value to the data generalization module404representative of the determined accuracy of the ML model using the generalized alternative feature. In some embodiments, the ML model evaluation module406also stores the accuracy value with the model history410in the memory408.

The data generalization module404compares the accuracy of the predictive model to a threshold performance value. In some embodiments, the threshold performance value is a baseline accuracy value of the predictive model using the original (pre-generalization) features. In some such embodiments, the threshold performance value and/or one or more prior accuracy values are stored as model history410in the memory408.

The data generalization module404is thus able to compare an accuracy of the predictive model using generalized features to a threshold performance value, such as an accuracy of the model using the original features. This allows the data generalization module404to determine whether the generalization reduced the accuracy of the predictive model.

In some embodiments, the threshold performance value is an accuracy of the model prior to generalization of any of the features. In some embodiments, the performance value is based on an accuracy of the model prior to generalization of any of the features and other factors, such as a user-configurable tolerance value (e.g., to allow for reduction of accuracy within a specified tolerance percentage). In some embodiments, the performance value is a user-configurable value that allows a user to specify a value for a performance metric, for example a user-specified accuracy value.

In some embodiments, the data generalization module404performs an iterative process that gradually increases the generalization of feature values while checking the accuracy of the predictive model such that the feature values are increasingly generalized until the accuracy of the model fails to satisfy the threshold performance value. This allows the data generalization module404to generalize feature values to values that are generalized to the extent possible while still allowing the predictive model to maintain a threshold level of accuracy.

With reference toFIG.5, this figure depicts a block diagram of an example data processing environment500in accordance with an illustrative embodiment. In a particular embodiment, the data processing environment500is a more detailed example of the data processing environment400ofFIG.4.

In the illustrated embodiment, the data processing environment500includes a ML runtime module402, a data generalization module404, and a ML model evaluation module406. The ML runtime module402includes a data preparation module502and a machine learning model504. The data generalization module404includes a global explainability module506, a feature selection module508, and a generalization module510. The ML model evaluation module406includes a testing data module512and a testing module514. In alternative embodiments, the data processing environment500can include some or all of the functionality described herein but grouped differently into one or more modules. In some embodiments, the functionality described herein is distributed among a plurality of systems, which can include combinations of software and/or hardware-based systems, for example Application-Specific Integrated Circuits (ASICs), computer programs, or smart phone applications.

In the illustrated embodiment, the ML runtime module402receives new data that includes data instances for which predictions by the machine learning model504are sought. In some embodiments, the new data is tabular data, such as data in a spreadsheet, database, or comma separated variable (CSV) file. In some embodiments, the new data is raw data. The data preparation502performs data preparation on the incoming data as necessary to prepare the data for processing by the machine learning model504.

The data preparation502then provides the prepared data to the machine learning model504. The ML runtime module402then uses the machine learning model504to infer and make predictions about the input new data.

In the illustrated embodiment, the data generalization module404determines a generalized version of feature data that is input to an ML model and still enables the model to make predictions above a threshold level of accuracy. In some embodiments, a generalized version of numerical feature data includes a list of ranges, and a generalized version of categorical feature data is one or more groups of values.

In the illustrated embodiment, the global explainability module506receives the trained ML predictive model machine learning model504from the ML runtime module402as well as one or more instances of the new data and associated predictions made by the machine learning model504. The global explainability module506performs a global explainability analysis and outputs global explainability data that includes feature importance values that are indicative of a level of importance for each feature. The “importance” of a feature refers to how significant or influential that feature was for the model in making its predictions. In some such embodiments, the global explainability analysis includes determining a normalized sum of feature importance values for each feature. In some such embodiments, the global explainability module506sums each feature's importance values across all of the training instances. In some embodiments, the global explainability module506then normalizes each sum of feature importance values, resulting in global explainability or global feature importance values for each of the features. In some embodiments, the feature importance values are Shapely additive explanation (SHAP) scores.

In the illustrated embodiment, the global explainability module506provides the global explainability data to the feature selection module508. In the illustrated embodiment, the global explainability module506provides the global explainability data to the feature selection module508. In some embodiments, the feature selection module508sorts the features in ascending order according to the global feature importance values. In some embodiments, additional scaling can be implemented to adjust the global feature importance values prior to sorting. For example, in some embodiments, the feature importance values are weighted for features involving sensitive data, meaning data that one would reasonably prefer to retain as private or confidential information is scaled to make it a higher priority for generalization in order to generalize such sensitive data. As another example, in some embodiments, feature importance values are scaled according to the feature's range, meaning features with a larger domain will be prioritized to be generalized ahead of features with smaller domains. For example, a categorical feature with four categories will be scaled by 1/4 while categorical feature with ten categories will be scaled by 1/10. This scaling works equivalently for continuous features. The assumption is that the feature with a higher initial domain can achieve better generalization. The feature selection module508will then select a feature as a generalization target based on the importance values subject to any scaling that has been done.

The feature selection module508then provides the selected feature to the generalization module510. The generalization module510evaluates feature values of the selected feature to identify candidate feature values for generalization. Since these features have been revealed to be good candidates for generalization, the feature values of these features are evaluated to identify actual feature values that can be generalized without degrading the quality of the predictive model. In some embodiments, this evaluation searches for feature values

In some embodiments, the generalization module510attempts to identify a set of candidate feature values of the selected feature. In some such embodiments, the generalization module510identifies the candidate feature values using a generalization function (e.g., k-means, decision tree, agglomerative clustering), meaning the generalization function minimizes the reduction in accuracy of the predictive model.

In some embodiments, the generalization function uses agglomerative hierarchical clustering. Using a bottom-up approach, the agglomerative hierarchical clustering starts without any generalizations and tries to generalize as much as possible. The algorithm will generalize/merge different values that minimize model's accuracy reduction. Each sample starts as an independent cluster. Aggregation attempts between different clusters are performed based on some distance metric (e.g., the distance between centroids), and the decision whether to aggregate or not is taken based on the criteria function (e.g., model's accuracy).

In some embodiments, the generalization function uses a decision tree. In such embodiments, the process trains a decision tree on the selected feature and prunes the decision tree accordingly. Thus, instead of aggregation that is done using the agglomerative hierarchical clustering, the decision tree embodiment prunes the decision tree.

In some embodiments, the generalization function uses a binary Search on K in a K-means algorithm. In such embodiments, the process tries to cluster the feature space into a minimal number of clusters as much as possible. For K_i (K in iteration i), the process applies the proposed generalization. Generalization is defined by the min and max values of each cluster. If the accuracy reduction is below the defined threshold, the process will try a smaller number of clusters (better generalization) K_j<K_i. Otherwise, the process will try to increase the number of clusters, with the next K being chosen based on a binary search algorithm.

In some exemplary embodiments, once the generalization module510identifies a feature as having one or more groups of feature values as generalization candidates, the generalization module510selects that feature for generalization. The generalization module510replaces the selected feature with an alternative feature that is a generalization of the selected feature. As an example, the selected feature may be representative of age based on the feature selection module508determining that the age feature has a relatively low feature importance value and the generalization module510identifying groups of ages that are good generalization candidates. The generalization module510then replaces the age feature, which includes any integer from 1 to 120 as a feature value, with an alternative feature representative of age ranges, for example [1-20], [21-25], [26-50], [51-120]. Thus, the granularity of the selected feature is reduced with the alternative feature, from a domain of 120 separate values, to a generalized domain of four separate values.

In the illustrated embodiment, the ML model evaluation module406evaluates the machine learning model504with the alternative generalized feature. The testing data module512updates the new data received from the ML runtime module402by replacing the selected feature with the alternative generalized feature. The testing module514then performs a validation process that checks the generalized alternative feature to determine what effect, if any, the generalization has on the accuracy of the machine learning model504. In some such embodiments, the testing module514evaluates the ML model's ability to generalize and accurately make predictions about the updated generalized data of the new data and outputs an accuracy value to the generalization module510and records the value in the model history410. The generalization module510compares an accuracy of the predictive model to a threshold performance value. In some embodiments, the threshold performance value is a baseline accuracy value of the predictive model using the original (pre-generalization) features. In some such embodiments, the threshold performance value and/or one or more prior accuracy values are stored as model history410in the memory408.

The generalization module510is thus able to compare an accuracy of the predictive model using generalized features to a threshold performance value, such as an accuracy of the model using the original features. This allows the generalization module510to determine whether the generalization reduced the accuracy of the predictive model.

In some embodiments, the threshold performance value is an accuracy of the model prior to generalization of any of the features. In some embodiments, the performance value is based on an accuracy of the model prior to generalization of any of the features and other factors, such as a user-configurable tolerance value (e.g., to allow for reduction of accuracy within a specified tolerance percentage). In some embodiments, the performance value is a user-configurable value that allows a user to specify a value for a performance metric, for example a user-specified accuracy value.

In some embodiments, the data generalization module404performs an iterative process that gradually increases the generalization of feature values while checking the accuracy of the predictive model such that the feature values are increasingly generalized until the accuracy of the model fails to satisfy the threshold performance value. This allows the data generalization module404to generalize feature values to values that are generalized to the extent possible while still allowing the predictive model to maintain a threshold level of accuracy.

With reference toFIG.6, this figure depicts a block diagram of an example data processing environment600in accordance with an illustrative embodiment. In some embodiments, the data processing environment600includes a ML runtime module602, a data generalization module604, a ML model evaluation module606, and a memory608, which are examples of the ML runtime module402, data generalization module404, ML model evaluation module406, and memory408, respectively, ofFIG.4. Thus, the descriptions of the ML runtime module402, data generalization module404, ML model evaluation module406, and memory408in connection withFIG.4apply equally to the ML runtime module602, data generalization module604, ML model evaluation module606, and memory608, respectively, except where described below. In still further embodiments, the ML training module202may be used in place of the ML runtime module602such that the value mapping data612and value mapping module614described below are used to map feature values in training and/or testing data and/or another dataset having a distribution that is acceptably similar according to user preferences

In the illustrated embodiment, the data generalization module604determines a generalized version of feature data that is input to an ML model and still enables the model to make predictions above a threshold level of accuracy. In some embodiments, the ML model evaluation module ML model evaluation module606then determines an accuracy of the ML model using the generalized alternative feature. The ML model evaluation module606then provides an accuracy value to the data generalization module604representative of the determined accuracy of the ML model using the generalized alternative feature. In some embodiments, the ML model evaluation module606also stores the accuracy value with the model history610in the memory608.

The data generalization module604compares the accuracy of the predictive model to a threshold performance value. In some embodiments, the threshold performance value is a baseline accuracy value of the predictive model using the original (pre-generalization) features. In some such embodiments, the threshold performance value and/or one or more prior accuracy values are stored as model history610in the memory608.

In the illustrated embodiment, if the data generalization module604determines that the accuracy of the predictive model using the alternative generalized feature values is acceptable, the data generalization module604generates value mapping data612. The value mapping data612provides data mapping instructions to a value mapping module614. In some embodiments, data mapping instructions include instructions to map feature values in a generalized range or group of values to a representative value for the generalized range or group. For example, if the generalization module604has replaced the age feature, which includes any integer from 1 to 120 as a feature value, with an alternative feature representative of age ranges, for example [1-20], [21-25], [26-50], [51-120], the value mapping data612will instruct the value mapping module614to map any age from 1 to 20 to a first representative value for the first range, map any age from 21 to 25 to a second representative value for the second range, map any age from 26 to 50 to a third representative value for the third range, and map any age from 51 to 120 to a fourth representative value for the fourth range.

In some embodiments, where the generalization is a numerical range, the representative value is a median value of the numerical range (e.g., the first representative value for the 1-20 range is 10.5). In alternative embodiments, where the generalization is a numerical range, any value in the numerical range is designated as the representative value and is thereafter used in place of any feature value in the numerical range. In some embodiments, where the generalization is for a categorical feature, such as blood type or state of residence, any value in the generalized group of categorical feature values is designated as the representative value and is thereafter used in place of any feature value in the generalized group of categorical feature values.

With reference toFIG.7, this figure depicts a block diagram of an example data processing environment700in accordance with an illustrative embodiment. In some embodiments, the data processing environment700includes a ML runtime module702, a data generalization module704, a ML model evaluation module706, and a memory708, which are examples of the ML runtime module402, data generalization module404, ML model evaluation module406, and memory408, respectively, ofFIG.4. Thus, the descriptions of the ML runtime module402, data generalization module404, ML model evaluation module406, and memory408in connection withFIG.4apply equally to the ML runtime module702, data generalization module704, ML model evaluation module706, and memory708, respectively, except where described below.

In the illustrated embodiment, the data generalization module704determines a generalized version of feature data that is input to an ML model and still enables the model to make predictions above a threshold level of accuracy. In some embodiments, the ML model evaluation module ML model evaluation module706then determines an accuracy of the ML model using the generalized alternative feature. The ML model evaluation module706then provides an accuracy value to the data generalization module704representative of the determined accuracy of the ML model using the generalized alternative feature. In some embodiments, the ML model evaluation module706also stores the accuracy value with the model history710in the memory708.

The data generalization module704compares the accuracy of the predictive model to a threshold performance value. In some embodiments, the threshold performance value is a baseline accuracy value of the predictive model using the original (pre-generalization) features. In some such embodiments, the threshold performance value and/or one or more prior accuracy values are stored as model history710in the memory708.

In the illustrated embodiment, the data processing environment700includes a user interface714. In some exemplary embodiments, the user interface714may include a Graphical User Interface (GUI), Voice User Interface (VUI), or the like. The user interface714may be utilized to obtain feature values from a user. In some such embodiments, the user interface714is a dynamic user interface that can be updated based on input provided thereto.

In the illustrated embodiment, if the data generalization module704determines that the accuracy of the predictive model using the alternative generalized feature values is acceptable, the data generalization module704generates dynamic display data712. The dynamic display data712provides instructions to update the user interface714to replace a user input field configured to request or receive the original feature values with an alternative input field configured to request or receive the alternative generalized feature values. For example, the generalization module704may replace an age feature, which includes any integer from 1 to 120 as a feature value, with an alternative feature representative of age ranges, for example [1-20], [21-25], [26-50], [51-120]. In this example, the dynamic display data712instructs the user interface714to replace a user input field that requests an age with a user input field that requests selection of one of the age ranges. The user interface714will then output to the ML runtime module702a representative value for the selected range.

In some embodiments, where the generalization is a numerical range, the representative value is a median value of the numerical range (e.g., the first representative value for the 1-20 range is 10.5). In alternative embodiments, where the generalization is a numerical range, any value in the numerical range is designated as the representative value and is thereafter used in place of any feature value in the numerical range. In some embodiments, where the generalization is for a categorical feature, such as blood type or state of residence, any value in the generalized group of categorical feature values is designated as the representative value and is thereafter used in place of any feature value in the generalized group of categorical feature values.

With reference toFIG.8, this figure depicts a block diagram of a data flow800of a data generalization module in accordance with an illustrative embodiment. In a particular embodiment, the data flow800is included in the data generalization module200.

In the illustrated embodiment, a training/testing dataset802includes several training instances804that each include predicted labels that a supervised training process can use to test the output of a model being trained. Each of the training instances804includes feature values806for a specified set of features. A data generalization process according to disclosed embodiments performs a global explainability analysis on each of the features. The global explainability analysis results in feature importance values808for each of the feature values806of each of the training instances804. In order to determine an overall importance of each feature, a sum810of all of the feature importance values808for each feature is calculated and then normalized812. In other words, a first normalized sum is calculated of all of the F_1 importance values for the first feature FEATURE_1 of training instances 1-N, and a second normalized sum is calculated of all of the F_2 importance values for the second feature FEATURE_2 of training instances 1-N, and so on through the F_M importance values for the Mth feature FEATURE_M of training instances 1-N(where M and N are implementation-specific integer values). The features are then arranged in a ranked feature list812according to importance using the normalized sum feature importance values.

With reference toFIG.9, this figure depicts a block diagram of a data flow900of a data generalization module in accordance with an illustrative embodiment. In a particular embodiment, the data flow900is includes the training dataset802having the training instances804with feature values806ofFIG.8.

In some embodiments, once the features have been ranked as shown inFIG.8, the generalization process attempts to identify one or more sets of candidate feature values of one or more selected feature. In some such embodiments, a separate generalization function (e.g., k-means, decision tree, agglomerative clustering) is trained and applied to each selected feature. In the example shown inFIG.9, the selected features are features1,2, and M. Thus, a first generalization function902is applied to the set of first feature values from each of the training instances 1-N, a second generalization function902is applied to the set of second feature values from each of the training instances 1-N, and a third generalization function902is applied to the set of Mth feature values from each of the training instances 1-N. Each generalization function902outputs a respective set of candidate feature values. In some embodiments, the generalization process applies one set of generalizing candidate feature values at a time and evaluates the predictive model before adding another set of generalizing candidate models. In some embodiments, the generalization process generates multiple generalizing candidate feature values for respective features, and then applies and evaluates two or more of the generalizing candidate feature values at a time.

With reference toFIG.10, this figure depicts a flowchart of an example process1000for generating an optimal data generalization in accordance with an illustrative embodiment. In a particular embodiment, the data generalization module200carries out the process1000.

In an embodiment, at block1004, the process computes global feature importance values. In some embodiments, the process applies a Global Explainability Analysis to a predictive model and dataset in order to determine the relative importance of each feature in the dataset, where the relative importance of a feature is indicative of a degree to which the predictive model was influenced by that feature compared to other features. In some embodiments, the global explainability analysis includes a SHapley Additive exPlanations (SHAP) analysis that results in Shapley values for each of the features in each of the input datasets (e.g., training instances). Each Shapley value represents the contribution of a given feature towards the prediction of the model for a given input dataset. The explanation presents how strongly, and which way, a given feature affects the prediction of the predictive model. In alternative embodiments, other explainability techniques may be used, for example Locally Interpretable Model Agnostic Explanations (LIME). In some embodiments, the process sums each feature's importance values across all of the training instances.

Next, at block1006, the process normalizes each sum of feature importance values, resulting in global explainability or global feature importance values for each of the features. In some embodiments, the feature importance values are Shapely additive explanation (SHAP) scores.

Next, at block1008, in some embodiments, additional scaling can be implemented to adjust the global feature importance values prior to sorting. For example, in some embodiments, the feature importance values are weighted for features involving sensitive data, meaning data that one would reasonably prefer to retain as private or confidential information is scaled to make it a higher priority for generalization in order to generalize such sensitive data. As another example, in some embodiments, feature importance values are scaled according to the feature's range, meaning features with a larger domain will be prioritized to be generalized ahead of features with smaller domains. For example, a categorical feature with four categories will be scaled by 1/4 while categorical feature with ten categories will be scaled by 1/10. This scaling works equivalently for continuous features. The assumption is that the feature with a higher initial domain can achieve better generalization.

Next, at block1010, the process sorts the features in ascending order according to the global feature importance values. Next, at bock1012, the process identifies a set of candidate feature values of one or more selected features. In some such embodiments, the process identifies the candidate feature values using a generalization function (e.g., k-means, decision tree, agglomerative clustering), meaning the generalization function minimizes the reduction in accuracy of the predictive model.

Next, at block1014, the process replaces each of the one or more selected features with an alternative feature that is a generalization of the selected feature. Then, at block1016, validation process checks the generalized alternative feature to determine what effect, if any, the generalization has on the accuracy of the predictive model. In some such embodiments, the process evaluates the ML model's ability to generalize and accurately make predictions about the updated generalized data of the training dataset and outputs an accuracy value. The process compares an accuracy of the predictive model to a threshold performance value.

In some embodiments, the process1000is an iterative process that gradually increases the generalization of feature values while checking the accuracy of the predictive model such that feature values are increasingly generalized until the accuracy of the model fails to satisfy the threshold performance value. In some embodiments, the increasing of the generalization includes increasing the level of generalization of a particular feature and/or generalizing levels of generalization of multiple features. Thus, at block1018, the process determines whether the predictive model still has a threshold level of accuracy with the generalized feature. If so, the process increases the generalization of feature values and returns to block1012. Thus, the process1000includes an iterative process that gradually increases the generalization of feature values while checking the accuracy of the predictive model such that the feature values are increasingly generalized until the accuracy of the model fails to satisfy the threshold performance value, at which point the process continues to block1022, where the generalization values up to last generalization features that resulted in an acceptable accuracy value are implemented and the process ends.

Embodiments of the present invention may also be delivered as part of a service engagement with a client corporation, nonprofit organization, government entity, internal organizational structure, or the like. Aspects of these embodiments may include configuring a computer system to perform, and deploying software, hardware, and web services that implement, some or all of the methods described herein. Aspects of these embodiments may also include analyzing the client's operations, creating recommendations responsive to the analysis, building systems that implement portions of the recommendations, integrating the systems into existing processes and infrastructure, metering use of the systems, allocating expenses to users of the systems, and billing for use of the systems. Although the above embodiments of present invention each have been described by stating their individual advantages, respectively, present invention is not limited to a particular combination thereof. To the contrary, such embodiments may also be combined in any way and number according to the intended deployment of present invention without losing their beneficial effects.