Model change management of online software as a medical device

Machine learning model change management in an online Software as a Medical Device (“SaMD”) is provided. One or more machine learning models implemented in a medical domain may be monitored where the one or more machine learning models are continuously updated. One or more changes to the one or more machine learning models. The one or more machine learning models, having the one or more changes, are certified as being in compliance with performance characteristics and compliance criteria.

BACKGROUND

The present invention relates in general to computing systems, and more particularly, to various embodiments for providing machine learning model change management in an online Software as a Medical Device (“SaMD”) in a computing systems using a computing processor.

SUMMARY

According to an embodiment of the present invention, a method for providing machine learning model change management in of an online Software as a Medical Device (“SaMD”) in a computing environment, by one or more processors, is depicted. One or more machine learning models implemented in a medical domain may be monitored where the one or more machine learning models are continuously updated. One or more changes to the one or more machine learning models. The one or more machine learning models, having the one or more changes, are certified as being in compliance with performance characteristics and compliance criteria.

An embodiment includes a computer usable program product. The computer usable program product includes a computer-readable storage device, and program instructions stored on the storage device.

An embodiment includes a computer system. The computer system includes a processor, a computer-readable memory, and a computer-readable storage device, and program instructions stored on the storage device for execution by the processor via the memory.

Thus, in addition to the foregoing exemplary method embodiments, other exemplary system and computer product embodiments are provided.

DETAILED DESCRIPTION OF THE DRAWINGS

In today's society, consumers, businesspersons, educators, and others communicate over a wide variety of mediums in real time, across great distances, and many times without boundaries or borders. The advent of computers and networking technologies have made possible the increase in the quality of life while enhancing day-to-day activities.

Computing systems may be found in the workplace, at home, or at school. Due to the recent advancement of information technology and the growing popularity of the Internet, a wide variety of computer systems have been used in machine learning. For example, Artificial intelligence (AI) and machine learning (ML) technologies provides critical assistance within the health care profession by generating new and important insights from the large amount of data generated during the delivery of health care every day. Medical device manufacturers use these technologies to innovate their products to better assist health care providers and improve patient care.

AI/ML in software within the medical domain is the ability to learn from real-world use and experience, and its capability to improve its performance. For example, the United States Food and Drug Administration (“FDA”) refers to software functions that are device functions as “device software functions” or “Software as a Medical Device (SaMD)” and “Software in a Medical Device (SiMD). That is, these AI/ML, systems used in medical settings are classed as “Software as a Medical Device” (SaMD) by regulators. Thus, in one aspect, as used herein, a SaMD may collect, process, diagnose, and analyze health care data to improve health, accelerate treatment, diagnose disease, or monitor various health care data. Moreover, the FDA further distinguishes AI/ML models as being either “locked” (e.g., unchanged), and “online” (e.g., those SaMD's that dynamically learn, retrain, and adapt based on new data).

Vendors of such AI/ML systems must seek pre-approval (e.g., regulatory approval) for any modifications to their system, if there are any changes in performance, inputs/outputs, or intended use. These regulators have several approval pathways for such systems and have a risk classification system.

As used herein, the term SaMD may be defined as software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device. Medical device may mean any instrument, apparatus, implement, machine, appliance, implant, reagent for in vitro use, software, material or other similar or related article, intended by the manufacturer to be used, alone or in combination, for human beings, for one or more of the specific medical purpose(s) such as, for example, diagnosis, prevention, monitoring, treatment, alleviation of disease, injury, physiological process, compensation for an injury, investigation, replacement, modification, or support of the anatomy or of a physiological process, supporting or sustaining health and life. Thus, a medical device may be any device used in the medical domain.

While machine model retraining can be automatic, several operations in retraining a machine model are not automatic. Thus, the present invention provides for automatically providing machine learning model change management in an online Software as a Medical Device (“SaMD”) in a computing environment. One or more machine learning models implemented in a medical domain may be monitored where the one or more machine learning models are continuously updated. One or more changes may be made to the one or more machine learning models. The one or more machine learning models, having the one or more changes, are certified as being in compliance with performance characteristics and compliance criteria.

The present invention provides for automatically assessing changes to clinical or analytical performance (e.g., improved specificity of method in detecting abnormal tissue in a scan) is not automatic, determining the impact of a change in disparate across various demographic groups, data signals in the model impact analytical performance. Also, embodiments of the present invention provide for operations such as, for example, determining the impact of the change disparate across various demographic groups, determining whether new data signals in the model impact analytical performance, determining if model updates involve a change to the scope of use, and if so, determining whether these changes require a re-certification of the product.

In an additional aspect, the present disclosure provides an AI/ML model management function for online AI models such as, for example, the ability to describe (e.g., in plain text or interpretable data) clinical context for an online AI/ML model. In an additional aspect, the present disclosure provides the ability to update/retrain model and trigger model validation (both clinical and analytical). In an additional aspect, the present disclosure provides the ability to configure the model (e.g., data selection, model tuning, and/or other features).

In one aspect, the present disclosure provides for model comparison functions such as, for example, the ability to compare a newly trained model with previous model and compare and contrast performance measures between AI/ML model versions.

In an additional aspect, the present disclosure provides for the ability to provide a human interpretable summary of changes, manage regulatory implications of AI/ML model changes (e.g., functionality to determine if changes have a material impact on compliance, and functions) to determine if changes may require re-certification of product. In another aspect, the present disclosure provides for an alerting component or signal to a vendor if an online/adaptable model has undergone substantial change. Thus, the AI/ML model may learn, collect, monitor, manage, store, and/or update input and training data.

As used herein, a user may be defined as, for example, a human (e.g., patient, expert, a care coordinator, nurses, doctors, counselors, etc.), vendor, manufacturer, a client and/or customer of a business (either an entity or human), and/or an entity/person that has previously interacted with one or more components of the present invention. The user may be an entity/person interacting with the present invention as described herein. Input data may include a health state of a user, historical medical data, patient/user profile and history, similarity of the patient/user with other (anonymized) client profiles, history of interactions with other (anonymized) patient/user, activities of daily living (“ADLs”), goals (formulated as text) defined by one or more domain experts.

In one aspect, the health state (e.g., wellness) may include at least one or more medical conditions of one or more clients, a health state (e.g., subjective health state “SWB”, emotional health state, mental health state, physical health state, or an overall health state) of the one or more clients, an emotional state of the one or more clients, biometric data, behavior patterns, a health profile of the client, or a combination thereof. In one aspect, health state may be generally described as a normal/standardized or satisfactory condition of existence of the client, or a state characterized by health, happiness, emotional stability, mental stability, physical stability, or success. As one of ordinary skill in the art will appreciate, “health state” may be dependent on a number of factors, including such factors as medical condition, emotional stability, mental stability, physical stability, financial stability, a degree or level of happiness, or other factors that may be learned. A health state of a client/patient may be defined. For example, a knowledge base or ontology may be used to define a health state for a client/patient and may include defining and/or indicating one or more correlations between a health state, a plurality of states, medical conditions, activities of daily living (ADL), and context of daily living (CDL).

Moreover, as used herein, ADLs may refer to the most common activities that people perform during a day. For example, activities of daily living may include many activities that take place throughout the day, particularly going to work, child-care, elderly care, health management, communication management, financial management, safety/emergency responses, shopping, visiting friends or family, traveling, housekeeping, grooming or personal hygiene practices, meal preparation/dining out, engaging in social media, and even using a computer. ADLs may also be used in terms of healthcare to refer to the person's daily self-care activities. The context of daily living (“CDL” or “CDLs”) may refer to the context in which one or more ADLs are executed or carried out. The CDL may also include one or more dimensions such as, for example, time, location, environment conditions, weather conditions, traffic conditions, and the like. A domain knowledge may provide one or more correlations or relationships between a person's health state and the ADLs and CDLs.

Some ADLs may also be applicable for one or more types of specific events. For example, a person having experienced a recent surgical procedure may require different or altered ADLs for treatment, recovery, or even resuming previously enjoyed ADLs. Each organism (e.g., person) may have different ADLs than other persons. Accordingly, the ADLs for each person may be learned, identified, and analyzed as part of the machine learning models implemented in a medical device. In one aspect, the ADLs for a person may be learned such as, for example, using machine learning or using a domain knowledge relating to information about the person's activities and behaviors, which may be stored in a patient profile.

It should be noted as used herein, “intelligent” (or “intelligence”) may refer to a mental action or process of acquiring knowledge and understanding through thought, experience, and one or more senses using machine learning (which may include using sensor-based devices or other computing systems that include audio or video devices). “Intelligence” may also refer to identifying patterns of behavior, leading to a “learning” of one or more events, operations, or processes. The term “intelligent” or “intelligence” may refer to an artificial intelligent/machine learning system. The intelligent system may be a specialized computer system, or set of computer systems, configured with hardware and/or software logic (in combination with hardware logic upon which the software executes) to emulate human cognitive functions. These intelligent systems apply human-like characteristics to convey and manipulate ideas which, when combined with the inherent strengths of digital computing, can solve problems with a high degree of accuracy (e.g., within a defined percentage range or above an accuracy threshold) and resilience on a large scale. An intelligent system may perform one or more computer-implemented cognitive operations that approximate a human thought process while enabling a user or a computing system to interact in a more natural manner. An intelligent system may comprise artificial intelligence logic, such as natural language processing (NLP) based logic, for example, and machine learning logic, which may be provided as specialized hardware, software executed on hardware, or any combination of specialized hardware and software executed on hardware. The logic of the intelligent system may implement the cognitive operation(s), examples of which include, but are not limited to, question answering, identification of related concepts within different portions of content in a corpus, and intelligent search algorithms, such as Internet web page searches.

In general, such intelligent systems are able to perform the following functions: 1) Navigate the complexities of human language and understanding; 2) Ingest and process vast amounts of structured and unstructured data; 3) Generate and evaluate hypotheses; 4) Weigh and evaluate responses that are based only on relevant evidence; 5) Provide situation-specific advice, insights, estimations, determinations, evaluations, calculations, and guidance; 6) Improve knowledge and learn with each iteration and interaction through machine learning processes; 7) Enable decision making at the point of impact (contextual guidance); 8) Scale in proportion to a task, process, or operation; 9) Extend and magnify human expertise and cognition; 10) Identify resonating, human-like attributes and traits from natural language; 11) Deduce various language specific or agnostic attributes from natural language; 12) Memorize and recall relevant data points (images, text, voice) (e.g., a high degree of relevant recollection from data points (images, text, voice) (memorization and recall)); and/or13) Predict and sense with situational awareness operations that mimic human cognition based on experiences.

It should be noted that one or more calculations may be performed using various mathematical operations or functions that may involve one or more mathematical operations (e.g., solving differential equations or partial differential equations analytically or computationally, using addition, subtraction, division, multiplication, standard deviations, means, averages, percentages, statistical modeling using statistical distributions, by finding minimums, maximums or similar thresholds for combined variables, etc.).

In general, such cognitive systems are able to perform the following functions: 1) Navigate the complexities of human language and understanding; 2) Ingest and process vast amounts of structured and unstructured data; 3) Generate and evaluate hypotheses; 4) Weigh and evaluate responses that are based only on relevant evidence; 5) Provide situation-specific advice, insights, estimations, determinations, evaluations, calculations, and guidance; 6) Improve knowledge and learn with each iteration and interaction through machine learning processes; 7) Enable decision making at the point of impact (contextual guidance); 8) Scale in proportion to a task, process, or operation; 9) Extend and magnify human expertise and cognition; 10) Identify resonating, human-like attributes and traits from natural language; 11) Deduce various language specific or agnostic attributes from natural language; 12) Memorize and recall relevant data points (images, text, voice) (e.g., a high degree of relevant recollection from data points (images, text, voice) (memorization and recall)); and/or13) Predict and sense with situational awareness operations that mimic human cognition based on experiences.

It should be noted that a cognitive system may also perform one or more calculations that may be performed using various mathematical operations or functions that may involve one or more mathematical operations (e.g., solving differential equations or partial differential equations analytically or computationally, using addition, subtraction, division, multiplication, standard deviations, means, averages, percentages, statistical modeling using statistical distributions, by finding minimums, maximums or similar thresholds for combined variables, etc.).

In general, as used herein, “optimize” may refer to and/or defined as “maximize,” “minimize,” “best,” or attain one or more specific targets, objectives, goals, or intentions. Optimize may also refer to maximizing a benefit to a user (e.g., maximize a trained machine learning scheduling agent benefit). Optimize may also refer to making the most effective or functional use of a situation, opportunity, or resource.

Additionally, optimizing need not refer to a best solution or result but may refer to a solution or result that “is good enough” for a particular application, for example. In some implementations, an objective is to suggest a “best” combination of operations, schedules, PE's, and/or machine learning models/machine learning pipelines, but there may be a variety of factors that may result in alternate suggestion of a combination of operations, schedules, PE's, and/or machine learning models/machine learning pipelines yielding better results. Herein, the term “optimize” may refer to such results based on minima (or maxima, depending on what parameters are considered in the optimization problem). In an additional aspect, the terms “optimize” and/or “optimizing” may refer to an operation performed in order to achieve an improved result such as reduced execution costs or increased resource utilization, whether or not the optimum result is actually achieved. Similarly, the term “optimize” may refer to a component for performing such an improvement operation, and the term “optimized” may be used to describe the result of such an improvement operation.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

As previously stated, the present invention provides novel solutions for providing online AI/ML models that provide the ability to describe clinical context for an online AI/ML model. The online AI/ML model may update/retrain the model and trigger model validation (both clinical and analytical). The online AI/ML model may configure an online machine learning model (e.g., configure data selection, model tuning, and any new features or parameters). The online AI/ML model may provide a model comparison functions, i.e., the ability to compare a newly trained model with previous AI/ML model. The online AI/ML model may compare and contrast performance measures between model versions. The online AI/ML model may provide the ability to provide a user-interpretable (e.g., understandable by a human) summary of changes. The online AI/ML model may manage regulatory implications of AI model changes, i.e., functionality to determine if changes have a material impact on compliance, operations to determine if changes may require re-certification of product, alternations to vendor if an online/adaptable model has undergone substantial change.

Turning now toFIG.4, a block diagram depicting exemplary functional components of system400for providing machine learning model change management in of an online Software as a Medical Device (“SaMD”) in a computing environment according to various mechanisms of the illustrated embodiments is shown. In one aspect, one or more of the components, modules, services, applications, and/or functions described inFIGS.1-3may be used inFIG.4. As will be seen, many of the functional blocks may also be considered “modules” or “components” of functionality, in the same descriptive sense as has been previously described inFIGS.1-3.

An interpretable model comparison service410is shown, incorporating processing unit420(“processor”) to perform various computational, data processing and other functionality in accordance with various aspects of the present invention. In one aspect, the processor420and memory430may be internal and/or external to the interpretable model comparison service410, and internal and/or external to the computing system/server12. The interpretable model comparison service410may be included and/or external to the computer system/server12, as described inFIG.1. The processing unit420may be in communication with the memory430. The interpretable model comparison service410may include a machine learning component440, an identification component450, a comparison component460, and an explainer component470.

In one aspect, the system400may provide virtualized computing services (i.e., virtualized computing, virtualized storage, virtualized networking, etc.). More specifically, the system400may provide virtualized computing, virtualized storage, virtualized networking and other virtualized services that are executing on a hardware substrate.

The interpretable model comparison service410may, using the machine learning component440, the identification component450, the comparison component460, and the explainer component470, monitor one or more machine learning models implemented in a medical domain, wherein the one or more machine learning models are continuously updated; identify one or more changes to the one or more machine learning models; and certify the one or more machine learning models having the one or more changes are in compliance with performance characteristics and compliance criteria.

Also, the interpretable model comparison service410may, using the machine learning component440, the identification component450, the comparison component460, and the explainer component470may track model versions, provenance, clinical context, and performance of a medical device. It should be noted that “provenance” may refer to or imply an origin of a specific model such as, for example, information about that original training data of the model, the original version of the algorithm used to train the model, and all original training parameters of the model.

The comparison component460, in association with the explainer component470, may provide an interpretable model comparison between a current more machine learning model and an updated machine learning model.

The identification component460may track each version of the one or more machine learning models in the medical domain. The identification component450may identify the one or more changes compromise the performance characteristics and compliance criteria of the one or more machine learning models.

The comparison component460, in association with the explainer component470, may update and retain the one or more machine learning models to achieve compliance with the performance characteristics and compliance criteria.

The comparison component460, in association with the explainer component470, may map the one or more changes of the one or more machine learning models to a risk assessment domain.

The interpretable model comparison service410may, using the machine learning component440, the identification component450, the comparison component460, and the explainer component470may build an interpretable rule set based on performance characteristics and compliance criteria, validate the one or more machine learning models against the interpretable rule set; and generate a certificate indicating the one or more machine learning models comply with interpretable rule set.

In one aspect, the machine learning component440as described herein, may perform various machine learning operations using a wide variety of methods or combinations of methods, such as supervised learning, unsupervised learning, temporal difference learning, reinforcement learning and so forth. Some non-limiting examples of supervised learning which may be used with the present technology include AODE (averaged one-dependence estimators), artificial neural network, backpropagation, Bayesian statistics, naive bays classifier, Bayesian network, Bayesian knowledge base, case-based reasoning, decision trees, inductive logic programming, Gaussian process regression, gene expression programming, group method of data handling (GMDH), learning automata, learning vector quantization, minimum message length (decision trees, decision graphs, etc.), lazy learning, instance-based learning, nearest neighbor algorithm, analogical modeling, probably approximately correct (PAC) learning, ripple down rules, a knowledge acquisition methodology, symbolic machine learning algorithms, sub symbolic machine learning algorithms, support vector machines, random forests, ensembles of classifiers, bootstrap aggregating (bagging), boosting (meta-algorithm), ordinal classification, regression analysis, information fuzzy networks (IFN), statistical classification, linear classifiers, fisher's linear discriminant, logistic regression, perceptron, support vector machines, quadratic classifiers, k-nearest neighbor, hidden Markov models and boosting. Some non-limiting examples of unsupervised learning which may be used with the present technology include artificial neural network, data clustering, expectation-maximization, self-organizing map, radial basis function network, vector quantization, generative topographic map, information bottleneck method, IBSEAD (distributed autonomous entity systems based interaction), association rule learning, apriori algorithm, eclat algorithm, FP-growth algorithm, hierarchical clustering, single-linkage clustering, conceptual clustering, partitional clustering, k-means algorithm, fuzzy clustering, and reinforcement learning. Some non-limiting example of temporal difference learning may include Q-learning and learning automata. Specific details regarding any of the examples of supervised, unsupervised, temporal difference or other machine learning described in this paragraph are known and are within the scope of this disclosure. Also, when deploying one or more machine learning models, a computing device may be first tested in a controlled environment before being deployed in a public setting. Also even when deployed in a public environment (e.g., external to the controlled, testing environment), the computing devices may be monitored for compliance.

Turning now toFIG.5, block diagram depicts exemplary operations of system500for providing machine learning model change management in of an online Software as a Medical Device (“SaMD”) in a computing environment. In one aspect, one or more of the components, modules, services, applications, and/or functions described inFIGS.1-4may be used inFIG.5. As shown, various blocks of functionality are depicted with arrows designating the blocks of system500relationships with each other and to show process flow (e.g., steps or operations). Additionally, descriptive information is also seen relating each of the functional blocks of system500. As will be seen, many of the functional blocks may also be considered “modules” of functionality, in the same descriptive sense as has been previously described inFIGS.1-4. Repetitive description of like elements employed in other embodiments described herein is omitted for sake of brevity.

With the foregoing in mind, the module blocks' of systems500may also be incorporated into various hardware and software components of a system integrating disaggregated memory in a cloud computing environment in accordance with the present invention. Many of the functional blocks of systems500may execute as background processes on various components, either in distributed computing components, or elsewhere.

In operation, training data502A,502B and raw labels504A and540B may be provided as input into an online machine learning model. That is, one or more raw labels504A and504B, along with the training data502A and502B, may be input training data502to the machine learning model A510and machine learning model B512, respectively. One or more labels (e.g., model labels) may be provided to an explainer (“BRCG”)520and522. In one aspect, BRCG (“Boolean Rules via Column Generation”) is an explanation method and used by way of example only. Also, the “grounding” between520and522refers to an optional operation to stabilize rules to enable the comparison (e.g., make a comparison feasible). Thus, the “grounding” operation may enhance or influences an interpretable learner with external information to achieve structural similar rules to enable comparison.

The interpretable rule set530and532may receive the data from an explainer520and530, respectively, and build an interpretable rule set based on performance characteristics and compliance criteria. That is, the interpreter rule set530and532may be global rule surrogates that are built in order to provide a rule comparison. This results in an interpretable clause set that identifies sub-populations impacted by the SaMD model change. In one aspect, a “global rule surrogate” may be a set of rules (e.g., a human-readable set of rules) that describes any underlying logic of any machine learning model (e.g., a complex machine learning model). For example, assume a bank uses a complex AI/ML model to approve and/or reject mortgage applications. Without the global rule surrogate, challenges are presented on determining why the AI/ML rejects some applications while approves other applications (as the underlying model logic is not known). Thus, for example, by analysis of several AI/ML mortgage approvals/rejections, the explainer520may learn and build the “global rule surrogate” that learns, detects, and understands patterns of model behavior. A surrogate rule may be, for example: Rule: If age is less than 27 years of age, income is less than $25,000, then reject the mortgage application (e.g., RULE: “age<26, income<$25K) THEN (Reject”).

Using the interpreter rule set530and532, the rule comparator540may provide an interpretable model comparison to describe qualitative differences between AI/ML model versions. The rule comparator540may map model changes to a regulatory risk assessment framework (e.g., regulatory information and risk classification) and determine if a model change has regulatory implications. For example, the risk assessment framework may be a checklist to answer if the change has been significant. In other implementations, the risk assessment framework may be a complex version involving reporting into a machine learning process or defined process to evaluate changes.

The rule comparator540may also use the regulatory risk assessment frameworks (e.g., regulatory information and risk classification), the machine learning model A510and machine learning model B512specifications and associated training data502A and502B, and raw labels504A and504B, to monitor and determine acceptable change thresholds based on tracking model versions, provenance, clinical context, and performance.

The rule comparator540may provide one or more ruleset differences550based on the comparison of the current AI/ML model and a newly trained AI/ML model. That is, the ruleset differences550show one or more differences between the comparison of the current AI/ML model and a newly trained AI/ML model and may validate the one or more machine learning models against the interpretable rule set. The rule comparator540may provide an interpretable model comparison to describe qualitative differences between model versions (e.g., the current AI/ML model and a newly trained AI/ML model).

For those of the newly trained AI/ML model that are validated by the one or more machine learning models against the interpretable rule set, the rule comparator540may generate a certificate indicating the one or more machine learning models comply with interpretable rule set.

Turning now toFIG.6, a method600for providing machine learning model change management in an online Software as a Medical Device (“SaMD”) in a computing environment using a processor is depicted, in which various aspects of the illustrated embodiments may be implemented. The functionality600may be implemented as a method (e.g., a computer-implemented method) executed as instructions on a machine, where the instructions are included on at least one computer readable medium or one non-transitory machine-readable storage medium. The functionality600may start in block602.

An interpretable model comparison (e.g., interpretable machine learning model comparison) between an existing/current model and a new model (e.g., a current machine learning model and a new machine learning model) may be provided, as in block604. The interpretable model comparison may be validated against one or more policy criteria (e.g., medical domain policy criteria), as in block606. A certificate to verify compliance and a length of applicable duration (e.g., the amount of time the certificate is valid for using) may be generated, as in block608. The functionality600may end, as in block610.

Turning now toFIG.7, a method700for providing machine learning model change management in an online Software as a Medical Device (“SaMD”) in a computing environment using a processor is depicted, in which various aspects of the illustrated embodiments may be implemented. The functionality700may be implemented as a method (e.g., a computer-implemented method) executed as instructions on a machine, where the instructions are included on at least one computer readable medium or one non-transitory machine-readable storage medium. The functionality700may start in block702.

One or more machine learning models implemented in a medical domain may be monitored where the one or more machine learning models are continuously updated, as in block704. One or more changes to the one or more machine learning models may be identified, as in block706. The one or more machine learning models, having the one or more changes, may be certified as being in compliance with performance characteristics and compliance criteria, as in block708. The functionality700may end, as in block710.

In one aspect, in conjunction with and/or as part of at least one blocks ofFIG.7, the operations of method700may include each of the following. The operations of method700may provide an interpretable model comparison between a current more machine learning model and an updated machine learning model. The operations of method700may track each version of the one or more machine learning models in the medical domain.

The operations of method700may identify the one or more changes compromise the performance characteristics and compliance criteria of the one or more machine learning models. The operations of method700may update and retain the one or more machine learning models to achieve compliance with the performance characteristics and compliance criteria.

The operations of method700may map the one or more changes of the one or more machine learning models to a risk assessment domain. The operations of method700may build an interpretable rule set based on performance characteristics and compliance criteria; validate the one or more machine learning models against the interpretable rule set; and generate a certificate indicating the one or more machine learning models comply with interpretable rule set.