Patent Publication Number: US-2023139803-A1

Title: Continuous Delivery in Cloud Platforms of Machine Learning Models with Human in the Loop

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims the benefit of U.S. Provisional Application No. 63/273,491 filed Oct. 29, 2021, which is incorporated by reference herein. 
    
    
     FIELD OF ART 
     This disclosure relates in general to machine learning models, and in particular to monitoring execution of machine learning models for continuous delivery of machine learning models, for example in cloud platforms. 
     BACKGROUND 
     Continuous Integration and Continuous Delivery (CI/CD) techniques are used to deploy software artifacts from a development environment to a production environment. For example, CI/CD techniques allow delivery of software artifacts in cloud platforms. Certain applications used in production environment use artificial intelligence techniques such as machine learning models for making predictions. These machine learning models may be trained using training data in a development environment and deployed in production, for example, in a cloud platform. Often there is a difference in the type of data processed by the machine learning models in a production environment compared to the type of data used for training the machine learning models. 
     If the machine learning model is trained using training data that does not reflect the type of data encountered in a production environment, the predictions made by the machine learning models in production may be less accurate compared to the predictions made in the development environment. This may result in issues in downstream systems that process the predictions made by the machine learning model. For example, if a machine learning model is executed in a manufacturing facility to recognize components and determine what actions to take during a workflow being executed, incorrect workflow actions may be performed as a result of incorrect predictions of the machine learning model. 
     SUMMARY 
     A system monitors execution of a machine learning model trained using a training dataset. The system initializes a review dataset based on elements of the training dataset. The machine learning model is being executed in a production environment. The system receives a production dataset based on values received from a production environment. 
     The system samples a subset of elements of the production dataset by performing the following steps repeatedly. The system identifies an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset. The identified element is added to the review dataset. 
     The system selects one or more elements of the review dataset that were not obtained from the training dataset and sends them for presentation via a user interface. The user interface is configured to present a result of execution of the machine learning model for each sample and receive user feedback indicating accuracy of the result of execution of the machine learning model. 
     The user feedback may be used for evaluation of the machine learning model. For example, if the user feedback indicates that the machine learning model has a measure of quality that is below a threshold value, the system may recommend re-training the machine learning model to improve the accuracy. 
     According to an embodiment, the elements selected from the review dataset are prioritized for presentation via the user interface, for example, for review by a user. The priority of an element is determined based on an order in which the sample was added to the review dataset. For example, an element added to the review dataset before another element has higher priority for presenting via the user interface compared to the other element. 
     Embodiments include methods that perform the above steps, non-transitory computer-readable storage media storing instructions for performing the above methods, and computer systems that include processors and non-transitory computer-readable storage media storing instructions for performing the above methods. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG.  1    is a block diagram of a system environment for configuring and using a machine learning based model for making predictions, according to one embodiment. 
         FIG.  2    illustrates the system architecture of an online system for configuring and using a machine learning based model, according to one embodiment. 
         FIG.  3 A  illustrates a user mode for deploying an ML model according to an embodiment. 
         FIG.  3 B  illustrates a shadow mode for deploying an ML model according to an embodiment. 
         FIG.  3 C  illustrates a production mode for deploying an ML model according to an embodiment. 
         FIG.  4 A  shows the screen shot of the user interface of the visual inspection application in shadow mode, according to an embodiment. 
         FIG.  4 B  shows the screen shot of the user interface of the visual inspection application in production mode, according to an embodiment. 
         FIG.  5    shows the system architecture of the sampling module according to an embodiment. 
         FIG.  6    is a flow chart illustrating the overall process for sampling data for presenting to users, according to an embodiment. 
     
    
    
     The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the embodiments described herein. 
     The Figures (FIGS.) and the following description relate to preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of what is disclosed. 
     DETAILED DESCRIPTION 
     A system uses user feedback on artificial intelligence (AI) solutions, for example, machine learning models for improving the AI solution. The system can be operated in various modes that allow model execution as well as user inspection to evaluate a machine learning model in various environments, for example, development environment or production environment. The system receives user feedback, thereby allowing users to inspect, intervene, override, and supervise the deployed AI solution. The model evaluation may be used for determining whether to promote the machine learning model in a continuous delivery process, for example, to determine whether a machine learning model can be promoted from a development environment to a production environment. The system uses sampling strategies for selecting an optimal set of samples for presenting to users for inspection. 
       FIG.  1    is a block diagram of a system environment for configuring and using a machine learning based model for making predictions, according to one embodiment. The system environment  100  includes a computing system  110  and one or more client devices  105 . The online system includes at least a machine learning (ML) model  120  and a control module  130 . 
     The computing system  110  may represent multiple computing systems even though illustrated as one block in  FIG.  1   . Accordingly, the modules shown in  FIG.  1    and  FIG.  2    may execute in one or more computing systems. A computing system  110  may be part of a cloud platform, for example, AWS (AMAZON Web Services), GCP (GOOGLE Cloud Platform), or AZURE cloud platform. Accordingly, one or more modules may execute in the cloud platform. Furthermore, multiple instances of a module may execute, for example, the ML model  120  may execute in a development environment as well as a production environment. 
     The ML model  120  is trained to predict some results. The computing system  110  may be used for machine learning applications that make decisions based on predictions of the machine learning model. For example, the ML model  120  may be configured to receive an image  115  as input and trained to recognize certain object within the image or a feature of an object within the image. According to an embodiment, the system may capture an image of an object and the ML model may make predictions re certain feature of the object. The prediction made by the ML model is indicated as the ML prediction  135  in  FIG.  1   . For example, the system may capture images of a component in a manufacturing facility and the ML model is trained to predict whether the component is faulty. The manufacturing facility may use the predictions to make decisions regarding the component, for example, determine whether the component should be routed to a department for further inspection or the component may be routed for being delivered as a final product. The control module  130  generates control signals to perform these actions based on the predictions. For example, the control module  130  may either send a signal to be displayed via a user interface provided to an operator for taking appropriate action or the control module  130  may automatically operate equipment that routes the component as necessary based on the prediction. 
     According to an embodiment, the image  115  is provided to a visual inspection application  170  displayed via the display of a client device  105 . The visual application  170  allows a user, for example, an expert or an operator to provide feedback regarding the feature of the image being monitored. The user feedback is indicated as the user prediction  125  in  FIG.  1   . According to an embodiment, the feature determined by a user via visual inspection application  170  is the same feature regarding which a prediction is being made by the ML model  120 . The computing system  110  uses the user prediction  125  and the ML prediction in various ways depending in the mode in which the computing system  110  is configured to operate. These modes are further described herein in connection with  FIGS.  3 A-C . 
       FIG.  2    illustrates the system architecture of an online system for configuring and using a machine learning based model, according to one embodiment. The computing system  110  includes a training module  210 , a sampling module  220 , the ML model  120 , a mode selection module  230 , an ML evaluation module  240 , an ML quality assurance module  250 , the control module  130 , a training dataset  260 , and a production dataset  270 . Other embodiments may include more or fewer modules. Actions indicated as being performed by a particular module herein may be performed by other modules than those indicated. The ML model  120  and the control module  130  is described in connection with  FIG.  1   . 
     The training module  210  is used for training the ML model  120 . The training dataset  260  is used for training the ML model  120 . The training dataset may comprise labelled data where users, for example, experts view input data for the ML model and provide labels representing the expected output of the ML model for the input data. The training module  210  may initialize the parameters of the ML model using random values and use techniques such as gradient descent to modify the parameters, so as to minimize a loss function representing the difference between a predicted output and expected output for inputs of the training dataset. 
     In some embodiments, the training module  210  uses supervised machine learning to train the ML model  120 . Different machine learning techniques—such as linear support vector machine (linear SVM), boosting for other algorithms (e.g., AdaBoost), neural networks, logistic regression, naïve Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps—may be used in different embodiments. The training module  210  can periodically re-train the ML model  120  using features based on updated training data. 
     The production dataset  270  stores data collected from a production environment. For example, an ML model  120  may be trained using training dataset  260  and deployed in a production environment. The values predicted by the ML model  120  in the production environment are stored in the production dataset. According to an embodiment, the data processed in the production environment is sampled by the sampling module  220 . The samples selected by the sampling module  220  are presented to a user, for example, an operator. The data presented to the user includes the input processed by the ML model and the results as predicted by the ML model via the visual inspection application  170 . The user can provide feedback regarding the prediction of the ML model. Accordingly, the user can indicate whether the prediction of the ML model  120  is accurate or poor. This feedback is used by the ML quality assurance module  250  for testing the quality of the ML model in production environment. Similar process may be used in a development or staging environment for evaluating the ML model by the ML evaluation module  240 . According to an embodiment, the ML evaluation module  240  determines metrics such as precision, recall, and accuracy of the ML model  120  based on production data to evaluate the ML model. 
       FIGS.  3 A-C  illustrate various modes in which the computing system  110  can operate for deploying an ML model. These modes may be used for example, in a manufacturing facility for controlling workflow related to some components  310 . An image  115  of the component  310  is captured and is used to determine what action to take for the component based on either visual inspection or ML model or both. 
       FIG.  3 A  illustrates a user mode for deploying an ML model according to an embodiment. In the user mode, the prediction of the value of a feature of the component  310  is made by a user via visual inspection. The control module uses the user predictions to make determinations re the actions taken with respect to the component  310 . 
     In this mode, the image  115  of the component  310  is sent by the computing system  110  to a visual inspection application  170  running on the client device  105 . A user makes a determination regarding a specific feature of the component, for example, whether the component is defective. The determination by the user is referred to as the user prediction  225 . The user prediction  225  is provided to the control module  130 . The control module  130  generates the signals necessary to take the appropriate action associated with the component based on the user prediction  225 . For example, a particular action A 1  may be taken if the user prediction  225  indicates a particular value of the feature (e.g., feature indicating that the component is faulty), and a different action A 2  may be taken if the user prediction  225  indicates a different value of the feature (e.g., feature indicating that the component is not faulty). 
       FIG.  3 B  illustrates a shadow mode for deploying an ML model according to an embodiment. In the shadow mode, the prediction of the value of the feature is made by a user via visual inspection. However, a prediction is also made by the ML model. The control module uses the user predictions to make determinations re the actions taken with respect to the component  310 . The two predictions can be compared to evaluate the ML model and see how it is likely to perform in production without actually using the predictions of the ML model for making decisions re the components. 
     As shown in  FIG.  3 B , the image  115  of the component is provided as input to both the visual inspection application  170  and the ML model  120 . The user views the visual inspection application  170  and make the user prediction  225  of the value of the feature of the component. The ML model  120  makes the ML prediction  235  of the value of the feature of the component. The user predictions are provided to the control module to control module  130  and the control module  130  generates the signals necessary to take the appropriate action associated with the component based on the user prediction  225 . The ML prediction  235  is used to evaluate the ML model  120 , for example, to measure the performance of the ML model when processing input data obtained in production. The evaluation may be performed by ML evaluation module  240 . The system may store the ML predictions  235  obtained by execution of the ML model and the user prediction  225  in logs for processing at a later stage. 
       FIG.  3 C  illustrates a production mode for deploying an ML model according to an embodiment. In production mode, the image obtained from a component is processed both by the ML model  120  and by a user performing visual inspection. However, control module uses the ML predictions to make determinations re the actions taken with respect to the component  310 . 
     As shown in  FIG.  3 C , the image  115  of the component is provided as input to both the visual inspection application  170  and the ML model  120 . The ML model  120  makes the ML prediction  235  of the value of the feature of the component. The ML predictions  235  are provided to the control module to control module  130  and the control module  130  generates the signals necessary to take the appropriate action associated with the component based on the ML prediction  235 . The user also views the visual inspection application  170  and make the user prediction  225  of the value of the feature of the component. 
     According to an embodiment, not all data values obtained in production are provided to the visual inspection application  170 . The system may store the user predictions  225  provided by the user and also the ML predictions  235  obtained by execution of the ML model in logs for processing at a later stage. The user prediction  225  is used for quality assurance purposes. For example, the ML Model quality assurance module  250  may process the logs to determine how the ML model  120  performed in production environment. If the ML model  120  performs poorly in certain contexts, the information may be provided, for example, to developers or testers to further evaluate the ML model. For example, a determination by the ML quality assurance module  250  that the ML model performs poorly for certain type of inputs may be used for obtaining training data based on that particular type of inputs and using for retraining the ML model  120 . 
     The system may operate in other modes not described in  FIGS.  3 A-C , for example, an experimental mode in which the ML model is used for processing all the inputs and the visual inspection application is not used. This mode may be used during development and testing of the ML model  120 . 
     The different modes of the system illustrated herein are used in a CI/CD pipeline for deploying ML models, for example, in a cloud platform. For example, an experimental mode may be used for building the ML model in a development environment. While the ML model is being developed, the production environment is handled using the user mode. When the ML model passes the criteria for being promoted to the next stage, for example, staging environment, the shadow mode may be used for evaluating the ML model  120 . When the ML model  120  is evaluated to determine that the ML model satisfies the required quality metrics for being promoted to a production stage, the system operates in the production mode. 
     According to an embodiment, the computing system  110  reconfigures the user interface of the visual inspection application  170  based on the mode of the system which in turn is determined based on the type of environment that the system is operating in. The automatic reconfiguration of the visual inspection application allows the system to automate a continuous integration/continuous deployment pipeline being executed for deployment of the ML models, for example, in cloud platforms. 
       FIGS.  4 A-B  show screen shots of the user interface used for performing visual inspection according to an embodiment.  FIG.  4 A  shows the screen shot of the user interface of the visual inspection application in shadow mode, according to an embodiment. The user interface presents an image  410  being processed to the user, for example, an image of a component in a manufacturing facility. The user is provided with buttons or any other widget for providing input for example, drop down lists, text boxes, and so on. For example, button  420  allows user to indicate that the component displayed in the image  410  is good (i.e., OK) and button  430  allows the user to indicate that the component displayed in the image is not good (i.e., NG). 
       FIG.  4 B  shows the screen shot of the user interface of the visual inspection application in production mode, according to an embodiment. The image  440  presented to the user includes the result of the processing performed by the ML model  120 . Widgets  450 ,  460  are provided to the user to provide inputs indicating whether the user accepts or rejects the prediction of the ML model respectively. 
     Sampling of Data for Model Evaluation 
     In a production environment, an ML model  120  may be invoked from hundreds to tens-of-thousands of times a day. Embodiments present the input processed by the ML model, for example, an image to users to receive user feedback for evaluating the model execution in production or another environment. Since an ML model may be invoked a very large number of times in a production environment, it is infeasible for a user to review every single prediction of the ML model. 
     The sampling module  220  samples a subset of the production data for review by users as shown in  FIG.  3 C . There are several sampling strategies that may be used for example, time-based, threshold-based, and class-based sampling. Several sampling strategies generate samples that do not cover the entire population distribution. These strategies typically generate a poor sample since they may use samples that are similar to the training dataset and as a result do not address the problem that the model may not perform well if the production data is different from the training data. Furthermore, these samples may all have similar features and leave out large portions of feature values that may be available in the production data. To achieve good coverage of the data using these strategies, a large number of samples may have to be selected. 
     In contrast, the system according to various embodiments, maximizes variety in the content of the input data. As a result, a small set of samples extracted from the production data is able to provide adequate coverage. 
       FIG.  5    shows the system architecture of the sampling module  220  according to an embodiment. The sampling module  220  includes a feature extraction module  510 , a feature vector distance module  520 , and a sample selection module  530 . Other embodiments may have more or fewer modules than those indicated in  FIG.  5   . 
     The feature extraction module  510  extracts features of the elements of the data processed by the ML model  120 . According to an embodiment, the data processed by the ML model comprises images, for example, images of components in a manufacturing facility or images of objects that are being monitored by a system performing computer vision. The ML model may be a model configured to process images, for example, a convolutional neural network. The feature extraction module  510  may extract either global (i.e., image-level) features that process the entire image or local (e.g., patch-level) features that process portions of images. Global features capture large-scale attributes of the image (e.g., lighting changes). Local features capture smaller, localized features like defects in an object observed in a portion of the image. 
     According to an embodiment, the system uses a convolutional neural network to extract global features from an image. The system extracts outputs of an intermediate (or hidden) layer of the neural network. The system may apply global max pooling across the height/width dimensions, to generate a single vector. The resulting vector summarizes the global content of the image and represents large-scale changes such as lighting changes. 
     In some embodiments the system extracts local features that are more useful in certain domains, for example, manufacturing facilities. The system obtains the entire feature volume (for example, a vector in three-dimensional space H×W×C), and processes it as a collection of H×W vectors, each of dimensionality C. In this representation, each vector corresponds spatially to a patch in the original input image. The system considers the feature representation of the image as the collection of these H×W vectors. In this way, the system preserves local information within the image. This however increases size of each feature representation. 
     The feature vector distance module  520  determines a measure of distance between two samples representing data processed by the ML model. According to an embodiment, the system generates feature vector representations of each sample and determines a measure of distance between two feature vectors, for example, based on an L 1  norm or L 2  norm. 
     The sample selection module  530  selects samples from production data based on techniques disclosed herein, for example, based on the process disclosed in  FIG.  6   . The sample selection module  530  selects samples representing a subset of the data that is a good representation of the production distribution. The sample selection module  530  determines an ordering of the sampled subset of production data. The system uses the order in which the samples are provided as an indication of priority of each sample. Accordingly, the system provides the samples to users in the order of priority so as to achieve the best utilization of the available resources. 
       FIG.  6    is a flow chart illustrating the overall process  600  for sampling data for presenting to users, according to an embodiment. The steps of the process may be executed in an order different from that indicated herein. The steps are indicated as executed by a system, for example, the computing system  110  and may be executed by modules indicated in  FIG.  1 ,  2   , or  5 . 
     The system receives  610  a machine learning model trained using a training dataset D T . The system initializes  620  a review dataset D R  based on elements of the training dataset. The review dataset may also be referred to as a core set. For example, the system may initialize the dataset D R  to the training dataset D T . The system receives  630  a production dataset D P  generated using values received from a production environment. For example, the system may extract inputs processed by the ML model executing in a production environment and use them as the production dataset D P . 
     The system samples a subset of elements of the production dataset by repeatedly executing the steps  640  and  650 . The system identifies  640  an element of the production dataset D P  that maximizes a measure of minimum distance of the element from elements of the review dataset D R . The system adds  650  the identified element to the review dataset D R . 
     The system selects  660  one or more elements of the review dataset that were not obtained from the training dataset. For example, the system may remove all elements of the training dataset D T  from the review dataset D R . The system sends 670 elements selected from the review dataset for presentation via the user interface of the visual inspection application  170 . The visual inspection application  170  presents a result of execution of the machine learning model for an element of the review dataset and receives user feedback indicating accuracy of the result of execution of the machine learning model. The user feedback may be logged and in addition or in the alternative may be further processed to evaluate the ML model. For example, if the user feedback indicates that the ML model has a measure of quality below a threshold value, the system may send a request to re-train the ML model. According to an embodiment, the system may analyze the user feedback to identify types if features of the production dataset that indicate lower accuracy of the ML model so that training dataset having these types of features is added to the training dataset while retraining the ML model. 
     The elements selected from the review dataset are prioritized for presentation via the user interface. The priority of a sample is determined based on an order in which the sample was added to the review dataset. Accordingly, an element E 1  added to the review dataset before an element E 2  has higher priority for presenting via the user interface compared to the element E 2 . The system may select a subset of elements of the review dataset based on the priority. The system may also make a selection of the users processing the elements based on the priority, for example, a more experienced user may be given elements with higher priority compared to a user with less experience. 
     A process similar to that shown in  FIG.  6    may be used at training time or at production time. At training time, the system may initialize the review dataset to empty, i.e., a set with no elements. The process of  FIG.  6    is optionally executed to generate a summarized training dataset that represents a subset of the training dataset with statistical properties that are similar to the original training dataset. The summarized training dataset is used for training the model, or substituted for the full training dataset in downstream tasks to improve computational efficiency. 
     At inference time (for example, in a production environment where the machine learning model is used), the system initializes the review dataset to a training dataset used for training the model. The elements of the training dataset are removed from the review dataset when sending elements for review. 
     If the training dataset is large, executing the process of  FIG.  6    at inference time may be computationally expensive. As an optimization, in some embodiments, the summarized training dataset is substituted for the entire training dataset at inference time to improve computational efficiency of execution. 
     Following is a pseudocode illustrating the process of  FIG.  6    according to an embodiment. The following process receives as input a set of feature vectors z, and selects K feature vectors that best cover the space spanned by z. The system also receives as input a set of feature vectors z_preexisting that is initialized to empty for generating a summarized training dataset, which can be substituted for the full training dataset in downstream tasks to reduce computational expense. Alternatively, z_preexisting is initialized to the training dataset or summarized training dataset for generating a review dataset (reviewdataset) that excludes elements of the received z_preexisting set for providing to users for review via visual inspection. 
     
       
         
           
               
               
             
               
                   
                   
               
             
            
               
                   
                 if z_preexisting is empty: 
               
               
                   
                  # If no preexisting vectors, then all are equally good. 
               
               
                   
                  # Just choose one at random to start. 
               
               
                   
                  let v_chosen = select one vector randomly from z 
               
               
                   
                  for each vector v in z: 
               
               
                   
                   let min_dist[v] = distance between v and v_chosen 
               
               
                   
                  add v_chosen to reviewdataset 
               
               
                   
                 else: 
               
               
                   
                  for each vector v in z: 
               
               
                   
                   let min_dist[v] = +infinity 
               
               
                   
                  for each vector v_p in z_preexisting: 
               
               
                   
                   for each vector v in z: 
               
               
                   
                    let dist[v] = distance between v and v_p 
               
               
                   
                    update min_dist[v] = min(min_dist[v], dist[v]) 
               
               
                   
                  select v_chosen maximizing min_dist[v_chosen] 
               
               
                   
                  add v_chosen to reviewdataset 
               
               
                   
                 repeat K − 1 times: 
               
               
                   
                  for each vector v in z: 
               
               
                   
                   let dist[v] = distance between v and v_chosen 
               
               
                   
                   update min_dist[v] = min(min_dist[v], dist[v]) 
               
               
                   
                  select v_chosen maximizing min_dist[v_chosen] 
               
               
                   
                  add v_chosen to reviewdataset 
               
               
                   
                 return reviewdataset 
               
               
                   
                   
               
            
           
         
       
     
     In the above process, the system repeatedly selects an unchosen feature vector that is furthest away from current reviewdataset, i.e., v_chosen is a feature vector that maximizes the value of min_dist (minimum distance) from elements of the reviewdataset. The system adds the v_chosen feature vector to the reviewdataset and updates the min_dist (minimum distance) values of vectors of z and review dataset. 
     The sampling strategy as disclosed by the above processes selects elements (e.g., images) that cover the production data well and lie outside the training set. This prevents the system from selecting elements that are similar to the training dataset. The process also builds up the review dataset in priority order that can be used to prioritize the review process. Accordingly, the first element sampled has the highest priority for review and the last element sampled has the lowest priority for review. 
     The ability to prioritize elements for review allows the system to select a subset of elements that are review, thereby resulting in improvement of efficiency of execution and efficiency of resource utilization. For example, improvement in efficiency of use of computational resources since fewer samples are processed, improvement in efficiency of use of storage resources since fewer samples need to be stored as well as improvement in efficiency of use of network since fewer samples are transmitted the user for review. Furthermore, the techniques disclosed improve user efficiency since fewer user resources are consumed while maximizing coverage for a given amount of resources. 
     Additional Configuration Considerations 
     Throughout this specification, plural instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. Structures and functionality presented as separate components in example configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein. 
     Certain embodiments are described herein as including logic or a number of components, modules, or mechanisms. Modules may constitute either software modules (e.g., code or instructions embodied on a non-transitory computer readable storage medium or machine-readable medium) or hardware modules. A hardware module is tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client or server computer system) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein. 
     In various embodiments, a hardware module may be implemented mechanically or electronically. For example, a hardware module may comprise dedicated circuitry or logic that is permanently configured (e.g., as a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware module may also comprise programmable logic or circuitry (e.g., as encompassed within a general-purpose processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations. 
     Accordingly, the term “hardware module” should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. As used herein, “hardware-implemented module” refers to a hardware module. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where the hardware modules comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respective different hardware modules at different times. Software may accordingly configure a processor, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time. 
     Hardware modules can provide information to, and receive information from, other hardware modules. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple of such hardware modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) that connect the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware modules may also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information). 
     The various operations of example methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. The modules referred to herein may, in some example embodiments, comprise processor-implemented modules. 
     Similarly, the methods described herein may be at least partially processor-implemented. For example, at least some of the operations of a method may be performed by one or processors or processor-implemented hardware modules. The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processor or processors may be located in a single location (e.g., within a home environment, an office environment or as a server farm), while in other embodiments the processors may be distributed across a number of locations. 
     The one or more processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs).) 
     The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the one or more processors or processor-implemented modules may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the one or more processors or processor-implemented modules may be distributed across a number of geographic locations. 
     Some portions of this specification are presented in terms of algorithms or symbolic representations of operations on data stored as bits or binary digital signals within a machine memory (e.g., a computer memory). These algorithms or symbolic representations are examples of techniques used by those of ordinary skill in the data processing arts to convey the substance of their work to others skilled in the art. As used herein, an “algorithm” is a self-consistent sequence of operations or similar processing leading to a desired result. In this context, algorithms and operations involve physical manipulation of physical quantities. Typically, but not necessarily, such quantities may take the form of electrical, magnetic, or optical signals capable of being stored, accessed, transferred, combined, compared, or otherwise manipulated by a machine. It is convenient at times, principally for reasons of common usage, to refer to such signals using words such as “data,” “content,” “bits,” “values,” “elements,” “symbols,” “characters,” “terms,” “numbers,” “numerals,” or the like. These words, however, are merely convenient labels and are to be associated with appropriate physical quantities. 
     Unless specifically stated otherwise, discussions herein using words such as “processing,” “computing,” “calculating,” “determining,” “presenting,” “displaying,” or the like may refer to actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information. 
     As used herein any reference to “one embodiment” or “an embodiment” means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment. 
     Some embodiments may be described using the expression “coupled” and “connected” along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. For example, some embodiments may be described using the term “connected” to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term “coupled” to indicate that two or more elements are in direct physical or electrical contact. The term “coupled,” however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other. The embodiments are not limited in this context. 
     As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present). 
     In addition, use of the “a” or “an” are employed to describe elements and components of the embodiments herein. This is done merely for convenience and to give a general sense of the invention. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise. 
     Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for a system and a process for improving training data of a machine learning model through the disclosed principles herein. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope defined herein.