Hybrid decision making automation

Techniques are provided for decision making tasks using a hybrid approach where cooperation between an AI assessor and a human labeler controls automation of the process. In one aspect, a method for hybrid decision making automation includes: monitoring interactions between an AI assistant and a human decision maker; tracking, from the interactions, agreement of the human decision maker with decision predictions made by the AI assistant; determining a predicted performance of data tasks by the AI assistant on unseen data based on the agreement of the human decision maker with the decision predictions over time; and assessing delegation of remaining data tasks on the unseen data to the AI assistant using the predicted performance.

FIELD OF THE INVENTION

The present invention relates to machine learning, and more particularly, to machine learning-based techniques for decision making tasks such as data labeling using a hybrid approach where cooperation between an artificial intelligence (AI) assessor and a human labeler controls automation of the process.

BACKGROUND OF THE INVENTION

Training a machine learning process for performing decision making tasks, such as data labeling, requires large amounts of labeled data. For instance, training a machine learning process to label images can require hundreds of thousands of labeled images to use as training data. Therefore, manually labeling the data using human labelers, is extremely time consuming and thus expensive. See, for example, Zhou et al., “Learning with Local and Global Consistency,” Proceedings of the 16thInternational Conference on Neural Information Processing Systems, December 2003 (8 pages) (hereinafter “Zhou”).

As such, alternative approaches to manual labeling have been explored. For instance, with artificial intelligence (AI)-assisted data labeling approaches, AI tooling is used to help the human labeler with everything from label look-up to making labeling suggestions. While AI-assisted data labeling enhances the productivity of the human labeler and speeds up the process, items still have to be evaluated individually. Thus, AI-assisted data labeling has scaling limitations.

Automated AI approaches automatically label the data based on a small amount of human input. Namely, a small subset of labeled data is used to derive the labels for the remaining data automatically. Some common automated data labeling approaches include weak supervision and semi-supervised learning. With weak supervision, noisy, limited, or imprecise sources are used as a supervision signal for labeling a large amount of training data. Semi-supervised learning combines a small amount of labeled data with a large amount of unlabeled data during training. While efficient and scalable, these automated approaches however require the human labeler to relinquish control of the process. Thus, there is no guarantee for the quality and accuracy of the labeled data that is produced.

Accordingly, machine learning-based techniques for performing decision making tasks such as data labeling that achieve the efficiency level of the automated approaches, but maintain level of human supervision and control as in the assisted approaches would be desirable.

SUMMARY OF THE INVENTION

The present invention provides machine learning-based techniques for decision making tasks such as data labeling using a hybrid approach where cooperation between an artificial intelligence (AI) assessor and a human labeler controls automation of the process. In one aspect of the invention, a method for hybrid decision making automation is provided. The method includes: monitoring interactions between an AI assistant and a human decision maker; tracking, from the interactions, agreement of the human decision maker with decision predictions made by the AI assistant; determining a predicted performance of data tasks (e.g., data labeling tasks) by the AI assistant on unseen data based on the agreement of the human decision maker with the decision predictions over time; and assessing delegation of remaining data tasks on the unseen data to the AI assistant using the predicted performance.

Accordingly, a delegation assessment can be presented to the human decision maker. A delegation decision can be obtained from the human decision maker, and performance of the remaining data tasks on the unseen data can be delegated to the AI assistant. Advantageously, the present techniques leverage the benefits of automated decision making such as scalability, while at the same time retain control by the human decision maker until a level of confidence in the predictions of the AI assistant has been achieved to ensure a quality and accuracy of the predictions.

Namely, the predicted performance of the data tasks can be determined using an agreement predictor model which is trained using the agreement of the human decision maker with the decision predictions over time. Once trained, the agreement predictor model can be applied to remaining unseen data to determine a predicted agreement accuracy of the AI assistant for the remaining unseen data, and an automation readiness of the AI assistant can be assessed using the predicted agreement accuracy.

In another aspect of the invention, another method for hybrid decision making automation is provided. The method includes: monitoring interactions between an AI assistant and a human decision maker; tracking, from the interactions, agreement of the human decision maker with decision predictions made by the AI assistant; determining a predicted performance of data tasks by the AI assistant on unseen data based on the agreement of the human decision maker with the decision predictions over time; assessing delegation of remaining data tasks on the unseen data to the AI assistant using the predicted performance; determining whether a pre-configured threshold in predicted AI-human agreement has been reached; and delegating performance of the remaining data tasks on the unseen data to the AI assistant when the pre-configured threshold in predicted AI-human agreement has been reached.

The above can be performed by an AI assessor. The AI assessor can operate in the background, i.e., in a silent mode, such that the human decision maker is unaware of the AI assessor's activity related to the labeling process, including delegation decisions. Doing so advantageously provides further automation assistance to the human decision maker.

In yet another aspect of the invention, yet another method for hybrid decision making automation is provided. The method includes: making decision predictions on data tasks (e.g., data labeling tasks) using an AI assistant based on decisions the AI assistant has already received from a human decision maker; presenting the decision predictions to the human decision maker for the human decision maker to agree or disagree with the decision predictions, wherein an AI assessor, which monitors interactions between the AI assistant and the human decision maker, tracks agreement of the human decision maker with the decision predictions made by the AI assistant from the interactions, determines a predicted performance of the data tasks by the AI assistant on unseen data based on the agreement of the human decision maker with the decision predictions over time, and assesses delegation of remaining data tasks on the unseen data to the AI assistant using the predicted performance; and performing the remaining data tasks on the unseen data when performance has been delegated to the AI assistant.

The decision predictions can be presented to the human decision maker along with a corresponding data item and a confidence value. Thus, as provided above, the human decision maker advantageously retains control of the process until a level of confidence in the predictions of the AI assistant has been achieved to ensure a quality and accuracy of the predictions.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Provided herein are machine learning-based techniques for decision making tasks using cooperation between an artificial intelligence (AI) assessor and a human labeler to control the level of automation of the process. The present techniques may be thought of as a hybrid approach between assisted and automated machine learning approaches (see above), leveraging the benefits of each approach. Thus, advantageously, by way of this hybrid approach, enhanced efficiency can be achieved without fully relinquishing control of the process. Thus, a greater quality and accuracy of the task performance can be guaranteed.

It is notable that the present techniques are generally applicable to any machine learning-assisted decision making process where data tasks are provided, and decisions are needed for the data tasks. For instance, as will be described in detail below, the present techniques can be employed for data labeling, where the tasks involve providing labels for data (such as image data). Other non-limiting applications include machine learning-assisted decision making in the field of medical diagnosis, where the tasks involve making diagnostic decisions based on image data such as x-ray image data to determine if a patient has certain factors related to a disease or other medical condition. Based on that information, physicians can perform related diagnostics and tests, devise appropriate treatment options, etc.

As will be described in detail below, the present techniques employ a three-party system, including an AI assistant, a human decision maker, and an AI assessor. The AI assistant takes input data and makes decision predictions based on decisions the AI assistant has already received from the human decision maker. For instance, in the context of data labeling, the AI assistant takes unlabeled input data and makes label predictions based on labels the AI assistant has already received from the human decision maker.

This process proceeds in an active learning refinement loop, where the AI assistant selects batches of the unprocessed (e.g., unlabeled) input data and presents them to the human decision maker. The human decision maker provides (true) human decisions (e.g., labels) for the presented items to the AI assistant, to be used by the AI assistant in making future decision predictions, and so on. Once a number of decisions have been made, the human decision maker or AI assessor can eventually delegate decision making (e.g., labeling) on the remainder of the data to the AI assistant (e.g., for auto-labeling) based on the delegation decision support provided by the AI assessor.

The AI assessor provides decision support by observing this interaction between the AI assistant and the human decision maker from the outside. Namely, the AI assessor tracks the AI assistant-to-human decision maker decision agreement and then predicts how well the agreement will continue to be on unseen input data. Based on this tracking and prediction, the AI assessor can assess the readiness of the AI assistant for automated operation (automation readiness) such as auto-labeling, and recommend delegating of the remaining data to the AI assistant.

In one exemplary embodiment described below, the human decision maker ultimately makes the delegation decision. However, the human decision maker advantageously has the data and statistics from the AI assessor about the tracked and predicted AI assistant-to-human decision maker agreement to aid in making the delegation decision. In an alternative embodiment described below, the AI assessor operates in ‘silent mode’ meaning that the AI assessor operates in the background such that the human decision maker is unaware of the AI assessor's activity related to the labeling process. In that case, instead of presenting the results to the human decision maker, it is the AI assessor that initiates delegation of data tasks to the AI assistant based, e.g., on reaching a pre-configured threshold in predicted AI-human agreement. By way of example only, the pre-configured threshold can be determined up front by the human decision maker. For instance, the human decision maker may decide that a 95% predicted agreement is sufficient to delegate to the AI Assistant. A 95% predicted agreement means that the AI assistant makes the same decisions (e.g., selects the same labels) as the human decision maker 95% of the time. For high stakes decisions, the threshold may be higher such as 99%.

In general, the AI assistant and the AI assessor represent any type of learning algorithm including, but not limited to, a machine learning process. The AI assistant and the AI assessor can be implemented in a computer-based apparatus such as apparatus800described in conjunction with the description ofFIG.8, below. One illustrative, non-limiting example of a machine learning process is a deep neural network. In machine learning and cognitive science, deep neural networks are a family of statistical learning models inspired by the biological neural networks of animals, and in particular the brain. Deep neural networks may be used to estimate or approximate systems and cognitive functions that depend on a large number of inputs and weights of the connections which are generally unknown.

Deep neural networks are often embodied as so-called “neuromorphic” systems of interconnected processor elements that act as simulated “neurons” that exchange “messages” between each other in the form of electronic signals. See, for example,FIG.1which provides a schematic illustration of an exemplary deep neural network100. As shown inFIG.1, deep neural network100includes a plurality of interconnected processor elements102,104/106and108that form an input layer, at least one hidden layer, and an output layer, respectively, of the deep neural network100. By way of example only, deep neural network100can be embodied in an analog cross-point array of resistive devices such as resistive processing units (RPUs).

Similar to the so-called ‘plasticity’ of synaptic neurotransmitter connections that carry messages between biological neurons, the connections in a deep neural network that carry electronic messages between simulated neurons are provided with numeric weights that correspond to the strength or weakness of a given connection. The weights can be adjusted and tuned based on experience, making deep neural networks adaptive to inputs and capable of learning. For example, a deep neural network for image labeling is defined by a set of input neurons (see, e.g., input layer102in deep neural network100) which may be activated by the pixels of an input image. After being weighted and transformed by a function determined by the network's designer, the activations of these input neurons are then passed to other downstream neurons, which are often referred to as ‘hidden’ neurons (see, e.g., hidden layers104and106in deep neural network100). This process is repeated until an output neuron is activated (see, e.g., output layer108in deep neural network100). The activated output neuron makes a label decision.

Instead of utilizing the traditional digital model of manipulating zeros and ones, deep neural networks such as deep neural network100create connections between processing elements that are substantially the functional equivalent of the core system functionality that is being estimated or approximated. For example, IBM's SyNapse computer chip is the central component of an electronic neuromorphic machine that attempts to provide similar form, function and architecture to the mammalian brain. Although the IBM SyNapse computer chip uses the same basic transistor components as conventional computer chips, its transistors are configured to mimic the behavior of neurons and their synapse connections. The IBM SyNapse computer chip processes information using a network of just over one million simulated “neurons,” which communicate with one another using electrical spikes similar to the synaptic communications between biological neurons. The IBM SyNapse architecture includes a configuration of processors (i.e., simulated “neurons”) that read a memory (i.e., a simulated “synapse”) and perform simple operations. The communications between these processors, which are typically located in different cores, are performed by on-chip network routers.

Given the above overview, an exemplary embodiment for hybrid decision making automation in accordance with the present techniques is now described by way of reference toFIGS.2and3. As highlighted above, the present scheme employs a three-party system, including an AI assistant, a human decision maker, and an AI assessor. The AI assistant coordinates with the human decision maker to make decision predictions.

FIG.2provides an exemplary methodology200for hybrid decision making automation one or more steps of which can be performed by the AI assistant when coordinating with the human decision maker and the AI assessor. In step202, the AI assistant takes input data and provides a set of decision predictions for a set of data tasks. When acting in this role of making decision predictions, the AI assistant may also be referred to herein generally as a ‘decision predictor.’ As will be described below, in one exemplary embodiment, the AI assistant (acting as a decision predictor) makes label predictions for data labeling tasks.

The decision predictions are made based on decisions the AI assistant has already received from the human decision maker such that the AI assistant and the human decision maker execute an active learning refinement loop. Namely, in step204, the AI assistant selects a batch of the (unprocessed) input data items and presents the data items to the human decision maker. According to an exemplary embodiment, the AI assistant uses a selection strategy such as low margin active learning to select the input data to present to the human decision maker. The notion here is that the AI assistant is given the task of selecting (unprocessed) input data to incorporate in the active learning refinement loop. One process which can assist in that selection is active learning. An active learning algorithm has both the capability of drawing random unlabeled examples from the underlying distribution and that of asking for the labels of any of these examples, and the hope is that a good classifier can be learned with significantly fewer labels by actively directing the queries to informative examples. See, for example, Maria-Florina Balcan et al., “Margin Based Active Learning,” Bshouty N. H., Gentile C. (eds) Learning Theory. International Conference on Computational Learning Theory COLT 2007, Lecture Notes in Computer Science, vol 4539, Springer, Berlin, Heidelberg (16 pages). In low margin (or simply margin) based active learning, the active learner chooses the most uncertain examples to maximize learning. The most uncertain examples are determined as the examples with the smallest margin between the top-2 predicted labels (or decisions).

Thus, the goal is to obtain (true) human decisions for the data from the human decision maker (see step206, described below). In that regard, it may be beneficial to assist the human decision maker with this process in order to enhance efficiency. Therefore, optionally, in step204decision predictions made by the AI assistant are also provided along with the data presented to the human decision maker. Doing so will assist the human decision maker in making the correct decision.

Presenting decision predictions made by the AI assistant to the human decision maker in step204(along with the corresponding data item and a confidence value—see below) permits the human decision maker to review the performance of the AI assistant by either agreeing or disagreeing (agreed/not agreed) with the predictions made by the AI assistant, also referred to herein as ‘AI assistant-to-human decision maker decision agreement’ or simply ‘AI-to-human agreement.’ As will be described in conjunction with the description ofFIG.3below, the AI assessor tracks this AI-to-human agreement over time. The agreement tracking by the AI assessor happens in the background during the active learning refinement loop. Thus, when human decisions are obtained in step206(described below), in the background, agreement between the human decision and prediction is measured and recorded by the AI assessor.

The decision predictions are the work of the AI assistant and the AI assessor needs to have access to them. Thus, the decision predictions always need to be shared with/visible to the AI assessor, such that the AI assessor can track agreement between the human decision and prediction in the background. However, as provided above, it is optional whether the decision predictions are also shown to the human decision maker.

In step206, (true) human decisions for the data items presented to the human decision maker are obtained from the human decision maker. These human decisions for the data items are then used by the AI assistant in making further decision predictions, and so on. Namely, as shown inFIG.2, the process is repeated iteratively, i.e., in an active learning refinement loop. In step206, the human decision maker will provide human decisions for the data items that have been presented to the human decision maker. For instance, in the case of data labeling, the human decision maker will provide labels for each of the data items. For those data items presented to the human decision maker along with decision predictions made by the AI assistant (see above), the human decision maker will either agree or disagree with the decision prediction from the AI assistant and, in the latter, change the decision (e.g., label).

Any type of semi-supervised learning, self-training/auto-labeling and/or weak supervision process can be employed to train the AI assistant using the (true) human decisions. For example, the tasks with human decisions obtained so far can be used as training data to train a supervised machine learning model to make predictions for the remaining tasks (self-training/auto-labeling). According to an exemplary embodiment, a semi-supervised learning process such as label spreading is employed. ‘Semi-supervised’ refers to learning from data processed and unprocessed data. For instance, in the context of data labeling, semi-supervised learning is learning from labeled and unlabeled data. The key to semi-supervised problems is the prior assumption of consistency, which means: (1) nearby points are likely to have the same label, and (2) points on the same structure (typically referred to as a cluster or a manifold) are likely to have the same label. See Zhou. A semi-supervised clustering technique such as label spreading may be used to propagate the clusters whereby the label information from each point is spread to its neighbors in the cluster. Semi-supervised learning uses both the decisions obtained so far as well as the information from the remaining tasks (such as proximity and neighbor information) to propagate and obtain labels for the remaining data. In label spreading, labels are propagated to neighboring items.

In step208, a decision is made as to whether to delegate performance of the data tasks on the remaining data to the AI assistant. As will be described in conjunction with the description ofFIG.3below, this delegation decision is supported by an analysis from the AI assessor which has been tracking AI-to-human agreement in the background and will predict performance of the AI assistant in making future decisions. This prediction serves as a metric for how well the performance will be once the decision making has been delegated to the AI assistant. In other words, the predicted performance is an assessment of the readiness of the AI assistant for automated operation (i.e., the automation readiness) such as auto-labeling.

Namely, at some point in the process, once a level of confidence has been met in the decisions being made by the (trained) AI assistant, the decision making capabilities can be fully delegated to the AI assistant. As highlighted above and as will be described in detail below, the decision to delegate the remaining tasks can reside wholly with the human decision maker or, alternatively, the decision to delegate the remaining data tasks to the AI assistant can be made by the AI assessor (i.e., when the AI assessor operates in a silent mode). When the delegation decision will be made by the human decision maker, the AI assessor can provide a recommendation to the human decision maker when automation readiness has been achieved. Alternatively, in silent mode, the AI assessor itself can determine when automation readiness has been achieved based, for example, on a pre-configured threshold in predicted AI-human agreement. In either case, however, the delegation decision is supported by the AI assessor which tracks the interactions between the AI assistant and the human decision maker and, from those interactions, assesses the automation readiness.

Based on the presented analysis from the AI assessor (which is provided at the end of each active learning loop iteration), if it is decided in step208that NO automation readiness has not yet been achieved, then delegation to the AI assistant is declined and training is continued with manual decision in the next iteration of the active learning refinement loop. On the other hand, if it is decided in step208that YES automation readiness has been achieved, then the decision making capabilities are fully delegated to the AI assistant and in step210the AI assistant performs the decision making on the remainder of the data tasks without receiving verification (agreed/not agreed) from the human decision maker.

FIG.3provides an exemplary methodology300for hybrid decision making automation one or more steps of which can be performed by the AI assessor when observing the interactions between the human decision maker and the AI assistant. In step302, the AI assessor observes the interactions between the human decision maker and the AI assistant. As provided above, the AI assistant makes decision predictions on input data based on decisions the AI assistant has already received from the human decision maker such that the AI assistant and the human decision maker execute an active learning refinement loop. The AI assessor monitors these AI assistant-human decision maker interactions in step302.

In step304, the AI assessor tracks the AI-to-human agreement over time. Namely, by observing the AI assistant-human decision maker interactions (see step302), the AI assessor can determine how often the (true) human decisions from the human decision maker confirms (or does not confirm) the decision predictions from the AI assistant. To use a data labeling scenario as a non-limiting example, in step304the AI assessor can track how often the predicted labels from the AI assistant concur with the labels assigned by the human decision maker on the same data items from interactions between the AI assistant and the human decision maker that have occurred over time.

Tracking the AI-to-human agreement over time enables the AI assessor to assess and characterize the quality of the AI predictions. For instance, one metric employed might be how often the human decision maker agrees with the top-N prediction from the AI assistant. To use an image labeling task as an illustrative, non-limiting example, when making a decision prediction the AI assistant will determine a probability distribution vector across possible labels for the image. If the AI assistant's top label prediction for an image is ‘dog,’ followed by ‘cat’ and ‘rabbit’ in that order, then the top-1 (where N=1) prediction is ‘dog.’ For top-2 accuracy (where N=2), the label of ‘cat’ would be considered correct, and so on. To look at it another way, the label ‘cat’ would not be considered as correct for top-1 accuracy.

In step306, based on the tracked agreement data, the AI assessor predicts the AI assistant-to-human agreement for the remaining (unseen) input data. Doing so permits an assessment to be made as to whether future predictions by the AI assistant will be made with a sufficient level of accuracy, or whether the AI assistant needs further training. That assessment will then support making the delegation decision. According to an exemplary embodiment, the predicted AI assistant-to-human agreement for the remaining (unseen) input data is made using a performance predictor model that uses the tracked agreement data to train an agreement predictor model. Through this training, the agreement predictor model will learn under what circumstances (i.e., for which data item characteristics and which AI assistant predictions and confidence value characteristics) the AI assistant is likely to receive an agreement from the human decision maker. An exemplary performance predictor model for use in accordance with the present techniques is described in conjunction with the description ofFIG.4below.

Based on the tracking (step304) and predicted performance (step306), in step308the AI assessor makes an assessment of the readiness of the AI assistant for automated operation (automation readiness). For instance, by way of example only, the assessment made in step308can be based on AI-to-human agreement, namely what percentage of the time the human decision maker agreed with the decision predictions of the AI assistant, e.g., there is 80% agreement on the last 50 data items tracked. Different agreement measures may be used. For example, only the top-1 prediction may be compared to the human decision to measure agreement. Alternatively, the entire probability distribution of the decision predictions may be compared to the human decision to determine a distance value between human decision and AI assistant prediction as a probability distribution. As highlighted above, the AI assistant will provide a confidence value along with its decision prediction on each of the data items. Thus, another potential metric is certainty, for example, how certain the AI assistant is about the accuracy of the predictions it has made over time. Using the performance predictor model (see below), the AI assistant can also determine how accurate it thinks it will be should the AI assistant begin automated operation in the dataset's current state.

As highlighted above, the delegation decision can be made by the human decision maker or by the AI assessor (e.g., when the AI assessor is operating in silent mode). With the former scenario, in step310the AI assessor provides its assessment (from step308) to the human decision maker. The human decision maker will then make a delegation decision based on the assessment provided by the AI assessor. For instance, using the examples provided above, if the predicted accuracy for automated operation of the AI assistant in the dataset's current state is low, then the human decision maker would at present choose not to delegate the remaining data tasks to the AI assistant. However, the accuracy (and other metrics of performance) are expected to improve over time, as the AI assistant sees more training data.

In step312, a decision is then made by the human decision maker as to whether to delegate performance of the remaining data items to the AI assistant. This delegation decision is obtained by the AI assessor. If it is decided in step312that NO performance of the remaining data items should not be delegated to the AI assistant, then the AI assessor continues to observe, track and predict AI-to-human agreement until a decision is obtained from the human decision maker to delegate performance of the remaining data items to the AI assistant. On the other hand, if it is decided in step312that YES performance of the remaining data items should be delegated to the AI assistant, then in step314, the AI assessor (upon obtaining this decision) delegates the remaining data tasks to the AI assistant.

Alternatively, when operating in silent mode, the AI assessor makes the decision as to whether to delegate the remaining data tasks to the AI assistant. By ‘silent mode,’ it is meant that the AI assessor operates in the background such that the human decision maker is unaware of the one or more functions the AI assessor performs in the decision making process. In this alternative ‘silent mode’ embodiment, the involvement of the human decision maker in the decision making process is the same as described above, except that the human decision maker does not control the automation delegation decision. Here, the AI assessor acts as like a second opinion for the AI assistant.

Namely, rather than presenting an assessment (based on the tracking and prediction data) to the human decision maker, in step316the AI assessor determines whether a pre-configured threshold in predicted AI-human agreement has been reached. As provided above, the pre-configured threshold can be determined up front by the human decision maker. For instance, by way of example only, if the predicted accuracy for automated operation of the AI assistant in the dataset's current state is greater than the pre-configured threshold, then the AI assessor would choose to delegate the remaining data tasks to the AI assistant. Thus, if it is determined in step316that YES the pre-configured threshold in predicted AI-human agreement has been reached, then in step318the AI assessor delegates the remaining data tasks to the AI assistant. On the other hand, if it is determined in step316that NO the pre-configured threshold in predicted AI-human agreement has not been reached, then the AI assessor continues to observe, track and predict AI-human agreement until the pre-configured threshold in predicted AI-human agreement has been reached. As noted above, the accuracy is expected to improve over time, as the AI assistant sees more training data.

An exemplary performance predictor model for use in accordance with the present techniques is now described in conjunction with the description of methodology400ofFIG.4. In the same manner as described in conjunction with the description of methodology200of FIG.2above, in step402the AI assistant is trained using (true) human decisions (e.g., labels) from the human decision maker via an iterative learning process, and in step404the trained AI assistant makes decision predictions (e.g., label predictions) for unseen (e.g., unlabeled) data.

Namely, in step404, the AI assistant takes batches of (e.g., unlabeled) data items as input and outputs decision (e.g., label) predictions along with confidence values. As provided above, when making a decision prediction the AI assistant will determine a probability distribution vector across possible decisions. The confidence value defines the probability that data items fall into the different classes, such as ‘cat’ or ‘dog’ to use the above image labeling task as an example. For instance, some machine learning models, such as neural nets, determine a probability distribution across the possible decisions (or classes in general). The probabilities are usually interpreted as confidences. Thus, if the top-1 prediction of the model has 90% probability, then it can be said that the model is 90% confident that the top-1 prediction is accurate.

As described in conjunction with the description of step206of methodology200ofFIG.2above, the data items together with the decision predictions and confidence values are then presented by the AI assistant to the human decision maker who either agrees or disagrees with the prediction. This data is collected over time. Thus, during each iteration of the active learning refinement loop, the following data is collected for each data item included in the batch:
[data item][AI assistant decision prediction][confidence value][agreed/not agreed],
also referred to herein as ‘decision prediction-agreement’ data. See step406. This data collected in step406is then used as training data for training an agreement predictor model. See step408. The agreement predictor model can be trained using any suitable process. According to an exemplary embodiment, the agreement predictor model is trained using a random forest decision model. A random forest decision model is an ensemble learning method by which a large number of individual decision trees operate as an ensemble. Each decision tree provides a prediction, and the prediction having the highest outcome amongst the decision trees becomes the overall prediction of the model. Through this training the agreement predictor model learns under what circumstances (i.e., for which data item characteristics and which AI assistant prediction and confidence value characteristics) the AI assistant is likely to receive an agreement from the human decision maker. The term ‘data item characteristics’ refers to the features extracted from the data item.

During training, the agreement predictor model takes output from the AI assistant, i.e., [data item][AI assistant decision prediction][confidence value]. Based on the training data collected in step406, the agreement predictor model outputs a predicted score, i.e., [agreed] or [not agreed].

Once trained, the agreement predictor model is applied in step410to the remaining unseen (e.g., unlabeled) data and their confidence values (from the AI assistant) to determine a predicted agreement accuracy for the remaining unseen (e.g., unlabeled) data. The predicted agreement accuracy is then used by the AI assessor in assessing the automation readiness of the AI assistant, and by the human decision maker or AI assessor in making a delegation decision as described, for example, in conjunction with the description of methodology200ofFIG.2above.

The agreement predictor model is trained on the agreed/not agreed outcomes of the AI assistant predictions. Once trained, the agreement predictor, given as input a new example together with the AI assistant predictions and confidences, can predict whether the human decision maker is likely to agree or not agree with the AI assistant's prediction.

To use the agreement predictor model on the remaining (e.g., unlabeled) examples, all of those examples are first run through the AI assistant decision predictor—in order to have the examples together with the AI assistant predictions and confidences. The agreement predictor model is then applied to each example with its prediction and confidence value, and either agreed or not agreed is obtained as a prediction from the agreement predictor model. That way, a percentage (say 98%) of the items for which it is predicted that the human decision maker would agree with the AI assistant prediction can be determined.

So, alternatively, instead of showing these examples with their predictions to the human to ‘measure’ the agreement, the agreement predictor model can instead be used to ‘predict’ the agreement. This is also called a meta-prediction, i.e., predictions are made about how good (how agreeable) the AI assistant predictions are.

FIG.5is a diagram illustrating an exemplary system500for hybrid decision making automation in accordance with the present techniques. As shown inFIG.5, system500is a three-party system including an AI assistant502, a human decision maker504, and an AI assessor506. As provided above, the AI assistant502and the AI assessor506can be implemented in a computer-based apparatus such as apparatus800described in conjunction with the description ofFIG.8, below.

In the same manner as described in conjunction with the description of methodology200ofFIG.2above, the AI assistant502takes input data such as unlabeled data from a database508(or other suitable repository) and provides a set of decision predictions, such as providing label predictions for a set of data labeling tasks. These decision predictions are made based on decisions the AI assistant502has already received from the human decision maker504as part of an active learning refinement loop. Namely, as shown inFIG.5, the AI assistant502selects a batch of the (e.g., unlabeled) data and presents it to the human decision maker504. See arrow510. As provided above, a selection strategy such as low margin active learning can be used to select the (e.g., unlabeled) data to present to the human decision maker504. In turn, the human decision maker504returns (true) human decisions (e.g., human labels) for the presented data. See arrow512.

These (true) human decisions are then used by the AI assistant502in making further decision predictions, and so on in an active learning refinement loop. As described above, the decision predictions from the trained AI assistant502are presented to the human decision maker along with the corresponding data item and a confidence value, and the human decision maker either agrees or disagrees (agreed/not agreed) with the predictions made by the AI assistant. As shown inFIG.5, the output from system500is a combination of human and AI assistant decision data (e.g., human-labeled and AI-labeled data). This output data is stored in a database514(or other suitable repository). According to an exemplary embodiment, the AI labeled data is obtained (after delegation) by assigning the top-1 prediction as the label, i.e., the data labeled using the top-1 (most probable) predicted label.

The AI assessor506observes the interactions between the AI assistant502and the human decision maker504. See arrow516. Namely, as described above, the AI assessor506tracks the AI assistant502-to-human decision maker504decision agreement (agreed/not agreed) over time in order to assess and characterize the quality of the AI predictions. Based on the tracked agreement data, the AI assessor506predicts the AI assistant-to-human agreement for the remaining (unseen) input data using, e.g., the performance predictor described in conjunction with the description of methodology400ofFIG.4above. Doing so permits an assessment to be made as to whether future predictions by the AI assistant502will be made with a sufficient level of accuracy and therefore support a decision to delegate labeling of the remaining data to the AI assistant502(see arrow518), or whether the AI assistant502still needs further training.

As described in detail above, the delegation decision can be made by the human decision maker504or by the AI assessor506(in silent mode). With the former, the AI assessor506provides its assessment to the human decision maker504. See arrow520. The human decision maker504will then make a delegation decision based on the assessment provided by the AI assessor506. However, this communication between the AI assessor506and the human decision maker504is optional. Namely, in the latter case, the AI assessor506itself makes the decision as to whether to delegate the remaining data tasks to the AI assistant502based, for example, on a pre-configured threshold in predicted AI-human agreement. In that case, the human decision maker504is likely unaware of the functions the AI assessor506performs in this decision making process. Namely, the AI assessor506operates in silent mode in the background.

As provided above, one type of decision making task is a data (e.g., image) labeling task. Thus, for illustrative purposes only, an exemplary methodology for hybrid data labeling automation is now described by way of reference toFIG.6andFIG.7. In the context of this data labeling example, the AI assistant is also referred to as a ‘AI label predictor’ and the human decision maker is also referred to as the ‘human labeler.’ Thus, the terms ‘AI assistant’ and ‘human decision maker’ are used herein synonymously with ‘AI label predictor’ and ‘human labeler,’ respectively.

FIG.6provides an exemplary methodology600for hybrid data labeling automation one or more steps of which can be performed by the AI label predictor when coordinating with the human labeler and the AI assessor. In step602, the AI label predictor takes unlabeled input data and provides a set of label predictions for a set of data labeling tasks.

The label predictions are made based on labels the AI label predictor has already received from the human labeler such that the AI label predictor and the human labeler execute an active learning refinement loop. Namely, in step604, the AI label predictor selects a batch of unlabeled data items and presents the selected unlabeled data items to the human labeler. According to an exemplary embodiment, the AI label predictor uses a selection strategy such as low margin active learning (see above) to select the unlabeled data items to present to the human labeler. Namely, the AI label predictor is given the task of selecting the unlabeled data items to incorporate in the active learning refinement loop. As provided above, the AI label predictor can optionally provide its label predictions along with the data presented to the human labeler to assist the human labeler in picking the correct label.

Thus, the goal is to obtain (true) human labels for the data from the human labeler (see step606, described below). In that regard, it may be beneficial to assist the human labeler with this process in order to enhance efficiency. Therefore, optionally, in step604label predictions made by the AI label predictor are also provided along with the data presented to the human labeler. Doing so will assist the human labeler in making the correct decision.

Presenting label predictions made by the AI label predictor to the human labeler in step604(along with the corresponding data item and a confidence value—see above) permits the human labeler to review the performance of the AI label predictor by either agreeing or disagreeing (agreed/not agreed) with the predictions made by the AI label predictor (i.e., AI-to-human agreement). As will be described in conjunction with the description ofFIG.7below, the AI assessor tracks this AI-to-human agreement over time. The agreement tracking by the AI assessor happens in the background during the active learning refinement loop. Thus, when human labels are obtained in step606(described below), in the background, agreement between the human label and prediction is measured and recorded by the AI assessor.

The label predictions are the work of the AI label predictor and the AI assessor needs to have access to them. Thus, the label predictions always need to be shared with/visible to the AI assessor, such that the AI assessor can track agreement between the human label and label prediction in the background. However, as provided above, it is optional whether the label predictions are also shown to the human labeler.

In step606, (true) human labels for the unlabeled data items presented to the human labeler are obtained from the human labeler. These human labels for the data items are then used by the AI label predictor in making further label predictions, and so on. Namely, as shown inFIG.6, the process is repeated iteratively, i.e., in an active learning refinement loop. In step606, the human labeler will provide human labels for each of the data items that have been presented to the human labeler. For those data items presented to the human labeler along with decision predictions made by the AI label predictor (see above), the human labeler will either agree or disagree with the decision prediction from the AI label predictor and, in the latter, change the label. As provided above, any type of semi-supervised learning, self-training/auto-labeling and/or weak supervision process can be employed to train the AI label predictor using the (true) human labels. For example, the tasks with human labels obtained so far can be used as training data to train a supervised machine learning model to make predictions for the remaining tasks (self-training/auto-labeling). According to an exemplary embodiment, a semi-supervised learning process such as label spreading (see above) is employed.

In step608, a decision is made as to whether to delegate performance of the data tasks on the remaining data to the AI label predictor. As will be described in conjunction with the description ofFIG.7below, this delegation decision is supported by an analysis from the AI assessor which has been tracking AI-to-human agreement in the background and will predict performance of the AI label predictor in making future labeling decisions. This prediction serves as a metric for how well the performance will be once the label decision making has been delegated to the AI label predictor. In other words, the predicted performance is an assessment of the readiness of the AI label predictor for automated operation (i.e., the automation readiness) such as auto-labeling.

Namely, once a level of confidence has been met in the label decisions being made by the (trained) AI label predictor, the label decision making capabilities can be fully delegated to the AI label predictor. As above, the decision to delegate the remaining data labeling tasks can reside wholly with the human labeler or, alternatively, the decision to delegate the remaining data labeling tasks to the AI label predictor can be made by the AI assessor (i.e., when the AI assessor operates in a silent mode). When the delegation decision will be made by the human labeler, the AI assessor can provide a recommendation to the human labeler when automation readiness has been achieved. Alternatively, in silent mode, the AI assessor itself can determine when automation readiness has been achieved based, for example, on a pre-configured threshold in predicted AI-human agreement. In either case, however, the delegation decision is supported by the AI assessor which tracks the interactions between the AI label predictor and the human labeler and, from those interactions, assesses the automation readiness.

Based on the presented analysis from the AI assessor (which is provided at the end of each active learning loop iteration), if it is decided in step608that NO automation readiness has not yet been achieved, then delegation to the AI label predictor is declined and training is continued with manual decision in the next iteration of the active learning refinement loop. On the other hand, if it is decided in step608that YES automation readiness has been achieved, then the label decision making capabilities are fully delegated to the AI label predictor and in step610the AI label predictor performs the label decision making on the remainder of the data tasks without receiving verification (agreed/not agreed) from the human labeler.

FIG.7provides an exemplary methodology700for hybrid data labeling automation one or more steps of which can be performed by the AI assessor when observing the interactions between the human labeler and the AI label predictor. In step702, the AI assessor observes the interactions between the human labeler and the AI label predictor. As provided above, the AI label predictor makes label predictions on unlabeled input data based on labels the AI label predictor has already received from the human labeler such that the AI label predictor and the human labeler execute an active learning refinement loop. The AI assessor monitors these AI label predictor-human labeler interactions in step702.

In step704, the AI assessor tracks the AI-to-human agreement over time. Namely, by observing the AI label predictor-human labeler interactions (see step702), the AI assessor can determine how often the (true) human labels from the human decision labeler confirms (or does not confirm) the label predictions from the AI label predictor. Tracking the AI-to-human agreement over time enables the AI assessor to assess and characterize the quality of the AI label predictions.

In step706, based on the tracked agreement data, the AI assessor predicts the AI label predictor-to-human agreement for the remaining unlabeled input data. Doing so permits an assessment to be made as to whether future label predictions by the AI label predictor will be made with a sufficient level of accuracy, or whether the AI label predictor needs further training. That assessment will then support making the delegation decision. According to an exemplary embodiment, the predicted AI label predictor-to-human agreement for the remaining unlabeled input data is made using a performance predictor (such as the performance predictor described in conjunction with the description ofFIG.4above) that uses the tracked agreement data to train an agreement predictor model. Through this training, the agreement predictor will learn under what circumstances (i.e., for which data item characteristics and which label predictions and confidence value characteristics) the AI label predictor is likely to receive an agreement from the human labeler.

Based on the tracking (step704) and predicted performance (step706), in step708the AI assessor makes an assessment of the readiness of the AI label predictor for automated operation (automation readiness) such as auto-labeling. For instance, by way of example only, the assessment made in step708can be based on AI-to-human agreement, namely what percentage of the time the human labeler agreed with the label predictions of the AI label predictor. As highlighted above, the AI label predictor will provide a confidence value along with its label prediction on each of the data items. Thus, another potential metric is certainty, for example, how certain the AI label predictor is about the accuracy of the label predictions it has made over time. Using the performance predictor, the AI label predictor can also determine how accurate it thinks it will be should the AI label predictor begin automated operation in the dataset's current state.

When the delegation decision is made by the human labeler, in step710the AI assessor provides its assessment (from step708) to the human labeler. The human labeler will then make a delegation decision based on the assessment provided by the AI assessor. Namely, in step712, a decision is then made by the human labeler as to whether to delegate performance of the remaining data items to the AI label predictor. This delegation decision is obtained by the AI assessor. If it is decided in step712that NO performance of the remaining data items should not be delegated to the AI label predictor, then the AI assessor continues to observe, track and predict AI-to-human agreement until a decision is obtained from the human labeler to delegate performance of the remaining data items to the AI label predictor. On the other hand, if it is decided in step712that YES performance of the remaining data items should be delegated to the AI label predictor, then in step714the AI assessor (upon obtaining this decision) delegates the remaining data tasks to the AI label predictor.

Alternatively, when the delegation decision is made by the AI assessor, in step716the AI assessor determines whether a pre-configured threshold in predicted AI-human agreement has been reached. If it is determined in step716that YES the pre-configured threshold in predicted AI-human agreement has been reached, then in step718the AI assessor delegates the remaining data tasks to the AI label predictor. On the other hand, if it is determined in step716that NO the pre-configured threshold in predicted AI-human agreement has not been reached, then the AI assessor continues to observe, track and predict AI-to-human agreement until the pre-configured threshold in predicted AI-human agreement has been reached. As noted above, the accuracy is expected to improve over time, as the AI label predictor sees more training data.

As will be described below, one or more elements of the present techniques can optionally be provided as a service in a cloud environment. For instance, by way of example only, the input and/or output data items can reside remotely on a cloud server. Also, the one or more functions of the AI assessor and/or AI assistant/AI label predictor can be performed on a dedicated cloud server to take advantage of high-powered CPUs and GPUs, after which the result is sent back to the local device.

Turning now toFIG.8, a block diagram is shown of an apparatus800for implementing one or more of the methodologies presented herein. By way of example only, apparatus800can be configured to implement one or more of the steps of methodology200ofFIG.2, one or more of the steps of methodology300ofFIG.3, one or more of the steps of methodology400ofFIG.4, one or more of the steps of methodology600ofFIG.6and/or one or more of the steps of methodology700ofFIG.7. For instance, according to an exemplary embodiment, the AI assistant/AI label predictor and/or the AI assessor are implemented in apparatus800.

Apparatus800includes a computer system810and removable media850. Computer system810includes a processor device820, a network interface825, a memory830, a media interface835and an optional display840. Network interface825allows computer system810to connect to a network, while media interface835allows computer system810to interact with media, such as a hard drive or removable media850.

Processor device820can be configured to implement the methods, steps, and functions disclosed herein. The memory830could be distributed or local and the processor device820could be distributed or singular. The memory830could be implemented as an electrical, magnetic or optical memory, or any combination of these or other types of storage devices. Moreover, the term “memory” should be construed broadly enough to encompass any information able to be read from, or written to, an address in the addressable space accessed by processor device820. With this definition, information on a network, accessible through network interface825, is still within memory830because the processor device820can retrieve the information from the network. It should be noted that each distributed processor that makes up processor device820generally contains its own addressable memory space. It should also be noted that some or all of computer system810can be incorporated into an application-specific or general-use integrated circuit.

Optional display840is any type of display suitable for interacting with a human user of apparatus800. Generally, display840is a computer monitor or other similar display.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows: