Patent ID: 12229663

DETAILED DESCRIPTION OF THE DISCLOSURE

A method for determining intents associated with human utterances at a voice response system is provided. The method may involve a machine learning system and a deep learning system.

The method may include receiving a predetermined number of labeled training utterances at a training module of the machine learning system. Each labeled training utterance may include an utterance and an intent. The intent may be included in a plurality of intents.

The method may include generating a plurality of sub-models that correspond to the plurality of intents. A feature engineering and extraction module at the training module of the machine learning system may generate the plurality of sub-models that correspond to the plurality of intents.

The method may include receiving a plurality of live unlabeled utterances at an execution module in the machine learning subsystem. Each live utterance may be transmitted by an entity.

The method may include identifying, at the execution module, a sub-model that corresponds to each unlabeled utterance.

The method may include, for each live utterance, presenting a series of steps associated with the intent, to the entity that transmitted the live utterance.

The method may include identifying, over a predetermined confident threshold, whether each identified intent was accurately assigned or inaccurately assigned. The identifying may be based on a plurality of series received during and after the presenting the series of steps.

The method may include training an artificial neural network of the deep learning system using the received live utterances and the associated intents. The method may include utilizing an active learning module to determine which intents should be transmitted to the artificial neural network.

The method may include receiving an unlabeled utterance directly at the deep learning system. The method may include accurately determining an intent for the unlabeled utterance.

The method may include inputting a new intent that is not included in the plurality of intents directly into the deep learning system by inputting a plurality of labeled intents, each of which corresponds to, and forms, the new intent.

The system described above may enable use of a deep learning system. Operation of the system obviates the need for the conventional, manually-intensive task of generating large amounts of training data required for operation of the deep learning system. It should be appreciated that experiments using this system produced unexpected, and surprisingly advantageous, results.

One experiment included inputting 100% accurately labeled training data into the machine learning classifier in order to train the machine learning classifier. The trained machine learning classifier, when used on unlabeled production data, labeled the production data with an accuracy rate of 81%. The production data that was labeled with an accuracy rate of 81% was inputted as labeled training data for a deep learning classifier. The deep learning classifier was then trained using the 81% accurately labeled production data.

The trained deep learning classifier, when used on unlabeled production data, labeled the production data with an accuracy rate of 79.5%. Therefore, the input to output accuracy ratio of the deep learning classifier was 17.5% better than the machine learning classifier. Furthermore, the machine learning classifier used in the experiment was a 2-3-year-old classifier, which means that it already achieved 2-3 years of learned behavior. The deep learning classifier used in the experiment was a newborn classifier. As a result of this experiment, it may be understood that this system produced unexpectedly advantageous results—i.e., that the deep learning classifier achieved a 17.5% input to output accuracy ratio increase.

Apparatus and methods described herein are illustrative. Apparatus and methods in accordance with this disclosure will now be described in connection with the figures, which form a part hereof. The figures show illustrative features of apparatus and method steps in accordance with the principles of this disclosure. It is to be understood that other embodiments may be utilized and that structural, functional and procedural modifications may be made without departing from the scope and spirit of the present disclosure.

The steps of methods may be performed in an order other than the order shown or described herein. Embodiments may omit steps shown or described in connection with illustrative methods. Embodiments may include steps that are neither shown nor described in connection with illustrative methods.

Illustrative method steps may be combined. For example, an illustrative method may include steps shown in connection with another illustrative method.

Apparatus may omit features shown or described in connection with illustrative apparatus. Embodiments may include features that are neither shown nor described in connection with the illustrative apparatus. Features of illustrative apparatus may be combined. For example, an illustrative embodiment may include features shown in connection with another illustrative embodiment.

FIG.1shows an illustrative neuron. The illustrative neuron includes input features102,104and106. Input102corresponds to X0. Input104corresponds to Xi. Input106corresponds to XM. Input features102,104and106may be features retrieved from an utterance.

X0may be assigned a weighting factor of W0. Ximay be assigned a weighting factor of Wi. XMmay be assigned a weighting factor of WM. The weighting factor may be manually set. The weighting factor may be automatically generated. The weighting factor may be tuned after the neuron's processing.

Inputs102,104and106may be values assigned to features. Preferably, the values may be between 0 and 1. Intent prediction108may be based on the output of the neuron. The output may be a sum of the values and the weights assigned to those values. The summation algorithm shown is:

∑i=0M⁢Wi⁢Xi=W·X

The summation algorithm shown above can be explained as follows: Starting with the input feature labeled 0 until the input feature labeled M: multiply the input value by the associated weight and add the computed value to the previous value. It should be appreciated that the above-described summation algorithm is equivalent to the dot product of W and X, where W and X are vectors whose components are weights and inputs, respectively.

The output of the summation algorithm and/or dot product algorithm is compared to a threshold. If the output of the summation algorithm, or the dot product algorithm, is greater than the threshold, the intent prediction may be set to one. Setting the intent prediction to one may be understood to mean that the identified intent is appropriate for the inputted utterance. If the output of the summation algorithm is less than the threshold, the intent prediction may be set to zero. Setting the intent prediction to zero may be understood to mean that the identified intent is appropriate for the inputted utterance. The threshold may be manually set, or computer-generated.

Another way to introduce the threshold into the algorithm is by using a bias. The bias is equivalent to the negative value of the threshold. The bias is then added to the dot product. If the value of the dot product plus the bias is less than or equal to zero, the intent prediction is set to zero. If the value of the dot product plus the bias is greater than zero, the intent prediction is set to one.

The bias and threshold algorithms described above produce a binary output, either a zero or a one. In order to utilize inputs between 0 and 1 and produce outputs between 0 and 1, a sigmoid neuron may be used. The output of a sigmoid neuron with inputs x1, x2, weights w1, w2, and bias b is as follows:

11+exp⁡(-Σj⁢Wj⁢Xj-b)

The output of the sigmoid neuron is a value between zero and one. If the value of the dot product of W and X plus b is a large and positive number, then the output of the sigmoid neuron is close to one. If the value of the dot product of W and X plus b is a large negative number, then the output of the sigmoid neuron is close to zero. When the value of the dot product of W and X plus b is a modest value, then the output ranges somewhere in between zero and one.

FIG.2shows an artificial neural network. The artificial neural network includes an input layer, shown at202, hidden layer1, shown at204, hidden layer2, shown at206and an output layer shown at208.

Input layer202may include input neurons210,212and214. Each of input neurons210,212and214may be similar to X0, Xiand XM, shown at102,104and106ofFIG.1.

The output of each of210,212and214may be input into each neuron included in hidden layer1. Because there were no manipulations or calculations executed up until this point on the input neurons, the output of input neurons210,212and214may be the same as the inputs to input neurons210,212and214. Hidden layer1may include neurons216,218,220and222. The output-input lines between input layer, shown at202, and hidden layer1, shown at204, may have weights associated with each of them. Therefore, neurons216,218,220and222, included in hidden layer1, may be inputted the value of each input neuron and the weight associated with each input neuron. It should be appreciated that each output-input line, even from the same input neuron, may be assigned a different weight. It should also be appreciated that the term ‘hidden layer’ may be defined as a layer, in a neural network, that is not an input layer nor an output layer.

The output of neurons216,218,220and222may be input, with its own assigned weight, into neurons224,226,228and230included in hidden layer 3. The output of neurons224,226,228and230, with its own assigned weight may be input into neuron232, included in output layer208.

It should be noted that each layer within a neural network may build upon the previous layers. For example, in a facial recognition application, initial layers may define pixels, secondary layers may define lines and curves, tertiary layers may define facial features and final layers may compare the facial features to previously recognized faces. The neural network may be explained as a hierarchical application which goes from pixels to lines and curves to facial features to a face to facial identification.

FIG.3shows an illustrative diagram. The diagram shows some differences between machine learning and deep learning. Although, deep learning is considered a subset of machine learning, the illustrative diagram defines machine learning as at least partially manually supervised. The illustrative diagram defines deep learning as a layered structure of artificial neurons with minimal or no human interaction.

Input302may be an utterance. The utterance may be What is my account balance? Input302may be labeled with the account balance intent. Input302may be inputted into feature extraction module304. Feature extraction module304may be, at least partially, manually supervised. Feature extraction module304may extract various features from the labeled input. For example, feature extraction module304may determine that an utterance includes one or more words, or groupings of words, which may be called n-grams, relating to an account balance intent. N-grams for an account balance intent may include What is my, is my account, my account balance, what is, is my, my account, account balance, what, is, my, account and balance.

Machine learning classification module306may classify an inputted non-labeled input based on features extracted from the non-labeled input. The features extracted from the non-labeled input are compared to the previously determined account balance intent features, such as the n-grams for the account balance intent. The classification module makes a decision whether the non-labeled input is either associated with the account balance intent or is not associated with the account balance intent. The decision is outputted via output module308.

It should be appreciated that a machine learning system may learn during execution. However, machine learning systems typically require manual tuning during the learning process.

Input310may be inputted to deep learning feature extraction and classification module312. Feature extraction and classification module312may utilize an artificial neural network trained by large amounts of labeled training data. The training data includes many utterances that are either labeled as being associated with the account balance intent or not being associated with the account balance intent. The feature extraction and classification module determines, absent human intervention, which features are important when determining whether an utterance is or is not associated with the account balance intent.

An unlabeled utterance inputted into feature extraction and classification module312is classified as either associated with the account balance intent or not associated with the account balance intent. The classification is outputted via output module314. The output may be combined with various other outputs associated with other intents in order to determine the appropriate intent.

FIG.4shows an illustrative diagram. The illustrative diagram includes graph400. Graph400shows a comparison between deep learning classifiers (406) and machine learning classifiers (408).

The x-axis of graph400is the amount of data, shown at402. The y-axis of graph400is the performance of the system, shown at404. The performance of the system may be the accuracy of the system.

As shown in graph400, the performance of machine learning classifiers plateau at a certain point with respect to the amount of training data received. However, the performance of deep learning classifiers keeps on climbing with larger amounts of training data.

FIG.5shows another illustrative diagram. The diagram shows the rise in the popularity of deep learning, as shown at502. The x-axis of the graph shows years, as shown at506. The y-axis of the graph shows a popularity variable. Line508shows an increase in the popularity of deep learning classifiers from approximately the year 2010 until approximately the year 2016.

FIG.6shows an illustrative diagram. Production data602may include millions of utterances and intents retrieved from a production environment. The calculated accuracy of the intents properly corresponding to the utterances is greater than 75%. Active learning can be applied to the production data, as shown at604. Active learning on the production data may raise the accuracy of the intents properly corresponding to the utterances to greater than 85%. Active learning may include weeding out utterances and corresponding intents that are determined to be below a predetermined confidence threshold. The confidence threshold may be a score of how confident the system is that the intent corresponds to the utterance. The confidence threshold may be based on signals, received from an entity that transmitted the utterance, upon transmission of the determined intent.

The following exemplary scenario may correspond to an utterance and determined intent that may be removed during the active learning process: An entity transmitted the utterance, What is my account balance?, the system determined the intent of the utterance to be a routing number intent, the system provided the entity with the routing number intent and the entity abandoned the steps necessary to retrieve the routing number.

The following exemplary scenario may correspond to an utterance and determined intent that may be retained during the active learning process: An entity transmitted utterance, What is may routing number?, the system determined the intent of the utterance to be a routing number intent, the system provided the entity with the routing intent number intent and the entity completed the steps necessary to retrieve the routing number.

The amount of production data that was retrieved via the active learning module may be between 500,000 and one million utterances. The production data retrieved via the active learning module may be inputted into deep learning model620.

Manually labeled data, shown at614, which has an accuracy of approximately 99% and a count of approximately 20,000 may also be inputted into deep learning model620. Public models616may also be inputted into deep learning model620.

Deep learning model620may utilize the labeled input to customize its artificial neural network. Deep learning model may be able to receive an unlabeled training data element and produce intent622.

There may be benefits associated with deep learning models trained by machine learning systems. The benefits may include learning from production data, shown at612. It should be appreciated that learning from production data may achieve a higher level of accuracy than learning from training data. Production data may be more accurate because it is received from a more diverse body of entities, when compared to training data that is generated by a select group of technical entity employees.

Another benefit associated with deep learning models trained by machine learning systems may be reducing manual tuning effort associated with machine learning systems, as shown at610.

Another benefit associated with deep learning models trained by machine learning systems may be the ability of the deep learning system to scale to more intents and data, as shown at608. New intents may be incorporated into the deep learning system by transmitting, to the deep learning system, labeled training data associated with the new intent.

Another benefit associated with deep learning models trained by machine learning systems may be the ability to reduce word sensitivity, as shown at606. Word sensitivity may be when a classifier thinks that a word is very important with respect to a specific intent. For example, if there are 500 intents and only three intents have the word please, the classifier may think that the word please is very important to these three intents. Therefore, if an utterance includes the word please it will weigh heavily towards the three specified intents. When the system is trained by production data instead of manually created intents, there is a wider range of inputs, and therefore, the word please will probably appear similarly across all intents. Therefore, inaccurate word sensitivity will be reduced.

FIG.7shows an illustrative flow chart. Flow chart700shows training data preparation. Interaction history702may be a database of labeled production data.

Active learning signal analysis704may be a process to determine whether the entity that transmitted the utterance was satisfied with the intent received. Active learning may include sentiment analysis. Sentiment analysis may include determining the reaction of the entity. An example of sentiment analysis that may lean towards not accurate may be if a transfer intent usually takes five steps to complete and the entity abandons the intent after two steps. Another example of sentiment analysis that may lean towards accurate may be if the entity completes the steps associated with the intent. Another example of sentiment analysis may be good or bad feedback from the entity.

One or more of the signals included in the sentiment analysis may be combined into a single active learning model. The active learning model may predict whether an intent was correctly or incorrectly associated with an utterance.

The active learning system may remove as much as 50% of the production data. However, because there is such a large amount of production data being produced, the system may be trained with 50% of the production data.

Database706shows the data that has been previously labeled in the production environment. The data included in database706may not include data that has been removed during the active learning process.

Step708shows that there may be an optional manual review of the data included in database706. The data included in auto labeled data706may be transmitted, via step708, or straight to join step712. Join step712may also include tuning request710. Tuning request710may include request to tune intents or tune training data. Manual training data716may also be inputted into the join step712. The join step712may produce combined training data database714.

FIG.8shows illustrative schematic800. Schematic chart800shows a model that gets better incrementally over time. Training data database714may be inputted into N-1 training set, shown at802.

Schematic800shows a high-level overview of the systems and methods for degradation-resistant tuning of an ML model. The tuning process shown in schematic800may include other elements and/or steps not shown in the schematic. The tuning process shown in schematic800shows one embodiment. Other embodiments may include different elements and/or steps.

The tuning process shown in schematic800may be an iterative process. Schematic800may show one iteration, beginning with the current (N-1) Training set802. Current Training set802may be used to train the current (N-1) ML model804. A set of inoperative utterances may be determined via a received tuning request806, or any other suitable determination. A test set810of utterances may be tested via an automated regression tool (“ART”)812. Test run reports814may indicate the overall accuracy of the system. At step818, the current training set808may be modified and tested along with confusion matrix816. If the test results show improved accuracy across the system, the modifications may be deployed into the next iteration of the training set, N (820).

FIG.9shows an illustrative diagram. Deep learning prediction classifier904may be trained using various labeled training data elements.

Deep learning prediction classifier may receive an unlabeled data element via pipe902. Deep learning prediction classifier904may determine and label an intent for the received data element. The data element may be transmitted out the pipe that corresponds to the label of the intent. The intent may be intent1, shows at906, intent2, shows at908or any other suitable intent, not shown.

The system may plot the newly labeled utterance with respect each intent. The label may be a confidence label, as shown at912. The pipe, shown at910(which may also be included for pipes906and908, however not shown) may be an output neuron of an artificial neural network associated with deep learning prediction system904. The deep learning prediction system904may determine a value for the correspondence between the intent and the utterance. When system904is more confident that the intent matches the utterance, the determined value may be closer to 1. When system904is less confident that the intent matches the utterance, the determined value may be closer to 0. The determined value of how confident the deep learning system is sure that the intent matches the utterances is where the newly labeled utterance is plotted on graph914. The output of a neuron, as explained above, is typically between 0 and 1. Therefore, an output close to 1 may indicate greater confidence than an output close to 0. In this scenario, the confidence level may be determined to be between 0 and 1.

Thus, a deep learning system is provided. Persons skilled in the art will appreciate that the present invention can be practiced by other than the described embodiments, which are presented for purposes of illustration rather than of limitation. The present invention is limited only by the claims that follow.