Patent ID: 12230248

DETAILED DESCRIPTION

Described herein are systems and methods for performing intent classification using a hierarchy of models. The hierarchical structure does not suffer from the same performance degradation experienced by a single intent classification model as the number of intents to classify increases. Further, the hierarchical structure can be automatically generated using an unsupervised learning algorithm, thus eliminating cost associated with manual efforts (e.g., by linguists and data scientists) to formulate the necessary ontology.

The following description sets forth exemplary methods, parameters, and the like. It should be recognized, however, that such description is not intended as a limitation on the scope of the present disclosure but is instead provided as a description of exemplary embodiments.

Although the following description uses terms “first,” “second,” etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first graphical representation could be termed a second graphical representation, and, similarly, a second graphical representation could be termed a first graphical representation, without departing from the scope of the various described embodiments. The first graphical representation and the second graphical representation are both graphical representations, but they are not the same graphical representation.

The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “comprises,” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The term “if” is, optionally, construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event],” depending on the context.

FIG.1Aillustrates a single-model approach to intent classification. With reference toFIG.1A, a single intent classification model114is configured to receive a user utterance and identify an intent associated with the user utterance. In the depicted example, the model114can classify m number of pre-defined intents. Specifically, the model provides m number of outputs corresponding to the m number of pre-defined intents, respectively. Each output can be a probabilistic value indicating the likelihood that the user utterance is associated with the respective intent, and the intent having the highest probabilistic value is selected as the final intent. It should be appreciated that the intent classification model114can be configured to provide other formats of outputs to indicate the identified intent.

The single-model approach to intent classification can suffer from performance degradation as the number of intents, m, increases. For example, when m increases, some intents may start to overlap. As such, additional datasets (e.g., with more specificity) may be needed to train the model114to classify an increased number of intents. However, the additional datasets may introduce more overlapping intents, further exacerbating the problem.

The hierarchical approach to intent classification can improve the performance of any underlying model, as the overall probability for the classification improves by limiting the number of classes required for classification at each stage and thus reducing the chances of intent overlap as additional datasets are included in the application. This is illustrated inFIG.1B, as described below.

With reference toFIG.1B, a hierarchy of multiple models100is used to classify a user utterance102with respect to m number of intents. The hierarchy100comprises two layers—the first layer comprises an intent cluster classification model104and the second layer comprises a plurality of intent classification models106a-n. The intent cluster classification model104is configured to receive a user utterance and identify an intent cluster associated with the user utterance. In the depicted example, there are n intent clusters1-n. Intent classification models106a-nare associated with the intent clusters1-n, respectively. Each intent classification model is configured to receive a user utterance and identify an intent associated with the user utterance within the respective intent cluster.

In operation, the system receives a user utterance102(e.g., via a conversational bot interface). The system invokes intent cluster classification model104and provides the user utterance102to the model104to identify an intent cluster associated with the user utterance102. Based on the identified user cluster, the system invokes the intent classification model associated with the identified user cluster to identify the intent. For example, the intent cluster classification model104can determine that the user utterance102is associated with Cluster n. Accordingly, the system invokes the intent classification model106n. The intent classification model106ncan receive the user utterance102and identify an intent within intent cluster n associated with the user utterance.

The system does not invoke the intent classification models not associated with the identified intent cluster. In the example above, the system only invokes the intent classification model106n, not the rest of the intent classification models in the second layer of the hierarchy.

The hierarchy of models100results in a number of technical improvements. First, for a given user utterance, the hierarchy of models100analyzes a given user utterance with respect to only a subset of intents1-m(i.e., only the intents in the identified intent cluster and not the intents in the other clusters). The reduction in classes for classification at each layer enables better training (and better performance) for each model in the hierarchy while reducing the chances of intent overlap. Thus, the overall performance for the classification improves. In other words, the performance of the model hierarchy100can better scale as the number of intents increases.

Further, the hierarchy of models100is automatically generated by an unsupervised learning algorithm, as described further with reference toFIG.2A. As described above, manually setting up a hierarchical structure can be very cumbersome, expensive, and potentially inaccurate especially when there is a limited amount of empirical data to guide the process (e.g., only a list of intents and the corresponding training utterances). In contrast, the hierarchy structure (e.g., the intent clusters) is automatically generated with limited dataset (normally available from conversational logs or other means through which users generate queries) and minimal linguist involvement.

FIG.2Aillustrates an exemplary workflow for creating clusters of intents to support the hierarchical approach, in accordance with some embodiments. Process200is performed, for example, using one or more electronic devices implementing a software platform. In some examples, process200is performed using a client-server system, and the blocks of process200are divided up in any manner between the server and a client device. In some embodiments, process200is performed using only a client device or only multiple client devices. In process200, some blocks are, optionally, combined, the order of some blocks is, optionally, changed, and some blocks are, optionally, omitted. In some examples, additional steps may be performed in combination with the process200. Accordingly, the operations as illustrated (and described in greater detail below) are exemplary by nature and, as such, should not be viewed as limiting.

At block204, an exemplary system (e.g., one or more electronic devices) generates first training data based on first utterance data202. The first utterance data202can be collected from conversation logs. In some embodiments, a minimal set of utterance can be generated, collected, and/or expanded upon through a user survey, through outsourced solutions (e.g., contractors), and/or through experts (e.g., linguists). In the first utterance data202, the utterances are associated with pre-defined intents. For example, the first utterance data202may include an utterance “What is my HSA balance” and an utterance “have I reached my HAS maximum” associated with the intent “hsa_balance,” an utterance “my 401k balance” and an utterance “what's my 401 maximum” associated with the intent “401k_balance,” an utterance “what's my capital balance” associated with intent “capital_balance.”

In some embodiments, the system performs embedding of the first utterance data. For example, the system can use either a sentence embedding, word embedding, or a combination of both to encode the first utterance data into embedding vectors. Pre-trained sentence and word embedding models can be employed to perform the embedding, thereby alleviating the need of a large training dataset.

For example, the system provides the first utterance data to a natural-language processing embedding model that is configured to create a vector in an n-dimensional space corresponding to each utterance. This embedding generator model can be pre-trained based on the desired language (e.g., English) and creates an n-dimensional vector for utterances based on semantic similarity. For example, if the utterances are specific to particular domain(s) such as medical science, the embedding generator model can be pre-trained on medical journal data to be able to derive semantic similarity among the utterances based on medical ontology.

At block209, the system identifies, based on the first training data, a first plurality of intent clusters1-N using an unsupervised machine-learning algorithm. For example, the embedded vectors can be used to create clusters of the dataset using the cosine similarity to find the distance between vectors in n-dimensions and cluster them, or using k-nearest neighbor, k-means, BIRCH algorithm, etc., for unsupervised clustering of the data. Using the unsupervised clustering approach eases any annotation requirements. Because each utterance is associated with a pre-defined intent, the clustering of utterances essentially creates clusters of intents. In some embodiments, the system can be further optimized to pre-define the number of clusters to be created based on the total number of intents and to achieve a nearly even distribution of intents within the clusters.

For example, the first utterance data202may include an utterance “What is my HSA balance” and an utterance “have I reached my HAS maximum” associated with the intent “hsa_balance,” an utterance “my 401k balance” and an intent “what's my 401 maximum” associated with the intent “401k_balance,” an utterance “what's my capital balance” associated with intent “capital_balance.” After clustering, all utterances (and thus associated intents) related to the capital balance can be in one cluster, all utterances (and thus associated intents) related to the HSA balance can be in another cluster, and all utterances (and thus associated intents) related to 401k can be in yet another cluster.

If an intent is defined by multiple utterances and all of the utterances (or the corresponding embeddings) are clustered into one particular intent cluster, then the intent is deemed to belong to that particular intent cluster. However, if the utterances associated with an intent are clustered into multiple different intent clusters, the system can automatically determine which intent cluster that the intent belongs to. In some embodiments, the system selects the highest frequency cluster (e.g., the cluster having the maximum number of utterances corresponding to the intent). This can occur because there can be multiple utterances that a user may use to invoke an intent and the utterances may be semantically different.

For example, an intent “insights_me_at_work” may be defined by utterances: “view my work profile,” “tell me about myself,” “I need my personal details,” “show me my profile details,” “display my financial summary,” “vacation time,” “about myself,” etc. These utterances may be grouped into different intent clusters. Thus, the system can select one intent cluster based on the distribution of the utterances among the various clusters.

At block212, the system trains an intent cluster classification model216. The intent classification model216is configured to receive a user utterance and identify an intent cluster from the intent clusters1-N. The intent classification model is the first layer in the hierarchy of models, as illustrated by intent cluster classification model104inFIG.1B. For example, the intent cluster classification model can be trained to receive a user utterance and determine if the user utterance is related to the HAS balance cluster, the 401k balance cluster, or the capital balance cluster.

At block214, the system trains an intent classification model for each intent cluster of clusters1-N. Thus, the system obtains N number of intent classification models. Each intent classification model is configured to receive a user utterance and identify an intent within the corresponding intent cluster. For example, intent classification model N is configured to receive a user utterance and identify an intent from the intents in the intent cluster N. The intent classification models218form the second layer in the hierarchy of models, as illustrated by intent classification models106a-ninFIG.1B.

For example, three separate intent classification models can be trained for the capital balance cluster, the HSA balance cluster, and the 401k balance cluster, respectively. When deployed, the intent cluster classification model may first be used to determine that the inquiry is related to the 401k balance cluster and forward the user inquiry to the 401k intent classification model, which in turn is specifically trained to distinguish between different 401k intents. In some embodiments, additional utterances (to the first utterance data202) can be generated to train the intent classification models. For example, additional utterances related to 401k inquiries can be generated to train the 401k intent classification model.

The intent classification models1-N are trained using different training datasets. For example, intent classification model1is trained using training utterances corresponding to intents within intent cluster1, while intent classification model N is trained using training utterances corresponding to intents within intent cluster N. In some embodiments, the training utterances for each intent classification model includes some utterances from the first utterance data202and/or additional utterances corresponding to intents within the respective cluster (e.g., formulated by a linguist).

In some embodiments, the intent classification models can comprise different types of models.

FIG.2Billustrates an exemplary workflow for updating clusters of intents to support the hierarchical approach, in accordance with some embodiments. Process250is performed, for example, using one or more electronic devices implementing a software platform. In some examples, process250is performed using a client-server system, and the blocks of process250are divided up in any manner between the server and a client device. In some embodiments, process250is performed using only a client device or only multiple client devices. In process250, some blocks are, optionally, combined, the order of some blocks is, optionally, changed, and some blocks are, optionally, omitted. In some examples, additional steps may be performed in combination with the process250. Accordingly, the operations as illustrated (and described in greater detail below) are exemplary by nature and, as such, should not be viewed as limiting.

With reference toFIG.2B, the system receives second utterance data. The second utterance data can correspond to new intents to be included in the system's NLU capabilities. For example, if the system comprises a conversational bot and its NLU capabilities are to be extended to include new intents, the second utterance may include exemplary utterances corresponding to these new intents. For example, the second utterance data can include “check my company's 401k benefits” and “what's my IRA balance.”

At block254, the system generates second training data based on the second utterance data252. In some embodiments, the system performs embedding of the second utterance data. For example, the system can use either a sentence embedding, word embedding, or a combination of both to encode the second utterance data into embedding vectors. Pre-trained sentence and word embedding models can be employed to perform the embedding, thereby alleviating the need of a large training dataset.

At block258, the system uses an unsupervised machine-learning algorithm to identify a second plurality of intent clusters based on the first training data and the second training data. In other words, the system combines the first and second training data and performs re-clustering. As a result, new clusters may be identified due to the addition of the second training data. For example, the system may identify a new intent cluster, the IRA cluster, corresponding to the intent associated with the new utterance “what's my IRA balance.” Further, a new intent associated with the second training data may be clustered into an existing intent cluster of the first plurality of clusters. For example, the system may determine that the intent associated with the new utterance “check my company's 401k benefits” should be clustered into the existing 401k cluster.

At block252, the system retrains the intent cluster classification model216based on the first and the second training data. The intent cluster classification model is configured to receive a user utterance and identify an intent cluster of the second plurality of intent clusters.

At block262, if a new intent cluster is identified in the second plurality of intent clusters that is not in the first plurality of intent clusters, the system trains a new intent classification model (e.g., model O in218) corresponding to the new intent cluster. For example, with reference toFIG.1C, if a new intent cluster o is identified, a new intent classification model106ois trained. The new intent classification model is trained based on only the training data corresponding to the intents within the new intent cluster. For example, the system may train a new intent classification model to process IRA inquiries.

At block264, if a new intent in the second utterance data belongs to an existing intent cluster of the first plurality of clusters, the system retrains the existing intent classification model. For example, with reference toFIG.1C, if a new intent in the second utterance data belongs to the intent cluster N, the intent classification model106nis retrained such that it can classify the new intent. For example, the system may retrain the 401k intent classification model such that it can process the new intent “check my company's 401k benefits.”

Accordingly, when new intents are added to the NLU capability of the system, not all intent classification models1061-nneed to be retrained. Further, re-clustering is performed automatically. This eliminates the significant cost required by prior art to for manually define new ontology to incorporate the new intents into an existing system.

The same model(s) can be retrained for the intent hierarchy to achieve a performance improvement without the need to rework on the underlying algorithms. The models at each layer of hierarchical NLU can now be trained with additional dataset, as needed, without degrading the performance due to intent overlaps, which is a common occurrence when a single model is trained to classify within a large number of intents. In one exemplary implementation, ˜10% performance improvement over the single layer architecture is observed.

FIG.3Aillustrates an exemplary process for creating clusters of intents to support the hierarchical approach, in accordance with some embodiments. Process300is performed, for example, using one or more electronic devices implementing a software platform. In some examples, process300is performed using a client-server system, and the blocks of process300are divided up in any manner between the server and a client device. In some embodiments, process300is performed using only a client device or only multiple client devices. In process300, some blocks are, optionally, combined, the order of some blocks is, optionally, changed, and some blocks are, optionally, omitted. In some examples, additional steps may be performed in combination with the process300. Accordingly, the operations as illustrated (and described in greater detail below) are exemplary by nature and, as such, should not be viewed as limiting.

At block302, the system (e.g., one or more electronic devices) receives first utterance data corresponding to a first plurality of intents; at block304, the system identifies, based on the first utterance data, a first plurality of intent clusters using an unsupervised machine-learning algorithm, wherein each intent cluster of the first plurality of intent clusters comprises a respective subset of the first plurality of intents; at block306, the system trains, based on the first utterance data, an intent cluster classification model, wherein the intent cluster classification model is configured to receive a user utterance and identify an intent cluster of the first plurality of intent clusters; and at block308, the system trains, based on the first utterance data, an intent classification model for each intent cluster of the first plurality of intent clusters to obtain a plurality of intent classification models, wherein each intent classification model is configured to receive the user utterance and identify an intent from the respective intent cluster.

FIG.3Billustrates an exemplary process for performing natural-language understanding, in accordance with some embodiments. Process350is performed, for example, using one or more electronic devices implementing a software platform. In some examples, process350is performed using a client-server system, and the blocks of process350are divided up in any manner between the server and a client device. In some embodiments, process350is performed using only a client device or only multiple client devices. In process350, some blocks are, optionally, combined, the order of some blocks is, optionally, changed, and some blocks are, optionally, omitted. In some examples, additional steps may be performed in combination with the process350. Accordingly, the operations as illustrated (and described in greater detail below) are exemplary by nature and, as such, should not be viewed as limiting.

At block352, the system (e.g., one or more electronic devices) receives a user utterance; at block354, the system inputs the user utterance to an intent cluster classification model to obtain a pre-defined intent cluster of a plurality of pre-defined intent clusters, wherein the plurality of pre-defined intent clusters are automatically identified using an unsupervised machine-learning algorithm; at block356, the system inputs the user utterance to an intent classification model corresponding to the obtained pre-defined intent cluster to obtain an intent associated with the user utterance.

The operations described above with reference toFIGS.3A and3Bare optionally implemented by components depicted inFIG.4.

FIG.4illustrates an example of a computing device in accordance with one embodiment. Device400can be a host computer connected to a network. Device400can be a client computer or a server. As shown inFIG.4, device400can be any suitable type of microprocessor-based device, such as a personal computer, workstation, server or handheld computing device (portable electronic device) such as a phone or tablet. The device can include, for example, one or more of processor410, input device420, output device430, storage440, and communication device460. Input device420and output device430can generally correspond to those described above, and can either be connectable or integrated with the computer.

Input device420can be any suitable device that provides input, such as a touch screen, keyboard or keypad, mouse, or voice-recognition device. Output device430can be any suitable device that provides output, such as a touch screen, haptics device, or speaker.

Storage440can be any suitable device that provides storage, such as an electrical, magnetic or optical memory including a RAM, cache, hard drive, or removable storage disk. Communication device460can include any suitable device capable of transmitting and receiving signals over a network, such as a network interface chip or device. The components of the computer can be connected in any suitable manner, such as via a physical bus or wirelessly.

Software450, which can be stored in storage440and executed by processor410, can include, for example, the programming that embodies the functionality of the present disclosure (e.g., as embodied in the devices as described above).

Software450can also be stored and/or transported within any non-transitory computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch instructions associated with the software from the instruction execution system, apparatus, or device and execute the instructions. In the context of this disclosure, a computer-readable storage medium can be any medium, such as storage440, that can contain or store programming for use by or in connection with an instruction execution system, apparatus, or device.

Software450can also be propagated within any transport medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch instructions associated with the software from the instruction execution system, apparatus, or device and execute the instructions. In the context of this disclosure, a transport medium can be any medium that can communicate, propagate or transport programming for use by or in connection with an instruction execution system, apparatus, or device. The transport readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic or infrared wired or wireless propagation medium.

Device400may be connected to a network, which can be any suitable type of interconnected communication system. The network can implement any suitable communications protocol and can be secured by any suitable security protocol. The network can comprise network links of any suitable arrangement that can implement the transmission and reception of network signals, such as wireless network connections, T1 or T3 lines, cable networks, DSL, or telephone lines.

Device400can implement any operating system suitable for operating on the network. Software450can be written in any suitable programming language, such as C, C++, Java or Python. In various embodiments, application software embodying the functionality of the present disclosure can be deployed in different configurations, such as in a client/server arrangement or through a Web browser as a Web-based application or Web service, for example.

Although the disclosure and examples have been fully described with reference to the accompanying figures, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the disclosure and examples as defined by the claims.

The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the techniques and their practical applications. Others skilled in the art are thereby enabled to best utilize the techniques and various embodiments with various modifications as are suited to the particular use contemplated.