Patent Description:
Dialog systems that communicate with users using natural language phrasing utilize language understanding models to discern the intent of a user so that a proper response can be given, or proper actions can be taken. Often times, the user will provide short utterances to the dialog system, which are passed along to the language understanding model for interpretation. Short utterances are often highly ambiguous and the discovery of the intent of an utterance is difficult.

It is within this context that the present embodiments arise. <CIT> relates to a spoken dialogue system that includes a spoken language understanding apparatus. The spoken language understanding apparatus can include an intent apparatus and a selection apparatus. The intent apparatus is configured to determine if a query comprises a global command, to determine if an intent associated with a query is or is not included in a domain that is supported by the spoken dialogue system, to determine if a query comprises a confirmation type, to tag one or more entities in a query, and to determine an intent probability distribution and a domain probability distribution that is associated with a query. When the query includes an entity that is included in two or more possible entities, the selection apparatus is configured to provide a score for each of the two or more possible entities. <CIT> relates to systems and methods that are provided for improving language models for speech recognition by adapting knowledge sources utilized by the language models to session contexts. A knowledge source, such as a knowledge graph, is used to capture and model dynamic session context based on user interaction information from usage history, such as session logs, that is mapped to the knowledge source. From sequences of user interactions, higher level intent sequences may be determined and used to form models that anticipate similar intents but with different arguments including arguments that do not necessarily appear in the usage history. In this way, the session context models may be used to determine likely next interactions or "turns" from a user, given a previous turn or turns. Language models corresponding to the likely next turns are then interpolated and provided to improve recognition accuracy of the next turn received from the user.

It is the object of the present invention to provide an improved method, system and computer readable media for language understanding.

The description that follows includes illustrative systems, methods, user interfaces, techniques, instruction sequences, and computing machine program products that exemplify illustrative embodiments. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the inventive subject matter. In general, well-known instruction instances, protocols, structures, and techniques have not been shown in detail.

The following overview is provided to introduce a selection of concepts in a simplified form that are further described below in the Description. This overview is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

Natural language interaction with dialog systems such as digital assistants, chat bots, automation systems, and other such systems, is now common place. The natural language input received from a user or other system is referred to in this disclosure as an utterance. An utterance can have a variety of formats such as verbal, textual, and so forth. In this disclosure, textual utterances will be used for illustration purposes, but the disclosure is not so limited and any format of utterance will be encompassed by the disclosure.

The dialog system receives an utterance and returns a reply to the user and/or initiates a desired action on behalf of the user. For example, a user may utter "turn on the lights" and expect that a digital assistant, automation system, or other such system will turn the lights on. As another example, a user may input "tell me about last nights baseball scores" and expect that the scores from last night's games will be displayed.

These dialog systems use a language understanding model to detect what the user wants. The language understanding model receives an utterance and returns an intent and/or entity. The language understanding model operates in a stateless manner, evaluating each utterance and returning each intent and/or entity based solely on the received utterance. As used herein, an "intent" is what the user wants to do. An "entity" is information related to the intent. For example, if the utterance is "order me a pizza" the intent is to order goods or services. The entity "pizza" is an entity and is what the user wants to order. As another example, if the utterance is "play the latest King Arthur trailer" the intent is to view media. The entities are "trailer" which it the type of media the user desires to view and "King Arthur" is the particular trailer the user desires to view. While there is generally only one intent associated with an utterance, there can be zero or more entities associated with an utterance.

Often users give short utterances, expecting the dialog system to understand what the user wants. However, short utterances are often highly ambiguous and they can be very difficult for the dialog system and language model to determine the intent of the utterance. For example, consider the utterance "Chicago" could mean many different things and it may be extremely difficult to determine an intent from such a short utterance.

Consider the following dialog exchange between a user and a dialog system:.

Since the language understanding model operates in a stateless manner, each utterance stands on its own and the intents and/or entities of the utterance evaluated only on the current input utterance. Thus, although the language understanding model may be able to determine that the intent in utterance U1 is to book a flight, the intent for utterance U2 and U3 would likely come back as "none" since the utterance contains only cities and states or countries and the language understanding model would be unable to discern any intent in the utterances.

Embodiments of the present disclosure utilize prior dialog context to help disambiguate intent and/or entities in an utterance. In the context of this disclosure, prior dialog context means prior intents and/or entities from the dialog. The language understanding models in embodiments of the present disclosure can be trained using utterance-context pairs. In some embodiments, the utterance-context pairs are an input utterance and one or more prior intents, an input utterance and one or more prior entities, and/or an input utterance and one or more prior intents and one or more prior entities.

When the language understanding model receives prior dialog context along with the utterance, the language model utilizes features from both the utterance and prior dialog context in order to improve the ability of the language understanding model to be able to disambiguate utterances and increase the accuracy of intent and/or entity detection.

Embodiments of the present disclosure can apply to a wide variety of systems whenever machine recognition comprises textual clues in the source image. Example systems in which embodiments of the present disclosure can apply include, but are not limited to, visual search systems, visual recognition systems, and any system where automated recognition of an item can be used to drive further processing such as an automated checkout system. All these are collectively referred to as recognition systems herein.

<FIG> illustrates an example architecture <NUM> of a representative dialog system <NUM> according to some aspects of the present disclosure. The dialog systems and/or dialog services of the present disclosure are simply referred to as a dialog system for ease of discussion. The representative architecture <NUM> can be applied to a variety of various situations and contexts.

As a first example, the architecture <NUM> can be an informational chatbot. In this context, a user interacts with the informational chatbot via a user device <NUM> such as a mobile phone, a computer (laptop, desktop, etc.), a tablet system, a gaming system, and/or so forth. In some contexts, the user may be required to authenticate through an authentication service <NUM> prior to accessing the informational chatbot. The authentication service <NUM> performs authentication and/or authorization of the user to ensure the user is entitled to access the chatbot, customize the chatbot to the user, and/or for other purposes.

Once the user has been authenticated and/or authorization checked, the user is allowed to access the dialog system <NUM>, which in this example is an informational chatbot, which answers questions, provides information to the user, and so forth.

The user can ask the dialog system <NUM> questions within the domains that the chatbot covers. For example, the dialog system <NUM> can provide assistance that helps the user understand what types of utterances can be effectively answered by the dialog system.

Upon receiving an utterance, the dialog system <NUM> submits the utterance and/or prior dialog context to the language understanding models <NUM> as described herein. The language understanding model returns the intents and/or entities corresponding to the utterance.

The dialog system <NUM> evaluates the intent and/or entities and utilizes its own resources and/or the resources of additional services <NUM> and/or data <NUM> to formulate a proper prompt and return it to the user.

As an example, suppose the chatbot is designed to help programmers in a particular programming environment. The user may submit a question such as "tell me about the addExample API. " The dialog system <NUM> can pass the utterance to the language understanding model <NUM> and the language understanding model <NUM> can return an intent such as "RetrieveAPI" and the entity "addExample.

The dialog system <NUM> will understand what the user desires and retrieve information about the "addExample" API either directly from a data store <NUM> or via another service <NUM>. The dialog system <NUM> formulates the prompt giving the details of the "addExample" API and returns it to the user.

In another example, the dialog system <NUM> can be a digital assistant that can retrieve information and/or accomplish tasks on behalf of the user through various additional services <NUM>, such as calendaring services, scheduling services, purchasing services, and/or any other type of service. In such an example, the dialog system <NUM> would interact with the language understanding model <NUM> in much the same way as previously described.

In yet another example, the dialog system can be an automation system or digital assistant that can influence the user's environment by turning on/off lights, adjusting the temperature, and other such tasks. In this situation, the additional services <NUM> may include a hub or other IoT (Internet of Things) device interface and can effectuate the user's intent by interacting with the hub and/or IoT device interface.

These examples represent a small fraction of the contexts in which the architecture of <FIG> can be used. In all contexts, the various aspects such as the dialog system <NUM>, language understanding model <NUM>, additional services <NUM> and/or data store <NUM> can be integrated into a single system, can utilize one or more services to implement one or more of the aspects, and/or combinations thereof. For example, Microsoft® provides online services for chatbots, language understanding models, and/or other services that can be used to implement one or more of the illustrated aspects.

The language understanding model <NUM> is trained via a model training process <NUM> as described herein. As described below, a developer can utilize a developer system <NUM> to submit model training data to a model training process <NUM>. The language understanding model <NUM> is then trained by the model training process <NUM> utilizing the training data.

<FIG> illustrates an example diagram depicting a dialog system and language model interaction <NUM> according to some aspects of the present disclosure. Example dialog systems <NUM> include a digital assistant or a chatbot or another such system which receives natural language input from a user via a user machine <NUM> and responds to the user and/or initiates action(s) on behalf of the user as discussed herein. These dialog systems <NUM> interact with a user by receiving user requests <NUM>, providing prompts to the user <NUM>, and/or performing actions on behalf of a user. At a high level, the dialog system <NUM> receives information <NUM> and, based on that information as well as other information in some instances formulates one or more prompts <NUM> and/or performs actions on behalf of the user. Thus, the sequence of a conversation can be viewed as a network map, with each node representing a current state of the conversation and moving from one node to another based on user input and/or other information. The state of the conversation can be based on the prior request/prompts as well as the input provided by the user.

The input from the user <NUM> (e.g., request) an utterance. In order to formulate a prompt <NUM> and/or initiate action on behalf of the user, the dialog system <NUM> uses a language understanding model <NUM> as described herein. While the current state of the conversation is possibly dependent upon past request/prompts, past conversation state, and so forth, the language understanding model <NUM> operates in a stateless manner. The current natural language input <NUM> is presented to the language understanding model <NUM> and the language understanding model <NUM> responds with a data structure that comprises the intent (if any) and entities (if any) of the utterance as understood by the language understanding model. In some embodiments the data structure also comprises a confidence level of the intent and/or entities.

As an example, suppose the user provides the natural language input "Book me a flight to Cairo. " The language understanding model <NUM> can evaluate the input and provide a data structure that comprises a set of item-value pairs. For example, the item-value pairs can comprise one or more of:.

These can be encoded into one or more data structures. For example, for the request "Book me a flight to Cairo" the data structure can comprise one or more of:
{
"query": "Book me a flight to Cairo",
"topScoringIntent": {
"intent": "BookFlight",
"score": <NUM>
},
"intents": [
{
"intent": "BookFlight",
"score": <NUM>
},
{
"intent": "None",
"score": <NUM>
},
{
"intent": "LocationFinder",
"score": <NUM>
},
{
"intent": "Reminder",
"score": <NUM>
},
{
"intent": "FoodOrder",
"score": <NUM>. 765154E-<NUM>
},
],
"entities": [
{
"entity": "cairo",
"type": "Location",
"startIndex": <NUM>,
"endIndex": <NUM>,
"score": <NUM>
}
]
}.

In the above data structure, the startIndex and endIndex are the character positions in the input utterance that correspond to the beginning and ending of the corresponding entity.

As an example of how the various nodes of the conversation map are traversed, consider that the node <NUM> represents the current (or source) node of a conversation. The parameters of the current state, including whatever prompt is sent to the user is represented by <NUM>. Based on the user response and/or other information (<NUM>, <NUM>), the dialog system will move to either destination node <NUM>, displaying a prompt and having parameters <NUM> or move to destination node <NUM>, with its associated prompt/parameters <NUM>. The destination node becomes the new source node and the process repeats for the duration of the conversation. Over time, as the conversation proceeds, a conversation map from the dialog system with each node in the map having a set of parameters will be traversed.

<FIG> illustrates an example flow diagram <NUM> for using prior dialog context according to some aspects of the present disclosure. This method can be implemented, for example, by the dialog systems of the present disclosure. The method begins at operation <NUM> and proceeds to operation <NUM> where an utterance is received from a user or other system.

In operation <NUM> the dialog system determines whether the utterance is part of an existing dialog. This is accomplished by considering one or more factors either as a set of rules or as inputs into a machine learning or other model that combines the factors into a probability, or some combination thereof. For example, factors such as length of time from the prior utterance and/or prompt, whether the system has received the utterance in response to a prompt that is part of an existing dialog, whether the utterance is in response to any prompt or not, whether a prior dialog has been closed, and so forth can indicate whether a received utterance is likely to be part of an existing dialog or whether the received utterance is not part of an existing dialog. To explain these factors further:.

Thus, the dialog system can either have a set of rules such as the above, or can input a combination of factors into a trained machine learning model or other type of model, or some combination thereof, in order to ascertain the likelihood that the utterance is part of an existing dialog or a new dialog. Since there are only two choices (the utterance is part of an existing dialog or the utterance is part of a new dialog), the two likelihoods are related, and the system can utilize either of these likelihoods to take action. Comparing the desired likelihood to a threshold can allow the system to make a choice as to whether the received utterance is part of an existing dialog or not.

If the utterance is part of an existing dialog, the method proceeds to operation <NUM> where the system identifies the prior dialog context. As noted herein, the prior dialog context can comprise one or more intents, one or more entities, or any combination thereof.

The system then sends the utterance and prior dialog context to the language understanding model in operation <NUM>.

If, on the other hand, the system determines that the received utterance is not part of an existing dialog, the method proceeds to operation <NUM> and the utterance (without prior dialog context) is sent to the language understanding model.

The dialog system receives any intents and/or utterances associated with received utterance or received utterance and prior dialog context from the language understanding model in operation <NUM> as described herein.

Based on the intents and/or entities, the dialog system determines what the next action should be based in its conversation state, internal programming, known information and/or so forth in operation <NUM>. The method terminates in operation <NUM>.

An example can help illustrate how the method of <FIG> operates. As noted in <FIG>, the dialog system has or creates a conversation map during a dialog with the user. Consider the example of booking a flight for the user. The user may initiate this action by an utterance that indicates to the dialog system that a flight should be booked. For example, an initial utterance which initiates the booking of a flight can be one or more of:.

or any number of similar utterances. The dialog system can submit the utterance to the language understanding model and receive in return an intent indicating the user wants to book a flight and any entities contained in the utterance. Some of the utterances above have one or more entities and some do not.

Once the dialog system understands that the user wants to book a flight, certain information needs to be collected before the action can be initiated and/or completed. In the example of booking a flight, the dialog system would need to know, for example:.

Not all such information may need to be collected, but there is a minimum set of information that the dialog system would need before travel options can be identified and presented to a user. Suppose this minimum set of data included the originating city, the destination city, and the dates of travel. The dialog system may be programmed so that the user is prompted to provide the minimum set of data and then present some options to the user. In such a situation, the dialog may proceed as follows:.

The dialog system receives utterance U1 at operation <NUM>. In operation <NUM> the dialog system determines whether the utterance is part of an existing dialog. Since the dialog system has no current dialog state, and since the utterance was not received in response to a dialog prompt (or was received in response to a general prompt such as "how can I help today?"), the system concludes that the utterance is not part of any existing dialog and proceeds to operation <NUM> and submits the utterance to the language understanding model.

At operation <NUM>, the dialog system receives an intent (BookFlight) and entity (date: Tuesday). The dialog system identifies the missing information needed to effectuate the user intent (in this case the destination and source cities) from its internal conversation maps and in operation <NUM> generates the prompt R1 "where would you like to travel?" and presents it to the user.

On the next turn of the dialog, the utterance "Redmond, Washington" is received (utterance U2). This time, the dialog system determines that the utterance is part of an existing dialog in operation <NUM> because the utterance was received in response to a prompt that was part of an existing dialog. Thus, the dialog system proceeds to operation <NUM> and identifies "BookFlight" as the prior dialog context.

In operation <NUM> the "BookFlight" intent as well as the utterance "Redmond, Washington" is sent to the language understanding model. In response, the language understanding model can identify the existing intent "BookFlight" and the entities of the utterance, "city: Redmond" and "state: Washington" which are received by the dialog system in operation <NUM>.

Since the prompt was asking for destination city, the dialog system can slot the city and state into the destination city entity in operation <NUM> and then prompt for the remaining missing information with prompt R2: "Where are you traveling from?".

The process repeats substantially as described above with the received utterance U3 "Cairo Egypt" being identified as part of the same dialog in operation <NUM> and the prior dialog context of "BookFlight" being passed to the language understanding model and the intent "BookFlight" and entities city: Cairo and Country: Egypt being identified in the utterance.

At operation <NUM>, the dialog system should have all the information it needs to present some travel choices to the user, in accord with the representative example.

Although not shown in <FIG>, one option for the dialog system is to present both the utterance with the prior dialog context and the utterance alone to the language understanding model and then evaluate the intents and/or entities that are returned and make a decision as to whether the utterance is part of the prior dialog or not. This can be implemented in several ways. For example, if the dialog system cannot decide (e.g., the likelihoods are "too close to call"), the factors considered point to both results (e.g., the a prompt that is part of a dialog was sent and the utterance was received in response to the prompt, but the time between prompt and utterance would otherwise indicate that the two are not related), and/or combinations thereof, and/or for other reasons, the dialog system can then submit both. This would turn operation <NUM> from a binary decision (yes, no) to a trinary decision (yes, no, maybe) and the "maybe" path would be to submit both the utterance alone and the utterance-prior dialog context pair to the language understanding model. The system can then use various criteria to decide when to take the "yes" path, when to take the "no" path, and when to take the "maybe" path.

In still another embodiment, the dialog system can submit both the utterance alone and the utterance-prior dialog context pair to the language understanding model based on some other criteria. In one representative example, the dialog system can always pick either the "yes" path. In another representative example, the dialog system can always pick the "maybe" path.

<FIG> illustrates an example <NUM> of deriving training data according to some aspects of the present disclosure. As discussed herein, the language understanding model can include one or more machine learning models that are trained using training data. To train a model to extract intents and entities from utterances alone, a set of example utterances <NUM> are collected. The example utterances <NUM> can be collected in several ways as discussed herein.

Operation <NUM> can examine the example utterances <NUM> one by one and extract features from an example utterance and assemble the features into a training vector <NUM> that corresponds to the example utterance. In this way a collection of training vectors <NUM> are extracted from the example utterances <NUM>.

Feature extraction operation <NUM> can extract features in a variety of ways. In one embodiment, stop words (e.g., common words that convey little or no meaning like articles, "a", "an", "the" etc.) can be removed and the remaining words can be assembled into a training vector. In other embodiments, other mechanisms can be used to extract features. For example, words can be stemmed, n-grams can be extracted from the input phrase, and other such processing can be part of the feature extraction process <NUM>. In another example, Word2vec or similar tools that produces word embeddings can be utilized to produce training vectors from the example utterances. When trained with a corpus of text, Word2vec produces a vector space where words are assigned positions in the vector space in a manner where words that share common context (such as being semantically similar) are positioned in proximity to one another. The Word2vec tools, methods, and variants thereof are well known and need not be discussed further here.

In some instances, the example utterances <NUM> are annotated, meaning that the example utterances and have identified intents and/or entities that are associated with the example utterance. When example utterances <NUM> are annotated, feature extraction method <NUM> can use the annotations to extract appropriate features, can encode the annotations into the resultant training vector <NUM>, and/or combinations thereof.

The resultant training data set <NUM> can be used to train a language understanding model to identify intents and/or entities associated with an utterance when the utterance is submitted without prior dialog context.

To train a language understanding model to identify intents and/or entities associated with an utterance when the utterance is submitted with prior dialog context, utterance-context pairs are used. The process of assembling utterance-context pairs is discussed in greater detail below. However, in summary, an example utterance from a set of example utterances <NUM> is associated with an example prior dialog context from a set of example prior example contexts <NUM>. The pair can be put through a feature extraction process <NUM> that operates substantially the same as feature extraction process <NUM> discussed above. The resultant output of feature extraction process <NUM> is a training vector <NUM> that is associated with the corresponding utterance-context pair. The training vectors <NUM> can be collected into a training data set <NUM> and used to train a language understanding model to recognize intents and/or entities as described herein.

<FIG> and <FIG> illustrates a representative architecture for training a machine learning model according to some aspects of the present disclosure. <FIG> illustrates training a machine learning model for example utterances while <FIG> illustrates training a machine learning model for example utterance-context pairs.

Turning first to <FIG>, example utterances <NUM> are collected. The example utterances <NUM> are the example utterances that will be used to train the machine learning model of the language understanding model. After training, the language understanding model will extract intents and/or entities as described herein.

There are numerous ways to collect a set of example utterances for training. In one embodiment, the example utterances are provided by the developer through a machine <NUM>. For example, the developer can simply provide a complete set of example utterances <NUM>.

In another embodiment, the example utterances are provided by one or more applications and/or services <NUM>. For example, applications and/or services <NUM> can monitor what users enter and collect the example utterances for initial training and/or updating of the language understanding model. In some instances, criteria can be used to collect example utterances. For example, the applications and/or services <NUM> can monitor the output of the language understanding model while the dialog system is in use and when the language understanding model is unable to identify an intent and/or entity associated with an utterance. This can be determined, for example, when the scores associated with the intents and/or entities identified in an utterance fall below a designated threshold. For example, if the user enters an utterance and the top scoring intent and/or entity has a score that falls below a threshold, the application and/or services <NUM> can capture the utterance for use in training.

The application and/or services <NUM> can be the dialog system itself, another system, application and/or service, and/or any combination thereof. <FIG>, discussed below, can be representative of a mechanism that helps extract information from a dialog system, either in a fully automated way, or in a developer assisted way, and/or some combination thereof, in order to collect example utterances and/or utterance-context pairs (as illustrated in <FIG>).

In yet another example, the example utterances can be collected from the developer in an interactive manner. For example, a user interface <NUM> can be presented to the developer that allows the developer to enter example utterances, annotate example utterances, initiate training of the model, run test cases to see how the trained model performs, and control other aspects of example utterance collection, model training, and/or model training evaluation. An example of this type of embodiment is discussed below, along with a set of representative APIs that can be used to collect information and interact with a system that stores example utterances (<NUM>) and/or example utterance-context pairs (<NUM>) in order to train an appropriate model.

Once a set of example utterances <NUM> are collected, a feature extraction process <NUM> extracts features to create a set of training data <NUM>. Feature extraction was discussed above in conjunction with <FIG>.

In some example embodiments, different machine-learning tools may be used. The language understanding model can be set up as either a classification problem or a matching problem. In the classification problem approach, the training data is used with an appropriate machine learning model to train the model to recognize intents and/or entities from utterances and/or features extracted from utterances. In the matching problem approach, the training data is used with an appropriate machine learning mode to train the model to select the matching intents and/or entitles associated with an utterances and/or features extracted from utterances.

For classification type systems example machine learning models include, but are not limited to, Logistic Regression (LR), Naive-Bayes, Random Forest (RF), neural networks (NN), matrix factorization, and Support Vector Machines (SVM) tools. For matching type systems, some form of neural network (e.g., Long Short Term Memory, bidirectional Long Short Term Memory, etc.) is generally used to encode both the utterance and candidate intents and/or entities and a matching score determined between the encoded utterance and encoded candidate intents/entities. The intent and/or entities with the highest matching score are selected. The machine learning models for both classification type problems and for matching type problems are known and any such models can be utilized with embodiments of the present disclosure.

The machine-learning algorithms utilize the training data <NUM> to find correlations among the identified features that affect the outcome or assessment <NUM>. In some example embodiments, the training data <NUM> includes annotated data (also referred to as labeled data), which is known data for one or more identified features and one or more outcomes, such as the intents and/or entities associated with an example utterance.

With the training data <NUM>, the machine learning model of the language understanding model is trained at operation <NUM>. The training process <NUM> appraises the value of the features as they correlate to the training data <NUM> and the associated intents and/or entities. This is typically done by minimizing an error function. Training of the various machine learning models is known and need not be further discussed here. The result of the training is the trained language understanding model <NUM>.

When the trained language understanding model <NUM> is used to perform an assessment, new data <NUM> (e.g., a new utterance) is provided as an input to the trained language understanding model <NUM>, and the language understanding model <NUM> generates the assessment <NUM> as output. In this case the assessment <NUM> are the intents and/or entities associated with the input utterance <NUM>.

<FIG> illustrates the same process using utterance-context pairs rather that utterances. Thus, the description of <FIG>, mutatis mutandis, applies to <FIG>.

There are several ways to collect the utterance-context pairs (<NUM>) in order to produce the training data. <FIG> illustrates a representative flow diagram <NUM> for deriving training data according to some aspects of the present disclosure. This flow diagram is an example method of how to collect utterance-context pairs to be used training a language understanding model as described herein.

The method begins at operation <NUM> and proceeds to operation <NUM> where prompts having prior dialog contexts are identified. As discussed in conjunction with <FIG>, dialog systems have or build a conversation network map during the course of a dialog. Prompts displayed by a dialog system to a user are programmed by a developer, created during operation of the dialog system, and/or some combination thereof. Each prompt can be designed to convey information to a user, elicit a response from a user, or both. Thus, prompts can be seen as steps in a dialog to begin the dialog, continue the dialog, and/or end the dialog.

Those prompts that are part of a dialog are those that are likely to benefit from prior dialog context. Thus, if a dialog system has a set of prompts that are part of a given dialog, collecting the prompts can be part of the implementation of operation <NUM>. For example, if the dialog system has a set of pre-determined prompts and/or prompts that are created during runtime (e.g., using a set of conditions, rules, and/or so forth), operation <NUM> can collect such prompts from information utilized by the dialog system, such as data structures where such prompts are stored, conditions, rules, and/or so forth where prompts are retrieved and/or created. As another example, if the dialog system creates prompts dynamically during run time, the prompts can be collected during run time and later retrieved by operation <NUM>. As yet another example, a developer can input the set of prompts, such as through an interactive user interface. As still another example, a developer can input the set of prompts in a batch. Other mechanisms can also be used to collect a set of prompts that have associated prior dialog context.

As a representative example, consider a simple book a flight dialog that may contain several prompts that are part of the dialog. For example, suppose in order to complete booking of a flight, a dialog system is programmed to accomplish the booking in several steps:.

In this representative example, step <NUM> is completed before step <NUM> and so forth, although one of the strengths of dialog systems is that they can be flexible in terms of such orders.

These different steps can be represented as rules, dialog flows, nodes in a conversation map, or in any other way. The dialog system will utilize one or more of these representations to present dialog prompts to gather the needed information, create dialog prompts to gather the needed information, and/or a combination thereof.

To see how the prompts with prior dialog context can be collected (e.g., operation <NUM>) from the information above, consider a conversation map that is created from the steps. In this example, step <NUM> must be completed before step <NUM> is accomplished. Step <NUM> will be entered based on an entered initial utterance such as "I need to travel for business next week. " The initial utterance may or may not have one or more of the needed entities <NUM> :a-c. Thus, an utterance that results in the identified intent "BookFlight" will initiate step <NUM>. Since at a minimum of the data for the entities in step <NUM> :a-c must be collected before step <NUM> can be marked as complete and step <NUM> started, the system will utilize prompts to collect information from the user until the entities in <NUM> :a-c are known. Thus, one or more prompts will be associated with <NUM> :a-c in order to gather the needed information. For example, a simple set of prompts may include the following:.

Of course, more complicated set of prompts may be used by a dialog system, such as asking for source and destination cities and only asking for clarification on the airport if more than one exist or simply searching for flights to all airports serving the city. The prompts may change depending on the initial utterance in some embodiments.

The prompts in operation <NUM> can be directly collected from the information in the dialog system, collected during runtime and later utilized as described herein, and/or identified in another way. As an alternative, an interactive user interface can be collected through interaction with the developer. For example, the system may identify entities <NUM>:a-c as being required and the system can prompt the developer for various prompts that can be presented to the user in order to gather the entity information. Other ways such as a batch upload and so forth can be used to collect the various prompts.

An example of an interactive embodiment for collecting examples and interacting with a model training system is presented below.

Operation <NUM> begins a loop that considers each prompt in the set collected in operation <NUM> in turn.

Operation <NUM> identifies one or more example utterances that a user might enter in response to the prompt under consideration. For example, if the prompt is part of a dialog to book a flight such as "when would you like to travel?" operation <NUM> would identify possible responses a user may enter. For example, a user may enter responses such as:.

Thus, each prompt in table <NUM> may give rise to a number of associated utterances.

Operation <NUM> can collect the example utterances that a user might enter in response the prompt under consideration in several ways. For example, in an interactive context, the prompt may be displayed to a developer and the developer asked to enter one or more example utterances that a user may enter in response to the prompt. These can be collected by operation <NUM>. In another example, during operation the dialog system, or another service/system, or a combination thereof can collect user responses to prompts and the data can be mined by operation <NUM> to extract example utterances associated with the prompt under consideration. Other options such as a developer entering example utterances in a batch can also be utilized.

Operation <NUM> identifies example prior dialog contexts for each of the example utterances. In some embodiments, the system examines the prompt under consideration and creates example contexts from dialog information associated with the prompt. For example, the prompt under consideration will either be part of no dialog or part of a dialog. In the case where the prompt is part of no dialog, the system can treat the prompt as having no prior dialog context, or can identify a prior dialog context by querying the developer, or identify a prior dialog context in some other fashion.

In the case where the prompt is part of a dialog, the dialog itself may be associated with an intent. For example, a prompt of "where do you want to travel" that is part of a flight booking dialog can be associated with the "BookFlight" intent. In such a situation, the intent can be extracted from dialog metadata, the dialog data store, or other location where the intent associated with the dialog can be found. Similarly, other prompts in the dialog may be mined for additional context data.

As a representative example of mining dialog data to discover prior dialog context, consider the conversation map of <FIG>, where each node in the map has associated data (e.g., <NUM>, <NUM>, <NUM>, <NUM>) that contains the prompt. Collecting all the prompts can give prior dialog contexts in terms of intents and/or entities. Similarly, the information that causes the state transition (e.g., <NUM>, <NUM>) can be mined for representative example utterances. This can be an example of mining data stored, accessed, and/or created by a dialog system to identify example contexts associated with a prompt under consideration.

As yet another example, a user interface can be presented to the developer and the developer asked to provide example prior dialog contexts that are to be associated with prompt under consideration.

As yet another example, consider the booking a flight example above. Example prior dialog contexts can include one or more intents and/or one or more entities. In the booking a flight example above, step <NUM> is completed before step <NUM> is accomplished and so forth. A minimum of the data in step <NUM> :a-c must be collected before step <NUM> can be marked as complete and step <NUM> started.

In one embodiment, the example context is simply the intent, BookFlight. In other embodiments various combinations of the entities in <NUM> :a-c can also be included in the example context. Unless the dialog system is designed so that users have to enter data in a particular order, the example contexts can account for entry of the entities in any order and all combinations can be used as example prior contexts. For example, assuming that order of previously entered entities do not matter, prior dialog context can comprise the intent and any combination of entities. In this case, the intent is BookFlight and the entities can be BeginTravelDate, EndTravelDate, SourceAirport, and/or DestinationAirport in any combination. Thus, prior context expressed in expressed in an intent. entities format can include:.

Along with the intent alone, each of these can represent a prior example context that is identified in operation <NUM>.

In operation <NUM> utterance-context pairs are assembled. To create the utterance-context pairs, a context is paired with the utterances that are designed to elicit the information not already represented by the prior dialog context. For example, if the prior dialog context is "BookFlight", the example utterances that are given in response to the prompts of Table <NUM> above can be paired with the intent BookFlight. For contexts that have associated entities, the utterances are those that are given in response to entities that are not part of the context. For example, if the context is (<NUM>) on the list above (BookFlight. DestinationAirport), the utterances associated with the prompts designed to collect SourceAirport, BeginTravelDate, and EndTravelDate can be paired in combination. If the context is (<NUM>) on the list (BookFlight. EndTravelDate. SourcAirport. DestinationAirport), the only missing entity is BeginTravelDate, so the utterances associated with the prompt to collect BeginTravelDate are paired in combination with the context (<NUM>).

Operation <NUM> ends the loop for the prompts and the method ends at operation <NUM>.

As discussed herein, one approach to gathering training examples, with or without prior dialog context is to allow a developer to interact with the system through a user interface such as shown in <FIG>:<NUM> and/or <FIG>:<NUM>. The user interface can collect information and use APIs to store the information, such as according to the database schema of <FIG> or in another data store. Such an interactive approach can be particularly useful when the language understanding model is provided by an infrastructure, service, or some other type of platform that the developer's dialog system relies on.

The user interface (e.g., <FIG>:<NUM> and/or <FIG>:<NUM>) can present options that allow a developer to enter an example, with or without prior dialog context, and then annotate and/or save the example. For example, the system can allow the developer to enter an example utterance, identify intents, entities, and/or prior dialog context associated with the utterance. Using the booking a flight example above, the developer can enter: "I need to travel for business next week" and identify the associated intent as BookFlight. Similarly, the user can enter "Book me a flight to Cairo next Tuesday" and identify the intent as BookFlight, the entity DestinationAirport as Cairo and the entity BeginTravelDate as Tuesday.

Similarly, prior dialog contexts can interactively be entered. For example, the user can enter "Friday" as the utterance and the prior dialog context as having the BookFlight intent and BeginTravelDate entity with a value of Tuesday.

In some embodiments, the interactivity can be assisted by the system. For example, if the training system was able to extract the prompts in Table <NUM> from the developer's dialog system (e.g., as part of operation <NUM>), the training system can walk through each of the prompts and allow the user to enter example utterances against each of the prompts (e.g., as part of operation <NUM>). The user can then identify example prior dialog contexts through the user interface as well for the different example utterances entered.

The interactive entry of example utterances and/or example utterance-context pairs can also be accompanied by an interactive loop through the training methodology outlined in <FIG> and <FIG>. For example the user can enter one or more example utterances and/or utterance-context pairs as described, the feature extraction (<NUM>, <NUM>) can extract the training data (<NUM>, <NUM>), the model can be trained and/or updated (<NUM>, <NUM>). The user can then enter a test utterance and/or utterance-context pair (<NUM>, <NUM>), and the test utterance and/or utterance-context pair evaluated by the trained model (<NUM>, <NUM>) and the resulting assessment (<NUM>, <NUM>) presented to the developer through the user interface (<NUM>, <NUM>). The developer can then add annotations to correct any errors made by the model and the annotated correction be used to further update the model.

<FIG> illustrates a representative flow diagram <NUM> for selecting and utilizing a trained machine learning model according to some aspects of the present disclosure. The trained machine learning models are those trained as part of the language understanding models as described herein. The process is performed, for example, by the language understanding model in some embodiments.

The process begins at <NUM> and proceeds to <NUM> where input parameters are received from the dialog system. The input parameters can comprise the input utterance and (optional) prior dialog context. Because the dialog system may or may not present prior dialog context (even when such is available), the language understanding model needs to accommodate scenarios where prior dialog context is received along with an input utterance and scenarios where no prior dialog context is received and only an input utterance is received.

As noted above, one way to accommodate both scenarios is to train two machine learning models, one on utterance-context pairs and one only on utterances. This is the embodiment illustrated in <FIG>. In another embodiment, only a single machine learning model is used, which has been trained using both the utterance only training data and the utterance-context pair training data.

Operation <NUM> determines whether prior dialog context has been received along with the utterance or only an utterance without any prior dialog context. Responsive to the prior dialog context having been received, execution proceeds to operation <NUM> where the model trained on utterance-context pairs is selected. If only an utterance without any prior dialog context is received, execution proceeds to operation <NUM> where the model trained only on utterances is selected.

The input (e.g., utterance or utterance/prior dialog context) is presented to the selected, trained machine learning model in operation <NUM>. In response, the model returns intents, entities, and related scores as desired. For example, the model can be trained to return a score associated with all possible intents and/or any identified entities along with their respective scores. As discussed herein, the scores represent the likelihood that the associated intent and/or entity is represented in the input utterance.

In operation <NUM> the desired parameters are assembled and returned to the dialog system. The dialog system may want all the intents/entities and associated scores, or may want only the top scoring intents and/or entities. For example a "verbose" set of parameters returned to the dialog system may look something like:
{
"query": "Book me a flight to Cairo",
"topScoringIntent": {
"intent": "BookFlight",
"score": <NUM>
},
"intents": [
{
"intent": "BookFlight",
"score": <NUM>
},
{
"intent": "None",
"score": <NUM>
},
{
"intent": "LocationFinder",
"score": <NUM>
},
{
"intent": "Reminder",
"score": <NUM>
},
{
"intent": "FoodOrder",
"score": <NUM>. 765154E-<NUM>
},
],
"entities": [
{
"entity": "cairo",
"type": "Location",
"startIndex": <NUM>,
"endIndex": <NUM>,
"score": <NUM>
}
]
}
On the other hand, a "top scoring" result may be something like:
{
"query": "Book me a flight to Cairo",
"topScoringIntent": {
"intent": "BookFlight",
"score": <NUM>
},
"entities": [
{
"entity": "cairo",
"type": "Location",
"startIndex": <NUM>,
"endIndex": <NUM>,
"score": <NUM>
}
]
}.

<FIG> illustrates a representative machine architecture suitable for implementing the systems and so forth or for executing the methods disclosed herein. The machine of <FIG> is shown as a standalone device, which is suitable for implementation of the concepts above. For the server aspects described above a plurality of such machines operating in a data center, part of a cloud architecture, and so forth can be used. In server aspects, not all of the illustrated functions and devices are utilized. For example, while a system, device, etc. that a user uses to interact with a server and/or the cloud architectures may have a screen, a touch screen input, etc., servers often do not have screens, touch screens, cameras and so forth and typically interact with users through connected systems that have appropriate input and output aspects. Therefore, the architecture below should be taken as encompassing multiple types of devices and machines and various aspects may or may not exist in any particular device or machine depending on its form factor and purpose (for example, servers rarely have cameras, while wearables rarely comprise magnetic disks). However, the example explanation of <FIG> is suitable to allow those of skill in the art to determine how to implement the embodiments previously described with an appropriate combination of hardware and software, with appropriate modification to the illustrated embodiment to the particular device, machine, etc. used.

While only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The example of the machine <NUM> includes at least one processor <NUM> (e.g., a central processing unit (CPU), a graphics processing unit (GPU), advanced processing unit (APU), or combinations thereof), one or more memories such as a main memory <NUM>, a static memory <NUM>, or other types of memory, which communicate with each other via link <NUM>. Link <NUM> may be a bus or other type of connection channel. The machine <NUM> may include further optional aspects such as a graphics display unit <NUM> comprising any type of display. The machine <NUM> may also include other optional aspects such as an alphanumeric input device <NUM> (e.g., a keyboard, touch screen, and so forth), a user interface (UI) navigation device <NUM> (e.g., a mouse, trackball, touch device, and so forth), a storage unit <NUM> (e.g., disk drive or other storage device(s)), a signal generation device <NUM> (e.g., a speaker), sensor(s) <NUM> (e.g., global positioning sensor, accelerometer(s), microphone(s), camera(s), and so forth), output controller <NUM> (e.g., wired or wireless connection to connect and/or communicate with one or more other devices such as a universal serial bus (USB), near field communication (NFC), infrared (IR), serial/parallel bus, etc.), and a network interface device <NUM> (e.g., wired and/or wireless) to connect to and/or communicate over one or more networks <NUM>.

The various memories (i.e., <NUM>, <NUM>, and/or memory of the processor(s) <NUM>) and/or storage unit <NUM> may store one or more sets of instructions and data structures (e.g., software) <NUM> embodying or utilized by any one or more of the methodologies or functions described herein. These instructions, when executed by processor(s) <NUM> cause various operations to implement the disclosed embodiments.

As used herein, the terms "machine-storage medium," "device-storage medium," "computer-storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms refer to a single or multiple storage devices and/or media (e.g., a centralized or distributed database, and/or associated caches and servers) that store executable instructions and/or data. The terms shall accordingly be taken to include storage devices such as solid-state memories, and optical and magnetic media, including memory internal or external to processors. Specific examples of machine-storage media, computer-storage media and/or device-storage media include non-volatile memory, including by way of example semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms machine-storage media, computer-storage media, and device-storage media specifically and unequivocally excludes carrier waves, modulated data signals, and other such transitory media, at least some of which are covered under the term "signal medium" discussed below.

The term "signal medium" shall be taken to include any form of modulated data signal, carrier wave, and so forth. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a matter as to encode information in the signal.

Claim 1:
A method for language understanding, comprising:
receiving (<NUM>, <NUM>) an input utterance;
ascertaining a likelihood that the input utterance is part of an existing dialog by considering one or more factors being at least one of i) a length of time from a prior utterance and/or prompt, ii) whether the utterance has been received in response to a prompt that is part of an existing dialog, iii) whether the utterance is in response to any prompt or not, or iv) whether a prior dialog has been closed, the factors being considered either as a set of rules or as inputs into a machine learning or other model that combines the factors into a probability, or some combination thereof;
determining (<NUM>) whether the input utterance is part of an existing dialog or not by comparing the likelihood to a threshold;
responsive to determining that the input utterance is part of an existing dialog:
identifying (<NUM>, <NUM>) prior dialog context information that was received in conjunction with the input utterance;
selecting (<NUM>) a first trained language understanding model, wherein the first trained language understanding model is trained using example utterances and associated prior dialog context;
determining (<NUM>) a first intent from the input utterance and the prior dialog context information using the first trained language understanding model; and returning (<NUM>) the first intent; and
responsive to determining that the input utterance is part of a new dialog:
selecting (<NUM>) a second trained language understanding model, wherein the second trained language understanding model is trained using example utterances without associated prior dialog context pairs;
determining (<NUM>) a second intent from the input utterance using the second trained language understanding model; and
returning (<NUM>) the second intent.