System and method for learning new concepts from input utterances

A method includes applying, by at least one processor, a natural language understanding (NLU) model to an input utterance in order to obtain initial slot probability distributions. The method also includes performing, by the at least one processor, a confidence calibration by applying a calibration probability distribution to the initial slot probability distributions in order to generate calibrated slot probability distributions. The calibration probability distribution has a higher number of dimensions than the initial slot probability distributions. The method further includes identifying, by the at least one processor, uncertainties associated with words in the input utterance based on the calibrated slot probability distributions. In addition, the method includes identifying, by the at least one processor, a new concept contained in the input utterance that is not recognized by the NLU model based on the identified uncertainties.

TECHNICAL FIELD

This disclosure relates generally to machine learning systems. More specifically, this disclosure relates to a system and method for learning new concepts from input utterances.

BACKGROUND

With the rise of modern artificially-intelligent voice-enabled personal assistants, natural language understanding (NLU) now plays an important role in understanding input utterances and in carrying out the intents of users. One common task in NLU is slot filling, which aims to extract semantic concepts from natural language input utterances. Existing slot filling approaches often rely on machine learning models that are trained using large amounts of pre-collected training data. However, it is infeasible for training data to cover all possible concepts that might be associated with input utterances, particularly considering the fast growth of new concepts. Because of this, it is common for trained machine learning models to incorrectly perform slot filling and intent detection for input utterances associated with new concepts. For instance, state-of-the-art NLU models are based on deep learning technologies and will often misinterpret a new concept as a random known concept.

SUMMARY

This disclosure relates to a system and method for learning new concepts from input utterances.

In a first embodiment, a method includes applying, by at least one processor, a natural language understanding (NLU) model to an input utterance in order to obtain initial slot probability distributions. The method also includes performing, by the at least one processor, a confidence calibration by applying a calibration probability distribution to the initial slot probability distributions in order to generate calibrated slot probability distributions. The calibration probability distribution has a higher number of dimensions than the initial slot probability distributions. The method further includes identifying, by the at least one processor, uncertainties associated with words in the input utterance based on the calibrated slot probability distributions. In addition, the method includes identifying, by the at least one processor, a new concept contained in the input utterance that is not recognized by the NLU model based on the identified uncertainties.

In a second embodiment, an apparatus includes at least one processor configured to apply an NLU model to an input utterance in order to obtain initial slot probability distributions. The at least one processor is also configured to perform a confidence calibration through application of a calibration probability distribution to the initial slot probability distributions in order to generate calibrated slot probability distributions. The calibration probability distribution has a higher number of dimensions than the initial slot probability distributions. The at least one processor is further configured to identify uncertainties associated with words in the input utterance based on the calibrated slot probability distributions. In addition, the at least one processor is configured to identify a new concept contained in the input utterance that is not recognized by the NLU model based on the identified uncertainties.

In a third embodiment, a non-transitory computer readable medium contains instructions that when executed cause at least one processor to apply an NLU model to an input utterance in order to obtain initial slot probability distributions. The medium also contains instructions that when executed cause the at least one processor to perform a confidence calibration through application of a calibration probability distribution to the initial slot probability distributions in order to generate calibrated slot probability distributions. The calibration probability distribution has a higher number of dimensions than the initial slot probability distributions. The medium further contains instructions that when executed cause the at least one processor to identify uncertainties associated with words in the input utterance based on the calibrated slot probability distributions. In addition, the medium contains instructions that when executed cause the at least one processor to identify a new concept contained in the input utterance that is not recognized by the NLU model based on the identified uncertainties.

DETAILED DESCRIPTION

FIGS.1through7, discussed below, and the various embodiments of this disclosure are described with reference to the accompanying drawings. However, it should be appreciated that this disclosure is not limited to these embodiments and all changes and/or equivalents or replacements thereto also belong to the scope of this disclosure.

As noted above, with the rise of modern artificially-intelligent voice-enabled personal assistants, natural language understanding (NLU) now plays an important role in understanding input utterances and in carrying out the intents of users. One common task in NLU is slot filling, which aims to extract semantic concepts from natural language input utterances. Existing slot filling approaches often rely on machine learning models that are trained using large amounts of pre-collected training data. However, it is infeasible for training data to cover all possible concepts that might be associated with input utterances, particularly considering the fast growth of new concepts. Because of this, it is common for trained machine learning models to incorrectly perform slot filling and intent detection for input utterances associated with new concepts. For instance, state-of-the-art NLU models are based on deep learning technologies and will often misinterpret a new concept as a random known concept (which is often referred to as “over-confidence” of the model).

As examples of this, it is common for unknown words or phrases to be used in input utterances, such as when untrained words or phrases are used or when new words or phrases appear over time. It is also common for unknown personalized expressions to be used in input utterances, such as when idiosyncratic or personalized words or phrases are used by a specific user or a specific subset of users. In this document, the phrases “new concept” and “new concepts” are generally used to refer to one or more concepts that have not been learned by an NLU model, such as when the concepts were not included or could not have been included in training data for the NLU model. This may be the result of the concepts not being included in the training data because of limitations on the training data, such as due to the limited size or nature of the training data. This may also be the result of the concepts not existing at the time of NLU model training, such as due to the creation of new concepts (like new restaurant names or new book names) over time. In order to provide improved user experiences, it may be highly desirable to detect and learn new concepts as they appear in input utterances. Each new concept can relate to any aspect of an input utterance for which a trained NLU model cannot derive a suitable slot label or otherwise understand.

Some approaches have used rule-based techniques to ask clarification questions when new concepts are detected in input utterances. However, these approaches are generally not applicable to modern deep learning NLU models. Other approaches are often designed based on the assumption that new concepts can be identified using labeled training data, but labeled training data may not exist in many cases. Still other approaches can operate to detect if an entire utterance is new to an NLU model, but these approaches cannot extract new concepts from the utterance. These issues can be particularly problematic when considering the fact that numerous new concepts can emerge quickly over time after an NLU model has been trained and deployed.

This disclosure provides techniques for learning new concepts from input utterances. As described in more detail below, a trained NLU base model can be applied to an input utterance in order to obtain initial slot probability distributions. The initial slot probability distributions may identify likelihoods of different words in the input utterance belonging to different slot types. A confidence calibration may be performed by applying a calibration probability distribution (such as a Dirichlet distribution) to the initial slot probability distributions in order to generate calibrated slot probability distributions. The calibration probability distribution has a higher number of dimensions than the initial slot probability distributions, which helps to overcome issues associated with over-confidence of the NLU base model. Uncertainties associated with words in the input utterance can be identified based on the calibrated slot probability distributions, such as by calculating entropies of the calibrated slot probability distributions. A new concept contained in the input utterance that is not recognized by the NLU model can be identified based on the identified uncertainties.

At that point, one or more actions may occur in order to process or otherwise use the input utterance. For example, a word or phrase from the input utterance associated with the new concept may be identified, such as by selecting a word or phrase having at least one uncertainty that is larger than a threshold value. The identified word or phrase can be labeled as having an unknown slot type. A prompt for clarification of the new concept may be made, such as when a user is asked for the intent or meaning of the new concept or when an online database or other data source is accessed for information related to the new concept. Based at least partially on the clarification, slot filling may be performed for the identified word or phrase, and/or an action associated with the input utterance may be identified and performed. The NLU base model and/or the confidence calibration may be retrained based at least partially on the clarification, thereby allowing future input utterances related to the same or similar concept to be understood more effectively.

In this way, new concepts (such as words or phrases) contained in input utterances can be detected, and meanings for the new concepts can be identified. Rather than relying on labeled training data, the new concepts can be detected in the input utterances being processed using a trained NLU base model, which allows learning to occur while the trained NLU base model is in use and input utterances are being received from one or more users. These techniques may also be seamlessly integrated with any deep learning-based NLU model. Using these techniques, artificially-intelligent voice-enabled personal assistants or other systems that utilize natural language understanding can achieve improved performance. These techniques may also allow artificially-intelligent voice-enabled personal assistants or other systems that utilize natural language understanding to learn new concepts continuously over time. Among other things, this may help to account for the creation of new concepts over time, changing interests by users over time, and other factors that alter the concepts contained in input utterances being processed.

Note that the phrase “input utterance” in this document refers to any suitable text-based, voice-based, or other natural language-based input. An input utterance may be provided by a user or other source in any suitable manner. For example, an input utterance may be provided by a user typing the input utterance into an electronic device or by a user speaking the input utterance to an electronic device (which can perform speech-to-text recognition). Also note that the functionality described below is often described as being used with an artificially-intelligent voice-enabled personal assistant, such as one supported by a user's mobile electronic device or other “smart” electronic device. However, the functionality described below may be used in any suitable NLU-based system with any suitable electronic device.

FIG.1illustrates an example network configuration100including an electronic device in accordance with this disclosure. The embodiment of the network configuration100shown inFIG.1is for illustration only. Other embodiments of the network configuration100could be used without departing from the scope of this disclosure.

According to embodiments of this disclosure, an electronic device101is included in the network configuration100. The electronic device101can include at least one of a bus110, a processor120, a memory130, an input/output (I/O) interface150, a display160, a communication interface170, and a sensor180. In some embodiments, the electronic device101may exclude at least one of these components or may add at least one other component. The bus110includes a circuit for connecting the components120-180with one another and for transferring communications (such as control messages and/or data) between the components.

The processor120includes one or more of a central processing unit (CPU), a graphics processor unit (GPU), an application processor (AP), or a communication processor (CP). The processor120is able to perform control on at least one of the other components of the electronic device101and/or perform an operation or data processing relating to communication. In some embodiments of this disclosure, for example, the processor120may obtain and process input utterances, identify new concepts in the input utterances, and perform slot filling, retraining, or other functions as described in more detail below.

The memory130can include a volatile and/or non-volatile memory. For example, the memory130can store commands or data related to at least one other component of the electronic device101. According to embodiments of this disclosure, the memory130can store software and/or a program140. The program140includes, for example, a kernel141, middleware143, an application programming interface (API)145, and/or an application program (or “application”)147. At least a portion of the kernel141, middleware143, or API145may be denoted an operating system (OS).

The kernel141can control or manage system resources (such as the bus110, processor120, or memory130) used to perform operations or functions implemented in other programs (such as the middleware143, API145, or application147). The kernel141provides an interface that allows the middleware143, the API145, or the application147to access the individual components of the electronic device101to control or manage the system resources. The application147may include one or more applications that, among other things, obtain and process input utterances, identify new concepts in the input utterances, and perform slot filling, retraining, or other functions as described in more detail below. These functions can be performed by a single application or by multiple applications that each carries out one or more of these functions.

The middleware143can function as a relay to allow the API145or the application147to communicate data with the kernel141, for instance. A plurality of applications147can be provided. The middleware143is able to control work requests received from the applications147, such as by allocating the priority of using the system resources of the electronic device101(like the bus110, the processor120, or the memory130) to at least one of the plurality of applications147. The API145is an interface allowing the application147to control functions provided from the kernel141or the middleware143. For example, the API145includes at least one interface or function (such as a command) for filing control, window control, image processing, or text control.

The I/O interface150serves as an interface that can, for example, transfer commands or data input from a user or other external devices to other component(s) of the electronic device101. The I/O interface150can also output commands or data received from other component(s) of the electronic device101to the user or the other external device.

The display160includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum-dot light emitting diode (QLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display. The display160can also be a depth-aware display, such as a multi-focal display. The display160is able to display, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display160can include a touchscreen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a body portion of the user.

The communication interface170, for example, is able to set up communication between the electronic device101and an external electronic device (such as a first electronic device102, a second electronic device104, or a server106). For example, the communication interface170can be connected with a network162or164through wireless or wired communication to communicate with the external electronic device. The communication interface170can be a wired or wireless transceiver or any other component for transmitting and receiving signals.

The electronic device101further includes one or more sensors180that can meter a physical quantity or detect an activation state of the electronic device101and convert metered or detected information into an electrical signal. For example, the sensor(s)180can include one or more cameras or other imaging sensors, which may be used to capture images of scenes. The sensor(s)180can also include one or more buttons for touch input, one or more microphones, a gesture sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor (such as a red green blue (RGB) sensor), a bio-physical sensor, a temperature sensor, a humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an ultrasound sensor, an iris sensor, or a fingerprint sensor. The sensor(s)180can further include an inertial measurement unit, which can include one or more accelerometers, gyroscopes, and other components. In addition, the sensor(s)180can include a control circuit for controlling at least one of the sensors included here. Any of these sensor(s)180can be located within the electronic device101.

The first external electronic device102or the second external electronic device104can be a wearable device or an electronic device-mountable wearable device (such as an HMD). When the electronic device101is mounted in the electronic device102(such as the HMD), the electronic device101can communicate with the electronic device102through the communication interface170. The electronic device101can be directly connected with the electronic device102to communicate with the electronic device102without involving with a separate network. The electronic device101can also be an augmented reality wearable device, such as eyeglasses, that include one or more cameras.

The first and second external electronic devices102and104and the server106each can be a device of the same or a different type from the electronic device101. According to certain embodiments of this disclosure, the server106includes a group of one or more servers. Also, according to certain embodiments of this disclosure, all or some of the operations executed on the electronic device101can be executed on another or multiple other electronic devices (such as the electronic devices102and104or server106). Further, according to certain embodiments of this disclosure, when the electronic device101should perform some function or service automatically or at a request, the electronic device101, instead of executing the function or service on its own or additionally, can request another device (such as electronic devices102and104or server106) to perform at least some functions associated therewith. The other electronic device (such as electronic devices102and104or server106) is able to execute the requested functions or additional functions and transfer a result of the execution to the electronic device101. The electronic device101can provide a requested function or service by processing the received result as it is or additionally. To that end, a cloud computing, distributed computing, or client-server computing technique may be used, for example. WhileFIG.1shows that the electronic device101includes the communication interface170to communicate with the external electronic device104or server106via the network162or164, the electronic device101may be independently operated without a separate communication function according to some embodiments of this disclosure.

The server106can include the same or similar components as the electronic device101(or a suitable subset thereof). The server106can support to drive the electronic device101by performing at least one of operations (or functions) implemented on the electronic device101. For example, the server106can include a processing module or processor that may support the processor120implemented in the electronic device101. In some embodiments of this disclosure, the server106may obtain and process input utterances, identify new concepts in the input utterances, and perform slot filling, retraining, or other functions as described in more detail below.

FIG.2illustrates an example technique200for learning new concepts from input utterances in accordance with this disclosure. For ease of explanation, the technique200ofFIG.2may be described as being used by the electronic device101ofFIG.1. However, the technique200may be used with any suitable device(s) and in any suitable system(s), such as when the technique200is used with the server106inFIG.1or in another system.

As shown inFIG.2, the technique200involves the use of a trained NLU base model202. The NLU base model202represents a machine learning model that has been trained to perform at least one NLU-related task. For example, the NLU base model202may be trained to perform slot filling and optionally intent detection operations. Slots refer to different parameters that may be included in input utterances being analyzed, and slot filling refers to identifying specific values for different slots based on the contents of the input utterances being analyzed. For instance, slots may refer to objects, locations, or times, and slot filling may be performed to identify the specific objects, locations, or times within the input utterances being analyzed. Intents refer to the overall meanings or purposes of the input utterances being analyzed. As a particular example of this, an input utterance of “reservation at Mario's Italiano for 1 pm” may be associated with a location slot (Mario's Italiano) and a time slot (1 pm), and the intent of this input utterance can be determined to be “reserve item.”

The NLU base model202represents any suitable type of machine learning model that has been trained to perform at least one NLU-related task. Example types of NLU base models202that could be used here can include recurrent neural network (RNN)-based models and bidirectional encoder representations from transformers (BERT)-based models. Note, however, that this disclosure is not limited to any particular type of machine learning model. Also note that the technique200shown here may involve the use of multiple NLU base models202if needed or desired.

When the NLU base model202is applied to an input utterance, the NLU base model202is used to obtain initial slot probability distributions204. Each initial slot probability distribution204represents a collection of probabilities that a specific word in the input utterance belongs to different slot types. For example, the NLU base model202might recognize nine different slot types, and each of the initial slot probability distributions204might represent nine probabilities for a specific word of the input utterance (where each probability is associated with a different one of the nine slot types). In typical NLU-based systems, a word would simply be assigned to a slot type based on the highest probability in its associated slot probability distribution204, and the word would be labeled to identify its assigned slot type. In some cases, the “IOB” format is used to label words in input utterances, where “B” and “I” respectively indicate a beginning word and an intermediate word of a slot and “O” indicates that a word does not belong to any slot. InFIG.2, however, the technique200processes the initial slot probability distributions204in order to identify any new concept in the input utterance that is not recognized by the NLU base model202.

A confidence calibration operation206processes the initial slot probability distributions204in order to generate calibrated slot probability distributions208. The confidence calibration operation206here can apply a calibration probability distribution to the initial slot probability distributions204in order to generate the calibrated slot probability distributions208. The calibration probability distribution has a higher number of dimensions than the initial slot probability distributions204. Because of this, possible slot labels for words of the input utterance are mapped from a lower-dimensional space associated with the initial slot probability distributions204to a higher-dimensional space associated with the calibration probability distribution. The use of the higher-dimensional space helps in learning the uncertainty of the slot probability distributions better so as to mitigate over-confidence of the NLU base model202.

In some embodiments, the calibration probability distribution applied by the confidence calibration operation206represents a Dirichlet distribution, which is often denoted as Dir(α). A Dirichlet distribution is a continuous multivariate probability distribution that is parameterized by a vector α of positive real numbers. Also, in some embodiments, the confidence calibration operation206itself may be implemented as a machine learning model, such as a prior RNN (which may be degenerated as a softmax-based RNN in sequence labeling). The confidence calibration operation206in these embodiments may be used to degenerate the RNN-based slot filling model or other NLU base model202for sequence uncertainty estimation. In some cases, the confidence calibration operation206is used to degenerate and replace a softmax layer, which is routinely used in the NLU base model202. Note that while the use of the Dirichlet distribution is often described in this patent document as being used to learn higher-dimensional distributions in order to replace a softmax function, any other suitable distribution may be used to learn higher-dimensional distributions, and the NLU base model202may or may not include a softmax function. For instance, histogram binning, Bayesian binning, or Gaussian generative model may be used by the confidence calibration operation206. Once a particular higher-dimensional distribution is selected for use, the parameters of the lower-dimensional distribution can be tied to that particular higher-dimensional distribution.

A word uncertainty score calculation operation210processes the calibrated slot probability distributions208(and possibly the initial slot probability distributions204) in order to identify top-ranked slot labels and uncertainty scores212for the associated words of the input utterance. The top-ranked slot labels represent slot labels having the highest probabilities (according to the calibrated slot probability distributions208and/or the initial slot probability distributions204) for the associated words of the input utterance. The uncertainty score for each top-ranked slot label represents how certain the overall system is regarding the identified slot label for the associated word. In some cases, higher uncertainty scores may represent less certainty (more uncertainty), and lower uncertainty scores may represent less uncertainty (more certainty). However, any other suitable scoring system may be used here. In general, the word uncertainty score calculation operation210identifies little or no uncertainty when an input utterance does not involve any new concepts and more uncertainty when an input utterance involves at least one new concept.

In some cases, the top-ranked slot labels for words having low uncertainty (such as word having uncertainty scores below a threshold value) can be identified based on the initial slot probability distributions204. For example, a softmax function or other function may be applied to the initial slot probability distribution204for any word of the input utterance that is determined to have a low uncertainty in order to identify that word's slot label. This allows slot labels to be obtained for any words of the input utterance that are likely to be interpreted correctly (since they have low uncertainty) using the initial slot probability distributions204.

The top-ranked slot labels and uncertainty scores212are provided to an uncertain concept extraction operation214, which uses the top-ranked slot labels and uncertainty scores212to identify a new concept (if any) in the input utterance being processed. For example, the uncertain concept extraction operation214may compare the uncertainty scores to a threshold value in order to determine whether any individual word in the input utterance has an excessive uncertainty. If an uncertainty score associated with a word exceeds the threshold value, the uncertain concept extraction operation214can determine that the input utterance contains a new concept, and the uncertain concept extraction operation214may identify that word and possibly a phrase containing that word as being related to the new concept. In this example, an output216of the uncertain concept extraction operation214represents the word or phrase associated with the new concept (if any).

The new concept or the identified word or phrase associated with the new concept may be used in any suitable manner. For example, the identified word or phrase may be labeled as having an unknown slot type (rather than incorrectly labeling the word or phrase with the wrong slot type due to over-confidence). The electronic device101may also prompt for clarification of the new concept. This may include the electronic device101asking a user to clarify the intent or meaning of the identified word or phrase or the input utterance, such as by asking the user one or more questions and receiving one or more answers from the user. This may also or alternatively include the electronic device101accessing an online database or other data source to identify a potential intent or meaning of the identified word or phrase or the input utterance. If information is obtained in response to the prompt, the electronic device101may perform slot filling for the identified word or phrase based on obtained information associated with the identified word or phrase.

In some embodiments, a dependency parser218may be used to generate a dependency tree220, which can be used by the uncertain concept extraction operation214. For example, the dependency parser218may analyze the grammatical structure of an input utterance in order to identify parts of speech and relationships between words in the input utterance. The dependency tree220breaks down an input utterance into different parts of speech and models the input utterance in a directed tree or directed graph format based on the relationships of the words. For instance, the dependency parser218may identify and the dependency tree220may represent any noun phrases (NPs), verb phrases (VPs), adjective phrases (ADJPs), adverb phrases (ADVPs), and/or prepositional phrases (PPs) in an input utterance. One or more words in each phrase can also be tagged with one or more labels, such as noun (NN), plural noun (NNS), verb (VB), coordinating conjunction (CD), or preposition or subordinating conjunction (IN). Any suitable dependency parser218may be used here to produce dependency trees220.

The dependency tree220for an input utterance may be provided to the uncertain concept extraction operation214for use during the identification of a word or phrase in the output216. For example, the uncertain concept extraction operation214may identify a word having an uncertainty above a threshold, and the uncertain concept extraction operation214can use the dependency tree220to determine whether that word is part of a larger phrase in the input utterance. If so, the uncertain concept extraction operation214may identify that entire phrase as being related to a new concept, rather than just the individual word.

In some embodiments, a multi-task calibration training operation222may be performed as part of the technique200in order to retrain the NLU base model202and/or to retrain the confidence calibration operation206. For example, after the technique200has identified at least one word or phrase related to at least one new concept, slot filling may occur for the word(s) or phrase(s) (such as based on prompting or other information). Using the correct slot filling labels for the word(s) or phrase(s) (or using other information), the multi-task calibration training operation222can retrain the NLU base model202during retraining224and/or retrain the confidence calibration operation206during retraining226to recognize the new concept(s). This allows the NLU base model202and/or the confidence calibration operation206to learn the new concept(s) and thereby more effectively process subsequent input utterances that include the same or similar concept(s).

In particular embodiments, the retraining may include minimizing the sequence loss associated with the NLU base model202and maximizing the entropy loss associated with the confidence calibration operation206. In some cases, the retraining of the confidence calibration operation206may enhance the robustness of uncertainty modeling by learning an adaptive calibration matrix that better separates existing (learned) and new (unlearned) concepts. Because of the retraining, subsequent input utterances related to the same or similar new concept may be processed more effectively, possibly without identifying any portion of the subsequent input utterances as being directed to any additional new concepts.

Again, note that the technique200shown inFIG.2may be performed using any suitable type(s) of NLU base model(s)202. Given the number of possible NLU model types, the technique200can therefore be applied effectively across a number of applications. Also, the technique200shown inFIG.2can effectively identify new concepts and optionally words and phrases associated with those new concepts so that the intents or meanings of those new concepts can be identified, which supports effective learning by an overall system. In addition, the technique200shown inFIG.2may have negligible computational overhead (relative to the computational load already associated with use of an NLU model) and therefore can be easily implemented in any suitable device or system.

Additional details regarding various operations in the technique200are provided below. Note that the operations and functions described above with reference toFIG.2can be implemented in an electronic device101,102,104, server106, or other device in any suitable manner. For example, in some embodiments, the operations and functions described above can be implemented or supported using one or more software applications or other software instructions that are executed by at least one processor120of a device. In other embodiments, at least some of the operations and functions described above can be implemented or supported using dedicated hardware components. In general, the operations and functions described above can be performed using any suitable hardware or any suitable combination of hardware and software/firmware instructions.

AlthoughFIG.2illustrates one example of a technique200for learning new concepts from input utterances, various changes may be made toFIG.2. For example, each operation206,210,214,222may occur any number of times as needed or desired in order to process input utterances and perform retraining. As a particular example, the operations206,210,214may occur repeatedly for each input utterance received and processed. If and when at least one new concept is identified a threshold number of times, the multi-task calibration training operation222may conduct the retraining224and/or the retraining226for the new concept(s). The threshold number can have any suitable value, including a value of one.

FIG.3illustrates an example application of the technique200for learning new concepts from input utterances in accordance with this disclosure. In particular,FIG.3illustrates how the technique200might operate in response to receiving a particular input utterance. Note, however, that this input utterance is for illustration only and does not limit this disclosure to any particular type of input utterance.

As shown inFIG.3, the NLU base model202is applied to an input utterance302of “reservation at Mario's Italiano for 1 pm.” The NLU base model202processes the input utterance302and generates the initial slot probability distributions204for the words of the input utterance302. Each initial slot probability distribution204can have a number of probabilities equal to the number of possible slot types. As a particular example, each initial slot probability distribution204may have nine probabilities that identify the likelihoods of the associated word belonging to nine slot types. The nine slot types might include B-restaurant, I-restaurant, B-time, I-time, B-game, I-game, B-country, I-country, and O. Of course, the number of slot types and the specific slot types can vary based on a number of factors, such as based on the domain associated with an input utterance.

Assume in this example that “Mario's Italiano” is the name of a new restaurant or is the name of a restaurant for which the NLU base model202was not trained to recognize. Thus, “Mario's Italiano” is a new concept from the perspective of the NLU base model202. Because of this, incorrect slot labels will likely be identified for the words “Mario's” and “Italiano.” For instance, the outputs of the NLU base model202might indicate a high probability that the word “Mario's” is associated with a game (such as the B-game slot type) and a high probability that the word “Italiano” is associated with a country (such as the B-country slot type). Both of those determinations would be incorrect, which is illustrated in predicted slot labels304associated with the initial slot probability distributions204(where the two incorrect slot labels are shown in strikethrough).

The confidence calibration operation206applies a calibration probability distribution (such as a Dirichlet distribution in this example) to the initial slot probability distributions204in order to generate the calibrated slot probability distributions208. As noted above, the calibration probability distribution has a higher dimensionality compared to the dimensions of the initial slot probability distributions204. This remaps the probability values into a higher-dimensional space where there is less over-confidence.

The resulting calibrated slot probability distributions208are provided to the word uncertainty score calculation operation210, which calculates the top-ranked slot labels and uncertainty scores212for the associated words in the input utterance302. For example, the word uncertainty score calculation operation210may calculate the entropy of each calibrated slot probability distribution208to identify the uncertainty score for the word associated with that calibrated slot probability distribution208. Here, the uncertainty scores are shown along with indicators identifying whether the uncertainty scores are below (L) or above (H) a threshold value. As can be seen inFIG.3, the uncertainty score associated with the word “Mario's” is above the threshold value, while the uncertainty score associated with the word “Italiano” is also high but is not necessarily above the threshold value. As noted above, in some embodiments, the slot labels for words having low uncertainty scores can be determined by applying a softmax function or other function to the initial slot probability distributions204of those words.

The uncertain concept extraction operation214receives the top-ranked slot labels and uncertainty scores212. The uncertain concept extraction operation214may also receive a dependency tree220for the input utterance302. The uncertain concept extraction operation214can determine that the uncertainty score associated with the word “Mario's” exceeds the threshold value, so the uncertain concept extraction operation214can determine that the input utterance302contains at least one new concept not recognized by the NLU base model202. The uncertain concept extraction operation214can also determine that the word “Italiano” is related to the word “Mario's,” such as by using the dependency tree220to determine that both words are contained in the same phrase. The uncertain concept extraction operation214can therefore identify the phrase “Mario's Italiano” as being related to the new concept, and the uncertain concept extraction operation214might output this phrase. Based on this, revised slot labels306might be produced that replace the labels for these two words in the slot labels304with “unknown” labels. If desired, prompting may occur to obtain information associated with this phrase (such as from a user or online data source) in order to identify correct slot labels for these words. As described above, however, any other or additional actions may occur based on the identification of a new concept in an input utterance.

AlthoughFIG.3illustrates one example of an application of the technique200for learning new concepts from input utterances, various changes may be made toFIG.3. For example, the specific input utterance, dependency tree, and numerical values shown inFIG.3are for illustration only and can vary widely based on the use of the technique200and how the NLU base model202is trained.

FIG.4illustrates an example confidence calibration operation206in the technique200for learning new concepts from input utterances in accordance with this disclosure. For ease of explanation, the confidence calibration operation206ofFIG.4may be described as being used by the electronic device101ofFIG.1. However, the confidence calibration operation206may be used with any suitable device(s) and in any suitable system(s), such as when the confidence calibration operation206is used with the server106inFIG.1or in another system.

As shown inFIG.4, for a word402(denoted x) of an input utterance, a predicted slot label404(denoted y) can be determined. A variable406(denoted z) represents the latent variable that is used to introduce a higher number of dimensions in order to calibrate the confidence for the predicted slot label404. Parameters408(denoted θ) represent the parameters of the model being used to introduce the higher number of dimensions (such as when the confidence calibration operation206is implemented as a machine learning model). The notation p(a|b) refers to the probability of a given b, and the notation p(c) refers to the probability of c. In the following discussion, ω is used to refer to the prediction confidence, meaning confidence in the prediction that word x has label y.

In some embodiments, for the word x in the input utterance, the confidence predicted by the NLU base model202can be expressed as P(ω|x, D), where D represents the training data used to train the NLU base model202. This confidence can be decomposed as follows:

P⁡(ω|x,D)=∫P⁡(ω|x,θ)︸Data⁢p⁡(θ|D)︸Model⁢d⁢⁢θ(1)
In Equation (1), the data uncertainty is described by the label-level posterior P(ω|x, θ), and the model uncertainty is described by the model-level posterior p(0θ|D). Equation (1) may be further decomposed as follows:

P⁡(ω|x,D)=∫P⁡(ω|μ)︸Data⁢p⁡(μ|x,θ)⁢p⁡(θ|D)︸Model⁢d⁢⁢μdθ≈∫P⁡(ω|μ)︸Data⁢p⁡(μ|x,θ^)︸Model⁢d⁢⁢μ(2)where:⁢p⁡(μ|x,θ)=∏i=1n⁢P⁡(μi|μ1,⁢…,ui-1,x;θ^)⁢P⁡(ωt|μ)=P⁡(ωi|μt)⁢(3)
Here, μ is a categorical distribution over a simplexkwith μ=[μ1, . . . , μK]=[P(y=ω(1), . . . , P(y=ω(K))]T. Thus, μ is the predicted distribution of all possible K labels for the word x, and t represents each word in the input utterance. As can be seen here, each ωtis only dependent on μt.

In some embodiments, the confidence calibration operation206can apply a calibration probability distribution in the form of a Dirichlet distribution to the initial slot probability distributions204in order to generate calibrated slot probability distributions208. A probability density function (PDF) of the Dirichlet prior distribution over all possible values of the K-dimensional categorical distribution μ may be expressed as:

B⁡(α)=∏iK⁢Γ⁡(α⁡(i))Γ⁡(∑ik⁢α⁡(i))(5)
In particular embodiments, the confidence calibration operation206can be implemented using a Dirichlet prior RNN that is implemented by a neural network function with parameters θ. The neural network function operates to generate n K-dimensional vectors (where αt∈k) at each timestamp t such that:
α=(α1, . . . ,αn)=g(x∪z,θ)  (6)
in which there is:
Dir(αt)=P(μt|μ1, . . . ,μt-1,z;{circumflex over (θ)})  (7)
A degeneration of the Dirichlet prior RNN may be defined so that traditional RNN training can be used to train the confidence calibration operation206. Here, posterior over labels at each timestamp t can be identified using the Dirichlet distribution as follows:

E(x,y)∈D[1n⁢∑j=1K⁢∑t=1n⁢yt⁡(j)⁢⁢log⁡(αt⁡(j)αt⁡(0))]⁢(9)
where E(x, y)represents a measure of cross-entropy. If mtdenotes the last layer output of each word t in the NLU base model202, it can be shown that maximizing LDiris equivalent to maximizing sequence cross-entropy loss in the NLU base model202if the following condition is set:
αt=Memt=Mexp(BaseNLU(xt;θ))  (10)
where M is a scale constant.

In some embodiments, the word uncertainty score calculation operation210can compute uncertainty scores for words in an input utterance by calculating the entropies of the calibrated slot probability distributions208produced by the confidence calibration operation206. Based on the definition of one possible implementation of the confidence calibration operation206described above, one possible implementation of the word uncertainty score calculation operation210can calculate the uncertainty score for a specific word that might be classified into one of k classes (one of K slot labels) as follows:

AlthoughFIG.4illustrates one example of the confidence calibration operation206in the technique200for learning new concepts from input utterances, various changes may be made toFIG.4. For example, the description ofFIG.4has provided the mathematical basis for using a Dirichlet distribution in order to increase the dimensionality of initial slot probability distributions204and generate calibrated slot probability distributions208. Other possible implementations of the confidence calibration operation206may be used, and those implementations may or may not rely on a Dirichlet distribution in this way. This can also have an impact on how the word uncertainty score calculation operation210calculate the uncertainty scores.

FIG.5illustrates an example multi-task calibration training operation222in the technique200for learning new concepts from input utterances in accordance with this disclosure. For ease of explanation, the multi-task calibration training operation222ofFIG.5may be described as being used by the electronic device101ofFIG.1. However, the multi-task calibration training operation222may be used with any suitable device(s) and in any suitable system(s), such as when the multi-task calibration training operation222is used with the server106inFIG.1or in another system.

In the example ofFIG.5, the NLU base model202is shown as including a number of layers502, where each layer can produce an output. This may be common, for example, when the NLU base model202is implemented using an RNN or other similar type of machine learning model. Note, however, that other structures for the NLU base model202may be used here.

As described above, the multi-task calibration training operation222may engage in retraining224of the NLU base model202and/or retraining226of the confidence calibration operation206(such as retraining of the machine learning model used by the confidence calibration operation206). The retraining224of the NLU base model202may involve attempting to minimize the low-dimensional sequence loss (expressed ascalibration) of the NLU base model202. The retraining226of the confidence calibration operation206may involve attempting to maximize the high-dimensional entropy loss (expressed ascalibration) of the outputs produced by the confidence calibration operation206. Performing these retraining operations can help make the confidence calibration operation206more clearly distinguish between known or “in distribution” (IND) concepts (concepts previously learned by the NLU base model202) and new or “out of distribution” (00D) concepts (concepts not previously learned by the NLU base model202).

In some embodiments, the traditional sequence loss in the NLU base model202can be minimized according to the following expression:

min⁢⁢ℒcalibration=𝔼(z,y)~IND⁢∑i=1S⁢∑t=1n⁢yt⁢⁢log⁢⁢Pt⁡(i)(13)
Here, Pt(i) is a prediction and is computed based on the design of the model used in the confidence calibration operation206. When the confidence calibration operation206is implemented using a Dirichlet prior RNN, Pt(i) may be expressed as:

In some embodiments, the entropy of the confidence calibration operation206can be maximized according to the following expression:

AlthoughFIG.5illustrates one example of a multi-task calibration training operation222in the technique200for learning new concepts from input utterances, various changes may be made toFIG.5. For example, the description ofFIG.5has provided the mathematical basis for retraining an NLU base model202and/or a confidence calibration operation206when the confidence calibration operation206supports use of a Dirichlet distribution. Other possible implementations of the confidence calibration operation206may be used (which may or may not rely on a Dirichlet distribution), so other retraining of the NLU base model202and/or the confidence calibration operation206may be used.

FIG.6illustrates example results obtained using the technique for learning new concepts from input utterances in accordance with this disclosure. In particular,FIG.6illustrates how a phrase of an input utterance may be identified as being related to a new concept. In this particular example, the input utterance is “What is the height of the Eiffel Tower?” A dependency tree600shown inFIG.6breaks down the input utterance while omitting two instances of “the” from the input utterance. The dependency tree600here can be generated using the dependency parser218, which allows the system to obtain the syntax of the input utterance.

The input utterance can be processed as described above in order to produce calibrated slot probability distributions208, where the calibrated slot probability distributions208are associated with the words of the input utterance shown inFIG.6. The word uncertainty score calculation operation210can identify top-ranked slot labels and uncertainty scores212for the words using their calibrated slot probability distributions208. For any word having an uncertainty score above a threshold, the uncertain concept extraction operation214might mark that word as having an “unknown” slot type.

The uncertain concept extraction operation214can also traverse the dependency tree600in order to identify any words related to the word or words having uncertainty scores above the threshold, and the uncertain concept extraction operation214can combine those words into a phrase associated with a new concept. For example, the uncertain concept extraction operation214might identify a word having an uncertainty score above the threshold and mark that word as having an unknown slot type. The uncertain concept extraction operation214can also determine that the nearest noun (N) or noun phrase (NP) root containing that word also contains one or more additional words. In response, the uncertain concept extraction operation214might mark each additional word as having an unknown slot type, and the uncertain concept extraction operation214can combine all of these words into a phrase associated with a new concept.

As one example of this, the uncertain concept extraction operation214might determine that the word “Eiffel” has an excessive uncertainty score and mark it as having an unknown slot type. The uncertain concept extraction operation214can also traverse the dependency tree600from that word and identify the nearest noun or noun phrase root containing the word “Eiffel.” Here, the uncertain concept extraction operation214would determine that the identified noun or noun phrase also contains the word “Tower,” so the uncertain concept extraction operation214can mark that word as having an unknown slot type, as well. In this way, the uncertain concept extraction operation214identifies a phrase602(“Eiffel Tower”) as having unknown slot types and as being associated with a new concept.

As another example of this, the uncertain concept extraction operation214might determine that the word “Eiffel” has an excessive uncertainty score and mark it as having an unknown slot type. The uncertain concept extraction operation214can also traverse the dependency tree600from that word and identify a higher noun or noun phrase root containing the word “Eiffel.” Here, the uncertain concept extraction operation214could determine that the identified noun or noun phrase also contains the words “height,” “of,” and “Tower,” so the uncertain concept extraction operation214can mark those words as having unknown slot types, as well. In this way, the uncertain concept extraction operation214identifies a phrase604(“height of Eiffel Tower”) as having unknown slot types and as being associated with a new concept.

AlthoughFIG.6illustrates one example of results obtained using the technique for learning new concepts from input utterances, various changes may be made toFIG.6. For example, the specific input utterance, dependency tree, and words identified as having an unknown slot type are for illustration only and can vary widely based on the use of the technique200and how the NLU base model202is trained.

As described above, possible prompting mechanisms that can be used to obtain information about a new concept can include (i) asking one or more questions to a user and receiving one or more responses related to the new concept and/or (ii) accessing an online or other data source for information related to the new concept. The following provides several example use cases of how these prompting mechanisms might be used to obtain information related to a new concept. Note, however, that these are example use cases only and do not limit the scope of this disclosure.

As a first example, assume a user provides an input utterance of “Show me game night in SPOTIFY.” A voice-enabled personal assistant or other system may determine that “game night” is a new concept using the approach above, so the voice-enabled personal assistant or other system may respond with “What is game night?” The user may answer with “Game night is the name of a playlist.” At that point, the voice-enabled personal assistant or other system can determine the intent of the initial utterance and start playing the “Game night” playlist, such as in a SPOTIFY app on the user's device.

As a second example, assume a user provides an input utterance of “Reserve at Mario's Italiano for 1 pm.” A voice-enabled personal assistant or other system may determine that “Mario's Italiano” is a new concept using the approach above, so the voice-enabled personal assistant or other system may access an online data source and learn that a new restaurant is located nearby named “Mario's Italiano.” At that point, the voice-enabled personal assistant or other system can automatically make a reservation at Mario's Italiano for 1 pm.

As a third example, assume a user provides an input utterance of “Find a subway and how to get there using subway.” A voice-enabled personal assistant or other system may determine that both instances of “subway” are new concepts using the approach above, so the voice-enabled personal assistant or other system may respond with “Which subway do you want to find?” The user may answer with “SUBWAY restaurant.” The voice-enabled personal assistant or other system may then respond with “Got it. Which subway to do want to use?” The user may answer with “Subway transportation.” At that point, the voice-enabled personal assistant or other system can determine the intent of the initial utterance and respond with “OK. Here is the nearest SUBWAY restaurant. You can get there via subway line 1.”

Again, these use cases are for illustration only. In general, the technique200described above may be used in conjunction with any suitable approach that allows a system to obtain additional information related to a new concept identified using the technique200. Depending on the implementation, this may allow for continuously learning of new concepts in order to, for instance, better understand how user interests change over time or to provide better personalized services (such as personalized NLU-based services, personalized recommendations, or targeted advertisements).

FIG.7illustrates an example method700for learning new concepts from input utterances in accordance with this disclosure. For ease of explanation, the method700ofFIG.7may be described as being performed by the electronic device101ofFIG.1, which may use the technique200ofFIG.2. However, the method700may be performed using any suitable device(s) and in any other suitable system(s), and the method700may be performed as part of any suitable NLU-based algorithm.

As shown inFIG.7, an input utterance is obtained at step702. This may include, for example, the processor120of the electronic device101receiving the input utterance from a user or other source. The input utterance may be obtained in any suitable manner. In some instances, a user may type the input utterance into the electronic device101or speak the input utterance to the electronic device101(which can perform speech-to-text recognition). An NLU model is applied to the input utterance in order to obtain initial slot probability distributions at step704. This may include, for example, the processor120of the electronic device101applying the NLU base model202to the input utterance and obtaining initial slot probability distributions204for words in the input utterance. The initial slot probability distributions204can identify the likelihoods of different words in the input utterance belonging to different slot types.

Confidence calibration is performed by applying a calibration probability distribution to the initial slot probability distributions in order to generate calibrated slot probability distributions at step706. This may include, for example, the processor120of the electronic device101performing the confidence calibration operation206and generating calibrated slot probability distributions208. The calibration probability distribution has a higher number of dimensions than the initial slot probability distributions204. In some embodiments, the calibration probability distribution is a Dirichlet distribution.

Uncertainties associated with words in the input utterance are identified based on the calibrated slot probability distributions at step708. This may include, for example, the processor120of the electronic device101performing the word uncertainty score calculation operation210and generating top-ranked slot labels and uncertainty scores212for words of the input utterance. In some embodiments, the uncertainties are identified by calculating entropies of the calibrated slot probability distributions208associated with the words in the input utterance. A new concept contained in the input utterance that is not recognized by the NLU model may be identified based on the identified uncertainties at step710. This may include, for example, the processor120of the electronic device101performing the uncertain concept extraction operation214and determining whether any of the identified uncertainties exceeds a threshold value.

If a new concept is identified, any number of operations may occur involving or associated with the new concept. For instance, a word or phrase in the input utterance associated with the new concept may be identified and labeled as having an unknown slot type at step712. This may include, for example, the processor120of the electronic device101labeling each word of the input utterance with an uncertainty score exceeding the threshold as having an unknown slot type. This may also include the processor120of the electronic device101traversing a dependency tree220associated with the input utterance to identify whether any other words of the input utterance are in the same phrase as a word marked as having an unknown slot type and, if so, marking each of the other words as having an unknown slot type.

A prompt for clarification of the new concept may be made at step714, and a clarification identifying the meaning or intent associated with the new concept is received at step716. This may include, for example, the processor120of the electronic device101providing one or more questions to a user, where the one or more questions ask the user to provide information about the new concept. This may also include the processor120of the electronic device101receiving one or more answers from the user. This may further include the processor120of the electronic device101accessing one or more online data sources in an attempt to identify additional information about the new concept.

Slot filling for the identified word or phrase related to the new concept can be performed based on the clarification at step718. This may include, for example, the processor120of the electronic device101identifying one or more slot labels for the word or phrase in the input utterance associated with the new concept based on the received information. At least one action associated with the input utterance may be performed at step720. This may include, for example, the processor120of the electronic device101identifying at least one desired action to be performed as instructed by the input utterance based on the clarification or the slot filling. Retraining of the NLU model and/or the confidence calibration may be performed at step722. This may include, for example, the processor120of the electronic device101performing the multi-task calibration training operation222to retrain the NLU base model202and/or the confidence calibration operation206based at least partially on the clarification. As a particular example, this may include the processor120of the electronic device101performing the multi-task calibration training operation222to minimize sequence loss in the NLU base model202and to maximize entropy loss based on the calibrated slot probability distributions208generated by the confidence calibration operation206.

AlthoughFIG.7illustrates one example of a method700for learning new concepts from input utterances, various changes may be made toFIG.7. For example, while shown as a series of steps, various steps inFIG.7may overlap, occur in parallel, occur in a different order, or occur any number of times. Also, various steps inFIG.7might be omitted, such as when one or some of the steps712-722are omitted following the identification of a new concept in an input utterance.