System and method for natural language processing

Systems and methods are provided for natural language processing. An exemplary method implementable by a server may comprise: obtaining, from a computing device, an audio input and a current interface, wherein the current interface is associated with a context; and determining a query associated with the audio input based at least on the audio input and the context of the current interface.

FIELD OF THE INVENTION

This disclosure generally relates to methods and devices for natural language processing in human-machine interaction.

BACKGROUND

Advances in human-machine interactions allow people to use their voices to effectuate control. For example, traditional instruction inputs via keyboard, mouse, or touch screen can be achieved with speeches. Nevertheless, many hurdles are yet to be overcome to streamline the process.

SUMMARY

Various embodiments of the present disclosure can include systems, methods, and non-transitory computer readable media configured to process natural language process. According to one aspect, a method for natural language processing, implementable by a server, may comprise: obtaining, from a computing device, an audio input and a current interface, wherein the current interface is associated with a context; and determining a query associated with the audio input based at least on the audio input and the context of the current interface.

In some embodiments, the computing device is configured to provide a plurality of inter-switchable interfaces, the plurality of interfaces comprise at least one of: an interface associated with navigation, an interface associated with media, or an interface associated with messaging, the context of the current interface comprises a first context and a second context, the first context comprises at least one of: the current interface as navigation, the current interface as media, or the current interface as messaging, and the second context comprises at least one of: an active route, a location, an active media session, or an active message.

In some embodiments, determining the query associated with the audio input based at least on the audio input and the context of the current interface comprises: feeding the audio input to an voice recognition engine to determine raw texts corresponding to the audio input, and feeding the raw texts and the context of the current interface to a natural language processing engine to determine the query associated with the audio input.

In some embodiments, feeding the raw texts and the context of the current interface to the natural language processing engine to determine the query associated with the audio input comprises: pre-processing the raw texts based on at least one of: lemmatizing, spell-checking, singularizing, or sentiment analysis to obtain pre-processed texts; matching the pre-processed texts against preset patterns; in response to not detecting any preset pattern matching the pre-processed texts, tokenizing the texts; and vectorizing the tokenized texts to obtain vectorized texts.

In some embodiments, feeding the raw texts and the context of the current interface to the natural language processing engine to determine the query associated with the audio input further comprises: dynamically updating one or more weights associated with one or more first machine learning models at least based on the first context; and applying the one or more first machine learning models to the first context and at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts, to obtain an intent classification of the audio input.

In some embodiments, applying the one or more first machine learning models to obtain the intent classification of the audio input comprises: applying a decision-tree-based model and a feedforward neural network model each to the first context and to the at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts to obtain corresponding output classifications; in response to determining that an output classification from the decision-tree-based model is the same as an output classification from the feedforward neural network model, using the either output classification as the intent classification of the audio input; and in response to determining that the output classification from the decision-tree-based model is different from the output classification from the feedforward neural network model, applying a directed acyclic graph-support vector machine (DAGSVM) model to the corresponding at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts to obtain the intent classification of the audio input.

In some embodiments, feeding the raw texts and the context of the current interface to the natural language processing engine to determine the query associated with the audio input further comprises: applying one or more second machine learning models to the second context and at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts to obtain a sub-classification prediction distribution of the audio input, the one or more second machine learning models comprising at least one of: a naive bayes model, a term frequency-inverse document frequency model, a N-gram model, a recurrent neural network model, or a feedforward neural network model; and comparing the sub-classification prediction distribution with a preset threshold and against an intent database to obtain a sub-classification of the audio input, wherein the sub-classification corresponds to a prediction distribution exceeding the preset threshold and matches an intent in the intent database.

In some embodiments, feeding the raw texts and the context of the current interface to the natural language processing engine to determine the query associated with the audio input further comprises: in response to multiple prediction distributions exceeding the preset threshold, determining that the audio input corresponds to multiple intents and applying a neural network model to divide the at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts correspondingly according to the multiple intents; and for each of the divided texts, applying the N-gram model to obtain the corresponding intent sub-classification.

In some embodiments, feeding the raw texts and the context of the current interface to the natural language processing engine to determine the query associated with the audio input further comprises: in response to determining that the intent classification and the intent sub-classification are consistent, extracting one or more entities from the tokenized texts; and in response to determining that the intent classification and the intent sub-classification are inconsistent, re-applying the one or more first machine learning models without the context of the current interface to the at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts to update the intent classification of the audio input.

In some embodiments, feeding the raw texts and the context of the current interface to the natural language processing engine to determine the query associated with the audio input further comprises: identifying one or more entities from the tokenized text based on at least one of the intent classification, the intent sub-classification, or the second context; determining contents associated with the one or more entities based on at least one of public data or personal data; and determining the query as an intent corresponding to at least one of the intent classification or the intent sub-classification, in association with the determined one or more entities and the determined contents.

According to another aspect, a system for natural language processing, implementable on a server, may comprise a processor and a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the system to perform a method. The method may comprise: obtaining, from a computing device, an audio input and a current interface, wherein the current interface is associated with a context; and determining a query associated with the audio input based at least on the audio input and the context of the current interface.

According to another aspect, a method for natural language processing may comprise: obtaining an audio input from a computing device, wherein the audio is inputted to the computing device when a first interface of the computing device is active, determining a context of the first interface, the first interface comprising an interface associated with media, an interface associated with navigation, or an interface associated with messaging, feeding the audio input and the context of the first interface to one or more algorithms to determine an audio instruction associated with the audio input, and transmitting a computing device instruction to the computing device based on the determined audio instruction, causing the computing device to execute the computing device instruction.

In some embodiments, transmitting the computing device instruction to the computing device based on the determined audio instruction, causing the computing device to execute the computing device instruction comprises: in response to determining that the audio instruction is empty, generating a first dialog based on the context of the first interface, causing the computing device to play the first dialog; in response to determining that the audio instruction comprises an entity, extracting the entity, and generating a second dialog based on the extracted entity, causing the computing device to play the second dialog; in response to determining that the audio instruction comprises a response, matching the response with a response database, and in response to detecting a matched response in the response database, causing the computing device to execute the matched response; and in response to determining that the audio instruction comprises a query, matching the query with a query database, and in response to detecting no matched query in the query database, feeding the audio input and the context of the first interface to the one or more of algorithms to determine an audio instruction associated with the query.

DETAILED DESCRIPTION

Voice control can readily replace traditional control methods such as touch control or button control when they are impractical or inconvenient. For example, a vehicle driver complying with safety rules may be unable to divert much attention to his mobile phone, nor to operate on its touch screen. In such situations, voice control can help effectuate the control without any physical or visual contact with the device. Enabled by voice control, the device can also play specific contents according to an instruction spoken by the user.

Voice control applications require high accuracies. In the driver's example, erroneous interpretations of the voice input may cause frustration and terrible user experience, or even accidents. So far, it has been challenging to achieve accurate machine comprehension of human voice inputs.

The disclosed systems and methods can at least improve the accuracy of understanding human voice inputs, that is, the accuracy of processing natural language. Various embodiments of the present disclosure can include systems, methods, and non-transitory computer readable media configured to process natural language process. Example methods can leverage context information of graphic user interface (GUI) and user-machine interactions to supplement natural language processing and improve the performance of user intention interpretation. By considering the context of the current interface, the system can dynamically adjust the weights of classification classes associated with the user's intentions, thus better interpret user's audio input and reduce the needs for further clarification from the user. Further, the methods can help generate appropriate dialogs based on the context, prioritize the active sessions, and offer personalized recommendations.

According to one aspect, a method for natural language processing, implementable by a server, may comprise: obtaining, from a computing device, an audio input and a current interface, wherein the current interface is associated with a context; and determining a query associated with the audio input based at least on the audio input and the context of the current interface.

According to another aspect, a method for natural language processing may comprise: obtaining an audio input from a computing device, wherein the audio is inputted to the computing device when a first interface of the computing device is active, determining a context of the first interface, the first interface comprising an interface associated with media, an interface associated with navigation, or an interface associated with messaging, feeding the audio input and the context of the first interface to one or more algorithms to determine an audio instruction associated with the audio input, and transmitting a computing device instruction to the computing device based on the determined audio instruction, causing the computing device to execute the computing device instruction.

FIG. 1illustrates an example environment100for processing natural language, in accordance with various embodiments. As shown inFIG. 1, the example environment100can comprise at least one computing system102that includes one or more processors104and memory106. The memory106may be non-transitory and computer-readable. The memory106may store instructions that, when executed by the one or more processors104, cause the one or more processors104to perform various operations described herein. The instructions may comprise various algorithms, models, and databases described herein. Alternatively, the algorithms, models, and databases may be stored remotely (e.g., on a cloud server) and accessible to the system102. The system102may be implemented on or as various devices such as mobile phone, tablet, server, computer, wearable device (smart watch), etc. The system102above may be installed with appropriate software (e.g., platform program, etc.) and/or hardware (e.g., wires, wireless connections, etc.) to access other devices of the environment100.

The environment100may include one or more data stores (e.g., a data store108) and one or more computing devices (e.g., a computing device109) that are accessible to the system102. In some embodiments, the system102may be configured to obtain data (e.g., music album, podcast, audio book, radio, map data, email server data) from the data store108(e.g., a third-party database) and/or the computing device109(e.g., a third-party computer, a third-party server). The map data may comprise GPS (Global Positioning System) coordinates of various locations.

The environment100may further include one or more computing devices (e.g., computing devices110and111) coupled to the system102. The computing devices110and111may comprise devices such as mobile phone, tablet, computer, wearable device (e.g., smart watch, smart headphone), home appliances (e.g., smart fridge, smart speaker, smart alarm, smart door, smart thermostat, smart personal assistant), robot (e.g., floor cleaning robot), etc. The computing devices110and111may each comprise a microphone or an alternative component configured to capture audio inputs. For example, the computing device110may comprise a microphone115configured to capture audio inputs. The computing devices110and111may transmit or receive data to or from the system102.

In some embodiments, although the system102and the computing device109are shown as single components in this figure, it is appreciated that the system102and the computing device109can be implemented as single devices, multiple devices coupled together, or an integrated device. The data store(s) may be anywhere accessible to the system102, for example, in the memory106, in the computing device109, in another device (e.g., network storage device) coupled to the system102, or another storage location (e.g., cloud-based storage system, network file system, etc.), etc. The system102may be implemented as a single system or multiple systems coupled to each other. In general, the system102, the computing device109, the data store108, and the computing device110and111may be able to communicate with one another through one or more wired or wireless networks (e.g., the Internet, Bluetooth, radio) through which data can be communicated. Various aspects of the environment100are described below in reference toFIG. 2toFIG. 5.

FIG. 2illustrates an example system200for processing natural language, in accordance with various embodiments. The operations shown inFIG. 2and presented below are intended to be illustrative. In various embodiments, the system102may obtain an audio input from a computing device, wherein the audio is inputted to the computing device when a first interface of the computing device is active; determine a context of the first interface, the first interface comprising an interface associated with media, an interface associated with navigation, or an interface associated with messaging; feed the audio input and the context of the first interface to one or more algorithms to determine an audio instruction associated with the audio input; and transmit a computing device instruction to the computing device based on the determined audio instruction, causing the computing device to execute the computing device instruction. Each step is described in further details below.

In some embodiments, the system102may obtain data202from the data store108and/or the computing devices109, and obtain audio204and information206from the computing devices110. The data202may be obtained in advance to, contemporaneous with, or after the audio204. The information206may be obtained in conjunction with or after the audio204. The audio204may comprise an audio input, and the information206may comprise a current interface of the computing device110. The data202may comprise public data (e.g., music albums, artists, audio books, radio, map data, locations of points-of-interest, operating hours of points-of-interest, etc.) and personal data (e.g., personal music albums, personal podcasts, personal audio books, personal radio, personal playlists (possibly created on a third-party software platform), personal media player references, personal map data, personal routes, personal locations, personal messages such as text messages or emails). The personal data may also include personal preferences (e.g., favorite music, saved locations, contacts) and histories (e.g., played music, past navigations, searched locations, message history). The public data may be stored in a public database106cof the memory106. The personal data may be stored in a personal database106dof the memory106. Although shown as separate databases, the public and personal databases may alternatively be integrated together.

In some embodiments, the system102may obtain, from a computing device (e.g., the computing device110), an audio input (e.g., the audio204) and a current interface (e.g., as a part of the information206), wherein the current interface is associated with a context. For example, a user may speak within a detection range of the microphone115, such that an audio input (e.g., “find me a coffee shop near ABC University,” “play my most recent playlist”) is captured by the computing device110. The system102may obtain from the computer device110the audio input and the current interface.

Referring toFIG. 3Awhich illustrates example interfaces of the computing device110. In some embodiments, the computing device is configured to provide a plurality of inter-switchable interfaces. The switching can be achieved, for example, by swiping on a touch screen or by voice control. The plurality of interfaces may comprise at least one of: an interface associated with navigation (e.g., a current interface312), an interface associated with media (e.g., other interface316), or an interface associated with messaging (e.g., other interface314). The current interface may be a currently active or selected interface on the computing device. For example when the interface312is currently active, the interface314and316are inactive. The audio input may be (but not necessarily) captured at the current interface. If the interface has switched several times as the user speaks to the microphone, the current interface obtained by the system102may be preset to a certain (e.g., the last) interface during the span of the audio input. In one example, a user may have triggered a “microphone trigger” associated with the current interface312to capture the audio input. In another example, the user may have triggered a generic button on the computing device to capture the audio input. In another example, the microphone may continuously capture audio, and upon detecting a keyword, the computing device may obtain the audio input following the keyword. In yet another example, the microphone may start capturing the audio after any interface becomes current.

Still referring toFIG. 3A, in some embodiments, the context of the current interface may comprise a first context and a second context. The first context may comprise at least one of: the current interface as navigation, the current interface as media, or the current interface as messaging. That is, the first context may provide an indication of the main category or theme of the current interface. The second context may comprise at least one of: an active route, a location (e.g., a current location of the computing device), an active media session, or an active message. The active route may comprise a selected route for navigation. The location may comprise a current location of the computing device, any location on a map, etc. The active media session may comprise a current media (such as music, podcast, radio, audio book) on the media interface. The active message may comprise any message on the messaging interface. The context of the current interface may comprise many other types of information. For example, if the current interface312is navigation, the context of the current interface may comprise an indication that the current interface is navigation, an active route, a location, etc. The current interface312inFIG. 3Ashows four current locations (home, work, gym, and beach chalet), which may be included in the second context.

Referring back toFIG. 2, the system102may determine an audio instruction associated with the audio input based at least on the audio input and the context of the current interface. The audio instruction may refer to the instruction carried in the audio input, which may comprise one or more of: an entity, a response, a query, etc. The system102may further transmit a computing device instruction to the computing device based on the determined audio instruction, causing the computing device to execute the computing device instruction. The data207may comprise the computing device instruction, which can be a command (e.g., playing a certain music), a dialog (e.g., a question played to solicit further instructions from the user), a session management (e.g., sending an message to a contact, starting a navigation to home), etc. The effect of the data207can also be referred to below where the system102causes the (user's) computing device110to perform various functions.

In some embodiments, transmitting the computing device instruction to the computing device based on the determined audio instruction, causing the computing device to execute the computing device instruction, may comprise the following cases depending on the audio instruction. (1) In response to determining that the audio instruction is empty, the system102may generate a first dialog based on the context of the first interface, causing the computing device to play the first dialog. If the user supplies additional information in response to the dialog, the system102may analyze the additional information as an audio input. (2) In response to determining that the audio instruction comprises an entity, the system102may extract the entity, and generate a second dialog based on the extracted entity, causing the computing device to play the second dialog (e.g., output303adescribed below). (3) In response to determining that the audio instruction comprises a response, the system102may match the response with a response database, and in response to detecting a matched response in the response database, cause the computing device to execute the matched response (e.g., output303bdescribed below). (4) In response to determining that the audio instruction comprises a query, the system102may match the query with a query database. In response to detecting a matched query in the query database, the matched query may be outputted (e.g., output303cdescribed below). In response to detecting no matched query in the query database, feed the audio input and the context of the first interface to the one or more of algorithms to determine an audio instruction associated with the query (e.g., output303ddescribed below). Further details of these cases and the associated algorithms (e.g., a voice recognition engine106aand a natural language processing engine106bin the memory106) are described below with reference toFIGS. 3B-3D.

FIG. 3Billustrates example algorithms for natural language processing, in accordance with various embodiments. The algorithms may be shown in association with an example flowchart320. The operations shown inFIG. 3Band presented below are intended to be illustrative. Depending on the implementation, the example flowchart320may include additional, fewer, or alternative steps performed in various orders or in parallel. Various steps described below which call for “matching” may be performed by algorithms such as rule-based pattern matching.

In some embodiments, the system102may feed the audio input (e.g., the audio204) to an voice recognition engine106ato determine raw texts301corresponding to the audio input. There can be many example algorithms to implement the voice recognition engine106a, for converting the audio input to corresponding texts. For example, the voice recognition engine106amay first apply an acoustic model (e.g., Viterbi Model, Hidden Markov Model). The acoustic model may have been trained to represent the relationship between the audio recording of the speech and phonemes or other linguistic units that make up the speech, thus relating the audio recording to word or phrase candidates. The training may feed the acoustic model with sample pronunciations with labelled phonemes, so that the acoustic model can identify phonemes from audios. The voice recognition engine106amay dynamically determine the start and end for each phoneme in the audio recording and extract features (e.g., character vectors) to generate speech fingerprints. The voice recognition engine106amay compare the generated speech fingerprints with a phrase fingerprint database to select the most matching word or phrase candidates. The phrase fingerprint database may comprise the mapping between the written representations and the pronunciations of words or phrases. Thus, one or more sequence candidates comprising various combinations of words or phrases may be obtained. Further, the voice recognition engine106amay apply a language model (e.g., a N-gram model) to the one or more sequence candidates. The language model represents a probability distribution over a sequence of phrase, each determined from the acoustic model. The voice recognition engine106amay compare the selected words or phrases in the candidate sequences with a sentence fingerprint database (e.g., a grammar and semantics model) to select the most matching sentence as the raw texts301. The above example acoustic model and language model and other alternative models and their training are incorporated herein by reference.

In some embodiments, the system102may further feed the raw texts301and the context of the current interface (e.g., a part of the information206) to a natural language processing engine106bto determine an audio instruction (e.g., an entity, a response, a query) associated with the audio input. The natural language processing engine106bmay comprise: pre-processing algorithm(s)322, first machine learning model group324, second machine learning model group326, and extraction algorithm(s)328, the details of which are described below with reference toFIGS. 3C-3D. Also shown inFIGS. 3C-3D, the natural language processing engine106bmay comprise various other algorithms to help implement the disclosed methods. The natural language processing engine106bmay produce output303(e.g., determined query, intent, entity structure data, empty message, failure message, outputs303a-303fdescribed below). Accordingly, the system102may utilize various algorithms described above with reference toFIG. 2to obtain the data207.

FIGS. 3C and 3Dillustrate example algorithms for natural language processing, in accordance with various embodiments. The algorithms may be shown in association with an example flowchart330(separated into algorithms330aand330bin respective figures). The operations shown inFIGS. 3C and 3Dand presented below are intended to be illustrative. Depending on the implementation, the example flowchart330may include additional, fewer, or alternative steps performed in various orders or in parallel.

As shown inFIG. 3C, pre-processing algorithm(s)332may be configured to pre-process the raw texts301, in light of information206at one or more steps. In some embodiments, feeding the raw texts and the context of the current interface to the natural language processing engine106bto determine the query associated with the audio input comprises: pre-processing the raw texts based on at least one of: lemmatizing, spell-checking, singularizing, or sentiment analysis to obtain pre-processed texts; matching the pre-processed texts against preset patterns; in response to not detecting any preset pattern matching the pre-processed texts, tokenizing the texts; and vectorizing the tokenized texts to obtain vectorized texts. Various pre-processing algorithms and associated steps are described below.

At block31, a mode determination algorithm may be applied to determine if the raw texts comprise only an “entity” (e.g., an entity name), only a “response” (e.g., a simple instruction), or a “query” (e.g., one or more queries), where the query may comprise an entity and/or a response.

In some embodiments, if the determination is “entity,” the flowchart may proceed to block32where a normalization algorithm can be applied to, for example, singularize, spell-check, and/or lemmatize (e.g., remove derivational affixes of words to obtain stem words) the raw texts. From block32, the flowchart may proceed to block34or proceed to block33before proceeding to block34. At block33, a part of speech tagger algorithm may be used to tag the part-of-speech of the each word. At block34, extraction algorithm328may be used to extract the entity as output303a. In one example, the system102may have obtained the current interface as being “media” and the user's intention to play music, and have asked the user in a dialog “which music should be played?” The user may reply “Beethoven's” in an audio input. Upon the normalization and part-of-speech tagging, the system102may normalize “Beethoven's” to “Beethoven” as a noun and output “Beethoven.” Accordingly, the system102can cause the user's computing device to obtain and play a Beethoven playlist. In another example, the system102may have obtain the current interface as being messaging and the user's intention to send an email, and have asked the user in a dialog “who should this email be sent to?” The user may reply “John Doe” in an audio input. The system102may recognize John Doe from the user's contacts. Accordingly, the system102may obtain John Doe's email address, and cause the user's computing device to start drafting the email.

In some embodiments, if the determination is “response,” the flowchart may proceed to block35where a match algorithm may be applied to match the raw texts again a database of generic intents (e.g., confirmation, denial, next). If the match is successful, the matched generic intent can be obtained as output303b. In one example, when a current interface is “media,” the user may say “stop” to cease the music or “next” to play the next item in the playlist. In another example, in a dialog, the system102may ask some simple “yes” or “no” question. The user's answer, as a confirmation or denial, can be parsed accordingly. In yet another example, if the current interface is navigation from which the user tries to look for a gas station and the system102has determined three closest gas stations, the system102may play information of these three gas stations (e.g., addresses and distances from the current location). After hearing about the first gas station, the user may say “next,” which can be parsed as described above, such that the system102will recognize and play the information of the next gas station.

In some embodiments, if the determination is “query,” the flowchart may proceed to block36where a sentence splitting algorithm may be applied to split the raw texts into sentences. At block37, for each sentence, a clean sentence algorithm may be applied to determine the politeness and/or remove noises. To both block36and block37, a sentiment analysis algorithm at block38may be applied. The sentiment analysis algorithm may classify the sentence as positive, neutral, or negative. At block37, if the determined politeness is above a preset threshold, the flowchart may proceed to block41where the normalization algorithm is applied. If the determined politeness is not above the preset threshold, the flowchart may proceed to block39where the normalization algorithm is applied, and then to block40where a filtering algorithm is applied to filter impolite words. After filtering, if the texts are empty, the audio input may be interpreted as a complaint. The system102may obtain a “user complaint” as output303fand cause the user's computing device to create a dialog to help resolve the complaint. If the texts are non-empty, the flowchart may proceed to block41. The raw texts301pre-processed by any one or more steps from block31to block41may be referred to as pre-processed texts. From block41, the flowchart may proceed to block42, where a pattern match algorithm may be applied to match the pre-processed texts against an intent database, and a direct match may be obtained as output303c. The intent database may store various preset intents. In one example, one of the preset intent “playing music” corresponds to detecting a text string of “play+[noun.]” when the current interface is “media.” Accordingly, if the pre-processed texts are determined to be “can you please play Beethoven,” the output303cmay be “play Beethoven.” If there is no direct match, the flowchart may proceed to block43, where a tokenization algorithm may be applied to obtain tokenized texts (e.g., an array of tokens each representing a word). The tokenized texts may be further vectorized by a vectorization algorithm to obtain vectorized texts (e.g., each word represented by strings of “0” and “1”).

Continuing fromFIG. 3CtoFIG. 3D, first machine learning model group324and/or second machine learning model group326may be configured to process the raw texts301, the pre-processed texts, the tokenized texts, and/or vectorized texts, in light of the information206. That is, any of the texts in the various forms may be used as inputs to the first and then to the second machine learning model group, or directly to the second machine learning model group.

In some embodiments, the first machine learning model group324may be applied to obtain a general classification of the intent corresponding to the audio input at block48. Feeding the raw texts and the context of the current interface to the natural language processing engine106bto determine the query associated with the audio input further comprises: dynamically updating one or more weights associated with one or more first machine learning models at least based on the first context described above (comprised in the information206); and applying the one or more first machine learning models to the first context and at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts, to obtain an intent classification of the audio input. The first machine learning models may comprise a decision-tree-based model, a feedforward neural network model, and a graph-support vector machine (DAGSVM) model, all of which and their training are incorporated herein by reference. Applying the one or more first machine learning models to obtain the intent classification of the audio input comprises: applying a decision-tree-based model (block44) and a feedforward neural network model (block45) each to the first context and to the at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts to obtain corresponding output classifications. The outputs of block44and block45are compared at block46. In response to determining that an output classification from the decision-tree-based model is the same as an output classification from the feedforward neural network model, either of the output classification (from block44or block45) can be used as the intent classification of the audio input (block48). In response to determining that the output classification from the decision-tree-based model is different from the output classification from the feedforward neural network model, the DAGSVM model can be applied to the corresponding at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts (block47) to obtain the intent classification of the audio input (block48). In the above steps, based on the context of the current interface, one or more weights of the class associated with the user's intention in the each machine learning model can be dynamically adjusted. For example, for a current interface being “media,” the “media” classification's weights may be increase in the various algorithms and models, thus improving the accuracy of the classification.

In some embodiments, the second machine learning model group326may be applied to obtain a sub-classification of the intent corresponding to the audio input at block57. Feeding the raw texts and the context of the current interface to the natural language processing engine106bto determine the query associated with the audio input further comprises: applying one or more second machine learning models326to the second context described above (comprised in the information206) and at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts to obtain a sub-classification prediction distribution of the audio input; and comparing the sub-classification prediction distribution with a preset threshold and against an intent database to obtain a sub-classification of the audio input, wherein the sub-classification corresponds to a prediction distribution exceeding the preset threshold and matches an intent in the intent database. In response to multiple prediction distributions exceeding the preset threshold, the audio input may be determined to correspond to multiple intents, and a neural network model may be applied to divide the at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts correspondingly according to the multiple intents. For each of the divided texts, the N-gram model to may be applied to obtain the corresponding intent sub-classification.

In some embodiments, at block49, the raw texts, the pre-processed text, the tokenized texts, the vectorized texts, the information206, and/or the classification from block48may be fed to a naive bayes model and/or a term frequency-inverse document frequency (TF-IDF) model to obtain a sub-classification prediction distribution (e.g., a probability distribution for each type of possible sub-classification). Alternatively or additionally, the raw texts, the pre-processed text, the tokenized texts, the vectorized texts, and/or the information206may bypass the first machine learning model group and be fed to the second machine learning model group. At block50, the prediction distribution may be applied with thresholding. If one or more prediction distribution exceeds the threshold, the flowchart may proceed to block51; if no prediction distribution exceeds the threshold, the flowchart may proceed to block52. At block51, if two or more sub-classification predictions exceed the threshold (e.g., when the audio input is “navigate home and play music” which corresponds to two intents), the flowchart may proceed to block52, where a neural network (e.g., feedforward neural network (FNN), recurrent neural network (RNN)) model may be applied to (1: following from block51) separate the corresponding input texts into various text strings based on the multiple sub-classification predictions and/or (2: following from block50) extract a sub-classification prediction. If just one sub-classification prediction exceeds the threshold, after the multiple sub-classification predictions are separated, or after the sub-classification prediction is extracted, the flowchart may proceed to block53where a N-gram model may be applied to convert the each text string (which corresponds to the sub-classification prediction) for approximate matching. By converting the sequence of text strings to a set of N-grams, the sequence can be embedded in a vector space, thus allowing the sequence to be compared to other sequences (e.g., preset intentions) in an efficient manner. Accordingly, at block54, the converted set of N-grams (corresponding to the sub-classification prediction) may be compared against an intent database to obtain a matching intent in the intent database. The matching intent(s) may be obtained as the sub-classification(s) of the audio input at block57.

In some embodiments, each sub-classification may represent a sub-classified intent, and the general classification described above at block48may represent a general intent. Each general classification may correspond to multiple sub-classification. For example, a general classification “media” may be associated with sub-classifications such as “play music,” “play podcast,” “play radio,” “play audio book,” “play video,” etc. For another example, a general classification “navigation may be associated with sub-classifications such as “points-of-interest,” “points-of-interest location search,” “start navigation,” “traffic,” “show route,” etc. For yet another example, a “messaging” classification may be associated with sub-classifications such as “email,” “send text message,” “draft social media message,” “draft social media post,” “read message,” etc.

If the intent match is unsuccessful at block54, a feedforward neural network model may be applied at block55. At block56, the outputs of the block49and the block55may be compared. If the two outputs are the same, the flowchart may proceed to block57; otherwise, the second machine learning model group326may render output303e(e.g., a fail message). The naive bayes model, the TF-IDF model, the N-gram model, the FNN, and the RNN, and their training are incorporated herein by reference. Based on the context of the current interface, one or more weights of the class associated with the user's intention in the each machine learning model can be dynamically adjusted, thus improving the accuracy of the classification.

In some embodiments, the classification from block48and the sub-classification from the block57may be compared. In response to determining that the intent classification (block48) and the intent sub-classification (block57) are consistent, extraction algorithm(s)328(e.g., conditional random field (CRF) incorporated herein by reference, name entity recognition (NER) algorithm incorporated herein by reference) may be applied to identify and extract one or more entities from the tokenized texts at block58. Each sub-classification may be associated with one or more preset entities. The entities may be extracted from the public database106c, the personal database106d, or other databases or online resources based on matching. In response to determining that the intent classification and the intent sub-classification are inconsistent, the one or more first machine learning models324, without the context of the current interface, may be re-applied at block59to the at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts to update the intent classification of the audio input. The inconsistency may arise when, for example, the user inputs a navigation-related audio when the current interface is not navigation (e.g., the user asks “how is the traffic to home” from the media interface). According to the flow of the first and second machine learning models, a general classification of “media” and a sub-classification of “traffic to home” may be obtained respectively and inconsistent with each other. Thus, the first machine learning models can be re-applied without the context information for adjusting the general classification.

In some embodiments, one or more entities from the tokenized text may be identified based on at least one of the intent classification, the intent sub-classification, or the second context; contents associated with the one or more entities may be determined based on at least one of public data or personal data; and the query may be determined as an intent corresponding to at least one of the intent classification or the intent sub-classification, in association with the determined one or more entities and the determined contents. From block58and block59respectively, an output303d(e.g., a classified intent with associated entity structured data) may be obtained. For example, if the audio input is “find me a coffee shop near ABC University” at a navigation interface, the disclosed systems and methods can obtain a general classification of “navigation,” a sub-classification of “points-of-interest location search,” and a search target (entity1of the sub-classification) of “coffee shop,” a search area (entity2of the sub-classification) of “ABC University.” With the above information, the system102can generate an appropriate response and cause the user's computing device to respond accordingly to the user.

As shown above, the disclosed systems and methods including the multi-layer statistical based models can leverage interface context to supplement natural language processing and significantly improve the accuracy of machine-based audio interpretation. The supported contents and services (e.g., maps, traffic, music stream, radio, email, messenger) can be grouped into main categories such as “navigation,” “media,” and “messaging” and consolidated into a software application (e.g., a mobile phone application similar toFIG. 3A). The software application can implement the disclosed methods. Application users can be empowered to navigate, access, and manage their favorite contents and services easily and quickly while traditional control methods are unavailable, dangerous, or inconvenient. Four main benefits, among others, are elaborated below.

The disclosed systems and methods can offer highly accurate intent prediction, by taking user's current context information such as the current interface into consideration. The natural language processing engine can dynamically adjust the weights towards each general classification candidate and sub-classification candidate, which means the current interface-related intents can have a higher numerical representation in the prediction distribution and more likely to be matched.

The disclosed systems and methods can also offer dialog conversation with the user. Understanding the current interface context can help the system generate appropriate dialogs to instantly engage the user and respond specifically. For example, if the user triggers the microphone at yjr “media” interface but does not give clear commands, the system can generate an instant voice dialog “do you want to play your most recent playlist” (as a voice) to quickly engage user's intention. The voice dialog's content can be generated based on the current interface context.

The disclosed systems and methods can also offer effective session management. For example, users may give generic commands like “stop”, “start,” and “next”, which can be used with different intents and at different interfaces. The above systems and methods can reduce the ambiguity in the interpretation and avoid excessive probing, by encompassing various scenarios in the above algorithms.

The disclosed systems and methods can further offer personalized recommendations. Giving recommendations is useful especially in a driving scenario where driver's hands are occupied and attention is focused on the road. The above systems and methods can provide personal recommendations for driving routes, media contents, schedule management, and instant contacts based on a current session, the user's settings, the user's preferences, and/or the user's histories such as past schedules and past routes.

FIG. 4Aillustrates a flowchart of an example method400, according to various embodiments of the present disclosure. The method400may be implemented in various environments including, for example, the environment100ofFIG. 1. The example method400may be implemented by one or more components of the system102(e.g., the processor104, the memory106). The example method400may be implemented by multiple systems similar to the system102. The operations of method400presented below are intended to be illustrative. Depending on the implementation, the example method400may include additional, fewer, or alternative steps performed in various orders or in parallel.

At block402, an audio input and a current interface may be obtained from a computing device, wherein the current interface is associated with a context. At block404, a query associated with the audio input may be determined based at least on the audio input and the context of the current interface. In some embodiments, the computing device is configured to provide a plurality of inter-switchable interfaces, the plurality of interfaces comprise at least one of: an interface associated with navigation, an interface associated with media, or an interface associated with messaging, the context of the current interface comprises a first context and a second context, the first context comprises at least one of: the current interface as navigation, the current interface as media, or the current interface as messaging, and the second context comprises at least one of: an active route, a location, an active media session, or an active message.

The block404may comprise block412and block414. At block412, the audio input may be fed to an voice recognition engine to determine raw texts corresponding to the audio input. At block414, the raw texts and the context of the current interface may be fed to a natural language processing engine to determine the query associated with the audio input. The block414may comprise the method420described below.

FIG. 4Billustrates a flowchart of an example method420, according to various embodiments of the present disclosure. The method420may be implemented in various environments including, for example, the environment100ofFIG. 1. The example method420may be implemented by one or more components of the system102(e.g., the processor104, the memory106). The example method420may be implemented by multiple systems similar to the system102. The operations of method420presented below are intended to be illustrative. Depending on the implementation, the example method420may include additional, fewer, or alternative steps performed in various orders or in parallel. Various modules described below may have been trained, e.g., by the methods discussed above.

At block421, the raw texts may be pre-processed based on at least one of: lemmatizing, spell-checking, singularizing, or sentiment analysis to obtain pre-processed texts. At block422, the pre-processed texts may be matched against preset patterns. At block423, in response to not detecting any preset pattern matching the pre-processed texts, the texts may be tokenized. At block424, the tokenized texts may be vectorized to obtain vectorized texts.

At block425, one or more weights associated with one or more first machine learning models may be dynamically updated at least based on the first context. At block426, the one or more first machine learning models may be applied to the first context and at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts, to obtain an intent classification of the audio input. In some embodiments, the block426comprises: applying a decision-tree-based model and a feedforward neural network model each to the first context and to the at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts to obtain corresponding output classifications; in response to determining that an output classification from the decision-tree-based model is the same as an output classification from the feedforward neural network model, using the either output classification as the intent classification of the audio input; and in response to determining that the output classification from the decision-tree-based model is different from the output classification from the feedforward neural network model, applying a directed acyclic graph-support vector machine (DAGSVM) model to the corresponding at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts to obtain the intent classification of the audio input.

At block427, one or more second machine learning models may be applied to the second context and at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts to obtain a sub-classification prediction distribution of the audio input, the one or more second machine learning models comprising at least one of: a naive bayes model, a term frequency-inverse document frequency model, a N-gram model, a recurrent neural network model, or a feedforward neural network model. At block428, the sub-classification prediction distribution may be compared with a preset threshold and matched against an intent database to obtain a sub-classification of the audio input, wherein the sub-classification corresponds to a prediction distribution exceeding the preset threshold and matches an intent in the intent database.

In some embodiments, the method420further comprises: in response to multiple prediction distributions exceeding the preset threshold, determining that the audio input corresponds to multiple intents and applying a neural network model to divide the at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts correspondingly according to the multiple intents; and for each of the divided texts, applying the N-gram model to obtain the corresponding intent sub-classification.

In some embodiments, the method420further comprises: in response to determining that the intent classification and the intent sub-classification are consistent, extracting one or more entities from the tokenized texts; and in response to determining that the intent classification and the intent sub-classification are inconsistent, re-applying the one or more first machine learning models without the context of the current interface to the at least one of: the raw texts, the pre-processed text, the tokenized texts, or the vectorized texts to update the intent classification of the audio input.

At block429, one or more entities may be identified from the tokenized text based on at least one of the intent classification, the intent sub-classification, or the second context. At block430, contents associated with the one or more entities may be determined based on at least one of public data or personal data. At block431, optionally, the query may be determined as an intent corresponding to at least one of the intent classification or the intent sub-classification, in association with the determined one or more entities and the determined contents.

FIG. 4Cillustrates a flowchart of an example method480, according to various embodiments of the present disclosure. The method480may be implemented in various environments including, for example, the environment100ofFIG. 1. The example method480may be implemented by one or more components of the system102(e.g., the processor104, the memory106). The example method480may be implemented by multiple systems similar to the system102. The operations of method480presented below are intended to be illustrative. Depending on the implementation, the example method480may include additional, fewer, or alternative steps performed in various orders or in parallel.

At block482, an audio input may be obtained from the computing device, wherein the audio is inputted to the computing device when a first interface of the computing device is active. At block484, a context of the first interface may be determined, the first interface comprising an interface associated with media, an interface associated with navigation, or an interface associated with messaging. At block486, the audio input and the context of the first interface may be fed to one or more algorithms to determine an audio instruction associated with the audio input. At block488, a computing device instruction may be transmitted to the computing device based on the determined audio instruction, causing the computing device to execute the computing device instruction.

The block488may comprise block492, block494, block496, and block498. At block492, in response to determining that the audio instruction is empty, a first dialog may be generated based on the context of the first interface, causing the computing device to play the first dialog. At block494, in response to determining that the audio instruction comprises an entity, the entity may be extracted, and a second dialog may be generated based on the extracted entity, causing the computing device to play the second dialog. At block496, in response to determining that the audio instruction comprises a response, the response may be matched with a response database, and in response to detecting a matched response in the response database, the computing device may be caused to execute the matched response. At block498, in response to determining that the audio instruction comprises a query, the query may be matched with a query database, and in response to detecting no matched query in the query database, the audio input and the context of the first interface may be fed to the one or more of algorithms to determine an audio instruction associated with the query. The block498may further comprise the method330or420described above.

FIG. 5is a block diagram that illustrates a computer system500upon which any of the embodiments described herein may be implemented. The system500may correspond to the system102described above. The computer system500includes a bus502or other communication mechanism for communicating information, one or more hardware processors504coupled with bus502for processing information. Hardware processor(s)504may be, for example, one or more general purpose microprocessors. The processor(s)504may correspond to the processor104described above.

The computer system500also includes a main memory506, such as a random access memory (RAM), cache and/or other dynamic storage devices, coupled to bus502for storing information and instructions to be executed by processor504. Main memory506also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor504. Such instructions, when stored in storage media accessible to processor504, render computer system500into a special-purpose machine that is customized to perform the operations specified in the instructions. The computer system500further includes a read only memory (ROM)508or other static storage device coupled to bus502for storing static information and instructions for processor504. A storage device510, such as a magnetic disk, optical disk, or USB thumb drive (Flash drive), etc., is provided and coupled to bus502for storing information and instructions. The main memory506, the ROM508, and/or the storage510may correspond to the memory106described above.

The computer system500also includes a communication interface518coupled to bus502. Communication interface518provides a two-way data communication coupling to one or more network links that are connected to one or more local networks. For example, communication interface518may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface518may be a local area network (LAN) card to provide a data communication connection to a compatible LAN (or WAN component to communicated with a WAN). Wireless links may also be implemented. In any such implementation, communication interface518sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

The computer system500can send messages and receive data, including program code, through the network(s), network link and communication interface518. In the Internet example, a server might transmit a requested code for an application program through the Internet, the ISP, the local network and the communication interface518.

The various operations of example methods described herein may be performed, at least partially, by an algorithm. The algorithm may be comprised in program codes or instructions stored in a memory (e.g., a non-transitory computer-readable storage medium described above). Such algorithm may comprise a machine learning algorithm or model. In some embodiments, a machine learning algorithm or model may not explicitly program computers to perform a function, but can learn from training data to make a predictions model (a trained machine learning model) that performs the function.