Patent Description:
A speech-enabled environment is one in which a user can speak a query or command aloud and a computer-based system will obtain the query, answer the query, and/or cause the command to be performed using speech recognition techniques. The speech-enabled environment may include a network of connected microphone devices distributed throughout the various rooms or areas of the environment. A user has the power to orally query the computer-based system from essentially anywhere in the speech-enabled environment without the need to have a computer or other device in front of him/her or even nearby. For example, while getting dressed in the morning, a user might ask the computer-based system "what's the temperature outside?" and, in response, receive an answer from the system, e.g., in the form of synthesized voice output. In other examples, a user might ask the computer-based system questions such as "what time does my favorite restaurant open?" or "where is the nearest gas station?".

For the speech-enabled environment, users can interact with the computer-based system primarily through voice input. As a result, the computer-based system may obtain potentially all utterances and noises made in the speech-enabled environment including those utterances not directed towards the computer-based system. Thus, the computer-based system must have some way to discern between utterances directed towards the computer-based system and utterances not directed towards the computer-based system, but rather, to other individuals present in the speech-enabled environment. One way to accomplish this is for the user to use a predetermined word, such as a hotword or keyword, to signify the attention of the computer-based system. Additionally, the other individuals in the speech-enabled environment understand that the use of the predetermined word is only for the computer-based system. For example, a user may speak "OK computer," followed by a query, to ask the computer-based system a question. However, the user having to speak a predetermined word each time he or she asks the computer-based system a question disrupts the flow of normal conversation and imposes a cognitive burden on the user. In order to alleviate this cognitive burden, the computer-based system can keep the microphone open for any follow on questions the user may have after a user asks a first question using the predetermined word. The follow on questions would not require the use of the predetermined word. However, the computer-based system would still need to discern follow on questions directed towards the computer-based system and other utterances obtained not directed towards the computer-based system. Accordingly, if the computer-based system determines a follow on utterance looks like a question likely directed to the computer-based system, then the follow on utterance is accepted and processed. Otherwise, the computer-based system closes the microphone and waits for the next utterance from a user that includes the predetermined word. Accordingly, an improved mechanism can be provided to facilitate user input, such as user commands, to the computer-based system. Other examples for determining whether a user vocal query is directed to a computer system can be found in <CIT>.

In some implementations, the computer-based system allows free form conversations with the user once the computer-based system listens for follow on questions. The user no longer needs to use the predetermined hotword to communicate with the computer-based system. The computer-based system uses both the acoustic characteristics of the obtained utterance and recognized text of the obtained utterance to determine whether the follow on question is directed towards the computer-based system.

In some implementations, a classification system includes a classifier to discern content of human speech. Specifically, the content includes a determination that the audience for the human speech is likely directed towards the automated assistant server. The classification system includes a speech recognizer, a transcription representation generator, an acoustic feature generator, a concatenation module, and a classifier to perform this determination function. The speech recognizer can obtain utterance information spoken by a user and generate a transcription of the spoken utterance from the user. The acoustic feature generator can receive the utterance information spoken by the user and output speech unit representations, such as feature vectors that describe the audio characteristics of the received spoken utterance. The transcription representation generator can receive the transcription of the spoken utterance from the speech recognizer and output transcriptions including word embeddings. The concatenation module can receive the acoustic information and the word embeddings, concatenate the acoustic information and the word embeddings into a matrix representation, and provide the matrix representation to the classifier. The classifier provides an indication that the audience for the obtained utterance is likely directed towards the automated assistant server.

In some implementations, the classifier includes one or more neural network elements to process the spoken utterance. The classifier trains the neural network elements with examples of queries previously spoken by users not likely directed towards the automated assistant server. This type of training allows the classifier to detect these types of queries and additionally, provide an indication that the audience for the obtained queries is likely directed towards the automated assistant server.

In one general aspect, a method according to independent claim <NUM> is proposed.

Other embodiments of this and other aspects of the disclosure include corresponding independent system claim <NUM> and non-transitory computer-readable medium claim <NUM>. Other preferred embodiments are set forth in the dependent claims.

The specification describes a technique for performing speech classification to determine whether an obtained utterance is likely directed towards the computer-based system. The computer-based system does not require that the obtained utterance include a predetermined word, such as a hotword, to invoke the computer-based system's attention. Rather, the computer-based system's microphone remains open for any obtained utterance of follow on questions following a user's first query that does include the predetermined word. Advantageously, the technique uses neural networks for both the acoustic characteristics and the recognized text of the obtained utterance to train a neural network to produce an indication of whether the audience for the obtained utterance is likely directed towards the computer-based system. This may be beneficial because users can converse in a more fluent manner with the computer-based system without the use of the hotword.

<FIG> is a block diagram that illustrates an example of a system <NUM> for speech classification using a classifier server <NUM>. The system <NUM> includes a classifier server <NUM> and an automated assistant server <NUM>. Each of the classifier server <NUM> and the automated assistant server <NUM> can include one or more computers. The classifier server <NUM> includes one or more neural network components, a speech recognizer, and an acoustic feature generator, which will be further explained below with respect to <FIG>. The classifier server <NUM> may include one or more servers connected locally or over a network. The classifier server <NUM> may be implemented in software, hardware, firmware, or a combination thereof. <FIG> illustrates various operations in stages (A) to (D) and (A') to (B') which can be performed in the sequence indicated or in another sequence.

The example of <FIG> shows an example of the classifier server <NUM> determining whether an utterance is likely directed towards a recipient. The processing shown in <FIG> achieves two important tasks. First, unlike systems that require a user to speak a hotword or keyword each time the user speaks to that system, the classifier server <NUM> does not require the utterance to include a hotword or a keyword after a first use of the hotword or keyword that signifies the audience is likely directed towards the automated assistant server <NUM>. The classifier server <NUM> requires a user to include a word, such as a hotword or keyword, the first time a user <NUM> speaks to the automated assistant server <NUM>. The classifier server <NUM> includes a microphone that turns on in response to hearing the hotword or keyword. Additionally, the microphone remains on for subsequent questions provided by the user <NUM>. The microphone turns off once the user <NUM> ceases asking questions.

In conversational settings, when a user includes a hotword or a keyword each time he or she signifies to a device, such as the classifier server <NUM>, the hotword can interrupt the flow of normal conversation. Therefore, the processing shown in <FIG> removes the use of a hotword or a keyword after a first use of the hotword or keyword to allow for conversation that is more fluent.

Second, the classifier server <NUM> can determine the audience that the utterance is likely directed to is the automated assistant server <NUM> without the use of a hotword. As a result, the classifier server <NUM> can distinguish between utterances directed towards the automated assistant server <NUM> and utterances not directed towards the automated assistant server <NUM>. In some implementations, the utterances not directed towards the automated assistant server <NUM> may include phrases potentially directed towards the automated assistant server <NUM>. For example, phrases such as "What time is it" may be included in phrases such as "Hey mom, I'm late for school, what time is it" that the classifier server <NUM> may obtain.

However, upon analyzing the recorded utterance, the classifier server <NUM> can determine that the utterance includes other phrases in connection with the question and indicates the audience for the utterance is not the automated assistant server <NUM>. In summary, the classifier server <NUM> can judge the likelihood and provide an indication that the audience for the obtained utterance is likely directed to the automated assistant server <NUM>.

In some implementations, the classifier server <NUM> can provide data indicating an obtained utterance and instructions to the automated assistant server <NUM>. The classifier server <NUM> creates instructions to convey to the automated assistant server <NUM> whether to process the obtained utterance or not. For example, the classifier server <NUM> may obtain an utterance spoken by a user, such as user <NUM>, and determine that the audience for the obtained utterance is likely directed to the automated assistant server <NUM>. In response, the classifier server <NUM> can provide data, indicating instructions and the obtained utterance, to the automated assistant server <NUM> over a network such as network <NUM>. The instructions request that the automated assistant server <NUM> process the obtained utterance and generate a response to the obtained utterance.

In some implementations, the automated assistant server <NUM> can generate a response to the obtained utterance. In particular, the automated assistant server <NUM> can provide an answer to the questions and/or statements provided by the classifier server <NUM>. For example, the automated assistant server <NUM> may obtain data indicating an utterance and instructions that require the automated assistant server <NUM> to process the utterance. The automated assistant server <NUM> determines that the utterance recites, "What time is it" and generates a response to the utterance. For example, the automated assistance server <NUM> determines the time is "<NUM>:<NUM> PM" and generates a response <NUM> to provide to the classifier server <NUM> over a network <NUM>. The response <NUM> may include the answer that recites, "The time is <NUM>:<NUM> PM. " The classifier server <NUM> can provide the response generated by the automated assistant server <NUM> to a client device, such as client device <NUM>, owned by the user <NUM>.

The examples of this system described above can be illustrated with regards to an exemplary use case shown in <FIG> as described below.

During stage (A), the classifier server <NUM> obtains a spoken utterance from a user <NUM>. The spoken utterance can include various types of phrases and/or a question directed towards the automated assistant server <NUM>. In other implementations, the classifier server <NUM> can obtain one or more spoken utterances from the user <NUM> that is not likely directed to the automated assistant server <NUM>. The spoken utterances may include audio waveform over a predetermined length of time.

In some implementations, the classifier server <NUM> may record utterances detected above a threshold level of volume. The levels of volume can be measured in decibels (dB). For example, the classifier server <NUM> may obtain audio recordings starting from a first time the classifier server <NUM> detects acoustical properties from users and/or devices nearby above a threshold to a second time when then the classifier server <NUM> no longer detects acoustical properties from the nearby users and/or devices. For example, the normal voice of a user in a typical area may vary between <NUM> dB and <NUM> dB, depending on the distance between the user and the classifier server <NUM>. At the lower end of the volume level, for example, the classifier server <NUM> may be recording faint background noise. At the higher end of the volume level, for example, the classifier server <NUM> may be recording loud music or loud talking, to name a few examples. In one example, user <NUM> may pose a question to the automated assistant server <NUM>, such as "Ok computer, what should I wear today?" at <NUM>:<NUM> PM. The classifier server <NUM> can detect the hotword from acoustical properties of the spoken question and process the question "what should I wear today?" Then, the classifier server <NUM>'s microphone remains on from <NUM>:<NUM> PM for a period of time to wait for follow on questions from user <NUM>. The time period can be <NUM> second, <NUM> seconds, or <NUM> seconds, to name a few examples. The microphone remains on for as long as the user <NUM> continues to ask follow on questions directed towards the automated assistant server <NUM>.

In some implementations, the classifier server <NUM> may record each audio utterances from various devices and/or users located near the location of the classifier server <NUM> above a particular threshold level. For example, the classifier server <NUM> may listen and record audio from users in the same room as the classifier server <NUM>. In one example, the classifier server <NUM> may listen and record one or more individuals communicating from a television program as displayed by the television. In another example, the classifier server <NUM> may listen and record music played from speakers in audible range of the classifier server <NUM>. In another example, the classifier server <NUM> may listen and record one or more individuals communicating over a client device, such as a cell phone or a laptop using a voice-to-talk or video chatting application. In particular, the audio recordings may be recorded from various devices and/or users located throughout a room where the classifier server <NUM> is located.

In the illustrated example, the classifier server <NUM> obtains an utterance <NUM> from user <NUM>. The utterance <NUM> poses a question <NUM> to the automated assistant server <NUM>. The question <NUM> recites, "What's the temperature outside?" The classifier server <NUM> obtains the utterance <NUM> and records the utterance <NUM> for further processing.

In some implementations, the user <NUM> may ask more than one question to the automated assistant server <NUM> in a short period. In other implementations, the user <NUM> may ask one question directed towards another individual in the area and a subsequent question directed towards the automated assistant server <NUM>. For example, the user <NUM> may ask the question to his or her significant other "What should I wear today, Rachel?" and subsequently ask another question to the automated assistant server <NUM> - "What is the temperature outside?" In this example, the classifier server <NUM> can record both questions asked by user <NUM> and determine following processing of the recorded utterances that the former question is not directed towards the automated assistant server <NUM> whereas the latter question is.

During stage (B), the classifier server <NUM> performs processing on the recorded utterance <NUM>. In particular, the classifier server <NUM> classifies the recorded utterance <NUM> and provides an output indicating a likelihood that the audience for the utterance is likely directed to the automated assistant server <NUM>. As further described below with respect to <FIG>, the classifier server <NUM> utilizes an acoustic representation and a textual representation of the utterance <NUM> to determine whether the audience for utterance <NUM> is likely directed to the automated assistant server <NUM>.

In this illustrated example, the classifier server <NUM> produces an indication that the audience for the utterance <NUM> is likely directed to the automated assistant server <NUM>. As seen in <FIG>, the checkbox illustrates the indication. In a practical application, the automated assistant server <NUM> may provide an indication, such as a flashing light, upon a determination by the classifier server <NUM> that the audience for utterance <NUM> is likely directed to the automated assistant server <NUM>. In some implementations, the automated assistant server <NUM> may provide the indication in response to receiving the selective instructions <NUM> from the classifier server <NUM>. In other implementations, the automated assistant server <NUM> may provide a beep, a ring, or a pre-determined tone as indicated by user <NUM> to denote the audience for the utterance <NUM> from user <NUM> is likely directed towards the automated assistant server <NUM>.

In alternate implementations, the classifier server <NUM> may obtain utterances and determine that the obtained utterances are not likely directed towards the automated assistant server <NUM>. This can be illustrated in the example shown in <FIG>.

During stage (A'), the classifier server <NUM> obtains another spoken utterance <NUM> from one or more devices. In this illustrated example, the television program from television <NUM> produces a spoken utterance <NUM> of a phrase <NUM> that includes the question <NUM>. For example, a character in the television program may recite, "Bob said, what's the temperature outside, and I shook. " In response, the classifier server <NUM> may obtain and record the spoken utterance <NUM> upon a determination that a volume of the utterance is above a decibel threshold level.

In some implementations, the classifier server <NUM> may obtain and record utterances, such as utterance <NUM> and utterance <NUM>, at substantially the same time. The classifier server <NUM> can distinguish between each of the recorded utterances. In other implementations, the classifier server <NUM> may obtain and record utterances, such as utterance <NUM> and utterance <NUM>, sequentially. In such a case where the classifier server <NUM> obtains utterances sequentially, the classifier server <NUM> processes each utterance in the order received. In other implementations, the classifier server <NUM> may produce an indication that the obtained and recorded utterance is not discernable. For example, the obtained and recorded utterance <NUM> may include various noise components, from external events such as other users speaking, the television's loud volume, a fan running in the background, and a dog barking, to name a few examples. The classifier server <NUM> may provide an indication such as "Please repeat the phrase," over a speaker to the user <NUM>.

During stage (B'), the classifier server <NUM> performs processing on the recorded utterance <NUM>. In particular, the classifier server <NUM> classifies the recorded utterance <NUM> and provides an output indicating a likelihood that the audience for utterance <NUM> was likely directed to the automated assistant server <NUM>. In this illustrated example, the classifier server <NUM> provides an indication that the audience for the recorded utterance <NUM> was not likely directed to the automated assistant server <NUM>. As seen in <FIG>, the "X" in the box provides the indication as shown inside the classifier server <NUM>. In a practical application, the classifier server <NUM> turns off a microphone to stop listening to conversation in response to the determination that the recorded utterance <NUM> was not likely directed to the automated assistant server <NUM>.

In other implementations, the classifier server <NUM> may perform additional actions in response to the determination that the audience for the recorded utterance was not likely directed to the automated assistant server <NUM>. For example, the classifier server <NUM> may discard the recorded utterance, such as recorded utterance <NUM>, and continue listening for other utterances. In another example, the classifier server <NUM> may transfer the recorded utterance, such as recorded utterance <NUM>, to the automated assistant server <NUM> with particular instructions.

In some implementations, instead of the classifier server <NUM> instructing the automated assistant server <NUM> to process the recorded utterance <NUM>, the classifier server <NUM> may instruct the automated assistant server <NUM> to store the utterance in a database. The database may store one or more utterances that the classifier server <NUM> can access to determine whether a current obtained utterance does not match any of the stored utterances. For example, the classifier server <NUM> can compare the obtained utterance <NUM> to one or more utterances stored in the database. Should any of the comparisons match within a predetermined threshold, the classifier server <NUM> can reject that particular utterance since the audience was not likely directed to the automated assistant server <NUM>. Alternatively, should none of the comparisons match, the classifier server <NUM> can process the recorded utterance through a classifier, as described below with respect to <FIG>.

During stage (C), the automated assistant server <NUM> obtains the data indicating the recorded utterance and the selective instructions from the classifier server <NUM>. In the illustrated example, the automated assistant server <NUM> obtains the recorded utterance <NUM> and not the recorded utterance <NUM>, because the classifier server <NUM> determined the audience for the former utterance is likely directed to the automated assistant server <NUM>, whereas the audience for the latter utterance is not likely directed to the automated assistant server <NUM>. The classifier server <NUM> does not provide an indication of who the likely recipient was of the recorded utterance <NUM>, but rather, that the recipient was not likely directed towards the automated assistant server <NUM>.

In some implementations, the automated assistant server <NUM> processes the recorded utterance, such as utterance <NUM>, in response to the instructions in the data <NUM>. For example, the instructions may convey to the automated assistant server <NUM> to process the recorded utterance <NUM> and provide a response to user <NUM>'s question <NUM>. The automated assistant server <NUM> can use network access to the internet to search for and obtain an answer to the user <NUM>'s question <NUM>. Continuing with the illustrated example in <FIG>, the automated assistant server <NUM> can obtain an answer to the question <NUM> - "What's the temperature outside?".

In some implementations, the automated assistant server <NUM> can use information associated with the classifier server <NUM> to assist in answering the question. For example, the automated assistant server <NUM> may obtain locational coordinates, a time, and a model type of the classifier server <NUM> to help with answering the question <NUM>. By knowing the location of the classifier server <NUM> and the time, the automated assistant server <NUM> can obtain an answer of "<NUM> degrees Farenheit" from the Internet or other databases.

In some implementations, the automated assistant server <NUM> generates a response <NUM> that includes the answer to the user <NUM>'s question <NUM>. For example, the response includes the answer to the user <NUM>'s question in a sentence-structured format. The response <NUM> may include the statement <NUM> that recites, "The temperature is <NUM> degrees Fahrenheit. " In some implementations, the statement <NUM> may be in a text form or an audio form. The automated assistant server <NUM> transmits the response <NUM> to the classifier server <NUM> over the network <NUM>.

During stage (D), the classifier server <NUM> obtains the response <NUM> from the automated assistant server <NUM>. The classifier server <NUM> may obtain the response <NUM> over the network <NUM>. In response to obtaining the response <NUM>, the classifier server <NUM> determines which client device to send the statement <NUM>. The classifier server <NUM> analyzes a list of one or more client devices connected to the classifier server <NUM>. In some implementations, a client device, such as client device <NUM>, may connect to the classifier server <NUM> via a short-range communication protocol, such as Bluetooth or Wi-Fi. In some implementations, the classifier server <NUM> may send the statement <NUM> to each of the connected devices.

In some implementations, the classifier server <NUM> may transmit the statement <NUM> to the client device <NUM> associated with a user, such as user <NUM>. For example, the classifier server <NUM> may determine from the obtained recorded utterance <NUM> that the speaker is user <NUM>. The classifier server <NUM> may store an audio representation of a user, such as user <NUM>, in a profile each time a new user speaks an utterance likely directed towards the automated assistant server <NUM>. This may be beneficial and improve delay in responding to a user's utterance because the classifier <NUM> can receive an utterance and determine which user is speaking. If an utterance is received does not have a corresponding utterance associated with a user profile, the classifier server <NUM> creates a new user profile and stores the newly obtained utterance with the new user profile in memory.

In some implementations, the classifier server <NUM> may send a message, such as a text message, an email, and a short message service (SMS) message, to name a few examples, to the client device <NUM> with the statement <NUM>. In other implementations, the classifier server <NUM> may play the statement <NUM> out of a speaker connected to the classifier server <NUM>.

The operations of stages (A) to (D) and (A') to (B') illustrate one or more iterations of using the classifier server <NUM> to determine an audience the obtained utterance is likely directed towards. The classifier server <NUM> can repeat the operations of stages (A) to (D) and (A') to (B') for many other iterations. In some implementations, the classifier server <NUM> may perform the operations illustrated in <FIG> simultaneously. Additionally, the classifier server <NUM> may determine the operations illustrated in <FIG> for more utterances than just two utterances at a time, such as utterance <NUM> and utterance <NUM>. <FIG> illustrates two utterances for exemplary purposes only.

<FIG> is a block diagram that illustrates an example of a classification system. In particular, <FIG> illustrates examples of the classifier server <NUM> of the system <NUM> discussed above. In some implementations, the classifier server <NUM> includes a microphone <NUM> for recording the obtained utterances, a speech recognizer <NUM>, an acoustic feature generator, a transcription representation generator, a concatenation module <NUM>, a neural network <NUM>, and a sigma <NUM>. As described in <FIG>, the classifier server <NUM> obtains a recorded utterance <NUM> from a user <NUM>. In some implementations, classifier server <NUM> propagates the recorded utterance <NUM> through the speech recognizer <NUM>, the acoustic feature generator <NUM>, the transcription representation generator <NUM>, the concatenation module <NUM>, the neural network <NUM>, and the sigma <NUM> to judge a likelihood that the audience for the obtained utterance <NUM> is likely directed towards the automated assistant server <NUM>.

The speech recognizer <NUM> may be a device including a combination of hardware, software, and firmware configured to identify words and phrases in a spoken language. In some implementations, the speech recognizer <NUM> converts the obtained utterance <NUM> to a machine-readable format. The machine-readable format may include one or more words in a sentence-structured format that represents the obtained utterance <NUM>. In some implementations, the speech recognizer <NUM> may use various combinations of methodologies to perform speech recognition. For example, the speech recognizer <NUM> may include a Hidden Markov model approach, dynamic time warping (DTW)-based, neural networks, a deep feedforward and recurrent neural network approach, or some combination of the various approaches. The classifier server <NUM> provides the output of the speech recognizer <NUM> to a transcription representation generator <NUM>. Simultaneously, the classifier server <NUM> provides the obtained utterance <NUM> to an acoustic feature generator <NUM>.

In some implementations, the acoustic feature generator <NUM> may be a device including a combination of hardware, software, and firmware configured to extract feature vectors from the obtained utterance <NUM> and provide the extracted feature vectors as input to the recurrent neural network elements. The acoustic feature generator <NUM> analyzes different segments or analysis windows of the obtained utterance <NUM>. These windows can be w<NUM>,. wn, are referred to as frames of the audio. In some implementations, each window or frame represents the same fixed-size amount of audio, e.g., five milliseconds (ms) of audio. The windows may partially overlap or may not overlap. For the obtained utterance <NUM>, a first frame w<NUM> may represent the segment from <NUM> to <NUM>; a second window w<NUM> may represent a segment from <NUM> to <NUM>, and so on.

In some implementations, a feature vector, or a set of acoustic feature vectors, may be determined for each frame of the obtained utterance <NUM>. For example, the acoustic feature generator can perform a Fast Fourier Transform (FFT) on the audio in each window w<NUM>,. wn; map the powers of the spectrum using a mel-frequency scale; take the logarithms of the powers at each mel-frequency; take the discrete cosine transform of the list of mel log powers; and, analyze the amplitude content of the features to determine the acoustic features for each window. The acoustic features may be mel-frequency cepstral coefficients (MFCCs), the features determined using a perceptual linear prediction (PLP) transform, or features determined using other techniques.

The classifier server <NUM> provides the feature vectors one at a time to the recurrent neural network elements in the acoustic feature generator <NUM>. The recurrent neural network elements can be one or more long short-term memory (LSTM) layers. The acoustic feature generator <NUM> may be a deep-layered LSTM neural network architecture built by stacking multiple LSTM layers. The classifier server <NUM> can train the neural network in the acoustic feature generator <NUM> to provide an output of a fixed-size speech unit representation or an embedding. For example, the embedding may be a <NUM>-unit vector. In some implementations, the units may be bits or bytes. One embedding is output for each feature vector.

In some implementations, the classifier server <NUM> includes the acoustic feature generator <NUM> along with the speech recognizer <NUM> to enhance the recognition of the context of the obtained utterance <NUM>. In the illustrated example from <FIG>, after the classifier server <NUM> classifies the question <NUM> and the user <NUM> does not ask a follow on question, sometimes the classifier server <NUM> records future utterances that may contain faint background noise or speech incorrectly recognized by the speech recognition system. For example, without the use of the acoustic feature generator <NUM>, the classifier server <NUM> may transcribe the misrecognitions to common phrases, such as "Stop" or "Play.

Additionally, without the acoustic feature generator <NUM>, the recognized text becomes insufficiently discriminative when the classifier server <NUM> classifies these utterances. By enabling an acoustic feature generator <NUM> in the classifier server <NUM>, the classifier server <NUM> can reduce the failure cases of misrecognitions. In addition, people's utterances include distinctive acoustic elements not captured by the text of the utterance. For example, acoustic elements may include characteristics such as pitch, speech tempo, and accent, to name a few. By including the acoustic feature generator <NUM>, the distinctive acoustic elements can assist in determining whether the audience for the utterance is likely directed towards the automated assistant server <NUM>.

The transcription representation generator <NUM> can include one or more neural network layers. For example, the transcription representation generator <NUM> can include a convolutional neural network (CNN) word-embedding model. Like the acoustic feature generator <NUM>, the transcription representation generator <NUM> can include one or more LSTM layers and may be a deep LSTM neural network architecture build by stacking multiple LSTM layers. In addition, the classifier server <NUM> trains the neural network in the transcription representation generator <NUM> to provide output of a transcription of the obtained utterance <NUM>. In some implementations, the transcription of the utterance includes fixed-size text unit representations or embeddings. For example, each embedding output can be a <NUM>-unit vector. In some implementations, the units may be floating point or integer values. One embedding output from the transcription representation generator <NUM> for each word of the sentence. The transcription includes each of the embedding units provided as output.

In some implementations, the classifier server <NUM> provides input that includes the sentence produced by the speech recognizer <NUM> to the transcription representation generator <NUM>. The classifier server <NUM> inputs one word at a time from the sentence into the CNN word-embedding model of the transcription representation generator <NUM>. In addition, the CNN word-embedding model may max pool the sentence data provided to the CNN word-embedding model to decrease the input data in order to reduce the computational complexity of the network. Max pooling allows for significant reduction in data throughput through the CNN word-embedding model by filtering and averaging the input data. This speeds up the process performed by the transcription representation generator <NUM> without suffering detail in output quality.

In order for the classifier server <NUM> to provide the maximum probability that the obtained utterance <NUM> is directed towards the automated assistant server <NUM>, the classifier server <NUM> utilizes both outputs of the transcription representation generator <NUM> and the acoustic feature generator <NUM>. The concatenation module <NUM> may be a device in software, hardware, firmware, or a combination of each that combines the embedding output from the transcription representation generator <NUM> and the embedding output from the acoustic feature generator <NUM>. For example, the concatenation module <NUM> combines the <NUM>-unit vector output from the transcription representation generator <NUM> and the <NUM>-unit vector output from the acoustic feature generator <NUM> output to create a <NUM>-unit vector output.

In some implementations, the concatenation module <NUM> may create a matrix of <NUM>-unit vector outputs. For example, the matrix may include one or more columns of <NUM>-unit vectors. This concatenation module <NUM> may convert the <NUM>-unit embeddings to a semantically meaningful vector that include one or more numbers. The concatenation module <NUM> converts the <NUM>-unit embeddings to the semantically meaningful vector that includes one or more numbers using one or more functional calls in a programming language, such as word2vec or GloVe.

In the illustrated example, the concatenation module <NUM> may generate a matrix of the numeric vectors from the transcription representation generator <NUM> and a matrix of embeddings from the acoustic feature generator <NUM>. In particular, given a <NUM>-word sentence where each word is associated with a <NUM>-unit embedding, the concatenation module <NUM> may create a 10x100 matrix and concatenate that with the matrix from the acoustic feature generator <NUM>. In the same example, the matrix from the acoustic feature generator <NUM> may include <NUM> feature vectors that each includes a <NUM>-unit embedding. The concatenation module <NUM> may create a 10x64 matrix to concatenate with a 10x100 matrix from the transcription representation generator <NUM>. The resultant matrix created by the concatenation module <NUM> may be 10x164.

The benefit of producing a matrix for the acoustic features of the utterance <NUM> and a matrix for the textual features of the utterance <NUM> is that the dimensions for each respective matrix is such that they can be combined into a proper matrix. For example, each matrix includes the same number of rows, which allow for a horizontal concatenation. Assuming the number of columns between respective matrices is similar, the concatenation module <NUM> would create a vertical concatenation. In some implementations, the concatenation module <NUM> provides the concatenated matrix to the neural network <NUM>.

In some implementations, the concatenation module <NUM> may buffer the number of <NUM>-unit vectors in the matrix until the neural network <NUM> has processed one <NUM>-unit vector. Once the neural network <NUM> processes one <NUM>-unit vector, the concatenation module <NUM> provides the next <NUM>-unit vector into the neural network <NUM>. The speech recognizer <NUM>, the acoustic feature generator <NUM>, the transcription representation generator <NUM>, and the concatenation module <NUM> may create <NUM>-unit vectors faster than the neural network <NUM> can process one <NUM>-unit vector. Therefore, the concatenation module <NUM> creates a matrix buffer of <NUM>-unit vectors to store and ready to be processed.

In some implementations, classifier server <NUM> includes a neural network <NUM> to process the concatenated matrix. In particular, the neural network <NUM> includes a classifier <NUM>-A and another LSTM <NUM>-B. The classifier server <NUM> can train the classifier <NUM>-A and the LSTM <NUM>-B to produce an output that indicates a likelihood that the audience for the obtained utterance <NUM> is likely directed towards the automated assistant server <NUM>. In some implementations, the classifier <NUM>-A and the LSTM <NUM>-B are trained using examples of queries previously spoken by users and recorded dialog not directed towards the automated assistant server <NUM>. For example, the classifier server <NUM> may retrieve multiple phrases spoken and in text format not directed towards the automated assistant server <NUM>.

In some implementations, the classifier server <NUM> may retrieve other phrases from one or more databases across the internet that include phrases not directed towards the automated assistant server <NUM>. For example, one of the phrases may include "Bob said, what's the temperature outside, and I shook" or "What time are your parents coming over?" Generally, the audience for these types of questions is another individual in the room, even though the classifier server <NUM> may interpret the audience to be likely directed towards the automated assistant server <NUM> for these types of questions. However, these types of questions are the optimal phrases to use for training the neural network <NUM>. The classifier <NUM>-A can learn to identify phrases that closely identify and seem to include an audience likely directed towards the automated assistant server <NUM>, yet are actually background noises, or directed towards other individuals in a room. Such examples include, "What time are your parents coming over," "How much do you weigh," or "What did you buy at the grocery store?" Each of these questions do not include an identifier indicating who the speaker is talking to but do include a pronoun that may indicate to the classifier <NUM>-A to look away when identifying phrases not directed towards the automated assistant server <NUM>.

In some implementations, the classifier server <NUM> updates the weights of the classifier <NUM>-A and the weights of the LSTM <NUM>-B during training. For example, the classifier server <NUM> can update the weights of the classifier <NUM>-A and the LSTM <NUM>-B using back-propagation of errors through time with stochastic gradient descent.

In some implementations, the output of the classifier <NUM>-A and the LSTM <NUM>-B can include an indication that the audience for the obtained utterance <NUM> was likely directed towards the automated assistant server <NUM>. For example, the indication may include a probability that the output of the <NUM>-unit vector indicates the audience for the obtained utterance is likely directed towards the automated assistant server <NUM>. In other implementations, the output of the classifier <NUM>-A and the LSTM 210B collectively may include a score ranging from <NUM> to <NUM>.

In some implementations, the sigma <NUM> sums each of the outputs provided by the LSTM <NUM>-B. For example, the sigma <NUM> receives output probabilities or output scores for each of the <NUM>-unit vectors for each utterance that propagates through the neural network <NUM>. The sigma <NUM> cumulatively sums each output probability or score from the neural network <NUM> for the entire obtained utterance <NUM>.

In some implementations, the sigma <NUM> compares the final output probability or score to a predetermined threshold. If the sigma <NUM> determines the final output probability or score exceeds the predetermined threshold, then the classifier server <NUM> indicates a likelihood that the audience for the obtained utterance <NUM> is likely directed towards the automated assistant server <NUM>. Alternatively, the classifier server <NUM> indicates a likelihood that the audience for the obtained utterance <NUM> is likely directed towards the automated assistant server <NUM> if the final output probability or score is below the predetermined threshold. For example, the predetermined threshold may be a probability of <NUM>% or a score of <NUM>/<NUM>.

In some implementations, the classifier server <NUM> may determine from the output of the sigma <NUM> whether to provide the obtained utterance <NUM> to the automated assistant server <NUM>. For example, should the classifier server <NUM> determine that the final output probability or score exceeds the predetermined threshold, the classifier server <NUM> generates instructions to provide to the automated assistant server <NUM>. The instructions convey to the automated assistant server <NUM> to process the obtained utterance <NUM>. The classifier server <NUM> packages the obtained utterance <NUM> and the generated instructions into data <NUM>, and transmits the data <NUM> to the automated assistant server <NUM> for processing.

Alternatively, the classifier server <NUM> may determine the final output probability or score does not exceed the predetermined threshold. In response, the classifier server <NUM> may delete the obtained utterance <NUM> from memory and proceed to wait until a new obtained utterance. In other implementations, the classifier server <NUM> may generate instructions for the automated assistant server <NUM> indicating to not process the obtained utterance <NUM>. In addition, the instructions may convey to the automated assistant server <NUM> to store the obtained utterance <NUM> in the database of stored utterances not directed towards the automated assistant server <NUM>. The classifier server <NUM> packages the obtained utterance <NUM> and the generated instructions into data <NUM>, and transmits the data to the automated assistant server <NUM> for processing.

<FIG> is a flow diagram that illustrates an example of a process <NUM> for speech classification. One or more computers, such as one or more computers of the classifier server <NUM>, may perform the process <NUM>.

In the process <NUM>, the one or more computers receive audio data corresponding to an utterance (<NUM>). For example, the classifier server <NUM> obtains a spoken utterance from a user <NUM>. The spoken utterance can include various types of phrase and/or questions directed towards the automated assistant server <NUM>. In other implementations, the classifier server <NUM> can obtain one or more spoken utterances from the user <NUM> not directed towards the automated assistant server <NUM>. In other implementations, the classifier server <NUM> may obtain audio utterances from various devices located near the classifier server <NUM>. For example, the classifier server <NUM> may listen to and record one or more individuals communicating in a television program as displayed by the television. In another example, the classifier server <NUM> may listen and record music played from speakers in audible range of the classifier server <NUM>.

The one or more computers provide transcription for an utterance (<NUM>). For example, the speech recognizer <NUM> included in the classifier server <NUM> converts the obtained utterance <NUM> to a machine-readable format. The machine-readable format may include one or more words in a structured format that represents the obtained utterance <NUM>. The classifier server <NUM> provides input that includes the sentence produced by the speech recognizer <NUM> to the transcription representation generator <NUM>.

The one or more computers generate a representation of the audio data and a representation of the transcription of the utterance (<NUM>). For example, the classifier server <NUM> includes an acoustic feature generator <NUM> configures to extract feature vectors from the obtained utterance <NUM> and provide the extracted feature vectors as input to the recurrent neural network elements in the acoustic feature generator <NUM>. In particular, the classifier server <NUM> provides the extracted feature vectors one at a time to the recurrent neural network elements.

In some implementations, the classifier server <NUM> can train the recurrent neural network in the acoustic feature generator <NUM> to provide output of a fixed-size speech representation or an embedding. For example, the embedding may be a <NUM>-unit vector. In some implementations, the units may be bits or bytes. One embedding is output for each feature vector.

In some implementations, the classifier server <NUM> inputs one word at a time from the sentence into a CNN word-embedding model of the transcription representation generator <NUM>. The classifier server <NUM> can train the neural network in the transcription representation generator <NUM> to provide output of a transcription of the obtained utterance <NUM>. The transcriptions of the utterance include fixed-size text unit representations or embeddings. For example, each embedding output can be a <NUM>-unit vector. The transcription includes each of the embedding units provided as output.

The one or more computers provide (i) the representation of the audio data and (ii) the representation of the utterance to a classifier (<NUM>). For example, the concatenation module <NUM> converts each of the <NUM>-unit embedding to a semantically meaning vector that includes one or more numbers. The concatenation modules <NUM> combines the embedding output of the transcription representation generator <NUM> and the embedding output from the acoustic feature generator <NUM>. For example, the concatenation module <NUM> combines the <NUM>-unit vector output from the transcription representation generator <NUM> and the <NUM>-unit vector output from the acoustic feature generator <NUM> output to create a <NUM>-unit vector output.

In some implementations, the concatenation module <NUM> may generate a matrix of the numeric vectors from the transcription representation generator <NUM> and a matrix of embeddings from the acoustic feature generator <NUM>. In particular, given a <NUM>-word sentence where each word is associated with a <NUM>-unit embedding, the concatenation module <NUM> may create a 10x100 matrix and concatenate that with the matrix from the acoustic feature generator <NUM>. In the same example, the matrix from the acoustic feature generator <NUM> may include <NUM> feature vectors that each includes a <NUM>-unit embedding. The concatenation module <NUM> may create a 10x64 matrix to concatenate with a 10x100 matrix from the transcription representation generator <NUM>. The resultant matrix created by the concatenation module <NUM> may be of size 10x164. In some implementations, the concatenation module <NUM> provides the concatenated matrix to the neural network <NUM>, one <NUM>-unit vector at a time.

The one or more computers receive an indication of whether the audience for the utterance is likely directed towards the automated assistant (<NUM>). For example, the classifier server <NUM> includes a neural network <NUM> to process the concatenated matrix. In particular, the neural network <NUM> includes a classifier <NUM>-A and another LSTM <NUM>-B. The classifier server <NUM> can train the classifier <NUM>-A and the LSTM <NUM>-B to produce an output that indicates the audience for the obtained utterance <NUM> is likely directed towards the automated assistant server <NUM>. In some implementations, the classifier <NUM>-A and the LSTM <NUM>-B are trained using examples of queries previously spoken by users and recorded dialog not directed towards the automated assistant server <NUM>.

In some implementations, the output of the classifier <NUM>-A and the LSTM <NUM>-B can include an indication that the audience for the obtained utterance is likely directed towards the automated assistant server <NUM>. For example, the indication may include a probability of how likely audience for the output of the <NUM>-unit vector is likely directed towards the automated assistant server <NUM>. In other implementations, the output of the classifier <NUM>-A and the LSTM 210B collectively may include a score ranging from <NUM> to <NUM>.

The one or more computers selectively instruct the automated assistant based on the indication that the utterance corresponding to the received audio data is likely directed to the automated assistant (<NUM>). For example, the classifier server <NUM> includes a sigma <NUM> at the output of the neural network <NUM>. The sigma <NUM> sums each of the outputs provided by the LSTM <NUM>-B. For example, the sigma <NUM> receives output probabilities or output scores for each of the <NUM>-unit vectors for each utterance that propagates through the neural network <NUM>. The sigma <NUM> cumulatively sums each output probability or score from the neural network <NUM> until the sigma <NUM> entirely processes the obtained utterance <NUM>.

In some implementations, the sigma <NUM> compares the final output probability or score to a predetermined threshold. If the sigma <NUM> determines the final output probability or score exceeds the predetermined threshold, then the classifier server <NUM> indicates a likelihood that the audience for the obtained utterance <NUM> is likely directed towards the automated assistant server <NUM>. Alternatively, the classifier server <NUM> indicates a likelihood that the audience for the obtained utterance <NUM> is likely directed towards the automated assistant server <NUM> if the final output probability or score is below the predetermined threshold.

In some implementations, the classifier server <NUM> may determine from the output of the sigma <NUM> whether to provide the obtained utterance <NUM> to the automated assistant server <NUM>. For example, should the classifier server <NUM> determine that the final output probability or score exceeds the predetermined threshold, the classifier server <NUM> generates instructions to provide to the automated assistant server <NUM>. The instructions instruct the automated assistant server <NUM> to process the obtained utterance <NUM>. The classifier server <NUM> packages the obtained utterance <NUM> and the generated instructions into data <NUM>, and transmits the data <NUM> to the automated assistant server <NUM> for processing.

Alternatively, the classifier server <NUM> may determine the final output probability or score does not exceed the predetermined threshold. In response, the classifier server <NUM> may delete the obtained utterance <NUM> from memory and proceed to wait until the next obtained utterance.

<FIG> shows an example of a computing device <NUM> and a mobile computing device <NUM> that can be used to implement the techniques described here.

The mobile computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

The computing device <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, a high-speed interface <NUM> connecting to the memory <NUM> and multiple high-speed expansion ports <NUM>, and a low-speed interface <NUM> connecting to a low-speed expansion port <NUM> and the storage device <NUM>. Each of the processor <NUM>, the memory <NUM>, the storage device <NUM>, the high-speed interface <NUM>, the high-speed expansion ports <NUM>, and the low-speed interface <NUM>, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a GUI on an external input/output device, such as a display <NUM> coupled to the high-speed interface <NUM>. Also, multiple computing devices may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The high-speed interface <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low-speed interface <NUM> manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface <NUM> is coupled to the memory <NUM>, the display <NUM> (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports <NUM>, which may accept various expansion cards (not shown). In the implementation, the low-speed interface <NUM> is coupled to the storage device <NUM> and the low-speed expansion port <NUM>. The low-speed expansion port <NUM>, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The memory <NUM> stores information within the mobile computing device <NUM>. An expansion memory <NUM> may also be provided and connected to the mobile computing device <NUM> through an expansion interface <NUM>, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory <NUM> may provide extra storage space for the mobile computing device <NUM>, or may also store applications or other information for the mobile computing device <NUM>. Specifically, the expansion memory <NUM> may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory <NUM> may be provided as a security module for the mobile computing device <NUM>, and may be programmed with instructions that permit secure use of the mobile computing device <NUM>.

The memory may include, for example, flash memory and/or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier, such that the instructions, when executed by one or more processing devices (for example, processor <NUM>), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory <NUM>, the expansion memory <NUM>, or memory on the processor <NUM>). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver <NUM> or the external interface <NUM>.

The mobile computing device <NUM> may communicate wirelessly through the communication interface <NUM>, which may include digital signal processing circuitry where necessary. The communication interface <NUM> may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication may occur, for example, through the transceiver <NUM> using a radio-frequency. In addition, a GPS (Global Positioning System) receiver module <NUM> may provide additional navigation- and location-related wireless data to the mobile computing device <NUM>, which may be used as appropriate by applications running on the mobile computing device <NUM>.

Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

Although a few implementations have been described in detail above, other modifications are possible. For example, while a client application is described as accessing the delegate(s), in other implementations the delegate(s) may be employed by other applications implemented by one or more processors, such as an application executing on one or more servers. In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other actions may be provided, or actions may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.

Claim 1:
A computer-implemented method comprising:
receiving audio data corresponding to an utterance;
obtaining a transcription of the utterance;
generating a representation of the audio data;
generating a representation of the transcription of the utterance;
providing (i) the representation of the audio data and (ii) the representation of the transcription of the utterance to a classifier that, based on a given representation of audio data and a given representation of a transcription of an utterance, is trained to output an indication of whether the utterance associated with the given representation is likely directed to an automated assistant or is likely not directed to an automated assistant;
receiving, from the classifier, an indication of whether the utterance corresponding to the received audio data is likely directed to the automated assistant or is likely not directed to the automated assistant; and
selectively instructing the automated assistant based at least on the indication of whether the utterance corresponding to the received audio data is likely directed to the automated assistant or is likely not directed to the automated assistant, wherein the classifier includes one or more neural network elements to process the spoken utterance, wherein the classifier trains the neural network elements with examples of queries previously spoken by users not likely directed towards the automated assistant server.