Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining, for each of multiple words or sub-words, audio data corresponding to multiple users speaking the word or sub-word; training, for each of the multiple words or sub-words, a pre-computed hotword model for the word or sub-word based on the audio data for the word or sub-word; receiving a candidate hotword from a computing device; identifying one or more pre-computed hotword models that correspond to the candidate hotword; and providing the identified, pre-computed hotword models to the computing device.

FIELD

The present disclosure generally relates to speech recognition.

BACKGROUND

The reality of a speech-enabled home or other environment—that is, one in which a user need only speak a query or command out loud and a computer-based system will field and answer the query and/or cause the command to be performed—is upon us. A speech-enabled environment (e.g., home, workplace, school, etc.) can be implemented using a network of connected microphone devices distributed throughout the various rooms or areas of the environment. Through such a network of microphones, a user has the power to orally query the system from essentially anywhere in the environment without the need to have a computer or other device in front of him/her or even nearby. For example, while cooking in the kitchen, a user might ask the system “how many milliliters in three cups?” and, in response, receive an answer from the system, e.g., in the form of synthesized voice output. Alternatively, a user might ask the system questions such as “when does my nearest gas station close,” or, upon preparing to leave the house, “should I wear a coat today?”

Further, a user may ask a query of the system, and/or issue a command, that relates to the user's personal information. For example, a user might ask the system “when is my meeting with John?” or command the system “remind me to call John when I get back home.”

In a speech-enabled environment, a user's manner of interacting with the system is designed to be primarily, if not exclusively, by means of voice input. Consequently, a system which potentially picks up all utterances made in the environment, including those not directed to the system, must have some way of discerning when any given utterance is directed at the system as opposed, e.g., to being directed an individual present in the environment. One way to accomplish this is to use a “hotword” (also referred to as an “attention word” or “voice action initiation command”), which by agreement is reserved as a predetermined term that is spoken to invoke the attention of the system.

In one example environment, the hotword used to invoke the system's attention is the word “Google.” Consequently, each time the word “Google” is spoken, it is picked up by one of the microphones, and is conveyed to the system, which performs speech recognition techniques to determine whether the hotword was spoken and, if so, awaits an ensuing command or query. Accordingly, utterances directed at the system take the general form [HOTWORD] [QUERY], where “HOTWORD” in this example is “Google” and “QUERY” can be any question, command, declaration, or other request that can be speech recognized, parsed and acted on by the system, either alone or in conjunction with a server over network.

SUMMARY

According to some innovative aspects of the subject matter described in this specification, a system can provide pre-computed hotword models to a mobile computing device, such that the mobile computing device can detect a candidate hotword spoken by a user associated with the mobile computing device through an analysis of the acoustic features of a portion of an utterance, without requiring the portion to be transcribed or semantically interpreted. The hotword models can be generated based on audio data obtained from multiple users speaking multiple words or sub-words, including words or sub-words that make up the candidate hotword.

In some examples, a user desires to make the words “start computer” a hotword to initiate a “wake up” process on a mobile computing device, such as a smartphone. The user speaks the words “start computer” and, in response, the system can identify pre-computed hotword models associated with the term “start computer,” or of the constituent words “start” and “computer.” The system can provide the pre-computed hotword models to the mobile computing device such that the mobile computing device can detect whether a further utterance corresponds to the hotword “start computer,” and correspondingly wake up the mobile computing device.

Innovative aspects of the subject matter described in this specification may be embodied in methods that include the actions of obtaining, for each of multiple words or sub-words, audio data corresponding to multiple users speaking the word or sub-word; training, for each of the multiple words or sub-words, a pre-computed hotword model for the word or sub-word based on the audio data for the word or sub-word; receiving a candidate hotword from a computing device; identifying one or more pre-computed hotword models that correspond to the candidate hotword; and providing the identified, pre-computed hotword models to the computing device.

These and other embodiments may each optionally include one or more of the following features. For instance, identifying the one or more pre-computed hotword models includes obtaining two or more sub-words that correspond to the candidate hotword; and obtaining, for each of the two or more sub-words that correspond to the candidate hotword, a pre-computed hotword model that corresponds to the sub-word. Receiving the candidate hotword from the computing device after training, for each of the multiple words or sub-words, the pre-computed hotword model for the word or sub-word. Receiving the candidate hotword from the mobile computing includes receiving audio data corresponding to the candidate hotword. Receiving the candidate hotword from the computing device includes receiving the candidate hotword including two or more words from the computing device. Identifying one or more pre-computed hotword models that correspond to each word of the two or more words of the candidate hotword; and providing the identified, pre-computed hotword models that correspond to each word of the two or more words of the candidate hotword to the computing device. Providing, to the computing device, instructions defining a processing routine of the identified, pre-computed hotword models that correspond to each word of the two or more words of the candidate hotword. The instructions include instructions to sequentially process the identified, pre-computed hotword models that correspond to each word of the two or more words of the candidate hotword. The instructions include a processing order to sequentially process the identified, pre-computed hotword models that correspond to each word of the two or more words of the candidate hotword.

The features further include, for instance, dynamically creating one or more hotword models that correspond to one or more words of the two or more words of the candidate hotword; and providing the dynamically created one or more hotword models that correspond to one or more words of the two or more words of the candidate hotword to the computing device. Dynamically creating the one or more hotword models that correspond to the one or more words of the two or more words of the candidate hotword after receiving the candidate hotword from the computing device. Training, for each of the multiple words or sub-words, the pre-computed hotword model for the word or sub-word, further includes for each word or sub-word of the multiple words or sub-words, obtaining, for each user of the multiple users, a transcription of the audio data of the user speaking the word or sub-word; associating, for each user of the multiple users, the audio data of the user speaking the word or sub-word with the transcription of the audio data of the user speaking the word or sub-word; and generating a particular pre-computed hotword model corresponding to the word or sub-word based on i) the audio data corresponding to each of the multiple users speaking the word or sub-word and ii) the transcription associated with the audio data corresponding to each of the multiple users speaking the word or sub-word.

In the drawings, like reference symbols indicate like elements throughout.

DETAILED DESCRIPTION

FIG. 1depicts a system100for providing pre-computed hotword models. In some examples, the system100includes mobile computing devices102,104,106, a speech recognition engine108, a vocabulary database110, and a hotword configuration engine112. In some examples, any of the mobile computing devices102,104,106may be a portable computer, a smartphone, a tablet-computing device, or a wearable computing device. Each of the mobile computing devices102,104,106are associated with a respective user114,116,118. The mobile computing devices102,104,106can include any audio detection means, e.g., a microphone, for detecting utterances from the respective associated user114,116,118. The mobile computing devices102and104are in communication with the speech recognition engine108, e.g., over one or more networks, and the mobile computing device106is in communication with the hotword configuration engine112, e.g., over one or more networks.

In some implementations, the speech recognition engine108obtains, for each of multiple words or sub-words, audio data that corresponds to multiple users speaking the word or sub-words, during operation (A). Specifically, the speech recognition engine108obtains audio data from the mobile computing devices102and104that correspond, respectively, to the users114and116speaking words or sub-words, e.g., over one or more networks. In some examples, the user114and the user116each say one or more words that the mobile computing device102and mobile computing device104, respectively detect. In some examples, the users114and116speak the words or sub-words during any interaction with the mobile computing devices102and104, respectively, e.g., submitting voice queries for voice commands. In some examples, in addition to obtain the audio data associated with the users114and116speaking the words or sub-words, the speech recognition engine108obtains a locale of the users114and116from the mobile computing device102and104. The locale can include the approximate current location of the user when speaking the words or sub-words, or a location associated with a profile of the user.

For example, the user114says the utterance150of “start my car” and the user116says the utterance152of “where do I buy a computer?” The mobile computing device102detects the utterance150of “start my car” to generate waveform data120that represents the detected utterance150; and the mobile computing device104detects the utterance152of “where do I buy a computer?” to generate waveform data122that represents the detected utterance152. The mobile computing devices102and104transmit the waveforms120and122, respectively, to the speech recognition engine108, e.g., over one or more networks.

In some examples, the speech recognition engine108, for each word or sub-word of the multiple words or sub-words, obtains, for each user of the multiple users, a transcription of the audio data of the user speaking the word or the sub-word, during operation (B). Specifically, the speech recognition engine108processes the received audio data, including generating a transcription of an utterance of words or sub-words associated with the audio data. Generating the transcription of the audio data of the user speaking the word or the sub-word can include transcribing the utterance into text or text-related data. In other words, the speech recognition engine108can provide a representation of natural language in written form of the utterance associated with the audio data. For example, the speech recognition engine108transcribes the waveforms120and122, as received from the mobile computing devices102and104, respectively. That is, the speech recognition engine108transcribes the waveform120to generate the transcription124of “start my car” and transcribes the waveform122to generate the transcription126of “where do I buy a computer?”

In some examples, the speech recognition engine108, for each word or sub-word of the multiple words or sub-words, associates, for each user of the multiple users, the audio data of the user speaking the word or sub-word with the transcription of the audio data of the user speaking the word or sub-word, during operation (C). For example, the speech recognition engine108associates the waveform160with the transcription124and associates the waveform162with the transcription126. In some examples, the waveform160is substantially the same as the waveform120, and the waveform162is substantially the same as the waveform122. In some examples, the waveform160is a processed version (e.g., by the speech recognition engine108) of the waveform120, and the waveform162is a processed version (e.g., by the speech recognition engine108) of the waveform122.

In some examples, the speech recognition engine108associates a portion of the waveform160with a corresponding portion of the transcription124. That is, for each word, or sub-word, of the waveform160, the speech recognition engine108associates the corresponding portion of the transcription124with the word, or sub-word. For example, the speech recognition engine108associates a portion of the waveform160for each of the words “start,” “my,” “car” with the corresponding portion of the transcription124. Similarly, the speech recognition engine108associates a portion of the waveform162for each of the words “where,” “do,” “I,” “buy,” “a,” “computer” with the corresponding portion of the transcription126. In some examples, the speech recognition engine108associates a portion of the waveform160for each of the sub-words, e.g., phoneme or tri-phone level, of each word, e.g., “st-ah-rt” of the word “start,” with the corresponding portion of the transcription. Similarly, in some examples, the speech recognition engine108associates a portion of the waveform162for each of the sub-words, e.g., phoneme or tri-phone level, of each word, e.g., “kom-pyu-ter” of the word “computer,” with the corresponding portion of the transcription.”

In some examples, associating the audio data of the user speaking the word or sub-word with the transcription of the audio data of the user speaking the word or sub-word includes storing the association in a database, or a table. Specifically, the speech recognition engine108provides the transcription124and the waveform160to the vocabulary database110such that the vocabulary database110stores an association between the waveform160and the transcription124. Similarly, the speech recognition engine108provides the transcription126and the waveform162to the vocabulary database110such that the vocabulary database110stores an association between the waveform162and the transcription126.

In some examples, the speech recognition engine108provides the locale associated with the word or sub-words of the transcription124(e.g., the locale of the user114) to the vocabulary database110such that the vocabulary database110additionally stores an association between the waveform160, the transcription124, and the respective locale. Similarly, the speech recognition engine108provides the locale associated with the word or sub-words of the transcription126(e.g., the locale of the user116) to the vocabulary database110such that the vocabulary database110additionally stores an association between the waveform162, the transcription126, and the respective locale

In some examples, the vocabulary database110indicates an association between a portion of the waveform160with a corresponding portion of the transcription124. That is, for each word, or sub-word, of the waveform160, the vocabulary database110stores an association of the portion of the waveform160with a corresponding portion of the transcription124with the word, or sub-word. For example, the vocabulary database110stores an association of a portion of the waveform160for each of the words “start,” “my,” “car” with the corresponding portion of the transcription124. Similarly, the vocabulary database110stores an association of a portion of the waveform162for each of the words “where,” “do,” “I,” “buy,” “a,” “computer” with the corresponding portion of the transcription126.

In some implementations, the hotword configuration engine112trains, for each of the multiple words or sub-words, a pre-computed hotword model for the word or sub-word, during operation (D). Specifically, the hotword configuration engine112is in communication with the vocabulary database110, and obtains, for each word or sub-word stored by the vocabulary database110, the audio data of each user of the multiple users speaking the word or sub-word and the associated transcription of the audio data. For example, the hotword configuration engine112obtains, from the vocabulary database110, the waveform160and the associated transcription124, and further obtains the waveform162and the associated transcription126.

In some examples, for each word or sub-word stored by the vocabulary database110, the hotword configuration engine112generates a pre-computed hotword model corresponding to the word or sub-word. Specifically, the hotword configuration engine112generates the pre-computed hotword model for each word or sub-word based on i) the audio data corresponding to each of the multiple users speaking the word or sub-word and ii) the transcription associated with the audio data corresponding to each of the multiple users speaking the word or sub-word. In some examples, the pre-computed hotword model can be a classifier such as a neural network, or a support vector machine (SVM).

For example, the hotword configuration engine112generates a pre-computed hotword model corresponding to each word or sub-word of the waveforms160and162. In some examples, for the word “start” of the waveform160, the hotword configuration engine112generates a pre-computed hotword model for the word based on i) the audio data corresponding to the user114speaking the word “start” (e.g., the portion of the waveform160corresponding to the user114speaking the word “start”) and ii) the transcription associated with the audio data corresponding to the user114speaking the word “start.” Additionally, the hotword configuration engine112can generate a pre-computed hotword model for the remaining words “my” and “car” of the waveform160, as well as each sub-word (of each word) of the waveform160, e.g., “st-ah-rt” of the word “start.”

Additionally, in some examples, for the word “computer” of the waveform162, the hotword configuration engine112generates a pre-computed hotword model for the word based on i) the audio data corresponding to the user116speaking the word “computer” (e.g., the portion of the waveform162corresponding to the user116speaking the word “computer”) and ii) the transcription associated with the audio data corresponding to the user116speaking the word “computer.” Additionally, the hotword configuration engine112can generate a pre-computed hotword model for the remaining words “where,” “do,” “I,” “buy” and “a” of the waveform162, as well as each sub-word of the waveform160, e.g., “kom-pyu-ter” of the word “computer.”

The hotword configuration engine112, after pre-computing the hotword models for one or more words stored by the vocabulary database110, provides the pre-computed hotword models128to the vocabulary database110such that the vocabulary database110stores or otherwise indicates an association between the words or sub-words and the corresponding pre-computed hotword models128. That is, for each word or sub-word of the waveforms160and162, the vocabulary database110stores an association between each of the words or sub-words (e.g., of the waveforms160and162) and the corresponding pre-computed hotword models128. In some examples, the vocabulary database110stores, for each word or sub-word of the waveforms160and162, an association between i) the portion of the waveform corresponding to the word or sub-word, ii) the corresponding transcription of the portion of the waveform, and iii) the corresponding pre-computed hotword model. For example, for the word “start” of the waveform160, the vocabulary database110stores i) an association of a portion of the waveform160corresponding to the word “start,” ii) a portion of the transcription124corresponding to the word “start,” and iii) a pre-computed hotword model128for the word “start.”

In some implementations, the hotword configuration engine112receives a candidate hotword129from the mobile computing device106, during operation (E). Specifically, the hotword configuration engine112receives, e.g., over one or more networks, data from the mobile computing device106that corresponds to the user118providing the candidate hotword129. In some examples, the mobile computing device106provides a graphical user interface180to the user118that provides for display of text182to prompt the user118to provide a hotword. For example, the text182includes “Please say your desired Hotword.” In response, the user118says the candidate hotword129that the mobile computing device106detects, and transmits to the hotword configuration engine112. For example, the user118says the utterance170of “start computer” that corresponds to the candidate hotword129. The mobile computing device106detects the utterance170of “start computer” and generates a waveform130that represents the detected utterance170. The mobile computing device106transmits the waveform130to the hotword configuration engine112, e.g., over one or more networks.

In some examples, the user118provides text-based input to the mobile computing device106, e.g., via a graphical user interface of the mobile computing device106, that corresponds to the candidate hotword129. For example, the user118inputs via a keyboard, virtual or tactile, the text of “start computer.” The mobile computing device106transmits the text-based candidate hotword129of “start computer” to the hotword configuration engine112, e.g., over one or more networks.

In some examples, the hotword configuration engine112receives the candidate hotword from the mobile computing device106after training, for each of the multiple words or sub-words, the pre-computed hotword model for the word or sub-word. Specifically, the hotword configuration engine112receives the candidate hotword129from the mobile computing device106after the hotword configuration engine112generates the pre-computed hotword models128corresponding to each of the words or sub-words stored by the vocabulary database110. For example, the hotword configuration engine112receives the candidate hotword129of “start computer” from the mobile computing device106after training, for each of the multiple words or sub-words of the waveforms160and162, the pre-computed hotword models128for the word or sub-word.

In some examples, the hotword configuration engine112receives the candidate hotword129that includes two or more words from the mobile computing device106. For example, the hotword configuration engine112receives the candidate hotword129of “start computer” that includes two words (e.g., “start” and “computer”). In some examples, the hotword configuration engine112receives the candidate hotword129that includes a single word from the mobile computing device106.

In some examples, the hotword configuration engine112obtains two or more sub-words that correspond to the candidate hotword129. That is, the hotword configuration engine112processes the candidate hotword129to identify sub-words of the candidate hotword129. For example, for the candidate hotword129of “start computer,” the hotword configuration engine112can obtain the sub-words of “st-ah-rt” for the word “start” of the candidate hotword129and further obtain the sub-words of “kom-pyu-ter” for the word “computer” of the candidate hotword129.

In some implementations, the hotword configuration engine112identifies one or more pre-computed hotword models that correspond to the candidate hotword129, at operation (F). Specifically, the hotword configuration engine112accesses the vocabulary database110to identify one or more of the pre-computed hotword models128that are stored by the vocabulary database110and that correspond to the candidate hotword129. The hotword configuration engine112retrieves the pre-computed hotword models128from the vocabulary database110, e.g., over one or more networks. In some examples, the hotword configuration engine112identifies the pre-computed hotword models128that are associated with the words, or sub-words, of the candidate hotword129. The hotword configuration engine112can identify the pre-computed hotword models128by matching the words, or sub-words, of the candidate hotword129with the words, or sub-words, that are stored by the vocabulary database110.

In some examples, the hotword configuration engine112identifies the pre-computed hotword models128that correspond to the utterance170of the candidate hotword129provided by the user118. That is, the hotword configuration engine112identifies the one or more pre-computed hotword models128based on the waveform130that represents the detected utterance170of the candidate hotword129. In the illustrated example, the hotword configuration engine112identifies one or more pre-computed hotword models128stored by the vocabulary database110that correspond to the utterance170of “start computer.”

In some examples, when the candidate hotword includes two or more words, the hotword configuration engine112identifies the pre-computed hotword models that correspond to each word of the two or more words. That is, each word of the two or more words of the candidate hotword129corresponds to a pre-computed hotword model128stored by the vocabulary database110. For example, the candidate hotword129includes two words, e.g., “start” and “computer.” To that end, the hotword configuration engine112identifies a first pre-computed hotword model128stored by the vocabulary database110corresponding to the word “start” and a second pre-computed hotword model128stored by the vocabulary database110corresponding to the word “computer.” In some examples, the hotword configuration engine112identifies a pre-computed hotword model128stored by the vocabulary database110corresponding to the both words “start computer.”

In some examples, the hotword configuration engine112identifies the one or more pre-computed hotword models128that correspond to the utterance170of the candidate hotword129by matching at least a portion of the waveform130to at least a portion of waveforms stored by the vocabulary database110. Matching the waveform130to waveforms stored by the vocabulary database110can include performing an audio-based comparison between the waveform130and the waveforms stored by the vocabulary database110to identify a matching waveform stored by the vocabulary database110to the waveform130. In some examples, the audio-based comparison between the waveform130and the waveforms stored by the vocabulary database110can be performed by an audio processing engine that is in communication with the hotword configuration engine112, e.g., over one or more networks. To that end, upon the hotword configuration engine112identifying a matching waveform stored by the vocabulary database110to the waveform130, the hotword configuration engine112identifies the pre-computed hotword models128associated with the matching waveform.

In some examples, the hotword configuration engine112identifies the one or more pre-computed hotword models128that correspond to the utterance170of the candidate hotword129by applying one or more of the pre-computed hotword models128stored by the vocabulary database110to the utterance170to identify the pre-computed hotword models128corresponding with a highest confidence score relative to the remaining pre-computed hotword models128. The confidence score indicates the likelihood that the identified pre-computed hotword model128corresponds to the utterance170.

For example, the hotword configuration engine112can match the waveform130to a portion of one or more of the waveforms160and162that are stored by the vocabulary database110. Specifically, the hotword configuration engine112can match a portion of the waveform130that corresponds to the word “start” with a portion of the waveform160stored by the vocabulary database110that corresponds to the word “start.” Based on this matching, the hotword configuration engine112can identify the corresponding pre-computed hotword model128that is associated with the portion of the waveform160for the word “start.” Similarly, the hotword configuration engine112can match a portion of the waveform130that corresponds to the word “computer” with a portion of the waveform162stored by the vocabulary database110that corresponds to the word “computer.” Based on this matching, the hotword configuration engine112can identify the corresponding pre-computed hotword model128that is associated with the portion of the waveform162for the word “computer.”

In some examples, the hotword configuration engine112identifies the one or more pre-computed hotword models128that correspond to the utterance of the candidate hotword129by matching at least a portion of a transcription of the waveform130to at least a portion of transcriptions stored by the vocabulary database110. Specifically, the hotword configuration engine112can provide the waveform130to a speech recognition engine, e.g., the speech recognition engine108, such that speech recognition engine108transcribes the waveform130. To that end, matching the transcription of the waveform130to transcriptions stored by the vocabulary database110can include comparing the transcription of the waveform130to the transcriptions stored by the vocabulary database110to identify a matching transcription stored by the vocabulary database110to the waveform130. To that end, upon the hotword configuration engine112identifying a matching transcription stored by the vocabulary database110to the transcription of the waveform130, the hotword configuration engine112identifies the pre-computed hotword models128associated with the matching transcription.

For example, the hotword configuration engine112can match the transcription of the waveform130to a portion of one or more of the transcriptions124and126that are stored by the vocabulary database110. Specifically, the hotword configuration engine112can match a portion of the transcription of the waveform130that corresponds to the word “start” with a portion of the transcription124stored by the vocabulary database110that corresponds to the word “start.” Based on this matching, the hotword configuration engine112can identify the corresponding pre-computed hotword model128that is associated with the portion of the transcription124for the word “start.” Similarly, the hotword configuration engine112can match a portion of the transcription of the waveform130that corresponds to the word “computer” with a portion of the transcription126stored by the vocabulary database110that corresponds to the word “computer.” Based on this matching, the hotword configuration engine112can identify the corresponding pre-computed hotword model128that is associated with the portion of the transcription126for the word “computer.”

In some examples, matching the words, or sub-words, of the candidate hotword129with words, or sub-words, that are stored by the vocabulary database110can include determining a full match between the words, or sub-words, of the candidate hotword129with words, or sub-words that are stored by the vocabulary database110. In some examples, matching the words, or sub-words, of the candidate hotword129with words, or sub-words, that are stored by the vocabulary database110can include determining a partial match between the words, or sub-words, of the candidate hotword129with words, or sub-words that are stored by the vocabulary database110.

In some examples, the hotword configuration engine112obtains the pre-computed hotword models128for the sub-words that correspond to the candidate hotword129. As mentioned above, for the candidate hotword129of “start computer,” the hotword configuration engine112identifies the sub-words of “st-ah-rt” for the word “start” of the candidate hotword129and further identifies the sub-words of “kom-pyu-ter” for the word “computer” of the candidate hotword129. To that end, the hotword configuration engine112accesses the vocabulary database110to identify the pre-computed hotword models128that are stored by the vocabulary database110and that correspond to the sub-words of the candidate hotword129. The hotword configuration engine112can identify the pre-computed hotword models128by matching the sub-words of the candidate hotword129with the sub-words that are stored by the vocabulary database110and are associated with the pre-computed hotword models128. For example, the hotword configuration engine112identifies the one or more pre-computed hotword models128stored by the vocabulary database110that correspond to the each of the sub-words of “st-ah-rt” for the word “start” of the candidate hotword129and each of the sub-words of “kom-pyu-ter” for the word “computer” of the candidate hotword129.

In some implementations, the hotword configuration engine112provides the identified, pre-computed hotword models to the mobile computing device106, at operation (G). Specifically, the hotword configuration engine112provides the pre-computed hotword models134, e.g., a subset of the pre-computed hotword models128, corresponding to the candidate hotword129to the mobile computing device106, e.g., over one or more networks. For example, the hotword configuration engine112can provide the pre-computed hotword models134that correspond to the candidate hotword129of “start computer” to the mobile computing device106.

In some examples, the hotword configuration engine112provides the identified, pre-computed hotword models134that correspond to each word of the two or more words of the candidate hotword129to the mobile computing device106. For example, the candidate hotword129includes two words, e.g., “start” and “computer,” and the hotword configuration engine112provides the pre-computed hotword models134that correspond to each word. That is, the hotword configuration engine112provides a first pre-computed hotword model134corresponding to the word “start” and a second pre-computed hotword model134corresponding to the word “computer” to the mobile computing device106.

In some examples, the identified, pre-computed hotword models134are provided to the mobile computing device106based on a type of the mobile computing device106. For example, a lower-end, or lower processing power, mobile computing device is more suitable to receive an appropriate version (e.g., smaller neural network) of the pre-computed hotword models134such that the mobile computing device is able to appropriately process the pre-computed hotword models134.

In some examples, the mobile computing device106can receive two or more pre-computed hotword models134in response to a command (or query) from the user118. That is, the user118can provide a command such as “navigate to coffee house” to the mobile computing device106. In response, the mobile computing device106can receive pre-computed hotword models134that correspond to two differing locations of coffee houses that are proximate to the user's118current location. For example, the mobile computing device106can receive a pre-computed hotword model135for “Palo Alto” and a pre-computed hotword model134for “Mountain View.” The mobile computing device106can provide both location options to the user118(e.g., via sound or the graphical user interface of the mobile computing device106). The user118can provide an utterance of one of the locations that the mobile computing device106can detect via the received pre-computed hotword models134, as described above.

In some examples, by generating the pre-computed hotword models128and providing the same to the vocabulary database110, the pre-computed hotword models are instantaneously available (or nearly instantaneously available) for identifying hotwords from utterances, e.g., by the mobile computing device106. For example, the mobile computing device106is able to instantaneously obtain the hotword models that correspond to the words “start” and “computer” such that the mobile computing device106is able to appropriately process the utterance170close to detection of utterance170.

In some examples, by generating the pre-computed hotword models128trained on utterances of other users (e.g., users114and116) that are not available to the mobile computing device106, the pre-computed hotword models128employed by the mobile computing device106to process the utterance170can be more robust as compared to hotword models128trained on utterances only provided by the user118.

In some examples, the hotword configuration engine112provides, to the mobile computing device106, instructions136that define a processing routine of the pre-computed hotword models134. That is, the instructions136define how the mobile computing device106is to appropriately process the pre-computed hotword models134. In some examples, the pre-computed hotword models134detect hotwords (e.g., of utterances) based on an analysis of underlying acoustic features (e.g., mel-frequency cepstrum coefficients) of an input utterance (e.g., utterance170).

In some examples, the instructions136include instructions to sequentially process the hotword models134, and further include a processing order of the hotword models134. For example, the instructions136can include instructions to initially process the pre-computed hotword model134corresponding to the word “start” and subsequently process the pre-computed hotword model134corresponding to the word “computer.” In some examples, the instructions136include instructions to parallel process the hotword models134. For example, the instructions136can include instructions to process the pre-computed hotword model134corresponding to the word “start” and process the pre-computed hotword model134corresponding to the word “computer” in parallel, e.g., at substantially the same time. In some examples, the instructions136include instructions to process the hotword models134such that a second hotword model134corresponding to the word “computer” is processed only when a first hotword model134detects the hotword “start.” In other words, upon detection of the word “computer” by the first hotword model134, the mobile computing device106triggers processing of the second hotword model134corresponding to the word “computer.”

The mobile computing device106receives the pre-computed hotword models134, and in some examples, the instructions136, from the hotword configuration engine112, e.g., over one or more networks. The mobile computing device106stores the pre-computed models134in memory of the mobile computing device106. Thus, upon detection of an utterance by the user118at a later time (e.g., after receiving the pre-computed hotword models134), the mobile computing device106can appropriately process the utterance in view of the pre-computed hotword models134to determine whether the utterance corresponds to the candidate hotword129.

In some further implementations, the hotword configuration engine112dynamically creates one or more of the hotword models that correspond to candidate hotword129. That is, in response to receiving the candidate hotword129from the mobile computing device106, the hotword configuration engine112dynamically creates a hotword model that corresponds to one or more words of the candidate hotword129. In some examples, the hotword configuration engine112dynamically creates the hotword models for the candidate hotword129based on i) the waveform130and ii) an obtained transcription of the waveform130, e.g., from the speech recognition engine108. For example, for the word “start” of the waveform130, the hotword configuration engine112dynamically creates a hotword model for the word based on i) a portion of the waveform130corresponding to the user118speaking the word “start” and ii) a transcription associated with the waveform130corresponding to the user118speaking the word “start.”

In some examples, as mentioned above, the hotword configuration engine112matches at least a portion of the waveform130to at least a portion of the waveforms stored by the vocabulary database110. Upon the matching, the hotword configuration engine112can further identify a portion of the corresponding transcription associated with the matched waveform that is stored by the vocabulary database110. To that end, the hotword configuration engine112dynamically creates the hotword model corresponding to the candidate hotword129based on i) the matched waveform and ii) the corresponding transcription associated with the matched waveform. For example, the hotword configuration engine112can identify a portion of the waveform160stored by the vocabulary database110corresponding to the word “start” and further identify the corresponding transcription124of the portion of the waveform160that includes the word “start.” The hotword configuration engine112can dynamically create the hotword model for the word “start” of the candidate hotword129based on i) the portion of the waveform160stored by the vocabulary database110corresponding to the word “start” and ii) the corresponding transcription124that includes the word “start.”

In some examples, as mentioned above, the hotword configuration engine112matches at least a portion of a transcription of the waveform130to at least a portion of transcriptions stored by the vocabulary database110. Upon the matching, the hotword configuration engine112can further identify the corresponding waveform associated with the matched transcription that is stored by the vocabulary database110. To that end, the hotword configuration engine112dynamically creates the hotword model corresponding to the candidate hotword129based on i) the matched transcription and ii) the corresponding waveform associated with the matched transcription. For example, the hotword configuration engine112can identify a portion of the transcription124stored by the vocabulary database110corresponding to the word “start” and further identify the corresponding portion of the waveform160that includes the word “start.” The hotword configuration engine112can dynamically create the hotword model for the word “start” of the candidate hotword129based on i) the portion of the transcription124stored by the vocabulary database110corresponding to the word “start” and ii) the corresponding portion of the waveform160that includes the word “start.”

In some examples, the hotword configuration engine112provides the dynamically created hotword models to the mobile computing device106, e.g., over one or more networks. For example, the hotword configuration engine112can provide dynamically created hotword models134that correspond to the word “start” of the candidate hotword “start computer” to the mobile computing device106. In some examples, the hotword configuration engine112can provide i) the dynamically created hotword model that corresponds to the word “start” of the candidate hotword129“start computer” and provide ii) the pre-computed hotword model134that corresponds to the word “computer” of the candidate hotword129“start computer” to the mobile computing device106.

In some examples, the hotword configuration engine112dynamically creates the hotword models that correspond to the candidate hotword129after receiving the candidate hotword129from the mobile computing device106. For example, the hotword configuration engine112dynamically creates the hotword models that correspond to the candidate hotword129of “start computer” after receiving the candidate hotword129from the mobile computing device106.

FIG. 2illustrates example graphical user interface (GUI)202of a mobile computing device204for identifying a user provided hotword. The mobile computing device204can be similar to the mobile computing device106ofFIG. 1. To that end, the mobile computing device204provides a first GUI202ato a user206associated with the mobile computing device204that provides for display initiation of the process to identify the user provided hotword to be associated with an action (e.g., a process performed at least partially by the mobile computing device204). In some examples, the first GUI202aincludes text208indicating to the user206to provide a hotword. For example, the text208includes “What would you like your Hotword to be to initiate a web search?” The user206provides an utterance210that the mobile computing device204detects. For example, the user206says the utterance210of “go gadget go” that the user206desires to be a hotword to initiate a web search.

In response to detecting the utterance210, the mobile computing device204provides a second GUI202bto the user206that provides for display a proposed transcription of the detected utterance210. In some examples, the second GUI202bincludes text212indicating to the user206to confirm, or reject, a transcription of the utterance210. For example, the text212includes “We think you said ‘Go gadget go.’ If yes, please hit confirm button. If no, please hit reject button, and speak Hotword again.” To that end, the second GUI202bfurther includes selectable buttons214and216that the user206is able to select to indicate whether to confirm that the transcription is correct, or reject the transcription. For example, upon selection of the selectable button214by the user206, the mobile computing device204receives a confirmation that the transcription of “Go gadget go” corresponds to the utterance210. Further, for example, upon selection of the selectable button216by the user206, the mobile computing device204receives a rejection of the transcription corresponding to the utterance210, e.g., an incorrect or inaccurate transcription. In some examples, the proposed transcription of the detected utterance210is not provided to the user206via the second GUI202b.

In response to receiving a confirmation that the transcription is correct, the mobile computing device204provides a third GUI202cto the user206that provides for display a confirmation of the transcription of the detected utterance210. In some examples, the third GUI202cincludes text218indicating to the user206that the user206has confirmed that the transcription of the utterance210is correct. For example, the text218includes “We have confirmed that your Hotword is ‘Go gadget go.’” Thus, the words “Go gadget go” have been established as associated with a hotword, and further associated with an action of initiating a web search.

After establishing the hotword by the user206, the user206can provide a hotword220, e.g., via an utterance or text-input, to the mobile computing device206, e.g., after identifying the user provided hotword. For example, the hotword220can include the words “go gadget go.” Thus, in response to receiving the hotword220of “go gadget go,” the mobile computing device206causes one or more actions to be performed, including initiating a web search, and provides a fourth GUI202dto the user206that provides for display a description of the action to be taken associated with receiving the hotword220. In some examples, the fourth GUI202dincludes text222of “Starting Search . . . . ”

FIG. 3depicts a flowchart of an example process300for providing hotword models. The example process300can be executed using one or more computing devices. For example, the mobile computing device102,104,106, the speech recognition engine108, and the hotword configuration engine112can be used to execute the example process500.

For each of multiple words or sub-words, audio data corresponding to multiple users speaking the word or sub-word is obtained (302). For example, the speech recognition engine208obtains the waveforms120and122from the mobile computing devices102and104, respectively, that correspond to the user114speaking the utterance150of “start my car” and the user116speaking the utterance152of “where do I buy a computer.” For each of the multiple words or sub-words, a pre-computed hotword model for the word or sub-word is trained based on the audio for the word or sub-word (304). For example, the hotword configuration engine112trains a pre-computed hotword model for each word or sub-word based on the waveforms120and122. A candidate hotword is received from a mobile computing device (306). For example, the hotword configuration engine112receives the candidate hotword129of “start computer” from the mobile computing device106. One or more pre-computed hotword models are identified that correspond to the candidate hotword (308). For example, the hotword configuration engine112identifies the pre-computed hotword models128stored by the vocabulary database110that correspond to the candidate hotword129of “start computer.” The identified, pre-computed hotword models are provided to the mobile computing device (310). For example, the hotword configuration engine112provides the pre-computed hotword models134to the mobile computing device106.

The memory404stores information within the computing device400. In one implementation, the memory404is a volatile memory unit or units. In another implementation, the memory404is a non-volatile memory unit or units. The memory404may also be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device406is capable of providing mass storage for the computing device400. In one implementation, the storage device406may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product may be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory404, the storage device406, or a memory on processor402.

The high speed controller408manages bandwidth-intensive operations for the computing device400, while the low speed controller412manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. In one implementation, the high-speed controller408is coupled to memory404, display416(e.g., through a graphics processor or accelerator), and to high-speed expansion ports410, which may accept various expansion cards (not shown). In the implementation, low-speed controller412is coupled to storage device406and low-speed expansion port414. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device400may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server420, or multiple times in a group of such servers. It may also be implemented as part of a rack server system424. In addition, it may be implemented in a personal computer such as a laptop computer422. Alternatively, components from computing device400may be combined with other components in a mobile device (not shown), such as device450. Each of such devices may contain one or more of computing device400,450, and an entire system may be made up of multiple computing devices400,450communicating with each other.

Computing device450includes a processor452, memory464, an input/output device such as a display454, a communication interface466, and a transceiver468, among other components. The device450may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components450,452,464,454,466, and468, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

The processor452may execute instructions within the computing device640, including instructions stored in the memory464. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor may provide, for example, for coordination of the other components of the device450, such as control of user interfaces, applications run by device450, and wireless communication by device450.

Processor452may communicate with a user through control interface648and display interface456coupled to a display454. The display454may be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface456may comprise appropriate circuitry for driving the display454to present graphical and other information to a user. The control interface458may receive commands from a user and convert them for submission to the processor452. In addition, an external interface462may be provide in communication with processor452, so as to enable near area communication of device450with other devices. External interface462may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

The memory464stores information within the computing device450. The memory464may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory454may also be provided and connected to device450through expansion interface452, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory454may provide extra storage space for device450, or may also store applications or other information for device450. Specifically, expansion memory454may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory454may be provide as a security module for device450, and may be programmed with instructions that permit secure use of device450. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

Device450may also communicate audibly using audio codec460, which may receive spoken information from a user and convert it to usable digital information. Audio codec460may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device450. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device450.

Thus, particular implementations of the present disclosure have been described. Other implementations are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. For example, various forms of the flows shown above may be used, with steps re-ordered, added, or removed. Accordingly, other implementations are within the scope of the following claims.