Initiating actions based on partial hotwords

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, receiving audio data; determining that an initial portion of the audio data corresponds to an initial portion of a hotword; in response to determining that the initial portion of the audio data corresponds to the initial portion of the hotword, selecting, from among a set of one or more actions that are performed when the entire hotword is detected, a subset of the one or more actions; and causing one or more actions of the subset to be performed.

FIELD

The present disclosure discusses initiating actions based on partial hotwords, and, more specifically, providing an initial action in response to detecting the partial hotword.

BACKGROUND

Vocal utterances may include a hotword, i.e., a predetermined reserved word that causes a system to perform a corresponding action or actions. A speech recognition service on an electronic device generally receives the vocal utterances that include spoken words from a user, and transcribes the spoken words into text. To accomplish this, the speech recognition service may attempt to match the sounds of the spoken input with phonetic representations of textual words.

SUMMARY

According to some innovative aspects of the subject matter described in this specification, the detection of an initial portion of a hotword causes an action to be performed. A user may say a hotword such as “OK Google” to a mobile computing device that causes actions to be performed by the mobile computing device. For example, the hotword “OK Google” causes the mobile computing device to wake from a sleep or stand-by state.

Upon detection of an initial portion of the hotword, e.g., the “OK” portion of the hotword “OK Google,” the mobile computing device can cause certain actions to be performed that are associated with the full hotword “OK Google.” For instance, upon detection of the partial hotword “OK,” the mobile computing device can open a network connection. By causing a subset of the actions that are associated with the hotword to be performed after detecting an initial portion of the hotword can reduce latency associated with the hotword, and thus, the processing time associated with the hotword is reduced, while being transparent to the user. That is, the user perceives a speed improvement of the mobile computing device.

Innovative aspects of the subject matter described in this specification may be embodied in methods that include the actions of receiving audio data; determining that an initial portion of the audio data corresponds to an initial portion of a hotword; in response to determining that the initial portion of the audio data corresponds to the initial portion of the hotword, selecting, from among a set of one or more actions that are performed when the entire hotword is detected, a subset of the one or more actions; and causing one or more actions of the subset to be performed

These and other embodiments may each optionally include one or more of the following features. For instance, identifying one or more acoustic features of the initial portion of the audio data; comparing the one or more acoustic features of the initial portion of the audio data to one or more acoustic features associated with one or more hotwords stored in a database; and based on the comparing, determining that the initial portion of the audio data corresponds to the initial portion of the hotword of the one or more hotwords stored in the database. Obtaining a transcription of the initial portion of the audio data, the audio data including a speech utterance; comparing the transcription of the initial portion of the audio to one or more transcriptions associated with one or more hotwords stored in a database; and based on the comparing, determining that the initial portion of the audio corresponding to the initial portion of the hotword of the one or more hotwords stored in the database. The initial portion of the hotword includes less than the entire hotword. The one or more actions of the subset include actions that are not visible to a user of a computing device performing the actions. The one or more actions of the subset of includes actions that are performed in a background computing environment of a computing device performing the actions. Transmitting an interrupt signal to an operating system executing on a computing device to perform the one or more actions. The interrupt signal causes the operating system to initialize a camera functionality of the computing device. The interrupt signal causes the operating system to activate the computing device from a sleep state. The interrupt signal causes the operating system to initialize a network connection on the computer device. The interrupt signal causes the operating system to preload an application on the computing device associated with the hotword.

The features further include, for example, receiving a subsequent portion of the audio data; determining that the initial portion of the audio data and the subsequent portion of the audio data, collectively correspond to the entire hotword; in response to determining that the initial portion of the audio data and the subsequent portion of the audio data, collectively, correspond to the entire hotword, selecting, from among the set of one or more actions that are performed when the entire hotword is detected, an additional subset of the one or more actions; and causing the one or more actions of the additional subset to be performed. The set of the one or more actions includes the subset of the one or more actions and the additional subset of the one more actions. Receiving a subsequent portion of the audio data; determining that the initial portion of the audio data and the subsequent portion of the audio data, collectively, do not correspond to the entire hotword; and in response to determining that the initial portion of the audio data and the subsequent portion of the audio data, collectively, do not correspond to the entire hotword, causing the one or more actions of the subset to be reversed. Selecting an additional subset of one or more actions that when performed, reverse the one or more actions of the subset of the one or more actions; and causing the one or more actions of the addition subset to be performed. Prior to receiving the audio data, receiving (i) data indicating the initial portion of the audio data corresponds to the initial portion of the hotword and (ii) data indicating the subset of the actions associated with the hotword. The hotword is a particular term that triggers semantic interpretation of an additional term of one or more terms that follow the particular term.

DETAILED DESCRIPTION

FIG. 1depicts a system100for detecting partial hotwords. In some examples, the system100includes an audio subsystem102, a partial hotword action initiator104, a hotword detector106, a hotword-to-action mapper108, a hotword-to-action database110, and an action engine112. In some examples, the system100is a mobile computing device, e.g., a mobile computing device100. In some examples, the mobile computing device includes a smartphone, or tablet-computing device. The mobile computing device100can include any audio detection means, e.g., a microphone, for detecting utterances from a user116.

In some examples, the user116says one or more words that the mobile computing device100detects. In some examples, the utterance includes one or more hotwords, or partial hotwords, that cause an action to be performed by the mobile computing device100. As depicted in the illustrated example, the user116says “OK Google.” The mobile computing device100detects the utterance “OK Google” with the audio subsystem102appropriately receiving audio data of the utterance “OK Google.”

In the illustrated example, the audio subsystem102receives the initial portion of the audio data associated with the utterance “OK” at a time t1. The initial portion of the audio data can include a first few audio frames of the utterance, or a first few hundred milliseconds of audio data associated with the utterance. In some examples, the audio subsystem102processes the utterance to generate waveform data that includes a portion of the utterance, or substantially a whole portion of the utterance.

In some examples, the audio subsystem102appropriately processes the received audio data. For example, the audio subsystem102identifies one or more acoustic features of the initial portion of the audio data. In the illustrated example, the audio subsystem102identifies acoustic features associated with the initial portion of the audio data associated with the utterance “OK.” In some examples, the audio subsystem102obtains a transcription of the initial portion of the audio data associated with the utterance “OK.” In the illustrated example, the audio subsystem102obtains a transcription of the initial portion of the audio data associated with the utterance “OK.”

In some examples, the audio subsystem102provides the processed audio data to the partial hotword action initiator104. For example, the audio subsystem102provides the processed audio data associated with the utterance “OK” to the partial hotword action initiator104. In some examples, the processed audio data can include acoustic features associated with the utterance “OK,” a transcription associated with the utterance “OK,” or both.

The partial hotword action initiator104receives the processed audio data from the audio subsystem102. For example, the processed audio data can include audio data associated with the utterance “OK,” acoustic features associated with the utterance “OK,” a transcription associated with the utterance “OK,” or any combination thereof. The partial hotword action initiator104provides the processed audio data to the hotword detector106.

The hotword detector106receives the processed audio data from the partial hotword action initiator104. In the illustrated example, the hotword detector106initially receives the processed audio data associated with the utterance “OK,” e.g., an initial portion of the audio data. The hotword detector106determines that the received audio data corresponds to an initial portion of a hotword. Specifically, the hotword detector106includes a partial hotword detector118. The partial hotword detector118determines that the initial portion of the audio data corresponds to an initial portion of a hotword. In the illustrated example, the partial hotword detector118determines that the initial portion of the audio data associated with the utterance “OK” corresponds to an initial portion of a hotword “OK Google.” In some examples, the initial portion of a hotword includes less than the entire hotword. For example, the initial portion “OK” is less than the entire hotword “OK Google.”

In some examples, the received audio data includes associated acoustic features, e.g., as identified by the audio subsystem102. The partial hotword detector118compares the acoustic features of the initial portion of the audio data associated with the utterance “OK” to acoustic features associated with one or more hotwords stored in a database. Specifically, each hotword is associated with one or more acoustic features, e.g., phonetics of the hotword. A database, or table, stores a mapping between the hotwords and an associated acoustic feature(s). In some examples, a hotword, is associated with two or more acoustic features. In some examples, an acoustic feature is associated with two or more hotwords.

Based on the comparing of acoustic features, the initial portion of the audio data associated with the utterance “OK” is determined to correspond to an initial portion of a hotword stored in the database. For example, it is determined that the acoustic features of the initial portion of the audio data associated with the utterance “OK” corresponds to an initial portion of a hotword “OK Google” based on the comparison of the acoustic features of the utterance “OK” and the acoustic features of the hotword “OK Google.”

In some examples, the processed audio data includes a transcription, e.g., as obtained by the audio subsystem102. The partial hotword detector118compares the transcription of the initial portion of the audio data associated with the utterance “OK,” to transcriptions associated with hotwords stored in a database. Specifically, a hotword is associated with a transcription. A database, or table, stores a mapping between the hotword and a transcription.

Based on the comparing of transcriptions, the transcription of the initial portion of the audio data associated with the partial utterance “OK” is determined to correspond to an initial portion of a hotword stored in a database. For example, it is determined that the transcription of the initial portion of the audio data associated with the utterance “OK” corresponds to an initial portion of a hotword “OK Google” based on the comparison of the transcription of the utterance “OK” and the transcription of the hotword “OK Google.”

The hotword detector106provides a result of the determination that the initial portion of the audio data corresponds to the initial portion of a hotword to the partial hotword action initiator104. Specifically, in the illustrated example, the hotword detector106provides the partial hotword “OK,” or a signal associated with the partial hotword “OK,” to the partial hotword action initiator104.

The partial hotword action initiator104receives the result of the determination from the hotword detector106. Specifically, in the illustrated example, the partial hotword action initiator104receives the partial hotword “OK,” or a signal associated with the partial hotword “OK,” from the hotword detector106. The signal can include an indication that the initial portion of the audio data associated with the utterance “OK” corresponds to an initial portion of a hotword. In response, the partial hotword action initiator104provides the partial hotword “OK,” or the signal associated with the partial hotword “OK,” to the hotword-to-action mapper108.

The hotword-to-action mapper108receives the result of the determination that the initial portion of the audio data corresponds to the initial portion of the hotword from the partial hotword action initiator104. Specifically, in the illustrated example, the hotword-to-action mapper108receives the partial hotword “OK,” or the signal associated with the partial hotword “OK,” from the partial hotword action initiator104. The hotword-to-action mapper108selects, from among a set of actions that are performed when the entire hotword is detected, a subset of the actions. Specifically, each hotword is associated with one or more actions, e.g., actions that are performed when the entire hotword is detected. A hotword-to-action database110stores mapping between hotwords and associated actions. In some examples, a hotword is mapped to two or more actions. In some examples, an action is mapped to two or more hotwords.

In the illustrated example, upon detection of the partial hotword “OK,” the hotword-to-action mapper108identifies a particular set of actions that are associated with the entire hotword “OK Google.” That is, the hotword-to-action mapper108is in communication with the hotword-to-action database110to identify a mapping between the entire hotword “OK Google” and the particular set of actions from multiple sets of actions that are associated with the entire hotword “OK Google.” The hotword-to-action mapper108selects a subset of actions from among the particular set of actions associated with the entire hotword “OK Google.” Specifically, the hotword-to-action mapper108selects the subset of actions that are further associated with the partial hotword “OK” of the entire hotword “OK Google.” In some examples, the hotword-to-action mapper108identifies the subset of actions that are associated with the partial hotword “OK.”

In some examples, the hotword-to-action mapper108can use multiple criteria for selecting the subset of actions. For example, the hotword-to-action mapper108can select actions from the particular set of actions that are not visible to the user. That is, select actions that are not visible to the user that include actions caused to be performed by the mobile computing device116that are not readably noticeable or detectable by the user116, e.g., visually or audible detectable by the user116. For example, the actions can include actions that are not shown on a user interface of the mobile computing device100, or actions that do not make noise by the mobile computing device100. In some examples, the selected actions can include actions that are performed in a background computing environment of the mobile computing device100. In the illustrated example, the subset of actions include an action of establishing a network connection, e.g., with a third-party computing server or a back-end computing server. In some examples, the selected actions can include actions that have no lasting or permanent effects, i.e., are reversible, on the mobile computing device

The hotword-to-action mapper108provides data identifying the subset of actions to the partial hotword action interior104. Specifically, the hotword-to-action mapper108provides data identifying the subset of actions associated with the hotword “OK Google” to the partial hotword action initiator104.

The partial hotword action initiator104receives the data identifying the subset of actions from the hotword-to-action mapper108, and in response, provides the data identifying the subset of actions to the action engine112. Specifically, by providing the data identifying the subset of actions to the action engine112, the partial hotword action initiator104causes the subset of actions to be performed.

The action engine112receives the data identifying the subset of actions from the partial hotword action initiator104. In the illustrated example, the action engine112receives the data identifying the subset of actions associated with the initial portion of the hotword “OK Google,” as selected by the hotword-to-action mapper108. Specifically, the subset of actions that are associated with the partial hotword “OK” of the entire hotword “OK Google” are performed at the time t1. The action engine112causes an action of establishing a network connection to be performed, e.g., by the operating system executing on the mobile computing device100.

In some examples, causing the subset of actions to be performed includes transmitting an interrupt signal to the operating system executing on the mobile computing device100. In some examples, the interrupt signal causes the operating system to initialize a camera functionality of the mobile computing device100. In some examples, the interrupt signal causes the operating stem to activate the mobile computing device100from a sleep state. That is, that state of the mobile computing device100is changed from a sleep state, or an “off” state, to a wake state, or an “on” state. In some examples, the interrupt signal causes the operating system to preload an application stored on the mobile computing device100that is associated with the hotword. That is, the application can be preloaded into a memory of the mobile computing device100.

In some further implementations, after receiving the initial portion of the audio data, the audio subsystem102receives a subsequent portion of the audio data. For example, the audio subsystem102receives the subsequent portion of the detected utterance, e.g., subsequent to the initial portion of the audio data. In the illustrated example, the audio subsystem102receives the subsequent portion of the audio data associated with the utterance “Google” at a time t2. The audio subsystem102appropriately processes the received audio data. In some examples, the audio subsystem102receives a whole portion of the audio, including the initial portion and the subsequent portion.

In some examples, the hotword is a particular term that triggers semantic interpretation of an additional term of one or more terms that follow the particular term. For example, by the initial portion of the audio data associated with the utterance “OK” corresponding to an initial portion of a hotword, the audio subsystem102provides a semantic interpretation of the subsequent portion of the audio data associated with the utterance “Google.”

In some examples, the audio subsystem102provides the processed audio data associated with the entire utterance to the partial hotword action initiator104. For example, the audio subsystem102provides the processed audio data associated with the utterance “OK Google” to the partial hotword action initiator104. In some examples, the processed audio data can include acoustic features associated with the utterance “OK Google,” a transcription associated with the utterance “OK Google,” or both.

The partial hotword action initiator104receives the processed audio data from the audio subsystem102. For example, the processed audio data can include the audio data associated with the utterance “OK Google,” acoustic features associated with the audio data associated with the utterance “OK Google,” a transcription associated with the audio data associated with the utterance “OK Google,” or any combination thereof. The partial hotword action initiator104provides the processed audio data to the hotword detector106.

The hotword detector106receives the processed audio data from the partial hotword action initiator104. In the illustrated example, the hotword detector106receives the processed audio data associated with the initial portion of the audio associated with the utterance “OK” and the subsequent portion of the audio data associated with the utterance “Google.” In other words, the hotword detector106receives the audio data associated with the entire utterance “OK Google.” In the illustrated example, the hotword detector106determines that the initial and the subsequent portion of the audio data, collectively, correspond to the entire hotword “OK Google.”

The hotword detector106provides a result of the determination that the initial portion and the subsequent portion of the audio data associated with the utterance “OK Google” collectively correspond to the entire hotword to the partial hotword action initiator104. Specifically, in the illustrated example, the hotword detector106provides the entire hotword “OK Google,” or a signal associated with the entire hotword “OK Google,” to the partial hotword action initiator104. The signal can include an indication that the initial portion and the subsequent portion of the audio data, collectively, correspond to an entire hotword.

The partial hotword action initiator104receives the result of the determination from the hotword detector106. Specifically, in the illustrated example, the partial hotword action initiator104receives the entire hotword “OK Google,” or a signal associated with the entire hotword “OK Google,” from the hotword detector106. In response, the partial hotword action initiator104provides the entire hotword “OK Google,” or a signal associated with the entire hotword “OK Google,” to the hotword-to-action mapper108.

The hotword-to-action mapper108receives the result, e.g., a signal, of the determination that the initial portion and the subsequent portion of the audio data associated with the utterance “OK Google” collectively correspond to the entire hotword from the partial hotword action initiator104. Specifically, in the illustrated example, the hotword-to-action mapper108receives the entire hotword “OK Google,” or a signal associated with the entire hotword “OK Google,” from the partial hotword action initiator104. The hotword-to-action mapper108selects, from among a set of actions that are performed when the entire hotword is detected, an additional subset of the actions.

In the illustrated example, upon receipt of the entire hotword “OK Google,” or the signal associated with the entire hotword “OK Google,” the hotword-to-action mapper108identifies a particular set of actions that are associated with the entire hotword “OK Google.” That is, the hotword-to-action mapper108is in communication with the hotword-to-action database110to identify a mapping between the entire hotword “OK Google” and the particular set of actions from multiple sets of actions. The hotword-to-action mapper108selects an additional subset of actions from among the particular set of actions associated with the entire hotword “OK Google.” In some examples, the set of actions associated with the entire hotword “OK Google” includes the subset of actions and the additional subset of actions. In some examples, the additional subset of actions includes the subset of actions. In some examples, the additional subset of actions includes one or more actions of the subset of actions. For example, upon detection of the entire hotword “OK Google,” each action of the subset of actions may have not been fully performed by the action engine112. Thus, any unperformed actions associated with the subset of actions can further be included by the additional subset of actions.

The hotword-to-action mapper108provides data identifying the additional subset of actions to the partial hotword action initiator104. That is, the hotword-to-action mapper108provides data identifying the additional subset of actions associated with the detection of the entire hotword “OK Google” to the partial hotword action initiator104.

The partial hotword action initiator104receives the data identifying the additional subset of actions from the hotword-to-action mapper108, and in response, provides the data identifying the additional subset of actions to the action engine112. Specifically, by providing the data identifying the additional subset of actions to the action engine112, the partial hotword action initiator104causes the additional subset of actions to be performed.

The action engine112receives the data identifying the additional subset of actions from the partial hotword action initiator104. In the illustrated example, the action engine112receives the data identifying the additional subset of actions associated with the detection of the entire hotword, as selected by the hotword-to-action mapper108. Specifically, the additional subset of actions that are associated with the entire hotword “OK Google” are performed at the time t2. In response to receiving the additional subset of actions, the action engine112causes an action of turning a display, or screen, of the mobile computing device100to be in an “on” state. In some examples, in response to receiving the additional subset of actions, the action engine112causes an action of displaying a user interface. In some examples, the set of actions associated with the detection of the partial hotword “OK” and the additional set of actions associated with the detection of the entire hotword “OK Google” are caused to be performed by the action engine112in parallel, or at substantially the same time.

In some further implementations, before receiving the audio data, additional data is received (i) indicating that the initial portion of the audio data corresponds to the initial portion of the hotword and additional data (ii) indicating the subset of the actions associated with the hotword. For example, prior to receiving the audio data “OK Google” from the user116, the mobile computing device100is “trained,” e.g., receives the additional data, such that the partial hotword detector118is able to determine whether an initial portion of the audio data, e.g., the “OK” utterance, corresponds to an initial portion of a hotword, e.g., “OK Google.” Additionally, for example, the mobile computing device100receives the additional data such that the hotword-to-action-mapper108and the hotword-to-action database110are able to identify a subset of actions associated with the hotword “OK Google.” That is, the hotword-to-action-mapper108receives data such that the hotword-to-action database110maps associations between hotwords, e.g., “OK Google,” and actions, e.g., establishing network connection actions and/or waking the mobile computing device100.

FIG. 2illustrates a swim-lane diagram of interactions between an audio subsystem295, a partial hotword action initiator296, a hotword detector297, a hotword-to-action mapper298, and an action engine299. In some examples, the audio subsystem295can include the audio subsystem102ofFIG. 1; the partial hotword action initiator296can include the partial hotword initiator104ofFIG. 1; the hotword detector297can include the hotword detector106, the partial hotword detector118, or both, ofFIG. 1; the hotword-to-action mapper298can include the hotword-to-action mapper108ofFIG. 1; and the action engine299can include the action engine112ofFIG. 1. Specifically, in the illustrated example, a user, e.g., the user116, provides the utterance “OK computer I mean OK Google call Mom” to a mobile computing device, e.g., the mobile computing device100.

As illustrated inFIG. 2A, at step201, the audio subsystem295receives and appropriately processes the partial utterance “OK.” At step202, the partial hotword action initiator296receives the processed partial utterance “OK” and forwards the same to the hotword detector297. At step203, the hotword detector297determines that the partial utterance “OK” corresponds to, at least, an initial portion of the hotword “OK Google.” In some examples, the partial utterance “OK” can correspond to an initial portion of two or more hotwords. At step204, the hotword detector297provides the determination, or result, to the partial hotword action initiator296. For example, the partial hotword detector296determines that the partial utterance “OK” correspond to an initial portion of a hotword “OK Google.” At step205, the partial hotword action initiator296receives the determination, or result, and at step206, in response, triggers the hotword-to-action mapper298. For example, in response to the partial utterance “OK” corresponding to an initial portion of a hotword, the partial hotword action initiator296triggers the hotword-to-action-mapper298by providing the partial hotword “OK,” or a signal associated with the partial hotword “OK,” to the hotword-to-action initiator298.

At step207, the hotword-to-action mapper298maps the partial hotword “OK” of the entire hotword “OK Google” to an action, e.g., utilizing a database. For example, the hotword-to-action mapper298maps the partial hotword “OK” of the hotword “OK Google” to a network connection action, e.g., initializing or opening a network connection. At step208, the hotword-to-action mapper298provides information associated with the network connection action to the partial hotword action initiator296. At step209, the partial hotword action initiator296receives the information associated with the network connection action. At step210, the partial hotword action initiator296causes the network connection action to be performed. That is, the partial hotword action initiator296provides the information associated with the network connection action to the action engine299. At step211, the action engine299appropriately processes the network connection action information, including transmitting an interrupt signal to an operating system executing on a mobile computing device to perform the network connection action.

As illustrated inFIG. 2B, at step215, the audio subsystem295receives and appropriately processes the partial utterance “computer.” At step216, the partial hotword action initiator296receives the processed partial utterance “computer” and forwards the same to the partial hotword detector118. At step217, the hotword detector297determines that the entire utterance “OK computer” does not correspond to, at least, the entire hotword “OK Google,” or any other hotword. Specifically, in the illustrated example, the hotword detector297determines that the initial portion and the subsequent portion of the utterance “OK computer” does not correspond to an entire hotword.

In some examples, the processed audio data includes associated acoustic features, e.g., as identified by the audio subsystem102. The hotword detector297can compare the acoustic features of the audio data associated with the entire utterance “OK computer” to acoustic features associated with one or more hotwords stored in a database. Based on the comparing of acoustic features, the audio data associated with the entire utterance “OK computer” is determined not to correspond to a hotword stored in the database.

In some examples, the processed audio data includes a transcription, e.g., as obtained by the audio subsystem295. The hotword detector297can compare the transcription of the audio data associated with the entire utterance “OK computer” to transcriptions associated with hotwords stored in a database. Based on the comparing of transcriptions, the audio data associated with the entire utterance “OK computer” is determined not to correspond to a hotword stored in a database.

At step218, the hotword detector297provides the determination, or result, to the partial hotword action initiator296. For example, the hotword detector297determines that the entire utterance “OK computer” does not correspond to a hotword. At step219, the partial hotword action initiator296receives the determination, or result, and at step220, in response, triggers the hotword-to-action mapper298. For example, in response to the utterance “OK computer” not corresponding to a hotword, the partial hotword action initiator297triggers the hotword-to-action-mapper298. At step221, the hotword-to-action mapper298selects an additional subset of actions that when performed, cause the open network connection action to be reversed, or cancelled. For example, each particular action corresponding to a hotword can further be associated with one or more additional actions that when selected, reverse the particular action that was previously performed. For example, the open network connection action, as performed at step211, can be associated with a close network connection action such that when the hotword “OK Google” is not detected, e.g., after detecting the partial hotword “OK,” the open network connection action is reversed. In some examples, each particular action can be associated with two or more additional actions that reverse the particular action, e.g., negate the particular action.

At step222, the hotword-to-action mapper298provides information associated with the close network connection action to the partial hotword action initiator296. At step223, the partial hotword action initiator296receives the information associated with the close network connection action. At step224, the partial hotword action initiator296causes the close network connection action to be performed. That is, the partial hotword action initiator296provides the information associated with the close network connection action to the action engine299. At step225, the action engine299appropriately processes the close network connection action information, including transmitting an interrupt signal to an operating system executing on a mobile computing device to perform the close network connection action.

As illustrated atFIG. 2C, at step230, the audio subsystem295receives and appropriately processes the utterance “I mean.” At step231, the partial hotword action initiator296receives the processed utterance “I mean” and forwards the same to the hotword detector297. At step232, the hotword detector297determines that the utterance “I mean” does not corresponds to, at least, an initial portion of a hotword or an entire hotword. At step233, the hotword detector297provides the determination, or result, to the partial hotword action initiator296. At step234, the partial hotword action initiator296receives the determination, or result, and, in response to the utterance “I mean” not corresponding to an initial portion of a hotword or an entire hotword, takes no further action.

At step235, the audio subsystem295receives and appropriately processes the partial utterance “OK.” At step236, the partial hotword action initiator295receives the processed partial utterance “OK” and forwards the same to the hotword detector297. At step237, the hotword detector297determines that the partial utterance “OK” corresponds to, at least, an initial portion of the hotword “OK Google.” At step238, the hotword detector297provides the determination, or result, to the partial hotword action initiator296. For example, the hotword detector297determines that the partial utterance “OK” correspond to an initial portion of a hotword “OK Google.” At step239, the partial hotword action initiator296receives the determination, or result, and at step240, in response, triggers the hotword-to-action mapper298. For example, in response to the partial utterance “OK” corresponding to an initial portion of a hotword, the partial hotword action initiator296triggers the hotword-to-action-mapper298by providing the partial hotword “OK,” or a signal associated with the partial hotword “OK,” to the hotword-to-action mapper298.

At step241, the hotword-to-action mapper298maps the partial hotword “OK” to an action, e.g., utilizing a database. For example, the hotword-to-action mapper298maps the partial hotword “OK” of the entire hotword “OK Google” to a network connection action, e.g., initializing or opening a network connection. At step242, the hotword-to-action mapper298provides information associated with the network connection action to the partial hotword action initiator296. At step243, the partial hotword action initiator296receives information associated with the network connection action. At step244, the partial hotword action initiator296causes the network connection action to be performed. That is, the partial hotword action initiator296provides the information associated with the network connection action to the action engine299. At step245, the action engine299appropriately processes the network connection action information, including transmitting an interrupt signal to an operating system executing on a mobile computing device to perform the network connection action.

As illustrated atFIG. 2D, at step250, the audio subsystem295receives and appropriately processes the partial utterance “Google.” At step251, the partial hotword action initiator296receives the processed partial utterance “Google” and forwards the same to the hotword detector297. At step252, the hotword detector297determines that the utterance “OK Google” corresponds to, at least, a hotword “OK Google.” At step253, the hotword detector297provides the determination, or result, to the partial hotword action initiator296. For example, the partial hotword detector297determines that the initial utterance “OK” and the subsequent utterance “Google,” collectively correspond to the hotword “OK Google.” At step254, the partial hotword action initiator296receives the determination, or result, and at step255, in response, triggers the hotword-to-action mapper298. For example, in response to the initial utterance “OK” and the subsequent utterance “Google,” collectively, corresponding to an entire hotword, the partial hotword action initiator296triggers the hotword-to-action-mapper298by providing the entire hotword “OK Google,” or a signal associated with the entire hotword “OK Google,” to the hotword-to-action mapper298.

At step256, the hotword-to-action mapper298maps the hotword to an action, e.g., utilizing a database. For example, the hotword-to-action mapper298maps the hotword “OK Google” to a wake mobile device action, e.g., waking the mobile computing device from a sleep state, or “off state,” to a wake state, or an “on” state. At step257, the hotword-to-action mapper298provides information associated with the wake mobile device action to the partial hotword action initiator296. At step258, the partial hotword action initiator296receives information associated with the wake mobile device action. At step259, the partial hotword action initiator296causes the wake mobile device action to be performed. That is, the partial hotword action initiator296provides the information associated with the wake mobile device action to the action engine299. At step260, the action engine299appropriately processes the wake mobile device action information, including transmitting an interrupt signal to an operating system executing on a mobile computing device to perform the wake mobile device action.

As illustrated atFIG. 2E, at step265, the audio subsystem295receives and appropriately processes the partial utterance “Call.” At step266, the partial hotword action initiator296receives the processed partial utterance “Call” and forwards the same to the hotword detector297. At step267, the hotword detector297determines that the partial utterance “Call” corresponds to, at least, an initial portion of a hotword. At step268, the hotword detector297provides the determination, or result, to the partial hotword action initiator296. For example, the hotword detector297determines that the partial utterance “Call” correspond to an initial portion of a hotword. At step269, the partial hotword action initiator296receives the determination, or result, and at step270, in response, triggers the hotword-to-action mapper298. For example, in response to the partial utterance “Call” corresponding to an initial portion of a hotword, the partial hotword action initiator296triggers the hotword-to-action-mapper298by providing the partial hotword “Call” to the hotword-to-action mapper108.

At step271, the hotword-to-action mapper298maps the partial hotword of the hotword to an action, e.g., utilizing a database. For example, the hotword-to-action mapper298maps the partial hotword “Call” to a launch telephone application action, e.g., preloading of an application on a mobile computing device. At step272, the hotword-to-action mapper298provides information associated with the launch telephone application action to the partial hotword action initiator296. At step273, the partial hotword action initiator296receives information associated with the launch telephone application action. At step274, the partial hotword action initiator296causes the launch telephone application action to be performed. That is, the partial hotword action initiator296provides the information associated with the launch telephone application action to the action engine299. At step275, the action engine299appropriately processes the launch telephone application action information, including transmitting an interrupt signal to an operating system executing on a mobile computing device to perform the launch telephone application action.

As illustrated atFIG. 2F, at step280, the audio subsystem295receives and appropriately processes the partial utterance “Mom.” At step281, the partial hotword action initiator296receives the processed partial utterance “Mom” and forwards the same to the hotword detector297. At step282, the hotword detector297determines that the initial utterance “Call” and the subsequent utterance “Mom,” collectively, correspond to, at least, the hotword “Call Mom.” At step283, the hotword detector297provides the determination, or result, to the partial hotword action initiator296. For example, the hotword detector297determines that the initial utterance “Call” and the subsequent utterance “Mom,” collectively, correspond to an entire hotword. At step284, the partial hotword action initiator296receives the determination, or result, and at step285, in response, triggers the hotword-to-action mapper298. For example, in response to the initial utterance “Call” and the subsequent utterance “Mom,” collectively, corresponding to an entire hotword, the partial hotword action initiator296triggers the hotword-to-action-mapper298by providing the entire hotword “Call Mom” to the hotword-to-action mapper298.

At step286, the hotword-to-action mapper298maps the entire hotword “Call Mom” to an action, e.g., utilizing a database. For example, the hotword-to-action mapper298maps the entire hotword “Call Mom” to a contact assigned with the utterance “Mom.” The user may have previously associated the utterance “Mom” with a contact stored by a mobile computing device. At step287, the hotword-to-action mapper298provides information associated with the place call to “Mom” action to the partial hotword action initiator296. At step288, the partial hotword action initiator296receives information associated with the place call to “Mom” action. For example, the information can include contact information associated with a contact “Mom.” At step289, the partial hotword action initiator296causes the call to be placed to “Mom” action to be performed. That is, the partial hotword action initiator296provides the information associated with the call to be placed to “Mom” action to the action engine299. At step290, the action engine299appropriately processes the call to be placed to “Mom” action information, including transmitting an interrupt signal to an operating system executing on a mobile computing device to perform the call to be placed to “Mom” action.

FIG. 3depicts a flowchart of an example process300for identifying partial hotwords. The example process300can be executed using one or more computing devices. For example, the audio subsystem102, the partial hotword action initiator104, the hotword detector106, the hotword to action mapper108, the hotword-to-action database110, the action engine112, and the partial hotword detector118can be used to execute the example process300.

Audio data is received (302). For example, the audio subsystem102receives the audio data of an utterance “OK Google” from the mobile computing device114. An initial portion of the audio data is determined to correspond to an initial portion of a hotword (304). For example, the hotword detector106, the partial hotword detector118, or both, determine that the initial portion “OK” of the audio data “OK Google” corresponds to an initial portion of a hotword. In response to determining that the initial portion of the audio data corresponds to the initial portion of the hotword, a subset of one or more actions are selected from among a set of one or more actions that are performed when the entire hotword is detected (306). For example, the hotword-to-action mapper108identifies a particular set of actions that are associated with the entire hotword “OK Google.” The hotword-to-action mapper108selects the subset of actions that are further associated with the partial hotword “OK” of the entire hotword “OK Google.” One or more actions of the subset are caused to be performed (308). For example, the action engine112receives the subset of actions associated with the entire hotword “OK Google” that are caused to be performed.

The memory404stores information within the computing device400. In one implementation, the memory404is a volatile memory unit or units. In another implementation, the memory404is a non-volatile memory unit or units. The memory404may also be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device406is capable of providing mass storage for the computing device400. In one implementation, the storage device406may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product may be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory404, the storage device406, or a memory on processor402.

The high speed controller408manages bandwidth-intensive operations for the computing device400, while the low speed controller412manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. In one implementation, the high-speed controller408is coupled to memory404, display416(e.g., through a graphics processor or accelerator), and to high-speed expansion ports410, which may accept various expansion cards (not shown). In the implementation, low-speed controller412is coupled to storage device406and low-speed expansion port414. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device400may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server420, or multiple times in a group of such servers. It may also be implemented as part of a rack server system424. In addition, it may be implemented in a personal computer such as a laptop computer422. Alternatively, components from computing device400may be combined with other components in a mobile device (not shown), such as device450. Each of such devices may contain one or more of computing device400,450, and an entire system may be made up of multiple computing devices400,450communicating with each other.

Computing device450includes a processor452, memory464, an input/output device such as a display454, a communication interface466, and a transceiver468, among other components. The device450may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components450,452,464,454,466, and468, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

The processor452may execute instructions within the computing device640, including instructions stored in the memory464. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor may provide, for example, for coordination of the other components of the device450, such as control of user interfaces, applications run by device450, and wireless communication by device450.

Processor452may communicate with a user through control interface648and display interface456coupled to a display454. The display454may be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface456may comprise appropriate circuitry for driving the display454to present graphical and other information to a user. The control interface458may receive commands from a user and convert them for submission to the processor452. In addition, an external interface462may be provide in communication with processor452, so as to enable near area communication of device450with other devices. External interface462may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

The memory464stores information within the computing device450. The memory464may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory454may also be provided and connected to device450through expansion interface452, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory454may provide extra storage space for device450, or may also store applications or other information for device450. Specifically, expansion memory454may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory454may be provide as a security module for device450, and may be programmed with instructions that permit secure use of device450. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

Device450may also communicate audibly using audio codec460, which may receive spoken information from a user and convert it to usable digital information. Audio codec460may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device450. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device450.

Thus, particular implementations of the present disclosure have been described. Other implementations are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. For example, various forms of the flows shown above may be used, with steps re-ordered, added, or removed. Accordingly, other implementations are within the scope of the following claims.