Patent Description:
The reality of a speech-enabled home or other environment - that is, one in which a user need only speak a query or command out loud and a computer-based system will field and answer the query and/or cause the command to be performed - is upon us. A speech-enabled environment (e.g., home, workplace, school, etc.) can be implemented using a network of connected microphone devices distributed throughout the various rooms or areas of the environment. Through such a network of microphones, a user has the power to orally query the system from essentially anywhere in the environment without the need to have a computer or other device in front of him/her or even nearby. For example, while cooking in the kitchen, a user might ask the system "how many milliliters in three cups?" and, in response, receive an answer from the system, e.g., in the form of synthesized voice output. Alternatively, a user might ask the system questions such as "when does my nearest gas station close," or, upon preparing to leave the house, "should I wear a coat today?" An example of speech-enabled environment can be found in <CIT>.

Further, a user may ask a query of the system, and/or issue a command, that relates to the user's personal information. For example, a user might ask the system "when is my meeting with John?" or command the system "remind me to call John when I get back home.

For a speech-enabled system, the users' manner of interacting with the system is designed to be primarily, if not exclusively, by means of voice input. Consequently, the system, which potentially picks up all utterances made in the surrounding environment including those not directed to the system, must have some way of discerning when any given utterance is directed at the system as opposed, e.g., to being directed at an individual present in the environment. One way to accomplish this is to use a hotword, which by agreement among the users in the environment, is reserved as a predetermined word that is spoken to invoke the attention of the system. In an example environment, the hotword used to invoke the system's attention are the words "OK computer. " Consequently, each time the words "OK computer" are spoken, it is picked up by a microphone, conveyed to the system, which may perform speech recognition techniques or use audio features and neural networks to determine whether the hotword was spoken and, if so, awaits an ensuing command or query. Accordingly, utterances directed at the system take the general form [HOTWORD] [QUERY], where "HOTWORD" in this example is "OK computer" and "QUERY" can be any question, command, declaration, or other request that can be speech recognized, parsed and acted on by the system, either alone or in conjunction with the server via the network.

In addition to detecting a hotword, a computing device may be configured to detect audio watermarks embedded in the audio of the hotword. The audio watermark may be a high frequency watermark that is inaudible to humans or a watermark that sounds similar to background noise or static. The computing device may be configured to respond differently to hotwords depending on the audio watermark. For example, a portion of media content may be created that includes an actor speaking, "Ok computer, give me directions to the train station. " To prevent any computing device within the vicinity of a television playing the media content from providing directions to the train station, the creators of the media content may include an audio watermark that overlaps the hotword. A computing device detecting the audio of the media content may extract the audio watermark and compare the audio watermark to an audio watermark database. The audio watermark database may include a rule for when a computing device should perform speech recognition on the audio following the hotword and this particular audio watermark and additional rules for other audio watermarks. As will be appreciated, at least some of the methods and systems described herein may enable a computing device to selectively determine when to respond to a spoken utterance that is output over a speaker in the vicinity of the computing device.

The rule for the audio watermark for the media content may include conditions that a computing device should satisfy before further processing the audio from the media content. An example rule may be that a computing device should respond if the computing device is not currently providing directions, is not moving greater than ten miles per hour, is a smart speaker, and is located at the user's home. Another example of a rule may be that a computing device should respond only if the computing device is within premises associated with the owner or creator of the media content and the watermark indicates that the utterance is high priority. If the computing device satisfies these rules, then the computing device may perform speech recognition on the portion following the hotword. If the transcription is, "give me directions to the train station," then the computing device may display or audibly provide directions to the train station. If the computing device does not satisfy the rules, then the computing devices does not perform speech recognition on the portion of the audio following the hotword and performs no further actions in response to the audio.

In some implementations, the audio watermark may encode data that may eliminate the need for the computing device to compare the audio watermark to an audio watermark database. The encoded data may include a rule for when the computing device should perform speech recognition, an identifier, an action, or any other similar data. In some implementations, the computing device may use the encoded data in combination with the audio watermark database to determine whether to perform speech recognition on the audio following the hotword.

According to an innovative aspect of the subject matter described in this application as set forth in independent claim <NUM>, a computer implemented method which when executed on data processing hardware of a user device causes the data processing hardware to perform operations comprises: receiving audio data captured by a microphone of the user device while the user device is in a sleep mode, the audio data corresponding to playback of an item of media content output from an audio source in a vicinity of the user device; processing, while the user device remains in the sleep mode, the audio data to determine a corresponding bitstream of an audio watermark in the audio data; and based on the determined corresponding bitstream of the audio watermark in the audio data, determining, without accessing an audio watermark database to identify a matching audio watermark, to bypass performing speech recognition on a portion of the audio data following a hotword included in the audio data.

These and other implementations can each optionally include one or more of the following features. The audio data comprises an utterance of a command preceded by the hotword. The action of determining to bypass performing speech recognition on the audio data comprises bypassing performing speech recognition on a portion of the received audio data corresponding to the utterance of the command. The audio watermark is in a portion of the audio data that includes the hotword.

The operations further comprise, prior to processing the audio data to determine the corresponding bitstream of the audio watermark in the audio data, determining that the received audio data includes the hotword. The act of determining that the received audio data includes the hotword comprises: computing, without performing speech recognition, a hotword confidence score indicating a likelihood that the audio data includes the hotword; and determining that the hotword confidence score satisfies a hotword confidence score threshold.

The action of processing the audio data while the user device remains in the sleep state further comprises processing the audio data to identify the audio watermark in the audio data by detecting a spread spectrum shaping type of watermark. The user device remains in the sleep mode after determining to bypass performing speech recognition on the audio data.

The operations further comprise: analyzing the audio watermark to identify the source of the watermark in the audio data, wherein determining to bypass performing speech recognition on the audio data is further based on the source of the watermark in the audio data. The operations further comprise updating a log file to indicate the source of the watermark in the audio data.

Another innovative aspect of the subject-matter described in this application includes a user device as set forth in independent claim <NUM>.

A computing device may respond to hotwords that include only specific audio watermarks, thus saving battery power and processing capacity of the computing device. Network bandwidth may be preserved with fewer computing devices performing search queries upon receiving hotwords with audio watermarks. In addition, an audio watermark may be utilized to cause a user's computing device to convey information (e.g. the response to the spoken query or an alert of some kind) to the user in situations when they might not be able to hear it if it were e.g. output via a loudspeaker located in the user's vicinity. Such users may include those with hearing impairments or those listening to other audio via personal speakers (e.g. headphones) connected to their audio devices. For instance, a particular audio watermark may be understood by the computing device as indicating high priority, in which case the computing device may respond to the query received via the primary audio.

<FIG> illustrates an example system <NUM> for suppressing hotword triggers when detecting a hotword in recorded media. Briefly, and as described in more detail below, computing devices <NUM> and <NUM> receive audio <NUM> output from an audio source <NUM> (e.g., a television). The audio <NUM> includes an utterance of a predefined hotword and an audio watermark. Both computing devices <NUM> and <NUM> process the audio <NUM>, and determine that the audio <NUM> includes the predefined hotword. The computing devices <NUM> and <NUM> identify the audio watermark. Based on the audio watermark and the context or characteristics of the computing devices <NUM> and <NUM>, each of the computing devices <NUM> and <NUM> may perform speech recognition on the audio.

In the example shown in <FIG>, the audio source <NUM> is playing media content associated with Nugget World. During the media content, an actor in the media content says the utterance <NUM>, "Ok computer, what's in a nugget?" The utterance <NUM> includes the hotword <NUM> "Ok computer" and a query <NUM> "what's in a nugget?" The audio source <NUM> outputs the utterance <NUM> through a speaker. Any computing device in the vicinity with a microphone is able to detect the utterance <NUM>.

The audio of the utterance <NUM> includes an audible portion <NUM> and an audio watermark <NUM>. The creator of the media content may add the audio watermark <NUM> to ensure particular computing devices responds correctly to the utterance <NUM>. In some implementations, the audio watermark <NUM> may include audio frequencies that are higher or lower than human hearing range. For example, the audio watermark <NUM> may include frequencies that are greater than <NUM> or less than <NUM>. In some implementations, the audio watermark <NUM> may include audio that is within the human hearing range but is not detectable by humans because of its sounds similar to noise. For example, the audio watermark <NUM> may include a frequency pattern between <NUM> and <NUM>. The strength of different frequency bands may be imperceptible to a human, but may be detectable by a computing device. As illustrated by the frequency domain representation <NUM> of the audio <NUM> includes an audio watermark <NUM> that is in a higher frequency range than the audible portion <NUM>.

The computing devices <NUM> and <NUM> may be any type of device that is capable of receiving audio through a microphone. For example, computing devices <NUM> and <NUM> can be a desktop computer, laptop computer, a tablet computer, a wearable computer, a cellular phone, a smart phone, a music player, an e-book reader, a navigation system, a smart speaker and home assistant, wireless (e.g., Bluetooth) headset, hearing aid, smart watch, smart glasses, activity tracker, or any other appropriate computing device. As illustrated in <FIG>, computing device <NUM> is a smart phone and computing device <NUM> is a desktop computer. The audio source <NUM> can be any source of audio such as, for example, a television, a radio, a music player, a desktop computer, laptop computer, a tablet computer, a wearable computer, a cellular phone, or a smart phone. As illustrated in <FIG>, the audio source <NUM> is a television.

The computing devices <NUM> and <NUM> each receive the audio through a microphone. With respect to computing device <NUM>, the microphone may be part of an audio subsystem <NUM>. The audio subsystem <NUM> may include buffers, filters, analog to digital converters that are each designed to initially process the audio received through the microphone. The buffer may store the current audio received through the microphone and processed by the audio subsystem <NUM>. For example, the buffer stores the previous five seconds of audio data. Similarly, the microphone of computing device <NUM> may be part of an audio subsystem <NUM>. The audio subsystem <NUM> may include buffers, filters, analog to digital converters that are each designed to initially process the audio received through the microphone. The buffer may store the current audio received through the microphone and processed by the audio subsystem <NUM>. For example, the buffer stores the previous three seconds of audio data.

The computing devices <NUM> and <NUM> each include a hotworder <NUM> and <NUM>, respectively. The hotworders <NUM> and <NUM> are each configured to identify hotwords in audio received through the microphone and/or stored in the buffer. In some implementations, the hotworders <NUM> and <NUM> may be active at any time that the computing devices <NUM> and <NUM> are powered on. The hotworder <NUM> continuously analyzes the audio data stored in the buffer. The hotworder <NUM> computes a hotword confidence score that reflects the likelihood that current audio data in the buffer includes a hotword. To compute the hotword confidence score, the hotworder <NUM> may extract audio features from the audio data such as filterbank energies or mel-frequency cepstral coefficients. The hotworder <NUM> may use classifying windows to process these audio features such as by using a support vector machine or a neural network. In some implementations, the hotworder <NUM> does not perform speech recognition to determine a hotword confidence score. The hotworder <NUM> determines that the audio includes a hotword if the hotword confidence score satisfies a hotword confidence score threshold. For example, the hotworder <NUM> determines that the audio that corresponds to utterance <NUM> includes the hotword <NUM> if the hotword confidence score is <NUM> and the hotword confidence score threshold is <NUM>. The hotworder <NUM> may functional similarly to the hotworder <NUM>.

The computing devices <NUM> and <NUM> each include an audio watermark identifier <NUM> and <NUM>, respectively. The audio watermark identifiers <NUM> and <NUM> are each configured to process the audio received through the microphone and/or stored in the buffer and identify audio watermarks that are included in the audio. The audio watermark identifiers <NUM> and <NUM> may each be configured to detect spread spectrum and psychacoustic shaping types of watermarks. These types of watermarks may be located in frequency bands that overlap the frequency bands of the corresponding audio. A human may perceive these types of watermark as noise. The audio watermark identifiers <NUM> and <NUM> may also each be configured to detect high frequency watermarks. These types of watermarks may be located in frequency bands that are above the frequency bands of the corresponding audio. The frequency bands of the high frequency watermarks may be above the threshold of human hearing. The audio watermark identifiers <NUM> and <NUM> may also each be configured to detect low frequency watermarks. These types of watermarks may be located in frequency bands that are below the frequency bands of the corresponding audio. The frequency bands of the low frequency watermarks may be below the threshold of human hearing. In some implementations, the audio watermark identifiers <NUM> and <NUM> process the audio in response to the respective hotworder <NUM> and <NUM> detecting a hotword.

The audio watermark identifiers <NUM> and <NUM> may each be configured to separate the audio watermark and the primary audio. The primary audio may be the portion of the audio to which the audio watermark was added. For example, the primary audio may include the audible portion <NUM> which includes the audio corresponding to "Ok computer, what's in a nugget?" without watermark <NUM>. The audio watermark identifier <NUM> separates the audio <NUM> that was received through the microphone of the computing device <NUM> into the primary audio <NUM> and the audio watermark <NUM>. Similarly, the audio watermark identifier <NUM> separates the audio <NUM> that was received through the microphone of the computing device <NUM> into the primary audio <NUM> and the audio watermark <NUM>. In some implementations, audio watermark and the primary audio may overlap in the time domain.

In some implementations according to the claimed invention, the audio watermark identifiers <NUM> and <NUM> process the audio watermarks <NUM> and <NUM>, respectively, to identify a corresponding bitstream of the audio watermark. For example, the audio watermark identifier <NUM> may process the audio watermark <NUM> and determine that the audio watermark <NUM> corresponds to a bitstream of <NUM>. The audio watermark identifier <NUM> may preform similar processing on the audio watermark <NUM>.

The audio watermark comparers <NUM> and <NUM> each compare the respective audio watermarks <NUM> and <NUM> to the audio watermarks <NUM> and <NUM>, respectively. For example, the audio watermark comparer <NUM> may compare the frequency pattern or bitstream of watermark <NUM> to the audio watermarks <NUM>. The audio watermark comparer <NUM> may determine that the audio watermark <NUM> matches an audio watermark of Nugget World. The audio watermark comparer <NUM> may make a similar determination.

The audio watermarks <NUM> and <NUM> may contain audio watermarks that various entities embed in the audio of media content, or other distributed or broadcast audio, that contain a hotword. Nugget World may include a watermark in the audio <NUM> to ensure that only particular devices respond to the hotword, perform speech recognition on the audio, and execute the query <NUM>. Nugget World may provide the audio watermark <NUM> for inclusion in the audio watermarks <NUM> and <NUM> along with instructions for when devices should respond to the hotword with the audio watermark <NUM>. For example, Nugget World may include in the audio watermarks <NUM> and <NUM> instructions for any device with a Nugget World user identifier that are located in a Nugget World restaurant to respond to the hotword with the audio watermark <NUM>. In some implementations, the audio watermarks <NUM> and <NUM> are stored on the computing devices <NUM> and <NUM> and updated periodically, for example, once per day. In some implementations, the audio watermarks <NUM> or <NUM>, the audio watermark identifiers <NUM> and <NUM>, and/or the audio watermark comparers <NUM> and <NUM> may be located on a remote server. In this instance, the computing device <NUM> or <NUM> may communicate with the remote server over a network.

The computing device <NUM> extracts the audio watermark <NUM> and matches the audio watermark <NUM> to a Nugget World watermark. Based on the instructions in the audio watermarks <NUM> for the Nugget World watermark, the computing device <NUM> may perform speech recognition on the primary audio <NUM> and execute any queries or commands included in the corresponding transcription. The instructions may include a set of rules for the computing device <NUM> to follow to determine whether to perform speech recognition.

The computing device <NUM> includes a location detector <NUM>. The location detector <NUM> may generate geographic location data that reflects the location of the computing device. The location detector <NUM> may use any geographic location technique, for example, GPS, triangulation, and/or any other similar locating technique. In some implementations, the location detector <NUM> may access map or location data that indicates locations of various points of interest. The location detector <NUM> may further identify a point of interest where the computing device is located. For example, the location detector <NUM> may determine that the computing device <NUM> is located at Nugget World.

The computing device <NUM> includes a device identifier <NUM>. The device identifier <NUM> includes a device identity <NUM> that identifies the type of device for computing device <NUM>. The device identity <NUM> may be a desktop computer, laptop computer, a tablet computer, a wearable computer, a cellular phone, a smart phone, a music player, an e-book reader, a navigation system, a smart speaker and home assistant, or any other appropriate computing device. For example, the device identity <NUM> of computing device <NUM> is phone.

The computing device <NUM> includes a user identifier <NUM>. The user identifier <NUM> includes a user identity <NUM> that identifies the user for computing device <NUM>. The user identity <NUM> may be an email address, phone number, or any other similar type of unique user identifier. For example, the user identity <NUM> of computing device <NUM> is user@example. The user identifier <NUM> may be input by the user <NUM>.

The computing device <NUM> includes user settings <NUM>. The user settings <NUM> may be provided by the user <NUM> and may include additional rules for how the computing device <NUM> should respond to a hotword. For example, the user settings <NUM> may include a rule that the computing device <NUM> does not respond to any hotwords that include audio watermarks unless the computing device <NUM> receives the hotword that includes an audio watermark while at home. As another example, the user settings <NUM> may include a rule that the computing device <NUM> does not responds to hotwords that include audio watermarks that correspond to particular entities (e.g. owners or creators of media content), such as Chicken World. In some implementations, the user <NUM> may agree to allow the computing device <NUM> to respond to hotwords with a watermark of a particular entity.

In the example illustrated in <FIG>, speech recognizer <NUM> remains inactive as indicated by the speech recognizer status <NUM>. The computing device <NUM> set the speech recognizer status <NUM> to inactive based on applying of the instructions corresponding to the audio watermark <NUM> as stored in the audio watermarks to the device location, user settings <NUM>, device identity <NUM>, and the user identity <NUM>. For example, the instructions corresponding to the audio watermark <NUM> may be to set the speech recognizer status <NUM> to active if the user identity <NUM> is a Nugget World identifier and the device is located at a Nugget World restaurant. For computing device <NUM>, the user identity <NUM> is not a Nugget World identifier. Therefore, the speech recognizer status <NUM> is inactive.

In some implementations, the user interface generator <NUM> of the computing device <NUM> may provide, to a display of the computing device, data for a graphical interface. The graphical interface may indicate the processes or actions of the computing device <NUM> while, in advance of, or after the computing device has performed the processes or actions. For example, the user interface generator <NUM> may display an interface indicating that the computing device <NUM> is processing received audio, that the computing device <NUM> is identifying the audio watermark <NUM>, the speech recognizer status <NUM>, and/or any properties or rules of the identified audio watermark <NUM>.

In some implementations, the user interface generator <NUM> may generate an interface indicating that the speech recognizer status <NUM> is inactive. The interface may also include a user selectable option to override the speech recognizer status <NUM>. For example, user <NUM> may select an option to set the speech recognizer status <NUM> to active. Having heard the query <NUM> "what's in a nugget," the user <NUM> may be curious and request that the computing device <NUM> process the query <NUM> and provide an output.

In some implementations, the computing device <NUM> may include an audio watermark log <NUM>. The audio watermark log <NUM> may include data indicating a number of times that the computing device <NUM> has received each audio watermark. For example, each time that the computing device <NUM> receives and identifies audio watermark <NUM>, the computing device <NUM> may store data indicating the receipt of the audio watermark <NUM> in the audio watermark log <NUM>. The data may include a timestamp, a device location, any relevant user settings, user identifiers, and any other similar information. In some implementations, the computing device <NUM> may provide the data of the audio watermark log <NUM> to an aggregate audio watermark log on a server that combines the audio watermark logs from different computing devices that receive audio watermarks. The aggregate audio watermark log may include the user identities for the receiving computing device, a device identifier, and data stored in the audio watermark log <NUM>. In some implementations, the data in the aggregate audio watermark log and the audio watermark log <NUM> may synchronize. In this instance, the audio watermark log <NUM> may include additional log data from different devices as well as data identifying the different devices, different users, location information, timestamp data, and other related information.

In some implementations, the instructions for a particular audio watermark may include instructions related to the data stored in the audio watermark log <NUM>. The instructions may relate to a particular number of times that a hotword marked with a particular audio watermark should activate a speech recognizer. For example, the instructions may indicate that within <NUM>-hour period, the audio watermark <NUM> should only activate the speech recognizer <NUM> once.

In some implementations, the creator of the media content on audio device <NUM> may access the aggregate audio watermark log to identify the details related to each time the hotword <NUM> and corresponding audio watermark <NUM> activated a speech recognizer. In some implementations, a user may indicate through the user settings on the device for the computing device not to upload the audio watermark log to the aggregate audio watermark log.

The computing device <NUM> processes the audio watermark <NUM> in a similar fashion to the computing device <NUM> processing the audio watermark <NUM>. In particular, the computing device <NUM> extracted the audio watermark <NUM> and matches the audio watermark <NUM> to a Nugget World watermark. Based on the instructions in the audio watermarks <NUM> for the Nugget World watermark, the computing device <NUM> may perform speech recognition on the primary audio <NUM> and execute any queries or commands included in the corresponding transcription. The instructions may include a set of rules for the computing device <NUM> to follow to determine whether to perform speech recognition.

The computing device <NUM> includes a location detector <NUM>. The location detector <NUM> may generate geographic location data that reflects the location of the computing device. The location detector <NUM> may use any geographic location technique, for example, GPS, triangulation, and/or any other similar locating technique. In some implementations, the location detector <NUM> may access map or location data that indicates locations of various points of interest. The location detector <NUM> may further identify a point of interest where the computing device <NUM> is located. For example, the location detector <NUM> may determine that the computing device <NUM> is located at Nugget World.

The computing device <NUM> includes a device identifier <NUM>. The device identifier <NUM> includes a device identity <NUM> that identifies the type of device for computing device <NUM>. The device identity <NUM> may be a desktop computer, laptop computer, a tablet computer, a wearable computer, a cellular phone, a smart phone, a music player, an e-book reader, a navigation system, a smart speaker and home assistant, or any other appropriate computing device. For example, the device identity <NUM> of computing device <NUM> is desktop computer.

The computing device <NUM> includes a user identifier <NUM>. The user identifier <NUM> includes a user identity <NUM> that identifies the user for computing device <NUM>. The user identity <NUM> may be an email address, phone number, or any other similar type of unique user identifier. For example, the user identity <NUM> of computing device <NUM> is store@nuggetworld. The user identifier <NUM> may be input by a user.

The computing device <NUM> includes user settings <NUM>. The user settings <NUM> may be provided by a user and may include additional rules for how the computing device <NUM> should respond to a hotword. For example, the user settings <NUM> may include a rule that the computing device <NUM> does not respond to any hotwords that include audio watermarks unless the computing device <NUM> is located at a Nugget World restaurant. As another example, the user settings <NUM> may include a rule that the computing device <NUM> does not respond to any hotwords other than those marked with audio watermarks from Nugget World. As another example, the user settings <NUM> may indicate that the computing device <NUM> does not respond to any hotwords with any type of audio watermark outside of Nugget World's opening hours.

In the example illustrated in <FIG>, speech recognizer <NUM> is active as indicated by the speech recognizer status <NUM>. The computing device <NUM> sets the speech recognizer status <NUM> to active based on applying of the instructions corresponding to the audio watermark <NUM> as stored in the audio watermarks to the device location, user settings <NUM>, device identity <NUM>, and the user identity <NUM>. For example, the instructions corresponding to the audio watermark <NUM> may be to set the speech recognizer status <NUM> to active if the user identity <NUM> is a Nugget World identifier and the device is located at a Nugget World restaurant. For computing device <NUM>, the user identity <NUM> is a Nugget World identifier and the location is at a Nugget World. Therefore, the speech recognizer status <NUM> is active.

The speech recognizer <NUM> performs speech recognition on the primary audio <NUM>. The speech recognizer <NUM> generates the transcription "what's in a nugget. " The computing device <NUM> may provide the transcription to a search engine if the transcription corresponds to a query. The computing device may execute a command if the transcription corresponds to a command. In the example in <FIG>, the computing device <NUM> provides the transcription of the primary audio <NUM> to a search engine. The search engine returns a result, and the computing device <NUM> may output the result through a speaker, which may be for example a speaker of the computing device, or personal speakers, such as headphones, earphones, earbuds etc., connected to the computing device. Outputting the result via personal speakers may be useful, for instance, for causing information to be provided to the user when they would be unable to hear the information if it were instead output as part of the primary audio. For example, in the Nugget World example, the computing device <NUM> may output the audio <NUM> of "a nugget contains chicken. " In some implementations, the user interface generator <NUM> may display the search results on the display of the computing device <NUM>. This may be particularly useful for causing information to be provided to users such as those with impaired hearing who might not be able to hear the information if it were instead output as part of the primary audio or via a speaker associated with the computing device.

In some implementations, the user interface generator <NUM> may provide additional interfaces. The graphical interface may indicate the processes or actions of the computing device <NUM> while, in advance of, or after the computing device <NUM> has performed the processes or actions. For example, the user interface generator <NUM> may display an interface indicating that the computing device <NUM> is processing received audio, that the computing device <NUM> is identifying the audio watermark <NUM>, the speech recognizer status <NUM>, and/or any properties or rules of the identified audio watermark <NUM>.

In some implementations, the user interface generator <NUM> may generate an interface indicating that the speech recognizer status <NUM> is active. The interface may also include a user selectable option to override the speech recognizer status <NUM>. For example, a user may select an option to set the speech recognizer status <NUM> to suppress any actions related to the transcriptions. In some implementations, the user interface generator <NUM> may generate an interface to update the user settings <NUM> based on the recently received override and the current properties of the computing device <NUM>. The user interface generator <NUM> may also provide a similar interface after receiving an override command.

In some implementations, the computing device <NUM> may include an audio watermark log <NUM>. The audio watermark log <NUM> may store data similar to audio watermark log <NUM> based on the audio watermarks received by the computing device <NUM>. The audio watermark log <NUM> may interact with an aggregate audio watermark log in a similar fashion to the audio watermark log <NUM>.

In some implementations, the computing devices <NUM> and <NUM> may perform speech recognition on the primary audio <NUM> and <NUM>, respectively, independent of the rules stored in the audio watermarks <NUM> and <NUM>. The audio watermarks <NUM> and <NUM> may include rules related to performing actions of the primary audio based in part on the transcription.

<FIG> illustrates an example process <NUM> for suppressing hotword triggers when detecting a hotword in recorded media. In general, the process <NUM> performs speech recognition on audio corresponding to media content based on the audio including a hotword and a watermark. The process <NUM> will be described as being performed by a computer system comprising one or more computers, for example, the computing devices <NUM> or <NUM> as shown in <FIG>.

The system receives audio corresponding to playback of an item of media content (<NUM>). In some implementations according to the claimed invention, the audio is received through a microphone of the system. The audio may correspond to the audio of media content played on a television or the radio.

The system determines that the audio includes an utterance of a predefined hotword and an audio watermark (<NUM>). In some implementations, the audio watermark is an inaudible portion of the audio. For example, the audio watermark may be located in frequency bands that are above or below those of human hearing. In some implementations, the audio watermark is audible, but sounds similar to noise. In some implementations, the audio watermark overlaps the audio of the predefined hotword. In some implementations, the system determines that the audio includes the predefined hotword. In response to that determination, the system processes the audio to determine whether the audio includes an audio watermark.

The system compares the audio watermark to one or more audio watermarks (<NUM>). In some implementations, the system may compare the audio watermark to a database of audio watermarks. The database may be stored on the system or on a different computing device. The system may compare a digital representation of the audio watermark or an analog representation of the audio watermark in the time and/or frequency domain. The system may identify a matching audio watermark and process the audio according to a rule specified in the database for the identified audio watermark. In some implementations, the system may identify a source, or owner, of the audio watermark. For example, the source, or owner, may be the entity, Nugget World. The system may update a log file to indicate that the system received a hotword with a Nugget World audio watermark.

The system based on comparing the audio watermark to the one or more audio watermarks, determines whether to perform speech recognition on a portion of the audio following the predefined hotword (<NUM>). Based on the rule specified in the database for identified audio watermark, the source of the audio watermark, and the context of the system, the system determines whether to perform speech recognition on the audio following the predefined hotword. The context of the system may be based on any combination of a type of the system, a location of the system, and any user settings. For example, the rule may specify that mobile phones that are located at the user's home should perform speech recognition on the audio upon receiving a hotword with a particular watermark from the management company of the apartment where the user lives. In some implementations, the system determines, based on comparing the audio watermark to the one or watermarks or based on analyzing the audio watermark, whether to perform natural language processing on the portion of the audio following the predefined hotword. The system may perform natural language processing in additional to the speech recognition or in place of the speech recognition.

In instances where the system determines to perform speech recognition, the system generates a transcription of the audio that follows the hotword. The system executes a command included in the transcription, such as adding a calendar appointment for an apartment building meeting, or submits a query to a search engine. The system may output the search results over a speaker of the system or on the display of the system or both.

In instances where the system determines not to perform speech recognition, the system may remain in sleep mode standby mode, or low-energy mode. The system may be in sleep mode standby mode, or low-energy mode while processing the audio and may remain in sleep mode, standby mode, or low-energy mode if the system does not perform speech recognition on the audio. In some implementations, the user <NUM> may be using the computing device <NUM> while the computing device <NUM> receives the audio <NUM>. For example, the user <NUM> may be listening to music or viewing a photo application. In this instance, the hotword and audio watermark processing may occur in the background, and the user's activities may be uninterrupted. In some implementations, the audio may not include an audio watermark. In this instance the system may perform speech recognition on the audio after the hotword and execute any command or query included in the audio.

In some implementations, the system may determine a type of media content for the audio. The system may compare the audio watermark to the audio watermarks included in the audio watermark database. The system may identify a matching audio watermark in the audio watermark database, and the matching audio watermark may identify a type of media content for that particular audio watermark. The system may apply the rule for the identified type of media content. For example, an audio watermark database may indicate that an audio watermark is included in sales media, directional media, commercial media, political media, or any other type of media. In this instance, the system may follow a general rule for the type of media. For example, the rule may be to only perform speech recognition for commercial media when the system is located at home. The rule may also be a rule that is specific to the received audio watermark. In some implementations, the system may also record the type of media content in the audio watermark log.

In some implementations, the system may analyze the audio watermark. The system may analyze the audio watermark in place of or in combination with comparing the audio watermark to the audio watermark database. The audio watermark may encode an action, an identifier, a rule, or any other similar data. The system may decode the audio watermark and process the audio according to the decoded audio watermark. The audio watermark may be encoded as a header and payload. The system may identify the header, which may be common to all or almost all audio watermarks or may identify particular groups of audio watermarks. A payload may follow the header and encode the action, identifier, rule, or other similar data.

The system may apply the rule encoded in the audio watermark. For example, the rule may be for the system to perform speech recognition on the portion of the audio following the hotword if the system is a smart speaker that is located at a location that corresponds to the user identifier stored in the system. In this instance and according to the claimed invention, the system may have no need to access an audio watermark database. In some implementations, the system may add the rule encoded in the audio watermark to an audio watermark database.

The system may use the data encoded in the audio watermark in combination with the audio watermark database. For example, data encoded in the audio watermark may indicate that the audio is political media content. The system may access a rule that corresponds to the audio watermark and that specifies for the system to perform speech recognition on audio that includes the political media content watermark or a commercial media content watermark when the system is a located in the user's home. In this instance, the audio watermark may include a header or other portion that the system may use to identify corresponding audio watermarks in the audio watermark database. The payload may encode the type of media content or other data such as action, identifier, or rule.

<FIG> shows an example of a computing device <NUM> and a mobile computing device <NUM> that can be used to implement the techniques described here. The mobile computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, wireless (e.g., Bluetooth) headsets, hearing aid, smart watches, smart glasses, activity trackers, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

The computing device <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, a high-speed interface <NUM> connecting to the memory <NUM> and multiple high-speed expansion ports <NUM>, and a low-speed interface <NUM> connecting to a low-speed expansion port <NUM> and the storage device <NUM>. Each of the processor <NUM>, the memory <NUM>, the storage device <NUM>, the high-speed interface <NUM>, the high-speed expansion ports <NUM>, and the low-speed interface <NUM>, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a GUI on an external input/output device, such as a display <NUM> coupled to the high-speed interface <NUM>. Also, multiple computing devices may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The high-speed interface <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low-speed interface <NUM> manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface <NUM> is coupled to the memory <NUM>, the display <NUM> (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports <NUM>, which may accept various expansion cards (not shown). In the implementation, the low-speed interface <NUM> is coupled to the storage device <NUM> and the low-speed expansion port <NUM>. The low-speed expansion port <NUM>, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, a microphone, speakers, or a networking device such as a switch or router, e.g., through a network adapter.

The mobile computing device <NUM> includes a processor <NUM>, a memory <NUM>, an input/output device such as a touch-enabled display <NUM>, a communication interface <NUM>, and a transceiver <NUM>, among other components.

The memory <NUM> stores information within the mobile computing device <NUM>. An expansion memory <NUM> may also be provided and connected to the mobile computing device <NUM> through an expansion interface <NUM>, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory <NUM> may provide extra storage space for the mobile computing device <NUM>, or may also store applications or other information for the mobile computing device <NUM>. Specifically, the expansion memory <NUM> may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory <NUM> may be provided as a security module for the mobile computing device <NUM>, and may be programmed with instructions that permit secure use of the mobile computing device <NUM>.

The memory may include, for example, flash memory and/or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier. that the instructions, when executed by one or more processing devices (for example, processor <NUM>), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory <NUM>, the expansion memory <NUM>, or memory on the processor <NUM>). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver <NUM> or the external interface <NUM>.

The mobile computing device <NUM> may communicate wirelessly through the communication interface <NUM>, which may include digital signal processing circuitry where necessary. The communication interface <NUM> may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication may occur, for example, through the transceiver <NUM> using a radio-frequency. In addition, a GPS (Global Positioning System) receiver module <NUM> may provide additional navigation- and location-related wireless data to the mobile computing device <NUM>, which may be used as appropriate by applications running on the mobile computing device <NUM>.

The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

Claim 1:
A computer-implemented method which when executed on data processing hardware of a user device causes the data processing hardware to perform operations comprising:
receiving audio data captured by a microphone of the user device while the user device is in a sleep mode, the audio data corresponding to playback of an item of media content output from an audio source in a vicinity of the user device;
processing, while the user device remains in the sleep mode, the audio data to determine a corresponding bitstream of an audio watermark in the audio data; and
based on the determined corresponding bitstream of the audio watermark in the audio data, determining, without accessing an audio watermark database to identify a matching audio watermark, to bypass performing speech recognition on a portion of the audio data following a hotword included in the audio data.