Patent Description:
A speech-enabled environment (e.g., home, workplace, school, automobile, etc.) allows a user to speak a query or a command out loud to a computer-based system that fields and answers the query and/or performs a function based on the command. The speech-enabled environment can be implemented using a network of connected microphone devices distributed through various rooms or areas of the environment. These devices may use hotwords to help discern when a given utterance is directed at the system, as opposed to an utterance that is directed to another individual present in the environment. Accordingly, the devices may operate in a sleep state or a hibernation state and wake-up only when a detected utterance includes a hotword. Once the device has awoken by the hotword within the detected utterance, the device performs further processing on the hotword and/or one or more terms that follow the hotword. In other words, the hotword and/or one or more terms form a query or a voice command to be performed by the device. As speech synthesizers become more prevalent within speech-enabled environments, a synthesized utterance containing a hotword, or including other words/sub-words that sound like the hotword, may cause the device to wake-up from the sleep/hibernation state and begin processing the synthesized utterance even though the synthesized utterance is not directed to that device. Stated another way, the synthesized speech unintentionally activates the device, often to the dismay of a user of a speech synthesizer. <CIT>, <NUM>. <NUM> discloses a device and method for detecting wake words and audio commands that should be disregarded, since they originate from multi-media audio data and not from the user. Consequently, a system that receives utterances within the environment must have some way of discerning between an utterance of human speech directed at the system and an utterance of synthesized speech output from a nearby device not directed at the system.

Aspects of the present disclosure are as set out in the accompanying claims.

A method for preventing initiation of a wake-up process on a user device. The method includes receiving, at data processing hardware of a speech synthesis device, text input data for conversion into synthesized speech; and determining, by the data processing hardware and using a hotword-aware model trained to detect a presence of at least one hotword assigned to a user device, whether a pronunciation of the text input data includes the hotword, the hotword, when included in audio input data received by the user device, configured to initiate a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the audio input data. When the pronunciation of the text input data includes the hotword, the method also includes generating an audio output signal from the text input data and providing, by the data processing hardware, the audio output signal to an audio output device to output the audio output signal. The audio output signal when captured by an audio capture device of the user device, configured to prevent initiation of the wake-up process on the user device.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, determining whether the pronunciation of the text input data includes the hotword includes determining that of at least one of a word, a sub-word, or a text-to-speech sequence of the text input data is associated with the hotword. The hotword-aware model may be trained on a text-to-speech sequence or audio representation of the hotword assigned to the user device. Additionally, the text input data may include a first language and the audio output signal may include a translation of the text input data in a different language.

In some examples, the method also includes detecting, by the data processing hardware, a presence of the user device within an operating environment of the speech synthesis device; and querying, by the data processing hardware, the user device to obtain the hotword assigned to the user device for training the hotword-aware model. Additionally or alternatively, the method may include querying a remote hotword repository to obtain at least the hotword assigned to the user device for training the hotword-aware model.

In some implementations, generating the audio output signal from the text input data includes inserting a watermark to the audio output signal that indicates the audio output signal corresponds to synthesized speech and instructs a hotword detector of the user device to ignore detection of the hotword in the synthesized speech. In other implementations, generating the audio data includes determining a speech waveform that represents a text-to-speech output for the text input data and altering the speech waveform by removing or altering any sounds associated with the hotword to evade detection of the hotword by a hotword detector of the user device. In yet another implementations, generating the audio data includes determining a speech waveform that represents the text input data and filtering the audio waveform to evade detection of the hotword by a hotword detector of the user device.

Another aspect of the present disclosure provides a method for preventing initiation of a wake-up process on a user device. The method includes receiving, at a hotword detector of a user device, audio input data containing a hotword, the hotword configured to initiate a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the audio input data; determining, by the hotword detector, whether the audio input data includes synthesized speech using a hotword detector model configured to detect the hotword in the audio input data and a presence of synthesized speech; and when the audio input data comprises synthesized speech, preventing, by the hotword detector, initiation of the wake-up process on the user device for processing the hotword and/or the one or more other terms following the hotword in the audio input data.

This aspect may include one or more of the following optional features. In some implementations, the hotword detector model is trained on a plurality of training samples that include positive training samples and negative training samples. The positive training samples include human-generated audio data corresponding to one or more users speaking the hotword assigned to the user device. The negative training samples include synthesized speech utterances output from one or more speech synthesizer devices. In some examples, at least one of the synthesized speech utterances of the negative training samples pronounce the hotword assigned to the user device. In other examples. None of the synthesized speech utterances of the negative training samples pronounce the hotword assigned to the user device. Determining whether the audio input data includes synthesized speech may include using the hotword detector model to detect the presence of synthesized speech in the audio input data through an analysis of acoustic features of the audio input data without transcribing or semantically interpreting the audio input data.

Another aspect of the present disclosure provides a system for preventing initiation of a wake-up process on a user device. The system includes data processing hardware of a speech synthesis device and memory hardware in communication with the data processing hardware. The memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations that include receiving text input data for conversion into synthesized speech and determining, using a hotword-aware model trained to detect a presence of at least one hotword assigned to a user device, whether a pronunciation of the text input data includes the hotword, The hotword when included in audio input data received by the user device, configured to initiate a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the audio input data. When the pronunciation of the text input data includes the hotword, the operations also include generating an audio output signal from the text input data and providing the audio output signal to an audio output device to output the audio output signal. The audio output signal when captured by an audio capture device of the user device, configured to prevent initiation of the wake-up process on the user device.

In some examples, the operations also include detecting a presence of the user device within an operating environment of the speech synthesis device and querying the user device to obtain the hotword assigned to the user device for training the hotword-aware model. Additionally or alternatively, the operations may also include querying a remote hotword repository to obtain at least the hotword assigned to the user device for training the hotword-aware model.

Another aspect of the present disclosure provides a system for preventing initiation of a wake-up process on a user device. The system includes data processing hardware of a user device and memory hardware in communication with the data processing hardware. The memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations that include receiving, at a hotword detector of a user device, audio input data containing a hotword, the hotword configured to initiate a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the audio input data; determining, by the hotword detector, whether the audio input data includes synthesized speech using a hotword detector model configured to detect the hotword in the audio input data and a presence of synthesized speech; and when the audio input data comprises synthesized speech, preventing, by the hotword detector, initiation of the wake-up process on the user device for processing the hotword and/or the one or more other terms following the hotword in the audio input data.

In a speech-enabled environment, a user's manner of interacting with a computer-based system, that may be implemented using a network of connected microphone devices distributed throughout the environment (e.g., rooms or other areas of a home, workplace, school, etc.), is designed to be primarily, if not exclusively, through voice input (i.e., audio commands). More devices are using audio commands to instruct operation of user devices. Through the use of a "hotword" (also referred to as an "attention word", "wake-up phrase/word", "trigger phrase", or "voice action initiation command"), in which by agreement a predetermined term (i.e., keyword) that is spoken to invoke the attention of the system is reserved, the system is able to discern between utterances directed to the system (i.e., to initiate a wake-up process for processing one or more terms following the hotword in the utterance) and utterances directed to an individual in the environment. To put another way, a user device may operate in a low power mode, but upon detecting a hotword, the user device may switch into a full power mode so that all audio data captured by a microphone is detected, processed, and analyzed. However, as the output of synthesized speech from speech synthesizers (e.g., text-to-speech (TTS) systems) becomes more prevalent within speech-enabled environments, synthesized speech including a hotword assigned to a nearby user device, or words or sub-words that make up or sound like the hotword, can unintentionally cause a hotword detector (e.g., hotworder) on the user device to detect the presence of the hotword and initiate the wake-up process for processing terms in the synthesized speech. As used herein, the terms "synthesized speech" and "synthesized utterance" are used interchangeably. As used herein, synthesized speech output from a TTS system or speech synthesis device includes a machine output from a non-audible originating data input. The machine output can inform a user of operations being performed by a device associated with the TTS system or to acknowledge instructions provided by the user to the device associated with the TTS system. Accordingly, synthesized speech is distinguishable from broadcasted audio output from a television, multimedia set-top box, stereo, radio, computer system, or other type of device capable of outputting broadcasted audio.

For instance, in a speech-enabled environment, such as a home of a user, the user may have one or more mobile devices (e.g., smart phone and/or tablet) and a smart speaker/display device. The smart speaker/display device may operate as a digital assistant for outputting synthesized speech as well as triggering processing of a voice query or a voice command to be performed when the voice query or the voice command is preceded by a hotword assigned to the corresponding user device. Scenarios may occur when synthesized speech output from one of the devices (e.g., the smart speaker) that is directed toward the user contains one or more words or sub-words that make up a hotword assigned to one of the other devices (e.g., the user's tablet) in the environment. For example, the term "Dog" may be assigned as a hotword to the user's tablet and a portion of the synthesized speech may recite the term "hotdog. " As a result, a microphone of the other device may capture the synthesized speech and a hotword detector may detect the term "dog" preceded by the term "hot" and trigger the user's tablet to unintentionally initiate the wake-up process. Thus, pronunciation of the hotword in the synthesized speech may unintentionally cause a nearby speech-enabled device to transition from a sleep/hibernation state to an active state where the nearby speech-enabled device begins processing (i.e., transcribing and/or semantically interpreting) the synthesized speech.

It is an object of the present disclosure to avoid initiation of the wake-up process of one or more other user devices caused by the use of a hotword, or other term that sounds like the hotword, generated by TTS audio (e.g., synthesized speech). This will prevent unintended initiation of the wake-up process, thereby allowing user devices to remain in a lower power state for longer to conserve power.

In order to prevent unintentional initiation of the wake-up process in response to detecting pronunciation of a hotword in a synthesized utterance, implementations herein are directed toward injecting hotwords assigned to nearby devices into a training pipeline of a TTS system to generate a hotword-aware model for use in detecting a presence of the hotwords. The hotword-aware model may be trained on any combination of hotwords assigned to nearby devices, a list of hotwords associated with one or more devices owned-controlled by a specific user, and/or a list of all potential hotwords that may be assigned to any given device for initiating the wake-up process. For instance, a speech synthesizer device may use the hotword-aware model to determine whether a pronunciation of text input data for conversion into synthesized speech includes the hotword. In some examples, the hotword-aware model is trained on an audio representation (e.g., acoustic features) of the hotword such as a sequence or string of the hotword. Accordingly, a speech synthesis device that receives text input data (text and content) for conversion into synthesized speech may pre-process the text input data to obtain individual sequences (TTS sequences) and use the hotword-aware model to recognize sequences that, when pronounced audibly, compose the hotword or compose sound-alike phrases of the hotword by identifying matches or similarities between the TTS sequences and a hotword sequence obtained from the hotword-aware model. For instance, text input data that includes the phrase "dawg" would compose a sound-alike phrase of a hotword for the term "dog" when pronounced audibly. Accordingly, the hotword-aware model is trained to detect whether a pronunciation of text input data includes the hotword (e.g., composes the hotword or composes sound-alike phrases of the hotword). The TTS system may include a multi-lingual TTS system trained on multiple languages such that the hotword aware model is trained to detect the hotword or sound-alike phrases of the hotword in multiple languages.

If the speech synthesis device simply held a log or a whitelist of known hotwords in text form, as opposed to using the hotword-aware model, the speech synthesis device would fail to recognize misspelled words in the text input data that compose the hotword and fail to recognize sub-words within a word that compose the hotword. For instance, if the speech synthesis device simply referenced a whitelist of known hotwords, the speech synthesis device would fail to recognize text input data for the phrase "dawg" composing a hotword for the term "dog" (unless the spelling of "dawg" were included in the whitelist) and would fail to recognize the sub-word "dog" in text input data for the phrase "hotdog" (unless "hotdog" was included in the whitelist).

Once the speech synthesis device determines that a pronunciation of text input data includes the hotword, implementations further include a waveform generator of the speech synthesis device generating an audio output signal for the synthesized speech that is configured to prevent initiation of a wake-up process on a nearby user device when the audio output signal is captured by a microphone of the nearby user device. In some examples, the waveform generator uses unit selection logic for generating the output audio signal. In these examples, the waveform generator may emit a known watermark over the audio sequence where the known watermark is identifiable to a hotword detector on the nearby user device; thus, the hotword detector on the nearby user device would simply ignore the audio output signal with the known watermark even though the audio output signal pronounces the hotword. Alternatively, the unit selection logic may select an alternative variation of units (or a subset of the units) used to generate the synthesized speech (e.g., audio output signal) that are known to be adversarial to a hotword detection model used by the hotword detector of the nearby user device. Here, the hotword detection model may be trained on these same adversarial units so that the hotword detector knows to ignore any utterances including these units during inference (i.e. a non-training mode of the hotword detector), and thereby prevent initiation of the wake-up process even when the utterance contains a hotword. Moreover, the waveform generator could distort the synthesized speech using a filter trained against the hotword detector of the nearby user device so that the hotword detector ignores or fails to detect the synthesized speech.

In other examples, the waveform generator may generate the output audio signal by using a neural network (e.g., based on WaveNet) to output an audio sequence of synthesized phonemes that represent the text input data. In these examples, when a portion of the synthesized phonemes forms the hotword, the waveform generator may provide additional conditioning information that causes the neural network to emit a known watermark over the audio sequence that would be identifiable to a hotword detector on the nearby user device so that the nearby user device would simply ignore the audio output signal even though the hotword is pronounced. In other words, the presence of a watermark is used to instruct the nearby user device to ignore the pronounced hotword. Alternatively, segments of synthesized speech output from the neural network that compose the hotword (or compose sound-alike phrases of the hotword) may be modified (e.g., distorted) to generate the output audio signal in a manner that is adversarial to detection by the hotword detector of the nearby user device.

Additionally or alternatively, implementations may further include injecting synthesized speech utterances into a training pipeline of a hotword detector to generate a hotword detector model. The hotword detector model is configured to detect a presence of synthesized speech in audio input data received by the hotword detector. For instance, a hotword detector trainer may train the hotword detector to detect hotwords in utterances and to further determine whether or not an utterance includes synthesized speech, e.g., audio data output from a speech synthesis device (e.g., TTS system). Thus, when a microphone on a user device captures an utterance that contains a hotword assigned to the user device, the hotword detector will simply ignore the presence of the hotword in the captured utterance if the hotword detector detects that the utterance includes synthesized speech, thereby preventing initiation of the wake-up process on the user device. In some examples, the hotword detector model is trained on positive training examples that include human-generated audio data corresponding to one or more users speaking the hotword assigned to the user device, and negative training samples including synthesized speech utterances output from one or more speech synthesizer devices. By training the hotword detector model to detect the presence of synthesized speech in audio input data, the hotword detector may advantageously use the hotword detector model to detect the presence of synthesized speech through an analysis of acoustic features of received audio input data without transcribing or semantically interpreting the audio input data.

Referring to <FIG>, in some implementations, a speech-enabled system <NUM> includes one or more user devices <NUM>, 110a-b. For example, the speech-enabled system <NUM> includes two user devices 110a, 110b located proximate to one another and connected to a remote server <NUM> (e.g., cloud computing enviornment) via a network <NUM>. The user devices 110a, 110b may or may not communicate with one another. Each user device <NUM> is configured to capture sounds corresponding to an utterance <NUM> from a user <NUM>. The user <NUM> may speak the utterance <NUM> out loud as a query or a command. The speech-enabled system <NUM> may field the query or the command by answering the query and/or causing the command to be performed. Each user device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and storing instructions, that when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations. Each user device <NUM> further includes an audio capture device (e.g., microphone) <NUM> for capturing and converting spoken utterances <NUM> within the speech-enabled system <NUM> into electrical signals and a speech output device (e.g., a speaker) <NUM> for communicating an audible audio signal (e.g., as output audio data from a user device <NUM>).

Each user device <NUM> may be associated with the user <NUM> and capable of processing the utterance <NUM> from the associated user <NUM> when the utterance <NUM> begins with a hotword <NUM>. A hotword <NUM> may be a spoken phrase that causes either of the user devices <NUM> to treat a subsequently spoken phase as a voice input for the system. In other words, a hotword <NUM> may be a spoken phrase that explicitly indicates that a spoken input is to be treated as a voice command. Namely, a hotword <NUM> may be a spoken phrase that triggers endpointing, automated speech recognition, or semantic interpretation on the hotword <NUM> or one or more terms that follow the hotword <NUM>. In other words, referring to a "hotword" refers to a word or phrase that is a designated hotword or sounds similar to at least a portion of the designated hotword (e.g., sounds similar to a hotword in other languages).

To detect the presence of hotwords <NUM> within the utterance <NUM>, each user device <NUM> includes a hotword detector <NUM>. The hotword detector <NUM> may receive sounds corresponding to the utterance <NUM> and determine whether the utterance <NUM> includes a term that has been designated or assigned as a hotword <NUM>. In some examples, the hotword detector <NUM> detects acoustic features of captured sound from the utterance <NUM>. Here, when the acoustic features are characteristic of the hotword <NUM>, the hotword detector <NUM> identifies the hotword <NUM>. With the detection of a hotword <NUM>, the hotword detector <NUM> may initiate the wake-up process and further processes for the user device <NUM>. In other configurations, the hotword detector <NUM> communicates the detection of the hotword <NUM> to other components of the user device <NUM>. In some implementations in order to efficiently and to effectively detect hotwords <NUM>, the hotword detector <NUM> is trained by a hotword detector model <NUM> with data or examples of speech to learn how to identify whether an utterance <NUM> includes a hotword <NUM>. For example, the hotword detector <NUM> is taught by a machine learning model to identify a hotword <NUM>.

In some examples, a user <NUM> or the user device <NUM> generates a hotword query <NUM> to identify hotwords <NUM> of interest to the user <NUM> and/or the user devices <NUM>. In some implementations, a user device <NUM> communicates with the remote server <NUM> via the network <NUM> to identify and/or to receive hotwords <NUM> from a hotword repository <NUM> in communication with the remote server <NUM>. In some examples, the hotword query <NUM> may include a user identifier that maps to all hotwords <NUM> assigned to user devices <NUM> owned by a user <NUM> associated with the user identifier. Additionally or alternatively, a user device <NUM> may obtain an identifier (e.g., media access control (MAC) identifier) associated with each nearby user device <NUM> and provide the identifier in the query <NUM> to obtain all hotwords <NUM> associated with each identifier from the repository <NUM>. The hotword repository <NUM> may include any combination of hotwords <NUM> assigned to nearby devices <NUM>, a list of hotwords <NUM> associated with one or more devices <NUM> owned and/or controlled by a specific user <NUM>, and/or a list of all potential hotwords <NUM> that may be assigned to any given device <NUM> for initiating the wake-up process (e.g., global hotwords associated with a particular type(s) of device(s) <NUM>). By generating a hotword may receive hotword(s) <NUM> to form a robust hotword training process for the hotword detector <NUM>. Referring to <FIG>, each user device <NUM> is configured to send and/or to receive hotword queries <NUM> to one or more other user devices <NUM> to understand and/or to compile hotword(s) <NUM> that are assigned to the other user devices <NUM>.

Each user device <NUM> may be further configured as a speech synthesis device. As a speech synthesis device, a user device <NUM> may further include a speech synthesizer <NUM>, such as a text-to-speech (TTS) system, that generates synthesized speech <NUM>. For instance, the synthesized speech <NUM> may audibly convey an answer to a query received from the user <NUM>. In some examples, all functionality of the speech synthesizer <NUM> may reside on the user device <NUM>. In other examples, a portion of the speech synthesizer <NUM> resides on the user device <NUM> and the remaining portion of the speech synthesizer <NUM> resides on a distributed environment, e.g., cloud computing environment <NUM>.

In some examples, the speech synthesizer <NUM> on one device <NUM> (e.g., the first user device <NUM>, 110a) is trained on a text-to-speech sequence or audio representation of a hotword <NUM> assigned to the other user device <NUM> (e.g., the second user device 110b). For instance, a training pipeline (e.g., a hotword-aware trainer <NUM>) of the speech synthesizer <NUM> (e.g., a TTS system) associated with one device <NUM> may generate a hotword-aware model <NUM> for use in detecting a presence of hotwords <NUM>. The hotword-aware model <NUM> may be trained on any combination of hotwords <NUM> assigned to nearby devices <NUM>, a list of hotwords <NUM> associated with one or more devices <NUM> owned and/or controlled by a specific user <NUM>, and/or a list of all potential hotwords <NUM> that may be assigned to any given device <NUM> for initiating the wake-up process (e.g., global hotwords associated with a particular type(s) of device(s) <NUM>). Additionally or alternatively, the hotword query <NUM> may be used to obtain the hotword(s) <NUM> for the hotword-aware model <NUM>. The speech synthesizer <NUM> of each user device <NUM> may further include a waveform generator <NUM> for producing synthesized speech <NUM>. The waveform generator <NUM> may use unit selection logic for generating synthesized speech <NUM> in the form of output audio data. In some examples, the waveform generator <NUM> uses a neural network for generating the output audio data. While examples are directed toward using the hotword-aware model <NUM> for detecting a presence of hotwords <NUM> in synthesized speech, the hotword-aware model <NUM> may be similarly trained for detecting hotwords <NUM> in other types of specified audio, such as, without limitation, broadcast audio.

In the example shown, the speech-enabled system <NUM> includes a first user device 110a and a second user device 110b. The second user device 110b may be considered a nearby device of the first user device 110a or vice versa. Here, the user devices 110a, 110b are considered "nearby" of one another when the corresponding audio capture device <NUM> on one device <NUM> is capable of capturing an utterance <NUM> directed to the other device <NUM>. Stated differently, "nearby" user devices 110a, 110b are within overlapping audio receiving proximity such that the speech output device <NUM> of one user device <NUM>, 110a is within detectable range of the audio capturing device <NUM> of a nearby user device <NUM>, 110b. Although the speech-enabled system <NUM> is shown to include two user devices 110a, 110b, in other examples, the speech-enabled system <NUM> includes additional user devices <NUM> without departing from the scope of the present disclosure. Some examples of user devices <NUM> are portable computers, smartphones, tablet-computing devices, smart speakers, smart displays, or wearable computing devices.

In some examples, the user devices <NUM>, 110a-b each correspond to a user <NUM> speaking words or sub-words, over one or more networks <NUM>. For instance, the user <NUM> may say a first utterance 150a detectable by the first user device 110a that includes "Ok Google: Remind me first thing tomorrow morning to Restart Computer at Work. " Here, the phrase "Ok Google" is a hotword <NUM> assigned to the user device 110a to thereby cause the hotword detector <NUM> to trigger the user device 110a to initiate a wake-up process for processing the hotword <NUM> and/or one or more other terms following the hotword <NUM> in the audio input data (e.g., the remainder of the first utterance 150a. , "Remind me first thing tomorrow to Restart Computer at Work"). In this example, the first user device 110a responds to the first utterance 150a. with synthesized speech <NUM> saying, "Ok, Jim. Reminder is set for tomorrow morning.

Similarly, the second user device <NUM>, 110b may be assigned the hotword <NUM> "Start Computer. " In this configuration, the user <NUM> desires the second user device <NUM>, 110b to initiate a wake-up process when the user <NUM> uses the hotword <NUM>, "Start Computer. " Accordingly, when the user <NUM> says a second utterance 150b detectable by the second user device 110b that includes "Start Computer: Play music from <NUM>'s music playlist," the phrase "Start computer" causes the hotword detector <NUM> to trigger the second user device 110b to initiate a wake-up process for processing the hotword <NUM> and/or one or more other terms following the hotword <NUM> in the audio input data - "Play music from <NUM>'s music playlist.

When two user devices <NUM> are nearby, synthesized speech <NUM> including a hotword <NUM> as output data from the first user device 110a may be inadvertently received by the audio capturing device <NUM>, 116b of the second user device 110b. In response to the inadvertently received synthesized speech <NUM> containing the hotword <NUM>, the user <NUM> does not intend for the hotword detector <NUM>, 200b of the second device <NUM>, 110b to wake-up and/or initiate further processing based on the inadvertently received synthesized speech <NUM>. To prevent a hotword detector <NUM> from activating the second user device <NUM>, 110b, the hotword detector <NUM> may be configured to identify synthesized speech <NUM> and ignore synthesized speech <NUM> containing the hotword <NUM>.

<FIG> is an example of the hotword detector <NUM> within a user device <NUM> of the speech-enabled system <NUM>. The hotword detector <NUM> is configured to determine whether audio input data, such as an utterance <NUM>, includes a hotword <NUM> (e.g., based on detecting that some or all of the acoustic features of a sound corresponding to the hotword <NUM> are similar to acoustic features characteristic of a hotword <NUM>). For example, the hotword detector <NUM> determines the utterance <NUM> begins with a hotword <NUM> and subsequently initiates a wake-up process for the user device <NUM> of the hotword detector <NUM>.

In some examples, the hotword detector <NUM> includes a hotword detector trainer <NUM> and a hotword detector model <NUM>. In addition to being trained on positive training samples <NUM>, <NUM>2b containing audio representations of hotwords, the hotword detector trainer <NUM> is trained on negative training examples <NUM>, 212a of synthesized speech <NUM> to generate the hotword detector model <NUM>, and thereby teach the hotword detector <NUM> to discern between human-generated utterances <NUM> (e.g., non-synthesized speech) and synthesized utterances <NUM> (generated by a speech synthesizer <NUM>). The hotword detector model <NUM> is a synthesized speech aware model <NUM> generated by the hotword detector trainer <NUM> based on training examples <NUM>, 212a-b.

In some implementations, the hotword detector trainer <NUM> trains the hotword detector model <NUM> by negative training examples 212a and positive training examples 212b. A negative training example 212a is a sample of audio that the hotword detector trainer <NUM> teaches the hotword detector model <NUM> to ignore. Here, in order to prevent inadvertent wake-up initiation for a user device <NUM> based on synthesized speech <NUM>, the negative training examples 212a are samples of audio corresponding to synthesized speech <NUM>. The synthesized speech <NUM> of one or more negative training exaniple(s) 212a may be synthesized speech <NUM> that includes the hotword <NUM> (i.e. pronounces the hotword <NUM>) or synthesized speech that does not include the hotword <NUM>. In either scenario, the hotword detector <NUM> is taught to disregard synthesized speech <NUM> so that a wake-up process based on utterances <NUM> is not inadvertently initiated by synthesized speech <NUM> containing a hotword or one or more words/sub-words that sound like the hotword <NUM>. By disregarding synthesized speech <NUM>, the hotword detector <NUM> prevents the initiation of the wake-up process on the user device <NUM> for processing the hotword <NUM> and/or the one or more other terms following the hotword <NUM> in the audio input data.

Optionally, the hotword detector trainer <NUM> may additionally or alternatively train the hotword detector model <NUM> by negative training examples 212a that include samples of other types of audio (e.g., broadcast audio). Accordingly, the hotword detector <NUM> may be similarly taught to disregard these other types of audio so that a wake-up process based on utterances <NUM> is not inadvertently initiated by these other types of audio containing a hotword or one or more words/sub-words that sound like the hotword <NUM>.

In contrast, a positive training example 212b is an audio sample of an utterance <NUM> of human speech that includes a hotword <NUM>. The hotword detector trainer <NUM> feeds the hotword detector model <NUM> positive training examples 212b to teach the hotword detector <NUM> examples where the hotword detector <NUM> should initiate the wake-up process. Additionally or alternatively, the hotword detector trainer <NUM> may train the hotword detector model <NUM> with training examples <NUM> that are audio samples of utterances <NUM> of human speech without the hotword <NUM> in order to expose the hotword detector <NUM> to further scenarios that may occur during operation of the hotword detector <NUM>. In some implementations, the more training examples <NUM> taught to the hotword detector model <NUM> by the hotword detector trainer <NUM>, the more robust and/or computationally efficient the hotword detector <NUM> becomes when implementing the hotword detector model <NUM>. Moreover, by training the hotword detector <NUM> with the hotword detector model <NUM> taught by training examples <NUM> from the hotword detector trainer <NUM>, the hotword detector model <NUM> allows detection of the presence of synthesized speech in utterances <NUM> (e.g., audio input data) through an analysis of acoustic features of the utterances <NUM> without transcribing or semantically interpreting the utterances <NUM>.

With continued reference to <FIG>, the hotword detector <NUM> of the user device <NUM> implements the hotword detector model <NUM> to determine whether the received audio input data of "reminder to restart computer when you arrive at work this morning" includes a hotword <NUM>. For example, the first user device <NUM>, 110a generates this audio input data as synthesized speech <NUM>. The second user device <NUM>, 110b, as a nearby user device to the first user device <NUM>, 110a may overhear this synthesized speech <NUM>, for example, at an audio capture device <NUM>, 116b of the second user device <NUM>, 110b. Here, instead of the hotword detector <NUM>, 200b initiating the wake-up process due to the hotword <NUM>, "start computer" as an acoustic feature of the synthesized speech <NUM>, the hotword detector <NUM>, 200b implements the hotword detector model <NUM> to identify the audio input data as synthesized speech <NUM> and thereby ignores the presence of the assigned hotword <NUM> "start computer" contained within the phrase "reminder to restart computer when you arrive at work this morning.

In some configurations, the hotword detector trainer <NUM> is configured to segregate training examples <NUM> into training and evaluation sets (e.g., <NUM>% training and <NUM>% evaluation). With these sets, the hotword detector trainer <NUM> trains the hotword detector model <NUM> with the audio samples until a performance of the hotword detector model <NUM> on the evaluation set stops decreasing. Once the performance stops decreasing on the evaluation set, the hotword detector model <NUM> is ready for modeling where the hotword detector model <NUM> allows the hotword detector <NUM> to accurately detect hotwords <NUM> received at the user device <NUM> that do not correspond to synthesized speech <NUM>.

Additionally or alternatively, the hotword detector model <NUM> is a neural network. The hotword detector model <NUM> may be a convolution neural network (CNN) or a deep neural network (DNN). In some examples, the hotword detector model <NUM> is a combination of a convolution neural network and a deep neural network such that the convolution neural network filters, pools, then flattens information to send to a deep neural network. Much like when the hotword detector model <NUM> is a machine learning model, a neural network is trained (e.g., by the hotword detector trainer <NUM>) to generate meaningful outputs that may be used for accurate hotword detection. In some examples, a mean squared error loss function trains the hotword detector model <NUM> when the hotword detector model <NUM> is a neural network.

<FIG> and <FIG> are examples of the speech synthesizer <NUM> of a user device <NUM>. As an example, the speech synthesizer <NUM> is a TTS system where an input to the speech synthesizer is text input data <NUM>. The speech synthesizer <NUM> may be configured to generate synthesized speech <NUM> from the text input data <NUM> by converting the text input data <NUM> into synthesized speech <NUM>. As illustrated by <FIG> and <FIG>, the speech synthesizer <NUM> can generate synthesized speech <NUM> by different processes, such as a unit selection process (<FIG>) or a neural network process (<FIG>). In either process, the speech synthesizer <NUM> includes a hotword-aware trainer <NUM> along with a hotword-aware model <NUM> to provide an audio output signal <NUM> identifiable by nearby user device(s) <NUM> to prevent an initiation of the wake-up process on the nearby user device <NUM>. In other words, although the audio output signal <NUM> may include a hotword <NUM> that should initiate the wake-up process for a nearby user device <NUM>, the speech synthesizer <NUM> identifies the audio output signal <NUM> as synthesized speech <NUM> to avoid/evade initiating wake-up processes associated with other nearby user devices <NUM>. In these examples, the speech synthesizer <NUM> uses the hotword-aware model <NUM> to detect a presence of a hotword <NUM> assigned to a user device <NUM> and to determine whether a pronunciation (e.g., the audio output signal <NUM> of synthesized speech <NUM>) of the text input data <NUM> includes the hotword <NUM>. When the pronunciation includes the hotword <NUM>, the speech synthesizer <NUM> generates the audio output signal <NUM> such that the pronunciation fails to trigger hotword detector(s) <NUM> from a different user device <NUM>.

In some examples, the hotword-aware trainer <NUM> utilizes the hotword query <NUM> to obtain a hotword <NUM> or a list of hotwords <NUM> (e.g., from a hotword repository <NUM> or directly from nearby user devices <NUM>). As stated previously, the hotword query <NUM> may obtain any combination of hot words <NUM> assigned to nearby devices <NUM>, a list of hotwords <NUM> associated with one or more devices <NUM>, 110a-n owner-controlled by a specific user <NUM>, and/or a list of all potential hotwords <NUM> that may be assigned to any given device <NUM> for initiating the wake-up process. In other examples, the user <NUM> or an administrator of the user devices <NUM> of the speech-enabled system <NUM> preprograms and/or updates the hotword-aware trainer <NUM> with hotword(s) <NUM>. The hotword-aware trainer <NUM> trains the hotword-aware model <NUM> based on received and/or obtained hotwords <NUM>. In some examples, the hotword-aware trainer <NUM> trains the hotword-aware model <NUM> based on a TTS sequence or an audio representation of at least one hotword <NUM>.

The speech synthesizer <NUM> may use the hotword-aware model <NUM> at any point during the speech synthesis process. In some examples, the speech synthesizer <NUM> first generates a text-to-speech output and then uses the hotword-aware model <NUM> to analyze the synthesized speech <NUM> for a hotword <NUM> or sound-alike phrases. In other examples, the speech synthesizer <NUM> uses the hotword-aware model <NUM> to analyze the text-to-speech output for a hotword <NUM> during the generation of the synthesized speech <NUM>.

When the hotword-aware model <NUM> identifies a hotword <NUM> during the speech synthesis process or within the synthesized speech <NUM>, the speech synthesizer <NUM> provides an indication that the synthesized speech <NUM> includes a hotword <NUM> (e.g., within the audio output signal <NUM>). In some examples, the speech synthesizer <NUM> emits a known watermark <NUM> over the audio sequence of the synthesized speech <NUM> that is identifiable to a hotword detector <NUM>. The speech synthesizer <NUM> may insert the watermark <NUM> into or over the synthesized speech <NUM> in any manner that is identifiable to the hotword detector <NUM>. For instance, the speech synthesizer <NUM> may insert a watermark by appending/prepending/overlaying the watermark or encoding the watermark within the synthesized speech <NUM>. The speech synthesizer <NUM> may insert a unique feature, such as the known watermark <NUM>, over the audio sequence in discrete intervals within the synthesized speech <NUM>. These discrete intervals may range anywhere from millisecond intervals to larger intervals spanning several seconds. For example, smaller intervals, such as millisecond intervals, allow even portions of the synthesized speech <NUM> received at a nearby user device <NUM> to be identifiable to prevent unwanted wake-up initiation. Inserting watermarks <NUM> at intervals may further prevent unwanted speech recognition in the event that the user device is active and already awake. In some implementations, the speech synthesizer <NUM> distorts the synthesized speech <NUM> using a filter <NUM> that has been trained against a given hotword detector <NUM>. In other words, the hotword detector <NUM> on the nearby device <NUM> is trained with the filter <NUM> to ignore filtered synthesized speech <NUM>. In some examples, the filter <NUM> obscures the hotword <NUM> within synthesized speech <NUM> to a hotword detector <NUM>. Similarly, the speech synthesizer <NUM> may alter a speech waveform corresponding to an audio output signal <NUM> associated with the synthesized speech <NUM> by removing or altering any sounds associated with a hotword <NUM> in order to evade hotword detection by a hotword detector <NUM>.

Referring to <FIG>, the speech synthesizer <NUM>, 300a uses unit selection logic to generate the synthesized speech <NUM>. Here, the speech synthesizer <NUM> is a TTS system where a unit expander <NUM> receives the text input data <NUM> and parses the text input data <NUM> into components compatible with speech units of a unit database <NUM>. A unit selector <NUM> is configured to interpret the parsed text input data from the unit expander <NUM> and to select speech units that correspond to the parsed text input data from the unit database <NUM> in communication with the unit selector <NUM>. The unit database <NUM> is a database that generally includes a collection of units of parsed text along with these units' corresponding audio signal form (i.e. speech units). The unit selector <NUM> constructs a sequence of units <NUM> from speech units relating to the parsed text input data to form the synthesized speech <NUM> for the text input data <NUM>. In some configurations, when the synthesized speech <NUM> includes a hotword <NUM>, the speech synthesizer <NUM>, 300a is configured to select alternative variations of the speech units to form the synthesized speech <NUM> such that a hotword detector <NUM> would fail to detect the hotword <NUM>.

<FIG> is an example of a speech synthesizer <NUM>, 300b similar to that of <FIG>, except that the speech synthesizer <NUM>, 300b receives text input data <NUM> and generates an input text sequence <NUM> to be input into a waveform neural network model <NUM>. The waveform neural network model <NUM>, unlike the unit selection process, does not require a unit database <NUM>. Without the unit database <NUM>, the waveform neural network model <NUM> may achieve greater computational efficiency and reduce a portion of the computational load when compared to the speech synthesizer <NUM>, 300a.

Similar to the hotword detector model <NUM>, the hotword-aware model <NUM> and/or waveform neural network model <NUM> may be machine learning models that may first undergo model training (e.g.,, in the case of the hotword-aware model <NUM> via the hotword-aware trainer <NUM>) and, once trained, may proceed to be implemented by the speech synthesizer <NUM>. During model training, models <NUM>, <NUM> receive data sets and result sets to predict its own output based on input data similar to the data sets. In the case of the hotword-aware model <NUM>, the data sets and results sets may be audio samples or text samples associated with the hotword <NUM>, such as a phrase, a word, a sub-word, a text-to-speech sequence, a linguistic variation, a language translation, etc. In the case of the waveform neural network model <NUM>, the data sets and results sets may be text samples configured to train the waveform neural network model <NUM> to generate synthesized speech <NUM> from the input text sequence <NUM><NUM>. In some examples, for training purposes, data is segregated into training and evaluation sets (e.g., <NUM>% training and <NUM>% evaluation). With these sets, the models <NUM>, <NUM> train until a performance on the evaluation set stops decreasing. Once the performance stops decreasing on the evaluation set, each respective model <NUM>, <NUM> is ready for modeling (e.g., identifying hotwords <NUM> for the hotword-aware model <NUM> or generating synthesized speech <NUM> for the waveform neural network model <NUM>).

Additionally or alternatively, each respective model <NUM>, <NUM> is a neural network. The model <NUM>, <NUM> may be a convolution neural network (CNN) (e.g., a modified WaveNet) or a deep neural network (DNN). In some examples, the model <NUM>, <NUM> is a combination of a convolution neural network and a deep neural network such that the convolution neural network filters, pools, then flattens information to send to a deep neural network. Much like when the model <NUM>, <NUM> is a machine learning model, a neural network is trained to generate meaningful audio output signals <NUM>. In some examples, a mean squared error loss function trains the model <NUM>, <NUM> when the model <NUM>, <NUM> is a neural network.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of determining a pronunciation of text input data <NUM> includes a hotword <NUM> assigned to a nearby device <NUM>. The data processing hardware <NUM> may execute the operations for the method <NUM> by executing instructions stored on the memory hardware <NUM>. At operation <NUM>, the method <NUM> includes receiving, at the data processing hardware <NUM> of a speech synthesis device <NUM>, text input data <NUM> for conversion into synthesized speech <NUM>. At operation <NUM>, the method <NUM> includes determining, by the data processing hardware <NUM> and using a hotword-aware model <NUM> trained to detect a presence of a hotword <NUM> assigned to a user device <NUM>, whether a pronunciation of the text input data <NUM> includes the hotword <NUM>, the hotword <NUM>, when included in audio input data received by the user device <NUM>, configured to initiate a wake-up process on the user device <NUM> for processing the hotword 130and/or one or more other terms following the hotword <NUM> in the audio input data.

At operation <NUM>, when the pronunciation of the text input data <NUM> includes the hotword <NUM>, the method <NUM> includes generating an audio output signal <NUM> from the input text data <NUM>. At operation <NUM>, when the pronunciation of the text input data <NUM> includes the hotword <NUM>, the method <NUM> includes providing, by the data processing hardware <NUM>, the audio output signal <NUM> to an audio output device <NUM> to output the audio output signal <NUM>, the audio output signal <NUM> when captured by an audio capture device <NUM> of the user device <NUM>, configured to prevent initiation of the wake-up process on the user device <NUM>.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> when the audio input data includes synthesized speech <NUM>, preventing initiation of a wake-up process on a user device <NUM> for processing audio input data when the audio input data includes synthesized speech <NUM>. The data processing hardware <NUM> may execute the operations for the method <NUM> by executing instructions stored on the memory hardware <NUM>. At operation <NUM>, the method <NUM> includes receiving, at a hotword detector <NUM> of the user device <NUM>, audio input data containing a hotword <NUM>. The hotword <NUM> is configured to initiate the wake-up process on the user device <NUM> for processing the hotword <NUM> and/or one or more other terms following the hotword <NUM> in the audio input data At operation <NUM>, the method <NUM> includes determining, by the hotword detector <NUM>, whether the audio input data includes synthesized speech <NUM> using a hotword detector model <NUM> configured to detect the hotword <NUM> in the audio input data and a presence of synthesized speech <NUM>. At operation <NUM>, when the audio input data includes synthesized speech <NUM>, the method <NUM> includes preventing, by the hotword detector <NUM>, initiation of the wake-up process on the user device <NUM> for processing the hotword <NUM> and/or the one or more other terms following the hotword <NUM> in the audio input data.

Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRANI), phase change memory (PCM) as well as disks or tapes.

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto optical disks; and CD ROM and DVD-ROM disks.

Claim 1:
A method (<NUM>) comprising:
receiving, at a hotword detector (<NUM>) of a user device (<NUM>), audio input data containing a hotword (<NUM>), the hotword (<NUM>) configured to initiate a wake-up process on the user device (<NUM>) for processing the hotword (<NUM>) and/or one or more other terms following the hotword (<NUM>) in the audio input data;
determining, by the hotword detector (<NUM>), whether the audio input data comprises synthesized speech (<NUM>) using a hotword detector model (<NUM>) that has been trained to detect the hotword (<NUM>) in the audio input data and a presence of synthesized speech (<NUM>), synthesized speech being machine output from a non-audible originating data input; and
when the audio input data comprises synthesized speech (<NUM>), preventing, by the hotword detector (<NUM>), initiation of the wake-up process on the user device (<NUM>) for processing the hotword (<NUM>) and/or the one or more other terms following the hotword (<NUM>) in the audio input data,
wherein the hotword detector model (<NUM>) is trained on a plurality of training samples comprising:
positive training samples (212b) comprising human-generated audio data corresponding to one or more users speaking the hotword (<NUM>) assigned to the user device (<NUM>); and
negative training samples (212a) comprising synthesized speech utterances (<NUM>) output from one or more speech synthesizer devices (<NUM>).