Patent Description:
A speech-enabled environment (e.g., home, workplace, school, automobile, etc.) allows a user to speak a query or a command out loud to a computer-based system that fields and answers the query and/or performs a function based on the command. The speech-enabled environment can be implemented using a network of connected microphone devices distributed through various rooms or areas of the environment. These devices may use hotwords to help discern when a given utterance is directed at the system, as opposed to an utterance that is directed to another individual present in the environment. Accordingly, the devices may operate in a sleep state or a hibernation state and wake-up only when a detected utterance includes a hotword. Once awake, the devices can proceed to perform more expensive processing such as full on-device automated speech recognition (ASR) or server-based ASR.

<CIT> relates to methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving audio data including an utterance, obtaining context data that indicates one or more expected speech recognition results, determining an expected speech recognition result based on the context data, receiving an intermediate speech recognition result generated by a speech recognition engine, comparing the intermediate speech recognition result to the expected speech recognition result for the audio data based on the context data, determining whether the intermediate speech recognition result corresponds to the expected speech recognition result for the audio data based on the context data, and setting an end of speech condition and providing a final speech recognition result in response to determining the intermediate speech recognition result matches the expected speech recognition result, the final speech recognition result including the one or more expected speech recognition results indicated by the context data.

<CIT> relates to a speech-to-speech translation method. The method comprises the following steps: receiving a speech wake-up instruction, and entering into, according to the speech wake-up instruction, an activation state; collecting speech information, and carrying out translation processing on the speech information; and outputting the translated speech information. According to the speech-to-speech translation method provided by an embodiment of the present invention, a translating machine is activated by means of speech control to carry out translation processing, thereby eliminating the key of the translating machine, and improving the convenience of operation.

<CIT> relates to techniques for managing power consumption for a computing device by one or more keywords. If an audio input obtained by the computing device includes a keyword, a network interface module and/or an application processing module of the computing device may be activated. The audio input may then be transmitted via the network interface module to a remote computing device, such as a speech recognition server. Alternately, the computing device may be provided with a speech recognition engine configured to process the audio input for on-device speech recognition.

One aspect of the disclosure provides a method for detecting freeze words. The method includes receiving, at data processing hardware, audio data that corresponds to an utterance spoken by a user and captured by a user device associated with the user. The method also includes processing, by the data processing hardware, using a speech recognizer, the audio data to determine that the utterance includes a query for a digital assistant to perform an operation. The speech recognizer is configured to trigger endpointing of the utterance after a predetermined duration of non-speech in the audio data. Before the predetermined duration of non-speech in the audio data, the method includes detecting, by the data processing hardware, a freeze word in the audio data. The freeze word follows the query in the utterance spoken by the user and captured by the user device. In response to detecting the freeze word in the audio data, the method includes triggering, by the data processing hardware, a hard microphone closing event at the user device to prevent the user device from capturing any audio subsequent to the freeze word.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the freeze word includes one of a predefined freeze word that includes one or more fixed terms across all users in a given language, a user-selected freeze word that includes one or more terms specified by the user of the user device, or an action-specific freeze word associated with the operation to be performed by the digital assistant. In some examples, detecting the freeze word in the audio data includes: extracting audio features from the audio data; generating, using a freeze word detection model, a freeze word confidence score by processing the extracted audio features; the freeze word detection model executing on the data processing hardware; and determining that the audio data corresponding to the utterance includes the freeze word when the freeze word confidence score satisfies a freeze word confidence threshold.

Detecting the freeze word in the audio data may include recognizing, using the speech recognizer that executes on the data processing hardware, the freeze word in the audio data. Optionally, the method may further include, in response to detecting the freeze word in the audio data: instructing, by the data processing hardware, the speech recognizer to cease any active processing on the audio data; and instructing, by the data processing hardware, the digital assistant to fulfill performance of the operation.

According to the invention, processing the audio data to determine that the utterance includes the query for the digital assistant to perform the operation further includes:
processing, using the speech recognizer, the audio data to generate a speech recognition result for the audio data; and performing semantic interpretation on the speech recognition result for the audio data to determine that the audio data includes the query to perform the operation. In response to detecting the freeze word in the audio data, the method also includes: modifying, by the data processing hardware, the speech recognition result for the audio data by stripping the freeze word from the speech recognition result; and instructing, by the data processing hardware, using the modified speech recognition result, the digital assistant to perform the operation requested by the query.

In some examples, prior to processing the audio data using the speech recognizer, the method further includes: detecting, by the data processing hardware, using a hotword detection model, a hotword in the audio data that precedes the query; and in response to detecting the hotword, triggering, by the data processing hardware, the speech recognizer to process the audio data by performing speech recognition on the hotword and/or one or more terms following the hotword in the audio data. In these examples, the method may also include verifying, by the data processing hardware, a presence of the hotword detected by the hotword detection model based on detecting the freeze word in the audio data. Optionally, detecting the freeze word in the audio data may include executing a freeze word detection model on the data processing hardware that is configured to detect the freeze word in the audio data without performing speech recognition on the audio data. Here, the freeze word detection model and the hotword detection model may each include the same or different neural network-based models.

Another aspect of the disclosure provides a system for detecting freeze words. The system includes data processing hardware and memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include receiving audio data that corresponds to an utterance spoken by a user and captured by a user device associated with the user. The operations also include processing, using a speech recognizer, the audio data to determine that the utterance includes a query for a digital assistant to perform an operation. The speech recognizer is configured to trigger endpointing of the utterance after a predetermined duration of non-speech in the audio data. Before the predetermined duration of non-speech in the audio data, the operations include detecting a freeze word in the audio data. The freeze word follows the query in the utterance spoken by the user and captured by the user device. In response to detecting the freeze word in the audio data, the operations include triggering a hard microphone closing event at the user device prevent the user device from capturing any audio subsequent to the freeze word.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the freeze word includes one of a predefined freeze word that includes one or more fixed terms across all users in a given language, a user-selected freeze word that includes one or more terms specified by the user of the user device, or an action-specific freeze word associated with the operation to be performed by the digital assistant. In some examples, detecting the freeze word in the audio data includes: extracting audio features from the audio data; generating, using a freeze word detection model, a freeze word confidence score by processing the extracted audio features; and determining that the audio data corresponding to the utterance includes the freeze word when the freeze word confidence score satisfies a freeze word confidence threshold. In these examples, the freeze word detection model executes on the data processing hardware.

Detecting the freeze word in the audio data may include recognizing, using the speech recognizer that executes on the data processing hardware, the freeze word in the audio data. Optionally, the operations may further include, in response to detecting the freeze word in the audio data: instructing the speech recognizer to cease any active processing on the audio data; and instructing the digital assistant to fulfill performance of the operation.

According to the invention, processing the audio data to determine that the utterance includes the query for the digital assistant to perform the operation further includes:
processing, using the speech recognizer, the audio data to generate a speech recognition result for the audio data; and performing semantic interpretation on the speech recognition result for the audio data to determine that the audio data includes the query to perform the operation. In response to detecting the freeze word in the audio data, the operations also include: modifying the speech recognition result for the audio data by stripping the freeze word from the speech recognition result; and instructing, using the modified speech recognition result, the digital assistant to perform the operation requested by the query.

In some examples, prior to processing the audio data using the speech recognizer, the operations further include: detecting, using a hotword detection model, a hotword in the audio data that precedes the query; and in response to detecting the hotword, triggering the speech recognizer to process the audio data by performing speech recognition on the hotword and/or one or more terms following the hotword in the audio data. In these examples, the operations also includes verifying a presence of the hotword detected by the hotword detection model based on detecting the freeze word in the audio data. Optionally, detecting the freeze word in the audio data may include executing a freeze word detection model on the data processing hardware that is configured to detect the freeze word in the audio data without performing speech recognition on the audio data. The freeze word detection model and the hotword detection model each include the same or different neural network-based models.

Speech-based interfaces such as digital assistants are becoming increasingly prevalent across a variety of devices including, without limitation, mobile phones and smart speakers/displays that include microphones for capturing speech. The general way of initiating voice interaction with an assistant-enabled device is to speak a fixed phrase, e.g., a hotword, that when detected by the speech-enabled device in streaming audio, triggers the assistant-enabled device to initiate a wake-up process to begin recording and processing subsequent speech to ascertain a query spoken by the user. Thus, the hotword is an important component in the overall digital assistant interface stack as it allows users to wake up their assistant-enabled devices from a low power state so that the assistant-enabled devices can proceed to perform more expensive processing such as full automated speech recognition ASR or server-based ASR.

A query spoken by a user that is directed toward an assistant-enabled device typically falls into two categories of queries: conversational queries and non-conversational queries. Conversational queries refer to standard digital assistant queries that query the digital assistant to perform operations such as "set a timer", "remind me to buy the milk", "how tall is Barack Obama", and so on. On the other hand, non-conversational queries refer to dictation-based queries which are longer-form queries where the user speaks to dictate an email, a message, document, social media post, or some other piece of content. For instance, the user may speak the query "send an email to Aleks saying" and then proceed to speak the contents of the email message that the digital assistant will dictate/transcribe and subsequently send from the user's email client to the recipient's (e.g., Aleks) email client.

ASR systems typically use endpointers to determine when a user has started and finished speaking. Once demarcated, the portion of the audio representing the user's speech can be processed to generate speech recognition results, and in some instances, semantic interpretation can be performed on the speech recognition results to ascertain a query spoken by the user. Endpointers typically evaluate a duration of pauses between words in determining when an utterance begins or ends. For instance, if a user says "what is <long pause> for dinner", the endpointer may segment the speech input at the long pause and cause the ASR system to only process the incomplete phrase "what is" instead of the complete phrase "what is for dinner". If an endpointer designates an incorrect end point for an utterance, the result of processing the utterance can be inaccurate and undesirable. At the same time, while allowing for longer duration of pauses between words in determining when an utterance begins or ends safeguards premature endpointing, a microphone of the user's assistant-enabled device detecting the utterance remains open and may detect sounds not intended for the user device. Additionally, delaying the closing of the microphone consequently delays execution of an action specified by the utterance. For instance, if the utterance spoken by the user was a query for a digital assistant to perform an action to "Call Mom", there will inevitably be a delay in the digital assistant initiating the call while the endpointer is waiting for an endpointing timeout duration to lapse to confirm that the user has likely stopped speaking. In this scenario, the assistant-enabled device may also detect additional unintended audio which may result in performance of an action different than what the user intended, which may result in wastage of computational resources in interpreting and acting on the additional audio detected due to the inability to determine in a timely fashion when the user has likely finished speaking.

To mitigate the drawbacks associated with endpointing timeout durations that are too short, e.g., potentially cutting off speech before the user has finished speaking, and endpointing time out durations that are too long, e.g., increasing the chance of capturing unintended speech and increased latency for executing an action specified by the utterance, implementations herein are directed toward freeze words, that when spoken at the end of an utterance, specify when the user is finished speaking to the assistant-enabled device. To some degree, a "freeze word" corresponds to an inverse of a hotword by permitting a user to manually endpoint an utterance and initiate a hard microphone closing event to end a voice-based conversation or a long-form utterance. That is, while a hotword will trigger an assistant-enabled device to wake-up from a sleep or hibernation state to begin processing speech, a freeze word will perform the inverse by causing all active processing on speech to cease and deactivating microphones on the assistant-enabled device, and thereby transition the assistant-enabled device back to the sleep or hibernation state.

In addition to turning off some or all of the speech processing in progress, once a freeze word is detected, the assistant-enabled device may additionally disable or adapt future processing for some amount of time to effectively make the assistant-enabled device less responsive. For instance, a hotword detection threshold may be temporarily raised to make it harder/less likely for a user to issue a subsequent query for some time window after speaking the freeze word. In this scenario, the raised hotword detection threshold may gradually reduce over time back to a default value. Additionally or alternatively, after detecting a freeze word spoken by a particular user, speech input is disabled for that particular user.

An assistant-enabled device executes an acoustic feature detection model configured to detect the presence of a freeze word in audio data corresponding to an utterance without performing speech recognition or semantic interpretation on the audio data. Here, the acoustic feature detection model may be a neural network-based model trained to detect one or more freeze words. The assistant-enabled device may employ the same or different acoustic feature detection model for detecting the presence of a hotword in the audio data. In the case that a same acoustic feature detection model is used for both hotword detection and freeze word detection, the functionality for only one of hotword detection and freeze word detection may be active at a time. Notably, acoustic feature detection models are capable of running on user devices due to their relatively compact size and lower processing requirements compared to ASR models.

In some configurations, in addition to triggering hard microphone closing events, the detection of a freeze word in audio data verifies a presence of a hotword recently detected in the audio data while an assistant-enabled device was in a sleep or hibernation state. Here, the detected hotword may be associated with a low hotword detection confidence score and the subsequent detection of the freeze word may serve as a proxy for verifying that the hotword is present in the audio data. In these configurations, the audio data may be buffered on the assistant-enabled device while the hotword detection and the freeze word detection is performed, and once the detection of the freeze word is detected in the buffered audio data, the assistant-enabled device may initiate a wake-up process to perform speech recognition on the buffered audio data.

In some additional implementations, freeze word detection leverages an automated speech recognizer that is currently executing on-device or server-side to recognize the presence of the freeze word. The speech recognizer may be biased to recognize one or more particular freeze words.

In some examples, a language model may be leveraged to determine if a freeze word is detected in the audio. In these examples, the language model may allow the assistant-enabled device to identify scenarios where the freeze word is actually part of the user's utterance/query, and not spoken by the user to endpoint the utterance/query. Further, the language model can also allow for of near matches of freeze words where a phrase is similar to a freeze word and the freeze word is unlikely to be part of the user's query/utterance per a language model score.

An assistant-enabled device may be capable of recognizing one or more different types/categories of freeze words such as, without limitation, predefined freeze words, custom freeze words, user-selected freeze words, action-specific freeze words, and query-specific freeze words. A predefined freeze word may include a phrase of one or more fixed terms across all users in a given language. For instance, for conversational queries of "Call Mom right now" and "Tell me the temperature outside, thanks Google" the phrases "right now" and "thanks Google" correspond to freeze words for allowing the user to manually endpoint the respective queries.

A user-selected freeze word may correspond to a freeze word specified by a particular user upfront, e.g., during setup of the digital assistant. For instance, the user may select a freeze word from a list of suggested freeze words. Optionally, the user may specify one or more custom terms for use as a freeze word by typing or speaking the terms. In some scenarios, the user specifies user-selected freeze words to be active for particular types of queries. Here, a user may assign different user-selected freeze words for dictation-based queries than for conversational queries. For example, in the dictation-based query "Hey Google send a message to Aleks saying 'I'll be late for our meeting' The End". In this example, "Hey Google" corresponds to a hotword, the phrase "send a message to Aleks saying" corresponds to a query for the digital assistant to dictate and send a message to a recipient, the phrase "I'll be late for our meeting" corresponds to the contents of the message, and the phrase "The End" includes a user-selected freeze word for manually endpointing the query. Thus, upon detecting the freeze word "The End", the assistant-enabled device will immediately endpoint the utterance and cause a speech recognizer to strip the freeze word from the dictated message before sending to the recipient. Alternatively, the phrase "send a message to Aleks saying" could instead correspond to a query for the digital assistant to facilitate an audio-based communication between the user and recipient in which the contents of the message "I'll be late for our meeting" or simply sent as a voice message to the recipient for audible playback on the recipient's device. Notably, the freeze word "The End" when detected by the assistant-enabled device will immediately endpoint the utterance and strip audio of the freeze word from the voice message before sending to the recipient.

An action-specific freeze word is associated with a particular operation/action specified by a query for a digital assistant to perform. For example, a user speaking the query "Hey Google broadcast I'm home end broadcast" includes the freeze word "end broadcast" specific to broadcast action performed by the digital assistant. In this example, the term "broadcast I'm home" specifies the action of broadcasting an audible notification through one or more speakers to indicate to other individuals that the user is home. The audible notification may include a specific melody or chime that permit individuals hearing the audible notification to ascertain that the user is home. In some implementations, the action-specific freeze word is enabled in parallel with a user-specified freeze word and/or a predefined freeze word.

A query-specific freeze word may be specified as part of a query spoken by a user. For example, the following utterance "Hey Google, dictate the following journal entry until I say I'm done <contents of journal entry> I'm done" includes a dictation-based query for the digital assistant to dictate contents spoken by the user for a journal entry. Additionally, the dictation-based query further specifies a freeze word "I'm Done" before the user starts speaking the contents of the journal entry. Here, the freeze word "I'm Done" specified as part of the dictation-based query instructs an endpointer to wait, or at least extend an endpoint timeout duration, to trigger endpointing until the freeze word "I'm Done" is detected. Extending the endpoint time duration allows for long pauses while the user is speaking the contents of the journal entry that would otherwise trigger endpointing. In some examples, the query-specific freeze word is enabled in parallel with a user-specified freeze word and/or a predefined freeze word.

Referring to <FIG>, in some implementations, an example system <NUM> includes an assistant-enabled device (AED) <NUM> associated with one or more users <NUM> and in communication with a remote system <NUM> via a network <NUM>. The AED <NUM> may correspond to a computing device, such as a mobile phone, computer (laptop or desktop), tablet, smart speaker/display, smart appliance, smart headphones, wearable, vehicle infotainment system, etc., and is equipped with data processing hardware <NUM> and memory hardware <NUM>. The AED <NUM> includes or is in communication with one or more microphones <NUM> for capturing utterances from the respective user <NUM>. The remote system <NUM> may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic computing resources <NUM> (e.g., data processing hardware) and/or storage resources <NUM> (e.g., memory hardware).

The AED <NUM> includes an acoustic feature detector <NUM> configured to detect the presence of hotword(s) <NUM> and/or freeze word(s) <NUM> in streaming audio <NUM> without performing semantic analysis or speech recognition processing on the streaming audio <NUM>. The AED <NUM> also includes an acoustic feature extractor <NUM> which may be implemented as part of the acoustic feature detector <NUM> or a separate component from the acoustic feature detector <NUM>. The acoustic feature extractor <NUM> is configured to extract acoustic features from utterances <NUM>. For instance, the acoustic feature extractor <NUM> may receive streaming audio <NUM> captured by the one or more microphones <NUM> of the AED <NUM> that corresponds to an utterance <NUM> spoken by the user <NUM> and extract acoustic features from audio data <NUM> corresponding to the utterance <NUM>. The acoustic features may include Mel-frequency cepstrum coefficients (MFCCs) or filter bank energies computed over windows of the audio data <NUM> corresponding to the utterance <NUM>.

The acoustic feature detector <NUM> may receive the audio data <NUM> including the acoustic features extracted by the acoustic feature extractor <NUM>, and based on the extracted features, a hotword classifier <NUM> is configured to classify whether the utterance <NUM> includes a particular hotword <NUM> spoken by the user <NUM>. The AED <NUM> may store the extracted acoustic features in a buffer of the memory hardware <NUM> and the hotword classifier <NUM> may use the acoustic features in the buffer to detect whether the audio data <NUM> includes the hotword <NUM>. The hotword classifier <NUM> may also be referred to as a hotword detection model <NUM>. The AED <NUM> may include multiple hotword classifiers <NUM> each trained to detect a different hotword associated with a particular term/phrase. These hotwords may be predefined hotwords and/or custom hotwords assigned by the user <NUM>. In some implementations, the hotword classifier <NUM> includes a trained neural network-based model received from the remote system <NUM> via the network <NUM>.

The acoustic feature detector <NUM> also includes a freeze word classifier <NUM> configured to classify whether the utterance <NUM> includes a freeze word <NUM> spoken by the user <NUM>. The freeze word classifier <NUM> may also be referred to as a freeze word detection model <NUM>. The AED <NUM> may include multiple freeze word classifiers <NUM> each trained to detect a different freeze word associated with a particular term/phrase. As described in the remarks above, the freeze words may include predefined freeze words, user-selected freeze words, action-specific freeze words, and/or query-specific freeze words. As with the hotword classifier <NUM>, the freeze word classifier <NUM> may include a trained neural network-based model received from the remote system <NUM>. In some examples, the freeze word classifier <NUM> and the hotword classifier <NUM> are incorporated into the same neural network-based model. In these examples, respective portions of the neural network model corresponding to the hotword classifier <NUM> and the freeze word classifier <NUM> are never simultaneously active. For example, while the AED <NUM> is in a sleep state, the hotword classifier <NUM> may be active to listen for a hotword <NUM> in streaming audio <NUM> and the freeze word classifier <NUM> may be inactive. Once the hotword <NUM> is detected to trigger the AED <NUM> to wake-up and process subsequent audio, the hotword classifier <NUM> may now be inactive and the freeze word classifier <NUM> may be active to listen for the freeze word <NUM> in the streaming audio <NUM>. The classifiers <NUM>, <NUM> of the acoustic feature detector <NUM> may run on a first processor of the AED <NUM>, such as a digital signal processor (DSP), and/or a second processor of the AED <NUM>, such as an application processor (AP) or CPU), that consumes more power while operating than the first processor.

In some implementations, the hotword classifier <NUM> is configured to identify hotwords that are in the initial portion of the utterance <NUM>. In the example shown, the hotword classifier <NUM> may determine that the utterance <NUM> "Ok Google, broadcast I'm home end broadcast" includes the hotword <NUM> "Ok Google" if the hotword classifier <NUM> detects acoustic features in the audio data <NUM> that are characteristic of the hotword <NUM>. For example, the hotword classifier <NUM> may detect that the utterance <NUM> "Ok Google, broadcast I'm home end broadcast" includes the hotword <NUM> "Ok Google" based on generating MFCCs from the audio data and classifying that the MFCCs include MFCCs that are similar to MFCCs that are characteristic of the hotword "Ok Google" as stored in a model of the hotword classifier <NUM>. As another example, the hotword classifier <NUM> may detect that the utterance <NUM> "Ok Google, broadcast I'm home end broadcast" includes the hotword <NUM> "Ok Google" based on generating mel-scale filterbank energies from the audio data and classifying that the mel-scale filterbank energies include mel-scale filterbank energies that are similar to mel-scale filterbank energies that are characteristic of the hotword "Ok Google" as stored in the model of the hotword classifier <NUM>.

At stage A of the acoustic feature detector <NUM>, when the hotword classifier <NUM> determines that the audio data <NUM> corresponding to the utterance <NUM> includes the hotword <NUM>, the AED <NUM> may trigger a wake-up process to initiate speech recognition on the audio data <NUM> that corresponds to the utterance <NUM>. For example, an automated speech recognition (ASR) engine <NUM> (interchangeably referred to as 'speech recognizer' <NUM>) running on the AED <NUM> may perform speech recognition or semantic interpretation on the audio data that corresponds to the utterance <NUM>. The speech recognizer <NUM> may include an ASR model <NUM>, a natural language understanding (NLU) module <NUM>, and an endpointer <NUM>. The ASR model <NUM> may process the audio data <NUM> to generate a speech recognition result <NUM> and the NLU module <NUM> may perform semantic interpretation on the speech recognition result <NUM> to determine that the audio data <NUM> includes a query <NUM> for the digital assistant <NUM> to perform an operation. In this example, the ASR model <NUM> may process the audio data <NUM> to generate a speech recognition result <NUM> for "broadcast I'm home end broadcast" and the NLU module <NUM> may identify "broadcast I'm home" as the query <NUM> for the digital assistant <NUM> to perform the operation of broadcasting an audible notification for audible output from one or more speakers that indicates to other individuals that the user is home. Alternatively, the query <NUM> may be for the digital assistant <NUM> to broadcast a voice message of the user speaking "I'm home" for audible output from the one or more speakers. The NLU module <NUM> may also be leveraged to determine if the presence of a freeze word detected in the audio data <NUM> was actually part of the query <NUM>, and thus not spoken by the user to endpoint the utterance. Thus, the NLU <NUM> may override the detection of a freeze word in scenarios where the freeze word is in fact part of the utterance. The NLU <NUM> may leverage a language model score in these scenarios.

In some implementations, the speech recognizer <NUM> is located on the remote system <NUM> in addition to, or in lieu, of the AED <NUM>. Upon the hotword classifier <NUM> triggering the AED <NUM> to wake-up responsive to detecting the hotword <NUM> in the utterance <NUM>, the AED <NUM> may transmit the audio data <NUM> corresponding to the utterance <NUM> to the remote system <NUM> via the network <NUM>. The AED <NUM> may transmit the portion of the audio data that includes the hotword <NUM> for the remote system <NUM> to confirm the presence of the hotword <NUM> performing speech recognition via the ASR model <NUM>. Alternatively, the AED <NUM> may transmit only the portion of the audio data <NUM> that corresponds to the portion of the utterance <NUM> after the hotword <NUM> to the remote system <NUM>. The remote system <NUM> executes the ASR model <NUM> to generate the speech recognition result <NUM> for the audio data <NUM>. The remote system <NUM> may also execute the NLU module <NUM> to perform semantic interpretation on the speech recognition result <NUM> to identify the query <NUM> for the digital assistant <NUM> to perform the operation. Alternatively, the remote system <NUM> may transmit the speech recognition result <NUM> to the AED <NUM> and the AED <NUM> may execute the NLU module <NUM> to identify the query <NUM>.

With continued reference to <FIG>, the endpointer <NUM> is configured to trigger endpointing of the utterance after a predetermined duration of non-speech in the audio data <NUM>. Here, the predetermined duration of non-speech may correspond to an endpointing timeout duration in which the endpointer <NUM> will endpoint utterances upon detecting a period of non-speech for at least a predetermined duration. That is, the endpointer <NUM> endpoints an utterance by making a hard microphone closing decision that instructs the one or more microphones <NUM> at the AED <NUM> to close and no longer capture streaming audio <NUM>. The endpointing timeout duration is typically set to a default value that is long enough to prevent prematurely endpointing utterances so that contents of utterances are not cut-off before a user is finished speaking. At the same time, while setting longer endpointing timeout durations allow for longer pauses between words in speech and prevent processing incomplete phrases, a microphone of the assistant-enabled device remains open and may detect sounds not directed toward the assistant-enabled device. Additionally, delaying the closing of the microphone consequently delays execution of the action/operation specified by the query.

While the speech recognizer <NUM> is processing the audio data <NUM> and before the endpointer <NUM> detects the predetermined duration of non-speech in the audio data, the freeze word classifier <NUM> simultaneously runs on the AED <NUM> and detects the freeze word <NUM> "end broadcast" in the audio data <NUM>. Here, the freeze word <NUM> "end broadcast" follows the query <NUM> at the end of the utterance <NUM> spoken by the user <NUM> and corresponds to an action-specific freeze word <NUM>. That is, the freeze word <NUM> "end broadcast" is specific to the action/operation of broadcasting a notification or message through acoustic speakers. In some examples, the NLU <NUM> provides an instruction <NUM> to the acoustic feature detector <NUM> to activate/enable the freeze word <NUM> "end broadcast" responsive to determining that the speech recognition result <NUM> for the audio data <NUM> includes the query <NUM> for the digital assistant <NUM> to perform the operation of broadcasting. In these examples, the acoustic feature detector <NUM> may activate/enable a freeze word detection model that is configured to detect the freeze word <NUM> "end broadcast".

In some implementations, the freeze word classifier <NUM> running on the AED <NUM> is configured to identify freeze words <NUM> that are at the end of the utterance <NUM> without performing speech recognition or semantic interpretation. For instance, in this example, the freeze word classifier <NUM> may determine that the utterance <NUM> "Ok Google, broadcast I'm home end broadcast" includes the freeze word <NUM> "end broadcast" if the freeze word classifier <NUM> detects acoustic features in the audio data <NUM> that are characteristic of the freeze word <NUM>. For example, the freeze word classifier <NUM> may detect that the utterance <NUM> "Ok Google, broadcast I'm home end broadcast" includes the freeze word <NUM> "end broadcast" based on generating MFCCs from the audio data and classifying that the MFCCs include MFCCs that are similar to MFCCs that are characteristic of the freeze word <NUM> "end broadcast" as stored in a model of the freeze word classifier <NUM>. As another example, the freeze word classifier <NUM> may detect that the utterance <NUM> "Ok Google, broadcast I'm home end broadcast" includes the freeze word <NUM> "end broadcast" based on generating mel-scale filterbank energies from the audio data and classifying that the mel-scale filterbank energies include mel-scale filterbank energies that are similar to mel-scale filterbank energies that are characteristic of the hotword "Ok Google" as stored in the model of the hotword classifier <NUM>. The freeze word classifier <NUM> may generate a freeze word confidence score by processing the extracted audio features in the audio data <NUM> and determine that the audio data <NUM> corresponding to the utterance <NUM> includes the freeze word <NUM> when the freeze word confidence score satisfies a freeze word confidence threshold.

At stage B of the acoustic feature detector <NUM>, in response to the freeze word classifier <NUM> detecting the freeze word <NUM> in the audio data <NUM> before the endpointer <NUM> detects the predetermined duration of non-speech in the audio data, the AED <NUM> may trigger a hard microphone closing event <NUM> at the AED <NUM> that prevents the AED <NUM> from capturing any streaming audio <NUM> subsequent to the freeze word <NUM>. For instance, triggering the hard microphone closing event <NUM> may include the AED <NUM> deactivating the one or more microphones <NUM>. Thus, the freeze word <NUM> is spoken by the user <NUM> as a manual cue to indicate when the user <NUM> is finished speaking the query <NUM>, and thereby trigger the hard microphone closing event <NUM> without waiting for the endpointer timeout duration to lapse so that the endpointer <NUM> can endpoint the utterance. Instead, triggering the hard microphone closing event <NUM> responsive to detecting the freeze word <NUM> causes the AED <NUM> to instruct the endpointer <NUM> and/or ASR model <NUM> to immediately endpoint the utterance. Triggering the hard microphone closing event <NUM> also causes the AED <NUM> to instruct the ASR system <NUM> cease any active processing on the audio data and instructs the digital assistant <NUM> to fulfill performance of the operation. As a result, speech recognition accuracy is improved since no subsequent speech or background noise is captured by the microphones <NUM> subsequent to the user speaking the freeze word <NUM>, while at the same time latency is improved, since the utterance <NUM> is manually endpointed to permit the digital assistant <NUM> start fulfilling performance of the operation specified by the query <NUM> without having to wait for the endpointing timeout duration to lapse. In the example shown, the ASR system <NUM> provides an output <NUM> to the digital assistant <NUM> that causes the digital assistant <NUM> to perform the operation specified by the query <NUM>. The output <NUM> may include an instruction to perform the operation.

In some scenarios, the output <NUM> also includes the speech recognition result <NUM> for the audio data <NUM> corresponding to the utterance <NUM>. These scenarios may occur when the query <NUM> identified by the ASR system <NUM> corresponds to a search query in which case the speech recognition result <NUM> for the search query <NUM> is provided as the output <NUM> to a search engine (not shown) to retrieve a search result. For instance, an utterance <NUM> of "Hey Google, tell me the weather for tomorrow now Google" may include the hotword "Hey Google", the conversational search query <NUM> "tell me the weather for tomorrow", and lastly a freeze word "now Google". The ASR system <NUM> may process audio data <NUM> to generate a speech recognition result <NUM> for the utterance <NUM> and perform semantic interpretation on the speech recognition result <NUM> to identify the search query <NUM>. Continuing with this example, responsive to the freeze word classifier <NUM> detecting the freeze word "now Google", the AED <NUM> may trigger the hard microphone closing event <NUM> and the ASR system <NUM> may strip the phrase "now Google" from the end of speech recognition result <NUM> (e.g., transcription <NUM>) and provide speech recognition result <NUM> as a search query to the search engine for retrieving a search result for tomorrow's weather forecast. In this example, the freeze word "now Google" may include a predefined freeze word <NUM> common to all users of a given language, that when spoken while speech recognition is active, manually triggers the hard microphone closing event <NUM>.

In some implementations, the digital assistant <NUM> is enabled for continued conversation in which the microphone <NUM> may be left open to accept follow-up queries from the user after the digital assistant <NUM> outputs a response to a previous query. For instance, using the example above, the digital assistant <NUM> may audibly output the search result for tomorrow's weather forecast as synthesized speech and then instruct the microphone <NUM> to remain open so that the user <NUM> can speak a follow-up query without having to repeat speaking the hotword <NUM> as a prefix to the follow-up query. In this example, if the user <NUM> does not have a follow-up query, the user <NUM> speaking the phrase "Thanks Google" (or other phrase of one or more fixed terms) may serve as a freeze word <NUM> to trigger the hard microphone closing event. Leaving the microphone <NUM> open for a fixed duration to accept a follow-up query that the user <NUM> may or may not speak inevitably requires increased processing since speech processing is active while the microphone <NUM> is open to thereby increase power consumption and/or bandwidth use. Thus, the user <NUM> speaking the freeze word may trigger the hard microphone closing event to prevent the AED <NUM> from capturing unintended speech and provide power and bandwidth savings since the AED <NUM> may transition to low-power sleep or hibernation state.

In some examples, if the user <NUM> speaks the freeze word to close the microphone <NUM> and end continued conversation, the AED <NUM> temporarily raises the hotword detection threshold and/or ignores subsequent speech spoken by the same user <NUM> for some period of time. The AED <NUM> may store a reference speaker embedding for the user <NUM> indicating voice characteristics of the user that can be compared with a verification speaker embedding extracted from an utterance. For instance, the verification speaker embedding can be text-dependent where the embedding is extracted from a spoken hotword and the reference speaker embedding can be extracted from the user <NUM> speaking the same hotword one or more times during enrollment and/or one or more previous interactions with the digital assistant <NUM>. When the verification speaker embedding extracted from a subsequent utterance matches the reference speaker embedding for the user <NUM>, the speech may be ignored if the subsequent utterance was provided shortly after the user <NUM> spoke the freeze word to trigger hard microphone closing.

<FIG> and <FIG> show the ASR engine <NUM> receiving a first instance of audio data 120a (<FIG>) corresponding a dictation-based query <NUM> for the digital assistant <NUM> to dictate audible contents <NUM> and a second instance 120b of the audio data <NUM> (<FIG>) corresponding to an utterance <NUM>, 119b of the audible contents <NUM>. Referring to <FIG>, the AED <NUM> captures a first instance 119a of an utterance <NUM> spoken by the user <NUM> that includes "Hey Google, dictate a message to Aleks until I say I'm done. " In this example, "Hey Google" corresponds to the hotword <NUM>, the phrase "dictate a message to Aleks" corresponds to a dictation-based query <NUM> for the digital assistant <NUM> to dictate a message to Aleks, and the phrase "until I say I'm done" specifies a freeze word for endpointing audible contents <NUM> for the message in which the phrase "I'm done" corresponds to the freeze word <NUM>.

The acoustic feature detector <NUM> receives streaming audio <NUM> captured by the one or more microphones <NUM> of the AED <NUM> that corresponds to the first instance 119a of the utterance <NUM>. The hotword classifier <NUM> determines that the streaming audio <NUM> includes the hotword <NUM>. For example, the hotword classifier <NUM> determines that the streaming audio <NUM> includes the hotword <NUM> "Hey Google. " After the hotword classifier <NUM> determines the streaming audio <NUM> includes the hotword <NUM>, the AED <NUM> triggers the wake-up process to initiate speech recognition on the first instance 120a of audio data <NUM> that corresponds to the first instance 119a of the utterance <NUM>.

The ASR <NUM> receives the first instance 120a of the audio data <NUM> from the acoustic feature detector <NUM>. The ASR model <NUM> may process the first instance 120a of the audio data <NUM> to generate a speech recognition result <NUM>. For example, the ASR model <NUM> receives the first instance 120a of the audio data <NUM> corresponding to the utterance 119a "dictate a message to Aleks until I say I'm done" and generates a corresponding speech recognition result <NUM>. The NLU module <NUM> may receive the speech recognition result <NUM> from the ASR model <NUM> and perform semantic interpretation on the speech recognition result <NUM> to determine that the first instance 120a of the audio data <NUM> includes the dictation-based query <NUM> for the digital assistant <NUM> to dictate the audible contents <NUM> spoken by the user <NUM>. In particular, the semantic interpretation performed by the NLU <NUM> on the speech recognition result <NUM> identifies the phrase "dictate a message to Aleks" as the dictation-based query <NUM> for the digital assistant <NUM> to dictate audible contents <NUM> for a message (e.g., electronic message or email) to a recipient, Aleks. In addition to messages, a dictation-based query <NUM> may be associated with dictating other types of content such as audible contents corresponding to a journal entry or notes to be stored in a document.

In some implementations, the ASR <NUM> further determines that the dictation-based query <NUM> specifies the freeze word <NUM> based on the semantic interpretation performed on the speech recognition result <NUM> for the first instance of the audio data <NUM>. For instance, in the example shown, the NLU <NUM> identifies the phrase "until I say I'm done" as an instruction to set the phrase "I'm done" as a freeze word <NUM> for endpointing the audible contents <NUM> of the message. In some examples, the NLU <NUM> provides an instruction <NUM> to the acoustic feature detector <NUM> to activate/enable the freeze word <NUM> "I'm done" responsive to determining that the speech recognition result <NUM> for the first instance of the audio data 120a specifies the freeze word <NUM>. In these examples, the acoustic feature detector <NUM> may activate/enable the freeze word classifier (e.g., freeze word detection model) <NUM> to detect the freeze word <NUM> "I'm done" in subsequent streaming audio <NUM> captured by the AED <NUM>.

In some examples, the freeze word classifier <NUM> and the hotword classifier <NUM> are never simultaneously active. The dotted line that surrounds the freeze word classifier <NUM> in <FIG> indicates that the freeze word classifier <NUM> is currently inactive while the solid line that surrounds the hotword classifier <NUM> indicates that the hotword classifier <NUM> is active. For example, before the NLU <NUM> sends the instruction <NUM> to the acoustic feature detector <NUM> to activate/enable the freeze word classifier <NUM> for detecting the freeze word <NUM> "I'm done", the hotword classifier <NUM> may be active (e.g., indicated by the solid line) to listen for the hotword <NUM> in streaming audio <NUM> and the freeze word classifier <NUM> may be inactive (e.g., indicated by the dotted line). Once the NLU <NUM> determines that the speech recognition result <NUM> includes the dictation-based query <NUM> and that the dictation-based query <NUM> specifies the freeze word <NUM>, the NLU <NUM> transmits the instruction <NUM> to the acoustic feature detector <NUM> to cause the acoustic feature detector <NUM> to activate the freeze word classifier <NUM> for detecting the freeze word "I'm done" in subsequent streaming audio <NUM> and deactivate the hotword classifier <NUM>.

In the example shown, the freeze word <NUM> "I'm done" corresponds to a query-specific freeze word that is specified as part of the query <NUM> spoken by the user <NUM>. Notably, the dictation-based query <NUM> specifies the freeze word <NUM> "I'm Done" before the user starts speaking the audible contents <NUM> of the message. Here, the freeze word "I'm Done" specified as part of the dictation-based query instructs an endpointer to wait, or at least extend an endpoint timeout duration, to trigger endpointing until the freeze word "I'm Done" is detected. In some implementations, the NLU <NUM> sends an instruction <NUM> to the endpointer <NUM> to increase the endpointing timeout duration. Extending the endpoint time duration allows for long pauses once the user <NUM> begins speaking the audible contents <NUM> of the message that would otherwise trigger endpointing. In some examples, the query-specific freeze word is enabled in parallel with an action-specific freeze word (e.g., "End Message") and/or a user-specified freeze word (e.g., "The End"), and/or a predefined freeze word (e.g., "Thanks Google").

In scenarios when the NLU <NUM> determines that the speech recognition result <NUM> includes the dictation-based query <NUM> but that the dictation-based query <NUM> does not specify the query-specific freeze word <NUM>, the NLU <NUM> would not send an instruction <NUM> to the acoustic feature detector <NUM> to cause the acoustic feature detector <NUM> to activate/enable the freeze word classifier <NUM> to detect any query-specific freeze word since none was specified by the query <NUM>. However, the NLU <NUM> could still send an instruction <NUM> to the acoustic feature detector <NUM> to activate/enable the freeze word classifier <NUM> to detect at least one of an action-specific freeze word, a user-defined freeze word, or a predefined freeze word. Optionally, the acoustic feature detector <NUM> may automatically activate/enable the freeze word classifier <NUM> to detect user-defined and/or predefined freeze words in subsequent streaming audio <NUM> upon detecting the hotword <NUM> in the streaming audio <NUM> corresponding to the first instance 119a of the utterance <NUM>.

Referring now to <FIG>, after the user <NUM> speaks the first instance 119a of the utterance <NUM> that conveys the hotword <NUM> and the dictation-based query <NUM> specifying the query-specific freeze word <NUM>, the user <NUM> subsequently speaks a second instance 119b of the utterance <NUM> to convey the audible contents <NUM> of the message the user <NUM> wants the digital assistant <NUM> to dictate followed by the query-specific freeze word <NUM> indicating that the user <NUM> has completed speaking the audible contents <NUM> of the message. Notably, the user <NUM> does not need to prefix the second instance 119b of the utterance <NUM> with the hotword <NUM> since the AED <NUM> is now awake and the ASR <NUM> remains active responsive to the hotword classifier <NUM> detecting the hotword <NUM> "Hey Google" in the first instance 119a of the utterance <NUM>. In the example shown, the second instance 119b of the utterance <NUM> includes "Aleks, I'm running late I'm done. " In this example, the phrase "Aleks, I'm running late" corresponds to the audible contents <NUM> of the message and the phrase "I'm done" corresponds to the query-specific freeze word <NUM> specified by the dictation-based query <NUM> in the first instance 119a of the utterance <NUM> spoken by the user <NUM> in <FIG>. In lieu of the query-specific freeze word "I'm done" following the audible contents <NUM>, other types of freeze words <NUM> could follow the audible contents <NUM> to similarly trigger endpointing of the audible contents <NUM>.

The acoustic feature detector <NUM> executing on the AED <NUM> receives streaming audio <NUM> captured by the one or more microphones <NUM> of the AED that corresponds to the second instance 119b of the utterance <NUM>. The hotword classifier <NUM> is now inactive (e.g., as indicated by the dotted lines) and the freeze word classifier <NUM> is now active (e.g., as indicated by the solid line) responsive to the acoustic feature detector <NUM> receiving the instruction <NUM> from the NLU <NUM> in <FIG> for activating/enabling the freeze word classifier <NUM> to listen for the presence of the query-specific freeze word <NUM> in the streaming audio <NUM>. The acoustic feature detector <NUM> utilizes the freeze word classifier <NUM> to determine whether the streaming audio <NUM> includes the freeze word <NUM>. The acoustic feature detector <NUM> transmits a second instance 120b of the audio data <NUM> to the ASR <NUM>. The ASR <NUM> receives the second instance 120b of the audio data <NUM> that corresponds to the second instance 119b of the utterance <NUM> of the audible contents <NUM> spoken by the user <NUM> and captured by the AED <NUM>. Moreover, the endpointer <NUM> is applying the extended endpointing timeout duration responsive to receiving the instruction <NUM> from the NLU <NUM> based on the dictation-based query <NUM> specifying the freeze word <NUM> "I'm Done" in the first instance 119a of the utterance <NUM> of <FIG>.

The ASR <NUM> processes the second instance 120b of the audio data <NUM> to generate a transcription <NUM> of the audible contents <NUM>. For example, the ASR <NUM> generates a transcription <NUM> of the audible contents <NUM>"Aleks, I'm running late". During the processing of the second instance 120b of the audio data <NUM> at the ASR <NUM>, the acoustic feature detector <NUM> detects the freeze word <NUM> in the second instance 120b of the audio data <NUM>. Specifically, the freeze word classifier (e.g., freeze word detection model) <NUM> detects the presence of the freeze word <NUM> in the second instance 120b of the audio data <NUM>. In the example shown, the freeze word <NUM> includes the query-specific freeze word "I'm done" to indicate the end of the audible contents <NUM>. The freeze word <NUM> follows the audible contents <NUM> in the second instance 119a of the utterance <NUM> spoken by the user <NUM>.

In response to detecting the freeze word <NUM> in the second instance 120b of the audio data <NUM>, the ASR <NUM> provides the transcription <NUM> of the audible contents <NUM> spoken by the user <NUM> for output from the AED <NUM>. The AED <NUM> may output the transcription <NUM> by transmitting the transcription <NUM> to a recipient device (not shown) associated with the recipient, Aleks. In scenarios when the transcription <NUM> dictates audible contents <NUM> related to notes or a journal entry, the AED <NUM> may provide the transcription <NUM> for output by storing the transcription <NUM> in a document or sending the transcription <NUM> to an associated application. Further, the AED <NUM> may output the transcription <NUM> by displaying the transcription on a graphical user interface (if available) of the AED. Here, the user <NUM> can view the transcription <NUM> before sending to the recipient device in case the user <NUM> wants to re-dictate the transcription <NUM>, fix any mis-transcribed terms, and/or change any of the contents of the message. Additionally or alternatively, the AED <NUM> may employ a text-to-speech (TTS) module to convert the transcription <NUM> to synthesized speech for audible playback to the user <NUM> so that the user <NUM> can confirm that the user <NUM> wants the transcription <NUM> sent to the recipient device. In configurations when the ASR <NUM> executes server-side on the remote system <NUM> (<FIG>), the ASR <NUM> may transmit the transcription <NUM> to the AED <NUM> and/or transmit the transcription <NUM> to the recipient device. That is the ASR <NUM> provides the transcription <NUM> "Aleks, I'm running late" that corresponds to the audible contents <NUM> of the second instance 120b of the audio data <NUM>.

In some examples, in response to detecting the freeze word <NUM> in the second instance 120b of the audio data <NUM>, the acoustic feature detector <NUM> initiates/triggers a hard microphone closing event <NUM> at the AED <NUM>. The hard microphone closing event <NUM> prevents the AED <NUM> from capturing any audio subsequent to the freeze word <NUM>. That is, triggering the hard microphone closing event <NUM> at the AED <NUM> may include the AED <NUM> deactivating the one or more microphones <NUM>. Thus, the freeze word <NUM> is spoken by the user <NUM> as a manual cue to indicate when the user <NUM> is finished speaking the audible contents <NUM> for the dictation-based query <NUM>, and thereby trigger the hard microphone closing event <NUM> without waiting for the endpointer timeout duration to lapse so that the endpointer <NUM> can endpoint the second instance 119b of the utterance <NUM> immediately. Instead, triggering the hard microphone closing event <NUM> responsive to detecting the freeze word <NUM> causes the AED <NUM> to instruct the endpointer <NUM> and/or ASR model <NUM> to immediately endpoint the utterance. Triggering the hard microphone closing event <NUM> also causes the AED <NUM> to instruct the ASR system <NUM> to cease any active processing on the second instance 120b of the audio data <NUM> and instructs the digital assistant <NUM> to fulfill performance of the operation.

In some additional implementations, the ASR system <NUM> detects the presence of the freeze word <NUM> in the second instance 120b of the audio data <NUM> in addition to, or in lieu of, the freeze word classifier <NUM> of the acoustic feature detector <NUM>. That is, since the ASR <NUM> is already active processing the second instance 120b of the audio data 120b responsive to the hotword classifier <NUM> detecting the hotword <NUM> in the first instance 120a of the audio data <NUM>, the ASR <NUM> is capable of recognizing the presence of the freeze word <NUM> in the second instance 120b of the audio data <NUM>. Accordingly, the ASR <NUM> may be configured to initiate the hard microphone closing event <NUM> at the AED <NUM>, cease active processing on the second instance 120b of the audio data <NUM>, and strip the recognized freeze word <NUM> from the end of the transcription <NUM>. Expanding on this capability of the ASR system <NUM> further, the freeze word classifier <NUM> may run on the AED <NUM> as a first-stage freeze word detector and the ASR system <NUM> may be employed as a second-stage freeze word detector to confirm the presence of a freeze word detected in audio data by the freeze word classifier <NUM>.

In some implementations, while processing the second instance 120b of the audio data <NUM> to generate a transcription <NUM> of the audible contents <NUM>, the ASR <NUM> also transcribes the freeze word <NUM> for inclusion in the transcription <NUM>. For example, the transcription <NUM> of the audible contents <NUM> may include "Aleks, I'm running late I'm done. " Here, the transcription <NUM> of the audible contents <NUM> inadvertently includes the freeze word <NUM> "I'm done" as a portion of the audible contents <NUM> of the message. That is, the user <NUM> did not intend for the digital assistant <NUM> to dictate the freeze word <NUM> as part of the audible contents <NUM> to be included in the transcription <NUM>, but rather was spoken to designate an end of the audible contents <NUM>. Accordingly, the hard microphone closing event <NUM> initiated at the AED <NUM> responsive to detecting the freeze word <NUM> may cause the ASR <NUM> to strip the freeze word <NUM> from the end of the transcription <NUM> prior to providing the transcription <NUM> of the audible contents <NUM> for output from the AED <NUM>. Additionally or alternatively, the ASR <NUM> may recognize the presence of the freeze word <NUM> at the end of the transcription <NUM> and accordingly strip the freeze word <NUM> from the end of the transcription <NUM>. In the example shown, the ASR <NUM> strips the freeze word <NUM> "I'm done" from the transcription <NUM> "Aleks, I'm running late I'm done" prior to providing the transcription <NUM> for output <NUM>. Thus, after the ASR <NUM> strips the freeze word <NUM> from the transcription <NUM>, the ASR <NUM> provides the transcription <NUM> "Aleks, I'm running late" for output <NUM> from the AED <NUM>.

<FIG> is a flowchart of an exemplary arrangement of operations for a method <NUM> for detecting freeze words. The method <NUM>, at operation <NUM>, includes receiving, at data processing hardware <NUM>, audio data <NUM> that corresponds to an utterance <NUM> spoken by a user <NUM> and captured by a user device <NUM> associated with the user <NUM>. At operation <NUM>, the method <NUM> includes processing, by the data processing hardware <NUM>, using a speech recognizer <NUM>, the audio data <NUM> to determine that the utterance <NUM> includes a query <NUM> for a digital assistant <NUM> to perform an operation. The speech recognizer <NUM> is configured to trigger endpointing of the utterance <NUM> after a predetermined duration of non-speech in the audio data <NUM>.

At operation <NUM>, before the predetermined duration of non-speech in the audio data <NUM>, the method <NUM> includes detecting, by the data processing hardware <NUM>, a freeze word <NUM> in the audio data <NUM>. The freeze word <NUM> follows the query <NUM> in the utterance <NUM> spoken by the user <NUM> and captured by the user device <NUM>. At operation <NUM>, in response to detecting the freeze word <NUM> in the audio data <NUM>, the method <NUM> includes triggering, by the data processing hardware <NUM>, a hard microphone closing event <NUM> at the user device <NUM>. The hard microphone closing event <NUM> prevents the user device <NUM> from capturing any audio subsequent to the freeze word <NUM>.

<FIG> is a flowchart of an exemplary arrangement of operations for a method <NUM> for detecting freeze words, not forming part of the invention. At operation <NUM>, the method <NUM> includes receiving, at data processing hardware <NUM>, a first instance 119a of audio data <NUM> that corresponds to a dictation-based query <NUM> for a digital assistant <NUM> to dictate audible contents <NUM> spoken by a user <NUM>. The dictation-based query <NUM> is spoken by the user <NUM> and captured by an assistant-enabled device (AED) <NUM> associated with the user <NUM>. At operation <NUM>, the method <NUM> includes receiving, at the data processing hardware <NUM>, a second instance 120b of the audio data <NUM> that corresponds to an utterance <NUM> of the audible contents <NUM> spoken by the user <NUM> and captured by the assistant-enabled device <NUM>. At operation <NUM>, the method <NUM> includes processing, by the data processing hardware <NUM>, using a speech recognizer <NUM>, the second instance 120b of the audio data <NUM> to generate a transcription <NUM> of the audible contents <NUM>.

At operation <NUM>, during the processing of the second instance 120b of the audio data <NUM>, the method <NUM> includes detecting, by the data processing hardware <NUM>, a freeze word <NUM> in the second instance 120b of the audio data <NUM>. The freeze word <NUM> follows the audible contents <NUM> in the utterance <NUM> spoken by the user <NUM> and captured by the assistant-enabled device <NUM>. At operation <NUM>, in response to detecting the freeze word <NUM> in the second instance 120b of the audio data <NUM>, the method <NUM> includes providing, by the data processing hardware <NUM>, for output <NUM> from the assistant-enabled device <NUM>, the transcription <NUM> of the audible contents <NUM> spoken by the user <NUM>.

The processor <NUM> may include the data processing hardware <NUM>, <NUM> of the user device <NUM> or the remote system <NUM>. The data processing hardware <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display <NUM> coupled to high speed interface <NUM>. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

The memory <NUM> can include the memory hardware <NUM>, <NUM> of the user device <NUM> or the remote system <NUM>. The memory hardware <NUM> may be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s).

Claim 1:
A method (<NUM>) comprising:
receiving, at data processing hardware (<NUM>), audio data (<NUM>) corresponding to an utterance (<NUM>) spoken by a user and captured by a user device (<NUM>) associated with the user;
processing, by the data processing hardware (<NUM>), using a speech recognizer (<NUM>), the audio data (<NUM>) to determine that the utterance (<NUM>) includes a query (<NUM>) for a digital assistant (<NUM>) to perform an operation, wherein the speech recognizer (<NUM>) is configured to trigger endpointing of the utterance (<NUM>) after a predetermined duration of non-speech in the audio data (<NUM>), wherein processing the audio data (<NUM>) to determine that the utterance (<NUM>) includes the query (<NUM>) for the digital assistant (<NUM>) to perform the operation comprises:
processing, using the speech recognizer (<NUM>), the audio data (<NUM>) to generate a speech recognition result (<NUM>) for the audio data (<NUM>); and
performing semantic interpretation on the speech recognition result (<NUM>) for the audio data (<NUM>) to determine that the audio data (<NUM>) includes the query (<NUM>) to perform the operation; and
before the predetermined duration of non-speech in the audio data (<NUM>):
detecting, by the data processing hardware (<NUM>), a freeze word (<NUM>) in the audio data (<NUM>), the freeze word (<NUM>) following the query (<NUM>) in the utterance (<NUM>) spoken by the user and captured by the user device (<NUM>); and
in response to detecting the freeze word (<NUM>) in the audio data (<NUM>):
triggering, by the data processing hardware (<NUM>), a microphone closing event (<NUM>) at the user device (<NUM>) to prevent the user device (<NUM>) from capturing any audio subsequent to the freeze word (<NUM>);
modifying, by the data processing hardware (<NUM>), the speech recognition result (<NUM>) for the audio data (<NUM>) by stripping the freeze word (<NUM>) from the speech recognition result (<NUM>); and
instructing, by the data processing hardware (<NUM>), using the modified speech recognition result (<NUM>), the digital assistant (<NUM>) to perform the operation requested by the query (<NUM>).