Patent Description:
A speech-enabled environment permits a user to only speak a query or command out loud and a digital assistant will field and answer the query and/or cause the command to be performed. A speech-enabled environment (e.g., home, workplace, school, etc.) can be implemented using a network of connected microphone devices distributed throughout various rooms and/or areas of the environment. Through such a network of microphones, a user has the power to orally query the digital assistant from essentially anywhere in the environment without the need to have a computer or other device in front of him/her or even nearby. For example, while cooking in the kitchen, a user might ask the digital assistant "please set a timer for <NUM>-minutes" and, in response, the digital assistant will confirm that the timer has been set (e.g., in a form of a synthesized voice output) and then alert (e.g., in the form of an alarm or other audible alert from an acoustic speaker) the user once the timer lapses after <NUM>-minutes. Often, there are many users in a given environment that query/command the digital assistant to perform various actions. These users can register a voice profile with the digital assistant so that the users can be identified to provide a more personalized experience for each user. For instance, one of multiple different users of the digital assistant might speak the command "please play my music playlist", and in response, the digital assistant can identify the particular user by matching voice characteristics with the voice profile for that user and then stream the music playlist for that particular user through an acoustic speaker.

<CIT> discloses techniques for enabling the use of "dynamic" or "context-specific" hot words to invoke an automated assistant. An automated assistant may be executed in a default listening state at least in part on a user's computing device(s). While in the default listening state, audio data captured by microphone(s) may be monitored for default hot words. Detection of the default hot word(s) transitions the automated assistant into a speech recognition state. Sensor signal(s) generated by hardware sensor(s) integral with the computing device(s) may be detected and analyzed to determine an attribute of the user. Based on the analysis, the automated assistant may transition into an enhanced listening state in which the audio data may be monitored for enhanced hot word(s). Detection of enhanced hot word(s) triggers the automated assistant to perform a responsive action without requiring detection of default hot word(s).

<CIT> discloses methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for contextual hotwords. In one aspect, a method, during a boot process of a computing device, includes the actions of determining, by a computing device, a context associated with the computing device. The actions further include, based on the context associated with the computing device, determining a hotword. The actions further include, after determining the hotword, receiving audio data that corresponds to an utterance. The actions further include determining that the audio data includes the hotword. The actions further include, in response to determining that the audio data includes the hotword, performing an operation associated with the hotword.

Preferred embodiments of the present invention are set out in the appended dependent claims.

One aspect of the disclosure provides a method for activating speaker-dependent warm words, as set out in independent claim <NUM>.

The method includes receiving, at data processing hardware, audio data corresponding to an utterance spoken by a user and captured by an assistant-enabled device associated with the user. The utterance includes a command for a digital assistant to perform a long-standing operation, wherein a long-standing operation is an operation performed by the digital assistant for an extended duration of time.

After receiving the audio data corresponding to the utterance, the method includes activating, by the data processing hardware, a set of one or more warm words each associated with a respective action for controlling the long-standing operation, and associating, by the data processing hardware, the activated set of one or more warm words with only the user that spoke the utterance. While the digital assistant is performing the long-standing operation, the method also includes: receiving, at the data processing hardware, additional audio data corresponding to an additional utterance captured by the assistant-enabled device; identifying, by the data processing hardware, in the additional audio data, one of the warm words from the activated set of one or more warm words; and performing, by the data processing hardware, speaker verification on the additional audio data to determine whether the additional utterance was spoken by the same user that is associated with the activated set of one or more warm words. The method further includes when it is determined by the speaker verification that the additional utterance was spoken by the same user that is associated with the activated set of one or more warm words, performing by the data processing hardware, the respective action associated with the identified one of the warm words for controlling the long-standing operation when the additional utterance is spoken by the same user that is associated with the activated set of one or more warm words.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, activating the set of one or more warm words includes activating, for each corresponding warm word in the activated set of one or more warm words, a respective warm word model to run on the assistant-enabled device associated with the user. In these implementations, identifying, in the additional audio data, the one of the warm words from the activated set of one or more warm words includes detecting, using the respective warm word model activated for the corresponding one of the warm words, the one of the warm words in the additional audio data without performing speech recognition on the additional audio data. Detecting the one of the warm words in the additional audio data may include: extracting audio features of the additional audio data; generating, using the respective warm word model activated for the corresponding one of the warm words, a warm word confidence score by processing the extracted audio features; and determining that the additional audio data corresponding to the additional utterance includes the corresponding one of the warm words when the warm word confidence score satisfies a warm word confidence threshold.

In some examples, activating the set of one or more warm words includes executing a speech recognizer on the assistant-enabled device. The speech recognizer is biased to recognize the one or more warm words in the activated set of one or more warm words. In these examples, identifying, in the additional audio data, the one of the warm words from the activated set of one or more warm words includes recognizing, using the speech recognizer executing on the assistant-enabled device, the one of the warm words in the additional audio data.

In some implementations, after receiving the audio data corresponding to the utterance spoken by the user, the method also includes performing, by the data processing hardware, speaker identification on the audio data to identify the user that spoke the utterance. The speaker identification includes extracting, from the audio data corresponding to the utterance spoken by the user, a first speaker-discriminative vector representing characteristics of the utterance spoken by the user, and determining whether the extracted speaker-discriminative vector matches any enrolled speaker vectors stored on the assistant-enabled device. Each enrolled speaker vector is associated with a different respective enrolled user of the assistant-enabled device. When the first speaker-discriminative vector matches one of the enrolled speaker vectors, the method includes identifying the user that spoke the utterance as the respective enrolled user associated with the one of the enrolled speaker vectors that matches the extracted speaker-discriminative vector. In these implementations, the utterance spoken by the user further includes a hotword preceding the command for the digital assistant to perform the long-standing operation; the first speaker-discriminative vector includes a text-dependent speaker-discriminative vector extracted from the portion of the audio data that includes the hotword; and each enrolled speaker vector includes a text-dependent enrolled speaker vector extracted from one or more audio samples of the respective enrolled user speaking the hotword. In additional implementations, performing the speaker verification on the additional audio data includes extracting, from the additional audio data corresponding to the additional utterance of the one of the warm words, a second speaker-discriminative vector representing characteristics of the additional utterance, and determining whether the second extracted speaker-discriminative vector matches a reference speaker vector for the respective enrolled user identified as the user that spoke the utterance. Here, when the extracted second speaker-discriminative vector matches the reference speaker vector, the method includes determining that the additional utterance was spoken by the same user that is associated with the activated set of one or more warm words. In these implementations, the reference speaker vector may include the enrolled speaker vector associated with the respective enrolled user. The reference speaker vector additionally or alternatively includes a text-dependent speaker vector extracted from one or more audio samples of the respective enrolled user speaking the identified one of the warm words.

In some examples, when the first speaker-discriminative vector does not match any of the enrolled speaker vectors, identifying the user that spoke the utterance as a guest user of the assistant-enabled device and performing the speaker verification on the additional audio data includes extracting, from the additional audio data, a second speaker-discriminative vector representing characteristics of the additional utterance and determining whether the second speaker-discriminative vector matches the first speaker-discriminative vector representing the characteristics. In these examples, the method includes determining that the additional utterance was spoken by the same user that is associated with the activated set of one or more warm words when the first and second extracted speaker-discriminative vectors match. In additional implementations, when the additional utterance was spoken by a different user than the user that is associated with the activated set of one or more warm words, the method includes suppressing, by the data processing hardware, performance of the respective action associated with the identified one of the warm words for controlling the long-standing operation.

In some examples, when the additional utterance was spoken by a different user than the user that is associated with the activated set of one or more warm words, the method includes prompting, by the data processing hardware, the user that is associated with the activated set of one or more warm words to authorize performance of the respective action associated with the identified one of the warm words for controlling the long-standing operation. The method also includes receiving, at the data processing hardware, an acknowledgement from the user authorizing performance of the respective action and performing, by the data processing hardware, the respective action associated with the identified one of the warm words for controlling the long-standing operation. Prompting the user may include identifying the different user that spoke the additional utterance. In some implementations, the method further includes determining, by the data processing hardware, when the digital assistant stops performing the long-standing operation and deactivating, by the data processing hardware, the set of one or more warm words.

Another aspect of the disclosure provides a system for activating speaker-dependent warm words, as set out in independent claim <NUM>.

A user's manner of interacting with an assistant-enabled device is designed to be primarily, if not exclusively, by means of voice input. Consequently, the assistant-enabled device must have some way of discerning when any given utterance in a surrounding environment is directed toward the device as opposed to being directed to an individual in the environment or originating from a non-human source (e.g., a television or music player). One way to accomplish this is to use a hotword, which by agreement among the users in the environment, is reserved as a predetermined word(s) that is spoken to invoke the attention of the device. In an example environment, the hotword used to invoke the assistant's attention are the words "OK computer. " Consequently, each time the words "OK computer" are spoken, it is picked up by a microphone, conveyed to a hotword detector, which performs speech understanding techniques to determine whether the hotword was spoken and, if so, awaits an ensuing command or query. Accordingly, utterances directed at an assistant-enabled device take the general form [HOTWORD] [QUERY], where "HOTWORD" in this example is "OK computer" and "QUERY" can be any question, command, declaration, or other request that can be speech recognized, parsed and acted on by the system, either alone or in conjunction with the server via the network.

In cases where the user provides several hotword based commands to an assistant-enabled device, such as a mobile phone or smart speaker, the user's interaction with the phone or speaker may become awkward. The user may speak, "Ok computer, play my homework playlist. " The phone or speaker may begin to play the first song on the playlist. The user may wish to advance to the next song and speak, "Ok computer, next. " To advance to yet another song, the user may speak, "Ok computer, next," again. To alleviate the need to keep repeating the hotword before speaking a command, the assistant-enabled device may be configured to recognize/detect a narrow set of hotphrases or warm words to directly trigger respective actions. In the example, the warm word "next" serves the dual purpose of a hotword and a command so that the user can simply utter "next" to invoke the assistant-enabled device to trigger performance of the respective action instead of uttering "Ok computer, next.

A set of warm words can be active for controlling a long-standing operation. As used herein, a long-standing operation refers to an application or event that a digital assistant performs for an extended duration and one that can be controlled by the user while the application or event is in progress. For instance, when a digital assistant sets a timer for <NUM>-minutes, the timer is a long-standing operation from the time of setting the timer until the timer ends or a resulting alert is acknowledged after the timer ends. In this instance, warm word such as "stop timer" could be active to allow the user to stop the timer by simply speaking "stop timer" without first speaking the hotword. Likewise, a command instructing the digital assistant to play music from a streaming music service is a long-standing operation while the digital assistant is streaming music from the streaming music service through a playback device. In this instance, an active set of warm words can be "pause", "pause music", "volume up", "volume down", "next", "previous", etc., for controlling playback of the music the digital assistant is streaming through the playback device. The long-standing operation may include multi-step dialogue queries such as "book a restaurant" in which different sets of warm words will be active depending on a given stage of the multi-step dialogue. For instance, the digital assistant <NUM> may prompt a user to select from a list of restaurants, whereby a set of warm words may become active that each include a respective identifier (e.g., restaurant name or number in list) for selecting a restaurant from the list and complete the action of booking a reservation for that restaurant.

One challenge with warm words is limiting the number of words/phrases that are simultaneously active so that quality and efficiency is not degraded. For instance, a number of false positives indicating when the assistant-enabled device incorrectly detected/recognized one of the active words greatly increases the larger the number of warm words that are simultaneously active. Moreover a user that seeds a command to initiate a long-running operation cannot prevent others from speaking active warm words for controlling the long-running operation.

Implementations herein are directed toward activating a set of one or more warm words associated with a long-standing operation in progress that are speaker-dependent on a user that spoke a command for a digital assistant to perform the long-standing operations. That is, the warm words that are active are associated with a high likelihood of being spoken by the user after the initial command for controlling the long-standing operation. As such, while digital assistant is performing the long-standing operations commanded by the user, only the user is authorized to speak any of the active warm words to trigger a respective action for controlling the long-standing operation. That is, if someone other than the user speaks one of the warm words, performance of the respective action for controlling the long-stranding operation will be suppressed, or require approval by the user. Stated differently, any processing of warm words captured in streaming audio is limited to the voice of the user that issued the initial command for the digital assistant to perform the long-standing operation. A warm word detector and speaker identification may run on the assistant-enabled device and consume low power.

By associating active warm words with a voice of a particular user so that the warm words are speaker dependent, accuracy in triggering respective actions upon detecting the warm words is improved since only the particular user is permitted to speak the warm words. Additionally, processing costs are improved since a number of times the assistant-enabled device wakes-up and potentially connects to a server reduces, as well as the number of false positives are reduced. Moreover, a user's experience with the digital assistant improves since the user's command to initiate performance of the long-standing operation cannot be overridden by others, unless explicit permission by the user is provided.

<FIG> and <FIG> illustrate example systems <NUM>, 100a, 100b for activating warm words associated with respective actions for controlling a long-standing operation and associating the warm words <NUM> with a user <NUM> that spoke an initial command for controlling the long-standing operation. Briefly, and as described in more detail below, an assistant-enabled device <NUM> begins to play music <NUM> in response to an utterance <NUM>, "Ok computer, play music," spoken by the user <NUM>. While the assistant-enabled device <NUM> is performing the long-standing operation of the music <NUM> as playback audio from a speaker <NUM>, the assistant-enabled device <NUM> is able to detect/recognize an active warm word <NUM> of "next" that is spoken by the user <NUM> as an action to control the long-standing operation, e.g., an instruction to advance to the next song in a playlist associated with the music <NUM>.

The systems 100a, 100b include the assistant-enabled device (AED) <NUM> executing a digital assistant <NUM> that the user <NUM> may interact with through speech. In the example shown, the AED <NUM> corresponds to a smart speaker. However, the AED <NUM> can include other computing devices, such as, without limitation, a smart phone, tablet, smart display, desktop/laptop, smart watch, smart appliance, headphones, or vehicle infotainment device. The AED <NUM> includes data processing hardware <NUM> and memory hardware <NUM> storing instructions that when executed on the data processing hardware <NUM> cause the data processing hardware <NUM> to perform operations. The AED <NUM> includes an array of one or more microphones <NUM> configured to capture acoustic sounds such as speech directed toward the AED <NUM>. The AED <NUM> may also include, or be in communication with, an audio output device (e.g., speaker) <NUM> that may output audio such as music <NUM> and/or synthesized speech from the digital assistant <NUM>.

<FIG> shows the user <NUM> speaking an utterance <NUM>, "Ok computer, play music" in the vicinity of the AED <NUM>. The microphone <NUM> of the AED <NUM> receives the utterance <NUM> and processes the audio data <NUM> that corresponds to the utterance <NUM>. The initial processing of the audio data <NUM> may involve filtering the audio data <NUM> and converting the audio data <NUM> from an analog signal to a digital signal. As the AED <NUM> processes the audio data <NUM>, the AED may store the audio data <NUM> in a buffer of the memory hardware <NUM> for additional processing. With the audio data <NUM> in the buffer, the AED <NUM> may use a hotword detector <NUM> to detect whether the audio data <NUM> includes the hotword. The hotword detector <NUM> is configured to identify hotwords that are included in the audio data <NUM> without performing speech recognition on the audio data <NUM>.

In some implementations, the hotword detector <NUM> is configured to identify hotwords that are in the initial portion of the utterance <NUM>. In this example, the hotword detector <NUM> may determine that the utterance <NUM> "Ok computer, play music" includes the hotword <NUM> "ok computer" if the hotword detector <NUM> detects acoustic features in the audio data <NUM> that are characteristic of the hotword <NUM>. The acoustic features may be mel-frequency cepstral coefficients (MFCCs) that are representations of short-term power spectrums of the utterance <NUM> or may be mel-scale filterbank energies for the utterance <NUM>. For example, the hotword detector <NUM> may detect that the utterance <NUM> "Ok computer, play music" includes the hotword <NUM> "ok computer" based on generating MFCCs from the audio data <NUM> and classifying that the MFCCs include MFCCs that are similar to MFCCs that are characteristic of the hotword "ok computer" as stored in a hotword model of the hotword detector <NUM>. As another example, the hotword detector <NUM> may detect that the utterance <NUM> "Ok computer, play music" includes the hotword <NUM> "ok computer" based on generating mel-scale filterbank energies from the audio data <NUM> and classifying that the mel-scale filterbank energies include mel-scale filterbank energies that are similar to mel-scale filterbank energies that are characteristic of the hotword "ok computer" as stored in the hotword model of the hotword detector <NUM>.

When the hotword detector <NUM> determines that the audio data <NUM> that corresponds to the utterance <NUM> includes the hotword <NUM>, the AED <NUM> may trigger a wake-up process to initiate speech recognition on the audio data <NUM> that corresponds to the utterance <NUM>. For example, a speech recognizer <NUM> running on the AED <NUM> may perform speech recognition or semantic interpretation on the audio data <NUM> that corresponds to the utterance <NUM>. The speech recognizer <NUM> may perform speech recognition on the portion of the audio data <NUM> that follows the hotword <NUM>. In this example, the speech recognizer <NUM> may identify the words "play music" in the command <NUM>.

In some implementations, the speech recognizer <NUM> is located on a server <NUM> in addition to, or in lieu, of the AED <NUM>. Upon the hotword detector <NUM> triggering the AED <NUM> to wake-up responsive to detecting the hotword <NUM> in the utterance <NUM>, the AED <NUM> may transmit the audio data <NUM> corresponding to the utterance <NUM> to the server <NUM> via a network <NUM>. The AED <NUM> may transmit the portion of the audio data <NUM> that includes the hotword <NUM> for the server <NUM> to confirm the presence of the hotword <NUM>. Alternatively, the AED <NUM> may transmit only the portion of the audio data <NUM> that corresponds to the portion of the utterance <NUM> after the hotword <NUM> to the server <NUM>. The server <NUM> executes the speech recognizer <NUM> to perform speech recognition and returns a transcription of the audio data <NUM> to the AED <NUM>. In turn, the AED <NUM> identifies the words in the utterance <NUM>, and the AED <NUM> performs semantic interpretation and identifies any speech commands. The AED <NUM> (and/or the server <NUM>) may identify the command for the digital assistant <NUM> to perform the long-standing operation of "play music". In the example shown, the digital assistant <NUM> begins to perform the long-standing operation of playing music <NUM> as playback audio from the speaker <NUM> of the AED <NUM>. The digital assistant <NUM> may stream the music <NUM> from a streaming service (not shown) or the digital assistant <NUM> may instruct the AED <NUM> to play music stored on the AED <NUM>.

The AED <NUM> (and/or the server <NUM>) may include an operation identifier <NUM> and a warm word selector <NUM>. The operation identifier <NUM> may be configured to identify one or more long-standing operations the digital assistant <NUM> is currently performing. For each long-standing operation the digital assistant <NUM> is currently performing, the warm word selector <NUM> may select a corresponding set of one or more warm words <NUM> each associated with a respective action for controlling the long-standing operation. In some examples, the warm word selector <NUM> accesses a registry or table (e.g., stored on the memory hardware <NUM>) that associates the identified long-standing operation with a corresponding set of one or more warm words <NUM> that are highly correlated with the long-standing operation. For example, if the long-standing operation corresponds to a set timer function, the associated set of one or more warm words <NUM> available the warm word selector <NUM> to activate includes the warm word <NUM> "stop timer" for instructing the digital assistant <NUM> to stop the timer. Similarly, for the long-standing operation of "Call [contact name]" the associated set of warm words <NUM> includes a "hang up" and/or "end call" warm word(s) <NUM> for ending the call in progress. In the example shown, for the long-standing operation of playing music <NUM>, the associated set of one or more warm words <NUM> available for the warm word selector <NUM> to activate includes the warm words <NUM> "next", "pause", "previous", "volume up", and "volume down" each associated with the respective action for controlling playback of the music <NUM> from the speaker <NUM> of the AED <NUM>. Accordingly, the warm word selector <NUM> activates these warm words <NUM> while the digital assistant <NUM> is performing the long-standing operation and may deactivate these warm words <NUM> once the long-standing operation ends. Similarly, different warm words <NUM> may activate/deactivate depending on a state of the long-standing operation in progress. For example, if the user speaks "pause" to pause the playback of music <NUM>, the warm word selector <NUM> may activate a warm word <NUM> for "play" to resume the playback of the music <NUM>. In some configurations, instead of accessing a registry, the warm word selector <NUM> examines code associated with an application of the long-standing operation (e.g., a music application running in the foreground or background of the AED <NUM>) to identify any warm words <NUM> that developers of the application want users <NUM> to be able to speak to interact with the application and the respective actions for each warm word <NUM>. The warm words <NUM> in the registry may also relate to follow-up queries that the user <NUM> (or typical users) tend to issue following the given query, e.g., "Ok computer, next track".

In some implementations, after activating the set of one or more warm words <NUM> correlated to the long-standing operation, an associator <NUM> executing on the AED <NUM> (or the server <NUM>) associates the activated set of one or more warm words <NUM> with only the user <NUM> that spoke the utterance <NUM> with the command <NUM> for the digital assistant <NUM> to perform the long-standing operation. That is, the associator <NUM> configures the activated set of warm words <NUM> to be dependent on a speaking voice of the particular user <NUM> that provided the initial command <NUM> to initiate the long-standing operation. As will become apparent, by depending the activated set of warm words <NUM> on the speaking voice of the particular user <NUM>, the AED <NUM> (e.g., via the digital assistant <NUM>) will only perform the respective action associated with one of the active warm words <NUM> when the active warm word is spoken by the particular user <NUM>, and thereby suppress performance (or at least require approval from the particular user <NUM>) of the respective action when the warm word <NUM> is spoken by a different speaker.

In order for the associator <NUM> to associate the activated set of one or more warm words <NUM> with only the user <NUM> that spoke the utterance <NUM> initiating the command <NUM>, the associator <NUM> must first resolve an identity of the user <NUM> that spoke the utterance <NUM>. In some scenarios, the user <NUM> is identified as an enrolled user <NUM> of the AED <NUM> that is authorized to access or control various functions of the AED <NUM> and digital assistant <NUM>. The AED <NUM> may have multiple different enrolled users <NUM> each having registered user accounts indicating particular permissions or rights regarding functionality of the AED <NUM>. For instance, the AED <NUM> may operate in a multi-user environment such as a household with multiple family members, whereby each family member corresponds to an enrolled user <NUM> having permissions for accessing a different respective set of resources. To illustrate, a father speaking the command "play my music playlist" would result in the digital assistant <NUM> streaming music from a rock music playlist associated with the father, as opposed to a different music playlists created by, and associated with, another enrolled user <NUM> of the household such as a teenage daughter whose playlist includes pop music.

<FIG> shows an example data store storing enrolled user data/information for each of multiple enrolled users 200a-n of the AED <NUM>. Here, each enrolled user <NUM> of the AED <NUM> may undertake a voice enrollment process to obtain a respective enrolled speaker vector <NUM> from audio samples of multiple enrollment phrases spoken by the enrolled user <NUM>. For example, a speaker-discriminative model <NUM> (<FIG> and <FIG>) may generate one or more enrolled speaker vectors <NUM> from the audio samples of enrollment phrases spoken by each enrolled user <NUM> that may be combined, e.g., averaged or otherwise accumulated, to form the respective enrolled speaker vector <NUM>. One or more of the enrolled users <NUM> may use the AED <NUM> to conduct the voice enrollment process, where the microphone <NUM> captures the audio samples of these users speaking the enrollment utterances and the speaker-discriminative model <NUM> generates the respective enrolled speaker vectors <NUM> therefrom. The model <NUM> may execute on the AED <NUM>, the server <NUM>, or a combination thereof. Additionally, one or more of the enrolled users <NUM> may enroll with the AED <NUM> by providing authorization and authentication credentials to an existing user account with the AED <NUM>. Here, the existing user account may store enrolled speaker vectors <NUM> obtained from a previous voice enrollment process with another device also linked to the user account.

In some examples, the enrolled speaker vector <NUM> for an enrolled user <NUM> includes a text-dependent enrolled speaker vector. For instance, the text-dependent enrolled speaker vector may be extracted from one or more audio samples of the respective enrolled user <NUM> speaking a predetermined term such as the hotword <NUM> (e.g., "Ok computer") used for invoking the AED <NUM> to wake-up from a sleep state. In other examples, the enrolled speaker vector <NUM> for an enrolled user <NUM> is text-independent obtained from one or more audio samples of the respective enrolled user <NUM> speaking phrases with different terms/words and of different lengths. In these examples, the text-independent enrolled speaker vector may be obtained over time from audio samples obtained from speech interactions the user <NUM> has with the AED <NUM> or other device linked to the same account.

<FIG> also shows the AED <NUM> (and/or server <NUM>) optionally storing one or more other text-dependent speaker vectors <NUM> each extracted from one or more audio samples of the respective enrolled user <NUM> speaking a specific term or phrase. For example, the enrolled user 200a may include a respective text-dependent speaker vector <NUM> for each of one or more warm words <NUM> that, when active, may be spoken to cause the AED <NUM> to perform a respective action for controlling a long-standing operation or perform some other command. Accordingly, a text-dependent speaker vector <NUM> for a respective enrolled user <NUM> represents speech characteristics of the respective enrolled user <NUM> speaking the specific warm word <NUM>. Described in greater detail below with reference to <FIG>, the text-dependent speaker vector <NUM> stored for a respective enrolled user <NUM> that is associated with a specific warm word <NUM> may be used to verify the respective enrolled user <NUM> speaking the specific warm word <NUM> to command the AED <NUM> to perform an action for controlling a long-standing operation.

Referring to <FIG>, in some examples, the associator <NUM> resolves the identity of the user <NUM> that spoke the utterance <NUM> by performing a speaker identification process 400a. The speaker identification process 400a may execute on the data processing hardware <NUM> of the AED <NUM>. The process 400a may also execute on the server <NUM>. The speaker identification process 400a identifies the user <NUM> that spoke the utterance <NUM> by first extracting, from the audio data <NUM> corresponding to the utterance <NUM> spoken by the user <NUM>, a first speaker-discriminative vector <NUM> representing characteristics of the utterance <NUM>. Here, the speaker identification process 400a may execute a speaker-discriminative model <NUM> configured to receive the audio data <NUM> as input and generate, as output, the first speaker-discriminative vector <NUM>. The speaker-discriminative model <NUM> may be a neural network model trained under machine or human supervision to output speaker-discriminative vectors <NUM>. The speaker-discriminative vector <NUM> output by the speaker-discriminative model <NUM> may include an N-dimensional vector having a value that corresponds to speech features of the utterance <NUM> that are associated with the user <NUM>. In some examples, the speaker-discriminative vector <NUM> is a d-vector. In some examples, the first speaker-discriminative vector <NUM> includes a set of speaker-discriminative vectors each associated with a different user who is also authorized to speak the activated set of warm words. For instance, aside from the user <NUM> that spoke the utterance <NUM>, other authorized users could include other individuals who were present when the user spoke <NUM> the utterance <NUM> issuing the long-standing operation and/or individuals that the user <NUM> added/specified as being authorized.

Once the first speaker-discriminative vector <NUM> is output from the model <NUM>, the speaker identification process 400a determines whether the extracted speaker-discriminative vector <NUM> matches any of the enrolled speaker vectors <NUM> stored on the AED <NUM> (e.g., in the memory hardware <NUM>) for enrolled users 200a-n (<FIG>) of the AED <NUM>. As described above with reference to <FIG>, the speaker-discriminative model <NUM> may generate the enrolled speaker vectors <NUM> for the enrolled users <NUM> during a voice enrollment process. Each enrolled speaker vector <NUM> may be used as a reference vector corresponding to a voiceprint or unique identifier representing characteristics of the voice of the respective enrolled user <NUM>.

In some implementations, the speaker identification process 400a uses a comparator <NUM> that compares the first speaker-discriminative vector <NUM> to the respective enrolled speaker vector <NUM> associated with each enrolled user 200a-n of the AED <NUM>. Here, the comparator <NUM> may generate a score for each comparison indicating a likelihood that the utterance <NUM> corresponds to an identity of the respective enrolled user <NUM>, and the identity is accepted when the score satisfies a threshold. When the score does not satisfy the threshold, the comparator may reject the identity. In some implementations, the comparator <NUM> computes a respective cosine distance between the first speaker-discriminative vector <NUM> and each enrolled speaker vector <NUM> and determines the first speaker-discriminative vector <NUM> matches one of the enrolled speaker vectors <NUM> when the respective cosine distance satisfies a cosine distance threshold.

In some examples, the first speaker-discriminative vector <NUM> is a text-dependent speaker-discriminative vector extracted from a portion of the audio data that includes the hotword <NUM> and each enrolled speaker vector <NUM> is also text-dependent on the same hotword <NUM>. The use of text-dependent speaker vectors can improve accuracy in determining whether the first speaker-discriminative vector <NUM> matches any of the enrolled speaker vectors <NUM>. In other examples, the first speaker-discriminative vector <NUM> is a text-independent speaker-discriminative vector extracted from the entire audio data that includes both the hotword <NUM> and the command <NUM> or from the portion of the audio data that includes the command <NUM>.

When the speaker identification process 400a determines that the first speaker-discriminative vector <NUM> matches one of the enrolled speaker vectors <NUM>, the process 400a identifies the user <NUM> that spoke the utterance <NUM> as the respective enrolled user <NUM> associated with the one of the enrolled speaker vectors <NUM> that matches the extracted speaker-discriminative vector <NUM>. In the example shown, the comparator <NUM> determines the match based on the respective cosine distance between the first speaker-discriminative vector <NUM> and the enrolled speaker vector <NUM> associated with the first enrolled user 200a satisfying a cosine distance threshold. In some scenarios, the comparator <NUM> identifies the user <NUM> as the respective first enrolled user 200a associated with the enrolled speaker vector <NUM> having the shortest respective cosine distance from the first speaker-discriminative vector <NUM>, provided this shortest respective cosine distance also satisfies the cosine distance threshold.

Conversely, when the speaker identification process 400a determines that the first speaker-discriminative vector <NUM> does not match any of the enrolled speaker vectors <NUM>, the process 400a may identify the user <NUM> that spoke the utterance <NUM> as a guest user of the AED <NUM>. Accordingly, the associator <NUM> may associate the activated set of one or more warm words <NUM> with the guest user and use the first speaker-discriminative vector <NUM> as a reference speaker vector representing the speech characteristics of the voice of the guest user. In some instances, the guest user could enroll with the AED <NUM> and the AED <NUM> could store the first speaker-discriminative vector <NUM> as a respective enrolled speaker vector <NUM> for the newly enrolled user.

In the example shown in <FIG>, the associator <NUM> associates the activated set of one or more warm words <NUM> with the first enrolled user 200a named Barb. In some examples, the AED <NUM> notifies the identified user <NUM> (e.g., Barb) associated with the activated set of one or more warm words <NUM> that the warm words <NUM> are active and that the user <NUM> can speak any of the warm words <NUM> to instruct the AED <NUM> to perform the respective action for controlling the long-standing operation. For instance, the digital assistant <NUM> may generate synthesized speech <NUM> for audible output from the speaker <NUM> of the AED <NUM> that states, "Barb, you may speak music playback controls without saying 'Ok computer'". In additional examples, the digital assistant <NUM> may provide a notification to a user device <NUM> (e.g., smart phone) linked back to user account of the identified user to inform the identified user <NUM> (e.g., Barb) which warm words <NUM> are currently active for controlling the long-standing operation.

A graphical user interface (GUI) <NUM> (<FIG>) executing on the user device <NUM> may display the active warm words <NUM> and associated respective actions for controlling the long-standing operation. Each warm word itself may serve as a descriptor identifying the respective action. <FIG> provides an example GUI <NUM> displayed on a screen of the user device <NUM> to inform the user <NUM> which warm words <NUM> are active for that user <NUM> to speak to control the long-standing operation, and which warm words <NUM> are inactive, and thus, unavailable for controlling the long-standing operation when spoken by the user <NUM>. Specifically, the GUI <NUM> renders the active warm words <NUM> "next", "pause", "previous", "volume up", and "volume down," and the inactive warm word <NUM> "play". If the user <NUM> were to pause playback of the music, the warm word for "play" may become an active warm word <NUM> and the warm word for "pause" may become an inactive warm word <NUM>. Each warm word <NUM> is associated with the respective action for controlling playback of the music <NUM> from the speaker <NUM> of the AED <NUM>.

Additionally, the GUI <NUM> may render for display an identifier of the long-standing operation (e.g., "Playing Track <NUM>"), an identifier of the AED <NUM> (e.g., smart speaker) that is currently performing the long-standing operation, and/or an identity of the active user <NUM> (e.g., Barb) that initiated the long-standing operation. In some implementations, the identity of the active user <NUM> includes an image <NUM> of the active user <NUM>. Accordingly, by identifying the active user <NUM> and the active warm words <NUM>, the GUI <NUM> reveals the active user <NUM> as a "controller" of the long-standing operation that may speak any of the active warm words <NUM> displayed in GUI <NUM> to perform a respective action for controlling the long-standing operation. As mentioned above, the active set of warm words <NUM> are dependent on the speaking voice of Barb <NUM>, since Barb <NUM> seeded the initial command <NUM> "play music," to initiate the long-standing operation. By depending the active set of warm words <NUM> on the speaking voice of Barb <NUM>, the AED <NUM> (e.g., via the digital assistant <NUM>) will only perform a respective action associated with one of the active warm words <NUM> when the active warm word <NUM> is spoken by Barb <NUM>, and will suppress performance (or at least require approval from Barb <NUM>) of the respective action when the active warm word <NUM> is spoken by a different speaker.

The user device <NUM> may also render graphical elements <NUM> for display in the GUI <NUM> for performing the respective actions associated with the respective active warm words <NUM> to playback of the music <NUM> from the speaker <NUM> of the AED <NUM>. In the example shown, the graphical elements <NUM> are associated with playback controls for the long-standing operation of playing music <NUM>, that when selected, cause the device <NUM> to perform a respective action. For instance, the graphical elements <NUM> may include playback controls for performing the action associated with the warm word <NUM> "next," performing the action associated with the warm word <NUM> "pause," performing the action associated with the warm word <NUM> "previous," performing the action associated with warm word <NUM> "volume up," and performing the action associated with the warm word <NUM> "volume down. " The GUI <NUM> may receive user input indications via any one of touch, speech, gesture, gaze, and/or an input device (e.g., mouse or stylist) to control the playback of the music <NUM> from the speaker <NUM> of the AED <NUM>. For example the user <NUM> may provide a user input indication indicating selection of the "next" control (e.g., by touching the graphical button in the GUI <NUM> that universally represents "next") to cause the AED <NUM> to perform the action of advancing to the next song in the playlist associated with the music <NUM>.

Referring back to <FIG> and <FIG>, activating the set of warm words <NUM> by the warm word selector <NUM> causes the AED <NUM> to activate, for each corresponding warm word <NUM> in the activated set of one or more warm words <NUM>, a respective warm word model <NUM> to run the AED <NUM>. Here, the warm word models <NUM> may be stored on the memory hardware <NUM> of the AED <NUM> or on the server <NUM>. If stored on the server, the AED <NUM> may request the server <NUM> to retrieve a warm word model <NUM> for a corresponding warm word <NUM> and provide the retrieved warm word model <NUM> so that the AED <NUM> can activate the warm word model <NUM>. As will be described in greater detail below, an active warm word model <NUM> running on the AED <NUM> may detect an utterance <NUM> of the corresponding active warm word <NUM> in streaming audio captured by the AED <NUM> without performing speech recognition on the captured audio. Further, a single warm word model <NUM> may be capable of detecting all of the active warm words <NUM> in streaming audio. Thus, a warm word model <NUM> may detect a set of active warm words, and a different warm word model <NUM> may detect a different set of active warm words.

In additional implementations, activating the set of warm words <NUM> by the warm word selector <NUM> causes the AED <NUM> to execute the speech recognizer <NUM> on the AED <NUM> in a low-power and low-fidelity state. Here, the speech recognizer <NUM> is constrained or biased to only recognize the one or more warm words <NUM> that are active when spoken in the utterance captured by the AED <NUM>. Since the speech recognizer <NUM> is only recognizing a limited number of terms/phrases, the number of parameters of the speech recognizer <NUM> may be drastically reduced, thereby reducing the memory requirements and number of computations needed for recognizing the active warm words in speech. Accordingly, the low-power and low-fidelity characteristics of the speech recognizer <NUM> may be suitable for execution on a digital signal processor (DSP). In these implementations, the speech recognizer <NUM> executing on the AED <NUM> may recognize an utterance <NUM> of an active warm word <NUM> in streaming audio captured by the AED <NUM> in lieu of using a warm word model <NUM>.

Referring to <FIG>, while the digital assistant <NUM> is performing the long-standing operation of playing music <NUM>, the user <NUM> speaks an utterance <NUM> that includes one of the warm words <NUM> from the activated set of one or more warm words <NUM>. In the example shown, the user <NUM> utters the active warm word <NUM> "next". Without performing speech recognition on the captured audio, the AED <NUM> may apply the warm word models <NUM> activated for the activated set of one or more warm words <NUM> to identify whether the utterance <NUM> includes any active warm words <NUM>. The active warm words <NUM> may be "next", "pause", "previous", "volume up", and "volume down". The AED <NUM> compares the audio data that corresponds to the utterance <NUM> to the activated warm word models <NUM> that correspond to the active warm words <NUM> "next," "pause," "previous," "volume up," and "volume down" and determines that the warm word model <NUM> activated for the warm word <NUM> "next" detects the warm word <NUM> "next" in the utterance <NUM> without performing speech recognition on the audio data. Based on identifying the warm word <NUM> "next" that corresponds to the instruction to advance to the next song, the AED <NUM> performs a speaker verification process 400b on the audio data corresponding to the utterance <NUM> to determine whether the utterance <NUM> was spoken by the same user <NUM> that is associated with the activated set of one or more warm words <NUM>. If the speaker verification process 400b on the audio data corresponding to the utterance <NUM> indicates that the utterance <NUM> was spoken by the same user <NUM> that is associated with the activated set of one or more warm words <NUM>, the AED <NUM> may proceed with performing the respective action associated with the identified warm word <NUM> for controlling the long-standing operation. Conversely, if the speaker verification process 400b on the audio data corresponding to the utterance <NUM> indicates that the utterance <NUM> was not spoken by the same user <NUM> that is associated with the activated set of one or more warm words <NUM>, the AED <NUM> may suppress performance (or at least require approval from the particular user <NUM> (e.g., in <FIG>)) of the respective action when the warm word <NUM> is spoken by the different speaker.

In some implementations, the AED <NUM> identifies warm words <NUM> that are not among the activated set of one or more warm words <NUM> but whose models are still stored in the warm word models <NUM>. In this instance, the AED <NUM> may provide an indication to the user device <NUM> to display in the GUI <NUM> that the warm word is not among (e.g., inactive) the activated set of the one or more warm words <NUM>. For example, the user <NUM> may speak "play" when the music <NUM> is playing. The AED <NUM> may identify the warm word <NUM> "play. " Because the warm word <NUM> "play" is not among the activated set of one or more warm words <NUM>, the AED <NUM> performs no action. However, the user device <NUM> may display in GUI <NUM> an indication that the warm word "play" is an inactive warm word <NUM> and indicate that the active warm words <NUM> are "next," "pause," "previous," "volume up," and "volume down.

The warm word models <NUM> may detect that the associated utterance <NUM> includes one of the warm words <NUM> from the activated set of one or more warm words <NUM> by extracting audio features of the audio data associated with the utterance <NUM>. The activated warm word models <NUM> may each generate a corresponding warm word confidence score by processing the extracted audio features and comparing the corresponding warm word confidence score to a warm word confidence threshold. For instance, the warm word models <NUM> may collectively generate corresponding warm word confidence scores for each of the active warm words <NUM> "play", "next", "pause", "previous", volume up", and "volume down". In some implementations, the speech recognizer <NUM> generates a warm word confidence score for each portion of processed audio data associated with the utterance <NUM>. If the warm word confidence score satisfies a threshold, then the warm word model <NUM> determines that the audio data corresponding to the utterance <NUM> includes a warm word <NUM> among the activated set of the one or more warm words <NUM>. For example, if the warm word confidence score generated by a warm word model <NUM> (or the speech recognizer <NUM>) is <NUM> and the warm word confidence threshold is <NUM>, then the AED <NUM> determines that the audio data corresponding with the utterance <NUM> includes a warm word <NUM>.

In some implementations, if the warm word confidence score is within a range below the threshold, the digital assistant <NUM> may generate synthesized speech <NUM> for audible output from the speaker <NUM> of the AED <NUM> that requests that the user <NUM> confirm or repeat the warm word <NUM>. In these implementations, if the user <NUM> confirms that the user <NUM> spoke the warm word <NUM>, the AED may use the audio data to update the corresponding warm word model <NUM>.

Referring to <FIG>, in response to identifying that the additional audio data <NUM> corresponding to the additional utterance <NUM> includes one of the warm words <NUM> from the activated set of warm words <NUM>, the associator <NUM> resolves the identity of the user <NUM> that spoke the utterance <NUM> by performing the speaker verification process 400b. The speaker verification process 400b may execute on the data processing hardware <NUM> of the AED <NUM>. The process 400b may also execute on the server <NUM> instead of or in combination with executing on the AED <NUM>. The speaker verification process 400b identifies the user <NUM> that spoke the utterance <NUM> by first extracting, from the audio data <NUM> corresponding to the utterance <NUM> spoken by the user <NUM>, a second speaker-discriminative vector <NUM> representing characteristics of the utterance <NUM>. Here, the speaker verification process 400b may execute the speaker-discriminative model <NUM> configured to receive the audio data <NUM> as input and generate, as output, the second speaker-discriminative vector <NUM>. As discussed above in <FIG>, the speaker-discriminative model <NUM> may be a neural network model trained under machine or human supervision to output speaker-discriminative vectors <NUM>. The second speaker-discriminative vector <NUM> output by the speaker-discriminative model <NUM> may include an N-dimensional vector having a value that corresponds to speech features of the utterance <NUM> that are associated with the user <NUM>. In some examples, the speaker-discriminative vector <NUM> is a d-vector.

Once the second speaker-discriminative vector <NUM> is output from the speaker-discriminator model <NUM>, the speaker verification process 400b determines whether the extracted speaker-discriminative vector <NUM> matches a reference speaker vector <NUM> associated with the first enrolled user 200a stored on the AED <NUM> (e.g., in the memory hardware <NUM>). The reference speaker vector <NUM> associated with the first enrolled user 200a may include the respective enrolled speaker vector <NUM> associated with the first enrolled user 200a. In other examples, the reference speaker vector <NUM> includes a text-dependent speaker vector <NUM> extracted from one or more audio samples of the respective enrolled user 200a speaking the active warm word <NUM> that was identified in the additional audio data <NUM> corresponding to the utterance <NUM>.

As described above with reference to <FIG>, the speaker-discriminative model <NUM> may generate the enrolled speaker vectors <NUM> for the enrolled users <NUM> during a voice enrollment process. Each enrolled speaker vector <NUM> may be used as a reference vector corresponding to a voiceprint or unique identifier representing characteristics of the voice of the respective enrolled user <NUM>. Additionally, during enrollment, the enrolled user 200a may train text-dependent speaker vectors <NUM> for each of one or more warm words <NUM> that, when active, may be spoken to cause the AED <NUM> to perform a respective action for controlling a long-standing operation or perform some other command. For example, the text-dependent speaker vector <NUM> for enrolled user 200a represents speech characteristics of the enrolled user 200a speaking the respective warm words <NUM> of "play," "pause," "next," "previous," "volume up," and "volume down.

In some implementations, the speaker verification process 400b uses a comparator <NUM> that compares the second speaker-discriminative vector <NUM> to the reference speaker vector <NUM> associated with the first enrolled user 200a of the enrolled users <NUM>. Here, the comparator <NUM> may generate a score for the comparison indicating a likelihood that the utterance <NUM> corresponds to the identity of the first enrolled user 200a, and the identity is accepted when the score satisfies a threshold. When the score does not satisfy the threshold, the comparator <NUM> may reject the identity. In some implementations, the comparator <NUM> computes a respective cosine distance between the second speaker-discriminative vector <NUM> and the reference speaker vector <NUM> associated with the first enrolled user 200a and determines the second speaker-discriminative vector matches the reference speaker vector <NUM> when the respective cosine distance satisfies a cosine distance threshold.

When the speaker verification process 400b determines that the second speaker-discriminative vector <NUM> matches the reference speaker vector <NUM> associated with the first enrolled user 200a, the process 400b identifies the user <NUM> that spoke the utterance <NUM> as the first enrolled user 200a associated with the reference speaker vector <NUM>. In the example shown, the comparator <NUM> determines the match based on the respective cosine distance between the second speaker-discriminative vector <NUM> and the reference speaker vector <NUM> associated with the first enrolled user 200a satisfying a cosine distance threshold. In some scenarios, the comparator <NUM> identifies the user <NUM> as the respective first enrolled user 200a associated with the reference speaker vector <NUM> having the shortest respective cosine distance from the second speaker-discriminative vector <NUM>, provided this shortest respective cosine distance also satisfies the cosine distance threshold.

With reference to <FIG> above, in some implementations, the speaker identification process 400a determines that the first speaker-discriminative vector <NUM> does not match any of the enrolled speaker vectors <NUM>, and identifies the user <NUM> that spoke the utterance <NUM> as a guest user of the AED <NUM> and associates the activated set of one or more warm words <NUM> with the user <NUM>. Accordingly, the speaker verification process 400b may first determine whether the user <NUM> that spoke the utterance <NUM> was identified by the speaker identification process 400a as an enrolled user <NUM> or a guest user. When the user <NUM> is a guest user, the comparator <NUM> compares the second speaker-discriminative vector <NUM> to the first speaker-discriminative vector <NUM> obtained during the speaker identification process 400a. Here, the first speaker-discriminative vector <NUM> represents the characteristics of the utterance <NUM> spoken by the guest user <NUM>, and thus, is used as a reference vector to verify whether or not the utterance <NUM> was spoken by the guest user <NUM> or another user. Here, the comparator <NUM> may generate a score for the comparison indicating a likelihood that the utterance <NUM> corresponds to the identity of the guest user <NUM>, and the identity is accepted when the score satisfies a threshold. When the score does not satisfy the threshold, the comparator <NUM> may reject the identity of the guest user that spoke the utterance <NUM>. In some implementations, the comparator <NUM> computes a respective cosine distance between the first speaker-discriminative vector <NUM> and the second speaker-discriminative vector <NUM> and determines the first speaker-discriminative vector <NUM> matches the second speaker-discriminative vector <NUM> when the respective cosine distance satisfies a cosine distance threshold.

Referring back to <FIG>, the AED <NUM> (e.g., via the digital assistant <NUM>) will only proceed to perform the respective action of advancing to the next song/track associated with the active warm word <NUM> "next" when the speaker verification process 400b verifies the speaker of the utterance <NUM>. In other words, after the word <NUM> "next" is detected in the additional audio <NUM> corresponding to the additional utterance <NUM>, the AED must verify that the additional utterance <NUM> was spoken by the same user <NUM> that is associated with the activated set of one or more warm words before performing the respective action associated with the detected warm word <NUM> "next" of advancing to the next song in the playlist associated with the music <NUM>. If the speaker verification process 400b is unable to verify that the additional utterance <NUM> was spoken by the same speaker that spoke the utterance <NUM>, the AED will suppress performance (or at least require approval from the user <NUM>) of the respective action when the warm word <NUM> "next" is spoken by a different speaker.

Referring now to <FIG>, another example system 100c shows another user <NUM> speaking an utterance <NUM> that includes the active warm word <NUM> "stop" from the activated set of one or more warm words <NUM> while the digital assistant <NUM> is performing the long-standing operation of playing music <NUM>. Without performing speech recognition on audio data that corresponds to the utterance <NUM>, the AED <NUM> may compare the audio data to the warm word models <NUM> activated for the activated set of one or more warm words <NUM> to identify whether the utterance <NUM> includes any active warm words <NUM>. The active warm words <NUM> may include "play", "next", "stop", "back", "volume up", and "volume down". In the example shown, the AED <NUM> determines that the warm word model <NUM> activated for the warm word <NUM> "stop" detects the warm word <NUM> "stop" in the utterance <NUM> without performing speech recognition on the audio data <NUM>. Based on identifying the warm word <NUM> "stop", the AED <NUM> performs the speaker verification process 400b on the audio data corresponding to the utterance <NUM> to determine whether the utterance <NUM> was spoken by the same user <NUM> (e.g., Barb) that is associated with the activated set of one or more warm words <NUM>.

In the example shown, performing speaker verification process 400b on the audio data <NUM> corresponding to the utterance <NUM> indicates that the utterance <NUM> was not spoken by Barb <NUM>. For example, the discriminative-speaker vector model <NUM> of the speaker verification process 400b of <FIG> may generate a second speaker-discriminative vector <NUM> representing characteristics of the utterance <NUM> and the comparator <NUM> may determine that the second speaker-discriminative vector <NUM> does not match the reference speaker vector <NUM> associated with the first enrolled user 200a (e.g., Barb). Accordingly, the speaker verification process 400b determines that the utterance <NUM> was spoken by a different user <NUM> than the user <NUM> (e.g., Barb) that is associated with the activated set of one or more warm words <NUM>. In some implementations, the AED <NUM> suppresses performance of the respective action associated with the identified warm word <NUM> "stop" for controlling the long-standing operation of playing music <NUM> when the utterance <NUM> was spoken by the different user <NUM>. In other implementations, in response to the verification process 400b determining that the utterance <NUM> was spoken by someone other than Barb <NUM>, the AED <NUM> requests approval from Barb <NUM> to perform the respective action associated with the identified warm word <NUM> "stop". In one example, the digital assistant <NUM> requests approval from Barb <NUM> by generating synthesized speech <NUM> for audible output from the speaker <NUM> of the AED <NUM> that prompts Barb <NUM> to authorize or reject the performance of the respective action associated with the warm word <NUM> "stop". For instance, the digital assistant <NUM> may generate synthesized speech <NUM> for audible output from the speaker <NUM> of the AED <NUM> that asks, "Barb, an action to stop playing the song was requested by someone else, do you accept?" In response, Barb <NUM> speaks an utterance <NUM> that includes an acknowledgement <NUM>. The acknowledgement <NUM> by the user <NUM> may be "yes" to authorize performance of the respective action, or "no" to reject performance of the respective action.

In the example shown, Barb <NUM> utters the acknowledgement <NUM> "yes" that corresponds to authorizing the performance of the respective action associated with the warm word <NUM> "stop". In some implementations, the AED <NUM> also performs the speaker verification process 400b on audio data corresponding to the utterance <NUM> to verify that the acknowledgement <NUM> was spoken by the authorized user <NUM> (e.g., Barb) that is associated with the activated set of one or more warm words <NUM>. Based on receiving the acknowledgement <NUM> "yes" from Barb <NUM> authorizing performance of the respective action "stop" and the speaker verification process 400b performed on the audio data corresponding to the utterance <NUM> indicating that the utterance <NUM> was spoken by Barb <NUM>, the AED <NUM> may proceed with performing the respective action associated with the identified warm word "stop" <NUM> spoken by the different user <NUM> for controlling the long-standing operation. Conversely, when the speaker verification process 400b performed on the audio data corresponding to the utterance <NUM> indicates that the utterance <NUM> was not spoken by Barb <NUM> or when the acknowledgement <NUM> spoken by Barb <NUM> was instead "no", the AED <NUM> may suppress performance of the respective action "stop" so that the music playing <NUM> from the AED <NUM> continues without stopping.

Additionally or alternatively, the digital assistant <NUM> may provide a notification to the user device <NUM> linked to the user account of Barb <NUM> that prompts Barb <NUM> to authorize or reject the performance of the respective action associated with the warm word <NUM> "stop". For instance, the GUI <NUM> displayed on the user device <NUM> may render the prompt as a notification to allow Barb <NUM> to authorize or reject the performance of the respective action associated with the warm word <NUM> "stop". In one example, the GUI <NUM> renders graphical elements/buttons that Barb <NUM> may select to authorize or reject the performance of the respective action. For instance, Barb <NUM> may provide a user input indication indicating selection of a graphical element/button for the acknowledgement <NUM> "yes" (e.g., by touching the graphical element for "yes" in the GUI <NUM>) that corresponds to authorizing the performance of the respective action associated with the warm word <NUM> "stop". In response to receiving the user input indication indicating selection of the graphical element/button for the acknowledgement <NUM> "yes" by Barb <NUM>, the AED <NUM> may proceed with performing the respective action associated with the identified warm word "stop" <NUM> spoken by the different user <NUM> for controlling the long-standing operation. Conversely, Barb <NUM> may provide a user input indication indicating selection of a graphical element/button for the acknowledgement <NUM> "no" (e.g., by touching the graphical element for "no" in the GUI <NUM>) to reject the performance of the respective action associated with the warm word <NUM> "stop", and thereby cause the AED <NUM> to suppress performance of the respective action.

In some implementations, identifying which user <NUM>, <NUM> spoke the utterance <NUM> may help Barb <NUM> determine whether to authorize or reject the action (e.g., when one or more users <NUM> is present in a room with Barb <NUM>). Continuing with the example shown in <FIG>, prompting Barb <NUM> to authorize or reject the performance of the respective action associated with the warm word <NUM> "stop" further includes identifying the different user <NUM> that spoke the utterance <NUM>. Here, when the AED <NUM> performs the speaker verification process 400b on the audio data <NUM> corresponding to the utterance <NUM>, the speaker verification process 400b may identify the user <NUM> as the second enrolled user 200b (e.g., Ed). For instance, the process 400b may determine that the speaker-discriminative vector <NUM> representing characteristics of the utterance <NUM> spoken by the user <NUM> match a respective enrolled speaker vector <NUM> stored on the AED <NUM> for the second enrolled user 200b. Accordingly, when the user <NUM> is identified as one of the enrolled users <NUM>, the digital assistant <NUM> may similarly generate synthesized speech <NUM> for audible output from the speaker <NUM> of the AED <NUM> that prompts the user <NUM>, "Barb, do you authorize Ed's request to stop the music? Is Ed authorized for controlling music playback settings?" In response, Barb <NUM> speaks the utterance <NUM> that includes the acknowledgement <NUM> "yes". In some implementations, Barb <NUM> authorizes Ed's <NUM> request to stop playing music <NUM> but rejects future control of the long-standing operation by Ed <NUM>. Barb <NUM> may also have the ability to authorize Ed <NUM> to control the long-standing operation for a predetermined period of time (e.g., one hour, one day, etc.).

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> for activating a set of one or more warm words <NUM> associated with a long-standing operation. At operation <NUM>, the method <NUM> includes receiving, at data processing hardware <NUM>, audio data <NUM> corresponding to an utterance <NUM> spoken by a user <NUM> and captured by an assistant-enabled device (AED) <NUM> associated with the user <NUM>. The utterance <NUM> includes a command for a digital assistant <NUM> to perform a long-standing operation. The long-standing operation may include setting a timer for a specified amount of time or playing music from a streaming service. The AED <NUM> may identify the user <NUM> that spoke the utterance <NUM> by performing a speaker identification process 400a.

At operation <NUM>, the method <NUM> includes, after receiving the audio data <NUM> corresponding to the utterance <NUM>, activating, by the data processing hardware <NUM>, a set of one or more warm words <NUM> and associating, by the data processing hardware <NUM>, the activated set of one or more warm words <NUM> with only the user <NUM> that spoke the utterance <NUM>. Each warm word <NUM> in the set of one or more warm words <NUM> is associated with a respective action for controlling the long-standing operation. For example, the respective actions for the long-standing operation of playing music <NUM> as playback audio from the speaker <NUM> of the AED <NUM> may include one or more of reverting to a previous song in the playlist, playing the music playlist, pausing the music playlist, skipping to a next song in the playlist, and adjusting a volume level of the speaker <NUM>.

At operation <NUM>, the method <NUM> also includes, while the digital assistant <NUM> is performing the long-standing operation, receiving, at the data processing hardware <NUM>, additional audio data <NUM> corresponding to an additional utterance <NUM> captured by the AED <NUM>. The AED <NUM> identifies, by the data processing hardware <NUM>, in the additional audio data <NUM>, one of the warm words <NUM> from the activated set of one or more warm words <NUM>. The AED <NUM> may identify that one of the warm words <NUM> from the activated set of one or more warm words <NUM> is in the additional audio data <NUM> without performing speech recognition on the captured audio. For example, the AED <NUM> may compare the audio data <NUM> that corresponds to the utterance <NUM> to activated warm word models <NUM> that correspond to the activated set of one or more warm words <NUM>.

At operation <NUM>, the method <NUM> further includes performing, by the data processing hardware <NUM>, speaker verification 400b on the additional audio data <NUM> to determine whether the additional utterance <NUM> was spoken by the same user <NUM> that is associated with the activated set of one or more warm words <NUM>. When the additional utterance <NUM> was spoken by the same user <NUM> that is associated with the activated set of one or more warm words <NUM>, the method <NUM> includes performing, by data processing hardware <NUM>, the respective action associated with the identified one of the warm words <NUM> for controlling the long-standing operation. When the speaker verification <NUM> indicates that the additional utterance <NUM> was not spoken by the same user <NUM> that is associated with the activated set of one or more warm words <NUM>, the AED <NUM> may suppress performance of the respective action.

<FIG> is a schematic view of an example computing device <NUM> that may be used to implement the systems and methods described in this document. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only.

Claim 1:
A method (<NUM>) for activating speaker-dependent warm words (<NUM>), the method (<NUM>) comprising:
receiving, at data processing hardware (<NUM>), audio data (<NUM>) corresponding to an utterance spoken by a user and captured by an assistant-enabled device (<NUM>) associated with the user, the utterance comprising a command (<NUM>) for a digital assistant (<NUM>) to perform a long-standing operation, wherein a long-standing operation is an operation performed by the digital assistant for an extended duration of time;
after receiving the audio data (<NUM>) corresponding to the utterance:
activating, by the data processing hardware (<NUM>), a set of one or more warm words (<NUM>) each associated with a respective action for controlling the long-standing operation; and
associating, by the data processing hardware (<NUM>), the activated set of one or more warm words (<NUM>) with only the user that spoke the utterance; and
while the digital assistant (<NUM>) is performing the long-standing operation:
receiving, at the data processing hardware (<NUM>), additional audio data (<NUM>) corresponding to an additional utterance captured by the assistant-enabled device (<NUM>);
identifying, by the data processing hardware (<NUM>), in the additional audio data (<NUM>), one of the warm words (<NUM>) from the activated set of one or more warm words (<NUM>);
performing, by the data processing hardware (<NUM>), speaker verification on the additional audio data (<NUM>) to determine whether the additional utterance was spoken by the same user (<NUM>) that is associated with the activated set of one or more warm words (<NUM>); and
when it is determined by the speaker verification that the additional utterance was spoken by the same user (<NUM>) that is associated with the activated set of one or more warm words (<NUM>), performing, by the data processing hardware (<NUM>), the respective action associated with the identified one of the warm words (<NUM>) for controlling the long-standing operation.