Patent Description:
In a speech-enabled environment, such as a home or automobile, a user may access information and/or control various functions using voice input. The information and/or functions may be personalized for a given user. It may therefore be advantageous to identify a given speaker from among a group of speakers associated with the speech-enabled environment.

Speaker identification (e.g., speaker verification and voice authentication) provides an easy way for a user of a user device to gain access to the user device Speaker identification allows the user to unlock, and access, the user's device by speaking an utterance without requiring the user manually enter (e.g., via typing or speaking) a passcode to gain access to the user device. Speaker verification also allows a digital assistant to identify authorized users from spoken utterances without requiring the users to provide authorization credentials.

In the prior art, it is known from the patent application <CIT>, speaker identification techniques using a d-vector feature and cosine distance scoring. It is further known according to the patent application <CIT>, techniques for speaker identification using a d-vector and attention modelling. It is further known from the publication <NPL>, techniques for speaker recognition using an i-vector space and applying a multi-condition training.

One aspect of the present disclosure provides a computer-implemented method for speaker identification that when executed on data processing hardware causes the data processing to perform operations that include receiving audio data corresponding to an utterance captured by a user device and processing, using a speaker identification model, the audio data to generate an evaluation attentive d-vector (ad-vector) representing voice characteristics of the utterance. The evaluation ad-vector includes ne component classes (also referred as style classes) each including a respective value vector concatenated with a corresponding routing vector, wherein each routing vector conveys environmental, channel and/or contextual information associated with the audio data. The operations also include generating, using a self-attention mechanism, at least one multi-condition attention score that indicates a likelihood that the evaluation ad-vector matches a respective reference ad-vector associated with a respective user. and identifying the speaker of the utterance as the respective user associated with the respective reference ad-vector based on the multi-condition attention score. The reference ad-vector includes nr component classes each including a respective value vector concatenated with a corresponding routing vector.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, identifying the speaker of the utterance includes determining whether the multi-condition attention score satisfies a threshold score, and when the multi-condition attention score satisfies the threshold score, determining that the speaker of the utterance includes the respective user associated with the respective reference ad-vector Each value vector may include a same first dimensionality and each routing vector may include a same second dimensionality that is less than the first dimensionality of each value vector. As such, a size of the second dimensionality may be one-third a size of the first dimensionality.

In some examples, the respective value vector of each style class of the n style classes in each one of the evaluation ad-vector and the reference ad-vector contains respective speaker-related information, while the respective routing vector of each style class of the n style classes in each one of the evaluation ad-vector and the reference ad-vector. contains channel and/or context information associated with a respective utterance the one of the evaluation ad-vector or the reference ad-vector was extracted from Additionally or alternatively, the routing vectors in the evaluation and reference ad-vectors may be configured to identify matching conditions between the utterance associated with the evaluation ad-vector and at least one reference utterance associated with the reference ad-vector.

In some additional implementations, generating the at least one multi-condition attention score includes using the self-attention mechanisms to generate multiple multi-condition attention scores each indicating a respective likelihood that the evaluation ad-vector matches a respective one of multiple reference ad-vectors, while identifying the speaker of the utterance includes identifying the speaker of the utterance as the respective enrolled user of the user device that is associated with the respective reference ad-vector corresponding to the greatest multi-condition attention score. Each reference ad-vector is associated with a respective one of one or more enrolled users of the user device. In these additional implementations, the utterance captured by the user device may include a query specifying an action to perform, each of the one or more different enrolled users of the user device may have permissions for accessing a different respective set of personal resources, and performance of the action specified by the query may require access to the respective set of personal resources associated with the respective enrolled user identified as the speaker of the utterance.

The speaker identification model may include a neural network having an input layer, a plurality of hidden layers, and an output layer including multiple sets of output nodes. Each set of output nodes in the multiple sets of output nodes of the output layer is designated to learn to generate speaker-related information specific to a respective one of the n style classes Here, processing the audio data to generate the evaluation ad-vector may include using the neural network to process the audio data to generate each of the n style classes for the evaluation ad-vector as output from the respective set of output nodes of the output layer that is designated to learn to generate the speaker-related information specific to the respective style class.

In some examples, processing the audio data to generate the reference ad-vector. generating, as output from the speaker identification model including a neural network, a non-attentive d-vector representing voice characteristics of the utterance; and applying a set of linear and non-linear transformations to transform the non-attentive d-vector into the reference ad-vector. The reference ad-vector may be generated by the speaker identification model in response to receiving one or more previous utterances spoken by the respective user. At least one style class of the nr and ne style classes may be dependent on a fixed term or phrase. Furthermore, the data processing hardware may execute both the speaker identification model and the self-attention mechanism, while residing on one of the user device or a distributed computing system in communication with the user device via a network.

In some implementations, the reference ad-vector is generated by: receiving, as input to the speaker identification model, m enrollment utterances spoken by the respective user; for each enrollment utterance of the m enrollment utterances, generating, as output from the speaker identification model, a respective enrollment ad-vector having ne style classes; and combining a superset of the ne style classes of the enrollment ad-vectors generated as output from the speaker identification model for the m enrollment utterances into the reference ad-vector. In these implementations, when generating the at least one multi-condition attention score that indicates the likelihood that the evaluation ad-vector matches the respective reference ad-vector associated with the respective user, the self-attention mechanism may automatically align the style classes among the evaluation ad-vector and the multiple reference ad-vectors.

Another aspect of the disclosure provides a system for speaker identification. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform any of the above described method implementations.

In a speech-enabled environment, such as a home, automobile, workplace, or school, a user may speak a query or command and a digital assistant may answer the query and/or cause the command to be performed. Such a speech-enabled environment may be implemented using a network of connected microphone devices distributed through various rooms or areas of the environment. Through the network of microphones, a user can query the digital assistant through a spoken utterance. without having to have a computer or other interface in front of them. In some instances, the speech-enabled environment is associated with multiple enrolled users, e.g., people who live in a household. These instances may apply when a single device is shared by multiple users such as a smart phone, smart speaker, smart display, tablet device, smart television, smart appliance, vehicle infotainment system, etc. Here, the speech-enabled environment may have a limited number of users, e.g., between two and six people in a speech-enabled home, office, or automobile As such, it is desirable to determine an identity of a particular user that is speaking the query. The process of determining the identities of particular speakers/users may be referred to as speaker verification, speaker recognition, speaker identification. or voice recognition.

Speaker verification/identification may allow a user to issue queries that act on behalf of the particular user and/or trigger personalized responses in multi-user environments. Speaker verification/identification (e. g, voice authentication) provides an easy way for a user of a user device to gain access to the user device. For instance, a user may unlock and access the user device by speaking an utterance without requiring the user to manually enter (e. g, via typing or speaking) a passcode to gain access to the user device.

In some scenarios, a user queries the digital assistant that relates to personal information of the user and/or requires access to a resource from a set of personal resources associated with the user. For instance, a particular user (e.g., one who is enrolled with the digital assistant) might ask the digital assistant "when is my meeting with Matt" or query the digital assistant "play my music playlist". Here, the user may be one of one or more multiple enrolled users who each have permission to access their own respective set of personal resources (e. g, calendar, music player, email, messaging, contact list, etc ) and are restricted from accessing the personal resources of the other enrolled users. For instance, if John and Meg are both enrolled users of the digital assistant, the digital assistant would need to discern whether John or Meg spoke the utterance "when is my meeting with Matt" in order to access the appropriate enrolled user's calendar to determine when a meeting is scheduled with Matt and respond with meeting details for the scheduled meeting with Matt Similarly, the digital assistant would need to discern which one of John or Meg spoke the utterance "play my music playlist" in order to access a music player and ultimately audibly output tracks from the appropriate music playlist since John and Meg have unique music playlists.

To determine which user is speaking in a multiuser, speech-enabled environment, speech-enabled systems may include speaker identification systems (e.g., speaker verification systems or voice authentication systems). A speaker verification system may employ a speaker identification (SID) model trained to extract an evaluation vector from audio data corresponding an utterance spoken by a particular user, whereby the extracted evaluation vector represents voice characteristics of the particular user. The evaluation vector may include a d-vector. The SID model may be a neural network model trained under machine or human supervision to output d-vectors. To resolve the identity of the particular user, a comparer determines whether the evaluation vector matches any reference vectors for enrolled and/or authorized users of the user device. Here, each reference vector may correspond to a voiceprint or unique identifier representing characteristics of the voice of the respective enrolled/authorized user. Each enrolled and/or authorized user may speak multiple enrollment utterances. and for each enrollment phrase, the SID model may generate a corresponding reference d-vector that may be combined, e.g., averaged or otherwise accumulated, to form the respective reference d-vector for the enrolled/authorized user.

A conventional technique for determining whether an evaluation d-vector matches a reference d-vector includes computing a cosine similarity score that represents a cosine distance between the evaluation and reference d-vectors. When computing cosine similarity scores the reference d-vector is not conditioned on the evaluation d-vector. Moreover, since the reference d-vector includes an average d-vector based on a combination of multiple enrollment utterances. useful speaker-related information may be lost during the combination.

To address some of the aforementioned shortcomings with using conventional d-vectors for speaker identification tasks, implementations herein are directed toward extracting an evaluation attentive d-vector (ad-vector) to represent voice characteristics of a captured utterance and leveraging a soft-attention mechanism to apply an attentive scoring function that computes a multi-condition attentive score (MCAS) between the evaluation ad-vector and a reference ad-vector associated with a respective authorized/enrolled user Similar to cosine similarity scores, the MCAS indicates a likelihood that the evaluation ad-vector matches the reference ad-vector associated with the authorized/enrolled user. For instance, the MCAS score may range from -<NUM> to <NUM> where an MCAS equal to <NUM> is indicative of a perfect match between the evaluation and reference ad-vectors. However, by contrast to conventional techniques that compute cosine similarity scores between evaluation and reference d-vectors, the soft attention mechanism conditions the reference ad-vector to the evaluation ad-vector when computing the MCAS in a similar way that Transformers condition weights to network inputs. Moreover, the soft attention mechanism may further condition a concatenation reference ad-vectors extracted from multiple enrollment utterances to the evaluation ad-vector when computing the MCAS score such that useful speaker-related information contained in each of the reference ad-vectors is preserved when computing the MCAS for identifying the speaker of the utterance.

Described in greater detail below, an ad-vector includes n component (style) classes that each include a respective value vector. concatenated with a corresponding routing vector. The value vector of each component (style) class may represent different speaker related information than the speaker related information represented by the value vectors of the other component (style) classes in each ad-vector The routing vectors convey environmental/channel and/or contextual information associated with audio data that corresponds to the utterance the ad-vector was generated from. The routing vectors permit the soft-attention mechanism to identify matching conditions between the evaluation and reference ad-vectors. As will become apparent, the respective routing vector is specific to its component (style) class and weights how important the respective value vector is for the style class when the attention mechanism computes the MCAS. That is, the routing vectors enable the attention mechanism to automatically align the component (style) classes among the evaluation and reference ad-vectors.

In speaker identification systems, it is desirable to condition a user profile and scoring between evaluation and reference speaker vectors to attributes from the utterances the speaker vectors were derived from such as audio length, near-vs-far field audio, noise conditions, and other attributes that may be helpful. In conventional speaker identification systems that perform speaker identification via cosine similarity scores between evaluation and reference d-vectors, the aforementioned attributes may be modeled by pre-defining a set of context classes passed as a side input to the speaker identification model However, this technique requires pre-defined granularity that may result in the propagation of many problems rendering the technique infeasible at scale. For instance, the pre-defined granularity may require using extra enrollment utterances to cover all the different context classes. managing multiple profiles, and mapping each evaluation utterance to the appropriate context class. Notably, the use of the ad-vectors and attentive scoring function applied by the soft-attention mechanism described herein addresses these issues by effectively learning context classes that represent the desirable attributes without ever predefining the context classes. That is, the number n of component (style) classes learned and represented by an ad-vector may be specified as a hyper parameter such that the component (style) classes are data driven and not manually defined Each context class may be learned during training of the SID model such that each context class is represented by a respective one of the n style classes. The routing vectors specific to the n component (style) classes permit the soft-attention mechanism to automatically align the component (style) classes among the evaluation and reference ad-vectors.

Referring to <FIG>, in some implementations, an example speech-enabled environment includes a user device <NUM> associated with one or more users <NUM> and in communication with a remote system <NUM> via a network <NUM> The user device <NUM> may correspond to a computing device, such as a mobile phone, computer (laptop or desktop), tablet, smart speaker/display, smart appliance, smart headphones, wearable, vehicle infotainment system, etc, and is equipped with data processing hardware <NUM> and memory hardware <NUM>. The user device <NUM> includes or is in communication with one or more microphones <NUM> for capturing utterances from the respective user <NUM>. The remote system <NUM> may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic computing resources <NUM> (e. g, data processing hardware) and/or storage resources <NUM> (e.g., memory hardware).

The user device <NUM> may include hotword detector (not shown) configured to detect the presence of a hotword in streaming audio <NUM> without performing semantic analysis or speech recognition processing on the streaming audio <NUM>. The user device <NUM> may include an acoustic feature extractor (not shown) which may be implemented as part of the hotword detector or as a separate component for extracting audio data <NUM> from utterances <NUM>. For instance, the acoustic feature extractor may receive streaming audio <NUM> captured by the one or more microphones <NUM> of the user device <NUM> that corresponds to an utterance <NUM> spoken by the user <NUM> and extract the audio data <NUM>. The audio data <NUM> may include acoustic features such as Mel-frequency cepstrum coefficients (MFCCs) or filter bank energies computed over windows of an audio signal In the example shown, the utterance <NUM> spoken by the user <NUM> includes "Ok Google, Play my music playlist".

The hotword detector may receive the audio data <NUM> to determine whether the utterance. <NUM> includes a particular hotword (e. g , Ok Google) spoken by the user <NUM>. That is, the hotword detector <NUM> may be trained to detect the presence of the hotword (e.g., Ok Google) or one or more variants of the hotword (e. g, Hey Google) in the audio data <NUM> to cause the user device <NUM> to wake-up from a sleep state or hibernation state and trigger an automated speech recognition (ASR) system <NUM> to perform speech recognition on the hotword and/or one or more other terms that follow the hotword, eg. , a voice query that follows the hotword and specified an action to perform. In the example shown, the query following the hotword in utterance <NUM> captured in the streaming audio includes "Play my music playlist" that specifies an action for the digital assistant to access a music playlist associated with a particular user (e. g , John) 10a and provide a response <NUM> including an audio track from Jolin's music playlist for the user device <NUM> (and/or one or more designated audio output devices) to playback for audible output from a speaker. Hotwords may be useful for "always on" systems that may potentially pick up sounds that are not directed toward the speech-enabled user device <NUM>. For example, the use of hotwords may help the device <NUM> discern when a given utterance. <NUM> is directed at the device <NUM>, as opposed to an utterance that is directed to another individual present in the environment or a background utterance.

The speech-enabled environment <NUM> includes a speaker identification (SID) system <NUM> that is configured to determine an identity of the user <NUM> that is speaking the utterance <NUM> by processing the audio data <NUM>. The SID system <NUM> may determine whether the identified user <NUM> is an authorized user such that the query is only fulfilled (e.g., the action specified by the query is performed) if the user is identified as an authorized user. Advantageously, the SID system <NUM> allows the user to unlock and access the user's device <NUM> by speaking the utterance without requiring the user to manually enter (e.g., via typing) or speak a passcode or provide some other means of verification (e.g., answer a challenge question, provide biometric verification data, etc.) to gain access to the user device <NUM>.

In some examples, the speech-enabled environment <NUM> includes a multi-user, speech-enabled environment in which multiple different users <NUM>, 10a-n are each enrolled with the user device <NUM> and have permission to access a respective set of personal resources (e.g., calendar, music player, email, messaging, contact list, etc.) associated with that user Enrolled users <NUM> are restricted from accessing personal resources from the respective sets of personal resources associated with the other enrolled users. Each enrolled user <NUM> may have a respective user profile that links to the respective set of personal resources associated with that user, as well as other pertinent information (e. g , user-specified preference settings) associated with that user <NUM> Accordingly, the SID system <NUM> may be used to determine which user is speaking the utterance <NUM> in the multiuser, speech-enabled environment <NUM>. For instance, in the example shown, John and Meg may both be enrolled users <NUM> of the user device <NUM> (or digital assistant interface running on the user device), and the digital assistant needs to discern whether John or Meg spoke the utterance <NUM> "Ok Google, Play my music playlist" in order to access a music player and ultimately audibly output tracks from the appropriate music playlist since Meg and John may each have unique music playlists. Here, the SID system <NUM> processes the audio data <NUM> corresponding to the utterance <NUM> to identify that John was the speaker of the utterance.

In the example shown, the SID system <NUM> includes a SID model <NUM>, a self-attention mechanism <NUM>, and a verifier <NUM> The SID model <NUM> is configured to process audio data <NUM> to generate an attentive d-vector <NUM> For instance, the SID <NUM> receives, as input, the audio data <NUM> corresponding to the utterance <NUM> and generates, as output, an evaluation ad-vector <NUM>, 200B that represents the voice characteristics of the utterance. <NUM> captured by the user device <NUM>. The soft-attention mechanism <NUM> is configured to apply an attentive scoring function that computes one or more multi-condition attentive scores (MCAS) <NUM> each indicating a likelihood that the evaluation ad-vector 200E matches a respective one of one or more reference ad-vectors <NUM>, 200Ra-Rn. Here, each reference ad-vector 200R is associated with a respective one of the one or more enrolled users <NUM> of the user device <NUM>.

<FIG> shows an example ad-vector <NUM> that includes n style classes <NUM>, 202a-n each including a respective value vector (V<NUM>-Vn) <NUM>, 220a-n concatenated with a corresponding routing vector (R<NUM>-Rn) <NUM>, 210a-n. Each value vector <NUM> includes a same first dimensionality dv and each routing vector <NUM> includes a same second dimensionality dr that is less than the first dimensionality dv of each value vector. In some examples, a size of the second dimensionality dr is one-third (<NUM>/<NUM>) a size of the first dimensionality dv.

For reference (enrollment) ad-vectors 200R, the routing vectors <NUM> correspond to keys, ne denotes the number of component (style) classes <NUM>, a matrix K represents a concatenation of the ne routing (key) vectors (R<NUM>-Rn) 210a-n, and a matrix E encompasses all the ne value vectors (V<NUM>-Vn) 220a-n. For evaluation (test) ad-vectors 200E, the routing vectors <NUM> correspond to queries, nt denotes the number of component (style) classes <NUM>, a matrix Q represents a concatenation of the nt routing (key) vectors (R<NUM>-Rn) 210a-n, and a matrix T encompasses all the nt value vectors (V<NUM>- Vn) 220a-n As described in greater detail below, the number of style classes nt, ne can be potentially different.

The respective value vector <NUM> of each component (style) class <NUM> of the n component (style) classes contains respective speaker-related information (e.g., speaker-related phonetic components such as vowels, consonants, and/or fricatives. The respective routing vector <NUM> of each component (style) class <NUM> includes environmental, channel, and/or contextual information associated with the captured utterance the respective ad-vector <NUM> was extracted from. The routing vectors <NUM> are configured to identify matching conditions between an utterance associated with an evaluation ad-vector 200E and at least one reference utterance (eg. , enrollment utterance) associated with a reference ad-vector 200R to allow the soft-attention mechanism <NUM> to condition the reference ad-vector 200R to the ad-vector 200E when computing the MCAS <NUM>. In one example, the routing vectors <NUM> could weight value vectors <NUM> when the evaluation and reference ad-vectors are both derived from utterances captured by a same type of user device and/or user devices executing the same type of operating system.

In some examples, the ad-vector <NUM> is a text-dependent ad-vector <NUM> extracted from audio data characterizing one or more particular terms, such as, for example, a predefined hotword. For instance, the evaluation ad-vector 200E may be extracted from the portion of the audio data <NUM> that characterizes the predetermined hotword "Hey Google" spoken by the user. Likewise, each reference ad-vector 200R may be extracted from a portion of one or more enrollment utterances spoken by the respective enrolled user <NUM> that characterizes the same predetermined hotword "Hey Google" That is, each reference ad-vector corresponds to a voiceprint or unique identifier representing characteristics of the voice of the respective enrolled user <NUM> speaking the predetermined hotword.

In other examples, the ad-vector <NUM> is a text-independent ad-vector that is generated from an utterance independent of the terms/text spoken in the utterance. For instance, a text-independent evaluation ad-vector 200E may be extracted from the query portion of the audio data <NUM> that characterizes the query "Play my music playlist" spoken by the user. The hotword portion of the audio data <NUM> may further contribute to the text-independent evaluation ad-vector 200E such that the evaluation ad-vector 200E represents voice characteristics of the entire utterance <NUM>. Likewise, a text-independent reference ad-vector 200R may be generated from one or more enrollment utterances spoken by the respective enrolled user <NUM> during a voice enrollment process. The text-independent reference ad-vector 200R may correspond to a voiceprint or unique identifier representing characteristics of the voice of the respective enrolled user independent of the text/terms spoken in the enrollment utterances.

Notably, the ad-vector <NUM> may also be configured to represent both text-independent and text-dependent speaker-related information. For example, the SID model <NUM> may learn to generate an ad-vector with n component (style) classes <NUM> whereby at least one of the component (style) classes <NUM> is dependent on a fixed term or phrase such that the respective value vector <NUM> in this component (style) class includes speaker-related information representative of voice characteristics of the spoken fixed term or phrase. For instance, the fixed term or phrase may include a predefined hotword. During training, training utterances used to train the SID model <NUM> may include a portion characterizing the fixed term or phrase such that the SID model <NUM> learns to depend at least one of the n component (style) classes <NUM> on the fixed term or phrase.

The use of ad-vectors <NUM> is not limited to speaker identification tasks, and may be employed in other technologies where evaluation embeddings are compared to reference embeddings. For instance, ad-vectors <NUM> may be used in image-based tasks such as face identification where an evaluation ad-vector 200E includes n style classes <NUM> with corresponding value vectors <NUM> each representing respective facial feature-related information extracted from an image of an individual's face is compared to a reference ad-vector 200R to determine whether or not a facial identification match is made by computing a similarity score (e.g., MCAS) between the evaluation and reference ad-vectors 200E, 200R.

Referring back to <FIG>, in some implementations, each enrolled user <NUM> of the user device <NUM> has permissions for accessing a different respective set of personal resources, where performance of a query characterized by a portion of the audio data <NUM> requires access to the respective set of personal resources associated with the enrolled user <NUM> identified as the speaker of the utterance <NUM>. Here, each enrolled user <NUM> of the user device <NUM> may undertake a voice enrollment process to obtain respective enrolled user reference ad-vectors 200R from audio samples of multiple enrollment phrases spoken by the enrolled user <NUM>. One or more of the enrolled users <NUM> may use the user device <NUM> to conduct the voice enrollment process, where the microphone <NUM> captures the audio samples of these users speaking the enrollment utterances and the SID model <NUM> generates the respective reference ad-vectors 200R Additionally, one or more of the enrolled users <NUM> may enroll with the user device <NUM> by providing authorization and authentication credentials to an existing user account with the user device <NUM>. Here, the existing user account may store a reference ad-vector 200E obtained from a previous voice enrollment process conducted by a respective user with another device also linked to the user account.

The reference ad-vectors 200R are not limited to being obtained from the enrolled users <NUM> explicitly undertaking an enrollment process and in which the users <NUM> are prompted to speak predefined enrollment phrases. For instance, a reference ad-vector 200R may be extracted from one or more audio samples of the respective enrolled user <NUM> speaking a predetermined term such as the hotword (e. g, "Ok Google" or "Hey Google") used for invoking the user device to wake up from a sleep state during previous interactions with the user device <NUM> or another user device <NUM> linked to an existing user account. Similarly, the reference ad-vector 200R for an enrolled user <NUM> may be obtained from one or more audio samples of the respective enrolled user <NUM> speaking phrases with different terms/words and of different lengths. For instance, the reference ad-vector 200R may be obtained over time from audio samples obtained from speech interactions the user <NUM> has with the user device <NUM> or other devices linked to the same account. In other words, the reference ad-vector 200R may be generated by the SID model <NUM> in response to receiving one or more previous utterances spoken by the enrolled user <NUM> of the user device <NUM> and/or other devices linked to the same account. In some examples, an enrolled user <NUM> uses one user device to capture a first set of one or enrollment utterances and then uses another user device to capture a second set of one or more enrollment utterances.

By contrast to conventional reference d-vectors which are not of the attentive type and are representative of an average of the speaker-related information extracted multiple enrollment phrases spoken by a particular speaker, <FIG> shows that the reference ad-vector 200R generated for each respective enrolled user can include a concatenation of multiple ad-vectors 200A-M each generated from a respective one of multiple M enrollment utterances A-M (or other previous utterances obtained from previous interactions) spoken by the enrolled user Here, each individual one of the multiple ad-vectors includes respective n component (style) classes <NUM>, 202a-n each including a respective value vector <NUM>, 220Aa-220Mn concatenated with a corresponding routing vector 210Aa-Mn. Each style class <NUM> may be interchangeably referred to as a "component class", such that 202a denotes a first component class, 202b denotes a second component class,. , and 202n denotes an nth component class. Accordingly, the reference ad-vector 200R combines a superset of component (style) classes <NUM> from all of the enrollment utterances A-M by concatenating the multiple ad-vectors into a single, much larger reference ad-vector 200R with M x n component (style) classes. For each enrollment utterance from the respective enrolled speaker, a respective matrix Ki may be generated that represents a concatenation of the n routing vectors and a respective matrix Ei may be generated that encompasses the n value vectors for the n component (style) classes. Accordingly, the concatenation of the multiple ad-vectors 200A-M into the single, much longer reference ad-vector R with M x n components may be expressed as follows. <MAT>
<MAT>.

As such, the reference ad-vector 200R may include ne component (style) classes while the evaluation ad-vector 200E includes less nt component (style) classes. Notably, the reference ad-vector 200R including the ne component (style) classes <NUM> retains all the speaker-related information extracted from the multiple M enrollment utterances, and as will be described in greater detail below, the soft-attention mechanism <NUM> is conditioned on both the evaluation ad-vector 200E having nt component (style) classes <NUM> and the reference ad-vector 200R having the greater number of ne component (style) classes <NUM> when applying the attentive scoring function to compute the MCAS <NUM>. As such, the reference ad-vector 200R permits the combining of multiple enrollment utterances by combining the superset of n components from all the ad-vectors 200R into the reference ad-vector 200R such that the reference ad-vector 200R is not representative of an average of the individual ad-vectors.

Referring to <FIG>, <FIG>, and <FIG>, in some implementations, the soft-attention mechanism <NUM> applies the attentive scoring function for computing the MCAS scores <NUM> by providing a soft alignment among the nt value vectors <NUM> in the evaluation ad-vector 200E and the nr value vectors <NUM> in the reference ad-vector 200R that is conditioned to the utterance <NUM> and the enrollment utterances. Simply put, the attentive scoring function may define a process of generating a single similarity number, i.e., the MCAS <NUM>, from two independent ad-vectors <NUM>, i.e., the evaluation and reference ad-vectors 200E, 200R. While examples below describe particular implementations of the soft-attention mechanism <NUM> computing MCAS <NUM>, the soft-attention mechanism <NUM> is not limited to computing MCAS <NUM> and may compute other types of metrics/scores for indicating similarity between the evaluation ad-vector 200E and respective ones of one or more reference ad-vectors <NUM>, 200Ra-Rn. As the soft-attention mechanism <NUM> closely resembles the attention mechanism similar to what is used by a Transformer, the soft-attention mechanism <NUM> may compute the MCAS as follows:
<MAT>
<MAT>
where Equation (<NUM>) represents the classical equation for attention weights α, similar to what is used in a Transformer, that includes a matrix with values between <NUM> and <NUM> to provide the soft alignment among the evaluation and reference value vectors <NUM>. That is, the attention weights α are fully defined the matrix K, representing the concatenation of the ne routing (key) vectors <NUM>, and the matrix Q, representing the concatenation of the nt routing (query) vectors <NUM>. The denominator <MAT> in Equation (<NUM>) is a constant and controls how much the sofmax is comparable to an argmmax operation according to a hyperparameter P In the attention weight matrix represented by Equation (<NUM>), each column among nt columns goes over its own softmax transformation. In Equation (<NUM>), the operation tr(T E' α) ∈ R access ne weights from the attention weight matrix α and an inner product is computed between each value vector <NUM> in the evaluation ad-vector 200E and the α-weighted average of the matrix E which represents the concatenation of the nt value vectors <NUM> in the evaluation ad-vector 200E. This process repeats for each of the remaining value vectors <NUM> among the nt value vectors <NUM> in the evaluation ad-vector 200E where the results are summed to provide the final MCAS <NUM> via the trace tr operation.

Notably, the soft-attention mechanism <NUM> using Equation <NUM> to compute the MCAS <NUM> results in different weight matrices α for different keys and value vectors. In other words, the weights of the attention mechanism <NUM> are conditioned to the input ad-vectors 200E, 200R, thereby permitting the weighted average of the reference value vectors <NUM> in the reference ad-vector 200R to be conditioned on both the evaluation (test) and reference (evaluation) utterances that the respective evaluation and reference ad-vectors 200E, 200R were extracted from. Thus, when generating each MCAS <NUM>, the self-attention mechanism <NUM> is conditioned on both the input evaluation and reference ad-vectors 200E, 200R and the reference ad-vector 200R is conditioned to the evaluation ad-vector 200E.

Additionally, as the number "n" of component (style) classes <NUM> represented by the ad-vectors <NUM> is arbitrary, the MCAS is well-suited to model multivariate distributions on the vector space of v which is not possible using conventional cosine distance scoring. The attentive scoring function when computing the MCAS doesn't require additional parameters that could be somehow described as being part of a Siamese topology, which is optimal for production releases of the SID system <NUM> in a distributed system.

With continued reference to <FIG>, the verifier <NUM> of the SID system <NUM> is configured to receive each MCAS <NUM> output from the soft-attention mechanism <NUM> and identify the speaker of the utterance <NUM> as the respective enrolled user <NUM> that is associated with the respective reference ad-vector 200R corresponding to the greatest MCAS <NUM>. In some examples, the verifier <NUM> compares each MCAS <NUM> to a threshold score such that the speaker of the utterance <NUM> is only identified as one of the respective enrolled users <NUM> when the respective reference ad-vector 200R associated with that user <NUM> satisfies the threshold score. Each MCAS <NUM> may include a value within a range of values between and including -<NUM> to <NUM>, whereby an MCAS <NUM> equal to <NUM> is indicative of a perfect match between the evaluation and reference ad-vectors 200E, 200R.

In the example shown, when the verifier <NUM> identifies the speaker of the utterance <NUM>, the verifier <NUM> may provide a SID confirmation <NUM> to the ASR system <NUM> that identifies the speaker of the utterance <NUM> as the respective enrolled user <NUM> associated with the MCAS <NUM> that satisfied the confidence threshold. The SID confirmation <NUM>, when received by the ASR system <NUM>, may instruct the ASR system <NUM> to initiate performance of the action specified by the query. In the example shown, the ASR system <NUM> may include an ASR model (not shown) that configured to perform speech recognition on the audio data <NUM> that characterizes the query. The ASR system <NUM> may also includes a natural language understanding (NLU) module configured to perform query interpretation on the speech recognition result output by the ASR model. Generally, the NLU module may perform semantic analysis on the speech recognition result to identify the action to perform that is specified by the query. In the example shown, the NLU module may determine that performance of the action specified by the query "Play my music playlist", requires access to a respective set of personal resources associated with a respective enrolled user <NUM> of the user device <NUM>. Thus, the NLU module determines that the action specified by the query is missing a necessary parameter, i.e., the identity of the user, needed to perform the action. Accordingly, the NLU module uses the SID confirmation <NUM> identifying a particular enrolled user (e.g., John) 10a as the speaker of the utterance <NUM>, and therefore initiates fulfillment of the query by providing an output instruction <NUM> to perform the action specified by the query. In the example shown, the output instruction <NUM> may instruct a music streaming service to stream a music track from the enrolled user John's music playlist. The digital assistant interface may provide the response <NUM> to the query that includes the music track for audible output from the user device <NUM> and/or one or more other devices in communication with the user device <NUM>.

<FIG> shows an example SID training process <NUM> for training the SID system <NUM>. The training process <NUM> may execute on the remote system <NUM> of <FIG>. The training process <NUM> obtains one or more training data sets <NUM> stored in data storage <NUM> I and trains the SID model <NUM> on the training data sets <NUM>. The data storage <NUM> may reside on the memory hardware <NUM> of the remote system <NUM>. Each training data set <NUM> includes a plurality of training utterances <NUM>, 320a-n spoken by different speakers. Each corresponding training utterance <NUM> may include a text-dependent portion <NUM> and a text-independent portion <NUM>. The text-dependent portion <NUM> includes an audio segment characterizing a predetermined word (e.g., "Hey Google") or a variant of the predetermined hotword (e.g., "Ok Google") spoken in the training utterance <NUM>. In additional implementations, the text-dependent portion <NUM> in some training utterances <NUM> includes audio segments characterizing other terms/phrases lieu of the predetermined word or variant thereof, such as custom hotwords or commonly used voice commands (e.g., play, pause, volume up/down, call, message, navigate/directions to, etc.). The text-dependent portion <NUM> is optional such that only a portion of the training utterances <NUM> may include the text-dependent portion or none of the training utterances <NUM> may include the text-dependent portion <NUM>.

The text-independent portion <NUM> in each training utterance <NUM> includes an audio segment that characterizes a query statement spoken in the training utterance <NUM> following the predetermined word characterized by the text-dependent portion <NUM>. For instance, the corresponding training utterance <NUM> may include "Ok Google, What is the weather outside?" whereby the text-dependent portion <NUM> characterizes the predetermined "Ok Google" and the text-independent portion <NUM> characterizes the query statement "What is the weather outside". While the text-dependent portions <NUM> in each training utterance <NUM> is phonetically constrained by the same predetermined word or variation thereof, the lexicon of the query statement characterized by each text-independent portion <NUM> is not constrained such that the duration and phonemes associated with each query statement is variable.

With continued reference to <FIG>, the training process <NUM> trains a neural network <NUM> on the training utterances <NUM>, 320a-n to generate a respective ad-vector <NUM> for each utterance <NUM> During training, additional information about each utterance <NUM> may be provided as input to the neural network <NUM>. For instance, SID targets <NUM>, such as SID target vectors, corresponding to ground-truth output labels for training the SID model <NUM> to learn how to predict may be provided as input to the neural network <NUM> during training with the utterances <NUM>. Thus, one or more utterances <NUM> from each particular speaker may be paired with a particular SID target vector <NUM>.

The neural network <NUM> may include a deep neural network including an input layer for inputting the training utterances <NUM>, multiple hidden layers for processing the training utterances, and an output layer <NUM>. In some examples, the neural network <NUM> generates an ad-vector <NUM> directly from an input training utterance <NUM> received by the neural network <NUM>. In these examples, each respective set among n sets of output nodes <NUM> of the output layer <NUM> is designated to generate a respective one of the n component (style) classes <NUM> for each input training utterance <NUM>. That is the number of n sets of output nodes <NUM> may be equal to the n number of component (style) classes specified for the ad-vectors <NUM>. That is, each respective set of output nodes <NUM> is configured to generate a respective value vector <NUM> specific to the respective component (style) class that is also concatenated with a corresponding routing vector <NUM> specific to the component (style) class. Stated differently, each set of output nodes in the n multiple sets of output nodes of the output layer <NUM> is designated to learn to generate speaker-related information specific to the respective component (style) class <NUM> of the ad-vector <NUM>. As mentioned previously, at least one component (style) class may be dependent on a fixed term or phrase, such as the text-dependent portion <NUM> of a training utterance. Employing the neural network <NUM> to generate the ad-vectors <NUM> as output from the output layer <NUM> may require enlarging the output layer <NUM> to include a greater number of output nodes <NUM> compared to if the output layer <NUM> were generating a conventional d-vector of the non-attentive type.

In other examples, a set of linear and non-linear transformations <NUM> is applied to the output of the neural network <NUM> to generate the ad-vector <NUM>. In these examples, the neural network <NUM> generates, as output from the output layer <NUM>, a conventional non-attentive d-vector and the set of linear and non-linear transformatinos <NUM> is applied to transform the non-attentive d-vector into the ad-vector <NUM> representing the concatenation of the n component (style) classes. For instance, using x ∈ R a to represent the non-attentive d-vector with a dimensions, the matrices K, Q, E, T may be computed as follows to transform the non-attentive d-vector into the ad-vector <NUM>:
<MAT>
<MAT>
<MAT>
<MAT>
where g() is a transformation function, typically and L2 normalization required to make the training process stable. With respect to the K and Q matrices, the function g() is not needed since the softmax function in Equation (<NUM>) already performs the scale normalization.

Training of the SID model <NUM> may begin by providing sequence of training utterances <NUM> to the neural network <NUM>. In some examples, the neural network <NUM> is trained using a pair-wise training technique in which a first training utterance 320a paired with a particular SID target vector <NUM> is input to the neural network <NUM> and processed to generate a respective first ad-vector 200a. Subsequently, a second training utterance <NUM> paired with a particular SID target vector is input to the neural network <NUM> and processed to generate a respective second as-vector 200b. The soft-attention mechanism <NUM> then compares the first and second ad-vectors 200a, 200b to determine whether or not the first and second ad-vectors 200a, 200b were derived from training utterances 320a, 320b that were uttered by the same speaker. As described above, the soft attention mechanism <NUM> may compute an MCAS <NUM> indicating a likelihood that the first and second ad-vectors 200a, 200b match one another. The MCAS <NUM> output from the soft-attention mechanism <NUM> provides an indication of whether the training utterances 320a, 320b were spoken by the same speaker. In one example, the MCAS <NUM> may simply include a binary value of '<NUM>' or a '<NUM>' in which the '<NUM>' indicates the utterance were spoken by different speakers and the '<NUM>' indicates the utterances were spoken by the same speaker. The parameters of the neural network <NUM> may then be adjusted based on the MCAS <NUM>. Multiple sets of training data may be processed in this manner. Once the SID model <NUM> is trained, the remote system <NUM> may transmit a copy of the SID model <NUM> through the network <NUM> to one or more respective user devices such as user device <NUM>. The trained SID model <NUM> may optionally execute on the remote system <NUM>.

<FIG> includes a flowchart of an example arrangement of operations for a method <NUM> of speaker identification. At operation <NUM>, the method <NUM> includes receiving audio data <NUM> corresponding to an utterance <NUM> captured by a user device <NUM>. At operation <NUM>, the method <NUM> includes processing, using a speaker identification model <NUM>, the audio data <NUM> to generate an evaluation attentive d-vector (ad-vector) 200E that represents voice characteristics of the utterance <NUM>. The evaluation ad-vector 200E includes ne component (style) classes <NUM> each including a respective value vector <NUM> concatenated with a corresponding routing vector <NUM>.

At operation <NUM>, the method <NUM> includes generating at least one multi-condition attention score (MCAS) <NUM> that indicates a likelihood that the evaluation ad-vector 200E matches a respective reference ad-vector 200R The reference ad-vector 200R includes nr component (style) classes <NUM> each including a respective value vector <NUM> concatenated with a corresponding routing vector <NUM>. At operation <NUM>, the method <NUM> includes identifying the speaker of the utterance <NUM> as the respective user <NUM> associated with the respective reference ad-vector 200R based on the at least one MCAS <NUM>.

Each of the components <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display <NUM> coupled to high speed interface <NUM>. In other implementations, multiple processors and/or multiple buses may be used, as appropriate. along with multiple memories and types of memory. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank. a group of blade servers, or a multi-processor system).

The memory <NUM> stores information non-transitority within the computing device <NUM> The memory <NUM> may be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). The non-transitory memory <NUM> may be physical devices used to store programs (e.g., sequences of instructions) or data (eg, program state information) on a temporary or permanent basis for use by the computing device <NUM>. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e. g, typically used for firmware, such as boot programs) Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM). static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.

In some implementations, the high-speed controller <NUM> is coupled to the memory <NUM>, the display <NUM> (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports <NUM>, which may accept various expansion cards (not shown) In some implementations, the low-speed controller <NUM> is coupled to the storage device <NUM> and a low-speed expansion port <NUM>. The low-speed expansion port <NUM>, which may include various communication ports (eg. , USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, eg. , through a network adapter.

The non-transitory memory may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device The non-transitory memory may be volatile and/or non-volatile addressable semiconductor memory.

Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e. g, magnetic, magneto optical disks, or optical disks. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, eg. , internal hard disks or removable disks, magneto optical disks; and CD ROM and DVD-ROM disks.

To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e. g, a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user and optionally a keyboard and a pointing device, e. g, a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user, for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

Claim 1:
A computer-implemented method (<NUM>) for speaker identification when executed on data processing hardware (<NUM>, <NUM>) causes the data processing hardware (<NUM>, <NUM>) to perform operations comprising:
receiving audio data (<NUM>) corresponding to an utterance (<NUM>) captured by a user device (<NUM>); said method being characterised by causing the data processing hardware to perform operations further comprising:
processing, using a speaker identification model (<NUM>), the audio data (<NUM>) to generate an evaluation attentive d-vector, ad-vector, (200E) representing voice characteristics of the utterance (<NUM>), the evaluation ad-vector (200E) comprising ne component classes (<NUM>) each comprising a respective value vector (<NUM>) concatenated with a corresponding routing vector (<NUM>), wherein each routing vector (<NUM>) conveys environmental, channel and/or contextual information associated with the audio data (<NUM>);
generating, using a self-attention mechanism (<NUM>), at least one multi-condition attention score (<NUM>) that indicates a likelihood that the evaluation ad-vector (200E) matches a respective reference ad-vector (200R) associated with a respective user (<NUM>), the reference ad-vector (200R) comprising nr component classes (<NUM>) each comprising a respective value vector (<NUM>) concatenated with a corresponding routing vector (<NUM>); and
identifying the speaker of the utterance (<NUM>) as the respective user (<NUM>) associated with the respective reference ad-vector (200R) based on the multi-condition attention score (<NUM>).