Patent Description:
Recently, computing devices that provide multiple user input modalities have become more prevalent. For example, smartphones and other user devices include speech recognition services that allow users to provide voice inputs to a device as an alternative to typing or pointing inputs. Voice-based inputs may be more convenient in some circumstances as a hands-free means for interacting with the computing device. Some devices require that a user's identity be verified before performing an action based upon voice input, in order to guard against breaches of privacy and security. Often, it may be difficult for this verification performed by devices to identify a user with little or limited information (e.g., audio data) about the voice of the user.

<NPL>" presents an end-to-end (E2E) speaker diarization architecture for automatic character recognition via convolutional neural networks and embedding matching procedures.

<NPL> describes techniques that augment utterances, e.g., by cutting, splitting, shuffling, etc..

One aspect of the disclosure provides method of generating an accurate speaker representation for an audio sample. The method includes receiving, at data processing hardware, a first audio sample from a first speaker and a second audio sample from a second speaker. For each audio sample of the first audio sample and the second audio sample, the method includes dividing, by the data processing hardware, the respective audio sample into a plurality of audio slices. For each audio sample of the first audio sample and the second audio sample, the method also includes based on the plurality of slices, generating, by the data processing hardware, a set of candidate acoustic embeddings where each candidate acoustic embedding includes a vector representation of acoustic features. For each audio sample of the first audio sample and the second audio sample, the method further includes removing, by the data processing hardware, a subset of the candidate acoustic embeddings from the set of candidate acoustic embeddings. For each audio sample of the first audio sample and the second audio sample, the method additionally includes generating, by the data processing hardware, an aggregate acoustic embedding from the remaining candidate acoustic embeddings in the set of candidate acoustic embeddings after removing the subset of the candidate acoustic embeddings. In some examples, the method also includes determining, by the data processing hardware, whether the aggregate acoustic embedding generated for the first audio sample from the first speaker corresponds to the aggregate acoustic embedding generated for the second audio sample from the second speaker and when the aggregate acoustic embedding generated for the first audio sample from the speaker corresponds to the aggregate acoustic embedding generated for the second audio sample from the second speaker, identifying, by the data processing hardware, that the first speaker and the second speaker are the same speaker. In some implementations, the method further includes determining, by the data processing hardware, whether a distance between the aggregate acoustic embedding generated for the first audio sample from the first speaker and the aggregate acoustic embedding generated for the second audio sample from the second speaker satisfies a distance threshold and when the distance between the aggregate acoustic embedding generated for the first audio sample from the first speaker and the aggregate acoustic embedding generated for the second audio sample from the second speaker satisfies the distance threshold, identifying, by the data processing hardware, that the first speaker and the second speaker are the same speaker.

Another aspect of the disclosure provides a system of generating an accurate speaker representation for an audio sample. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include receiving a first audio sample from a first speaker and a second audio sample from a second speaker. For each audio sample of the first audio sample and the second audio sample, the operations include dividing the respective audio sample into a plurality of audio slices. For each audio sample of the first audio sample and the second audio sample, the operations also include, based on the plurality of slices, generating a set of candidate acoustic embeddings where each candidate acoustic embedding includes a vector representation of acoustic features. For each audio sample of the first audio sample and the second audio sample, the operations further include removing a subset of the candidate acoustic embeddings from the set of candidate acoustic embeddings. For each audio sample of the first audio sample and the second audio sample, the operations additionally include generating an aggregate acoustic embedding from the remaining candidate acoustic embeddings in the set of candidate acoustic embeddings after removing the subset of the candidate acoustic embeddings. In some examples, the operations also include determining whether the aggregate acoustic embedding generated for the first audio sample from the first speaker corresponds to the aggregate acoustic embedding generated for the second audio sample from the second speaker and when the aggregate acoustic embedding generated for the first audio sample from the speaker corresponds to the aggregate acoustic embedding generated for the second audio sample from the second speaker, identifying that the first speaker and the second speaker are the same speaker. In some implementations, the operations further include determining whether a distance between the aggregate acoustic embedding generated for the first audio sample from the first speaker and the aggregate acoustic embedding generated for the second audio sample from the second speaker satisfies a distance threshold and when the distance between the aggregate acoustic embedding generated for the first audio sample from the first speaker and the aggregate acoustic embedding generated for the second audio sample from the second speaker satisfies the distance threshold, identifying that the first speaker and the second speaker are the same speaker.

Implementations of either the system or the method may include one or more of the following optional features. In some implementations, each candidate acoustic embedding comprises a respective d-vector. In some examples, generating the set of candidate acoustic embeddings based on the plurality of audio slices comprises generating each candidate acoustic embedding in the set of candidate acoustic embeddings by reordering the audio slices in the plurality of audio slices divided from the respective audio sample into an order that is different from the respective audio sample, concatenating the reordered audio slices, and generating the corresponding candidate acoustic embedding based on the concatenation of the reordered audio slices. Here, an order of the audio slices in the concatenation of the reordered audio slices associated with each candidate acoustic embedding is different. In some of these examples, concatenating the reordered audio slices includes determining that the concatenation of the reordered audio slices satisfies a time threshold. In some configurations, generating the set of candidate acoustic embeddings includes generating the set of candidate acoustic embeddings using a neural network acoustic model where the neural network acoustic model configured to receive, as input, audio data and to generate, as output, an acoustic embedding.

In some implementations, removing the subset of the candidate acoustic embeddings from the set of candidate acoustic embeddings includes the following operations. For each candidate acoustic embedding in the set of candidate acoustic embeddings, the operations include determining a distance from the respective candidate acoustic embedding to each other candidate acoustic embedding in the set of candidate acoustic embeddings and generating a distance score for the respective candidate acoustic embedding based on the distances determined from the respective candidate acoustic embedding to each other candidate acoustic embedding of the set of candidate acoustic embeddings. The operations also includes selecting a threshold number of the candidate acoustic embeddings in the set of candidate acoustic embeddings that are associated with the lowest distance score.

In some examples, removing the subset of the candidate acoustic embeddings from the set of candidate acoustic embeddings includes the following operations. For each candidate acoustic embedding in the set of candidate acoustic embeddings, the operations include determining a distance from the respective candidate acoustic embedding to each other candidate acoustic embedding in the set of candidate acoustic embeddings and generating a distance score for the respective candidate acoustic embedding based on the distances determined from the respective candidate acoustic embedding to each other candidate acoustic embedding of the set of candidate acoustic embeddings. The operations also includes selecting each candidate acoustic embedding in the set of candidate acoustic embeddings whose distance score fails to satisfy a distance score threshold.

Generally, speaker identification refers to a process of identifying a speaker based on one or more audio samples. One such form of speaker identification is speaker verification. Speaker verification refers to a process of verifying whether two or more utterances originated from the same speaker. To perform this verification, a speaker identification system compares audio samples (e.g., two audio samples) and determines whether a first audio sample corresponding to a first utterance spoken by a speaker matches or closely resembles a second audio sample corresponding to another spoken utterance. When the first utterance matches or closely resembles the other spoken utterance, the speaker identification system identifies that both utterances are likely from the same speaker. On the other hand, when the first utterance fails to match or to closely resemble the other spoken utterance, the speaker identification system identifies that each utterance is likely from a different speaker. When comparing two audio samples, a speaker identification system may use a vector-based approach or a model-based approach. In a vector-based approach, the speaker identification system compares a first vector for the first audio sample to a second vector for the second audio sample. The vector, which can also be referred to as a d-vector or acoustic embedding, is a vector generated by, or received at, the speaker identification system that represents the acoustic characteristics of the audio sample. To determine whether the speaker of one audio sample is the same as a speaker for another audio sample, the vector-based approach generates a d-vector for each audio sample and compares these d-vectors in order to determine whether each audio sample originates from the same audio source (i.e., from the same speaker). In other words, when the first audio sample has a d-vector that closely resembles the d-vector from the second audio sample, the speaker identification system determines that the similar d-vectors indicate that the audio samples likely originate from the same speaker.

In comparison, the model-based approach inputs the two audio samples into a speaker identification model and uses the model to generate a prediction of whether the speakers from the two audio samples are the same speaker. In other words, the model is trained to identify when two input audio samples are likely to be the same speaker or different speakers. Although the vector-based approach and the model-based approach function to perform speaker identification, both of these approaches share a common setback that either approach is contingent on the quality of the two audio samples provided. For instance, although the model may be trained on a larger corpus of samples, the model predicates its prediction on the ability of the input audio sample to represent the speech characteristics of its corresponding speaker. Likewise, the vector-based approach is confined to how well the vector representation of the audio sample represents the speech characteristics of the speaker. But unfortunately, a particular audio sample may not include the audio characteristics which optimally represent a speaker. For example, if a speaker has a particular British accent, but the speaker's British accent is not as pronounced or distinguishable when the speaker says a particular phrase, an audio sample of the particular phrase may not be a good d-vector representation (e.g., for a vector-based approach) or input audio sample (e.g., for a model-based approach) of the speaker to compare to other spoken phrases (i.e., audio samples) by the speaker. With this in mind, when a speaker identification system performs speaker identification using a single sample audio, the speaker identification system may not always have the best input of audio information to identify a speaker or a similarity between speakers. In fact, a single audio sample is unlikely to be an optimal acoustic representation of the speaker.

To overcome this issue that a particular audio sample may not be an optimal representation of the speaker, a speaker identification system may use a single audio sample to generate multiple variations of the audio sample. By generating multiple variations of the audio sample, there is likely a greater probability that at least one of the many variations of the audio sample accurately represents the speaker. In other words, by having more than one audio sample, the speaker identification system may increase its likelihood that it correctly performs speaker verification. To generate multiple variations from a single audio sample, the speaker identification may use various audio sample augmentation processes.

For a vector-based approach, the audio sample augmentation process generates multiple variations of a single audio sample that, in turn, generates multiple d-vectors for each variation of the single audio sample. With more d-vectors, there is likely a greater probability that at least one of the many d-vectors accurately represents the speaker. To generate multiple d-vectors from a single audio sample, the speaker identification system utilizes the fact that any length of an audio sample may generate a d-vector. For instance, a single d-vector may be generated for a ten minute audio sample or a single d-vector may be generated for a half second (<NUM> second) audio sample. In other words, the generation of the d-vector is irrespective of the length of the audio sample. Therefore, a single audio sample that is three seconds long may form a single d-vector corresponding to the spoken audio during the three seconds or the three second audio sample may be divided into one second (<NUM> second) audio slices and the speaker identification system generates a d-vector for each of the audio slices. This means that, in this example, instead of having a single d-vector with the hope that the single d-vector accurately represents the speech characteristics of the speaker, the speaker identification system has three d-vectors that each may have some degree of accuracy to represent the speech characteristics of the speaker.

When a speaker identification system generates a greater number of d-vectors, the speaker identification system may be configured to use the multiple d-vectors to identify which d-vector or set of d-vectors are the most accurate representation(s) of the speaker. Here, with a greater number of d-vectors or vector samples that represent the speaker of the audio sample, the speaker identification system may compare each of these samples to each other to identify outlier d-vectors that are unlikely to represent the speaker accurately. For instance, if each of the multiple d-vectors accurately represented to speaker, the multiple d-vectors would appear to spatially converge in a dimensional space. In other words, a spatial representation of the multiple d-vectors would illustrate a tight cluster of d-vectors around a theoretical perfect d-vector representation of the speaker. In contrast, a system that only generates a single d-vector from an audio sample for speaker identification is not capable of performing this relative comparison of multiple d-vectors to determine whether the single d-vector is an accurate representation of the speaker. To extend the scenario further, without knowing whether a single d-vector is an accurate representation of the speaker, a speaker identification system may inevitably use a d-vector that poorly represents the speaker to verify the speaker. With this poor representation, there becomes an increased probability that the speaker identification system fails to correctly verify the speaker. When a speaker's identity becomes tied to various permissions or rights, the speaker identification system may incorrectly prevent a speaker from accessing functionality that the speaker should be able to access based on his or her permissions/rights.

For a model-based approach, the audio sample augmentation process preforms spectrogram augmentation on an audio sample to produce several variations of the spectrogram. In other words, since the input to the model is based on the audio sample, the spectrogram augmentation process generates spectrogram variations of the audio sample. Like the vector-based approach, by generating multiple spectrogram variations, the model is able to receive multiple inputs for each audio sample. With multiple inputs rather than a single input corresponding to the audio sample, the model is more likely to be more informed and, thus, to base its prediction on more representations of the speaker of the audio sample. In other words, this approach of multiple inputs per audio sample provides the model with a greater understanding of the speech characteristics for the speaker of the audio sample, which, in turn may result in a better prediction for speaker identification and/or verification.

<FIG> is an example of a speech environment <NUM> that includes one or more users <NUM> communicating a spoken utterance <NUM> to a speech-enabled device <NUM> (also referred to as a device <NUM> or a user device <NUM>). The user <NUM> (i.e., speaker of the utterance <NUM>) may speak the utterance <NUM> as a query or a command to solicit a response from the device <NUM>. The device <NUM> is configured to capture sounds from one or more users <NUM> within the speech environment <NUM>. Here, the audio sounds may refer to a spoken utterance <NUM> by the user <NUM> that functions as an audible query, a command for the device <NUM>, or an audible communication captured by the device <NUM>. Speech-enabled systems of the device <NUM> or associated with the device <NUM> may field the query for the command by answering the query and/or causing the command to be performed.

Here, the device <NUM> is configured to detect utterances <NUM> and to invoke a local or a remote speaker identification process. The device <NUM> may correspond to any computing device associated with the user <NUM> and capable of receiving audio signals corresponding to spoken utterances <NUM>. Some examples of user devices <NUM> include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-book readers, etc.), computers, wearable devices (e.g., smart watches), music player, casting devices, smart appliances (e.g., smart televisions) and internet of things (IoT) devices, remote controls, smart speakers, etc. The device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and storing instructions, that when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations related to utterance detection or some other form of utterance/speech processing (e.g., speech identification and/or speech verification).

In some examples, the device <NUM> includes one or more applications (i.e., software applications) where each application may utilize one or more speech processing systems (e.g., a speech recognition system, a text-to-speech system, a speaker identification system <NUM>, etc.) associated with device <NUM> to perform various functions within the application. In some implementations, the device <NUM> may detect an utterance <NUM> and provide data characterizing the utterance <NUM> to the one or more speech processing systems. For instance, the device <NUM> includes a speech identification application configured to identify the speaker <NUM> of an utterance <NUM>. The speech identification application may perform a speaker verification process that verifies an identity of a speaker <NUM> of the utterance <NUM>. For instance, speaker verification involves accepting or rejecting an identity claim of a speaker <NUM> based on characteristics of the speaker's voice, as determined by one or more utterances <NUM> from the speaker <NUM>. In some examples, the device <NUM> is configured with the application locally to perform local speaker verification or remotely to utilize remote resources to perform some portion of speaker verification.

The device <NUM> further includes an audio subsystem with an audio capturing device (e.g., a microphone) <NUM> for capturing and converting spoken utterances <NUM> within the speech environment <NUM> into electrical signals. While the device <NUM> implements a single audio capturing device <NUM> in the examples shown, the device <NUM> may implement an array of audio capturing devices <NUM> without departing from the scope of the present disclosure, whereby one or more audio capturing devices <NUM> in the array may not physically reside on the device <NUM>, but be in communication with the audio subsystem (e.g., peripherals of the device <NUM>). For example, the device <NUM> may correspond to a vehicle infotainment system that leverages an array of microphones positioned throughout the vehicle. Additionally or alternatively, the device <NUM> also includes a speech output device (e.g., a speaker) <NUM> for communicating an audible audio signal from the device <NUM>. For instance, the device <NUM> is configured to generate a synthesized playback signal in response to a detected utterance <NUM>. In other words, an utterance <NUM> may correspond to a query that the device <NUM> answers with synthesized audio generated by the device <NUM> and communicated via the speech output device <NUM>.

Furthermore, the device <NUM> is configured to communicate via a network <NUM> with a remote system <NUM>. The remote system <NUM> may include remote resources <NUM>, such as remote data processing hardware <NUM> (e.g., remote servers or CPUs) and/or remote memory hardware <NUM> (e.g., remote databases or other storage hardware). The device <NUM> may utilize the remote resources <NUM> to perform various functionality related to speech processing such as speech recognition and/or speaker identification/verification. For instance, the device <NUM> is configured to perform speaker identification using a speaker identification system <NUM>. This system <NUM> may reside on the device <NUM> (referred to as on-device systems) or reside remotely (e.g., reside on the remote system <NUM>), but in communication with the device <NUM>. In some examples, some portions of the system <NUM> reside locally or on-device while others reside remotely. For instance, the verifier <NUM> that is configured to perform speech verification for the speaker identification system <NUM> resides remotely or locally. In some examples, the speaker identification system <NUM> may be combined with other speech processing systems such as speech recognition systems, diarization systems, text-to-speech systems, etc. In some configurations, the location of where the speaker identification system <NUM> resides is based on processing requirements. For example, when the system <NUM> is rather large in size or processing requirements, the system <NUM> may reside in the remote system <NUM>. Yet when the device <NUM> may support the size or the processing requirements of the system <NUM>, the one or more systems <NUM> may reside on the device <NUM> using the data processing hardware <NUM> and/or the memory hardware <NUM>.

The speaker identification system <NUM> is generally configured to process data characterizing an utterance <NUM> and to provide a response <NUM> to the device <NUM> that indicates a result of a speech verification process performed by the verifier <NUM> of the speaker identification system <NUM>. For instance, the speaker identification system <NUM> is the system that performs speech verification for a speech identification application of the device <NUM>. In other words, the speaker identification system <NUM> is configured to perform a speaker verification process using a verifier <NUM> to verify an identity of a speaker <NUM> of the utterance <NUM>. For instance, the response <NUM> may indicate whether a speaker <NUM> is registered with the device <NUM> (i.e., a registered speaker) based on a spoken utterance <NUM> by the speaker <NUM>. In some examples, the speaker identification system <NUM> generates a response <NUM> that identifies the identity of the speaker <NUM> based on a verification process at the verifier <NUM>.

Still referring to <FIG>, the device <NUM> may use the speaker identification system <NUM> to enroll one or more users 10a-c with the device <NUM>. By enrolling a user <NUM> with the device <NUM>, the enrollment serves as a type of speech registration process to identify an enrolled user <NUM>, <NUM>E as an authorized user of the device <NUM>. The device <NUM> may be configurable such that an enrolled user <NUM>E of the device <NUM> may have access to or control of various functions of the device <NUM> while an unauthorized user <NUM> that is not enrolled with the device <NUM> is prohibited from performing one or more functions that the device <NUM> is capable of performing. Optionally, the device <NUM> may enroll multiple users <NUM>. With multiple users <NUM>, each enrolled user <NUM>E may configure a user account on the device <NUM> that has particular permissions or rights regarding functionality of the device <NUM>. For example, the three users 10a-c in <FIG> correspond to a family of three with a husband, wife, and a nine-year old daughter. Here, when each adult enrolls with the device <NUM>, the adults may setup parental controls that allow each adult to access or to control all the functionality of the device <NUM>, but restrict their daughter, who is also an enrolled user <NUM>E, from having entire control of the device <NUM>. For example, the parents setup their daughter's account to prevent their daughter from modifying home automation controls such as the thermostat schedule controlled by the device <NUM>. This means that one enrolled user <NUM>E may have particular permissions or rights that overlap or are different from the permissions or rights of another enrolled user <NUM>E. Moreover, instead of only generating permissions for enrollees, the device <NUM> may also be configured to designate permissions for a user <NUM> of the device <NUM> who is not an enrolled user <NUM>E. For instance, when a user <NUM> of the device <NUM> is not an enrolled user <NUM>E, the device <NUM> may be configured to perform limited functionality (e.g., a guest mode) or to entirely prevent the unenrolled user <NUM> from using the device <NUM>. Without limitation, authorizing an enrolled user <NUM>E may permit the device <NUM> to access resources only that enrolled user <NUM>E has permission to access. For instance, in a household with at least two enrolled users <NUM>E, in which one speaks the voice command "Play my music playlist" captured by the device <NUM>, the verifier <NUM> can identify the identity of the particular enrolled speaker <NUM>E and permit the device <NUM> to access the particular music playlist associated the identified speaker <NUM> and not the music playlists of the other enrolled users <NUM>E.

In some configurations, the device <NUM> uses the speaker identification system <NUM> to perform the enrollment process of enrolling a user <NUM> as a registered speaker for the device <NUM>. For example, a speaker identification application associated with the speaker identification process <NUM> prompts a user <NUM> to speak one or more enrollment utterances <NUM> from which a speaking signature <NUM> can be generated for the user <NUM>. In some implementations, the enrollment utterances <NUM> are short phrases of, for example, one, two, three, four, or more words. The speaker identification system <NUM> may prompt the user <NUM> to speak pre-defined phrases as the enrollment utterances <NUM>, or the user <NUM> may spontaneously speak and provide enrollment utterances <NUM> based on phrases that that were not specifically provided for the user <NUM>. In some examples, the user <NUM> may speak multiple enrollment utterances <NUM> where each enrollment utterance is the same phrase or a different phrase. The enrollment utterances <NUM> could include the user <NUM> speaking a predefined hotword configured to trigger the device <NUM> to wake-up from a sleep state for processing spoken audio received after the predefined hotword. While the example shows the users <NUM> providing the spoken enrollment utterance(s) <NUM> to the device <NUM>, other examples may include one or more of the users <NUM> accessing the speech identification system <NUM> from another device (e.g., a smart phone) to provide the enrollment utterance(s) <NUM>. Upon receiving the enrollment utterances <NUM>, the speaker identification system <NUM> processes the enrollment utterances <NUM> to generate a speaker representation for each enrollment utterance <NUM>. The speaker identification system <NUM> may generate a speaker signature <NUM> for the user <NUM> from all, some, or one of the speaker representations for the enrollment utterances <NUM>. In some examples, the speaker signature <NUM> is an average of the respective speaker representations for the multiple enrollment utterances <NUM>. In other examples, the speaker signature <NUM> corresponds to a particular speaker representation from a particular enrollment utterance <NUM> that is selected based on one or more criteria (e.g., based on an audio or voice quality of the audio for the selected enrollment utterance <NUM>). Once a speaker signature <NUM> is generated for a speaker <NUM>, the speaker signature <NUM> may be stored locally on the device <NUM> or stored in the remote system <NUM> (e.g., in the remote memory hardware <NUM>).

After enrollment, when the device <NUM> detects a query utterance <NUM> by a user <NUM> within the speech environment <NUM>, the speaker identification system <NUM> is configured to identify whether or not the speaker <NUM> of the query utterance <NUM> is an enrolled user <NUM>E of the device <NUM> based on the query utterance <NUM>. A query utterance <NUM> may refer to a special type of utterance or spoken phrase, such as a text-dependent verification phrase, or more generally refer text-independent phrases that may include any utterance <NUM> spoken by a user <NUM> subsequent to the completion of the enrollment process for one or more user <NUM>. Here, a verification process performed by the verifier <NUM> identifies whether the speaker <NUM> of the detected query utterance <NUM> is an enrolled user <NUM>E and generates the response <NUM> to indicate whether or not the speaker <NUM> is an enrolled user <NUM>E. In some examples, the verifier <NUM> has access to speaker signatures <NUM> that have been generated for enrolled users <NUM>E and compares the detected query utterance <NUM> by the speaker <NUM> to the speaker signatures <NUM> to determine whether the query utterance <NUM> corresponds to a particular speaker signature <NUM>. In these examples, when the query utterance <NUM> corresponds to a particular speaker signature <NUM>, the verifier <NUM> determines that the query utterance <NUM> was spoken by an enrolled user <NUM>E and generates a response <NUM> that indicates that the speaker <NUM> of the query utterance <NUM> is an enrolled user <NUM>E.

In some implementations, when the speaker identification system <NUM> generates a response <NUM> that the speaker <NUM> is not an enrolled user <NUM>E, the speaker identification system <NUM> prompts the speaker <NUM> to determine if the user <NUM> wants to become an enrolled user <NUM>E on the device <NUM>. In some configurations, prior to prompting the unenrolled user <NUM> to become an enrolled user <NUM>E, the device <NUM> is configured with criteria, such as security criteria, to ensure that an owner of the device <NUM> has given the unenrolled user <NUM> or guest user permission to become an enrolled user <NUM>E of the device <NUM>. This may prevent anyone from simply enrolling and gaining unwanted control of the device <NUM>.

<FIG> illustrates three users 10a-c that first enrolled with the device <NUM> by performing the enrollment process. In other words, <FIG> depicts at least one enrollment utterance <NUM>, 144a-c being sent to the device <NUM> by each user <NUM> to enroll with the device <NUM>. Following the enrollment process, the third user 10c speaks a query utterance <NUM> to the device <NUM>. When the device <NUM> receives the query utterance <NUM>, the device <NUM> communicates the query utterance <NUM> along with any speaker signatures <NUM> to the speaker identification system <NUM> to enable the verifier <NUM> to verify that the third speaker 10c is an enrolled user <NUM>E. Here, when the verifier <NUM> verifies that the third speaker 10c is indeed an enrolled user <NUM>E, the speaker identification system <NUM> communicates the response <NUM> to the device <NUM> to indicate that the third speaker 10c is an enrolled user <NUM>E with the device <NUM>. Once verified, the third speaker 10c may use the device <NUM> or access some aspect of computing functionality offered by the device <NUM> that an enrolled user <NUM>E has permission to access or, more particularly, designated to the third user 10c. In some examples, the query utterance <NUM> includes a hotword followed by a query. In these examples, the verifier <NUM> may verify the third speaker 10c based on only a portion of the audio corresponding to the hotword, only a portion of the audio corresponding to the query, or the entire audio including both the hotword and the query. In additional examples, the query utterance <NUM> may be a particular verification phrase that an enrolled user <NUM>E provides to the verifier <NUM> to verify the identity of the enrolled user <NUM>E.

<FIG> is an example speech environment <NUM> that uses the speaker identification system <NUM> to identify speakers <NUM> or changes between speakers <NUM> for a diarization system associated with the device <NUM>. Here, the device <NUM> detects a first utterance 12a and then detects a subsequent second utterance 12b and determines whether the first utterance 12a and the second utterance 12b correspond to the same speaker <NUM> or different speakers <NUM>. By differentiating whether an utterance <NUM> is from the same speaker <NUM> or not, the speaker identification system <NUM> is able to assist, for example, a diarization system in identifying a speaker <NUM> during an audio stream received by the device <NUM>. In other words, when the verifier <NUM> verifies that the speaker <NUM> of the second utterance 12b is a different speaker <NUM> than the speaker <NUM> of the first utterance 12b, the verifier <NUM> may generate a response <NUM> to inform the diarization system of this difference. In the example shown, speech systems associated with the device <NUM> may use the response <NUM> from the verifier <NUM> to generate speaker labels when a speech processing system associated with the device <NUM> is generating a transcript for the audio stream. For example, <FIG> illustrates a meeting with six users 10a-f of the device <NUM>. During the meeting, the device <NUM> is being used to generate a transcript for the meeting. As part of the transcript, the transcript includes speaker labels that identify who is speaking what information. By using the speaker identification system <NUM> during the meeting, the device <NUM> is able to use the verifier <NUM> to verify that a label for a given speaker in the transcript should change or should remain the same based on the verifier <NUM> determining whether two subsequent utterances <NUM> are spoken by the same speaker <NUM> or different speakers <NUM>. For instance, <FIG> depicts that a first utterance 12a by the speaker 10a labeled "speaker <NUM>" is followed by a second utterance 12b by the speaker 10c labeled "speaker <NUM>. " Since, the verifier <NUM> confirms that these two utterances 12a-b are from different speakers <NUM> in its response <NUM>, the device <NUM> uses the response <NUM> to indicate that a new speaker label needs to occur in the transcript for the second utterance 12b. In some examples, besides verifying whether the speakers are the same or different, the speaker identification system <NUM> is configured to generate a response <NUM> that includes the identity of the speaker <NUM>. For instance, referring back to <FIG>, if the speaker identification system <NUM> includes speaker signatures <NUM> that are labeled as to the identity of the speaker <NUM>, when an utterance <NUM> corresponds (e.g., matches or closely resembles) to a particular labeled signature <NUM>, the speaker identification system <NUM> may include the labeled identity for that particular signature in its response <NUM>.

To more broadly refer to multiple potential applications of the speaker identification system <NUM>, all types of utterances (e.g., enrollment utterances <NUM>, query utterance <NUM>, or just general speaking utterances <NUM>) and speaker signatures <NUM> may be more generally referred to as audio samples <NUM> (<FIG>). An audio sample refers to any length of audio data provided to the device <NUM> (e.g., by a user <NUM>) or to the speaker identification system <NUM>. For example, an enrollment utterance <NUM> is an audio sample <NUM> spoken by a particular user <NUM> that includes some length of audio data to perform the enrollment process. In some examples, an audio sample <NUM> may correspond to an entire utterance <NUM> spoken by a user <NUM> or some portion of the utterance <NUM> spoken by the user <NUM>. Therefore, the device <NUM> receives the audio sample <NUM> from a user <NUM> of the device <NUM> (e.g., at the audio capturing device <NUM>) and communicates the audio sample <NUM> or some derivative of the audio sample <NUM> to the speaker identification system <NUM>.

Referring to <FIG>, the verifier <NUM> is configured to perform a vector-based approach to speaker verification. In the vector-based approach, the verifier <NUM> includes a variator <NUM>, a generator <NUM>, and a comparator <NUM>. The variator <NUM> is configured to receive an audio sample <NUM> and to generate multiple sample variations <NUM>, 212a-n of the audio sample <NUM>. Each sample variation <NUM> corresponds to a version of the audio sample <NUM> that has undergone some augmentation technique. In some examples, the variator <NUM> uses an augmentation technique that divides the audio sample <NUM> into slices <NUM> such that each slice <NUM> corresponds to a sample variation <NUM>. For example, <FIG> depicts the audio sample <NUM> being divided into four audio slices 214a-d to form four audio sample variations 212a-d.

The generator <NUM> is configured to receive each sample variation <NUM> of the audio sample <NUM> and to generate a speaker representation <NUM> for each sample variation <NUM>. In other words, although the audio sample <NUM> from the speaker <NUM> has undergone some type of augmentation technique at the variator <NUM>, each sample variation <NUM> will still include speech characteristics derived from the audio sample <NUM>. For instance, when the variator <NUM> forms the sample variation <NUM> by dividing the audio sample <NUM> into slices <NUM>, each slice <NUM>, as a subset of the audio sample <NUM>, will include a subset of speech characteristics corresponding to that particular slice <NUM>. In some implementations, such as the vector-based approach, the speaker representation <NUM> generated by the generator <NUM> is an acoustic embedding <NUM> of the sample variation <NUM>. An acoustic embedding <NUM> is a type of speaker representation <NUM> that refers to an n-dimensional vector where each dimension of the vector represents some form of a speech characteristic according to its acoustic features. In other words, the acoustic embedding <NUM> corresponds to a vector representation of speech characteristics for the sample variation <NUM> since the sample variation <NUM> is a derivative of an audio sample <NUM> spoken by a speaker <NUM>. The acoustic embedding <NUM> may include a d-vector. In some configurations, the generator <NUM> generates the acoustic embedding <NUM> by leveraging an acoustic model (AM) of a speech recognition system in communication with the speaker identification system <NUM>. Here, the generator <NUM> may include a version of the AM or communicate sample variations <NUM> to the AM of a speech recognition system in order for the AM to use its model that maps segments of audio (i.e., frames of audio) to phonemes to generate the acoustic embeddings <NUM> for the generator <NUM>.

In some implementations, since the verifier <NUM> performs the verification process on two audio samples 202a-b, the generator <NUM> generates a first set of acoustic embeddings <NUM> for the first audio sample 202a and a second set of acoustic embeddings <NUM> for the second audio sample 202b. In other words, the generator <NUM> generates an acoustic embedding <NUM> for each sample variation <NUM> of the audio sample <NUM> to form a set of acoustic embeddings <NUM> for that particular audio sample <NUM>. With multiple acoustic embeddings <NUM> for each audio sample <NUM>, the comparator <NUM> functions to determine which acoustic embedding <NUM> or subset of acoustic embeddings <NUM> are likely the best acoustic embeddings <NUM> to represent the speaker <NUM> of the audio sample <NUM>. As previously stated, instead of relying on, for example, a single acoustic embedding <NUM> for the audio sample <NUM> to represent the speaker <NUM> accurately, the verifier <NUM> produces multiple variations <NUM> of the audio sample <NUM> such that there is likely a greater probability that at least one of the many variations <NUM> of the audio sample <NUM>, or some combination of the variations <NUM>, accurately represent the speaker <NUM>. This means that the multiple sample variations <NUM> represented by multiple acoustic embeddings <NUM> should be evaluated to determine one or more acoustic embeddings <NUM> that appear to best represent the speech characteristics of the speaker <NUM> of the audio sample <NUM>.

To perform this role, the comparator <NUM> is configured to evaluate each acoustic embedding <NUM> from the generator <NUM> as a candidate acoustic embedding <NUM> and to determine which single candidate <NUM> or set of candidates 232a-n would best represent the speech characteristics of the speaker <NUM> of the audio sample <NUM>. In some examples, the comparator <NUM> functions by removing a subset of candidate acoustic embeddings <NUM> from the set of candidate acoustic embeddings <NUM> and generating an aggregate acoustic embedding from the remaining candidate acoustic embeddings <NUM>. For instance, <FIG> depicts four candidate acoustic embeddings 232a-d for the first audio sample <NUM> that correspond to the acoustic embeddings 222a-d from the generator <NUM> and four acoustic embeddings 232e-h for the second audio sample <NUM> that corresponding to the acoustic embeddings 222e-h from the generator <NUM>. Here, the comparator <NUM> reviews each candidate acoustic embedding <NUM> with respect to the other candidate acoustic embedding <NUM> in the set and makes the determination of which candidate acoustic embedding(s) <NUM> to remove. For instance, <FIG> illustrates the comparator <NUM> removing two candidate acoustic embeddings 222a,d in the first set for the first audio sample 202a and two candidate acoustic embeddings 222e,f in the second set for the second audio sample 202b. From the remaining candidate acoustic embeddings 232b,c in the first set and the remaining candidate acoustic embeddings 232e,f in the second set, the comparator <NUM> generates a first aggregate acoustic embedding 234a for the first audio sample 202a and a second aggregate acoustic embedding 234b for the second audio sample 202b. In some examples, the comparator <NUM> generates the aggregate acoustic embedding <NUM> by determining an average vector for the remaining candidate acoustic embeddings <NUM>.

In some examples, the comparator <NUM> evaluates the set of candidate acoustic embeddings <NUM> by determining a score for each candidate acoustic embedding <NUM> in the set. In some configurations, the score corresponds to a function of the average cosine similarity between a given candidate acoustic embedding <NUM> and the other candidate acoustic embeddings <NUM> in a set for a particular audio sample <NUM>. The cosine similarity refers to a metric that measures the cosine of the angle between two vectors in dimensional space. By generating a cosine similarity between a given candidate acoustic embedding <NUM> and each other candidate acoustic embedding <NUM> in a set of candidate acoustic embeddings <NUM>, all of the cosine similarities for the given candidate may be averaged together to generate the average cosine similarity score. In some implementations, the score corresponds to a function of the Euclidean distance between a given candidate acoustic embedding <NUM> and the other candidate acoustic embeddings <NUM> in a set for a particular audio sample <NUM>. For instance, like the cosine similarity, the comparator <NUM> determines the Euclidean distance between a given candidate <NUM> and each other candidate <NUM>. From these multiple Euclidean distances for the given candidate <NUM>, the score is set equal to the average of all of the multiple Euclidean distances to represent an overall Euclidean distance score for the candidate <NUM>. After generating a score by any method, the comparator <NUM> may rank or order the set of candidates <NUM> based on the score. For example, the scores are ordered in descending order from the greatest score to the least score where the greatest score represents that the candidate acoustic embedding <NUM> with the greatest score is the closest on average to every other candidate acoustic embedding <NUM> in the set in the dimensional vector space. After ordering the set of candidate acoustic embeddings <NUM> for a given audio sample <NUM>, the comparator <NUM> may be configured to select N number of candidates <NUM> from the ordered list and to remove the candidates <NUM> not selected. For instance, <FIG> shows the comparator <NUM> having ordered the set of candidate acoustic embeddings <NUM> for the first audio sample 202a and the set of candidate acoustic embeddings <NUM> for the second audio sample 202b. Here, N=<NUM> and the comparator <NUM> selects two of the candidate acoustic embeddings <NUM> in the ordered list with the greatest score while removing the rest of the candidate acoustic embeddings <NUM>. The comparator <NUM> then uses the selected N acoustic embeddings <NUM> to generate the aggregate acoustic embedding <NUM> for the audio sample <NUM>. Other selection criteria are also possible. For instance, instead of selecting N candidate acoustic embeddings <NUM> to for the aggregate acoustic embedding <NUM>, the comparator may remove N candidate acoustic embeddings <NUM> from the set. As another example, the comparator may remove T-N candidate acoustic embeddings <NUM> from the set where T is the total number of candidate acoustic embeddings <NUM> in the set.

Alternatively, instead of selecting N candidate acoustic embeddings <NUM> with the greatest score, the comparator <NUM> is configured with a threshold score value such that the comparator <NUM> generates the aggregate acoustic embedding <NUM> using all candidate acoustic embeddings <NUM> that satisfy the threshold score value (e.g., equal or exceed the set threshold score value). By using a scoring process, the comparator <NUM> may ensure that outlier acoustic embeddings <NUM> of the sample variations <NUM> for the audio sample <NUM> that are likely inaccurate representations of speech characteristics for the speaker <NUM> of the audio sample <NUM> have minimal impact on the verifier <NUM>. In some configurations, the comparator <NUM> performs some combination of the N selection and the threshold score value. For example, in knowing that N number of candidate acoustic embeddings <NUM> will form the aggregate acoustic embedding <NUM>, the comparator <NUM> determines a score that corresponds to the Nth candidate acoustic embedding <NUM> in the ordered list of candidate acoustic embeddings <NUM> and sets the threshold score to this value. In this approach, the comparator <NUM> may also review the threshold score that corresponds to the Nth candidate <NUM> to determine if the number N should be updated (e.g., increased or decreased based on the threshold score).

With the aggregate acoustic embedding <NUM> for each audio sample 202a-b, the comparator <NUM> may then compare each aggregate acoustic embedding <NUM> to determine whether the first audio sample 202a and the second audio sample 202b are from the same speaker <NUM> or not. In some examples, the comparator <NUM> determines that the first audio sample 202a and the second audio sample 202b are from the same speaker <NUM> when the first aggregate acoustic embedding 234a for the first audio sample 202a matches or closely resembles the second aggregate acoustic embedding 234b of the second audio sample 202b.

In some implementations, such as <FIG>, in order to determine whether the first aggregate acoustic embedding 234a for the first audio sample 202a matches or closely resembles the second aggregate acoustic embedding 234b of the second audio sample 202b, the comparator <NUM> determines the distance (e.g., the cosine distance) between the first aggregate acoustic embedding 234a and the second aggregate acoustic embedding 234b. Here, the comparator <NUM> may be configured such that when the distance between the first aggregate acoustic embedding 234a and the second aggregate acoustic embedding 234b satisfies a distance threshold <NUM>, the comparator <NUM> determines that the first audio sample 202a and the second audio sample 202b are from the same speaker <NUM>. Otherwise, when the distance between the first aggregate acoustic embedding 234a and the second aggregate acoustic embedding 234b fails to satisfy the distance threshold <NUM>, the comparator <NUM> determines that the first audio sample 202a and the second audio sample 202b are not from the same speaker <NUM>. The distance threshold <NUM> refers to a value that is set to indicate a confidence level that the speaker <NUM> of the first audio sample 202a is likely the same speaker <NUM> as the second audio sample 202b.

In some implementations, the augmentation technique of the variator <NUM> has some limitations. For instance, when the variator <NUM> generates the sample variations <NUM> using the augmentation technique that divides the audio sample <NUM> into slices <NUM>, the size of the slices <NUM> cannot be so small that an individual slice <NUM> includes very little speech characteristic data to form a meaningful speaker representation <NUM>. If the slice <NUM> is too small, the speech characteristics corresponding to the slice <NUM> may become attenuated in their representation of the speaker <NUM>. Due to this limitations, a sample variation <NUM> that has a length of less than some time threshold (e.g., one second) may not form a meaningful speaker representation <NUM>. Therefore, the slicing augmentation technique may be constrained to prevent the size of a given slice <NUM> from being less than the time threshold. Unfortunately, an audio sample <NUM> that corresponds to enrollment utterances <NUM> or query utterances <NUM> is often only a few seconds long. This would mean that the technique of slicing would only generate a few speaker representations <NUM> instead of a larger number that would likely increase the accuracy of the verifier <NUM>.

To overcome this issue, the variator <NUM> may combine the slicing technique with other augmentation techniques (e.g., a shuffle technique and/or a concatenation technique). For example, as shown in <FIG>, the variator <NUM> may divide an audio sample <NUM> that is two seconds long into three slices 214a-c (such that each slice <NUM> is about <NUM> seconds). Here, if the time threshold was equal to one second, the generator <NUM> would generate a speaker representation <NUM> for each slice <NUM> that is unlikely to improve the accuracy of the verifier <NUM>. Instead, after slicing the audio sample <NUM>, the variator <NUM> may reconstruct a number of slices <NUM> together (i.e., concatenate some number of slices <NUM> together) such that the total length of the concatenated slices <NUM> is greater than the time threshold (e.g., one second). This way, the reconstructed sample variation <NUM> can have enough slices <NUM> combined together to avoid suffering from poor speaker characteristics. Furthermore, this approach capitalizes on the fact that the generator <NUM> may generate a speaker representation <NUM> for any length of audio. Therefore, the variator <NUM> does not need to reconstruct slices <NUM> to have a length equal to that of the original audio sample <NUM>, but instead the reconstructed slices may have a different length of time when compared to the original audio sample <NUM>. For example, <FIG> depicts each sample variation 212a-f constructed from two slices <NUM> to have a total length of <NUM> seconds, which is less than the original length of the audio sample <NUM>, two seconds.

Moreover, <FIG> illustrates that, when reconstructing more than one slice <NUM> together to form the sample variation <NUM>, the slices <NUM> may be shuffled in an order that is different from the order that the slices <NUM> appeared in the audio sample <NUM>. For instance, the variator <NUM> forms the third sample variation 212c from an ordered combination of the second slice 214b and the first slice 214a which is the reverse of the original order for the audio sample <NUM> that is represented by the first sample variation 212a. In <FIG>, the variator <NUM> is able to utilize the reshuffling or mixing of slices <NUM> and the concatenation of slices <NUM> together to form six sample variations 212a-f. The variator <NUM> is able to shuffle and reconstruct the slices <NUM> in any and all permutations as long as the variator <NUM> avoids forming a sample variation <NUM> with a length of time less than the time threshold. For example, if the sample audio <NUM> were sliced into five slices <NUM>, the variator <NUM> would be able to form one-hundred and twenty sample variations <NUM> because the number of permutations may be represented as P(n, r) where n is the number of slices <NUM> and r is equal to the number of slices <NUM> being concatenated together to form the sample variation <NUM>. Stated differently, the variator <NUM> would be able to form one-hundred and twenty sample variations <NUM> because n = <NUM> and r = <NUM> (i.e. P(<NUM>,<NUM>)).

<FIG> is an example of the model-approach for the speaker verification process. Here, the verifier <NUM> does not include a comparator <NUM>, but instead includes a model <NUM> that is configured to generate a prediction <NUM> of whether the speaker <NUM> of the first audio sample 202a is the same speaker <NUM> as the second audio sample 202b. In the model approach, the variator <NUM> is configured to generate sample variations <NUM> by performing several augmentation techniques on a frequency representation of the audio sample <NUM>. For instance, the variator <NUM> performs mulitple spectrogram augmentation techniques <NUM>, 216a-n on a spectrogram of the audio sample <NUM>. A spectrogram generally refers to a visual representation of a spectrum of frequencies for an audio signal corresponding to the audio sample <NUM>. A spectrogram may also sometimes be referred to as a sonograph, voiceprint, or voicegram. As a visual representation, the variator <NUM> is configured to augment the visual time sequence of a spectrogram for the audio sample <NUM>. Some examples of spectrogram augmentation techniques <NUM> include time masking or adding <NUM>, 216a, frequency masking <NUM>, 216b, and time warping <NUM>, 216c (i.e., spectrogram stretching). When performing time masking 216a, the variator <NUM> may set particular parameters such as an initial offset, a removal width corresponding to the amount of data (e.g., frames) that will be removed from the spectrogram, and a width to keep that specifies one or more frames of the spectrogram to keep undisrupted. In contrast, time masking includes parameters that specify frames to duplicate or to add to the spectrogram.

As shown in <FIG>, when the variator <NUM> receives the first audio sample 202a and the second audio sample 202b, the variator <NUM> performs the same spectrogram augmentation techniques on each audio sample <NUM>. For instance, when the variator <NUM> performs time masking 216a, frequency masking <NUM>, and time warping 216c on a spectrogram corresponding to the first audio sample 202a, the variator <NUM> performs the same augmentation techniques <NUM> of time masking 216a, frequency masking <NUM>, and time warping 216c on a spectrogram corresponding to the second audio sample 202b. By performing the same spectrogram augmentation techniques <NUM> on each audio sample <NUM>, the verifier <NUM> ensures that the audio samples 202a-b are comparable by the generator <NUM> and/or model <NUM>. As a result from each spectrogram augmentation technique <NUM>, the variator <NUM> generates a corresponding sample variation <NUM>.

The generator <NUM> is configured to receive all of the sample variations <NUM> from the variator <NUM> and to generate a score <NUM> for each spectrogram augmentation technique <NUM>. For instance, the generator <NUM> compares a first sample variation 212a generated by a first spectrogram augmentation technique 216a on the first audio sample 202a to a second sample variation 212d generated by the first spectrogram augmentation technique 216a on the second audio sample 202b. For the second spectrogram augmentation technique 216b, the generator <NUM> compares a third sample variation 212b generated by the second spectrogram augmentation technique 216b on the first audio sample 202a to a fourth sample variation 212e generated by the second spectrogram augmentation technique 216b on the second audio sample 202b. For the third spectrogram augmentation technique 216c, the generator <NUM> compares a fifth sample variation 212c generated by the third spectrogram augmentation technique 216c on the first audio sample 202a to a sixth sample variation 212f generated by the third spectrogram augmentation technique 216c on the second audio sample 202b. As shown in <FIG>, the generator's comparison of the first spectrogram augmentation technique 216a results in a first score 224a. The generator's comparison of the second spectrogram augmentation technique 216b results in a second score 224b. The generator's comparison of the third spectrogram augmentation technique 216c results in a third score 224c. This process may repeat depending on the number of spectrogram augmentation techniques being used. For example, although the variator <NUM> is performing three techniques <NUM>, the variator <NUM> may perform four or even five techniques in a scalable manner. In some examples, the score <NUM> determined by the generator <NUM> is a cosine similarity score <NUM>.

The model <NUM> is configured to receive the scores <NUM> as input and to generate a prediction <NUM> of whether the speaker <NUM> of the first audio sample 202a is the same speaker <NUM> as the second audio sample 202b as output. In some implementations, the prediction <NUM> corresponds to a probability that the first audio sample 202a and the second audio sample 202b belong to the same speaker <NUM>. In some configurations, the model <NUM> is a machine learning model or neural network that is configured to process data characterizing an audio sample <NUM> (e.g., a score <NUM> from the generator <NUM>). The model <NUM> may include one or more layers of nonlinear units to generate the prediction <NUM> based on the received input. In some implementations, the model <NUM> lacks a softmax or other classification layer. In some examples, the model <NUM> is a Long Short-Term Memory (LSTM) neural network that includes one or more LSTM memory blocks. Each LSTM memory block can include one or more memory cells, and each memory cell can include an input gate, a forget gate, and an output gate that allow the cell to store previous states for the cell, e.g., for use in generating a current activation or to provide to other components of the model <NUM>. The model <NUM> may be a feedforward neural network, a convolutional neural network, a recurrent neural network, or may be a deep neural network having several portions of different types.

Before the model <NUM> is deployed for real-time or inference prediction, the model <NUM> undergoes a training process to teach the model <NUM> how to generate an accurate prediction <NUM>. The model <NUM> may learn how to generate predictions <NUM> by iteratively updating current values of internal parameters (e.g., of its neural network) over a series of training cycles. In each training cycle, the model <NUM> processes a batch of training examples. The output of the model <NUM> in each cycle is a set of predictions <NUM> that has been generated for each training example in the batch. During training, the model <NUM> may be trained to optimize a loss function or other objective function. The loss function is generally formulated to minimize variation among the outputs or predictions <NUM> for training examples of the same speaker, while maximizing differences among predictions <NUM> for training examples of different speakers.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of generating a speaker representation <NUM> for an audio sample <NUM>. At operation <NUM>, the method <NUM> receives a first audio sample <NUM>, 202a from a first speaker <NUM>, 10a and a second audio sample <NUM>, 202b from a second speaker <NUM>, 10b. At operation <NUM>, the method <NUM> includes sub-operations 304a-d for each audio sample <NUM> of the first audio sample 202a and the second audio sample 202b. At operation 304a, the method <NUM> divides the respective audio sample <NUM> into a plurality of slices <NUM>. At operation 304b, based on the plurality of slices <NUM>, the method <NUM> generates a set of candidate acoustic embeddings <NUM> where each candidate acoustic embedding <NUM> includes a vector representation <NUM> of acoustic features. At operation 304c, the method <NUM> removes a subset of the candidate acoustic embeddings <NUM> from the set of candidate acoustic embeddings <NUM>. At operation 304d, the method <NUM> generates an aggregate acoustic embedding <NUM> from the remaining candidate acoustic embeddings <NUM> in the set of candidate acoustic embeddings <NUM> after removing the subset of the candidate acoustic embeddings <NUM>.

<FIG> is schematic view of an example computing device <NUM> that may be used to implement the systems (e.g., the speaker identification system <NUM> and/or verifier <NUM>) and methods (e.g., method <NUM>) described in this document.

The computing device <NUM> includes a processor <NUM> (e.g., data processing hardware), memory <NUM> (e.g., memory hardware), a storage device <NUM>, a high-speed interface/controller <NUM> connecting to the memory <NUM> and high-speed expansion ports <NUM>, and a low speed interface/controller <NUM> connecting to a low speed bus <NUM> and a storage device <NUM>. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

For example, it may be implemented as a standard server 400a or multiple times in a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

Claim 1:
A method (<NUM>) comprising:
receiving, at data processing hardware (<NUM>), a first audio sample (202a) from a first speaker (10a) and a second audio sample (202b) from a second speaker (10b);
for each audio sample (<NUM>) of the first audio sample (202a) and the second audio sample (202b):
a) dividing, by the data processing hardware (<NUM>), the respective audio sample (<NUM>) into a plurality of audio slices (<NUM>);
b) based on the plurality of slices (<NUM>), generating, by the data processing hardware (<NUM>), a set of candidate acoustic embeddings (<NUM>), each candidate acoustic embedding (<NUM>) comprising a vector representation of acoustic features;
c) determining, by the data processing hardware (<NUM>), a similarity score for each candidate acoustic embedding (<NUM>) in the set of candidate acoustic embeddings (<NUM>);
d) removing, by the data processing hardware (<NUM>), based on the similarity score for each candidate acoustic embedding (<NUM>) from the set of candidate acoustic embeddings (<NUM>), a subset of the candidate acoustic embeddings (<NUM>) from the set of candidate acoustic embeddings (<NUM>); and
e) generating, by the data processing hardware (<NUM>), an aggregate acoustic embedding (<NUM>) from the remaining candidate acoustic embeddings (<NUM>) in the set of candidate acoustic embeddings (<NUM>) after removing the subset of the candidate acoustic embeddings (<NUM>); and
identifying, by the data processing hardware (<NUM>), whether the first speaker (10a) and the second speaker (10b) are the same speaker (<NUM>) or different speakers (<NUM>) based on the aggregate acoustic embedding (<NUM>) generated for each audio sample (<NUM>) of the first audio sample (202a) and the second audio sample (202b).