Patent Description:
Speaker diarization is the process of partitioning an input audio stream into homogenous segments according to speaker identity. In an environment with multiple speakers, speaker diarization answers the question "who is speaking when" and has a variety of applications including multimedia information retrieval, speaker turn analysis, audio processing, and automatic transcription of conversational speech to name a few. For example, speaker diarization involves the task of annotating speaker turns in a conversation by identifying that a first segment of an input audio stream is attributable to a first human speaker (without particularly identifying who the first human speaker is), a second segment of the input audio stream is attributable to a different second human speaker (without particularly identifying who the second human speaker is), a third segment of the input audio stream is attributable to the first human speaker, etc. It is known from <CIT> how to implement end-to-end speech diarization using a segmentation module, an embedding module, and a speaker label predictor.

The invention is as defined in appended independent claims <NUM> and <NUM>.

Automatic speech recognition (ASR) systems generally rely on speech processing algorithms that assume only one speaker is present in a given input audio signal. An input audio signal that includes a presence of multiple speakers can potentially disrupt these speech processing algorithms, thereby leading to inaccurate speech recognition results output by the ASR systems. As such, speaker diarization is the process of segmenting speech from a same speaker in a larger conversation to not specifically determine who is talking (speaker recognition/identification), but rather, determine when someone is speaking. Put another way, speaker diarization includes a series of speaker recognition tasks with short utterances and determines whether two segments of a given conversation were spoken by the same individual or different individuals, and repeated for all segments of the conversation.

Existing speaker diarization systems generally include multiple relatively independent components, such as, without limitation, a speech segmentation module, an embedding extraction module, and a clustering module. The speech segmentation module is generally configured to remove non-speech parts from an input utterance and divide the input utterance into small fixed-length segments, while the embedding extraction module is configured to extract, from each fixed-length segment, a corresponding speaker-discriminative embedding. The speaker-discriminative embeddings may include i-vectors or d-vectors. The clustering modules employed by the existing speaker diarization systems are tasked with determining the number of speakers present in the input utterance and assign speaker identifies (e.g., labels) to each fixed-length segment. These clustering modules may use popular clustering algorithms that include Gaussian mixture models, mean shift clustering, agglomerative hierarchical clustering, k-means clustering, links clustering, and spectral clustering. Speaker diarization systems may also use an additional re-segmentation module for further refining the diarization results output from the clustering module by enforcing additional constraints.

These existing speaker diarization are limited by the fact that the extracted speaker-discriminative embeddings are not optimized for diarization, and therefore may not necessarily extract relevant features for disambiguating speakers in the presence of overlap. Moreover, the clustering modules operate in an unsupervised manner such that all speakers are assumed to be unknown and the clustering algorithm needs to produce new "clusters" to accommodate the new/unknown speakers for every new input utterance. The drawback with these unsupervised frameworks is that they are unable to improve by learning from large sets of labeled training data that includes fine-grained annotations of speaker turns (i.e., speaker changes), time-stamped speaker labels, and ground truth. Since this labeled training data is readily obtainable in many domain-specific applications and diarization training datasets, speaker diarization systems could benefit from the labeled training data by becoming more robust and accurate in producing diarization results. Moreover, existing state-of-the-art clustering algorithms mostly execute offline, thereby making it difficult to produce diarization results by clustering in real-time scenarios. Speaker diarization systems are also required to perform on long sequences of speech (i.e., several minutes), however, training speaker diarization systems over long speech sequences with a large batch size may be difficult due to memory constraints.

To overcome the limitations of typical diarization systems discussed above, implementations herein are directed toward a Diarization by Iterative Voice Embedding (DIVE) system. The DIVE system includes an end-to-end neural diarization system that combines and jointly trains three separate components/modules: a temporal encoder tasked with projecting an input audio signal to a downsampled embedding space that includes a sequence of temporal embeddings each representing current speech content at a corresponding time step; a speaker selector tasked with performing an iterative speaker selection process to select long-term speaker vectors for all speakers in the input audio stream; and a voice activity detector (VAD) tasked with detecting voice activity for each speaker at each of the plurality of time steps.

Referring to <FIG>, a system <NUM> includes a user device <NUM> capturing speech utterances <NUM> from a group of speakers (e.g., users) <NUM>, 10a-n and communicating with a remote system <NUM> via a network <NUM>. The remote system <NUM> may be a distributed system (e.g., cloud computing environment) having scalable/elastic resources <NUM>. The resources <NUM> include computing resources <NUM> (e.g., data processing hardware) and/or storage resources <NUM> (e.g. memory hardware). In some implementations, the user device <NUM> and/or the remote system <NUM> executes the DIVE system <NUM> (also referred to as an end-to-end neural diarization system <NUM>) that is configured to receive an input audio signal (i.e., audio data) <NUM> that corresponds to the captured utterances <NUM> from the multiple speakers <NUM>. The DIVE system <NUM> encodes the input audio signal <NUM> into a sequence of T temporal embeddings <NUM>, 220a-t and iteratively selects a respective speaker embedding <NUM>, 240a-n for each respective speaker <NUM>. Using the sequence of T temporal embeddings <NUM> and each selected speaker embedding <NUM>, the DIVE system <NUM> predicts a respective voice activity indicator <NUM> for each respective speaker <NUM> during each of a plurality of time steps. Here, the voice activity indicator <NUM> indicates at each time step whether a voice of the respective speaker is active or inactive. The respective voice activity indicator <NUM> predicted for each respective speaker <NUM> during each of the plurality of time steps may provide diarization results <NUM> indicating when the voice of each speaker is active (or inactive) in the input audio signal <NUM>. Each time step may correspond to a respective one of the temporal embeddings. In some examples, each time step includes a duration of <NUM> millisecond. As such, the diarization results <NUM> may provide time-stamped speaker labels based on the per-speaker voice activity indicators <NUM> predicted at each time step that not only identify who is speaking at a given time, but also identify when speaker changes (e.g., speaker turns) occur between adjacent time steps.

In some examples, the remote system <NUM> further executes an automated speech recognition (ASR) module <NUM> that is configured to receive and transcribe the audio data <NUM> into a corresponding ASR result <NUM>. The user device <NUM> may similarly execute the ASR module <NUM> on-device in lieu of the remote system <NUM>, which may be useful when network connections are unavailable or quick (albeit lower-fidelity) transcriptions are preferable. Additionally or alternatively, the user device <NUM> and the remote system <NUM> may both execute corresponding ASR modules <NUM> such that the audio data <NUM> can be transcribed on-device, via the remote system <NUM>, or some combination thereof. In some implementations, the ASR module <NUM> and the DIVE system <NUM> both execute entirely on the user device <NUM> and do not require any network connection to the remote system <NUM>. The ASR result <NUM> may also be referred to as a 'transcription' or simply 'text'. The ASR module <NUM> may communicate with the DIVE system <NUM> to utilize the diarization results <NUM> associated with the audio data <NUM> for improving speech recognition on the audio data <NUM>. For instance, the ASR module <NUM> may apply different speech recognition models (e.g., language models, prosody models) for different speakers identified from the diarization results <NUM>. Additionally or alternatively, the ASR module <NUM> and/or the DIVE system <NUM> (or some other component) may index a transcription <NUM> of the audio data <NUM> using the per-speaker, per-time step voice activity indicators <NUM>. For instance, a transcription of a conversation between multiple co-workers (e.g., speakers <NUM>) during a business meeting may be indexed by speaker to associate portions of the transcription with the respective speaker for identifying what each speaker said.

The user device <NUM> includes data processing hardware <NUM> and memory hardware <NUM>. The user device <NUM> may include an audio capture device (e.g., microphone) for capturing and converting the speech utterances <NUM> from the speakers <NUM> into the audio data <NUM> (e.g., electrical signals). In some implementations, the data processing hardware <NUM> is configured to execute a portion of the DIVE system <NUM> locally while a remaining portion of the diarization system <NUM> executes on the remote system <NUM>. Alternatively, the data processing hardware <NUM> may execute the DIVE system <NUM> in lieu of executing the DIVE system <NUM> on the remote system <NUM>. The user device <NUM> can be any computing device capable of communicating with the remote system <NUM> through the network <NUM>. The user device <NUM> includes, but is not limited to, desktop computing devices and mobile computing devices, such as laptops, tablets, smart phones, smart speakers/displays, smart appliances, internet-of-things (IoT) devices, and wearable computing devices (e.g., headsets and/or watches). The user device <NUM> may optionally execute the ASR module <NUM> to transcribe the audio data <NUM> into corresponding text <NUM>. For instance, when network communications are down or not available, the user device <NUM> may execute the diarization system <NUM> and/or the ASR module <NUM> locally to produce the diarization results for the audio data <NUM> and/or generate a transcription <NUM> of the audio data <NUM>.

In the example shown, the speakers <NUM> and the user device <NUM> may be located within an environment (e.g., a room) where the user device <NUM> is configured to capture and convert speech utterances <NUM> spoken by the speakers <NUM> into the audio signal <NUM> (also referred to as audio data <NUM>). For instance, the speakers <NUM> may correspond to co-workers having a conversation during a meeting and the user device <NUM> may record and convert the speech utterances <NUM> into the audio signal <NUM>. In turn, the user device <NUM> may provide the audio signal <NUM> to the DIVE system <NUM> for predicting the voice activity indicators <NUM> for each of the speakers <NUM> during each of the plurality of time steps. Thus, the DIVE system <NUM> is tasked with processing the audio signal <NUM> to determine when someone is speaking without specifically determining who is talking via speaker recognition/identification.

In some examples, at least a portion of the utterances <NUM> conveyed in the audio signal <NUM> are overlapping such that at a given instant in time voices of two or more of the speakers <NUM> are active. Notably, a number N of the multiple speakers <NUM> may be unknown when the input audio signal <NUM> is provided as input to the DIVE system <NUM> and the DIVE system <NUM> may predict the number N of the multiple speakers <NUM>. In some implementations, the user device <NUM> is remotely located from the speakers <NUM>. For instance, the user device <NUM> may include a remote device (e.g., a network server) that captures speech utterances <NUM> from speakers that are participants in a phone call or video conference. In this scenario, each speaker <NUM> would speak into their own device (e.g., phone, radio, computer, smartwatch, etc.) that captures and provides the speech utterances <NUM> to the remote user device <NUM> for converting the speech utterances <NUM> into the audio data <NUM>. Of course in this scenario, the utterances <NUM> may undergo processing at each of the user devices and be converted into corresponding audio signals <NUM> that are transmitted to the remote user device <NUM> which may additionally processes the audio signal <NUM> provided as input to the DIVE system <NUM>.

In the example shown, the DIVE system <NUM> includes a temporal encoder <NUM>, an iterative speaker selector <NUM>, and a voice activity detector (VAD) <NUM>. The temporal encoder <NUM> is configured to receive the audio signal <NUM> and encode the input audio signal <NUM> into the sequence of temporal embeddings h <NUM>, 220a-t. Each temporal embedding h <NUM> may be associated with a corresponding time step t and represent speech content extracted from the input audio signal <NUM> during the corresponding time step t. The temporal encoder <NUM> sends the sequence of temporal embeddings <NUM> to the iterative speaker selector <NUM> and the VAD <NUM>.

During each of a plurality of iterations i each corresponding to a respective speaker of the multiple speakers <NUM>, the iterative speaker selector <NUM> is configured to select a respective speaker embedding <NUM>, 240a-n for each respective speaker <NUM>. For simplicity, the audio signal <NUM> in <FIG> includes utterances <NUM> spoken by only two distinct speakers <NUM>, however, the iterative speaker selector <NUM> may select speaker embeddings <NUM> for any number N of distinct speakers <NUM> present in the input audio signal <NUM>. Thus, in the example two-speaker scenario, the iterative speaker selector <NUM> selects a first speaker embedding s<NUM> <NUM> for the first speaker 10a during an initial first iteration (i=<NUM>), and during a subsequent second iteration (i=<NUM>), the iterative speaker selector <NUM> selects a second speaker embedding s<NUM> <NUM> for the second speaker 10b. During each iteration i, the iterative speaker selector <NUM> selects the respective speaker embedding <NUM> by determining, for each temporal embedding <NUM> in the sequence of T temporal embeddings <NUM>, a probability that the corresponding temporal embedding <NUM> includes a presence of voice activity by a single new speaker <NUM> for which a speaker embedding <NUM> was not previously selected during a previous iteration. Thereafter, the iterative speaker selector <NUM> selects, during the corresponding iteration i, the respective speaker embedding <NUM> for the respective speaker <NUM> as the temporal embedding <NUM> in the sequence of T temporal embeddings <NUM> that is associated with the highest probability for the presence of voice activity by the single new speaker <NUM>. That is, the iterative speaker selector <NUM> selects the respective speaker embedding <NUM> that has the highest probability of being associated with the speech content of the respective T temporal embeddings <NUM>.

The VAD <NUM> receives the temporal embedding <NUM> and speaker embeddings <NUM> (e.g., s<NUM> and s<NUM> in the two-speaker scenario of <FIG>) and predicts, at each time step, the respective voice activity indicator <NUM> for each respective speaker of the multiple N speakers. In particular, the VAD <NUM> predicts the voice activity indicator <NUM> based on the temporal embedding <NUM> that represents the speech content at the respective time step t, a speaker embedding <NUM> that represents the identity of the speaker of interest, and another speaker embedding <NUM> that represents all of the speakers <NUM>. Here, the respective voice activity indicator <NUM> indicates whether a voice of the respective speaker <NUM> is active or inactive at the corresponding time step. Notably, the VAD <NUM> predicts the voice activity indicator <NUM> without particularly identifying the respective speaker <NUM> from the multiple speakers <NUM>. The DIVE system <NUM> may use the voice activity indicator <NUM> at each time step to provide diarization results <NUM>. As shown in <FIG>, the diarization results <NUM> include the voice activity indicator yi,t of speaker i at time step t. Accordingly, the voice activity indicator yi,t <NUM> of the diarization results <NUM> provides per-speaker, per-timestep VAD results with a value of "<NUM>" when the speaker <NUM> is inactive and a value of "<NUM>" when the speaker <NUM> is active during time step t. As shown at time step (t=<NUM>), multiple speakers <NUM> may be active at the same time.

<FIG> shows the temporal encoder <NUM>, the iterative speaker selector <NUM>, and the VAD <NUM> of the DIVE system <NUM>. The temporal encoder <NUM> encodes the input audio signal <NUM> into the sequence of temporal embeddings <NUM>, 220a-t each associated with a corresponding time step t. The temporal encoder <NUM> may project the sequence of temporal embeddings <NUM> encoded from the input audio signal <NUM> into a downsampled embedding space. The temporal encoder <NUM> may cascade residual blocks of dilated 1D-convolutions, with parametric rectified linear unit (PReLU) activations and layer normalization, and perform downsampling by introducing 1D average pooling layers between the residual blocks. Thus, the input audio signal <NUM> may correspond to an input waveform x such that the temporal encoder <NUM> produces T temporal embeddings <NUM> (e.g., latent vectors) each having a dimension D. Accordingly, each temporal embedding <NUM> may be represented as <MAT>.

The iterative speaker selector <NUM> outputs the respective speaker embedding <NUM>, 240a-n for each speaker <NUM> detected in the input audio signal <NUM>. During each of the plurality of iterations i, the iterative speaker selector <NUM> receives the temporal embeddings <NUM> and selects a respective speaker embedding <NUM> that was not selected in a previous iteration i (i.e., a new speaker embedding <NUM>). In some examples, the speaker selector <NUM> receives, as input, the previously selected speaker embedding <NUM> along with the sequence of T temporal embeddings <NUM> during each iteration i, and outputs the probability as a confidence c that each corresponding temporal embedding <NUM> includes the presence of voice activity by a single new speaker. The previously selected speaker embedding <NUM> may include an average of each respective speaker embedding <NUM> previously selected during previous iterations. Notably, the previously selected speaker embedding <NUM> would be zero during the initial first iteration since no speaker embedding <NUM> has been previously selected. Advantageously, the iterative process performed by the iterative speaker selector <NUM> does not require a particular speaker ordering to training to select the speaker embeddings <NUM>, and therefore does not require Permutation-Invariant Training (PIT) to avoid penalties for choosing speaker orders. PIT suffers from inconsistent assignments when applied on long audio sequences, and therefore is not preferable for use in learning long-term speaker representations/embeddings.

For simplicity, <FIG> illustrates the audio signal <NUM> including utterances <NUM> spoken by only two distinct speakers <NUM>, however, this is a non-limiting example and the audio signal <NUM> may include utterances spoken by any number of distinct speakers <NUM>. In the example shown, the iterative speaker selector <NUM> includes a first speaker selector <NUM>, 230a that, at a first iteration (i=<NUM>), receives the sequence of T temporal embeddings <NUM> and selects a first speaker embedding s<NUM> <NUM>, 240a. The first speaker embedding s<NUM> 240a includes a first confidence c<NUM> indicating the likelihood that the temporal embedding <NUM> includes the first speaker embedding s<NUM> 240a. Here, because there are no previously selected speaker embeddings <NUM> the first speaker selector 230a may select any of the speaker embeddings <NUM>. Continuing with the example, the iterative speaker selector <NUM> includes a second speaker selector <NUM>, 230b that, at a subsequent iteration (i=<NUM>), receives the sequence of T temporal embeddings <NUM> and the previously selected first speaker embedding s<NUM> 240a and selects a second speaker embedding s<NUM> <NUM>, 240b. The second speaker embedding s<NUM> 240b includes a second confidence c<NUM> indicating the likelihood that the temporal embedding <NUM> includes the second speaker embedding s<NUM> 240b. Here, the second speaker selector 230b may select any of the speaker embeddings <NUM> besides the previously selected speaker embeddings (e.g., the first speaker embedding s<NUM> 240a).

The iterative speaker selector <NUM> may include any number of speaker selectors <NUM> to select the speaker embeddings <NUM>. In some examples, the iterative speaker selector <NUM> determines whether the confidence c associated with the speaker embedding <NUM> for the corresponding temporal embedding <NUM> that is associated with the highest probability for the presence of voice activity by the single new speaker satisfies a confidence threshold. The iterative speaker selector <NUM> may continue iteratively selecting speaker embeddings <NUM> until the confidence c fails to satisfy the confidence threshold.

In some implementations, the iterative speaker selector <NUM> includes a multi-class linear classifier with a fully-connected network that is configured to determine a probability distribution of possible event types e for each corresponding temporal embedding <NUM> during each iteration i. The possible event types et may include four possible types: the presence of voice activity by the single new speaker <NUM>; a presence of voice activity for a single previous speaker <NUM> for which another respective speaker embedding <NUM> was previously selected during a previous iteration; a presence of overlapped speech; and a presence of silence. Accordingly, the multi-class linear classifier with the fully-connected network may include a <NUM>-by-D matrix gµ(µi) that represents a <NUM>-class linear classifier that maps each temporal embedding ht <NUM> to one of the four possible event types et. Here, each temporal embedding <NUM> may be mapped to the event type having the highest probability in the probability distribution of possible event types during each iteration i. The probability distribution may be represented as follows. <MAT> In Equation <NUM>, et represents the event type, ht represents the respective temporal embedding at time t, ui represents the average embedding of each previously selected speaker at iteration i, and gh represents a fully-connected neural network. During inference, the confidence c for a respective speaker embedding <NUM> may be represented as follows. <MAT> where <MAT> corresponds to the temporal embedding <NUM> that is associated with the highest probability for the presence of voice activity by the single new speaker <NUM>. Thus, the selected speaker embedding <NUM> during each iteration corresponds to the temporal embedding that reaches the maximal confidence (i.e., highest probability) according to Equation <NUM>. Selecting a speaker embedding <NUM> may be conditioned on <MAT> satisfying a confidence threshold. If the confidence threshold is not satisfied, the iterative speaker selector <NUM> may bypass selection during the corresponding iteration and not perform any subsequent iterations. In this scenario, the DIVE system <NUM> may determine a number N of the multiple speakers <NUM> based on the number of speaker embeddings <NUM> previously selected during iterations prior to the corresponding iteration that bypasses selection of speaker embeddings <NUM>. During training, <MAT> is not output by the iterative speaker selector <NUM>, but instead the temporal embedding ht <NUM> is sampled uniformly from times with a novel speaker marked as active in the labeled training data. The iterative speaker selector <NUM> is trained in a supervised manner by a training process and parameters of training process learn to minimize the negative log likelihood of the <NUM>-way linear classifier as follows.

After the speaker embeddings <NUM> are selected (e.g., s<NUM> and s<NUM> in the two-speaker scenario of <FIG>), the VAD <NUM> predicts, at each time step, the respective voice activity indicator <NUM> for each respective speaker of the multiple N speakers based on the respective speaker embeddings <NUM>, a mean of all the speaker embeddings <NUM> previously selected, and the temporal embedding <NUM> associated with the corresponding time step as follows. <MAT> where i = <NUM>, <NUM>,. The respective voice activity indicator (yi,t) <NUM> indicates whether a voice of the respective speaker (indexed by iteration i) <NUM> is active (yi,t =<NUM>) or inactive (yi,t =<NUM>) at the corresponding time step (indexed by time step t). The respective voice activity indicator <NUM> may correspond to a binary per-speaker voice activity mask that provides a value of "<NUM>" when the respective speaker is inactive and a value of "<NUM>" when the respective speaker is active during the time step t. The voice activity indicator (yi,t) <NUM> predicted at each time step t for each speaker i may be based on the temporal embedding ht associated with the corresponding time step, the respective speaker embedding si selected for the respective speaker, and a mean of all the speaker embeddings s selected during the plurality of iterations.

In some implementations, the VAD <NUM> contains two parallel fully-connected neural networks fh and fs with PReLU activations with layer normalization, except for a last linear projection layer which includes a linear projection. In these implementations, to predict the voice activity indicator yi,t of speaker i at time step t, fh and fs project the temporal embedding <MAT> at the corresponding time step and the speaker embeddings <MAT> as follows. <MAT> In Equation <NUM>, [si; s] represents the concatenation along a channel axis of the respective speaker embedding si selected for the corresponding speaker i and <MAT> the mean of all of the speaker embeddings <NUM>. Notably, the mean speaker embedding calls the VAD <NUM> to exploit contrasts between the respective speaker embedding <NUM> associated with the speaker i of interest as well as all other speakers present in the sequence of temporal embeddings <NUM>. In the example shown, the VAD <NUM> predicts a voice activity inidicators <NUM> for a first and second speaker <NUM> at time step (t = <NUM>). The voice activity indicators <NUM> may provide diarization results <NUM> indicating that the first speaker <NUM> is active at the time step (t = <NUM>) (e.g., y<NUM>,<NUM> = <NUM>) and the second speaker <NUM> is inactive at the time step (t = <NUM>) (e.g., y<NUM>,<NUM> = <NUM>).

Referring to <FIG>, a schematic view <NUM> illustrates an example training process <NUM> and inference <NUM> for the DIVE system <NUM>. In some implementations, the training process <NUM> jointly trains the temporal encoder <NUM>, the iterative speaker selector <NUM> that includes the multi-class linear classifier with the fully-connected network, and the VAD <NUM> of the DIVE system <NUM> on fully-labeled training data <NUM> that includes a corpus of training audio signals x* each including utterances <NUM> spoken by multiple different speakers <NUM>. The training audio signals x* may include long speech sequences representing speech for several minutes. In some examples, the training process <NUM> samples W fixed-length windows per training audio signal x*, encodes the training audio signal x* using the temporal encoder <NUM>, and concatenates W fixed-length windows along the temporal axis. By concatenating the W fixed-length windows, the fully-labeled training data <NUM> increases speaker diversity and speaker turns for each training audio signal x* and keeps while keeping the memory usage low. That is, the training audio signal x* may represent same speakers over windows far apart during the long speech sequences. Some training audio signals x* may include portions where utterances <NUM> spoken by two or more different speakers <NUM> overlap. Each training audio signal x* is encoded by the temporal encoder <NUM> into a sequence of training temporal embeddings <NUM> that are each assigned a respective speaker label <NUM> indicating an active speaker or silence.

The speaker labels <NUM> may be represented as a sequence of training speaker labels ŷ = (ŷ<NUM>, ŷ<NUM>,. ,ŷT), where entry ŷt in the sequence represents the speaker label <NUM> assigned to training temporal embedding <NUM> at time step t. In the example shown, the training process <NUM> provides the sequence of training temporal embeddings 220T encoded by the temporal encoder <NUM> and assigned speaker labels <NUM> for training the iterative speaker selector <NUM> during each of the plurality of i iterations, and subsequently, the VAD <NUM> based on the speaker embeddings <NUM> selected by the iterative speaker selector <NUM> during the plurality of iterations.

As the temporal encoder <NUM>, the iterative speaker selector <NUM>, and the VAD <NUM> are trained jointly, the VAD <NUM> is also trained on the corpus of training audio signals x*, where each training audio signal x* is encoded into a sequence of training temporal embeddings each including a corresponding voice activity indicator (i.e., a speaker label ŷ) indicating which voice is present/active in the corresponding training temporal embedding. The training process may train the VAD <NUM> on the following VAD loss. <MAT> where the training process backpropagates the per-speaker, per-time step VAD loss of Equation <NUM> as independent binary classification tasks. The DIVE system <NUM> may be evaluated in terms of a diarization error rate (DER) of the diarization results <NUM>. In some examples, the training process applies a collar that provides a tolerance around speaker boundaries such that the training VAD loss of Equation <NUM> does not penalize the VAD <NUM> for small annotation errors in the training data. In some examples, a typical value for the tolerance representing the collar is about <NUM> on each side of a speaker turn boundary (<NUM>) specified in the labeled training data. Accordingly, the training process may compute a masked VAD loss by removing VAD losses associated with frames/time steps that fall inside the collar from the total loss as follows. <MAT> where Br includes the set of audio frames/time steps that lie within a radius r around speaker turn boundaries. The training process may backpropagate the masked VAD loss computed by Equation <NUM>. During training, a total loss for the DIVE system <NUM> may be computed for jointly training the temporal encoder <NUM>, the iterative speaker selector <NUM>, and the VAD <NUM> as follows. <MAT> The total loss may similarly be computed without application of the collar loss by substituting the VAD loss of Equation <NUM>.

The iterative speaker selector <NUM> may be trained based on the speaker selector loss represented by Equation <NUM> and the VAD <NUM> may be trained based on the VAD loss represented by Equation <NUM> or Equation <NUM> when the training collar is applied. That is, the training process <NUM> may include a collar-aware training process that removes losses associated with any of the training temporal embeddings <NUM> that fall inside a radius around speaker turn boundaries. The collar-aware training process does not penalize or train the DIVE system <NUM> on small annotation errors. For example, the radius around speaker turn boundaries may include <NUM> on each side of the speaker turn boundary (<NUM> in total). Thus, the DIVE system <NUM> may be trained on a total loss computed by Equation <NUM>.

The separate components <NUM>, <NUM>, <NUM> of the DIVE system <NUM> may include respective neural networks such that the training process <NUM> generates hidden nodes, weights of connections between the hidden nodes and input nodes that correspond to the to the fully-labeled training data <NUM>, weights of connections between the hidden nodes and output nodes, and weights of connections between layers of the hidden nodes themselves to minimize the losses of Equations <NUM>, <NUM>, <NUM>, and <NUM>. Thereafter, during inference <NUM>, the fully trained DIVE system <NUM> may be employed against input data (e.g., raw audio signals <NUM>) to generate unknown output data (e.g., voice activity indicators <NUM>) corresponding diarization results <NUM>.

<FIG> shows a plot <NUM> of raw diarization error rate (DER) (%) evaluation for a standard training process and a collar-aware training process used to train the DIVE system <NUM>. In plot <NUM>, the standard training process outperforms the collar-aware training process when using the raw DER (%) evaluation. <FIG> shows a plot <NUM> of the collar-aware DER evaluation that includes a collar of <NUM> applied on each side of speaker turn boundaries for the standard training process and the collar-aware training process. Here, the <NUM> is applied on each side of the speaker turn boundaries according to Equation <NUM>. Notably, the collar-aware training process outperforms the standard training process when evaluating using the collar-aware DER evaluation. Thus, <FIG> illustrates that when the evaluation technique includes the collar-aware DER evaluation it is beneficial to integrate the collar-aware training to train the DIVE system <NUM>.

<FIG> is a flowchart of an exemplary arrangement of operations for a method <NUM> of performing speaker diarization on a received utterance <NUM> of speech. The data processing hardware <NUM>, <NUM> may execute the operations for the method <NUM> by executing instructions stored on the memory hardware <NUM>, <NUM>. At operation <NUM>, the method <NUM> includes receiving an input audio signal <NUM> corresponding to utterances <NUM> spoken by multiple speakers <NUM>, 10a-n. At operation <NUM>, the method <NUM> includes encoding the input audio signal <NUM> into a sequence of T temporal embeddings <NUM>, 220a-t. Here, each temporal embedding <NUM> is associated with a corresponding time step t and represents speech content extracted from the input audio signal <NUM> at the corresponding time step t.

During each of a plurality of iterations i that each correspond to a respective speaker <NUM> of the multiple speakers <NUM>, the method <NUM>, at operation <NUM>, includes selecting a respective speaker embedding <NUM>, 240a-n for the respective speaker <NUM>. For each temporal embedding <NUM> in the sequence of T temporal embeddings <NUM>, the method <NUM>, at operation <NUM>, includes determining a probability (e.g., confidence c) that the corresponding temporal embedding <NUM> includes a presence of voice activity by a single new speaker <NUM> for which a speaker embedding <NUM> was not previously selected during for during a previous iteration i. At operation <NUM>, the method <NUM> includes selecting the respective speaker embedding <NUM> for the respective speaker <NUM> as the temporal embedding <NUM> in the sequence of T temporal embeddings <NUM> associated with the highest probability for the presence of voice activity by the single new speaker <NUM>. At operation <NUM>, the method <NUM>, at each time step t, includes predicting a respective voice activity indicator <NUM> for each respective speaker <NUM> of the multiple speakers <NUM> based on the respective speaker embeddings <NUM> selected during the plurality of iterations i and the temporal embedding <NUM> associated with the corresponding time step t. Here, the respective voice activity indicator <NUM> indicates whether a voice of the respective speaker <NUM> is active or inactive at the corresponding time step t.

The processor <NUM>, i.e., the data processing hardware <NUM>, <NUM> of <FIG>, can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM>, i.e., the memory hardware <NUM>, <NUM> of <FIG>, or on the storage device <NUM>, i.e., the memory hardware <NUM>, <NUM> of <FIG>, to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display <NUM> coupled to high speed interface <NUM>. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

For example, it may be implemented as a standard server 700a or multiple times in a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

Claim 1:
A computer-implemented method (<NUM>) when executed on data processing hardware (<NUM>) causes the data processing hardware (<NUM>) to perform operations comprising:
receiving an input audio signal (<NUM>) corresponding to utterances (<NUM>) spoken by multiple speakers (<NUM>);
encoding the input audio signal (<NUM>) into a sequence of T temporal embeddings (<NUM>), each temporal embedding (<NUM>) associated with a corresponding time step of a plurality of time steps and representing speech content extracted from the input audio signal (<NUM>) at the corresponding time step;
during each of a plurality of iterations each corresponding to a respective speaker of the multiple speakers (<NUM>), selecting a respective speaker embedding (<NUM>) for the respective speaker by:
for each temporal embedding (<NUM>) in the sequence of T temporal embeddings (<NUM>), determining a probability that the corresponding temporal embedding (<NUM>) includes a presence of voice activity by a single new speaker for which a speaker embedding (<NUM>) was not previously selected during a previous iteration; and
selecting the respective speaker embedding (<NUM>) for the respective speaker as the temporal embedding (<NUM>) in the sequence of T temporal embeddings (<NUM>) associated with the highest probability for the presence of voice activity by the single new speaker; and
at each time step in the plurality of time steps, predicting a respective voice activity indicator (<NUM>) for each respective speaker of the multiple speakers (<NUM>) based on the respective speaker embeddings (<NUM>) selected during the plurality of iterations and the temporal embedding (<NUM>) associated with the corresponding time step, the respective voice activity indicator (<NUM>) indicating whether a voice of the respective speaker is active or inactive at the corresponding time step.