A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising: receiving an input audio signal corresponding to utterances spoken by multiple speakers; encoding the input audio signal into a sequence of T temporal embeddings, each temporal embedding associated with a corresponding time step of a plurality of time steps and representing speech content extracted from the input audio signal at the corresponding time step; during each of a plurality of iterations each corresponding to a respective speaker of the multiple speakers, selecting a respective speaker embedding for the respective speaker by: for each temporal embedding in the sequence of T temporal embeddings, determining a probability that the corresponding temporal embedding includes a presence of voice activity by a single new speaker for which a speaker embedding was not previously selected during a previous iteration; and selecting the respective speaker embedding for the respective speaker as the temporal embedding in the sequence of T temporal embeddings associated with the highest probability for the presence of voice activity by the single new speaker; and at each time step in the plurality of time steps, predicting a respective voice activity indicator for each respective speaker of the multiple speakers based on the respective speaker embeddings selected during the plurality of iterations and the temporal embedding associated with the corresponding time step, the respective voice activity indicator indicating whether a voice of the respective speaker is active or inactive at the corresponding time step. receiving an input audio signal corresponding to utterances spoken by multiple speakers; encoding the input audio signal into a sequence of T temporal embeddings, each temporal embedding associated with a corresponding time step of a plurality of time steps and representing speech content extracted from the input audio signal at the corresponding time step; during each of a plurality of iterations each corresponding to a respective speaker of the multiple speakers, selecting a respective speaker embedding for the respective speaker by: for each temporal embedding in the sequence of T temporal embeddings, determining a probability that the corresponding temporal embedding includes a presence of voice activity by a single new speaker for which a speaker embedding was not previously selected during a previous iteration; and selecting the respective speaker embedding for the respective speaker as the temporal embedding in the sequence of T temporal embeddings associated with the highest probability for the presence of voice activity by the single new speaker; and for each temporal embedding in the sequence of T temporal embeddings, determining a probability that the corresponding temporal embedding includes a presence of voice activity by a single new speaker for which a speaker embedding was not previously selected during a previous iteration; and selecting the respective speaker embedding for the respective speaker as the temporal embedding in the sequence of T temporal embeddings associated with the highest probability for the presence of voice activity by the single new speaker; and at each time step in the plurality of time steps, predicting a respective voice activity indicator for each respective speaker of the multiple speakers based on the respective speaker embeddings selected during the plurality of iterations and the temporal embedding associated with the corresponding time step, the respective voice activity indicator indicating whether a voice of the respective speaker is active or inactive at the corresponding time step.