A method for training a recurrent neural network (RNN) model, the method being implemented by one or more processors, the method comprising: identifying a training instance that comprises: a sequence of audio data frames for corresponding audio data, and ground truth speaker labels for each of the audio data frames; for each of the audio data frames, and in the sequence: applying frame features for the audio data frame as input to an RNN model , and processing the frame features for the audio data frame using the RNN model to generate output that includes a corresponding probability for each of the ground truth speaker labels; generating a loss as a function of: the generated probabilities for the audio data frames, and the ground truth speaker labels; and updating one or more weights of the RNN model based on the generated loss to generate a trained RNN model, wherein generating the loss comprises comparing the generated probabilities to the ground truth speaker labels, by calculating: L = ∑ i = 1 N ∑ i − j N ∑ k = 1 N ! M k ⊙ P ̃ wherein: L is the loss; N is a quantity of audio data frames in the sequence of audio data frames ; i is a row index; j is a column index; M k are binary mappings from the ground truth speaker labels to the generated probabilities for the audio data frames ; k represents an index of the binary mappings; P is a cross-entropy tensor between the generated probabilities for audio data frames and the ground truth speaker labels ; and ⊙ represents matrix element-wise multiplication. identifying a training instance that comprises: a sequence of audio data frames for corresponding audio data, and ground truth speaker labels for each of the audio data frames; a sequence of audio data frames for corresponding audio data, and ground truth speaker labels for each of the audio data frames; for each of the audio data frames, and in the sequence: applying frame features for the audio data frame as input to an RNN model , and processing the frame features for the audio data frame using the RNN model to generate output that includes a corresponding probability for each of the ground truth speaker labels; applying frame features for the audio data frame as input to an RNN model , and processing the frame features for the audio data frame using the RNN model to generate output that includes a corresponding probability for each of the ground truth speaker labels; generating a loss as a function of: the generated probabilities for the audio data frames, and the ground truth speaker labels; and the generated probabilities for the audio data frames, and the ground truth speaker labels; and updating one or more weights of the RNN model based on the generated loss to generate a trained RNN model, wherein generating the loss comprises comparing the generated probabilities to the ground truth speaker labels, by calculating: L = ∑ i = 1 N ∑ i − j N ∑ k = 1 N ! M k ⊙ P ̃ wherein: L is the loss; N is a quantity of audio data frames in the sequence of audio data frames ; i is a row index; j is a column index; M k are binary mappings from the ground truth speaker labels to the generated probabilities for the audio data frames ; k represents an index of the binary mappings; P is a cross-entropy tensor between the generated probabilities for audio data frames and the ground truth speaker labels ; and ⊙ represents matrix element-wise multiplication. L is the loss; N is a quantity of audio data frames in the sequence of audio data frames ; i is a row index; j is a column index; M k are binary mappings from the ground truth speaker labels to the generated probabilities for the audio data frames ; k represents an index of the binary mappings; P is a cross-entropy tensor between the generated probabilities for audio data frames and the ground truth speaker labels ; and ⊙ represents matrix element-wise multiplication. <EOS>