A method for training a recurrent neural network model, the method being implemented by one or more processors, the method comprising: identifying a training instance that comprises: a sequence of audio data frames for corresponding audio data, and ground truth speaker labels for each of the audio data frames; for each of the audio data frames, and in the sequence: applying frame features for the audio data frame as input to an RNN model, and processing the frame features for the audio data frame using the RNN model to generate output that includes a corresponding probability for each of the ground truth speaker labels; generating a loss as a function of: the generated probabilities for the audio data frames, and the ground truth speaker labels; and updating one or more weights of the RNN model based on the generated loss to generate a trained RNN model, wherein generating the loss comprises comparing the generated probabilities to the ground truth speaker labels, by calculating: L = ∑ i = 1 N ∑ i − j N ∑ k = 1 N ! M k ⊙ P ̃ wherein: L is the loss; N is a quantity of audio data frames in the sequence of audio data frames; i is a row index; j is a column index; M k are binary mappings from the ground truth speaker labels to the generated probabilities for the audio data frames; k represents an index of the binary mappings; P is a cross-entropy tensor between the generated probabilities for audio data frames and the ground truth speaker labels; and ⊙ represents matrix element-wise multiplication. identifying a training instance that comprises: a sequence of audio data frames for corresponding audio data, and ground truth speaker labels for each of the audio data frames; a sequence of audio data frames for corresponding audio data, and ground truth speaker labels for each of the audio data frames; for each of the audio data frames, and in the sequence: applying frame features for the audio data frame as input to an RNN model, and processing the frame features for the audio data frame using the RNN model to generate output that includes a corresponding probability for each of the ground truth speaker labels; applying frame features for the audio data frame as input to an RNN model, and processing the frame features for the audio data frame using the RNN model to generate output that includes a corresponding probability for each of the ground truth speaker labels; generating a loss as a function of: the generated probabilities for the audio data frames, and the ground truth speaker labels; and the generated probabilities for the audio data frames, and the ground truth speaker labels; and updating one or more weights of the RNN model based on the generated loss to generate a trained RNN model, wherein generating the loss comprises comparing the generated probabilities to the ground truth speaker labels, by calculating: L = ∑ i = 1 N ∑ i − j N ∑ k = 1 N ! M k ⊙ P ̃ wherein: L is the loss; N is a quantity of audio data frames in the sequence of audio data frames; i is a row index; j is a column index; M k are binary mappings from the ground truth speaker labels to the generated probabilities for the audio data frames; k represents an index of the binary mappings; P is a cross-entropy tensor between the generated probabilities for audio data frames and the ground truth speaker labels; and ⊙ represents matrix element-wise multiplication. L is the loss; N is a quantity of audio data frames in the sequence of audio data frames; i is a row index; j is a column index; M k are binary mappings from the ground truth speaker labels to the generated probabilities for the audio data frames; k represents an index of the binary mappings; P is a cross-entropy tensor between the generated probabilities for audio data frames and the ground truth speaker labels; and ⊙ represents matrix element-wise multiplication.