A computer-implemented method, comprising: selecting, at a computing system, multiple different subsets of training data for training a neural network, each subset of training data comprising a plurality of first components that characterize respective utterances of a first speaker and a second component that characterizes an utterance of the first speaker or a second speaker; and for each of the selected subsets of training data: inputting each of the first components into the neural network to generate a respective first speaker representation corresponding to each of the first components; inputting the second component into the neural network to generate a second speaker representation corresponding to the second component; determining a simulated speaker model for the first speaker based on an average of the respective first speaker representations for the plurality of first components; comparing the second speaker representation with the simulated speaker model to classify the utterance characterized by the second component as an utterance of the first speaker or as an utterance of a speaker other than the first speaker; and adjusting the neural network based on whether the utterance characterized by the second component was correctly classified as an utterance of the first speaker or as an utterance of a speaker other than the first speaker. selecting, at a computing system, multiple different subsets of training data for training a neural network, each subset of training data comprising a plurality of first components that characterize respective utterances of a first speaker and a second component that characterizes an utterance of the first speaker or a second speaker; and for each of the selected subsets of training data: inputting each of the first components into the neural network to generate a respective first speaker representation corresponding to each of the first components; inputting the second component into the neural network to generate a second speaker representation corresponding to the second component; determining a simulated speaker model for the first speaker based on an average of the respective first speaker representations for the plurality of first components; comparing the second speaker representation with the simulated speaker model to classify the utterance characterized by the second component as an utterance of the first speaker or as an utterance of a speaker other than the first speaker; and adjusting the neural network based on whether the utterance characterized by the second component was correctly classified as an utterance of the first speaker or as an utterance of a speaker other than the first speaker. inputting each of the first components into the neural network to generate a respective first speaker representation corresponding to each of the first components; inputting the second component into the neural network to generate a second speaker representation corresponding to the second component; determining a simulated speaker model for the first speaker based on an average of the respective first speaker representations for the plurality of first components; comparing the second speaker representation with the simulated speaker model to classify the utterance characterized by the second component as an utterance of the first speaker or as an utterance of a speaker other than the first speaker; and adjusting the neural network based on whether the utterance characterized by the second component was correctly classified as an utterance of the first speaker or as an utterance of a speaker other than the first speaker.