A method of verifying an utterance comprising: receiving, by a computing device, an utterance from a user; accessing, by the computing device, a trained neural network for generating a speaker representation for the received utterance, the neural network comprising a long short-term memory, LSTM, layer and a fully-connected linear layer, the LSTM layer configured to receive as input data that characterizes the received utterance, the data comprising a plurality of frames, the plurality of frames having been segmented from the utterance, and the fully-connected linear layer configured to: receive, as input, an output of the LSTM layer; and generate, as output, a speaker representation for the received utterance; providing, as input to the trained neural network, the data that characterizes the received utterance to obtain a speaker representation that indicates distinctive characteristics of the user's voice; accessing, by the computing device, a speaker model; comparing the speaker representation with the speaker model and determining a measure of similarity between the speaker model and the speaker representation for the utterance, and based on the measure of similarity, the speaker model either authenticates the user if the user's voice is sufficiently similar to an enrolled user's voice, or generates an indication that the user is not authenticated if a similarity of the user's voice does not meet at least a threshold similarity score with respect to the enrolled user's voice; and taking an action based on whether or not the user is authenticated. receiving, by a computing device, an utterance from a user; accessing, by the computing device, a trained neural network for generating a speaker representation for the received utterance, the neural network comprising a long short-term memory, LSTM, layer and a fully-connected linear layer, the LSTM layer configured to receive as input data that characterizes the received utterance, the data comprising a plurality of frames, the plurality of frames having been segmented from the utterance, and the fully-connected linear layer configured to: receive, as input, an output of the LSTM layer; and generate, as output, a speaker representation for the received utterance; receive, as input, an output of the LSTM layer; and generate, as output, a speaker representation for the received utterance; providing, as input to the trained neural network, the data that characterizes the received utterance to obtain a speaker representation that indicates distinctive characteristics of the user's voice; accessing, by the computing device, a speaker model; comparing the speaker representation with the speaker model and determining a measure of similarity between the speaker model and the speaker representation for the utterance, and based on the measure of similarity, the speaker model either authenticates the user if the user's voice is sufficiently similar to an enrolled user's voice, or generates an indication that the user is not authenticated if a similarity of the user's voice does not meet at least a threshold similarity score with respect to the enrolled user's voice; and taking an action based on whether or not the user is authenticated.