Patent Description:
Speaker verification generally relates to verifying the identity of a person based on characteristics of the person's voice. Some computing devices allow a user to "enroll" with the device by providing to the device one or more samples of speech spoken by the user, from which a speaker model representing the user's voice is determined. Subsequent speech samples received at the device may then be processed and evaluated with respect to the speaker model to verify a user's identity.

"Modeling Speaker Variability Using Long Short- Term Memory Networks for Speech Recognition" (by Li Xiangang et al) may also be relevant.

The invention is defined by the appended independent claims, with the dependent claims providing further preferred embodiments.

This document generally describes systems, methods, devices, and other techniques for training and using neural networks, or other types of models, for speaker verification. In some implementations, the neural network may be a component of a speaker verification model that is accessible by a computing device performing speaker verification. Generally, the neural network may be trained in iterations that each simulate speaker enrollment and verification of an utterance. For example, in each training iteration, a speaker representation generated by the neural network for a given utterance may be evaluated with respect to a speaker model. Based on a comparison of the speaker representation for a simulated verification utterance to a combination (e.g., average) of speaker representations for one or more simulated enrollment utterances, the parameters of the neural network may be updated so as to optimize the ability of the speaker verification model to classify a given utterance as having been spoken by the same person or by a different person than an enrolled person. It will be appreciated that this has advantages in terms of increasing the reliability of the system. The neural network may be further configured to process data characterizing an entire utterance in a single pass through the neural network, rather than processing frames of the utterance individually or sequentially. These and other implementations are described more fully below, and depicted in the Figures.

<FIG> is a schematic diagram of an example system <NUM> for training a neural network for a speaker verification model, and for carrying out a process of speaker verification using the model. Generally, speaker verification is the task of accepting or rejecting the identity claim of a speaker based on characteristics of the speaker's voice, as determined from one or more utterances of the speaker. As depicted in <FIG>, speaker verification can generally include three phases, namely (i) training of a neural network for the speaker verification model, (ii) enrollment of a new speaker, and (iii) verification of the enrolled speaker.

The system <NUM> includes a client device <NUM>, a computing system <NUM>, and a network <NUM>. In some implementations, the computing system <NUM> may provide a speaker verification model <NUM> based on a trained neural network <NUM> to the client device <NUM>. In some implementations, the speaker verification model <NUM> may be pre-installed on the client device <NUM>, for example, as a component of an operating system or application. In other implementations, the speaker verification model <NUM> may be received over the network <NUM>. The client device <NUM> may use the speaker verification model <NUM> to enroll the user <NUM> to the speaker verification process. When the identity of the user <NUM> needs to be verified at a later time, the client device <NUM> may receive a speech utterance of the user <NUM> to verify the identity of the user <NUM> using the speaker verification model <NUM>. Because the speaker verification model <NUM> may be stored locally at the client device <NUM>, the client device <NUM> may be able to make a speaker verification decision without communication over the network <NUM>.

Although not shown in <FIG>, in some implementations, the computing system <NUM> may store the speaker verification model <NUM> based on the trained neural network <NUM>, rather than or in addition to the neural network <NUM> being stored on the client device <NUM>. In these implementations, the client device <NUM> may communicate with the computing system <NUM> via the network <NUM> to remotely access and use the speaker verification model <NUM> for enrollment of the user <NUM>. When the identity of the user <NUM> needs to be verified at a later time, the client device <NUM> may receive a speech utterance of the user <NUM>, and may communicate with the computing system <NUM> via the network <NUM> to verify the identity of the user <NUM> using the remotely located speaker verification model <NUM>. The computing system <NUM> and the computing device <NUM> may be distinct and physically separate from each other.

In the system <NUM>, the client device <NUM> can be, for example, a desktop computer, laptop computer, a tablet computer, a watch, a wearable computer, a cellular phone, a smart phone, a music player, an e-book reader, a navigation system, or any other appropriate computing device that a user may interact with. In some implementations, the client device <NUM> may be a mobile computing device. The computing system <NUM> can include one or more computers, and may perform functions on individual ones of the computers, or the functions may be distributed for performance across multiple computers. The network <NUM> can be wired or wireless or a combination of both and can include the Internet.

In some implementations, a client device <NUM>, such as a phone of a user, may store a speaker verification model <NUM> locally on the client device <NUM>, allowing the client device <NUM> to verify a user's identity without relying on a model at a remote server (e.g., the computing system <NUM>) for either the enrollment or the verification process, and therefore may save communication bandwidth and time. Moreover, in some implementations, when enrolling one or more new users, the speaker verification model <NUM> described here does not require any retraining of the speaker verification model <NUM> using the new users, which can also be computationally efficient. In other implementations, utterances of a given user that are provided for enrollment, verification, or both, may be provided to the computing system <NUM> and added to the training data so that the neural network (and thus the speaker verification model) may be regularly updated based using newly collected training data.

It is desirable that the size of the speaker verification model <NUM>, including the trained neural network <NUM>, be compact because the storage and memory space on the client device <NUM> may be limited. As described below, the speaker verification model <NUM> is based on a trained neural network <NUM>. The speaker verification model <NUM> may include the neural network <NUM> to generate, based on data that characterizes an utterance, a speaker representation that indicates distinctive features of the voice of a speaker of the utterance. The speaker verification model <NUM> may include further components to process the speaker representation and to determine whether the voice of the speaker of the utterance is sufficiently similar to the voice of an enrolled user, such that an identity claim of the speaker of the utterance can be verified.

In some implementations, the neural network may be trained using a large set of training data. Various techniques may be applied during pre-processing of the training data, during training itself, or during a post-training stage to enforce and/or reduce a size of the neural network so as to achieve a compact model size. For example, the speaker verification model <NUM> may be constructed by selecting only certain layers of the neural network <NUM>, which may result in a compact speaker verification model suitable for storage on the client device <NUM>. Enrollment may be performed without a softmax or other classification layer in generating the speaker representations for the speaker model.

<FIG> also illustrates an example flow of data, shown in stages (A) to (F). Stages (A) to (F) may occur in the illustrated sequence, or they may occur in a sequence that is different than in the illustrated sequence. In some implementations, one or more of the stages (A) to (F) may occur offline, where the computing system <NUM> may perform computations when the client device <NUM> is not connected to the network <NUM>. Stages (A) and (B) generally occur during the training phase that was referred to above. Stage (D) generally occurs during the enrollment phase. Stages (E)-(G) generally occur during the verification phase.

At stage (A), the computing system <NUM> selects samples of training utterances to provide to the neural network <NUM> for supervised training of the neural network <NUM>. In some implementations, the utterances in the training samples <NUM> may each consist of one or more predetermined words spoken by many different training speakers, the utterances having been previously recorded and made accessible for use by the computing system <NUM>. Each training speaker may speak a predetermined utterance to a computing device, and the computing device may record an audio signal that includes the utterance. For example, each training speaker may be prompted to speak the training phrase "Hello Phone. " In some implementations, each training speaker may be prompted to speak the same training phrase multiple times. The recorded audio signal of each training speaker may be transmitted to the computing system <NUM>, and the computing system <NUM> may collect the recorded audio signals from many different computing devices and many different training speakers. In some implementations, the neural network <NUM> may be optimized for text-dependent speaker verification, in that a user's identity may be verified based on characteristics of the user's voice determined from an utterance of the pre-defined training phrase. In such implementations, the neural network <NUM> may be trained on utterances that all, or substantially all, include the pre-defined training phrase. In other implementations, the neural network <NUM> may be trained to allow for text-independent speaker verification, in that a user's identity may be verified based on characteristics of the user's voice determined from an utterance of a wide variety of words or phrases, which may not be pre-defined. For example, a user could independently decide which words or phrases that he or she wishes to speak to verify his or her identity, and the speaker verification model based on the trained neural network <NUM> could then authenticate the user given the spoken words or phrases. To allow for text-independent speaker verification, the neural network <NUM> may be trained on utterances of a wide variety of words or phrases spoken by many different training speakers.

At stage (B), the neural network <NUM> may be trained in a manner that parallels the enrollment and verification of users at a client device. Accordingly, the computing system <NUM> can select in each training sample <NUM> a set of simulated enrollment utterances 122b and a simulated verification utterance 122a. The simulated enrollment utterances 122b may all be utterances of the same training speaker, such that a simulated speaker model can be determined for each training sample <NUM>. The simulated verification utterance 122a may be an utterance of the same speaker as the speaker of the simulated enrollment utterances 122b, or may be an utterance of a different speaker. The training samples <NUM> can then be provided to the neural network <NUM>, and a classification can be made based on outputs of the neural network <NUM> as to whether the simulated verification utterance 122a was spoken by the same speaker as the speaker of the simulated enrollment utterances 122b, or by a different speaker from the speaker of the simulated enrollment utterances 122b. The neural network <NUM> can then be updated based on whether the speaker determination was correct. In some implementations, each training sample <NUM> may be labeled as belonging to one of two classes: a matching speakers class 141a (for samples where the speakers of the simulated verification and enrollment utterances are the same) and a non-matching speakers class 141b (for samples where the speakers of the simulated verification and enrollment utterances are different). These labels indicate the ground truth of whether the same speaker spoke the utterances 122a and the utterances 122b. The correctness of the classification of a training sample <NUM> can be determined based on the sample's label. In some implementations, the adjustments to the neural network may not be based strictly on the correctness of a classification of an input sample, but may generally be based on one or more metrics determined from a comparison of speaker representations generated by the neural network for the simulated verification utterance 122a and the simulated enrollment utterances 122b. In some implementations, the training samples <NUM> may be selected from a repository of training data, which may be organized into utterance pools <NUM>. Each of the utterance pools <NUM> may include training utterances that are grouped by the training speaker of the utterances.

The neural network <NUM> may include an input layer for inputting information about the utterances in the training samples <NUM>, and several hidden layers for processing the samples <NUM>. The weights or other parameters of one or more hidden layers may be adjusted so that the trained neural network <NUM> produces output that causes the speaker verification model <NUM> to generate the desired classification of the training samples <NUM> as having either matching or non-matching speakers among the simulated verification and enrollment utterances of the samples <NUM>. In some implementations, the parameters of the neural network <NUM> may be adjusted automatically by the computing system <NUM>. In some other implementations, the parameters of the neural network <NUM> may be adjusted manually by an operator of the computing system <NUM>. The training phase of a neural network is described in more details below in the descriptions of <FIG>, <FIG>, <FIG>, and <FIG>, for example.

At stage (C), once the neural network <NUM> has been trained, a speaker verification model <NUM> based on the trained neural network <NUM> is transmitted from the computing system <NUM> to the client device <NUM>, for example, through the network <NUM>. In some implementations, the trained neural network <NUM>, or a portion thereof, may be a component of the speaker verification model <NUM>. The speaker verification model <NUM> can be configured to verify an identity of the user <NUM> based on characteristics of the user's voice determined from one or more utterances of the user <NUM>. The model <NUM> may be configured to provide data characterizing an utterance of the user <NUM> as input to the trained neural network <NUM>, in order to generate a speaker representation for the user <NUM> that indicates distinctive features of the user's voice. The speaker representation can then be compared to a model of the user's voice that has been previously determined. If the speaker representation is sufficiently similar to the user's speaker model, then the speaker verification model <NUM> can output an indication that the identity of the user <NUM> is valid. In contrast, if the speaker representation is not sufficiently similar to the user's speaker model, then the speaker verification model <NUM> can output an indication that the identity of the user <NUM> is invalid (not verified).

At stage (D), a user <NUM> who desires to enroll his or her voice with the client device <NUM> provides one or more enrollment utterances <NUM> to the client device <NUM> in the enrollment phase. In general, the user <NUM> is not one of the training speakers whose voices were used in training the neural network <NUM>. In some implementations, the client device <NUM> may prompt the user <NUM> to speak an enrollment phrase that is the same phrase spoken by the set of training speakers in the utterances of the training samples <NUM>. In some implementations, the client device <NUM> may prompt the user to speak the enrollment phrase several times, and may record audio signals for the spoken enrollment utterances as the enrollment utterances <NUM>.

The client device <NUM> uses the enrollment utterances <NUM> to enroll the user <NUM> in a speaker verification system of the client device <NUM>. In general, the enrollment of the user <NUM> is done without retraining the neural network <NUM>. Respective instances of the same speaker verification model <NUM> may be used at many different client devices, and for enrolling many different speakers, without requiring that changes be made to the weight values or other parameters in the neural network <NUM>. Because the speaker verification model <NUM> can be used to enroll any user without retraining the neural network <NUM>, enrollment may be performed at the client device <NUM> with limited processing requirements.

In some implementations, information about the enrollment utterances <NUM> is input to the speaker verification model <NUM>, and the speaker verification model <NUM> may output a reference vector or other set of values corresponding to the user <NUM>. The reference vector or other set of values may constitute a speaker model that characterizes distinctive features of the user's voice. The speaker model may be stored on the client device <NUM>, or at a computing system remote from the client device <NUM>, so that speaker representations generated based on utterances later received by the client device <NUM> may be compared against the speaker model to verify whether respective speakers of the later-received utterances are the user <NUM> or are other speakers.

At stage (E), the user <NUM> attempts to gain access to the client device <NUM> using voice authentication. The user <NUM> provides a verification utterance <NUM> to the client device <NUM> in the verification phase. In some implementations, the verification utterance <NUM> is an utterance of the same phrase that was spoken as the enrollment utterance <NUM>. The verification utterance <NUM> is used as input to the speaker verification model <NUM>.

At stage (F), the client device <NUM> determines whether the user's voice is a match to the voice of the enrolled user. In some implementations, the neural network <NUM> may process data that characterizes the verification utterance <NUM>, and may output a speaker representation for the user <NUM> based on the verification utterance <NUM>. In some implementations, the client device <NUM> may compare the speaker representation for the user <NUM> with the speaker model for the enrolled user to determine whether the verification utterance <NUM> was spoken by the enrolled user. The verification phase of a neural network is described in more detail below with respect to <FIG>, for example.

At stage (G), the client device <NUM> provides an indication that represents a verification result <NUM> to the user <NUM>. In some implementations, if the client device <NUM> has accepted the identity of the user <NUM>, the client device <NUM> may send the user <NUM> a visual or audio indication that the verification is successful. In some other implementations, if the client device <NUM> has accepted the identity of the user <NUM>, the client device <NUM> may prompt the user <NUM> for a next input. For example, the client device <NUM> may output a message "Device enabled. Please enter your search" on the display. In some other implementations, if the client device <NUM> has accepted the identity of the user <NUM>, the client device <NUM> may perform a subsequent action without waiting for further inputs from the user <NUM>. For example, the user <NUM> may speak "Hello Phone, search the nearest coffee shop" to the client device <NUM> during the verification phase. The client device <NUM> may verify the identity of the user <NUM> using the verification phrase "Hello Phone. " If the identity of the user <NUM> is accepted, the client device <NUM> may perform the search for the nearest coffee shop without asking the user <NUM> for further inputs. Generally, in some implementations, if the client device <NUM> has accepted the identity of the user <NUM>, the client device <NUM> may respond by transitioning from a locked state, in which one or more capabilities of the client device <NUM> are disabled or blocked, to an unlocked state, in which the capabilities are enabled or otherwise made available to the user <NUM> to access. Similarly, the client device <NUM> may "wake" or transition from a low-power state to a more fully-featured state in response to a successful verification.

In some implementations, if the client device <NUM> has rejected the identity of the user <NUM>, the client device <NUM> may send the user <NUM> a visual or audio indication that the verification is rejected. In some implementations, if the client device <NUM> has rejected the identity of the user <NUM>, the client device <NUM> may prompt the user <NUM> for another utterance attempt. In some implementations, if the number of attempts exceeds a threshold, the client device <NUM> may block the user <NUM> from further attempting to verify his or her identity.

Turning to <FIG>, a block diagram is shown of an example system <NUM> for training a neural network <NUM>. At a completion of the training phase illustrated by <FIG>, the trained neural network <NUM> may be capable of processing data that characterizes an utterance of a speaker, and generating a speaker representation for the speaker that indicates distinctive features of the speaker's voice. The speaker representation may then be used by a speaker verification model in either generating a speaker model for the speaker during the enrollment phase, or in verifying an identity of the speaker during the verification phase.

Generally, <FIG> illustrates that the neural network <NUM> may be trained in a manner that parallels the enrollment and verification phases that later occur at client devices performing a speaker verification task. Unlike some approaches that train the neural network <NUM> to classify training utterances from a finite number of speakers into corresponding classes for each of the speakers, the neural network <NUM> in <FIG> is not trained to determine the particular speaker of a given utterance. Instead, the neural network <NUM> is trained to generate speaker representations that are distinctive and usable to determine whether or not the speaker of a given utterance is the same as the speaker of another set of utterances, without necessarily matching any of the utterances to a specific speaker identity. In this way, the loss function optimized during training is the same function utilized by the speaker verification model during the verification phase. In other words, during verification, a speaker representation based on a verification utterance is compared to a speaker model for an enrolled user. If the speaker representation is sufficiently similar to the speaker model, then an identity of the speaker of the verification utterance is verified. The approach depicted in <FIG> employs similar techniques during training. Namely, a simulated speaker model <NUM> is generated based on speaker representations for one or more enrollment utterances, and a speaker representation <NUM> is also generated for a simulated verification utterance <NUM>. The weight values and other parameters of the neural network <NUM> are adjusted during training so as to minimize the error in classifying the simulated verification utterance <NUM> as being spoken by a same or different speaker as the simulated enrollment utterances 204a-n.

<FIG> depicts a forward pass of a single training iteration based on a sample of training data that includes data characterizing a simulated verification utterance <NUM> and data characterizing one or more simulated enrollment utterances 204a-n. In practice, the neural network <NUM> is trained over many iterations and many different samples of training data. With each iteration, the neural network <NUM> may be adjusted based on results of processing the corresponding sample of training data for the respective iteration. <FIG>, described further below, depict example techniques by which the simulated verification utterance <NUM> and the simulated enrollment utterances 204a-n may be selected. The simulated enrollment utterances 204a-n for a particular sample are generally all utterances spoken by the same training speaker. Although the speaker of the simulated enrollment utterances 204a-n may be different among different samples of training data for different training iterations, within a given training sample for a given training iteration all of the enrollment utterances 204a-n are generally spoken by the same training speaker. The simulated verification utterance <NUM> may have been spoken by the same training speaker as the speaker of the simulated enrollment utterances 204a-n, or may have been spoken by a different training speaker than the speaker of the simulated enrollment utterances 204a-n. For samples of training data in which the speaker is the same among both the simulated verification utterance <NUM> and the simulated enrollment utterances 204a-n, the sample may be labeled as a "matching" sample. For samples of the training data in which the speaker is different among the simulated verification utterance <NUM> and the simulated enrollment utterances 204a-n, the sample may be labeled as a "non-matching" sample. The labels may represent true classifications of the training samples, and may be determined in a pre-processing phase before training. In some implementations, the estimated classification of a training sample based on output of the neural network <NUM> may be compared to the true classification indicated by the label for the training sample to determine whether to adjust the neural network <NUM>.

In some implementations, the data in the training sample may not be the raw audio signals for the simulated verification and enrollment utterances <NUM>, 204a-n. Instead, the utterances <NUM>, 204a-n may have been processed and converted into an appropriate format for processing by the neural network <NUM>. For example, the data in the training sample may characterize respective features of the simulated verification and enrollment utterances <NUM>, 204a-n, rather than the raw audio signals themselves. In some implementations, the data representing each of the simulated utterances <NUM>, 204a-n in the training sample may include one or more log-filterbanks for the respective utterance. In some implementations, each utterance may be segmented in time into a plurality of frames for the utterance, and separate log-filterbanks can be generated for each frame of the utterance. For example, each frame of the utterance may be represented by, say, forty log-filterbanks.

In some implementations, the data characterizing the simulated verification utterance <NUM> and the data characterizing each of the simulated enrollment utterances 204a-n can be processed at once (i.e., in a single pass) through the neural network <NUM>. Thus, even though the training data for a given utterance is segmented into multiple frames that are each represented by a respective set of log-filterbanks, the data characterizing all of the frames for an entirety of the utterance can be inputted into the neural network <NUM> (e.g., as an 80x40 feature vector for <NUM> frames with <NUM> log-filterbanks each) for processing in a single pass through the neural network. This stands in contrast to individually inputting data for each frame of the utterance into the neural network <NUM> for separate processing of the frames. In other implementations, data characterizing individual frames of the utterances <NUM>, 204a-n can be provided as input to the neural network <NUM>, rather than training the neural network <NUM> to process data characterizing an entirety of each utterance <NUM>, 204a-n in a single pass through the neural network <NUM>.

In some implementations, the simulated verification and enrollment utterances <NUM>, 204a-n may be pre-processed according to one or more additional techniques. For example, the structure of the neural network <NUM> may require that the training utterances all have a fixed length (e.g., <NUM> seconds of audio). At least some of the utterances <NUM>, 204a-n may thus be the result of cropping longer utterances to a uniform length, and/or padding some shorter utterances to make longer clips. In other implementations, however, the neural network <NUM> may be capable of processing variable length utterances, in which case the utterances <NUM>, 204a-n in the training data may not be cropped or padded to a fixed length. The audio for the utterances <NUM>, 204a-n may also have been equalized, and noise may have been added or removed from the training utterances <NUM>, 204a-n to ensure that the neural network is trained to perform robustly in the presence of noise.

The portion of the system <NUM> within dashed-line box <NUM> simulates the enrollment phase of a speaker verification process, in that data characterizing a plurality of simulated enrollment utterances 204a-n are used to generate a simulated speaker model <NUM> for the particular training speaker of the simulated enrollment utterances 204a-n. The respective data characterizing each of the simulated enrollment utterances 204a-n is separately inputted into the neural network <NUM> at an input layer of the neural network <NUM>. The neural network <NUM> processes the data through one or more hidden layers, and generates a respective speaker representation 210a-n for each of the simulated enrollment utterances 204a-n. For example, as shown in <FIG>, speaker representation <NUM> (210a) is generated by the neural network <NUM> based on simulated enrollment utterance <NUM> (204a). Likewise, speaker representation <NUM> (210b) is generated by the neural network <NUM> based on simulated enrollment utterance <NUM> (204b). A speaker representation can thus be generated by the neural network <NUM> for each of the simulated enrollment utterances 204a-n. In some implementations, the speaker representations 210a-n may be generated by serially processing each of the simulated enrollment utterances 204a-n through the neural network <NUM>. In some implementations, the speaker representations 210a-n can be generated concurrently by parallel processing the data that characterizes the utterances 204a-n with respective instances of the neural network <NUM> for each of the simulated enrollment utterances 204a-n. The speaker representations 210a-n generally each include a collection of values that represent distinctive characteristics of the simulated-enrollment training speaker's voice, as determined by the neural network <NUM> based on a corresponding one of the simulated enrollment utterances 204a-n. In some implementations, the speaker representations 210a-n may indicate the weight values or other parameters of a last hidden layer of the neural network <NUM>. In some implementations, the speaker representations 210a-n may be a final output of the neural network <NUM> when the neural network <NUM> is configured without a softmax output layer.

To generate the simulated speaker model <NUM>, the speaker representations 210a-n can be averaged, as shown in box <NUM> of <FIG>. Accordingly, the simulated speaker model <NUM> may define a collection of values that represent the distinctive characteristics of the voice of the training speaker of the simulated enrollment utterances 204a-n. By averaging multiple speaker representations 210a-n to determine the simulated speaker model <NUM>, variations in the speaker's voice among the different simulated enrollment utterances 204a-n can be smoothed. The simulated speaker model <NUM> may thus be a more reliable representation of the speaker's voice than any of the individual speaker representations 210a-n, which may individually reflect idiosyncrasies of a given simulated enrollment utterance 204a-n.

In some implementations, the total number of simulated enrollment utterances 204a-n in each sample of training data for each training iteration may vary. For example, a first training sample for a first training iteration may include <NUM> simulated enrollment utterances 204a-n. A second training sample for a second training iteration, however, may include only <NUM> simulated enrollment utterances 204an. In other implementations, the total number of simulated enrollment utterances 204a-n in each sample of training data for each training iteration may be fixed. For example, the neural network <NUM> may be trained over a series of iterations in which the set of training data for each iteration includes a total of <NUM> simulated enrollment utterances 204a-n. In some implementations, one, some, or all of the training iterations may be performed with training samples that include just a single simulated enrollment utterance 204a-n.

In the same manner that the speaker representations 210a-n were generated from the data that characterizes the simulated enrollment utterances 204a-n, a speaker representation <NUM> can be generated from data that characterizes the simulated verification utterance <NUM>. The data that characterizes the simulated verification utterance <NUM> (e.g., log-filterbank values characterizing features of the verification utterance <NUM>) can be provided to an input layer of the neural network <NUM>. The neural network <NUM> then processes the input through one or more hidden layers of the network. The output of the neural network <NUM> is a speaker representation <NUM> that defines a collection of values indicating distinctive characteristics of a voice of a speaker who spoke the simulated verification utterance <NUM>.

To further parallel the verification phase during training of the neural network <NUM>, the speaker representation <NUM> based on the simulated verification utterance <NUM> can be compared to the simulated speaker model <NUM> in the same manner that would occur on a client device, for example, by the speaker verification model during the verification phase. In some implementations, the comparison can be performed by taking the cosine distance (as shown in block <NUM>) of (<NUM>) a first vector defining the collection of values for the simulated speaker representation <NUM> and (<NUM>) a second vector defining the collection of values for the simulated speaker model <NUM>. A logistic regression <NUM> can then be applied to the distance to estimate whether the training speaker who spoke the simulated verification utterance <NUM> is the same or different than the training speaker who spoke the simulated enrollment utterances 204a-n. This is represented in <FIG> by a first block 220a for a matching speakers class, and a second block 220b for a non-matching speakers class. In some implementations, classification techniques other than a logistic regression <NUM> may be applied to make a determination as to whether the training speaker who spoke the simulated verification utterance <NUM> is the same or different than the training speaker who spoke the simulated enrollment utterances 204a-n. For example, a hinge layer or a softmax layer may be used for the classification in some alternatives. In a two-class model like that shown in <FIG>, the softmax and logistic regression techniques may use a same or similar optimization function.

The weight values or other parameters of the neural network <NUM> can then be adjusted, as represented by block <NUM>, based on a result of the comparison of the speaker representation <NUM> for the simulated verification utterance <NUM> with the simulated speaker model <NUM>. For example, if the training sample were labeled as truly having non-matching speakers, incorrectly classified the training sample as having matching speakers, then the neural network <NUM> may be automatically adjusted to correct the error. More generally, the neural network <NUM> may be optimized so as to maximize the similarity score for matching speakers samples or to optimize a score output by the logistic regression, and the neural network <NUM> may also be optimized so as to minimize the similarity score for non-matching speakers samples or to optimize the score output by the logistic regression. In some implementations, adjustments to the neural network <NUM> can occur in response to the results of each training sample for each training iteration, or the neural network <NUM> may be adjusted based on the results of only some of the training iterations. In some implementations, the neural network <NUM> may be adjusted so as to maximize the distance (i.e., maximize differences) between the speaker representation <NUM> and the simulated speaker model <NUM> for non-matching speakers, while minimizing the distance (i.e., minimize differences) between the speaker representation <NUM> and the simulated speaker model <NUM> for matching speakers. Note that, in some implementations, a hard decision to classify a training sample as belonging to either the matching speakers class 220a or the non-matching speakers class 220b may not be made during the training phase. Rather, the neural network <NUM> may be adjusted in a manner that optimizes the scores output by the logistic regression layer <NUM>, or that optimizes one or more other metrics.

Referring now to <FIG>, a flowchart is shown of an example process <NUM> for training a neural network that may be used in a speaker verification model. In some implementations, the process <NUM> may be carried out by the computing systems described herein, such as the computing system <NUM> from <FIG> and the computing system <NUM> from <FIG>.

The process <NUM> commences at stage <NUM>, where a first set of training data is selected (i.e., a first training sample). The first set of training data can include data characterizing a simulated verification utterance and data characterizing one or multiple simulated enrollment utterances. The utterances in the training set are "simulated" in that they are used in the training process in a manner that parallels, or "simulates," the enrollment and verification phases of speaker verification during the training phase. However, the utterances themselves are generally real snippets of recorded speech spoken by training speakers. The training speakers are generally not the same speakers who provide utterances during the actual enrollment and verification phases of the speaker verification process. <FIG>, which are described further below, depict example techniques for selecting the simulated verification and enrollment utterances.

The selected set of training data (i.e., the selected sample) may be labeled according to whether it represents speech of matching speakers or a sample for non-matching speakers. If the speaker of the simulated verification utterance is the same as the speaker of the simulated enrollment utterances, then the set of training data is labeled as a matching speaker sample. If the speaker of the simulated verification utterance is different from the speaker of the simulated enrollment utterances, then the set of training data is labeled as a non-matching speaker sample. In some implementations, the labels can be used later in the training process <NUM> to determine whether an estimated classification of the set of training data as either being a matching or non-matching sample is accurate or not.

In some implementations, the selected set of training data may include not the raw audio signal for the simulated verification and enrollment utterances, but instead data that characterizes features of the utterances. For example, each utterance represented in the set of training data can be characterized by a set of log-filterbanks determined for fixed-length frames of the utterance. The log-filterbanks for each frame of the utterance may then be concatenated into a single set of input values that are provided as input to the neural network and that characterize an entirety of the utterance.

At stages <NUM> and <NUM> of the process <NUM>, speaker representations are determined for each of the utterances characterized in the first set of training data. The speaker representations can each be a collection of values that indicate distinctive features of a voice of the training speaker who spoke the corresponding utterance for the respective speaker representation. For example, a first speaker representation may be generated based on the simulated verification utterance, and respective second speaker representations may be generated based on each of the simulated enrollment utterances. To generate the speaker representations, the data characterizing an utterance is provided to an input layer of the neural network being trained. The neural network then processes the input data through one or more hidden layers of the network. The speaker representation is then an output of the neural network. In some implementations, the output is output at an output layer that is not a softmax layer. The final layer providing the output may be a fully connected, linear layer. In some implementations, the speaker representation may include the values generated at or activations of a last hidden layer of the neural network, rather than the output of a sofmax output layer. The neural network may be configured without a softmax output layer in some implementations.

At stage <NUM>, the speaker representations corresponding to the simulated enrollment utterances are combined to create a simulated speaker model. The simulated speaker model can be an average of the speaker representations for the simulated enrollment utterances. By averaging the speaker representations, a reliable model characterizing the voice of the training speaker can be determined. For example, variations in the manner that the speaker spoke each of the simulated enrollment utterances may be smoothed so that the speaker model can be used a robust baseline to which the speaker representation for the simulated verification utterance is compared. In some implementations, the process <NUM> may select only a subset of the speaker representations for the simulated enrollment utterances to combine in generating the simulated speaker model. For example, a measure of quality of each of the simulated enrollment utterances or the corresponding simulated enrollment utterances may be determined. The process <NUM> may then select only those speaker representations that meet a threshold quality score, or those speaker representations whose corresponding utterances meet a threshold quality score, for inclusion in the set of representations used to generate the simulated speaker model.

At stage <NUM>, the speaker representation for the simulated verification utterance is compared to the simulated speaker model. In some implementations, a binary classifier is used to classify the data sample as representing matching speakers or not. In some implementations, the comparison can include determining a measure of similarity between the speaker representation for the simulated verification utterance and the simulated speaker model. For example, the measure of similarity may be a cosine distance between a vector of values for the speaker representation and a vector of values for the simulated speaker model. The measure of similarity may then be used to estimate a classification of the first set of training data as either a matching speakers sample or a non-matching speakers sample. For example, if the measure of similarity is sufficiently high (e.g., meets a threshold similarity score), then a logistic regression may be used to map the set of training data to a class of matching speakers. On the other hand, if the measure of similarity is too low (e.g., does not meet the threshold similarity score), then the logistic regression may be used to map the set of training data to a class of non-matching speakers.

Next, at stage <NUM>, one or more parameters of the neural network may be adjusted based on a result of the comparison at stage <NUM> between the speaker representation for the simulated verification utterance and the simulated speaker model. For example, the weights of the various nodes in the hidden layers, or other parameters of the neural network may be adjusted so as to increase the distance (reduce the similarity score) between the speaker representation and the simulated speaker model if the training data was labeled as a non-matching speakers sample. Additionally, the weights or other parameters of the neural network may be adjusted to reduce the distance (increase the similarity score) between the speaker representation and the simulated speaker model if the training data was labeled as a matching speakers sample. Generally, as each iteration of the training process <NUM> is intended to simulate a respective enrollment phase and respective verification phase, the neural network may be adjusted to optimize a same loss function as that which is applied during actual enrollment and verification phases during speaker verification. One benefit of this approach is that the neural network is trained to better generate speaker representations that can be used in a speaker verification model for more accurate verification of a speaker's identity. For example, in some implementations, no additional post-processing steps are taken during actual verification of an utterance that are not taken in to account when training the neural network. These techniques may thus be considered an "end-to-end" approach to training the neural network.

Lastly, at stage <NUM>, a next set of training data is selected for another iteration of training the neural network. Again, the set of training data selected at this stage may include data that characterizes a simulated verification utterance and data that characterizes one or more simulated enrollment utterances. The process <NUM> may then repeat stages <NUM>-<NUM>, and continue selecting additional sets of training data for additional training iterations until a limit is reached. In some implementations, the limit may result from expiring all of the available training data. In some implementations, the process <NUM> may continue until a target performance level is reached. For example, after a number of training iterations, the neural network may be tested against a held-out set of data that was not used during the training process <NUM>. Training may continue until tests on the held-out set indicate that the neural network has achieved at least the target performance level.

Referring now to <FIG>, schematic diagrams are shown that illustrate example techniques for selecting sets of training data to use in training a neural network for a speaker verification model. In some implementations, the techniques described with respect to <FIG> can ensure diversity in the training utterances that are selected across many training iterations, which may result in a better performing neural network for a given number of training utterances.

In some implementations, all or a portion of the available training utterances may be clustered into a plurality of groups 410a-n. The groups 410a-n may be further arranged into an utterance pool <NUM> that includes a collection of groups of training utterances. The training utterances may be grouped by speaker in some implementations. For example, group 410a includes a plurality of utterances that were all spoken by a first speaker, whereas group 410n includes a plurality of utterances that were all spoken by another speaker. Accordingly, each of the groups 410a-n may correspond to different speakers. The groups 410a-n may all contain the same number of training utterances, or the number of training utterances may vary among different ones of the groups 410a-n.

For each training iteration, the utterance pool <NUM> may be accessed, and particular utterances may be selected for the sample of training data that will be applied as input in the respective training iteration. For example, <FIG> shows one set of training data that was randomly selected from the utterance pool <NUM> for a training iteration as input sample <NUM>. A first group of utterances, corresponding to a first speaker, can be selected from the groups 410a-n in the utterance pool <NUM> for use in generating the simulated speaker model. The group may be selected randomly or in another manner. From the selected group, e.g., group 410a in <FIG>, a subset of the utterances of the first speaker are selected as simulated enrollment utterances <NUM> in the input sample <NUM>. This subset generally includes multiple utterances, and may include the same or a different number of utterances from one training iteration to another. Utterances from the selected group, e.g., group 410a, may be selected randomly so that different combinations of the utterances are used to generate different simulated speaker models for the first speaker in different training iterations.

An utterance <NUM> is also selected as a simulated verification utterance. The utterance <NUM> may be an utterance of the first speaker or of a different speaker, depending on whether the training iteration is an example of a match or a non-match with the enrollment utterances <NUM>. Both matching and non-matching examples are used in training. As a result, for some training iterations, the utterance <NUM> is an utterance of the first speaker, e.g., an utterance from group 410a. For other training iterations, the utterance <NUM> is an utterance of a second speaker that is different from the first speaker, as shown in <FIG>, so that the input sample <NUM> does not represent a match between the simulated verification utterance <NUM> and the simulated enrollment utterances <NUM>.

In the example of <FIG>, a particular utterance is selected (e.g., randomly selected) from a second group 410n of utterances as the simulated verification utterance <NUM>. In some implementations, the second group of utterances (from which the utterance <NUM> is selected) may be selected randomly from among the groups 410a-n in the utterance pool <NUM>, or according to a pattern of varying selection of the groups 410a-n. In other implementations, a random selection may be made as to whether another utterance from the same speaker as the speaker of the simulated enrollment utterances should be applied as the simulated verification utterance. Thus, perhaps the random selection is biased so that a fifty percent probability exists that the simulated verification utterance <NUM> will be an utterance of the same speaker as the speaker of the simulated enrollment utterances <NUM>. If a result of the random selection is that the input sample <NUM> is to be a matching speaker sample, then the simulated verification utterance <NUM> can be selected from the same group of utterances <NUM> as the group of utterances from which the simulated enrollment utterances <NUM> were selected. But if a result of the random selection is that the input sample <NUM> is to be a non-matching speaker sample, then the simulated verification utterance <NUM> can be selected from a different group of utterances <NUM> corresponding to a different speaker than the group of utterances from which the simulated enrollment utterances <NUM> were selected.

Generally, the selection techniques indicated by <FIG> can allow utterances from different combinations of speakers to be applied in different training iterations. For example, in a first training iteration, the simulated enrollment utterances may have been spoken by a first speaker, and the simulated verification utterance also may have been spoken by the first speaker. In a second training iteration, the simulated enrollment utterances may have been spoken by a second speaker, and the simulated verification utterance may have been spoken by a third speaker. Then in a third training iteration, the simulated enrollment utterances may have been spoken by the first speaker, and the simulated verification utterance may have been spoken by the second speaker. In some implementations, a selection algorithm may be employed that does not randomly select groups of utterances 410a-n, but that instead determinatively selects groups of utterances 410a-n in a manner that creates different permutations or maximizes a number of permutations in the input samples <NUM> between speakers of the simulated verification and enrollment utterances. As a simple example, if three groups of utterances A, B, and C from three different training speakers were available in the utterance pool <NUM>, then nine different input samples <NUM> may be generated for nine training iterations: (A, A), (A, B), (A, C), (B, A), (B, B), (B, C), (C, A), (C, B), and (C, C). Training iterations can also occur with these same pairings of groups, but with different utterances within the groups being selected.

One benefit of the training approach described herein, in contrast to other approaches that involve training a neural network to classify inputs as belonging to a particular speaker among a number of pre-selected speakers, is that a greater number and variety of speakers may be used to train the network. Additionally, there is no minimum number of training utterances that are required for each training speaker to ensure reliable training (other than the one or more utterances that are actually used for each training speaker), because the network is not trained to specific speakers, but is instead trained based on whether a given input sample <NUM> has matching speakers or non-matching speakers among the simulated verification and enrollment utterances.

<FIG> depicts a schematic diagram 400b of a shuffling technique for the selection of utterances for input samples during training of the neural network. As shown in the figure, the samples in a batch of training samples can all come from different pools to obtain better shuffling and diversity of utterances among the training samples in the batch. The shuffling technique may result in more robust and reliable training of the neural network.

Turning to <FIG>, block diagrams are shown of example neural networks <NUM>, <NUM> that may be employed in a speaker verification model. In some implementations, either of the neural networks <NUM>, <NUM> may be used to implement the techniques described with respect to <FIG> and <FIG>, including the training techniques described with respect to <FIG>.

The architecture of the deep neural network <NUM> in <FIG> includes a locally connected layer <NUM>, followed by one or more fully connected hidden layers 506a-n. The locally connected layer <NUM> and fully connected layers 506a-n may have rectified linear units (ReLUs). The last layer of the network <NUM> is a fully connected, linear layer <NUM>, which outputs a speaker representation 510a based on the input utterance (or a frame of an utterance) 503a. The last layer <NUM> before the representation 510a is a linear layer in order to map the non-negative activations into the full space, and to determine projections in some implementations. The full space refers to the notion that ReLu activations can be functions such as y = max(x, <NUM>). Therefore, the activations (y) that form the speaker representation may always be a positive vector. If such an activation function is changed by a linear activation function y = x, then the speaker representation can be made as a vector with potentially positive and negative values. The latter can be a more suitable representation of the speaker when it followed by a cosine distance comparison function, for example.

The configuration of the neural network <NUM> is generally capable of processing fixed length training utterances, or fixed number of frames of utterances. When the neural network <NUM> is trained and later used during runtime in the enrollment and verification phases, utterances may be cropped or padded, as appropriate, to ensure that the utterance has the fixed length required to be processed by the neural network <NUM>. As a result, the neural network <NUM> can compute a speaker representation in a single pass, e.g., a single forward propagation through the deep neural network <NUM>. This allows the speaker representation to be generated with lower latency than techniques that involve sequential processing of different portions of an utterance.

Next, the neural network <NUM> depicted in <FIG> is a recurrent neural network that is used in the method/system according to the invention. Unlike the architecture of neural network <NUM>, the neural network <NUM> is capable of processing variable length input utterances. For example, utterance 503b may be a training utterance, an enrollment utterance, or a verification utterance depending on the context in which the neural network <NUM> is being used. The utterance 503b may be segmented into a plurality of frames, which may have a fixed length. The number of frames inputted to the neural network <NUM> may be a function of the overall length of the utterance 503b. In other words, longer utterances may have more frames, and shorter utterances may have fewer frames. The frames of the utterance 503b are inputted to a long-short-term-memory (LSTM) layer <NUM>. One or more additional hidden layers may follow the LSTM layer <NUM>. The last layer of the network <NUM> is again a fully connected, linear layer <NUM>. The fully connected, linear layer <NUM> may output a speaker representation 510b by mapping the non-negative activations into the full space, and determining projections in some cases. Because the neural network <NUM> is capable of handling variable length utterances, it may be well-suited for text-independent speaker verification in which the words or phrase of an utterance are not pre-defined and may vary among different utterances.

Although the neural networks <NUM> and <NUM> depicted in <FIG> are shown as having particular configurations, the neural networks that may be employed with the techniques described herein are not limited by these examples. For example, the hidden topology of the neural networks may have different numbers and arrangements of layers, which may or may not include fully connected layers, locally connected layers, or any recurrent layers such as long short-term memory layers. The neural network may be a convolutional neural network in some implementations.

<FIG> is a flowchart of an example process <NUM> for verifying an utterance using a speaker verification model and a neural network that has been trained according to the invention. The process <NUM> generally corresponds to the verification phase (stages E-G) depicted in <FIG>. The neural network referred to in <FIG> may be trained according to the techniques described with respect to <FIG>, and has a structure as shown in <FIG>.

At stage <NUM>, an utterance is received from a user of a computing device. For example, a user may wish to unlock his smartphone or perform some other function with a computing device. However, the smartphone may require the user to authenticate himself or herself before the phone will be unlocked, or before the desired function is performed. The authentication may be performed based on characteristics of the user's voice using a speaker verification model on the phone, in some implementations. The phone may prompt the user to speak a verification utterance, which may be received and recorded by the phone at stage <NUM>.

At stage <NUM>, the phone accesses a neural network to generate a speaker representation based on the received utterance. The neural network may be stored locally on the phone, or may be accessed on a remote computing system via an application programming interface (API), for example. The neural network may be trained according to the techniques described herein, and may have been trained based on samples of data that each include a simulated verification utterance and a plurality of simulated enrollment utterances. The neural network may be configured to process, in a single pass through the neural network, data that characterizes an entirety of an utterance. At stage <NUM>, data that characterizes the received utterance is provided as input to the neural network. The neural network processes the input and generates a speaker representation that indicates distinctive characteristics of the user's voice.

At stage <NUM>, a speaker model is accessed on the phone. The speaker model may indicate distinctive features of the voice of an enrolled user. In some implementations, the speaker model may be based on an average of multiple speaker representations generated by the neural network from data that characterizes respective utterances of the enrolled user. At stage <NUM>, the speaker representation that was generated at stage <NUM> based on the verification utterance is compared to the speaker model, or is otherwise evaluated with respect to the speaker model. In some implementations, the comparison or other evaluation is performed by a speaker verification model on the user's phone. The speaker verification model may determine a distance or other measure of similarity between the speaker model and the speaker representation for the verification utterance. Based on the distance or other measure of similarity, the speaker verification model may authenticate the user if the user's voice is sufficiently similar to the enrolled user's voice. Otherwise, the speaker verification model may generate an indication that the user is not authenticated if a similarity of the user's voice does not meet at least a threshold similarity score with respect to the enrolled user's voice.

In some implementations, if the speaker verification model determines with sufficient confidence that the verification utterance was spoken by the enrolled speaker, the speaker model for the enrolled user may then be updated based on the verification utterance. Consider how the device may respond to the following three verification utterances, for example. The similarity score for the first of three verification utterances is below a first threshold value such that the speaker verification model rejects the identity of the user who spoke the first verification utterance (e.g., therefore the device may refuse to unlock in response to the first verification utterance). The similarity score for the second of the three verification utterances may meet the first threshold value such that the identity of the user who spoke the second verification utterance is accepted. However, the similarity score for the second verification utterance is not sufficiently high for the enrolled user's speaker model to be updated based on the second verification utterance. Finally, the similarity score for the third of the verification utterances satisfies the first threshold value, such that the identity of the user who spoke the third verification utterance is accepted (e.g., and a first set of actions such as unlocking a device may be performed), and also satisfies the higher, second threshold value, such that the speaker model for the enrolled user may be updated based on the third verification utterance. The speaker model may be updated by combining (e.g., averaging) the speaker representation generated by the neural network for the third verification utterance with other speaker representations from enrollment utterances of the user that were used to create the speaker model in the first instance.

At stage <NUM>, the phone takes an action based on whether or not the user is authenticated. For example, the phone may wake up or unlock in response to a determination that the user who provided the utterance is the enrolled user. But if the user who provided the utterance is determined to not be the enrolled user, or is not one of a plurality of enrolled users, then the phone may remain locked or may otherwise block performance of one or more functions that the user has selected to perform. In another application, the speaker verification techniques described herein may be employed on a user device (e.g., smartphone, notebook computer, wearable device) to reject speech input detected by the device from non-authorized users (e.g., users whose voices have not been enrolled with the device). For example, when the device is in an unlocked state, the device may listen for voice commands spoken by an authorized user of the device that indicate an action that the user wishes the device to perform (e.g. "Navigate to the football game" or "Play my music collection. In some implementations, the device may only perform the requested action indicated by the voice command if it can be determined that the voice command was spoken by the authorized user. In this way, side speech from other, non-authorized users, for example, may be rejected.

<FIG> shows an example of a computing device <NUM> and a mobile computing device that can be used to implement the techniques described herein. The mobile computing device is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices.

The computing device <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, a high-speed interface <NUM> connecting to the memory <NUM> and multiple high-speed expansion ports <NUM>, and a low-speed interface <NUM> connecting to a low-speed expansion port <NUM> and the storage device <NUM>. Each of the processor <NUM>, the memory <NUM>, the storage device <NUM>, the high-speed interface <NUM>, the high-speed expansion ports <NUM>, and the low-speed interface <NUM>, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a GUI on an external input/output device, such as a display <NUM> coupled to the high-speed interface <NUM>. Also, multiple computing devices may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The computer program product can also be tangibly embodied in a computer- or machine-readable medium, such as the memory <NUM>, the storage device <NUM>, or memory on the processor <NUM>.

The high-speed interface <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low-speed interface <NUM> manages lower bandwidth-intensive operations. In some implementations, the high-speed interface <NUM> is coupled to the memory <NUM>, the display <NUM> (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports <NUM>, which may accept various expansion cards (not shown). In the implementation, the low-speed interface <NUM> is coupled to the storage device <NUM> and the low-speed expansion port <NUM>. The low-speed expansion port <NUM>, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The memory <NUM> stores information within the mobile computing device <NUM>. An expansion memory <NUM> may also be provided and connected to the mobile computing device <NUM> through an expansion interface <NUM>, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory <NUM> may provide extra storage space for the mobile computing device <NUM>, or may also store applications or other information for the mobile computing device <NUM>. Specifically, the expansion memory <NUM> may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory <NUM> may be provide as a security module for the mobile computing device <NUM>, and may be programmed with instructions that permit secure use of the mobile computing device <NUM>.

The memory may include, for example, flash memory and/or NVRAM memory (non-volatile random access memory), as discussed below. The computer program product can be a computer- or machine-readable medium, such as the memory <NUM>, the expansion memory <NUM>, or memory on the processor <NUM>. In some implementations, the computer program product can be received in a propagated signal, for example, over the transceiver <NUM> or the external interface <NUM>.

The mobile computing device <NUM> may communicate wirelessly through the communication interface <NUM>, which may include digital signal processing circuitry where necessary. The communication interface <NUM> may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication may occur, for example, through the transceiver <NUM> using a radio-frequency. In addition, a GPS (Global Positioning System) receiver module <NUM> may provide additional navigation- and location-related wireless data to the mobile computing device <NUM>, which may be used as appropriate by applications running on the mobile computing device <NUM>.

Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

Claim 1:
A method of verifying an utterance comprising:
receiving (<NUM>), by a computing device, an utterance from a user;
accessing (<NUM>), by the computing device, a trained neural network for generating a speaker representation for the received utterance, the neural network comprising a long short-term memory, LSTM, layer and a fully-connected linear layer, the LSTM layer configured to receive as input data that characterizes the received utterance, the data comprising a plurality of frames, the plurality of frames having been segmented from the utterance, and the fully-connected linear layer configured to:
receive, as input, an output of the LSTM layer; and
generate, as output, a speaker representation for the received utterance;
providing (<NUM>), as input to the trained neural network, the data that characterizes the received utterance to obtain a speaker representation that indicates distinctive characteristics of the user's voice;
accessing (<NUM>), by the computing device, a speaker model;
comparing (<NUM>) the speaker representation with the speaker model and determining a measure of similarity between the speaker model and the speaker representation for the utterance, and based on the measure of similarity, the speaker model either authenticates the user if the user's voice is sufficiently similar to an enrolled user's voice, or generates an indication that the user is not authenticated if a similarity of the user's voice does not meet at least a threshold similarity score with respect to the enrolled user's voice; and
taking (<NUM>) an action based on whether or not the user is authenticated.