Text-dependent speaker identification

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for speaker verification. The methods, systems, and apparatus include actions of inputting speech data that corresponds to a particular utterance to a first neural network and determining an evaluation vector based on output at a hidden layer of the first neural network. Additional actions include obtaining a reference vector that corresponds to a past utterance of a particular speaker. Further actions include inputting the evaluation vector and the reference vector to a second neural network that is trained on a set of labeled pairs of feature vectors to identify whether speakers associated with the labeled pairs of feature vectors are the same speaker. More actions include determining, based on an output of the second neural network, whether the particular utterance was likely spoken by the particular speaker.

TECHNICAL FIELD

This specification generally relates to speaker verification.

BACKGROUND

In some instances, it may be useful to verify a user's identify based on the user's voice. A user's identity may be verified by voice based on determining that characteristics of a user's voice from a particular utterance match previously stored characteristics of the user's voice.

SUMMARY

According to one innovative aspect of the subject matter described in this specification, speaker verification may be performed using an evaluation vector similarity model. The evaluation vector similarity model may receive an evaluation vector representing characteristics of a speaker's voice from a particular utterance and a reference vector representing characteristics of a particular speaker's voice. The evaluation vector similarity model may then determine whether it is likely that the particular utterance was spoken by the particular speaker.

The evaluation vector similarity model may be a neural network that is trained using labeled pairs of feature vectors representing characteristics of speakers' voices from utterances. The pairs of feature vectors may be labeled as either representing feature vectors for the same speaker or for different speakers.

In some aspects, the subject matter described in this specification may be embodied in methods that may include the actions of inputting speech data that corresponds to a particular utterance to a first neural network and determining an evaluation vector based on output at a hidden layer of the first neural network. Additional actions include obtaining a reference vector that corresponds to a past utterance of a particular speaker. Further actions include inputting the evaluation vector and the reference vector to a second neural network that is trained on a set of labeled pairs of feature vectors to identify whether speakers associated with the labeled pairs of feature vectors are the same speaker. More actions include determining, based on an output of the second neural network, whether the particular utterance was likely spoken by the particular speaker.

Other versions include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

These and other embodiments may each optionally include one or more of the following features. For instance, in some implementations speakers associated with one or more of the labeled pairs of feature vectors are different speakers. In some aspects, a speaker associated with one or more of the labeled pairs of feature vectors is the particular speaker. In certain aspects, actions include inputting the set of labeled pairs of feature vectors to a neural network trainer and obtaining the second neural network based on an output of the neural network trainer.

In some implementations, each of the labeled pairs of the set of labeled pairs of feature vectors comprises a first feature vector, a second feature vector, and a label indicating whether the second neural network is to output that utterances corresponding to the first feature vector and the second feature vector were likely spoken by a same speaker. In some aspects, actions include generating the set of labeled pairs of feature vectors.

In certain aspects, generating the set of labeled pairs of feature vectors includes inputting speech data that corresponds to a first utterance spoken by the particular speaker to the first neural network, in response to inputting the speech data that corresponds to the first utterance spoken by the particular speaker to the first neural network, determining a first feature vector based on output at the hidden layer of the first neural network, inputting speech data that corresponds to a second utterance spoken by the particular speaker to the first neural network, in response to inputting the speech data that corresponds to the second utterance spoken by the particular speaker to the first neural network, determining a second feature vector based on output at the hidden layer of the first neural network, and labeling the first feature vector and the second feature vector with an indication that the second neural network is to output that the utterances corresponding to the first feature vector and the second feature vector were likely spoken by the same speaker.

In some implementations, generating the set of labeled pairs of feature vectors includes determining that a first feature vector and a second feature vector correspond to utterances that are not spoken by the same speaker, in response to determining that the first feature vector and the second feature vector correspond to utterances that are not spoken by the same speaker, generating a third feature vector based on adding a predetermined amount of random noise to the first feature vector, and labeling the first feature vector and the third feature vector with an indication that the second neural network is to output that the utterances corresponding to the first feature vector and the third feature vector were likely spoken by the same speaker.

In certain aspects, the particular utterance corresponds to a hotword. In some aspects, determining, based on the output of the second neural network, whether the particular utterance was likely spoken by the particular speaker includes determining a cosine distance between the evaluation vector and the reference vector, and determining, based on an output of the second neural network and the determined cosine distance, whether the particular utterance was likely spoken by the particular speaker.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other potential features and advantages will become apparent from the description, the drawings, and the claims.

DETAILED DESCRIPTION

FIG. 1is a block diagram of an example system100that can verify a user's identity using a speaker verification model based on a neural network. Briefly, a speaker verification process is the task of accepting or rejecting the identity claim of a speaker based on the information from his/her speech signal. In general, the speaker verification process includes three phases, (i) training of the speaker verification model and an evaluation vector similarity model, (ii) enrollment of a new speaker, and (iii) verification of the enrolled speaker.

The system100includes a client device110, a computing system120, and a network130. In some implementations, the computing system120may provide a speaker verification model144based on a trained neural network142to the client device110and an evaluation vector similarity model146. The client device110may use the speaker verification model144to enroll the user102to the speaker verification process. When the identity of the user102needs to be verified at a later time, the client device110may receive speech utterance of the user102to verify the identity of the user102using the speaker verification model144and the evaluation vector similarity model146.

Although not shown inFIG. 1, in some other implementations, the computing system120may store the speaker verification model144based on the trained neural network142and the evaluation vector similarity model146. The client device110may communicate with the computing system120through the network130to use the speaker verification model144to enroll the user102to the speaker verification process. When the identity of the user102needs to be verified at a later time, the client device110may receive speech utterance of the user102, and communicate with the computing system120through the network130to verify the identity of the user102using the speaker verification model144and the evaluation vector similarity model146.

In the system100, the client device110can be, for example, a desktop computer, laptop computer, a tablet computer, a wearable computer, a cellular phone, a smart phone, a music player, an e-book reader, a navigation system, or any other appropriate computing device. The functions performed by the computing system120can be performed by individual computer systems or can be distributed across multiple computer systems. The network130can be wired or wireless or a combination of both and can include the Internet.

In some implementations, a client device110, such as a phone of a user, may store a speaker verification model144and an evaluation vector similarity model146locally on the client device110, allowing the client device110to verify a user's identity without reaching out to a remote server (e.g., the computing system120) for either the enrollment or the verification process, and therefore may save communications bandwidth and time. Moreover, in some implementations, when enrolling one or more new users, the speaker verification model144and the evaluation vector similarity model146described here do not require any retraining of the speaker verification model144and the evaluation vector similarity model146using the new users, which also is computationally efficient.

It is desirable that the size of the speaker verification model144be compact because the memory space on the client device110may be limited. As described below, the speaker verification model144is based on a trained neural network. The neural network may be trained using a large set of training data, and may generate a large amount of data at the output layer. However, the speaker verification model144may be constructed by selecting only certain layers of the neural network, which may result in a compact speaker verification model suitable for the client device110.

FIG. 1also illustrates an example flow of data, shown in stages (A) to (H). Stages (A) to (H) may occur in the illustrated sequence, or they may occur in a sequence that is different than in the illustrated sequence. In some implementations, one or more of the stages (A) to (H) may occur offline, where the computing system120may perform computations when the client device110is not connected to the network130.

During stage (A), the computing system120obtains a set of training utterances122, and inputs the set of training utterances122to a supervised neural network140. In some implementations, the training utterances122may be one or more predetermined words spoken by the training speakers that were recorded and accessible by the computing system120. Each training speaker may speak a predetermined utterance to a computing device, and the computing device may record an audio signal that includes the utterance. For example, each training speaker may be prompted to speak the training phrase “Hello Phone.” In some implementations, each training speaker may be prompted to speak the same training phrase multiple times. The recorded audio signal of each training speaker may be transmitted to the computing system120, and the computing system120may collect the recorded audio signals and select the set of training utterances122. In other implementations, the various training utterances122may include utterances of different words.

During stage (B), the computing system120uses the training utterances122to train a neural network140, resulting in a trained neural network142. In some implementations, the neural network140is a supervised deep neural network.

During training, information about the training utterances122is provided as input to the neural network140. Training targets124, for example, different target vectors, are specified as the desired outputs that the neural network140should produce after training. For example, the utterances of each particular speaker may correspond to a particular target output vector. One or more parameters of the neural network140are adjusted during training to form a trained neural network142.

For example, the neural network140may include an input layer for inputting information about the training utterances122, several hidden layers for processing the training utterances122, and an output layer for providing output. The weights or other parameters of one or more hidden layers may be adjusted so that the trained neural network produces the desired target vector corresponding to each training utterance122. In some implementations, the desired set of target vectors may be a set of feature vectors, where each feature vector is orthogonal to other feature vectors in the set. For example, speech data for each different speaker from the set of training speakers may produce a distinct output vector at the output layer using the trained neural network.

The neural network that generates the desired set of speaker features may be designated as the trained neural network142. In some implementations, the parameters of the supervised neural network140may be adjusted automatically by the computing system120. In some other implementations, the parameters of the supervised neural network140may be adjusted manually by an operator of the computing system120. The training phase of a neural network is described in more details below in descriptions ofFIGS. 2A, 2B, 2C, and 3.

During stage (C), the computing system120obtains labeled pairs of feature vectors126, and inputs the labeled pairs of feature vectors126to a second supervised neural network141. The labeled pairs of feature vectors may represent characteristics of voices of multiple different speakers. In some implementations, the labeled pairs of feature vectors126may be outputs from inputting speech data corresponding to utterances from multiple different speakers to the speaker verification model144.

Each labeled pair of feature vectors126may include a first feature vector corresponding to characteristics of a speaker's voice corresponding to a first utterance, a second feature vector corresponding to characteristics of a speaker's voice corresponding to a second utterance, and a label indicating if the feature vectors are for the same speaker or different speakers. For example, the feature vectors may be labeled as for the same speaker if the first and second utterances are from the same speaker. In another example, the feature vectors may be labeled as for different speakers if the first and second utterances are from different speakers.

The computing system120uses the labeled pairs of feature vectors126to train a second neural network141, resulting in a second trained neural network143. In some implementations, the second neural network141is a supervised deep neural network. During training, the labeled pairs of feature vectors126are provided as input to the second neural network141. One or more parameters of the second neural network141are adjusted during training to form a second trained neural network143. The one or more parameters of the second neural network141may be adjusted similarly to how the one or more parameters of the neural network140may be adjusted during training, as described above. The training phase of the second neural network141is described in more detail below in descriptions ofFIG. 8.

During stage (D), once the neural network140and the second neural network141have been trained, a speaker verification model144based on the trained neural network142is transmitted from the computing system120to the client device110through the network130. In some implementations, the speaker verification model144may omit one or more layers of the neural network142, so that the speaker verification model144includes only a portion of, or subset of, the trained neural network142. For example, the speaker verification model144may include the input layer and the hidden layers of the trained neural network142, and use the last hidden layer of the trained neural network142as the output layer of the speaker verification model144. As another example, the speaker verification model144may include the input layer of the trained neural network142, and the hidden layers that sequentially follow the input layer, up to a particular hidden layer that has been characterized to have a computation complexity exceeding a threshold.

During stage (E), a user102who desires to enroll her voice with the client device110provides one or more enrollment utterances152to the client device110in the enrollment phase. In general, the user102is not one of the training speakers that generated the set of training utterances122. In some implementations, the user client device110may prompt the user102to speak an enrollment phrase that is the same phrase spoken by the set of training speakers. In some implementations, the client device110may prompt the user to speak the enrollment phrase several times, and record the spoken enrollment utterances as the enrollment utterances152.

The client device110uses the enrollment utterances152to enroll the user102in a speaker verification system of the client device110. In general, the enrollment of the user102is done without retraining the speaker verification model144or any other neural network. The same speaker verification model144may be used at many different client devices, and for enrolling many different speakers, without requiring changing the weight values of other parameters in a neural network. Because the speaker verification model144can be used to enroll any user without retraining a neural network, enrollment may be done at the client device110with limited processing requirements. In some implementations, information about the enrollment utterances152is input to the speaker verification model144, and the speaker verification model144may output a reference vector corresponding to the user102. The output of the speaker verification model144may represent characteristics of the user's voice. The client device100stores this reference vector for later use in verifying the voice of the user102. The enrollment phase of a neural network is described in more details below in descriptions ofFIGS. 4 and 5.

During stage (F), the user102attempts to gain access to the client device110using voice authentication. The user102provides a verification utterance154to the client device110in the verification phase. In some implementations, the verification utterance154is an utterance of the same phrase that was spoken as the enrollment utterance152. The verification utterance154is used as input to the speaker verification model144.

During stage (G), the client device110determines whether the user's voice is a match to the voice of the enrolled user. In some implementation, the speaker verification model144may output an evaluation vector that corresponds to the verification utterance154. In some implementations, the client device110may compare the evaluation vector with the reference vector of the user102using the evaluation vector similarity model144to determine whether the verification utterance154was spoken by the user102. In some implementations, the client device110may input the evaluation vector and the reference vector of the user120into an evaluation vector similarity model144to determine whether the verification utterance154was spoken by the user102. The verification phase of a neural network is described in more details below inFIGS. 6, 7, 8, and 9B.

During stage (H), the client device110provides an indication that represents a verification result156to the user102. In some implementations, if the client device110has accepted the identity of the user102, the client device110may send the user102a visual or audio indication that the verification is successful. In some other implementations, if the client device110has accepted the identity of the user102, the client device110may prompt the user102for a next input. For example, the client device110may output a message “Device enabled. Please enter your search” on the display. In some other implementations, if the client device110has accepted the identity of the user102, the client device110may perform a subsequent action without waiting for further inputs from the user102. For example, the user102may speak “Hello Phone, search the nearest coffee shop” to the client device110during the verification phase. The client device110may verify the identity of the user102using the verification phrase “Hello Phone.” If the identity of the user102is accepted, the client device110may perform the search for the nearest coffee shop without asking the user102for further inputs.

In some implementations, if the client device110has rejected the identity of the user102, the client device110may send the user102a visual or audio indication that the verification is rejected. In some implementations, if the client device110has rejected the identity of the user102, the client device110may prompt the user102for another utterance attempt. In some implementations, if the number of attempts exceeds a threshold, the client device110may disallow the user102from further attempting to verify her identity.

FIG. 2Ais a block diagram of an example neural network200for training a speaker verification model. The neural network200includes an input layer211, a number of hidden layers212a-212k, and an output layer213. The input layer211receives data about the training utterances. During training, one or more parameters of one or more hidden layers212a-212kof the neural network are adjusted to form a trained neural network. For example, one or more hidden layers may be adjusted to obtain different target vectors corresponding to the different training utterances122until a desired set of target vectors are formed. In some implementations, the desired set of target vectors may be a set of feature vectors, where each feature vector is orthogonal to other feature vectors in the set. For example, for N training speakers, the neural network200may output N vectors, each vector corresponding to the speaker features of the one of the N training speakers.

A set of input vectors201for use in training is determined from sample utterances from multiple speakers. In the example, the value N represents the number of training speakers whose speech samples are used for training. The input vectors201are represented as {uA, uB, uC, . . . , uN}. The input vector uArepresents characteristics of an utterance of speaker A, the input vector uArepresents characteristics of an utterance of speaker B, and so on. For each of the different training speakers, a corresponding target vector215A-215N is assigned as a desired output of the neural network in response to input for that speaker. For example, the target vector215A is assigned to Speaker A. When trained, the neural network should produce the target vector215A in response to input that describes an utterance of Speaker A. Similarly, the target vector215B is assigned to Speaker B, the target vector215C is assigned to Speaker C, and so on.

In some implementations, training utterances may be processed to remove noises associated with the utterances before deriving the input vectors201from the utterances. In some implementations, each training speaker may have spoken several utterances of the same training phrase. For example, each training speaker may have been asked to speak the phrase “hello Google” ten times to form the training utterances. An input vector corresponding to each utterance, e.g., each instance of the spoken phrase, may be used during training. As an alternative, characteristics of multiple utterances may be reflected in a single input vector. The set of training utterances201are processed sequentially through hidden layers212a,212b,212c, to212k, and the output layer213.

In some implementations, the neural network200may be trained under machine or human supervision to output N orthogonal vectors. For each input vector201, the output at the output layer213may be compared to the appropriate target vector215A-215N, and updates to the parameters of the hidden layers212a-212kare made until the neural network produces the desired target output corresponding to the input at the input layer211. For example, techniques such as backward propagation of errors, commonly referred to as backpropagation, may be used to train the neural network. Other techniques may additionally or alternatively be used. When training is complete, for example, the output vector215A may be a 1-by-N vector having a value of [1, 0, 0, . . . , 0], and corresponds to the speech features of utterance uA. Similarly, the output vector215B is another 1-by-N vector having a value of [0, 1, 0, . . . , 0], and corresponds to the speech features of utterance uA.

The hidden layers212a-212kcan have various different configurations, as described further with respect toFIGS. 2B and 2Cbelow. For example, rectified linear units may be used as the non-linear activation function on hidden units and a learning rate of 0.001 with exponential decay (0.1 every 5M steps). Alternatively, a different learning rate (e.g., 0.1, 0.01, 0.0001, etc.) or a different number of steps (e.g., 0.1M, 1M, 10M, etc.) may be used.

In some implementations, once the neural network200is trained, a speech verification model may be obtained based on the neural network200. In some implementations, the output layer213may be excluded from the speech verification model, which may reduce the size of the speech verification model or provide other benefits. For example, a speech verification model trained based on speech of 500 different training speakers may have a size of less than 1 MB.

FIG. 2Bis a block diagram of an example neural network200having a hidden layer212athat implements the maxout feature.

In some implementations, the neural network200may be trained as a maxout neural network. Maxout networks differ from the standard multi-layer perceptron (MLP) networks in that hidden units, e.g., nodes or neurons, at each layer are divided into non-overlapping groups. Each group may generate a single activation via the max pooling operation. For example, the hidden layer212ashows four hidden nodes226a-226d, with a pool size of three. Each of the nodes221a,221b, and221cproduces an output, but only the maximum of the three outputs is selected by node226ato be the input to the next hidden layer. Similarly, each of the nodes222a,222b, and222cproduces an output, but only the maximum of the three outputs is selected by node226bto be the input to the next hidden layer.

Alternatively, a different number of layers (e.g., 2, 3, 5, 8, etc.) or a different number of nodes per layer (e.g., 16, 32, 64, 128, 512, 1024, etc.) may be used. A pool size of 2 is used per layer, but the pool size used may be greater or fewer than this, e.g., 1, 3, 5, 10, etc.

FIG. 2Cis a block diagram of an example neural network200having a hidden layer212athat implements a maxout neural network feature using the dropout feature.

In some implementations, the neural network200may be trained as a maxout neural network using dropout. In general, dropout is a useful strategy to prevent over-fitting in neural network fine-tuning when using a small training set. In some implementations, the dropout training procedure may include randomly selecting certain hidden nodes of one or more hidden layers, such that output from these hidden nodes are not provided to the next hidden layer.

In some implementations, dropout techniques are used at fewer than all of the hidden layers. For example, the initial hidden layers may not use dropout, but the final layers may use drop out. As another example, the hidden layer212ashows four hidden nodes226a-226d, with a pool size of three, and a dropout rate of 50 percent. Each of the nodes221a,221b, and221cproduces an output, but only the maximum of the three outputs is selected by node226ato be the input to the next hidden layer. Similarly, each of the nodes222a,222b, and222cproduces an output, but only the maximum of the three outputs is selected by node226bto be the input to the next hidden layer. However, the hidden layer212adrops 50 percent of activations as a result of dropout. Here, only the outputs of nodes226aand226dare selected as input for the next hidden layer, and the outputs of nodes226band226care dropped. As an alternative, at layers where dropout is used, the amount of activations dropped may be, for example, 10 percent, 25 percent, 40 percent, 60 percent, 80 percent, etc.

FIG. 3is a flow diagram that illustrates an example process300for training a speaker verification model. The process300may be performed by data processing apparatus, such as the computing system120described above or another data processing apparatus.

The system receives speech data corresponding to utterances of multiple different speakers (302). For example, the system may receive a set of training utterances. As another example, the system may receive feature scores that indicate one or more audio characteristics of the training utterances. As another example, using the training utterances, the system may determine feature scores that indicate one or more audio characteristics of the training utterances. In some implementations, the feature scores representing one or more audio characteristics of the training utterances may be used as input to a neural network.

The system trains a neural network using the speech data (304). In some implementations, the speech from each of the multiple different speakers may be designated as corresponding to a different output at an output layer of the neural network. In some implementations, the neural network may include multiple hidden layers.

In some implementations, training a neural network using the speech data may include a maxout feature, where for a particular hidden layer of the multiple hidden layers, the system compares output values generated by a predetermined number of nodes of the particular hidden layer, and outputs a maximum output value of the output values based on comparing the output values.

In some implementations, training a neural network using the speech data may include a dropout feature, where for a particular node of a particular hidden layer of the multiple hidden layers, the system determines whether to output an output value generated by the particular node based on a predetermined probability.

The system obtains a speech verification model based on the trained neural network (306). In some implementations, a number of layers of the speech verification model is less than a number of layers of the trained neural network. As a result, the output of the speech verification model is the outputs from a hidden layer of the trained neural network. For example, the speaker verification model may include the input layer and the hidden layers of the trained neural network, and use the last hidden layer of the trained neural network as the output layer of the speaker verification model. As another example, the speaker verification model may include the input layer of the trained neural network, and the hidden layers that sequentially follow the input layer, up to a particular hidden layer that has been characterized to have a computation complexity exceeding a threshold.

FIG. 4is a block diagram of an example speaker verification model400for enrolling a new user. In general, the new user is not one of the training speakers that generated the set of training utterances. In some implementations, a user client device storing the speaker verification model400may prompt the new user to speak an enrollment phrase that is the same phrase spoken by the set of training speakers. Alternatively, a different phrase may be spoken. In some implementations, the client device may prompt the new user to speak the enrollment phrase several times, and record the spoken enrollment utterances as enrollment utterances. The output of the speaker verification model400may be determined for each of the enrollment utterances. The output of the speaker verification model400for each enrollment utterance may be accumulated, e.g., averaged or otherwise combined, to serve as a reference vector for the new user.

In general, given a set of utterances Xs={Os1, Os2, . . . Osn} from a speaker s, with observations Osi={o1, o2, . . . , om}, the process of enrollment may occur as follows. First, every observation ojin utterance Osi, together with its context, may be used to feed a speech verification model. In some implementations, the output of the last hidden layer may then be obtained, normalized, and accumulated for all the observations ojin Osi. The resulting accumulated vector may be referred to as a reference vector associated with the utterance Osi. In some implementations, the final representation of the speaker s may be derived by averaging all reference vectors corresponding for utterances in Xs.

For example, a speaker verification model410is obtained from the neural network200as described inFIG. 2A. The speaker verification model410includes the input layer211, and hidden layers212a-212kof the neural network200. However, the speaker verification model410does not include the output layer213. When speech features for an enrollment utterance402are input to the speaker verification model, the speaker verification model410uses the last hidden layer212kto generate a vector404.

In some implementations, the vector404is used as a reference vector, e.g., a voiceprint or unique identifier that represents characteristics of the user's voice. In some implementations, multiple speech samples are obtained from the user, and a different output vector is obtained from the speaker verification model410for each of the multiple speech samples. The various vectors resulting from the different speech samples can be combined, e.g., averaged or otherwise accumulated, to form a reference vector. The reference vector can serve as a template or standard that can be used to identify the user. As discussed further below, outputs from the speaker verification model410can be compared with the reference vector to verify the user's identity.

Here, the reference vector404is a 1-by-N vector. The reference vector may have the same dimension as any one of the vectors215A-215N, or may have a different dimension, since the reference vector404is obtained from layer212kand not output layer213shown inFIG. 2A. The reference vector404has values of [0, 1, 1, 0, 0, 1 . . . , 1], which represent the particular characteristics of the user's voice. Note that the user speaking the enrollment utterance402is not included in the set of training speakers, and the speech verification model generates a unique reference vector404for the user without retraining the neural network200.

In general, the completion of an enrollment process causes the reference vector404to be stored at the client device in association with a user identity. For example, if the user identity corresponds to an owner or authorized user of the client device that stores the speaker verification model400, the reference vector404can be designated to represent characteristics of an authorized user's voice. In some other implementations, the speaker verification model400may store the reference vector404at a server, a centralized database, or other device.

FIG. 5is a flow diagram that illustrates an example process500for enrolling a new speaker using the speaker verification model. The process500may be performed by data processing apparatus, such as the client device110described above or another data processing apparatus.

The system obtains access to a neural network (502). In some implementations, the system may obtain access to a neural network that has been trained to provide an orthogonal vector for each of the training utterances. For example, a speaker verification model may be, or may be derived from, a neural network that has been trained to provide a distinct 1×N feature vector for each speaker in a set of N training speakers. The feature vectors for the different training speakers may be orthogonal to each other. A client device may obtain access to the speaker verification model by communicating with a server system that trained the speaker verification model. In some implementations, the client device may store the speaker verification model locally for enrollment and verification processes.

The system inputs speech features corresponding to an utterance (504). In some implementations, for each of multiple utterances of a particular speaker, the system may input speech data corresponding to the respective utterance to the neural network. For example, the system may prompt a user to speak multiple utterances. For each utterance, feature scores that indicate one or more audio characteristics of the utterance may be determined. The one or more audio characteristics of the training utterances may then be used as input to the neural network.

The system then obtains a reference vector (506). In some implementations, for each of multiple utterances of the particular speaker, the system determines a vector for the respective utterance based on output of a hidden layer of the neural network, and the system combines the vectors for the respective utterances to obtain a reference vector of the particular speaker. In some implementations, the reference vector is an average of the vectors for the respective utterances.

FIG. 6is a block diagram of an example speaker verification model600for verifying the identity of an enrolled user. As discussed above, a neural network-based speaker verification method may be used for a small footprint text-dependent speaker verification task. As refers to in this Specification, a text-dependent speaker verification task refers to a computation task where a user speaks specific words or phrase that is predetermined. In other words, the input used for verification may be a predetermined word or phrase expected by the speaker verification model. The speaker verification model600may be based on a neural network trained to classify training speakers with distinctive feature vectors. The trained neural network may be used to extract one or more speaker-specific feature vectors from one or more utterances. The speaker-specific feature vectors may be used for speaker verification, for example, to verify the identity of a previously enrolled speaker.

For example, the enrolled user may verify her identity by speaking the verification utterance602to a client device. In some implementations, the client device may prompt the user to speak the verification utterance602using predetermined text. The client device may record the verification utterance602. The client device may determine one or more feature scores that indicate one or more audio characteristics of the verification utterances602. The client device may input the one or more feature scores in the speaker verification model410. The speaker verification model410generates an evaluation vector604. A comparator620compares the evaluation vector604to the reference vector404to verify the identity of the user. In some implementations, the comparator620may generate a score indicating a likelihood that an utterance corresponds to an identity, and the identity may be accepted if the score satisfies a threshold. If the score does not satisfy the threshold, the identity may be rejected.

In some implementations, a cosine distance between the reference vector404and the evaluation vector604may then be computed. A verification decision may be made by comparing the distance to a threshold. In some implementations, the comparator620may be implemented on the client device110. In some other implementations, the comparator620may be implemented on the computing system120. In some other implementations, the comparator620may be implemented on another computing device or computing devices.

In some implementations, the client device may store multiple reference vectors, with each reference vector corresponding to a respective user. Each reference vector is a distinct vector generated by the speaker verification model. In some implementations, the comparator620may compare the evaluation vector604with multiple reference vectors stored at the client device. The client device may determine an identity of the speaker based on the output of the comparator620. For example, the client device may determine that the enrolled user corresponding to a reference vector that provides the shortest cosine distance to the evaluation vector604to be the identity of the speaker, if the shortest cosine distance satisfies a threshold value.

In some implementations, a neural network-based technique and a vector-based technique can be used together to verify speaker identity. The reference vector system and the vector system can each generate a score indicating a likelihood that an utterance corresponds to an identity. The individual scores can be normalized, and the normalized scores may then be summed or otherwise combined to produce a combined score. A decision about the identity can then be made based on comparing the combined score to a threshold. In some instances, the combined use of a vector approach and a reference-vector approach may outperform either approach used individually.

In some implementations, a client device stores a different reference vector for each of multiple user identities. The client device may store data indicating which reference vector corresponds to each user identity. When a user attempts to access the client device, output of the speaker verification model may be compared with the reference vector corresponding to the user identity claimed by the speaker. In some implementations, the output of the speaker verification model may be compared with reference vectors of multiple different users, to identify which user identity is most likely to correspond to the speaker or to determine if any of the user identities correspond to the speaker.

FIG. 7is a flow diagram that illustrates an example process700for verifying the identity of an enrolled user using the speaker verification model. The process700may be performed by data processing apparatus, such as the client device110described above or another data processing apparatus.

The system inputs speech data that correspond to a particular utterance to a neural network (702). In some implementations, the neural network includes multiple hidden layers that are trained using utterances of multiple speakers, where the multiple speakers do not include the particular speaker.

The system determines an evaluation vector based on output at a hidden layer of the neural network (704). In some implementations, the system determines an evaluation vector based on output at a last hidden layer of a trained neural network. In some other implementations, the system determines an evaluation vector based on output at a hidden layer of a trained neural network that optimizes the computation efficiency of a speaker verification model.

The system compares the evaluation vector with a reference vector that corresponds to a past utterance of a particular speaker (706). In some implementations, the system compares the evaluation vector with the reference vector by determining a distance between the evaluation vector and the reference vector. For example, determining a distance between the evaluation vector and the reference vector may include computing a cosine distance between the evaluation vector and the reference vector. In some implementations, the system inputs the evaluation vector and the reference vector into an evaluation vector similarity model, as further described below inFIG. 8.

The system verifies the identity of the particular speaker (708). In some implementations, based on comparing the evaluation vector and the reference vector, the system determines whether the particular utterance was spoken by the particular speaker. In some implementations, the system determines whether the particular utterance was spoken by the particular speaker by determining whether the distance between the evaluation vector and the reference vector satisfies a threshold. In some implementations, the system determines an evaluation vector based on output at a hidden layer of the neural network by determining the evaluation vector based on activations at a last hidden layer of the neural network in response to inputting the speech data.

In some implementations, the neural network includes multiple hidden layers, and the system determines an evaluation vector based on output at a hidden layer of the neural network by determining the evaluation vector based on activations at a predetermined hidden layer of the multiple hidden layers in response to inputting the speech features.

FIG. 8is a block diagram of an example800of speaker verification using an evaluation vector similarity model830. Initially, evaluation vector similarity model training data810may be obtained. The evaluation vector similarity model training data810may be labeled pairs of feature vectors that represent characteristics of voices of multiple different speakers. Each labeled pair of feature vectors may include a first feature vector corresponding to characteristics of a speaker's voice corresponding to a first utterance, a second feature vector corresponding to characteristics of a speaker's voice corresponding to a second utterance, and a label indicating if the feature vectors are for the same speaker or different speakers.

For example, the labeled pairs of feature vectors may include a first pair of feature vectors (FV1, FV2) labeled as for the same speaker and including a first and second feature vector that are respective outputs from the speaker verification model850in response to input of speech data representing an utterance from a first speaker and speech data representing another utterance from the same speaker. The labeled pairs of feature vectors may include a second pair of feature vectors (FV2, FV3) labeled as for different speakers and including a second and third feature vector that are respective outputs from the speaker verification model850in response to input of speech data representing an utterance from the first speaker and speech data representing another utterance from a second speaker. The labeled pairs of feature vectors may include a third pair of feature vectors (FV3, FV4) labeled as for the same speaker and including a third and fourth feature vector that are respective outputs from the speaker verification model850in response to input of speech data representing an utterance from the second speaker and speech data representing another utterance from the same speaker.

The labeled pairs of feature vectors may be obtained by providing existing speech data of utterances labeled by speaker as inputs to the speaker verification model850and pairing the feature vectors output from the speaker verification model850by the labeled speakers. For example, pairs of feature vectors that are output based on speech data labeled as from the same speaker may be labeled as for the same speaker, and pairs of features vectors that are output based on speech data labeled as from different speakers may be labeled as for different speakers. In some implementations, the labels may be associated with similarity scores. For example, a label that indicates that a pair of feature vectors is for the same speaker may be associated with a similarity score of “1.0” and a label that indicates that a pair of feature vectors is for different speakers may be associated with a similarity score of “0.0.”

In some implementations, there may be more utterances spoken by different speakers than utterances spoken by the same speaker. For example, if there are ten speakers and each speaker has two utterances, for each speaker, there may be only a single pair of utterances spoken by the speaker but eighteen pairs of utterances spoken by different speakers. Training an evaluation vector similarity model with fewer pairs of feature vectors that are labeled as for the same speaker may reduce the accuracy of the evaluation vector similarity model. Accordingly, additional pairs of feature vectors that are labeled as for the same speaker may be generated.

An additional pair of feature vectors labeled as from the same speaker may be generated for each pair of feature vectors that is labeled as for different speakers. For example, if there are eighteen pairs of feature vectors labeled as for different speakers, eighteen pairs of additional feature vectors labeled as for the same speaker may be generated. The additional labeled feature vectors may be generated by adding random noise, e.g., 1%, 3%, or 5%, to a first feature vector to generate a second feature vector. The amount of random noise added may also be randomly selected between a predetermined range, e.g., 0-5%, 1-8%, or 2-5%. The first feature vector and second feature vector may then be paired and labeled as for the same speaker.

The evaluation vector similarity model training data810may be provided to an evaluation vector similarity model trainer820. The evaluation vector similarity model trainer820may use the evaluation vector similarity model training data810to train a neural network. In some implementations, training a neural network using the evaluation vector similarity model training data810may include a maxout feature or a dropout feature. For example, a feed forward neural network may be trained using rectified linear activation functions using dropout features.

The evaluation vector similarity model trainer820may train the neural network so that when a first feature vector and a second feature vector of a labeled pair of feature vectors is received as an input to the neural network, the neural network outputs a result that matches the label for the labeled pair of feature vectors. For example, the evaluation vector similarity model trainer820may train a neural network so that when FV1and FV2are received as inputs, the neural network outputs “SAME SPEAKER,” and when FV2and FV3are received as inputs, the neural network outputs “DIFFERENT SPEAKER.” The trained neural network may be the evaluation vector similarity model830.

During speaker verification, speech data840of an utterance from a speaker to be verified as a particular speaker may be obtained and received by the speaker verification model850. In response to receiving the speech data840, the speaker verification model850may output an evaluation vector860. For example, the speaker verification model410trained as described inFIG. 3may receive speech data for an utterance602and output the evaluation vector604as described inFIG. 6.

The evaluation vector similarity model830may receive the evaluation vector860output by the speaker verification model850and obtain a reference vector870corresponding to the particular speaker. For example, the evaluation vector similarity model830may receive the reference vector404designated to represent characteristics of an authorized user's voice. As described above, the reference vector870may be an output of the speaker verification model850in response to input of speech data for an enrollment utterance from the particular speaker, or an average of multiple outputs of the speaker verification model850in response to inputs of speech data for utterances from the particular speaker.

The evaluation vector similarity model830may process the evaluation vector860and the reference vector870and provide an output that indicates whether the utterance was likely spoken by the particular speaker. For example, the evaluation vector similarity model830may output, “SAME SPEAKER” or “DIFFERENT SPEAKER.”

In some implementations, the evaluation vector similarity model830may output a similarity score that may be compared to a similarity threshold to classify whether the utterance was likely spoken by the particular speaker. For example, in the case where labels of the evaluation vector similarity model training data810that indicate that pairs of feature vectors are for the same speaker are associated with a similarity score of “1.0” and labels of the evaluation vector similarity model training data810that indicate that pairs of feature vectors are for different speakers are associated with a similarity score of “0.0,” the evaluation vector similarity model830may output a similarity score that ranges from “0.0” to “1.0,” and compare the similarity score to a similarity threshold of “0.7,” “0.8,” or 0.9,” to classify whether the utterance was likely spoken by the particular speaker.

In one particular example, the evaluation vector similarity model830may output a similarity score of “0.9,” which may then be determined to be greater than a similarity threshold of “0.8,” and based on the determination that the similarity score is greater than the similarity threshold, the utterance may be classified as likely spoken by the particular speaker. In another particular example, the evaluation vector similarity model830may output a similarity score of “0.6,” which may then be determined to be less than a similarity threshold of “0.7,” and based on the determination that the similarity score is less than the similarity threshold, the utterance may be classified as likely not spoken by the particular speaker.

FIG. 9Ais a flow chart illustrating an example process900for training an evaluation vector similarity model. The process900may be performed by data processing apparatus, such as the computing system120described above or another data processing apparatus.

The computing system120may receive labeled pairs of feature vectors (902). For example, the computing system120may obtain the evaluation vector similarity model training data810. As described above, the evaluation vector similarity model training data810may include labeled pairs of feature vectors, where each labeled pairs of feature vectors may include a first feature vector, a second feature vector, and a label indicating whether the first feature vector and second feature vector are for the same speaker or different speakers.

The computing system120may train a neural network using the labeled pairs of feature vectors (904). For example, the computing system120may provide the evaluation vector similarity model training data810to the evaluation vector similarity model trainer820. More details of training the neural network may be found above in relation to the evaluation vector similarity model trainer820.

The computing system120may obtain an evaluation vector similarity model (906). For example, the evaluation vector similarity model830may be a neural network trained by the evaluation vector similarity model trainer820using the evaluation vector similarity model training data. More details of obtaining the evaluation vector similarity model may be found above in relation to the evaluation vector similarity model trainer820.

FIG. 9Bis a flow chart illustrating an example process950for speaker verification using an evaluation vector similarity model. The process950may be performed by data processing apparatus, such as the client device110described above or another data processing apparatus.

The client device110may input speech data to a speaker verification model (952). For example, the client device110may input, to the speaker verification model850, speech data840for an utterance from a speaker to be verified as a particular speaker.

The client device110may determine an evaluation vector (954). For example, in response to the speech data840, the speaker verification model850on the client device110may output the evaluation vector860.

The client device110may input the evaluation vector and a reference vector to an evaluation vector similarity model (956). For example, the evaluation vector860and the reference vector870may be provided to the evaluation vector similarity model830on the client device110for the evaluation vector similarity model830to output an indication whether the speaker is likely the particular speaker. The evaluation vector similarity model830may output, “SAME SPEAKER” or “DIFFERENT SPEAKER,” or a similarity score that may be compared to a similarity threshold to classify whether the utterance was likely spoken by the particular speaker. For example, the evaluation vector similarity model830may output a similarity score of “0.95.”

The client device110may verify the speaker identity (958). For example, if the evaluation vector similarity model830outputs an indication that the speaker is likely the particular speaker, the client device110may verify that the speaker is the particular speaker. In another example, if the evaluation vector similarity model830outputs an indication that the speaker is likely not the particular speaker, the client device110may verify that the speaker is not the particular speaker. In yet another example, the client device110may determine that a similarity score of “0.95” output by the evaluation vector similarity model830is greater than a similarity threshold of “0.9,” and based on the determination that the similarity score is greater than the similarity threshold, the client device110may verify that the speaker is the particular speaker.

In some implementations, the client device110may process voice commands based on speaker verification using the evaluation vector similarity model830. For example, utterances may include a term that is a hotword followed by terms for a voice command. A hotword may indicate to the client device110that terms that follow the hotword are terms for a voice command for the client device110. According, the client device110may not process audio input for voice commands until the client device110detects a hotword. However, background utterances from other speakers may include a hotword. To avoid having a particular user's client device110detect a hotword in a background utterance and then process audio input for voice commands, the client device110may verify using the evaluation vector similarity model830that any detected hotword was spoken by the particular user and only process audio input for voice commands when the detected hotword is verified as being spoken by the particular user.

In some implementations, the evaluation vector similarity model training data may also include a corresponding pair of output vectors. The evaluation vector similarity model may receive an evaluation vector, a reference vector, an output vector corresponding to input speech features, and a reference output vector. The evaluation vector similarity model may then output a similarity score based on the four vectors.

In some implementations, the system100may use mel-frequency cepstrum coefficients (MFCC) for speaker verification. The system100may use a speech recognizer to force align audio input based on phonemes. For example, the speech recognizer may force align an audio signal hotword “OK COMPUTER” with the following sequence of phonemes, “O,” “K,” “COM,” PU,” “TER.” The system100may split the audio signal into segments based on the boundaries of the alignments and, for each segment, compute a MFCC frame using the audio signal across the whole width of the segment. The system100may compute MFCC coefficients of the frequency spectrum in a range of [100 Hz, 4000 Hz], then discard the first coefficient and keep the first N coefficients, skipping the first coefficient. N may be a number including 6, 8, 12, or some other number. For example, the result of a computation for the audio signal hotword “OK COMPUTER” using an N of 12 is a 12×5 matrix, holding 5 MFCC frames, each frame holding 12 coefficients. The system100may then normalize each frame. For example, the system100may normalize each MFCC frame by first subtracting its arithmetic mean, and then dividing the obtained frame by its Euclidean distance. The system100may then reshape the matrix into a single feature vector of length60.

Alternatively to splitting the audio signal hotword based on phonemes, the system100may split the audio signal hotword into a predetermined number, e.g., 4, 6, 10, or some other number, of equally long, possibly overlapping, parts. Additionally, instead of using MFCC features, the system100may generate feature vectors based on mel-frequency filterbank coefficients.

The system100may generate a single composite feature vector from pairs of feature vectors from the same or different speakers. The system100may generate the single composite feature vector by comparing a pair of feature vectors using a partial distance. The composite feature vector may be computed based on determining distances of the pair of feature vectors' pairwise scalar components. Different distance functions may be used for comparing corresponding scalar components. For example, a scalar distance may be used or a squared scalar distance used. Alternatively, instead of using a distance function, a similarity function may be used to generate the single composite feature vector. For example, a cosine similarity function may be used. The system100may train a classifier using training data including single composite feature vectors that are labeled as either from a pair of feature vectors from the same speaker or from a pair of feature vectors from different speakers.

To verify a user's identity as an enrolled user, the system100may obtain a feature vector for the user, a feature vector for the enrolled user, generate a single composite feature vector using the distance or similarity function, and then use the trained classifier to classify the single composite feature vector as generated from a pair of feature vectors from the same speaker, thus verifying the user's identity as the enrolled user, or from different speakers.

Thus, particular embodiments have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results.