Modeling device and method for speaker recognition, and speaker recognition system

A modeling device comprises a front end which receives enrollment speech data from each target speaker, a reference anchor set generation unit which generates a reference anchor set using the enrollment speech data based on an anchor space, and a voice print generation unit which generates voice prints based on the reference anchor set and the enrollment speech data. By taking the enrollment speech and speaker adaptation technique into account, anchor models with a smaller size can be generated, so reliable and robust speaker recognition with a smaller size reference anchor set is possible.

TECHNICAL FIELD

The present disclosure is related to speech (audio) processing and recognition technology, and also related to the technology of speaker verification, telephone conference, and digital network audio & video.

BACKGROUND

Speaker recognition technique is very useful for many applications, e.g. speaker tracking, audio index and segmentation. Recently, it has been proposed to model a speaker using several anchor (speaker) models. The speaker voice is projected on the anchor models to constitute vector representing acoustic characteristics of the speaker.

FIG. 1shows a block diagram of a conventional device for speaker recognition. As shown in theFIG. 1, an anchor space is created by training speeches from a lot of general speakers. In a reference anchor set generation unit102, a number of virtual anchor speakers which are the centriods of clusters are selected from the anchor space to form a reference anchor set, or the nearest anchor speaker to the centroid of each cluster is selected to form the reference anchor set. A front end101receives an enrollment speech by a target speaker and converts the enrollment speech into feature parameters, and sends the feature parameters to a voice print generation unit103. The voice print generation unit103generates a voice print based on the feature parameters sent from the front end101and the reference anchor set generated by the reference anchor set generation unit102. Then, the generated voice print is stored into a voice print database104for further use for speaker recognition. As can be seen from theFIG. 1the reference anchor set generated by the device100can reflect the distribution of the anchor space itself only. Accordingly, a larger number of anchors are needed to describe the target speaker better, which makes the computation load higher and more difficult to be used in the embedded system.

SUMMARY

In one aspect of the present disclosure, there is provided a modeling device for speaker recognition, comprising: a front end which receives enrollment speeches from target speakers; a reference anchor set generation unit which generates a reference anchor set using the enrollment speeches based on an anchor space; and a voice print generation unit which generates voice prints based on the reference anchor set and the enrollment speeches.

In another aspect of the present disclosure, there is provided a modeling method for speaker recognition, comprising steps of: receiving enrollment speeches from target speakers; generating a reference anchor set using the enrollment speeches based on an anchor space; and generating voice prints based on the reference anchor set and enrollment speeches.

In a further aspect of the present disclosure, there is provided a speaker recognition system comprises: a front end which receives enrollment speeches and/or testing speeches from target speakers; a reference anchor set generation unit which generates a reference anchor set using the enrollment speeches based on an anchor space; a voice print generation unit which generates voice prints based on the reference anchor set and the enrollment speeches and/or testing speeches; a matching unit which compares the voice prints generated from the testing speeches with the voice prints generated from the enrollment speeches; and a decision unit which recognizes the identity of the target speakers based on the result of the comparison.

In another further aspect of the present disclosure, there provided a speaker recognition system comprises a modeling device and a recognition device, wherein, the modeling device comprising: a first front end which receives enrollment speeches from target speakers; a reference anchor set generation unit which generates a reference anchor set using the enrollment speeches based on an anchor space; and a first voice print generation unit which generates first voice prints based on the reference anchor set and the enrollment speeches, and the recognition device comprising: a second front end which receives testing speeches from the target speakers; a second voice print generation unit which generates second voice prints based on the reference anchor set and the testing speeches; a matching unit which compares the first voice prints with the second voice prints; and a decision unit which recognizes the identity of the target speakers based on the result of comparison.

With the modeling device, method, and speaker recognition system of the present disclosure, by taking the enrollment speech and speaker adaptation technique into account, anchor models with smaller size can be generated, and reliable and robust speaker recognition with smaller size reference anchor set is possible. It brings great advantages for computation speed improvement and great memory reduction, such low computation and small reference anchor set is more suitable for embedded application.

The foregoing is a summary and thus contains, by necessity, simplifications, generalizations, and omissions of details; consequently, those skilled in the art will appreciate that the summary is illustrative only and is not intended to be in any way limiting. Other aspects, features, and advantages of the devices and/or processes and/or other subject matters described herein will become apparent in the teachings set forth herein. The summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

DETAILED DESCRIPTION

The main terms used in the present disclosure will be introduced as follows:

1) Anchor Database

Generally, there are voice data from hundreds or thousands of speakers in the training corpus. Anchor model (e.g. Gaussian mixture model) representing the acoustic characteristic of each speaker can be trained. The anchor database is constructed if all anchor models are put together.

2) Reference Anchor Set

Reference anchor set are defined as those generated from the anchor database according to specific generation rules, which is used for speaker verification/identification system.

3) Anchor Space

If each anchor model represents one dimension of the space, the anchor space can be constructed when all anchor models in the anchor database are used. The dimension of the anchor space is equal to the total number of anchor models in the anchor database,

4) Principal Anchor Model

The principal anchor model is defined as the nearest model with respect to one speakers enrollment speech.

5) Associate Anchor Model

Except the principal anchor, the remainder models in the reference anchor set are defined as associate anchor models.

FIG. 2shows a block diagram of a modeling device for speaker recognition according to one embodiment of the present disclosure.

As shown in theFIG. 2, the modeling device200for speaker recognition according to one embodiment of the present disclosure includes: a front end201, a reference anchor set generation unit202, a voice print generation unit203and a voice print database204. In the above structure, for example, the voice print generation unit203is connected to the front end201, reference anchor set generation unit202and voice print database204. The front end201is also connected to the reference anchor set generation unit202.

According to one embodiment of the present disclosure, in the modeling device200, the front end201receives enrollment speeches from target speakers, the reference anchor set generation unit202generates a reference anchor set using the enrollment speeches based on an anchor space, and the voice print generation unit203generates voice prints based on the reference anchor set and the enrollment speeches.

Hereinafter, the description will be given to the operations of the above respective units of the modeling device200according to the embodiment of the present disclosure.

An anchor space is firstly created by training lots of speeches from general speakers, and the anchor space includes a plurality of anchor models which represent the acoustic characteristics of those general speakers. The anchor space can be stored in a database as a form of the anchor database. The front end201receives enrollment speeches by target speakers, and converts the enrollment speeches into feature parameters, and sends the feature parameters to the reference anchor set generation unit202and the voice print generation unit203. The reference anchor set generation unit202generates a reference anchor set from the anchor space based on the enrollment speeches by target speakers. The voice print generation unit203generates voice prints by applying the feature parameters extracted from the enrollment speeches and sent from the front end201onto the anchor models of the reference anchor set generated by the reference anchor set generation unit202, and stores the generated voice prints into the voice print database204for further use for speaker recognition.

Since the operations of the front end201and the voice print generation unit203are well known by those skilled in the art, the details thereof will be omitted in order not to confuse the main points of the present disclosure. In the following, the operation of the reference anchor set generation unit202will be described in detail.

According to another embodiment of the present disclosure, the reference anchor set generated by the reference anchor set generation unit202is composed of at least one principal anchors and at least one associate anchors, and the reference anchor set generation unit202generates the principal anchors by using the enrollment speeches based on the anchor space, and generates the at least one associate anchors based on the principal anchors.

It is assumed that if there are enrollment speeches of “n” target speakers, then the number of the principal anchors which are generated could be 1 to n, and it is preferred that n principal anchors are generated based on the enrollment speeches from the n target speakers.

FIG. 3shows a block diagram of the reference anchor set generation unit according to one embodiment of the present disclosure. The functions of the reference anchor set generation unit302is the same as those of the reference anchor set generation unit202as shown in theFIG. 2. As shown in theFIG. 3, the reference anchor set generation unit302includes a principal anchor generation unit3021, an associate anchor generation unit3022and a combination unit3023.

Specifically, the principal anchor generation unit3021generates a principal anchor by finding an anchor model with the nearest distance to the model of enrollment speech of the target speaker from the anchor space. Here, the model of enrollment speech can also be a GMM (Gaussian mixture model) which represents the acoustic characteristic of the target speaker, and can be calculated by the front end201or the reference anchor set generation unit302with the feature parameters of the enrollment speech of the target speaker. At least one principal anchors are generated dependent on the number of the target speakers. The associate anchor generation unit3022generates the associate anchors by splitting the anchor space into a plurality of dusters based on the principal anchors, and finding anchor models with the nearest distance to the centroids of the plurality of dusters containing no principal anchors. Here, the distance can be a likelihood value. The combination unit3023combines the generated principal anchors and associate anchors together as the reference anchor set.

Hereinafter, the description will be given to the operations of the above respective units of the reference anchor set generation unit302according to one embodiment of the present disclosure.

Firstly, in the principal anchor generation unit3021the principal anchors are generated by using the enrollment speeches by target speakers. Specifically, the principal anchor for one target speaker can be obtained by finding the anchor model with the nearest distance to the GMM model of enrollment speech of the target speaker in the anchor space. More specifically and according to one embodiment of the present disclosure, likelihood distance may be used as the above distance.

For example, the likelihood distance between the target speaker and each general speaker existing in the anchor space is computed with the enrollment speech from the target speaker. It is firstly assumed that each speakeriin the anchor space is modeled by a probability model such as a GMM (Gaussian mixture model) Λiwith parameters {pij, μij, Σij}, j=1, 2, . . . , Mi, where Midenotes the number of mixture components, pijdenotes the mixture gain, and μijand Σijare the mean vector and the diagonal covariance matrix, respectively, of each Gaussian mixture component. The enrollment speech X of the target speaker has voice frames [x1, x2, x3, . . . , xT] where T is the total frame number. Therefore, the likelihood distance is obtained as the equation (1) below.

Subsequently, the nearest speaker model of general speaker is found as the principal anchor for example by computing arg maxiP(X|Λi). It should be noted that if more than one target speakers exist, the above operations can be repeated to get multiple principle anchors. Thus the number of generated principle anchors is not limiting of scope of the present disclosure. Also, the distance is not limited to the likelihood distance.

Then, in the associate anchor generation unit3022, the associate anchors are generated based on the principal anchors generated by the principal anchor generation unit3021.

According to one embodiment of the present disclosure, the process of the anchor space being split into the plurality of clusters is as follows: firstly, the anchor space is split into N clusters, here, N equals to the number of the principal anchors. One cluster with the biggest intra-class distance among the N clusters are found to be further split, here, if the cluster has a principal anchor, the cluster is split into two sub-clusters based on the principal anchor and an anchor with the farthest distance to the principal anchor in the cluster, while if the cluster has no principal anchor, the cluster is split into two sub-clusters based on two farthest anchors in the cluster with respect to the farther principal anchor. The above process is repeated until the plurality of clusters is obtained.

Specifically, the anchor space is initially split into N clusters (N equals the number of the principal anchors) according to the distance between each anchor in the anchor space and the generated principal anchors. Each of the obtained N clusters contains one principal anchor generated by the principal anchors generation unit3021. Here, the distance can be a likelihood value. For example, Kullback-Leibler (KL) divergence distance or Euclidean distance can be used as the above distance. Taking Euclidean distance as an example, the Euclidean distance between two GMMs Λ1and Λ2is given as the equation (2) below,

Herein, k, m=1, 2, only representing the indexes of the two GMMs.

Next, the operation of finding one cluster with the biggest intra-class distance to be further split is repeated until the total number of dusters is satisfied. The number of clusters can be predetermined by requirements of the practical applications. The step of splitting is based on the following rule:if the cluster selected to be split has a principal anchor, an anchor with the farthest distance to the principal anchor in this cluster is found, each of rest anchors in this duster is compared with the two anchors (the anchor with the farthest distance and the principal anchor) for classification, and this cluster is split into two sub-clusters based on the comparison; otherwiseif the cluster selected to be split has no principal anchor, two farthest anchors in the cluster with respect to the father principal anchor, that is, the principal anchor included in the father cluster from which the cluster is split are found, each of rest anchors in this cluster is compared with these two anchors for classification, and this cluster is split into two sub-clusters based on the comparison.

For instance, the comparisons are performed according to the distance between two anchors as described above. The anchor space is split into the desired number of clusters after the above operation is repeated as described above.

Subsequently, anchors nearest to the centroids of the clusters other than the clusters containing principle anchors are found as associate anchors. According to one embodiment of the present disclosure, the centroids may be obtained according to scaled Bhattacharyya distance. For example, the distance measure for combining the GMMs in each cluster to obtain each centroid is based on the scaled Bhattacharyya distance which is given as the equation (4) below,
BSD=BscaleBdistance=Bscale└−log ∫√{square root over (ΛiΛj)}dx┘(4)

The selection of anchors nearest to the centriods can, for example, use the above KL/Euclidean distance.

Finally, in the combination unit3023, the principal anchors generated by the principal anchors generation unit3021and the associate anchors generated by the associate anchors generation unit3022are combined as a reference anchor set for further use for speaker recognition.

FIG. 4shows a schematic diagram of an anchor space400according to one embodiment of the present disclosure. In theFIG. 4, by taking a case of two target speakers as an example, a result after processing by the reference anchor set generation unit302of theFIG. 3is shown. As shown in theFIG. 4, there are totally six split clusters401,402,403,404,405and406in the anchor space400. Two principal anchors4011and4041respectively located in the clusters401and404, are generated by the principal anchor generation unit3021based on the enrollment speech data4012and4042of speaker1and speaker2as described above. By the above operation of splitting based on the two principal anchors4011and4041, the associate anchors generation unit3022generates four associate anchors4021,4031,4051and4061respectively located in the clusters402,403,405and406which containing no principal anchor. The combination of the two principal anchors4011and4041as well as four associate anchors4021,4031,4051and4061is the reference anchor set generated by the reference anchor set generation unit302. Note that the total number of the split clusters is not limited to six, and those skilled in the art should understand that the total number of clusters can be appropriately set to be any other numerals.

According to one embodiment of the present disclosure, a reference anchor set generation unit may further include an adaptation unit.FIG. 5shows a block diagram of a reference anchor set generation unit502in the modeling device200according to one embodiment of the present disclosure.

As can be seen from theFIG. 5, the reference anchor set generation unit502is different from the reference anchor set generation unit302of theFIG. 3in that an adaptation unit5024is added between the principal anchor generation unit3021and the associate anchor generation unit3022.

Specifically, the principal anchors generated by the principal anchor generation unit3021are input into the adaptation unit5024. In the adaptation5024, speaker adaptation techniques such as MLLR (Maximum Likelihood Linear Regression)/MAP (Maximum A Posteriori) etc. can be used on the generated principal anchors, and the refined principal anchors after the adaptation is output to the associate anchor generation unit3022. In the associate anchor generation unit3022, the refined principal anchors guide to split the anchor space and to find the associate anchors with the operation described before. Since the adaptation technology is a common technology, the details thereof will not be described hereinafter.

That is to say, the principal anchors can be adapted with the enrollment speech by adaptation method such as MAP or MLLR in the adaptation unit5024, and the associate anchors can be generated based on the principal anchors after adaptation in the associate anchor generation unit3022.

It should be noted that the adaptation is not limited to be used on the generated principal anchors, instead can be used on the generated reference anchor set.

FIG. 6shows a block diagram of the reference anchor set generation unit in the modeling device200according to another embodiment of the present disclosure. An adaptation unit can be directly located between the principal anchor generation unit and the combination unit of the reference anchor set generation unit.

As can be seen from theFIG. 6, the reference anchor set generation unit602is different from the reference anchor set generation unit502of theFIG. 5in that the adaptation unit5024and the associate anchor generation unit3022are located in parallel between the principal anchor generation unit3021and the combination unit3023.

Specifically, the operations of the respective units shown inFIG. 6are as follows. After the principal anchors are generated by the principal anchor generation unit3021, the obtained principal anchors are input to the adaptation unit5024and the associate anchor generation unit3022, respectively. In the adaptation unit5024, some speaker adaptation techniques such as MLLR/MAP etc. can be used on the principal anchors for adaptation processing. In the associate anchor generation unit3022, the generated principal anchors guide to split the anchor space and to find the associate anchors with the operations as described before with reference toFIG. 3. Then, the adapted principal anchors output from the adaptation unit5024and the associate anchors output from the associate anchor generation unit3022are input to the combination unit3023for combination processing, and the refined reference anchor set is obtained from combining the adapted principal anchors and the associate anchors, and is input to the voice print generation unit203for generating the voice prints of the target speakers, and for further use for speaker recognition.

FIG. 7shows a flow chart of a method for speaker recognition according to one embodiment of the present disclosure

As shown in theFIG. 7, the method for speaker recognition according to the embodiment of the present disclosure may include the following steps. In the step S701, the enrollment speeches are received from target speakers. In the step S702, a reference anchor set is generated by using the enrollment speeches based on an anchor space. In the step S703, voice prints of the target speakers are generated based on the reference anchor set and enrollment speeches. According to the present embodiment, the step S701can be performed by the front end201, the step S702can be performed by any one of the reference anchor set generation units202,302,502and602as describe above, and the step S703can be performed by the voice print generation unit203.

FIG. 8shows a flow chart of a method for generating the reference anchor set according to one embodiment of the present disclosure. As described above, the reference anchor set is composed of principal anchors and associate anchors, and the associate anchors are generated based on the principal anchors.

Specifically, the step S702of generating the reference anchor set as shown in theFIG. 7may further includes sub-steps as shown in theFIG. 8. As shown in theFIG. 8, in the step S801, the principal anchors are generated by using the enrollment speeches based on an anchor space. In the step S802, the associate anchors are generated based on the generated principle anchors. In the step S803, the principal anchors and the associate anchors are combined as the reference anchor set. According to the embodiment, the steps S801to S803may be performed by the principal anchor generation unit3021, the associate anchor generation unit3022and the combination unit3023respectively as described above.

According to another embodiment of the present disclosure, the step S801further comprises a step of generating the principal anchors by finding anchor models with the nearest distance to the model of enrollment speeches from the anchor space.

According to another embodiment of the present disclosure, the step S802further comprises steps of generating the associate anchors by splitting the anchor space into a plurality of clusters based on the principal anchors and finding anchor models with the nearest distance to the centroids of the plurality of clusters containing no principal anchors.

According to another embodiment of the present disclosure, the step S802further comprises steps of: firstly splitting the anchor space into N clusters, wherein N equals to the number of the principal anchors, finding one cluster with the biggest intra-class distance among the N clusters to be further split, wherein if the cluster has a principal anchor, the cluster is split into two sub-clusters based on the principal anchor and an anchor with the farthest distance to the principal anchor in the cluster, while if the cluster has no principal anchor, the cluster is split into two sub-clusters based on two farthest anchors in the cluster with respect to the father principal anchor, and repeating the above process until the plurality of clusters are obtained.

According to one embodiment of the present disclosure, the method of generating the reference anchor set as shown in theFIG. 8may further include a step of adding adaptation process after the step S801. On one hand, the principle anchors generated in the step S801may be adapted with the enrollment speeches by using adaptation technique such as MAP and MLLR before being used to generate the associate anchors in the step S802, and the associate anchors are generated based on the principal anchors after adaptation in the step S802. On the other hand, the adaptation technique such as MAP and MLLR may be used on the principal anchors generated in the step S801, and the associate anchors are generated based on the principal anchors without adaptation in the step S802, so the combination unit3023combines the associate anchors and the adapted principal anchors together to obtain the reference anchor set. According to the embodiment of the present disclosure, the step of adaptation can be performed by the adaptation unit5024as describe above.

The execution of the above steps of the method is not limited to the above sequence, and the steps can be executed in any sequence and/or in a parallel manner. It is also possible that not all of the steps as shown are necessary to be executed.

FIG. 9shows a block diagram of a speaker recognition system according to one embodiment of the present disclosure.

As shown in theFIG. 9, the whole speaker recognition system900is composed of two phases, one of which is an enrollment phase, and the other is a testing phase. The structure of the enrollment phase is the same as described with reference to theFIG. 2, so the details thereof are omitted hereinafter. The testing phase includes a front end901, a voice print generation unit902, a matching unit903and a decision unit904. The front end901receives testing speeches by target speakers, extracts feature parameters from the testing speeches, and sends the feature parameters to the voice print generation unit902. The front end901and the front end201can be worked as one unit, which is not limited to the above construction. The voice print generation unit902generates voice prints based on the feature parameters sent from the front end901and the reference anchor set generated by the reference anchor set generation unit202as describe above. Similar with the front ends901and201, the voice print generation unit902and the voice print generation unit203can be worked as one unit, which is not limited to the above construction. The matching unit903compares the voice prints generated from the test phase with the voice prints generated from the enrollment phase, and sends the result of comparison to the decision unit904. The decision unit904recognizes the identity of the target speaker based on the result, that is, if the result of comparison is more than a predetermined threshold, the identity of the target speaker will be recognized, and if the result of comparison is less than the predetermined threshold, the identity of the target speaker will be denied.

Thereby, the speaker recognition system900according to the embodiment of the present disclosure can be constructed as comprising: a front end201or901which receives enrollment speeches and/or testing speeches from target speakers, a reference anchor set generation unit202which generates a reference anchor set by using the enrollment speeches based on an anchor space, a voice print generation unit203or902which generates voice prints based on the reference anchor set and the enrollment speeches and/or testing speeches, a matching unit903which compares the voice prints generated from the testing speeches with the voice prints generated from the enrollment speeches, and a decision unit904which recognizes the identity of the target speaker based on the result of the comparison.

According to another embodiment of the present disclosure, the speaker recognition system900can be constructed as comprising: a modeling device and a recognition device, wherein, the modeling device comprises a first front end201which receives enrollment speeches from target speakers, a reference anchor set generation unit202which generates a reference anchor set by using the enrollment speeches based on an anchor space, and a first voice print generation unit203which generates first voice prints based on the reference anchor set and the enrollment speeches, the recognition device comprises a second front end901which receives testing speeches from the target speakers, a second voice print generation unit902which generates second voice prints based on the reference anchor set and the testing speeches, a matching unit903which compares the first voice prints with the second voice prints, and a decision unit904which recognizes the identity of the target speaker based on the result of comparison.

In this way, by taking the enrollment speeches and speaker adaptation technique into account, anchor models with smaller size can be generated, so reliable and robust speaker recognition with smaller size reference anchor set is possible. It brings great advantages for computation speed improvement and great memory reduction, and low computation and small reference anchor set is more suitable for embedded applications.

Several experiments are conducted to validate the effectiveness of the present disclosure. In the first experiment, the speaker identification and verification experiments on the specific size of reference anchor set are used. As shown in Table 1, the present disclosure can outperform the traditional method.

In the next experiment, the influence of the reference anchor set size on speaker identification system is checked. It is observed fromFIG. 10that the present disclosure outperforms the traditional method greatly in the small size of the reference anchor set and speaker adaptation can boost the performance. In addition, it is shown there are two factors influencing the system performance. One factor is the discriminative capability of anchor models on target speakers. The initial performance of the traditional method is not good because its discriminative capability is low. With embodiment, the performance is improved because the discriminative capability is considered during anchor generation process. Furthermore, speaker adaptation on anchors could boost this capability and the performance is further improved. Besides principal anchors, associate anchors can provide additional discriminative capability, especially for rejection purpose. That is why the performance improves when anchor size increases without adaptation, shown from the curves of tradition method and embodiment inFIG. 10. The positive effect of adaptation to improve the discriminative capability reaches its maximum in small anchor case. Another factor is how stable the speaker print vector generated is, which is affected by the enrollment data size. From the curve of embodiment with adaptation inFIG. 10, the performance is degraded because it needs much more enrollment data to generate stable higher dimensional speaker print vector as anchor size increases.

When the anchor size is small, the adaptation is dominant since little enrollment data is needed for speaker print generation. When that anchor size is getting larger, much more enrollment data is necessary to generate a stable speaker print vector and the effect of adaptation decreases. In conclusion, more dimensions of voice print have more rejection capability and fewer dimensions of voice print require less enrollment data. The effect of fewer dimensions exceeds the effect of more dimensions because of additional effect of this boosting in fewer dimensions condition.