Speech feature extraction apparatus, speech feature extraction method, and computer-readable storage medium

A speech feature extraction apparatus 100 includes a voice activity detection unit 103 that drops non-voice frames from frames corresponding to an input speech utterance, and calculates a posterior of being voiced for each frame, a voice activity detection process unit 106 calculates a function value as weights in pooling frames to produce an utterance-level feature, from a given a voice activity detection posterior, and an utterance-level feature extraction unit 112 that extracts an utterance-level feature, from the frame on a basis of multiple frame-level features, using the function values.

This application is a National Stage Entry of PCT/JP2018/024933 filed on Jun. 29, 2018, the contents of all of which are incorporated herein by reference, in their entirety.

TECHNICAL FIELD

The present invention relates to a speech feature extraction apparatus, speech feature extraction method, and a computer-readable storage medium storing a program for realizing these.

BACKGROUND ART

In speaker recognition, the system input is a sequence of raw features (acoustic features) of variable number of frames. They are frame-level, while the system output expected to be speaker ID in speaker identification or target/non-target (same speaker/different speakers) in speaker verification.

Both of the output speaker ID and target/non-target are in the utterance level. To produce such utterance-level output from the frame-level input, a pooling (sum-up) process over all valid frames is necessary in somewhere in the speaker recognition system. Equally weighted pooling is commonly used for such a purpose.

For example, Non-Patent Documents (NPL) 1 and 2 disclose a speaker recognition system.FIG.18is a block diagram of the speaker recognition system disclosed in NPL 2.

CITATION LIST

Non Patent Literature

SUMMARY OF INVENTION

Technical Problem

In speaker recognition, no matter the standard ivector-based methods disclosed in the NPL1, or recently popular DNN-based speaker embedding methods disclosed in the NPL2, equally weighted pooling is used for such purpose of obtaining an output of utterance-level speaker recognition results from frame-level acoustic feature information.

In ivector-based methods of the NPL1, given an utterance with a L frame feature sequence {y1, y2, . . . , yL}, an utterance-level feature x is extracted in accord with Math. 1 and 2. In the Math. 1, M means supervector M. Supervector M is generated by concatenating all the Mc. In the Math. 2, c is the index of Gaussian components in GMM-UBM. All frames are treated equally, just in the manner of summation over all frames.
M=μ+Tx,[Math. 1]

In DNN-based methods as shown in [NPL2], an average pooling layer gives the same importance to every frame while in reality some frames do have more speaker information than others. This results in that the embeddings are not the accurate representation of speakers, so that speaker recognition performance is degraded no matter what model is used in backend.

An object of the present invention is to resolve the foregoing problem and provide a speech feature extraction apparatus, speech feature extraction method, and a computer-readable recording medium that can provide a more accurate representation of an utterance for speaker recognition task.

Solution to Problem

In order to achieve the foregoing object, a speech feature extraction apparatus according to one aspect of the present invention includes:

a voice activity detection unit that drops non-voice frames from frames corresponding to an input speech utterance, and calculates a posterior of being voiced for each frame;

a voice activity detection process unit that calculates a function value as weights in pooling frames to produce an utterance-level feature, from a given a voice activity detection posterior;

an utterance-level feature extraction unit that extracts an utterance-level feature, from the frame on a basis of multiple frame-level features, using the function values.

In order to achieve the foregoing object, a speech feature extraction method according to another aspect of the present invention includes:

(a) a step of dropping non-voice frames from frames corresponding to an input speech utterance, and calculates a posterior of being voiced for each frame;

(b) a step of calculating a function value as weights in pooling frames to produce an utterance-level feature, from a given a voice activity detection posterior;

(c) a step of extracting an utterance-level feature, from the frame on a basis of multiple frame-level features, using the function values.

In order to achieve the foregoing object, a computer-readable recording medium according to still another aspect of the present invention has recorded therein a program, and the program includes an instruction to cause the computer to execute:

(a) a step of dropping non-voice frames from frames corresponding to an input speech utterance, and calculates a posterior of being voiced for each frame;

(b) a step of calculating a function value as weights in pooling frames to produce an utterance-level feature, from a given a voice activity detection posterior;

(c) a step of extracting an utterance-level feature, from the frame on a basis of multiple frame-level features, using the function values.

Advantageous Effects of Invention

As described above, according to the present invention, it is possible to provide a more accurate representation of an utterance for speaker recognition task.

DESCRIPTION OF EMBODIMENTS

Principle of the Invention

This invention is to give weights using functions of Voice Activity Detection (VAD) posteriors for frames in pooling features from frame-level to utterance-level. It is a common sense that frames with higher VAD posteriors have more speaker information than those with low VAD posteriors which are likely to be silence or noise. So giving more weightage to those frames which have higher VAD posteriors will get a final representation for the utterance more appropriate for speaker recognition task.

EMBODIMENT

Each exemplary embodiment of the present invention will be described below with reference to the figures. The following detailed descriptions are merely exemplary in nature and are not intended to limit the invention or the application and uses of the invention. Furthermore, there is no intention to be bound by any theory presented in the preceding background of the invention or the following detailed description.

First Embodiment

A speech feature extraction apparatus of a first embodiment can utilize the posteriors of the VAD which is already applied in frame selections in most speech processing systems including speaker recognition, to give more weight to more voice-like frames. It can extract more appropriate utterance-level feature with existing VAD.

Hereinafter, a speech feature extraction apparatus, a speech feature method, and a program of the first embodiment of the present invention will be described with reference toFIGS.1to7.

Device Configuration

First, a schematic configuration of a speech feature extraction apparatus100according to the first embodiment will be described usingFIG.1.FIG.1is a block diagram schematically showing the configuration of the speech feature extraction apparatus according to the first embodiment of the present invention.

As shown inFIG.1, the speech feature extraction apparatus100includes a voice activity detection (VAD) unit103, a voice activity detection (VAD) process unit106, and a utterance-level feature extraction unit112.

The VAD unit103drops non-voice frames from frames corresponding to an input speech utterance, and calculates a posterior of being voiced for each frame. The VAD process unit106calculates a function value as weights in pooling frames to produce an utterance-level feature, from a given a voice activity detection posterior. The utterance-level feature extraction unit112extracts an utterance-level feature, from the frame on a basis of multiple frame-level features, using the function values.

According to the speech feature extraction apparatus100, it is possible to provide a more accurate representation of an utterance for speaker recognition task.

Next, the configuration of the speech feature extraction apparatus of the first embodiment will be described in detail with reference toFIGS.2to4as well.FIG.2is a block diagram showing the specific configuration of the speech feature extraction apparatus according to the first embodiment of the present invention.

In the first embodiment of the present invention, the speech feature extraction apparatus100using existing VAD will be described. The speech feature extraction apparatus100includes a training part100A and an utterance-level feature extraction part100B. But the training part100A and the utterance-level feature extraction part100B are not necessarily to be tied together. The utterance-level feature extraction part100B can be used alone with the training part in the prior arts disclosed in the NPL1 and the NPL2.

As shown inFIG.2, the training part100A includes a speech data storage101, an acoustic feature extraction unit102A, a VAD unit103A, a VAD threshold storage104A, a selected acoustic features storage105, a VAD process unit106A, a frame weights storage107, a utterance-level feature extractor training unit108and an utterance-level feature extractor parameter storage109.

The speech feature extraction part100B includes an acoustic feature extraction unit102B, a VAD unit103B, a VAD threshold storage104B, an acoustic features of selected frames storage110, a VAD posterior unit106B, a frame weights storage111, an utterance-level feature extraction unit112and an utterance-level features storage113.

The acoustic feature extraction unit102A and the acoustic feature extraction unit102B have the same function. The VAD unit103A and the VAD unit103B have the same function. The VAD unit103A and the VAD unit103B function as the above described the VAD unit103inFIG.1. The VAD process unit106A and the VAD process unit106B have the same function. The VAD process unit106A and the VAD process unit106B function as the above described the VAD process unit106.

The VAD threshold storage104A and the VAD threshold storage104B may be configured with the same storage, meaning that the same threshold is used in the training part and speech feature extraction part. Note that in case of VAD threshold storage, it is also possible to have different components in the training part100A and utterance-level feature extraction part100B.

The acoustic feature extraction unit102extracts acoustic feature vectors f from data in speech data storage101. VAD unit103A applies VAD to the acoustic features and obtains a VAD posterior P for each frame.

The VAD unit103A compares VAD posteriors with a pre-determined VAD threshold THETA stored in VAD threshold storage104A, and drops those frames whose VAD posteriors are smaller than the threshold (P<THETA), then stores the acoustic features of the remaining frames{fi|Pi>=THETA} in selected acoustic feature storage105.

The VAD process unit106A passes the VAD posteriors P to a function and obtains weights for those frames w=F(P), then VAD process unit106A stores them in frame weights storage107.

The utterance-level feature extractor training unit108reads the acoustic features of the selected frames from the selected acoustic feature storage105, and corresponding weights from the frame weights storage107, trains an utterance-level feature extractor, and finally stores the extractor in the utterance-level feature extractor parameters storage109.

In the speech feature extraction part, the acoustic feature extraction unit102B extracts acoustic feature vectors from the input speech data. The VAD unit103B applies VAD to the acoustic feature vectors and obtains a VAD posterior for each frame. The VAD unit103B compares the VAD posteriors with a pre-determined VAD threshold stored in the VAD threshold storage104B, and drops those frames whose VAD posteriors are smaller than the threshold.

The acoustic features of the remaining frames are stored in selected acoustic feature storage110. The VAD process unit106B passes the VAD posteriors to the function F(P) and obtains weights and stores them in the frame weights storage111.

The utterance-level feature extraction unit112reads the acoustic features of the selected frames from the selected acoustic feature storage110and the corresponding weights from the frame weights storage111, and extractor parameters from the utterance-level feature extractor parameter storage109. It extracts one feature vector for the input utterance, and stores it in the utterance-level feature storage113.

In one example of NN (Neural Network)-based speaker embedding, the NN at least has one input layer, one output layer and multiple hidden layers. As shown in the NN structure figure (FIG.3), the hidden layers include at frame-level process layers, a pooling layer and utterance-level process layers. To train such an NN-based speaker embedding extractor, utterance-level feature extractor training unit108passes the acoustic features from storage105and corresponding frame weights from storage107to the input layer.

The training unit108also passes the speaker IDs to the output layer of the NN. Then it trains the NN and obtains the parameters of hidden layers and stores them in the storage109(FIG.4). So, in the NN-based speaker embedding case, besides the acoustic feature of the selected frames are stored in storage105, speaker ID is also carried together with the acoustic features from speech data storage101.

The utterance-level feature extractor parameter storage109stores NN parameters (FIG.4), which at least includes frame-level process layers parameters, pooling layer parameters and utterance-level process layers parameters. In the speech feature extraction part100B, input layer is the acoustic features stored in storage110and frame weights from storage111, and the output layer is removed. Hidden layers are from the storage109. The NN passes the inputs forward and one of the output of the utterance-level process layers is used as the speaker embedding-utterance-level feature.

In the example of i-vector, speaker IDs are not necessary in the utterance-level feature extractor training unit108. The utterance-level feature extractor training unit108trains a total variability matrix (T matrix) and stores it in the utterance-level feature extractor parameter storage109(FIG.5). In the utterance-level feature extraction part, utterance-level feature extraction unit112extracts i-vectors from the acoustic features in storage110, using the T-matrix stored in storage109, given the frame weights from storage111.

The function in VAD process unit106A and106B is monotonically increasing by the VAD posteriors, to make sure that more likely the frames is to be voice, more weights is given to it in pooling. In addition, it should also satisfy Math. 3 over all frames selected for one utterance. The function has a wide range of choices. The simplest example is Math. 4, where the weight of a frame is linear to the its VAD posterior.

We can also choose function that contains parameters, for example, the Math. 5. Larger ALPHA means that more trust is given to the VAD. We have many other choices like Odds, log Odds, and so on.

F⁡(Pi)=Piα∑jF⁡(PJ´α).[Math.5]
Operations of Apparatus

Next, operations performed by the speech feature extraction apparatus100according to the first embodiment of the present invention will be described with reference toFIGS.6to8.FIGS.1to5will be referenced as necessary in the following description. Also, in the first embodiment, a speech feature extraction method is implemented by causing the speech feature extraction apparatus to operate. Accordingly, the following description of operations performed by the speech feature extraction apparatus100will substitute for a description of the speech feature extraction method of the first embodiment.

The whole operation of speech feature extraction apparatus100will be described by referring toFIG.6.FIG.6is a flowchart showing operations performed by the speech feature extraction apparatus according to the first embodiment of the present invention.FIG.6contains operations of the training part100A and the speech feature extraction part100B. However, this shows an example, the operation of the training and feature extraction can be executed continuously or time interval can be inserted, or the operation of feature re extraction can be executed with other training operation, for example, prior arts disclosed in the NPL1 and NPL2.

First, as shown inFIG.6, in the training part100A, the utterance-level feature extraction unit108trains an utterance-level feature extractor and stores its parameters in storage109(step A01). In the case of NN-based speaker embedding, the NN parameters are stored. And in the case of i-vector, T matrix is stored.

Next, in the utterance-level feature extraction part100B, the utterance-level feature extraction unit112uses the extractor parameters stored in storage109, and extracts utterance-level features from the acoustic features from storage110together with their corresponding frame weights in storage111(step A02).

FIG.7is a flowchart showing specific operation of the training part of the speech feature extractor according to the first embodiment. First, the acoustic feature extraction unit102A reads speech data from storage101(step B01). Then, the acoustic feature extraction unit102A extracts frame-level acoustic features (step B02).

Next, the VAD unit103A applies a VAD and obtains posteriors for all frames (step B03). Next, the VAD unit103A compares the posteriors with a pre-determined threshold and drops frames whose posteriors are smaller than the threshold (step B04).

Next, the VAD process unit106A passes the VAD posteriors to a function F(P) and stores them as frame weights (step B05). Next, the utterance-level feature extractor training unit108trains an utterance-level feature extractor (step B06). Finally, the training unit108stores the extractor parameters in storage109(step B07).

FIG.8is a flowchart showing specific operation of the speech feature extraction part using the same VAD posteriors for dropping frames according to the first embodiment. First, the acoustic feature extraction unit102B reads the input speech data (step C01). Then, the acoustic feature extraction unit102B extracts frame-level acoustic features (step C02).

Next, the VAD unit103B applies VAD and obtains posteriors for all frames (step C03). Next, the VAD unit103B compares the posteriors with a pre-determined threshold and drops frames whose posteriors are smaller than the threshold (step C04).

Next, the VAD process unit106B passes the VAD posteriors to a function F(P) and stores them as frame weights (step C05). Next, the utterance-level feature extraction unit112reads the utterance-level feature extractor parameter in storage109(step C06). Finally, the extraction unit112extracts utterance-level features (step C07).

Effect of First Exemplary Embodiment

The first embodiment can extract more appropriate utterance-level features using weighted pooling in a process converting frame level to utterance level. It uses a function of VAD posteriors as weights. The VAD posteriors are already commonly used in most speech processing systems including speaker recognition to drops frames which are likely to be non-voice. So, the first embodiment doesn't need extra information but can improve the features of utterances.

Program

A program of the first embodiment need only be a program for causing a computer to execute steps A01to A02shown inFIG.6, steps B01to B07shown inFIG.7, and steps C01to C07shown inFIG.8. The speech feature extraction apparatus100and the speech feature extraction apparatus method according to the first embodiment can be realized by installing the program on a computer and executing it. In this case, the Processor of the computer functions as the training part100A and the speech feature extraction part100B, and performs processing.

The program according to the first embodiment may be executed by a computer system constructed using a plurality of computers. In this case, for example, each computer may function as a different one of the training part100A and the speech feature extraction part100B.

Second Embodiment

The first embodiment uses the same posteriors in weighted pooling and frame dropping. However, the VAD often used in frame dropping in speaker recognition is an energy-based method, which is a vulnerable to loud background noise or diverse noisy condition. So, the VAD posteriors are not accurate enough to be used for weighting frames in pooling. The second embodiment allows to use a different VAD to obtain posteriors for weighting frames in pooling, for example, NN-based VAD which is more accurate in various conditions.

Device Configuration

First, a schematic configuration of a speech feature extraction apparatus200according to the second embodiment will be described usingFIG.9.FIG.9is a block diagram showing the specific configuration of the speech feature extraction apparatus according to the second embodiment of the present invention.

In the second embodiment of the present invention, a speech feature extraction apparatus using a new VAD will be described. The speech feature extraction apparatus200includes training part200A and utterance-level feature extraction part200B. But they are not necessarily to be tied together. The utterance-level feature extraction part can be used alone with the training part of the prior arts disclosed in the NPL1 and the NPL2.

As shown inFIG.9, in the speech feature extraction apparatus200of the second embodiment, the training part200A includes a speech data storage201, a acoustic feature extraction unit202A, a first VAD unit203A, a first VAD threshold storage204A, an acoustic feature of selected frames storage205, a second VAD unit206A, a VAD process unit207A, a frame weighs storage208, an utterance-level feature extractor training unit209and an utterance-level feature extractor parameter storage210.

The speech feature extraction part200B includes an acoustic feature extraction unit202B, a first VAD unit203B, a first VAD threshold storage204B, an acoustic feature of selected frames storage211, an second VAD unit206B, a VAD process unit207B, a frame weighs storage212, an utterance-level feature extraction unit213and an utterance-level feature storage214.

The acoustic feature extraction unit202A and202B have the same function. The first VAD unit203A and203B have the same function. The second VAD unit206A and206B have the same function. VAD process unit207A and207B have the same function. The VAD threshold storage204A and204B may be configured with the same storage, meaning that the same threshold is used in the training part and speech feature extraction part. Note that in case of VAD threshold storage, it is also possible to have different components in the training part and utterance-level feature extraction part.

In the training part200A, the acoustic feature unit201extracts acoustic feature vectors f from data in speech data storage201. The first VAD unit203A applies the first VAD to the acoustic features and obtains a VAD posterior P1for each frame. Then, the first VAD unit203A compares the posteriors P1with a pre-determined first VAD threshold THETA stored in the VAD threshold storage204A, and drops those frames whose VAD posteriors are smaller than the threshold (P1<THETA). The acoustic features of the remaining frames{fi|Pi>=THETA} are stored in the acoustic feature selected frames storage205.

The second VAD unit206A applies the second VAD to the acoustic features and obtains the second sets of VAD posteriors P2. The VAD process unit207A passes the second sets of VAD posteriors P2to a function and obtains weights for those frames w=F(P2) and stores them in the frame weights storage208.

The utterance-level feature extractor training unit209reads the acoustic features of the selected frames from the acoustic feature of selected frames storage205, and corresponding weights from the frame weights storage208, then trains an utterance-level feature extractor, and finally stores the extractor parameters in the utterance-level feature extractor parameters storage210.

In the speech feature extraction part200B, the acoustic feature extraction unit202B extracts acoustic feature vectors from the input speech data. The first VAD unit203B applies the first VAD to the acoustic feature vectors and obtains a VAD posterior for each frame.

Comparing with the pre-determined VAD threshold stored in the first VAD threshold storage204B, those frames whose VAD posteriors are smaller than the threshold are dropped and the acoustic features of the remaining frames are stored in the acoustic feature of selected frames storage211.

The second VAD unit206B applies the second VAD to the acoustic feature vectors and obtain another VAD posteriors for each frame. The VAD process unit207B passes the second VAD posteriors to a function and obtain weights for the frame and stores them in the frame weights storage212.

The utterance-level feature extraction unit213reads the acoustic features of the selected frames from selected acoustic feature storage211, the corresponding weights from the frame weights storage212, and extract parameters from the utterance-level feature extractor parameter storage210. The utterance-level feature extraction unit213extracts one feature vector for the input utterance, and stores it in the utterance-level feature storage214.

The second embodiment can also be applied to the case of NN-based speaker embedding and the case of i-vector as well in the same way as the first exemplary embodiment. (See the first embodiment).

The function in the VAD process unit207A and207B is monotonically increasing by the VAD posteriors, to make sure that more likely the frames is to be voice, more weights is given to it in pooling. In addition, it should also satisfy above Math. 3 over all frames selected for one utterance. (See first embodiment).

Operation of Apparatus

Next, the operation of performed by the speech feature extraction apparatus200according to the second embodiment of the present invention will be described with referenceFIGS.10to12.FIG.9will be referenced as necessary in the following description. Also, in the second embodiment, a speech feature extraction method is implemented by causing the speech feature extraction apparatus to operate. Accordingly, the following description of operations performed by the speech feature extraction apparatus200will substitute for a description of the speech feature extraction method of the second embodiment.

The whole operation of speech feature extraction200will be described by referring toFIG.10.FIG.10is a flowchart showing operations performed by the speech feature extraction apparatus according to the second embodiment of the present invention.FIG.10contains operations of the training part200A and the speech feature extraction part200B. However, this shows an example, the operation of the training and feature extraction can be executed continuously or time interval can be inserted, or the operation of feature re extraction can be executed with other training operation, for example, prior art disclosed in the NPL1 and the NPL2.

First, as shown inFIG.10, in the training part200A, the utterance-level feature extractor training unit209trains an utterance-level feature extractor and stores its parameters in storage210(step D01). In the case of NN-based speaker embedding, the NN parameters are stored. And in the case of i-vector, T matrix is stored.

Next, in the utterance-level feature extraction part200B, the utterance-level feature extraction unit213uses the extractor parameters stored in storage210, and extracts utterance-level features from the acoustic features from storage211together with their corresponding frame weights in storage212(step B02).

FIG.11is a flowchart showing specific operation of the training part of the speech feature extractor using another VAD to obtain posteriors for weighted pooling, different from dropping frames, according to the second embodiment.

First, the acoustic feature extraction unit202A reads speech data from storage201(step E01). Then, the acoustic feature extraction unit202A extracts frame-level acoustic features (step E02).

Next, first VAD unit203A applies the first VAD and obtains posteriors for all frames (step E03). Then, the first VAD unit203A compares the posteriors with a pre-determined threshold and drops frames whose posteriors are smaller than the threshold (step E04).

Next, the second VAD unit206A applies the second VAD and obtains the second set of posteriors for all frames (step E05). Then, the VAD process unit207A passes the second set of VAD posteriors to a function F(P2) and stores them as frame weights (step E06).

Next, the utterance-level feature extractor training unit209trains the extractor (step E07). Finally, the training unit209stores the extractor parameters in storage210(step E08).

FIG.12is a flowchart showing specific operation of the speech feature extraction part using another VAD to obtain posteriors for weighted pooling, different from dropping frames, according to the second embodiment.

First, the acoustic feature extraction unit202B reads the input speech data (step F01). Then, the acoustic feature extraction unit202B extracts frame-level acoustic features (step F02).

Next, the first VAD unit203B applies the first VAD and obtains posteriors for all frames (step F03). Then, the VAD unit203B compares the posteriors with a pre-determined threshold and drops frames whose posteriors are smaller than the threshold (step F04).

Next, the second VAD unit206B applies the second VAD and obtains the second set of posteriors for all frames (step F05). Then, the VAD process unit208B passes the second set of VAD posteriors to a function F(P2) and stores them as frame weights (step F06).

Effect of Second Embodiment

The second embodiment can extract more appropriate utterance-level features using weighted pooling in a process converting frame-level to utterance-level. It uses a function of different VAD (generally with higher performance) posteriors as weights, from the VAD used in dropping frames. The VAD which produces posteriors for frame weights can have many choices, for example, NN-based VAD. Such VAD usually have more sophisticated structure than the VAD used in frame dropping. So, its posteriors are also more accurate to use for weights.

Program

A program of the second embodiment need only be a program for causing a computer to execute steps D01to D02shown inFIG.10, steps E01to E08shown inFIG.11, and steps F01to F07shown inFIG.12. The speech feature extraction apparatus200and the speech feature extraction apparatus method according to the second embodiment can be realized by installing the program on a computer and executing it. In this case, the Processor of the computer functions as the training part200A and the speech feature extraction part200B, and performs processing.

The program according to the second embodiment may be executed by a computer system constructed using a plurality of computers. In this case, for example, each computer may function as a different one of the training part200A and the speech feature extraction part200B.

Third Embodiment

The second embodiment uses the posteriors from a more advanced VAD (second VAD) in weighted pooling, other than the VAD used in frame dropping (first VAD). However, sometimes different VAD have very different posteriors for the same frames, which means some non-voice frames may fool one of the VADs that take it as voice frames. Among those frames which are selected by first VAD may have very low posteriors in second VAD. Even though the second embodiment will give low weights for such frames, large amount of such frames still affect the final utterance-level feature. The third embodiment drops frames using both first VAD and second VAD, and then uses the advanced second VAD to give weights in pooling. It will remove the non-voice frames better, so that the final utterance-level features are more accurate.

Device Configuration

First, a schematic configuration of a speech feature extraction apparatus300according to the third embodiment will be described usingFIG.13.FIG.13is a block diagram showing the specific configuration of the speech feature extraction apparatus according to the third embodiment of the present invention.

In the third embodiment of the present invention, a speech feature extraction apparatus using a new VAD for both weighted pooling and frame dropping will be described. The speech feature extraction apparatus300includes training part300A and utterance-level feature extraction part300B. But they are not necessarily to be tied together. The utterance-level feature extraction part can be used alone with the training part of the prior arts disclosed in the NPL1 and the NPL2.

As shown inFIG.13, in the speech feature extraction apparatus300, the training part300A includes a speech data storage301, an acoustic feature extraction unit302A, a first VAD unit303A, a first VAD threshold storage304A, an acoustic feature of selected frames storage305, a second VAD unit306A, a second VAD threshold storage307A, a VAD process unit308A, a frame weighs storage309, an utterance-level feature extractor training unit310and an utterance-level feature extractor parameter storage311.

The speech feature extraction part300B includes an acoustic feature extraction unit302B, a first VAD unit303B, a first VAD threshold storage304B, an acoustic feature of selected frames storage312, an second VAD unit306B, an second VAD a threshold storage307B, an VAD process unit308B, a frame weighs storage313, an utterance-level feature extraction unit314and an utterance-level feature storage315.

The acoustic feature extraction unit302A and302B have the same function. The first VAD unit303A and303B have the same function. second VAD unit306A and306B have the same function. The VAD process unit308A and308B have the same function. The first VAD threshold storage304A and304B have the same storage, and the second VAD threshold storage307A and307B may be configured with the same storage, meaning that the same threshold is used in the training part and speech feature extraction part. Note that in case of VAD threshold storage, it is also possible to have different components in the training part and utterance-level feature extraction part.

In the training part300A, the acoustic feature unit302A extracts acoustic feature vectors f from data in the speech data storage301. The first VAD unit303A applies the first VAD to the acoustic features and obtains a VAD posterior P1for each frame. Then, the first VAD unit303A compares the posteriors P1with a pre-determined first VAD threshold THETA′ stored in the first VAD threshold storage304A, and drops those frames whose VAD posteriors are smaller than the threshold (P1<THETA1).

The second VAD unit306A applies the second VAD to the acoustic features and obtains the second sets of VAD posteriors P2. The second VAD unit306A compares the second sets of posteriors P2. with a pre-determined second VAD threshold THETA2stored in second VAD threshold storage307A, and further drops more frames whose second VAD posteriors P2are smaller than the threshold (P2<THETA2). The acoustic features of the remaining frames {fi|(P1i>=THETA1) &&(P2i>=THETA2)} are stored in selected acoustic feature storage305.

The VAD posterior process unit308A passes the second sets of VAD posteriors P2to a function and obtains weights for those frames w=F(P2) and stores them in the frame weights storage309. The utterance-level feature extractor training unit310reads the acoustic features of the selected frames from selected acoustic feature storage305, and corresponding weights from frame weights storage309, then trains an utterance-level feature extractor, and finally stores the extractor parameters in the utterance-level feature extractor parameters storage311.

It should be noted that the two posteriors P1and P2can be compared with a single threshold THETA by linear combining P1and P2in such a way as Math. 6.
κP1+ΔP2θ.  [Math. 6]

In the speech feature extraction part300B, the acoustic feature extraction unit302B extracts acoustic feature vectors from the input speech data. first VAD unit303B applies the first VAD to the acoustic feature vectors and obtains a VAD posterior for each frame. The second VAD unit306B applies the second VAD to the acoustic feature vectors and obtains another VAD posteriors for each frame.

The first VAD unit303B compares the first set of posteriors with a pre-determined first VAD threshold stored in first VAD threshold storage304B, and drops those whose first VAD posteriors are smaller than the first threshold. The second VAD unit306B compares the second set of posteriors of remaining frames, and further drops more frames whose second VAD posteriors are smaller than second VAD threshold.

The frames remained after two selections are stored in acoustic feature of selected frames storage312. The VAD posterior process unit308B passes the second VAD posteriors to a function and obtain weights for the frame and stores them in frame weights storage313. The utterance-level feature extraction unit314reads the acoustic features of the selected frames from selected acoustic feature storage312, the corresponding weights from frame weights storage313, and extractor parameters from utterance-level feature extractor parameter storage311. It extracts one feature vector for the input utterance, and stores it in utterance-level feature storage315.

The third exemplary embodiment can also be applied to the case of NN-based speaker embedding and the case of i-vector as well in the same way as the first and the second exemplary embodiment (see first embodiment).

The function in VAD process unit308A and308B is monotonically increasing by the VAD posteriors, to make sure that more likely the frames is to be voice, more weights is given to it in pooling. In addition, it should also satisfy above Math. 3 over all frames selected for one utterance. (See first embodiment).

Operation of Apparatus

Next, the operation of performed by the speech feature extraction apparatus300according to the third embodiment of the present invention will be described with referenceFIGS.14to16.FIG.13will be referenced as necessary in the following description. Also, in the third embodiment, a speech feature extraction method is implemented by causing the speech feature extraction apparatus to operate. Accordingly, the following description of operations performed by the speech feature extraction apparatus300will substitute for a description of the speech feature extraction method of the third embodiment.

The whole operation of speech feature extraction300will be described by referring toFIG.14.FIG.14is a flowchart showing operations performed by the speech feature extraction apparatus according to the third embodiment of the present invention.FIG.14contains operations of the training part300A and the speech feature extraction part300B. However, this shows an example, the operation of the training and feature extraction can be executed continuously or time interval can be inserted, or the operation of feature re extraction can be executed with other training operation, for example, prior arts disclosed in the NPL1 and the NPL2.

First, as shown inFIG.14, in the training part300A, the utterance-level feature extractor training unit310trains an utterance-level feature extractor and stores its parameters in storage311(step G01). In the case of NN-based speaker embedding, the NN parameters are stored. And in the case of i-vector, T matrix is stored.

Next, in the utterance-level feature extraction part300B, the utterance-level feature extraction unit314uses the extractor parameters stored in storage311, and extracts utterance-level features from acoustic features from storage312together with their corresponding frame weights in storage313(step G02).

FIG.15is a flowchart showing specific operation of the training part of the speech feature extractor using another VAD to obtain posteriors for weighted pooling, and also used in dropping frames in addition to a VAD originally used for dropping frames, according to the third embodiment.

First, the acoustic feature extraction unit302A reads speech data from storage301(step H01). Then, the acoustic feature extraction unit302A extracts frame-level acoustic features (step H02).

Next, the first VAD unit303A applies the first VAD and obtains posteriors for all frames (step H03). Then, the first VAD unit303A compares the posteriors with a pre-determined threshold and drops frames whose posteriors are smaller than the threshold (step H04).

Next, the second VAD unit306A applies the second VAD and obtains the second set of posteriors for all frames (step H05). Then, the second VAD unit306A compares the second set of posteriors with a pre-determined second threshold and further drops more frames whose posteriors are smaller than the second threshold (step H06).

Next, the VAD process unit308A passes the second set of VAD posteriors to a function F(P2) and stores them as frame weights (step H07). Then, the utterance-level feature extractor training unit310trains the extractor (step H08). Finally, the training unit310stores the extractor parameters in storage311(step H09).

FIG.16is a flowchart showing specific operation of the speech feature extraction part using another VAD to obtain posteriors for weighted pooling, and also used in dropping frames in addition to a VAD originally used for dropping frames, according to the third embodiment.

First, the acoustic feature extraction unit302B reads the input speech data (step I01). Then, the acoustic feature extraction unit302B extracts frame-level acoustic features (step I02).

Next, the first VAD unit303B applies the first VAD and obtains posteriors for all frames (step I03). Then, the VAD unit303B compares the posteriors with a pre-determined first threshold and drops frames whose posteriors are smaller than the threshold (step I04).

Next, the second VAD unit306B applies the second VAD and obtains the second set of posteriors for all frames (step I05). Then, the second VAD unit306A compares the second set of posteriors with a pre-determined second threshold and further drops more frames whose posteriors are smaller than the second threshold (step I06).

Next, the VAD process unit308B passes the second set of VAD posteriors to a function F(P2) and stores them as frame weights (step I07). Then, the utterance-level feature extraction unit314reads the utterance-level feature extractor parameter in storage312(step I08). Finally, the extraction unit314extracts utterance-level features (step I09).

Effect of Third Embodiment

The third exemplary embodiment can extract more appropriate utterance-level features using weighted pooling in a process converting frame-level to utterance-level. It uses two VADs for dropping frames and uses a function of posteriors from the more advance one of the two VADs as weights.

The VAD which produces posteriors for frame weights, same as that in the second embodiment, can have many choices, for example, NN-based VAD. Such VAD usually have more sophisticated structure than the VAD used in frame dropping.

Two VADs are used for dropping frames to avoid some noisy frames fooling one VAD as voice. So, the final utterance-level feature is obtained by pooling the voiced frames with weights indicating the amount of voice posteriors, and it is more accurate.

Program

A program of the third embodiment need only be a program for causing a computer to execute steps G01to G02shown inFIG.14, steps H01to H09shown inFIG.15, and steps I01to109shown inFIG.16. The speech feature extraction apparatus300and the speech feature extraction apparatus method according to the second embodiment can be realized by installing the program on a computer and executing it. In this case, the Processor of the computer functions as the training part300A and the speech feature extraction part300B, and performs processing.

The program according to the third embodiment may be executed by a computer system constructed using a plurality of computers. In this case, for example, each computer may function as a different one of the training part300A and the speech feature extraction part300B.

Physical Configuration

The following describes a computer that realizes the speech feature extraction apparatus by executing the program of the first, second or third embodiment, with reference toFIG.17.FIG.17is a block diagram showing an example of a computer that realizes the speech feature extraction apparatus according to the first to the third embodiment of the present invention.

As shown inFIG.17, the computer10includes a CPU (Central Processing Unit)11, a main memory12, a storage device13, an input interface14, a display controller15, a data reader/writer16, and a communication interface17. These units are connected via a bus21so as to be capable of mutual data communication.

The CPU11carries out various calculations by expanding programs (codes) according to the present embodiment, which are stored in the storage device13, to the main memory12and executing them in a predetermined sequence. The main memory12is typically a volatile storage device such as a DRAM (Dynamic Random-Access Memory). Also, the program according to the present embodiment is provided in a state of being stored in a computer-readable storage medium20. Note that the program according to the present embodiment may be distributed over the Internet, which is connected to via the communication interface17.

Also, specific examples of the storage device13include a semiconductor storage device such as a flash memory, in addition to a hard disk drive. The input interface14mediates data transmission between the CPU11and an input device18such as a keyboard or a mouse. The display controller15is connected to a display device19and controls display on the display device18.

The data reader/writer16mediates data transmission between the CPU11and the storage medium20, reads out programs from the storage medium20, and writes results of processing performed by the computer10in the storage medium20. The communication interface17mediates data transmission between the CPU11and another computer.

Also, specific examples of the storage medium20include a general-purpose semiconductor storage device such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), a magnetic storage medium such as a flexible disk, and an optical storage medium such as a CD-ROM (Compact Disk Read Only Memory).

The pulse rate estimation apparatus100according to the present exemplary embodiment can also be realized using items of hardware corresponding to various components, rather than using the computer having the program installed therein. Furthermore, a part of the pulse rate estimation apparatus100may be realized by the program, and the remaining part of the pulse rate estimation apparatus100may be realized by hardware.

The above-described embodiment can be partially or entirely expressed by, but is not limited to, the following Supplementary Notes 1 to 24.

A speech feature extraction apparatus comprising:

a voice activity detection unit that drops non-voice frames from frames corresponding to an input speech utterance, and calculates a posterior of being voiced for each frame;

a voice activity detection process unit that calculates a function value as weights in pooling frames to produce an utterance-level feature, from a given a voice activity detection posterior;

an utterance-level feature extraction unit that extracts an utterance-level feature, from the frame on a basis of multiple frame-level features, using the function values.

The speech feature extraction apparatus according to supplementary note 1, further comprising

a utterance-level feature extractor training unit that trains the utterance-level feature extraction unit to generate utterance-level feature extraction parameters using the multiple frame-level features and weights as the function values calculated by the voice activity detection process unit.

The speech feature extraction apparatus according to supplementary note 1, further comprising

a second voice activity detection unit that drops non-voice frames and calculates a second posterior of being voiced for each frame,

Wherein the utterance-level feature extraction unit utilizes weights from functions of the second posteriors while the posteriors are utilized for frame dropping.

The speech feature extraction apparatus according to supplementary note 2,

Wherein the utterance-level feature extractor training unit utilizes weights from functions of the second posteriors while the posteriors are used in frame dropping.

The speech feature extraction apparatus according to supplementary note 3,

Wherein the utterance-level feature extraction unit also utilizes a voice activity detection for obtaining weights for pooling to drop frames.

The speech feature extraction apparatus according to supplementary note 2,

Wherein utterance-level feature extractor training unit also utilizes a voice activity detection for obtaining weights for pooling to drop frames.

The speech feature extraction apparatus according to supplementary note 1,

Wherein the voice activity detection process unit employs a monotonically increasing and non-linear function defined as one of normalized Odds, and normalized log Odds, and the utterance-level feature extraction unit extracts an i-vector as a feature.

The speech feature extraction apparatus according to supplementary note 1,

Wherein the voice activity detection process unit employs a monotonically increasing function, and the utterance-level feature extraction unit extracts a feature using a neural network with at least one pooling layer.

A speech feature extraction method comprising:

(a) a step of dropping non-voice frames from frames corresponding to an input speech utterance, and calculates a posterior of being voiced for each frame;

(b) a step of calculating a function value as weights in pooling frames to produce an utterance-level feature, from a given a voice activity detection posterior;

(c) a step of extracting an utterance-level feature, from the frame on a basis of multiple frame-level features, using the function values.

The speech feature extraction method according to supplementary note 9, further comprising(d) a step of training the utterance-level feature extraction in the step (c) to generate utterance-level feature extraction parameters using the multiple frame-level features and weights as the function values calculated by the step (b).
(Supplementary Note 11)

The speech feature extraction method according to supplementary note 9, further comprising

(e) a step of dropping non-voice frames and calculating a second posterior of being voiced for each frame,

Wherein in the step (c), utilizing weights from functions of the second posteriors while the posteriors are utilized for frame dropping.

The speech feature extraction method according to supplementary note 10,

Wherein in the step (c), utilizing weights from functions of the second posteriors while the posteriors are used in frame dropping.

The speech feature extraction method according to supplementary note 11,

Wherein in the step (c), also utilizing a voice activity detection for obtaining weights for pooling to drop frames.

The speech feature extraction method according to supplementary note 10,

Wherein in the step (d), also utilizing a voice activity detection for obtaining weights for pooling to drop frames.

The speech feature extraction method according to supplementary note 9,

Wherein in the step (b), employing a monotonically increasing and non-linear function defined as one of normalized Odds, and normalized log Odds, and in the step (c), extracting an i-vector as a feature.

The speech feature extraction method according to supplementary note 9,

Wherein in the step (b), employing a monotonically increasing function, and in the step (c), extracting a feature using a neural network with at least one pooling layer.

A computer-readable storage medium storing a program that includes commands for causing a computer to execute:

(a) a step of dropping non-voice frames from frames corresponding to an input speech utterance, and calculates a posterior of being voiced for each frame;

(b) a step of calculating a function value as weights in pooling frames to produce an utterance-level feature, from a given a voice activity detection posterior;

(c) a step of extracting an utterance-level feature, from the frame on a basis of multiple frame-level features, using the function values.

The computer-readable storage medium according to supplementary note 17,

Wherein the program further includes commands causing the computer to execute (d) a step of training the utterance-level feature extraction in the step (c) to generate utterance-level feature extraction parameters using the multiple frame-level features and weights as the function values calculated by the step (b).

The computer-readable storage medium according to supplementary note 17,

Wherein the program further includes commands causing the computer to execute (e) a step of dropping non-voice frames and calculating a second posterior of being voiced for each frame,

In the step (c), utilizing weights from functions of the second posteriors while the posteriors are utilized for frame dropping.

The computer-readable storage medium according to supplementary note 18,

Wherein in the step (c), utilizing weights from functions of the second posteriors while the posteriors are used in frame dropping.

The computer-readable storage medium according to supplementary note 19,

Wherein in the step (c), also utilizing a voice activity detection for obtaining weights for pooling to drop frames.

The computer-readable storage medium according to supplementary note 18,

Wherein in the step (d), also utilizing a voice activity detection for obtaining weights for pooling to drop frames.

The computer-readable storage medium according to supplementary note 17,

Wherein in the step (b), employing a monotonically increasing and non-linear function defined as one of normalized Odds, and normalized log Odds, and in the step (c), extracting an i-vector as a feature.

The computer-readable storage medium according to supplementary note 17,

Wherein in the step (b), employing a monotonically increasing function, and in the step (c), extracting a feature using a neural network with at least one pooling layer.

Although the invention of the present application has been described above with reference to the embodiment, the invention of the present application is not limited to the above embodiment. Various changes that can be understood by a person skilled in the art can be made to the configurations and details of the invention of the present application within the scope of the invention of the present application.

INDUSTRIAL APPLICABILITY

As described above, according to the present invention, it is possible to provide a more accurate representation of an utterance for speaker recognition task. The present invention is useful in fields, e.g. speaker verification.

REFERENCE SIGNS LIST