Patent Publication Number: US-6993481-B2

Title: Detection of speech activity using feature model adaptation

Description:
This application claims the benefit of U.S. Provisional Patent No. 60/251,749 filed on Dec. 4, 2000. 

   BACKGROUND OF THE INVENTION 
   This invention relates in general to systems for transmission of speech and, more specifically, to detecting speech activity in a transmission. 
   The purpose of some speech activity detection algorithms, or VAD algorithms, for transmission systems is to detect periods of speech inactivity during a transmission. During these periods a substantially lower transmission rate can be utilized without quality reduction to obtain a lower overall transmission rate. A key issue in the detection of speech activity is to utilize speech features that show distinctive behavior between the speech activity and noise. A number of different features have been proposed in prior art. 
   Time Domain Measures 
   In a low background noise environment, the signal level difference between active and inactive speech is significant. One approach is therefore to use the short-term energy and tracking energy variations in the signal. If energy increases rapidly, that may correspond to the appearance of voice activity, however it may also correspond to a change in background noise. Thus, although that method is very simple to implement, it is not very reliable in relatively noisy environments, such as in a motor vehicle, for example. Various adaptation techniques and complementing the level indicator with another time-domain measures, e.g. the zero crossing rate and envelope slope, may improve the performance in higher noise environments. 
   Spectrum Measures 
   In many environments, the main noise sources occur in defined areas of the frequency spectrum. For example, in a moving car most of the noise is concentrated in the low frequency regions of the spectrum. Where such knowledge of the spectral position of noise is available, it is desirable to base the decision as to whether speech is present or absent upon measurements taken from that portion of the spectrum containing relatively little noise. 
   Numerous techniques are known that have been developed for spectral cues. Some techniques implement a Fourier transform of the audio signal to measure the spectral distance between it and an averaged noise signal that is updated in the absence of any voice activity. Other methods use sub-band analysis of the signal, which are close to the Fourier methods. The same applies to methods that make use of cepstrum analysis. 
   The time-domain measure of zero-crossing rate is a simple spectral cue that essentially measures the relation between high and low frequency contents in the spectrum. Techniques are also known to take advantage of periodic aspects of speech. All voiced sounds have determined periodicity—whereas noise is usually aperiodic. For this purpose, autocorrelation coefficients of the audio signal are generally computed in order to determine the second maximum of such coefficients, where the first maximum represents energy. 
   Some voice activity detection (VAD) algorithms are designed for specific speech coding applications and have access to speech coding parameters from those applications. An example is the G729 application, which employs four different measurements on the speech segment to be classified. The measured parameters are the zero-crossing rate, the full band speech energy, the low band speech energy, and 10 line spectral frequencies from a linear prediction analysis. 
   Problems with Conventional Solutions 
   Most VAD features are good at separating voiced speech from unvoiced speech. Therefore the classification scenario is to distinguish between three classes, namely, voiced speech, unvoiced speech, and inactivity. When the background noise becomes loud it can be difficult to distinguish between active unvoiced speech and inactive background noise. Virtually all VAD algorithms have problems with the situation where a single person is also talking over background noise that consists of other people talking (often referred to as babble noise) or an interfering talker. 
   Likelihood Ratio Detection 
   A classic detection problem is to determine whether a received entity belongs to one of two signal classes. Two hypotheses are then possible. Let the received entity be denoted r, then the hypotheses can be expressed:
 
H 1 :rεS 1 
 
H 0 :rεS 0 
 
where S 1  and S 0  are the signal classes. A Bayes decision rule, also called a likelihood ratio test, is used to form a ratio between probabilities that the hypotheses are true given the received entity r. A decision is made according to a threshold τ B :
 
           L   B     ⁡     (   r   )       =         P   ⁢           ⁢     r   ⁡     (     r   |     H   1       )           P   ⁢           ⁢     r   ⁡     (     r   |     H   0       )           ⁢     {           ≥     τ   B             choose   ⁢           ⁢     H   1                 &lt;     τ   B             choose   ⁢           ⁢     H   0                     
 
The threshold τ B  is determined by the a priori probabilities of the hypotheses and costs for the four classification outcomes. If we have uniform costs and equal prior probabilities then τ B= 1 and the detection is called a maximum likelihood detection. A common variant used for numerical convenience is to use logarithms of the probabilities. If the probability density functions for the hypotheses are known, the log likelihood ratio test becomes: 
         L   ⁡     (   r   )       =       log   ⁡     (       P   ⁢           ⁢     r   ⁡     (     r   |     H   1       )           P   ⁢           ⁢     r   ⁡     (     r   |     H   0       )           )       =       log   ⁡     (         f     H   1       ⁡     (   r   )           f     H   0       ⁡     (   r   )         )       ⁢     {           ≥   τ           choose   ⁢           ⁢     H   1                 &lt;   τ           choose   ⁢           ⁢     H   0                       
 
   Gaussian Mixture Modeling 
   Likelihood ratio detection is based on knowledge of parameter distributions. The density functions are mostly unknown for real world signals, but can be assumed to be of a simple, e.g. Gaussian, distribution. More complex distributions can be estimated with more general probability density function (PDF) models. In speech processing, Gaussian mixture (GM) models have been successfully employed in speech recognition and in speaker identification. 
   A Gaussian mixture PDF for d-dimensional random vectors, x, is a weighted sum of densities: 
           f   x     ⁢     (   x   )       =       ∑     k   =   1     M     ⁢       ρ   k     ⁢       f       μ   k     ,     Σ   k         ⁢     (   x   )               
 
where ρ k  are the component weights, and the component densities to ƒ μ     k     ,Σ     k    (x) are Gaussian with mean vectors μ k  and covariance matrices Σ k . The component weights are constrained by 
           ρ   k     &gt;     0   ⁢           ⁢   and   ⁢           ⁢       ∑     k   =   1     M     ⁢     ρ   k           =   1.       
 
   Adaptive Algorithms 
   The GM parameters are often estimated using an iterative algorithm known as an expectation-maximum (EM) algorithm. In classification applications, such as speaker recognition, fixed PDF models are often estimated by applying the EM algorithm on a large set of training data offline. The results are then used as fixed classifiers in the application. This approach can be used successfully if the application conditions (recording equipment, background noise, etc) are similar to the training conditions. In an environment where the conditions change over time, however, a better approach utilizes adaptive techniques. A common adaptive strategy in signal processing is called gradient methods where parameters are updated so that a distortion criterion is decreased. This is achieved by adding small values to the parameters in the negative direction of the first derivative of the distortion criterion with respect to the parameters. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The present invention is described in conjunction with the appended figures: 
       FIG. 1  presents an overview block diagram of an embodiment of a transmitting part of a speech transmitter system; 
       FIG. 2A  presents an overview block diagram of a first embodiment of a VAD algorithm system; 
       FIG. 2B  presents an overview block diagram of a second embodiment of a VAD algorithm system; 
       FIG. 3  presents an overview block diagram of an embodiment of a feature extraction unit; 
       FIG. 4A  presents an overview block diagram of the first embodiment of a classification unit; 
       FIG. 4B  presents an overview block diagram of the second embodiment of a classification unit; 
       FIG. 5  presents a flow diagram of an embodiment of a hangover algorithm; and 
       FIG. 6  presents an overview block diagram of an embodiment of a model update unit. 
   

   In the appended figures, similar components and/or features may have the same reference label. 
   DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
   The ensuing description provides preferred exemplary embodiment(s) only, and is not intended to limit the scope, applicability or configuration of the invention. Rather, the ensuing description of the preferred exemplary embodiment(s) will provide those skilled in the art with an enabling description for implementing a preferred exemplary embodiment of the invention. It being understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the invention as set forth in the appended claims. 
   An ideal speech detector is highly sensitive to the presence of speech signals while at the same time remaining insensitive to non-speech signals, which typically include various types of environmental background noise. The difficulty arises in quickly and accurately distinguishing between speech and certain types of noise signals. As a result, voice activity detection (VAD) implementations have to deal with the trade-off situation between speech clipping, which is speech misinterpreted as inactivity, on one hand and excessive system activity due to noise sensitivity on the other hand. 
   Standard procedures for VAD try to estimate one or more feature tracks, e.g. the speech power level or periodicity. This gives only a one-dimensional parameter for each feature and this is then used for a threshold decision. Instead of estimating only the current feature itself, the present invention dynamically estimates and adapts the probability density function (PDF) of the feature. By this approach more information is gathered, in terms of degrees of freedom for each feature, to base the final VAD decision upon. 
   In one embodiment, the classification is based on statistical modeling of the speech features and likelihood ratio detection. A feature is derived from any tangible characteristic of a digitally sampled signal such as the total power, power in a spectral band, etc. The second part of this embodiment is the continuous adaptation of models, which is used to obtain robust detection in varying background environments. 
   The present invention provides a speech activity detection method intended for use in the transmitting part of a speech transmission system. One embodiment of the invention includes four steps. The first step of the method consists of a speech feature extraction. The second step of the method consists of log-likelihood ratio tests, based on an estimated statistical model, to obtain an activity decision. The third step of the method consists of a smoothing of the activity decision for hangover periods. The fourth step of the method consists of adaptation of the statistical models. 
   Referring first to  FIG. 1 , a block diagram for the transmitting part of a speech transmitter system  100  is shown. The sound is picked up by a microphone  110  to produce an electric signal  120 , which is sampled and quantized into digital format by an A/D converter  130 . The sample rate of the sound signal is chosen to be adequate for the bandwidth of the signal and can typically be 8 KHz, or 16 KHz for speech signals and 32 KHz, 44.1 KHz or 48 KHz for other audio signals such as music, but other sample rates may be used in other embodiments. The sampled signal  140  is input to a VAD algorithm  150 . The output  160  of the VAD algorithm  150  and the sampled signal  140  is input to the speech encoder  170 . The speech encoder  170  produces a stream of bits  180  that are transmitted over a digital channel. 
   VAD Procedure 
   The VAD approach taken by the VAD algorithm  150  in this embodiment is based on a priori knowledge of PDFs of specific speech features in the two cases where speech is active or inactive. The observed signal, u(t), is expressed as a sum of a non-speech signal, n(t), and a speech signal, s(t), which is modulated by a switching function, θ(t):
 
 u ( t )=θ( t ) s ( t ) +n ( t )θ( t )ε{0,1}
 
   The signals contain feature parameters, x s  and x n , and the observed signal can be written as:
 
 u ( t,x ( t ))=θ( t ) s ( t,x   s ( t ))+ n ( t,x   n ( t ))
 
   It is assumed that the feature parameters can be extracted from the observed signal by some extraction procedure. For every time instant, t, the probability density function for the feature can be expressed as:
 
ƒ x ( x )=ƒ x|θ=0 ( x|θ= 0) Pr(θ= 0)+ƒ x|θ=1 ( x|θ= 1) Pr(θ= 1)
 
   With access to the speech and non-speech conditional PDFs, we can regard the problem as a likelihood ratio detection problem: 
         L   ⁡     (     x   0     )       =       log   ⁡     (         f       x   |   θ     =   1       ⁡     (     x   0     )           f       x   |   θ     =   0       ⁡     (     x   0     )         )       ⁢           ⁢     {           ≥   τ           choose   ⁢           ⁢     H   1                 &lt;   τ           choose   ⁢           ⁢     H   0                     
 
where x 0  is the observed feature and τ is the threshold. The higher the ratio, generally, the more likely the observed feature corresponds to speech being present in the sampled signal. It is possible to adjust the decision to avoid false classification of speech as inactivity by letting τ&lt;0. The threshold can also be determined by the a priori probabilities of the two classes, if these probabilities are assumed to be known. The PDFs for speech and non-speech are estimated offline in a training phase for this embodiment.
 
   With reference to  FIGS. 2A and 2B , embodiments of VAD algorithm systems  150  are shown. The embodiment of  FIG. 2A  includes a model update unit  260  to adapt the models to various signal conditions over time to increase likelihood. In contrast, the embodiment of  FIG. 2B  does not adapt over time. The VAD algorithm system  150  consists of four major parts, namely, a feature extraction unit  210 , classification unit  230 , a hangover smoothing function  250 , and a model update function  260 . The VAD algorithm function  150  generally operates according to the following four steps. First, a set of speech features are extracted by the feature extraction unit  210 . Second, features  220  produced by the feature extraction function  210  are used as arguments in the first classification  230 . Third, an initial decision  240  that is produced from the classification unit  230  is smoothened by the hangover smoothing function  250 . Fourth, the statistical models in the model update function  260  are updated based on the current features such that the models are iteratively improved over time. Below each of these four steps are described in further detail. 
   Feature Extraction 
   An embodiment of the feature extraction unit  210  is depicted in  FIG. 3 . The sampled speech signal  140  is divided into frames  315  of N ƒr  samples by the framing unit  320 . If the frame power  330 , as determined by a power calculation unit  325 , is below a certain threshold, T E , a binary decision variable  215 , V P , is set to zero by a threshold tester  315  for later use in the classification. In this embodiment, an N ƒt  (N ƒt  &gt;N ƒr ) samples-long discrete fast Fourier transform (FFT)  350  operates upon a zero-padded and windowed frame produced by the padding and windowing unit  345 . The signal powers in N bands, x j , (the “N powers”)  220  are calculated by adding the logarithms of the absolute values of the Fourier coefficients in each band and normalizing them with the length of the band with the squared absolute values  15  block  220  and the partial sums block  370 . These N powers  220  are the features used in the classification. 
   Likelihood Ratio Tests 
   Two embodiments of the classification unit  230  are shown in  FIGS. 4A and 4B . The embodiment of  FIG. 4A  interfaces with the embodiment of the VAD algorithm system  150  of  FIG. 2A  and includes adaptive inputs  270 . The embodiment of  FIG. 4B  interfaces with the embodiment of the VAD algorithm system  150  of  FIG. 2B  and does not have an adaptive feature. In these embodiments, the N powers  220  or N features  220 , x j , are used in N C  parallel N m -dimensional likelihood ratio generators  420 , where 
       N   =       ∑     m   =   1       N   C       ⁢       N   m     .           
 
A likelihood ratio  430 , η m , is calculated with the likelihood ratio generators  420  by taking the logarithm of a ratio between the activity PDF value and the inactivity PDF value obtained by using the feature as arguments to the PDFs: 
               η   m     =     log   ⁡     (         f   m     (   S   )       ⁡     (     x   m     )           f   m     (   N   )       ⁡     (     x   m     )         )               m   =     1   ⁢           ⁢   …   ⁢           ⁢     N   C                 
 
where ƒ m   (S)  denotes the activity PDF, ƒ m   (N)  denotes the inactivity PDF, and x m  are N m -dimensional vectors formed by grouping the features x j . A weight calculation unit  425  determines a weighting factor  440 , v m , for each likelihood ratio  430 . A test variable  460 , y, is then calculated as a weighted sum of the ratios: 
       y   =       ∑     m   =   1       N   C       ⁢       η   m     ⁢     v   m             
 
Experimentation may be used to determine the best weighting for each likelihood ratio  430 . In one embodiment, each likelihood ratio  430  is equally weighted.
 
   The test variable  460  is compared to a certain threshold, τ I , by a first decision block  465  to obtain a decision variable  470 , V L ,: 
       y   ⁢     {           ≥     τ   I               V   L     =   1               &lt;     τ   I               V   L     =   0                 
 
If an individual channel indicates strong activity by having a large likelihood ratio  430 , η m , greater than another threshold, τ 0 , then a corresponding variable  450 , V m , is set to equal one in a second decision block  445 . The initial activity classification  240 , V I , is calculated as the logical OR of the corresponding and decision variables  450 ,  470 .
 
   This embodiment of the invention utilizes Gaussian mixture models for the PDF models, but the invention is not to be so limited. In the following description of this embodiment, N m =1 and N C =N will be used to imply one-dimensional Gaussian mixture models. It is entirely in the spirit of the invention to employ a number of multivariate Gaussian mixture models. 
   Hangover Smoothing 
   With reference to  FIG. 5 , an embodiment of a hangover algorithm  250  is used to prevent clipping in the end of a talk spurt. The hangover time is dependent of the duration of the current activity. If the talk spurt, n A , is longer than n AM  frames, the hangover time, n O , is fixed to N 1  frames, otherwise a lower fixed hangover time of N 2  frames is used as shown in steps  508 ,  516  and  520 . A logical AND between the output of the hangover smoothing, V H , and the frame power binary variable  215 , V P , yields the final VAD decision  160 , V F . If V I =1 then V H =1 in step  536  and a counter, n A , is incremented in step  532  to count the number of consecutive active frames. Otherwise, if V I , became 0 within the last N 1  or N 2  frames then V H =1 shown in steps  512 ,  524  and  528 . If V I , has been 0 longer than N 1  or N 2  frames, then V H =0 in steps  512 ,  524  and  540 . 
   Model Update 
   The parameters of the active and the inactive PDF models are updated after every frame in the adaptive embodiment shown in  FIG. 2A . Feature data is sampled over time by the model update unit  260  to affect operation in the classification unit  230  to increase likelihood. The stages of updates are performed by the model update unit  260  depicted in  FIG. 6 . Both the PDF models are first updated by a gradient method for a likelihood ascend adaptation using an inactivity likelihood ascend unit  610  and a speech likelihood ascend unit  620 . The inactive PDF model parameters are then adapted to reflect the background by a long-term correction  630 . Finally, a test is performed to assure a minimum model separation  640 , where the active PDF model parameters may be further adapted. 
   Likelihood Ascend 
   The PDF parameters are updated to increase the likelihood. The parameters are the logarithms of the component weights, α j,k   (N)  and α j,k   (S) , the component means, μ j,k   (N)  and μ j,k   (S) , and the variances, λ j,k   (N)  and λ j,k   (S) . For notation convenience the symbol a+=b will in the following denote a(n+1)=a(n)+b(n), where n is an iteration counter. For the update equations we calculate the following probabilities 
               H     0   ,   j       =         f   j     (   N   )       ⁡     (       x   j     ⁡     (   n   )       )       =       ∑     k   =   1     M     ⁢       ρ     j   ,   k       (   N   )       ⁢       f     j   ,   k       (   N   )       ⁡     (       x   j     ⁡     (   n   )       )                       H     1   ,   j       =         f   j     (   S   )       ⁡     (       x   j     ⁡     (   n   )       )       =       ∑     k   =   1     M     ⁢       ρ     j   ,   k       (   S   )       ⁢       f     j   ,   k       (   S   )       ⁡     (       x   j     ⁡     (   n   )       )                           p     j   ,   k       (   N   )       =         ρ     j   ,   k       (   N   )       ⁢       f     j   ,   k       (   N   )       ⁡     (       x   j     ⁡     (   n   )       )           H     0   ,   j                   p     j   ,   k       (   S   )       =         ρ     j   ,   k       (   S   )       ⁢       f     j   ,   k       (   S   )       ⁡     (       x   j     ⁡     (   n   )       )           H     1   ,   j                   
 
   The logarithms of the component weights are updated according to 
               α     j   ,   k       (   N   )       +=       v   α     ⁢     p     j   ,   k       (   N   )                   α     j   ,   k       (   S   )       +=       v   α     ⁢     p     j   ,   k       (   S   )                       ρ     j   ,   k       (   N   )       =     exp   ⁢           ⁢     α     j   ,   k       (   N   )                   ρ     j   ⁢           ⁢   k       (   S   )       =     exp   ⁢           ⁢     α     j   ,   k       (   S   )                   
 
where V α  is some constant controlling the adaptation. The component weights are restricted not to fall below a minimum weight ρ min . They must also add to one and this is assured by 
               ρ     j   ,   k       (   N   )       =       ρ     j   ,   k       (   N   )           ∑     i   =   1     M     ⁢     ρ     i   ,   k       (   N   )                     ρ     j   ,   k       (   S   )       =       ρ     j   ,   k       (   S   )           ∑     i   =   1     M     ⁢     ρ     i   ,   k       (   S   )                         α     j   ,   k       (   N   )       =     ln   ⁢           ⁢     ρ     j   ,   k       (   N   )                   α     j   ,   k       (   S   )       =     ln   ⁢           ⁢     ρ     j   ,   k       (   S   )                   
 
   The variance parameters are updated as standard deviations 
         σ     j   ,   k       (   N   )       +=         v   σ     ⁢     p     j   ,   k       (   N   )       ⁢       (           (         x   j     ⁢     (   n   )       -     μ     j   ,   k       (   N   )         )     2       λ     j   ,   k       (   N   )         -   1     )       σ     j   ,   k       (   N   )         ⁢           ⁢     σ     j   ,   k       (   S   )         +=       v   σ     ⁢     p     j   ,   k       (   S   )       ⁢       (           (         x   j     ⁢     (   n   )       -     μ     j   ,   k       (   S   )         )     2       λ     j   ,   k       (   S   )         -   1     )       σ     j   ,   k       (   S   )                 
         λ     j   ,   k       (   N   )       =           (     σ     j   ,   k       (   N   )       )     2     ⁢           ⁢     λ     j   ,   k       (   S   )         =       (     σ     j   ,   k       (   S   )       )     2           
 
   The variance parameters, λ j,k , are restricted not to fall below a minimum value of λ min . 
   The component means are updated similarly 
         μ     j   ,   k       (   N   )       +=         v   μ     ⁢       p     j   ,   k       (   N   )       ⁢     (           x   j     ⁢     (   n   )       -     μ     j   ,   k       (   N   )           λ     j   ,   k       (   N   )         )       ⁢           ⁢     μ     j   ,   k       (   S   )         +=       v   μ     ⁢       p     j   ,   k       (   S   )       ⁢     (           x   j     ⁢     (   n   )       -     μ     j   ,   k       (   S   )           λ     j   ,   k       (   S   )         )               
 
   As with the component weights, the update equations for the means and the standard deviations also contain adaptation constants, v μ  and ν σ , controlling the step sizes. 
   Long Term Correction 
   In a sufficiently long window there is most likely some inactive frames. The frame with the least power in this window is likely a non-speech frame. To obtain an estimate of the average background level in each band we take the average of the least N sel  power values of the latest N back  frames: 
         b   j     =       0.99   ·     1     N   sel         ⁢       ∑     i   =   1       N   sel       ⁢     x   j     (   i   )               
 
where x j   (i) &lt;x j   (i+1)  are the sorted past feature (power) values {x j (n), x j (n−1), . . . , x j (n−N back )}. The mixture component means of the non-speech PDF are then adapted towards this value according to the equation: 
         μ     j   ,   k       (   N   )       +=       ɛ   back     ⁢     (       b   j     -     m   j     (   N   )         )           
 
where the GMM “global” mean is given by 
         m   j     (   N   )       =       ∑     k   =   1     M     ⁢       ρ     j   ,   k       (   N   )       ⁢     μ     j   ,   k       (   N   )               
 
and the adaptation is controlled by the factor ε back .
 
   Minimum Model Separation 
   In order to keep the speech and non-speech PDFs well separated the mixture component means of the active PDF are then adjusted according to the equations: 
         Δ   j     (   m   )       =       m   j     (   S   )       -     m   j     (   N   )             
             Δ   j     (   m   )       &lt;     Δ   j     (   min   )         ⇒     μ     j   ,   k       (   S   )         +=       (       Δ   j     (   min   )       -     Δ   j     (   m   )         )     ·   0.95         
           where   ⁢           ⁢     m   j     (   N   )         =       ∑     k   =   1     M     ⁢       ρ     j   ,   k       (   N   )       ⁢     μ     j   ,   k       (   N   )             ,     
     ⁢       m   j     (   S   )       =       ∑     k   =   1     M     ⁢       ρ     j   ,   k       (   S   )       ⁢     μ     j   ,   k       (   S   )             ,     and   ⁢           ⁢     Δ   j     (   min   )       ⁢           ⁢   a   ⁢           ⁢   pre   ⁢     -     ⁢   defined         
 
minimum distance. In one embodiment, an additional 5% separation is provided by applying the above technique.
 
   While the principles of the invention have been described above in connection with specific apparatuses and methods, it is to be clearly understood that this description is made only by way of example and not as limitation on the scope of the invention.