Patent Publication Number: US-2004044531-A1

Title: Speech recognition system and method

Description:
FIELD OF INVENTION  
       [0001] The invention relates to speech recognition system and method, particularly suitable where robustness to variant speech characteristics for example gender, accent, age and level of noise is required.  
       BACKGROUND TO INVENTION  
       [0002] Automated speech recognition is a difficult problem, particularly in applications requiring speech recognition to be free from the constraints of different speaker genders, ages, accents, speaker vocabularies, level of noise and different environments.  
       [0003] Human speech generally comprises a sequence of single sounds or phones. Phonetically similar phones are grouped into phonemes which differentiate between utterances. One method of speech recognition involves building a Hidden Markov Model (HMM) for each word in the expected vocabulary. The various parts of words in the expected vocabulary are represented as states in a left-right HMM .  
       [0004] Methods of implementing and training such HMMs for speech recognition are described in W. H. Abdulla and N. K. Kasabov, “The Concepts of Hidden Markov Model in Speech Recognition”, Technical Report TR99/09, University of Otago, July 1999; W. H. Abdulla and N. K. Kasabov, “Two Pass Hidden Markov Model for Speech Recognition Systems”, Paper #175, Proceedings of the ICICS&#39;99, Singapore, December 1999; and L. R. Rabiner, “A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition”, Proceedings of the IEEE, Vol. 77, No. 2, pp. 257-286, February 1989.  
       SUMMARY OF INVENTION  
       [0005] In broad terms in one form the invention comprises a method of speech recognition comprising the steps of receiving a signal comprising one or more spoken words; extracting a spoken word from the signal using a Hidden Markov Model; passing the spoken word to a plurality of word models, one or more of the word models based on a Hidden Markov Model comprising one or more states, each state including one or more word observations; determining the word model most likely to represent the spoken word; and outputting the word model representing the spoken word.  
       [0006] In broad terms in another form the invention comprises a speech recognition system comprising a receiver configured to receive a signal comprising one or more spoken words; an extractor configured to extract one or more spoken words from the signal using a Hidden Markov Model; a plurality of word models to which the spoken word is passed, one or more of the word models based on a Hidden Markov Model comprising one or more states, each state including one or more word observations; a probability calculator configured to determine the word model most likely to represent the spoken word; and an output device configured to output the word model representing the spoken word.  
       [0007] In broad terms in another form the invention comprises a speech recognition computer program comprising a receiver module configured to receive a signal comprising one or more spoken words; an extractor module configured to extract one or more spoken words from the signal using a Hidden Markov Model; a plurality of word models stored in a memory to which the spoken word is passed, one or more of the word models based on a Hidden Markov Model comprising one or more states, each state including one or more word observations; a probability calculator configured to determine the word model most likely to represent the spoken word; and an output module configured to output the word model representing the spoken word.  
       [0008] In broad terms in yet another form the invention comprises a method of initialising a word model for speech recognition comprising the steps of extracting one or more versions of a spoken word from one or more signals; segmenting the spoken word into one or more states, each state including one or more word observations; and calculating a probability function to represent the word model based on the states and the word observations.  
     
    
    
     BRIEF DESCRIPTION OF THE FIGURES  
     [0009] Preferred forms of the method and system of speech recognition will now be described with reference to the accompanying figures in which:  
     [0010]FIG. 1 is a schematic view of the preferred system;  
     [0011]FIG. 2 is a further schematic view of the system of FIG. 1;  
     [0012]FIG. 3 is the topology of the underlying Markov chain of the models;  
     [0013]FIGS. 4A and 4B show a preferred method for training the models of FIG. 3; and  
     [0014]FIG. 5 shows a preferred method of denoising a speech signal. 
    
    
     DETAILED DESCRIPTION OF PREFERRED FORMS  
     [0015] Referring to FIG. 1, the preferred system  2  comprises a data processor  4  interfaced to a main memory  6 , the processor  4  and the memory  6  operating under the control of appropriate operating and application software or hardware. The processor  4  is interfaced to one or more input devices  8  and one or more output devices  10  with an I/O controller  12 . The system  2  may further include suitable mass storage devices  14 , for example floppy, hard disk or CD Rom drives or DVD apparatus, a screen display  16 , a pointing device  18 , a modem  20  and/or network controller  22 . The various components could be connected via a system bus  24 .  
     [0016] The preferred system is configured for use in speech recognition and is also configured to be trained on model speech signals. The input devices  8  could comprise a microphone and/or a further storage device in which audio signals or representations of audio signals are stored. Output devices  10  could comprise a printer for displaying the speech or language processed by the system, and/or a suitable speaker for generating sound. Speech or language could also be displayed on display device  16 .  
     [0017]FIG. 2 illustrates the computer implemented aspects of the system indicated at  20  stored in memory  6  and arranged to operate with processor  4 . A signal  22  is input into the system through one or more of the input devices  8 . The preferred signal  20  comprises one or more spoken words from one or more speakers of differing genders, ages and/or accents and could further comprise background noise.  
     [0018] Where the signal  22  comprises a high proportion of static or background noise, the speech signal could optionally be processed by signal denoiser  24  before being input to the system  20 . The signal denoiser could comprise a software module installed and operating on a memory, or could comprise a specific hardware device. The preferred signal denoiser  24  uses a wavelet technique both to reduce the dynamic behaviour of the speech signal and to remove unwanted background noise or static. The signal denoiser may, for example, decompose the signal  22  into low frequency and high frequency coefficients and then set all high frequency coefficients below a threshold level to zero followed by reconstruction of the decomposed signal based on the low frequency coefficients and the threshold high frequency coefficients. The signal denoiser  24  is further described below.  
     [0019] The preferred system may further comprise a combination word and feature extractor  25 , a 3 state HMM for speech/background discrimination also arranged to extract one or more spoken words from the signal  22  by discriminating the speech from the background environment in the signal  22 . The extractor  25  is preferably trained on a data set comprising words from different spoken entities in different background environments and normally selected in the range of 50 to 100 words. The extractor  25  is further described below. It could comprise a software module installed and operating on a memory, or could comprise a specific hardware device.  
     [0020] The extracted word or series of extracted words indicated at  28  is then passed to a word probability calculator  30  interfaced to one or more word models  32  stored in a memory. The system  20  preferably comprises a separate word model  32  for each word requiring recognition by the system. Each word model calculates a certain likelihood that the extracted word  28  passed to it is the word represented by the word model.  
     [0021] The probability calculator  30  assesses the respective likelihoods calculated by the word model  30 . A decision maker forming part of the probability calculator determines the word model most likely to represent the extracted word. The model that scores maximum log likelihood log[P(O/λ)] represents the submitted input, where P(O/λ) is the probability of observation O given a model λ. The duration factor is incorporated through an efficient formula which results in improved performance. During recognition, the states&#39; duration are calculated from the backtracking procedure using the Viterbi Algorithm. The log likelihood value is incremented by the log of the duration probability value as follows:  
         log        [       P   ^          (     q   ,     O      λ       )       ]       =       log        [     P        (     q   ,     O      λ       )       ]       +     η   (       length        (   O   )              ∑     j   =   1     N          log        [       p   j                     (     τ   j     )       ]                             
 
     [0022] where η is a scaling factor and τ j  is the normalised duration of being in state j as detected by the Viterbi algorithm.  
     [0023] The recognised word indicated at  34  is then output by the system through output device(s)  10 . The probability calculator could comprise a software module installed and operating on a memory, or could comprise a specific hardware device.  
     [0024] The preferred word model  32  is based on a nine state Continuous Density Hidden Markov Model which is described with reference to FIG. 3. Human speech generally comprises a sequence of single sounds or phones. Each word is preferably segmented uniformly into N states. Speech is produced by the slow movements of articulatory organs. The speech articulators taking up a sequence of different positions produce a stream of sounds forming the speech signal. Each articulatory position in a spoken word, for example, could be represented by a state of different and varying duration.  
     [0025]FIG. 3 shows a HMM  100  representing the underlying structure of the Markov chain. The model is shown as having five different states indicated at  102 A,  102 B,  102 C,  102 D and  102 E respectively, modeled by a mixture of probability density functions, for example Gaussian mixture models. Five states are shown for the purpose of illustration, although there are preferably 9 states and 12 mixtures. The transition between different articulatory positions or states is represented as a ij , the state transition probability. In other words, a ij  is the probability of being in state S j  given state S i .  
     [0026] The model  100  is preferably constrained with a left-right topology to reduce the number of possible paths. When positioned at one state, the model assumes that the next state visited will be either the same state, the state one to the right, or the state two to the right. The left-right topology constraint may be defined as:  
       a   ij =0 for all  j&gt;i+ 2 and  j&lt;i    
     [0027] The same word could be pronounced differently depending on the individual speaker, the accent of the speaker, the language of the speaker and so on. The resulting model has one or more observations in each state, due to the variations in the pronunciation of each word. The training data set preferably comprises 50-100 utterances, from any language, of the same word taken from different speakers.  
     [0028] The model  100  is preferably implemented as a continuous Hidden Markov Model (CHMM) in which the probability density function (pdf) of certain observations O being in a state is considered to be of Gaussian Distribution.  
     [0029] Model parameter initialisation in accordance with the invention uses the following definitions:  
     [0030]  is the pdf distribution which is considered to be Gaussian in this example;  
     [0031] μ im  is the mean of the m-th mixture in state i;  
     [0032] U im  is the covariance of the m-th mixture in state i;  
     [0033] b im (O t ) is the probability of being in state i with mixture m and given observation sequence O t ;  
     [0034] b i (O t ) represents the probability of being in state i given observation sequence O t ;  
     [0035] c im  is the probability of being in state i with mixture m (gain coefficient);  
     [0036] T i  is the total number of observations in state i;  
     [0037] T im  is the total number of observations in state i with mixture m;.  
     [0038] N is the number of states;  
     [0039] M is the number of mixtures in each state.  
     [0040]FIGS. 4A and 4B show a preferred method  200  for training each model to recognise a particular word. FIG. 4A shows those aspects of the method provided by the invention. The remaining aspects of the method shown in FIG. 4B are described in the prior art. Referring to FIG. 4A, the first step, as indicated at  202  is to obtain several versions or observations of individual words, for example the word “zero” spoken several times by different speakers.  
     [0041] As indicated at  203  the next step is to extract feature vectors which are composed of 28 mel scale coefficients (10 mels and one power+9 delta-mels and one delta-power+6 delta-delta mels and one delta-delta-power.  
     [0042] As shown at  204 , each input word is segmented uniformly into N states. Preferably there are 9 states and 12 mixtures. Each speech frame is preferably of window length 23 ms taken every 9 ms. Some prior art techniques use a Viterbi algorithm to detect the states of each version of the training spoken word. These prior art techniques require a previously prepared model which is then optimised based on the training words. These previously prepared models could have been formed from just one speaker.  
     [0043] The present invention does not require a previously prepared model. At step  204 , the invention creates a new model by segmenting each word into N states. We have found that the invention performs better than prior art systems, particularly when it is applied to varying and even unanticipated speakers, accents and languages, as new models are created from the training words.  
     [0044] After segmentation each state will contain several observations, each observation resulting from a different version or observation of individual words. As indicated at  206 , each observation within each state is placed into a different cell. Each cell represents the population of a certain state derived from several observation sequences of the same word.  
     [0045] The resulting populations of each cell are represented by continuous vectors. It is however more useful to use a discrete observation symbol density rather than continuous vectors. Preferably a vector quantizer is arranged to map each continuous observation vector into a discrete code word index. In one form the invention could split the population into 128 code words, indicated at  208 , identify the M most populated code words as indicated at  210 , and calculate the M mixture representatives from the M most populated code words as indicated at  212 .  
     [0046] As shown at  214 , the population of each cell is then reclassified according to the M code words. In other words, the invention calculates W m  classes for each state from M mixtures.  
     [0047] Referring to step  216 , the median of each class is then calculated and considered as the mean μ im . The median is a robust estimate of the centre of each class as it is less affected by outliers. The covariance, U im , is also calculated for each class.  
     [0048] The remaining steps of the model initialisation method are performed as described in the prior art. Referring to FIG. 4B, the gain factor C im  is calculated as indicated at  218  as follows:  
         C     i                 m       =       number                 of                 observations                 being                 in                 state                 i                 and                 mixture                 m       total                 number                 of                 observations                 in                 state                 i                     
 
     [0049] Referring to step  220 , the probability of being in state i with mixture m and given O t  (b im (O t )) and the probability of being in state i given observation sequence O t  (b i (O t )) are calculated as follows:  
       b   im ( O   t )= ( O   t ,μ im   ,U   im )  
     [0050]           b   i          (     O   t     )       =       ∑     m   =   1     M                       C   im            b     i                 m            (     O   t     )                           
     [0051] The probability function of being in a mixture class W im  given O t  in state i is represented as Φ(W im |O t ). Referring to step  222 , it is calculated as follows:  
         Φ        (       W     i                 m            O   t       )       =       C     i                 m       ·         b     i                 m            (     O   t     )           b   i          (     O   t     )                         
 
     [0052] Using maximum likelihood, next estimates of mean, covariance and gain factor indicated at  224  are calculated as follows:  
           C   ^       i                 m       =       1     T   i              ∑     t   =   1       T   i                       Φ        (       w     i                 m            O   t       )                         
 
       T   im   =T   i .ĉ im    
     [0053]           μ   ^       i                 m       =       1     T     i                 m                ∑     t   =   1       T   i                         Φ        (       w     i                 m            O   t       )       ·     O   t                       U   ^       i                 m       =       1     T     i                 m                ∑     t   =   1       T                 i                           Φ        (       w     i                 m            O   t       )       ·     (       O   t     -       μ   ^     im       )              (       O   t     -       μ   ^       i                 m         )     ′                           b   ^       i                 m            (     O   t     )       =       ∑     m   =   1     M                         c   ^       i                 m               (       O   ;       μ   ^       i                 m         ,       U   ^       i                 m         )           ,     1   ≤   i   ≤   N                   b   ^     i          (     O   t     )       =       ∑     m   =   1     M              c   ^       i                 m                b   ^       i                 m            (     O   t     )                           
     [0054] As indicated at step  226 , the next estimate of Φ is calculated as follows:  
                    Φ   ^          (       W   im          O   t       )       =           c   ^       i                 m                b   ^       i                 m            (     O   t     )             ∑     n   =   1     M              c   ^       i                 n                b   ^       i                 n            (     O   t     )                             
 
     [0055] Referring to step  228 , if |Φ(W im |O t )−{circumflex over (Φ)}(W im |O t )|≦ε, where ε is a small threshold, then there is no significant difference between the actual and estimated rates and the model is considered adequately trained.  
     [0056] On the other hand, where there is a significant difference as shown at  229 , the value of Φ(W tm |O t ) is set to the predicted value {circumflex over (Φ)}(W im |O t ) as indicated at  230  and the next estimates of mean, covariance and gain factor are recalculated.  
     [0057] Referring to FIG. 2, the speech signal could optionally be processed by signal denoiser  24  before being input to the system. FIG. 5 illustrates a flow diagram of the preferred denoising method  300 . As indicated at  302 , an input speech signal is received by the input device(s)  8 .  
     [0058] As shown at  304 , the signal is decomposed into high scale low frequency coefficients or approximations and low scale high frequency coefficients or details. Decomposition is preferably performed by a wavelet, for example a symlet of form SYM4 which is decomposed up to level 8. This preferred wavelet is a modification of the Daubechies family of wavelets. The advantage of this form of wavelet is that it has more symmetry than other wavelets and has greater simplicity.  
     [0059] The input signal is preferably decomposed into approximations and details coefficients in a tree of depth 8. Preferably this decomposition may be repeated for more than one level and is preferably performed up to level 8.  
     [0060] The next stage in denoising the signal, as indicated at  306 , is to apply an appropriate threshold to the decomposed signal. The purpose of thresholding is to remove small details from the input signal without substantially affecting the main features of the signal. All details coefficients below a certain threshold level are set to zero.  
     [0061] A fixed form thresholding level is preferably selected for each decomposition level from 1 to 8 and applied to the details coefficients to mute the noise. The threshold level could be calculated using any one of a number of known techniques or suitable functions depending on the type of noise present in the speech signal. One such technique is the “soft thresholding” technique which follows the following sinusoidal function:  
       y   =     {               sgn        (   x   )            (          x        -   Δ     )                   for                      x          &gt;   Δ                 0                 for                      x          ≤   Δ           }                   
 
     [0062] where y is the denoised signal and x is the noisy input signal.  
     [0063] As indicated at  308 , the signal is then reconstructed. Preferably the signal is reconstructed based on the original approximation coefficients of level 8 and the detail coefficients of levels 1 to 8 which have been modified by the thresholding described above. The resulting reconstructed signal is substantially free from noise, this noise having been removed by thresholding.  
     [0064] As indicated at  310 , the reconstructed denoised signal is then output to the speech recognition system. The benefit of denoising is that of reducing background noise and dynamic behaviour in a speech signal. Such noise can be annoying in speaker to speaker conversation in wireless communications. Furthermore, in the field of automated speech recognition, the presence of background noise or static in a speech signal may prevent a speech recognition system correctly determining the beginning and end of spoken words.  
     [0065] Referring to FIG. 2, the speech signal could optionally be processed by a word extractor  26  arranged to extract one or more spoken words from the speech signal. The word extractor is preferably a computer implemented speech/background discrimination model (SBDM) based on a left-right continuous density Hidden Markov Model (CDHMM) described above having three states representing presilence, speech and postsilence respectively.  
     [0066] Unimodal data modelling is used in the parameter estimation. The observations are Mel scale coefficients of the speech signal frames with only 13 coefficients (12 Mels plus one power coefficient). The dynamic delta coefficients are preferably omitted to make the model insensitive to the dynamic behaviour of the signal and this gives more stable background detection. The speech frames for building the model are preferably of length 23 ms taken each 9 ms.  
     [0067] The invention provides a method and system of speech recognition which is particularly suitable where robustness to variant speech characteristics caused by for example gender, accent, age and different types of noise is required. The possible fields of application of the invention are in systems which use speech recognition to execute commands, wheelchair control, vehicles which respond to driver enquiries such as asking the driver about oil level, engine temperature or any other meter reading, interactive games which use speech commands, elevator control, domestic and industrial appliances arranged to be controlled by voice, and communication apparatus such as cellular phones.  
     [0068] The foregoing describes the invention including preferred forms thereof. Alterations and modifications as will be obvious to those skilled in the art are intended to be incorporated within the scope hereof, as defined by the accompanying claims.