Abstract:
The present invention defines a pitch-synchronous parametrical representation of speech signals as the basis of speech recognition, and discloses methods of generating the said pitch-synchronous parametrical representation from speech signals. The speech signal is first going through a pitch-marks picking program to identify the pitch periods. The speech signal is then segmented into pitch-synchronous frames. An ends-matching program equalizes the values at the two ends of the waveform in each frame. Using Fourier analysis, the speech signal in each frame is converted into a pitch-synchronous amplitude spectrum. Using Laguerre functions, the said amplitude spectrum is converted into a unit vector, referred to as the timbre vector. By using a database of correlated phonemes and timbre vectors, the most likely phoneme sequence of an input speech signal can be decoded in the acoustic stage of a speech recognition system.

Description:
[0001]    The present application is a continuation in part of patent application Ser. No. 13/692,584, entitled “System and Method for Speech Synthesis Using Timbre Vectors”, filed Dec. 3, 2012, by inventor Chengjun Julian Chen. 
     
    
     FIELD OF THE INVENTION 
       [0002]    The present invention generally relates to automatic speech recognition, in particular to automatic speech recognition using pitch-synchronous spectral parameters, for example timbre vectors. 
       BACKGROUND OF THE INVENTION 
       [0003]    Speech recognition is an automatic process to convert the voice signal of speech into text, which has three steps. The first step, acoustic processing, reduces the speech signal into a parametric representation. The second step is to find the most possible sequences of phonemes from the said parametrical representation of the speech signal. The third step is to find the most possible sequence of words from the possible phoneme sequence and a language model. The current invention is related to a new type of parametric representation of speech signal and the process of converting speech signal into that parametric representation. 
         [0004]    In current commercial speech recognition systems, the speech signal is first multiplied by a shifting process window, typically a Hamming window of duration about 25 msec and a shifts about 10 msec, to form a frame, see  FIG. 2(A) . A set of parameters is produced from each windowed speech signal. Therefore, for each 10 msec, a set of parameters representing the speech signal in the 25 msec window duration is produced. The most widely used parameter representations are linear prediction coefficients (LPC) and mel-frequency cepstral coefficients (MFCC). Such a method has flaws. First, the positions of the processing windows are unrelated to the pitch periods. Therefore, pitch information and spectral information cannot be cleanly separated. Second, because the window duration is typically 2.5 times greater that the shift time, a phoneme boundary is always crossed by two or three consecutive windows. In other words, large number of frames cross phoneme boundaries, see  FIG. 2(A) . 
         [0005]    A better way of parameterizing the speech signal is first to segment the speech signals into frames that are synchronous to the pitch periods, see  FIG. 2(B) . For voiced section of the speech signals,  211 , each frame is a single pitch period,  213 . For unvoiced signals,  212 , the frames  214  are segmented for convenience, typically into frames approximately equal to the average pitch periods of the voiced sections. The advantages of the pitch-synchronous parameterization are: First, the speech signal in a single frame only represent the spectrum or timbre of the speech, decoupled from pitch. Therefore, timbre information is cleanly separated from pitch information. Second, because a phoneme boundary must be either a boundary between a voiced section and an unvoiced section, or at a pitch-period boundary, each frame has a unique phoneme identity. Therefore, each parameter set has a unique phoneme identity. The accuracy of speech recognition can be improved. (See Part E of Springer Handbook of Speech Processing, Springer Verlag 2008). 
       SUMMARY OF THE INVENTION 
       [0006]    The present invention defines a pitch-synchronous parametrical representation of the speech signals as the basis for speech recognition, and discloses methods of generating the said pitch-synchronous parametrical representation from speech signals. 
         [0007]    According to an exemplary embodiment of the invention, see  FIG. 1 , a speech signal is first going through a pitch-marks picking program to pick the pitch marks. The pitch marks are sent to a process unit to generate a complete set of segmentation points. The speech signal is segmented into pitch-synchronous frames according to the said segmentation points. An ends-meeting program is executed to make the values at the two ends of every frame equal. Using Fourier analysis, the speech signal in each frame is converted into a pitch-synchronous amplitude spectrum, then Laguerre functions are used to convert the said pitch-synchronous amplitude spectrum into a unit vector characteristic to the instantaneous timbre, referred to as the timbre vector. Those timbre vectors constitute the parametrical representation of the speech signal. 
         [0008]    Using recorded speech by a speaker or a number of speakers reading a prepared text which contains all phonemes of the target language, an acoustic database can be formed. The speech signal of the read text is converted into timbre vectors. The phoneme identity of each timbre vector is determined by correlating to the text. The average timbre vector and variance for each individual phoneme is collected from the paired record, which forms an acoustic database. 
         [0009]    During speech recognition, the incoming speech signal is first converted into a sequence of timbre vectors. Those timbre vectors are then compared with the timbre vectors in the database to find the most likely phoneme sequence. The possible phoneme sequence is then sent to a language decoder to find out the most likely text. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0010]      FIG. 1  is a block diagram of a speech recognition systems using pitch-synchronous spectral parameters. 
           [0011]      FIG. 2  shows the fundamental difference between the prior-art signal processing methods using a overlapping and shifting process window and the pitch-synchronous method of the present invention. 
           [0012]      FIG. 3  is an example of the asymmetric window for finding pitch marks. 
           [0013]      FIG. 4  is an example of the profile function for finding the pitch marks. 
           [0014]      FIG. 5  is a chart of number of pitch marks as a function of the window scale for optimizing the window scale. 
           [0015]      FIG. 6  shows the ends-meeting program to equalize the values of two ends of the waveform in a pitch period. 
           [0016]      FIG. 7  is an example of amplitude spectrum in a pitch period, including the raw data, those after interpolation, and those recovered from a Laguerre transform. 
           [0017]      FIG. 8  is a graph of the Laguerre functions. 
           [0018]      FIG. 9  is an example of the proximity indices. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0019]    Various exemplary embodiments of the present invention are implemented on a computer system including one or more processors and one or more memory units. In this regard, according to exemplary embodiments, steps of the various methods described herein are performed on one or more computer processors according to instructions encoded on a computer-readable medium. 
         [0020]      FIG. 1  is a block diagram of the automatic speech recognition system according to an exemplary embodiment of the present invention. The input signal is the digitized speech  101 , typically in PCM (pulse-code modulation) format. If simultaneously acquired electroglottograph (EGG) signal  102  is available, pitch marks  104  can be extracted from the EGG signal  102 . Otherwise, through a pitch-mark picking program  103 , the pitch marks  104  in the voiced sections of the speech signal is detected. The list of pitch marks and the speech signal are sent to a processing unit  105  to extend into the unvoiced sections to form a complete list of segmentation points. The unvoiced section is segmented into frames of sizes approximately equal to the average pitch periods in the voiced sections. The speech signal is segmented by the segmenter  107  based on the segmentation points  106  into pitch-synchronous frames  108 . During the segmentation process, the two ends of each said pitch-synchronous are matched by a ends-matching program, ready for Fourier analysis  109 , to generate a pitch-synchronous amplitude spectrum  110 . The pitch-synchronous amplitude spectrum  110  is sent to a Laguerre transform unit  111  to generate timbre vectors  112 . 
         [0021]    The set of the said timbre vectors  112  are sent to the remaining part of the speech recognition engine,  113  through  118 . In the acoustic decoder  113 , the timbre vectors are compared with a database comprising a correlation tables of phonemes or subphoneme units versus timbre vectors. A number of most likely phoneme sequences  115  is generated. The most likely phoneme sequence  115  is sent to language decoder  116 , assisted with language model  117 , to find the most likely output text  118 . 
         [0022]    The fundamental difference of the present invention from prior arts is the way of segmenting speech signals into frames, see  FIG. 2(A) . In prior-art speech recognition systems, the speech signals  201  and  202  are first multiplied by a shifting process window  203 , typically a Hamming window of duration about 25 msec and a shifts about 10 msec, to form a frame. A set of parameters is produced from each windowed speech signal. Therefore, for each 10 msec, a set of parameters representing the speech signal in the 25 msec window duration is produced. The most widely used parameter representations are linear prediction coefficients (LPC) and mel-frequency cepstral coefficients (MFCC). Such a method has flaws. First, the position of the processing window is unrelated to the pitch periods. Pitch information and timbre information cannot be separated cleanly. Second, many frames  203  cross phoneme boundaries, as shown in  FIG. 2(A) , some frames cross the boundary of voiced section  201  and unvoiced section  202 . 
         [0023]    The present invention starts with a different segmentation method. see  FIG. 2(B) . The speech signals  211  and  212  are first segmented into frames that are synchronous to pitch periods,  213  and  214 . For voiced sections of the speech signal,  211 , each frame is a single pitch period,  213 . For unvoiced signals,  212 , the frames  214  are segmented for convenience, typically into frame sizes approximately equal to the average pitch periods of the voiced sections. The advantages of the pitch-synchronous parameterization are: First, the speech signal in a single frame only represent the spectrum or timbre of the speech, decoupled from pitch. Therefore, timbre information is cleanly separated from pitch information. Second, because a phoneme boundary must be either a boundary between a voiced section and an unvoiced section, or at a pitch-period boundary, each frame has a unique phoneme identity, and therefore, each parameter set has a unique phoneme identity. The accuracy of speech recognition can be improved. (See Part E of Springer Handbook of Speech Processing, Springer Verlag 2008). 
         [0024]    To segment the speech signal into pitch-synchronous frames, one known method is to rely on the simultaneously acquired electroglottograph (EGG) signals,  102 . For speech recognition, in most cases there is no electroglottograph instrument. However, to segment the speech signals into pitch-synchronous frames, one does not require the exact glottal closure instants. It only requires the identification of a section in a pitch period where the variation is weak. Based on the observed waveforms, a method to identify the weakly varying section in a pitch period is designed. It is based on the fact that at the starting moment of a pitch period, the signal variation is the greatest. Therefore, by convoluting the speech signal with a asymmetric window function w(n) shown in  FIG. 3 , the location with weakest variation can be found. An example of asymmetric window function is defined on an interval (−N&lt;n&lt;N), with a formula 
         [0000]    
       
         
           
             
               w 
                
               
                 ( 
                 n 
                 ) 
               
             
             = 
             
               
                 ± 
                 exp 
               
                
               
                 { 
                 
                   
                     - 
                     
                       
                         π 
                         2 
                       
                       
                         N 
                         2 
                       
                     
                   
                    
                   
                     n 
                     2 
                   
                 
                 } 
               
                
               
                 
                   sin 
                    
                   
                     ( 
                     
                       
                         π 
                          
                         
                             
                         
                          
                         n 
                       
                       N 
                     
                     ) 
                   
                 
                 . 
               
             
           
         
       
     
         [0025]    The ±sign is used to accommodate the polarity of the pcm signals. If a positive sign is taken, the value is positive for 0&lt;n&lt;N, but becomes zero at n=N; and it is negative for −N&lt;n&lt;0, again becomes zero at n=−N. Denoting the pcm signal as p(n), A profile function is generated 
         [0000]    
       
         
           
             
               f 
                
               
                 ( 
                 m 
                 ) 
               
             
              
             
               
                 ∑ 
                 
                   n 
                   = 
                   
                     - 
                     N 
                   
                 
                 
                   n 
                   &lt; 
                   N 
                 
               
                
               
                 
                   
                     w 
                      
                     
                       ( 
                       n 
                       ) 
                     
                   
                    
                   
                     [ 
                     
                       
                         p 
                          
                         
                           ( 
                           
                             m 
                             + 
                             n 
                           
                           ) 
                         
                       
                       - 
                       
                         p 
                          
                         
                           ( 
                           
                             m 
                             + 
                             n 
                             - 
                             1 
                           
                           ) 
                         
                       
                     
                     ] 
                   
                 
                 . 
               
             
           
         
       
     
         [0026]    Typical result is shown in  FIG. 4 . Here,  401  is the voice signal.  402  indicates the starting point of each pitch period, where the variation of signal is the greatest.  403  is the profile function generated using the asymmetric window function w(n). As shown, the peak positions  404  of the profile function  403  are pointing to the locations with weak variation  405 . The reason why this simple method works is also shown in  FIG. 4 : Each pitch period starts with a large variation of pcm signal at  402 . The variation decreases gradually and becomes weak near the end of each pitch period. 
         [0027]    In order to generate accurate results, the size of the window, N, should be properly chosen. This can be done with a simple test: For a sentence of a given speaker, do the pitch mark finding procedure with a number of different widths N, and count the total number of pitch marks thus generated. If the polarity is correct, for a broad range of window scales, the total number of pitch marks should be stable, within a few percents.  FIG. 5  shows a typical result.  501  is a curve with the correct polarity. When the window size it too small, there are many spurious pitch marks, shown in  501 . When the window scale is approximately correct,  502 , there is a wide range of window scales where the output is stable, here from 9 msec to 15 msec. Choosing a window size of 12 msec is good. If the window scale is too large,  503 , the number of pitch marks reduces quickly. If the polarity is incorrect,  504 , for relatively small window scales, the number of spurious pitch marks increases rapidly. With a wrong polarity, the pitch mark is in the middle of a pitch period. Therefore, the test can also determine the correct polarity. 
         [0028]    As shown in  FIG. 4 , values of the voice signal at two adjacent pitch marks, for example two adjacent  405  points, may not match. The following is an algorithm to equalize the ends. Let the number of sampling points between two adjacent pitch marks be N, and the original voice signal be x 0 (n). The smoothed signal x(n) in a small interval 0&lt;n&lt;M is defined as 
         [0000]    
       
         
           
             
               x 
                
               
                 ( 
                 
                   N 
                   - 
                   n 
                 
                 ) 
               
             
             = 
             
               
                 
                   
                     x 
                     0 
                   
                    
                   
                     ( 
                     
                       N 
                       - 
                       n 
                     
                     ) 
                   
                 
                  
                 
                   n 
                   M 
                 
               
               + 
               
                 
                   
                     x 
                     0 
                   
                    
                   
                     ( 
                     
                       - 
                       n 
                     
                     ) 
                   
                 
                  
                 
                   
                     
                       M 
                       - 
                       n 
                     
                     M 
                   
                   . 
                 
               
             
           
         
       
     
         [0029]    Where M is about N/10. Otherwise x (n)=x 0 (n).  FIG. 6  shows an example. The starting pcm  601  and the end pcm  602  of the original waveform do not match. Fourier analysis cannot be done. By cutting a small section of waveform  603  from the previous pitch period, making a linear interpolation with the small section at the end of the current pitch period 604 to make a new small section  605 , to replace the small section  604  in the original waveform, as shown in the right-hand side waveform, the new ends  606  and  607  are matched. Because the ends-matching is executed in the weakly varying region, the disturbance to the spectrum is negligible. 
         [0030]    For unvoiced sections, see  212  of  FIG. 2 , the signals are segmented for convenience. For example, an unvoiced section is segmented into frames of sizes approximately equal to the average pitch periods in the voiced sections, see  212  in  FIG. 2 . An ends-matching procedure is also applied to the unvoiced frames. 
         [0031]    After the ends are matched,  606  and  607 , the waveform in a frame is processed by Fourier analysis to generate an amplitude spectrum, see  FIG. 7 . Because the number of points in a pitch period is finite, according to the sampling theorem, the number of amplitude spectrum is one half of the number of points in a pitch period, see  701  of  FIG. 7 . For the conversion into timbre vectors using numerical integration, the discrete points  701  is interpolated by various methods, including quadratic, cubic, and the Whittaker-Shannon algorithm, into many more points in the frequency axis, represented by the dotted curve  702 . The continuous spectrum is further processed by a Laguerre transform unit to generate timbre vectors. The continuous amplitude spectrum can be recovered from the timbre vectors with high accuracy,  703 . Therefore, the timbre vectors represents the amplitude spectrum with high accuracy, see following. 
         [0032]    Laguerre functions,  FIG. 8 , are defined as 
         [0000]    
       
         
           
             
               
                 
                   Φ 
                   n 
                 
                  
                 
                   ( 
                   x 
                   ) 
                 
               
               = 
               
                 
                   
                     
                       n 
                       ! 
                     
                     
                       
                         ( 
                         
                           n 
                           + 
                           k 
                         
                         ) 
                       
                       ! 
                     
                   
                 
                  
                 
                   e 
                   
                     
                       - 
                       x 
                     
                     / 
                     2 
                   
                 
                  
                 
                   x 
                   
                     k 
                     / 
                     2 
                   
                 
                  
                 
                   
                     L 
                     n 
                     
                       ( 
                       k 
                       ) 
                     
                   
                    
                   
                     ( 
                     x 
                     ) 
                   
                 
               
             
             , 
           
         
       
     
         [0033]    where k is an integer, typically k=0, 2 or 4; and the associated Laguerre polynomials are 
         [0000]    
       
         
           
             
               
                 L 
                 n 
                 
                   ( 
                   k 
                   ) 
                 
               
                
               
                 ( 
                 x 
                 ) 
               
             
             = 
             
               
                 
                   
                     e 
                     x 
                   
                    
                   
                     x 
                     
                       - 
                       k 
                     
                   
                 
                 
                   n 
                   ! 
                 
               
                
               
                 
                    
                   n 
                 
                 
                    
                   
                     x 
                     n 
                   
                 
               
                
               
                 
                   ( 
                   
                     
                       e 
                       - 
                     
                      
                     
                       x 
                       
                         n 
                         + 
                         k 
                       
                     
                   
                   ) 
                 
                 . 
               
             
           
         
       
     
         [0034]    In  FIG. 8 , the argument of the Laguerre functions is scaled to frequency,  801 . The low-order Laguerre functions are concentrated in low-frequency region,  802 . For higher-order Laguerre functions, in the low-frequency region, there is an oscillation,  803 ; and in high-frequency region, there are broad peaks. Therefore, the Laguerre functions closely resembles the frequency-response curve of human ears. 
         [0035]    The amplitude spectrum A( ω ) is expended into Laguerre functions 
         [0000]    
       
         
           
             
               
                 A 
                  
                 
                   ( 
                   ω 
                   ) 
                 
               
               = 
               
                 
                   ∑ 
                   
                     n 
                     = 
                     0 
                   
                   N 
                 
                  
                 
                   
                     C 
                     n 
                   
                    
                   
                     
                       Φ 
                       n 
                     
                      
                     
                       ( 
                       κω 
                       ) 
                     
                   
                 
               
             
             , 
           
         
       
     
         [0036]    where the coefficients are calculated by 
         [0000]    
       
         
           
             
               
                 C 
                 n 
               
               = 
               
                 
                   ∫ 
                   0 
                   ∞ 
                 
                  
                 
                   κ 
                    
                   
                       
                   
                    
                   
                     A 
                      
                     
                       ( 
                       ω 
                       ) 
                     
                   
                    
                   
                     
                       Φ 
                       n 
                     
                      
                     
                       ( 
                       κω 
                       ) 
                     
                   
                    
                   
                      
                     ω 
                   
                 
               
             
             , 
           
         
       
     
         [0037]    and x is a scaling factor to maximize accuracy. The norm of the vector C is the intensity parameter I, 
         [0000]    
       
         
           
             
               I 
               = 
               
                 
                   
                     ∑ 
                     
                       n 
                       = 
                       0 
                     
                     N 
                   
                    
                   
                     C 
                     n 
                     2 
                   
                 
               
             
             , 
           
         
       
     
         [0038]    and the normalized Laguerre coefficients are defined as 
         [0000]    
       
      
       c 
       n 
       =C 
       n 
       /I.  
      
     
         [0039]    The amplitude spectrum can be recovered from the Laguerre coefficients. By using sufficient number of Laguarre functions, the accuracy can be sufficientley high, see  703  of  FIG. 7 . 
         [0040]    In addition to the normalized Laguerre coefficients, the voicedness index indicating whether the frame is voiced or unvoiced, the duration of the frame (pitch period for voiced sections), and the intensity of that frame are retained as part of the parameters of a frame. Those parameters are useful in the decoding process,  113 . 
         [0041]    The pitch-synchronous parametric representation based on timbre vectors represents the timbre of each pitch period. A timbre distance δ between two frames can be defined as 
         [0000]    
       
         
           
             
               δ 
               = 
               
                 
                   ∑ 
                   
                     n 
                     = 
                     0 
                   
                   N 
                 
                  
                 
                   
                     [ 
                     
                       
                         c 
                         n 
                         
                           ( 
                           1 
                           ) 
                         
                       
                       - 
                       
                         c 
                         n 
                         
                           ( 
                           2 
                           ) 
                         
                       
                     
                     ] 
                   
                   2 
                 
               
             
             , 
           
         
       
     
         [0042]    where c (1)   n  and c( 2 ) n  are elements of the two timbre vectors. Experiments have shown that for two timbre vectors of the same phoneme (not diphthong), the distance is less than 0.1. For timbre vectors of different vowels, the distance is 0.1 to 0.6. For a vowel and a consonant, the distance is even greater. A more convenient parameter, the timbre proximity index, can be defined 
         [0000]        P=− log(δ+ε),
 
         [0043]    where ε is a small positive number (here ε=0.1) to avoid infinity. The timbre proximity index is greater if the two phonemes are similar.  FIG. 9  shows an example of the variation of timbre proximity index with the frame index. Showing is a sequence of three IPA phonemes, [iao].  901  is the variation of P with regard to the base phoneme of [i],  902  is the variation of P with regard to the base phoneme of [a], and  903  is the variation of P with regard to the base phoneme of [o]. Therefore, the phoneme identity of each pitch period can be identified. A speech recognition system of high accuracy can be built based on this method. 
         [0044]    While this invention has been described in conjunction with the exemplary embodiments outlined above, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, the exemplary embodiments of the invention, as set forth above, are intended to be illustrative, not limiting. Various changes may be made without departing from the spirit and scope of the invention.