Abstract:
A new method for enhancing the speech of multiple speakers in an enclosure (e.g., home, office, etc) using a microphone array is developed. In the method, the direction of arrival of speech sources and non-speech sources are determined and a beamformer-response mask to enhance and suppress the desired and non-desired acoustic sources, respectively, is constructed. To obtain a beamformer that closely approximates the mask, combinations of pre-computed beamformers are optimally combined together.

Description:
RELATED APPLICATIONS 
       [0001]    This application claims priority to U.S. Provisional Patent Application No. 62/018,663, filed Jun. 30, 2014, entitled DETECTION AND ENHANCEMENT OF MULTIPLE SPEECH SOURCES, the contents of which are incorporated by reference herein in their entirety for all purposes. 
     
    
     BACKGROUND 
       [0002]    This invention generally relates to detection and enhancement of acoustic sources. More particularly, embodiments of this invention relate to the detection and enhancement of speech of multiple talkers or acoustic sources from different directions in an indoor environment, such as a home or an office. 
         [0003]    Detection and enhancement of speech sources in an indoor environment is a challenge. Interference may come from many sources including music system, television, babble noise, refrigerator hum, washing machine, lawn mower, printer, and vacuum cleaner. 
         [0004]    When used in an indoor environment a microphone may be used to receive sound from occupants within the environment. As the distance increases, the signal becomes more susceptible to noise and distortion. 
         [0005]    When focusing on cost, power consumption or mobility, a manufacturer may limit the processing power of the devices or the size of the power-supply battery. A manufacturer&#39;s desire to keep costs down may reduce the accuracy and quality to a point that is much lower than their customers&#39; expectations. There is room for improvement for a speech detection and enhancement system, especially in indoor environments. There is a need for a system that detects and enhances multiple speech sources at a low computational cost and at the same time is sensitive, accurate, and has minimal latency. 
         [0006]    It will be appreciated that these systems and methods are novel, as are applications thereof and many of the components, systems, methods and algorithms employed and included therein. It should be appreciated that embodiments of the presently described inventive body of work can be implemented in numerous ways, including as processes, apparata, systems, devices, methods, computer readable media, computational algorithms, embedded or distributed software and/or as a combination thereof. Several illustrative embodiments are described below. 
       SUMMARY 
       [0007]    A system that enhances speech from desired multiple speakers in an indoor environment using a microphone array. The system includes a method for determining the direction of arrival of speech sources and non-speech sources. A beamformer-response mask is constructed to enhance and suppress the desired and non-desired acoustic sources, respectively. To obtain a beamformer that closely approximates the mask, several pre-computed perfect (or near perfect) linear-phase beamformers are then optimally combined together. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0008]    The inventive body of work will be readily understood by referring to the following detailed description in conjunction with the accompanying drawings, in which: 
           [0009]      FIG. 1  illustrates a beamformer with capability for processing and update of coefficients; 
           [0010]      FIG. 2  illustrates a realization of  FIG. 1  in greater detail; 
           [0011]      FIG. 3  illustrates an alternate realization of  FIG. 1  in greater detail; 
           [0012]      FIG. 4  illustrates an acoustic activity detector; 
           [0013]      FIG. 5  illustrates a speech detector; 
           [0014]      FIG. 6  illustrates an exemplary method to compute the acoustic-magnitude profile from various directions; 
           [0015]      FIG. 7  illustrates an exemplary beamformer mask across the frequency and angular directions; 
       
    
    
     DETAILED DESCRIPTION 
       [0016]    A detailed description of the inventive body of work is provided below. While several embodiments are described, it should be understood that the inventive body of work is not limited to any one embodiment, but instead encompasses numerous alternatives, modifications, and equivalents. In addition, while numerous specific details are set forth in the following description in order to provide a thorough understanding of the inventive body of work, some embodiments can be practiced without some or all of these details. Moreover, for the purpose of clarity, certain technical material that is known in the related art has not been described in detail in order to avoid unnecessarily obscuring the inventive body of work. 
         [0017]    In the text which follows a reference to a “beamformer” is a reference to a spatial filter that operates on the output of an array of sensors in order to enhance the amplitude of a coherent wavefront relative to background noise and directional interference. In the text which follows an abbreviation “DOA” is used as an acronym for “direction of arrival”. In the text which follows reference to “beamformer-coefficient” is intended as a reference to adaptive beamforming algorithms with real-value coefficients. 
         [0018]      FIG. 1  illustrates a block diagram of a system  100  for processing and updating the coefficients of a beamformer so as to detect and enhance desired speech sources from multiple talkers from different directions in the presence of noise. The system  100  includes a microphone array  102 , a beamformer-coefficient processing module  104 , and a beamformer  106 . 
         [0019]    The beamformer-coefficient processing module  104  uses the signal from the microphone array  102  to detect the presence of speech and non-speech sources from various directions, and then computes coefficients to enhance desired speech sources. 
         [0020]    The beamformer module  106  is updated with the coefficients computed by module  104  to enhance the desired speech sources. 
         [0021]      FIG. 2  illustrates a more detailed block diagram of the beamformer-coefficient processing module  104 . The processing module  104  includes a speech detector  104 AA, a speech-detector delay alignment  104 AB, a speech DOA processor  104 AC, a non-speech detector  104 AD, a non-speech detector delay alignment  104 AE, a non-speech DOA processor  104 AF, a beamformer mask processor  104 AG, and a beamformer coefficient processor  104 AH. 
         [0022]    The speech detector  104 AA detects if the incoming signal from the microphone array  102  is speech; if it is speech it then the speech DOA processor  104 AC computes the direction and magnitude of the speech source. The processor  104 AC also stores the DOAs and magnitudes of the recent speech sources that are then passed on to the beamformer mask processor  104 AG. The speech detector  104 AA can also have a more detailed classifier to classify if the speech signal is from a male or female speaker, or whether it came from a certain individual. 
         [0023]    The non-speech detector  104 AD detects if the incoming signal from the microphone array  102  is not speech; if it is not speech, the non-speech DOA processor  104 AF computes the direction of the speech source. The processor  104 AF also stores the DOAs and magnitudes of the recent non-speech sources that are then passed on to the beamformer mask processor  104 AG. The non-speech detector  104 AD can also have a classifier to classify the non-speech signals in greater detail, such as from different appliances, electronic audio systems, and various types of transients and noise. 
         [0024]    The beamformer mask processor  104 AG takes in the recently detected speech and non-speech sources from modules  104 AC and  104 AF, respectively. Depending upon the application, the beamformer mask processor  104 AG may select certain desired speech sources while suppressing the other speech and non-speech sources. In other application, it may also be possible that the processor  104 AG may select certain types of non-speech sources while suppressing the other non-speech sources and speech sources. 
         [0025]    Depending upon the application, the beamformer mask processor  104 AG may use several criteria to select the speech or non-speech sources; one criteria is to select signals that are greater than a prescribed threshold with DOA lying between prescribed angular bounds. The output of the mask processor  104 AG is a beamformer-response mask that is then passed on to the beamformer coefficient processor  104 AH. 
         [0026]    The beamformer coefficient processor  104 AH uses the beamformer mask from the beamformer mask processor  104 AG and computes the beamformer coefficients so that the beamformer response closely replicates the beamformer mask. 
         [0027]      FIG. 3  illustrates a more detailed alternate realization of the block diagram of the beamformer-coefficient processing module  104 . In the realization, the estimation module  104  includes an acoustic activity detector  104 BA, an acoustic-activity-detector delay alignment  104 BB, a speech detector  104 BC, a speech-detector delay alignment  104 BD, a speech DOA processor  104 BE, a magnitude-profile processor across different directions  104 BF, a beamformer mask processor  104 BG, and a beamformer-coefficient processor  104 BH. 
         [0028]    The acoustic activity detector  104 BA ensures that the computation of the beamformer coefficients is carried out only when the acoustic signal at the microphones is at a certain level above the background noise. 
         [0029]    The speech detector  104 BC detects if the incoming signal from the microphone array  102  is speech; if it is speech it then the speech DOA processor  104 BE computes the direction and magnitude of the speech source. The processor  104 BE also stores the DOAs and magnitudes of the recent speech sources that are then passed on to the beamformer mask processor  104 BG. The speech detector  104 BC may also have a more detailed classifier to classify if the speech signal is from a male or female speaker, or whether it came from a certain individual. 
         [0030]    The magnitude-profile processor  104 BF scans the acoustic signal across different directions and creates an acoustic-magnitude profile across different directions. The profile is then passed on to the beamformer mask processor  104 BG. 
         [0031]    The beamformer mask processor  104 BG takes in the recently detected speech sources from the speech DOA processor  104 BE and the acoustic magnitude profile from the magnitude-profile processor  104 BF. Depending upon the application, the beamformer mask processor  104 AG may select certain desired speech sources while suppressing the other speech and non-speech sources. 
         [0032]    The beamformer coefficient processor  104 BH uses the beamformer mask from the beamformer mask processor  104 BG and computes the beamformer coefficients so that the beamformer response closely replicates the beamformer mask. 
         [0033]      FIG. 4  illustrates a block diagram of a simple implementation of an acoustic activity detector  104 BA that includes a smooth energy processor  104 BAA, a background noise estimator  104 BAB, and decision logic  104 BAC. 
         [0034]    The decision logic  104 BAC uses the outputs of the smooth energy processor  104 BAA and the background noise processor  104 BAB to decide if the acoustic signal is above the estimated background noise level. For more precise detection of the acoustic activity, subband-based methods where the energy is detected across each subband using frequency-domain or wavelet-transform based analysis can also be used. In another implementation, a beamformer may also be incorporated within the acoustic activity detector  104 BA so that only acoustic signals from preferred spatial directions are analyzed. 
         [0035]      FIG. 5  illustrates a speech detector  104 BC that includes a summer  104 BCA, a single channel noise remover  104 BCB, and a speech detection model  104 BCC. 
         [0036]    The summer  104 BCA combines the signal from the microphone array to a single channel signal and passes it on to the single-channel noise remover  104 BCB. The summer  104 BCA may also be replaced by a beamformer so that only signals from preferred spatial directions are selected for analysis. The cleaned output from the single-channel noise remover  104 BCB is then passed to a speech detection module  104 BCC. The speech detection module  104 BCC detects whether the input signal is speech. If speech, it outputs a TRUE value and if not a FALSE value. The speech detection module  104 BCC may incorporate more detailed detectors that detect whether the speech signal corresponds to a male or a female speaker or to a particular individual. 
         [0037]      FIG. 6  illustrates a flowchart of the acoustic-magnitude profile processor  104 BF to obtain the magnitude profile across various directions. In the flowchart, the beamformer is uploaded with coefficients that are pre-computed to focus in a certain direction. Then, after a prescribed interval the beamformer is update with a new set of coefficients that gradually shifts the direction of focus by a small prescribed angle. In this way, by gradually varying the beamformer angular focus across prescribed directions, the beamformer scans for acoustic signals within the indoor environment. The magnitudes of the acoustic signal scanned across the different directions are stored in a vector, mVec. A temporal leaky average of mVec is then taken to obtain a smooth profile of the magnitude of the acoustic signal across the various directions, which is stored in the vector mSmVec. 
         [0038]      FIG. 7  illustrates a typical desired beamformer mask, M d (θ, ω), across the frequency and angular directions is shown. As can be seen, the mask has two angular passbands, with frequency band lying between flow and fHigh. 
         [0039]    The next step is to obtain a beamformer that has a magnitude response that closely replicates the mask. One new method is to optimally combine pre-computed beamformers. In the method, perfect (or near perfect) linear phase beamformer for different directions are constructed; if M i (θ, ω) is the magnitude response of the pre-computed beamformer for look-direction d(i), then the corresponding linear-phase beamformer response is given by 
         [0000]        B   i (θ, ω)= M   i (θ, ω) e   −jωτ 
 
         [0000]    A linear combination of the various linear-phase beamformers with different magnitude response is given by 
         [0000]    
       
         
           
             
               
                 
                   
                     B 
                      
                     
                       ( 
                       
                         θ 
                         , 
                         ω 
                       
                       ) 
                     
                   
                   = 
                     
                    
                   
                     
                       ∑ 
                       i 
                       
                           
                       
                     
                      
                     
                         
                     
                      
                     
                       
                         c 
                         i 
                       
                        
                       
                         
                           B 
                           i 
                         
                          
                         
                           ( 
                           
                             θ 
                             , 
                             ω 
                           
                           ) 
                         
                       
                     
                   
                 
               
             
             
               
                 
                   = 
                     
                    
                   
                     
                       ∑ 
                       i 
                       
                           
                       
                     
                      
                     
                         
                     
                      
                     
                       
                         c 
                         i 
                       
                        
                       
                         
                           M 
                           i 
                         
                          
                         
                           ( 
                           
                             θ 
                             , 
                             ω 
                           
                           ) 
                         
                       
                        
                       
                          
                         
                           - 
                           jωτ 
                         
                       
                     
                   
                 
               
             
             
               
                 
                   = 
                     
                    
                   
                     
                       M 
                        
                       
                         ( 
                         
                           θ 
                           , 
                           ω 
                         
                         ) 
                       
                     
                      
                     
                        
                       
                         - 
                         jωτ 
                       
                     
                   
                 
               
             
           
         
       
       
         
           where 
         
       
       
         
           
             
               M 
                
               
                 ( 
                 
                   θ 
                   , 
                   ω 
                 
                 ) 
               
             
             = 
             
               
                 ∑ 
                 i 
                 
                     
                 
               
                
               
                   
               
                
               
                 
                   c 
                   i 
                 
                  
                 
                   
                     M 
                     i 
                   
                    
                   
                     ( 
                     
                       θ 
                       , 
                       ω 
                     
                     ) 
                   
                 
               
             
           
         
       
     
         [0000]    and c i  are the weights. One way to obtain the weights, c i , is to minimize the least-square error between M(θ, ω) and the beamformer mask M d (θ, ω); i.e., 
         [0000]      minimize Σ i   |M (θ i , ω i )− M   d (θ i , ω i )| 2 , θ i ∈Θ and ω i ∈Ω
 
         [0000]    Ifm is a vector containing the magnitude responses of the beamformer we have 
         [0000]    
       
         
           
             m 
             = 
               
              
             
               
                 
                   [ 
                   
                     
                       M 
                        
                       
                         ( 
                         
                           
                             θ 
                             1 
                           
                           , 
                           
                             ω 
                             1 
                           
                         
                         ) 
                       
                     
                     , 
                     … 
                      
                     
                         
                     
                     , 
                     
                       M 
                        
                       
                         ( 
                         
                           
                             θ 
                             K 
                           
                           , 
                           
                             ω 
                             K 
                           
                         
                         ) 
                       
                     
                   
                   ] 
                 
                 T 
               
               = 
               Ac 
             
           
         
       
       
         
           where 
         
       
       
         
           
             A 
             = 
             
               [ 
               
                 
                   
                     
                       
                         M 
                         1 
                       
                        
                       
                         ( 
                         
                           
                             θ 
                             1 
                           
                           , 
                           
                             ω 
                             1 
                           
                         
                         ) 
                       
                     
                   
                   
                     … 
                   
                   
                     
                       
                         M 
                         L 
                       
                        
                       
                         ( 
                         
                           
                             θ 
                             1 
                           
                           , 
                           
                             ω 
                             1 
                           
                         
                         ) 
                       
                     
                   
                 
                 
                   
                     ⋮ 
                   
                   
                     ⋱ 
                   
                   
                     ⋮ 
                   
                 
                 
                   
                     
                       
                         M 
                         1 
                       
                        
                       
                         ( 
                         
                           
                             θ 
                             K 
                           
                           , 
                           
                             ω 
                             K 
                           
                         
                         ) 
                       
                     
                   
                   
                     … 
                   
                   
                     
                       
                         M 
                         L 
                       
                        
                       
                         ( 
                         
                           
                             θ 
                             K 
                           
                           , 
                           
                             ω 
                             K 
                           
                         
                         ) 
                       
                     
                   
                 
               
               ] 
             
           
         
       
       
         
           
             c 
             = 
             
               
                 [ 
                 
                   
                     c 
                     1 
                   
                   , 
                   … 
                    
                   
                       
                   
                   , 
                   
                     c 
                     L 
                   
                 
                 ] 
               
               T 
             
           
         
       
     
         [0000]    parameters K and L are the length of the rows and columns of A. Using matrix notation the optimization problem can be expressed as 
         [0000]      minimized ∥ Ac−m   d μ 2   2  
 
         [0000]    where vector c is the optimization variable and 
         [0000]        m   d   =[M   d (θ 1 , ω 1 ), . . . ,  M   d (θ K , ω K )] T  
 
         [0000]    A closed formed solution of the optimal weights, c opt , for the optimization problem is given by 
         [0000]        C   opt =( A   T   A ) −1   A   T   m   d    
         [0040]    Although the foregoing has been described in some detail for purposes of clarity, it will be apparent that certain changes and modifications may be made without departing from the principles thereof. It should be noted that there are many alternative ways of implementing both the processes and apparatuses described herein. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the inventive body of work is not to be limited to the details given herein, which may be modified within the scope and equivalents of the appended claims.