Patent Publication Number: US-6983055-B2

Title: Method and apparatus for an adaptive binaural beamforming system

Description:
RELATED APPLICATIONS 
   The present application is a continuation-in-part of U.S. patent application Ser. No. 09/593,266, filed Jun. 13, 2000, the disclosure of which is incorporated herein in its entirety for any and all purposes. 

   FIELD OF THE INVENTION 
   The present invention relates to digital signal processing, and more particularly, to a digital signal processing system for use in an audio system such as a hearing aid. 
   BACKGROUND OF THE INVENTION 
   The combination of spatial processing using beamforming techniques (i.e., multiple-microphones) and binaural listening is applicable to a variety of fields and is particularly applicable to the hearing aid industry. This combination offers the benefits associated with spatial processing, i.e., noise reduction, with those associated with binaural listening, i.e., sound location capability and improved speech intelligibility. 
   Beamforming techniques, typically utilizing multiple microphones, exploit the spatial differences between the target speech and the noise. In general, there are two types of beamforming systems. The first type of beamforming system is fixed, thus requiring that the processing parameters remain unchanged during system operation. As a result of using unchanging processing parameters, if the source of the noise varies, for example due to movement, the system performance is significantly degraded. The second type of beamforming system, adaptive beamforming, overcomes this problem by tracking the moving or varying noise source, for example through the use of a phased array of microphones. 
   Binaural processing uses binaural cues to achieve both sound localization capability and speech intelligibility. In general, binaural processing techniques use interaural time difference (ITD) and interaural level difference (ILD) as the binaural cues, these cues obtained, for example, by combining the signals from two different microphones. 
   Fixed binaural beamforming systems and adaptive binaural beamforming systems have been developed that combine beamforming with binaural processing, thereby preserving the binaural cues while providing noise reduction. Of these systems, the adaptive binaural beamforming systems offer the best performance potential, although they are also the most difficult to implement. In one such adaptive binaural beamforming system disclosed by D. P. Welker et al., the frequency spectrum is divided into two portions with the low frequency portion of the spectrum being devoted to binaural processing and the high frequency portion being devoted to adaptive array processing. ( Microphone - array Hearing Aids with Binaural Output - part II: a Two - Microphone Adaptive System,  IEEE Trans. on Speech and Audio Processing, Vol. 5, No. 6, 1997, 543–551). 
   In an alternate adaptive binaural beamforming system disclosed in co-pending U.S. patent application Ser. No. 09/593,728, filed Jun. 13, 2000, two distinct adaptive spatial processing filters are employed. These two adaptive spatial processing filters have the same reference signal from two ear microphones but have different primary signals corresponding to the right ear microphone signal and the left ear microphone signal. Additionally, these two adaptive spatial processing filters have the same structure and use the same adaptive algorithm, thus achieved reduced system complexity. The performance of this system is still limited, however, by the use of only two microphones. 
   SUMMARY OF THE INVENTION 
   An adaptive binaural beamforming system is provided which can be used, for example, in a hearing aid. The system uses more than two input signals, and preferably four input signals, the signals provided, for example, by a plurality of microphones. 
   In one aspect, the invention includes a pair of microphones located in the user&#39;s left ear and a pair of microphones located in the user&#39;s right ear. The system is preferably arranged such that each pair of microphones utilizes an end-fire configuration with the two pairs of microphones being combined in a broadside configuration. 
   In another aspect, the invention utilizes two stages of processing with each stage processing only two inputs. In the first stage, the outputs from two microphone pairs are processed utilizing an end-fire array processing scheme, this stage providing the benefits of spatial processing. In the second stage, the outputs from the two end-fire arrays are processed utilizing a broadside configuration, this stage providing further spatial processing benefits along with the benefits of binaural processing. 
   In another aspect, the invention is a system such as used in a hearing aid, the system comprised of a first channel spatial filter, a second channel spatial filter, and a binaural spatial filter, wherein the outputs from the first and second channel spatial filters provide the inputs for the binaural spatial filter, and wherein the outputs from the binaural spatial filter provide two channels of processed signals. In a preferred embodiment, the two channels of processed signals provide inputs to a pair of transducers. In another preferred embodiment, the two channels of processed signals provide inputs to a pair of speakers. In yet another preferred embodiment, the first and second channel spatial filters are each comprised of a pair of fixed polar pattern units and a combining unit, the combining unit including an adaptive filter. In yet another preferred embodiment, the outputs of the first and second channel spatial filters are combined to form a reference signal, the reference signal is then adaptively combined with the output of the first channel spatial filter to form a first channel of processed signals and the reference signal is adaptively combined with the output of the second channel spatial filter to form a second channel of processed signals. 
   In yet another aspect, the invention is a system such as used in a hearing aid, the system comprised of a first channel spatial filter, a second channel spatial filter, and a binaural spatial filter, wherein the binaural spatial filter utilizes two pairs of low pass and high pass filters, the outputs of which are adaptively processed to form two channels of processed signals. 
   A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
       FIG. 1  is an overview schematic of a hearing aid in accordance with the present invention; 
       FIG. 2  is a simplified schematic of a hearing aid in accordance with the present invention; 
       FIG. 3  is a schematic of a spatial filter for use as either the left spatial filter or the right spatial filter of the embodiment shown in  FIG. 2 ; 
       FIG. 4  is a schematic of a binaural spatial filter for use in the embodiment shown in  FIG. 2 ; and 
       FIG. 5  is a schematic of an alternate binaural spatial filter for use in the embodiment shown in  FIG. 2 . 
   

   DESCRIPTION OF THE SPECIFIC EMBODIMENTS 
     FIG. 1  is a schematic drawing of a hearing aid  100  in accordance with one embodiment of the present invention. Hearing aid  100  includes four microphones; two microphones  101  and  102  positioned in an endfire configuration at the right ear and two microphones  103  and  104  positioned in an endfire configuration at the left ear. 
   In the following description, “RF” denotes right front, “RB” denotes right back, “LF” denotes left front, and “LB” denotes left back. Each of the four microphones  101 – 104  converts received sound into a signal; x RF (n), x RB (n), x LF (n) and x LB (n), respectively. Signals x RF (n), x RB (n), x LF (n) and x LB (n) are processed by an adaptive binaural beamforming system  107 . Within system  107 , each microphone signal is processed by an associated filter with frequency responses of W RF (f), W RB (f), W lF (f) and W LB (f), respectively. System  107  output signals  109  and  110 , corresponding to z R (n) and z L (n), respectively, are sent to speakers  111  and  112 , respectively. Speakers  111  and  112  provide processed sound to the user&#39;s right ear and left ear, respectively. 
   To maximize the spatial benefits of system  100  while preserving the binaural cues, the coefficients of the four filters associated with microphones  101 – 104  should be the solution of the following optimization equation:
 
min W     RF     (f),W     RB     (f),W     LF     (f),W     LB     (f) E[|z L (n)| 2 +|z R (n) 2 |]  (1)
 
where C T  W=g, E(f)=0, and L(f)=0. In these equations, C and g are the known constrained matrix and vector; W is a weight matrix consisting of W RF (f), W RB (f), W lF (f) and W LB (f); E(f) is the difference in the ITD before and after processing; and L(f) is the difference in the ILD before and after processing. As Eq. (1) is a nonlinear constrained optimization problem, it is very difficult to find the solution in real-time.
 
     FIG. 2  is an illustration of a simplified system in accordance with the present invention. In this system, processing is performed in two stages. In the first stage of processing, spatial filtering is performed individually for the right channel (ear) and the left channel (ear). Accordingly, x RF (n) and x RB (n) are input to right spatial filter (RSF)  201 . RSF  201  outputs a signal y R (n). Simultaneously, during this stage of processing, x LF (n) and X LB (N) are input to left spatial filter (LSF)  203  which outputs a signal y L (n). In the second stage of processing, output signals y R (n) and y L (n) are input to a binaural spatial filter (BSF)  205 . The output signals from BSF  205 , z R (n)  109  and z L (n)  110 , are sent to the user&#39;s right and left ears, respectively, typically utilizing speakers  111  and  112 . 
   In the embodiment shown in  FIG. 2 , the design and implementation of RSF  201  and LSF  203  can be similar, if not identical, to the spatial filtering used in an endfire array of two nearby microphones. Similarly, the design and implementation of BSF  205  can be similar, if not identical, to the spatial filtering used in a broadside array of two microphones (i.e., where y R (n) and y L (n) are considered as two received microphones signals). 
   An advantage of the embodiment shown in  FIG. 2  is that there are no binaural issues (e.g., ITD and ILD) in the initial processing stage as RSF  201  and LSF  203  operate within the same ear, respectively. The combination of the binaural cues with spatial filtering is accomplished in BSF  205 . As a result, this embodiment offers both design simplicity and a means of being implemented in real-time. 
   Further explanation will now be provided for the related adaptive algorithms for RSF  201 , LSF  203  and BSF  205 . With respect to the adaptive processing of RSF  201  and LSF  203 , preferably a fixed polar pattern based adaptive directionality scheme is employed as illustrated in  FIG. 3  and as described in detail in co-pending U.S. patent application Ser. No. 09/593,266, the disclosure of which is incorporated herein in its entirety. It should be understood that although the description provided below refers to the structure and algorithm used in LSF  203 , the structure and algorithm used in RSF  201  is identical. Accordingly, RSF  201  is not described in detail below. The related algorithms will apply to RSF  201  with replacement of x LF (n) and x LB (n) by x RF (n) and x RB (n), respectively. 
   The adaptive algorithm for two nearby microphones in an endfire array for LSF  203  is primarily based on an adaptive combination of the outputs from two fixed polar pattern units  301  and  302 , thus making the null of the combined polar-pattern of the LSF output always toward the direction of the noise. The null of one of these two fixed polar patterns is at zero (straight ahead of the subject) and the other&#39;s null is at 180 degrees. These two polar patterns are both cardioid. The first fixed polar pattern unit  301  is implemented by delaying the back microphone signal x LB (n) by the value d/c with a delay unit  303  and subtracting it from the front microphone signal, x LF (n), with a combining unit  305 , where d is the distance separating the two microphones and c is the speed of the sound. Similarly, the second fixed polar pattern unit is implemented by delaying the front microphone signal x LF (n) by the value d/c with a delay unit  307  and subtracting it from the back microphone signal, x LB (n), with a combining unit  309 . 
   The adaptive combination of these two fixed polar patterns is accomplished with combining unit  311  by adding an adaptive gain following the output of the second polar pattern. This combination unit provides the output y L (n) for next stage BSF  205  processing. By varying the gain value, the null of the combined polar pattern can be placed at different degrees. The value of this gain, W, is updated by minimizing the power of the unit output y L (n) as follows: 
               W   opt     =       R   12       R   22               (   2   )             
 
where R 12  represents the cross-correlation between the first polar pattern unit output x L1 (n) and the second polar pattern unit x L2 (n) and R 22  represents the power of X L2 (n).
 
   In a real-time application, the problem becomes how to adaptively update the optimization gain W opt  with available samples x L1 (n) and x L2 (n) rather than cross-correlation R 12  and power R 22 . Utilizing available samples x L1 (n) and x L2 (n), a number of algorithms can be used to determine the optimization gain W opt  (e.g., LMS, NLMS, LS and RLS algorithms). The LMS version for getting the adaptive gain can be written as follows:
 
 W ( n+ 1)= W ( n+ 1)+λ x   L2 ( n ) y   L ( n )  (3)
 
where λ is a step parameter which is a positive constant less than 2/P and P is the power of x L2 (n).
 
   For improved performance, λ can be time varying as the normalized LMS algorithm uses, that is, 
               W   ⁡     (     n   +   1     )       =       W   ⁡     (   n   )       +       μ       P   L2     ⁡     (   n   )         ⁢       x   L2     ⁡     (   n   )       ⁢       y   L     ⁡     (   n   )                   (   4   )             
 
where μ is a positive constant less than 2 and P L2 (n) is the estimated power of x L2 (n).
 
   Equations (3) and (4) are suitable for a sample-by-sample adaptive model. 
   In accordance with another embodiment of the present invention, a frame-by-frame adaptive model is used. In frame-by-frame processing, the following steps are involved in obtaining the adaptive gain. First, the cross-correlation between x L1 (n) and x L2 (n) and the power of x L2 (n) at the m&#39;th frame are estimated according to the following equations: 
                   R   ^     12     ⁡     (   m   )       =       1   M     ⁢       ∑     n   =   1     M     ⁢         x   L1     ⁡     (   n   )       ⁢       x   L2     ⁡     (   n   )                     (   5   )                     R   ^     22     ⁡     (   m   )       =       1   M     ⁢       ∑     n   =   1     M     ⁢       x   L2   2     ⁡     (   n   )                   (   6   )             
 
where M is the sample number of a frame. Second, R 12  and R 22  of Equation (2) are replaced with the estimated {circumflex over (R)} 12  and {circumflex over (R)} 22  and then the estimated adaptive gain is obtained by Eqn.(2).
 
   In order to obtain a better estimation and achieve smoother frame-by-frame processing, the cross-correlation between x L1 (n) and x L2 (n) and the power of x L2 (n) at the m&#39;th frame can be estimated according to the following equations: 
                   R   ^     12     ⁡     (   m   )       =         α   M     ⁢       ∑     n   =   1     M     ⁢         x   L1     ⁡     (   n   )       ⁢       x   L2     ⁡     (   n   )             +     β   ⁢           ⁢         R   ^     12     ⁡     (     m   -   1     )                   (   7   )                     R   ^     22     ⁡     (   m   )       =         α   M     ⁢       ∑     n   =   1     M     ⁢       x   L2   2     ⁡     (   n   )           +     β   ⁢           ⁢         R   ^     22     ⁡     (     m   -   1     )                   (   8   )             
 
where α and β are two adjustable parameters and where 0≦α≦1, 0≦β≦1, and α+β=1. Obviously if α=1 and β=0, Equations (7) and (8) become Equations (5) and (6), respectively.
 
   As previously noted, the adaptive algorithms described above also apply to RSF  201 , assuming the replacement of x LF (n) and x LB (n) with x RF (n) and x RB (n), respectively. 
   Since BSF  205  has only two inputs and is similar to the case of a broadside array with two microphones, the implementation scheme illustrated in  FIG. 4  can be used to achieve the effective combination of the spatial filtering and binaural listening. In this implementation of BSF  205 , the reference signal r(n) comes from the outputs of RSF  201  and LSF  203  and is equivalent to y R (n)-y L (n). Reference signal r(n) is sent to two adaptive filters  401  and  403  with the weights given by:
 
 W   R ( n )=[ W   R1 ( n ),  W   R2 ( n ), . . . ,  W   RN ( n )] T  and
 
 W   L ( n )=[ W   L1 ( n ),  W   L2 ( n ), . . . ,  W   LN ( n )] T 
 
Adaptive filters  401  and  403  provide the outputs  405  (a R (n)) and  407  (a L (n)), respectively, as follows: 
                 a   R     ⁡     (   n   )       =         ∑     m   =   1     M     ⁢         W   Rm     ⁡     (   n   )       ⁢     r   ⁡     (     n   -   m   +   1     )           =         W   R   T     ⁡     (   n   )       ⁢     R   ⁡     (   n   )                   (   9   )                   a   L     ⁡     (   n   )       =         ∑     m   =   1     M     ⁢         W   Lm     ⁡     (   n   )       ⁢     r   ⁡     (     n   -   m   +   1     )           =         W   L   T     ⁡     (   n   )       ⁢     R   ⁡     (   n   )                   (   10   )             
 
where R(n)=[r(n), r(n− 1 ), . . . , r(n−N+1)] T  and N is the length of adaptive filters  401  and  403 . Note that although the length of the two filters is selected to be the same for the sake of simplicity, the lengths could be different. The primary signals at adaptive filters  401  and  403  are y R (n) and y L (n). Outputs  109  (z R (n)) and  110  (z L (n)) are obtained by the equations:
 
 z   R ( n )= y   R ( n )− a   R ( n )  (11)
 
 z   L ( n )= y   L ( n )− a   L ( n )  (12)
 
The weights of adaptive filters  401  and  403  are adjusted so as to minimize the average power of the two outputs, that is, 
                 min       W   R     ⁡     (   n   )         ⁢     E   ⁡     (              z   R     ⁡     (   n   )       2          )         =       min       W   R     ⁡     (   n   )         ⁢     E   ⁡     (                y   R     ⁡     (   n   )       -       a   R     ⁡     (   n   )              2     )                 (   13   )                   min       W   L     ⁡     (   n   )         ⁢     E   ⁡     (              z   L     ⁡     (   n   )       2          )         =       min       W   L     ⁡     (   n   )         ⁢     E   ⁡     (                y   L     ⁡     (   n   )       -       a   L     ⁡     (   n   )              2     )                 (   14   )             
 
   In the ideal case, r(n) contains only the noise part and the two adaptive filters provide the two outputs a R (n) and a L (n) by minimizing Equations (13) and (14). Accordingly, the two outputs should be approximately equal to the noise parts in the primary signals and, as a result, outputs  109  (i.e., z R (n)) and  110  (i.e., z L (n)) of BSF  205  will approximate the target signal parts. Therefore the processing used in the present system not only realizes maximum noise reduction by two adaptive filters but also preserves the binaural cues contained within the target signal parts. In other words, an approximate solution of the nonlinear optimization problem of Equation (1) is provided by the present system. 
   Regarding the adaptive algorithm of BSF  205 , various adaptive algorithms can be employed, such as LS, RLS, TLS and LMS algorithms. Assuming an LMS algorithm is used, the coefficients of the two adaptive filters can be obtained from:
 
 W   R ( n+ 1)= W   R ( n )+η R ( n ) z   R ( n )  (15)
 
 W   L ( n+ 1)= W   L ( n )+η R ( n ) x   L ( n )  (16)
 
where η is a step parameter which is a positive constant less than 2/P and P is the power of the input r(n) of these two adaptive filters. The normalized LMS algorithm can be obtained as follows: 
                 W   R     ⁡     (     n   +   1     )       =         W   R     ⁡     (   n   )       +       μ            R   ⁡     (   n   )            2       ⁢     R   ⁡     (   n   )       ⁢       z   R     ⁡     (   n   )                   (   17   )                   W   L     ⁡     (     n   +   1     )       =         W   L     ⁡     (   n   )       +       μ            R   ⁡     (   n   )            2       ⁢     R   ⁡     (   n   )       ⁢       z   L     ⁡     (   n   )                   (   18   )             
 
where μ is a positive constant less than 2.
 
   Based on the frame-by-frame processing configuration, a further modified algorithm can be obtained as follows: 
                 W   Rk     ⁡     (     n   +   1     )       =         W   Rk     ⁡     (   n   )       +       μ            R   ⁡     (   n   )            2       ⁢     R   ⁡     (   n   )       ⁢       z   Rk     ⁡     (   n   )                   (   19   )                   W   Lk     ⁡     (     n   +   1     )       =         W   Lk     ⁡     (   n   )       +       μ            R   ⁡     (   n   )            2       ⁢     R   ⁡     (   n   )       ⁢       z   Lk     ⁡     (   n   )                   (   20   )             
 
where k represents the k&#39;th repeating in the same frame. It is noted that the frame-by-frame algorithm in LSF is different from that for the BSF primarily because in LSF only an adaptive gain is involved.
 
     FIG. 5  illustrates an alternate embodiment of BSF  205 . In this embodiment, output y R (n) of RSF  201  is split and sent through a low pass filter  501  and a high pass filter  503 . Similarly, the output y L (n) of LSF  203  is split and sent through a low pass filter  505  and a high pass filter  507 . The outputs from high pass filters  503  and  507  are supplied to adaptive processor  509 . Output  510  of adaptive processor  509  is combined using combiner  511  with the output of low pass filter  501 , the output of low pass filter  501  first passing through a delay and equilization unit  513  before being sent the combiner. The output of combiner  511  is signal  109  (i.e., z R (n)). Similarly, output  510  is combined using combiner  515  in order to output signal  110  (i.e., z L (n)). 
   In yet another alternate embodiment of BSF  205 , a fixed filter replaces the adaptive filter. The fixed filter coefficients can be the same in all frequency bins. If desired, delay-summation or delay-subtraction processing can be used to replace the adaptive filter. 
   In yet another alternate embodiment, the adaptive processing used in RSF  201  and LSF  203  is replaced by fixed processing. In other words, the first polar pattern units x L1 (n) and x R1 (n) serve as outputs y L (n) and y R (n), respectively. In this case, the delay could be a value other than d/c so that different polar patterns can be obtained. For example, by selecting a delay of 0.342 d/c, a hypercardioid polar pattern can be achieved. 
   In yet another alternate embodiment, the adaptive gain in RSF  201  and LSF  203  can be replaced by an adaptive FIR filter. The algorithm for designing this adaptive FIR filter can be similar to that used for the adaptive filters of  FIG. 4 . Additionally, this adaptive filter can be a non-linear filter. 
   As will be understood by those familiar with the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. For example, although an LMS-based algorithm is used in RSF  201 , LSF  203  and BSF  205 , as previously noted, LS-based, TLS-based, RLS-based and related algorithms can be used with each of these spatial filters. The weights could also be obtained by directly solving the estimated Wienner-Hopf equations. Accordingly, the disclosures and descriptions herein are intended to be illustrative, but not limiting, of the scope of the invention which is set forth in the following claims.