Patent Publication Number: US-2023164274-A1

Title: Post-mixing acoustic echo cancellation systems and methods

Description:
CROSS-REFERENCE 
     This application is a continuation of U.S. patent application Ser. No. 16/523,070, filed on Jul. 26, 2019, which is a continuation of U.S. patent application Ser. No. 15/406,172, now U.S. Pat. No. 10,367,948, filed on Jan. 13, 2017. The contents of these applications are incorporated herein in their entireties. 
    
    
     TECHNICAL FIELD 
     This application generally relates to acoustic echo cancellation performed after the mixing of audio signals from a plurality of acoustic sources, such as microphones used in a conferencing system. In particular, this application relates to systems and methods for cancelling and suppressing acoustic echo from the output of a mixer while efficiently utilizing computation resources. 
     BACKGROUND 
     Conferencing environments, such as boardrooms, conferencing settings, and the like, can involve the use of microphones for capturing sound from audio sources and loudspeakers for presenting audio from a remote location (also known as a far end). For example, persons in a conference room may be conducting a conference call with persons at a remote location. Typically, speech and sound from the conference room may be captured by microphones and transmitted to the remote location, while speech and sound from the remote location may be received and played on loudspeakers in the conference room. Multiple microphones may be used in order to optimally capture the speech and sound in the conference room. 
     However, the microphones may pick up the speech and sound from the remote location that is played on the loudspeakers. In this situation, the audio transmitted to the remote location may therefore include an echo, i.e., the speech and sound from the conference room as well as the speech and sound from the remote location. If there is no correction, the audio transmitted to the remote location may therefore be low quality or unacceptable because of this echo. In particular, it would not be desirable for persons at the remote location to hear their own speech and sound. 
     Existing echo cancellation systems may utilize an acoustic echo canceller for each of the multiple microphones, and a mixer can subsequently mix and process each echo-cancelled microphone signal. However, these types of systems may be computationally intensive and complex. For example, separate and dedicated processing may be needed to perform acoustic echo cancellation on each microphone signal. Furthermore, a typical acoustic echo canceller placed after a mixer would work poorly due to the need to constantly readapt to the mixed signal generated by the mixer should the mixer be dynamic, i.e., the gains on one or more of the mixer channels changes over time. 
     Accordingly, there is an opportunity for acoustic echo cancellation systems and methods that address these concerns. More particularly, there is an opportunity for acoustic echo cancellation systems and methods that cancel and suppress acoustic echo and work with a mixer that has mixed the audio of multiple acoustic sources, while being computationally efficient and resource-friendly. 
     SUMMARY 
     The invention is intended to solve the above-noted problems by providing acoustic echo cancellation systems and methods that are designed to, among other things: (1) generate an echo-cancelled mixed audio signal based on a mixed audio signal from a mixer, information gathered from the audio signal from each of the plurality of acoustic sources, and a remote audio signal; (2) generate the echo-cancelled mixed audio signal by selecting various tap coefficients of a background filter performing a normalized least-mean squares algorithm, a hidden filter, and a mix filter, based on comparing a background error power and a hidden error power; and (3) use a non-linear processor to generate an echo-suppressed mixed audio signal from the echo-cancelled mixed audio signal when the background filter and hidden filter have not yet converged. 
     In an embodiment, a system includes a memory, a plurality of acoustic sources, a mixer in communication with the plurality of acoustic sources and the memory, and an acoustic echo canceller in communication with the mixer, the memory, and a remote audio signal. The plurality of acoustic sources may each be configured to generate an audio signal. The mixer may be configured to mix the audio signal from each of the plurality of acoustic sources to produce a mixed audio signal. The acoustic echo canceller may be configured to generate an echo-cancelled mixed audio signal based on the mixed audio signal, information gathered from each of the plurality of acoustic sources, and the remote audio signal. 
     In another embodiment, a method includes receiving an audio signal from each of a plurality of acoustic sources; receiving a remote audio signal; mixing the audio signal from each of the plurality of acoustic sources using a mixer to produce a mixed audio signal; and generating an echo-cancelled mixed audio signal based on the mixed audio signal, information gathered from the audio signal from each of the plurality of acoustic sources, and the remote audio signal, using an acoustic echo canceller. 
     These and other embodiments, and various permutations and aspects, will become apparent and be more fully understood from the following detailed description and accompanying drawings, which set forth illustrative embodiments that are indicative of the various ways in which the principles of the invention may be employed. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a schematic diagram of a communication system including an acoustic echo canceller, in accordance with some embodiments. 
         FIG.  2    is a schematic diagram of an acoustic echo canceller for use in the communication system of  FIG.  1   , in accordance with some embodiments. 
         FIG.  3    is a flowchart illustrating operations for performing acoustic echo cancellation using the communication system of  FIG.  1   , in accordance with some embodiments. 
         FIG.  4    is a flowchart illustrating operations for running a background filter and a hidden filter while performing acoustic echo cancellation using the communication system of  FIG.  1   , in accordance with some embodiments. 
         FIG.  5    is a flowchart illustrating operations for running a non-linear processor to generate an echo-suppressed mixed audio signal using the communication system of  FIG.  1   , in accordance with some embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     The description that follows describes, illustrates and exemplifies one or more particular embodiments of the invention in accordance with its principles. This description is not provided to limit the invention to the embodiments described herein, but rather to explain and teach the principles of the invention in such a way to enable one of ordinary skill in the art to understand these principles and, with that understanding, be able to apply them to practice not only the embodiments described herein, but also other embodiments that may come to mind in accordance with these principles. The scope of the invention is intended to cover all such embodiments that may fall within the scope of the appended claims, either literally or under the doctrine of equivalents. 
     It should be noted that in the description and drawings, like or substantially similar elements may be labeled with the same reference numerals. However, sometimes these elements may be labeled with differing numbers, such as, for example, in cases where such labeling facilitates a more clear description. Additionally, the drawings set forth herein are not necessarily drawn to scale, and in some instances proportions may have been exaggerated to more clearly depict certain features. Such labeling and drawing practices do not necessarily implicate an underlying substantive purpose. As stated above, the specification is intended to be taken as a whole and interpreted in accordance with the principles of the invention as taught herein and understood to one of ordinary skill in the art. 
     The acoustic echo cancellation systems and methods described herein can generate an echo-cancelled mixed audio signal based on a mixed audio signal from a mixer, information gathered from the audio signal from each of the plurality of acoustic sources, and a remote audio signal, while being computationally efficient and resource-friendly. The systems and methods may eliminate the need for separate acoustic echo cancellers for each acoustic source, e.g., microphone, while maintaining the cancellation benefits of separate acoustic echo cancellers. Moreover, the decreased computational load may allow the use of less expensive hardware (e.g., processor and/or DSP), and/or enable other features to be included in the communication system  100 . User satisfaction may be increased through use of the communication system  100  and acoustic echo canceller  112 . 
       FIG.  1    is a schematic diagram of a communication system  100  for capturing sound from audio sources in an environment using microphones  102  and presenting audio from a remote location using a loudspeaker  104 .  FIG.  2    is a schematic diagram of the acoustic echo canceller  112  included in the communication system  100 . The communication system  100  may generate an echo-cancelled mixed audio signal using the acoustic echo canceller  112  that processes a mixed audio signal from a mixer  106 . The echo-cancelled mixed audio signal may mitigate the sound received from the remote location that is played on the loudspeaker  104 . In this way, the echo-cancelled mixed audio signal may be transmitted to the remote location without the undesirable echo of persons at the remote location hearing their own speech and sound. 
     Environments such as conference rooms may utilize the communication system  100  to facilitate communication with persons at the remote location, for example. The types of microphones  102  and their placement in a particular environment may depend on the locations of audio sources, physical space requirements, aesthetics, room layout, and/or other considerations. For example, in some environments, the microphones may be placed on a table or lectern near the audio sources. In other environments, the microphones may be mounted overhead to capture the sound from the entire room, for example. The communication system  100  may work in conjunction with any type and any number of microphones  102 . Various components included in the communication system  100  may be implemented using software executable by one or more servers or computers, such as a computing device with a processor and memory, and/or by hardware (e.g., discrete logic circuits, application specific integrated circuits (ASIC), programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc. 
       FIGS.  3 - 5    illustrate embodiments of methods for utilizing the communication system  100  and the acoustic echo canceller  112 . In particular,  FIG.  3    illustrates a process  300  for performing acoustic echo cancellation using the communication system  100 ,  FIG.  4    illustrates a method  324  for running a background filter  202  and a hidden filter  204  in the acoustic echo canceller  112 , and  FIG.  5    illustrates a method  312  for conditionally running a non-linear processor  212  in the acoustic echo canceller  112 . In general, a computer program product in accordance with the embodiments includes a computer usable storage medium (e.g., standard random access memory (RAM), an optical disc, a universal serial bus (USB) drive, or the like) having computer-readable program code embodied therein, wherein the computer-readable program code is adapted to be executed by a processor (e.g., working in connection with an operating system) to implement the methods described below. In this regard, the program code may be implemented in any desired language, and may be implemented as machine code, assembly code, byte code, interpretable source code or the like (e.g., via C, C++, Java, Actionscript, Objective-C, Javascript, CSS, XML, and/or others). 
     Referring to  FIG.  1   , the communication system  100  may include the microphones  102 , the loudspeaker  104 , a mixer  106 , a switch  108 , a memory  110 , the acoustic echo canceller  112 , fast Fourier transform (FFT) modules  114 ,  116 ,  118 , and an inverse fast Fourier transform module  120 . Each of the microphones  102  may detect sound in the environment and convert the sound to an audio signal. In embodiments, some or all of the audio signals from the microphones  102  may be processed by a beamformer (not shown) to generate one or more beamformed audio signals, as is known in the art. Accordingly, while the systems and methods are described herein as using audio signals from microphones  102 , it is contemplated that the systems and methods may also utilize any type of acoustic source, such as beamformed audio signals generated by a beamformer. 
     The audio signals from each of the microphones  102  may be received by the mixer  106 , such as at step  318  of the process  300  shown in  FIG.  3   , to generate a mixed audio signal, such as at step  326 . The mixed audio signal generated by the mixer  106  may conform to a desired audio mix such that the audio signals from certain microphones are emphasized and the audio signals from other microphones are deemphasized or suppressed. Exemplary embodiments of audio mixers are disclosed in commonly-assigned patents, U.S. Pat. Nos. 4,658,425 and 5,297,210, each of which is incorporated by reference in its entirety. The mixed audio signal generated at step  326  may be converted into the frequency domain using a fast Fourier transform module  116 , such as at step  328 . 
     In parallel, the audio signals from each of the microphones  102  may be converted to the frequency domain by fast Fourier transform modules  114 , such as at step  320 . One of these converted audio signals may be selected and conveyed at step  322  by a signal selection mechanism, such as a switch  108 , for example. The signal selection mechanism may gather information about each acoustic source (or subset of acoustic sources), e.g., audio signals from the microphones  102  or beamformed audio signals, in order to optimize the adaptation for a mix of all of the acoustic sources. While a switch  108  is illustrated in  FIG.  1   , other signal selection mechanisms are contemplated, such as a second mixer that could select the audio signal from a particular microphone  102  by attenuating some or all of the audio signals from the other microphones  102 . 
     Each of the audio signals from the microphones  102  can be selected by the switch  108  and processed in turn, such that a background filter  202  and a hidden filter  204  (in the acoustic echo canceller  112 ) work on one of the audio signals at a time. The switch  108  may enable adaptation on each of the audio signals from the microphones  102  within a particular duration so that the communication system  100  may properly perform echo cancellation regardless of the type of mixer  106 , the current state of the mixer  106 , or if the mixer  106  is undergoing a change in state. At step  324 , the background filter  202  and the hidden filter  204  in the acoustic echo canceller  112  may run on the selected audio signal. Step  324  is described below in more detail with respect to  FIG.  4   . 
       FIG.  4    describes further details of an embodiment of step  324  for running a background filter  202  and a hidden filter  204  in the acoustic echo canceller  112 . The background filter  202  may be a finite impulse response filter that runs a normalized least-mean squares algorithm on the selected audio signal, such as at step  402 , and may generate an estimate ĥ m [n] of the impulse response of a sample n for a microphone m in the environment. The background filter  202  may also measure a background error power of the selected audio signal, such as at step  404 . The background filter  202  may have tap coefficients h that are used to scale a finite series of delay taps. A background error e[n] of the selected audio signal may be measured by the background filter  202  according to the equation: 
     
       
      
       e[n]=d[n]−ĥ 
       † 
       [n]x[n] 
      
     
     where d[n] is the audio signal, x[n] is a vector of samples from a remote audio signal, and † denotes a conjugate transpose operation. The background error power may be measured based on the background error e[n], such as by using a time average of the magnitude of the squared background error. 
     The hidden filter  204  may be a finite impulse response filter that is run at step  406 , on a remote audio signal and a previous unweighted estimate of the echo-path impulse response made by the background filter  202 . The unweighted previous estimate corresponds to an unweighted portion of the selected audio signal within a mix filter  208  (described below). The hidden filter  204  may measure a hidden error of the selected audio signal, such as at step  408 , by subtracting the remote audio signal from the selected audio signal. A hidden error power may be measured based on the hidden error, such as by using a time average of the magnitude of the squared hidden error. The hidden filter  204  may have tap coefficients h that are used to scale a finite series of delay taps. 
     The background error power measured at step  404  and the hidden error power measured at step  408  may be compared at step  410  by an error comparison module  206 . The error comparison module  206  may determine at step  410  whether the background error power is greater than the hidden error power. If it is determined that the background error power is greater than the hidden error power at step  410 , then the process  324  may continue to step  412 . At step  412 , the tap coefficients of the background filter  202  may be selected and stored in a memory  110 . At step  414 , the stored tap coefficients from step  412  may be copied from the memory  110  and used to replace the tap coefficients of the hidden filter  204 . The stored tap coefficients from step  412  may also be copied at step  414  from the memory  110  and used to update the tap coefficients of the mix filter  208 , as described in more detail below. 
     Following step  414 , the process  324  may continue to step  416 . In addition, if it is determined at step  410  that the background error power is not greater than the hidden error power, then the process  324  may continue to step  416 . At step  416 , it may be determined whether a channel scaling factor a of the mixer  106  has changed. The channel scaling factor of the mixer  106  may change automatically or manually (e.g., by a user adjustment). If the channel scaling factor of the mixer  106  has changed at step  416 , then the process  324  may continue to step  418 . At step  418 , the tap weights of the mix filter  208  may be updated corresponding to the changed channel scaling factor, such as by adding a difference in weight multiplied by a channel impulse response estimate, as described in more detail below. 
     Following step  418 , the process  324  may continue to step  420 . In addition, if it is determined that the channel scaling of the mixer  106  has not changed at step  416 , then the process  324  may continue to step  420 . At step  420 , the tap coefficients of the background filter  202  may be updated, according to the equation: 
     
       
         
           
             
               
                 h 
                 ^ 
               
               [ 
               
                 n 
                 + 
                 1 
               
               ] 
             
             = 
             
               
                 
                   h 
                   ^ 
                 
                 [ 
                 n 
                 ] 
               
               + 
               
                 
                   α 
                   
                     
                        
                       
                         x 
                         [ 
                         n 
                         ] 
                       
                        
                     
                     2 
                   
                 
                 ⁢ 
                 
                   
                     e 
                     * 
                   
                   [ 
                   n 
                   ] 
                 
                 ⁢ 
                 
                   x 
                   [ 
                   n 
                   ] 
                 
               
             
           
         
       
     
     where α is a step-size parameter, * denotes a complex conjugation operation, and ∥·∥ denotes a    2  norm. The process  324  may then return to the process  300  and in particular, to step  308 , as described below. 
     Returning to the process  300  of  FIG.  3   , while the audio signals are received from the microphones  102  and processed in steps  318 - 328  of the process  300  and steps  402 - 420  of the process  324 , a remote audio signal may be received from a remote location, i.e., a far end, such as step  302 . The remote audio signal may be output on the loudspeaker  104  in the environment, such as at step  304 . At step  306 , the remote audio signal may also be converted into the frequency domain using a fast Fourier transform module  118 . At this point, it can be seen that the acoustic echo canceller  112  may receive the mixed audio signal from the mixer  106 , the selected audio signal from the switch  108 , and the remote audio signal from the remote location (far end). Each of the mixed audio signal from the mixer  106 , the selected audio signal from the switch  108 , and the remote audio signal may have been converted into the frequency domain, as previously described, by the respective FFT modules  114 ,  116 ,  118 . Accordingly, the acoustic echo canceller  112  may operate in the frequency domain so that the acoustic echo cancellation is performed faster and with high quality. 
     The acoustic echo canceller  112  may run a mix filter  208  at step  308 . The mix filter  208  may be a weighted sum ĥ mix [n] of the finite impulse responses of all the audio signals of the microphones  102 , such that: 
     
       
         
           
             
               
                 
                   h 
                   ^ 
                 
                 
                   m 
                   ⁢ 
                   i 
                   ⁢ 
                   x 
                 
               
               [ 
               n 
               ] 
             
             = 
             
               
                 ∑ 
                 
                   m 
                   = 
                   0 
                 
                 
                   M 
                   - 
                   1 
                 
               
               
                 
                   a 
                   m 
                 
                 ⁢ 
                 
                   
                     
                       h 
                       ^ 
                     
                     m 
                   
                   [ 
                   n 
                   ] 
                 
               
             
           
         
       
     
     where α m  is the channel scaling (weight or gain) of a particular microphone  102 . The mix filter  208  processes the remote audio signal received from the far end and generates a filtered remote audio signal that is an estimate of the echo signal generated at the output of the mixer. In particular, the mix filter models the coupling between the echo paths detected by the microphones  102  and the mixer  106 . 
     As described previously, the tap coefficients of the mix filter  208  may be updated by the tap coefficients of the background filter at step  414  of the process  324 , if the background error power is greater than the hidden error power at step  410 . When this occurs, the weighted sum ĥ mix [n+1] for the next sample n+1 may be given by: 
     
       
         
           
             
               
                 
                   h 
                   ^ 
                 
                 
                   m 
                   ⁢ 
                   i 
                   ⁢ 
                   x 
                 
               
               [ 
               
                 n 
                 + 
                 1 
               
               ] 
             
             = 
             
               
                 
                   ∑ 
                   
                     m 
                     ≠ 
                     
                       m 
                       ′ 
                     
                   
                   
                     M 
                     - 
                     1 
                   
                 
                 
                   
                     a 
                     m 
                   
                   ⁢ 
                   
                     
                       
                         h 
                         ^ 
                       
                       m 
                     
                     [ 
                     n 
                     ] 
                   
                 
               
               + 
               
                 
                   a 
                   
                     m 
                     ′ 
                   
                 
                 ⁢ 
                 
                   
                     
                       h 
                       ^ 
                     
                     
                       m 
                       ′ 
                     
                   
                   [ 
                   
                     n 
                     + 
                     1 
                   
                   ] 
                 
               
             
           
         
       
     
     where m′ is the selected audio signal of a particular microphone  102 . 
     As also described previously, the tap weights of the mix filter  208  may be updated at step  418  of the process  324 , if the channel scaling factor of the mixer  106  has changed at step  416 . When this occurs, the update may be performed by adding the difference in weight multiplied by the channel impulse response estimate ĥ m′ . In particular, the weighted sum ĥ mix [n+1] for the next sample n+1 may be given by: 
     
       
         
           
             
               
                 
                   h 
                   ^ 
                 
                 
                   m 
                   ⁢ 
                   i 
                   ⁢ 
                   x 
                 
               
               [ 
               
                 n 
                 + 
                 1 
               
               ] 
             
             = 
             
               
                 
                   ∑ 
                   
                     m 
                     ≠ 
                     
                       m 
                       ′ 
                     
                   
                   
                     M 
                     - 
                     1 
                   
                 
                 
                   
                     a 
                     m 
                   
                   ⁢ 
                   
                     
                       
                         h 
                         ^ 
                       
                       m 
                     
                     [ 
                     n 
                     ] 
                   
                 
               
               + 
               
                 
                   ( 
                   
                     
                       
                         a 
                         
                           m 
                           ′ 
                         
                       
                       [ 
                       
                         n 
                         + 
                         1 
                       
                       ] 
                     
                     - 
                     
                       
                         a 
                         
                           m 
                           ′ 
                         
                       
                       [ 
                       n 
                       ] 
                     
                   
                   ) 
                 
                 ⁢ 
                 
                   
                     
                       h 
                       ^ 
                     
                     
                       m 
                       ′ 
                     
                   
                   [ 
                   n 
                   ] 
                 
               
             
           
         
       
     
     After the mix filter  208  generates the filtered remote audio signal at step  308 , the process  300  may continue to step  310 . At step  310 , the echo-cancelled mixed audio signal may be generated by the acoustic echo canceller  112 . In particular, the filtered remote audio signal generated by the mix filter  208  may be subtracted from the mixed audio signal from the mixer  106 , as denoted by the summing point  214  shown in  FIG.  2   . The echo-cancelled mixed audio signal may be processed by a non-linear processor at step  312 , depending on the coherence of the filtered remote audio signal from the mix filter  208  and the estimated residual echo power of the echo-cancelled mixed audio signal output from the summing point  214 . Details of step  312  are described below with respect to  FIG.  5   . 
       FIG.  5    describes further details of an embodiment of step  312  for running a non-linear processor  212  in the acoustic echo canceller  112  to generate an echo-suppressed mixed audio signal. In particular, after the echo-cancelled mixed audio signal is generated at step  310 , it can be determined whether to run the non-linear processor  212  to further suppress any echo and generate comfort noise (e.g., synthetic background noise), as necessary. The non-linear processor  212  may run, for example, in situations when there is only speech and sound from the remote location (far end) and when the background filter  202  and the hidden filter  204  have not yet converged. 
     At step  502 , the output coherence of the filtered remote audio signal from the mix filter  208  may be measured by mix estimators  210 . The output coherence is a measure of the relationship between the frequency content of the filtered remote audio signal and the audio signals from the microphones  102 . The mix estimators  210  may measure the coherence from the output of the mixer  106  prior to echo cancellation at the summing point  214  and after echo cancellation at the summing point  214 . If the coherence is high, then the signals may be deemed to be related in the frequency domain. The residual echo power of the echo-cancelled mixed audio signal output from the summing point  214  may be estimated at step  504  by the mix estimators  210 . The non-linear processor  212  may process the echo-cancelled mixed audio signal at step  508  to generate an echo-suppressed mixed audio signal if (1) the output coherence is greater than a predetermined threshold (e.g., signifying that there is only an echo signal present in the microphones  102 ); or (2) the residual echo power is greater than half of the power of the mixed audio signal from the mixer  106 . Following step  508 , the process  312  may continue to step  314  of the process  300 . However, if neither of these conditions is satisfied, then the process  312  may continue from step  506  to step  314  of the process  300 . 
     Returning to  FIG.  3   , at step  314 , the (1) echo-cancelled mixed audio signal generated at step  310  (if the non-linear processor  212  was not executed at step  312 ) or (2) the echo-suppressed mixed audio signal generated at step  508  (if the non-linear processor  212  was executed at step  312 ) may be converted to the time domain. The resulting echo-cancelled or echo-suppressed audio signal may be transmitted to the remote location (far end) at step  316 . The process  300  may return to step  322  to select and convey another of the audio signals from the microphones  102  for processing at steps  324  and  308 - 316 , as described previously. In this way, information from the audio signal from each of the plurality of microphones  102  may be utilized when generating the echo-cancelled or echo-suppressed audio signal. 
     Any process descriptions or blocks in figures should be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process, and alternate implementations are included within the scope of the embodiments of the invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those having ordinary skill in the art. 
     This disclosure is intended to explain how to fashion and use various embodiments in accordance with the technology rather than to limit the true, intended, and fair scope and spirit thereof. The foregoing description is not intended to be exhaustive or to be limited to the precise forms disclosed. Modifications or variations are possible in light of the above teachings. The embodiment(s) were chosen and described to provide the best illustration of the principle of the described technology and its practical application, and to enable one of ordinary skill in the art to utilize the technology in various embodiments and with various modifications as are suited to the particular use contemplated. All such modifications and variations are within the scope of the embodiments as determined by the appended claims, as may be amended during the pendency of this application for patent, and all equivalents thereof, when interpreted in accordance with the breadth to which they are fairly, legally and equitably entitled.