Provided is a method and computer program product for producing an enhanced audio signal for an output device from audio signals received by 2 or more microphones in close proximity to each other. For example, one embodiment of the present invention comprises the steps of receiving a first input audio signal from the first microphone, digitizing the first input audio signal to produce a first digitized audio input signal, receiving a second input audio input signal from the second microphone, digitizing the second input audio input signal to produce a second digitized audio input signal, using the first digitized audio input signal as a reference signal to an adaptive prediction filter, using the second digitized audio input signal as input to said adaptive prediction filter and finally adding a prediction result signal from the adaptive prediction filter to the first digitized audio input signal to produce the enhanced audio signal. In other embodiments, any number of microphones can be used, and in all embodiments there is no requirement to detect or locate the source or direction of arrival of the input audio signals.

FIELD OF THE INVENTION

The present invention is related to audio signal processing and more specifically to system and method of adaptive multi-microphone beamforming to enhance speech/audio far-field pickup.

SUMMARY OF THE INVENTION

It is quite natural for human beings to use their own voices as an effective means of communication. Indeed children start to use their voices long before they develop other communication skills, such as reading or writing. The broad adoption of mobile devices is another example that demonstrates the proliferation and importance of voice enabled communications throughout the modern world.

Telephony applications have progressed through a long evolution from wired devices to wireless mobile units, and from operator assisted calls, to fully automated end-to-end user calls across the globe. Increasingly, users appreciate the flexibility and freedom afforded by modern telecommunication devices and services. Another step to further this evolution is to completely liberate users' hands from the operation of their mobile communication devices. The use of hands-free modes for phone calls is not only convenient in many situations, but is often required and frequently enforced by the law, for example, as is the case when using mobile phones while driving.

Another rapidly growing technological area that is currently gaining enormous momentum is the vast array of smart or connected devices (also referred to as the Internet of Things or “IoT”, that can be installed almost anywhere including residential homes, office buildings, public spaces, transportation vehicles, and even implanted in human beings. These devices generally include sensors, actuators and the like, and are connected to the Web or other cloud-based services and/or to each other in some fashion. Some residential examples include audio/video equipment, thermostats, appliances, and lighting. IoT devices can be designed and manufactured to respond to voice commands in order to provide increased flexibility and freedom to users.

Major problems that must be overcome when implementing hands-free communications or voice controlled devices are inefficiencies due to the inherent nature of sound waves that degrade when propagating through the air. Specifically, because the strength or intensity of sound waves is inversely proportional to the square of the distance from the source, it becomes increasingly difficult to achieve acceptable results the further away a user is from the input device or microphone.

When a user holds a phone close to his or her mouth, it is not difficult to achieve a sufficiently high signal to noise ratio (SNR), and thus produce acceptable results for voice recognition or noise reduction applications, even in a noisy environment. For example, the volume level of normal speech (as measured close to the human mouth) is approximately 85 dB(A). A background noise level of 70 dB(A) is generally considered a noisy environment, such as a crowded restaurant or bar. This example leads to a SNR of 15 dB, which is large enough to achieve acceptable results for most applications. Examples of such applications include voice recognition accuracy for a voice-controlled device, or a typical noise suppression module for a high quality telephony call.

However, if the user moves only three meters away from the microphone, and still speaks at the same volume, the strength of his or her voice (as measured at the microphone) would now be reduced to around 55 dB(A). Thus, even with a much lower noise level of 50 dB(A), (a level in which most users would describe as quiet), the resulting SNR is only 5 dB, which makes it extremely difficult for applications to produce acceptable results.

In order to mitigate this issue, it is a common industry practice to use multiple microphones, or a microphone array, combined with advanced techniques such as beamforming, to enhance the SNR to produce better results. Traditional beamforming techniques use a “Delay-Sum” approach, which analyze a talker's voice arrival time at each microphone, delays early-arrived speech signals, aligns each of the signals with the latest arrival speech signal, and finally sums up all of the speech signals to create a maximum correlated output speech signal. While this approach is simple and effective, it requires accurate tracking of the user's location relative to the microphones or microphone array to determine the angle of arrival of the speech signals. Errors in determining the user's location relative to the microphones will quickly diminish the beamforming gains, resulting in rapid speech level variations.

Persons skilled in the art would appreciate that, while techniques exist for determining a user's location fairly accurately using multiple microphone inputs, it is nonetheless a very challenging task when ambient noises are present, especially at low SNR conditions. Also, when a user moves around rapidly, such as when walking back and forth inside a home for example, timely and accurate detection of the user's location represents another challenge.

Another difficulty with traditional approaches is that due to design constraints and the like, multiple microphones are not necessarily aligned in a straight line. This makes the estimation of the talker's location even more difficult to calculate and therefore further limits the applicability of traditional methods.

Thus, in order to resolve the limitations of conventional methods and systems and to improve user experience, the present invention provides an adaptive multi-microphone beamforming technique that does not require calculations for the user's location or the direction of arrival of audio signals. In addition, the present invention provides an additional benefit of allowing arbitrary placement of microphones in products without impacting the beamforming performance.

DETAILED DESCRIPTION

The present invention may be described herein in terms of functional block components and various processing steps. It should be appreciated that such functional blocks may be realized by any number of hardware components or software elements configured to perform the specified functions. For example, the present invention may employ various integrated circuit components, e.g., memory elements, digital signal processing elements, logic elements, look-up tables, and the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices. In addition, those skilled in the art will appreciate that the present invention may be practiced in conjunction with any number of data and voice transmission protocols, and that the system described herein is merely one exemplary application for the invention.

FIG. 1illustrates the principle of a beamforming technique that enhances a speech signal from a talker101. In this example, the talker101is speaking directly in front of the two microphones111/112, such that he is 0° or directly perpendicular to the two microphones. In this example, the sound wave front102arrives at the two microphones111/112at exactly the same time. This causes the resulting electronic speech signals121/122to be the same, or at least close enough to be considered the same. Adding such two similar (or highly correlated) signals together results in an output signal130with 2 times (2×) amplification of the signal sample values in the time domain, obtaining therefore an energy increase of 4 time (4×), which corresponds to a gain of 6 dB.

Referring now toFIG. 2, the talker201is now at an angle of approximately 45° to the two microphones,211and212. The sound wave front252arrives at the two microphones at different times, resulting in an early arrival speech signal261, and a late arrival speech signal262. With a delay module260, the early arrival speech signal261can be delayed by a certain amount265in order to align with the late arrival speech signal262. Next, adding the delay adjusted signals together results in an enhanced signal280with the same 2× amplification gain or a 6 dB energy gain. Note that the example inFIG. 1can be considered a special case of the more generic case ofFIG. 2, where a delay of zero is applied between the two speech signals121and122.

Referring now toFIG. 3, assuming the source of noise302is located at a certain angle, and the noise picked up by the microphones have a certain difference in time as shown in321and322. Since the talker301is directly in front or perpendicular to the two microphones, no delay adjustment is performed for the identified speech signal from the talker301, and therefore the two noise signals remain uncorrelated.

Thus, because the two noise signals321and322remain uncorrelated, their sum does not create a 2× sample value effect in the output signal330, as does the voice signal from the talker301. Therefore, the two uncorrelated noise signals added together is simply a noise energy increase of 2, and a noise level increase of 3 d B.

FIG. 4shows a case where the source of noise402is directly in front of the two microphones and the talker401is at 45° from the microphones. In this example, the noise arrives at the two microphones at approximately the same time. However, due to the fact that the talker is at a 45° angle, the delay module460applies a certain delay465to the signal461in order to correlate the voice signal from talker's voice401. This results in a 6 dB energy gain in the voice signal. However, the delay460also has the beneficial effect of “de-correlating” the two noise signals402. Adding the two uncorrelated noise signals together does not achieve the 2× sample value effect as does the voice signal, resulting in a simple increase in the energy of the noise signal by 3 dB in the uncorrelated noise output signal480.

In an ideal case, with a speech signal energy level increase of 6 dB, and a noise level increase of 3 dB, the maximum gain of a two-microphone based delay-sum beamforming approach is 3 dB SNR. However, as previously mentioned, this traditional method requires extremely accurate knowledge regarding the location of the talker in order to calculate the exact time delay required to create a perfectly correlated speech signal. As would be appreciated by persons skilled in the art, it is often very difficult to accurately and precisely detect a talker's location. When such location information is not accurate or unavailable, the performance of such traditional beamforming systems and methods are dramatically reduced as is often the case when a talker is not stationary.

Another difficulty with traditional delay-sum beamforming is that, due to design constraints, such as required product size, and other form factor considerations, multiple microphones are not necessarily aligned in a straight line. This makes the estimation of the talker's location even more difficult to calculate and therefore further limits the applicability of traditional methods. These types of problems are illustrated inFIG. 5.

As shown by the examples depicted inFIG. 5, it is often the case that portable products containing multiple microphones are not necessarily configured in a straight line or in any predictable manner as such products are subject to constant orientation changes by the user. Examples of such products can be seen in504,506and508. In such cases, it is very difficult to determine the direction of arrival (DOA) of the talker's voice501and the noise sources502. Thus traditional methods of delay-sum beamforming as described above, are extremely difficult to implement under these conditions, and are subject to rapid deterioration in performance and quality by miscalculated DOA estimates

The present invention alleviates the problems found in traditional microphone beamforming methods and systems by not requiring any determination of the direction of arrival of the audio sources. Further, because the orientation of the device and the placement of the microphones are irrelevant, the present invention works equally well under all conditions and may be implemented with less complexity than traditional methods.

FIG. 6shows an exemplary embodiment of the present invention using an adaptive prediction filter module670. In a preferred embodiment of the present invention a normalized least mean square (NLMS) based adaptive filter is used, however, any type of adaptive prediction filter may be also be used without departing from the scope and breadth of the present invention. Examples of such adaptive filters can be found in the following article, which is incorporated herein by reference as if set forth in full: “Comparison between Adaptive filter Algorithms (LMS, NMLS and RLS)” by Jyoti Dhiman, Shadab Ahmad, and Kuldeep Gulia, published by the Internal Journal of Science, Engineering and Technology Research (IJSETR), Volume 2, Issue 5, May 2013, ISSN:2278-7998.

In general, as stated by the above-referenced article, an adaptive filter is a filter that self adjusts its transfer function according to an optimizing algorithm. It adapts the performance based on the input signal. Such filters incorporate algorithms that allow the filter coefficients to adapt to the signal statics. Adaptive techniques use algorithms, which enable the adaptive filter to adjust its parameters to produce an output that matches the output of an unknown system. This algorithm employs an individual convergence factor that is updated for each adaptive filter coefficient at each iteration.

As shown inFIG. 6, the present invention uses a signal from a first microphone as a reference signal, which is used by the adaptive prediction filter to minimize, in an iterative fashion, a prediction error signal692, which represents the difference between the two input speech signals681and682. Over time, as the adaptive filter670learns the transfer function of the reference signal691(also the first input speech signal681), the prediction error signal693approaches zero, and the prediction result signal692approaches the reference signal691. This results in an alignment or a convergence of the second input speech signal that is highly correlated with the first input speech signal681. The prediction signal result is added together with the reference signal to produce the desired energy gain. In other words, the result of the adaptive prediction filter is an audio signal from the second microphone (prediction result), that is now closely correlated or aligned with the audio signal from the first microphone (the reference signal). This is performed iteratively and automatically and does not require detecting the direction of arrival (DOA) of the audio signals as do traditional methods.

Referring back now toFIG. 6, for a more detailed description, the audio signal from the first microphone input602is digitized by the analog to digital convertor (A/D)610to become the first input speech signals681. This first input speech signal681is used as a reference single691for the adaptive prediction module670.

The audio signal from the second microphone input604is digitized by the A/D converter608to become the second input speech signal682, and is the input to the adaptive prediction module670. The prediction result692is subtracted from the reference signal691to obtain the prediction error693. This prediction error693is then used to drive the adaptive prediction module670, which acts to minimize the prediction error as an objective for the adaptation. The sum of the first input speech signal681and the prediction result signal692forms the desired output signal680, which is output to an output device such as a speaker, headphones or the like. Adding such highly correlated signals together results in an output signal692with an approximate amplification of 2×.

Please note that in the examples used herein, speech signals are used as examples (such as input speech signals681and682) of the desired type of signals that are enhanced by an embodiment of the present invention. However, in other embodiments, any type of audio signal can be enhanced by the improved techniques described herein, such as music signals and the like, without departing from the scope and breadth of the present invention.

FIG. 7is an alternative embodiment of the present invention using a symmetric arrangement that uses both the first and the second microphone inputs as reference signals for multiple adaptive prediction modules. This embodiment is used to minimize the potential impact to the resulting signals, when the microphone inputs are not consistent, for example, when one of the microphone inputs represents the original audio signal better than the other microphone input. In this fashion, because both microphone inputs722and721are used as reference signals to the adaptive prediction modules772and771in the first stage, any impact caused by inconsistent inputs are minimized.

InFIG. 7, a symmetric arrangement is illustrated for the first level of prediction according to one embodiment of the present invention. The digitized first input speech signal721is used as reference signal for the second adaptive prediction module772. The second adaptive prediction module772takes the digitized second input speech signal722as input, and produces an optimized prediction result732, which acts to minimize the prediction error between the first input speech signal721and the prediction result732. The sum of732and721forms the first enhanced signal742.

Similarly, the digitized second input speech signal722is used as reference for the first adaptive prediction module771, which takes the digitized first input speech signal721as input to produce an optimized prediction result signal731that minimizes the prediction error between the reference signal722and the prediction result signal731. The sum of731and722forms the second enhanced signal741.

The second enhanced signal741is used as the reference signal for a second level of prediction according to an example embodiment of the present invention. The first enhanced signal742is input to the third adaptive prediction module773that produces an optimized prediction result733by minimizing the prediction error between second enhanced signal741and the prediction result733. Finally, the sum of741and733is the desired output signal798, with is subsequently output to an output audio device.

It should be noted that in this example embodiment, it is assumed that there is a high level of consistency between the first input signal722and the second input signal721. As such, in this example, the second enhanced signal741is selected to act as the reference signal to the third adaptive prediction module773. Indeed, in most cases, were the microphones that comprise the microphone array are closely spaced relative to each other, this consistency is expected. However, in order to minimize any negative effects from inconsistent inputs and to maximize the performance of the present invention, another stage may be added to the embodiment shown inFIG. 7. This alternative embodiment is shown with reference toFIG. 7B.

As shown inFIG. 7B, the first step is to determine which of the audio signals742or741is the “better” or “stronger” signal. There are many ways to make this determination including finding the signal with the greatest energy, lowest noise component, or better sensitivity, as is the case for example, when a microphone's input is blocked and covered by dust or other objects. In addition, the better or stronger signal can also be determined based on longer term measurements that are well known in the art. Indeed any method for determining a better or stronger signal as a best candidate for a reference signal can be used without departing from the scope and breadth of the present invention. Note that in the examples used herein, such signals are referred to as either “stronger” or “better.” Similarly, the term “weaker” is used to describe signals other than those that have been determined to be stronger or better in accordance with the principles of the present invention as disclosed herein.

In this example, the better or stronger single is detected in the first step702, for example, the signal with the highest energy, or other criteria as discussed above is identified in the first step702. Once this determination is made, the better signal is used as the reference signal and the other signal or weaker signal, is used as the input signal to the third adaptive prediction module773. In particular, in step702, if it is determined that signal742is better than741, then as shown in step704, the signal742is used as the reference signal and the signal741is used as the input signal to the adaptive prediction module773. Similarly, if the Signal741is better than (or equal to)742, then as shown in step703, the signal741is the reference signal and the signal742is the input signal to the adaptive prediction module773. In practice, if the signals are equivalent and neither one is better or stronger than the other, than it makes no difference which signal is used as the reference signal and which signal is used as the input signal.

In yet another embodiment of the present invention, this technique ofFIG. 7Bcan be used in the embodiment discussed above with reference toFIG. 6. That is, inFIG. 6, rather than assigning the first input signal681as the reference signal and the second input signal682as the input signal to the adaptive prediction module670, the technique described above inFIG. 7Bis used to determine which of the signals681or682is the better or stronger signal. Once that determination is made, the better or stronger signal is used as the reference signal and the other signal is used as the input signal to the adaptive prediction module670.

FIG. 8illustrates another exemplary embodiment of the present invention using multiple microphones (N). In this example embodiment, the number of microphones can be any number greater than 2. The digitized first microphone input is the first input speech signal831. The first input speech signal831is used as the reference signal851for each of the N−1 adaptive prediction modules, such as the adaptive prediction modules878and879shown inFIG. 8.

The digitized second microphone input is the input speech signal872that is the input to the second adaptive prediction module878. Adaptive prediction module878functions to minimize the prediction error signal894between the reference signal851and the prediction result882. As shown and indicated by the ellipses inFIG. 8, the proceeding modules or steps are repeated for each of the remaining input speech signals until the Nthinput speech signal873is reached. That is, the digitized Nthmicrophone input is the Nthinput speech signal873that is the input to the Nthadaptive prediction module879. Adaptive prediction module879acts to minimize the prediction error signal895between the reference signal851and the prediction result signal883.

Finally, the sum of the first input speech signal831(also the reference signal), and each of the prediction result signals associated with each of the N−1 adaptive prediction filter modules, (such as those shown in882and883), form the desired output signal898, which is output to an output device.

In yet another embodiment of the present invention, the technique ofFIG. 7Bcan be used in the embodiment discussed above with reference toFIG. 8. That is, inFIG. 8, rather than assigning the first input signal831as the reference signal851to each of the N−1 adaptive prediction modules (such as shown in878and879), the technique described above inFIG. 7Bis used to determine which of the input speech signals831,872, . . .873is the stronger or better signal. Once that determination is made, the stronger signal is used as the reference signal for each of the N−1 adaptive prediction modules, and the other signals or weaker signals are used as inputs to their respective adaptive prediction modules.

The present invention may be implemented using hardware, software or a combination thereof and may be implemented in a computer system or other processing system. Computers and other processing systems come in many forms, including wireless handsets, portable music players, infotainment devices, tablets, laptop computers, desktop computers and the like. In fact, in one embodiment, the invention is directed toward a computer system capable of carrying out the functionality described herein. An example computer system901is shown inFIG. 9. The computer system901includes one or more processors, such as processor904. The processor904is connected to a communications bus902. Various software embodiments are described in terms of this example computer system. After reading this description, it will become apparent to a person skilled in the relevant art how to implement the invention using other computer systems and/or computer architectures.

Computer system901also includes a main memory906, preferably random access memory (RAM), and can also include a secondary memory908. The secondary memory908can include, for example, a hard disk drive910and/or a removable storage drive912, representing a magnetic disc or tape drive, an optical disk drive, etc. The removable storage drive912reads from and/or writes to a removable storage unit914in a well-known manner. Removable storage unit914, represent magnetic or optical media, such as disks or tapes, etc., which is read by and written to by removable storage drive912. As will be appreciated, the removable storage unit914includes a computer usable storage medium having stored therein computer software and/or data.

In alternative embodiments, secondary memory908may include other similar means for allowing computer programs or other instructions to be loaded into computer system901. Such means can include, for example, a removable storage unit922and an interface920. Examples of such can include a USB flash disc and interface, a program cartridge and cartridge interface (such as that found in video game devices), other types of removable memory chips and associated socket, such as SD memory and the like, and other removable storage units922and interfaces920which allow software and data to be transferred from the removable storage unit922to computer system901.

Computer system901can also include a communications interface924. Communications interface924allows software and data to be transferred between computer system901and external devices. Examples of communications interface924can include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, etc. Software and data transferred via communications interface924are in the form of signals which can be electronic, electromagnetic, optical or other signals capable of being received by communications interface924. These signals926are provided to communications interface via a channel928. This channel928carries signals926and can be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, such as WiFi or cellular, and other communications channels.

In this document, the terms “computer program medium” and “computer usable medium” are used to generally refer to media such as removable storage device912, a hard disk installed in hard disk drive910, and signals926. These computer program products are means for providing software or code to computer system901.

Computer programs (also called computer control logic or code) are stored in main memory and/or secondary memory908. Computer programs can also be received via communications interface924. Such computer programs, when executed, enable the computer system901to perform the features of the present invention as discussed herein. In particular, the computer programs, when executed, enable the processor904to perform the features of the present invention. Accordingly, such computer programs represent controllers of the computer system901.

In an embodiment where the invention is implemented using software, the software may be stored in a computer program product and loaded into computer system901using removable storage drive912, hard drive910or communications interface924. The control logic (software), when executed by the processor904, causes the processor904to perform the functions of the invention as described herein.