Audio source localization system and method

Systems and methods are described that perform audio source localization in a manner that provides increased robustness and responsiveness in the presence of acoustic echo. The systems and methods calculate a difference between a signal level associated with one or more of the audio signals generated by a microphone array and an estimated level of acoustic echo associated with one or more of the audio signals. This information is then used to determine whether and/or how to perform audio source localization. For example, a controller may use the difference to determine whether or not to freeze an audio source localization module that operates on the audio signals. As another example, the audio source localization module may incorporate the difference (or the estimated level of acoustic echo used to calculate the difference) into the logic that is used to determine the location of a desired audio source.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to systems that automatically determine the location of one or more desired audio sources based on audio input received via an array of microphones.

As used herein, the term audio source localization refers to a technique for automatically determining the location of at least one desired audio source, such as a talker, in a room or other area.FIG. 1is a block diagram of an example system100that performs audio source localization. System100may represent, for example and without limitation, a speakerphone, a teleconferencing system, a video gaming system, or other system capable of both capturing and playing back audio signals.

As shown inFIG. 1, system100includes an output audio processing module102that processes at least one audio signal for playback via loudspeakers104. The audio signal processed by audio output processing module102may be received from a remote audio source such as a far-end talker in a speakerphone or teleconferencing scenario. Additionally or alternatively, the audio signal processed by output audio processing module102may be generated by system100itself or some other source connected locally thereto. For example, in a video gaming scenario, the audio signal processed by output audio processing module102may represent music and/or sound effects associated with a video game being executed by system100.

As further shown inFIG. 1, system100further includes an array of microphones106that converts sound waves produced by local audio sources into audio signals. These audio signals are then processed by an audio source localization module108. Depending upon the implementation, the audio signals generated by microphone array106may first be processed by other logic (e.g., acoustic echo cancellers (AECs)) prior to being received by audio source localization module108.

Audio source localization module108periodically processes the audio signals generated by microphone array106to estimate a current location of a desired audio source114. Desired audio source114may represent, for example, a near-end talker in a speakerphone or teleconferencing scenario or a video game player in a video gaming scenario. The estimated current location of desired audio source114as determined by audio source localization module108may be defined, for example, in terms of an estimated current direction of arrival of sound waves emanating from desired audio source114.

System100also includes a steerable beamformer110that is configured to process the audio signals generated by microphone array106to produce a single audio signal. In producing the audio signal, steerable beamformer110performs spatial filtering based on the estimated current location of desired audio source114such that signal components attributable to sound waves emanating from locations other than the estimated current location of desired audio source114are attenuated relative to signal components attributable to sound waves emanating from the estimated current location of desired audio source114. This tends to have the beneficial effect of attenuating undesired audio sources relative to desired audio source114, thereby improving the overall quality and intelligibility of the output audio signal. In a speakerphone or teleconferencing scenario, the audio signal produced by steerable beamformer110is transmitted to a far-end listener.

The information produced by audio source localization module108may also be useful for applications other than steering a beamformer used for acoustic transmission. For example, the information produced by audio source localization module108may be used in a video gaming system to integrate the estimated current location of a player within a room into the context of a game (e.g., by controlling the placement of an avatar that represents the player within a scene rendered by a video game based on the estimated current location of the player) or to perform proper sound localization in surround sound gaming applications. Various other beneficial applications of audio source localization also exist. These applications are generally represented in system100by the element labeled “other applications” and marked with reference numeral112.

One problem for system100and certain other systems that perform audio source localization is the presence of acoustic echo116. Acoustic echo116is generated when system100plays back audio signals via loudspeakers104, an echo of which is picked up by microphone array106. In a speakerphone or teleconferencing system, such echo may be attributable to speech signals representing the voices of one or more far end talkers that are played back by the system. Such echo is typically intermittent. In a video gaming system, the echo may be attributable to music, sound effects, and/or other audio content produced by a game. This type of echo is typically more continuous in nature.

The presence of acoustic echo can cause audio source localization module108to perform poorly, since the module may not be able to adequately distinguish between desired audio source114whose location is to be determined and the echo. This may cause audio source localization module108to incorrectly estimate the location of desired audio source114.

There are some known techniques that may be used to deal with this issue. For example, acoustic echo cancellation may be performed on each of the microphone input signals using transversal filters. However, there are problems with this approach. For example, transversal filters require time to converge to an accurate acoustic impulse response and during this convergence time, echo cancellation performance may be poor. Furthermore, it is likely that the acoustic echo can never be canceled completely because of factors such as background noise/interference118and/or non-linearities associated with system loudspeakers or with other audio processing logic that is located outside of system100. For example, where system100is a video gaming system that is part of a home theater installation, audio output produced by the system may be processed by audio processing logic located in a receiver and/or in external speakers.

These problems may render the acoustic echo cancellation insufficiently robust. As a result, residual echo may be delivered to audio source localization module108, impairing its performance.

Another approach known in the art is to “freeze” the operation of audio source localization module108whenever audio content is being played back by system100. This ensures that the estimated location of desired audio source114will not be changed based on acoustic echo. However, this approach negatively impacts the responsiveness of audio source localization module108, since that module cannot track the location of desired audio source114during periods when audio content is being played back by system100. Such lack of responsiveness is especially damaging in a video gaming application where the audio played back by the video gaming system may be virtually continuous.

What is needed, then, is a system for performing audio source localization in the presence of acoustic echo that addresses one or more of the aforementioned shortcomings associated with prior art solutions.

BRIEF SUMMARY OF THE INVENTION

Systems and methods are described herein that perform audio source localization in a manner that provides both increased robustness and responsiveness in the presence of acoustic echo as compared to conventional approaches. As will be described in more detail herein, system and methods in accordance with various embodiments of the present invention calculate a difference between a signal level associated with one or more of the audio signals generated by a microphone array and an estimated level of acoustic echo associated with one or more of the audio signals. The systems and methods then use this information to determine whether and/or how to perform audio source localization. For example, a controller may use the difference to determine whether or not to freeze an audio source localization module that operates on the audio signals. As another example, the audio source localization module may incorporate the difference (or the estimated level of acoustic echo used to calculate the difference) into the logic that is used to determine the location of a desired audio source.

By using the difference and/or estimated level of acoustic echo to determine whether and/or how to perform audio source localization, systems and methods in accordance with embodiments of the present invention can advantageously reduce the adverse effect of acoustic echo on the performance of audio source localization, thereby providing improved robustness. Furthermore, by using the difference and/or estimated level of acoustic echo to determine whether and/or how to perform audio source localization, systems and methods in accordance with embodiments of the present invention advantageously allow audio source localization to be performed in the presence of echo, thereby providing improved responsiveness.

DETAILED DESCRIPTION OF THE INVENTION

The following detailed description of the present invention refers to the accompanying drawings that illustrate exemplary embodiments consistent with this invention. Other embodiments are possible, and modifications may be made to the embodiments within the spirit and scope of the present invention. Therefore, the following detailed description is not meant to limit the invention. Rather, the scope of the invention is defined by the appended claims.

B. First Example System for Performing Audio Source Localization in Accordance with an Embodiment of the Present Invention

FIG. 2is a block diagram of a first example system200for performing audio source localization in accordance with an embodiment of the present invention. As shown inFIG. 2, system200includes a number of interconnected components including a microphone array202, an array of analog-to-digital (A/D) converters204, an audio source localization module206, a location-based application208, an audio source localization controller210, an output audio source212, an output audio processing module214, and one or more loudspeakers216. Each of these components will now be described.

Output audio processing module214is configured to receive an audio signal from output audio source212and to process the received audio signal for playback via loudspeaker(s)216. Among other operations, output audio processing module214may perform one or more of audio decoding, frame buffering, amplification, and digital-to-analog conversion to generate a processed audio signal that is in a form suitable for playback by loudspeaker(s)216.

Output audio source212is intended to broadly represent any component or entity that is capable of producing an audio signal for playback by system200. For example, in an embodiment in which system200is part of a speakerphone or teleconferencing system, output audio source212may comprise a receiver that is configured to receive an audio signal representative of a voice of a far-end talker over a communications network. In an embodiment in which system200is part of a video gaming system, output audio source212may comprise a video game that, when executed by the appropriate system elements, generates music and/or sound effects for playback. These examples are not intended to be limiting and persons skilled in the relevant art(s) will appreciate that output audio source212may represent other types of audio sources as well.

Each of loudspeaker(s)216comprises an electro-mechanical transducer that operates in a well-known manner to convert an analog representation of an audio signal into sound waves for perception by a user.

Microphone array202comprises two or more microphones that are mounted or otherwise arranged in a manner such that at least a portion of each microphone is exposed to sound waves emanating from audio sources proximally located to system200. Each microphone in array202comprises an acoustic-to-electric transducer that operates in a well-known manner to convert such sound waves into a corresponding analog audio signal. The analog audio signal produced by each microphone in microphone array202is provided to a corresponding A/D converter in array204. Each A/D converter in array204operates to convert an analog audio signal produced by a corresponding microphone in microphone array202into a digital audio signal comprising a series of digital audio samples prior to delivery to audio source localization module206.

Audio source localization module206is connected to array of A/D converters204and receives digital audio signals therefrom. Audio source localization module206is configured to periodically process time-aligned segments of the digital audio signals to determine a current location of a desired audio source. A variety of algorithms are known in the art for performing this function. In one example embodiment, audio source localization module206is configured to determine the current location of the desired audio source by determining a current direction of arrival (DOA) of sound waves emanating from the desired audio source. After determining the current location of the desired audio source, audio source localization module206passes this information to location-based application208.

Location-based application208is intended to broadly represent any application that is configured to perform operations based on the location information received from audio source localization module206. For example, in an embodiment in which system200comprises a speakerphone or teleconferencing system, application208may comprise a steerable beamformer that processes the audio signals generated by microphone array202to produce a single audio signal for acoustic transmission. In producing the audio signal, the steerable beamformer may perform spatial filtering based on the current location of a desired audio source, such as a desired talker, as determined by audio source localization module206. As another example, in an embodiment in which system200comprises a video teleconferencing system, location-based application208may comprise an application that uses the location information provided by audio source localization module206to control a video camera to point at and/or zoom in on a desired audio source, such as a desired talker. As a further example, in an embodiment in which system200comprises a video gaming system, location-based application208may comprise a video gaming application that uses location information provided by audio source localization module206to integrate the current location of a player into the context of a game or may comprise a surround sound application that uses location information provided by audio source localization module206to perform proper sound localization. These examples are provided by way of illustration only and are not intended to be limiting.

Depending upon the implementation, location-based application208may be proximally or remotely located with respect to the other components of system100. For example, location-based application208may be an integrated part of single device that includes the other components of system100or may be located in close proximity to the other components of system100(e.g., in the same room). Alternatively, location-based application208may be located in a different room, home, city or country than the other components of system100. In either case, a suitable wired or wireless communication link is provided between audio source localization module206and location-based application208so that location information can be passed there between.

As described in the Background Section above, the performance of audio source localization module206may be adversely impacted by acoustic echo generated by sound waves emanating from loudspeaker(s)216. To address this issue, system200includes an audio source localization controller210. Audio source localization controller210selectively enables audio source localization module206to produce updated location information when it determines that the impact of acoustic echo upon the performance of the module is likely to be acceptable and selectively disables audio source localization module206from producing updated location information when it determines that the impact of acoustic echo upon the performance of the module is likely to be unacceptable. To determine the impact of acoustic echo upon the performance of audio source localization module206, audio source localization controller includes a signal-to-echo ratio (SER) calculator222that calculates at least one SER upon which the disabling/enabling decision is premised. To calculate the at least one SER, SER calculator222uses information obtained from output audio processing module214and array of A/D converters204.

The operation of audio source localization controller210and SER calculator222in accordance with one embodiment of the present invention will now be explained with reference to flowchart300ofFIG. 3. Although the method of flowchart300will be described herein with reference to components of example system200, it is to be understood that the method is not limited to that implementation and may be performed by other components or systems entirely.

As shown inFIG. 3, the method of flowchart300begins at step302in which SER calculator222determines an estimated level of acoustic echo associated with one or more of the audio signals generated by microphone array202. In one embodiment, SER calculator222performs this function by estimating an echo return loss (ERL) associated with one or more of the audio signals generated by microphone array202and then subtracting in the log domain the estimated ERL from a level of an output audio signal that is processed by output audio processing module214for playback via loudspeaker(s)216. Various methods for determining an ERL are known in the art and thus need not be described herein. In one implementation, the level of the audio signal that is processed by output audio processing module214for playback via loudspeaker(s) is measured by output audio processing module214and passed to SER calculator222.

At step304, SER calculator222determines a signal level associated with one or more of the audio signals generated by microphone array202. The signal level may comprise, for example, the level of an audio signal generated by a designated microphone within microphone array202or an average of the levels of the audio signals generated by two or more of the microphones within microphone array202. The digital representation of the microphone signals produced by array of A/D converters204may be used to perform the necessary signal level measurements.

At step306, SER calculator222calculates a difference between the signal level determined during step304and the estimated level of acoustic echo determined during step302in the dB domain. As will be appreciated by persons skilled in the relevant art(s), this operation is the mathematical equivalent of calculating a ratio between the signal level and the estimated level of acoustic echo in the linear domain.

At step308, audio source localization controller210selectively disables or enables audio source localization module206based at least on the difference calculated during step306. This step may include, for example, selectively disabling or enabling audio source localization module206based at least on a determination of whether the difference exceeds a threshold.

Depending upon the implementation, disabling audio source localization module206may comprise, for example, preventing audio source localization module206from determining a new current location of a desired audio source or preventing audio source localization module206from providing a new current location of a desired audio source to location-based application208. In either case, the effect is to “freeze” the output of audio source localization module206such that the determined location of the desired audio source will not change. Conversely, enabling audio source localization module206may comprise, for example, enabling audio source localization module206to determine a new current location of a desired audio source or enabling audio source localization module206to provide a new current location of a desired audio source to location-based application208.

The foregoing embodiment thus uses at least one SER to determine if the proportion of acoustic echo present in the audio input being received via microphone array202is small enough such that module206can use the audio input to perform audio source localization in a reliable manner. If it is, then module206is enabled and if it is not, module206is disabled. This helps to ensure that the location information produced by audio source localization module206is reliable even when the module is operating in the presence of acoustic echo. Furthermore, in contrast to certain prior art solutions, this advantageously allows audio source localization to be performed even when an output audio signal is being played back via loudspeaker(s)216.

FIG. 4depicts a flowchart400of one particular technique for implementing the general method of flowchart300ofFIG. 3. The method of flowchart400is provided herein by way of example only and is not intended to be limiting. Persons skilled in the relevant art(s) will appreciate that other techniques may be used to implement the general method of flowchart300ofFIG. 3. Furthermore, although the method of flowchart400will also be described herein with continued reference to components of example system200, it is to be understood that the method is not limited to that implementation and may be performed by other components or systems entirely.

As shown inFIG. 4, the method of flowchart400begins at step402in which SER calculator222determines an estimated level of acoustic echo for each of a plurality of frequency sub-bands for each of the audio signals generated by microphone array202. In one embodiment, SER calculator222performs this function by estimating an ERL for each of the plurality of frequency sub-bands for each of the audio signals generated by microphone array202. Then for each audio signal, SER estimator222subtracts the estimated ERL for each frequency sub-band for that audio signal from a corresponding frequency sub-band signal level of an output audio signal that is processed by output audio processing module214for playback via loudspeaker(s)216, thereby generating an estimated level of acoustic echo for each of the plurality of frequency sub-bands for each audio signal. The subtraction is performed in the log domain.

At step404, SER calculator222determines a signal level for each of the plurality of frequency sub-bands for each of the audio signals generated by microphone array202. In one embodiment, SER calculator222performs this function by measuring the level of an audio signal generated by each microphone in each of the plurality of frequency sub-bands.

At step406, SER calculator222calculates a difference between the signal level determined in step404and the estimated level of acoustic echo determined in step402in the dB domain for each of the plurality of frequency sub-bands for each of the audio signals generated by microphone array202. As will be appreciated by persons skilled in the relevant art(s), this operation is the mathematical equivalent of calculating a ratio between the signal level and the estimated level of acoustic echo in the linear domain for each of the plurality of frequency sub-bands for each of the audio signals generated by microphone array202.

At step408, audio source localization controller210identifies the frequency sub-bands in which the difference calculated during step406exceeds a threshold for every audio signal generated by microphone array202. In one example implementation, the threshold is in the range of 6-10 decibels (dB), and in a particular example implementation, the threshold is 6 dB.

At step410, audio source localization controller210selectively disables or enables audio source localization module206based at least on the frequency sub-bands identified during step408. For example, in one embodiment, if the number of frequency sub-bands identified during step408does not exceed a threshold, then audio source localization controller210will disable audio source localization module206from generating or outputting new location information whereas if the number of frequency sub-bands identified during step408does exceed the threshold, then audio source localization controller210will enable audio source localization module206to generate or output new location information. In a further embodiment, if the number of frequency sub-bands identified during step408exceeds the threshold, then audio source localization controller210will enable audio source localization module206to generate or output new location information based only on components of the digital audio signals produced by arrays202and204that are located in the identified frequency sub-bands, since these are the frequency sub-bands that may be deemed reliable for performing audio source localization.

One advantage of the foregoing sub-band-based approach is that it can make use of both the time and frequency separation between acoustic echo and the desired components of the audio input received by microphone array202to render a disabling/enabling decision and to identify reliable frequency sub-bands for performing audio source localization. It is noted that other sub-band based approaches may be used than those previously described. For example, in one implementation, only certain frequency sub-bands may be considered in rendering a disabling/enabling decision or for use in performing audio source localization. In another implementation, all frequency sub-bands may be considered but the contribution of each frequency sub-band to the ultimate disabling/enabling decision and/or to the audio source localization processing may be weighted. However, these are only examples and various other approaches may be used.

C. Second Example System for Performing Audio Source Localization in Accordance with an Embodiment of the Present Invention

FIG. 5is a block diagram of a second example system500for performing audio source localization in accordance with an embodiment of the present invention. In contrast to system200ofFIG. 2, which uses at least one calculated SER to determine whether or not to disable or enable an audio source localization module, system500includes an audio source localization module that estimates a level of acoustic echo present in time-aligned segments of audio signals generated by a microphone array and then uses both the time-aligned segments and the estimated level of acoustic echo in determining the location of a desired audio source. This approach also allows system500to provide improved audio source localization performance in the presence of acoustic echo as compared to the conventional solutions described in the Background Section above. System500will now be described in more detail.

As shown inFIG. 5, system500includes a number of interconnected components including a microphone array502, an array of A/D converters504, an audio source localization module506, a location-based application508, an output audio source510, an output audio processing module512, and one or more loudspeakers514. Each of these components will now be described.

Output audio source510, output audio processing module512and loudspeaker(s)514are intended to represent essentially the same structures, respectively, as output audio source212, output audio processing module214and loudspeaker(s)216as described above in reference to system200and are configured to perform like functions. For example, output audio processing module512is configured to receive an audio signal from output audio source510and to process the received audio signal for playback via loudspeaker(s)514.

Microphone array502and array of A/D converters504are intended to represent essentially the same structures, respectively, as microphone array202and array of A/D converters204as described above in reference to system200and are configured to perform like functions. For example, each microphone in microphone array502operates to convert sound waves into a corresponding analog audio signal and each A/D converter in array504operates to convert an analog audio signal produced by a corresponding microphone in microphone array502into a digital audio signal comprising a series of digital audio samples prior to delivery to audio source localization logic506.

Audio source localization module506is connected to array of A/D converters504and receives digital audio signals therefrom. Like audio source localization module206of system200, audio source localization module506periodically processes the digital audio signals to determine a current location of a desired audio source. However, in contrast to audio source localization module206which may utilize a conventional audio source localization algorithm, audio source localization module506includes an acoustic echo level estimator522that estimates a level of acoustic echo present in time-aligned segments of the digital audio signals received from array504. Audio source localization module506then uses both the time-aligned segments and the estimated level of acoustic echo in determining the location of a desired audio source. Acoustic echo level estimator522is configured to determine the estimated level of acoustic echo associated with the time-aligned segments of the digital audio signals by processing information obtained from both output audio processing module512and from array504.

After determining the current location of the desired audio source, audio source localization module506passes this information to location-based application508. Like location-based application208described above in reference to system200, location-based application508is intended to broadly represent any application that is configured to perform operations based on the location information received from audio source localization module506. Various examples of such applications have already been provided herein as part of the description of system200and thus will not be repeated here for the sake of brevity.

A general method by which audio source localization module506may operate to determine the location of a desired audio source will now be described with reference to flowchart600ofFIG. 6. Although the method of flowchart600will be described herein with reference to components of example system500, it is to be understood that the method is not limited to that implementation and may be performed by other components or systems entirely.

As shown inFIG. 6, the method of flowchart600begins at step602in which audio source localization module506obtains time-aligned segments of audio signals generated by microphone array502. These time-aligned segments may comprise, for example, time-aligned frames of the digital audio signals produced by array of A/D converters504. Each frame may comprise a fixed number of digital samples obtained at a fixed sampling rate.

At step604, acoustic echo level estimator522determines an estimated level of acoustic echo associated with the time-aligned segments obtained during step602. In one embodiment, acoustic echo level estimator222performs this function by estimating an echo return loss (ERL) associated with one or more of the time-aligned segments and then subtracting in the log domain the estimated ERL from a level of an audio signal that was processed by output audio processing module512for playback via loudspeaker(s)514. Various methods for determining an ERL are known in the art and thus need not be described herein. In one implementation, the level of the audio signal that was processed by output audio processing module512for playback via loudspeaker(s) is measured by output audio processing module512and passed to acoustic echo level estimator522.

At step606, audio source localization module506determines a location of a desired audio source based at least on the time-aligned segments and the estimated level of acoustic echo associated therewith. Various methods by which step606may be performed in accordance with various embodiments of the present invention will now be described in reference to flowcharts700,800,900,1000and1100depicted inFIGS. 7,8,9,10and11, respectively.

For example,FIG. 7depicts a flowchart700of a first method for determining a location of a desired audio source based at least on time-aligned segments of audio signals generated by a microphone array and an estimated level of acoustic echo associated therewith in accordance with an embodiment of the present invention. Although the method of flowchart700will also be described herein with continued reference to components of example system500, it is to be understood that the method is not limited to that implementation and may be performed by other components or systems entirely.

As shown inFIG. 7, the method of flowchart700begins at step702in which acoustic echo level estimator522calculates a difference between a signal level associated with the time-aligned segments and the estimated level of acoustic echo associated with the time-aligned segments in the dB domain. As will be appreciated by persons skilled in the relevant art(s), this operation is the mathematical equivalent of calculating a ratio between the signal level associated with the time-aligned segments and the estimated level of acoustic echo associated with the time-aligned segments in the linear domain. Acoustic echo level estimator522may obtain the signal level associated with the time-aligned segments, for example, by measuring a signal level associated with a designated one of the time-aligned segments or by calculating an average measure of the signal levels associated with two or more of the time-aligned segments.

At step704, acoustic echo level estimator522associates the difference calculated during step702with the time-aligned segments.

At step706, audio source localization module506processes the time-aligned segments to determine a potential location of the desired audio source. Any of a variety of known audio source localization methods may be used to perform this step.

At step708, audio source localization module506controls a degree to which the potential location determined during step706is used to determine the location of the desired audio source based at least on the difference. For example, in one embodiment, audio source localization module506determines the location of the desired audio source based on the potential location determined during step706and also on one or more locations determined for one or more previously-received sets of time-aligned segments. Each of the previously-received sets of time-aligned segments is also associated with a corresponding difference. In such an embodiment, audio source localization module506may combine the potential location associated with the current set of time-aligned segments as determined during step706and the previously-determined location(s) associated with the previously-received sets of time-aligned segments in some manner to select the new location of the desired audio source. In performing the combination, audio source localization module506may weight the contribution of each set of time-aligned segments based on the difference associated with that set. For example, if the difference associated with a particular set of time-aligned segments is relatively low (which indicates that the segments are less reliable for performing audio source localization) then audio source localization module506may apply a lesser weight to the contribution of that set, whereas if the difference associated with a particular set of time-aligned segments is relatively high (which indicates that the segments are more reliable for performing audio source localization), then audio source localization module506may apply a greater weight to the contribution of that set. The difference associated with each set of time-aligned segments can thus advantageously be used as a “trust factor” for determining the reliability of information generated by processing each set.

Persons skilled in the relevant art(s) will readily appreciate that step702may be carried out in the frequency sub-band domain, such that a difference, or SER, is obtained for each frequency sub-band. In this case, in step708, determining the degree to which the potential location is used to determine the location of the desired audio source may include, but is not limited to, considering the number of frequency sub-bands that provide what is deemed a reliable or unreliable difference, considering the differences associated with only certain frequency sub-bands, considering weighted versions of the differences associated with the frequency sub-bands, or any combination of the foregoing.

FIG. 8depicts a flowchart800of a second method for determining a location of a desired audio source based at least on time-aligned segments of audio signals generated by a microphone array and an estimated level of acoustic echo associated therewith in accordance with an embodiment of the present invention. Although the method of flowchart800will also be described herein with continued reference to components of example system500, it is to be understood that the method is not limited to that implementation and may be performed by other components or systems entirely.

As shown inFIG. 8, the method of flowchart800begins at step802, in which acoustic echo level estimator522calculates a difference between a signal level associated with the time-aligned segments and the estimated level of acoustic echo associated with the time-aligned segments. At step804, acoustic echo level estimator522associates the difference calculated during step802with the time-aligned segments. These steps are intended to represent essentially the same processes that were described above in reference to steps702and704of flowchart700.

At step806, audio source localization module506processes the time-aligned segments in a beamformer to generate a measure of a parameter associated with each of a plurality of look directions. For example, if audio source localization module506uses the well-known Steered Response Power (SRP) approach to performing localization, then step806may comprise processing the time-aligned segments in a beamformer to generate a measure of response power associated with each of a plurality of look directions. As another example, if audio source localization module506uses an approach to localization that is described in commonly-owned, co-pending U.S. patent application Ser. No. 12/566,329 (entitled “Audio Source Localization System and Method,” filed on Sep. 24, 2009, the entirety of which is incorporated by reference herein), then step806may comprise processing the time-aligned segments in a beamformer to generate a measure of distortion associated with each of the plurality of look directions.

At step808, audio source localization module506selects one of the plurality of look directions based at least on the measures of the parameter generated during step806, wherein the degree to which the measures of the parameter are used to select one of the plurality of look directions is controlled based at least on the difference. For example, in one embodiment, audio source localization module506selects the look direction based on the measures of the parameter generated during step806and also measures of the parameter generated for one or more previously-received sets of time-aligned segments. Each of the previously-received sets of time-aligned segments is also associated with a corresponding difference. In such an embodiment, audio source localization module506may combine the measures of the parameter associated with the current set of time-aligned segments as determined during step806and the previously-determined measures of the parameter associated with the previously-received sets of time-aligned segments in some manner to select the look direction. In performing the combination, audio source localization module506may weight the contribution of each set of time-aligned segments based on the difference associated with that set. The difference associated with each set of time-aligned segments can thus advantageously be used as a “trust factor” for determining the reliability of information generated by processing each set.

At step810, audio source localization module506determines the location of the desired audio source based at least on the look direction selected during step808.

Persons skilled in the relevant art(s) will readily appreciate that step802may be carried out in the frequency sub-band domain, such that a difference is obtained for each frequency sub-band. In this case, in step808, determining the degree to which the measures of the parameter are used to select one of the plurality of look directions may include, but is not limited to, considering the number of frequency sub-bands that provide what is deemed a reliable or unreliable difference, considering the differences associated with only certain frequency sub-bands, considering weighted versions of the differences associated with the frequency sub-bands, or any combination of the foregoing. The measures associated with different sets of time-aligned segments may also be combined on a frequency sub-band basis, with only certain frequency sub-bands being combined, or with different weights applied to different frequency sub-bands.

FIG. 9depicts a flowchart900of a third method for determining a location of a desired audio source based at least on time-aligned segments of audio signals generated by a microphone array and an estimated level of acoustic echo associated therewith in accordance with an embodiment of the present invention. In contrast to the methods of flowcharts700and800, which utilize an estimated level of acoustic echo to calculate a signal-to-echo ratio for a plurality of time-aligned segments and then use the ratio to weight or otherwise control the contribution of the plurality of time-aligned segments to a function used for generating a location decision, the method described in flowchart900actually applies the estimated level of acoustic echo to the level of the time-aligned segments directly. Although the method of flowchart900will also be described herein with continued reference to components of example system500, it is to be understood that the method is not limited to that implementation and may be performed by other components or systems entirely.

As shown inFIG. 9, the method of flowchart900begins at step902, in which audio source localization module506reduces a level of each of the time-aligned segments by the estimated level of acoustic echo as determined by acoustic echo level estimator522to generate modified time-aligned segments.

At step904, audio source localization module506processes the plurality of modified time-aligned segments to determine the location of the desired audio source.

FIG. 10depicts a flowchart1000of one method by which audio source localization module506may perform step904in an embodiment in which audio source localization module506uses a variant of the well known SRP-based approach for performing audio source localization.

As shown inFIG. 10, the method of flowchart1000begins at step1002in which audio source localization module506processes the modified time-aligned segments in a beamformer to identify a look direction that provides a maximum response power.

At step1004, audio source localization module506compares the maximum response power determined during step1002to a threshold.

At step1006, audio source localization module506determines the location of the desired audio source based at least on the look direction identified during step1002if the maximum response power exceeds the threshold.

In accordance with this embodiment, the level of the modified time-aligned segments that are used to generate the maximum response power will be low when the estimated level of acoustic echo is high relative to the signal level and will be high when the estimated level of acoustic echo is low relative to the signal level. By selecting the proper threshold for step1004, this will have the beneficial effect of ignoring a selected look direction when the audio input includes a disproportionally large amount of acoustic echo and is thus unreliable.

It is noted that in the methods described in reference to flowcharts900and1000, the estimated level of acoustic echo may be determined on a frequency sub-band basis. Thus, the level of the time-aligned segments can be determined for each frequency sub-band and then reduced by the estimated level of acoustic echo in the same frequency sub-band. The processing of the modified sub-bands signals can then be carried out on a frequency sub-band basis to determine the location of the desired audio source. For example, in step1002of flowchart1000, the response power for each look direction can be determined on a frequency sub-band basis. Furthermore, the threshold comparison in step1004may be carried out on a frequency sub-band basis.

It is further noted that in the embodiment described above in reference to flowchart1000, in which the estimated level of acoustic echo is applied directly to the level of the time-aligned segments and the modified time-aligned segments are then processed in a beamformer, it is critical that the same estimated level of acoustic echo is applied is applied to each segment. Applying a different estimated level of acoustic echo to each segment would negatively impact the beamformer since beamforming takes into account the relative magnitude and phase differences between the audio signals on each microphone channel. It is conceivable that a different estimated level of acoustic echo could be applied to each frequency sub-band when the implementation is in the frequency sub-band domain—however, the same overall estimated level of acoustic echo must be applied to all microphone channels.

FIG. 11depicts a flowchart1100of a fourth method for determining a location of a desired audio source based at least on time-aligned segments of audio signals generated by a microphone array and an estimated level of acoustic echo associated therewith in accordance with an embodiment of the present invention. The method of flowchart1100may be implemented in an embodiment in which audio source localization module506uses a variant of the well known SRP-based approach for performing audio source localization. Although the method of flowchart1100will also be described herein with continued reference to components of example system500, it is to be understood that the method is not limited to that implementation and may be performed by other components or systems entirely.

As shown inFIG. 11, the method of flowchart1100begins at step1102, in which audio source localization module506processes the time-aligned segments in a beamformer to identify a look direction that provides a maximum response power.

At step1104, audio source localization module506reduces the maximum response power determined during step1102by the estimated level of acoustic echo as determined by acoustic echo level estimator522to generate a modified maximum response power.

At step1106, audio source localization module506compares the modified maximum response power to a threshold.

At step1108, audio source localization module506determines the location of the desired audio source based at least on the identified look direction if the modified maximum response power exceeds the threshold.

In accordance with this embodiment, the level of the modified maximum response power will be low when the estimated level of acoustic echo is high relative to the signal level and will be high when the estimated level of acoustic echo is low relative to the signal level. By selecting the proper threshold for step1106, this will have the beneficial effect of ignoring a selected look direction when the audio input includes a disproportionally large amount of acoustic echo and is thus unreliable.

It is noted that in the method described in reference to flowchart1100, the estimated level of acoustic echo may be determined on a frequency sub-band basis. Thus, step1102can encompass determining the steered response power associated with each look direction in each frequency sub-band and step1104can encompass reducing the steered response power associated with the identified look direction in each frequency sub-band by the estimated level of acoustic echo in the same frequency sub-band. As a result, the comparison of the maximum response power to a threshold in step1106can be carried out on a frequency sub-band basis if desired.

D. Example Embodiments Including Acoustic Echo Cancellers

Although example systems200and500described above in reference toFIGS. 2 and 5, respectively, did not include acoustic echo cancellers, embodiments of the present invention may also be implemented in systems that include acoustic echo cancellers. For example,FIG. 12is a block diagram of such a system1200.

As shown inFIG. 12, system1200includes an array of microphones1202, an array of A/D converters1204, a location-based application1210, an output audio source1214, an output audio processing module1216and one or more loudspeakers1218. These components are intended to represent essentially the same structures, respectively, as array of microphones202, array of A/D converters204, location-based application208, output audio source212, output audio processing module214and loudspeaker(s)216as described above in reference to system200and are configured to perform like functions.

As further shown inFIG. 12, system1200includes an array of acoustic echo cancellers1206that operate to receive the digital representations of the audio signals produced by arrays1202and1204and to perform acoustic echo cancellation thereon. As will be appreciated by persons skilled in the relevant art(s), the acoustic echo cancellation function is performed based at least in part on information concerning an output audio signal processed by output audio processing module1216. The signals generated by array1206are then provided to an audio source localization module1208which processes the signals to determine a current location of a desired audio source and passes the location information to location-based application1210.

System1200also includes an audio source localization controller1212. Audio source localization controller1212selectively enables audio source localization module1208to produce updated location information when it determines that the impact of acoustic echo upon the performance of the module is likely to be acceptable and selectively disables audio source localization module1208from producing updated location information when it determines that the impact of acoustic echo upon the performance of the module is likely to be unacceptable. To determine the impact of acoustic echo upon the performance of audio source localization module1208, audio source localization controller includes an SER calculator1222that calculates at least one SER upon which the disabling/enabling decision is premised.

However, unlike SER calculator222of system200which determines an SER by calculating a difference in the dB domain between a signal level associated with one or more of the audio signals generated by a microphone array and an estimated level of acoustic echo associated with one or more of those signals, SER calculator1222determines an SER by calculating a difference in the dB domain between a signal level associated with one or more of the audio signals generated by microphone array1202after application of acoustic echo cancellation thereto to and an estimated level of residual echo associated with one or more of those signals after application of acoustic echo cancellation thereto.

In one embodiment, the estimated level of residual echo is determined by estimating an ERL associated with one or more of the audio signals generated by microphone array1202after application of acoustic echo thereto and then subtracting the ERL from the level of an output audio signal processed by output audio processing module1216. In this case, ERL refers to the combined loss between the echo path and the echo cancellation operation. In another embodiment, the estimated level of residual echo is determined by estimating an ERL associated with one or more of the audio signals generated by microphone array1202and an estimate of the amount of echo cancellation that is obtained by the echo cancellers (which may be referred to as the echo return loss enhancement (ERLE)) and then subtracting the estimated ERL and ERLE from the level of an output audio signal processed by output audio processing module1216.

Aside from the manner in which the SER is calculated as described above, the operation of system1200may be otherwise identical to that described above in reference to system200ofFIG. 2and in reference to flowcharts300and400as described above in reference toFIGS. 3 and 4. It is noted that the inclusion of acoustic echo cancellers in system1200ofFIG. 12may provide improved performance since the estimated level of residual echo will generally be lower than the estimated level of echo.

FIG. 13is a block diagram of another system1300that includes acoustic echo cancellers and performs audio source localization in accordance with an embodiment of the present invention. As shown inFIG. 13, system1300includes an array of microphones1302, an array of A/D converters1304, a location-based application1310, an output audio source1312, an output audio processing module1314and one or more loudspeakers1316. These components are intended to represent essentially the same structures, respectively, as array of microphones502, array of A/D converters504, location-based application508, output audio source510, output audio processing module512and loudspeaker(s)514as described above in reference to system500and are configured to perform like functions.

As further shown inFIG. 13, system1300includes an array of acoustic echo cancellers1306that operate to receive the digital representations of the audio signals produced by arrays1302and1304and to perform acoustic echo cancellation thereon. As will be appreciated by persons skilled in the relevant art(s), the acoustic echo cancellation function is performed based at least in part on information concerning an output audio signal processed by output audio processing module1314. The signals generated by array1306are then provided to an audio source localization module1308which processes the signals to determine a current location of a desired audio source and passes the location information to location-based application1310.

Audio source localization module1308includes an acoustic echo level estimator1322that estimates a level of acoustic echo present in time-aligned segments of the digital audio signals received from array1306. Audio source localization module1308then uses both the time-aligned segments and the estimated level of acoustic echo in determining the location of a desired audio source. Any of the methods described above in reference to flowcharts600,700,800,900,1000and1100ofFIGS. 6,7,8,9,10and11, respectively, may be used to perform this function.

However, unlike acoustic echo level estimator522of system500which determines an estimated level of acoustic echo associated with the time-aligned segments of the audio signals generated by a microphone array, acoustic echo level estimator1322determines an estimated level of residual echo associated with the time-aligned segments of audio signals generated by microphone array1302after application of acoustic echo cancellation thereto. Various methods for determining an estimated level of residual echo were previously described in reference to SER calculator1222of system1200. In embodiments of system1300in which an SER is also calculated, the signal level refers to a signal level associated with the time-aligned segments of audio signals generated by microphone array1302after application of acoustic echo thereto. The inclusion of acoustic echo cancellers in system1300ofFIG. 13may provide improved performance since the estimated level of residual echo will generally be lower than the estimated level of echo.

E. Example Computer System Implementation

It will be apparent to persons skilled in the relevant art(s) that various elements and features of the present invention, as described herein, may be implemented in hardware using analog and/or digital circuits, in software, through the execution of instructions by one or more general purpose or special-purpose processors, or as a combination of hardware and software.

The following description of a general purpose computer system is provided for the sake of completeness. Embodiments of the present invention can be implemented in hardware, or as a combination of software and hardware. Consequently, embodiments of the invention may be implemented in the environment of a computer system or other processing system. An example of such a computer system1400is shown inFIG. 14. Various components depicted inFIGS. 2 and 5, for example, can execute on one or more distinct computer systems1400. Furthermore, any or all of the steps of the flowcharts depicted inFIGS. 3,4and6-11can be implemented on one or more distinct computer systems1400.

Computer system1400includes one or more processors, such as processor1404. Processor1404can be a special purpose or a general purpose digital signal processor. Processor1404is connected to a communication infrastructure1402(for example, a bus or network). Various software implementations are described in terms of this exemplary computer system. After reading this description, it will become apparent to a person skilled in the relevant art(s) how to implement the invention using other computer systems and/or computer architectures.

Computer system1400also includes a main memory1406, preferably random access memory (RAM), and may also include a secondary memory1420. Secondary memory1420may include, for example, a hard disk drive1422and/or a removable storage drive1424, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, or the like. Removable storage drive1424reads from and/or writes to a removable storage unit1428in a well known manner. Removable storage unit1428represents a floppy disk, magnetic tape, optical disk, or the like, which is read by and written to by removable storage drive1424. As will be appreciated by persons skilled in the relevant art(s), removable storage unit1428includes a computer usable storage medium having stored therein computer software and/or data.

In alternative implementations, secondary memory1420may include other similar means for allowing computer programs or other instructions to be loaded into computer system1400. Such means may include, for example, a removable storage unit1430and an interface1426. Examples of such means may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM, or PROM) and associated socket, and other removable storage units1430and interfaces1426which allow software and data to be transferred from removable storage unit1430to computer system1400.

Computer system1400may also include a communications interface1440. Communications interface1440allows software and data to be transferred between computer system1400and external devices. Examples of communications interface1440may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, etc. Software and data transferred via communications interface1440are in the form of signals which may be electronic, electromagnetic, optical, or other signals capable of being received by communications interface1440. These signals are provided to communications interface1440via a communications path1442. Communications path1442carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link and other communications channels.

As used herein, the terms “computer program medium” and “computer readable medium” are used to generally refer to media such as removable storage units1428and1430or a hard disk installed in hard disk drive1422. These computer program products are means for providing software to computer system1400.

Computer programs (also called computer control logic) are stored in main memory1406and/or secondary memory1420. Computer programs may also be received via communications interface1440. Such computer programs, when executed, enable the computer system1400to implement the present invention as discussed herein. In particular, the computer programs, when executed, enable processor1400to implement the processes of the present invention, such as any of the methods described herein. Accordingly, such computer programs represent controllers of the computer system1400. Where the invention is implemented using software, the software may be stored in a computer program product and loaded into computer system1400using removable storage drive1424, interface1426, or communications interface1440.

In another embodiment, features of the invention are implemented primarily in hardware using, for example, hardware components such as application-specific integrated circuits (ASICs) and gate arrays. Implementation of a hardware state machine so as to perform the functions described herein will also be apparent to persons skilled in the relevant art(s).