Audio zoom

An audio equivalent of a video zoom feature for video recording and communication applications, as well as video post production processes. The audio zoom may operate in conjunction with a video zoom feature or independently. The audio zoom may be achieved by controlling reverberation effects of a signal, controlling a gain of the signal, as well as controlling the width of a directional beam which is used to select the particular audio component to focus on. The audio zoom may operate in response to user input, such as a user selection of a particular direction, or automatically based a current environment or other factors.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to audio processing, and more particularly to a noise suppression processing of an audio signal.

2. Description of Related Art

A common feature in video and images is a “zoom” function. When engaging a video zoom, a user of the product may either view a smaller portion of the screen as the full size image or “step back” and view a wider portion of the current image.

Currently, there is no “zoom” mechanism for an audio source within an audio signal containing multiple sources. A user may increase a volume of an audio signal, but this does not differentiate between different source directions within the received signal. There is a need for an audio zoom feature for audio signals.

SUMMARY OF THE INVENTION

The present technology provides an audio equivalent of a video zoom feature for video recording and communication applications, as well as video post-production processes. The audio zoom may operate in conjunction with a video zoom feature or independently. The audio zoom may be achieved by controlling reverberation effects of a signal, controlling a gain of the signal, as well as controlling the width and direction of a directional beam pointing towards the particular audio component to focus on. The audio zoom may operate in response to user input, such as a user selection of a particular direction, or automatically based on a current environment or other factors. An embodiment for performing audio zoom may receive one or more acoustic signals by one or more microphones on a device. Each acoustic signal may be associated with a spatial source within an environment. An indication of a spatial area within the environment may be received. The energy of acoustic signal components may be reduced. The acoustic signal components may be associated with a source positioned outside the spatial area. A reverberation level associated with acoustic signal components may be adjusted. The acoustic signal components may be associated with a source positioned inside the spatial area based on the indication.

An embodiment of a system for performing an audio zoom may include a memory, a beamformer module, and a reverb module. The beamformer module may be stored in the memory and executed by a processor to identify sub-band signals associated with an audio source within a spatial area associated with an audio source to zoom in. The reverb module stored in the memory and executed by a processor to cancel at least a portion of the sub-band signals.

Additionally, a computer readable storage medium may be implemented in which a program is embodied, the program being executable by a processor to perform a method for performing an audio zoom.

DETAILED DESCRIPTION OF THE INVENTION

The present technology provides an audio equivalent of a video zoom feature for video recording and communication applications, as well as video post-production processes. The audio zoom may operate in conjunction with a video zoom feature or independently. The audio zoom may be achieved by controlling reverberation effects of a signal, controlling a gain of the signal, as well as controlling the width and direction of a directional beam pointing towards the particular audio component to focus on. The audio zoom may operate in response to user input, such as a user selection of a particular direction, or automatically based on a current environment or other factors.

FIG. 1is a block diagram of an exemplary environment100in which the present technology can be used. The environment100ofFIG. 1includes audio device104, audio sources112,114and116, all within an environment having walls132and134.

A user of a audio device104may choose to focus on or “zoom” into a particular audio source from the multiple audio sources within environment100. Environment100includes audio source112,114, and116which all provide audio in a multidirectional audio, including towards audio device104. Additionally, reflections from audio sources112and116as well as other audio sources may provide audio which reflects off the walls132and134of the environment and is directed at audio device104. For example, reflection128is a reflection of an audio signal provided by audio source112and reflected from wall132, and reflection129is a reflection of an audio signal provided by audio source116and reflected from wall134, both of which travel towards audio device104.

The present technology allows the user to select an area to “zoom.” By performing an audio zoom on a particular area, the present technology detects audio signals having a source within the particular area and enhances those signals with respect to signals from audio sources outside the particular area. The area may be defined using a beam, such as for example beam140inFIG. 1. InFIG. 1, beam140contains an area that includes audio source114. Audio sources112and116are outside the beam area. As such, the present technology would emphasize or “zoom” on the audio signal provided by audio source114and de-emphasize the audio provided by audio sources112and116, including any reflections provided by environment100, such as reflections128and129.

The primary microphone106and secondary microphone108of audio device104may be omni-directional microphones. Alternatively embodiments may utilize other forms of microphones or acoustic sensors, such as directional microphones.

While the microphones106and108receive sound (i.e. acoustic signals) from the audio source114, the microphones106and108also pick up noise from audio source112. Although the noise122is shown coming from a single location inFIG. 1, the noise122may include any sounds from one or more locations that differ from the location of audio source114, and may include reverberations and echoes. The noise122may be stationary, non-stationary, and/or a combination of both stationary and non-stationary noise.

Some embodiments may utilize level differences (e.g. energy differences) between the acoustic signals received by the two microphones106and108. Because the primary microphone106is much closer to the audio source116than the secondary microphone108in a close-talk use case, the intensity level for noise126is higher for the primary microphone106, resulting in a larger energy level received by the primary microphone106during a speech/voice segment, for example.

The level difference may then be used to discriminate speech and noise in the time-frequency domain. Further embodiments may use a combination of energy level differences and time delays to discriminate speech. Based on binaural cue encoding, speech signal extraction or speech enhancement may be performed.

FIG. 2is a block diagram of an exemplary audio device. In some embodiments, audio device ofFIG. 2provides more detail for audio device104ofFIG. 1.

In the illustrated embodiment, the audio device104includes a receiver210, a processor220, the primary microphone106, an optional secondary microphone108, an audio processing system230, and an output device240. The audio device104may include further or other components necessary for audio device104operations. Similarly, the audio device104may include fewer components that perform similar or equivalent functions to those depicted inFIG. 2.

Processor220may execute instructions and modules stored in a memory (not illustrated inFIG. 2) in the audio device104to perform functionality described herein, including noise reduction for an acoustic signal. Processor220may include hardware and software implemented as a processing unit, which may process floating point operations and other operations for the processor220.

The exemplary receiver210is an acoustic sensor configured to receive a signal from a communications network. In some embodiments, the receiver210may include an antenna device. The signal may then be forwarded to the audio processing system230to reduce noise using the techniques described herein, and provide an audio signal to the output device240. The present technology may be used in one or both of the transmit and receive paths of the audio device104.

The audio processing system230is configured to receive the acoustic signals from an acoustic source via the primary microphone106and secondary microphone108and process the acoustic signals. Processing may include performing noise reduction within an acoustic signal. The audio processing system230is discussed in more detail below. The primary and secondary microphones106,108may be spaced a distance apart in order to allow for detecting an energy level difference, time difference or phase difference between them. The acoustic signals received by primary microphone106and secondary microphone108may be converted into electrical signals (i.e. a primary electrical signal and a secondary electrical signal). The electrical signals may themselves be converted by an analog-to-digital converter (not shown) into digital signals for processing in accordance with some embodiments. In order to differentiate the acoustic signals for clarity purposes, the acoustic signal received by the primary microphone106is herein referred to as the primary acoustic signal, while the acoustic signal received from by the secondary microphone108is herein referred to as the secondary acoustic signal. The primary acoustic signal and the secondary acoustic signal may be processed by the audio processing system230to produce a signal with an improved signal-to-noise ratio. It should be noted that embodiments of the technology described herein may be practiced utilizing only the primary microphone106.

The output device240is any device which provides an audio output to the user. For example, the output device240may include a speaker, an earpiece of a headset or handset, or a speaker on a conference device.

In various embodiments, where the primary and secondary microphones are omni-directional microphones that are closely-spaced (e.g., 1-2 cm apart), a beamforming technique may be used to simulate forwards-facing and backwards-facing directional microphones. The level difference may be used to discriminate speech and noise in the time-frequency domain which can be used in noise reduction.

FIG. 3is a block diagram of an exemplary audio processing system. The block diagram ofFIG. 3provides more detail for the audio processing system230in the block diagram ofFIG. 2. Audio processing system230includes FCT modules302and304, beam former module310, multiplicative gain expansion module320, reverb module330, mixer module340, and zoom control module350.

FCT modules302and304may receive acoustic signals from audio device microphones and convert the acoustic signals to frequency range sub-band signals. FCT modules302and304may be implemented as one or more modules which create one or more sub-band signals for each received microphone signal. FCT modules302and304may receive an acoustic signal from each microphone contained in audio device104. These received signals are illustrated as signals X1-XI, wherein X1is a primary microphone signal and XIrepresents the remaining microphone signals. The audio processing system230ofFIG. 3may perform audio zoom on a per frame and per sub band basis.

Beam former module310may receive the frequency sub-band signals as well as a zoom indication signal. The zoom indication is received from zoom control module350. The zoom indication communicated by zoom indicator signal K may be generated in response to user input, analysis of a primary microphone signal or other acoustic signals received by audio device104, a video zoom feature selection, or some other data. In operation, beam former module310receives sub-band signals, processes the sub-band signals to identify which signals are within a particular area to enhance (or “zoom”), and provide data for the selected signals as output to multiplicative gain expansion module320. The output may include sub-band signals for the audio source within the area to enhance. Beam former module310also provides a gain factor to multiplicative gain expansion module320. The gain factor may indicate whether multiplicative gain expansion module320should perform additional gain or reduction to the signals received from beam former module310. In some embodiments, the gain factor is generated as an energy ratio based on the received microphone signals and components. The gain indication output by beam former module310is a ratio of how much energy that one has reduced in the primary versus an output energy. Hence, the gain may be a boost or cancellation gain expansion factor. The gain factor is discussed in more detail below.

Beam former module310can be implemented as a null processing noise subtraction (NPNS) module, multiplicative module, or a combination of these modules. Beam former module310may implement an array processing algorithm described below with respect to the beam former module ofFIG. 4. When an NPNS module is used in microphones to generate a beam and achieve beam forming, the beam may be focused by narrowing constraints of alpha and gamma. For a rider beam, the constraints may be made larger. Hence a beam may be manipulated by putting a protective range around the preferred direction. Beam former module310may be implemented by a system described in the U.S. patent application No. 61/325,764, entitled “Multi-Microphone Robust Noise Suppression System,” the disclosure of which is incorporated herein by reference. Additional techniques for reducing undesired audio components of a signal are discussed in U.S. patent application Ser. No. 12/693,998, entitled “Adaptive Noise Reduction Using Level Cues,” the disclosure of which is incorporated herein by reference.

Multiplicative gain expansion module320receives the sub-band signals associated with audio sources within the selected beam, the gain factor from beam former module310, and the zoom indicator signal. Multiplicative gain expansion module320applies a multiplicative gain based on the gain factor received. In effect, module320filters the beam former signal provided by beam former module310.

The gain factor may be implemented as one of several different energy ratios. For example, the energy ratio may be the ratio of a noise reduced signal to a primary acoustic signal received from a primary microphone, the ratio of a noise reduce signal and a detected noise component within the primary microphone signal, the ratio of a noise reduce signal and a secondary acoustic signal, or the ratio of a noise reduce signal compared to the intra level difference between a primary signal and another signal. The gain factors may be an indication of signal strength in a target direction versus all other directions. Put another way, the gain factor may be an indication of multiplicative expansions due, and whether additional expansion or less expansion should be performed at the multiplicative gain expansion module320. Multiplicative gain expansion module320outputs the modified signal and provides signal to reverb module330(which may also function to de-reverb).

Reverb module330receives the sub-band signals output by Multiplicative gain expansion module320, as well as the microphone signals which were also received by beam former module310, and performs reverberation or dereverberation to the sub-band signals output by Multiplicative gain expansion module320. Reverb module330may adjust a ratio of direct energy to remaining energy within a signal based on the zoom control indicator provided by zoom control module350.

Adjusting the reverb for a signal may involve adjusting the energy of different components of the signal. An audio signal has several components in a frequency domain, including a direct component, early reflections, and a tail component. A direct component typically has the highest energy level, followed by a somewhat lower energy level of reflections within the signal. Also included within a very particular signal is a tail which may include noise and other low energy data or low energy audio. A reverberation is defined as reflections of the direct audio component. Hence, a reverberation with many reflections over a broad frequency range account for a more noticeable reverberation. A signal with fewer reflection components has a smaller reverberation component.

Typically, the further away a listener is from an audio source, the larger the reverberation in the signal. The closer a listener is to an audio source, the smaller the intensity of the reverberation signal (reflection components). Hence, based on the zoom indicator received from zoom control module350, reverb module330may adjust the reverberation components in the signal received from multiplicative gain expansion module320. Hence, if the zoom indicator received indicates that a zoom in operation is to be performed on the audio, the reverberation will be decreased by minimizing the reflection components of the received signal. If the zoom indicator indicates that a zoom out is to be performed on the audio signal, the early reflection components are gained to increase these components to make it appear as if there is additional reverberation within the signal. After adjusting the reverberation of the received signal, reverb module330provides the modified signal to mixing module340.

The mixing module340receives the reverberation adjusted signal and mixes the signal with the signal from the primary microphone. Mixing module340may increase the energy of the signal appropriately where there was audio present in the frame and decreased where there was little audio energy present in the frame.

FIG. 4is a block diagram of an exemplary beam former module. The beam former module310may be implemented per tap (i.e., per sub-band). Beam former module310receives FCT output signals for a first microphone (such as a primary microphone) and a second microphone. The first microphone FCT signal is processed by module410according to the function:

to generate a first differential array with parameters.

The secondary microphone FCT signal is processed by module420according to the function:

to generate a second differential array with parameters.

The output of module410is then subtracted from the secondary microphone FCT signal at combiner440and the output of module420is then subtracted by the primary microphone FCT signal at combiner430. A cardioid signal Cfis output from combiner430and provided to module450where the following function is applied:

A cardioid signal Cbis output from combiner430and provided to module450where the following function is applied:

The difference of the outputs of modules450and460is determined4element470and output as an ILD cue. The ILD cue may be output by beam former module310to a post filter, for example a filter implemented by multiplicative gain expansion module320.

FIG. 5is a flow chart of an exemplary method for performing an audio zoom. An acoustic signal is received from one or more sources at step510. The acoustic signals may be received through one or more microphones on audio device104. For example, acoustic signals from audio sources112-116and reflections128-129may be received through microphones106and108of audio device104.

A zoom indication is then received for a spatial area at step520. The zoom indication may be received from a user or determined based on other data. For example, the zoom indication may be received from a user by one of a video zoom setting, pointing an audio device in a particular direction, an input for video zoom, or in some other manner.

Acoustic signal component energy levels may be enhanced based on the zoom indication at step530. Acoustic signal component energy levels may be enhanced by increasing the energy levels for audio source sub-band signals that originate from a source device within a selected beam area. Audio signals from a device outside a selected beam area are de-emphasized. Enhancing acoustic signal component energy levels is discussed in more detail below with respect to the method ofFIG. 6.

Reverberation signal components associated with a position inside the spatial area are adjusted based on the received indication at step540. As discussed above, the adjustments may include modifying the ratio of a direct component with respect to reflection components for the particular signal. When a zoom in function is to be performed, reverberation should be decreased by increasing the ratio of the direct component to the reflection components in the audio signal. When a zoom out function is performed for the audio signal, the direct component is reduced with respect to the reflection components to decrease the ratio of direct to reflection components of the audio signal.

A modulated gain is applied to the signal component at step550. The gain may be applied by mixing a reverb processed acoustic signal with a primary acoustic signal (or another audio signal received by audio device104). The mixed signal which has been processed by audio zoom is in output at step560.

As discussed above, sub-band signals may be enhanced based on a zoom indication.FIG. 6is a flow chart of an exemplary method for enhancing acoustic signal components. In some embodiments, the method inFIG. 6provides more detail for step530of the method inFIG. 5. An audio source is detected in the direction of a beam at step610. This may be performed by a null-processing noise subtraction mechanism or some other module that is able to identify a spatial position of a source based on audio signals received by two or more microphones.

Acoustic signal sources located outside the spatial area are attenuated at step620. The acoustic sources outside the spatial area may include certain audio sources (e.g.,112inFIG. 1) and reflected audio signals such as reflections128and129. Adaptation constraints are then used to steer the beam based on the zoom indication at step630. The adaptation constraints may include α and σ constraints used in a null processing noise suppression system. The adaptation constraints may also be derived from multiplicative expansion or selection of a region around a preferred direction based on a beam pattern.

Energy ratios are then determined at step640. The energy ratios may be used to derive multiplicative masks that boost or reduce a beam former cancellation gain for signal components. Next, multiplicative masks are generated based on energy ratios at step650. Generating multiplicative masks based on an energy ratio is discussed in more detail below with respect to the method ofFIG. 7.

FIG. 7is a flow chart of an exemplary method for generating a multiplicative mask. The method ofFIG. 7provides more detail for step650in the method ofFIG. 6. Differential arrays are generated from microphone signals at step710. The arrays may be generated as part of a beam former module310. The beam pattern may be a cardiod pattern generated based at least in part from the differential output signals. Next, a beam pattern is generated from the differential areas at step720. Energy ratios are then generated from beam patterns at step730. The energy ratios may be generated as any of a combination of signals. Once generated, an ILD map may be generated per frequency from energy ratios. An ILD range corresponding to the desired selection may be selected. An ILD window may then be applied to a map by boosting the signal components within the window and attenuating the signal components positioned outside the window. A filter, such as a post filter, may be derived from the energy ratio at step740.

The above described modules, including those discussed with respect toFIG. 3, may include instructions stored in a storage media such as a machine readable medium (e.g., computer readable medium). These instructions may be retrieved and executed by the processor220to perform the functionality discussed herein. Some examples of instructions include software, program code, and firmware. Some examples of storage media include memory devices and integrated circuits.

While the present invention is disclosed by reference to the preferred embodiments and examples detailed above, it is to be understood that these examples are intended in an illustrative rather than a limiting sense. It is contemplated that modifications and combinations will readily occur to those skilled in the art, which modifications and combinations will be within the spirit of the invention and the scope of the following claims.