Multi-band integrated speech separating microphone array processor with adaptive beamforming

A speech separating digital signal processing system and algorithms for implementing speech separation combine beam-forming with residual noise suppression, such as computational auditory scene analysis (CASA) using a beam-former that has a primary lobe steered toward the source of speech by a control value generated from an adaptive filter. An estimator estimates the ambient noise and provides an input to the residual noise suppressor, and a post-filter may be used to noise-reduce the output of the estimator using a time-varying filter that compares two or more outputs of the beam-former with a quasi-stationary model of the speech and ambient noise.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to audio communication systems, and more specifically, to techniques for separating speech from ambient acoustic noise.

2. Background of the Invention

The problem of separation of speech from one or more persons speaking in a room or other environment is central to the design and operation of systems such as hands-free telephone systems, speaker phones and other teleconferencing systems. Further, the separation of speech from other sounds in an ambient acoustic environment, such as noise, reverberation and other undesirable sounds such as other speakers can be usefully applied in other non-duplex communication or non-communication environments such as digital dictation devices, computer voice command systems, hearing aids and other applications in which reduction of sounds other than desired speech provides an improvement in performance.

Processing systems that separate desired speech from undesirable background sounds and noise may use a single microphone, or two or more microphones forming a microphone array. In single microphone applications, the processing algorithms typically rely entirely on source-attribute filtering algorithms that attempt to isolate the speech (source) algorithmically, for example computational auditory scene analysis (CASA). In some implementations, two or more microphones have been used to estimate the direction of desired speech. The algorithms rely on separating sounds received by the one or more microphones into types of sounds, and in general are concerned with filtering the background sound and noise from the received information.

However, when practical, a microphone array can be used to provide information about the relative strength and arrival times of sounds at different locations in the acoustic environment, including the desired speech. The algorithm that receives input from the microphone array is typically a beam-forming processing algorithm in which a directivity pattern, or beam, is formed through the frequency band of interest to reject sounds emanating from directions other than the speaker whose speech is being captured. Since the speaker may be moving within the room or other environment, the direction of the beam is adjusted periodically to track the location of the speaker.

Beam-forming speech processing systems also typically apply post-filtering algorithms to further suppress background sounds and noise that are still present at the output of the beam-former. However, until recently, the source-attribute processing techniques were not used in beam-forming speech processing systems. The typical filtering algorithms employed are fast-Fourier transform (FFT) algorithms that attempt to isolate the speech from the background, which have relatively high latency for a given signal processing capacity.

Since source-attribute filtering techniques such as CASA rely on detecting and determining types of the various sounds in the environment, inclusion of a beam-former having a beam directed only at the source runs counter to the detection concept. For the above reason, combined source-attribute filtering and location-based techniques typically use a wideband multi-angle beam-former that separates the scene being analyzed by angular location, but still permits analysis of the entire ambient acoustic environment. The wideband multi-angle beam-formers employed do not attempt to cancel all signals other than the direct signal from the speech source, as a narrow beam beam-former would, and therefore loses some signal-to-noise-ratio reduction by not providing the highest possible selectivity through the directivity of a single primary beam.

Therefore, it would be desirable to provide improved techniques for separating speech from other sounds and noise in an acoustic environment. It would further be desirable to combine source-attribute filtering with narrow band source tracking beam-forming to obtain the benefits of both. It would further be desirable to provide such techniques with a relatively low latency.

SUMMARY OF THE INVENTION

The above stated objective of separating a particular speech source from other sounds and noise in an acoustic environment is accomplished in a system and method. The method is a method of operation of the system, which may be a digital signal processing system executing program instructions forming a computer program product embodiment of the present invention.

The system receives multiple microphone signals from microphones at multiple positions and filters each of the microphone signals to split them into multiple frequency band signals. A spatial beam is formed having a primary lobe with a direction adjusted by a beam-former. The beam-former receives the multiple frequency band signals for each of the multiple microphone signals. At least one of the multiple frequency band signals is adaptively filtered to periodically determine a position of the speech source and generate a steering control value. The direction of the primary lobe of the beam-formed is adjusted by the steering control value toward the determined position of the speech source. The ambient acoustic noise is estimated and at least one output of the beam-former is processed using a result of the estimating to suppress residual noise to obtain the separated speech.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENT

The present invention encompasses audio processing systems that separate speech from an ambient acoustic background (including other speech and noise). The present invention uses a steering-controlled beam-former in combination with residual noise suppression, such as computational auditory scene analysis (CASA) to improve the rejection of unwanted audio signals in the output that represents the desired speech signal. In the particular embodiments described below, the system is provided in a mobile phone that enables normal phone conversation in a noisy environment. In implementation such as the mobile telephone depicted herein, the present invention improves speech quality and provides more pleasant phone conversation in a noisy acoustic environment. Also, the ambient sound is not transmitted to the distant talker, which improves clarity at the receiving end and efficiently uses channel bandwidth, particularly in adaptive coding schemes.

Referring now toFIG. 1, a mobile telephone8in accordance with an embodiment of the present invention is shown. Signals provided from a first microphone101and a second microphone102provide inputs to respective analog-to-digital converter (ADC)103and ADC104. Microphones101and102are closely-spaced, according to the dimensions of packaging of depicted mobile telephone8. A digital signal processor (DSP)10receives the outputs of ADCs103and104. DSP10includes a processor core12, a data memory (DMEM)14and an instruction memory (IMEM)16, in which program instructions are stored. Program instructions in IMEM16operate on the values received from ADCs103and104to generate signals for transmission by a global system for mobile communications (GSM) radio18, among other operations performed within mobile telephone8. In accordance with an embodiment of the invention, the program instructions within IMEM16include program instructions that implement an ambient noise suppressor (ANS)105, details of which will be described below. IMEM16also includes program instructions that implement an adaptive multi-rate codec106that encodes the output of ANS105for transmission by GSM radio18, and will generally include other program instructions for performing other functions within mobile telephone8and operating on the output of ANS105, including acoustic echo cancellers (AEC) and automatic gain control circuits (AGCs). The present invention concerns structures and methodologies applied in ANS105, and therefore details of other portions of mobile telephone8are omitted for clarity.

Referring now toFIG. 2, details of ANS105are shown in a block diagram. While ANS105in the illustrative embodiment is a set of program instructions, i.e, a set of software modules that implement a digital signal processing method, the information flow within the software modules can be represented as a block diagram, and further a system in accordance with an alternative embodiment of the present invention comprises logic circuits configured as shown in the following block diagrams. Some or all of the signal processing in an embodiment of the present invention may be performed in dedicated logic circuits, with the remainder implemented by a DSP core executing program instructions. Therefore, the block diagrams depicted inFIGS. 2-8are understood to apply to both software and hardware implementations of the algorithms forming ANS105in mobile telephone8.

Signals XMLand XMR, which are digitized versions of the outputs of microphones101and102, respectively, are received by ANS105from ADCs103and104. A pair of gammatone filter banks201and202respectively filter signals XMLand XMR, splitting signals XMLand XMRinto two sets of multi-band signals XLand XR. Gammatone filter banks201and202are identical and have n channels each. In the exemplary embodiment depicted herein, there are sixty-four channels provided from each of gammatone filter banks201and202, with the frequency bands spaced according to the Bark scale. The filters employed are fourth-order infinite impulse response (IIR) bandpass filters, but other filter types including finite impulse response (FIR) filters may alternatively be employed. Multi-band signals XLand XRare provided as inputs to a reference generator204.

Reference generator204generates an estimate of the ambient noise XN, which includes all sounds occurring in the acoustic ambient environment of microphones101and102, except for the desired speech signal. Reference generator204, as will be shown in greater detail below, generates an adaptive control signal Cθas part of the process of cancelling the desired speech from the estimate of the ambient acoustic noise XN, which is then used as a steering control signal provided to a steering controlled beam-former (SCBF)203. SCBF203processes multi-band signals XLand XRaccording to the direction of the speaker's head as specified by adaptive control signal Cθ, which in the depicted embodiments is a vector representing parameters of an adaptive filter internal to SCBF203. The output of SCBF203is a multichannel speech signal XSwith partly suppressed ambient acoustic noise due to the directional filtering provided by SCBF203.

Multichannel speech signal XSand the estimated ambient acoustic noise XNare provided to post-filter205that implements a time-varying filter similar to a Wiener filter that suppresses the residual noise from multi-channel speech signal XSto generate another multi-channel signal XW. Multi-channel signal XWis mostly the desired speech, since the estimated noise is removed according to post-filter205. However, residual interference is further removed by a computational auditory scene analysis (CASA) module206, which receives the multi-channel speech signal XS, the reduced-noise speech signal XW, and an estimated fundamental frequency f0of the speech as provided from a fundamental frequency estimation block207. The output of CASA module206is a fully processed speech signal XOUTwith ambient acoustic noise removed by directional filtering, filtering according to quasi-stationary estimates of the speech and the ambient acoustic noise, and final post-processing according to CASA. In particular, the post-filtering applied by post-filter205provides a high degree of noise filtering not present in other beam-forming systems. Pre-filtering using the directionally filtered speech and the estimated noise according to quasi-stationary filtering techniques provides additional signal-to-noise ratio improvement over scene analysis techniques that are operating on direct microphone inputs or inputs filtered by a multi-source beam-forming technique.

Referring now toFIG. 3, details of reference generator204and SCBF203are shown. A filter301having parameters Cθand a subtractor302form a normalized least-means-squared (NLMS) adaptive filter that is controlled by a voice activity detector304. The adaptive filter suppresses speech in multichannel signal XLby using multichannel signal XRas reference. Subtractor302subtracts the output of filter301, which filters multichannel signal XR, from multichannel signal XL. An adaption control block303tunes filter301by adjusting parameters Cθ, so that at the output of subtractor302the desired speech signal is canceled, effectively steering a directivity null formed by subtractor302that tracks the speaker's head. There is high correlation between the ambient acoustic noise components of multichannel signals XLand XRsignals, particularly in the low frequency channels, where wavelengths are long compared to the distance between microphones101and102.

Adaption control block303can adapt parameters Cθaccording to minimum energy in error signal e, which may be qualified by observing only the lower frequency bands. Error signal e is by definition given by E(t)=XL(t)−CθXR(t), where t is an instantaneous time value, and a NLMS algorithm can be used to estimate Cθaccording to:

C^θ⁡(t)=C^θ⁡(t-1)+μ⁢E⁡(t)XR⁡(t)2+δ2⁢XR*⁡(t)
where μ is a positive scalar that control the convergence rate of time-varying parameters Cθ(t), δ is a positive scalar that provides stability for low magnitudes of multichannel signal XR. Adaptation can be stopped during non-speech intervals, according to the output of VAD304, which decides whether speech is present from the instantaneous power of multichannel signal XR, trend of the signal power, and dynamically estimated thresholds.

As noted above, in addition to providing input to adaptation block303, error signal e is also used for estimation of the ambient acoustic noise. While the speech signal is highly suppressed in error signal e, the ambient noise is also, since microphones101and102are closely spaced and the ambient acoustic noise in multichannel signals XLand XRis therefore highly correlated. A gain control block306calculates a gain factor that compensates for the noise attenuation caused by the adaptive filter formed by subtractor302and filter301. The output of multiplier307, which multiplies error signal e by a gain factor g(t), is estimated ambient acoustic noise signal XN.

Referring now toFIG. 4, details of post-filter205ofFIG. 2are shown. The inputs to postfilter205are multichannel speech signal XSand estimated acoustic ambient noise XN. Post-filter205has a noise reducing filter block408that estimates a Wiener filter transfer function defined by:

HW=ϕs⁢⁢sϕs⁢⁢s+ϕn⁢⁢n
where φss=E(ss*) is short time speech power given s as the speech signal, and φnn=E(nn*) is short time noise power, given n as the instantaneous noise. Filter block408receives multichannel speech signal XSand generates reduced-noise multi-channel speech signal XW. Both φssand φnn, which are provided from computation blocks406and407, respectively, are estimated from both of multichannel speech signal XSand estimated acoustic ambient noise XN. The short-term power Φxsof multichannel speech signal XScan be modeled by:
φxs=E(XSX*S)=φss+φnn
where φss=E(ss*) is short-term power of the speech component in multichannel speech signal XS, and φnn=E(nn) is the short-term power of the noise component in multichannel speech signal XS. The short-term power of estimated acoustic ambient noise XNcan be modeled by:
φxn=E(XNXN*)=αsφss+αnφnn,αs<<αn
Speech is highly attenuated in signal XN, αs<<1 while the noise power attenuation is partly compensated by gain factor g(t). Therefore, αn≈1. With the assumption that φxs, φxn, αsand αnare known, then the short-term power of the speech and noise can be reduced to:

ϕs⁢⁢s=ϕxn-αn⁢ϕxsαs-αn,ϕnn=αs⁢ϕxs-ϕxnαs-αn,
which are computed by computation blocks406and407, respectively. Since values φxnand φxsare time-varying, they can be estimated by first order IIR filters401and402, respectively, according to:
{circumflex over (φ)}sx(t)=λ{circumflex over (φ)}xs(t−1)+(1−λ)xS*(t)xS(t)
{circumflex over (φ)}xn(t)=λ{circumflex over (φ)}xn(t−1)+(1−λ)xN*(t)xN(t),
where λ=0.99 is an exponential forgetting factor. As αsand αnare unknown, they are estimated using auxiliary variable φaux(t) calculated in divider403as:

ϕa⁢⁢u⁢⁢x⁡(t)=ϕ^x⁢⁢n⁡(t)ϕ^x⁢⁢s⁡(t)
First φaux(t) is processed by a first order IIR filter404according to:
{circumflex over (φ)}aux(t)=λ1{circumflex over (φ)}aux(t−1)+(1−λ1)φaux(t),0<λ1<1,
where λ1is a constant. Then αs, which is the expected value of φaux(t) over the non-speech interval, is estimated by recursive minimum estimation using another IIR filter with two different forgetting factors according to:

α^s⁡(t)={0.9⁢⁢α^s⁡(t-1)+0.1⁢ϕ^a⁢⁢u⁢⁢x⁡(t),for⁢⁢α^s⁡(t-1)<ϕ^a⁢⁢u⁢⁢x⁡(t)0.999⁢⁢α^s⁡(t-1)+0.001⁢ϕ^a⁢⁢u⁢⁢x⁡(t),for⁢⁢α^s⁡(t-1)≥ϕ^a⁢⁢u⁢⁢x⁡(t)
Similarly, αnis estimated by recursive maximum estimation using an IIR filter405with two different forgetting factors according to:

α^n⁡(t)={0.999⁢⁢α^n⁡(t-1)+0.001⁢ϕ^a⁢⁢u⁢⁢x⁡(t),for⁢⁢α^n⁡(t-1)<ϕ^a⁢⁢u⁢⁢x⁡(t)0.9⁢⁢α^n⁡(t-1)+0.1⁢ϕ^a⁢⁢u⁢⁢x⁡(t),for⁢⁢α^n⁡(t-1)≥ϕ^a⁢⁢u⁢⁢x⁡(t)
At output of the filters404and405there are estimates of αsand αn, respectively. By providing αsand αnas inputs to each of computation blocks406and407, estimates of speech and noise powers φssand φnnare obtained at their respective outputs. Noise powers φssand φnnare then used to estimate the Wiener filter, as noted above.

Referring now toFIG. 5, details of f0estimation block207ofFIG. 2are shown. A bandpass filter501limits the frequency range of microphone signal XMLto a frequency range of approximately 70 Hz to 1000 Hz. The output of bandpass filter is partitioned into overlapping segments 43 ms wide and a window function is applied by block502. A fast-fourier transform503is performed on the output of window function and an autocorrelation module504computes the autocorrelation of the windowed and bandlimited microphone signal XML. A compensation filter505compensates for the influence of the window function, e.g., longer autocorrelation lag in windowed and bandlimited microphone signal XML, and then multiple candidates for fundamental frequency f0are tested by selection of local minima, computation of local strength and computation of a transition cost associated with every candidate. Finally for a dynamic programming algorithm module507selects the best candidate and estimates fundamental frequency f0.

Referring now toFIG. 6, details of CASA module206ofFIG. 2are shown. CASA module206has two stages and determines three masks at the first stage. A segment mask is computed from reduced-noise multichannel speech signal XWby a segment mask computation block601. A target mask is computed by estimated fundamental frequency f0and reduced-noise multichannel speech signal XWand an onset-offset mask is also computed from reduced-noise multi-channel speech signal XW. The three first-stage masks are combined into a unique final mask in final mask calculation module604. The final mask is used for speech enhancement and suppression of interference in a speech synthesis module605that generates fully processed speech signal XOUT. Synthesis of speech from masked channel signals is performed using a time alignment method, without requiring computation intensive FIR filtering. The total analysis/synthesis delay time in the depicted embodiment is 4 ms, which in mobile phone applications is a desirably short delay.

The output of target mask computation block602is 64-channel vector of binary decisions of whether the time-frequency elements of reduced-noise multi-channel speech signal XWcontain a component of estimated fundamental frequency f0. An autocorrelation is calculated for each channel using a delay that corresponds to the estimated f0. The autocorrelation value is normalized by signal power and compared to a threshold. If the resultant value exceeds a predefined threshold, the decision is one (true), otherwise the decision is zero (false). For the channels of reduced-noise multi-channel speech signal XWhaving a center frequency greater than 800 Hz, the autocorrelation function is calculated on a complex envelope, which reduces the influence of the residual noise on the mask estimation.

Segment mask computation block601computes a measure of similarity of spectra in neighboring channels of reduced-noise multi-channel speech signal XW. Since the formant structure of speech spectra concentrates signal around formants, non-formant interferences can be identified on the basis of rapid changes in power of adjacent channels. Typical segment mask computation techniques use autocorrelation, which is computation intensive. While such techniques may be used in certain embodiments of the present invention, according to the exemplary embodiment described herein, a spectral distance measure that does not use autocorrelations is employed. A correlation index is calculated using time-domain waveform data on the channels of reduced-noise multi-channel speech signal XWthat have a center frequency below 800 Hz. For channels having a central frequency over 800 Hz, an amplitude envelope of the complex signal is used to compute the correlation index calculation according to the following:

Dc⁡(t,fi,fi+1)=∑n=0N-1⁢x~W⁡(t-n,fi)⁢x~W⁡(t-n,fi+1)∑n=0N-1⁢x~W⁡(t-n,fi)⁢x~W⁡(t-n,fi)⁢∑n=0N-1⁢x~W⁡(t-n,fi+1)⁢x~W⁡(t-n,fi+1),
where DCis the spectral distance measure, N is the number of samples, and fi, fi+1the center frequencies of two adjacent channels. The segment mask is a real-valued number between zero and one. Unlike autocorrelation-based spectral measures that are insensitive to phase difference between neighboring channels, the spectral measure of the exemplary embodiment is sensitive to the phase differences of neighboring channels.

Onset-offset mask computation block603separates speech segments from background noise using a time-frequency model that has a rapid increase in signal energy indicating the beginning of a speech interval that then ends with fall of the signal energy below the noise floor. The ambient acoustic noise may be stationary as a fan-noise which has no onset and offset, which can be easily separated from speech using the above-described time-frequency model. Also, ambient acoustic noise may be non-stationary, for example the sound of a ball bouncing against a gym floor. In the non-stationary case, a rule for the segment length is used to separate speech from noise.

While reduced-noise multi-channel speech signal XWis used for mask calculation in CASA module206, multi-channel signal Xs is used for speech synthesis. Using multi-channel signal Xs as the basis for output speech synthesis instead of reduced-noise multi-channel speech signal XWprevents double filtering and possibility of the speech distortion due to the double filtering as CASA module206interacts with the filtering action in post-filter205.

Referring now toFIG. 7, details of onset-offset mask computation block603ofFIG. 6are depicted. Onset-offset mask computation block603identifies speech segments that begin with an onset and end with an offset. A segment energy estimation block estimates the energy in the channels of reduced-noise multi-channel speech signal XW, and in the exemplary embodiment, are calculated on segments 64 samples long. Next, the energy estimates are low-pass filtered in time by a time filtering block702and across the channels by a frequency filtering block703. Time derivatives of low-pass filtered (smoothed) energy values are used to enhance rapid changes in signal power and are computed by a differentiation block704. Onset/offset detection is performed on the output of differentiation block704in an onset-offset detection module705. If the time derivative of the smoothed energy values exceed the onset threshold, onset is detected. Onset-offset detection module705then searches for the offset. When the time derivative of the smoothed energy falls below the offset threshold, offset is detected. Certain rules have been imposed in the exemplary embodiment that have produced enhanced results:1. Speech segments are not permitted to be less than 40 ms. Segments less then 40 ms are enlarged to 40 ms.2. The offset threshold is provided as a time-varying value by offset threshold estimation module707. Immediately after an onset, offset threshold is set to a high value to prevent early offset detection. The offset threshold decreases with time to increase the probability of the offset detection. Decrease of the offset threshold prevents long speech segments. Speech segments of the channel signals are alternated with pauses after a change of phoneme. Very long speech segments in channel signals rarely occur in normal speech.3. Onset threshold is estimated by onset threshold estimation module706using ambient noise power determined after offset detection. Accurate noise power estimate provides better estimate of the ideal onset threshold that increases the probability of the onset detection.

Referring now toFIG. 8, details of final mask calculation block604ofFIG. 6are depicted. Final mask calculation block604calculates a final mask on basis of the target, segment and onset/offset masks described above. The target and segment masks are used to form an auxiliary mask at output of auxiliary mask computation module801. A union mask is formed at output of a union mask computation module802from the onset/offset and the auxiliary mask. The union mask is real valued. The union mask requires some post-processing due to non-zero element groups that have too few time-frequency (TF) units due to mis-estimation of the frequency width and duration of the speech segment. Therefore, segment grouping module803searches for groups having less than eight TF units and sets them to zero to further suppress noise. The output of segment grouping module803is a final mask that is used for speech synthesis by speech synthesis module605ofFIG. 6.