System for distinguishing desired audio signals from noise

A system distinguishes a primary audio source and background noise to improve the quality of an audio signal. A speech signal from a microphone may be improved by identifying and dampening background noise to enhance speech. Stochastic models may be used to model speech and to model background noise. The models may determine which portions of the signal are speech and which portions are noise. The distinction may be used to improve the signal's quality, and for speaker identification or verification.

PRIORITY CLAIM

This application claims the benefit of priority from European Patent Application No. 07021933.2, filed Nov. 12, 2007, which is incorporated by reference.

BACKGROUND OF THE INVENTION

1. Technical Field

This disclosure is related to a speech processing system that distinguishes background noise from a primary audio source for speech recognition and speaker identification/verification in noisy environments.

2. Related Art

Speech recognition may confirm or reject speaker identities. When recognizing speech, the audio that includes the speech is processed to identify high-quality speech signals, rather than background noise. Speech signals detected by microphones may be distorted by background noise that may or may not include speech signals of other speakers. Some systems may not distinguish sound from a primary source, such as a foreground speaker, from background noise.

SUMMARY

A system distinguishes a primary audio source, such as a speaker, from background noise to improve the quality of an audio signal. A speech signal from a microphone may be improved by identifying and dampening background noise to enhance speech. Stochastic models may be used to model speech and to model background noise. The models may determine which portions of the signal are speech and which portions are noise. The distinction may be used to improve the signal's quality, and for speaker identification or verification.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Speech recognition and speaker identification/verification may utilize segmentation of detected verbal utterances to discriminate or distinguish between speech and non speech (e.g., significant speech pause segments). The temporal evolution of microphone signals comprising both speech and speech pauses may be analyzed. For example, the energy evolution in the time or frequency domain of the signal may be analyzed. Abrupt energy drops may indicate significant speech pauses. However, background noise or perturbations with energy levels that are comparable to the ones of the speech contribution to the microphone signal may be recognized in the signal as speech, which may result in a deterioration of the microphone signal. Utilizing the pitch and/or other associated harmonics may also be used for identifying speech passages and distinguishing background noise that may have a high-energy level. However, perturbations that include both non-verbal and verbal noise/perturbations (also known as “babble noise”) may not be detected. For example, those perturbations may be relatively common in the context of conference settings, meetings and product presentations, e.g., in trade shows. The use of stochastic models for the primary audio source, such as the speaker, and stochastic models the secondary audio, such as any background noise, may distinguish the desirable audio from the audio signal. The stochastic models may be combined with energy and/or pitch analysis for speech recognition, or speaker identification and verification.

FIG. 1is a recording environment in which a microphone102may receive an audio input signal104. The microphone102may be any device or instrument for receiving or measuring sound. The microphone102may be a transducer or sensor that converts sound/audio into an operating signal that is representative of the sound/audio at the microphone. The microphone102receives the audio input signal104. The audio input signal104may include any acoustic signals or vibrations that may be detected when the signal lie in an aural range. The audio input signal104may be characterized by wave properties, such as frequency, wavelength, period, amplitude, speed, and direction. These sound signals may be detected by the microphone102or an electrical or optical transducer. The audio input signal104may include audio or sound from a primary source106. The primary source106may include a foreground speaker or other intended source of audio. For simplicity, the primary source106may be described as a speaker and the primary source audio may be described as a speech signal, however, the primary source106may include sound emissions other than just a speaker. The system determines audio from the primary source106by identifying all other audio from the audio input signal104. The other audio may include other speakers112, such as background or unintended speakers. Likewise, background noise108and other sounds110, such as perturbations may also be part of the audio input signal104. As described, background audio, background sound, or background noise may be used to describe and include any audio (including other speakers/sounds) other than audio from the primary source106.

FIG. 2is a system for analyzing audio. The microphone102receives audio from the primary source106, as well as background audio202. The microphone102generates a microphone signal from the received audio. The microphone signal may include speech and no speech portions. In both signal portions background audio, such as perturbations, may be present. The microphone signal is passed to an audio analyzer204. The audio analyzer204may be a computing device that receives and analyzes audio signals as shown inFIG. 5. As described below, the audio analyzer204may analyze the microphone signal and distinguish audio from the primary source106from the background audio202. This distinction may be used to produce the output208.

FIG. 3is an audio analysis system illustrating the output208from the audio analyzer204. The output208may include speech recognition302, speaker identification304, speaker verification306, and/or enhanced audio308. Speech recognition302may include identifying the words that are spoken into the microphone. Speaker identification304may include determining the identity of a speaker based on the speech received by the microphone. Likewise, speaker verification306may include determining the identity of a speaker for verification. In some systems, an additional self-learning speaker identification system may enable the unsupervised stochastic modeling of unknown speakers and the recognition of known speakers, such as is described in commonly assigned U.S. patent application Ser. No. 12/249,089, entitled “Speaker Recognition System,” filed on Oct. 10, 2008, the entire disclosure of which is incorporated by reference.

The distinction determined by the audio analyzer204may also be used for generating enhanced audio308. In particular, the audio/speech input into the microphone may include background audio, and after that background audio is distinguished, it may be removed or suppressed to improve the audio from the primary source. Alternatively, after identifying segments of an audio signal from the primary source, those segments may be attenuated by noise reduction filtering means, such as a Wiener filter or a spectral subtraction filter. Conversely, segments of the audio signal that are background audio may be dampened for enhancing the audio.

The audio analyzer204may utilize training data206for distinguishing audio.FIG. 4is exemplary training data206. The training data206may include a primary source stochastic model402and a background audio stochastic model404. As described below with respect toFIG. 7, a stochastic model may characterize the audio. The primary source stochastic model402characterizes the audio from the primary source and the background audio stochastic model404characterizes the background audio. A stochastic model may include a probability analysis in which multiple results may occur because of the presence of a random element. Even if an initial condition is known, the stochastic model may identify multiple possibilities in which some are more probable than others. An audio signal, such as a speech signal, may be modeled with a stochastic model because it fluctuates over time.

The training may be performed off-line on the basis of feature vectors from the primary source and from background audio, respectively. Characteristics or feature vectors may include feature parameters, such as the frequencies and amplitudes of signals, energy levels per frequency range, formants, the pitch, the mean power and the spectral envelope, etc., or other characteristics for received speech signals. The feature vectors may comprise cepstral vectors.

In one example, a stochastic model will be associated with each of a plurality of potential speakers. The stochastic models for each speaker may be used for improving or enhancing the speech from the speaker. Stochastic models for both the utterances of a foreground speaker and the background noise may produce a more reliable segmentation of portions of the microphone signal that contains speech and portions that contain significant speech pauses (no speech) as further discussed below. Significant speech pauses may occur before and after a foreground speaker's utterance. The utterance itself may include short pauses between individual words. These short pauses may be considered part of speech present in the microphone signal. The segmentation that identifies the beginning and end of the foreground speaker's utterance may be utilized for distinguishing the speaker's utterance from background noise.

A stochastic model for the background audio202may comprise a stochastic model for diffuse non-verbal background noise108and verbal background noise due to background speaker112. A stochastic model for the primary source106, which may be a foreground speaker whose utterance corresponds to the wanted signal. The foreground may be an area close (e.g., several meters) to the microphone102used to obtain the microphone signal. Even if a second speaker112is as close to the microphone102as the foreground speaker, the foreground speaker's utterances may be identified through the use of different stochastic models for each speaker.

FIG. 5is an exemplary audio analyzer204. The audio analyzer204may include a processor502, memory504, software506and an interface508. The interface508may include a user interface that allows a user to interact with any of the components of the audio analyzer204. For example, a user may modify or provide the stochastic models that are used by the audio analyzer204to distinguish audio from the primary source. In one example, data that is used for determining stochastic models, as well as parameters of those models may be stored in a database510. In some systems, the database510may be a part of or the same as the memory504.

The processor502in the audio analyzer204may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP) or other type of processing device. The processor502may be a component in any one of a variety of systems. For example, the processor502may be part of a standard personal computer or a workstation. The processor502may be one or more general processors, digital signal processors, application specific integrated circuits, field programmable gate arrays, servers, networks, digital circuits, analog circuits, combinations thereof, or other now known or later developed devices for analyzing and processing data. The processor502may operate in conjunction with a software program, such as code generated manually (i.e., programmed).

The processor502may communicate with a local memory504, or a remote memory504. The interface508and/or the software506may be stored in the memory504. The memory504may include computer readable storage media such as various types of volatile and non-volatile storage media, including to random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media and the like. In one system, the memory504includes a random access memory for the processor502. In alternative systems, the memory504is separate from the processor502, such as a cache memory of a processor, the system memory, or other memory. The memory504may be an external storage device, such as the database510, for storing audio data, model parameters, model data, etc. Examples include a hard drive, compact disc (“CD”), digital video disc (“DVD”), memory card, memory stick, floppy disc, universal serial bus (“USB”) memory device, or any other device operative to store data. The memory504is operable to store instructions executable by the processor502.

The functions, acts or tasks illustrated in the figures or described here may be processed by the processor executing the instructions stored in the memory504. The functions, acts or tasks are independent of the particular type of instruction set, storage media, processor or processing strategy and may be performed by software, hardware, integrated circuits, firm-ware, micro-code and the like, operating alone or in combination. Processing strategies may include multiprocessing, multitasking, or parallel processing. The processor502may execute the software506that includes instructions that analyze audio signals.

The interface508may be a user input device or a display. The interface508may include a keyboard, keypad or a cursor control device, such as a mouse, or a joystick, touch screen display, remote control or any other device operative to interact with the audio analyzer204. The interface508may include a display that communicates with the processor502and configured to display an output from the processor502. The display may be a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, a cathode ray tube (CRT), a projector, a printer or other now known or later developed display device for outputting determined information. The display may act as an interface for the user to see the functioning of the processor502, or as an interface with the software506for providing input parameters. In particular, the interface508may allow a user to interact with the audio analyzer204to generate and modify models for audio data received from the microphone102.

FIG. 6is another audio analysis system. A microphone array602may replace the microphone102discussed above. In particular, the microphone array602may comprise a plurality of microphones102that each measure and/or receive audio signals. A beamformer604may be coupled with the microphone array602for improving the measured audio. The beamformer604may be utilized for steering the microphone array602to the direction of the primary source106or foreground speaker. The microphone signal from the microphone array602may represent a beamformed microphone signal that may be analyzed by the audio analyzer204.

The beamforming may be performed by a “General Sidelobe Canceller” (GSC). The GSC may include two signal processing paths: a first (or lower) adaptive path with a blocking matrix and an adaptive noise cancelling means and a second (or upper) non-adaptive path with a fixed beamformer. The fixed beamformer may improve the signals pre-processed, e.g., by a means for time delay compensation using a fixed beam pattern. Adaptive processing methods may be characterized by an adaptation of processing parameters such as filter coefficients during operation of the system. The lower signal processing path of the GSC may be optimized to generate noise reference signals used to subtract the residual noise of the output signal of the fixed beamformer. The lower signal processing means may comprise a blocking matrix that may be used to generate noise reference signals from the microphone signals. Based on these interfering signals, the residual noise of the output signal of the fixed beamformer may be subtracted applying some adaptive noise cancelling means that employs adaptive filters.

The distinction or discrimination of the primary source106audio (such as a foreground speaker) from the background audio202may include stochastic models and assigning scores to feature vectors from the microphone signal as discussed below. The score may be determined by assigning the feature vector to a class of the stochastic models. If the score for assignment to a class of the primary source stochastic speaker model exceeds a predetermined limit, the associated signal portion may be determined to be from the primary source. In particular, a score may be assigned to feature vectors extracted from the microphone signal for each class of the stochastic models, respectively. Scoring of the extracted feature vectors may provide a method for determining signal portions of the microphone signal that include audio from the primary source.

FIG. 7is an exemplary process for distinguishing speech in a microphone signal. An audio signal is detected by a microphone in block702. The microphone signal may include a verbal utterance by a speaker positioned near the microphone and may also include background audio. The background audio may include diffuse non-verbal noise and babble noise, as well as utterances by other speakers. The other speakers may be positioned away from the microphone or further away than the foreground speaker. The microphone signal may be obtained by one or more microphones, in particular, a microphone array steered to the direction of the foreground speaker. In the case of a microphone array, the microphone signal obtained in block702may be a beamformed signal as discussed with respect toFIG. 6.

From the microphone signal obtained in block702ofFIG. 1one or more characteristic feature vectors may be extracted from the audio signal. According to one example, Mel-frequency cepstral coefficients (MFCCs) may be determined. In particular, the digitized microphone signal y(n) (where n is the discrete time index due to the finite sampling rate) is subject to a Short Time Fourier Transformation employing a window function, e.g., the Hann window, in order to obtain a spectrogram. The spectrogram represents the signal values in the time domain divided into overlapping frames, weighted by the window function and transformed into the frequency domain. The spectrogram may be processed for noise reduction by the method of spectral subtraction, i.e., by subtracting an estimate for the noise spectrum from the spectrogram of the microphone signal, as known in the art. The spectrogram may be supplied to a Mel filter bank modeling the MEL frequency sensitivity of the human ear and the output of the Mel filter bank is logarithmized to obtain the cepstrum in block704for the microphone signal y(n). The obtained spectrum may show a strong correlation in the different bands due to the pitch of the speech contribution to the microphone signal y(n) and the associated harmonics. Therefore, a Discrete Cosine Transformation applied to the cepstrum may obtain the feature vectors x as in block706. The feature vectors may comprise feature parameters, such as the formants, the pitch, the mean power and the spectral envelope.

At least one stochastic primary source model and at least one stochastic model for background audio are used for determining speech parts in the microphone signal. These models may be trained off-line in blocks714,716. The training may occur before the signal processing is performed. Training may include preparing sound samples that can be analyzed for feature parameters as described above. For example, speech samples may be taken from a plurality of speakers positioned close to a microphone used for taking the samples in order to train a stochastic speaker model.

In some systems, Hidden Markov Models (HMM) may be used. HMM may be characterized by a sequence of states each of which has a well-defined transition probability. If speech recognition is performed by HMM, in order to recognize a spoken word, a likely sequence of states through the HMM may be computed. This calculation may be performed by the Viterbi algorithm, which may iteratively determine the likely path through the associated trellis.

Alternatively, in some systems, Gaussian Mixture Models (GMM) may be used. GMM may model transition probabilities and may improve the modeling of feature vectors that are expected to be statistically independent from one another. A GMM may include N classes each consisting of a multivariate Gauss distribution Γ{x|μ, Σ} with the average μ and the covariance matrix Σ. A probability density of a GMM may be given by

p(x⁢λ)=∑i=1N⁢wi⁢Γ⁢{x⁢μi,Σi}
with the a priori probabilities p(i)=wi(weights), with

For the GMM training of both the stochastic primary source model in block714and the stochastic background audio model in block716the Expectation Maximization (EM) algorithm or the K-means algorithm may be used. Starting from an arbitrary initial parameter set comprising, e.g., equally Gaussian distributed weights wiand arbitrary feature vectors as the means pi with covariant unit matrices, feature vectors of training samples may be assigned to classes of the initial models by means of the EM algorithm, i.e. by means of a posteriori probabilities, or the K-means algorithm according to the least Euclidian distance. The iterative training of the stochastic models may include the parameter sets of the models are estimated and adopted for the new models until a predetermined abort criterion is fulfilled. In some systems, one or more speaker-independent, Universal Speaker Model (USM), or speaker-dependent models may be used. The USM may serve as a template for speaker-dependent models generated by an appropriate adaptation as discussed below.

One speaker-independent stochastic speaker model for the primary source may be characterized by λUSMand one stochastic model for the background audio (the Diffuse Background Model (DBM)) may characterized by λDBM. A total model including the parameter set of both models may be formed λ={λUSM, λDBM}. The total model may be used to determine scores SUSM, as in block708, for each of the feature vectors xtextracted in block706from the MEL cepstrum. In this context, t denotes the discrete time index. In some systems, the scores may be calculated by the a posteriori probabilities representing the probability for the assignment of a given feature vector xtat a particular time to a particular one of the classes of the total model for given parameters λ, where indices i and j denote the class indices of the USM and DBM, respectively:

p⁡(i|xt,λ)=wUSM,i⁢Γ⁢{xt|μUSM,i,ΣUSM,i}∑i⁢wUSM,i⁢Γ⁢{xt|μUSM,i,ΣUSM,i}+∑j⁢wDBM,j⁢Γ⁢{xt|μDBM,j,ΣDBM,j}
in the form of

With the likelihood function

p⁡(xt,λ)=∑i⁢wi⁢Γ⁢{xt|μi,Σi},
the above formula may be re-written as

S~USM⁡(xt)=11+exp⁡(αln⁢⁢p⁢(xt|λDBM)-βln⁢⁢p⁡(xt|λUSM)+γ));0≤S~USM⁡(xt)≤1
in order to weight scores in a particular range (damp or raise scores) or to compensate for some biasing. Such a modification (smoothing) may be carried out for each frame to avoid a time delay and for real time processing as in block710. In some systems, the scoring may occur only for those classes that show a likelihood for exceeding a suitable threshold for a respective frame.

The smoothing in block710may be performed to avoid outliers and strong temporal variations of the sigmoid. The smoothing may be performed by an appropriate digital filter, e.g., a Hann window filter function. In some systems, the time history of the above described score may be divided into very small overlapping time windows and an average value may be determined adaptively, along with a maximum value and a minimum value of the scores. A measure for the variations in a considered time interval (represented by multiple overlapping time windows) may be given by the difference of maximum to minimum values. This difference may be subsequently subtracted (after some appropriate normalization in some systems) from the average value to obtain a smoothed score for the primary source as in block710.

Based on the scores (with or without the smoothing in block710) primary source audio from the microphone signal may be determined in block712. Depending on whether the determined scores exceed or fall below a predetermined threshold L the audio in question may be from the primary source or from background audio. In some systems, when the audio is from the primary source, such as a speaker, the score for that audio signal exceeds the threshold L. For example, a binary mapping may be employed for the detection of primary source audio activity

FSAD⁡(xt)={1,if⁢⁢S~USM⁡(xt)≥L0,else..
Short speech pauses between detected speech contributions may be considered part of the speech from the primary source. A short pause between two words of a command uttered by the foreground speaker, e.g., “Call XY”, “Delete z”, etc., may be passed by the segmentation between speech and no speech.

Some systems may relate to a singular stochastic primary source model and a singular stochastic model for background audio. In alternative systems, a plurality of models may be employed, respectively. In some systems, the plurality of stochastic models for the background audio may be used to classify the background audio present in the microphone signal. K models for different types of background audio (perturbances) may be trained in combination with a singular primary source speaker model λ={λUSM, λ1, . . . , λK}. Accordingly, the above formulae may read

SUSM⁡(xt)=∑i⁢wUSM,i⁢Γ⁢{xt|μUSM,i,ΣUSM,i}∑i⁢wUSM,i⁢Γ⁢{xt|μUSM,i,ΣUSM,i}+∑k=1K⁢∑j⁢wk,j⁢Γ⁢{xt|μk,j,Σk,j}
and

The characteristics of the sigmoid may be controlled by parameters, namely, α, β and γ as described above and δk, k=1, . . . , K for weighting the individual models for perturbations characterized by λk

In some systems, speaker-dependent stochastic speaker models may be used additionally or in place of the above-mentioned USM in order to perform speaker identification or speaker verification. Therefore, each of the USM's is adapted to a particular foreground speaker. Exemplary methods for speaker adaptation may include the Maximum Likelihood Linear Regression (MLLR) and the Maximum A Priori (MAP) methods. The latter may represent a modified version of the EM algorithm. According to the MAP method, starting from a USM the a posteriori probability

p⁡(i|xt,λ)=wi⁢Γ⁢{xt|μi,Σi}∑i=1N⁢wi⁢Γ⁢{xt|μi,Σi}
may be calculated. According to the a posteriori probability, the extracted feature vectors may be assigned to classes for modifying the model. The relative frequency of occurrence ŵ of the feature vectors in the classes that they are assigned to may be calculated as well as the means {circumflex over (μ)} and covariance matrices {circumflex over (Σ)}. These parameters may be used to update the GMM parameters. Adaptation of only the means μiand the weights wimay be utilized to avoid problems in estimating the covariance matrices. With the total number of feature vectors assigned to a class i,

ni=∑t=1T⁢p⁡(i|xt,λ),
one obtains

The new GMM parameterswiandμimay be obtained from the previous ones (according to the previous adaptation) and the above ŵiand {circumflex over (μ)}i. This may be achieved by employing a weighting function such that classes with less adaptation values may be adapted slower than classes to which a greater number of feature vectors are assigned:

w_i=wi⁡(1-αi)+w^i⁢αi∑i=1N⁢(wi⁡(1-αi)+w^i⁢ai)μ_i=μi⁡(1-αi)+μ^i⁢αi
with predetermined positive real numbers

αi=nini+const.
that are smaller than 1.

The system and process described may be encoded in a signal bearing medium, a computer readable medium such as a memory, programmed within a device such as one or more integrated circuits, one or more processors or processed by a controller or a computer. If the methods are performed by software, the software may reside in a memory resident to or interfaced to a storage device, synchronizer, a communication interface, or non-volatile or volatile memory in communication with a transmitter. A circuit or electronic device designed to send data to another location. The memory may include an ordered listing of executable instructions for implementing logical functions. A logical function or any system element described may be implemented through optic circuitry, digital circuitry, through source code, through analog circuitry, through an analog source such as an analog electrical, audio, or video signal or a combination. The software may be embodied in any computer-readable or signal-bearing medium, for use by, or in connection with an instruction executable system, apparatus, or device. Such a system may include a computer-based system, a processor-containing system, or another system that may selectively fetch instructions from an instruction executable system, apparatus, or device that may also execute instructions.