Patent Description:
Noise suppression algorithms for voice communications have been effectively implemented on edge devices, such as phones, laptops and conferencing systems. A common problem with bi-directional voice communications is that the background noise at each user's location is transmitted with the user's voice signal. If the signal-to-noise ratio (SNR) of the combined signal received at the edge device is too low, the intelligibility of the reconstructed voice will be degraded resulting in a poor user experience.

Document <CIT> relates to method and a system of noise suppressing an audio signal comprising a combination of at least two audio system input signals each having a sound source signal portion and a background noise portion, the method and system comprising steps and means of: Extracting at least two different types of spatial sound field features from the input signals such as discriminative speech and/or background noise features, computing a first intermediate spatial noise suppression gain on the basis of the extracted spatial sound field features, computing a second intermediate stationary noise suppression gain, combining the two intermediate noise suppression gains to form a total noise suppression gain, wherein the two intermediate noise suppression gains are combined by comparing their values and dependent on their ratio or relative difference, determining the total noise suppression gain, applying the total noise suppression gain to the audio signal to generate a noise suppressed audio system output signal. Document <CIT> relates to method of post-processing banded gains for applying to an audio signal, an apparatus to post-processed banded gains, and a tangible computer-readable storage medium comprising instructions that when executed carry out the method. The banded gains are determined by input processing one or more input audio signals. The method includes post-processing the banded gains to generate post-processed gains, generating a particular post-processed gain for a particular frequency band including percentile filtering using gain values from one or more previous frames of the one or more input audio signals and from gain values for frequency bands adjacent to the particular frequency band.

In the drawings, specific arrangements or orderings of schematic elements, such as those representing devices, units, instruction blocks and data elements, are shown for ease of description. However, it should be understood by those skilled in the art that the specific ordering or arrangement of the schematic elements in the drawings is not meant to imply that a particular order or sequence of processing, or separation of processes, is required. Further, the inclusion of a schematic element in a drawing is not meant to imply that such element is required in all embodiments or that the features represented by such element may not be included in or combined with other elements in some implementations.

Further, in the drawings, where connecting elements, such as solid or dashed lines or arrows, are used to illustrate a connection, relationship, or association between or among two or more other schematic elements, the absence of any such connecting elements is not meant to imply that no connection, relationship, or association can exist. In other words, some connections, relationships, or associations between elements are not shown in the drawings so as not to obscure the disclosure. In addition, for ease of illustration, a single connecting element is used to represent multiple connections, relationships or associations between elements. For example, where a connecting element represents a communication of signals, data, or instructions, it should be understood by those skilled in the art that such element represents one or multiple signal paths, as may be needed, to affect the communication.

The same reference symbol used in various drawings indicates like elements.

In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the various described embodiments. It will be apparent to one of ordinary skill in the art that the various described implementations may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits, have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. Several features are described hereafter that can each be used independently of one another or with any combination of other features.

As used herein, the term "includes" and its variants are to be read as open-ended terms that mean "includes, but is not limited to. " The term "or" is to be read as "and/or" unless the context clearly indicates otherwise. The term "based on" is to be read as "based at least in part on. " The term "one example implementation" and "an example implementation" are to be read as "at least one example implementation. " The term "another implementation" is to be read as "at least one other implementation. " The terms "determined," "determines," or "determining" are to be read as obtaining, receiving, computing, calculating, estimating, predicting or deriving. In addition, in the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.

Traditional noise suppression solutions use two or more microphones to capture background noise, where one microphone is closer to the user's mouth and the other is further away. The signals from the two microphones are subtracted to remove the background noise that is common to both signals. This technique, however, does not work with edge devices having a single microphone, or in the case of a mobile phone, the user is shaking or turning the phone while they speak. Other noise suppression algorithms attempt to continuously find a noise pattern in the audio signal and adapt to it by processing the audio frame by frame or block by block, where each block can include two or more frames. These existing adaptive algorithms work well in certain use cases but do not scale to a large variety and variability of background noise.

Recently, deep neural networks have been used to suppress noise in voice communications. These solutions, however, require significant computation power, and are hard to implement on real-time communication systems. The disclosed embodiments use a combination of directionality and machine learning (e.g., deep neural networks) to provide low cost, high quality noise estimation and suppression for voice communication applications.

<FIG> is a block diagram of a machine learning assisted spatial noise estimation and suppression system <NUM>, according to some embodiments. System <NUM> includes filterbank <NUM>, banding unit <NUM>, machine learning classifier <NUM> (e.g., a neural network such as a deep neural network (DNN)), direction detection unit <NUM>, speech/noise directionality model <NUM>, noise level model <NUM>, noise suppression gain unit <NUM>, multiplying unit <NUM> and inverse filterbank <NUM>. In some embodiments, filterbank <NUM> (e.g., a short-time Fourier transform (STFT)) receives a time-domain input audio signal and converts the time domain input audio signal into subbands, each subband having a different frequency spectra (e.g., time/frequency tiles). The subbands of each block/frame are input into banding unit <NUM> which combines the subbands of the block/frame into bands according to a psychoacoustic model and outputs banded power <NUM> (e.g. banded power expressed in dB). The subbands output by filterbank <NUM> are also input into direction detection unit <NUM>, which generates and outputs microphone covariance vector <NUM>. In some embodiments, there is one covariance vector for each band and a common banding matrix is used in banding unit <NUM> direction detection unit <NUM>. In some implementations, banding unit <NUM> and direction detection unit <NUM> are a single unit which produces a band power in decibels and a covariance vector for each of the bands.

Banded power <NUM> is input into a machine learning classifier <NUM>. In an embodiment, machine learning classifier <NUM> is a pre-trained neural network classifier that estimates and outputs probabilities <NUM> of a plurality of classes, including but not limited to a speech class, a stationary noise class, a nonstationary noise class and a reverberation class for each block/frame and each band. In some embodiments, the stationary noise class is used to drive speech/noise directionality model <NUM> and noise level model <NUM>.

In the disclosure that follows, it assumed that the machine learning classifier estimated and outputs probabilities for speech and stationary noise. In an embodiment where the machine learning classifier outputs a probability of reverberation, the estimated probabilities of speech, stationary noise, nonstationary noise and reverberation are used to separate the reverberation from the noise. In some embodiments, any time-frequency tiles classified as reverberation are added neither to the speech/noise directionality model <NUM> nor noise level model <NUM>.

Probabilities <NUM> are input into speech/noise directionality model <NUM> together with the microphone covariance vectors <NUM>. Probabilities <NUM> are also input into speech/noise level model <NUM> together with banded power <NUM>. For each band and each block/frame, speech/noise directionality model <NUM> estimates and outputs a respective mean and/or covariance <NUM> for speech and noise for each band, based on microphone covariance vectors <NUM>. Noise level model <NUM> estimates and outputs a mean and variance <NUM> of noise power for each band for each block/frame. In some embodiments, noise level model <NUM> also outputs a mean and variance of speech.

Noise suppression gain unit <NUM> receives outputs <NUM>, <NUM> of speech/noise directionality model <NUM> and noise level model <NUM>, respectively, banded power <NUM> and microphone covariance vectors <NUM>. Noise suppression gain unit <NUM> calculates and outputs suppression gains, which are used by multiplying unit <NUM> to scale the subbands output by filterbank <NUM> to suppress the noise in each subband. The subbands are then converted to a time domain output audio signal by inverse filterbank <NUM>. Each of the components of system <NUM> will now be described in further detail below. In some embodiments, the output of machine learning classifier <NUM> is directly used as a noise gain or to calculate noise suppression gains.

As described above, machine learning classifier <NUM> takes banded power <NUM> as input and provides as output probabilities <NUM> of speech and noise given by: <MAT> where t is the block/frame number and f is the band index, the subscript "s" represents speech and the subscript "n" represents noise. In some embodiments, the noise is stationary noise, based on the assumption there is one talker of interest who does not move around a lot, and that the noise field is therefore relatively stationary. If these assumptions are violated, the modelling performed by <NUM> and <NUM> may be inappropriate. In some embodiments, the output of machine learning classifier <NUM> could also include probabilities of non-stationary noise and reverberation and noise suppression gains that can be directly used by noise suppression gain unit <NUM>.

In the illustrated example, the machine learning classifier <NUM> comprises a neural network (NN). Input to example NN <NUM> includes <NUM> banded powers of the current frame and an input linear dense layer maps the banded powers to <NUM> features. The <NUM> features go through a set of GRU layers, each of which contains <NUM> hidden units, and at last a dense layer with some nonlinear kernel. NN <NUM> outputs the relative weights for each class and for each band which are converted to probabilities using a softmax function. NN <NUM> is trained using labeled voice with noise mixed and using cross entropy as the cost function. In an embodiment, an Adam optimizer is used for the training the neural network. Other examples of machine learning classifiers include but are not limited to k-nearest neighbors, support vector machine or decision tree.

In an embodiment, NN <NUM> can be trained as follows:.

In some embodiments, filterbank <NUM> is implemented using a short-time Fourier transform (STFT). Let Xt(k) be the kth bin data (subband) of STFT of block/frame t for all microphone inputs. Xt(k) is a vector of length N, where N is the number of microphones, M is the number of subbands and α is a weighting factor weighting contributions of past and estimated covariances. In real-time audio processing, the microphone covariance of band f is calculated by: <MAT> where <MAT> where B(f) is the set of all STFT bins (i.e., subbands) that belong to bandf.

Note that Equations [<NUM>] and [<NUM>] hold for "rectangular banding", in which each subband contributes to exactly one output band with a gain of <NUM>. In general, there is some weight wkf in the range [<NUM>, <NUM>] which describes how much input subband k contributes to output band f For one subband k all the weights wkf must sum to <NUM> over all the bands f In this way arbitrary-shaped banding can be performed. For example banding that is cosine or triangular shaped in linear, log or Mel frequency can be used in addition to rectangular banding. <MAT> <MAT>.

In an embodiment, the banded power can be computed directly from filterbank <NUM>. For example, a non-uniform filterbank that has the banding scheme embedded. In such an embodiment, decimation is not used and a block rate defines the block frame t.

To simplify notation, a normalized covariance matrix is rearranged to be a real vector <NUM>, vt,f, because the covariance matrix is Hermitian and its diagonal elements are real. A normalized covariance is the covariance matrix divided by its trace so that it represents direction only, with any level component removed.

If an element of this covariance matrix is denoted as cm,n, a re-arranged vector for a <NUM> microphone system is given by, where m and n index the covariance matrix elements: <MAT>.

In some embodiments, vt,f is normalized. Systems with more or fewer microphones would have more or fewer elements, and thus the system is scalable to systems with any number of microphones.

In some embodiments, speech/noise directionality model <NUM> takes covariance vectors (one per band), vt,f, and probabilities <NUM> output by machine learning classifier <NUM> as input, and estimates means and/or covariance matrix of vt,f. There are at least two embodiments for estimating the mean and/or covariance matrix of vt,f, which are described as follows:.

The noise level model <NUM> takes banded power <NUM> (Lt,f) in dB as input, and estimates the mean and variance of the noise for band f and block/frame t: <MAT> <MAT>.

Note that in Equations [<NUM>]-[<NUM>] a time average is used to estimate the mean and covariance of a random variable (either level or directionality) under the assumption that the random process is ergodic and is stationary for a period of time. The average is achieved using a first order low pass filter model, where, for each frame, the input to the low pass filter (mean or covariance) is weighted by the probability of speech or noise.

<FIG> is a diagram illustrating a noise suppression gain, GL(b, f), calculation based on noise level model <NUM>, according to some embodiments. The vertical axis is gain in dB and the horizontal axis is level in dB In <FIG>, G<NUM> is the maximum suppression gain, and the slope of the gain ramp, β, and k are tuning parameters: <MAT>.

If the current signal to noise level is greater than a pre-defined signal to noise ratio threshold (See Equation [<NUM>]), the noise suppression gain is GL, otherwise, the speech/noise directionality model <NUM> is used to calculate GS. The system calculates the probability of speech for band f and for block/frame t using one of at least two methods:.

<FIG> is a diagram illustrating a noise suppression gain calculation based on speech/noise directionality model <NUM>. The vertical axis is gain in dB, the horizontal axis is the probability of speech, and the noise suppression gain is given by: <MAT> where γ and ks are tuning parameters.

The final suppression gain G(t, f) is calculated (e.g., by noise suppression gain unit <NUM>) for each band and f each block/frame t as: <MAT> where SNR<NUM> is a pre-defined signal to noise ratio threshold and <MAT> is the estimated signal to noise ratio for block/frame t.

In some embodiments, the estimated <MAT> in [<NUM>] can be obtained by using a VAD (Voice activity detector) output to drive an automatic gain control (AGC) component (not shown) to level the speech signal to a pre-defined power level (in dB), and subtract the estimated noise level µL(t, f) from the pre-defined power level, and estimate the speech level using a similar method as was used estimate noise, where: <MAT> <MAT> is calculated a: <MAT>.

The final suppression gain G(t,f) is used in multiplication unit <NUM> to scale the subbands output by filterbank <NUM> to suppress the noise in each subband. For example, the gain G(t,f) is applied to all subbands k that belong to band f: <MAT> wherein Yt(k) is the output of multiplication unit <NUM>.

The output of multiplication unit <NUM> is then converted to a time domain output audio signal by inverse filterbank <NUM> to arrive at a noise suppressed output signal.

<FIG> and <FIG> are a flow diagram of a process <NUM> noise estimation and suppression in voice communications using directionality and deep neural networks, according to some embodiments. Process <NUM> can be implemented using system <NUM> shown in <FIG>.

Process <NUM> begins receiving an input audio signal comprising a number of blocks/frames (<NUM>). For each block/frame, process <NUM> continues by converting the block/frame into subbands (<NUM>), where each subband has a different spectra than the other subbands, combining he subbands into bands and determining a power in each band (<NUM>), and determining a microphone covariance based on the subbands (<NUM>).

Process <NUM> continues by, for each band and each block/frame, estimating, using a machine learning classifier (e.g., a neural network), respective probabilities of speech and noise (<NUM>), estimating, using a directionality model, a set of means for speech and noise, or a set of means and covariances for speech and noise, based on the microphone covariance and the probabilities (<NUM>), estimating, using a level model, a mean and variance of noise power based on the probabilities and the band powers (<NUM>), calculating a first noise suppression gain based on first outputs of the directionality model (<NUM>), determining a second noise suppression gain based on second outputs of the level model (<NUM>), and selecting one of the first noise suppression gain or the second noise suppression gain, or a sum of the first noise suppression gain and the second noise suppression gain, based on a signal to ratio of the input audio signal (<NUM>).

Process <NUM> continues by scaling each subband in each band by the selected first or second noise suppression gain for the band (<NUM>), and converting the scaled subbands into an output audio signal (<NUM>).

<FIG> shows a block diagram of an example system for implementing the features and processes described in reference to <FIG>, according to an embodiment. System <NUM> includes any devices that are capable of playing audio, including but not limited to: smart phones, tablet computers, wearable computers, vehicle computers, game consoles, surround systems, kiosks.

As shown, the system <NUM> includes a central processing unit (CPU) <NUM> which is capable of performing various processes in accordance with a program stored in, for example, a read only memory (ROM) <NUM> or a program loaded from, for example, a storage unit <NUM> to a random access memory (RAM) <NUM>. In the RAM <NUM>, the data required when the CPU <NUM> performs the various processes is also stored, as required. The CPU <NUM>, the ROM <NUM> and the RAM <NUM> are connected to one another via a bus <NUM>. An input/output (I/O) interface <NUM> is also connected to the bus <NUM>.

The following components are connected to the I/O interface <NUM>: an input unit <NUM>, that may include a keyboard, a mouse, or the like; an output unit <NUM> that may include a display such as a liquid crystal display (LCD) and one or more speakers; the storage unit <NUM> including a hard disk, or another suitable storage device; and a communication unit <NUM> including a network interface card such as a network card (e.g., wired or wireless).

In some implementations, the input unit <NUM> includes one or more microphones in different positions (depending on the host device) enabling capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

In some implementations, the output unit <NUM> include systems with various number of speakers. As illustrated in <FIG>, the output unit <NUM> (depending on the capabilities of the host device) can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).

The communication unit <NUM> is configured to communicate with other devices (e.g., via a network). A drive <NUM> is also connected to the I/O interface <NUM>, as required. A removable medium <NUM>, such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive or another suitable removable medium is mounted on the drive <NUM>, so that a computer program read therefrom is installed into the storage unit <NUM>, as required. A person skilled in the art would understand that although the system <NUM> is described as including the above-described components, in real applications, it is possible to add, remove, and/or replace some of these components and all these modifications or alteration all fall within the scope of the present disclosure.

Aspects of the systems described herein may be implemented in an appropriate computer-based sound processing network environment for processing digital or digitized audio files. Portions of the adaptive audio system may include one or more networks that comprise any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route the data transmitted among the computers. Such a network may be built on various different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

In accordance with example embodiments of the present disclosure, the processes described above may be implemented as computer software programs or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program including program code for performing methods. In such embodiments, the computer program may be downloaded and mounted from the network via the communication unit <NUM>, and/or installed from the removable medium <NUM>, as shown in <FIG>.

Generally, various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits (e.g., control circuitry), software, logic or any combination thereof. For example, the units discussed above can be executed by control circuitry (e.g., a CPU in combination with other components of <FIG>), thus, the control circuitry may be performing the actions described in this disclosure. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device (e.g., control circuitry). While various aspects of the example embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representation, it will be appreciated that the blocks, apparatus, systems, techniques or methods described herein may be implemented in, as nonlimiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

Additionally, various blocks shown in the flowcharts may be viewed as method steps, and/or as operations that result from operation of computer program code, and/or as a plurality of coupled logic circuit elements constructed to carry out the associated function(s). For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program containing program codes configured to carry out the methods as described above.

In the context of the disclosure, a machine readable medium may be any tangible medium that may contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine readable medium may be a machine readable signal medium or a machine readable storage medium. A machine readable medium may be non-transitory and may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

Computer program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus that has control circuitry, such that the program codes, when executed by the processor of the computer or other programmable data processing apparatus, cause the functions/operations specified in the flowcharts and/or block diagrams to be implemented. The program code may execute entirely on a computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server or distributed over one or more remote computers and/or servers.

Claim 1:
A method of audio processing, comprising:
receiving, using at least one processor, bands of power spectra of an input audio signal and a microphone covariance for each band, wherein the microphone covariance is based on a configuration of microphones used to capture the input audio signal;
for each band:
estimating, using a machine learning classifier, respective probabilities of speech and noise;
estimating, using a directionality model, a set of means for speech and noise, or a set of means and covariances for speech and noise, based on the microphone covariance for the band and the probabilities;
estimating, using a level model, a mean and covariance of noise power based on the probabilities and the power spectra;
determining, using the at least one processor, a first noise suppression gain based on the set of means for speech and noise, or the set of means and covariances for speech and noise, respectively, estimated using the directionality model;
determining, using the at least one processor, a second noise suppression gain based on the mean and covariance estimated using the level model;
selecting, using the at least one processor, one of the first noise suppression gain or the second noise suppression gain, or a sum of the first noise suppression gain and the second noise suppression gain, based on a signal to noise ratio of the input audio signal;
scaling, using the at least one processor, a time-frequency representation of the input signal by the selected first or second noise suppression gain or the sum thereof for the band; and
converting, using the at least one processor, the time-frequency representation into an output audio signal.