Patent Description:
In a related art, an intelligent product device mostly adopts a Microphone (MIC) array for sound-pickup, and a MIC beamforming technology is adopted to improve quality of voice signal processing to increase a voice recognition rate in a real environment. However, a multi-MIC beamforming technology is sensitive to a MIC position error, resulting in relatively great influence on performance. In addition, increase of the number of MICs may also increase product cost.

Therefore, more and more intelligent product devices are configured with only two MICs at present. For the two MICs, a blind source separation technology that is completely different from the multi-MIC beamforming technology is usually adopted for voice enhancement. How to improve quality of a voice signal separated based on the blind source separation technology is a problem urgent to be solved at present.

<CIT> provides a selective audio source enhancement system. The system includes a processor and a memory, and a pre-processing unit configured to receive audio data including a target audio signal, and to perform sub-band domain decomposition of the audio data to generate buffered outputs. In addition, the system includes a target source detection unit configured to receive the buffered outputs, and to generate a target presence probability corresponding to the target audio signal, as well as a spatial filter estimation unit configured to receive the target presence probability, and to transform frames buffered in each sub-band into a higher resolution frequency-domain. The system also includes a spectral filtering unit configured to retrieve a multichannel image of the target audio signal and noise signals associated with the target audio signal, and an audio synthesis unit configured to extract an enhanced mono signal corresponding to the target audio signal from the multichannel image.

<NPL> propose a method for underdetermined blind source separation of convolutive mixtures.

<CIT> provides a speech recognition system. The speech recognition system includes: a sound source separating section which separates mixed speeches from multiple sound sources; a mask generating section which generates a soft mask which can take continuous values between <NUM> and <NUM> for each separated speech according to reliability of separation in separating operation of the sound source separating section; and a speech recognizing section which recognizes speeches separated by the sound source separating section using soft masks generated by the mask generating section.

The present invention provides a method and device for processing audio signals, and a storage medium, as set out in the appended independent claims.

Preferred embodiments of the present invention are set out in the appended dependent claims.

The technical solutions provided by the embodiments of the present invention may have the following beneficial effects.

In the embodiments of the present invention, the original noisy signals of the at least two microphones are separated to obtain the respective time-frequency estimated signals of sounds emitted from the at least two sound sources in each microphone, so that preliminary separation may be implemented by use of dependence between signals from different sound sources to separate the sounds emitted from the at least two sound sources in the original noisy signal. Therefore, compared with separating signals from different sound sources by use of a multi-MIC beamforming technology in the related art, this manner has the advantage that positions of these microphones are not required to be considered, so that the audio signals of the sounds emitted from different sound sources may be separated more accurately.

In addition, in the embodiments of the present invention, the mask values of the at least two sound sources in each microphone are obtained based on the time-frequency estimated signals, and the updated time-frequency estimated signals of the sounds emitted from the at least two sound sources are acquired based on the respective original noisy signals of the microphones and the mask values. Therefore, in the embodiments of the present invention, the sounds emitted from the at least two sound sources are further separated according to the original noisy signals and the preliminarily separated time-frequency estimated signals. Moreover, the mask value is a proportion of the time-frequency estimated signal of each sound source in the original noisy signal of each microphone, so that part of bands that are not separated by preliminary separation may be recovered into the audio signals of the corresponding sound sources, voice damage degree of the audio signal after separation may be reduced, and the separated audio signal of each sound source is higher in quality.

It is to be understood that the above general descriptions and detailed descriptions below are only exemplary and explanatory and not intended to limit the present invention.

The implementations set forth in the following description of exemplary embodiments do not represent all implementations consistent with the present invention. Instead, they are merely examples of devices and methods consistent with aspects related to the present invention as recited in the appended claims.

<FIG> is a flow chart showing a method for processing audio signal, according to some embodiments of the invention. As shown in <FIG>, the method includes the following operations.

At block S11, audio signals emitted from at least two sound sources respectively are acquired through at least two MICs to obtain respective original noisy signals of the at least two MICs.

At block S12, sound source separation is performed on the respective original noisy signals of the at least two MICs to obtain respective time-frequency estimated signals of the at least two sound sources.

At block S13, a mask value of the time-frequency estimated signal of each sound source in the original noisy signal of each MIC is determined based on the respective time-frequency estimated signals of the at least two sound sources.

At block S14, the respective time-frequency estimated signals of the at least two sound sources are updated based on the respective original noisy signals of the at least two MICs and the mask values.

At block S15, the audio signals emitted from the at least two sound sources respectively are determined based on the respective updated time-frequency estimated signals of the at least two sound sources.

The method of the embodiment of the present invention is applied to a terminal. Herein, the terminal is an electronic device integrated with two or more than two MICs. For example, the terminal may be a vehicle terminal, a computer or a server. In an embodiment, the terminal may be an electronic device connected with a predetermined device integrated with two or more than two MICs, and the electronic device receives an audio signal acquired by the predetermined device based on this connection and sends the processed audio signal to the predetermined device based on the connection. For example, the predetermined device is a speaker.

During a practical application, the terminal includes at least two MICs, and the at least two MICs simultaneously detect the audio signals emitted from the at least two sound sources respectively to obtain the respective original noisy signals of the at least two MICs. Herein, it can be understood that, in the embodiment, the at least two MICs synchronously detect the audio signals emitted from the two sound sources.

The method for processing audio signal according to the embodiment of the present invention may be implemented in an online mode and may also be implemented in an offline mode. Implementation in the online mode refers to that acquisition of an original noisy signal of an audio frame and separation of an audio signal of the audio frame may be simultaneously implemented. Implementation in the offline mode refers to that audio signals of audio frames in a predetermined time are started to be separated after original noisy signals of the audio frames in the predetermined time are completely acquired.

In the embodiment of the present invention, there are two or more than two MICs, and there are two or more than two sound sources.

In the embodiment of the present invention, the original noisy signal is a mixed signal including sounds emitted from the at least two sound sources. For example, there are two MICs, i.e., a first MIC and a second MIC respectively; and there are two sound sources, i.e., a first sound source and a second sound source respectively. In such case, the original noisy signal of the first MIC includes the audio signals from the first sound source and the second sound source, and the original noisy signal of the second MIC also includes the audio signals from both the first sound source and the second sound source.

For example, there are three MICs, i.e., a first MIC, a second MIC and a third MIC respectively, and there are three sound sources, i.e., a first sound source, a second sound source and a third sound source respectively. In such case, the original noisy signal of the first MIC includes the audio signals from the first sound source, the second sound source and the third sound source, and the original noisy signals of the second MIC and the third MIC also include the audio signals from the first sound source, the second sound source and the third sound source, respectively.

Herein, the audio signal may be a value obtained after inverse Fourier transform is performed on the updated time-frequency estimated signal.

Herein, if the time-frequency estimated signal is a signal obtained by a first separation, the updated time-frequency estimated signal is a signal obtained by a second separation.

Herein, the mask value refers to a proportion of the time-frequency estimated signal of each sound source in the original noisy signal of each MIC.

It can be understood that, if a signal from a sound source is an audio signal in a MIC, a signal from another sound source is a noise signal in the MIC. According to the embodiment of the present invention, the sounds emitted from the at least two sound sources are required to be recovered through the at least two MICs.

In the embodiment of the present invention, the original noisy signals of the at least two MICs are separated to obtain the time-frequency estimated signals of sounds emitted from the at least two sound sources in each MIC, so that preliminary separation may be implemented by use of dependence between signals of different sound sources to separate the sounds emitted from the at least two sound sources in the original noisy signals. Therefore, compared with the solution in which signals from the sound sources are separated by use of a multi-MIC beamforming technology in the related art, this manner has the advantage that positions of these MICs are not required to be considered, so that the audio signals of the sounds emitted from the sound sources may be separated more accurately.

In addition, in the embodiments of the present invention, the mask values of the at least two sound sources with respect to the respective MIC may also be obtained based on the time-frequency estimated signals, and the updated time-frequency estimated signals of the sounds emitted from the at least two sound sources are acquired based on the original noisy signals of each MIC and the mask values. Therefore, in the embodiments of the present invention, the sounds emitted from the at least two sound sources may further be separated according to the original noisy signals and the preliminarily separated time-frequency estimated signals. Moreover, the mask value is a proportion of the time-frequency estimated signal of each sound source in the original noisy signal of each MIC, so that part of bands that are not separated by preliminary separation may be recovered into the audio signals of the respective sound sources, voice damage degrees of the separated audio signals may be reduced, and the separated audio signal of each sound source is higher in quality.

In addition, if the method for processing audio signal is applied to a terminal device with two MICs, compared with the conventional art that voice quality is improved by use of a beamforming technology based on at least more than three MICs, the method also has the advantages that the number of the MICs is greatly reduced, and hardware cost of the terminal is reduced.

It can be understood that, in the embodiment of the present invention, the number of the MICs is usually the same as the number of the sound sources. In some embodiments, if the number of the MICs is smaller than the number of the sound sources, a dimensionality of the number of the sound sources may be reduced to a dimensionality equal to the number of the MICs.

In some embodiments, the operation that the sound source separation is performed on the respective original noisy signals of the at least two MICs to obtain the respective time-frequency estimated signals of the at least two sound sources includes the following actions.

A first separated signal of a present frame is acquired based on a separation matrix and the original noisy signal of the present frame. The separation matrix is a separation matrix for the present frame or a separation matrix for a previous frame of the present frame.

The time-frequency estimated signal of each sound source is obtained by a combination of the first separated signal of each frame.

It can be understood that, when the MIC acquires the audio signal of the sound emitted from the sound source, at least one audio frame of the audio signal may be acquired and the acquired audio signal is the original noisy signal of each MIC.

The operation that the original noisy signal of each frame of each MIC is acquired includes the following actions.

A time-domain signal of each frame of each MIC is acquired.

Frequency-domain transform is performed on the time-domain signal of each frame, and the original noisy signal of each frame is determined according to a frequency-domain signal at a predetermined frequency point.

Herein, frequency-domain transform may be performed on the time-domain signal based on Fast Fourier Transform (FFT). In an example, frequency-domain transform may be performed on the time-domain signal based on Short-Time Fourier Transform (STFT). In an example, frequency-domain transform may also be performed on the time-domain signal based on other Fourier transform.

In an example, if a time-domain signal of an n th frame of the p th MIC is <MAT>, the time-domain signal of the n th frame of is converted into a frequency-domain signal, and the original noisy signal of the n th frame is determined to be: <MAT>, where m is the number of discrete time points of time-domain signal of the n th frame, and k is the frequency point. Therefore, according to the embodiment, the original noisy signal of each frame may be obtained by conversion from a time domain to a frequency domain. Of course, the original noisy signal of each frame may also be acquired based on another FFT formula. There are no limits made herein.

In the embodiment of the present invention, the original noisy signal of each frame may be obtained, and then the first separated signal of the present frame is obtained based on the separation matrix and the original noisy signal of the present frame. Herein, the operation that the first separated signal of the present frame is acquired based on the separation matrix and the original noisy signal of the present frame may be implemented as follows: the first separated signal of the present frame is obtained based on a product of the separation matrix and the original noisy signal of the present frame. For example, if the separation matrix is W (k) and the original noisy signal of the present frame is X (k,n) , the first separated signal of the present frame is <MAT>.

In an embodiment, if the separation matrix is the separation matrix for the present frame, the first separated signal of the present frame is obtained based on the separation matrix for the present frame and the original noisy signal of the present frame.

In another embodiment, if the separation matrix is the separation matrix for the previous frame of the present frame, the first separated signal of the present frame is obtained based on the separation matrix for the previous frame and the original noisy signal of the present frame.

In an embodiment, if a frame length of the audio signal acquired by the MIC is n, n being a natural number more than or equal to <NUM>, in case of n=<NUM>, the previous frame is a first frame.

In some embodiments, when the present frame is a first frame, the separation matrix for the first frame is an identity matrix.

The operation that the first separated signal of the present frame is acquired based on the separation matrix and the original noisy signal of the present frame includes the following action.

The first separated signal of the first frame is acquired based on the identity matrix and the original noisy signal of the first frame.

Herein, if the number of the MICs is two, the identity matrix is <MAT>; if the number of the MICs is three, the identity matrix is <MAT>; and by parity of reasoning, if the number of the MICs is N, the identity matrix may be <MAT>. <MAT> is an N×N matrix.

In some other embodiments, if the present frame is an audio frame after the first frame, the separation matrix for the present frame is determined based on the separation matrix for the previous frame of the present frame and the original noisy signal of the present frame.

In an embodiment, an audio frame may be an audio band with a preset time length.

In an example, the operation that the separation matrix for the present frame is determined based on the separation matrix for the previous frame of the present frame and the original noisy signal of the present frame may specifically be implemented as follows. A covariance matrix of the present frame may be calculated at first according to the original noisy signal and a covariance matrix of the previous frame. Then the separation matrix for the present frame is calculated based on the covariance matrix of the present frame and the separation matrix for the previous frame.

If it is determined that the n th frame is the present frame and the n - 1th frame is the previous frame of the present frame, the covariance matrix of the present frame may be calculated at first according to the original noisy signal and the covariance matrix of the previous frame. The covariance matrix is <MAT>, where β is a smoothing coefficient, Vp(k,n-<NUM>) is an updated covariance of the previous frame, ϕp (k,n) is a weighting coefficient, Xp(k,n) is the original noisy signal of the present frame, and <MAT> is a conjugate transpose matrix of the original noisy signal of the present frame. Herein, the covariance matrix of the first frame is a zero matrix. In an embodiment, after the covariance matrix of the present frame is obtained, the following eigenproblem may further be solved: V<NUM>(k,n)ep(k,n) = λp (k,n)V<NUM>(k,n)ep(k,n) , and the separation matrix of the present frame is calculated to be <MAT>, where λp (k,n) is an eigenvalue, and ep (k,n) is an eigenvector.

In the embodiment, in the case that the first separated signal is obtained according to the separation matrix of the present frame and the original noisy signal of the present frame, since the separation matrix is an updated separation matrix of the present frame, a proportion of the sound emitted from each sound source in the corresponding MIC may be dynamically tracked, so the obtained first separated signal is more accurate, which may facilitate obtaining a more accurate time-frequency estimated signal. In the case that the first separated signal is obtained according to the separation matrix of the previous frame of the present frame and the original noisy signal of the present frame, the calculation for obtaining the first separated signal is simpler, so that a calculation process for calculating the time-frequency estimated signal is simplified.

In some embodiments, the operation that the mask value of the time-frequency estimated signal of each sound source in the original noisy signal of each MIC is determined based on the respective time-frequency estimated signals of the at least two sound sources includes the following action.

The mask value of a sound source with respect to a MIC is determined to be a proportion of the time-frequency estimated signal of the sound source in the MIC and the original noisy signal of the MIC.

For example, there are three MICs, i.e., a first MIC, a second MIC and a third MIC respectively, and there are three sound sources, i.e., a first sound source, a second sound source and a third sound source respectively. The original noisy signal of the first MIC is X1 and the time-frequency estimated signals of the first sound source, the second sound source and the third sound source are Y1, Y2 and Y3 respectively. In such case, the mask value of the first sound source with respect to the first MIC is Y1/X1, the mask value of the second sound source with respect to the first MIC is Y2/X1, and the mask value of the third sound source with respect to the first MIC is Y3/X1.

Based on the example, the mask value may also be a value obtained after the proportion is transformed through a logarithmic function. For example, the mask value of the first sound source with respect to the first MIC is α×log (Y<NUM>/X<NUM>), the mask value of the second sound source with respect to the first MIC is α×log (Y<NUM>/X<NUM>), and the mask value of the third sound source with respect to the first MIC is α×log (Y<NUM>/X<NUM>), where α is an integer. In an embodiment, α is <NUM>. In the embodiment, transforming the proportion through the logarithmic function may synchronously reduce a dynamic range of each mask value to ensure that the separated voice is higher in quality.

In an embodiment, a base number of the logarithmic function is <NUM> or e. For example, in the embodiment, log (Y<NUM>/X<NUM>) may be log<NUM>(Y<NUM>/X<NUM>) or loge (Y<NUM>/X<NUM>).

In another embodiment, if there are two MICs and two sound sources, the operation that the mask value of the time-frequency estimated signal of each sound source in the original noisy signal of each MIC is determined based on the respective time-frequency estimated signals of the at least two sound sources includes the following action.

A ratio of the time-frequency estimated signal of a sound source and the time-frequency estimated signal of another sound source in the same MIC is determined.

For example, there are two MICs, i.e., a first MIC and a second MIC respectively, and there are two sound sources, i.e., a first sound source and a second sound source respectively. The original noisy signal of the first MIC is X<NUM>, and the original noisy signal of the second MIC is X<NUM>. The time-frequency estimated signal of the first sound source in the first MIC is Y<NUM>, and the time-frequency estimated signal of the second sound source in the second MIC is Y<NUM>. In such case, the time-frequency estimated signal of the second sound source in the first MIC is obtained to be Y<NUM>=X<NUM>-Y<NUM> by calculations, and the time-frequency estimated signal of the first sound source in the second MIC is obtained to be Y<NUM>=X<NUM>-Y<NUM> by calculations. Furthermore, the mask value of the first sound source in the first MIC is obtained based on Y<NUM>/Y<NUM>, and the mask value of the first sound source in the second MIC is obtained based on Y<NUM>/Y<NUM>.

The operation that the mask value of the time-frequency estimated signal of each sound source in the original noisy signal of each MIC is determined based on the respective time-frequency estimated signals of the at least two sound sources includes the following actions.

A proportion value is obtained based on the time-frequency estimated signal of a sound source in each MIC and the original noisy signal of the MIC.

Nonlinear mapping is performed on the proportion value to obtain the mask value of the sound source in each MIC.

The operation that nonlinear mapping is performed on the proportion value to obtain the mask value of the sound source in each MIC includes the following action.

Nonlinear mapping is performed on the proportion value by use of a monotonic increasing function to obtain the mask value of the sound source in each MIC.

Nonlinear mapping is performed on the proportion value according to a sigmoid function to obtain the mask value of the sound source in each MIC.

Herein, the sigmoid function is a nonlinear activation function. The sigmoid function is used to map an input function to an interval (<NUM>, <NUM>). In an embodiment, the sigmoid function is sigmoid <MAT>, where x is the mask value. In another embodiment, the sigmoid function is sigmoid <MAT>, where x is the mask value, a is a coefficient representing a degree of curvature of a function curve of the sigmoid function, and c is a coefficient representing translation of the function curve of the sigmoid function on the axis x.

In another embodiment, the monotonic increasing function is sigmoid <MAT> , where x is the mask value and a<NUM> is greater than <NUM>.

In an example, there are two MICs, i.e., a first MIC and a second MIC respectively, and there are two sound sources, i.e., a first sound source and a second sound source respectively. The original noisy signal of the first MIC is X<NUM>, and the original noisy signal of the second MIC is X<NUM>. The time-frequency estimated signal of the first sound source in the first MIC is Y<NUM>, and the time-frequency estimated signal of the second sound source in the second MIC is Y<NUM>. In such case, the time-frequency estimated signal of the second sound source in the first MIC is obtained to be Y<NUM>=X<NUM>-Y<NUM> by calculations. The mask value of the first sound source in the first MIC may be α×log (Y<NUM>/Y<NUM>), and the mask value of the first sound source in the second MIC may be α×log (Y<NUM>/Y<NUM>). Alternatively, α×log (Y<NUM>/Y<NUM>) is mapped to the interval (<NUM>, <NUM>) through the nonlinear activation function sigmoid to obtain a first mapping value as the mask value of the first sound source in the first MIC, and the first mapping value is subtracted from <NUM> to obtain a second mapping value as the mask value of the second sound source in the first MIC. α×log (Y<NUM>/Y<NUM>) is mapped to the interval (<NUM>, <NUM>) through the nonlinear activation function sigmoid to obtain a third mapping relationship as the mask value of the first sound source in the second MIC, and the third mapping relationship is subtracted from <NUM> to obtain a fourth mapping value as the mask value of the second sound source in the second MIC.

It should be appreciated that in another embodiment, the mask value of the sound source in the MIC may also be mapped to another predetermined interval, for example (<NUM>, <NUM>) or (<NUM>, <NUM>), through another nonlinear mapping function relationship. In such case, when the updated time-frequency estimated signal is subsequently calculated, division by a coefficients with corresponding multiples is required.

In the embodiment of the present invention, the mask value of any sound source in a MIC may be mapped to the predetermined interval by a nonlinear mapping function such as the sigmoid function, so that excessive mask value appeared in some embodiments may be dynamically reduced to simplify calculation, and a reference standard may further be unified for subsequent calculation of the updated time-frequency estimated signal to facilitate subsequent acquisition of a more accurate updated time-frequency estimated signal. In particular, if the predetermined interval is limited to be (<NUM>, <NUM>) and only two MICs are involved in mask value calculation, a calculation process of the mask value of the other sound source in the same MIC may be greatly simplified.

Of course, in another embodiment, the mask value may also be acquired in another manner if the proportion of the time-frequency estimated signal of each sound source in the original noisy signal of the same MIC is acquired. The dynamic range of the mask value may be reduced through the logarithmic function or in a nonlinear mapping manner, etc. There are no limits made herein.

In some embodiments, there are N sound sources, N being a natural number more than or equal to <NUM>.

The operation that the respective time-frequency estimated signals of the at least two sound sources are updated based on the respective original noisy signals of the at least two MICs and the mask values includes the following actions.

An xth numerical value is determined based on the mask value of the Nth sound source in the xth MIC and the original noisy signal of the xth MIC, x being a positive integer less than or equal to X and X being the total number of the MICs.

The updated time-frequency estimated signal of the Nth sound source is determined based on a first numerical value to an Xth numerical value.

In an example, the first numerical value is determined based on the mask value of the Nth sound source in the first MIC and the original noisy signal of the first MIC.

The second numerical value is determined based on the mask value of the Nth sound source in the second MIC and the original noisy signal of the second MIC.

The third numerical value is determined based on the mask value of the Nth sound source in the third MIC and the original noisy signal of the third MIC.

The rest numerical values are determined in the same manner.

The Xth numerical value is determined based on the mask value of the Nth sound source in the Xth MIC and the original noisy signal of the Xth MIC.

The updated time-frequency estimated signal of the Nth sound source is determined based on the first numerical value, the second numerical value to the Xth numerical value.

Then, the updated time-frequency estimated signal of the other sound source is determined in a manner similar to the manner of determining the updated time-frequency estimated signal of the Nth sound source.

For further explaining the example, the updated time-frequency estimated signal of the Nth sound source may be calculated through the following calculation formula: YN(k,n) = X<NUM>(k,n)gmask<NUM>N + X<NUM> (k,n)gmask<NUM>N + X<NUM> (k,n)gnask3N +L + Xx (k,n)gmaskXN , where YN (k,n) is the updated time-frequency estimated signal of the Nth sound source, k is the frequency point and n is the audio frame; X<NUM> (k,n), X<NUM> (k,n), X<NUM> (k,n),. and Xx (k,n) are the original noisy signals of the first MIC, the second MIC, the third MIC,. and the Xth MIC respectively; and mask<NUM>N, mask<NUM>N, mask<NUM>N ,. and maskXN are the mask values of the Nth sound source in the first MIC, the second MIC, the third MIC,. and the Xth MIC respectively.

In the embodiment of the present invention, the audio signals of the sounds emitted from different sound sources may be separated again based on the mask values and the original noisy signals. Since the mask value is determined based on the time-frequency estimated signal obtained by first separation of the audio signal and the ratio of the time-frequency estimated signal in the original noisy signal, band signals that are not separated by first separation may be separated and recovered to the corresponding audio signals of the respective sound sources. In such a manner, the voice damage degree of the audio signal may be reduced, so that voice enhancement may be implemented, and the quality of the audio signal from the sound source may be improved.

In some embodiments, the operation that the audio signals emitted from the at least two sound sources respectively are determined based on the respective updated time-frequency estimated signals of the at least two sound sources includes the following action.

Time-domain transform is performed on the respective updated time-frequency estimated signals of the at least two sound sources to obtain the audio signals emitted from the at least two sound sources respectively.

Herein, time-domain transform may be performed on the updated frequency-domain estimated signal based on Inverse Fast Fourier Transform (IFFT). The updated frequency-domain estimated signal may also be converted into a time-domain signal based on Inverse Short-Time Fourier Transform (ISTFT). Time-domain transform may also be performed on the updated frequency-domain signal based on other inverse Fourier transform.

For helping the abovementioned embodiments of the present invention to be understood, descriptions are made herein with the following example. As shown in <FIG>, an application scenario of the method for processing audio signal is disclosed. A terminal includes a speaker A, the speaker A includes two MICs, i.e., a first MIC and a second MIC respectively, and there are two sound sources, i.e., a first sound source and a second sound source respectively. Signals emitted from the first sound source and the second sound source may be acquired by both the first MIC and the second MIC. The signals from the two sound sources are aliased in each MIC.

<FIG> is a flow chart showing a method for processing audio signal, according to some embodiments of the invention. In the method for processing audio signal, as shown in <FIG>, sound sources include a first sound source and a second sound source, and MICs include a first MIC and a second MIC. Based on the method for processing audio signal, audio signals from the first and second sound sources are recovered from original noisy signals of the first MIC and the second MIC.

If a frame length of a system is Nfft, a frequency point is K=Nfft/<NUM>+<NUM>.

In S301, W (k) and Vp (k) are initialized.

Initialization includes the following steps.

In S302, an original noisy signal of the n th frame of the p th MIC is obtained. <MAT> is windowed to perform STFT based on Nfft points to obtain a corresponding frequency-domain signal: <MAT> , where m is the number of points selected for Fourier transform, STFT is short-time Fourier transform, and <MAT> is a time-domain signal of the n th frame of the p th MIC. Herein, the time-domain signal is the original noisy signal.

Then, an observed signal of Xp(k,n) is X(k,n)=[X<NUM>(k,n),X<NUM>(k,n)]T, where [X<NUM>(k,n), X<NUM> (k,n)]T is a transposed matrix.

In S303, a priori frequency-domain estimate for the signals from the two sound sources is obtained by use of W (k) of a previous frame.

It is set that the priori frequency-domain estimate for the signals from the two sound sources is Y(k,n)=[Y<NUM>(k,n),Y<NUM>(k,n)]T, where Y<NUM>(k,n),Y<NUM>(k,n) are estimated values for the first sound source and the second sound source at a frequency-frequency point (k, n) respectively.

A observation matrix X(k,n) is separated through the separation matrix W(k) to obtain that Y(k,n)=W'(k)X(k,n) , where W' (k) is a separation matrix for the previous frame (i.e., a previous frame of a present frame).

Then, a priori frequency-domain estimate for the n th frame of the signal from the p th sound source is: Yp (n) = [Yp(<NUM>,n),L Yp(K,n)]T.

In S304, a weighted covariance matrix Vp(k,n) is updated.

The updated weighted covariance matrix is calculated to be: <MAT>, where β is a smoothing coefficient, β being <NUM> in an embodiment; Vp(k,n-<NUM>) is a weighted covariance matrix of the previous frame; <MAT> is a conjugate transpose matrix of Xp (k,n) ; <MAT> is a weighting coefficient, <MAT> being an auxiliary variable; and G(Yp(n))= -logp(Yp(n)) is a contrast function.

p(Yp (n)) represents a whole-band-based multidimensional super-Gaussian priori probability density function of the p th sound source. In an embodiment, <MAT>. In such case, if <MAT>.

In S305, an eigenproblem is solved to obtain an eigenvector ep (k,n).

Herein, ep (k,n) is an eigenvector corresponding to the p th MIC.

The eigenproblem V<NUM> (k,n)ep (k,n) = λp (k,n)V<NUM> (k,n)ep (k,n) is solved to obtain:
<MAT>
<MAT>
<MAT> and
<MAT>
where <MAT>.

In S306, an updated separation matrix W(Zc) for each frequency point is obtained.

The updated separation matrix for the present frame is obtained to be <MAT> based on the eigenvector of the eigenproblem.

In S307, a posteriori frequency-domain estimate for the signals from the two sound sources is obtained by use of W (k) of the present frame.

The original noisy signal is separated by use of W (k) of the present frame to obtain the posteriori frequency-domain estimate Y(k,n)=[Y<NUM>(k,n),Y<NUM>(k,n)]T =W(k)X(k,n) for the signals from the two sound sources.

It can be understood that calculation in subsequent steps may be implemented by use of the priori frequency-domain estimate or the posteriori frequency-domain estimate. Using the priori frequency-domain estimate may simplify a calculation process, and using the posteriori frequency-domain estimate may obtain a more accurate audio signal of each sound source. Herein, the process of S301 to S307 may be considered as first separation for the signals from the sound sources, and the priori frequency-domain estimate or the posteriori frequency-domain estimate may be considered as the time-frequency estimated signal in the abovementioned embodiment.

It can be understood that, in the embodiment of the present invention, for further reducing voice damages, the separated audio signal may be re-separated based on a mask value to obtain a re-separated audio signal.

In S308, a component of the signal from each sound source in an original noisy signal of each MIC is acquired.

Through the step, the component Y<NUM> (k, n) of the first sound source in the original noisy signal X<NUM>(k,n) of the first MIC may be obtained.

The component Y<NUM>(k,n) of the second sound source in the original noisy signal X<NUM> (k,n) of the second MIC may be obtained.

Then, the component of the second sound source in the original noisy signal X<NUM>(k,n) of the first MIC is <MAT>.

The component of the first sound source in the original noisy signal X<NUM>(k,n) of the second MIC is <MAT>.

In S309, a mask value of the signal from each sound source in the original noisy signal of each MIC is acquired, and nonlinear mapping is performed on the mask value.

The mask value of the first sound source in the original noisy signal of the first MIC is obtained to be <MAT>.

Nonlinear mapping is performed on the mask value of the first sound source in the original noisy signal of the first MIC as follows: mask<NUM>(k,n)=sigmoid(mask<NUM>(k,n),<NUM>,<NUM>).

Then the mask value of the second sound source in the first MIC is mask12 (k, n)=<NUM>-mask<NUM>(k,n).

The mask value of the first sound source in the original noisy signal of the second MIC is obtained to be <MAT>.

Nonlinear mapping is performed on the mask value of the first sound source in the original noisy signal of the second MIC as follows: mask<NUM>(k,n)=sigmoid(mask<NUM>(k,n),<NUM>,<NUM>).

Then the mask value of the second sound source in the original noisy signal of the second MIC is mask22(k,n) = <NUM>- mask21(k,n).

Herein, sigmoid <MAT>. In the embodiment, a=<NUM> and c is <NUM>. Herein, x is the mask value, a is a coefficient representing a degree of curvature of a function curve of the sigmoid function, and c is a coefficient representing translation of the function curve of the sigmoid function on the axis x.

In S310, updated time-frequency estimated signals are acquired based on the mask values.

The updated time-frequency estimated signal of each sound source may be acquired based on the mask value of the sound source in each MIC and the original noisy signal of each MIC:
<MAT> where Y<NUM>(k,n) is the updated time-frequency estimated signal of the first sound source; and
<MAT> where Y<NUM>(k,n) is the updated time-frequency estimated signal of the second sound source.

In S311, time-domain transform is performed on the updated time-frequency estimated signals through inverse Fourier transform.

ISTFT and overlapping-addition are performed on Yp(n)=[Yp(<NUM>,n),. Yp(K,n)]T to obtain an estimated time-domain audio signal <MAT> respectively.

In the embodiment of the present invention, the original noisy signals of the two MICs are separated to obtain the time-frequency estimated signals of sounds emitted from the two sound sources in each MIC respectively, so that the time-frequency estimated signals of the sounds emitted from the two sound sources in each MIC may be preliminarily separated from the original noisy signals. Furthermore, the mask values of the two sound sources in the two MICs respectively may further be obtained based on the time-frequency estimated signals, and the updated time-frequency estimated signals of the sounds emitted from the two sound sources are acquired based on the original noisy signals and the mask values. Therefore, according to the embodiment of the present invention, the sounds emitted from the two sound sources may further be separated according to the original noisy signals and the preliminarily separated time-frequency estimated signals. In addition, the mask values is a proportion of the time-frequency estimated signal of a sound source in the original noisy signal of a MIC, so that part of bands that are not separated by preliminary separation may be recovered into the audio signals of their corresponding sound sources, voice damage degrees of the separated audio signals may be reduced, and the separated audio signal of each sound source is higher in quality.

Moreover, only two MICs are used, compared with the conventional art that a beamforming technology based on three or more MICs is adopted to implement sound source separation, the embodiment of the present invention has the advantages that, on one hand, the number of the MICs is greatly reduced, which reduces hardware cost of a terminal; and on the other hand, positions of multiple MICs are not required to be considered, which may implement more accurate separation of the audio signals emitted from different sound sources.

<FIG> is a block diagram of a device for processing audio signal, according to some embodiments of the invention. Referring to <FIG>, the device includes a detection module <NUM>, a first obtaining module <NUM>, a first processing module <NUM>, a second processing module <NUM> and a third processing module <NUM>.

The detection module <NUM> is configured to acquire audio signals emitted from at least two sound sources respectively through at least two MICs to obtain respective original noisy signals of the at least two MICs.

The first obtaining module <NUM> is configured to perform sound source separation on the respective original noisy signals of the at least two MICs to obtain respective time-frequency estimated signals of the at least two sound sources.

The first processing module <NUM> is configured to determine a mask value of the time-frequency estimated signal of each sound source in the original noisy signal of each MIC based on the respective time-frequency estimated signals of the at least two sound sources.

The second processing module <NUM> is configured to update the respective time-frequency estimated signals of the at least two sound sources based on the respective original noisy signals of the at least two MICs and the mask values.

The third processing module <NUM> is configured to determine the audio signals emitted from the at least two sound sources respectively based on the respective updated time-frequency estimated signals of the at least two sound sources.

In some embodiments, the first obtaining module <NUM> includes a first obtaining unit <NUM> and a second obtaining unit <NUM>.

The first obtaining unit <NUM> is configured to acquire a first separated signal of a present frame based on a separation matrix and the present frame of the original noisy signal. The separation matrix is a separation matrix for the present frame or a separation matrix for a previous frame of the present frame.

A second obtaining unit <NUM> is configured to combine the first separated signal of each frame to obtain the time-frequency estimated signal of each sound source.

The first obtaining unit <NUM> is configured to acquire the first separated signal of the first frame based on the identity matrix and the original noisy signal of the first frame.

In some embodiments, the first obtaining module <NUM> further includes a third obtaining unit <NUM>.

The third obtaining unit <NUM> is configured to, when the present frame is an audio frame after the first frame, determine the separation matrix for the present frame based on the separation matrix for the previous frame of the present frame and the original noisy signal of the present frame.

The first processing module <NUM> includes a first processing unit <NUM> and a second processing unit <NUM>.

The first processing unit <NUM> is configured to obtain a proportion value based on the time-frequency estimated signal of any of the sound sources in each MIC and the original noisy signal of the MIC.

The second processing unit <NUM> is configured to perform nonlinear mapping on the proportion value to obtain the mask value of the sound source in each MIC.

In some embodiments, the second processing unit <NUM> is configured to perform nonlinear mapping on the proportion value by use of a monotonic increasing function to obtain the mask value of the sound source in each MIC.

In some embodiments, there are N sound sources, N being a natural number more than or equal to <NUM>, and the second processing module <NUM> includes a third processing unit <NUM> and a fourth processing unit <NUM>.

The third processing unit <NUM> is configured to determine an xth numerical value based on the mask value of the Nth sound source in the xth MIC and the original noisy signal of the xth MIC, x being a positive integer less than or equal to X and X being the total number of the MICs.

The fourth processing unit <NUM> is configured to determine the updated time-frequency estimated signal of the Nth sound source based on a first numerical value to an Xth numerical value.

With respect to the device in the above embodiment, the specific manners for performing operations for individual modules therein have been described in detail in the embodiment regarding the method, which will not be elaborated herein.

The embodiments of the present invention also provide a terminal, which includes:.

The memory may include any type of storage medium, and the storage medium is a non-transitory computer storage medium and may keep information stored thereon when a communication device is powered off.

The processor may be connected with the memory through a bus and the like, and is configured to read an executable program stored in the memory to implement, for example, at least one of the methods shown in <FIG> and <FIG>.

The embodiments of the present invention further provide a computer-readable storage medium having stored therein an executable program, the executable program being executed by a processor to implement the method for processing audio signal in any embodiment of the present invention, for example, for implementing at least one of the methods shown in <FIG> and <FIG>.

<FIG> is a block diagram of a terminal <NUM>, according to some embodiments not being part of the invention. For example, the terminal <NUM> may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a gaming console, a tablet, a medical device, exercise equipment, a personal digital assistant and the like.

Referring to <FIG>, the terminal <NUM> may include one or more of the following components: a processing component <NUM>, a memory <NUM>, a power component <NUM>, a multimedia component <NUM>, an audio component <NUM>, an Input/Output (I/O) interface <NUM>, a sensor component <NUM>, and a communication component <NUM>.

The processing component <NUM> typically controls overall operations of the terminal <NUM>, such as the operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component <NUM> may include one or more processors <NUM> to execute instructions to perform all or part of the steps in the abovementioned method. Moreover, the processing component <NUM> may include one or more modules which facilitate interaction between the processing component <NUM> and the other components. For instance, the processing component <NUM> may include a multimedia module to facilitate interaction between the multimedia component <NUM> and the processing component <NUM>.

Examples of such data include instructions for any application programs or methods operated on the terminal <NUM>, contact data, phonebook data, messages, pictures, video, etc. The memory <NUM> may be implemented by any type of volatile or non-volatile memory devices, or a combination thereof, such as an Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic memory, a flash memory, a magnetic or an optical disk.

The power component <NUM> provides power for various components of the terminal <NUM>. The power component <NUM> may include a power management system, one or more power supplies, and other components associated with generation, management and distribution of power for the terminal <NUM>.

The multimedia component <NUM> includes a screen providing an output interface between the terminal <NUM> and a user. The screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes the TP, the screen may be implemented as a touch screen to receive an input signal from the user. The TP includes one or more touch sensors to sense touches, swipes and gestures on the TP. The touch sensors may not only sense a boundary of a touch or swipe action but also detect a duration and pressure associated with the touch or swipe action. The multimedia component <NUM> includes a front camera and/or a rear camera. The front camera and/or the rear camera may receive external multimedia data when the device <NUM> is in an operation mode, such as a photographing mode or a video mode. Each of the front camera and the rear camera may be a fixed optical lens system or have focusing and optical zooming capabilities.

The audio component <NUM> is configured to output and/or input an audio signal. For example, the audio component <NUM> includes a MIC, and the MIC is configured to receive an external audio signal when the terminal <NUM> is in the operation mode, such as a call mode, a recording mode and a voice recognition mode. The received audio signal may further be stored in the memory <NUM> or sent through the communication component <NUM>. The audio component <NUM> further includes a speaker configured to output the audio signal.

The sensor component <NUM> includes one or more sensors configured to provide status assessment in various aspects for the terminal <NUM>. For instance, the sensor component <NUM> may detect an on/off status of the device <NUM> and relative positioning of components, such as a display and small keyboard of the terminal <NUM>. The sensor component <NUM> may further detect a change in a position of the terminal <NUM> or a component of the terminal <NUM>, presence or absence of contact between the user and the terminal <NUM>, orientation or acceleration/deceleration of the terminal <NUM> and a change in temperature of the terminal <NUM>. The sensor component <NUM> may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication component <NUM> is configured to facilitate wired or wireless communication between the terminal <NUM> and another device. The terminal <NUM> may access a communication-standard-based wireless network, such as a Wireless Fidelity (WiFi) network, a 2nd-Generation (<NUM>) or 3rd-Generation (<NUM>) network or a combination thereof. The communication component <NUM> receives a broadcast signal or broadcast associated information from an external broadcast management system through a broadcast channel. The communication component <NUM> further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on a Radio Frequency Identification (RFID) technology, an Infrared Data Association (IrDA) technology, an Ultra-Wide Band (UWB) technology, a Bluetooth (BT) technology and another technology.

The terminal <NUM> may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components, and is configured to execute the abovementioned method.

In some embodiments of the invention, there is also provided a non-transitory computer-readable storage medium including instructions, such as the memory <NUM> including instructions, and the instructions may be executed by the processor <NUM> of the terminal <NUM> to implement the abovementioned method. For example, the non-transitory computer-readable storage medium may be a ROM, a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disc, an optical data storage device and the like.

In the description of the present invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," and the like can indicate a specific feature described in connection with the embodiment or example, a structure, a material or feature included in at least one embodiment or example. In the present invention, the schematic representation of the above terms is not necessarily directed to the same embodiment or example.

In some embodiments, the control and/or interface software or app can be provided in a form of a non-transitory computer-readable storage medium having instructions stored thereon is further provided. For example, the non-transitory computer-readable storage medium can be a ROM, a CD-ROM, a magnetic tape, a floppy disk, optical data storage equipment, a flash drive such as a USB drive or an SD card, and the like.

Implementations of the subject matter and the operations described in this invention can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this invention can be implemented as one or more computer programs, i.e., one or more portions of computer program instructions, encoded on one or more computer storage medium for execution by, or to control the operation of, data processing apparatus.

The operations described in this invention can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

The devices in this invention can include special purpose logic circuitry, e.g., an FPGA (field-programmable gate array), or an ASIC (application-specific integrated circuit).

The processes and logic flows described in this invention can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output.

Other embodiments of the invention will be apparent to those skilled in the art in view of the specification and drawings of the present invention.

Various modifications of, and equivalent acts corresponding to, the disclosed aspects of the example embodiments, in addition to those described above, can be made by a person of ordinary skill in the art, having the benefit of the present invention, without departing from the scope of the invention defined in the following claims, the scope of which is to be accorded the broadest interpretation.

It should be understood that "a plurality" or "multiple" as referred to herein means two or more. "And/or," describing the association relationship of the associated objects, indicates that there may be three relationships, for example, A and/or B may indicate that there are three cases where A exists separately, A and B exist at the same time, and B exists separately. The character "/" generally indicates that the contextual objects are in an "or" relationship.

In the present invention, it is to be understood that the terms "lower," "upper," "under" or "beneath" or "underneath," "above," "front," "back," "left," "right," "top," "bottom," "inner," "outer," "horizontal," "vertical," and other orientation or positional relationships are based on example orientations illustrated in the drawings, and are merely for the convenience of the description of some embodiments, rather than indicating or implying the device or component being constructed and operated in a particular orientation. Therefore, these terms are not to be construed as limiting the scope of the present invention.

In the description of the present invention, "a plurality" indicates two or more unless specifically defined otherwise.

In the present invention, a first element being "on" a second element may indicate direct contact between the first and second elements, without contact, or indirect geometrical relationship through one or more intermediate media or layers, unless otherwise explicitly stated and defined. Similarly, a first element being "under," "underneath" or "beneath" a second element may indicate direct contact between the first and second elements, without contact, or indirect geometrical relationship through one or more intermediate media or layers, unless otherwise explicitly stated and defined.

Claim 1:
A method for processing audio signals, the method comprising
acquiring (S11), by at least two microphones of a terminal, a plurality of audio signals emitted respectively from at least two sound sources, to obtain respective original noisy signals of the at least two microphones;
performing (S12), by the terminal, sound source separation on the respective original noisy signals of the at least two microphones to obtain respective time-frequency estimated signals of the at least two sound sources;
determining (S13), by the terminal, a mask value of the time-frequency estimated signal of each sound source in the original noisy signal of each microphone based on the respective time-frequency estimated signals of the at least two sound sources, comprising:
obtaining, by the terminal, a proportion value based on the time-frequency estimated signal of any of the sound sources and the original noisy signal of each microphone; and
performing, by the terminal, nonlinear mapping on the proportion value based on a sigmoid function to obtain the mask value of the sound source in each microphone;
updating (S14), by the terminal, the respective time-frequency estimated signals of the at least two sound sources based on the respective original noisy signals of the at least two microphones and the mask values; and
determining (S15), by the terminal, the plurality of audio signals emitted respectively from the at least two sound sources based on the respective updated time-frequency estimated signals of the at least two sound sources,
wherein performing the nonlinear mapping on the proportion value based on the sigmoid function to obtain the mask value of the sound source in each microphone consists in:
performing the nonlinear mapping on the proportion value to obtain the mask value of the sound source in each microphone by using one of the following sigmoid functions: <MAT> wherein x is the mask value;
<MAT> wherein x is the mask value, a is a coefficient representing a degree of curvature of a function curve of the sigmoid function, and c is a coefficient representing translation of the function curve of the sigmoid function on an axis x; or
<MAT> wherein x is the mask value, and a<NUM> is greater than <NUM>.