Patent Publication Number: US-10770090-B2

Title: Method and device of audio source separation

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to a method and a device of audio source separation, and more particularly, to a method and a device of audio source separation capable of being adaptive to a spatial variation of a target signal. 
     2. Description of the Prior Art 
     Speech input/recognition is widely exploited in electronic products such as mobile phones, and multiple microphones are usually utilized to enhance performance of speech recognition. In a speech recognition system with multiple microphones, an adaptive beamformer technology is utilized to perform spatial filtering to enhance audio/speech signals from a specific direction, so as to perform speech recognition on the audio/speech signals from the specific direction. An estimation of direction-of-arrival (DoA) corresponding to the audio source is required to obtain or modify a steering direction of the adaptive beamformer. A disadvantage of the adaptive beamformer is that the steering direction of the adaptive beamformer is likely incorrect due to a DoA estimation error. In addition, a constrained blind source separation (CBSS) method is proposed in the art to generate the demixing matrix, which is able/utilized to separate a plurality of audio sources from signals received by a microphone array. The CBSS method is also able to solve a permutation problem among the separated sources of a conventional blind source separation (BSS) method. However, a constraint of the CBSS method in the art is not able to be adaptive to a spatial variation of the target signal(s), which degrades performance of target source separation. Therefore, it is necessary to improve the prior art. 
     SUMMARY OF THE INVENTION 
     It is therefore a primary objective of the present invention to provide a method and a device of audio source separation capable of being adaptive to a spatial variation of a target signal, to improve over disadvantages of the prior art. 
     An embodiment of the present invention discloses a method of audio source separation, configured to separate audio sources from a plurality of received signals. The method comprises steps of applying a demixing matrix on the plurality of received signals to generate a plurality of separated results; performing a recognition operation on the plurality of separated results to generate a plurality of recognition scores, wherein the plurality of recognition scores is related to a matching degree between the plurality of separated results and a target signal; generating a constraint according to the plurality of recognition scores, wherein the constraint is a spatial constraint or a mask constraint; and adjusting the demixing matrix according to the constraint; wherein the adjusted demixing matrix is applied to the plurality of received signals to generate a plurality of updated separated results from the plurality of received signals. 
     An embodiment of the present invention further discloses an audio separation device, configured to separate audio sources from a plurality of received signals. The audio separation device comprises a separation unit, for applying a demixing matrix on the plurality of received signals to generate a plurality of separated results; a recognition unit, for performing a recognition operation on the plurality of separated results to generate a plurality of recognition scores, wherein the plurality of recognition scores is related to a matching degree between the plurality of separated results and a target signal; a constraint generator, for generating a constraint according to the plurality of recognition scores, wherein the constraint is a spatial constraint or a mask constraint; and a demixing matrix generator, for adjusting the demixing matrix according to the constraint; wherein the adjusted demixing matrix is applied to the plurality of received signals to generate a plurality of updated separated results from the plurality of received signals. 
     These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a schematic diagram of an audio source separation device according to an embodiment of the present invention. 
         FIG. 2  is a schematic diagram of an audio source separation process according to an embodiment of the present invention. 
         FIG. 3  is a schematic diagram of a constraint generator according to an embodiment of the present invention. 
         FIG. 4  is a schematic diagram of an update controller according to an embodiment of the present invention. 
         FIG. 5  is a schematic diagram of a spatial constraint generation process according to an embodiment of the present invention. 
         FIG. 6  is a schematic diagram of a constraint generator according to an embodiment of the present invention. 
         FIG. 7  is a schematic diagram of an update controller according to an embodiment of the present invention. 
         FIG. 8  is a schematic diagram of a mask constraint generation process according to an embodiment of the present invention. 
         FIG. 9  is a schematic diagram of an audio source separation device according to an embodiment of the present invention. 
         FIG. 10  is a schematic diagram of a recognition unit according to an embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  is a schematic diagram of an audio source separation device  1  according to an embodiment of the present invention. The audio source separation device  1  may be an application specific integrated circuit (ASIC) , configured to separate audio sources z 1 - z M  from received signals x 1 -x M . Target signals s 1 -s N  may be speech signals and exist within the audio sources z 1 -z M . The audio sources z 1 -z M  may have various types. For example, the audio sources z 1 -z M  may be background noise, echo, interference or speech from speaker(s). In embodiments of the present invention, the target signals s 1 -s N  may be speech signals from a target speaker for a specific speech content. Hence, in an environment with the audio sources z 1 -z M , the target signals s 1 -s N  do not always exist. For illustrative purpose, the following description is under an assumption that there is only one single target signal s n . The audio source separation device  1  may be applied for speech recognition or speaker recognition, which comprises receivers R 1 -R M , a separation unit  10 , a recognition unit  12 , a constraint generator  14  and a demixing matrix generator  16 . The receivers R 1 -R M  may be microphones, which receive received signals x 1 -x M  and deliver the received signals x 1 -x M  to the separation unit  10 . The received signals x 1 -x M  may be represented as a received signal set x, i.e., x=[x 1 , . . . , x M ] T . The separation unit  10  is coupled to the demixing matrix generator  16 . The separation unit  10  is configured to multiply the received signal set x by a demixing matrix W generated by the demixing matrix generator  16 , so as to generate a separated result set y. The separated result set y comprises separated results y 1 -y M , i.e., y[y 1 , . . . , y M ] T =Wx, wherein the separated results y 1 -y M , corresponding to the audio sources z 1 -z M , are separated from the received signals x 1 -x M . The recognition unit  12  is configured to perform a recognition operation on the separated results so as to generate recognition scores q 1 -q M , related to the matching degree corresponding to the target signal s n , and deliver the recognition scores q 1 -q M  to the constraint generator  14 . The higher the recognition scores q m , the higher the matching degree (the more similar) between the separated result y m  and the target signal s n . The constraint generator  14  may generate a constraint CT according to the recognition scores q 1 -q M , and deliver the constraint CT to the demixing matrix generator  16 , wherein the constraint CT is utilized as a control signal corresponding to a specific direction in a particular space. The demixing matrix generator  16  may generate a renewed/adjusted demixing matrix W according to the constraint CT. The adjusted demixing matrix W may then be applied to the received signals x 1 -x M  to separate the audio sources z 1 -z M . In an embodiment, the demixing matrix W may be generated by the demixing matrix generator  16  via a constrained blind source separation (CBSS) method. 
     The recognition unit  12  may comprise a feature extractor  20 , a reference model trainer  22  and a matcher  24 , as shown in  FIG. 10 . The feature extractor  20  may generate feature signals b 1 -b M  according to the separated results y 1 -y M . Take speech recognition as an example, the feature extracted by the feature extractor  20  may be Mel-frequency cepstral coefficients (MFCC). When a training flag FG indicates that the recognition unit  12  is in a training phase, the feature extractor  20  extracts features related to the target signal s n  from the separated results y 1 -y M , and delivers the features to the reference model trainer  22 , so as to generate a reference model of the target signal s n . On the other hand, when the training flag FG indicates that the recognition unit  12  is in a testing phase, the matcher  24  compares features extracted from the separated results y 1 -y M (in the testing phase) with the reference model, so as to generate the recognition scores q 1 -q M . In other words, the reference model trainer  22  may establish the reference model corresponding to the target signal s n  during the training phase. Then, in the testing phase, the matcher compares the feature signals b 1 -b M  extracted by the feature extractor  20  (in the testing phase) with the reference model, to output the recognition scores q 1 -q M  and obtain the degree of similarity in between. Other details of the recognition unit  12  are known by the art, which are not narrated herein. 
     In short, since the recognition scores q 1 -q M  may change with spatial characteristic of the target signal(s) related to the receivers R 1 -R M , the audio source separation device  1  generates different constraint CT, according to the recognition scores q 1 -q M  generated by the recognition unit  12  at different time instants, as a control signal corresponding to some specific direction in the space, and adjusting the demixing matrix W according to the updated constraint CT, so as to separate the audio sources z 1 -z M  more properly, and obtain the updated results y 1 -y M . Therefore, the constraint CT and the demixing matrix W generated by the audio source separation device  1  are adaptive in response to the spatial variation of the target signal(s), which improves performance of target source separation. Operations of the audio source separation device  1  may be summarized as an audio source separation process  20 . As shown in  FIG. 2 , the audio source separation process  20  comprises the following steps:
     Step  200 : Apply the demixing matrix W on the received signals x 1 -x M , to generate the separated results y 1 -y M .   Step  202 : Perform the recognition operation on the separated results y 1 -y M , to generate the recognition scores q 1 -q M  corresponding to the target signal s n .   Step  204 : Generate the constraint CT according to the recognition scores q 1 -q M  corresponding to the target signal s n .   Step  206 : Adjust the demixing matrix W according to the constraint CT.   

     In an embodiment, the constraint generator  14  may generate the constraint CT as a spatial constraint c, and the demixing matrix generator  16  may generate the renewed demixing matrix W according to the spatial constraint c. The spatial constraint c may be configured to limit a response of the demixing matrix W along with a specific direction in the space, such that the demixing matrix W has a spatial filtering effect on the specific direction. Methods of the demixing matrix generator  16  generating the demixing matrix W according to the spatial constraint c are not limited. For example, the demixing matrix generator  16  may generate the demixing matrix W such that w m   H c=c 1 , where c 1  may be an arbitrary constant, and w m   H  represents a row vector of the demixing matrix W (i.e., the demixing matrix W may be represented as 
     
       
         
           
             
               
                 W 
                 = 
                 
                   [ 
                   
                     
                       
                         
                           w 
                           1 
                           H 
                         
                       
                     
                     
                       
                         ⋮ 
                       
                     
                     
                       
                         
                           w 
                           M 
                           H 
                         
                       
                     
                   
                   ] 
                 
               
               ) 
             
             . 
           
         
       
     
     In detail,  FIG. 3  and  FIG. 4  are schematic diagrams of a constraint generator  34  and an update controller  342  according to an embodiment of the present invention. The constraint generator  34  may generate the spatial constraint c according to the demixing matrix W and the recognition scores q 1 -q M , which comprises the update controller  342 , a matrix inversion unit  30  and an average unit  36 . The update controller  342  comprises a mapping unit  40 , a normalization unit  42 , a maximum selector  44  and a weighting combining unit  46 . The matrix inversion unit  30  is coupled to the demixing matrix generator  16  to receive the demixing matrix W, and performs a matrix inversion operation on the demixing matrix W, to generate an estimated mixing matrix W −1 . The update controller  342  generates an update rate α and an update coefficient c update  according to the estimated mixing matrix W −1  and the recognition scores q 1 -q M , and the average unit  36  generates the spatial constraint c according to the update rate α and the update coefficient c update . 
     Specifically, the estimated mixing matrix W −1  may represent an estimate of a mixing matrix H. The mixing matrix H represents corresponding relationship between the audio sources z 1 -z M  and the received signals x 1 -x M , i.e., x=Hz and z=[z 1 , . . . , z M ] T . The mixing matrix H comprises steering vectors h 1 -h M , i.e. , H=[h 1 . . . h M ]. In other words, the estimated mixing matrix w −1  comprises estimated steering vectors ĥ 1 -ĥ M , which may be represented as W −1 =└ĥ 1  . . . ĥ M ┘. In addition, the update controller  342  may generate weightings ω 1 -ω M  according to the recognition scores q 1 -q M , and generate the update coefficient c update  as 
               c   update     =       ∑     m   =   1     M     ⁢           ⁢       ω   m     ⁢         h   ^     m     .               
In addition, the update controller  342  performs a mapping operation on the recognition scores q 1 -q M  via the mapping unit  40 , which is to map the recognition scores q 1 -q M  onto an interval between 0 and 1, linearly or nonlinearly, to generate mapping values {tilde over (q)} 1 -{tilde over (q)} M  corresponding to the recognition scores q 1 -q M  (each of the mapping values {tilde over (q)} 1 -{tilde over (q)} M  is between 0 and 1). Further, the update controller  342  performs a normalization operation on the mapping values {tilde over (q)} 1 -{tilde over (q)} M  via the normalization unit  42 , to generate the weightings ω 1 -ω M  
 
               (       i   .   e   .     ,       ω   m     =         q   ~     m     /       ∑     n   =   1     M     ⁢       q   ~     n             )     .         
In addition, the update controller  342  may generate the update rate α as a maximum value among the mapping values {tilde over (q)} 1 -{tilde over (q)} M  via the maximum selector  44 , i.e., α=max m {tilde over (q)} m  . Therefore, the update controller  342  may output the update rate α and the update coefficient c update  to the average unit  36 , and the average unit  36  may compute the spatial constraint c as c=(1−α)c+αc update . The constraint generator  34  delivers the spatial constraint c to the demixing matrix generator  16 , and the demixing matrix generator  16  may generate the renewed demixing matrix W according to the spatial constraint c, to separate the audio sources z 1 -z M  even more properly.
 
     Operations of the constraint generator  34  may be summarized as a spatial constraint generation process  50 , as shown in  FIG. 5 . The spatial constraint generation process  50  comprises the following steps:
     Step  500 : Perform the matrix inversion operation on the demixing matrix W, to generate the estimated mixing matrix W −1 , wherein the estimated mixing matrix W −1  comprises the estimated steering vectors ĥ 1 -ĥ M .   Step  502 : Generating the weightings ω 1 -ω M  according to the recognition scores q 1 -q M .   Step  504 : Generate the update rate α according to the recognition scores q 1 -q M .   Step  506 : Generate the update coefficient c update  according to the weightings ω 1 -ω M  and the estimated steering vectors ĥ 1 -ĥ M .   Step  508 : Generate the spatial constraint c according to the update rate α and the update coefficient c update .   

     In another embodiment, the constraint generator  14  may generate the constraint CT as a mask constraint Λ, and the demixing matrix generator  16  may generate the renewed demixing matrix W according to the mask constraint Λ. The mask constraint Λ may be configured to limit a response of the demixing matrix w toward a target signal, to have a masking effect on the target signal. Method of the demixing matrix generator  16  generating the demixing matrix w according to the mask constraint Λ is not limited. For example, the demixing matrix generator  16  may use a recursive algorithm (such as a Newton method, a gradient method, etc.) to estimate an estimate of the mixing matrix H between the audio sources z 1 -z M  and the received signals x 1 -x M , and use the mask constraint Λ to constraint a variation of the estimated mixing matrix from one iteration to the next iteration. In other words, the estimated mixing matrix Ĥ k+1 , at the (k+1) -th iteration can be represented as Ĥ k+1 =Ĥ k +ΔH·Λ, wherein the demixing matrix generator  16  may generate the demixing matrix W as W=Ĥ k+1   −1 , and ΔH is related to the algorithm the demixing matrix generator  16  uses to generate the estimated mixing matrix Ĥ k+1 . In addition, the mask constraint Λ may be a diagonal matrix, which may perform a mask operation on an audio source z n*  among the audio sources z 1 -z M , where the audio source z n*  is regarded as the target signal s n , and the index n* is regarded as the target index. In detail, the constraint generator  14  may set the n*-th diagonal element of the mask constraint Λ as a specific value G, where the specific value G is between 0 and 1, and set the rest of diagonal elements as (1-G). That is, the i-th diagonal element [Λ] i,i  of the mask constraint Λ may be expressed as 
     
       
         
           
             
               
                 [ 
                 Λ 
                 ] 
               
               
                 i 
                 , 
                 i 
               
             
             = 
             
               { 
               
                 
                   
                     
                       
                         
                             
                         
                         ⁢ 
                         
                           G 
                           , 
                         
                       
                     
                     
                       
                         i 
                         = 
                         
                           n 
                           * 
                         
                       
                     
                   
                   
                     
                       
                         
                           1 
                           - 
                           G 
                         
                         , 
                       
                     
                     
                       
                         i 
                         ≠ 
                         
                           n 
                           * 
                         
                       
                     
                   
                 
                 . 
               
             
           
         
       
     
     In detail,  FIG. 6  and  FIG. 7  are schematic diagrams of a constraint generator  64  and an update controller  642  according to an embodiment of the present invention. The constraint generator  64  may generate the mask constraint Λ according to the separated results y 1 -y M  and the recognition scores q 1 -q M , which comprises the update controller  642 , an energy unit  60 , a weighted energy generator  62 , a reference energy generator  68  and a mask generator  66 . The update controller  642  comprises a mapping unit  70 , a normalization unit  72  and a transforming unit  74 . The energy unit  60  receives the separated results y 1 -y M  and computes audio source energies P 1 -P M  corresponding to the separated results y 1 -y M  (also corresponding to the audio sources z 1 -z M ). The update controller  642  generates the weightings ω 1 -ω M  and weightings ß 1 -ß M  according to the recognition scores q 1 -q M . The weighted energy generator  62  generates a weighted energy P wei  according to the weightings ω 1 -ω M  and the audio source energies P 1 -P M . The reference energy generator  68  generates a reference energy P ref  according to the weightings ß 1 -ß M  and the audio source energies P 1 -P M . The mask generator  66  generates the mask constraint Λ according to the weightings ω 1 -ω M , the weighted energy P wei  and the reference energy P ref . 
     Specifically, the weighted energy generator  62  may generate the weighted energy P wei  as 
               P   wei     =       ∑     m   =   1     M     ⁢           ⁢       ω   m     ⁢       P   m     .               
The reference energy generator  68  may generate the reference energy P ref  as
 
               P   ref     =       ∑     m   =   1     M     ⁢           ⁢       β   m     ⁢       P   m     .               
The mapping unit  70  and the normalization unit  72  comprised in the update controller  642  are the same as the mapping unit  40  and the normalization unit  42 , which are not narrated further herein. In addition, the transforming unit  74  may transform the weightings ω 1 -ω M  into the weightings ß 1 -ß M , Method of the transforming unit  74  generating the weightings ß 1 -ß M  is not limited. For example, the transforming unit  74  may generate/transform the weightings ß M  as β m =1−ω m , which is not limited thereto.
 
     On the other hand, the mask generator  66  may generate the specific value G in the mask constraint Λ according to the weighted energy P wei  and the reference energy P ref . For example, the mask generator  66  may compute the specific value G as 
             G   =     {             1   ,             P   wei     &gt;     γ   ⁢           ⁢     P   ref                   0   ,             P   wei     ≤     γ   ⁢           ⁢     P   ref               ,             
where the ratio γ may be adjusted according to practical situation. In addition, the mask generator  66  may compute the specific value G as G=P wei /P ref  or G=P wei /(P ref +P wei ), and not limited thereto. In addition, the mask generator  66  may determine the target index n* of the target signal according to the weightings ω 1 -ω M  (i.e., according to the recognition scores q 1 -q M ) . For example, the mask generator  66  may determine the target index n* as an index corresponding to a maximum weighting among the weightings ω 1 -ω M , i.e., the target index n* may be expressed as n*=arg m  max ω m . Thus, after obtaining the specific value G and the target index n*, the mask generator  66  may generate the mask constraint Λ as
 
                 [   Λ   ]       i   ,   i       =     {                     ⁢     G   ,             i   =     n   *                   1   -   G     ,           i   ≠     n   *             .             
The constraint generator  64  may deliver the mask constraint Λ to the demixing matrix generator  16 , and the demixing matrix generator  16  may generate the renewed demixing matrix W according to the mask constraint Λ, so as to separate the audio sources z 1 -z M  more properly.
 
     Operations of the constraint generator  64  may be summarized as a mask constraint generation process  80 . As shown in  FIG. 8 , the mask constraint generation process  80  comprises the following steps:
     Step  800 : Compute the audio source energies P 1 -P M  corresponding to the audio sources z 1 -z M  according to the separated results y 1 -y M .   Step  802 : Generate the weightings ω 1 -ω M  and the weightings ß 1 -ß M  according to the recognition scores q 1 -q M .   Step  804 : Generate the weighted energy P wei  according to the audio source energies P 1 -P M  and the weightings ω 1   -ω   M .   Step  806 : Generate the reference energy P ref  according to the audio source energies P 1 -P M  and the weightings ß 1 -ß M .   Step  808 : Generate the specific value G according to the weighted energy P wei  and the reference energy P ref .   Step  810 : Determine the target index n* according to the weightings ω 1 -ω M .   Step  812 : Generate the mask constraint Λ according to the specific value G and the target index n*.   

     In another perspective, the audio separation device is not limited to be realized by ASIC.  FIG. 9  is a schematic diagram of an audio source separation device  90  according to an embodiment of the present invention. The audio separation device  90  comprises a processing unit  902  and a storage unit  904 . The audio source separation process  20 , the spatial constraint generation process  50 , the mask constraint generation process  80  stated in the above may be compiled as a program code  908  stored in the storage unit  904 , to instruct the processing unit  902  to execute the processes  20 ,  50  and  80 . The processing unit  902  may be a digital signal processor (DSP), and not limited thereto. The storage unit  904  may be a non-volatile memory (NVM), e.g., an electrically erasable programmable read only memory (EEPROM) or a flash memory, and not limited thereto. 
     In addition, to be more understandable, a number of M is used to represent the numbers of the audio sources z, the target signal s, the receivers R, or other types of output signals (such as the audio source energies P, the recognition scores q, the separated results y, etc.) in the above embodiments. Nevertheless, the numbers thereof are not limited to be the same. For example, the numbers of the receivers R, the audio sources z, and the target signal s, may be 2, 4, and 1, respectively. 
     In summary, the present invention is able to update the constraint according to the scores, and adjust the demixing matrix according to the updated constraint, which may be adaptive to the spatial variation of the target signal(s) , so as to separate the audio sources z 1 -z M  more properly. 
     Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.