Abstract:
A method separates acoustic signals generated by multiple acoustic sources, such as mixed speech spoken simultaneously by several speakers in the same room. For each source, the acoustic signals are combined into a mixed signal acquired by multiple microphones, at least one for each source. The mixed signal is filtered, and the filtered signals are summed into a signal from which features are extracted. A target sequence through a factorial HMM is estimated, and filter parameters are optimized accordingly. These steps are repeated until the filter parameters converge to optimal filtering parameters, which are then used to filter the mixed signal once more, and the summed output of this last filtering is the acoustic signal for a particular acoustic source.

Description:
FIELD OF THE INVENTION  
         [0001]    The present invention relates generally separating mixed acoustic signals, and more particularly to separating mixed acoustic signals acquired by multiple channels from multiple acoustic sources, such as speakers.  
         BACKGROUND OF THE INVENTION  
         [0002]    Often, multiple speech signals are generated simultaneously by speakers so that the speech signals mix with each other in a recording. Then, it becomes necessary to separate the speech signals. In other words, when two or more people speak simultaneously, it is desired to separate the speech from the individual speakers from recordings of the simultaneous speech. This is referred to as a speaker separation problem.  
           [0003]    In one method, the simultaneous speech is received via a single channel recording, and the mixed signal is separated by time-varying filters, see Roweis, “One Microphone Source Separation,” Proc. Conference on Advances in Neural Information Processing Systems, pp. 793-799, 2000, and Hershey et al., “Audio Visual Sound Separation Via Hidden Markov Models,” Proc. Conference on Advances in Neural Information Processing Systems, 2001. That method uses extensive a priori information about the statistical nature of speech from the different speakers, usually represented by dynamic models like a hidden Markov model (HMM), to determine the time-varying filters.  
           [0004]    Another method uses multiple microphones to record the simultaneous speech. That method typically requires at least as many microphones as the number of speakers, and the source separation problem is treated as one of blind source separation (BSS). BSS can be performed by independent component analysis (ICA). There, no a priori knowledge of the signals is assumed. Instead, the component signals are estimated as a weighted combination of current and past samples taken from the multiple recordings of the mixed signals. The estimated weights optimize an objective function that measures an independence of the estimated component signals, see Hyväarinen, “Survey on Independent Component Analysis,” Neural Computing Surveys, Vol. 2., pp. 94-128, 1999.  
           [0005]    Both methods have drawbacks. The time-varying filter method, with known signal statistics, is based on the single-channel recording of the mixed signals. The amount of information present in the single-channel recording is usually insufficient to do effective speaker separation. The blind source separation method ignores all a priori information about the speakers. Consequently, in many situations, such as when the signals are recorded in a reverberant environment, the method fails.  
           [0006]    Therefore, it is desired to provide a method for separating mixed speech signals that improves over the prior art.  
         SUMMARY OF THE INVENTION  
         [0007]    The method according to the invention uses detailed a prior statistical information about acoustic speech signals, e.g., speech, to be separated. The information is represented in hidden Markov models (HMM). The problem of signal separation is treated as one of beam-forming. In beam-forming, each signal is extracted using an estimated filter-and-sum array.  
           [0008]    The estimated filters maximize a likelihood of the filtered and summed output, measured on the HMM for the desired signal. This is done by factorial processing using a factorial HMM (FHMM). The FHMM is a cross-product of the HMMs for the multiple signals. The factorial processing iteratively estimates the best state sequence through the HMM for the signal from the FHMM for all the concurrent signals, using the current output of the array, and estimates the filters to maximize the likelihood of that state sequence.  
           [0009]    In a two-source mixture of acoustic signals, the method according to the invention can extract a background acoustic signal that is 20 dB below a foreground acoustic signal when the HMMs for the signals are constructed from the acoustic signals. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0010]    [0010]FIG. 1 is a block diagram of a system for separating mixed acoustic signals according to the invention;  
         [0011]    [0011]FIG. 2 is a block diagram of a method for separating mixed acoustic signals according to the invention;  
         [0012]    [0012]FIG. 3 is flow diagram of factorial HMMs used by the invention;  
         [0013]    [0013]FIG. 4A is a graph of a mixed speech signal to be separated; and  
         [0014]    FIGS.  4 B-C are graphs of separated speech signals according to the invention. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT  
       [0015]    System Structure  
         [0016]    [0016]FIG. 1 shows the basic structure of a system  100  for multi-channel acoustic signal separation according to our invention. In this example, there are two sources, e.g., speakers  101 - 102 , generating a mixed acoustic signal, e.g., speech  103 . More sources are possible. The object of the invention is to separate the signal  190  of a single source from the acquired mixed signal.  
         [0017]    The system includes multiple microphones  110 , at least one for each speaker or other source. Connected to the multiple microphones are multiple sets of filter  120 . There is one set of filters  120  for each speaker, and the number of filters in each set  120  is equal to the number of microphones  110 .  
         [0018]    The output  121  each set of filters  120  is connected to a corresponding adder  130 , which provides a summed signal  131  to a feature extraction module  140 .  
         [0019]    Extracted features  141  are fed to a factorial processing module  150  having its output connected to an optimization module  160 . The features are also fed directly to the optimization module  160 . The output of the optimization module  160  is fed back to the corresponding set of filters  120 . Transcription hidden Markov models (HMMs)  170  for each speaker also provide input to the factorial processing module  150 . It should be noted that HMMs do not need to be transcription based, e.g., the HMMs can be derived directly from the acoustic content, in whatever form or source, music, machinery sounds, natural sounds, animal sounds, and the like.  
         [0020]    System Operation  
         [0021]    During operation, the acquired mixed acoustic signals  111  are first filtered  120 . An initial set of filter parameters can be used. The filtered signal  121  is summed, and features  141  are extracted  140 . A target sequence  151  is estimated  150  using the HMMs  170 . An optimization  160 , using a conjugate gradient descent, then derives optimal filter parameters  161  that can be used to separate the signal  190  of a single source, for example a speaker.  
         [0022]    The structure and operation of the system and method according to our invention is now described in greater detail.  
         [0023]    Filter and Sum  
         [0024]    We assume that the number of sources is known. For each source, we have a separate filter-and-sum array. The mixed signal  111  from each microphone  110  is filtered  120  by a microphone-specific filter. The various filtered signals  121  are summed  130  to obtain a combined  131  signal. Thus, the combined output signal y i [n]  131  for source i is:  
                 y   i          [   n   ]       =       ∑     j   =   1     L                         h     i                 j            [   n   ]       *       x   j          [   n   ]                   (   1   )                               
 
         [0025]    where L is the number of microphones  110 , x j [n] is the signal  111  at the j th  microphone, and h ij [n] is the filter applied to the j th  filter for speaker i. The filter impulse responses h ij [n] is optimized by optimal filter parameters  161  such that the resultant output y i [n]  190  is the separated signal from the i th  source.  
         [0026]    Optimizing the Filters for a Source  
         [0027]    The filters  120  for the signals from a particular source are optimized using available information about their acoustic signal, e.g., a transcription of the speech from the speaker.  
         [0028]    We can use a speaker-independent hidden Markov model (HMM) based speech recognition system that has been trained on a 40-dimensional Mel-spectral representation of the speech signal. The recognition system includes HMMs for the various sound units in the acoustic signal.  
         [0029]    From these, and perhaps, the known transcription for the speaker&#39;s utterance, we construct the HMM  170  for the utterance. Following this, the parameters  161  for the filters  120  for the speaker are estimated to maximize the likelihood of the sequence of 40-dimensional Mel-spectral vectors determined from the output  141  of the filter-and-sum array, on the utterance HMM  170 .  
         [0030]    For the purpose of optimization, we express the Mel-spectral vectors as a function of the filter parameters as follows.  
         [0031]    First we concatenate the filter parameters for the i th  source, for all channels, into a single vector h i . A parameter Z i  represent the sequence of Mel-spectral vectors extracted  141  from the output  131  of the array for the i th  source. The parameter z it  is the t th  spectral vector in Z i . The parameter z it  is related to the vector h i  by:  
           z   it =log ( M|DFT ( y   it )| 2 )=log ( M ( diag ( FX   t   h   i   h   i   T   X   t   T   F   H )))  (2)  
         [0032]    where Y it  is a vector representing the sequence of samples from y i [n] that are used to determine Z it , M is a matrix of the weighting coefficients for the Mel filters, F is the Fourier transform matrix, and X t  is a super matrix formed by the channel inputs and their shifted versions.  
         [0033]    Let Λ i  represent the set of parameters for the HMM for the i th  source. In order to optimize the filters for the i th  source, we maximize L i (Z i )=log (P(Z i |Λ i )), the log-likelihood of Z i  on the HMM for that source. The parameter L i (Z i ) is determined over all possible state sequences through the HMMs  170 .  
         [0034]    To simplify the optimization, we assume that the overall likelihood of Z i  is largely represented by the likelihood of the most likely state sequence through the HMM, i.e., P(Z i |Λ i )=P(Z i , S i |Λ i ), where S i  represents the most likely state sequence through the HMM. Under this assumption, we get  
                 L   i          (     Z   i     )       =         ∑     t   =   1     T                     log        (     P        (       z     i                 t            s     i                 t         )       )         +     log        (     P        (       s     i                 1       ,     s     i                 2       ,              …              ,     s     i                 T         )       )                 (   3   )                               
 
         [0035]    where T represents the total number of vectors in Z i , and s ij  represents the state at time t in the most likely state sequence for the i th  source. The second log term in the sum does not depend on z ij , or the filter parameters, and therefore does not affect the optimization. Hence, maximizing Equation 3 is the same as maximizing the first log term.  
         [0036]    We make the simplifying assumption that this is equivalent to minimizing the distance between Z i  and the most likely sequence of vectors for the state sequence S i .  
         [0037]    When state output distributions in the HMM are modeled by a single Gaussian, the most likely sequence of vectors is simply the sequence of means for the states in the most likely state sequence.  
         [0038]    Hereinafter, we refer to this sequence of means as a target sequence  151  for the speaker. An objective function to be optimized in the optimization step  160  for the filter parameters  161  is defined by  
               Q   i     =       ∑     t   =   1     T          (         (       z     i                 t       -     m     s     i                 t       i       )     T          (       z     i                 t       -     m     s     i                 t       i       )       )               (   4   )                               
 
         [0039]    where the t th  vector in the target sequence m s     ij     t  is the mean of s it , the t th  state, in the most likely state sequence S i .  
         [0040]    Equations 2 and 4 indicate that Q i  is a function of h i . However, direct optimization of Q i  with respect to h i  is not possible due to the highly non-linear relationship between the two. Therefore, we optimize Q using an optimization method such as conjugate gradient descent.  
         [0041]    [0041]FIG. 2 shows the steps of the method  200  according to the invention.  
         [0042]    First, initialize  201  the filter parameters to h i [0]=1/N, and h i [k]=0 for k≠0-. and filter and sum the mixed signals  111  for each speaker using Equation 1.  
         [0043]    Second, extract  202  the feature vectors  141 .  
         [0044]    Third, determine  203  the state sequence, and the corresponding target sequence  151  for an optimization.  
         [0045]    Fourth, estimate  204  optimal filter parameters  161  with an optimization method such as conjugate gradient descent to optimize Equation 4.  
         [0046]    Fifth, re-filter and sum the signals with the optimized filter parameters. If the new objective function has not converged  206 , then repeat the third and fourth step  203 , until done  207 .  
         [0047]    Because the process minimizes a distance between the extracted features  141  and the target sequence  151 , the selection a good target is important.  
         [0048]    Target Estimation  
         [0049]    An ideal target is a sequence of Mel-spectral vectors obtained from clean uncorrupted recordings of the acoustic signals. All other targets are only approximations to the ideal target. To approximate this ideal target, we derive the target  151  from the HMMs  170  for that speaker&#39;s utterance. We do this by determining the best state sequence through the HMMs from the current estimate of the source&#39;s signal.  
         [0050]    A direct approach finds the most likely state sequence for the sequence of Mel-spectral vectors for the signal. Unfortunately, in the initial iterations of the process, before the filters  120  are fully optimized, the output  131  of the filter-and-sum array for any speaker contains a significant fraction of the signal from other speakers as well. As a result, naive alignment of the output to the HMMs results in a poor estimate of the target.  
         [0051]    Therefore, we also take into consideration the fact that the array output is a mixture of signals from all the sources. The HMM that represents this signal is a factorial HMM (FHMM) that is a cross-product of the individual HMMs for the various sources. In the FHMM, each state is a composition of one state from the HMMs for each of the sources, reflecting the fact that the individual sources&#39; signal can be in any of their respective states, and the final output is a combination of the output from these states.  
         [0052]    [0052]FIG. 3 shows the dynamics of the FHMM for the example of two speakers with two chains of HMMs  301 - 302 , one for each speaker. The HMMs operate with the feature vectors  141   
         [0053]    Let S i   k  represent the i th  state of the HMM for the k th  speaker, where kε[1,2]. S ij   kl  represents the factorial state obtained when the HMM for the k th  speaker is in state i, and that for the l th  speaker is in state j. The output density of S ij   kl  is a function of the output densities of its component states  
           P ( x|S   ij   kl )=ƒ( P ( X|S   i   k ),  P ( X|S   j   l ))  (5)  
         [0054]    The precise nature of the function θ( ) depends on the proportions to which the signals  103  from the speakers are mixed in the current estimate of the desired speaker&#39;s signal. This in turn depends on several factors including the original signal levels of the various speakers, and the degree of separation of the desired speaker effected by the current set of filters. Because these are difficult to determine in an unsupervised manner, ƒ( ) cannot be precisely determined.  
         [0055]    We do not attempt to estimate ƒ( ). Instead, the HMMs for the individual sources are constructed to have simple Gaussian state output densities. We assume that the state output density for any state of the FHMM is also a Gaussian whose mean is a linear combination of the means of the state output densities of the component states.  
         [0056]    We define m ij   kl , the mean of the Gaussian state output density of S ij   kl  as  
           m   ij   kl   =A   k   m   i   k   +A   l   m   j   l   (6)  
         [0057]    where m i   k  represents the D dimensional mean vector for S k , and A k  is a D×D weighting matrix.  
         [0058]    We consider three options for the covariance of a factorial state S ij   kl .  
         [0059]    All factorial states have a common diagonal covariance matrix C. i.e. the covariance of any factorial state S ij   kl  is given by C ij   kl =C. The covariance of S ij   kl  is given by C ij   kl =B(C i   k +C j   l ) where C i   k  is the covariance matrix for S i   k , and B is a diagonal matrix. is given by C ij   kl =B k C j   l +B l C j   l , where B k  is a diagonal matrix,  
         [0060]    B k =diag(b k ).  
         [0061]    We refer to the first approach as the global covariance approach and the latter two as the composed covariance approaches. The state output density of the factorial state S ij   kl  is now given by  
           P ( Z   t   |S   ij   kl )=| C   ij   kl | −1/2 (2π) −D/2   e   −1/2(Z     t     −m     ij       kl     )     t     (C     ij       kl     )     −1     (Z     t     −m     ij       kl     )   (7)  
         [0062]    The various A k  values and the covariance parameter values (C, B, or B k , depending on the covariance option considered) values are unknown, and are estimated from the current estimate of the speaker&#39;s signal. The estimation is performed using an expectation maximization (EM) process.  
         [0063]    In the expectation (E) step of the process, the a posteriori probabilities of the various factorial states, and thereby the a posteriori probabilities of the states of the HMMs for the speakers, are found. The factorial HMM has as many states as the product of the number of states in its component HMMs. Thus, direct computation of the (E) step is prohibitive.  
         [0064]    Therefore, we take a variational approach, see Ghahramani et al., “Factorial Hidden Markov Models,” Machine Learning, Vol. 29, pp. 245-275, Kluwer Academic Publishers, Boston  1997 . In the maximization (M) step of the process, the computed a posteriori probabilities are used to estimate the A k  as  
             A   =       ∑     i   =   1       N   k                         ∑     j   =   1       N   l                         ∑   t            (       Z   t              P     i                 j            (   t   )       ′          M   ′       )            (     M          ∑   t            (         P     i                 j            (   t   )                P     i                 j            (   t   )       ′       )          M   ′           )       -   1                       (   8   )                               
 
         [0065]    where A is a matrix composed by A 1  and A 2  as A=[A 1 , A 2 ], P ij  ( t ) is a vector whose i th  and (N k +j) th  values equal P(Z i |S i   k ) and P(Z i |S j   l ), and M is a block matrix in which blocks are formed by matrices composed by the means of the individual state output distributions.  
         [0066]    For the composed variance approach where C ij   kl =B k C i   k +B l C j   l , the diagonal component b k  of the matrix B k  is estimated in the n th  iteration of the EM algorithm as  
               b   n   k     =       ∑     t   ,   i   ,     j   =   1         T   ,     N   k     ,     N   l                             (       Z   t     -     m     i                 j       k                 l         )     ′            (     I   +         (       B     n   -   1     k          C   i   k       )       -   1            B     n   -   1     l          C   j   l         )       -   1            (       Z   t     -     m     i                 j       k                 l         )            p     i                 j            (   t   )                   (   9   )                               
 
         [0067]    where p ij  (t)=P(Z i |S ij   kl ).  
         [0068]    The common covariance C for the global covariance approach, and B for the first composed covariance approach can be similarly computed.  
         [0069]    After the EM process converges and the A k S, the covariance parameters (C, B, or B k , as appropriate) are determined, the best state sequence for the desired speaker can also be obtained from the FHMM, also using the variational approximation.  
         [0070]    The overall system to determine the target sequence  151  for a source works as follows. Using the feature vectors  141  from the unprocessed signal and the HMMs found using the transcriptions, parameters A and the covariance parameters (C, B, or B k , as appropriate) are iteratively updated using Equations 8 and 9, until the total log-likelihood converges.  
         [0071]    Thereafter, the most likely state sequence through the desired speaker&#39;s HMM is found. After the target  151  is obtained, the filters  120  are optimized, and the output  131  of the filter-and-sum array is used to re-estimate the target. The system converges when the target does not change on successive iterations. The final set of filters obtained is used to separate the source&#39;s acoustic signal.  
         [0072]    Effect of the Invention  
         [0073]    The invention provides a novel multi-channel speaker separation system and method that utilizes known statistical characteristics of the acoustic signals from the speakers to separate them.  
         [0074]    With the example system for two speakers, the system and method according to the invention improves the signal separation ratios (SSR) by 20 dB over simple delay-and-sum of the prior art. For the case where the signal levels of the speakers are different, the results are more dramatic, i.e., an improvement of 38 dB.  
         [0075]    [0075]FIG. 4A shows a mixed signal, and FIGS. 4B and 4C show two separated signals obtained by the method according to the invention. The signal separation obtained with the FHMM-based methods is comparable to that obtained with ideal-targets for the filter optimization. The composed-variance FHMM method converges to the final filters in fewer iterations than the method that uses a global covariance for all FHMM states.  
         [0076]    Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications may be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.