SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhong-Qiu Wang and Shinji Watanabe
Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA
wang.zhongqiu41@gmail.com
In reverberant conditions with multiple concurrent speakers, each microphone acquires a mixture signal of multiple speakers at a different location. In over-determined conditions where the microphones out-number speakers, we can narrow down the solutions to speaker images and realize unsupervised speech separation by leveraging each mixture signal as a constraint (i.e., the estimated speaker images at a microphone should add up to the mixture). Equipped with this insight, we propose UNSSOR, an algorithm for unsupervised neural speech separation by leveraging over-determined training mixtures. At each training step, we feed an input mixture to a deep neural network (DNN) to produce an intermediate estimate for each speaker, linearly filter the estimates, and optimize a loss so that, at each microphone, the filtered estimates of all the speakers can add up to the mixture to satisfy the above constraint. We show that this loss can promote unsupervised separation of speakers. The linear filters are computed in each sub-band based on the mixture and DNN estimates through the forward convolutive prediction (FCP) algorithm. To address the frequency permutation problem incurred by using sub-band FCP, a loss term based on minimizing intra-source magnitude scattering is proposed. Although UNSSOR requires over-determined training mixtures, we can train DNNs to achieve under-determined separation (e.g., unsupervised monaural speech separation). Evaluation results on two-speaker separation in reverberant conditions show the effectiveness and potential of UNSSOR.
In many machine learning and artificial intelligence applications, sensors, while recording, usually capture a mixture of desired and undesired signals. One example is the cocktail party problem (or speech separation) [1, 2], where, given a recorded mixture of the concurrent speech by multiple speakers, the task is to separate the mixture to individual speaker signals. Speech separation [3] has been dramatically advanced by deep learning, since deep clustering [4] and permutation invariant training (PIT) [5] solved the label permutation problem. They (and their subsequent studies [6-26]) are based on supervised learning, requiring paired clean speech and its corrupted signal generated via simulation, where clean speech is mixed with, for example, various noises and competing speakers at diverse energy and reverberation levels in simulated rooms [3]. The clean speech can provide an accurate, sample-level supervision for model training. Such simulated data, however, may not match the distribution of real-recorded test data in the target domain, and the resulting supervised learning based models would have generalization issues [27, 28]. How to train unsupervised neural speech separation systems on unlabelled target-domain mixtures is hence an important problem to study.
Training unsupervised speech separation models directly on monaural mixtures is an ill-posed task [2], since there is only one mixture signal observed but multiple speaker signals to reconstruct. The separation model would lack an accurate supervision (or regularizer) to figure out what desired sound objects (e.g., clean speaker signals) are, as there are infinite solutions where in each solution the
estimated sources can sum up to the mixture. Supposing that the separation model does not separate well and outputs a clean speaker signal plus some competing speech, noise or reverberation, would this output be viewed as a desired sound object? This is clear to humans, clear to supervised learning based models (by comparing the outputs with training labels), but not really clear to an unsupervised model. On the other hand, many studies [3-26] have observed that deep learning based supervised learning can achieve remarkable separation. In other words, with proper supervision, modern DNNs are capable of separating mixed speakers, but, in an unsupervised setup, there lacks an accurate supervision to unleash this capability. The key to successful unsupervised neural separation, we believe, is designing a clever supervision that can inform the model what desired sound objects are, and penalize the model if its outputs are not good and reward otherwise.
Our insight is that, in multi-microphone over-determined conditions where the microphones outnumber speakers, the ill-posed problem can be turned into a well-posed one, where a unique solution to the speakers exists (up to speaker permutation). This well-posed property (that a unique solution exists) can be leveraged as a supervison (or regularizer) to design loss functions that could inform the unsupervised separation model what desired sound objects are and promote separation of speakers.
Equipped with this insight, we perform unsupervised neural speech separation by leveraging multi-microphone over-determined training mixtures. Our DNNs can be trained directly on over-determined mixtures to realize over- and under-determined separation. The proposed algorithm, named UNSSOR, obtains strong separation performance on two-speaker separation. Our contributions include:
- We enforce a linear-filter constraint between each speaker's reverberant images at each microphone pair, turning the ill-posed problem into a well-posed one that can promote separation of speakers.
- We formulate unsupervised neural speech separation as a blind deconvolution problem, where both the speaker images and linear filters need to be estimated. We design loss functions motivated by the blind deconvolution problem, and propose a DNN approach to optimize the loss functions, where the speaker images are estimated via DNNs and the linear filters are estimated via a sub-band linear prediction algorithm named FCP [29] based on the mixture and DNN estimates.
- We propose a loss term, which minimizes a metric named intra-source magnitude scattering, to address the frequency permutation problem incurred when using sub-band FCP.
- Based on over-determined training mixtures, UNSSOR can be trained to perform under-determined separation (e.g., monaural unsupervised speech separation).
Various unsupervised neural separation algorithms, which do not require labelled mixtures, have been proposed. The most notable one is mixture invariant training (MixIT) [30-34], which first synthesizes training mixtures, each by mixing two existing mixtures, and then trains a DNN to separate the resulting mixed mixtures to underlying sources such that the separated sources can be partitioned into two groups and the separated sources in each group can sum up to one of the two existing mixtures (used for mixing). Care needs to be taken when synthesizing mixtures of mixtures. First, the sources in an existing mixture could have similar characteristics (e.g., similar reverberation patterns as the sources in an existing mixture are recorded in the same room) that are informative about which sources belong to the same existing mixture, and this would prevent MixIT from separating the sources [30, 35]. Second, it is unclear how to mix existing multi-channel mixtures, which are usually recorded by devices with different microphone geometry and number of microphones. Third, mixing existing mixtures with different reverberation characteristics would create unrealistic mixtures.
UNSSOR avoids the above issues by training unsupervised neural separation models directly on existing mixtures rather than on synthesized mixtures of mixtures. An earlier study related to this direction is the reverberation as supervision (RAS) algorithm [36], which addresses monaural two-speaker separation given binaural (two-channel) training mixtures. RAS performs magnitude-domain monaural separation directly on the left-ear mixture and then linearly filters the estimates through time-domain Wiener filtering so that the filtered estimates can approximate the right-ear mixture. RAS essentially does monaural separation and is effective at separating speakers in a semi-supervised learning setup, where a supervised PIT-based model is first trained and then used to bootstrap unsupervised training. It however fails completely in fully-unsupervised setup [36], unlike UNSSOR.
Conventional algorithms such as independent component analysis [37-41], independent vector analysis (IVA) [41-44] and spatial clustering [45-48] can perform unsupervised separation directly on existing mixtures. They perform separation based on a single test mixture at hand and are not
designed to learn speech patterns from large training data, while UNSSOR leverages DNNs to model speech patterns through unsupervised learning, which could result in better separation. Another difference is that UNSSOR can be configured for monarual, under-determined separation, while ICA, IVA and spatial clustering cannot. There are studies [49-52] training DNNs to approximate pseudolabels produced by conventional signal processing based separation models such as spatial clustering and blind source separation (BSS) algorithms. Their performance is however often limited since spatial clustering and BSS themselves are not good enough at separation. There are studies [53-55] training DNNs to separate multi-speaker mixtures such that the likelihood of observed mixtures under a probabilistic distribution derived based on the DNN separation results can be maximized. Such methods rely on statistical models for separation. They require costly iterative estimation of signal statistics at run time. In addition, the DNNs are only leveraged in estimating target magnitude, while phase estimation is only realized by spatial filtering.
Given a  $P$ -microphone mixture with  $C$  speakers in reverberation conditions, the physical model can be formulated using a system of linear equations in the short-time Fourier transform (STFT) domain:
$$
Y _ {p} (t, f) = \sum_ {c = 1} ^ {C} X _ {p} (c, t, f) + \varepsilon_ {p} (t, f), \text {f o r} p \in \{1, \dots , P \}, \tag {1}
$$
where  $t$  indexes  $T$  frames,  $f$  indexes  $F$  frames, and at microphone  $p$ , time  $t$  and frequency  $f$ ,  $Y_{p}(t,f)$ ,  $X_{p}(c,t,f)$  and  $\varepsilon_{p}(t,f) \in \mathbb{C}$  respectively denote the STFT coefficients of the mixture, reverberant image of speaker  $c$ , and non-speech signals. In the rest of this paper, we refer to the corresponding spectrogram when dropping the index  $c$ ,  $p$ ,  $t$  or  $f$ . We assume that  $\varepsilon$  is weak and stationary (e.g., a time-invariant Gaussian noise or simply modelling errors). Without loss of generality, we designate microphone 1 as the reference microphone. Our goal is to, in an unsupervised way, estimate each speaker's image at the reference microphone (i.e.,  $X_{1}(c)$  for each speaker  $c$ ) given the input mixture. We do not aim at dereverberation, instead targeting at maintaining the reverberation of each speaker.
Unsupervised separation based only on the observed mixture is difficult. There are infinite solutions to the above linear system where there are  $T \times F \times P$  equations (we have a mixture observation for each  $Y_{p}(t,f)$ ) but  $T \times F \times P \times C$  unknowns (we have one unknown for each  $X_{p}(c,t,f)$ ).
Our insight is that the number of unknowns can be dramatically reduced, if we enforce constraints to the speaker images at different microphones. Since  $X_{1}(c)$  and  $X_{p}(c)$  are both convolved versions of the dry signal of speaker  $c$ , there exists a linear filter between them such that convolving  $X_{1}(c)$  with the filter would reproduce  $X_{p}(c)$ . This convolutive relationship is a physical constraint, which can be leveraged to reduce the number of unknowns. Specifically, we formulate (1) as
$$
Y _ {1} (t, f) = \sum_ {c = 1} ^ {C} X _ {1} (c, t, f) + \varepsilon_ {1} (t, f),
$$
$$
Y _ {p} (t, f) = \sum_ {c = 1} ^ {C} \mathbf {g} _ {p} (c, f) ^ {\mathrm {H}} \widetilde {\mathbf {X}} _ {1} (c, t, f) + \varepsilon_ {p} ^ {\prime} (t, f), \text {f o r} p \in \{2, \dots , P \}, \tag {2}
$$
where  $\widetilde{\mathbf{X}}_1(c,t,f) = [X_1(c,t - A,f),\ldots ,X_1(c,t,f),\ldots ,X_1(c,t + B,f)]^\top \in \mathbb{C}^{A + 1 + B}$  stacks a window of  $E = A + 1 + B$  T-F units,  $\mathbf{g}_p(c,f)\in \mathbb{C}^E$  is the relative room impulse response (relative RIR) relating  $X_{1}(c)$  to  $X_{p}(c)$ , and  $(\cdot)^{\mathrm{H}}$  computes Hermitian transpose.  $\mathbf{g}_p(c,f)$  is not long (i.e.,  $E$  is small) [56] if microphone 1 and  $p$  are placed close to each other, which is the case for compact arrays.
An implication of this constraint is that the number of unknowns is reduced from  $T \times F \times P \times C$  to  $T \times F \times C + F \times (P - 1) \times E \times C^1$ , which can be smaller than the number of equations (i.e.,  $T \times F \times P$ ) when  $P > C$  (i.e., over-determined conditions) and when  $T$  is sufficiently large (i.e., the input mixture is reasonably long). In other words, this formulation suggests that (1) there exists a solution for separation, which is most consistent with the above linear system; and (2) in over-determined cases, it is possible to estimate the speaker images in an unsupervised way.
${}^{1}T \times  F \times  C$  is because there is one unknown for each  ${X}_{1}\left( {c,t,f}\right)$  ,and  $F \times  \left( {P - 1}\right)  \times  E \times  C$  is because  ${\mathbf{g}}_{p}\left( {c,f}\right)$  is  $E$  -tap and we have one such filter for each of  $P - 1$  microphone pairs for each frequency and speaker.
Figure 1: Illustration of UNSSOR (assuming  $P > C$  during training).
As  $\varepsilon$  is assumed weak, time-invariant and Gaussian, one way to find the solution is to compute an estimate that is most consistent with the linear system in (2) by solving the following problem:
$$
\underset {\mathbf {g} \cdot (\cdot , \cdot), X _ {1} (\cdot , \cdot , \cdot)} {\operatorname {a r g m i n}} \sum_ {t, f} \left| Y _ {1} (t, f) - \sum_ {c = 1} ^ {C} X _ {1} (c, t, f) \right| ^ {2} + \sum_ {p = 2} ^ {P} \sum_ {t, f} \left| Y _ {p} (t, f) - \sum_ {c = 1} ^ {C} \mathbf {g} _ {p} (c, f) ^ {\mathrm {H}} \widetilde {\mathbf {X}} _ {1} (c, t, f) \right| ^ {2}. \tag {3}
$$
This is a blind deconvolution problem [57], which is non-convex in nature and difficult to be solved if no prior knowledge is assumed about the relative RIRs or the speaker images, because both of them are unknown. In the next section, we propose a DNN-based approach, which can model speech patterns through unsupervised learning (and hence model speech priors), to tackle this problem.
Fig. 1 illustrates the proposed system. The DNN takes in the mixture at all the  $P$  microphones or at the reference microphone 1 as input and produces an intermediate estimate  $\hat{Z}(c)$  for each speaker  $c$ . FCP [29] is then performed on  $\hat{Z}(c)$  at each microphone  $p$  to compute a linear-filtering result, denoted as  $\hat{X}_p^{\mathrm{FCP}}(c)$ , which, we will describe, is essentially an estimate of the speaker image  $X_p(c)$ . After that, two loss functions are computed and combined for DNN training. This section describes the DNN configuration, loss functions, FCP filtering, and an extension for monaural separation.
4.1 DNN configurations
The intermediate estimate  $\hat{Z}(c)$  for each speaker  $c$  is obtained via complex spectral mapping [23, 58], where we stack the real and imaginary (RI) parts of the input mixture as features for the DNN to predict the RI parts of  $\hat{Z}(c)$ . For the DNN architecture, we employ TF-GridNet [23], which obtains strong results on supervised speech separation benchmarks. See Appendix I for more DNN details.
4.2 Mixture-constraint loss on filtered estimates
Following formulation in (3), we propose mixture-constraint (MC) loss, which is computed by filtering the DNN estimate  $\hat{Z}(c)$  of each speaker  $c$  to approximate the  $P$ -channel input mixture:
$$
\mathcal {L} _ {\mathrm {M C}} = \alpha_ {1} \sum_ {t, f} \mathcal {F} \left(Y _ {1} (t, f), \sum_ {c = 1} ^ {C} \hat {Z} (c, t, f)\right) + \sum_ {p = 2} ^ {P} \alpha_ {p} \sum_ {t, f} \mathcal {F} \left(Y _ {p} (t, f), \sum_ {c = 1} ^ {C} \hat {\mathbf {g}} _ {p} (c, f) ^ {\mathrm {H}} \widetilde {\hat {\mathbf {Z}}} (c, t, f)\right). \tag {4}
$$
In (4),  $\hat{\mathbf{Z}}(c, t, f)$  stacks a window of T-F units around  $\hat{Z}(c, t, f)$ , and  $\hat{\mathbf{g}}_p(c, f)$  is an estimated relative RIR computed based on  $\hat{Z}(c, \cdot, f)$  and the mixture  $Y_p(\cdot, f)$  through FCP [29]. Both of them will be described in the next sub-section.  $\alpha_p \in \mathbb{R}$  is a weighting term for microphone  $p$ . Following [23],  $\mathcal{F}(\cdot, \cdot)$  in (4) computes an absolute loss on the estimated RI components and their magnitude:
$$
\begin{array}{l} \mathcal {F} \left(Y _ {p} (t, f), \hat {Y} _ {p} (t, f)\right) = \frac {1}{\sum_ {t ^ {\prime} , f ^ {\prime}} | Y _ {p} \left(t ^ {\prime} , f ^ {\prime}\right) |} \left(\left| \operatorname {R e} \left(Y _ {p} (t, f)\right) - \operatorname {R e} \left(\hat {Y} _ {p} (t, f)\right) \right| \right. \\ \left. + \left| \operatorname {I m} \left(Y _ {p} (t, f)\right) - \operatorname {I m} \left(\hat {Y} _ {p} (t, f)\right) \right| + \left| \left| Y _ {p} (t, f) \right| - \left| \hat {Y} _ {p} (t, f) \right| \right|\right), \tag {5} \\ \end{array}
$$
where  $\mathrm{Re}(\cdot)$  and  $\mathrm{Im}(\cdot)$  respectively extract RI components and  $|\cdot|$  computes magnitude. The term  $1 / \sum_{t', f'} |Y_p(t', f')|$  balances the losses at different microphones and across training mixtures.
According to the discussion in Section 3, minimizing  $\mathcal{L}_{\mathrm{MC}}$  would encourage separation of speakers. We illustrate the loss surface of  $\mathcal{L}_{\mathrm{MC}}$  in Appendix B.
Compared to the mixture consistency term proposed in [59], our mixture-constraint loss has very different physical meanings and mathematical forms. See Appendix E for detailed discussions.
4.3 FCP for relative RIR estimation
To compute  $\mathcal{L}_{\mathrm{MC}}$ , we need to first estimate each of the relative RIRs,  $\hat{\mathbf{g}}_p(c,f)$ . In [29, 60], FCP is proposed to estimate the relative RIR relating direct-path signal to reverberant image for speech dereverberation. In this study, we employ FCP to estimate the relative RIR relating  $\hat{Z}(c)$  to the speaker image captured at each microphone  $p$  (i.e.,  $X_p(c)$ ).
Assuming speakers are non-moving, we estimate relative RIRs by solving the following problem:
$$
\hat {\mathbf {g}} _ {p} (c, f) = \underset {\mathbf {g} _ {p} (c, f)} {\arg \min } \sum_ {t} \frac {1}{\hat {\lambda} _ {p} (c , t , f)} \left| Y _ {p} (t, f) - \mathbf {g} _ {p} (c, f) ^ {\mathrm {H}} \tilde {\hat {\mathbf {Z}}} (c, t, f) \right| ^ {2}, \tag {6}
$$
where  $\mathbf{g}_p(c,f)\in \mathbb{C}^{I + 1 + J}$  is a  $K$ -tap (with  $K = I + 1 + J$ ) time-invariant FCP filter,  $\hat{\mathbf{Z}}(c,t,f) = [\hat{Z}(c,t - I,f),\dots,\hat{Z}(c,t,f),\dots,\hat{Z}(c,t + J,f)]^{\top}\in \mathbb{C}^{K}$  stacks  $I$  past and  $J$  future T-F units with the current one. Since the actual number of filter taps (i.e.,  $A$  and  $B$  defined in the text below (2)) is unknown, we set them to  $I$  and  $J$ , both of which are hyper-parameters to tune.  $\hat{\lambda}_p(c,t,f)$  is a weighting term balancing the importance of each T-F unit. Following [29], we define it as  $\hat{\lambda}_p(c,t,f) = \left(\frac{1}{P}\sum_{p' = 1}^{P}|Y_{p'}(t,f)|^2\right) + \xi \times \max \left(\frac{1}{P}\sum_{p' = 1}^{P}|Y_{p'}|^2\right)$ , where  $\xi (= 10^{-4}$  in this study) is used to floor the weighting term and  $\max(\cdot)$  extracts the maximum value of a spectrogram. (6) is a weighted linear regression problem. A closed-form solution can be readily computed:
$$
\hat {\mathbf {g}} _ {p} (c, f) = \left(\sum_ {t} \frac {1}{\hat {\lambda} _ {p} (c , t , f)} \widetilde {\hat {\mathbf {Z}}} (c, t, f) \widetilde {\hat {\mathbf {Z}}} (c, t, f) ^ {\mathsf {H}}\right) ^ {- 1} \sum_ {t} \frac {1}{\hat {\lambda} _ {p} (c , t , f)} \widetilde {\hat {\mathbf {Z}}} (c, t, f) (Y _ {p} (t, f)) ^ {*}, \quad (7)
$$
where  $(\cdot)^*$  computes complex conjugate. We then plug  $\hat{\mathbf{g}}_p(c,f)$  into (4) and compute the loss.
Note that to compute the relative RIR, ideally we should filter  $\hat{Z}(c)$  to approximate  $X_{p}(c)$  (i.e., replacing  $Y_{p}$  in (6) with  $X_{p}(c)$ ), but  $X_{p}(c)$  is unknown. In (6), we instead linearly filter  $\hat{Z}(c)$  to approximate  $Y_{p}$ , and earlier studies [29, 60] suggest that the resulting  $\hat{\mathbf{g}}_{p}(c,f)^{\mathrm{H}}\widetilde{\hat{\mathbf{Z}}} (c,t,f)$  would be an estimate of  $X_{p}(c,t,f)$ , if  $\hat{Z} (c)$  is reasonably accurate (see Appendix C for the derivation). We name the speaker image estimated this way as FCP-estimated image:
$$
\hat {X} _ {p} ^ {\mathrm {F C P}} (c, t, f) = \hat {\mathbf {g}} _ {p} (c, f) ^ {\mathrm {H}} \tilde {\hat {\mathbf {Z}}} (c, t, f). \tag {8}
$$
It is therefore reasonable to sum up the FCP-estimated images of all the speakers and define a loss between the summation and  $Y_{p}$  as in (4).
Although (6) appears similar to multi-channel linear prediction (MCLP) [61, 62] which is popular in conventional speech separation algorithms, we point out that they have very different physical meanings. We consider that (6) does forward filtering, where source estimates are filtered to approximate mixtures so that relative RIRs can be estimated, while MCLP does inverse filtering, where mixtures are filtered to approximate target sources and the filters are designed to suppress non-target signals. This difference results in non-trivial changes of the physical meanings of the computed filters (see also discussions in Section V.C of [29]).
Although there were earlier efforts in estimating relative RIRs [56], they are based on conventional signal processing techniques and the performance is usually limited due to strong assumptions on, and inaccurate estimation of, signal statistics.
4.4 Time alignment issues and alternative loss functions
In (4), we do not filter the DNN estimates when computing the loss on the first (reference) microphone. We expect this to result in a  $\hat{Z}(c)$  time-aligned with the speaker image  $X_{1}(c)$  (i.e.,  $\hat{Z}(c)$  is an estimate
of  $X_{1}(c)$ ). Since the reference microphone may not be the microphone closest to speaker  $c$ , it is best to use non-causal filters when filtering  $\hat{Z}(c)$  to approximate the reverberant image  $X_{p}(c)$  at non-reference microphones that are closer to source  $c$  than the reference microphone, and instead use causal filters for non-reference microphones that are farther². Since estimating which non-reference microphones are closer or farther to a source than the reference microphone is not an easy task and doing this would complicate our system, we can just choose to use non-causal filters for all the non-reference microphones. This could, however, limit the DNN's capability at separating the speakers, because the relative RIRs for some non-reference microphones (farther to source  $c$  than the reference microphone) are causal, and it may not be a good idea to assume non-causal filters.
To address this issue, we make a simple modification to the loss function in (4):
$$
\mathcal {L} _ {\mathrm {M C}} = \sum_ {p = 1} ^ {P} \alpha_ {p} \mathcal {L} _ {\mathrm {M C}, p} = \sum_ {p = 1} ^ {P} \alpha_ {p} \sum_ {t, f} \mathcal {F} \left(Y _ {p} (t, f), \sum_ {c = 1} ^ {C} \hat {\mathbf {g}} _ {p} (c, f) ^ {\mathrm {H}} \tilde {\hat {\mathbf {Z}}} (c, t, f)\right), \tag {9}
$$