Title: UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures
Venue: NeurIPS 2023
Accepted: yes
Keywords: Speech separation

microphone array processing

deep learning
Abstract: In reverberant conditions with multiple concurrent speakers, each microphone acquires a mixture signal of multiple speakers at a different location. In over-determined conditions where the microphones out-number speakers, we can narrow down the solutions to speaker images and realize unsupervised speech separation by leveraging each mixture signal as a constraint (i.e., the estimated speaker images at a microphone should add up to the mixture). Equipped with this insight, we propose UNSSOR, an algorithm for $\underline{u}$nsupervised $\underline{n}$eural $\underline{s}$peech $\underline{s}$eparation by leveraging $\underline{o}$ver-determined training mixtu$\underline{r}$es. At each training step, we feed an input mixture to a deep neural network (DNN) to produce an intermediate estimate for each speaker, linearly filter the estimates, and optimize a loss so that, at each microphone, the filtered estimates of all the speakers can add up to the mixture to satisfy the above constraint. We show that this loss can promote unsupervised separation of speakers. The linear filters are computed in each sub-band based on the mixture and DNN estimates through the forward convolutive prediction (FCP) algorithm. To address the frequency permutation problem incurred by using sub-band FCP, a loss term based on minimizing intra-source magnitude scattering is proposed. Although UNSSOR requires over-determined training mixtures, we can train DNNs to achieve under-determined separation (e.g., unsupervised monaural speech separation). Evaluation results on two-speaker separation in reverberant conditions show the effectiveness and potential of UNSSOR.

UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures

Abstract

In reverberant conditions with multiple concurrent speakers, each microphone acquires a mixture signal of multiple speakers at a different location. In overdetermined conditions where the microphones out-number speakers, we can narrow down the solutions to speaker images and realize unsupervised speech separation by leveraging each mixture signal as a constraint (i.e., the estimated speaker images at a microphone should add up to the mixture). Equipped with this insight, we propose UNSSOR, an algorithm for unsupervised neural speech separation by leveraging over-determined training mixtures. At each training step, we feed an input mixture to a deep neural network (DNN) to produce an intermediate estimate for each speaker, linearly filter the estimates, and optimize a loss so that, at each microphone, the filtered estimates of all the speakers can add up to the mixture to satisfy the above constraint. We show that this loss can promote unsupervised separation of speakers. The linear filters are computed in each sub-band based on the mixture and DNN estimates through the forward convolutive prediction (FCP) algorithm. To address the frequency permutation problem incurred by using sub-band FCP, a loss term based on minimizing intra-source magnitude scattering is proposed. Although UNSSOR requires over-determined training mixtures, we can train DNNs to achieve under-determined separation (e.g., unsupervised monaural speech separation). Evaluation results on two-speaker separation in reverberant conditions show the effectiveness and potential of UNSSOR.

1 Introduction

In many machine learning and artificial intelligence applications, sensors, while recording, usually capture a mixture of desired and undesired signals. One example is the cocktail party problem (or speech separation) [1, 2], where, given a recorded mixture of the concurrent speech by multiple speakers, the task is to separate the mixture to individual speaker signals. Speech separation [3] has been dramatically advanced by deep learning, since deep clustering [4] and permutation invariant training (PIT) [5] solved the label permutation problem. They (and their subsequent studies [6-26]) are based on supervised learning, requiring paired clean speech and its corrupted signal generated via simulation, where clean speech is mixed with, for example, various noises and competing speakers at diverse energy and reverberation levels in simulated rooms [3]. The clean speech can provide an accurate, sample-level supervision for model training. Such simulated data, however, may not match the distribution of real-recorded test data in the target domain, and the resulting supervised learning based models would have generalization issues [27, 28]. How to train unsupervised neural speech separation systems on unlabelled target-domain mixtures is hence an important problem to study.

Training unsupervised speech separation models directly on monaural mixtures is an ill-posed task [2], since there is only one mixture signal observed but multiple speaker signals to reconstruct. The separation model would lack an accurate supervision (or regularizer) to figure out what desired sound objects (e.g., clean speaker signals) are, as there are infinite solutions where in each solution the

estimated sources can sum up to the mixture. Supposing that the separation model does not separate well and outputs a clean speaker signal plus some competing speech, noise or reverberation, would this output be viewed as a desired sound object? This is clear to humans, clear to supervised learning based models (by comparing the outputs with training labels), but not really clear to an unsupervised model. On the other hand, many studies [3-26] have observed that deep learning based supervised learning can achieve remarkable separation. In other words, with proper supervision, modern DNNs are capable of separating mixed speakers, but, in an unsupervised setup, there lacks an accurate supervision to unleash this capability. The key to successful unsupervised neural separation, we believe, is designing a clever supervision that can inform the model what desired sound objects are, and penalize the model if its outputs are not good and reward otherwise.

Our insight is that, in multi-microphone over-determined conditions where the microphones outnumber speakers, the ill-posed problem can be turned into a well-posed one, where a unique solution to the speakers exists (up to speaker permutation). This well-posed property (that a unique solution exists) can be leveraged as a supervison (or regularizer) to design loss functions that could inform the unsupervised separation model what desired sound objects are and promote separation of speakers.

Equipped with this insight, we perform unsupervised neural speech separation by leveraging multimicrophone over-determined training mixtures. Our DNNs can be trained directly on over-determined mixtures to realize over- and under-determined separation. The proposed algorithm, named UNSSOR, obtains strong separation performance on two-speaker separation. Our contributions include:

We enforce a linear-filter constraint between each speaker's reverberant images at each microphone pair, turning the ill-posed problem into a well-posed one that can promote separation of speakers.

We formulate unsupervised neural speech separation as a blind deconvolution problem, where both the speaker images and linear filters need to be estimated. We design loss functions motivated by the blind deconvolution problem, and propose a DNN approach to optimize the loss functions, where the speaker images are estimated via DNNs and the linear filters are estimated via a sub-band linear prediction algorithm named FCP [29] based on the mixture and DNN estimates.

We propose a loss term, which minimizes a metric named intra-source magnitude scattering, to address the frequency permutation problem incurred when using sub-band FCP.

Based on over-determined training mixtures, UNSSOR can be trained to perform under-determined separation (e.g., monaural unsupervised speech separation).

2 Related work

Various unsupervised neural separation algorithms, which do not require labelled mixtures, have been proposed. The most notable one is mixture invariant training (MixIT) [30-34], which first synthesizes training mixtures, each by mixing two existing mixtures, and then trains a DNN to separate the resulting mixed mixtures to underlying sources such that the separated sources can be partitioned into two groups and the separated sources in each group can sum up to one of the two existing mixtures (used for mixing). Care needs to be taken when synthesizing mixtures of mixtures. First, the sources in an existing mixture could have similar characteristics (e.g., similar reverberation patterns as the sources in an existing mixture are recorded in the same room) that are informative about which sources belong to the same existing mixture, and this would prevent MixIT from separating the sources [30, 35]. Second, it is unclear how to mix existing multi-channel mixtures, which are usually recorded by devices with different microphone geometry and number of microphones. Third, mixing existing mixtures with different reverberation characteristics would create unrealistic mixtures.

UNSSOR avoids the above issues by training unsupervised neural separation models directly on existing mixtures rather than on synthesized mixtures of mixtures. An earlier study related to this direction is the reverberation as supervsion (RAS) algorithm [36], which addresses monaural twospeaker separation given binaural (two-channel) training mixtures. RAS performs magnitude-domain monaural separation directly on the left-ear mixture and then linearly filters the estimates through time-domain Wiener filtering so that the filtered estimates can approximate the right-ear mixture. RAS essentially does monaural separation and is effective at separating speakers in a semi-supervised learning setup, where a supervised PIT-based model is first trained and then used to bootstrap unsupervised training. It however fails completely in fully-unsupervised setup [36], unlike UNSSOR.

Conventional algorithms such as independent component analysis [37-41], independent vector analysis (IV A) [41-44] and spatial clustering [45-48] can perform unsupervised separation directly on existing mixtures. They perform separation based on a single test mixture at hand and are not

designed to learn speech patterns from large training data, while UNSSOR leverages DNNs to model speech patterns through unsupervised learning, which could result in better separation. Another difference is that UNSSOR can be configured for monarual, under-determined separation, while ICA, IVA and spatial clustering cannot. There are studies [49-52] training DNNs to approximate pseudolabels produced by conventional signal processing based separation models such as spatial clustering and blind source separation (BSS) algorithms. Their performance is however often limited since spatial clustering and BSS themselves are not good enough at separation. There are studies [53-55] training DNNs to separate multi-speaker mixtures such that the likelihood of observed mixtures under a probabilistic distribution derived based on the DNN separation results can be maximized. Such methods rely on statistical models for separation. They require costly iterative estimation of signal statistics at run time. In addition, the DNNs are only leveraged in estimating target magnitude, while phase estimation is only realized by spatial filtering.

3 Problem formulation

Given a P -microphone mixture with C speakers in reverberation conditions, the physical model can be formulated using a system of linear equations in the short-time Fourier transform (STFT) domain:

where t indexes T frames, f indexes F frames, and at microphone p , time t and frequency f , Y p ( t, f ) , X p ( c, t, f ) and ε p ( t, f ) ∈ C respectively denote the STFT coefficients of the mixture, reverberant image of speaker c , and non-speech signals. In the rest of this paper, we refer to the corresponding spectrogram when dropping the index c , p , t or f . We assume that ε is weak and stationary (e.g., a time-invariant Gaussian noise or simply modelling errors). Without loss of generality, we designate microphone 1 as the reference microphone. Our goal is to, in an unsupervised way, estimate each speaker's image at the reference microphone (i.e., X 1 ( c ) for each speaker c ) given the input mixture. We do not aim at dereverberation, instead targeting at maintaining the reverberation of each speaker.

Unsupervised separation based only on the observed mixture is difficult. There are infinite solutions to the above linear system where there are T × F × P equations (we have a mixture observation for each Y p ( t, f ) ) but T × F × P × C unknowns (we have one unknown for each X p ( c, t, f ) ).

Our insight is that the number of unknowns can be dramatically reduced, if we enforce constraints to the speaker images at different microphones. Since X 1 ( c ) and X p ( c ) are both convolved versions of the dry signal of speaker c , there exists a linear filter between them such that convolving X 1 ( c ) with the filter would reproduce X p ( c ) . This convolutive relationship is a physical constraint, which can be leveraged to reduce the number of unknowns. Specifically, we formulate (1) as

where ˜ X 1 ( c, t, f ) = [ X 1 ( c, t -A,f ) , . . . , X 1 ( c, t, f ) , . . . , X 1 ( c, t + B,f )] T ∈ C A +1+ B stacks a window of E = A +1+ B T-F units, g p ( c, f ) ∈ C E is the relative room impulse response (relative RIR) relating X 1 ( c ) to X p ( c ) , and ( · ) H computes Hermitian transpose. g p ( c, f ) is not long (i.e., E is small) [56] if microphone 1 and p are placed close to each other, which is the case for compact arrays.

An implication of this constraint is that the number of unknowns is reduced from T × F × P × C to T × F × C + F × ( P -1) × E × C 1 , which can be smaller than the number of equations (i.e., T × F × P ) when P > C (i.e., over-determined conditions) and when T is sufficiently large (i.e., the input mixture is reasonably long). In other words, this formulation suggests that (1) there exists a solution for separation, which is most consistent with the above linear system; and (2) in over-determined cases, it is possible to estimate the speaker images in an unsupervised way.

Microphone

!

…

Microphone

"

DNN

multi-channel

input:

!

&

, … , !

!

, … , !

'

or

monaural

input:

!

&

"

#(%)

"

#(+)

"

# 1

FCP

FCP

FCP

,

3

!

()*

1

, 3

!

()*

%

, 3

!

()*

+

⨁

, !

!

ℒ

"),!

!

!

ℒ

.45/6.6

⨁

…

…

…

…

ℒ

#+"+,!

As ε is assumed weak, time-invariant and Gaussian, one way to find the solution is to compute an estimate that is most consistent with the linear system in (2) by solving the following problem:

This is a blind deconvolution problem [57], which is non-convex in nature and difficult to be solved if no prior knowledge is assumed about the relative RIRs or the speaker images, because both of them are unknown. In the next section, we propose a DNN-based approach, which can model speech patterns through unsupervised learning (and hence model speech priors), to tackle this problem.

4 Method

Fig. 1 illustrates the proposed system. The DNN takes in the mixture at all the P microphones or at the reference microphone 1 as input and produces an intermediate estimate ˆ Z ( c ) for each speaker c . FCP [29] is then performed on ˆ Z ( c ) at each microphone p to compute a linear-filtering result, denoted as ˆ X FCP p ( c ) , which, we will describe, is essentially an estimate of the speaker image X p ( c ) . After that, two loss functions are computed and combined for DNN training. This section describes the DNN configuration, loss functions, FCP filtering, and an extension for monaural separation.

4.1 DNN configurations

The intermediate estimate ˆ Z ( c ) for each speaker c is obtained via complex spectral mapping [23, 58], where we stack the real and imaginary (RI) parts of the input mixture as features for the DNN to predict the RI parts of ˆ Z ( c ) . For the DNN architecture, we employ TF-GridNet [23], which obtains strong results on supervised speech separation benchmarks. See Appendix I for more DNN details.

4.2 Mixture-constraint loss on filtered estimates

Following formulation in (3), we propose mixture-constraint (MC) loss, which is computed by filtering the DNN estimate ˆ Z ( c ) of each speaker c to approximate the P -channel input mixture:

In (4), ˜ ˆ Z ( c, t, f ) stacks a window of T-F units around ˆ Z ( c, t, f ) , and ˆ g p ( c, f ) is an estimated relative RIR computed based on ˆ Z ( c, · , f ) and the mixture Y p ( · , f ) through FCP [29]. Both of them will be described in the next sub-section. α p ∈ R is a weighting term for microphone p . Following [23], F ( · , · ) in (4) computes an absolute loss on the estimated RI components and their magnitude:

where Re ( · ) and Im ( · ) respectively extract RI components and |·| computes magnitude. The term 1 / ∑ t ′ ,f ′ | Y p ( t ′ , f ′ ) | balances the losses at different microphones and across training mixtures.

According to the discussion in Section 3, minimizing L MC would encourage separation of speakers. We illustrate the loss surface of L MC in Appendix B.

Compared to the mixture consistency term proposed in [59], our mixture-constraint loss has very different physical meanings and mathematical forms. See Appendix E for detailed discussions.

4.3 FCP for relative RIR estimation

To compute L MC, we need to first estimate each of the relative RIRs, ˆ g p ( c, f ) . In [29, 60], FCP is proposed to estimate the relative RIR relating direct-path signal to reverberant image for speech dereverberation. In this study, we employ FCP to estimate the relative RIR relating ˆ Z ( c ) to the speaker image captured at each microphone p (i.e., X p ( c ) ).

Assuming speakers are non-moving, we estimate relative RIRs by solving the following problem:

where g p ( c, f ) ∈ C I +1+ J is a K -tap (with K = I +1+ J ) time-invariant FCP filter, ˜ ˆ Z ( c, t, f ) = [ ˆ Z ( c, t -I, f ) , . . . , ˆ Z ( c, t, f ) , . . . , ˆ Z ( c, t + J, f )] T ∈ C K stacks I past and J future T-F units with the current one. Since the actual number of filter taps (i.e., A and B defined in the text below (2)) is unknown, we set them to I and J , both of which are hyper-parameters to tune. ˆ λ p ( c, t, f ) is a weighting term balancing the importance of each T-F unit. Following [29], we define it as ˆ λ p ( c, t, f ) = ( 1 P ∑ P p ′ =1 | Y p ′ ( t, f ) | 2 ) + ξ × max ( 1 P ∑ P p ′ =1 | Y p ′ | 2 ) , where ξ ( = 10 -4 in this study) is used to floor the weighting term and max ( · ) extracts the maximum value of a spectrogram. (6) is a weighted linear regression problem. A closed-form solution can be readily computed:

where ( · ) ∗ computes complex conjugate. We then plug ˆ g p ( c, f ) into (4) and compute the loss.

Note that to compute the relative RIR, ideally we should filter ˆ Z ( c ) to approximate X p ( c ) (i.e., replacing Y p in (6) with X p ( c ) ), but X p ( c ) is unknown. In (6), we instead linearly filter ˆ Z ( c ) to approximate Y p , and earlier studies [29, 60] suggest that the resulting ˆ g p ( c, f ) H ˜ ˆ Z ( c, t, f ) would be an estimate of X p ( c, t, f ) , if ˆ Z ( c ) is reasonably accurate (see Appendix C for the derivation). We name the speaker image estimated this way as FCP-estimated image :

It is therefore reasonable to sum up the FCP-estimated images of all the speakers and define a loss between the summation and Y p as in (4).

Although (6) appears similar to multi-channel linear prediction (MCLP) [61, 62] which is popular in conventional speech separation algorithms, we point out that they have very different physical meanings. We consider that (6) does forward filtering , where source estimates are filtered to approximate mixtures so that relative RIRs can be estimated, while MCLP does inverse filtering , where mixtures are filtered to approximate target sources and the filters are designed to suppress non-target signals. This difference results in non-trivial changes of the physical meanings of the computed filters (see also discussions in Section V.C of [29]).

Although there were earlier efforts in estimating relative RIRs [56], they are based on conventional signal processing techniques and the performance is usually limited due to strong assumptions on, and inaccurate estimation of, signal statistics.

4.4 Time alignment issues and alternative loss functions

In (4), we do not filter the DNN estimates when computing the loss on the first (reference) microphone. We expect this to result in a ˆ Z ( c ) time-aligned with the speaker image X 1 ( c ) (i.e., ˆ Z ( c ) is an estimate

of X 1 ( c ) ). Since the reference microphone may not be the microphone closest to speaker c , it is best to use non-causal filters when filtering ˆ Z ( c ) to approximate the reverberant image X p ( c ) at non-reference microphones that are closer to source c than the reference microphone, and instead use causal filters for non-reference microphones that are farther 2 . Since estimating which non-reference microphones are closer or farther to a source than the reference microphone is not an easy task and doing this would complicate our system, we can just choose to use non-causal filters for all the non-reference microphones. This could, however, limit the DNN's capability at separating the speakers, because the relative RIRs for some non-reference microphones (farther to source c than the reference microphone) are causal, and it may not be a good idea to assume non-causal filters.

To address this issue, we make a simple modification to the loss function in (4):

where the difference is that we also filter the DNN estimates when computing the loss on the reference microphone, and we constrain ˆ g p ( c, f ) to be causal and that ˜ ˆ Z ( c, t, f ) only stacks current and past frames. This way, the resulting ˆ Z ( c ) would not be time-aligned with the revererberant image captured at the reference microphone (i.e., X 1 ( c ) ) or any other non-reference microphones. Because of the causal filtering, ˆ Z ( c ) would be more like an estimate of the reverberant image captured by a virtual microphone that is closer to speaker c than all the P microphones (see Appendix G.1 for an interpretation). It would contain less reverberation of speaker c than any of the speaker images captured by the P microphones due to the causal filtering.

To produce an estimate that is time-aligned with the reverberant image at a microphone (e.g., X p ( c ) ), we use the FCP-estimated image computed in (8) (i.e., ˆ X FCP p ( c ) ) as the output.

4.5 Addressing frequency permutation problem

In (4) and (9), FCP is performed in each frequency independently from the others. Even though the speakers are separated at each frequency, the separation results of the same speaker at different frequencies may however not be grouped into the same output spectrogram (see an example in Appendix D). This is known as the frequency permutation problem [41], which has been studied for decades in frequency-domain blind source separation algorithms such as frequency-domain independent component analysis [37-41] and spatial clustering [45-48]. Popular solutions for frequency alignment are designed by leveraging cross-frequency correlation of spectral patterns [47, 64] and direction-of-arrival estimation [65]. However, these solutions are often empirical and have a complicated design. They can be used to post-process DNN estimates for frequency alignment, but it is not easy to integrate them with UNSSOR for joint training. This section proposes a loss term, with which the trained DNN can learn to produce target estimates without frequency permutation.

To deal with frequency permutation, IVA [42-44] assumes that, at each frame, the de-mixed outputs at all the frequencies follow a complex Gaussian distribution with a shared variance term across frequencies: w ( c, f ) H Y ( t, f ) ∼ N (0 , D ( t, c )) , where w ( c, f ) ∈ C P is the de-mixing weight vector (in a time-invariant de-mixing matrix) for speaker c at frequency f , and D ( t, c ) ∈ R is the shared variance term, which is assumed time-variant. When maximum likelihood estimation is performed to estimate the de-mixing matrix, the variance term shared across all the frequencies is found very effective at solving the frequency permutation problem [41-44].

Motivated by IVA, we design the following loss term, named intra-source magnitude scattering (ISMS), to alleviate the frequency permutation problem in DNN outputs:

where ˆ X FCP p is computed via (8), ˆ X FCP p ( c, t, · ) ∈ C F , and var ( · ) computes the variance of the values in a vector. At each frame, we essentially want the the magnitudes of the estimated spectrogram of each speaker (i.e., ˆ X FCP p ( c, t, · ) ) to have a small intra-source variance. The rationale is that, when

frequency permutation happens, ˆ X FCP p ( c, t, · ) would contain multiple sources, and the resulting variance would be larger than that computed when ˆ X FCP p ( c, t, · ) contains only one source. L ISMS echoes IVA's idea of assuming a shared variance term across all the frequencies. If the ratio in (10) becomes smaller, it indicates that the magnitudes of ˆ X FCP p ( c, t, · ) are more centered around their mean. This is similar to optimizing the likelihood of ˆ X FCP p ( c, t, · ) under a Gaussian distribution with a variance term shared across all the frequencies. In (10), a logarithmic compression is applied, since log-compressed magnitudes better follow Gaussian distribution than raw magnitudes [66].

We combine L ISMS with L MC in (9) for DNN training, using a weighting term γ ∈ R :

4.6 Training UNSSOR for monaural unsupervised separation

UNSSOR can be trained for monaural unsupervised separation by only feeding the mixture at the reference microphone to the DNN but still computing the loss on multiple microphones. Fig. 1 illustrates the idea. At run time, the trained system performs monaural under-determined separation, and multi-microphone over-determined mixtures are only required for DNN training. The loss computed at multiple microphones could guide the DNN to exploit monaural spectro-temporal patterns for separation, even in an unsupervised setup.

5 Experimental setup

We validate the proposed algorithms on two-speaker separation in reverberant conditions based on the six-channel SMS-WSJ dataset [67] (see Appendix A for its details). This section describes the baseline systems and evaluation setup. See Appendix I for miscellaneous system and DNN setup.

5.1 Baselines

The baselines include conventional unsupervised separation algorithms, an improved version of RAS, MixIT, and supervised learning based models.

We include spatial clustering [46-48] for comparison. We use a public implementation 3 [68], which leverages complex angular-central Gaussian mixture models [48] for sub-band spatial clustering and exploits inter-frequency correlation of cluster posteriors [46, 64] for frequency alignment. The number of sources is set to three, one of which is used for garbage collection, following [44]. After obtaining the estimates, we discard the one with the lowest energy. The STFT window size is tuned to 128 ms and hop size to 16 ms.

We include IVA [41, 44] for comparison. We use the public implementations provided by the torchiva toolkit [69]. We use the default spherical Laplacian model to model source distribution. In overdetermined cases, the number of sources is set to three and we discard the estimate with the lowest energy, similarly to the setup in the spatial clustering baseline 4 . The STFT window size is tuned to 256 ms and hop size to 32 ms.

We propose a novel variant of the RAS algorithm [36] for comparison. Appendix H discusses the differences between UNSSOR and RAS. Since RAS cannot achieve unsupervised separation, we improve it by computing loss on multi-microphone mixtures, and name the new algorithm as improved RAS (iRAS). We employ the time-domain Wiener filtering (WF) technique in [36] to filter re-synthesized time-domain estimates ˆ z ( c ) = iSTFT ( ˆ Z ( c )) , where ˆ Z ( c ) is produced by TF-GridNet. The loss is defined as:

with ∗ denoting linear convolution, y p the time-domain mixture at microphone p , and ˆ h p ( c ) a time-domain Wiener filter computed by solving the following problem:

Notes : the rows shows in grey indicate using oracle information or using supervised models.

which is quadratic and has a closed-form solution. The separation result is computed as ˆ x WF p ( c ) =

ˆ h p ( c ) ∗ ˆ z ( c ) . Following [36], we use 512 filter taps, and filter the future 100 , the current, and the past 411 samples (i.e., non-causal filtering). We can also filter the current and the past 511 samples (i.e., causal filtering), and additionally experiment with a filter length (in time) same as the length of the FCP filters (see Appendix J).

For comparison, we also include MixIT [30], which requires using synthetic mixtures of mixtures.

We report the result of using supervised learning, where PIT [5] is used to address the permutation problem. This result can be viewed as a performance upper bound of unsupervised separation.

We use the same DNN and training configurations as those in UNSSOR for a fair comparison.

5.2 Evaluation setup and metrics

We designate the first microphone as the reference microphone, and use the time-domain signal corresponding to X 1 ( c ) of each speaker c for metric computation. The evaluation metrics include signal-to-distortion ratio (SDR) [70], scale-invariant SDR (SI-SDR) [71], perceptual evaluation of speech quality (PESQ) [72], and extended short-time objective intelligibility (eSTOI) [73]. SI-SDR and SDR evaluate the sample-level accuracy of predicted signals, and PESQ and eSTOI are objective metrics of speech quality and intelligibility respectively. For all the metrics, the higher, the better.

6 Evaluation results

This section reports evaluation results on SMS-WSJ and compares the performance of various setups.

6.1 Effectiveness of UNSSOR at promoting separation

Table 1 and 2 respectively report the results of using six- and three-microphone input and loss. After hyper-parameter tuning, in default we use the loss in (9) for DNN training, set I = 19 and J = 0 (defined below (6)) for FCP (i.e., causal FCP filtering with 20 taps), and set α p = 1 (meaning no

weighting is applied for different microphones). For the 3-microphone case, we use the mixtures at the first, third, and fifth microphones for training and testing. Notice that for two-speaker separation, the cases with six or three microphones are both over-determined.

In both tables, from row 1a we observe that UNSSOR produces reasonable separation of speakers, improving the SDR from 0 . 1 to, for example, 12 . 5 dB in Table 1, but its output suffers from the frequency permutation problem (see Appendix D for an example). In row 1c, we use oracle target speech to obtain oracle frequency alignment and observe much better results over 1a. This shows the effectiveness of L MC at promoting separation of speakers and the severity of the frequency permutation problem. In row 1b, we use a frequency alignment algorithm (same as that used in the spatial clustering baseline) [46, 64] to post-process the separation results of 1a. This algorithm leads to impressive frequency alignment (see 1b vs. 1c), but it is empirical and has a complicated design.

6.2 Effectiveness of ISMS loss at addressing frequency permutation problem

We train DNNs using L MC+ISMS defined in (11). In each case (i.e, six- and three-microphone), we separately tune the weighting term γ in (11) based on the validation set. In both table, comparing row 2a-2c with 1a-1c, we observe that including L ISMS is very effective at dealing with the frequency permutation problem, yielding almost the same performance as using oracle frequency alignment.

6.3 Results of training UNSSOR for monaural unsupervised separation

̸

Table 3 and 4 use the mixture only at the reference microphone 1 as the network input, while computing the loss respectively on three and six microphones. We tune J to 1 (i.e., non-causal FCP filter), considering that, for a specific target speaker, the reference microphone may not be the microphone closest to that speaker 5 . See an interpretation on why non-causal filtering is needed in Appendix G.2. We still set the microphone weight α p to 1 . 0 for non-reference microphones (i.e., when p = 1 ), but tune α 1 to a smaller value based on the validation set. Without using a smaller α 1 , we found that the DNN easily overfits to microphone 1 , as we use the mixture at microphone 1 as the only input and compute the MC loss also on the mixture at microphone 1 . The DNN can just aggressively optimize L MC ,p to zero at microphone 1 and not optimize that at other microphones.

From row 1a of both tables, strong performance is observed in this under-determined setup, indicating that the multi-microphone loss can inform the DNN what desired target sound objects are and the DNN can learn to model spectral patterns in speech for unsupervised separation. In addition, the result in 1a of Table 3 is better than that in Table 4. This indicates that using more microphones as constraints in the proposed MC loss can elicit better separation.

6.4 Comparison with other methods and supplementary results

In Table 1-4, we compare the performance of UNSSOR with spatial clustering, IVA, iRAS, and supervised PIT-based models. In Appendix J, we compare UNSSOR and iRAS when they use the same filter length (in time). UNSSOR shows better performance than previous unsupervised separation models that can be performed or trained directly on mixtures. A sound demo is available at this link 6 . UNSSOR is worse than supervised PIT but the performance is reasonably strong. For example, in row 2a of Table 2, UNSSOR obtains 15 . 4 dB SDR on the test set, which is close to the 16 . 8 dB result obtained by supervised PIT in 4a. In Appendix L, we compare the results of UNSSOR with MixIT [30] and observe better performance.

Wethink that the effectiveness of the proposed system comes from both the strong modeling capability of TF-GridNet and the UNSSOR mechanism itself. In Appendix K, we use UNSSOR with other separation models such as TCN-DenseUNet [74] and DPRNN [75], which are known to have weaker modelling capability than TF-GridNet in supervised separation tasks (see for example [23]). We observe that the separation performance is worse but still reasonable.

7 Limitations

Our study shows the strong potential of UNSSOR for unsupervised speech separation. There are, however, several weaknesses we need to address in future research. First, we assume that sources are directional point sources so that each relative RIR can be modelled using a short filter, and diffuse sources are not considered. Second, we assume that sources are non-moving within each utterance so that we can use time-invariant FCP filters. Third, we assume that the number of sources is known and the sources are fully-overlapped. Fourth, only measurement or modeling noise is considered and realistic directional or diffuse background noises with strong energy are not included. Although these assumptions are also made in many algorithms such as IVA, spatial clustering, RAS and iRAS, they need to be addressed to realize more practical and robust speech separation systems.

8 Conclusion

We have proposed UNSSOR for unsupervised neural speech separation. We show that it is possible to train unsupervised models directly on mixtures, if the mixtures are over-determined. We have proposed mixture-constraint loss functions, which leverage multi-microphone mixtures as constraints, to promote separation of speakers. We find that minimizing ISMS can alleviate the frequency permutation problem. Although UNSSOR requires over-determined training mixtures, it can be trained to perform under-determined unsupervised separation. Future research will combine UNSSOR with semi-supervised learning, evaluate it on real-recorded noisy-reverberant data such as CHiME6 [76], AMI [77] and AliMeeting [78], and address the limitations described in Section 7.

In closing, we emphasize that a key scientific contribution of this paper is that the over-determined property afforded by having more microphones than speakers can narrow down the solutions to the underlying sources, and this property can be leveraged to design a supervision to train DNNs to model speech patterns via unsupervised learning and realize unsupervised separation. This meta-idea, we believe, would motivate the design of many algorithms in future research on neural source separation.

Acknowledgments and Disclosure of Funding

We would like to thank Dr. Robin Scheibler at LINE Corporation for constructive discussions on IV A, and Dr. Samuele Cornell for helpful discussions on MixIT. This research is part of the Delta research computing project, which is supported by the National Science Foundation (Award OCI 2005572 ) and the State of Illinois. Delta is a joint effort of the University of Illinois at Urbana-Champaign and its National Center for Supercomputing Applications. We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the RTX 8000 GPUs used in this research.

References

C. E. Cherry, 'Some Experiments on the Recognition of Speech, with One and with Two Ears,' Journal of the Acoustical Society of America , vol. 25, no. 5, pp. 975-979, 1953.

J. H. McDermott, 'The Cocktail Party Problem,' Current Biology , vol. 19, no. 22, pp. 1024-1027, 2009.

D. Wang and J. Chen, 'Supervised Speech Separation Based on Deep Learning: An Overview,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 26, no. 10, pp. 1702-1726, 2018.

J. R. Hershey, Z. Chen, J. Le Roux, and S. Watanabe, 'Deep Clustering: Discriminative Embeddings for Segmentation and Separation,' in Proc. ICASSP , 2016, pp. 31-35.

M. Kolbæk, D. Yu, Z.-H. Tan, and J. Jensen, 'Multitalker Speech Separation with Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 25, no. 10, pp. 1901-1913, 2017.

Y. Luo, Z. Chen, and N. Mesgarani, 'Speaker-Independent Speech Separation with Deep Attractor Network,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 26, no. 4, pp. 787-796, 2018.

Z.-Q. Wang and D. Wang, 'Combining Spectral and Spatial Features for Deep Learning Based Blind Speaker Separation,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 27, no. 2, pp. 457-468, 2019.

A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein, 'Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation,' ACM Transactions on Graphics , vol. 37, no. 4, 2018.

Y. Liu and D. Wang, 'Divide and Conquer: A Deep CASA Approach to Talker-Independent Monaural Speaker Separation,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 27, no. 12, pp. 2092-2102, 2019.

Y. Luo and N. Mesgarani, 'Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 27, no. 8, pp. 1256-1266, 2019.

Z.-Q. Wang, K. Tan, and D. Wang, 'Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspective,' in Proc. ICASSP , 2019, pp. 71-75.

K. Žmolíková, M. Delcroix, K. Kinoshita, T. Ochiai, T. Nakatani, L. Burget, and J. ˇ Cernocký, 'SpeakerBeam: Speaker Aware Neural Network for Target Speaker Extraction in Speech Mixtures,' IEEE Journal of Selected Topics in Signal Processing , vol. 13, no. 4, pp. 800-814, 2019.

I. Kavalerov, S. Wisdom, H. Erdogan, B. Patton, K. Wilson, J. Le Roux, and J. R. Hershey, 'Universal Sound Separation,' in Proc. WASPAA , 2019, pp. 175-179.

D. Yin, C. Luo, Z. Xiong, and W. Zeng, 'PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network,' in Proc. AAAI , 2020, pp. 9458-9465.

Y. Xu, M. Yu, S.-X. Zhang, L. Chen, C. Weng, J. Liu, and D. Yu, 'Neural Spatio-Temporal Beamformer for Target Speech Separation,' in Proc. Interspeech , 2020, pp. 56-60.

T. Jenrungrot, V. Jayaram, S. Seitz, and I. Kemelmacher-Shlizerman, 'The Cone of Silence: Speech Separation by Localization,' in Proc. NeurIPS , 2020.

C. Tang, C. Luo, Z. Zhao, W. Xie, and W. Zeng, 'Joint Time-Frequency and Time Domain Learning for Speech Enhancement,' in Proc. IJCAI , 2020, pp. 3816-3822.

E. Nachmani, Y. Adi, and L. Wolf, 'Voice Separation with An Unknown Number of Multiple Speakers,' in Proc. ICML , 2020, pp. 7121-7132.

N. Zeghidour and D. Grangier, 'Wavesplit: End-to-End Speech Separation by Speaker Clustering,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, pp. 2840-2849, 2021.

Z. Chen, T. Yoshioka, L. Lu, T. Zhou, Z. Meng, Y. Luo, J. Wu, X. Xiao, and J. Li, 'Continuous Speech Separation: Dataset and Analysis,' in Proc. ICASSP , 2020, pp. 7284-7288.

Z. Zhang, Y. Xu, M. Yu, S. X. Zhang, L. Chen, D. S. Williamson, and D. Yu, 'Multi-Channel Multi-Frame ADL-MVDR for Target Speech Separation,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, pp. 3526-3540, 2021.

J. Rixen and M. Renz, 'SFSRNet: Super-Resolution for Single-Channel Audio Source Separation,' in Proc. AAAI , 2022.

Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe, 'TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 31, pp. 3221-3236, 2023.

Y. Masuyama, X. Chang, W. Zhang, S. Cornell, Z.-Q. Wang, N. Ono, Y. Qian, and S. Watanabe, 'Exploring The Integration of Speech Separation and Recognition with Self-Supervised Learning Representation,' in Proc. WASPAA , 2023, pp. 1-5.

K. Saijo, W. Zhang, Z.-Q. Wang, S. Watanabe, T. Kobayashi, and T. Ogawa, 'A Single Speech Enhancement Model Unifying Dereverberation, Denoising, Speaker Counting, Separation, and Extraction,' in Proc. ASRU , 2023.

W. Zhang, K. Saijo, Z.-Q. Wang, S. Watanabe, and Y. Qian, 'Toward Universal Speech Enhancement for Diverse Input Conditions,' in Proc. ASRU , 2023.

A. Pandey and D. Wang, 'On Cross-Corpus Generalization of Deep Learning Based Speech Enhancement,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 28, pp. 2489-2499, 2020.

W. Zhang, J. Shi, C. Li, S. Watanabe, and Y. Qian, 'Closing The Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditions,' in Proc. WASPAA , 2021, pp. 146-150.

Z.-Q. Wang, G. Wichern, and J. Le Roux, 'Convolutive Prediction for Monaural Speech Dereverberation and Noisy-Reverberant Speaker Separation,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, pp. 3476-3490, 2021.

S. Wisdom, E. Tzinis, H. Erdogan, R. J. Weiss, K. Wilson, and J. R. Hershey, 'Unsupervised Sound Separation using Mixture Invariant Training,' in Proc. NeurIPS , 2020.

E. Tzinis, S. Wisdom, A. Jansen, S. Hershey, T. Remez, D. P. W. Ellis, and J. R. Hershey, 'Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds,' in Proc. ICLR , 2021.

E. Tzinis, Y. Adi, V. K. Ithapu, B. Xu, P. Smaragdis, and A. Kumar, 'RemixIT: Continual Self-Training of Speech Enhancement Models via Bootstrapped Remixing,' IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1329-1341, 2022.

E. Tzinis, S. Wisdom, T. Remez, and J. R. Hershey, 'AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation,' in Proc. ECCV , 2022, pp. 368-385.

K. Saijo and T. Ogawa, 'Self-Remixing: Unsupervised Speech Separation via Separation and Remixing,' in Proc. ICASSP , 2023.

A. Sivaraman, S. Wisdom, H. Erdogan, and J. R. Hershey, 'Adapting Speech Separation To Real-World Meetings Using Mixture Invariant Training,' in Proc. ICASSP , 2022, pp. 686-690.

R. Aralikatti, C. Boeddeker, G. Wichern, A. S. Subramanian, and J. Le Roux, 'Reverberation as Supervision for Speech Separation,' in Proc. ICASSP , 2023.

S.-I. Amari, A. Cichocki, and H. Yang, 'A New Learning Algorithm for Blind Signal Separation,' in Proc. NeurIPS , vol. 8, 1996, pp. 757-763.

A. Hyvärinen and E. Oja, 'Independent Component Analysis: Algorithms and Applications,' in Neural Networks , vol. 45, no. 4-5, 2000, pp. 411-430.

C. G. Puntonet and E. W. Lang, 'Blind Source Separation and Independent Component Analysis,' Neurocomputing , vol. 6, no. 1, pp. 1-57, 2006.

T. Adali, M. Anderson, and G.-s. Fu, 'Diversity in Independent Component and Vector Analyses,' IEEE Signal Processing Magazine , vol. 31, no. 3, pp. 18-33, 2014.

H. Sawada, N. Ono, H. Kameoka, D. Kitamura, and H. Saruwatari, 'A Review of Blind Source Separation Methods: Two Converging Routes to ILRMA Originating from ICA and NMF,' APSIPA Transactions on Signal and Information Processing , vol. 8, pp. 1-14, 2019.

T. Kim, T. Eltoft, and T.-W. Lee, 'Independent Vector Analysis: An Extension of ICA to Multivariate Components,' in International Conference on Independent Component Analysis and Signal Separation , 2006, pp. 165-172.

A. Hiroe, 'Solution of Permutation Problem in Frequency Domain ICA, using Multivariate Probability Density Functions,' in International Conference on Independent Component Analysis and Signal Separation , 2006, pp. 601-608.

C. Boeddeker, F. Rautenberg, and R. Haeb-Umbach, 'A Comparison and Combination of Unsupervised Blind Source Separation Techniques,' in ITG Conference on Speech Communication , 2021, pp. 129-133.

S. Rickard, 'The DUET Blind Source Separation Algorithm,' in Blind Speech Separation , 2007, pp. 217-237.

D. H. T. Vu and R. Haeb-Umbach, 'Blind Speech Separation Employing Directional Statistics in An Expectation Maximization Framework,' in Proc. ICASSP , 2010, pp. 241-244.

H. Sawada, S. Araki, and S. Makino, 'Underdetermined Convolutive Blind Source Separation via Frequency Bin-Wise Clustering and Permutation Alignment,' IEEE Transactions Audio, Speech, and Language Processing , vol. 19, no. 3, pp. 516-527, 2011.

N. Ito, S. Araki, and T. Nakatani, 'Complex Angular Central Gaussian Mixture Model for Directional Statistics in Mask-Based Microphone Array Signal Processing,' in Proc. EUSIPCO , 2016, pp. 1153-1157.

E. Tzinis, S. Venkataramani, and P. Smaragdis, 'Unsupervised Deep Clustering for Source Separation: Direct Learning from Mixtures Using Spatial Information,' in Proc. ICASSP , 2019, pp. 81-85.

L. Drude, D. Hasenklever, and R. Haeb-Umbach, 'Unsupervised Training of a Deep Clustering Model for Multichannel Blind Source Separation,' in Proc. ICASSP , 2019, pp. 695-699.

P. Seetharaman, G. Wichern, J. Le Roux, and B. Pardo, 'Bootstrapping Single-Channel Source Separation via Unsupervised Spatial Clustering on Stereo Mixtures,' in Proc. ICASSP , 2019, pp. 356-360.

M. Togami, Y. Masuyama, T. Komatsu, and Y. Nakagome, 'Unsupervised Training for Deep Speech Source Separation with Kullback-Leibler Divergence Based Probabilistic Loss Function,' in Proc. ICASSP , vol. 2020-May, 2020, pp. 56-60.

L. Drude, J. Heymann, and R. Haeb-Umbach, 'Unsupervised Training of Neural Mask-based Beamforming,' in Proc. Interspeech , 2019, pp. 1253-1257.

Y. Bando, K. Sekiguchi, Y. Masuyama, A. A. Nugraha, M. Fontaine, and K. Yoshii, 'Neural Full-Rank Spatial Covariance Analysis for Blind Source Separation,' IEEE Signal Process. Lett. , vol. 28, pp. 16701674, 2021.

Y. Bando, T. Aizawa, K. Itoyama, and K. Nakadai, 'Weakly-Supervised Neural Full-Rank Spatial Covariance Analysis for a Front-End System of Distant Speech Recognition,' in Proc. Interspeech , 2022, pp. 3824-3828.

R. Talmon, I. Cohen, and S. Gannot, 'Relative Transfer Function Identification using Convolutive Transfer Function Approximation,' IEEE Transactions Audio, Speech, and Language Processing , vol. 17, no. 4, pp. 546-555, 2009.

A. Levin, Y. Weiss, F. Durand, and W. T. Freeman, 'Understanding Blind Deconvolution Algorithms,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 33, no. 12, pp. 2354-2367, 2011.

Z.-Q. Wang, P. Wang, and D. Wang, 'Multi-Microphone Complex Spectral Mapping for Utterance-Wise and Continuous Speech Separation,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, pp. 2001-2014, 2021.

S. Wisdom, J. R. Hershey, K. Wilson, J. Thorpe, M. Chinen, B. Patton, and R. A. Saurous, 'Differentiable Consistency Constraints for Improved Deep Speech Enhancement,' in Proc. ICASSP , 2019, pp. 900-904.

Z.-Q. Wang, G. Wichern, and J. Le Roux, 'Convolutive Prediction for Reverberant Speech Separation,' in Proc. WASPAA , 2021, pp. 56-60.

T. Yoshioka, T. Nakatani, M. Miyoshi, and H. G. Okuno, 'Blind Separation and Dereverberation of Speech Mixtures by Joint Optimization,' IEEE Transactions Audio, Speech, and Language Processing , vol. 19, no. 1, pp. 69-84, 2011.

R. Haeb-Umbach, J. Heymann, L. Drude et al. , 'Far-Field Automatic Speech Recognition,' Proc. IEEE , 2020.

A. V. Oppenheim, Discrete-Time Signal Processing . Pearson Education India, 1999.

H. Sawada, S. Araki, and S. Makino, 'A Two-Stage Frequency-Domain Blind Source Separation Method for Underdetermined Convolutive Mixtures,' in Proc. WASPAA , 2007, pp. 139-142.

H. Sawada, R. Mukai, S. Araki, and S. Makino, 'A Robust and Precise Method for Solving The Permutation Problem of Frequency-Domain Blind Source Separation,' IEEE Transactions on Speech and Audio Processing , vol. 12, no. 5, pp. 530-538, 2004.

S. Gannot, E. Vincent, S. Markovich-Golan, and A. Ozerov, 'A Consolidated Perspective on MultiMicrophone Speech Enhancement and Source Separation,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 25, pp. 692-730, 2017.

L. Drude, J. Heitkaemper, C. Boeddeker, and R. Haeb-Umbach, 'SMS-WSJ: Database, Performance Measures, and Baseline Recipe for Multi-Channel Source Separation and Recognition,' in arXiv preprint arXiv:1910.13934 , 2019.

C. Boeddeker, 'https://github.com/fgnt/pb_bss/blob/master/examples/mixture_model_example.ipynb,' 2019.

R. Scheibler and K. Saijo, 'https://github.com/fakufaku/torchiva,' 2022.

E. Vincent, R. Gribonval, and C. Févotte, 'Performance Measurement in Blind Audio Source Separation,' IEEE Transactions Audio, Speech, and Language Processing , vol. 14, no. 4, pp. 1462-1469, 2006.

J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, 'SDR - Half-Baked or Well Done?' in Proc. ICASSP , 2019, pp. 626-630.

A. Rix, J. Beerends, M. Hollier, and A. Hekstra, 'Perceptual Evaluation of Speech Quality (PESQ)-A New Method for Speech Quality Assessment of Telephone Networks and Codecs,' in Proc. ICASSP , vol. 2, 2001, pp. 749-752.

C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, 'An Algorithm for Intelligibility Prediction of Time-Frequency Weighted Noisy Speech,' IEEE Transactions Audio, Speech, and Language Processing , vol. 19, no. 7, pp. 2125-2136, 2011.

Z.-Q. Wang, G. Wichern, and J. Le Roux, 'Leveraging Low-Distortion Target Estimates for Improved Speech Enhancement,' arXiv preprint arXiv:2110.00570 , 2021.

Y. Luo, Z. Chen, and T. Yoshioka, 'Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation,' in Proc. ICASSP , 2020, pp. 46-50.

S. Watanabe, M. Mandel, J. Barker, E. Vincent, A. Arora et al. , 'CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings,' in arXiv preprint arXiv:2004.09249 , 2020.

J. Carletta, S. Ashby, S. Bourban, M. Flynn et al. , 'The AMI Meeting Corpus: A Pre-Announcement,' in International Workshop on Machine Learning for Multimodal Interaction , 2006, pp. 28-39.

F. Yu, S. Zhang, Y. Fu, L. Xie, S. Zheng, Z. Du, W. Huang, P. Guo, Z. Yan, B. Ma, X. Xu, and H. Bu, 'M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge,' in Proc. ICASSP , 2022, pp. 6167-6171.

K. Tan and D. Wang, 'Learning Complex Spectral Mapping With Gated Convolutional Recurrent Networks for Monaural Speech Enhancement,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 28, pp. 380-390, 2020.

Y. Hu, Y. Liu, S. Lv et al. , 'DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement,' in Proc. Interspeech , 2020, pp. 2472-2476.

Y. Liu and D. Wang, 'Causal Deep CASA for Monaural Talker-Independent Speaker Separation,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , pp. 1270-1279, 2020.

A. Pandey and D. Wang, 'Densely Connected Neural Network with Dilated Convolutions for Real-Time Speech Enhancement in The Time Domain,' in Proc. ICASSP , 2020, pp. 6629-6633.

H. Taherian, K. Tan, and D. Wang, 'Multi-Channel Talker-Independent Speaker Separation Through Location-Based Training,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 30, pp. 2791-2800, 2022.

J. Zhang, C. Zorila, R. Doddipatla, and J. Barker, 'On End-to-End Multi-Channel Time Domain Speech Separation in Reverberant Environments,' in Proc. ICASSP , 2020, pp. 6389-6393.

Appendix

This appendix is organized as follows:

Appendix A describes the SMS-WSJ dataset.

Appendix B visualizes the surface of the proposed mixture-constraint loss to show that minimizing the loss can promote unsupervised separation of speakers.

Appendix C provides the derivation that FCP can be used to approximate speaker images.

Appendix D illustrates the frequency permutation problem.

Appendix E describes differences between the MC loss and the mixture consistency concept.

Appendix F illustrates the cases on when to use causal and non-causal filtering.

Appendix G provides interpretations of the physical meanings of intermediate DNN estimates ˆ Z .

Appendix H discusses differences between UNSSOR and RAS.

Appendix I presents miscellaneous system and DNN configurations.

Appendix J experiments with alternative filters taps for iRAS and UNSSOR.

Appendix K experiments UNSSOR with DNN architectures with lower modelling capabilities.

Appendix L reports the results of MixIT.

A SMS-WSJ dataset

SMS-WSJ [67] is a popular corpus for evaluating two-speaker separation algorithms in reverberant conditions. The clean speech is sampled from the WSJ0 and WSJ1 datasets. The corpus contains 33 , 561 ( ∼ 87 . 4 h), 982 ( ∼ 2 . 5 h), and 1 , 332 ( ∼ 3 . 4 h) two-speaker mixtures respectively for training, validation, and testing. The simulated microphone array has six microphones arranged uniformly on a circle with a diameter of 20 cm. For each mixture, the speaker-to-array distance is drawn from the range [1 . 0 , 2 . 0] m, and the reverberation time (T60) is sampled from [0 . 2 , 0 . 5] s. A weak white noise is added to simulate microphone self-noises, and the energy level between the sum of the reverberant speech signals and the noise is sampled from the range [20 , 30] dB. The sampling rate is 8 kHz.

B Visualization of loss surface

Fig. 2 visualizes the values of L MC in (4) based on a six-channel noisy-reverberant two-speaker mixture sampled from the SMS-WSJ dataset (see Appendix A for the dataset details). Let C = 2 and suppose that the DNN estimates are

where µ and ν ∈ R are bounded in the range [0 , 1] . Essentially, we use µ and ν to mimic the cases that the DNN produces (1) good separation (i.e., when µ ≈ 1 and ν ≈ 0 , or µ ≈ 0 and ν ≈ 1 ); and (2) bad separation (i.e., when µ and ν are both away from 0 and 1 , meaning that each estimate contains multiple speakers, and when µ ≈ 0 and ν ≈ 0 , or µ ≈ 1 and ν ≈ 1 , meaning that the two speakers are merged into one output and the other output does not contain any speakers). Fig. 2 enumerates µ and ν and plots the resulting separation results against the loss value of (4). We can see the loss values are smallest when µ ≈ 1 and ν ≈ 0 or when µ ≈ 0 and ν ≈ 1 (i.e., when the speakers are successfully separated), and clearly larger otherwise. This indicates that minimizing the proposed loss function can encourage separation.

0.0

0.33

0.66

1.0

1.0

0.66

0.33

0.0

0.30

0.35

0.40

0.45

0.50

0.55

C Effectiveness of FCP at approximating speaker images

Following the derivation in [29], let us define the mixture as Y p = X p ( c ) + V p ( c ) , where V p ( c ) consists of the signals of all the sources but c . We can formulate (6) as

where the derivation from the first row to the second is based on the assumption that, as the DNN training continues, ˆ Z ( c ) would become more and more accurate so that, after some epochs, it can become uncorrelated (or little correlated) with V p ( c ) , meaning that the cross-term would be small:

From the third row of (15), the resulting ˆ g p ( c, f ) H ˜ ˆ Z ( c, t, f ) would approximate X p ( c, t, f ) .

D Illustration of frequency permutation problem

We use three-channel input and loss for DNN training. Fig. 3(a) and (b) show an example separation result of the model trained with L MC in (9). Comparing the separated speech in Fig. 3(a) and (b) with the clean speech in (c) and (d), we can see that the separated speech suffers from the frequency permutation problem approximately in the range [1 . 6 , 2 . 9] kHz. Fig. 3(e) and (f) show the separation result of the model trained with L MC+ISMS in (11), which effectively addresses the frequency permutation problem.

(a)

(b)

Frequency (kHz)

(c)

(d)

4

Frequency (kHz)

2

0

(e)

(f)

4

Frequency (kHz)

2

0

0

1.6

3.2

0

1.6

3.2

Time (s)

Time (s)

E Differences between mixture-constraint loss and mixture consistency

The proposed mixture-constraint loss, L MC, has very different physical meanings and mathematical forms compared to the mixture consistency term proposed in [59]. First, in [59], DNN estimates are strictly constrained to add up to the mixture (see Eq. (7) and (9) in [59]), while our MC loss only encourages the filtered DNN estimates to add up to the mixture. Second, our MC loss is applied to filtered DNN estimates rather than directly to DNN estimates. Third, we deal with multi-microphone MC, while [59] only addresses single-channel cases. Fourth, the motivation of the proposed MC loss is to use mixtures as constraints to regularize DNN estimates so that the estimates can approximate source images. This motivation is completely different from that of the mixture consistency term.

F Illustration of using causal and non-causal filtering

This section illustrates that the relative RIR relating a signal to its delayed version is causal and the relative RIR relating a signal to its advanced version is noncausal.

In Fig. 4, suppose that the blue signal is the DNN estimate for speaker c , and the orange signal is speaker c 's image at another microphone, which is a delayed version (i.e., reaching the microphone later). To filter the blue signal to approximate the oracle one, we only need a causal filter to delay the blue signal.

Reversely, suppose that the orange signal is the DNN estimate for speaker c , and the blue signal is speaker c 's image at another microphone, which is an advanced version (i.e., reaching the microphone earlier). To filter the orange signal to approximate the blue signal, we need a non-causal filter to advance the orangle signal.

4

2

0

2

4

6

8

10

12

Time steps

G Interpretation of intermediate DNN estimates ˆ Z

This section provides an interpretation of the physical meanings of the intermediate DNN estimates ˆ Z in the cases of using multi-channel input and loss, and single-channel input and multi-channel loss.

G.1 Multi-channel-input case

In (9), ˆ Z ( c ) is constrained such that it can be filtered by a causal filter ˆ g p ( c ) to approximate X p ( c ) . Since there could be an infinite number of combinations of ˆ Z ( c ) and ˆ g p ( c ) whose convolution results would well approximate X p ( c ) , ˆ Z ( c ) cannot be interpreted as the dry source signal and we think it more similar to a signal captured by a virtual microphone that is closer to speaker c than all the P microphones. See Fig. 5(a) for an example, where each virtual microphone captures the direct-path signal of a target speaker earlier than any other microphones so that we can use causal FCP filters.

G.2 Monaural-input case

In the monaural-input case, each ˆ Z ( c ) would be aligned to the speaker's image at the input microphone (i.e., the reference microphone), since the DNN only has monaural input and in this case the DNN is not likely to align its outputs to a virtual microphone or any actual microphones.

We give an example in Figure 5(b), where the reference microphone captures speaker 2 's direct-path signal later than all the other microphones. In this case, we just need to use non-causal FCP filters when filtering ˆ Z ( c ) (which is estimated based on the monaural signal at the reference microphone) to approximate speaker 2 's images captured at the other microphones.

Speaker #1

Speaker #2

Virtual mic 1

Virtual mic 2

Speaker #1

Speaker #2

Reference mic

(a)

(b)

H Differences from RAS

In many aspects, UNSSOR differs from RAS [36], which deals with monaural two-speaker separation given binaural (two-channel) training mixtures. RAS first performs DNN-based monaural separation on the left-ear mixture in the magnitude T-F domain, then linearly filters the DNN estimates at the left ear through time-domain Wiener filtering such that the filtered estimate can approximate the right-ear mixture, and the training loss is computed between the summation of the filtered DNN estimates and the right-ear mixture. Besides differences in, for example, the DNN architectures, linear filtering algorithms, how phase estimation (important for estimating relative RIRs) is handled, how frequency permutation problem is dealt with, and training data curation (difficult training examples need to be removed to train RAS), the key difference is that RAS fails to be trained in an unsupervised way [36]. It still needs labelled mixtures so that a supervised PIT-based model can be trained and then used as the starting point for their unsupervised algorithm.

We think that the ineffectiveness of RAS in fully-unsupervised setup is likely because the loss is computed only on the right-ear mixture. Following our analysis in Section 3, in RAS there are N × 1 equations (because the loss is only computed on the right-ear time-domain signal, assumed N -sample) but N × C +(2 -1) × 512 × C unknowns (where the 512 term is because the filter is assumed 512 -tap in [36], and the (2 -1) term is because there is only one filter for each speaker in the binaural setup, i.e., only one non-reference microphone). This is an ill-posed problem, not likely to be solved via the current RAS algorithm.

I Miscellaneous system and DNN setup

In default, for STFT, the window size is 32 ms, the hop size is 8 ms, and the square-root Hann window is used as the analysis window. For 8 kHz sampling rate, a 256 -point discrete Fourier transform is applied to extract 129 -dimensional complex STFT spectra at each frame. This STFT setup is very common in modern deep learning based separation. Differently, the window and hop sizes of some baselines such as IVA and spatial clustering are considerably larger (see detailed setup in Section 5.1), as we observe that this leads to better separation performance, likely because more reverberation can be covered in each frame and, this way, their model assumptions can be better satisfied.

Our DNN architecture is TF-GridNet [23]. Using the symbols defined in Table I of [23], we set its hyper-parameters to D = 48 , B = 4 , I = 4 , J = 1 , H = 192 , L = 4 and E = 4 for 8 kHz sampling rate. Please do not confuse the symbols in TF-GridNet with the ones in this paper.

In each epoch, we sample an l -second mixture segment from each training mixture for model training. If a mixture is shorter than l seconds, we pad zeros in the front rather than in the end, since padding in the end would result in a mixture that has abrupt stop of reverberation, which is not realistic and would be detrimental to FCP-based relative RIR estimation. In comparison, padding zeros in the front can avoid this problem. If a mixture is longer than l seconds, we randomly pick an l -second segment. In default, l is set to four seconds.

We normalize the sample variance of each sampled mixture segment to 1 . 0 , before feeding them to DNN for training. Adam (with the default setup in Pytorch v1.9) is used as the optimizer. The

L 2 norm for gradient clipping is set to 1 . 0 . The learning rate starts from 10 -3 and is halved if the validation loss is not improved in two epochs. We terminate training once the learning rate is reduced to 6 . 25 × 10 -5 . The batch size is set to four, with each segment being 4 -second long. For each model, an Nvidia A100 40GB GPU is used for training, and the model converges in three to four days.

We sweep γ in (11) based on the set of { 0 . 02 , 0 . 04 , 0 . 06 , 0 . 1 , 0 . 3 , 1 . 0 } . The microphone weight α p in (4), (9), (10) and (12) is set to 1 . 0 in default for all microphones (i.e., no weighting), and, in the monaural input case (e.g., in Table 3 and 4), we sweep α 1 at the reference microphone 1 based on the set of { 1 / 2 , 1 / 3 , 1 / 4 , 1 / 5 , 1 / 6 , 1 / 8 , 1 / 10 } to alleviate overfitting to microphone 1 .

J Alternative filter length for iRAS and UNSSOR

We also provide the results of iRAS that uses the same filter length (in seconds) as that in UNSSOR. Given 8 kHz sampling rate, 32 ms window size, 8 ms hop size, and K = I +1+ J (defined below (6)) filter taps in FCP, we use M = (( K -1) × 8 + 32) / 1000 × 8000 filter taps for each ˆ h p ( c ) in (12), and configure ˆ h p ( c ) to filter the past M -8 / 1000 × 8000 -1 , the current, and the future 64 ( = 8 / 1000 × 8000 ) samples. We filter the future 8 ms of samples, because, in the STFT case, the hop size is 8 ms. We report the results in Table 5-8, each respectively corresponding to the results in Table 1-4. We observe that the performance is not as good as that of UNSSOR.

In Fig. 6, we make further comparisons of using different filter taps between UNSSOR and iRAS, following the experimental setup in the previous paragraph. Fig. 6(a) uses six-microphone input, Fig. 6(b) uses monaural input, and both of them use the six-microphone L MC+ISMS (or L iRAS) loss. For UNSSOR, in Fig. 6(a) we set J = 0 and sweep K ∈ { 5 , 9 , 13 , 17 , 20 , 25 } and in Fig. 6(b) we set J = 1 and sweep K ∈ { 5 , 9 , 13 , 17 , 21 , 25 } . For iRAS, we configure the filter to always filter the future 64 samples, and in Fig. 6(a) we sweep the filter taps M ∈ { 128 , 256 , 384 , 512 , 768 , 1024 , 1280 , 1472 } and in Fig. 6(b) we sweep M ∈ { 128 , 256 , 384 , 512 , 768 , 1024 , 1280 , 1536 } . Notice that in Fig. 6, the filter taps M in iRAS and K in UNSSOR are vertically corresponding to each other. That is, some swepted filter taps M are computed based on the swepted K (e.g., for M = 1024 and K = 13 , we have 1024 = ((13 -1) × 8 + 32) / 1000 × 8000 ) so that they can result in the same filter length in time. From the figures, we observe that the best filter length (in seconds) is different for UNSSOR and iRAS, and the best performance of UNSSOR is higher than that of iRAS.

128

256

384

512

768

1024

1280

1472

M

in iRAS (filter future 64 samples)

0

2

4

6

8

10

12

14

16

SDR (dB)

5

9

13

17

20

25

K

( =

I

+1+0) in UNSSOR

(a)

128

256

384

512

768

1024

1280

1536

M

in iRAS (filter future 64 samples)

0.0

2.6

5.2

7.8

10.4

13.0

SDR (dB)

5

9

13

17

21

25

K

( =

I

+1+1) in UNSSOR

(b)

We point out that using a very long filter (i.e., a large M ) for time-domain Wiener filtering would prevent separation, since, in that case, the filter would have a large degrees of freedom to filter ˆ z ( c ) to fit y p very well and obtain a small L iRAS (see (13)), even though ˆ z ( c ) is not a good separation result. From Fig. 6(a), it can be observed that a very large M (e.g., 1472 ) does not yield good separation; and in addition, a very small M (e.g., 128 ) is also not good, as the linear filter could be just too short to fit the mixture y p well. Similar trend can also be observed in the results of UNSSOR. For example, in Fig. 6(a), setting K to 5 and 25 produces worse performance than to 20 . We can see that the filter length is an important hyper-parameter to tune.

We emphasize that, to compute the closed-form solution of time-domain Wiener filtering (see (13)), we need to invert a big M × M matrix for each mixture, while in FCP (see (6)), we only need to invert a much smaller K × K matrix for each of the F frequencies. Using FCP is clearly less computationally-expensive, given that the time complexity of matrix inversion is typically O ( n 3 ) . This also indicates that if the same amount of computation is required for linear filtering, FCP can use a much longer filter (in time) than time-domain Wiener filtering.

K UNSSOR's effectiveness when used with other DNN architectures

We observe that the effectiveness of the proposed system comes from both the strong modelling capabilities of TF-GridNet and the contributions of the UNSSOR mechanism itself. Without using UNSSOR to deal with the ill-posed problem, the modelling capability of strong DNNs cannot be unleashed to separate speakers; and without using a strong DNN, the patterns in speech cannot be modelled well to realize good separation.

We expect UNSSOR to work with many DNN architectures, as long as the architecture is reasonably strong and can effectively deal with reverberation. To validate this, we experiment UNSSOR with DNN arachitectures with lower modelling capability. We select two representative separation models from the literature, TCN-DenseUNet [74] and DPRNN [75].

We replace TF-GridNet with the TCN-DenseUNet described in [74], and train the network using the same training configurations. TCN-DenseUNet [74] contains a temporal convolution network sandwiched by a UNet with DenseNet blocks. It is a reasonably strong separation model, which is fully convolutional and shares many similarities with many contemporary DNN architectures [79-83]. According to [23], it is worse than TF-GridNet in supervised separation tasks. We provide the unsupervised separation results in Table 9, which are obtained by using six-channel input and loss. We observe that UNSSOR works to some extent with TCN-DenseUNet, and the results are not as good as the ones in Table 1 obtained by using TF-GridNet.

The DPRNN architecture [75] in our experiments has a window size of 4 ms and a hop size of 1 ms. It has 6 layers. The number of bases is 256 . The bottleneck dimension is 128 . The number of hidden units in each BLSTM in each direction is 128 . We apply ReLU as the encoder non-linearity and as the non-linearity for embedding masking. The chunk size is set to 64 and the chunk overlap is 50% . To leverage spatial information for model training, we follow the strategy proposed in [84] (see its Fig. 2 to get the idea), where spectral embeddings are learned together with spatial embeddings and

DNN-estimated masks are used to mask spectral embeddings. In the six-channel case, the spatial embedding dimension is set to 360 , following [84]. Differently from the Fig. 2 of [84], we do not use microphone-pair-wise Conv1D layers to obtain spatial embeddings. Instead, we obtain them by using a Conv1D layer with P ( = 6 ) input channels and 360 output channels. We first use the DPRNN to obtain intermediate separation results in the time domain, and then apply STFT (with a window size of 32 ms, a hop size of 8 ms, and the square-root of Hann window) to obtain ˆ Z ( c ) for each speaker c . The network is trained by only using the MC loss in (9) and not using the ISMS loss in (10). Although the ISMS loss is not used, we only observe minor frequency permutation on the SMS-WSJ dataset. All the other procedures are the same as the TF-GridNet based UNSSOR system. The results on SMS-WSJ, obtained by using six-channel input and loss, are presented in Table 10. We observe that UNSSOR also works with DPRNN to some degrees, and the results are not as good as the ones in Table 1.

L Comparison with MixIT

This section compares the results of UNSSOR with MixIT [30], which also deals with unsupervised separation. We put this comparison only in appendix, considering that MixIT may not be an ideal baseline to UNSSOR. Notice that MixIT needs to be trained on synthetic mixtures of mixtures (MoM), which ideally would require the two mixtures used in creating each MoM to be recorded in the same room using the same device. Differently, UNSSOR is designed to be trained directly on existing mixtures. In this case, the two models would be trained on different training examples, and this would make the comparison difficult. In the main body of this paper, we hence only consider methods that can be trained (or performed) directly on existing mixtures (such as IV A, spatial clustering and iRAS) as baselines.

To report the results of MixIT, we create a particular scenario (ideal for MixIT), where, for each existing SMS-WSJ mixture ( y = x (1) + x (2)+ n with n denoting noise), we randomly add two extra speakers in the same simulated room and use the same array placed at the same location so that we can have two 2-speaker mixtures (i.e., an existing SMS-WSJ mixture 1 : y (1) = x (1) (1) + x (1) (2) + n (1) and a newly-simulated mixture 2 : y (2) = x (2) (1) + x (2) (2) + n (2) where the superscript (1) denotes mixture 1 and (2) denotes mixture 2 ) to create an MoM for training MixIT for 4-speaker separation. Similarly to n (1) , n (2) is sampled such that the SNR between x (2) (1) + x (2) (2) and n (2) is in the range of [20 , 30] dB. The DNN architecture and training configurations for MixIT are the same as that in UNSSOR and PIT. The loss function is defined similarly to (5) on the real, imaginary and magnitude of the reconstructed mixtures weighted by the summation of mixture magnitudes. Similarly to our proposed technique, we can feed 1-, 3- or 6-channel mixtures to TF-GridNet-based MixIT. At run time, the trained MixIT model is used to separate the existing two-speaker mixtures in SMS-WSJ to four outputs, and the two outputs with the highest energy are selected for evaluation.

This way, the evaluation scores can be pretty much directly compared with the ones obtained by UNSSOR.

The results are shown in Table 11. They are not as good as the ones reported in Table 1, 2, 3 and 4. For unknown reasons, three-channel MixIT fits the loss well but fails at separating speakers.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an innovative approach to unsupervised neural speech separation, leveraging the conditions where the number of microphones surpasses the number of speakers. The authors propose a method, named UNSSOR, which transforms an originally ill-posed problem - one that does not have a unique solution - into a well-posed problem - one with a unique solution - thereby facilitating the separation of speakers.\n\nKey contributions of the paper include:\n\n- The authors establish a linear-filter constraint between each speaker's reverberant images at each microphone pair, which converts the ill-posed problem into a well-posed one, thereby enhancing the separation of speakers.\n\n- The authors devise loss functions inspired by the blind deconvolution problem and propose a DNN-based approach to optimize these functions. The speaker images are determined via DNNs, while the linear filters are estimated using a sub-band linear prediction algorithm named FCP, based on the mixture and DNN estimates.\n\n- To address the frequency permutation issue that arises when using sub-band FCP, the authors propose a loss term that minimizes a measure known as intra-source magnitude scattering.\n\nThe authors claim that UNSSOR can be trained to perform under-determined separation, such as monaural unsupervised speech separation, based on over-determined training mixtures.\n\nSTRENGTHS:\nThe authors propose a novel unsupervised neural speech separation method. The proposed method, UNSSOR, utilizes the multimicrophone over-determined condition to solve the unsupervised learning problem of speech separation. This is a completely new perspective compared to previous unsupervised speech separation methods. The authors have designed innovative loss functions that guide the unsupervised separation model about the desired sound objects and encourage the separation of speakers. This approach is original and shows a creative combination of existing ideas.\n\nWEAKNESSES:\nHowever, there are several limitations to this paper that undermine its true potential. Please see below for concerns and questions for the authors.\n\n1. The article mentions that MixIT may have two mixtures that are not in the same scene under reverberant conditions, which in turn may result in providing spatial a priori information, or providing a priori information due to differences in the configuration and number of microphones. However, I feel that this statement is a bit too hypothetical. I think these problems are more a result of how the dataset is constructed and not a deficiency of the MixIT method itself. More specifically, these problems should be solved by constructing a room-specific dataset. The SMS-WSJ dataset mentioned in the paper can control the room's parameters, the arrangement of microphones, and the number of microphones. By finely designing and controlling these parameters, the resulting problems can be effectively circumvented, allowing the MixIT method to be evaluated in a more consistent environment. Therefore, I believe the problems mentioned in the paper do not serve as a weakness of the MixIT method.\n\n2. In Section 4.2, I noticed that the authors proposed the MC loss function. However, as far as I understand, the MC loss function is not presented in this paper and has been described in detail in reference [1]. Cite [1] as a reference and clarify whether the MC loss function in the paper is an innovation from [1] and where exactly the innovation is.\n\n3. The authors chose TF-GridNet as the separation DNN structure for the UNSSOR method.TF-GridNet has shown excellent performance in supervised separation tasks. However, the question I would like to raise is: Is the UNSSOR method's effectiveness due to the separation model TF-GridNet's strong performance, or is it due to the contribution of the UNSSOR method itself? I believe that to elaborate the capability of the UNSSOR method better; it is necessary to try to use different separation structures and compare their performance on each structure. If the UNSSOR method performs well on other separation structures, then we can be more confident that the UNSSOR method itself is highly robust and general. Such an analysis is necessary to assess and understand the actual value of the UNSSOR method.\n\n4. I note that the approach in the paper combines traditional methods (Spatial clustering and IVA) with the DNN-based method iRAS. In addition, UNSSOR, a DNN-based scheme, is compared to include only one DNN method, which may not be comprehensive, in my opinion. Of particular note is that MixIT is a popular and widely used method for unsupervised speech separation, as shown in references [2-5]. I was puzzled while reading the paper as to why the authors did not include MixIT in the comparison.\n\n5. The authors set different STFT window sizes and hop lengths in the comparison, which may raise questions about the fairness of the results. For the separation model of DNN, different settings of STFT window sizes and hop lengths can significantly affect the model's performance [6].\n\n### References\n[1]. Wisdom S, Hershey J R, Wilson K, et al. Differentiable consistency constraints for improved deep speech enhancement[C]//ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019: 900-904.\n\n[2]. Wisdom S, Tzinis E, Erdogan H, et al. Unsupervised sound separation using mixture invariant training[J]. Advances in Neural Information Processing Systems, 2020, 33: 3846-3857.\n\n[3]. Tzinis E, Adi Y, Ithapu V K, et al. RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing[J]. IEEE Journal of Selected Topics in Signal Processing, 2022, 16(6): 1329-1341.\n\n[4] Tzinis E, Casebeer J, Wang Z, et al. Separate but together: Unsupervised federated learning for speech enhancement from non-iid data[C]//2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2021: 46-50.\n\n[5] Zhang J, Zorila C, Doddipatla R, et al. Teacher-student MixIT for unsupervised and semi-supervised speech separation[J]. arXiv preprint arXiv:2106.07843, 2021.\n\n[6]  Peer T, Gerkmann T. Phase-aware deep speech enhancement: It's all about the frame length[J]. JASA Express Letters, 2022, 2(10): 104802.\n\nQUESTIONS:\nMy detailed questions are as described above.\n\n(1) Is the MC loss function in this paper an innovation from [1] and where is the innovation marked?\n\n(2) Is the effectiveness of the UNSSOR method due to the powerful performance of the separation model TF-GridNet, or is it due to the contribution of the UNSSOR method itself?\n\n(3) Why not compare MixIT method?\n\n(4) Why not use the same STFT settings, especially compared to the DNN model iRAS.\n\nLIMITATIONS:\nThe authors discuss the limitation of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe most popular method for training neural networks for speech separation is by artificially mixing sources, since neural network based training requires supervision. However, this kind of supervised training creates a mismatch since the mixtures seen at test time contain real overlapping speech and noise. This paper formalizes the problem of unsupervised speech separation, and posits that it becomes a well-posed problem for over-determined conditions (i.e., when the number of microphones is more than the number of speakers). The authors propose a method (inspired from signal processing) that uses the images at each microphone as supervision for the neural network training, and a new loss function to address the frequency permutation problem that often happens in blind source separation (BSS). They also propose a method to use these models for monoaural separation.\n\nSTRENGTHS:\nOverall, I believe this is an extremely strong paper with landmark results for unsupervised speech separation. As mentioned above, the majority of literature in the field of speech separation, since permutation-invariant training (PIT) was proposed, has only revolved around better neural network architectures. While the SI-SDR on the synthetic WSJ-Mix benchmark has consequently gone up, significantly less attention has been paid to real overlapping speech. The paradigm of continuous speech separation (CSS) tries to address some of the issues (such as sparse overlaps), but it is also trained with PIT on synthetic mixtures. The result is that there exists no public speech separation models that can perform well on multi-speaker data such as AMI, CHiME-6, AliMeeting, etc. I think UNSSOR is quite promising and should change the face of speech separation research.\n\nThe paper is strong on many levels, such as the following.\n\n1. The authors correctly identify that the problem with training separation networks has to do with a lack of accurate supervision, and propose that this supervision may come from estimating each speaker’s reverberant image at every microphone. Under the assumption of a linear-filter constraint between a speaker’s images at the microphones, the resulting linear system has a unique solution, as described in Section 3.\n\n2. The authors design loss functions which are inspired from signal processing formulation of the problem. This includes (i) mixture consistency loss for the filtered estimates, (ii) forward convolutive prediction (FCP) for relative RIR estimation, (iii) causal/non-causal stacking to handle time misalignment, and (iv) an intra-source magnitude scattering loss to address the frequency permutation problem. Overall, all these losses have clear motivations and design.\n\n3. The recently proposed TF-GridNet encoder is used as the backbone of the model. This architecture has been shown to obtain state-of-the-art results on PIT-based separation (even surpassing time-domain models), and it is interesting to see that it can be trained well with the unsupervised technique.\n\n4. The authors compare their method with extremely strong baselines, which makes their conclusions stronger. For example, they implemented a novel variant of the recently proposed “Reverberation as Supervision” (RAS) method, called iRAS, which can perform unsupervised separation. This modification could very well be a short paper in itself.\n\n5. The results on SMS-WSJ show that UNSSOR obtains results approaching supervised separation models. For example, for the 6-channel setting, the SDR on the test set is 15.6 dB (compared to 19.4 dB for PIT model). As a comparison, the next unsupervised method is IVA, which obtains 10.6 dB. Moreover, on reducing the number of channels from 6 to 3, UNSSOR’s performance degrades only marginally (to 15.4 dB), whereas PIT degrades to 16.8 dB.\n\n6. The authors even find a way to use models trained with multi-channel inputs for performing mono-aural separation.\n\n7. The code and models for UNSSOR will be released.\n\nA side effect of all of the above is that the paper is quite dense and may not be accessible to readers who do not have a source separation background. However, it is a case study in how neural methods can be designed by incorporating the knowledge from other domains (in this case, signal processing).\n\nWEAKNESSES:\n1. The main point of concern about the paper is that results are shown just for the SMS-WSJ dataset. Such an evaluation defeats the original motivation for unsupervised training, since SMS-WSJ contains synthetically mixed sources. Clearly, the PIT models outperform UNSSOR on this benchmark, which is expected. The real test of UNSSOR would have been an evaluation on real mixtures, such as AMI, CHiME-6, or AliMeeting, where PIT-based models fail completely. Nevertheless, since the paper proposes a completely novel paradigm of speech separation, I am willing to forgo this point, but at the cost of docking a point from the ratings.\n\n2. Can UNSSOR be trained with data containing different numbers of channels or different array geometries? It seems that the choice of the hyper-parameters $I$ and $J$ may be heavily dependent on the microphone configuration.\n\n3. The authors delegate the “Limitations” section to the appendix. I think it should be put in the main paper, since an important part of presenting our research is talking about its limitations. This would also be useful for other researchers to get ideas for building upon UNSSOR for their own work.\n\nQUESTIONS:\n1. How robust is UNSSOR to change in array configuration or the number of channels for the case of mono-aural separation?\n\n2. In Section 5.2, can you briefly explain what these metrics are, for the unfamiliar reader.\n\nLIMITATIONS:\nNone, except those presented in Appendix H.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an unsupervised method for training neural networks to separate speech from multi-microphone recordings. The idea is to separate a reference microphone into separate sources, then use each microphone as a mixture signal that must match filtered versions of the reference microphone separated sources. The linear filters are computed in subbands, which results in a well-known frequency permutation problem, for which an additional \"intra-source magnitude\" loss term is proposed to ameliorate. Though the method requires \"overdetermined\" mixtures (i.e. more mics than sources), the method can also be used for underdetermined cases, e.g. 2 speakers with 1 mic, where a single mic is used as input, and multiple mics are used in the loss function. For 3-mic and 6-mic mixtures, the proposed method is shown to achieve better performance in terms of SDR, SI-SDR, PESQ, and eSTOI versus a baseline (the RAS algorithm) that works on a similar principle on the SMS-WSJ dataset. The method does not outperform a supervised PIT method, which is expected.\n\nSTRENGTHS:\nS1) The method is intuitive and is explained well, and the paper provides thorough description of the method and chosen hyperparamters.\n\nS2) The proposed method can perform unsupervised training on single mixtures, unlike MixIT which requires combining at least two mixtures into a training examples.\n\nS3) The authors clearly describe the differences of the proposed UNSSOR method with the RAS method, particularly why UNSSOR can successfully train unsupervised, while RAS cannot.\n\nWEAKNESSES:\nW1) The dataset used, SMS-WSJ, consists entirely of synthetic mixtures that use synthetic simulated RIRs. Thus, there is some concern that the method may not generalize to real-world acoustic scenarios. Some evaluation on real data would further improve the paper. Also, one of the main advantages of unsupervised algorithms is to be able to adapt separation models to unsupervised real-world data, so that models can work better on those data domains. Of course, evaluation on real domains cannot be evaluated by intrusive metrics, as used in this paper, and require subjective human listening tests. I am glad the authors are considering this as a future direction, and I encourage them towards that goal.\n\nW2) As acknowledged by the authors, the method assumes stationary sources, which is potentially quite limiting when considering real scenarios where speakers may be moving. Even if speakers are seated, head movement can still produce acoustic effects. There are likely interesting extensions of the method to handle nonstationary spatial scenarios, such as allowing for slowly-varying linear filters. \n\nW3) No audio demos are provided. Providing such demos would make it a lot easier to readers to evaluate the quality of the predictions and improve understanding of the method.\n\nW4) It would be interesting to see how a single-channel unsupervised method like MixIT compares to the multi-microphone method. Also, since this paper was submitted, a multichannel version of MixIT seems to have been proposed (https://arxiv.org/abs/2305.11151), which seems to operate differently (network with multichannel input and multichannel output, and MixIT applied directly to multichannel outputs using multi-channel mixtures-of-mixtures). It would be interesting to compare both single-channel and multi-channel MixIT to the proposed approach; these MixIT methods will likely be mismatched to test time since they train on mixtures-of-mixtures, but it would be interesting to compare.\n\nMinor comments and typos\n\na) \"boot-start\" -> \"bootstrap\" ?\n\nb) \"comptue\" -> \"compute\"\n\nc) Would be good to mention what \\mathcal{F} is in description of equation (4)\n\nd) \"doing this would complicates\" -> \"doing this would complicate\"\n\ne) \"assumed time-variant.\" -> \"assumed time-variance.\"\n\nf) Maybe bold the best numbers in each column for Tables 1 and 2?\n\nQUESTIONS:\nQ1) How much tuning of the number of filter taps was done? This seems like it could have a big effect on the performance of the model. Also, does tuning the number of taps suggest some knowledge about the unsupervised data? I suppose a practitioner could use a blind T60 model to estimate the distribution of T60 across an unsupervised dataset, which could give some insight into optimal filter taps. Besides T60 (i.e. duration of expected RIRs to model), are there other issues that affect the optimal choice of filter taps?\n\nQ2) How certain is it that frequency permutation is the primary cause of the drop in performance going from rows 1a->1b and 2a->2b in Tables 1 and 2? The appendix provides an illustrative example, but none of the objective metrics are directly measuring frequency permutation. It seems that an objective metric could be formulated that directly measures the degree of frequency permutation, but perhaps such a metric is not necessary if spot checks of the predictions indicated that frequency permutation was present?\n\nLIMITATIONS:\nLimitations are discussed well.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors tackle speech source separation, in which multiple fixed speech sources $X(c)$ are recorded by an array of p microphones, resulting in p observable mixtures $Y_p$. The authors propose an STFT domain source separation algorithm called UNSSOR, leveraging a complex-valued deep neural network (TF-GridNet) to estimate the individual sources at a reference microphone (e.g., mic 1). Typically the problem is solved in a supervised fashion using PIT (permutation invariant training) where the clean sources must be known, but the authors propose an unsupervised method, where only the mixtures $Y_p$ are required at training time. The key idea is to relax the under-determined problem, by leveraging physical constraints, i.e. the sources combining at a microphone p are the same sources at the reference microphone convolved with a relative room impulse response (relative RIR). As such the authors estimate virtual sources $Z(c)$ and relative RIRs at each microphone for each source $g_p(c)$. The relative RIRs are computed in closed form using FCP (a least squares optimization problem), leveraging only the mixtures $Y_p$ and the estimates $Z(c)$. Multiplying $g_p(c)$ with $Z(c)$ the authors obtain the FCP-estimated source images, which are summed to obtain mixtures estimates $\\hat Y_{p}$ at each microphone, obtaining a mixture consistency loss $L_{MC}$ for training the neural network. The authors notice that solutions obtained with the model trained on $L_{MC}$ only exhibit the frequency permutation problem, where estimates on certain spectral bands are swapped between estimates. To address this problem, the authors regularize the variance of the magnitude of the FCP-estimated source images over frequency bands during training time, with a novel intra-source magnitude scattering loss $L_{ISMS}$. Combining $L_{MC}$ and $L_{ISMS}$ results in improved separation metrics. Finally, the authors try the algorithm in the monoaural unsupervised source separation setting, where they provide only one mixture at training time, but optimize with multiple microphones in the loss.\n\nSTRENGTHS:\n- The idea of reducing the number of equations using physical constraints is natural, leading to a well-structured unsupervised multi-mixture speech source separation algorithm. Computing the RIRs with least squares also is very useful, given that in such a way the mixture consistency loss depends only on the observed mixtures and the estimated virtual sources. The authors present such an idea in a very clear and formal manner, with relevant links to the existing literature. I believe that this approach is way more founded theoretically than the MixIt approach, leading to a better methodological line of research on unsupervised neural source separation.\n- The intra-source magnitude scattering regularizer ($L_{ISMS}$) seems very interesting because such a loss can be incorporated not only in the presented multi-microphone speech scenario but in every source separation task working on STFT spectrograms (e.g, music, universal). Improving the results for more than 4dB gives strong empirical results to the effectiveness of such a technique.\n- The method achieves good empirical results with respect to a plethora of metrics (SDR, SI-SDR, PESQ, eSTOI) on SMS-WSJ compared to other unsupervised multi-mic algorithms (Spatial clustering, IVA, RAS) and fares well with respect to a supervised PIT baseline (using the same neural architecture).\n\nWEAKNESSES:\n- UNSSOR, while improving the results in the monoaural speech separation setting with respect to iRAS (improves 1.4 dB on the SMS-WSJ test set both with 3 and 6 channels loss), still faces a large gap with respect to the supervised Monaural PIT baseline (~ 4 dB). This is expected since not relying on supervision can negatively impact the separation results. The authors should have included at least a comparison with some other popular unsupervised source separation baseline such as MixIt, in order to prove the superiority of UNSSOR in the unsupervised source separation arena.\n- As with other unsupervised neural source separators such as MixIt, it is not really clear why one should go unsupervised if the supervised metrics are already better on the same dataset. These types of studies (including MixIt or its regularized versions such as https://arxiv.org/abs/2106.00847), should experimentally showcase that performing unsupervised source separation can benefit over supervised source separation as the dataset size increase. For example in https://arxiv.org/abs/2106.00847, authors cannot beat a supervised baseline trained on FUSS, using an order of magnitude more data (Audioset or YFCC100m) with an unsupervised method (regularized MixIt). Providing a scaling law should be of uttermost importance in papers such as the presented one and in future papers, if not we can continue inventing unsupervised methods that will never surpass supervised or weakly supervised methods (using labels, learning Bayesian priors).\n\nQUESTIONS:\nI will add suggestions for the improvement of the paper and ask relevant questions.\n\n- Line 65: The authors should use a term like `metric` instead of `measure`, given the precise meaning of such a term in measure-theory. I know they can be synonyms in an applied field such as audio processing but it is better to be more precise.\n- Line 120: `Hermitian transpose` instead of `Hermittan`\n- Line 146: the term mixture consistency is defined here: https://arxiv.org/pdf/1811.08521.pdf, I think it's ok to use the same term but could create a little bit of confusion.\n- Line 161: Can the authors explain briefly in the text, for better understanding, the rationale of matching the FCP-estimated images with the mixture in Eq. (6)? \n- Line 167: Regarding the $\\xi$ hyperparameter, I don't understand if it multiplies only the max or all the expression\n- Line 179: I found the causal analysis well developed, but I still do not well understand why if a source sound c reaches the reference microphone earlier it should be processed with future values (non-causal filtering). \n- Line 193: Given that $Z(c)$ is learned, why it is interpreted like a virtual microphone estimate and not the real dry signal at speaker $c$?\n- Line 232: In the monoaural speech source separation setting, I don't understand what you match on the different microphones in the mixture consistency loss, when using only one input $Y_p=1$? Do you match the only available $Y_p=1$ at all \"virtual\" microphones?\n\nLIMITATIONS:\nThe authors address the limitations of their work in Appendix H, namely that they assume to know the number of sources, they assume they are directional point sources at fixed positions. I do not believe these limitations are much problematic as the main limitation is the unavailability of data required for scaling up the algorithm with respect to the supervised baseline (if it scales).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe focus of this paper is unsupervised speech separation by exploiting training mixtures with more microphones than speakers (over-determined). The proposed neural separator is using the input mixture to constrain the estimated images of each speaker. It is proposed to also train the system for under-determined scenarios, e.g., for single-channel speech separation, by separating a single-channel signal, while using multichannel loss function.\n\nSTRENGTHS:\nThe idea of using multichannel mixtures for unsupervised training is novel and reasonably effective compared to the supervised and unsupervised baselines.\nThe experimental evaluation is sufficient and demonstrates the effectiveness and includes sufficient data to support some of the design choices.\nThe paper is well written.\n\nWEAKNESSES:\nIt would be useful for the readers less familiar with speech processing to include references to a few papers very relevant to relative IR estimation, mixture consistency and linear prediction.\n\nQUESTIONS:\nOnly measurement/modeling noise is included in the model, and background noise or discrete noise sources would complicate things, e.g., since the same RTF could not be applied to the noise component. Is there any initial data on robustness in presence of noise?\n\nll. 119\nIt is claimed that the relative room impulse response in (2) is typically short. However, it should be noted that this is not necessarily true, even for the microphone array and reverberation times used in the experiments (20 cm diameter, up to 0.5s RT60). A useful datapoint is paper by [Talmon, 2009], Fig. 7 in particular. I think this it would be helpful for many readers at NeurIPS to mention this (or a similar paper) and point out that this assumption does not always hold, and has been investigated in the literature.\n\n[Talmon, 2009] Talmot, Cohen, Gannot, Relative Transfer Function Identification Using Convolutive Transfer Function Approximation, IEEE Tr. ASLP, 2009.\n\nll. 128\n- Typo in “comptue”\n\nSection 4.2\nMixture consistency has already been proposed in [Wisdom, 2019]. This paper should be clearly mentioned. Also, consider renaming to multichannel mixture consistency, since you’re using P channels and RTFs.\n\n[Wisdom, 2019] Wisdom et al., Differentiable Consistency Constraints for Improved Deep Speech Enhancement, 2019.\n\nSection 4.3\nFormulation is (6) is basically multichannel linear prediction with using estimated sources Z instead of the original mixtures Y. Relevant works solving a problem analogous to (6) should be at least briefly mentioned, such as earlier work in [Yoshioka, 2010] and many later works.\n\n[Yoshioka, 2010] Yoshioka et al., Blind Separation and Dereverberation of Speech Mixtures by Joint Optimization, IEEE Tr. ASLP 2011.\n\nLIMITATIONS:\nThe main limitation is that the model does not take into account background or discrete noise, and is only demonstrated to work in presence of uncorrelated (measurement) noise. This severely limits applicability to using real-world recordings to train a separation model.\nIt would be very useful to include results on robustness in presence of higher levels of noise. However, this is not essential and it may be out of scope of this paper.\nAnother limitation for real-world use is the assumption of a static scenario (fixed relative IRs), which would not hold in real recordings.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}