UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures
Authors: Zhong-Qiu Wang
Shinji Watanabe
Venue: NeurIPS 2023 poster (2023)
Keywords: Speech separation
microphone array processing
deep learning
TL;DR: We show that DNNs can be trained directly on multi-speaker mixtures to realize unsupervised neural speech separation if the training mixtures are over-determined (i.e., more microphones than speakers).

Abstract: In reverberant conditions with multiple concurrent speakers, each microphone acquires a mixture signal of multiple speakers at a different location. In over-determined conditions where the microphones out-number speakers, we can narrow down the solutions to speaker images and realize unsupervised speech separation by leveraging each mixture signal as a constraint (i.e., the estimated speaker images at a microphone should add up to the mixture). Equipped with this insight, we propose UNSSOR, an algorithm for $\underline{u}$nsupervised $\underline{n}$eural $\underline{s}$peech $\underline{s}$eparation by leveraging $\underline{o}$ver-determined training mixtu$\underline{r}$es. At each training step, we feed an input mixture to a deep neural network (DNN) to produce an intermediate estimate for each speaker, linearly filter the estimates, and optimize a loss so that, at each microphone, the filtered estimates of all the speakers can add up to the mixture to satisfy the above constraint. We show that this loss can promote unsupervised separation of speakers. The linear filters are computed in each sub-band based on the mixture and DNN estimates through the forward convolutive prediction (FCP) algorithm. To address the frequency permutation problem incurred by using sub-band FCP, a loss term based on minimizing intra-source magnitude scattering is proposed. Although UNSSOR requires over-determined training mixtures, we can train DNNs to achieve under-determined separation (e.g., unsupervised monaural speech separation). Evaluation results on two-speaker separation in reverberant conditions show the effectiveness and potential of UNSSOR.

UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures
Zhong-Qun Wang and Shinji Watanabe
Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA wang.zhongee 2honggiu4100mail
Abstract
In reverberant conditions with multiple concurrent speakers. each microphone acquires a mixture signal of multiple speakers at different location. In over- determined conditions where the microphones out-number speakers, we can narrow down the solutions to speaker images and realize unsupervised speech separation by leveraging each mixture signal as a constraint (i.e., the estimated speaker images at microphone should add up to the mixture). Equipped with this insight, we propose UNSSOR, an algorithm for unsupervised neural speech separation by leveraging over-determined training mixtures. At each training step, we feed an input mixture to a deep neural network (DNN) to produce an intermediate estimate for each speaker, linearly filter the estimates, and optimize a loss that, at each microphone, the filtered estimates of all the speakers can add up to the mixture satisfy the above constraint. We show that this loss can promote unsupervised separation of speakers. The linear filters are computed in each sub- band based the mixture and DNN estimates through the forward convolutive prediction (FCP) algorithm. To address the frequency permutation problem incurred by using sub-band FCP, loss term based minimizing intra-source magnitude scattering is proposed. Although UNSSOR requires over-determined training mixtures, we can train DNNs to achieve under-determined separation (e.g., unsupervised monaural speech separation). Evaluation results on two-speaker separation in reverberant conditions show the effectiveness and potential of UNSSOR.
Introduction
In many machine learning and artificial intelligence applications, sensors, while recording, usually capture a mixture of desired and undesired signals. One example is the cocktail party problem (or speech separation)
22. where, given a recorded mixture of the concurrent speech by multiple speakers, the task is to separate the mixture to individual speaker signals. Speech separation [ has been dramatically advanced by deep learning, since deep clustering 4 and permutation invariant training (PIT) 5 solved the label permutation problem. They (and their subsequent studies 6 2
are based on supervised learning, requiring paired clean speech and its corrupted signal generated simulation, where clean speech is mixed with, for example, various noises and competing speakers diverse energy and reverberation levels in simulated rooms accurate, sample-level supervision for model training. Such simulated data, however, may not match distribution of real-recorded test data in the target domain, and the resulting supervised learning based models would have generalization issues 28) How to train unsupervised neural speech separation systems on unlabelled target-domain mixtures is hence an important problem to study.
B. The clean speech can provide an
Training unsupervised speech separation models directly on monaural mixtures is an ill ill-posed task (2). since there is only one mixture signal observed but multiple speaker signals to reconstruct. The separation model would lack an accurate supervision (or regularizer) to figure out what desired sound objects (e.g., clean speaker signals) are, there are infinite solutions where each solution the
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

estimated sources can sum up to the mixture. Supposing that the separation model does not separate well and outputs a clean speaker signal plus some competing speech, noise or reverberation, would this output be viewed as desired sound object? This is clear to humans, clear to supervised learning based models (by comparing the outputs with training labels), but not really clear to an unsupervised model. On the other hand, many studies [3426] have observed that deep learning based supervised learning can achieve remarkable separation. In other words, with proper supervision, modern DNNs capable of separating mixed speakers, but, in an unsupervised setup, there lacks an accurate supervision to unleash this capability. The key to successful unsupervised neural separation, we believe, is designing clever supervision that can inform the model what desired sound objects are, and penalize the model if its outputs are not good and reward otherwise.
Our insight is that, in multi- microphone weee-detrri conditions where the microphones out- number speakers, the ill ill-posed problem can be turned into well-posed one, where a unique solution to the speakers exists (up to speaker permutation). This well-posed property (that a unique solution exists) can be leveraged as supervison (or regularizer) to design loss functions that could inform the unsupervised separation model what desired sound objects are and promote separation of speakers.
Equipped with this insight, we perform unsupervised neural speech separation by leveraging multi- microphone over-determined training mixtures. Our DNNs can be trained directly on over-determined mixtures realize over- and under-determined separation. The proposed algorithm, named UNSSOR, obtains strong separation performance on two-speaker separation. Our contributions include:
We enforce a linear-filtrr constraint between each speaker's reverberant images at each microphone pair, turning the ill-posed problem into a well-posed one that can promote separation of speakers. blind deconvolution problem, where both
We formulate unsupervised neural speech separation the speaker images and linear filters need to be estimated. We design loss functions motivated by the blind deconvolution problem, and propose a DNN approach to optimize the loss functions, where the speaker images are estimated via DNNs and the linear filters are estimated via a sub-band linear prediction algorithm named FCP 29 based on the mixture and DNN estimates.
We propose a loss term, which minimizes a metric named intra-source magnitude scattering, to address the frequency permutation problem incurred when using sub-band FCP.
Based on oeer-determined training mixtures, UNSSOR can be trained to perform under-determined separation (e.g., monaural unsupervised speech separation).
2 Related work
Various unsupervised neural separation algorithms, which do not require labelled mixtures, have been proposed. The most notable one is mixture invariant training (MixIT) [3] which first synthesizes training mixtures, each by mixing two existing mixtures, and then trains a DNN to separate the resulting mixed mixtures to underlying sources such that the separated sources can be partitioned into two groups and the separated sources in each group can sum up to one of the two existing mixtures (used for mixing). Care needs to be taken when synthesizing mixtures of mixtures. First, the sources in an cxisting mixture could have similar characteristics (c.g., similar reverberation patterns as the sources in existing mixture are recorded in the same room) that are informative about which sources belong to the same existing mixture, and this would prevent MixIT from separating the sources [00 35 Second, it is unclear how mix existing multi-channel mixtures, which are usually recorded by devices with different microphone geometry and number of microphones. Third, mixing existing mixtures with different reverberation characteristics would create unrealistic mixtures.
UNSSOR avoids the above issues by training unsupervised neural separation models directly on existing mixtures rather than on synthesized mixtures of mixtures. An earlier study related to this direction is the reverberation as supervsion (RAS) algorithm [36]. which addresses monaural two- speaker separation given binaural (two-channel) training mixtures. RAS performs magnitude-domant monaural separation directly on the left-ear mixture and then linearly filters the estimates through time-domain Wiener filtering that the filtered estimates can approximate the right-ear mixture. RAS essentially does monaural separation and is effective at separating speakers in semi-supervised learning setup, where supervised PiT-based model is first trained and then used to bootstrap unsupervised training. It however fails completely in ually-unsupervised setup (36], unlike UNSSOR.
independent vector analysis (IVA) 441 and spatial clustering 48 can perform unsupervised separation directly on existing mixtures. They perform separation based on single test mixture at hand and are not
Conventional algorithms such as independent component analysis 37 41

---

designed to learn speech patterns from large training data, while UNSSOR leverages DNNs to model speech patterns through unsupervised learning, which could result in better separation. Another difference is that UNSSOR can be configured for monarual, under-determined separation, while ICA, IVA and spatial clustering cannot. There are studies [29 training DNNs to approximate pseudo- labels produced by conventional signal processing based separation models such as spatial clustering and blind source separation (BSS) algorithms. Their performance is however often limited since spatial clustering and BSS themselves are not good enough at separation. There are studies [53 training DNNs to separate multi-speaker mixtures such that the likelihood observed mixtures under probabilistic distribution derived based the DNN separation results can be maximized. Such methods rely on statistical models for separation. They require costly iterative estimation of signal statisties run time. In addition, the DNNs are only leveraged in estimating target magnitude, while phase estimation is only realized by spatial filtering.
3 Problem formulation
Given P-microphone mixture with C' speakers in reverberation conditions, the physical model can be formulated using system of linear equations in the short-time Fourier transform (STFT) domain:
E ft c + forp E PI,
(I)
where indexes T frames, f indexes F frames, and at microphone p. time I and frequency f. Y,,,, J). Xp(c, 1, J') and sp(t, f) E C respectively denote the STFT coefficients of the mixture, reverberant image of speaker c, and non-speech signals. In the rest of this paper, we refer to the corresponding spectrogram when dropping the index c, p, f. We assume that is weak and stationary (e.g., time-invariant Gaussian noise or simply modelling errors). Without loss of generality, we designate microphone 1 as the reference microphone. Our goal is to, in an unsupervised way, estimate each speaker's 's image at the reference microphone (i.e., (c) for each speaker e) given the input mixture. We do not aim at dereverberation, instead targeting at maintaining the reverberation of each speaker.
Unsupervised separation based only on the observed mixture is difficult. There are infinite solutions to the above linear system where there are T x F x P equations (we have a mixture observation for each (t,,, ()) but F x C unknowns (we have one unknown for each Xp(c, L, f)).
Our insight is that the number of unknowns can be dramatically reduced, if we enforce constraints to the speaker images different microphones. Since (c) and Xp(c) are both convolved versions of the dry signal of speaker c, there exists linear filter between them such that convolving X1 (c()) with the filter would reproduce X (p(c). This convolutive relationship a physical constraint, which can be leveraged to reduce the number of unknowns. Specifically, we formulate (T) as
Ki({)) E X1(e,, +eil()
[t(t,))= = Σ f) &1 + (p(t, forp E PJ.
where (e,,,, = [x1(,,--,,1
CA+1+B stacks window of E === 1--++++B T-F units, f) C is the relative room impulse response (relative RIR) relating Xi) to Xp(c). and (-)H computes Hermitian transpose. g, (c, is not long (i.e., E is small) [56 if microphone and p are placed close to each other, which is the case for compact arrays.
implication of this constraint that the number of unknowns reduced from T x F P C to T x F (i.e.. T x P) when P C (i.e., over-determined conditions) and when T' sufficiently large (i.e., the input mixture is reasonably long). In other words, this formulation suggests that (1) there exists solution for separation, which most consistent with the above linear system; and (2) in over-determined cases, it is possible to estimate the speaker images in an unsupervised way.
C x (P - 1) E x c which can be smaller than the number of equations
'T'S x F' x C' is because there is one unknown for each Xi f, f), and x (P - x E x C' is because go f) is E--ap and we have such filter for each of P microphone pairs for each frequency and speaker.
3

---

Microphane
Microphonep
LISMS.P
2(1)
FCP
(L)
multi-channel input: 18,... Yp.... ... Ypl
DNN
-(2)
RECP (c)
McC+1SMS
monaural input: 1/
2(C)
FCP
23 (C)
LMCP
Figure I: Illustration of UNSSOR (assuming P > C' during training).
As is assumed weak, estimate that is most consistent with the linear system in 2 by solving the following problem:
and Gaussian, one way find the solution is to compute
P
C
argmin Σ Eitt, Σ (ictt)) (.6)..)()rr)) of
+ 2= t.
ΣΣ
Ytt,,
Σ sp f) X,(ctt (x(ctt
This is a blind deconvolution problem [57] which is non-convex in nature and difficult to be solved if no prior knowledge is assumed about the relative RIRs the speaker images, because both of them are unknown. In the next section, we propose a DNN-based approach, which can model speech patterns through unsupervised learning (and hence model speech priors), to tackle this problem.
4 Method
Fig. D Jillustrates the proposed system. The DNN takes in the mixture at all the P microphones or at the reference microphone 1 as input and produces an intermediate estimate (cc for each speaker FCP (29] is then performed on (cc at each microphone p to compute a linear-filtering result, denoted (CF) which, we will describe, is essentially estimate of the speaker image X (c). After that, two loss functions are computed and combined for DNN training. This section describes the DNN configuration, loss functions, FCP filtering, and extension for monaural separation.
4.1 DNN configurations
The intermediate estimate (c) for each speaker e obtained via complex spectral mapping (23 58) where we stack the real and imaginary (RI) parts of the input mixture as features for the DNN to predict the RI parts of (c). For the DNN architecture, we employ TF-GridNet [23], which obtains strong results on supervised speech separation benchmarks. See Appendix
| for more DNN details.
4.2 Mixture-constraint loss on filtered estimates
Following formulation in (3 we propose mixture-constrraint (MC) loss, which is computed by
filtering the DNN estimate Z(c) of each speaker e to approximate the P-channel input mixture:
LMC = a1 Σ F(Y f), c=1 4.
(c,t.) +
F(Y Σ (c, 2(e, t, f)). (4)
ap
2=2
In 44. (cc, t, f) stacks a window of T-F units around (c, t, f). and (c, f) is an estimated relative RIR computed based on (c.. f) and the mixture Yp(- f) through FCP [29]. Both of them will be
described in the next sub-section. ap R is a weighting term for microphone p. Following [23].
F(..)) in (4) computes an absolute loss on the estimated RI components and their magnitude:
(),
Ex. f7] (Re(Y,(t,)) f)) (o((,(t,,))
+ Im((), Im Im(()(( - - (5)

---

where Re(-) and Im(-) (-) respectively extract RI components and |-| computes magnitude. The term 1/Er, (t,', f') balances the losses at different microphones and across training mixtures.
According to the discussion in Section 3 minimizing LMC would encourage separation of speakers. We illustrate the loss surface of LMc in Appendix B
Compared to the mixture consistency term proposed in (59,, our different physical meanings and mathematical forms. See Appendix for detailed discussions.
loss has very
4.3 FCP for relative RIR estimation
To compute Luc, we need to first estimate each of the relative RIRs, (p(c, f)- In 29 60. FCP is proposed to estimate the relative RIR relating direct-path signal reverberant image for speech dereverberation. In this study, we employ FCP to estimate the relative RIR relating (c) to the speaker image captured at each microphone (i.e., (p(c)).
Assuming speakers are non- non-moving. we estimate relative RIRs by solving the following problem:
(p,c, = argmin
E. By (c.f)
Xp Xp(e, f) - f) t(c,,
(6)
where f) E C++1+J is a K tapp (with = = time-invariant FCP filter, ((c, t, f) (c f).)..... fcc,t.f)....
(cc, + J. [ E C stacks past and J future T-F units with the current one. Since the actual number of filter taps (i.c., A and B defined in the text below (2) is unknown, we set them and J, both of which are hyper-parameters to tune. Ap(c, t, f) weighting term balancing the importance of each T-F unit. Following (29]. we define it as f) = E (ty + x max (+ - where & 10 in this study) used to floor the weighting term and max(-) extracts the maximum value of a spectrogram. (6) is weighted linear regression problem. A closed-form solution can be readily computed:
(c (E, (Eiideet, (GLL "E, 'E. iple,tt,f (c, (LG (7)
1
ip(e,t,f)
where (.)* computes complex conjugate. We then plug Bo(c, f) into (4 and compute the loss.
Note that to compute the relative RIR, ideally we should filter ((c) to approximate Xp(c) (i.c., replacing Yp in (6) with Xp(c)), but Xp(c) is unknown. In (6). we instead linearly filter (cc to
approximate Yp. and earlier studies 29 60
suggest that the resulting fc) Z(c,t, f, would be
an estimate of t,, (c) reasonably accurate (see Appendix C for the derivation). We name the speaker image estimated this way as FCP TCP-estimated image:
X (c,t,f) = Ep(c, f) Z(c, f).
(8)
It is therefore reasonable to sum up the FCP-estimated images of all the speakers and define loss between the summation and Yp as in (4).
Although (6) appears similar to multi-channel linear prediction (MCLP) [61 62] which is popular in conventional speech separation algorithms, we point out that they have very different physical meanings. We consider that (6) does forward filtering, where source estimates are filtered to approximate mixtures so that relative RIRs can be estimated, while MCLP does inverse filtering, where mixtures are filtered to approximate target sources and the filters are designed to suppress non-target signals. This difference results in non-trivial changes of the physical meanings of the computed filters (see also discussions in Section V.C of (2
Although there were earlier efforts in estimating relative RIRs (66], they are based on conventional signal processing techniques and the performance is usually limited due to strong assumptions on, and inaceurate estimation of, signal statisties.
4.4 Time alignment issues and alternative loss functions
In 4). we do not filter the DNN estimates when computing the loss on the first (reference) microphone. We expect this to result in (cc aime-aligned with the speaker image X2 (c) (i.e., 2(c) is estimate
5

---

of (c)))) Since the reference microphone may not be the microphone closest to speaker c, it is best to use non-causal filters when filtering (c) to approximate the reverberant image X₂ Xp(c) at non-reference microphones that are closer to source c than the reference microphone, and instead use causal filters for non-reference microphones that farther Since estimating which non-reference microphones are closer or farther to a source than the reference microphone is not an easy task and doing this would complicate our system, the non-reference microphones. This could, however, limit the DNN's capability at separating the speakers, because the relative RIRs for some non-reference microphones (farther to source e than the reference microphone) are causal, and it may not be a good idea to assume non-causal filters.
just choose to non-causal filters for all
To address this issue, make simple modification to the loss function in
P - p=1 MpCMC.p = =E,r P
Sc f) p=1 Op F(sllt E_,
LMc
(9)
where the difference is that we also filter the DNN estimates when computing the loss on the reference microphone, and we constrain Bp(c, f) to be causal and that Z(c, t, f) only stacks current and past frames. This way, the resulting (c) would not be time-aligned with the revererberant image captured at the reference microphone (i.e., (c) or any other non-reference microphones. Because of the causal filtering. (c) would be more like an estimate of the reverberant image captured by virtual microphone that is closer to speaker e than all the P microphones (see Appendix [G.] for an interpretation). It would contain less reverberation of speaker than any of the speaker images captured by the microphones due to the causal filtering.
To produce an estimate that is time-aligned with the reverberant image at a microphone (e.g., xc()),, we use the FCP-estimated image computed in (8) (i.e., XFC((c) as the output.
4.5 Addressing frequency permutation problem
In ( and (9). FCP is performed in each frequency independently from the others. Even though the speakers are separated at each frequency, the separation results of the same speaker at different frequencies may however not be grouped into the same output spectrogram (see an example in Appendix D) This is known the frequenqy permutation problem [41], which has been studied for decades in rrequency-domall independent component analysis [3 and spatial clustering 48). Popular solutions for frequency aligument are designed by leveraging cross- croous-fr correlation of spectral patterns [47] 64 and ooaectionnoo-arrivaa estimation [65]. However, these solutions are often empirical and have complicated design. They can be used to post-process DNN estimates for frequency alignment, but it is not easy to integrate them with UNSSOR for joint training. This section proposes loss term, with which the trained DNN can learn to produce target estimates without frequency permutation.
blind source separation algorithms such as frequency-domen
To deal with frequency permutation, IVA 42 assumes that, at each frame, the de-mixed outputs at all the frequencies follow a complex Gaussian distribution with a shared variance term across frequencies: w(c, YY,, ~ (0, D(t, c)), where w(c,)) E CP is the de-mixing weight vector (in a inne-invariant de- mixing matrix) for speaker c at frequency f. and D(t, c) E R is the shared variance term, which is assumed time-variant. When maximum likelihood estimation is performed to estimate the de-mixing matrix, the variance term shared across all the frequencies is found very effective solving the frequency permutation problem 41 44
Motivated by IVA, we design the following loss term, named intra-source magnitude scattering (ISMS), to alleviate the frequency permutation problem in DNN outputs:
P
E Ec var (ox (c,
LISMS = Σ ap LISMS.. = Σ
app
(10)
E, var var((o
p=1
where X-FCP is computed via (8) &FCP (c,t,) .) E CF, and var(-) computes the variance of the values p in vector. At each frame, we essentially want the the magnitudes of the estimated spectrogram of each speaker (i.e.. XFCP (c, t, t)) to have a small intra-source variance. The rationale is that, when
Note that the relative RIR relating signal to its delayed version is causal and the relative RIR relating a signal to its advanced version is non causca (63 See Appendix F for intuitive explanation.

---

frequency permutation happens, X, X (c,t,, would contain multiple sources, and the resulting variance would be larger than that computed when X (c,t,-) (,-) contains only one source. LISMS echoes IVA's idea of assuming shared variance term across all the frequencies. If the ratio in (T0) becomes smaller, it indicates that the magnitudes of x (c,t,-) are more centered around their
mean. This is similar to optimizing the likelihood of X (c, (,, (, under a Gaussian distribution with variance term shared across all the frequencies. In (TO). logarithmic compression is applied, since log-compressed magnitudes better follow Gaussian distribution than raw magnitudes [66].
We combine LIsMS with LmC in ② for DNN training, using weighting term 7 E R:
LMCHIMM LMC + ? LISMS.
(11)
4.6 Training UNSSOR for monaural unsupervised separation
UNSSOR can be trained for monaural unsupervised separation by only feeding the mixture at the reference microphone to the DNN but still computing the loss on multiple microphones. Fig. [ illustrates the idea, At run time, the trained system performs monaural under-determined separation, and multi-microphone over-determined mixtures are only required for DNN training. The loss computed at multiple microphones could guide the DNN to exploit monaural spectro-temporal patterns for separation, even in an unsupervised setup.
5 Experimental setup
We validate the proposed algorithms on two-speaker separation in reverberant conditions based on the six-channel SMS-WSJ dataset (67] (see Appendix for its details). This section deseribes the baseline systems and evaluation setup. See Appendix for miscellaneous system and DNN setup.
5.1 Baselines
The baselines include conventional unsupervised separation algorithms, an improved version of RAS, MixIT, and supervised learning based models.
We include spatial clustering 46 48 for comparison. We use public implementatior
68 which leverages complex angular-central Gaussian mixture models [88 for sub-band spatial clustering and exploits inter-frequency correlation of cluster posteriors number of sources is set to three, one of which is used for garbage collection, following [44]. After discard the one with the lowest energy. The STFT window size is tuned
46 64 for frequency alignment. The
obtaining the estimates, to 128 ms and hop size to 16 ms.
We include IVA 41 44 for comparison. We use the public implementations provided by the torchiva toolkit [69]. We use the default spherical Laplacian model to model source distribution. In over- determined cases, the number of sources is set three and we discard the estimate with the lowest energy, similarly to the setup in the spatial clustering baseline The STFT window size is tuned to 256 ms and hop size to 32 ms.
We propose a novel variant of the RAS algorithm 36 for comparison. Appendix H discusses the differences between UNSSOR and RAS. Since RAS cannot achieve unsupervised separation, we improve it by computing loss on multi-microphone mixtures, and name the new algorithm as improved RAS (iRAS). We employ the time-domain Wiener filtering (WF) technique in (36) to filter re-synthesized time-domain estimates (cc) iSTFT(2(c)) where Z(c) is produced by TF-GridNet. The loss is defined as:
CIAS = Σ p=1 ag CpLIRAS.
Σ
p=1 Olp
Up
- c -_ hp(c)) * {(c) h(()))
(12)
Tppll1
with * denoting linear convolution, yp the time-domain mixture at microphone p, and (c) a time-domain Wiener filter computed by solving the following problem:
h)(Cc) trgmin hp(c) ((cc) * 2(c)
(13)
httpa://github. http://ggituuu..mm/ggggeegg com/fgat/pb httggggttpbbbb/bbeeebeeeeee wws/blob/mastee/eeeeeeeeeeeeeeee hhaa///ol//aaster//rraaa/s////as sotiree_mmde____e wttpmmmddeeeepleelmlmmeee ipynb For IVA and spatial clustering, using garbage source leads to better separation in experiments.
7

---

Table 1: Averaged results of 2-speaker separation on SMS-WSJ (6-channel input and loss).
Val. set
sest
Row
Systems
I
J
Loss
SDR (dB)
SDR (dB)
SI-SDR (dB)
PESQ
cSTOI
On
Mixture
0.1
0.1
0.0
1.87
0.603
UNSSOR
19
0
Case
12.5
11.9
10.2
2.61
0.735
1b
UNSSOR Conz. based treq, align. UNSSOR Oracle freq. align.
19
0
Luc
16.1
15.7
14.7
3.47
0.884
Eme
15.8
UNSSOR
19
0
Cascasss
16.0
15.8
14.6
3.44
0.885
UNSSOR Corr. based freq, align.
19
Cocc-isss
16.0
15.6
14.7
3.44
0.885
2c
UNSSOR + Oracle freq. align.
19
0
16.0
15.6
14.7
3.11
0.886
Spatial clustering Con, based freq. align. 44
8.8
8.6
7.4
2.44
0.726
IVA [4]
10.3
10.8
8.9
2.58
0.764
IRAS w/ causal 2-ttp filters
CaRA
7.8
7.6
5.7
2.14
0.642
3d
IRAS w/ noo-causal 512-tap filters (100 future taps)
CRAS
8.0
7.8
5.7
2,13
0.637
PIT (supervised)
26
19.9
19.4
18.9
4.08
0.949
show's grey using ornele informarion using supervised models.
Table 2: Averaged results of 2 2-speaker separation on SMS WSJ (3-channel input and loss).
Val. set
Test set
Row
Systems
I J
Loss
SDR (dB)
SDR (dB)
SI-SDR (dB)
PESQ
cSTOI
Mixture
0.1
0.1
0.0
1.87
0.603
UNSSOR
19
Las
7.4
1b
UNSSOR Corr. based freq. align. UNSSOR Oracle freq. align.
19
0
CMc
15.3
15.0
13.9
3.18
0.867
19
Lise
15.5
15.2
14.1
3.19
0.871
Za
UNSSOR
19
0
Lvcc usMs
15.7
15.4
14.4
3.20
0.874
UNSSOR Corz. freq. align. UNSSOR + Oracle freq. align.
19
Excc-isss
15.7
15.4
2c
19
0
15.8
15.4
14.5
3.20
0.876
Spatial clustering Com, based freq. alignn 44
9.6
9.5
8.5
2.52
0.759
3b
IVA [+
11.6
12.0
10.7
2.67
0,802
512-lup filters
Caas CRRA
2.7
3d
iRAS w/ non causal 512-tu filters (100 fature tups)
4.6
4.5
2.2
1.87
0.579
PIT (supervised)
25
17.4
16.8
16.3
3.91
0.924
which is quadratic and has closed- solution. The separation result is computed A:F (c)
(c)(c) + ((). Following [36], we use 512 filter taps, and filter the future 100, the current, and the past 411 samples (i.e., non-causal filtering). We can also filter the current and the past 511 samples (i.e., causal filtering), and additionally experiment with a filter length (in time) same as the length of the FCP filters (see Appendix D.
For comparison, we also include MixIT 30). which requires using synthetic mixtures of mixtures.
We report the result of using supervised learning, where PIT S is used to address the permutation problem. This result can be viewed as a performance upper bound of unsupervised separation.
We use the same DNN and training configurations as those in UNSSOR for a fair comparison.
5.2 Evaluation setup and metrics
We designate the first microphone as the reference microphone, and use the time-domain signal corresponding to X, (c) of each speaker for metric computation. The evaluation metrics include iignal-o-ddistortoon ratio (SDR) (70), scale-invariant SDR (SI-SDR) [7T), perceptual evaluation of speech quality (PESQ) (72). and extended short-time objective intelligibility (cSTOI) [73]. SI-SDR and SDR evaluate the sample-level accuracy of predicted signals, and PESQ and eSTOI are objective metrics of speech quality and intelligibility respectively. For all the metrics, the higher, the better.
Evaluation results
This section reports evaluation results on SMS-WSJ and compares the performance of various setups.
6.1 Effectiveness of UNSSOR at promoting separation
Table I and respectively report the results of using six- and three-microphone input and loss. After hyper-parameter tuning, in default we use the loss in (9) for DNN training, set 19 and (defined below (6)) for FCP (i.c., causal FCP filtering with 20 taps), and set ap = 1 (meaning
8

---

Table 3: Averaged results of 2-speaker separation on SMS-WSJ (1-channel input and 6-channel loss).
Val. set
sest
Row
Systems
I
J
Loss
SDR (dB)
SDR (dB)
SI-SDR (dB)
PESQ
cSTOI
On
Mixture
0.1
0.1
0.0
1.87
0.003
UNSSOR
19
1
Cocc-isas
13.0
12.5
11.9
3.27
0.832
causal 2-tupp filters
CIRAS Caas
7.2
5.6
2.03
0.641
2b
IRAS non causal 512 filters (100 Tuture tups)
10.7
10.5
9.7
2.80
0.778
Monaural (supervised)
23
15.7
15.3
3.79
0.907
Table 4: Averaged results of 2-speaker separation on SMS-WSJ (1-channel input and 3-channel loss).
Val. set
Test set
Row
Systems
1
J
Loss
SDR (dB)
SDR (dB)
SI-SDR (dB)
PESQ
cSTO1
Mixture
0.1
0.1
0.0
1.87
0.603
UNSSOR
19
Cxccaass
12.5
12.0
11.4
3.18
0.822
2a iRAS w/ causal 512-tup üiliers
CaaA
-0.1
-0.3
-3.0
1.62
0.453
iRAS non causal 512 lup filters future tups)
CIRAS
11.0
10.7
9.9
2.81
0.783
3a
Monaural PIT (supervised)
23
16.2
15.7
15.3
3.79
0.907
weighting is applied for different microphones). For the 3-microphone case, we use the mixtures at the first, third, and fifth microphones for training and testing. Notice that for two-speaker separation, the cases with six or three microphones are both over-determined.
In both tables, from row la we observe that UNSSOR produces reasonable separation of speakers, improving the SDR from 0.1 to, for example, 12.5 in Table but its output suffers from the frequency permutation problem (see Appendix for an example). In row 1c, we use oracle target speech obtain oracle frequency alignment and observe much better results over 1a. This shows the effectiveness of LMc at promoting separation of speakers and the severity of the frequency permutation problem. In row 1b, we use a frequency alignment algorithm (same as that used in the spatial clustering bascline) [46] 64 to post-process the separation results of la. This algorithm leads to impressive frequency alignment (see lb vs. Ic), but it is empirical and has a complicated design.
6.2 Effectiveness of ISMS loss at addressing frequency permutation problem
We train DNNs using LMCHIMS defined in TT.. In each case (i.e, six- and three- prre--iicrophone), we separately tune the weighting term y in (TT based on the validation set. In both table, comparing row 2a-2c with 1 la-lc, we observe that including LIsMS is very effective at dealing with the frequency permutation problem, yielding almost the same performance as using oracle frequency alignment.
6.3 Results of training UNSSOR for monaural unsupervised separation
Table and use the mixture only at the reference microphone I as the network input, while computing the loss respectively on three and six microphones. We tune J to 1 (i.e., non-causal FCP filter), considering that, for a specific target speaker, the reference microphone may not be the microphone closest to that speaker See interpretation on why non-causal filtering is needed in Appendix G.2 We still set the microphone weight a, to 1.0 for non-reference microphones (i.e., when p * T), but tune on to smaller value based on the validation set. Without using smaller on, we found that the DNN easily overfits to microphone 1, as we use the mixture at microphone 1 as the only input and compute the MC loss also on the mixture at microphone 1. The DNN can just aggressively optimize LMC,p to zero at microphone 1 and not optimize that at other microphones.
From row la of both tables, strong performance is observed in this under-determined setup, indicating that the multi-microp microphone loss can inform the DNN what desired target sound objects are and the DNN can learn to model spectral patterns in speech for unsupervised separation. In addition, the result in la of Table iss better than that in Table 4 This indicates that using more microphones as constraints in the proposed MC loss can elicit better separation.
'We do not need many future taps, considering that the hop size is 8 ms in our system and the microphone array in SMS- WSJ is a compact array with diameter of 20 em. In air, sound would travel 340 x 0.008 = 2.72 meters in 8 ms if its speed is 340 meters per second. This distance is far larger than the array aperture size.

---

6.4 Comparison with other methods and supplementary results
In Table 14 we compare the performance of UNSSOR with spatial clustering, IVA, iRAS, and supervised PIT-based models. In Appendix J we compare UNSSOR and iRAS when they use the same filter length (in time). UNSSOR shows better performance than previous unsupervised separation models that can be performed or trained directly on mixtures. A sound demo is available this link UNSSOR is worse than supervised PIT but the performance is reasonably strong. For example, in row 2a of T UNSSOR obtains 15.4 dB SDR on the test set, which is close to the 16.8 dB result obtained by supervised PIT in 4a. In Appendix L with MixIT 30 and observe better performance.
we compare the results of UNSSOR
We think that the effectiveness of the proposed system comes from both the strong modeling capability
of TF-GridNet and the UNSSOR mechanism itself. In Appendix K we use UNSSOR with other separation models such as TCN-DenseUNet (74) and DPRNN [75), which known have weaker modelling capability than TF-GridNet in supervised separation tasks (see for example 23 We observe that the separation performance is worse but still reasonable.
Limitations
Our study shows the strong potential of UNSSOR for unsupervised speech separation. There are, however, several weaknesses we need to address in future research. First, we assume that sources are directional point sources so that each relative RIR can be modelled using a short filter, and diffuse sources are not considered. Second, we assume that sources are non-moving within each utterance so that we can use time-invariant FCP filters. Third, we assume that the number of sources known and the sources are fully-overlapped. Fourth, only measurement or modeling noise is considered and realistic directional or diffuse background noises with strong energy are not included. Although these assumptions are also made in many algorithms such as IVA, spatial clustering, RAS and iRAS, they need to be addressed to realize more practical and robust speech separation systems.
8 Conclusion
We have proposed UNSSOR for unsupervised neural speech separation. We show that it is possible to train unsupervised models directly on mixtures, if the mixtures over-determined. We have proposed mixture-constrriit loss functions, which leverage multi-microphone mixtures as constraints, to promote separation of speakers. We find that minimizing ISMS can alleviate the frequency permutation problem. Although UNSSOR requires over-determined training mixtures, it can be trained to perform under-determined unsupervised separation. Future research will combine UNSSOR with semi-supervised learning, evaluate it on real-recorded noveyrrevetterant data such as CHiME- 766. AMI [77) and AliMeeting (78). and address the limitations described in Section 7
In closing, we emphasize that key scientific contribution of this paper is that the over-determined property afforded by having more microphones than speakers can narrow down the solutions the underlying sources, and this property can be leveraged to design a supervision to train DNNs to model speech patterns via unsupervised learning and realize unsupervised separation. This meta-idea, we believe, would motivate the design of many algorithms in future research on neural source separation.
Acknowledgments and Disclosure of Funding
We would like to thank Dr. Robin Scheibler at LINE Corporation for constructive discussions on IVA, and Dr. Samuele Corelll for helpful discussions MixIT. This research is part of the Delta research computing project, which is supported by the National Science Foundation (Award OCT 2005572) and the State of Illinois. Delta is joint effort of the University of Illinois at Arbaan-Chaampain and its National Center for Supercomputing Applications. We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the RTX 8000 GPUs used in this research.
https://zqwang// .ithhb. httpa////.oooo//////-ttttttt
10

---

References
[ C. E. Chery, "Some Experiments the Recognition of Speech, with One and with Two Ears," Journal of the Acoustical Sociery of America, vol. 25, no. 5, pp. 975-979, 1953.
[2] J. H. McDermott, "The Cocktail Party Problem," Current Biology, vol. 19, no. 22, pp. 1024-1027, 2009.
[3] D. Wang and J. Chen, "Supervised Speech Separation Based on Deep Learning: An Overview." IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 10, pp.
2018.
[4] R. Hershey, Z. Chen, Le Roux, and S. Watanabe, "Deep Clustering: Discriminative Embeddings for Segmentation and Separation," in Proc. ICASSP, 2016, pp. 31-35.
[5] M. Kolbæk,, D. Yu, Z.-H. Tan, and J. Jensen, "Multitalker Speech Separation with Utterance- Permutation Invariant Training of Deep Recurrent Neural Networks," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 10, pp. 1901-1913, 2017.
[6] Y. Luo, Z. Chen, and N. Mesgarani, gopeaker-Independemn Speech Separation with Deep Attractor Network," JEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 4, pp. 787-796, 2018,
[7] Z.-Q. Wang and D. Wang, "Combining Spectral and Spatial Features for Deep Learning Based Blind Speaker Separation." IEEÉ/ACM Transactions on Audio, Speech, and Language Procexsing, vol. 27, no. 2, pp. 457-468, 2019.
[8] A. Ephrat, 1. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein, Audio- Model for Speech Separa- Graphics, vol. 37, no. 4, 2018.
"Looking to Listen at the Cocktail Party: A tion," ACM Pransactions
[9] Y. Liu and D. Wang, "Divide and Conquer: A Deep CASA Approach to Talker-Independent Monaural Speaker Separation," /EEF/ACM Transactions Audio, Speech, and Language Processing, vol. 27, no. 12, pp. 2092-2102, 2019.
[10] Y. Luo and N. Mesgarani, "Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation," IEEE/ACM Transactions Audio, Speech, and Language Processing, vol. 27, no. 8, pp. 1256 1266, 2019.
[11] Z.-Q. Wang, K. Tan, and D. Wang, "Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspective," in Proc. ICASSP, 2019, pp. 71-75.
[12] K. žmoliková, M. Deleroix, K. Kinoshita, T. Ochiai, T. Nakatani, L. Burget, and J. Čemocký, "Speaker- Beam: Speaker Aware Neural Network Target Speaker Extraction Speech Mixtures," " IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 4, pp. 800-814, 2019.
[13] Kavalerov, S. Wisdom, H. Erdogan, B. Patton, K. Wilson, Le Roux, and J. R. Hershey, "Universal Sound Separation," in Proc. WASPAA, 2019, pp. 175-179.
[14] D. Yin, C. Luo, Xiong, and W. Zeng, "PHASEN: A Phase-and- Haressdss--mmoiss-AAAaaar Speech Enhancement Network," " Proc. AAAI, 2020, pp. 9458-9465.
[15] Y. Xu, M. Yu, S.-X. Zhang, L. Chen, C. Weng, J. Liu, and Yu, "Neural Spatio-Temporal Beamformer for Target Speech Separation," in Proc. Interspeech, 2020, pp. 56 60-
[16] T. Jenrungrot, V. Jayaram, S. Seitz, and I. Saanelmachee-Shhizemmee "The Cone of Silence: Speech Separation by Localization." " in Proc. NewrIPS, 2020.
[17] C. Tangg C. Luo, 7. Zhao, W. Xie, and W. Zeng, "Joint Time- Frequency and Time Domain Learning for Speech Enhancement," in Proc. IJCAI, 2020, pp. 3816-3822.
[18] E. Nachmani, Y. Adi, and L. Wolf, "Voice Separation with An Unknown Number of Multiple Speakers," in Proc. ICML, 2020, pp. 7121-7132.
[19] N. Zeghidour and D. Grangier, "Wavesplit: End-to-End Specch Separation by Speaker Clustering," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2840 28400222,,990 2021.
[20] 7. Chen, T. Yoshioka, I., Lu, T. Zhou, Z. Meng, Luo, J. Wu, X. Xiao, and J. Li, "Continuous Speech Separation: Dataset and Analysis," in Proc. ICASSP, 2020, pp. 7284-7288
[21] Z. Zhang. Y. Xu, M. Yu, S. X. Zhang, L. Chen, D. S. Williamson, and D. Yu, "Multi- Channel Multi-Fram ADL- ADL-MVDR for Target Speech Separation," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2526-3540, 2021.

---

[22] J. Rixen and M. Renz, "SFSRNet: Super-Resolation for Single-Channel Audio Source Separation." in Proc. AAAI, 2022.
[23] Z.-Q. Wang. S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe, "TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31. pp. 3221-3236, 2023.
[24] Y. Masuyama, X. Chang, W. Zhang, S. Cornell, Z.-Q. Wang. N. Ono, Y. Qian, and S. Watanabe, "Exploring The Integration Speech Separation and Recognition with Self-Supervised Learning Representation, in Proc. WASPAA, 2023, pp. 1-5.
[25] K. Saijo. W. Zhang, Z.-Q. Wang, S. Watanabe, Kobayashi, and Ogawa, "A Single Speech Enhancement Model Unitying Dereverberation, Denoising, Speaker Counting, Separation, and Extraction," in Proc. ASRU, 2023.
[26] W. Zhang. K. Saijo. Z.-Q. Wang. S. Watanabe, and Y. Qian, "Toward Universal Speech Enhancement for Diverse Input Conditions," in Proc. ASRU, 2023.
[27] A. Pandey and D. Wang. "On Cross-Corpus Generalization of Deep Learning Based Speech Eshancement," IEEC/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2489-2499, 2020.
[28] W. Zhang, J. Shi, C. Li, S. Watanabe, and Y. Qian, "Closing The Gap Between Time-Domain Domain Julti-Channel Speech Enhancement on Real and Simulation Conditions," in Proc. WASPAA, 2021, pp. 146-150.
[29] Z.-Q. Wang, G. Wichern, and J. Le Roux, "Convolutive Prediction for Monaural Speech Dereverberation and Noisy- Reverhrant Speaker Separation," IEEF/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3476-344 34960 2021.
[30] S. Wisdom, E. Tzinis, H. Erdogan, R. J. Weiss. K. Wilson, and J. R. Hershey, "Unsupervised Sound Separation using Mixture Invariant Training," " in Proc. NeurlPS, 2020.
[31] F. Tzinis, S. Wisdom, A. Jansen, S. Hershey, T. Remez, D. P. W. Ellis, and J. R. Hershey, "Into the Wild with AudioScope: Unsupervised Visial Separation of On-Screen Sounds," in Proc. ICLR, 2021.
[32] E. Tzinis, Y. Adi, V. K. Ithapu, B. Xu, P. Smaragdis, and A. Kumar, "RemixlT: Continual Self-Training of Speech Enhancement Models via Bootstrapped Remixing," " IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1329-1341, 2022.
[33] E. Tzinis, Wisdom, T. Remez, and J. R. Hershey, "AudioScopeV2: Audio- Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation," " Pruc. ECCV, 2022, pp. 368-385.
[34] K. Saijo and T. Ogawa, "Self-Remixxng: Unsupervised Speech Separation via Separation and Remixing," in Proc. ICASSP, 2023.
[35] A. Sivaraman, Wisdom, H. Erdogan, and Hershey, "Adapting Speech Separation To Real-World Meetings Using Mixture Invariant Training," " Proc. ICASSP, 2022, pp. 686-690.
[36] R. Aralikatti, C. Boeddeker, G. Wichem, A. S. Suhramanian, and J. Le Roux, "Reverheration as Supervision for Speech Separation," in Proc. ICASSP, 2023.
[37] S.-l. Amari, A. Cichocki, and H. Yang, "A New Learning Algorithm for Blind Signal Separation," in Proc. NeurIPS, vol. 8, 1996, pp. 757-763.
[38] A. Hyvärinen and E. Oja, "Independent Component Analysis: Algorithms and Applications," " in Neural Networks, vol. 45, no. 4-5, 2000, pp. 411-430.
[39] C. Puntonet and E. Lang. "Blind Source Separation and Independent Component Analysis," Neuro- computing, vol. 6, no. 1, pp. 1-57, 2006.
[40] T. Adali, M. Anderson, and G.-s. Fu, "Diversity in Independent Component and Vector Analyses," IEEE Signal Processing Magazine, vol. 31, no. 3, pp. 18-33 13,, 2014.
[41] H. Sawada, N. Ono, H. Kameoka. Kitamura, and H. Saruwatari, "A Review of Blind Source Separation Methods: Two Converging Routes to ILRMA Originating from ICA and NMF." APSIPA Transactions on Signal and Information Processing, vol. 8, pp. 1-14, 2019.
[42] T. Kim, T. Eltoft, and T.WW. Lee, "Independent Vector Analysis: An Extension of ICA to Multivariate Components," in International Conference on Independent Component Analysis and Signal Separation, 2006, pp. 162-
12

---

[43] A. Hiroe, "Solution of Permutation Problem in Frequency Domain ICA, using Multivariate Probability Den- sity Functions," in International Conference on Independent Component Analysis and Signal Sepuration. 2006. pp. 601-608.
[44] C. /Boeddeker, F. Rautenberg, and R. Haeb-Umbach, "A Comparison and Combination of Unsupervised Blind Source Separation Techniques." in ITG Conference on Speech Communication, 2021, pp. 133.
[45] S. Rickard, "The DUET Blind Source Separation Algorithm." in Blind Speech Separation, 2007, pp. 217-237.
[46] D. H. T. Vu and R. Hacb-Umbach, "Blind Speech Separation Employing Directional Statistics in An Expectation Maximization Framework," Proc. ICASSP, 2010, pp. 241-244.
[47] H. Sawada, S. Araki, and S. Makino, "Underdetermined Convolutive Blind Source Separation via Frequency Bin-Wise Clustering and Permutation Alignment," JEEE Transactions Audio, Speech, and Language Processing, vol. 19, no. 3. pp. 516-527, 2011.
[48] N. Ito, Araki, and T. Nakatani, "Complex Angular Central Gaussian Mixture Model for Directional Statistics in Mask-Based Microphone Array Signal Processing," in Proc. EUSIPCO, 2016, pp. 1153-1157.
[49] E. Tzinis, S. Venkataramani, and P. Smaragdis, "Unsupervised Deep Clustering for Source Separation: Direct Learning from Mixtures Using Spatial Information." in Proc. ICASSP, 2019, pp. 81-85.
[50] L. Drude, D. Hasenklever, and R. Haeb-Umbach, "Unsupervised Training a Deep Clustering Model for Multichannel Blind Source Separation," in Proc. ICASSP, 2019, pp. 695-699.
[51] P. Scetharaman, G. Wichern, J. Le Roux, and B. Pardo, "Bootstrapping Single- Channel Source Separation via Unsupervised Spatial Clustering on Sterco Mixtures," in Proc. ICASSP, 2019, pp. 356-360.
[52] M. Togami, Y. Masuyama, T. Komatsu, and Y. Nakagome, "Unsupervised Training for Deep Speech Source Separation with Kullhack-Leeibee Divergence Based Probabilistic Loss Function," in Proc. ICASSP, vol. 2020-May, 2020, pp. 56-60.
[53] L. Drude, J. Heymann, and R. Umbach, "Unsupervised Training of Neural Mask-based Beamforming," in Proc. Interspeech, 2019, pp. 1253-1257.
[54] Y. Bando, K. Sekiguchi, Y. Masuyama, A. A. Nugraha, M. Fontaine, and K. Yoshii, "Neural Full-Rank Spatial Covariance Analysis for Blind Source Separation," IEEE Signal Process. Lett., vol. 28, pp. 1670- 1674, 2021.
[55] Y. Bando, T. Aizawa, K. Itoyama, and K. Nakadai, "Weakly-Supervised Neural Full-Rank Spatial Covari- ance Analysis for Front-End System of Distant Speech Recognition," in Proc. Interspeech, 2022, pp- 3824-3828.
[56] R. Talmon, I. Cohen, and Gannot, "Relative Transfer Function Identification using Convolutive Transfer Function Approximation," " JEEE Transactions Audio, Speech, and Language Processing, vol. 17, no. 4, pp. 546-555, 2009.
[57] A. Levin, Y. Weiss, F. Durand, and W. Freeman, "Understanding Blind Deconvolution Algorithms," " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 12, pp. 2367-2367, 2011.
[58] Z.-Q. Wang, P. Wang, and D. Wang, "Multi-Microphone Complex Spectral Mapping for Utterance Wise and Continuous Speech Separation," JEEF/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2001-2014, 2021.
[59] S. Wisdom, J. R. Hershey, K. Wilson, J. Thorpe, M. Chinen, B. Patton, and R. A. Saurous, "Differentiable Consistency Constraints for Improved Deep Speech Enhancement," in Proc. ICASSP, 2019, pp. 900-904.
[60] 7.-Q. Wang, G. Wichem, and J. Le Roux, "Convolutive Prediction for Reverherant Speech Separation," in Proc. WASPAA, 2021, pp. 56-60.
[61] T. Yoshioka, T. Nakatani, M. Miyoshi, and H. G. Okuno, "Blind Separation and Dereverberation of Speech Mixtures by Joint Optimization," IEEE Transactions Audio, Speech, and Language Processing, vol. 19, no. pp. 69-84, 2011.
[62] R. Hacb- Umbach, J. Heymann, L. Drude et al., "Far-Field Automatic Speech Recognition," Proc. IEEE, 2020.
[63] A. V. Oppenheim, Discrete- Time Signal Processing. Pearson Education India, 1999.
13

---

[64] H. Sawada. S. Araki, and S. Makino, "A Two-Stage Frequency-Domain Blind Source Separation Method for Underdeternined Convolutive Mixtures," in Proc. WASPAA, 2007, pp. 139-142.
[65] H. Sawada, R. Mukai, S. Araki, and S. Makino, "A Robust and Precise Method for Solving The Permutation Problem of Sesquency-Domaalle Blind Source Separation." IEEE Transactions on Speech and Audio Processing, vol. 12, no. 5. pp. 530-538, 2004.
[66] S. Gannot, E. Vincent, S. Markovich- and A. Ozerov, "A Consolidated Perspective Multi- Microphone Speech Enhancement and Source Separation." " IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, pp. 692-730, 2017.
[67] L. Drude, Heitkaemper. Boeddeker, and R. Haeb-Umbach, "SMS-WSJ: Database, Performance Measures, and Baseline Recipe for Multi-Channel Source Separation and Recognition," in arXiv preprint onXiv:1910.1 13934, 2019.
[68] C. Boeddeker, 2019.
model
[69] R. Scheibler and K. Saijo,
2022.
[70] E. Vincent, R. Gribonval, and C. Févotte. "Performance Measurement in Blind Audio Source Separation." IEEE Transactions Audio, Speech, and Language Processing, vol. 14, no. 4, pp. 1462-1469, 2006.
[71] Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, "SDR Half-Baked or Well Done?" Proc. ICASSP, 2019, pp. 626-630.
[72] A. Rix, Beerends. M. Hollier, and A. Hekstra, "Perceptual Evaluation Speech Quality (PESQ)-A New Method for Speech Quality Assessment of Telephone Networks and Codecs," " in Proc. ICASSP, vol. 2, 2001, pp. 749-752.
[73] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, "An Algorithm for Intelligibility Prediction of Time- Time-Frequency Weighted Noisy Speech," IEEE Transactions Audio, Speech, and Language Processing. vol. 19, no. 7. pp. 2125-2136, 2011.
[74] Z.-Q. Wang, G. Wichern, and J. Le Roux, "Leveraging Low-Distortion Target Estimates for Improved Speech Enhancement," arXiv preprint arXiv:21 2021.
[75] Y. Luo, Z. Chen, and T. Yoshioka. "Dual-Path RNN: Efficient Long Sequence Modeling for Single-Channel Speech Separation." in Proc. ICASSP, 2020, pp. 46-50.
[76] S. Watanabe, M. Mandel, J. Barker, F. Vincent, A. Arora et al., "CHiME-6 Challenge: Tackling Multi- speaker Speech Recognition for Unsegmented Recordings." in arXiv preprint arXiv:2004.092.99 2020. 2020.
[77] J. Carletta, S. Ashby, S. Bourban, M. Flynn et al., "The AMI Meeting Corpus: A Prmouuucnmmmmmt"" in International Workshop Machine Learning for Multimodal Interaction, 2006, pp. 28-39.
[78] F. Yu, S. Zhang. Y. Fu, I. Xie, S. Zheng, 7. Du, W. Huang, P. Guo, 7. Yan, B. Ma, X. Xu, and H. Bu, "M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge," in Proc. ICASSP, 2022, pp. 6167-6171.
[79] K. Tan and D. Wang. "Learning Complex Spectral Mapping With Gated Convolutional Recurrent Networks for Monaural Speech Enhancement," JEEE/ACM Transactions on Audio, Speech, and Language Processing. vol. 28, pp. 380 380-390, 2020.
[80] Y. Hu, Liu, S. Lv et al., "DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement," in Proc. Interspeech, 2020, pp. 2472-2476.
[81] Y. Liu and D. Wang, "Causal Deep CASA for Monaural Talker-Independent Speaker Separation," " IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 1270-1279, 2020.
[82] A. Pandey and D. Wang, "Densely Connected Neural Network with Dilated Convolutions for Real-Time Speech Enhancement in The Time Domain," in Proc. ICASSP, 2020, pp. 6629-6633.
[83] H. Taherian, K. Tan, and Wang, "Multi-Channe" Talker- Independntt Speaker Separation Through Location-Based Training," JEEE/ACM Trunsactions on Audio, Speech, and Language Processing, vol. 30, pp. 2790- 2022.
[84] J. Zhang, C. Zorila, R. Doddipatla, and J. Barker, "On End-to-End Multi-CCannel Time Domain Speech Separation in Reverberant Environments." in Proc. ICASSP, 2020, pp. 6389-6393
14

---

Appendix
This appendix is organized as follows:
Appendix
describes the SMS-WSJ dataset.
Appendix visualizes the surface of the proposed mixture-constraint loss to show that minimizing
the loss can promote unsupervised separation of speakers.
Appendix
provides the derivation that FCP can be used to approximate speaker images.
Appendix illustrates the frequency permutation problem.
Appendix describes differences between the MC loss and the mixture consistency concept.
Appendix illustrates the cases on when to use causal and non-causal filtering.
Appendix provides interpretations of the physical meanings of intermediate DNN estimates 2.
Appendix discusses differences between UNSSOR and RAS.
Appendix presents miscellaneous system and DNN configurations.
Appendix experiments with alternative filters taps for iRAS and UNSSOR.
Appendix experiments UNSSOR with DNN architectures with lower modelling capabilities.
Appendix reports the results of MixIT.
A SMS-WSJ dataset
SMS- WSJ 67 is a popular corpus for evaluating two-speaker separation algorithms in reverberant conditions. The clean speech is sampled from the WSJO and WSJ1 datasets. The corpus contains 33, 561 (~87.4 h), 982 (~2..5 b); and 1, 1,332 (~3.4 h) two-speaker mixtures respectively for training, validation, and testing. The simulated microphone array has six microphones arranged uniformly on a circle with a diameter of 20 cm. For each mixture, the speaker-to-ane torkerr-t-array distance is drawn from the range [1.0,2. 2.0] m. and the reverberation time (T60) is sampled from [0.2, 0.5] s. A weak white noise is added to simulate microphone self-noises, and the energy level between the sum of the reverberant speech signals and the noise sampled from the range [20, 30] dB. The sampling rate is kHz.
B Visualization of loss surface
Fig. visualizes the values of LMc in (4) based on a six-channel noisy-reverberant two-speaker mixture sampled from the SMS-WSJ dataset (see Appendix for the dataset details). Let C = 2 and suppose that the DNN estimates are
(1) n Xi(1) + (2) 21/2
2(2) = (1 - 1) x x1 + (1 v) x (2) + E1/2,
(14)
where f and E R are bounded in the use μ and / to mimic the cases that the DNN produces (1) good separation (i.e., when μ and v R O, or μ R and x 1); and (2) bad separation (i.e., when μ and away from 0 and 1, meaning that each esti- mate contains multiple speakers, and when R and v R O, or RO 1 and v & 1, merged
10
range [0, 1]. Essentially,
0.50
both
0.60
0.45
meaning that the two speakers into one output and the other output does not contain any speakers). Fig. 2 enumer- ates μ and y and plots the resulting separa- tion results against the loss value of 4 We can see the loss values are smallest when or when μ 0 and
0.35
0.30
0.0
1 and v I ~ 1 (i.e., when the speakers are success- fully separated), and clearly larger other- wise. This indicates that minimizing the proposed loss function can encourage sep- aration.
0.66
Figure 2: Loss surface of Luc in with P - 6 and C' = 2 (i.e., over-determined conditions) against hypothesized separa- tion outputs generated by using various μ and v. Best viewed in color.
15

---

C Effectiveness of FCP at approximating speaker images
Following the derivation in [2)]. let us define the mixture as Y, Xp(c) + Vi(c), where Vi(e)
consists of the signals of all the sources but c. We can formulate
argmin E. so
Xplc,t,f) + Velct, f) gp(c, t(
Bp (c,f)
argmin sple,))
xp(c,t,) s) fl
splo,t,f)
Sp(c,t,f) xx(e4,1) (xlct,,))x((LL))))
8p(c, f) ( (s
argmin sple,))
(15)
where the derivation from the first row to the second is based on the assumption that, as the DNN training continues, (c)) would become more and more accurate that, after some epochs, it can become uncorrelated (or little correlated) with Vp(c), meaning that the cross-temn would small:
1 Σ, Ex {p(c,1,1)
-aalf)-gp(cc.. f) (gl(.) ((ctt.))) Volc,t,)) =0
(16)
From the third row of 15). b, the resulting 8p(c, f) 2(c, (,,, would approximate Xp(c, f)..
D Illustration of frequency permutation problem
We use three-channel input and loss for DNN training. Fig. 3aa and (b) show an example separation result of the model trained with LMC in (9). Comparing the separated speech in Fig. (3) and (b) with the clean speech in (c) and (d), we can see that the separated speech suffers from the frequency permutation problem approximately in the range [1.6, 2.9) kHz. Fig. B(e) and (f) show the separation result of the model trained with LMC+ISMS in which effectively addresses the frequency permutation problem.
(a)
Frequency (KHz2)
(c)
(d)
Frequency Frequency (4H2)
(e)
1.5
32
1.6 Time (s)
3.2
Time (s)
Figure 3: Example spectrograms of (a)-(b): FCP-estimated speaker image and 2 with SDR scores of 8.7 and 7.7 dB (using Exe in 9 for training): (c)-(d): oracle speaker image and 2; and (e)-(f): FCP-estimated speaker image and with SDR scores of 17.1 and 16.8 (using LAccusss (TT) training). The blue rectangles in (a) and (b) mark the region with frequency permutation. The mixture SDR scores of the two speakers are respectively 0.2 and - dB, Best viewed in color.
16

---

E Differences between mixture-constraint loss and mixture consistency
The proposed mixture-constrrint loss, Lvc, has very different physical meanings and mathematical forms compared the mixture consistency term proposed in [59]. First, in [59], DNN estimates strictly constrained to add up to the mixture (see Eq. (7) and (9) in (5 while our MC loss only encourages the filtered DNN estimates to add up to the mixture. Second, our MC loss applied to filtered DNN estimates rather than directly to DNN estimates. Third, we deal with multi-microphone MC, while [59] only addresses single-channel cases. Fourth, the motivation is to use mixtures as constraints to regularize DNN estimates so that the estimates can approximate source images. This motivation is completely different from that of the mixture consistency term.
the proposed MC loss
F Illustration of using causal and non-causal filtering
This section illustrates that the relative RIR relating a signal to its delayed version causal and the relative RIR relating signal to its advanced version is non- causal.
wwm
In Fig. 4 suppose that the blue signal is the DNN es- timate for speaker c, and the orange signal is speaker c's image another microphone, which is a delayed version (i.e., reaching the microphone later). To filter the blue signal to approximate the oracle one, only need a causal filter to delay the blue signal.
Reversely, suppose that the orange signal is the DNN estimate for speaker c, and the blue signal is speaker c's image at another microphone, which is an ad- vanced version (i.e., reaching the microphone earlier). To filter the orange signal to approximate the blue signal, we need non-causal filter to advance the orangle signal.
10
Time stops
Figure 4: Example for illustrating causal and non- causal filtering. Best viewed in color.
G Interpretation of intermediate DNN estimates 2
This section provides an interpretation of the physical meanings of the intermediate DNN estimates Ž in the cases of using multi-chane malti-channel input and loss, and single-channel input and multi multi-channel loss.
G.1 Multi-chamnel-input case
In (c) constrained such that it can be filtered by a causal filter Ep(c) to approximate Xp).). Since there could be an infinite number of combinations of (c) and (p(c) whose convolution results would well approximate Xp)c), (() cannot be interpreted as the dry source signal and think it more similar to a signal captured by a virtual microphone that closer to speaker e than all the P microphones. See Fig. 5ja) for example, where each virtual microphone captures the direct-path signal of a target speaker earlier than any other microphones so that we can use causal FCP filters.
G.2 Monaural-input case
In the monaural-input case, each (c) would be aligned to the speaker's 's image at the input microphone (i.e., the reference microphone), since the DNN only has monaural input and in this case the DNN is not likely to align its outputs to a virtual microphone or any actual microphones.
We give an example in Figure 5bb), where the reference microphone captures speaker 2's direct- -ath signal later than all the other microphones. In this case, we just need to use non-causal FCP filters when filtering (c) (which estimated based on the monaural signal at the reference microphone) to approximate speaker 2's images captured the other microphones.
17

---

Speaker
Speaker #1
Virtual mic
Speaker
Vittallic 2
Reference mic
(a)
(b)
Figure 5: Illustration of signal alignment in (a) caatiichhanel--nnge multi-channel-loss case; and (b) single-
channel-inpul inputt chatne-hhnnel-sss case. Best viewed in color.
H Differences from RAS
In many aspects, UNSSOR differs from RAS (36, which deals with monaural two-speaker separation given binaural (two-channel) training mixtures. RAS first performs DNN-based monaural separation on the left-ear mixture in the magnitude T-F domain, then linearly filters the DNN estimates at the left ear through time-domain Wiener filtering such that the filtered estimate can approximate the right-ear mixture, and the training loss is computed between the summation of the filtered DNN estimates and the right-ear mixture. Besides differences in, for example, the DNN architectures, linear filtering algorithms, how phase estimation (important for estimating relative RIRs) is handled, how frequency permutation problem dealt with, and training data curation (difficult training examples need to be removed train RAS), the key difference is that RAS fails to be trained in unsupervised way [3). still needs labelled mixtures so that a supervised PIT pas-d model can be trained and then used as the starting point for their unsupervised algorithm.
We think that the ineffectiveness of RAS fully-unsupervised setup likely because the loss computed only on the right-ear mixture. Following analysis in Section 3 in RAS there
N 1 equations (because the loss is only computed on the right-ear time-domain signal, assumed N N-sample) but N C + (2 1) 512 x C' unknowns (where the 512 term is because the filter is assumed 512-tap in [36. and the (2 1) term is because there is only one filter for each speaker in the binaural setup, i.e., only non-reference microphone). This is ill-posed problem, not likely to be solved via the current RAS algorithm.
I Miscellaneous system and DNN setup
In default, for STFT, the window size is 32 ms, the hop size is 8 ms, and the square-root Hann window used the analysis window. For 8 kHz sampling rate, a 256-point discrete Fourier transform is applied to extract -i9-dimensional complex STFT spectra at each frame. This STFT setup is very common in modern deep learning based separation. Differently, the window and hop sizes some baselines such as IVA and spatial clustering are considerably larger (see detailed setup in Section 5.T). as we observe that this leads better separation performance, likely because more reverberation can be covered in each frame and, this way, their model assumptions can be better satisfied.
Our DNN architecture TF-GridNet 23) Using the symbols defined in Table of (23). set its hyper-parameters to D = 48, B = 4, / 4, = = 192, I. 4 and E for 8 kHz sampling rate, Please do not confuse the symbols TF-GridNet with the ones in this paper.
In each epoch, we sample an (-second mixture segment from each training mixture for model training. If mixture shorter than / seconds, we pad zeros in the front rather than in the end, since padding in the end would result in a mixture that has abrupt stop of reverberation, which is not realistic and would be detrimental front can avoid this problem. mixture is longer than seconds, we randomly pick an l-second segment. In default, is four seconds.
based relative RIR estimation. In comparison, padding zeros in the
We normalize the sample variance of each sampled mixture segment 1.0, before feeding them to DNN for training. Adam (with the default setup in Pytorch v1.9) is used as the optimizer. The
18

---

Table 5: Supplementary averaged results of 2-speaker separation SMS-WSJ (6-channel input and loss).
Val. set
Test set
Row
Systems
J
Loss
SDR (dB)
SDR (dB)
SI-SDR (dB)
PESQ
cSTOI
2a
UNSSOR
19
0
16.0
15.6
14.6
3.44
0.885
30
IRAS w/ non-causal 1472-tap filters (64 future taps)
Caas
2.4
2.4
0.2
1.89
0.549
Table 6: Supplementary averaged results of 2-speaker separation SMS-WSJ (3-channel input and loss).
Val. set
Test set
Row
Systems
J
Loss
SDR (dB)
SDR (dB3)
SI-SDR (dB3)
PESQ
cSTOI
Za
UNSSOR
19
AMCHSSss
15.7
15.4
14.4
3.20
0.874
30 iRAS w/ non-causal 1472-tap filters (64 future taps)
Cass
7.7
7.2
5.4
1.87
0.621
Table Supplementary averaged results of 2-speaker separation on SMS-WS (I-ch input and 6-ch loss).
Val. set
Test set
Row
Systems
J
Loss
SDR (dB)
SDR (dB)
SI-SDR (dB3)
PESQ
eSTOI
UNSSOR
11.9
3.27
0.832
2e
IRAS w/ non-causal 1536-tap filters (64 future taps)
Lass
1.7
1.7
0.0
1.90
0.561
Table Supplementary averaged results of 2-speaker separation on SMS- WSJ (1-ch input and 3-ch loss).
Val.
Test set
Row
Systems
Loss
SDR (dB)
SDR (dB)
SI-SDR (dB)
PESQ
eST01
la
UNSSOR
19
AMcusses
12.5
12.0
11.4
3.18
0.822
20
IRAS w/ non-causal 1536-tap fitters (64 future taps)
Luus
1.3
1.3
-0.3
1.87
0.549
To norm for gradient clipping is set to 1.0. The learning rate starts from 10-3 and is halved if the validation loss is not improved in two epochs. We terminate training once the learning rate is reduced to 6.25 x 10-5, The batch size is set four, with each segment being 4-second long. For each model, an Nvidia A100 40GB GPU is used for training, and the model converges in three to four days.
We sweep y in based on the set of {0.02, 0.04, 0.06, 0.3, 1.0]. The microphone weight Op in TOP and (2) is set to 1.0 in default for all microphones (i.e., no weighting). and, in the monaural input case (e.g.. in Table and 4 we sweep 01 at the reference microphone 1 based on the set of {1/2, 1/3, 1/ 1/5, 1/ 1 1 8, 1/10] to alleviate overfitting to microphone 1.
Alternative filter length for iRAS and UNSSOR
We also provide the results of iRAS that uses the same filter length (in seconds) as that in UNSSOR.
Given 8 kHz sampling rate, 32 ms window size, 8 ms hop size, and K = - +1+1 + (defined below
filter taps in FCP, we use M ((K- - 1) x 8 + (2)/1000 x 8000 filter taps for each h (c) in 12 and configure (c) filter the past M- 8/1000 x 8000 the current, and the future 64 8/ 8/1000 x 8000) samples. We filter the future 8 ms of samples, because, in the STFT case, the hop size is ms. We report the results in Table 5 g each respectively corresponding to the results in We observe that the performance is not as good as that of UNSSOR.
(
Table
4
In Fig- 6 we make further comparisons of using different filter taps between UNSSOR and iRAS, following the experimental setup in the previous paragraph. Fig. ([a) uses six-microphone input, Fig. b) uses monaural input, and both of them use the six-microphone LMCHIMS (or LIAAS) loss. For UNSSOR, in Fig. 66 () we set J = 0 and sweep K E {5, 9, 13, 17, 20, 25} and in F(g. we set / = 1 and sweep K 15.0 9, 13, 17, 21, 253. For iRAS, we configure the filter always filter the future 64 samples, and in Fig. aa we sweep the filter taps M E {128, 256. 384, 512, 768, 1024, 1280, 1472) and in Fig. b()) we sweep M {128, 256 384, 512, 768, 1024, 1280, 1536]. Notice that in Fig. 6 the filter taps M in iRAS and K in UNSSOR are vertically corresponding to each other. That iS. some swepted filter taps M are computed based on the swepted K (e.g. for M = 1024 and K 13, we have 1024- = (113 1) x 8 + 32) /1000 x 8000) so that they can result in the same filter length time. From the figures, we observe that the best filter length (in seconds) is different for UNSSÖR and iRAS, and the best performance of UNSSOR is higher than that of iRAS.
19

---

(==1+ 1+ 1+ (a)
in
UNSSOR
K( 1 1 1) in UNSSOR
5
13
17
20
25
5
13
17
25
16
135
14.
104
12-
SDR (dB)
SDR (dB)
2-
128
256
364
512
768
1024 1280 1472
128
256
384
512
1024 1280 1536
M in IRAS (filter future 64 samples)
M in IRAS (filter future 64 samples)
Figure 6: Averaged SDR (dB) results of UNSSOR and iRAS using various filter taps in cases of using (a) 6-channel input and loss; and (b) 1-channel input and 6-channel loss on SMS-WSJ. Vertically-correspondd M and K mean that the filter lengths in time are the same. Best viewed in color.
We point out that using very long filter (i.e., a large M) for time-domain Wiener filtering would prevent separation, since, in that case, the filter would have a large degrees of freedom to filter {c) to fit y/ very well and obtain small LIRAS (see 13) ) even though ²(c) is not good separation result. From Fig. G(a) a). it can be observed that a very large M (e.g.. 1472) does not yield good separation; and in addition, very small M (e.g., 128) is also not good, the linear filter could be just too short to fit the mixture Up well. Similar trend can also be observed in the results of UNSSOR. For example, in Fig. [[), setting K to and 25 produces worse performance than to 20. We can see that the filter length is an important paper-parraeter to tune.
We emphasize that, to compute the closed-form solution of time-domain Wiener filtering (see (3) we need to invert a big M x M matrix for each mixture, while in FCP (see (6). we only need to invert a much smaller K K matrix for each of the F' frequencies. Using FCP is clearly less computationalll-exxensivv
given that the time complexity of matrix inversion is typically O(n").
This also indicates that if the same amount of computation is required for linear filtering, FCP can use a much longer filter (in time) than time-domain Wiener filtering.
K UNSSOR's effectiveness when used with other DNN architectures
We observe that the effectiveness of the proposed system comes from both the strong modelling capabilities of TF-GridNet and the contributions of the UNSSOR mechanism itself. Without using UNSSOR deal with the ill-posed problem, the modelling capability of strong DNNs cannot be unleashed to separate speakers; and without using a strong DNN, the patterns in speech cannot be modelled well to realize good separation.
We expect UNSSOR to work with many DNN architectures, as long as the architecture is reasonably strong and can effectively deal with reverberation. To validate this, experiment UNSSOR with DNN arachitectures with lower modelling capability. We select two representative separation models from the literature, (CN-DenseUNe (74 and DPRNN (75].
We replace TF-GridNet with the ICN-DenseUNer described in 74,, and train the network using the same training configurations. TCN-DenseUNet (74) contains temporal convolution network sandwiched by a UNet with DenseNet blocks. It is a reasonably strong separation model, which is fully convolutional and shares many similarities with many contemporary DNN architectures (7//83]. According to [23]. it is worse than TF-GridNet in supervised separation tasks. We provide the unsupervised separation results in Table 9 which are obtained by using six-channel input and loss. We observe that UNSSOR works to some extent with TCN-DenscUNG and the results are not as good the ones in Table obtained by using Tr-GrieNet.
The DPRNN architecture 75 in our experiments has a window size of ms and hop size of ms. It has 6 layers. The number of bases is 256. The bottleneck dimension is 128. The number of hidden
units in each BLSTM each direction is 128. We apply ReLU as the encoder non-linearity and as the lin-liiyyy for embedding masking. The chunk size is set to 64 and the chunk overlap is 50%. To leverage spatial information for model training, we follow the strategy proposed in [44 (see its Fig, 2 to get the idca), where spectral embeddings are learned together with spatial embeddings and
20

---

Table 9: Averaged results of 2-speaker separation on SMS-WSJ (6-channel input and loss), obtained by using Uet 74] architecture.
Val. set
Test set
Row
Systems
r
J
Loss
SDR (dB)
SDR (dB)
SDR (dB)
PESQ
eSTOI
Mixture
0.1
0.1
0.0
1.87
0.603
UNSSOR
0
Cac-asss
11.1
10.7
9.7
2.92
0.770
2b
UNSSOR + Corr. based freq. align.
19 0
Euc isss
11.1
10.7
9.7
2.92
0.770
UNSSOR + Orucle frey. align.
19
0
LMc-sss
11.2
10.8
9.8
2.93
0.773
4a
PIT (supervised)
14.6
13.9
13.4
3.58
0.878
Table 10: Averaged results of 2-speaker separation on SMS-WSJ (6-channel input and loss), obtained by using DPRNN (7] architecture.
Val. set
Test set
Row
Systems
r J
Loss
SDR
SDR (dB)
SD (dB)
PESQ
eSTOI
Mixture
0.1
0.0
1.87
0.603
Za
UNSSOR
19 0
LMc-sMss
9.2
8.9
8.0
2.68
0.724
UNSSOR Corr. based freq. align.
0
Csic isMs
9.2
8.9
8.0
2.68
0.724
2c
UNSSOR Oracle frey. align.
19 0
EMc-sMM
9.3
9.0
8.1
2.68
0.727
PIT (supervised)
12.3
11.7
11.3
3.00
0.820
DNN-estimateds masks are used to mask spectral embeddings. In the six-channel case, the spatial embedding dimension set to 360, following (84). Differently from the Fig. 2 of [84]. we do not use microphone-pair-we
Conv ID layers to obtain spatial embeddings. Instead, we obtain them by using Conv ID layer with (= 6) input channels and 360 output channels. We first use the DPRNN to obtain intermediate separation results in the time domain, and then apply STFT (with window size of 32 ms, a hop size of 8 ms, and the square-root of Hann window) to obtain Z(c) for each speaker c. The network is trained by only using the MC loss in 2 and not using the ISMS loss in (1O). Although the ISMS loss is not used, we only observe minor frequency permutation the SMS- WSJ dataset. All the other procedures are the same as the TF-GridNet based UNSSOR system. The results SMS-W WS,, obtained by using six- six-channel input and loss, are presented in Table [0] We observe that UNSSOR also works with DPRNN to some degrees, and the results are not as good the ones in Table 1
L Comparison with MixIT
This section compares the results of UNSSOR with MixIT 30, which also deals with unsupervised separation. We put this comparison only appendix, considering that MixIT may not be ideal baseline to UNSSOR. Notice that MixIT needs to be trained on synthetic mixtures of mixtures (MoM), which ideally would require the two mixtures used in creating each MoM to be recorded in the same room using the same device. Differently. UNSSOR is designed to be trained directly existing mixtures. In this case, the two models would be trained on different training examples, and this would make the comparison difficult. In the main body of this paper, we hence only consider methods that can be trained (or performed) directly on existing mixtures (such as IVA, spatial clustering and iRAS) as baselines.
To report the results of MixIT, we create a particular scenario (ideal for MixIT), where, for each existing SMS-WSS mixture (y/ xx11) ) + .x( + with denoting noise), we randomly add two extra speakers in the same simulated room and use the same array placed at the same location so that we can have two 2-speaker mixtures (i.e., an existing SMS- WSJ mixture 1: y (1 (1) + xx(1(2) + nl)) + (2) (2) + n2() where the superscript (1) denotes mixture 1 and (2) denotes mixture 2) to create an MoM for training MixIT for 4-speaker separation.
and newly-simulate mixture (2) (2))
Similarly to n (1), n (2( is sampled such that the SNR between (²() + (²)) (2) and n2) is in the range of [20, 30] dB. The DNN architecture and training configurations for MixIT are the same as that in UNSSOR and PIT. The loss function is defined similarly to (5) on the real, imaginary and magnitude of the reconstructed mixtures weighted by the summation of mixture magnitudes. Similarly to our proposed technique, we can feed 1-, 3- or 6-channel mixtures to MixIT. At run time, the trained MixIT model used to separate the existing two-speaker mixtures in SMS-W WSJ to four outputs, and the two outputs with the highest energy are selected for evaluation.
21

---

Table 11: Averaged results of 2-speaker separation on SMS-WSJ, obtained by using MixIT 30
Val. set
sest
Row
Systems
SDR (dB)
SDR (dB)
SI-SDR (dB)
PESQ
cST01
0a
Mixture
0.1
0.1
0.0
1.87
0.603
5a
One-channel MixIT
6.7
6.3
2.21
0.691
5b
Threecharmne MixiT
0.1
0.1
0.0
1.91
0.608
channel MixIT
This way, the evaluation scores can be pretty much directly compared with the ones obtained by UNSSOR.
The results are shown in Table 11 They are not good as the ones reported in Table |
2 and
For unknown reasons, three-channel MixIT fits the loss well but fails at separating speakers.
22


Reviews:
[{"review_id": "JvvL16XSfp", "reviewer": "Reviewer_2uqn", "summary": "The focus of this paper is unsupervised speech separation by exploiting training mixtures with more microphones than speakers (over-determined). The proposed neural separator is using the input mixture to constrain the estimated images of each speaker. It is proposed to also train the system for under-determined scenarios, e.g., for single-channel speech separation, by separating a single-channel signal, while using multichannel loss function.", "questions": "Only measurement/modeling noise is included in the model, and background noise or discrete noise sources would complicate things, e.g., since the same RTF could not be applied to the noise component. Is there any initial data on robustness in presence of noise?\n\nll. 119\nIt is claimed that the relative room impulse response in (2) is typically short. However, it should be noted that this is not necessarily true, even for the microphone array and reverberation times used in the experiments (20 cm diameter, up to 0.5s RT60). A useful datapoint is paper by [Talmon, 2009], Fig. 7 in particular. I think this it would be helpful for many readers at NeurIPS to mention this (or a similar paper) and point out that this assumption does not always hold, and has been investigated in the literature.\n\n[Talmon, 2009] Talmot, Cohen, Gannot, Relative Transfer Function Identification Using Convolutive Transfer Function Approximation, IEEE Tr. ASLP, 2009.\n\nll. 128\n- Typo in “comptue”\n\nSection 4.2\nMixture consistency has already been proposed in [Wisdom, 2019]. This paper should be clearly mentioned. Also, consider renaming to multichannel mixture consistency, since you’re using P channels and RTFs.\n\n[Wisdom, 2019] Wisdom et al., Differentiable Consistency Constraints for Improved Deep Speech Enhancement, 2019.\n\nSection 4.3\nFormulation is (6) is basically multichannel linear prediction with using estimated sources Z instead of the original mixtures Y. Relevant works solving a problem analogous to (6) should be at least briefly mentioned, such as earlier work in [Yoshioka, 2010] and many later works.\n\n[Yoshioka, 2010] Yoshioka et al., Blind Separation and Dereverberation of Speech Mixtures by Joint Optimization, IEEE Tr. ASLP 2011.", "limitations": "The main limitation is that the model does not take into account background or discrete noise, and is only demonstrated to work in presence of uncorrelated (measurement) noise. This severely limits applicability to using real-world recordings to train a separation model.\nIt would be very useful to include results on robustness in presence of higher levels of noise. However, this is not essential and it may be out of scope of this paper.\nAnother limitation for real-world use is the assumption of a static scenario (fixed relative IRs), which would not hold in real recordings.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of using multichannel mixtures for unsupervised training is novel and reasonably effective compared to the supervised and unsupervised baselines.\nThe experimental evaluation is sufficient and demonstrates the effectiveness and includes sufficient data to support some of the design choices.\nThe paper is well written.", "weaknesses": "It would be useful for the readers less familiar with speech processing to include references to a few papers very relevant to relative IR estimation, mixture consistency and linear prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> It would be useful for the readers less familiar with speech processing to include references to a few papers very relevant to relative IR estimation, mixture consistency and linear prediction.\n\n Will include, especially [Talmon, 2019].\n\n> Only measurement/modeling noise is included in the model, and background noise or discrete noise sources would complicate things, e.g., since the same RTF could not be applied to the noise component. Is there any initial data on robustness in presence of noise?\n\nYes, we need to consider background noises. We haven't had comprehensive results on this yet, and plan to address this in a follow-up study.\n\nWe could use a number of garbage sources to absorb directional sources.\nThat is, we can train UNSSOR to separate the mixture to $C+N$ sources (where $C$ is the hypothesized number of speakers and $N$ the hypothesized number of directional noise sources) so that we can have a separate RTF for each source.\nWe envision that our method could be effective at separating a large number of directional sources (including both speech and noise sources) if there is a sufficient number of microphones to afford over-determined conditions.\nIn this case, we can assume $\\varepsilon$ to be a measurement/modelling noise, as the speech and noise signals are all modelled by the $N+C$ sources.\n\n> ll. 119 \n> It is claimed that the relative room impulse response in (2) is typically short. However, it should be noted that this is not necessarily true, even for the microphone array and reverberation times used in the experiments (20 cm diameter, up to 0.5s RT60). A useful datapoint is paper by [Talmon, 2009], Fig. 7 in particular. I think this it would be helpful for many readers at NeurIPS to mention this (or a similar paper) and point out that this assumption does not always hold, and has been investigated in the literature.\n\n> [Talmon, 2009] Talmot et al., Relative Transfer Function Identification Using Convolutive Transfer Function Approximation, IEEE TASLP, 2009.\n\nThanks for pointing this out. We realize that our sentence \"Note that $\\mathbf{g}_p(c,f)$ is very short (i.e., $E$ is small)\" may be misleading. \n\nJust to clarify. We intended to say that $E$ is small. In our paper, given that the STFT hop size is $8$ ms, $E=I+1+J$ equals $20$ in in Table 1-2, and equals $21$ in Table 3-4. We don't mean that the RIR in the time domain is very short.\n\nThanks for sharing the paper by Talmon et al. Based on our understanding of the paper, its Fig. 7(a) suggests that, in low-noise cases, when the microphone distance is larger, CTF, which can use multiple taps, can better estimate the RTF than MTF, which is restricted to only use one tap. In [Talmon, 2009], the number of the filter taps of CTF is set to $1/8$ of the T60. So given a T60 of $0.5$ s, the number of filter taps is roughly $0.5/8/0.016=3.9$, where $0.016$ is the STFT hop size in seconds. In other words, this setup echos our claims that $E$ is small.\n\nWe will improve the sentence and cite the referred paper.\n\n> ll. 128 Typo in “comptue”\n\nWill correct!\n\n> Section 4.2 Mixture consistency has already been proposed in [Wisdom, 2019]. This paper should be clearly mentioned.\nAlso, consider renaming to multichannel mixture consistency, since you’re using P channels and RTFs.\n> [Wisdom, 2019] Wisdom et al., Differentiable Consistency Constraints for Improved Deep Speech Enhancement, 2019.\n\nWe now realize that it is a bad idea to use the same name, and we will change the name of our loss to \"mixture-constraint\" loss to differentiate it from the \"mixture consistency\" term proposed in [Wisdom, 2019].\n\nWe will highlight their differences in the paper.\nAs you mentioned, our loss differs in the number of channels used and, in addition, we filter the DNN estimates before loss computation.\nAnother major difference is that [Wisdom, 2019] constrains the estimated sources to strictly add up to the mixture (see their Eq. (7) and (9)), while our method only ``encourages'' the filtered source estimate to add up to the mixture.\n\n> Section 4.3 Formulation is (6) is basically multichannel linear prediction with using estimated sources Z instead of the original mixtures Y. Relevant works solving a problem analogous to (6) should be at least briefly mentioned, such as earlier work in [Yoshioka, 2010] and many later works.\n\n> [Yoshioka, 2010] Yoshioka et al., Blind Separation and Dereverb. of Speech Mixtures by Joint Optimization, TASLP 2011.\n\nWill mention!\n\nAlthough (6) appears similar to conventional multichannel linear prediction (MCLP), we would like to emphasize that it has very different physical meanings. We consider that (6) does \"forward filtering\", where source estimates are filtered to approximate mixtures, while MCLP does \"inverse filtering\", where mixtures are filtered to approximate sources.\nThis modification leads to non-trivial changes of the physical meanings of the computed filters (see also discussions in Section V.C of [A1] listed below).\n\n[A1] Z.-Q. Wang et al., Convolutive Prediction for Monaural Speech Dereverb. and Noisy-Reverb. Speaker Separation, in TASLP, 2021.\n\n> The main limitation is that the model does not take into account background or discrete noise, and is only demonstrated to work in presence of uncorrelated (measurement) noise. This severely limits applicability to using real-world recordings to train a separation model. It would be very useful to include results on robustness in presence of higher levels of noise. However, this is not essential and it may be out of scope of this paper. \n\nSee our earlier response to this point. We also think this may be out of scope of this paper. \n\n> Another limitation for real-world use is the assumption of a static scenario (fixed relative IRs), which would not hold in real recordings.\n\nThis is indeed a tricky issue, which is a common problem also existed in many other algorithms. We could model time-varying filters in some way, and we will investigate this in a future work."}, {"review_id": "LUoMncu3Xo", "reviewer": "Reviewer_jddy", "summary": "In this paper the authors tackle speech source separation, in which multiple fixed speech sources $X(c)$ are recorded by an array of p microphones, resulting in p observable mixtures $Y_p$. The authors propose an STFT domain source separation algorithm called UNSSOR, leveraging a complex-valued deep neural network (TF-GridNet) to estimate the individual sources at a reference microphone (e.g., mic 1). Typically the problem is solved in a supervised fashion using PIT (permutation invariant training) where the clean sources must be known, but the authors propose an unsupervised method, where only the mixtures $Y_p$ are required at training time. The key idea is to relax the under-determined problem, by leveraging physical constraints, i.e. the sources combining at a microphone p are the same sources at the reference microphone convolved with a relative room impulse response (relative RIR). As such the authors estimate virtual sources $Z(c)$ and relative RIRs at each microphone for each source $g_p(c)$. The relative RIRs are computed in closed form using FCP (a least squares optimization problem), leveraging only the mixtures $Y_p$ and the estimates $Z(c)$. Multiplying $g_p(c)$ with $Z(c)$ the authors obtain the FCP-estimated source images, which are summed to obtain mixtures estimates $\\hat Y_{p}$ at each microphone, obtaining a mixture consistency loss $L_{MC}$ for training the neural network. The authors notice that solutions obtained with the model trained on $L_{MC}$ only exhibit the frequency permutation problem, where estimates on certain spectral bands are swapped between estimates. To address this problem, the authors regularize the variance of the magnitude of the FCP-estimated source images over frequency bands during training time, with a novel intra-source magnitude scattering loss $L_{ISMS}$. Combining $L_{MC}$ and $L_{ISMS}$ results in improved separation metrics. Finally, the authors try the algorithm in the monoaural unsupervised source separation setting, where they provide only one mixture at training time, but optimize with multiple microphones in the loss. ", "questions": "I will add suggestions for the improvement of the paper and ask relevant questions.\n\n- Line 65: The authors should use a term like `metric` instead of `measure`, given the precise meaning of such a term in measure-theory. I know they can be synonyms in an applied field such as audio processing but it is better to be more precise.\n- Line 120: `Hermitian transpose` instead of `Hermittan`\n- Line 146: the term mixture consistency is defined here: https://arxiv.org/pdf/1811.08521.pdf, I think it's ok to use the same term but could create a little bit of confusion.\n- Line 161: Can the authors explain briefly in the text, for better understanding, the rationale of matching the FCP-estimated images with the mixture in Eq. (6)? \n- Line 167: Regarding the $\\xi$ hyperparameter, I don't understand if it multiplies only the max or all the expression\n- Line 179: I found the causal analysis well developed, but I still do not well understand why if a source sound c reaches the reference microphone earlier it should be processed with future values (non-causal filtering). \n- Line 193: Given that $Z(c)$ is learned, why it is interpreted like a virtual microphone estimate and not the real dry signal at speaker $c$?\n- Line 232: In the monoaural speech source separation setting, I don't understand what you match on the different microphones in the mixture consistency loss, when using only one input $Y_p=1$? Do you match the only available $Y_p=1$ at all \"virtual\" microphones?", "limitations": "The authors address the limitations of their work in Appendix H, namely that they assume to know the number of sources, they assume they are directional point sources at fixed positions. I do not believe these limitations are much problematic as the main limitation is the unavailability of data required for scaling up the algorithm with respect to the supervised baseline (if it scales).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The idea of reducing the number of equations using physical constraints is natural, leading to a well-structured unsupervised multi-mixture speech source separation algorithm. Computing the RIRs with least squares also is very useful, given that in such a way the mixture consistency loss depends only on the observed mixtures and the estimated virtual sources. The authors present such an idea in a very clear and formal manner, with relevant links to the existing literature. I believe that this approach is way more founded theoretically than the MixIt approach, leading to a better methodological line of research on unsupervised neural source separation.\n- The intra-source magnitude scattering regularizer ($L_{ISMS}$) seems very interesting because such a loss can be incorporated not only in the presented multi-microphone speech scenario but in every source separation task working on STFT spectrograms (e.g, music, universal). Improving the results for more than 4dB gives strong empirical results to the effectiveness of such a technique.\n- The method achieves good empirical results with respect to a plethora of metrics (SDR, SI-SDR, PESQ, eSTOI) on SMS-WSJ compared to other unsupervised multi-mic algorithms (Spatial clustering, IVA, RAS) and fares well with respect to a supervised PIT baseline (using the same neural architecture).", "weaknesses": "- UNSSOR, while improving the results in the monoaural speech separation setting with respect to iRAS (improves 1.4 dB on the SMS-WSJ test set both with 3 and 6 channels loss), still faces a large gap with respect to the supervised Monaural PIT baseline (~ 4 dB). This is expected since not relying on supervision can negatively impact the separation results. The authors should have included at least a comparison with some other popular unsupervised source separation baseline such as MixIt, in order to prove the superiority of UNSSOR in the unsupervised source separation arena.\n- As with other unsupervised neural source separators such as MixIt, it is not really clear why one should go unsupervised if the supervised metrics are already better on the same dataset. These types of studies (including MixIt or its regularized versions such as https://arxiv.org/abs/2106.00847), should experimentally showcase that performing unsupervised source separation can benefit over supervised source separation as the dataset size increase. For example in https://arxiv.org/abs/2106.00847, authors cannot beat a supervised baseline trained on FUSS, using an order of magnitude more data (Audioset or YFCC100m) with an unsupervised method (regularized MixIt). Providing a scaling law should be of uttermost importance in papers such as the presented one and in future papers, if not we can continue inventing unsupervised methods that will never surpass supervised or weakly supervised methods (using labels, learning Bayesian priors).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The authors should have included at least a comparison with some other popular unsupervised source separation baseline such as MixIt\n\nSee our response to all the reviewers.\n\n> As with other unsupervised neural source separators such as MixIt, it is not really clear why one should go unsupervised if the supervised metrics are already better on the same dataset. These types of studies (including MixIt or regularized MixIt https://arxiv.org/abs/2106.00847 [Wisdom'21]) should experimentally show that performing unsupervised separation can benefit over supervised separation as the dataset size increase. For example in [Wisdom'21], authors cannot beat a supervised baseline trained on FUSS, using an order of magnitude more data with an unsupervised method. \n> Providing a scaling law should be of uttermost importance in papers such as the presented one and in future papers, if not we can continue inventing unsupervised methods that will never surpass supervised or weakly supervised methods.\n\nInsightful comment! We will pay more attention to the scaling law in follow-up studies.\n\nIn speech separation, a major motivation of using unsupervised separation is that the models can be trained directly on real mixtures (where the clean speech signals are not available) and hence could have better generalizability on real data than supervised models, which need to be trained on simulated data (often mismatched with real-recorded test data).\nAs is also pointed out by reviewer ``XWBZ'', supervised models such as PIT have limited success so far on real-recorded multi-speaker datasets such as AMI, CHiME-6 and AliMeeting, even if a lot of data can be simulated to train PIT.\nA possible initial step towards solving this problem could be using an algorithm like UNSSOR which can be trained directly on real mixture and avoid using unrealistic synthetic mixtures for training.\n\nOn the other hand, unsupervised separation and supervised separation may not be mutually exclusive, and they could be combined. We could, for example, adapt a supervised model to new domains via unsupervised mechanisms such as UNSSOR, or fine-tune unsupervised models such as UNSSOR via supervised mechanisms like PIT in a target domain, where some high-quality labelled data is available.\n\n> Line 65: Ue metric instead of measure\n> Line 120: Hermitian transpose instead of Hermittan\n\nWill change.\n\n> Line 146: the term mixture consistency is defined in [Wisdom 2019], I think it ok to use the same term but could create confusion.\n\nWill rename the loss to \"mixture-constraint\" loss to avoid any confusion.\n\n> Line 161: Can the authors explain briefly the rationale of matching the FCP-estimated images with the mixture in Eq. (6)?\n\nWill explain.\n\nIf $Y_p$ only contains $X_p(c)$, (6) can estimate the relative RIR relating $\\hat{Z}(c)$ to $X_p(c)$.\n\nIf $Y_p$ contain other sources besides $X_p(c)$, (6) could still estimate the relative RIR following the derivations in Appendix C.\n\n> Line 167: Regarding the $\\xi$, I don't understand if it multiplies only the max or all the expression\n\nOnly to the max expression. We will make the equation clearer.\n\n> Line 179: I found the causal analysis well developed, but I still do not well understand why if a source sound c reaches the reference mic earlier it should be processed with future values (non-causal filtering).\n\nWe give an example below.\n\nIn Fig. 1(a) (see attached .pdf file), suppose that the blue signal is the DNN estimate for speaker $c$, and the orange signal is speaker $c$'s image at another microphone, which is a delayed version (i.e., reaching the microphone later). To filter the blue signal to approximate the oracle signal, we only need a causal filter.\n\nReversely, suppose that the orange signal is the DNN estimate for speaker $c$, and the blue signal is speaker $c$'s image at another microphone, which is an advanced version (i.e., reaching the microphone earlier). To filter the orange signal to approximate the blue signal, we need a non-causal filter.\n\nWe will add this example to the Appendix of the paper.\n\n> Line 193: Given that $Z(c)$ is learned, why it is interpreted like a virtual microphone estimate and not the real dry signal at speaker $c$?\n\nIn Eq. (9), $\\hat{Z}(c)$ is constrained such that it can be filtered by a causal filter $\\hat{\\mathbf{g}}_p(c)$ to approximate $X_p(c)$, and it is not explicitly constrained to be a dry signal. Since there could be an infinite number of $\\hat{Z}(c)$ and $\\hat{\\mathbf{g}}_p(c)$ whose convolution results would well approximate $X_p(c)$, $\\hat{Z}(c)$ is likely not the dry source signal.\n\nSee Fig. 1(b)  (see attached .pdf file) for an example, where each virtual microphone captures the direct-path signal of a target speaker earlier than any other microphones so that we can use causal FCP filters.\n\n> Line 232: In the monoaural speech source separation setting, I don't understand what you match on the different microphones in the mixture consistency loss, when using only one input $Y_{p=1}$? Do you match the only available $Y_{p=1}$ at all \"virtual\" microphones?\n\nWe now think that, in the monaural case, $\\hat{Z}(c)$ would all be aligned to the speakers' images at the reference microphone $1$, since the DNN only has monaural input and in this case the DNN is not likely to align its outputs to a virtual microphone different from microphone $1$.\n\nTo help understanding, we give an example in Fig. 1(c) (see attached .pdf file), where the reference microphone captures speaker $2$'s direct-path signal later than all the other microphones. In this case, we need to use non-causal FCP filters when filtering $\\hat{Z}(c)$ (which is estimated based on the monaural signal at the reference microphone) to approximate speaker $2$'s images captured at the other microphones.\n\n> ... the main limitation is the unavailability of data required for scaling up the algorithm with respect to the supervised baseline (if it scales).\n\nSee our responses to your earlier comment on this.\n"}, {"review_id": "XhZjHrXRxv", "reviewer": "Reviewer_XDQW", "summary": "This paper proposes an unsupervised method for training neural networks to separate speech from multi-microphone recordings. The idea is to separate a reference microphone into separate sources, then use each microphone as a mixture signal that must match filtered versions of the reference microphone separated sources. The linear filters are computed in subbands, which results in a well-known frequency permutation problem, for which an additional \"intra-source magnitude\" loss term is proposed to ameliorate. Though the method requires \"overdetermined\" mixtures (i.e. more mics than sources), the method can also be used for underdetermined cases, e.g. 2 speakers with 1 mic, where a single mic is used as input, and multiple mics are used in the loss function. For 3-mic and 6-mic mixtures, the proposed method is shown to achieve better performance in terms of SDR, SI-SDR, PESQ, and eSTOI versus a baseline (the RAS algorithm) that works on a similar principle on the SMS-WSJ dataset. The method does not outperform a supervised PIT method, which is expected.", "questions": "Q1) How much tuning of the number of filter taps was done? This seems like it could have a big effect on the performance of the model. Also, does tuning the number of taps suggest some knowledge about the unsupervised data? I suppose a practitioner could use a blind T60 model to estimate the distribution of T60 across an unsupervised dataset, which could give some insight into optimal filter taps. Besides T60 (i.e. duration of expected RIRs to model), are there other issues that affect the optimal choice of filter taps?\n\nQ2) How certain is it that frequency permutation is the primary cause of the drop in performance going from rows 1a->1b and 2a->2b in Tables 1 and 2? The appendix provides an illustrative example, but none of the objective metrics are directly measuring frequency permutation. It seems that an objective metric could be formulated that directly measures the degree of frequency permutation, but perhaps such a metric is not necessary if spot checks of the predictions indicated that frequency permutation was present?", "limitations": "Limitations are discussed well.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1) The method is intuitive and is explained well, and the paper provides thorough description of the method and chosen hyperparamters.\n\nS2) The proposed method can perform unsupervised training on single mixtures, unlike MixIT which requires combining at least two mixtures into a training examples.\n\nS3) The authors clearly describe the differences of the proposed UNSSOR method with the RAS method, particularly why UNSSOR can successfully train unsupervised, while RAS cannot.", "weaknesses": "W1) The dataset used, SMS-WSJ, consists entirely of synthetic mixtures that use synthetic simulated RIRs. Thus, there is some concern that the method may not generalize to real-world acoustic scenarios. Some evaluation on real data would further improve the paper. Also, one of the main advantages of unsupervised algorithms is to be able to adapt separation models to unsupervised real-world data, so that models can work better on those data domains. Of course, evaluation on real domains cannot be evaluated by intrusive metrics, as used in this paper, and require subjective human listening tests. I am glad the authors are considering this as a future direction, and I encourage them towards that goal.\n\nW2) As acknowledged by the authors, the method assumes stationary sources, which is potentially quite limiting when considering real scenarios where speakers may be moving. Even if speakers are seated, head movement can still produce acoustic effects. There are likely interesting extensions of the method to handle nonstationary spatial scenarios, such as allowing for slowly-varying linear filters. \n\nW3) No audio demos are provided. Providing such demos would make it a lot easier to readers to evaluate the quality of the predictions and improve understanding of the method.\n\nW4) It would be interesting to see how a single-channel unsupervised method like MixIT compares to the multi-microphone method. Also, since this paper was submitted, a multichannel version of MixIT seems to have been proposed (https://arxiv.org/abs/2305.11151), which seems to operate differently (network with multichannel input and multichannel output, and MixIT applied directly to multichannel outputs using multi-channel mixtures-of-mixtures). It would be interesting to compare both single-channel and multi-channel MixIT to the proposed approach; these MixIT methods will likely be mismatched to test time since they train on mixtures-of-mixtures, but it would be interesting to compare.\n\nMinor comments and typos\n\na) \"boot-start\" -> \"bootstrap\" ?\n\nb) \"comptue\" -> \"compute\"\n\nc) Would be good to mention what \\mathcal{F} is in description of equation (4)\n\nd) \"doing this would complicates\" -> \"doing this would complicate\"\n\ne) \"assumed time-variant.\" -> \"assumed time-variance.\"\n\nf) Maybe bold the best numbers in each column for Tables 1 and 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1) The dataset used, SMS-WSJ, consists entirely of synthetic mixtures that use synthetic simulated RIRs. Thus, there is some concern that the method may not generalize to real-world acoustic scenarios. Some evaluation on real data would further improve the paper.\nAlso, one of the main advantages of unsupervised algorithms is to be able to adapt separation models to unsupervised real-world data, so that models can work better on those data domains. Of course, evaluation on real domains cannot be evaluated by intrusive metrics, as used in this paper, and require subjective human listening tests. I am glad the authors are considering this as a future direction, and I encourage them towards that goal.\n\nGreat comment!\n\nWe are currently working towards that goal, and will share our findings.\n\n> W2) As acknowledged by the authors, the method assumes stationary sources, which is potentially quite limiting when considering real scenarios where speakers may be moving. Even if speakers are seated, head movement can still produce acoustic effects. There are likely interesting extensions of the method to handle nonstationary spatial scenarios, such as allowing for slowly-varying linear filters.\n\nYes, we will investigate the moving-source case, which is a common problem also in spatial processing.\n\n> W3) No audio demos are provided. Providing such demos would make it a lot easier to readers to evaluate the quality of the predictions and improve understanding of the method.\n\nAn audio demo is available at https://anonymauth.github.io/\n\nWe will add this link to the paper.\n\n> W4) It would be interesting to see how a single-channel unsupervised method like MixIT compares to the multi-microphone method. Also, since this paper was submitted, a multichannel version of MixIT seems to have been proposed (\\url{https://arxiv.org/abs/2305.11151}), which seems to operate differently (network with multichannel input and multichannel output, and MixIT applied directly to multichannel outputs using multi-channel mixtures-of-mixtures). It would be interesting to compare both single-channel and multi-channel MixIT to the proposed approach; these MixIT methods will likely be mismatched to test time since they train on mixtures-of-mixtures, but it would be interesting to compare.\n\nSee our response to all the reviewers.\n\nThanks for pointing us the paper on multi-channel PIT. We plan to compare with multi-channel MixIT in a future study, as it was online after the submission of this paper.\n\n> a) \"boot-start\" -> \"bootstrap\" ?\n\n> b) \"comptue\" -> \"compute\"\n\n> c) Would be good to mention what $\\mathcal{F}$ is in description of equation (4)\n\n> d) \"doing this would complicates\" -> \"doing this would complicate\"\n\n> e) \"assumed time-variant.\" -> \"assumed time-variance.\"\n\n> f) Maybe bold the best numbers in each column for Tables 1 and 2?\n\nWill change following the comments.\n\n> Q1) How much tuning of the number of filter taps was done?\n> This seems like it could have a big effect on the performance of the model.\n> Also, does tuning the number of taps suggest some knowledge about the unsupervised data?\n> I suppose a practitioner could use a blind T60 model to estimate the distribution of T60 across an unsupervised dataset, which could give some insight into optimal filter taps.\n> Besides T60 (i.e. duration of expected RIRs to model), are there other issues that affect the optimal choice of filter taps?\n\nGreat question!\n\nThe filter taps in Fig. 4 of the Appendix show the filter taps we tuned. It indeed has a big effect.\n\nOur experience is that the filter tap can not be set too long or too short. If it is set too long, the model would optimize the loss well but not separate speakers (i.e., overfit); and if it is set too short, the model would not fit the loss well (i.e., underfit).\n\nWe see what you mean by using a blind T60 model to estimate the filter length for each training example, but the FCP filters in the study are the relative RIRs among speaker images at closely-placed microphones rather than the RIRs between sound source and its far-field images. We currently do not have a good way to determine the optimal choice of filter taps.\n\n> Q2) How certain is it that frequency permutation is the primary cause of the drop in performance going from rows 1a->1b and 2a->2b in Tables 1 and 2? \n> The appendix provides an illustrative example, but none of the objective metrics are directly measuring frequency permutation.\n> It seems that an objective metric could be formulated that directly measures the degree of frequency permutation, but perhaps such a metric is not necessary if spot checks of the predictions indicated that frequency permutation was present?\n\nWe are confused by the first question. In Table 1 and 2, going from 1a to 1b and going from 2a to 2b both do not have performance drop.\n\nA direct metric of frequency permutation may not be necessary, as you suggested. Our spot checks are that frequency permutation is always presented, and its presence also makes sense as FCP is performed independently in each frequency. From the results in row 1a and 1c of Table 1 and 2, we can see that using oracle frequency permutation produces large improvement. This largely indicates how severe the frequency permutation problem is."}, {"review_id": "m6nVEBiGke", "reviewer": "Reviewer_XWBZ", "summary": "The most popular method for training neural networks for speech separation is by artificially mixing sources, since neural network based training requires supervision. However, this kind of supervised training creates a mismatch since the mixtures seen at test time contain real overlapping speech and noise. This paper formalizes the problem of unsupervised speech separation, and posits that it becomes a well-posed problem for over-determined conditions (i.e., when the number of microphones is more than the number of speakers). The authors propose a method (inspired from signal processing) that uses the images at each microphone as supervision for the neural network training, and a new loss function to address the frequency permutation problem that often happens in blind source separation (BSS). They also propose a method to use these models for monoaural separation.", "questions": "1. How robust is UNSSOR to change in array configuration or the number of channels for the case of mono-aural separation?\n\n2. In Section 5.2, can you briefly explain what these metrics are, for the unfamiliar reader.\n", "limitations": "None, except those presented in Appendix H.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Overall, I believe this is an extremely strong paper with landmark results for unsupervised speech separation. As mentioned above, the majority of literature in the field of speech separation, since permutation-invariant training (PIT) was proposed, has only revolved around better neural network architectures. While the SI-SDR on the synthetic WSJ-Mix benchmark has consequently gone up, significantly less attention has been paid to real overlapping speech. The paradigm of continuous speech separation (CSS) tries to address some of the issues (such as sparse overlaps), but it is also trained with PIT on synthetic mixtures. The result is that there exists no public speech separation models that can perform well on multi-speaker data such as AMI, CHiME-6, AliMeeting, etc. I think UNSSOR is quite promising and should change the face of speech separation research.\n\nThe paper is strong on many levels, such as the following.\n\n1. The authors correctly identify that the problem with training separation networks has to do with a lack of accurate supervision, and propose that this supervision may come from estimating each speaker’s reverberant image at every microphone. Under the assumption of a linear-filter constraint between a speaker’s images at the microphones, the resulting linear system has a unique solution, as described in Section 3.\n\n2. The authors design loss functions which are inspired from signal processing formulation of the problem. This includes (i) mixture consistency loss for the filtered estimates, (ii) forward convolutive prediction (FCP) for relative RIR estimation, (iii) causal/non-causal stacking to handle time misalignment, and (iv) an intra-source magnitude scattering loss to address the frequency permutation problem. Overall, all these losses have clear motivations and design.\n\n3. The recently proposed TF-GridNet encoder is used as the backbone of the model. This architecture has been shown to obtain state-of-the-art results on PIT-based separation (even surpassing time-domain models), and it is interesting to see that it can be trained well with the unsupervised technique.\n\n4. The authors compare their method with extremely strong baselines, which makes their conclusions stronger. For example, they implemented a novel variant of the recently proposed “Reverberation as Supervision” (RAS) method, called iRAS, which can perform unsupervised separation. This modification could very well be a short paper in itself.\n\n5. The results on SMS-WSJ show that UNSSOR obtains results approaching supervised separation models. For example, for the 6-channel setting, the SDR on the test set is 15.6 dB (compared to 19.4 dB for PIT model). As a comparison, the next unsupervised method is IVA, which obtains 10.6 dB. Moreover, on reducing the number of channels from 6 to 3, UNSSOR’s performance degrades only marginally (to 15.4 dB), whereas PIT degrades to 16.8 dB.\n\n6. The authors even find a way to use models trained with multi-channel inputs for performing mono-aural separation.\n\n7. The code and models for UNSSOR will be released.\n\nA side effect of all of the above is that the paper is quite dense and may not be accessible to readers who do not have a source separation background. However, it is a case study in how neural methods can be designed by incorporating the knowledge from other domains (in this case, signal processing).\n", "weaknesses": "1. The main point of concern about the paper is that results are shown just for the SMS-WSJ dataset. Such an evaluation defeats the original motivation for unsupervised training, since SMS-WSJ contains synthetically mixed sources. Clearly, the PIT models outperform UNSSOR on this benchmark, which is expected. The real test of UNSSOR would have been an evaluation on real mixtures, such as AMI, CHiME-6, or AliMeeting, where PIT-based models fail completely. Nevertheless, since the paper proposes a completely novel paradigm of speech separation, I am willing to forgo this point, but at the cost of docking a point from the ratings.\n\n2. Can UNSSOR be trained with data containing different numbers of channels or different array geometries? It seems that the choice of the hyper-parameters $I$ and $J$ may be heavily dependent on the microphone configuration.\n\n3. The authors delegate the “Limitations” section to the appendix. I think it should be put in the main paper, since an important part of presenting our research is talking about its limitations. This would also be useful for other researchers to get ideas for building upon UNSSOR for their own work.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Overall, I believe this is an extremely strong paper with landmark results for unsupervised speech separation. As mentioned above, the majority of literature in the field of speech separation, since permutation-invariant training (PIT) was proposed, has only revolved around better neural network architectures. While the SI-SDR on the synthetic WSJ-Mix benchmark has consequently gone up, significantly less attention has been paid to real overlapping speech. The paradigm of continuous speech separation (CSS) tries to address some of the issues (such as sparse overlaps), but it is also trained with PIT on synthetic mixtures. The result is that there exists no public speech separation models that can perform well on multi-speaker data such as AMI, CHiME-6, AliMeeting, etc. I think UNSSOR is quite promising and should change the face of speech separation research.\n\n> The paper is strong on many levels, such as the following.\n\n> ...\n\n> A side effect of all of the above is that the paper is quite dense and may not be accessible to readers who do not have a source separation background. However, it is a case study in how neural methods can be designed by incorporating the knowledge from other domains (in this case, signal processing).\n\nWe feel so excited after reading the comments!\n\nThis is such a big encouragement for us to invest more on improving the proposed methods in follow-up studies.\n\n> The main point of concern about the paper is that results are shown just for the SMS-WSJ dataset. Such an evaluation defeats the original motivation for unsupervised training, since SMS-WSJ contains synthetically mixed sources. Clearly, the PIT models outperform UNSSOR on this benchmark, which is expected. The real test of UNSSOR would have been an evaluation on real mixtures, such as AMI, CHiME-6, or AliMeeting, where PIT-based models fail completely. Nevertheless, since the paper proposes a completely novel paradigm of speech separation, I am willing to forgo this point, but at the cost of docking a point from the ratings.\n\nInsightful point!\n\nWe made an effort to evaluate on AMI, CHiME-6, and AliMeeting datasets, which are real-recorded meeting-style data. However, to obtain strong performance, we need to deal with many other problems (such as sparse speaker overlap, varying number of speaker, unknown number of speakers etc.), which would make the paper much less focused. We hence leave the evaluation as a future work, and this paper focuses on showing the potential of UNSSOR, which will be the core technique we will build upon in our future work.\n\n> Can UNSSOR be trained with data containing different numbers of channels or different array geometries? It seems that the choice of the hyper-parameters and may be heavily dependent on the microphone configuration.\n\nGood point!\n\nWe think that UNSSOR can be trained with the mentioned data, and this could be a good future extension.\n\nWe could use DNNs that can handle variable number of input channels and different array geometries.\n\nThe loss can also be computed on training examples that have different numbers of channels, by configuring the filter taps in a smart way to deal with different array geometries (i.e., to cover a wide range of microphone distances).\n\n> The authors delegate the “Limitations” section to the appendix. I think it should be put in the main paper, since an important part of presenting our research is talking about its limitations. This would also be useful for other researchers to get ideas for building upon UNSSOR for their own work.\n\nWill change.\n\n> How robust is UNSSOR to change in array configuration or the number of channels for the case of mono-aural separation?\n\nAt run time, the trained model performs monaural separation. We think that it would be invariant to changes in array configurations or the number of channels.\n\nAt training time, we expect that UNSSOR can robustly deal with training examples with various array configurations or number of channels.\nNotice that the DNN is trained to only exploit monaural spectro-temporal patterns for separation. As long as there is a sufficient number of microphone mixtures to help pinpoint the solutions to speaker images, we expect the DNN to be trained well, although the shape of the loss surface afforded by different array configurations or numbers of channels could influence the training. This investigation could be a follow-up paper.\n\nWe expect that using more microphones for loss computation would lead to clearly better separation, but we don't observe this in our current evaluations (e.g., see Table 3 and 4). We will investigate this in a future study.\n\n> In Section 5.2, can you briefly explain what these metrics are, for the unfamiliar reader.\n\nThey are popular metrics in speech separation. SI-SDR and SDR measure the quality of predictions at the sample level, and PESQ and eSTOI are objective metrics of speech quality and intelligibility respectively.\n\nWe will add this description to the paper.\n"}, {"review_id": "gDBJIxz2FB", "reviewer": "Reviewer_ux7k", "summary": "This paper presents an innovative approach to unsupervised neural speech separation, leveraging the conditions where the number of microphones surpasses the number of speakers. The authors propose a method, named UNSSOR, which transforms an originally ill-posed problem - one that does not have a unique solution - into a well-posed problem - one with a unique solution - thereby facilitating the separation of speakers.\n\nKey contributions of the paper include:\n\n- The authors establish a linear-filter constraint between each speaker's reverberant images at each microphone pair, which converts the ill-posed problem into a well-posed one, thereby enhancing the separation of speakers.\n\n- The authors devise loss functions inspired by the blind deconvolution problem and propose a DNN-based approach to optimize these functions. The speaker images are determined via DNNs, while the linear filters are estimated using a sub-band linear prediction algorithm named FCP, based on the mixture and DNN estimates.\n\n- To address the frequency permutation issue that arises when using sub-band FCP, the authors propose a loss term that minimizes a measure known as intra-source magnitude scattering.\n\nThe authors claim that UNSSOR can be trained to perform under-determined separation, such as monaural unsupervised speech separation, based on over-determined training mixtures.", "questions": "My detailed questions are as described above.\n\n(1) Is the MC loss function in this paper an innovation from [1] and where is the innovation marked?\n\n(2) Is the effectiveness of the UNSSOR method due to the powerful performance of the separation model TF-GridNet, or is it due to the contribution of the UNSSOR method itself?\n\n(3) Why not compare MixIT method?\n\n(4) Why not use the same STFT settings, especially compared to the DNN model iRAS.", "limitations": "The authors discuss the limitation of the proposed method.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors propose a novel unsupervised neural speech separation method. The proposed method, UNSSOR, utilizes the multimicrophone over-determined condition to solve the unsupervised learning problem of speech separation. This is a completely new perspective compared to previous unsupervised speech separation methods. The authors have designed innovative loss functions that guide the unsupervised separation model about the desired sound objects and encourage the separation of speakers. This approach is original and shows a creative combination of existing ideas.", "weaknesses": "However, there are several limitations to this paper that undermine its true potential. Please see below for concerns and questions for the authors.\n\n1. The article mentions that MixIT may have two mixtures that are not in the same scene under reverberant conditions, which in turn may result in providing spatial a priori information, or providing a priori information due to differences in the configuration and number of microphones. However, I feel that this statement is a bit too hypothetical. I think these problems are more a result of how the dataset is constructed and not a deficiency of the MixIT method itself. More specifically, these problems should be solved by constructing a room-specific dataset. The SMS-WSJ dataset mentioned in the paper can control the room's parameters, the arrangement of microphones, and the number of microphones. By finely designing and controlling these parameters, the resulting problems can be effectively circumvented, allowing the MixIT method to be evaluated in a more consistent environment. Therefore, I believe the problems mentioned in the paper do not serve as a weakness of the MixIT method.\n\n2. In Section 4.2, I noticed that the authors proposed the MC loss function. However, as far as I understand, the MC loss function is not presented in this paper and has been described in detail in reference [1]. Cite [1] as a reference and clarify whether the MC loss function in the paper is an innovation from [1] and where exactly the innovation is.\n\n3. The authors chose TF-GridNet as the separation DNN structure for the UNSSOR method.TF-GridNet has shown excellent performance in supervised separation tasks. However, the question I would like to raise is: Is the UNSSOR method's effectiveness due to the separation model TF-GridNet's strong performance, or is it due to the contribution of the UNSSOR method itself? I believe that to elaborate the capability of the UNSSOR method better; it is necessary to try to use different separation structures and compare their performance on each structure. If the UNSSOR method performs well on other separation structures, then we can be more confident that the UNSSOR method itself is highly robust and general. Such an analysis is necessary to assess and understand the actual value of the UNSSOR method.\n\n4. I note that the approach in the paper combines traditional methods (Spatial clustering and IVA) with the DNN-based method iRAS. In addition, UNSSOR, a DNN-based scheme, is compared to include only one DNN method, which may not be comprehensive, in my opinion. Of particular note is that MixIT is a popular and widely used method for unsupervised speech separation, as shown in references [2-5]. I was puzzled while reading the paper as to why the authors did not include MixIT in the comparison.\n\n5. The authors set different STFT window sizes and hop lengths in the comparison, which may raise questions about the fairness of the results. For the separation model of DNN, different settings of STFT window sizes and hop lengths can significantly affect the model's performance [6].\n\n### References\n[1]. Wisdom S, Hershey J R, Wilson K, et al. Differentiable consistency constraints for improved deep speech enhancement[C]//ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019: 900-904.\n\n[2]. Wisdom S, Tzinis E, Erdogan H, et al. Unsupervised sound separation using mixture invariant training[J]. Advances in Neural Information Processing Systems, 2020, 33: 3846-3857.\n\n[3]. Tzinis E, Adi Y, Ithapu V K, et al. RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing[J]. IEEE Journal of Selected Topics in Signal Processing, 2022, 16(6): 1329-1341.\n\n[4] Tzinis E, Casebeer J, Wang Z, et al. Separate but together: Unsupervised federated learning for speech enhancement from non-iid data[C]//2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2021: 46-50.\n\n[5] Zhang J, Zorila C, Doddipatla R, et al. Teacher-student MixIT for unsupervised and semi-supervised speech separation[J]. arXiv preprint arXiv:2106.07843, 2021.\n\n[6]  Peer T, Gerkmann T. Phase-aware deep speech enhancement: It's all about the frame length[J]. JASA Express Letters, 2022, 2(10): 104802.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> (1) Is the MC loss function in this paper an innovation from [1] and where is the innovation marked?\n\n> [1]. Wisdom et al. Differentiable consistency constraints for improved deep speech enhancement, ICASSP, 2019.\n\nSorry for the confusion. We think that it is a novel contribution, and we should have marked the innovations.\n\nWe will emphasize in the paper that the proposed MC loss is very different from [1] in the following aspects.\n\nFirst, in [1], DNN estimates are strictly constrained to add up to the mixture (See Eq. (7) and (9) in [1]), but our MC loss only ``encourages'' the filtered DNN estimates to add up to the mixture.\n\nSecond, our MC loss is applied to filtered DNN estimates rather than DNN estimates.\n\nThird, we are dealing with multi-microphone MC, while [1] only addresses single-channel.\n\nOverall, the proposed MC loss and the one in [1] have very different motivations and physical meanings.\n\nWe now realize that it is not a good idea to use the same name, and we will change to ``mixture-constraint'' loss to indicate differences.\n\n> (2) Is the effectiveness of the UNSSOR method due to the powerful performance of the separation model TF-GridNet, or is it due to the contribution of the UNSSOR method itself?\n\n> I believe that to elaborate the capability of the UNSSOR method better; it is necessary to try to use different separation structures and compare their performance on each structure. If the UNSSOR method performs well on other separation structures, then we can be more confident that the UNSSOR method itself is highly robust and general. Such an analysis is necessary to assess and understand the actual value of the UNSSOR method.\n\nWe think that both are important. Without using UNSSOR to deal with the ill-posed problem, the modelling capability of strong DNNs cannot be unleased to separate speakers; and without using a strong DNN, the patterns in speech cannot be modelled well to realize good separation.\n\nWe think that the proposed UNSSOR method would not just work with a particular DNN architecture. We expect that the proposed UNSSOR mechanism can work with many DNN architectures, as long as the architecture is reasonably strong and can effectively handle reverberation, and that stronger DNN architectures would likely produce better separation.\n\nTo further address the comments, we replace TF-GridNet with TCN-DenseUNet [A1], and train the network using the same training configurations. TCN-DenseUNet contains a temporal convolution network (TCN) sandwiched by a UNet with DenseNet blocks. It is a reasonably strong separation model, which is fully convolutional and shares many similarities with many modern DNN architectures in the literature, and, according to [A2], it is worse than the recent TF-GridNet architecture in supervised separation tasks. We provide the unsupervised results in Table 1 of the attached .pdf file. We observe that it obtains reasonably-good separation results in unsupervised tasks.\n\n[A1] Wang et al., Leveraging Low-Distortion Target Estimates for Improved Speech Enhancement, arXiv preprint arXiv:2110.00570, 2021.\n\n[A2] Wang et al., TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation, arXiv preprint arXiv:2211.12433, 2022.\n\n> (3) Why not compare MixIT method?\n\n> The article mentions that MixIT may have two mixtures that are not in the same scene under reverberant conditions, which in turn may result in providing spatial a priori information, or providing a priori information due to differences in the configuration and number of microphones. However, I feel that this statement is a bit too hypothetical. I think these problems are more a result of how the dataset is constructed and not a deficiency of the MixIT method itself. More specifically, these problems should be solved by constructing a room-specific dataset. Therefore, I believe the problems mentioned in the paper do not serve as a weakness of the MixIT method.\n\n> I note that the approach in the paper combines traditional methods with the DNN-based method iRAS. UNSSOR, a DNN-based scheme, is compared to include only one DNN method, which may not be comprehensive. Of particular note is that MixIT is a popular and widely used method for unsupervised speech separation, as shown in [2-5]. I was puzzled while reading the paper as to why the authors did not include MixIT in the comparison.\n\nSee our response to all the reviewers.\n\n> (4) Why not use the same STFT settings, especially compared to the DNN model iRAS.\n\n> The authors set different STFT window sizes and hop lengths in the comparison, which may raise questions about the fairness of the results. For the separation model of DNN, different settings of STFT window sizes and hop lengths can significantly affect the model's performance [6].\n\n> [6] Peer et al. Phase-aware deep speech enhancement: It's all about the frame length, JASA, 2022.\n\nJust to clarify. For UNSSOR and iRAS, we use exactly the same STFT setting, i.e., 32 ms window size and 8 ms hop size.\nThis STFT setting is very common in STFT-domain separation algorithms.\n\nIf the reviewer meant different filter taps (in time) are used in UNSSOR and iRAS, Appendix G (see Fig. 4) is provided to address the concerns.\nThe idea is that, since UNSSOR performs filtering in the time-frequency domain and iRAS performs filtering in the time domain, we configure the filter lengths (in time) to be the same for comparison.\n\nIf the reviewer meant different STFT window and hop sizes are used for IVA and spatial clustering, we emphasize that it is very common for IVA and spatial clustering to use longer window so that enough amount of reverberation is covered in each frame. This way, their model assumption can be better satisfied and better separation can be achieved.\nIt is very common in IVA and spatial clustering to tune STFT window and hop sizes.\nWe will emphasize this in the paper.\n\nThe referred paper [6] only considers non-reverberant cases, and therefore may not tell the full story."}]

Author rebuttal:
We would like to thank the reviewers for their valuable feedback towards improving this manuscript.
Here we collectively address some common concerns raised by the reviewers:

$\textbf{1. Reasons for not comparing with MixIT in the first submission}$

We carefully considered using MixIT as a baseline, since MixIT also deals with unsupervised separation, but we realize that MixIT may not be a good baseline to UNSSOR out of the following considerations:

$\textbf{1.1.}$ MixIT needs to be trained on synthetic mixtures of mixtures (MoM), while UNSSOR is designed to be trained directly on existing mixtures. The two models would be trained on different training examples, and our concern is that this would make the comparison difficult. We hence consider methods that can be trained (or performed) directly on existing mixtures (such as IVA, spatial clustering and iRAS) as baselines.

$\textbf{1.2.}$ One could argue that we could, for example, mix the existing 2-speaker mixtures in SMS-WSJ to train MixIT, and compare it with UNSSOR trained directly on the existing 2-speaker mixtures in SMS-WSJ. However, this would require the mixtures used for creating each MoM to be recorded in the same room, by the same array, and at the same location in the room, each of which would incur restrictions to the dataset that can be used for training MixIT models, while UNSSOR does not have such restrictions.

$\textbf{1.3.}$ We could simulate a particular scenario, where, in each simulated room, we generate 4 speaker sources so that we can have several 2-speaker mixtures to create MoM for training MixIT models (for 4-speaker separation) and then compare the performance with the performance of UNSSOR trained on the same 2-speaker mixtures for 2-speaker separation. However, this simulated scenario would be very ideal for MixIT. Note that the eventual MixIT system should synthesize MoM based on real-recorded mixtures. Many times, the procedure for synthesizing MoM is very tricky, since real-recorded mixtures are usually not recorded in the same room, using the same array, and at the same location in the room.

$\textbf{1.4.}$ A possibly good way to compare UNSSOR and MixIT is by using real-recorded datasets such as AMI, AliMeeting and CHiME-{5,6,7} recorded in meeting scenarios where concurrent speech naturally happens. However, for both algorithms, to achieve good performance we need to solve many other problems (such as sparse speaker overlap, varying number of speaker, unknown number of speakers etc.) and there do not exist mature solutions yet; and, in addition, including the solutions to many other problems in this paper would make the paper much less focused. We hence lean towards leaving this investigation to a future study, and focus on showing the potential of UNSSOR, which avoids using synthetic MoM.

We will add these discussions and considerations to the paper.

$\textbf{2. Performance comparison with MixIT}$

Following the reviewers' suggestions, in this rebuttal, we have been trying to provide a performance comparison with MixIT.
However, due to time limit, the training cannot finish before the rebuttal deadline, and we will update the results during the discussion phase (by the end of Aug. 13 ET).

What we have been doing is to create the particular scenario (**ideal for MixIT**) described in paragraph $\textbf{1.3}$ above, where, for each existing SMS-WSJ mixture ($y = s_1 + s_2 + n$ where $n$ denotes noise), we randomly add two extra speakers in the same simulated room and use the same array placed at the same location so that we can have two 2-speaker mixtures (i.e., mixture 1: $s_1 + s_2 + n/2$ and mixture 2: $s_3 + s_4 + n/2$) to create MoM for training MixIT for 4-speaker separation.
The DNN architecture and training configurations for MixIT are the same as that in UNSSOR and PIT, the loss function is defined similarly on the real, imaginary and magnitude of the reconstructed mixtures, and the DNN can take single- or multi-channel input.
At run time, the trained MixIT model is used to separate the existing two-speaker mixtures in SMS-WSJ to four outputs, and we select the two outputs with the highest energy for evaluation. This way, the numbers can be pretty much directly compared with the existing ones obtained by UNSSOR.

We will update the results during the discussion phase (by the end of Aug. 13 ET). We will also add the results to the paper, if the paper is accepted.

Meta-review: This paper proposed an algorithm named unsupervised neural speech separation by leveraging over-determined training mixtures.  It received acceptance from all reviewers, with reviewer ux7k particularly appreciating the rebuttal and changing his/her score from 5 to 6. There is no doubt that this paper is recommended for acceptance.
Decision: Accept (poster)