SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the difference is that we also filter the DNN estimates when computing the loss on the reference microphone, and we constrain  $\hat{\mathbf{g}}_p(c,f)$  to be causal and that  $\widetilde{\hat{\mathbf{Z}}} (c,t,f)$  only stacks current and past frames. This way, the resulting  $\hat{Z} (c)$  would not be time-aligned with the reverberant image captured at the reference microphone (i.e.,  $X_{1}(c)$ ) or any other non-reference microphones. Because of the causal filtering,  $\hat{Z} (c)$  would be more like an estimate of the reverberant image captured by a virtual microphone that is closer to speaker  $c$  than all the  $P$  microphones (see Appendix G.1 for an interpretation). It would contain less reverberation of speaker  $c$  than any of the speaker images captured by the  $P$  microphones due to the causal filtering.
To produce an estimate that is time-aligned with the reverberant image at a microphone (e.g.,  $X_{p}(c)$ ), we use the FCP-estimated image computed in (8) (i.e.,  $\hat{X}_{p}^{\mathrm{FCP}}(c)$ ) as the output.
4.5 Addressing frequency permutation problem
In (4) and (9), FCP is performed in each frequency independently from the others. Even though the speakers are separated at each frequency, the separation results of the same speaker at different frequencies may however not be grouped into the same output spectrogram (see an example in Appendix D). This is known as the frequency permutation problem [41], which has been studied for decades in frequency-domain blind source separation algorithms such as frequency-domain independent component analysis [37-41] and spatial clustering [45-48]. Popular solutions for frequency alignment are designed by leveraging cross-frequency correlation of spectral patterns [47, 64] and direction-of-arrival estimation [65]. However, these solutions are often empirical and have a complicated design. They can be used to post-process DNN estimates for frequency alignment, but it is not easy to integrate them with UNSSOR for joint training. This section proposes a loss term, with which the trained DNN can learn to produce target estimates without frequency permutation.
To deal with frequency permutation, IVA [42-44] assumes that, at each frame, the de-mixed outputs at all the frequencies follow a complex Gaussian distribution with a shared variance term across frequencies:  $\mathbf{w}(c,f)^{\mathrm{H}}\mathbf{Y}(t,f)\sim \mathcal{N}(0,D(t,c))$ , where  $\mathbf{w}(c,f)\in \mathbb{C}^P$  is the de-mixing weight vector (in a time-invariant de-mixing matrix) for speaker  $c$  at frequency  $f$ , and  $D(t,c)\in \mathbb{R}$  is the shared variance term, which is assumed time-variant. When maximum likelihood estimation is performed to estimate the de-mixing matrix, the variance term shared across all the frequencies is found very effective at solving the frequency permutation problem [41-44].
Motivated by IVA, we design the following loss term, named intra-source magnitude scattering (ISMS), to alleviate the frequency permutation problem in DNN outputs:
$$
\mathcal {L} _ {\mathrm {I S M S}} = \sum_ {p = 1} ^ {P} \alpha_ {p} \mathcal {L} _ {\mathrm {I S M S}, p} = \sum_ {p = 1} ^ {P} \alpha_ {p} \frac {\sum_ {t} \frac {1}{C} \sum_ {c = 1} ^ {C} \operatorname {v a r} \left(\log \left(| \hat {X} _ {p} ^ {\mathrm {F C P}} (c , t , \cdot) |\right)\right)}{\sum_ {t} \operatorname {v a r} \left(\log \left(| Y _ {p} (t , \cdot) |\right)\right)}, \tag {10}
$$
where  $\hat{X}_p^{\mathrm{FCP}}$  is computed via (8),  $\hat{X}_p^{\mathrm{FCP}}(c,t,\cdot)\in \mathbb{C}^F$ , and  $\operatorname{var}(\cdot)$  computes the variance of the values in a vector. At each frame, we essentially want the the magnitudes of the estimated spectrogram of each speaker (i.e.,  $\hat{X}_p^{\mathrm{FCP}}(c,t,\cdot)$ ) to have a small intra-source variance. The rationale is that, when
Note that the relative RIR relating a signal to its delayed version is causal and the relative RIR relating a signal to its advanced version is non-causal [63]. See Appendix F for an intuitive explanation.
frequency permutation happens,  $\hat{X}_p^{\mathrm{FCP}}(c,t,\cdot)$  would contain multiple sources, and the resulting variance would be larger than that computed when  $\hat{X}_p^{\mathrm{FCP}}(c,t,\cdot)$  contains only one source.  $\mathcal{L}_{\mathrm{ISMS}}$  echoes IVA's idea of assuming a shared variance term across all the frequencies. If the ratio in (10) becomes smaller, it indicates that the magnitudes of  $\hat{X}_p^{\mathrm{FCP}}(c,t,\cdot)$  are more centered around their mean. This is similar to optimizing the likelihood of  $\hat{X}_p^{\mathrm{FCP}}(c,t,\cdot)$  under a Gaussian distribution with a variance term shared across all the frequencies. In (10), a logarithmic compression is applied, since log-compressed magnitudes better follow Gaussian distribution than raw magnitudes [66].
We combine  $\mathcal{L}_{\mathrm{ISMS}}$  with  $\mathcal{L}_{\mathrm{MC}}$  in (9) for DNN training, using a weighting term  $\gamma \in \mathbb{R}$ :
$$
\mathcal {L} _ {\mathrm {M C} + \text {I S M S}} = \mathcal {L} _ {\mathrm {M C}} + \gamma \times \mathcal {L} _ {\text {I S M S}}. \tag {11}
$$
4.6 Training UNSSOR for monaural unsupervised separation
UNSSOR can be trained for monaural unsupervised separation by only feeding the mixture at the reference microphone to the DNN but still computing the loss on multiple microphones. Fig. 1 illustrates the idea. At run time, the trained system performs monaural under-determined separation, and multi-microphone over-determined mixtures are only required for DNN training. The loss computed at multiple microphones could guide the DNN to exploit monaural spectro-temporal patterns for separation, even in an unsupervised setup.
We validate the proposed algorithms on two-speaker separation in reverberant conditions based on the six-channel SMS-WSJ dataset [67] (see Appendix A for its details). This section describes the baseline systems and evaluation setup. See Appendix I for miscellaneous system and DNN setup.
The baselines include conventional unsupervised separation algorithms, an improved version of RAS, MixIT, and supervised learning based models.
We include spatial clustering [46-48] for comparison. We use a public implementation[68], which leverages complex angular-central Gaussian mixture models [48] for sub-band spatial clustering and exploits inter-frequency correlation of cluster posteriors [46, 64] for frequency alignment. The number of sources is set to three, one of which is used for garbage collection, following [44]. After obtaining the estimates, we discard the one with the lowest energy. The STFT window size is tuned to  $128~\mathrm{ms}$  and hop size to  $16~\mathrm{ms}$ .
We include IVA [41, 44] for comparison. We use the public implementations provided by the torchiva toolkit [69]. We use the default spherical Laplacian model to model source distribution. In overdetermined cases, the number of sources is set to three and we discard the estimate with the lowest energy, similarly to the setup in the spatial clustering baseline<sup>4</sup>. The STFT window size is tuned to 256 ms and hop size to 32 ms.
We propose a novel variant of the RAS algorithm [36] for comparison. Appendix H discusses the differences between UNSSOR and RAS. Since RAS cannot achieve unsupervised separation, we improve it by computing loss on multi-microphone mixtures, and name the new algorithm as improved RAS (iRAS). We employ the time-domain Wiener filtering (WF) technique in [36] to filter re-synthesized time-domain estimates  $\hat{z}(c) = \mathrm{iSTFT}(\hat{Z}(c))$ , where  $\hat{Z}(c)$  is produced by TF-GridNet. The loss is defined as:
$$
\mathcal {L} _ {\mathrm {i R A S}} = \sum_ {p = 1} ^ {P} \alpha_ {p} \mathcal {L} _ {\mathrm {i R A S}, p} = \sum_ {p = 1} ^ {P} \alpha_ {p} \frac {1}{\| y _ {p} \| _ {1}} \left\| y _ {p} - \sum_ {c = 1} ^ {C} \hat {h} _ {p} (c) * \hat {z} (c) \right\| _ {1}, \tag {12}
$$
with  $*$  denoting linear convolution,  $y_{p}$  the time-domain mixture at microphone  $p$ , and  $\hat{h}_p(c)$  a time-domain Wiener filter computed by solving the following problem:
$$
\hat {h} _ {p} (c) = \operatorname {a r g m i n} _ {h _ {p} (c)} \| y _ {p} - h _ {p} (c) * \hat {z} (c) \| _ {2} ^ {2}, \tag {13}
$$
<sup>3</sup>https://github.com/fgnt/pb_bss/blob/master/examples/mixture_model_example.ipynb<sup>4</sup>For IVA and spatial clustering, using a garbage source leads to better separation in our experiments.
Table 1: Averaged results of 2-speaker separation on SMS-WSJ (6-channel input and loss).
Row	Systems	I	J	Loss	Val. set SDR (dB)	SDR (dB)	Test set
SI-SDR (dB)	PESQ	eSTOI
0a	Mixture	-	-	-	0.1	0.1	0.0	1.87	0.603
1a	UNSSOR	19	0	LMC	12.5	11.9	10.2	2.61	0.735
1b	UNSSOR + Corr. based freq. align.	19	0	LMC	16.1	15.7	14.7	3.47	0.884
1c	UNSSOR + Oracle freq. align.	19	0	LMC	16.2	15.8	14.9	3.48	0.889
2a	UNSSOR	19	0	LMC+ISMS	16.0	15.6	14.6	3.44	0.885
2b	UNSSOR + Corr. based freq. align.	19	0	LMC+ISMS	16.0	15.6	14.7	3.44	0.885
2c	UNSSOR + Oracle freq. align.	19	0	LMC+ISMS	16.0	15.6	14.7	3.44	0.886
3a	Spatial clustering + Corr. based freq. align. [44]	-	-	-	8.8	8.6	7.4	2.44	0.726
3b	IVA [44]	-	-	-	10.3	10.6	8.9	2.58	0.764
3c	iRAS w/ causal 512-tap filters	-	-	LIRAS	7.8	7.6	5.7	2.14	0.642
3d	iRAS w/ non-causal 512-tap filters (100 future taps)	-	-	LIRAS	8.0	7.8	5.7	2.13	0.637
4a	PIT (supervised) [23]	-	-	-	19.9	19.4	18.9	4.08	0.949
Notes: the rows show in grey indicate using oracle information or using supervised models.
Table 2: Averaged results of 2-speaker separation on SMS-WSJ (3-channel input and loss).
Row	Systems	I	J	Loss	Val. set SDR (dB)	Test set
SDR (dB)	SI-SDR (dB)	PESQ	eSTOI
0a	Mixture	-	-	-	0.1	0.1	0.0	1.87	0.603
1a	UNSSOR	19	0	LMC	9.9	9.4	7.4	2.12	0.672
1b	UNSSOR + Corr. based freq. align.	19	0	LMC	15.3	15.0	13.9	3.18	0.867
1c	UNSSOR + Oracle freq. align.	19	0	LMC	15.5	15.2	14.1	3.19	0.871
2a	UNSSOR	19	0	LMC+ISMS	15.7	15.4	14.4	3.20	0.874
2b	UNSSOR + Corr. based freq. align.	19	0	LMC+ISMS	15.7	15.4	14.4	3.20	0.875
2c	UNSSOR + Oracle freq. align.	19	0	LMC+ISMS	15.8	15.4	14.5	3.20	0.876
3a	Spatial clustering + Corr. based freq. align. [44]	-	-	-	9.6	9.5	8.5	2.52	0.759
3b	IVA [44]	-	-	-	11.6	12.0	10.7	2.67	0.802
3c	iRAS w/ causal 512-tap filters	-	-	LIRAS	5.1	4.8	2.7	1.88	0.588
3d	iRAS w/ non-causal 512-tap filters (100 future taps)	-	-	LIRAS	4.6	4.5	2.2	1.87	0.579
4a	PIT (supervised) [23]	-	-	-	17.4	16.8	16.3	3.91	0.924
which is quadratic and has a closed-form solution. The separation result is computed as  $\hat{x}_p^{\mathrm{WF}}(c) = \hat{h}_p(c)*\hat{z} (c)$ . Following [36], we use 512 filter taps, and filter the future 100, the current, and the past 411 samples (i.e., non-causal filtering). We can also filter the current and the past 511 samples (i.e., causal filtering), and additionally experiment with a filter length (in time) same as the length of the FCP filters (see Appendix J).
For comparison, we also include MixIT [30], which requires using synthetic mixtures of mixtures.
We report the result of using supervised learning, where PIT [5] is used to address the permutation problem. This result can be viewed as a performance upper bound of unsupervised separation.
We use the same DNN and training configurations as those in UNSsOR for a fair comparison.
5.2 Evaluation setup and metrics
We designate the first microphone as the reference microphone, and use the time-domain signal corresponding to  $X_{1}(c)$  of each speaker  $c$  for metric computation. The evaluation metrics include signal-to-distortion ratio (SDR) [70], scale-invariant SDR (SI-SDR) [71], perceptual evaluation of speech quality (PESQ) [72], and extended short-time objective intelligibility (eSTOI) [73]. SI-SDR and SDR evaluate the sample-level accuracy of predicted signals, and PESQ and eSTOI are objective metrics of speech quality and intelligibility respectively. For all the metrics, the higher, the better.
6 Evaluation results
This section reports evaluation results on SMS-WSJ and compares the performance of various setups.
6.1 Effectiveness of UNSSOR at promoting separation
Table 1 and 2 respectively report the results of using six- and three-microphone input and loss. After hyper-parameter tuning, in default we use the loss in (9) for DNN training, set  $I = 19$  and  $J = 0$  (defined below (6)) for FCP (i.e., causal FCP filtering with 20 taps), and set  $\alpha_{p} = 1$  (meaning no
Table 3: Averaged results of 2-speaker separation on SMS-WSJ (1-channel input and 6-channel loss).
Row	Systems	I	J	Loss	Val. set SDR (dB)	SDR (dB)	Test set
SI-SDR (dB)	PESQ	eSTOI
0a	Mixture	-	-	-	0.1	0.1	0.0	1.87	0.603
1a	UNSSOR	19	1	\( \mathcal{L}_{\text{MC+ISMS}} \)	13.0	12.5	11.9	3.27	0.832
2a	iRAS w/ causal 512-tap filters	-	-	\( \mathcal{L}_{\text{iRAS}} \)	7.5	7.2	5.6	2.03	0.641
2b	iRAS w/ non-causal 512-tap filters (100 future taps)	-	-	\( \mathcal{L}_{\text{iRAS}} \)	10.7	10.5	9.7	2.80	0.778
3a	Monaural PIT (supervised) [23]	-	-	-	16.2	15.7	15.3	3.79	0.907
Table 4: Averaged results of 2-speaker separation on SMS-WSJ (1-channel input and 3-channel loss).
Row	Systems	I	J	Loss	Val. set SDR (dB)	SDR (dB)	Test set
SI-SDR (dB)	PESQ	eSTOI
0a	Mixture	-	-	-	0.1	0.1	0.0	1.87	0.603
1a	UNSSOR	19	1	\( \mathcal{L}_{\text{MC+ISMS}} \)	12.5	12.0	11.4	3.18	0.822
2a	iRAS w/ causal 512-tap filters	-	-	\( \mathcal{L}_{\text{iRAS}} \)	-0.1	-0.3	-3.0	1.62	0.453
2b	iRAS w/ non-causal 512-tap filters (100 future taps)	-	-	\( \mathcal{L}_{\text{iRAS}} \)	11.0	10.7	9.9	2.81	0.783
3a	Monaural PIT (supervised) [23]	-	-	-	16.2	15.7	15.3	3.79	0.907
weighting is applied for different microphones). For the 3-microphone case, we use the mixtures at the first, third, and fifth microphones for training and testing. Notice that for two-speaker separation, the cases with six or three microphones are both over-determined.
In both tables, from row 1a we observe that UNSSOR produces reasonable separation of speakers, improving the SDR from 0.1 to, for example,  $12.5\mathrm{dB}$  in Table 1, but its output suffers from the frequency permutation problem (see Appendix D for an example). In row 1c, we use oracle target speech to obtain oracle frequency alignment and observe much better results over 1a. This shows the effectiveness of  $\mathcal{L}_{\mathrm{MC}}$  at promoting separation of speakers and the severity of the frequency permutation problem. In row 1b, we use a frequency alignment algorithm (same as that used in the spatial clustering baseline) [46, 64] to post-process the separation results of 1a. This algorithm leads to impressive frequency alignment (see 1b vs. 1c), but it is empirical and has a complicated design.
6.2 Effectiveness of ISMS loss at addressing frequency permutation problem
We train DNNs using  $\mathcal{L}_{\mathrm{MC + ISMS}}$  defined in (11). In each case (i.e., six- and three-microphone), we separately tune the weighting term  $\gamma$  in (11) based on the validation set. In both table, comparing row 2a-2c with 1a-1c, we observe that including  $\mathcal{L}_{\mathrm{ISMS}}$  is very effective at dealing with the frequency permutation problem, yielding almost the same performance as using oracle frequency alignment.
6.3 Results of training UNSSOR for monaural unsupervised separation
Table 3 and 4 use the mixture only at the reference microphone 1 as the network input, while computing the loss respectively on three and six microphones. We tune  $J$  to 1 (i.e., non-causal FCP filter), considering that, for a specific target speaker, the reference microphone may not be the microphone closest to that speaker<sup>5</sup>. See an interpretation on why non-causal filtering is needed in Appendix G.2. We still set the microphone weight  $\alpha_{p}$  to 1.0 for non-reference microphones (i.e., when  $p \neq 1$ ), but tune  $\alpha_{1}$  to a smaller value based on the validation set. Without using a smaller  $\alpha_{1}$ , we found that the DNN easily overfits to microphone 1, as we use the mixture at microphone 1 as the only input and compute the MC loss also on the mixture at microphone 1. The DNN can just aggressively optimize  $\mathcal{L}_{\mathrm{MC},p}$  to zero at microphone 1 and not optimize that at other microphones.
From row 1a of both tables, strong performance is observed in this under-determined setup, indicating that the multi-microphone loss can inform the DNN what desired target sound objects are and the DNN can learn to model spectral patterns in speech for unsupervised separation. In addition, the result in 1a of Table 3 is better than that in Table 4. This indicates that using more microphones as constraints in the proposed MC loss can elicit better separation.
<sup>5</sup>We do not need many future taps, considering that the hop size is 8 ms in our system and the microphone array in SMS-WSJ is a compact array with a diameter of 20 cm. In air, sound would travel  $340 \times 0.008 = 2.72$  meters in 8 ms if its speed is 340 meters per second. This distance is far larger than the array aperture size.
6.4 Comparison with other methods and supplementary results
In Table 1-4, we compare the performance of UNSSOR with spatial clustering, IVA, iRAS, and supervised PIT-based models. In Appendix J, we compare UNSSOR and iRAS when they use the same filter length (in time). UNSSOR shows better performance than previous unsupervised separation models that can be performed or trained directly on mixtures. A sound demo is available at this link<sup>6</sup>. UNSSOR is worse than supervised PIT but the performance is reasonably strong. For example, in row 2a of Table 2, UNSSOR obtains 15.4 dB SDR on the test set, which is close to the 16.8 dB result obtained by supervised PIT in 4a. In Appendix L, we compare the results of UNSSOR with MixIT [30] and observe better performance.
We think that the effectiveness of the proposed system comes from both the strong modeling capability of TF-GridNet and the UNSSOR mechanism itself. In Appendix K, we use UNSSOR with other separation models such as TCN-DenseUNet [74] and DPRNN [75], which are known to have weaker modelling capability than TF-GridNet in supervised separation tasks (see for example [23]). We observe that the separation performance is worse but still reasonable.
Our study shows the strong potential of UNSSOR for unsupervised speech separation. There are, however, several weaknesses we need to address in future research. First, we assume that sources are directional point sources so that each relative RIR can be modelled using a short filter, and diffuse sources are not considered. Second, we assume that sources are non-moving within each utterance so that we can use time-invariant FCP filters. Third, we assume that the number of sources is known and the sources are fully-overlapped. Fourth, only measurement or modeling noise is considered and realistic directional or diffuse background noises with strong energy are not included. Although these assumptions are also made in many algorithms such as IVA, spatial clustering, RAS and iRAS, they need to be addressed to realize more practical and robust speech separation systems.
We have proposed UNSSOR for unsupervised neural speech separation. We show that it is possible to train unsupervised models directly on mixtures, if the mixtures are over-determined. We have proposed mixture-constraint loss functions, which leverage multi-microphone mixtures as constraints, to promote separation of speakers. We find that minimizing ISMS can alleviate the frequency permutation problem. Although UNSSOR requires over-determined training mixtures, it can be trained to perform under-determined unsupervised separation. Future research will combine UNSSOR with semi-supervised learning, evaluate it on real-recorded noisy-reverberant data such as CHiME-6 [76], AMI [77] and AliMeeting [78], and address the limitations described in Section 7.
In closing, we emphasize that a key scientific contribution of this paper is that the over-determined property afforded by having more microphones than speakers can narrow down the solutions to the underlying sources, and this property can be leveraged to design a supervision to train DNNs to model speech patterns via unsupervised learning and realize unsupervised separation. This meta-idea, we believe, would motivate the design of many algorithms in future research on neural source separation.
We would like to thank Dr. Robin Scheibler at LINE Corporation for constructive discussions on IVA, and Dr. Samuele Cornell for helpful discussions on MixIT. This research is part of the Delta research computing project, which is supported by the National Science Foundation (Award OCI 2005572) and the State of Illinois. Delta is a joint effort of the University of Illinois at Urbana-Champaign and its National Center for Supercomputing Applications. We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the RTX 8000 GPUs used in this research.
$^{6}$ https://zqwang7.github.io/demos/UNSSOR-demo/index.html.
- Appendix A describes the SMS-WSJ dataset.
- Appendix B visualizes the surface of the proposed mixture-constraint loss to show that minimizing the loss can promote unsupervised separation of speakers.
- Appendix C provides the derivation that FCP can be used to approximate speaker images.
- Appendix D illustrates the frequency permutation problem.
- Appendix E describes differences between the MC loss and the mixture consistency concept.
- Appendix F illustrates the cases on when to use causal and non-causal filtering.
- Appendix G provides interpretations of the physical meanings of intermediate DNN estimates  $\hat{Z}$ .
- Appendix H discusses differences between UNSSOR and RAS.
- Appendix I presents miscellaneous system and DNN configurations.
- Appendix J experiments with alternative filters taps for iRAS and UNSSOR.
- Appendix K experiments UNSSOR with DNN architectures with lower modelling capabilities.
- Appendix L reports the results of MixIT.
A SMS-WSJ dataset
SMS-WSJ [67] is a popular corpus for evaluating two-speaker separation algorithms in reverberant conditions. The clean speech is sampled from the WSJ0 and WSJ1 datasets. The corpus contains 33,561 ( $\sim$ 87.4 h), 982 ( $\sim$ 2.5 h), and 1,332 ( $\sim$ 3.4 h) two-speaker mixtures respectively for training, validation, and testing. The simulated microphone array has six microphones arranged uniformly on a circle with a diameter of  $20\mathrm{cm}$ . For each mixture, the speaker-to-array distance is drawn from the range  $[1.0, 2.0]\mathrm{m}$ , and the reverberation time (T60) is sampled from  $[0.2, 0.5]\mathrm{s}$ . A weak white noise is added to simulate microphone self-noises, and the energy level between the sum of the reverberant speech signals and the noise is sampled from the range  $[20, 30]\mathrm{dB}$ . The sampling rate is  $8\mathrm{kHz}$ .
B Visualization of loss surface
Fig. 2 visualizes the values of  $\mathcal{L}_{\mathrm{MC}}$  in (4) based on a six-channel noisy-reverberant two-speaker mixture sampled from the SMS-WSJ dataset (see Appendix A for the dataset details). Let  $C = 2$  and suppose that the DNN estimates are
$$
\hat {Z} (1) = \mu \times X _ {1} (1) + \nu \times X _ {1} (2) + \varepsilon_ {1} / 2
$$
$$
\hat {Z} (2) = (1 - \mu) \times X _ {1} (1) + (1 - \nu) \times X _ {1} (2) + \varepsilon_ {1} / 2, \tag {14}
$$
where  $\mu$  and  $\nu \in \mathbb{R}$  are bounded in the range [0, 1]. Essentially, we use  $\mu$  and  $\nu$  to mimic the cases that the DNN produces (1) good separation (i.e., when  $\mu \approx 1$  and  $\nu \approx 0$ , or  $\mu \approx 0$  and  $\nu \approx 1$ ); and (2) bad separation (i.e., when  $\mu$  and  $\nu$  are both away from 0 and 1, meaning that each estimate contains multiple speakers, and when  $\mu \approx 0$  and  $\nu \approx 0$ , or  $\mu \approx 1$  and  $\nu \approx 1$ , meaning that the two speakers are merged into one output and the other output does not contain any speakers). Fig. 2 enumerates  $\mu$  and  $\nu$  and plots the resulting separation results against the loss value of (4). We can see the loss values are smallest when  $\mu \approx 1$  and  $\nu \approx 0$  or when  $\mu \approx 0$  and  $\nu \approx 1$  (i.e., when the speakers are successfully separated), and clearly larger otherwise. This indicates that minimizing the proposed loss function can encourage separation.