Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments
Authors: Shitong Xu
Yiyuan Yang
Niki Trigoni
Andrew Markham
Venue: NeurIPS 2025 poster ()
Keywords: Target Speech Extraction
Few-Shot Inference
Primary area: applications
TL;DR: Target Speech Extraction conditioned on positive audio enrolment (where target speaker speaks) and negative audio enrolment (where target speaker does not speak).

Abstract: Target speaker extraction focuses on isolating a specific speaker's voice from an audio mixture containing multiple speakers. To provide information about the target speaker's identity, prior works have utilized clean audio samples as conditioning inputs. However, such clean audio examples are not always readily available. For instance, obtaining a clean recording of a stranger's voice at a cocktail party without leaving the noisy environment is generally infeasible. Limited prior research has explored extracting the target speaker's characteristics from noisy enrollments, which may contain overlapping speech from interfering speakers. In this work, we explore a novel enrollment strategy that encodes target speaker information from the noisy enrollment by comparing segments where the target speaker is talking (Positive Enrollments) with segments where the target speaker is silent (Negative Enrollments). Experiments show the effectiveness of our model architecture, which achieves over 2.1 dB higher SI-SNRi compared to prior works in extracting the monaural speech from the mixture of two speakers. Additionally, the proposed two-stage training strategy accelerates convergence, reducing the number of optimization steps required to reach 3 dB SNR by 60\%. Overall, our method achieves state-of-the-art performance in the monaural target speaker extraction conditioned on noisy enrollments. Our implementation is available at
https://github.com/xu-shitong/TSE-through-Positive-Negative-Enroll .

Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments
Shitong Xu Yiyuan Yang
Niki Trigoni
Andrew Markham
Department of Computer Science, University of Oxford shitong. xu@cs. ac.uk uk
Abstract
Target speaker extraction focuses on isolating a specific speaker's voice from audio mixture containing multiple speakers. To provide information about the target speaker's identity, prior works have utilized clean audio samples as conditioning inputs. However, such clean audio examples not always readily available. For instance, obtaining a clean recording of a stranger's voice at cocktail party without leaving the noisy environment is generally infeasible. Limited prior research has explored extracting the target speaker's characteristics from noisy enrollments, which may contain overlapping speech from interfering speakers. In this work, we explore novel enrollment strategy that encodes target speaker information from the noisy enrollment by comparing segments where the target speaker is talking (Positive Enrollments) with segments where the target speaker is silent (Negative Enrollments). Experiments show the effectiveness of our model architecture, which achieves over 2.1 dB higher SI-SNRi compared to prior works in extracting the monaural speech from the mixture of two speakers. Additionally, the proposed two-stage training strategy accelerates convergence, reducing the number of optimization steps required to reach 3 dB SNR by 60%. Overall, our method achieves state-of-the-an performance in the monaural target speaker extraction conditioned on noisy enrollments. Our implementation is available at https://github..com// com/x/uuuuoooouuuooaaaeoemo wwtp:/osstoooooooooooooooo TSE tatttthhrrroeeee Dositive-Negative-te
Introduction
In the target speaker extraction task, the model required to extract the target speaker's voice from a mixture of multiple speakers and ambient noise. To specify the characteristics of the target speaker, prior works have explored using conditional information of the target speaker in multiple modalities, including visual [1 (2), contextual 3 4) acoustic modality (5) 6 g). Though prior works using conditional information in the acoustic modality have achieved significant good extraction performance, most of these works only considered using clean audio examples of the target speaker 10 [2]. This strong assumption prevents these models from performing well when only noisy audio examples are available. For example, consider cocktail party, where the user meets a stranger who has not provided an audio sample before. To extract the target stranger speaker's voice from the noisy environment to assist clear conversation, the user will have to ask the speaker to step outside to record the clean audio enrollment. Enrolling target speakers in this way is often impractical in real-world applications.
Although prior works have attempted to perform target speaker extraction using noisy enrollments, they either still exploit the timesteps where only the target speaker speaking the noisy enrollment [3] [4] or assume that the user has participated in the audio mixture recording [6]., limiting extraction from arbitrary audio mixtures. In this work, we present method for performing target speaker extraction conditioned on noisy audio enrollments where both the target speaker and the
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Miture
Task Sconario
Target Speakerss Crean Speech
only hoir
Arbitaryy Audio Cilp
Extraction Mocel
Negative
Enniimmnt Enonder
bJ Enroliment Design
c) Model Architecture
oengmeen
regnent B
aegment
Negrennt
Proelivw Enndikreet Zeegmment B C)
Negathe Enrollmont
IsegmentA- + DJ
Ennollment Enooder
Fis Enrol ment Encoder
Atton Comparison
mnrt Encoder
Negothve
Fos tive Ennolmmnt
Nogativve Enroliment- (argaa apassker ourraitt saluet
(whrn largel speakkrr spaaking)
Audio Midure
Clean Speech
Target Speaker
Blue Inturferens Ramoved tirough compariing sugmants B
Extraction
www.t
100 intw Heriiniing Speeioers
Red and Groen Interfoners. Removed through comparing sogmens C
a) Task scenario explored in the work. Users identify a speaker of interest in an audio mixture. and labeling when the target speaker speaks (Positive Enrollment) or remains silent (Negative Enrollment) in the audio mixture. b) Decomposition of each speaker's voice the audio mixture. Due to the stochasticity in human conversation, the interfering speaker will either remain silent in some of the segments the Positive Enrollment or speak in the Negative Enrollment, leaving the target speaker the only speaker who talks throughout the Positive Enrollments but not in the Negative Enrollment. c) The model performs self-attention between the encoded Positive and Negative Enrollments to extract the target speaker's characteristic, which serves as the conditional information for the following extraction model. The model then extract the target speaker from the Audio Mixture.
Figure 1:
interfering speakers overlap, as shown in Figure (a). To resolve the ambiguity of which speaker in the noisy enrollment is the target speaker, we exploit the stochastic nature of different speakers' speaking patterns, which allows us to assume that speakers won't 't't start and stop talking in perfect synchrony in a noisy conversational environmen As shown in Figure [ (b), based on this assumption, we structure the enrollment input as a combination of two segments: a Positive Enrollment (where the target speaker is speaking) and a Negative Enrollment (where the target speaker remains silent). During training, we encourage the model to encode the identity of the speaker who consistently speaks during Positive Enrollments and remains silent during Negative Enrollments. Interfering speakers, from the assumption above, will be inactive during parts of the Positive Enrollment (e-g., Blue speaker in segment C) and/or present during the Negative Enrollment (e.g.. Red and Green speakers in segments and D). By exploiting these temporal misalignments, the model can distinguish and encode the target speaker's iss characteristics, despite the presence of interfering speech and noise throughout both enrollments. We detail the problem formulation and the training pipeline to achieve this in Section 3
By performing target speaker extraction in this manner, our model supports a broad range of real- world applications. Enrollment could easily be achieved in reality by pressing a button on an app or tapping an earbud to indicate positive segment. Note that precise labeling is not required. Similarly, for offline speaker extraction (e.g., from an audio recording of meeting), a few seconds of audio can be easily labeled as Positive or Negative samples.
In conclusion, our contributions include:
Exploring a novel enrollment strategy for target speaker extraction with noisy positive/negative enrollments and imprecise labels.
Design of two-branch encoder and an associated two-stage training strategy for the task. Ablation experiments show that the proposed training method allows our model to achieve the same level of performance (3.0 dB SNR on the validation set) with 60% fewer optimization steps.
3. Demonstrate and discuss our model's performance across challenging and realistic application scenarios, including different numbers of interfering speakers, significant overlap between target and interfering speakers in enrollments, and inaccuracies in the positive and negative enrollment labeling.
We quantitatively verify the feasibility of this assumption in Appendix A.
2

---

2 Related Work
2.1 Target Speaker Extraction in Challenging Scenarios
In target speaker extraction, prior works have explored using visual textural 3 4 examples [ 6 7 g of the target speaker as extraction conditions. Our work belongs to the third category, which learns the acoustic characteristics from given audio examples. Prior works this category have attempted to improve extraction quality by modifying the fusion method [9] [7], leveraging multi-channel information in the audio mixture input [10 18] and processing audio the temporal domain [6 7
and audio
To improve the models' robustness and applicability in real-world applications, prior works have
explored extracting multiple target speakers' speech simultancously [19] 3. [2)]. transferring to
extract speech for different languages [2]] and disentangling irrelevant audio characteristics (c.g..
reverberation effect) from the enrollment [21 22 23 24
25 26]. In particular, Zhao et al. [[]]
addressed the problem of variable interfering speaker numbers in the audio mixture to be separated. In our experiment, we also show our model's s generalizability to extract with different numbers of target and interfering speakers. Ranjan et al. [22] adopted curriculum learning to gradually increase the extraction difficulty. After fixed number of epochs, samples with lower signal-to-noise ratio (SNR) and higher similarity between target and interfering speakers are added to the training dataset. Our method adopts a similar training paradigm to pretraining the audio encoder. Both ours and the prior work from Ranjan et al. [22] show that the multi-stage training improves model convergence speed and leads to better final performance.
2.2 Target Audio Extraction Conditioned on Enrollments with Interfering Speakers
Though prior works have achieved significant progress in target audio extraction, most of these works assume the availability of clean audio enrollment. However, in real-world application scenarios, such enrollment examples are not necessarily available.
Among those prior works that attempted to extract target speech conditioned on noisy audio enroll- ments, TCE [15] explored the turn-tal turn-taking dynamics in the conversation. TCE model accepts an audio encoding of the user's clean voice and performs extraction by considering the speakers who cross-talk with the user as the interfering speakers. However, TCE is limited to performing target speaker extraction when the user is participating in the conversation and does not allow specifying audio segments where the target speaker is present. A closely related work to ours is ADEnet [13, which also utilizes target speaker activity labeling to perform speech extraction. However, ADEnet only validated its performance enrollment contains mostly clean speech. As a result, it does not explore the challenges unstable optimization and high overlap ratio between the target speaker and the interferers in the enrollment segment, which we explicitly address in our work. LookOnce ToHaar [66 leverages binaural audio to extract the target speaker's 's voice. During the enrollment stage, when the user is looking toward the target speaker, the model performs beamforming at a 90-degree azimuth to disambiguate the target speaker from the interferers talking simultaneously.
two-speaker mixtures with an average overlap ratio of 38.5%, where the
In comparison to the previous target speaker extraction methods, our model does not assume prior knowledge of any speaker in the mixed audio or the target speaker's spatial location in the enrollment stage. This allows our model to extract the target speaker from an arbitrary mono-channel sound mixture, and only use easily obtained binary labels of positive and negative audio segments in the temporal domain to extract the target speech. Besides, our model can flexibly perform extraction using either clean noisy enrollments without retraining and get comparable performance with the previous method trained on clean enrollment, as shown in the Appendix J.
3 Method
3.1
Problem Formulation
In the target speaker extraction (TSE) task, the model first encodes the target speaker's 's characteristic from an enrollment input using an encoding branch. Conditioned on the encoding branch output, the model extracts the target speaker's voice from an Audio Mixture A To allow the model to extract the target speaker conditioned on noisy enrollments, we formulate our enrollment as pair of Positive Enrollment a and Negative Enrollment N After encoding the target speaker characteristic
3

---

F(a",a' from the enrollment pair, the model extracts the target speaker's voice from the audio mixture G(a F(a", ))
To simulate the audio mixture in noisy environment, we model the signal as the sum of an individual speaker's voice and the background noise. Thus, the noisy Positive Enrollment a Negative Enrollment a²", and the Audio Mixture a are constructed as:
M E
aj + M a
E (ESTRU(A
+n
Σ JESTR
a +
+n
(1)
NESIM Uff}
where af (.P.NM.) Mixture. n (P,N,. represents the background noise in the three mixtures. represents the target speaker ID, and ST S represent the interfering speakers in the enrollments and the Audio Mixture, respectively.
represents the speech signal of speaker in the Positive, Negative, and Audio
3.2 Speaker Disambiguation via Positive and Negative Enrollments
To show how the Positive and Negative Enrollment pair resolves the ambiguity of which speaker is the target speaker in the noisy enrollments, we introduce the following assumption about speakers' temporal dynamics:
Assumption: In a long audio mixture where multiple speakers speak independently, no two speakers will consistently start and stop talking at exactly the same times throughout the mixture.
This assumption is easily satisfied: For two speakers to always talk simultaneously, one speaker would have to deliberately start and stop speaking at the exact same times as the other, effectively aiming to sabotage the other's conversation. Such speakers interfering with intention are unlikely to exist in typical vocktail-party scenario, when interfering speakers in the environment are talking independently from the target speaker. We quantitatively verify this assumption on the real-world audio dataset in Appendix A.
Based on the assumption above, the Positive and Negative Enrollments divide interfering speakers into 4 categories, depending on their existence in both enrollments:
1) Negative Interferer (NI): These are speakers who talk consistently throughout the Positive Enroll- ment and are present in the Negative Enrollment. The model should learn their voice characteristics from the Negative Enrollment and exclude them from the encoding.
2) Positive Interferer (PI): These are speakers who are not present in the Negative Enrollment and talk in fraction of the Positive Enrollment. The model should identify those segments when PIs are absent in the Positive Enrollment and exclude these speakers in the encoding. We explore how sensitive our model is to this fractional overlap in Section[.44
3) Hybrid Interferer (HI): These speakers appear partially in both the Positive and Negative Enroll- ments. They resemble a combination of Positive and Negative Interferers, providing the model with two potential cues for suppression: their absence in some Positive Enrollment segments and their presence in some of the Negative Enrollment segments.
4) Requect-Required Interferer (NRI): Similar to the Negative Interferers, the model encoder should avoid include these speakers' voice characteristic in the extracted embedding.
These speakers appear exclusively in the Negative Enrollment.
Since Positive and Negative Interferers represent the two primary scenarios in which the model must learn to distinguish target speakers from interfering ones, we focus our training and evaluation on these two types. We demonstrate our model's generalizability when Hybrid and Neglect-Required Interferers exist in Appendix H and I, respectively.
3.3 Model Architecture
As shown in Figure 2 we adopt TF-GridNet 27 as the backbone for both our Encoding and Extraction Branches. The TF-GridNet architecture consists of a 2D convolution layer followed by stacks of three TF-GridNet blocks. Each block consists of two BiL-TM modules that capture the freeuefyy and temporal information, followed by a Full-band Self-attention Module designed to capture the longerange frame information between frames. To adapt the original TF-GridNet, which

---

Clean Speech
Encoding Branch
Siamese Encoder
Spos
Protrained TFGrdNet Encoder
GindfNee Blacks
Epos
Enuoed
Envoliment
Shared Parameter
Sneg
(Self-Altn)
Module
Fusion
Pooling
TFGrdNect Blocks
Negative Enrollment
Frozen
Extraction Branch
Extraction Branch Black
TFGrldNet Block
Extraction Fusion Module
8MR Loss
(Crose-Altn)
Predictinn
Figure 2: Encoding and Extraction Branch model architecture and training pipeline.
is designed to perform sound separation, to the Target Speaker Extraction task, we introduce an attention-based Encoder Fusion Module and an Extraction Fusion Module. The Encoder Fusion Module performs comparison between Positive and Negative Enrollment embeddings to generate the embedding of the target speaker, while the Extraction Fusion Module integrates enrollment encoding into the extraction model. The model architecture and training method for encoding the target speaker's 's feature are shown in Figure 2
The Positive and Negative Enrollments are first encoded by pair of TF-GridNet encoders in the Siamese Encoder. Since The goal of both branches the Siamese Encoder is to encode multiple speakers' characteristics from the noisy Positive or Negative Enrollments, we share the parameters between the two TF-GridNet encoders of the Siamese Encoder to reduce model parameter size. The Siamese encoder results in two sequences of embeddings Epss and Eneg with shapes [Tpas D] and [Tneg, D], respectively. Tpos: Tneg are the numbers of frames the positive and negative enrollment embeddings, and D is the feature dimension at each frame.
The Encoder Fusion Module then extracts the target speaker's s voice embedding from the Siamese Encoders' Input: output Ep and Fneg- The pseudo-code for the Encoder Fusion Module is shown in Figure In Step 1, two seg- ment embeddings, Spos and Sneg with shape [1, D). are first element-wise added to the input embeddings to allow model to distinguish which enrollment each embed- ding originates from. The resulting two embeddings are concatenated along the temporal dimension Step 2, and passed through two Full-band Self-attention layers in Step In these two layers, the self-attention calculation be- tween the embeddings of different Positive Enrollment frames allows the model to identify the Positive Interfer- ers, who remain silent in some of the Positive Enrollment frames, and exclude these speakers in the extracted em- bedding. Similarly, attention between the Positive and Negative Enrollments embeddings enables the model to identify and exclude Negative Interferers, (i.e., speakers present only in the negative enrollments), from the output embedding. Finally, in Step 4, we truncate the output retain only the embeddings corresponding to the positive enrollment as the extracted target speaker embedding, which reduces the output a [T D] shaped embedding Ejsed This reduction in embedding size ensures the embedding feature is the same size as the pretrained encoder's output, allowing the model benefit from the fast convergence brought by training the encoder using knowledge distillation (as shown in Section 4.3) To further reduce the computation load in the extraction branch, we apply non-overlapping average pooling with 40 kernel size along the temporal dimension to Frused. resulting in a 170, DJ shaped feature Frused pooled- We discuss the effect of different kernel sizes on the model performance and inference time in Appendix D.
__pos: shape [T_pos, D]
E_neg: shape [T_eeg, D]
Learnable paransters:
Segnant Enheddings:
s_pea, $_neg: shape [1, D]
# N: 2 Full-band Self-attention layers
Encoder Fusion Module:
Step 1: Elenentvise add via broadcasting E_pos E_pas S_pos shape [T_pas, D] B_neg E_neg + S_neg shape [T_neg, D]]
Step 2: Concatenste along the temporal dimeasion concatt [E_pos, E_neg] shape [T_pos T_nog,
Stop 3: Apply two Pull-band Solf-attention layers connaatt M(F_cancat) shape [T_pan T_neg, D]
Step Truncate eubeddinge B_ fused E.concat [_concat[:T_poo]]] shape [7_pas, D]
Figure 3: Encoder Fusion Module pseudo-code.
5

---

Conditioned on the pooled enrollment encoder output Enused-pooled- we use a TF-GridNeet-B Extraction Branch to extract the larget speaker's voice from the Audio Mixture. To perform causal target speaker extraction, we modify the TF-GridNet block by adopting the same modification as LookOnceTooeea [16]. Three causal TF-GridNee blocks are used the Extraction Branch, Two cross-attention-bal based Extraction Fusion Blocks are added after the first two causal TF-GridNeet blocks to integrate the pooled target speaker embedding Efused-pooled into the extraction model. In each fusion module, the pooled target speaker embeddings poused-pooled serve as the Key and Value, while the output from the previous TF-GridNet block serves as the Query for the fusion module. The output from the last ee-GridNett block passes through transposed convolution layer and ISTFT module to generate the speech waveform.
In addition to the monaural model described above, which processes single-channel audio, we propose a binaural variant designed to encode and extract binaural audios. The difference between the monaural and binaural model architectures lies solely in the input. The binaural model takes in 4 channels input (stacked real and imaginary components from both channels) and the monaural model takes in only 2 channels (stacked real and imaginary component). By stacking the two channels' STFT, the binaural model can capture the inter-channel temporal differences between the two channels, which encode the directional information of the target speaker since the target speaker speaks at 90 degree azimuth angle in the enrollment. Both model architectures do not differ after the first convolution layer.
3.4 Training Method
Due to the additional speaker and background noise in the enrollments, the noisy Positive and Negative Enrollment pairs introduce significant variability in the training data in comparison to the clean enrollments. As shown in Section 4.3) when the Encoding and the Extraction branches are trained together using these noisy training samples, the model converges much more slowly and reaches worse final performance. Motivated by the prior works [16] 28 29] that successfully trained extraction models conditioned on the clean target speaker embeddings, we separately train the Encoding and Extraction branches in two-stage training strategy.
As shown in Figure 2 the first stage trains the Siamese Encoder and the Encoder Fusion Module from scratch, using knowledge distillation, extract the target speaker's s embedding from the noisy positive and negative audio enrollments. We obtain the ground-truth target speaker's embedding by passing the clean target speaker's 's voice through trained frozen TF-GridNet encoder [16]. Formally, the first stage training loss is written as
Lstage
Eglean
Ejused
|
(2)
where Eetean is the embedding of the clean target speaker voice aclem encoded by trained TF-GridNet encoder from [6].
In the second stage, we train the Extraction Branch to extract the target speaker from the Audio Mixture. The training loss for the second stage is the negative SNR value of the extracted audio åtgt and the ground-truth target speaker speech digt:
L Lsuge 2
SNR (ārgt; aegt).
More training and model details are discussed in Appendix C.
4 Experiment
4.1 Datasets and Baselines
Datasets We construct our Positive, Negative, and Mixed Audios from the samples in the Lib- riSpeech dataset (30], with background noise (PNNN from the WHAM! dataset [3I. Following the experiment configuration in LookOneeTTo [6]] we also construct binaural Positive / Negative / Mixed Audio samples by convolving the speech of each speaker with the binaural RIR data from
different datasets: CIPIC [32), RRBRIR [33), ASH-Listening-Set (34), and CATTRIR (35).

---

Table 1: Comparison between method and the baselines' performance when extracting from an audio mixture of 2-3 speakers conditioned on an enrollment mixture of 2-4 speakers. NMF's performance does not change the number of speakers in the Enrollments changes, as it performs unconditional sound separation. Differences in speaker similarity across combinations substantially impact extraction difficulty, leading the high standard deviation across samples.
2 Speakers Mixture
Speakers in Mixture
Method
Audio Type
Metric
Speakers
Speakers
Speakers Enroll
Speakers
Speakers
Speakers
SNRi
1.24
5.65+t 1.85
SI-SNRi
-1.651.378
-1 1.500=3..57
NMH
Mono
PESQ
STOI
0,36240.
1.55:0.32
0,296::0.1
1,2210.42
DNSMOS
1.5620.44
1.55-#0.41
WER
0.98+00.55
0.99+0002
SNRI
3.42 + 3.49
3:45 + 3.58
3.30 + 3.50
4.31 + 2.52
4.15 + 2.47
4,23+ 2.58
SLSKRI
.0.03 + 5.97
-0.03 6.42
-0.14 + 6.00
0:29 3.38
0.11
3,36
0.09 3.24
USEF-TFGGddeet
T
Mono
PESQ
STOI
0.43++00 0.17
0.43 0.18
1.54 0.49
0.17
0.33
0.35
1.32
0.37
0.11
0.36++001 0.1.1
DNSMOS
1.37
0.45
1.37 0.45
1.36
0.45
1,38 + 041
1:33
0.41
1.34 0.42
WER
0.66 à: 0.33
0.66 à: 0,36
0.68 = 0.32
0.85 4: 0.26
0.86 = 0.21
0,86 :: 0.20
SNRi
8.48:42.34
7.84122.56
7.52-22.68
8.474:2.37
8.57=2.34
8.07422.0
SESNNI
6.67.13.69
5.57144.47
4.77
5.13
TCE S
Mono
PESQ
STOI
0.682:001200
0.34
0.650:0.140
1.80
0.624-0.100
1.73 0.46
0.682::0.100
0.551=0.144
1.53 0.39
0.521
0.40
DNSMOS
1.5510.44
1.4400.4
183-10.44
1.85.10.4
1.65
1.6511.41
WER
0.73 0.16
0.76 0.24
0.77 0.24
0:225
0,88 = 0.21
0.88 0.17
SNRi
8.5342.30
8.391.2.37
8.37
9.03.12.23
9.00=2.31
8.89112.38
SI-SNRi
6,7013.43
6.45:3,74
6.31
6.25::3.35
6 6.06=3.68
5,761:3.95
Ours (Film fusion)
Mono
PESQ
STOI
0.68120.111
1.84 0.32
0.67510.121
L83 +: 0.34
0.665 127
1.84 0.33
0.58110.123
1,57 0.31
0.578 129
1.56 = 0.32
0,565 133
1.54
0.35
DNSMOS
1.9210.39
1.90100399
1.90:0.41
1.6910.38
1,68=0038
167110.38
WER
0.28
0.25
SNRi
10.14::2.57
9.93::2.71
9,79-2.86
10.42:2.49
10.37:22.64
10.22::2.79
SI-NNRI
8.85::3.67
8.54::.011
8.30=44447
8.42++3.62
8.29-
7.82:14..2
Ours (Monaural)
Mono
PESQ
STOI
0.758:0.107
2.07 1 0.34
0,749::0.121
2.06 1: 0,36
0.742:00.126
2,05 =: 0.36
0.668:20.125
L.79 1: 0.33
0.665=0.130
1.78 =: 0.34
0.648:0.116
1.75 0.37
DNSMOS
0.14:00.77
2.13::0.37
0.02:0.88
1.93::0.37
1.92-0.38
1.90:00.8
WER
0.35
0.43
0.28
0.61
In evaluation, 5000 samples are generated from the LibriSpeech dataset test-clean component using the same strategy as the training samples. We report the average improvement in SNR (SNRi) and SI-SNR (SI-SNRi), and the extracted audio's ' STOI and DNSMOS, along with the standard deviation across evaluation samples calculated using the built-in Pytorch function under the assumption that model performance metrics follow a normal distribution. We detail the training and testing data generation in Appendix B.
Baselines We compare our monaural model's performance with TCE 115, USEF-TFGridnet [17, and non-negative matrix factorization (NMF) method (36). We compare our binaural model's reverberant target speech extraction performance with the CookOnceToHeee model [16].
TCE [15] considers speakers whose voices do not overlap with a given speaker as the target speakers and extracts their voices. The model takes in a d-vector embedding of the given speaker and removes all interfering speakers who talk the same time as the given speaker.
LookOnceToHHear [66 is a binaural target speech extraction method conditioned on noisy audio enrollments. This method uses beamforming extract the target speaker's characteristics from the noisy Positive Enrollment, which contains audio from multiple additional speakers.
3. USEF-TFGridne [7] is target speech extraction model using clean target speaker enrollment as the extraction condition. We include this model as a baseline to show the influence of having interfering speakers in the audio enrollment on the model performance.
NMF (36] is a non-deep- mea-deep-learing--eer based audio separation technique. Since the method could not perform target speaker extraction conditioned on noisy audio examples, we report its extraction quality by selecting the audio with the highest SNR among its output as its extracted audio.
4.2 Result on Monaural and Binaural Reverberant Target Speech Extraction
We compare our model with baseline methods under scenarios where different numbers of speakers are present. Table shows the monaural target speaker extraction performance. USFF-TFGridnet 17 shows significantly worse performance, especially the number of interfering speaker increase
7

---

Table 2: Comparison between our method and the baselines' performance when extracting binaural audio from mixtures of 2-3 speakers conditioned on an enrollment mixture of 2-4 speakers.
2 Speakers in Mixture
Speakers in Mixture
Method
Andio Type
Metric
2 Speakees in Enroll
3 Speakers in Euroll
4 Speakers in Euroll
2 Speakees in Enroll
3 Speakers in Enroll
4 Speakers in Enroll
SNRi
9.36=
8.90=3.57
9.12+3.57
9.30=3.39
9.28++3.49
9.30+
SI-SNRi
5.75+5535
7.24++5.43
7:53+5535
6.58+5.37
6.49+5.62
6.51+5.64
LookOnce TaHear
Binaural
PESQ
STOI
0.136:0.152
2.28 0.47
0.723:0.162
2.25 + 0.52
0.732:0.153
2.28 0.45
1.87 + 0.48
0.620:0.175
0.610:0.180
188 0..4
0.611:0.187
1,90 0.47
DNSMOS
1.79+0.56
1,7740.55
1.76++0.55
1.59:00.47
1.00::0.48
1,58++0.99
WER
0.48 0.41
0.44 0.35
0.47 0.68
0.66 0.36
0.63 0.33
0.64 0.33
SNRi
9.84
9.601:3.57
9,5913363
9.811:328
9.78:13.23
9.83:3333
SI-SNRi
8.18:4481
5.84
5.75
6.76::5:27
6.72:15.08
6.71
Ours (Binaural)
Binaural
PESQ
STOI
0.735=0.1500
2.24 0.47
0.7291:0.155
2.22 0.47
0.720 1.59
2.25 0.47
0.60820.117
1.85
0.44
0.605::0.177
1.85 0.42
1.87 0.43
0.85:20.1822
DNSMOS
1.82:0.60
1,80=0.59
1.80=0.59
1.59=0.50
1.6020.49
1,59:0,48
0.48
0.45 0.34
0.35
0.33
0.64 0.43
0.64 0.42
in the enrollment. TCE [15] achieves better performance than USEF-TFGridnet [17 baseline since does not rely on the clean target speaker's 's enrollment. However, apart from the STOI metric in one scenario, the TCE model shows worse performance than our method under all different numbers of interfering speakers in the Audio Mixture and in enrollments.
In the multi-channel target speech extraction, prior work [16] has explored using beamforming to extract the target speaker's characteristics. In this experiment, we show that additionally including the Negative Enrollment helps improve model performance. As shown in Table 2 in comparison to the MookOnceTooHaa [16] baseline, based on two-sample t-tests over the 5000 test samples at the 90% confidence level, our model achieves statistically significant improvements in SNRi and SI-SNRi under all conditions. The worse STOI performance might be caused by the difference in parameter size, as shown in Appendix C. In contrast to the monaural experiment results, the binaural models show lower performance in most of the metrics and scenarios. This performance drop might be caused by the difference in training objective, since the binaural models need to predict the reverberant binaural audio, while the monaural models are trained to extract non-reverberant speech (to achieve fair comparison with the prior works focusing on non-reverberant extraction). The DNSMOS, PESQ, and WER metric are also affected by the reverberation effect in the prediction, thus providing less effective information for comparing model performance. We include them here only for completeness.
4.3 Ablation Study
Effectiveness of the two-stage model training In this section, we show the effectiveness of the two-stage train- ing by showing the validation loss curve of our model with and without the first training stage. As shown in Figure 4
SNK Valxiaation Los (dB))
end-to-end training reaches 3 dB SNR on the validation set after 600k optimization steps (around 125 hours), while
the combined training time for the two-staged training
takes 240k optimization steps (around 50 hours) based on a single Nvidia A10 24GB GPU.
With training
Wilhout bvo-olage touining
100
200
500
400
500
600k
Such a performance difference might be caused by the high difficulty in encoding the expected target speaker from the noisy enrollment pair. Since its very challenging for the model to draw similarity between the ground-truth single speaker audio and the noisy pair of enrollments input, training using an end-to-end learning method provides little guidance the encoder on which speaker is expected to be encoded. In contrast, the two-stage training explicitly guides the model to extract the target speaker's embedding the first stage, significantly accelerating convergence.
Opdimissation Starps
Figure 4: Validation loss values for the optimization step. The curve for the two- stage training begins at the 200k step to account for the 200k optimization steps performed during the first training stage.
Cross-Attention based Fusion over Film Fusion Method Film fusion is widely applied in the prior target audio extraction works (37 16). However, Film fusion passes the condition embedding
8

---

Spenkor'ss Sprech w.rdh Pusélive Interferet' a Spech
6-
Target Spealerr's is Speech W.r.t. Negative Interieren's Speech
SI.SNR (AR)
SLSNRI (dB)
0g
-5
-5
-10
-10
15
-16
Positive Interferer Sporch Duration Positlive Envallment
Negative Interferer Speech Duration the Negative Encolimens
Figure 4: Model performance w.r.t length of Positive Interferer in the Positive Enrollment.
Figure 5: Model performance w.r.f length of Negative Interferer in the Negative Enrollment.
Table 3: SI-SDRi of 10 Sample using Ground-Truie or Inaccurate User Enrollment Labelings.
Enrollment
3
6
7
9
Labeling Source
Ground Truth 10.98
7.60
8.35
11.71
10.31
12.31
11.01
8.00
11.81
10,90
7.60
8.40
11.65
10.23
12.14
10.82
7.32
8.04
11.15
User-Labelings + 0.04
+0.05
+0.02
+0.04
+0.07
00.12
+0,24
+1.32
+0.10
+2.10
through linear layer, and element-wise multiplies the output with the input tensor to perform fusion. This limits the model to use a fixed embedding size, which might not be capable of encoding the fine-grained details of the target speaker's voice characteristics. In this experiment, show that our attention-based fusion method is more optimal for our target speech extraction task.
We keep the rest of the model architecture intact and only change the model fusion method. In particular, we perform global average pooling on the encoder head output along the temporal dimension to obtain an embedding as the condition input to the Film fusion block. As shown in Table I. our attention-bane shows that the tesss-attention-basee fusion method is more preferable for the target speech extraction conditioned on noisy enrollments.
fusion method achieves higher performance all application scenarios. This
4.4 Model Performance in Challenging Application Scenarios
Model Performance under different PI and NI's speech length As the Positive Interferer (PI)'s speech duration in the Positive Enrollment increases, fewer frames are available for the model to distinguish the Target Speaker from the Similarly, reducing the Negative Interferer (NI)'s 's speech length in the Negative Enrollment results in less available conditional information to remove NI's voice characteristics in the target speaker encoding. In this section, we explicitly examine how the model performance varies with the duration of the PI and NI's speech in the enrollment.
We focus on the scenario where one PI and NI exist in the enrollments, and the Audio Mixture contains the same interferer as those in the enrollments. Let the length of the PI's speech in the Positive Enrollment be lpoe. and the NI's speech in the Negative Enrollment be Ineg- Figure we report the average model performance when extracting 2000 test samples with kneg randomly sampled between 1s to 3s and lpos takes different values between Os to 3s. We evaluate the extracted audio's s SI-SNRi with respect to the target speaker and the Pl's speech in the Audio Mixture. When :pos = 3, the model fails distinguish the two speakers as expected, since both speakers have the same speech length in each enrollment. As pos decreases below 2.7s, the model correctly identifies the target speaker's characteristic, as shown by the increasing SI-SNRi of the extracted audio w.r.t. the target speaker's speech and the decreasing SI-SNRi w.r.t. the Pl's speech. Alternatively, we randomly sample lpos between 1s to 3s and evaluate the model performance when Ineg takes value between Os to 3s. As shown in Figure 5, the model correctly extracts the target speaker and removes the NI's speech when kneg increases above 0.3s. These experiments demonstrate that the model indeed leverages the diffferences in speakers' speech length in both enrollments to encode the target speaker, and could distinguish speakers from as short as 0.3 seconds of misalignment between their speech.
Model performance under inaccurate Positive and Negative Labeling In the real-worls appli- cation scenario, inaccurate Positive and Negative Enrollment labeling from the users could lead to

---

unwanted inclusion of the target speaker's voice in the Negative Enrollments, or including timesteps when the target speaker remains silent in the Positive Enrollment. Both cases violate our model input assumption. In this section, we demonstrate our model's tolerance to the inaccuracy in the user-provided Enrollments' labeling.
We randomly select 2-3 speakers and overlay their voices so that each speaker spans 5-7 seconds of a 10-second audio mixture. For 10 such randomly generated samples, we instruct 10 users to label the start and stop timesteps of a certain speaker following a speaker hint, e.g, the same speaker heard in another clean audio example. The segments not labeled by the user are considered negative enrollments. Using the enrollment labeling provided by users, we perform extraction on another audio mixture containing the target speaker. We report the SI-SNRi of each sample across different users' labeling, as well as the SI-SNRi if the ground truth labeling is used. As shown in Table 3 our model shows little performance degradation when using the inaccurate positive enrollment labeling provided by the user, despite each start and stop timestep of the positive enrollment being on average 0.30 0. 0.19 seconds away from the ground truth timestep. In addition, the model shows small variation in metric value across different users' labeling. This validates our model's robustness to the inaccuracy in different users' labeling.
Model Performance on Real-Worlh Audio Mixtures In this section, we compare our model and the baseline methods on naturally recorded real-world audio mixtures. We evaluate the Mean Opinion Score (MOS) of differ- ent models using five naturally recorded audio mixtures sourced from Freesound.ov and the VoxConverse dataset. These sounds are noisy real-world speech recordings cap- tured in pubs, metro stations, urban areas, and city coun- cil meetings. For the evaluation, we manually labelled the Positive and Negative Enrollment and extracted target speech from Audio Mixture clips taken from the same recording. The duration of the enrollment and Audio Mix- ture clips range from 3 to 8 seconds.
Table 4: SI-SDRi under Inaccurate User labelings.
Method
MOS
Ours (Monaural)
3.35
Ours (FiLM Fusion)
2.60
USFF-TFGridnet
1.45
Unprocessed
2.10
10 participants are asked rate the original (unprocessed) Audio Mixture, and the audio extracted by three models. Given a textual description the target speaker (c.g., the female speaker talking over the crowd), they rated the clarity of the target speech relative background noise 1 1-4 scale. As shown in Table 4 our model achieved a MOS 3.35, outperforming all baseline methods. These results demonstrate the superiority and practical applicability of our model in real-world target speech extraction scenarios.
5 Conclusion and Limitation
In this paper, we present a method for performing target speech extraction conditioned on noisy enrollments where interfering speakers' speech overlaps significantly with the target speaker. We optimize our model encode the target speaker from easily obtained noisy positive and negative enrollments, without using any clean audio enrollment input. Experiments show that our method achieves SOTA performance under the challenging and realistic application scenario.
Though the proposed two-stage training strategy for the encoding branch significantly accelerates convergence, may limit the encoder's 's performance that of the frozen encoder used for knowledge distillation. Further work on encoding branch training is required to remove such potential perfor- mance limitations while maintaining the high convergence speed. In addition, our model extraction still includes artifacts noticeable to the human ear, as shown by the low DNSMOS score reported. This hinders the model's applicability to downstream tasks such as audio recognition. Further work is required to reduce the artifacts, for example, by employing more advanced extraction branch architectures. Finally, although our model can flexibly perform extraction using clean or noisy audio enrollments, it does not outperform SOTA models explicitly trained to extract conditioned on clean enrollments. Further work required develop models with strong generalizability across both enrollment strategies.
10

---

References
[1] Zexu Pan, Ruijic Tao, Chenglin Xu, and Haizhou Li. Selective listening by synchronizing speech with lips, 2022.
[2] Zexu Pan, Ruijie Tao, Chenglin Xu, and Haizhou Li. extraction with visual cues, 2021.
Muse: Multi-modal target speaker
[3] Hao Ma, Zhiyuan Peng. Xu Li, Mingjie Shao, Xixin Wu, and Ju Liu. Clapsep: Leveraging contrastive pre-trained model for multi-modal
target sound extraction, 2024.
[4] Xiang Hao, Jibin Wu, Jianwei Yu, Chenglin Xu, and Kay Chen Tan. Typing to listen at the cocktail party: Text- -uideg target speaker extraction, 2024.
[5] Katerina Zmoliková, Mare Delcroix, Keisuke Kinoshita, Tsubasa Ochiai, Tomohiro Nakatani, Lukáš Burget, and Jan Černocký. Speakerbeam: Speaker aware neural network for target speaker extraction in speech mixtures. IEEE Journal of Selected Topics in Signal Processing, 13(4):800-814 2019.
[6] Chenglin Xu, Wei Rao, Eng Siong Chng, and Haizhou Li. Spex: Multi-scale time domain speaker extraction network. IEEF/ACM Transactions on Audio, Speech, and Language Process- ing. 28:1370-1388 2020.
[7] Meng Ge, Chenglin Xu, Longbiao Wang, Eng Siong Chng. Jianwu Dang, and Haizhou Li. Spex+: A complete time domain speaker extraction network, 2020.
[8] Ke Zhang, Junjie Li, Shuai Wang, Yangjie Wei, Yi Wang, Yannan Wang, and Haizhou Li. Multi-level speaker representation for target speaker extraction, 2024.
[9] Shulin He, Huaiwen Zhang, Wei Rao, Kanghao Zhang, Yukai Ju, Yang Yang, and Xueliang Zhang, Hierarchical speaker representation for target speaker extraction, 2024.
[10] Hanyu Meng, Qiquan Zhang, Xiangyu Zhang, Vidhyasaharan Sethu, and Eliathamby Am- bikairajah. Binaural selective attention model for target speaker extraction, 2024.
[11) He Zhao, Hangting Chen, Jianwei Yu, and Yuehai Wang. Continuous target speech extraction: Enhancing personalized diarization and extraction on complex recordings, 2024.
[12] The Hieu Pham, Phuong Thanh Tran Nguyen, Xuan Tho Nguyen, Tan Dat Nguyen, and Duc Dung Nguyen. Wanna hear your voice: Adaptive, effective, and language-agnostic approach in voice extraction, 2024.
[13] Marc Delcroix, Katerina Zmolikova, Tsubasa Ochiai, Keisuke Kinoshita, and Tomohiro Nakatani. Speaker activity driven neural speech extraction, 2021.
[14] Yiru Zhang, Linyu Yao, and Qun Yang. Or-tse: An overlap-robust speaker encoder for target speech extraction. pages 587-591, 09 2024.
[15] Tuochao Chen, Qirui Wang, Bohan Wu, Malek Itani, Emre Sefik Eskimez, Takuya Yoshioka, and Shyamnath Gollakota. Target conversation extraction: Source separation using turn-taking dynamics. In Interspeech 2024, page 3550-3554. ISCA, September 2024.
[16] Bandhav Veluri, Malek Itani, Tuochao Chen, Takuya Yoshioka, and Shyamnath Gollakota. Look once hear: Target speech hearing with noisy examples. In Proceedings of the CHll Conference on Human Factors in Computing Systems, page 1-16. ACM, May 2024.
[17] Bang Zeng and Ming Li. Usef-tse: Universal speaker embedding free target speaker extraction, 2024.
[18] Ashutosh Pandey, Sanha Lee, Juan Azcarreta, Daniel Wong, and Buye Xu. All neural low- latency directional speech extraction, 2024.
[19] Rajeev Rikhye, Quan Wang, Qiao Liang, Yanzhang He, and McGraw, Multi-user voicefilter- lite via attentive speaker embedding, 2021.

---

[20] Bang Zeng, Hongbing Suo, Yulong Wan, and Ming Li. Simultaneous speech extraction for multiple target speakers under the meeting scenarios, 2023.
[21] Yun Liu, Xuechen Liu, Xiaoxiao Miao, and Junichi Yamagishi. Target speaker extraction with curriculum learning, 2024.
[22] Shivesh Ranjan and John H. I.. Hansen. Curriculum learning based approaches for noise robust speaker recognition. JEEE/ACM Transactions on Audio, Speech, and Language Processing. 26(I):197-210, 2018.
[23] Woon-Haeng Heo, Joongyu Maeng, Yoseb Kang, and Namhyun Cho. Centroid estimation with transformer-based speaker embedder for robust target speaker extraction. pages 4333-4337, 09 2024.
[24] Ashutosh Pandey and DeLiang Wang. Attentive training: A new training framework for speech enhancement. IEEE/ACM Transactions Audio, Speech, and Language Processing, 31:1360-1370, 2023.
[25] Marvin Borsdorf, Zexu Pan, Haizhou Li, and Tanja Schultz. wTIMIT2mix: A Cocktail Party Mixtures Database to Study Target Speaker Extraction for Normal and Whispered Speech. In Proceedings of the 25th Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 5038-5042, 2024.
[26] Zhaoxi Mu, Xinyu Yang, Sining Sun, and Qing Yang. Self-supervised disentangled representa- tion learning for robust target speech extraction, 2024.
[27] Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong- Yeol Kim, and Shinji Watanabe. Tf-gridnet: Making frequeer domain models great again for monaural speaker separation, 2023.
[28] Woon-Haeng Heo, Joongyu Maeng, Yoseb Kang, and Namhyun Cho. Centroid estimation with transformer-based speaker embedder for robust target speaker extraction. pages 4333-4337, 09 2024.
[29] Quan Wang, Hannah Muckenhirn, Kevin Wilson, Prashant Sridhar, Zelin Wu, John Hershey, Rif A. Saurous, Ron J. Weiss, Ye Jia, and Ignacio Lopez Moreno. Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking, 2019.
[30] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210, 2015.
[31] Gordon Wichern, Joe Antognini. Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. Wham!: Extending speech separation to noisy environments. In Proc. Interspeech, September 2019.
[32] V.R. Algazi, R.O. Duda, D.M. Thompson, and C. Avendano. The cipic hrtf database. In Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01TH8575), pages 902102, 2001.
[33] IoSR-Surrey. i Binaural impulse responses captured in real rooms. 2016. (2016).
[34] Shanon Pearce. haapmmpparcc/ass--sseennnneegeee A dataset of filters for headphone correction and binaural synthesis of spatial audio systems on headphones, 2022, (2022).
[35] IoSR-Surey. Simulated room impulse responses. http:///iooo...utttttatrrrreaeee 2023. (2023).
[36] S. Abdali and B. NaserSharif. nen-negative matrix factorization for speech/music separation using source dependent decomposition rank, temporal continuity term and filtering. Biomedical Signal Processing and Control, 36: 167-175, 2017.
[37] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville, Film: Visual reasoning with general conditioning layer, 2017.
12

---

[38] David Wiseman. py-webrtevad: Python interface to the google webrtc voice activity detector (vad), 2018.
[39] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers, 2021.
[40] Zifeng Zhao, Dongchao Yang, Rongzhi Gu, Haoran Zhang, and Yuexian Zou. Target confusion in end-to-end speaker extraction: Analysis and approaches, 2022.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly describe the main contributions of the paper, including using noisy positive and negative enrollments for target speaker extraction, a novel model architecture, and two-stage training method. These claims are well supported by the experiments and analysis in the main text, and the paper also discusses assumptions and limitations, matching the stated scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses several limitations in the main experimental results section and the final "Conclusion and Limitation" section. It acknowledges challenges such as performance ceiling posed by the proposed two-stage training strategy, the gap compared to models trained with clean enrollments, and the difficulty predicting reverberant audio. also notes potential issues like artifacts in extracted speech and the need for better encoding architectures. Additionally, the paper reflects on the assumption that speakers do not start and stop speaking in perfect synchrony, and empirically validates this assumption in the "Model Performance in Challenging Application Scenarios" section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ter system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
14

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
The paper does not include any formal theoretical results, theorems, or proofs. While it introduces an assumption about speaker temporal dynamics (i.c., that speakers do not start and stop speaking in perfect synchrony), this is empirically validated on the real-world datasets Appendix A rather than formally proven. Therefore, this question is not applicable.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides sufficient details to reproduce the main experimental results. It describes the dataset construction process using LibriSpeech and WHAM!, outlines the model architecture clearly, and explains the training procedure, including the two-stage training strategy. Hyperparameters, evaluation metrics, and comparison baselines are also specified. Additionally, the paper mentions that code, model checkpoints, pretrained models, and usage instructions (readme file) are included in the supplementary material, supporting reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
15

---

reproducible, they While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code, model checkpoints, and the instructions for reproducing the result are given in the supplementary file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https www.eoooweeeeeeoooooooooooo
/ //nips. //nips.cc/pul
ssi.iioliooyyyyic for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides detailed descriptions of the training and testing setup. It specifies the datasets used, the construction of positive, negative, and mixed audio samples, and the baseline models for comparison. Also, some specifics (e.g., hyperparameters, optimizer details) are deferred to the appendix B, C. We use the official train/validationnnestt of the LibriSpeech dataset to construct different dataset splits.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports standard deviations alongside the main evaluation metrics in the result tables, indicating variability across test samples. It also states that the metrics are assumed to follow a normal distribution and that the standard deviations are computed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative етror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies that training was conducted using single Nvidia A10 24GB GPU and provides concrete information about training time and optimization steps in Appendix C. This level of detail is sufficient estimate the computational requirements for reproducing the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
17

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://ttws.aaa.p....o.ooo////aa
Answer: [Yes]
Justification: Our training samples involve only publicly available datasets, with personal identifiable information. We designed and performed our user study so that fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed the social impact of the work in the main paper. Our paper focuses on improving making it easier for users to focus on desired voices in complex auditory scenes, an application scenario already widely used. No social impact feel should be explicitly addressed here.
Justification: We discuss both potential positive and negative societal impacts the Ap- pendix. Our work aims to help users better focus on desired voices in complex auditory scenes, which an application scenario already widely used. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA].
Question:
18

---

Justification:
The paper does not release models or datasets that are clearly high risk for misuse. It uses publicly available datasets and does not involve sensitive or scraped data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper uses publicly available datasets and properly cites the original sources. It also credits the TF-GridNeet model as the architectural backbone with appropriate references. While the paper does explicitly mention license types URLs, the cited resources are standard in the community, and their terms of use are widely known. There is indication of improper use lack of attribution.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, wwwppsaittchooeoooopppp thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces new model and training pipeline for target speaker extraction and provides code, pretrained checkpoints, and instructions in the supplementary material. While does not create new dataset, the documentation appears sufficient for users to understand and apply the new assets. The assets are anonymized for submission, aligning with NeurlPS guidelines.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
19

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We performed the experiment 10 more human volunteers verify the feasibility of obtaining the Positive and Negative Enrollments. The experiment details and the instruction given the human participants given in the Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [Yes]
Justification: We have obtained the approval from the institution where all the authors are from.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is used solely to refine wording for clarity in this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy https:://neurpss cttp.c//eeeeeeeeee/22/////////// for what should or should not be described.
20

---

The appendix is structured as follow: Section A quantitatively verifies the feasibility of encoding the target speaker using the Positive and Negative Enrollments pair. Section B-D provide training details and justification for the hyperparameter selection. Section E-I present model performance under different numbers and speech lengths of interfering speakers. Section J-L. reports the model's performance when clean audio enrollments are available. Section M show model's generalizability to speech data from other datasets. Seciton N shows the successful and failure cases. Section O states the societal impact of the work.
Section Quaantitative Evaluation on the Overlapping Ratio between the Target Speaker and the Interfering Speakers
Section B Data Simulation Method
Section C Moddll Architecture and Training Detail
Section D :Embedding Pooling Size
Section E :Affect of Increasing Number of Positive and Negative Interferers on Model Performance
Section :Model Performance under Different Lengths of Positive and Negative Enrollment
Section G :Model Performance with respect to the Input Audio Quality
Section H :Model Performance under presence of Hybrid Interferers
Section I Model Performance under presence of Neglect-Required Interferers
Section J :Model Performance for Target speaker Confusion Problem
Section K :Model Performance when Clean Target Speaker's Enrollment is available
Section L Model performance WSJO Dataset
Section M :Audio Visualization and Failure Cases
Section N :Societal Impact
A Quantitative Evaluation on the Overlapping Ratio between the Target Speaker and the Interfering Speakers
As shown in Section 3.2, we base our problem formulation on the assumption that speakers will not always start and stop talking at the same time. In this section, we show such temporal misalignment between speakers in real-world noisy audio mixtures is prevalent, which allows our model to identify the target speaker from the noisy Positive and Negative Enrollment.
We validate the assumption using the MSDWild dataset and the VoxConverse dataset. Both datasets are activity detection datasets consisting of real-world conversational audios, and provide ground truth activity labeling indicating which speaker speaks when in each conversation. To investigate the over- lapping ratio between speakers from different conversations groups, we perform pairwise comparison between conversation samples from each dataset. Specifically, for pair of conversation containing speakers (#a1, 802, {am) } and (8b1, 862; ..., 8bn respectively, we consider all the speaker pairs from different conversation groups {(Soi sby)l E [1..m].] E [1..n]] }. and calculate the mean Inter- section Over Union (IOU) for the speech of these speaker pairs _i IoU (Sai, 8b;).
The mean IOU and the standard deviation of IOU across different samples is 0.25 + 0.20 for the many-talker val split of the MSDWild dataset, and 0.22 0.06 for the test split of VoxConverse dataset. This indicates that, on average, the non-overlapping speech between any two independent speakers spans more than half of their combined speech segments. These non-overlapped regions offer rich information for our Positive and Negative Enrollment strategy, enabling the system to identify the target speaker and perform accurate speaker extraction. Note that the overllapgg regions are defined with respect to pairs of speakers. In scenes with multiple interfering speakers, different interferers may overlap with the target speaker different times, resulting mixture where no single timestep contains only one active speaker. However, by obtaining pairwise speaker difference from the non-overlapping speech between two speakers, Positive and Negative Enrollment pair still provide sufficient information to distinguish the target speaker from interfering speakers.
21

---

Table 5: Speech length of different types of speakers' speech in enrollments when 3 seconds of Positive and 3 seconds of Negative Enrollments are used.
Speaker Type
Length in Positive Enrollment
Length in Negative Enrollment
Target Speaker
seconds
0 seconds
Positive Interferer
1-2 seconds
seconds
Negative Interferer
seconds
1-3 seconds
Hybrid Interferer
1-2 seconds
1-3 seconds
Deglect-Requines Interferer
scconds
1-3 seconds
B Data Simulation Method
In this section, we detail the method for simulating the training and testing audio.
As shown in Section 3.2 in the main paper, based on the presence/absence of the interfering speaker in all/subset of the segments the enrollments, the interfering speakers are classified into types. Table summarizes the length of each type of speaker in the Enrollments. Target Speaker speaks throughout the Positive Enrollment and does not exist in the Negative Enrollment. Negative Interfer- ers' speech fully overlap with the target speaker in the Positive Enrollment and exist in the Negative Enrollment. Alternatively, Positive Interferers' speech does not exist in the Negative Enrollment, and exist in a subset of segments in the Positive Enrollment. Hybrid Interferer exist in subset of the Positive Enrollment, and also exist in the Negative Enrollment. Reglect-Requice exist only in the Negative Enrollment and are excluded from the Positive Enrollment. In order to ensure each speaker's s voice span the desired length of the Enrollment, we use WebRTC Voice Activity Detector (38] to detect and remove zeros in the LibriSpeech dataset samples, before repeating or cutting to construct their voice of desired length in the Audio Mixture, the Positive Enrollment and the Negative Enrollment.
Interferers
We generate our training, validation, and testing data from the train-clean 360, dev-clean, and test-clean components from the LibriSpeech dataset (30), respectively. The sampling rate is 16000. We generate the training samples so that the enrollments and the Audio Mixture all have one target speaker and two interfering speakers. The target speaker is shared between the enrollments and the Audio Mixture, while the interfering speakers in the Audio Mixtures are two randomly sampled speakers different from the target speaker, without being required to be the same those in the enrollments. To reduce the instability caused by the variation of different speaker types, the Hybrid and Peglect-RRquired Interferers are not included in the enrollments in the training samples, so the two interfering speakers are randomly designated as either the Positive Interferer or the Negative Interferer. We enforce the target and interfering speakers to talk throughout the Audio Mixture to increase the extraction difficulty. All training samples are generated on the fly to enhance variability and diversity during training. The testing data generation follow the same strategy as the training samples. All the Audio Mixtures in the test samples are 6 seconds long.
To simulate the background environment noise, we use the WHAM! [1 noise dataset. Different noise samples from the WHAM! noise dataset are recorded in different environments. As a result, additional interfering sounds might exist in one WHAM! noise example but not the other. For instance, some WHAM! noise dataset samples contain music playing in the background while others don't. tt Using WHAM! noise sample containing music in the Positive Enrollment and WHAM! sample of ambient environment noise in the Negative Enrollment will confuse our model on whether the music is the target sound to be extracted. We resolve such ambiguity by sampling the n (M,PP.N from the same WHAM! noise sample but at different temporal segments. Selecting random segments from the WHAM! noise prevents the model from assuming the background noise is the same across the Positive, Negative, and Mixed Audio. We scaled the WHAM! noise to be between 2-2.5, 2.5] SNR with respect to the ground truth target speaker's 's voice in each sample.
To simulate the binaural reverberant training and testing samples, we follow the same data simulation method as the Torr [16]. In particular, we convolve each speaker's voice with BRIR from four binaural room impulse response datasets 32
33 35 The binaural RIRs used for constructing one data sample are randomly selected such that All BRIR used in single data
sample generation are from the same scene, and 2. The BRIR for the target speaker in the Positive
22

---

Table 6: Comparison of model parameter size, inference time, and memory usage when extracting 1 --sccnnd audio on an Intel Xeon Silver 4314 @ 2.40GHz CPU. We report the average inference time for extraction 50 repeated experiments. Replacing the Film fusion module with the cross-attention based fusion module in our final model architecture could significantly reduce the parameter size.
Model
Param Size
Inference Time
Inference Memory Usage
Ours (cross attn)
1.88 M
0.36s
2.30 GB
Ours (Film fusion)
3,94
0.3s
2.31 GB
TCE
2.54 M
0.1 Is
1.89 GB
LooeOnce
4.41 M
0.35s
2.32 GB
Enrollment has the direction of arrival of 0 degrees. The re-processsing strategy for each speaker's voice and the generation of background noise remains the same as the monaural dataset.
C Model Architecture and Training Detail
The TF-GridNet [27] Encoder in the Encoding Branch uses the following configuration x 4 kernel size and x stride in the first Conv2D, 64 hidden units in all three BiLSTM layers, 8 attention head numbers in the Full-band Self-attention Module. The input audio is processed by Short-Time Fourier Transform (STFT) with 128 window size and 64 hop length.
To perform causal inference in the extraction branch, we follow the same modification as in the LookOnce [16]. In particular, remove the global layer normalization after the first convolution layer, change the BiLSTM in the TF-GridNet blocks to unidirectional LSTM, and constrain the Full-band Self-attention Modules to calculate the causal attention value of one-time frame with only frames before it. The parameter configurations of causal TF-GridNet blocks in the Extraction Branch are the same as the Encoding Branch, apart from using 1 x kernel size in its first 2Dnv2D layer. Three causal TF-GridNet blocks are used in the Extraction Branch. The output from the last TF- GridNet block passes through a transposed convolution layer and an ISTFT module to generate the speech waveform.
All the training is done on single Nvidia A10 24GB GPU with batch size of 2. In both training stages, we use the Adam optimizer and decay the learning rate by half when the validation loss does not decrease for more than 50 epochs. Motivated by MOCOv3 [39], which observes that changes in the parameters of shallower layers can lead to instability in the loss, we assign lower learning rates to the earlier layers of the model and higher learning rates the deeper layers. In particular, the initial learning rate is set to 5e-4 for the whole Siamese Encoder, le-3 for the Encoder Fusion Module in the first training stage, and 2e-3 for the whole extraction branch in the second training stage. 500 epochs (200k optimization steps) are used in the first pretraining stage, and 1000 epochs (400k optimization steps) are used in the second stage.
Since TCE is not trained on the LibriSpeech dataset, we fine-tune the TCE model on our simulated data till convergence (for 120k optimization steps) using the Adam optimizer with 5e-4 learning rate. To train and test the TCE model on the proposed extraction task, we modify our dataloader's ss output to match TCE's application scenario by adding a known speaker's 's voice in the Negative Enrollment and concatenating the Mixed Audio with this modified Negative Enrollment. The model then performs extraction conditioned on the known speaker's d-vector embedding.
In addition, despite the LookOnceToHers model is trained on audio examples from the LibriSpeech dataset, it uses Scaper toolkit to load and generate audio mixtures. We notice this leads to slightly different audio input to the model if the audio is otherwise loaded by torchaudio. and mixed by addition. The distribution shift in input audio leads to noticable difference of around SI-SNRi decrease in performance. For fair comparison, we tune the MookOnneTooHarl extraction model on our training data with an initial learning rate set to 5e-4 and decay by half if the validation error does not decrease for more than 50 epochs. The model performance converges after 300 epochs fine-tuning (120k optimization steps) when the Ir decreased to around 6e-5.
We load the USER-FFGridne model from the open-source project https///tthab.com/////nn///11 TSE.
23

---

Table 7: Model performance with different pooling sizes. The inference time the average time taken for the model main branch to extract 1-second audio in 50 repeated experiments on an Intel Xeon Silver 4314 @ 2.40GHz CPU.
2 Speakers in Mixture
3 Speakers Mixture
Inf.
Pooling Size
Metric
Speakers in Enroll
3 Speakers in Enroll
4 Speakers Enroll
Speakers Emroll
Speakers in Enroll
Speakers in Enroll
Time
SNRi SI-SNRi
9,5742261
9.35-42.67
9.27+2279
9.96-22.56
9.93+2.57
9.87422.6
8.10:33.90
7.82444033
7.7244440
7.80:13.99
7.60:-4.12
7.35-44.46
0.365
STOI
0.736:10.120
0.728:10.11 125
0.713:10.199
0.641.10.133
0.637.10.131
0.627:10.140
DNSMOS
2.02+0042
2.00+0042
1.78++0.4
2.00--0043
1.78++0.42
1.76++0.41
SNRi
10.14:2.57
9.93=22.71
9.79-12.86
10.42:22.49
10.37=2.64
10.22:22799
40
SI-SNRI
8.85:13.67
8.54:14.01
8.30:14.477
8.42::3.62
8,29:1-.006
7.82:1-4.62
0.36
STOI
0.758:1.107
0.749:10112
0.122
0.668 1.112
0,665 : 0.130
0.648 i 0,146
DNSMOS
2.14:10.37
2.13:10.37
2.02:10.38
1.93:0037
1.92:10.38
1.90:10.38
SNRi 9.93:22.55
9.75422.55
9.71+:2.74
10.18:2.49
10,20=2.59
10.10=2.72
SI-SNRI
8.59-43.77
8.28+44.22
8.22-44.38
8.02+33822
8.00++4.15
7. 7.66-4-4.69
0.33
STOI 0,749+0.110 109
0,742 122
0.739:00124 124
0,654:0.11 128
0,654 1333
0,642 144
DNSMOS
2.09:10.4
2.0620.41
1.95:0.44
1.861:040
1.8610.4
1.84:10:40
Table 8: Model performance under different number of Positive Interferers (PI).
2 Speakers Mixture
Speakers in Mixture
Method
Metric
1
PT
3
PT
2
PT
SNRi
10.23:12.56
10. 10.10.12.58
10.05:12.6
10.46:12.43
10.47-2.54
10.49-12.63
Ours
SI-SNRi
8.98133.6
8.87:13.60
8.7913.76
8.5313.56
8.53::.76
8.44:14.09
(Monaural)
STOI
0.763++0.10
0.759:0.106
0.758+0.105
0.673:0.119
0.673=+0.122
0.667=00129
DNSMOS
2.15+0037
2.14+0.36
2.14+0037
1.93+0.37
1.93+0038
1.93+00.77
In Table 6 we compare our model architecture's parameter size, extraction branch's 's inference time, and memory usage inference, with the two prior works on target speaker extraction using noisy enrollments. Our model uses smaller parameter size. In particular, we notice that the Film fusion module substantially increases the parameter size, as shown by the significant increase in model parameter size when we replace our cross-attention fusion module with the Film fusion module.
D Embedding Pooling Size
To reduce the model computation time in the Extraction Branch, we perform an average pooling of 40 on the embedding extracted by the encoder head. In this section, we analyze the trade-offs between model performance and inference time under different pooling configurations. Specifically, compare model (40 pooling size) with two variants using pooling sizes of 20 and 80, as shown in Table
As the pooling size increases, the length of the extracted target embedding sequence decreases, which reduces the inference time for the extraction branch. Larger pooling size also results in more information loss, which leads to worse extraction performance. To our surprise, smaller pooling size also results in a worse performance, despite it retains more detailed information of the target speaker in the embedding sequence. We hypothesize that smaller pooling size introduces instability in model training, as the resulting target speaker embedding sequence has larger variation across different timesteps. The unstable model learning could lead to worse final model performance. As result, we selected 40 pooling sizes as the final model configuration.
E Affect of Increasing Number of Performance
Positive and Negative Interferers on Model
In Section 4.2 in the main paper, we randomly assign the interferers in the enrollments as Positive or Negative Interferers, and show the model performance as the number of interferers increase. this section, we separatedly evaluate how Positive and Negative Interferers will affect model performance. In the first experiment, we include only Positive Interferers the enrollments and gradually increase their number. In the second experiment, we follow the same procedure but only include Negative
24

---

Table 9: Model performance under different number of Negative Interferers (NI).
2 Speakers in Mixture
3 Speakers in Mixture
Method
Metric
1 II
NI
3
N
N
N
SNRi
10.07=2.70
9.724:2.85
9.334:3.1
10.32::2.55
10.19-2.73
9.93
Ours
SI-SNRi
8.69:14.06
8.19:44.48
7.43:55399
8.23:t4.02
7.89:4.59
7.21
(Monaural)
STOI
0.754::0.118
0.739+00.19
0.717+001122
0.664:0.129
0.653:+0.122 142
0.630=00161
DNSMOS
2.13+-038
2.10+0.39
2.07+00.0
1.91+0.38
1.89++0.39
1.87+0.39
Table 10: Model performance under different monaural enrollment lengths. We vary the conditional Positive and Negative Enrollment length between to 10 seconds.
Condition
Metric
Pos. sec
Pos. 3 sec
Pos. 5 sec Pos. 10 sec
SNRi 9.6412.66
10.22::2.53
10.31=2.51
10.43:22.43
SI-SNRi
7.07:t4.45
8.161:3.94
8.33
8.54
Neg. sec
STOI
0.626:10.1 142
0.662:10.122
0.66810.121
0.67420.117
DNSMOS
1.86:10.38
1.93
1.941:.37
1.95
SNRi 9.68:22.66
10.37:t2.64
10.38=2.56
10.47+2.40
SI-SNRi
7.16:L4.38
8.29:14.06
8.39:1..3
8.63+3.42
Neg. sec
STOI
0.629 140
0.66510.130
0.67110.122
0.677=0.114
DNSMOS
1.87:L0.39
1.92:10338
1.94:1.37
1.9710.36
SNRi 9.71+22.70
10.26:#2.63
10.21=2.54
10.49:2.41
SI-SNRi
7.16114.64
8.1744.13
8.4543.64
8.66
Neg. 5 sec
STOI
0,629:10.133
0.662:10.128
0.67110.118
0.677+0.115
DNSMOS
1.87:10.39
1.93
1.9410.37
1.96
SNRi 9.74+2..70
10.22::2.59
10.41=2.43
10.5042.36
S1-SNRi
7.28:L4.35
8.1114.06
8.54
8.71
Neg. 10 sec
STOI 0.632.10.136
0.66010.126
0.67410.114
0.679=0.110
DNSMOS
1.88
1.94.10.37
1.95:10.36
1.9611.36
Interferers in enrollments. In both cases, we evaluate the model's performance when extracting a target speaker from Audio Mixtures containing 2-3 speakers.
As shown in Table 8 and Table 9 our model shows only 0.2 dB SI-SNRi decrease performance the number of Positive Interferer increases from to 3. In comparison, when the number of Negative Interferers increase, the model shows more significant performance decrease of over dB SI-SNRi. This suggests that, in comparison removing interfering speakers' characteristic given in the Negative Enrollment, our model more capable of identifying the target speaker's s' characteristic, which is common voice characteristic that exists across different segments in the Positive Enrollment.
F Model Performance under Different Lengths of Positive and Negative Enrollment
We train our model solely on samples with seconds of Positive and Negative Enrollments, and tested the model performance solely on these scenarios. In this section, we test the model performance on 10 seconds. We keep the ratio of the Positive Interferer
different enrollment lengths between in the Positive Enrollment, and the ratio of the Negative Interferer in the Negative Enrollment unchanged. This means the Positive Enrollment will still span 1/3 to 2/3 of the Positive Enrollment, and Negative Interferer span the 1/3 to full length of the Negative Enrollment. As shown in Table 10 the model performance improves in general as both the length the Positive Enrollment and the Negative Enrollments increase. We also notice that increasing the Negative Enrollment length when less than seconds of the Positive Enrollment is provided could adversely affect the model performance. This might be because the model has too little information of the target speaker
25

---

7.5
Qutpit 9 SKA 6080
SI-SNRI (dB)
5
-20
No Finn-trriing
-15
Fine-tuned on -15 S1-SNRA
Fine-tuged -20 SI-SNR
-20
0
25
20
15
-10
0
Pusitive Enrollment SLSNN (dB)
34
Irput M Audio S1SM (95)
Figure 7: SI-SNRi of the extracted audio with
Figure SI-SNR of the extracted audio with respect to the Positive Enrollment SI-SNR. respect to the Mixed Audio. The dashed black line represents the zero-improvement line.
Table 11: Model performance under different number of Hybrid Interferers (HI)
Speakers Mixture
Speakers in Mixture
Method
Metric
III
III
2
SNRi SI-SNRi
10.23+22.55
10.06++2.59
10.05++2.63
10.47+22.43
10.49-2.52
10.53-+2.60
Ours
8.99.13.63
8.82.13.61
8.831.3.58
8.54.13.56
8.57.13.7
8.53113.92
(Monaural)
STOI
0.763.10.105
0.757.10.108
0.75910.103
0.67310.119
0.674.10.119
0.669110.126
DNSMOS
2.15:0037
2.14:0.36
2.14:0037
1.93:0.37
1.93-0.38
1.93:10.37
form the Positive Enrollment, while the Negative Enrollment might guides the model to remove excessive speaker characteristics from the Positive Enrollment, resulting in the decrease in model performance. This suggests that and Negative enrollments if the results are unsatisfactory, and increasing the Positive Enrollment length is preferable.
enhance extraction quality by providing longer Positive
G Model Performance with respect the Input Audio Quality
Model Performance with respect to the Audio Mixture Quality In this section, show model performance relative to the quality of input Audio Mixture. As shown in Figure we plot the extracted audio's 's SI-SDR values for 5000 randomly generated test samples input SI-SDR. 96.6% of the audio mixtures' SI-SNR were improved after the extraction.
function of their
Model Performance with respect the Positive Enrollment Quality Figure 7 presents our model's extraction performance as function of enrollment audio quality. To evaluate this, we scale clean enrollment audio of the target speaker (Positive Enrollment) to specific SI-SNR values, while leaving the Negative Enrollment and the Mixed Audio unchanged. We then measure the average of 5000 test samples' extracted audios' SI-SNRi each Positive Enrollment SI-SNR value.
The model achieves optimal performance when the Positive Enrollment SI-SNR is between -5 dB to dB, and the performance deteriorates significantly when the SI-SNR of the Positive Enrollment falls below -10 dB. To address this limitation, we fine-tune our model by scaling the training data's ' Positive Enrollment -20 and -15 SI-SNR dB. As shown by the red line and the orange line in Figure this fine-tuning shifts the model's peak performance to lower SI-SNR regions, which means the model achieves optimum extraction performance under worse Positive Enrollment quality. Further research is necessary to develop model with strong performance across the full SI-SNR range of the Positive Enrollment.
7
H Model Performance under presence of Hybrid Interferers
As defined in Section 3.2 the main paper, when an interfering speaker speaks in subset of segments in the Positive Enrollment, and also exist in the Negative Enrollment, it is Hybrid Interferer. Model
26

---

Table 12: Model performance under different number of Neglect-Requires Interferers (NRI).
2 Speakers in Mixture
3 Speakers in Mixture
Method
Metric
1 NRI
NRI
NR
NRI
2 NRI
3
SNRi
10.29:2.52
10.20=2..54
10.25:22.55
10.45=2.44
10.51=2.49
10.57=2.54
Ours (Monaural)
SI-SNRi
9.07+:3.55
8.9943.56
9.08:13.43
8.51
8.63:13.56
8.63=3.77
STOI
0.764:+0.103
0,761+0.105
0.764:40.098
0.672+#0120
0.675=0.117
0.671=0.125
DNSMOS
2.15+0036
2.14+0.35
2.15+0036
1.93++037
1,93++0.38
1.92+0.37
Table 13: Comparison between our method and the TD-SpeakerBeam baselines' extracting conditioned on clean or noisy enrollments.
performance when
2 Speakers in Mixture
Speakers Mixture
4 Speakers in Mixture
Method
Metric
Clean Enroll
Noisy Enroll
Clean Enroll
Noisy Enroll
Clean Enroll
Noisy Euroll
SNRi SI-SNRi
10.19::2.59
9.93:12.71
10.46:2.46
10.37 : 2.64
10.48:
10.50:
Ours
9.12#:3.50
8.54.14.01
8,50-3.63
8.29:14.06
7.79-13.0
7.64:14.11
(Monaural)
STOI
0.764::0.102
0.749:10.121
0.670::0.122
0.665:10.130
0.592:10.131
0.144:0.141
DNSMOS
2.15=10.37
2.13:10.37
1.93:0.37
0.92:10.38
1.76::0.37
1665=0.40
SNRi
12.87:13.63
4.84:12.83
10.01
5.58:12.48
9.08:12.69
6.63:+2.60
TD- TD-SpeakeBBeam
SI-SNRi
61.85:6.36
1.75
7.63:6.06
-0,33=8.70
5,49:5.34
-1,26=7.17
STOI
0.834::0.145
0.595:1.2270
0.654:10182
0,451:0,212
0.53410.177
0.372:0.177
DNSMOS
2.81 +0.32
1,73+0.559
2.63+-0.3
1.51++0.40
2.45-10.35
1.46+00.48
have two ways to distinguish these interfering speakers from the target speaker: either through noticing that these speakers were silent in of the segments in the Positive Enrollment, when the target speaker is supposed to talk continuously; or noticing that these speakers exist in the Negative Enrollment, when the target speaker supposed remain silent. In this section, we explicitly show the model performance when such interfering speakers exist.
As shown in Table 11. focus the scenario containing only Hybrid Interferers, and gradually increase their number from to 3. In comparison to the increase in the Positive and Negative Interferer number, model shows higher robustness the increase in the number of Hybrid Interferers. This indicates that our model generalizes well in removing interference from Hybrid Interferers by leveraging the fact that such speakers can be effectively suppressed when treated as either Positive Negative Interferers.
I Model Performance under presence of Neglect-Required Interferers
When recording the Negative Enrollments, additional interfering speakers not captured in the Positive Enrollments might be present. Since these interferers' speech does not overlap with the target speaker, their voice should be neglected (thus named Peglect-Requiree Interferers). In this section, we investigate their affect on our model performance in this section.
As shown in Table [2] similar to the result observed when the number of Hybrid Interferers increase, model does not show statistically significant performance differences when the number of Neglect- Required Interferers increases from one to three. This demonstrates our model effectively generalizes to prevent the
Interferers from adversely affecting the quality of the extracted target
speaker speech.
J Model Performance when Clean Target Speaker's Enrollment is available
If applied without modification, our model fails when the enrollment consists solely of the clean target speaker's voice. This is due to the model's input normalization, where all-zero Negative Enrollment leads to a NaN output. To address this, we randomly select a fixed WHAM! noise sample as the Negative Enrollment, and add it with the Positive Enrollment to construct poiudo-noisy Positive Enrollment. Using such modification, we construct secudo-Positive and Negative Enrollment from different target speakers' clean audio enrollment, and perform extraction.
As shown in Table [3 in comparison to extracting target speaker's characteristic from noisy Enroll- ments, our model achieves higher performance when extracting conditioned on the pseudo-Positive
27

---

Table 14: Model performance when extracting conditioned on noisy single-speaker enrollments.
Table 15: Speaker configuration for eval- uating the Target Speaker Confusion Problem.
2 Speakers Mixture
Speakers in Mixture
Method
Metric
SNRi
10.19:12.59
10.48-12.46
Ours
SI-SNRi
9.12++3.50
8.50+3.63
Target Speaker
Interfering Speaker
(Monaural)
STOI
0.76.10.10
0.67 0.112
Audio Mixture
B
DNSMOS
2.15:+0.37
0.93
Fired
B
Fit Jor
SNRi 3.54:1:.45
4.29:12.46
B
A
SI-SNRi
0.50:t5.58
0.56+
USEF-TFGridnet
STOI
0.47:-0.17
0.38++0.11
DNSMOS
1.35+00444
1.29-+0.37
and Negative Enrollment. In comparison to the TD-SpeakerBeam [5], which is trained to extract using clean audio enrollment, our model achieves higher SNRi, SI-SNRi, and STOI when extracting from the mixture of 3 or more mixtures. These results verifies our model's applicability to extraction using clean audio enrollments. The reason why our model achieves higher performance under larger number of interfering speaker might be the difference in training data generation. TD-SSeakerBeam is trained predefined mixtures of different speakers' speech, while our training samples generated by dynamic mixing, which creates more diverse mixture of different speakers' voices. This training data generation strategy acts a form of data augmentation, helping the model generalize better to larger number of interfering speaker. However, there is still noticeable performance gap between model and the Beaa when extracting from the mixture of two speakers using clean enrollments. Further work is required to reduce the performance gap between models using noisy enrollment and using clean enrollment.
We want to emphasize that our model tackles the more challenging task of target speech extraction (TSE) with noisy enrollment, realistic scenario where prior works have shown significantly degraded performance. Although our model has worse performance under clean audio enrollment, our method still correctly identifies the target speaker users want, shown by the strongly positive SI-SNRi values. Indeed, the performance gap between our method and other models under clean enrollment could bring novel insights to improve
model, which will explore in the further works.
In addition, we evaluate our model and the baseline models' performance when using noisy single speaker enrollments. The noise in the enrollment is from the WHAM! dataset and set at 0 SNR level. As shown Table 14, TSE models trained with clean enrollments show significant performance decrease under noisy single speaker enrollments. In comparison, our model consistently outperforms the baselines across all metrics except DNSMOS, demonstrating greater robustness to enrollment noise.
K Model Performance for Target speaker Confusion Problem
Prior works have shown that the auxiliary speaker encoder may sometimes generate ambiguous speaker embeddings, leading to the extraction model extracting interferer's voice as the output. This known as the target confusion problem 44) To further investigate whether our model suffers from the target speaker confusion problem, we follow the experimental setup proposed by Zhao et al. [40]. Specifically, we construct 5000 test samples where both the audio mixture and the enrollment contain two speakers, and share the same interfering speaker. Let Ppred be the enrollment pair in the sample, where speaker is the target speaker and speaker B is the interfering speaker. After performing extraction, we identify the enrollment pairs that result in the extracted audio being more similar (in terms of SNR) to the interfering speaker B's speech than to the target speaker A's. We refer to these samples as target confusion samples. For each target confusion sample, we construct two additional enrollment pairs: Ptge and Pinterfer- In Ptgt, same as the target confusion sample, speaker A is the target and speaker B is the interferer. In Pinterfer speaker B is the target and speaker A is the interferer. Table [5] below summarizes the role speaker A and speaker in the audio mixture and each enrollment pair.
28

---

Table 16: Performance our model and the TCE model before and after fine-tuning, using monaural samples generated from the WSJO dataset.
Speakers in Mixture
Speakers in Mixture
Method
Metric
NI
2
NT
N
N
SNRi
8.33++2,97
8.14=3.03
7.93:33.0
8.69+22.64
8.50=2.73
8,24+22.81
Ours
SI-SNRi
5.66:16.04
5.33=6625
4.79:16.52
5.0416.03
4.5126.34
3.8416663
(Monaural)
STOI
0.672::0.14
0.664=0.171
0.647:001881
0.591=0.172
0.577=0.177
0.554=0.187
DNSMOS
1,93+=0.39
1.91=0040
1899+0.41
1.74-0.37
1.72++0.37
1.69=0.37
SNRi
8.19:13.00
7.97::3.07
7.75::3.06
8.96-2.70
8.80:1
8.58: 2.88
Ours
SI-SNRi
5.33:16.17
4.93-66477
4,45:16.50
5.27
4.84
4,20
(Fine-tuned)
STOI
0.647+0.177
0.640+0.184
0.621+0.190
0.580-0.181
0,567++0.188
0,544+0. 0.198
DNSMOS
1.87:10:46
1.84:10.46
181100477
1.70=0.42
1.68:10.42
1.65:10.41
6.65+2.79
6,20+2281
5.7912295
6.64=2.79
6.43-1:2335
6.11
SI-SNRi
4.00::4.93
3.1345.28
2.2145.87
3.99=4.93
2.19:44.98
14645.20
TCE
STOI
0.593=0.149
0.567=0.158
0.546+0.169
0.593=0.149
0.471=0.150
0.453=0.159
DNSMOS
1,74++0.38
1.73++0.37
1,75+0.37
1.74 0.38
1,60+0,335
1.63++0.35
SNRi
6.59112.62
6.20:12.69
6.03.12.81
6,58-2,62
6.54-12.33
6.3411241
TCE (Fine-tuned)
SI-SNRi
3,99+-4.41
3,21++4.77
2.67+55.25
3.97-4.41
2.55++4.43
2.10++4.62
STOI
0.576
0.548+0.154
0.533++0.13
0.462++0.133
0.446+0.154
DNSMOS
1.59:0.40
1.56:10.39
1.57:10.38
1.59=0.40
1.46:10.44
1,46.10.3
To verify if the model mistakenly encode the interfering speaker as the target speaker and the interfering speaker, we flatten and normalize the embeddings extracted by our model from each enrollment pair, obtaining E(Pprea), E(Pttt), E E(Pinterfer). and compute the cosine similarity between E(Prred and E( as well as between E(Ppred) and E ((Panterfer).
Out of the 5000 tested samples, 74 of them are the target confusion samples. 34 samples out of the 74 target confusion samples (45.9%) have enrollment embedding closer to the interfer- ing speaker than to the target speaker (i.e. 34 of the samples have cos E (Prpp))), ((Ptgt)) < cos( cos(FF(ppred) F PiPPnterrfe fer) These findings confusion study [40). where 45. % of target confusion samples showed the extracted embedding being closer the interferer. This means that significant percentage of target confusion samples have encoder embeddings being more close to the interfering speaker. However, it is important to emphasis that only 74 out of the 5000 samples (1.48%) showed evidence of target speaker confusion in our evaluation, suggesting that this is a rare occurrence and does not pose significant issue for our model's overall performance.
consistent with those reported in the target speaker
L Model performance on WSJO Dataset
In the main paper, we focus our experiments the samples constructed from the LibriSpeech dataset. In this section, we show our model's applicability to other datasets. We select the WSJO dataset as the source of speaker speech.
As shown in Table 16) our model correctly encode and extract the target speaker's voice, as shown by the positive SNRi and SI-SNRi metric, and outperforms the TCE baseline in most scenarios. However, after fine-tuning on samples containing two interfering speakers in both the Audio Mixture and enrollments, our model only improves performance when extracting from three-speaker mixtures, while performance degrades in two-speaker cases. Further exploration is required to train model with strong generalizability across multiple datasets and scenarios.
M Audio Visualization and Failure Cases
In this section, we show four successful cases and four failing cases of our model. For each case, we visualize the waveform and STFT spectrogram of the Mixed Audio, the extraction ground truth, and the model prediction of a one-second audio segment.
As shown in Figure our model correctly predicts silent sound in the extracted audio when the target speaker not speaking and extracts the target speaker's voice despite the frequency range largely overlaps with the interfering speakers.
29

---

Rige
Truth Rigi
Succasshi Cana
Pradiction
VisedAudiio
Truth fige
(12 48 SHGNI))
Prodiction
Grousd Truth Aige
Seczeushi SHSGI
Figure 8: Successful cases of model. We selected four samples where the model SI-SNRi performance above dB. We report the extracted audio SI-SNRi under the case number.
In several cases, our model less optimum performance, with less than dB SI-SNRi performance. We show four of these cases in Figure 9 In these cases, the model fails to extract the full target speaker's s voice some timesteps (Failure Cases 2), includes interfering the speaker's voice in its extraction (Failure Case 1, 3, 4), and introduces artifacts that obscure the intelligibility of the target speech. These non non-cxhaustive failure cases serve as a reference for future improvements.
N Societal Impact
This paper presents work whose goal is advance the field of Machine Learning and Audio Processing. This technology can potentially improve communication aids, hearing devices, and audio editing tools, making it easier for users focus on desired voices in complex auditory scenes. We acknowledge that such technology could raise privacy concerns if misused to extract a speaker's voice without consent. However, labels, which presupposes that the target speaker is already engaged in face-to-face conversation or otherwise clearly audible the user. As a result, our approach serves to enhance human auditory perception rather than introduce new capabilities that could pose societal or ethical risks. While our work may have broader socictal implications, we do not identify any that require specific emphasis.
method requires prior access to the target speaker's speech activity
30

---

Audio Warsforra
apedrrggen lop magehbude
spectrogrem pheae argle
Agg
ETT (NRR)
Prediction
Aipitidd
Wovd Audioo ght
Fonms Nu!
Auge
Palkea Crose
Prediction
AbseiAudio
Ground Age
Palura
Woud Audio
Grousd Trush Aig
Palure Cose (5.00
Wind Austoo
Figure 9: Failure cases of our model. We selected four samples where the model SI-SNRi performance below 8 dB. We highlight the timesteps where the model has poor performance with red boxes.
31


Reviews:
[{"review_id": "itpvp6UZbT", "reviewer": "Reviewer_anep", "summary": "This paper proposes a novel target speaker extraction (TSE) model capable of utilizing mixture enrollments with multiple speakers for effective extraction of the target speaker's voice. The enrollment feature is first enhanced by extracting positive and negative enrollment representations from segments with and without the target speaker respectively. An self-attention mechanism is then employed to filter out interference speakers from the enrollment feature, resulting in a fused enrollment feature focused on the target speaker. It is finally integrated into the TSE model via cross-attention to extract the target speaker's voice. The proposed model is mostly based on the TF-GridNet model architecture, and the core contribution is the ability of utilizing mixture enrollments for effective TSE. Experiments are conducted on Librispeech and WSJ0 (Appendix) based simulation datasets as well as a real-world dataset based on VoxConverse, which validates the efficacy of the proposed model, especially in scenarios with multiple speakers in the enrollment.", "questions": "1. When the input signal is binaural (as described in Section 4.1), it is unclear whether the proposed model is capable of modeling the spatial information between two microphone channels, as Section 3 only discusses monaural scenarios.\n\n2. Table 1 primarily evaluates the extraction performance when there are multiple speakers in the enrollment, which matches the motivation of this paper. However, it is still necessary to report the performance with single-speaker noisy enrollments, as this constitutes a large percentage of real-world scenarios. In Appendix J, it is shown that the proposed model underperforms TD-SpeakerBeam with single-speaker clean enrollments. But it is still unclear whether the same tendency will be observed for noisy enrollments.\n\n3. While line 323 claims that the model “correctly extracts the target speaker” based on the improved performance shown in Figure 5, it is not explicitly verified. In particular, the target confusion problem [R1] is commonly observed in target speaker extraction systems and can be analyzed quantitatively. Therefore, it is highly recommended that the authors present in-depth analysis on the accuracy of target speaker extraction, which can be more difficult in the context of mixture enrollments as discussed in this paper.\n\n[R1] Z. Zhao, D. Yang, R. Gu, H. Zhang, and Y. Zou, “Target confusion in end-to-end speaker extraction: Analysis and approaches,” in Proc. ISCA Interspeech, 2022, pp. 5333–5337.\n\n4. There are two major limitations of the experiments:\n    1. Only the non-intrusive DNSMOS metric is used for the evaluation on real-world speech data in Appendix K. It is nice to see evaluation on real-world data. However, as reported by various prior studies, solely using an NN-based non-intrusive metric for evaluation can be risky as they may not reflect the true speech quality due to domain mismatch. Additional evaluation metrics such as word error rate (WER), speaker similarity, or even MOS based on subjective listening tests should be additionally reported for more comprehensive assessment.\n    2. There is no demo page or subjective listening to subjectively validate the performance of the proposed method. For example, the artifacts mentioned in line 365 cannot be easily understood without an example.", "limitations": "The potential negative societal impact of this work is not discussed in Appendix N. It is recommended that the authors include a discussion on the potential misuse of the proposed method and how they can avoided. For example, the proposed method could raise privacy concerns if used to extract the voice of a target speaker without their consent.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The problem addressed by the proposed method is important and novel, which is highly related to real-world applications.\n2. The proposed model is compact and performant, achieving better performance in complex scenarios with mixture enrollments with only 1.88 M parameters, which is smaller than baseline models such as TCE and LookOnceToHear.\n3. Extensive experimental validation and ablation studies are provided on both simulated and real-world datasets, validating the model's robustness and generalizability.\n\nWeaknesses:\n1. Some technical details are missing, such as the definition and implementation of the learnable segmentation embeddings in the Encoder Fusion Module.\n2. The organization of the paper could be improved by moving some key experiments (e.g., Appendix K) to the main text to better highlight the effectiveness of the proposed method.\n3. Some claims lack a solid experimental validation, such as line 323, which should be rephrased or supported by additional experiments.\n4. The writing needs improvement, as some sentences are overly complex or even grammatically incorrect, leading to poor readability.\n5. More advanced TSE baselines, such as X-TF-GridNet, instead of the outdated SpeakerBeam model, should be compared in the experiments to better assess the performance of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: When the input signal is binaural (as described in Section 4.1), it is unclear whether the proposed model is capable of modeling the spatial information between two microphone channels, as Section 3 only discusses monaural scenarios.\n\n**A1**: Thank you for raising this question. The difference between the monaural and binaural audio architectures lies solely in the input. The binaural model takes in 4 channels input (stacked real and imaginary components from both channels) and the monaural model takes in only 2 channels (stacked real and imaginary component). By stacking the two channels' STFT, the binaural model can capture the inter-channel temporal differences between the two channels, which encode the directional information of the target speaker since the target speaker speaks at 90 degree azimuth angle in the enrollment. Both model architectures do not differ after the first convolution layer.\n\nTo verify that our model leverages spatial information for speaker extraction, we evaluate the binaural model by randomly varying the azimuth angle of the target Speaker in the Positive Enrollment. As shown in the table below, this results in a significant drop in performance. This indicates that our binaural model leverages the spatial information (target speaker direction) to perform enrollment.\n\n| Model | Metric    | Extract from 2 speakers, Random Azimuth   | Extract from 2 speakers, Azimuth = 90 degree | Extract from 3 speakers, Random Azimuth | Extract from 3 speakers, Azimuth = 90 degree |\n|-|-|-|-|-|-|\n| Ours (Binaural) | SNRi      | 6.91 $\\pm$ 3.56 | 9.60 $\\pm$ 3.57        | 7.96 $\\pm$ 3.46 | 9.78 $\\pm$ 3.23 |\n| | SI-SNRi   | 2.22 $\\pm$ 8.01 | 7.84 $\\pm$ 5.20 | 1.81 $\\pm$ 8.03 | 6.72 $\\pm$ 5.08 |\n| | PESQ      | 1.80 $\\pm$ 0.62 | 2.22 $\\pm$ 0.47 | 1.54 $\\pm$ 0.54 | 1.85 $\\pm$ 0.42 |\n| | STOI      | 0.58 $\\pm$ 0.22 | 0.73 $\\pm$ 0.16 | 0.47 $\\pm$ 0.23 | 0.61 $\\pm$ 0.18 |\n| | DNSMOS    | 1.68 $\\pm$ 0.52 | 1.80 $\\pm$ 0.59 | 1.53 $\\pm$ 0.46 | 1.60 $\\pm$ 0.49 |\n| | WER       | 0.65 $\\pm$ 0.40 | 0.45 $\\pm$ 0.34 | 0.78 $\\pm$ 0.33 | 0.64 $\\pm$ 0.43 |\n    \n\n\n\n**Q2**: Model performance with single-speaker noisy enrollments.\n\n**A2**: Thank you for your suggestion on additional experiments. We re-evaluate our model and two baseline models using noisy single speaker enrollments. The noise in the enrollment is from the WHAM! dataset and set at 0 SNR level. The table below shows the model performance when extracting from the mixture of two or three speakers, TSE models trained with clean enrollments show significant performance decrease under noisy single speaker enrollments. In comparison, our model consistently outperforms the baselines across all metrics except DNSMOS, demonstrating greater robustness to enrollment noise.\n\n\n| Model          | Metric       | Extract from 2 speakers | Extract from 3 speakers |\n|-|-|-|-|\n| Ours (Monaural) | SNRi         | **10.19 $\\pm$ 2.59**         | **10.48 $\\pm$ 2.46**         |\n| | SI-SNRi | **9.12 $\\pm$ 3.50**          | **8.50 $\\pm$ 3.63**          |\n| | PESQ | **2.08 $\\pm$ 0.32**          | **1.81 $\\pm$ 0.33**          |\n| | STOI | **0.76 $\\pm$ 0.10**          | **0.67 $\\pm$ 0.12**          |\n| | DNSMOS | 2.15 $\\pm$ 0.37          | 1.93 $\\pm$ 0.37          |\n| | WER | **0.43 $\\pm$ 0.29**          | **0.61 $\\pm$ 0.35**          |\n| USEF-TFGridnet | SNRi         | 3.54 $\\pm$ 3.45          | 4.29 $\\pm$ 2.46          |\n| | SI-SNRi      | 0.50 $\\pm$ 5.58          | 0.56 $\\pm$ 2.85          |\n| | PESQ         | 1.52 $\\pm$ 0.48          | 1.33 $\\pm$ 0.36          |\n| | STOI         | 0.47 $\\pm$ 0.17          | 0.38 $\\pm$ 0.11          |\n| | DNSMOS       | 1.35 $\\pm$ 0.44          | 1.29 $\\pm$ 0.37          |\n| | WER          | 0.64 $\\pm$ 0.33          | 0.84 $\\pm$ 0.25          |\n| TD-SpeakerBeam    | SNRi         | 8.88 $\\pm$ 5.05          | 7.21 $\\pm$ 3.44          |\n| | SI-SNRi      | 4.88 $\\pm$ 10.82         | 1.77 $\\pm$ 8.24          |\n| | PESQ         | 1.81 $\\pm$ 0.71          | 1.32 $\\pm$ 0.57          |\n| | STOI         | 0.62 $\\pm$ 0.23          | 0.47 $\\pm$ 0.19          |\n| | DNSMOS       | **2.50 $\\pm$ 0.43**      | **2.33 $\\pm$ 0.41**      |\n| | WER          | 0.63 $\\pm$ 0.31          | 0.83 $\\pm$ 0.26          |\n\n\n**Q3**: Line 323: Explanation of \"correctly extracts the target speaker\" and analysis of the target speaker confusion problem.\n\n**A3**： Thank you for your suggestion. By \"correctly extracts the target speaker\", we mean the extracted audio achieving a higher SI-SNR with respect to the target speaker's voice than with respect to the interferer's speech. If the model mistakenly encodes the interfering speaker in the enrollment, it should instead show a high SI-SNR for the interfering speaker. As shown in Figures 4 and 5 in the original paper, the extracted speech has less than -10 Si-SNRi with respect to the interfering speaker's speech, verifying that the interfering speakers' voice is effectively suppressed. \n\nTo further investigate whether our model suffers from the target speaker confusion problem, we follow the experimental setup proposed by Zhao et al. [1]. Specifically, we construct 5000 test samples where both the audio mixture and the enrollment contain two speakers, and share the same interfering speaker. Let $P_{pred}$ be the enrollment pair in the sample, where speaker A is the target speaker and speaker B is the interfering speaker. After performing extraction, we identify the enrollment pairs that result in the extracted audio being more similar (in terms of SNR) to the interfering speaker B's speech than to the target speaker A's. We refer to these samples as target confusion samples. For each target confusion sample, we construct two additional enrollment pairs: $P_{tgt}$ and $P_{interfer}$. In $P_{tgt}$, same as the target confusion sample, speaker A is the target and speaker B is the interferer. In $P_{interfer}$, speaker B is the target and speaker A is the interferer. The table below summarises the role of speaker A and speaker B in the audio mixture and each enrollment pair. \n\n| | Audio Mixture | $P_{pred}$ | $P_{tgt}$ | $P_{interfer}$ |  \n|-|-|-|-|-|\n| Target Speaker | A | A | A | B |\n| Interfering Speaker | B | B | B | A | \n\nTo verify if the model mistakenly encode the interfering speaker as the target speaker and the interfering speaker, we flatten and normalize the embeddings extracted by our model from each enrollment pair, obtaining $E(P_{pred})$, $E(P_{tgt}), E(P_{interfer})$, and compute the cosine similarity between $E(P_{pred})$ and $E(P_{tgt})$, as well as between $E(P_{pred})$ and $E(P_{interfer})$.\n\nOut of the 5000 tested samples, 74 of them are the target confusion samples. 34 samples out of the 74 target confusion samples (45.9%) have enrollment embedding closer to the interfering speaker than to the target speaker (i.e. 34 of the samples have $cos(E(P_{pred}), E(P_{tgt})) < cos(E(P_{pred}), E(P_{interfer}))$). These findings are consistent with those reported in the target speaker confusion study [1], where 45.1% of target confusion samples showed the extracted embedding being closer to the interferer. This means that significant percentage of target confusion samples have encoder embeddings being more close to the interfering speaker. However, it is important to emphasise that only 74 out of the 5000 samples (1.48%) showed evidence of target speaker confusion in our evaluation, suggesting that this is a rare occurrence and does not pose a significant issue for our model's overall performance.\n\n[1] Z. Zhao, D. Yang, R. Gu, H. Zhang, and Y. Zou, “Target confusion in end-to-end speaker extraction: Analysis and approaches,” in Proc. ISCA Interspeech, 2022, pp. 5333–5337.\n\n**Q4**: Additional evaluation metrics and baseline comparison on real-world speech data.\n\n**A4**: We additionally evaluate our model with WER and PESQ on the synthetic data. Please see the response to reviewer iQ1f Q3 for the results. In addition, since we do not have access to the ground truth of the extracted audio in audio mixtures recorded in the real-world, we focus our evaluation on the Mean Opinion Score for the real-world speech data. Please see the response to reviewer 1MM1 Q1 for the results.\n\n**Q5**: The artefacts mentioned in line 365 cannot be easily understood without an example.\n\n**A5**: We provided 5 audio examples for each of successful and failure cases of our model in the original supplementary file. We will implement a demo page to present the extraction results on both synthetic and real-world audio mixtures. \n\n**Q6**: More advanced TSE baselines, such as X-TF-GridNet, instead of the outdated SpeakerBeam model, should be compared in the experiments to better assess the performance of the proposed method.\n\n**A6**: Thank you for your suggestion. As USEF-TFGridnet[2] reports higher performance in comparison to the X-TFGridnet in their paper, we compare with USEF-TFGridnet and SoloAudio. Please see the response to reviewer iQ1f Q2 for the results.\n\n[2] B. Zeng and M. Li, “USEF‑TSE: Universal Speaker Embedding Free Target Speaker Extraction,” IEEE Trans. Audio, Speech and Language Processing, 2025, pp. 2110–2124.\n\n**Paper Formatting Concerns and Clarifying Technical Details**\nThank you for raising these concerns. We will refine the grammar and explanation in the corresponding paragraphs in the revised version. In particular, we will include pseudo-code of the Encoder Fusion module in the response to reviewer iQ1f Q5. We will also discuss the potential negative social impact mentioned."}, {"review_id": "Tw0vHB6Z3R", "reviewer": "Reviewer_iQ1f", "summary": "The paper addresses monaural target speaker extraction using only noisy enrollment audio. Instead of requiring clean clips, the authors propose using both Positive (target speaking) and Negative (target silent) segments from noisy enrollments. An attention-based Siamese Encoder Fusion Module generates target-speaker embeddings to guide a TF-GridNet-based Extraction Fusion Module. The two-stage training first distills embeddings from clean speech, then trains the extraction network, leading to faster convergence. The method achieves state-of-the-art results in this setting.", "questions": "See Weaknesses for details.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a novel enrollment strategy using paired positive and negative enrollments for addressing monaural target speaker extraction with noisy enrollment.\n2. The paper demonstrates robust experiments and comparisons to baseline models.\n3. The authors use thorough evaluation on their dataset using objective metrics (SNR, SI-SNR, STOI, DNSMOS) on both monaural and binaural audio.\n4. The paper has shown extensive ablation studies on using two-stage training strategy and different fusion strategies to encode the target speaker’s characteristics.\n\nWeaknesses:\n1. I think it is important to report performance when a clean enrollment is available. Without this, we cannot judge whether the noisy strategy matches or degrades relative to the ideal case.\n2. In my view, comparing only to SpeakerBeam for the monaural task is too weak. Including recent SOTA discriminative and generative TSE methods (e.g. USEF-TSE, SoloAudio) would give a fairer picture of the gains.\n3. The current metrics (SNR, SI-SNR, STOI, DNSMOS) omit key measures of intelligibility (WER or CER) and perceptual quality (e.g. PESQ). I think adding these would strengthen the assessment.\n4. Objective gains do not always reflect perceptual quality. I would like to see human listening tests or a comparison demo to confirm that improvements hold up for end users.\n5. Some notation in Section 3.3 (encoder fusion) is dense. A short pseudo-code snippet or an extra diagram could help readers grasp the self-attention comparison more easily.", "was_revised": true, "final_justification": "This paper proposes a negative sampling method for target speaker extraction. The experiments demonstrate its effectiveness. However, in my opinion, negative sampling is not new and incremental in 2025. I therefore recommend Borderline Accept, but I will also not be surprised if the paper does not get accepted.", "rebuttal": "**Q1**: I think it is important to report performance when a clean enrollment is available. Without this, we cannot judge whether the noisy strategy matches or degrades relative to the ideal case.\n\n**A1**: We reported our model performance when clean enrollments are available in Appendix J in the original supplementary material. Our model outperforms the TSE model trained on clean enrollment when extracting from the mixtures of three or more speakers, but shows lower performance when extracting from the mixtures of two speakers. In addition, we evaluate our model and the baseline models' performance when using noisy single speaker enrollments. Please see the response to reviewer anep Q2 for the result. \n\n**Q2**: Including recent SOTA discriminative and generative TSE methods (e.g. USEF-TSE, SoloAudio) would give a fairer picture of the gains.\n\n**A2**: Thank you for your suggestion. We did not focus our comparison with the SOTA TCE methods because of the difference in problem formulation. Since the enrollments contain multiple speakers, SOTA TSE models trained on clean enrollment cannot perform accurate extraction without additional information to distinguish target speakers from the interfering speakers in the noisy enrollment. \n\nTo verify this, we additionally evaluate the USEF-TFGridnet and SoloAudio under the proposed task. As shown in the table below, both model performance degrade significantly under our problem formulation, highlighting the increased difficulty of our task and demonstrating the effectiveness of our proposed model. In particular, we notice that the SoloAudio model trained to perform target sound extraction struggles to distinguish the subtle difference between different speakers' characteristics. As a result, it often produces outputs with large silent regions, leading to notably low SI-SNRi scores.\n\n| Model | Metric | Extract from 2 spk., 2 spk. in Enroll | Extract from 2 spk., 3 spk. in Enroll | Extract from 2 spk., 4 spk. in Enroll |  Extract from 3 spk., 2 spk. in Enroll | Extract from 3 spk., 3 spk. in Enroll | Extract from 3 spk., 4 spk. in Enroll |\n|-|-|-|-|-|-|-|-|\n| USEF-TFGridnet | SNRi | 3.42 $\\pm$ 3.43 | 3.45 $\\pm$ 3.58 | 3.30 $\\pm$ 3.50 | 4.31 $\\pm$ 2.52 | 4.15 $\\pm$ 2.47 | 4.23 $\\pm$ 2.58 |\n| | SI-SNRi | -0.03 $\\pm$ 5.97 | -0.03 $\\pm$ 6.42    | -0.14 $\\pm$ 6.00 | 0.29 $\\pm$ 3.38 | 0.11 $\\pm$ 3.36 | 0.09 $\\pm$ 3.24 |\n| | PESQ | 1.52 $\\pm$ 0.49 | 1.54 $\\pm$ 0.49 | 1.52 $\\pm$ 0.49 | 1.32 $\\pm$ 0.38 | 1.32 $\\pm$ 0.37 | 1.31 $\\pm$ 0.36 |\n| | STOI | 0.43 $\\pm$ 0.17 | 0.43 $\\pm$ 0.18 | 0.43 $\\pm$ 0.17 | 0.36 $\\pm$ 0.11 | 0.35 $\\pm$ 0.11 | 0.36 $\\pm$ 0.11 |\n| | DNSMOS | 1.37 $\\pm$ 0.45 | 1.37 $\\pm$ 0.45 | 1.36 $\\pm$ 0.45 | 1.35 $\\pm$ 0.41 | 1.33 $\\pm$ 0.41 | 1.34 $\\pm$ 0.42 |\n| | WER | 0.66 $\\pm$ 0.33 | 0.66 $\\pm$ 0.36 | 0.68 $\\pm$ 0.32 | 0.85 $\\pm$ 0.26 | 0.86 $\\pm$ 0.21 | 0.86 $\\pm$ 0.20 |\n| SoloAudio | SNRi | 1.55 $\\pm$ 2.26 | 1.39 $\\pm$ 2.17 | 1.57 $\\pm$ 2.35 | 3.27 $\\pm$ 2.41 | 3.45 $\\pm$ 2.79 | 3.59 $\\pm$ 2.58 |\n| | SI-SNRi | -47.05 $\\pm$ 11.04  | -46.40 $\\pm$ 12.13  | -46.46 $\\pm$ 10.13  | -46.93 $\\pm$ 12.77  | -44.46 $\\pm$ 11.98  | -45.41 $\\pm$ 12.03  |\n| | PESQ | 0.85 $\\pm$ 0.90 | 0.60 $\\pm$ 0.69 | 0.71 $\\pm$ 0.85 | 0.65 $\\pm$ 0.76 | 0.69 $\\pm$ 0.72 | 0.78 $\\pm$ 0.88 |\n| | STOI | 0.06 $\\pm$ 0.09 | 0.07 $\\pm$ 0.09 | 0.06 $\\pm$ 0.09 | 0.04 $\\pm$ 0.08 | 0.05 $\\pm$ 0.09 | 0.05 $\\pm$ 0.08 |\n| | DNSMOS | 1.57 $\\pm$ 0.26 | 1.59 $\\pm$ 0.24 | 1.58 $\\pm$ 0.25 | 1.55 $\\pm$ 0.22 | 1.51 $\\pm$ 0.26 | 1.54 $\\pm$ 0.25 |\n| | WER | 0.96 $\\pm$ 0.03 | 0.97 $\\pm$ 0.02 | 0.98 $\\pm$ 0.01 | 0.98 $\\pm$ 0.02 | 0.98 $\\pm$ 0.04 | 0.99 $\\pm$ 0.04 |\n    \n\n**Q3**: The current metrics (SNR, SI-SNR, STOI, DNSMOS) omit key measures of intelligibility (WER or CER) and perceptual quality (e.g. PESQ). I think adding these would strengthen the assessment.\n\n**A3**: Thank you for your suggestion on additional metrics.  We include further evaluations of our models and baseline methods using PESQ and WER in the table below. Our monaural model outperforms all the baseline methods in both metrics. However, as mentioned in the Conclusion and Limitation section, the generated audio still contains artifacts, which negatively impact the PESQ and WER performance. Since the binaural model is trained to preserve the reverberation characteristics of the target speaker, and reverberation can negatively impact PESQ and WER, we report the binaural models' results here solely for completeness.\n\n| Model | Metric | Extract from 2 spk., 2 spk. in Enroll | Extract from 2 spk., 3 spk. in Enroll | Extract from 2 spk., 4 spk. in Enroll |  Extract from 3 spk., 2 spk. in Enroll | Extract from 3 spk., 3 spk. in Enroll | Extract from 3 spk., 4 spk. in Enroll |\n|-|-|-|-|-|-|-|-|\n| Ours (Monaural) | PESQ | **2.07 $\\pm$ 0.34** | **2.06 $\\pm$ 0.36** | **2.05 $\\pm$ 0.36** | **1.79 $\\pm$ 0.33** | **1.78 $\\pm$ 0.34** | **1.75 $\\pm$ 0.37** |\n| | WER | **0.42 $\\pm$ 0.35** | **0.43 $\\pm$ 0.28** | **0.45 $\\pm$ 0.28** | **0.61 $\\pm$ 0.28** | **0.61 $\\pm$ 0.28** | **0.62 $\\pm$ 0.28** |\n| Ours (FiLM Fusion)   | PESQ | 1.84 $\\pm$ 0.32 | 1.83 $\\pm$ 0.34 | 1.84 $\\pm$ 0.33 | 1.57 $\\pm$ 0.31 | 1.56 $\\pm$ 0.32 | 1.54 $\\pm$ 0.35 |\n| | WER | 0.54 $\\pm$ 0.37 | 0.54 $\\pm$ 0.29 | 0.54 $\\pm$ 0.28 | 0.73 $\\pm$ 0.24 | 0.72 $\\pm$ 0.25 | 0.73 $\\pm$ 0.24 |\n| TCE | PESQ   | 1.91 $\\pm$ 0.34 | 1.80 $\\pm$ 0.41 | 1.73 $\\pm$ 0.46 | 1.91 $\\pm$ 0.34 | 1.53 $\\pm$ 0.39 | 1.48 $\\pm$ 0.40 |\n| | WER | 0.73 $\\pm$ 0.16 | 0.76 $\\pm$ 0.24 | 0.77 $\\pm$ 0.24 | 0.73 $\\pm$ 0.25 | 0.88 $\\pm$ 0.21 | 0.88 $\\pm$ 0.17 |\n| USEF-TFGridnet | PESQ | 1.52 $\\pm$ 0.49 | 1.54 $\\pm$ 0.49 | 1.52 $\\pm$ 0.49 | 1.32 $\\pm$ 0.38 | 1.32 $\\pm$ 0.37 | 1.31 $\\pm$ 0.36 |\n| | WER | 0.66 $\\pm$ 0.33 | 0.66 $\\pm$ 0.36 | 0.68 $\\pm$ 0.32 | 0.85 $\\pm$ 0.26 | 0.86 $\\pm$ 0.21 | 0.86 $\\pm$ 0.20 |\n\n| Model | Metric | Extract from 2 spk., 2 spk. in Enroll | Extract from 2 spk., 3 spk. in Enroll | Extract from 2 spk., 4 spk. in Enroll |  Extract from 3 spk., 2 spk. in Enroll | Extract from 3 spk., 3 spk. in Enroll | Extract from 3 spk., 4 spk. in Enroll |\n|-|-|-|-|-|-|-|-|\n| Ours (Binaural) | PESQ | 2.24$\\pm$ 0.47 | 2.22 $\\pm$ 0.47 | 2.25 $\\pm$ 0.47 | 1.85 $\\pm$ 0.44 | 1.85 $\\pm$ 0.42 | 1.87 $\\pm$ 0.43 |\n| | WER | 0.45 $\\pm$ 0.48 | 0.45 $\\pm$ 0.34 | 0.44 $\\pm$ 0.35 | 0.63 $\\pm$ 0.33 | 0.64 $\\pm$ 0.43 | 0.64 $\\pm$ 0.42 |\n| LookOnceToHear| PESQ   | 2.28 $\\pm$ 0.47 | 2.25 $\\pm$ 0.52 | 2.28 $\\pm$ 0.45 | 1.87 $\\pm$ 0.48 | 1.88 $\\pm$ 0.44 | 1.90 $\\pm$ 0.47 |\n| | WER | 0.45 $\\pm$ 0.41 | 0.44 $\\pm$ 0.35 | 0.47 $\\pm$ 0.68 | 0.66 $\\pm$ 0.36 | 0.63 $\\pm$ 0.33 | 0.64 $\\pm$ 0.33 |\n\n\n**Q4**: Objective gains do not always reflect perceptual quality. I would like to see human listening tests or a comparison demo to confirm that improvements hold up for end users.\n\n**A4**: \nWe provided five audio examples each for the successful and failure cases of our model in the supplementary materials. \n\nAdditionally, we evaluate the Mean Opinion Score (MOS) of different models using naturally recorded audio mixtures sourced from Freesound.org and the VoxConverse dataset. These sounds are real-world audio mixtures captured in pubs, metro stations, urban areas, and city council meeting. Please see the response to reviewer 1MM1 Q1 for the results.\n\n**Q5**: Some notation in Section 3.3 (encoder fusion) is dense. A short pseudo-code snippet or an extra diagram could help readers grasp the self-attention comparison more easily.\n\n**A5**: Thank you for your suggestion. We provide the pseudo-code of the Encoder Fusion Module below. Two segmentation embeddings, $S_{pos}, S_{neg}$, are first element-wise added to the input embeddings $E_{pos}, E_{neg}$ to allow the model to distinguish which enrollment each embedding originates from. The resulting two embeddings are concatenated along the temporal dimension and passed through two Full-band Self-attention layers. \n\nThe self-attention calculation between the embeddings of different Positive Enrollment frames allows the model to identify the Positive Interferers, who remain silent in some of the Positive Enrollment frames, and exclude these speakers in the extracted embedding. Similarly, attention between the Positive and Negative Enrollments embeddings enables the model to identify and exclude Negative Interferers, (i.e., speakers present only in the negative enrollments), from the output embedding. \n\nDuring Stage 1 training, we simulate the enrollments such that the number of Positive and Negative Interferers varies between 0 and 2. Distilling the encoder to produce an embedding that retains only the target speaker’s characteristics under these enrollments input encourages the model to exclude both types of interferers in the extracted speaker embedding.\n\n```python\n# Input: \n#     E_pos: Positive Enrollment embedding, shape [T_pos, D]\n#     E_neg: Negative Enrollment embedding, shape [T_neg, D]\n# Learnable parameters: \n#     S_pos: segmentation embedding for E_pos, shape [1, D]\n#     S_neg segmentation embedding for E_neg, shape [1, D]\n#     M: 2 Full-band Self-attention layers\nEncoder Fusion Module:\n  # Elementwise add segmentation embedding via broadcasting\n  E_pos = E_pos + S_pos # shape [T_pos, D]\n  E_neg = E_neg + S_neg # shape [T_neg, D]\n\n  # Concatenate along the temporal dimension \n  E_concat = [E_pos, E_neg] # shape [T_pos + T_neg, D]\n\n  # Apply two Full-band Self-attention calculation\n  E_concat = M(E_concat) # shape [T_pos + T_neg, D]\n\n  # Truncate embeddings to match the teacher model's embedding shape\n  output = E_concat[:T_pos] # shape [T_pos, D]\n```"}, {"review_id": "JIjOShH9Qa", "reviewer": "Reviewer_1MM1", "summary": "This paper studies the target speaker extraction (TSE) problem where the task is to extract the target speaker's voice from an audio mixture provided that some enrollment audio segments are given as the conditional input to the TSE model. Different from prior works that typically assume availability of the clean speech of the target speaker for the enrollment audio, this paper focuses on the more realistic scenario where the enrollment audio segments can contain both the target speaker's voice and interfering speakers' speech as well as background noise. The main idea is to utilized positive and negative noisy enrollments by exploiting the temporal misalignment of speech signals. To this end, a two-branch encoder design and an associated two-stage training strategy are introduced to effectively learn the extraction functionality. Experimental results on synthetic audio mixture data are provided to demonstrate the effectiveness of the proposed method over existing TSE models that have not sufficiently leveraged noisy enrollments as the conditioning input for realistic use cases.", "questions": "1. The loss function for training the extraction branch, i.e., eq. (3) seems relatively simple -- it is just the SNR between the ground truth signal and the output of the TSE model. I wonder if this would lead to sub-optimal performance and what motivated you to use this loss? Also, have you experimented with other more sophisticated loss functions, e.g., time-frequency domain MSE losses or some other perceptually motivated losses?\n\n2. Will you make the code public if the paper is accepted?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The problem that the paper aims to address is interesting and important, i.e., TSE with noisy enrollments, which is an essential step towards more realistic scenarios and applications of target speaker extraction.\n\n2. The main idea of exploiting the temporal misalignment of voices from different speakers seems novel and makes sense, which also leads to effective TSE with noisy enrollments.\n\n3. Extensive ablation studies on one-stage vs. two-stage training schemes, different enrollment segment lengths, accuracy of the labeling, etc. offer further insights into understanding the behavior of the proposed approach.\n\n4. The paper is well-written and easy to follow, with code and audio samples provided to facilitate reproducibility and qualitative comparison.\n\n**Weaknesses**\n\n1. The experimental results are mostly relying on artificially mixed audio data. Given that this paper aims to address the limitation of prior works for real-world TSE applications, it is expected to see more evaluation results of the method on naturally recorded mixture audio, to further confirm its effectiveness. Although there are some relevant results presented in Appendix K, it seems that the mixture signals in the experiment are still manually added up. In addition, there is no comparison with other existing approaches.", "was_revised": true, "final_justification": "During the rebuttal period, the authors provided additional evaluation results on naturally recorded real-world audio mixtures, which addressed my earlier concerns about the method’s performance in realistic scenarios. The accompanying subjective evaluation further supports the effectiveness of their approach. I am therefore happy to raise my score to Accept.", "rebuttal": "**Q1**: The experimental results are mostly relying on artificially mixed audio data. Given that this paper aims to address the limitation of prior works for real-world TSE applications, it is expected to see more evaluation results of the method on naturally recorded mixture audio, to further confirm its effectiveness. Although there are some relevant results presented in Appendix K, it seems that the mixture signals in the experiment are still manually added up. In addition, there is no comparison with other existing approaches.\n\n**A1**: Thank you for raising this question. To effectively demonstrate our model's performance under strong overlap between the target and interfering speakers, all audio mixtures used in our original submission were synthetically generated or manually added up from real-world audio recordings. \n\nWe additionally compare our model and the baseline methods on naturally recorded real-world audio mixtures. We evaluate the Mean Opinion Score (MOS) of different models using five naturally recorded audio mixtures sourced from Freesound.org and the VoxConverse dataset. These sounds are noisy real-world speech recordings captured in pubs, metro stations, urban areas, and city council meetings. For the evaluation, we manually labelled the Positive and Negative Enrollment and extracted target speech from Audio Mixture clips taken from the same recording. The duration of the enrollment and Audio Mixture clips range from 3 to 8 seconds. \n\n10 participants are asked to rate the original (unprocessed) Audio Mixture, and the audio extracted by three models. Given a textual description of the target speaker (e.g., the female speaker talking over the crowd), they rated the clarity of the target speech relative to background noise on a 1–4 scale. As shown in the table below, our model achieved a MOS of 3.35, outperforming all baseline methods. These results demonstrate the superiority and practical applicability of our model in real-world target speech extraction scenarios.\n\n| Model | Average User Mark |\n|-|-|\n| Ours (Monaural) | 3.35 |\n| Ours (FiLM Fusion) | 2.60 |\n| USEF-TFGridnet | 1.45 |\n| Unprocessed | 2.10 |\n\n**Q2**: The loss function for training the extraction branch, i.e., eq. (3) seems relatively simple -- it is just the SNR between the ground truth signal and the output of the TSE model. I wonder if this would lead to sub-optimal performance and what motivated you to use this loss? Also, have you experimented with other more sophisticated loss functions, e.g., time-frequency domain MSE losses or some other perceptually motivated losses?\n\n**A2**: Thank you for sharing your insight. Training using only SNR loss is a common practice used in numerous prior works [1, 2, 3, 4]. We kept the loss simple and consistent with the prior works to improve the reproducibility and comparability of our work. Therefore, we did not conduct extensive ablation studies on alternative loss functions. Incorporating perceptual losses is indeed a valuable direction for future work to enhance model performance.\n\n[1] J. Huang, X. Wang, and D. Wang, “Multi-level speaker representation for target speaker extraction,” in Proc. ISCA Interspeech, 2021, pp. 1459–1463.\n\n[2] A. Ephrat, I. Mosseri, T. M. Remez, and S. Freeman, “Target conversation extraction: Source separation using turn-taking dynamics,” in Proc. ISCA Interspeech, 2020, pp. 1401–1405.\n\n[3] Y. Xu, M. Yu, C. Yu, and D. Yu, “Listen to extract: Onset-prompted target speaker extraction,” in Proc. ISCA Interspeech, 2022, pp. 2963–2967.\n\n[4] W. Zhou, S. Chen, Y. Liu, and D. Su, “Look once to hear: Target speech hearing with noisy examples,” in Proc. ISCA Interspeech, 2023, pp. 3597–3601.\n\n**Q3**: Will you make the code public if the paper is accepted?\n\n**A3**: Yes. The code and checkpoints will be made public once accepted."}, {"review_id": "mMZm2PXD8D", "reviewer": "Reviewer_5AuH", "summary": "This paper addresses the TSE problem in a challenging setting: the target user cannot provide their enrollment audio in a clean acoustic environment, but have to provide samples in the noisy condition with background noise and interference speakers. It is a very interesting topic in speech that's rarely discussed, but is of actually good use cases in reality. The contribution is mainly about training an Encoder Fusion Module in the first stage, then training an Extraction Fusion Module in a second stage. The second stage is pretty standard. The first stage is very interesting.", "questions": "What is the mechanism to remove Positive Interferer (PI) in the Encoder Fusion Module? This part is very unclear to me. Both the model architecture or the loss function of stage 1 do not seem to have a built-in mechanism to remove positive interferer. I'd like to see more clarification or explanation on this.\n\nLine 173: I assume T_pos is number of frames of positive enrollment audio? Worth clarifying.", "limitations": "The real limitation of this work is that it is not evaluated with any realistic data. This should be clearly stated.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nPaper is very well written. Easy to read in general except for the Encoder Fusion Module section. Figure 1 and Figure 2 are super helpful. \n\nAblation studies are very well designed, and answered important questions, i.e. Effectiveness of the two-stage model training, and Cross-Attention based Fusion over Film Fusion Method. The experiments on inaccurate Positive and Negative Labeling are very helpful.\n\nThe Encoder Fusion Module work (stage 1) is novel. I never saw similar work.\n\nThe assumption about the start/end time being unique is easy to satisfy in reality.\n\n\nWeaknesses:\n\nLine 175: For the entire \"The Encoder Fusion Module\" section, this is the most important part, but it is a bit difficult to follow. I would suggest using equations wherever possible. Also, consider adding a new diagram explaining how this module works.\n\n\"Model performance under inaccurate Positive and Negative Labeling\" - it is really great to have this section. But I think it can be improved. First, 0.30 ± 0.19 seconds do not seem to be super inaccurate. What if the speaker boundary errors are above 1 second? Also, I am not sure I understand Figure 6. What is x-axis? Each column is one sample? Why the red square can be sometimes high (sample 7) and sometimes low (sample 10)? Please consider remaking this Figure or replace with a table.\n\nAll the experiments are based on synthetic datasets. Although it is common, it's not ideal, especially this paper aims to addess a special use case in real world applications.", "was_revised": true, "final_justification": "The authors have addressed all of my questions. My final rating is 5.", "rebuttal": "**Q1**: Line 175: For the entire \"The Encoder Fusion Module\" section, this is the most important part, but it is a bit difficult to follow. I would suggest using equations wherever possible. Also, consider adding a new diagram explaining how this module works. Line 173: I assume T_pos is number of frames of positive enrollment audio? Worth clarifying.\n\n**A1**: Thank you for your suggestions. We provide further explanation on the Fusion module in the response to reviewer iQ1f Q5, and will include the pseudo-code of the module in the revised version. T_pos indeed represents the number of STFT frames in the Positive Enrollment.\n\n\n**Q2**: \"Model performance under inaccurate Positive and Negative Labeling\" - it is really great to have this section. But I think it can be improved. First, 0.30 ± 0.19 seconds do not seem to be super inaccurate. What if the speaker boundary errors are above 1 second? Also, I am not sure I understand Figure 6. What is x-axis? Each column is one sample? Why the red square can be sometimes high (sample 7) and sometimes low (sample 10)? Please consider remaking this Figure or replace with a table.\n\n**A2**: Thank you for your suggestions. In the user study on inaccurate positive and negative labeling, among the 100 user-provided labels across 10 samples, 18 labels for audio samples [1, 3, 4, 6, 10] deviate more than one second from the ground truth boundaries. We report the average model SI-SNRi on these inaccurate labels in the table below. Sample [1,3,4,6] all show less than 1 dB SI-SNRi performance degradation compared to the model performance using ground truth boundaries, despite a temporal labeling inaccuracy of 1.36 $\\pm$ 0.44 seconds. For sample 10, only one user label deviates by more than one second, reaching 4.1 seconds. This results in 5.16 dB SI-SNRi. Overall, these results demonstrate the robustness of our model to inaccuracies in user-provided boundary annotations.\n\n| sample ID  | 1 | 3 | 4 | 6 | 10 |\n|----------------|--------|-------|-------|--------|--------|\n| Ground Truth Labeling    | 10.98 | 8.38 | 11.71 |12.31 | 11.81 |\n| User Labeling (> 1s deviation) | 10.89 | 8.35 | 11.62 | 11.90 | 5.16 | \n\nRegarding Figure 6, you are correct that the x-axis represents the sample ID, and each column corresponds to the model performance on an individual sample. The red markers indicate outliers, which appear above or below the box plot when a model’s performance on one of the 10 user labelings for that sample exceeds the upper quartile or falls below the lower quartile, respectively. To improve clarity, we will revise Figure 6 as a table reporting the mean and standard deviation of model performance for each sample.\n\n**Q3**: All the experiments are based on synthetic datasets. Although it is common, it's not ideal, especially this paper aims to addess a special use case in real world applications.\n\n**A3**: Thank you for mentioning this. Indeed, to effectively demonstrate our model's performance under strong overlap between the target and interfering speakers, all audio mixtures used in our original submission were synthetically generated or manually added up from real-world audio recordings. We additionally compare our model and the baseline methods on naturally recorded real-world audio mixtures. Please refer to the response to reviewer 1MM1 Q1 for the result.\n\n**Q4**: What is the mechanism to remove Positive Interferer (PI) in the Encoder Fusion Module? This part is very unclear to me. Both the model architecture or the loss function of stage 1 do not seem to have a built-in mechanism to remove positive interferer. I'd like to see more clarification or explanation on this.\n\n**A4**: Sorry for the confusion. We based the model Encoder Fusion Module on self-attention layers. The self-attention operation allows model to perform pairwise comparison between the embeddings of different frames in the Positive Enrollment. As a result, this architecture design allows the model to identify the speakers who do not talk in some of the Positive Enrollment frames and exclude them in the extracted embedding. To guide the model exclude PI's characteristic in its embedding, we simulated the Positive Enrollments so that the number of PI varies between 0 to 2 in the stage 1 training. Distilling the encoder to only encode the target speaker's voice in the Positive Enrollment forces the model to exclude the PI's voice characteristics in its extraction. Please see the response to reviewer iQ1f Q5 for more details."}]

Author rebuttal:

Meta-review: In this work, the authors tackle the problem of extracting a target speaker's voice from a mixture when clean reference audio is not available. They propose a method that learns speaker characteristics from noisy enrollment signals, which include both speech and silence segments.

Key strengths include:
(i) A novel enrollment strategy that leverages paired positive and negative samples to address monaural target speaker extraction under noisy conditions;
(ii) Comprehensive experiments demonstrating that the proposed approach outperforms prior methods, achieving over 2.1 dB improvement in SI-SNRi and significantly faster training convergence.

Main weaknesses:
The presentation quality could be improved. Conceptually, the idea is akin to negative sampling, which is not entirely new and may be considered incremental. The experiments presented during the rebuttal phase were understandably absent from the initial submission and should be incorporated into the revised manuscript before publication.

The rebuttal phase was productive, with both authors and nearly all reviewers actively engaging in discussion. Unfortunately, Reviewer anep, despite providing a solid initial review and my requestes to participate in the discussion, did not participate in the discussion. Nevertheless, their comments were addressed by the authors. Overall, reviewers’ feedback primarily focused on the need for additional experiments to strengthen the empirical validation. The authors conducted these experiments as requested. In reaching my recommendation, I placed less weight on Reviewer anep's review due to their lack of engagement during the discussion phase. While I agree with Reviewer iQ1f’s observation that the approach is not fully novel, I believe the work merits acceptance as a poster presentation. Notably, some reviewers increased their scores following the discussion.
Decision: Accept (poster)