Title: Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments
Venue: NeurIPS 2025
Accepted: yes
Keywords: Target Speech Extraction

Few-Shot Inference
Abstract: Target speaker extraction focuses on isolating a specific speaker's voice from an audio mixture containing multiple speakers. To provide information about the target speaker's identity, prior works have utilized clean audio samples as conditioning inputs. However, such clean audio examples are not always readily available. For instance, obtaining a clean recording of a stranger's voice at a cocktail party without leaving the noisy environment is generally infeasible. Limited prior research has explored extracting the target speaker's characteristics from noisy enrollments, which may contain overlapping speech from interfering speakers. In this work, we explore a novel enrollment strategy that encodes target speaker information from the noisy enrollment by comparing segments where the target speaker is talking (Positive Enrollments) with segments where the target speaker is silent (Negative Enrollments). Experiments show the effectiveness of our model architecture, which achieves over 2.1 dB higher SI-SNRi compared to prior works in extracting the monaural speech from the mixture of two speakers. Additionally, the proposed two-stage training strategy accelerates convergence, reducing the number of optimization steps required to reach 3 dB SNR by 60\%. Overall, our method achieves state-of-the-art performance in the monaural target speaker extraction conditioned on noisy enrollments. Our implementation is available at
https://github.com/xu-shitong/TSE-through-Positive-Negative-Enroll .


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the TSE problem in a challenging setting: the target user cannot provide their enrollment audio in a clean acoustic environment, but have to provide samples in the noisy condition with background noise and interference speakers. It is a very interesting topic in speech that's rarely discussed, but is of actually good use cases in reality. The contribution is mainly about training an Encoder Fusion Module in the first stage, then training an Extraction Fusion Module in a second stage. The second stage is pretty standard. The first stage is very interesting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nPaper is very well written. Easy to read in general except for the Encoder Fusion Module section. Figure 1 and Figure 2 are super helpful. \n\nAblation studies are very well designed, and answered important questions, i.e. Effectiveness of the two-stage model training, and Cross-Attention based Fusion over Film Fusion Method. The experiments on inaccurate Positive and Negative Labeling are very helpful.\n\nThe Encoder Fusion Module work (stage 1) is novel. I never saw similar work.\n\nThe assumption about the start/end time being unique is easy to satisfy in reality.\n\n\nWeaknesses:\n\nLine 175: For the entire \"The Encoder Fusion Module\" section, this is the most important part, but it is a bit difficult to follow. I would suggest using equations wherever possible. Also, consider adding a new diagram explaining how this module works.\n\n\"Model performance under inaccurate Positive and Negative Labeling\" - it is really great to have this section. But I think it can be improved. First, 0.30 ± 0.19 seconds do not seem to be super inaccurate. What if the speaker boundary errors are above 1 second? Also, I am not sure I understand Figure 6. What is x-axis? Each column is one sample? Why the red square can be sometimes high (sample 7) and sometimes low (sample 10)? Please consider remaking this Figure or replace with a table.\n\nAll the experiments are based on synthetic datasets. Although it is common, it's not ideal, especially this paper aims to addess a special use case in real world applications.\n\nQUESTIONS:\nWhat is the mechanism to remove Positive Interferer (PI) in the Encoder Fusion Module? This part is very unclear to me. Both the model architecture or the loss function of stage 1 do not seem to have a built-in mechanism to remove positive interferer. I'd like to see more clarification or explanation on this.\n\nLine 173: I assume T_pos is number of frames of positive enrollment audio? Worth clarifying.\n\nLIMITATIONS:\nThe real limitation of this work is that it is not evaluated with any realistic data. This should be clearly stated.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the target speaker extraction (TSE) problem where the task is to extract the target speaker's voice from an audio mixture provided that some enrollment audio segments are given as the conditional input to the TSE model. Different from prior works that typically assume availability of the clean speech of the target speaker for the enrollment audio, this paper focuses on the more realistic scenario where the enrollment audio segments can contain both the target speaker's voice and interfering speakers' speech as well as background noise. The main idea is to utilized positive and negative noisy enrollments by exploiting the temporal misalignment of speech signals. To this end, a two-branch encoder design and an associated two-stage training strategy are introduced to effectively learn the extraction functionality. Experimental results on synthetic audio mixture data are provided to demonstrate the effectiveness of the proposed method over existing TSE models that have not sufficiently leveraged noisy enrollments as the conditioning input for realistic use cases.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The problem that the paper aims to address is interesting and important, i.e., TSE with noisy enrollments, which is an essential step towards more realistic scenarios and applications of target speaker extraction.\n\n2. The main idea of exploiting the temporal misalignment of voices from different speakers seems novel and makes sense, which also leads to effective TSE with noisy enrollments.\n\n3. Extensive ablation studies on one-stage vs. two-stage training schemes, different enrollment segment lengths, accuracy of the labeling, etc. offer further insights into understanding the behavior of the proposed approach.\n\n4. The paper is well-written and easy to follow, with code and audio samples provided to facilitate reproducibility and qualitative comparison.\n\n**Weaknesses**\n\n1. The experimental results are mostly relying on artificially mixed audio data. Given that this paper aims to address the limitation of prior works for real-world TSE applications, it is expected to see more evaluation results of the method on naturally recorded mixture audio, to further confirm its effectiveness. Although there are some relevant results presented in Appendix K, it seems that the mixture signals in the experiment are still manually added up. In addition, there is no comparison with other existing approaches.\n\nQUESTIONS:\n1. The loss function for training the extraction branch, i.e., eq. (3) seems relatively simple -- it is just the SNR between the ground truth signal and the output of the TSE model. I wonder if this would lead to sub-optimal performance and what motivated you to use this loss? Also, have you experimented with other more sophisticated loss functions, e.g., time-frequency domain MSE losses or some other perceptually motivated losses?\n\n2. Will you make the code public if the paper is accepted?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses monaural target speaker extraction using only noisy enrollment audio. Instead of requiring clean clips, the authors propose using both Positive (target speaking) and Negative (target silent) segments from noisy enrollments. An attention-based Siamese Encoder Fusion Module generates target-speaker embeddings to guide a TF-GridNet-based Extraction Fusion Module. The two-stage training first distills embeddings from clean speech, then trains the extraction network, leading to faster convergence. The method achieves state-of-the-art results in this setting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper proposes a novel enrollment strategy using paired positive and negative enrollments for addressing monaural target speaker extraction with noisy enrollment.\n2. The paper demonstrates robust experiments and comparisons to baseline models.\n3. The authors use thorough evaluation on their dataset using objective metrics (SNR, SI-SNR, STOI, DNSMOS) on both monaural and binaural audio.\n4. The paper has shown extensive ablation studies on using two-stage training strategy and different fusion strategies to encode the target speaker’s characteristics.\n\nWeaknesses:\n1. I think it is important to report performance when a clean enrollment is available. Without this, we cannot judge whether the noisy strategy matches or degrades relative to the ideal case.\n2. In my view, comparing only to SpeakerBeam for the monaural task is too weak. Including recent SOTA discriminative and generative TSE methods (e.g. USEF-TSE, SoloAudio) would give a fairer picture of the gains.\n3. The current metrics (SNR, SI-SNR, STOI, DNSMOS) omit key measures of intelligibility (WER or CER) and perceptual quality (e.g. PESQ). I think adding these would strengthen the assessment.\n4. Objective gains do not always reflect perceptual quality. I would like to see human listening tests or a comparison demo to confirm that improvements hold up for end users.\n5. Some notation in Section 3.3 (encoder fusion) is dense. A short pseudo-code snippet or an extra diagram could help readers grasp the self-attention comparison more easily.\n\nQUESTIONS:\nSee Weaknesses for details.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel target speaker extraction (TSE) model capable of utilizing mixture enrollments with multiple speakers for effective extraction of the target speaker's voice. The enrollment feature is first enhanced by extracting positive and negative enrollment representations from segments with and without the target speaker respectively. An self-attention mechanism is then employed to filter out interference speakers from the enrollment feature, resulting in a fused enrollment feature focused on the target speaker. It is finally integrated into the TSE model via cross-attention to extract the target speaker's voice. The proposed model is mostly based on the TF-GridNet model architecture, and the core contribution is the ability of utilizing mixture enrollments for effective TSE. Experiments are conducted on Librispeech and WSJ0 (Appendix) based simulation datasets as well as a real-world dataset based on VoxConverse, which validates the efficacy of the proposed model, especially in scenarios with multiple speakers in the enrollment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The problem addressed by the proposed method is important and novel, which is highly related to real-world applications.\n2. The proposed model is compact and performant, achieving better performance in complex scenarios with mixture enrollments with only 1.88 M parameters, which is smaller than baseline models such as TCE and LookOnceToHear.\n3. Extensive experimental validation and ablation studies are provided on both simulated and real-world datasets, validating the model's robustness and generalizability.\n\nWeaknesses:\n1. Some technical details are missing, such as the definition and implementation of the learnable segmentation embeddings in the Encoder Fusion Module.\n2. The organization of the paper could be improved by moving some key experiments (e.g., Appendix K) to the main text to better highlight the effectiveness of the proposed method.\n3. Some claims lack a solid experimental validation, such as line 323, which should be rephrased or supported by additional experiments.\n4. The writing needs improvement, as some sentences are overly complex or even grammatically incorrect, leading to poor readability.\n5. More advanced TSE baselines, such as X-TF-GridNet, instead of the outdated SpeakerBeam model, should be compared in the experiments to better assess the performance of the proposed method.\n\nQUESTIONS:\n1. When the input signal is binaural (as described in Section 4.1), it is unclear whether the proposed model is capable of modeling the spatial information between two microphone channels, as Section 3 only discusses monaural scenarios.\n\n2. Table 1 primarily evaluates the extraction performance when there are multiple speakers in the enrollment, which matches the motivation of this paper. However, it is still necessary to report the performance with single-speaker noisy enrollments, as this constitutes a large percentage of real-world scenarios. In Appendix J, it is shown that the proposed model underperforms TD-SpeakerBeam with single-speaker clean enrollments. But it is still unclear whether the same tendency will be observed for noisy enrollments.\n\n3. While line 323 claims that the model “correctly extracts the target speaker” based on the improved performance shown in Figure 5, it is not explicitly verified. In particular, the target confusion problem [R1] is commonly observed in target speaker extraction systems and can be analyzed quantitatively. Therefore, it is highly recommended that the authors present in-depth analysis on the accuracy of target speaker extraction, which can be more difficult in the context of mixture enrollments as discussed in this paper.\n\n[R1] Z. Zhao, D. Yang, R. Gu, H. Zhang, and Y. Zou, “Target confusion in end-to-end speaker extraction: Analysis and approaches,” in Proc. ISCA Interspeech, 2022, pp. 5333–5337.\n\n4. There are two major limitations of the experiments:\n    1. Only the non-intrusive DNSMOS metric is used for the evaluation on real-world speech data in Appendix K. It is nice to see evaluation on real-world data. However, as reported by various prior studies, solely using an NN-based non-intrusive metric for evaluation can be risky as they may not reflect the true speech quality due to domain mismatch. Additional evaluation metrics such as word error rate (WER), speaker similarity, or even MOS based on subjective listening tests should be additionally reported for more comprehensive assessment.\n    2. There is no demo page or subjective listening to subjectively validate the performance of the proposed method. For example, the artifacts mentioned in line 365 cannot be easily understood without an example.\n\nLIMITATIONS:\nThe potential negative societal impact of this work is not discussed in Appendix N. It is recommended that the authors include a discussion on the potential misuse of the proposed method and how they can avoided. For example, the proposed method could raise privacy concerns if used to extract the voice of a target speaker without their consent.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}