Title: Improving Target Sound Extraction via Disentangled Codec Representations with Privileged Knowledge Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Target Sound Extraction

Privileged Knowledge distillation

Disentangled Representation Learning

Neural Audio Codec

Feature-level Knowledge Distillation
Abstract: Target sound extraction aims to isolate target sound sources from an input mixture using a target clue to identify the sounds of interest. To address the challenge posed by the wide variety of sounds, recent work has introduced privileged knowledge distillation (PKD), which utilizes privileged information (PI) about the target sound, available only during training. While PKD has shown promise, existing approaches often suffer from overfitting of the teacher model for the overly rich PI and ineffective knowledge transfer to the student model. In this paper, we propose Disentangled Codec Knowledge Distillation (DCKD) to mitigate these issues by regulating the amount and the flow of target sound information within the teacher model. We begin by extracting a compressed representation of the target sound using a neural audio codec to regulate the amount of PI. Disentangled representation learning is then applied to remove class information and extract fine-grained temporal information as PI. Subsequently, an n-hot vector as the class information and the class-independent PI are used to condition the early and later layers of the teacher model, respectively, forming a regulated coarse-to-fine target information flow. The resulting representation is transferred to the student model through feature-level knowledge distillation. Experimental results show that DCKD consistently improves existing methods across model architectures under the multi-target selection condition.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel framework, Disentangled Codec Knowledge Distillation (DCKD), to improve Target Sound Extraction (TSE). Recent methods use Privileged Knowledge Distillation (PKD), where a \"teacher\" model trained with extra information (PI) guides a \"student\" model. However, existing PKD approaches often suffer from the teacher model overfitting to overly rich PI, leading to ineffective knowledge transfer. DCKD addresses this by regulating the PI's amount and flow. It first uses a neural audio codec to create a compressed representation of the target sound. It then employs Disentangled Representation Learning (DRL) to separate this representation into a static (class-related) factor and a dynamic (class-independent temporal) factor. The teacher model is conditioned in a coarse-to-fine manner: it receives general class information in early layers and the disentangled temporal PI in later layers. This prevents overfitting and facilitates effective feature-level knowledge distillation to the student model. Experimental results show that DCKD consistently improves separation performance across different TSE model architectures, demonstrating that regulating and disentangling privileged information enables more effective knowledge transfer and better student model performance.\n\nSTRENGTHS AND WEAKNESSES:\n**[Strengths]**\n1. This work proposes a novel method for regulating privileged information using a neural codec for compression and disentangled learning to isolate class-independent temporal details for more effective knowledge transfer, which mitigates the teacher overfitting issue and ensures better student model generalization.\n2. The method's effectiveness is empirically validated across multiple distinct model architectures, demonstrating its robustness and general applicability for improving various target sound extraction systems, not just a single setup.\n\n**[Weaknesses]**\n1. Clarity. The main motivation of this paper is to mitigating the teacher overfitting issue by disentangling the class-independent temporal details. If my understanding is correct, the overfitting issue of PKD is mainly resulted by the class-dependent informations (the shared features of $l$ and $x$). But currently, the text in Section 1 and 3 does not clearly describe this point, which may raise difficulties in understanding the motivation of this paper quickly. For example, when I read \n2. Some out-date modules. Some modules in Figure 2 (a) adopts out-date modules like LSTM and TCN. A minor suggestion is that the authors may try to compare current backbones with Transformer based architecture for the Disentangled Codec and Diffusion Transformer based architecture for the teacher and student model.\n3. The baseline selection issue. The authors compare their work with baselines like SoundSelector and WaveFormer in Section 4.3. I agree that this experiment demonstrate the effectiveness of the proposed DCKD method. However, the comparison beyond the proposed framework is missing. For example, the authors could add comparisons with baselines based on other frameworks like SoloAudio [1], AudioSep [2]  (based on Language-oriented TSE), which would make this work more convincing.\n4. Since the concept of global and local features, mutual information-based disentanglement, privilege knowledge distillation is widely used in the speech fields, the main contribution of this work are identifying and  solving the teacher overfitting issue. \n\n\n**Reference:**  \n[1] SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer, Helin Wang et al.  \n[2] Separate Anything You Describe, Xubo Liu et al.\n\nQUESTIONS:\nMy questions are listed in the weakness part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis papers propose a novel privileged knowledge distillation framework for target sound extraction. The authors proposes a coarse-to-fine conditioning mechanism to mitigate overfitting by providing class-level information to earlier layers and class-independent information to later layers. The experimental results demonstrates the effectiveness of the proposed framework in improving the performance of the SoundSelector-based and Waveformer-based student model.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The thorough ablation study demonstrates the necessities of each proposed component.\n- The experiment results are well-organized and clearly explained.\n\n### Weaknesses\n\n- The proposed method was only evaluated on one dataset and one backbone model for Table 2 and 4 (two backbone models for Table 3). Additional experiments on another dataset would help examine the generalizability of the proposed framework.\n- The error bars are missing, making it hard to see if any of these differences are significant.\n- The effectiveness of the feature-level KD loss is unclear as the performance remains almost the same when $L_{KD}^{feature}$ is removed.\n\nQUESTIONS:\n- (Line 73) \"achieving notable gains under acoustically challenging conditions.\" -> It's unclear what \"acoustically challenging conditions\" mean here. Please be specific.\n- (Figure 2) It would be helpful to highlight which components are being trained and which are frozen.\n- (Line 111) $l \\in \\mathbb{R}^M$ -> Should it be $l \\in [0, 1]^M$ to be more precise?\n- (Line 146-147) \"To further remove class information while preserving fine-grained temporal information, we apply instance normalization,\" -> It is not straightforward to me how instance normalization can remove class information. Please clarify this.\n- (Line 156-157) \"To address this, we guide the static factor to retain class-relevant information by minimizing the cosine distance to the embedding vector of the n-hot class label.\" -> If I understand it correctly, this will make the static factor into class-level features rather than instance-level features, unlike the claimed \"temporal-variant\" vs \"temporal-invariant\" features.\n- (Table 2) How about the Waveformer architecture? Did you observe a similar trend?\n- (Line 331) \"Lastly, the removal of feature-level KD loss led to non-trivial performance drops,\" -> Is this true? The numbers are super close (10.40/9.42 vs 10.40/9.43). Am I missing something here?\n- (Line 360-361) \"DRL contribute significantly to improving performance over the baseline.\" -> Please weaken this claim as there's no significance test.\n\nLIMITATIONS:\nDiscussion on the limitations is missing. -> The authors have promised to extend the discussion on the limitations in the revised manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for Target Sound Extraction in a PKD frameword that overcomes the possibility of overfitting that usually happens in PKD. The key contributions are generating a disentangled representation of the target sound. The disentanglement also acts as a regularization that helps avoid overfitting. In order to learn meaningful disentanglement, mutual information objectives are used.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The problem is well-motivated and the proposed solution is explained clearly. Overall the paper is well-written and easy to understand.\n2. The authors provide ablation studies explaining the effect of each of the proposed component.\n3. The problem the paper is trying to solve is significant: reducing overfitting in PKD\n4. Although the proposed methodologies are not novel, it is a good solution to the problem in hand.\n\nWeakness:\n1. The authors do not compare the proposed method with other TSE models/baselines from related work. From my understanding, table 3 only compares distillation based methods w/wo the proposed solution.\n2. The experimental section is also limited to one synthetic dataset that the authors curate. To understand the true potential of this model, we need to experiment with more datasets.\n\nQUESTIONS:\n1. What makes this problem definition only limited to the scope of TSE? Is the solution generalizable to any domain or are we making certain assumptions to make the solution work only for TSE.\n2. What is the effect of the codec model on the performance? It would be nice to analyze a variety of neural codec models and understand its impact.\n\nLIMITATIONS:\nI was not able to find a limitations section in the paper. In the checklist, it is mentioned that section 6 has the limitations, but I am not able to find a section 6.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel Disentangled Codec Knowledge Distillation (DCKD) framework for target sound extraction (TSE). TSE is the task of isolating a specific sound source from a mixture given a clue about the target (e.g., class label or reference). The authors leverage privileged knowledge distillation (PKD) – a strategy where a teacher model has access to additional information (privileged information, PI) during training that the student model doesn’t get at test time. The key idea in DCKD is to carefully regulate the amount and flow of target information provided to the teacher: they compress the target audio through a neural audio codec to limit information, and apply disentangled representation learning (DRL) to separate class-related vs. fine-grained details. The teacher model is trained with both the usual target class label (an n-hot vector indicating which sound classes to extract) and the disentangled codec representation of the target audio as PI. The student model, which at test time only receives the class label, learns from the teacher via feature-level knowledge distillation. Experiments on a multi-target sound extraction dataset (synthetic mixtures from FSD Kaggle2018 + TAU Urban Acoustic Scenes 2019) show that DCKD yields consistent improvement in signal-to-distortion ratio (SDR) for the student model across different architectures and difficult scenarios. Notably, DCKD outperforms prior PKD approaches that use richer PI (like full target audio) by avoiding teacher overfitting and achieving better student generalization. \n\nIn summary, the DCKD approach integrates these pieces: privileged info in the form of a compressed, disentangled target audio representation, delivered to the teacher in a coarse-to-fine manner and then apply feature level knowledge distillation to train the student.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\nThis work extends PKD to a new domain (general sound separation) with a much richer form of PI (actual audio content) than prior TSE studies. The coarse-to-fine conditioning strategy is insightful it acknowledges that not all information should be dumped in at once. By structuring the flow (class clue first, then details), the teacher’s utilization of privileged data is more controlled. This is a non-trivial improvement over earlier PKD approaches that might simply concatenate an extra feature and risk overfitting. \n\nThe authors validate DCKD on two model architectures and different numbers of target sources. The consistent gains (approximately +1 to +2 dB SI-SDRi across the board) show the method is not tied to a specific network or scenario. This implies the approach will be robust under a range of scenarios. Moreover, demonstrating improvement even for a lightweight real-time model (Waveformer) is important: it indicates DCKD can help in practical settings where computational resources are limited. \n\nThe inference overhead is also low since student model has the same input requirements and the model size is same as the TSE model where the approach improves the performance without increasing the runtime cost or the model size for the deployment. \n\nWeakness: \n\nComputational inefficiency: The DCKD framework is quite complex. It requires training a pretrained neural codec, a DSDE module with multiple loss terms and and a large teacher model, followed by a student. This is overall computationally heavy in case the models are not available. The teacher training runs for up to 500 epochs. In comparison, simpler PKD methods (like using just a timestamp) are much easier to implement and train. The authors acknowledge the computational inefficiency as a limitation and propose to work on a lightweight compression model in future. \n\nThe privileged data used here is specifically audio of the target source itself. This is a strong assumption – essentially, one needs isolated target recordings for training as PI (which they simulate by knowing the mix composition). In real applications, obtaining a clean example of the exact sound one wants to extract (for training) may not be straightforward. By contrast, using timestamps or class names as PI (like prior works) doesn’t require providing the sound itself. The paper demonstrates the superiority of using the target audio as PI (via codec) over those simpler PI types but it doesn’t solve how to get such audio in a realistic setting unless synthetic data or extra sensors are available. If one has a library of isolated sounds for each class, one could use that, but then generalizing to new sounds not in the library might be an issue. In real life, training with real mixtures might not come with ground-truth isolated tracks unless it’s a task like speech where data can be simulated. This is not a flaw in the paper but rather a technical limitation on the application of this method. Would be great to see author's comments on this aspect. \n\nClarification needed: The authors did not report standard deviations or statistical significance. Given that they use a fixed test set of 5000 mixtures, this is fine, but some indication of variance (perhaps via multiple runs) could strengthen confidence. In the NeurIPS checklist, they marked that they reported statistical significance, but in the paper’s content, we mainly see mean SDRi. However, since improvements are fairly large relative to any plausible variance, this is a minor issue.\n\nQUESTIONS:\nThe paper should better explain how the privileged information is integrated into the teacher model. Currently, it’s only stated that the n-hot label conditions early layers and the disentangled dynamic factors condition later layers. However, it’s unclear how this conditioning is implemented. The architecture details should be cleared in the supplementary material for the paper if not in the main paper to share the details. \n\nThe authors themselves note that the use of DAC introduces a significant computational increase. It would be great if authors can detail the overhead from model size, training time details and discuss the computational requirements.  For instance, how much longer (in epochs or wall-clock time) does training DCKD take compared to a baseline? Was the teacher model significantly larger in parameters or just in input dimension? \n\nThe proposed PKD framework assumes access to isolated target audio examples for every sound class during training. In the synthetic experiment this is feasible (mixtures are generated with known sources), but in real-world scenarios it might be hard to obtain clean target samples for every sound of interest. For example, is it assumed that one has a collection of prototypical sounds for each class (as in FSD Kaggle) to use as PI? If so, the authors should clarify this and perhaps note that the method is limited to closed-set target classes. Relatedly, how well would the student model generalize to a new sound class that was not seen in training or had no privileged example? Since the student only uses class labels, in principle it could handle new classes if trained appropriately – but without a teacher for those, it’s unclear.\n\nLIMITATIONS:\nThe paper itself notes one main limitation: the reliance on the heavy DAC (neural codec) and the additional training cost. A future direction they plan is to design a lightweight feature compression method to serve the role of DAC. Perhaps a smaller autoencoder trained jointly could replace the off-the-shelf codec, or dimensionality reduction techniques could be used to create a bottleneck without a full codec.\n\nIt would also be interesting to apply DCKD to speech domain explicitly (target speaker extraction). In that case, the privileged info could be a clean recording of the target speaker’s voice. Actually, that scenario is common (speaker extraction often assumes a reference voice recording is available at runtime). But if we assume a case where at runtime you only know the speaker’s identity or name (not a voice snippet), one could train a teacher with a reference voice audio (privileged) and then distill a student that only needs speaker ID (like a class label). This mirrors what DCKD does (because in DCKD student only gets class label, teacher heard the actual voice). This could allow speaker extraction without needing enrollment audio at test time, which is a powerful capability. The technique would be largely the same (codec + DSDE on the voice). This cross-application potential is worth exploration.\n\nReal world application: The dataset here is synthetic. Real recordings might have more variation (reverberation, non-additive noise, etc.). The method might need to prove itself on real sound separation benchmarks (though few exist for arbitrary classes). Possibly, using datasets like AVE (Audio-Visual Event) or overlapping ESC-50 might be a next step. The reliance on a known set of classes and available example recordings per class is a constraint – expanding beyond class-conditioned separation to, say, text-based queries (like “extract the sound of glass breaking”) could be a future extension. Perhaps a language model could generate an “imagined” audio representation as privileged info for a teacher, but that’s quite speculative.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}