Weakly-supervised Audio Separation via Bi-modal Semantic Similarity
Authors: Tanvir Mahmud
Saeed Amizadeh
Kazuhito Koishida
Diana Marculescu
Venue: ICLR 2024 poster (2024)
Keywords: Audio-language learning
conditional audio separation
unsupervised learning
weakly supervised learning
semi-supervised learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a weakly supervised learning framework for conditional audio separation that significantly outperforms the baselines in unsupervised and semi-supervised settings.

Abstract: Conditional sound separation in multi-source audio mixtures without having access to single source sound data during training is a long standing challenge. Existing mix-and-separate based methods suffer from significant performance drop with multi-source training mixtures due to the lack of supervision signal for single source separation cases during training. However, in the case of language-conditional audio separation, we do have access to corresponding text descriptions for each audio mixture in our training data, which can be seen as (rough) representations of the audio samples in the language modality. That raises the curious question of how to generate supervision signal for single-source audio extraction by leveraging the fact that single-source sounding language entities can be easily extracted from the text description. To this end, in this paper, we propose a generic bi-modal separation framework which can enhance the existing unsupervised frameworks to separate single-source signals in a target modality (i.e., audio) using the easily separable corresponding signals in the conditioning modality (i.e., language), without having access to single-source samples in the target modality during training. We empirically show that this is well within reach if we have access to a pretrained joint embedding model between the two modalities (i.e., CLAP). Furthermore, we propose to incorporate our framework into two fundamental scenarios to enhance separation performance. First, we show that our proposed methodology significantly improves the performance of purely unsupervised baselines by reducing the distribution shift between training and test samples. In particular, we show that our framework can achieve 71% boost in terms of Signal-to-Distortion Ratio (SDR) over the baseline, reaching 97.5% of the supervised learning performance. Second, we show that we can further improve the performance of the supervised learning itself by 17% if we augment it by our proposed weakly-supervised framework. Our framework achieves this by making large corpora of unsupervised data available to the supervised learning model as well as utilizing a natural, robust regularization mechanism through weak supervision from the language modality, and hence enabling a powerful semi-supervised framework for audio separation. Code is released at https://github.com/microsoft/BiModalAudioSeparation.

Published as a conference paper at ICLR 2024
WEAKLY-SUPERVISED AUDIO SEPARATION VIA BI- MODAL SEMANTIC SIMILARITY
Tanvir Mahmud t Saeed Amizadeh Kazuhito Koishida" & Diana Marculescu' 'The University of Texas at Austin, USA, "Microsoft Corporation sanvirmammu, dianam] edu, (saamizad, kazukeljemdcrosoft.com
ABSTRACT
Conditional sound separation in multi-source audio mixtures without having ac- cess to single source sound data during training is a long standing challenge. Ex- isting mix-and-separre based methods suffer from significant performance drop with multi-source training mixtures due to the lack of supervision signal for sin- gle source separation cases during training. However, in the case of language- conditional audio separation, we do have access to corresponding text descrip- tions for each audio mixture in training data, which can be seen (rough) representations of the audio samples in the language modality. That raises the curious question of how to generate supervision signal for single-source audio ex- traction by leveraging the fact that single-source sounding language entities can be easily extracted from the text description. To this end, in this paper, we pro- pose a generic bi-modal separation framework which can enhance the existing unsupervised frameworks to separate single-source signals a target modality (i.e.. audio) using the easily separable corresponding signals in the conditioning modality (i.e., language), without having access to single-source samples the target modality during training. We empirically show that this is well within reach if we have access to a pretrained joint embedding model between the two modalities (i.e., CLAP). Furthermore, into two fundamental scenarios to enhance separation performance. First, we show that our proposed methodology significantly improves the performance of purely unsupervised baselines by reducing the distribution shift between train- ing and test samples. In particular, we show that our framework can achieve 71% boost in terms of -ooisttto- Ratio (SDR) over the baseline, reach- ing 97.5% of the supervised learning performance. Second, we show that we can further improve the performance of the supervised learning itself by 17% if we augment it by our proposed weakly-supervise framework. Our framework achieves this by making large corpora of unsupervised data available to the super- vised learning model as well as utilizing a natural, robust regularization mecha- nism through weak supervision from the language modality, and hence enabling a powerful semi-supervised framework for audio separation. Code is released at https:///itthu com crosoft. Modoal Audi oSeparationn
propose incorporate framework
INTRODUCTION
Environmental sounds often appear in mixtures containing diverse sounding events from differ- ent sources (Kim et al. 2019). Separating sounds from mixtures has significant applications in speech (Liu & Wang) 2018), audio (Chen et al., 2022), and music processing (Stöter et al.) 2019). Humans gather the perception
every single sounding source through experience and interactions with environment. To train sound separation models, most prior works require well-curated, sin- gle source samples (Yu et al. 2017): Sawata et al.) 2021). However, procuring large-scale single source datasets for all possible sounding entities can be quite costly and time-consuming in many real scenarios (Wisdom et al., [2020] Dong et al.) 2022). Nevertheless, classical unsupervised at- tempts separate all sounding sources in mixture typically require complex post-processing to
Work done in part during an internship at Microsoft Corporation, Redmond, USA 1 Equal contribution.

---

Published as a conference paper at ICLR 2024
Mastawd
Unaupervised Mix-ani-S Training Pig
LErR
Nitton
Crwz
Weakly Supervised Treaving (8ea S.B. Fig 31
CoRi Auuee
Twst 1000
(Guter and Thuns
Proposed Framework
Figure 1: (Left) The proposed conditional audio separation framework. (Right) The comparison of our framework and the mix-and-separate baseline in unsupervised and semi-supervises settings. isolate the target source which limits their applicability (Wisdom et al. 2020). In contrast, condi- tional sound separation directly separates the target sound from the input mixture via conditioning signals possibly presented in another modality. Prior work introduced image (Gao & Grauman 2019). video (Zhao et al. 2018), reference audio (Chen et al. 2022). and language conditions (Dong) et al. 2022) to represent the sounds to be separated. Most of these works are built upon the idea of aax--nndssppaace originally introduced by Zhao et al. (2018).
Despite the success of mix-and- and-separate approach, the model does not see single-source cases during training which means that the single-source conditioning signals are not seen at the training time either. This creates a shift between training and test distributions, which, in turn, harms the model's generalization. The problem is further exacerbated by the increasing number of components in the training mixtures, because now the model has to indirectly discover single-source patterns from even more complex training patterns. For instance, as shown in Figure I the performance of mix-and- separate method on 2-component test mixtures drops below 35% of that of the supervised approach when the number of components in training data increases four.
Aside from mix- mix-and-separate, incorporating sound classification has been proposed to generate weak supervision for single source samples during unsupervised training (Tzinis et al. 2020 Pishdadian et 2020). However, such methods suffer from fixed and limited number of dingle-souree sounding entities (i.e., the number of classes) as well as dependency on seeing prototypical, single source examples for each class during training, which severely restricts their applicability in real world scenarios.
similar vein, in this paper, we seek methodology to generate weak supervision, but without having to deal with the restrictions of the classífication based methods. To this end, we propose a wcakly-supervised audio separation framework that relics on incorporating pretrained audio-language embedding models (specifically CLAP (Wu et al., 2023)) into the unsupervised learning training loop. Since models such as CLAP (i) show promising open-ended recognition and zero-shot capabilities, and (ii) are already pretrained on large corpora of audio-language pairs, our proposed framework breaks the classification-base methods' barrier while at the same time (unlike mix-and-separatt)) provides weak supervision for single-source audio separation during training. More generally. the main contributions of our work are:
(1) We propose a weakly supervised source separation framework capable of extracting single- source signals from a mixture in a target modality (e.g., audio) using a conditioning modality (e.g., language) if (i) we easily separate the corresponding entities in the conditioning modal- ity, and (ii) we have access to a pretrained joint embedding model between the two modalities. In this paper, adapt this general framework for segguage-conittioaa audio separation.
(2) We incorporate mix-and-separate, our method achieves up to 71%, 102% and 129% SDR boost on 2-component separation test when trained 2-, 3-, and 4-component mixture data, respectively (Figure I-
framework into pure unsupervised setting and show that, compared to
We further propose to use our methodology to augment supervised learning- based audio separa- tion into a powerful semi-supervised learning (SSL) counterpart. We show that in moderate SSL. scenarios, we achieve 17%, 8%, and 6% SDR boost over supervised learning for 2-component separation when trained on 2-, 3-, and 4-component mixture data, respectively; whereas for ex- treme SSL scenarios, our method still outperforms the supervised training by 8% and 3% when trained on 2- and 3-component mixture data, respectively. (Figure T).
(4) We conduct extensive experiments and ablation studies on MUSIC Zhaoo et al. 2018), VG- GSound (Chen et al. 2020), and AudioCaps (Kim al. 2019 datasets, and report the results.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORKS
Unconditional Sound Separation Most prior work focused on unconditional sound separation
on speech (Wang& & Chen 2018
Yu Yu et al. 2017 Zhang et al. 2021 Luo & Mesgarani] 2018). and 2021 Takahashi & Mitsufuji 2021). For these methods,
music (Stöter et al. 2019 Sawata et al.
post-processing mechanisms have been employed to pick the target sound. Kavalerov et al. (2019) used permutation invariant training (PIT) for universal sound separation, originally introduced by Yu et al. (017). PIT measures permutations of all predictions to match the ground truth sounds. Nevertheless, these methods still need single source samples for training. Later, Wisdom et al. (2020) proposed a mixture invariant training (MixIT) to use multi-source mixtures during training, but suffers from over-separation of sounding sources, and still requires post-processing separate the sounds. Later, Wisdom et al. (202T) proposed to use a pretrained sound-classifier to separate the sounds while training. Such a classifier, however, requires single-source samples for training. MixPIT (Karamath & Kirbiz 2022) proposed direct predictions of mixtures from mixture of mix- tures (MoM) in training but it suffers from under-separation. |Pishdadian et al. (2020) proposed a weakly supervised training by applying sound classifiers on estimated sources; however, the model restricted by the fixed number of classes. In contrast, our framework achieves single-source audio separation with open-ended natural language prompts, without relying on any post-processing or having access to mongle-souree samples during training.
Conditional Sound Separation Prior work on conditional sound separation utilized visual infor- Chatterjee et 2021 [u et al.)
mation (Gao & Grauman 2019 Zhao et 2018) Tian ct al. 2021
2022 Kilgour et al. 2022 Ian et al. 2023 Liu et al. 2023) of sources in mixtures for separation. Most of these methods employ the mix-and-separaa (2022); y) Tan et al. (2023) introduced modality inversion conditioning by leveraging CLIP model with mix-and- -eexarttsepartte, where video/image conditioning is used for training while both video and text can be used for test conditioning. However, with increasing number of sources in training mix- tures, the performance of these methods significantly drop compared to supervised training. [aa & Grauman (2019) introduced sound classifier in mix-and-separate for clustering single source samples. AudioScope (Tzinis et al., 2020) added a classifier in the MixIT baseline for separating on-screen samples based on visual conditioning. Another line of research uses reference audio sig- nal for conditioning in mixture separation (Chen et al., 2022 Gfeller et al.] (021). However, it is generally costly to gather reference signals for single sources. In contrast, our approach can provide competitive results of single source training under completely unsupervised scenarios.
2018). as well as text information (Dong et al. 2022 Liu al.
framework originally introduced by Zhao et al. (2018). Recently, Dong et al.
Contrastive Learning on Audio, Language, and Vision Foundation Models Radford et al. (2021) first introduced CLIP model which learns joint visual- visual-language embedding from 400M
image-text pairs. CLIP has been extensively studied in diverse visual-language applications, such
as zero-shot classification (Radford al. (2021), open-vocabulary segmentation (Luo et al.) 2023
Xu et al. 2023), and object detection (Zareian et al. [021] Du et al. 2022). Guzhov et al. (2022) later integrates audio modality to learn joint representations of audio, vision, and language. Wu
et al. (2023) introduced joint audio-language embedding with CLAP model by large-scale pretrain- ing with 633, 526 audio-caption pairs. In this work, we leverage pretrained CLAP model to generate weak supervision with representative texts in multi-source sound separation training.
THE PROPOSED FRAMEWORK
We propose robust language-conditional sound source separation framework capable of separat- ing single sources from mixtures withoul having access to single-source audio data during train- ing. To this end, we develop generic supervsed training framework to separate/segment a single-source signal from a mixture in modality A conditioned on single-source, concurrent signal in modality T assuming that we have access to (i) single-source signals in modality T. and (ii) pretrained joint embedding model between modalities A and T. If these requirements are met, our framework can learn a conditional source separation model in modality without needing single-source instances from this modality during training. In this sense, proposed framework is supull-supprviiee. In this paper, we focus on the specific scenario where modality AA is audio and modality T is language. As such, requirement (i) boils down to having access to single-source entities in language (e.g., "the sound of violin") which is casily obtainable from the caption associ- ated with audio mixture sample using standard NL.P techniques such Name Entity Recognition
3

---

Published as a conference paper at ICLR 2024
(NER) (Li et al. 2020) combined with basic prompt engineering. Also requirement (ii) is met do have access to language-audin pretrained
such as CLAP (Wu al. (023).
Nevertheless, driving training purely based weak supervision will only get far. In practice, pure weak supervision through language is quite often coarse and therefore produces low fidelity au- dio samples during inference. To overcome this problem, the conventional wisdom is to incorporate some form of reconstruction loss minimization during training. However, traditionally, reconstruc- tion loss relies on having access to supervised training data (i.e., single-source audio samples) which we are trying to circumvent. As a result, here we propose to incorporate the unsupervised version of the reconstruction loss minimization, aka the mix-and-sepant and-separate approach (Zhao et al., 2018 in combination with our weakly-supervised training, as depicted in Figure
3.1 PROBLEM FORMULATION
Let D {(M1, Ti), (M2, T2)...., (M).........))) Tp)} be set of P audio mixture and text description pairs. We assume each audio mixture M; is composed of K > I single source audio sounds S L.e M₁ - St which we do not have access to during training. Furthermore, we assume each audio single source St corresponds to a sounding language entity T, E Tr, which is either given or easily obtainable from the textual prompts Tr. For example, for a music audio mixture accompanying with the textual description "a trio of violin, piano, and cello", the sounding language entities are "violin"" "piinno", and "cello". Given this setting, our goal is to train a neural network model So using DD such that, at inference time, given mixture M and sounding language entity T it extracts the corresponding single-source audio S = fe(M, Th) from M More precisely, we design fe as residual conditional U-Net that operates on the magnitude spectrogram of the input audio mixture:
fo(M, = S "[I((M)) o go () ( ((S(M)]
(I)
where S(.) is the Short Term Fourier Transform (STFT) function, |. and Φ6-) are the magnitude and phase functions, 9o is the masking U-Net function parameterized by 6, E-.(-) is the (frozen) language embedding model, and @ is the Hadamard product. Note that our design choices for modeling fo specifically guided by the nature of the problem at hand. First, our U-Net operates only on the magnitude of the input mixture as it is known that the phase information is not crucial to many audio prediction tasks. Second, we have intentionally chosen the mask architecture for fo as opposed to directly predicting the output in order to encode the inductive bias that the output of fo should be a component of its input mixture. This inductive bias is crucial for the case of weakly supervised learning, as the supervision signal does not contain the fine-graine information necessary for reconstructing the output from scratch.
The masking function go is a conditional U-Net that predicts a (soft) binary mask for an input mag- nitude spectrogram mixture conditioned on the encoding of the text prompt condition (via ey()). We have extended the common audio U-Net architecture to incorporate the conditioning signal at different resolutions of the input by using multi-scale cross attention modules. For more details of the proposed architecture and our architecture ablation study, see Appendices and G.1
3.2 UNSUPERVISED MIX-AND SEPARAEE TRAINING
common methodology to train the neural model presented in equation T is to synthetically mix two or more single-source audio samples at the training time and have the model predict each component by incorporating the é reconstruction loss on the prediction of the model and the original single- source ground truth components. However, in our setting, we do not have access to single-source audio samples during training: instead, we can mix two or more multi-source mixtures and have the model separate them. This is the essence of the unsupervised Mix-and-S and-Saparaaeee approach. In particular, during training, we synthetically mix two or more audio mixtures to form a mixture of mixtures (MoM) which is then fed to the model to separate the original ingredient mixtures.
For example, suppose we have two audio mixtures M1, with description T1 as "a duet of piano and cello", ", and Ma with description To as "a duet of guitar and violin". ". We can mix Mi and M2 to produce the MoM M = M + M2,, which is fed to the model conditioned by either of Ti or T2 to estimate Mj or M2, respectively. Then the unsupervised reconstruction loss (URL) for M is:
(unl(M: II Ti) - Mille, + IIfolM T2) Mallé,
(2)
Note that in practice, the sounding entity is presented prompt phrase to the model, e.g., "the sound of violin" instead of just "violin". For more details of our prompt engineering pipeline, see Appendices G.8 &
4

---

Published as a conference paper at ICLR 2024
The sound mixture of guiter and flute
Predicted Spectrogram (Mixture-1)
Ground Truth Spectrogram [guitar and fluit)
Mixture-1 1 (Gultar and Flute) t
Conditional
Shared
CURL RL
U-Net
Moxture-ffmikbures Spectrogram
Mixtur-22 Drum and Violin)
Predicted Spectrogram (Mixture-2)
Ground Truth Spectrogram (drum and violin)
sound
Z: Language Encoder
Figure 2: Unsupervised mix-and-separeeeeee training with language conditioning (for =, 2)
The generalization of this loss function to more than two components then straightforward.
3.3 WEAKLY SUPERVISED AUDIO-LANGUAGE TRAINING
Despite its merits, the mix-and-separate approach presented in the previous section suffers from a major drawback: due to our unsupervised restriction, the model never sees any single-source conditioning prompt during training; whereas, during inference, most realistic scenarios involve querying for separation of single-source sound from the mixture input. This creates a distribution shift between the training and testing samples which significantly harms the model generalization. Therefore, the main question is, if we introduce single-source text prompts during training, how can we generate supervision withou requiring to see the corresponding single-source audio samples?
Our key idea in response to this question is to use the single-source text prompts themselves for supervision! In order to achieve this, we note that the conditioning text prompts and the model's predictions belong two different modalities (i.e., language and audio) which means that in order to define any notion similarity between them (which is needed to define loss function), we would need to map both common semantic space. Fortunately, this is solved problem already with the recent rise joint multi-modal embedding frameworks such as CLIP (Radford et al. [022]] and CLAP (Wu al., 2023). In particular, we propose to the pretrained CLAA language encoder (denoted by (-)) and the pretrained CLAP audio encoder (denoted by Ex(-)) to calculate cosine similarity between the the model's prediction and the conditioning text prompts and subsequently generate weak supervision signal for single-source text prompts. However, directly using (negative) cosine similarity as the loss function can result in degenerate outputs, which leads to incorporating cosine similarity within discriminative setting, i.e., the contrastive loss (Radford et al. 2021). In particular, given a batch of N mixture and text description pairs B [(M1, Ti...... (MN. (N)), if each text description Ti consists of K; single-source sounding language entities T. A E 1..K; (e.g., "the duet of piano and cello" consists of "piano" and "cello" single-source sounding language entities), then our flavor of contrastive loss for mixture M, is defined as:
K
CONT
log [Cr (caaki {Ciago a L..K,jj L.,N, L.,K}))++
2K;
k=1
log [Sr (caski {Cybia : L..K,,j 1..N,b L.,KH)))
(3)
where
(z; exp(x/T)
(f( T: (T,)))
Eyer exp(y/7)) and Cijjt |EA( T,t)) e (T))
(4)
the Softmax function with (leamable) temperature parameter and the cosine similarity, respec- tively. If M, is MoM with M component mixtures (that is, M, MM-- Miy), then define CoNT(M, 0) Σ}= (oNT(Me, We emphasize that the weak supervision signal generated as above does not contain enough information to reconstruct the fine-grain details of the queried single-source audio signal; in fact, the contrastive loss in equation 3 only enforces the predicted audio contain the "characteristic featurre features" such that it be encoded close the corresponding text prompt in the CLAP semantic space. That why we refer to this process as "weak" supervision and why it is crucial this loss function combined with the unsupervised reconstruction loss in
5

---

Published as a conference paper at ICLR 2024
Gueni Cood
Spectruttr
CLAP Language Encoder
Vistore) (oubbrand Flute)
Coni
Spectrogaa
Matured Drum and Vistnn
Figure Our proposed weakly-supervised audio-language training framework: bi-modal con- trastive loss (LENt) combined with consistency reconstruction loss (LCRs.).
equation 2] which requires us to use MoMs instead of the original mixtures during training. More- over, since now we can query for single-source audio components, we can add an additional (soft) consistency constraint have the predicted single source samples sum up the original mixture. More precisely, for a MoM M, = -E Mij where each component mixture May itself consists of Kıj (unknown) single-source component Sty for which we have access to the sounding language entities T.j. we define the consistency reconstruction loss (CRL) as:
Kis
Len (M;
MiM
So T.
(5)
Putting everything together. at each training step, given batch of audio mixtures and their language descriptions B = {MM1, (MM.) Ji....., (MN, w. Tx)]. first we randomly combine pairs of mixtures and their text prompts from B to form the synthetic batch of MoMs B' {(Mi, T(),... (M''s Tyy ) Once we have B', we minimize the Total Weak-supervision Loss (TWL) to find the optimal parameter values for our segmentation model fe: N'
CTWI(B',C
1
Σ [a- LM,, (M + LURL (M,, 0)]
(6)
N
where B and are the scalar relative weights for each loss term. Note that the parameters of the CLAP encoder models are kept frozen during training.
3.4
SEMI-SUPERVISED LEARNING
As mentioned earlier, the proposed learning framework not only boosts the performance of unsu- pervised audio separation by introducing weak supervision, but also improves the performance of supervised audio separation by incorporating potentially large corpora of mixture audio data which are not naturally accessible to supervised learning methods. The latter scenario effectively presents remi-supervised learning (SSL. scheme which can be significantly impactful in practice, as mixture audio datasets are generally way more abundant than their well-curated single source counterparts. Moreover, as will later in experiments, the introduction of unsupervised samples in the supervised training loop via the weak supervision signal provides an elegant regularization mecha- nism which protects the supervised method from potential overfitting.
In the SSL scenario, in addition to the mixture data, we also have portion of single-source audio samples in each batch, S, from which simply augment our total loss in equation to include the standard reconstruction loss for S':
generate the synthetic set of mixtures S'. Then
(B' U U', 0) =>,- Lu
>,, LTwr.
where > and A are the relative weights. Note that, as opposed to the unsupervised case, the LURL here is computed over S' for which we have access to the ground-truth single sources (i.e., S).
EXPERIMENTS
Datasets: We experiment on synthetic mixtures produced from single source MUSIC (Zhao et al. 2018) and VGGSound (Chen al. 2020) datasets by mixing samples from sources. We the

---

Published as a conference paper at ICLR 2024
Table 1: Comparison on MUSIC Dataset under the unsupervised setup. The supervised column is also provided as an upperbound. SDR on 2-Source separation test set is reported for all cases. All methods are reproduced under the same setting. * denotes implementation with our improved U-Net model. Bold and blue represents the best and second best performance in each group, respectively.
Single-Source (Supervised)
Multi-Source (Unsupervised)
Method
2-Source 3-Source
4-Source
Unconditional 2017
PIT eYu al.
8.0
MixIT Wisdom al.
2020
3.2 0.34
2.3 057
1.4 035
MixPIT Karamati & Kirbiz
2022
3.6 0.46
2.1 1014
1.7 0:35
Image Conditional
CIIPSep--Ig (Dong et al. 2022
6.8 0.25
3.8 0.27
2.9 003
2.1 =
CLIPSep-Img (aingg(DDnn (Dong ef al
2022
7.4 0.22
4.6 0.31
3.8 0:2
2.9 =
CoSep Gao & Grauman
2019
7.9 0:28
4,9 0.37
4.0 1 0:29
3.1 0.36
SOP*
Zhao al. 2018
6,5 0.233
4.1 1.0.1
3,5 1 0.26
2.7 1:022
Language Conditional
CLIPSep-Texs (Dong et al.
2022
7.7 +
4.6 0.35
3.5 #277
2.7 0.45
CLIPSep-Tex(" (Dog al. 2022
8,3 + 0.27
5,4 0.41
4.7 +.32
3.8 + 0.2
BertSep
7.9
5.3 +:31
4.0 0
3.1 00
CLAPSep
8.1
5.5 0.3%
4.3 0.28
3.5 003
I.ASS-Net (Liu et al. 2022
7.8
5.2
4.2 0029
0.6
Weak-Sup (Pishdadian et al. 2020
3.14
2.2 0.38
1.9 203
Proposed (w/ Timbre Classifier concurrent training)
5,0 1.229
4.5 1 0.32
3.5 1.07
Proposed (w/ Timbre Classifier- pretrained)
6.1 1.033
5.2 1 0.37
4.1 1:05
Proposed (w/ Bi-modal CLAP)
7,9 0.35
1: 0.42
6.2 0.38
same test set containing samples of sources for each dataset in all experiments. We also experiment with AudioCaps (Kim et al., 2019). a natural mixture dataset containing ~ 6 sounding sources in each mixture with full-length captions, where we use Constituent-Tres language parser (Halvani) 2023) to extract single source text phrases. See Appendix D
for more details on dataset preparation.
Training: All the models are trained for 50 epochs with initial learning rate of 0.001. The learning rate drops by a factor of 0.1 after every 15 epochs. Adam optimizer (Kingma & Ba 2014 is used with 31 0.9, B2 0.999 and e 10- The training was carried out with 8
GPUs
with 48GB memory. For more implementation and training details, see Appendix D
Evaluation: We primarily use signal-to-distortion ratio (SDR) (Vincent al. 2006 for evaluating different models. However, we have also calculated other evaluation metrics for comparisons, the details of which can be found Appendices Fland G.3
Setup:
Every instance of training in our experiments is carried out under one of these three settings: (I) supervised, where the training data consists of only single-source samples, (II) unsupervised, where the training data consists of only multi-source samples either natural or synthetic, and finally (III) cemi-supervisede where the training data is a mix single-source and multisouuce samples.
4.1 THE UNSUPERVISED SETTING
We have compared our proposed framework on MUSIC dataset with various types of state-of-the-art baselines, as presented in Table For comparison studies on VGGSound and AudioCaps datasets, see Appendix G.4 Here, the performance is measured for different complexities of training sce- narios containing ~ 4 single source components in training mixtures. We use the same test set for all training scenarios containing two components in mixtures. For comparison test results on 3-component mixtures, see Appendix G..5 There are three baseline categories: unconditional, image-conditional, and language-conditionaa methods. Most conditional models are primarily built on top of the mix-and -ed-saadeee method introduced in [Zhao et al. (2018). We have reproduced the results for all baselines under the same training and test conditions. For a fair comparison, we have incorporated our improved conditional U-Net architecture in most baselines.
Comparison to Unconditional Methods: Unconditional methods rely on complex post-procest processing selection to extract the target sound from the predicted sounds. Following their training methods, use similar selection methods on test set to find the best match with ground truths which can be po- tential limitation in real-time scenarios. PIT (Yu et al. 2017) can only be used in supervised training
7

---

Published as a conference paper at ICLR 2024
on single source sounds that gives comparable results with conditional methods. MixIT (Wisdom jet al. 2020) and MixPIT (Karamath & Kirbiz 2022) are built for multi-source training on top of the PIT framework. MixIT suffers from over-separation by predicting more components than the actual input mixtures. Though MixPIT reduces the over-separation by directly predicting the mixture in training, it often suffers under-separation the test set. Our framework, however, achieves +4.7 and +4.3 SDR improvements compared to MixIt and MixPIT, respectively.
Comparison to Image Conditional Methods: The MUSIC dataset also comes with correspond- ing YouTube videos for audio samples which can be used as visual conditioning signal for audio separation. To this end, we use the mean embedding of multiple sounding source images extracted with corresponding pretrained image encoders for conditioning. CLIPSep (Dong et al. (022) intro- duces IIme--maaee encoder for conditioning in the SOP framework (Zhao et al.) 2018) substituting its ResNet-18 18 encoder. CoSep (Gao & Grauman 2019) further adds a pre-trained object detector for finer conditioning and uses co-separation loss for classifying each sounding source. However, CoSep is limited by the number of training classes and pretraining of the object detectors on the tar- get classes. On 2-source training, our method achieves +3.8, +3.0, and +3.3 SDR improvements compared to SOP, CoSep, and CLIPSep methods, which use our improved U-Net model.
Comparison to Language Conditional Methods: CLIPSep (Dong et al., 2022) also includes the CLIP-text encoder which can be used for language conditioning. Additionally. we train separate CLIPSep models with its CLIP-text encoder replaced by frozen Bert (Devlin et al. 2018). and CLAP (Wu et al.] (023) text encoders denoted by BertSep and CLAPSep, respectively. LASS- Net (Liu et al.) (022) is another baseline that uses Bert text encoder with custom ResUnet model architecture. Note that all these methods lack fine-grain conditioning on single source predictions which results in sigmificant performance drop on multi-source training. As for weak supervision, we experimented with Weak -Sp (Pishdadian et al., 2020) which introduces a separate classifier on top of separation models. Apart from restricting the model to certain number of classes, such fine- fine-grain grain weak supervision often results in spectral loss; plus, without proper pre-training of the classifier on single-source samples, such methods face convergence issues that deteriorates the performance. In comparison, our method consistently achieves significant performance improvements in all chal- lenging multi-source training scenarios over all these baselines. Notably, we achieve 97.5% of the supervised method's performance trained on 2-source mixtures.
Bi-modal Embedding vs. Timbre Classification: A key question one might ask is whether can get similar gain if instead of using the bi-modal embedding CLAP model, we incorporate a simpler timbre classification model to generate weak-supervision for single source prompts, similar to CoSep (Gao & Grauman 2019) and Weak-Sup (Pishdadian et al. 2020). After all, CLAP can be seen zero-shot audio classifier with an open-ended set of classes. To test this idea, we have replaced our CLAP-basce loss with a timbre classification-ber based loss, where the classifier shares the exact same architecture as that of the CLAP audio encoder but the contrastive loss is replaced by the cross-entropy loss. Since, in general. the conditioning prompt can contain more than one classes, we have treated this problem as multi-label classification problem with C' binary classification outputs in the last layer, where C is the number of classes. Furthermore, we have trained our classifier- loss under two scenarios: (I) concurrently with the separation model, and (II) pretrained beforehand. In both cases, the training dataset is the same as that of the original separation task. The results are shown in TableT As the results show, the concurrent version performs worse than some of the baselines that do not even have weak-supervision. And while the pretrained version does better than the baselines, its performance is still significantly lower than our proposed framework using the CLAP loss, not to mention its restricted applicability due to the fixed number of classes. We hypothesize the superiority of CLAP-based supervision comes from the large-scale pretraining of CLAP which enables us to transfer that knowledge to source separation. In other words, in the limit for large-scale training data and gigantic number of classes, the classification approach should perform as well as the CLAP-based loss, but at that point, we might as well use CLAP.
4.2 THE SEMI-SUPERVISED SETTING
To demonstrate semi-supervised performance on synthetic mixtures, we form the training set by combining supervised and unsupervised training subsets. The results are given in Table 2 As shown in the table, when for both our semi cumi-supervised method and the supervised baseline, while letting our semi-supervieed framework use the rest of 95% unsupervised data, we get the dramatic 6.2 SDR boost over the su-
only 5% of MUSIC and VGGSound datasets as supervised data
8

---

Published as a conference paper at ICLR 2024
Table 2: Comparisons of the proposed semi-supervised learning with different portions of single- source and multi-source subsets. Bold and blue represents the best and second best performance.
Training
Test Set
Single-source Data
Multi-source Mixture Data
Performance
Method
Mixture
Dataset
Fraction
Dataset
Fraction
#Source
(SDR)
Supervised
MUSIC-2Mix
MUSIC MUSIC
100%
8.1 0.31
Supervised
MUSIC-2Mix
5%
2.6 0:35
Unsupervised
MUSIC-2Mix
MUSIC
100%
2
7.9 =: 0.15
Semi-Supervised
MUSIC-2Mix
MUSIC
5%
MUSIC
95%
2
8.8 = 0.28
Semi- eupe-Superiised
MUSIC-2Mix
MUSIC
5%
MUSIC
95%
3
8.2 0.22
Semi -upervuuee
MUSIC-2Mix
MUSIC
5%
MUSIC
95%
4
7.4 2 0.31
Semi- Supervised
MUSIC-2Mix
MUSIC
10%
MUSIC
90%
2
8.9 = 0.26
Semi Suni-vuper
MUSIC-2Mix
MUSIC
25%
MUSIC
75%
2
9.2 = 0.24
Somi-Sun -uperviidd
MUSIC-2Mix
MUSIC
75%
MUSIC
25%
9.5 0.29
Semi Semi--upeere -Speerised
MUSIC-2Mix
MUSIC
100%
VGGSound
100%
2
9,9 0.35
Semi-Supervised
MUSIC-2Mix
VGGSound
100%
MUSIC
100%
2
9.7 = 0.35
Scmi Suni-Super
MUSIC-2Mix
VGGSound
100%
MUSIC
100%
9.2 = 0.31
Somi Supervised
MUSIC-2Mix
VGGSound
100%
MUSIC
100%
8.9 0.42
Supervised
VGGSound-2MM
VGGSound
100%
2.3 0.23
Supervised
VGGSound-2M0 -ixx
VGGSound
5%
0.4 0.35
Unsupervised
VGGSound-22 2ixx
VGGSound
100%
2
2.2 0.79
Scmi-Superee Supervissd
VGGSound-2MM
VGGSound
5%
VGGSound
95%
2
3.1 = 0.31
Semi Supervised
VGGSound -Mix
VGGSound
75%
VGGSound
25%
2
3.4 0:36
Unsupervised
AudioCups-2MMa 2Mix
AudioCaps AudioCaps
100%
1-6
2.9 0.3
Semi- Supe-Super
AudioCaps- 2Mxx
VGGSound
100%
100%
1~6
4.3 0.34
pervised method. This shows that our semi-supervised framework can make a significant difference in scenarios where well-curated, single source samples are scarce and costly acquire by lever- aging large corpora of unsupervised data. More interestingly, even when we let the supervised baseline use 100% of the data as supervised single-source data, our semi-supervised approach still beats it by +0.7, and +0.8 SDR boost on the two datasets using only 5% of the supervised data! Based on this result, we hypothesize that, in addition data augmentation for training, our proposed framework offers powerful regularization mechanism that boosts the generalization of the super- vised method by encouraging it to also discover salient patterns from unsupervised data. For more comprehensive study of our framework's 's regularization effects, see Appendices G.2 G.5 and G.7
Finally, we note that for realistic, natural mixture datasets where single source audio samples are not available for semi-supervised learning, we can still utilize our semi-supervised scheme by running it across multiple datasets and let supervised and unsupervised training samples come from different datasets. To this end, we have trained our model using the AudioCaps natural mixtures the unsupervised subset and the VGGSound dataset as the supervised subset, and have achieved +1.4 SDR boost over the weakly-superve
baseline trained on AudioCaps only.
CONCLUSION AND FUTURE WORK
In this paper. we proposed a weakly supervised learning framework for language conditional audio separation when single source audio samples modal similarity between the audio and language modalities through the pretrained CLAP model, our methodology is capable of generating weak supervision signal for single-source prompts during training, which in turn, enables the model to reduce the shift between the training and test data distributions. By conducting extensive experiments, we demonstrate the superiority of our proposed framework over the SOTA baselines by significant margin, shrinking the gap between unsupervised and supervised methods. Furthermore, by incorporating our framework into the semi-supervised setting, we showed that our framework beats the supervised method itself. More interestingly, as shown by the experiments, our framework still maintains superiority even when the supervised baseline had access to 20x more supervised single source data than our framework. This latter result highlights the ability of our framework to employ natural regularization through the language modality during training.
available during training. By leveraging cross
As mentioned earlier, aside from the implementation details, the core idea of our proposed frame- work is generic and -ineaitt--ddeeeeeent, and therefore can be adopted for conditional segmenta- tion tasks in other modalities as well (e.g., unsupervised image segmentation in Computer Vision). This property provides fertile ground for future research directions and novel applications.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported in part by ONR Minerva program, iMAGiNE the Intelligent Machine Engineering Consortium UT Austin, and UT Cockrell School of Engineering Doctoral Fellow- ship.
REFERENCES
Moitreya Chatterjee, Jonathan Le Roux, Narendra Ahuja, and Anoop Cherian. Visual scene graphs for audio souree separation. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 1204-1213, 2021.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp- 721-725. IEEE, 2020.
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg- KingKKikkpatrick, and Shlomo Dubnov. Zero- shot audio source separation through query-based learning from weakly-labeled data. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1444-4449, 2022.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 180.04405 2018.
Hao-Wen Wen Dong, Naoya Takahashi, Yuki Mitsufuji, Julian McAuley, and Taylor Berg- Kirg-Kirkpatrick. Clipsep: Learning text-queried sound separation with noisy unlabeled videos. arXiv preprint arXiv:2212.07065. 2022.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14084 14093, 2022.
Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3879-3888 2019.
Beat Gfeller, Dominik Roblek, and Marco Tagliasacchi. One-shot conditional audio filtering of arbitrary sounds. In ICASSP 2021-2021 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pp- 501-505. IEFE, 2021.
Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980. IEEE, 2022.
Oren Halvani.
Constituent Treelib A Lightweight Python Library for Constructing, Process- ing, and Visualizing Constituent Trees., 2023. URL httttt:/ggtttuu......... com Halmmnii
Constituent treelib
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ertuğ Karamath and Serap Kırbi. Mixcycle: Unsupervised speech separation via cyclic mixture permutation invariant training. IEEE Signal Processing Letters, 29:2637-2641, 2022.
Ilya Kavalerov, Scott Wisdom, Hakan Erdogan, Brian Patton, Kevin Wilson, Jonathan Le Roux, and John R Hershey. Universal sound separation. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 175-179. IEEE, 2019.
Kevin Kilgour, Beat Gfeller, Qingqing Huang, Aren Jansen, Scott Wisdom, and Marco Tagliasacchi. Text-driven separation of arbitrary sounds. arXiv preprint arXiv:2204.05733 2022.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume I (Long and Short Papers), pp. 119-132, 2019.
10

---

Published as a conference paper at ICLR 2024
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering, 34(1):50-70, 2020.
Xuho Liu, Haohe Liu, Qiugiang Kong, Xinhao Mei, Jinzheng Zhao, Qiushi Huang, Mark D Plumb- ley, and Wenwu Wang. Separate what you describe: Language-queriel audio source separation. arXiv preprint arXiv:2203. 15147, 2022.
Xubo Liu, Qiugiang Kong. Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D Plumbley, and Wenwu Wang, Separate anything you describe. arXiv preprint arXiv:2308.00 05037, 2023.
Yuzhou Liu and DeLiang Wang. A casa approach to deep learning based speaker-independen co- channel speech separation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5399-5403. IEFE, 2018.
Rui Lu, Zhiyao Duan, and Changshui Zhang. Listen and look: Audio-visual matching assisted speech source separation. IEEE Signal Processing Letters, 2509):115-1139, 2018.
Huaishao Luo, Junwei Bao. Youzheng Wu, Xiaodong He, and Tianrui Li. Segelip: Patch aggregation with learnable centers for vocabulcyy semantic segmentation. International Conference on Machine Learning, pp. 20044. PMLR, 2023.
Yi Luo and Nima Mesgarani. Tasnet: time-domain audio separation network for real-time, single- channel speech separation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 696-700. IEEE, 2018.
Juan Montesinos. Toreh-mir-eval: Pytorch implementation of mir-eval, 2021. URL https:
77
mir _ir_eval
eval
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Fatemeh Pishdadian, Gordon Wichern, and Jonathan Le Roux. Finding strength in weakness: Learn- ing to separate sounds with weak supervision. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2386-2399, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Colin Raffel, Brian McFee, Eric Humphrey,Jusee Justin Salamon, Oriol Nieto, Dawen Liang, Daniel PW Ellis, and C Colin Raffel. Mir. eval: transparent implementation of common mir metrics. In ISMIR, volume 10, pp. 2014, 2014.
Ryosuke Sawata, Stefan Uhlich, Shusuke Takahashi, and Yuki Mitsufuji. All for one and one for all: Improving music separation by bridging networks. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 51-55. IEEE, 2021.
Fabian-Robert Stöter, Stefan Uhlich, Antoine Liutkus, and Yuki Mitsufuji. Open-unmix-a reference implementation for music source separation. Journal of Open Source Software, 4(41):1667, 2019.
Naoya Takahashi and Yuki Mitsufuji. Densely connected multi-dilated convolutional networks for dense prediction tasks. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 993-1002, 2021.
Reuben Tan, Arijit Ray, Andrea Burns, Bryan A Plummer, Justin Salamon, Oriol Nieto, Bryan Rus- sell, and Kate Saenko. Language-guics guided audio-visual source separation via trimodal consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10575-10584, 2023.
11

---

Published as a conference paper at ICLR 2024
Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2754, 2021.
Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel PW Ellis, and John R Hershey. Into the wild with audioscope: Unsupervised audio-visual separation of on- screen sounds. arXiv preprint arXiv:201 143, 2020.
Efthymios Tzinis, Gordon Wichern, Paris Smaragdis, and Jonathan Le Roux. Optimal condition training for target source separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE transactions audio, speech, and language processing, 14(4): 1462- 1469, 2006.
DeLiang Wang and Jitong Chen. Supervised speech separation based on deep learning; An overview. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(10):1702-172 2018.
Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron Weiss, Kevin Wilson, and John Hershey. Unsupervised sound separation using mixture invariant training. Advances in Neural Information Processing Systems, 33:3846-3853 2020.
Scott Wisdom, Aren Jansen, Ron J Weiss, Hakan Erdogan, and John R Hershey. Sparse, efficient, and semantic mixture invariant training: Taming in-the-wild unsupervised sound separation. In 202/ IEEE Workshop Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 51-55. IEEE, 2021.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and peword-to-captionn augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2945-2954, 2023.
Yao-Y Yuan Yang, Moto Hira, Zhaoheng Ni. Artyom Astafurov, Caroline Chen, Christian Puhrsch, David Pollack, Dmitriy Genzel, Donny Greenberg, Edward Yang, et al. Torchaudio: Building blocks for audio and speech processing. In ICASSP 2022-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1982.6986. IEEE, 2022.
Dong Yu, Morten Kolbæk, Zheng- Hua Tan, and Jesper Jensen. Permutation invariant training of deep models for speaker-independent multi-talker speech separation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 241-245 IEFF, 2017.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14393-14402, 2021.
Zhuohuang Zhang. Yong Xu, Meng Yu, Shi-Xiong Zhang, Lianwu Chen, and Dong Yu. Add-mvdrr: All deep learning mvdr beamformer for target speech separation. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6089-6093. IEEE, 2021.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pp. 570-586, 2018.
12

---

Published as a conference paper at ICLR 2024
A
OVERVIEW OF SUPPLEMENTARY MATERIALS
We cover the following topics in our supplementary materials:
All notations used the paper are summarized in Appendix
The proposed conditional U-Net architecture is detailed in Appendix C
The details of the training process for the proposed framework are illustrated in Appendix D
The datasets' details and the data prepossessing pipeline presented in Appendix
The details of evaluation metrics are presented in Appendix
Additional experimental ablation studies are reported in Appendix G
And finally, qualitative analysis of the model's performance can be found in Appendix !
Table 3: The glossary of all notations used in the paper.
Type
Description
Notation
Scalar Parameters
Total number of mixtures datasct Number sampled mixtures a batch
N
Number of single source components Temperature parameter in contrastive loss
mixture
K
Unsupervised reconstruction loss
LURL
Contrastive loss Consistency reconstruction loss
LONT
CORL
Total reak-supervisse loss
CrwL
Total semi-supervised learning loss
Eest
Weak supervision loss weights
a, 3.
Semi-supervised learning loss weights
Ass Au
Vectors/Matrices
Complete dataset
D
Batch of mixtures
B
Batch of synthetic mixture-of-mixtunee
B
tt Sound mixture
M;
Mixture of mixture
M'
Language prompt of 1th mixture
T
The 6tth single source component the 2th mixture
S
Language prompts of single source sound
T
Models
Frozen CLAP language encoder
cx(-)
Frozen CLAP audio encoder
EA(-)
Conditional U-Net audio source separation model
fo(.)
Conditional U-Net mask model
86(-)
Operators/Functions
Magnitude function
Phase function
01-)
Short Term Fourier Transform
Si.
Softmax function with temperature parameter Audio-language Cosine Similarity
G()
loss
-
Hadamard product
B THE NOTATION GLOSSARY
Table 3 presents the glossary of all notations used in the paper. We have divided the notations into four groups: scalars, vectors/matrices, models, and operators/functions.
C THE PROPOSED ARCHITECTURE
C.I THE LANGUAGE-CCNDITTONN U-NET
To extract rich features for faithful reconstruction of the audio sources conditioned the input prompt, we propose an enhanced conditional U-Net architecture. Our U-Net model operates on the magnitude spectrum of input mixtures, and estimates the segmentation mask for the corresponding
13

---

Published as a conference paper at ICLR 2024
source(s) based conditional feature embedding. Prior works on conditional sound separation, mostly used unconditional U-Net with post conditioning on final generated features from the U- Net (Dong et al.; 2022 Zhao et al. 2018). Some works (Gao & Grauman, 2019) used simple conditional feature concatenation at the innermost layer of the U-Net. Since most of these methods primarily built for supervised separation, which is much simpler problem, the vanilla U-Net architecture is often sufficient. However, since post-conditioning methods cannot leverage the con- ditional language features through the network, their performance can degrade significantly in the unsupervised setting. To overcome this, we redesign the conditional U-Net architecture by intro- ducing multi-scale cross attention conditioning on the intermediate feature maps of the U-Net. The architecture is shown in Figure 5
We incorporate three main building blocks into the proposed conditional U-Net: residual block (ResBlock), self-attention (SA), and cross-attentin (CA) modules. The residual block is used for enhancing model capacity following He et al. (2016) at every scale of feature processing. For the input z, the operation can be represented by.
x + ConvBlock(#)
where ConvBlock represents two successive convolutional layers. The self-attention and cross- attention modules are designed using the multi-head attention (MHA) operations introduced by (Vaswani et al.] (017). Self-attention re-calibrates the feature space before applying the con- ditioning modulation. The self-attention operation for input is given by.
x= + MHA(Q = K z, V x))
where Q, K, V represent query, key, and values used in the MHA operation, respectively. In contrast, cross-attention selectively filters the relevant features based on conditional features. For condition embedding y with input x, the cross-attention mechanism given by,
x = x + MHA(Q = x, K = y, V = y)
(10)
We divide the conditional U-Net model into two sub-networks: the Head network and the Modula- tor network. The Head network operates on the fine-grat ine-grain features of the higher signal resolutions to generate coarse- coarse-grain features to be conditioned later by the language modality. Only ResBlocks with traditional skip connections ulator network applies the feature modulation based on the conditional language embedding. We incorporate the self-attention and cross-attention operations in the skip connections of every Modu- lator network layer. In total, the U-Net contains 7 layers of encoder and decoder. The Head network contains top four layers of encoding and decoding, and the Modulator network contains the remain- ing three layers. Table 4 shows the architectural details of each block in our enhanced conditional U-Net.
used each scale of the Head network. In contrast, the Mod-
C.2 THE INFERENCE PIPELINE
For the inference, use the similar pipeline baseline methods. As shown in Figure 4 the condi- tional U-Net takes the input mixture and the language prompt (querying for the target source), and generates the (soft) magnitude mask. The mask is applied on the mixture's magnitude spectrogram, while the phase is directly copied from the input.
D THE TRAINING DETAILS
All the models are trained for 50 epochs with initial learning rate of 0.001. The learning rate drops by the factor of 0.1 after every 15 epochs. Adam optimizer (Kingma & Ba 2014) is used with 3 = 0.9, 8n = 0.999 and 6 = 10- for backpropagation. All the training was carried out with RTX-A6000 GPUs with 48GB memory. We validate the model after every training epoch. We use the batch size of 32 for the MUSIC dataset, and batch size of 64 for the VGGSound and AudioCaps datasets. We reproduce all baselines under the same settings. PyTorchh library (Paszke et al.) 2019) is used to implement all the models. The complete training algorithm of the proposed framework is illustrated in Algorithm
Furthermore, in Figure 6 we have visualized the detailed loss curves over the training course of the proposed weakly supervised training (Fig. (oap and its semi-supervised flavor (Fig. 6b). We have
14

---

Published as a conference paper at ICLR 2024
Table 4: Architectural details of proposed building blocks in the conditional U-Net model.
Block Type
Parameters
Values
No. of attention heads
8
No. of channels
512
Head dimension
64
Condition dimension
77x768
Cross Attention Block (CA)
No. of linear layers in attention
Attention type
Softmax
Normalization Layer
LayerNorm
No. of linear layers in MLP
2
MLP intermediate channels
1024
MLP intermediate activation
GeLU
Num of attention heads
8
Channel dimension
512
Head dimension
64
No. of linear layers
4
Self Attention Block
Attention type
Softmax
(SA)
Normalization layer
LayerNorm
No. linear layers MLP
2
MLP intermediate channels
1024
MLP intermediate activation
GeLU
Conv kernel size
(3,3)
Residual Block (ResBlock)
No. convolutions
Normalization layer
BatchNorm Leaky ReLU (th=0.2)
Activation
Channel expansion ratio
Operator
Strided Convolution
Encoder Downsampler Module
Kernel size
4x4
Strides
2x2
Channel expansion ratio
2
Spatial upsampler
Bilinear upsampling
Decoder Upsampler
Scale
2.0
Module
Channel compressor
Convolution 2.0
Channel compression ratio
15

---

Published as a conference paper at ICLR 2024
The sound of accordion
Condition Embedding
STFT
Mixture Audio
Magnitude Spectrogram
Predicted
Spectrogram
Phase Residual
Figure 4: Inference pipeline for the proposed language conditional sound separation framework.
Predicted Spectrogram
Mixture Spectrogram
Predicted
Text Condition
ResBlock
ResBlock
Head Network
ResBlock
ResBlock
ResBlook
ResBlock
Modulator
ResBlock
SA
CA
ResBlock
Figure 5: Proposed conditional U-Net architecture. We incorporate three building blocks: residual block (ResBlock), self-attention (SA), and cross-attention (CA) blocks. The model is divided into two modules: the head and the modulator. The head network operates on fine-grained features and generates latent embedding. The modulator network modulates latent features based cross- attention conditioning.
combined both unsupervised reconstruction loss Lu Rz and consistency reconstruction loss (LCRl) in the reconstruction loss plot. For further analysis of the loss components as well as their weight hyper-parameter tuning details, please refer Appendix G.2
16

---

Published as a conference paper at ICLR 2024
16
Total Loss
Contrastive Loss
14
Recons Loss
12
Loss Value
10
Epoch
(a) Weakly supervised training
Total Loss
Sup Recons
12
Contrastive Loss
Unsup Recons Loss
10
Loss Value
10
15
20
25
30
35
40
Epoch
(b) Semi-supervised training
Figure 6: Visualization of loss curves over training epochs in proposed (a) weakly supervised train- ing, and (b) supi-supe training.
17

---

Published as a conference paper at ICLR 2024
Algorithm The Proposed Weakly Supervised Training for Audio Source Separation
Input: Dataset D {M; Ti) Single source text prompts [T-' K;
of sources per
mixture, masking U-Net go, Pre-trained joined embedding encoders (EI, eA)- Require: Initialize weights of ge. keep pre-trained joint embedding encoders (EI, frozen.
1: for t E [1, do
T: Training iteration Batch size +- N
2: Sample N Mixture with text prompts {Ma, Ta
E
3: for [1, NJ do
4:
Sample another mixture {Mm, Tm
D No single sound source overlaps in Mn. Mm
5: Prepare MoM M' Mn + +m
6:
Predict Mn + fo Tn)
7:
Predict Mm + fo(M', Tm)
8:
Compute LURL with (M,, Mn: Mm: Mn) using equation 2
9:
for E [1, do
10:
Compute single source sound 5% t- so(Mn: Txt) Compute single source sound Sk t [o(Mm, Tyi) end for
11:
12:
Compute CONT(M,, 0) with SS T St. Thi using equation and equation 4
14:
Reconstruct mixture M,, + E Sh and Mm + E S Compute LORL with (Ma, Mni Mm Mm) using equation end for
16:
Compute total loss CrwL using LU RLR LCRL, and LeNT for all N mixtures Back-propagate VLwwr and update weights of go
18:
19:
end for
E DATASET PREPARATION
E. DATASETS DESCRIPTION
MUSIC Dataset Following prior works, we experiment with MUSIC dataset (Zhao et al. 2018) for musical instrument separation task. Instead of using the original instrument datasets, we use its extended version of MUSIC-21 containing around 1, 200 videos from 21 musical instruments. Since some videos are not available, our aggregated version contains 1, 086 videos total. The video duration ranges 1 5 minutes. We extract audios and class labels annotations from each video. We use 80% videos of each classes for training and the remaining for testing. For training, randomly sample around 6s duration segments from each audio, while for testing, we prepare non-overlapping samples from the whole length audio. The dataset only contains sounds single- source musical instruments. To this datset for unsupervised training, we create synthetic training mixtures by sampling different combinations of K single source sounds. Text prompts are then generated using the class labels of the single source sounds. We use the common template for representing the single and multi-source language condition prompts, as presented in Table 6
VGGSound Dataset VGGSound (Chen al. 2020) is large-scale environmental sound datasets containing more than 190, 000 videos from 309 classes. Since many corresponding videos are not available in YouTube. our aggregated subset contains 175, 599 videos. We use the official train and test split of VGGSound that contains 162, 199 training videos and 13, 398 test videos. Every video contains audio, mostly with a single source. Each video duration is 10s that contains single source audio collected from different environments corrupted by natural noise. The sounding event duration varies from 10s. Because of that, use the full-length audio samples in VGGSound for our experiments. In order to use VGGSound for the unsupervised learning scenario, we mix K random single source samples for each training mixture. The corresponding text prompts are generated using the class labels of the the sounding sources in the mixture, similar to Table
AudioCaps Dataset AudioCaps (Kim et al. 2019) contains around 50, 000 natural sound mix- tures of 10s duration each. It also includes the complete captions of the prominent sources in each mixture. We use the official train and test splits that contain 45, 182 and 4, 110 mixtures, respec-
18

---

Published as a conference paper at ICLR 2024
tively. In general, each mixture contains 6 single source components. To cover all sounding events included in the text caption, we use full-length audios of 10s. We use Constituent- Tree li- brary (Halvani 2023) to extract fine-grain phrases representing each sounding source from the full caption. We initially extract several sentence and noun phrases, then perform simple post-processing on them eliminate the overlapping phrases. Some examples of extracted phrases from the full cap- tions are given in Table [ To handle different number of mixture components, we sample fixed number of phrases from each caption. In case there not enough sounding phrases in the text prompt, we re-sample some of the phrases, and introduce weighted reconstruction to ensure proper reconstruction of the mixture.
AudioCaps is primarily used to measure training performance on natural mixtures containing diverse sounding events, as opposed to synthetic mixtures. However, to evaluate the performance of the model, we prepare synthetic mixture-of-mixtuee test set. the test time, the model queried with one full-length caption representing one of the mixtures in synthetic MoM, and evaluated using the corresponding mixture.
by combining two mixtures from the AudioCaps
E.2 DATA PREPROCESSING PIPELINE
We use the sampling rate of 11kHz for audio samples used. The audio clip length is chosen to be 65, 535 for MUSIC dataset, and 110, 000 for the AudioCaps and VGGSound datasets. Since AudioCaps and VGGSound datasets are noisy, and usually contain sounding regions on small portion of 10s duration, use full length audio samples for these two datasets. For the MUSIC dataset, we extract consecutive 65, 535 segments from the complete duration of the samples representing 6s audio. We compute the spectrogram for each sample using short-term Fourier transform (STFT) with window size of 1024, filter length of 1024, and hop size of 256.
datasets. Only mono-channel audio
The CLAP model is pre-trained with 10s duration audios of 48KHz sampling rate and has different pre- rre-processing pipeline than ours. To integrate the pretrained CLAP model in our training pipeline, we initially reconstruct the sound waveform from the predicted spectrogram. For audio samples extracted from MUSIC dataset that contains duration segments, we repeat the waveform to extract equivalent 10s duration of audios. Then, the audio waveform is resampled with 48KHz sampling rate. We use Torchaudio package (Yang et al., 2022) to process predicted audio samples in the training loop. To estimate the contrastive loss (CONT) with the CLAP model, we follow the same pipeline of CLAP with the pretrained temperature value for т. We note that the CLAP model is kept frozen throughout the entire training, as it is only used to generate weak supervision. For text conditioning signals, instead of the projected mean-pooled token representation of language prompts, the complete language embedding of dimension (77 x 768) representing 77 tokens, generated by the CLAP language encoder.
F EVALUATION METRICS
We use three evaluation metrics in our experiments: SDR, SIR, and SAR. Here, we provide the detailed equations as well as the explanation of each metric. We note that a predicted sound Spred can be represented as combination of the true sound Strue- the interference of other sources in the mixture Enterrf and the artifacts generated during reconstruction Eartifacti that is, Spred = Strue + Einters + Emoise + Eartifact- According to Vincent et al. (2006), these evaluation metrics are described follows:
Signal-to-Distoriie Ratio (SDR): SDR is the primary metric used for evaluating sound sep- aration performance in most prior work. It represents the overall measure of the sound quality considering all kinds distortions. It is given by
|| 1|
SDR 100ogg0
(11)
Einters Enoise
Eartifact
Signal-to-Inteeffrrnnce Ratio (SIR): SIR is also widely used evaluation metric in sound sepa- ration. It represents the "leakage" "bleed" from other sounding sources in the mixture to the
19

---

Published as a conference paper at ICLR 2024
Table 5: Examples of some extracted phrases from full-length AudioCaps (Km et al. tions.
2019 cap-
Complete Audio Captions
Extracted Phrases
A young female speaks, followed by spraying
A young female speaks. Spraying and a female screaming
and a female screaming
Motor noise is followed by a horn honking and siren wailing
Motor noise. horn honking.
A siren wailing.
Rustling occurs, ducks quack and water splashes, followed by an adult female and adult male speaking and duck calls being blown
Rustling occurs. Ducks quack and water splashes. An adult female and adult male speaking. Duck calls being blown.
An audience gives applause as man yells and group sings
An audience gives applause. A man yells. A group sings.
A man speaks over intermittent keyboard taps
A man speaks. Intermittent keyboard taps.
An airplane engine runs
An airplane engine.
predicted sound. SIR measures the quality of the predicted sound considering the amount of cross- interference from other sources. is given by
Strue
SIR = 10 logi0 [ 2
(12)
Signal-to-Artifact Ratio (SAR): SAR mostly used to measure how realistic the predicted sound is. It measures the amount of synthetic artifacts present in the predicted audio. Without any separa- tion applicd, the original mixture usually have very high SAR, as does not contain that many of artifacts. However, as the model learns to separate single source components from the mixture, is expected to introduce more artifacts.
SAR = 10 log Strue + Einterf + Enoise
(13)
&artifoct
|
In order to calculate these metrics, we have used the Python package torch-mir-eval (Montesinos 2021 which the Pytorch implementation of mir-eval (Raffel al. 2014).
G ADDITIONAL EXPERIMENTAL STUDIES
In this section, we present additional experimental ablation studies for a deeper analysis of the proposed framework compared to the state-of-the-ard baseline methods as well as some of our design choices.
G.1 ABLATION STUDY ON CONDITIONAL U-NET ARCHITECTURE
We have studied the contribution of all three building blocks in the proposed conditional U-Net architecture. We have experimented with both supervised and unsupervised training settings
Table 6: Text query templates with examples for single and synthetic multi-souree sounds
Source
Query Template
Example
Single Source
The sound of {source)
The sound of guitar. The sound mixture of guitar and piano.
Multi-Source
The sound mixture of {source-1] } and [source-2]
(2-Source)
Multi-Source
The sound mixture of [source-1] 1), (source-2), and [source-3)
The sound mixture of guitar, piano, violin.
(3-Source)
20

---

Published as a conference paper at ICLR 2024
Proponed Ni. Rovand (SLP540)
14
Test SAR
Test
23
23
Epoch
Epoch
Epoch
(8)
(b)
(c)
Figure 7: Test metric plots during training with two-source mixtures on MUSIC dataset. We use CLIPSep the mix-and-separate baseline method. Our proposed method achieves significant im- provement in terms of SDR and SIR by largely reducing noise and cross-interference in predictions, respectively. As the model tries to learn to separate single-source sounds, some artifacts are intro- duced, which in turn result in low SAR value in the early stages of training. However, most of these artifacts get removed over the course of training which in turn causes SAR to increase to the same level as the baseline method by the end of training. In other words, by the end of training, both our method and the baseline produce audio samples with reasonable quality regardless of their separa- tion performance.
Table 7: Ablation on three building blocks of proposed conditional U-Net architecture: ResBlock, self-attention (SA), and cross-attention(CA)) The vanilla U-Net contains single convolutional layer instead of ResBlock, and simple skip connections instead of attention modules. Test SDR on 2- source mixture reported for various single and multi-source training scenarios on MUSIC dataset. For the single source training, simple mix-an mix-and-seppan reparae based on CLAPSep is used. For the multi- source training, proposed weakly supervised training is used. All three blocks contribute to consid- erable performance gain mostly in challenging multi-source scenarios. Bold and blue represents the best and second best performance in each group. respectively.
ResBlock
SA
CA
Total Params (M) Single Source
2-Source
3-Source
4-Source
X
X
30.7M
6.7
5.8
4.9
AXM
/
37.4M
7.7
7.1
6.3
5.5
AAAAKE
1V
44.7M
7.8
7.3
6.5
5.7
36.6M
6.9
6.1
5.3
42.9M
7.5
6.8
5.9
5.1
74.3M
7.6
6.9
6.1
5.2
118.8M
7.1
6.4
5.6
5.0
43.7M
7.9
7,4
6.7
5.8
57.7M
7.2
6.6
5.9
81.4M
8.1
7.9
7.1
6.2
the MUSIC dataset. The test set contains 2-source mixtures as before. The baseline vanilla U-Net contains single convolutional layer instead of ResBlock, and direct skip connections instead of self- attention and cross-attention modules. The results are given in Table Utilizing all three building blocks results in +0.7, +1.2, +1.3, and +1.3 SDR improvements on single source, 2-source, 3- source, and 4-source training settings, respectively. We note that the performance improvements are comparatively higher in the challenging unsupervised setting compared to the supervised setting. Since unsupervised training is mostly guided by weak supervision generated by the bi-modal CLAP model, we hypothesize that the multi-scale feature modulation based on conditional embedding becomes more critical in such cases.
Note that the increased number of parameters by the way of adding new blocks can be seen as the major contributor to the performance gain. To control for this effect, in some of the ablation scenarios, we have inserted the target block(s) twice in row merely to increase the capacity of the model (shown by // in Table 7). As the results show, while increasing the number of model's parameters contributes to the performance gain, it is not the major driver. In fact, some our smaller candidates beat the larger ones by simply incorporating
block type. This shows that
21

---

Published as a conference paper at ICLR 2024
Table 8: Ablation on loss components of proposed weakly supervised training method with multi- source training mixtures from the MUSIC dataset. Test SDR on 2-source mixtures is reported for all cases. Unsupervised reconstruction loss (LURL) underperforms of fine-grain supervision. Contrastive loss (LCNT), on the other hand, produces weak supervision that performs the best combined with proposed consistency reconstruction loss (LC))) Combining all three loss components achieves significant performance improvements. Bold and blue represents the best and second best performance each group, respectively.
higher mixtures due to the lack
LURL.
LONT
LCRL.
2-Source
3-Source
4-Source
AYKYA
X
5.5
4.3
3.5
V
X
4.9
2.8
1.7
V
X
V
2.4
1.3
0.6
V
6.8
5.9
5.3
V
V
7.9
7.1
6.2
Table 9: Ablation study loss weights with proposed weakly supervised and semi-supervised learning. For the weakly supervised training, 2-source mixtures from the (a) MUSIC dataset, (b) VGGSound dataset, and (c) natural mixtures from the AudioCaps dataset are used. (d) For the semi- supervised training, 5% single-source and 95% 2-source mixtures are used. Test SDR on 2-source mixture separation is reported. Bold and blue represents the best and second best performance in each group, respectively.
(a) Weakly supervised training on MUSIC
(b) Weakly supervised training on VGGSound
a(LeNT) M(CCLL)
(LUUL)
SDR
aCCNN))
3(CCRL)
(LCuI,)
SDR
0.1
5
7.7
0.5
2.0
0.1
10
2.1
0.1
15
7.7
2,2
0.2
10
7.8
0.1
5
15
1.9
0.1
10
10
7.6
0.3
5
1.7
(d) Semi-supervised training on MUSIC
(c) Weakly supervised training on AudioCaps
a(CeNr) B(CCRL)
7CuRL.
SDR
(a(LUIL.)
Au (cTWL)
SDR
0.2
0.5
10
2,8
0.5
8.5
0.2
10
2,9
8.8
0.2
10
2.7
8.6
0.1
15
2.5
2.5
8.7
0.3
5
2.4
0.5
8.2
our proposed blocks encode important inductive bias for our problem which can boost the model performance without overfitting.
Furthermore, for a fair comparison with the baseline methods, we have reproduced most of the prior work with our improved U-Net architecture, shown in Table [ and [2] We observe consistent improvements of performance by leveraging our modified U-Net. Nonetheless, the improved U-Net architecture increases computational burden in general, but the proposed weakly supervised training can still be applied in resource constrained scenarios by simply using the vanilla U-Net architecture.
G.2 EFFECTS DIFFERENT LOSS COMPONENTS
We have also studied the effects of different loss components in the proposed framework under the challenging unsupervised settings, as shown in Table By only using the unsupervised reconstruc- tion loss RI.) we get sub-optimal performance due to the training and test distribution shift. On the other hand, the contrastive loss by itself (CONT) results in performance drops with significant spectral loss due to the lack of fine-grain supervision to reconstruct the target signal. Similarly, the consistency reconstruction loss(Lert. by itself suffers from convergence issues due to the lack of any supervision to encourage the model for conditional single source separation. In other words,
22

---

Published as a conference paper at ICLR 2024
since the final reconstruction administered by the consistency reconstruction loss (CORl) greatly depends on the quality of single-source predictions, a significant performance drop is inevitable without using any single-source supervision. However, by combining LONT and LCRL. we achieve +1.3, +1.6, and 1.8 SDR improvements over the Lu no-- scenario for 2-source, 3-source, and 4-source training settings, respectively. Furthermore, by combining all three losses, achieve sig- nificant performance improvements of +2.4, +2.8, +2.7 SDR over the LU LuRL-only approach for 2-source, 3-source, and 4-source training settings, respectively.
In addition to the elimination study of different loss terms, we have performed an ablation study on 2-component mixtures from the MUSIC and VGGSound datasets, as well as on natural mixtures of AudioCaps dataset, to find the optimal relative weights of these components (i.e. a, 8, and y in interesting to observe that, in the optimal setting, LENT is weighed two orders of magnitude less than LU,L, which suggests that the weak suaervsunniisionn mechanism in our framework acts as an effective regularizer while the backprop- agated supervision signal is mostly dominated by the reconstruction error. And yet this relatively small regularization effect makes a significant improvement to the final performance of the model during inference. Moreover, VGGSound, and AudioCaps dataset contain significant amount of envi- ronmental noises that result in noisy training particularly with the consistency reconstruction losses. As result, the corresponding weight 7 of the consistency reconstruction loss LCRS. is relatively reduced in VGGSound and AudioCaps dataset, while LCNT coefficient a is slightly increased for the best performance. Similar study has been performed to find the optimal relative weights of the supervised and weakly-supervised components for the semi-supervised loss (CSst) (i.e. A and X in equation 7..
equation 6). Table aa), [b), and ( shows these results.
The results are summarized TablegTb)
G.3 ANALYSIS OF EVALUATION METRICS FOR UNSUPERVISED TRAINING
The metric plots in Figure 7 demonstrate comparative analysis of the evaluation metric curves during the unsupervised (2-source) training. To represent the baseline hii--and-separaa framework, we have used the CLIPSep (Dong al.] (022) method. The -id-----see baseline attempts to extract the single-source components from the mixture without having any supervision on single- source predictions during training; this results in large noise and cross-interference. In contrast, our proposed weakly-supervised method significantly reduces noise and cross-interference during unsupervised training by leveraging weak supervision through the language modality, and results in much higher SDR (Figure [ and SIR(Figure (])), respectively. However, separating single- source components is susceptible to producing artifacts that cause lower SAR in the early stages of training, shown in Figure Cc) Nevertheless, the training continues, such artifacts eliminated which subsequently improves SAR. In general, SAR represents an style metric measuring the amounts of artifacts presents in audio. Also note that SAR can be quite high even for an audio mixture that doesn't 't contain any artifact. Initial drops of SAR followed by subsequent improvements demonstrate that our proposed method attempts learn extract single-source components from the very early stages of training.
largely
G.4 PERFORMANCE COMPARISON ON ADDITIONAL DATASETS
We have also conducted additional comparisons between the proposed method and other baselines on VGGSound (Chen et al. 2020), and AudioCaps (Kim al. 2019) datasets. Both datasets contain a large variety of sounding source categories well as significant environmental noise types that make the single-source separation task particularly challenging. For a fair comparison, we have reproduced all the baselines under the same setting. Moreover, we have replaced the vanilla U-Net with our improved U-Net in most baselines.
Comparisons on VGGSound: We have conducted performance comparison on VGGSound dataset under supervised and unsupervised with synthetic, 2- & 3-source mixtures training see- narios, as shown Table I0 We observe unconditional methods suffer from convergence issues during unsupervised training that results in significant performance drops. Conditional methods, on the other hand, achieve considerably higher performance in general compared to their uncon- ditional counterparts. Since the dataset contains variable length of sounding events with large amount of noise components, image-conditional methods in general achieves lower SDR compared to language-conditional methods. However, we observe our method achieves 88%, 68% of the
23

---

Published as a conference paper at ICLR 2024
Table 10: Performance comparison on VGGSound Dataset under supervised and unsupervised training scenarios. Same set of 2-Source separation is used for all cases. All methods are reproduced under the same setting. denotes implementation with our improved U-Net model. Our proposed method largely closes the performance gap between supervised and unsupervised settings. Bold and blue represents the best and second best performance in each group, respectively.
Single-Source (Supervised)
Multi-Source (Unsupervised)
Method
2-Source
3-Source
Unconditional
PIT (Yu et al. 2017)
2.1 + 0.33
MixIT (Wisdom al.
2020
-1.7 0.44
-2.9 +: 0.51 -3.1 + 0.39
MixPIT Karamattı & Kirbiz
2022)
-1.4 0.51
Image Conditional
CLIPSep-Img (Dong al.
2022
1.3 i 0.34
-0.5 0.27
-1.2 1: 0.35
CLIPSep-Img' (Dong et al.
2022
1,7 0.36
0.4 0.31
-0.6 + 0.28
SOP*
(hao et al. 2018
1.6 = 0.23
0.3 0.41
-0.9 - 0.26
Language Conditional
CLIPSep-Text (Dong et al.
2022
2.1 =
0.8 0.31
-0.1 0.27
CLIPSep-Texte (Dong et al. 2022
2.5 + 0.29
1.2 0.44
0.5 + 0.38
BertSep*
2.0 0.27
0.7 0.31
0.3 0.22
CLAPSep*
2.3 = 0.32
1.1 0.36
0.5 + 0.28
LASS-Net (Liu 2022
2.2 1 0.31
0.9 1 0.28
0.2 1 0.29
Weak-Sup Pishdadian et a.
2020
0.6 0.39
-0.8 1 0.33
Proposed Framework
2.2
1.7 0.39
Table 11: Performance comparison AudioCaps Dataset representing natural multi-source mix- ture training. Same test set 2- separation used for all cases. All methods are reproduced under the same setting. denotes implementation with our improved U-Net model. Our proposed method significantly improves the performance over the baselines. Bold and blue represents the best and second best performance in each group, respectively.
Method
Test SDR
Image Conditional
CLIPSep-Image (Dong et al. 2022 2022
-0.7 + 0.47
LLIPSep-Imager Dong
0.4 0.33
SOP*
Zhao et al. 2018
0.2 = 0.25
Language Conditional
CLIPSep- Text (Dong et al. 2022
0.7 + 0.36
CLIPSep-Tees Text* (Dong al. 2022)
1.3 0.31
3ertSep
0.9 = 0.29
CLAPSep"
1.2 0.41
LASS-Net (Luu et al. 2022
0.8 : 0.38
Proposed Framework
2.9 0.35
24

---

Published as a conference paper at ICLR 2024
Table 12: SDR comparison on 3-source separation test set form the MUSIC Dataset under super- vised and unsupervised training scenarios. All methods are reproduced under the same setting. denotes implementation with our improved U-Net model. Bold and blue represents the best and second best performance in each group, respectively.
Single-Source (Supervised)
Multi-Source (Unsupervised)
Method
2-Source 3-Source
4-Source
Unconditional
PIT eY al. 2017
2.3
MixIT WWsdomm al.
2020
-2.3
-3.1 4
4.2
MixPIT (Karamati Kiibiz
2022
-1.9
-2.8 1.41
-3.9 i
Image Conditional
CIIPSep-Img (Dong et al.
2022
0.7 0:05
-0.8 = 0.27
-1.7 0.35
-2.9 0.3
CLIPSep-Img (Dong et al 2022
1.6 0:02
0.1 = 0.31
-0.9 0.28
-1.8 = 0.4
CoSept (Gao & Grauman
2019
1.8 0:028
0.4 : 0.37
-0.2 0.29
-0.7 0.6
SOP*
Zhao et al. 2018
1.3 1:023
-0.5 1 0.41
-1.6 1.2.6
-2.6 1.42
Language Conditional
CLIPSep-Text (Dong al.
2022
1.8 021
-0.2
-1.1 0.27
-2.1 0:45
CLIPSep-TexI (Dong et 2022
2.4 022
0.9 + 0.41
0.3 0.33
-0.4 + 0.41
BertSep*
1.9 002
0.4 0.31
-0.2 0.22
-1.1 0
CLAPPSpp*
2.2 0311
0.6 + 0.36
0.1 +028
-0.8
LASS-Net [Liu et al.
2022
2.1 025
0.5 + 0.26
0.3
-0.9 +
Weak-Sup (Pishdadian al. 2020
-1.1 0.47
2.38
+332
Proposed
3.5 1 0.35
2.7 1.422
0.8
pervised method's performance 2-source separation test in 2-source and 3-source training scenarios, respectively. Furthermore, we achieve 1.83x, and 3.4x SDR improvements over the sec- ond best method in 2-source and 3-source training scenarios, respectively, while using the same model architecture.
Comparisons AudioCaps: AudioCaps contains natural mixtures with 1 6 single source components in each mixture which makes the separation task particularly challenging. To test on AudioCaps, we prepare synthetic mixture-of-mixtures (MoM) test set by mixing random mixture pairs. Table TT shows the comparison results. Due to severe convergence issues unconditional methods, we only present comparisons for image and language conditional methods. Since the au- dio contains variable number of sounding sources with different durations, it becomes increasingly difficult condition with images compared text prompts which results in lower SDR for image conditional baselines. Nonetheless, our proposed method achieves superior performance outper- forming the second highest baseline by achieving 2.3x SDR improvement under the same setting.
G.5 COMPARISONS ON HIGHER ORDER MIXTURE TEST SETS
common test set of 2-source mixtures. Here, we present the same comparisons on more challenging test mixtures containing combinations of three single source components from the MUSIC dataset. The results reported in Table [2] In general, we observe significant performance drops compared to 2-source test setting for all methods including ours. In particular, the mix-and-sepprrate-bas in the supervised setting in the 2-source mixture training scenario. Interestingly, our weakly su- pervised approach outperforms the supervised method achieving 1.5x and 1.1x higher SDR on the 3-source test set when trained on 2-source, and 3-source mixtures, respectively. This result reveals key feature of our framework that is consistent with our observations through our other ablation studies; namely, the weak supervision proposed in our framework acts as effective regularization mechanism, that can significantly improve the model's generalization, especially when we test on the 3-source mixture set. In other words, the supervised method tends to overfit to the separation task on its training distribution, and that is why it experiences larger performance drop when the test distribution shifts. Whereas, in our framework, due to the inherent regularization properties of the weak supervision mechanism, the performance drop is less dramatic when the test distribution shifts.
So far, have shown all performance comparisons
baselines suffer from 62.5% SDR drop
25

---

Published as a conference paper at ICLR 2024
Table 13: Ablation on the effect of CLAP-constraint supervised single-source training. Same test set of 2-Source separation is used for all cases. All methods are reproduced under the same setting. denotes implementation with our improved U-Net model. Bi-modal semantic CLAP constraint introduces additional regularization in supervised training, which results in notable performance improvement. Bold and blue represents the best and second best performance each group, respec- tively.
MUSIC Dataset
VGGSound Dataset
Method
w/o CLAP
w/ CLAP
w/o CLAP
w/ CLAP
CLIPSep-Text (Dong al. 2022)
7.7 0:2
8.2 0.32
2.1 0.26
2.5 + 0.11
CLIPSep-Text* (Dong et al. 2022 BertSep*
8.3 0:27
8.8 0.41
2.5 + 0.29
2.9 + 0.44
7.9 0:27
8.3 0.35
2.0 0.27
2.5 1.31
CLAPSep*
8.1 0.31
8.7 0.34
2.3 + 0.32
2.8 0.31
G.6 EFFECT OF BI-MODAL CLAP CONSTRAINT ON SUPERVISED TRAINING
Apart from using bi-modal CLAP constraint as weak-supervision for multi-source (unsupervised) training, study its impact single source (supervised) training the baseline methods. In particular, we add the bi-modal semantic CLAP-constraint in the form LENT loss to the mix-and- separate LURL loss, while training using supervised single-source samples from the MUSIC and VGGSound datasets. The results are reported in Table [3 As the results show, there is consis- tent SDR improvement across the board when incorporate the CLAP constraint in supervised learning, even though, intuitively speaking, the weak supervision obtained from LeNr should be impertinent the presence of the strong supervision signal coming from the supervised loss. We hypothesize that the integration of CLAP constraint here introduces additional regularization to su- pervised training by transferring the knowledge obtained through CLAP's large-scale pre-training to the problem of audio source separation. This result further shows that proposed framework not only boost the separation quality in unsupervised and semi-supervised training scenarios, it can also help the supervised training itself by introducing extra dross-domain regularization.
G.7 ADDITIONAL COMPARISONS FOR SEMI- TRAINING
Table 14 depicts additional performance comparisons between supervised training on single source sounds, unsupervised training on multi-source mixture sounds, and proposed semi-supervised train- ing on both single-source and multi-source mixture sounds. We split the MUSIC training dataset with different ratios for single-source and multi-source training as mentioned before. Multi-source mixtures are composed of two single-source components here. In general, semi superviper training significantly outperforms both supervised and unsupervised training. With the increase in single source data portion, we note the performance improves in general. Similarly, unsupervised perfor- mance on multi-source mixtures also depend on available training data. Also note that the unsu- pervised performance with different splits of training data largely closes the performance gap in comparison with single-source supervised training, which is consistent with our prior observations. More notably, however, by combining both single-source and multi-source training mixtures in the proposed semi-supervised learning framework, we achieve considerable performance improvement compared 100% supervised performance reaching 9.5 SDR, which is 28% higher than the 100% scenario for the supervised baseline. This result, again, suggests the regularization effects of the pro- posed framework which can significantly reduce the reliance on single-source data and supervised training for conditional sound separation.
G.8 EFFECTS OF PROMPT TUNING FOR THE CLAP MODEL
We primarily use the bi-modal CLAP model to generate weak supervision for single-source sepa- ration from the corresponding language entity. Since the CLAP model is trained on large corpora of audio-language pairs, it can effectively generate weak supervision signals for a target dataset based hand-crafted language prompts. We have studied the performance impacts of the CLAP model customization on a target dataset by tuning language prompts with few-shot single-source reference audio samples. The results given in Table 15
For this experiment, first we separate
26

---

Published as a conference paper at ICLR 2024
Table 14: Performance comparisons between supervised, unsupervised, and semi-supervised set- tings using both single source and multi source (2-source) training data from the MUSIC dataset. Test SDR on 2-source mixtures is reported. x% of training data is used for single-source super- vised training, while (1 - x)% data used for multi-source weakly-supervised training with synthetic mixtures. CLAPSep is used for supervised training, while our proposed weakly super- vised training is applied for the multi-source data. Lastly, the semi-supervised training is applied on the combination of single source and multi-source data. Semi-supervised learning consistently achieves better performance in all data splits. Training on single-source, multi-source, and joint single- and multi-source data referred "Supervvsed", "Unsupervised", and "Semi-superrised"" method, respectively. Bold and blue represents the best and second best performance in each group, respectively.
Single Source Split
Two Source Split
SDR Performance
Supervised
Unsupervised
Semi-Supervised
100%
7.9
5%
95%
2.6
7.6
8.8
10%
90%
3.9
7.4
8.9
25%
75%
5.3
7.1
9.2
50%
50%
6.6
6.2
9.4
75%
25%
7.4
4.9
9.5
100%
8.1
Table 15: The effects of prompt tuning for our proposed framework. We use the 2-source training mixtures from MUSIC dataset. Test SDR on 2-source mixtures is reported. We initially separate several full length single source audio samples from each category for training learnable prompts. Bold and blue represents the best and second best performance each group, respectively.
(a) Ablation study on learnable prompt length with number of training audio samples per category. Each full-length audio represents single source audio. We extract several overlap- ping frames from audio samples to train the Icarnable text prompts.
Prompt Length #Audion/Category
Test SDR
(b) Comparison between the learnable and
8.1
the template-base prompts.
2
8.4
5
8.6
Prompt Type
Test SDR
Template-bused
7.9
8.2
Learnable
8.8
16
2
8.5
OCT Tzinis 2023
8.7
5
8.7
OCT+
Tzims
9.0
8.0
32
8.6
5
8.8
27

---

Published as a conference paper at ICLR 2024
Table 16: Subjective evaluation performance analysis. denotes implementation with our improved U-Net model. Bold and blue represent best and second best performance, respectively.
Training Data
Method
Correct (%) 7
Wrong (%) &
Both (%)
None (%) /
Supervised
CIIPPSepText* (ong et
2022
71.1
0.9
26.9
1.1
CLIPSep-Text Text*
Dong el 2022
30.4
20.5
40.6
8.5
Unsupervised
Proposed Framework
68.9
1.5
27.4
2.2
Sepi-supervised
Proposed Framework
82.6
0.4
16.2
0.8
few samples of full-length single-source audio samples for each category to incorporate learnable language prompts instead of the template-basee based ones. We then randomly sample single-source audio segments from a hold-out dataset to train learnable language prompts. By learning such prompts, can customize the CLAP model for our target dataset generate more informative supervision signal. In Table 15a we report the effects of the prompt lengths as well as the number of full-length audio samples per category on the 2-Source test set using the proposed weakly supervised training on 2-source mixtures. By using 5-shot single-source audio samples per category and the prompt length of 32, we achieve around 12% SDR improvement compared the template-based prompts (Table 15b).
Apart from the text-based prompt tuning using CLAP model, our proposed framework can also integrate heterogeneous prompting with other cues of the target source. Following [Tinis et al. (2023), we experiment with heterogeneous training conditions, such as text description, signal en- ergy, and harmonicity of the target sound for source separation. We use the hold-out single source samples (5/Category) for each category to estimate the additional cues for prompting target sounds in mixtures. The baseline OCT method performs on-par with our learnable text prompting technique (8.7 vs. 8.8). We note that OCT with the embedding refinement approach (OCT++) achieves the best performance of 9.0 SDR. Hence, our proposed framework can effectively integrate advanced prompting techniques to separate the target sounds from the mixture.
H SUBJECTIVE EVALUATION
We conduct a subjective human evaluation to compare different models' performances based on human perception. Following prior work (Zhao et al.) 2018 Dong et al., (022), we have randomly sampled separated sounds from 2-source mixtures and presented them to the evaluators, who are then asked "Which sound do you hear? 1. A, 2.B, 3. Both, or 4. None of them". Here, A and are replaced by the single-source sounding entities present in the input mixture, e.g. A. cat meowing, B. pigeon, dove cooing. In Table [6. we present the percentages of predicted samples that are correctly identified by the evaluator the source class (Correct), which are incorrectly perceived by the evaluator (Wrong), which contains audible sounds of both sources (Both), and which doesn't 't contain any of the target sounds (None). We use the same 30 sample predictions on 2 source mixture test sets for comparing models trained with supervised single-source, unsupervised multi- source, and semi-supervised single with multi-source data. 20 human evaluators have participated in this evaluation. We use the CLIPSep (Dong et al. 2022) method as the competitive baseline of the mix-and- aed-rree framework with the text prompts.
As the results show, our proposed framework improves over the CLIPSep baseline's 's correct percent- age statisties of 30.4% in the unsupervised setting by more than twice, reaching 68.9% and almost closing the gap with the performance of CLIPSep under the supervised regime (i.e. 71.1%). Further- more, our framework's 's performance under the semi-supervised training setup goes even beyond that of the supervised setting by significantly reducing the number of under-separated instances from 26.9% to 16.2%. leading to 11.5% increase in correct percentage statistics to the total of 82.6%. This result shows the efficacy of our weakly-supervised training strategy under both unsupervised and semi-supervised training regimes. But more importantly, these results are consistent with our quantitative evaluation results, which further corroborate conclusions.
28

---

Published as a conference paper at ICLR 2024
I QUALITATIVE COMPARISONS
In this appendix, we present qualitative comparisons between the proposed method and the mix-and- separate approach represented by the CLIPSep Dong al. (2022) framework under the unsupervised training scenario. The MUSIC dataset is used for this analysis. The models are trained and tested us- ing 2-source mixtures. The results are given Figure - Figure Due to the lack of single-source supervision in the mix-and-separate approach, most of its predictions exhibit significant spectral leakage, and large inss-interferrece. spectral loss and cross interference. Also, separation under challenging cases of spectral overlap produces reasonable performance. These examples demonstrate the effectiveness of the proposed weakly supervised training method in disentangling single-source audio components form the input mixture.
contrast, our proposed method significantly reduces the
29

---

Published as a conference paper at ICLR 2024
Ground Truth Source #1
Ground Truth Source #2
Mixture
Predicted Source #1
Predicted Source #2
Proposed
Predicted Source #1
Predicted Source #2
CLIPSep
Figure 8: Qualitative comparisons between the proposed method and the mix-and-separate approach (CLIPSep (Dong et al., 2022)): The input mixture contains piano (source 1) and violin(source 2) sounds. For the lack of single source supervision in CLIPSep, large cross-intefference visible in its prediction for the piano source. In contrast, our method significantly reduces cross-interference.
30

---

Published as a conference paper at ICLR 2024
Ground Truth Source #1
Ground Truth Source #2
Mixture
Predicted Source #1
Predicted Source #2
Proposed
Predicted Source #1
Predicted Source #2
CLIPSep
Figure 9: Qualitative comparisons between the proposed method and the mix-and-separate approach (CLIPSep (Dong et al. 2022)): The input mixture contains accordion (source 1) and ukulele(source 2) sounds. For accordion sound separation, CLIPSep exhibits significant spectral loss, while for ukulele sound separation, it shows cross-interference. However, our method largely reduces both the spectral loss for accordion sound segmentation and the cross-interference for ukulele sound segmentation.
31

---

Published as a conference paper at ICLR 2024
Ground Truth Source #1
Ground Truth Source #2
Mixture
Predicted Source #1
Predicted Source #2
Proposed
Predicted Source #1
Predicted Source #2
CLIPSep
Figure 10: Qualitative comparisons between the proposed method and the mix-and-separate ap- proach (CLIPSep (Dong et al. (022)): The input mixture contains xylophone (source and accor- dion(source 2) sounds. CLIPSep can hardly differentiate between these two sources due to a large spectral overlap. Our method, however, reasonably separates the two sounds despite showing some spectral loss for the accordion prediction.
32

---

Published as a conference paper at ICLR 2024
Ground Truth Source #1
Ground Truth Source #2
Mixture
Predicted Source #1
Predicted Source #2
Proposed
Predicted Source #1
Predicted Source #2
CLIPSep
Figure 11: Qualitative comparisons between the proposed method and the mix-and-separate ap- proach (CLIPSep (Dong et al. (2022)): The input mixture contains tuba (source 1) and congas(source 2) sounds. CLIPSep cannot properly identify the tuba sound due to the lack of single source train- ing. Whereas, our method achieves considerable results in separating the tuba sound. nevertheless, some spectral loss can be observed for the tuba sound separation, which reasonable considering the significant spectral overlaps of the two sounds.
33


Reviews:
[{"review_id": "cuu95by8Bk", "reviewer": "Reviewer_ScdR", "summary": "In this paper, a novel weakly supervised learning framework is introduced for language-conditioned audio separation, particularly when single-source audio samples are absent during training. Leveraging the cross-modal similarity between audio and language using the pretrained CLAP model, this approach generates weak supervision signals for single-source prompts during training, effectively bridging the gap between training and test data distributions. The authors perform experiments to showcase the superiority of this framework over other baseline models and they show that they can bridge the gap between the performance of the unsupervised and the supervised methods.", "questions": "Would it be possible to add some human evaluations for the best baseline and the proposed method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally speaking well written with a lot of information needed to be reproduced by the researchers in the field of sound separation.\n- The authors have conducted several experiments to show that their algorithm performs better than some other existing text-based separation algorithms.\n- It is very important that the proposed method yields a significant performance improvement over the unsupervised baseline of MixIT.\n- I also like the novel extension of MixIT with conditioning.", "weaknesses": "1. The authors should make clear the distinction of when the proposed method is trained using only weak supervision and when it is semi-supervised trained. For instance, in Table 1, I think the proposed framework row refers to the semi-supervised version of the method, thus the authors should rename the column to ‘Fully supervised’ from ‘Supervised’. Maybe a better idea is to specify the data used to train ALL the parts of each model and have two big columns ‘Mixture training data’ and ‘Single source data’ which will make it much more prevalent of what is which. \n\n2. Building upon my previous argument, I think that when one is using these large pre-trained networks on single-source data like CLAP, the underlying method becomes supervised in a sense, or to put it more specifically supervised with unpaired data. The authors should clearly explain these differences throughout the manuscript.\n\n3. I think the authors should include stronger text-based sound separation baselines like the model and ideally the training method that uses heterogeneous conditions to train the separation model in [A] which has already shown to outperform LASS-Net (Liu et. al 2022) which is almost always the best performing baseline in this paper.\n\nI would be more than happy to increase my score if all the above weaknesses are addressed by the authors. \n\n[A] Tzinis, E., Wichern, G., Smaragdis, P. and Le Roux, J., 2023, June. Optimal Condition Training for Target Source Separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ITHTHybcCA", "reviewer": "Reviewer_SLiU", "summary": "This paper introduces a novel conditional source separation model designed to improve separation performance in weakly-labeled data. The approach leverages CLAP embeddings to enforce semantic consistency in the separation results, even when individual source tracks are not explicitly provided but merged within the data.\n\nThe proposed model comprises three key components:\n\n1. Mix-Separation Paradigm: This component creates separation datasets from weakly-labeled noisy data, where more than one source is present in a single audio clip.\n\n2. CLAP Embeddings: The model employs CLAP, a joint embedding of text and audio modalities, to establish a constraint that ensures the separation results align with the same semantic characteristics, primarily focusing on timbre. This constraint is enforced through the CLIPLoss between the separation audio embeddings and the associated single-source text embeddings (e.g., 'This is the sound of [class]').\n\n3. Training: The entire separation model is trained by integrating the above two components with corresponding loss functions.\n\nThe experimental results demonstrate that the incorporation of CLAP embeddings significantly enhances separation performance by making the model sensitive to mixed and overlapping events in the data, ultimately leading to superior separation outcomes.", "questions": "Two questions are proposed in the weaknesses section, regarding the question of CLAP vs. timbre classifier, and the latest result of LASSNet either using CLAP or CLIP as the query embedding.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents a compelling and well-articulated examination of a critical limitation within the existing zero-shot/few-shot conditional source separation pipeline: the dataset. It addresses this issue within the context of cross-modal learning (CLAP).\n\nThe paper's technical novelty is commendable. It ingeniously employs CLAP to establish a method for validating the consistency of separation outputs, integrating loss objectives that effectively converge both intermediate semantic consistency and separation content accuracy. Furthermore, it extends this innovative approach into a semi-supervised learning framework, broadening the potential applications of the data.\n\nThe paper demonstrates a robust and comprehensive experimental performance evaluation. It goes beyond merely listing state-of-the-art results by including self-implemented ablations, which, while not previously explored, offer valuable points of comparison. The extensive analysis of existing work enriches the paper's conclusion and strengthens its contribution to the field", "weaknesses": "I assigned a 'weak reject' score to this paper due to what I consider to be a critical issue in the design of the separation pipeline and an empirical problem in the experimental results.\n\nFirstly, the paper introduces the utilization of CLAP embeddings as a constraint to enforce semantic similarity between the separation output and the source constraint. While this is a technically novel approach, it raises two important concerns:\n\n(1) The use of CLAP, or similar contrastive learning models, to enforce semantic similarity is not as solid as it appears. This is because CLAP is also trained with the same noisy, weakly-labeled data. When the paper uses prompts like '[this is the sound of flute],' these prompts have been trained on audio clips that contain more than just flute sounds, potentially including piano and guitar sounds. Therefore, the data used in the current setting for training this proposed separation model, is exactly the data that CLAP was trained on (and even more). This raises questions about whether the constraint effectively operates as intended, especially in cases where the separation sample contains multiple sources, one of which is the target source. To ascertain the true effectiveness of CLAP as an exclusive text embedding, further experiments are necessary.\n\n(2) The semantic alignment is a fancy phrase but in this audio separation case, primarily restricts the timbre of the audio rather than aligning the semantic content. While a high CLAP score indicates similarity in timbre, it doesn't guarantee similarity in content. Semantic alignment comes more in the case when the text itself contains different arrangement and logics. However, here the model only uses one or two single prompt such as [this is the mixture/sound of [class]]. A worth-noting question to ask is: why not use a timbre classier to verify it? You can send the separation result to an audio classification model to force the output to be in one timbre. Or a more direct question is: did you try this as an ablation study to show that using CLAP to align the semantic similarity is better than training a timbre classifier to enhance the separation results? I think this ablation study is a good point to show the technical solid of this paper. Such an ablation study would strengthen the paper's technical foundation without compromising its flexibility. Because for every data trained in this pipeline, you should at least know the source type to make the prompt text. A timbre classifier seems can be trained in a fixed (but large) candidacy slot.\n\nFrom another perspective, the paper presents a substantial number of experiments to demonstrate the efficacy of the separation results. The appendix is comprehensive and aids in clarifying various aspects. However, the experimental results on the MUSIC dataset, achieving an SDR performance of 7.9 and 9.9 under the semi-supervised learning paradigm, do not significantly outperform the results in the recent LASSNet paper [1] (9.75 dB and 10.45 dB). While I acknowledge that both papers were produced during the same time period, this raises concerns about the necessity of CLAP as an embedding for enforcing semantic similarity. It prompts a reconsideration of whether CLAP could be superseded by a simpler text- or audio-query model with CLAP or CLIP embeddings, but a different separation design that doesn't require semantic similarity.\n\nBesides, any listenable demo for this paper? It would be beneficial to provide a demo page to assess the performance of the separation and we might see more interesting findings, such as how CLAP is trying to enhance the separation result. \n\nIn light of these concerns, I eagerly anticipate the authors' response to address these questions. Until it can be convincingly demonstrated that CLAP is a crucial element in enhancing performance or that it offers greater functionality than a single classifier constraint, I believe this paper may fall slightly below the acceptance threshold for ICLR.\n\n\n[1] Separate Anything You Describe, Arxiv.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l4ja2ZrtjT", "reviewer": "Reviewer_fPmw", "summary": "The paper presents a framework designed to utilize text-audio data pairs, utilizing the text-audio encoding network, CLAP, for source separation. There are two primary training objectives:\n\n1. Unsupervised Training: Two mixed audio recordings are combined, and the separation network aims to reconstruct each original mixture.\nWeakly-supervised \n2. Weakly-supervised Audio-language Training: Here again, two mixed audios are combined. The goal is to separate each individual source present in both mixtures. This is achieved with the bi-modal contrastive loss and a reconstruction loss. The bi-modal contrastive loss is employed to enhance the similarity between the predicted single source and its corresponding language prompt. The reconstruction loss, mirroring the unsupervised training, aims to reconstruct the original individual mixture prior to combining.\nThe paper provides comprehensive experiments, demonstrating that the proposed method surpasses baseline methods in performance.", "questions": "Does the proposed method necessitate knowledge of all components present in the mixture? Furthermore, if the data contains background noises not mentioned in the text prompt, could this impact the model's performance? In a weakly-supervised setting, it's assumed that the individual sources contribute to the mix. For instance, if the prompt mentions people talking and birds chirping, but the recording was made in a city environment with background motor sounds, how would the model handle such discrepancies?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a unique framework that integrates unsupervised, weakly supervised, and semi-supervised training for source separation, conditional on language prompts. The experimental results validate the efficacy of the proposed method.\n2. The paper conducts detailed experiments, contrasting the proposed approach with various baselines. Ablation tests further underscore the superior performance of the proposed method compared to existing techniques.", "weaknesses": "Although it might not be a weakness, it is a bit questionable how the model performance will improve in large-dataset settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)