CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos
Authors: Hao-Wen Dong
Naoya Takahashi
Yuki Mitsufuji
Julian McAuley
Taylor Berg-Kirkpatrick
Venue: ICLR 2023 poster (2023)
Keywords: universal sound separation
source separation
contrastive language-image pre-training
multi-modal learning
self-supervised learning

Abstract: Recent years have seen progress beyond domain-specific sound separation for speech or music towards universal sound separation for arbitrary sounds. Prior work on universal sound separation has investigated separating a target sound out of an audio mixture given a text query. Such text-queried sound separation systems provide a natural and scalable interface for specifying arbitrary target sounds. However, supervised text-queried sound separation systems require costly labeled audio-text pairs for training. Moreover, the audio provided in existing datasets is often recorded in a controlled environment, causing a considerable generalization gap to noisy audio in the wild. In this work, we aim to approach text-queried universal sound separation by using only unlabeled data. We propose to leverage the visual modality as a bridge to learn the desired audio-textual correspondence. The proposed CLIPSep model first encodes the input query into a query vector using the contrastive language-image pretraining (CLIP) model, and the query vector is then used to condition an audio separation model to separate out the target sound. While the model is trained on image-audio pairs extracted from unlabeled videos, at test time we can instead query the model with text inputs in a zero-shot setting, thanks to the joint language-image embedding learned by the CLIP model. Further, videos in the wild often contain off-screen sounds and background noise that may hinder the model from learning the desired audio-textual correspondence. To address this problem, we further propose an approach called noise invariant training for training a query-based sound separation model on noisy data. Experimental results show that the proposed models successfully learn text-queried universal sound separation using only noisy unlabeled videos, even achieving competitive performance against a supervised model in some settings.

Published as a conference paper at ICLR 2023
CLIPSEP: LEARNING TEXT-QUERIED SOUND SEPARA- TION WITH NOISY UNLABELED VIDEOS
Hao- Wen Dong'
Naoya Takahashi' i Yuki Mitsufuji'
Julian McAuley?
Taylor Berg-Kirkpatrick'
Sony Group Corporation University of California San Diego
hwdong@uesd.edu (Naoya. Takahashi, Yuhki wwuki..iitsuuul Mitsuuuuj jentsuffjj@@soy.... Jesony. com, {jmcauley, tberg) tberg)@uusd. @uesdd
edu
ABSTRACT
Recent years have seen progress beyond domain-specific sound separation for speech music towards universal sound separation for arbitrary sounds. Prior work on universal sound separation has investigated separating a target sound out of an audio mixture given a text query. Such text-queried sound separation systems provide a natural and scalable interface for specifying arbitrary target sounds. However, supervised text-queried sound separation systems require costly labeled audio-text pairs for training. Moreover, the audio provided in existing datasets is often recorded in controlled environment, causing a considerable generalization gap to noisy audio in the wild. In this work, we aim to approach text-queried universal sound separation by using only unlaheled data. We propose to leverage the visual modality as a bridge to learn the desired audio-textual correspondence. The proposed CLiPSep model first encodes the input query into a query vector using the contrastive amguage-image pretraining (CLIP) model, and the query vector is then used to condition an audio separation model to separate out the target sound. While the model is trained on image-audio pairs extracted from unlabeled videos, at test time we can instead query the model with text inputs in a zero-shot setting, thanks to the joint language-image embedding learned by the CLIP model. Further, videos in the wild often contain off-screen sounds and background noise that may hinder the model from learning the desired audio-textual correspondence. To address this problem, we further propose an approach called noise invariant training for training a query- noisy data. Experimental results show that the proposed models successfully learn text-queried universal sound separation using only noisy unlabeled videos, even achieving competitive performance against a supervised model in some settings.
sound separation model on
INTRODUCTION
Humans can focus on to specific sound in the environment and describe it using language. Such abilities are learned using multiple calalities-auditory for selective listening, vision for learning the concepts of sounding objects, and language for describing the objects or scenes for communication. In machine listening, selective listening is often cast as the problem of sound separation, which aims to separate sound sources from an audio mixture (Cherry, 1953; Bach & Jordan, 2005). While text queries offer a natural interface for humans to specify the target sound to separate from a mixture (Liu et al., 2022; Kilgour et al., 2022), training a text-queried sound separation model in supervised manner requires labeled audio-text paired data single-soure types, which can be costly acquire. Moreover, such isolated sounds often recorded in controlled environments and have a considerable domain gap to recordings in the wild, which usually contain arbitrary noise and reverberations. In contrast, humans often leverage the visual modality to assist learning the sounds of various objects (Baillargeon, 2002). For instance, by observing a dog barking, a human can associate the sound with the dog, and can separately learn that the animal is called "dog." Further, such learning is possible even if the sound is observed in a noisy environment, e.g.,
recordings of a vast number of sound
Work done during an internship at Sony Group Corporation
1 "Corrsponning author

---

Published as a conference paper at ICLR 2023
when car is passing by or someone is talking nearby, where humans can still associate the barking sound solely with the dog. Prior work in psychophysics also suggests the intertwined cogmition of vision and hearing (Sekuler et al., 1997; Shimojo & Shams, 2001; Rahne et al., 2007).
Motivated by this observation, we aim to tackle text-queried sound separation using only unlabeled videos in the wild. We propose text-queried sound separation model called CLIPSep that leverages abundant unlabeled video data resources by utilizing the contrastive (CLIP) (Radford al., 2021) model to bridge the audio and text modalities. As illustrated in Figure I, during training, the image feature extracted from a video frame by the CLiP-image encoder is used to condition a sound separation model, and the model is trained to separate the sound that corresponds to the image query in sule-vue setting. Thanks to the properties of the CLIP model, which projects corresponding text and images to close embeddings, at test time we instead use the text. feature obtained by the CLIP-text encoder from a text query in zero-shot setting.
pretraining
However, such zero-shot modality transfer can be challenging when we use videos in the wild for training as they often contain off-screen sounds and voice overs that can lead to undesired audio- visual associations. To address this problem, we propose the noise invariant training (NIT), where query-based separation heads and permutation in- variant separation heads jointly estimate the noisy target sounds. We validate in our experiments that the proposed noise invariant training reduces the zero-shot modality transfer gap when the model is trained on a noisy dataset, sometimes achieving competitive results against a fully supervised text-queried sound separation system.
.(u. Audio
"(IE"
Audo
Dosinel comesponnence (Fow daza)
Abundent video data resounces
Tes time
Tnwining time
CLIP pectuairiig
AS
CLIP joint
Z Image
Tix
Image
Toxtt
enbbolding
Figure 1: An illustration of modality transfer.
Our contributions can be summarized as follows: 1) We propose the first text-queried universal sound separation model that can be trained on unlabeled videos. 2) We propose new approach called noise invariant training for training query- -eeyy-baeed sound separation model on noisy data in the wild. Audio samples can be found on an our demo website.' For reproducibility, all source code, hyperparameters and pretrained models are available at: https:// /gi thub. com/ /CLS/CL
2 RELATED WORK
Universal sound separation Much prior work on sound separation focuses on separating sounds for a specific domain such as speech (Wang & Chen, 2018) or music (Takahashi & Mitsufuji, 2021; Mitsufuji et al., 2021). Recent advances in domain specific sound separation lead several attempts to generalize to arbitrary sound classes. Kavalerov et al. (2019) reported successful results on separating arbitrary sounds with a fixed number of sources by adopting the permutation invariant training (PIT) (Yu et al., 2017), which was originally proposed for speech separation. While this approach does not require labeled data for training, a post-selection process is required as we cannot not tell what sounds are included in each separated result. Follow-up work (Ochiai et al., 2020: Kong et al., 2020) addressed this issue by conditioning the separation model with class label to specify the target sound in a supervised setting. However, these approaches still require labeled data for training, and the interface for selecting the target class becomes cumbersome when we need a large number of classes to handle open-domain data. Wisdom et al. (2020) later proposed an unsupervised method called mixture invariant training (MixIT) for learning sound separation on noisy data. MixIT is designed to separate all sources at time and also requires post-selection process such as using pre-trained sound classifier (Scott al., 2021), which requires labeled data for training, to identify the target sounds. We summarize and compare related work in Table 1.
Query-based sound separation Visual information has been used for selecting the target sound in speech (Ephrat et al., 2019; Afouras et al., 2020), music (Zhao et al., 2018; 2019; Tian et al., 2021) and universal sounds (Owens & Efros, 2018; Gao et al., 2018; Rouditchenko et al., 2019). While many image-queried sound separation approaches require clean video data that contains isolated sources, Tzinis et al. (2021) introduced an unsupervised method called AudioScope for separating on-screen sounds using noisy videos based the MixIT model. While image queries can serve as
hhttps://sony.. wig.in..ggttt.u..... wtt.hb...o/CLIPS... io/CLIPSep/
2

---

Published as a conference paper at ICLR 2023
Method
Query type
Unlabeled data
Noisy data
USS (Kavalerov al., 2019) MixIT (Wisdom al., 2020)
x
X
Universal Sound Selector (Ochiai al., 2020)
Label
USS-Label (Kong et al., 2020)
Label
3 $
PixelPlayer (Zhao al., 2018)
Image
AudioScope (Tzinis et al., 2021)
Image
SoundFilter (Gfeller et al., 2021)
Audio
Zero-shot audio separation (Chen et al., 2022)
Audio
Text- Lext-Querie (Liu et al., 2022)
Text
Text/Audio- Queried (Kilgour al., 2022)
Text / Audio
CLIPSep (ours)
Text Image
CLIPSep-NIT (ours)
Text Image
Table I: Comparisons of related work in sound separation. '(V)" indicates that the problem of noisy training data is partially addressed. CLIPSep-NIT denotes the proposed CLIPSep model trained with the noise invariant training. To the best of our knowledge, no previous work has attempted the problem of label-free text-queried sound separation.
natural interface for specifying the target sound in certain use cases, images of target sounds become unavailable in low-light conditions and for sounds from out-of-screen objects.
Another line of research uses the audio modality to query acoustically similar sounds. Chen et al. (2022) showed that such approach can generalize to unseen sounds. Later, Gfeller al. (2021) cropped two disjoint segments from single recording and used them as a query-target pair to train a sound separation model, assuming both segments contain the same sound source. However, in many cases, it is impractical to prepare a reference audio sample for the desired sound as the query.
Most recently, text-queried sound separation has been studied as provides a natural and scalable interface for specifying arbitrary target sounds as compared to systems that use fixed set of class labels. Liu et al. (2022) employed pretrained language model to encode the text query, and condition the model to separate the corresponding sounds. Kilgour et al. (2022) proposed a model that accepts audio or text queries in a hybrid manner. These approaches, however, require labeled text-audio paired data for training. Different from prior work, our goal is learn text-queried sound separation for arbitrary sound without labeled data, specifically using unlabeled noisy videos in the wild.
Contrastive language-image-audee been used as a pretraining of joint embedding spaces among text, image and audio modalities for downstream tasks such audio classification (Wu et al., 2022; Guzhov et al., 2022) and sound guided image manipulation (Lee et al., 2022). Pretraining is done either in a supervised manner using labels (Guzhov al., 2022; Lee et al., 2022) or self-supervised manner by training an additional audio encoder to map input audio to the pretrained CLIP embedding space (Wu et al., 2022). In contrast, we explore the zero-shot modality transfer capability of the CLIP model by freezing the pra-trdined CLIP model and directly optimizing the rest of the model for the target sound separation task.
pretraining The CLIP model (Radford al., 2021) has
3 METHOD
3.1 CLIPSEP- CHIPSEP-LEARNING TEXT-QUERIED SOUND SEPARATION WITHOUT LABELED DATA
In this section, we propose the CLIPSep model for text-queried sound separation without using labeled data. We base the CLIPSep model on Sound-of-Pixels (SOP) (Zhao et al., 2018) and replace the video analysis network of the SOP model. As illustrated in Figure 2, during training, the model takes inputs audio mixture æ Ei=1 8;, where 8 Sn are the n audio tracks, along with their corresponding images yı...., Yn extracted from the videos. We first transform the audio mixture æ into magnitude spectrogram X and pass the spectrogram through an audio U-Net bRonneberger et al., 2015; Jansson et al., 2017) to produce l (2 n)) intermediate masks M Mr. On the other stream, each image is encoded by the pretrained CLIP model (Radford et al., 2021) into an embedding e; 5R522 The CLIP embedding e; will further be projected query vector
3

---

Published as a conference paper at ICLR 2023
Query
ploto playing didgeridoo
(est tims))
CLIP model (preraized; fixed)
Projection chamdss
Query vestor (4.)
Predicted mask (R)
Souree
Audio model
Audio U-Net
Spectrogram
chomnels)
Source
Predicted mask 2 (8,)
Projection layer (4 chamdss
Query vector 2 (a.)
pholo of motocboat
CLIP model (precaised; liech
acceleration
speedboat
(ber time)
Query model
Figure 2: An illustration of the proposed CLIPSep model for 2. During training, we mix audio from two videos and train the model to separate each audio source given the corresponding video frame as the query. At test time, we instead use a text query the form of "a photo of [user input query/" to query the sound separation model. Thanks to the properties of the pretrained CLIP model, the query vectors we obtain for the image and text queries are expected to close.
qi E R by projection layer, which is expected to extract only audio-relevant information from e., Finally, the query vector qi will be used to mix the intermediate masks into the final predicted masks M, = - a (Wiyiy , + b:). where w, E R is a learnable scale vector, b, E R a learnable bias, and (-) the sigmoid function. Now, suppose M, is the ground truth mask for source s,. The training objective of the model is the sum of the weighted binary cross entropy losses for each source:
CCLIPSOP Σ WBCE(M, M.) = Σ x X (-
(-M, M, lMg log M. (1 -MM (1 (1-M) (-.M.) ).)
(1)
At test time, thanks to the joint image-text embedding offered by the CLIP model, we feed a text query instead of an image to the query model to obtain the query vector and separate the target sounds accordingly (see Appendix A for an illustration). As suggested by Radford et al. (2021), we prefix the text query into the form of photo of [user input query]" J" reduce the generalization gap.
3.2 NOISE INVARIANT TRAINING- HANDLING NOISY DATA IN THE WILD
While the CLIPSep model separate sounds given image text queries, it assumes that the sources are clean and contain few query-irrelevant sounds. However, this assumption does not hold for videos the wild as many of them contain out-of-screen sounds and various background noises. Inspired by the mixture invariant training (MixIT) proposed by Wisdom et al. (2020), we further propose the noise invariant training (NIT) to tackle the challenge training with noisy data. As illustrated in Figure we introduce additional permutation invariant heads called noise heads to the CLIPSep model, where the masks predicted by these heads are interchangeable during loss computation. Specifically, we introduce additional projection layers, and each of them takes as input the sum of all query vectors produced by the query heads (i.e., Σ q:) and produce a vector that later used to mix the intermediate masks into the predicted noise mask. In principle, the query masks produced by the query vectors are expected to extract query query-relever relevant sounds due to their stronger correlations to their corresponding queries, while the interchangeable noise masks should 'soak up' other sounds.
We extract three frames with 1-see intervals and compute their CLIP embedding as the input the projection layer to reduce negative effects when the selected frame does not contain the objects of interest. "Similar to how prepare the image queries, create four queries from the input text query using four query templates (see Appendix and take their mean CI.IP emhedding as the input to the projection layer.
4

---

Published as a conference paper at ICLR 2023
pf of (est
model
Producted murk
dideesson
model
Projectior layer
Query vecter (9))
Noion incariant training
Tranting
mIck ER RD
Profksed
Eroond moth mast:
Poodicted case
Audio model
Source
Mir
Seurce
Projuation layur
M
Sportmgram
Audio U-Net
L+ (cchmooh))
Prodicted suivo raak
Noise invariat
1 (R,
CLIP medel Iperaierii Good)
Projectior layer chommer))
Query vector 2 fec)
ylosso of secoloation (wslwwr))
Query
nignal
Producted
Figure 3: An illustration of the proposed CLIPSep-NIT model for n 2. Similar to CLIPSep, we train the model to separate each audio source given the corresponding query image during training and switch to using a text query at test time. The two predicted noise masks are interchangeable for loss computation during training, and they are discarded at test time (grayed out paths).
Mathematically, let MP ..... M be the predicted query masks and MN ..... MA be the predicted noise masks. Then, the noise invariant loss defined as:
Mi, min (1. + +a,)) )
CNIT
min
WBCE
(2)
where Σn denotes the set of all permutations of {l....,,., ,n.' Take n = 2 for example." We consider the two possible ways for combining the query heads and the noise heads:
(Arrangement 1)
M₁ min (1, M Mi) Mr min (1, + M))
(3)
(Arrangement 2)
M min A1 + M,,"). Mi min (1, + +MN).
(4)
Then, the noise invariant loss is defined as the smallest loss achievable:
N/r = min (WBCE(M, (1, M)) + WBCE M2), WBCE (11 Mi) + WBCE Mc).
(5)
Once the model is trained, we discard the noise heads and use only the query heads for inference (see Appendix A for an illustration). Unlike the MixIT model (Wisdom et al., 2020), our proposed noise invariant training still allows us to specify the target sound by input query, and it does not require any post-selection process as we only use the query heads during inference.
In practice, we find that the model tends to assign part of the target sounds to the noise heads as these heads can freely enjoy the optimal permutation to minimize the loss. Hence, we further introduce a regularization term to penalize producing high activations on the noise masks:
LaSG
max
0. mumm Ar," )
(6)
where 7 [0, ni] is hyperparameter that we will refer to as the noise regularization level. The proposed regularization has no effect when the sum of the means of all the noise masks is lower than a predefined threshold y, while having a linearly growing penalty when the sum higher than γ- Finally, the training objective of the CLIPSep-NIT model is a weighted sum of the noise invariant loss and regularization term: CCLIPSep-Ne NIT L NIT + XCREG, where We set À = 0.1 for all experiments, which we find work well across different settings.
a weight hyperparameter.
"We note that CLIPSep-NII -IT considers 2n sources in total as the model has n queried heads and n noise heads. While PIT (Yu et al.. 2017) and MixIT (Wisdom et al., 2020) respectively require (((2)!! and ()22") search consider 2n sources, the proposed NIT only requires O(re)) permutation in the loss computation. "Since our goal is not to further separate the noise into individual sources but to separate the sounds that correspond to the query, n may not need to be large. In practice, we find that the CLIPSep-NIT model with = 2 already learns to handle the noise properly and can successfully transfer to the text-queried mode. Thus, we throughout this paper and leave the testing on larger n future work.
5

---

Published as a conference paper at ICLR 2023
Unlabeled data
Post-proe. free
Query type
SDR [dB]
Model
Training
Test
Mean
Median
Mixture
0.00 + 0.89
0.00
Text-queried models CLIPSep
Image
Text
5,49 + 0.72
4.97
CLIPSep-Text
Text
Text
7.91 + 0.81
7.46
CLIPSep-Hybrid
Text + Image
Text
8.36 + 0.83
8.72
Image-queried models
SOP (Zhao et al., 2018)
Image Image
Image Image
6.59 + 0.85
6.22
CLIPSep
7.03 0.70
5.85
CLIPSep-Text CLIPSep-HYybri
Text Text Image
Image Image
6.25 + 0.72
6.19
8.06 0.79
8.01
Nonqueried models
LabelSep
Label
Label
8.18 0.80
7.82
(Yu al., 2017)
X
8.68 0.76
7.67
Table 2: Results on the MUSIC dataset. Standard emoor are reported in the mean SDR column. Bold values indicate the largest SDR achieved per group.
4 EXPERIMENTS
We base implementations the code provided by Zhao al. (2018) (https://gi http:///gtttuu.ccoo com/ hangzhaomi wtt///SundoofPiiii Implementation details can be found in Appendix C.
4.1 EXPERIMENTS ON CLEAN
We first evaluate the proposed CLIPSep model without the noise invariant training musical instrument sound separation task using the MUSIC dataset, as done in (Zhao et al., 2018). This experiment is designed focus evaluating the quality of the learned query vectors and the shot modality transferability of the CLIPSep model on a small, clean dataset rather than showing its ability to separate arbitrary sounds. The MUSIC dataset collection of 536 video recordings of people playing musical instrument out of 11 instrument classes. Since no existing work has trained text-queried sound separation model using only unlabeled data to our knowledge, we compare the proposed CLIPSep model with two baselines that serve as upper bounds the PIT model (Yu et al., 2017, see Appendix for an illustration) and a version of the CLIPSep model where the query model replaced by learnable embeddings for the labels, which addition, we also include the SOP model (Zhao al., 2018) to investigate the quality of the query vectors the CLIPSep and SOP models share the same network architecture except the query model.
will refer to as the LabelSep model. In
We report the results Table 2. Our proposed CLIPSep model achieves mean momull-o-distortton ratio (SDR) (Vincent et al., 2006) of 5.49 dB and median SDR of 4.97 dB using text queries in zero-shot modality transfer setting. When using image queries, the performance of the CLIPSep model is comparable that of the SOP model. This indicates that the CLIP embeddings are as informative as those produced by the SOP model. The performance difference between the CLIPSep model using text and image queries at test time indicates the zero-shot modality transfer gap. We observe 1 1.54 dB and 0.88 dB differences on the mean and median SDRs, respectively. Moreover, we also report in Table 2 and Figure 4 the performance of the CLIPSep models trained on different modalities to investigate their modality transferability in different settings. We notice that when we train the CLIPSep model using text queries, dubbed CLIPSep-Text the mean SDR using text queries increases to 7.91 dB. However, when we test this model using image queries, we observe 1.66 dB difference the mean SDR compared that using text queries, which is close to
quary
[mage query
SDR
CLIFSep
CLIPSep Text
CLiPSep-Ilybnd
Figure 4: Mean SDR and standard errors of the models trained and tested on dif- ferent modalities.

---

Published as a conference paper at ICLR 2023
MUSIC
VGGSound-Clent
Unlabeled data
Post-proc. free
Median SDR
Median
Model
Mean SDR
Mean SDR
SDR
Mixture
4.49 1.41
2.04
-0.77 = 1.31
-0.84
Text-queried models
CLIPSep
9.71 1.21
8.73
2.76 + 1.00
CLIPSep-NIT
10.27 1.04
10.02
3.05 - 0.73
3.26
BERTSep
4.67
0.44
4.41
5,09 0.80
5.49
CLIPSep-Tens
10.73 0.99
9.93
5,49 + 0.82
5.06
Image-queried models
SOP (Zhao et al., 2018)
11.44 1.18
11.18
2.99 + 0.84
3,89
CLIPSep
12.20
1.17
12.42
5,46 1 0.79
5.35
CLIPSep-NIT
11.28
1.08
10.83
4.84 + 0.66
3.57
CLIPSep-Text
9.89
1.04
8.09
2.45 1 0.70
1.74
Nonqueried models PIT (Yu et al., 2017)
12.24
1,20
12.53
5.73 + 0.79
4.97
LabelSep
5.55 = 0.81
5.29
Table 3: Results of the MUSIC and VGGSound-Cleann evaluations (see Section 4.2). Standard errors are reported the mean SDR [dB] columns. Bold values indicate the largest SDR achieved per group. We use 7 0.25 for CLIPSep-NIT, Note that the LabelSep model does not work on the MUSIC dataset due to the different label taxonomies the MUSIC and VGGSound datasets.
the mean SDR difference we observe for the model trained with image queries. Finally, we train CLIPSep model using both text and image queries in alternation, dubbed as CLIPSep-Hybbrd, We that it leads the best test performance for both text and image modalities, and there is only mean SDR difference of 0.30 dB between using text and image queries. As a reference, the LabelSep model trained with labeled data performs worse than the CLIPSep-Hybrig model using text queries. Further, the PIT model achieves a mean SDR of 8.68 dB and a median SDR 7.67 dB, but it requires post-processing figure out the correct assignments.
4.2 EXPERIMENTS ON NOISY DATA
Next, we evaluate the proposed method on a large-scale dataset aiming universal sound separation. We use the VGGSound dataset (Chen et al., 2020), a large-scale audio-visual dataset containing more than 190,000 10-second videos in the wild out of more than 300 classes. We find that the audio in the VGGSound dataset often noisy and contains off-screen sounds and background noise. Although train the models such noisy data, it is not suitable to use the noisy data targets for evaluation because fails to provide reliable results. For example, if the target sound labeled as "dog barking" also contains human speech, separating only the dog barking sound provides a lower SDR value than separating the mixture of dog barking sound and human speech even though the text query is "dog the labels only for evaluation but not for training.) To avoid this issue,
barking". (Note that we consider the following two evaluation settings:
Samples in the MUSIC dataset are used as clean targets and mixed with sample in the VGGSound dataset as an interference. The separation quality is evaluated on the clean target from the MUSIC dataset.
MUSIC
we do not use the MUSIC dataset for training, this can be considered as zero-shot transfer to new data domain containing unseen sounds (Radford et al., 2019; Brown et al., 2020). To avoid the unexpected overlap the target sound types in the MUSIC and VGGSound datasets caused by the label mismatch, we exclude all the musical instrument playing videos from the VGGSound dataset in this setting.
We manually collect 100 clean samples that contain distinct target sounds from the VGGSound test set, which we will refer to as VGGSound- sample in VGGSound-Clens with another in the test set of VGGSound. Similarly, consider the VGGSound audio as an interference sound added to the relatively cleaner VGGSound-Cles audio and evaluate the separation quality the VGGSound- stem.
VGGSound-Cles
We mix an audio
7

---

Published as a conference paper at ICLR 2023
(a) Input mixture (b) Ground truth (c) Interference
(d) Prediction
(e) Noise head (I) Noise head
Figure 5: Example results of the proposed CLIPSep-NIT model with ? 0.25 on the MUSIC dataset. We mix the an audio sample ("violin" in this example) in the MUSIC dataset with an interference audio sample ("people sobbing" in this example) in the VGGSound dataset to create an artificial mixture. (b) and (c) show the reconstructed signals using the ground truth ideal binary masks. The spectrograms are shown in the log frequency scale. We observe that the proposed model successfully separates the desired sounds (i.e., (d) from query-irrelevant noise (i.e., (e) and (f)).
Table 3 shows the evaluation results. First, CLIPSep successfully learns text-queried sound separation even with noisy unlabeled data, achieving 5.22 dB and 3.53 dB SDR improvements over the mixture on MUSIC and VGGSound-
respectively. By comparing CLIPSep and CLIPSep-NIT, we observe that NIT improves the mean SDRs in both settings. Moreover, on MUSIC* CLIPSep-NIT's performance matches that of CLIPSep-Ters dB lower mean SDR and even 0.05 dB higher median SDR This result suggests that the proposed self-supervised text-queried sound separation method can learn separation capability competitive with the fully supervised model in some target sounds. In contrast, there still a gap between them on VGGSound-Clean" possibly because the videos of non-music-ing inntriiennttumentt objects are more noisy in both audio and visual domains, thus resulting in more challenging zero-shot modality transfer. This hypothesis is also supported by the higher zero-shot modality transfer gap (mean SDR difference of image- and text-queried mode) of 1.79 dB on VGGSound-Clent for CLI. In addition, we consider another baseline model that replaces the CLIP model in CLIPSep with a BERT encoder (Devlin et al., 2019), which we call BERTSep. Interestingly, although BERTSep performs similarly to CLIPSep- Text VGGSound Clean + the performance of BERTSep is significantly lower than that of CLIPSep-Text on MUISC* indicating that BERTSep fails to generalize to unseen text queries. We hypothesize that the CLIP text embedding captures the timbral similarity of musical instruments better than the BERT embedding do, because the CLIP model is aware of the visual similarity between musical instruments during training. Moreover, is interesting to see that CLIPSep outperforms CLIPSep-NIT when an image query is used at test condition), possibly because images contain richer context information such as objects nearby and backgrounds than labels, and the models can use such information to better separate the target sound. While CLIISSp has to fully utilize such information, CLIPSep-NIT can use the noise heads to model sounds that are less relevant to the image query. Since we remove the noise heads from CLIPSep-NIT during the evaluation, it can rely less on such information from the image, thus improving the zero-shot modality transferability. Figure 5 shows example of the separation results on MUSIC* (see Figures 12 to 15 for more examples). We observe that the two noise heads contain mostly background noise. Audio samples can be found on our demo website.'
which utilizes labels for training, achieving only a 0.46
than that of 1.01 dB on MUSIC+
time
4.3 EXAMINING THE EFFECTS OF THE NOISE REGULARIIZTION LEVEL 7
In this experiment, we examine the effects of the noise regularization level y in Equation (6) by changing the value from to I. As we can see from Figure 6 (a) and (b), CLIPSep-NIT with = 0.25 achieves the highest SDR on both evaluation settings. This suggests that the optimal y value is not sensitive to the evaluation dataset. Further, we also report in Figure 6 (c) the total mean noise head activation, Σ mean((M. ), on the validation set. As MNN is the mask estimate for the noise, the total mean noise head activation value indicates to what extent signals are assigned to the noise head. We observe that the proposed regularizer successfully keeps the total mean noise head activation close to the desired level, 7 for & 0.5. Interestingly, the total mean noise head activation still around 0.5 when y 1.0, suggesting that the model inherently tries to use both the query- heads and the noise heads to predict the noisy target sounds. Moreover, while we discard the noise heads during evaluation in our experiments, keeping the noise heads can lead to higher SDR shown in
8

---

Published as a conference paper at ICLR 2023
11.5
CUIPSpp-IT (wl ialaddas heati
16.0
64
1.3
85
(a)
Make wwg.laiiiiin
Modan bation
(c)
Noiaa www.hetaationiioo
Figure 6: Effects of the noise regularization level y for the proposed CLIP model mcan SDR for the (a) MUSIC and (b) VGGSound -Ceann"" evaluations, and (c) the total mean noise head activation, D (M,". on the validation set. The shaded areas show standard errors.
Figure 6 (a) and (b), which can be helpful in certain use cases where a post-processing procedure similar to the PIT model (Yu et al., 2017) is acceptable.
5 DISCUSSIONS
For the experiments presented in this paper, we work on labeled datasets so that we can evaluate the performance of the proposed models. However, our proposed models do not require any labeled data for training, and can thus be trained larger unlabeled video collections in the wild. Moreover, observe that the proposed model shows the capability of combing multiple queries, e.g., "a photo of [query AJ and (query BJ," to extract multiple target sounds, and report the results the demo website. This offers a more natural user interface against having to separate each target sound and mix them via additional post-processing step. We also show in Appendix G that our proposed model is robust to different text queries and can extract the desired sounds.
In experiments, we often observe modality transfer gap greater than 1 dB difference of SDR. A future research direction is to explore different approaches to reduce the modality transfer gap. For example, the CLIP model is pretrained on different dataset, and thus finetuning the CLIP model on the target dataset can help improve the underlying modality transferability within the CLIP model. Further, while the proposed noise invariant training is shown improve the training on noisy data and reduce the modality transfer gap, it still requires sufficient audio-visual correspondence for training video. In other words, if the audio and images are irrelevant in most videos, the model will struggle to learn the correspondence between the query and target sound. In practice, we find that the data in the VGGSound dataset often contains off-screen sounds and the labels sometimes correspond to only part of the video content. Hence, filtering the training data to enhance its audio-visual correspondence can also help reduce the modality transfer gap. This can achieved by self-supervised andio-visuall correspondence prediction (Arandjelović & Zisserman, 201 20;b) or temporal synchronization (Korbar et al., 2018; Owens & Efros, 2018).
Another future direction is explore the semi- supervied setting where small subset of labeled data can be used to improve the modality transferability. We can also consider the proposed method pretraining unlabeled data for other separation tasks in the low-resource regime. We include in Appendix H a preliminary experiment in this aspect using the ESC-50 dataset (Piczak, 2015).
CONCLUSION
In this work, we have presented a novel text-queried universal sound separation model that can be trained on noisy unlabeled videos. In this end, we have proposed use the contrastive image- language pretraining to bridge the audio and text modalities, and proposed the noise invariant training for training a query-based sound separation model on noisy data. We have shown that the proposed models can leamn to separate an arbitrary sound specified by text query out of mixture, even achieving competitive performance against fully supervised model in some settings. We believe proposed approach closes the gap between the ways humans and machines learn to focus sound in a mixture, namely, the multi-modal supervise learning paradigm of humans against the supervised learning paradigm adopted by existing label-based machine learning approaches.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We would like to thank Stefan Uhlich, Giorgio Fabbro and Woosung Choi for their helpful comments during the preparation of this manuscript. We also thank Mayank Kumar Singh for supporting the setup of the subjective test in Appendix F. Hao- Wen thank J. Yang and Family Foundation and Taiwan Ministry of Education for supporting his PhD study.
REFERENCES
Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self-supervised learning of audio-visual objects from video. In Proc. ECCV, 2020. (Cited page 2.)
Relja Arandjelović and Andrew Zisserman. Look, listen and learn. In Proc. ICCV, 2017a. (Cited on page 9.)
Relja Arandjelović and Andrew Zisserman. Objects that sound. In Proc. ECCV, 2017b. (Cited on page 9.)
Francis R. Bach and Michael I. Jordan. Blind one-microphone speech separation: A spectral learning approach. In Proc. NIPS, 2005, (Cited on page 1.)
Renée Baillargeon. The acquisition of physical knowledge in infancy: A summary in eight lessons. Blackwell handbook of childhood cognitive development, 2002. (Cited on page 1.)
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In Proc. NeurIPS, 2020. (Cited on page 7.)
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. VGGSound: A large-scale audio-visual dataset. In Proc. ICASSP, 2020. (Cited on page 7.)
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg- Borg-Kirkpatrick, and Shlomo Dubnov. Zero- shot audio source separation through query-based learning from weakly-labeled data. Proc. AAAI, 2022. (Cited on page 3.)
E Colin Cherry. Some experiments on the recognition of speech, with one and with two ears. The Journal of the acoustical society of America, 25, 1953. (Cited on page 1.)
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL, 2019. (Cited on page 8.)
Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independee audio-visual model for speech separation. ACM Transactions on Graphics, 37, 2019. (Cited on page 2.)
Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proc. ECCV, 2018. (Cited on page 2.)
Beat Gfeller, Dominik Roblek, and Marco Tagliasacchi. One-shot conditional audio filtering of arbitrary sounds. In Proc. ICASSP, 2021. (Cited page 3.)
Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel, AudioCLIP: Extending CLIP to image, text and audio. In Proc. ICASSP, 2022. (Cited page 3.)
Andreas Jansson, Eric Humphrey, Nicola Montecchio, Rachel Bittner, Aparn Kumar, and Tillman Weyde. Singing voice separation with deep U-Net convolutional networks. In Proc. ISMIR, 2017. (Cited on page.3. 3.)
10

---

Published as a conference paper at ICLR 2023
Hyy Kavalerov, Scott Wisdom, Hakan Erdogan. Brian Patton, Kevin Wilson, Jonathan Le Roux, and John R. Hershey. Universal sound separation. In Proc. WASPAA, 2019. (Cited on pages 2 and 3.)
Kevin Kilgour, Beat Gfeller, Qingqing Huang, Aren Jansen, Scott Wisdom, and Marco Tagliasacchi. Text-driven separation of arbitrary sounds. In Proc. INTERSPEECH, 2022. (Cited on pages and 3.)
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proc. ICLR, 2015. (Cited on page 13.)
Qiugiang Kong, Yuxuan Wang, Xuchen Song, Yin Cao, Wenwu Wang, and Mark D. Plumbley. Source separation with weakly labelled data: An approach to computational auditory scene analysis. In Proc. ICASSP, 2020. (Cited on pages and 3.)
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In Proc. NeurIPS, 2018. (Cited on page 9.)
Seung Hyun Lee, Wonseok Roh, Wonmin Byeon, Sang Ho Yoon, Chan Young Kim, Jinkyu Kim, and Sangpil Kim. Sound-guided semantic image manipulation. In Proc. CVPR, 2022. (Cited on page 3.)
Xubo Liu, Haohe Liu, Qiuqiang Kong, Xinhao Mei, Jinzheng Zhao, Qiushi Huang, MarkD Plumbley, and Wenwu Wang. Separate what you describe: Language-quest audio source separation. In Proc. INTERSPEECH, 2022. (Cited on pages and 3.)
Yuki Mitsufuji, Giorgio Fabbro, Stefan Uhlich, Robern Stöter, Alexandre Défossez, Minseok Kim, Woosung Choi, Chin-Yun Yu, and Kin Cheuk. Music demixing challenge 2021, 2021. (Cited on page 2.)
Tsubasa Ochiai, Mare Delcroix, Yuma Koizumi, Hiroaki Ito, Keisuke Kinoshita, and Shoko Araki. Listen what you want: Neural network-based universal sound selector. In Proc. INTERSPEECH, 2020. (Cited on pages 2 and 3.)
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proc. ECCV, 2018. (Cited on pages 2 and 9.)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Proc. NewrIPS, 2019. (Cited on page 13.)
Karol J. Piczak. ESC: Dataset for environmental sound classification. In Proc. MM, 2015. (Cited on pages and 15.)
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical Report of OpenAl, 2019. (Cited on page 7.)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proc. ICML, 2021.
(Cited on pages 2, 3, 4, and 12.)
Torsten Rahne, Martin Böckmann, Hellmut von Specht, and Elyse S Sussman. Visual cues can modulate integration and segregation of objects in auditory scene analysis. Brain research, 2007. (Cited on page 2.)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Proc. MICCAI, 2015. (Cited on pages and 13.)
Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh McDermott, and Antonio Torralba. Self- supervised audio-visual co-segmentation. In Proc. ICASSP, 2019. (Cited page 2.)
Wisdom Scott, Aren Jansen, Ron J. Weiss, Hakan Erdogan, and John R. Hershey. Sparse, efficient, and semantic mixture invariant training: Taming in-the-wild unsupervised sound separation. In Proc. WASPAA, 2021. (Cited page 2.)
11

---

Published as a conference paper at ICLR 2023
Robert Sekuler, Allison B. Sekuler, and Renee Lau. Sound alters visual motion perception. Nature, 385, 1997. (Cited on page 2.)
Shinsuke Shimojo and Ladan Shams. Sensory modalities are not separate modalities: plasticity and interactions. Current Opinion in Newrobiology, 11, 2001. (Cited on page 2.)
Fabian-Robert Stöter, Antoine Liutkus, and Nobutaka Ito. The 2018 signal separation evaluation campaign. Proc. LVA/CA, 2018. (Cited on page 13.)
Naoya Takahashi and Yuki Mitsufuji. Densely connected multidilated convolutional networks for dense prediction tasks. In Proc. CVPR, 2021. (Cited on page 2.)
Yapeng Tian, Di Hu, and Chenliang Xu, Cyclic co-learning sounding object visual grounding and sound separation. In Proc. CVPR, 2021. (Cited on page 2.)
Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel P. W. Ellis, and John Hershey. the wild with AudioScope: Unsupervised audio-visual separation of on-screen sounds. In Proc. ICLR, 2021. (Cited on pages 2 and 3.)
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE Transactions on Audio, Speech, and Language Processing, 14(4): 1462-1469, 2006. (Cited page 6.)
DeLiang Wang and Jitong Chen. Supervised speech separation based on deep learning: An overview. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(10):102--722 2018. (Cited on page 2.)
Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron J. Weiss, Kevin Wilson, and John R. Hershey. Unsupervised sound separation using mixture invariant training. In Proc. NeurlPS, 2020. (Cited on pages 2, 3, 4, and 5.)
Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2CLIP: Learning robust audio representations from CLIP. In Proc. ICASSP, 2022. (Cited page 3.)
Dong Yu, Morten Kolbæk, Zheng-Hur Hua Tan, and Jesper Jensen. Permutation invariant training of deep models for gpeaker-nndependen multi-talker speech separation. In Proc. ICASSP, 2017. (Cited on pages 2, 6, 7, 14, 18, 19, 20, and 21.)
Jingzhao Zhang. Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: theoretical justification for adaptivity. Proc. ICLR, 2020. (Cited on page 13.)
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proc. ECCV, 2018. (Cited on pages 2, 3, 6, 7, and 14.)
Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. Proc. ICCV, 2019. (Cited page 2.)
A INFERENCE PIPELINE OF CLIPSEP AND CLIPSEP-NIT
Figure 7 illustrates the inference pipeline for the proposed CLIPSep and CLIPSep-NIT models. For the CLIPSep-NIT model, we discard the noise heads at test time. The masked spectrogram is combined with the input phase and converted to the waveform by the inverse short-time Fourier transform (STFT).
B QUERY ENSEMBLING
Radford et al. (2021) suggest that using a prompt template in the form "a photo of [user input query/" helps bridge the distribution gap between text queries used for zero-shot image classification and text in the training dataset for the CLIP model. They further show that the ensemble of various prompt templates improve the generalizability. Motivated by this observation, adopt similar idea and use several query templates at test time (see Table 4). These query templates are heuristically chosen handle the noisy images extracted from videos.
12

---

Published as a conference paper at ICLR 2023
Query model
Predicted mask
Masked spectrogram
CLIP model (poetained; fixed weigits)
Projection layer (2 chaanels)
Query vactor
Reconstructed waveform
Query phoso playing didgeridoo
Spectrogram
Audio model
Mixture
Audio U-Net
Figure 7: Inference pipeline of the proposed CLIPSep and CLIPSep-NIT models. Note that the mixture spectrogram and the masks are shown in log frequency scale, while the masked spectrogram shown linear frequency scale.
Query template
Example query for the label "dog barking"
a photo of [user input query)
photo dog barking
a photo of the small [user input query)
photo of the small dog barking
a low resolution photo of [user input query]
low resolution photo of dog barking
photo of many [user input query]
photo of many dog barking
Table 4: Query templates used in our experiments.
C IMPLEMENTATION DETAILS
We implement the audio model as a 7-layer U-Net (Ronneberger et al., 2015). We use k 32. We use binary masks as the ground truth masks during training while using the raw, real-valued masks for evaluation. We train all the models for 400,000 steps with batch size of 32. We use the Adam optimizer (Kingma & Ba, 2015) with 31 = 0.9, B2 0.999 and c 10-8 In addition, we clip the norm of the gradients 1.0 (Zhang et al., 2020). We adopt the following learning rate schedule with warm-up-the learning rate starts from 0 and grows to 0.001 after 5,000 steps, and then it linearly drops to 0.0001 at 100,000 steps and keeps this value thereafter. We validate the model every 10,000 steps using image queries
do not assume labeled data is available for the validation set. We use a sampling rate of 16,000 Hz and work on audio clips of length 65,535 samples (+= 4 seconds). During training, randomly sample center frame from video and extract three frames (images) with 1-sec intervals and 4-sec audio around the center frame. During inference, for image-queried models, extract three frames with 1-sec intervals around the center of the test clip. For the spectrogram computation, we use filter length of 1024, a hop length of 256 and window size of 1024 in the short-time Fourier transform (STFT). We resize images extracted from video to a size of 224-by-222 pixels. For the CLIPSep-Hyy Hybrid model, we alternatively train the model with text and image queries, i.e., one batch with all image queries and next with all text queries, and so on. We implement all the models using the Py PoToch library (Paszke et al., 2019). We compute the signal-to-distortoo ratio (SDR) using museval (Stöter et al., 2018).
In our preliminary experiments, we also tried directly predicting the final mask by conditioning the audio model on the query vector. We applied this modification for both SOP and CLIPSep models, however, observe that this architecture is prone to overfitting, We hypothesize that this is because the audio model is powerful enough to remember the subtle clues in the query vector, which hinder the generalization over-determined masks and then combines them on the basis of the query vector, which avoids the overfitting problem due the simple fusion step.
sound and query. In contrast, the proposed architecture first predicts
13

---

Published as a conference paper at ICLR 2023
Permutation invariant training
Predicted
Ground truth
Source
(Ixl
Audio
Source
Spectrogram
Predicted mask 2
Ground truth
An illustration of the PIT model for n 2. The two predicted masks are interchangeable during the loss computation. Since the predicted masks are interchangeable, the PIT model requires an additional post-selection step to obtain the target sound.
Figure 8:
D PERMUTATION INVARIANT TRAINING
Figure 8 illustrates the permuatation invariant training (PIT) model (Yu et al., 2017). The permutation invariant loss is defined follows for = 2.
LPIT = min [WBCE(M), Mi) + WBCE(M,, M2), WBC M2) + WBCE(M,, MI).
(7)
where M and M2 are the predicted masks. Note that the PIT model requires additional post- selection step to obtain the target sound.
E QUALITATIVE EXAMPLE RESULTS
We show in Figures 12 to 15 some example results. More results and audio samples can be found at https://sony. wit.ub.io/CLIPS.... i
F SUBJECTIVE EVALUATION
We conduct subjective test evaluate whether the SDR results aligned with perceptual quality. As done in the Sound of Pixel (Zhao et al., 201 8), separated audio samples are randomly presented to evaluators, and the following question asked: "Which sound do you hear? 1. A. B, Both, or 4. None of them". Here A and B are replaced by labels of their mixture sources, e.g. A=accordion, B=engine accelerating. Ten samples (including naturally occurring mixture) are evaluated for each model and 16 evaluators have participated in the evaluation. Table 5 shows the percentages of samples which are correctly identified the target sound class (Correct), which are incorrectly identified the target sound sources (Wrong), which are selected as both sounds are audible (Both), and which are selected as neither of the sounds are audible (None). The results indicate that the evaluators more often choose the correct sound source for CLIPSep-NIT (83.8%) than CLIPSep (66.3%) with text queries. Notably, CLIPSep-NIT with text-query obtained a higher correct score than that with image-query, which matches the training mode. This is probably because image queries often contain information about backgrounds and environments, hence, some noise and off-screen sounds are also suggested by the the information of target sounds, thus, the query head more aggressively extract the target sounds.
and leak to the query head. In contrast, text-queries purely contain
G ROBUSTNESS TO DIFFERENT QUERIES
To examine the model's 's robustness to different queries, we take the same input mixture and query the model with different text queries. We use the CLIPSep-NIT model the MUSIC dataset and
14

---

Published as a conference paper at ICLR 2023
Model
Query type
Correct [%]
Wrong [%]
Both [%]
None
Mixture
17.5
10.0
72.5
0.0
Image Text
70.6
0.0
29.4
0.0
CLIPSep
66.3
3.8
30.0
0.0
Image Text
68.8
1.9
28.1
1.3
CLIPSep-NIT
83.8
0.6
15.0
0.6
Table 5: Subjectíve test results.
Model
Post-processing free
Dataset
SDR
Training
Finetuning
Mean
Median
Mixture
0.00 0.44
0.00
PIT
VGGSound
4.90 + 0.26
2.44
VGGSound
1.07 + 0.28
2.34
CLIPSep
FSC-50
5.18 0.26
5.09
VGGSound
ESC-50
6.73 0.26
5.89
Table 6: Results on the ESC-50 dataset. Standard errors are reported in the mean SDR column.
report in Figure 16 the results. We that the model is robust different queries and can extract the desired sounds. Audio samples can be found at hottps://sony/ github. itriii////// /CLIPPPpp/
H FINETUNING EXPERIMENTS ON THE ESC-50 DATASET
In this experiment. we aim to examine the possibilities of having a clean dataset for further finetuning, We consider the ESC-50 dataset (Piczak, 2015), collection of 2.000 high-quality environmental audio recordings, as the clean dataset here. We report the experimental results in Table 6. We can see that the model pretrained VGGSound does not generalize well to the ESC-50 dataset as the ESC-50 contains much cleaner sounds, i.e., without quere-irrelevant sounds and background noise. Further, we train the CLIPSep model from scratch on ESC-50 dataset, can only achieve mean SDR of 5.18 8 dB and median SDR of 5.09 dB. However, if we take the model pretrained on the VGGSound dataset and finetune it on the ESC-50 dataset, can achieve a mean SDR of 6.73 dB and median SDR of 4.89 dB, resulting in improvement of 1.55 dB on the mean SDR.
I TRAINING BEHAVIORS
We present in Figure the training and validation losses along the training progress. Please note that we only show the results obtained using text queries for reference but do not use them for choosing the best model. We also evaluate the intermediate checkpoints every 10,000 steps and present in Figure 10 the test SDR along the training progress. addition, for the CLIPSep-NIT model, we visualize in Figure 11 the total mean noise head activation, Σ mean (M,, along the training progress. We can see that the total mean noise head activation stays around the desired level for = 0.1, 0.25. For = 0.5 and the unregularized version, the total mean noise head activation converges to a similar value around 0.55.
15

---

Published as a conference paper at ICLR 2023
0.380
6.310
Vald
0.333
Saat
0:729
0.33
0.20
0233
0.25
6.225
0,22
0:230
0229
503e0
100060
130003
230030
59382
100600
130003
200000
50030
320038
190000
200000
Sapa
Staps
Sape
(a) CLIPSep
(b) PIT
(c) CLIPSep-NIT (unregularized)
0.36
Vilid
Vead
0.325
Veld tast
6.325
Vias Redt
0.22
Yold Hobd
0.310
0.38
6.38
0.29
19200
39393
176035
199000
Depe
Staps
(d) CLIPSep-NIT (7 0.5)
(e) CLIPSep-NIT (y 0.25)
CLIPSep-NIT ( 0.1)
Figure 9: Training and validation losses along the training progress the VGGSound dataset. We also include the losses computed using text queries instead of image queries. The y-axes are intentionally set to the same range for easy comparison. Note that we do not use the validation results obtained with text queries for choosing the best model.
Iaue
Bien
Bepe
(a) CLIPSep
(b) PIT
(c) CLIPSep-NIT (unregularized)
500
Telk adeo Seado
Text
Teck fel beado
Teal
That colea hoadd
SECERD
51003s
1800C0
150000
22000
200000
(d) CLIPSep-NIT (y 0.5)
(e) CLIPSep-NIT (y = 0.25)
CLIPSep-NIT (? (y= 0.1)
Figure 10: Test SDR along the training progress on the VGGSound-Cless dataset. The y-axes are intentionally set to the same range for easy comparison.
16

---

Published as a conference paper at ICLR 2023
Total mesn muise lesed activation
0.7
0.1
0.5
= 0.25
Unregularized
0.6
0.5
50000
100000 Stops
150000
200000
Figure Total mean noise head activation, Σ mcan(MM)) on the validation set for the CLIPSep- NIT models along the training progress.
17

---

Published as a conference paper at ICLR 2023
Ground truth
Mixture
Ground truth
(Interference)
CLIPSep-NIT (7 0.25)
PIT (Yu al., 2017)
CLIPSep
Noise head (CLIPSep-NIT)
Noise head 2 (CLIPSep-NNT)
Figure 12: Example results on the MUSIC dataset. Target murce-"violin"" interference-""eoplee
sobbing"; quoly-"violin".. The spectrograms and masks are shown in the log and linear frequency scales, respectively.
18

---

Published as a conference paper at ICLR 2023
Ground truth (Interference)
Mixture
Ground truth
CLIPSep-NIT (7 0.25)
PIT (Yu al., 2017)
CLIPSep
Noise head (CLIPSep-NIT)
Noise head 2 (CLIPSep-NIT)
Figure 13: Example results on the MUSIC+ dataset. Target source- "acousticc guitar"; interference "cheetah chirrup", query-""ccoustic guitar". The spectrograms and masks are shown in the log and linear frequency scales, respectively.
19

---

Published as a conference paper at ICLR 2023
Ground truth (Interference)
Mixture
Ground truth
CLIPSep-NIT (7 = 0.25)
PIT (Yu al., 2017)
CLIPSep
LabelSep
Noise head (CLIPScp-NIT)
Noise head 2 (CLIPSep-NNT)
Figure 14: Example results on the VGGSound-Clean+ dataset. Target source-"ca growling": seterferencee"rallleeee car train wagon":; query-"cal growling". The spectrograms and masks are shown in the log and linear frequency scales, respectively.
20

---

Published as a conference paper at ICLR 2023
Ground truth (Interference)
Mixture
Ground truth
CLIPSep-NIT (7 = 0.25)
PIT (Yu et al., 2017)
CLIPSep
LabelSep
Noise head (CLIPSep-NIT)
Noise head 2 (CLIPScp-NIT)
Figure 15: Example results on the VGGSound C dataset. Target pource-"electric grinder grinding": interference- horn car horn honking": query-"elecctrcc grinder grinding". The spectrograms and masks are shown in the log and linear frequency scales, respectively. Note that the PIT model requires a post-selection step to get the correct source. Without the selc-sclection step, the PIT model return the right source in only a 50% chance.
21

---

Published as a conference paper at ICLR 2023
Ground truth (Interference)
Mixture
Ground truth
Prediction (Query: "acoustic guitar")
Prediction (Query: "guitar")
Prediction
(Query: "a man is playing acoustic guitar")
Prediction
Prediction (Query: "car engine")
QQery: "a man playing acoustic guitar in room")
Figure 16: Query robustness experiment on the MUSIC dataset. Target source "acoustic guitar"; interference "chhett
"cheetah chirrup". The spectrograms are shown the log frequency scale.
22


Reviews:
[{"review_id": "wfPxWV7muaD", "reviewer": "Reviewer_LZWB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "### Summary\n\nThis paper proposes a text-queried universal sound separation model that can be trained on noisy in-the-wild videos (i.e. videos that contain both on-screen and off-screen sounds). Two versions are proposed: CLIPSep and CLIPSep-NIT (CLIPSep with noise invariant training).\n\nCLIPSep: during training, mix audio from two videos. Extract the CLIP embedding of an image frame; from the spectrogram of the audio mixture, predict k masks; predict a k-dim query vector q_i from the CLIP embedding; predict overall mask for source i using query vector q_i to combine across the k masks, with an additional k-dimensional scaling weight w_i and scalar bias b_i; audio is reconstructed using inverse STFT on masked STFT. Training loss is weighted binary cross-entropy between estimated mask and ground-truth mask (so training requires isolated source audio from on-screen-only video). During inference, CLIP embedding is computed from text (assuming this will be close to CLIP embedding of image), and just one mask is predicted for the source described by the text.\n\nCLIPSep-NIT: same as CLIPSep, except that for each of the n sources during training, an additional \"noise\" mask is predicted, which is an additional query vector that combines the k predicted masks with a noise query vector. Then during training, all permutations of the noise masks added to the source masks are considered, and the permutation with the minimum error is used. It seems the purpose of the noise masks is to \"soak up\" sounds not related to the CLIP embedding. At test time, the noise masks are discarded.\n\n### Contributions\n\n1. First text-driven separation model (to my knowledge) that can be trained on noisy videos, enabled by the NIT trick.\n\n2. NIT is a contribution, though I feel its novelty is relatively minor, since it's just a constrained version of permutation invariant training (PIT).", "summary_of_the_review": "Overall, a nice paper that accomplishes training text-driven separation on noisy in-the-wild data. Achieves good performance compared to prior approaches, and qualitative demos are convincing.", "strength_and_weaknesses": "### Strengths\n\n1. To my knowledge, this is the first method to train text-queried separation on noisy mixtures.\n\n2. The evaluation is done on both MUSIC+ and VGGSound-Clean+, measuring performance on both music separation and universal separation, and these results are convincing.\n\n3. Paper includes link to anonymized demo page, which is convincing.\n\n### Weaknesses\n\n1. I think the paper makes the post-selection step required for a MixIT model to be harder than it actually is. For a MixIT-trained model with N outputs, it's pretty easy to pick a source, e.g. with a sound classification network. This setup was actually proposed with a classification-regularized loss in:\nWisdom, Scott, Aren Jansen, Ron J. Weiss, Hakan Erdogan, and John R. Hershey. \"Sparse, efficient, and semantic mixture invariant training: Taming in-the-wild unsupervised sound separation.\" In 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 51-55. IEEE, 2021. (https://arxiv.org/pdf/2106.00847.pdf)\nAnother advantage of MixIT is that the outputs are more interpretable, compared to models that rely on conditioning, such as the one described in this paper. Thus, I think it may be good to discuss the pros and cons of separate-then-select versus conditional separation in the paper.\n\n2. This statement is a bit incorrect:\n\n    \"However, AudioScope still requires a post-selection process if there is more than one predicted on-screen channel.\"\n\n    The goal of AudioScope is to recover all on-screen sounds in a single channel, which is what the model does: it uses on-screen probabilities as mixing weights across the sources.\n\n3. \"where s1, . . . , sn are the n audio sources,\": in practice, these are mixtures, right? The model is just assuming that they are single sources. it might be good to refine the terminology here a bit.\n\n4. Some explanation of why k masks are predicted, then combined, would be good. I think this is kind of analogous to the multiple output sources in MixIT, which can be combined for a particular user interface or output goal, e.g. AudioScope combines with on-screen probabilities to get an estimate of on-screen sound.\n\n5. The equation for computing the overall source mask from the k masks is confusing. What does the \\odot versus the \\cdot mean? If w_i is k-dimensional, I don't see a sum over k, since it's \\odot'ed with scalar q_{ij} times \\tilde{M}_j. Should this actually be w_{i,j}? Please specify how this is done.\n\n6. The model uses mask-based losses, which, in my own experience, are often suboptimal compared to signal based losses (i.e. computed loss in time domain, backpropping through iSTFT applied to masked mixture STFT). Also, in the NIT loss, adding masks together and applying a ceil of 1 does not exactly correspond to adding signals in the time domain, because of STFT consistency. it would be interesting to try time-domain based losses for this network, and see if that provides any improvement. Also, the architecture in the MixIT paper used mixture consistency, so that output sources sum up to the original input mixture. This might also be a useful constraint on the architecture here.\n\n7. I think best practice for reporting units in decibels is to use only one decimal place. Humans can often not even hear 0.1 dB of difference. Thanks, by the way, for reporting std dev from the mean and median.\n\n8. More explanation of the motivation of NIT would be very welcome. My intuition is that it helps \"soak up\" extra noise by providing additional output sources, but this might not be right. Please add some explicit discussion of the motivation.\n\n### Typos and minor comments\n\na. \"For eaxmple,\" -> \"For example,\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WBq109SWE4", "reviewer": "Reviewer_8zRs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a source separation system for text-queried source separation. The authors propose to train the system with a picture query during training time, however in inference time they use text for the query. In addition to the basic system, they also propose to add a mixit layer at the end of the pipeline to increase the noise robustness of the system. ", "summary_of_the_review": "I think this paper proposes an interesting training methodology. I think it's above the acceptance threshold. My only problem with it is the lack of comparison with text-query-only models. (See my comment above) \n\n\n---------------\n\n\nUpdate after rebuttal: The authors provided a BERT based baseline, and I increased my score. \n", "strength_and_weaknesses": "Strengths: \n- The proposed problem is definitely interesting, and I can see the practical applications of this system. \n- The results (shared in the link https://dezimynona.github.io/separation/) seems to suggest that the system is doing what is intended. \n\nWeaknesses: \n- I think it would have been nice to also compare with a baseline system which uses sentence embeddings as a guide. This paper could be a nice point of comparison https://arxiv.org/pdf/2203.15147.pdf. You could have done this comparison in two ways. 1) On your experiments you can directly train this model and compare 2) You could have taken a pretrained systems for both your approach, and the baseline and compare in a zero-shot manner. The VGGSound+None experiment that you have on your demo page is a nice option for this. \n- There is little difference between the separation quality of Clipsep and Clipsep+NIT. In some of the examples on your demo page the two methods sound very similar. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ROJnpULtdHF", "reviewer": "Reviewer_CpEt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "CLIPSep demonstrates how a pretrained CLIP model can be used to train a source separation model using unlabeled videos and achieve competitive results in some settings.", "summary_of_the_review": "CLIPSep shows a novel approach to training source separation on unlabeled videos. However, I am concerned that the main value to this approach will come from scaling up the training dataset to many in-the-wild videos, but that setting was not attempted in this paper. As shown by the VGGSound results, it's possible that there will be problems with the noisiness of in-the-wild videos that prevent this technique from working without additional insights and modifications.\n\n*Update*: As mentioned above, I am now much less concerned about the ability of this technique to scale up.", "strength_and_weaknesses": "Strengths:\n\n- This model shows a path toward training a sound separation model that is text queryable for arbitrary sources and can be trained on unlabeled video data.\n- Results are competitive with labeled approaches in some settings.\n\nWeaknesses:\n\n- The goal of this approach is to be able to scale up training on an arbitrary number of in-the-wild videos. However, the model is trained and evaluated only on relatively small and clean datasets. Even when the data is somewhat noisy (e.g., the offscreen noises in VGGSound), the model starts to exhibit difficulties using only text queries. The authors acknowledge these issues in the Discussion section and provide some ideas for improvement, but I'm concerned that we don't know how well the model will actually scale up to in-the-wild video datasets. It's possible that entirely different techniques will end up being needed to get to that level of unlabeled training.\n  - *Update*: After discussion with the authors, I realized I misunderstood the scale of VGGSound and how representative it is of \"in the wild\" audio, so I am much less concerned with how well this technique will scale up.\n\n- Motivation for some of the architecture design choices is not fully explained and alternatives are not fully explored (details below).\n  - *Update*: After discussion with the authors, they have updated the paper to explain some of these choices. I found the discussion around \"early fusion\" vs. \"late fusion\" particularly interesting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F2W7idAm7g", "reviewer": "Reviewer_dcmL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a self-supervised way to do sound separation using a frozen pre-trained CLIP, along with video data (assumed to also have audio). \n\nThe core method of CLIPSep is shown in Fig 2. During training, they run the frames of two different videos through CLIP in order to independently get embeddings, which are then projected into another space by a learnable projection mapping. In parallel, they add together the audio streams of both videos, encode this as a spectrogram, and then run that through an Audio UNet. They then independently combine the output of the UNet with each video's projections in order to predict an audio mask. That audio mask is compared against the true audio mask for the video in order to get a loss.\n\nFigure 3 expands on CLIPSep and introduces CLIPSep-NIT in order to better account for noisy streams of audio. It's more complicated, but the gist is to create audio masks that account for the noise found in in-the-wild videos. This is patterned after the MixIT approach from Wisdom et al.\n\nThey then show that this self-supervised approach can be comparable to supervised datasets on two different tasks involving mixing test VGGSound and eval MUSIC+ with VGGSound.", "summary_of_the_review": "Before I increase my score, I would want to see the paper improve significantly wrt clarity, notably section 4.1 and Figure 3. I would also like to see a better explanation for the evaluation approach in 4.2 and perhaps something to support it elsewhere in the literature. If those are satisfied I will increase my score because I do like this paper and think the underlying method deserves to be recognized.", "strength_and_weaknesses": "Strengths:\n1. The main strength is that the method is novel. I like this idea a lot and think there's something materially interesting if you ramp up the dataset size. \n2. The comparisons are also clear. The tables show the delineations between the models that you compare and I don't have trouble understanding what's going on wrt numbers.\n\n\nWeaknesses:\n1. The explanation of the model feels like some info is left out, notably from where the images are extracted with respect to the audio. As I understand, there is a singular image per video (2 total to be exact), but it's unclear how the audio is determined around that. It can't be instantaneous. Is it 10 seconds around it? Maybe I'm missing it, but this seems important for reproduction.\n2. There should be audio samples here. It's hard to truly evaluate what's going on without audio samples. I don't see any such links in the paper.\n3. I don't understand at all what is section 4.1. What is the task? I read through it a few times and it's unclear to me what you're actually doing there.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "9lwxAOuwnR", "reviewer": "Reviewer_vU71", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper under review proposes a method of selecting a single sound source from a mixture of sounds in a video via a text description of the visual component of the video. The system can be trained on unlabeled data, aka unsupervised training. This is a novel configuration of using a pre-trained audio-visual correspondence model to allow text queries to select the single audio source to separate from a mixture in the video. Unlike what is claimed in the paper though in section 4.1, work was published this year on querying by text to separate a source from an audio mixture (this is understandable given timing). There is also a contribution of a form of noise invariant training that allows for the model to account for sounds in the mixture that have no correspondence in the video. The results are conducted on test sets, MUSIC and VGGSound-Clean, that have audio collected from the wild (YouTube), however they have been artificially mixed to yield multiple sound sources. The results are competitive with PIT, although PIT has a \"post-processing\" requirement.\n\n", "summary_of_the_review": "Overall the paper is novel in a narrow sense. It builds on the Sound Of Pixels work but adding a method of textual query. The results are good, demonstrating the approach is viable, however in the opinion of this reviewer, not overwhelming excellent (other reviewers may disagree). It feels more incremental than ground breaking, hence the recommendation to marginally accept.", "strength_and_weaknesses": "Strengths:\n* A new configuration of querying by text to separate out an audio source in a video with sources that have corresponding audio and visual signals. \n* Shows performance competitive with state-of-the-art in sound separation\n\nWeaknesses\n* Tests are made by artificially combining samples of YouTube videos. Can you conduct test results on naturally occurring mixtures?\n* Results report an automatically computed quantitative metric, ie SDR. It is unclear whether how this corresponds to actual user preferences. Since the results are close, could a qualitative survey be conducting comparing the results of PIT with CLIPSep, similar to how they were done in Sound of Pixels using Mechanical Turk?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster