OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup
Authors: Xize Cheng
Siqi Zheng
Zehan Wang
Minghui Fang
Ziang Zhang
Rongjie Huang
Shengpeng Ji
Jialong Zuo
Tao Jin
Zhou Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: sound separation
composed query
negative query
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: OmniSep enables sound separation using multi-modal composed queries.

Abstract: Query-based sound separation (QSS) effectively isolate sound signals that match the content of a given query, enhancing the understanding of audio data. However, most existing QSS methods rely on a single modality for separation, lacking the ability to fully leverage homologous but heterogeneous information across multiple modalities for the same sound signal. To address this limitation, we introduce Omni-modal Sound Separation (**OmniSep**), a novel framework capable of isolating clean soundtracks based on omni-modal queries, encompassing both single-modal and multi-modal composed queries. Specifically, we introduce the **Query-Mixup** strategy, which blends query features from different modalities during training. This enables OmniSep to optimize multiple modalities concurrently, effectively bringing all modalities under a unified framework for sound separation. We further enhance this flexibility by allowing queries to influence sound separation positively or negatively, facilitating the retention or removal of specific sounds as desired. Finally, OmniSep employs a retrieval-augmented approach known as **Query-Aug**, which enables open-vocabulary sound separation. Experimental evaluations on MUSIC, VGGSOUND-CLEAN+, and MUSIC-CLEAN+ datasets demonstrate effectiveness of OmniSep, achieving state-of-the-art performance in text-, image-, and audio-queried sound separation tasks. For samples and further information, please visit the demo page at \url{https://omnisep.github.io/}.

Published as a conference paper at ICLR 2025
OMNISEP: UNIFIED OMNI-MODALITY SOUND SEPARA- TION WITH QUERY-MIXUP
Xize Cheng'
Siqi Zheng" Zehan Wang'
Minghui Fang'
Ziang Zhang' Zhou Zhao !*
Rongjie Huang' Shengpeng Ji Jialong Zuo' Tao Jin'
'Zhejiang University Alibaba Group
chengxizeézju.ec edu.
ABSTRACT
Query-based sound separation (QSS) effectively isolate sound signals that match the content of a given query. enhancing the understanding of audio data. However, most existing QSS methods rely on a single modality for separation, lacking the ability to fully leverage homologous but heterogeneous information across multiple modalities for the same sound signal. To address this limitation, we introduce Omni-modes Sound Separation (OmniSep), a novel framework capable of isolating clean soundtracks based on omni-modal queries, encompassing both single-modal and multi-modal composed queries. Specifically, we introduce the Query rareiiy-iii xup strategy, which blends query features from different modalities during training. This enables OmniSep to optimize multiple modalities concurrently, effectively bringing all modalities under a unified framework for sound separation. We further enhance this flexibility by allowing queries to influence sound separation positively or negatively, facilitating the retention or removal of specific sounds as desired. Finally, OmniSep employs a retrieval-ane -aggmenned approach known as Query-Aug, which enables vocabuocyyl sound separation. Experimental evaluations on MUSIC, BEESSUNDD-CLEAN
and MUSIC-CLEAN: datasets demonstrate effectiveness of OmniSep, achieving state-of-the-an performance in text-, image-, and audio-queried sound separation tasks. For samples and further information, please visit the demo page at wtlhttps://oom htopisse/////..p..//mm/m atitthibioo/
INTRODUCTION
The development of sound separation (Kavalerov et al., 2019: Wisdom et al. 2020) has significantly advanced the ability to isolate and analyze specific sound signals from complex audio mixtures. Early sound separation methods focused on isolating specific audio signals like vocals, drums, and bass (Défossez et al.) [2199] Zeghidour & Grangier [202]] [Wang et al. (0236). Later, by employing a variety of queries to guide sound separation, Query-Based Sound Separation (QSS) effectively isolates sound signals that align with the given query, thereby enhancing the semantic understanding and interpretability of audio. Researchers delved into using natural language queries to extract semantically consistent audio tracks from audio sources, known as Text-Queried Sound Separation
(TQSS) (Ochia et al. 2020 Kong et al. 2020 Liu et al. 2022). However, the limitations of textual descriptions conveying nuanced scene information spurred further exploration into leveraging visual content as queries for extracting object-emitted sounds from images, termed Ietage-Queried Sound Separation (IQSS) (Tzinis et al., 2020). Dong et al. 2022 Chen et al.) 2023). Additionally, some researchers also attempted to employ audio references queries for extracting similar audio tracks, denoted as audio-Queried Sound Separation (AQSS) (Lee et al. 2019 Chen et al. 2022).
Despite these advances, still face several challenges to scale up audio data with sound separation: (I) Lack of a unified model to handle composed queries from multiple modalities. Current sound separation methods rely solely on modll queries, limiting their effectiveness in accurately ex- pressing the target sound signal. To overcome this limitation, we need a model that can accommodate multi-modal composed queries, thereby improving its ability capture and express the nuances of the desired sound signal. (2) Limited sound manipulation flexibility. Current methods are limited
Cooresponding author.

---

Published as a conference paper at ICLR 2025
to preserving specific sound signals based queries but unable to filter out specific sounds based on information corresponding to undesired sounds. (3) Incapable of handling open-vocabulary queries. Most of the existing audio-text datasets (Chen et al., 2020 Gemmeke et al., 2017) only offer predefined, limited class labels, rendering most works unfeasible to employ unrestricted text descriptions for sound separation (Liu al. 2023).
To address these challenges, we introduce an omni-modal sound separation model, OmniSep. to concurrently leverage omni-modal information. Specifically, we propose a Query -errry--xuu mix up query features from different modalities, enabling OmniSep to optimize each modality concurrently and achieve unified sound separation model capable of handling composed queries from diverse modalities. Expanding on this feature, we introduce the negative query, which identifies undesired sound information to eliminate specific sounds and enhance the flexibility of sound separation. Moreover, the reliance on predefined class labels in datasets confines current text-queried sound separation to predetermined categories, restricting the application of unrestricted text beyond the designated domain. To overcome this limitation, propose Query-Aue method inspired by Lewis et al. (2020). This method retrieves the most similar in-domain class queries from the query set based similarity reference query, facilitating open- open-vocabulary sound separation.
-uggeeted
Our experimental results MUSIC (hao et al. 2018 VGGSOUND-cleum (Dong et al. 2022). and MUSIC-clean+ (Dong et al. 2022) datasets demonstrate the superior sound separation performance of our OmniSep across Text Query Sound Separation (TQSS), Image Query Sound Separation (IQSS), and Audio Query Sound Separation (AQSS) tasks, solidifying its position as an omni- modal sound separation model. OmniSep+ Neg Query introduces information corresponding to unnecessary sounds, achieves the elimination of specific sounds, and enhances the flexibility and performance. Additionally, by adopting the Query-Aug strategy, the model's ''s robustness to out-of- domain unrestricted text is improved, achieving good separation of open vocabularies. The code and models will be released, and the main contributions of this paper are as follows:
We propose OmmiSep, an omni-modal sound separation model that can separate sound based on queries of arbitrary modality, including single-modal queries such text, images, and audio, as well as multi-modal composed queries.
We introduce the negative query, leveraging undesired sound information information to filter out specific sound signals and further enhancing the flexibility of the sound separation model.
We introduce Ouery-AAg,, a retrieval augmented method, to achieve open-vocabulary sound separation, allowing querying with unrestricted natural language descriptions.
Our experiments demonstrate that the OmniSep model achieves state-of-the-an sound separation performance across TQSS, IQSS, and AQSS.
2
RELATED WORKS
2.1
UNIVERSAL SOUND SEPARATION
The universal sound separation (Kavalerov al.) 2019 Liu et al.
2022 Pons al. 2024) aims
to extract distinct audio tracks from mixed audio, a critical task for audio understanding, Initially,
research efforts were concentrated on specific domains such speech (Wang & Chen 2018 Luo
et al.
2020
Li et al.] 2023 Pegg et al. 2023] Wang et al. 2023b Li et al. 2022) or music (Détossez
al.
2019 Manilow al. 2022] Rouard al. 2023 Luo & Yu 2023) separation. Later, Kavalerov
2017) to separate mixed audio into multiple sound tracks of unidentified categories, yet remained limited music, speech, and certain artificial sounds, lacking applicability to complex real-world sound understanding. To enhance comprehension of real-world audio, some researchers introduced extensive labeled audio datasets (Gemmeke et al. 20e7,Chen Chen al., 2020), gradually achieving universal sound separation.
et al.
2019) employed permutation invariant training (PIT) Yu et al.
Some (Ochiai et al. 2020 Kong et al.] 2020 Liu et al. 2022) suggest utilizing class labels as queries for corresponding sound separation. MixIT (Wisdom al. 2020) leverages pre-trained sound classification model to conduct unsupervised training on unlabeled audio data. CLIPSEP (Dong et al. 2022) integrates visual data to enhance text query sound separation model training.
2

---

Published as a conference paper at ICLR 2025
However, universal sound separation still faces two limitations: (1) Existing sound separation models are primarily trained on data with predefined class labels (Gemmeke et al. 2017 Chen et al. 2020), which restricts their ability to separate sounds based on out-of-domain text queries. (2) Previous separation methods have failed to utilize information related to interfering sounds, resulting in limited introduce the
model performance and flexibility. To overcome these limitations, strategy to enhance the robustness of sound separation models and enable open vocabulary sound separation. In addition, we propose the negative queries to handle undesired sound information, thereby enhancing the flexibility and performance of sound separation models.
2.2 QUERY-BASED SOUND SEPARATION
Query- based sound separation (QSS) (Ochiai et al. 2020 Kong et al., [020 Liu et al. 2022 Dong et al. 2022 extracts specific audio tracks from mixed audio that match given query. Previous research can be categorized into three main types: text query sound separation, image query sound separation, and audio query sound separation. Sound separation based on text queries (Ochiai et al.) 202g et al., [2000] Liu et al. 2022) extracts relevant audio content from mixed sounds using textual descriptions. These methods are suitable for scenarios where a single text label can accurately describe the target sound. However, in more complex scenes such as outdoor environments where multiple sound sources are mixed, single label may not suffice to describe the sound accurately. To address this, researchers (Tzinis et al. 2020 Dong et al. 2022 Chen et al. 2023) have utilized image as query to extract corresponding sounds. Moreover, certain sounds, like sound effects or abstract noises, are challenging to describe and may not be linked to visual content. Hence, researchers (Lee) ct al. 2019 Chen et al. 2022) have proposed audio queried methods to separate such abstract sounds.
However, existing sound separation models typically tailored for single-modal queries and encounter difficulties in separating sound with composed queries. To tackle this limitation, we introduce the query-mixup training strategy, which mixes up the features from different modalities during training. This allows OmniSep to unify the training objectives and optimize each modality concurrently, thereby facilitating sound separation based on omni-modal query.
2.3
MULTI-MODALITY REPRESENTATION LEARNING
In recent years, numerous studies (Girdaar al. 2023 Wang et al. 2023a 2024d b/c Zhang et al. 2025) have employed self--upervised learning to establish alignment across multiple modalities. Initially, CLIP (Radford al., (02I) was trained on extensive public data to construct alignment between images and texts. Subsequently, CLAP (Eiialdee et al. 2023) adopted a similar methodology to learn alignment between audio and text modalities. However, these works only focused on alignment between two modalities. On the other hand, Imagebind (Girdhar et al. 2023) aligns diverse modalities individually to unified image modality, thereby enabling the unified alignment of multiple modalities, including text, images, audio, depth, and more. Consequently, numerous studies (Cheng et al.| 22233a bl Huang et al., [003] Lei et al. 2024 Fu et al. (024) have emerged to investigate various multimodal tasks leveraging these unified representations. For instance, Xu ct al. 2023);; Liang et al. 2023)) achieve the open vocabulary segmentation with the joint text-image representations, while Han al. (2023) expands upon this groundwork to enhance multimodal comprehension. Meanwhile, Dong et al. (2022) delves into the domain of zero-shot -uided sound separation with the aid of image guidance.
Despite the rapid advancements in multi-modal representation learning, there is currently no dedicated multi-modal unified model tailored for sound separation tasks capable of handling text, audio, and image modal queries. In response, we introduce the first omni-modal sound separation model.
3 METHODS
3.1
OVERVIEW
The Omni-mode sound separation model (OmniSep) aims to perform sound separation based on different modal queries, where the query can originate from the audio modality (A), text modality (T), or visual modality (V). It separates noisy mixed audio into clean audio consistent with the query, described in Section 3.2 Additionally, in Section 3.3
we introduce a method to incorporate
3

---

Published as a conference paper at ICLR 2025
Query-Mixup W
Predicted Mask M
Child Singing
V/1
7
ImageBind
W
Linear
[Child Singing]
W
A
"Child Singing"
WBBC
N1
"Applause Sound"
[Applause] -il-
A2
STFT
Ground Truth Masks
Mixed Audio
A1
Audio U-NET
[Chiinsingginn]
d
Intermediate Masks M Qr
Qy
ON
Figure Illustration of Omni-modal Sound Separation (OmniSep). The OmniSep employs the curameter-frozen ImageBind model to extract features from diverse modal queries, denoted as Qr. Qv, and QA within the figure. Negative query N1, which aligns semantically with the interference audio A2, is adopted to aid sound separation during inference. Note that during testing time for IQSS, TQSS, and AQSS, only a single modal query is employed.
interference noise information into the model based on the composed query of the original query and the negative query to assist the model in sound separation. To further achieve open vocabulary sound separation, we propose Query-Aug in Section 3.4 which allows the model to querying with any unrestricted natural language description.
3.2 OMNISEP: OMNI-MODAL SOUND SEPARATION.
OmniSep adopts an architecture similar to CLIPSEP Dong et al. 2022). as illustrated in Figure I The Query-Net converts data from various modalities into corresponding query features Q E 18xx1022 where n denotes the number of audio samples in the mixed audio. Subsequently, we employ the Separate-Net transform the audio mixture Amix, consisting of audio sources, into clean sound Actean based on the query features Q. However, since CLIPSEP periodically switches the query modality during training, the training objective remains unfixed, posing challenges in achieving optimal performance across all modalities. We address this issue using the Query Mi xup strategy.
Query-Net To train a unified sound separation model adaptable to queries of various modalities, selected the imagebind modef"] pre-trained on multiple modalities, to extract query features: QA, Qv, QT = ImageBina((A (A, V,T" where QA, Qv, Qr E 1Tx1024 with the model parameters kept frozen during training. Subsequently,
adopt the Query-Mi xup strategy to mix up query features from different modalities, enabling OmniSep to optimize each modality concurrently and unify the training objectives with randomly sampled weight factors was w., and wy:
Wrn Q^ + W. Qv + w. QT WP + w. + we
Q
Was Wop we E [0, 1].
(I)
Separate-Net For mixed audio signal Amix composed of n audio sources {A1, A2. A,). each audio source A; is associated with corresponding video V, and textual query T,, forming n triplets {(A), Vi. Ti).
(An, V, }. The query Q; for sound separation is derived from the i-th triplet ((A,, V.,TT), T2). The mixed audio is first converted into the magnitude spectrum X E RCx 1ixFx xT using the Short-Time Fourier Transform (STFT), where C represents the number of channels, F is the frequency dimension and T' is the time dimension. For single-channel audio in this paper, C' = 1. Subsequently, we input the spectrogram into the audio U-Net (Ronneberger et al.) 2015 Jansson al. 2017) and obtain intermediate masks M [M1,"- M,,.. M₁ where M R XX
M, is the j-th intermediate mask and & 2 n. In this work. following the setting of CLIPSEP (Dong et al. 2022), & is set to 32. At the same time, each query feature Q obtained from the (A,, V,, T,)
https:
com http://cco...ccccooeeeeeeeeeeeee hwt.aaarrchhmagggiiiiin Imageting
4

---

Published as a conference paper at ICLR 2025
transformed into k-dimensional channel-wise weight 9; through the fully connected mapping layer: / -Lieer(Q: ( where 9 E TR Finally, the final predicted masks M, RCXFXT corresponding A, can be obtained with the channel-wise weight yi and the intermediate masks M:
M, a (wisj15, + b,).
(2)
where wy E R is learnable scale vector, by E R learnable bias, 9ij is the channel-wise weight of the j-th intermediate mask M,, and a-)) the sigmoid function. The training objective of the entire model is the sum of the weighted binary cross entropy (WBCE) losses for each query source:
L= Σ (BCE(M,, M. = Σx (-M, log M, (1 - M,) log(1 - M.).
(3)
==1
where M, is the ground truth mask for audio source A,. The predicted magnitude spectrum is first obtained by applying the predicted mask M, to the magnitude spectrum of the mixed audio. Finally, the noisy phase is combined with the predicted magnitude, and the inverse short-time Fourier transform (iSTFT) is applied convert the spectrogram back to a time-domain waveform, resulting in the separated audio signal. During inference, it's important to note that for the sound separation of single-mon modal queries, we input the single-modal query into the Sep-net without employing the Query-Mixup strategy. Furthermore, the subsequent subsections introduce two training- free strategies for query operations aimed at further enhancing the performance of the sound separation model.
3.3 NEGATIVE QUERY: ELIMINATE INTERFERENCE INFORMATION FROM THE ORIGINAL QUERY.
For noise information, we treat it as a novel form of query and extract it into the corresponding Query feature Qv, employing the same method as for other modal queries. With the aid of multi-modal pre-training (Girdhar et al. 2023) and the Query-Mi xup, features linked to different modal queries can be adeptly mapped into the multi-modal joint retrieval (Wang et al., 2024a). we employ the vector subtraction technique to climinate the information associated with QN from the original Query Q. However, it's worth noting that compared to retrieval tasks, sound separation tasks demand a more fine-graince manipulation of representations. Direct subtraction may lead to the loss of local information in Q, where any local information may be linked to sounds on certain frequency bands. Thus, when employing negative queries, we aim to maintain the stability of the frequency bands corresponding to non-key content by introducing a negative query weight a, which is expressed as:
space, facilitating sound separation. Drawing inspiration from
Q' (1 + aa -0QQQQQQ.
Subsequently, the final query feature Q' is inputted into the Separate- Net for sound separation. In particular, when a = 0, Q' Q, which means that no noise information is applicable during sound separation.
3.4 QUERY-AUG: TOWARDS THE QUERY OF UNRESTRICTED NATURE LANGUAGE DESCRIPTIONS.
Given that the majority texts in current audio-text paired datasets are accompanied by predefined category labels, the sound separation model encounters challenges when confronted with more authentic and unrestricted natural language descriptions during inference. To tackle this issue, we introduce the Query- Aug method, which facilitates open-vocabulary sound separation. In this method, a query set (Query-Set E RMM210 is constructed, comprising query features extracted using ImageBind for each class label, where M represents the number of class labels utilized during training. For an unrestricted natural language description, the query feature Qoug E R 1024 in the set that is most closely related to its corresponding ImageBind query feature Quts E 1021 is identified:
Qaug
arg max cos( Qutes Qm),
(5)
QmeQuery-SS
where cos represents the cosine similarity between the query features. Subsequently. Quey serves as the query for Separate-Net, which uses it to generate the corresponding separated audio signals.
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We conducted experiments on two datasets, VGGSOUND (Chen et al. 2020) and MUSIC (Zhao al., 2018), to evaluate the sound separation effect under different modal queries. VGGSOUND stands out as the largest audio-video consistent sound description dataset, boasting 550 hours of audio across 330 different sound categories, thereby representing various sound separation scenarios. Conversely, MUSIC focuses on instrument sounds, providing 10 hours of recordings featuring diverse instruments alongside corresponding performance videos, thus delineating a specific domain within sound separation. For comparison with previous studies, we conducted sound separation experiments following the CLIPSEP. This entailed training our model on the MUSIC dataset and evaluating its performance on itself. Additionally, evaluated its performance on the BEGSOUND-LLEAN and MUSIC-CLEAN+ datasets, which contain manually processed clean sound separation evaluation samples. For more details about the dataset, please refer to Appendix D
trained the model on the VGGSOUND dataset and
Following the experimental settings of CLIPSEP, a 4-second audio segment is randomly selected from the entire audio as the audio source. For queries, we extract the audio feature of the entire audio as the audio query; the average image feature of 4 frames with 1-second interval from the corresponding video serves as the image query, and the class label associated with the audio as the text query. Note that, during the inference, we extract S audio samples from the training set for each class as its audio query (where S = 5 for VGGSOUND, and S = for MUSIC) according to the setting of (Lee et al. 2019), for audio query sound separation (AQSS). More implementation details are shown in the Appendix A
4.2
MAIN RESULTS
Sound Separation for Omni-modal Queries. The OmniSep is designed to address sound separation tasks with diverse modal queries. Table J the SDR comparison of sound separation across various modal queries. Specifically, TQSS, and AQSS represent text-, image-, and audio-queried sound separation, respectively, while Non- Non-Queried Models denotes the method of implicitly learning sounds within the model. Across each task in TQSS, TQSS, and AQSS, our OmniSep achieves state-of-the-art sound separation performance, improving the Mean SDR by 0.43 to 4.36. This demonstrates that OmniSep can be queried with any single modality query. Furthermore, when queried with composed Omni-modal query, OmniSep can refer to information from different modalities to achieve more robust sound separation performance, with the Mean SDR of 7.46 on VGGSOUND-CLEANY
OmniSep:
Additionally, negative queries further enhance the sound separation performance by leveraging noise information during test-time. Compared to OmniSep without negative queries, the Mean SDR is further enhanced by 0.10 to 1.60. A detailed analysis of negative queries is provided in subsection
Query-Mixup: Enhancing Multi-modal Joint Training. To further validate the effectiveness of proposed method, we conducted detailed ablation experiments as shown in Table (1) Multi- modal joint training for different modal queried sound separation: The comparison between methods trained using modal data and multi-modal data highlights the adaptability of the latter in handling sound separation tasks with different modal queries. While the single-modality training method may yield better results on specific tasks (e.g., #1 trained with only text queries achieves a 0.73 higher Mean SDR on TQSS than #3), the #3 model, jointly trained with multi-modal queries, achieves an AVG SDR of 6.42, consistently outperforming #1 and #2 with improvements of 0.39 ~ 1.30. Furthermore, the introduction of audio modality data during training (#4) leads to further performance enhancement, with a 0.03 increase in performance from #3 to #4. (2) Query- mixup for unified omni-modal sound separation: To handle sound separation tasks with various modal queries, we employ Query-Min Mixup. In Experiment #5, we observed that the average SDR of 6.70 surpassed Experiment #4 of 6.45 by 0.25. Remarkably, this enables us to achieve comparable sound separation performance in TQSS as the text-specific training method (#1). This indicates that the Query-Mixx method circumvents the problem of fluctuating training objectives in traditional multimodal joint training methods, which often affects the performance of single-mode queried sound separation.

---

OmniSep(ours) OmniSep(ours)+NQ Published as a conference paper at ICLR 2025
Comparison of sound separation performance among different methods on MUSIC, MUSIC-CLEAN+, and YGGSOUND-CLEANS Experiments marked with are reproduced using CLIPSEP's dataset partitioning for comparison. All +NQ results are outcomes when a 0.5.
Table 1:
MUSIC
VGGSOUND-CLEAN+
MUSIC-CLEAN+
Method
Mean SDR
Med
Mean SDR
Med SDR
Mean SDR
Med SDR
Non- Queried Sound Separation
LabelSep
0.18
7.82
5.55+0.81
5.29
PIT Kavalerov et
2019
8.68++0.76
7.67
5.73++0.79
4.97
12.24
1.20
12.53
TDANet
10.31 0.799
10.18
6.43+00.2
5.83
13.09
1.11
12.74
Text- cuar-iueiied Sound Separation
BERTSep
5,09:10.00
5.49
4.67::0.44
4.41
CLIPSEP-NIT Dong ctal.
2022
3.05:10.73
3.26
10.27=1.04
10.02
CLIPSEP (Dong
8.36+1083
8.72
2.76+1.00
3.95
9.71#1.21
8.73
CLIPSEP- Text Dong etal.
2022
7.91
7.46
5,49:00.2
5,06
10.73=0.99
9.93
AudioSEP Liu 2023
9.82+t0.89
8.76
6.26+0087
5.57
11.23=0.99
10.28
OmniSep(ours)
10.65+1.007
9.97
6.70+H0.66
5.73
12.68
ON
1.09
9.97
7.57+
6.52
14.15 0.95
14.46
Image- Queereed Sound Separation
SOP Zhao al.
2018
6.5
6.22
2.99:10.8
3.89
11.44=1.18
11.18
CLIPSEP (ong et
2022
8.06:10.79
8.01
5,46:10.9
5.35
12.20=1.17
12.42
Query Chen 2023
10.54=00.2
9.73
OmniSep(ours)
1..97+1.03
10.21
6.69:#0.77
6.43
13.15=0.92
13.89
ONO
11.08++1.02
10.22
7.68+0..9
6.60
13.78=1.02
13.99
Madio-Queriin Sound Separation
AQSS
2
6.43++0.92
5.73
5.34
5.30
8.56++0.5
7.82
OmniSeptours)
10.26+11113
10.06
7.12+00.5
5.45
12.92-0.89
14.04
ONn
10.40
10.14
7.22:10.68
5.07
13.43
14.14
Composed Onni-Modal Queried Sound Separation
11.03
10.21
7.46:10.65
6.32
13.49 0.94
14.04
11.16:11.04
10.33
8.00
6.43
13.82 0.98
14.07
Table 2: Comparison of SDR values on TQSS and IQSS among sound separation models trained with diverse modality data (Text, Image and Audio) and training methods (MixUP). AVG SDR represents the average SDR across different sound separation models queried with and image.
TQSS
IQSS
ID Text Image Audio MixUP
AVG SDR
Mean SDR
Med SDR
Mean SDR
Med SDR
6.70=0.68
5.81
3.53+:0.49
2.86
5.129
#2
5.72++0.83
5.41
6.33
5.74
6.03
#3
6.33
5.83
6.51+:0.72
5.77
6.42++0.72
#4
6.37
5.68
6.53
6.33
6.45100.8
#5
6.70-10.66
6.22
6.69110.7
6.43
6.70
4.3
NEGATIVE QUERY: ENHANCING SOUND SEPARATION WITH NOISE INFORMATION.
To further enhance the efficacy of sound separation, we introduce negative queries to leverage noise information. Illustrated in Figure D are the variations of SDR with the negative query weight a on VGGSOUND-CLEAN and MUSIC-CLEAN+. We compare two distinct methods of negative query augmentation: firstly, the naive subtraction method (Q' = Q a Q,v). which employs negative representation processing akin to retrieval tasks; and secondly. (1+a)Q Q - aQN)). ), which additionally enhances the original query with proportionally weight. Please note that in both methods, when 0, the negative query not adopted.
proposed method (Q'
Here are the conclusions drawn from the figure: (1) Adaptability of Negative Query: Across all sound separation tasks (TQSS, IQSS, and AQSS), the negative query significantly enhances sound
7

---

Published as a conference paper at ICLR 2025
Text-Querled Sound Separation
ImageeQueried Sound Separation
Audio-Queries Sound Separation
8.0
8.0
7.5
7.5
7.37
7.57
7.37
7.32
7.69
7.22
7.07
12
7.24...
122
7.08
6.79
8
6.56.
6.0
6.69
5.5-
5.0
Mean SDR (naive)
Mean SDR (naive)
5.0
Mean SDR Inaive)
4.5
Mean SDR (ours)
Meen SDR lours)
Mean 5DR tours)
4.5
2-²
2-3
2
2-1
a neqative query
of negative cuery
of negative cuery
(a) VGGSOUND-CLEAN-
Text eext-Qeeieed Sound Separation
Image-Quertn Sound Separation
Audio- Madio--uerred Sound Separation
16
(naive) Mean SDR (ours)
(naive)
SDR (naive) Mean SDR tours)
15
15
Meen SDR (ours)
15
14
14.15
14.20
14:36
14-
24-
13.85
13469
19.78 13.81
13.79
13:3.15
2..92
13.36
13.43
13.34
13.06
17
12555
11
11
20
of negaative quary
of negative query
of negative query
(b) MUSIC-CLEAN+
Figure 2: The variation of SDR with the negative query weight on VGGSOUND-CLEAN: and MUSIC-CLEAN+. The x-axis represents the weight a of negative query, while the y-axis denotes the SDR. The shaded area indicates the standard deviation of the SDR. The dashed and solid lines respectively represent the results of naive subtraction (Q==QaaQ Q.v and ours (Q==(++a) (1ataa--aaCC
separation performance. For instance, on MUSIC-CLEAN, when a 1, the Mean SDR stands at 14.20, marking notable improvement of 1.65 from the 12.55 achieved without negative queries. (2) Proportional Weighting vs. Direct Subtraction: Comparative analysis reveals that our proposed proportional weighting method consistently outperforms traditional direct subtraction method. Across all tasks and varying alpha values, the solid line representing our method consistently surpasses the performance depicted by the dotted line, which represents direct subtraction. This demonstrates the efficacy of our proportional weighting approach in mitigating interference on the original query information when negative queries are employed.
(3) Robustness of Weight Selection: As training- free method, it offers considerable flcxibility in utilizing negative queries during the inference process, allowing for the use of any weight to negative query. However, this flexibility also presents challenges in determining the optimal weight a. The Mean SDR of naive direct subtraction shows significant fluctuations with changes in the weight a highlighting the considerable impact of on the naive method. Moreover, the optimal a value for achieving the best performance for naive direct subtraction varies across different tasks and datasets. Hence, finding a universally applicable fixed weight for the naive direct subtraction method is not feasible. On the contrary, even across various tasks and datasets, the mean SDR range resulting from the introduction of negative queries using our proposed proportional weighting method is more than 0.45 for different weights, thus obviating the need for excessive adjustment of the weight a. (For the TQSS task on VGGSOUND-CLEAT when a=2, the Mean SDR of our method remains 7.37, while the naive sound separation method has already decreased to 4.32), demonstrating significant robustness.
4.4 SOUND SEPARATION WITH UNRESTRICTED TEXTUAL DESCRIPTIONS.
As mentioned in Liu et al. 2023), previous methods (Ochiai et al. 2020 Veluri et al. 2023 for text- queried sound separation relied on predefined class labels as queries, constraining their adaptability to accommodate unrestricted natural language descriptions. To evaluate sound separation performance with such unrestricted descriptions as queries, we utilized GPT-3.5 to rewrite all predefined class labels into varied yet semantically consistent textual deseriptions. Further details provided in Appendix
In Table 3 conduct comparative analysis of the sound separation performance achieved by various methods using different text queries: (1) Predefined tags vs. Unrestricted descriptions: Training on predefined class labels limits the model's ability generalize to dut-oi- text. A
8

---

Published as a conference paper at ICLR 2025
significant drop in performance is observed when inferring out-of-domain text (Mean SDR drops from 5,49 in Experiment #6 to 3.53 in Experiment #8). (2) Multi-modal joint training vs. Single- modal training: Joint training with multi-modal queries (Experiment #9) helps fill the gap between the limited representations of text queries, enhancing generalization to out-of-domain text. (3) Open-vocabulary sound separation with Query-Aug: Despite the improvement in robustness achieved by multi-modal joint training, a performance gap remains due to cross-modal differences. The Mean SDR of 4.95 Experiment #10 is significantly lower than that of Experiment #6, which achieved a sound separation perfor- mance of 5.49 with in-domain class la- bels. Our proposed Query-Aus Aug method addresses this by selecting suitable domain text for text enhancement based on text similarity. The Query-Aus method significantly enhances the robustness of the sound separation model against out- of-domain text. When employing unre- stricted out-of-domain text descriptions for sound separation, the Mean SDR is further improved by 1.37 compared to OmniSep (Experiment #11 yields an SDR of 6.32, whereas Experiment #10 achieves 4.95). It's noteworthy that the performance of OmniSep+Querry
Table 3: The performance comparison of sound separa- tion between queries using predefined class labels and unrestricted textual descriptions on the VGGSOUND- CLEAN+ dataset.
ID
Method
Mean SDR
Med SDR
Query with predefined class labels.
#6 CLIPSEP-Text
5.49-10.82
5.06
#
OmniSep
6.7010.66
5.73
Query with unrestricted textual descriptions.
# 8
CLIPSEP-Text
3.53:0.52
2.91
#
+Query-Auu
5,2410079
4.87
10
OmniSep
4.95=10.81
4.45
(#11) with out-of- domain text is comparable to that achieved with predefined in-domain class labels (#6, #7). Specifically, while Experiment #6 achieves Mean SDR of 5.49, Experiment attains a Mean SDR of 6.32, which is superior performance despite being in more challenging condition, achieving open- open-vocabulary sound separation.
11
+Query-Aug
6.32:10.64
5.97
4.5 How QUERY-MIXE ENABLES OMNI-MODAL SOUND SEPARATION?
Figure 3 iillstrates the spatial distribution relationship of imagebind representations across different modalities, showing that features from various modalities cluster inde- pendently with significant cross moosl gaps, discussed in Liang et al. (2022). This makes it challenging for sin- gle sound separation model to process data from different modalities simultaneously. In previous methods, such CLIPSEP, the model constantly switches between text and image features, making it difficult simultaneously ac- commodate the distinct distribution requirements of both text and image representations. Consequently, the model fails to achieve optimal performance for both modalities the same time.
Our proposed OmniSep uses query-mixup strategy to combine features of the same semantics from different modalities, effectively bridging the gap between their dis- tributions with "mixed embeddings". can optimize the mixed embeddings, composed of different modal embed- dings, to accommodate both uni-modality queries and multi-modality composed queries. In omni-modal com- posed sound separation, queries from different modalities correspond to the same semantics and can complement each other, ensuring the model has more comprehensive understanding of the query information, thereby improving sound separation.
Figure 3: UMAP visualization of three different modal imagebind embeddings. The mix embedding is a weighted em- bedding of the three modalities.
In particular, functions such omni-modal composed sound separation and negative query rely on the additive and subtractive characteristics of embeddings between different modalities. The query embeddings required for composed query and negative-query all in the mixed embedding area between modalities. Therefore, the Query-Mixus xup strategy forms the basis of composed-quers and negative query capabilities, enhancing the model's controllability.

---

Published as a conference paper at ICLR 2025
Table 4: Qualitative results of sound separation using different modal queries. Interference and Target denote the reconstructed signals of interference audio and ground truth audio using the ground truth ideal binary masks, respectively. All spectra are presented on logarithmic frequency scale.
Query
Mixture
Interference
Target
Prediction
Child Singing
4.6 QUALITATIVE ANALYSIS
To demonstrate the effectiveness of OmniSep, we conduct qualitative analysis of sound separation with various modal queries, as depicted Table 4 The results showcase our OmniSep success- fully separate audio signals corresponding to the queries across TQSS, IQSS, and AQSS tasks. Additionally, in Appendix C.I we present qualitative results of sound separation with and without negative queries in TQSS. It's noteworthy that in sample of child singing audio mixed with child crying noise interference, traces of the crying persist in the separation results, posing challenges to effective separation due to the association of both sounds with the child. However, results obtained with negative queries demonstrate that OnniSep+Negative Query can effectively eliminate noise by leveraging the undesired sound information. The introduction of negative query information significantly enhances the removal of interfering sound signals and improves the flexibility of the sound separation model. addition, as demonstrated in the comparison of sound separation using unrestricted natural language queries in Appendix C.2, Query-Aug enhances the model's ability to generalize to out-of-domain unrestricted descriptions, thereby achieving open- vocabulary sound separation. Please visit the demo page at /itpttss///.. / / sep. github. 1o/ to see more sound separation results and learn more about OmniSep.
CONCLUSION
Researchers have adopted sound separation to scaling up audio datasets, but existing sound separation methods are limited to single-modal queries and have certain limitations. In this study, we introduce the multi-modal query mixing strategy, Query seery-Mixup.. and propose the first Omni-Modal Sound Separation model (OmniSep). OmniSep can perform sound separation based on any modal query, including single-modal queries such as text, images, and audio, as well as multi-modal composed queries. Additionally, we propose a method to remove undesired sound information according to negative queries from the original query, thereby improving sound separation performance and model flexibility. To overcome the challenge of limited text labels, we introduce Query-Aug, achieving open- vecabuacayy sound separation and facilitating the of unrestricted natural language queries. Experimental results on different modal sound separation tasks (TQSS, IQSS, AQSS) demonstrate that our model achieves tate-of-the-art performance in omni-modal sound separation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by National Natural Science Foundation of China under Grant No. 62222211 and No.624B2128 Thank Ziyang Ma and Slytherin Wang for their insightful discussions and valuable suggestions regarding this project.
REPRODUCIBILITY STATEMENT
All our code, and model weights will be open-sourced. In Section we provide a detailed description of the OmniSep modules and training instructions. Section 4.1 and Appendix [ offer additional details on the training process.
REFERENCES
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
Jiaben Chen, Renrui Zhang, Dongze Lian, Jiaqi Yang. Ziyao Zeng, and Jianbo Shi. iquery: Instru- ments as queries for audio-visual sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14675-14686, 2023.
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrrcke and Shlomo Dubnov. Zero-shot audio source separation through query-based learning from weakly-labeled data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4441-4449, 2022.
Xize Cheng, Rongjie Huang. Linjun Li, Tao Jin, Zehan Wang, Aoxiong Yin, Minglei Li, Xinyu Duan, Zhou Zhao, et al. Transface: Unit-based audio-visual speech synthesizer for talking head translation. arXiv preprint ar3iv: 12. 151197 2023a.
Xize Cheng, Tao Jin, Rongjie Huang. Linjun Li, Wang Lin, Zehan Wang, Ye Wang, Huadai Liu, Aoxiong Yin, and Zhou Zhao Mixspeech: Cross-modalit self-learning with audio-visual stream mixup for visual speech translation and recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 35735-15743 2023b.
Xize Cheng, Tao Jin, Linjun Li, Wang Lin, Xinyu Duan, and Zhou Zhao. Opensr: Open-modality speech recognition via maintaining multi-modality alignment. arXiv preprint 2023c.
Xize Cheng, Ziang Zhang, Zehan Wang, Minghui Fang. Rongjie Huang, Siqi Zheng, Ruofan Hu, Bai Jionghao, Tao Jin, and Zhou Zhao. Avset-10m: An open large-scale audio-visual dataset with high correspondence. 2024.
Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Demucs: Deep extractor for music sources with extra unlabeled data remixed. arXiv preprint arXiv: 19Xix1900.01172 2019.
Hao-Wen Dong, Naoya Takahashi, Yuki Mitsufuji, Julian McAuley, and Taylor Berg-Kirkpatrick. Clipsep: Learning text-queried sound separation with noisy unlabeled videos. arXiv preprint arXiv:2212.07065. 2022.
Benjamin Elizalde, Soham Deshmukh, Mahmoud AI Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Dongjie Fu, Xize Cheng, Xiaoda Yang, Wang Hanting, Zhou Zhao, and Tao Jin. Boosting speech recognition robustness modality-distoriin with contrast-augmented prompts. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1933-3847,, 2024.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and hubenel dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
11

---

Published as a conference paper at ICLR 2025
Rohit Girdhar, Alaacldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Haonan Han, Xiangzuo Wu, Huan Liao, Zumnan Xu, Zhongyuan Hu, Ronghui Li, Yachao Zhang, and Xiu Li. Atom: Aligning text-to motion model at event-level with gpt-4vision reward, 2024a. URL https: http///xiiiioorraa//////1111/ 18654
Haonan Han, Rui Yang, Huan Liao, Jiankai Xing, Zunnan Xu, Xiaoming Yu, Junwei Zha, Xiu Li, and Wanhua Li. Reparo: Compositional 3d assets generation with differentiable 3d layout alignment,
2024b. URL
abss/445555 18525
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llmg Mudti-modality instruction tuning. arXiv preprint Nat.23309003005. 2023.
Rongjie Huang, Huadai Liu, Xize Cheng, Yi Ren, Linjun Li, Zhenhui Ye, Jinzheng He, Lichao Zhang, Jinglin Liu, Xiang Yin, et al. Av-transpeech: Audio-visual robust seeech-to-speech translation. arXiv preprini arXiv:2305. /5403, 2023.
Andreas Jansson, Eric Humphrey, Nicola Montecchio, Rachel Bittner, Aparna Kumar, and Tillman Weyde. Singing voice separation with deep u-net convolutional networks. 2017.
Iyya Kavalenov, Scott Wisdom, Hakan Erdogan, Brian Patton, Kevin Wilson, Jonathan Le Roux, and John R Hershey. Universal sound separation. In 20/9 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 175-179. IEEE, 2019.
Qiuqiang Kong, Yuxuan Wang, Xuchen Song. Yin Cao, Wenwu Wang, and Mark D Plumbley. Source separation with weakly labelled data: An approach to computational auditory scene analysis. In ICASSP 2020-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 101-105. IEBE, 2020.
Anurag Kumar, Andrew Perrault, and Donald S Williamson. Using rlhf to align speech enhancement approaches to mean-opinion quality scores, arXiv preprint arXiv:2410.1311 13182, 2024.
Jie Hwan Lee, Hyeong-Seok Choi, and Kyogu Lee. Audio query-based music source separation. arXiv preprini arXiv: /908:99080000 066593, 2019.
Songju Lei, Xize Cheng, Mengjiao Lyu, Jianqiao Hu, Jintao Tan, Runlin Liu, Lingyu Xiong, Tao Jin, Xiandong Li, and Zhou Zhao. Uni-dubbing: Zero-shot speech synthesis from visual articulation. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10082-10099, 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval- genera- tion for mnowledge-intenstt nlp tasks. Advances in Neural Information Processing Systems, 33: 2020.
Kai Li. Runxuan Yang, and Xiaolin Hu. An efficient encoder-decoder architecture with top-down attention for speech separation. arXiv preprint arXiv:2209. (at::22299.122000 15000, 2022.
Xiang Li, Yiwen Wang, Yifan Sun, Xihong Wu, and Jing Chen. Pgss: pitch-guided speech separation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 13130-13138, 2023.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulee semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612-17625 2022.
12

---

Published as a conference paper at ICLR 2025
Xubo Liu, Haohe Liu, Qiuqiang Kong, Xinhao Mei, Jinzheng Zhao, Qiushi Huang, Mark D Plumbley, and Wenwu Wang. Separate what you describe: Language-quena audio source separation. arXiv preprint arXiv:2203.15147, 2022.
Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D. Plumbley, and Wenwu Wang. Separate anything you describe, 2023.
Yi Luo and Jianwei Yu. Music source separation with band-split rnn. JEEE/ACM Trunsactions on Audio, Speech, and Language Processing, 2023.
Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path rn: efficient long sequence modeling for time- domain single-channel speech separation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 46-50. IEEE, 2020.
Ethan Manilow, Patrick O' Reilly, Prem Seetharaman, and Bryan Pardo. Source separation by steering pretrained music models. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 126-130. IEEE, 2022.
Tsubasa Ochiai, Marc Delcroix, Yuma Koizumi, Hiroaki Ito, Keisuke Kinoshita, and Shoko Araki. Listen what you want: Neural network-based universal sound selector. arXiv preprint arXiv:2006.05712, 2020.
Samuel Pegg, Kai Li, and Xiaolin Hu. Rtfs-net: Recurrent free-ery modelling for efficient audio-visual speech separation. arXiv preprint arXiv:230.117189, 2023.
Jordi Pons, Xiaoyu Liu, Santiago Pascual, and Joan Serrà. Gass: Generalizing audio source separation with large-scale data. In /CASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 546-550. IEEE, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention -MICCA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234 241. Springer, 2015.
Simon Rouard, Francisco Massa, and Alexandre Défossez. Hybrid transformers for music source separation. In ICASSP 2023-2023 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Fabian-Robert Stöter, Antoine Liutkus, and Nobutaka Ito. The 2018 signal separation evaluation campaign. In Latent Variable Analysis and Signal Separation: 14th International Conference, LVA/ICA 2018, Guildford, UK, July 2-5, 2018, Proceedings 14, pp. 295-305. Springer, 2018.
Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel PW Ellis, and John R Hershey. Into the wild with audioscope: Unsupervised audio-visual separation of on-screen sounds. arXiv preprint arXiv:201 143, 2020.
Bandhav Veluri, Justin Chan, Malek Itani, Tuochao Chen, Takuya Yoshioka, and Shyamnath Gol- lakota. Real-time target sound extraction. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEBE, 2023.
DeLiang Wang and Jitong Chen. Supervised speech separation based on deep learning: An overview. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 1702 1726. Oct 2018. doi: 10.1 209.tttsl..018.288.8 2842159
URL http:://da doi. org 10. 1109/tas wt..://///............. 2010
Haofan Wang, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. Instantstyle: Free lunch towards style-preserving in text-to-image generation. arXiv preprint arXiv:2404.02733. 2024a.
13

---

Published as a conference paper at ICLR 2025
Zehan Wang, Yang Zhao, Haifeng Huang, Jiageng Liu, Aoxiong Yin, Li Tang, Linjun Li, Yongqi Wang, Ziang Zhang, and Zhou Zhao. Connecting muultimodaa contrastive representations. Ad- vances in Neural Information Processing Systems, 06:22099-221111 2023a.
Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang, Luping Liu, Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, et al. Freebind: Free lunch in unified multimodal space via knowledge fusion. arXiv preprint arXiv:2405.0445 04883, 2024b.
Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang. Luping Liu, Zhenhui Ye, Haifeng Huang. Yang Zhao, Tao Jin, Peng Gao, et al. Molecule-space: Free lunch in unified multimodal space via knowledge fusion. arXiv e-prints, pp. arXiv-2405, 2024c.
Zehan Wang, Ziang Zhang, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Hengshuang Zhao, and Zhou Zhao. Omnibind: Large-scale omni multimodal representation via binding spaces. arXiv preprint arXiv:2407.11 1895, 2024d.
Zhong- Wang, Samuele Coreell, Shukjae Choi, Younglo Lee, Byeong- Yeol Kim, and Shinji Watanabe. Tf-gridnet: Making time-frequency domain models great again for monaural speaker separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023b.
Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron Weiss, Kevin Wilson, and John Hershey. Unsupervised sound separation using mixture invariant training. Advances in Neural Information Processing Systems, -388866-3877 2020.
Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open- vocabulary semantic segmentation models from natural language supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2935-2944 2023.
Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen. Permutation invariant training of deep models for speakcr-indepennent multi-talker speech separation. In 2017 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 241-245. IEEE, 2017.
Neil Zeghidour and David Grangier. Wavesplit: End-to-end speech separation by speaker clustering. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:2840-2849, 2021.
Ziang Zhang, Zehan Wang, Luping Liu, Rongjie Huang, Xize Cheng, Zhenhui Ye, Huadai Liu, Haifeng Huang. Yang Zhao, Tao Jin, al. Extending multi-modal contrastive representations. Advances in Neural Information Processing Systems, 37:91 11911880-9100. 2025.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pp. 570-586, 2018.
14

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
Same as the experimental setting Dong et al. (2022), for all audio samples, we conducted exper- iments samples of length 65535 (approximately 4 seconds) at sampling rate of 16 kHz. For spectrum computation, we employed a short-time Fourier transform (STFT) with a filter length of 1024, a hop length of 256, and a window size of 1024. All images were resized to 224 x 224 pixels. The audio model in this paper is a wildly used 7-layer U-Net network with l: = 32, generating 32 intermediate masks. All models were trained with batch size of 128, using the Adam optimizer with parameters 31 = 0.9, 82 = 0.999, and = 10-8 for 200,000 steps. Additionally, we employed warm-up and gradient clipping strategies. following Dong et (2022). We compute the signal-to- distortion ratio (SDR) using museval (Stöter et al. 2018). 8) All experiments were conducted on single A800 GPU.
B SAMPLE OF UNRESTRICTED NATURAL LANGUAGE DESCRIPTIONS.
We utilized ChatGPT 3.5 to transform each class label from VGGSOUND into natural language descriptions without constraints, ensuring semantic consistency. Table 5 displays the instruction prompt employed in ChatGPT. For further analysis of the Query-Aug, Table 6 presents numerous examples of these unrestricted descriptions.
Table 5: Instruction prompt when using chatgpt generate unlimited natural language descriptions.
User:
Could you please assist me in rephrasing these descriptions of audio content to convey the same meaning but with different wording? Of course! Please provide the descriptions you'd 'd like me to rephrase, and I'll help you with alternative wording.
Chatgpt:
User:
tapping guitar
Chatgpt:
Tapping on a guitar to produce rhythms dog barking
User:
Chatgpt:
A dog vocalizing with a bark.
Table 6: Samples of class labels along with corresponding unrestricted natural language descriptions.
Predefined Class Label
Unrestricted Nature Language Description
playing didgeridoo
Performing on a didgeridoo instrument.
golf driving
Swinging golf club on the course.
dog barking
A dog letting out barking noises.
playing timpani
Beating drums timpani performance.
magpie calling
The melodious call of magpie bird.
dog bow-wow
A dog letting out a woofing sound.
subway metro underground cat growling
The rumbling noise of a subway train passing.
A cat emitting a growling noise.
C MORE EXPERIMENTS
C.I QUALITATIVE ANALYSIS ON SOUND SEPARATION WITH NEGATIVE QUERY.
In Table 7 we provide comparison of the sound separation effect between employing negative query (OmniSep+Neg query) and not using it (OmniSep). We exclusively showcase TQSS as an illustrative example, acknowledging the consistent performance of negative query across different modal queries in sound separation tasks. As illustrated in the Mel spectrogram comparison of the separation results, integrating negative query significantly enhances the removal of interfering audio signals from the mixed audio, effectively isolating the content associated with the negative query, Additional examples and corresponding audios are available on the demo page. Comparing the raw files of the two methods, it's evident that using negative query greatly enhance the flexibility of the sound separation model, particularly when addressing challenges such as the inability to effectively differentiate between two sound signals using single text query.
15

---

Published as a conference paper at ICLR 2025
Table Qualitative comparison of sound separation models with and without negative queries. All OmniSep+Neeg Query experiments are obtaned with 1. We highlighted the enhanced separation effect achieved by employing negative queries with red boxes.
Mixture
Interference
Target
OmniSep
OmniSep+NNeg Query
Table 8: Qualitative comparison of sound separation models with and without negative queries. We
highlighted the enhanced separation effect achieved by employing negative queries with red boxes.
Mixture
Interference
Target
OmniSep
OmniSep+Query-Auug
C.2 QUALITATIVE ANALYSIS ON SOUND SEPARATION WITH QUERY-AUG.
In Table we display the sound separation outcomes for an unrestricted natural language description query. It's clear from the mel spectrogram that when the aueey-augmenttiion method is not employed, the model (OmniSep) faces difficulties in grasping the sound information linked to the query. This results reduced capability to effectively separate the sound signals associated with the unrestricted natural language description. Conversely, with the integration of the query- augyeettgmntttoonn method, the model "OmniSep+Query-iiee sound information content and removing interfering sound signals.
showcases enhanced effectiveness in segregating the corresponding
C.3 OPEN VOCABULARY SOUND SEPARATION ON DIVERSE EVALUATION SETS.
To further demonstrate the effectiveness of our proposed Query-Aug method for open-vocabulary sound separation, we integrated with AudioSep and evaluated its performance on the datasets you suggested. The results are presented in Table 9 It is important to note that AudioSet and VGGSOUND use class labels as queries, which are in-domain queries already encountered during training. Consequently, when applying Query-Aug to these datasets, the retrieved query remains identical to the original, resulting in no performance change. Therefore, we did not include evaluations on AudioSet and VGGSOUND. Across other datasets, including AudioCaps, MUSIC, ESC-50, and
16

---

Published as a conference paper at ICLR 2025
Table Performance comparison of open-vocabulary sound separation on diverse evaluation sets. For fair evaluation, we compare the results of AudioSep and AudioSep Query-Aug.
AudioCaps
Music
ESC-50
Clotho
Method
SDRi
SI-SDR
SDRi
SI-SDR
SDRi
SI-SDR
SDRi
SI-SDR
AudioSep
7.68
6.45
9.75
8.45
10.24
9.16
6.51
4.84
+Query-Aug(ours)
8.22
7.19
10.72
9.71
10.72
9.84
7.20
5.80
Clotho, Query-Aug led to significant performance improvements, demonstrating its effectiveness in enhancing upen-vocabulat capabilities.
C.4 THE ABLATION PERFORMANCE ON AQSS
To provide more comprehensive comparison, we present the omitted AQSS results in Table IO which were excluded from Table due to space constraints. These results offer additional insights into the system's 's performance under different modality settings and further validate the contributions of multimodal integration to QSS tasks.
Table 10: Comparison of SDR values on AQSS among sound separation models trained with diverse modality data (Text, Image and Audio) and training methods (MixUP).
Audio
Text
Image
MixUP
Mean SDR (AQSS)
Med SDR (AQSS)
5.79 + 0.78
5.19
6.67 + 0.71
5.38
/
/
6.97 + 0.66
5.40
7.12 + 0.65
5.45
C.5 IMAGEBIND ABLATION EXPERIMENTS
To evaluate the contribution ImageBind pretraining to the performance of our model, we conducted ablation experiments focused on the role of ImageBind. As shown in Table ??Our experiments indicate that without a pretrained model, performance drops significantly, as the model cannot effectively extract features align representations across modalities. While fine-tuning ImageBind (E2) slightly improves in-domain performance, it hinders generalization to out-of-domain data, resulting in a 1.94 SDR drop on the MUSIC test set. Freezing ImageBind (E3) and using linear layer provides a balance between performance and generalization.
Table 11: ImageBind Ablation Experiments.
VGGSOUND-CLEAN
MUSIC SDR(TQSS)
ID Pretrained
Imagebind
SDR(TQSS)
SDR(QSS)
SDR(AQSS)
E1
X
tuning
2.33
2.31
1.94
E2
tuning
6.81
6.73
7.22
4.76
E3
freeze
6.70
6.69
7.12
6.70
C.6 ABLATION ON NEGATIVE QUERIES
When constructing the final query features with negative queries, it is crucial not only ensure stability but also to remove the information corresponding to the negative query from the original query. If we use (1 Q + aQn both the negative query and the positive query content retained, which contradicts the goal of using negative queries. On the other hand, directly subtracting ( Q aQv can lead to instability and difficulty in choosing the coefficient o, discussed in
17

---

Published as a conference paper at ICLR 2025
Section 4.2 and Figure of the paper. As shown in Table 12 provide the results for three methods with a = 0.5 for more intuitive comparison:
Table 12: Performance Comparison of Different Negative Query Embedding Strategies
Query Embedding
SDR(IQSS) SDR(TQSS)
SDR(AQSS)
- ( + aQN
3.96
3.23
3.77
Q-aQx (1+a)Q - -QQN
6.77
6.92
6.65
7.57
7.68
7.22
D DATASETS
VGGSOUND (Chen et al., 2020) VGGSOUND is large-scale audio-visual dataset containing over 200,000 videos spanning 309 diverse sound categories, designed to support research in audio event classification, sound separation, and other multi-modal learning tasks.
MUSIC (Zhao et al. 2018) The MUSIC dataset a collection of 536 video recordings of people playing musical instrument of instrument classes.
Music-Cleant (Dong et al. 2022)
A refined subset of the music audio data, focusing on clean and distinct tracks with minimal noise or overlapping sounds. To prevent unintended overlaps of target sound types between the MUSIC and VGGSound datasets caused by label mismatches, all videos of musical instrument performances are excluded from the VGGSound dataset in this setting.
VGGSound-Cleant (Dong et al. 2022) A refined subset of the VGGSound dataset, comprising audio clips with reduced background noise and more distinct sound events. Dong et al. (2022) manually selects 100 clean samples containing clear and distinct target sounds from the VGGSound test set, referred to (GGSound-Cleam
E LIMITATIONS
This study still lacks more generalized validation. For the sake of comparison with prior models, we primarily conduct experiments the VGGSOUND and MUSIC datasets. While these datasets well-suited for omni-modal sound separation tasks, and VGGSOUND already encompasses over 300 sound events across various categories, they still fall short of representing all real-world audio events, particularly some abstract sound events like pop music, which are not adequately represented the dataset. Moving forward, we aim to construct a more extensive dataset (Cheng et al. 2024) to address this limitation. We also plan to use RLHF (Han et al. 2024b a Kumar et al. 2024) to optimize sound separation and improve its generalization.
F ETHICAL DISCUSSION
The sound separation method proposed in this paper enables the separation of target queries according to user preferences, but may be susceptible to misuse, such separating background music (BGM) and vocals from copyrighted songs, extracting dialogue and various sound effects from copyrighted movies, and clear copyright ownership of various audio and video products, the potential social impact is not expected to be significant.
However, given the long history of development in this field and the relatively
18


Reviews:
[{"review_id": "UjEMNhOqXg", "reviewer": "Reviewer_7ccM", "summary": "This paper presents OmniSep, a framework for omni-modal sound separation, which supports sound isolation using queries from multiple modalities, such as text, image, and audio, either independently or in combination. The key method of omni-modal source separation is the introduction of Query-Mixup, a strategy that mixes query features from different modalities, using a pre-trained ImageBind. OmniSep further enables open-vocabulary sound separation with Query-Aug, a retrieval-augmented method that enhances adaptability, particularly for text-based queries. Experimental evaluations on MUSIC, VGGSOUND-CLEAN+, and MUSIC-CLEAN+ datasets showcase OmniSep’s SOTA performance across various modality-based separation tasks.", "questions": "- What happens when the query is some class that’s not present inside the input mixture audio?\n- Table 2: why no results on AQSS?\n- Query-Aug Adaptability:\n    - Is the Query-Aug method adaptable only to text queries $Q_T$, or does it extend to other modalities as well?\n    - Additionally, how will the model handle text prompts containing multiple sound sources (e.g., “the sound of a baby with her parent’s soothing voice”)? Is the model trained to handle multi-source separation?\n- Number of Audio Sources in Mixtures: For AQSS VGGSOUND, does $\\mathcal{S}=5$ mean $A_\\text{mix}=\\sum_{n=1}^6A_n$? then for all other setups, is $A_\\text{mix}=A_1+A_2$?\n- Did the authors use the pre-trained ImageBind weights from the official repository (https://github.com/facebookresearch/ImageBind), or did they train ImageBind from scratch? They should mention the repository if pre-trained weights were used. If trained from scratch, they should include the details of the pre-training specifications.\n- How does OmniSep respond when a query references a class that is not present within the input audio mixture?\n- Why are there no AQSS results reported in Table 2?\n\nMinor comments\n- typo @Figure 1: “…, donated as $\\text{Q}_T$, …”\n- Figure 1: inside the Query-Mixup block, the weight factors are denoted as $W$, where it should be $w$\n- grammar @line 217: “The training …”\n- Table 1: should note the source of VGGSOUND-CLEAN+ and MUSIC-CLEAN+ subsets. Also provide a couple of sentences for details of these subsets.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents an omni-modal sound separation approach, allowing users to conduct sound separation tasks using queries across various modalities, including text, image, and audio, both independently and jointly. Additionally, the authors conducted a comprehensive evaluation across several benchmarks. Further, the inclusion of extensive ablation studies provides insight into the contributions of each component: impact of Query-Mixup, negative query weighting, long text description-queried separation, and analysis on query embeddings.", "weaknesses": "While the paper presents a solid framework, there are some details missing that make it difficult to give a firm acceptance (please answer Questions). \n\nThe use of full-length video as a query, which includes the target segment, raises questions about the potential for information leakage. It’s unclear if the QueryNet architecture ensures a bottleneck to prevent the model from “cheating” by directly accessing target audio features. To better demonstrate the model’s robustness, an alternative setup might involve using a different video as the query that shares the same class identity as the input audio.\n\nAdditionally, the comparison in Section 4.3 between the proposed weighting and a naive weighting approach on $Q_N$ may not be particularly insightful, as both methods use the same model. The proposed approach always has a weight difference of 1 ($1+\\alpha$ vs. $\\alpha$), while the comparison (naive) approach reduces the weight gap between $Q$ and $Q_N$, hence the results in Figure 2 are somewhat predictable. However, the insights provided by varying $\\alpha$ are valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ErzcX4QUg", "reviewer": "Reviewer_PPUR", "summary": "This paper presents OmniSep, a novel unified framework for query-based sound separation that accommodates multiple query modalities (text, image, and audio) within a single model. The authors introduce three key technical contributions: Query-Mixup for simultaneous multi-modal query processing, a negative query mechanism for unwanted sound suppression, and Query-Aug for handling natural language descriptions beyond predefined class labels. The model's architecture represents an advance over previous approaches, which were typically constrained to single-modality queries.\n\nExperimental validation on several datasets demonstrates OmniSep's performance compared to existing methods across all query modalities. The model exhibits robust separation capabilities in complex, multi-source scenarios and achieves state-of-the-art results on MUSIC, VGGSOUND-CLEAN+, and MUSIC-CLEAN+ benchmarks. OmniSep's multi-modal query capability enables enhanced separation performance through the simultaneous application of different query types, such as combining textual and visual queries.", "questions": "1. Separate-Net only converts to magnitude spectrum X, ignoring phase. Modern time-frequency methods (RTFS-Net, TF-GridNet) use retain both magnitude and phase information via concatenated real/imaginary STFT outputs. Some methods additionally concatenate the magnitude to create a three channel representation. Have you tried these approaches? \n\n2. Why keep ImageBind parameters frozen? What if it was trained from scratch with the model? What if it was initialized and fine tuned with the model using a lower learning rate? \n\n3. For the Negative Query, why (1+alpha)Q-alphaQ_N instead of (1-alpha)Q+alphaQ_N, Q-alphaQ_N or a scale-and-shift approach such as in stable diffusion - please add some justification in this section or some experimental evidence. \n\n4. And the problems raised in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The paper presents a clear and well-motivated problem statement, addressing three fundamental limitations in current sound separation approaches: the absence of unified multi-modal query handling, insufficient flexibility in sound manipulation (particularly for unwanted sound removal), and restricted vocabulary constraints that preclude natural language descriptions. The authors construct a compelling narrative throughout the introduction and literature review, effectively contextualizing their contributions within the field through comprehensive citations and thorough analysis of related work.\n\nThe methodology is generally well-documented to begin with. The experimental validation is particularly robust, encompassing diverse tasks and datasets that demonstrate the method's versatility. The authors provide extensive ablation studies that illuminate the model's internal mechanisms and justify some of the architectural choices. Their commitment to reproducibility through code release further strengthens the paper's contribution to the field.", "weaknesses": "There are several issues with the paper, which can broadly be classified into two areas.  Most of these issues can be fixed with better, scientific writing, and more explanation, rigour and experimentation. \n\nTechnical Clarity Issues: \n1. Significant documentation gaps in core variables and operations in Separate-Net: \n   - q_i and q_ij are poorly explained or completely undefined in Separate-Net \n   - Many variables are undefined and or with no dimensions specified. All variables should be clearly defined with dimensions given.  \n   - M(hat) is defined but never used. \n   - Mechanism of how masks are used/applied is not explained.\n   - Audio U-Net lacks both citation and architectural explanation.\n2. Query-Aug undefined components: \n   - Q_des: completely undefined without dimensions and no explanation of how these features are obtained (T5/Bert?).\n   - Query-Set: is undefined and dimensions not specified.\n   - Q_aug defined as argmax but its integration into model never explained.\n   - sim(.,.) is not a standard operator, it should be properly defined (cosine similarity?).\n\nExperimental Validation Problems: \n1. Table 1 uses a 2017 model called PIT for non-queried sound separation. State-of-the-art audio-only separation methods (i.e. TDANet, TF-GridNet) would serve stronger baselines (while these models are for speech separation, so is PIT. Additionally, PIT was proposed so solve the permutation problem, not as a strong speech separation model).  \n2. Table 2 does not show that the query-mixup method works, only that it scales across different modalities. Validating its effectiveness would mean comparing to other methods and/or different strategies. \n\nHowever, there are some other issues. While the method is interesting, the paper's novel work can be summarised as a weighted average of modalities, a linear layer and (1+alpha)Q-alphaQ_N for negative queries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UTBOV3s94S", "reviewer": "Reviewer_Mat5", "summary": "This paper proposes an audio source separation model that can be queried via audio, language and image. To construct such model, this paper proposes query mixup which blends query from different modalities during training and ensure queries from different modalities reside in the same semantic space. This paper also propose a negative query mechanism to enhance the flexibility of query in inference stage. Last, to enable open-vocabulary text query input to the model which trained on close-set class name, this paper propose a query-augmentation method at inference time which retrieves nearest class name. Experiment result shows the model outperforms existing model in all modalities on MUSIC, VGGSOUND-CLEAN+, and MUSIC-CLEAN+ datasets.", "questions": "The introduction of the Separate-Net is missing some details:\n1. Line 210: what is k and what does k corresponds to? \n2. Does the query conditioned to the U-Net? If not, why query is not conditioned to UNet? If possible, it would be helpful to visualize the qi in some examples. It feels odd to me that the query only results in a channel-wise weight.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. A novel model for source separation queried by audio, text and image. Propose a novel query-mixup method to enable such model.\n2. Propose novel negative query and query-aug method to improve performance and flexibility at inference time.\n3. State-of-the-art performance on separation tasks queried by all modalities", "weaknesses": "1. Presentation: Certain aspects of the paper’s presentation feel oversimplified, particularly in explaining key contributions and technical details. Please refer to the questions section for specific areas needing clarification.\n\n2. Contribution of Query-Aug: The significance of the query-augmentation (query-aug) contribution seems overstated. For example, lines 054–056 suggest that current systems cannot handle open-vocabulary queries. However, Liu et al. (2023) (\"Separate Anything You Describe\") demonstrates that open-vocabulary language queries are achievable using audio-text datasets, such as Clotho or AudioCaps. Furthermore, the model proposed in this paper should be capable of training on these audio-text datasets and potentially on datasets lacking complete audio, text, or image pairs. An expanded discussion comparing this work to Liu et al. (2023), along with a rationale for the choice of datasets, would strengthen the contribution.\n\n3. Experimental Results: The presentation of experimental results lacks some necessary detail:\n - What training settings were used for the models in Table 2, and how do these models (particularly model #5) correspond to those in Table 1?\n - For Table 4, a comparison of results from querying the same audio mixture across different modalities would provide valuable insights.\n\n4. Metrics for Comparison: In prior works cited in the paper, metrics such as SI-SDR and SDRi are commonly used for source separation tasks, as they better reflect model performance in recent studies. Including these metrics or providing justification for the metrics chosen would enhance the comparative analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "98nIHDtyqS", "reviewer": "Reviewer_nQk6", "summary": "This paper attempts to cover text-, image- and audio-queried sound separation all at once with one model by exploiting ImageBind as its encoder. Several training techniques are introduced and verified its effectiveness in terms of signal-to-noise ratio as well as spectral similarity, which demonstrates the superiority of OminiSep to other existing query-based sound separation methods", "questions": "The motivation of Query-mixup is clear. On the other hand, its effectiveness compared to finetuning of ImageBind is not clear. Why don't you simply finetune your encoder so that all the modalities align well each other to the inputs of your interests. It is also unclear if the performance improvement really comes from the proposed three methods because the encoders in CLIPSep and OmniSep are different. Table 3 is a nice way to prove Query-Aug is actually important to increase SDR. Not sure about other two. You could have tried the rest of two methods on CLIPSep and see the performance improvements of CLIPSep?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Three novel training techniques, Query-mixup, NQ, Query-Aug, are proposed and evaluated in the standardized experimental settings\n- Consistent improvements of median SDR in Table 2, which demonstrates the SOTA performance of OmniSep in the query-based sound separation field\n- Good visualization of ImageBind embeddings to show the clear motivation of the Query-mixup", "weaknesses": "- Inappropriate motivation: I felt puzzled when I was reading the introduction part because the original motivation of the paper was how we could increase the number of high-quality training dataset although this motivation was never addressed anywhere in the other sections of the paper. If the objective is like the above, why don't we simply compare your scheme to other data augmentation methods or artificial data creation methods other than sound separation? I agree that denoising with sound separation would serve as one of many methods to achieve the same goal. Then, I believe the author should prove that OmniSep is the one of the best methods to increase the number of data. That being said, while assessing the paper, I somehow felt that this was not your intention. Since the technical contributions of the paper are fair enough, I suggest rewriting the introduction part in a way like your motivation is to improve the quality or coverage of text-, image-, or audio-queried sound separation and NOT to use sound separation to increase the number of training datasets for other purposes. \n- As an expert in this field, I don't agree relying only on SDR to measure the performance of any sound separation methods. Rather than showing pairs of spectrums, I suggest conducting a listening test to doublecheck if the performance of OmniSep really dominates others and the difference is recognizable. If the goal of the paper is not to make separated data available to human, as you originally stated in the introduction part, i.e., data increase is the objective, I don't think having a subjective listening test enriches the content. Otherwise, I strongly recommend this because this is a common practice in the sound separation community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)