OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup

## Abstract

Abstract The scaling up has brought tremendous success in the fields of vision and language in recent years. When it comes to audio, however, researchers encounter a major challenge in scaling up the training data, as most natural audio contains diverse interfering signals. To address this limitation, we introduce Omni -modal Sound Sep aration ( OmniSep ), a novel framework capable of isolating clean soundtracks based on omni-modal queries, encompassing both single-modal and multi-modal composed queries. Specifically, we introduce the Query-Mixup strategy, which blends query features from different modalities during training. This enables OmniSep to optimize multiple modalities concurrently, effectively bringing all modalities under a unified framework for sound separation. We further enhance this flexibility by allowing queries to influence sound separation positively or negatively, facilitating the retention or removal of specific sounds as desired. Finally, OmniSep employs a retrieval-augmented approach known as Query-Aug , which enables open-vocabulary sound separation. Experimental evaluations on MUSIC, VGGSOUND-CLEAN+, and MUSIC-CLEAN+ datasets demonstrate effectiveness of OmniSep, achieving state-of-the-art performance in text-, image-, and audio-queried sound separation tasks. For samples and further information, please visit the demo page at https://omnisep.github.io/ .

## 1 Introduction

The scaling up (Hu et al., ; Kaplan et al., ) has brought great success to the generation and understanding tasks for images and text. However, in the case of audio modality, wild data typically comprises sound signals from multiple sources (Tzinis et al., ), making it challenging to serve as training data. This difficulty presents a significant obstacle to expanding audio datasets. In response, some researchers (Jiang et al., ; Copet et al., ) have turned to sound separation (Kavalerov et al., ; Wisdom et al., ) to expand audio datasets by separating complex real-world audio mixtures into clear and distinct tracks.
Early sound separation methods
focused on isolating specific audio signals like vocals, drums, and bass (Défossez et al., ; Zeghidour & Grangier, ; Wang et al., ). Later, researchers delved into using natural language queries to extract semantically consistent audio tracks from audio sources, known as Text-Queried Sound Separation (TQSS) (Ochiai et al., ; Kong et al., ; Liu et al., ). However, the limitations of textual descriptions in conveying nuanced scene information spurred further exploration into leveraging visual content as queries for extracting object-emitted sounds from images, termed Image-Queried Sound Separation (IQSS) (Tzinis et al., ; Dong et al., ; Chen et al., ). Additionally, some researchers also attempted to employ audio references as queries for extracting similar audio tracks, denoted as Audio-Queried Sound Separation (AQSS) (Lee et al., ; Chen et al., ).

Despite these advances, we still face several challenges to scale up audio data with sound separation:
(1) Lack of a unified model to handle composed queries from multiple modalities. Current sound separation methods rely solely on single-modal queries, limiting their effectiveness in accurately expressing the target sound signal. To overcome this limitation, we need a model that can accommodate multi-modal composed queries, thereby improving its ability to capture and express the nuances of the desired sound signal.
(2) Limited sound manipulation flexibility. Current methods are limited to preserving specific sound signals based on queries but are unable to filter out specific sounds based on information corresponding to undesired sounds.
(3) Incapable of handling open-vocabulary queries. Most of the existing audio-text datasets (Chen et al., ; Gemmeke et al., ) only offer predefined, limited class labels, rendering most works unfeasible to employ unrestricted text descriptions for sound separation (Liu et al., ).

To address these challenges, we introduce an omni-modal sound separation model, OmniSep, to concurrently leverage omni-modal information.
Specifically, we propose a Query-Mixup to mix up query features from different modalities, enabling OmniSep to optimize each modality concurrently and achieve a unified sound separation model capable of handling composed queries from diverse modalities.
Expanding on this feature, we introduce the negative query, which identifies undesired sound information to eliminate specific sounds and enhance the flexibility of sound separation. Moreover, the reliance on predefined class labels in datasets confines current text-queried sound separation to predetermined categories, restricting the application of unrestricted text beyond the designated domain.
To overcome this limitation, we propose Query-Aug, a retrieval-augmented method inspired by Lewis et al. (). This method retrieves the most similar in-domain class queries from the query set based on similarity as a reference query, facilitating open-vocabulary sound separation.

Our experimental results on MUSIC (Zhao et al., ), VGGSOUND-clean+ (Dong et al., ), and MUSIC-clean+ (Dong et al., ) datasets demonstrate the superior sound separation performance of our OmniSep across Text Query Sound Separation (TQSS), Image Query Sound Separation (IQSS), and Audio Query Sound Separation (AQSS) tasks, solidifying its position as an omni-modal sound separation model. OmniSep+Neg Query introduces information corresponding to unnecessary sounds, achieves the elimination of specific sounds, and enhances the flexibility and performance. Additionally, by adopting the Query-Aug strategy, the model’s robustness to out-of-domain unrestricted text is improved, achieving good separation of open vocabularies. The code and models will be released,
and the main contributions of this paper are as follows:

- •
We propose OmniSep, an omni-modal sound separation model that can separate sound based on queries of arbitrary modality, including single-modal queries such as text, images, and audio, as well as multi-modal composed queries.
- •
We introduce the negative query, leveraging undesired sound information information to filter out specific sound signals and further enhancing the flexibility of the sound separation model.
- •
We introduce Query-Aug, a retrieval augmented method, to achieve open-vocabulary sound separation, allowing querying with unrestricted natural language descriptions.
- •
Our experiments demonstrate that the OmniSep model achieves state-of-the-art sound separation performance across TQSS, IQSS, and AQSS.

## 2 Related Works

### 2.1 Universal Sound Separation

The universal sound separation (Kavalerov et al., ; Wisdom et al., ; Liu et al., ; Pons et al., ) aims to extract distinct audio tracks from mixed audio, a critical task for audio understanding. Initially, research efforts were concentrated on specific domains such as speech (Wang & Chen, ; Luo et al., ; Subakan et al., ; Zeghidour & Grangier, ; Wang et al., ; Li et al., ) or music (Défossez et al., ; Manilow et al., ; Rouard et al., ; Luo & Yu, ) separation. Later, Kavalerov et al. () employed permutation invariant training (PIT) (Yu et al., ) to separate mixed audio into multiple sound tracks of unidentified categories, yet remained limited to music, speech, and certain artificial sounds, lacking applicability to complex real-world sound understanding. To enhance comprehension of real-world audio, some researchers introduced extensive labeled audio datasets (Gemmeke et al., ; Chen et al., ), gradually achieving universal sound separation. Some (Ochiai et al., ; Kong et al., ; Liu et al., ) suggest utilizing class labels as queries for corresponding sound separation. MixIT (Wisdom et al., ) leverages a pre-trained sound classification model to conduct unsupervised training on unlabeled audio data. CLIPSEP (Dong et al., ) integrates visual data to enhance text query sound separation model training.

However, universal sound separation still faces two limitations: (1) Existing sound separation models are primarily trained on data with predefined class labels (Gemmeke et al., ; Chen et al., ), which restricts their ability to separate sounds based on out-of-domain text queries. (2) Previous separation methods have failed to utilize information related to interfering sounds, resulting in limited model performance and flexibility. To overcome these limitations, we introduce the Query-Aug strategy to enhance the robustness of sound separation models and enable open vocabulary sound separation. In addition, we propose the negative queries to handle undesired sound information, thereby enhancing the flexibility and performance of sound separation models.

### 2.2 Query-based Sound Separation

Query-based sound separation (QSS) (Ochiai et al., ; Kong et al., ; Liu et al., ; Dong et al., ) extracts specific audio tracks from mixed audio that match a given query. Previous research can be categorized into three main types: text query sound separation, image query sound separation, and audio query sound separation.
Sound separation based on text queries (Ochiai et al., ; Kong et al., ; Liu et al., ) extracts relevant audio content from mixed sounds using textual descriptions. These methods are suitable for scenarios where a single text label can accurately describe the target sound. However, in more complex scenes such as outdoor environments where multiple sound sources are mixed, a single label may not suffice to describe the sound accurately. To address this, researchers (Tzinis et al., ; Dong et al., ; Chen et al., ) have utilized image as query to extract corresponding sounds. Moreover, certain sounds, like sound effects or abstract noises, are challenging to describe and may not be linked to visual content. Hence, researchers (Lee et al., ; Chen et al., ) have proposed audio queried methods to separate such abstract sounds.

However, existing sound separation models are typically tailored for single-modal queries and encounter difficulties in separating sound with composed queries. To tackle this limitation, we introduce the query-mixup training strategy, which mixes up the features from different modalities during training.
This allows OmniSep to unify the training objectives and optimize each modality concurrently, thereby facilitating sound separation based on omni-modal query.

### 2.3 Multi-Modality Representation Learning

In recent years, numerous studies (Radford et al., ; Elizalde et al., ; Girdhar et al., ; Zhu et al., ) have employed self-supervised learning to establish alignment across multiple modalities. Initially, CLIP (Radford et al., ) was trained on extensive public data to construct alignment between images and texts. Subsequently, CLAP (Elizalde et al., ) adopted a similar methodology to learn alignment between audio and text modalities. However, these works only focused on alignment between two modalities.
On the other hand, Imagebind (Girdhar et al., ) aligns diverse modalities individually to a unified image modality, thereby enabling the unified alignment of multiple modalities, including text, images, audio, depth, and more. Consequently, numerous studies (Dong et al., ; Xu et al., ; Liang et al., ) have emerged to investigate various multimodal tasks leveraging these unified representations.
For instance, Xu et al. (); Liang et al. () achieve the open vocabulary segmentation with the joint text-image representations, while Han et al. () expands upon this groundwork to enhance multimodal comprehension. Meanwhile, Dong et al. () delves into the domain of zero-shot text-guided sound separation with the aid of image guidance.

Despite the rapid advancements in multi-modal representation learning, there is currently no dedicated multi-modal unified model tailored for sound separation tasks capable of handling text, audio, and image modal queries. In response, we introduce the first omni-modal sound separation model.

Figure: Figure 1: Illustration of Omni-modal Sound Separation (OmniSep). The OmniSep employs the parameter-frozen ImageBind model to extract features from diverse modal queries, donated as $\mathbf{Q}_{T}$, $\mathbf{Q}_{V}$, and $\mathbf{Q}_{A}$ within the figure. Negative query $N_{1}$, which aligns semantically with the interference audio $A_{2}$, is adopted to aid sound separation during inference. Note that during testing time for IQSS, TQSS, and AQSS, only a single modal query is employed.
Refer to caption: extracted/5959815/Styles/images/omnisep.png

## 3 Methods

### 3.1 Overview

The Omni-modal sound separation model (OmniSep) aims to perform sound separation based on different modal queries, where the query can originate from the audio modality ($A$), text modality ($T$), or visual modality ($V$). It separates noisy mixed audio into clean audio consistent with the query, as described in Section 3.2. Additionally, in Section 3.3, we introduce a method to incorporate interference noise information into the model based on the composed query of the original query and the negative query to assist the model in sound separation. To further achieve open vocabulary sound separation, we propose Query-Aug in Section 3.4, which allows the model to querying with any unrestricted natural language description.

### 3.2 OmniSep: Omni-modal Sound Separation.

OmniSep adopts an architecture similar to CLIPSEP (Dong et al., ), as illustrated in Figure 1. The Query-Net converts data from various modalities into corresponding query features $\mathbf{Q}$. Subsequently, we employ the Separate-Net to transform the audio mixture $A_{\text{mix}}$, consisting of $n$ audio sources, into clean sound $A_{\text{clean}}$ based on the query features $\mathbf{Q}$. However, since CLIPSEP periodically switches the query modality during training, the training objective remains unfixed, posing challenges in achieving optimal performance across all modalities. We address this issue using the Query-Mixup strategy.

#### Query-Net

To train a unified sound separation model adaptable to queries of various modalities, we selected the imagebind model, pre-trained on multiple modalities, to extract query features: $\mathbf{Q}_{A},\mathbf{Q}_{V},\mathbf{Q}_{T}=\texttt{ImageBind}(A,V,T)$
where $\mathbf{Q}_{A},\mathbf{Q}_{V},\mathbf{Q}_{T}\in\mathbb{R}^{1024}$, with the model parameters kept frozen during training.
Subsequently, we adopt the Query-Mixup strategy to mix up query features from different modalities, enabling OmniSep to optimize each modality concurrently and unify the training objectives with randomly sampled weight factors $w_{a}$, $w_{v}$, and $w_{t}$:

$$ $\mathbf{Q}=\frac{{w_{a}\mathbf{Q}_{A}+w_{v}\mathbf{Q}_{V}+w_{t}\mathbf{Q}_{T}} }{{w_{a}+w_{v}+w_{t}}},\quad w_{a},w_{v},w_{t}\in[0,1].$ (1) $$

#### Separate-Net

For a mixed audio signal $A_{\text{mix}}$ composed of $n$ audio sources $\{A_{1},A_{2},\cdots,A_{n}\}$, it is first converted into the magnitude spectrum $X$ with the Short-Time Fourier Transform (STFT). Subsequently, we input the spectrogram into the audio U-Net and obtain $k$ intermediate masks $\tilde{M}=\{\tilde{M}_{1},\cdots,\tilde{M}_{k}\}$, where $k\geq n$.
At the same time, each query feature $\mathbf{Q}_{i}$ obtained from the $(A_{i},V_{i},T_{i})$ corresponding to $i$-th sample is transformed into $k$-dimensional channel-wise weight $q_{i}$ through the fully connected mapping layer: $q_{i}=\text{Linear}(\mathbf{Q}_{i})$. Finally, the final predicted masks $\hat{M}_{i}$ can be obtained with the channel-wise weight $q_{i}$ and the intermediate masks ${\tilde{M}}$:

$$ $\hat{M}=\sum_{j=1}^{k}\sigma(w_{ij}q_{ij}\tilde{M}_{j}+b_{i}),$ (2) $$

where $w_{ij}\in\mathbb{R}^{k}$ is a learnable scale vector, $b_{i}\in\mathbb{R}$ a learnable bias, and $\sigma(\cdot)$ the sigmoid function. The training objective of the entire model the sum of the weighted binary cross entropy (WBCE) losses for each query source:

$$ $\mathcal{L}=\sum_{i=1}^{n}\texttt{WBCE}(M_{i},\hat{M}_{i})=\sum_{i=1}^{n}X \odot\left(-M_{i}\log\hat{M}_{i}-(1-M_{i})\log(1-\hat{M}_{i})\right),$ (3) $$

where $M_{i}$ is the ground truth mask for audio source $A_{i}$. During inference, it’s important to note that for the sound separation of single-modal queries, we input the single-modal query into the Sep-net without employing the Query-Mixup strategy. Furthermore, the subsequent subsections introduce two training-free strategies for query operations aimed at further enhancing the performance of the sound separation model.

### 3.3 Negative Query: Eliminate Interference Information from the Original Query.

For noise information, we treat it as a novel form of query and extract it into the corresponding Query feature $\mathbf{Q}_{N}$, employing the same method as for other modal queries. With the aid of multi-modal pre-training (Girdhar et al., ) and the Query-Mixup, features linked to different modal queries can be adeptly mapped into the same space, facilitating sound separation. Drawing inspiration from multi-modal joint retrieval (Wang et al., ), we employ the vector subtraction technique to eliminate the information associated with $\mathbf{Q}_{N}$ from the original Query $\mathbf{Q}$. However, it’s worth noting that compared to retrieval tasks, sound separation tasks demand a more fine-grained manipulation of representations. Direct subtraction may lead to the loss of local information in $\mathbf{Q}$, where any local information may be linked to sounds on certain frequency bands. Thus, when employing negative queries, we aim to maintain the stability of the frequency bands corresponding to non-key content by introducing a negative query weight $\alpha$, which is expressed as:

$$ $\mathbf{Q}^{\prime}=(1+\alpha)\mathbf{Q}-\alpha\mathbf{Q}_{N}.$ (4) $$

Subsequently, the final query feature $\mathbf{Q}^{\prime}$ is inputted into the Separate-Net for sound separation. In particular, when $\alpha=0$, $\mathbf{Q}^{\prime}=\mathbf{Q}$, which means that no noise information is applicable during sound separation.

### 3.4 Query-Aug: Towards the Query of Unrestricted Nature Language Descriptions.

Given that the majority texts in current audio-text paired datasets are accompanied by predefined category labels, the sound separation model encounters challenges when confronted with more authentic and unrestricted natural language descriptions during inference. To tackle this issue, we introduce the Query-Aug method, which facilitates open-vocabulary sound separation. In this method, we establish a query set containing the query feature corresponding to each class label. For a query feature $\mathbf{Q}_{des}$ of the unrestricted natural language descriptions, we identify the most closely related query feature $\mathbf{Q}_{aug}$ within the Query-Set, selecting it as the query for sound separation:

$$ $\mathbf{Q}_{{aug}}=\underset{\mathbf{Q}_{{i}}\in\text{Query-Set}}{\arg\max}\, \text{sim}(\mathbf{Q}_{{des}},\mathbf{Q}_{{i}}),$ (5) $$

where $\text{sim}(.,.)$ denotes the similarity between the query features.

## 4 Experiments

### 4.1 Implementation Details

We conducted experiments on two datasets, VGGSOUND (Chen et al., ) and MUSIC (Zhao et al., ), to evaluate the sound separation effect under different modal queries. VGGSOUND stands out as the largest audio-video consistent sound description dataset, boasting 550 hours of audio across 330 different sound categories, thereby representing various sound separation scenarios. Conversely, MUSIC focuses on instrument sounds, providing 10 hours of recordings featuring diverse instruments alongside corresponding performance videos, thus delineating a specific domain within sound separation.
For comparison with previous studies, we conducted sound separation experiments following the CLIPSEP. This entailed training our model on the MUSIC dataset and evaluating its performance on itself. Additionally, we trained the model on the VGGSOUND dataset and evaluated its performance on the VGGSOUND-CLEAN+ and MUSIC-CLEAN+ datasets, which contain manually processed clean sound separation evaluation samples.

Following the experimental settings of CLIPSEP, a 4-second audio segment is randomly selected from the entire audio as the audio source. For queries, we extract the audio feature of the entire audio as the audio query; the average image feature of 4 frames with a 1-second interval from the corresponding video serves as the image query, and the class label associated with the audio as the text query. Note that, during the inference, we extract $\mathcal{S}$ audio samples from the training set for each class as its audio query (where $\mathcal{S}=5$ for VGGSOUND, and $\mathcal{S}=1$ for MUSIC) according to the setting of  (Lee et al., ), for audio query sound separation (AQSS). More implementation details are shown in the Appendix A.

**Table 1: Comparison of sound separation performance among different methods on MUSIC, MUSIC-CLEAN+, and VGGSOUND-CLEAN+. Experiments marked with * are reproduced using CLIPSEP’s dataset partitioning for comparison. All +NQ results are outcomes when $\alpha=0.5$.**
|  | MUSIC | VGGSOUND-CLEAN+ | MUSIC-CLEAN+ |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | Mean SDR | Med SDR | Mean SDR | Med SDR | Mean SDR | Med SDR |
| Non-Queried Sound Separation |  |  |  |  |  |  |
| LabelSep | 8.18$\pm$0.80 | 7.82 | 5.55$\pm$0.81 | 5.29 | - | - |
| PIT (Yu et al., ) | 8.68$\pm$0.76 | 7.67 | 5.73$\pm$0.79 | 4.97 | 12.24$\pm$1.20 | 12.53 |
| Text-Queried Sound Separation |  |  |  |  |  |  |
| BERTSep | - | - | 5.09$\pm$0.80 | 5.49 | 4.67$\pm$0.44 | 4.41 |
| CLIPSEP-NIT (Dong et al., ) | - | - | 3.05$\pm$0.73 | 3.26 | 10.27$\pm$1.04 | 10.02 |
| CLIPSEP (Dong et al., ) | 8.36$\pm$0.83 | 8.72 | 2.76$\pm$1.00 | 3.95 | 9.71$\pm$1.21 | 8.73 |
| CLIPSEP-Text (Dong et al., ) | 7.91$\pm$0.81 | 7.46 | 5.49$\pm$0.82 | 5.06 | 10.73$\pm$0.99 | 9.93 |
| AudioSEP (Liu et al., )* | 9.82$\pm$0.89 | 8.76 | 6.26$\pm$0.87 | 5.57 | 11.23$\pm$0.99 | 10.28 |
| OmniSep(ours) | 10.65$\pm$1.07 | 9.97 | 6.70$\pm$0.66 | 5.73 | 12.55$\pm$0.77 | 12.68 |
| OmniSep(ours)+NQ | 10.92$\pm$1.06 | 9.97 | 7.57$\pm$0.67 | 6.52 | 14.15$\pm$0.95 | 14.46 |
| Image-Queried Sound Separation |  |  |  |  |  |  |
| SOP (Zhao et al., ) | 6.59$\pm$0.85 | 6.22 | 2.99$\pm$0.84 | 3.89 | 11.44$\pm$1.18 | 11.18 |
| CLIPSEP (Dong et al., ) | 8.06$\pm$0.79 | 8.01 | 5.46$\pm$0.79 | 5.35 | 12.20$\pm$1.17 | 12.42 |
| i-Query (Chen et al., )* | 10.54$\pm$0.72 | 9.73 | - | - | - | - |
| OmniSep(ours) | 10.97$\pm$1.03 | 10.21 | 6.69$\pm$0.67 | 6.43 | 13.15$\pm$0.92 | 13.89 |
| OmniSep(ours)+NQ | 11.08$\pm$1.02 | 10.22 | 7.68$\pm$0.69 | 6.60 | 13.78$\pm$1.02 | 13.99 |
| Audio-Queried Sound Separation |  |  |  |  |  |  |
| AQSS (Lee et al., )* | 6.43$\pm$0.92 | 5.73 | 5.34$\pm$0.71 | 5.30 | 8.56$\pm$0.75 | 7.82 |
| OmniSep(ours) | 10.26$\pm$1.13 | 10.06 | 7.12$\pm$0.65 | 5.45 | 12.92$\pm$0.89 | 14.04 |
| OmniSep(ours)+NQ | 10.40$\pm$1.11 | 10.14 | 7.22$\pm$0.68 | 5.07 | 13.43$\pm$0.98 | 14.14 |
| Composed Omni-Modal Queried Sound Separation |  |  |  |  |  |  |
| OmniSep(ours) | 11.03$\pm$1.05 | 10.21 | 7.46$\pm$0.65 | 6.32 | 13.49$\pm$0.94 | 14.04 |
| OmniSep(ours)+NQ | 11.16$\pm$1.04 | 10.33 | 8.00$\pm$0.69 | 6.43 | 13.82$\pm$0.98 | 14.07 |

**Table 2: Comparison of SDR values on TQSS and IQSS among sound separation models trained with diverse modality data (Text, Image and Audio) and training methods (MixUP). AVG SDR represents the average SDR across different sound separation models queried with text and image.**
| ID | Text | Image | Audio | MixUP | TQSS | IQSS | AVG SDR |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mean SDR | Med SDR | Mean SDR | Med SDR |  |  |  |  |  |  |
| #1 | ✓ |  |  |  | 6.70$\pm$0.68 | 5.81 | 3.53$\pm$0.49 | 2.86 | 5.12$\pm$0.59 |
| #2 |  | ✓ |  |  | 5.72$\pm$0.83 | 5.41 | 6.33$\pm$0.68 | 5.74 | 6.03$\pm$0.76 |
| #3 | ✓ | ✓ |  |  | 6.33$\pm$0.71 | 5.83 | 6.51$\pm$0.72 | 5.77 | 6.42$\pm$0.72 |
| #4 | ✓ | ✓ | ✓ |  | 6.37$\pm$0.68 | 5.68 | 6.53$\pm$0.68 | 6.33 | 6.45$\pm$0.68 |
| #5 | ✓ | ✓ | ✓ | ✓ | 6.70$\pm$0.66 | 6.22 | 6.69$\pm$0.67 | 6.43 | 6.70$\pm$0.66 |

### 4.2 Main Results

#### OmniSep: Sound Separation for Omni-modal Queries.

The OmniSep is designed to address sound separation tasks with diverse modal queries. Table 1 presents the SDR comparison of sound separation across various modal queries. Specifically, TQSS, IQSS, and AQSS represent text-, image-, and audio-queried sound separation, respectively, while Non-Queried Models denotes the method of implicitly learning sounds within the model.
Across each task in TQSS, IQSS, and AQSS, our OmniSep achieves state-of-the-art sound separation performance, improving the Mean SDR by 0.43 to 4.36. This demonstrates that OmniSep can be queried with any single modality query. Furthermore, when queried with a composed Omni-modal query, OmniSep can refer to information from different modalities to achieve a more robust sound separation performance, with the Mean SDR of 7.46 on VGGSOUND-CLEAN+.
Additionally, negative queries further enhance the sound separation performance by leveraging noise information during test-time. Compared to OmniSep without negative queries, the Mean SDR is further enhanced by 0.10 to 1.60. A detailed analysis of negative queries is provided in subsection 4.3.

#### Query-Mixup: Enhancing Multi-modal Joint Training.

To further validate the effectiveness of our proposed method, we conducted detailed ablation experiments as shown in Table 2:
(1) Multi-modal joint training for
different modal queried sound separation:
The comparison between methods trained using single-modal data and multi-modal data highlights the adaptability of the latter in handling sound separation tasks with different modal queries. While the single-modality training method may yield better results on specific tasks (e.g., #1 trained with only text queries achieves a 0.73 higher Mean SDR on TQSS than #3), the #3 model, jointly trained with multi-modal queries, achieves an AVG SDR of 6.42, consistently outperforming #1 and #2 with improvements of $0.39\sim 1.30$. Furthermore, the introduction of audio modality data during training (#4) leads to further performance enhancement, with a 0.03 increase in performance from #3 to #4.
(2) Query-mixup for unified omni-modal sound separation: To handle sound separation tasks with various modal queries, we employ Query-Mixup. In Experiment #5, we observed that the average SDR of 6.70 surpassed Experiment #4 of 6.45 by 0.25. Remarkably, this enables us to achieve comparable sound separation performance in TQSS as the text-specific training method (#1). This indicates that the Query-Mixup method circumvents the problem of fluctuating training objectives in traditional multimodal joint training methods, which often affects the performance of single-modal queried sound separation.

Figure: (a) VGGSOUND-CLEAN+
Refer to caption: extracted/5959815/Styles/images/NQ/VGGSOUND-text.png

### 4.3 Negative Query: Enhancing Sound Separation With Noise Information.

To further enhance the efficacy of sound separation, we introduce negative queries to leverage noise information. Illustrated in Figure 2 are the variations of SDR with the negative query weight $\alpha$ on VGGSOUND-CLEAN+ and MUSIC-CLEAN+. We compare two distinct methods of negative query augmentation: firstly, the naive subtraction method ($\mathbf{Q}^{\prime}=\mathbf{Q}-\alpha\mathbf{Q}_{N}$), which employs negative representation processing akin to retrieval tasks; and secondly, our proposed method ($\mathbf{Q}^{\prime}=(1+\alpha)\mathbf{Q}-\alpha\mathbf{Q}_{N}$), which additionally enhances the original query with proportionally weight. Please note that in both methods, when $\alpha=0$, the negative query is not adopted.

Here are the conclusions drawn from the figure:
(1) Adaptability of Negative Query: Across all sound separation tasks (TQSS, IQSS, and AQSS), the negative query significantly enhances sound separation performance. For instance, on MUSIC-CLEAN+, when $\alpha=1$, the Mean SDR stands at 14.20, marking a notable improvement of 1.65 from the 12.55 achieved without negative queries. (2) Proportional Weighting vs. Direct Subtraction: Comparative analysis reveals that our proposed proportional weighting method consistently outperforms traditional direct subtraction method. Across all tasks and varying alpha values, the solid line representing our method consistently surpasses the performance depicted by the dotted line, which represents direct subtraction. This demonstrates the efficacy of our proportional weighting approach in mitigating interference on the original query information when negative queries are employed.
(3) Robustness of Weight Selection:
As a training-free method, it offers considerable flexibility in utilizing negative queries during the inference process, allowing for the use of any weight $\alpha$ to negative query. However, this flexibility also presents challenges in determining the optimal weight $\alpha$.
The Mean SDR of naive direct subtraction shows significant fluctuations with changes in the weight $\alpha$ , highlighting the considerable impact of $\alpha$ on the naive method. Moreover, the optimal $\alpha$ value for achieving the best performance for naive direct subtraction varies across different tasks and datasets. Hence, finding a universally applicable fixed weight $\alpha$ for the naive direct subtraction method is not feasible. On the contrary, even across various tasks and datasets, the mean SDR range resulting from the introduction of negative queries using our proposed proportional weighting method is no more than 0.45 for different $\alpha$ weights, thus obviating the need for excessive adjustment of the weight $\alpha$.
(For the TQSS task on VGGSOUND-CLEAN+, when $\alpha$=2, the Mean SDR of our method remains at 7.37, while the naive sound separation method has already decreased to 4.32), demonstrating significant robustness.

### 4.4 Sound Separation with Unrestricted Textual Descriptions.

As mentioned in Liu et al. (), previous methods (Ochiai et al., ; Veluri et al., ) for text-queried sound separation relied on predefined class labels as queries, constraining their adaptability to accommodate unrestricted natural language descriptions. To evaluate sound separation performance with such unrestricted descriptions as queries, we utilized GPT-3.5 to rewrite all predefined class labels into varied yet semantically consistent textual descriptions. Further details are provided in Appendix B.

In Table 3, we conduct a comparative analysis of the sound separation performance achieved by various methods using different text queries: (1) Predefined tags vs. Unrestricted descriptions: Training on predefined class labels limits the model’s ability to generalize to out-of-domain text. A significant drop in performance is observed when inferring out-of-domain text (Mean SDR drops from 5.49 in Experiment #6 to 3.53 in Experiment #8). (2) Multi-modal joint training vs. Single-modal training: Joint training with multi-modal queries (Experiment #9) helps fill the gap between the limited representations of text queries, enhancing generalization to out-of-domain text. (3) Open-vocabulary sound separation with Query-Aug: Despite the improvement in robustness achieved by multi-modal joint training, a performance gap remains due to cross-modal differences. The Mean SDR of 4.95 in Experiment #10 is significantly lower than that of Experiment #6,

**Table 3: The performance comparison of sound separation between queries using predefined class labels and unrestricted textual descriptions on the VGGSOUND-CLEAN+ dataset.**
| ID | Method | Mean SDR | Med SDR |
| --- | --- | --- | --- |
| Query with predefined class labels. |  |  |  |
| # 6 | CLIPSEP-Text | 5.49$\pm$0.82 | 5.06 |
| # 7 | OmniSep | 6.70$\pm$0.66 | 5.73 |
| Query with unrestricted textual descriptions. |  |  |  |
| # 8 | CLIPSEP-Text | 3.53$\pm$0.52 | 2.91 |
| # 9 | +Query-Aug | 5.24$\pm$0.79 | 4.87 |
| # 10 | OmniSep | 4.95$\pm$0.81 | 4.45 |
| # 11 | +Query-Aug | 6.32$\pm$0.64 | 5.97 |

which achieved a sound separation performance of 5.49 with in-domain class labels. Our proposed Query-Aug method addresses this by selecting suitable in-domain text for text enhancement based on text similarity. The Query-Aug method significantly enhances the robustness of the sound separation model against out-of-domain text. When employing unrestricted out-of-domain text descriptions for sound separation, the Mean SDR is further improved by 1.37 compared to OmniSep (Experiment #11 yields an SDR of 6.32, whereas Experiment #10 achieves 4.95). It’s noteworthy that the performance of OmniSep+Query-Aug (#11) with out-of-domain text is comparable to that achieved with predefined in-domain class labels (#6, #7). Specifically, while Experiment #6 achieves a Mean SDR of 5.49, Experiment #11 attains a Mean SDR of 6.32, which is a superior performance despite being in a more challenging condition, achieving open-vocabulary sound separation.

### 4.5 How Query-Mixup Enables Omni-Modal Sound Separation?

Figure: Figure 3: UMAP visualization of three different modal imagebind embeddings. The mix embedding is a weighted embedding of the three modalities.
Refer to caption: x1.png

Figure 3 illustrates the spatial distribution relationship of imagebind representations across different modalities, showing that features from various modalities cluster independently with significant cross-modal gaps, as discussed in Liang et al. (). This makes it challenging for a single sound separation model to process data from different modalities simultaneously. In previous methods, such as CLIPSEP, the model constantly switches between text and image features, making it difficult to simultaneously accommodate the distinct distribution requirements of both text and image representations. Consequently, the model fails to achieve optimal performance for both modalities at the same time.

Our proposed OmniSep uses a query-mixup strategy to combine features of the same semantics from different modalities, effectively bridging the gap between their distributions with "mixed embeddings". It can optimize the mixed embeddings, composed of different modal embeddings, to accommodate both uni-modality queries and multi-modality composed queries. In omni-modal composed sound separation, queries from different modalities correspond to the same semantics and can complement each other, ensuring the model has a more comprehensive understanding of the query information, thereby improving sound separation performance.

In particular, functions such as omni-modal composed sound separation and negative query rely on the additive and subtractive characteristics of embeddings between different modalities. The query embeddings required for composed-query and negative-query are all in the mixed embedding area between modalities. Therefore, the Query-Mixup strategy forms the basis of composed-query and negative query capabilities, enhancing the model’s controllability.

**Table 4: Qualitative results of sound separation using different modal queries. Interference and Target denote the reconstructed signals of interference audio and ground truth audio using the ground truth ideal binary masks, respectively. All spectra are presented on a logarithmic frequency scale.**
| Query | Mixture | Interference | Target | Prediction |
| --- | --- | --- | --- | --- |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |

### 4.6 Qualitative analysis

To demonstrate the effectiveness of OmniSep, we conduct a qualitative analysis of sound separation with various modal queries, as depicted in Table 4. The results showcase our OmniSep successfully separate audio signals corresponding to the queries across TQSS, IQSS, and AQSS tasks.
Additionally, in Appendix C.1, we present qualitative results of sound separation with and without negative queries in TQSS. It’s noteworthy that in a sample of child singing audio mixed with child crying noise interference, traces of the crying persist in the separation results, posing challenges to effective separation due to the association of both sounds with the child. However, results obtained with negative queries demonstrate that OmniSep+Negative Query can effectively eliminate noise by leveraging the undesired sound information. The introduction of negative query information significantly enhances the removal of interfering sound signals and improves the flexibility of the sound separation model. In addition, as demonstrated in the comparison of sound separation using unrestricted natural language queries in Appendix C.2, Query-Aug enhances the model’s ability to generalize to out-of-domain unrestricted descriptions, thereby achieving open-vocabulary sound separation. Please visit the demo page at [https://omnisep.github.io/](https://omnisep.github.io/) to see more sound separation results and learn more about OmniSep.

## 5 Conclusion

Researchers have adopted sound separation to scaling up audio datasets, but existing sound separation methods are limited to single-modal queries and have certain limitations. In this study, we introduce the multi-modal query mixing strategy, Query-Mixup, and propose the first Omni-Modal Sound Separation model (OmniSep). OmniSep can perform sound separation based on any modal query, including single-modal queries such as text, images, and audio, as well as multi-modal composed queries. Additionally, we propose a method to remove undesired sound information according to negative queries from the original query, thereby improving sound separation performance and model flexibility.
To overcome the challenge of limited text labels, we introduce Query-Aug, achieving open-vocabulary sound separation and facilitating the use of unrestricted natural language queries. Experimental results on different modal sound separation tasks (TQSS, IQSS, AQSS) demonstrate that our model achieves state-of-the-art performance in omni-modal sound separation.

## Appendix A Implementation Details

Same as the experimental setting of Dong et al. (), for all audio samples, we conducted experiments on samples of length 65535 (approximately 4 seconds) at a sampling rate of 16 kHz. For spectrum computation, we employed a short-time Fourier transform (STFT) with a filter length of 1024, a hop length of 256, and a window size of 1024. All images were resized to $224\times 224$ pixels. The audio model in this paper is a wildly used 7-layer U-Net network with $k=32$, generating 32 intermediate masks. All models were trained with a batch size of 128, using the Adam optimizer with parameters $\beta_{1}=0.9$, $\beta_{2}=0.999$, and $\epsilon=10^{-8}$, for 200,000 steps. Additionally, we employed warm-up and gradient clipping strategies, following Dong et al. (). We compute the signal-to-distortion ratio (SDR) using museval (Stöter et al., ). All experiments were conducted on a single A800 GPU.

## Appendix B Sample of Unrestricted Natural Language Descriptions.

We utilized ChatGPT 3.5 to transform each class label from VGGSOUND into natural language descriptions without constraints, ensuring semantic consistency. Table 5 displays the instruction prompt employed in ChatGPT. For further analysis of the Query-Aug, Table 6 presents numerous examples of these unrestricted descriptions.

**Table 5: Instruction prompt when using chatgpt to generate unlimited natural language descriptions.**
| User: | Could you please assist me in rephrasing these descriptions of audio content to convey |
| --- | --- |
|  | the same meaning but with different wording? |
| Chatgpt: | Of course! Please provide the descriptions you’d like me to rephrase, and I’ll help you |
|  | with alternative wording. |
| User: | tapping guitar |
| Chatgpt: | Tapping on a guitar to produce rhythms |
| User: | dog barking |
| Chatgpt: | A dog vocalizing with a bark. |

**Table 6: Samples of class labels along with corresponding unrestricted natural language descriptions.**
| Predefined Class Label | Unrestricted Nature Language Description |
| --- | --- |
| playing didgeridoo | Performing on a didgeridoo instrument. |
| golf driving | Swinging a golf club on the course. |
| dog barking | A dog letting out barking noises. |
| playing timpani | Beating drums in a timpani performance. |
| magpie calling | The melodious call of a magpie bird. |
| dog bow-wow | A dog letting out a woofing sound. |
| subway metro underground | The rumbling noise of a subway train passing. |
| cat growling | A cat emitting a growling noise. |

## Appendix C More Experiments

### C.1 Qualitative Analysis on Sound Separation with Negative Query.

In Table 7, we provide a comparison of the sound separation effect between employing negative query (OmniSep+Neg query) and not using it (OmniSep). We exclusively showcase TQSS as an illustrative example, acknowledging the consistent performance of negative query across different modal queries in sound separation tasks. As illustrated in the Mel spectrogram comparison of the separation results, integrating negative query significantly enhances the removal of interfering audio signals from the mixed audio, effectively isolating the content associated with the negative query. Additional examples and corresponding audios are available on the demo page. Comparing the raw files of the two methods, it’s evident that using negative query can greatly enhance the flexibility of the sound separation model, particularly when addressing challenges such as the inability to effectively differentiate between two sound signals using a single text query.

### C.2 Qualitative Analysis on Sound Separation with Query-Aug.

In Table 7, we display the sound separation outcomes for an unrestricted natural language description query. It’s clear from the mel spectrogram that when the query-augmentation method is not employed, the model (OmniSep) faces difficulties in grasping the sound information linked to the query. This results in a reduced capability to effectively separate the sound signals associated with the unrestricted natural language description. Conversely, with the integration of the query-augmentation method, the model (OmniSep+Query-Aug) showcases enhanced effectiveness in segregating the corresponding sound information content and removing interfering sound signals.

**Table 7: Qualitative comparison of sound separation models with and without negative queries. All OmniSep+Neg Query experiments are obtaned with $\alpha=1$. We highlighted the enhanced separation effect achieved by employing negative queries with red boxes.**
| Mixture | Interference | Target | OmniSep | OmniSep+Neg Query |
| --- | --- | --- | --- | --- |
|  |  |  |  |  |
|  |  |  |  |  |

**Table 8: Qualitative comparison of sound separation models with and without negative queries. We highlighted the enhanced separation effect achieved by employing negative queries with red boxes.**
| Mixture | Interference | Target | UMSS | UMSS+Query-Aug |
| --- | --- | --- | --- | --- |
|  |  |  |  |  |
|  |  |  |  |  |

## Appendix D Limitations

This study still lacks more generalized validation. For the sake of comparison with prior models, we primarily conduct experiments on the VGGSOUND and MUSIC datasets. While these datasets are well-suited for omni-modal sound separation tasks, and VGGSOUND already encompasses over 300 sound events across various categories, they still fall short of representing all real-world audio events, particularly some abstract sound events like pop music, which are not adequately represented in the dataset. Moving forward, we aim to construct a more extensive dataset to address this limitation.

## Appendix E Ethical Discussion

The sound separation method proposed in this paper enables the separation of target queries according to user preferences, but it may be susceptible to misuse, such as separating background music (BGM) and vocals from copyrighted songs, extracting dialogue and various sound effects from copyrighted movies, and so on. However, given the long history of development in this field and the relatively clear copyright ownership of various audio and video products, the potential social impact is not expected to be significant.

## Reproducibility Statement

All our code, and model weights will be open-sourced. In Section 3, we provide a detailed description of the OmniSep modules and training instructions. Section 4.1 and Appendix A offer additional details on the training process.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - What happens when the query is some class that’s not present inside the input mixture audio?
- Table 2: why no results on AQSS?
- Query-Aug Adaptability:
    - Is the Query-Aug method adaptable only to text queries $Q_T$, or does it extend to other modalities as well?
    - Additionally, how will the model handle text prompts containing multiple sound sources (e.g., “the sound of a baby with her parent’s soothing voice”)? Is the model trained to handle multi-source separation?
- Number of Audio Sources in Mixtures: For AQSS VGGSOUND, does $\mathcal{S}=5$ mean $A_\text{mix}=\sum_{n=1}^6A_n$? then for all other setups, is $A_\text{mix}=A_1+A_2$?
- Did the authors use the pre-trained ImageBind weights from the official repository (https://github.com/facebookresearch/ImageBind), or did they train ImageBind from scratch? They should mention the repository if pre-trained weights were used. If trained from scratch, they should include the details of the pre-training specifications.
- How does OmniSep respond when a query references a class that is not present within the input audio mixture?
- Why are there no AQSS results reported in Table 2?

Minor comments
- typo @Figure 1: “…, donated as $\text{Q}_T$, …”
- Figure 1: inside the Query-Mixup block, the weight factors are denoted as $W$, where it should be $w$
- grammar @line 217: “The training …”
- Table 1: should note the source of VGGSOUND-CLEAN+ and MUSIC-CLEAN+ subsets. Also provide a couple of sentences for details of these subsets.
soundness: 3
strengths: The paper presents an omni-modal sound separation approach, allowing users to conduct sound separation tasks using queries across various modalities, including text, image, and audio, both independently and jointly. Additionally, the authors conducted a comprehensive evaluation across several benchmarks. Further, the inclusion of extensive ablation studies provides insight into the contributions of each component: impact of Query-Mixup, negative query weighting, long text description-queried separation, and analysis on query embeddings.
summary: This paper presents OmniSep, a framework for omni-modal sound separation, which supports sound isolation using queries from multiple modalities, such as text, image, and audio, either independently or in combination. The key method of omni-modal source separation is the introduction of Query-Mixup, a strategy that mixes query features from different modalities, using a pre-trained ImageBind. OmniSep further enables open-vocabulary sound separation with Query-Aug, a retrieval-augmented method that enhances adaptability, particularly for text-based queries. Experimental evaluations on MUSIC, VGGSOUND-CLEAN+, and MUSIC-CLEAN+ datasets showcase OmniSep’s SOTA performance across various modality-based separation tasks.
weaknesses: While the paper presents a solid framework, there are some details missing that make it difficult to give a firm acceptance (please answer Questions). 

The use of full-length video as a query, which includes the target segment, raises questions about the potential for information leakage. It’s unclear if the QueryNet architecture ensures a bottleneck to prevent the model from “cheating” by directly accessing target audio features. To better demonstrate the model’s robustness, an alternative setup might involve using a different video as the query that shares the same class identity as the input audio.

Additionally, the comparison in Section 4.3 between the proposed weighting and a naive weighting approach on $Q_N$ may not be particularly insightful, as both methods use the same model. The proposed approach always has a weight difference of 1 ($1+\alpha$ vs. $\alpha$), while the comparison (naive) approach reduces the weight gap between $Q$ and $Q_N$, hence the results in Figure 2 are somewhat predictable. However, the insights provided by varying $\alpha$ are valuable.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 1
questions: 1. Separate-Net only converts to magnitude spectrum X, ignoring phase. Modern time-frequency methods (RTFS-Net, TF-GridNet) use retain both magnitude and phase information via concatenated real/imaginary STFT outputs. Some methods additionally concatenate the magnitude to create a three channel representation. Have you tried these approaches? 

2. Why keep ImageBind parameters frozen? What if it was trained from scratch with the model? What if it was initialized and fine tuned with the model using a lower learning rate? 

3. For the Negative Query, why (1+alpha)Q-alphaQ_N instead of (1-alpha)Q+alphaQ_N, Q-alphaQ_N or a scale-and-shift approach such as in stable diffusion - please add some justification in this section or some experimental evidence. 

4. And the problems raised in the weaknesses section.
soundness: 3
strengths: The paper presents a clear and well-motivated problem statement, addressing three fundamental limitations in current sound separation approaches: the absence of unified multi-modal query handling, insufficient flexibility in sound manipulation (particularly for unwanted sound removal), and restricted vocabulary constraints that preclude natural language descriptions. The authors construct a compelling narrative throughout the introduction and literature review, effectively contextualizing their contributions within the field through comprehensive citations and thorough analysis of related work.

The methodology is generally well-documented to begin with. The experimental validation is particularly robust, encompassing diverse tasks and datasets that demonstrate the method's versatility. The authors provide extensive ablation studies that illuminate the model's internal mechanisms and justify some of the architectural choices. Their commitment to reproducibility through code release further strengthens the paper's contribution to the field.
summary: This paper presents OmniSep, a novel unified framework for query-based sound separation that accommodates multiple query modalities (text, image, and audio) within a single model. The authors introduce three key technical contributions: Query-Mixup for simultaneous multi-modal query processing, a negative query mechanism for unwanted sound suppression, and Query-Aug for handling natural language descriptions beyond predefined class labels. The model's architecture represents an advance over previous approaches, which were typically constrained to single-modality queries.

Experimental validation on several datasets demonstrates OmniSep's performance compared to existing methods across all query modalities. The model exhibits robust separation capabilities in complex, multi-source scenarios and achieves state-of-the-art results on MUSIC, VGGSOUND-CLEAN+, and MUSIC-CLEAN+ benchmarks. OmniSep's multi-modal query capability enables enhanced separation performance through the simultaneous application of different query types, such as combining textual and visual queries.
weaknesses: There are several issues with the paper, which can broadly be classified into two areas.  Most of these issues can be fixed with better, scientific writing, and more explanation, rigour and experimentation. 

Technical Clarity Issues: 
1. Significant documentation gaps in core variables and operations in Separate-Net: 
   - q_i and q_ij are poorly explained or completely undefined in Separate-Net 
   - Many variables are undefined and or with no dimensions specified. All variables should be clearly defined with dimensions given.  
   - M(hat) is defined but never used. 
   - Mechanism of how masks are used/applied is not explained.
   - Audio U-Net lacks both citation and architectural explanation.
2. Query-Aug undefined components: 
   - Q_des: completely undefined without dimensions and no explanation of how these features are obtained (T5/Bert?).
   - Query-Set: is undefined and dimensions not specified.
   - Q_aug defined as argmax but its integration into model never explained.
   - sim(.,.) is not a standard operator, it should be properly defined (cosine similarity?).

Experimental Validation Problems: 
1. Table 1 uses a 2017 model called PIT for non-queried sound separation. State-of-the-art audio-only separation methods (i.e. TDANet, TF-GridNet) would serve stronger baselines (while these models are for speech separation, so is PIT. Additionally, PIT was proposed so solve the permutation problem, not as a strong speech separation model).  
2. Table 2 does not show that the query-mixup method works, only that it scales across different modalities. Validating its effectiveness would mean comparing to other methods and/or different strategies. 

However, there are some other issues. While the method is interesting, the paper's novel work can be summarised as a weighted average of modalities, a linear layer and (1+alpha)Q-alphaQ_N for negative queries.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: The introduction of the Separate-Net is missing some details:
1. Line 210: what is k and what does k corresponds to? 
2. Does the query conditioned to the U-Net? If not, why query is not conditioned to UNet? If possible, it would be helpful to visualize the qi in some examples. It feels odd to me that the query only results in a channel-wise weight.
soundness: 3
strengths: 1. A novel model for source separation queried by audio, text and image. Propose a novel query-mixup method to enable such model.
2. Propose novel negative query and query-aug method to improve performance and flexibility at inference time.
3. State-of-the-art performance on separation tasks queried by all modalities
summary: This paper proposes an audio source separation model that can be queried via audio, language and image. To construct such model, this paper proposes query mixup which blends query from different modalities during training and ensure queries from different modalities reside in the same semantic space. This paper also propose a negative query mechanism to enhance the flexibility of query in inference stage. Last, to enable open-vocabulary text query input to the model which trained on close-set class name, this paper propose a query-augmentation method at inference time which retrieves nearest class name. Experiment result shows the model outperforms existing model in all modalities on MUSIC, VGGSOUND-CLEAN+, and MUSIC-CLEAN+ datasets.
weaknesses: 1. Presentation: Certain aspects of the paper’s presentation feel oversimplified, particularly in explaining key contributions and technical details. Please refer to the questions section for specific areas needing clarification.

2. Contribution of Query-Aug: The significance of the query-augmentation (query-aug) contribution seems overstated. For example, lines 054–056 suggest that current systems cannot handle open-vocabulary queries. However, Liu et al. (2023) ("Separate Anything You Describe") demonstrates that open-vocabulary language queries are achievable using audio-text datasets, such as Clotho or AudioCaps. Furthermore, the model proposed in this paper should be capable of training on these audio-text datasets and potentially on datasets lacking complete audio, text, or image pairs. An expanded discussion comparing this work to Liu et al. (2023), along with a rationale for the choice of datasets, would strengthen the contribution.

3. Experimental Results: The presentation of experimental results lacks some necessary detail:
 - What training settings were used for the models in Table 2, and how do these models (particularly model #5) correspond to those in Table 1?
 - For Table 4, a comparison of results from querying the same audio mixture across different modalities would provide valuable insights.

4. Metrics for Comparison: In prior works cited in the paper, metrics such as SI-SDR and SDRi are commonly used for source separation tasks, as they better reflect model performance in recent studies. Including these metrics or providing justification for the metrics chosen would enhance the comparative analysis.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: The motivation of Query-mixup is clear. On the other hand, its effectiveness compared to finetuning of ImageBind is not clear. Why don't you simply finetune your encoder so that all the modalities align well each other to the inputs of your interests. It is also unclear if the performance improvement really comes from the proposed three methods because the encoders in CLIPSep and OmniSep are different. Table 3 is a nice way to prove Query-Aug is actually important to increase SDR. Not sure about other two. You could have tried the rest of two methods on CLIPSep and see the performance improvements of CLIPSep?
soundness: 3
strengths: - Three novel training techniques, Query-mixup, NQ, Query-Aug, are proposed and evaluated in the standardized experimental settings
- Consistent improvements of median SDR in Table 2, which demonstrates the SOTA performance of OmniSep in the query-based sound separation field
- Good visualization of ImageBind embeddings to show the clear motivation of the Query-mixup
summary: This paper attempts to cover text-, image- and audio-queried sound separation all at once with one model by exploiting ImageBind as its encoder. Several training techniques are introduced and verified its effectiveness in terms of signal-to-noise ratio as well as spectral similarity, which demonstrates the superiority of OminiSep to other existing query-based sound separation methods
weaknesses: - Inappropriate motivation: I felt puzzled when I was reading the introduction part because the original motivation of the paper was how we could increase the number of high-quality training dataset although this motivation was never addressed anywhere in the other sections of the paper. If the objective is like the above, why don't we simply compare your scheme to other data augmentation methods or artificial data creation methods other than sound separation? I agree that denoising with sound separation would serve as one of many methods to achieve the same goal. Then, I believe the author should prove that OmniSep is the one of the best methods to increase the number of data. That being said, while assessing the paper, I somehow felt that this was not your intention. Since the technical contributions of the paper are fair enough, I suggest rewriting the introduction part in a way like your motivation is to improve the quality or coverage of text-, image-, or audio-queried sound separation and NOT to use sound separation to increase the number of training datasets for other purposes. 
- As an expert in this field, I don't agree relying only on SDR to measure the performance of any sound separation methods. Rather than showing pairs of spectrums, I suggest conducting a listening test to doublecheck if the performance of OmniSep really dominates others and the difference is recognizable. If the goal of the paper is not to make separated data available to human, as you originally stated in the introduction part, i.e., data increase is the objective, I don't think having a subjective listening test enriches the content. Otherwise, I strongly recommend this because this is a common practice in the sound separation community.

Meta-review: This paper presents a novel system for query-based sound separation, called OmniSep, that accommodates multiple query modalities (text, image, and audio), either independently or in combination. Technical contributions consist in Query-Mixup for simultaneous multi-modal query processing, which is also supported by a negative query mechanism for sound suppression at inference, and Query-Aug for handling natural language descriptions beyond naive class labels, so enabling open-vocabulary sound separation.
Experimental validation on several datasets demonstrates OmniSep's iincreased performance as compared to existing methods (single-modality) across all query modalities. 

This work is appreciated under several respects, including the interesting addressed task, the technically sound and novel methodological contributions, and the experimental analysis provided.

Weak aspects regard, in general, several requests of better addressing some specific parts of the methodology (for all three main contributions, but mainly for Query-Aug), and clarifications/revision of the experimental section. 

The authors replied properly to all such comments and the initial scores (6, 5, 5, 5) became all positive (6, 6, 6, 6).

Therefore, this paper can be considered acceptable for publication to ICLR 2025.

Decision: Accept (Poster)