Title: Continual Audio-Visual Sound Separation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Audio-Visual Learning

Sound Separation

Continual Learning
Abstract: In this paper, we introduce a novel continual audio-visual sound separation task, aiming to continuously separate sound sources for new classes while preserving performance on previously learned classes, with the aid of visual guidance. This problem is crucial for practical visually guided auditory perception as it can significantly enhance the adaptability and robustness of audio-visual sound separation models, making them more applicable for real-world scenarios where encountering new sound sources is commonplace. The task is inherently challenging as our models must not only effectively utilize information from both modalities in current tasks but also preserve their cross-modal association in old tasks to mitigate catastrophic forgetting during audio-visual continual learning. To address these challenges, we propose a novel approach named ContAV-Sep ($\textbf{Cont}$inual $\textbf{A}$udio-$\textbf{V}$isual Sound $\textbf{Sep}$aration). ContAV-Sep presents a novel Cross-modal Similarity Distillation Constraint (CrossSDC) to uphold the cross-modal semantic similarity through incremental tasks and retain previously acquired knowledge of semantic similarity in old models, mitigating the risk of catastrophic forgetting. The CrossSDC can seamlessly integrate into the training process of different audio-visual sound separation frameworks. Experiments demonstrate that ContAV-Sep can effectively mitigate catastrophic forgetting and achieve significantly better performance compared to other continual learning baselines for audio-visual sound separation. Code is available at: https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024.

Continual Audio-Visual Sound Separation

Abstract

In this paper, we introduce a novel continual audio-visual sound separation task, aiming to continuously separate sound sources for new classes while preserving performance on previously learned classes, with the aid of visual guidance. This problem is crucial for practical visually guided auditory perception as it can significantly enhance the adaptability and robustness of audio-visual sound separation models, making them more applicable for real-world scenarios where encountering new sound sources is commonplace. The task is inherently challenging as our models must not only effectively utilize information from both modalities in current tasks but also preserve their cross-modal association in old tasks to mitigate catastrophic forgetting during audio-visual continual learning. To address these challenges, we propose a novel approach named ContAV-Sep ( Cont inual A udioV isual Sound Sep aration). ContAV-Sep presents a novel Cross-modal Similarity Distillation Constraint (CrossSDC) to uphold the cross-modal semantic similarity through incremental tasks and retain previously acquired knowledge of semantic similarity in old models, mitigating the risk of catastrophic forgetting. The CrossSDC can seamlessly integrate into the training process of different audio-visual sound separation frameworks. Experiments demonstrate that ContAV-Sep can effectively mitigate catastrophic forgetting and achieve significantly better performance compared to other continual learning baselines for audio-visual sound separation. Code is available at: https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024 .

1 Introduction

Humans can effortlessly separate and identify individual sound sources in daily experience [25, 7, 64, 33]. This skill plays a crucial role in our ability to understand and interact with the complex auditory environments that surround us [34]. However, replicating this capability in machines remains a significant challenge due to the inherent complexity of real-world auditory scenes [7, 77]. Inspired by the multisensory perception of humans [62, 60], audio-visual sound separation tackles this challenge by utilizing visual information to guide the separation of individual sound sources in an audio mixture.

Recent advances in deep learning have led to significant progress in audio-visual sound separation [84, 23, 21, 67, 14, 65, 81, 63, 11, 71]. Benefiting from more advanced architectures ( e.g., U-Net [84, 23], Transformer [14], and diffusion models [27]) and discriminative visual cues ( e.g., grounded visual objects [67], motion [83], and dynamic gestures [21]), audio-visual separation models are able to separate sounds ranging from domain-specific speech, musical instrument sounds to open-domain general sounds within training sound categories. However, a limitation of these studies is their focus on scenarios where all sound source classes are presently known, overlooking the potential inclusion of unknown sound source classes during inference in real-world applications. This oversight leads to the catastrophic forgetting issue [32, 3], where the fine-tuning of models on new classes detrimentally impacts their performance on previously learned classes. Despite Chen et al. [14] demonstrating that their iQuery model can generalize to new classes well through simple fine-tuning, it still suffers from the catastrophic forgetting problem on old classes. This prevents the trained models from

continuously updating in real-world scenarios, impeding their adaptability to dynamic environments. The question how to effectively leverage visual guidance to continuously separate sounds from new categories while preserving separation ability for old sound categories remains open.

To bridge this gap, we introduce a novel continual audio-visual sound separation task by integrating audiovisual sound separation with continual learning principles. The goal of this task is to develop an audio-visual model that can continuously separate sound sources in new classes while maintaining performance on previously learned classes. The key challenge we need to address is catastrophic forgetting during continual audio-visual learning, which occurs when the model is updated solely with data from new classes or tasks, resulting in a significant performance drop on old ones. We illustrate our new task and the catastrophic forgetting issue in Fig. 1.

Unlike typical continual learning problems such as task-, domain-, or classincremental classification in visual domains [2, 57, 38, 53, 85], which result in progressively increasing logits (or probability distribution) across all ob-

Task 4

Task 1:{'accordion', 'bassoon', 'clarinet'...}

…

Task 4:{ 'erhu', 'guzheng', 'pipa'...}

Clarinet

Clarinet

Bassoon

Pipa

…

Task 1

Task 1:{'accordion', 'bassoon', 'clarinet'...}

Model

Model

served classes at each incremental step, our task uniquely produces fixed-size separation masks throughout all incremental steps. In this context, each entry in the mask does not directly correspond to any specific classes. Additionally, the new task involves both audio and visual modalities. Therefore, simply applying existing visual-only methods cannot fully exploit and preserve the inherent cross-modal semantic correlations. Very recently, Pian et al. [53] and Mo et al. [44] extended continual learning to the audio-visual domain, but both focused on classification tasks.

To address these challenges, in this paper, we propose a novel approach named ContAV-Sep ( Cont inual A udioV isual Sound Sep aration). Upon the framework, we introduce a novel Cross-modal Similarity Distillation Constraint (CrossSDC) to not only maintain the cross-modal semantic similarity through incremental tasks but also preserve previously learned knowledge of semantic similarity in old models to counter catastrophic forgetting. The CrossSDC is a generic constraint that can be seamlessly integrated into the training process of different audio-visual sound separators. To evaluate the effectiveness of our proposed ContAV-Sep, we conducted experiments on the MUSIC-21 dataset within the framework of continual learning, using the state-of-the-art audio-visual sound separation model iQuery [14] and a representative audio-visual sound separation model Co-Separation [23], as our separation base models. Experiments demonstrate that ContAV-Sep can effectively mitigate catastrophic forgetting and achieve significantly better performance than other continual learning baselines. In summary, this paper contributes follows:

(i) To explore more practical audio-visual sound separation, in which the separation model should be generalized to new sound source classes continually, we pose a Continual Audio-Visual Sound Separation task that trains the separation model under the setting of continual learning. To the best of our knowledge, this is the first work on continual learning for audio-visual sound separation.

(ii) We propose ContAV-Sep for the new task. It uses a novel cross-modal similarity distillation constraint to preserve cross-modal semantic similarity knowledge from previously learned models.

(iii) Experiments on the MUSIC-21 dataset can validate the effectiveness of our ContAV-Sep, demonstrating promising performance gain over baselines.

2 Related Work

Audio-Visual Sound Separation. Audio-visual sound separation aims to separate individual sound sources from an audio mixture guided by visual cues. A line of research emerges under various scenarios, such as separating musical instruments [23, 84, 79, 21, 83, 67], human speech [20, 1, 18, 49, 15], or sound sources in in-the-wild videos [22, 70]. Many frameworks and methods have been proposed to address challenges within specific problem settings. For instance, the extraction of face embeddings proves beneficial for speech audio separation [18]. Moreover, incorporating object detection can provide an additional advantage [23, 22]. The utilization of trajectory optical flows to leverage temporal motion information in videos, as demonstrated by [83], also yields improvements. In this work, not competing on designing stronger separators, we would advance the exploration of the audio-visual sound separation within the paradigm of continual learning. We investigate how a model can learn to consistently separate sound sources from sequential separation tasks without forgetting previously acquired knowledge.

Continual Learning. The field of continual learning has drawn significant attention, especially in visual domains, with various approaches addressing this challenge. Notable among these are regularization-based methods, exemplified in works such as [32, 3, 31, 39]. These approaches involve applying regularization to crucial parameters associated with old tasks to maintain the model's capabilities and during incremental steps, less important parameters are given higher priority for updates compared to important ones. Conversely, several works [57, 9, 5, 26, 12, 54, 8, 10, 40] applied rehearsal-based pipelines to enable the model review previously learned knowledge. For instance, Rebuffi et al. [57] proposed one of the most representative exemplars selection strategy Nearest-Meanof-Exemplars (NME), selects the most representative exemplars in each class based on the distance to the feature center of the class. Meanwhile, pseudo-rehearsal [47, 48, 66] employs generative models to create pseudo-exemplars based on the estimated distribution of data from previous classes. Moreover, architecture-based/dynamic architecture methods[52, 4, 46, 24, 28, 37, 40] proposed to modify the model architecture itself to enable the model to acquire new knowledge while mitigating the forgetting of old knowledge. Specifically, Pham et al. [52] proposed a dual network architecture, in which one is to learn new tasks while the other one is for retaining knowledge learned from old tasks. Wang et al. [72] combined the dynamic architecture and distillation constraint to mitigate the issue of continual-increasing overhead problem in dynamic architecture-based continual learning method. However, above studies mainly concentrate on the area of continual image classification. Recently, researchers also explored other continual learning scenarios beyond image classification. For instance, Park et al. [50] extend the knowledge distillation-based [2, 17] continual image classification method to the domain of video by proposing the time-channel distillation constraint. Douillard et al. [16] proposed to tackle the continual semantic segmentation task with multi-view feature distillation and pseudo-labeling. Xiao et al. [78] further addressed the continual semantic segmentation problem through weights fusion strategy between old and current models. Wang et al. [75] addressed the continual sound classification task through generative replay. Furthermore, continual learning has also been explored in the domain of language/vision-language learning tasks [30, 43, 59, 61, 19, 86], self-supervised representation learning [19, 42, 55, 80, 35, 74], audio classification [6, 73] and fake audio detection [41, 82], etc. Despite the success of existing continual learning methods in various scenarios, their applicability in the domain of continual audio-visual sound separation is still unexplored. Although Pian et al. [53] and Mo et al. [44] proposed to tackle the catastrophic problem in audio-visual learning, their studies mainly concentrated in the area of audio-visual video classification. In contrast to existing works in continual learning, in this paper, we delves into the continual audio-visual sound separation, aiming to tackle the challenge of catastrophic forgetting specifically in the context of separation mask prediction for complicated mixed audio signals within joint audio-visual modeling.

3 Method

3.1 Problem Formulation

Audio-Visual Sound Separation. Audio-visual sound separation aims to separate distinctive sound signals according to the given associated visual guidance. Following previous works [14, 23, 67, 21, 79], we adopt the common 'mix-and-separation' training strategy to train the model. Given two videos V 1 ( s 1 , v 1 ) and V 2 ( s 2 , v 2 ) , we can obtain the input mixed sound signal S by mixing two

video sound signals s 1 and s 2 , and then we can have the ratio masks M 1 = s 1 / S and M 2 = s 2 / S 1 . The goal of the task is to utilize the corresponding visual guidance v 1 and v 2 to predict the ratio masks for reconstructing the two individual audio signals. This process can be formulated as:

where F Θ is the separation model with trainable parameters Θ . And then, the original sound signals s 1 and s 2 are used to calculate the loss function for optimizing the model:

where D denotes the training set, and L is the loss function between the prediction and ground-truth.

Continual Audio-Visual Sound Separation. Our proposed continual audio-visual sound separation task aims to train a model F Θ continually on a sequence of T separation tasks {T 1 , T 2 , ..., T T } . For the t -th task T t (incremental step t ), we have a training set D t = { V i ( s i , v i ) , y i t } n t i =1 , where i and n t denote the i -th video sample and the total number of samples in D t respectively, and y i t ∈ C t is the corresponding sound source class of video V i , where C t is the training sound class label space of task T t . For any two tasks T t 1 and T t 2 and their corresponding training sound class label space C t 1 and C t 2 , we have C t 1 ∩ C t 2 = ∅ . Following previous works in continual learning [57, 2, 53, 44, 29, 76], for a task T t , where t > 1 , holding a small size of memory/exemplar set M t to store some data from old tasks is permitted in our setting. Therefore, with the memory/exemplar set M t , all available data that can be used for training in task T t ( t > 1 ) can be denoted as D ′ t = D t ∪M t . Finally, the training process of Eq. 2 in our continual audio-visual sound separation setting can be denoted as:

which means that the new model Θ t is obtained by updating the old model Θ t -1 which was trained on the previous task, using the current task's available data D ′ t . After the training process for task T t with D ′ t , the updated model will be evaluated on a testing set which includes video samples from all seen sound source classes up to continual step t (task T t ). And the evaluation also follows the common 'mix-and-separation' strategy. During this continual learning process, the model's separation performance on the previously learned tasks drops significantly after training on new tasks. This learning issue is referred to as the catastrophic forgetting [32, 38, 3] problem, which poses a considerable challenge in continual audio-visual sound separation.

3.2 Overview

To address the challenge of catastrophic forgetting in continual audio-visual sound separation, we introduce ContAV-Sep . This new framework, illustrated in Fig. 2, consists of three key components: a separation base model, an output mask distillation module, and our proposed Cross-modal Similarity Distillation Constraint (CrossSDC) . We use a recent state-of-the-art audio-visual separator: iQuery [14] as the base model of our approach, which contains a video encoder to extract the global motion feature, an object detector and image encoder to obtain the object feature, a U-Net [58] for mixture sound encoding and separated sound decoding, and an audio-visual Transformer to get the separated sound feature through multi-modal cross-attention mechanism and class-aware audio queries. For the object detector, we follow iQuery [14] and use the pre-trained Detic [87], a universal object detector, to detect the sound source objects in each frame. For the video encoder and the image encoder, inspired by the excellent generalization ability of recent self-supervised pre-trained models, which has been proven to be effective and appropriate in continual learning [53], we apply two self-supervised pre-trained models VideoMAE [69] and CLIP [56] as the video encoder and the image encoder, respectively. Note that, during the training process, the object detector, video encoder, and image encoder are frozen.

ˆ

Video

Encoder

Object

Detector

&

Encoder

Audio-Viual

Transformer

&

Learnable

Audio Query

Audio

Encoder

Audio

Decoder

Projection

MLP

/circledot

f

a

t

f

m

t

f

o

t

Cross-modal

Contrastive Loss

…

…

…

Cross-task Cross-modal Contrastive Loss

Cross-modal Similarity Distillation Constraint

f

a

t

-

1

,

1

f

a

t

-

1

,n

f

m

t

-

1

,n

f

m

t

-

1

,

1

f

o

t

-

1

,

1

f

o

t

-

1

,n

Output Mask Distillation

Projection

Video

Encoder

Object

Detector

&

Encoder

Audio-Viual

Transformer

&

Learnable

Audio Query

Audio

Encoder

Audio

Decoder

Projection

MLP

/circledot

Projection

F

Θ

t

F

Θ

t

-

1

f

m

t

-

1

f

a

t

-

1

f

o

t

-

1

❄

❄

❄

🔥

🔥

🔥

🔥

…

…

…

🔥

🔥

f

m

t,

1

f

a

t,

1

f

a

t,n

f

o

t,

1

f

m

t,n

f

o

t,n

STFT

Mix Audio

/circledot

iSTFT

Separated Audio

M

t

-

1

ˆ

M

t

ˆ

M

t

-

1

ˆ

M

t

(

ˆ

M

t

,

ˆ

M

t

1 )

L

-

Given a pair of videos V 1 ( s 1 , v 1 ) and V 2 ( s 2 , v 2 ) , at incremental step t (task T t ), the U-Net audio encoder F AE t takes the mixed audio signal S obtained by mixing s 1 and s 2 as input, and generates the latent mixed audio feature. This process can be expressed as:

Then, the audio-visual Transformer F Trans. t is employed to generate the separated sound feature by taking the latent mixed audio feature and visual features as inputs:

where f a, 1 t denotes the separated sound feature of video V 1 ; Obj. 1 and Mo. 1 denote the object and motion features extracted by the frozen pre-trained image and video encoders respectively from the visual signal v 1 of video V 1 , U o t ( · ) and U m t ( · ) are learnable projection layers to map the object and motion features into the same dimension. Similarly, we can also obtain the separated sound feature of V 2 guided by the associated visual features.

The extracted separated sound feature and the latent mixed audio feature are combined to generate a mask. This mask is subsequently applied to the mixed audio, leading to the reconstruction of the separated sound spectrogram.

where ˆ M 1 t and ˆ M 2 t denote the predicted masks for audio signals of video V 1 and V 2 , respectively; F AD t is the U-Net decoder at incremental step t ; MLP t ( · ) denotes a MLP module; and ⊙ denotes channel-wise multiplication. The sound s 1 at this incremental step can be reconstructed by applying S ⊙ ˆ M 1 t and then performing an inverse STFT to obtain the audio waveform.

3.3 Cross-modal Similarity Distillation Constraint

Recent studies [53, 45] have highlighted the importance of cross-modal semantic correlation in audio-visual modeling. However, this correlation tends to diminish during subsequent incremental phases, which leads to catastrophic forgetting in our continual audio-visual sound separation task. To address this challenge, we propose a novel Cross-modal Similarity Distillation Constraint (CrossSDC)

that serves two crucial purposes (1) maintaining cross-modal semantic similarity through incremental tasks, and (2) preserving previous learned semantic similarity knowledge from old tasks.

CrossSDC preserves cross-modal semantic similarity from two perspectives: instance-aware semantic similarity and class-aware semantic similarity. Both similarities are enforced by integrating contrastive loss and knowledge distillation. Instead of exclusively focusing on the similarities within current and memory data generated by the current training model, CrossSDC incorporates the cross-modal similarity knowledge acquired from previous tasks into the contrastive loss. This integration not only facilitates the learning of cross-modal semantic similarities in new tasks but also ensures the preservation of previously acquired knowledge. In the incremental step t ( t > 1 ), the instance-aware part of our CrossSDC can be formulated as:

where ✶ [ i = j ] is an indicator that equals 1 when i = j , denoting that video samples V i and V j are the same video; The sim function represents the cosine similarity function with temperature scaling; The modalities mod 1 and mod 2 , where ( mod 1 , mod 2 ) ∈ { ( a, o ) , ( a, m ) , ( m,o ) } , denote different pairs of features to be compared: separated sound and object features, sound and motion features, and motion and object features. Here, τ denotes the incremental step, for which we have:

which means that, for current task's data D t , we calculate the contrastive loss using features from the current model ( τ 1 = τ 2 = t ), while for memory set data M t , we use features from both the old and current models ( e.g., τ 1 = t and τ 2 = t -1 ). In this way, knowledge distillation would be integrated into the cross-modal semantic similarity constraint for the current task, which ensures better preservation of learned cross-modal semantic similarity from previous tasks.

While the instance-aware similarity provides valuable semantic correlation modeling, it does not account for the class-level semantic correlations, which is also crucial for audio-visual similarity modeling. To capture and preserve the semantic similarity within each class across incremental tasks, we also incorporate a class-aware component specifically designed for inter-class cross-modal semantic similarity, which can be formulated as:

✶ In this context, visual and audio features from two videos are encouraged to be close when they belong to the same class. The overall formulation of our CrossSDC is as follows:

where λ ins and λ cls are two scalars that balance the two loss terms. In this way, the model captures and preserves semantic correlations not just between instances but also within the same classes.

3.4 Overall Loss Function

In the previous subsection, we introduced our proposed CrossSDC constraint. To effectively combine CrossSDC with the overall objective, we incorporate it alongside output distillation and the main separation loss function.

Output distillation is a widely used technique in continual learning [38, 2, 53] to preserve the knowledge gained from previous tasks while learning new ones. In our approach, we utilize the output of the old model as the distillation target to preserve this knowledge. Note that we only distill knowledge for data from the memory set, as represented by:

where ˆ M 1 t -1 and ˆ M 2 t -1 are predicted masks generated by the old model that is trained at incremental step t -1 . For the loss function here, we follow [84, 14] and adopt the per-pixel L 1 loss [84]. For

the main separation loss function, we also apply the per-pixel L 1 loss:

Finally, our overall loss function is denoted as:

3.5 Management of Memory Set

In alignment with the work of [76], our framework maintains a compact memory set throughout incremental updates. Each old class is limited to a maximum number of exemplars. After completing training for each task, we adopt the exemplar selection strategies in [2, 53] by randomly selecting exemplars for each current class and combining these new exemplars with the existing memory set.

4 Experiments

In this section, we first introduce the setup of our experiments, i.e. , dataset, baselines, evaluation metrics, and the implementation details. After that, we present the experimental results of our ContAV-Sep compared to the baselines, as well as ablation studies. We also conduct experiments on the AVE [68] and the VGGSound [13] datasets, which contain sound categories beyond the music domain. We put the experimental results on the A VE and the VGGSound datasets, the comparison to the uni-modal semantic similarity preservation method, the performance evaluation on old classes in incremental tasks, and the visualization of separating results in the Appendix.

4.1 Experimental Setup

Dataset. Following common practice [83, 88, 14], we conducted experiments on MUSIC-21 [83], which contains solo videos of 21 instruments categories: accordion, acoustic guitar, cello, clarinet, erhu, flute, saxophone, trumpet, tuba, violin, xylophone, bagpipe, banjo, bassoon, congas, drum, electric, bass, guzheng, piano, pipa, and ukulele. In our experiments, we randomly selected 20 of them to construct the continual learning setting. Specifically, we split the selected 20 classes into 4 incremental tasks, each of which involves 5 classes. The total number of available videos is 1040, and we randomly split them into training, validation, and testing sets with 840, 100, and 100 videos, respectively. To further validate the efficacy of our method across a broader sound domain, we conduct experiments using the AVE [68] and the VGGSound [13] datasets in the appendix.

Baselines. We compare our proposed approach with vanilla Fine-tuning strategy, and continual learning methods EWC [32] and LwF [38]. As we mentioned before, typical continual learning methods, e.g. , class-incremental learning methods, which yield progressively increasing logits (or probability distribution) across all observed classes at each incremental step and design specific technique in the classifier, we consider that these methods are not an optimal choice for our proposed continual audio-visual sound separation problem. Thus, considering that continual semantic segmentation problem has a more similar form compared to conventional class-incremental learning, we also select two state-of-the-art continual semantic segmentation methods PLOP [16] and EWF [78] as our baselines. Moreover, we compare our method to the recently proposed audio-visual continual learning method AV-CIL [53], in which we adapt the original class-incremental version to the form of continual audio-visual sound separation by replacing their task-wise logits distillation with the output mask distillation. Further, we also present the experimental results of the Oracle/Upper Bound, which means that using the training data from all seen classes to train the model. For fair comparison, all compared continual learning methods and our ContAV-Sep use the same state-of-the-art separator, i.e. iQuery [14], as the base separation model . Further, we also incorporate our proposed and baseline methods into another representative audio-visual sound separation model Co-Separation [23]. Notably, the Co-Separation model does not utilize the motion modality. Therefore, when CrossSDC is applied to Co-Separation, the ( mod 1 , mod 2 ) in Eq. 7 and 9 is constrained to ( mod 1 , mod 2 ) = ( a, o ) . For baselines that involve memory sets, we ensure that each of them is allocated the same number of memory as our proposed method for fair comparison.

Implementation Details. Following [14], we use a 7-layers U-Net [58] as the audio net, and subsample the audio at 11kHz, each of which is approximately 6 seconds. We apply the STFT with

the Hann window size of 1022 and the hop length of 256, to obtain the 512 × 256 Time-Frequency representation of each audio signal, followed by a re-sampling on the log-frequency scale to generate the magnitude spectrogram with T, F = 256 . We set the video frame rate (FPS) to 1, and detect the object using the pre-trained universal detector Detic [87] to detect the sound source object on each frame, and then, each detected object is resized and randomly cropped to the size of 224 × 224 . For the image encoder and the video encoder, we apply the self-supervised pre-trained CLIP [56] and VideoMAE [69] to yield the object feature and motion feature, respectively. For the audio-visual Transformer module, we follow the design in [14]. For all the baseline methods, we apply the same model architecture and modules with ours for them, including the mentioned Detic, CLIP, VideoMAE, audio-visual Transformer, etc. Please note that, during our training process, the pre-trained Detic, CLIP, and VideoMAE are frozen. In our proposed Cross-modal Similarity Distillation Constraint (CrossSDC), the balance weights λ ins and λ cls are set to 0.1 and 0.3, respectively. And the balance weight λ dist. for the output distillation loss is set to 0.3 in our experiments. For the memory set, we set the number of samples in each old class to 1, so as other baselines that involve the memory set. All the experiments in this paper are implemented by Pytorch [51]. We train our proposed method and all baselines on a NVIDIA RTX A5000 GPU. We follow previous works [67, 14] in sound separation, and evaluate the performance of all the methods using three common metrics in sound separation tasks: Signal to Distortion Ratio (SDR), Signal to Interference Ratio (SIR), and Signal to Artifact Ratio (SAR). The SDR measures the interference and artifacts, while SIR and SAR measure the interference and artifacts, respectively. In our experiments, we report the SDR, SIR, and SAR of all the methods after training at last incremental steps, i.e. , testing results on all classes. For all these three metrics, higher values denote better results.

4.2 Experimental Comparison

The main experimental comparisons are shown in Tab. 1. Our proposed method, ContAV-Sep, outperforms the state-of-the-art baselines by a substantial margin. Notably, compared to baselines using state-of-the-art audio-visual sound separator iQuery [14] as the separation base model, ContA VSep achieves a 0.3 improvement in SDR over the compared best-performing method. Additionally, our method surpasses the top baseline by 0.25 in SIR and 0.41 in SAR. Furthermore, compared to continual learning baselines with Co-Separation [23], our ContAV-Sep still outperforms other approaches. This consistent superior performance across different model architectures highlights not only the effectiveness but also the broad applicability and generalizability of our proposed CrossSDC.

Our observations further demonstrate that retaining a small memory set significantly enhances the performance of each baseline method. For instance, for the iQuery-based continual learning methods, equipping LwF [38] with a small memory set results in improvements of 3.31, 3.99, and 1.94 on SDR, SIR, and SAR, respectively. Similarly, the addition of a small memory set to EWC [32] leads to enhancements of 2.98, 3.43, and 1.43 in the respective metrics. The memory-augmented version of PLOP [16] exhibits superior performance with margins of 3.21, 3.24, and 1.68 for SDR, SIR, and SAR, respectively. Finally, incorporating memory into EWF [78] results in improvements of 1.37,

5

10

15

20

Classes

3

4

5

6

7

8

SDR

Fine-tuning

LwF w/o memory

EWC w/o memory

PLOP w/o memory

EWF w/o memory

LwF w/ memory

EWC w/ memory

PLOP w/ memory

EWF w/ memory

Ours

(a)

5

10

15

20

Classes

10

12

14

16

SIR

Fine-tuning

LwF w/o memory

EWC w/o memory

PLOP w/o memory

EWF w/o memory

LwF w/ memory

EWC w/ memory

PLOP w/ memory

EWF w/ memory

Ours

(b)

5

10

15

20

Classes

9

10

11

12

13

14

SAR

Fine-tuning

LwF w/o memory

EWC w/o memory

PLOP w/o memory

EWF w/o memory

LwF w/ memory

EWC w/ memory

PLOP w/ memory

EWF w/ memory

Ours

(c)

1.67, and 0.29 for the three metrics. This phenomenon can be attributed to the inherent nature of the sound separation training process. In training, the audio signal from each sample mixes with others, giving a composite audio signal. This mixed audio signal, coupled with the corresponding visual data pair for each separated audio, constitutes the actual training sample for the separation task. As a result, even a single memory sample can be associated with multiple samples from the current training set, generating a diverse array of effective training pairs.

We also present the testing results of SDR, SIR, and SAR at each incremental step in Figures 3a, 3b, and 3c, respectively. Our method is consistently observed to outperform others in terms of SDR at all incremental steps. While our approach may not always produce the best SIR and SAR results at the intermediate steps (specifically, steps 2 and 3 for SIR, and step 3 for SAR), it ultimately achieves the highest performance at the final step. This demonstrates the robustness of our method, indicating minimal forgetting throughout the incremental learning process.

4.3 Ablation Study on CrossSDC and Memory Size

In this subsection, we conduct an ablation study to investigate the effectiveness of our proposed CrossSDC. By removing single or multiple components of the CrossSDC, we evaluate the impact of each on the final results. The results of the ablation study are presented in Tab. 2. From the table, we can see that our full model achieves the best performance compared to the variants, which further demonstrates the effectiveness of our proposed CrossSDC.

Moreover, we also discuss the effect of memory size on our proposed ContAV-Sep. In our main experiments, the default setting of the memory size is 1 sample per old class. In this subsection, we conduct experiments by increasing the memory size from 1 sample per old class to 30 samples per old class. The experimental results are shown in Tab. 3 and Figure 4. Observations from the table indicate a positive correlation between the size of the memory and the overall performance metrics. As the memory size increases, there is a discernible trend of improvement in the results.

4.4 Limitation and Discussion

Our experimental findings reveal that the utilization of a small memory set, even a single sample per old class, markedly improves the performance of each baseline method. This improvement is attributed to the ability of a single memory sample to pair with diverse samples from the current training set, thereby generating numerous effective training pairs. Consequently, this process enables the model to acquire new knowledge for old classes in subsequent tasks, as the memory data can be

0

5

10

15

20

25

30

# of samples per class

6

7

8

9

10

SDR

(a)

0

5

10

15

20

25

30

# of samples per class

13

14

15

16

17

SIR

(b)

0

5

10

15

20

25

30

# of samples per class

12.0

12.5

13.0

13.5

14.0

14.5

SAR

(c)

paired with data from previously unseen new classes - this is different from conventional continual learning tasks, where old classes do not acquire new knowledge in new tasks. This could be a potential reason why the baseline continual learning methods do not perform well in our continual audio-visual sound separation problem. In this work, our method also mainly focuses on preserving old knowledge of old tasks, which may prevent the model from acquiring new knowledge of old classes when training in new tasks. Recognizing this, we identify the exploration of this problem as a key avenue for future research in this field.

Additionally, the base model architectures used in our approach and baselines require object detectors to identify sounding objects. Although iQuery [14] can supplement object features with global video representations, it may still suffer from undetected objects. It is a fundamental limitation of the object-based audio-visual sound separators [23, 14]. While our work, unlike previous efforts, does not compete on designing a stronger audio-visual separation base model, enhancing the robustness of sounding object detection presents a promising direction for future research.

5 Conclusion

In this paper, we explore training audio-visual sound separation models under a more practical continual learning scenario, and introduce the task of continual audio-visual sound separation. To address this novel problem, we propose ContAV-Sep, which incorporates a Cross-modal Similarity Distillation Constraint to maintain cross-modal semantic similarity across incremental tasks while preserving previously learned semantic similarity knowledge. Experiments on the MUSIC-21 dataset demonstrate the effectiveness of our method in this new continual separation task. This paper opens a new direction for real-world audio-visual sound separation research.

Broader Impact. Our proposed continual audio-visual sound separation allows the model to adapt to new environments and sounds without full retraining, which could enhance efficiency and privacy by reducing the need to transmit and store sensitive audio data.

Acknowledgments. We thank the anonymous reviewers and area chair for their valuable suggestions and comments. This work was supported in part by a Cisco Faculty Research Award, an Amazon Research Award, and a research gift from Adobe. The article solely reflects the opinions and conclusions of its authors but not the funding agents.

References

Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. The conversation: Deep audio-visual speech enhancement. arXiv preprint arXiv:1804.04121 , 2018.

Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. SSIL: separated softmax for incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 824-833, 2021.

Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV) , pages 139-154, 2018.

Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual learning method based on complementary learning system. arXiv preprint arXiv:2201.12604 , 2022.

Eden Belouadah and Adrian Popescu. Il2m: Class incremental learning with dual memory. In Proceedings of the IEEE/CVF international conference on computer vision , pages 583-592, 2019.

Ruchi Bhatt, Pratibha Kumari, Dwarikanath Mahapatra, Abdulmotaleb El Saddik, and Mukesh Saini. Characterizing continual learning scenarios and strategies for audio analysis. arXiv preprint arXiv:2407.00465 , 2024.

Albert S Bregnian. Auditory scene analysis: Hearing in complex environments. Thinking in Sounds , pages 10-36, 1993.

Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems , 33:15920-15930, 2020.

Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 233-248, 2018.

Sungmin Cha, Sungjun Cho, Dasol Hwang, Sunwon Hong, Moontae Lee, and Taesup Moon. Rebalancing batch normalization for exemplar-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20127-20136, 2023.

Moitreya Chatterjee, Narendra Ahuja, and Anoop Cherian. Learning audio-visual dynamics using scene graphs for audio source separation. Advances in Neural Information Processing Systems , 35:16975-16988, 2022.

Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc'Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486 , 2019.

Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 721-725. IEEE, 2020.

Jiaben Chen, Renrui Zhang, Dongze Lian, Jiaqi Yang, Ziyao Zeng, and Jianbo Shi. iquery: Instruments as queries for audio-visual sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14675-14686, 2023.

Soo-Whan Chung, Soyeon Choe, Joon Son Chung, and Hong-Goo Kang. Facefilter: Audiovisual speech separation using still images. arXiv preprint arXiv:2005.07074 , 2020.

Arthur Douillard, Yifu Chen, Arnaud Dapogny, and Matthieu Cord. Plop: Learning without forgetting for continual semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4040-4050, 2021.

Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16 , pages 86-102. Springer, 2020.

Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speakerindependent audio-visual model for speech separation. arXiv preprint arXiv:1804.03619 , 2018.

Enrico Fini, Victor G Turrisi Da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models are continual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9621-9630, 2022.

Aviv Gabbay, Asaph Shamir, and Shmuel Peleg. Visual speech enhancement. arXiv preprint arXiv:1711.08789 , 2017.

Chuang Gan, Deng Huang, Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10478-10487, 2020.

Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 35-53, 2018.

Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3879-3888, 2019.

Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. arXiv preprint arXiv:1903.04476 , 2019.

Simon Haykin and Zhe Chen. The cocktail party problem. Neural computation , 17(9):18751902, 2005.

Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 831-839, 2019.

Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Davis: High-quality audio-visual separation with generative diffusion models. arXiv preprint arXiv:2308.00122 , 2023.

Ching-Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and ChuSong Chen. Compacting, picking and growing for unforgetting continual learning. Advances in Neural Information Processing Systems , 32, 2019.

Minsoo Kang, Jaeyoo Park, and Bohyung Han. Class-incremental learning by knowledge distillation with adaptive feature consolidation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16050-16059, 2022.

Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual learning of language models. In International Conference on Learning Representations (ICLR) , 2023.

Sanghwan Kim, Lorenzo Noci, Antonio Orvieto, and Thomas Hofmann. Achieving a better stability-plasticity trade-off via auxiliary networks in continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11930-11939, 2023.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 114(13):3521-3526, 2017.

Saksham Singh Kushwaha. Analyzing the effect of equal-angle spatial discretization on sound event localization and detection. In Proceedings of the 7th Detection and Classification of Acoustic Scenes and Events 2022 Workshop (DCASE2022) , 2022.

Saksham Singh Kushwaha and Magdalena Fuentes. A multimodal prototypical approach for unsupervised sound classification. arXiv preprint arXiv:2306.12300 , 2023.

Jaewoo Lee, Jaehong Yoon, Wonjae Kim, Yunji Kim, and Sung Ju Hwang. Lifelong audio-video masked autoencoder with forget-robust localized alignments. arXiv preprint arXiv:2310.08204 , 2023.

Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 312-321, 2019.

Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning , pages 3925-3934. PMLR, 2019.

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence , 40(12):2935-2947, 2017.

Yan-Shuo Liang and Wu-Jun Li. Adaptive plasticity improvement for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7816-7825, 2023.

Zilin Luo, Yaoyao Liu, Bernt Schiele, and Qianru Sun. Class-incremental exemplar compression for class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11371-11380, 2023.

Haoxin Ma, Jiangyan Yi, Jianhua Tao, Ye Bai, Zhengkun Tian, and Chenglong Wang. Continual learning for fake audio detection. arXiv preprint arXiv:2104.07286 , 2021.

Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. In International Conference on Learning Representations (ICLR) , 2022.

Fei Mi, Liangwei Chen, Mengjie Zhao, Minlie Huang, and Boi Faltings. Continual learning for natural language generation in task-oriented dialog systems. In Findings of the Association for Computational Linguistics: EMNLP 2020 , volume EMNLP 2020, pages 3461-3474, 2020.

Shentong Mo, Weiguo Pian, and Yapeng Tian. Class-incremental grouping network for continual audio-visual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7788-7798, 2023.

Shentong Mo and Yapeng Tian. Audio-visual grouping network for sound localization from mixtures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10565-10574, 2023.

Xing Nie, Shixiong Xu, Xiyan Liu, Gaofeng Meng, Chunlei Huo, and Shiming Xiang. Bilateral memory consolidation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16026-16035, 2023.

Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning , pages 2642-2651. PMLR, 2017.

Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11321-11329, 2019.

Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European conference on computer vision (ECCV) , pages 631-648, 2018.

Jaeyoo Park, Minsoo Kang, and Bohyung Han. Class-incremental learning for action recognition in videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13698-13707, 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems , 34:16131-16144, 2021.

Weiguo Pian, Shentong Mo, Yunhui Guo, and Yapeng Tian. Audio-visual class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7799-7811, 2023.

Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 524-540. Springer, 2020.

Senthil Purushwalkam, Pedro Morgado, and Abhinav Gupta. The challenges of continuous self-supervised learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI , pages 702-721. Springer, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2001-2010, 2017.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 6107-6122, 2022.

Charles Spence. Audiovisual multisensory integration. Acoustical science and technology , 28(2):61-70, 2007.

Tejas Srinivasan, Ting-Yun Chang, Leticia Leonor Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. CLiMB: A continual learning benchmark for vision-andlanguage tasks. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Barry E Stein and Terrence R Stanford. Multisensory integration: current issues from the perspective of the single neuron. Nature reviews neuroscience , 9(4):255-266, 2008.

Yiyang Su, Ali Vosoughi, Shijian Deng, Yapeng Tian, and Chenliang Xu. Separating invisible sounds toward universal audiovisual scene-aware sound separation. arXiv preprint arXiv:2310.11713 , 2023.

Elyse S Sussman. Integration and segregation in auditory scene analysis. The Journal of the Acoustical Society of America , 117(3):1285-1298, 2005.

Reuben Tan, Arijit Ray, Andrea Burns, Bryan A Plummer, Justin Salamon, Oriol Nieto, Bryan Russell, and Kate Saenko. Language-guided audio-visual source separation via trimodal consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10575-10584, 2023.

Yu-Ming Tang, Yi-Xing Peng, and Wei-Shi Zheng. Learning to imagine: Diversify memory for incremental learning using unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9549-9558, 2022.

Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2745-2754, 2021.

Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European conference on computer vision (ECCV) , pages 247-263, 2018.

Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems , 35:10078-10093, 2022.

Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel PW Ellis, and John R Hershey. Into the wild with audioscope: Unsupervised audio-visual separation of on-screen sounds. arXiv preprint arXiv:2011.01143 , 2020.

Efthymios Tzinis, Scott Wisdom, Tal Remez, and John R Hershey. Audioscopev2: Audio-visual attention architectures for calibrated open-domain on-screen sound separation. In European Conference on Computer Vision , pages 368-385. Springer, 2022.

Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. FOSTER: feature boosting and compression for class-incremental learning. In In Proceedings of the European Conference on Computer Vision (ECCV) , pages 398-414, 2022.

Yu Wang, Nicholas J Bryan, Mark Cartwright, Juan Pablo Bello, and Justin Salamon. Fewshot continual learning for audio classification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 321-325. IEEE, 2021.

Zhepei Wang, Cem Subakan, Xilin Jiang, Junkai Wu, Efthymios Tzinis, Mirco Ravanelli, and Paris Smaragdis. Learning representations for new sound classes with continual self-supervised learning. IEEE Signal Processing Letters , 29:2607-2611, 2022.

Zhepei Wang, Cem Subakan, Efthymios Tzinis, Paris Smaragdis, and Laurent Charlin. Continual learning of new sound classes using generative replay. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) , pages 308-312. IEEE, 2019.

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 139-149, 2022.

Adam Weisser. Complex acoustic environments: Concepts, methods, and auditory perception . PhD thesis, PhD thesis). Macquarie University, Sydney, Australia. doi: 1959.14/1266534, 2018.

Jia-Wen Xiao, Chang-Bin Zhang, Jiekang Feng, Xialei Liu, Joost van de Weijer, and Ming-Ming Cheng. Endpoints weight fusion for class incremental semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7204-7213, 2023.

Xudong Xu, Bo Dai, and Dahua Lin. Recursive visual sound separation using minus-plus net. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 882-891, 2019.

Shipeng Yan, Lanqing Hong, Hang Xu, Jianhua Han, Tinne Tuytelaars, Zhenguo Li, and Xuming He. Generative negative text replay for continual vision-language pretraining. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI , pages 22-38. Springer, 2022.

Yuxin Ye, Wenming Yang, and Yapeng Tian. Lavss: Location-guided audio-visual spatial audio separation. arXiv preprint arXiv:2310.20446 , 2023.

Xiaohui Zhang, Jiangyan Yi, Jianhua Tao, Chenglong Wang, and Chu Yuan Zhang. Do you remember? overcoming catastrophic forgetting for fake audio detection. In International Conference on Machine Learning , pages 41819-41831. PMLR, 2023.

Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proceedings of the IEEE International Conference on Computer Vision , pages 1735-1744, 2019.

Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In The European Conference on Computer Vision (ECCV) , September 2018.

Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class-incremental learning: A survey. arXiv preprint arXiv:2302.03648 , 2023.

Da-Wei Zhou, Yuanhan Zhang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for vision-language models. arXiv preprint arXiv:2305.19270 , 2023.

Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vision , pages 350-368. Springer, 2022.

Lingyu Zhu and Esa Rahtu. Visually guided sound source separation and localization using self-supervised motion representations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1289-1299, 2022.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in Sec. 4.4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not contain theoretical results or proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include an implementation paragraph section in Sec. 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our source code has been released. Datasets used in our experiments are public dataset.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include an implementation paragraph section in Sec. 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Because of the limitation of computation resource, we didn't report the error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides sufficient information of the computer resources needed to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our paper does not violate any respect of the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses the potential societal impacts. We include it in Sec. 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are introduced in the paper

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel continual audio-visual sound separation task, aimed at continuously separating new categories of sound sources while maintaining the performance of previously learned categories.\n\nSTRENGTHS:\n1. The structure of the entire paper is clear, and the expression is fluent.\n2. The experimental results can demonstrate the effectiveness of the method, and the quality of the separated audio shown in the visual video is also good.\n\nWEAKNESSES:\n1. The Cross-modal Similarity Distillation Constraint proposed in the paper includes two main innovations: 1) instance-aware semantic similarity and 2) class-aware semantic similarity. However, in terms of mathematical expression, compared to the Dual-Audio-Visual Similarity Constraint in reference [1], I believe the innovations of the two papers are almost identical.\n[1] Pian W, Mo S, Guo Y, et al. Audio-visual class-incremental learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 7799-7811.\n2. Although the paper has conducted experimental validation on the MUSIC-21 dataset, it does not provide sufficient information to assess the model's generalization ability on other datasets or in real-world scenarios, making it difficult to demonstrate its enhanced applicability in real scenarios where new sound sources are encountered.\n\nQUESTIONS:\n1. In Fig. 1, the authors illustrate the continual audio-visual sound separation task. Is the significant performance gap between ContAV-Sep and the fine-tuning method due to the small data scale and limited number of categories?\n\n2. I am curious whether this paper has been submitted twice, as I noticed the \"Anonymous ECCV Submission Paper ID 6024\" in the uploaded video materials. If that is the case, this paper will be directly rejected.\n\n3. Please clarify the difference between the innovation of the Cross-modal Similarity Distillation Constraint in this paper and the Dual-Audio-Visual Similarity Constraint in [1].\n\nLIMITATIONS:\nThe innovation of the Cross-modal Similarity Distillation Constraint in this paper may overlap with the Dual-Audio-Visual Similarity Constraint from the previous work. It suggests that while the paper introduces a novel approach, there might not be a clear distinction or significant difference from the existing method in terms of the core innovative aspects.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel task termed \"Continual Audio-Visual Sound Separation,\" aiming to address the practical challenge of separating sound sources for new classes in audio-visual scenarios while retaining performance on previously learned classes. This task is inherently challenging due to the inherent risk of catastrophic forgetting, where models trained on new data often exhibit performance degradation on previously learned classes. To tackle this challenge, the authors propose ContAV-Sep, a novel framework incorporating a Cross-modal Similarity Distillation Constraint (CrossSDC). This constraint preserves cross-modal semantic similarity across incremental tasks by enforcing both instance-aware and class-aware similarities through a combination of contrastive loss and knowledge distillation. Notably, CrossSDC integrates knowledge from past tasks into the contrastive learning process, ensuring the retention of previously acquired cross-modal correlations. Experiments on the MUSIC-21 dataset demonstrate that ContAV-Sep significantly outperforms existing continual learning baselines in terms of standard sound separation metrics (SDR, SIR, SAR) across multiple audio-visual sound separation base models. This work highlights the importance of cross-modal semantic correlation in continual audio-visual learning and provides a novel, effective solution for mitigating catastrophic forgetting in this domain.\n\nSTRENGTHS:\nHere are four strengths of the paper, presented as a list:\n- Clearly identifies and addresses a novel and practical problem in audio-visual sound separation: continual learning in this domain. It is important to mention that this might be an important problem in the field and not so many people have provided good solutions for this problem.\n- Proposes a novel framework, ContAV-Sep,  with a well-defined Cross-modal Similarity Distillation Constraint (CrossSDC) to tackle catastrophic forgetting.\n- Empirically demonstrates the effectiveness of ContAV-Sep on the MUSIC-21 dataset, showcasing significant performance improvements over strong baselines.\n- Provides a thorough analysis of the results, including ablation studies and exploration of memory size effects, highlighting the contributions of different components of the proposed method.\n\nWEAKNESSES:\n- Please replace the “mask” variable with something appropriate like \\widehat{\\mathbf{m}}\n- Please fix weird fontsizes like the one in equation 9, in general the manuscript does not seem polished enough for a NeurIPS submission.\n- Differences smaller than <0.1 dB in terms of SNR metrics are not either significant nor hearable (I would even argue for 0.5 dB but let’s follow the literature in this one), thus I would suggest rounding up all those performance numbers to a one decimal precision (it would also make the Tables less cluttered).\n- I would like to see an even larger ablation in Table 3 to show the full extent of how the performance deviates with an even larger amount of samples per class and not only 4, a graph would make the visualization better here cause currently it does not convey any meaningful message. \n- The authors do not make a thorough investigation on previous works in the literature that employ continual learning techniques for related sound processing tasks. I will refer here only a couple of the works that I am aware of like in [A, B] but I am almost certain that there is no lack of thereof to try to include other works and try to make some empirical or theoretical conclusions on how those methods can become interconnected, employed together and in general how they relate.\n\n\n\n[A] Wang, Z., Subakan, C., Tzinis, E., Smaragdis, P. and Charlin, L., 2019, October. Continual learning of new sound classes using generative replay. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) (pp. 308-312). IEEE.\n\n[B] Wang, Z., Subakan, C., Jiang, X., Wu, J., Tzinis, E., Ravanelli, M. and Smaragdis, P., 2022. Learning representations for new sound classes with continual self-supervised learning. IEEE Signal Processing Letters, 29, pp.2607-2611.\n\nQUESTIONS:\nThe classes seem a bit limited, would the authors want to generalize their experiments to a larger dataset like audioset and show how their method performs there? I think that would be even more interesting since the distribution of classes in Audioset has a long tail for multiple not so prevalent sounds (e.g. consider sounds that are not represented by Speech or music)\n\nLIMITATIONS:\nI think the authors include several limitations of their work, they should also identify the memorization of individual user’s data inside the memory of their continual learning method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
The paper introduces ContAV-Sep, the goal is to continuously separate new sound classes while maintaining performance on previously learned classes, addressing the challenge of catastrophic forgetting in continual learning. ContAV-Sep employs a Cross-modal Similarity Distillation Constraint (CrossSDC) to preserve cross-modal semantic similarity across tasks and retain old knowledge, integrated into an audio-visual sound separation framework.

STRENGTHS:
1. The proposed method is a good solution to help maintain cross-modal semantic similarity across incremental tasks.
2. The paper has a clear writing structure, and the figures and tables are easy to understand.

WEAKNESSES:
1. The performance improvement of ContAV-Sep (with iQuery) is not significant.
2. What does iSTFT stand for in Figure 2? The symbols appearing in the figure need to be clearly explained in the caption.

QUESTIONS:
See weakness.

LIMITATIONS:
See weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis approach defines an audio-visual sound separation task where sound separation is the task and during fine-tuning novel classes are added, in the regime of continuous learning. The goal is to avoid catastrophic forgetting which typically leads to decreased performance in task performance on classes which were learned early but then little or no instances arrive in later training stages. They then present a method to solve this task with audio-visual data. Several losses are defined to achieve cross-modal similarity of embeddings through incremental tasks and preserve previous learned knowledge. \n\nTheir approach does not increase the number of logits to be able to accommodate more classes but instead creates separation masks, more suitable to audio separation than typical classification tasks.\n\nResults are compared on meaningful baselines even though this is basically a new task definition and ablating the different loss components shows that the combination yields the best performance.\n\nAdditional videos and investigations in the supplementary work complete the work.\n\nSTRENGTHS:\nEspecially in the videos the performance on very similar instruments is impressive. The instruments share a very similar frequency profile and their separation is a very hard problem.\n\nThe paper is very clearly written, the task is clearly stated and well contrasted and localized within the domain.\n\nThe mathematical notation is very clear to follow.\n\nIn general the paper defines an interesting and realistic task, does a thorough investigation and states clearly its limitations.\n\nWEAKNESSES:\nThe paper defines a task and then solves it, which is always a bit easier due to limited competition. A direct comparison between sound separation was excluded by the authors stating that they do not compete in that sense, which is a limitation.\n\nFigure 2 is very small and could be more self complete. The relation of the right side's illustration to the left side is not clear from the caption at all which only states the names of the concepts. Also, there is space left and right to increase the size. Probably this resizing was reduced to gain space. This leaves the text very small. It would be good to find space in a different way because the fonts are already extremely small within Figure 2. Some of the dataset details could probably go into the appendix.\n\nThis is not a summary paper. That the author's found so much related work is commendable but I would have preferred more detailed contrast to existing work, e.g. by picking out certain ideas and contributions, explaining them and then contrasting this work against them. Just having lists of 7 works and summarizing them as \"semantic segmentation\" seems more like a homework chore than a contribution to the paper. But my fellow reviewers may disagree.\n\nQUESTIONS:\n- It seems from the problem definition that all tasks have distinct classes. That seems like a special, even though the hardest possible, case. Do you have an intuition or looked into variations where actually a smaller part of instances from a specific class is part of all tasks? It seems also more realistic to assume that such a network is not necessarily fine-tuned on a dataset that is completely different from the previous one.\n\n- What are the hardware requirements for this? It seems CLIP is used even twice, once for each task and then the features are fed into transformers. Yet the NVIDIA RTC A5000 does seem to have only 24GB. Are there some compromises, e.g. image resolution, which were made due to the hardware available?\n\n- On a related note: What is the memory size? How does the memory and computation scale to new tasks as the memory keeps growing beyond 4 tasks?\n\n- The choice of 11kHz audio sampling seems odd compared with defaults in audio encoding or other approaches for other audio-visual tasks. What is the reasoning behind it?\n\n- How does this scale to hard examples, where both instruments are the same. The second example is already impressive with two woodwinds but it would be interesting how much the approach is able to derive from movement and how much from appearance in the visual modality.\n\n- It is interesting how the approach scales to n-classes in Figure 3 but how does it scale to n-tasks? In the appendix there is an investigation into old classes but an investigation of the performance measures for, say, one chosen class after 4,5,6,7... tasks would be interesting to judge the trend.\n\nLIMITATIONS:\nThe limitations are properly listed and the authors are upfront about them.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

comments: SUMMARY:
This work proposes a continual audio-visual sound separation framework to mitigate the catastrophic forgetting problem

STRENGTHS:
As I learned, this is the first work that focuses on the catastrophic forgetting problem in audio-visual separation task.

WEAKNESSES:
1. The font size in Figure 2 is too small, reducing the readability of this paper.
2. The technique novelty is limited, the model architecture is totally the same as iQuery, and the most brightness point of this work is just proposes a cross-modal similarity distillation constraint, however, it just the combination of the contrastive loss implemented on the modalities and features extracted from different training step.
3. Experiments are not enough, all experiments are conducted on Music21, which just contains limited data among 21 classes. Experiments conducted on Music [1], VGGSound [4], and AVE datasets [2-3] can provide a more comprehensive evaluation.
[1] The sound of pixels.
[2] Audioset: An ontology and human-labeled dataset for audio events.
[3] Audio-visual event localization in unconstrained videos
[4] Vggsound: A large-scale audio-visual dataset

QUESTIONS:
Please seem the weakness part.

LIMITATIONS:
1. Limited novelty.
2. Insufficient experiments
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-23