Continual Audio-Visual Sound Separation
Authors: Weiguo Pian
Yiyang Nan
Shijian Deng
Shentong Mo
Yunhui Guo
Yapeng Tian
Venue: NeurIPS 2024 poster (2024)
Keywords: Audio-Visual Learning
Sound Separation
Continual Learning
Primary area: speech_and_audio
TL;DR: In this paper, we introduce a novel continual audio-visual sound separation task and an approach named ContAV-Sep for the proposed task.

Abstract: In this paper, we introduce a novel continual audio-visual sound separation task, aiming to continuously separate sound sources for new classes while preserving performance on previously learned classes, with the aid of visual guidance. This problem is crucial for practical visually guided auditory perception as it can significantly enhance the adaptability and robustness of audio-visual sound separation models, making them more applicable for real-world scenarios where encountering new sound sources is commonplace. The task is inherently challenging as our models must not only effectively utilize information from both modalities in current tasks but also preserve their cross-modal association in old tasks to mitigate catastrophic forgetting during audio-visual continual learning. To address these challenges, we propose a novel approach named ContAV-Sep ($\textbf{Cont}$inual $\textbf{A}$udio-$\textbf{V}$isual Sound $\textbf{Sep}$aration). ContAV-Sep presents a novel Cross-modal Similarity Distillation Constraint (CrossSDC) to uphold the cross-modal semantic similarity through incremental tasks and retain previously acquired knowledge of semantic similarity in old models, mitigating the risk of catastrophic forgetting. The CrossSDC can seamlessly integrate into the training process of different audio-visual sound separation frameworks. Experiments demonstrate that ContAV-Sep can effectively mitigate catastrophic forgetting and achieve significantly better performance compared to other continual learning baselines for audio-visual sound separation. Code is available at: https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024.

Continual Audio-Visual Sound Separation
Weiguo Pian'
Yiyang Nan?
Shijian Deng'
Shentong Mo
Yunhui Guo'
Yapeng Tian
The University of Texas at Dallas 2 Brown University Carnegie Mellon University
Abstract
In this paper, introduce novel continual visual sound separation task, aiming to continuously separate sound sources for new classes while preserving per- formance on previously learned classes, with the aid of visual guidance. This prob- lem crucial for practical visually guided auditory perception as it can significantly enhance the adaptability and robustness audio-visual sound separation models, making them more applicable for well-world scenarios where encountering new sound sources is commonplace. The task is inherently challenging as our models must not only effectively utilize information from both modalities in current tasks but also preserve their cross-modal association in old tasks mitigate catastrophic forgetting during audio-visua continual learning. To address these challenges, we propose a novel approach named ContAV-Sey (Continual Audio-Visual Sound Separation). ContAV-Sep presents a novel Cross-modal Similarity Distillation Constraint (CrossSDC) to uphold the cross-modal semantic similarity through cremental tasks and retain previously acquired knowledge of semantic similarity in old models, mitigating the risk of catastrophic forgetting. The CrossSDC can seam- lessly integrate into the training process of different audio-visual sound separation frameworks. Experiments demonstrate that ContAV-Sep can effectively mitigate catastrophic forgetting and achieve significantly better performance compared to other continual learning baselines for audio-visual sound separation. Code is Neur IPS2244.
available at: https: //t////i thub. cow/weigguee www.wwoiiinnCooooo iaap/CinnnCC/ Mee.AV--eee..euuP222O0
1 Introduction
Humans can effortlessly separate and identify individual sound sources in daily experience [25, 7, 64, 33]. This skill plays erucial role in ability understand and interact with the complex auditory environments that surround us [34]. However, replicating this capability in machines remains a significant challenge due to the inherent complexity of real-world auditory scenes [7, 77]. Inspired by the multisensory perception of humans [62, 60), audio-visual sound separation tackles this challenge by utilizing visual information guide the separation of individual sound sources in audio mixture.
Recent advances in deep learning have led to significant progress audio-visual sound separation [84, 23, 21, 67, 14, 65, 81, 63, 11, 71]. Benefiting from more advanced architectures (e.g.. U-Net [84, 23], Transformer [14], and diffusion models [27]) and discriminative visual cues (e.g., grounded visual objects [67], motion [83]. and dynamic gestures [21]), audio-visual separation models are able to separate sounds ranging from domain- specifcc speech, musical instrument sounds to open-domain general sounds within training sound categories. However, limitation of these studies is their focus on scenarios where all sound source classes are presently known, overlooking the potential inclusion of unknown sound source classes during inference in real-world applications. This oversight leads to the catastrophic forgetting issue [32, 3), where the fine-tuning of models on new classes detrimentally impacts their performance previously learned classes. Despite Chen et al. [14] demonstrating that their iQuery model can generalize to new classes well through simple fine-tuning, it still suffers from the catastrophic forgetting problem old classes. This prevents the trained models from
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

continuously updating in real-world scenarios, impeding their adaptability to dynamic environments. The question how to effectively leverage visual guidance continuously separate sounds from new categories while preserving separation ability for old sound categories remains open.
To bridge this gap, we introduce a novel continual andio-visual sound separation task by integrating audio- visual sound separation with contin- ual learning principles. The goal of this task is to develop an audio-visiie visuull model that can continuously separate sound sources in new classes while maintaining performance on previously learned classes. The key challenge we need to address is catastrophic for- getting during continual visual learning, which occurs when the model is updated solely with data from new classes or tasks, resulting in a signifi- cant performance drop on old ones. We illustrate our new task and the catas- trophic forgetting issue in Fig. 1.
Task
I:precornioon"
aacandioon (cannet..)
Task 'cehe', "gatheng" 'pipa...
Clarinct
Clariert
Model
Pipa
Purformance Comparison
(Dradl))
Fin-hining
Tak
Figure 1: Top: Illustration of the continual audio-visual sound separation task, where the model (separator) learns from sequential audio-visual sound separation tasks. Bot- tom: Illustration of the catastrophic forgetting problem in continual audio-visual sound separation and its mitigation by our proposed method. Fine-tuning: Directly fine-tune the separation model on new sound source classes; Upper bound: Train the model using all training data from seen sound source classes.
Unlike typical continual learning prob- lems such as task-, domain-, or class- incremental classification in visual do- mains [2. 57, 38, 53, 85], which result in progressively increasing logits (or probability distribution) across all ob- served classes at each incremental step, our task uniquely produces fixed-size separation masks throughout all incremental steps. In this context, each entry in the mask does not directly correspond to any specific classes. Additionally, the new task involves both audio and visual modalities. There- fore, simply applying existing visual-only methods cannot fully exploit and preserve the inherent cross-modal semantic correlations. Very recently, Pian et al. [53] and Mo et al. [44] extended continual learning to the audio-visual domain, but both focused classification tasks.
To address these challenges, in this paper, we propose a novel approach named ContAV-Sep (Continual Audio-Visual Sound Separation). Upon the framework, we introduce novel Cross-modal Similarity Distillation Constraint (CrussSDC) to not only maintain the cross-modal semantic similar- ity through incremental tasks but also preserve previously learned knowledge of semantic similarity in old models to counter catastrophic forgetting. The CrossSDC is generic constraint that can be scamlessly integrated into the training process of different audio-visual sound separators. To evaluate the effectiveness of our proposed ContAV-Sep. we conducted experiments on the MUSIC-21 dataset within the framework of continual learning, using the state-of-the-are audio-visual sound separation model iQuery [14] and a representative audio-visual sound separation model Co-Separation [23], as our separation base models. Experiments demonstrate that ContAV-Sep can effectively mitigate catastrophic forgetting and achieve significantly better performance than other continual learning baselines. In summary, this paper contributes follows:
(i) To explore more practical audio-visual sound separation, in which the separation model should be generalized to new sound source classes continually, pose Continual Audio- Visual Sound Separation task that trains the separation model under the setting of continual learning. To the best of knowledge, this is the first work on continual learning for audio-visual sound separation.
(ii) We propose ContAV-Sep for the new task. uses a novel cross-modal similarity distillation constraint to preserve cross-modal semantic similarity knowledge from previously learned models.
(iii) Experiments on the MUSIC-21 dataset can validate the effectiveness of our ContAV-Sep, demon- strating promising performance gain over baselines.
2

---

2 Related Work
Audio-Visual Sound Separation. Audio-visual sound separation aims to separate individual sound sources from an audio mixture guided by visual cues. A line of research emerges under various scenarios, such as separating musical instruments [23, 84, 79, 21, 83, 67], human speech [20, 1, 18, 49, 15], sound sources in in-the-wild videos [22, 70J. Many frameworks and methods have been proposed to address challenges within specific problem settings, For instance, the extraction of face embeddings proves beneficial for speech audio separation [18]. Moreover, incorporating object detection can provide an additional advantage [23, 22]. The utilization of trajectory optical flows to leverage temporal motion information in videos, demonstrated by [83], also yields improvements. In this work, not competing on designing stronger separators, we would advance the exploration of the audio-visual sound separation within the paradígm of continual learning. We investigate how a model can learn to consistently separate sound sources from sequential separation tasks without forgetting previously acquired knowledge.
Continual Learning. The field of continual learning has drawn significant attention, especially in visual domains, with various approaches addressing this challenge. Notable among these are regularization-base methods, exemplified in works such as [32, 3, 31, 39J. These approaches involve applying regularization to crucial parameters associated with old tasks to maintain the model's capabilities and during incremental steps, less important parameters are given higher priority for updates compared to important ones. Conversely, several works [57, 5, 26, 12, 54, 8, 10, 40] applied rehearsal-bar -aaed pipelines to enable the model review previously learned knowledge. For instance, Rebuffi al. [57] proposed one of the most representative exemplars selection strategy Nearest-Mean of of-Exxmlaars (NME), selects the most representative exemplars in each class based on the distance to the feature center of the class. Meanwhile, pseudo-rehearsal [47, 48, 66] employs generative models to create pseudo-cxemplars based on the estimated distribution of data from previous classes. Moreover, architecture- sawuuur--aaeeddyyaamnia modify the model architecture itself to enable the model to acquire new knowledge while mitigating the forgetting of old knowledge. Specifically, Pham et al. [52] proposed a dual network architecture, in which one is to learn new tasks while the other one is for retaining knowledge learned from old tasks. Wang et al. [72] combined the dynamic architecture and distillation constraint to mitigate the issue of continull-increasinn
architecture methods[52, 4, 46, 24, 28, 37, 40] proposed to
overhead problem in dynamic architecture-based continual learning method. However, above studies mainly concentrate on the area of continual image classification. Recently, researchers also explored other continual learning scenarios beyond image classification. For instance, Park et al. [50] extend the knowledge distillation-based [2, 17] continual image classification method to the domain of video by proposing the time-channel distillation constraint. Douillard et al. [16] proposed to tackle the continual semantic segmentation task with multi-view feature distillation and pseudo-labeling. Xiao el al. [78] further addressed the continual semantic segmentation problem through weights fusion strategy between old and current models. Wang et al. [75] addressed the continual sound classification task through generative replay. Furthermore, continual learning has also been explored in the domain of
wawnug//viiiionnangaaaae learning tasks [30, 43, 59, 61, 19, 86], self-supervised representation learning [19, 42, 55, 80, 35, 74), audio classification [6, 73] and fake audio detection [41, 82), etc. Despite the success of existing continual learning methods in various scenarios, their applicability in the domain of continual audio-visual sound separation is still unexplored. Although Pian et al. [53] and Mo et al. [44] proposed to tackle the catastrophic problem in audio-visual learning, their studies mainly concentrated in the area of audio-visual video classification. In contrast to existing works in continual learning, in this paper, we delves into the continual audio-visual sound separation, aiming to tackle the challenge of catastrophic forgetting specifically in the context of separation mask prediction for complicated mixed audio signals within joint -viuull modeling.
3 Method
3.1 Problem Formulation
Audio-Visual Sound Separation. Audio-visual sound separation aims to separate distinctive sound signals according to the given associated visual guidance. Following previous works [14, 23, 67, 21, 79], we adopt the common "mix-and- "nix-and-separaatoon" training strategy to train the model. Given two videos Vi V1(11, v1) and (2(52, v2), we can obtain the input mixed sound signal S by mixing two
3

---

= video sound signals $1 and 82, and then we can have the ratio masks M s1/S and M = s2/S¹ S'. The goal of the task is to utilize the corresponding visual guidance 21 and V22 to predict the ratio masks for reconstructing the two individual audio signals. This process can be formulated as:
M' Fe(S,01),
M (e, v2),
where Fe is the separation model with trainable parameters @ And then, the original sound signals 81 and $2 are used calculate the loss function for optimizing the model:
θ x = argmin VVv [C(M' M³ + C(M , M3).
(2)
where D denotes the training set, and is the loss function between the prediction and ground-truth.
Continual Visual Sound Separation. Our proposed continual audio-visual sound separation task aims train a model Fe continually on a sequence of 'T separation tasks {Ti, ..,,.. Tr]. For the f-th task Ta (incremental step t), we have a training set D. {V'(s', ,'),, y where and ne denote the i-th video sample and the total number of samples in Dj respectively, and y E Ce is the corresponding sound source class of video V where C, is the training sound class label space of task Tr For any two tasks Tr, and Tr, and their corresponding training sound class label space C, and Cig. we have Cr, Ct2 0. Following previous works in continual learning [57, 2, 53, 44, 29, 76), for a task Te. where & > 1, holding a small size of memory/exemplar set Mr to store some data from old tasks permitted
our setting. Therefore, with the memory/exemplar set MM, all available data that can be used for training in task Tr (f > 1) can be denoted as Dj- = D, U,M Finally, the training process of Eq. in our continual audio-visual sound separation setting can be denoted as:
+ argmin E (V, [C(M' MM (AMT.M))
0
S.L. M = Fe. (S, M² = Fe, (S,02).
which means that the new model O is obtained by updating the old model 0,-1 which was trained on the previous task, using the current task's available data D/. After the training process for task Tr with Di, the updated model will be evaluated on a testing set which includes video samples from all seen sound source classes up to continual step / (task T.). And the evaluation also follows the common mnixx-and-eepaattion" strategy. During this continual learning process, the model's s separation performance on the previously learned tasks drops significantly after training on new tasks. This learning issue is referred to as the catastrophic forgetting [32, 38, 3] problem, which poses considerable challenge in continual audio-visiee sound separation.
3.2 Overview
To address the challenge of catastrophic forgetting in continual audio-visual sound separation, introduce ContAV-Sep. This new framework, illustrated in Fig. 2, consists of three key components: separation base model, an output mask distillation module, and our proposed Cross-modal Simi- larity Distillation Constraint (CrossSDC)) We use a recent state-of-the- audio- iuuery [14] as the base model of our approach, which contains video encoder to extract the global motion feature, an object detector and image encoder to obtain the object feature, a U-Net [58] for mixture sound encoding and separated sound decoding, and an audio-visual Transformer get the separated sound feature through multi-modal cross-attention mechanism and class-aware audio queries. For the object detector, we follow iQuery [14] and use the pre-trained Detic [87], universal object detector, to detect the sound source objects in each frame. For the video encoder and the image encoder, inspired by the excellent generalization ability of recent self-supervised pre-trained models, which has been proven to be effective and appropriate in continual learning [53], we apply two self-supervised pre-trained models VideoMAE [69] and CLIP [56] as the video encoder and the image encoder, respectively. Note that, during the training process, the object detector, video encoder, and image encoder are frozen.
separator:
In practice, the audio signal is first processed using the Short-Time Fourier Transform (STFT) to generate a spectrogram, For brevity, we will denote spectrogram magnitudes 81, $2, and S.

---

Crossmoddal Similarity Distillation Constraint
Adda
Cros--mdal Cuntrattiv
Mix Audio
Separated Audin
INTFT
Fee
Output Mask Distillation
M
X7,
Mix Spectram
Auadon hova
Lauraabl Quar)
Ali Spectrure
C(M M,-1)
Figure 2: Overview our proposed ContAV-Sep, which consists of an audio-visual sound separation base model architecture, Output Mask Distillation, and our proposed Cross-modal Similarity Distillation Constraint. The fire icon denotes the module is trainable, while the snowflake icon denotes that the module is frozen. The (i)STT stands for (inverse) Short-Time Fourier Transform. Please note that, the old model Fe.., is frozen during training.
Given a pair of videos (1 01) and ($2, v2). at incremental step (task Tr). the U-Net audio encoder JAL takes the mixed audio signal S obtained by mixing 81 and as input, and generates the latent mixed audio feature. This process can expressed as:
flat FFF
(4)
Then, the audio-visual Transformer J.rrane is employed to generate the separated sound feature by taking the latent mixed audio feature and visual features as inputs:
f = Ff rans. (fler. fo f s.L. S = Ug(b)) fth (Mo.))
(5)
where fa.) denotes the separated sound feature of video V1; Obj. and Mo. denote the object and motion features extracted by the frozen pre-trained image and video encoders respectively from the visual signal v1 of video V1. U-(-) and U (-) are learnable projection layers map the object and motion features into the same dimension. Similarly, we can also obtain the separated sound feature of V2 guided by the associated visual features.
The extracted separated sound feature and the latent mixed audio feature are combined to generate mask. This mask is subsequently applied to the mixed audio, leading the reconstruction of the separated sound spectrogram.
M, = J.AD (flat @ MLP.(S..) )
(6)
M, F.AD (flat O MLP.(S..)
where ! and M denote the predicted masks for audio signals of video V1 and V2, respectively; FAD is the U-Net decoder at incremental step t; (.P,,(.) denotes a MLP module; and o denotes channel-wise multiplication. The sound at this incremental step be reconstructed by applying
S 3 M, and then performing an inverse STFT to obtain the audio waveform.
3.3
Cross-modal Similarity Distillation Constraint
Recent studies [53, 45] have highlighted the importance of cross-modal semantic correlation in audio-visual modeling. However, this correlation tends to diminish during subsequent incremental phases, which leads catastrophic forgetting in our continual audio-visual sound separation task. To address this challenge, propose novel Cross-modal Similarity Distillation Constraint (CrossSDC)
5

---

that serves two crucial purposes (1) maintaining cross-modal semantic similarity through incremental tasks, and (2) preserving previous learned semantic similarity knowledge from old tasks.
CrossSDC preserves cross-modal semantic similarity from two perspectives: instance-awar semantic similarity and class-aware semantic similarity. Both similarities enforced by integrating contrastive loss and knowledge distillation. Instead of exclusively focusing on the similarities within current and memory data generated by the current training model, CrossSDC incorporates the cross-modal similarity knowledge acquired from previous tasks into the contrastive loss. This integration not only facilitates the learning of cross-modal semantic similarities in new tasks but also ensures the preservation of previously acquired knowledge. In the incremental step (t > 1), the instance- part of our CrossSDC can be formulated as:
Σ E1(i log exp(sim(frm))
Fnd & Σx exp(sim (madd frii
Linst. = -EV
(7)
where [ = .] is an indicator that equals when j. denoting that video samples V and V are the same video; The sim function represents the cosine similarity function with temperature scaling; The modalities modj and mod2, where (mod,, mod2) E {(a o), (a, m), (m, J) denote different pairs of features to be compared: separated sound and object features, sound and motion features, and motion and object features. Here, T denotes the incremental step, for which we have:
71 T2 T, where T T- T=
{t, 1}, if eMc. {t},
8
if E D.,
which means that, for current task's data D,, we calculate the contrastive loss using features from the current model (Ti 72 1), while for memory set data M,, we use features from both the old and current models (e.g., TI = and T2 = 1). In this way, knowledge distillation would be integrated into the cross-modal semantic similarity constraint for the current task, which ensures better preservation of learned cross-mode semantic similarity from previous tasks.
While the instance-aware similarity provides valuable semantic correlation modeling, it does not account for the class-level semantic correlations, which is also crucial for audio-visual similarity modeling. To capture and preserve the semantic similarity within each class across incremental tasks, we also incorporate a class- component specifically designed for inter-class mooal semantic similarity, which can be formulated as:
exp(sim( mods 71.8 s smodd))
Cols. -vi wD
Σ, 1ly w01 EAp'= = o'g Σn exp(sim( (fmal) smook))
(9)
In this context, visual and audio features from two videos are encouraged to close when they belong to the same class. The overall formulation of our CrossSDC is as follows:
LCroseSDC
Ainn
Acte
(10)
where Aine and Aota are two scalars that balance the two loss terms. In this way, the model captures and preserves semantic correlations not just between instances but also within the same classes.
3.4 Overall Loss Function
In the previous subsection, we introduced our proposed CrossSDC constraint. To effectively combine CrossSDC with the overall objective, we incorporate it alongside output distillation and the main separation loss function.
Output distillation is a widely used technique in continual learning [38, 2, 53) to preserve the knowledge gained from previous tasks while learning new ones. In approach, utilize the output of the old model as the distillation target to preserve this knowledge. Note that we only distill knowledge for data from the memory set, as represented by:
Ldist. = EV [ Mi + IIM - -M,
(11)
where M ; and M. (-1 are predicted masks generated by the old model that is trained incremental
step t 1. For the loss function here, we follow [84, 14] and adopt the per-pixel L1 loss [84]. For

---

the main separation loss function, we also apply the per-pixel L1 loss:
Emain E (vi |I - M 1 || 1
(12)
Finally, our overall loss function is denoted as:
AV = Emain + Adist, Caist. + LCrossSDC.
(13)
3.5 Management of Memory Set
In alignment with the work of [76], our framework maintains compact memory set throughout incremental updates. Each old class limited a maximum number of exemplars. After completing training for each task, we adopt the exemplar selection strategies in [2. 53J by randomly selecting exemplars for each current class and combining these new exemplars with the existing memory set.
Experiments
In this section, first introduce the setup of experiments, i.e., dataset, baselines, evaluation metrics, and the implementation details. After that, we present the experimental results of our ContAV-Sep compared to the baselines, as well as ablation studies. We also conduct experiments on the AVE [68] and the VGGSound [13] datasets, which contain sound categories beyond the music domain. We put the experimental results on the AVE and the VGGSound datasets, the comparison to the uni-modal semantic similarity preservation method, the performance evaluation on old classes in incremental tasks, and the visualization of separating results in the Appendix.
4.1 Experimental Setup
Dataset. Following common practice [83, 88, 14], we conducted experiments on MUSIC-21 [83], which contains solo videos of 21 instruments categories: accordion, acoustic guitar, cello, clarinet, erhu, flute. saxophone, trumpet, tuba, violin, xylophone, bagpipe, banjo, bassoon, congas, drum, electric, bass, guzheng, piano, pipa, and ukulele. In our experiments, we randomly selected 20 of them to construct the continual learning setting. Specifically, we split the selected 20 classes into incremental tasks, each of which involves 5 classes. The total number of available videos 1040, and we randomly split them into training, validation, and testing sets with 840, 100, and 100 videos. respectively. To further validate the efficacy of our method across a broader sound domain, we conduct experiments using the AVE [68] and the VGGSound [13] datasets in the appendix.
Baselines. We compare our proposed approach with vanilla Fine-tuning strategy, and continual leam- ing methods EWC [32] and LwF [38]. As we mentioned before, typical continual learning methods, e.g. class-incremental learning methods, which yield progressively increasing logits (or probability distribution) across all observed classes at each incremental step and design specific technique in the classifier, we consider that these methods are not an optimal choice for our proposed continual audio-visual sound separation problem. Thus, considering that continual semantic segmentation problem has more similar form compared to conventional Mlas-inccemmntal learning. we also tite--of-thearr
select two state-of- the-art art continual semantic segmentation methods PLOP [16] and EWF [78] as our baselines. Moreover, we compare our method to the recently proposed audio-visual continual learning method AV-CIL [53], in which we adapt the original class-incremental version to the form of continual audio-visual sound separation by replacing their task-wise logits distillation with the output mask distillation. Further, we also present the experimental results of the Oracle/Upper Bound, which means that using the training data from all seen classes to train the model. For fair comparison, all compared continual learning methods and our ContAV-Sep use the same state-of-the-ars sepa- rator, i.e. iQuery [14], as the base separation model. Further, we also incorporate our proposed and baseline methods into another representative audio-visun sound separation model Co-Separation [23]. Notably, the Co- Co-Separation model does not utilize the motion modality. Therefore, when CrossSDC is applied to Co-Separation, the (mod1, mod2) in Eq. 7 and 9 is constrained to (mod1, mod2) (a. o). For baselines that involve memory sets, we ensure that each of them is allocated the same number of memory our proposed method for fair comparison.
Implementation Details. Following [14], we use a 7-layers U-Net [58] as the audio net, and sub- sample the audio at 1 1kHz, each of which is approximately 6 seconds. We apply the STFT with
7

---

Table 1: Main results of different methods on MUSIC-21 dataset under the setting of Continual
Audio- Visiol Sound Separation with base separation models of iQuery [14] and Co-Separation [23],
respectively. The bold part denotes the best results. Our proposed ContAV-Sep achieves the best
performance among all baselines.
Method
SDRT
SIR
SART
Method
SDRT
SIR
SAR*
w/o memory
w/o memorry
iQuery [14] + Fine-tuning
3.46
9.30
10.57
Co-Sep. [23] + Fine-tuning
1.93
8.75
9.75
iQuery [14] + L.F [38]
3.45
8.78
10.66
Co-Sep. [23] + LwF [38]
2.32
7.84
10.28
iQuery [14] + EWC [36]
3.67
9.58
10.30
Co-Sep. [23] + BWC [36]
2.01
8.36
9.61
iQuery [14] PLOP [16]
3.82
10.06
10.22
Co-Sep. [23] + PLOP [16]
3.24
9.17
9.59
iQuery [14] FWF [78)
3.98
9.68
11,52
Co-Sep. [23] + RWF [78]
2.61
7.77
10.85
w'menory
w/ memory
iQuery [14] + LWF [38]
6.76
12.77
12,60
Co-Sep. [23] + LwF [38]
3.85
9.62
10.74
iQuery [14] +WWC [36]
6.65
13.01
11.73
Co-Sep. [23] + BWC [36]
3.31
9.55
9.80
iQuery [14] PLOP [16]
7.03
13.30
11,90
Co-Sep. [23] PLOP [16]
3.88
9.92
9.99
iQuery [14] EWF [78]
5.35
11.35
11.81
Co-Sep. [23] + EWF [78]
3.63
9.07
10.58
iQuery [14] + AV-CIL [53)
6.86
13.13
12.31
Co-Sep. [23] + AV-CIL [53]
3.61
9.76
9.68
ContAV-Sep (with iQuery [14])
7.33
13.55
13.01
ContAV-Sep (with Co-Sep. (23])
4.06
10.06
11.07
Upper Bound (with iQuery)
10.36
16.64
14.68
Upper Bound (with Co-Sep.)
7.30
14.34
11.90
the Hann window size of 1022 and the hop length of 256, to obtain the 512 x 256 Fime--Fry representation of each audio signal, followed by a re-sampling on the log-frequency scale to generate the magnitude spectrogram with T, F' = 256. We set the video frame rate (FPS) to I, and detect the object using the pre-trained universal detector Detic [87] to detect the sound source object on each frame, and then, each detected object is resized and randomly cropped to the size of 224 x 224. For the image encoder and the video encoder, we apply the self-supervised pre-trained CLIP [56] and VideoMAE [69] to yield the object feature and motion feature. respectively. For the oudio-visual Transformer module, we follow the design in [14]. For the baseline methods, we apply the same model architecture and modules with ours for them, including the mentioned Detic, CLIP, VideOMAF, audio-visual Transformer, etc. Please note that, during our training process, the pre-trained Detic, CLIP, and VideoMAH are frozen. In our proposed Cross-moda Similarity Distillation Constraint (CrossSDC), the balance weights Xine and Aate are set to 0.1 and 0.3, respectively. And the balance weight Adist. for the output distillation loss is set to 0.3 in our experiments. For the memory set, we set the number of samples in each old class to 1, so as other baselines that involve the memory set. All the experiments in this paper implemented by Pytorch [51]. We train proposed method and all baselines on a NVIDIA RTX A5000 GPU. We follow previous works [67, 14] in sound separation, and evaluate the performance of all the methods using three common metrics in sound separation tasks: Signal to Distortion Ratio (SDR). Signal to Interference Ratio (SIR), and Signal to Artifact Ratio (SAR). The SDR measures the interference and artifacts, while SIR and SAR measure the interference and artifacts, respectively. In our experiments, we report the SDR, SIR, and SAR of all the methods after training at last incremental steps, i.e., testing results on all classes. For all these three metrics, higher values denote better results.
4.2 Experimental Comparison
The main experimental comparisons are shown in Tab. 1. Our proposed method, ContAV-Sep. outperforms the state-of-the-ar baselines by a substantial margin. Notably, compared to baselines using state-of-the-art audio-visual sound separator iQuery [14] as the separation base model, ContAV- Sep achieves a 0.3 improvement in SDR over the compared best-performing method. Additionally, our method surpasses the top baseline by 0.25 in SIR and 0.41 in SAR. Furthermore, compared to continual learning basclines with Co-Separation [23]. our ContAV-Sep still outperforms other approaches. This consistent superior performance across different model architectures highlights not only the effectiivnness but also the broad applicability and generalizability of our proposed CrossSDC.
Our observations further demonstrate that retaining a small memory set significantly enhances the performance of each baseline method. For instance, for the iQuery- based continual learning methods, equipping LwF [38] with small memory set results in improvements of 3.31, 3.99, and 1.94 on SDR, SIR, and SAR, respectively. Similarly, the addition of a small memory set to EWC [32] leads to enhancements of 2.98, 3.43, and 1.43 in the respective metrics. The memory-augmented version of PI.PP [16] exhibits superior performance with margins of 3.21, 3.24, and 1.68 for SDR, SIR, and SAR, respectively, Finally, incorporating memory into EWF [78] results in improvements of 1.37,
8

---

KLO" wereey
Clasee
Clasme
Classes
(a)
(b)
(c)
Figure 3: Testing results of different continual learning methods with iQuery [14] the metrics of (a) SDR, (b) SIR, and (c) SAR at each incremental step.
1.67, and 0.29 for the three metrics. This phenomenon can be attributed to the inherent nature of the sound separation training process. In training, the audio signal from each sample mixes with others, giving a composite audio signal. This mixed audio signal, coupled with the corresponding visual data pair for each separated audio, constitutes the actual training sample for the separation task. As result, even single memory sample can be associated with multiple samples from the current training set, generating diverse array of effective training pairs.
We also present the testing results of SDR. SIR, and SAR at each incremental step in Figures 3a, 3b, and 3c, respectively. Our method is consistently observed outperform others in of SDR at all incremental steps. While our approach may not always produce the best SIR and SAR results at the intermediate steps (specifically, steps 2 and 3 for SIR, and step for SAR), ultimately achieves the highest performance at the final step. This demonstrates the robustness of our method, indicating minimal forgetting throughout the incremental learning process.
4.3 Ablation Study on CrossSDC and Memory Size
In this subsection, conduct ablation study to investigate the effectiveness of our proposed CrossSDC. By removing single or multiple components of the CrossSDC, we evaluate the impact of each on the final results. The results of the ablation study are presented in Tab. 2. From the table, we can see that our full model achieves the best performance compared to the variants, which further demonstrates the effecttivneess of our proposed CrossSDC.
Moreover, we also discuss the effect of memory size on our proposed ContAV-Sep. In our main experiments, the default setting of the memory size is 1 sample per old class. In this subsection, we conduct experiments by increasing the memory size from sample per old class to samples per old class. The experimental results are shown in Tab. 3 and Figure 4. Observations from the table indicate a positive correlation between the size of the memory and the overall performance metrics. As the memory size increases, there is discernible trend of improvement in the results.
Table 2: Ablation study on our proposed ContAV-Sep. Our full approach achieves best results compared to the variants.
Edist.
Linst.
Lets.
SDR
SIRT
SARt
x
X
6.32
12.99
11.82
ContAV-Sep
/
X
6.01
11.92
11.74
x
6.86
13.12
12.25
7.33
13.55
13.01
4.4 Limitation and Discussion
Our experimental findings reveal that the utilization of a small memory set, even single sample per old class, markedly improves the performance of each baseline method. This improvement attributed to the ability of a single memory sample to pair with diverse samples from the current training set, thereby generating numerous effective training pairs. Consequently, this process enables the model to acquire new knowledge for old classes in subsequent tasks, the memory data can be

---

Table Experimental results of our proposed ContAV-Sep with different memory size from to 30 samples per memory class.
of samples per class
SDR
SIRT
SART
1
7.33
13.55
13.01
2
7.26
13.10
12.65
3
7.88
13.66
13.43
ContAV-Sep
4
8.16
14.16
13.21
10
8.97
15.16
13.72
20
9.39
15.93
13.69
30
10.09
16.34
14.10
0.4
al sempphe diase
#et sumpples cheos
sumpphrr per cleer
(a)
(c)
Figure 4: Testing results with different memory size (number of samples per class in the memory) the metrics of (a) SDR, (b) SIR, and (c) SAR at each incremental step.
paired with data from previously unseen new classes - this is different from conventional continual learning tasks, where old classes do not acquire new knowledge in new tasks. This could be potential reason why the baseline continual learning methods do not perform well in our continual audio-visual sound separation problem. In this work, our method also mainly focuses on preserving old knowledge of old tasks, which may prevent the model from acquiring new knowledge of old classes when training new tasks. Recognizing this, we identify exploration of this problem as key avenue for future research in this field.
Additionally, the base model architectures used in our approach and baselines require object detectors to identify sounding objects. Although iQuery [14] can supplement object features with global video representations, it may still suffer from undetected objects. is fundamental limitation of the object-based based audio-visual sound separators [23, 14]. While our work, unlike previous efforts, does compete on designing stronger audio-visual separation base model, enhancing the robustness of sounding object detection presents promising direction for future research.
5 Conclusion
In this paper, we explore training audio-visure sound separation models under a more practical continual learning scenario, and introduce the task of continual audio-visual sound separation. To address this novel problem, we propose ContAV-Sep, which incorporates a Cross-modal Similarity Distillation Constraint maintain cross-modal semantic similarity across incremental tasks while preserving previously learned semantic similarity knowledge. Experiments on the MUSIC-21 dataset demonstrate the effectiveness of our method in this new continual separation task. This paper opens new direction for real-world audio-visual sound separation research.
Broader Impact. Our proposed continual audio-visual sound separation allows the model to adapt to new environments and sounds without full retraining, which could enhance efficiency and privacy by reducing the need to transmit and store sensitive audio data.
Acknowledgments. We thank the anonymous reviewers and area chair for their valuable suggestions and comments. This work was supported in part by Cisco Faculty Research Award, an Amazon Research Award, and a research gift from Adobe. The article solely reflects the opinions and conclusions of its authors but not the funding agents.
10

---

References
[1] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. The conversation: Deep audio-visual speech enhancement. arXiv preprint arXiv: 1804.04104 2018.
[2] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. SS- IL: separated softmax for incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 824-833, 2021.
[3] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) forget. In Proceedings of the European conference on computer vision (ECCV), pages 139-154, 2018.
[4] Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general contin- ual learning method based on complementary learning system. arXiv preprint arXiv:2201. 12604, 2022.
[5] Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory. In Proceedings of the IEEE/CVF international conference computer vision, pages 583-592, 2019.
[6] Ruchi Bhait, Pratibha Kumari, Dwarikanath Mahapatra, Abdulmotaleb El Saddik, and Mukesh Saini. Characterizing continual learning scenarios and strategies for audio analysis. arXiv preprint arXiv:2407.004665 2024.
[7] Albert Bregnian. Auditory scene analysis: Hearing in complex environments. Thinking in Sounds, pages 10-36, 1993.
[8] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 11115920-1990,, 2020.
[9] Francisco Castro, Manuel Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 233-248, 2018.
[10] Sungmin Cha, Sungjun Cho, Dasol Hwang, Sunwon Hong, Moontae Lee, and Taesup Moon. Rebalancing batch normalization for exemplar- based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20127-20136. 2023.
[11] Moitreya Chatterjee, Narendra Ahuja, and Anoop Cherian. Learning audio-visual dynamics using scene graphs for audio source separation. Advances in Neural Information Processing Systems, 35:16975-16988, 2022.
[12] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint 190.: 2019.
[13] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020.
[14] Jiaben Chen, Renrui Zhang, Dongze Lian, Jiaqi Yang. Ziyao Zeng, and Jianbo Shi. iquery: Instruments as queries for audio-visual sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14675-14686, 2023.
[15] Soo-Whan Chung, Soyeon Choe, Joon Son Chung, and Hong- Kang. Facefilter: Audio- visual speech separation using still images. arXiv preprint arXin:2005.070744 2020.
[16] Arthur Douillard, Yifu Chen, Amaud Dapogny, and Matthieu Cord. Plop: Learning without forgetting for continual semantic segmentation. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4040-4050, 2021.

---

[17] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Eccon-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 80- Springer, 2020.
[18] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen the cocktail party: speaker- independent audio-visual model for speech separation. arXiv preprint arXiv: 1804 00004.036// 2015 8.
[19] Enrico Fini, Victor G Turriis Da Costa, Xavier Alameda-Pinedas Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models are continual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9621-96223 1930,, 2022.
[20] Aviv Gabbay, Asaph Shamir, and Shmuel Peleg. Visual speech enhancement. arXiv preprint 17 2017.
[21] Chuang Gan, Deng Huang, Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10478-10487, 2020.
[22] Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV), pages 35 53, 2018.
[23] Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1189-38885 2019.
[24] Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. arXiv preprint arXiv: / /9033 04476, 2019.
[25] Simon Haykin and Zhe Chen. The cocktail party problem. Neural computation, 17(9))18755 1902, 2005.
[26] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 831-839, 2019.
[27] Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Davis: High-quality audio-visual separation with generative diffusion models. arXiv preprint arXiv:2308.00 222 2023.
[28) Ching Yi Hung, Cheng- Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu- Song Chen. Compacting, picking and growing for unforgetting continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[29] Minsoo Kang, Jaeyoo Park, and Bohyung Han. inssemnncreeetall learning by knowledge distillation with adaptive feature consolidation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16050-16059, 2022.
[30] Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual learning of language models. In International Conference on Learning Representations (ICLR), 2023.
[31] Sanghwan Kim, Lorenzo Noci, Antonio Orvieto, and Thomas Hofmann. Achieving better stability-plastice plasticity trade-off via auxiliary networks in continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11930-11939, 2023.
[32] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):52113522 2017.
et al.
12

---

[33] Saksham Singh Kushwaha. Analyzing the effect of equal-angl spatial discretization on sound event localization and detection. In Proceedings of the 7th Detection and Classification of Acoustic Scenes and Events 2022 Workshop (DCASF2002)), 2022
[34] Saksham Singh Kushwaha and Magdalena Fuentes. A multimodal prototypical approach for unsupervised sound classification. arXiv preprint arXiv:2306. 16601,000, 2023.
[35] Jaewoo Lee, Jachong Yoon, Wonjae Kim, Yunji Kim, and Sung Ju Hwang. Lifelong audio-video masked autoencoder with forget-robust localized alignments. arXiv preprint 10.00224 2023.
[36] Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data the wild. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 312-321, 2019.
[37] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. International Conference Muchine Leurning, pages 3925-3934. PMLR, 2019.
[38] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2955-2447, 2017.
[39] Yan-Shuo Liang and Wu-Jun Li. Adaptive plasticity improvement for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7816-7825, 2023.
[40] Zilin Luo, Yaoyao Liu, Bernt Schiele, and Qianru Sun. Class-incremental exemplar compression for class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11371-11333 1-11880, 2023.
[41] Haoxin Ma, Jiangyan Yi, Jianhua Tao, Ye Bai, Zhengkun Tian, and Chenglong Wang. Continual learning for fake audio detection. arXiv preprint arXiv:2104.07 004802 2021.
[42] Divyam Madaan, Jachong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representa- tional continuity for unsupervised continual learning. In International Conference on Learning Representations (ICLR), 2022.
[43] Fei Mi, Liangwei Chen, Mengjie Zhao, Minlie Huang, and Boi Faltings. Continual learning for natural language generation in rask-oriented dialog systems. In Findings of the Association for Computational Linguistics: EMNLP 2020, volume EMNLP 2020, pages 3461-3474, 2020.
[44] Shentong Mo, Weiguo Pian, and Yapeng Tian. Class-incremental grouping network for continual audio-visual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7788-7798. 2023.
[45] Shentong Mo and Yapeng Tian. Audio-visual grouping network for sound localization from mixtures. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 10565-10574, 2023.
[46] Xing Nie, Shixiong Xu, Xiyan Liu, Gaofeng Meng, Chunlei Huo, and Shiming Xiang. Bilateral memory consolidation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16026-16035, 2023.
[47] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference machine learning, pages
PMLR, 2017.
[48] Olcksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 11321-11329, 2019.
[49] Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisen- sory features. Proceedings of the European conference on computer vision (ECCV), pages 631-648, 2018.
13

---

[50] Jaeyoo Park, Minsoo Kang, and Bohyung Han. Clase-nncr learning for action recognition in videos. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 13698-137000 13707, 2021.
[51] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[52] Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34:16131-11144, 2021.
[53] Weiguo Pian, Shentong Mo, Yunhui Guo, and Yapeng Tian. Audio-visual class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7799-7811, 2023.
[54] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part II 16, pages 524 Springer, 2020.
[55] Senthil Purushwalkam, Pedro Morgado, and Abhinav Gupta. The challenges of continuous self-supervised learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, pages 702-721. Springer, 2022.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 7748-8763. PMLR, 2021.
[57] Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: incremental classifier and representation learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[58] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Enervenntion-MICAA. 2015: 18th International Conference, Munich, Germany, October 5.9, 2015, Proceedings, Part III 18, pages 234-3 144.21..6 Springer, 2015.
[59] Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 6107-6122, 2022.
[60] Charles Spence. Audiovisual multisensory integration. Acoustical science and technology, 28(2):61-70, 2007.
[61] Tejas Srinivasan, Rostami, and Jesse Thomason. CLiMB: A continual learning benchmark for vision-and- language tasks. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Ting- Yun Chang, Leticia Leonor Pinto Alva, Georgios Chochlakis, Mohammad
[62] Barry E Stein and Terrence R Stanford. Multisensory integration: current issues from the perspective of the single neuron. Nature reviews neuroscience, 9(4):255-266 2008
[63] Yiyang Su, Ali Vosoughi, Shijian Deng, Yapeng Tian, and Chenliang Xu. Separating in- visible sounds toward universal audiovisual scene-aware sound separation. arXiv preprint 2rXiv:230..11713. 2023.
[64] Elyse S Sussman. Integration and segregation in auditory scene analysis. The Journal of the Acoustical Society of America, 117(3):1285-1) 2005.
[65] Reuben Tan, Arijit Ray, Andrea Burns, Bryan A Plummer, Justin Salamon, Oriol Nieto, Bryan Russell, and Kate Saenko. Language -giided audio-visual source separation via trimodal consistency. In Proceedings of the IEEF/VV Conference on Computer Vision and Pattern Recognition, pages 10575-10584, 2023.
14

---

[66] Yu-Ming Tang, Yi-Xing Peng. and Wei-Shi Zheng. Learning to imagine: Diversify memory for incremental learning using unlabeled data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9549-9558, 2022.
[67] Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pages 2745-2755 2021.
[68] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization unconstrained videos. In Proceedings of the European conference on computer vision (ECCV), pages 247-263, 2018.
[69] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 15:10088-1009,, 2022.
[70] Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel PW Ellis, and John R Hershey. Into the wild with audioscope: Unsupervised audio-visual separation of on-screen sounds. arXiv preprint arXiv:2011.011433 2020.
[71] Efthymios Tzinis, Scott Wisdom, Tal Remez, and John R Hershey. Audioscopev2: Audio-visual attention architectures for calibrated open-domain on-screen sound separation. In European Conference Computer Vision, pages 368-385. Springer, 2022.
[72] Fu-Yun Yun Wang, Da- ReiWei Zhou, Han-Jia Ye, and De-Chuan Zhan. FOSTER: feature boosting and compression for caass-nncrementll learning. In In Proceedings of the European Conference on Computer Vision (ECCV), pages 398 414, 2022.
[73] Yu Wang, Nicholas Bryan, Mark Cartwright, Juan Pablo Bello, and Justin Salamon. Few- shot continual learning for audio classification. In ICASSP 2021-2021 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 321-325. IEEE, 2021.
[74] Zhepei Wang, Cem Subakan, Xilin Jiang, Junkai Wu, Efthymios Tzinis, Mirco Ravanelli, and Paris Smaragdis. Learning representations for new sound classes with continual self-supervised learning. IEEE Signal Processing Letters, 29:2607-2611, 2022.
[75] Zhepei Wang, Cem Subakan, Efthymios Tzinis, Paris Smaragdis, and I. Laurent Charlin. Continual learning of new sound classes using generative replay. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 308-312. IEEE, 2019.
[76] Zifeng Wang. Zizhao Zhang, Chen-Yu Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[77] Adam Weisser. Complex acoustic environments: Concepts, methods, and auditory perception. PhD thesis, PhD thesis). Macquarie University, Sydney, Australia. doi: 1959. 1559.14/1266534 2018.
[78] Jia-Wen Xiao, Chang- Zhang, Jickang Feng, Xialei Liu, Joost van de Weijer, and Ming-Ming Cheng. Endpoints weight fusion for class incremental semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7213, 2023.
[79] Xudong Xu, Bo Dai, and Dahua Lin. Recursive visual sound separation using minus- plus net. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 282-891, 2019.
[80] Shipeng Yan, Lanqing Hong, Hang Xu, Jianhua Han, Tinne Tuytelaars, Zhenguo Li, and Xuming He. Generative negative text replay for continual vision-language pretraining. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 22-38. Springer, 2022.
[81] Yuxin Ye, Wenming Yang, and Yapeng Tian. Lavss: Location-guided audio-visual spatial audio separation. arXiv preprint arXiv:2310 2023.
15

---

[82] Xiaohui Zhang, Jiangyan Yi, Jianhua Tao, Chenglong Wang, and Chu Yuan Zhang. Do you remember? overcoming catastrophic forgetting for fake audio detection. International Conference Machine Learning, pages 41819-41831. PMLR, 2023.
[83] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proceedings of the IEEE International Conference on Computer Vision, pages 1735-1744, 2019.
[84] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In The European Conference on Computer Vision (ECCV), September 2018.
[85] Dei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu, Deep class- incremental learning: A survey. arXiv preprint arXiv:2302.033644 2023.
[86] DeiWei Zhou, Yuanhan Zhang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for vision-language models. arXiv preprint arXiv:2305. 19270, 2023.
[87] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vision, pages 350-368. Springer, 2022.
[88] Lingyu Zhu and Esa Rahtu. Visually guided sound source separation and localization using self-supervised motion representations. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 1289- 1299, 2022.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Sec. 4.4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our paper does not contain theoretical results or proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We include implementation paragraph section in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: Our source code has been released. Datasets used our experiments are public dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
httpps.ccppublic//ee / ssi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include an implementation paragraph section in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Because of the limitation of computation resource, we didn't 't report the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information of the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eerripppepe htcppullic/ttiisstiiiie cecuuidieeedddeeeeeee
Answer: [Yes]
Justification: Our paper does not violate any respect of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses the potential societal impacts. We include Sec. 5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models), used the paper, are properly credited and the license and terms of use explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thwode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced in the paper
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "ar9AJzFRnq", "reviewer": "Reviewer_NCtq", "summary": "This work proposes a continual audio-visual sound separation framework to mitigate the catastrophic forgetting problem", "questions": "Please seem the weakness part.", "limitations": "1. Limited novelty.\n2. Insufficient experiments", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "As I learned, this is the first work that focuses on the catastrophic forgetting problem in audio-visual separation task.", "weaknesses": "1. The font size in Figure 2 is too small, reducing the readability of this paper.\n2. The technique novelty is limited, the model architecture is totally the same as iQuery, and the most brightness point of this work is just proposes a cross-modal similarity distillation constraint, however, it just the combination of the contrastive loss implemented on the modalities and features extracted from different training step.\n3. Experiments are not enough, all experiments are conducted on Music21, which just contains limited data among 21 classes. Experiments conducted on Music [1], VGGSound [4], and AVE datasets [2-3] can provide a more comprehensive evaluation.\n[1] The sound of pixels.\n[2] Audioset: An ontology and human-labeled dataset for audio events.\n[3] Audio-visual event localization in unconstrained videos\n[4] Vggsound: A large-scale audio-visual dataset", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions! We address the raised concerns below. If there are any additional questions, we are willing to address them and revise our paper.\n\n> ### **Q1: The font size in Figure 2 is too small.**\n\nThank you for your suggestion! We have enlarged the font size in Figure 2.\n\n\n> ### **Q2: Same architecture as iQuery.**\n\nIn our work, we address the proposed continual audio-visual sound separation problem, focusing on the challenge of **training audio-visual sound separation models continuously while mitigating the catastrophic forgetting typically associated with continual learning**. To benchmark and fairly compare with existing continual learning approaches, we do not introduce new separation architectures. Instead, we concentrate on developing new learning approaches based on current state-of-the-art separators, i.e., iQuery [1], to tackle the catastrophic forgetting problem in separation, aligning with similar research goals in the broader continual learning literature.\n\nFurthermore, besides iQuery [1], in Table 1 of Section 4.2, we present experimental results based on another model architecture, Co-Separation [2]. Our method outperforms baseline methods in this context as well, demonstrating the generalizability of our proposed approach to different separation model architectures.\n\n\n> ### **Q3: Cross-modal Similarity Distillation Constraint is just the combination of the contrastive loss implemented on the modalities and features extracted from different training step.**\n\nWe would like to note that the proposed Cross-modal Similarity Distillation Constraint (CrossSDC) is a non-trivial contribution. CrossSDC is the first method to integrate previously learned features into the contrastive learning process of the current task. This enables maintaining cross-modal semantic similarity incrementally while distilling prior knowledge from older tasks directly into the current model. This simple yet effective unified training target/constraint effectively addresses catastrophic forgetting in our Continual Audio-Visual Sound Separation task. Extensive experiments with two different separation models validate its effectiveness.\n\n> ### **Q4: Experiments conducted on other datasets can provide a more comprehensive evaluation.**\n\nThank you for suggesting additional datasets such as Music, VGGSound, and AVE! Following common practice in recent audio-visual sound separation works, we use the Music-21 dataset as our main benchmark. Note that Music [3] is a smaller subset of Music-21. To further validate our method, we conducted experiments on the AVE dataset and have included the results in the Appendix. The results for Fine-tuning/LwF/PLOP/AV-CIL/ContAV-Sep (Ours) are 2.07/2.19/2.45/2.53/**2.72** for SDR and 5.64/6.43/6.11/6.64/**7.32** for SIR, respectively, which further demonstrate the effectiveness of our approach. For more details, please refer to Section A.3 in the Appendix. We will move these results to the main paper. Additionally, experiments on larger datasets such as VGGSound and AudioSet would be very interesting for future work, and we will discuss these in the main paper.\n\n\n[1] iQuery: Instruments as Queries for Audio-Visual Sound Separation. In *CVPR* 2023.\n\n[2] Co-Separating Sounds of Visual Objects. In *ICCV* 2019.\n\n[3] The Sound of Pixels. In *ECCV* 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DjPXb93t4i", "reviewer": "Reviewer_49mS", "summary": "This approach defines an audio-visual sound separation task where sound separation is the task and during fine-tuning novel classes are added, in the regime of continuous learning. The goal is to avoid catastrophic forgetting which typically leads to decreased performance in task performance on classes which were learned early but then little or no instances arrive in later training stages. They then present a method to solve this task with audio-visual data. Several losses are defined to achieve cross-modal similarity of embeddings through incremental tasks and preserve previous learned knowledge. \n\nTheir approach does not increase the number of logits to be able to accommodate more classes but instead creates separation masks, more suitable to audio separation than typical classification tasks.\n\nResults are compared on meaningful baselines even though this is basically a new task definition and ablating the different loss components shows that the combination yields the best performance.\n\nAdditional videos and investigations in the supplementary work complete the work.", "questions": "- It seems from the problem definition that all tasks have distinct classes. That seems like a special, even though the hardest possible, case. Do you have an intuition or looked into variations where actually a smaller part of instances from a specific class is part of all tasks? It seems also more realistic to assume that such a network is not necessarily fine-tuned on a dataset that is completely different from the previous one.\n\n- What are the hardware requirements for this? It seems CLIP is used even twice, once for each task and then the features are fed into transformers. Yet the NVIDIA RTC A5000 does seem to have only 24GB. Are there some compromises, e.g. image resolution, which were made due to the hardware available?\n\n- On a related note: What is the memory size? How does the memory and computation scale to new tasks as the memory keeps growing beyond 4 tasks?\n\n- The choice of 11kHz audio sampling seems odd compared with defaults in audio encoding or other approaches for other audio-visual tasks. What is the reasoning behind it?\n\n- How does this scale to hard examples, where both instruments are the same. The second example is already impressive with two woodwinds but it would be interesting how much the approach is able to derive from movement and how much from appearance in the visual modality.\n\n- It is interesting how the approach scales to n-classes in Figure 3 but how does it scale to n-tasks? In the appendix there is an investigation into old classes but an investigation of the performance measures for, say, one chosen class after 4,5,6,7... tasks would be interesting to judge the trend.", "limitations": "The limitations are properly listed and the authors are upfront about them.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Especially in the videos the performance on very similar instruments is impressive. The instruments share a very similar frequency profile and their separation is a very hard problem.\n\nThe paper is very clearly written, the task is clearly stated and well contrasted and localized within the domain.\n\nThe mathematical notation is very clear to follow.\n\nIn general the paper defines an interesting and realistic task, does a thorough investigation and states clearly its limitations.", "weaknesses": "The paper defines a task and then solves it, which is always a bit easier due to limited competition. A direct comparison between sound separation was excluded by the authors stating that they do not compete in that sense, which is a limitation.\n\nFigure 2 is very small and could be more self complete. The relation of the right side's illustration to the left side is not clear from the caption at all which only states the names of the concepts. Also, there is space left and right to increase the size. Probably this resizing was reduced to gain space. This leaves the text very small. It would be good to find space in a different way because the fonts are already extremely small within Figure 2. Some of the dataset details could probably go into the appendix.\n\nThis is not a summary paper. That the author's found so much related work is commendable but I would have preferred more detailed contrast to existing work, e.g. by picking out certain ideas and contributions, explaining them and then contrasting this work against them. Just having lists of 7 works and summarizing them as \"semantic segmentation\" seems more like a homework chore than a contribution to the paper. But my fellow reviewers may disagree.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and encouraging remarks! We address your questions below. If there are any additional questions, we are happy to address them and revise our paper.\n\n> ### **Q1: Direct comparison between sound separation.**\n\nThank you for your comment! Indeed, our paper focuses on developing continual learning methods to address catastrophic forgetting in audio-visual sound separation, rather than introducing new separation models. Our research demonstrates that existing audio-visual sound separation models suffer from catastrophic forgetting. We explored two separators, including the state-of-the-art iQuery model, and found that without advanced continual learning mechanisms, performance degrades significantly (e.g., SDR drops to 3.46, while our method achieves 7.33).\n\nTherefore, we believe a fair comparison involves applying various continual learning methods to the same separator model, as directly comparing different models without continual learning would result in significantly lower separation performance. This approach allows us to isolate the impact of continual learning techniques on the task. In future work, we will also explore developing a more general and robust audio-visual sound separation model architecture that inherently mitigates catastrophic forgetting.\n\n\n> ### **Q2: Figure 2 is too small.**\n\nThanks for the suggestion! We have found more space for Figure 2 and enlarged it, by moving the dataset details into the Appendix.\n\n\n> ### **Q3: More detailed contrast to existing work in related work.**\n\nThank you for your suggestion! We have added more details of existing works in the related work, such as \"...Park et al. [47] extend the knowledge distillation-based [2, 15] continual image classification method to the domain of video by proposing the time-channel distillation constraint. Douillard et al. [14] proposed to tackle the continual semantic segmentation task with multi-view feature distillation and pseudo-labeling...\". Due to space constraints, we cannot include the entire section here.\n\n\n> ### **Q4: All tasks have distinct classes. Is it a special case? Do you have an intuition or looked into variations where actually a smaller part of instances from a specific class is part of all tasks?**\n\nNo, this is not a special case. In continual learning, models are typically trained on a sequence of *new* tasks with distinct classes. This setup is standard for evaluating a method's ability to tackle catastrophic forgetting.\n\nIn scenarios where a smaller subset of instances from a specific class appears across all tasks, the forgetting issue for that specific class would be minimal or nonexistent. However, our proposed methods could still be applied to mitigate catastrophic forgetting in other classes and leverage knowledge from previous tasks to enhance fine-tuning on the shared classes in new tasks.\n\n\n> ### **Q5: Are there any compromises due to the hardware limitation?**\n\nGreat question! We did not compromise the original data quality, such as image resolution or compression. Since the video encoder, object detector, and image encoder are frozen, we can pre-extract these features offline and use them to train the subsequent parts of the model, which allows us to remove the computational load of these three frozen components from the GPU, enabling the training process on a single RTX A5000 GPU with 24GB of memory.\n\n> ### **Q6: What is the memory size? How does the memory and computation scale to new tasks as the memory keeps growing beyond 4 tasks?**\n\nGreat question! The memory size is set to 1 sample per old class. By keeping only 1 sample for each old class in memory, it becomes easy to scale to new tasks.\n\n> ### **Q7: Why choose 11kHz as the audio sampling rate?**\n\nWe choose 11kHz as the audio sampling rate as it is a common setting for the MUSIC-21 dataset in existing audio-visual sound separation papers [1,2,3].\n\n> ### **Q8: How does this scale to hard examples, where both instruments are the same? How much the approach is able to derive from movement and how much from appearance in the visual modality?**\n\nIn our model architecture iQuery [1], it consists of a spatial-temporal video encoder to extract the motion information from the given video. This allows the model to differentiate between instruments based on their motion patterns, even when they are from the same category and have similar audio characteristics.\n\nIn hard cases where instruments have a similar appearance, the separation capability primarily relies on motion cues from the visual modality. However, when instruments have distinct appearances, the visual modality's appearance features can provide sufficient visual cues to effectively guide sound separation.\n> ### **Q9: How does Figure 3 scale to n-tasks? One chosen class after following tasks.**\n\nNice suggestion! We randomly select one class (\"accordion\") from the first task and report its performance after training for each task. The results are shown in the following table (please see the following Official Comment with the title of \"Results table of question Q9: How does Figure 3 scale to n-tasks? One chosen class after following tasks\"), in which we can see that our method has an overall better performance compared to baselines. We can also observe that with incremental step increases, the performance of each method on this class tends to improve. This is because the memory data can be paired with data from previously unseen new classes to acquire new knowledge for old classes in subsequent tasks, as discussed in our paper. However, the performance drop compared to the upper bound still exists, demonstrating that catastrophic forgetting still occurs.\n\n[1] iQuery: Instruments as Queries for Audio-Visual Sound Separation. In *CVPR* 2023.\n\n[2] The Sound of Pixels. In *ECCV* 2018.\n\n[3] Co-Separating Sounds of Visual Objects. In *ICCV* 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z2PuN7ehX1", "reviewer": "Reviewer_5v9b", "summary": "The paper introduces ContAV-Sep, the goal is to continuously separate new sound classes while maintaining performance on previously learned classes, addressing the challenge of catastrophic forgetting in continual learning. ContAV-Sep employs a Cross-modal Similarity Distillation Constraint (CrossSDC) to preserve cross-modal semantic similarity across tasks and retain old knowledge, integrated into an audio-visual sound separation framework.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is a good solution to help maintain cross-modal semantic similarity across incremental tasks.\n2. The paper has a clear writing structure, and the figures and tables are easy to understand.", "weaknesses": "1. The performance improvement of ContAV-Sep (with iQuery) is not significant.\n2. What does iSTFT stand for in Figure 2? The symbols appearing in the figure need to be clearly explained in the caption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer highlighting our proposed approach and writing. We address the raised questions below and are happy to answer further questions.\n\n> ### **Q1: The performance improvement of ContAV-Sep (with iQuery) is not significant.**\n\nThanks for the comment! We would like to clarify that the performance improvement of our ContAV-Sep (with iQuery) is non-trivial. Compared to the baseline continual learning methods LwF/EWC/PLOP/EWF/AV-CIL, our proposed ContAV-Sep shows improvements of 0.57/0.68/0.30/1.98/0.47 in SDR and 0.78/0.54/0.25/2.20/0.42 in SIR, respectively. Furthermore, as demonstrated in the demo included in our supplementary material, our ContAV-Sep (with iQuery) achieves better separation results and sound quality compared to the baselines, highlighting its effectiveness in mitigating catastrophic forgetting.\n\n\n> ### **Q2: What does iSTFT stand for in Figure 2?**\n\nThank you for your question. iSTFT stands for Inverse Short-Time Fourier Transform, a common technique used to convert spectrograms back into audio signals. We have added this explanation to the figure caption for clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3VnXMcAUyx", "reviewer": "Reviewer_wmVP", "summary": "This paper introduces a novel task termed \"Continual Audio-Visual Sound Separation,\" aiming to address the practical challenge of separating sound sources for new classes in audio-visual scenarios while retaining performance on previously learned classes. This task is inherently challenging due to the inherent risk of catastrophic forgetting, where models trained on new data often exhibit performance degradation on previously learned classes. To tackle this challenge, the authors propose ContAV-Sep, a novel framework incorporating a Cross-modal Similarity Distillation Constraint (CrossSDC). This constraint preserves cross-modal semantic similarity across incremental tasks by enforcing both instance-aware and class-aware similarities through a combination of contrastive loss and knowledge distillation. Notably, CrossSDC integrates knowledge from past tasks into the contrastive learning process, ensuring the retention of previously acquired cross-modal correlations. Experiments on the MUSIC-21 dataset demonstrate that ContAV-Sep significantly outperforms existing continual learning baselines in terms of standard sound separation metrics (SDR, SIR, SAR) across multiple audio-visual sound separation base models. This work highlights the importance of cross-modal semantic correlation in continual audio-visual learning and provides a novel, effective solution for mitigating catastrophic forgetting in this domain.", "questions": "The classes seem a bit limited, would the authors want to generalize their experiments to a larger dataset like audioset and show how their method performs there? I think that would be even more interesting since the distribution of classes in Audioset has a long tail for multiple not so prevalent sounds (e.g. consider sounds that are not represented by Speech or music)", "limitations": "I think the authors include several limitations of their work, they should also identify the memorization of individual user’s data inside the memory of their continual learning method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Here are four strengths of the paper, presented as a list:\n- Clearly identifies and addresses a novel and practical problem in audio-visual sound separation: continual learning in this domain. It is important to mention that this might be an important problem in the field and not so many people have provided good solutions for this problem.\n- Proposes a novel framework, ContAV-Sep,  with a well-defined Cross-modal Similarity Distillation Constraint (CrossSDC) to tackle catastrophic forgetting.\n- Empirically demonstrates the effectiveness of ContAV-Sep on the MUSIC-21 dataset, showcasing significant performance improvements over strong baselines.\n- Provides a thorough analysis of the results, including ablation studies and exploration of memory size effects, highlighting the contributions of different components of the proposed method.", "weaknesses": "- Please replace the “mask” variable with something appropriate like \\widehat{\\mathbf{m}}\n- Please fix weird fontsizes like the one in equation 9, in general the manuscript does not seem polished enough for a NeurIPS submission.\n- Differences smaller than <0.1 dB in terms of SNR metrics are not either significant nor hearable (I would even argue for 0.5 dB but let’s follow the literature in this one), thus I would suggest rounding up all those performance numbers to a one decimal precision (it would also make the Tables less cluttered).\n- I would like to see an even larger ablation in Table 3 to show the full extent of how the performance deviates with an even larger amount of samples per class and not only 4, a graph would make the visualization better here cause currently it does not convey any meaningful message. \n- The authors do not make a thorough investigation on previous works in the literature that employ continual learning techniques for related sound processing tasks. I will refer here only a couple of the works that I am aware of like in [A, B] but I am almost certain that there is no lack of thereof to try to include other works and try to make some empirical or theoretical conclusions on how those methods can become interconnected, employed together and in general how they relate.\n\n\n\n[A] Wang, Z., Subakan, C., Tzinis, E., Smaragdis, P. and Charlin, L., 2019, October. Continual learning of new sound classes using generative replay. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) (pp. 308-312). IEEE.\n\n[B] Wang, Z., Subakan, C., Jiang, X., Wu, J., Tzinis, E., Ravanelli, M. and Smaragdis, P., 2022. Learning representations for new sound classes with continual self-supervised learning. IEEE Signal Processing Letters, 29, pp.2607-2611.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions! We appreciate your recognition of the novelty in our work's problem formulation and approach, as well as its effectiveness demonstrated in our experiments. We address your questions below and welcome any further inquiries.\n\n> ### **Q1: Replace the \"**mask**\" variable.**\n\nThank you for your suggestion. We have replaced the \"**mask**\" variable with a variable **m**.\n\n> ### **Q2: Weird fontsizes like the one in equation 9.**\n\nThank you for your suggestion. We have fixed this fontsize issue.\n\n> ### **Q3: Rounding up all performance numbers to a one decimal precision.**\n\nNice suggestion. We have updated performance numbers to one decimal precision in our paper.\n\n> ### **Q4: Larger amount of samples per class in memory. A graph would make the visualization better.**\n\nThank you for your suggestion! We conducted experiments with the number of samples per class in memory with 10, 20, and 30. The results are shown below. Moreover, as per your advice, we have included a graph in our paper to better show the impact of different sample sizes per class in memory.\n\n\n| # of samples per class | SDR  | SIR  | SAR  |\n|------------------------|------|------|------|\n| 10                     | 9.0  | 15.2 | 13.7 |\n| 20                     | 9.4  | 15.9 | 13.7 |\n| 30                     | 10.1 | 16.6 | 14.1 |\n\n\n\n> ### **Q5: Missing references in continual learning for sound processing tasks.**\n\nThanks for the suggestion! We have added relevant references on continual learning for sound processing tasks, including the two suggested works and others [1, 2, 3, 4], and discussed them in our revised paper:\n\n[1] Ma, Haoxin, Jiangyan Yi, Jianhua Tao, Ye Bai, Zhengkun Tian, and Chenglong Wang. \"Continual learning for fake audio detection.\" arXiv preprint arXiv:2104.07286 (2021).\n\n[2] Bhatt, Ruchi, Pratibha Kumari, Dwarikanath Mahapatra, Abdulmotaleb El Saddik, and Mukesh Saini. \"Characterizing Continual Learning Scenarios and Strategies for Audio Analysis.\" arXiv preprint arXiv:2407.00465 (2024).\n\n[3] Wang, Yu, Nicholas J. Bryan, Mark Cartwright, Juan Pablo Bello, and Justin Salamon. \"Few-shot continual learning for audio classification.\" In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 321-325. IEEE, 2021.\n\n[4] Zhang, Xiaohui, Jiangyan Yi, Jianhua Tao, Chenglong Wang, and Chu Yuan Zhang. \"Do you remember? overcoming catastrophic forgetting for fake audio detection.\" In International Conference on Machine Learning, pp. 41819-41831. PMLR, 2023.\n\n> ### **Q6: Generalize the experiments to a larger dataset like AudioSet.**\n\nNice suggestion! Following common practice, we used the MUSIC dataset as our primary benchmark and included an additional dataset, AVE, in the appendix to validate audio-visual sound separation performance. Extending our approach to very large datasets like AudioSet would indeed be interesting. However, due to limited computational resources and time constraints, conducting extensive experiments on AudioSet is currently challenging. We will investigate this in our future work.\n\n> ### **Q7: Identify the memorization of individual user’s data inside the memory.**\n\nThank you for your suggestion. We will address the potential memorization of individual users' data within our continual learning method in the limitations section. This discussion will include the importance of future work to empirically analyze and prevent data memorization, ensuring the development of robust and privacy-preserving models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a7FbQENohq", "reviewer": "Reviewer_tY1x", "summary": "The paper proposes a novel continual audio-visual sound separation task, aimed at continuously separating new categories of sound sources while maintaining the performance of previously learned categories.", "questions": "1. In Fig. 1, the authors illustrate the continual audio-visual sound separation task. Is the significant performance gap between ContAV-Sep and the fine-tuning method due to the small data scale and limited number of categories?\n\n2. I am curious whether this paper has been submitted twice, as I noticed the \"Anonymous ECCV Submission Paper ID 6024\" in the uploaded video materials. If that is the case, this paper will be directly rejected.\n\n3. Please clarify the difference between the innovation of the Cross-modal Similarity Distillation Constraint in this paper and the Dual-Audio-Visual Similarity Constraint in [1].", "limitations": "The innovation of the Cross-modal Similarity Distillation Constraint in this paper may overlap with the Dual-Audio-Visual Similarity Constraint from the previous work. It suggests that while the paper introduces a novel approach, there might not be a clear distinction or significant difference from the existing method in terms of the core innovative aspects.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. The structure of the entire paper is clear, and the expression is fluent.\n2. The experimental results can demonstrate the effectiveness of the method, and the quality of the separated audio shown in the visual video is also good.", "weaknesses": "1. The Cross-modal Similarity Distillation Constraint proposed in the paper includes two main innovations: 1) instance-aware semantic similarity and 2) class-aware semantic similarity. However, in terms of mathematical expression, compared to the Dual-Audio-Visual Similarity Constraint in reference [1], I believe the innovations of the two papers are almost identical.\n[1] Pian W, Mo S, Guo Y, et al. Audio-visual class-incremental learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 7799-7811.\n2. Although the paper has conducted experimental validation on the MUSIC-21 dataset, it does not provide sufficient information to assess the model's generalization ability on other datasets or in real-world scenarios, making it difficult to demonstrate its enhanced applicability in real scenarios where new sound sources are encountered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments! We appreciate the reviewer highlighting the clear structure, fluent expression, and effective experimental results of our paper. We address the raised concerns below and are willing to answer any further questions.\n\n> ### **Q1: Difference between the innovation of the Cross-modal Similarity Distillation Constraint (CrossSDC) and Dual-Audio-Visual Similarity Constraint (D-AVSC) in [1].**\n\nThank you for your question! The Dual-Audio-Visual Similarity Constraint (D-AVSC) [1] only focuses on the feature similarities within data generated by the current training model, while our proposed CrossSDC further integrate features learned by previous model into the contrastive loss. In this way, our CrossSDC incorporates the cross-modal similarities knowledge acquired from previous tasks into the contrastive loss, which not only facilitates the learning of cross-modal semantic similarities in new tasks but also ensures the preservation of previously acquired knowledge in audio-visual sound separation.\n\nMoreover, the experimental results also demonstrate the superiority of our method with CrossSDC (**7.33/13.55/13.01** for SDR/SIR/SAR) compared to the AV-CIL with D-AVSC [1] (6.86/13.13/12.31 for SDR/SIR/SAR).\n\n\n> ### **Q2: Generalization ability on other datasets.**\n\nThanks for the suggestion! Besides the MUSIC-21 dataset, we conducted experiments on the AVE [2] dataset and have included the experimental results in Appendix. The results of Fine-tuning/LwF/PLOP/AV-CIL/ContAV-Sep(Ours) are 2.07/2.19/2.45/2.53/**2.72** and 5.64/6.43/6.11/6.64/**7.32** for SDR and SIR, respectively, which further demonstrates the effectiveness of our proposed method. For more details, please kindly see Section A.3 in Appendix.\n\n\n> ### **Q3: Is the significant performance gap between ContAV-Sep and the fine-tuning method due to the small data scale and limited number of categories?**\n\nWe would like to clarify that the significant performance gap between ContAV-Sep and fine-tuning is not due to the small data scale and limited number of categories. Experiments on a larger dataset, AVE [2], demonstrate a similar performance gap. On AVE, fine-tuning achieves an SDR of 2.07 and SIR of 5.64, while ContAV-Sep achieves **2.72** and **7.32**, respectively. This persistent gap underscores that the difference is not solely due to the dataset. For further details, please refer to Section A.3 in the Appendix.\n\n\n> ### **Q4: Whether this paper has been submitted twice?**\n\n We confirm that this paper is not a dual submission. The current version is the revised submission of our previously withdrawn paper. We apologize for the oversight in the teaser of the demo video, which contained a typo. This issue has been corrected in our demo.\n\n\n\n[1] Audio-Visual Class-Incremental Learning. In *ICCV* 2023.\n\n[2] Audio-Visual Event Localization in Unconstrained Videos. In *ECCV* 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes the continual audio separation task. The goal of this task is to adapt an audio-visual separation model to new source classes while maintaining separation performance on previously-learned tasks, in a continual fashion. The proposed approach uses the iQuery model as a foundation, and adds a "cross-modal similarity distillation constraint (CrossSDC)", which enforces both instance-aware and class-aware cross-modal similarities through a combination of contrastive loss and knowledge distillation, which prevent catastrophic forgetting. The approach is evaluated on MUSIC21, and after rebuttal, also VGGSound.

The majority of reviewers recommend at least borderline accept, especially after the author rebuttal and discussion. Particular strengths mentioned include:

1. Clearly written paper that is easy to understand, with good motivation for the problem addressed (continual audio-visual separation) and for the proposed solution (CrossSDC to prevent catastrophic forgetting)

2. Demos in the included supplementary are impressive and convincing.

3. Thorough evaluation and analysis of results, including ablations, especially of memory size.

Weaknesses mentioned include the following, along with corresponding 

1. Concern about novelty, as the method is based on the prior iQuery method, and also that similar contrastive methods to the proposed CrossSDC have been used in potentially similar contexts before. The most critical reviewer (NCtq) considers the combination of these two prior works and the application of CrossSDC to the related task of continual audio-visual separation to fall short as a significant contribution. I agree this isn't a completely novel contribution, but to me (and other reviewers) it seems that the application to a new task is interesting and novel enough.

2. Inital concern about lack of generalization, as the initial results were only on MUSIC21. The authors presented results on more general VGGSound data in the rebuttal, which convinced reviewers that the method is more generally applicable.

3. Some lack of discussion of prior works, e.g. audio-only continual separation works.

Given that the majority of reviewers find value in this work, especially after the author rebuttal and dsicussion, I think it is over the bar of acceptance, and an interesting contribution to the areas of audio-visual source separation and continual learning.
Decision: Accept (poster)