Continual Audio-Visual Sound Separation

Authors: weiguo  pian
yiyang  nan
shijian  deng
shentong  mo
yunhui  guo
yapeng  tian
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 09:51:06

Abstract: In this paper, we introduce a novel continual audio-visual sound separation task, aiming to continuously separate sound sources for new classes while preserving performance on previously learned classes, with the aid of visual guidance. This problem is crucial for practical visually guided auditory perception as it can significantly enhance the adaptability and robustness of audio-visual sound separation models, making them more applicable for real-world scenarios where encountering new sound sources is commonplace. The task is inherently challenging as our models must not only effectively utilize information from both modalities in current tasks but also preserve their cross-modal association in old tasks to mitigate catastrophic forgetting during audio-visual continual learning. To address these challenges, we propose a novel approach named ContAV-Sep ($\textbf{Cont}$inual $\textbf{A}$udio-$\textbf{V}$isual Sound $\textbf{Sep}$aration). ContAV-Sep presents a novel Cross-modal Similarity Distillation Constraint (CrossSDC) to uphold the cross-modal semantic similarity through incremental tasks and retain previously acquired knowledge of semantic similarity in old models, mitigating the risk of catastrophic forgetting. The CrossSDC can seamlessly integrate into the training process of different audio-visual sound separation frameworks. Experiments demonstrate that ContAV-Sep can effectively mitigate catastrophic forgetting and achieve significantly better performance compared to other continual learning baselines for audio-visual sound separation. Code is available at: https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024.

Main paper: 
 introduction: 
 Humans can effortlessly separate and identify individual sound sources in daily experience [25, 7, 64, 33] . This skill plays a crucial role in our ability to understand and interact with the complex auditory environments that surround us [34] . However, replicating this capability in machines remains a significant challenge due to the inherent complexity of real-world auditory scenes [7, 77] . Inspired by the multisensory perception of humans [62, 60] , audio-visual sound separation tackles this challenge by utilizing visual information to guide the separation of individual sound sources in an audio mixture.Recent advances in deep learning have led to significant progress in audio-visual sound separation [84, 23, 21, 67, 14, 65, 81, 63, 11, 71] . Benefiting from more advanced architectures (e.g., U-Net [84, 23] , Transformer [14] , and diffusion models [27] ) and discriminative visual cues (e.g., grounded visual objects [67] , motion [83] , and dynamic gestures [21] ), audio-visual separation models are able to separate sounds ranging from domain-specific speech, musical instrument sounds to open-domain general sounds within training sound categories. However, a limitation of these studies is their focus on scenarios where all sound source classes are presently known, overlooking the potential inclusion of unknown sound source classes during inference in real-world applications. This oversight leads to the catastrophic forgetting issue [32, 3] , where the fine-tuning of models on new classes detrimentally impacts their performance on previously learned classes. Despite Chen et al. [14] demonstrating that their iQuery model can generalize to new classes well through simple fine-tuning, it still suffers from the catastrophic forgetting problem on old classes. This prevents the trained models from 38th Conference on Neural Information Processing Systems (NeurIPS 2024). continuously updating in real-world scenarios, impeding their adaptability to dynamic environments. The question how to effectively leverage visual guidance to continuously separate sounds from new categories while preserving separation ability for old sound categories remains open. To bridge this gap, we introduce a novel continual audio-visual sound separation task by integrating audiovisual sound separation with continual learning principles. The goal of this task is to develop an audio-visual model that can continuously separate sound sources in new classes while maintaining performance on previously learned classes. The key challenge we need to address is catastrophic forgetting during continual audio-visual learning, which occurs when the model is updated solely with data from new classes or tasks, resulting in a significant performance drop on old ones. We illustrate our new task and the catastrophic forgetting issue in Fig. 1 .Unlike typical continual learning problems such as task-, domain-, or classincremental classification in visual domains [2, 57, 38, 53, 85] , which result in progressively increasing logits (or probability distribution) across all observed classes at each incremental step, our task uniquely produces fixed-size separation masks throughout all incremental steps. In this context, each entry in the mask does not directly correspond to any specific classes. Additionally, the new task involves both audio and visual modalities. Therefore, simply applying existing visual-only methods cannot fully exploit and preserve the inherent cross-modal semantic correlations. Very recently, Pian et al. [53] and Mo et al. [44] extended continual learning to the audio-visual domain, but both focused on classification tasks.To address these challenges, in this paper, we propose a novel approach named ContAV-Sep (Continual Audio-Visual Sound Separation). Upon the framework, we introduce a novel Cross-modal Similarity Distillation Constraint (CrossSDC) to not only maintain the cross-modal semantic similarity through incremental tasks but also preserve previously learned knowledge of semantic similarity in old models to counter catastrophic forgetting. The CrossSDC is a generic constraint that can be seamlessly integrated into the training process of different audio-visual sound separators. To evaluate the effectiveness of our proposed ContAV-Sep, we conducted experiments on the MUSIC-21 dataset within the framework of continual learning, using the state-of-the-art audio-visual sound separation model iQuery [14] and a representative audio-visual sound separation model Co-Separation [23] , as our separation base models. Experiments demonstrate that ContAV-Sep can effectively mitigate catastrophic forgetting and achieve significantly better performance than other continual learning baselines. In summary, this paper contributes follows:(i) To explore more practical audio-visual sound separation, in which the separation model should be generalized to new sound source classes continually, we pose a Continual Audio-Visual Sound Separation task that trains the separation model under the setting of continual learning. To the best of our knowledge, this is the first work on continual learning for audio-visual sound separation.(ii) We propose ContAV-Sep for the new task. It uses a novel cross-modal similarity distillation constraint to preserve cross-modal semantic similarity knowledge from previously learned models.(iii) Experiments on the MUSIC-21 dataset can validate the effectiveness of our ContAV-Sep, demonstrating promising performance gain over baselines. 
related work: 
 Audio-Visual Sound Separation. Audio-visual sound separation aims to separate individual sound sources from an audio mixture guided by visual cues. A line of research emerges under various scenarios, such as separating musical instruments [23, 84, 79, 21, 83, 67] , human speech [20, 1, 18, 49, 15] , or sound sources in in-the-wild videos [22, 70] . Many frameworks and methods have been proposed to address challenges within specific problem settings. For instance, the extraction of face embeddings proves beneficial for speech audio separation [18] . Moreover, incorporating object detection can provide an additional advantage [23, 22] . The utilization of trajectory optical flows to leverage temporal motion information in videos, as demonstrated by [83] , also yields improvements. In this work, not competing on designing stronger separators, we would advance the exploration of the audio-visual sound separation within the paradigm of continual learning. We investigate how a model can learn to consistently separate sound sources from sequential separation tasks without forgetting previously acquired knowledge.Continual Learning. The field of continual learning has drawn significant attention, especially in visual domains, with various approaches addressing this challenge. Notable among these are regularization-based methods, exemplified in works such as [32, 3, 31, 39] . These approaches involve applying regularization to crucial parameters associated with old tasks to maintain the model's capabilities and during incremental steps, less important parameters are given higher priority for updates compared to important ones. Conversely, several works [57, 9, 5, 26, 12, 54, 8, 10, 40] applied rehearsal-based pipelines to enable the model review previously learned knowledge. For instance, Rebuffi et al. [57] proposed one of the most representative exemplars selection strategy Nearest-Meanof-Exemplars (NME), selects the most representative exemplars in each class based on the distance to the feature center of the class. Meanwhile, pseudo-rehearsal [47, 48, 66] employs generative models to create pseudo-exemplars based on the estimated distribution of data from previous classes. Moreover, architecture-based/dynamic architecture methods [52, 4, 46, 24, 28, 37, 40] proposed to modify the model architecture itself to enable the model to acquire new knowledge while mitigating the forgetting of old knowledge. Specifically, Pham et al. [52] proposed a dual network architecture, in which one is to learn new tasks while the other one is for retaining knowledge learned from old tasks. Wang et al. [72] combined the dynamic architecture and distillation constraint to mitigate the issue of continual-increasing overhead problem in dynamic architecture-based continual learning method. However, above studies mainly concentrate on the area of continual image classification. Recently, researchers also explored other continual learning scenarios beyond image classification. For instance, Park et al. [50] extend the knowledge distillation-based [2, 17] continual image classification method to the domain of video by proposing the time-channel distillation constraint. Douillard et al. [16] proposed to tackle the continual semantic segmentation task with multi-view feature distillation and pseudo-labeling. Xiao et al. [78] further addressed the continual semantic segmentation problem through weights fusion strategy between old and current models. Wang et al. [75] addressed the continual sound classification task through generative replay. Furthermore, continual learning has also been explored in the domain of language/vision-language learning tasks [30, 43, 59, 61, 19, 86] , self-supervised representation learning [19, 42, 55, 80, 35, 74] , audio classification [6, 73] and fake audio detection [41, 82] , etc. Despite the success of existing continual learning methods in various scenarios, their applicability in the domain of continual audio-visual sound separation is still unexplored. Although Pian et al. [53] and Mo et al. [44] proposed to tackle the catastrophic problem in audio-visual learning, their studies mainly concentrated in the area of audio-visual video classification. In contrast to existing works in continual learning, in this paper, we delves into the continual audio-visual sound separation, aiming to tackle the challenge of catastrophic forgetting specifically in the context of separation mask prediction for complicated mixed audio signals within joint audio-visual modeling. The goal of the task is to utilize the corresponding visual guidance v 1 and v 2 to predict the ratio masks for reconstructing the two individual audio signals. This process can be formulated as:EQUATIONwhere F Θ is the separation model with trainable parameters Θ. And then, the original sound signals s 1 and s 2 are used to calculate the loss function for optimizing the model:EQUATIONwhere D denotes the training set, and L is the loss function between the prediction and ground-truth.Continual Audio-Visual Sound Separation. Our proposed continual audio-visual sound separation task aims to train a model F Θ continually on a sequence of T separation tasks {T 1 , T 2 , ..., T T }. For the t-th task T t (incremental step t), we have a training setD t = {V i (s i , v i ), y i t } nt i=1, where i and n t denote the i-th video sample and the total number of samples in D t respectively, and y i t ∈ C t is the corresponding sound source class of video V i , where C t is the training sound class label space of task T t . For any two tasks T t1 and T t2 and their corresponding training sound class label space C t1 and C t2 , we have C t1 ∩ C t2 = ∅. Following previous works in continual learning [57, 2, 53, 44, 29, 76] , for a task T t , where t > 1, holding a small size of memory/exemplar set M t to store some data from old tasks is permitted in our setting. Therefore, with the memory/exemplar set M t , all available data that can be used for training in task T t (t > 1) can be denoted as D ′ t = D t ∪ M t . Finally, the training process of Eq. 2 in our continual audio-visual sound separation setting can be denoted as:EQUATIONwhich means that the new model Θ t is obtained by updating the old model Θ t-1 which was trained on the previous task, using the current task's available data D ′ t . After the training process for task T t with D ′ t , the updated model will be evaluated on a testing set which includes video samples from all seen sound source classes up to continual step t (task T t ). And the evaluation also follows the common "mix-and-separation" strategy. During this continual learning process, the model's separation performance on the previously learned tasks drops significantly after training on new tasks. This learning issue is referred to as the catastrophic forgetting [32, 38, 3] problem, which poses a considerable challenge in continual audio-visual sound separation. 
overview: 
 To address the challenge of catastrophic forgetting in continual audio-visual sound separation, we introduce ContAV-Sep. This new framework, illustrated in Fig. 2 , consists of three key components: a separation base model, an output mask distillation module, and our proposed Cross-modal Similarity Distillation Constraint (CrossSDC). We use a recent state-of-the-art audio-visual separator: iQuery [14] as the base model of our approach, which contains a video encoder to extract the global motion feature, an object detector and image encoder to obtain the object feature, a U-Net [58] for mixture sound encoding and separated sound decoding, and an audio-visual Transformer to get the separated sound feature through multi-modal cross-attention mechanism and class-aware audio queries. For the object detector, we follow iQuery [14] and use the pre-trained Detic [87] , a universal object detector, to detect the sound source objects in each frame. For the video encoder and the image encoder, inspired by the excellent generalization ability of recent self-supervised pre-trained models, which has been proven to be effective and appropriate in continual learning [53] , we apply two self-supervised pre-trained models VideoMAE [69] and CLIP [56] as the video encoder and the image encoder, respectively. Note that, during the training process, the object detector, video encoder, and image encoder are frozen. Given a pair of videos V 1 (s 1 , v 1 ) and V 2 (s 2 , v 2 ), at incremental step t (task T t ), the U-Net audio encoder F AE t takes the mixed audio signal S obtained by mixing s 1 and s 2 as input, and generates the latent mixed audio feature. This process can be expressed as:f a t-1,1 f a t-1,n f m t-1,n f m t-1,1 f o t-1,1 f o t-1,n⊙ Projection F Θt F Θt-1 f m t-1 f a t-1 f o t-1 ❄ ❄ ❄ 🔥 🔥 🔥 🔥 … … … 🔥 🔥 f m t,1 f a t,1 f a t,n f o t,1 f m t,n f o t,n STFT Mix Audio Mix Spectrum ⊙ Mix Spectrum iSTFT Separated Audio M t-1 M t M t-1 M t L( M t , M t-1 )EQUATIONThen, the audio-visual Transformer F T rans.t is employed to generate the separated sound feature by taking the latent mixed audio feature and visual features as inputs:EQUATIONwhere f a,1 t denotes the separated sound feature of video V 1 ; Obj. 1 and M o. 1 denote the object and motion features extracted by the frozen pre-trained image and video encoders respectively from the visual signal v 1 of video V 1 , U o t (•) and U m t (•) are learnable projection layers to map the object and motion features into the same dimension. Similarly, we can also obtain the separated sound feature of V 2 guided by the associated visual features.The extracted separated sound feature and the latent mixed audio feature are combined to generate a mask. This mask is subsequently applied to the mixed audio, leading to the reconstruction of the separated sound spectrogram.EQUATIONwhere M 1 t and M 2 t denote the predicted masks for audio signals of video V 1 and V 2 , respectively; F AD t is the U-Net decoder at incremental step t; M LP t (•) denotes a MLP module; and ⊙ denotes channel-wise multiplication. The sound s 1 at this incremental step can be reconstructed by applying S ⊙ M 1 t and then performing an inverse STFT to obtain the audio waveform.Recent studies [53, 45] have highlighted the importance of cross-modal semantic correlation in audio-visual modeling. However, this correlation tends to diminish during subsequent incremental phases, which leads to catastrophic forgetting in our continual audio-visual sound separation task. To address this challenge, we propose a novel Cross-modal Similarity Distillation Constraint (CrossSDC) that serves two crucial purposes (1) maintaining cross-modal semantic similarity through incremental tasks, and (2) preserving previous learned semantic similarity knowledge from old tasks.CrossSDC preserves cross-modal semantic similarity from two perspectives: instance-aware semantic similarity and class-aware semantic similarity. Both similarities are enforced by integrating contrastive loss and knowledge distillation. Instead of exclusively focusing on the similarities within current and memory data generated by the current training model, CrossSDC incorporates the cross-modal similarity knowledge acquired from previous tasks into the contrastive loss. This integration not only facilitates the learning of cross-modal semantic similarities in new tasks but also ensures the preservation of previously acquired knowledge. In the incremental step t (t > 1), the instance-aware part of our CrossSDC can be formulated as:EQUATIONwhere 1[i = j] is an indicator that equals 1 when i = j, denoting that video samples V i and V j are the same video; The sim function represents the cosine similarity function with temperature scaling;The modalities mod 1 and mod 2 , where (mod 1 , mod 2 ) ∈ {(a, o), (a, m), (m, o)}, denote different pairs of features to be compared: separated sound and object features, sound and motion features, and motion and object features. Here, τ denotes the incremental step, for which we have:EQUATIONwhich means that, for current task's data D t , we calculate the contrastive loss using features from the current model (τ 1 = τ 2 = t), while for memory set data M t , we use features from both the old and current models (e.g., τ 1 = t and τ 2 = t -1). In this way, knowledge distillation would be integrated into the cross-modal semantic similarity constraint for the current task, which ensures better preservation of learned cross-modal semantic similarity from previous tasks.While the instance-aware similarity provides valuable semantic correlation modeling, it does not account for the class-level semantic correlations, which is also crucial for audio-visual similarity modeling. To capture and preserve the semantic similarity within each class across incremental tasks, we also incorporate a class-aware component specifically designed for inter-class cross-modal semantic similarity, which can be formulated as:EQUATIONIn this context, visual and audio features from two videos are encouraged to be close when they belong to the same class. The overall formulation of our CrossSDC is as follows:EQUATIONwhere λ ins and λ cls are two scalars that balance the two loss terms. In this way, the model captures and preserves semantic correlations not just between instances but also within the same classes.In the previous subsection, we introduced our proposed CrossSDC constraint. To effectively combine CrossSDC with the overall objective, we incorporate it alongside output distillation and the main separation loss function.Output distillation is a widely used technique in continual learning [38, 2, 53] to preserve the knowledge gained from previous tasks while learning new ones. In our approach, we utilize the output of the old model as the distillation target to preserve this knowledge. Note that we only distill knowledge for data from the memory set, as represented by:EQUATIONwhere M 1 t-1 and M 2 t-1 are predicted masks generated by the old model that is trained at incremental step t -1. For the loss function here, we follow [84, 14] and adopt the per-pixel L 1 loss [84] . For the main separation loss function, we also apply the per-pixel L 1 loss:EQUATIONFinally, our overall loss function is denoted as:EQUATIONIn alignment with the work of [76] , our framework maintains a compact memory set throughout incremental updates. Each old class is limited to a maximum number of exemplars. After completing training for each task, we adopt the exemplar selection strategies in [2, 53] by randomly selecting exemplars for each current class and combining these new exemplars with the existing memory set.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiments: 
 In this section, we first introduce the setup of our experiments, i.e., dataset, baselines, evaluation metrics, and the implementation details. After that, we present the experimental results of our ContAV-Sep compared to the baselines, as well as ablation studies. We also conduct experiments on the AVE [68] and the VGGSound [13] datasets, which contain sound categories beyond the music domain. We put the experimental results on the AVE and the VGGSound datasets, the comparison to the uni-modal semantic similarity preservation method, the performance evaluation on old classes in incremental tasks, and the visualization of separating results in the Appendix.Dataset. Following common practice [83, 88, 14] , we conducted experiments on MUSIC-21 [83] , which contains solo videos of 21 instruments categories: accordion, acoustic guitar, cello, clarinet, erhu, flute, saxophone, trumpet, tuba, violin, xylophone, bagpipe, banjo, bassoon, congas, drum, electric, bass, guzheng, piano, pipa, and ukulele. In our experiments, we randomly selected 20 of them to construct the continual learning setting. Specifically, we split the selected 20 classes into 4 incremental tasks, each of which involves 5 classes. The total number of available videos is 1040, and we randomly split them into training, validation, and testing sets with 840, 100, and 100 videos, respectively. To further validate the efficacy of our method across a broader sound domain, we conduct experiments using the AVE [68] and the VGGSound [13] datasets in the appendix.Baselines. We compare our proposed approach with vanilla Fine-tuning strategy, and continual learning methods EWC [32] and LwF [38] . As we mentioned before, typical continual learning methods, e.g., class-incremental learning methods, which yield progressively increasing logits (or probability distribution) across all observed classes at each incremental step and design specific technique in the classifier, we consider that these methods are not an optimal choice for our proposed continual audio-visual sound separation problem. Thus, considering that continual semantic segmentation problem has a more similar form compared to conventional class-incremental learning, we also select two state-of-the-art continual semantic segmentation methods PLOP [16] and EWF [78] as our baselines. Moreover, we compare our method to the recently proposed audio-visual continual learning method AV-CIL [53] , in which we adapt the original class-incremental version to the form of continual audio-visual sound separation by replacing their task-wise logits distillation with the output mask distillation. Further, we also present the experimental results of the Oracle/Upper Bound, which means that using the training data from all seen classes to train the model. For fair comparison, all compared continual learning methods and our ContAV-Sep use the same state-of-the-art separator, i.e. iQuery [14] , as the base separation model. Further, we also incorporate our proposed and baseline methods into another representative audio-visual sound separation model Co-Separation [23] .Notably, the Co-Separation model does not utilize the motion modality. Therefore, when CrossSDC is applied to Co-Separation, the (mod 1 , mod 2 ) in Eq. 7 and 9 is constrained to (mod 1 , mod 2 ) = (a, o).For baselines that involve memory sets, we ensure that each of them is allocated the same number of memory as our proposed method for fair comparison.Implementation Details. Following [14] , we use a 7-layers U-Net [58] as the audio net, and subsample the audio at 11kHz, each of which is approximately 6 seconds. We apply the STFT with the Hann window size of 1022 and the hop length of 256, to obtain the 512 × 256 Time-Frequency representation of each audio signal, followed by a re-sampling on the log-frequency scale to generate the magnitude spectrogram with T, F = 256. We set the video frame rate (FPS) to 1, and detect the object using the pre-trained universal detector Detic [87] to detect the sound source object on each frame, and then, each detected object is resized and randomly cropped to the size of 224 × 224. For the image encoder and the video encoder, we apply the self-supervised pre-trained CLIP [56] and VideoMAE [69] to yield the object feature and motion feature, respectively. For the audio-visual Transformer module, we follow the design in [14] . For all the baseline methods, we apply the same model architecture and modules with ours for them, including the mentioned Detic, CLIP, VideoMAE, audio-visual Transformer, etc. Please note that, during our training process, the pre-trained Detic, CLIP, and VideoMAE are frozen. In our proposed Cross-modal Similarity Distillation Constraint (CrossSDC), the balance weights λ ins and λ cls are set to 0.1 and 0.3, respectively. And the balance weight λ dist. for the output distillation loss is set to 0.3 in our experiments. For the memory set, we set the number of samples in each old class to 1, so as other baselines that involve the memory set. All the experiments in this paper are implemented by Pytorch [51] . We train our proposed method and all baselines on a NVIDIA RTX A5000 GPU. We follow previous works [67, 14] in sound separation, and evaluate the performance of all the methods using three common metrics in sound separation tasks: Signal to Distortion Ratio (SDR), Signal to Interference Ratio (SIR), and Signal to Artifact Ratio (SAR). The SDR measures the interference and artifacts, while SIR and SAR measure the interference and artifacts, respectively. In our experiments, we report the SDR, SIR, and SAR of all the methods after training at last incremental steps, i.e., testing results on all classes. For all these three metrics, higher values denote better results.The main experimental comparisons are shown in Tab. 1. Our proposed method, ContAV-Sep, outperforms the state-of-the-art baselines by a substantial margin. Notably, compared to baselines using state-of-the-art audio-visual sound separator iQuery [14] as the separation base model, ContAV-Sep achieves a 0.3 improvement in SDR over the compared best-performing method. Additionally, our method surpasses the top baseline by 0.25 in SIR and 0.41 in SAR. Furthermore, compared to continual learning baselines with Co-Separation [23] , our ContAV-Sep still outperforms other approaches. This consistent superior performance across different model architectures highlights not only the effectiveness but also the broad applicability and generalizability of our proposed CrossSDC.Our observations further demonstrate that retaining a small memory set significantly enhances the performance of each baseline method. For instance, for the iQuery-based continual learning methods, equipping LwF [38] with a small memory set results in improvements of 3.31, 3.99, and 1.94 on SDR, SIR, and SAR, respectively. Similarly, the addition of a small memory set to EWC [32] leads to enhancements of 2.98, 3.43, and 1.43 in the respective metrics. The memory-augmented version of PLOP [16] exhibits superior performance with margins of 3.21, 3.24, and 1.68 for SDR, SIR, and SAR, respectively. Finally, incorporating memory into EWF [78] results in improvements of 1.37, We also present the testing results of SDR, SIR, and SAR at each incremental step in Figures 3a, 3b , and 3c, respectively. Our method is consistently observed to outperform others in terms of SDR at all incremental steps. While our approach may not always produce the best SIR and SAR results at the intermediate steps (specifically, steps 2 and 3 for SIR, and step 3 for SAR), it ultimately achieves the highest performance at the final step. This demonstrates the robustness of our method, indicating minimal forgetting throughout the incremental learning process.In this subsection, we conduct an ablation study to investigate the effectiveness of our proposed CrossSDC. By removing single or multiple components of the CrossSDC, we evaluate the impact of each on the final results. The results of the ablation study are presented in Tab. 2. From the table, we can see that our full model achieves the best performance compared to the variants, which further demonstrates the effectiveness of our proposed CrossSDC.Moreover, we also discuss the effect of memory size on our proposed ContAV-Sep. In our main experiments, the default setting of the memory size is 1 sample per old class. In this subsection, we conduct experiments by increasing the memory size from 1 sample per old class to 30 samples per old class. The experimental results are shown in Tab. 3 and Figure 4 . Observations from the table indicate a positive correlation between the size of the memory and the overall performance metrics.As the memory size increases, there is a discernible trend of improvement in the results. Our experimental findings reveal that the utilization of a small memory set, even a single sample per old class, markedly improves the performance of each baseline method. This improvement is attributed to the ability of a single memory sample to pair with diverse samples from the current training set, thereby generating numerous effective training pairs. Consequently, this process enables the model to acquire new knowledge for old classes in subsequent tasks, as the memory data can be paired with data from previously unseen new classes -this is different from conventional continual learning tasks, where old classes do not acquire new knowledge in new tasks. This could be a potential reason why the baseline continual learning methods do not perform well in our continual audio-visual sound separation problem. In this work, our method also mainly focuses on preserving old knowledge of old tasks, which may prevent the model from acquiring new knowledge of old classes when training in new tasks. Recognizing this, we identify the exploration of this problem as a key avenue for future research in this field.Additionally, the base model architectures used in our approach and baselines require object detectors to identify sounding objects. Although iQuery [14] can supplement object features with global video representations, it may still suffer from undetected objects. It is a fundamental limitation of the object-based audio-visual sound separators [23, 14] . While our work, unlike previous efforts, does not compete on designing a stronger audio-visual separation base model, enhancing the robustness of sounding object detection presents a promising direction for future research. 
conclusion: 
 In this paper, we explore training audio-visual sound separation models under a more practical continual learning scenario, and introduce the task of continual audio-visual sound separation. To address this novel problem, we propose ContAV-Sep, which incorporates a Cross-modal Similarity Distillation Constraint to maintain cross-modal semantic similarity across incremental tasks while preserving previously learned semantic similarity knowledge. Experiments on the MUSIC-21 dataset demonstrate the effectiveness of our method in this new continual separation task. This paper opens a new direction for real-world audio-visual sound separation research.Broader Impact. Our proposed continual audio-visual sound separation allows the model to adapt to new environments and sounds without full retraining, which could enhance efficiency and privacy by reducing the need to transmit and store sensitive audio data.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper discusses the potential societal impacts. We include it in Sec. 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: In practice, the audio signal is first processed using the Short-Time Fourier Transform (STFT) to generate a spectrogram. For brevity, we will denote spectrogram magnitudes as s1, s2, and S. 


Decision: This paper proposes the continual audio separation task. The goal of this task is to adapt an audio-visual separation model to new source classes while maintaining separation performance on previously-learned tasks, in a continual fashion. The proposed approach uses the iQuery model as a foundation, and adds a "cross-modal similarity distillation constraint (CrossSDC)", which enforces both instance-aware and class-aware cross-modal similarities through a combination of contrastive loss and knowledge distillation, which prevent catastrophic forgetting. The approach is evaluated on MUSIC21, and after rebuttal, also VGGSound.

The majority of reviewers recommend at least borderline accept, especially after the author rebuttal and discussion. Particular strengths mentioned include:

1. Clearly written paper that is easy to understand, with good motivation for the problem addressed (continual audio-visual separation) and for the proposed solution (CrossSDC to prevent catastrophic forgetting)

2. Demos in the included supplementary are impressive and convincing.

3. Thorough evaluation and analysis of results, including ablations, especially of memory size.

Weaknesses mentioned include the following, along with corresponding 

1. Concern about novelty, as the method is based on the prior iQuery method, and also that similar contrastive methods to the proposed CrossSDC have been used in potentially similar contexts before. The most critical reviewer (NCtq) considers the combination of these two prior works and the application of CrossSDC to the related task of continual audio-visual separation to fall short as a significant contribution. I agree this isn't a completely novel contribution, but to me (and other reviewers) it seems that the application to a new task is interesting and novel enough.

2. Inital concern about lack of generalization, as the initial results were only on MUSIC21. The authors presented results on more general VGGSound data in the rebuttal, which convinced reviewers that the method is more generally applicable.

3. Some lack of discussion of prior works, e.g. audio-only continual separation works.

Given that the majority of reviewers find value in this work, especially after the author rebuttal and dsicussion, I think it is over the bar of acceptance, and an interesting contribution to the areas of audio-visual source separation and continual learning.

Reviews: review: paper_summary: paper_summary: This work proposes a continual audio-visual sound separation framework to mitigate the catastrophic forgetting problem
strength_weakness: strength_weakness: 1. The font size in Figure 2 is too small, reducing the readability of this paper.
2. The technique novelty is limited, the model architecture is totally the same as iQuery, and the most brightness point of this work is just proposes a cross-modal similarity distillation constraint, however, it just the combination of the contrastive loss implemented on the modalities and features extracted from different training step.
3. Experiments are not enough, all experiments are conducted on Music21, which just contains limited data among 21 classes. Experiments conducted on Music [1], VGGSound [4], and AVE datasets [2-3] can provide a more comprehensive evaluation.
[1] The sound of pixels.
[2] Audioset: An ontology and human-labeled dataset for audio events.
[3] Audio-visual event localization in unconstrained videos
[4] Vggsound: A large-scale audio-visual dataset
questions: questions: Please seem the weakness part.
limitations: limitations: 1. Limited novelty.
2. Insufficient experiments
paper_summary: paper_summary: This approach defines an audio-visual sound separation task where sound separation is the task and during fine-tuning novel classes are added, in the regime of continuous learning. The goal is to avoid catastrophic forgetting which typically leads to decreased performance in task performance on classes which were learned early but then little or no instances arrive in later training stages. They then present a method to solve this task with audio-visual data. Several losses are defined to achieve cross-modal similarity of embeddings through incremental tasks and preserve previous learned knowledge. 

Their approach does not increase the number of logits to be able to accommodate more classes but instead creates separation masks, more suitable to audio separation than typical classification tasks.

Results are compared on meaningful baselines even though this is basically a new task definition and ablating the different loss components shows that the combination yields the best performance.

Additional videos and investigations in the supplementary work complete the work.
strength_weakness: strength_weakness: The paper defines a task and then solves it, which is always a bit easier due to limited competition. A direct comparison between sound separation was excluded by the authors stating that they do not compete in that sense, which is a limitation.

Figure 2 is very small and could be more self complete. The relation of the right side's illustration to the left side is not clear from the caption at all which only states the names of the concepts. Also, there is space left and right to increase the size. Probably this resizing was reduced to gain space. This leaves the text very small. It would be good to find space in a different way because the fonts are already extremely small within Figure 2. Some of the dataset details could probably go into the appendix.

This is not a summary paper. That the author's found so much related work is commendable but I would have preferred more detailed contrast to existing work, e.g. by picking out certain ideas and contributions, explaining them and then contrasting this work against them. Just having lists of 7 works and summarizing them as "semantic segmentation" seems more like a homework chore than a contribution to the paper. But my fellow reviewers may disagree.
questions: questions: - It seems from the problem definition that all tasks have distinct classes. That seems like a special, even though the hardest possible, case. Do you have an intuition or looked into variations where actually a smaller part of instances from a specific class is part of all tasks? It seems also more realistic to assume that such a network is not necessarily fine-tuned on a dataset that is completely different from the previous one.

- What are the hardware requirements for this? It seems CLIP is used even twice, once for each task and then the features are fed into transformers. Yet the NVIDIA RTC A5000 does seem to have only 24GB. Are there some compromises, e.g. image resolution, which were made due to the hardware available?

- On a related note: What is the memory size? How does the memory and computation scale to new tasks as the memory keeps growing beyond 4 tasks?

- The choice of 11kHz audio sampling seems odd compared with defaults in audio encoding or other approaches for other audio-visual tasks. What is the reasoning behind it?

- How does this scale to hard examples, where both instruments are the same. The second example is already impressive with two woodwinds but it would be interesting how much the approach is able to derive from movement and how much from appearance in the visual modality.

- It is interesting how the approach scales to n-classes in Figure 3 but how does it scale to n-tasks? In the appendix there is an investigation into old classes but an investigation of the performance measures for, say, one chosen class after 4,5,6,7... tasks would be interesting to judge the trend.
limitations: limitations: The limitations are properly listed and the authors are upfront about them.
paper_summary: paper_summary: The paper introduces ContAV-Sep, the goal is to continuously separate new sound classes while maintaining performance on previously learned classes, addressing the challenge of catastrophic forgetting in continual learning. ContAV-Sep employs a Cross-modal Similarity Distillation Constraint (CrossSDC) to preserve cross-modal semantic similarity across tasks and retain old knowledge, integrated into an audio-visual sound separation framework.
strength_weakness: strength_weakness: 1. The performance improvement of ContAV-Sep (with iQuery) is not significant.
2. What does iSTFT stand for in Figure 2? The symbols appearing in the figure need to be clearly explained in the caption.
questions: questions: See weakness.
limitations: limitations: See weakness.
paper_summary: paper_summary: This paper introduces a novel task termed "Continual Audio-Visual Sound Separation," aiming to address the practical challenge of separating sound sources for new classes in audio-visual scenarios while retaining performance on previously learned classes. This task is inherently challenging due to the inherent risk of catastrophic forgetting, where models trained on new data often exhibit performance degradation on previously learned classes. To tackle this challenge, the authors propose ContAV-Sep, a novel framework incorporating a Cross-modal Similarity Distillation Constraint (CrossSDC). This constraint preserves cross-modal semantic similarity across incremental tasks by enforcing both instance-aware and class-aware similarities through a combination of contrastive loss and knowledge distillation. Notably, CrossSDC integrates knowledge from past tasks into the contrastive learning process, ensuring the retention of previously acquired cross-modal correlations. Experiments on the MUSIC-21 dataset demonstrate that ContAV-Sep significantly outperforms existing continual learning baselines in terms of standard sound separation metrics (SDR, SIR, SAR) across multiple audio-visual sound separation base models. This work highlights the importance of cross-modal semantic correlation in continual audio-visual learning and provides a novel, effective solution for mitigating catastrophic forgetting in this domain.
strength_weakness: strength_weakness: - Please replace the “mask” variable with something appropriate like \widehat{\mathbf{m}}
- Please fix weird fontsizes like the one in equation 9, in general the manuscript does not seem polished enough for a NeurIPS submission.
- Differences smaller than <0.1 dB in terms of SNR metrics are not either significant nor hearable (I would even argue for 0.5 dB but let’s follow the literature in this one), thus I would suggest rounding up all those performance numbers to a one decimal precision (it would also make the Tables less cluttered).
- I would like to see an even larger ablation in Table 3 to show the full extent of how the performance deviates with an even larger amount of samples per class and not only 4, a graph would make the visualization better here cause currently it does not convey any meaningful message. 
- The authors do not make a thorough investigation on previous works in the literature that employ continual learning techniques for related sound processing tasks. I will refer here only a couple of the works that I am aware of like in [A, B] but I am almost certain that there is no lack of thereof to try to include other works and try to make some empirical or theoretical conclusions on how those methods can become interconnected, employed together and in general how they relate.



[A] Wang, Z., Subakan, C., Tzinis, E., Smaragdis, P. and Charlin, L., 2019, October. Continual learning of new sound classes using generative replay. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) (pp. 308-312). IEEE.

[B] Wang, Z., Subakan, C., Jiang, X., Wu, J., Tzinis, E., Ravanelli, M. and Smaragdis, P., 2022. Learning representations for new sound classes with continual self-supervised learning. IEEE Signal Processing Letters, 29, pp.2607-2611.
questions: questions: The classes seem a bit limited, would the authors want to generalize their experiments to a larger dataset like audioset and show how their method performs there? I think that would be even more interesting since the distribution of classes in Audioset has a long tail for multiple not so prevalent sounds (e.g. consider sounds that are not represented by Speech or music)
limitations: limitations: I think the authors include several limitations of their work, they should also identify the memorization of individual user’s data inside the memory of their continual learning method.
paper_summary: paper_summary: The paper proposes a novel continual audio-visual sound separation task, aimed at continuously separating new categories of sound sources while maintaining the performance of previously learned categories.
strength_weakness: strength_weakness: 1. The Cross-modal Similarity Distillation Constraint proposed in the paper includes two main innovations: 1) instance-aware semantic similarity and 2) class-aware semantic similarity. However, in terms of mathematical expression, compared to the Dual-Audio-Visual Similarity Constraint in reference [1], I believe the innovations of the two papers are almost identical.
[1] Pian W, Mo S, Guo Y, et al. Audio-visual class-incremental learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 7799-7811.
2. Although the paper has conducted experimental validation on the MUSIC-21 dataset, it does not provide sufficient information to assess the model's generalization ability on other datasets or in real-world scenarios, making it difficult to demonstrate its enhanced applicability in real scenarios where new sound sources are encountered.
questions: questions: 1. In Fig. 1, the authors illustrate the continual audio-visual sound separation task. Is the significant performance gap between ContAV-Sep and the fine-tuning method due to the small data scale and limited number of categories?

2. I am curious whether this paper has been submitted twice, as I noticed the "Anonymous ECCV Submission Paper ID 6024" in the uploaded video materials. If that is the case, this paper will be directly rejected.

3. Please clarify the difference between the innovation of the Cross-modal Similarity Distillation Constraint in this paper and the Dual-Audio-Visual Similarity Constraint in [1].
limitations: limitations: The innovation of the Cross-modal Similarity Distillation Constraint in this paper may overlap with the Dual-Audio-Visual Similarity Constraint from the previous work. It suggests that while the paper introduces a novel approach, there might not be a clear distinction or significant difference from the existing method in terms of the core innovative aspects.

Comments: title: Response to reviewer tY1x
Thank you all for your time and consideration
Response to reviewer wmVP
Response to Reviewer tY1x
Response to the authors' rebuttal
Author-reviewer discussion ending soon
More details of the experiments on VGGSound
Response to Reviewer NCtq
Reply
Experiments on larger dataset with broader range of sound categories
Response to Reviewer NCtq
Reply
Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
Follow-up on the Rebuttal
Results table of question Q9: How does Figure 3 scale to n-tasks? One chosen class after following tasks.
comment: Thank you for raising your rating! We sincerely appreciate your insightful questions and feedback, and we're grateful for your patience in reviewing our detailed responses.

You've raised an excellent point. Our approach could potentially be extended to address other audio-visual prediction tasks such as sound source visual localization and audio-visual segmentation. However, these tasks present additional challenges, as they require learning very fine-grained correspondences between individual sound sources and their corresponding visual objects in order to accurately predict the visual regions associated with specific sounds in video frames. Furthermore, these tasks may only have access to weak supervision in the form of video-level labels during training, while needing to predict precise sounding regions or segmentation masks in testing. This would necessitate the development of novel multimodal continual learning approaches, opening up new technical challenges and exciting avenues for future research.
Based on the author's answer, although it elaborates a lot on the differences between continuous audio-visual separation and continuous audio-visual classification, it feels partially answering the question: **how significant is the value of combining two existing tasks to create a new one**, as tasks such as Continual Audio-Visual Segmentation may come up later, and they may all be similar in approach to this paper, making it it is difficult to assess their impact. But I've decided to raise my rating as you are actively working on these issues.
Dear Area Chair and Reviewers,

Thank you all for your time and effort in reviewing our submission!

We are sincerely grateful for the constructive feedback provided by all reviewers, which has significantly improved our work. We also appreciate the insightful discussions and the professional and respectful manner in which they were conducted throughout the rebuttal and discussion process. We have taken care to ensure our responses respect the opinions of all reviewers.

While we understand that some concerns may remain, we hope our explanations have addressed them satisfactorily. We remain committed to providing any further clarification or addressing additional questions that may arise.

Thank you again for your time and consideration!

Best regards,

The Authors of Paper 3807
Thank you so much for your positive support of our work! Regarding your question about how different methods use the same architecture, we would like to clarify that continual learning methods typically involve different strategies for training on new tasks, such as selective weight updating, memory buffer usage, or the incorporation of regularization techniques to prevent catastrophic forgetting. As a result, most continual learning methods can be viewed as specific training strategies that are independent of the model architecture. Therefore, a common experimental setting in continual learning research is to hold the model architecture constant across the proposed method and all baseline methods, allowing for a fair comparison of their training performance.

We will provide more details about the implementation details in appendix and release our source code for our model and the implemented baselines. We truly appreciate your support!
Thank you for your insightful questions! We want to clarify that solving audio-visual sound separation in a continual learning setting is non-trivial. Extending continual learning from classification to this sound separation is far from straightforward due to fundamental differences between the two.

In continual audio-visual classification, the goal is to accurately identify the category of an audio-visual clip by extracting and integrating discriminative features across tasks, while avoiding catastrophic forgetting. In contrast, audio-visual sound separation is inherently more complex. It requires using visual signals to disentangle mixed sounds into visually-aligned audio features, ultimately predicting ratio sound spectrogram masks for separating individual sources. Here, learning discriminative features alone is insufficient. Instead, preserving the relative correlations or similarities between audio and visual components is more critical and difficult in the separation task, as various sound sources can be mixed with a source from a known or unknown class.

Therefore, we consider Continual Audio-Visual Sound Separation a more challenging task than continual audio-visual classification. The core challenge lies in maintaining these relative audio-visual correlations or similarities to leverage visually indicated sound representations for separation, all while mitigating forgetting. This ensures the preservation of separation abilities learned from previous classes. Otherwise, catastrophic forgetting affects not only old classes but also new class separation, particularly when new sound is mixed with data from previously learned classes.

Moreover, in continual classification, the model is trained to predict continually incrementing logits among all previously seen classes. However, our proposed Continual Audio-Visual Sound Separation task focuses on addressing catastrophic forgetting in the context of separation mask prediction within joint audio-visual modeling. Unlike in continual classification, where the model adapts to an expanding output space as new classes are introduced, our task requires the model to consistently predict a fixed-size separation mask with values between 0 and 1 across all tasks. This presents a greater challenge because the model must retain its ability to effectively separate audio sources even as new tasks are introduced, without expanding the output space. Consequently, directly applying continual learning methods from classification to our proposed task is not feasible and requires significant adaptation, especially for methods involving techniques focused on partially updating the classifier for new classes, as evidenced by our experiments and demo.

Our CrossSDC, unlike D-AVSC in [1], goes beyond focusing solely on feature similarities within the current task’s training model. It integrates previously learned features into the contrastive learning process, establishing a cross-task audio-visual correlation/similarity constraint. This enables the preservation and maintenance of relative audio-visual correlation/similarities across current and all seen tasks. Our experiments demonstrate that CrossSDC significantly outperforms AV-CIL with D-AVSC on various datasets and with different separation architectures.

[1] Audio-Visual Class-Incremental Learning. In ICCV 2023.
Thanks for addressing some of my concerns and my questions. I congratulate the authors' rebuttal and the extra experiments especially for the larger dataset of VGGsound. I will thus, increase my score, although the comparisons with other methods in the literature are still not really easy to me how they used same model capacity etc.
Dear authors and reviewers,

Thank you for the robust discussions so far.

Reviewers 5v9b and wmVP: it would be very valuable to have your input to the discussion, and the authors have posted detailed responses to your reviews. Could you please respond to acknowledge that you have read and considered the reviews and author rebuttal, and post any remaining questions that would be useful to discuss with the authors? We only have about a day left in the discussion period, so please prioritize this.

Thanks in advance.

All the best,
Your AC
We’d like to provide more details about our VGGSound experiments to address the reviewer’s doubts.
We randomly selected the following 100 categories:

['playing theremin', 'donkey, ass braying', 'playing electronic organ', 'zebra braying', 'people eating noodle', 'airplane flyby', 'playing double bass', 'cat growling', 'footsteps on snow', 'playing tennis', 'black capped chickadee calling', 'bouncing on trampoline', 'playing steelpan', 'waterfall burbling', 'subway, metr', 'people clapping', 'chipmunk chirping', 'chopping food', 'people shuffling', 'elk bugling', 'alarm clock ringing', 'people booing', 'canary calling', 'chopping wood', 'people humming', 'lathe spinning', 'playing tuning fork', 'playing violin, fiddle', 'singing choir', 'playing timbales', 'children shouting', 'chicken crowing', 'car passing by', 'driving motorcycle', 'bull bellowing', 'lawn mowing', 'playing bugle', 'mouse squeaking', 'child singing', 'playing tympani', 'hair dryer drying', 'basketball bounce', 'driving snowmobile', 'train whistling', 'thunder', 'dog bow-wow', 'ocean burbling', 'cuckoo bird calling', 'sheep bleating', 'splashing water', 'air conditioning noise', 'cattle mooing', 'eagle screaming', 'air horn', 'playing bass guitar', 'sloshing water', 'tap dancing', 'running electric fan', 'playing ukulele', 'playing guiro', 'playing shofar', 'people sniggering', 'people whispering', 'people finger snapping', 'car engine idling', 'bathroom ventilation fan running', 'police car (siren)', 'roller coaster running', 'playing french horn', 'swimming', 'lighting firecrackers', 'playing electric guitar', 'playing castanets', 'people babbling', 'arc welding', 'wood thrush calling', 'wind rustling leaves', 'playing darts', 'planing timber', 'crow cawing', 'shot football', 'writing on blackboard with chalk', 'people slapping', 'using sewing machines', 'raining', 'dog howling', 'playing cello', 'playing trumpet', 'fox barking', 'bowling impact', 'people crowd', 'pumping water', 'ice cracking', 'baby crying', 'playing bass drum', 'playing bongo', 'tornado roaring', 'playing steel guitar, slide guitar', 'playing squash', 'typing on typewriter'].

As you can see, our dataset is diverse, containing not only musical instruments but also human sounds, sports, traffic, animals, and various other categories. To the best of our knowledge, recent state-of-the-art methods, including the two suggested by the reviewer, have rarely been tested on such a diverse range of categories in the context of audio-visual sound separation.

The total number of videos in this subset is 61,195. For each category, we randomly selected 20 videos for validation, 20 videos for testing, and the remainder for training. This results in 57,195 videos for training, 2000 mixtures for validation, and 2000 mixtures for testing. We divided the 100 categories into 4 incremental tasks, each containing 25 categories. Both our method and the baseline methods were trained for 100 epochs per task.

We hope these details further demonstrate the breadth and rigor of our experiments on VGGSound, strengthening our contributions.
Thank you for the author's further response and the keen interest in delving deeper into the reasons. As the author mentioned, the paper introduces a new task, 'Continual Audio-Visual Sound Separation,' by combining continual learning with audio-visual sound separation. However, this approach originates from the combination of continual learning and audio-visual classification tasks, making it easily extendable. For instance, one could combine continual learning with audio-visual segmentation or audio-visual event localization, which makes proposing this task relatively straightforward. As for CrossSDC, most of its content is derived from Audio-Visual Class-Incremental Learning, which is not a significant innovation. 

**My core question is: how significant is the value of combining two existing tasks to create a new one? Can extending the innovative aspects from previous similar methods to this new task be considered a major innovation?**
Thank you for your prompt response! We’d like to clarify that we began experiments with VGGSound at the end of July after receiving the reviewer’s feedback. However, due to the time required for data preparation and model training, we couldn’t provide results earlier. We initially believed the results on MUSIC-21 and AVE sufficiently validated our findings.

**We also want to highlight that the two works mentioned by the reviewer utilize only small subsets of VGGSound and AudioSet, as described in their papers or appendices.** For example, [3] uses only 15 musical instrument categories from AudioSet, and [4] focuses on 49 music categories from VGGSound-Music (even though the full set was adopted in this work for sound source localization task and their separation task only used this small subset).  **In contrast, our VGGSound subset consists of 100 significantly more broad and diverse categories, extending beyond just musical instruments.** We believe these results clearly strengthen our contributions.

We will release all of our source code and pre-trained models. While we respectfully disagree with your assessment of the technical contribution, we leave the final judgment to you, the other reviewers, and the AC. We appreciate your perspective and value your comments and engagement in the discussions. Your suggestion on a broader range of data classes has indeed made our paper stronger.

[3] Language-Guided Audio-Visual Source Separation via Trimodal Consistency. In *CVPR* 2023.

[4] A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition. In *ICML* 2023.
Many thanks to the reviewers for their responses, I was surprised by how quickly the authors downloaded and processed the 100-class VGGSound dataset from YouTube. The 100 classes may contain roughly 90K videos. I was also surprised by the training speed of the authors. However, I am still not confident enough to recommend this paper for acceptance, **mainly because of its low technical contribution.** I will maintain my original score.
Dear Reviewer wmVP,

To demonstrate that our approach can generalize to larger dataset with broader range of sound categories, we have conducted experiments on the VGGSound dataset, in which we use randomly selected 100 categories for our continual audio-visual sound separation. We train both our method and the baseline methods using iQuery as the separator architecture. The results for Fine-tuning/LwF/PLOP/AV-CIL/ContAV-Sep (Ours) are 3.69/4.71/4.56/4.66/**4.90** for SDR and 7.23/8.89/8.32/8.61/**9.25** for SIR, respectively, which further demonstrate the effectiveness of our approach on a broader range of data categories.

Best Regards,

The Authors
We thank your further response to our rebuttal and your willing to revise your evaluation based on the results on the dataset with larger number of categories.

> ### **Regarding the contribution**

We would like to further emphasize our contributions. Beyond our proposed approach, the Continual Audio-Visual Sound Separation task itself is a significant contribution of our paper.

In the approach, our technical contribution is not to introduce a separator framework, such as iQuery, for sound separation. Instead, similar to the research goals in the broader continual learning literature, where the main concentration is to explore **"how to mitigate the catastrophic forgetting probelm when training the model continually across tasks?"**, the primary research direction in this paper is to address the catastrophic forgetting problem when continually training the audio-visual sound separation model. Further, our technical contribution can be seamlessly adapted to different model architectures for Continual Audio-Visual Sound Separation. The experimental results in Table 1 demonstrate that our approach consistently outperforms baseline methods, even when applied to other architectures like Co-Separation [1], highlighting the generalizability of our approach across various separation models.

While the knowledge distillation methods in [2, 3] have some similar characteristics with our CrossSDC, there are key differences that set our approach apart. Unlike the method in [2], which employs a separate distillation loss to align the prediction distribution/logits across different modalities and classes, and unlike the approach in [3], which uses contrastive loss to ensure class similarities between teacher and student models without preserving any cross-modal semantic similarities, our CrossSDC construct an unified cross-task cross-modal contrastive loss by integrating previously learned audio and visual features into the computing process of cross-modal contrastive learning, allowing the cross-modal similarity calculation process across different tasks, enabling not only the distilation of previous learned cross-modal knowledges/similarities but also the enhancement of the learning of current task's audio-visual similarities. We have cited and discussed these papers in our revised version.


> ### **Experiments on a broader range of sound categories**

To demonstrate that the approach can handle a broad range of categories, we have conducted experiments on the VGGSound dataset, in which we use randomly selected 100 categories for our continual audio-visual sound separation. We train both our method and the baseline methods using iQuery as the separator architecture. The results for Fine-tuning/LwF/PLOP/AV-CIL/ContAV-Sep (Ours) are 3.69/4.71/4.56/4.66/**4.90** for SDR and 7.23/8.89/8.32/8.61/**9.25** for SIR, respectively, which further demonstrate the effectiveness of our approach on a broader range of data categories.


[1] Co-Separating Sounds of Visual Objects. In *ICCV* 2019.

[2] Distilling Audio-Visual Knowledge by Compositional Contrastive Learning. In *CVPR* 2021.

[3] Learning from Students: Online Contrastive Distillation Network for General Continual Learning. In *IJCAI* 2021.
Thank you to the authors for their response. However, I still have concerns regarding the technical contribution and the adequacy of the experiments conducted.

**Limmited Technical Contribution**: I remain unconvinced about the significance of the technical contribution in this paper. As the authors mentioned, they employ iQuery as their primary framework to address the issue of catastrophic forgetting in audio separation. If we consider the overview in the manuscript as a simplified introduction to iQuery's workflow, it suggests that the framework is not novel. The authors argue that the Cross-modal Similarity Distillation Constraint (CrossSDC) is their main contribution, highlighting it as the first method to integrate previously learned features into the contrastive learning process for the current task. However, I find this claim unconvincing because similar techniques have been applied to other tasks in previous work [1][2]. This makes the contribution seem more like an extension for achieving the audio speparation rather than a novel innovation.

**Insufficient Experimental demonstration**: There are also concerns regarding the sufficiency of the experimental validation. While [3][4] conduct audio separation experiments on VGGsound, Music21 only includes 21 categories, and the AVE dataset used by the authors comprises around 20 categories. This limited scope raises doubts about whether the experiments are comprehensive enough to convincingly demonstrate the method's effectiveness, as the author emphasized they focus on the catastrophic forgetting problem. I would be more inclined to improve my evaluation if the method's effectiveness were demonstrated across a broader range of data categories."

[1] Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
[2 ]Learning from Students: Online Contrastive Distillation Network
[3] Language-Guided Audio-Visual Source Separation via Trimodal Consistency
[4] A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition
Dear Reviewer wmVP,

We sincerely appreciate your valuable feedback, which has greatly improved our paper! We kindly ask if you could confirm whether our response has adequately addressed your concerns. If so, we would be grateful if you might consider raising your rating. Please do not hesitate to let us know if there are any remaining issues.

Thank you once again for your insightful feedback!

Best Regards,

The Authors
Dear Reviewer 5v9b,

We sincerely appreciate your valuable feedback, which has greatly improved our paper! We kindly ask if you could confirm whether our response has adequately addressed your concerns. If so, we would be grateful if you might consider raising your rating. Please do not hesitate to let us know if there are any remaining issues.

Thank you once again for your insightful feedback!

Best Regards,

The Authors
Dear Reviewer NCtq

We sincerely appreciate your valuable feedback, which has greatly improved our paper! We kindly ask if you could confirm whether our response has adequately addressed your concerns. If so, we would be grateful if you might consider raising your rating. Please do not hesitate to let us know if there are any remaining issues.

Thank you once again for your insightful feedback!

Best Regards,

The Authors
Thank you, Reviewer tY1x, for your response! We respect your opinion but respectfully disagree that the technical novelty is limited. In this work, we introduce not only the novel task of Continual Audio-Visual Sound Separation but also CrossSDC to address it. We’d like to emphasize that, in addition to the model, the new task itself is an important contribution to the field.

Furthermore, we believe our CrossSDC is indeed a novel approach within continual audio-visual sound separation. By integrating features from previous tasks into the cross-modal contrastive loss, we are pioneering the maintenance of audio-visual similarity across tasks. This allows us to learn audio-visual similarity within the current task and distill similarity from previous tasks, all without requiring a specific model architecture designed to combat catastrophic forgetting. Our approach is simple yet effective, and experiments with two different separators on two datasets demonstrate its superiority.

We respectfully leave the final judgment to you, the other reviewers, and the AC. We sincerely appreciate your comments and engagement with our rebuttal!
Thank for authors' response. However, the technique novelty is limited. The proposed CrossSDC just integrates features learned by previous model into the contrastive loss upon the D-AVSC,  so I hold on my score without any modification.
Thank you so much for your positive support!
I have read the author's responses to my and the other reviewer's questions. Due to its focus on continual learning and added investigations of other datasets and answers to the questions I am confident in my rating.
Dear Reviewers,


Thank you for all of the valuable comments and suggestions provided in the reviews. We have addressed your questions in our rebuttal and would appreciate it if you could take a look and share any further questions or suggestions you may have. We’re happy to address them!


Best regards,

Paper 3807 Authors
| Method              | step 1 |        |        | step 2 |        |        | step 3 |        |        | step 4 |        |        |
|---------------------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|
|                     | SDR    | SIR    | SAR    | SDR    | SIR    | SAR    | SDR    | SIR    | SAR    | SDR    | SIR    | SAR    |
| LwF                 | 3.65   | 6.16   | 12.54  | 4.97   | 7.53   | 10.50  | 4.09   | 9.68   | 6.90   | 4.50   | 10.19  | 7.42   |
| PLOP                | 3.65   | 6.16   | 12.54  | 4.22   | 6.85   | 9.92   | 4.93   | 10.16  | 7.70   | 5.79   | 11.31  | 8.01   |
| EWF                 | 3.65   | 6.16   | 12.54  | 2.62   | 4.42   | 9.71   | 4.02   | 9.16   | 7.84   | 5.01   | 11.03  | 7.37   |
| ContAV-Sep (Ours)   | 4.46   | 6.80   | 11.86  | 5.14   | 6.60   | 13.09  | 6.62   | 10.49  | 9.89   | 6.02   | 10.15  | 10.05  |
| Upper bound         | 3.65   | 6.16   | 12.54  | 6.84   | 9.44   | 11.73  | 7.68   | 11.37  | 11.83  | 10.43  | 13.99  | 13.89  |