MMAL: Multi-Modal Analytic Learning for Exemplar-Free Audio-Visual Class Incremental Tasks

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 18:33:38

Abstract: Class-incremental learning poses a significant challenge under an exemplar-free constraint, leading to catastrophic forgetting and sub-par incremental accuracy. Previous attempts have focused primarily on single-modality tasks, such as image classification or audio event classification. However, in the context of Audio-Visual Class-Incremental Learning (AVCIL), the effective integration and utilization of heterogeneous modalities, with their complementary and enhancing characteristics, remains largely unexplored. To bridge this gap, we propose the Multi-Modal Analytic Learning (MMAL) framework, an exemplar-free solution for AVCIL that employs a closed-form, linear approach. To be specific, MMAL introduces a modality fusion module that re-formulates the AVCIL problem through a Recursive Least-Square (RLS) perspective. Complementing this, a Modality-Specific Knowledge Compensation (MSKC) module is designed to further alleviate the under-fitting limitation intrinsic to analytic learning by harnessing individual knowledge from audio and visual modality in tandem. Comprehensive experimental comparisons with existing methods show that our proposed MMAL demonstrates superior performance with the accuracy of 76.71%, 78.98% and 76.19% on AVE, Kinetics-Sounds and VGGSounds100 datasets, respectively, setting new state-of-the-art AVCIL performance. Notably, compared to those memory-based methods, our MMAL, being an exemplar-free approach, provides good data privacy and can better leverage multi-modal information for improved incremental accuracy.

Main paper: 
 introduction: 
 The inherent correlation between audio and visual signals empowers humans to associate sounds with their sources, such as identifying a barking dog. By leveraging the enhancing and complementary nature of audio and vision, these endeavors aim to enhance the understanding and capabilities of models for tasks such as speech recognition [1, 8, 37, 52, 61] , sound localization [12, 14, 22, 23, 31, 39, 47, 48, 50] and audio-visual event classification [7, 16, 55, 60, 65] . The aforementioned studies demonstrated the effectiveness of jointly modeling audio-visual modalities in capturing meaningful cross-modal semantic correlations. Inspired by their success, in this paper, we tackle the problem of identifying the sounding objects in synchronized audio-video streams. To be mentioned, it is crucial for real-world applications where the set of classes (e.g., types of sounds or visual events) may evolve or where the model needs to integrate new information into an ever-growing knowledge base. By focusing on preserving semantic similarity and learned correlations between modalities, we aim to mitigate catastrophic forgetting (i.e., a model loses previously acquired knowledge or experiences a significant performance decline when learning new information) and enhance the resilience and adaptability of audio-visual systems.Class-incremental learning (CIL) [49, 62] progressively updates network parameters by incorporating training data from different unseen classes over time. Existing methodologies have concentrated mainly on employing exemplar-free [25, 28, 32] and memorybased [15, [34] [35] [36] 62] solutions to mitigate catastrophic forgetting. Specifically, the former one introduces additional regularization terms to the loss function. While the later one leverages external memory modules or knowledge distillation to selectively replay past data samples, thereby achieving superior results.As a viable alternative for exemplar-free CIL, Analytic Learning (AL)-based [68] [69] [70] [71] methods address the issue of catastrophic forgetting by identifying the iterative mechanism as its main cause and substituting it with linear recursive tools. These methods achieve results comparable to those of memory-based techniques and exhibit robust performance. Nonetheless, the current AL-based methods may encounter the challenge of under-fitting due to their reliance on a single linear projection and the frozen backbone.While prior Class-incremental learning (CIL) works are mostly limited to a single modality and cannot acquire concise multi-modal representations with the awareness of different classes, there is a growing need to extend these methods to broader domains and modalities. To this end, a few attempts have been made in Audiovisual Class-incremental learning (AVCIL), especially in the field of audio-visual event classification [40, 46] . Specifically, they utilize class token distillation and continual grouping to prevent knowledge forgetting of previous tasks, thereby improving the model's adaptability to capture new and discriminative audio-visual event categories. However, these works explore the data from previous tasks, making them not feasible in privacy-sensitive scenarios with confidential data (e.g., in private property and restricted area), as illustrated in Figure 1 .To solve the aforementioned issues, we propose a novel yet practical framework, namely Multi-Modal Analytic Learning (MMAL), for audio-visual CIL. Specifically, MMAL contains one modality fusion module re-formulating the AVCIL problem in a Recursive Least-Square (RLS) manner, and one Modality-Specific Knowledge Compensation (MSKC) module remedying the under-fitting limitation of the fusion module due to the frozen audio-visual backbone and linearity (i.e., the inherent properties of analytic learning). This study advances AL-based CIL methods by introducing multi-modal incremental learning for the first time and proposes a compensation module that simultaneously leverages individual modality knowledge (i.e., A-MSKC and V-MSKC), thereby addressing their inherent limitations without sacrificing the fundamental advantages. Our key contributions can be summarized as follows.(1) This paper is a pioneering work to provide the technical route of analytic learning to address the challenging multimodal exemplar-free AVCIL problem. (2) The fusion module of MMAL re-formulates the AVCIL problem into a RLS problem by freezing the audio-visual backbone. This allows the module to emulate its joint training counterpart that adopts data from both current and historical incremental steps. (3) To address the intrinsic under-fitting issue associated with analytic learning, we propose the Modality-Specific Knowledge Compensation (MSKC) module. This allows for the simultaneous integration of distinct audio and visual modality information for compensation, enhancing the model's overall learning capability. (4) Through extensive experiments on three widely-used benchmark audio-visual datasets, AVE, Kinetics-Sounds, and VG-GSound100, we demonstrate the superiority of our MMAL over state-of-the-art competitors in AVCIL scenarios. 
related work 2.1 audio-visual learning: 
 Previous research has extensively explored the field of audio-visual learning, with numerous methods [3, 5, 30, 38, 42, 50] aiming to explore their temporal correlations, enhancing and complementary characteristics in synchronized streams. A comprehensive review has been presented [57] , which indicates the key point in audiovisual learning is establishing cross-modal alignment by repelling non-matching audio-visual embeddings while attracting the matching ones. This draws increasing insight into interdisciplinary fields such as speech separation [17-19, 43, 63] , active speaker detection [10, 29, 33, 53] and sound localization [41, 47, 48, 51, 58] . In this paper, our primary objective is to develop compact audio-visual representations with non-stationary audio-visual pairs, specifically for CIL tasks. This task poses greater challenges compared to the aforementioned tasks because of the complexities arising from sequential tasks and feature adaptation to changing audio-visual inputs over time. There are a few attempts [40, 46] on AVCIL addressing above challenges, however, they still require to store historical samples.The primary objective of CIL is to develop strategies that mitigate the negative impact of forgetting previously learned classes while accommodating the introduction of new classes. Existing CIL methods can be categorized into two main types, i.e., memory-based, and exemplar-free methods.Memory-based Work. Memory-based methods [6, 9, 49, 64] leverage additional data, such as exemplars/memory, to tackle CIL. In particular, they leverage the stored data from previous tasks to reinforce learning and prevent catastrophic forgetting. The mechanism was first introduced by iCaRL [49] , followed by various attempts due to its superior performance. In [59] , an additional trainable layer was designed to correct the bias towards new classes. The LUCIR [21] creates an innovative adaptation which replaces the softmax layer with a cosine layer. In PODNet [15] , a spatial-based distillation loss is introduced to retain previously acquired knowledge while accommodating new information. While the above memorybased methods have obtained satisfactory results, they still have the requirement of storing previous samples. Exemplar-free Work. Exemplar-free methods avoid revisiting historical samples during training, which can be mainly categorized into regularization-, and prototype-, and the recent analytic learning-based types, which are discussed as follows. 1) Regularization-based methods introduce additional constraints to the learning process to minimize the impact of new tasks on previously learned knowledge. For example, EWC [28] protects the critical parameters associated with previous tasks, ensuring their stability and minimizing their susceptibility to change during training for new tasks. Based on EWC, RWalk [11] defines a Riemannian metric to calculate a penalty term for the loss function, thereby restricting the magnitude of changes in the parameters. Other examples include LfL [25] which penalizes differences in network activations, and LwF [32] prevents activation changes between old and new networks.2) Prototype-based methods use past class prototypes to prevent forgetting. For example, prototype augmentation is proposed in PASS [66] to improve the discrimination of the classes learned in different incremental steps. A prototype selection mechanism is introduced in SSRE [67] . FeTrIL [45] combines a fixed feature extractor and a pseudo-feature generator to improve the stabilityplasticity balance.3) Analytic learning-based methods draw inspiration from [20, 69] which leverage least squares to obtain closed-form solutions for network training. In particular, ACIL [71] first transforms CIL into a recursive analytic learning process which releases the need of storing exemplars by preserving a correlation matrix. DS-AL [68] employs a dual-stream approach to further enhance the fitting ability of ACIL. The following work GKEAL [70] specializes in the few-shot CIL setting by introducing a Gaussian kernel process.As an emerging and promising CIL branch, AL-based methods demonstrate impressive performance, even on few-shot scenarios. However, existing AL-based CIL works primarily focus on exploring single-modality solutions, neglecting the potential benefits of multi-modal learning, particularly in audio-visual CIL scenarios. Thus, the integration of audio and visual modalities remains largely unexplored. Furthermore, the AL-based methods require freezing the backbone during the incremental steps, which will cause serious under-fitting limitation due to the complexity of audio-visual data. As a result, we are motivated to explore the multi-modal analytical CIL solution in audio-visual settings.  
analytic audio-visual incremental learning: 
 In this subsection, we will give details of the fusion module of our proposed method. The fusion module aims to transform the traditional AVCIL task into a RLS task and give its analytic solution in a one-epoch training style. It starts with an audio-visual fusion backbone via conventional backpropagation (BP) iterative training, followed by the audio-visual analytic re-alignment, and finally the audio-visual CIL procedures, as illustrated in Figure 2 . Audio-Visual Fusion Backbone Training via BP. Given an input sequence of visual frames x 𝑣 and the corresponding audio signal x 𝑎 , we first use the pre-trained audio and visual encoder (e.g., Au-dioMAE [24] and VideoMAE [56] ) to extract high-level visual and audio embeddings, respectively. Afterward, the audio-guided visual attention mechanism is employed to aggregate the visual embedding by considering both spatial and temporal dimensions. This has been demonstrated to be highly effective in capturing correlations between audio and visual features [54] . Lastly, we add the audio-guided visual embedding and the audio embedding as the joint audio-visual embeddings and then feed into the classifier. The model is trained on the base classes using backpropagation (BP) for multiple epochs. Subsequently, when provided with an input sample, the network produces the following output:EQUATIONwhere F Θ (𝑋 𝑎 , 𝑋 𝑣 ) denotes the output of the audio-visual backbone, 𝑊 FFN are the parameters of the linear feed-forward classifier, and 𝑓 softmax denotes the softmax function.Once the audio-visual fusion backbone is trained on the base classes, we freeze this backbone and replace the final linear classifier with a 2-layer analytic network for re-alignment and the forthcoming incremental steps, as detailed in the following section. Audio-Visual Analytic Re-alignment. With the network trained on the base classes, MMAL seeks to detach the backbone and attach it with a 2-layer linear feed-forward network, in which the first layer conducts embedding up-sampling before feeding to the subsequent layer for classification. This procedure enables the network's learning to match the learning dynamics of analytic learning.Specifically, we feed the input samples (𝑋 train 0,𝑎 , 𝑋 train 0,𝑣 ) of the base classes into the backbone to extract the joint audio-visual embeddings, and then up-sample the embeddings followed by an activation function:EQUATIONwhere 𝑊 𝑎𝑣 𝑢𝑝 denotes the parameters of the up-sampling layer, 𝑋 𝑎𝑣 ∈ R 𝑁 0 ×𝑑 𝑎𝑣 with 𝑑 𝑎𝑣 being the up-sampling size, and 𝑓 𝜎 is the activation function. Here, we simply follow other analytic methods [69, 71] to adopt the random projection for 𝑊 𝑎𝑣 𝑢𝑝 and ReLU for 𝑓 𝜎 .We then map the up-sampled embeddings into the label matrix 𝑌 train 0 via the linear classifier layer, whose weights can be computed by solving: argminEQUATIONwhere ∥ • ∥ 𝐹 is the Frobenius form, and 𝜂 is the regularization term.Then, the optimal solution can be obtained as:EQUATIONwhere Ŵ 𝑎𝑣 0 is the estimated parameters of the linear classifier layer, and • 𝑇 indicates the transpose operation. Analytic Audio-Visual Class-Incremental Learning. Following the analytic re-alignment of the joint audio-visual embeddings, we then move to the AVCIL steps in an analytic learning manner.Specifically, the learning problem using all seen data at step 𝑘 -1 can be extended from (3) to: argminEQUATIONwhere (4) , the solution to (5) can be obtained as:𝑌 train 0:𝑘 -1 =          𝑌 train 0 0 0 • • • 0 0 𝑌 train 1 0 • • • 0 . . . 0 0 • • • 0 𝑌 train 𝑘 -1          , 𝑋 𝑎𝑣 0:𝑘 -1 =          𝑋 𝑎𝑣 0 𝑋 𝑎𝑣 1 . . . 𝑋 𝑎𝑣 𝑘 -1          Similar toEQUATIONwhereŴ 𝑎𝑣 𝑘 -1 ∈ R 𝑑 𝑎𝑣 × 𝑘 -1 𝑖=1 𝑑 𝑦 𝑖 .To goal of AVCIL under exemplar-free constraint is to sequentially learn new tasks on D train 𝑘 given a network trained on D train 0:𝑘 -1 . Nevertheless, the equation above shows that previous data is still required. To reduce this dependency, let 𝑅 𝑎𝑣 𝑘 -1 = ((𝑋 𝑎𝑣 𝑘 -1 ) 𝑇 𝑋 𝑎𝑣 𝑘 -1 + 𝜂𝐼 ) -1 , we redefine the CIL process as a RLS task as outlined in the subsequent theorem. Theorem 1. Given training data 𝐷 train 𝑘 and estimated weights of the final classifier layer Ŵ 𝑎𝑣 𝑘 -1 of task T 𝑘 -1 , Ŵ 𝑎𝑣 𝑘 can be recursively obtained by:EQUATIONwhere𝑅 𝑎𝑣 𝑘 = 𝑅 𝑎𝑣 𝑘 -1 -𝑅 𝑎𝑣 𝑘 -1 (𝑋 𝑎𝑣 𝑘 ) 𝑇 (𝑋 𝑎𝑣 𝑘 𝑅 𝑎𝑣 𝑘 -1 (𝑋 𝑎𝑣 𝑘 ) 𝑇 + 𝐼 ) -1 𝑋 𝑎𝑣 𝑘 𝑅 𝑎𝑣 𝑘 -1 , (8) proof.See the Supplementary Material.Theorem 1 suggests that the weights of joint training can be obtained by recursively training on the data from D train 1 to D train 𝑘 sequentially, which implies that, by freezing the audio-visual backbone, the AVCIL is equalized to its joint training counterpart as shown in the theorem. This means the model trained incrementally yields the same weights as that trained on both current and all previous data.The audio-visual fusion module is built on the joint audio-visual embeddings. Moreover, since the AVCIL problem is reformulated to a RLS task, the audio-visual backbone has to be freezed, except the final linear classifier. However, when the training samples are complex, the fusion module might not be sufficient to capture the complementary representations of the multi-modal data, thus the under-fitting might occur. To alleviate this limitation, we introduce the MSKC module by leveraging the individual information from audio and visual modality to enhance the fusion module.This MSKC module contains an audio MSKC (A-MSKC) and a visual MSKC (V-MSKC) sub-module, which operate in a similar manner to the main fusion module, except that the label matrix for updating Ŵ 𝑎𝑣 𝑘 is generated using the residue from the fusion module. For brevity, in the following part, we take the A-MSKC for explanation while the V-MSKC follows the same rules. Audio-Visual Analytic Re-labelling. Without loss of generality, assume that we have conducted the fusion stream at step 𝑘, (i.e., obtaining Ŵ 𝑎𝑣 𝑘 and 𝑅 𝑎𝑣 𝑘 ) and the A-MSKC sub-module at step 𝑘 -1 (i.e., obtaining the weights matrix Ŵ 𝑎 𝑘 -1 and its corresponding 𝑅 𝑎 𝑘 -1 ). Let 𝑌 𝑘 be the residue after conducting the fusion module, i.e.,EQUATIONwhere the zero matrix is due to the mutually exclusive AVCIL setting. Let 𝑋 𝑎 𝑘 be the output the corresponding up-sampled audio embedding, i.e.,EQUATIONwhere 𝑊 𝑎 𝑢𝑝 represents the weights of the up-sampling layer of the A-MSKC sub-module.The 𝑌 𝑘 can be viewed as the residual error, where the joint audiovisual embedding cannot reach. The key idea of the MSKC module is to leverage the individual information from both audio and visual modality to remedy this error, attempting to further reduce the i) Update 𝑅 𝑎𝑣 𝑘 , 𝑅 𝑎 𝑘 , and 𝑅 𝑣 𝑘 , using ( 8), ( 13) and ( 15) respectively.𝑅 𝑎𝑣 0 = ((𝑋 𝑎𝑣 0 ) 𝑇 𝑋 𝑎𝑣 0 + 𝜂𝐼 ) -1 , 𝑅 𝑎 0 = ((𝑋 𝑎 0 ) 𝑇 𝑋 𝑎 0 + 𝜂𝐼 ) -1 , 𝑅 𝑣 0 = ((𝑋 𝑣 0 ) 𝑇 𝑋 𝑣 0 + 𝜂𝐼 ) -6:ii) Update Ŵ 𝑎𝑣 𝑘 , Ŵ 𝑎 𝑘 , and Ŵ 𝑣 𝑘 , using ( 7), (12) , and ( 14) respectively. 7: end for forgetting and improve the incremental accuracy. To achieve this, we construct an audio recursive CIL and a visual recursive CIL procedures for audio and video, respectively. Akin to the audiovisual fusion module indicated in Theorem 1, the A-MSKC module follows a similar recursive procedure.Before conducting the following step, note that 𝑌 𝑘 is the label residual error containing both the current and all previous data, however, under the exemplar-free constraint, we only have access to the data D train 𝑘 at step 𝑘. Therefore, to prevent the false supervision for the data at step 𝑘, letEQUATIONonly contain the last 𝑑 𝑦 𝑘 columns for present data D train 𝑘 .Audio MSKC. Upon obtaining the input 𝑋 𝑎 𝑘 and new labels 𝑌 𝑘 , we can proceed to recursively update the weight 𝑊 𝑎 𝑘 of A-MSKC following the same procedure indicated in Theorem 1. Then we haveEQUATIONwhereEQUATION) Visual MSKC. Similarly, to leverage visual modality to compensate the fusion module, we haveEQUATIONwhereEQUATIONFinally, during the inference stage, the predictions can be obtained as follows:EQUATIONwhere 𝜆 𝑎 and 𝜆 𝑣 denote the compensation ratio of audio and visual modality respectively, indicating the degree to which the network relies on the audio and video to compensate and make more accurate prediction. The MMAL is summarized in Algorithm 1. 
experiments 4.1 datasets: 
 We conduct experiments with our proposed method compared to state-of-the-art baselines on three public datasets i.e., AVE [54] , Kinetics-Sounds [4] and VGGSound [13] . Specifically, the AVE dataset consists of 4K 10-seconds videos from 28 audio-visual event classes. The Kinetics-Sounds dataset contains around 24K 10-seconds videos from 31 human action classes, while the VGGSound dataset contains around 200K 10-seconds YouTube videos from 309 classes. We follow the experimental protocol adopted in [46] , in which 30 classes are randomly selected from the Kinetics-Sounds containing 23K samples in total, and 100 classes (i.e., VGGSound100) are randomly selected from the original VGGSound dataset containing 60K samples in total. For each class of the VGGSound100, 50 samples are randomly selected for validation and testing, respectively. In terms of class-incremental setting, since the AVE and Kinetics-Sounds datasets contain few classes, we evenly divide the AVE into 4 incremental steps, each of which contains 7 classes, and divide the Kinetics-Sounds into 5 incremental steps, each of which contains 6 classes. While for the VGGSound100 dataset, we explore two different incremental scenarios: 0Base-10Task, and 50Base-𝑛Task. The former evenly splits 100 classes into 10 tasks, which mainly follows the setting in [46] , while the latter first selects 50 classes as the base classes and then distributes the other across 𝑛 tasks. It shoud be noted that for the AVE, Kinetics-Sounds and the 0Base-10Task setting on VGGSound100, the base classes are considered to be those classes of the first task. Most existing methods only report small-phase results, e.g., those of 𝐾 = 4, 5, 10, we include 𝐾 = 25, 50 as well to validate MMAL's large-step performance.The key to successful incremental learning lies in maintaining a delicate balance between plasticity and stability, enabling the acquisition of new knowledge without forgetting previously learned information. Therefore, two metrics, namely average incremental accuracy (𝐴𝑐𝑐) and performance drop rate (𝑃𝐷), are used to evaluate CIL methods. 𝐴𝑐𝑐 = 1 𝐾+1 𝐾 𝑘=0 𝐴𝑐𝑐 𝑘 , where 𝐴𝑐𝑐 𝑘 indicates the average test accuracy of the model incrementally trained at step 𝑘 by testing it on all seen classes (i.e., D test 0:𝑘 ), evaluates the overall performance of CIL algorithms. A higher 𝐴𝑐𝑐 score is preferred. 𝑃𝐷 = 𝐴𝑐𝑐 0 0 -𝐴𝑐𝑐 0 𝐾 , where 𝐴𝑐𝑐 0 𝐾 denotes the average accuracy at last step 𝐾 by testing it on the base classes D test 0 , reveals the degree to which a CIL method forgets the base classes in the first step, which can reflect the degree of the model's retention ability of the old knowledge.We conduct all our experiments with PyTorch [44] . For the audio encoder and visual encoder, we use the recent self-supervised pre-trained AudioMAE [24] and VideoMAE [56] , respectively. Following the protocol of AudioMAE, the raw audio waveform is transformed into 128-dimensional spectrogram with a 25ms Hanning window and a 10ms shift before feeding into the audio encoder. Similar to VideoMAE, 16 frames are randomly selected from the video clip and then feed into the visual encoder. During the conventional BP training on the base classes, we freeze the pre-trained audio and visual encoder, and only train the remaining parts containing audio-guided visual attention layer, the fusion layer and the final classifier, with a maximum training epochs of 200. We use Adam [27] to optimize the model with learning rate and weight decay of 1e-3 and 1e-4, respectively, with a batch size of 256. All experiments are conducted using one Nvidia A100 GPU with the results averaged over 3 runs. Note that in our MMAL, the audiovisual backbone is only trained during the BP-based training. After which, the parameters of the backbone are fixed and used as a feature extractor during the following incremental steps.For the regularization parameter 𝜂, we fix it at 𝜂 = 1 for all three datasets. For the embedding up-sampling dimension of the fusion module, audio MSKC and visual MSKC (i.e., 𝑑 𝑎𝑣 , 𝑑 𝑎 and 𝑑 𝑣 ), we set to (8K, 15K, 15K), (8K, 15K, 15K), and (20K, 20K, 20K) for AVE, Kinetics-Sounds and VGGSound100, respectively. For the compensation ratio of audio and visual modality (𝜆 𝑎 , 𝜆 𝑣 ) in ( 16), we set to (0.8, 0.9), (0.5, 1.0), (0.7, 1.0) for AVE, Kinetics-Sounds and VGGSound100, respectively.To demonstrate the effectiveness of the proposed MMAL, we comprehensively compare it to previous representative and state-ofthe-art baselines: 1) Fine-tuning: the simplest incremental learning method which initializes the model with the parameters trained from the last step and re-train it on the current step without any constraints to prevent the catastrophic forgetting issue. 2) LwF [32] , an exemplar-free method that preserves outputs of previous examples to reduce the forgetting of the old task and act as a regularizer for the new task. 3) ACIL [71] : an exemplar-free AL-based method that identifies the iterative mechanism as the primary cause of catastrophic forgetting and replace it with linear recursive tools. 4) iCaRL [49] : an memory-based method that uses the exemplars in combination with distillation to avoid forgetting. We report the experimental results with both nearest-mean-of-exemplars (NME) classification strategy and the classifier, denoted as iCaRL-NME and iCaRL-FC, respectively. 5) SS-IL [2] : an exemplar-based method that consists of separated softmax output layer combined with task-wise knowledge distillation network. 6) AFC [26] : a knowledge distillation method that minimizes the upper bound of the expected loss increased over the previous tasks. We also report the experimental results with both NME classification strategy and the classifier, denoted as AFC-NME and AFC-LSC, respectively. 7) AV-CIL [46] : an audio-visual incremental learning method that incorporates the dual-audio-visual similarity constraint and visual attention distillation. We compare the proposed MMAL with the above methods in Table 1. We can see that our proposed MMAL outperforms recent stateof-the-art methods significantly, including both memory-based and exemplar-free methods. Specifically, on the AVE dataset, our MMAL outperforms the state-of-the-art Acc and PD results by 2.67 and 3.30, respectively. For the Kinetics-Sounds dataset, our method outperforms the state-of-the-art method, i.e., AV-CIL, by 5.92 and 3.32 for Acc and PD, respectively. For the 0Base-10Task setting on the VGGSound100 dataset, our method has the improvement of 2.32 and 1.81 for Acc and PD over the ACIL. While for the 50Base-10Task scenario, our method yields 2.32 and 2.64 improvement for Acc and PD compared the strong baseline ACIL. These experimental results demonstrate the effectiveness of our proposed method in AVCIL.Furthermore, we show the testing accuracy at each incremental step of our MMAL and other baselines in Figure 3 . It can be observed that our method achieves the best performance at each incremental step on three datasets, showing less forgetting and better accuracy. Moreover, comparing two different settings on the VG-GSound100 dataset, the accuracy at last step of the 50Base-10Task achieves 70.28%, even though the accuracy on the base classes is only 81.60% which is much lower than that of the 0Base-10Task (i.e., 90.40%), indicating less forgetting. This might be due to that, in the 50Base-10Task setting, we have more data to train the backbone during the base training, thus encouraging to extract more meaningful representations for the following incremental steps. Note that the backbone is freeze once the base training is finished in our MMAL. In summary, our method has a significant superiority for exemplar-free AVCIL compared to others, which demonstrates the effectiveness of our proposal.In Section 3, we have demonstrated that the MMAL exhibits a stepinvariant property. To empirically validate this claim, we conduct The MSKC module utilizes the individual information from audio and visual simultaneously to compensate the limitation of the fusion module. To help understand this strategy, we give specific examples (e.g., 0Base-4Task experiments on the AVE dataset) during the MMAL training and testing. We plot the top-5 predictions with before and after the MSKC module's contribution. As shown in Figure 4 , the original prediction for the sample class is inaccurate in the fusion stream. While the MSKC module provides an extra gain, thereby correcting the predicted result. In this example, we can observe a significant prediction change, suggesting a non-trivial enhancement on the MMAL's ability by leveraging the individual information from audio and visual modality. We also quantitatively analyze the MSKC's impact on the stability and plasticity of the model. To show this, after training on all steps, we evaluate the corresponding model on the base classes D test 0 and the incremental classes D test 1:𝐾 separately. As shown in Table 3 , by introducing the MSKC module, new classes learned during the incremental steps receive a significant improvement (plasticity) with few performance loss on the base classes (stability). For instance, the MKSC module improves the newly learned classes by 9.31% while only losing the accuracy 0.96% on the base classes of the AVE dataset, suggesting a more reasonable stability-plasticity balance for overall improvement.To quantitatively evaluate the effect of the MSKC module, we conduct ablation experiments on three datasets. The MSKC contains an audio sub-module and a visual sub-module, aiming to leverage information from audio and video modality, respectively. As shown in Table 4 , without the audio compensation (i.e., A-MSKC), the performance will drop 2.50%, 0.99%, and 1.99% on the AVE, Kinetics-Sounds and VGGSound100 respectively. If we further remove the visual compensation (i.e., V-MSKC), the average accuracy will yield a significant drop of 6.15%, 6.42% and 3.03% compared to the full model. This demonstrates the effectiveness of our proposed MSKC module.To explore the appropriate compensation ratio of audio and visual modality to enhance the overall performance, here we conduct ablation experiment to investigate the effect of different ratio on the Kinetics-Sound dataset. As shown in Table 5 , when the 𝜆 𝑣 is set to 1, the best performance is obtained at 𝜆 𝑎 = 0.5. When increasing the ratio of audio modality, the performance starts to decrease, indicating over-compensation could mislead the model. When the 𝜆 𝑎 is set to 0.5, the performance will drop as the ratio of visual modality becomes small, which suggests that visual modality play an important role to correct the inaccurate predictions of the fusion module.Learning informative audio-visual representations with categoryaware semantics is critical for classifying audio-visual pairs. To better evaluate the quality of learned category-aware features, we visualize the learned joint audio-visual representations of 9 categories after finishing 4 incremental tasks on Kinetics-Sounds by t-SNE, as shown in Figure 5 . It should be noted that each color denotes one class of the audio-visual pair. As can be seen in the last column, audio-visual embeddings extracted by the proposed MMAL are both intra-class compact and inter-class separable. In contrast to our representations in the audio-visual semantic space, mixtures of multiple audio-visual categories still exist among features learned by LwF, iCaRL and AV-CIL. These meaningful visualization results further showcase the superiority of our MMAL in extracting compact audio-visual incremental representations with class-aware semantics for incremental audio-visual learning.  
conclusion: 
 In this paper, we propose a Multi-Modal Analytic Learning (MMAL) to tackle the challenging exemplar-free AVCIL problem for the first time. Specifically, MMAL comprises a fusion module that redefines the AVCIL task through a RLS solution, and a Modality-Specific Knowledge Compensation (MSKC) module that alleviates the underfitting stemming from the static backbone and inherent linearity within the fusion module. Hence, the fusion and compensation module can complement each other for better category-aware semantics during the incremental steps, yielding improved incremental performance. Experimental results on three audio-visual class-incremental datasets AVE, Kinetics-Sounds and VGGSound100 show that our proposed approach outperforms state-of-the-art methods significantly. In the future, we plan to extend our MMAL for tri-modal learning and beyond.  
 
 Appendix:  


Decision: All four reviewers are positive. The proposed method is novel and effective.

Reviews: review: paper_summary: paper_summary: The authors study the problem of class-incremental learning with an exemplar-free constraint. The authors reformulate audio-visual class incremental learning by recursive least-square and design modlaity-specific knowledge compensation module. The experiments are conducted using AVE, Kinetics-Sounds and VGGSound. They use pre-trained AudioMAE and VideoMAE as audio encoder and visual encoder. The main concern is about the novelty of this paper, I cannot find a lot of novel contribution to audio-visual, compared with visual based ones.
strength_weakness: strength_weakness: 1. The problem is interesting. 
2. The experimental results are good on AVE, Kinetics-Sounds and VGGSound100.
3. There is some contribution about reformulation of AVCIL problem with RLS, which is interesting.
limitations: limitations: 1. The dataset used in this paper is not large enough and it is better for the authors to explore the incremental learning for large audio-visual model.
paper_summary: paper_summary: This paper discusses the challenge of class-incremental learning, particularly in the context of Audio-Visual Class-Incremental Learning (AVCIL). The authors propose a new framework, Multi-Modal Analytic Learning (MMAL), which is an exemplar-free solution that uses a closed-form, linear approach. The MMAL introduces a modality fusion module that redefines the AVCIL problem through a Recursive Least-Square (RLS) perspective. It also includes a Modality-Specific Knowledge Compensation (MSKC) module to alleviate the under-fitting limitation of analytic learning by utilizing individual knowledge from audio and visual modality. The MMAL has shown superior performance in experiments, achieving high accuracy on AVE, Kinetics-Sounds, and VGGSounds100 datasets. Notably, the MMAL provides good data privacy and can better use multi-modal information for improved incremental accuracy.
strength_weakness: strength_weakness: 1. MMAL provides the technical route of analytic learning to address the challenging multimodal exemplar-free AVCIL problem.
2. MMAL re-formulates the AVCIL problem into a RLS problem by freezing the audio-visual backbone.
3. MMAL proposes a MSKC module, which allows for the simultaneous integration of distinct audio and visual modality information for compensation, enhancing the model’s overall learning capability.
limitations: limitations: 1. The backbone is not frozen in the fusion backbone training phase of the MMAL method, does this have an impact on the generalizability of the whole model? Has it been considered to freeze the backbone itself and only use training with less number of parameters such as linear layers instead of phase (a)? If not, can some explanation be given?
2. Is it possible to add experiments under different backbone?
paper_summary: paper_summary: This work proposed a Multi-Modal Analytic Learning (MMAL) framework for the effective utilization of heterogeneous modalities for the AVCIL task. MMAL re-formulates the AVCIL problem via Recursive Least-Square (RLS) perspective. Then, a Modality-Specific Knowledge Compensation (MSKC) module is to alleviate the under-fitting limitation intrinsic by harnessing individual knowledge from audio and visual modality. Experimental comparisons demonstrate the effectiveness of the proposed MMAL.
strength_weakness: strength_weakness: 1. This work explores the novel problem that extends Class-Incremental Learning to Multi-modal Audio-Visual task.
2. Extensive experiments demonstrate the superiority of MMAL over state-of-the-art competitors in AVCIL scenarios.
3. Overall, the paper is well-written.
limitations: limitations: 1. Line 361: what is base class? What is the meaning of analytic learning in Audio-Visual Analytic Re-alignment? The object of re-alignment is really confusing.
2. Line 392: "the learning problem using all seen data at step k-1", it seems that Eq. (5) utilizes all data from step 0 to step k-1, which is inconstent with the classical CIL setting, i.e. old data before step k-1 is unseen. Please further clarify it.
3. One of the most important issue is catastrophic forgetting. However, the connection between such issue and the proposed method is weak. How does MMAL address forgetting issue?
4. Please check the Eq. (9), what dose the left term mean?
5. What is the optimization objective of MMAL? 
6. The CIL baselines are not up-to-date (LwF and iCaRL)? Any the most recent CIL baselines?
paper_summary: paper_summary: This paper introduce Multi-Modal Analytic Learning (MMAL) to address the challenging exemplar-free AVCIL for the first time. This method redefines the AVCIL task using a Recursive Least Squares (RLS) and incorporates a Modality-Specific Knowledge Compensation (MSKC) module. This module can mitigate the under-fitting issues caused by the frozen audio-visual backbone and the inherent linearity typical of analytic learning.
strength_weakness: strength_weakness: This paper effectively addresses the challenging exemplar-free AVCIL for the first time. Furthermore, they logically redefine the CIL process as an RLS task through clear mathematical derivation.
limitations: limitations: [1] In Figure 2, I believe additional details may be required for clarity. While the text specifies that there are two layers following the audio-visual backbone, the figure appears to depict multiple layers. Could you please clarify the discrepancy in this representation? Furthermore, in the text (Line 370-371), it is mentioned that the feature $X^{av}$ emerges after upsampling, but this does not seem to be clearly depicted in the figure. Could you please address these discrepancies?

[2] In Line 410-411, it appears that the term $X_{k-1}^{av}$ in the formula for $R_{k-1}^{av}$ should be revised to $X_{0:k-1}^{av}$. Could you please confirm if this modification is necessary?