Continual Learning in the Frequency Domain

Authors: ruiqi  liu
boyu  diao
libo  huang
zijia  an
zhulin  an
yongjun  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 07:34:30

Abstract: Continual learning (CL) is designed to learn new tasks while preserving existing knowledge. Replaying samples from earlier tasks has proven to be an effective method to mitigate the forgetting of previously acquired knowledge. However, the current research on the training efficiency of rehearsal-based methods is insufficient, which limits the practical application of CL systems in resource-limited scenarios. The human visual system (HVS) exhibits varying sensitivities to different frequency components, enabling the efficient elimination of visually redundant information. Inspired by HVS, we propose a novel framework called Continual Learning in the Frequency Domain (CLFD). To our knowledge, this is the first study to utilize frequency domain features to enhance the performance and efficiency of CL training on edge devices. For the input features of the feature extractor, CLFD employs wavelet transform to map the original input image into the frequency domain, thereby effectively reducing the size of input feature maps. Regarding the output features of the feature extractor, CLFD selectively utilizes output features for distinct classes for classification, thereby balancing the reusability and interference of output features based on the frequency domain similarity of the classes across various tasks. Optimizing only the input and output features of the feature extractor allows for seamless integration of CLFD with various rehearsal-based methods. Extensive experiments conducted in both cloud and edge environments demonstrate that CLFD consistently improves the performance of state-of-the-art (SOTA) methods in both precision and training efficiency. Specifically, CLFD can increase the accuracy of the SOTA CL method by up to 6.83% and reduce the training time by 2.6×.

Main paper: 
 introduction: 
 Continual learning (CL) enables machine learning models to adjust to new data while preserving previous knowledge in dynamic environments [25] . Traditional training methods often underperform in CL because the adjustments to the parameters prioritize new information over old information, leading to what is commonly known as catastrophic forgetting [36] . While recent CL methods primarily concentrate on addressing the issue of forgetting, it is imperative to also consider learning efficiency when implementing CL applications on edge devices with constrained resources [37, 31] , such as the NVIDIA Jetson Orin NX.To mitigate catastrophic forgetting, a wide range of methods have been employed: regularizationbased methods [49, 42, 30, 11, 3] constrain updates to essential parameters, minimizing the drift in network parameters that are crucial for addressing previous tasks; architecture-based methods [14, 24, T T T T raining T raining T raining T raining Time (hours) ime (hours) ime (hours) ime (hours) Right: On the NVIDIA Jetson Orin NX edge device, CLFD demonstrates a notable enhancement in both accuracy and efficiency compared to ER [4] on the split CIFAR-10 dataset.To this end, we propose a novel framework called Continual Learning in the Frequency Domain (CLFD), which comprises two components: Frequency Domain Feature Encoder (FFE) and Class-aware Frequency Domain Feature Selection (CFFS). To reduce the size of input images, we propose the FFE. This method utilizes Discrete Wavelet Transform (DWT) to transform the original RGB image input into the frequency domain, thereby preserving both the frequency domain and spatial domain features of the image, which facilitates data augmentation. Furthermore, acknowledging that distinct tasks exhibit varying sensitivities to different frequency components, we propose the CFFS method to balance the reusability and interference of frequency domain features. CFFS calculates the frequency domain similarity between inputs across different classes and selects distinct frequency domain features for classification. This method promotes the use of analogous frequency domain features for categorizing semantically similar inputs while concurrently striving to diminish the overlap of frequency domain features among inputs with divergent semantics. Our framework avoids introducing any cross-task learnable parameters, thereby reducing the risk of catastrophic forgetting. Simultaneously, by optimizing only the input and output features of the feature extractor, it facilitates the seamless integration of CLFD with various rehearsal-based methods. Figure 1 (right) demonstrates that CLFD significantly improves both the training efficiency and accuracy of rehearsal-based methods when implemented on the edge device.In summary, our contributions are as follows:• We propose the CLFD, a novel framework designed to improve the efficiency of CL. This framework enhances the training by mapping input features in the frequency domain and compressing these frequency domain features. To the best of our knowledge, this study represents the first attempt to utilize frequency domain features to enhance the performance and the efficiency of CL on edge devices.• CLFD stores and replays encoded feature maps instead of original images, thereby enhancing the efficiency of storage resource utilization. Concurrently, CLFD minimizes interference among frequency domain features, significantly boosting the performance of rehearsal-based methods across all benchmark datasets. These improvements can increase accuracy by up to 6.83% compared to the SOTA methods. • We evaluate the CLFD framework on an actual edge device, showcasing its practical feasibility.The results indicate that our framework can achieve up to a 2.6× improvement in training speed and a 3.0× reduction in peak memory usage.2 Related Work 
continual learning: 
 The current CL methods can be categorized into three primary types: Regularization-based methods [42, 30, 11, 22, 23] limit updates to key parameters to minimize drift in network parameters essential for previous tasks. Architecture-based methods [14, 24, 35, 45, 52] assign distinct parameters to each task or add network components for new tasks to decouple task-specific knowledge.Rehearsal-based methods [2, 6, 8, 12, 10] mitigate forgetting by maintaining an episodic memory buffer and continuously replaying samples from previous tasks to approximate the joint distribution of tasks during training. Among these, our framework focuses on rehearsal-based methods, as these methods are acknowledged as the most effective in mitigating catastrophic forgetting [6] . ER [38] enhances CL by integrating training samples from both the current and previous tasks. Expanding upon this concept, DER++ [6] enhances the learning process by retaining previous model output logits and utilizing a consistency loss during the model update. ER-ACE [7] safeguards learned representations and minimizes drastic adjustments required for adapting to new tasks, thereby mitigating catastrophic forgetting. Moreover, CLS-ER [4] mimics the interaction between rapid and prolonged learning processes by maintaining two supplementary semantic memories.A limited number of works explore training efficiency in CL [46, 27, 16] . Among these methods, SparCL [46] reduces the FLOPs required for model training through the implementation of dynamic weight and gradient masks, along with selective sampling of crucial data. These methods accelerate the training process through pruning and sparse training. Nevertheless, our framework enhances efficiency by reducing the size of the input feature map, which is an orthogonal optimization to pruning.Some studies [48, 15, 18, 21] utilize DCT to map images into the frequency domain and enhance the inference speed of the models. However, these methods are not conducive to enhancing rehearsalbased methods. Previous research [6] indicates that data augmentation can significantly boost the performance of rehearsal-based methods. Nevertheless, utilizing DCT results in a total loss of spatial information, thereby restricting the application of data augmentation. Other studies [29, 33, 32, 47, 17, 13] employ DWT to improve the classification performance of models. While wavelet transform effectively preserves the spatial features of images, these methods are not well-suited for CL due to the substantial increase in learnable parameters they introduce. In CL, this proliferation of parameters significantly raises the risk of catastrophic forgetting across tasks. MgSvf [51] utilizes the frequency domain in the context of CL, focusing on the influence of different frequency components on model performance. In contrast, our framework delves into the differences in redundancy between the spatial and frequency domains. Compared to the spatial domain, CL in the frequency domain can more effectively remove redundant information from images, thereby improving the efficiency of CL. which balances the reusability and interference of frequency domain features. The entire framework is illustrated in Figure 2 . 
problem setting: 
 The problem of CL involves sequentially learning T tasks from a dataset, where in each task t corresponds to a training set D t = (x i , y i ) Nt i=1 . Each task is characterized by a task-specific data distribution represented by the pairs (x i , y i ). To improve knowledge retention from previous tasks, we employ a fixed-size memory buffer, M = (x i , y i ) B i=1 , which stores data from tasks encountered earlier. Given the inherent limitations in CL, the model's storage capacity for past experiences is finite, thus B ≪ N t . To address this constraint, we utilize reservoir sampling [44] to efficiently manage the memory buffer. In the simplest testing configuration, we assume that the identity of each upcoming test instance is known, a scenario defined as Task Incremental Learning (Task-IL). If the class subset of each sample remains unidentified during CL inference, the situation escalates to a more complex Class Incremental Learning (Class-IL) setting. This research primarily focuses on the more intricate Class-IL setting, while the performance of Task-IL is used solely for comparative analysis.DWT offers effective signal representation in both spatial and frequency domains [28] , facilitating the reduction of input feature size. Compared with the DWT, DCT coefficients predominantly capture the global information of an image, but they fail to preserve the spatial continuity that is typical in normal images. In DCT, local spatial information is mixed, resulting in a loss of distinct local features. In contrast, DWT effectively integrates both spatial and frequency domain information, maintaining a balance between the two. Furthermore, the DWT method can be seamlessly integrated with data augmentation techniques in rehearsal-based methods, enhancing its applicability and effectiveness.For 2D signal X ∈ R N ×N , The signal after DWT can be represented as:EQUATIONwhere L and H represent the low-frequency and high-frequency filters of orthogonal wavelets, respectively. These filters are truncated to the size of N 2 × N . The term X ll refers to the low-frequency component, while X lh , X hl , X hh represents the high-frequency components. We select the Haar wavelet as the basis for the wavelet transform because of its superior computational efficiency [28] , which is well-suited for our tasks. Previous methods [29, 47, 28] typically discarded high-frequency components X lh , X hl , X hh and retained low-frequency component X ll . However, focusing solely on low-frequency component leaves many potentially useful frequency components unexplored. Low-frequency component compress the global topological information of an image at various levels, while high-frequency components reveal the image's structure and texture [20] . Therefore, we employ three 1×1 point convolutions to integrate various frequency components. As shown in Figure 3 , we use a 1×1 point convolution to merge low-frequency component X ll to obtain low-frequency features, another one to merge high-frequency components X lh , X hl , X hh to obtain high-frequency features, and a final one to merge all frequency components to obtain global features. These three merged features compose the input feature maps. By utilizing both low and high-frequency components in CL, we can better prevent the loss of critical information while reducing input feature size. Since each merged feature's width and height are half of the original image, another advantage of working in the frequency domain is that the spatial size of the original image (H × W × 3) is reduced by half in both width and height (H/2 × W/2 × 3) after FFE. With the reduced spatial size, the computational load and peak memory requirements of CL models decrease. Moreover, the reduction in spatial size means that more replay samples can be stored under the same storage resources, while also reducing the bandwidth required for accessing data. However, setting a specific frequency domain feature encoder for each task may result in significant catastrophic forgetting. Therefore, we freeze the FFE at the end of the first task's training.Considering that tasks are predominantly sensitive to specific frequency domain features extracted by a feature extractor, different tasks prioritize distinct frequency domain features. To this end, we propose the CFFS, designed to manage the issue of overlap in frequency domain features among samples from different classes. This method promotes comparable classes to utilize similar frequency domain features for classification, while also ensuring that samples from dissimilar classes employ divergent features. Consequently, this method reduces interference among various tasks and mitigates overfitting issues. For specific classes, we select a predetermined number of frequency domain features based on the absolute values of these features. Subsequently, unselected features are masked to prevent their interference in the classification process. We utilize a counter F ∈ R C×N to track the number of selections for each frequency domain feature among samples associated with a specific class. N and C denote the dimensions and the classes of frequency domain features, respectively. We utilize cosine similarity to evaluate the frequency domain similarity between two class samples. To decrease computational complexity, only low-frequency component X ll is utilized for calculating cosine similarity. The similarity between class i and class j is expressed as follows:EQUATION)The value of cosine similarity is determined solely by the direction of the features, regardless of their magnitude. Consequently, f i represents the sum of the low-frequency component of samples in class i. We then select the class that exhibits the greatest similarity and the class that displays the least similarity to the current class:EQUATIONwhere K represents the total number of classes in the preceding task. Several studies [39, 43] employ Heterogeneous Dropout [1] to enhance the selection of underutilized features for subsequent tasks. While this method helps manage the overlap of feature selection across different tasks, it overlooks for e = 1, . . . , K t do3: if e < E then 4: if t > 1 then 5:Dropout features based on frequency dropout probabilities 1 -P f 6:if e == 1 then Update F similarities among classes. This oversight can negatively impact the effectiveness of selecting features in the frequency domain. To this end, we propose the Frequency Dropout method, which adjusts the probability of discarding frequency domain features based on class similarity. Specifically, let [F y ] j denote the number of the j-th frequency domain feature when learning class y. The probability of selecting this feature in class c while learning a new task is expressed as follows:EQUATIONwhere S c denotes the average cosine similarity between class c and all classes in previous tasks. The parameters α - c and α + c control the intensity of the selection process. A higher value indicates a greater overlap of activated features with analogous classes and a reduced overlap with non-analogous classes. The coefficient λ serves as a weighting factor that adjusts the selection of frequency domain features, determining whether the emphasis is more towards similar classes or less towards dissimilar ones. The Frequency Dropout probability is updated at the beginning of each task. After completing training for E epochs on a given task, Semantic Dropout [39] is employed instead of Frequency Dropout. It encourages the model to use the same set of frequency domain features for classification by setting the retention probability of frequency domain features in each class. This probability is proportional to the number of times that frequency domain feature has been selected in that class so far:EQUATIONwhere β c controls the strength of dropout. The probability of Semantic Dropout is updated at the end of each epoch, thereby enhancing the model's established frequency domain feature selection. This adjustment effectively regulates the extent of overlap in the utilization of frequency domain features. Algorithm 1 outlines the procedure for the CFFS.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: We provide code to reproduce the primary experimental results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setup: 
 Datasets. We conduct comprehensive experimental analyses on extensively used public datasets, including Split CIFAR-10 (S-CIFAR-10) [6] and Split Tiny ImageNet (S-Tiny-ImageNet) [9] . The S-CIFAR-10 dataset is structured into five tasks, each encompassing two classes, while the S-Tiny-ImageNet dataset is divided into ten tasks, each comprising twenty classes. Additionally, the standard input image size for these datasets is 32 × 32 pixels. Evaluation metrics. We use the average accuracy on all tasks to evaluate the performance of the final model:EQUATIONWe denote the classification accuracy on the τ -th task after training on the t-th task as R t,τ . Moreover, we evaluate the training time, training FLOPs and peak memory footprint [46] to demonstrate the efficiency of each method. More experimental results can be found in Appendix F.Baselines. We compare CLFD with several representative baseline methods, including three regularization-based methods: oEWC [41] , SI [50] and LwF [30] , as well as four rehearsal-based methods: ER [38] , DER++ [6] , ER-ACE [7] and CLS-ER [4] . In our evaluation, we incorporate two non-continual learning benchmarks: SGD as the lower bound and JOINT as the upper bound.Implementation Details We expand the Mammoth CL repository in PyTorch [6] . For the S-CIFAR-10 and S-Tiny-ImageNet datasets, we utilize a standard ResNet18 [19] without pretraining as the baseline model, following the method outlined in DER++ [6] . All models are trained using the Stochastic Gradient Descent optimizer with a fixed batch size of 32. Additional details regarding other hyperparameters are detailed in Appendix D and E. For the S-Tiny-ImageNet dataset, models undergo training for 100 epochs, whereas for the S-CIFAR-10 dataset, training lasts for 50 epochs per task. In rehearsal-based methods, each training batch consists of an equal mix of new task samples and samples retrieved from the buffer. To ensure robustness, all experiments are conducted 10 times with different initializations, and the results are averaged across these runs.Table 1 presents a comparative analysis of the results on the S-CIFAR-10 and S-Tiny-ImageNet datasets, evaluated under Class-IL and Task-IL settings. The results elucidate that CLFD significantly We evaluate the acceleration performance of the CLFD utilizing the NVIDIA Ampere architecture GPU and Octa-core Arm CPU on the NVIDIA Jetson Orin NX 16GB platform. We measure the training time and accuracy of various methods using the S-CIFAR-10 dataset with a buffer size of 125. To expedite the training process, data augmentation techniques were omitted, resulting in accuracy results that vary from those reported in Table 1 . Figure 4 illustrates the training time and average accuracy of various methods. When combined with various CL methods, CLFD significantly reduces training time while simultaneously enhancing model accuracy. By achieving approximately 2.4 × training acceleration, CLFD can attain the highest average accuracy of 47.64% when integrated with ER-ACE. This suggests that CLFD significantly enhances the efficiency of CL on edge devices by reducing the input feature map size.In Table 2 , we present a comprehensive ablation study of CLFD-ER employing a buffer size of 50 on the S-CIFAR-10 dataset. The results indicate that each component of our method makes a substantial contribution to enhancing accuracy. Comparing row 1 and 2, we can see that FFE enhances the utilization of storage resources by storing encoded features from the frequency domain rather than the original images. This method significantly improves the accuracy of rehearsal-based methods by optimizing the representation of stored data. Comparing rows 1 and 3, we can see that CFFS enhances the model's accuracy by mitigating the interference among frequency domain features across different tasks. CLFD achieves superior average accuracy by comprehensively integrating all components, thereby substantiating the efficacy of its individual elements. In this section, we provide an in-depth analysis of CLFD.Sparse Training We compare our method with SparCL [46] , a SOTA sparse training method. Table 3 presents the average accuracy and training FLOPs for CLFD and SparCL. Our method significantly enhances the average accuracy compared to SparCL. Although our method incurs higher training FLOPs than SparCL, this does not directly correlate with actual training speed. Our method accelerates training speed without requiring additional optimization. Conversely, pruning and sparse training methods that utilize masks often fail to translate into actual training time savings without optimizations at the compiler level. Furthermore, the combination of CLFD and SparCL not only achieves the highest accuracy but also leads to the lowest training FLOPs. The successful integration of CLFD and SparCL serves as an example of CLFD's adaptability to sparse training and pruning methods.To explore the influence of different frequency components, we employ them as input feature maps and assess the CLFD-DER++ accuracy on the S-CIFAR-10 dataset under 50 buffer size. Table 4 presents the average accuracy across various frequency components. Utilizing the low-frequency components of images as input feature maps yields the highest accuracy among different frequency components. This suggests that CNN models demonstrate greater sensitivity to low-frequency channels compared to high-frequency channels. This finding aligns with the characteristics of the HVS, which also prioritizes low-frequency information. Despite this, FFE achieves the highest average accuracy, suggesting that high-frequency components are also significant. Optimal preservation of image information during downsampling is achieved only through the integration of components across different frequencies. Some examples of the encoded frequency domain feature maps are visualized in Figure 6 .Frequency Domain Feature Selection To evaluate the effectiveness of frequency dropout in reducing interference among frequency domain features across diverse tasks, we calculate the selection of specific frequency domain features for different classes. Specifically, we track the classification activities on the test set and conduct normalized counts of selections for the frequency domain features, as illustrated in Figure 5 . We observe that feature selection patterns exhibit higher correlations among semantically similar classes. For instance, the classes "cat" and "dog" often select identical sets of features. Similarly, significant similarities in feature selection patterns are evident between "auto" and "truck". This result demonstrates the effectiveness of CFFS. 
conclusion: 
 Inspired by the human visual system, we propose CLFD, a comprehensive framework designed to enhance the efficiency of CL training and augment the precision of rehearsal-based methods. To effectively reduce the size of feature maps and optimize feature reuse while minimizing interference across various tasks, we propose the Frequency Domain Feature Encoder and the Class-aware Frequency Domain Feature Selection. The FFE employs wavelet transform to convert input images into the frequency domain. Meanwhile, the CFFS selectively uses different frequency domain features for classification depending on the frequency domain similarity of classes. Extensive experiments conducted across various benchmark datasets and environments have validated the effectiveness of our method, which enhances the accuracy of the SOTA method by up to 6.83%. Moreover, it achieves up to a 2.6× increase in training speed and a 3.0× reduction in peak memory usage. We discuss the limitations and broader impacts of our method in Appendix A and B, respectively. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Section 4.1, Appendix D and Appendix E. In addition, we provide code to reproduce the primary experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: See Table 1 , Table 2 , Table 3 , Table A2 and Figure 4 Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: See Appendix B Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We mentioned and cited the datasets (S-CIFAR-10 and S-Tiny-ImageNet), as well as all comparative methods with their respective papers. The licenses for the datasets and models used are provided in the cited references and explicitly stated in Appendix C.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: One limitation of our framework is its focus on optimizing rehearsal-based methods. While the use of a rehearsal buffer throughout the CL process is widely accepted, there exist scenarios where rehearsal buffers are prohibited. Nonetheless, by reducing the size of the input feature map, our framework has the potential to accelerate various CL methods, though the implications for model performance require further investigation. Furthermore, our analysis was limited to scenarios with finite rehearsal buffer sizes, whereas many current studies focus on scenarios with infinite rehearsal buffer sizes. This shift is primarily due to memory constraints being less significant compared to computational costs. We will also continue to investigate the performance of our method in scenarios involving infinite rehearsal buffer sizes.Inspired by HVS, we propose a novel framework called CLFD, which utilizes frequency domain features to enhance the performance and efficiency of CL training. Its success has opened up opportunities for enhancing existing CL methods in the frequency domain. CLFD contributes to the responsible and ethical deployment of artificial intelligence technologies by improving CL performance and efficiency. This is accomplished through the efficient ability of models to update and refine their knowledge without the need for extensive retraining. This further facilitates the real-world application of CL.Although CLFD serves as a comprehensive framework aimed at improving the efficiency of various CL methods, we must remain cognizant of its potential negative societal impacts. While CLFD improves model stability, it does so by compromising the expense of model plasticity, resulting in reduced accuracy when applied to new tasks. This trade-off requires specific consideration in applications where accuracy is crucial, such as healthcare [46] . Furthermore, as a powerful tool for enhancing the efficiency of CL methods, CLFD could also strengthen models designed for malicious applications [5] . Therefore, it is recommended that the community devise additional regulations to mitigate the malicious use of artificial intelligence.• CIFAR-10 [26] is licensed under the MIT license.• The licensing information for Tiny-ImageNet is unavailable. However, the dataset is accessible to researchers for non-commercial purposes.Table A1 presents the selected optimal hyperparameter combinations for each method in the main paper. The hyperparameters include the learning rate (lr), batch size (bs), and minibatch size (mbs) for rehearsal-based methods. Other symbols correspond to specific methods. It should be noted that the batch size and minibatch size are held constant at 32 for all CL benchmarks.We conduct comprehensive experiments utilizing the NVIDIA GTX 2080Ti GPU paired with the Intel Xeon Gold 5217 CPU, as well as the NVIDIA Jetson Orin NX 16GB, boasting NVIDIA Ampere architecture GPU and Octa-core Arm CPU.We set λ = 0.5, β c = 2 in equation ( 4) and equation ( 5). We also set E at a value of 0.4 of the training epochs. In CFFS, we only select 60% of the frequency domain features for classification. We utilize the code from DER++ [6] . We extend our gratitude to the authors for their support and for providing In line with [6] , we employ random crops and horizontal flips as data augmentation techniques for both examples from the current task and the replay buffer. To ensure uniformity in data augmentation between the original images and the input features encoded with the FFE, random cropping is restricted to even pixels only.If the CL model can retain previously learned information, it is considered stable; if it can effectively acquire new information, it is considered plastic. To better understand how various methods balance stability and plasticity, we investigate the evolution of task performance as the model learns tasks sequentially. Figure 7 shows that CLFD consistently enhances the balance of all CL methods, delivering more uniform performance across all tasks. In addition, to further demonstrate the effectiveness of our method, we introduce a trade-off measure [40] that approximates how the model balances its stability and plasticity. Upon the model's completion of the final task T , its stability is evaluated by calculating the average performance across all preceding T -1 tasks as follows:EQUATIONThe plasticity of the model (P) is evaluated by computing the average performance of each task after its initial learning i.e. the diagonal of the heatmap:EQUATIONThus, the trade-off measure determines the optimal balance between the model's stability (S) and plasticity (P ). This measure is calculated as the harmonic mean of S and P .Trade-off = 2SP S + PFigure 8 provides the stability-plasticity trade-off measure for different CL methods. ER and DER++ exhibit high plasticity, enabling them to rapidly adapt to new information. However, they lack the ability to effectively retain previously acquired knowledge, leading to task recency bias issues. CLFD consistently improves the stability of CL methods, thereby reducing task reception bias and enhancing the balance between stability and plasticity.We use the Final Forgetting (FF) to measure the model's anti-forgetting performance:EQUATIONA smaller FF value indicates that the model exhibits less forgetting of previous knowledge, thereby demonstrating stronger anti-forgetting performance. Table A2 presents the FF result. Our method consistently reduces forgetting in CL methods, demonstrating the effectiveness of CLFD in preserving prior knowledge rather than solely focusing on improving the accuracy of subsequent tasks.To further enhance the credibility of our results, we conduct additional experiments on ImageNet-R dataset. ImageNet-R, an extension of the ImageNet dataset, comprises 200 classes and a total of 30,000 images, with 20% designated for the test set. We divide ImageNet-R into ten tasks, each containing 20 classes. Input images are resized to 224 × 224 pixels. We test the Class-IL accuracy of each task with a buffer size of 500, maintaining consistent hyperparameters across all methods as those used in Split Tiny ImageNet. Figure 9 presents the experimental results, demonstrating that our method still significantly enhances the performance of various rehearsal-based CL methods, We conduct additional ablation experiments to investigate the impact of feature selection proportions on model performance. We focus on two methods: CLFD-ER and CLFD-ER-ACE, as Figure 8 shows that ER is the most plastic method, while ER-ACE is the most stable. We conduct tests on the S-CIFAR-10 dataset with a buffer size of 50. Table A3 presents the result. In the Task-IL setting, both CLFD-ER and CLFD-ER-ACE achieve higher accuracy with higher feature selection proportions, as the inclusion of more features enables the model to better distinguish between classes within the tasks. It is important to highlight that in CLFD-ER, lower feature selection proportions can also lead to positive outcomes. This phenomenon can be attributed to the high plasticity of the CLFD-ER method, where reducing feature selection proportions aids in mitigating potential interference among various tasks. In the Class-IL setting, CLFD-ER achieves higher accuracy with lower feature selection proportions, while CLFD-ER-ACE exhibits superior performance with higher feature selection proportions. This observation suggests that for methods with high plasticity, we need to decrease the feature selection proportions to mitigate the overlap of frequency domain features, thereby minimizing the impact of new tasks on old tasks. Conversely, for methods with high stability, increasing the feature selection proportions allows us to utilize more frequency domain features to learn the current classes effectively.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: This work leverages the frequency domain to enhance the performance and training efficiency of rehearsal-based continual learning methods.  For the input features of the feature extractor, CLFD employs wavelet transform to map the original input image into the frequency domain, thereby effectively reducing the size of input feature maps. Regarding the output features of the feature extractor, CLFD selectively utilizes output features for distinct classes for classification, thereby balancing the reusability and interference of output features based on the frequency domain similarity of the classes across various tasks. 

Initially the paper received mixed reviews. Reviewers have appreciation for the method design, extensive experiments, and writing, while raising concerns on lack of experiments on more complex datasets, method scalability, etc. The authors have provided a strong rebuttal that convinced the reviewer enough to support acceptance. The AC-panel concurs with their decision and strongly encourages the authors to include all additional clarifications and important experiments in the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel framework designed to enhance the efficiency and effectiveness of continual learning (CL) systems by leveraging frequency domain representations, inspired by the human visual system's varying sensitivity to different frequency components. This approach aims to address the limitations of existing rehearsal-based methods in CL, particularly under constraints like limited resources on edge devices. The framework, named Continual Learning in the Frequency Domain (CLFD), uses a wavelet transform to convert input images into frequency domain representations, optimizing both the input and output features of the feature extractor. This allows for better management of memory usage and computational demands, leading to improvements in both accuracy and training efficiency. Extensive experiments demonstrate that CLFD can enhance the performance of state-of-the-art rehearsal-based methods, achieving higher accuracy and reduced training times.
strength_weakness: strength_weakness: 1.	While the framework shows promising results in specific settings and datasets (like Split CIFAR-10 and Split Tiny ImageNet), the paper does not thoroughly discuss its performance across a wider range of scenarios or more complex datasets. 
2.	The effectiveness of the proposed method is somewhat dependent on the buffer size used for rehearsal in continual learning. The reliance on buffer size might limit its utility in extremely constrained environments where memory is severely limited.
3.	Freezing the FFE based on the first task might introduce scalability issues, as the encoder might not efficiently handle the complexity introduced by a broader set of tasks or more diverse data.
4.	In Table 1, it is observed that under the Task Incremental Learning (Task-IL) setting, the integration of the CLFD framework leads to worsened performance in some experimental results. This highlights a potential weakness, as the paper does not sufficiently explain why the framework underperforms in these specific scenarios.
5.	The paper exhibits several technical and typographical issues that affects its formal presentation and readability. For example, the multiplication symbol in section 3.3 is represented as “x” instead of using a standard “$\times$”. In Eq. 2, symbols are not bolded. Figure 4’s caption lacks a period at the end. Other typos like writing “FFE” as “FFD” in the conclusion section, introduces confusion and can be misleading about key terms and components described in the paper. In addition, there are errors in the formulation of some equations. For instance, Eq. 7’s subscript should start from 1 instead of 0. Similarly, Eq. 8 is also incorrectly formulated.
questions: questions: Please refer to the weaknesses.
limitations: limitations: The limitations have been discussed in Appendix A.
paper_summary: paper_summary: In this paper, the authors proposed a novel replay-based continual learning, which is named continual learning in the frequency domain (CLFD). The framework consists of two main modules, frequency domain feature encoder (FFE) and class-aware frequency domain feature selection (CFFS). FFE utilizes discrete wavelet transform (DWT) to transform the RGB images into a frequency domain. CFFS computes similarity and selects suitable frequency domain features for classification.
strength_weakness: strength_weakness: The weaknesses of this paper are listed below:
- Some parts are not presented clearly. More details may included. Including a pseudocode may help.
- Some notations are not explained clearly, e.g. sec. 3.4.
- Even though there are many experiments, more ablation studies about hyperparameters may provide more information.
questions: questions: How is the historical data stored? What format is the data in a memory buffer? How do you use it for replay?

Could you please give some more details about CFFS? The overall idea can be understood, but the details are unclear because of confusing notations.
limitations: limitations: As the authors have discussed, replay-based methods are unsuitable for all scenarios.
paper_summary: paper_summary: Based on the research that human visual system (HVS) exhibits varying sensitivities to different frequency components, this paper proposes to do continual learning in the wavelet frequency domain to reduce the size of inputs. The proposed CLFD module includes feature extractor and feature encoder, where the feature encoder generate low-frequency features, global features and high-frequency features, and the feature extractor selects class-specific frequency features. The generated features are used to do continual learning.
strength_weakness: strength_weakness: 1. It is better to include larger databases such as imagenet-1k into the experiments to make the results more convincing.
2. Table-1 leaves some unclear items. For instance, the meaning of class-IL (class incremental learning) and task-IL (task incremental learning) should be explained. The meaning of shadowed lines and bold texts in the table should be included. 
3. The classification performance of the proposed method seems have a large gap with the state-of-the-art methods on both cifar-10 and tiny-imagenet.
questions: questions: 1. How about the time-consuming of the processes of wavelet transform and the forward and backward of CLFD module?
2. Please clearly explain table-1 for better understanding.
limitations: limitations: 1. The authors may consider to do some discuss about the different frequency domains such as fourier domain and discrete cosine domains.
paper_summary: paper_summary: Inspired by the human visual system (HVS), this paper proposes a new framework called Continual Learning in the Frequency Domain (CLFD) for edge devices. For the input features of the feature extractor, CLFD employs wavelet transforms to map the original input image to the frequency domain, thereby reducing the size of the input feature map. In experiments on two public datasets, the performance of the proposed and conventional methods is discussed in terms of both accuracy and learning efficiency.
strength_weakness: strength_weakness: Throughout, the explanation of the proposed method needs to be more comprehensive. Experiments also need to be more comprehensive to demonstrate the effectiveness of the proposed method. Specifically, the paper has the following rooms for improvement.
- In line 60, there needs to be a clear explanation of why using frequency space is adequate. While it is interesting to get inspiration from HVS, there is no apparent reason why it is a means to the challenge of the proposed method (continuous learning on edge devices). In other words, the introduction needs a more logical structure.
- In Figure 2, the meaning of the symbols (e.g., \otimes, etc.) is unclear, so a specific explanation is needed. The clarity of the figure needs to be improved.
- In the description of the method, there is the following statement: "Considering that tasks are predominantly sensitive to specific frequency domain features extracted by a feature extractor, different To this end, we propose the CFFS, designed to manage the issue of overlap in frequency domain features among samples from different classes." However, no results from the analysis support this issue (fact). Furthermore, it needs to state why CFFS is the idea to solve this fact. Therefore, the design of the proposed method needs to be more convincing.
- Many grammatical errors need to be corrected. A comma or period is needed after the formula. (e.g., line 210: Where->where)
- The method's design throughout is very ad hoc and heuristic. For example, in Equation 4, no clear reason is given for the algorithm's design.
- Only the overall ACC represented by Equation 6 is evaluated in the experiment. In continuous learning, there are other evaluation measures (e.g., Forgetting measure). In addition, task-specific ACCs and other measures need to be evaluated. "[Chaudhry et al., 2018b] Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In Proc. ICLR, 2018."
- In Table 1, there are no experiments for larger Buffer numbers. For example, previous studies (CLS-ER and SparCL-ER) have more buffers in their paper. Why are there no comparisons for larger numbers of buffers? It is unfair from the point of view of experiments in academic papers to publish only comparisons of conditions in which the proposed method is superior. (For example, edge devices that will be satisfied with larger buffer sizes may be developed.)
- Furthermore, it is difficult to judge the effectiveness of the proposed method since the experiments do not evaluate larger datasets; in this paper, only smaller datasets such as CIFAR and TinyImageNet are used.
- When the number of tasks increases, the proposed method will likely perform poorly for the latter tasks.
questions: questions: The paper and the proposed method need both clarity and experimentation. 
In particular, the proposed method's effectiveness is difficult to determine because of its ad hoc design and insufficient explanation.

- Please explain more about the necessity of utilizing frequency space for continuous learning for edge devices.
- Furthermore, I'm curious about the absence of experiments with larger Buffer numbers in Table 1. I think edge devices that will be satisfied with larger buffer sizes may be developed in the future.
- The overall ACC, represented by Equation 6, is the only evaluation. Other evaluation measures (e.g., forgetting measures) also exist in continuous learning. In addition, task-specific ACCs, etc., need to be evaluated. Please explain why you did not use these evaluation measures.
limitations: limitations: - For large data sets, the proposed method may need to be revised.
- When the number of buffers is large, the proposed method may be inferior to the conventional method.
- When the number of tasks increases, the proposed method is likely not to perform well for the latter tasks.

Comments: title: Reply to Reviewer wmJG
Supplementary experiments on the Split ImageNet-R dataset
Reply to Reviewer KM8C
More ablation studies about hyperparameters
Reply to Reviewer wmJG
Reply to Reviewer KM8C
Reply to Reviewer wmJG
Official Comment by Reviewer KM8C
comment: Thank you for your response. I will keep the rating.
Thank you for your timely response, which provides a more comprehensive understanding of the proposed method. Given these additional experiments, I have increased my rating to 6: Weak Accept. I also appreciate the authors' patient and detailed response, and I have no further questions.
We express our sincere appreciation for the thoughtful feedback offered by the reviewer and the constructive evaluation of our work. We have addressed all the technical details and typographical issues as per your suggestions. Additionally, following your recommendation, we conduct a detailed comparison and ablation study on the other hyperparameters presented in Section E.2. Since our framework can be integrated with various continual learning methods, we do not use grid search; instead, we individually investigate the impact of each hyperparameter on continual learning performance. We focus on the CLFD-ER-ACE method for this study, as it performs well across all datasets and does not introduce any additional hyperparameters. We conduct tests on the S-CIFAR-10 dataset with a buffer size of 50. The accuracy results under the Class-IL setting are as follows:

| $\beta_c$ in Eq. 5           | 0.5 | 1 | 1.5 | 2 | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 |
|-------------------------------|-----|---|-----|---|-----|---|-----|---|-----|---|
| CLFD-ER-ACE                   | 50.70 | 50.19 | 51.47 | 52.74 | 50.29 | 52.48 | 51.65 | 51.41 | 50.73 | 52.36 |

| $\lambda$ in Eq. 4           | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 |
|-------------------------------|-----|-----|-----|-----|-----|-----|-----|-----|-----|
| CLFD-ER-ACE                   | 50.85 | 52.08 | 51.24 | 51.68 | 52.74 | 51.77 | 50.81 | 50.68 | 51.81 |


| $\mathcal{E}$ in CFFS           | 5  | 10  | 15  | 20  | 25  | 30  | 35  | 40  | 45  |
|---------------------------------|----|-----|-----|-----|-----|-----|-----|-----|-----|
| CLFD-ER-ACE                     | 52.06 | 52.27 | 52.16 | 52.74 | 52.43 | 52.18 | 52.65 | 52.64 | 52.01 |


Our choice of hyperparameters in the paper yields optimal performance, and our framework demonstrates robustness to these hyperparameter selections. We hope that these ablation studies will provide the reviewer with a more comprehensive understanding of our work. We once again extend our gratitude for your thoughtful and highly constructive feedback.
We have completed the experiments with other methods on the Split ImageNet-R dataset under the 20-task setting. The results are as follows:
| Methods           | Class-IL Accuracy (%) | Task 10 | Task 11 | Task 12 | Task 13 | Task 14 | Task 15 | Task 16 | Task 17 | Task 18 | Task 19 | Task 20 |
|-------------------|-----------------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
| ER                | 5.22                      |    9.49     |    8.82     |    7.44     |    6.87     |    5.93     |    5.34     |    4.70     |    5.63     |    5.13     |    5.33     |    5.22     |
| CLFD-ER           | 5.23                      |    9.51     |    10.70     |    8.79     |    7.32     |    6.32     |    5.94     |    4.77     |    5.76     |    5.46     |   5.97      |    5.23     |
| DER++             | 9.47                  |    16.37     |    15.23     |    9.58     |    12.15     |    11,31     |    10.64     |    9.79     |    9.78     |    9.66     |    6.10     |    9.47     |
| CLFD-DER++        | 10.05                 |     18.04    |     17.64    |    12.16     |    15.66     |    12.56     |    12.13     |    11.00     |    10.48     |    10.17     |    7.86     |    10.05     |
| ER-ACE            | 10.63                 |  15.38  |  15.37  |  15.26  |  12.75  |  11.60  |  12.22  |  8.63   |  7.48   |  10.07  |  10.00  |  10.63  |
| CLFD-ER-ACE       | 13.44                 |  15.93  |  16.43  |  17.20  |  13.59  |  15.42  |  13.08  |  9.75   |  14.69  |  13.26  |  12.79  |  13.44  |
| CLS-ER            | 8.72                      |    6.83     |    6.38     |    6.09     |    6.63     |    6.38     |    7.17     |    6.91     |    7.16     |    7.35     |    8.07     |    8.72     |
| CLFD-CLS-ER       | 11.12                      |    9.77     |     10.30    |    9.99     |   10.64      |    10.72     |    11.32     |    11.14     |    11.22     |    11.36     |    11.43     |   11.12      |

These results provide a more comprehensive demonstration of the scalability of our framework. We have included a comprehensive analysis of the framework's scalability in the final version of the paper.
I appreciate your timely response and insightful analysis. The authors have addressed most of the concerns I raised in my initial review. I recommend that the authors focus on resolving the remaining technical details that I initially mentioned, particularly ensuring all technical and typographical issues are thoroughly addressed. Additionally, further enhancement of the manuscript could be achieved by including detailed comparative and ablation studies about other hyperparameters as indicated in E.2. These additions would provide a more comprehensive understanding of the proposed method, and I believe that addressing these final points will make it a solid contribution to the field.
We sincerely thank you for your thoughtful and highly constructive feedback, which has significantly contributed to improving the quality of our manuscript. Both FSDR [1] and Mgsvf [2] utilize the discrete cosine transform to incorporate frequency domain information, with a focus on how different frequency components influence the model. In contrast, our framework explores the differences in redundancy between the spatial and frequency domains. The frequency domain facilitates the more effective elimination of redundant information from images compared to the spatial domain, thereby enabling more efficient continual learning—an aspect not explored in previous work. Consequently, while Mgsvf primarily aims to enhance the model’s resistance to forgetting, our framework not only improves the accuracy of continual learning but also significantly increases training efficiency. We will expand the discussion on these two works in Section 3.2 to clarify our position. Once again, we sincerely appreciate your feedback in helping us to enhance the quality of our manuscript. If there are any other areas that require further clarification, we are ready to provide additional explanations as needed.

[1] Jiaxing Huang et al. "FSDR: Frequency Space Domain Randomization for Domain Generalization," in Proc. CVPR , 2022

[2] Zhao, Hanbin, et al. "Mgsvf: Multi-grained slow versus fast framework for few-shot class-incremental learning," IEEE TPAMI, 2021
We conduct supplementary ablation experiments to examine the influence of different feature selection proportions on model performance. Our analysis concentrated on two methods: CLFD-ER and CLFD-ER-ACE. Figure 8 in the appendix illustrates that ER exhibits the highest degree of plasticity, whereas ER-ACE demonstrates the greatest stability. We conduct tests on the S-CIFAR-10 dataset with a buffer size of 50. The accuracy results under the Task-IL setting are as follows:

| Feature Selection Proportions | 10%  | 20%  | 30%  | 40%  | 50%  | 60%  | 70%  | 80%  | 90%  |
|-------------------------------|------|------|------|------|------|------|------|------|------|
| CLFD-ER                        |   85.67   |   85.47   |   84.91   |   84.89   |   83.88   |   84.45   |   84.49   |   84.99   |   87.97   |
| CLFD-ER-ACE                    |   85.06   |   85.86   |   86.74   |   86.54   |   87.05   |   87.13   |   87.30   |   88.12   |   89.83   |


And the accuracy results under the Class-IL setting are as follows:

| Feature Selection Proportions | 10%  | 20%  | 30%  | 40%  | 50%  | 60%  | 70%  | 80%  | 90%  |
|-------------------------------|------|------|------|------|------|------|------|------|------|
| CLFD-ER                        |   51.03   |   49.97   |   48.91   |   46.67   |   45.69   |   45.56   |   43.88   |   41.79   |   39.58   |
| CLFD-ER-ACE                    |   50.37   |   50.50   |   50.64   |   50.87   |   52.12   |   52.74   |   52.84   |  53.97    |   54.20   |

We also conduct an ablation study on the other hyperparameters presented in Section E.2. Since our framework can be integrated with various continual learning methods, we do not use grid search; instead, we individually investigate the impact of each hyperparameter on continual learning performance. We focus on the CLFD-ER-ACE method for this study, as it performs well across all datasets and does not introduce any additional hyperparameters. We conduct tests on the S-CIFAR-10 dataset with a buffer size of 50. The accuracy results under the Class-IL setting are as follows:

| $\beta_c$ in Eq. 5           | 0.5 | 1 | 1.5 | 2 | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 |
|-------------------------------|-----|---|-----|---|-----|---|-----|---|-----|---|
| CLFD-ER-ACE                   | 50.70 | 50.19 | 51.47 | 52.74 | 50.29 | 52.48 | 51.65 | 51.41 | 50.73 | 52.36 |

| $\lambda$ in Eq. 4           | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 |
|-------------------------------|-----|-----|-----|-----|-----|-----|-----|-----|-----|
| CLFD-ER-ACE                   | 50.85 | 52.08 | 51.24 | 51.68 | 52.74 | 51.77 | 50.81 | 50.68 | 51.81 |

| $\mathcal{E}$ in CFFS           | 5  | 10  | 15  | 20  | 25  | 30  | 35  | 40  | 45  |
|---------------------------------|----|-----|-----|-----|-----|-----|-----|-----|-----|
| CLFD-ER-ACE                     | 52.06 | 52.27 | 52.16 | 52.74 | 52.43 | 52.18 | 52.65 | 52.64 | 52.01 |

Our choice of hyperparameters in the paper yields optimal performance, and our framework demonstrates robustness to these hyperparameter selections. We hope that these ablation studies will provide the reviewer a more thorough comprehension of our work. We extend our appreciation for the feedback provided. Should there be any aspects requiring further elucidation, we are prepared to offer further explanations.
We sincerely appreciate the thoughtful feedback provided by the reviewer and the constructive evaluation of our work. Following your suggestion, we conduct additional ablation experiments to investigate the impact of feature selection proportions on model performance. We focus on two methods: CLFD-ER and CLFD-ER-ACE, as Figure 8 in the appendix shows that ER is the most plastic method, while ER-ACE is the most stable. We conduct tests on the S-CIFAR-10 dataset with a buffer size of 50. The accuracy results under the Task-IL setting are as follows:

| Feature Selection Proportions | 10%  | 20%  | 30%  | 40%  | 50%  | 60%  | 70%  | 80%  | 90%  |
|-------------------------------|------|------|------|------|------|------|------|------|------|
| CLFD-ER                        |   85.67   |   85.47   |   84.91   |   84.89   |   83.88   |   84.45   |   84.49   |   84.99   |   87.97   |
| CLFD-ER-ACE                    |   85.06   |   85.86   |   86.74   |   86.54   |   87.05   |   87.13   |   87.30   |   88.12   |   89.83   |


And the accuracy results under the Class-IL setting are as follows:

| Feature Selection Proportions | 10%  | 20%  | 30%  | 40%  | 50%  | 60%  | 70%  | 80%  | 90%  |
|-------------------------------|------|------|------|------|------|------|------|------|------|
| CLFD-ER                        |   51.03   |   49.97   |   48.91   |   46.67   |   45.69   |   45.56   |   43.88   |   41.79   |   39.58   |
| CLFD-ER-ACE                    |   50.37   |   50.50   |   50.64   |   50.87   |   52.12   |   52.74   |   52.84   |  53.97    |   54.20   |


In the Task-IL setting, both CLFD-ER and CLFD-ER-ACE achieve the highest accuracy with the highest feature selection proportions, which aligns with our previous analysis. It is noteworthy that in CLFD-ER, lower feature selection proportions can also yield favorable results. This is due to the high plasticity of the ER method, where reducing feature selection proportions helps to avoid interference between different tasks. In the Class-IL setting, CLFD-ER achieves higher accuracy with lower feature selection proportions, while CLFD-ER-ACE performs better with higher feature selection proportions. This phenomenon indicates that for methods with high plasticity, we need to lower the feature selection proportions to reduce the overlap of frequency domain features, thereby minimizing the impact of new tasks on old tasks. Conversely, for methods with high stability, increasing the feature selection proportions allows us to utilize more frequency domain features to learn the current classes effectively. We once again express our gratitude for your feedback, which will assist us in improving the quality of our manuscript. If there are any aspects that necessitate additional clarification, we are prepared to offer further explanations as needed.
Thank you very much for your thoughtful and meaningful discussion. 

---
> Our framework is “continual learning with the frequency domain (including for edge devices).”

OK. 
I understand that the proposed framework is "continual learning with the frequency domain (including for edge devices)."  However, an algorithm that take advantage of frequency domains have already been proposed in the context of continual learning [a]. Also, exploiting frequency domains is one of the most common approaches in machine learning areas closely related to continual learning (e.g., domain generalization [b] and domain adaptation). We strongly recommend clarifying the position with such papers, e.g., in Sec. 2.3.

[a] Zhao, Hanbin, et al. "Mgsvf: Multi-grained slow versus fast framework for few-shot class-incremental learning," IEEE TPAMI, 2021

[b] Jiaxing Huang et al. "FSDR: Frequency Space Domain Randomization for Domain Generalization," in Proc. CVPR
, 2022

---
Please note that I believe that the authors' responses can solve most of my concerns about this paper, and I would like to re-rate this paper to 5: borderline accept.
Thank you for your timely and detailed response, which addresses most of my concerns. Given these clarifications, I am adjusting my rating to 5: borderline accept. However, I encourage the authors to include additional ablation studies that explore the effect of feature selection proportions on both Class-IL and Task-IL settings. It would also be beneficial to provide a deeper analysis of how these adjustments specifically impact performance across the different settings.
We thank the reviewer for taking the time to review our rebuttal. Below, we provide further clarification on the concerns you have raised.

# Scalability.

To validate the effectiveness of CLFD in addressing scalability issues, we divide the Split ImageNet-R dataset into 20 tasks and evaluate the performance of both ER-ACE and CLFD-ER-ACE. Additionally, we evaluate the average Class-IL accuracy after the completion of each task as the number of tasks increased from 10 to 20. The results are as follows:

| Methods           | Class-IL Accuracy (%) | Task 10 | Task 11 | Task 12 | Task 13 | Task 14 | Task 15 | Task 16 | Task 17 | Task 18 | Task 19 | Task 20 |
|-------------------|-----------------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
| ER-ACE            | 10.63                  |    15.38     |     15.37    |    15.26     |    12.75     |    11.60     |    12.22     |    8.63     |    7.48     |    10.07     |    10.00     |    10.63     |
| CLFD-ER-ACE       | 13.44                 |     15.93    |     16.43    |    17.20     |     13.59    |     15.42    |    13.08     |    9.75     |    14.69     |    13.26     |    12.79     |    13.44     |


The results illustrate that our framework consistently enhances the performance of continual learning methods on the intricate Split ImageNet-R dataset, regardless of whether the dataset is divided into 10 tasks or 20 tasks. Furthermore, at any task boundary between 10 and 20, the accuracy of our framework consistently outperforms the baseline method. This indicates that our framework demonstrates strong scalability. We hope to provide results for additional methods before the end of the discussion period; however, time constraints and limited computational resources are significant challenges. In any case, we will incorporate these results into our final revision.

# Positioning.

We apologize for any confusion. Our framework is “continual learning with the frequency domain (including for edge devices).” The purpose of this framework is to leverage the frequency domain to enhance the performance and training efficiency of continual learning methods. Even when training in the cloud, our framework significantly reduces training time while improving accuracy. On edge devices, the advantages of our framework are even more pronounced due to the stricter memory constraints. Our framework can greatly increase training speed and reduce peak memory usage without requiring any additional optimization, an outcome that previous work has not achieved, thereby facilitating the deployment of continual learning on edge devices. We will revise the corresponding statements to avoid any potential confusion.

We once again express our gratitude to the reviewer for the valuable feedback. Please let us know if any concerns persist. We would be more than willing to provide additional information to ensure a thorough understanding of our work.
We thank the reviewer for swift response. Based on your suggestion, we conducted additional experiments.

# CLFD can effectively overcome scalability issues.

To validate the effectiveness of CLFD in overcoming scalability challenges, we divide the Split ImageNet-R dataset into 20 tasks and evaluate the performance of both ER-ACE and CLFD-ER-ACE. Additionally, we evaluate the average Class-IL accuracy after the completion of each task as the number of tasks increased from 10 to 20. The results are as follows:

| Methods           | Class-IL Accuracy (%) | Task 10 | Task 11 | Task 12 | Task 13 | Task 14 | Task 15 | Task 16 | Task 17 | Task 18 | Task 19 | Task 20 |
|-------------------|-----------------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
| ER-ACE            | 10.63                  |    15.38     |     15.37    |    15.26     |    12.75     |    11.60     |    12.22     |    8.63     |    7.48     |    10.07     |    10.00     |    10.63     |
| CLFD-ER-ACE       | 13.44                 |     15.93    |     16.43    |    17.20     |     13.59    |     15.42    |    13.08     |    9.75     |    14.69     |    13.26     |    12.79     |    13.44     |

The results demonstrate that, on the complex Split ImageNet-R dataset, our framework consistently enhances the performance of continual learning methods, whether the dataset is divided into 10 tasks or 20 tasks. This indicates that our framework exhibits good scalability. We hope to provide results for additional methods before the end of the discussion period; however, time constraints and limited computational resources are significant challenges. In any case, we will incorporate these results into our final revision.

# The performance of CLFD on the S-CIFAR-10 dataset under the Task-IL setting.

Since we primarily focus on performance under the Class-IL setting, we select only 60% of the frequency domain features for classification in CFFS. When we select 90% of the frequency domain features, the accuracy results under the Task-IL setting with a buffer size of 50 are as follows:

| Methods           | Task-IL Accuracy (%) |
|-------------------|-------------------|
| ER           | 86.36               |
| DER++       | 83.51              |
| ER-ACE       | 85.78              |
| CLS-ER       | 89.71              |
| CLFD-ER           | 87.97               |
| CLFD-DER++       | 83.91              |
| CLFD-ER-ACE       | 89.83              |
| CLFD-CLS-ER       | 90.74              |


The results indicate that our framework consistently improves accuracy under the Task-IL setting, which is also validated in Table 2 of the ablation study. When tested on the simpler S-CIFAR-10 dataset, removing the CFFS module enhances accuracy under the Task-IL setting, consistent with our previous explanations regarding CFFS. By adjusting the CFFS module, our framework is able to consistently improve accuracy under both Class-IL and Task-IL settings across all datasets.

# Comparison with SCoMMER [1] and TriRE [2].

Given that our framework efficiently integrates with various continual learning methods, we conduct integration tests of CLFD with SCoMMER [1] and TriRE [2]. We strictly maintain all hyperparameters and experimental settings consistent with those in the original papers. Using the Mammoth CL repository [3], we perform the tests on the S-CIFAR-10 dataset with a buffer size of 200 to ensure a fair comparison. The results are as follows:

| Methods           | Class-IL Accuracy (%) |
|-------------------|-------------------|
| SCoMMER | 61.34               |
| TriRE       | 56.18              |
| CLFD-SCoMMER | 63.69               |
| CLFD-TriRE       | 60.03              |


The experimental results demonstrate that our framework consistently enhances the accuracy and training efficiency of both SCoMMER and TriRE. The observed variations in accuracy are attributed solely to the adjustments made to the data augmentation technique within the Mammoth CL repository, as a unified experimental setup is utilized. The original data augmentation technique from the repository cannot be directly applied to the frequency domain. Hence, we implement a simpler data augmentation technique, which is detailed in Sections E.2 and E.3 of the appendix. 

[1] Sarfraz, Fahad, Elahe Arani, and Bahram Zonooz. "Sparse coding in a dual memory system for lifelong learning." AAAI. 2023.

[2] Vijayan, Preetha et al. “TriRE: A Multi-Mechanism Learning Paradigm for Continual Knowledge Retention and Promotion.” NeurIPS. 2023.

[3] Buzzega, Pietro, et al. "Dark experience for general continual learning: a strong, simple baseline." NeurIPS. 2020.

Once again, we express our gratitude to the reviewer for the valuable feedback provided. If there are any remaining concerns, please inform us. We are prepared to provide additional information to ensure a thorough understanding of our work.
Thank you very much for your kind feedback. I deeply appreciate very thoughtful feedback. However, I still have a few concerns about the following;


1) Scalability.:

In the rebuttal, we discuss the validity of increasing the number of tasks with results of ten tasks. However, as noted in the paper below, in my understanding, the performance is also discussed with a larger number of tasks, e.g., 50 or 20 tasks, in a replay-based manner.

- [i] Rebuffi, Sylvestre-Alvise, et al. "icarl: Incremental classifier and representation learning." in Proc. CVPR 2017. (https://arxiv.org/pdf/1611.07725) 

- [ii] Wu, Yue, et al. "Large scale incremental learning." in Proc.CVPR. 2019.(https://openaccess.thecvf.com/content_CVPR_2019/papers/Wu_Large_Scale_Incremental_Learning_CVPR_2019_paper.pdf)


2) Positioning.:

It is unclear whether the proposed method is "continual learning with the frequency domain (only) for edge devices" or "continual learning with the frequency domain (including for edge devices)". I am a bit confused about whether it is the former or the latter, as the explanation is inconsistent throughout the paper text, title, and rebuttal.
Overall, I am satisfied with the responses provided to the first and second weaknesses outlined in my initial review. 

However, regarding the third weakness, I noticed that there still lacks experimental evidence specifically validating that CLFD can effectively overcome scalability issues. This echoes the last weakness highlighted by Reviewer KM8C. 

Additionally, I am particularly concerned about the performance degradation observed in the results for the S-CIFAR-10 dataset under the Task-IL setting when CLFD is integrated. Considering that Task-IL is generally regarded as a simpler scenario compared to Class-IL, **the inferior performance in an easier setting raises doubts about the overall efficacy of the CLFD framework**. Moreover, when compared to SparCL [1], SCoMMER [2] and TriRE [3], the CLFD results still show an obvious gap. This discrepancy further accentuates my concerns regarding the effectiveness and robustness of CLFD.

After considering the responses to the weaknesses I outlined and feedback from other reviewers, I maintain my original rating with a score of 4: borderline reject.

[1] Wang, Zifeng, et al. "Sparcl: Sparse continual learning on the edge." NeurIPS. 2022.

[2] Sarfraz, Fahad, Elahe Arani, and Bahram Zonooz. "Sparse coding in a dual memory system for lifelong learning." AAAI. 2023.

[3] Vijayan, Preetha et al. “TriRE: A Multi-Mechanism Learning Paradigm for Continual Knowledge Retention and Promotion.” NeurIPS. 2023.
I thank the authors for answering my comments.

I believe that the authors' responses can solve most of my concerns of this paper, and I would like to re-rate this paper to weak-accept.