Continual Learning in the Frequency Domain
Authors: RuiQi Liu
Boyu Diao
Libo Huang
Zijia An
Zhulin An
Yongjun Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Continual Learning
Catastrophic Forgetting
Experience Replay
Efficient Deep Learning System
Frequency Domain Learning
Primary area: machine_vision
TL;DR: A novel framework for mapping input images into the frequency domain and minimizing interference between frequency domain features to enhance the performance and efficiency of continual learning.

Abstract: Continual learning (CL) is designed to learn new tasks while preserving existing knowledge. Replaying samples from earlier tasks has proven to be an effective method to mitigate the forgetting of previously acquired knowledge. However, the current research on the training efficiency of rehearsal-based methods is insufficient, which limits the practical application of CL systems in resource-limited scenarios. The human visual system (HVS) exhibits varying sensitivities to different frequency components, enabling the efficient elimination of visually redundant information. Inspired by HVS, we propose a novel framework called Continual Learning in the Frequency Domain (CLFD). To our knowledge, this is the first study to utilize frequency domain features to enhance the performance and efficiency of CL training on edge devices. For the input features of the feature extractor, CLFD employs wavelet transform to map the original input image into the frequency domain, thereby effectively reducing the size of input feature maps. Regarding the output features of the feature extractor, CLFD selectively utilizes output features for distinct classes for classification, thereby balancing the reusability and interference of output features based on the frequency domain similarity of the classes across various tasks. Optimizing only the input and output features of the feature extractor allows for seamless integration of CLFD with various rehearsal-based methods. Extensive experiments conducted in both cloud and edge environments demonstrate that CLFD consistently improves the performance of state-of-the-art (SOTA) methods in both precision and training efficiency. Specifically, CLFD can increase the accuracy of the SOTA CL method by up to 6.83% and reduce the training time by 2.6×.

Continual Learning in the Frequency Domain
Ruiqi Liu Boyu Diao! 1.2,, Libo Huang', Zijia An An!:?, Zhulin An! Yongjun Xu 1.2
"Institute of Computing Technology, Chinese Academy Sciences, Beijing, China Uniiersiiyy of Chinese Academy of Sciences, Beijing, China
1iuruiqi23@maills uuca::/ettmmma.uuu..a........... uca.u ac. cn.
{diaoboyu2012, anzijia23p, anzhulin, xyj)@ict. ac. cn,
www.huanglibo@gne. com
Abstract
Continual learning (CL) is designed to learn new tasks while preserving existing knowledge. Replaying samples from earlier tasks has proven to be an effective method to mitigate the forgetting of previously acquired knowledge. However, the current research on the training efficiency of rehearsal- which limits the practical application of CL systems in resource- The human visual system (HVS) exhibits varying sensitivities to different frequency components, enabling the efficient elimination of visually redundant information. Inspired by HVS, we propose a novel framework called Continual Learning in the Frequency Domain (CLFD). To knowledge, this is the first study uti- lize frequency domain features to enhance the performance and efficiency of CL training edge devices. For the input features of the feature extractor, CLFD employs wavelet transform to map the original input image into the frequency domain, thereby effectively reducing the size of input feature maps. Regarding the output features of the feature extractor, CLFD selectively utilizes output features for distinct classes for classification, thereby balancing the reusability and interference of output features based on the frequency domain similarity of the classes across various tasks. Optimizing only the input and output features of the feature extractor allows for seamless integration of CLFD with various rehearsal -based methods. Extensive experiments conducted in both cloud and edge environments demon- strate that CLFD consistently improves the performance of state-of-the-an (SOTA) methods in both precision and training efficiency. Specifically, CLFD can increase the accuracy of the SOTA CL method by to 6.83% and reduce the training time by 2.6x. Code is available hhttps//ggithub... Sttp:/hhwwb..o/EEELLSCCCCCAACAAC
methods insufficient, scenarios.
Introduction
Continual learning (CL) enables machine learning models to adjust to new data while preserving previous knowledge in dynamic environments (25). Traditional training methods often underperform in CL because the adjustments to the parameters prioritize new information over old information, leading to what is commonly known as catastrophic forgetting (36). While recent CL methods addressing the issue of forgetting, it is imperative to also consider learning efficiency when implementing CL applications on edge devices with constrained resources [37 31
primarily concentrate
such the NVIDIA Jetson Orin NX.
To mitigate catastrophic forgetting. wide range of methods have been employed: regularization- based methods [49 42 30 11
constrain updates to essential parameters, minimizing the drift in network parameters that are crucial for addressing previous tasks; architecture-bac methods [14] 24
Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

35 45 52
allocate distinct parameters for each task or incorporate additional network components upon the arrival of new tasks decouple task sees--seeffic knowledge; and rehearsal- based methods (2)
8 12 10) effectively prevent forgetting by maintaining an episodic memory buffer and continuously
replaying samples from previous tasks, Among these methods, Athearsal-basee methods have been proven to be the most effective in mitigating catastrophic forgetting [6]. However, when the buffer size is constrained by memory limitations (e.g., on edge devices), accurately approximating the joint distribution using limited samples becomes challenging. Moreover, rehearsal-based methods often require frequent data retrieval from buffers. This process significantly increases both computational demands and memory usage, consequently limiting the practical application of rehearsal- -ased methods in resource-constraing
environments.
By reducing the size of the input image, both the training FLOPs and peak memory usage can be significantly decreased, thereby enhancing the training efficiency of rehearsal-based methods. Concurrently, this method allows rehearsal-baset methods to store more samples within the same buffer. However, directly downsampling the input image can significantly degrade the model's performance due to information loss. Owing to the natural smoothness of images, the human visual system (HVS) exhibits greater sensitivity to low-frequency components than to high-frequency enabling the efficient elimination of visually redundant information. Inspired by HVS, we transfer the CL methods from the spatial domain to the frequency domain and reduce the size of imput feature maps in the frequency domain. Several studies [48] [5 [3] have focused on accelerating model training in the frequency domain. However, two primary limitations hinder their direct application to CL: (1) These studies utilize Discrete Cosine Transform (DCT) to map images into the frequency domain, resulting in a complete loss of spatial information, which prevents the use of data augmentation techniques in rehearsal- baseed methods. (2) These studies introduce significant number of cross-task learnable parameters, consequently increasing the risk of catastrophic forgetting.
components [88 34
To this end, we propose novel framework called Continual Learning in the Frequency Domain (CLFD), which comprises two components: Frequency Domain Feature Encoder (FFE) and Class-aware Frequency Do- main Feature Selection (CFFS). To reduce the size of input images, we propose the FFE. This method utilizes Discrete Wavelet Transform (DWT) to transform the original RGB im- age input into the frequency domain, thereby preserving both the frequency domain and spatial domain features of the image, which facilitates data augmentation. Furthermore, acknowl- edging that distinct tasks exhibit vary- ing sensitivities to different frequency components, we propose the CFFS method to balance the reusability and interference of frequency domain features. CFFS calculates the frequency domain similarity between inputs across different classes and selects distinct frequency domain features for classification. This method promotes the use of analogous frequency domain features for catego- rizing semantically similar inputs while concurrently striving to diminish the overlap of frequency domain features among inputs with divergent semantics. Our framework avoids introducing any cross-task learnable parameters, thereby reducing the risk of catastrophic forgetting. Simultancously, by optimizing only the input and output features of the feature extractor, it facilitates the seamless integration of CLFD with various rchearsal-based methods. Figure [ (right) demonstrates that CLFD significantly improves both the training efficiency and accuracy of rehearsal- based methods when implemented on the edge device.
ssiiig
Faoture Fatewtkrr
Traking
CFFS
Classiter
Figure Left: Overview of CLFD. CLFD consists of two components: Frequency Domain Feature Encoder (FFE) and Class-aware Frequency Domain Feature Selection (CFFS). Right: On the NVIDIA Jetson Orin NX edge device, CLFDD demonstrates notable enhancement in both accuracy and efficiency compared to ER [4] on the split CIFAR-10 10 dataset.
In summary, our contributions are as follows:
We propose the CLFD, a novel framework designed to improve the efficiency of CL. This framework enhances the training by mapping input features in the frequency domain and compressing these frequency domain features. To the best of our knowledge, this study represents the first attempt to
2

---

utilize frequency domain features to enhance the performance and the efficiency of CL on edge devices.
CLFD stores and replays encoded feature maps instead of original images, thereby enhancing the efficiency of storage resource utilization. Concurrently, CLFD minimizes interference among frequency domain features, significantly boosting the performance of rehearsal-based methods across all benchmark datasets. These improvements can increase accuracy by up to 6.83% compared the SOTA methods.
actual edge device, showcasing its practical feasibility. The results indicate that our framework can achieve up to 2.6x improvement in training speed and 3.0x reduction in peak memory usage.
We evaluate the CLFD framework
2
Related Work
2.1
Continual Learning
The current CL methods can be categorized into three primary types: Regularization-basse meth-
22, 23] limit updates to key parameters to minimize drift in network parameters assign distinct param- eters to each task or add network components for new tasks to decouple task-specific knowledge.
ods 42 30
essential for previous tasks. Architecture-based methods [14 24
35
45
52
fiteersal-based methods (2 8 12 10] mitigate forgetting by maintaining episodic memory buffer and continuously replaying samples from previous tasks to approximate the joint distribution of tasks during training. Among these, our framework focuses on rehearsal-based methods, these methods are acknowledged as the most effective in mitigating catastrophic forgetlnng (6). ER (38) enhances CL by integrating training samples from both the current and previous tasks. Expanding upon this concept, DER+ [6] enhances the learning process by retaining previous model output logits and utilizing consistency loss during the model update. ER-ACE (7) safeguards learned repre- sentations and minimizes drastic adjustments required for adapting to new tasks, thereby mitigating catastrophic forgetting. Moreover, CL-S-R 4 mimics the interaction between rapid and prolonged learning processes by maintaining two supplementary semantic memories.
A limited number of works explore training efficiency in CL 46 16.. Among these methods, SparCL [46) reduces the FLOPs required for model training through the implementation of dynamic weight and gradient masks, along with selective sampling of crucial data. These methods accelerate the training process through pruning and sparse training. Nevertheless, our framework enhances efficiency by reducing the size of the input feature map, which is an orthogonal optimization to pruning.
2.2 Frequency domain learning
Some studies 448 15| 18 21] utilize DCT to map images into the frequency domain and enhance the inference speed of the models. However, these methods are not conducive to enhancing rehearsal- based methods. Previous research (6) indicates that data augmentation can significantly boost the performance of rehearsal-basee methods. Nevertheless, utilizing DCT results in a total loss of spatial information, thereby restricting the application of data augmentation. Other studies [2 33 32 47
17 13
employ DWT to improve the classification performance of models. While wavelet transform effectively preserves the spatial features of images, these methods are not well-suited for CL due to the substantial increase in learnable parameters they introduce. In CL, this proliferation of parameters significantly raises the risk of catastrophic forgetting across tasks. MgSvf (5T) utilizes the frequency domain in the context of CL, focusing on the influence of different frequency components on model performance. In contrast. our framework delves into the differences in redundancy between the spatial and frequency domains. Compared to the spatial domain, CL in the frequency domain can more effectively remove redundant information from images, thereby improving the efficiency of CL..
3 Method
Our method, called Continual Learning in the Frequency Domain, is unified framework that integrates two components: the Frequency Domain Feature Encoder, which transforms the initial RGB image inputs into the wavelet domain, and the Clase-aware Frequency Domain Feature Selection,
3

---

class:
Class-aware Frequency
Domain Feature
Frequency Domain Feature Encoder
Selection
Input
1+1
Reservoir
Rehearsal
Feature
DINT
concal
Extractor
Classifier
CORTY
Masked Feature
Selected Feature
Figure 2: Illustration of the CLFD workflow. Initially, the original RGB image input is transformed into the wavelet domain through Frequency Domain Feature Encoder. Subsequently, the feature extractor extracts the frequency domain features of these input feature maps. We propose Class- aware Frequency Domain Feature Selection to selectively utilize specific frequency domain features, which are then inputted into the classifier for subsequent classification.
which balances the reusability and interference of frequency domain features. The entire framework
is illustrated in Figure 2
3.1 Problem Setting
The problem of CL involves sequentially learning T tasks from a dataset, where in each task corresponds training set D' (I,, y1 21 Each task characterized by a task-specific data distribution represented by the pairs (I,, 14). To improve knowledge retention from previous tasks, we employ a fixed-size memory buffer, MM (I:; M)) which stores data from tasks encountered earlier. Given the inherent limitations in CL, the model's storage capacity for past experiences is finite, thus B < Nt. To address this constraint, we utilize reservoir sampling (44 to efficiently manage the memory buffer. In the simplest testing configuration, we assume that the identity of each upcoming test instance is known, a scenario defined as Task Incremental Learning (Task-II.). the class subset of each sample remains unidentified during CL inference, the situation escalates to more complex Class Incremental Learning (Class-IL.) setting. This research primarily focuses on the more intricate Class-IL setting, while the performance of Task-IL is used solely for comparative analysis.
3.2 Discrete Wavelet Transform
DWT offers effective signal representation in both spatial and frequency domains (28), facilitating the reduction of input feature size. Compared with the DWT, DCT coefficients predominantly capture the global information of image, but they fail to preserve the spatial continuity that is typical in normal images. In DCT, local spatial information is mixed, resulting in a loss of distinct local features. In contrast, DWT effectively integrates both spatial and frequency domain information, maintaining a balance between the two. Furthermore, the DWT method can be seamlessly integrated with data augmentation techniques in cicherrsal-baeed methods, enhancing its applicability and effectiveness.
For 2D signal E N The signal after DWT can be represented as:
x' = I
LXIT LXHT HXLT HXHT
X XM
X
x
IT
HT H
(I)
X
where L and H represent the low-frequency and high-frequency filters of orthogonal wavelets, respectively. These filters are truncated to the size of N x N. The term X₁ refers to the low-frequency component, while X XMs, Xnn represents the high fighffreguency components. We

---

select the Haar wavelet as the basis for the wavelet transform because of its superior computational
efficiency 28
which well-suited for our tasks.
3.3
Frequency Domain Feature Encoder
Previous methods 29
47
28 typically
discarded high-frequency components Nu,,At XRa and retained low-frequency component Xu- However, focusing solely low-frequency component leaves many poten- tially useful frequency components unexplored. Low-frequency component compress the global topological information of an image at various levels, while high-frequency components reveal the image's structure and texture (20). Figure 3: The Utilization of DWT in FFE.
X11
1x1Conv
low-frequency features
Input image x
DWT
Ih
1x1Conv
global features
XM
1x1Conv
high-trequency
X
features
Therefore, we employ three 1x1 1 point convolutions to integrate various frequency components. As shown in Figure we use a xx I point convolution to merge low-frequency component X to obtain low-frequency features, another one to merge high-frequency components Xin, Xnt, X to obtain high-frequency features, and a final one to merge all frequency components to obtain global features. These three merged features compose the input feature maps. By utilizing both low and high-frre frequency components in CL, we can better prevent the loss of critical information while reducing input feature size. Since each merged feature's 's width and height are half of the original image, another advantage of working in the frequency domain is that the spatial size of the original image (H x W x 3) is reduced by half in both width and height (H/2 x W/2 x 3) after FFE. With the reduced spatial size, the computational load and peak memory requirements of CL models decrease. Moreover, the reduction in spatial size means that more replay samples can be stored under the same storage resources, while also reducing the bandwidth required for accessing data. However, setting specific frequency domain feature encoder for each task may result in significant catastrophic forgetting. Therefore, we freeze the FFE at the end of the first task's 's training.
3.4 Class-aware Frequency Domain Feature Selection
Considering that tasks are predominantly sensitive to specific frequency domain features extracted by a feature extractor, different tasks prioritize distinct frequency domain features. To this end, we propose the CFFS, designed to manage the issue of overlap in frequency domain features among samples from different classes. This method promotes comparable classes to utilize similar frequency domain features for classification, while also ensuring that samples from dissimilar classes employ divergent features. Consequently, this method reduces interference among various tasks and mitigates overfitting issues. For specific classes, select predetermined number of frequency domain features based on the absolute values of these features. Subsequently, unselected features are masked to prevent their interference in the classification process. We utilize a counter F E CgCN to track the number of selections for each frequency domain feature among samples associated with a specific class. N and denote the dimensions and the classes of frequency domain features, respectively. We utilize cosine similarity to evaluate the frequency domain similarity between two class samples. To decrease computational complexity, only low-frequency component Xu is utilized for calculating cosine similarity. The similarity between class and class j is expressed as follows:
T.T [E [
S =
(2)
The value of cosine similarity is determined solely by the direction of the features, regardless of their magnitude. Consequently, f, represents the sum of the low-frequency component of samples in class i. We then select the class that exhibits the greatest similarity and the class that displays the least similarity to the current class:
VJ
argmax Sij SE (E
3i = argmin Sij, iE(
where K represents the total number of classes in the preceding task. Several studies (39 43 employ Heterogeneous Dropous (1 to enhance the selection of underutilized features for subsequent tasks. While this method helps manage the overlap of feature selection across different tasks, it overlooks
5

---

Algorithm Class-aware Frequency Domain Feature Selection Algorithm
Input: number of tasks T, training epochs of the (-th task Kr, dropout parameter À and Be.
frequency dropout epochs 3 Initialize: Pf = P. =
for t ....., do
2: for I.....,, K, do
3:
if e < & then if t > then Dropout features based on frequency dropout probabilities 1 Ps
4:
5:
6:
if 1 then Update P₁ at the end first epoch
7:
(Eq.
8:
else
9:
Dropout features based on semantic dropout probabilities 1
10:
Update P. at the end of each epoch (Eq. 5
11:
Select the top 60% of frequency domain features by response values for classification Update F
12:
similarities among classes. This oversight can negatively impact the effectiveness selecting features in the frequency domain. To this end, we propose the Frequency Dropout method, which adjusts the probability of discarding frequency domain features based on class similarity. Specifically, let [Fyli denote the number of the j-th frequency domain feature when learning class y. The probability of selecting this feature in class e while learning new task is expressed as follows:
|F
000 )
[Pyles = Aexp
+(1 - A)
1 exp
max; F
max,
[F vf!
Sc
S Sey S.
0
a
Segs
where 5e denotes the average cosine similarity between class and all classes in previous tasks. The parameters og and ox control the intensity of the selection process. A higher value indicates a greater overlap of activated features with analogous classes and reduced overlap with non-analogous classes. The coefficient À serves as a weighting factor that adjusts the selection of frequency domain features, determining whether the emphasis is towards similar classes less towards dissimilar ones. The Frequency Dropout probability is updated at the beginning of each task. After completing training for epochs on given task, Semantic Dropout (39 employed instead of Frequency Dropout. It encourages the model to use the same set of frequency domain features for classification by setting the retention probability of frequency domain features in each class. This probability is proportional to the number of times that frequency domain feature has been selected in that class so far:
[Fll
max; [Fc). 3. ).).
[Pale.. = exp
where Se controls the strength of dropout. The probability of Semantic Dropout is updated at the end of each epoch, thereby enhancing the model's established frequency domain feature selection. This adjustment effectively regulates the extent of overlap in the utilization of frequency domain features. Algorithm Notlinne the procedure for the CFFS.
Experiment
4.1 Experimental Setup
Datasets. We conduct comprehensive experimental analyses on extensively used public datasets, including Split CIFAR-10 (S-CIFAR-10) (6) and Split Tiny ImageNet (m-Teny- (9. The S-CIFAR-10 dataset is structured into five tasks, each encompassing two classes, while the S-Tiny- ImageNet dataset is divided into ten tasks, each comprising twenty classes. Additionally, the standard input image size for these datasets is 32 32 pixels.

---

Table 1: Comparison different CL methods. CLFD consistently reduces the peak memory footprint of corresponding CL methods while simultaneously improving average accuracy. The highest results marked in bold, and shadowed lines indicate the results from framework.
S-CIFAR-10
S-Tiny-Iman ImageNet
Buffer
Method
Class-IL
Task-IL
Mem
Class-IL
Task-IL
Mem
JOINT
15
98.3
59. 19.99
82.04aa:0
SGD
19.62s005
61.0243.3
7.92
18.3
oEWC 41
19.49
68.294a.2
530MB
7.
19. 19.20aa31
970MB
[50
19.48
68, 05.551
573MB
6.58
36.32tm.1
1013MB
LwF 30
19.61
63.2942.5
316MB
8.46
15.85
736MB
ER [38
29,424351
86.364143
497MB
8. 14
26.80an94
1333MB
DEK++
42.15
8
646MB
8.00
23.53s267
1889MB
50
ER-ACE 40,961600
85.78
502MB
6.68.277
35.93x266
1314MB
CLS-ER
45.91
89.71am 12187
1016MB
11 11.0911.52
40.76
3142MB
CLFD-ER CLFD-DER 51.02x2.76
45,56
84,45
205MB
7.61 20.00
34.67
514MB
81.15x192
241MB
10.69am.21
31.55
658MB
50
CLFD-ER-ACE
52.74.191
87.13
204MB
1
38.05=11.9
514MB
CLFD-CLS-ER
50.13x367
85.30&101
401MB
12.61
37.80s3.09
1032MB
ER [38
38.49 68
89. 89.12Hu0 12H922
497MB
8.30
34.82t682
1333MB
DER+
53,09a3.5
88.3441.08
646MB
11. 11.29
32.92s201
1889MB
125
ER-ACE
12.12
90.49
502MB
11. 11.09x3.8
41.85x3.46
1314MB
CLS-ER 4
53,57a273
90.75x2.6
1016MB
16.35
46.1 46.11
3142MB
CLFD-ER
55.76
88.29
205MB
8.89
42.40
514MB
CLFD-DER++ 18.81
84.7640.66
241MB
15,42an.37
40.94
658MB
125
CLFD-ER-ACE
58.68
89.35x0.34
204MB
15. 15.88
44.71
514MB
CLFD-CLS-ER
59.98s1.8 38
87. 87.0920.43
401MB
18.73s091
49.75x201
1032MB
Evaluation metries. final model:
We use the average accuracy on all tasks evaluate the performance of the
ACC, Σ R
(6)
We denote the classification accuracy the --th task after training the t-th task as R1, Morcover, we evaluate the training time, training FLOPs and peak memory footprint (46) to demonstrate the efficiency of each method. More experimental results can be found in Appendix
Baselines. We compare CLFD with several representative baseline methods, including three regularization-basel methods; oEWC [41), SI (50) and LwFF (30). well four rehearsal-based methods: ER [38]. DER++ (6). ER-ACE (7) and CLS-ER 4 In our evaluation, we incorporate two non-continual learning benchmarks: SGD the lower bound and JOINT as the upper bound.
Implementation Details We the Mammoth CL repository in PyTorch 66) For the S-CIFAR-
10 and S-Tiny-ImageNet datasets, we utilize standard ResNet18 [19] without pretraining as the baseline model, following the method outlined in DER++ (6]. All models are trained using the Stochastic Gradient Descent optimizer with fixed batch size of 32. Additional details regarding other hyperparameters are detailed in Appendix and and E For the
dataset, models undergo training for 100 epochs, whereas for the S-CIFAR-II 10 dataset, training lasts for epochs per task. In rehearsal- -hased methods, each training batch consists of an equal mix of new task samples and samples retrieved from the buffer. To ensure robustness, all experiments are conducted 10 times with different initializations, and the results are averaged across these runs.
4.2 Experimental Result
Table presents a comparative analysis of the results on the S-CIFAR-10 10 and datasets, evaluated under Class-IL and Fask-IL settings. The results elucidate that CLFD significantly
7

---

Table Ablation Study: The Influence of systematically removing different components of CLFD- ER on model performance in S S-CIFAR-10.
Frequency Domain
Class-aware Frequency Domain
Class- IIL
Task-IL
Feature Encoder
Feature Selection
X
X
29.4 29.42
86.36.143
X
39.1920at
88.01
37.80
85.78-241
/
V
45.56
84.45-08s
enhances the performance of various rehearsal-based CL methods. Specifically, the CLFD model has notably achieved SOTA accuracies across all buffer sizes in benchmark evaluations. By reducing the peak memory footprint by 2.4 CLFD augment the average accuracy of the ER method by up to 16.14%. Furthermore, when integrated with the SOTA method CLS-ER, CLFD can also increase its average accuracy by 6.41% % and reduce its peak memory footprint by 2.5 x. The superior performance of CLFD indicates that our proposed framework effectively mitigates catastrophic forgetting by improving the efficiency of storage resource utilization and minimizing interference among various frequency domain features. Moreover, the improvements implemented by CLFD across four different established rehearsal-based methods underscore its adaptability unified framework, highlighting its potential for integration with diverse CL methods.
4.3 Edge Device Results
We evaluate the acceleration performance of the CLFD utilizing the NVIDIA Ampere archi- tecture GPU and Octa-core Arm CPU on the NVIDIA Jetson Orin NX 16GB platform. We measure the training time and accuracy of vari- ous methods using the S-CIFAR-I0 dataset with a buffer size of 125. To expedite the training pro- cess, data augmentation techniques were omit- ted, resulting in accuracy results that vary from those reported in Table I Figure illustrates the training time and average accuracy of var- ious methods. When combined with various CL methods, CLFD significantly reduces train- ing time while simultaneously enhancing model accuracy. By achieving approximately 2.4 x training acceleration, CLFD can attain the high- est average accuracy of 47.64% when integrated with ER-ACE. This suggests that CLFD signif- icantly enhances the efficiency of CL. on edge devices by reducing the input feature map size.
14
Average Acturacy Thaining Time
12
Teaining (hours)
Average Accurscy
35
25
15
%/7 -2044
0/7
Figure 4: Comparison of different methods for S-CIFAR-10 dataset using Nvidia Jetson Orin NX with a buffer size of 125. When combined with various CL methods, CLFD significantly re- duces training time while simultaneously enhanc- ing model accuracy.
4.4 Ablation Study
In Table 2 we present a comprehensive ablation study of CLFD-ER employing a buffer size of 50 on the S-CIFAR-10 dataset. The results indicate that each component our method makes substantial contribution to enhancing accuracy. Comparing row 1 and 2, we can see that FFE enhances the utilization of storage resources by storing encoded features from the frequency domain rather than the original images. This method significantly improves the accuracy of rehearsal based methods by optimizing the representation of stored data. Comparing rows and 3, we can see that CFFS enhances the model's accuracy by mitigating the interference among frequency domain features across different tasks. CLFD achieves superior average accuracy by comprehensively integrating all components, thereby substantiating the efficacy of its individual elements.

---

Table Comparison of CLFD and SparCL on S-CIFAR-10 dataset (Sparsity Ratio: 0.75, Buffer Size: 50).
Table 4: Comparison of different
frequency components.
S-CIFAR-I0
S-CIFAR-10
Method
Method
FLOPs Train
Class-IL(t)
Task-IL(T)
Class-II(()
Task-II((1)
x (1)
Xu
49.59
83,92
ER
38
29.42a3.53
86.36.1.43
11.1
Xis 41.77
79.92
SparCL-ER (46
43.74a2.11
85.01
2.0
XM
44.45
82.76
CLFD-ER
45.56.3.11
84.45
2.8
34.19
74.27
CLFD-SparCI-ER
55.15
88.52.0.29
0.6
FFE
51.02
81.15
600
image
In fo mamanay
globel
high-roguerry
Figure Frequency domain feature counts of the feature extractor trained on S-CIFAR10 with buffer size of 125.
Figure 6: Visualization results of the FFE encoded image.
4.5
Model Analysis
In this section, we provide an in-depth analysis of CLFD.
Sparse Training We compare our method with SparCL 46 a SOTA sparse training method. Table presents the average accuracy and training FLOPs for CLFD and SparCL. Our method significantly enhances the average accuracy compared to SparCL. Although our method incurs higher training FLOPs than SparCL, this does not directly correlate with actual training speed. Our method accelerates training speed without requiring additional optimization. Conversely, pruning and sparse training methods that utilize masks often fail translate into actual training time savings without optimizations at the compiler level. Furthermore, the combination of CLFD and SparCI. not only achieves the highest accuracy but also leads to the lowest training FLOPs. The successful integration of CLFD and SparCL serves as an example of CLFD's adaptability to sparse training and pruning methods.
The impact different frequency components To explore the influence different frequency components, we employ them as input feature maps and assess the CLFD-DER++ accuracy on the S-CIFAR-10 dataset under 50 buffer size. Table 4 presents the average accuracy across various frequency components. Utilizing the low-frequency components of images input feature maps yields the highest accuracy among different frequency components. This suggests that CNN models demonstrate greater sensitivity to low- low-frequency channels compared to high-frequency channels. This finding aligns with the characteristics the HVS, which also prioritizes low- fww-frequency information. Despite this, FFE achieves the highest average accuracy, suggesting that high-frequency components are also significant. Optimal preservation of image information during downsampling is achieved only through the integration of components across different frequencies. Some examples of the encoded frequency domain feature maps are visualized in Figure 6
Frequency Domain Feature Selection evaluate the effectiveness of frequency dropout in reducing interference among frequency domain features across diverse tasks, calculate the

---

selection of specific frequency domain features for different classes Specifically, we track the classification activities on the test set and conduct normalized counts of selections for the frequency domain features, as illustrated in Figure We observe that feature selection patterns exhibit higher correlations among semantically similar classes. For instance, the classes "cat" and "dog" often select identical sets of features. Similarly, significant similarities in feature selection patterns are evident between "auto" and "truck". This result demonstrates the effectiveness of CFFS.
Conclusion
Inspired by the human visual system, we propose CLFD, comprehensive framework designed to enhance the efficiency of CL training and augment the precision of rehearsal- based methods. To effectively reduce the size of feature maps and optimize feature reuse while minimizing interference across various tasks, we propose the Frequency Domain Feature Encoder and the Class-aware Frequency Domain Feature Selection. The FFE employs wavelet transform convert input images into the frequency domain. Meanwhile, the CFFS selectively uses different frequency domain features for classification depending on the frequency domain similarity of classes. Extensive experiments conducted across various benchmark datasets and environments have validated the effectiveness of our method, which enhances the accuracy of the SOTA method by up 6.83%. Moreover, it achieves up to 2.6x increase in training speed and a 3.0x reduction in peak memory usage. We discuss the limitations and broader impacts our method in Appendix A and respectively.
Acknowledgement: This work was supported by the Chinese Academy of Sciences Project for Young Scientists Basic Research (YSBR-107) and the Beijing Natural Science Foundation (4244098).
References
[1] Ali Abbasi et al. "Sparsity and heterogeneous dropout for continual learning in the null space of neural activations". In: Conference on Lifelong Learning Agents. PMLR. 2022, pp. 617-628.
[2] Rahaf Aljundi et al. "Gradient based sample selection for online continual learning". In: Advances in neural information processing systems 32 (2019).
[3] Rahaf Aljundi et al. "Memory aware synapses: Learning what (not) to forget". In: Proceedings of the European conference on computer vision. 2018, pp. 139-154.
[4] Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. "Learning fast, learning slow: A general con- tinual learning method based on complementary learning system". In: International Conference on Learning Representations (2022).
[5] Miles Brundage et al. "The malicious use of artificial intelligence: Forecasting, prevention, and mitigation". In: arXiv preprint arXiv: /002.0000000000 0802002228 (2018).
[6] Pietro Buzzega et al. "Dark experience for general continual learning: a strong, simple haseline".
In: Advances in neural information processing systems 33 (2020), pp. 15920-15930.
[7] Lucas Caccia et al. "New insights on reducing abrupt representation change in online continual learning". In: International Conference on Learning Representations (2022).
[8] Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. "Co2l: Contrastive continual learning". In: Pro- ceedings of the JEEE/CVF International conference on computer vision. 2021, pp. 9516- 9525.
[9] Arslan Chaudhry al. "Continual learning with tiny episodic memories". In: Workshop on Multi- Task and Lifelong Reinforcement Learning. 2019.
[10] Arslan Chaudhry et al. "Efficient lifelong learning with a-gem". In: International Conference on Learning Representations (2019).
[11] Arslan Chaudhry et al. "Riemannian walk for incremental learning: Understanding forgetting and intransigence". In: Proceedings of the European conference on computer vision. 2018, pp. 532-547.
[12] Arslan Chaudhry et al. "Using hindsight to anchor past knowledge in continual learning". In: P Proceedings of the AAAI conference on artificial intelligence. Vol. 35. 8. 2021, pp. 6993-7001.
[13] Tianshui Chen et al. "Learning a wavelet- like auto-encoder to accelerate deep neural networks".
In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. 1. 2018.
10

---

[14] Sayna Ebrahimi et al. "Adversarial continual learning". In: Proceedings of the European conference on computer vision. Springer. 2020, pp. 386-402.
[15] Max Ehrlich and Larry S Davis. "Deep residual learning in the jpeg transform domain". In: Proceedings of the IEEE/CVF international conference on computer vision. 2019, pp. 3484 3493.
[16] Utku Evci et al. "Rigging the lottery: Making all tickets winners". In: International conference on machine learning. PMLR. 2020, pp. 2943-2952.
[17] Shin Fujieda, Kohei Takayama, and Toshiya Hachisuka. "Wavelet convolutional neural net- works for texture classification". In: arXiv preprint w7Xiv:/707.07394 (2017).
[18] Lionel Gueguen al. "Faster neural networks straight from jpeg". In: Advances in Neural Information Processing Systems 31 (2018).
[19]
Kaiming He et al. "Deep residual learning for image recognition", In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 770-778.
[20] Huaibo Huang et al. "Wavelet-srnee A wavelet-based enn for multi-scale face super resolution". In: Proceedings of the IEEE international conference on computer vision. 2017, pp. 1689 1697.
[21]
Jiaxing Huang et al. "Fsdr: Frequency space domain randomization for domain generalization". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, pp. 6891-6902.
[22] Libo Huang et al. "e'Tag: Class-Incremental Learning via Embedding Distillation and Task- Oriented Generation". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. 11. 2024, pp. 12591-12599.
[23] Libo Huang et al. "KFC: Knowledge Reconstruction and Feedback Consolidation Enable Efficient and Effective Continual Generative Learning.. In: The Second Tiny Papers Track at ICLR 2024. 2024.
[24] Zixuan Ke, Bing Liu, and Xingchang Huang. "Continual learning of a mixed sequence of similar and dissimilar tasks". In: Advances in neural information processing systems 33 (2020), pp. 18493-18504.
[25] James Kirkpatrick et al. "Overcoming catastrophic forgetting in neural networks". In: Proceed- ings of the national academy of sciences 114.13 (2017), pp. 3521-3522
[26] Alex Krizhevsky, Geoffrey Hinton, al. "Learning multiple layers of features from tiny images". In: (2009).
[27] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. "Snip: Single-shot network prun- ing based on connection sensitivity". In: International Conference on Learning Representations (2019).
(28) A Levinskis. "Convolutional neural network feature reduction using wavelet transform". In:
Elektronika ir Elektrotechnika 19.3 (2013), pp. 61-64.
[29] Qiufu Li et al. "Wavelet integrated CNNs for noise-robust image classification". In: Pro- ceedings of the IEEE/CVF conference on computer vision and puttern recognition. 2020, pp. 7245-7254.
[30] Zhizhong Li and Derek Hoiem. "Learning without forgetting". In: IEEE transactions on pattern analysis and machine intelligence 40.12 12 (2017), pp. 2935-2947.
[31] Hangda Liu et al. "A resource-awand workload scheduling method for unbalanced GEMMs on GPUs". In: The Computer Journal (2024), bxael 100.
[32] Lin Liu et al. "Wavelet-based dual-branch network for image demoiréing". In: Proceedings of the European conference computer vision. Springer. 2020, pp. 86-102.
[33] Pengju Liu et al. "Multi-level Wavelet-CNN for image restoration". In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops 2018, pp. 773-782.
[34] Zhenhua Liu et al. crequeency-domale dynamic pruning for convolutional neural networks".
In: Advances in neural information processing systems 31 (2018).
[35] Noel Loo, Siddharth Swaroop, and Richard E Turner. "Generalized variational continual learning". In: International Conference on Learning Representations (2021).
[36] Michael McCloskey and Neal J Cohen. "Catastrophic interference in connectionist networks:
The sequential learning problem".. In: Psychology of learning and motivation. Vol. 24. Elsevier,
1989, pp. 109-165.

---

[37] Lorenzo Pellegrini et al. "Continual learning at the edge: Real-time training on smartphone devices". In: arXiv preprint arXiv:210S.13127 (2021).
[38]
Anthony Robins. "Catastrophic forgetting, rehearsal and pseudorchearsal" Connection Science 7.2 (1995), pp- 123-146.
[39] Fahad Sarfraz, Flahe Arani, and Bahram Zonooz. "Sparse coding in dual memory system for lifelong learning". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. 8. 2023, pp. 9714-9722.
[40] Fahad Sarfraz, Elahe Arani, and Bahram Zonooz. "Synergy between synaptic consolidation and experience replay for general continual learning". In: Conference on Lifelong Learning Agents. PMLR. 2022, pp. 920-936.
[41] Jonathan Schwarz et al. "Progress & compress: A scalable framework for continual learning".
In: International conference on machine learning. PMLR. 2018, pp. 4528-4537.
[42] Joan Serra al. "Overcoming catastrophic forgetting with hard attention to the task". In: International conference on machine learning. PMLR. 2018, pp. 4548 4557.
[43] Preetha Vijayan al. "TriRE: A Multi Meetti-Mechansmm Learning Paradigm for Continual Knowl- edge Retention and Promotion". . In: Advances in Neural Information Processing Systems 36 (2024).
[44] Jeffrey S Vitter. "Random sampling with reservoir", In: ACM Transactions on Mathematical Software (TOMS) 11.1 (1985), pp. 37-57.
[45] Zifeng Wang et al. "leesreprune--haree for lifelong learning". In: 2020 IEEE International Conference on Data Mining (ICDM). IEEE, 2020, pp. 641-650.
[46] Zifeng Wang et al. "Sparcl: Sparse continual learning on the edge". In: Advances in Neural Information Processing Systems 35 (2022), pp. 20366-20380.
[47] Travis Williams and Robert Li. "Wavelet pooling for convolutional neural networks". In: International conference on learning representations. 2018.
[48] Kai Xu et al. "Learning in the frequency domain". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020, pp. 1740-1749.
[49] Lu Yu et al. "Semantic drift compensation for class dass-incremental learning". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020, pp. 6982-6991.
[50] Friedemann Zenke, Ben Poole, and Surya Ganguli. "Continual learning through synaptic intelligence". In: International conference on machine learning. PMLR. 2017, pp. 3987-3995.
[51] Hanbin Zhao et al. "Mgsvf: Multi buuti--raiined slow versus fast framework for few-shot class- incremental learning". In: IEEE Transactions on Pattern Analysis and Machine Intelligence 46.3 (2021), pp. 1576-1588.
[52] Tingting Zhao et "Deep bayesian unsupervised lifelong learning". In: Neural Networks 149
(2022), pp. 95-106.
12

---

A Limitations
One limitation of our framework is its focus on optimizing rchearsal-based methods. While the use of a rehearsal buffer throughout the CL process is widely accepted, there exist scenarios where rehearsal buffers prohibited. Nonetheless, by reducing the size of the input feature map, our framework has the potential to accelerate various CL methods, though the implications for model performance require further investigation. Furthermore, our analysis was limited scenarios with finite rehearsal buffer sizes, whereas many current studies focus on scenarios with infinite rehearsal buffer sizes. This shift is primarily due memory constraints being less significant compared computational costs. We will also continue to investigate the performance of our method in scenarios involving infinite rehearsal buffer sizes.
B Broader Impacts
Inspired by HVS, we propose novel framework called CLFD, which utilizes frequency domain features enhance the performance and efficiency of CL. training. Its success has opened up opportunities for enhancing existing CL. methods in the frequency domain. CLFD contributes to the responsible and ethical deployment of artificial intelligence technologies by improving CL performance and efficiency. This is accomplished through the efficient ability of models to update and refine their knowledge without the need for extensive retraining. This further facilitates the real-world application of CL.
Although CLFD serves as a comprehensive framework aimed improving the efficiency of various CL methods, we must remain cognizant of its potential negative societal impacts. While CLFD improves model stability, it does by compromising the expense of model plasticity. resulting in reduced accuracy when applied to new tasks. This trade-off requires specific consideration in applications where accuracy is crucial, such as healthcare [46). Furthermore, as a powerful tool for enhancing the efficiency of CL. methods, CLFD could also strengthen models designed for malicious applications (5). Therefore, it is recommended that the community devise additional regulations to mitigate the malicious use of artificial intelligence.
C Dataset Licensing Information
CIFAR-10 266
licensed under the MIT license.
The licensing information for Tiny-ImageNet is unavailable. However, the dataset is accessi- ble to researchers for non-commercial purposes.
D Hyperparameter Selection
Table A1 presents the selected optimal hyperparameter combinations for each method in the main paper. The hyperparameters include the learning rate (Ir), batch size (bs), and minibatch size (mbs) for rehearsal-based methods. Other symbols correspond specifie methods. It should be noted that the batch size and minibatch size are held constant at 32 for all CL benchmarks.
E Experiment Details
E.1 Experiment Platform
We conduct comprehensive experiments utilizing the NVIDIA GTX 2080Ti GPU paired with the Intel Xeon Gold 5217 CPU, well as the NVIDIA Jetson Orin NX 16GB, boasting NVIDIA Ampere architecture GPU and Octa-core Arm CPU.
F.2 Implementation Details
We set À 0.5, B. 2 in equation (4) and equation We also set E at value of 0..4 of the training epochs. CFFS, we only select 60% of the frequency domain features for classification. We utilize the code from DER++ [6]. We extend our gratitude to the authors for their support and for providing
13

---

Table Al: Hyperparameters selected for our experiments.
Method
Buffer
Split Tiny ImageNet
Buffer
Split CIFAR-10
SGD
Ir: 0.03
Ir: 0.1
oEWC
Ir: 0.03
A: 90
7: 1.0
Ir: 0.03
A: 10
7: 1.0
SI
Ir: 0.03
1.0
{: 0.9
Ir: 0.03
c: 0.5
{: 1.0
LwF
Ir: 0.01
a: 2.2.0
Ir: 0.03
ar 0.5
T.2.0
ER
50
Ir: 0.1
epoch: 100
50
Ir: 0.1
epoch: 50
125
Ir: 0.03
epoch: 100
125
Ir: 0.1
epoch: 50
CLFD-ER
50
Ir: 0.1
epoch: 100
50
Ir: 0.1
epoch: 50
125
Ir: 0.03
epoch: 100
125
Ir: 0.1
epoch: 50
+ER+
50
Ir: 0.03
a:
8: 1.0
50
Ir: 0.03
a: 0.1
|:: 0.5
epoch: 100
epoch: 50
125
Ir: 0.03 a: 0.2
3: 0.5
125
Ir: 0.03
0.2
[: 0.5
epoch: 100
epoch: 50
CLFD-DER+
50
Ir: 0.03 o: 0.1
3: 0.5
50
0.03 ar 0.1
[3r 0.5
epoch: 100
epoch: 50
125
Ir: 0.03 epoch: 100
o: 0.1
3: 0.5
125
0.03
ar 0.2
|: 0.5
epoch: 50
ER-ACE
50
Ir: 0.03
epoch: 50
50
In: 0.03
epoch: 50
125
Ir: 0.03
epoch: 50
125
Ir: 0.03
epoch: 50
CLED-ER-ACE
50
Ir: 0.03
epoch: 50
50
Ir: 0.03
epoch: 50
125
Ir: 0.03
epoch: 50
125
Ir: 0.03
epoch: 50
CLS-ER
50
Ir: 0.1
rs: 0.04
rp: 0.08
50
Ir: 0.03
rs: 0.05
Tp: 0.2
as: 0.999
ap: 0.999
as: 0.999
ap: 0.999
A: 0.1
epoch: 50
A: 0.15
epoch: 50
125
Ir: 0.1
rs: 0.05
rp: 0.08
125
Ir: 0.03
rg: 0.1
rp: 0.9
ag: 0.999
ap: 0.999
as: 0.999
ap: 0.999
A: 0.1
epoch: 50
A: 0.15
epoch: 50
CLFD-CLS-ER
50
Ir: 0.1
rs: 0.08
rp: 0.16
50
Ir: 0.03
rs: 0.1 rpt 0.5
osg 0.999
op: 0.999
as: 0.999
0.999
X: 0.15
epoch: 50
A: 0.15
epoch: 50
125
Ir: 0.1
rs: 0.1 rp: 0.16
125
Ir: 0.03
rs: 0.2 rp: 0.9
as: 0.999
ap: 0.999
aş; 0.999
ap: 0.999
A: 0.15
epoch: 50
A: 0.15
epoch: 50
the research community with the Mammoth framework, which facilitates a fair comparison of various CL methods under standardized experimental conditions. ensure a fair comparison, we endeavor to closely align the experimental settings with those used in the Mammoth framework. However, we modified the data augmentation techniques within the Mammoth framework. The details of our data augmentation techniques are presented as follows.
E.3 Data Augmentation
In line with (6), we employ random crops and horizontal flips as data augmentation techniques for both examples from the current task and the replay buffer. To ensure uniformity in data augmentation between the original images and the input features encoded with the FFE, random cropping is restricted pixels only.
F
Additional Experiment Results
F11
Stability Trade-off
the CL. model can retain previously learned information, is considered stable; if can effectively acquire new information, it is considered plastic. To better understand how various methods balance
14

---

O6
0.5
03:40
20.25
9.20
0.00
5.00
90.00
6.00
8.00
0.0
$7.33
606
000
600
000
95.55
600
0.4
57:40
8.09
000
50.75
10.05
0C0
0.00
099
Aft Bale4
28.85
1.00
22.90
+4.00
600
21.25
20.46
$7.70
12.22
644
06.45
10.25
42.65
606
$0.82
180
28.00
85
Bal
B44
D6l1
946
CUPDEER
CLFD-DEM1
CLPD-EB-ACE
CLPO-4L--E1
50.00
640
646
6
15.43
640
0:00
644
65.15
$4.00
0.00
8.0
666
92
16.45
646
0000
Afr Takk
$2.35
27.00
30.30
$4.00
000
50.30
36.65
45.70
22.23
6.0
60.15
34:05
96.6
0.0
12.99
46.08
53.00
79
10
1al
Bal
B10
Tal:
310
Bal-3
Ta0
B16
Figure 7: Performance of different methods by task. The heatmaps display the test set results for each task (x-axis) evaluated the end of each sequential learning task (y-axis). We conducted experiments on the S-CIFAR-10 dataset using a buffer size of 50.
100
100
CLFD-ER CLFD-DER++ CLFD-CLS-ER
DER++
80
80
CLS-ER
60
60
40
20
Stability
Plasticity
Tradeoff
0
Stability
Plasticity
Tradeoff
Figure 8: Stability-laastiityy Trade-off for CL models trained on S-CIFAR 10 under 50 buffer size.
stability and plasticity, we investigate the evolution of task performance as the model learns tasks sequentially. Figure 7 shows that CLFD consistently enhances the balance of all CL. methods, delivering more uniform performance across all tasks. addition, to further demonstrate the effectiveness of our method, we introduce trade-off measure [40] that approximates how the model balances its stability and plasticity. Upon the model's 's completion the final task T, its stability is evaluated by calculating the average performance across all preceding T 1 tasks follows:
- RT.
S
T-1
The plasticity of the model (P) is evaluated computing the average performance of each task after its initial learning i.c. the diagonal of the heatmap:
R
(8)
T
Thus, the trade-off measure determines the optimal balance between the model's stability (S) and plasticity (P). This measure is calculated the harmonic mean of S and P.
2SP
Trade-off S
Figure g provides the stability-plasticnn trade-off measure for different CL. methods. ER and DER++ exhibit high plasticity, enabling them to rapidly adapt to new information. However, they lack the
15

---

Table A2: Forgetting results on S-CIFAR-I0 dataset.
S-CIFAR-10
Method
Class-IL(
(ask-II(1)
Buffer Size
50
125
50
125
38
83.61
22.24
12.55+2.1
9.13-1.5
+
60.67
48.80a721
15.83+441
10.12
ER-ACE
32.81 124330
26.42-18.77
13.32r4.46
7.54-0s8
CLS-ER
43.96.9492
48.23
6.07+2.6
6.52=064
CLFD-ER
45.01
33,9 166.5
5.2021.32
2.74
CLFD-DER++
41.93
32.8221.06
12.62x3.98
9.702094
CLAD-ER-ACE
24.98-634
21.13-1.45
4.92+2.19
3.04-0.5
CLFD-CLS-ER
18.19.2.19
22.20
4.22:3.09
3.82.0.54
50
ER (4.61%)
ERAACE (6. 86%)
CLFD-ER BEDDER (6.66%)
CLFD ER-ACE (14.06%)
(6.95%)
(12.01%)
Class- Accuracy (%6)
40
CLFD DERPP (9.05%)
CLFD CLS ER (15. (15.39%)
30
20
0
4
8
10
Tasks
Figure 9: Comparison Class-IL. accuracy for different methods on the Split ImageNet-R dataset, divided into 10 tasks. The figure reports the accuracy of individual tasks at the end of CL training. The values in parentheses in the legend indicate the average accuracy.
ability to effectively retain previously acquired knowledge, leading to task recency bias issues. CLFD consistently improves the stability of CL. methods, thereby reducing task reception bias and enhancing the balance between stability and plasticity.
F.2 Forgetting
We the Final Forgetting (FF) measure the model's anti-forgettig performance:
FF. = Σ max (Ri.j (E(1.....-1)
Re,)),
(10)
A smaller FF value indicates that the model exhibits less forgetting of previous knowledge, thereby demonstrating stronger anti-forgetting performance. preseets the FF result. Our method consistently reduces forgetting in CL methods, demonstrating the effectiveness of CLFD in preserving prior knowledge rather than solely focusing on improving the accuracy of subsequent tasks.
F.3
Split ImageNet-R Result
To further enhance the credibility of our results, we conduct additional experiments on ImageNet-R dataset. ImageNet-R, an extension of the ImageNet dataset, comprises 200 classes and total of 30,000 images, with 20% designated for the test set. We divide ImageNet-R into ten tasks, each containing 20 classes. Input images are resized to 224 224 pixels. We test the Class-IL. accuracy of each task with buffer size of 500, maintaining consistent hyperparameters across all methods as those used in Split Tiny ImageNet. Figure that our method still significantly enhances the performance of various rehearsal-bace CL methods,
D presents the experimental results, demonstrating
16

---

Table A3: Task- IL and Class-IL accuracy under different feature selection proportions.
Feature Selection Proportions
CLFD-ER
CLFD-ER-ACE
Task-IL: IL: 85.67
Task- IL: 85.06
10%
Class-IL: 51.03
Class-IL: 50.37
Task-IL: 84.91
Task IL: 86.74
30%
Class-IL: 48.91
Class-IL 50.64
Task-IL: 83.88
Task-IL: 87.05
50%
Class- Il.: 45.69
Class-II. 52.12
Task IL: 87.97
Task- Mask-IL: 89.83
90%
Class-II:: 39.58
Class-II. 54.20
even on more complex datasets. It is worth noting that the improvement in accuracy is not the sole advantage of our framework. By integrating our framework with rehearsal- -based methods on the Split ImageNet-R dataset, training speed increased by up to 1.7x, and peak memory usage decreased by up x.5x. This demonstrates that our framework can significantly enhance the training efficiency of CL, thereby promoting its application on edge devices.
F44 Feature Selection Proportions in CFFS
We conduct additional ablation experiments to investigate the impact of feature selection proportions on model performance. We focus on two methods: CLFD-ER and CLFD-ER-ACE, as Figure shows that ER is the most plastic method, while ER-ACE the most stable. We conduct tests on the S-CIFAR-I1 10 dataset with buffer size of 50. Table A3 presents the result. In the Task-IL setting, both CLFD-ER and CLFD-ER-ACE achieve higher accuracy with higher feature selection proportions, as the inclusion of more features enables the model to better distinguish between classes important highlight that in CLFD-ER, lower feature selection proportions
within the tasks. also lead to positive outcomes. This phenomenon can be attributed to the high plasticity of the CLFD-ER method, where reducing feature selection proportions aids in mitigating potential interference among various tasks. In the Class-IL setting, CLFD-ER achieves higher accuracy with lower feature selection proportions, while CLFD-ER-ACE exhibits superior performance with higher feature selection proportions. This observation suggests that for methods with high plasticity, need to decrease the feature selection proportions to mitigate the overlap of frequency domain features, thereby minimizing the impact of new tasks old tasks. Conversely, for methods with high stability, increasing the feature selection proportions allows us to utilize more frequency domain features to learn the current classes effectively.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims align with the experimental results. Based on the various experi- ments detailed in our paper, is expected that our method can be generalized.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
Our paper is based on experimental results, without any inclusion of theoretical
results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Section 4.1 and Appendix
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
We provide code to reproduce the primary experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee //nips.com htte:/ublll/lliiiee/e//eeeuuaasl ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See Section 4.1 Appendix Daand Appendix In addition, we provide code to reproduce the primary experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Table T Table
Table
Table
A2 and Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics standards in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We mentioned and cited the datasets (S-CIFAR-10 and S-Tiny-ImageNet), as well all comparative methods with their respective papers. The licenses for the datasets and models used are provided in the cited references and explicitly stated in Appendix C
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
22

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the code for proposed method in the supplement.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper is not about research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "7rPyKS1Yd6", "reviewer": "Reviewer_wmJG", "summary": "This paper introduces a novel framework designed to enhance the efficiency and effectiveness of continual learning (CL) systems by leveraging frequency domain representations, inspired by the human visual system's varying sensitivity to different frequency components. This approach aims to address the limitations of existing rehearsal-based methods in CL, particularly under constraints like limited resources on edge devices. The framework, named Continual Learning in the Frequency Domain (CLFD), uses a wavelet transform to convert input images into frequency domain representations, optimizing both the input and output features of the feature extractor. This allows for better management of memory usage and computational demands, leading to improvements in both accuracy and training efficiency. Extensive experiments demonstrate that CLFD can enhance the performance of state-of-the-art rehearsal-based methods, achieving higher accuracy and reduced training times.", "questions": "Please refer to the weaknesses.", "limitations": "The limitations have been discussed in Appendix A.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe proposed method is inspired by the human visual system, which is sensitive to different frequency components and efficiently reduces visually redundant information. This represents a novel shift from traditional spatial domain methods to frequency domain methods in CL, suggesting a significant departure from established methods.\n2.\tCLFD significantly enhances the performance and training efficiency of continual learning systems on edge devices, achieving up to 6.83% higher accuracy and reducing training time by 2.6 times.\n3.\tThe paper is structured to clearly present the problem of catastrophic forgetting in CL and how the proposed method addresses it by reducing the input feature map size and optimizing feature reuse. The methodology is described in detail, providing clarity on how the approach works and is implemented. \n4.\tBy reducing training time and memory usage while improving accuracy, the framework can integrate with existing rehearsal-based methods without extensive modification, which underscores its practical significance and potential impact on the field.", "weaknesses": "1.\tWhile the framework shows promising results in specific settings and datasets (like Split CIFAR-10 and Split Tiny ImageNet), the paper does not thoroughly discuss its performance across a wider range of scenarios or more complex datasets. \n2.\tThe effectiveness of the proposed method is somewhat dependent on the buffer size used for rehearsal in continual learning. The reliance on buffer size might limit its utility in extremely constrained environments where memory is severely limited.\n3.\tFreezing the FFE based on the first task might introduce scalability issues, as the encoder might not efficiently handle the complexity introduced by a broader set of tasks or more diverse data.\n4.\tIn Table 1, it is observed that under the Task Incremental Learning (Task-IL) setting, the integration of the CLFD framework leads to worsened performance in some experimental results. This highlights a potential weakness, as the paper does not sufficiently explain why the framework underperforms in these specific scenarios.\n5.\tThe paper exhibits several technical and typographical issues that affects its formal presentation and readability. For example, the multiplication symbol in section 3.3 is represented as “x” instead of using a standard “$\\times$”. In Eq. 2, symbols are not bolded. Figure 4’s caption lacks a period at the end. Other typos like writing “FFE” as “FFD” in the conclusion section, introduces confusion and can be misleading about key terms and components described in the paper. In addition, there are errors in the formulation of some equations. For instance, Eq. 7’s subscript should start from 1 instead of 0. Similarly, Eq. 8 is also incorrectly formulated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude to the reviewer for offering thoughtful feedback and providing a constructive evaluation of our work. The valuable input has significantly contributed to the improvement of our paper.\n\n# The paper does not thoroughly discuss its performance across a broader range of scenarios or more complex datasets.\n\nWe agree with the reviewer that conducting experiments on a broader dataset is essential. Accordingly, we have supplemented our experiments with the Split Imagenet-R dataset and provided a detailed discussion in the general response.\n\n# The reliance on buffer size might limit its utility in extremely constrained environments where memory is severely limited.\n\nWhile the dependence on buffers does consume some storage resources, our method can achieve excellent performance even with smaller buffer sizes by preserving the frequency domain features of the image instead of the original image, as shown in Table 1. In contrast, many continual learning methods [1,2] that do not rely on buffers need extra parameters, such as teacher models, to prevent model forgetting, which use significantly more storage than rehearsal-based methods.\n\nMore importantly, for continual learning on resource-constrained devices, the limitation of memory resources is much more critical than that of storage resources. This necessitates continual learning methods to have lower peak memory usage. As shown in Table 1, our method can achieve up to a 3.0 $\\times$ reduction in peak memory usage. This effectively promotes the application of continual learning methods on resource-constrained devices.\n\n[1] Smith, James, et al. \"Always be dreaming: A new approach for data-free class-incremental learning.\" *ICCV*. 2021.\n\n[2] Li, Zhizhong, and Derek Hoiem. \"Learning without forgetting.\" *IEEE transactions on pattern analysis and machine intelligence* 40.12 (2017): 2935-2947.\n\n# Freezing the FFE based on the first task might introduce scalability issues.\n\nWe agree with the reviewer that freezing the FFE does affect the plasticity of the model. However, not freezing the FFE leads to severe forgetting problems. To balance the plasticity and stability of the model, it is crucial to ensure that no cross-task learnable parameters are introduced. As shown in Figure 8 in the appendix, our method consistently improves the balance between plasticity and stability in continual learning methods.\n\n# The framework underperforms in the Task-IL setting.\n\nWe appreciate the reviewers for identifying the lack of the Task-IL accuracy analysis. This result can be explained from two aspects: (1) FFE encodes the frequency domain features of the input image, reducing the size of the input feature map to one-fourth of its original size. This leads to fewer learnable features and increases the difficulty of classifying classes within the task. (2) To minimize the overlap of frequency domain features between inputs with different semantics across tasks, CFFS selects only a subset of frequency domain features for classification. This selection also increases the difficulty of classifying classes within tasks. However, as the task difficulty increases, the performance of our framework becomes more pronounced in the Task-IL setting. For instance, as shown in Table 1 in the attached PDF, our framework significantly enhances Task-IL accuracy on the Split Tiny ImageNet dataset with buffer sizes of 200 and 500.\n\nConsidering that Task-IL is generally regarded as an easier CL scenario compared to Class-IL, this research, consistent with previous work [1], primarily focuses on the more intricate Class-IL setting (as mentioned in line 150). Furthermore, we emphasize that the novelty of this work lies not only in improving accuracy but also in significantly enhancing the training efficiency of continual learning. Additionally, we have validated our framework on edge devices, thereby promoting the application of continual learning in such environments.\n\n[1] Wang, Zifeng, et al. \"Sparcl: Sparse continual learning on the edge.\" NeurIPS. 2022.\n\n# The technical and typographical issues.\n\nThanks for pointing out these technical and typographical issues, we will revise them in the next release.\n\nWe once again thank the reviewer for providing detailed and insightful feedback. Please let us know if there are any open points that we may have overlooked.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PLCpvZMvoh", "reviewer": "Reviewer_2hDB", "summary": "In this paper, the authors proposed a novel replay-based continual learning, which is named continual learning in the frequency domain (CLFD). The framework consists of two main modules, frequency domain feature encoder (FFE) and class-aware frequency domain feature selection (CFFS). FFE utilizes discrete wavelet transform (DWT) to transform the RGB images into a frequency domain. CFFS computes similarity and selects suitable frequency domain features for classification.", "questions": "How is the historical data stored? What format is the data in a memory buffer? How do you use it for replay?\n\nCould you please give some more details about CFFS? The overall idea can be understood, but the details are unclear because of confusing notations.", "limitations": "As the authors have discussed, replay-based methods are unsuitable for all scenarios.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The strengths of this paper are listed below:\n- The paper proposed a novel method that utilizes the frequency domain to decode the information of inputs.\n- It can reduce the storage requirement to store a sample, which leads to less memory or more stored samples.\n- They ran many experiments to show the performance.", "weaknesses": "The weaknesses of this paper are listed below:\n- Some parts are not presented clearly. More details may included. Including a pseudocode may help.\n- Some notations are not explained clearly, e.g. sec. 3.4.\n- Even though there are many experiments, more ablation studies about hyperparameters may provide more information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for dedicating time to thoroughly review our work. We value the positive feedback provided on our manuscript. Below, we address the weaknesses and queries raised:\n\n# More ablation studies about hyperparameters.\n\nWe appreciate the feedback from the reviewers and acknowledge the importance of analyzing the robustness of hyperparameter selection. However, our framework can integrate with various continual learning methods, ablation studies on hyperparameters require substantial computational resources and extensive experimentation. Given this, we have demonstrated the effectiveness of our framework by using default hyperparameters for all datasets, methods, and buffer sizes.\n\n# The ways to replay data.\n\nWe use the widely accepted continual learning repository Mammoth CL [1] for data replay. We adopt a reservoir sampling strategy [2] to save samples, storing all data in tensor format. During each batch’s training, we read an additional batch of samples from the memory buffer and train them together with the current batch.\n\n[1] Buzzega, Pietro, et al. \"Dark experience for general continual learning: a strong, simple baseline.\" NeurIPS. 2020.\n\n[2] Vitter, Jeffrey S. \"Random sampling with a reservoir.\" *ACM Transactions on Mathematical Software (TOMS)* 11.1 (1985): 37-57.\n\n# More details about CFFS.\n\nWe apologize for any confusion. In the final version of the paper, we will optimize the details of CFFS. For a given task, we select only 60% of frequency domain features for classification. Each frequency domain feature is monitored for its selection frequency during training. Essentially, for each class, each frequency domain feature is assigned a selection counter $\\mathcal{F}$ that increments when the feature's value is among the top 60%. Before selecting frequency domain features, each feature is assigned a dropout probability. During the initial epoch of each task training, we apply frequency dropout. In the later epochs of training, we use semantic dropout and update its probability after each epoch (Eq. 5). After completing the first task training, for subsequent tasks, we calculate the similarity of the classes (Eq. 2) and update the frequency dropout probability (Eq. 4) at the end of the first epoch. The pseudocode is provided in the attached PDF.\n\nWe hope that the clarification provided has resolved your concerns and inquiries. Should you require further assistance or elaboration, we are willing to provide additional information to ensure a comprehensive understanding of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9NlVzNapBM", "reviewer": "Reviewer_wRV3", "summary": "Based on the research that human visual system (HVS) exhibits varying sensitivities to different frequency components, this paper proposes to do continual learning in the wavelet frequency domain to reduce the size of inputs. The proposed CLFD module includes feature extractor and feature encoder, where the feature encoder generate low-frequency features, global features and high-frequency features, and the feature extractor selects class-specific frequency features. The generated features are used to do continual learning.", "questions": "1. How about the time-consuming of the processes of wavelet transform and the forward and backward of CLFD module?\n2. Please clearly explain table-1 for better understanding.", "limitations": "1. The authors may consider to do some discuss about the different frequency domains such as fourier domain and discrete cosine domains.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper introduces the wavelet frequency domain features into CL. By encoding the low-frequency features and high-frequency features respectively, the proposed CLFD may have potential to mimic the human visual system for better learning results.\n2. The feature extractor considers the class information of the frequency features to help the process of CL.", "weaknesses": "1. It is better to include larger databases such as imagenet-1k into the experiments to make the results more convincing.\n2. Table-1 leaves some unclear items. For instance, the meaning of class-IL (class incremental learning) and task-IL (task incremental learning) should be explained. The meaning of shadowed lines and bold texts in the table should be included. \n3. The classification performance of the proposed method seems have a large gap with the state-of-the-art methods on both cifar-10 and tiny-imagenet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere gratitude for your thorough review of our paper. Below, we have diligently addressed each of your concerns to the best of our understanding, aiming to enhance the paper's overall contribution.\n\n# It is better to include larger databases.\n\nWe agree with the reviewer that introducing larger databases can make the results more convincing. Therefore, we introduced the Split Imagenet-R dataset as an alternative and provided a detailed discussion in the general response.\n\n# Table-1 leaves some unclear items.\n\nWe appreciate the reviewer's feedback regarding the shortcomings in Table 1. The highest results are marked in bold, and shadowed lines indicate the results from our framework. We introduced the definitions of Class-IL and Task-IL on line 147. In the simplest testing configuration, we assume that the identity of each upcoming test instance is known, a scenario defined as Task Incremental Learning (Task-IL). If the class subset of each sample remains unidentified during CL inference, the situation escalates to a more complex Class Incremental Learning (Class-IL) setting. We will update Figure 1 in the final version and provide definitions for Class-IL and Task-IL in the experiment.\n\n# The classification performance of the proposed method seems have a large gap with the state-of-the-art methods.\n\nThe reason for the low classification performance is that we only use the polar buffer size. When we adjust the buffer size to be consistent with state-of-the-art methods, the classification performance significantly improves, and our method continues to enhance the performance of various rehearsal-based methods. We provide a detailed analysis in the general response.\n\n# The time-consuming of the processes of wavelet transform and CLFD module.\n\nWe appreciate the reviewers for identifying the lack of training time analysis. We calculate the training time for the wavelet transform, CLFD module, ER method and CLFD-ER method throughout the entire Split CIFAR-10 training process. The wavelet transform accounts for only 1.5% of the ER training time, while the CLFD module accounts for only 13.1% of the ER training time. It is worth noting that when we integrated the CLFD module with the ER method, the training time was reduced by 2.3$\\times$. This demonstrates the efficiency of our framework. \n\n| Methods | Training Time (s) |\n|---------|---------------------------------|\n| wavelet transform  |   127      |\n| CLFD     | 1125        |\n| ER     | 8542        |\n| CLFD-ER     | 3732        |\n\n# Some discuss about the different frequency domains.\n\nWe discussed different frequency domains in sections 2.2 and 3.2. The discrete cosine transform and discrete Fourier transform both cause a complete loss of spatial information in images, making it difficult to utilize data augmentation techniques. Therefore, they are challenging to apply in continual learning.\n\nOnce again, we express our gratitude for your thoughtful evaluation and consideration. We have conscientiously endeavored to address each of the concerns you raised, and we are dedicated to ensuring that our paper makes a meaningful contribution to the conference proceedings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A9ZfObewra", "reviewer": "Reviewer_KM8C", "summary": "Inspired by the human visual system (HVS), this paper proposes a new framework called Continual Learning in the Frequency Domain (CLFD) for edge devices. For the input features of the feature extractor, CLFD employs wavelet transforms to map the original input image to the frequency domain, thereby reducing the size of the input feature map. In experiments on two public datasets, the performance of the proposed and conventional methods is discussed in terms of both accuracy and learning efficiency.", "questions": "The paper and the proposed method need both clarity and experimentation. \nIn particular, the proposed method's effectiveness is difficult to determine because of its ad hoc design and insufficient explanation.\n\n- Please explain more about the necessity of utilizing frequency space for continuous learning for edge devices.\n- Furthermore, I'm curious about the absence of experiments with larger Buffer numbers in Table 1. I think edge devices that will be satisfied with larger buffer sizes may be developed in the future.\n- The overall ACC, represented by Equation 6, is the only evaluation. Other evaluation measures (e.g., forgetting measures) also exist in continuous learning. In addition, task-specific ACCs, etc., need to be evaluated. Please explain why you did not use these evaluation measures.", "limitations": "- For large data sets, the proposed method may need to be revised.\n- When the number of buffers is large, the proposed method may be inferior to the conventional method.\n- When the number of tasks increases, the proposed method is likely not to perform well for the latter tasks.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Continuous learning in edge devices is a significant study from a practical point of view.\n- The proposed method is simple and effective under limited conditions.\n- A minimal survey of previous research is provided.", "weaknesses": "Throughout, the explanation of the proposed method needs to be more comprehensive. Experiments also need to be more comprehensive to demonstrate the effectiveness of the proposed method. Specifically, the paper has the following rooms for improvement.\n- In line 60, there needs to be a clear explanation of why using frequency space is adequate. While it is interesting to get inspiration from HVS, there is no apparent reason why it is a means to the challenge of the proposed method (continuous learning on edge devices). In other words, the introduction needs a more logical structure.\n- In Figure 2, the meaning of the symbols (e.g., \\otimes, etc.) is unclear, so a specific explanation is needed. The clarity of the figure needs to be improved.\n- In the description of the method, there is the following statement: \"Considering that tasks are predominantly sensitive to specific frequency domain features extracted by a feature extractor, different To this end, we propose the CFFS, designed to manage the issue of overlap in frequency domain features among samples from different classes.\" However, no results from the analysis support this issue (fact). Furthermore, it needs to state why CFFS is the idea to solve this fact. Therefore, the design of the proposed method needs to be more convincing.\n- Many grammatical errors need to be corrected. A comma or period is needed after the formula. (e.g., line 210: Where->where)\n- The method's design throughout is very ad hoc and heuristic. For example, in Equation 4, no clear reason is given for the algorithm's design.\n- Only the overall ACC represented by Equation 6 is evaluated in the experiment. In continuous learning, there are other evaluation measures (e.g., Forgetting measure). In addition, task-specific ACCs and other measures need to be evaluated. \"[Chaudhry et al., 2018b] Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In Proc. ICLR, 2018.\"\n- In Table 1, there are no experiments for larger Buffer numbers. For example, previous studies (CLS-ER and SparCL-ER) have more buffers in their paper. Why are there no comparisons for larger numbers of buffers? It is unfair from the point of view of experiments in academic papers to publish only comparisons of conditions in which the proposed method is superior. (For example, edge devices that will be satisfied with larger buffer sizes may be developed.)\n- Furthermore, it is difficult to judge the effectiveness of the proposed method since the experiments do not evaluate larger datasets; in this paper, only smaller datasets such as CIFAR and TinyImageNet are used.\n- When the number of tasks increases, the proposed method will likely perform poorly for the latter tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we extend our gratitude for the time and attention devoted to reviewing our paper. Below, we have carefully addressed each of your concerns to the best of our knowledge to improve the overall contribution of the paper.\n\n# The necessity of utilizing frequency space for continuous learning for edge devices.\n\nThank you for this great question. Edge devices are constrained by limited memory and computing resources, underscoring the significance of training efficiency and peak memory usage in continual learning. Presently, continual learning primarily operates in the spatial domain, where the filters in a CNN are generally smooth due to the local pixel smoothness in natural images, resulting in spatial redundancies that impact the efficiency and memory usage of continual learning. Conversely, in the frequency domain, it is possible to train on images using distinct frequency domain features, significantly reducing redundancy in the learning process and improving the performance of continual learning methods.\n\n# The symbols in Figure 2 is unclear.\n\nSorry for the unclear expression. $\\otimes$ represent this feature being masked. In the final version of the paper, we will improve the clarity of the figure.\n\n# The design of the CFFS needs to be more convincing.\n\nIn a previous study [1], it was found that continual learning has task-sensitive parameters for different tasks, and reducing the overlap of these parameters can significantly improve the performance of continual learning. Similarly, in the frequency domain, we observed that it also has task-sensitive parameters. As shown in Table 3, when we pruned 75% of the weights, the performance of the CLFD method improved significantly. Considering the presence of task-sensitive parameters in the classifier, which causes different tasks to be sensitive to specific frequency domain features, we propose CFFS to balance the reusability and interference of frequency domain features.\n\n[1] Sarfraz, Fahad, Elahe Arani, and Bahram Zonooz. \"Sparse coding in a dual memory system for lifelong learning.\" AAAI. 2023.\n\n# Grammatical errors need to be corrected.\n\nWe appreciate the reviewer's feedback regarding the grammatical errors. We will update the paper according to your suggestions in the final revision.\n\n# The method's design throughout is very ad hoc and heuristic.\n\nWe agree with the reviewer that the method's design is very heuristic. However, the design of our method is well-founded. Heterogeneous Dropout has already been demonstrated to be effective in continual learning within the spatial domain [1,2]. Building on this foundation, we propose Frequency Dropout (Eq. 4). Additionally, in this work, we emphasize that our primary focus is on enhancing the efficiency of continual learning training by leveraging the frequency domain, thereby promoting its application on edge devices. Our research is among the first in this area, and we hope it will inspire the continual learning research community to further explore the frequency domain.\n\n[1] Sarfraz, Fahad, Elahe Arani, and Bahram Zonooz. \"Sparse coding in a dual memory system for lifelong learning.\" AAAI. 2023.\n\n[2] Vijayan, Preetha et al. “TriRE: A Multi-Mechanism Learning Paradigm for Continual Knowledge Retention and Promotion.” NeurIPS. 2023.\n\n# Other evaluation measures.\n\nIn Appendix Table A2, we provide the forgetting measure, while Figure 7 presents the task-specific ACCs, and Figure 8 illustrates the stability and plasticity of our framework. We will incorporate the key experimental results into the main text.\n\n# There are no experiments for larger Buffer numbers.\n\nWe agree with the reviewer that conducting experiments on larger buffer sizes is essential. We conduct re-experiments under the buffer settings of SparCL-ER [1] and provide a detailed discussion in the general response.\n\n[1] Wang, Zifeng, et al. \"Sparcl: Sparse continual learning on the edge.\" NeurIPS. 2022.\n\n# The experiments do not evaluate larger datasets.\n\nWe agree with the reviewer's suggestion that incorporating larger datasets can enhance the credibility of the results. Therefore, we introduced the Split Imagenet-R dataset as an alternative and provided a detailed discussion in the general response.\n\n# The proposed method will likely perform poorly when the number of tasks increases.\n\nWe sincerely thank the reviewers for raising this concern. On the Split Tiny ImageNet and Split ImageNet-R datasets, each comprising 10 tasks, our framework consistently demonstrates strong performance, further validating the effectiveness of our framework. We emphasize that although our accuracy remains high even with a large number of classes, the principal contribution of our work lies in introducing the concept of continual learning in the frequency domain, which significantly improves training efficiency and facilitates the application of continual learning on edge devices.\n\nWe once again thank the reviewer for providing detailed feedback. We have made an utmost effort to resolve all the concerns raised. Please let us know in case we have missed something.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers' valuable comments and concerns. In the response below, we hope to have addressed all the points raised. Should there be any further questions or clarifications needed, please inform us so we can fully address any aspects of the paper during the rebuttal period. We plan to use additional pages to provide detailed clarifications on the issues raised by the reviewers, as outlined in the following responses. In this general response, we have clarified the reviewers' concerns regarding the accuracy of the proposed method with larger buffer sizes and on other datasets.

# The performance of our framework on larger buffer sizes.

We maintained the same buffer size selection as the state-of-the-art efficient continual learning method, SparCL [1], and conducted re-experiments on the Split CIFAR-10 and Split Tiny ImageNet datasets. Table 1 in the attached PDF presents the experimental results, demonstrating that our method still significantly enhances the performance of various rehearsal-based continual learning methods. It is important to note that while improving accuracy is a notable outcome, it is not the primary goal of this work. The novelty of this work lies in introducing the concept of continual learning in the frequency domain, which effectively reduces the training time and peak memory usage of continual learning methods, thereby facilitating their application on edge devices.

[1] Wang, Zifeng, et al. "Sparcl: Sparse continual learning on the edge." NeurIPS. 2022.

# The performance of our framework on other datasets.

We appreciate the reviewers' suggestion that incorporating larger datasets can enhance the credibility of our results. However, we consider Split Tiny ImageNet to be a substantial dataset. Numerous recent studies [1,2] have utilized Split Tiny ImageNet as the largest dataset in their experiments. Additionally, Mammoth CL [3], a widely adopted continual learning repository, also uses Split Tiny ImageNet as its largest dataset.

To further enhance the credibility of our results, we conducted additional experiments on Split ImageNet-R. We tested the Class-IL accuracy of each task with a buffer size of 500. Figure 1 in the attached PDF presents the experimental results, demonstrating that our method still significantly enhances the performance of various rehearsal-based continual learning methods, even on more complex datasets. It is worth noting that the improvement in accuracy is not the sole advantage of our framework. By integrating our framework with rehearsal-based methods on the Split ImageNet-R dataset, training speed increased by up to 1.7 $\times$, and peak memory usage decreased by up to 2.5 $\times$. This demonstrates that our framework can significantly enhance the training efficiency of continual learning, thereby promoting its application on edge devices.

[1] Gao, Qiang, et al. "Enhancing knowledge transfer for task incremental learning with data-free subnetwork." NeurIPS. 2023.

[2] Vijayan, Preetha et al. “TriRE: A Multi-Mechanism Learning Paradigm for Continual Knowledge Retention and Promotion.” NeurIPS. 2023.

[3] Buzzega, Pietro, et al. "Dark experience for general continual learning: a strong, simple baseline." NeurIPS. 2020.

Meta-review: This work leverages the frequency domain to enhance the performance and training efficiency of rehearsal-based continual learning methods.  For the input features of the feature extractor, CLFD employs wavelet transform to map the original input image into the frequency domain, thereby effectively reducing the size of input feature maps. Regarding the output features of the feature extractor, CLFD selectively utilizes output features for distinct classes for classification, thereby balancing the reusability and interference of output features based on the frequency domain similarity of the classes across various tasks. 

Initially the paper received mixed reviews. Reviewers have appreciation for the method design, extensive experiments, and writing, while raising concerns on lack of experiments on more complex datasets, method scalability, etc. The authors have provided a strong rebuttal that convinced the reviewer enough to support acceptance. The AC-panel concurs with their decision and strongly encourages the authors to include all additional clarifications and important experiments in the camera-ready version.
Decision: Accept (poster)