Tensor Decomposition Based Memory-Efficient Incremental Learning
Authors: Yuhang Li
Guoxu Zhou
Zhenhao Huang
Xinqi Chen
Yuning Qiu
Qibin Zhao
Venue: ICML 2025 poster (2025)
Keywords: Class Incremental Learning; Continual Learning
Primary area: general_machine_learning->online_learning_active_learning_and_bandits
TL;DR: a model-friendly and data-friendly memory-efficient replay methods based on tensor decomposition

Abstract: Class-Incremental Learning (CIL) has gained considerable attention due to its capacity to accommodate new classes during learning. Replay-based methods demonstrate state-of-the-art performance in CIL but suffer from high memory consumption to save a set of old exemplars for revisiting. To address this challenge, many memory-efficient replay methods have been developed by exploiting image compression techniques. However, the gains are often bittersweet when pixel-level compression methods are used. Here, we present a simple yet efficient approach that employs tensor decomposition to address these limitations. This method fully exploits the low intrinsic dimensionality and pixel correlation of images to achieve high compression efficiency while preserving sufficient discriminative information, significantly enhancing performance. We also introduce a hybrid exemplar selection strategy to improve the representativeness and diversity of stored exemplars. Extensive experiments across datasets with varying resolutions consistently demonstrate that our approach substantially boosts the performance of baseline methods, showcasing strong generalization and robustness.

Tensor Decomposition Based Memory-Efficien Incremental Learning
Yuhang Li Guoxu Zhou 12 Zhenhao Huang'
Xinqi Chen'
Yuning Qiu Qibin Zhao 31
3
Abstract
world, data is continuously generated and constantly chang- ing (Gomes et al., 2017), resulting in consistent delay in incorporating new knowledge by those advanced mod- els (e.g., large language models such as GPT-4 (Achiam et al., 2023)). To address this, researchers have turned to a promising approach known as Increm-nnlremtttlll Learn- ing" (CIL) (Masana et al., 2022; Zhou et al., 2023b). CIL aims to learn from streaming data continually without for- getting previously learned. Despite this vision, when the model directly trains new data, it tends to forget sub- stantial amount of previously captured knowledge, leading to irreversible performance degradation phenomenon known as catastrophic forgetting (McCloskey & Cohen, 1989; Robins, 1993). Therefore, effectively combating catas- trophic forgetting is the central challenge in developing CIL methods.
Learning (CIL) has gained con- siderable attention due to its capacity to accommo- date new classes during learning. Replay-basee methods demonstrate state-of-the-ar performance in CIL but suffer from high memory consumption to save a set of old exemplars for revisiting. To address this challenge, many memory-efficient replay methods have been developed by exploit- ing image compression techniques. However, the gains are often bittersweet when pixel-level com- pression methods are used. Here, we present a simple yet efficient approach that employs tensor decomposition to address these limitations. This method fully exploits the low intrinsic dimension- ality and pixel correlation of images achieve high compression efficiency while preserving suf- ficient discriminative information, significantly enhancing performance. We also introduce a hy- brid exemplar selection strategy to improve the representativeness and diversity of stored exem- plars. Extensive experiments across datasets with varying resolutions consistently demonstrate that our approach substantially boosts the performance of baseline methods, showcasing strong general- ization and robustness.
Replay-based methods (Rebuffi et al., 2017: Lopez-Paz & Ranzato, 2017; Isele & Cosgun, 2018; Zhao et al., 2020; Bang et al., 2021; Wang et al., 2022a), inspired by human learning processes (Robins, 1993; Kumaran et al., 2016; Kudithipudi et al., 2022), have demonstrated state-of-the- art performance in various incremental learning scenarios. These methods explicitly store subset of previously en- countered data (exemplars) in memory. When training on new tasks, the model can revisit these exemplars to help mitigate catastrophic forgetting. While intuition suggests that storing more old exemplars could enhance performance, practical constraints, e.g., fixed memory capacity (Krempl et al., 2014; Rebuffi al., 2017), restrict only few data can be kept for replay. Consequently, there is severe imbalance between new and old data, with the training process always dominated by new ones.
1. Introduction
In recent years, deep learning has achieved remarkable ad- vancements, with neural networks reaching or even surpass- ing human-level performance in many domains (Silver et al., 2016; Jumper et al., 2021). Traditionally, deep networks rely heavily on offline training, requiring multiple epochs over a static, pre-collected dataset. However, in the open
To alleviate the above problem, various Memory-Efficieee Replay Methods (Wang et al., 2024) have been proposed. For instance, MECIL (Zhao al., 2021) proposes using low-fidelity samples instead of the original ones to transfer old class knowledge. CIM (Luo et al., 2023) suggested downsampling non-diseriminative pixels (e.g., background) while retaining discriminative pixels (c.g., foreground) in the original image. MAF-CII (Zhai et al., 2023) demonstrates that Masked Autoencoders (MAE) (He et al., 2022) are efficient incremental learners, which store random image patches as exemplars and they can reconstruct high-quality images from only partial information for replay. Although
School of Automation, Guangdong University of Technol- ogy. Guangzhou, CHINA Key Laboratory of Intelligent Detection and the Internet of Things in Manufacturing, Ministry of Educa- tion. Guangdong University of Technology, Guangzhou, China p PRIKEN AIP, Tokyo, JAPAN. Correspondence to: Guoxu Zhou <gx.zhou@ edut.cdu.cn
Proceedings of the 42 International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Tensor Decomposition Based Memory-Efficient Incremental Learning
these methods achieve notable performance, they struggle to reconcile fettd and model-friendllinss (see Sec. 2 for more details).
solutions the challenges posed by CIL. 2) Distinguished from most existing memory-efficient methods, our approach can be effectively applied to data of various resolutions and easily integrated with other rehearsal methods as plug-and. play solution, serving as a model-friendly and data-friendly method. 3) In the context of memory-efficient CIL meth- ods, we introduce an innovative exemplar selection strategy that effectively balances the quantity and quality of samples and improves performance to some extent. This strategy is adaptable and can be generalized to other similar meth- ods. 4) Extensive experiments consistently indicate that the proposed method achieves excellent performance across di- verse experimental settings while exhibiting low sensitivity to parameter variations, underscoring the method's strong generalization and robustness.
Pixel-level compression methods, such as MECIL and CIM, directly compress images in the original high-dimensional pixel space, overlooking the fact that natural images typi- cally have low intrinsic dimensionality (Levina & Bickel, 2004) and exhibit local connectivity. To illustrate, in im- age reconstruction tasks, Masked Autoencoder (He al., 2022) divides an image into multiple patches and recon- structs the complete image by randomly masking a certain proportion of these patches. Experimental results show that with 75% of the patches masked, the original image can still be reconstructed. Similarly, low-rank tensor decomposition- based completion methods can recover images with high fidelity, even when over 80% of their data is missing (Yokota et al., 2016; Qiu et al., 2024). Additionally, the Maximum Likelihood Estimation of the intrinsic dimensionality of im- ages on the ImageNet (Deng et al., 2009) indicates that, although each image contains 150,528 pixels, its intrinsic dimension is only between 26 and 43 (Levina & Bickel, 2004). These results suggest that compressing data directly in high-dimensional pixel space is inefficient, and it can be represented approximated with much less complexity.
2. Related Works
There are three principal strategies to mitigate catastrophic forgetting in incremental learning (Van de Ven et al., 2022; Zhou et al., 2023b). Regularication--aseed Methods (Kirk- patrick et al., 2017; Wang et al., 2022c) introduce additional regularization terms into the loss function penalize signif- icant changes in the weights crucial for previously learned tasks. This ensures that the model retains its performance on old tasks while learning new ones. Architecture-based Methods (Yan et al., 2021; Zhou et al., 2022) aims to miti- gate catastrophic forgetting by dynamically modifying the fundamental concept here is to preserve the knowledge of past tasks by freezing the pa- rameters of previously trained sub-networks (called "old model"). New network structures are then introduced to ac- commodate new knowledge from incoming tasks. Replay- based Methods (Rebuffi et al., 2017; Lopez-Paz & Ranzato, 2017; Wang et al., 2022a) maintain a memory buffer with a fixed budget to store a few examples from previous tasks. When training on new tasks, the model can revisit these examples to reinforce its memory of old knowledge, thereby preventing forgetting.
Building on these insights, we propose introducing low- rank tensor decomposition (TD) (Kolda& & Bader, 2009) for image compression. This method takes full advantage of natural images' local correlation and low intrinsic dimen- sionality. Not only can it achieve considerable compression efficiency, but it also effectively captures the internal multi- dimensional structure of the image (e.g., the spatial distribu- tion of pixels and the correlation between chamnels) (Kilmer et al., 2021) so that more discriminative information can be retained after compressed, which is an attribute crucial for memory-efficied replay methods. Furthermore, recog- nizing the importance of balancing the quality and quantity of exemplars for this kind of method (Zhao et al., 2021; Wang et al., 2022b), we propose a novel exemplar selection strategy to enhance the representativeness and diversity of the exemplars.
network architecture.
Beyond this, some Memory-Efficient Replay Methods seek to improve storage efficiency by compressing raw im- ages or storing feature-level data, achieving impressive per- formance. Works centred on feature storage (Iscen et al., 2020; Hayes et al., 2020; Toldo & Ozay, 2022), referred to as feature replay (Wang et al., 2024), offer considerable efficiency and privacy benefits by preserving feature-Ievel distributions rather than raw data. However, major hurdle is representation drift resulting from the feature extractor's sequential updating. a phenomenon indicative of feature- level catastrophic forgetting (Wang et al., 2024). Knowledge Distillation (KD) (Hinton et al., 2015) and the partially fixed feature extractor (Hayes et al., 2020) are compromised solu- tions for this problem.
To evaluate the proposed method, we integrate it into several advanced CIL. methods (i.e., iCaRL. (Rebuffi et al., 2017), FOSTER (Wang et al., 2022a), DER (Yan et al., 2021), MOME (Zhou et al., 2022)) and perform extensive experi- ments on different-scale datasets. The results demonstrate that our proposed method consistently delivers substantial improvements. To summarize, our contributions are as fol- lows: 1) We propose a memory-efficient CIL method based on TD. To the best of our knowledge, tensor methods have not been used in CIL to mitigate catastrophic forgetting. Our work represents a new attempt, and we anticipate that intro- ducing the tensor approach will yield valuable insights and
2

---

Tensor Decomposition Based Memory-Efficient Incremental Learning
Another work line improves storage efficiency by compress- ing images (Caccia et al., 2020; Wang al., 2022b; Luo et al., 2023; Zhai et al., 2023). For example, MECIL (Zhao et al.. 2021) employs an additional auto-encoder to trans- form high-fidelity images into lower-fidelity ones and in- troduces an adaptive training scheme to mitigate perfor- mance degradation caused by domain shifts. CIM (Luo al., 2023) suggests downsampling dinermriimiiteeee pix- els (e.g., background) while retaining discriminative pixels (e.g., foreground) in the original image and introduces addi- tional Paué Activation Units (PAU) (Molina et al., 2019) to determine the activation areas at each incremental stage dy- namically, but this method is not efficient for low-resolution datasets, such as CIFAR (Krizhevsky, 2009). MAE-CIL (Zhai et al., 2023) stores random image patches as exem- plars, and they can reconstruct high-quality images from only partial information for replay. This method achieved offe-of-theearr performance, but the reconstruction heavily depends on MAE (He et al., 2022).
A
a
a
a
Figure I. Illustration of a CP decomposition of third-order tensor A into a sum of rank-1 tensors
There three common kinds of incremental learn- ing scenarios: Domain-incremente learning (DIL), Task- Incremental Learning (TIL), and Class-Incremental Learn- ing (CIL) (Van de Ven et al., 2022; Zhou et al., 2023b). In the DIL setting, all tasks share the same label space, the training data distributions for different tasks may differ (e.g.. various image styles), and the model does not receive information about which task the test sample belongs to during inference. For TIL and CIL settings, the label spaces for different tasks are disjoint; the model receives informa- tion about task identity during testing for TIL but not for CIL. In this paper, we focus on the more representative and challenging CIL.
Although these methods achieve notable performance, they necessitate the introduction of well-crafted additional struc- tures or strategies to compensate for the negative impacts of compression. In contrast, tensor decomposition- compression methods characterize the image using set of factors with fewer parameters. These factors are stored and can be reconstructed during subsequent task training. Thanks to the images' low-rank property, the reconstruction error remains minimal, ensuring that the reconstructed im- ages retain the same quality as the originals. This method serves training plugin, significantly boosting the perfor- mance of existing replay-based CIL techniques.
Tensor Decomposition a mathematical technique for ana- lyzing multidimensional data represented by tensors (Kolda & Bader, 2009). It breaks down a complex, higher-order ten- sor into combination of simpler, lower-order tensors. Here, we briefly introduce the CANDECOMP/PARATAC (CP) (Carroll & Chang, 1970) decomposition for an -order tensor A E TR x x-ll-xx Formally, the CP decomposi- tion decomposes A into sum of R rank-one factor tensors:
A
a()) aa(2)
c...o a
(N)
3. Methodology
(I)
This section introduces the basic concepts and notations, presents the classical incremental learning setup, and then describes how our proposed Tensor Decomposition- Memory- -ffiicient Replay (TDMER) works. Fig. 2 illus- trates the franework of our methods.
(A(0) (1) A
where rank R predefined positive integer, the symbol "o" represents the vector outer product, and ( E R a(?)
R at (N) E /R'N for r = -,..., R. These vectors can be grouped into N matrices, i.e. {A (n) E 1 Jn=1. where (n) [a[") a ..... .ap (m) for L.
3.1. Preliminary
and [[-] is defined as the CP decomposition operator (Kolda & Bader, 2009). Elementwise, the CP decomposition is written as:
Incremental Learning aims to train model on a sequence of tasks, represented by their datasets {D1, D²,.... " }. and achieve good performance on unseen test data from those tasks. The training set of task t contains N, distinct data-label pairs, D' = {(*h, N where x/, E X, and v E Yú represent the input instance and their corre- sponding label, respectively. The goal is to let the model, parameterized 0, learn a mapping from the whole input space X7- = U X, to the label space y = U Yr. i.e. fo X' + y At the test stage, the model should be able to predict the correct label y for a new unseen sample drawn from any of the previous tasks.
R
Oig A2...N
(wx)
(2)
For better clarity, a schematic representation of the CP de- composition of a third-order tensor is shown in Fig. 1
3.2. TD-Based Memory-Efficiiet Replay Method.
We introduce TD methods to factorize the images into a set of factor matrices or tensors. By selecting an appro-
3

---

Tensor Decomposition Based Memory-Effiieenl Incremental Learning
Memory
Write
Write
Write
D
D
D
Training
Training
Training
Training
Training
D2e
1
Dir1
D
learning stage
2-nd learning stage
I-th learning stage
Read
Read
Memory
Figure 2. Illustration of the proposed tensor decomposition bbased pemmory-efficient CIL method with two-stage exemplar selected strategy. We use D' represent training data 1-th task, D' represent the partial reconstructed data D', Di, represent the exemplar stored in memory, and Di, represent the exemplar reconstructed by the stored factor. The notion of "Write", "Read," and "Training" refers writing raw samples
factors obtained from other sample decomposition into memory, reading exemplars and factors set from memory, and training the model using new task data, stored exemplars, and reconstructed samples from the factors, respectively. All these exemplars are chosen by our exemplars selection strategy.
priate rank (crucial for balancing the fidelity and compact- ness of the factors), we can significantly reduce the size of these components while preserving crucial information. We first investigate two widely used TD methods: CAN- DECOMP/PARATAC (Carroll & Chang, 1970) and Tucker (Tucker, 1966). However, maintaining decomposition accu- racy with Tucker decomposition results in component sizes close to the original data for low-resolution datasets such as CIFAR- 11TAR--10110 10/100 (Krizhevsky, 2009), which are not effec- tively handled by many memory-efficcce replay methods (Wang et al., 2022c; Luo et al., 2023). This desirable for purposes. To ensure generalizability, chose to CP decomposition exclusively.
follows:
W 1 C
(3)
Note that with Tensorly (Kossaifi et al., 2019) for tensor decomposition, we first transform the original image (in integer format) to float-point format. As result, the factors from the decomposition ensure the Eq. (3) holds, need convert these float-point factors to 8-bit. By doing this, compression ratio of less than 1, where represents the memory required to store the raw image. Therefore, within the same memory budget, we can store more decomposed factors.
in floating-pons -oinngt-ponntttt format. To
consistently achieve
From CP Decomposition to Image Compression. For the data 5 E CC in D' given a rank R, the CP decomposition factorizes it into a sum of R rank-1 tensor factors. According to Eq. (I), we can infer size these components (be grouped into matrices) (1) e RCxR A (2) E H A (3) E R respectively. Therefore, the compression ratio 7/ of this process can be defined
Adaptive Training. Following the above concepts out- lined, focus the CIL setting, where for all t * t', X, n X4. = Ø and }} n Ø. The goal is to conduct continuous training without forgetting previous knowledge. Specifically, for conditioning the current task f, the setting with CIL remains rather tricky. The model must maintain stability for retaining the knowledge from all previous tasks

---

Tensor Decomposition Based Memory-Efficient Incremental Learning
(1, | 1), also requiring continuous learning to adapt to the new task This results in the stability-plastiiit dilemma. To overcome this bottleneck, appaa--based meth- ods employ a small-sized memory buffer, denoted as DM- Old tasks' exemplars are sampled, stored, and updated after completing training on each task. Data from the current task l is presented along with the exemplar set to the model. Thus, the objective function for replay-based CIL methods can be written as:
evaluate the quality of reconstruction, which is denoted as:
||3g - ||
RE,
(6)
To ensure the quality of the reconstructed data, we intro- duce hard threshold if the reconstruction error is larger than this threshold, which means that most of the critical information may have been lost, this data is discarded, and another one is re-selected. Thus, the final objective function is formulated as:
arg min
Σ
((f()))))) Xrregg
(Z.
(4)
Creg
Σ
Lreg (fo(z),
arg min
Σ
[ffo(z). + Ar + AgGreg:
(a.))E
(a.y)ED* U Ereg :=
Σ (a.))ED'r
Krg (fo(z).
where breg denotes the regularization terms based the old exemplars, and À is hyperparameter controlling the strength of the regularization.
(7)
where a and A2 is the hyperparameter making trade-off between D'r and D'r ,rey means the same regularization function.
Note that both the tensor decomposition and the data type conversion introduce some minor errors. To mitigate these effects, we adopt an adaptive training strategy. The objective function is as follows:
We introduce a parameter to control the proportion of samples between these two stages. When c is equal to O, which means that we only use the exemplars in Dir and X is set to O: when is equal to 1, it means that we only use the exemplars in " and A2 is equal to O. Notably, the number of samples selected in the first stage typically constitutes only modest part, meaning a focus on quantity.
arg min
Σ (fo(z).)) + XCreg UDD
(5)
Namely, in each training epoch, we select a subset of D', denoted as(af, -.. ,xy), and decompose each data low-rank factors, which can be reconstructed as {29 ...... x = D'. This allows the model to emulate training with processed images and learn to overcome the effects of error.
4. Experiments
4.1. Experimental Settings
Exemplar Selection Strategy. We notice that existing meth- ods almost always keep only compressed samples, but not all compressed samples have a positive effect; on the con- trary, some have negative impact (as shown in Fig. 3). It means that emphasizing quantity the expense of quality is counterproductive in memory-efficient CIL. To achieve a better trade-off, we propose a two-step hybrid sample selection strategy.
Datasets. We conducted experiments on two widely used datasets. CIFAR- 100 (Krizhevsky, 2009) is a low-resolution and widely used dataset for image classification tasks. It comprises 60,000 32x32 RGB images categorized into 100 classes, each containing 500 training and 100 testing im- ages. ImageNet-100 (Deng et al., 2009) is a subset of the larger ImageNet-
containing 100 randomly selected categories. Each category has approximately 1,300 training samples and 50 test samples, all high-resolution but varying in spatial size.
Specifically, in the first step, we use a common strategy called herding (Rebuffi et al., 2017) to select part of raw data xt. ... ),), denoted as D Herding helps identify and choose the samples that best represent the char- acteristics of their respective classes, meaning focus on quality.
Protocols. We conduct our experiments following two pro- tocols: learning from seratch (LFS) and learning from half (LFH), which are two different ways to split the classes into incremental phases. LFS: divides all the classes equally in each incremental stage. For example, if there are M stages and N classes, each task has N/M classes for train- ing. LFH: the first increment task divides half of the total classes, and the rest are equally divided among the subse- quent stages. Namely, the N/2 classes are distributed as the first task, and the N/2 - 1) classes are distributed as the remaining tasks.
In the second step, we randomly select a portion of the sam- ples that were not chosen in the first stage, e.g. D = {#), .... ;,,,), })),, composition factors and store them, denoted as Di-1 In the training, factors in D'r11 can be reconstructed flexibly, which denoted as (81, "...". ..).. We use relative error to
and D n D'=1 = Ø, get their de-
5

---

Tensor Decomposition Based Memory-Efficient Incremental Learning
Table average accuracy (%) and last accuracy (%) of four baseline methods on CIFAR-100, with without our TDMER plugged-in, the memory budget M- = 2k.
BaseO Inc 10
BaseO Inc20
Base50 Inc
Method
Avg
Last
Avg
Last
Avg
Last
iCaRL
60.83
42.99
62.85
46.67
55.99
46.67
iCaRL w/ ours 67.22 (+6.39)
51.98
(+8.99)
68.05
56.07 (+9.40)
61.59
(+5.60)
53.37 (+6.70)
FOSTER
66.67
54.24
69.50
60.11
66.75
58.76
FOSTER w/ ours
70.61 (+3.94)
58.25
(+4.01)
72.16
(+2.66)
62.41 (+2.30)
69.17
(+2.42)
61.79 (+3.03)
MEMO
69.57
58.13
69.61
60.60
65.95
58.45
MEMO w/ ours
72.67 (+3.10)
62.44
(+4.31)
72.47
(+2.86)
64.40 64.40
(+3.80)
68.61
(+2.66)
63.28 (+4.83)
DDER
771.05
59.64
71.11
63.17
69.37
62.71
DER w/ ours
73.69 (+2.64)
63.21 (+3.57)
73.57 (+2.43)
66.14 66.14(+2.97)
70.79 (+1.42) 65.16 (+2.45)
Both protocols are widely adopted in the current CIL com- munity (Yan et al., 2021; Wang et al., 2022a; Zhou et al., 2022; Luo al., 2023). To better distinguish, we use 'Base- Inc-c' denote different data splits, where represents the number of classes in the first stage and represents the number of courses in each incremental task. For example, "Base O, Inc 10" indicates that the LFS protocol is followed, and ten classes are assigned to each incremental task.
Table 2. average accuracy (%) and last accuracy (%) on CIFAR- 100 of MEMO and DER with without our TDMER integrated. under two memory budgets, M = 1k, M = 500.
BaseO Ine
Budget
Method
Avg
Last
MEMO
60.53
45.67
MEMO w/ ours
67.61
55.76
M 500
DER
64.74
50.95
Baseline Methods. We incorporate our method into four baseline CIL methods, and they are listed as: iCaRL (Re-
DER w/ ours
69.47
56.89
buffi al., 2017) combines knowledge distillation and pro- totype rehearsal with several innovative components, i.e.,
MEMO
65.15
52.08
MEMO w/ ours
70.51
59.30
M = 1k
seurest-mean-or exeempllllas classifier and prioritized exem- plar selection; DER (Yan et al., 2021) first freeze previously learned representation and expand a new feature extractor when new tasks come, then use a linear layer to aggregate the feature; MEMO (Zhou et al., 2022) proposes extending only the deeper layers specific the task, sharing more generalized shallow network for enabling model expansion with the least budget cost; FOSTER (Wang et al., 2022a) utilize a new module to fit the residuals between the target and the output of the original model, after which it removes redundant parameters and feature dimensions via a perfor- mant distillation strategy to maintain the single backbone model.
DDER
68.50
55,86
DER w/ ours
71.57
61.02
Implementation Details, Our implementation is based on the deep learning library PyTorch (Paszke et al., 2019), the CIL toolbox PyCIL (Zhou et al., 2023a), and the tensor development library Tensorly (Kossaifi al., 2019). Fol- lowing (Yan al., 2021; Wang al., 2022a; Zhou et al., 2022; Luo et al., 2023), we employ the same backbone, i.e., 18-layer ResNet (He et al., 2016) for ImageNet- 100 and 32- layer ResNet for CIFAR-100, with a fully-connected layer as the classifier across all our experiments, and all training hyperparameters remain the same with PyCII. More details will be provided in the appendix. A.
Evaluation Metrics. We use two metrics to evaluate the performance. First, Last Accuracy (Last) is used to estimate the final performance of CIL models, which calculates the classification accuracy for all seen classes up to the current task. However, reporting only the final accuracy ignores per- formance varying over the entire learning process. Hence, we employ Average Accuracy (Avg) to reflect the histori- cal performance, which is defined AA, = Σi= dt, where oz,; E [0, 1] denotes the accuracy of task after learn- ing task t (i S t). We shall report the final average accuracy, i.e., AAr. This metric considers the performance over the entire learning trajectory. Note that we only report the top-1 1 accuracy in all experimental results.
4.2. Results and Analyses
We evaluate the performance of our proposed method in- tegrated into four baseline methods on CIFAR-100 and ImageNet- 100. More results will be provided in the ap- pendix. B.
Results on CIFAR-100. From Tab. we have the follow- ing observations: (1) Our TDMER, when integrated into the four baseline methods, substantially improves both last accuracy and average accuracy across different data splits.
6

---

Tensor Decomposition Based Memory-Efficient Incremental Learning
Table average accuracy (%) and last accuracy (%) of four baseline methods on ImageNet-100, with without our TDMER plugged-in, the memory budget M- = 2k.
BaseO Incl0
BaseO Inc20
Base50 Inc
Method
Avg
Last
Avg
Last
Avg
Last
iCaRL
60.24
41.40
66.83
50.
55.20
43.32
iCaRL w/ ours
64.38 (+4.14)
47.34 (+5.94)
69.38 (+2.55)
54.40 (+4.28)
58.80 (+3.60)
47.12 (+3.80)
FOSTER
69.91
60.60
76.57
69.74
70.07
63.85
FOSTER w/ ours
74.51 (+4.60)
66.40
(+5.80)
78.59
(+2.02)
72.30 (+2.56)
75.05 (+4.98)
69.58 (+5.73)
MEMO
72.41
62.56
76.14
68.42
70.09
63.42
MEMO w/ ours
74.97 (+2.56)
66.94
(+4.38)
77.21 (+1.07)
(42.688)
70.63 (+0.54)
65.38 (+1.96)
"DER
77.08
66.84
78.56
72.10
77.37
71.10
DER w/ ours
79.75 (+2.677
73.26 (+6.42)
79.69 (+1.13)
73.90 (+1.80) 79.96 (+2.39) 75.46 (+4.36)
Table 4. average accuracy (%) and last accuracy (%) ImageNet- 100 MEMO and DER with and without our TDMER integrated, under two memory budgets.M = 1k, M
Table 5. Comparison of our method with other memory replay methods on the ImageNet-100 under the 'Base0 Inc10' setting.
500.
BaseO Incl0
BascO Incl0
Budget
Method
Budget
Method
Avg
Last
Avg
Last
MEMO
63.78
50.68
MRDC (ICLR'22)
76.02
MEMO w/ ours
67.64
57.66
CIM (CVPR'2) 23)
77.94
M- = 500
M = 2k
DER
69.86
55.86
MAE-CIL (ICCV'23)
79.54
70.29
DER w/ ours
72.66
62.54
Ours
79.75
73.26
MEMO
68.99
57.92
MEMO w/ ours
71.77
62.26
M = 1k
DER
72.41
62.22
Results on ImageNet- Keeping the same experimental settings as CIFAR- 100, we report the best average accu- racy and last accuracy on ImageNet- in Tab.3, and Tab. 4 shows the average accuracy and last accuracy of MEMO and DER for two memory budgets on the ImageNet- The experimental results on ImageNet- 100 corroborate the find- ings from CIFAR- Our proposed method consistently improves the performance of various baseline methods, with the magnitude of improvement varying across methods, in- creasing with the number of tasks and achieving higher improvement on a stricter memory budget. These consistent trends across different datasets underscore the robustness and effectiveness of our approach.
DER w/ ours
74.95
65.36
Specifically, the highest observed increases are 6.39 % for average accuracy and 8.99 % for last accuracy under 10- task setting; (2) Although our improvements are signifi- cant across all baseline methods, the extent of enhance- ment varies, with the largest gains observed for the iCaRL baseline, while the performance boost for the already high- performing DER is comparatively smaller; (3) Furthermore, as the number of tasks increases, the improvements provided by our method also grow, indicating that our approach be- comes increasingly beneficial in more complex incremental learning scenarios.
Comparing with other efficient replay methods. We com- pare our method with variety of memory-efficient replay methods, such as MRDC (Wang et al., 2022b), CIM (Luo et al., 2023), and MAE-CII. (Zhai et al., 2023). All these methods use the same amount of storage, i.e., 20 original images per class. Tab. shows the results on the ImageNet- 100 under the 'Basc0 Inc10' setting. Our method achieves consistently higher performance.
Tab. 2 shows the results of MEMO and DER for two mem- ory budgets on the CIF We can see that our method consistently boosts MEMO and DER, achieving higher per- formance growth with tighter memory budget. Specifically, improves the last accuracy of MEMO by 10.09% when M = 500, with an increment of 7.28% when M 1A (the increment is 4.31 % when M = 2k). DER also shows simi- lar results, but with a slightly smaller increase compared to MEMO, namely, for the three memory budgets M = 500, M = 1k, and M = 2k, the increments are 5.94%. 5.16%, and 3.57%, respectively.
Effectiveness of Compression Methods. Memory-efficient replay methods improve storage efficiency by retaining more compressed data, and the quality of the compressed samples is critical to whether such methods work. To evaluate the compression techniques, we processed the CIFAR-101 train-
7

---

Tensor Decomposition Based Memory-Effiiien Incremental Learning
NBMO
Proportion
Proportion
Rank 10
Rank 12
Rank 14
Rank 16
(a) Results CIFAR-100.
100
AA PNO
AAA
1830 Ales
Proportton
Proportion
Proportion
Proportion
Rank 70
Rank 80
Rank 90
Rank 100
(b) Results on ImageNet-100.
Figure 3. Results of different ranks R and proportion average accuracy (Avg %) with methods integrated into MEMO. (a)
CIFAR-100, (b) ImageNet-1000
larly, Fig. (b) presents the results on the ImageNet-100 1000. We observe that ImageNet-100 exhibits relatively stable performance across different e. However, the model's per- formance improves with increasing c, requiring a larger e to achieve optimal performance. We attribute the different performances of the two datasets their complexity and di- versity. Overall, our exemplar selection strategy effectively balances the quality and quantity of exemplars, thereby en- hancing performance. This approach provides a flexible mechanism to adapt to different dataset characteristics, en- suring optimal memory use and maintaining high model performance.
Table 6. Classification accuracy CIFAR 100 processed with different compression methods. "Accu." means the accuracy (%), "Comp." means the compressed ratio. "Down Sample" means setting downsampling rate to 2.56, and "CIM" means keeping the 16*16 centre region the original image, and the other parts image with a downsampling rate
Upper Bound
Down Smaple
TDMER
Method
CIM
12
16
Accu.
72.3
44.1
66.9
67.8
69.9
Comp.
1.0
0.39
0.56
0.26
0.35
5. Conclusion
ing set using various compression methods, then trained 32-layer ResNet from scratch the original (upper bound) and processed training sets, respectively. The original test set was used for classification evaluation. As shown in Tab. 6, the TD-based compression method achieves con- siderable compression efficiency, and the gap between its performance and the upper bound is small. This indicates that our method retains sufficient discriminative information, allowing perform comparably to the original training set.
In this work, we propose a data-friendly and model- friendly memory efficient replay method based on CAN- DECOMP/PARATAC (CP) decomposition. This method effectively leverages the local correlation and low intrin- sie dimensionality natural images exhibit, achieving high compression efficiency while preserving discriminative in- formation. Additionally, we introduced an innovative exem- plar selection strategy to balance the quality and quantity of examples further. Extensive experiments consistently demonstrate that our methods significantly improve replay- based methods. Overall, our work represents a novel attempt to integrate tensor methods with CIL. We expect that the in- troduction of tensor methods will provide valuable insights and solutions the challenges faced by the CIL. community.
Sensitivity Analyses. We conducted experiments with dif- ferent ranks and which controls the proportion of the raw and compressed data. As shown Fig. 3 (a), different e yield varying performance on the CIFAR-100, with the best performance observed around 6 of 0.1 or 0.2. Simi-
8

---

Tensor Decomposition Based Memory-Efficient Incremental Learning
Impact Statement
Hayes, T. L.., Kafle, K., Shrestha, R., Acharya, M., and
Kanan, C. Remind your neural network prevent catas- trophic forgetting. In European conference computer Springer, 2020.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
vision, pp.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Acknowledgements
This work is supported in part by the Natural Science Foundation of China (NSFC) under Grant 62203124 and 62406077. We also thank the anonymous reviewers for their helpful comments.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
References
Hinton, G., Vinyals, O., and Dean, Distilling the knowledge in neural network. arXiv preprint 2015.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Iscen, A., Zhang, J., Lazebnik, S., and Schmid, C. Memory- efficient incremental learning through feature adaptation. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pp. 699-715. Springer, 2020.
Bang, J., Kim, H., Yoo, Y., Ha, J..W., and Choi, Rainbow memory: Continual learning with memory of diverse samples. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8218- 8227, 2021.
Isele, and Cosgun, A. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Caccia, L., Belilovsky, E., Caccia, M., and Pineau, Online learned continual compression with adaptive quantiza- tion modules. In International conference on machine learning, pp. 1240-1250 PMLR, 2020.
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-58) 2021.
Carroll, J. D. and Chang, J. Analysis of individual differ- ences in multidimensional scaling via an n-way general- ization of "eckart-ye "eckart-young" decomposition. Psychometrika, 35(3):283-311 1970.
Kilmer, M. E., Horesh, L., Avron, H., and Newman, Tensor-tensse tensor algebra for optimal representation and com- pression of multiway data. Proceedings of the National Academy of Sciences, 11 28828)):001881111 2021.
Deng, J., Dong, W., Socher, R., Li, I.-J., J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 255. Icee, 2009.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Baabska-Barwinska A., al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 11((13)332213333 2017.
Dosovitskiy, An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Kolda, G. and Bader, B. W. Tensor decompositions and applications. SIAM review, (1(3):455-50K 2009.
Douillard, A., Ramé, A., Couairon, G., and Cord, M. Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9285- 9295, 2022.
Kossaifi, J., Panagakis, Y., Anandkumar, A., and Pantic, M.
Tensorly: Tensor learning in python. Journal of Machine Learning Research, 20(26):166, 2019.
Krempl, G., Žliobaite, I., Brzeziński, D., Hüllermeier, E., Last, M., Lemaire, V., Noack, T., Shaker, A., Sievi, S., Spiliopoulou, M., et al. Open challenges for data stream mining research. ACM SIGKDD explorations newsletter, 16(1):1-10, 2014.
Gomes, H. M., Barddal, J. P., Enembreck, F., and Bifet, A. A survey on ensemble learning for data stream classification. ACM Computing Surveys (CSUR), 10(2):1-36, 2017.

---

Tensor Decomposition Based Memory-Effiiien Incremental Learning
Krizhevsky, A. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Kudithipudi, D., Aguilar-Simon, M., Babb, J., Bazhenov, M.. Blackiston, D., Bongard, J., Brna, A. P., Chakravarthi Raja, S., Cheney, N., Clune, J., et al. Biolog- ical underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196-210, 2022.
Robins, A. Catastrophic forgetting in neural networks: the role of rehearsal mechanisms. In Proceedings 1993 The First New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems, pp. 68-68.15 IEEE, 1993.
Kumaran, D., Hassabis, D., and McClelland, J. L. What learning systems do intelligent agents need? complemen- tary learning systems theory updated. Trends cognitive sciences, 20(7):512-534 2016.
Silver, D., Huang, A., Maddison, C. Guez, A., Sifre, L.,, Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Levina, E. and Bickel, Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems, 17, 2004.
Toldo, M. and Ozay, M. Bring evanescent representations life in lifelong class incremental learning. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 16732-16741, 2022.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Luo, Z., Liu, Y., Schiele, B., and Sun, Q. class-incremental exemplar compression for class-incremental learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pullern Recognition, pp. 11371-11380 2023.
Tucker, L. R. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):279 1966.
Van de Ven, G. M., Tuytelaars, T., and Tolias, A. S. Three types of incremental Icarning. Nature Machine Intelli- gence, 4(12):1 185-1185-9999 1197, 2022.
Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A. D., and Van De Weijer, J. Class-incremental learning: survey and performance evaluation on image classifica- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-55333 2022.
Wang, F-Y.,, Zhou, D.W.,, Ye, H.-J., and Zhan, D.-C. Foster: Feature boosting and compression for caas-incrrmmental learning. European conference on computer vision, pp. 398-414. Springer, 2022a.
McCloskey, M. and Cohen, N. J. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, vol- ume 24, pp. 109-165. Elsevier, 1989.
Wang, L.,, Zhang, X., Yang, K., Yu, L.,, Li, C., Hong, L., Zhang. S., Li, Z., Zhong, Y., and Zhu, J. Memory re- play with data compression for continual learning, arXiv preprint 22Xi::2002.06592 2022b.
Molina, A., Schramowski, P., and Kersting, K. Padl'e activation units: End-to-end learning of flexible ac- tivation functions in deep networks. arXiv preprint 1rXhv:1907.06732. 2019.
Wang, L., Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: theory, method and application. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
Oseledets, I. V. Tensor-train decomposition. SIAM Journal Scientific Computing, 3315):2295-2317,2000 2011.
Wang, Z., Liu, L., Duan, Y., and Tao, D. Continual learning through retrieval and imagination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8594-8602, 2022c
Paszke, A., Gross, S. Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Yan, S., Xie, J., and He, X. Der: Dynamically expandable representation for class incremental learning. In Proceed- ings of the IEEE/CVF conference computer vision and 2021.
pattern recognition, pp.
Qiu, Y., Zhou, G., Zhao, Q. and Xie, S. Noisy tensor completion via low-rank tensor ring. IEEE Transactions on Neural Networks and Learning Systems. 35(1):1127- 1141, 2024.
Yokota, T., Zhao, Q., and Cichocki, A. Smooth parafac decomposition for tensor completion. IEEE Transactions on Signal Processing, 64(20):5423-5444 5436, 2016.
10

---

Tensor Decomposition Based Memory-Efficient Incremental Learning
Zhai, J.-T., Liu, X., Bagdanov, A. D., Li, K., and Cheng, M.- M. Masked autoencoders efficient class incremental learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19104-19113, 2023.
Zhao, B., Xiao, X., Gan, G., Zhang, B., and Xia, S.-T. Main- taining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13208- 13217, 2020.
Zhao, H., Wang, H., Fu, Y., Wu, F., and Li, X. Memory- efficient dass-incremental learning for image classifica- tion. IEEE Transactions on Neural Networks and Learn- ing Systems, 33(10):5966-577, 2021.
Zhao, Q. Zhou, G., Xie, S., Zhang, L., and Cichocki, A. Tensor ring decomposition. arXiv preprint 1606:/606.0555 05535, 2016.
Zhou, D.-.., Wang, Q.- Ye. H.-J., and Zhan, D.-C. A model 603 exemplars: Towards memory-efficient class- incremental learning. arXiv preprint ar arXiv:2205.13221, 2022.
Zhou, D.-W., Wang, F.-.., Ye, H.J.,, and Zhan, D.-C. Pycil: python toolbox for class- maassinncremetall learning. 2023a.
Zhou, D.- Wang, Q-W., Qi. Z.-H., Ye, H.-J., Zhan, D.-C., and Liu, Z. Deep class-incremenall learning: A survey. arXiv preprint arXh::3302.0364. 2023b.

---

Tensor Decomposition Based Memory-Efficient Incremental Learning
A. Implementation Details
A.1. Training Details
Our implementation is based on the deep learning library PyTorch (Paszke et al., 2019) and the class-incremental learning toolbox PyCIL (Zhou et al., 2023a). Following (Yan et al., 2021; Wang et al., 2022a; Zhou et al., 2022; Luo al., 2023), we employ the same backbone, i.e., 18-layer ResNet (He et al., 2016) for ImageNet-100 and 32-layer ResNet for CIFAR-100, with fully-connected layer the classifier across all our experiments. We utilize the Stochastic Gradient Descent (SGD) optimizer for training with the following hyperparameters:
Base stage: We used SGD with an initial learning rate of 0.1, momentum of 0.9, and weight decay of 0.0005. The training epoch for all datasets is set to 200 with a batch size of 128. The learning rate is scheduled to decay by 0.1 at epochs 60, 120, and 170.
Incremental stage: We maintain the training epoch at 170, with a learning rate and momentum that remain constant concerning the base stage. Yet the learning rate decays with factor 0.1 at epochs 80 and 120, and the weight decay is adjusted to 0.0002. The learning rate decayed one more time in epoch 150 for MEMO.
Note that all hyperparameters remained consistent for FOSTER, except that it utilized cosine annealing strategy for learning rate decay. Besides, for all baseline methods, kept other hyperparameters consistent with those in PyCIL (Zhou et al., 2023a). Regarding the parameter of the CP decomposition in the main results, Tab. 7 lists them.
A.2. The Details of Decomposition
This study presents memory memory-efficient efficiennt replay method based on Tensor Decomposition (Kolda & Bader, 2009). Specifically, we intend to introduce the tensor decomposition method, factorizing the raw image (a third-order tensor) as a set of factor matrices or tensors, and with the hyperparameter called 'Rank' to balance the quantity and quality of the decomposition. Hence, we need to solve two problems, i.e., the selection of the decomposition method and the Rank. Here, we briefly introduce several common tensor decomposition methods (taking a third-order tensor as an example).
First is the CANDECOMPIPARAAAC (CP) (Carroll & Chang, 1970), which expresses tensor as a sum of rank-one tensors, each represented by column vector from corresponding factor matrices along each mode. For third-order tensor X E R the CP decomposition decomposes X into sum of R rank-one factor tensors:
R a o by o.,
X' A
(8)
where a, E RR', by E RR RRR R is the rank, and 'o' is the outer product. The advantage of CP decomposition its low storage complexity (only requires (ddr)) parameters) and computational complexity. Tucker decomposition (Tucker, 1966), which generalizes CP by allowing factor matrices to have different sizes, factorizes the tensor core tensor multiplied by a set of factor matrices along each mode. Formally, we can express it as:
R X1 Ax
(9)
where G E R R2 is the core tensor, A e I B Jx C R Rs is the factor matrices, 'x," represent the n-mode product, R1, R2, Rg is a set of rank. Compared with CP decomposition, Tucker Decomposition provides a more flexible way to represent the data but needs more parameters, i.e., Oddrr + rd). Tensor-Train (TT) Decomposition (Oseledets, 201 decomposes a tensor into a series of chained products third-order core tensors, which requires (dnr²) parameters. It can be represented as:
X(i,j,k) E((rr) (ali.ni)-e((r..i.r))) Ge(ri.jr2)) Ga(r2, k)
(10)
Ry
where G1 E RIXR, G2 E RixxR Gg IR R2XK is the core tensors, R1, Rg is the rank. TT decomposition is widely used for high-dimensional data compression. The last one the Tensor- Ring (TR) Decomposition (Zhao et al., 2016). TR
12

---

Tensor Decomposition Based Memory-Efficient Incremental Learning
generalization of the TT decomposition, which decomposes tensor into series of cyclic products of third-order core tensors with the same parameter complexity 'TT decomposition. Formally, it can be expressed as:
X(i,j,k) R
Σ G((r,i,r)) G2(11,3.22) G5(r, k3)
(11)
Ra.
where R G2 xRxRR, 93 E gRoxkxla is the core tensor, R1, R2, R3 the rank.
All of these methods can be used for compression, but mainly consider two points: storage complexity and universality. It is obvious that CP decomposition has the lowest storage complexity. About the universality, although Tucker decomposition, TT decomposition, and TR decomposition offer greater flexibility, for low-resolution datasets like CIFAR-I0/100 the size of the components will approach the original size to maintain the accuracy of the decomposition, which is undesirable in our case. To ensure generality, we opt to use only CP decomposition.
A.3. The Details of Memory Information
About the detailed memory information, for CIFAR-100, when R = c 0.2, M = 2k, according to Eq. 3 compression rate 7j R 0,26, which means we saved 400 original samples and about 6100 sets of sample factors. Similarly, for ImageNet- 100, when R 80, c = 0.4, .M 2k, the compression ratio 1j = 0.24, which means we saved 800 original samples and about 5000 sets of sample factors
B. More Experimental Results
As an extension of replay-based methods (Rebuffi et al., 2017; Lopez-Paz & Ranzato, 2017: Wang et al., 2022a), memory- efficient methods (Caccia et al., 2020; Zhao et al., 2021; Wang et al., 2022b; Luo et al., 2023; Zhai et al., 2023) improve storage efficiency by compressing the original image. Ideally, such methods should fully adaptable to baseline methods. However, as mentioned earlier, often challenging for them balance example, CIM (Luo al., 2023) only works well high-resolution datasets but struggles with low-resolution ones (e.g.. 32x32 CIFAR-100). MAE-CIL (Zhai al., 2023) heavily dependent on MAE (He et al., 2022) for the image reconstruction. Here we provide more results to demonstrate further that our method is data- friendly and model-friendy: Tab. 7 lists the details of the datasets.
with data- mata-frrendllinsss. For
Table 7. Details of the three datasets. R is the rank of decomposition, r is reconstruction error threshold, and e is the proportion of raw samples between decomposition factors.
Dataset
Classes
Training images
Test images
Size
R
CIFAR-
100
50000
10000
32x32
12
0.07
0.2
Tiny- ImageNet
200
100000
10000
64x64
24
0.07
0.2
ImageNet-100
100
129395
5,000
224x224
80
0.08
0.4
B.1. Results on Timy-ImageNet
We evaluate our method on three datasets with low, medium, and high resolutions, respectively. In the main text, we provide results on CIFAR-100 and ImageNet- 100 (low and high ones), and Tab. 8 presents results on -iny-ImageNet (M = 2k, 10-tasks setting, i.e., 'Base 0 Inc 20'). As shown, our method consistently improves baseline performance across datasets of various resolutions.
Table average accuracy (%) last accuracy (%) of four baseline methods on Kiny-ImageNet, with or without our TDMER plugged-in.
iCaRL
FOSET
MEMO
DER
Method
Avg
Last
Avg
Last
Avg
Last
Avg
Last
w/o ours
35.84
15.38
50.86
39.90
50.22
37.91
52.51
41.01
w ours
39.40
18.35
54.32
41.20
57.85
48.81
58.48
50.54
13

---

Tensor Decomposition Based Memory-Effiiient Incremental Learning
B.2. Results on ImageNet-Ik
Here we provide the numerical incremental performance of DER (Yan et al., 2021) ImageNet-IK, Ik, Tab. 9 lists the incremental and average accuracy. Note that the parameter configurations of ImageNet- 1k remain the same ImageNet-100 except that we set the memory budget M 20k.
Table Incremental and average accuracy comparison DER under ImageNetl 000 BascO Inc setting, the memory budget M 204.
000
Accuracy in each session (%)
Method
Avg
2
3
4
5
8
9
Last
DER 83.16
78.64
74.21
71.44
68.13
13
65.80
62.90
61.21
59.84
58.30
68.36
DER W ours
83.24
77.55
74.89
72.84
71.23
68.57
66.12
64.97
63.76
62.98
70.62
B.3. Combine With Vision Transformer Based Methods
In recent years, the Vision Transformer (ViT) (Dosovitskiy, 2020) has garnered significant attention within the computer vision community, with many approaches to Class- Innsemnncreettal Learning (CII.) being based on ViT. DyTox (Douillard et al., 2022) is pioneering work that explores the use of ViT in CIL, which dynamically expands Transformer tokens allocate independent representations for each task, effectively addressing catastrophic forgetting. Here, combine method with Box, and the results shown in Tab. 10. As demonstrated, our method consistently improves Dy Tox performance across all settings, achieving greater improvements compared to the ViT-based efficient replay method MAE-CIL (Zhai et al., 2023).
Table 10. Results CIFAR-100 100 in average accuracy (%) and last accuracy (%) on 10., 20- and 50-task scenarios.
=10
N =20
N 550
Method
Avg
Last
Avg
Last
Avg
Last
Dytox MAE-CIL
77.10
64.53
76.57
62.44
75.45
58.76
79.12
68.40
78.76
65.22
76.95
63.12
Dytox w ours
80.21
70.56
81.41
70.68
80.51
66.80
B.4. Comparing with Other Compression Methods.
Supplementary to Tab. 6. To evaluate the effectiveness the TD-based compression method, we processed the training set of CIFAR-100 (Krizhevsky, 2009) with different compression methods and tested with the original test set. Specifically, for downsampling (Zhao et al., 2021), can implemented using the resize() function, "2.56" and "4" represent the different downsampling ratio. For CIM (Luo al., 2023), we did take a dynamic determination of the activation region, We directly selected 16*16 region in the centre and down-sampled the rest, settling the downsampling ratio to 4. More detailed results are shown in Tab. 11.
Down-Sample
TDMER
Method
Upper Bound
CIM
2.56
4.0
10
12
16
20
Accu.
72.3
44.1
33.1
66.9
66.1
67.8
69.9
70.8
Comp.
1.00
0.39
0.25
0.56
0.22
0.26
0.35
0.44
Table 11, Classification accuracy on CIFAR-100 processed with different compression methods. "Accu." means the accuracy (%), "Comp." means the compressed ratio.
B.5. Reconstructed Image visualization
Fig. 4 illustrates 15 randomly selected images from the original dataset (ImageNet- 100) and the reconstructed images under different rank parameters.
14

---

Tensor Decomposition Based Memory-Efficiee Incremental Learning
Original Image.
(b) Reconstructed Image, A = 80.
(c) Reconstructed Image, R 100.
Figure 4. ImageNet- 100 image visualization. (a) Randomly selected 15 original images. (b), (c) Showing the reconstructed image at CP rank R 80 and R 100, respectively.
15


Reviews:
[{"review_id": "RJwIVwuipU", "reviewer": "Reviewer_3VFZ", "summary": "This paper introduces a novel approach to Class-Incremental Learning (CIL) that addresses memory efficiency challenges in replay-based methods. By employing tensor decomposition techniques instead of traditional pixel-level compression, the method exploits the low intrinsic dimensionality and pixel correlations in images to achieve better compression while preserving critical discriminative information. Combined with a hybrid exemplar selection strategy that enhances representativeness and diversity, the approach significantly improves upon baseline methods across multiple datasets of varying resolutions, demonstrating robust generalization capabilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions. Below, we provide a point-by-point response to the comments raised.\n\n### **1. Inconsistent terminology for evaluation metrics**\n\nIn the revised manuscript, we will align our metric naming with the community standards.**\"Average Incremental Accuracy (AIA)\"** will be replaced with **\"Average Accuracy \"**, and **\"Average Accuracy \"** will be renamed to  **\"Last Accuracy''**\n\n### **2. More comprehensive comparisons and analysis for efficient replay methods**\n\nWe thank the reviewer for highlighting the importance of comparative analysis. The initial manuscript has provided the comparison results in Tab. 5 and Tab. 9, and all the results demonstrate our method's superiority. Here, we offer more comparison results (10-tasks) and analysis.\n| Method | M=2k  | M=1k  |\n| :- | -| -|\n| MRDC   | 76.02 | 72.78 |\n| CIM    | 77.94 | 73.91 |\n| Ours   | 79.75 | 74.95 |\n\nAs our introduction and related work noted, pixel-level compression methods (e.g., MRDC[1], CIM[2]) directly compress images in the high-dimensional pixel space, often neglecting the low intrinsic dimensionality and local correlations inherent to natural images. This oversight leads to a significant loss of discriminative information.\n\nIn contrast, our Tensor Decomposition (TD)-based method explicitly leverages these properties by factorizing images into low-rank components. This not only achieves lower storage complexity (e.g., a compression ratio of **0.34** vs. CIM’s **0.56** in Tab. 6) but also preserves more discriminative information through high-fidelity reconstruction. As shown in Tab. 6, training on TD-compressed data achieves **69.9% accuracy** on CIFAR-100, significantly closer to the upper bound (**72.3%**) than pixel-level methods like downsampling (**44.1%**) or CIM (**66.9%**). This demonstrates TD’s ability to retain essential information while drastically reducing memory costs. \n\nThe superiority of our method stems from two key factors:\n\n1. **Efficient compression**: TD captures multi-dimensional correlations (spatial, channel-wise) in images, preserving more discriminative information while keeping a great compression ratio, avoiding the \"brittle\" compression of pixel-level methods.\n2. **Adaptability**: Unlike methods reliant on fixed heuristics (e.g., CIM’s center cropping), TD flexibly adapts to varying resolutions and dataset complexities, as evidenced by consistent gains across CIFAR-100, Tiny-ImageNet, and ImageNet-100 (Tab. 1, 3, 8).\n\nThese revisions will solidify our method’s advantages over existing efficient replay techniques, and we hope this can address the reviewer’s concern for deeper comparative insights.\n\n### **3.Combing with online continual learning**\n\nThis is an excellent suggestion. While our current focus is on Class-Incremental Learning, we recognize the importance of online settings. In the revised manuscript, we will add a discussion about the applicability of our method to online continual learning and outline plans for future work in this direction.\n\n### **4. Effect of varying ranks and $\\epsilon$ on reconstruction quality**\n\nRegarding the reconstruction quality, which depends on the CP rank $R$, we provide some results on CIFAR-100(evaluated by mean **r**elative **s**quared **e**rror). In the table, ''comp'' means compression ratio, and ''accuracy'' represents the classification accuracy after finishing offline training on compressed data. About the effect of $\\epsilon$, in the initial manuscript, Fig. 3 has shown its influence on final performance.\n\n| rank     | 10     | 12     | 14     | 16     | 18     | 20     |\n| -------- | ------ | ------ | ------ | ------ | ------ | ------ |\n| rse      | 0.0613 | 0.0472 | 0.0402 | 0.0343 | 0.0276 | 0.0243 |\n| comp     | 0.22   | 0.26   | 0.31   | 0.35   | 0.39   | 0.44   |\n| accuracy | 66.11  | 67.84  | 69.26  | 69.99  | 70.37  | 70.81  |\n\n### **5. Lacks visualization of reconstructed images**\n\nWe will provide figures showcasing the original images and their corresponding reconstructed images at different ranks in the next version.\n\n[1]Memory replay with data compression for continual learning\n[2]Class-incremental exemplar compression for class-incremental learning", "claims_and_evidence": "The claims made in the paper are basically convincing.", "methods_and_evaluation_criteria": "A minor issue: I find the terminology for evaluation metrics in this paper somewhat confusing, as it deviates from standard naming conventions in continual learning literature. Specifically, what this paper refers to as \"average incremental accuracy (AIA)\" is typically called \"average accuracy\" in most publications, while what the authors term \"average accuracy\" generally corresponds to what the field commonly refers to as \"final accuracy\" or \"last accuracy\" (the performance after learning all incremental tasks). This inconsistency in metric naming might create confusion for readers familiar with the established terminology in continual learning research.", "theoretical_claims": "No theoretical claims included in this paper.", "experimental_designs_or_analyses": "1. The experimental design is fundamentally sound. However, the paper would benefit from more comprehensive comparisons between the proposed approach and other efficient replay methods in the field. I would particularly encourage the continual learning community to focus their attention on the results presented in Table 5, which offer more meaningful insights than those in Tables 1 and 3. This comparative analysis would provide a clearer understanding of the method's relative advantages within the broader context of efficient replay techniques.\n\n2. It's suggested that authors may include some online continual settings in the experimental part since in this area, the memory efficiency may be more important.", "relation_to_broader_scientific_literature": "Maybe some literatures in online continual learning can be cited but it's not a necessity.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths**\n1. This paper proposes a remarkably straightforward yet highly intuitive method for memory-efficient continual learning. The manuscript is well-structured with clear organization, making it easy to comprehend. Additionally, the proposed methodology is presented in a manner that facilitates straightforward implementation or replication.\n2. As the authors note, this represents the first application of tensor decomposition techniques in this specific sub-area of continual learning (which aligns with my understanding of the current literature).\n\n**Weaknesses**\n1. See the previous parts.\n2. The primary limitation of this paper is the insufficient integration of tensor decomposition techniques with the unique challenges of continual learning. Furthermore, as mentioned earlier, the comparative analysis against other memory-efficient replay methods is somewhat lacking.\n3. The paper would benefit from additional visualizations of reconstructed images, particularly demonstrating the effects of varying rank and $\\epsilon$ parameters on reconstruction quality.", "other_comments_or_suggestions": "No other specific suggestions.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CvtHUCtmoV", "reviewer": "Reviewer_qf47", "summary": "This paper applied tensor decomposition on the replay-based continual learning methods. To minimize the influence of the reconstruction error on the training, the reconstructed images with low reconstruction error are selected for storage. The method is validated combined with other replay-based methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful evaluation and constructive feedback. Below, we address each concern raised and outline revisions to strengthen the manuscript.\n\n### **1. Additional experiments are needed to validate the importance of incorporating reconstructed samples into training**\n\nIn experiments, we observed that pure replay-based methods, such as iCaRL and FOSTER, achieved strong performance even without incorporating reconstructed samples during training. This is primarily because they operate on a fixed network structure and employ strategies like knowledge distillation to mitigate forgetting. \n\nHowever, methods like DER and MEMO adopt a dynamical model structure. Their primary mechanism for combating forgetting is freezing previously trained sub-networks, although they do retain some old samples for replay. Nevertheless, their reliance on sub-network freezing is more significant. \n\nTo illustrate, when training on the first task, the corresponding sub-network lacks the ability to recognize reconstructed samples unless they are included in the training.  When the second task arrives, although some reconstructed samples from previous tasks are preserved and added to training, relying solely on this limited number of old samples does not ensure enough generalization, and there may be some contradiction between the outputs of these two networks. Therefore, for methods like this, not incorporating reconstructed samples during training leads to a drastic decrease in performance. \n\nHere, we report model performance under both scenarios in the table below(10-task,$M = 2k$, $R = 12$, $\\epsilon = 0.4$). For consistency, we choose to include reconstructed samples in training across all cases.\n\n| Scenario  | iCaRL | FOSTER | MEMO  | DER   |\n| - | - | - | -| -|\n| w Reconstructed Sample   | 67.32 | 70.34  | 71.88 | 72.43 |\n| w/o Reconstructed Sample | 67.67 | 70.09  | 63.19 | 64.37 |\n\n### **2.  Setting an appropriate reconstruction error threshold and a decomposed tensor rank can be challenging**\n\nWe have provided the ablation experiments of Rank $R$ in Fig. 3 in the manuscript, and the results show that our method always produces positive results when $R \\in [\\frac{H}{3}, \\frac{H}{2}]$.\n\nAs for the reconstruction error threshold $\\tau$, which is primarily introduced to filter out samples with failed decompositions (rarely occur), which typically exhibit reconstruction errors exceeding 0.1. Indeed, for RGB images, our experimental results show that the reconstruction error is minor when $R \\in [\\frac{H}{3}, \\frac{H}{2}]$, essentially no noisier samples are introduced.  For instance, on CIFAR-100, when using a CP rank $R = 12$, the average relative squared error (RSE) is already below 0.05. Increasing $R$ to 16 further reduces the average RSE to 0.034. Under a memory budget of $M = 2k$ and $\\epsilon = 0.1$, each class retains 2 raw images and approximately 50 sets of decomposition factors(only 10% of the total data). For a threshold $\\tau$ of around 0.07, its effect on the final result is negligible.  \n\n### **3.Experimental results on ImageNet-R**\nHere we provide experimental results on ImageNet-R under 5, 10, and 20-task settings; we set $M = 2k$, $\\epsilon = 0.4$, $R = 80$. It can be seen that our approach also responds effectively in this scenario.\n\n| Method   | Base 0 Inc10 | Base 0 Inc20 | Base 0 Inc40 |\n| -| - | -| - |\n| MEMO        | 46.93        | 50.73        | 51.37        |\n| MEMO w/ours | 53.21        | 54.56        | 55.82        |\n| DER         | 47.77        | 51.95        | 52.61        |\n| DER w/ours  | 55.84        | 56.02        | 57.31        |\n\n### **4. Combining with recent replay-based methods**\n\nHere, we provide experimental results (10-task) on CIFAR-100 with our method integrating into **MRFA**[1], we set $M = 2k$,$\\epsilon = 0.1$, $R = 16$, we report **Average Accuracy (AA)** and **Last Accuracy (Last)**. It can be seen that our method also provides positive gains.\n\n| Method      | AA    | Last  |\n| ----------- | ----- | ----- |\n| MRFA        | 76.23 | 63.80 |\n| MRFA w/ours | 79.83 | 69.88 |\n\n### **5. Lacks visualization of reconstructed images**\nWe will provide figures showcasing the original image and its corresponding reconstructed image at different ranks in the revised version.\n\n[1] Multi-layer Rehearsal Feature Augmentation for Class-Incremental Learning, ICML 2024", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No issues are found.", "experimental_designs_or_analyses": "No issues are found.", "relation_to_broader_scientific_literature": "Previous methods have increased storage capacity by utilizing low-quality JPEG compression, reconstructing images from partial raw patches, or encoding image information within trained parameters. In contrast, this paper proposes leveraging tensor decomposition to store some samples in tensor form, offering a more efficient approach.", "essential_references_not_discussed": "No issues are found.", "other_strengths_and_weaknesses": "Strengths: This study finds that applying tensor decomposition (TD) to a subset of stored samples and preserving them in tensor form not only enhances storage efficiency but also improves model performance. To mitigate the potential adverse effects of reconstructed samples on the model, this paper incorporates them into the training process, enabling the model to be more robust to reconstruction errors. Furthermore, this paper validates the proposed approach by integrating it with various replay-based methods across different backbone architectures, demonstrating its effectiveness and broad applicability.\n\nWeaknesses: \nIn practical applications, setting an appropriate reconstruction error threshold and a decomposed tensor rank can be challenging when dealing with images of varying resolutions.", "other_comments_or_suggestions": "1. Additional experiments are needed to validate the importance of incorporating reconstructed samples into training. It would be insightful to examine how performance degrades if these samples are excluded from training.\n\n2. The paper lacks an ablation study on the reconstruction error threshold. If the domain gap between tasks is too large, the method may introduce noisier samples, potentially affecting the model’s overall performance. Moreover, for datasets with significant variations in image resolution, such as ImageNet-R, setting an appropriate threshold and ranks may be more challenging.\n\n3. It is necessary to validate the method on replay-based methods from the recent two years.\n\n4. Providing visualizations of reconstructed samples and comparing them with those from other methods would strengthen the empirical evidence and enhance the credibility of the paper.", "questions_for_authors": "If the effectiveness of the proposed method can be validated on datasets with greater resolution variations and recent replay-based methods, I would reconsider my evaluation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "n/a"}}, {"review_id": "X634RD5Aoo", "reviewer": "Reviewer_gCZN", "summary": "The paper addresses the challenge of catastrophic forgetting in Class-Incremental Learning (CIL), where models struggle to retain previous knowledge when incrementally learning new classes. While replay-based methods mitigate this by storing old exemplars, their high memory consumption limits practicality. Existing memory-efficient approaches using pixel-level compression face trade-offs between compression efficiency and retaining discriminative information. To overcome this, the authors propose a novel method leveraging low-rank tensor decomposition (TD) to exploit natural images' low intrinsic dimensionality and spatial correlations, achieving high compression while preserving critical features. Experiments on classic CIL datasets validate the method’s effectiveness.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's constructive feedback and valuable suggestions. Below, we address the concerns raised.\n\n### **1. Failing to explicitly report the actual memory costs**\n\nIn our manuscript, we have provided parameter configurations of different datasets in Tab. 7, e.g., for CIFAR, $R = 12$, $\\epsilon = 0.2$, when $M = 2k$, according to ep.3 compression rate $\\eta \\approx 0.26$, which means we saved 400 original samples and about 6153 sets of sample factors. We will elaborate on this in the next release.\n\n### **2. Lacks comparative experiments under fixed memory of varying sizes**\n\nWe have indeed provided comparative experiments under fixed memory of varying sizes (see Tab. 2 and Tab. 4 for two different fixed memories) in our manuscript.   We can add more explanations for clarity.\n\n### **3. Computational latency of TD for image compression/decompression**\n\nOn a single GPU (NVIDIA 3090), decomposing a CIFAR-100 image takes approximately 17 ms, while reconstruction takes approximately 2 ms, since the decomposition can be done in parallel with the training process, this part of the delay is negligible. The main computational delay is caused by the inclusion of reconstructed samples in the training, which requires about 40% extra computation for DER and MEMO. Furthermore, as we pointed out in our response to reviewer qf47, the extra calculations for incorporating reconstructed samples during training are not necessary for iCaRL and FOSTER.\n\n### **4. Lacks experiments on ImageNet-1k**\n\nWhile ImageNet-1k is a valuable benchmark, our current experiments on high-resolution datasets (e.g., ImageNet-100) and complex scenarios (e.g., 200-class Tiny-ImageNet in Table 8) already demonstrate the scalability and robustness of our method. Specifically:\n\n1. **ImageNet-100**: As a standard high-resolution benchmark in CIL literature, our method achieves consistent improvements (e.g., **+6.42%** AA for DER in Table 3), validating its effectiveness under realistic settings.\n2. **Tiny-ImageNet**: With 200 classes and 64×64 resolution, this dataset mimics the complexity of large-scale tasks. Our method boosts DER’s accuracy by **9.53%** (Table 8), illustrating strong generalization.\n3. **Community Practices**: For ImageNet-1k, prior works (e.g., iCaRL, DER) typically adopt a memory budget of 20k and 10-/20-task splits, which aligns with our experimental protocols. While computational constraints limited direct validation on ImageNet-1k, the consistent gains across varying resolutions and class numbers (ImageNet-100 to Tiny-ImageNet) suggest scalability to larger datasets.\n\nWe acknowledge the value of ImageNet-1k experiments and will provide results as time permits. For now, the results on ImageNet-R (provided in response to qf47) further corroborate our method’s adaptability to domain-shifted and large-scale scenarios.\n\n### **5.Other questions for authors**\nWe acknowledge our oversight and clarify it here. CP is the abbreviation of \"CANDECOMP/PARAFAC.\"[1]. We will ensure terms are properly introduced when first mentioned and provide visualizations of reconstructed samples in the revised manuscript.\n\n[1]Tensor decompositions and applications.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Yes", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n1. The proposed method mainly focuses on efficient memory represent, enabling direct integration into existing CIL methods.\n2. Experiments on CIL datasets (CIFAR-100 and ImageNet-100) validate that the proposed approach can improve the performance of previous baselines such as MEMO and DER.\n\nWeakness\n1. This paper focuses on memory-efficient CIL, but it just conduct experiments under different memory budgets (the method presented in this paper uses both stored real data and tensor components), failing to explicitly report the actual memory costs of compared methods or quantify the additional memory overhead introduced by the proposed approach.\n\n2. There is a lack of comparative experiments and analysis of different methods under fixed memory of varying sizes, which are critical for evaluating the memory efficiency of different methods.\n\n3. The method includes exemplar selection strategy to select high quality of reconstruction during training, while the computational latency of TD for image compression/decompression during exemplar storage and rehearsal is not discussed.\n\n3. The experimental results on ImageNet-1k is missing, which is important to demonstrate the effectiveness on large-scale case.", "other_comments_or_suggestions": "No", "questions_for_authors": "Questions：\n1. Could you provide qualitative examples of TD-reconstructed images under different hyperparameter settings?\n2. What does 'CP' mean, as it appears in line 135 for the first time without explaination.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8OuS0H2Eae", "reviewer": "Reviewer_UizA", "summary": "This paper presents a new memory-efficient method for CIL. Different from previous papers, tensor decomposition is used to compress original image. Besides, a new exemplar selection strategy is proposed to ignore the influence of negative compressed samples. Extensive experiments on different datasets demonstrate this method's robustness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's thoughtful feedback and constructive suggestions. Below, we address the concerns raised. \n\n### **1. Lacks ablation experiments for the proposed method**\n\nIn our manuscript, we have conducted experiments (Section 4.2, Fig. 3, and Tab. 6) to evaluate the impact of key components, e.g., decomposition rank $R$, proportion $\\epsilon$, as shown in Fig.3 when $\\epsilon = 0$, we only use reconstructed samples without the two-stage selection strategy. Tab. 6 demonstrated the effect of different compression methods on performance.\n\n### **2.Specific experimental setups in Tab. 1 and Tab. 3**\n\nWe acknowledge this oversight. For the experimental setups in Tab. 1 and Tab. 3, we set the memory budget to 2k and explained other settings in the section \"Protocols.\" \n\n### **3. Performance discrepancy on iCaRl and DER**\nFor the performance discrepancy in DER, since they chose resnet18 as the backbone network for CIFAR(for resnet32, their results are close to ours), we used more lightweight resnet32 (that's what most do.). Thus, there is a performance gap; for ImageNet, we use the same backbone network with similar performance. For iCaRL, we have double-checked it without any changes. \n\n### **4. Lacks comparison results with memory-efficient methods**\nIn our manuscript, we have provided comparison results with some recent memory-efficient replay methods in Tab. 5 and Tab. 9. All the results demonstrate our method's superiority.\n\n### **5.Exlaination of Herding Strategy**\n\nFor exemplar selection, we clarify that herding is employed independently within each class, ensuring that selected samples best represent the class distribution. While herding traditionally minimizes the difference between selected samples and the overall dataset, its application per class in our method aligns with class-incremental learning settings. We will further clarify this in the manuscript.\n\nWe deeply appreciate your rigorous review and constructive feedback. All suggested revisions will be incorporated to strengthen the manuscript’s clarity, technical depth, and experimental validation. Thank you again for your time and consideration.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "No.\nInadequate Experiments: The article lacks ablation experiments for the proposed method. As I understand, there are two main innovations in the article: tensor decomposition and a new selection strategy. Providing ablation experiments similar to these would help prove the effectiveness of both components.", "theoretical_claims": "Yes.\nQuestion: The paper uses herding to select a subset of samples, but my understanding is that the criterion for herding is to minimize the difference between the selected samples and the overall dataset. In incremental learning, however, there are data of different class categories. Is it reasonable to use the herding selection strategy for samples of different classes?", "experimental_designs_or_analyses": "Yes.\nAccuracy of the experimental results: \n1.What are the specific experimental setups in Tab1 and Tab3? For methods such as iCaRL and DER, were they kept to their original results with 2k samples, or were there any modifications made to the methods? The reproduced results of iCaRL and DER by PyCIL should not be this low, and there is a significant discrepancy compared to the results reported in the original paper.[1][2][3][4]\n2.For memory-efficient methods, there are many other approaches. The article should compare and highlight the advantages of the proposed method over other existing methods (this can be done if time permits and is not intended to be a part of the final scoring criteria).[5]\n\n[1] DER: Dynamically Expandable Representation for Class Incremental Learning\n[2] Multi-layer Rehearsal Feature Augmentation for Class-Incremental Learning\n[3] Dynamic Residual Classifier for Class incremental Learning\n[4] FOSTER: Feature Boosting and Compression for  Class-Incremental Learning\n[5] A MODEL OR 603 EXEMPLARS: TOWARDS MEMORY EFFICIENT CLASS-INCREMENTAL LEARNING", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "The paper uses herding to select a subset of samples, but my understanding is that the criterion for herding is to minimize the difference between the selected samples and the overall dataset. In incremental learning, however, there are data of different class categories. Is it reasonable to use the herding selection strategy for samples of different classes?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qn1iZ84gKD", "reviewer": "Reviewer_8qn7", "summary": "This paper addresses the challenge of memory efficiency in Class-Incremental Learning (CIL), whose goal is continuously learning new classes over time. Replay-based methods, a prominent approach in CIL, suffer from high memory consumption due to the need to store past exemplars. To mitigate this, the authors propose a novel method that leverages Tensor Decomposition (TD) to compress images efficiently, thereby reducing memory footprint while preserving sufficient discriminative information. Furthermore, they introduce a two-stage exemplar selection strategy to enhance the representativeness and diversity of the stored samples. In the first stage, herding is used to sample exemplars most representative of the class characteristics, focusing on central, high-quality samples. The second stage then samples from the remaining, unselected data, prioritizing samples with low reconstruction error after tensor decomposition. This aims to select diverse and noise-robust exemplars. Experimental results on two image datasets demonstrate that the proposed sampling strategy improves the performance of several existing CIL techniques.\n\n## update after rebuttal\n\nThank you for your response. Your rebuttal has addressed the two questions to a reasonable extent, so I will revise my rating. However, having also reviewed the comments from the other reviewers, I believe their concerns are valid as well. Therefore, I do not intend to strongly oppose their opinions. I mistakenly posted my comment in the Official Comment section. My apologies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's insightful feedback and constructive suggestions. Below, we address the key concerns raised.\n\n### **1. Why diversity can be ensured by the second-stage sampling and its contribution to performance**\n\nThe paper proposes a novel exemplar selection strategy that enhances representativeness and diversity. Specifically, In the first stage, we use herding to select a small, equal number of representative original samples per class, prioritizing high-quality exemplars. In the second stage, we choose similarly an equal number of sample factors per class, focusing on sample quantity. The diversity of the selected exemplars is primarily ensured through the more stored samples and the uniform sampling strategy.\n\nAbout the diversity's contribution to performance, as shown in Fig. 3, when $\\epsilon$ remains within a small range, meaning a sufficient number of samples are retained in the second stage, this strategy consistently yields great performance improvements. However, as $\\epsilon$ increases further(means fewer samples are stored and lower diversity ), the model’s final performance gradually gets close to the original. For MEMO, when $\\epsilon$ exceeds 0.6, the performance gain on CIFAR remains below 1%.\n\n### **2.  Can you theoretically or experimentally demonstrate that actively selecting samples with low reconstruction error after tensor decomposition contributes to performance improvements?**\n\nYes, we can. Here, we provide results (Average Accuracy) for the sample selection strategy prioritized by the minimum reconstruction error (M= 2k, 10-task); note that we only use reconstructed samples. It can be seen that AA gradually improves as the reconstruction error decreases. \n\n| Method      | R = 10 ( rse 0.061) | R = 12 (rse 0.047) | R = 16 (rse 0.034) | R = 20 (rse 0.024) |\n| ----------- | :-----------------: | :----------------: | :----------------: | :----------------: |\n| DER w/ours  |        71.94        |       72.43        |       72.91        |      73.21       |\n| MEMO w/ours |        70.80        |       71.88        |       71.91        |    72.20        |\n\n### **3.Clarification of notation in the exemplar selection strategy**\n\nWe will revise the description of the Exemplar Selection Strategy to distinguish samples selected in each stage.", "claims_and_evidence": "The selection of central exemplars using herding is reasonable and well-justified. However, while sampling from the remaining data (i.e., those not selected by herding) can be expected to provide some degree of diversity, the extent to which this random sampling strategy ensures diversity is not demonstrated. Furthermore, while a low reconstruction error after Tensor Decomposition indicates that the original image features are well-preserved, it does not necessarily guarantee robustness to noise.", "methods_and_evaluation_criteria": "It's unclear whether random sampling adequately ensures diversity, and the evaluation doesn't assess how much diversity contributes to performance gains. Similarly, it's uncertain whether tensor decomposition ensures robustness, and the evaluation doesn't assess this aspect either.", "theoretical_claims": "The paper primarily discusses the memory efficiency of the proposed method, centered around Equation (3), which makes sense. However, there is a lack of theoretical analysis regarding the diversity ensured by random sampling and the robustness ensured by tensor decomposition.", "experimental_designs_or_analyses": "The paper demonstrates the effectiveness of the proposed method empirically through evaluation experiments using two different datasets and several key existing CIL techniques, which is good. However, as mentioned in the Methods and Evaluation Criteria section, the evaluation does not assess the extent to which the performance improvements are attributable to diversity, nor does it evaluate the robustness ensured by tensor decomposition.", "relation_to_broader_scientific_literature": "By demonstrating the potential of tensor decomposition to enhance CIL performance through robustness, this work offers a novel perspective on the application of tensor decomposition's noise resilience.", "essential_references_not_discussed": "The contribution of random sampling to improved predictive performance in machine learning appears to be a central idea in the proposed method. Discussing and citing relevant prior work on this topic could enhance the persuasiveness of the paper. For example, mentioning Random Forests [Breiman '01] would be one possibility.\n\nBreiman, L. Random Forests. Machine Learning 45, 5-32 (2001).", "other_strengths_and_weaknesses": "As mentioned above, this paper offers a novel perspective on the application of tensor decomposition's noise resilience. However, the persuasiveness of the proposed method could be enhanced by providing a more detailed discussion or evaluation experiments regarding the extent to which the performance improvements are attributable to diversity and the degree to which robustness is ensured by tensor decomposition.", "other_comments_or_suggestions": "There are several instances where the writing is potentially confusing and could lead to misinterpretations. For example, in the Exemplar Selection Strategy section, the description of the second stage (line 263) uses the same notation for the samples (e.g., x_1^t) as in the first stage, and it only specifies that the number of selected samples, j, is different from i. This could mistakenly suggest that the samples are not necessarily different from those selected in the first stage. The authors should refine the expression to convey that the samples are distinct from those selected in the first stage.", "questions_for_authors": "1. Can you demonstrate how the diversity ensured by the second-stage random sampling contributes to performance improvements? Experimentally showing the diversity of the sample distribution within a class might be one approach.\n\n2. Can you theoretically or experimentally demonstrate that actively selecting samples with low reconstruction error after tensor decomposition contributes to performance improvements?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors proposed to use tensor decomposition for memory-efficient incremental learning. The paper was reviewed by five expert reviewers, and three of them recommended accepting this paper. I think this is, overall, an interesting paper with solid results. The proposed method is simple yet effective. Therefore, I tend to accept this paper. There are also some remaining concerns, e.g., missing detailed memory information and experiments on ImageNet-1k. The authors should fix all these problems before the final camera-ready version.
Decision: Accept (poster)