A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning
Authors: Da-Wei Zhou
Qi-Wei Wang
Han-Jia Ye
De-Chuan Zhan
Venue: ICLR 2023 notable top 25% (2023)
Keywords: class-incremental learning

Abstract: Real-world applications require the classification model to adapt to new classes without forgetting old ones. Correspondingly, Class-Incremental Learning (CIL) aims to train a model with limited memory size to meet this requirement. Typical CIL methods tend to save representative exemplars from former classes to resist forgetting, while recent works find that storing models from history can substantially boost the performance. However, the stored models are not counted into the memory budget, which implicitly results in unfair comparisons. We find that when counting the model size into the total budget and comparing methods with aligned memory size, saving models do not consistently work, especially for the case with limited memory budgets. As a result, we need to holistically evaluate different CIL methods at different memory scales and simultaneously consider accuracy and memory size for measurement. On the other hand, we dive deeply into the construction of the memory buffer for memory efficiency. By analyzing the effect of different layers in the network, we find that shallow and deep layers have different characteristics in CIL. Motivated by this, we propose a simple yet effective baseline, denoted as MEMO for Memory-efficient Expandable MOdel. MEMO extends specialized layers based on the shared generalized representations, efficiently extracting diverse representations with modest cost and maintaining representative exemplars. Extensive experiments on benchmark datasets validate MEMO's competitive performance. Code is available at: https://github.com/wangkiw/ICLR23-MEMO

Published as a conference paper at ICLR 2023
A MODEL OR 603 EXEMPLARS: TOWARDS MEMORY- EFFICIENT CLASS-INCREMENTE LEARNING
Da-Wei Zhou, Qi-Wei Wang, Han-Jia Ye; De-Chuan Zhan State Key Laboratory for Novel Software Technology, Nanjing University (zhoudw, wanggiwei, yehj, zhande) è amda. aoju.edu. cn
ABSTRACT
Real-world applications require the classification model to adapt new classes without forgetting old ones. Correspondingly, aims train model with limited memory size meet this requirement. Typ- ical CIL methods tend to save representative exemplars from former classes to resist forgetting, while recent works find that storing models from history can substantially boost the performance. However, the stored models are not counted into the memory budget, which implicitly results in unfair comparisons. We find that when counting the model size into the total budget and comparing methods with aligned memory size, saving models do not consistently work, especially for the case with limited memory budgets. As result, need to holistically evaluate different CIL methods at different memory scales and simultaneously consider accuracy and memory size for measurement. On the other hand, we dive deeply into the construction of the memory buffer for memory efficiency. By analyzing the effect of different layers in the network, we find that shallow and deep layers have different characteristics in CIL. Motivated by this, we pro- pose a simple yer effective baseline, denoted as MEMO for Memory-efficien Expandable MOdel. MEMO extends specialized layers based on the shared gener- alized representations, efficiently extracting diverse representations with modest cost and maintaining representative exemplars. Extensive experiments on bench- mark datasets validate MEMO'S competitive performance. Code is available at: attps://gite com/ http://annkiwwIIIIS ICWp//CCRR33- MEMO
Learning (CIL)
INTRODUCTION
In the open world, training data is often collected in stream format with new classes appearing (Gomes
al. 20177 Geng et al.) 20200) Due to storage constraints (Krempl et al. 2014 Gaber 2012 or privacy issues (Chamikara et al. 2018 Ning et al. (02T), a practical Class- Incremental Learning (CIL) (Reuuffi et al. | (017) model requires the ability to update with incoming instances from new classes without revisiting former data. The absence of previous training data results in catastrophic forgetting (French) 1999) in CII. - fitting the pattern of new classes will erase that of old ones and
result in a performance decline. The research about CIL. has attracted much interest (Zhou et al.
20216 2022a by Liu et al. 2021b 2023 Zhao et al. 2021a b) in the machine learning field.
Saving all the streaming data for offline training is known as the performance upper bound of CIL algorithms, while it requires an unlimited memory budget for storage. Hence, in the early years, CIL
algorithms designed in striet setting without retaining any instances from the former classes (Li & Hoiem| 2017 Kirkpatrick et al.] 20077; Aljjnni et al., 2017]] Lee et al., 2017). It only keeps a classification model in the memory, which helps save the memory budget and meanwhile preserves privacy in the deployment. Afterward, some works noticed that saving limited exemplars from former
classes can boost the performance of CIL models (Rebuffi et al. 2017 Chaudhry et al.) (019). Various exemplar-bar based methodologies have been proposed, aiming to prevent forgetting by revisiting the old during new class learning, which improves the performance of CIL tasks steadily (Rolnick et al.) 2019) Castro et al. 2018 Wu et al.) 2019: Isele & Cosgun, 2018). The utilization of exemplars has drawn the attention of the community from the strict setting to update the model with restricted memory
"Correspondence Han-Jia Ye (ychj @ crchj@lamma.nju.dduu.

---

Published as a conference paper at ICLR 2023
70
Average Accuracy (%)
Average Accuracy (%)
70
65
Intersection
Intersection
60
60
50
DER WA
Replay iCaRL.
MEMO
DER WA
Replay iCaRL
MEMO
55
7.6
12.4
16.0
19.8
23.5
329
493
755
872
1180
1273
Memory Size (MB)
Memory Size (MB)
(a) CIFARI00, BaseO Inc10
(b) ImageNet100, Base50 Inc5
Figure 1: The average accuracy of different methods by varying memory size from small large. The start point corresponds to the memory size of exemplar- basel methods with benchmark backbone (WA (Zhao et al.) 2020), iCaRL (Rebuthi ct al. 2017), Replay (Chaudhry ct al. 2019)), and the endpoint corresponds to the memory cost of model-based methods with benchmark backbone (DER (Yan al., [202T) and MEMO (our proposed method)). We align the memory cost by using the small model for model-based methods or adding exemplars for exemplar-based methods. 'Base' stands the number of classes in the first task, and 'Inc' represents the number of classes in each incremental new task. See Section
and for more details.
size (Castro et al., 2018 Rebuffi et al. 2017). Rather than storing exemplars, recent works Yan al. 2021] Wang et al. 2022 Li et al. 2021 Douillard al., 202T) find that saving backbones from the history pushes the performance by one step towards the upper bound. These model-based methods propose to train multiple backbones continually and aggregate their representations as the feature representation for final prediction. Treating the backbones from history 'unforgettable checkpoints, this line of work suffers less forgetting with the help of these diverse representations.
Model-based CIL methods push the performance towards the upper bound, but does that mean catastrophic forgetting is solved? Taking a close look at these methods, we find that they implicitly introduce extra memory budget, namely model buffer for keeping old models. The additional buffer implicitly results in an unfair comparison to those methods without storing models. Take CIFAR100 (Krizhevsky et al. 2009) for example; if we exchange the model buffer of ResNet32 (He) et al. 2015) into exemplars of equal size and append them to iCaRL (Rebuffi et al.) 2017) (a baseline without retaining models), the average accuracy drastically improves from 62% to 70%. How fairly measure the performance of these methods remains long- standing problem since saving exemplars or models will both consume the memory budget. In this paper, we introduce an extra dimension to evaluate CII. methods by considering both incremental performance and memory cost. For those methods with different memory costs, we need align the performance measure at the same memory scale for a fair comparison.
How to fairly compare different methods? There are two primary sources of memory cost in CIL, i.e, exemplar and model buffer. We can align the memory cost by switching the size of extra backbones into extra exemplars for a fair comparison. For example, a ResNet32 model has the same memory size with 603 images for CIFAR 100, and 297 ImageNet (Deng al. 2009) images have the same memory size with a ResNet18 backbone. Figure sshows the fair comparison on benchmark datasets, e.g., CIFAR 100 and ImageNet100. We report the average accuracy of different models by varying the memory size from small to large. The memory size of the start point corresponds to the cost of an exemplar-based method with single backbone, and the endpoint denotes the cost of a model-based method with multiple backbones. As we can infer from these figures, there is an intersection between these methods saving models less effective when the total budget is limited while more effective when the total budget is ample.
In this paper, we dive deeply into the empirical evaluations of different CIL methods considering the incremental performance and memory budget. Towards fair comparison between different approaches, we propose several new measures that simultaneously consider performance and memory size, e.g., area under the performance performance-memory curve and accuracy per model size. On the other hand, how to organize the memory buffer efficiently so that we can save more exemplars and meanwhile maintain diverse representations? We analyze the effect of different layers of the network by counting the gradients and shifting range in incremental learning, and find that shallow layers tend to leam generalized features. By contrast, deep layers fit specialized features for corresponding tasks and yield very different characteristies from task to task. As a result, sharing the shallow layers and only creating deep layers for new tasks helps save the memory budget in CIL..
2

---

Published as a conference paper at ICLR 2023
Furthermore, the spared space can be exchanged for an equal number of exemplars to further boost the performance. Intuitively, we propose a simple yer effective baseline MEMO to simultancously consider extending diverse features with the most modest memory cost. MEMO shows competitive results against state-of-the-ar methods under the fair comparison on vast benchmark datasets and various settings, which obtains the best performance in most cases of Figure
2 RELATED WORK
We roughly divide current CIL methods into two groups, i.e., exemplar- and model-based methods. The former group seeks rehearse former knowledge when learning new, and the latter saves extra model components to assist incremental learning. Obviously, some methods do not fall
into these two groups (Kirkpatrick al. 2017 Li & Hoiem 2017 Jin al. 2021 Smith et al. 2021). and we refer the readers to (Zhou et al. 2023 for a holistic review.
Exemplar-Basee Methods: Exemplars are representative instances from former classes (Welling 2009), and CIL. models can selectively save a relatively small amount of exemplars for rehearsal during updating (Isele & Cosgun 2018). Like natural cognitive systems, rehearsal helps revisit former tasks to resist catastrophic forgetting (Parisi et al. (011)). Apart from direct replay, there are other methods addressing utilizing exemplars in CIL. iCaRL (Rebuffi et al., (2077) builds knowledge distillation (Zhou et al.) 2003 Zhou & Jiang [2004] Hinton
20115) regularization with exemplars to align the predictions of old and new models. On the other hand, (Lopez-Pat Papez-Paz& & Ranzato 2017) treats
the loss on the exemplar set as indicator of former tasks' performance and solves the quadratic program problem as regularization. (Wu et al., [019] Castro et al., (218)) utilize exemplars for balanced finetuning or bias correction. Note that exemplars can be directly saved or be generated with extra generative models (Shin et al. 22017b] He et al. 2018)), which indicates the equivalency between models and exemplars. Consistent with our intuition, there has been much research addressing that saving more exemplars will improve the performance of CIL models correspondingly (Iscen et al. 2020 Ahn et al. (02T).
Model-Based Methods: There are some methods that consider increasing model components incrementally to meet the requirements of new classes. (Liu et al.] (2021a) adds residual blocks as mask layers to balance stability and plasticity. (Ostapenko al. (022]) expands dynamic modules with dynamic routing to generalize to related tasks. (Yoon et al. (018)) creates new neurons to depict the features for new classes when needed, and (Xu & Zhu, 2018) formulates it reinforcement learning problem. Instead of expanding the neurons, some works (Serra et al. 2018: Rajasegaran et al. 2019| Abbai et al. 2020) propose to learn masks and optimize the task-specific sub-netword These methods increase a modest amount of parameters to be optimized. Recently, (Yan et al. 2021) addresses that aggregating the features by training a single backbone for each incremental task can substantially improve the performance. Since there could be numerous incremental tasks in CIL, saving backbone per task implicitly shifts the burden of storing exemplars into retaining models. Memory-Efficien CIL: Memory cost is an important factor when deploying models into real-world applications. (Iscen et al.) (020) addresses saving extracted features instead of raw images can help model learning. Similarly. (Zhao et al.] (021c) proposes to save low-fidelity exemplars to reduce the memory cost. (Smith et al. 202 Choi et al. (02T) release the burden of exemplars by data-free knowledge distillation (Lopes et al. 2017). To our knowledge, we are the first to address the memory-efficient problem in CIL from the model buffer perspective.
3 PRELIMINARIES
3.1 PROBLEM DEFINITION
Class-incremental learning was proposed to learn a stream of data continually with new classes (Re- buffi et al. 2017). Assume there are a sequence of B training tasks "D",DD,.... .. without
overlapping classes, where Dè = (x): } is the b-th incremental step with n, instances. x E IR is training instance of class yi E Yo. Y, is the label space of task b, where Y,nY, Ø for b b V. A fixed number of representative instances from the former classes are selected as exemplar only access
set E, |8| K is the fixed exemplar size. During the training process of task b, data from D and E. The aim of CIL at each step is not only to acquire the knowledge from the current task D but also to preserve the knowledge from former tasks. After each task, the trained
3

---

Published as a conference paper at ICLR 2023
model is evaluated over all seen classes 3% Y1 U... Yy. The incremental model is unaware of the task id, i.e., b, during inference. We decompose the model into the embedding module and linear layers, ie., f(x) WT ((x), where u(.) TR. Rd W E Rdx[D8)
3.2 OVERCOME FORGETTING IN MAAS--ICCRMMNTAL LEARNING
In this section, we separately introduce two baseline methods in CIL. The former baseline belongs to the exemplar-based method, while the latter belongs to the model-based approach.
Knowledge Distillation: To make the updated model still capable classifying the old class instances, common approach in CIL combines cross-entropy loss and knowledge distillation loss (Zhou et al.) 2003; Zhou & Jiang 2004; Hinton et al. 2013). builds a mapping between the former and the current model:
Dal
[(x, = =1 A1 Σ
1(y = log 6(x)) +1
S.(W 5(x)) log Sa(W ((x)), (1)
Cruss Entropp
Knowledge Distilaatonn
where 2/-1 = Y1 U.... 1-- denotes the set of old classes, À is trade-off parameter, and (-) denotes the k-th class probability after softmax operation. W and correspond frozen classifier and embedding before learning D Aligning the output of the old and current models helps maintain discriminability and resist forgetting. The model optimizes Eq. the current dataset and exemplar set DD U E. It depicts simple way to simultaneously consider learning new class and preserving old class knowledge, which is widely adopted exemplar-base methods (Rebuffi et al. 2017 Wu et al. Zhao et al. 2020).
2019
Restricted by the representation ability, methods with a single backbone cannot depiet the dynamic features of new classes. For example, if the first task contains 'tigers,' the CIL model will pay attention to the features to trace the beards and stripes. the next task contains 'birds,' the features will then be adjusted for beaks and feathers. Since single backbone can depict a limited number of features, learning and overwriting new features will undoubtedly trigger the forgetting of old ones. To this end, DER (Yan et al. (02T)) proposes to add backbone to depict the new features for new tasks. For example, in the second stage, it initializes new feature embedding Pnew IR D + dRd and freezes the old embedding Φold. also initializes a new linear layer Wnew |2dx|Mbl inherited from the linear layer Wata of the last stage, and optimizes the model with typical cross-entropy loss:
Feature Aggregation:
D&
L(x, v) = Σ
- Ily = k) log (W Toooooo Unew (x)]).
Similar Eq. the loss is optimized over D'UE, aiming to learn new classes and remember old ones. It also includes an auxiliary loss to differentiate the embeddings of old and new backbones, which will be discussed in the supplementary. Eq. sequentially optimizes the newest added backbone &new and fixes old ones. It can be seen as fitting the residual term to obtain diverse feature representations among all seen classes. Take the aforementioned scenario for an example. The model will first the features for beards and strides to capture tigers in the first task with daad. Afterward, it optimizes the features for beaks and feathers to recognize birds in the second task with &new- Training the new features will not harm the performance of old ones, and the model can obtain diverse feature representations as time goes by. However, since it creates new backbone per new task, it requires saving all the embeddings during inference and consumes much larger memory cost compared to exemplar- based methods.
4 ANALYSIS
4.1 EXPERIMENTAL. SETUP
As our paper is heavily driven by empirical observations, we first introduce the three main datasets experiment on, the neural network architectures we use, and the implementation details.
4

---

Published as a conference paper at ICLR 2023
Dataset: Following the benchmark setting (Rebuffi al. 2017: Wu et al. 2019). we evaluate the performance on CIFAR100 (Krizhevsky et 2009), and ImageNet100/ 1000 (Deng et al. 2009). CIFARIO0 contains 50,000 training and 10,000 testing images, with total of 100 classes. Each image is represented by 32 x 32 pixels. ImageNet is a large-scale dataset with 1,000 classes, with about 1.28 million images for training and 50,000 for validation. We also sample subset of 100 classes according to (Wu et al. 2019), denoted as ImageNet100.
Dataset Split: According to the common setting in CIL (Rebufli et al. 2017), the class order of training classes is shuffled with random seed 1993. There are two typical class splits in CIL. The former [Rebufti et al. 2017) equally divides all the classes into B stages. The latter (Hou et al.) 2019)
Yu et al. 2020) treats half of the total classes in the first stage (denoted as base classes) and equally divides the rest classes into the incremental stages. Without loss of generality, we use Base-r, Inc-y to represent the setting that treats x classes as base classes and learns y new classes per task. = 0 denotes the former setting.
Implementation Details: All models are deployed with PyTorch (Paszke et al., 2019) and Py- CIL (Zhou et al. 2021a) on NVIDIA 3090. If not specified otherwise, we use the same network backbone (Rebulfi et al. 2017) for all compared methods, i.e., ResNer32 (He et al.] 2013) for CI- FAR100 and ResNet18 for ImageNet. The model is trained with a batch size of 128 for 170 epochs, and we use SGD with momentum for optimization. The learning rate starts from 0.1 and decays by 0.1 at 80 and 150 epochs. The source code of MEMO will be made publicly available upon acceptance. We use the herding (Welling) 2009) algorithm select exemplars from each class.
4.2 HOw TO FAIRLY COMPARE CIL METHODS?
As discussed in Section 3.2 exemplar-based methods and model-based methods consume different memory sizes when learning CII. tasks. Aiming for fair comparison, we argue that these methods should be aligned to the same memory cost when comparing the results. Hence, we vary the total memory cost from small to large and compare different methods at these selected points. Figurel I shows corresponding results, which indicates the performance given different memory budget. Take Figure for an example; we first introduce how to set the start point and endpoint. The memory size of the start point corresponds to a benchmark backbone, i.e., ResNet32. The memory size of the endpoint corresponds to the model size of model-based methods using the same backbone, i.e., saving 10 ResNet32 backbones in this setting. For other points of the X-axis, we can easily extend exemplar-bar based methods to these memory sizes by adding exemplars of equal size. For example, saving a ResNet32 model costs 463, 504 parameters (float), while saving a CIFAR image costs 3 x 32 32 integer numbers (int). The budget of saving a backbone is equal to saving 463, 504 floats x4 bytes/float (3 x 32 32) bytes/image z 603 instances for CIFAR. We cannot use the same backbone as the exemplar-base based methods for model-based ones when the memory size small. Hence, we divide the model parameters into ten equal parts (since there are ten incremental tasks in this setting) and look for a backbone with similar parameter numbers. For example, we separately use ConvNet, ResNet14, ResNet20, and ResNet26 as the backbone for these methods to match different memory scales. Please refer to supplementary for more details.
Given these curves, there are two questions. First, what is a good performance measure consid- ering memory cost? We argue that a good CIL method with ex- tendability should work for any
WA
DER MEMO
foemge Aoemge-Accumao Accurayy (5))
88
Average Accuracy (16)
72-
KaRL
71-
-+- WA
ICURL
20
DER
memory cost. As result, is in-
MEMO
tuitive to measure the area under the curve (AUC) for these meth- ods for holistic overview. Ob- serving that model-based meth- ods work at large memory costs while failing at small costs in Fig- ure [ argue that measuring performance the start point and endpoint can also be representative measures. Secondly, given a specific memory cost, should we use a larger model or more ex- emplars? Denote the ratio of model size as p (aaeMMoooe))/Si(Tooee We vary this ratio
10
20
50
40
50
60
20
40
60
10
Model Mimec Size (%)
Model Mezur Size (1%)
Memory Size 12.4MB
(b) Mcmory Size 23.5MB
Figure Performance of different methods when fixing total budget and varying the ratio of model size to total memory size on CIFAR100.
5

---

Published as a conference paper at ICLR 2023
Shallow Festures
1.0
Mor Square Earor
Block Indox
Deop Reatures
3o-t
12-
04
1+3-
9-
1800
<200
5600
9000
1.5
02
Denston
Block Index
Gradient norm (log scalc)
(b) Shift of different blocks
(c) CKA between backbones
Figure 3: Left: gradient norm of different residual blocks when optimizing Eq. Deeper layers have larger
gradients. while shallow layers have small gradients. Middle: Shift between first and last epoch different residual blocks. Deeper layers change more, while shallow layers change less. Right: feature similarity (CKA) of different backbones learned by Eq. 2 The lower triangular matrix denotes the similarity between deeper layers; the upper triangular matrix denotes the similarity between shallow layers.
for exemplar-basee methods at different memory scales in Figure We switch ResNet32 to ResNer44/56/1 10 to enlarge p. Results indicate that the benefit from exemplars shall converge when larger model is memory-
the budget is large enough (i.e., Figure 2(b)), where switching efficient. However, this trend does not hold when the total memory budget is limited, i.e. Figure and there is no consistent rule in these figures. As result, report results with the benchmark backbone in Figure for consistency.
4.3 Do WE NEED A NEW BACKBONE PER TASK?
Aggregating the features from multiple stages can obtain diverse representations while sacrificing the memory size for storing backbones. From the memory--fficiemn perspective, wonder if all layers are equal in CIL-it if the storage of some layers is unnecessary, switching them for exemplars will be more memory-efficient for the final performance. In detail, analyze from three aspects, i.e., block-wise gradient, shift, and similarity on CIFAR with ResNet32. Experiments with other backbones (e.g.. ViT) and datasets (e.g.. NLP) can be found in Section C.10
Block-Wise Gradient: We first conduct experiments to analyze the gradient of different residual blocks when optimizing Eq. [ We show the gradient norm of different layers in a single task in Figure (a) A larger block index corresponds to deeper layers. It indicates that the gradients of shallow layers are much smaller than deep layers. As result, deeper layers shall face stronger adjustment within new tasks, while shallow layers tend to stay unchanged during CIL.
To quantitatively measure the adjustment of different layers, we calculate the mean square error (MSE) per block between the first and last epoch for every incremental stage in Figure (bb shows that the shift of deeper layers is much higher than shallow layers, indicating that deeper layers change more while shallow layers change less. It should be noted that MSE in the first task D calculated for the randomly initialized network, which shows different trends than others. These results consistent with the observations of gradients in Figure (3))
Block-Wise Shift:
Feature Similarity: Observations above imply the differences between shallow and deep layers in CII. We also train model with Eq. for 10 incremental stages, resulting in 10 backbones. We use centered kernel alignment (CKA) (Kornblith et al. 2019). an effective tool to measure the similarity network representations evaluate the relationships between these backbones. We can get corresponding feature maps by feeding the same batch of instances into these backbones. Afterward, CKA applied measure the similarity between these feature maps. We separately calculate the similarity for deep (i.e., residual block 15) and shallow features (i.e., residual block 5) and report the similarity matrix in Figure 3(c) The similarities between deep features are shown in the lower triangular matrix, and the similario similaiitiss between shallow features are shown in the upper triangular matrix. Results indicate that the features of shallow layers among all backbones are highly similar, while diverse for deeper layers.
To summarize, we empirically find that not all layers are equal in CIL, where shallow layers yield higher similarities than deeper layers. A possible reason is that shallow layers tend to provide general-purpot representations, whereas later layers specialize (Maennel et al. 2020 Ansuini et al. 2019 Arpit al. 20177] Yosinski et al. [014] [Zhang al. 2019). Hence, expanding general layers would be less effective since they are highly similar. On the other hand, expanding and saving the specialized features is essential, which helps extract diverse representations continually.

---

Published as a conference paper at ICLR 2023
Exemplar- Based Method
Model-Based Method
MEMO
Specialized Blocks
Model Buffer
Generalized Blocks
Exemplar Buffer
MAR
Figure 4: An overview of three typical methods. Left: Exemplar- based methods train single model. Middle: Model-based methods train a new model per new task. Right: MEMO trains new specialized black per new task. When aligning the memory cost of these methods, exemplar-bar based methods can save the most exemplars, while model-based methods have the least. MEMO strikes trade-off between exemplar and model buffer.
4.4 MEMO: MEMORY EFFICIET EXPANDABLE MODEL
Motivated by the observations above, seek simultaneously consider saving exemplars and model extension in memory efficcient manner. We ask:
Given the same memory budget, we share the generalized blocks and only extend specialized blocks for new tasks, can we further improve the performance?
Concretely, we redefine the model structure in Eq. 2 by decomposing the embedding module into specialized and generalized blocks, i.e., (xx) Φ Og (x) Specialized block (-(-) corresponds to the deep layers in the network (the last basic layer in our setting, see Section B. for details), while generalized block (- corresponds to the rest shallow layers. We argue that the features of shallow layers can be shared across different incremental stages, i.., there is no need to create an extra 6, for every new task. To this end, we can extend the feature representation by only creating specialized blocks oe based on shared generalized representations. We can modify the loss function in Eq. into:
((x, = Σ
T(y kb) log Se aeall & Dsnew (x)
Effect of block sharing: We illustrate the framework of MEMO in Figure There are two advantages of MEMO. Firstly, it enables a model to extract new features by adding specialized blocks continually, Hence, we can get holistic view of the instances from various perspectives, which in turn facilitates classification. Secondly, saves the total memory budget by sharing the generalized blocks compared extra memory budget to extract similar feature maps of
to Eq. [2] It is less effective to sacrifice these homogeneous features. Since only the last basic block is created for new tasks, we can exchange the saved budget of generalized blocks for equal size of exemplars. In other words, these extra exemplars will facilitate model training more than creating those generalized blocks.
Which Block Should be
Frozen? By comparing Eq. 2 to
Eq. 3 we can find that the net-
70
Accmmyy (K)
work structure is decomposed, and only the specialized blocks are extended. It should be noted that the old backbone is fixed when learning However, since the generalized blocks are shared during the learning stages, should they fixed in Eq. or or be dynamic to learn new classes? We conduct corresponding experiments on CIFAR100 and report the incremental performance by changing the learnable blocks in Figure 5 In detail, we fix/unfix the specialized and generalized blocks in B when learning new tasks, which yields four combinations. We separately evaluate the results in two settings by changing the number of base classes. As shown in Figure there are two core observations. Firstly, by comparing the results of whether freezing the
60-
+ -v(id)()))
50
(5,(0))
tasks in Eq. _
100
Numberr clasacs
Numther clissess
(a) CIFAR100 BaseO Inc10
(b) CIFAR100 Base50 Inc10
Figure 5: Experiments about specialized and generalized blocks. Specialized blocks should be fixed; while fixing or not generalized blocks depends on the number of classes in the base stage. Block with @ means frozen, while without bar means trainable.
We illustrate the implementation with ResNet, which can also be applied to other network structures like VGGNet and Inception. See Section B.2 for details.
7

---

Published as a conference paper at ICLR 2023
100
100
90
ICaRL, (59.24)
ICaRL (58.2))
90
DER
90
(50.260225)
Memory Size: 25B3MB
Replay (58. 75)
Replay (55.63)
Accuncc (%)
WA (6), 00)
Accuracy (P)
WIA (59.26)
Aconnccy (%)
80-
(62.55
MEMD
20
(58.37)
3.521
70-
70
1.72
DER (59, 8-1
60
Replay (36 20)
60-
60
WA (60.49)
Memory Size: 41.2MB
Memory Size: 23.5MB
MEMO (63336
50
78
100
100
20
Number of classes
Number of classes
Number of classes
(a) CIFARIO0 BaseO Inc5
(b) CIFARIC0 BaseO Inc10
(c) CIFARI00 Base50 Inc5
90
100
90
Memory Size: 16.5MB
Memory Size: 756MB
Memory Size: 3297MB
80
80
Accunay
70-
188
(%)
80
28
Ansumey (36)
70
-0.9 J
KCALL (S7.92)
1CaRL (57.30)
(41.37
60
DER (62.48)
60
DER (70.26)
DER (S8. 34)
Raplay (53)
(51.30)
Replay (39.40)
30
50
(67.96)
MEMO (63.66)
MEMO (71.54)
MEMO (57.44)
20
40
60
8D
100
50
75
100
2550
500
750
1000
Number of classes
Number classes
Numbor of classes
(d) CIFARI00 Base50 Inc
(e) ImageNet100 Base50 Inc5
(f) ImageNet1000 BaseO Inc
Figure Top-1 accuracy along incremental stages, All methods are compared under the same memory budget as denoted in the image (aligned with DER). We report the performance gap after the last task of MEMO and the runner-up method at the end of the line. We list the last accuracy of each method in the legend, and report the incremental accuracy and configurations in Section A.3
Table 1: Memory- aware performance measures for CIL. AUC depicts the dynamic ability with the change of memory size, and APM depicts the capacity at some specific memory cost.
CIFARI00 AUC-A AUC-L APM-S APM-E
ImageNet100
AUC-A AUC-I APM-S APM-E
Replay
10.49
8.02
7.68
2.97
Replay
553.6
470.1
0.137
5.2e-2
iCaRL
10.81
8.64
8.32
3.00
iCaRL
607.1
527.5
0.164
5.4e-2
WA
10.80
8.92
8.57
2.95
WA
666.0
581.7
0.195
5,8e-2
DER
10.74
8.95
7.05
2.97
DER
699.0
639.1
0.192
5.8c-2
MEMO
10.85
9.03
7.18
3.06
MEMO
713.0
654.6
0.196
6.1e-2
specialized blocks, we can tell that methods freezing the specialized blocks of former tasks have stronger performance than those do not freeze specialized blocks. It indicates that the specialized blocks of former tasks should be frozen to obtain diverse feature representations. Secondly, when the base classes are limited (e.g., 10 classes), the generalized blocks are not generalizable and transferable enough to capture the feature representations, which need to be incrementally updated. By contrast, vast base classes (e.g., 50 classes) can build transferable generalized blocks, and freezing them can get better performance under such a setting. See Section C.2 for more results.
EXPERIMENT
5.1 REVISITING BENCHMARK COMPARISON FOR CIL
Former works evaluate the performance with the accuracy trend along incremental stages, which lack consideration of the memory budget and compare models at different X coordinates. As result, in this section, we strike a balance between different methods by aligning the memory cost of different methods to the endpoint in Figure [ (which is also the memory cost of DER). We show 6 typical results in Figure containing different settings discussed in Section 4.1 on three benchmark datasets. We summarize two main conclusions from these figures. Firstly, the improvement of DER over other methods is not so much as reported in the original paper. which outperforms others substantially by 10% more. In observation, the improvement of DER than others under the fair comparison is much less, indicating that saving models shows slightly greater potential than saving exemplars when the memory budget is large. Secondly, MEMO outperforms DER by substantial margin in most
8

---

Published as a conference paper at ICLR 2023
(a) 6.1 (x)))
(b) (x)(),(x))
(c) (ob (x)) d (=)(0x(()]]
Figure 7: t-SNE visualizations on CIFAR 100 of different specialized blocks learned by MEMO. Classes 1-5 are shown in dots, and classes 6-10 are shown in triangles.
cases, indicating ours is a simple yet effective way to organize CIL models with memory efficiency. These conclusions are consistent with our observations in Figure
2 and 3
5.2 How TO MEASURE THE PERFORMANCE OF CIL MODELS HOLISTICALLY?
As discussed in Section 4.2 suitable CIL model should be able to handle the task with any memory budget. Changing the model size from small to large enables holistically. We observe that all the methods benefit from the increased memory size and perform better as it becomes larger in Figure I Hence, new performance measures should be proposed considering the model capacity. We first suggest the area under the performance-mentt memory curve (AUC) since the curve of each method indicates the dynamic ability with the change of model size. We calculate AUC-A and AUC-L, standing for the AUC under the average aefformancc-memmon curve and last performance-mentee near the start point of Figure T As result, start point and endpoint, denoted as APM-S and APM-E separately. They represent the performance of the algorithm different memory scales. It should be noted that these measures into the centesimal scale, and the ranking among different methods is more important than the relative values. We report these measures in Table 0 where MEMO obtains the best performance in most cases (7 out of 8). Since all the methods are compared under the same budget, MEMO achieves the performance improvement free, verifying memory efficiency.
evaluate different methods
curve. Similarly, we find that the intersection usually emerges
also calculate the accuracy per model size the
normalized
5.3 WHAT IS LEARNED BY SPECIALIZED BLOCKS?
Adding the specialized blocks helps extract diverse feature representations of single instance. In this section, we train our model on CIFAR 100 between two incremental stages; each contains five classes. We t-SNE (Van der Maaten & Hinton, 2008) to visualize the property of these blocks. Classes from the first task are shown in dots, and classes from the second task are shown in triangles. We visualize the learned embedding of these separate specialized blocks in Figure 7(a) and 7(b] We can infer that the specialized blocks are optimized to discriminate the corresponding task, i.e., (o, (x)) can recognize classes 1~5 clearly, and (22(0, (x)) can tackle classes 6~ 10 casily. When we aggregate the embeddings from these two backbones, i.e., [ (x)),) 222 (0, (x))]. the concatenated features are able to capture all the classes seen before. Results indicate that specialized blocks, which are fixed after learning the corresponding task, act as 'unforgettable checkpoints. They will not lose discrimination as data evolves. Hence, we can aggregate diverse feature representations in the aggregated high dimension and divide decision boundaries easily.
CONCLUSION
dlas-incremmnnal learning ability is of great importance to real-world learning systems, requiring a model to learn new classes without forgetting old ones. this paper, we answer two questions in CIL. Firstly, we fairly compare different methods by aligning the memory size at the same scale. Secondly, we find that not all layers are needed be created and stored for new tasks, and propose a simple yet effective baseline, obtaining state-of-the-at performance for free in the fair comparison. Experiments verify the memory efficiency of our proposed method.
Limitations: CIL methods can also be divided by whether using extra memory. Apart from the methods discussed in this paper, there are other methods that do not save exemplars or models. We only concentrate the methods with extra memory and select several typical methods for evaluation.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
This work is partially supported by NSFC (61921006, 62006112, 62250069), NSP of Jiangsu Province (BK20200313), Collaborative Innovation Center of Novel Software Technology and Industrialization, China Scholarship Council (49)202006190134).
REFERENCES
Davide Abati, Jakub Tomezak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pp. 3931-3940, 2020.
Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. Ss-il: Separated softmax for incremental learning. In ICCV, pp. 144-853, 2021.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In CVPR, pp. 3366-3375, 2017.
Alessio Ansuini, Alessandro Laio, Jakob Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. NeurlPS, 32, 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv: 1711:7770. 2017.
Wasserstein gan. arXiv preprini
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et A closer look at memorization in deep networks. In ICML, pp. 233-242. PMLR, 2017.
Francisco M Castro, Manuel
Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In ECCV, pp. 233-248, 2018.
Mahawaga Arachchige Pathum Chamikara, Peter Bertók, Dongxi Liu, Seyit Camtepe, and Ibrahim Khalil. Efficient data perturbation for privacy preserving and accurate data stream mining. Pervasive and Mobile Computing, 48:1-19, 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Ranzato. Continual learning with tiny episodic memories. 2019.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Dual-teacher class-incremental learning with data-free generative replay. In CVPR, pp. 3543-3552, 2021.
Yulai Cong, Miaoyun Zhao, Jianqiao Li, Sijia Wang, and Lawrence Carin. Gan memory with no forgetting. NeurlPS, 33:1 33:16481-16494 2020.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-24-82 248-255, 2009.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 3rXiv:1810.0480) 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2020.
Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthiu Cord. Dytox: Transformers for continual learning with dynamic token expansion. arXiv preprint arXiv:2 2021.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 14):128-135, 1999.
10

---

Published as a conference paper at ICLR 2023
Mohamed Medhat Gaber. Advances in data stream mining. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2(1):79- 85, 2012.
Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. TPAMI, pp. press, 2020.
Heitor Murilo Gomes, Jean Paul Barddal, Fabrício Enembreck, and Albert Bifet. A survey on ensemble learning for data stream classification. CSUR, 50(2):1-36, 2017.
Ian Goodfellow, Jean Pouget-Abadie,, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, 27, 2014.
Stephen T Grossberg. Studies of mind and brain: Neural principles of learning, perception, develop- ment, cognition, and motor control, volume 70. Springer Science & Business Media, 2012.
Chen He, Ruiping Wang, Shiguang Shan, and Xilin Chen. Exemplar-supported generative reproduc- tion for class incremental learning. In BMVC, pp. 98, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0253/ 2015.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In CVPR, pp. 831-839, 2019.
Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-effiiiet incremental learning through feature adaptation. In ECCV, pp. 699-715, 2020.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In AAAI, 2018.
Xisen Jin, Arka Sadhu, Junyi Du, and Xiang Ren. Gradient- based editing of memory examples for online task-free continual learning. NeurIPS, 34, 2021.
Zixuan Ke, Hu Xu, and Bing Liu. Adapting bert for continual learning a sequence of aspect sentiment classification tasks. In NAACL, pp. 4746-4755. 2021.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sama,, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. NeurIPS, 33:18661-
18673, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwins, Barwinska, et al. Overcoming catastrophic forgetting in neural networks. PNAS, 11<(13):55113352, 2017.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. ICML, pp. 3519-3529, PMLR, 2019.
Georg Krempl, Indre Žliobaite, Dariusz Brzeziński,, Eyke Hüllermeier, Mark Last, Vincent Lemaire, Tino Noack, Ammar Shaker, Sonja Sievi, Myra Spiliopoulou, et al. Open challenges for data stream mining research. KDD, 16(1):1-10, 2014.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, 2009.
Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung- Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. NIPS, 30:4652-4662, 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI, 40(12):2935-291) 2017.
Zhuoyun Li, Changhong Zhong, Sijia Liu, Ruixuan Wang, and Wei-Shi Zheng. Preserving earlier knowledge in continual learning with the help of all previous feature extractors. arXiv preprint arXiv:2 104. 13614, 2021.
11

---

Published as a conference paper at ICLR 2023
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi-class incremental learning without forgetting. In CVPR, pp. 12245-12254, 2020.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. CVPR, pp. 2544-2553, 2021a.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Rmm: Reinforced memory management for class- incremental learning. NeurIPS, 34, 2021b.
Yaoyao Liu, Yingying Li, Bernt Schiele, and Qianru Sun. Online hyperparameter optimization for inass-incrementl learning. arXiv preprint arXiv:2301.05033 2023.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535. 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In NeurIPS, pp. 6467-6476, 2017.
Hartmut Maennel, Ibrahim M Alabdulmohsin, Ilya O Tolstikhin, Robert Baldock, Olivier Bousquet, Sylvain Gelly, and Daniel Keysers. What do neural networks learn when trained with random labels? NeurIPS, 33: 19693-197044 2020.
Jingyi Ning, Lei Xie, Chuyu Wang, Yanling Bu, Fengyuan Xu, Da-Wei Zhou, Sanglu Lu, and Baoliu Ye. Rf-badge: Vital sign-based authentication via rfid tag array on badges. IEEE Transactions on Mobile Computing, 2021.
Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. NeurIPS, 34:30298-30312, 2021.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 11::54-71, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, pp. 8026-8037, 2019.
Mozhgan PourKeshavarzi, Guoying Zhao, and Mohammad Sabokrou. Looking back on learned experiences for class/task incremental learning. In ICLR, 2022.
Jathushan Rajasegaran, Munawar Hayat, Salman Khan, Fahad Shahbaz Khan, and Ling Shao.
Random path selection for incremental learning. NeurlPS, 3, 2019.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, pp. 2001-2010, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicrap, and Greg Wayne. Experience replay for continual learning. In NeurlPS, pp. 350-360, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 106447160, 2016.
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In ICASSP, pp. 6655-6659. IEEE, 2013.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In ICML, pp. 4548-4557. PMLR, 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NeurIPS, pp. 2990-2999, 2017a.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. NIPS, 30, 2017b.
12

---

Published as a conference paper at ICLR 2023
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint WrXiv:1409.1530 2014.
James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming; A new approach for data- free ctassincremmnntl learning. ICCV, pp. 2021.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In CVPR, pp. 403-412, 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pp. 2818-2826, 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11). 2008.
Fu-Yun Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and
compression for glass-incremental learning. arXiv preprint arXiv:2204. 04662, 2022.
Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. Deep metric learning with angular loss. In ICCV, pp. 2593-2601, 2017.
Max Welling.
Herding dynamical weights to learn. In JCML, pp. 1121-1128, 2009.
Tz- Ying Wu, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vasconcelos, Rahul Bhotika, and Stefano Soatto. Class-incremental learning with strong pre-trained models. In CVPR, pp. 9601-9610, 2022.
Yue WU, Yinpeng Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, pp. 372, 2019.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In NeurIPS, pp. 899-908, 2018.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. CVPR, pp. 3014-3023, 2021.
Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng. Jiangiang Huang, and Xian-sheng Hua. Quantization networks. In CVPR, pp. 7308-7316, 2019.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In ICLR, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? NIPS, 27, 2014.
Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. CVPR, pp. 6982-6991, 2020.
Qing Yu and Kiyoharu Aizawa. Unsupervised out-of-distribution detection by maximum classifier discrepancy. In ICCV, pp. 9518-9526, 2019.
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? arXiv preprint 1rX209:902.01996, 2019.
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In CVPR, pp. 13208-13217 13217, 2020.
Hanbin Zhao, Yongjian Fu, Mintong Kang, Qi Tian, Fei Wu, and Xi Li. Mgsvf: Multi- -rainegr slow vs. fast framework for few-shot class-incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021a.
Hanbin Zhao, Xin Qin, Shihao Su. Yongjian Fu, Zibo Lin. and Xi Li. When video classification meets incremental classes. In ACM MM, pp. 880-889, 2021b.
13

---

Published as a conference paper at ICLR 2023
Hanbin Zhao, Hui Wang. Yongjian Fu, Fei Wu, and Xi Li. Memory efficient caas-inccrmmenal learning for image classification. IEEE Transactions on Neural Networks and Learning Systems, 2021c.
Da-Wei Zhou, Yun Wang, Han-Jia Ye, and De-Chuan Zhan. Pycil: A python toolbox for class- incremental learning. arXiv preprint arXiv:2 12533, 2021a.
Da-We Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Co-transport for class-incremental learning. In ACM MM, pp. 1645-1654, 2021b.
Da-Wei Wei Zhou, Fu- Yun Wang. Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. Forward compatible few-shot class-incremental learning. In CVPR, pp. (046-9056) 2022a.
Dei Zhou, Han-Jia Ye, Liang Ma, Di Xie, Shiliang Pu, and De-Chuan Zhan. Few-shot class- incremental learning by sampling multi-phas tasks. IEEE Transactions on Pattern Analysis and Machine Intellience, 2022b.
DaiWei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu, Deep class-incremental learning: A survey. arXiv preprint wxXiv:2302.03648, 2023.
Zhi-Hua Zhou and Yuan Jiang. Nec4. 5: neural ensemble based c4. 5. JEEE Transactions knowledge and data engineering. 16(6):770-773. 2004.
Zhi-Hua Zhou, Yuan Jiang, and Shi-Fu Chen. Extracting symbolic rules from trained neural network ensembles. Ai Communications, 16(1):3-15, 2003.
14

---

Published as a conference paper at ICLR 2023
Supplementary Material
Class-incremental learning (CIL. is of great importance to the machine learning community. In the main paper, we answer two questions in CIL. The first is about how to fairly compare different methods, and we achieve this goal by aligning the memory cost. The second is about organizing the model with memory efficiency. and our proposed MEMO maintains the diverse feature representations with a modest memory cost. In the supplementary, we report more details about the experimental results mentioned in the main paper. We also provide more empirical evaluations and discussions. The supplementary material organized according to the two questions above we first supply details to fairly compare different methods and then discuss how to manage the model in a memory-efficien manner. Afterward, we give the additional experimental and implementation details.
Section/AJreport the implementation details of the models and exemplars in the performance- memory curve of the main paper, and numerical results in the benchmark comparison;
Section B discusses the variations of MEMO, including the definition of specialized and generalized blocks and other deep network structures;
Section provides providess extra illustrative experimental evaluations that cannot be included in the main paper due to page limit, including the last ceuuracc-memovy curve, gradient norms for all tasks, CKA visualizations of different blocks, the ablations about which layer to freeze, running time comparison, and CIL performance with multiple runs;
Section D Diolistically discusses the implementations of related work and ours, the choice of compared methods, and broader impacts.
A IMPLEMENTATION DETAILS OF PERFORMANCE-MEMORT CURVE
We give the performance-memory curve in the main paper as one main contribution to fairly com- paring different CIL methods. In this section, we give the detailed implementations of each point on the X coordinate. We will start with CIFAR100 (Krizhevsky et al. 2009), and then discuss ImageNet100 (Deng et al. (209)).
In the following discussions, we use & to represent the exemplar set. |E| denotes the number of exemplars, and S(E) represents the memory size (in MB) that saving these exemplars consume. Following the benchmark implementation (Rebuffi et al., 2017), 7) 2,000 exemplars are saved for every method for CIFARIOM and ImageNet100, Hence, the exemplar ar size of each method is denoted as |8| = 2000- + E, where E corresponds to the extra exemplars exchanged from the model size, as discussed in the main paper. We use '# Parameters' to represent the number of parameters and 'Model Size' to represent the memory budget (in MB) it costs to save this model in memory. The total memory size (i.e., numbers on the X coordinate) is the sum of exemplars and the models.
Since iCaRL (Rebuffi et al. 2017), Replay (Chaudhry et al. 2019) and WA (Zhao et al. 2020) are typical exemplar-base based methods, they the same network backbone with the same model size. Hence, they have equal memory sizes. DER (Yan et al., 2021) sacrifices the memory size to store the backbone from history, and it has the least exemplars. Compared to DER, MEMO does not keep the duplicated generalized blocks from history and saves much memory size to change into exemplars.
In the following discussions, first give the tables to illustrate the implementation of different methods and report their incremental performance with figures. We report the improvement of MEMO against the runner-up method at the end of each line in the figures and analyze the empirical evaluations after the tables and figures.
A.1 IMPLEMENTATIONS OF CIFARIA 100
There are five X coordinates in the curve of CIFAR100, e.g., 7.6, 12.4, 16.0, 19.8, and 23.5 MB. Following, we show the detailed implementation of different methods at these scales.
15

---

Published as a conference paper at ICLR 2023
7.6BB
[8]
S(E)
Model Type # Parameters Model Size
KCRL
WA
Replay
2000 5.85MB
ResNet32
0.46M
1.76MB
Accuracy (%)
80
Riplay
iCaRL
2000 5.85MB
ResNet32
0.46M
1.76MB
WA
2000 5.85MB
ResNet32
0.46M
1.76MB
80
DER
2096 6.14MB
ConvNet2
0.38M
1.48MB
40
-9.55_
МЕMО
2118 6,20MB
ConvNet2
0.37M
1.42MB
25
50
100
Number of classes
Table Implementation details when memory size= 7.6 MB
Figure 8: CIL performance
CIFARI00 with 7.6 MB Memory Size: The implementations are shown in Table 2 and Figure 7.6 MB is a relatively small memory size. Since we need to align the total budget of these methods, we are only able to use small backbones for DER and MEMO. These small backbones, i.e. ConvNet with two convolutional layers, has much fewer parameters than ResNet32, and saving 10 ConvNets matches the memory size single ResNet32 (1 188MMBB versus 1. 1.76MB). We can infer from the table that DER and MEMO are restricted by the learning ability of the inferior backbones, which perform poorly in the base session. These results are consistent with the conclusions in the main paper that model-based methods are inferior to exemplar based methods with small memory budget.
12.4MB
[E]
S(E)
Model Type # Parameters Model Size
IaaL
WA MEMO
Replay
3634
10.64MB
ResNet32
0.46M
1.76MB
Accornyy (9%
80
DER
Replay
iCaRL
3634 10.64MB
ResNet32
0.46M
1.76MBB
WA
3634 10.64MB
ResNet32
0.46M
1.76MB
80
0.23f
DER
2000 5.85MB
ResNet14
1.70M
6,55MB
MEMO
2495 7.32MB
ResNet14
1.33M
5.10MB
25
50
75
100
Number of classes
Table 3: Implementation details when memory size= 12.4 MB
Figure 9: CIL performance
CIFARI00 with 12.4 MB Memory Size: The implementations are shown in Table 3 Figure By raising the total memory cost to 12.4 MB, exemplar-based methods can utilize the extra memory size to exchange 1634 exemplars, and model-based methods can switch to more powerful backbones to get better representation ability. We use ResNet14 for DER and MEMO in this setting. We can infer that model-based methods show competitive results with stronger backbones and outperform cemplar-based methods in this setting. These results are consistent with the conclusions in the main paper that the intersection between these two groups of methods exists near the start point.
16.0MB
[3]
S(E)
Model Type # Parameters Model Size
KaRRi.
WA MEMO
Replay
4900
14.3MB
ResNet32
0.46M
1.76MB
Acameyy (emmacy
DER
Raplay
iCaRL.
4900
14.3MB
ResNet32
0.46M
1.76MB
00244
WA
4900
14.3MB
ResNet32
0.46M
1.76MB
50
DER
2000 5.85MB
ResNet20
2.69M
10.2MB
МЕMO
2768 8.1MMB
ResNet20
2.
8.01MB
40
25
100
Numher of classes
Table 4: Implementation details when memory size= 16.0 MB
Figure 10: CIL performacee
CIFAR100 with 16.0 MB Memory Size: The implementations are shown in Table and Figure TO
By raising the total memory cost to 16.0 MB, exemplar- based methods can utilize the extra memory size exchange 2900 exemplars, and model-based methods can switch to larger backbones get better representation ability. We use ResNet20 for DER and MEMO in this setting. The results are consistent with the former setting, where we can infer that model-based methods show competitive results with stronger backbones and outperform exemplar- based methods.
16

---

Published as a conference paper at ICLR 2023
19.8MB
[8]
S(8)
Model Type # Parameters Model Size
KCRL Riplay
WA
Replay
6165 18. 8.06MB
ResNet32
0.46M
1.76MBB
Accuracy (%)
iCaRL
6165 18.06MB
ResNet32
0.46M
1.76MB
1.53g
WA
6165
18.06MB
ResNet32
0.46M
1.76MB
80
DER
2000 5.85MBB
ResNet26
3.60M
13.9MB
MEMO
3040 8.91MB
ResNet26
2.86M
10.92MB
25
50
100
Number of clusses
Table 5: Implementation details when memory size= 19.8 MB
Figure 11: CIL performance
CIFARI00 with 19.8 MB Memory Size: The implementations are shown in Table and Figure By raising the total memory cost to 19.8 MB, exemplar-based methods can utilize the extra memory size to exchange 4165 exemplars, and model-based methods can switch to larger backbones to get better representation ability. We use ResNet26 for DER and MEMO in this setting. The results are consistent with the former setting, where we can infer that model-based methods show competitive results with stronger backbones and outperform exemplar-based methods.
23.5MB
[
S(E)
Model Type # Parameters Model Size
CaRI,
WA
Replay
7431 21.76MB
ResNet32
0.46M
1.75MB
80
Riplay
iCaRL
7431 21.76MB
ResNet32
0.46M
1.75MB
Accmmcy
2.171
WA
7431 21.76MB
ResNet32
0.46M
1.75MB
60
DER
2000 5.86MB
ResNet32
4.63M
17.68MB
MЕMO
3312 9.7MB
ResNet32
3.62M
13.83MB
25
50
75
100
Number of classes
Table 6: Implementation details when memory size= 23.5 MB
Figure 12: CIL performance
CIFARI00 with 23.5 MB Memory Size: The implementations are shown in Table and Figure 12
By raising the total memory cost to 23.5 MB, exemplar-based methods can utilize the extra memory size to exchange 5431 exemplars, and model-based methods can switch to larger backbones to get better representation ability. We use ResNet32 for DER and MEMO in this setting. The results are consistent with the conclusions in Section 5.1. Model-based methods are better than exemplar- based methods with large memory sizes, but the performance gap is not so large reported in (Yan et al.) 2021 when fairly compared.
To summarize, conduct fair comparisons for exemplar saase and model-based methods by varying the memory size from small to large. Results indicate that exemplar-base methods are competitive with small memory sizes, while model-based methods are competitive with large ones. Our proposed MEMO obtains the best performance in most cases in these settings.
A.2 IMPLEMENTATIONS OF IMAGENET100
Similar to CIFAR100, we can conduct an exchange between the model and exemplars on Ima- geNet100. For example, saving a ResNet18 model costs 11, 176, 512 parameters (float), while saving an ImageNet image costs 3 x 224 x 224 integer numbers (int). The budget of saving backbone is equal to saving 11, 176, 512 floats x bytes/float :33 x 224 x 224) bytes/image zs 297 images for ImageNet. We conduct the experiment with ImageNet100, Base50 Inc5, as discussed in the main paper. There are six X coordinates in the curve of ImageNet100, e.g., 329, 493, 755, 872, 1180 and 1273 MB. Following, we show the detailed implementation of different methods at these scales.
17

---

Published as a conference paper at ICLR 2023
329MB
[E]
S(E)
Model Type Parameters Model Size
80
A 1omnnn (%)
Replay
2000 287MB
ResNet18
11.17M
42.6MB
&CaRL
1,51
iCaRL
2000 287MB
ResNet18
11.17M
42.6MB
WA
2000 287MB
ResNet18
11.17M
42.6MB
Replay
WA
DER
2032 291MB
ConvNet4
9.96M
38.0MB
40
MEMO
MEMO
2115 303MB
ConvNet4
6.81M
26.0MB
25
30
75
100
Table 7: Implementation details when memory size=329MB
Figure 13: CIL performance
ImageNet100 with 329 MB Memory Size: The implementations are shown in Table 7 and Figure 13 329 MB is relatively small memory size. Since need to align the total budget of these methods, we are only able to use small backbones for DER and MEMO. These small backbones, i.e., ConvNet with four convolutional layers, has much fewer parameters than ResNet18, and saving 10 ConvNets matches the memory size of single ResNet18. We can infer from the table that DER and MEMO are restricted by the learning ability the inferior backbones, which perform poorly the base session. However, our proposed MEMO outperforms these better backbones by saving the 'unforgettable checkpoints," which obtains the best last accuracy and average accuracy in this case.
493MB
[3]
S(E)
Model Type Parameters Model Size
80
Replay
3136 450MB
ResNet18
11.17M
42.6MB
1.11
iCaRL
3136 450MB
ResNet18
11.17M
42.6MB
70
ICaRL
WA
3136 450MB
ResNet18
11.17M
42.6MB
60
Replay
DER
2000 287MB
ResNet10
53.96M
205MB
MEMO
MEMO
2327 334MB
ResNet10
41.63M
158MB
40
75
Number of classes
Table 8: Implementation details when memory size=493MB
Figure 14: CII. performance
ImageNet100 with 493 MB Memory Size: The implementations are shown in Table and Figure 14 By raising the total memory cost 493 MB, exemplar based methods can utilize the extra memory size to exchange 1 136 exemplars, and model- -ased methods can switch to larger backbones to get better representation ability. We use ResNet10 for DER and MEMO this setting. We can infer from the figure that model-based methods show competitive results with stronger backbones and outperform exemplar-based methods in this setting.
755MB
[]]
S(E)
Model Type # Parameters Model Size
90
28
Replay
4970
713.5MB
ResNet18
11.17M
42.6MB
iCaRL
4970 713.5MB
ResNet18
11.17M
42.6MB
Accmmey
70-
ICaRL
DER
WA
4970 713.5MB
ResNet18
11.17M
42.6MB
60
Ruplay
DER
2000 287MB
ResNet18
122.9M
468MB
WA
MEMO
MEMO
2739 393.2MB
ResNet18
95.11M
362.8MB
40
Number of classes
Table 9: Implementation details when memory sizc=755MB
Figure 15: CIL performance
ImageNet100 with 755 MB Memory Size: The implementations are shown in Table and Figure 15 By raising the total memory cost to 755 MB, exemplar-bass methods can utilize the extra memory size to exchange 2970 exemplars, and model-base better representation ability. We use ResNet18 for DER and MEMO in this setting. The results are consistent with the former setting, where we can infer that model-based methods show competitive results with stronger backbones and outperform exemplar-bate based methods.
methods switch to larger backbones to get
18

---

Published as a conference paper at ICLR 2023
872MB
[E]
S(E)
Model Type # Parameters Model Size
Replay
5779 829MB
ResNet18
11.17M
42.6MB
-50
iCaRL
5779 829MB
ResNet18
11.17M
42.6MB
70
70
iGaRL
DER
0.96
WA
5779 829MB
ResNet18
11.17M
42.6MB
Replay
DER
2000 287MB
ResNet26
153.4M
585.2MB
60
WA
MEMO
MEMO
2915 417MB
ResNet26
119.0M
453MB
50
01
25
50
75
100
Number of classes
Table 10: Implementation details when memory size=872MB
Figure 16: CIL performance ImageNet100 with 872 MB Memory Size: The implementations are shown in Table [0] and Figure T6 By raising the total memory cost to 872 MB, exemplar-based methods can utilize the extra memory size to exchange 3779 exemplars, and model- based methods can switch to larger backbones for better representation ability. We use ResNet26 for DER and MEMO in this setting.
1180MB
|3]
S(E)
Model Type # Parameters Model Size
Replay
7924 1137MB
ResNet18
11.17M
42.6MBB
80
2.98
iCaRL
7924 1137MB
ResNet18
11.17M
42.6MB
20
iCaRL
A
DER
WA
7924 1137MB
ResNet18
11.17M
42.6MB
Replay
DER
2000 287MB
ResNet34
234.1M
893MB
60
WA
MEMO
[4170 598MB
ResNet34
152.4M
581MB
0
25
50
75
100
Number of classes
Table 11: Implementation details when memory size=1180MB
Figure 17: CIL performance ImageNet100 with 1180 MB Memory Size: The implementations are shown Table [TT and Figure 17 By raising the total memory cost 1180 MB, exemplar methods can utilize the extra memory size to exchange 5924 exemplars, and model-based methods can switch to larger backbones to get better representation ability. We use ResNet34 for DER and MEMO in this setting.
1273MB
[8]
S(E)
Model Type # Parameters Model Size
90
Replay
8574 1230MB
ResNet 8
11.17M
42.6MB
80
3.04
iCaRL
8574 1230MB
ResNet18
11.17M
42.6MBB
(CaRL
DER
WA
8574 1230MB
ResNet18
11.17M
42.6MB
Replay
DER
2000 287MB
ResNet50
258.6M
986MB
60
WK
MEMO
MEMO
4270 612MB
ResNet50
173.2M
660MB
0
25
50
75
100
Number classes
Table 12: Implementation details when memory size=1273MB
Figure 18: CIL performance ImageNet100 with 1273 MB Memory Size: The implementations are shown in Table 12 and Figure 18 By raising the total memory cost 1273 MB, exemplar- methods can utilize the extra memory size to exchange 6574 exemplars, and model-based methods can switch to larger backbones to get better representation ability. We use ResNet50 for DER and MEMO in this setting. The results are consistent with the conclusions in Section 5.1. Model-based methods are better than exemplar-bir based methods with large memory sizes, but the performance gap is not so large as reported in (Yan et al. (021)) when fairly compared.
Discussion about Backbones: should be noted that ResNet18 the benchmark backbone for ImageNet, and the memory size for 872, 1180, and 1273 MB are larger than the benchmark setting. We conduct these experiments for two reasons. First, handling large-scale image inputs require more convolutional layers, and it is hard to find typical models with small memory budgets. Second, we would like to investigate the performance when the model is large enough to see whether the improvement of stronger backbones will converge, and the results successfully verify our assumptions.
To summarize, we conduct fair comparisons for exemplar-bate based and model- -base methods by varying the memory size from small to large. Results indicate that exemplar- based methods competitive with small memory sizes, while model-based methods are competitive with large ones. Our proposed MEMO obtains the best performance in most cases in these settings.
19

---

Published as a conference paper at ICLR 2023
A.3 NUMERICAL RESULTS AND CONFIGURATIONS FOR SECTION 5.1
In this section, we give the numerical incremental performance of different methods and confi configurations
in Section 5.1 We first list the incremental and average accuracy in Table T3. 14
15
T6
17
18
and
give the setting in Table 19 20
21
22
23 24
As we discussed in the main paper, a fair comparison among different methods should be aligned to
the same memory budget. Since DER (Yan et al. 2021) requires saving multiple backbones, training DER requires the largest memory budget. Hence, we align the budget of other methods to DER (as shown in Table 19
24) by saving more exemplars for them.
Table 13: Incremental and average accuracy comparison of different methods under CIFARI00 BaseO Inc5 setting.
Accerasy cash souscn (55]
Average
Replay
95.80
$4.00
29.77
15.55
14.44
14.57
76.14
73.12
71.23
70.66
70-42
58.63
65.55
16. 19
6465
62.34
61:3
60.24
60.17
58.78
70.81
KaRL
93.80 80
10.67
77.18
76.97
7501
74.23
7261
70:44
59.13
68.09
06.39
03.0
63.38
60.1
59.24
MENO
95.00
89.50
64.33
76.35
79.6
76.17
76.71
74.88
73116
71.76
70.33
8682
67.6
88 41
61.07
65.1
64.55
63.75
65.77
62.39
75.16
Table 14: Incremental and average accuracy comparison of different methods under CIFAR100 BaseO Incl0 setting.
Accuracy in each session (%) f
Method
Average
2
3
4
6
7
8
9
10
Replay
90.40
79.90
79.20
74.08
70.32
67.03
64.76
60.31
58. 14
55.61
69.98
iCaRL
90.40
80.25
78.00
72.80
70.88
68.88
66.37
62.82
60.51
58.52
70.94
WA
90.40
76.65
74.57
70.92
69.20
66.48
65.24
62.41
60.37
59.26
69.55
DER
90.40
80.55
77.37
73.47
70.78
68.38
67.43
64.22
61.93
60.26
71.47
MEMO
90.40
80.30
78.33
74.65
71.74
69.67
68.19
65.34
63.10
61.98
72.37
Table 15: Incremental and average accuracy comparison of different methods under CIFAR100
Base50 Inc5 setting.
Accuracy in each session (%) f
Method
Average
1
2
3
4
6
7
9
10
11
Replay
76.32
68.9
67.60
65.34
63.84
59.39
58.12
56.20
63.55
iCaRL.
76.32
70.00
69.28
67.38
67.00
65.45
63.55
60.95
60.47
59.31
58.37
65.28
WA
76.32
70.09
68.58
67.54
67.04
65.52
64.24
62.49
62.00
61.21
60.49
65.95
DER
76.32
74.71
72.15
70.92
69.54
68.03
65.26
63.00
61.80
61.73
59.84
67.57
MEMO
76.32
74.05
72.77
71.28
70.93
69.29
67.38
65.88
64.36
64.29
63.36
69.08
Table 16: Incremental and average accuracy comparison of different methods under CIFAR100
Base50 1nc10 setting.
Accuracy in each session (%) t
Method
Average
2
3
4
6
Replay
76.32
64.57
61.43
56.91
54.72
53.23
61.19
iCaRL
76.32
68.43
66.83
63.52
59.89
57.92
65.48
WA
76.32
71.25
69.30
64.79
62.37
60.62
67.44
DER
76.32
73.17
70.99
67.51
64,49
62.48
69.16
MEMO
76.32
74.37
71.87
68.50
65.81
63.66
70.09
20

---

Published as a conference paper at ICLR 2023
Table 17: Incremental and average accuracy comparison of different methods under ImageNet100 Base50 Inc5 setting.
Accuracy each session (%)
Method
Average
3
4
6
7
9
10
11
Replay
84.44
66.76 76.76
64.10
61.26
59.00
58.67
58.40
56.56
56.67
54.36
51.30
61 .0
iCaRL
84.44
73.27
69.70
67.29
65.46
62.80
63.55
61.53
60.36
59.26
57.30
65.90
WA
84.44
68.65
67.73
65.38
64.71
63.63
63.52
61.60
60.02
59.09
57.96
65.15
DER
84.44
82.22
81.30
78.22
77.03
75.23
74.10
74.14
72.11
71.07
70.26
76.37
MEMO
84.44
83.71
81.60
78.68
77.43
76.53
75.42
74.49
73.56
72.63
71.54
77.27
Table 18: Incremental and average accuracy comparison of different methods under ImageNet1000 BaseO Inc setting.
Accuracy in cach session (%) t
Method
Average
10
Replay
83.16
73.47
64.72
59.93
54.05
50.78
46.91
44.24
41.22
39.40
55.78
iCaRL.
81.16
74.56
67.63
61.98
57.03
53.15
49.72
47.07
43.88
41.30
57.94
WA
81.16
76.87
70.09
64.84
59.38
55.75
51.22
48.37
45.72
43.23
59.86
DER
83.16
78.64
74.21
71.44
68.13
65.80
62.9
61.21
59.84
58.30
68.36
MEMO
83.16
78.47
73.72
70.93
68.05
65.78
62.91
61.24
59.22
57.40
68.09
Table 19: Configurations of different methods under CIFAR100 BaseO Inc5 setting.
41.22MB
|3|
S(E)
Model Type # Parameters Model Size
Replay
13466
39.47MB
ResNet32
0.46M
1.75MB
iCaRL
13466
39.47MB
ResNet32
0.46M
1.75MB
WA
13466
39.47MB
ResNet32
0.46M
1.75MB
DER
2000
5.86MB
ResNet32
9.27M
35,36MB
MEMO
4771
13.98MB
ResNet32
7.14M
27.24MB
Table 20: Configurations of different methods under CIFAR100 BaseO Inc10 setting.
23.5MB
|8]
S(E)
Model Type # Parameters Model Size
Replay
7431 21.76MB
ResNet32
0.46M
1.75MB
iCaRL
7431 21.76MB
ResNet32
0.46M
1.75MB
WA
7431 21.76MB
ResNet32
0.46M
1.75MB
DER
2000 5.86MB
ResNet32
4.63M
17.68MB
MEMO
3312
9.7MB
ResNet32
3.62M
13.83MB
Table 21: Configurations of different methods under CIFARI00 Base50 Inc5 setting.
25.3MB
|8]
S(E)
Model Type # Parameters Model Size
Replay
8035 23.53MB
ResNet32
0.46M
1.75MB
iCaRL
8035 23.53MB
ResNet32
0.46M
1.75MB
WA
8035 23.53MB
ResNet32
0.46M
1.75MB
DER
2000 5.86MB
ResNet32
5.1M
19.45MB
MEMO
3458 10.1
ResNet32
3,98M
15.17MB
21

---

Published as a conference paper at ICLR 2023
Table 22: Configurations of different methods under CIFARI00 Base50 Incl0 setting.
16.45MB
[8]
S(E)
Model Type # Parameters Model Size
Replay
5017 14.7MB
ResNet32
0.46M
1.75MB
iCaRL
5017 14.7MB
ResNet32
0.46M
1.75MB
WA
5017 14.7MB
ResNet32
0.46M
1.75MB
DER
2000 5,86MB
ResNer32
2.78M
10.61MB
MEMO
2729 8.0MB
ResNet32
2.22M
8,45MB
Table 23: Configurations of different methods under ImageNet100 Base50 Inc5 setting.
756. IMB
[1]
S(E)
Model Type # Parameters Model Size
Replay
4970 713.47MB
R8sNet18
11.17M
42.63MB
iCaRL
4970 713.47MB
ResNet18
11.17M
42.63MB
WA
4970 713.47MB
ResNet18
11.17M
42.63MB
DER
2000 287.1MB
ResNet18
122.94M
168.99MB
MEMO
2739 393.2MB
ResNet18
95.11M
362.83MB
Table 24: Configurations of different methods under ImageNet1000 BaseO 1nc100 setting.
3297MB
[8]
S(E)
Model Type # Parameters Model Size
Replay
22672
3254.67MB
ResNet18
11.17M
42.63MB
iCaRL
22672 3254.67MB
ResNet18
11.17M
42.63MB
WA
22672 3254.67MB
ResNet18
11.17M
42.63MB
DER
20000
2871MB
ResNet18
111.76M
426.35MB
MRMO
20666 2967MB
ResNet18
86.72M
33
B VARIATIONS OF MEMO
In this section, we discuss the variations of MEMO, iincluding the choice of specialized blocks and implementation with other backbones.
B. How TO DEFINE THE SPECIALIZE AND GENERALIZE BLOCKS?
In the main paper, we observe the differences between shallow and deep layers in terms of gradient, MSE, and similarity. Hence, we decouple the network structure into deep and shallow layers and treat the deep layers as specialized blocks and the others as generalized ones. It worth exploring how to decompose the model into specialized and generalized blocks. In this section, we conduct vast experiments to give the rule of thumb for defining the specialized and generalized blocks.
We first take ResNet32 as example. There are three groups of residual blocks in ResNet32 i.e. residual blocks ~ 5 are encapsulated as group I; residual blocks 10 are encapsulated as group 2 and 11 15 as group 3. We treat these groups as the minimal unit when decoupling the network.
Since there are only three groups of blocks in ResNet32, there are only two ways to decouple the network structure (cutoff after group 1 or 2). Therefore, performance when treating the last group as the specialized block (as discussed in the main paper) and the last two groups as specialized blocks (denoted as MEMO-2). MEMO-2 extends two groups of
conduct experiments to compare the
These groups are denoted as 'layers' in the implementation (nttps: / / ggithbb.bb//gggggggggg/ vis visien/blot http:///bt..aaa//ooooo.....ooooo
resnet .py,, and each 'layer' may contain several resid- ual blocks. We interchangeably use 'layers' and 'groups' in this paper, and both of them denote the combination of residual blocks.
22

---

Published as a conference paper at ICLR 2023
90
90
MEMO, 2000
2000
MEMO-2, 2000
MEMO, 3033
Accuracy (%)
Accuracy (%)
MEMO-3, 2000
60
70
MEMO-2,2140
MEMO, 2698
50
65
20
40
60
80
100
20
40
60
80
100
Number classes
Number classes
(a) ResNet32
(b) ResNet18
100
80
MEMO-BI B14, 5252
90
BHEMO-B33,44
MEMO-B12, 3518
Accuracy (%)
70
Accuracy (%)
MEMO-BIL 2615
80
MEMO, 2000
60
70
MEMO-4, 2000
MEMO-3, 2019
50
MEMO-2, 2098
60
MEMO, 2412
40
50
20
40
60
80
100
25
50
75
100
Number of classes
Number of classes
(c) VGG8
(d) ResNet32, fine-grained decouple
Figure 19: Experiments about the choice of specialized and generalized blocks. The digit after model name denotes the number of exemplars used during model training. Choosing the last block groups as specialized blocks is more memory-efficient.
residual blocks at a time, which consumes more memory budget than MEMO. We follow the same evaluation protocol Section 5.1 in the main paper to compare under the CIFAR100 BaseO Inc10 setting, and report the results in Figure [9(a)
There are three lines in Figure (9) and the number of exemplars is shown in the legend. It is obvious that MEMO-2 uses more memory size than MEMO with the same exemplar size. Hence, we follow the description in the main paper and align the memory cost of MEMO to MEMO-2, and denote the method with aligned exemplars as MEMO, 3033. There are two main conclusions in this figure. Firstly, MEMO-2 has little better performance than MEMO with the same exemplar size, which means extending more generalized blocks can improve the performance, although they are highly similar. But should noted that these methods are not fairly compared since MEMO-2 uses more model size than MEMO. Secondly, when aligning the memory cost of MEMO, 3033 to MEMO-2, we can infer that MEMO has better performance than MEMO-2, verifying that treating the last layer as specialized blocks is memory-efficies In other words, saving the generalized blocks per task is less efficient than saving exemplars of equal size.
We also explore the strategy with ResNet18 under the ImageNet100 Base50 Inc5 setting and report the results in Figure [9(b) Since there are four layers in ResNet18, we choose to decouple the network after the first, second, and third layers, resulting in MEMO-3, MEMO-2, and MEMO, respectively. Furthermore, we align the memory budget of different methods to MEMO-3 since it costs the largest budget. The ranking of different variations is the same as Figure 19(a) and find treating the
23

---

Published as a conference paper at ICLR 2023
90
Accuracy (%)
70
Accuracy (%)
80
10.261
4.761
70
50
DER
60
DER
MEMO
MЕMO
20
40
100
20
40
60
80
100
Number of classes
Number of classes
(a) VGG8
(b) Inception-V V3
Figure 20: Experiments when varying the network structure. We evaluate MEMO and DER with VGG8 and Inception-V3 on ImageNet100. MEMO consistently outperforms DER with different network structures.
last group of blocks as the generalized block is more memory-efficient, which consistently shows the best performance. We also verify this conclusion regarding other backbone structures, i.e., VGGNet (Simonyan & Zisserman 2014). There are five layers in it, and we follow the same protocol conduct experiments and show the results Figure [T]c] Observing the results with different network structures and datasets, empirically find that choosing the last layer the specialized block more memory-efiin -ffficeenf.............
Since the groups are combined with residual blocks, we can also decouple the network from the middle of the layers. Taking ResNet32 as an example, we can decouple the network after residual block 11. after residual block 12, etc. We also experiment following the former settings, and the results are shown in Figure [9(d) We denote the variation which decouples the network after the n-th residual block MEMO-Bn. It implies that decoupling from the middle of the group is not good choice. A probable reason is that the decoupling harms the inner characteristics of the residual groups, resulting in inferior performance. Hence, do not conduct fine-grained model decoupling and treat the residual groups as the minimal unit in model decoupling.
To summarize, we suggest treating the last residual group in the network as specialized blocks when decoupling the layers, which is proven to be most memory efficccent among different network structures and datasets. This rule is consistent with the observations in Figure 3 that the last layer shifts more than others.
B.2 MEMO WITH OTHER BACKBONES
As discussed in the main paper, our implementation is based on ResNet, but the concept of MEMO can be applied any other deep network structure that relies on deep and shallow features. In this section, we conduct experiments with VGGNet (Simonyan & Zisserman 2014) and Inception (Szegedy et al. 2016) on ImageNet100. and compare MEMO with DER with the same backbone. Results are shown Figure 20
Other settings the same as in the main paper.
use VGG8 and Inception-V3. for implementation. VGG8 a relatively small backbone, and we can exchange the extra model size of DER into 412 exemplars for MEMO. Inception V3 is much larger, and the additional number of exemplars 2512 for Inception V3. We can infer from these figures that MEMO shows consistent improvement over DER these different network backbones, verifying that MEMO is a generalized protocol and can be applied to various kinds CIL tasks with various network structures.
24

---

Published as a conference paper at ICLR 2023
C EXTRA EXPERIMENTAL EVALUATIONS
In this section, give the extra experimental evaluations, including the last accuracy-memory curve. the gradient norm of all tasks, CKA visualization of different layers, multiple runs, and running time. We also conduct experiments ImageNet discuss which layer should be frozen in MEMO.
C.I LAST ACCURACY-MEMORT CURVE
There are two commonly used performance measures for class incremental learning. Denote the test accuracy after the b-th stage as As, the average accuracy A - 1 Σi= A, represents model's average performance with streaming data. The last accuracy Ag denotes the performance after the last learning stage. These two accuracy measures are commonly used to measure the CIL performance former works (Rebufh et al. 2017] Zhao et al. 2020; Yu Aizawa 2019 Wu et al.) 2019). We provide the performance-ment with the last accuracy Figure 21
curve with average accuracy in the main paper and report the curve
We can infer from these figures that the observations in the main paper still hold, e.g., we observe the intersection between exemplar-bar based methods and model-based methods in CIFAR100. We do observe the intersection on ImageNet100, but the performances of DER and WA are relatively close at the start point. The main reason is that the performance of the 4-layer ConvNet with a million parameters is enough to obtain diverse feature representations (as discussed in Figure [3]. We can infer from Figure 21 that the intersection will be observed given a smaller memory size. Besides, the rankings between methods not changed between these methods, and MEMO outperforms others by a substantial margin in most cases.
80
60
Last Accuracy (%)
Accuracy (%6
55
60
50
Intersection
45
DER
Replay iCaRL
MEMO
40
Replay iCaRL
MEMO
WA
40
30
7.6
12.4
16.0
19.8
23.5
329
493
755
872
1180
1273
Memory Size (MB)
Memory Size (MB)
(a) CIFAR100
(b) ImageNct100
Figure 21:
Last accuracy-ment memory curve for CIFAR 100 and ImageNet100. Other settings are the same as the main paper. The order between these methods still holds.
C.2 WHICH LAYER SHOULD FROZEN?
In the main paper, discuss the choice of fixing/umfixing the specialized and generalized blocks with empirical evaluations on CIFAR100. We report similar observations on ImageNet100 Figure 22
The main conclusions of these datasets consistent with the former Firstly, specialized blocks should be frozen. This conclusion is observed by comparing the results that Φ (x)) has better performance than d. (x)),, and d. (0, (x)) has better performance than d. y (x)). Secondly, by comparing the best strategy the different settings,
infer that fixing not the generalized block depends on the number of classes the first incremental task. It can be seen that &. (0y has the best performance with base classes, while de (Ø (x)) has the best performance with 50 base classes. The main reason behind these phenomena is the definition of 'good' generalized block. When the base classes are large enough, training these classes can obtain diverse and transferable generalized representations. By contrast, if the base classes small with few classes, training these
25

---

Published as a conference paper at ICLR 2023
100
80
Accuracy (%)
Accuracy (%)
60
60
40
(x(),(2))
+- + (x)
(x()((2))
b (0)
(1))
F, (6 x))
(1,)(a)
6, (6 (2))
20
20
20
40
60
80
100
20
60 Number of classes
40
80
100
Number of classes
(a) ImageNet100. BascO Inc10
(b) ImageNet100, Base50 Inc10
Figure 22: Experiments about specialized and generalized blocks. Specialized blocks should be fixed, while fixing not generalized blocks depends on the number of classes in the base stage.
100
110
180
+.18/011 316/41
6,66/01
10.0
Gap2257
Acaraay tho
G.p-6.90
28
Narber
(a) Base5 Inc5
(b) Basel0 Inc5
(c) Basc20 Inc5
(d) Basc25 Inc5
Gap--194
Gap-1.77
G.p-2.74
G.
Number of chancr
Namber afbbroofchan chena
Namber chancs
Narber of ckac
(c) Basc30 Inc5
(f) Base35 Inc5
(g) Base40 Inc5
(h) Basc50 Inc5
Figure 23: Comparison of (x, and (,, given diffcrrt base classes on CIFAR 100 dataset. We annotate the gap of the last accuracy between é, (0, and 5. (,, (x)) at the end of each line. Fewer base classes prefer trainable generalized blocks, while more base classes prefer frozen generalized blocks. The intersection among them emerges around 20 base classes.
classes cannot obtain diverse and transferable generalized representations, and fixing the generalized block will harm the representation ability of the model and the final performance.
In real-world applications, we may not know the incremental learning implementations advance. Hence, it requires the model to design suitable strategy automatically choose the frozen layers. In Figure 23 we show a preliminary experiment by changing the number of base classes in CIFAR Specifically, vary the number of base classes among [5, 10, 20, 25, 30, 35, 40, 50) and show the incremental performance. In the former experiments, we already know that specialized blocks should be frozen. Hence, we only compare Φ. (x())(x)) and (,, (x)) in these figures.
As can infer from these figures, 6a(6,, (x)) (i.e, do not freeze the generalized blocks) shows better performance when the base classes are limited, e.g., fewer than 20. The gap between these strategies becomes smaller the number of base classes increases. When the base class number reaches 20, the intersection among them emerges. After that, 6, (í.e., freezing the generalized blocks) shows better performance. Hence, can treat 20
empirical threshold to define the learning protocol.
26

---

Published as a conference paper at ICLR 2023
1.0
1.0
Dynumic Generulized Blocks
Dynamic Generulized Blocks
Base Class Accuracy
New Class Accuracy
0.9
Frozen Generalized Blocks
Frozen Generalized Blocks
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.4
0.5
0
4
8
10
0
2
10
Incremental Stage
Incremental Stage
(a) CIFARIO0 B50 Inc5, Base class accuracy
(b) CIFAR 100 B50 Inc5. New class accuracy
1.0
1.0
Dynamic Generalized Blocks
Dynamic Generalized Blocks
Base Class Accuracy
New Class Accuracy
0.9
Frozen Generalized Blocks
0.9
Frozen Generalized Blocks
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
Incremental Stage
Incremental Stage
(c) CIFAR 100 BO Inc10. Base class accuracy
(d) CIFARIO0 BO Inc10, New class accuracy
Figure 24: Base and new class accuracy of different strategies. Freezing the generalized blocks helps to overcome catastrophic forgetting, while changing generalized blocks helps to adapt to new classes efficiently.
must noted that these figures only correspond to a specific case of the CIFAR1O0 100 dataset, which may be different in other learning scenarios. Designing proper metrics to measure the generalizability of the shallow layers is interesting future work, and a possible solution is to utilize the hold-out or wild data for evaluation. On the other hand, it would be interesting to explore how to strengthen the generalization ability of shallow layers, e.g., via meta-learning (Sun et al. 2019). self-supervised learning (Khosla et al.; 2020) and deep metric-learning (Wang al. 2017
C.3 INFLUENCE OF FREEZING GENERALIZED BLOCKS
In Section we empirically verify that the strategy to freeze not the generalized blocks is related to the number of base classes. this section, we explore the influence when freezing the generalized blocks. Specifically, we run the experiment under CIFAR1O0 BO Inc10 and CIFAR100 B50 Inc5 setting and report the results in Figure 24
In these figures, we separately record the accuracy of the 'Base' and 'New' classes along incremental stages. 'Base' classes denote the classes in the first incremental stage, i.e., in D while 'New" classes represent the classes in the latest incremental stage, i.e., in D³. As result, the accuracy of base classes represents the ability to resist catastrophic forgetting, while the accuracy of new classes implics the ability of the model adapt to new classes. These abilities are also known Stability-plasticiy
27

---

Published as a conference paper at ICLR 2023
dilemma' (Grosseerg 2012). It should be noted that incremental performance jointly considers the performance among all seen classes, and both abilities are essential in incremental learning.
As can infer from these figures, freezing generalized blocks shows better performance the base classes, which means can better resist catastrophic forgetting. However, when comes to new classes, dynamic generalized blocks show have better performance. The reason is intuitive that freezing the generalized blocks restricts the model from adapting to new patterns of new classes. In other words, the general layers are trained with the current task, which is not generalizable enough for new tasks if the current class data is insufficient. Under such circumstances, freezing these layers harms the learning ability of the model to adapt to new classes. On the other hand, since the model keeps an exemplar set of old class instances, jointly optimizing the general layers with exemplars and the current dataset helps to rectify it, making generalize all classes and enhancing the model's ability. It must be noted that the joint learning process can resist forgetting by rehearsing former instances.
To summarize, CIL requires the model to perform well among all seen classes. Hence, if the benefits of learning new classes surpass the loss of old classes, we should enable the model to adapt to new classes and not freeze the generalized blocks. By contrast, if there are numerous base classes and forgetting them shall significantly damage the performance, frrezzin the generalized blocks is a better solution. These results are consistent with Section C.2)
90
90
Replay
DER
iCaRL DER
+ Replay
MEMO
WAA
MEMO
WA
80
iCaRL
80-
Accuracy (%)
Accuracy (%)
70
60
60
50
20
40
60
80
100
50
Average Accuracy
LastAccuracy
Number classes
(a) Multiple runs
(b) Incremental performance
Figure 25: Left: The average and last accuracy of different methods. Error bars denote standard deviation. Right: Average incremental performance among five runs. The standard deviation is shown in the shadow region. The ranking of these methods is the same with different class orders.
C.4 INCREMENTAL LEARNING WITH MULTIPLE RUNS
typical setting of CIL defines the comparison protocol (Rebuffi al. 2017 Hou et al. 2019 Wu
let al. 2019 Liu et al.) 2020: Yu et al.] 2020; İhhao et al., 22000) Yan et al. 2021 Douil lard et al. 2021 to shuffle the class order with random seed 1993, and we follow this protocol to conduct the benchmark comparison in the main paper. In this section, we run the experiment multiple times with different random seeds and report the results in Figure 25(a) We choose random seeds from { 110, 20, 30, 40. 50} and run the experiments five times with CIFAR100, Base50 Inc5. We also show the incremental performance of each method in Figure 25(6) We plot the average performance and standard deviation (shown in the shadow region) of each method among five runs. As we can infer from the figure, the results are consistent with the conclusions in the main paper, and the order of different classes remains the same with the change of random seeds.
C.5 RUNNING TIME COMPARISON
Exemplar- based methods rely on revisiting former instances during new class learning. i.e., the model optimizes the loss term over D U in every incremental stage. Hence, adding exemplar size will
28

---

Published as a conference paper at ICLR 2023
15000
Replay
DER
WA
MEMO
Running Time (
iCaRL
10000
5000
Figure 26: Running time comparison of different methods on CIFARIO0. The running time of MEMO is at the same scale other methods.
correspondingly increase the running time these methods. On the other hand, model-based methods save the old backbones, which requires forwarding the same batch of instances multiple times with different backbones. These methods will increase the running time of class- inssemnncreeetal models, and we empirically analyze the running time of these methods on CIFAR100 BaseO Inc10 in this section. We report the running time in Figure 26
As we can infer from the figure, simply replaying with exemplars consume the least running time, while it gets the worst performance among methods. Adding the knowledge distillation can relieve catastrophic forgetting while substantially increasing the running time. On the other hand, expanding the models and saving old backbones obtains better performance. At the same time, it also increases the running time since single instance should be forwarded by multiple backbones in the learning process. To summarize, we find that exemplar-based and model-based methods have the same scale running time. In other words, MEMO achieves the best performance with the competitive running time, which more efficient for developing CIL models in real-world applications.
C.6 GRADIENT NORM OF ALL INCREMENTAL TASKS
In the main paper, provide the gradient norm analysis for single incremental task due to the page limit. We give the full gradients of all incremental tasks in this section, as shown in Figure 27 We infer from these figures that the trend of gradient norm still holds for other incremental tasks. To be specific, the gradients of deeper layers are larger than shallow layers for all incremental tasks.
C.7 CKA VISUALIZZATION OF DIFFERENT LAYERS
In the main paper, we use CKA (Kornblith al. 2019) to measure the similarity between different backbones learned during the incremental stages with ResNet32. We calculate the pair-wise feature similarities between the shallow layers (i.e., after residual block 5) and deep layers (i.e., after residual block 15) in the main paper. In this section, we provide the full CKA visualization of these residual blocks in Figure 28 As we can infer from these figures, the features of different backbones the same depth yield different similarities. The features are highly similar for the shallow layers, i.e., after residual block and residual block 10. In contrast, the similarity is diverse for deeper layers, i.e. after residual block 15. These results consistent with the choice of generalized and specialized blocks discussed in the main paper and the empirical evaluations Figure 19
C.8 COMPARISON WITH GAN- BASED METHODS
Generative models capable of capturing the distribution of the data and generating instances. A typical line work using GAN (Goodfellow et al., 2014) to memorize the distribution of former tasks and then replay them when learning (Shin et al. 2017a) considers saving an extra GAN model as the generator and incrementally updates when new data arrives. The classification model is optimized jointly with Dnew U Doen, where
tasks (He et al. 2018 Shin et al.) [207aa). Specifically,
29

---

Published as a conference paper at ICLR 2023
Block Iadox
12-
15
15
9000
4200
(a) Task
(b) Task
(c) Task
Block badox
-2
12-
12-
12-
4200
fiom
foom
(d) Task 4
(e) Task
(l) Task 6
Block lindos
6.
12
12:
12
1600
4210
5600
9000
1800
4200
5400
9000
0
1800
4208
5600
9000
Thention
Domition
Doeratin
(g) Task 7
(h) Task
(i) Task 9
Figure 27: Gradient norm of different tasks. The numbers are reported in log scale. Deeper layers have larger gradients while shallow layers have smaller ones.
1.0
0.75
Baddhone Index
0.90
08925
10:900
0.900
0.875
0.855
0.850
0.850
Bockhone Index
Bockhonse Index
Backbone Index
(a) Residual Block 5
(b) Residual Block 10
(c) Residual Block 15
Figure 28: CKA visualization of different layers. The larger block index indicates deeper layers. The features extracted by deeper layers of different backbones are dissimilar, while that of shallow layers are similar.
Dncw stands for the incoming dataset and Døen stands for the generated dataset from the former distribution. However, incrementally updating single GAN model will also incur catastrophic
30

---

Published as a conference paper at ICLR 2023
forgetting. Hence, (He et al. the forgetting phenomena in GAN updating. It requires saving multiple generative models in the memory, which costs more memory the data stream evolves. Apart from saving GANs, (He et al., 2018) finds it useful to save exemplars from the former distribution, and optimizes the model with D
2018) considers training GAN per incremental class to resist
U Dgen U.
In this section, we re-implement GR (Shin et al. 2017a) and ESGR (He et al. 2018) and compare them to our proposed MEMO with CIFARI00 dataset. We follow the implementations in the main paper to organize a 'Base O Inc 10' setting. ESGR requires saving multiple GANs, which costs a much higher memory budget, and we do not align the cost of it to the others. We follow the original paper to use WGAN (Arjovsky et al.] 2017) as the generative model and implement it with four transposed convolutional layers. The optimization details (rounds, learning rate, optimizer) are set according to the original paper. We report the results in Table 25 and Figure 29
We can infer from these results that training a GAN costs a large number of parameters, which is also hard to optimize for complex image inputs. Our proposed MEMO outperforms these methods by substantial margin, even with much fewer memory budget. As a result, we do not compare MEMO to these GAN-based methods in the main paper.
+
MEMO
80
ESGR
Method|
[E]
S(E)
Model Type
# Parameters Model Size
20.391
Accuncy
60
GR
3658
10.7IMB
ResNer32 + WGAN
3,35M
12.78MB
ESGR
3300 9.66MB
ResNet32 100"WGAN
166.39M
634.75MB
MEMO
|3300 9.66MB
ResNet32
3.62M
13.83MB
20
25
50
75
100
Table 25: Implementation details when comparing to GAN-hased methods.
Number of classos
Figure 29: CIL performance
100
100
100
SPM
MEMO
SPM
MEMO
MEMO
Accuracy (%)
80
Accurscy (96)
Accuracy (%)
50
17:541
10.11
7.141
60-
60
60
20-
20
20
0
25
50
75
100
25
50
75
100
25
50
75
100
Nomber of classes
Number of classes
Number classes
(a) CIFAR100 BaseO Inc5
(b) CIFAR100 100 BascO Inc10
(c) CIFAR100 BaseO Inc20
SPM
MEMO
100
SPM
MEMO
100
SPM
MEMO
80
3,021
Accuracy (%)
80
4.987
Accuracy (%)
Accuracy
60
3.48
40
40
20-
20-
60
0
25
50
75
100
30
75
100
25
100
Number of classes
Number of classes
Number of classes
(d) CIFAR100 Base50 Inc10
(c) CIFAR100 Base80 Inc10
(f) CIFAR100 Basc80 Inc5
Figure 30: Experiments when comparing our proposed method to SPM. MEMO consistently outperforms
SPM with different benchmark settings.
C.9 COMPARISON WITH MULTI-BRANCH MODEL
A recent work SPM Wu et al. 2022) also considers the multi-branch model for itasseinnreemttlll learning. should be noted that SPM is designed for CIL with vast base classes, egg., with 500 or 800 classes in the first stage. However, there are not so many base classes in the benchmark CIL. setting,
31

---

Published as a conference paper at ICLR 2023
and re-implement SPM under the benchmark setting for comparison. In the implementation, train a model from scratch with the base dataset for incremental learning.
Since SPM also decouples the backbone network into two parts, follow the implementation in the original paper (Wu et al. ) to decouple the representation, i.e., before the last convolutional block. Hence, the way of expansion in SPM is the same MEMO, making the model size almost the same (the only difference lies the fully connected layers, which is negligible).
Following the comparison protocol defined in the main paper, with the same exemplar size. Since the model size and memory size of SPM and MEMO is equal, the comparison is fair for them. The results are reported in Figure (30] We infer from these figures that MEMO consistently outperforms SPM on these benchmark settings. The comparison to the contemporaneous multi-branch method verifies the effectiveness of our proposed method.
compare these methods fairly
5e-5
5e-5
+D'
+
D
+
+DD'
D²
-
D
Dt
- D²
+
Mean Square Ervor
Mean Square Error
3e-5
3e-S
le-5
le-5
5
Layer Index
Layer Index
(a) Layer-wise shift (VIT. Wo, WK. Wv. MLP)
(b) Layer-wise shift (VIT, We, WK. Wv)
Se-6
Sc-6
D'
D'
+-2
D
Mean Square Lror
Mean Square Error
3e-6
3e-6
le-6
1e-6
10
10
Layer Index
Layer Index
(c) Layer-wise shift (Bert, Wo. WK, Wv. MLP)
(d) Layer-wise shift (VIT, Wo, WX, Wv)
Layer- wise shift of VIT and Bert. The observations in residual networks still hold for these network structures and NLP datasets.
Figure 31:
C.10 DIFFERENT BACKBONES AND DATASETS
In the main paper, we explore the network behavior with ResNet (He et al. 2015) on image datasets, and find shallow layers stay unchanged while deep layers change in class dass-incremental learning. In this section, we verify this conclusion with different network structures and different incremental datasets.
Specifically, we first explore the behaviors Vision Transformer (Dosovitskiy et al. 2020) in incremental tasks. We directly the experiments CIFAR100 BaseO Incl0 with 6- VIT.
32

---

Published as a conference paper at ICLR 2023
Following the experiments in the main paper, we trace the weights in each Transformer encoder (including the weights of multi-head attention and MLP) and show the trends in Figure(31(a)) (1 In Figure (ag we observe that the layer-wise shift of deeper layers is more obvious than shallow ones. We also trace the shift of multi-head attention blocks, i.e,, Wo, WK. Wy in Figure (1 and find the trend is consistent.
Additionally, we explore the behaviors of the network in NLP tasks. We follow (Ke et al. 2021 to use Bert (Devlin et al. 2018) for ASC dataset (Ke et al., 202I) (the aspect sentiment classification dataset containing 19 products), which is benchmark NLP task. We keep the other settings the same and trace the shift of Transformer layers in Bert and draw the trends Figure 31(c) 3l(d) These results show the same trend as VIT.
As we can infer from these figures, the layer-wise change shows that shallow layers change less while deep layers change more, sharing the same trend as in residual networks. Hence, we can summarize that in various settings, shallow layers stay unchanged while deep layers change more in class-incremental learning.
D IMPLEMENTATIONS
D.1 DISCUSSIONS ABOUT RELATED WORK
glass-incrementee learning is now a hot topic in the machine learning field, where new methodologies emerge frequently. We discuss CIL methods by dividing them into two groups, e.g., exemplar-based and model-based. These groups either save exemplars or models to boost their performance.
However, there are other methods that do not fall into these groups, e.g., (Kirkpatrick et al. 2017 Li
& Hoiem| 2017 Jin et al., 2021; Pourkeshavarzi et al. 2022). Since they are designed to conduct incremental learning without extra memory budget, these methods often perform inferior to those discussed in the main paper, even equipped with additional exemplars. On the other hand, there methods that address memory-efficiiene addresses memory-efficient CIL by saving the embeddings instead of raw images, (Zhao et al.) 2021c
from other aspects. For example, (Iscen et al.) 2020
suggests saving low-fidelity exemplars, and (He et al. 2018 Shin al., 2017a) address the problem by generating exemplars with generative models. It should be noted that saving embeddings will suffer the embedding drift phenomenon (Yu al. (2020) and requires an extra finetuning process to fix such drift. The bias of the embedding adaptation process will accumulate, which works poorly incremental learning with multiple stages. We re-implement (Iscen et al., 2020) and find it shows inferior performance than saving exemplars. Generating exemplars with generative models will consume the memory budget to save generative models, which also suffer catastrophic forgetting (Cong et al. (020) and fail for large-scale images (Iscen et al.) 20200 (See Section C.8 There are other methods designing multi-branch network structures, e.g., A AANtts (Liu et al.) 2021a), PNN (Rusu et al.) 2016) and SPM (Wu et al. 2022). The extra branch is designed to re-scale the network outputs (Liu et al. 2021a or get multi-hear predictions (Wu et al. 2022). which requires an extra tuning process routing design. By contrast, our proposed method does require the extra model tuning and shows stronger performance (See Section C.9). As result, we mainly concentrate on the discussions about typical CIL methods that rely on extra memory from the efficiency and model complexity perspective. Below are the introductions to compared methods in this paper.
Replay (Chaudhry et al. 2019) is an exemplar- based baseline that simply optimizes the cross-entropy loss with 2 E in every incremental stage;
iCaRL (Rebuffi et al. 2017) builds knowledge distillation (Zhou et al.) 2003 Zhou &
Jiang 2004 Hinton et al. 2015) regularization term to regularize former classes from being
forgotten. The loss (cross-entropy and knowledge distillation) is optimized with D U E in every incremental stage;
WA (Zhao al. 2020) extends iCaRL with weight aligning, which normalizes the linear
layers to reduce the negative bias;
DER Yan et al. 2021) is a madel-based method that saves backbones to resist catastrophic forgetting. Apart from the loss term discussed in the main paper, it also introduces an auxiliary loss to encourage diverse representations and a sparse loss to conduct network
pruning.
33

---

Published as a conference paper at ICLR 2023
In this paper, implement DER with two modifications the original implementation. Firstly, the original DER utilizes different backbone networks than other methods, e.g., modified ResNet18 for CIFAR LOFARI0O, while other methods utilize ResNet32 as the benchmark backbone. In this paper, report the benchmark comparison when using the same backbones for DER and other exemplar-based methods (in Section 5.1 of the main paper). Secondly, DER claims reduce the parameter number. But it shows that the pruning has little effect on the total parameters and harms the final performance. On the other hand, the pruning code is not open-sourced yet As result, we re- re-implement DER without the pruning loss to report the results in this paper.
pruning algorithm to
Recently, SPM (Wu et al.] (022) finds that fixing the generalized block is effective for class- incremental learning with vast base classes and proposes a multi-branch method for CIL with strong pre-trained models. The findings SPM only correspond to a tiny portion of ours, and we list the differences below:
Different Settings: SPM concentrates on the incremental learning scenario with a strong pre-trained model, which requires 500 or 800 base classes in their setting. However, we are focusing on the typical class-incremental learning setting, where the model should be trained from scratch with few classes (say, 5 or 10). Our analysis in Section 4.3 is also irrelevant to the strong pre-trained models.
Different Observations: The observation in SPM is highly driven by their setting, i.c., they focus on how to design incremental models with vast base classes and find freezing the generalized block can facilitate model learning. However, the findings in SPM are only a tiny portion of our conclusions in Figure 5. Apart from the aforementioned findings, we also summarize that if the base classes are insufficient, fixing the generalized block will lead to inferior performance. Besides, we also find that the specialized blocks of former tasks should be frozen in all cases, which is also not discussed in SPM.
Different Network Behaviors: It should be noted that none of the conclusions in Section 4.3 can be found in SPM, where we analyze from the aspect of network behaviors (gradient norm, shift between blocks, and CKA between backbones). As a result, the conclusions in SPM only correspond to a tiny portion of ours and should not weaken our novelty.
Different Methodologies:
Although our MEMO and SPM both adopt the buati- structure; the core difference that SPM adds a new specialized block and fully-connected layer for each new task. It requires calibration between old and new classes, while ours will not face this problem since we update a larger fully-connected layer as Eq. 2 after each incremental stage. Besides, facilitated by the simple and effective training protocol, our method does not need any hyper-parre parameter to control the trade-off between loss terms.
We give the comparison of our proposed method SPM in Section C99 where we verify that our proposed method is more suitable for the benchmark class-incremental learning scenario.
D.2 IMPLEMENTATION DETAILS OF MEMO
Similar to DER, there two loss terms in MEMO. Apart from the cross-entropy loss discussed in the main paper, the auxiliary loss aims to differentiate between old and new classes. Denote the b-th incremental stage dataset Do contains |Y| classes, we create an extra classifier WA E [dxx||]++1
The auxiliary loss is represented by:
11++1
Lx(x,0)
Σ
- I[ū k) log SW (x()))
(4)
where da is the b-th embedding created for Db_ ÿ reassigns the ground-truth label into |Y/] classes, and treat y & Y, as class [Y)] + 1. Eq. 44 helps to acquire diverse feature representations for each embedding backbone, and the auxiliary classifier WA is dropped after each task training. The final optimization MEMO combines the cross-entropy loss discussed in the main paper and the auxiliary loss in Eq. We also weight normalization to eliminate the bias in the classifier layer.
pyttrch
34

---

Published as a conference paper at ICLR 2023
D.3 EXEMPLAR SELECTION
As discussed in the main paper, we follow (Rebuffi et al. 2017) to select the exemplars in the exemplar set & with the herding algorithm (Welling 2009). It aims select the most representative instances per class by the distance to the class center. Denote the current embedding as ((-). Given the instance X (x1. X2, Xn from class y. the target is select representative instances from X. We first calculate the class mean via: < 1 _i=1 (xx)). We then calculate and rank the distance of |l
each instance to the class center || <(x;)] ) in ascending order. The exemplar set the collection
of the top-m instances with the least distance.
D.4 MODEL COMPRESSION
This paper mainly discusses how to allocate the memory budget for meass-incrementll models, i.e, investigating memory- memory-efficient way allocate the model and exemplar given specific memory budget. We do not aim to design extra model compression algorithms or post tuning methods to reduce the total parameters. By contrast, concentrate saved//roopped. Given the fact that there are many methods to conduct model compression (Deng al., 2020), e.g., knowledge distillation (Wang et al., 2022). pruning (aan al. 2021). low-rank factoniaation (Sainath et al.) 2013) and quantization Yang et al. (019), we believe they can be combined with our method orthogonally interesting future work.
which part of the model should be
D.5 BROADER IMPACT
In this work, study the maassinncremntall learning problem, fundamental problem in machine learning. Specifically, we first address the fair comparison among different methods by aligning the memory budget and propose several performance measures for holistic evaluation. We then observe the differences between different layers in the incremeacreett learning process and propose simple yet effective baseline method efficiently organize the memory budget. Our work will give instructions for applications with difficulties managing the memory size for CIL models, At the same time, there is still much room for exploration in this work. We hope our work can inspire more discussions about glass-incremental learning in real-world applications and drive more research to build practical and memory-efficien CIL models.
Meanwhile, we are aware that the abuse of this technology can pose ethical issues. In particular, we note that people expect that learning systems will not save any personal information for future rehearsal. While there risks with this kind of AI rescarch, we believe that developing and demonstrating such techniques is essential for understanding valuable and potentially troubling applications of the technology. We hope to stimulate discussion about best practices and controls on these methods around responsible technology uses.
35


Reviews:
[{"review_id": "nHv8T-zFZuS", "reviewer": "Reviewer_aBa9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets the problem of Class-Incremental-Learning with two major contributions. First, the authors propose to evaluate previous methods with aligned memory budgets to fairly compare exampler-based and model-based approaches. Second, the authors conduct empirical observations on the effect of different layers during model updates. Based on the observation, the authors propose MEMO to create deep layers for new tasks to save memory budget.", "summary_of_the_review": "Overall, this paper could bring useful insight to the community. (1) It provides a practical evaluation perspective to compare orthogonal direction. (2) Its empirical observation on the gradient norm and shifting range could help understand the training dynamic of Class-Incremental-Learning. (3) The proposed method is a simple and intuitive next step for model-based methods. ", "strength_and_weaknesses": "Strength:\n\n(1)\tReasonable experimental analysis. The authors conduct extensive experiments to investigate the pros and cons of exampler-based methods and model-based methods given the same memory budget. It will also be interesting to know where the regularization-based methods sit.\n\n(2)\tSimple method based on interesting observation. Observation of the effect of different layers is informative. Given the observation, MEMO seems to be a reasonable proposal to save memory for the model-based method.\n\n\nWeakness: \n\n(1)\tThis paper focuses on vision benchmarks, which is understandable when comparing performance. However, for observational conclusions such as \"shallow layers stay unchanged while deeper layers change more during incremental learning,\" experiments could cover more settings. For example, have the authors looked at Vision Transformer as the transformer is also popular architecture for vision tasks? Have the authors investigated datasets from another domain, e.g., NLP? Does the observation conclusion still hold in various settings?\n\nQuestions:\n\n(1)\tMEMO only extend specialized layers to save memory budget for exemplars. One choice here is what layers are considered specialized. The last layer? The last two-layer? The authors provide a study in SectionB.1 using ResNet32. Does the result here generalize to other ResNets and other architectures? What would the authors recommend as the rule of thumb in deciding the specialized layer?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "i8AhF8t_nlA", "reviewer": "Reviewer_dAkF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors explore how to balance the memory budget usage between exemplars and model parameters in class-incremental learning. Based on their findings, the authors propose a simple yet effective class-incremental learning method named Memory-efficient Expandable Model (MEMO). MEMO extends specialized layers based on the shared generalized representations, efficiently extracting diverse representations with modest cost and maintaining representative exemplars. Extensive experiments on benchmark datasets validate MEMO’s competitive performance.", "summary_of_the_review": "Overall, I think this is a high-quality paper. It is well-written and easy to follow. The task studied is very meaningful in my view. I recomannd acceptance. ", "strength_and_weaknesses": "### Strengths\n\n- It is very meaningful to explore how to balance the memory budget usage between exemplars and model parameters in CIL. Saving the model parameters definitely requires a memory budget.\n\n- This paper is well-written and easy to follow. \n\n- Extensive visualizations and figures are provided. \n\n### Weaknesses \n\n- It would be better to discuss some strategies that help to decide which blocks should be frozen. In Figure 5, we can observe that different blocks should be frozen in different settings (e.g., Base0 vs. Base 50). However, we cannot know these settings in advance in real-world applications. Thus, it is important to design some methods that help us to decide how many blocks should be frozen. \n\n- It would be better to include a table in the main paper to show the accuracies of different baselines. Currently, Figure 6 shows the Top-1 accuracy. However, it is more clear to have a table with numerical results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "_ZGssYlmPS", "reviewer": "Reviewer_GaiS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to tackle the class-incremental learning (CIL) problem from a novel perspective. The current class-incremental learning community is somehow chaotic, where more and more methods tend to design extra model components. These additional components are not counted in the total memory budget, resulting in an unfair comparison between different methods. To this end, this paper unifies the comparison of different methods by introducing the extra dimension for comparison, i.e., memory budget. By varying the memory budget of different methods and designing new performance measures, this paper aims to evaluate incremental models holistically. The results are diverse, where strong models shall fail with limited memory budgets. These results inspire the authors to analyze the importance of each part in CIL and design a simple yet effective method that suits the characteristics of CIL models. Vast experiments on benchmark datasets verify the effectiveness of the proposed method.\nI really like the way that the authors study the CIL problem. They performed very systematic and reasonable experiments. The second part of the paper, which looks at the characteristics of different layers, is interesting. Also, the proposed approach (MEMO) just decouples the representations and expands the task-specific layers to strike a trade-off between model expansion and exemplar saving. It is a very interesting way to solve the problem.", "summary_of_the_review": "Finally, I found the paper clear and easy to read. All the experiments are detailed, as well as the thought process behind them. I think this is a thorough study of a very important problem and can improve many papers that are based on it. As a result, I vote for acceptance.", "strength_and_weaknesses": "Strengths:\n- New empirical findings shall shed light on the class-incremental learning community, e.g., the experimental evaluations in Figure 3 show that deep and shallow layers have different characteristics in CIL. The results are clear and straightforward, which correspond to the intuitive solutions in model expansion.\n- Intuitive method to solve CIL problem. Observing that shallow layers yield higher similarities than deeper layers, the authors argue that expanding deep layers is enough for CIL. The extra memory budget for expanding shallow layers can be exchanged for saving exemplars, which is more memory-efficient. Experimental results verify that the current method can boost performance for free with the same budget.\n- Novel performance measures to holistically and fairly compare different CIL methods. I agree with the point that current CIL methods are not compared fairly, especially for the model-based methods with multiple backbones. I think the new performance measure and the way of model evaluation shall inspire the community to design fair comparisons in the future.\n\nWeaknesses:\n- The current evaluation protocol depends on some specific and unusual network structures, e.g., ConvNet2, ResNet14, etc. As a result, it would be better for the authors to release the systematical codebase for this research in the final version, which will ease the burden when calculating these new performance measures.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ZxP5TEY-i1C", "reviewer": "Reviewer_FkFJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "It is known that methods that store examples in memory have positive results for the Continual Learning problem, specifically in Class Incremental Learning. Instead of saving examples from previous tasks, some works recommended storing model checkpoints to retain the information. The authors of this paper point out that unfair comparisons are often performed since the model buffer is not always considered part of the saved memory. Because of this, the authors propose to compare the methods by adding both memories (exemplar and model), revealing that weight-saving methods tend to suffer when memory is limited. With these results, the authors propose MEMO, a more efficient method to store checkpoints. MEMO takes advantage of the fact that the first tends to have generic information, keeping those shared across tasks and only adding later layers to incorporate specific knowledge. The newly freed memory can then be used to save more examples of previous tasks. MEMO achieves comparable results and sometimes outperforms current methods in different scenarios.", "summary_of_the_review": "Despite having limited contributions in my opinion, the ideas presented are good and may be interesting for the community.\n\nHowever, the authors leave many questions raised that, if answered, could significantly improve the work. In my opinion, this research work is being developed and has not yet reached its maximum potential.", "strength_and_weaknesses": "S:\n- Based on current works, the community agrees with the authors that there is a need to improve how we compare methods. A suitable example of this is the framework proposed in [1].\n- The idea of reducing the number of layers saved in the \"memory buffer\" to save more examples or using better models is good. The authors' experiments that show that these layers store general information and take advantage of that, are intriguing.\n\nW:\n- The CL definition is unclear, making it unclear whether or not the authors employ task-ids during inference. If they use the same definition as Yan et al., 2021. One can assume they do not use the task id. However, it would be helpful to make it clear.\n- \"… Since a single backbone can depict a limited number of features, learning and overwriting new features will undoubtedly trigger the forgetting of old ones …\" I would be careful with these kinds of statements. It may seem accurate at first, but we do not know how many concepts can be stored in each feature. For example, the model rarely used the entire weights based on the Lottery Tickets Hypothesis. Therefore, treating this as a limitation without giving further reasons is complex.\n- In implementation details, it would be helpful to know how many times the experiments were run. Running the experiments only once is not good practice in CL, since the variance between different initializations and task orders can be significant.\n- The motivation of Fig1 is straightforward but may be out of place. Several things are not explained until Sec4 (for example, Base50, Inc5, or MEMO). The explanation in the text could be improved or Fig1 could be left in Sec4.\n- In sec4.2. Since the authors advocate a fair comparison, I do not think it is right to change the base model. Each model has its characteristics that can harm or benefit some methods or scenarios. The results in Fig2 validate this concern.\n- Sec4.3. The experiments are interesting, but as the authors mention, it is something that is known. It may be possible to reduce this section and add experiments about how to properly divide general and specific layers. There is no mention in the paper of how this division is made.\n- Sec5.2. If models are in a similar memory context, why not just compare ACC? The motivation for using AUC is not straightforward.\n\nQ:\n- Understand that this paper and [2] are different. In your opinion, what is the motivational difference between them?\n- Could give an intuition for why not freezing the general layers improves the acc? If there is general knowledge in those layers, why change it? How much can we modify those layers and still prevent forgetting?\n\n[1] Mundt, Martin, et al. \"CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability.\" arXiv preprint arXiv:2110.03331 (2021).\n[2] Ostapenko, Oleksiy, et al. \"Continual learning via local module composition.\" Advances in Neural Information Processing Systems 34 (2021): 30298-30312.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%