Wakening Past Concepts without Past Data: Class-incremental Learning from Placebos
Authors: Yaoyao Liu
Bernt Schiele
Qianru Sun
Venue: ICLR 2022 Submitted (2022)
Keywords: incremental learning
continual learning
class-incremental learning

Abstract: Not forgetting knowledge about previous classes is one of the key challenges in class-incremental learning (CIL). A common technique to address this challenge is knowledge distillation (KD) that penalizes inconsistencies across models of subsequent phases. As old-class data is scarce, the KD loss mainly uses new class data. However, we empirically observe that this both harms learning of new classes and also underperforms to distil old class knowledge from the previous phase model. To address this issue, we propose to compute the KD loss using placebo data chosen from a free image stream (e.g., Google Images), which is both simple and surprisingly effective even when there is no class overlap between the placebos and the old data. When the image stream is available, we use an evaluation function to quickly judge the quality of candidate images (good or bad placebos) and collect good ones. For training this function, we sample pseudo CIL tasks from the data in the 0-th phase and design a reinforcement learning algorithm. Our method does not require any additional supervision or memory budget, and can significantly improve a number of top-performing CIL methods, in particular on higher-resolution benchmarks, e.g., ImageNet-1k and ImageNet-Subset, and with a lower memory budget for old class exemplars, e.g., five exemplars per class.

Under review as a conference paper at ICLR 2022
WAKENING PAST CONCEPTS WITHOUT PAST DATA: CLASS-INCREMENTAL LEARNING FROM PLACEBOS
Anonymous authors Paper under double-blind review
ABSTRACT
Not forgetting knowledge about previous classes is one of the key challenges in inassinncrementll learning (CIL). A common technique address this challenge is knowledge distillation (KD) that penalizes inconsistencies across models of subse- quent phases. As old-class data is scarce, the KD loss mainly uses new class data. However, we empirically observe that this both harms learning of new classes and also underperforms to distil old class knowledge from the previous phase model. To address this issue, we propose to compute the KD loss using placebo data cho- sen from a free image stream (e.g., Google Images), which is both simple and surprisingly effective even when there is no class overlap between the placebos and the old data. When the image stream is available, we use an evaluation func- tion to quickly judge the quality of candidate images (good or bad placebos) and collect good ones. For training this function, we sample pseudo CIL tasks from the data in the 0-th phase and design reinforcement learning algorithm. Our method does not require any additional supervision or memory budget, and can significantly improve a number of top-performing CIL methods, in particular on higher-resolution benchmarks, e.g., ImageNet- Ik and ImageNet-Subsse and with a lower memory budget for old class exemplars, e.g., five exemplars per class.
INTRODUCTION
AI learning systems expected to learn new concepts while maintaining the ability to recognize old ones. In many practical scenarios, they cannot access the past data due to the limitations such as storage data privacy but required to recognize all seen classes. Motivated by this, Rebuffi jet al. (207)) formulated the so-called class-incremental learning (CIL) problem, where training data of different classes gradually comes phase-by-phase and the model keeps re-training on new class data and is evaluated on both new and old classes. It is obvious that re-training in each new phase tends to override knowledge acquired from old data (McCloskey & Cohen 1989 McRae & Hetherington 1993 Ratcliff 1990), which is usually called "catastrophic forgetting" To alleviate this issue, most class- indremennaeetttlll learning methods (Rebuffi et al.] [2017] [Li & Hoiem 2016 Hou et al. 2019 Douillard et al. 2020 Liu et al. [220aa] 2 are equipped with knowledge distilla- tion (KD) loss that penalizes any feature or prediction inconsistency between the models adjacent phases. The ideal KD loss in each phase should be computed on the original old class data, which, however, is impossible in the constrained settings of CIL. Existing methods use new class data as the substitute (to compute the KD loss). We argue that this 1) hampers the learning of new classes it distracts the model from fitting the ground truth labels of new classes, and 2) falls short of distilling the knowledge of old classes since using new class data can not generate the same soft labels (or features) as using old data. Quantitative results are shown in Figure (hhre "old" denotes the upper bound (when using old data) and "new" shows rather poor results when using new data for KD.
In this paper, instead, we propose to search placebo images online to compute the KD loss for old classes. Ideally, the distillation effect of using our placebo images is close to that of using original old data. To optimize this effect, we propose a reinforcement learning (RL) algorithm where we define the state, action and policy function under the constraints of the CIL setting. The aim of the policy is to produce phase-specific evaluation functions (for evaluating the quality of placebo images) that are able handle the dynamic states along with incremental phases, e.g., the function
We made the corresponding changes in the revised paper and colorized these changes deep hlue.

---

Under review as a conference paper at ICLR 2022
KD on
hew
placebos (ours)
old (upper bound)
new model ouiputs
old model ourputs
labels
60
descentt
=CeB (0.11 0. 0.6,0.2
50
+ ALKDD (0.5,
0.2,
ascent
(a) The CIL loss new class training sample
40
.ascent
C = CCE [0.11
0.1, 1 0,
0,
30
+ +ACKD( (0.4,
New
Old
New
Old
ascent
LwF
iCaRL
(b) The CHI. loss for an old class training somple
Average accuracy (%) of new and old classes in the last phase, respectively us- ing LwF (Li & Hoiem 2016) and iCaRL (Rebuffi et al.) (017). The dark blue, light blue, and green bars indicate the results of using new data, placebos, and old data to compute KD losses, respectively. "KD on new" means computing the KD loss on both new data and old exemplars. All hyperparameters follow Hou et al. (219)) Right: Conceptual illustrations of the loss functions applied to new and old samples. The dark blue and orange numbers denote the predictions of old and new classes, respectively. It is clear in (a) that the objectives are different when using new class sample to do KD (the oracle case is to have both "ascent")), e.g., the ground truth label for the second old class is 0, while the "KD label" at this position is 0.8. This is not issue when using an old class sample, c.g., in (b), its ground truth label and "KD label" have consistent magnitudes at the same position (1 and 0.7, respectively). Please note the KD loss is softmax KI. divergence loss.
Figure 1: Left:
in later phase expected to handle more complex evaluations on the placebos of more old classes. The intuitive objective of the policy is to maximize the cumulative accuracy across all incremental phases. However, this does not comply with the CIL setting, where neither past nor future data are accessible. To overcome this problem, we propose to pre-train the policy on pseudo CIL tasks which can be built on any available categorical data, e.g., the data in the (-th When RI. is done (e.g., in the O-th phase), the policy function will be deployed in each incremental phase. Specifically, leverage this policy to generate an evaluation function according to the state of the phase and then use the function to estimate the quality of each candidate image (a good or bad placebo) in the batch loaded from a free and unlabeled data stream. We keep only high-quality placebos to compute the KD loss, after which we discard all of them and load another batch (and repeat the above steps).
first
We highlight that our method does not require any additional supervision or annotation. Storing the free data batch needs only tiny amount of additional memory, which few new class samples without harming the performance. This memory is reused for every batch, as the batch data are immediately discarded after computing the KD loss. Figure [ shows that our method (light blue) indeed alleviates the KD issue caused by using new class data (dark blue). encouraging that our method achieves the upper bound performance for recognizing the old (new) classes when plugged into LwF (Li & Hoiem 2016) (iCaRL (Rebuffi et al. (017)). In the experiments below, we evaluate more extensively by incorporating it into multiple strong baselines such as PODNet (Douillard et al. 2020). LUCIR (Hou et al. (019) and AANets (Liu et al. (021a)) and inspect its performance carefully by conducting a series of ablative studies. Our results on three popular CIL benchmarks show clear and consistent improvements over the baselines, especially on higher-resolution image datasets (such as ImageNet-Subss and ImageNet-Ik) and with lower memory budgets for old class exemplars (e.g., five exemplars per class).
obtain by deleting
Our technical contribution is three-fold. 1) A generic CIL method that leverages unlabeled placebo data from free image stream to improve the KD effect for both logit distillation and feature dis- tillation methods. 2) A -enrfrrcement-lerrinnggg based algorithm that learns a policy to adaptively produce phase-specific functions to evaluate the quality of placebos. 3) Extensive comparisons and visualizations for our method in three CIL benchmarks, taking perfefforn models as baselines.
RELATED WORK
Incremental learning (also called "continual learning") trains a model using the data coming in a sequence of learning phases. Recent methods are focused either task-incrementae learning where in each phase a new task (dataset) comes that contains a new split across all classes, i.c., classes are all seen before (Chaudhry et al. 2019 Davidson & Mozer 2020 Hu et al. 2019) Riemer et al. 2019
2

---

Under review as a conference paper at ICLR 2022
Shin al. 2017 Zhao et al. 2020), or class- stass-incremental learning (CIL) where each phase contains the data of a set of new classes, i.c., classes are unseen before. This paper is focused on CIL. The key challenge in CIL is the forgetting problem-oled classes are forgotten in later phases. There are three main lines of work. Memory-based methods (Rebufti et al. 2017; Shin et al. 2017) preserve a small number of old class data (called exemplars) and replay the model on them together with new class data. Regularization-bee to consolidate previous knowledge when training the model on new data. The key idea is to enforce model prediction logits (Li & Hoiem] 2016 Rebuffi et al., 2017), feature maps (Douillard et al.) 2020 Hou et al., 2019). the topology in the feature space (Tao et al., 2020),
methods introduce regularization terms in the loss function
low-dimensional manifolds (Simon et al. 2021) of the new-phase model to be close to that in the pre-phase model.
Network-arrchitectur----sss
methods (Rusu et al. 2016 Xu & Zhu| 2018 Abbaii et al.) 2020 Liu
et al., 20) design "incremental network architectures" " by expanding the network capacity for new data or freezing partial network parameters to keep the knowledge of old classes. For example,
Yan et al. (202T) froze the previously learned representation and augmented it with additional feature dimensions from a new learnable feature extractor. Our method belongs to the Regularization-base
group. Some prior papers use unlabeled data to improve regularization. Lee et al. (2019) proposed a confidence-ben based sampling method to leverage unlabeled external data for their global distillation loss. Zhang al. (020)) trained separate model only for the new classes and consolidated both old and new models by exploiting unlabeled auxiliary data. Liu et al. (2020b) integrated an ensemble of auxiliary classifiers to estimate regularization constraints and used unlabeled data maximize the classifier discrepancy. Our method differs in two aspects: 1) we design an RL-based algorithm that learns policy to adaptively produce thase-specific functions evaluate the quality of the unlabeled data; 2) our method is more generic and can be easily plugged in both logit distillation and feature distillation methods to boost the performance.
Reinforcement learning aims learn how to let intelligent agents take actions in an environment to maximize a cumulative reward. Some incremental learning papers also deploy reinforcement learn- ing algorithms in their frameworks. Xu & Zhu (018) used RL to adaptively expand its backbone network when a new task arrives, i.e., use RL to determine how many and where to add convolution filters. Liu et al. 202Tb) used RI. to learn a policy to adjust the memory allocation between old and new class data dynamically along with the learning phases. In this work, they also proposed to create pseudo tasks using the dataset observed in the 0-th phase or transferred from another dataset. Our method differs in two aspects: I) our RL algorithm is designed for data selection rather than memory allocation; 2) RL is not mandatory in our method. Using heuristic distances such as cosine distance also works for our data section, and incorporating RL can boost the model performance.
PRELIMINARIES
Class-incremental learning (CIL) usually assumes (N+1) learning phases: an initial phase and N incremental phases during which the number of classes gradually increases till the maximum (Douil- lard et al. 2020; Hou et al. 2019 Hu et al. 2021 Liu et al. 2020a). In the initial (0-th) phase, data D 1:0 D ..... D., F. containing the traming samples of co classes, are used to learn the initial classification model Oo After this phase, only a small subset of Disco (i.e., exemplars denoted as Eleog = {E1, ..., E. ) can be stored in the memory and used as replay samples in later phases. We use C to denote the number of classes we have observed from the O-th phase to the i-th phase. In the i-th incremental phase, we get new class data D = (De..+1:.... ..., De, } of (c: q-1) classes and load exemplars E from the memory. Then, we initialize θ with O,- and train it The resulting model θ; will be evaluated with a test set for D Di:e.-
using E U U.D.-I+lse
Reinforcement learning (RL) aims to learn an optimal policy no (parameterized by の) for an agent interacting in an unknown environment (Williams) 1992, Xu & Zhu 2018 Zoph & Le 2017). In the CIL scenario, the agent observes the current state s; from the environment in each incremental phase, and then takes an action a: for this phase according to the policy To (a, Subsequently, the environment updated to new state 81+1 for the next phase and the reward r: is calculated to optimize the parameters of Ta (a,) (s:. Specifically, the learning objective of (T. is to max- imize the expected cumulative reward (i.e., the cumulative validation accuracy of all training CIL tasks) (() = E, [R] = E,, EE>. rı]. We use the reward because the target of the CIL system is at any time providing competitive classifier for the classes observed so far (Rebuffi et al. 2017).
3

---

Under review as a conference paper at ICLR 2022
data for CE loss new class batch d
new
target data stream
sample
new model e
class
CE loss
exemplars
batch
sample
memory
prototypes
placebo batch p
old model 1
unlabeled data U
placebos P
KD loss
free
sample
image
stream
data for KD loss
evaluation functions
(M)( Jm-1
Figure 2: The computation flow our method PlaceboCIL in the i-th phase, We have three kinds of input data: old class exemplars, new class data,and free stream of unlabeled images. We compute two losses: CE loss on e U d and KD loss on e U The way to construct placebos is as follows: 1) We use the evaluation functions {S (u) ) m=1 to output the score for each unlabeled sample. 2) We load a batch unlabeled data LA from the free image stream, and add K placebos with the highest scores for each old class to P. 3) We compute KD loss using the placebos then delete them from P. 4) When we use up the selected placebos, we repeat the selection steps.
CLASS-INCREMENTAL LEARNING WITH PLACEBOS (PLACEBOCIL)
we illustrate the computation flow our method PlaceboCIL in each incremental phase. We have three kinds of data: old class exemplars stored in the memory, new class data coming in the current phase, and a free stream of unlabeled images. To update the new model, we compute two losses: cross-entropy (CE) loss on the sampled batch of old exemplars and new class data; and KD loss on the sampled exemplars as well as the selected placebos. We elaborate the update process in Section 4.1 As shown in the lower branch of Figure 2 the evaluation is based on the prototypes of all classes observed so far, and the evaluation functions are generated by the policy pre-trained by RL (or heuristic distances without RL). We introduce our RL algorithm in Section 4.2
In Figure 2
4.1 TRAINING CIL MODELS WITH PLACEBOS
In the following, introduce how to evaluate the unlabeled image to get high-quality placebos, and how to use placebos to compute the KD loss in CIL. We summarize the overall training process of incremental phase in Algorithm
Building evaluation functions. We argue that high-quality placebos for the m-th old class should meet two requirements: (1) the placebos should be close to the prototype of the m-th class in the feature space because they will be used to activate the related neurons of the m-th old class in the model; and (2) the placebos should be far from the prototypes of other classes in feature space so that they will not cause the KD issues shown in Figure [ To achieve these, we design the following evaluation function Sm, (z) for the m-th old class in the i-th phase:
Ci-1
Sm (x) = - cos (x),
Σ Fo.(2)
+B:
Σ Cos Fo, TEnl SEE,
Σ Fe,
Em
G-1
ZEE.
n=1 ngf-
ProtoDist(a,Eon)
ProtoDist(arE_m))
CO8 Fo)
Foi(2) sED:
% C Ci-1 l=0-1+1
|Di]
ProtoDist(a.DD, 1+1:e)
4

---

Under review as a conference paper at ICLR 2022
Algorithm Training the CIL model in Phase (i 2 1)
Algorithm 2 Pre-trant tre-traiiing the policy functions
Input: New data D +he,, old exemplars E1
Input: Data D for generating pseudo CIL tasks.
model 1 and pre- pre-trained policy fuention me
Output: Policy function
Output: New model θ,, new exemplars Ext.s
3: Initialize di
Initialize O2 with 0;-1:
4: for w RL epochs do
4: Observe 8; and produce (3,,7) sT(()));
5: for in 1.... G do
5: for epochs do
6: Create a new pseudo task T using D;
6: Create (=)) using Eq. | and set P 2:
7: for in 1..., H do
7: while p == Ø do
Initialize classification model Og: for in 0,.., do
8: Sample L/ from the free image stream;
9:
Select placebos P c. 2 using Eq. 2
10:
if == 1 do
10:
for iterations do
11:
Train @ a conventional classification task;
11:
Sample
p. c. and
12:
else do
12:
Calculate loss L using Eq.
13:
Ohserve 8; and train O using Algorithm I;
13:
Update the CIL model O θ e.
14:
Compute validation accuracy r;;
14:
Update knockoffs P P p:
15:
Compute R Ei and update b;
15:
Select new exemplars S.., using herding.
16:
Compute (d)( (Eq. and update & (Eq.
where denotes unlabeled input image, Fe, (.) denotes the encoder of O and cos(;;) denotes the cosine similarity. ProtoDist(z, ), ProtoDist(z, E_m), and ProtoDist(r: (x, Dec1+ are the (average) cosine similarities in feature space between and of the m-th old class prototype, other old class prototypes, new class prototypes, respectively. 8 and 7/2 are two hyperparameters to balance the above three parts. If we set 3 = % 1. S,(z) will be simple and heuristic evaluation function, which is also effective according to our empirical results (see Section 5) To make the evaluation function more efficient, we propose train a policy using RI to generate phase-specific
B and 7/. and the details are given in Section
Selecting the placebos. At the beginning of the i-th phase, we empty the placebo buffer P. When- ever P empty, we load a batch of unlabeled samples U from the free image stream, and add K placebos for each old class to P as follows,
P = [
14-1K
arg max BLEl
Sm (Tx).
(2)
1=1
Calculating the loss with the placebos. After selecting the placebos, we sample a batch of new class data C De, 1:e, a batch old class exemplars e C Elsog and batch of placebos p C P. We calculate the overall loss follows,
f = LCB due) + ACx
where LCe and LKD denote the CE loss and KD loss, respectively. À hyperparameter balance the two losses (Rebuffi et al.) (017)). For feature distillation methods (Hou et al. 2019 Douillard al.] (020)). the KD issue is not a major limitation. However, we empirically find our placebos are still effective for retaining the old knowledge; thus, we use p U d UUe to compute the feature KD loss.
Memory management. We need a small amount of additional memory to store the unlabeled image batch U. To obtain this memory, we randomly delete [ samples from the new class data D., 1 1+:e The empirical results show the above step hardly degrades performance (see Section (5.2 Ablation study). During training, we delete from P immediately after calculating the loss. When we run out of the selected placebos in P, we load another batch of unlabeled data 2/ from the free image stream and repeat the selecting steps to get new placebos P.
4.2 LEARNING THE POLICY FOR EVALUATION FUNCTIONS
Because the ratio between old and new classes changes significantly when the number of phases increases, the evaluation functions should also change accordingly. To achieve this, we design an
5

---

Under review as a conference paper at ICLR 2022
RL system and train policy build phase-speciffe evaluation functions. The overall objective is to maximize the cumulative evaluation accuracy across all incremental phases. However, this is not compatible with the standard CIL protocol (Rebutfi et al., 2017) where neither past nor future data are accessible. To tackle this issue, we sample pseudo CII tasks and optimize the policy on them. In the following, first introduce the formulation of our RL system, including the definitions of the states, actions, and rewards. Then we show how to create the pseudo CIL tasks and train the policy. We summarize the learning process of the policy in Algorithm 2
Formulations of the RL system. States. We define the states based on the following two require- ments. (1) The states should be transferable between different CIL tasks, e.g., from a small-scale CIL. task including 100 classes to large one including 000 classes. It is because we need to transfer the policy learned from pseudo CIL tasks to the target task. So the states should also be transferable. (2) The state should be distinct in each incremental phase. This is to enable the state variable to represent a specific forgetting degree at each phase. To fulfill these requirements, we define the state in the i-th incremental phase as s; = 1 where (a - ca-1) is the number of new classes we observe in the i-th phase, and c,-1 is the number of old classes we have seen in all the
previous phases, Actions. We use a vector that consists of the phase-specific hyperparameters evaluation function the action, i.e., a, (3:: %)). When we take action a, we build evaluation functions (Smm [2)] with (8,,7) and deploy them in our CIL pipeline. Rewards. The objective of CIL is to train model that is efficient to recognize all classes seen so far. So it is intuitive to use the validation accuracy as the reward in each phase. In the i-th phase, the objective of the RL system
is to maximize the cumulative reward, i.e., R = E Ti, where denotes the validation accuracy in the i-th phase.
Creating pseudo CIL tasks. As we need to compute the cumulative rewards, need to access all training and validation data of the pscudo CIL tasks. Thus, an intuitive solution is using Di... (available in the 0-th phase) generate pscudo CIL tasks. Based on the CIL protocol (Douillard et al. 2020 Hou et al. 2019 Hu et al. 2021 Liu et al. (021a), Dicop contains half of the classes of the whole dataset, e-g., 300 classes on ImageNet et- Ik, which supplies enough data build downsized CIL tasks. When building the tasks, we randomly choose 10% training samples of each class (from Diseg) ) to compose pseudo validation set (note that we are not allowed to use the original validation set in training). To train more robust policy we generate pseudo CIL tasks with different class orders and different numbers of phases.
Training with REINFORCE. We elaborate the steps of learning policy as (parameterized by () in the following. The goal is to optimize & by maximizing the expected cumulative reward J(Ø). We denote any pseudo CIL task as T and its cumulative reward as R, and have,
J(~) = E, [R].
(4)
Policy gradient estimation. According the policy gradient theorem Williams 1992) we can com- pute the gradients for J [[], 6) follows,
J = BT
B V log n (a:ls:))
(5)
Following the REINFORCE algorithm Williams 1992). we replace the expectations Ey[:] and Ex, with sample averages (i.e., the Monte Carlo method (Hammersley (003)). Specifically, in each RL epoch, we create G pseudo tasks and run each task for H times. Thus we can derive the empirical approximation of (() as,
G H V Io T (a:lss)))) b),
N
ΣΣΣ GH
(6)
g=1h-1
where Rh: denotes the g-th reward for the h-th pseudo task Th. and b denotes the baseline function- the moving average of previous rewards. Using this bascline function is common trick in RL to
reduce the variance of estimated policy gradients (Rennie al. 2017 Zoph & Le 2017).
Updating policy parameters. We update @ (i.e., the parameters in π&) in each RL epoch using
gradient ascent (Xu & Zhu 2018 Zoph & Le 2017):
e = + V ((),,
(7)
where 17 is the learning rate. We iterate this update for w RL epochs in total.

---

Under review as a conference paper at ICLR 2022
20 exemplars/class
10 exemplars/class
exemplars/class
Method
Average
Last
Average
Last
Average
Last
LwF
53.19
43.18
45.96
34.10
35.41
24.91
w/ ours
59.29 46.30
49.64 16.46
53.48 47.22
38.03 43.93
41.67 46.26
28.60 43.09
iCaRL
57.12
47.49
53.43
41.49
43.73
34.33
w/ ours
61.17 +4.05
50.96 13.47
59.32 45.89
46.48 +4.99
51.19 +7.46
39.29 +4.96
LUCIR
63.17
53.71
60.50
49.08
51.36
39.37
w/ ours
65.48 +2.31
56.77 +3.06
64.93 43.89
55.54 +6,45
63.01 +11,5
53.09 +13.72
LUCIR+AANets
66.72
57.77
65,46
55.17
60.28
48,2
w/ ours
67.33 40.61
59.32 +1.55
65.51 40.05
55.42
40.25
64.10 43.82
53.41 +5.18
POD+AANets
66.12
55.27
61.12
48.83
53.81
42.93
w/ ours
67.47 +1.35
58.91
+3.64
64.56 +3.44
52.60
+3.77
60.35
46.54
48.53 45.60
Table 1: Evaluation results (%) on 5-Phase CIFAR-100 using POD+AANets (Liu et al. 2021a),
LUCIR+AANets (Liu et al. 2021a). LUCIR (Hou al., 2019). iCaRL (Rebuffi et al.) (017)) 7). and
LwF (Li & Hoiem| 2016) w/ and w/o our PlaceboCIL plugged in. "Average" and "Last" denote the
average accuracy over five phases and the last last-phase (5-th) accuracy, respectively.
5 EXPERIMENTS
We evaluate our method on three CIL benchmarks (CIFAR-100, ImageNet-Subset, and ImageNet- Ik) and achieve clear and consistent improvements. Below we describe the datasets and imple- mentation details, followed by results and analyses including the comparison to the state-of-the-an methods, an ablation study and the visualization of our placebo samples.
5.1 DATASETS AND IMPLEMENTATION DETAILS
Datasets and free image streams. We use three benchmarks based two datasets, CIFAR- 100 (Krizhevsky et al., 2009) and ImageNet-Ik Ik (Russakovsky et al. 2015). and follow the same data splits in related works (Douillard et 2020 Rebuffi al. 2017; Liu al. 2021a). There are two CIL settings on ImageNet-Ik: Ik: ImageNet-Subsel using a subset of 100 classes, and ImageNet- 1k using the full set of 000 classes. The 100-class data for the ImageNet- Subset are sampled from ImageNet- lk. For CIFAR-100, we use ImageNet- 1k as the free image stream. For ImageNet- Subset, we use a 900-class subset of ImageNet-Ik, 1k, which is the complement of ImageNet- in ImageNet- 1k. For ImageNet- we use a 1, 000-class subset of Iet (Deng et al. 2009) without any overlapping class (totally different super classes from those in ImageNet-IIk,
Configurations. Following Hou et al. 2019)); Douillard et al. (020). we a 32-layer ResNet Re-- buffi et al. 2017) for CIFAR- 100 dataset and an 18 18-layer ResNet He et al. 2016) for ImageNet datasets. We follow the benchmark protocol used in Douillard et al. (2020); ; Hou et al. (2019). The 0-th phase model is trained on the data of half of the classes. Then, it learns the remaining classes evenly in the subsequent phases. The number of exemplars for each class 20 in the default setting. The training batch size is 128. [ and K are set as 1, 000 and 200, respectively.
5.2 RESULTS AND ANALYSES
Results on five baselines. Table T shows the average and last-phase accuracy for five baselines (i.e.,
POD+AANets, LUCIR+AANets (Liu al. 2021a). LUCIR (Hou et al. 2019). iCaRL (Rebuffi ct al. 2017). and LwF (Li & Hoiem 2016). From the table, we make the following observations. I) Using our PlaceboCII. boosts the performance of the baselines clearly and consistently in all settings, indicating that our method is general and robust. 2) When the number of exemplars decreases, the improvement brought by our method becomes more significant. For example, the average last-phase accuracy improvement increases from 3.63 to 6.63 percentage points when the number of exemplars per class decreases from 20 to 5. reveals that the superiority of our method is more obvious when the forgetting problem is more serious due tighter memory budget (more challenging for CIL). 3) Our PlaceboCIL can boost the performance of all KD regularization, i.c., not only for prediction logits-based KD (Rebuffi et al. 2017) but also for feature-based KD (Hou al. 2019)

---

Under review as a conference paper at ICLR 2022
CITAR-I00
ImageNettSubbse
ImageNet-Ik
Method
N=5
10
25
10
25
5
10
25
TPCIL Tao ct al.
2020)
65.34
63.58
76.27
74.81
64.89
62.88
PODNet (Dou lard
al.
2020
64.83
63.19 60.72
75.54
74.33
68.31
66.95
64.13
59.17
DDE Hu el al.
2021
65.42
64.12
76.71
75.41
66.42
64.71
GeoDL (Simon al. 2021
65.14
63.12 65.03
76.63
75.40
71.43
65.23
64.79
60.97
DER Yan el al.
2021
68.65
67.48
66.18
78.20
POD-AANets (Liu et al.
202
66.12
64.11
62.12
76.63
75.40
71.43
67.60
64.79
60.97
w/ PlaceboCIL
(ours)
67.47
65.70
64.53
78.14
77.08
75.50
68.61
65.69
61.76
DER Yan et al.
2021 requires additional memory store the store all learned encoders (in all phases).
Table 2: Average accuracy (%) across all phases using the state-of-the-arr method (POD+AANets)
w/ and w/o our PlaceboClI. plagged in. The upper block is for recent CIL methods
LwF
iCaRL
LUCIR
AUCIR+AANets
No. Setting
Average
Last
Average
Last
Average
Last
Average
Last
Baseline
53.19
43.18
57.12
47.49
63.17
53.71
66.72
57.77
2
PlaceboCIL (budget)
59.10
48.28
61.09
50.81
65.14
56.95
67.31
59.26
3
PlaceboCIL (non-budget)
59.29
49.64
61.17
50.96
65.48
56.77
67.33
59.32
4
Overlapping
58.95
48.71
62.15
52.62
65.73
57.26
67.48
59,06
5
Non-overlapping
58.59
49.28
61.52
51.70
65.48
56.99
67.01
58.53
6
New data
54.06
43.94
57.70
47.51
64.21
53.98
66.69
57.33
7
Old data (upper bound)
61.41
51.16
66.64
58.03
67.02
58.98
68.82
61,52
8
w/o RL
57.06
46.12
60.27
50.57
63.83
55,33
66,91
58.88
9
w/ Transferred RL
58.36
48.98
61.58
52.51
65,04
56.48
67.18
57.84
10
CE loss on placebos
56.23
45.74
56.40
44.70
65.03
55.84
66.05
56.02
11
Higher confidence
56.98
46.20
60.43
49.36
63.60
55.50
66.97
58. 12
12
Random placebos
50.99
40.22
56.27
46.64
64.16
55.40
66.23
57.22
Table 3: Ablation results (%) on CIFAR-100 -N=5.""5vreaaaa "Average" and "Last" denote the average accu- racy over all phases and the last-phase accuracy, respectively. The four blocks show the results of baselines, using different free image streams, using RI. (or not), and using placebos to compute CE loss, respectively. Please refer to more details in Section 5.2 Ablation study.
Comparison to the state-of-the-art. Table shows the results of ttate-of-the-ar methods and our best model (taking PlaceboCII. as a plug-in module in the top method (Liu et al. (021a)). We can see that using our PlaceboCIL outperforms previous methods except for DER Yan et al. [02T], which requires lots of additional memory to store the encoders of all previous phases. Intriguingly, find that we can surpass others more when the number of phases is frnger-where there are more serious forgetting problems. For example, when N=25, we improve POD-AANets by 4.1% on the ImageNet-Subsee while this number is only 1.5% when N=5 (which is an easier setting with more saturated results). This reflects the encouraging efficiency of our method for reducing the forgetting of old class knowledge in CIL models.
Ablation study. Table shows ablation study. First block: baselines. Row I shows baselines. Rows 2 and 3 show the baselines with our PlaceboCIL as plug-in module. Row 2 "budget" means the combined size of episodic memory and the placebo buffer matching to the episodic memory size in other comparable methods. To this end, we randomly delete [24] samples from the new class data D., lse; obtain the buffer memory storing the unlabeled data batch 4. Row "non- budget" means we are allowed to use a little additional memory to save the unlabelled data batch 24 without deleting other samples. Comparing Row 3 to Row we can see that the performance of our PlaceboCIL only reduces 0.16 percentage points on average under the "budget" setting. The reason is that the amount of new data is much larger than that of old samples, and removing a small batch of new data does not hurt performance much (Liu et al. 2021b). If not specified, our reported results in other tables are from the setting of budget". Second block: different free data streams. Rows 4-7 show the ablation results for the following free data streams. (1) "Overlapping" means including samples from the overlapping classes between CIFAR- 10R and ImageNet. (2) "Non-overlapping" means using only the samples of non-overlapping classes between CIFAR- 100 and ImageNet (more realistic than "Overlapping"). means using only the purrent-phase new class data (i.e., without using any free data stream) as
(3) "New data"
8

---

Under review as a conference paper at ICLR 2022
Road
#03977966 police
809064766
telescope
2020980055
Table
now
00320794] - .diltwasher
00345274] - grand piazo
004515003 - spright
102250055
estertaiiment
placebos
(a) Selected placebos and GradCAM visualization
(b) 1-SNE visualization
Figure 3: (a) The selected placebos for two classes ("road" and "table") and their activation maps using GradCAM (Selvaraju et al., 2017) on CIFAR-100 (N=5). The free image stream is non- matching ImageNet-1k, 1k. (b) The t-SNE (Maaten & Hinton 2008) results on CIFAR-100 (N=5). For clear visualization. We randomly pick five new classes and five old classes. The purple, light blue, and dark blue points denote the new data, old data, and the selected placebos, respectively.
candidates to seleet placebos. (4) "Old data" means the original old class data are all accessible when computing KD loss (i.e.. upper bound of KD effect). Please note that in (1) and (2), two classes considered "overlapping" if their classes super-classes overlap. For example, "n02640022 sturgeon" in ImageNet- k is regarded as an overlapping class of the "fish" in CIFAR-100 because they overlap the level of super-class (i.e., "fish"). When comparing Row 5 with Row 3, we can find that our method is robust to the change of data streams: even if all overlapping classes are removed, our method can still achieve the same-level performance. Comparing Row 6 with Row 3, we can get the clear sense that using additional unlabeled data is definitely helpful. Comparing Row 7 with Row 3, we see that our method achieves comparable results to the upper bound.
Third block: RL. Row 8 is for using heuristic evaluation functions (3,=x=1) instead of RI. Row 9 uses the policy transferred from ImageNet-Sub, which means on the target CIL dataset, there is no RL training. Comparing Row 8 with Row 3 shows that using the RL algorithm successfully boosts the model performance. Comparing Row 9 with Row 3, we are happy to see that the RL policy learned is generalizable and can be transferred to the placebo selection Fourth block: CE loss. Row 10 assigns pseudo labels to the selected placebos and compute CE loss instead of KD loss in CIL. Comparing Row 10 with Row shows using placebo data compute CE loss significantly hurts the performance. The reason is that soft labels (used for KD) contain not only useful logits but noisy logits. When computing CE loss, soft labels are converted to hard (one-hot) labels that could be completely wrong labels-making the model overfitted to noisy labels.
other datasets.
Fifth block: Different Placebo Selection Strategies. Row 11 chooses the placebos with higher confidence (Lee et al. 2019). Row 12 chooses the placebos randomly like (Zhang et al., (020). Comparing Row (Row 12) with Row 3 shows using our RL-learned placebo selection strategy achieves better performance. The reason is that our evaluation functions change accordingly when the number of phases increases and the ratio between old and new classes changes.
Visualization results. Figure 3 (a) demonstrates the activation maps visualized by Grad-CAM for the placebos of two old classes on CIFAR-100 ("road" and "table"). ImageNet-Ik, 1k is the free data stream. We can see that the selected placebos contain the parts of "road" and "table" even though their original labels (on ImageNet- kk) are totally different classes. While this is not always the case, our method seems to find sufficiently related images to old classes that activate the related neurons for old classes ("road" and "table"). To illustrate that, figure (b) shows t-SNE results for placebos, old data (not visible during training), and new data. We can see that the placebo samples locate near
the old data and far away from the new data. This is why placebos can recall the old knowledge and do not harm the learning of new classes.
6 CONCLUSIONS
We proposed a novel method PlaceboCIL. for tackling the forgetting problem in CIL tasks. Place- boCIL selects high-quality placebo data from free data streams, and use them to improve the effect of KD between the models learned in adjacent phases, while not harming the learning of new classes in the new model. We designed an RL algorithm to make the selection of placebos more adaptive in different phases. Extensive experiments on multiple baselines show that our method is general and efficient. We believe it is promising to leverage unlabeled or free data in solving the data scarce problems in continual learning tasks, and will continue explore this direction.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Computational costs. RL-based methods require intensive usage of computing resources, which is not climate-friemee calls for future research into proposing more effective RL training strategies that can speed up the training.
Privacy issues. Keeping old class exemplars has the issue of data privacy. This calls for future research that explicitly forgets or mitigates the identifiable feature of the data.
Licenses. We use the open-source code for the following papers: AANets (Liu et al. 202 iCaRL (Rebuffi et al. 2017). Mnemonics (Liu et al. 2020a), and PODNet (Douillard et al. 2020 They are licensed under the MIT License.
Datasets. We two datasets in our paper: CIFAR-100 (Krizhevsky al. 2009) and ImageNet- 1k (Russakovsky et al., 2013). The data for both datasets are downloaded from their official websites and allowed to for non-commercial research and educational purposes.
REPRODUCIBILITY STATEMENT
We include the details of reproducing experiments in Section 5.1 pendix) and Section D (appendix). We also promise to release the code if this paper is accepted.
(main paper), Section C (ap-
REFERENCES
Davide Abati, Jakub Tomezak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learn- ing. In CVPR, pp. 3931-3940, 2020.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In ICI.R, 2019.
Guy Davidson and Michael C Mozer. Sequential mastery of multiple visual tasks: Networks natu- rally learn to learn and forget to forget. In CVPR, pp. 9282-9293, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In ECCV, 2020.
John Hammersley. Monte carlo methods. Springer Science & Business Media, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In CVPR, pp. 831- 831-839, 2019.
Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting for continual learning via model adaptation. In ICLR, 2019.
Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. Distilling causal effect of data class-incremental learning. In CVPR, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In ICCV, pp. 312-321, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. In ECCV, pp. 614-629, 2016.
10

---

Under review as a conference paper at ICLR 2022
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi- class incremental learning without forgetting. In CVPR, pp. 12245-12254, 2020a.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. CVPR, 2021a.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Rmm: Reinforced memory management for class- incremental learning. In NeurIPS, 2021b.
Yu Liu, Sarah Parisot, Gregory Slabaugh, Xu Jia, Ales Leonardis, and Tinne Tuytelaars. More classifiers, less forgetting: A generic multi-llassfier paradigm for incremental learning, In ECCV, pp. 699-716. Springer, 2020b.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov) 2008
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pp. 109-165. Elsevier, 1989.
K. McRae and Hetherington. Catastrophic interference is eliminated pre-trained networks. In CogSci, 1993.
R. Ratcliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, 97:285-308, 1990.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. CVPR, pp. 5533-5542, 2017.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In CVPR, pp. 7008-7024, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. In ICLR, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Ko- ray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv, 1606.04671, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In CVPR. pp. 618-626, 2017.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NIPS. pp. 2990-2999, 2017.
Christian Simon, Piotr Koniusz, and Mehrtash Harandi. On learning the geodesic path for incre- mental learning. In CVPR, pp. 1591-1600, 2021.
Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong, Xing Wei, and Yihong Gong. class-incremental learning. In ECCV, 2020.
Ronald J Williams. Simple statistical gralient-ollowingg algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):22--256, 1992.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, pp. 374-382, 2019.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In NeurlPS, pp. 899-908, 2018.

---

Under review as a conference paper at ICLR 2022
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. CVPR. pp. 3014-3023, 2021.
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In WACV, pp. 1131- 1140, 2020.
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. CVPR, pp. 12208-13217 2020.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
12

---

Under review as a conference paper at ICLR 2022
These appendices include the additional experiment results (AA). benchmark protocols ($ net- work architecture details ( training configurations ($D). equal-size size split results ( learning curve area results ($F), ablation results for different unlabeled data sources ($G), hardware informa- tion () and additional plots (
A ADDITIONAL EXPERIMENT RESULTS
This is supplementary to Section "Results on five baselines." In Table A1 we supplement the average and last-phase accuracy on five baselines (i.e., POD+AANets, LUCIR+AANets (Liu et al. 2021a), LUCIR (Hou et al.) 2019), iCaRL. (Rebuth et al.J 2017), and LwF (Li Hoiem] (016)) on CIFAR-100 (N=25). We can observe that using our PlaceboCII. boosts the performance of the baselines clearly and consistently all settings.
20 exemplars/class
10 exemplarsiclass
5 exemplars/class
Method
Average
Last
Average
Last
Average
Last
iCaRL
48.22
39.56
40.40
33.14
32.53
19.35
w/ ours
52.46 +1.24
42.01 +2,45
42.48 +2.08
33.20 +0.06
36.40 +3,87
22.70 +3.35
LUCIR
57.54
48.32
49.78
40.63
45.64
32.16
w/ ours
63.66 +6.12
53.77
+3.45
61.22 +11.44
50.84
+10.21
59.82 +14.18
48.83 +16.67
LUCIR+AANets
64.29
52.99
59.75
48.19
55.26
39.60
ours
65.96 +1.67
57.67
+4.68
63.60 +3.85
52.67 +4,48
63.01 +7.75
51.88 +12.28
POD+AANets
62.12
51.73
53.31
42.83
45.09
32.97
w/ ours
64.53 +2.41
54.95
+3.22
58.75
+5.44
46.83
+4.00
47.86
+2.77
33.93
+0.6
Table Al: Supplementary POD+AANets (Liu et al. 2021a). LUCIR+AANets (Liu et al. 2021a). LUCIR (Hou et al. 2019).
Table Evaluation results (%) on 25-Phase CIFAR- 100 using
and iCaRL. (Rebuffi et al., (017) w/ and w/o our PlaceboCII. plugged in. "Average" and "Last" denote the average accuracy over five phases and the last-phase (25-th) accuracy, respectively.
B BENCHMARKS PROTOCOL.
We follow the benchmark protocol used in Douillard et al. (2220) : Hou et al. (2019); Liu et al. 2021a 2020a). Given a dataset, the initial model (in the (-th phase) is trained on the data of half of the classes. Then, it learns the remaining classes evenly in the subsequent N phases. The learned model in each phase is evaluated on the test set containing seen classes. In the tables, we report average accuracy over all phases and the last-phase accuracy.
C NETWORK ARCHITECTURE DETAILS
This is supplementary to Section ing (Hou al. 2019 Liu et al.] 202 la
"Network architectures and configurations." Follow- Rebuffi et al. 2017 Wu et al. 2019). use 32-layer ResNet (Rebuffi et al. 2017) for CIFAR- 100 and an 18-layer ResNet (He et al. 2016) for ImageNet- shallower ResNet for ImageNet- 1k. The 32-layer ResNet
1k. Please note that standard to consists of 1 initial convolution layer and 3 residual blocks (in single branch). Each block has 10 convolution layers with 3 3 kernels. The number of filters starts from 16 and is doubled every next block. After these 3 blocks, there is an wveraee-pooling layer to compress the output feature maps feature embedding. The 18-layer ResNet follows the standard settings in (He et al. (016).. We deploy AANets using the same parameters as its original paper (Liu et al. 2021a).
D TRAINING CONFIGURATIONS
This is supplementary to Section 5.1 'Network architectures and configurations." The training of the classification model θ exactly follows the uniform setting in (Douillard et al. 2020 Hou al.
13

---

Under review as a conference paper at ICLR 2022
Liu et al. 20 2020a On CIFAR-100 (ImageNet-Subset//) we train it for 160 (90) epochs in each phase, and divide the learning rate by 10 after 80 (30) and then after 120 (60) epochs. If the bascline is POD+AANets (Liu et al. 202
2019
fine-tune the model for 20 epochs using only
exemplars. We apply different forms of distillation losses on different baselines: (1) if the baselines LwF (Li & Hoiem 20T6) and iCaRL. (Rebuffi et al. 2017),
the softmax KI. divergence loss; (2) if the baselines are LUCIR (Hou et al. 2019) and LUCIR+AANets (Liu et al., 202 we
use the cosine embedding loss (Hou al. ac.a]] 2019 and (3) if the baseline PODNet (Douillard al.
2020 and POD+AANets (Liu et al. (02 we use pooled outputs distillation loss (Douillard et al.
2020
E EQUAL-SIZE SPL.TT RESULTS
This is supplementary to Section 5. .2 A Ablation study." In Table A2 we supplement the results using the equal-size split on CIFAR 100, 10-phase (10 classes/phase, 20 exemplars/class). We pre- train the RL policy on IuaseNet-Subset. We can observe that using placebos selected by heuristic evaluation functions (3, = % = 1) consistently improves the results on three baselines. Using the RL. algorithm can further boost performance. The observations are similar using the original
setting in the main paper.
LwF
iCaRL
LUCIR
No. Setting
Average
Last
Average
Last
Average
Last
Baseline
53.85
41.00
59.70
45.29
56.71
42.78
PlaceboCIL (ours)
57.68
42.44
62.32
46.52
57.89
44.08
PlaceboCIL (ours, w/o RL)
56.31
41.76
61.05
46.02
57.01
43.13
Table A2: Supplementary to Table 3
Ablation results (%) using equal-size split on CIFAR-100,
10-phase (10 classes/phase).
F LEARNING CURVE AREA RESULTS
This is supplementary to Section
5.2 "Ablation study. Table A3 we show the Learning Curve Area (LCA1a) (Chaudhry et al. 2019) CIFAR-100, 5-phase. We can observe that our PlaceboCIL. achieves much better LCA10 compared to the baselines. shows that our PlaceboCIL. improves the ability to learn new knowledge.
LUCTR +AANets
No.
Setting
LwF
iCaRI.
LUCIR
1
Baseline
40.02 50.82
47.79
52.06
56.44
2
PlaceboCIL (ours)
55.24
57.48
59.69
Table A3: Supplementary to Table
Learning Curve Area (L.CA1o)
CCaudhhry et al. 2019 on
CIFAR-100, 5-phase.
G ABLATION RESULTS FOR DIFFERENT UNLABELED DATA SOURCES
This is supplementary to Section "Ablation study." We believe that key to success design of our method instead of the choice of unlabeled data sources. To verify this, provide the results for a new ablative setting: "using random unlabeled data" (to compute the distillation loss) in Table A4 We can observe that matter what unlabeled data sources use, method consistently performs better than using random unlabeled data.
H HARDWARE INFORMATION
We experiments using GPU workstations follows,
14

---

Under review as a conference paper at ICLR 2022
LwF
iCaRL
LUCIR
LUCIR+AANC
No. Setting
Average
Last
Average
Last
Average
Last
Average
Last
Baselinc
53.19
43.18
57.12
47.49
63.17
53.71
66.72
57.77
2 PlaceboCIL (ours, all)
59.29
49.64
61.17
50.96
65.48
56.77
67.33
59.32
3
PlacchoCII (ours, overlapping)
58.95
48.71
62.15
52.62
65.73
57.26
67.48
59.06
4
Random unlabeled data (all)
50.99
40.22
56.27
46.64
64.16
55.45
66.23
$7.22
4
Random unlabeled data (overlapping)
50.80
40.94
55.70
46.47
64.23
54.68
66.58
57.08
Table A4: Supplementary to Table Ablation results (%) on CIFAR-100, N=5. "Averaae and
"Last" denote the average accuracy over all phases and the last-phase accuracy, respectively. "All" denotes using all data from ImageNet, and "overlapping" means including samples from the over- lapping classes between CIFAR-100 and ImageNet.
CPU: Ix AMD EPYC 7502P 32-Core Processor
GPU: 4x NVIDIA Quadro RTX 8000, 48 GB GDDR6
Memory: 1024 GiB = 8x 128GiB, DDR4, 3200 MHz, ECC
I ADDITIONAL PLOTS
This is supplementary to Section 5 Compared to the state-of-the-art. In Figures Al
present the phase-wise accuracy obtained on CIFAR-100 and ImageNet-Subset. "Upper Bound" shows results of joint training with all previous data accessible in every phase. We can observe that our method achieves the highest accuracy in almost every phase of different settings.
UpperBound
PlaceboCIL (ours)
PODNet
Mnemonics
LUCIR
iCaRL
70
accuracy
60
50
40
8
100
8
#phase (N=5)
#phase (N=10)
#phase (N=25)
(a) CIFAR- 100 (100 classes).
80
securacy (56)
70
2
3
100
2
4
8
10
12
#phase (N=5)
#phase (N=10)
#phase (N=25)
(b) ImageNet-Subset (100 classes).
Figure Al: Supplementary to Table 2. Phase-wise accuracies (%) CIFAR-100 and ImageNet- Subset.
15


Reviews:
[{"review_id": "rvEgEZK0C36", "reviewer": "Reviewer_vLx7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n- The idea of introducing unlabeled placebo data for alleviating catastrophic forgetting seems interesting and novel for class incremental learning. Such extra training data does not require additional data memory.  \n- This paper is well-written and easy to follow.\n- The proposed method achieves strong performances across multiple benchmarks, in particular for high-resolution image classification tasks, with higher memory efficiency for old data.\n\nConcerns\n- The related work is a bit lacking and many recent related works are missing:\n\n  [A] DER: Dynamically Expandable Representation for Class Incremental Learning (SOTA Method, better than PODNet+AANet)\n\n  [B] On Learning the Geodesic Path for Incremental Learning (better than or comparable to PODNet+AANet on CIFAR100)\n\n- Some of the previous work has explored other types of unlabeled data in the class incremental learning, which should be discussed:\n\n  [C] Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild (ICCV2019)\n\n  [D] More Classifiers, Less Forgetting: A Generic Multi-classifier Paradigm for Incremental Learning (ECCV2020)\n\n- The effect of RL seems limited. In Table 3, for the method LUCIR+AANets, the performance of RL is only marginally better than w/o RL (<0.5%).  It is unclear if the RL strategy is necessary. \n\n- Some of the experiment details are unclear/missing: \n1) The experimental setting of Figure 1 seems unclear: Does the experiment of 'KD on new' keep old data? What are the values of temperature and distillation coefficients and how does the method determine them?\n2) The values of some hyperparameters are missing:  What is the batch size for training? How many samples are taken from the free image stream (|\\mathcal{U}|) as candidates? In each iteration, how many unlabeled data samples (K) are selected from \\mathcal{U} for training?\n3) RL comparison: In Table.3, What sampling policy is used in No.8 w/o RL? Is it random sampling? ", "summary_of_the_paper": "This paper proposes to apply distillation loss on unlabeled data to tackle the catastrophic forgetting problem for class incremental learning. In each training iteration, this work first retrieves a candidate pool of unlabeled data from a free image stream and then adopts a learned RL strategy to select high-quality unlabeled data to augment the sampled mini-batch. It conducts extensive experiments and achieves consistent improvements on CIFAR-100, ImageNet-100, and ImageNet-1k datasets.", "summary_of_the_review": "The proposed CIL strategy, which exploits distillation on unlabeled placebo data, seems novel and effective. However, several SOTA methods are missing in the related work, and the experimental settings and the efficacy of RL need to be clarified. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jFNTMI3jwm9", "reviewer": "Reviewer_Dwjy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- S1: Novel idea of using unlabeled data for knowledge distillation for class incremental learning (CIL).\n- S2: Compelling empirical performance with large datasets.\n- S3: Clear presentations (i.e., figures, texts and equations)\n\n**Weaknesses**\n- W1: Out of protocol of CIL -- using large unlabeled data requires large supplementary data source and require additional computational cost, which may limits the applicability of the CIL method. The rationale behind this claim is that the CIL is for the realistic recognition scenario that the data is given in a stream and the model update should be on the fly, adaptively to the new data. If the computation of incremental update step is large, the benefit of the CIL could disappear.\n- W2: Large unlabeled source may or may not contain sufficient information (i.e., even the future information) depending on the choice of source of unlabeled data -- Google searched image could be the superset of the ImageNet-1k as ImageNet dataset is collected from Google search. Thus, the choice of unlabeled data might be the key to success of the proposed method, not the design of the method. Lack of study about the source is a weak point of significance of the proposed method\n- W3: The evaluation protocol of using small memory budget is largely beneficial to the proposed method. As the proposed method is using large unlabeled data as an extra source of information, the small memory budget is easily compensated by the proposed method while other methods inevitably performs poorly due to lack of information source.\n- W4: Similar to W3, the combined size of episodic memory (i.e., containing samples from the tasks) and the placebo buffer P (i.e., containing samples from the free image stream) should be matched to the episodic memory size in other comparable methods. The memory budget of the compared methods is not clearly stated. In addition, the \"budget\" and \"non-budget\" in the ablation study are also not clearly described (what do you mean by \"keep the strict memory budget in each phase\"? \n- W5: The proposed method seems promoting learning new knowledge. As there is a measure to evaluate the ability to learn new knowledge (e.g., intransigence (Chaudhry et al., 2018)), it is suggested to empirically validate the benefit of the proposed approach by the measure.\n- W6: As the method employs reinforcement learning (RL), the computational cost would be considerable. But there is no study about the computational complexity.", "summary_of_the_paper": "The paper uses data chosen from a free image stream (without labels) to supply knowledge (by the \"placebo\" samples) for class incremental learning. The empirical results shows that the proposed method exhibits good accuracy in large and high resolution image benchmark datasets such as ImageNet-1k with small memory budget. The paper also presents comprehensive ablation study for each proposed component.", "summary_of_the_review": "Given the strength and the weaknesses, the paper has decent novelty of using extra free data stream for improving continual learning methods but may be less fairly evaluated (i.e., the evaluation protocol is designed favorably towards the proposed method) with other compared methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YUG3j_jUDg7", "reviewer": "Reviewer_9D1K", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive points:\na) the paper confirms earlier observations that external data can significantly improve results on a wide range of KD-based CL methods.\n\nb) paper is well written, and motivation (fig 1 is nice)\n\nc) ablation shows that RL improves results (though as said above I would have liked more insights and stronger baselines).\n\nThe paper has the following weaknesses:\n\na) Related work: the usage of external data has been studied before in continual learning. I am aware of two relevant works:\n\n[1] Zhang, J., Zhang, J., Ghosh, S., Li, D., Tasci, S., Heck, L., Zhang, H. and Kuo, C.C.J., 2020. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1131-1140).\n[2] Lee, K., Lee, K., Shin, J. and Lee, H., 2019. Overcoming catastrophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 312-321).\n\nThese two methods (and especially [2]) already show the main contribution of the proposed paper, namely that external data can help significantly improve CL. [2] also provides a method to sample from the stream (called 'Confidence calibration for sampling.'). Comparison with these methods is of great importance to evaluate the proposed method. Also, the novelty claim should be adjusted according to these works. \n\nb) I think the RL algorithm was interesting. However, I would have liked to see a more in-depth analysis of its results, comparison also to simpler baselines (choosing only samples with higher confidence for example). \n\nc) It would be nice to also see results for the equal-size split on CIFAR 100 (i.e. 10 tasks of 10 classes). I do not see why the proposed method would perform worse in this setting (because of the RL training ?). Anyways even if underperforming for this setting, it would be insightful to include it. ", "summary_of_the_paper": "The paper proposes to use a separate available data stream (called placebos) to distill information during the continual learning. They show that this is almost as efficient as using ‘old data’, and much more efficient than using ‘new data’ (the normal distillation setting). To balance the different losses they propose a reinforcement learning algorithm that judges the quality of placebos.  They obtain good results on several standard datasets. ", "summary_of_the_review": "Regretfully the authors missed very important related work already showing the potential of external data. This drastically reduces the novelty of the paper. This work also proposes a method to select data from the data stream which could be compared to the proposed RL algorithm. Given that comparison with the most relevant related work is missing, I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XtGbxGswENv", "reviewer": "Reviewer_4jmy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The RL-based sampling algorithm looks interesting. The paper is well-written, but some parts could be improved (see suggestions below).\n\n- The first contribution, leveraging \"unlabeled placebo data from a free image stream to improve the KD effect in CIL models\" has already been studied in a prior work, [Lee et al.]. The second contribution, the RL-based sampling algorithm is not in this prior work, so maybe the authors can take [Lee et al.] as another baseline, though it seems the main performance gain is from the usage of the free image stream rather than the sampling algorithm (by looking at the ablation study in Table 3 & performance gain shown in [Lee et al.]).\n\n- Knowledge distillation can be done in many different forms, so the formulation should be clarified. I assume that the KL divergence loss is used, as 1) they referred to a series of works using it like LwF and iCaRL, and 2) Fig 1 matches the input of the CE loss and KD loss.\n\n- Fig 1(b) is technically incorrect. The KL divergence loss is defined only when two inputs are probability vectors. You should normalize the inputs. If the loss is really computed in this way, the method is technically incorrect.\n\n- Fig 2 (and even the paragraph referencing Fig 2) is not self-explainable, e.g., a description about the way to construct placebos is necessary.\n\n- Could you elaborate more on models used in No. 8 and 9 in Table 3?\n\n[Lee et al.] Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild. In ICCV, 2019.\n\nJust to note, my initial recommendation is under an assumption that the authors address all my concerns well.\n\n## Post rebuttal\n\nAs other reviewers agreed, this paper missed some important prior works like [Lee et al.] and [Zhang et al.] at the time of submission. The first contribution \"to compute the KD loss using placebo data chosen from a free image stream\" is duplicated with them, so the abstract and contribution paragraph in the intro had to be significantly rewritten but they didn't.\n\nHowever, apart from it, additional comparisons made during the rebuttal period is impressive. Also, I think the second contribution with an RL algorithm is interesting, and the ablation study on it (row 8 and 9 in Table 3) justify its effectiveness properly.\n", "summary_of_the_paper": "This paper proposes to sample placebo data from a free image stream and use them for knowledge distillation during class-incremental learning. For effective sampling, an RL algorithm is also proposed. Experimental results show that the proposed method improves state-of-the-art methods.", "summary_of_the_review": "The idea of sampling data from a free image stream for class-incremental learning is not new, but the RL-based sampling algorithm looks new. The paper is well-written, but some parts could be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel method for class-incremental learning (CIL) with the help of placebo data chosen from a free image stream. Such placebo data are unlabeled and easy to obtain in practice. To adaptively generate phase-specific functions as the accurate estimation of placebos' quality for KD, this paper applies reinforcement learning based on the constraints of the CIL. The effectiveness of the method has been verified on multiple datasets, including ImageNet-1k and ImageNet-Subset with both lower memory and higher accuracy than baselines. The major concern is about the novelty that the unlabeled auxiliary data is not quite new for CIL despite the minor difference in settings and methods. Moreover, the improvements over the baselines are not significant enough, which is a minor concern.
Decision: Reject