Online Placebos for Class-incremental Learning
Authors: Yaoyao Liu
Yingying Li
Bernt Schiele
Qianru Sun
Venue: Submitted to ICLR 2023 (2023)
Keywords: incremental learning
continual learning
class-incremental learning

Abstract: Not forgetting old class knowledge is a key challenge for class-incremental learning (CIL) when the model continuously adapts to new coming classes. A common technique to address this is knowledge distillation (KD) which penalizes prediction inconsistencies between old and new models. Such prediction is made with almost new class data, as old class data is extremely scarce due to the strict memory limitation in CIL. In this paper, we take a deep dive into KD losses and find that “using new class data for KD” not only hinders the model adaption (for learning new classes) but also results in low efficiency for preserving old class knowledge. We address this by “using the placebos of old classes for KD”, where the placebos are chosen from a free image stream, such as Google Images, in an automatical and economical fashion. To this end, we train an online placebo selection policy to quickly evaluate the quality of streaming images (good or bad placebos) and use only good ones for one-time feed-forward computation of KD. We formulate the policy training process as an online Markov Decision Process (MDP), and introduce an online learning algorithm to solve this MDP problem without causing much computation costs. In experiments, we show that our method 1) is surprisingly effective even when there is no class overlap between placebos and original old class data, 2) does not require any additional supervision or memory budget, and 3) significantly outperforms a number of top-performing CIL methods, in particular when using lower memory budgets for old class exemplars, e.g., five exemplars per class. The code is available in the supplementary.  

Under review as a conference paper at ICLR 2023
ONLINE PLACEBOS FOR CLASS-INCREMENTAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Not forgetting old class knowledge is key challenge for class-incremental learn- ing (CIL) when the model continuously adapts to new coming classes. A common technique to address this is knowledge distillation (KD) which penalizes predic- tion inconsistencies between old and new models. Such prediction is made with almost new class data, as old class data is extremely scarce due to the strict mem- ory limitation in CIL. In this paper, we take a deep dive into KD losses and find that "using new class data for KD" not only hinders the model adaption (for learning new classes) but also results in low efficiency for preserving old class knowledge. We address this by "using the placebos of old classes for KD", where the placebos are chosen from a free image stream, such as Google Images, in an automatical and economical fashion. To this end, train an online placebo selection pol- icy to quickly evaluate the quality of streaming images (good or bad placebos) and use only good ones for one-time feed-forward computation of KD. We for- mulate the policy training process as an online Markov Decision Process (MDP), and introduce an online learning algorithm to solve this MDP problem without causing much computation costs. In experiments, we show that our method 1) is surprisingly effective even when there is no class overlap between placebos and original old class data, 2) does not require any additional supervision or memory budget, and 3) significantly outperforms a number of top-performing CIL meth- ods, in particular when using lower memory budgets for old class exemplars, e.g., five exemplars per class. The code is available in the supplementary.
INTRODUCTION
AI learning systems expected to learn new concepts while maintaining the ability recognize old ones. In many practical scenarios, they cannot access the past data due to the limitations such as storage or data privacy but are expected be able recognize all seen classes. A pioncer work (Rebuffi et al., 2017) formulated this problem in the class-increment incremental learning (CIL) pipeline: training samples of different classes are loaded into the memory phase-by-pee by-phase, and the model keeps re-training with new class data (while discarding old class data) and is evaluated on the testing data both new and old classes. The key challenge is that re-training the model on the new class data tends to override the knowledge acquired from the old classes (McCloskey & Cohen, 1989; McRae & Hetherington, 1993; Ratcliff, 1990), and the problem is called "catastrophic forgetting". To alleviate this problem, most of CIL methods (Rebuffi et al., 2017; Hou et al., 2019; Douillard et al., 2020; Liu et al., 2020a; 2021a; Zhu et al., 2021) are equipped with knowledge distillation (KD) losses that penalize any feature and/or prediction inconsistencies between the models in adjacent phases.
The ideal KD losses should be computed on old class data since the teacher model (i.e., the model in the last phase) was trained on them. This is, however, impossible in the CIL. setting, where almost all old class data are inaccessible in the new phase. Existing methods have to use new class data as a substitute to compute KD losses. We argue that this 1) hampers the learning of new classes as it distracts the model from fitting the ground truth labels of new classes, and 2) can not achieve the ideal result of KD, as the model can not generate the same soft labels (or features) on new class data as on old class data. We justify this from an empirical perspective as shown in Figure I (a):
We made the corresponding changes in the revised paper and colorized these changes blue.

---

Under review as a conference paper at ICLR 2023
Newclass data
new model outputs
old model outputs
labels
KD on
Placebos (ours)
class data
Road
C =-CE [0.1. 0.2]
60
+ACKD( ( [0.5, 0
[0.2,
Aocuracy (%)
55
(i) The CIL loss for new class training sample
50
45
C =CCE A.1.0.1.1....11
Table
40
+ +ACKD( ( [0.4,
[0.3,
plaoo
(ii) The CIL loss for an old class training sample
New Class Old Class
(a) Average accuracy
(b) Conceptual illustrations of the CIL losses
(c) Selected placebos
Figure 1: (a) Average accuracy when computing KD loss on different data using iCaRL (Rebuffi et al., 2017) CIFAR-100. The KD loss (softmax KL divergence loss) is computed new class data (dark blue), placebos (of old class data) selected by our method (light blue), and old class data green), i.e., the ideal case. (b) Conceptual illustrations of the loss problem if using new class data for KD. The dark blue and orange numbers denote the predictions of old and new classes, respectively. It is clear in (i) that the objectives are different when using new class sample for KD (the oracle case is to have both "ascent"), ), e.g., the ground truth label for the second old class is O, while the "KD label" this position is 0.8. This is not issue when using the old class sample, e.g., in (ii), its ground truth label and "KD label" have consistent magnitudes at the same position (1 and 0.7, respectively). (c) Our selected placebos for two old classes ("road"" and "table") and their activation maps using GradCAM (Selvaraju et al., 2017) on CIFAR-100. The free image stream is ImageNet- 1k that does not have class overlap with CIFAR-100. They are selected because their partial image regions contain similar visual cues with old classes.
the upper bound of KD is achieved when using "Old class data", and if compared to it, using "New class data" sees a clear performance drop for recognizing both old and new classes. In Figure (b), we show the reason by diving into loss computation details: when using new class samples (as substitutes) to compute CE and KD losses simultaneously, these two losses are actually weakening each other, which does not happen in the ideal case of using old class samples.
Using unlabeled external data (called placebos in this paper) has been shown to be practical and effective in solving the above issue. Compared to existing works (Lee et al., 2019; Zhang et al., 2020), this paper aims to solve two open questions. Q1: How to adapt the placebo selection process in the non-stationary CIL pipeline. The ideal selection method needs to handle the dynamics of increasing classes in CIL. e.g., in later incremental phase, it is expected to handle more complex evaluation on the placebos of more old classes. Q2: How control the computational and memory- related costs during the selection and utilization of placebos. It is not intuitive how to process external data without encroaching the memory allocated for new class data or breaking the strict assumption of memory budget in CIL.
We solve these questions by proposing a new method called PlacoboCIL that adjusts the policy of selecting placebos for each new incremental phase, in an online and automatic fashion and without needing extra memory. Specifically, tackle Q1, we formulate the PlaceboCIL as an online Markov Decision Process (MDP) and introduce novel online learning algorithm to learn dynamic policy. In each new phase, this policy produces phase-specific function evaluate the quality of incoming placebos. The policy itself gets updated before the next phase. For Q2, we propose a mini-batch- based memory reusing strategy for PlaceboCII. Given a free data stream, we sample a batch of unlabeled data, evaluate their quality by using our phase-specific evaluation function (generated by the learned policy), and keep only the high-quality placebos to compute KD losses. After this, we remove this batch totally from the memory before loading a new batch. In our implementation, this batch can be very small, e.g., 200 images. We randomly remove the same size of new class data for intaking this batch to keep the strict assumption of memory budget.
We evaluate PlaceboCIL by incorporating it into multiple strong baselines such as PODNet (Douil- lard et al., 2020), LUCIR (Hou et al., 2019), AANets (Liu et al., 2021a), and FOSTER (Wang
2

---

Under review as a conference paper at ICLR 2023
et al., 2022), and conducting series of ablative studies. Our results three popular CIL bench- marks show the clear and consistent superiority of PlaceboCIL, especially when using a low memory budget for old class exemplars. For example, centage points on average when keeping only 5 exemplars per old class in the memory. In addition, it is worth mentioning that PlaceboCII. is surprisingly efficient even when there is no class overlap between placebos and original old class data. The reason is that PlaceboCIL can make use of the local visual cues in placebos, e.g., similar visual cues of "table" are found on the local regions of an "piano" (and "dishwasher") image as shown in Figure (c).
method boosts the last-phase accuracy by 6.9 per-
Our contributions are fhree-fold. 1) A generic PlacoboCIL method that selects placebo images from free image stream to solve the KD issue in existing methods. 2) A novel online learning algorithm for training a placeho selection policy and a mini-batch--asee memory reusing strategy to avoid extra memory usage. 3) Extensive comparisons and visualizations on three CII. benchmarks, taking top- pe-pefforming CIL models as baselines and with the same strict assumption on memory.
RELATED WORK
Class-incremental learning (CIL) methods can be divided into three categories. Distillation-based methods introduce different knowledge distillation (KD) losses to consolidate previous knowledge. The key idea is to enforce prediction logits (Li Hoiem, 2016; Rebuffi et al., 2017), feature maps (Douillard et al., 2020), or other essential information (Tao et al., 2020; Wang et al., 2022; Simon et al., 2021; Joseph et al., 2022) to be close to those of the pre-phase model. Memory-based methods use small number of preserved old class data (called exemplars) (Rebuffi et al., 2017; Shin et al., 2017; Liu et al., 2020a; Prabhu et al., 2020) or augmented data (Zhu et al., 2021) to recall the old class knowledge. Retwork-archiiecttre--bsss 2018; Abati et al., 2020; Yan et al., 2021) design incremental network architectures by expanding the network capacity for new class data freezing partial network parameters keep the old class knowledge. Our method can be used to improve different iitiillation-basele CIL methods
methods (Rusu et al., 2016; Xu & Zhu,
Some prior works used unlabeled external data for class- incremental learning. Lee et al. (2019) pro- posed fefidence-based sampling method to select unlabeled external data to compute specially- designed global distillation loss. Zhang et al. (2020) randomly selected unlabeled samples and used them to compute KD) loss for model consolidation. Liu et al. (2020b) used unlabeled data to max- imize the classifier discrepancy when integrating an ensemble of auxiliary classifiers. Our method differs from them in two aspects. 1) Our method use the unlabeled data in a more generic way and can be applied to improve different iistillation-base methods (Hou et al., 2019; Rebuffi et al., 2017; Wang et al., 2022), while the existing methods use unlabeled data to assist their ppecially-dssined loss terms or components. 2) We train an online policy to select better unlabeled data to adapt to the non-stationary CIL. pipeline while existing methods select unlabeled data by applying fixed (i.e., non-adaptive) rules in all incremental phases.
Online learning observes a stream of samples and makes a prediction for each element in the stream. There are mainly two settings in online learning: full feedback and bandit feedback. Full feedback means that the full reward function is given at each stage. It can be solved by Best-Expert algo- rithms (Even- -ar et al., 2005). Bandit feedback means that only the reward of the implemented decision is revealed. If the rewards are independently drawn from a fixed and unknown distribution, may use e.g., Thompson sampling (Agrawal & Goyal, 2012) and UCB (Auer & Ortner, 2010) to solve it. If the rewards are generated in a non-stochastic version, we can solve it by c.g., Exp3 (Auer al., 2002). Online MDP is an extension of online learning. Many studies (Even-Dar et al., 2009; Li et al., 2019) aim to solve it by converting it to online learning. In our case, we formulate the CIL as an online MDP and convert into a classic online learning problem. The rewards in our MDP are non-stochastic because the training and validation data change in each phase. Therefore, we design our algorithm based on Exp3 (Auer et al., 2002).
ONLINE PLACEBOS FOR CLASS-NCCRMMENTAL LEARNING (PLACEBOCIL)
CIL has multiple soauiinng-testingg phases during which the number of classes gradually increases to the maximum. In the 0-th phase, data D D1...= = D J. including the training samples of co classes, are used to learn the model On. After this phase, only a small subset of D Di:co (i.e., exemplars
3

---

Under review as a conference paper at ICLR 2023
denoted as f1:c0=(E1, Eeg }) can be stored in the memory and used replay samples in later phases. In the i-th phase, we use C₁ to denote the number of classes we have observed so far. We get new class data loe (D.
De, of (a Q-11) classes and load exemplars &1 from the memory. Then, we initialize θ with - and train it using T1sc;= UD_,UDDe++t The model θ will be evaluated with a testing set Q1:,,= ( 2c, for all classes seen far. Please note that in any phase of PlaceboCIL, we assume we can access a free image stream, where we can load unlabeled images and select placebos.
PlaceboCIL. formulates the CIL task as an online MDP. In each phase, we update a policy, for which we sample glass-balancee subset from training data as the testing set, and use the updated policy to produce phase-specific evaluation function. During model training, we sample unlabeled images, use the evaluation function to quickly judge the image quality (good or bad placebos), and select the good ones compute KD loss. In this section, we introduce the formulation of online MDP in Section 3.1, show how to apply the policy select placebos and compute KD loss in Section 3.2, and provide online learning algorithm to update the policy in Section 3.3. The pseudocode is given Appendix (Algorithms and 2).
3. ONLINE MDP FORMULATION FOR CIL
The placebo selection process in CII should be online inherently: training data (and classes) get updated in each phase, so the placebo selection policy should update accordingly. Thus, it is intuitive to formulate the CIL as an online MDP. the following, we provide detailed formulations.
Stages. Each phase the CIL task can be viewed as a stage in the online MDP.
States. The state should define the current situation of the intelligent agent. In CIL, we use the model θ as the state of the i-th phase (i.e., stage). We use to denote the state space.
We define the action as a, yx), consisting of the hyperparameters (3, and 7)) used to create evaluation function. As 3 and vary in a continuous range, we discretize them to define a finite action space. We will claborate on how to take an action and deploy the hyperparameters in Section 3.2.
Actions.
Policy x= (p(E is a probability distribution over the action space A, given the current state O We will elaborate how to update the policy using our proposed online learning algorithm in Section 3.3.
Environments. We take the training an testing data in each phase as the environment. In the i-th phase, the environment is (T= Q1se;) where Ti, the training data and Q1.e, is the testing data. The environment is time-varying because we observe different training data (and classes) in each new phase.
Rewards. CIL aims train model that is efficient in recognizing all classes is intuitive to use the testing accuracy as the reward in each phase. We cannot observe any reward (i.e., testing accuracy) directly because the testing data is not accessible during training. We solve this by building a local testing set using subset of training data (see details in Section 3.3). Our objective is to maximize cumulative reward, i.e., R Σm= I'H denotes the i-th phase reward, The reward function rxH changes with H4, so time-varying.
far. Therefore,
(H,((,; (0,, a:), where r'H (9,, a:)
3.2 PLACEBO SELECTION WITH MINI-BATCH BASED MEMORY REUSING STRATEGY
In the following, we introduce how to build phase-specific evaluation functions using the policy, select high- high-quality placebos without breaking memory constraints, and compute KD loss with the selected placebos. The computation flow (in each phase) is illustrated in Figure
Computing prototypes. Our placebo selection based on the distance from placebo to the class prototype, i.c., the mean feature of each class (Snell et al., 2017). First, we compute the prototypes of all seen classes. We use exemplars to compute the prototypes of old classes, and use new class
'Though discretization suffers the curse of dimensionality, experiments show that with a coarse grid, already have significant improvements over fre-fixed hyperparameters.
4

---

Under review as a conference paper at ICLR 2023
New data
New class batch
Data for C.. loss
Target data
Sample
New model
stream
Old class batch
CE loss
Old exemplars
Sample
Memory
Prototypes
Placebo batch P
Old model
Unlabeled data
Placebos P
KD loss
Free
Sample
image
stream
Data for KD loss
Evaluation functions (1) (1)) 1
Figure 2: The computation flow of our PlaceboCIL in the i-th phase. At the beginning of this phase, build sirasse-peecfic evaluation functions {S.m. (u)) [m)] During training, we select placebos as follows. 1) We load an unlabeled data batch LI from the free image stream. 2) We compute scores using {Sm (4) m for all samples U. 3) For each old class m, we select K placebos with the
highest scores and add them to P. 4) We delete used placebos from P after computing the loss. 5) When we use up the selected placebos in P, we repeat the selection steps.
training data for class prototypes, follows,
Σ Fo, Pro(Di) = [Di Pro(E,) = |Eml ZEE.
Σ Fe)
(I)
=ED:
where Fe,() (-) denotes the encoder (i.e., the feature extractor) of θ, Pro(En) and Pro(Di) denote the prototypes of the n-th old class and the I-th new class, respectively.
Building evaluation functions. We argue that high- high-quality placebos for the m-th old class should meet two requirements: (1) being close to the prototype of the m-th class in the feature space because they will be used to activate the related neurons the m-th old class the model; and (2) being far from the prototypes of all the other classes in the feature space so that they will not cause the KD issue (as shown in Figure 1). To achieve these, we design the following evaluation function (x) for the m-th old class in the i-th phase:
(x) = Sim (ro(E))) + -B Sim (x), Pro(En))
C-1 -1
==1 my
(2)
Sim
C - C:-1
1=0-1+1
where x denotes an unlabeled input image, and (im(. denotes cosine similarity. 3 and y are two hyperparameters from the action ( 74), sampled by the policy æ.
Allocating mini-batch-band based memory for placebos. We need to allocate small amount of mem- ory to store unlabeled images (before evaluating them). At the beginning of the i-th phase, we allocate memory buffers U and P respectively for the unlabeled image candidates and the selected placebos. order to not exceed the memory budget, we randomly remove the same number, i.e., 1 + Pl. of samples from the training data of new classes. Our empirical results show this "remove" does not degrade the model performance on new classes. Please kindly refer to Appendix G for detailed results and comparisons.
Selecting placebos. Whenever the placebo buffer P is empty, we load a batch of unlabeled samples from the free image stream, and choose K placebos for each old class to add into P, as follows,
K
p = (xx
1II = arg max
Σ m=
Σ
(xx).
(3)
24 Ell
5

---

Under review as a conference paper at ICLR 2023
Calculating loss with placebos. After selecting placebos, sample batch of new class data d C Dey-- +- batch of old class exemplars C &lseg and a batch of placebos p C P. We calculate the overall loss as follows,
C = CCB due) + 1
(4)
where LCE and LKD denote the CE loss and KD loss, respectively. À a hyperparameter to balance the two losses (Rebuffi et al., 2017). To control the memory usage, we delete p from P immediately after calculating the loss. When P is empty, we repeat the placebo selection operation.
3.3 ONLINE POLICY LEARNING ALGORITHIM
A common approach to solving an online MDP is approximate it as an online learning problem and solve it using online learning algorithms (Even-Dai et al., 2005; Agrawal & Goyal, 2012; Auer al., 2002). We also follow this idea in PlaceboCIL, and our approximation follows the work by Even-Dar et al. (2009) that is theoretically proved to have the optimal regret. Specifically, Even- Dar et al. (2009) relax the Markovian assumption of the MDP by decoupling the cumulative reward function and letting it be time-dependent so that they can solve online MDP by standard online learning algorithms.
However, we cannot directly apply the algorithms proposed in (Even-Dar et al., 2009) to our prob- lem. is because they assume full feedback meaning that the model can observe the rewards of all actions in every learning phase (which is also why its online learning problem can be solved by Best Expert algorithms (Even-Dar et al., 2005)). While in CIL, we cannot observe any reward (i.c., the testing accuracies) because the testing data Q iee are not accessible in any phase i. To address this problem, we split the training data we have in each phase into two subsets: one for training and an- other for validation. Once have validation set, we can solve online learning problem based on Exp3 (Auer et al., 2002)-a simple and effective bandit algorithm. In the following, we elaborate how we do this data splitting in each local dataset (i.e., the entire data we have in each training phase of CIL), compute the decoupled cumulative reward, and learn the policy π with Exp3.
subset Bise, from the training data TE: Brs,, contains the same number of samples for both the old and new classes. In this way, we rebuild the local training and validate sets, and update the environment from the oracle H. HT Q bea)) (which is unavailable in CIL) the local environment h BB Blse.).
Rebuilding local datasets. To compute reward, we sample a
Decoupled cumulative reward. We create the decoupled cumulative reward function R based on the original cumulative reward function Σ ru, ag). In the i-th phase, we compute R as follows.
i+n
R(a, h1) = Σ The (0,, + costant, i=1
(5)
where the "constant" denotes the historical rewards from the 1-st phase to the (i-1)-th phase. It doesn't influence policy optimization. R(a, h;) is the long-term reward of a time-invariant local MDP based on the local environment hú We use R(a,, hu) as estimation of the final cumulative reward, following (Even-Dar et al., 2009). Because we don't know the total number of phases N during training, we assume there will be n phases in the future. Furthermore, we fix the action a, to simplify the training process. R(a, h2) is a function of a; and h,
Training policy with Exp3. Exp3 (Auer et al., 2002) introduces an auxiliary variable w (w(a) JacA. It is updated follows. In the 1-st phase, we initialize as {I,...,1}, 1]. phase (12 1), we update w for T' iterations. In the f-th iteration, we sample an action a apply the action a, to the CIL system, and compute R(a,, h2) using Eq. 5. After that, we update w(a;) in w as,
In each
w(ax)) w(a)) exp(e pt)p
(6)
where& & can be regarded as the learning rate. After updating w, we get the policy x=w/|[w]] The pseudocode is available in Appendix A (Algorithms and 2).

---

Under review as a conference paper at ICLR 2023
20 exemplarsiclass
10 exemplarsíclass
exemplarsiclass
Method
Average
Last
Average
Last
Average
Last
LwF (2016)
53.19
43.18
45,96
34.10
35,41
24.91
w/ ours
59.08
49.15 +5.97
53.61 +7.65
38.36 +1.26
41.55
28.66 +3.77
iCaRL. (2017)
57.12
47.49
53.43
41.49
43.73
34.33
w/ ours
61.24 44.12
51.47 +3,98
59.11 +5.68
46.42 .4.93
51.55 47,82
39.35 45.02
LUCIR (2019)
63.17
53.71
60.50
49.08
51.36
39.37
w/ ours
65,28 +2.11
56.23 +2.52
64.79 +1.29
55.44 +6.36
62.74 +11.35
53.25 +13.88
LUCIR-AANeis (2021a)
66.12
55,27
61.12
48.83
53.81
42.93
ours
67.65 41.53
59.18 +3.91
64.30 v3.18
52.92
60.27 46.46
48.45 45.52
FOSTER (2022)
70.62
62.97
62.03
52.23
56.80
43.11
w/ ours
71.97 +1.35
64.43
+1.46
65.12
+3.09
54.81
+2.48
62.78
45.08
50.72 47.61
Table 1: Evaluation results (%) on CIFAR-100 (N=5) using different baselines w/ and w/o our PlaceboCIL. "Average" denotes the average accuracy over all phases. "Last" denotes the last phase (5-th phase) accuracy.
EXPERIMENTS
We evaluate our method on three CII. benchmarks and achieve consistent improvements over multi- ple baseline methods. Below we introduce datasets and implementation details, followed by results and analyses, including the comparison to the state-of-the-art, an ablation study, and the visualiza- tion of our placebos.
Datasets and free image streams. We use three datasets: CIFAR- 100 (Krizhevsky et al., 2009), ImageNet- 100 (Rebuffi et al., 2017), and ImageNet- Ik (Russakovsky et al., 2015). ImageNet- 100, which contains 100 classes, is sampled from ImageNet- 1k. We use exactly the same classes orders as the related works (Rebuffi et al., 2017: Hou et al., 2019). For CIFAR-100, 100, we use ImageNet- 1 as the free image stream. For ImageNet- 1000 we use 900-class subset of ImageNet-IK, Ik, which is the complement of ImageNet-100 in ImageNet-Ik. Ik. For ImageNet-Ik, we use a 1, 000-class subset of ImageNet-21k (Deng et al., 2009) without any overlapping class (different super-classes from those in ImageNet-1k).
Implementation details. Following (Hou al., 2019; Douillard al., 2020; Liu et al., 202 2021a:b) labb), modified 32-layer ResNet for CIFAR-100 and an 18-layer ResNet for ImageNet datasets. The number of exemplars for each class is 20 in the default setting. The training batch size is 128. On CIFAR- 100 (maageNetSSubset/1k we train it for 160 (90) epochs in each phase, and divide the learning rate by 10 after 80 (30) and then after 120 (60) epochs. If the baseline is POD-AANets (Liu al., 2021a), we fine-tune the model for 20 epochs using only exemplars. We apply different forms of distillation losses on different baselines: (1) if the baselines are LwF and iCaRL, we use the soft- max KL divergence loss; (2) if the baselines are LUCIR and AANets,
the cosine embedding loss (Hou et al., 2019); and (3) if the baseline is POD- AADets, we use pooled outputs distillation loss (Douillard al., 2020). For our PlaceboCIL. [24] and [P] are set as 1, 000 and 200, respectively. All experiments of our PlaceboCII. in the main paper use the "strict budget" setting, i.e., deleting I Pl samples from training data in order to not exceed the memory budget. An ablation study on the memory budget setting is provided in Appendix G. More implementation details are provided in Appendices (benchmark protocol), D (network architecture), and E (training configurations).
Results on five baselines. Table shows the average and laste accuracy for five baselines (i.e., LwF (Li & Hoiem, 2016). iCaRL (Rebuffi et al., 2017), LUCIR (Hou et al., 2019), AANets (Liu et al., 2021a), and FOSTER (Wang et al., 2022)). From the table, we make the following observa- tions. 1) Using our PlaceboCII. boosts the performance of the baselines clearly and consistently in all settings, indicating that our method is generic and efficient. 2) When the number of exemplars decreases, the improvement brought by our method becomes more significant. For example, the last-phase accuracy improvement of LUCIR increases from 2.52 to 13.88 percentage points when the number of exemplars per class decreases from 20 to 5. This reveals that the superiority of our method is more obvious when the forgetting problem is more serious (with fewer exemplars) due to tighter memory budget in CIL, 3) Our PlaceboCII. can boost the performance of all KD terms, i.e.,
7

---

Under review as a conference paper at ICLR 2023
CIFAR-100
ImageNer- 100
ImageNet- Ik
Method
N=5
10
25
5
10
25
5
10
LwF (2016)
53.19
46.98
45.51
53.62
47.64
44.32
44.35
38.90
iCaRL (2017)
57.12
52.66
48.22
65.44
59.88
52.97
51.50
46.89
TPCIL (2020)
65.34
63.58
76.27
74.81
64.89
62.88
DDE (2021)
65.34
63.58
76.27
74.81
64.89
62.88
GeoDL (2021)
65.14
65.03
63.12
76.63
75.40
71.43
65,23
64,46
DER (2021)
68.65
67.48
66.18
78.40
78.20
75.40
68.13
65.97
ELI (2022)
68.78
66.62
64.72
73.54
71.82
70.32
GD+ext (2019)
63.17 58.
51.79
=042
5:
72.08
65, 55
MUC-LwF (2020b)
72.31
10
98
62.93
POD-AANets (2021a)
42 12
76.63
40.47 45.40
71.43++0.2
67.60
64.79
PlaceboCIL (ours) 67 97.65=nm 40 65. 65.78
78.24
77. 17 75.85
68.55 65.49
FOSTER (2022)
70.62 68.43_0,45633 2..2
80.21 60 77.63.10.33 69.27.10.0
69.32.10.7
66.07 1066
w/ PlaceboCIL (ours) 71.97
67.02 2..5 82 79.52 22.79 71.02 2:039 60.8
Average accuracy (%) across all phases. The first block shows top-performing CIL methods. The second block shows CIL methods that use unlabeled data. The third block shows our method.
Table 2:
not only for logits-based KD (Rebuffi et al., 2017) but also for feature-based KD (Hou al., 2019; Douillard et al., 2020).
Comparisons to the state-of-the-art. Table 2 (Blocks 1&3) shows the results of our best model (taking PlaceboCII.
plug-in module in the top method (Wang et al., 2022)) and some recent top-performing methods. We can see that using our PlaceboCIL outperforms all previous methods. Intriguingly, we find that there are more serious forgetting problems. For example, when N=25, we improve POD-AANets by 4.4% the ImageNet- 100, while this number is only 1.6% when N=5 (which is easier setting with more saturated results). This reflects the encouraging efficiency of our method for reducing the forgetting of old class knowledge in CIL models.
surpass others more when the number of phases is larger-whers
Comparisons to the CIL methods ing unlabeled data. Table 2 (Blocks 2&3) shows the results of our best model and CIL methods using unlabeled data
iCaRL
LUCIR-AANets
No. Setting
Average
Last
Average
Last
(GD+ext (Lee al., 2019) and MUC- LwF (Liu et al., 2020b)). We can see that method consistently performs bet-
Baseline
57.12
47.49
66.72
57.77
2
PlacehoCII.
61.01
51.45
67.16
59.14
3
Overlapping
62.15
52.62
67.48
59.06
ter than others. For another related work,
Non-overlapping
61.52
51.70
67.01
58.53
DMC (Zhang al., 2020), we didn't 't find the public code. So, we compare ours with DMC using their paper's setting: iCaRL
5
New data
57.70
47.51
66.69
57.33
6
Old data (oracle)
66.64
58.03
68.82
61.52
7
w/o Online learning
60.27
50.57
66.91
58.88
w/ ours achieves 62.3%, while the result of DMC 59.1% (CIFAR-I00 -100, 10 phases, 10 classes/phase). Please kindly refer to Ap- pendix B for more analyses.
8
Offline RL
61.09
50.81
67.31
59,26
9
Higher confidence
60.43
49.36
66.97
58.12
10
Random placebos
56.27
46.64
66.23
57.22
Ablation study. Tables 3 (and A1) show the ablation results for free data streams, policy learning methods, and placebo se- lection strategies.
Table 3: Ablation results (%) on CIFAR-100, N=5. (1) First block: baselines. Row 1 shows the base- lines. Row 2 shows our method. All the follow- ing ablation settings (Rows 3-10) are based on Row 2. (2) Second block: different free data streams.
1) First block. Row shows the baselines. Row 2 shows method.
Rows 3-6 show the ablation results for the following free data streams. (3) Third block: different policy learning methods. Row 7 is for using fixed evaluation functions (B:=y==)). Row 8 uses the offline RI (the REINFORCE algorithm) to train the selection policy. (4) Fourth block: different placebo selection strate- gies. Row 9 uses unlabeled data with higher confi- dence. Row 10 uses them randomly.
2) Second block: different free data streams. Rows 3-6 show the ablation re- sults for the following free data streams. (1) "Overlapping" means including sam- ples from the overlapping classes between (2) "Non-
CIFAR- 100 and ImageNet.
8

---

Under review as a conference paper at ICLR 2023
CIFAR 100 1000llass class: soad
CIFAR- closs table
S heerian Pusky
unieycle
marimha
home theater
new old dana plaoehox
radio telescope
upright
(a) Selected placebos
(b) I-SNE visualization
Figure 3: (a) Selected placebos for two CIFAR-100 classes and their GradCAM activation maps. The free image stream is non-matching ImageNet-Ik, Ik. (b) The t-SNE results on CIFAR- (N=5). For clear visualization, we randomly pick five new classes and five old classes. The purple, light blue, and dark blue points denote the new data, old data, and selected placebos, respectively.
overlapping" means using only the samples of non-overlapping classes between CIFAR- 100 and ImageNet (more realistic than "Overlapping"). (3) "New data" means using only the current-phase new class data (i.c., without using any free data stream) as candidates to select placebos. (4) "Old data" means the original old class data are all accessible when computing KD loss (i.e., upper bound of KD effect). Please note that in (1) and (2), two classes are considered ""verlapping" if their classes or super-classes overlap. For example, "n02640242 sturgeon" " in ImageNet-Ik, is regarded as overlapping class of the "fish" in CIFAR-100 100, because they overlap at the level of super-class (i.e., "fish"). When comparing Row with Row 2, we can find that our method robust to change of data streams: even if all overlapping classes same-level performance. Comparing Row 5 with Row 2, we can get a clear sense that using addi- tional unlabeled data is definitely helpful. Comparing Row 6 with Row 2, we see that method achieves comparable results to the upper bound.
removed, our method can still achieve the
3) Third block: different policy learning methods. Row 7 is for using fixed evaluation functions (3,=x=1). Row uses the offiine RL (the REINFORCE algorithm (Liu et al., 2021b)) to train the selection policy. Comparing Row 7 with Row 2 shows that using online learning successfully boosts the model performance. Comparing Row 8 with Row 2, we are happy to see that our online learning method achieves the same-level performance the offline RL while the training time is much less. The training time of the bascline (without learning policy) is 2.7 hours. It becomes around 650 hours we solve the MDP by offline RI. In contrast, using our online method takes only 4.5 hours.
4) Fourth block: different placebo selection strategies. Row uses unlabeled data with higher confidence following (Lee et al., 2019). Row uses them randomly following (Zhang al., 2020). Comparing these results with Row 2 shows our superiority. The "mini-batch-base memory reusing strategy" is applied in Rows and 10.
Visualization results. Figure (a) demonstrates the activation maps visualized by Grad-CAM for the placebos of two old classes on CIFAR-100 ("road" and "table"), ImageNet-Ik is the free data stream. We can see that the selected placebos contain the parts of "road" and "table" even though their original labels (on ImageNet- 1k) are totally different classes. While this is not always the case, our method seems to find sufficiently related images to classes that activate the related neurons for old classes ("road" and "table"). To illustrate that, Figure 3 (b) shows t-SNE results for placebos, old class data (not visible during training), and new class data. We can see that the placebos are located near the old class data and far away from the new class data. This is why placebos can recall the old knowledge without harming the new class learning.
CONCLUSIONS
We proposed novel method, PlaceboCIL, which selects high-quality placebo data from free data streams and uses them to improve the effect of KD in CIL. We designed an online learning method memory
to make the selection of placebos more adaptive in different phases and mini-batch reusing strategy to control memory usage. Extensive experiments show that our method is general and efficient.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
Computational costs. Deep learning methods require intensive usage of computing resources, which is not climate-friendly. It calls for future research into proposing more effective training strategies that can speed up the training.
Privacy issues. Keeping old class exemplars has the issue of data privacy. This calls for future research that explicitly forgets or mitigates the identifiable feature the data.
Licenses. We use the open-source code for the following papers: AANets (Liu et al., 2021a), (Rebuffi
(Rebuff et al., 2017), Mnemonics (Liu et al., 2020a), PODNet (Douillard et al., 2020), FOS- TER (Wang et al., 2022). They are licensed under the MIT License.
Datasets. We use three datasets in our paper: CIFAR-100 (Krizhevsky al., 2009), ImageNet- 100 (Rebuffi et al., 2017) and ImageNet- 1k (Russakovsky et al., 2015). The data for both datasets are downloaded from their official websites and allowed to use for comcommercial research and educational purposes.
REPRODUCIBILITY STATEMENT
We include the details of reproducing our experiments in Section (main paper), Appendix C, Appendix D, and Appendix E. We also provide the code in the supplementary.
REFERENCES
Davide Abati, Jakub Tomezak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learn- ing. In CVPR, pp. 3931-3940, 2020. 3
Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi multi-armed bandit prob- lem. In COLT, volume 23, pp. 39.1-39.26, 2012. 3,6
Peter Auer and Ronald Ortner. Ucb revisited: Improved regret bounds for the stochastic multi-armed bandit problem. Periodica Mathematica Hungarica, 61(1-2):55-65, 2010. 3
Peter Auer, Nicolo Besa-Bianchi, Yoav Freund, and Robert Schapire. The nonstochastic multi- armed bandit problem. SIAM journal on computing, 32:1):444-111 2002. 3, 6
Charilaos Christopoulos, Athanassios Skodras, and Touradj Ebrahimi. The jpeg2000 still image coding system: overview. IEEE transactions consumer electronics, 46(4): (66(4)1:100 1111110311277 2000. 14
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning, In ECCV, pp. 86-102, 2020. 1, 2, 8, 10, 14
Eyal Even-Dar. Sham M Kakade, and Yishay Mansour. Experts in markov decision process. In NIPS, pp. 401-408, 2005. 3,
Eyal Even-Dar Sham M Kakade, and Yishay Mansour. Online markov decision processes. Mathe- matics of Operations Research, 14(33:726-7366 2009. 3,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016. 14
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In CVPR, pp. 831-839, 2019. 1, 2, 3, 7, 8, 14, 15
10

---

Under review as a conference paper at ICLR 2023
Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. Distilling causal effect of data in class-incrementi learning. In CVPR, 2021. 8
KJ Joseph, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Vineeth N Balasub- ramanian. Energy-based latent aligner for incremental learning. In CVPR, pp. 7452-7461, 2022. 3, 8
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. 7, 10
Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In ICCV, pp. 312-321, 2019. 3, 8, 9, 13, 14
Yingying Li, Aoxiao Zhong, Guannan Qu, and Na Li. Online markov decision processes with time-varying transition probabilities and rewards. In ICML workshop on Real-world Sequential Decision Making, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. In ECCV, pp. 614-629, 2016. 3, 7, 8, 15
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi- class incremental learning without forgetting. In CVPR, pp. 12245-12254, 2020a. 1, 10, 14
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for clas-inncrmmental learning. CVPR, pp. 2544-2553, 2021a. 1, 2, 7, 8, 10, 14,
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Rmm: Reinforced memory management for class-
incremental learning. In NewrlPS, 2021b. 7, 15
Yu Liu, Sarah Parisot, Gregory Slabaugh, Xu Jia, Ales Leonardis, and Tinne Tuytelaars. More classifiers, less forgetting: A generic multi-classffier paradigm for incremental learning, In ECCV, pp. 699-716, 2020b. 8,
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pp. 109-165. Elsevier, 1989.
K. McRae and P. Hetherington. Catastrophic interference is eliminated in pre-trained networks. In CogSei, 1993.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: simple approach that questions our progress in continual learning. In ECCV, pp. 524-540, 2020. 3
R. Ratcliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, 97:285-3083 -308 1990.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In CVPR, pp. 5533-5542, 2017. 1, 2, 6, 8, 10, 14
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252, 2015. 7, 10
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Ko- ray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv, 1606.04671, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In CVPR, pp. 618-626, 2017.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NIPS, pp. 2990-299999 2017.

---

Under review as a conference paper at ICLR 2023
Christian Simon, Piotr Koniusz, and Mehrtash Harandi. On learning the geodesic path for incre- mental learning. In CVPR, pp. 1591-1600, 2021. 3, 8
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NIPS, pp. 4077-4087, 2017. 4
Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong. Xing Wei, and Yihong Gong. Hopology-presereee class-incremental learning. In ECCV, pp. 254-270, 2020. 3,8
Fu- Yun Wang, Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and com- pression for caassincremmntal learning. In ECCV, 2022. 2, 3, 7, 8, 10
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.
Large scale incremental learning. In CVPR, pp. 374-382, 2019. 14
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In NeurIPS, pp. 899 908. 2018. 3
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning, In CVPR, pp. 3014-3023, 2021. 3,
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. WACV, pp. 1131- 1140, 2020. 2, 3, 8, 9, 13, 14
Fei Zhu, Zhen Cheng. Yao Zhang, and Cheng-lin Liu. Class-incremental learning via dual augmentation. NeurlPS, pp. 14306-1431, 2021. 1.3
12

---

Under review as a conference paper at ICLR 2023
APPENDICES
A ALGORITHMS
Supplementary Section 3. Algorithm summarizes the overall training steps of the proposed method the i-th phase (i 2 1). Algorithm presents the training with placebos for a specific action
Algorithm/ 1: Our online learning algorithm the i-th phase (i21)
Input Old model O training data 71.., testing data Q1 learnable parameters numbers of epochs Mj and M2.
Output: New model G, new exemplars Eus, learnable parameters // Policy learning
i=1 then
L Initialize w (1....,14 1}:
for L in i, i.... do Randomly sample a class-balanced subset Bise, from Ti:cyri Create the local environment h4 ((TI:e,) \ B Bi
Set the policy π = w/l w|l;
Sample an action a, π for j in i...., i + do
Train θ, for Mj epochs by Algorithm with inputs 9 ae. hi; Collect the reward rh. (O,,
Compute the cumulative reward R(a,, h2) by Eq. 5;
11
12
Update w by Eq. 6;
// CIL training
13 Sample an action a; m;
14 Train θ for M2 epochs by Algorithm 2 with inputs ,,-1, a, Hz = (Tree 21:;:):
Select new exemplars Esc; from Ti....
Algorithm 2: Training with placebos for action a
Input : Old model Ooi, action {,,,7) }. environment (T, Q).
Output: New model θ, reward ry (O,a) a) (i.e., the testing accuracy).
Initialize θ with Qali
Create {Sm (=) based on a {B, 7 using Eq. 2;
for epochs do
Set p Ø;
while p = Ø do
Sample 2/ from the free image stream; Select placebos P c li using Eq. 3; for iterations do
Sample mini-batches p. d, and e; Compute the loss L by Eq. and update θ; Update placbo buffer P p:
11
12 Compute the reward (O, a) on Q
B COMPARISONS TO THE CIL METHODS USING UNLABELED DATA.
This is supplementary to Section 4 "Comparisons to the CIL methods using unlabeled data."
Part I: Comparing with GD+ext (Lee et al., 2019) and DMC (Zhang et al., 2020). Lee et al. (2019) use the unlabeled data to compute the distillation loss in CIL. Our work also uses unlabeled data but differs from (Lee al., 2019) in three aspects:
13

---

Under review as a conference paper at ICLR 2023
Our method is focused the learnable strategies of sampling helpful unlabeled data for CIL. It is generic and can be plugged into different distillation-bar based frameworks, including both logit distillation (LwF and iCaRL) and feature distillation (LUCIR and PODNet). In Table 2, we also show the empirical results (comparison) to demonstrate our superiority.
We propose online learning algorithm. Its learned policy can adaptively produce phase- specific functions to evaluate the quality of the unlabeled samples. So our selection func- tions change accordingly when the number of phases increases. Lee et al. (2019) 1)'s selection strategy remains unchanged when the number of phases increases. Zhang et al. (2020) use the unlabeled data without selection. In Table 3, we show our learnable selection functions generally perform better than (Lee et al., 2019) and (Zhang et al., 2020). Please kindly refer to Section "Ablation study" for the results.
Our employment of unlabeled data is dynamic and takes little memory out of the tal memory budget. We maintain the memory budget strictly the same as the baselines (LUCIR (Hou et al., 2019). PODNet (Christopoulos et al., 2020a)), by loading fewer samples of new classes. In contrast, (Lee et al., 2019) requires double-size memory compared the baselines (LUCIR (Hou et al., 2019), POD- Net (Christopoulos et al., 2000)), and Mnemonics (Liu et al., 2020a)).
al., 2000), and Mnemonics (Liu
Part II: Comparing with MUC (Liu al., 2020b). Our method and (Liu al., 2020b) use unla- beled data for different purposes and in different ways. We use the unlabeled placebo data from free image stream to improve the positive effect of KD training CIL models. Liu et (2020b) integrate the ensemble of multiple FC classifiers to set regularization-be based constraints. They use the unlabeled data to compute the classifier discrepancy loss by using the predictions of multiple classifiers.
C BENCHMARK PROTOCOLS
This is supplementary to Section 4 "Implementation details." We follow the benchmark protocol used in (Douillard al., 2020; Hou al., 2019; Liu al., 2021a; 2020a). Given dataset, the initial model (in the O-th phase) is trained on the data of half of the classes. Then, it learns the remaining classes evenly in the subsequent N phases. For N. there are three options (5, 10, and 25), and the corresponding settings are called " "N-Phase". The learned model in each phase is evaluated on the test set containing all seen classes. In the tables, we report average accuracy over all phases and the last-phase accuracy.
D NETWORK ARCHITECTURE DETAILS
This is supplementary to Section 4 "Implementation details." Following (Hou et al., 2019; Liu et al., 202 Rebuffi et al., 2017; Wu et al., 2019), we modified 32-layer ResNet (Rebuffi et al., 2017) for CIFAR-100 100 and an 18-layer ResNet (He et al., 2016) for ImageNet- and ImageNet- 1k Please note that it is standard
shallower ResNet for ImageNet-100 and ImageNet- 1k in CIL. The 32-layer ResNet consists of initial convolution layer and 3 residual blocks (in a single branch). Each block has 10 convolution layers with 3 3 kernels. The number of filters starts from 16 and is doubled every next block. After these 3 blocks, there is an average- average-pooling layer to compress the output feature maps feature embedding. The 18-layer ResNet follows the standard settings in (He et al., 2016). We deploy AANets using the same parameters as its original paper (Liu et al., 2021a).
E MORE TRAINING CONFIGURATIONS
This is supplementary to Section 4 "Implementation details." The training of the classification model exactly follows the uniform setting in (Douillard et al., 2020; Hou et al., 2019; Liu et al., 2021a; 2020a).
14

---

Under review as a conference paper at ICLR 2023
F ABLATION STUDY ON MORE BASELINES
This is supplementary to Section 4 "Ablation study." In Table A1, provide the ablation results on another two baselines, LwF Li & Hoiem (2016) and LUCIR Hou et al. (2019).
LwF
LUCIR
No.
Setting
Average
Last
Average
Last
Baseline
53.19
43.18
63.17
53.71
PlaceboCIL.
59.06
48.56
63.42
56.18
Overlapping
58.95
48.71
65.73
57.26
Non-overlapping
58.59
49.28
65.48
56.99
New data
54.06
43.94
64.21
53.98
Old data (oracle)
61.41
51.16
67.02
58.98
w/o Online learning
57.06
46.12
63.83
55.33
Offline RL
59.01
48.18
65.14
56.95
Higher confidence Random placebos
56.98 50.99
46.20
63.60
55.50
10
40.22
64.16
55.40
Table A1: Ablation results (%) on CIFAR-100, N=5.
G ABLATION STUDY FOR THE MEMORY BUDGET SETTINGS
This is supplementary to Section 4. In Table A2, Row shows baselines. Rows 2 and 3 show the baselines with our PlacehoCII as a plug-in module. Row "budget" means the combined size of episodic memory and the placebo buffer is (strictly) equal to the episodic memory size in baseline methods. To this end, we randomly delete 22 P| samples from the new class data Peli+ to obtain the buffer memory storing the unlabeled data batch U and P. Row 3 mnonbbugget" means are allowed to use a little additional memory to save the unlabelled data 2/ and P without deleting other samples. Comparing Row Row 2, only reduces 0.04 percentage points on average under the strict "budget" setting. The reason the amount of new data greatly larger than that of old samples, and removing a small batch of new data does not harm the performance much, as observed in (Liu et al., 2021b).
that the performance of our PlaceboCIL.
LWF
iCaRL
LUCIR
AANets
No. Setting
Average
Last
Average
Last
Average
Last
Average
Last
Bascline
53.19
43.18
57.12
47.49
63.17
53.71
66.72
57.77
2
PlaceboCIL (budget)
59,06
48.56
61.01
51.45
65.42
56.18
67.16
59.14
PlacehoCII. (non-budget)
59.08
49.15
61.24
51.47
65.28
56.23
67.27
59.15
Table A2:
Ablation results (%) CIFAR-100, "Averaaer" and "Last" denote the average curacy over all phases and the last-phase accuracy, respectively.
H EQUAL-SIZE SPLIT RESULTS
This is supplementary to Section 4 "Ablation study." In Table A3, we supplement the results using the equal-size split on CIFAR 100, 10-phase (10 classes/phase, 20 exemplars/llass) We can observe that using placebos selected by heuristic evaluation functions (3,=x=1) consistently improves the results on three baselines. Using the online learning algorithm can further boost performance. The observations are similar to using the original setting in the main paper.
I ABLATION RESULTS FOR DIFFERENT UNLABELED DATA SOURCES
This is supplementary to Section 4 "Ablation study." We believe that the key to success is the design of our method instead of the choice of unlabeled data sources. To verify this, provide
15

---

Under review as a conference paper at ICLR 2023
LwF
iCaRL
LUCIR
No. Setting
Average
Last
Average
Last
Average
Last
Bascline
53.85
41.00
59.70
45.29
56.71
42.78
PlaceboCIL (ours)
57.68
42.44
62.32
46.52
57.89
44.08
PlaceboCIL (ours, wo online learning)
56.31
41.76
61.05
46.02
57.01
43.13
Table A3: Supplementary to Table
Ablation results (%) using equal-size split on CIFAR-100,
10-phase (10 classes/phase).
the results for new ablative setting: "using random unlabeled data" (to compute the distillation loss) in Table A4. We can observe that no matter what unlabeled data sources we use, our method consistently performs better than using random unlabeled data.
LwF
iCaRL
LUCIR
AANets
No. Setting
Average
Last
Average
Last
Average
Last
Average
Last
1 Baseline
$3.19
43.18
$7.12
47.49
63.17
53.71
66.72
57.77
2
PlaceboCII. (ours, all)
59.29
49.64
61.17
50.96
65.48
56.77
67.33
59.32
3
PlaceboCIL (ours, overlapping)
58.95
48.71
62.15
52.62
65.73
57.26
67.48
59.06
4
Random unlabeled data (all)
50.99
40.22
56.27
46.64
64.16
55.45
66.23
57.22
4
Random unlabeled data (overlapping)
50.80
40.94
55.70
46.47
64.23
54.68
66.58
57.08
Table A4: Supplementary to Table 3. Ablation results (%) on CIFAR-100, N "Averrer and
"Last" denote the average accuracy over all phases and the last-phase accuracy, respectively. "All" denotes using all data from ImageNet, and "overlapping" means including samples from the over- lapping classes between CIFAR-100 and ImageNet.
J REHEARSAL-FREE EXPERIMENTS
This is supplementary to Section "Ablation study."
In Table A5, we provide the "rehearsal-fres results for two variants of our method: 1) PlacboCIL. (original) w/o exemplars: we don't use any exemplars to compute the loss, only storing one exem- plar each class to compute the evaluation function (Eq. 4); 2) PlacboCIL (higher confident) w/o exemplars: we don't 't use any exemplars, and select unlabeled data with higher confidence as placebos.
We can observe that using placebos is effective in the "rchearsal-free" setting. The original Place- boCIL improves the average accuracy of the baseline by 6.30 percentage points. The "higher confi- dent" version performs a little worse than the original, but still improves the average accuracy of the baseline by 3.40 percentage points.
No. Setting
Average
Last
Baseline
66.72
57.77
PlacchoCII. (original)
67.16
59.14
PlaceboCIL (higher confidence)
66.97
58.12
Baseline w/o exemplars
50.01
39.12
PlacehoCIL. (original) w/o exemplars
56.31
44.02
PlaceboCIL (higher confidence) w/o exemplars
53.41
41.93
Table A5: Ablation results (%) CIFAR-100, N=5. Baseline: LUCIR+AANets Liu et al. (2021a).
16


Reviews:
[{"review_id": "qxvb6mGV-RP", "reviewer": "Reviewer_4WyN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The setting of this paper is to use wild data (free data) as an additional resource to do class-incremental learning. This paper proposes an online placebo (free images) selection policy to select good images from the free data stream. The selected images are used for knowledge distillation. Thus the proposed method can be combined with existing CIL methods to improve their performances. To train the evaluation policy, this paper introduces an online reinforcement learning algorithm, which reduces training computation costs compared with offline reinforcement learning. Experiments show that the proposed method can enhance the performances of many existing methods.", "summary_of_the_review": "There are concerns about the results. The baseline methods compared are without additional data, which is somehow unfair. Moreover, the proposed offline RL method takes extra costs but only brings marginal improvement. ", "strength_and_weaknesses": "Strengths:\n(1) This paper proposes an evaluation function to select good images from the free data stream. The proposed methods can provide a plug-in component for existing CIL methods.\n(2) The experiments validate the effectiveness of the proposed framework.\n(3) The paper is well organized, clearly motivated, and contributes novelty.\n\nWeaknesses:\n(1) The results are not so significant. There are some existing works using wild/free images as additional data for incremental learning. Compared with previous works, this work is a universal plug-in component to enhance many existing CIL works. However, this paper uses the methods without additional data as baselines, which obviously have improvements. In fact, one intuitive way of using free images is to find a confidence score to select good images or generate pseudo labels, which should be the true baseline. From Table.3 and A1, we can find that \"Higher confidence\" seems a baseline that does not use the proposed method but also can select good images for knowledge distillation. The gaps between the proposed method and \"Higher confidence\" is even close to random errors for LUCIR-AANet and iCaRL. From Table.3, using the proposed methods gets only 0.58% improvement in average accuracy for iCaRL, and the results for LUCIR-AANet is only 0.37%, which means \"Higher confidence\" is the basic baseline that should be compared with. This indicates that the overall good performance is largely due to the additional data, even a simple confident selection metric can provide relatively good improvements. \n(2) It would be helpful to report the mean and standard deviation to take out any concerns related to randomness.\n(3) How to use ImageNet-1K as additional dataset for modified ResNet-32? Because the image size of CIFAR-100 is 32x32. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SjWI9r90JN", "reviewer": "Reviewer_rXB7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method to leverage free unlabeled image as \"placebos\" to calculate knowledge distillation loss for class incremental learning. A corresponding online learning algorithm and a memory reusing strategy are proposed to avoid extra memory usage. Comprehensive experiments on benchmark datasets demonstrate the effectiveness of the method.", "summary_of_the_review": "The paper presents an interesting idea of levering free/public available data to help class incremental learning. However, the empirical evaluation of the method itself needs careful check and explanation. Therefore, I recommend weak reject initially.", "strength_and_weaknesses": "The main strength of the paper is the idea of levering free/public available data to help class incremental learning.\n\nHowever, there are multiple weaknesses that influence the overall quality of the paper:\n- I actually have a fundamental question of the problem setting. Since the idea is to use the placebos of old classes, is it possible to totally get rid of old classes, i.e., rehearsal-free? \n- In (4), the old class exemplars still play an important role in both the rehearsal loss (first term) and the distillation loss (second term). What if the exemplars are totally removed from (4)?\n- I doubt that the results in Table 1 and 2 are strictly fair. Since saving the free data batch requires additional memory, shouldn't the competing methods get the same amount of memory to save buffered examples? For example, if $|\\mathcal{U}|  + |\\mathcal{P}| = 1200$, the buffer size for other methods should be $1200$ + \"buffer size used in the results\". \n- The ablation study in Table 3 is very unclear.\n- The layout of Figure 1 needs improvement, especially (b).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NO5-OmEE6jw", "reviewer": "Reviewer_5n8q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes using placebos (chosen from a free image stream such as Google Images) in class incremental learning. It formulates the policy training process as an online Markov Decision Process, and achieve improvement over prior works.", "summary_of_the_review": "Reject is rated, given the crucial assumption of accessing free image stream is not carefully validated. Another concern is missing an important setup (see weakness).", "strength_and_weaknesses": "Strength:\n1. The paper is written well.\n2. Clear improvement is achieved over baseline.\n\nWeakness:\n1. The assumption of accessing free image stream (like Google Images) is NOT carefully validated. If the free image stream includes images of old classes or highly correlated to old classes, this essentially increases the size of memory which is not fair.  A clear distinction in terms of statistics between memory and placebo should be provided. (Please do NOT just say they are from different data sources.)\n\n2. Another CL setup is missing: the 0th phrase does not have half of classes, but just the number of incremental class (e.g. 5 or 10 classes).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JYW-cdmQXOD", "reviewer": "Reviewer_DLTo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to address the catastrophic forgetting problem in class-incremental learning (CIL). As most of the old samples are not accessible in CIL, the authors propose to use unlabeled external data (i.e., placebos) to compute the knowledge distillation loss to consolidate the old knowledge. To achieve online placebo selections, the authors propose to formulate the CIL task as an online MDP and design a novel online learning algorithm for learning an online policy to control the placebo selection. Further, they propose a mini-batch-based memory-reusing strategy to avoid extra memory usage. The authors conduct extensive experiments on three CIL datasets, CIFAR-100, ImageNet-100, and ImageNet-1k, and show their method achieves state-of-the-art performance. ", "summary_of_the_review": "Overall, I think this is a high-quality paper with a novel method and extensive experiment results. The proposed direction, selecting extra data to help to improve CIL, is promising in my view. Further, this paper achieves state-of-the-art performance and provides open-source code. So, I recommend acceptance. ", "strength_and_weaknesses": "Strengths\n\n1. The proposed online MDP formulation for CIL is novel and interesting. I think it makes sense to formulate the CIL as an online MDP and learn a policy to control the learning process, e.g., placebo selection. I think this formulation can also be used to learn policies to adjust other components in CIL. \n\n2. The idea of using extra unlabeled data is interesting and reasonable. Because unlabeled data can be obtained easily from the Internet without much effort. I think this is a promising direction to improve performance when training with imbalanced data, e.g., CIL. \n\n3. This paper is well-written and easy to follow. The authors clearly explain their motivation in the introduction. Besides, the authors also provide extensive figures and visualizations to make their explanations clearer. For example, Figure 1(b) clearly visualizes the problem when using the new data to compute the KD loss. Figure 1(c) demonstrates the effectiveness of the proposed placebo selection method. \n\n4. Extensive experiment results are provided to show the effectiveness of the proposed method. For example, in Table 1, the authors combine their method with five different CIL baselines and achieve consistent improvements. It is impressive to see that their method can work with many different methods and KD losses. In Table 2, the authors show their method achieves state-the-art performance on three datasets. Furthermore, the authors also provide detailed ablation results and visualizations. In summary, I think the experiments are very impressive. \n\n5. The authors provide open-source code and detailed configures. These materials are helpful for the following researchers. \n\nWeaknesses\n\n1. It would be better to move more training details (e.g., Appendix E training configures) from the appendices to the main papers. Because I think these details are very important for reproducing the results. \n\n2. The authors should also compare some famous CIL methods (e.g., LUCIR (Hou et al., 2019) and iCaRL (Rebuffi et al., 2017)) in Table 2 because they are very important baselines for class-incremental learning. \n\n3. The authors can provide more visualization results for different classes (like Figure 3(a)) in the appendix. These visualization results can help the following researchers to understand how the placebo images work in the class-incremental learning models. \n\n4. Some recent related papers published in CVPR 2022 should also be discussed and compared. For example, [1] also uses the same benchmark protocol and should be compared in Table 2. \n\n[1] Joseph, K. J., et al. \"Energy-based Latent Aligner for Incremental Learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject