Effective Decision Boundary Learning for Class Incremental Learning
Authors: KunChi Li
Jun Wan
Sergio Escalera
Zhen Lei
Shan Yu
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Class Incremental Learning
Catastrophic Forgetting
Long Tail
Mixup
Knowledge Distillation
Influence Balance

Abstract: Rehearsal approaches in class incremental learning (CIL) suffer from decision boundary overfitting to new classes, which is caused by two factors: insufficiency of old classes data for knowledge distillation (KD) and imbalanced data between the old and new classes because of the limited storage memory. In this work, we present a simple but effective approach to deal with these two factors to optimize the decision boundary. First, we employ the mixup knowledge distillation (MKD)  and re-sampling strategy to improve the performance of KD, which would greatly alleviate the overfitting problem. Specifically, it utilizes mixup and re-sampling to synthesize adequate data that are more consistent with the latent distribution between the learned and new classes. Second, inspired by the influence balanced (IB) loss used in handling the long-tailed data, we propose a novel incremental influence balanced (IIB) method for CIL to address the classification on imbalanced data, which re-weights samples by their influences to create a proper decision boundary. With these two improvements, we present the effective decision boundary learning (EDBL) algorithm which improves the performance of KD and deals with the imbalanced data classification simultaneously. Experiments show that the proposed EDBL achieves state-of-the-art performances on several CIL benchmarks. 

Effective Decision Boundary Learning for Class Incremental Learning
Anonymous Author(s)" Affiliations emails
Abstract
Rehearsal approaches in class incremental learning (CIL) suffer from decision boundary overfitting to new classes, which mainly caused by two factors: insuffi- ciency of old classes data for knowledge distillation and imbalanced data learning between the learned and new classes because of the limited storage memory. In this work, we present simple but effective approach to tackle these two factors. First, we employ re-sampling strategy and Mixup Knowledge Distillation (Re-MKD) improve performances of KD, which would greatly alleviate the overfitting problem. Specifically, we combine mixup and re-sampling strategy to synthesize adequate data used in KD training that are more consistent with the latent distribu- tion between the learned and new classes. Second, we propose a novel Incremental imbalanced
Influence Balance (IIB) method for CIL tackle the classification data by extending the influence balance method into the CIL setting, which re- weights samples by their influences to create a proper decision boundary. With these two improvements, we present the Effective Decision Boundary Learning algorithm (EDBL) which improves the performance of KD and deals with the imbalanced data learning simultaneously. Experiments show that the proposed EDBL achieves sate-of-the-asd performances on several CIL benchmarks.
1 Introduction
Applications of deep neural networks (DNNs) in real world require the ability of the system to learn new classes incrementally 3 However, DNNs typically suffer from drastic performance degradation on the previously learned tasks after learning new classes when the past data are unavailable, which well documented as catastrophic forgetting 4 5 Recently, many methods in CIL have been proposed to try to deal with this problem. Owing to their superior performances, Rehearsal-based approaches utilizing KD 7 (RKD) 8 10 M have been widely applied in CIL. However, recent works revealed that the RKD methods suffer from decision boundary overfitting to new classes, which is referred to as the (task-recency) bias towards new classes
As shown in Fig. I basic RKD approaches typically have two training objectives: transferring the existing knowledge from the old model by KD and learning new classes via minimizing the KD loss [ and the cross entropy (CE) classification loss. However, RKD approaches usually suffer from the decision boundary overfitting problem caused by insufficient KD and imbalanced data between the old and new classes.
Problem of Insufficient KD. In CIL setting, the stored exemplars of old classes limited because of the small memory budget. it well known that the high capacity of DNNs sufficient to memorize the entire training data [14 15), so RKD methods suffer from insufficient data for KD training. RKD methods commonly use the stored exemplars of old classes and data of new classes to compute the KD
"Uee footnote for providing further information about author (webpage, alternative address) not for acknowledging funding agencies.
Preprint. Under review.

---

Nih Coors
modil
700 mode
Sence
Insffidenn
Inobolorcedd dota lectning
Re4/K0
13 Iteelood
basic RDD
EULL
Figure 1: Comparisons between the basic RKD and the proposed methods (EDBL). Illustration of the two factors in RKD causing decision boundary overlitting classes.
loss to preserve the existing knowledge. However, data of new classes are out of distribution (OOD) with the original training dataset of the old model and Beyer al. [T6] demonstrated that OOD data work in KD to some extent while their performance suffers from great degradation, compared with using the original training data. The combination of the scantness of exemplars for the old classes and OOD between the learned classes and the new classes (data of old classes and new classes are typically OOD in the CIL setting) lead to the new model overfitting to new classes. This phenomenon is also demonstrated in our experiments (c.f. Sec. 5.5.2) and we refer to it as insufficient KD.
Problem of Imbalanced Learning. Due the limited memory budget for storing exemplars of the old classes in RKD approaches, there exists a serious problem of imbalanced data for learning (few samples for the leamed/old tasks are available while we have large number of samples for the current/new task), which would lead to decision boundary overfitting to the dominant (new) classes 11 18 19].
In order to tackle the insufficient KD, we employ MKD and re-sampling strategy. We over-sample the exemplars of the old classes and mix the samples from the old classes and the new classes to synthesize mixed data for KD training. The interpolated data by an old class and a new class are more consistent with the distribution of the original training data of old classes than the data of new classes, which can improve KD in CIL and form a smoother decision boundary (c.f. Fig. Re-MKD).
In order to deal with the imbalanced data learning problem, we attempt to attenuate the influence of samples that cause the overfitting. To this end, we extend the method of influence balance (IB) [9] to CIL setting and propose the incremental inffuence-balanced (IIB) method for CIL. Specifically, we first derive a metric that measures how much each sample influences the biased decision boundaries. Then, decompose the metric into two parts which are referred to the classification weighting factor and the KD weighting factor, respectively. Finally, we design the incremental influence- balanced (IIB) loss function for CIL, which adaptively assigns different weights to samples according to their influence on decision boundaries (c.f. Fig. I IIB Method).
Based on the previous proposal, we divide our method into two phases after re- re-sampling Mixup-based data augmentation. Phase 1 MKD training, which improves the efficacy of KD to form better decision boundary through generating sufficient and more related data with the old classes used in KD training. Phase balances training by utilizing the derived IIB loss, attenuating the influence of the samples that cause overfitting of the decision boundary. Through the two training phases, we learn a decision boundary with better gencralization capability.
In summary, the main contributions of this work are as follows: (i) We combine MKD with re- sampling strategy to relieve the insufficient KD problem in RKD methods through synthesizing data that are more related with the original training data of old classes than data of new classes. (ii) We extend the IB method into the CIL setting and propose novel IIB loss function tackle the imbalanced learning effectively in CIL. (iii) We propose the EDBL algorithm via combining Re-MKD and IIB by deriving the incremental influence factor for mixed data. Experiments demonstrate that the proposed EDBL method achieves state-of-the-al performances on several CIL benchmarks.
2 Related Work
2.1 Bias-Correction Approaches
CIL belongs to continual learning and the main challenge catastrophic forgetting 4 Previous approaches to tackle catastrophic forgetting can be divided into three categories [3] iregularizatio 220]

---

21, parameter isolation 12 23 and replay approaches [24 25].. RKD belongs to replay approaches
and achieves the sfate-of-the- art performances. RKD methods [2
9 8 I0 26 commonly train the
new model by preserving the existing knowledge and learning classes incrementally via minimizing the KD loss and the CE loss. We introduce the problem formulation and summarize the popular strategies of RKD training in Appendix A.
RKD methods suffer from the task-recency bias and the ideas in previous works alleviate this problem in CIL are very similar to the approaches in the long-tail learning. For example, iCaRL [2] employs representation learning strategy, and EEIL [9] utilizes classical image transformation e-g- random cropping, mirroring etc. to make data augmentation and fine-ture the newly trained model with balanced dataset. BiC [8] trains bias-correction classification layer with a balanced dataset, which is similar to the approaches of decouple learning and LUCIR [TO] belongs to cost-sensitive learning. Different from the previous works, our method employs re-sampling and MKD strategy to relieve insufficient KD problem and propose IIB method to deal with the imbalance data learning. Both improvements are complementary to the previous bias-correction approaches.
2.2 Classes Imbalance
Classes Imbalance is a significant challenge for machine learning 27 28
In long-tail learning, the
trained models usually bias towards the dominant classes. Previous efforts tackle the overfitting problem can be roughly divided into three groups: (i) data re-balance including re-sampling [29, 30
data augmentation 32) etc.; (ii) cost-sensitive learning including re-weighting strategy [3 19
etc.; (iii) module improvement including representation learning [34 decoupled training 36
etc.. In this work, we combine mixup with re-sampling to make data augmentation and utilize the re-weighting strategy to relieve overfitting. But, we mainly use mixed data in KD training to relieve insufficient KD problem in CIL and we extend the re- weighting method in [19 (referred to as IB method in this work) into CIL and combine it with the re-sampling&mixup based data augmentation to tackle imbalanced data learning.
and ensemble learning 37 38
2.3 Mixup and Knowledge Distillation
Mixup is first proposed in 39 which generates an interpolated sample (ë, () by (I1, vi), (Z), Vi)
according to Eq.
Az; (1 ))2,, = Ay. + (1 Alay.
(I)
where We, Ty are images and ys, B/j are labels, respectively, À is randomly drawn by the Beta function. In 39), the CE loss (Ler) for mixed data is computed as follows:
(2e 9 Xhee (a. vn) (1- (f,
After that, some other label mixing methods are proposed such as cutmix
40 manifold mixup 41
Remix [3I], etc.. Mixup-based data augmentation can greatly improve the generalization of DNNs in image classification learning while there are some works utilize label mixing methods in various scenarios such as the long-tail learning (3I, continual learning 42 43]. Recently, some works validate that label mixing methods improve the performance of KD e.g.
16 44
which compute the
KD loss with the mixed sample (ă. 9) as follows:
Lea(#, -o.( T(2)
Noglo: 8(a))
(3)
where T, S' are the teacher model and the student model, t is the temperature, m is the number of the learned classes of T and a is the softmar. These works utilize Mixup to improve KD performances in model compressing setting where the training data are with the original training dataset of the teacher model. However, they cannot verify the effectiveness of their methods in the CIL setting, where the training data are imbalanced and new classes are OOD with the old classes. In this work, we employ re-sampling in Mixup and validate effectivvnesss of mixup between old classes (IID dataset) and new classes (OOD dataset) for KD in the CII. setting.
(IID)
3 Preliminaries
robust statistics was proposed to find the influential instance of sample to model. Recent works have used influence function in DNNs,
Influence Function. The influence function 45 46
3

---

e.g.. 47
Given empirical risk R(0) = N E L(x, O), of which the optimal parameter is O- argming R()), where N is the number of training data, θ (0, W) are the parameters of the network and 0, W are the parameters of the feature extractor and the linear classifier (the last full connected layer, FC), respectively. According to the results in 47.. the influence function of the point (2, v) is given by:
1(x, 0) - -H (VoL(z, 9)),
(4)
where Ve is the gradient operator and H A E Vg θ) the Hessian, which is positive definite based on the assumption that L is strictly convex in a local convex basin in the vicinity of O
Influence Balance Method. Park et al. [9] firstly apply influence function to a learning scheme, where they design the influence-balanced (IB) loss by utilizing the influence function during training in long-tailes classification. They treat the influential instance of sample to model of point (æ, y) as the parameters of the model change when the distribution of the training data at the point (a, y) is slightly modified and referred to as the IB weighting factor. IB method uses L1 norm of Eq. to qualify the influence of the point (z, y) and further ignores the inverse Hessian reasonably. The IB weighting factor is computed as follows:
IB 0) li L2, B)lla
(5)
IB method focuses on the change in the FC layer of DNNs and the IB weighting factor can be simplified by:
IB(r, 8) = [1f(()) vll [|A]|a
(6)
where f is the network and is the hidden feature vector of (z, v). Finally, the IB loss, which is used in balancing training to attenuate the influence of samples that cause an overfitted decision boundary is given by:
L(x, y. 9)
L v. 6) Au Tz.( O)
(7)
where in IB method, L is CE loss function, À = >m E/ ni is the class-wise re-weighting term, k is the label of (x, nı is the number of samples in the k-th class and 7 is the hyper-parameter. IB method has two training phases:(i) Normal classification training via minimizing the CE loss; (ii)Balaciing training, where IB method fine-tunes the model via minimizing LIB by Eq. 7
4
The Proposed Method
4.1 Re-sampling MKD
Beyer et al. 16 demonstrates that KD training with OOD data suffers from great degradation and they validate empirically that the data, which are related overlapped with the original training data (consistent with the latent distribution of original training data) can perform as good as the original training data in KD training. So the mixed data generated by mixup using samples from old classes and new classes, which are usually more related with old classes than the data of new classes (basic RKD methods directly use data of new classes in KD training) may improve KD performances in CIL.
Re-sampling Mixup.
Given that the data between the old and new classes are seriously imbalanced in RKD methods, consider the ratio between the old classes from the past tasks and the new classes of the current task. We generate three kinds of mixed data in batch: mixup among old classes, mixup between old classes and new classes and mixup among new classes. Owing to the limited data per old class in RKD methods, mixup data more frequently for the first two types. Specially, we mixup 7/2 times for the first two types and time mixup for the last type of mixed data, where N' is the proportion of the data between per tail and head class, and we make sure the number of samples from old classes in a batch isn't 't less than fixed number (32 in this work).
N
Finally, ce-samlling-basee Mixup can generate much data through label mixing between old classes and new classes. These generated data increases the diversity and the number of data used in KD, which improves the performance of KD in CIL. Our experiments demonstrate that re- re-sampling-bue based Mixup outperforms the original Mixup in 39 (referred to as Vanilla- Vanilla-Mixup in this work) significantly (c.f. Sec. 1.5.1).
MKD Training. After data augmentation by re-sampling-Mixun we train the new model through minimizing two losses; the CE loss (Lee) ) and the KD loss (Lea) with the synthesized data in the

---

similar way as in the basic RKD method. We compute Lee and Lid according to Eq. 2 and B respectively and we replace the teacher and student model with the old model (/ and the new model ( in Eq. 3 respectively. In this paper, we denote the strategy of Re-sampling Mixup and MKD training (Re-sampling MKD) as Re-MKD while we denote the strategy of vanilla Mixup and MKD training as vanilla-MKID.
4.2 IIB Method
4.2.1 IIB Weighting Factor
We extend the IB method into CIL and propose a novel IIB loss function to attenuate the influ- ence of the samples to relieve the overfitting. The experience risk at task in RKD is R(O)=
N Ei=1 L(x,0) Q), where L a Lee fe + Laa(ff (z;). (z,). (xx)) and O- argming is the optimal parameter after initial training. We first utilize Eq. 4 to compute the influence of the sample (x. y) for CIL. Then, like in IB method, we ignore the inverse of the Hessian, which given
by Hr. A > multiplied by all the training samples and just the relative influences of the training samples are needed and we use Li norm to qualify the influence to obtain the incremental influence-balanced (IIB) weighting factor. Therefore, according to Eq. in the IB method, the IIB weighting factor of the point (z, y) for CIL can be represented by:
21=1 VgL(Z, θ) - Σi Vo 6) + ViLaa θ) because it is commonly
IIB(e, 8) V e) (1 9) + VaLaa(a, O)
(8)
At task t, let h = [h.,..... h be hidden feature vector (an input to the FC layer), and f(r, θ) [f1,...;fm .a....,,mm--. fm+n] 1 be the output denoted by fx - ( h). where a is the softmax function, m, are the number of learned classes and the new classes, respectively. The weight matrix of giv is denoted by W [wi,.... Wm...... Wm+nl R(m-n)xf, Then, VeLoo(E, θ) and Ve Lna(E, +)
computed as below, respectively.
Blee 0)
(f) velhi, mn]
(9)
Owni
8 9Lea (at, 0)
(ffla) - (z))). & m1
(10)
Owas
0.
e [m + 1, 1. n]
Finally, we apply Eq. and 10 to Eq.
Thus, the IIB weighting factor can be computed as:
TIB(a, e) = [1]22 fa(x)
(x)] + III (x) vlm [b]
(11)
where [V denotes the slice [u:..... v 2 i) of the vector V = [p1.... uN)
4.2.2 Decomposition of IIB Weighting Factor
The Stability-Plastice Dilemma. The stability-plaasiiity dilemma [48] is well-known constraint for artificial and biological neural systems. The basic idea is that a learning system requires plasticity for the integration of new knowledge, but also stability in order prevent the forgetting of previous knowledge. Too much plasticity will result in previously encoded data being constantly forgotten, whereas too much stability will impede the efficient coding of this data at the level of the synapses.
In RKD, the KD loss is used to preserve the existing knowledge by encouraging the new model to mimic the output of the old model and the CE loss is used to learn to recognize new classes. Eq. II considers the influence of the CE loss and the KD loss on the decision boundary, but sample may actually perform differently on the classification and KD training. For example, we use data of motorbike to train the new model to recognize motorbike incrementally while preserving the knowledge of knowing bike. Because of the similarity between motorbike and bike, samples of motorbike may improve the performance of KD and preserve the previous knowledge well. That is because the data which are related or overlapped with the original training data perform well in KD 16]. But in the classification training, the new model may bias to motorbike.
dilemma, we decompose the IIB weighting factor into two factors: the classification weighting factor for learning new tasks and the KD weighting factor for preserving the existing knowledge, which measure how much each sample influences on forming new boundaries by the classification training and the previous decision boundaries of the old model preserving training (KD training), respectively.
Decompasing IIB Factor. Considering the
5

---

Definition 4.1. In CIL, the classification weighting factor and the KD weighting factor of a sample (a, y) are defined as the magnitude of the gradient vector of the CE loss and the KD loss on that point, respectively:
(z..(a, 6) = (2 e)ll @) = VVVLaa( O)1l
(12)
Then, a haper-parameter to make trade off on these two factors. Therefore, IIB weighting factor is computed via:
IIB(a, @) IB..(.. θ) + aIBea 0) (x) - wills + - (x))U))),
(13)
where IIB(,, θ) < Ve (z, 1] + VeLua( 0) and a s 1.
4.2.3 IIB Loss
During the balancing training, attempt to fine-tune the decision boundary create more generalized one, so we multiply the CE loss function by the inverse of IIB weighting factor, resulting IIB loss as follows:
(1 0) A (z. y. 2) [IB(x, @)
(14)
where Dg is the class-wise re-weighting term (c.f. Sec. 3 and Eq.
4.3 EDBL Algorithm
We propose the EDBL algorithm combine Re-MKD and IIB method improve knowledge transferring and deal with imbalanced data classification synthetically to generate more generalized
decision boundaries CIL. After MKD training, considering an mixed sample (é, D), we apply Eq to Eq. 13 to compute the IIB weighting factor as follows:
ZIB @) = (ll(s"(a) (á) Ay; (1 - A)æslll
- -" (a)lb))ii
(15)
Then,
this IIB weighting factor to compute LIIB according Eq.
Inspired by [16], which validates empirically that a large number of epochs improves the performance of KD, we add the KD loss to the IIB loss. Then the overall loss Loverall for balancing training is as follows:
(overait
LII 9. 0) Lea(d, 0)
Overall, EDBL has two training phases: MKD training and balancing training. During the two training phases, Led is continually used for distillation encourage the decision boundary in the new model to mimic the one in the old model, and LIIB re-weights the interpolated samples by their influences on decision boundary in the balancing training phase. The pseudo-code of EDBL is presented in Algorithm in Appendix B.
Experiments
5.1
Datasets
We conduct experiments on CIFAR-10, CIFAR-100 49 and Tiny-Imagene (50] validate our approach. We pad four pixels for images in CIFAR- 10 and CIFAR- and then random crop them into 32 32 pixels. We use horizontal flip in all image pre-processings 12
5.2 Baselines and Protocols
We compare EDBL with some SOTA methods in CIL such as Mnemonics [51], PODNet-CNN [52
and SS-IL (26), etc.. We adopt two protocols to conduct experiments: (1) Base-half: we start from a model trained on half of all classes in the dataset, and the remaining half of classes come
and 10 phases and stores 20 samples for each old class [T]]. (2) Base-O: we follow the protocol in iCaRL 12] to conduct experiments, where all of classes come in 2, 5, or 10 phases and the memory budget is fixed value. We report the results of two inference strategies: CNN output and the cearest-mean-ot of ofannnof--eemmee of-exmr strategy (NME) [2] to evaluate our methods (denoted as EDBL-CNN and EDBL-NME, respectively).
We follow the exemplar management of iCaRI. to select exemplars for each old class. We use two common metrics: average accuracy and average incremental accuracy [2] to measure performances.

---

CIFAR-105 phssee))
CIFAR-10 100 1O0 phasse]
Tiry-Imagenet 55 phanes)
amy-inggenttteeo phaset)
Mumber taste
Numbarr
(a)
(b)
(c)
(d)
Figure 2: Comparison results of average accuracy on Base-half experiments with 5, 10 phases on CIFAR-100, Imag-em
5.3
Implementation Details and Hyper-parameters Tuning
The in Eq. D is randomly sampled from the Beta function B = (1 for all the experiments.
To make comparison fairly, we use the same networks for our methods and baselines. We follow [12 88 [0] and [43] to use ResNet32 (d=64) for experiments on CIFAR- 10/100 and R (d=512) for Imy-Imaaenet in this work. When training networks, we follow the standard prac- tices for fine-tuning existing networks. We use stochastic gradient descent (SGD). As for the training typer-parameter such as batch size, weight decay, momentum, learning rate, epochs etc., we set these parameters semi-heuristically and tune a few of parameters on CIFAR-100 and Ima--emaeenet experiments with 5 incremental learning phases, then use the same set- ting of these parameters for other experiments on CIFAR-10/100 and Tiny-Imagenet, respectively. We mainly tune As in Eq. [4 and a in Eq. 15 by grid searching. All the hyper-parameters are given in Appendix C. The code will be available at https///ithub.oom/xxx/IIII
Table I: Average incremental accuracy (%) on Base-half experiments. Models with an asterisk denotes using the results in [T]. The two mod- els with or I are reported directly from [53) and [52] respectively. Because SS-IL 26 with
5.4 Results
5.4.1 Results Base-half
Table presents the comparisons of our methods with the baselines on Base-half experiments. From Ta-
NME performs much better than original SS IL
in this work, we alternatively select SS- IL with NME (SSIL-NME) as baseline.
ble [ we can find that EDBL-NME outperforms all the compared methods on all the experiments signifi- cantly. EDBL-CNN also surpasses all the compared methods on all the experiments except the experiment on CIFAR-100 with 10 phases. The comparison re- sults of average accuracy at each incremental learning phase are given in Fig. Fig. 2 demonstrates that our methods (EDBL-CNN and EDBL-NME) surpass all the baselines at nearly each incremental learning phase.
Base half
CIFAR 100
Thy imagenet
Phases
10
Lwi [5]]
49.59
46.98
ICaRL*
57.12
52.66
BIC*
59.36
54.2
LUCIR*
[10
63.17
PODNet-CNNI
64.83
63.19
Mnemonics*
51
63,34
62.28
TPCILt
65,34
63.58
/
iCaRI. 12
59.67
56.13
48.98
39.27
BiC [
61.14
58.4
49.23
47.67
LUCIR [0
49.31
47.56
SSIL-NME 26
64.94
60.99
48.93
45.74
5.4.2 Results on Base-O
EDBL-CNNiOurs)
66.57
62.06
52.43
53,80
EDBL-NME(ous)
66.65
64.73
50.99
49.97
We further conducted experiments on CIFAR-10/100
following Base-O protocol to evaluate our method. The memory budgets on CIFAR-10 10 and CIFAR-100 are fixed to 200 and 2000 [22, respectively. CIFAR-10 is
Table 2: Average incremental accuracy (%) on Base-O experiments.
split into 2 and 5 phases while CIFAR- 100 is split into 2, and 10 phases. Our methods employ Mixup strategy which may help to relieve the overfitting problem while Remix [11 adapted Mixup to generate more effective interpolated data to tackle the long-tail learning. To compare our method with Remix, we directly employ Remix to train the new model to learn new classes incrementally with the optimal hyper-parameters given in and Remix-NME From Table 2 we can find that EDBL-CNN and EDBL-NME outperform all
Base-O
CIFAR-10
CIFAR-100
iCaRL [2]
89.7
77.13
68.35
67.24
63.98
BiC
X
89.29
76.5
70.15
68.06
65.9
PODNa--C
RemiX- CNN
78.52
70.36
64.98
64.37
60.85
RemiX-NNM NME
17
62.07
77.41
67.19
67.18
64.46
SSIL-NME 26]
89.97
77.9
68.7
65.83
56.11
EDBL-CNNNuurs)
72.28
EDBL-NME/Ous)
89.82
78.51
70.71
69.33
67.99
3I.
We report the results of Remix-CNN
7

---

[ranel,
proset class)
prasez, clase
glasez,
[Twwe) clasc?
(a) Basic RKD in [9
(b) EDBL
Figure 3: t-SNE visualization on random subset (4 classes) from CIFAR-10 (2 inermental phases).
Table 3: The effectiveness of Re-MKD&IIB of EDBL (Average Accuracy on each incremental phase).
Dataset
CIFAR-10
CIFAR-100
phase
2
3
3
4
5
Baseline
0.9894
0.775
0.3344
0.464
0.4427
0.848
0.7085
0.3606
0.4891
0.392
VunillaMMDD
0.9905
0.8095
0.587
0,4894
0.4189
0.8355
0.718
0.5658
0.4836
0.3625
IIB
0.9894
0.8527
0.6678
0.5851
0.5068
0.4476
Rc-MKD
0.9905
0.8387
0.6446
0.6066
0.5926
0.8624
0.761
0,5744
0.5882
0.5629
1IB-KI)
0.989
0.819
0.7143
0.6782
0.6329
0.8624
0.7355
0.636
0.5788
0.5362
EDBL-CNN(ur))
0,992
0.8727
0.7245
0,6794
0.6618
0.85
0.767
0.7093
0.6573
0.6051
baselines on nearly all experiments except the experiment with phases and 2 phases on CIFAR- 10, respectively. Both EDBL-CNN and EDBL-NME outperform Remix-CNN and Remix-NME on all the experiments by large margins about [1.1%, 7%]. We also give the results of average accuracy at each incremental learning phase and the results show that our methods surpass all the baselines at nearly each incremental learning phase, similarly (c.f. Appendix D).
5.5
Ablation Study
5.5.1
Analysis Re-MKD and IIB
The base work in 9 is a vanilla basic RKD which trains the new model via minimizing the KD loss and the CE loss without any other strategies. We use it as the baseline in ablation study. The memory size in the ablation study is fixed 200 and 2000 for CIFAR-10/100, respectively and the experiments are conducted in the same way as in the Base-O experiments. Table[]demonstratee the effectiveness of Re-MKD and IIB components in the proposed EDBL algorithm, where IIB-KD training strategy that uses the IIB loss and the KD loss to fine-tune the new model in the balancing training phase.
In summary, we can observe that: (1) The Re-MKD component can improve the performance of the baseline each incremental batch by large margins, about [1%, 16%] and Rc-MKD outperforms Monilla-MKI
MKD) significantly. This validates that the generated data by ---samlling-bee based Mixup are more related with old classes improve the performance of KD. This also demonstrates that the Re-MKD would relieve the problem of insufficient KD; (2) The IIB component can improve the performance of the baseline at each incremental batch remarkably, and surpasses baseline by up to 5.5%, 7.5%] on CIFAR- 10 and CIFAR- 100 at the last phase. Besides, IIB-KD further improves the performance based on the improvement by IIB, which exceeds baseline by 19% and 13% CIFAR- 10 and CIFAR- 10F, respectively at the last phase. All of these reflect the
Sailll-Mixup--aase

---

effectiveness of IIB loss for CIL: (3) EDBL achieves the best performances, which implies EDBL can learn more generalized decision boundaries effectively.
Further, we conduct visualization experiments with 2 incremental learning phases on a ran- domly selected subset with 4 classes from CIFAR-10 and the memory is fixed to 200. As shown in Fig. 3 the features extracted by the new model trained by EDBL has the less mu- tual intrusion between the learned classes (the first batch classes) and the new classes (the second batch classes), which reflects indirectly that the decision boundaries learned by EDBL are with more generalizability than the ones generated by the Basic RKD method.
Table 4: Sensitivity on memory (Average Ac-
5.5.2 Study on Memory Budget
curacy on the last phase of the experiments), *
denotes using the results in 42]
We further study the sensitivity of EDBL on the size of memory budget by conducting experiments on CIFAR- 100 with 5 phases following Base-O protocol. The av- erage accuracy on the last phase of the experiments demonstrated in Table 4 and the table shows that EDBL-CNN obtains the best performances. CLDR [42) utilizes manifold mixup make regularization to alleviate catastrophic forgetting while our method outperforms CLDR by large margins about [8%,,15 J.
Memory 0.2822
200
500
1000
2000
GEM
0.32144
013546
ER-Res.
150
0,2869
0.3190
0.3406
/
CLDR*
42
0.3434
0.3770
0.4020
0,4189
0.5133
0.5431
SSIL-NME 26
0.4984
0.5217
0.5371
(
0,4268
0.5265
0,5874
0.6051
5.5.3 Sensitivity a
In IIB method, we introduce the hyper- parameter a to trade off the classification weighting factor and the KD weighting factor compute IIB loss. Here, we choose the strategy, which is RKD IIB- KD in table 3 and a = (1, le 3, le 5, 0} to conduct experiments with 5 in- cremental phases on CIFAR- 10/100 and the memory is fixed to 200, 2000, respectively.
Table Sensitivity study a (Average Accuracy the last phases of the experiments).
Datnset
CIFAR-10
CIFAR- 100
le:3
le:5
le:3
le:5
CNN
0.6034
0.596
0.6309
06176
0,4955
0.4977
0.5018
NME
0.6485
0.6599
0.6577
0.6424
0.4836
0.4883
0.5016
0.5015
From Table we can observe that IIB method is sensitive to the setting of a and setting a small value for a improves the performance significantly while a is not the optimal value, where a denotes that IIB degenerates into the IB method. This implies that stability-plasticity dilemma really exists in CIL and we should carefully treat this issue to achieve a better performance.
6 Conclusion
In this paper, we analyzed the causes of the decision boundary overfitting problem in CIL by two factors: insufficient KD and imbalanced data learning. We proposed the EDBL algorithm to address these problems. First, in order to deal with insufficient data for KD, we presented the re-sampling MKD strategy, which generates much data by Mixup and re-sampling to be used in KD training, which are more consistent with the latent distribution of old classes. Second, we extended IB method into CIL through deriving a novel IIB loss, which re- weeightt samples by their influence on decision boundary to alleviate the problem of learning with imbalanced data. On top of this, we propose EDBL algorithm to combine re-sampling MKD and IIB method which can improve the knowledge transferring and deal with the imbalanced data classification simultaneously. Experiments show that EDBL achieves slate-of-the-ar performances on several benchmarks and ablation study validates that both of Rc-MKD and IIB are effective in CIL.
Limitation and Broader Impacts. One limitation is that more GPU memory and training time are needed (icause:()) re- re-sampling Mixup generates much mixed data during training. (ii)EDBL has two training phases, so EDBL nearly requires twice as much training time basic RKD method. Fortunately, both of re-sampling MKD or IIB are effective in CIL, so we can employ them in real applications flexibly. Moreover, EDBL significantly relieves catastrophic forgetting, which promotes the practical use of DNNs. The negative impacts may occur in some malicious or misuse scenarios. The appropriate proposes of employing EDBL (re-sampling MKD/IIB) are supposed to be ensured with attentions.

---

References
[1] Gregory Ditzler, Manuel Roveri, Cesare Alippi, and Robi Polikar. Learning in nonstationary environments: A survey, IEEE Computational Intelligence Magazine, 10(4):12-25, 2015.
[2] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
[3] Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[4] Yoshua Bengio, Mehdi Mirza, Ian Goodfellow, Aaron Courville, and Xia Da. An empirical investigation of catastrophic forgeting in gradient-based neural networks. 2013.
[5] Robert M French. Interactive tandem networks and the sequential learning problem. In submitted to the 1995 Cognitive Science Society Conference. Citeseer, 1995.
[6] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[7] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1503.020 2015.
[8] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 374-382, 2019.
[9] Francisco M Castro, Manuel J
Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pages 233-248 2018.
[10] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang. and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pages 831-839, 2019.
[11] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficient incremental learning through feature adaptation. European Conference on Computer Vision, pages 699-715. Springer, 2020.
[12] Sylvestre-Alvis Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings the IEEE conference Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[13] Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-incremental learning: survey and performance evaluation on image classification. arXiv preprint arXiv:2010.1527: 2020.
[14] Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, al. A closer look at memorization in deep networks. In International Conference on Machine Learning, pages 233 233-242. PMLR, 2017.
[15] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
[16] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 10925-10934, 2022.
10

---

[17] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314. 2020.
[18) Zhiliang Peng, Wei Huang, Zonghao Guo, Xiaosong Zhang, Jianbin Jiao, and Qixiang Ye. Long-tailed distribution adaptation. In Proceedings of the 29th ACM International Conference on Mullimedia, pages 3275-3282, 2021.
[19] Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. Influence-balanced loss for imbalanced visual classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 735-744. 2021.
[20] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwingg et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13)3521-3352 2017.
[21] Sang-Woo Lee, Jin-Hwa Kim, Jachyun Jung-Juggoooo Ha, and Byoung-Taie Zhang. Overcoming catastrophic forgetting by incremental moment matching. arXiv preprint 1rrxvv1703.003 2017.
[22] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734 2017.
[23] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765-7773, 2018.
[24] David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceed- ings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[25] Arslan Chaudhry, Naeemullah Khan, Puneet Dokania, and Philip HS Torr. Continual learning in low-rank orthogonal subspaces. arXiv preprini arXiv:2010. 1635, 2020.
[26] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. Ss-il: Separated softmax for incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 844-853, 2021.
[27] Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21) 121--128424 2009.
[28] Nathalie Japkowicz and Shaju Stephen. The class imbalance problem: A systematic study. Intelligent data analysis, 6(5):429 449, 2002.
[29] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321 357, 2002.
[30] Xu- BinY Liu, Jianxin Wu, and Zhi-Hua Zhou. Exploratory undersampling for imnalan learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 39(2):539- 550, 2008
[31] Hsin-Ping Chou, Shih-Chich Chang, Jia- Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: rebalanced mixup. In European Conference on Computer Vision, pages 95-110. Springer, 2020.
[32] Yuhang Zang, Chen Huang, and Chen Change Loy. Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation. In Proceedings the IEEF/CVF International Conference on Computer Vision, pages 3457-3466, 2021.
[33] Yin Cui, Menglin Jia, Tsung Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 9268-9277 2019.

---

[34] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-taibe recognition an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2537-2546, 2019.
[35] Xiao Zhang, Zhiyuan Fang, Yandong Wen, Zhifeng Li, and Yu Qiao. Range loss for deep face recognition with long-taie onngtailed training data. In Proceedings of the IEEE International Conference on Computer Vision, pages 5409-5418, 2017.
[36] Alakh Desai, Tz-Ying Wu, Subarna Tripathi, and Nuno Vasconcelos. Learning of visual relations: The devil is in the tails. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15404-15413, 2021.
[37] Hao Guo and Song Wang. Long-tailed tailed multi-labed labell visual recognition by collaborative training on uniform and re- re-balanced samplings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 15089- 15098, 2021.
[38] Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving long-tailed recognition in one-shot. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 112-121, 2021.
[39] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 171009442, 2017.
[40] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023-6032. 2019.
[41] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Joannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages
PMLR, 2019.
[42] Xuejun Han and Yuhong Guo. Continual learning with dual regularizations. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 619-634. Springer, 2021.
[43] Fei Zhu, Zhen Cheng, Xu-yao Zhang, and Cheng-lin Liu. Class-incremental learning via dual augmentation. Advances in Neural Information Processing Systems, 34, 2021.
[44] Dongdong Wang, Yandong Li, Liqiang Wang, and Boqing Gong. Neural networks are more productive teachers than human raters: Active mixup for data-efficient knowledge distillation from a blackbox model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1498-1507, 2020.
[45]
Anthony Robinson. Residuals and influence in regression, 1984.
[46] Frank R Hampel, Elvezio M Ronchetti, Peter J Rousseeuw, and Werner A Stahel. Robust statistics: the approach based on influence functions, volume 196. John Wiley & Sons, 2011.
[47] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pages 1894-1894. PMLR, 2017.
[48] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The vestigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 4:504, 2013.
dilemma: In-
[49] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[50] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[51] Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi- class incremental learning without forgetting. Proceedings of the IEFF/CVF conference on Computer Vision and Pattern Recognition, pages 12245-12254, 2020.
12

---

[52] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Ecion-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 80-102. Springer, 2020.
[53] Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong, Xing Wei, and Yihong Gong. Topology- preserving class-incrent incremenallt learning. In European Conference on Computer Vision, pages 254-270. Springer, 2020.
[54] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12)::933-2999 2017.
[55] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[56] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and M Ranzato. Continual learning with tiny episodic memories. 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13

---

Appendix
A Problem Formulation and RKD Methods
A.1 Problem Formulation of CIL.
In the CII. setting, a dataset D = {( y)lar E X E y) is split to T subsets: DD = D'UDDRU...UD where X' of images with labels y and the subsets have no overlapped classes, then learning system is trained to learn each subset incrementally. At task t. we have model fe.w which has incrementally learned the old classes Čt {
where 0, W denote the parameters of the feature extractor and the linear layer of the network, respectively and C denotes the classes in subset (task i). Now, given the new subsets DD with the new classes C', the goal is to train new model fē Jew that can perform classification on all the classes C¹ = Čt- U In rehearsal- baasd (RKD) methods, they store a small number of image exemplars of the old classes after the completion of each incremental learning task for experience replay at the future tasks. We denote E* as the selected exemplars of the current task (the new classes) to be stored after the completion of task L. We denote H = E--1 U Et-1, D' = D' U E', g {ar y) E D'), }. yr = (x, y) E DD' as all the stored exemplars of the old classes, all the observable dataset, all the available images and labels task respectively.
A.2 Training Strategy of RKD Method.
g 26 of RKD methods have the common process that uses all the available data to train the new model by minimizing two losses: the classification cross-entropy (CE) loss and the knowledge distilation (KD) loss. The CE loss is used to learn new classes and The KD loss is used to encourage the new network fú, w to mimic the output of the previous task model ft-1 The CE loss (LCg) and the KD loss (Lxd) are typically computed as follows:
Most previous works 12
min
LCE = Σ
(17)
(x
1=1
m
Lid
Lf (4))oolo)) (x))].
(18)
where (z()) is the label indicator function, m, n are the number of learned and new classes respectively and a is either the softmar or sigmoid function. So the new model fe, fe.w are trained by the overall loss:
Lid + ALCE
(19)
where X is the hyper parameter. Note that few continually updated task t, whereas the network fo. 1 is frozen and will not be stored after the completion of task t.
However, the dataset of the new classes (D') in the new task are out-of distribution (OOD) with the original training data (Dt-1) of the old model fow so the performances of KD suffer from huge degradation. Moreover, RKD methods suffer from the task-recency bias [3] After training the new model, to tackle the task task-recency bias, various RKD methods have different subsequent processing. For example, iCaRL [2] takes the seweeestmman-o---eeeeee strategy make inference, BiC g trains bias-correction layer with a balanced dataset and EEIL [9] further fine-tunes the whole model by using the balanced dataset of stored exemplars.
(NME) classification
B EDBL Algorithm
The training process of our method (EDBL) shown in Algorithm I task, we first make data augmentation by re-sampling Mixup, then we train the new model with the mixed data in the same way as in the basic RKD method.At last, we fine-tunes the whole model by Eq. 16 in the balancing training phase.
At each incremental learning
14

---

Algorithm EDBL Algorithm for CIL
Input: Exemplars (F², 1) and data of new classes (D"), ft-1
Output: Exemplars E+ Es U E², The New Model f
Mixup with Re-sampling:
Employ Mixup and Re-sample old classes to generate interpolated dataset D. Phase 1: MKD Training
for i 1 to Ti do
sample mini-batch D.,. from D
Los E-Eq. 2, Lid f- Eq. 3,
(( = -, 2 (D.)) Loe (., () mi Lna(#, O)
Update @ = O' 71 VL())
end for
Phase 2: Balancing Training
for i = to T2 do
sample mini-batch D.,. from D IIB i, Q) +- Eq. 15
Loverall (0) in E Di.j)e
A I7B(2.).))
(4.(0.)))) + (. Lees Lıd is computed by Eq.2
and 3
Update @ = O' 72 Loverall(())
end for
Exemplar Management: Utilize the strategy in 12 to make exemplar management to select
E (maybe also remove some samples from E*) to generate E++1
C Implement Detail
C.1 Typical Training Hyper-parameters Selection
We draw À in Eq. randomly from the Beta function B = (1, 1) for all the experiments. re-ampling Mixup, we heuristically make sure the number of samples from old classes in batch isn't 't less than 32, We semi- hemi-heurittically batch-size, etc. and tune a few of parameters on CIFAR- 100 and Tiny-Imagenet experiments with incremental learning phases, then we use the same setting these parameters for other experiments on CIFAR- 10/100 and Imay-ema respectively. All experiments use the same batch size, weight decay, momentum: 128, 0.0002, 0.9, respectively. Other training details of the two stages are as below:
the typical training hyper-parametent e-g. epoch, learning rate,
MKD Training. In Mixup-based Knowledge distillation (MKD) We train the new model with different hyper parameters on different datasets. For CIFAR-10/100 we train the network for 150 epochs each task. The learning rate epochs. As for Imng-ema the number of training epochs is 250 at each task. The learning rate is set to 0. and reduced by a factor of 10 epochs 75, 125, 175 and 225.
0.1, and reduced by factor of 10 60, 100, 130
Balancing Training. For 1IFAR-10/100 the training epoch is 100, the learning rate is set to 0.01 and reduced by factor of 10 at 30, 60, 80 epochs. For Tiny-Imagenet, the number of training epochs is 150 for each task. The learning rate is set to 0.01, and reduced by factor of 10 at epochs 60, 100 and 130.
Table
C.2 Tuning d and
Dataset
Experiments
a
We mainly make tuning on 0g in Eq. 14 and o in Eq. 16. A origins from the IB method and where k is the label, n; is the number of samples in the k-th class and y the hyper-parameter. We
Base-O-2 Phases
10
Ic-6
CIFAR-10
Basc-0-5 Phases
100
5c-6
K n tune y and by grid searching and adopt dif- ferent values on different dataset and different experiments, which are given in Table
given by A yym Em=n,
Base-0-2 Phases
100
5e-6
Base-0-5 Phases
300
5e-6
CIFAR-100
Base-0-10 Phuses
100
5e-6
Base-half-5 Phases
300
5e-6
Base-half-10 Phases
100
5e-6
Basc-half-5 Phases
10
Ic-6
Tiny- Imagenel
Basc-half-1 Phases
10
5c-6
15

---

D Comparison Results of Average Accuracy
CIFAR-10 (2 Phases)
CIFAR-10 (5 Phases)
100
95
90
AAcouracy(%)
85
Accuracy
-CaRR
80
BIC
-ICaaRL
80
SSIL-NME
75
BIC
SSIL-NME
75
Remix- CNN
Remix-NMME
70
Remix-CNN
RemixKD-ONN CNN
RemixKD-
70
RemixKD-NMEE
RemixKD-NME
(NOL-C)N(ours)
65
EDBL (MMEour)))
(D
UpperBound
UpperBound
60
50
Number of tasks
Number of tasks
(a)
(b)
CIFAR-100 (2 Phases)
CIFAR-100 Phases)
CIFAR-100 (10Phases)
99
AccursitSS
TB
Acturaoy(10 (%)
>0
AccursroyS)
BC
BC
DSLAMME
-
Berte CNN
Burte H918
BarkdD cAN
COOL AMitlan
43
Uppartlund
Upperbrursl
10
Number of taaks
(c)
(d)
Figure 4: Comparison results of average accuracy of Base-O experiments CIFAR-10 with 2, 5
phases and CIFAR-I00 with 2, 5, 10 pahses.
We conducted experiments on CIFAR- following Base-O protocol to evaluate our method. The memory budgets on CIFAR-IO and CIFAR- 100 are fixed to 200 and 2000, respectively. CIFAR-10 10 split into 2 and 5 phases while CIFAR- 100 is split into 2, 5 and 10 phases. Remix[31] adapted Mixup to generate more effective interpolated data to tackle the long-tail learning. Our method employs Mixup technique, so we use Remix compared method and directly employ Remix to train the new model to learn new classes incrementally with the optimal prreeepprraeters given in BI. this supplement, to compare Remix with our method fairly, we further combine Remix with knowledge distillation (KD) to conduct experiments and report the results of CNN output and the nearest-mest -ean-of-eeeeeeeeeeee of-exemplar strategy (NME) (denoted as RemixKD-CN and Remix respectively).
The comparison of the results shown in Fig. From Fig. 4 we can find that our methods (EDBL- CNN and EDBL-NME) outperform all the basclines nearly every incremental task except EDBL- CNN loses some baselines at the experiment on CIFAR- with 10 phases. Especially, our and Remix CNNN RemixKD-NME significantly by large margins about [1.1% 22%) at the last incremental phase. We re-conducted the experiment of our methods on CIFAR-10 10 with 5 phases and we got better results, compared with the results given in Table 2. The incremental average accuracies of EDBL-CNN and EDBL-NME on CIFAR- 10 with 5 phases are 80.4% and 81. 894%, respectively, which surpass all the baselines significantly.
methods surpass Remix-CNN Remix-NNi
16


Reviews:
[{"review_id": "uX3krpK_vsJ", "reviewer": "Reviewer_Fq7t", "summary": "This paper first points out that the scantness of exemplars for the old classes and OOD between the previous classes and the new classes\ncause insufficient knowledge distillation (KD). Then it proposes a two-phase method (EDBL) for class incremental learning. In the first phase, re-sampling strategy and Mixup Knowledge Distillation are used to generate mixed data and to form a better decision boundary.  The second phase uses the incremental influence-balanced (IIB) method to mitigate the imbalanced learning problem.  Experiments show that EDBL outperforms baselines by a large margin.", "questions": "Question 1:  The EDBL method generates more mixed data for the two types of mixup among old classes and mixup between old\nclasses and new classes than the type of mixup among new classes. But the rate ($N/2$) set for the first two types is just an empirical value. Please try to investigate the performance of EDBL at different rates and illustrate the results. Also, what does the \"tail and head class\" mean (Line 162)?\n\nQuestion 2: Can the EDBL method help the KD method to alleviate the forgetting problem further? Please calculate and compare the average forgetting rate of the EDBL method and other baselines. \n\nQuestion 3: The paper says the classification weighting factor is for learning new tasks. But it also calculates the classification weighting factor of the mixed data generated by the samples from old classes. Can the author explain the role of the classification weighting factor in this case?\n\nQuestion 4: That is just a minor question: does the EDBL method also work well in the few-shot continual learning setting? In this setting, the model is over-fitting to the new data severely. And the IIB loss may have a significant impact in this setting.\n\n\n", "limitations": "I don't see any potential negative societal impact in this paper", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The idea of generating mixed data by mixup using samples from old classes and new classes has been used in class incremental learning [1].  So that is not a new contribution. The introduction of the Influence Balance method and the decomposition of the Incremental Influence Balance factor are novel. The novelty of this paper is somewhat reduced.\n\n[1] Class-Incremental Learning via Dual Augmentation, NeurIPS2021.\n\nQuality: This paper is technically sound and the claims are supported. However, the experimental setup is flawed. The authors should discuss and compare more batch continual learning methods which use the KD mechanism [2,3,4].\n\n[2] Dark experience for general continual learning: a strong, simple baseline, NeurIPS2020\n\n[3] Co2L: Contrastive Continual Learning, ICCV2021\n\nThe method outperforms many baselines in the setting of CIFAR100 and Tiny-ImageNet. But I don't know if the EDBL method also has the lowest average forgetting rate.\n\nClarity: This paper is well organized. I suggest the author give a simple introduction for task/domain incremental learning in Section 2 and a brief comparison of other approaches like the regularization-based approach.\n\nSignificance: The EDBL method outperforms many baselines in the setting of CIFAR100 and Tiny-ImageNet.  And the idea of calculating the incremental influence-balance weighting factor for each sample and then using the factor to adjust the loss is interesting. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a05Kr9nI0MF", "reviewer": "Reviewer_PDXX", "summary": "The authors address the problem of imbalance between old and new\nclasses in class-incremental learning due to the limited memory for\nold classes.  For re-sampling, they added 3 types of Mixup instances:\namong old classes, between old and new classes, and among new classes,\nwith proportionally more for the first 2 types to yield balanced\nclasses.  The loss function has cross entropy and knowledge\ndistillation.  They adapted influence balanced (IB) method to include\nboth the classification and knowledge distillation terms from the loss\nfunction, which they call IIB.  They further use hyperparameter alpha\nto weight the IIB factor for knowledge distillation (to decompose\nknowledge distillation and cross entropy).  The loss function L_IIB is\nlambda * L_ce / IIB, where lambda is class-wise reweighting.\n\nThe overall 2-phase EDBL algorithm combines mixing and IIB.  The first\nphase has knowledge distillation.  The second phase has L_IIB, which\nincludes classification and knowledge distillation.\n\nWith 3 datasets, empirical results indicate the the proposed EDBL\ngenerally outperforms a number of existing techniques.  An ablation\nstudy also indicates the contribution of each of the proposed\ncomponents.\n", "questions": "According to Algorithm 1 in the Appendix, the main difference between\nPhase 1 (MKD) and Phase 2 (Balancing training) is whether IIB is\nincorporated or not.  Why not having only Phase 2?  A comparison of\nthe last 2 rows in Table 3 (if I understand correctly) seems to\nindicate the benefit of adding Phase 1 (but seems to be not discussed\nin Section 5.5.1).  Further insights would be important.\n\nEq 11: the first term seems to be missing 2* for f^t(x) and h. If so,\nEq 13 and Eq 15 seem to have similar issues.\n\nLines 223, 234, and 236: balancing training ->  balanced training?\n\nPhases are used in both EDBL and CIL.  Using different terms could be\neasier to separate the two concepts; e.g. stages in EDBL and phases in\nCIL.\n", "limitations": "Limitations and negative societal impact were discussed.\n\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe combination of Resampling with Mixup and IIB seem interesting.\n\nEmpirical results indicate the the proposed EDBL generally outperforms\na number of existing techniques.\n\nWeaknesses\n\nWhile the paper is generally well written, the motivation and\ndescription of the overall 2-phase EDBL algorithm are not clear.  I\nhad to consult Algorithm 1 in the Appendix.  It could be improved by\ndescribing and motivating the 2 phases first, and then the respective\nloss functions that are involved.\n\nPhase 2 can be considered as a superset of Phase 1--the motivation for\nthe need of Phase 1 could be further explained. (See questions below)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O93uLpEtan2", "reviewer": "Reviewer_1CA1", "summary": "In this paper, the authors find that the main problem with the rehearsal methods is decision boundary overfitting to new classes. To solve this problem, they propose to combine the mixup and re-sampling strategy to synthesize adequate data used in knowledge distillation. Extensive experiment results are provided.", "questions": "Please address the issues raised in the \"weaknesses\" part.", "limitations": "It would be better to add a section to discuss the potential negative societal impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n\n- The proposed method is technically sound. \n\n- This paper is well-organized and easy to follow.\n\n- Extensive experimental results and visualizations are provided.\n\n&nbsp;\n\n### Weaknesses\n\n- **The authors only provide results on small-scale datasets, e.g., CIFAR-100.** However, it is quite common to evaluate class-incremental learning on large-scale datasets. For example, iCaRl [12], BiC [8], LUCIR [10], Mnemonics [51], and PODNet-CNN [52] all provide result on ImageNet (full size, 1000 classes). Thus, it is not reasonable to ignore large-scale datasets for a top-tier conference submission. \n\n&nbsp;\n\n- **The technical novelty of this paper is somewhat limited.** This paper is based on re-sampling and mixup. Both strategies have already been widely used in many related topics, such as long-tailed recognition and few-shot learning. Thus, the technical contributions of this paper are somewhat limited. \n\n&nbsp;\n\n- **The state-of-the-art method is compared in Table 1.** For example, [A] archives much better performance than other baselines, but it is not compared in this paper. It seems the proposed method cannot achieve better performance than [A]. \n\n&nbsp;\n\n[A] Yan, Shipeng, Jiangwei Xie, and Xuming He. \"Der: Dynamically expandable representation for class incremental learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n\n&nbsp;\n\n=== **After rebuttal** ===\n\nThanks for the active responses from the authors. The authors provide some new results in the rebuttal period and address some of my concerns. However, my major concerns about large-scale datasets and the technical novelty still remain. Thus, I tend to keep my initial rating. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NEBx4JaoBN", "reviewer": "Reviewer_JLqd", "summary": "This paper addresses the problem of class incremental learning with an emphasis on adjusting the decision boundary. The authors claim that the deficiency of old data and the imbalanced training mainly cause the ineffective decision boundary. The paper proposes Re-MKD, which combines re-sampling strategy and Mixup, and Incremental Influence Balanced (IIB) loss to tackle the aforementioned problem. The experimental results on various benchmarks show the effectiveness of the proposed framework.", "questions": "- As mentioned in the weakness section, I wonder how the overall incremental learning performance would be affected when we exclude the mixed data of the newly added classes when computing $L_{kd}$.\n- Why is the IIB factor only multiplied to $L_{ce}$? Since $L_{kd}$ also can affect the overall decision boundary and IIB factor also considers $L_{kd}$ as in eq (10), IIB weighting factor may be helpful to $L_{kd}$ in eq (16), too. If possible, it would be nice to see how the final performance will be affected if IIB factor is also considered to $L_{kd}$. Or, if I missed something, please let me know.\n- Conventionally, many CIL methods [9, 10, 52] utilize additional “class balanced fine-tuning (CBF).” It seems that the second phase of EDBL algorithm, which is balanced training, can be a good replacement for CBF, since CBF only utilizes small number of examples unlike the proposed framework. Thus, it would be nice if the ablation study of balanced training is added to the paper. Or, is “Re-MKD” row in Table 3 indicates the Re-MKD + CBF?\n- Figure 3 can be improved if it contains more classes than two classes. Even though it is a minor issue for me, it seems demonstrating only a single case may weaken the claim of the paper.", "limitations": "The paper sufficiently deals with the limitation of the paper and broader impacts.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n    - Overall, the paper is well-written and clear. The authors’ claim that imbalanced training is critical issue for the ineffective decision boundary is well supported throughout the paper. Both proposed methods (Re-MKD and IIB loss) may not be super-novel, but they are reasonable solution for the problem. In particular, the motivation of the proposed solutions are clear. Re-MKD makes good use of the rate rebalancing of the old data and can populate many diverse examples from the old data. IIB also seems suitable for the class incremental learning since class incremental learning can be viewed as long-tail problem, where the old data is extremely scarce.\n\n- Weaknesses\n    - I have an issue with the claim made in the paper, which leave some room for improvement.\n        - The authors repeatedly emphasize that data of newly added classes, which are OOD data from the perspective of the previously learned classes, harm the knowledge distillation process. However, the proposed framework still uses the mixed data of the newly-added data for knowledge distillation. It would be nice if the paper includes the ablation study about the 3 types of mixing classes mentioned in Line 158-159. In other words, I wonder how the overall incremental learning performance will be affected when we exclude the mixed data of the newly added classes from computing $L_{kd}$.\n    - There are some serious formatting issues throughout the paper.\n        - Font size of the captions and equations is significantly smaller than the original submission format. This should be revised.\n    - Minor notation error\n        - The notation $L$ is used multiple times in different contexts. In general, $L$ denotes the loss, but in Line 188, $L$ denotes the feature dimension. Moreover, $f$ in Line 191 also needs to be corrected since $f$ does not indicate the dimension of the feature vector.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject