CPR: Classifier-Projection Regularization for Continual Learning
Authors: Sungmin Cha
Hsiang Hsu
Taebaek Hwang
Flavio Calmon
Taesup Moon
Venue: ICLR 2021 Poster (2021)
Keywords: continual learning
regularization
wide local minima

Abstract: We propose a general, yet simple patch that can be applied to existing regularization-based continual learning methods called classifier-projection regularization (CPR). Inspired by both recent results on neural networks with wide local minima and information theory, CPR adds an additional regularization term that maximizes the entropy of a classifier's output probability. We demonstrate that this additional term can be interpreted as a projection of the conditional probability given by a classifier's output to the uniform distribution. By applying the Pythagorean theorem for KL divergence, we then prove that this projection may (in theory) improve the performance of continual learning methods. In our extensive experimental results, we apply CPR to several state-of-the-art regularization-based continual learning methods and benchmark performance on popular image recognition datasets. Our results demonstrate that CPR indeed promotes a wide local minima and significantly improves both accuracy and plasticity while simultaneously mitigating the catastrophic forgetting of baseline continual learning methods. The codes and scripts for this work are available at https://github.com/csm9493/CPR_CL.

Published as a conference paper at ICLR 2021
CPR: CLASSIFIER-PROJECTIOO REGULARIZATION FOR CONTINUAL LEARNING
Sungmin Cha', Hsiang Hsu", Taebaek Hwang', Flavvio P. Calmon', and Taesup Moon 'Sungkyunkwan University HHarvard University "Scoul National University csm94930skku. edu, hsi asianghsugg harvard. edu, gxq91068 1mm00mmmaall....mm com, fcalmongg, harvard, edu, tsmoon@snu. ac. kr
ABSTRACT
We propose a general, yet simple patch that can be applied existing regularization- based continual learning methods called classifier-projection regularization (CPR). Inspired by both recent results on neural networks with wide local minima and information theory, CPR adds an additional regularization term that maximizes the entropy of classifier's output probability. We demonstrate that this additional term can be interpreted as a projection of the conditional probability given by a classifier's output to the uniform distribution. By applying the Pythagorean theorem for KI. divergence, we then prove that this projection may (in theory) improve the performance of continual learning methods. In our extensive experimental results, we apply CPR to several state-of-the-ar methods and benchmark performance on popular image recognition datasets. Our results demonstrate that CPR indeed promotes wide local minima and signifi- cantly improves both accuracy and plasticity while simultaneously mitigating the catastrophic forgetting of baseline continual learning methods. The codes and scripts for this work are available at htttss//gihuub.coo//m////////
egguarization-basee continual learning
CL
INTRODUCTION
Catastrophic forgetting (McCloskey & Cohen 1989) is central challenge in continual learning (CL): when training a model on a new task, there may be loss of performance (e.g.. decrease in accuracy) when applying the updated model previous tasks. At the heart of catastrophic forgetting is the stability-laastciity dilemma (Carpenter & Grossberg. 1987 Mermillod et al. 2013), where a model exhibits high stability on previously trained tasks, but suffers from low plasticity for the integration of new knowledge (and vice- versa). Attempts to overcome this challenge in neural betwork-basee CL can be grouped into three main strategies: regularization methods (Li & Hoiem 2017 Kirkpatrick et al. 2017 Zenke et al., 2017 Nguyen et al. 2018 Ahn et al., 2019: Aljundi et 2019),, memory replay (Lopez-Paz & Ranzato 2017] Shin et al. 201 Rebuff et al. 2017| Kemker & Kanan 2018). and dynamic network architecture Rusuu et al., 2016 Yoon al. 2018 Golkar et al., 2019)). In particular, regularization methods that control model weights bear the longest history due to its simplicity and efficiency to control the trade-off for a fixed model capacity.
In parallel, several recent methods seek to improve the generalization of neural network models trained single task by promoting wide local minima (Keskar et al.) [2017] Chaudhari et al.) 2019 Pereyra et al. 2017 Zhang et al., 2018). Broadly speaking, these efforts have experimentally shown that models trained with wide local minima- poromoiingoooiggggggom regularizers achieve better generalization and higher accuracy (Keskar et al. 2017 |Pereyra et al.) [2017] Chaudhari et al.; 2019 Zhang et al.) (0188). and can be more robust to weight perturbations (Zhang et al.) (2018) when compared to usual training methods. Despite the promising results, methods that promote wide local minima have yet to be applied to CL.
In this paper, we make novel connection between wide local minima in neural networks and regularization-bast CL methods. The typical regularizaaion-basce CL aims to preserve important weight parameters used in past tasks by penalizing large deviations when learning new tasks. As
"Corresponding author (F-mail: camoonesnu.ac. ac..

---

Published as a conference paper at ICLR 2021
te entr bo Tase
8%
8
6
Typical regularization-basee CL
6%
07
Regularization-based CL local minima
Figure I: In typical regularization- based CL. (top), when the low-error ellipsoid around local minima sharp and narrow, the space for candidate model parameters that perform well on all tasks (iee. the intersection of the ellipsoid for each task) quickly becomes very small as learning continues, thus, an inevitable trade-off between stability and plasiticty
In contrast, when the wide local minima exists for each task (bottom). it is likely the cllipsoids will significantly overlap even when the lcarning continues, hence, finding well performing model for all tasks becomes more feasible.
shown in the top of Fig. popular geometric intuition (as first given in EWC (Kirkpatrick et al. 2017)) for such CL methods to consider the (uncertainty) ellipsoid of parameters around local minima. When learning new tasks, parameter updates are selected in order to not significantly hinder model performance on past tasks. Our intuition is that promoting a wide local minima-wiin conceptually stands for local minima having a flar, rounded uncertainty ellipsoid can be particularly beneficial for regularization-bant based CL methods by facilitating diverse update directions for the new tasks (i.e,, improves plasticity) while not hurting the past tasks (i.e., retains stability). As shown in the bottom of Fig. [ when the ellipsoid containing the parameters with low-error is wider, i.e., when the wide local minima exists, there is more flexibility finding a parameter that performs well for all tasks after learning sequence of new tasks. We provide further details in Section 2.1
Based the above intuition, we propose general, yet simple patch that be applied existing regularization-base CL methods dubbed as Classifier-Projection Regularization (CPR). Our method implements an additional regularization term that promotes wide local minima by maximizing the entropy of the classifier's output distribution. Furthermore, from theory standpoint, we make an observation that our CPR term can be further interpreted in terms of information projection (I-projection) formulations (Cover & Thomas 2012] Murphy| 2012 Csiszár & Matus [003 Walsh Regalial 2010 Amari et al.] 2001] Csiszár & Matus (0003] Csiszár & Shields 2004) found in information theory. Namely, argue that applying CPR corresponds to projecting classifier's output onto a Kullback-Leible (KL) divergence ball of finite radius centered around the uniform distribution. By applying the Pythagorean theorem for KL divergence, then prove that this projection may (in theory) improve the performance of continual learning methods.
Through extensive experiments several benchmark datasets, demonstrate that applying CPR can significantly improve the performance of the ftate-of-the-ee regularization- based CL: using our simple patch improves both the stability and plasticity and, hence, achieves better average accuracy almost uniformly across the tested algorithms and lanasets-confirmiigg our intuition of wide local minima Fig. I Furthermore, we use feature map visualization that compares methods trained with and without CPR to further corroborate the effectiveness of our method.
CPR: MINIMUM
REGULARIZATION FOR WIDE LOCAL
In this section, elaborate in detail the core motivation outlined in Fig. then formalize CPR the combination two regularization terms: one stemming from prior regularization based CL methods, and the other that promotes wide local minima. Moreover, provide interpretation (Csiszár 1984; Cover & Thomas 2012; Murphy 2012) for the observed gain in performance when applying CPR CI..
2

---

Published as a conference paper at ICLR 2021
We consider continual learning of T classification tasks, where each task contains N training sample- label pairs {xx,, Vn 1n
1. t [1,... .I] with x", E RR, and the labels of each task has M, classes, i.e. yn [1,... .Ml. Note that task boundaries are given in evaluation time; ie., we consider task-aware setting. We denote fo Rd > AM as a neural network- based classification model with softmax output layer parameterized by 0.
2.1 MOTIVATION: INTRODUCING WIDE LOCAL MINIMA IN CONTINUAL. LEARNING
Considering the setting of typical regularization-bassee CL (top of Fig. denote 0 parameters that achieve local minima for specific task and 0 is that obtained with regularization terms. Assuming that 0 is learnt, when learning task 2, an appropriate regularization updates the parameters from i to 0₂ instead of fr since 0g achieves low-errors
both tasks and 2. However, when the low-error regimes (ellipsoids in Fig. [ are narrow, it is often infeasible to obtain a parameter that performs well all three tasks. This situation results in the trade-off between stability and plasticity in regularization-bant based CL (Carpenter & Grossberg 1987). Namely, stronger regularization strength (direction towards past tasks) brings more stability (0z), and hence less forgetting on past tasks. In contrast, weaker regularization strength (direction towards future tasks) leads to more plasticity so that the updated parameter 03 performs better on recent tasks, at the cost compromising the performance of past tasks.
key problem in the previous setting is that the parameter regimes that achieve low error for each task often narrow and do not overlap with each other. Therefore, straightforward solution is to enlarge the low-error regimes such that they have non-empty intersections with higher chance. This observation motivates to consider wide local minima for each task in CL (bottom of Fig. [1]. CL can more easily find parameter, 03 that is close to the the local minimas for each task, i.e., {0: 10;);--- Moreover, it suggests that once we promote the wide local minima of neural networks during continual learning, both the stability and plasticity could potentially be improved and result in simultaneously higher accuracy for all task which is later verified in our experimental (see See. 3 In the next section, we introduce the formulation of wide local minima in CL.
With wide local minima for each task, a regularization-bast
2.2 CLASSIFIER PROJECTION REGULARIZATION FOR CONTINUAL LEARNING
Regularization-bane continual learning Typical regularization-bue based CL methods attach regu- larization term that penalizes the deviation important parameters learned from past tasks order to mitigate catastrophic forgetting. The general loss form for these methods when learning task / is
[ (0) = I(CE(() (0) + Σ
Ω 10, - 8;-
(1)
where (0) ()) is the ordinary cross-entropy loss function for task t, À is the dimensionless regular- ization strength, 2 { is the of estimates of the weight importance, and (0--1) is the parameter learned until task f 1. A variety of previous work, e.g., EWC (Kirkpatrick et al. 2017).
SI (Zenke al. 2017), MAS (Aljundi al., 20118), and RWalk (Chaudhry al. 2018). proposed different ways of calculating S2' to measure weight importance.
Single-task wide local minima Several recent schemes have been proposed (Pereyra et al. 2017 (Szegedy et al. 2016 Zhhng et al.] 2018) to promote wide local minima of neural network for solving single task. These approaches can be unified by the following common loss form
TwLM(() = T-CE
(fe((xxx))loo)
N
where is some probability distribution in AM that regularizes the classifier output fe. 3 is a trade-off parameter, and (-L(-ll-) g is uniform distribution Py in Ass the regularization term corresponds entropy maximization proposed in Pereyra et al. (0177). and when is another classifier's output fo. Eq. (2) becomes equivalent to the loss function in Zhang al. (2018).
[I] is the KL divergence (Cover & Thomas 2012). Note that, for example, when
3

---

Published as a conference paper at ICLR 2021
CPR: Achieving wide local minima in continual learning Combining the above two regulariza- tion terms, we propose the CPR as the following loss form for learning task t:
3 Σ (f + + +
(() (0)
2 10.-0--11 02-
(3)
where À and 3 are the regularization parameters. The first regularization term promotes the wide local minima while learning task by using Py as the regularizing distribution (2) and the second term is from the typical regularization-basse
CL. Note that this formulation is oblivious to f and, hence, it can be applied to any state-of-the-ar regularization-ban based CL methods. In our experiments, we show that the simple addition of the KL-term can significantly boost the performance of several representative state-of-ther the-hrt methods, confirming our intuition on wide local minima for CL given in Section 2.1]and and [ig Furthermore, show in the next section that the KL-term can be geometrically interpreted in terms information projections (Csiszár) 1984 Cover & T'homas) 2012) Murphy 2012), providing additional argument (besides promoting wide local minima) for the benefit of using CPR in continual learning.
2.3 INTERPRETATION BY INFORMATION PROJECTION
Minimizing the KL divergence terms in 2 and (3) can be expressed as the optimization min (Q] where P is a given distribution and Q is a convex set of distributions in the QEO
probability simplex Am - (p E [0, 1|" [0. P: 1]. In other words, the optimizer P* is distribution in Q that is "closest" to P. where the distance measured by the KL divergence, and is termed the information projection (also called I-projection), i.c.,
P arg min (Q)
QEQ
The quantity P has several operational interpretations in information theory (e.g., in universal source coding (Cover & Thomas 2012)). In addition. the information projection enables a "geometric" interpretation of KI. divergence, where DKl(Q]))) behaves as the squared Fuclidean distance, and (Q, P", P) form "right triangle". The following lemma resembles the Pythagoras' triangle inequality theorem (not satisfied general by the KL divergence) in information geometry (Cover Thomas 2012).
Lemma 1, Suppose 3P* E EQ such that (KL(P" = min DKL (K](P))), then QEQ
DKL(QIIP) 2 (KL(Q]IP)) DKL [P). AQ Q.
(5)
In this sense, when minimizing the KI. divergence terms 2 and (3), we project the classifier output fo(-) towards a given distribution (e.g., a uniform distribution). Since fo(-) can be viewed as conditional probability distribution Qy where Y is the class label and X the input, we consider an extension of the information projection to conditional distributions. We call this extension as the classifier projection, which seeks a classifier output Py in a set that is closest (measured by the expected KI. divergence) to a given conditional distribution PY|x- Formally, given convex set C of conditional distributions, the classifier projection is defined as
pyix = arg min Epx Qrix EC
[DL (xxxx)))))(
(6)
The KL divergence terms in 2 and 3 are exactly the empirical version of the objective in (6) by X) g or Pu.
taking () fo(-) and (
Now, we explain why classifier projection can help overcoming catastrophic forgetting. Let the classifier after training task 1 be pyix C Am, and suppose we have two classifiers for task 2, one is x & C and the other is p2- by projecting P²ix to C using (6), the triplet (P/1 x forms right triangle, and P1+ fix and Pžrx are closer to each other measured by the KI. divergence (Lemma I). Therefore, when evaluating on task 1, the classifier of task 2 after projection is more similar to each other in terms of cross coss-entropyy (transformed from the KL divergence), guaranteeing smaller change in training loss and accuracy. The following proposition formally summarizes this paragraph.
4

---

Published as a conference paper at ICLR 2021
Proposition 1. For any classifier P C for task - with data distribution Px and let any classifier for task t be Py|x & and Pyix be the projected classifier by (6). then
E Pp--1 PX [ -log log P P
Epttxi
log p P'
To implement the CPR, we need to pre-define the set of possible classifiers C for projection. An intuitively best choice is the set of
classifiers that perform well on all tasks (can be obtained by training all tasks simultaneously). However, in CL setting, such classifiers are not available; therefore, pick the set of possible classifiers C to be a KL divergence ball centered the uniform distribution Pt. i.e.
RP
C(Pb,e) & {Qyix E AA Ex [DRI(QrIx[1EH]
S c).
Mia KL divergesen ball C(P Probehiliyy Siupos day
We select Po since it is the centroid of A and, hence, the worst- case divergence between any distribution and Py is at most log M. From the vantage point of classifier projection, the CPR regulariza- tion term in (3) can be viewed as the Lagrange dual of the constraint Qyix E C(Pu.c) the term that projects the classifier of individual tasks towards the uniform distribution in order to minimize changes when training sequential tasks (See Fig.
Figure 2: CPR can be understood as projection onto finite radius ball around Pv.
EXPERIMENTAL RESULTS
We apply CPR to four regularization-basee supervised CL methods: EWC (Kirkpatrick et al. 2017) SI (Zenke al. 2017), MAS (Aljundi et al., 2018). RWalk (Chaudhry al.) 2018) and AGS-CL (Jung et al. 2222)) and further analyze CPR via ablation studies and feature map visualizations.
3.1 DATA AND EVALUATION METRICS
We select CIFAR-100, CIFAR- 10/100 (Krizhevsky et 2009), Omniglot (Lkee et al. 2015). and CUB200 (Welinder et al. 2010) as benchmark datasets. Note that we ignore the MerMuted-MNIST dataset (LeCun et al.] 1998) since most state-of- oftte-of-theaass art algorithms can already achieve near perfect accuracy on it. CIFAR-100 is divided into 10 tasks where each task has 10 classes. CIFAR-10/100 additionally uses CIFAR-I0 for pre-training before learning tasks from CIFAR- 1000 Omniglot has 50 tasks, where each task is binary image classification on a given alphabet. For these datasets, used a simple feed-forward convolutional neural network (CNN) architecture. For the more challenging CUB200 dataset, which has 10 tasks with 20 classes for each task, used pre-trained ResNet- 18 (He et al.) 2016) as the initial model. Training details, model architectures, hyperparameters tuning, and source codes are available in the Supplementary Material (SM).
For evaluation, we first let dg.j E [0, 1] be the j-th task accuracy after training the k-th task (i S k). Then, used the following three metrics measure the continual learning performance:
Average Accuracy (A) is the average accuracy Aş on the first k tasks after training the k-th task, i.e., A Σ) akj. While being natural metric, Average Accuracy fails to explicitly measure the plasticity and stability of CL method.
Forgetting Measure (F) evaluates stability. Namely, we define the forgetting measure f of the j-th task after training k-tth task ft lE(j...kk-1)
@t.j Ok.j. Vj k, and the average forgetting measure F3 of a CL method as F3- k Σ 5
max
Intransigence Measure (I) measures the plasticity. Let aj be accuracy of a model trained by fine-tuning for the j-the task without applying any regularization (in other words, only task task-specific loss is used). The intransigence measure I, is then defined as La,k - Ej= ij. where in ab -jj.
The F and I metrics were originally proposed in (Chaudhry al. 2018), and slightly modified their definitions for our usage. Note that a low F; and I, implies high stability (low forgetting) and high plasticity (good forward transfer) of CL. method, respectively.
5

---

Published as a conference paper at ICLR 2021
626
LOSA Landicap Cook
024
rEXNIpECA MAMUU 666
EWC 21
200
EWIS
TINC
010
1.2
8.00
100
0.04
200
235
C00
1//
(a) Selecting regularization strength for CPR
(b) Adding Gaussian noise
(c) Analysis using Pyllessia
Figure 3: Verifying the regularization for wide local minima
3.2 QUANTIFYING THE ROLE OF WIDE LOCAL MINIMA REGULARIZATION
We first demonstrate the effect of applying CPR with varying trade-off parameter 3 by taking EWC (Kirkpatrick et a (017) trained on CIFAR-100 as running example. Fig. 3(a] shows how the aforementioned metrics varies as B changes over (0.1,...,11 1].
First, we observe that A1o certainly increases as & increases. Moreover, we can break down the gain in terms of 1, and Fio: observe 1, monotonically decreases as 3 increases, but F10 does not show the similar monotonicity although it also certainly decreases with B. This suggests that enlarged wide local minima is indeed helpful for improving both plasticity and stability. In the subsequent experiments, we selected B using validation sets by considering all three metries; among the I's that achieve sufficiently high A1o. we chose one that can reduce P1o more than reducing 11.10 since it turns out improving the stability seems more challenging. (In fact, in some experiments, when we simply consider A1o, the chosen 3 will result in the lowest 10 but with even higher F10 than the case without CPR.) For comparison purposes, we also provide experiments using Deep Mutual Learning (Zhang et al. (2018) and Label Smoothing (Szegedy et al.] (016) regularizer for achieving the wide local minima in the SM; their performance was slightly worse than CPR.
With the best / in hand, Fig. 3(b) experimenntyl verifies whether using CPR indeed makes the local minima wide. Following the methodology in Zhang et al. (2018), we perturb the network parameters, after learning the final task, of EWC and EWC+CPR by adding Gaussian noise with increasing o, then measure the increase in test loss for each task (for --FAR-100). From the figure, we clearly observe that EWC+CPR has smoother increase in test loss compared with EWC (without CPR) in each task. This result empirically confirms that CPR indeed promotes wide local minima for each task in CL settings and validates initial intuition given in Sec. 2.1 In the SM, we repeat the same experiment with MAS (Aljundi et al. 2018).
To plot the loss landscape of each model directly, we used PyHessian Yao et 2019), which is the framework that can plot the loss landscape of NNs by perturbing the model parameters across the first and second Hessian eigenvectors. As an example, Figure plots and compares the loss landscapes on the training data of the 2nd task of CIFAR- 100, for the networks trained with EWC and EW C+CPR, respectively. clearly shows that, by applying CPR, the loss landscape becomes much wider than that of the vanilla EWC. We consistently observe such trend across all tasks, of which are visualized in the SM, and we believe this confirms our intuition for adding the CPR term.
3.3 COMPARISON WITH STATE-OF-TIIE-AR
Next, we apply CPR to the of ofte-offthhear regularization-batt based CL on the benchmark datasets and measure the performance improvement with the three metries in Section 3.]] For the regularization strengths, we first select the best X without CPR, then choose 3 according to the procedure in Section 3.2 The results in Table I are averaged over 10 repeated experiments with different random initialization and task sequence using the chosen (A, 3). The hyperparameters reported in the SM.
CIFAR-100 and CIFAR-10/100 In Table Fig. (a) we observe that CPR consistently improves all regularization-base methods for all tested datasets terms of increasing Ajo and decreasing I1,00, and also consistently decreases Fig- Additionally, we find that for CIFAR-10/100 the orders of the 10 tasks in CIFAR- 100 and CIFAR-10 affect the performance of the CPR; namely, in the SM, we show that when CIFAR- 10 tasks are positioned in different positions rather than at the beginning. the gain due to CPR got much bigger.

---

Published as a conference paper at ICLR 2021
Table 1: Experimental results CL benchmark dataset with and without CPR.
Average Accuracy (Ama)
Focpetine Meusure (Ein)
Intransigenee Measure EA ml
Dataset
Method
Wio CPR
W/
dilf
Wio
W/
diil
Wio
W/
dil
CPR
(W-W/o)
CPR
CPK
(W/-W/o)
CPR
CPK
(W-Wo)
EWC
DONT
0:035
4UM27
(-8.7%)
0.1419
0.117
-.0301
CIFARIOD
(T' = 10)
Rwalk
MAS
SI
0.6172
0.5784
0.6141
0.6476
0.6442
0.6366
+0,0551
+0.0270
+0.0336
(-10.0%)
(+4.4%)
(+5.5%)
0.416
0.1166
0.0937
0.070
0.0460
0.0999
-00169 (-1800%)
-0.0011
-0.0107
(-2.6%)
(-9.7%)
0.0066
0.1155
0.1074
00027
0.044
0.7788
-0,9257
-8,9330
-00239
(-40.8%
(-42.2%)
(-22.2%)
AGS CL
0,6369
0.6615
+0,0246
(+3.9%
0.0259
0,0247
+0,0022
(-4.63%)
0,1100
00865
-0,0235
(-24.4%)
FWC
0.6950
0.2055
+0.0105
(+1.5%)
0.028
601k1
0.0048
(-21.1%)
01121
0. 10008
-8.0062
(-5.5%)
CIPARIOIIO
MAS
SI
0.7239
0.7127
0.7257
0.7185
+0.0059
.0,0017 (+0.2%)
(+0.8%
0.0499
0.0199
03477
0,090
400051 (-11.1%
.00001 (-0.6%
0.0733
0,0605
00021
00538
-0.0015
-0.0012
(-2.5%)
(-1.%%
Kwalk
0.6954
0.7046
+0,0112
(+1.6%)
0.0788
00000
4.0031
(-4.2%)
0,0072
00500
-0,0084 (-12.5%)
(+0.4%)
6,0000
00007
-0,0034 (-4.7%)
EWC
0.6632
0.8387
+0.1755 (+26.5%)
0.30%
031121
8.177
(.84.%%
400211
M20239
0.0012
Omniglot
MAS
St
0.8401
0.8778
0.9679
0.8621
+0,0278 (+3.3%)
+0,0143 (+1.7%)
0.0316
0,0247
09167
10000
.0,079
0.0215 (-68.0%
(-32.0%
-00358
00214
-3,038%
-0,0067
.0,065
(-27.1
(-25.3%)
+0,0440 (+5.5%)
0.0614
0,004
(-59.0%)
40026
-0,006 (-30.1%)
AGS CL
0,8905
+0,0253 (+3.0%
9
0.0323
0/146
.0,0277 (-85,8%)
EWC
0.5746
0.60%
+0,0345 (+6.1%
00811
0,0807
4.0004 (-0,5%)
0,1011
00060
-8,0345
(-34.1%)
CUUR200
+0,0157
(+2.6%)
0.0549
0.0474
-0,0755 (-13,7%
0.0918
00000
1,,091 (-9.9%)
(7 10)
Rwalk
0.6078
0.6334
+0,0281
+8,0247 (+4.1
(+48%)
0.1188
0.0811
6.0601
0.1080
-0,0210 (-25.9%)
0.0158 (-13.3%
0,0575
0.0679
92M00
12000
4013 (-24.2%)
-UA0SS (-8.5%)
AGS-CL CL
0,5403
0.5623
+0.0230 (+1.07%
0.0750
0.0692
+0,058 (-7.7%)
0.1408
0.1241
-0.0667 (-11.7%)
EWC
0.08
MAS
Rwalk
Rwakk
AGS-CL
ADSHCL
4 0.65
0.000
2.0.5
BWC
EWC EPR
EWC
EWC + CAR
0.50
51
51 CPR
0.50
S1
Si+ CAR
MAS
MAS CPR
MAS
MAS OPR
0.35
0.00
Healt: CPH
Reak
Reale CMH
ACS-CL
AGS-C1- EPR
AGS-CL
ACS-CL CAR
Taskit)
Taski!)
Taskio
(a) 100
(b) Omniglot
(C) CUB200
Figure 4: Experimental results on CL benchmark dataset
Omniglot This dataset is well-suited to evaluate CL with long task sequences (50 tasks). In Table D it is clear that the CPR considerably increases both plasticity and stability in long task sequences. In particular, CPR significantly decreases Fio for EWC and leads to a huge improvement in A1o. Interestingly, unlike the previous datasets, 41,10 is negative, implying that past tasks help in learning new tasks for the Omniglot dataset; when applying CPR, the gains in 1,10 are even better. Furthermore, Fig. 4(b) indicates that applying CPR leads to less variation in A, curves.
CUB200 The results in Table and Fig. 4(c) show that CPR is also effective when using pre- preitrained ResNet model for all methods and metrics.
3.4
ABLATION STUDY
6.70
DMC
IWC- (dhiaak
(ulo
6.50
CRA
DWC- CPA
0.2
EMC OH
0:55
EMC CP
0.730
0.13
0.50
0.00
< 6.42
6.11
0.1
1875
6:00
6.02
0.50
182
012
0.55
0.02
1-11
1-3
012
[-1
Cm-5
1+5
063
14.3
(a)
fio
(b) Ie+1,10
(c) A1o and 10
Figure 5: Ablation studies on CL with wide local minima
We study the ablation of the CPR on the regularization-baste methods using CIFAR-100 with the best (A, (3) found previously, and report the averaged results over 5 random initializations and task sequences in Fig. [S] The ablation is performed in two cases: (i) using CPR only at task t, denoted as EWC + CPR (only t-th task), and (ii) using CPR except task t, denoted as EWC + CPR (w/o f-th task). Fig. ((a) shows fi the amount of forgetting for task /. after learning the task 10, and Fig. (bb) shows
7

---

Published as a conference paper at ICLR 2021
I1+1,10, the amount of gap with fine-tuning after task t. In Fig. 5(a) we observe that CPR helps to decrease fio for each task whenever it is used (except for task 3), but fio of EWC+ + CPR (w/o f-th task) shows more random tendency. On average, EWC + CPR does reduce forgetting in all tasks, demonstrating the effectiveness of applying CPR to all tasks. Notably in Fig. ((b) 1, 10 of EWC + CPR (onty 1-th task) is lower than that of EWC+ CPR (w/o 1-th task) only when 7 1; this indicates that CPR is most beneficial in terms of plasticity when CPR is applied as early as possible to the learning sequence. EWC+ + CPR again achieves the lowest (i.e., most favorable) 1, Fig. [(c) as a further evidence, also suggests that applying CPR for f = 1 gives a better accuracy. Moreover, the accuracy of EWC + CPR (w/o t-th task) gets closer the optimal EWC + CPR, which is consistent with the decreasing difference of 10+1,10 between EWC+ + CPR (w/o I-th task) and EWC + CPR in Fig. 5(b) The EWC + CPR still gives the best Aιo and individual 0x accuracy. We emphasize that model converging to a wide local minima from the first task onwards considerably helps the training of future tasks as well, i.e., a significant increase in the plasticity can be achieved. By using this finding, we conducted an experiment on the case where CPR have to learn unscheduled additional tasks and got the impressive experimental result which is reported in the SM.
In the SM, we also visualize how the learned representations change with the additional CPR term. We utilized UMAP (MeInnes et al.) 2018) and show that the learned representations less drift as the learning continues with CPR, which again indirectly corroborates the existence of wide local minima. Due to the space constraint, we refer the detailed explanation and visualization to the SM.
3.5 APPLYING CPR TO CONTINUAL REINFORCEMENT LEARNING
To show the effectiveness of CPR in various do- main. we applied CPR to EWC (Kirkpatrick al. 2017). MAS (Aljundi et al. 2018) and AGS-CL (Jung al.) 2020) in continual RL. We followed exactly same experimental settings with AGS-CL, therefore, we did experiments on different Atari (Brockman et al. 2016) tasks, i.e., (StarGunmer Boring VideoPinball - Crazyelimber--appeeee -oboaar
16
fine-tuning
EWC
Accumulated performance
MAS
12
AGS-CL
FWC EPR
10:
MAS
88
AGS-CL CPR
We used PPO (Schulman et al., 2017) for reinforcement learning and simply applied CPR to PPO by adding KI. divergence term in Eq _ We trained each method on three different seeds and we report the averaged result. More detailed experimental settings are in the SM.
AtmonAttack - CameeThisaamee
Task] Task2 Task3
Taskel Tasks Tasks Task?
Taska
Figure 6: Accumulated normalized reward
Figure the experimental results and fine-tuning means the result of continual learning without regularization. x-axis is the training step and y-axis means the normalized accumulated reward, which is the sum of each task reward normalized by the reward of fine-tuning. We observe that CPR increases the accumulated reward of each method. From the analysis, we found out that, training with the best hyperparameter for each method already do not suffer from catastrophic forgetting on previous tasks but each method shows the difference on the ability to learn a new task well. However, we check that CPR can averagely increase the average reward of each task by 27% (EWC), (MAS) and 5% 5%(AA--CI and we believe that this is the reason why CPR leads to the improvement of the accumulated reward of each method.
4 RELATED WORK
Several methods have been recently proposed to reduce catastrophic forgetting (see Parisi et al. (2119) for a survey). In this paper, we mainly focus on the regularization-basse CL. methods (Li & Hoiem
2017 Kirkpatrick et al. 2017 Aljundii et al. 2018 Chaudhry et al. 2018 Zenke et al. 2017 Nguyen ct al. 2018: Ahn et al. 2019 Aljundi et al. 2019 Jung et al. 2020)) Broadly speaking, the motivation behind the regularizationnbased CL is to measure the importance of model parameters in previous tasks. This measure is then used in a regularization term for overcoming catastrophic forgetting when training for new tasks. Consequently, the main research focus of the regularization-bat based CL is
creating metrics for quantifying weight importance on previous tasks (e.g.. KKirpptrrck et al. (2017);
Aljundi et al. (2018); Chaudhry et al. (2018); Zenke al. (2017); Nguyen al. (2018) Ahn et al.
8

---

Published as a conference paper at ICLR 2021
(2019) ; Jung et al. (2020). ) In contrast, here we focus developing general method for augmenting regularization based CL instead of proposing (yet another) new metric for measuring the weight importance.
The work that shares similar philosophy as ours is Aljundi et al. (201)). which encourages sparsity of representations for each task by adding additional regularizer to the egularizationnbased CL methods. Note that the motivation of Aljundi et al. (2019) imposing sparsity of neuron activations is considerably different from ours that promotes wide local minima. Moreover, whereas Aljundi et al. (2019) focuses on average accuracy, we carefully evaluate the advantage of the added CPR regularization in terms of increasing both plasticity and stability of CL in addition to accuracy.
Several papers have recently proposed methods that promote wide local minima in neural networks in order to improve single-task generalization, including using small mini-batch size (Keskar et al. 2017), regularizing the output of the softmax layer in neural networks (Szegedy et al. al. 2017), using a newly proposed optimizer which constructs a local-entropy-barn based objective function (Pereyra et al. 2017) and distilling knowledge from other models (Zhang et al. 2018). We expand upon this prior work and investigate here the role of wide local minima in CL.
Mirzadeh et al. (2020) recently proposed a similar point of view with ours the advantages of wide local minina in continual learning. However, our work significantly different from theirs in the following aspects. Firstly, the core motivations are different. Mirzadeh et al. (2020) begins with defining a metric for forgetting, then approximates it with the second order Taylor expansion. They then mainly focus the stability and argue that, if the model converges minima during continual learning, the forgetting would decrease. However, as shown in Fig. [ our paper is motivated from geometric intuition, from which we further explain that if the model achieves a wide local minima for each task during continual learning, not only stability but also plasticity will improve (in other words, the trade-off between stability and plasticity in continual learning can be improved simultaneously). Secondly, the proposed method for converging to wide local minima is different. Mirzadeh et al. (2020) proposed control three elements, such as, learning rate, mini-batch size and drop out. On the other hand, we used classifier projection as a regularization that promotes wide local minima. Therefore, our method requires only one additional hyperparameter to be controlled, so the complexity of our method is much lower. Thirdly, while Mirzadeh et al. (2020) only empirically analyzed the forgetting of CL, we proposed a more principled theoretical interpretation of the role of CPR in terms of information projection. Fourthly, unlike Mirzadeh et al. (2020) which only considered a single epoch setting on a limited benchmark sets, we conducted extensive experiments four different classification datasets (CIFAR 100, CIFAR 1 10/000, Omniglot, CUB200) and continual reinforcement learning using Atari 8 tasks. Finally, there is a difference in experimental analyses. We conducted a more detailed experimental analysis of the effect of wide local minima for continual learning. This is because, by using UMAP, we verified the change of feature maps of model with or without CPR and how the plasticity and stability were improved after applying CPR through ablation study.
wide local
multiple epochs setting and diverse settings, such as,
CONCLUDING REMARK
We proposed a simple classifier projfecti--oocctiionn regularization (CPR) which can be combined with any regularization-basse continual learning (CL) method. Through extensive experiments in supervised and reinforcement learning, we demonstrated that, by converging to wide local minima each task, CPR can significantly increase the plasticity and stability of CL. These encouraging results indicate that wide local sunima-promoting regularizers have a critical role in successful CL. As a thcoretical interpretation, we argue that the additional term found in CPR can be understood as projection of the conditional probability given by a classifier's 's output onto a ball centered around the uniform distribution.
ACKNOWLEDGMENT
This work supported in part by NRF Mid-Carec Research Program [NRF-202IRIAZC20023 and IITP grant 201. 0-01396, Development of framework for analyzing, detecting, mitigating of bias in AI model and training data], funded by the Korean govermment (MSIT).
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. snccertainty-basee continual learning with adaptive regularization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4394-4404, 2019.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 139-154, 2018.
Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars. Selfless sequential learning. In International Conference on Learning Representations (ICLR), 2019.
Shun-ichi Amari, S Ikeda, and H Shimokawa. Information geometry of-projection in mean field approximation. Advanced Mean Field Methods, pp. 241-255 2001.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym. a60iv 01540, 2016.
Gail A Carpenter and Stephen Grossberg. Art 2: Self-organization of stable category recognition codes for analog input patterns. Applied Optics, 26(23):4919-4930 1987.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgdt Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124116 2019.
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 532-547, 2018.
Thomas M Cover and Joy Thomas. Elements of information theory. John Wiley & Sons, 2012.
Imre Csiszár, Sanov property, generalized L-projection and conditional limit theorem. The Annals of Probability, pp. 768-793, 1984.
Imre Csiszár and Frantisek Matus. Information projections revisited. IEEE Transactions on Informa- tion Theory, (996))11444411111 2003.
Imre Csiszár and Paul C Shields. Information theory and statistics: A tutorial. Now Publishers Inc, 2004.
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. arXiv preprint arXiv:1903: 144.449933.4476, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 770-778, 2016.
Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node- importance based adaptive group sparse regularization. In Advances in Neural Information Processing Systems (NeuriPS), volume 33, pp. 3647-3658. Curran Associates, Inc., 2020.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. In International Conference Learning Representations (ICLR), 2018.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference Learning Representations (ICLR), 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barekka-BBaiisaaa, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overeoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, (15113):352135242 2017.
ISSN 0027-8424. doi: 10. 10.1003/p 161 1ee.e116111111 1835114
10

---

Published as a conference paper at ICLR 2021
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, (1006266):1332-1133 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( 86(11):278--2324 1998.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-290) 2017
David Lopez-Paz and Marc Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing System (NeurIPS), pp. 6467-6476. 2017.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprini 180.:2200, 2018.
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plasticiii dilemma: Inves- tigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in Psychology, 4:504, 2013.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 7308-7320. Curran Associates, Inc., 2020.
Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In International Conference on Learning Representations (ICL.R) 2018.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.0654. 2017.
Sylvestre-AIvis -Aliis Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampeet. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 2001-2010, 2017.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.046% 04671, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances Neural Information Processing System (NeurIPS), pp. 2990-2999, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 281 2818-2826, 2016.
John MacLaren Walsh and Phillip A Regalia. Belief propagation, dykstra's 's algorithm, and iterated information projections. IEEE Transactions on Information' Theory, 56(8):4114 2010.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. 2010.

---

Published as a conference paper at ICLR 2021
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael Mahoney. Pyhessian: Neural networks through the lens of the hessian. arXiv preprint arXiv: arXiv:1912.07145,222 2019.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations (ICLR), 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning (ICML), pp. 3987-3995, 2017.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 4320 4328, 2018.
12


Reviews:
[{"review_id": "gWN6hbGyKhI", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to add a KL-divergence regularization to the objective of regularized continual learning in order to encourage the output prediction to be close to a uniform distribution over classes (i.e., increasing the entropy). They argue that this regularization makes the local minima flat and thus less prone to forgetting. They try to build a theoretical connection using results from information projection but there is still a large gap. In experiments, they show on several benchmarks that applying the KL divergence regularization to different regularization-based continual learning brings improvements. \n\nIt is always interesting to find a simple method that can consistently bring improvements. However, the simple method in this paper lacks sufficient explanation and theoretical justification. The experiments are also less convincing if without comparison to other types of continual learning methods.\n\n(1) The paper uses too much space (~4 pages) to explain the motivation of preferring flat minima and KL divergence and make them overcomplicated, which are actually quite straightforward to understand. However, **it uses less than 1 page for theoretical justification of the main idea (Section 2.3)** with few explanations about the equations and a large gap in building the connection between \"flat minima\" and KL divergence regularization. In particular, in Proposition 1:\n\n(i) The cross-entropy on the left or right-hand side of Eq.7 is defined on two joint distributions in the form of $P(X,Y)=P(Y|X)P(X)$, i.e., it shows that two joint distributions P1(X,Y) and P2(X,Y) becomes closer after projecting one of them to the KL ball. **However, what we truly care is whether P1(Y|X) and P2(Y|X) are close in expectation of X drawn from P(X), i.e., whether $E_{X~\\sim P(X)}[KL(P1|P2)]$ or $E_{X\\sim P(X)}[CE(P1|P2)]$ becomes smaller after the projection**, where E is expectation and CE is cross-entropy. To close this gap, the authors need to prove this very specifically.\n\n(ii) The result is mainly due to **an artifact of constraining all previous tasks' classifiers' outputs close to the uniform distribution**. It is obvious that the distance of distribution outside the ball to distribution inside becomes smaller if we project the outside distribution onto the ball. However, previous tasks' classifiers might already perform very poorly on their data distributions because their predictions are too close to uniform sampling, and the KL regularization accumulates the error and only make the prediction quality worse. In the extreme case, one can think that the classifier for task t-1 produces almost uniform predictions so it performs very poorly. Now making the classifier at task t also produce similar uniform predictions for data drawn from task t-1 does not help.\n\n(iii) Proposition 1 holds only when Lemma 1 holds, and Lemma 1 holds only when P* is the minimum solution for minimizing the KL divergence, i.e., Eq.6. However, the KL divergence is only a regularization term in the objective of Eq.3 (and there are two other terms) so it almost cannot be minimized. So **both Lemma 1 and Proposition 1 do not hold rigorously**.\n\n(2) It is encouraging to see that the proposed KL divergence regularization can improve regularization-based continual learning. However, as shown in recent works, other types of continual learning (e.g., memory replay based and dynamic architecture based) usually performs better. Since the proposed regularization can easily work within these methods, it is more convincing to compare their performance before and after adding the regularization.\n\n(3) The experiments do not cover several standard continual learning benchmarks such as permuted/rotated MNIST/FMNIST and sequential MNIST/Tiny-ImageNet. \n\n(4) Encouraging flat minima can alleviate forgetting in continual learning has been analyzed and studied in a recent paper: \nMirzadeh et al., \"Understanding the Role of Training Regimes in Continual Learning\", NeurIPS 2020. A discussion and comparison to it are recommended. Its analysis is in the model parameter space instead of the output space and is rigorous.\n\n(5) In Figure 3(c), the EWC+CPR achieved flat minima has much worse(larger) loss. Does this indicate that the stability is achieved with the price of significantly degraded loss?\n\n(6) Minor: the notations need to be more rigorous, for example (i) expectations E_{P_X} and E_X are both used in the paper at several positions, but they should be E_{X~P(X)}; (ii) to make the paper and appendix consistent, Q* in the proof of Lemma 1 of the appendix should be P*; (iii) The superscripts and subscripts can be simplified.", "extra_scores": {}, "extra_text": {}}, {"review_id": "N2PlrrSMifT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The viewpoint is quite novel and is built with nice connections to neural network optimization literature.\n\nThe paper has claimed several times that one can improve stability and plasticity simultaneously. This somehow intuitively makes less sense because it seems that there is an inherent tradeoff between them. If the model is well calibrated then increasing stability should decrease plasticity and vice versa. That's why it's usually referred to as the stability-plasticity dilemma. It might be the case that improvement in both measures is a sign of a sup-optimal regime.\n\nThe paper has also claimed the first one to study the connection between widness of local minima and continual learning. There is at least one paper that has studied this relation before:Mirzadeh, Seyed Iman, et al. \"Understanding the role of training regimes in continual learning.\" arXiv preprint arXiv:2006.06958 (2020).\nSince it's been around for a short while the authors were probably not aware of this paper but it basically connects widness of the local minima to stability of continual training. It's worth elaborating on the differences from and similarities to their approach.\n\nThe paper is generally well written. But, some notations are a little unclear and can be improved. For example, the difference between \\theta_i^start and \\hat \\theta_i is wauge. Also, citations should be improved. The authors should note the difference between \\citet and \\citep, etc. A proofreading can improve some typos throughout the paper. \n\nSection 2.3 is not straightforward to follow. For example, it's good to elaborate (e.g. in an appendix section) why proposition 1 is a consequence of lemma 1? Alos, how P_{Y|X}^{t} are related to continual learning steps? The role of P_u is unclear and it's not that obvious where does it appear in equation 7?\n\nExperiments look convincing but it can be improved in terms of clarity of the setup. For example, in measuring intransigence it's mentioned no regularization is applied. It's unclear whether this \"regularization\" is the proposed CPR (using KL) or the normal EWC-like regularization (like last term in equation 3)? Reporting confidence interval in results could have improved the empirical validation. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ExbdvDuYpp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors make the intuitive argument that finding wide local minima in DNN training is particularly desirable for Continual Learning and thus propose a simple loss-modification that is applicable out-of-the-box to a wide array of CL methods. The intuitive argument is supported by a good experimental analysis and a theoretical argument.\n\nPros:\n- The method is trivial to implement and thus very general.\n- The authors show very impressive improvements across three performance measures for various recent algorithms, indicating a benefit in almost all cases. I was pleased to see an application in RL, which is often shunned by most works in CL. \n- Experimental analysis is to a very high standard, investigating the source of aforementioned performance improvements with accepted measures of minima depth. \n- The paper is overall well written, intuitive and easy to follow\n\nCons:\n- The main idea behind this paper is arguably rather simple, straight-forwardly applying a known technique in a new setting. Nevertheless, the CL community may be ignorant to the particular effectiveness of the technique in this setting, which is why this paper makes for a suitable publication.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0bddwZHJk3Z", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors argue that achieving wide local minima during the training of tasks, is beneficial for continual learning. The plausible intuition (explained in Fig. 1) is that it is easier to find a parameter setting that is beneficial for all tasks when tasks have wide local minima. They enforce wide local minima by adding an entropy loss to the classifier ( a known strategy). The loss is further combined with any weight regularization loss, like EWC, SI, MAS. \n\npros.\n- motivation is plausible\n- results show that the simple method improves results over a wide range of weight regularization methods. \n- the simplicity of the method could mean that will be widely adopted in CL losses. \n- experiments also includes results for reinforcement learning \n\ncons:\n- novelty is rather low, it is known that wide minima are desirable. Here they show that they are also (or maybe even more) desirable for CL, and show that enforcing them (with a known method) improves results.\n- improve discussion/results of other techniques to encourage classifiers to be less certain about predictions.\n\nRemarks: \nThe entropy loss seems to enforce the classifier to be less certain about its predictions. Other methods that also have this effect could be discussed in more detail. Also, I could not find the promised results Label Smoothiing in Sup. Mat. \n\nMinor remarks:\n- would the loss also work when combined with non-weight regularization losses like LwF ? \n- I found the ablation study of less interest and could be moved to sup. mat. if needed. \n- At the start of section 2, it would be good to mention that methods are evaluated in a task-aware setting (not task agnostic). \n- many citations are in the wrong format (the authors should check how to correctly use the cite, citep, citet (I am not sure))\n\nFinal recommendation: The proposed method is very simple, but could be widely applied in CL losses and the paper could therefore have an impact.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was some positive consensus towards this paper, which slightly improved after the very strong author rebuttal. Reviewers, in general, appreciate the simplicity of the approach as well as its effectiveness. The most acute criticisms derived from several theoretical and technical points, similarity with [Mizadeh, 2020], and missing baseline comparisons. The author rebuttal responds to each of these points very clearly and convincingly, as well as with new experimental baseline comparisons that clearly demonstrate the effectiveness of the CPR approach. I encourage the authors to include the extensive comparison with [Mizadeh, 2020] provided in the rebuttal, especially given the similarity to the proposed approach. and to also tone down the strong claims of novelty in light of the similarities.

Decision: Accept (Poster)