Nonconvex Continual Learning with Episodic Memory
Authors: Seungyub Han
Yeongmo Kim
Jungwoo Lee
Venue:  (2021)
Keywords: continual learning
nonconvex optimization

Abstract: Continual learning aims to prevent catastrophic forgetting while learning a new task without accessing data of previously learned tasks. 
The memory for such learning scenarios build a small subset of the data for previous tasks and is used in various ways such as quadratic programming and sample selection. 
Current memory-based continual learning algorithms are formulated as a constrained optimization problem and rephrase constraints as a gradient-based approach.
However, previous works have not provided the theoretical proof on convergence to previously learned tasks.
In this paper, we propose a theoretical convergence analysis of continual learning based on stochastic gradient descent method.
Our method, nonconvex continual learning (NCCL), can achieve the same convergence rate when the proposed catastrophic forgetting term is suppressed at each iteration.
We also show that memory-based approaches have an inherent problem of overfitting to memory, which degrades the performance on previously learned tasks, namely catastrophic forgetting.
We empirically demonstrate that NCCL successfully performs continual learning with episodic memory by scaling learning rates adaptive to mini-batches on several image classification tasks.  

Under review as a conference paper at ICLR 2021
NONCONVEX CONTINUAL LEARNING WITH EPISODIC MEMORY
Anonymous authors Paper under double-blind review
ABSTRACT
Continual learning aims to prevent catastrophic forgetting while learning new task without accessing data of previously learned tasks. The memory for such learning scenarios build small subset of the data for previous tasks and is used in various ways such as quadratic programming and sample selection. Current memory-basee based continual learning algorithms are formulated as a constrained op- timization problem and rephrase constraints as gradient-basee ever, previous works have not provided the theoretical proof on convergence previously learned tasks. In this paper, we propose a theoretical convergence analysis of continual learning based on stochastic gradient descent method. Our method, nonconvex continual learning (NCCL), can achieve the same convergence rate when the proposed catastrophic forgetting term is suppressed at each iteration. We also show that memory- based approaches have an inherent problem of over- fitting to memory, which degrades the performance on previously learned tasks, namely catastrophic forgetting. We empirically demonstrate that NCCL success- fully performs continual learning with episodic memory by scaling learning rates adaptive to mini-batches on several image classification tasks.
approach. How-
INTRODUCTION
Learning new tasks without forgetting previously learned tasks is a key aspect of artificial intelli- gence to be as versatile as humans. Unlike the conventional deep learning that observes tasks from an i.i.d. distribution, continual learning train sequentially model on non-stationary stream of data (Ring) 1995 Thrun 1994p. The continual learning Al systems struggle with catastrophic forgetting when the data acess of previously learned tasks is restricted (French & Chater 2002).
To overcome catastrophic forgetting, continual learning algorithms introduce a memory to store and replay the previously learned examples (Lopez-Paz & Ranzato 2017 Aljundi et al. 2019b
Chaudhry et al., 2019a), penalize neural networks with regularization methods (Kirkpatrick et al, Zenke et al. (017), use Bayesian approaches (Nguyen et al.] 2018 Ebrahimi et al. 2020), and other novel methods (Yoon et al. 20188] Lee et al., 2019). Although Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato 2017) first formulated the continual learning constrained op- timization problem, the theoretical convergence analysis of the performance of previously learned tasks, which implies a measure of catastrophic forgetting, has not been investigated yet.
Continual learning with episodic memory utilizes small subset of the data for previous tasks to keep the model staying in a feasible region corresponding moderate suboptimal region. GEM-based approaches use the rephrased constraints, which are inequalities based on the inner product of loss gradient vectors for previous tasks and a current task. This intuitive reformulation of constrained optimization does not provide theoretical guarantee to prevent catastrophic forgetting. In addition, the memory-based approaches have the critical limitation of overfitting to memory. Choosing the perfect memory for continual learning is an NP-hard problem (Knoblauch et al., 2020). then the inductive bias by episodic memory is inevitable. This problem also degrades the performance on previously learned tasks like catastrophic forgetting but has not been discussed quantitatively to analyze backward transfer (BWT).
In this paper, address the continual learning with episodic memory smooth nonconvex finite- sum optimization problem. This generic form is well studied to demonstrate the convergence and complexity of stochastic gradient methods for the nonconvex setting (Zhou & Gu 2019 Lei al.

---

Under review as a conference paper at ICLR 2021
2017 Reddi al., 2016 Zaheer al. 2018). Unlike the convex case, the convergence is gener- ally measured by the expectation of the squared norm of the gradient E| f= The theoretical complexity is derived from the e-accurate solution, which is also known stationary point with E| [x]]] < 6. We formulate the proposed continual learning algorithm as a Stochastic gradient descent (SGD) based method that updates both previously learned tasks from episodic memory and the current task simultancously. By leveraging the update method, we can introduce a theoretical analysis of continual learning problems.
We highlight our main contributions as follows.
We develop convergence analysis for continual learning with episodic memory
We show the degradation of backward transfer theoretically and experimentally as prob- lems of catastrophic forgetting and overfitting to memory.
We propose a nonconvex continual learning algorithm that scales learning rates based on sampled mini-batch
1.1
RELATED WORK
The literature in continual learning can be divided into episodic learning and task-free learning. Episodic learning based methods assume that training model is able access clear task bound- aries and stores observed examples in the task-wise episodic memory (Lopez-Paz & Ranzato [017] Chaudhry al., 2019a). On the other hand, Al system experiences arbitrarily shifting data streams, which we are not able to access task boundaries in the real world. Task-free continual learning studies the general scenario without the task-boundary assumption. Aljundi al. (20T9a) introduces Memory-aware Synapses (MAS) and applies learning protocol without waiting until task finished. Furthermore, the following work (Aljundi et al.) 2019b) adopt the memory system of GEM selecting observed examples to store for preventing catastrophic forgetting.
Smooth nonconvex finite-sum optimization problem has been widely employed to derive the theo- retical complexity of computation for stochastic gradient methods (Ghadimi & Lan 2013 et al. 2017 Zaheer et al., 2018 Reddi et al. 2016). Unlike the convex optimization, the gradient based algorithms are not expected to converge to the global minimum but are evaluated by measur- ing the convergence rate to the stationary points in the nonconvex case. The complexity to reach stationary point is a key aspect of building new stochastic gradient method for nonconvex op- timization. In constrast with general optimization, memory-basee continual learning methods have limited data pool for previously learned tasks, which causes overfitting problem to memory. Knnollauch et al., (020)) found that optimal continual learning algorithms and building a perfect memory is equivalent. Furthermore, the authors proved that these two problems Na-hard. The theoretical result shows that overfitting to memory is inevitable.
Lei
2 PRELIMINARIES
We consider a continual learning problem with episodic memory where a learner can access the boundary between the previous task and the current task. The continuum of data in (Lopez-Paz & Ranzato 2017) is adopted as our task description continual learning. First, we formulate our goal the smooth nonconvex finite-sum optimization problems with two objectives,
min F(x) = fx) + g(x) = nt Σ f((z) ZERd
9 (x)
18g
where z E Rd is the model parameter, each objective component fz). 9, (,(x) is differentiable and nonconvex, and ny, ny are the numbers of components. We define two different components of the finite-sum optimization objectives from a sample i of previously learned tasks fi(x) and sample j of the current task gj (z).
Unlike the general stochastic optimization problem, we assume that the initial point in continual learning is e-accurate solution of f(x) with E|| V < for some < 1. By the property of nonconvex optimization, we know that there might exist multiple local optimal points that satisfy moderate performance the previously learned task (Garipov al. 2018). This implies that the
2

---

Under review as a conference paper at ICLR 2021
model parameter stays in the neighborhood of x0 or usually moves from initial local optimal point xd to the other local optimal point at the f-th iteration, x' over iterations of a successful continual learning scenario.
The continual learning algorithm with an episodic memory with size m cannot access the whole dataset of the previously learned tasks with n samples but limited samples in the memory when a learner trains on the current task. This limited access allows us to prevent catastrophic forgetting partially. However the fixed samples from memory cause biased gradient and the overfitting prob- lem. In Section 3 we provide the convergence analysis of the previously learned tasks (x), which are vulnerable to catastrophic forgetting.
We denote (z) as the component, which indicates the loss of sample from the previously learned tasks with the model parameter and (z) as its gradient. We use 4t, J7 as the mini-batch of samples at iteration t and denote b/ by as the mini-batch size [Le), [Je] for brevity throughout the paper. We also note that Oj from the current task holds the above and following assumptions.
To formulate the convergence over iterations, we introduce the Incremental First-order Oracle unit of cost by sampling the pair
(IFO) framework (Ghadimi & Lan 2013), which is defined fi For example, a stochastic gradient descent algorithm requires the cost as much as the batch size b, at each step, and the total cost is the sum of batch sizes Σt= by Let T(e) be the minimum number of iterations to guarantee e-accurate solutions. Then the average bound of IFO
complexity is less than or equal to T(() by-
To analyze the convergence and compute the IFO complexity, we define the loss gap between two local optimal points A as
= f(=)) inf f('), US1S7"
which might be much smaller than the loss gap of SGD. Suppose that the losses of all optimal points have the same values, i.e., f(x²) )(²)), then we have A, S O. This implies that A is not reason for moving away from a stationary point of f. which we will explain details in Section B
We also define vy.0g for f. g, respectively, the upper bounds on the variance of the stochastic gradients of given mini-batch. For brevity, we write only one of them of
by Vf((t) vf(=)
jf = sup
(3)
bf
Throughout the paper, we assume the L-smoothness.
Assumption Sa is L-smooth that there exists constant L > 0 such that for any xr y E Rd.
[V)x( Vf((i)ll s L| - yll
(4)
where ||-|| denotes the Euclidean norm. Then the following inequality directly holds that
L f()) (V)))),
L
2 1 - 3
y11².
In this paper, consider the framework of continual learning with episodic memory. By the sumption of GEM, we assign each task sample from i.i.d. distribution within its episode to the same memory budget m. In the learning phase task E 21,2..-. ,K) we sample batch with size ny from memories of all task with size [m (k- - 1)].
3 NONCONVEX CONTINUAL LEARNING
In this section, we present the convergence analysis of continual learning in the nonconvex setting. The theoretical result shows why catastrophic forgetting occurs in view the nonconvex optimiza- propose the Mon-Convex Continual Learning (NCCL) algorithm,
tion problem. As result, where the learning rates for the previously learned tasks and the current tasks are scaled by the value of the inner product by their gradients for the parameter in Section 3.3
3

---

Under review as a conference paper at ICLR 2021
3.1 ONE EPISODE ANALYSIS
The key element behind preventing catastrophic forgetting is to use gradient compensation on the training step of the current task. It can be considered as an additive gradient, in turn, applied to the gradient of the current task, although GEM (Lopez-Paz & Ranzato (017) uses the quadratic programming and EWC KKirkpatrick et al.) (017) introduces the auxiliary loss function. First, we present the proposed gradient compensation, which uses samples of the episodic memory for single new task episode. We define the gradient update
(6) where 31,, are learning rates scaled by the sampled mini-batches for H, = 1, U Jr and V Vin, (x²), Vgn, (x²) are the estimates of the gradient V [a(x",, Vg(a')) respectively. Equation 6 implies that the parameter is updated on the current task g with gradient compensation on pre- viously learned tasks f by aH, fI, (x*). Our goal is explain the effect of the gradient update 3H, Vg,, (x") on the convergence to stationary points of f(z) and observe the properties of the expectation of each element over It. For iteration E [1, T] and constant L, we define the catas- trophic forgetting term C₃ to be the expectation in terms of Vg,,(²"):
x'+1 = =xx= aH, ff, - 3H, (z",
1 L
C E
[V,r, (=²)11 - Bir, (V)())) Vyu, (19)
which we derive in Appendix
A
We temporally assume the following to show the convergence
analysis of continual learning.
Assumption Suppose that the episodic memory M contains the entire data points of previously learned tasks [k - 1] on the k-th episode and replays the mini-batch I M. Then V f1, (x") is an unbiased estimate that Ble,] = 0 for et = ff. (x') f(xx))
In the next section, do not Assumption and investigate the biasedness of the episodic memory M that causes the overfitting on memory. Our first main result is the following theorem that provides the stepwise change of convergence of our algorithm.
Theorem Suppose that La oZ, S ? ffr some > O and O., S 2. Under Assumption we have
-__ (BIf(=") f(a CC) + aHH L 2bg 03
E \((')) -
(8)
Han.
aH.
We present the proof in Appendix A. Note that the catastrophic forgetting term C2 exists, unlike the general SGD, and this term increases the IFO complexity. Fortunately, we can tighten the upper bound of Equation (8) by minimizing C.. Now we telescope over single episode for the current task. Then we obtain the following theorem.
Theorem 2 Let OHH = =a=
/ for some > O and all t. E [T] and 1 a 1 > 0 for some A.
Under Theorem 1 have
1 (
( A Σ
T-1
Le 2b
min E(V VT
C
+
of
This theorem can explain the theoretical background of catastrophic forgetting. The cumulative summation of catastrophic forgetting terms C,, increases drastically over iterations. This fact implies that the stationary point x can diverge. An immediate consequence of Equation 9 is that we can consider the amount of catastrophic forgetting as an optimization--ieve the additive catastrophic forgetting term. Theorem is equivalent to the result for SGD with fixed learning rate (Ghadimi & Lan| 2013). Similar to SGD, the upper bound of Equation can be made O A + C.) when we assume that Lec
factor. Without
2b, oj = 0(1).
Conversely, we consider the convergence analysis of g(x:) by changing roles for f and g in Theorem
2 In the very beginning of iterations, A, is dominant in Equation | and its catastrophic forget- Tin term Gt,g with regard to Vfr, (xr) is relatively small because z is the neighborhood of the
4

---

Under review as a conference paper at ICLR 2021
stationary point. When we consider Assumption 2 and the samples from previously learned tasks are constantly provided, the norm of gradients fl. (a'))l is bounded. Therefore, g(x) can reach a stationary point by the same rate SGD. However, We cannot access the full dataset of previously learned tasks because of the setting of continual learning. There exists an extra term that interrupts the convergence of g(rr), which is called the overfitting. We now ignore the extra term to conjecture that || V9,, (x)) is at least bounded. Then we have the following corollary.
Corollary Let the expected stationary of g(z) be 0 (T) for a constant 8 0 and the upper bound of learning rate for g(z) be 3 > 0. The cumulative sum of the catastrophic forgetting term C' is ((855V/)) Nonconvex contimual learning by Equation (66 does not converge as iterating the algorithm for the worst case, where min Bi] ff(x")) is ((32d) < PöVT. When 3²8 < VT' we have
t
min [(T\" (
(10)
Then, the IFO complexity for achieving an e-accurate solution of f(z) is 0 (1/2)-
We would like to emphasize that catastrophic forgetting is inevitable in the worst case scenario be- cause the stationary of ((x) is not decreasing and the convergence on f(x: cannot be recovered no matter how long we proceed training. Building a tight bound of C' is the key to preventing catas- trophic forgetting. Note that the general setting to minimize C' is scaling down the learning rate 3 to 328 < 1/VT. Then we have the decreasing C = O(1/VT). However, this method is slowing down the convergence of the current task g(æ) and not an appropriate way. The other option is to minimize C itself rather than tightening the loose upper bound O(328VT). We discuss how to minimize this term by scaling two learning rates in Section 3.3 The constrained optimization problem of GEM provided useful rephrased constraint but cannot explain and guarantee the catastrophic forgetting in the nonconvex setting. Our convergence analysis of continual learning is the first quantitative result of catastrophic forgetting in the manner of nonconvex optimization.
3.2 OVERFITTING TO EPISODIC MEMORY
In section discussed the theoretical convergence analysis of continual learning for smooth non- convex finite-sum optimization problems. The practical continual learning tasks have the restriction on full access to the entire data points of previously learned tasks, which is different from Assump- tion 2 The episodic memory with limited size [M] incurs the bias on V fx("). Suppose that we sample a mini-batch of previously learned tasks from episodic memory M. Then we can formulate
this bias E[eM] as
[leM] = (V/, - V/M((')-
Vf(x))
(11)
This equation shows that the bias depends on the choice of M. In the optimization, the bias drag the convergence of f(z) to fz). This fact is considered as the overfitting to the memory M. (Knoblauch et al. 2020) prove that selecting a perfect memory is hard. We can conclude that EleM] # O. Now we extract the overfitting bias on M from the ignored element in Equation 21 at Appendix A and the catastrophic forgetting term in Equation 7 The bias related term Bi is added to the upper bound of Equation Iaaiiionnaan gand reformulates the catastrophic forgetting term to prac- tical catastrophic forgetting term C2 as
Bi ((*)))) fM vf((')) + SR, (VMM(=') - Vf(a'), (x),
(12)
Br, L - (V.. VgJ. (2'))
C = 2
(13)
Note that the upper bound of C₁ is the same as C, even if we modify it to the version with the limited memory size scenario. The cumulative sum of Bi, over iterations is the amount of disturbance by overfitting to memory. This inherent defect of a memory-based continual learning framework can be considered generalization gap phenomenon Keskar al. (2016), and small mini-batch size can resolve this problem. In Section 4 we demonstrate the effect of different mini-bant batch sizes to alleviate the overfitting problem on the memory M.
5

---

Under review as a conference paper at ICLR 2021
(x)
(x
XPuc
Xpuc
as. Fi, (xf)) By, gi
9 Jepo (x²)
Xp x
Xp x
Bac Vii (x²) + Bug Biises (x.)
4 Itneg (t
Figure I: Geometric illustration of Non-Convex Continual Learning (NCCL). In the continual learn- ing setting, the model parameter starts from the moderate local optimal point for the previously learned tasks zpp. Over T' iterations, we expect to reach the new optimal point 2puc which has good performance on both previously learned and current tasks. In f-th iteration, the model param- eter x² encounters either VgJ.,pos (x") or VeJ,,neg(x"), These two different cases indicate whether (z*), VgJ, (x")) is positive or not. To prevent x' from escaping the feasible region, i.c., catas- trophic forgetting, we propose the theoretical condition on learning rates for f and g.
3.3 SCALING LEARNING RATES
The result of convergence analysis provides simple continual learning framework that only scales two learning rates in the gradient update of Equation As we proved in the above, we should tighten the upper bound of Ö, to prevent catastrophic forgetting. We propose an adaptive scaling method for learning rates that can minimize or reduce Č₁ in the both case of (V fr, (x"), VgJ, (z,,(x")) < 0 O. We note that Equation i3 is quadratic polynomial of 8H, where S1r, > 0. First, we can solve the minimum of the polynomial on By, when (Vfr, (I"), Vg,, (x")) > 0. By differentiating on 3H. can easily find the minimum Oj and the optimal learning rate Bir
and fV (x"), (x'))
37. (V (x), (x)))
(V,,, (z')
O;=
(14)
2L Vgn. (x']
A direct consequence Cj < implies that the optimal catastrophic forgetting surprisingly helps fx) to decrease the upper bound of stationary. For (V (x²), (x')) 0, however, Su. should be negative to achieve the global minimum of Öž, which violates our assumption. Instead, we propose a surrogate of Vg,,(*')
V S1, (xt) fi (2*))l Vg
(o', (a),
(15)
The surrogate borrows the gradient Vír, (x*) to cancel out the negative component /r, (x*) from reduce the catastrophic forgetting term drastically by boosting learning rate
VgJ, (z*). Now aH, without correcting V.g,, (z*) directly. The remaining non-negative value of C₁ is caused by the magnitude of VJJ, (9),(x*) itself. This phenomenon cannot be inevitable when we should learn the current task for all continual learning framework.
We summarize our results as follows.
(11 (V (w'), Vov.
(V), Vgh. (x)) <
OH,
(16)
(x') Vg.. (x))) >
aa
V (x'),, (x) 20
8H.-
Von (x)) L [1
[V VgJ, (x*)) > 0
(17)
We derive the details of the result in this section in Appendix B The existing GEMebased al- gorithms have only focused on canceling out the negative direction of (. from Vg,, (x') with the highly computation cost for the only case (Vir, VgJ, (x²)) S 0. The proposed

---

Under review as a conference paper at ICLR 2021
Algorithm Nonconvex Continual Learning (NCCL)
Input: K task data stream (Di, Dx] initial model x0, memory (M& with each size m
for to K do
for 1 0 to 7 do
Uniformly sample a mini-bater I C. [m (k 1)] with [1 = by Uniformly sample a mini- mati-batch Jr C DR with [Jal = by and store Jr into Ma Compute learning rates 0'H,, 3r, with V (x*), (g,(('))
x'+1 f- r² aH, f1,, (x.) BH, Vg,,(x) end for
xº f- x--1
end for
method has the advantage over both leveraging C₁ achieve the better convergence for the case (V (x"), (x)) > 0 and even reducing the effect of catastrophic forgetting by the term fR,"" VV, (2²) for the case (v'r, gg,, (x")) < 0 Figure || illustrates intuitively how scaling learning rates achieve the convergence to a mutual stationary point x puc as we proved the theoretical complexity in Corollary
EXPERIMENTS
Based on theoretical analysis of continual learning, evaluate the proposed NCCL model in episodic continual learning with 3 benchmark datasets. We run our experiments on GPU server with Intel 19-9900K, 64 GB RAM, and 2 NVIDIA Geforce RTX 2080 Ti GPU.
4.1 EXPERIMENTAL SETUP
Baselines. We compare NCCL to the following continual learning algorithms. Fine-tune is a basic baseline that the model trains data naively without any support, such as memory. Elastic Weight Consolidation (EWC) (Kirkpatrick et al. 2017) uses the regularized loss by Fisher Information. Reservoir Sampling (Chaudhry al.) 9b) show that simple experience replay can be power continual learning algorithm. It randomly selects a fixed number of examples from the stream of data tasks, which is similar with GEM and A-GEM. GEM and A-GEM Lopez-Paz, & Ranzato Chaudhry et al. (2019a) is the original and variant of Gradient Episodic Learning.
(207));
Datasets. We use the following datasets. Kirkpatrick et al. (2017) design Permuted-MNIIST MNIST (LeCun et al., 1998) based dataset, where apply fixed permutation of pixels to trans- form data point to make the input data distribution unrelated.
2) Zenke et al. (2017) introduce Split-MNIST dataset, which splits MNIST dataset into five tasks. Each task consists of two classes, for example (1, 7), and has approximately 12K images. 3) Split-CIFAR10 is one of most commonly
used continual learning datasets based on CIFARIO 10 dataset (Krizhevsky al. 2009), respectively
et 2020 Rebufñi et al. 2017 Zenke et al. 2017 Lopez-Paz & Ranzato 2017 Aljundi et al. 2019b).
Zenke
Training details. with ReLU activation. For CIFARI0 datasets, use a smaller viersion of ResNet18 from the setting in GEM. To show the empirical result of our theoretical analysis, we apply vanilla SGD into all train networks.
We use fully-connected neural networks with two hidden layers of [100, 100]
Performance measurement. We conduct our experiment on K tasks. We evaluate our exper- iments by two measures, ACC and BWT. ACC is the average test accuracy of all tasks after the whole learning is finished. Backward Transfer (BWT) is measure of for forgetting, which shows how much learning new tasks has affected the previously learned tasks. When BWT < 0, implies that catastrophic forgetting happens. Formally, we define ACC and BWT as
K ACC =
K Σ
Σ
ACCARS
BWT = K
ACC - ACCKA.
(18)
K
k=1
k=
where ACC,,j is the accuracy of task at the end of episode j.

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison ACC and BWT on Permuted-MNIST, Split-MNIST, Sp on
epochs per task over runs. For A-GEM, we report the result in (Chaudhry et al. 2019a
Permuted-MNIST
Split-MNIST
Split-CIFAROO
Method
ACC (%)
BWT (%)
ACC (%)
BWT (%)
ACC
BWT (%)
Finue-tune
2.43
12.10
19.31
13.00
18,08
9.22
EWC
68.30
0.29
19.80
4,20
42.40
0.26
Reservoir Sampling
10.01
1.00
43.82
44,00
GEM
89.50
0.06
92.20
61.20
0.06
A-GEM
89.10
0.06
93.10
62.30
0.07
GSS
77.30
84.80
33.56
NCCL (ours)
68.52
0.22
63.26
0.33
23.11
0.21
Table 2: Comparison on ACC of Permuted-MNIST with 5 permutation tasks on single epoch per task with multiple choices of hyperparmeters over runs. We define m as the memory budget for each task, by as a batch size for the current task, by as a batch size for single previous task.
m
bg
by
ACC (%)
m
b9
by
ACC (%)
500
200
10
77.02
200
200
75.82
500
200
20
73.95
200
200
20
74.98
500
200
50
71.78
200
200
50
72.35
4.2 RESULTS
Table and Table show our main experimental results. We explain the property of Split dataset first. Split dataset divide the whold dataset by the number of tasks, so get partial version of dataset. For example, 5 Split-MNIST, we can consider the number data points per task as the call single epoch of SpLit-MNIST datapoints for a task. We conduct experiments on 20 Permuted-MNIS' 5-Split MNIST, and 5-Split CIFAR10. We can notice that NCCL. does not outperform GEM and A-GEM. We conjecture that the reason of the lower performance is the differences of optimization techniques for new task. GEM- based methods apply the quadratic programming algorithm to continual learning, which spends more iterations to find a better surrogate for the negative direction between the previous task and the current task, but this procedure requires the very longer computation time which is not effective. We also expect that the theoretical convergence analysis for GEM surrogates can be achieved in future work. Compared to other reported methods, the performance of NCCL has a reasonable result. By these observations, we conclude the followings.
number of 0.2 epoch. Then,
5 repeated sets of its
Our theoretical convergence of analysis is reasonable for explaining catastrophic forgetting.
NCCL has both theoretical and empirical supports.
We observe that the small mini-batch size from memory is effective.
5 CONCLUSION
In this paper, we have presented the first generic theoretical convergence analysis of continual learn- ing. Our proof shows that training model can circumvent catastrophic forgetting by suppressing the disturbance term on the convergence of previously learned tasks. We also demonstrate theoretically and empirically that the performance of past tasks by nonconvex continual learning with episodic memory is degraded by two separate reasons, catastrophic forgetting and overfitting to memory. To tackle this problem, nonconvex continual learning applies two methods, scaling learning rates adaptive to mini-batches and sampling mini-batches from the episodic memory. Compared to other constrained optimization methods, the mechanism of NCCL utilizes both positive and negative di- rections between two stochastic gradients from the memory and the current task to keep stable performance on previous tasks. Finally, it is expected the proposed nonconvex framework helpful to analyze the convergence rate of other continual learning algorithms.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11254-11263, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, pp. 11816- 11825, 2019b.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Tomr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 10486, 2019b.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Mareus Rohrbach. Uncertainty-guide continual learning with bayesian neural networks. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Robert M. French and Nick Chater. Using noise to compute error surfaces in connectionist net- works: A novel means of reducing catastrophic forgetting. Neural Compulation, 14(7):1755- 1769, 2002. doi: 10. 10.1162/000 166/0099976002662289000 URL. https:: org.
089976602600222
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 31, pp. 8789- 8798. Curran Associates, Inc., 2018. URL http: http::///wapprr..pps..c.cceo
pdf
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochas- tic programming. SIAM Journal Optimization, 23(4):2341-2365 2013.
Saeed Ghadimi and Guanghui Lan. Accelerated gradient methods for nonconvex nonlinear and stochastic programming. Mathematical Programming, 156(1-2):59-99, 2016.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 04836, 2016.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Baabbskk--aaiiskaa,..a et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-333 2017.
Jeremias Knoblauch, Hisham Husain, and Tom Diethe. Optimal continual learning has perfect memory and is hardard. arXiv preprint arXiv:2006.09 08.888 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio. and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2322 1998.
Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overeoming catastrophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 312-321, 2019.

---

Under review as a conference paper at ICLR 2021
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Lihua Lei, Cheng Ju, Jianbo Chen, and Michael Jordan. Non-convex finite-sum optimization via
Scsg methods. In Advances in Neural Information Processing Systems, pp. 2348-2358, 2017.
David Lopez-Paz. and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 0467-6476 2017.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner Variational continual learn- ing. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Sylvestre- Aelvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg
Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Sashank Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. In International conference on machine learning, pp. 314 323, 2016.
Mark B. Ring. Continual learning in reinforcement envirommnns PhD thesis, University of Texas at Austin, TX, USA, 1995. httt:://ddnb.inf//999...
Sebastian Thrun. A lifelong learning perspective for mobile robot control. In Intelli- gent Robots and Systems, Selections of the International Conference on Intelligent Robots and Systems 1994. IROS 94, Mumich, Germany, 12-16 September 1994, pp. 201-214, 1994. doi:
URL https:///c:: :ttt:s:://ci.org/10... ooo.ooogg.11...
0978-0444822005555015
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive meth- ods for nonconvex optimization. In Advances in neural information processing systems. pp. 9793- 9803, 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings of machine learning research, 70:3987, 2017.
Dongruo Zhou and Quanquan Gu. Lower bounds for smooth nonconvex finite-sum optimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7574-7583, Long Beach, California, USA, 09-15 Jun 2019, PMLR.
10

---

Under review as a conference paper at ICLR 2021
APPENDIX
A THEORETICAL ANALYSIS
Proof of Theorem We analyze the convergence of nonconvex continual learning with episodic memory here. Recall that the gradient update is the following
net x² -an,
an, VIn, (a') - 3n, VgJ, (x*)
for all t E {1, T. by applying Equation 5
Since assume that f. is L-smooth, we have the following inequality
f(x s f(x)) (Vf(''), - x²) + L -1 x2
= -(x') (VS(2') an, (z) + Br, Vgs, + + lan, Vir,(x') + BH, Van, (x')) S f(x') an (V/(2') f., (HH (Vf(x'), (gy)
L 2 VVr, (z 2
L 3M [[,,
(19)
2
Let (x') (z²) and define
Č = マタ s/. (g),((")1 - (Vf(', (z))
2
fort 2 1. We have
f(x s f(x') -HH, (V fxx),, (x'() + L a. fv,,) - Ee
L a || (aH, - LV ex) + L
f(xx') (HH.
2 a + C..
2
Taking expectations with respect to I on both sides, noting that
C== = =[CC]
.e obtain
L 2 (IV[(1))) S S(x') - 1(a--1) (an, Lain (E(()((()))))))))))))))))))))))) ex)) + - a ||e/11 2 + E(C,) OHH -aam,
2
$ - -((x-1)) + C₁ + 2 oH ||e 1|2 + (La - an, E. ex))
Rearranging the terms and assume that La
oHH S y and a. > 0. have
L
2 a ||e
|[ < an, (1 fan)
(f(x') - f(x + C. + (La an. EV(f(( (x))) +
4
2 a ||ea
L
an (1 aam) (s(x') - 1(x + C. + EVV +
2 La..
(20)
Note that under Assumption E((V)(x*)) 0. we conclude
1
1 La 1e
|V <
4 1 fan.) (f(x*) f(x
+ C.) +
(21)
a1
1 4 5aH,
Furthermore, the batch size b
Proof of Theorem
Suppose that the learning rate aH, constant a c/VT, for c 0. a 1/4 0. Then, by summing Equation 21 from = 0 to T have

---

Under review as a conference paper at ICLR 2021
1 T
Vf
-
E
V f2²)|?
t=
T-1
1
f(z))-
- -f(x²)
Σ
Ci)
+ a00 ao3
I
L
2bg
t=0
T-1
1
1
A +
Σa
Σ Ci
Le
ga
CVT
2b
2b,
T-1
Le
A
+
C2
of
VT
c
2bF
t=
Lemma Let a constant 0 and an upper bound B BH, I.
The sum of the catastrophic
forgetting term over iterations T Ei C. is For S VT. we have O(1).
Proof
The upper bound of the catastrophic forgetting term is
L 3
C = E
VVg. (z 3H. (V)(''), (en
2
L
E
Vig, (x²)] 3n. BI
O (E [[Vg. (x*) (x))
Since
V9 (1 < [g(<')) VVg. (x') - g(x0)112
V || y
where og is analogous Equation Bnnd by is the mini-batch size of g. Then we have
C = O (BIVg(x'))1
628
where E [T] and for some
0. Summing over time t, we have
C Σ C, T
T-1
8 VT
0 (BPOVT).
1=0
Therefore, we obtain O(1) when 388T 1.
Proof of Corollary
To formulate the IFO calls, let T(e)
T(e) = min (T: min E V S c).
Recall that E 2 0 I by Theorem
Then by Lemma we have
328VT 0(828).
min E [
VT
It implies that min B|IV/(>*)) is not decreasing when 1 388T. Then, xf cannot reach the stationary point.
On the other hand, fxx can be converged to the stationary point when $28 <
V such that
1
1
min Vz -(()))))))))))))))))))))))))))))
(22)
/T
12

---

Under review as a conference paper at ICLR 2021
To derive bound for T(c), note that
1 Sc
VT
Then we have
T(e) 0
The IFO call is defined E/e by,t- Therefore, the IFO call is (
B
DERIVATION OF EQUATIONS IN SECTION 3
Proof of Equations
15
Let the surrogate (ir)("")
(z,
Vfr
(o', (x')
(+")|l Vo)
(23)
[+
Then, have
18,, L
G = E
(19) (x') Von. (
19HLL L
(VVgr
(T (x'), Fg
(x'),,),, Vys, ( Bn. (x'), Ve,,(21)
E
2
fu. (z)))
[V'11('')1'
1891 L
(z', "x,,
E
(["))(
BH. ((VI (x'), Vgn( - (V Vgu.
S4 (x:)]
19,, L
(V. (
(24)
2
[[V]]
13


Reviews:
[{"review_id": "mIB18bDk8Jv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper takes an interesting nonconvex optimization perspective on the continual learning problem. More specifically, the authors pose continual learning with episodic memory as a smooth nonconvex finite sum problem. They then consider the requirements for a theoretical proof of convergence to a stationary point for previously learned tasks. This results in the proposed NCCL method that leverages these ideas to modulate learning rates for the current and previous tasks to prevent escape from the feasible region. Overall, the strength of this paper is its theoretical analysis and I find the idea of connecting continual learning with the associated nonconvex optimization problem compelling. I am not an expert in nonconvex optimization, but my understanding is that the analysis itself is not that unique for the field. Rather, what is novel is the interesting application of the ideas to the continual learning problem. I find the theoretical aspect of this paper strong, but still lean towards rejection in its current form as I am very skeptical that the idea is at all validated by the experiments. This potentially suggests that the theory may lack relevance in these domains. \n\nThere are some comparisons to baselines and prior work that I found a bit questionable. On the bottom of page 6, the authors state that existing GEM based algorithms only focus on canceling the negative direction, but actually maximizing transfer even when gradient dot products align was explored in [1]. The authors also suggest in section 4.2 that, despite worse empirical results, the NCCL approach is superior to GEM because of its inefficient quadratic program computation. However, this was already addressed in A-GEM [2], so it is not so clear that there is a significant computational advantage to NCCL. I would think that the authors should actually compare compute times inline with prior work. I also am almost 100% sure that the comparison to reservoir sampling is incorrect. If you look at results in [1] and [3] you see that reservoir sampling consistently performs right around GEM and sometimes better than GEM on exactly these same benchmarks. The 10% number seems unfathomable to me and at the very least needs an explanation about how this could be true. \n\n[1] \"Learning to Learn Without Forgetting By Maximizing Transfer and Minimizing Interference\" Riemer et al., ICLR 2019. \n[2] \"Efficient Lifelong Learning With A-GEM\" Chaudhry et al., ICLR 2019.\n[3] \"On Tiny Episodic Memories in Continual Learning\" Chaudhry et al., 2019. \n\nThis last point is related to my biggest overall concern, which is that it is not clear that the learning rate weighting scheme proposed in this work actually helps in comparison to generic replay. For example, it would be a really important ablation to try the very same buffer setup but with no learning rate modulation. My experience leads me to believe that the gap between the GEM based approaches and NCCL is likely larger than the gap between these approaches and vanilla replay. As a result, I am very skeptical that the learning rate modulation component adds value based on the current results. Additionally, it would be very interesting to look deeper into how the model is working to understand its effect on learning. For example, the authors should detail patterns with the chosen modulated learning rates over time. \n\nWhile I appreciate the theoretical analysis of this paper, I think the experiment section is too short and leaves many important questions unexplored. Unfortunately, I feel that I must support rejection of this paper in its current form as my doubts about the experiments leave me unsure that the approach works at all in practice. \n\nAfter The Rebuttal: I really appreciate the author response and it is a shame that the revisions do not seem to be correctly uploaded. Unfortunately, the responses to my comments rely heavily on references to the revision that I cannot see, making it impossible for me to validate if my concerns were actually adequately addressed. The other reviewers have mentioned some very valid concerns about the submitted draft as well. As such, I continue to lean towards rejection of the submitted paper as significant revisions are certainly needed. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "kJWBlPQDLB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of paper**\n\nThis paper analyses the convergence of episodic memory-based continual learning methods by looking at it as a nonconvex optimisation problem. They analyse the convergence rates for the case where all memory from past tasks is stored, and then consider the case where there is only a subset of past data, leading to overfitting on the episodic memory. They then introduce a method that scales the learning rates of the their update method, with the goal of tightening the bound obtained in the convergence analysis. Finally, experiments are shown on different benchmarks, and the proposed method is compared to some competing baselines.\n\n**Summary of review**\n\nI am recommending rejecting this paper. Although the goal of the paper is commendable (convergence analysis for nonconvex episodic memory-based continual learning), I feel like there are many parts of the paper that can be improved (see later in the review).\n\n**Pros of paper**\n\n1. The paper attempts to analyse the convergence of continual learning methods theoretically (especially Section 3.1). This is very important to do, so that we can understand the problem of nonconvex continual learning better. This has not been attempted enough in the literature, partly because this is a very difficult problem. \n2. The work appears to be well-positioned with related work on convergence rates (as far as I am aware).\n3. The paper builds nicely, from Introduction to Preliminary Work to Theoretical Results to Experiments.\n\n**Cons of paper (/questions for the authors)**\n\n4. Although the aim of the paper is great, it appears to me as if the methods the paper mentions are not instances of the update that the paper analyses (Equation 6). Specifically, GEM and EWC (mentioned in the first paragraph of Section 3.1): GEM has a different optimisation technique (quadratic programming algorithm), and EWC does not store any episodic memory (only stores previous model parameters).\n5. I am struggling to see the significance of Section 3.2 (\"Overfitting to Episodic Memory\"). It appears like the authors are just pointing out that there is a bias introduced by storing only a subset of past data, without sufficiently commenting on the effects or significance of this bias.\n6. Appendix A (proof of Theorem 1) is incomplete.\n7. Something seems wrong to me with the BWT metric in Section 4.1:\na) My own experience with Fine-tune and EWC strongly suggests that both methods should have BWT<0. This is because the methods first learn the task well and then forget it slowly over time, and is fully expected from such algorithms. However, the authors report BWT>0.\nb) Fine-tune on Permuted-MNIST (Table 1) has an ACC of 2.43% but a BWT of 12.10%. Surely, be definition, BWT<=ACC always (Equation 18)?\nc) A final point on BWT: A BWT<0 does not \"imply that catastrophic forgetting happens\" (final paragraph page 7). Although it does imply *forgetting*, this is not necessarily *catastrophic forgetting*, which is only when BWT is extremely negative. For example, the concept of *graceful forgetting* will still have BWT<0 (but is usually distinguished from catastrophic forgetting).\n8. Can the authors comment on why the proposed method performs better with 1 epoch per task than with 5 epochs per task (Tables 1 vs 2, Permuted-MNIST)? This result appears to indicate that, despite the correction terms of the method, the method is forgetting tasks as it trains for longer.\n\n**Additional (minor) feedback**\n\n9. I would strongly recommend proof-reading the paper (or else asking a native English speaker to do so).\n10. Figure 1 is a nice sketch visually, but I did not see how it shows the benefit/key idea of NCCL specifically (which is about finding optimal learning rates). There is no visual/diagramatic element of how those learning rates might be chosen. (Alternatively put, a similar figure could be used to describe eg GEM).\n\n**Update to review**\n\nThanks to the authors for responding. They did clear up point 5 (above) for me. However, I shall keep my score of 4. Unfortunately I cannot see the new revision of the paper that the authors refer to, meaning I cannot change my score. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "bvYymw5lZQc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to provide a convergence analysis for nonconvex continual learning with episodic memories, and try to theoretically show the degradation of backward transfer caused by  overfitting to memorized samples.  It further proposes an algorithm for learning rate scheduling in  nonconvex continual learning based on these results.\n\nThe reason of the score of the paper is that the theoretical proof is wrong in my understanding and cannot support the main contribution claimed in this paper,  the main problems are as below.  \n\nThe proof of the main theorems is questionable regarding the nonconvex assumption, which is the most important contribution claimed in this paper.  Regarding the inequality in  Eq.(5),  in my understanding it is hold to be true only when f is a convex function [1].  And the theorems are based on this inequality which cannot be hold  for nonconvex case if this inequality is not true for nonconvex functions.  If I'm wrong, authors please provide proof of how to get Eq.(5) by L-smooth nonconvex functions.  \n\nMoreover, in the proof of Theorem 1, Eq.19  (Appendix A), the inequality of the last step cannot be hold unless the inner product of gradients < \\Delta f,  \\Delat g > is always positive, which cannot be guaranteed. Otherwise, there is no reason to develop gradient-based approaches in continual learning, such as  GEM [2] or AGEM [3].  So even if Eq.(5) can hold for nonconvex case, the theorem is still questionable. Therefore, the main claim of this paper is highly suspicious to me.  If authors cannot clarify these issues, this paper would be considered as with significant flaws. \n\nDespite the questions on the main theorem, the assumption of the initial state of the model is quite strong as it assumes the initial values of parameters are close to the optimal values, which is not very practical unless a pre-trained model is applied.  So the  significance of this paper is further limited.  \n\nAs the theoretical part is incorrect, I haven't reviewed the experiments part of this paper. If the authors can clarify all above main concerns, I'm willing to make another round of review. \n\n[1] Nesterov, Yurii. \"Introductory lectures on convex programming volume i: Basic course.\" Lecture notes 3.4 (1998): 5.\n[2] Lopez-Paz, David, and Marc'Aurelio Ranzato. \"Gradient episodic memory for continual learning.\" Advances in neural information processing systems. 2017.\n[3] Chaudhry, Arslan, et al. \"Efficient lifelong learning with a-gem.\" arXiv preprint arXiv:1812.00420 (2018).\n\n############################feedback to authors' response#############################\n\nI'm aware of the non-convex setting is valid, but since the corrected proof of the theorem is not uploaded, I will raise my score to 3. \n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WMUXQh87kP2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors provide theoretical justifications for memory-based continual learning (CL) methods and provide a scaling learning rate method NCCL to improve the practical performance. The results look quite exciting (there is quite scant theoretical paper for CL), however, after looking into the details of the paper, I was confused by many places and would say the authors need to further improve their manuscript in order to qualify for the ICLR standard.\n\n1. The theoretical analysis is not very impressive. The theory just split out the catastrophic forgetting term C and demonstrated that performance degradation depends on C. However, where C comes from (I know it is an additional term directly from mathematical derivation, but what's the meaning and intuition) is not clearly discussed. Also, the theorem based on the unrealistic assumption e_t is unbiased (Assumption 2), which can never happen in memory-based CL methods. The authors do mention approaches such as NCCL without Assumption 2, but no theory is provided. Probability section 3.2 is on theory without Assumption 2, then please provide a complete theorem instead of just waving hands.\n\n2. Moreover, there are many flaws in the proof, I just list a few of them here (or correct me if I misunderstand). \n- In proof of Theorem 1, second inequality of eq (19), why does the cross product term disappear? i.e., why $||\\nabla f + \\nabla g||^2 <= ||\\nabla f||^2 + ||\\nabla g||^2$?\n- why $C_t = E_I (\\tilde{C}_t)$, when taking an expectation over $I_t$? $C_t$ is defined in eq (7), and there is no randomness over $J_t$ (already with $E_J$). But $E_I (\\tilde{C}_t)$ still has randomness over $J_t$.\n- why $E||e_t||^2$ is written as $||e_t||^2$, we also have randomness in $e_t$ over $I_t$, see definition of $e_t$.\n- In proof of Lemma 1, why $E(||\\nabla f||) = O(E(||\\nabla g||))$ or how do we get the second equality?\n- How do we get the relation of $E||\\nabla g||^2 = O(\\beta^2\\delta / \\sqrt{T})$? I see it is directly assumed in Corollary 1 (expected stationary of $g$ be $\\delta/\\sqrt{T}$). But I think we should derive this instead of simply making an assumption. Actually, $f$ and $g$ are equivalent and interchangeable, if we assume $g$ already converge, does that mean $f$ also assumed converge? But if we directly apply results derived from $f$ this will be circular reasoning. So I am not sure, the authors better make more discussions on this.\n\n3. For practical performance, if we compare NCCL (68.52 accuracies in Table I) with GEM (89.50), A-GEM (89.10), GSS (77.30), or even EWC (68.30), there is no performance improvement at all. The authors further claim their methods are faster in computation, then please also include a time comparison, instead of just mentioning it. Otherwise, it is hard to quantify the contribution of the new method.\n\nOverall speaking, I am afraid that such work does not have sufficient theoretical or algorithmic contributions. And I doubt the true value of designing a new method without any performance improvement. However, I still appreciate the motivation of the paper and will be more tolerant since there are quite scant theory papers for CL. So I would be happy to adjust my rating if all my concerns were properly addressed. If there is any misunderstanding, please also let me know.\n\nupdate: Thanks for the response. However, there is no updated revision in the revision history of this paper. Based on the flaws that I have previously pointed out, it is impossible for me to validate if my concerns were actually adequately addressed without seeing the updated version. I will keep my score unchanged. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes to analyse convergence of episodic memory-based continual learning methods by looking at this problem through the lense of nonconvex optimisation. Based on the analysis a method is proposed to scale learning rates such that the bounds on the convergence rate are improved.

Pros:
- I agree with the reviewers that this is an interesting and novel perspective on continual learning

Cons:
- Reviewers point out concerns/issues with the clarity of the manuscript with respect to several parts: 
- reviewers raise concerns with respect to the significance of the evaluation 
- reviewers point out that the theoretical analysis itself is somewhat standard and not novel in itself, and 2 reviewers raise concerns with respect to the analysis made

Unfortunately the authors seem to have missed the upload of the revised version. The reviewers have nevertheless considered the rebuttal by the authors and the consensus is that this manuscript is not ready yet in it's current form. 
Decision: Reject