On the Convergence of Nonconvex Continual Learning with Adaptive Learning Rate
Authors: Seungyub Han
Yeongmo Kim
Jungwoo Lee
Venue: ICLR 2022 Submitted (2022)

Abstract: One of the objectives of continual learning is to prevent catastrophic forgetting in learning multiple tasks sequentially.
The memory based continual learning stores a small subset of the data for previous tasks and applies various methods such as quadratic programming and sample selection.
Some memory-based approaches are formulated as a constrained optimization problem and rephrase constraints on the objective for memory as the inequalities on gradients.
However, there have been little theoretical results on the convergence of continual learning.
In this paper, we propose a theoretical convergence analysis of memory-based continual learning with stochastic gradient descent.
The proposed method called nonconvex continual learning (NCCL) adapts the learning rates of both previous and current tasks with the gradients.
The proposed method can achieve the same convergence rate as the SGD method for a single task when the catastrophic forgetting term which we define in the paper is suppressed at each iteration.
It is also shown that memory-based approaches inherently overfit to memory, which degrades the performance on previously learned tasks. Experiments show that the proposed algorithm improves the performance of continual learning over existing methods for several image classification tasks. 

Under review as a conference paper at ICLR 2022
ON THE CONVERGENCE OF NONCONVEX CONTINUAL LEARNING WITH ADAPTIVE LEARNING RATES
Anonymous authors Paper under double-blind review
ABSTRACT
One of the objectives of continual learning is to prevent catastrophic forgetting in learning multiple tasks sequentially. The memory based continual learning stores small subset of the data for previous tasks and applies various methods such as quadratic programming and sample selection. Some memory based approaches are formulated as a constrained optimization problem and rephrase constraints on the objective for memory as the inequalities on gradients. However, there have been little theoretical results on the convergence of continual learning. this paper, we propose a theoretical convergence analysis of memory-based continual learning with stochastic gradient descent. The proposed method called nonconvex continual learning (NCCL) adapts the learning rates of both previous and current tasks with the gradients. The proposed method can achieve the same convergence rate the SGD method for single task when the catastrophic forgetting term which we define in the paper is suppressed at each iteration. It is also shown that memory-based approaches inherently overfit memory, which degrades the performance on previously learned tasks. Experiments show that the proposed algorithm improves the performance of continual learning over existing methods for several image classification tasks.
INTRODUCTION
Learning new tasks without forgetting previously learned tasks is a key aspect of artificial intelligence to be as versatile as humans. Unlike the conventional deep learning that observes tasks from an i.i.d. distribution, continual learning train sequentially a model on non-stationaay stream of data (Ring) 1995] Thrun 1994). The continual learning AI systems struggle with catastrophic forgetting when the data access of previously learned tasks is restricted (French Chater 2002). To overcome catastrophic forgetting, continual learning algorithms introduce novel methods such as replay memory to store and replay the previously learned examples (Lopez-Paz & Ranzato 2017 Aljundi
al. 2019 Chaudhry et al. (2019a), regularization methods that penalize neural networks (Kirkpatrick ct al.
2017 Zenke et al.] (011)). Bayesian methods that utilize the uncertainty of parameters or data points (Nguyen al. 2018 Ebrahimi al. 2020), and other recent approaches (Yoon al. 2018
Lee et al. 2019). In this paper, we focus on the online continual learning with replay memory. The learner stores small subset of the data for previous tasks into memory and utilizes the memory by replaying samples to keep the model staying in a feasible region corresponding to moderate suboptimal region. Gradient episodic memory (GEM) (Lopez-Paz & Ranzato 2017) first formulated the replay based continual learning as a constrained optimization problem. This formulation rephrases the constraints on objectives for previous tasks as the inequalities based on the inner product of loss gradient vectors for previous tasks and current task. However, the theoretical convergence analysis of the performance previously learned tasks, which implies measure of catastrophic forgetting, has not been rigorously studied in the literature. Without convergence analysis, this intuitive reformulation constrained optimization does not provide theoretical guarantee to prevent catastrophic forgetting.
Nonconvex finite-sum optimization problem offers a solution to analyze catastrophic forgetting by measuring the convergence of previously learndd tasks, which is related to the performance. Most deep learning problems are defined as nonconvex optimization, and the target objective composed of the of objectives for each data point. Now we express continual learning problem of the

---

Under review as a conference paper at ICLR 2022
form
min f(x) = 1 Σ M(x); =ER
(1)
where we assume that each objective f1) with a model I and data point i is nonconvex with Lipschitz gradient. Here, we expect that a stochastic gradient descent based algorithm reaches a stationary point instead of the global minimum in the nonconvex optimization. This generic form well studied demonstrate the convergence and complexity of stochastic gradient methods for nonconvex setting (Zhou & Gu. 2019; Lei et al., 2017; Reddi al., 2016a; Zaheer al. 2018). Unlike the convex case, the convergence is generally measured by the expectation of the squared norm of the gradient E (+))) The theoretical complexity is derived from the e-accurate solution, which is also known as a stationary point with V ff (z) I c. Suppose we divide the entire sum of objectives into two terms for previous tasks and current tasks, and measure the convergence on each term. Then, we can observe the transition of convergences on the previous and current tasks respectively while learning sequentially from data stream. We consider this transition of convergence on the previous task as catastrophic forgetting if EE| ([)Vfp(r)) fP))]] with the set of data points from previous tasks 1' increases over iterations.
In this work, we formulate continual learning problem as nonconvex finite-sum optimization with stochastic gradient descent algorithm that updates both previously learned tasks from replay memory and the current task simultaneously, and present theoretical convergence analysis of continual learning problems by leveraging the replay-based update method. This extends the continual learning algorithm such as ER-Reservoir (experience replay with reservoir sampling) with fixed and same learning rates on both two tasks to our adaptive method which controls the relative importance between tasks at each step with theoretical guarantee. In addition, the replay-based continual learning has the critical limitation of overfitting to memory, which also degrades the performance of previously learned tasks like catastrophic forgetting by interference. It is known that choosing the perfect memory for continual learning to prevent catastrophic forgetting is an NP-hard problem by (Knoblauch et al.) 2020). We present that the inductive bias by replay memory, which prevents perfect continual learning is inevitable in view of optimization.
2 BACKGROUNDS
The continual learning algorithm with a replay memory with size m cannot access the whole dataset of the previously learned tasks with ny samples, but uses limited samples in the memory when learner trains on the current task. This limited access does not guarantee to completely prevent catastrophic forgetting, and causes the overfitting problem with the biased gradient on a memory. In Section 3 we provide the convergence analysis of the previously learned tasks f(x), which are vulnerable to catastrophic forgetting.
As we denote fz(z) as the component, which indicates the loss of sample from the previously learned tasks with the model parameter x, we define (z) as its gradient. We use L,, Jr as the mini-batch of samples at iteration / and denote the mini-batch size [T, and |J&] as by; by throughout the paper. We also note that 9, (x), which denotes the loss for the current task and will be defined in Section 3.1 satisfies the above and following assumptions.
To formulate the convergence over iterations, we introduce the Incremental First-order Oracle (IFO) framework (Ghadimi & Lan 2013), which is defined as a unit of cost by sampling the pair fV., f(x)). For example, a stochastic gradient descent algorithm requires the cost as much as the batch size be at each step, and the total cost is the sum of batch sizes Σ by Let T(e) be the minimum number of iterations to guarantee c-accurate solutions. Then the average bound of IFO complexity is less than or equal to E/14) by-
To analyze the convergence and compute the IFO complexity, define the supremum of loss gap between a local optimal point z and the global optimum x²' as
A f = supf(=") ((x=))
Suppose that sup f(x") is same with f(=), then we have Ay = O, which might be much smaller than the loss gap of general SGD. Without the continual learning scenario, a general nonconvex SGD
2

---

Under review as a conference paper at ICLR 2022
updates the parameters from randomly initialized point, which is highly likely to have the large loss f(x"). Then, Ay > 0 is the key constant to determine the IFO complexity for convergence as Ay is in the numerator of Equation [T] However, continual learning algorithm has already converged to local optimal point zº for the previous task f(x) and might get a much smaller A than the general SGD. It means that Ay for nonconvex continual learining in Equation 11 dose not have large impact on the IFO complexity. To generalize the theoretical result, we define the worst local minimum to explain the upper bound of convergence rate in Equation 2 This implies that A, is not critical reason for moving away from stationary points of by catastrophie forgetting, which we will explain in Section
We also define of and 0g for f and g, respectively, the upper bounds the variance of the stochastic gradients of a given mini-batch. For brevity, we write only one of them f.
sup
Σ -V(o)) x ff 1=1
(3)
Throughout the paper, we assume the T-smoothness. Assumption 1. fo is L-smooth that there exists a constant >> O such that for any 2, y E Idd,
[v) - V(((v)ll s L||z - -yll
(4)
where ||-|| denotes the Euclidean norm. Then the following inequality directly holds that
L x v12 = f((x) f(()) (.))))) v)
L
y1
(5)
2
We derive the inequality in Appendix B
With Assumption 1 successfully handle individual
noncovex objectives for each data point. In the next section, we investigate nonconvex continual learning with adaptive learning rates overcome catastrophic forgetting.
NONCONVEX CONTINUAL LEARNING
We first present theoretical convergence analysis of memory based continual learning in nonconvex setting. We use the convergence rate of stochastic gradient methods, which denotes the IFO complex- ity to reach e-accurate solution for smooth nonconvex finite-sum problem (Reddi et al.) (016a). This generic form enables both deep learning and optimization communities to formulate various accelerated gradient methods with theoretical guarantee. We seek understand why catastrophic forgetting happens in terms of the convergence rate, and propose non-convex continual learning (NCCL) algorithms with theoretical convergence analysis.
3.1 PROBLEM FORMULATION
Given two finite sets P and C' at the initial time step t = O, we let two sets denote the sets of indices for previously learned data points and upcoming data points, respectively. Note that the task description for a continual leamner is two separate sets. In this section, analysis of the model parameter that we have trained on P and starts to learn C. Thus, we simply denote data stream of continual learning two consecutive sets P and C.
will show convergence
We consider our goal as a smooth nonconvex finite-sum optimization problem with two objectives
min F(z) f(x) )(x) FERS
Σ f((x) ny iEP
1
1 Σ Eg((x,
Th jEC
where (x( and g; (x) denote the objectives of data points i E P and j E C, respectively. addition, ny and ny the numbers of elements for P and C. To ease exposition, use different notation gj (x) for a data point j E C, which is usually the same objective function for a data point E P.
To formulate theoretical convergence analysis of continual learning, consider replay memory based method of which memory is subset of P U C. Let a random variable M, C P U C' be the replay memory time step E [0, 7), whose union of the form M == U, M2 We focus both the
3

---

Under review as a conference paper at ICLR 2022
episodic memory and the replay memory with dropping rule. The episodic memory based methods include GEM (Aljundi et al. 2019). A-GEM (Chaudhry et al. (019a), and ORTHOG-SUBSPACE (Chaudhry et al. (020). ER- Reservoir (Chaudhry al. (2019b] is a replay memory based method with dropping rule, which replaces the dropped sample d E M, with a sample in the stream for C.
We now define the gradient update of continual learning
x'+1 x² an, V fi, BH, Vg),
where 4e C. M, and Jr C. C' denote the mini-batcher from the replay memory and the current data stream, respectively. Here, H2 is the union of 1, and Jr The adaptive learning rates of f. (x*) and Vg,, (a*) denoted by aH,, and 8H, which are the functions of H,. Strictly speaking, the mini-batch Ig from Mj might contain a datapoint d E C' for ER-Reservoir. We describe the details of the problem in Appendix and assume that the notation Ie indicates subset of P for convenience. is a generalized version of continual learning algorithms, which is our novelty to prove the convergence rates in the nonconvex setting for the proposed method, A-GEM, and Reseevrrv later.
Equation 7
3.2 MEMORY-BASED NONCONVEX CONTINUAL LEARNING
Unlike the conventional smooth nonconvex finite-sume sum problem where each mini-batch iid-sampled from the dataset P U C, the replay memory based continual learning encounters a non-iid stream of data C and has access to a small sized memory M. Algorithm [ presents the pseudocode of NCCL. by Equation 7 By the limited access to P. the gradient update for fz) in Equation 7 estimate of the gradient V fa²). Specifically, at the timestep t,
is biased
(z') - Er, (z,((= = E1, f(x') + M ff(²') + EM,
where e, and eM, denote the error term Sr, (a*) - f((()) and the expectation value over It given Mr. respectively. We note that the given replay memory M, with small size at timestep induces the inevitable overfitting bias. We first state an intermediate result for a single gradient update of NCCL. For case of exposition, we define the overfitting term B, and the catastrophic forgetting term I as follows:
B. (La a..)) + B.. (Vg,,
3..LL L V99, 8H, a.. (x"), Vgn,
r, 2
where T. is a constant for Lipschitz smoothness. Under Assumption
a single gradient update by
Equation satisfies the following bound by letting x <- x²+11 and y x²:
I (an. 2 -0, |V S f(x') - f(x + +T, + B, 2 a 1eo1..
L
(8)
This reveals the basic qualitative difference between the conventional nonconvex SGD and NCCL in the convergence rate. Compared to the nonconvex SGD, there exist two terms B, and I, in Equation We group the terms containing 6, as B, and the other terms as Ft, We note that the catastrophic forgetting term F has (V fI, (xr) VgJ, (a')), which is the key aspect of interference and transer (Riemer et al.] 2018), and B, includes the error term between the batch of M and the entire dataset P. Then, we can quantify the amount of overfitting by tracing B..
g.
To compute the expectation over the stochasticity of NCCL, we derive the expectation of V / (a*) over the sampling rule of M,. The episodic memory M, MD for all is uniformly sampled once from the random sequence of P, and ER-reservoir iteratively samples the replay memory M, by the selection probability (MM[|M,-1))
Lemma 1. Let the history of M, be Maa:) = (Mo.... ... (M)) If Mo is uniformly sampled from P. then both episodic memory and ER-reservoir satisfies
E EM
[ (x*) = Vs(x²) and Empey [eM.] = 0.
We provide the detailed proof in Appendix B Note that taking expectation iteratively with respect to the history Mo is needed to compute the expected value of gradients for M,. Since taking the expectation over the stochasticity of NCCL implies the total expectation
E
EMpea [E, [E,, [- [7)] [Mxs]]
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Nonconvex Continual Learning (NCCL)
Input: Previous task P. K task data stream (Di,-.. DR initial model z², memory Mo C for to K do
for = 0 to - do
Uniformly sample mini-batch T C M2 with [T. by Uniformly sample a mini-batch J Dx with [ = bg Compute learning rates aH,, and 3H, with V fr, (x*) and (x²)
x'+1 f x² a., V fi, (x') 3μ. VgJ,
Store Jr into Mc+1 by the rule of replay memory scheme
end for 2 T-
ト PUU,
Mo t Mr-1 end for
we know that [[B,) is also O. We note that considering random choice of Mo allows us to analyze the convergence on f f(r), even if Mo cannot access the all datapoints in to compute || (Vf((s f(x')l² Our first main result is the following lemma that provides the stepwise change of upper bound.
Lemma 2. Suppose that Assumption holds and O<aH, S t. Then for xf updated by Algorithm we have the following bound
1 = aH. (1 { (f(x*) f((0(t)()))))))))))))))))))))))- + B, +T) + an I 2(1 2 aa.
of
(10)
where the effect of B, vanishes by TE] B
The proof is presented in Appendix B Surprisingly, taking the expectation over Mo P in Equation to handle the stochasticity of choosing M, allows us to analyze the convergence of f(x) in Section
We also note that the individual trial with randomly given Mo cannot cancel the effect by B,, although its total expectation over the whole possible trials B. is zero. More specifically, the worst case sup Mr V (=])l definitely contains non-zero value B, in the upper bound term. We discuss the more details of the overfitting to memory in Appendix D
3.3 THEORETICAL CONVERGENCE ANALYSIS
We now describe our convergence analysis of the stochastic gradient descent by Equation 7 Without any continual learning scheme, the model z' eventually converges M U C, not PU C. A local optimal point n for P at time step / = 0, which we have already trained before the continual learning step starts, loses the performance on P very quickly over time. This phenomenon is called catastrophic forgetting. We also note that the overfitting to replay memory is another reason of degrading performance for each trial when the model trains on samples in M with replacement after learning on P for previous tasks. We obtain the convergence rate of NCCL to stationary point of as the following theorem.
Tv and E (0.... ,, T Theorem 1. Let on, =a vJ for some < D- iterates of NCCL satisfy
1}. By Lemma the
A
T-1
Le + Le 2 20
EinE\
A
Σ
[T
(11)
VT
where 1 Lo/2).
For completeness we present a proof in Appendix A. The loss gap AS and the variance of gradients o are fixed values. We first note that there exists the cumulative sum of ET] unlike the convergence rate of SGD. Most of novelty in analysis lies in dealing with Σ B[[T.] The ET.)/V
5

---

Under review as a conference paper at ICLR 2022
not guaranteed converge to 0. This fact gives rise to catastrophic forgetting in terms of the nondecreasing upper bound.
Before showing bound IFO calls made by NCCL, we present the convergence rate for g(x)). To simplify the proof, assume that learning rates o SH, same fixed value 3 c VT. The assumption is reasonable, because we can observe that RHS of Equation is not perturbed drastically by the learning rates with small value 0 an, Br,, S 2 < 1. Then have the following lemma.
Lemma 3. Suppose that I J4 00 and the datapoints E M P P use the same functiio
ga Sa. Taking expectation over It C. M2 and Je C C, we have
24
min E Vg(''(>
T og:
(12)
where Ag and 0g is the version of loss gap and the variance for on M UCC, respectively. In fact, we note that the convergence rate of g on M U.C. Despite the convergence rate is not on C. also converges to C trivially.
Now we can get the upper bound of Σ [[[ by Lemma 3
Then we have the following main result.
Corollary 1. The IFO complexity of Algorithm to obtain c-accurate solution is:
((1/e)), = O(T)
IFO calls
(13)
The proofs of both Lemma 3 and Corollary Mre presented in Appendix B Here, we denote co as the impossibility of convergence. This result reveals that catastrophic forgetting is inevitable without minimizing the cumulative sum Σ ETy)). With replay memory, we know that the inner product in D be negative by the result of A-GEM (Chaudhry et al.) 2019a). This observation shows that there exists the chance that Ir becomes negative, and E[T can converge to a constant ((1) as + co. Now have the proper convergence rate of f(x), and the model keep the performance after learning on C. On the other hand, when awz = O for all time step. E[T;] {] is always larger than Intuitively, this case implies SGD on C' without the replay memory M or other supports, and the cumulative sum Σ E[T r] monotonically increases over time. Then, f==") diverges as t + co. We emphasize that the reason of catastrophic forgetting can be explained remarkably by the nonconvex optimization perspective.
P
The optimization problem nonconvex continual learning. The local optimal point x² after learning on C' might be different from .²", because x' moves towards x xuuc as described in Figure The above observation motivates the following formulation continual learning to induce Σ E to converge t > and keep e-accuracy of f during T iterations. More specifically, solve the following problem over T:
T-1 Σr. T
minimize an Sy, 1=
subject to 0 a. SH, 2/L for all t T.
(14)
Recall that we have the condition of two learning rates to prove Theorem D and Lemma condition needs to be constraints of the above optimization problem.
This
Finally, we note that our result first shows a theoretical convergence analysis of memory based nonconvex continual learning in the setting of Corollary [ method by Problem 14 with theoretical guarantee.
More precisely, we can now explain this
3.4 ADAPTIVE LEARNING RATES
As discussed in Section 3.3 the minimization problem 14 leads to tight the upper bound of conver- gence rate in nonconvex setting. The result of convergence analysis provides a simple continual learning framework that only adjusts two learning rates in Equation [ First, we review A-GEM
and other methods with fixed learning rate such as GSS (Aljundi et al. 2019) and ER-Resservoir (Chaudhry et al. 2019b) through result.

---

Under review as a conference paper at ICLR 2022
A-GEM (Chaudhry al. 2019a) propose surrogate of Vgg, (2') as the following equation avoid violating the constraint when VgJ, (x")) S 0:
(fi( Vii,, van,
(g),
We can interpret the surrogate as boosting the learning rate an, to cancel out the negative com- ponent of fr, (a') on VgJ, (x"). After applying the surrogate, E[T's] is reduced but still non- negative, which we show in Appendix A-GEM also fails to fully use the advantage of the case (V (z)), Vg,, (x")) > 0.
Fixed learning rates imply that building replay memory the key success continual learning. It means that this memory setting is an abundant pool to sample It, which satisfies (Vfr, VgJ, (x*)) > O, as much as possible. This can reduce [[
However, two above methods cannot enjoy the case (V,,,z²), Vg,, (z')) > 0 enough. As discussed above, we note that H, is a quadratic polynomial of Br,, where 3y, > O. We can solve the minimum of the polynomial on 81r, when (V (ar),, VgJ, (x²)) >> 0. By differentiating on BH,, we can easily find the minimum i and the optimal learning rate Pi.
9/. (1 on, LVVVI, (x²), Vga, (a"))
r (1 on, L)(V Sr, (x) 2L VZ.1V (21)
Vl (2))
We integrate the above result to propose better nonconvex continual learning algorithm with the theoretical guarantee. More specifically, we propose an adaptive learning rate method that can reduce Σ E['r] in both cases of (V fí, (x'), Vg., (x')) < 0 and (Vfi, (z)), Vgs, (x')) 0. Then, the
proposed adaptive learning rate scheme is as follows.
a(1 (f', (r'))
ff,, Vg.. (x')) <
OH,
(
(15)
a
(z') VgJ. (2²)) >
[a. (V- (n."" (g),
Vf,,(x'') VgJ, (x")) <0
BR.-
(16)
(V (x'), VgJ, (x")) >
We derive the details of the above result in Appendix C In addition, our result shows that A-GEM can be viewed as adaptive learning rate scheme using samples in M, directly. This fact implies that there no difference between A-GEM and experience replay methods. Figure Iillustrates intuitively how scaling learning rates achieve the convergence to mutual stationary point PuC as we proved the theoretical complexity in Corollary [
4 EXPERIMENTS
We evaluate the proposed NCCL and the state of the art online continual learning basclines on the following benchmarks. We report detailed experimental materials and results Appendix.
Datasets. We demonstrate the experimental results on standard continual learning benckmarks: Permuted-MNISS (Kirkpatrick al.) (017) is a MNIST (LeCun et al.) 1998) based dataset, where each task has fixed permutation of pixels and transform data points by the permutation to make each task distribution unrelated. MNIIS-MNIST (Zenke al., (017)) splits MNIST dataset into five tasks. Each task consists of two classes, for example (1, 7). (3, 4), and has approximately 12K images. Split-CIFARII F10 and 100 also split CIFAR-10 and 100 datasets (Krizhevsky et al. 2009) into five tasks and 20 tasks, respectively.
Baselines. We report the experimental evaluation on the online continual setting which implies a model is trained with single epoch. We compare with the following continual learning baselines. Fine-tune a simple method that a model trains observed data naively without any support, such replay memory. Elastic weight consolidation (EWC) is regularization based method by Fisher Information (Kirkpatrick et al. 2017). ER- Reserverr chooses samples to store from a data stream with probability proportional to the number of observed data points. The replay memory returns
7

---

Under review as a conference paper at ICLR 2022
random subset of samples each iteration for experience replay. ER-Reservoir (Chaudhry et al. 2019bb shows a powerful performance in continual learning scenario. GEM and A-GEM (Lopez- Paz & Ranzato 2017: Chaudhry et al.) 2019a) use gradient episodic memory to overcome catastrophic forgetting. The key idea GEM is gradient projection with quadratic programming and A-GEM simplifies this procedure. We also compare with iCarl, MER, ORTHOG-SUBSPACK (Chaudhry ct al. 2020), and stable-SGD (Mizzadeh et al. 2020)..
Architecture and Training detail. For fair comparison, we follow the commonly used model architecture and hyperparameters Lee et al. (2020); Chaudhry et al. (2020). For PerMuted-MNIST and Split-MNIST, we use fully-connected neural networks with two hidden layers of [400, 400 or [256, 256] and ReLU activation. ResNet-18 18 with the number of filters n 64, 20 (He et al. 2016) applied for Split CIFAR-10 and 100. All experiments conduct a single-pass over the data stream. is also called epoch or 0.2 epoch (in the case of split tasks). We deal both cases with and without the task identifiers in the results of split-tasks to compare fairly with baselines. Batch sizes of data stream and memory are both 10. All reported values are the average values of 5 runs with diffrent seeds, and we also provide standard deviation. Other miscellaneous settings are the same as in Chaudhry et al. (2020). We implement the baselines and the propose method Tensorflow 1. For evaluation, an NVIDIA 2080ti GPU along with 3.60 GHz Intel 19-9900K CPU and 64 GB RAM.
4.1 RESULTS
The following tables show main experimental result, which is averaged over runs. We denote the number of example per class per task at the top of each column.
Table 1: Multi-headed apit-CIFARROO, reduced size Resnet-18 n, 20. Accuracy and forgetting results.
memory size
1
Method
memory
accuracy
forgetting
accuracy
forgetting
EWC
X
42.7 (1.89)
0.28 (0.03)
42.7 (1.89)
0.28 (0.03)
Fintune
X
40.4 (2.83)
0.31 (0.02)
40.4 (2.83)
0.31 (0.02)
Stable SGD
X
59.9 (1.81)
0.08 (0.01)
59.9 (1.81)
0.08 (0.01)
A-GEM
V
50.7 (2.32)
0.19 (0.04)
59.9 (2.64)
0.10 (0.02)
ER-Ring
56.2 (1.93)
0.13 (0.01)
62.6 (1.77)
0.08 (0.02)
ER-Reservoir
46.9 (0.76)
0.21 (0.03)
65.5 (1.99)
0.09 (0.02)
ORTHOG-subspace
58.81 (1.88)
0.12 (0.02)
64.38 (0.95)
0.055 (0.007)
NCCL + Ring NCCL Reservoir
54.63 (0.65)
0.059 (0.01)
61.09 (1.47)
0.02 (0.01)
52.18 (0.48)
0.118 (0.01)
63.68 (0.18)
0.028 (0.009)
Table 2: Permuted-MNIs (23 tasks 60000 examples per task), FC-1256,256).
Accuracy and
forgetting results.
memory size
1
5
Method
memory
accuracy
forgetting
accuracy
forgetting
multi-task
X
83
83
Fine-tune
X
53.5 (1.46)
0.29 (0.01)
47.9
0.29 (0.01)
EWC
X
63.1 (1.40)
0.18 (0.01)
63.1 (1.40)
0.18 (0.01)
MER
/
69.9 (0.40)
0..400.01)
78.3 (0.19)
0.06
A-GEM
/
62.1 (1.39)
0.21 (0.01)
64.1 (0.74)
0.19 (0.01)
ER-Ring
70.2 (0.56)
0.12 (0.01)
75.8 (0.24)
0.07 (0.01)
ER-Reservoir
68.9 (0.89)
0.15 (0.01)
76.2 (0.38)
0.07 (0.01)
ORHOG-subspace
84.32 (1.10)
0.12 (0.01)
84.32 (1.1)
0.11 (0.01)
NCCL + Ring
/
74.22 (0.75)
0.13 (0.007)
84.41 (0.32)
0.053 (0.002)
NCCL+Reservoir
/ 79.36 (0.73)
0.12 (0.007)
88.22 (0.26)
0.028 (0.003)
8

---

Under review as a conference paper at ICLR 2022
Overall, NCCL variants outperforms baseline methods especially in the forgetting metric. Our goal is to demonstrate the usefulness of the adaptive learning rate scheme to reduce the catastrophic forgetting, and verify the proposed theoretical convergence analysis. We remark that our adaptive learning rates successfully suppress forgetting by large margin compared to baselines. Note that NCCL also outperform A-GEM, which does not maximize transfer fr, (x*), VgJ, (x²)) 0. Now, we can empirically demonstrate our theoretical guaranteed method by minimizing Σ I' is valid.
We clipped SH,, increase the performance. As we discussed earlier, we can prevent forgetting when (VJr, (x"), VgJ, (x<)) > O. However, we observe that Vi, [a²)l2 suddenly increases because of the interference at the previous step
The very large learning rate 3H, by the increased fi, (x')ll can force the model to fall into an arbitrary point that is likely to increase the loss of f. Clipping the learning rate reduces this problem and still has the effect of reducing the catastrophic forgetting term r. By the property of quadratic polynomial, the catastrophic forgetting term is negative because the clipped value is smaller than the original learning rate.
We show that NCCL is potentially powerful alternative for continual learning. Even with tiny replay memory, NCCL still performs better than some baselines. We note that NCCL shows the best performance on the forgetting metric. It implies that NCCL prevent catastrophic forgetting more efficiently than others by minimizing the catastrophic forgetting term in the proposed optimization problem. However, the accuracy is slightly lower than other baselines, which include experience replays. The purpose of our adaptive learning rate scheme is to prevent catastrophic forgetting, so the performance of current task is slightly lower than ER-Ring, stable-SGD, and ORTHÖG--uuspaed This result shows that the plasticity to learn new task is restricted by NCCL variants with tiny memory. In particular, we would expect that NCCL would benefit from the additional enhancements in ORTHOG-SUBSPACE and stable SGD by introducing their techniques. In appendix, we add more results with larger sizes of memory, which shows that NCCL outperforms on the average accuracy. We conclude that the transfer effect by the small size of memory for NCCL is less effective.
5 RELATED WORK
Chaudhry et al. 2019a Riemer et al. 2018 Chaudhry et al. 2019b) to the nonconvex optimization problem to provide the theoretical guarantee of previous methods. Our problem setting is related to the theoretical convergence analysis of smooth nonconvex optimization. Smooth nonconvex finite-sum optimization problem has been widely employed to derive the theoretical complexity of computation for stochastic
We extend the memory based continual learning [opez- & Ranzato 2017
gradient methods (Ghadimi & Lan 2013 2016 Lei et al. 2017 Zaheer et al. 2018 Reddi et al., 2016a). Unlike the convex optimization, the gradient based algorithms are not expected to converge to the global minimum but are evaluated by measuring the convergence rate to the stationary points in the nonconvex case. Prior works have only focused analyze the problems with two objectives where the additional term is for regularization (Yao & Kwok 2016). Here, we instead focuses on the continual learning problem, which has two objectives for both datapoints of the previous task and the current task. This setting is interesting since we can observe the convergence of each task respectively. Otherwise, (Knoblauch et al., 2020) first develops a theoretical approach on continual learning by set theory, and shows that optimal continual learning algorithms and building a perfect memory is equivalent. This is analogous to our result that we cannot handle the overfitting bias for each trial of continual learning.
6 CONCLUSION
In this paper, we have presented the first generic theoretical convergence analysis of continual learning. Our proof shows that a training model can circumvent catastrophic forgetting by suppressing the disturbance term the convergence of previously learned tasks. We also demonstrate theoretically and empirically that the performance of past tasks by nonconvex continual learning with replay memory is degraded by two separate reasons, catastrophic forgetting and overfitting to memory. To tackle these problems, nonconvex continual learning uses two methods, scaling learning rates adaptive to mini-batches and sampling mini-batches with small size from the replay memory. Finally, it is expected the proposed nonconvex framework is helpful to analyze the convergence rate of other continual learning algorithms.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning, Advances in Neural Information Processing Systems, pp. 11816- 11825, 2019.
Arslan Chaudhry, Mare Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny. Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint
2019b.
Arslan Chaudhry, Nacemullah Khan, Puneet Dokania, and Philip Torr. Continual learning in low-rank orthogonal subspaces. Advances in Neural Information Processing Systems, 33, 2020.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Mareus Rohrbach. Uncertainty-guidel continual learning with bayesian neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Robert M. French and Nick Chater. Using noise to compute error surfaces connectionist networks: networs:: A novel means of reducing catastrophic forgetting. Neural Computation, 14(7):1755-769,, 2002.
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2366 2013.
Saeed Ghadimi and Guanghui Lan. Accelerated gradient methods for nonconvex nonlinear and stochastic programming. Mathematical Programming, (56(12):559999 2016.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barkkkskaaainka,,,.a et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Jeremias Knoblauch, Hisham Husain, and Tom Diethe. Optimal continual learning has perfect memory and is np-hard. arXiv preprint arXiv:2006.051 08006 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-232) 1998.
Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 312-321, 2019.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Lihua Lei, Cheng Ju, Jianbo Chen, and Michael Jordan. Non-convex finite-sum optimization via scsg methods. In Advances in Neural Information Processing Systems, pp. 2348-2358, 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning. Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020.
10

---

Under review as a conference paper at ICLR 2022
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In 6th International Conference on Learning Representations. ICLR 2018, Vancouver. BC, Canada, April 30 May 3, 2018. Conference Track Proceedings. OpenReview.net, 2018.
Sashank Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. In International conference on machine learning, pp. 314-323, 2016a.
Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, and Alexander J. Smola. Stochastic variance reduction for nonconvex optimization. In Proceedings of the 33nd International Confer- ence on Machine Learming, ICML 2016, New York City, NY, USA, June 19-24, 2016, pp. 314-323, 2016b. URL hRRphttt://ppoeeedii wwe.s v
mir
reddil6 6. html
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2018.
Mark Ring. Continual learning in reinforcement environments. PhD thesis, University of Texas at Austin, TX, USA, 1995. http::::/ddnbb.nooo/////e/
Sebastian Thrun. A lifelong learning perspective for mobile robot control. In Intelligent Robots and Systems, Selections of the International Conference on Intelligent Robots and Systems 1994, IROS 94, Munich, Germany, 12-16 September 1994, pp. 201 1994. doi: 10.1016/ 0004444402000500000555555 /ttps///////////////./.......... 50015-3
Quanming Yao and James Kwok. Efficient learning with family of nonconvex regularizers by redistributing nonconvexity. In International Conference on Machine Learning, pp. 2645-2654. PMLR, 2016.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. In Advances in neural information processing systems, pp. 9793-9803, 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings of machine learning research, 70:3987, 2017.
Dongruo Zhou and Quanquan Gu. Lower bounds for smooth nonconvex finite-sum optimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp- 7574-7583, Long Beach, California, USA, 09-15 Jun 2019, PMLR

---

Under review as a conference paper at ICLR 2022
APPENDIX
A ADDITIONAL EXPERIMENTAL RESULTS AND DETAILS
A.I HYPERPARAMETER GRIDS
We report the hyper- hyper-param paramters grid we used in our experiments below. Except for the proposed algorithm, we adopted the hyper-paramters that are reported in the original papers. We used grid search to find the optimal parameters for each model.
finetune learning rate [0.003, 0.01, 0.03 (CIFAR), 0.I (MNIST), 0.3, 1.0]
EWC learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] regularization: [0.1, 1, 10 (IFAR,, 100, 1000]
AGEM learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0]
-RR-Reseroiir- -ecervoviir-erringg learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0]
ORTHOG-SUESPACE: learning rate: [0.003, 0.01, 0.03, 0.1 (MNIST), 0.2, 0.4 (CIFAR), 1.0]
MER learning rate: [0.003, 0.01, 0.03 (MNIST, CIFAR), 0.1, 0.3, 1.0] within batch meta-learning rate: [0.01, 0.03, 0.1 (MNIST, CIFAR), 0.3, 1.0] current batch learning rate multiplier: [1, 2,5 (CIFAR), 10 (MNIST))
iid-offline and iid-online learning rate [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0]
ER-reservoir - learning rate: [0.003, 0.01, 0.03, 0.1 (MNIST, CIFAR), 0.3, 1.0] NCCL-Ring (default) learning rate a: [0.003, (COOR))) 0.01, 0.03, 0.1, 0.3, 1.0]
NCCL.-Reservor learning rate a: [O.003(CIFAR), 0.001, 0.01, 0.03, 0.1, 0.3, 1.0]
A.2 ADDITIONAL RESULTS
Table 3: Multi-headed split-CIFAR100, full size Resnet-18 n, = 64. Accuracy and forgetting results.
memory size
Method
memory
accuracy
forgetting
accuracy
forgetting
Fintune
X
42.6 (2.72)
0.27 (0.02)
42.6 (2.72)
0.27 (0.02)
EWC
X
43.2 (2.77)
0.26 (0.02)
43.2 (2.77)
0.26 (0.02)
ICRAL
V
46.4 (1.21)
0.16 (0.01)
A-GEM
V
51.3 (3.49)
0.18 (0.03)
60.9 (2.5)
0.11 (0.01)
MER
(2.97)
0.19 (0.03)
ER-Ring
V
59.6 (1.19)
0.14 (0.01)
67.2 (1.72)
0.06 (0.01)
ER-Reservoir
51.5 (2.15)
0.14 (0.09)
62.68 (0.91)
0.06 (0.01)
ORTHOG-subspace
64.3 (0.59)
0.07 (0.01)
67.3 (0.98)
0.05 (0.01)
NCCL Ring
V
59.06 (1.02)
0.03 (0.02)
66.58 (0.12)
0.004 (0.003)
NCCL + Reservoir
54.7 (0.91)
0.083 (0.01)
66.37 (0.19)
0.004 (0.001)
12

---

Under review as a conference paper at ICLR 2022
Table 4: permuted-MNial (23 tasks 10000 examples per task), FC-[256,256]. Accuracy and
forgetting results.
memory size
Method
memory
accuracy
forgetting
accuracy
forgetting
multi-task
X
91.3
83
Fine-tune
X
50.6 (2.57)
0.29 (0.01)
47.9
0.29 (0.01)
EWC
X
68.4 (0.76)
(0.01)
63.1 (1.40)
0.18 (0.01)
MER
78.6 (0.84)
(0.01)
88.34 (0.26)
0.049 (0.003)
A-GEM
V
78.3 (0.42)
0.21 (0.01)
64.1 (0.74)
0.19 (0.01)
ER-Ring
79.5 (0.31)
0..0()
75.8 (0.24)
0.07 (0.01)
ER-Reservoir
68.9 (0.89)
(0.01)
76.2 (0.38)
0.07 (0.01)
ORHOG-subspace
86.6 (0.91)
(0.01)
87.04 (0.43)
0.04 (0.003)
NCCL + Ring
74.38 (0.89)
0.05 (0.009)
83.76 (0.21)
0.014 (0.001)
NCCL=Reservoir
76.48 (0.29)
0.1 (0.002)
86.02 (0.06)
0.013 (0.002)
Table 5: Single-headed split-MNIST, FC-[256,256). Accuracy and forgetting results.
Method
memory size
50
memory
nocuracy
forgetting
accuracy
forgetting
accuracy
forgetting
multi-task
X
95.2
Fine-tune
X
52.52 (5.24)
0.41 (0.06)
FWC
X
56.48 (6.46)
0.31 (0.05)
A-GEM
V
34.04 (7.10)
0.23 (0.11)
33.57 (6.32)
0.18 (0.03)
33.35 (4.52)
0.12 (0.04)
ER-Reservoir
34.63 (6.03)
0.79 (0.07)
63.60 (3.11)
0.42 (0.05)
86.17 (0.99)
0.13 (0.016)
NCCL + Ring
34.64 34.64
0.55 (0.03)
61.02 (6.21)
0.207 (0.07)
81.35 (8.24)
-0.03 (0.1)
NCCL+Reservoir
37.02 (0.34)
0.509 (0.009)
65.4
0.16 (0.006)
88.9 (0.28)
-0.125 (0.004)
Table 6: Single-headed split-MNIST, FC-[400,400] and mem. size=500(50) / cls.). Accuracy and
forgetting results.
Method
accuracy
multi-task
96.18
Fine-tune
50.9 (5.53)
EWC
55.40 (6.29)
A-GEM
26.49 (5.62)
Re-Reservoir
85.1 (1.02)
CN-DPM
93.23
Gdumb
(0.5)
NCCL Reservoir
95.15 (0.91)
Table Single-heade split-CIFARI0, full size Resnet-18 and mem. size=500(50/ cls.). Accuracy and forgetting results.
Method
accuracy
iid-offline
93.17
iid-online
36,65
Fine-tune
12.68
EWC
53.49 (0.72)
A-GEM
54.28 (3.48)
GSS
33.56
Reservoir Sampling
37.09
CN-DPM
41.78
NCCL Ring
54.63 (0.76)
NCCL Reservoir
55.43 (0.32)
13

---

Under review as a conference paper at ICLR 2022
Table 8: PerMuted-MNIST (23 tasks 10000 examples per task), FC-[256,256] and Multi-headed Split-CIFAR100, full size Resnet-18. Accuracies with different clipping rate on NCCL Ring.
8maz
Permuted-MNIST
Split-CIFAR100
0.001
72.52(0.59)
49.43(0.63)
0.01
72.93(1.38)
56.95(1.02)
0,05
72.18(0.77)
56.35(1.42)
0.1
(2.29(1.34)
58.20(0.155)
0.2
74.38(0.89)
57.60(0.36)
0.5
72.95(0.50)
59.06(1.02)
1
72.92(1.07)
57.43(1.3))
5
72.31(1.79)
57.75(0.24)
Table 9: Permuted-MNIST (23 tasks 10000 examples per task), FC-1256,256] and Multi-headed split-CIFAR100, full size Resnet-18. Training time.
Training time [s]
Methods
PerMuted-MNIST Split-CIFAR100
fine-tune
91
92
EWC
95
159
A-GEM
180
760
ER-Ring
109
129
ER-Reservoir
95
113
ORTHOG-SUBSPACE
90
581
NCCL+Ring
167
248
NCCL+Reservoir
168
242
14

---

Under review as a conference paper at ICLR 2022
(x)
(x
XPuc
Xpuc
an Vi, ) Bu, Bi (x+5
9 Jepor (x²)
Xp o
Xp x
BuC Vii (x*) + Bug Vilcoss (x.)
4 glnne (t9)
Figure Geometric illustration of Non-Convex Continual Learning (NCCL.. In the continual learning setting, the model parameter starts from the moderate local optimal point for the previously learned tasks zp.. Over T' iterations, we expect to reach a new optimal point TPuC which has good performance
both previously learned and current tasks. In the t-th iteration, the model parameter x' encounters either VgJ..poo (z") or nee, (z*). These two different cases indicates whether (f1, (x'), VgJ, (x*)) is positive not. To prevent x² from escaping the feasible region, i.e., catastrophic forgetting, we impose theoretical condition on learning rates for ʃ and
B THEORETICAL ANALYSIS
In this section, we provide the proofs of the results for nonconvex continual learning. We first start with the derivation of Equation 5 Slin Assumption [
B. ASSUMPTION AND ADDITIONAL LEMMA
Derivation of Equation 5 Recall that
T 2 1r yl
ff(()) f(() - (V)i()), w)1
(17)
Note that f. is differentiable and nonconvex. We define function g f. + t(x y)) for t [0, 1] and an objective function ft. By the fundamental theorem calculus,
J d((()dt = f(x) f(y).
(18)
By the property, we have
(f((z) ((()) - (Viil)), x si
(Viily t(x y)), v)dt - (VV), -
(Vii 1(x -)) v,(y), x -1)d0
Using the Cauchy-Schwartz inequality.
1/' (V t(x -)) - y/itt
V -(--v) ff((1))l 1| w/(dt
Since fi satisfies Equation
4 then we have
f((()
f((y) (V x - 3)
% L + t(x - y) - vll || - vllot
<
y1l 1/" tdt
L|Ie
| y
15

---

Under review as a conference paper at ICLR 2022
Lemma B.1. Letp= = (-,,...-ppl q= lg........ .qpl be two statistically independent random vectors with dimension D. Then the expectation of the inner product of two random vectors Blpp. 9))
Ed=1 Elpal Blga].
Proof. By the property of expectation,
E Kp, g) = E
>
Paqal
d=1
E pazd
d=1
Σ Elpa]EEaal.
B.2 PROOF MAIN RESULTS
We now show the main results of our work.
Proof of Lemma 7 To clarify the issue of EM, [,, [e|[M]] = 0, let us explain the details of constructing replay-memory as follows. We have considered episodic memory and reservoir sampling in the paper. We will first show the case of episodic memory by describing the sampling method for replay memory. We can also derive the case of reservoir sampling by simply applying the result of episodic memory.
Episodic memory (ring buffer). We divide the entire dataset continual learning into the previous task P and the current task C' on the time step f = 0. For the previous task P. the data stream of P i.i.d., and its sequence is random on every trial (episode). The trial (episode) implies that continual learning agent learns from an online data stream with two consecutive data sequences of P and C. Episodic memory takes the last data points of the given memory size m by the First In First Out (FIFO) rule, and holds the entire data points until learning
C' is finished. Then, Mc = My for all 2 and Mo is uniformly sampled from the i.i.d. sequence of P. By the law of total expectation, we derive EM.CP [E,, fV, (x*) [M]] for any xf, Vt 2 0 as follows.
note that
EMOPP [E,, [Vfn('"))Mu]] EMOCP [V (x'))
We know that Mo uniformly sampled from P each trial before training the current task C. Then, we take expectation with respect to every trial that implies the expected value over the memory distribution Mo. We get
EM.PP [ [=a) (Vfx'
for any xi, Vt. We can consider VM, (x*) as sample mean of P on every trial for any x², Vt > O. Although x² is constructed iteratively, the expected value of the sample mean for any z1, EMAPP fM, (x")] ]] also derived as fz(²).
Reservoir sampling. To clarify the notation for reservoir sampling first, we denote the expectation with respect to the history of replay memory Mpa = (M..... Mt.) as EM This is the revised version of EM, Reservoir sampling is the more tricky case than episodic memory, but EM [E, MM]] = still holds. Suppose that Mo full of the data points from P as the episodic memory sampled and the mini-batch size from C' is I for simplicity. The reservoir sampling algorithm drops a data point in M,-1 and replaces the dropped data point with data point in the current mini-bated from C with probability p m m/n, where m is the memory size and is the number of visited data points so far. The exact pseudo-code for reservoir sampling is described in [1]. The replacement procedure uniformly chooses the data point which will be dropped. We can
16

---

Under review as a conference paper at ICLR 2022
also consider the replacement procedure follows. The memory M, for P is reduced in size from Mc-1, and the replaced data point de from C' contributes in terms of Vgde (a") ) if de is sampled from the replay memory. Let M,-1 [d,.... d where denotes the cardinality of the
memory. The sample mean of M,-1 is given as
v VMM I (x'-1)=
(z'a) M di
(19)
By the rule of reservoir sampling, we assume that the replacement procedure reduces the memory from M:-1 to M, with size |M,-1] ||| - and the set of remained upcoming data points C, E C from the current data stream for online continual learning is reformulated into C--1 U [de]. Then, de can resampled from Ct-1 U [dc] to be composed of the minibatch of reservoir sampling with the dfferent probability. However, we ignore the probability issue now to focus the effect of replay-memory on f. Now, we sample M, from M,-1, then we get the random vector V JM, (x²)
M 1l Σ ==1
(M,
Wa,
(20)
M.
where the index uniformly sampled from ~ [1,... [M and Wiy is the indicator function that Wis is 0 if i else 1.
The above description implies the dropping rule, and M, can be considered as an uniformly sampled set with size JM,. from M,-1- There could also be M, M,-1 with probability p m /n. Then the expectation of V fMM, (x*) given M is derived as
M
M
FM, (x
(3') =p
Σ
1
Σ
Wi Va + + - -2) (VSa, (x'))
M
M
j=1
VMM
When we consider the mini-batch sampling, we can formally reformulate the above equation as
EM.sp(MM|MM-)) [Ei.M, (3)))[) [M+i V (f,_((('))
(21)
Now, apply the above equation recursively. Then,
EM (M)) [... (MMeen(M,,MM.... ELICM, [Vfr(z')[M, [M,1]] ... Mo)] fM,, (x²).
(22)
Similar to episodic memory, Mo is uniformly sampled from P. Therefore, we conclude that
Eso.... MM. / (x,, -
(23)
by taking expectation over the history Mp::] = (M1, M2. M,).
Note that taking expectation iteratively with respect the history My is needed to compute the expected value of gradients for M, However, the result EMo... ,M. [E,, [e. still holds in terms of expectation.
Proof of Lemma 2
We analyze the convergence of nonconvex continual learning with replay mem- ory here. Recall that the gradient update is the following
1+1 -aH, Vfi.(') SH. (g).
aH,
17

---

Under review as a conference paper at ICLR 2022
for all t E (1,2. ,T.. Let e, (x', Vz')). have the following inequality by applying Equation 5
Since we assume that f, 9 i L-smooth,
f( S s(())
x²) +
| ++1 - x
= - - an. V.L + 3H. (x)) + _ IlaH. (x., Sr, Ves
f(x') - -a. (VF()) - 3n. (Vf(z') (x'))
+ an. VVL 1 3/. Vgu. [| LaH, 3H. fr, (z'),, Vgn)
=s(²') an,
V\(z')) am, VV/ 8r, (VJr, (x'), ) Vys (2"1) + Br, (Vg,, (x)),e))
Lo [V + La (Vf(x'). +
La ||e.11 +
L
V (x )12 + LoH, Bu, (VSr, (x'), (es,
2
2
=f(z²) - (aH.. - L a |[
L [VV
2 1,,,1 (x'))1 - 3H. (1 an aH LV Vg..
+ +LaaH, -aH.) (Vf(z'), + 3H. Vas, (x)), ex) + - n |ee.122.
(24)
To show the proposed theoretical convergence analysis of nonconvex continual learning, we define the catastrophic forgetting term r, and the overfitting term B, as follows:
B: = (La - a. (Vn))(((='),, + 3n. (Vg,, (x'),
84. L
V (x'))1 BH, (1 ou, ( (g.))
Then, rewrite Equation 24 as
1att11 S f(xt) OHH, - Lain [V + + B, + 2 an lea1
L
L
(25)
We first note that B, is dependent of the error term e, with the batch I, In the continual learning step, an training agent cannot access V Vf(x'), then we cannot get the exact value of ez. Furthermore, T, dependent of the gradients (x,,"")), Vgr, (a²) and the learning rates on, 3tr.
Taking expectations with respect to I on both sides given Jr, we have
E, [f( < Bi,
f(x')
aи.
2 ain
[ff(2)| +I, + B.
L
a l|e, Je
2
S Ed, f(xt) OHH
a
|Vf(=9)1 + a 1|ee11
+ Er, [T. + Bd
Now, taking expectations over the whole stochasticity we obtain
[[ s E s(xt) - [HH,
L [Vf("") B&+ + 7 a 1la
2 2
Rearranging the terms and assume that I-Lan,72 /2
O, we have
Lan
L 2 S E [(x') ((())) ((att)++r, B, + 2 a lee11
aH.
2
and
aH. (1 5(HH) (f(x*) - f(x++11) +I+ B₂) + O L
E
{
E <E
1|ee
2(1 4 4aH
(f(x*) - f(x + 1'2 + B,) + aH L
E
2(1 1 fan!
18

---

Under review as a conference paper at ICLR 2022
Proof of Theorem 7 Suppose that the learning rate aи, is constant =c/VT, for
2 Ā > 0. Then, by summing Equation from / = 0 to T 1, we have
min B
Σ
/(
T
1=U
1
- f(xx)
T-1
Σ
E B, + +Ti))
+ L
2000
2 Q
aT
1
T-1
(B [B + +ri)))
+ Le 2V7"3)
A
T
t=
T-1
A VT
Le 2 of
A +
E B + +T/]
(26)
1=
We note that batch I is sampled from memory M, C M which is a random vector whose element a datapoint E PUG C. Then, taking expectation over I, M, C P UC implies that E [B, Therefore, get the minimum of expected square of the norm of gradients as follows:
min Vl
A
A
3++EETY
+ 2
Le
of
Proof of Lemma 3 By the assumption, it is equivalent to update on M U C. Then, the non-convex
finite sum optimization is given as
min g(x) ZERd
1
Σ
g:(2),
(27)
ng + M
iEMUC
where gi the same function as fi, and M is the number of clements in M. This problem can be solved by simple SGD algorithm (Reddi al. 2016b). Thus, we have
24
min Bl| Vg(+")> <
T gg.
(28)
Lemma B.2. Let an upper bound B >> BH,, > O. For the worst case, the expectation of summing the catastrophic forgetting term over iterations T is
T-1 Σr.=
T=U
Proof. First, we derive the rough upper bound of E[T,]:
82,
E E
E
Vgn, (xt)1 Br, (1-an, L)(S),VVV (x*), Vgn (2t1)
(29)
< E
V (xt))² |12 + B1.(1 - 10.1- VI,, (o)
(30)
2
L
= 0
E
(
(31)
2
where V (I+) 2 (Vfr((""'))
19

---

Under review as a conference paper at ICLR 2022
In addition, the supremum of the variance of the mini-batch gradient VgJ, (x*) is derived as
Vug - Vg(x")]1 = sup Rg - bg
(g)(x')
Vg(x')]
(32)
(Tg 1)bg
ng
n bg og.
(33)
(ng 1)b9
where By and by denotes the size of C and minibatch J. respectively. The detailed derivation is shown in [1]. By the triangular inequality, we get
El1gs, (x²)]] S Ellgn g(x")11 + Fl(g(x') )1 s Ellg(>")|1 + ng - b,
(34)
(ng 1)bg og
(35)
By plugging the above equation into E[T,), we conclude that
8
L
ETT -
E
|
Va
(36)
2
32
L
8 L - 0g)
0
F.
I
2(ng 1)bg 02
(37)
2
The sum of catastrophic forgetting term Σr, is corrected as Σ E]T,). We use the technique for summing up in the proof of Theorem 1, then the cumulative sum of catastrophic forgetting term derived as
T-1
T-1
L
(ng bg) d
Σ E.T.
Σ
F. [[[[((('')11
(38)
2
(ny 1)by 9)
L
T-1
[g(x*) - -g( + L33
(ng bg)
Σ
0
0/ +
(ng 1 yb9 0
(39)
2
3
2
(=)
L
1
TLB
09 + T(n bg)
0
Ag
+
(ng 1)69 02
(40)
2
2
I
(ng 3
= 0 BAn +
og
+
T
(41)
2
(ng 1)bb
Rearranging the above equation, we get
(0
L
(ng 6,)
Σ
0
=
+
T + 84g
(42)
2
(ng 1)b9
0=0
Therefore, we can write ΣtD E[T,] = OT)). We note that the rough upper bound of Σ increases monotonically with training step as in the previous result in the paper.
Now we provide the rigorous derivation of the convergence rate by the result of Corollary as follows:
0
L
(ng by)
B4g O(VT).
VT
(43)
(ng 1)b,
VT
This result is obtained by dividing Σ E[T.] by vT as in the proof of Thm. 1.
On the other hand, I, can be negative when (V Vg,, > O. It implies that the cumulative sum of T2 does not increase monotonically. Therefore, for some large number N, we can denote the cumulative sum of D, over the finite steps T as follows:
T
Σ
r. N = 0(1).
(44)
20

---

Under review as a conference paper at ICLR 2022
Proof of Corollary
To formulate the IFO calls, Recall that T(c)
T(e) = min (T: min E| Vxx''|| - e).
Additionally, single IFO call is invested in calculating each step. As seen in Theorem a convergence rate of
NCCL has
Σ 1== re
o
(45)
VT
We note that the convergence rate the worst case is
o (VT).
(46)
where the given model diverges on the convergence of f (a)) Then, IFO calls are denoted as co.
For the case of Equation 44 we obtain the convergence rate O(1/VT). Thus we get 0(1/e) this
case.
C DERIVATION OF EQUATIONS IN SECTION 3.4
Derivation for A-GEM
Let the surrogate VgJ, (x*) as
(='(
Vl,, (x*)
VğJ, (x') = Vgs, (x*)
(x')| (on)
(47)
Vfr, (x!)
(V (=)) (z))) IV,,, (x*)|*
where oH, a(1
and BH. a for Equation
Then, we have
L
ETT
E
(v')(
3M. (V V
2
L L
(1V9n
(V (x*) Vg.. (2t))²
(x')), Vg.. Vg,,(1919) - BH. (V (
EE
2
V (1))|
13, L
(V (x'),, Vgs, yg,, (x)))²
EE
[IV (20)1²
BH, (VVSr,(z') (on, V), Vgg
[x*)
13, T
(V Vgs, (29))) ] (x*))
| (x²)
EE
(48)
2
Note that this result is smaller than the original ET..
Derivation of optimal r and 8h,
For a fixed learning rate a, we have
0 TETT
ar,
E
03n.
03
= E.
[8H, I.|| (x)) - (I-aa))VViiC (z.))
Thus, we obtain
(1 - OH, LVVV,, (x'),, Vas, (a),
Bn,
9l, (I1)
r O.H. (V (xt), VgJ, (x'))
2L Vos (2))
D OVERFITTING TO REPLAY MEMORY
In the main text, discussed theoretical convergence analysis of continual learning for smooth nonconvex finite-sum optimization problems. The practical continual learning tasks have the restric- tion on full access the entire data points of previously learned tasks. Unlike taking expectation
21

---

Under review as a conference paper at ICLR 2022
over 1 M and M PU C. have to compute the given memory in the practical scenario.
Then, we note that M +
Now rewirte Equation 26 for the worst case as follows.
1
(Br (B)+ L 2 2
T [uup]\
A ( E (B, (B.+ + Le o9)
(49)
a(1 aT./2)
A VT
(50)
We note that Σ Br is random variable, which is unpredictible, and choosing V fM V[(x() over entire period impossible. Then, the cumulative sum of B, increasing over T. Therefore, we conclude that for the overfitting to memory degrades the convergence rate of NCCL empirically.
22


Reviews:
[{"review_id": "pWlXpOKIW9q", "reviewer": "Reviewer_UMAT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The authors propose a theoretical analysis of the memory based lifelong learning algorithms which is an important problem since currently most of the memory based lifelong learning algorithms lack  theoretical  guarantee.\n\n2. The authors make a good connection to existing memory based lifelong learning algorithms such as GEM and A-GEM and discussed their drawbacks.\n\nWeaknesses:\n\n1. The writing of the paper needs improvement. There are many unclear sentences and typos in the paper. The notations are also confusing. \n\n2. The main issues of paper are the experimental results. In table 1, NCCL + Reservoir and NCCL + Ring underperforms ER-Ring, could the authors give some discussion on this?\n\n3. The authors argue that \"the adaptive learning rate scheme is to prevent catastrophic forgetting\", but the average accuracy maybe is a more important metric？Otherwise a method which only focuses on improving the accuracy on the memory would perform best for preventing catastrophic forgetting.\n\n\n\nUnclear sentences and typos, just to list a few:\n\ni. This implies that ∆f is not a reason for moving away from stationary\npoints of f by catastrophic forgetting\n\nTypos:\n\ni. gradient descent based algorithm reach -> gradient descent based algorithm reaches\n\nii. an replay memory -> a replay memory\n\niii. but use limited -> but uses limited\n\niv. g_j is not defined in page 2. \n\nv.  we has trained ->  we have trained\n\n", "summary_of_the_paper": "In this paper, the authors analyses the convergence rate of episodic memory-based continual learning methods. The authors formulate the continual learning problem as a nonconvex finite-sum optimization problem. Based on the analysis, the authors propose an adaptive learning rate scheduling methods to adjust the learning rates based on the gradients computed in each iteration. The results on several benchmarks show that the proposed method can achieve better results  than the baselines.", "summary_of_the_review": "In this paper, the authors conduct a theoretical analysis of the memory based lifelong learning algorithms which is an important problem. However, the writing of the paper needs improvement and  the experimental results are not convincing enough to support the theory. \n\nAfter response:\n\nThe authors' responses partly address my concerns. But in the current form of the paper, it is not ready to be published. The authors should conduct more experiments to make the theory more convincing. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "euyiXVotnd3", "reviewer": "Reviewer_TD2U", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My detailed comments are provided as below. \n\nStrength:\n\n1, Continual learning is a timely and important topic for modern machine learning. This paper provides a good step to understand the optimization properties of this direction. The analysis captures something that has not been encountered in conventional optimization problem, such as the overfitting error term and the catastrophic forgetting term, although they do not provide a quantitative characterization on the catastrophic forgetting error. \n\n2. The design of the adaptive learning rates motivated by the analysis is interesting. It can help to reduce the catastrophic forgetting error and seem to perform well in the experiments. \n\nWeakness:\n\n1. This paper is not well written. For example, it is really hard for me to understand Lemma 1, which is an important result to show how the overfitting error vanish when the initial memory setup $M_0$ is properly selected, because the authors do not do a good job in explaining the episodic memory and the replay memory with sample dropping. Even after I carefully read their proofs, I figure out in the episodic memory case, $M_t=M_0$ for all $t$, whereas for the dropping case, $M_{t}$ samples from previous $M_{t-1}$ with one random sample dropping. Given such settings, it is not very surprising to me to get such a result. \n\n2. The authors do not provide a quantitative characterization on the overfitting error (for the general case without the initialization as in Lemma 1)  and the catastrophic forgetting error. For example, it is more interesting to characterize, even under some extra assumptions, how and when such errors can be bounded. So far I do not find such results. \n\n3. I am wondering whether the authors can provide some figures to show whether their proposed algorithms indeed achieve a faster convergence rate in terms of the iteration number and the catastrophic forgetting error. \n\n4. Since in practice, there is a long sequence of tasks (beyond two as studied in this paper) for continual learning, I am wondering whether the analysis here can be further generalized to this more general case. This is because based on my knowledge, catastrophic forgetting error may also depend on the how far between two tasks. \n\n5. Some typos: \n1) page 4 line 7: \"the the replay memory with...\" ->  \"the replay memory with...\"\n2) page 8 last row but three: \"Table ??\": no quotation number. \n\n\n\n\n\n\n\n\n\n\n\n\n\n", "summary_of_the_paper": "This paper provides a convergence rate characterization for a continual learning problem where the objective functions (for two consecutive tasks) are generally nonconvex and have a finite-sum form over data samples (which belong to current and previous tasks). The analysis uses the tools from nonconvex optimization, with a goal to find a stationary point for current and previous tasks. Under the setting of continual learning, the authors consider the standard replay memory based method, focusing on both the episodic memory and the replay memory with sample dropping. For the analysis, the authors first address two important types of errors: gradient estimation bias at time t and the catastrophic forgetting error. They show that for these two memory based methods, with a good initialization of replay memory, the gradient estimation error vanishes. They further show that in term of the convergence rate of previous task, i.e., $f$ in its context, the catastrophic forgetting error seems to be inevitable and can be even unbounded. To address this error, they further propose some adaptive learning rate schemes, which show some effectiveness in several experiments.     ", "summary_of_the_review": "Overall, I feel this paper provides some initial steps to understand the convergence of continual learning, but given some weakness I list above, I am slightly negative about it. However, I am open to increase my score based on the authors' feedback and other reviewers' comments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "asHNrAhYgHe", "reviewer": "Reviewer_jqxZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths \n\nThe use of adaptive learning rates in SGD provides a novel formulation on the continual learning problem. Experimental results show the merits of the proposes approach as compared to other state-the-art algorithms while a theoretical analysis of convergence is presented.\n\nRemarks/Weaknesses\n- The authors should make more clear the contributions of the current paper as compared to other relevant state-of-the-art algorithms possibly enumerating them.\n \n- The writing in poor in some parts of the paper making it difficult for the reader to grasp the meaning. \nE.g.  \n       a) last 3 sentences  of page 2 . What the authors mean by saying \"..smaller than the loss gap of general SGD...\"\n       b) Section 3, First paragraph : \"We use the convergence rate..., which denotes the IFO complexity ...\"\n- The two figs in Figure 2 are not well explained in the caption provided.\n\n- The overfitting and catastrophic forgetting terms $B_t, \\Gamma_t$ appears rather abruptly in the text and are not well explained.\n\n- Subsection 3.3. First sentence:  \"...the model $x^t $eventually converges to $M\\cup C$, not $P\\cup C$. It is not clear what the authors mean by saying that $x^t$ converges to $M\\cup C$. \n\n- Lemma 3: What does it mean that the convergence rate is ON $M\\cup C$? This looks rather weird, again it is not clear at all what the author mean by that.\n\n\n\n\n", "summary_of_the_paper": "The authors propose a stochastic gradient descent algorithm with adaptive learning rates for continual learning providing an  analysis of convergence. The main idea is to formulate the problem as a finite sum of non-convex objective functions with each component of the sum corresponding to a different task. The authors claim that the use of adaptive learning rates allow them to better control the relative importance of different tasks. ", "summary_of_the_review": "The paper proposes a  stochastic gradient descent algorithms applied on finite sums of nonconvex functions. The main novelty of the paper lies in the application of  the SGD ideas in the context of the continual learning.  In principle, this seems to be an interesting direction. However the  writing is rather poor in general and the authors fail to give an intuition of theoretical results presented in the paper. There also seems to be some wordings that make it difficult to understand some technical details of the paper (see above). I believe that the paper in its current form does not meet the high standards of ICLR.\n\n--------------------------------------------\nComment after authors' responses:\nI appreciate the efforts of the reviewers to address my comments and the changes they have made. However, I will keep my score to 5 since I believe the paper still lacks clarity of presentation and the significance of the contributions is not sufficiently illustrated. Moreover, the writing needs improvement and I encourage the authors to further work on it and resubmit an updated version of it to a future conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KMDHDcrnAfT", "reviewer": "Reviewer_qew6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n - The paper is well-written, clear, and straightforward.\n - The results are new and interesting\n - Theoretical analysis is well-conducted\n\nCons:\n - The guarantees not surprising and maybe a bit incremental \n - Proof techniques are standard\n\nOther than the small issue in Lemma 2: note that \\alpha is allowed to be 0 but then (10) is not well-defined, so it should be strictly larger than 0, I did not find any problems in the proofs in the appendix or in the mathematical writing.", "summary_of_the_paper": "The paper proposes mini-batch stochastic gradient method of the framework of nonconvex memory-based continual learning, and conducts  a theoretical convergence analysis. Additionally, the paper studies catastrophic forgetting and outfitting in the context of the proposed approach.\n\n", "summary_of_the_review": " The paper is interesting, provides new insights and theoretical guarantees, and in my opinion should be accepted for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper theoretically studies the convergence of memory-based continual learning with stochastic gradient descent, and suggested several methods based on adaptive learning rates.

The reviewers appreciated the novelty of the direction, and some of them thought the experimental results are promising.

However, most reviewers (3/4) were negative. I think the main reason was the paper presentation and clarity, which they found lacking (and I agree). One reviewer thought the experimental evaluation should be improved, but there might have been some misunderstanding there. Lastly, even the positive reviewer thought the results were somewhat incremental and non-surprising.

I hope the authors improve their paper and re-submit.
Decision: Reject