Nonconvex Continual Learning with Episodic Memory

1 INTRODUCTION . Learning new tasks without forgetting previously learned tasks is a key aspect of artificial intelligence to be as versatile as humans . Unlike the conventional deep learning that observes tasks from an i.i.d . distribution , continual learning train sequentially a model on a non-stationary stream of data ( Ring , 1995 ; Thrun , 1994 ) . The continual learning AI systems struggle with catastrophic forgetting when the data acess of previously learned tasks is restricted ( French & Chater , 2002 ) . To overcome catastrophic forgetting , continual learning algorithms introduce a memory to store and replay the previously learned examples ( Lopez-Paz & Ranzato , 2017 ; Aljundi et al. , 2019b ; Chaudhry et al. , 2019a ) , penalize neural networks with regularization methods ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ) , use Bayesian approaches ( Nguyen et al. , 2018 ; Ebrahimi et al. , 2020 ) , and other novel methods ( Yoon et al. , 2018 ; Lee et al. , 2019 ) . Although Gradient Episodic Memory ( GEM ) ( Lopez-Paz & Ranzato , 2017 ) first formulated the continual learning as a constrained optimization problem , the theoretical convergence analysis of the performance of previously learned tasks , which implies a measure of catastrophic forgetting , has not been investigated yet . Continual learning with episodic memory utilizes a small subset of the data for previous tasks to keep the model staying in a feasible region corresponding to moderate suboptimal region . GEM-based approaches use the rephrased constraints , which are inequalities based on the inner product of loss gradient vectors for previous tasks and a current task . This intuitive reformulation of constrained optimization does not provide theoretical guarantee to prevent catastrophic forgetting . In addition , the memory-based approaches have the critical limitation of overfitting to memory . Choosing the perfect memory for continual learning is an NP-hard problem ( Knoblauch et al. , 2020 ) , then the inductive bias by episodic memory is inevitable . This problem also degrades the performance on previously learned tasks like catastrophic forgetting but has not been discussed quantitatively to analyze backward transfer ( BWT ) . In this paper , we address the continual learning with episodic memory as a smooth nonconvex finitesum optimization problem . This generic form is well studied to demonstrate the convergence and complexity of stochastic gradient methods for the nonconvex setting ( Zhou & Gu , 2019 ; Lei et al. , 2017 ; Reddi et al. , 2016 ; Zaheer et al. , 2018 ) . Unlike the convex case , the convergence is generally measured by the expectation of the squared norm of the gradient E‖∇f ( x ) ‖2 . The theoretical complexity is derived from the -accurate solution , which is also known as a stationary point with E‖∇f ( x ) ‖2 ≤ . We formulate the proposed continual learning algorithm as a Stochastic gradient descent ( SGD ) based method that updates both previously learned tasks from episodic memory and the current task simultaneously . By leveraging the update method , we can introduce a theoretical analysis of continual learning problems . We highlight our main contributions as follows . • We develop convergence analysis for continual learning with episodic memory • We show the degradation of backward transfer theoretically and experimentally as prob- lems of catastrophic forgetting and overfitting to memory . • We propose a nonconvex continual learning algorithm that scales learning rates based on sampled mini-batch . 1.1 RELATED WORK . The literature in continual learning can be divided into episodic learning and task-free learning . Episodic learning based methods assume that a training model is able to access clear task boundaries and stores observed examples in the task-wise episodic memory ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2019a ) . On the other hand , an AI system experiences arbitrarily shifting data streams , which we are not able to access task boundaries in the real world . Task-free continual learning studies the general scenario without the task-boundary assumption . Aljundi et al . ( 2019a ) introduces Memory-aware Synapses ( MAS ) and applies a learning protocol without waiting until a task is finished . Furthermore , the following work ( Aljundi et al. , 2019b ) adopt the memory system of GEM selecting observed examples to store for preventing catastrophic forgetting . Smooth nonconvex finite-sum optimization problem has been widely employed to derive the theoretical complexity of computation for stochastic gradient methods ( Ghadimi & Lan , 2013 ; 2016 ; Lei et al. , 2017 ; Zaheer et al. , 2018 ; Reddi et al. , 2016 ) . Unlike the convex optimization , the gradient based algorithms are not expected to converge to the global minimum but are evaluated by measuring the convergence rate to the stationary points in the nonconvex case . The complexity to reach a stationary point is a key aspect of building a new stochastic gradient method for nonconvex optimization . In constrast with general optimization , memory-based continual learning methods have a limited data pool for previously learned tasks , which causes an overfitting problem to memory . ( Knoblauch et al. , 2020 ) found that optimal continual learning algorithms and building a perfect memory is equivalent . Furthermore , the authors proved that these two problems are NP-hard . The theoretical result shows that overfitting to memory is inevitable . 2 PRELIMINARIES . We consider a continual learning problem with episodic memory where a learner can access the boundary between the previous task and the current task . The continuum of data in ( Lopez-Paz & Ranzato , 2017 ) is adopted as our task description of continual learning . First , we formulate our goal as the smooth nonconvex finite-sum optimization problems with two objectives , min x∈Rd F ( x ) = f ( x ) + g ( x ) = 1 nf nf∑ i=1 fi ( x ) + 1 ng ng∑ j=1 gj ( x ) ( 1 ) where x ∈ Rd is the model parameter , each objective component fi ( x ) , gj ( x ) is differentiable and nonconvex , and nf , ng are the numbers of components . We define two different components of the finite-sum optimization as objectives from a sample i of previously learned tasks fi ( x ) and a sample j of the current task gj ( x ) . Unlike the general stochastic optimization problem , we assume that the initial point x0 in continual learning is an -accurate solution of f ( x ) with E‖∇f ( x ) ‖2 ≤ for some 1 . By the property of nonconvex optimization , we know that there might exist multiple local optimal points that satisfy moderate performance on the previously learned task ( Garipov et al. , 2018 ) . This implies that the model parameter x stays in the neighborhood of x0 or usually moves from an initial local optimal point x0 to the other local optimal point at the t-th iteration , xt over T iterations of a successful continual learning scenario . The continual learning algorithm with an episodic memory with size m can not access the whole dataset of the previously learned tasks with nf samples but use limited samples in the memory when a learner trains on the current task . This limited access allows us to prevent catastrophic forgetting partially . However the fixed samples from memory cause a biased gradient and the overfitting problem . In Section 3 , we provide the convergence analysis of the previously learned tasks f ( x ) , which are vulnerable to catastrophic forgetting . We denote fi ( x ) as the component , which indicates the loss of sample i from the previously learned tasks with the model parameter x and ∇fi ( x ) as its gradient . We use It , Jt as the mini-batch of samples at iteration t and denote bft , b g t as the mini-batch size |It| , |Jt| for brevity throughout the paper . We also note that gj from the current task holds the above and following assumptions . To formulate the convergence over iterations , we introduce the Incremental First-order Oracle ( IFO ) framework ( Ghadimi & Lan , 2013 ) , which is defined as a unit of cost by sampling the pair ( ∇fi ( x ) , fi ( x ) ) . For example , a stochastic gradient descent algorithm requires the cost as much as the batch size bt at each step , and the total cost is the sum of batch sizes ∑T t=1 bt . Let T ( ) be the minimum number of iterations to guarantee -accurate solutions . Then the average bound of IFO complexity is less than or equal to ∑T ( ) t=1 bt . To analyze the convergence and compute the IFO complexity , we define the loss gap between two local optimal points ∆f as ∆f = f ( x 0 ) − inf 0≤t≤T f ( xt ) , ( 2 ) which might be much smaller than the loss gap of SGD . Suppose that the losses of all optimal points have the same values , i.e. , f ( x∗ ) = f ( x0 ) , then we have ∆f ≤ 0 . This implies that ∆f is not a reason for moving away from a stationary point of f , which we will explain details in Section 3 . We also define σf , σg for f , g , respectively , as the upper bounds on the variance of the stochastic gradients of a given mini-batch . For brevity , we write only one of them σf , σf = sup x 1 bf bf∑ i=1 ‖∇fi ( x ) −∇f ( x ) ‖2 . ( 3 ) Throughout the paper , we assume the L-smoothness . Assumption 1 fi is L-smooth that there exists a constant L > 0 such that for any x , y ∈ Rd , ‖∇fi ( x ) −∇fi ( y ) ‖ ≤ L‖x− y‖ ( 4 ) where ‖·‖ denotes the Euclidean norm . Then the following inequality directly holds that −L 2 ‖x− y‖2 ≤ fi ( x ) − fi ( y ) − 〈∇fi ( y ) , x− y〉 ≤ L 2 ‖x− y‖2 . ( 5 ) In this paper , we consider the framework of continual learning with episodic memory . By the assumption of GEM , we assign each task sample from i.i.d . distribution within its episode to the same memory budget m. In the learning phase at task k ∈ { 1 , 2 , · · · , K } , we sample a batch with size nf from memories of all task with size [ m · ( k − 1 ) ] . 3 NONCONVEX CONTINUAL LEARNING . In this section , we present the convergence analysis of continual learning in the nonconvex setting . The theoretical result shows why catastrophic forgetting occurs in view of the nonconvex optimization problem . As a result , we can propose the Non-Convex Continual Learning ( NCCL ) algorithm , where the learning rates for the previously learned tasks and the current tasks are scaled by the value of the inner product by their gradients for the parameter in Section 3.3 . 3.1 ONE EPISODE ANALYSIS . The key element behind preventing catastrophic forgetting is to use gradient compensation on the training step of the current task . It can be considered as an additive gradient , in turn , is applied to the gradient of the current task , although GEM ( Lopez-Paz & Ranzato , 2017 ) uses the quadratic programming and EWC ( Kirkpatrick et al. , 2017 ) introduces the auxiliary loss function . First , we present the proposed gradient compensation , which uses samples of the episodic memory for a single new task episode . We define the gradient update xt+1 = xt − αHt∇fIt ( xt ) − βHt∇gJt ( xt ) ( 6 ) where αHt , βHt are learning rates scaled by the sampled mini-batches for Ht = It ∪ Jt and ∇fHt ( xt ) , ∇gHt ( xt ) are the estimates of the gradient ∇f ( xt ) , ∇g ( xt ) respectively . Equation 6 implies that the parameter is updated on the current task g with a gradient compensation on previously learned tasks f by αHt∇fIt ( xt ) . Our goal is to explain the effect of the gradient update βHt∇gJt ( xt ) on the convergence to stationary points of f ( x ) and observe the properties of the expectation of each element over It . For iteration t ∈ [ 1 , T ] and a constant L , we define the catastrophic forgetting term Ct to be the expectation in terms of∇gJt ( xt ) : Ct = E [ β2HtL 2 ‖∇gJt ( xt ) ‖2 − βHt〈∇f ( xt ) , ∇gJt ( xt ) 〉 ] , ( 7 ) which we derive in Appendix A . We temporally assume the following to show the convergence analysis of continual learning . Assumption 2 Suppose that the episodic memory M contains the entire data points of previously learned tasks [ k − 1 ] on the k-th episode and replays the mini-batch It ⊂ M . Then ∇fIt ( xt ) is an unbiased estimate that E [ et ] = 0 for et = ∇fIt ( xt ) −∇f ( xt ) . In the next section , we do not use Assumption 2 and investigate the biasedness of the episodic memory M that causes the overfitting on memory . Our first main result is the following theorem that provides the stepwise change of convergence of our algorithm . Theorem 1 Suppose that Lα2Ht − α 2 Ht ≤ γ for some γ > 0 and αHt ≤ 2L . Under Assumption 1 , 2 , we have E‖∇f ( xt ) ‖2 ≤ 1 1− L2 αHt ( 1 αHt ( E [ f ( xt ) − f ( xt+1 ) ] + Ct ) + αHtL 2bf σ2f ) . ( 8 ) We present the proof in Appendix A . Note that the catastrophic forgetting term Ct exists , unlike the general SGD , and this term increases the IFO complexity . Fortunately , we can tighten the upper bound of Equation ( 8 ) by minimizing Ct. Now we telescope over a single episode for the current task . Then we obtain the following theorem . Theorem 2 Let αHt = α = c√T for some c > 0 and all t ∈ [ T ] and 1− L 2 α = 1 A > 0 for some A . Under Theorem 1 , we have min t E‖∇f ( xt ) ‖2 ≤ A√ T ( 1 c ( ∆f + T−1∑ t=0 Ct ) + Lc 2bf σ2f ) . ( 9 ) This theorem can explain the theoretical background of catastrophic forgetting . The cumulative summation of catastrophic forgetting terms ∑ Ct increases drastically over iterations . This fact implies that the stationary point x0 can diverge . An immediate consequence of Equation 9 is that we can consider the amount of catastrophic forgetting as an optimization-viewed factor . Without the additive catastrophic forgetting term , Theorem 2 is equivalent to the result for SGD with a fixed learning rate ( Ghadimi & Lan , 2013 ) . Similar to SGD , the upper bound of Equation 9 can be made O ( A√ T ( ∆f + ∑ Ct ) ) when we assume that Lc2bf σ 2 f = O ( 1 ) . Conversely , we consider the convergence analysis of g ( x ) by changing roles for f and g in Theorem 2 . In the very beginning of iterations , ∆g is dominant in Equation 9 , and its catastrophic forgetting term Ct , g with regard to ∇fIt ( xt ) is relatively small because xt is the neighborhood of the stationary point . When we consider Assumption 2 and the samples from previously learned tasks are constantly provided , the norm of gradients ‖fIt ( xt ) ‖ is bounded . Therefore , g ( x ) can reach a stationary point by the same rate as SGD . However , We can not access the full dataset of previously learned tasks because of the setting of continual learning . There exists an extra term that interrupts the convergence of g ( x ) , which is called the overfitting . We now ignore the extra term to conjecture that ‖∇gJt ( x ) ‖ is at least bounded . Then we have the following corollary . Corollary 1 Let the expected stationary of g ( x ) be O ( δ√ T ) for a constant δ > 0 and the upper bound of learning rate for g ( x ) be β > 0 . The cumulative sum of the catastrophic forgetting term C is O ( β2δ √ T ) . Nonconvex continual learning by Equation ( 6 ) does not converge as iterating the algorithm for the worst case , where min t E‖∇f ( xt ) ‖2 isO ( β2δ ) for 1 β2δ √ T . When β2δ ≤ 1√ T , we have min t E‖∇f ( xt ) ‖2 = O ( 1√ T ) . ( 10 ) Then , the IFO complexity for achieving an -accurate solution of f ( x ) is O ( 1/ 2 ) . We would like to emphasize that catastrophic forgetting is inevitable in the worst case scenario because the stationary of f ( x ) is not decreasing and the convergence on f ( x ) can not be recovered no matter how long we proceed training . Building a tight bound of C is the key to preventing catastrophic forgetting . Note that the general setting to minimize C is scaling down the learning rate β to β2δ ≤ 1/ √ T . Then we have the decreasingC = O ( 1/ √ T ) . However , this method is slowing down the convergence of the current task g ( x ) and not an appropriate way . The other option is to minimize Ct itself rather than tightening the loose upper bound O ( β2δ √ T ) . We discuss how to minimize this term by scaling two learning rates in Section 3.3 . The constrained optimization problem of GEM provided a useful rephrased constraint but can not explain and guarantee the catastrophic forgetting in the nonconvex setting . Our convergence analysis of continual learning is the first quantitative result of catastrophic forgetting in the manner of nonconvex optimization .

Summary: This paper analyses the convergence of episodic memory-based continual learning methods by looking at it as a nonconvex optimisation problem. They analyse the convergence rates for the case where all memory from past tasks is stored, and then consider the case where there is only a subset of past data, leading to overfitting on the episodic memory. They then introduce a method that scales the learning rates of the their update method, with the goal of tightening the bound obtained in the convergence analysis. Finally, experiments are shown on different benchmarks, and the proposed method is compared to some competing baselines.