On the Convergence of Nonconvex Continual Learning with Adaptive Learning Rate

1 INTRODUCTION . Learning new tasks without forgetting previously learned tasks is a key aspect of artificial intelligence to be as versatile as humans . Unlike the conventional deep learning that observes tasks from an i.i.d . distribution , continual learning train sequentially a model on a non-stationary stream of data ( Ring , 1995 ; Thrun , 1994 ) . The continual learning AI systems struggle with catastrophic forgetting when the data access of previously learned tasks is restricted ( French & Chater , 2002 ) . To overcome catastrophic forgetting , continual learning algorithms introduce novel methods such as a replay memory to store and replay the previously learned examples ( Lopez-Paz & Ranzato , 2017 ; Aljundi et al. , 2019 ; Chaudhry et al. , 2019a ) , regularization methods that penalize neural networks ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ) , Bayesian methods that utilize the uncertainty of parameters or data points ( Nguyen et al. , 2018 ; Ebrahimi et al. , 2020 ) , and other recent approaches ( Yoon et al. , 2018 ; Lee et al. , 2019 ) . In this paper , we focus on the online continual learning with replay memory . The learner stores a small subset of the data for previous tasks into a memory and utilizes the memory by replaying samples to keep the model staying in a feasible region corresponding to moderate suboptimal region . Gradient episodic memory ( GEM ) ( Lopez-Paz & Ranzato , 2017 ) first formulated the replay based continual learning as a constrained optimization problem . This formulation rephrases the constraints on objectives for previous tasks as the inequalities based on the inner product of loss gradient vectors for previous tasks and a current task . However , the theoretical convergence analysis of the performance of previously learned tasks , which implies a measure of catastrophic forgetting , has not been rigorously studied in the literature . Without convergence analysis , this intuitive reformulation of constrained optimization does not provide theoretical guarantee to prevent catastrophic forgetting . Nonconvex finite-sum optimization problem offers a solution to analyze catastrophic forgetting by measuring the convergence of previously learned tasks , which is related to the performance . Most deep learning problems are defined as nonconvex optimization , and the target objective is composed of the sum of objectives for each data point . Now we express our continual learning problem of the form min x∈Rd f ( x ) = 1 n n∑ i=1 fi ( x ) , ( 1 ) where we assume that each objective fi ( x ) with a model x and a data point i is nonconvex with Lipschitz gradient . Here , we expect that a stochastic gradient descent based algorithm reaches a stationary point instead of the global minimum in the nonconvex optimization . This generic form is well studied to demonstrate the convergence and complexity of stochastic gradient methods for a nonconvex setting ( Zhou & Gu , 2019 ; Lei et al. , 2017 ; Reddi et al. , 2016a ; Zaheer et al. , 2018 ) . Unlike the convex case , the convergence is generally measured by the expectation of the squared norm of the gradient E‖∇f ( x ) ‖2 . The theoretical complexity is derived from the -accurate solution , which is also known as a stationary point with E‖∇f ( x ) ‖2 ≤ . Suppose we divide the entire sum of objectives into two terms for previous tasks and current tasks , and measure the convergence on each term . Then , we can observe the transition of convergences on the previous and current tasks respectively while learning sequentially from a data stream . We consider this transition of convergence on the previous task as catastrophic forgetting if E‖∇fP ( x ) ‖2 with the set of data points from previous tasks P increases over iterations . In this work , we formulate continual learning problem as a nonconvex finite-sum optimization with stochastic gradient descent algorithm that updates both previously learned tasks from replay memory and the current task simultaneously , and present a theoretical convergence analysis of continual learning problems by leveraging the replay-based update method . This extends the continual learning algorithm such as ER-Reservoir ( experience replay with reservoir sampling ) with fixed and same learning rates on both two tasks to our adaptive method which controls the relative importance between tasks at each step with theoretical guarantee . In addition , the replay-based continual learning has the critical limitation of overfitting to memory , which also degrades the performance of previously learned tasks like catastrophic forgetting by interference . It is known that choosing the perfect memory for continual learning to prevent catastrophic forgetting is an NP-hard problem by ( Knoblauch et al. , 2020 ) . We present that the inductive bias by replay memory , which prevents perfect continual learning is inevitable in view of optimization . 2 BACKGROUNDS . The continual learning algorithm with a replay memory with size m can not access the whole dataset of the previously learned tasks with nf samples , but uses limited samples in the memory when a learner trains on the current task . This limited access does not guarantee to completely prevent catastrophic forgetting , and causes the overfitting problem with the biased gradient on a memory . In Section 3 , we provide the convergence analysis of the previously learned tasks f ( x ) , which are vulnerable to catastrophic forgetting . As we denote fi ( x ) as the component , which indicates the loss of sample i from the previously learned tasks with the model parameter x , we define ∇fi ( x ) as its gradient . We use It , Jt as the mini-batch of samples at iteration t and denote the mini-batch size |It| and |Jt| as bf , bg throughout the paper . We also note that gj ( x ) , which denotes the loss for the current task and will be defined in Section 3.1 , satisfies the above and following assumptions . To formulate the convergence over iterations , we introduce the Incremental First-order Oracle ( IFO ) framework ( Ghadimi & Lan , 2013 ) , which is defined as a unit of cost by sampling the pair ( ∇fi ( x ) , fi ( x ) ) . For example , a stochastic gradient descent algorithm requires the cost as much as the batch size bt at each step , and the total cost is the sum of batch sizes ∑T t=1 bt . Let T ( ) be the minimum number of iterations to guarantee -accurate solutions . Then the average bound of IFO complexity is less than or equal to ∑T ( ) t=1 bt . To analyze the convergence and compute the IFO complexity , we define the supremum of loss gap between a local optimal point x0 and the global optimum x∗ as ∆f = sup x0 f ( x0 ) − f ( x∗ ) . ( 2 ) Suppose that sup f ( x0 ) is same with f ( x∗ ) , then we have ∆f = 0 , which might be much smaller than the loss gap of general SGD . Without the continual learning scenario , a general nonconvex SGD updates the parameters from an randomly initialized point , which is highly likely to have the large loss f ( x0 ) . Then , ∆f > 0 is the key constant to determine the IFO complexity for convergence as ∆f is in the numerator of Equation 11 . However , a continual learning algorithm has already converged to a local optimal point x0 for the previous task f ( x ) and might get a much smaller ∆f than the general SGD . It means that ∆f for nonconvex continual learining in Equation 11 dose not have a large impact on the IFO complexity . To generalize the theoretical result , we define the worst local minimum to explain the upper bound of convergence rate in Equation 2 . This implies that ∆f is not a critical reason for moving away from stationary points of f by catastrophic forgetting , which we will explain in Section 3 . We also define σf and σg for f and g , respectively , as the upper bounds on the variance of the stochastic gradients of a given mini-batch . For brevity , we write only one of them σf , σf = sup x 1 nf nf∑ i=1 ‖∇fi ( x ) −∇f ( x ) ‖2 . ( 3 ) Throughout the paper , we assume the L-smoothness . Assumption 1. fi is L-smooth that there exists a constant L > 0 such that for any x , y ∈ Rd , ‖∇fi ( x ) −∇fi ( y ) ‖ ≤ L‖x− y‖ ( 4 ) where ‖·‖ denotes the Euclidean norm . Then the following inequality directly holds that −L 2 ‖x− y‖2 ≤ fi ( x ) − fi ( y ) − 〈∇fi ( y ) , x− y〉 ≤ L 2 ‖x− y‖2 . ( 5 ) We derive the inequality in Appendix B . With Assumption 1 , we can successfully handle individual noncovex objectives for each data point . In the next section , we investigate nonconvex continual learning with adaptive learning rates to overcome catastrophic forgetting . 3 NONCONVEX CONTINUAL LEARNING . We first present a theoretical convergence analysis of memory-based continual learning in nonconvex setting . We use the convergence rate of stochastic gradient methods , which denotes the IFO complexity to reach an -accurate solution for smooth nonconvex finite-sum problem ( Reddi et al. , 2016a ) . This generic form enables both deep learning and optimization communities to formulate various accelerated gradient methods with theoretical guarantee . We seek to understand why catastrophic forgetting happens in terms of the convergence rate , and propose non-convex continual learning ( NCCL ) algorithms with theoretical convergence analysis . 3.1 PROBLEM FORMULATION . Given two finite sets P and C at the initial time step t = 0 , we let two sets denote the sets of indices for previously learned data points and upcoming data points , respectively . Note that the task description for a continual learner is two separate sets . In this section , we will show a convergence analysis of the model parameter that we have trained on P and starts to learn C. Thus , we simply denote a data stream of continual learning as two consecutive sets P and C. We consider our goal as a smooth nonconvex finite-sum optimization problem with two objectives min x∈Rd F ( x ) = f ( x ) + g ( x ) = 1 nf ∑ i∈P fi ( x ) + 1 ng ∑ j∈C gj ( x ) , ( 6 ) where fi ( x ) and gj ( x ) denote the objectives of data points i ∈ P and j ∈ C , respectively . In addition , nf and ng are the numbers of elements for P and C. To ease exposition , we use a different notation gj ( x ) for a data point j ∈ C , which is usually the same objective function for a data point i ∈ P . To formulate a theoretical convergence analysis of continual learning , we consider a replay memory based method of which memory is a subset of P ∪ C. Let a random variable Mt ⊂ P ∪ C be the replay memory at time step t ∈ [ 0 , T ] , whose union is of the form M : = ∪tMt . We focus both the episodic memory and the replay memory with dropping rule . The episodic memory based methods include GEM ( Aljundi et al. , 2019 ) , A-GEM ( Chaudhry et al. , 2019a ) , and ORTHOG-SUBSPACE ( Chaudhry et al. , 2020 ) . ER-Reservoir ( Chaudhry et al. , 2019b ) is a replay memory based method with dropping rule , which replaces the dropped sample d ∈Mt with a sample in the stream for C. We now define the gradient update of continual learning xt+1 = xt − αHt∇fIt ( xt ) − βHt∇gJt ( xt ) , ( 7 ) where It ⊂Mt and Jt ⊂ C denote the mini-batches from the replay memory and the current data stream , respectively . Here , Ht is the union of It and Jt . The adaptive learning rates of ∇fIt ( xt ) and ∇gJt ( xt ) are denoted by αHt and βHt which are the functions of Ht . Strictly speaking , the mini-batch It from Mt might contain a datapoint d ∈ C for ER-Reservoir . We describe the details of the problem in Appendix B and assume that the notation It indicates a subset of P for convenience . Equation 7 is a generalized version of continual learning algorithms , which is our novelty to prove the convergence rates in the nonconvex setting for the proposed method , A-GEM , and ER-Reservoir later .

Summary: The authors propose a stochastic gradient descent algorithm with adaptive learning rates for continual learning providing an  analysis of convergence. The main idea is to formulate the problem as a finite sum of non-convex objective functions with each component of the sum corresponding to a different task. The authors claim that the use of adaptive learning rates allow them to better control the relative importance of different tasks.