On the Convergence of Nonconvex Continual Learning with Adaptive Learning Rate

1 INTRODUCTION . Learning new tasks without forgetting previously learned tasks is a key aspect of artificial intelligence to be as versatile as humans . Unlike the conventional deep learning that observes tasks from an i.i.d . distribution , continual learning train sequentially a model on a non-stationary stream of data ( Ring , 1995 ; Thrun , 1994 ) . The continual learning AI systems struggle with catastrophic forgetting when the data access of previously learned tasks is restricted ( French & Chater , 2002 ) . To overcome catastrophic forgetting , continual learning algorithms introduce novel methods such as a replay memory to store and replay the previously learned examples ( Lopez-Paz & Ranzato , 2017 ; Aljundi et al. , 2019 ; Chaudhry et al. , 2019a ) , regularization methods that penalize neural networks ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ) , Bayesian methods that utilize the uncertainty of parameters or data points ( Nguyen et al. , 2018 ; Ebrahimi et al. , 2020 ) , and other recent approaches ( Yoon et al. , 2018 ; Lee et al. , 2019 ) . In this paper , we focus on the online continual learning with replay memory . The learner stores a small subset of the data for previous tasks into a memory and utilizes the memory by replaying samples to keep the model staying in a feasible region corresponding to moderate suboptimal region . Gradient episodic memory ( GEM ) ( Lopez-Paz & Ranzato , 2017 ) first formulated the replay based continual learning as a constrained optimization problem . This formulation rephrases the constraints on objectives for previous tasks as the inequalities based on the inner product of loss gradient vectors for previous tasks and a current task . However , the theoretical convergence analysis of the performance of previously learned tasks , which implies a measure of catastrophic forgetting , has not been rigorously studied in the literature . Without convergence analysis , this intuitive reformulation of constrained optimization does not provide theoretical guarantee to prevent catastrophic forgetting . Nonconvex finite-sum optimization problem offers a solution to analyze catastrophic forgetting by measuring the convergence of previously learned tasks , which is related to the performance . Most deep learning problems are defined as nonconvex optimization , and the target objective is composed of the sum of objectives for each data point . Now we express our continual learning problem of the form min x∈Rd f ( x ) = 1 n n∑ i=1 fi ( x ) , ( 1 ) where we assume that each objective fi ( x ) with a model x and a data point i is nonconvex with Lipschitz gradient . Here , we expect that a stochastic gradient descent based algorithm reaches a stationary point instead of the global minimum in the nonconvex optimization . This generic form is well studied to demonstrate the convergence and complexity of stochastic gradient methods for a nonconvex setting ( Zhou & Gu , 2019 ; Lei et al. , 2017 ; Reddi et al. , 2016a ; Zaheer et al. , 2018 ) . Unlike the convex case , the convergence is generally measured by the expectation of the squared norm of the gradient E‖∇f ( x ) ‖2 . The theoretical complexity is derived from the -accurate solution , which is also known as a stationary point with E‖∇f ( x ) ‖2 ≤ . Suppose we divide the entire sum of objectives into two terms for previous tasks and current tasks , and measure the convergence on each term . Then , we can observe the transition of convergences on the previous and current tasks respectively while learning sequentially from a data stream . We consider this transition of convergence on the previous task as catastrophic forgetting if E‖∇fP ( x ) ‖2 with the set of data points from previous tasks P increases over iterations . In this work , we formulate continual learning problem as a nonconvex finite-sum optimization with stochastic gradient descent algorithm that updates both previously learned tasks from replay memory and the current task simultaneously , and present a theoretical convergence analysis of continual learning problems by leveraging the replay-based update method . This extends the continual learning algorithm such as ER-Reservoir ( experience replay with reservoir sampling ) with fixed and same learning rates on both two tasks to our adaptive method which controls the relative importance between tasks at each step with theoretical guarantee . In addition , the replay-based continual learning has the critical limitation of overfitting to memory , which also degrades the performance of previously learned tasks like catastrophic forgetting by interference . It is known that choosing the perfect memory for continual learning to prevent catastrophic forgetting is an NP-hard problem by ( Knoblauch et al. , 2020 ) . We present that the inductive bias by replay memory , which prevents perfect continual learning is inevitable in view of optimization . 2 BACKGROUNDS . The continual learning algorithm with a replay memory with size m can not access the whole dataset of the previously learned tasks with nf samples , but uses limited samples in the memory when a learner trains on the current task . This limited access does not guarantee to completely prevent catastrophic forgetting , and causes the overfitting problem with the biased gradient on a memory . In Section 3 , we provide the convergence analysis of the previously learned tasks f ( x ) , which are vulnerable to catastrophic forgetting . As we denote fi ( x ) as the component , which indicates the loss of sample i from the previously learned tasks with the model parameter x , we define ∇fi ( x ) as its gradient . We use It , Jt as the mini-batch of samples at iteration t and denote the mini-batch size |It| and |Jt| as bf , bg throughout the paper . We also note that gj ( x ) , which denotes the loss for the current task and will be defined in Section 3.1 , satisfies the above and following assumptions . To formulate the convergence over iterations , we introduce the Incremental First-order Oracle ( IFO ) framework ( Ghadimi & Lan , 2013 ) , which is defined as a unit of cost by sampling the pair ( ∇fi ( x ) , fi ( x ) ) . For example , a stochastic gradient descent algorithm requires the cost as much as the batch size bt at each step , and the total cost is the sum of batch sizes ∑T t=1 bt . Let T ( ) be the minimum number of iterations to guarantee -accurate solutions . Then the average bound of IFO complexity is less than or equal to ∑T ( ) t=1 bt . To analyze the convergence and compute the IFO complexity , we define the supremum of loss gap between a local optimal point x0 and the global optimum x∗ as ∆f = sup x0 f ( x0 ) − f ( x∗ ) . ( 2 ) Suppose that sup f ( x0 ) is same with f ( x∗ ) , then we have ∆f = 0 , which might be much smaller than the loss gap of general SGD . Without the continual learning scenario , a general nonconvex SGD updates the parameters from an randomly initialized point , which is highly likely to have the large loss f ( x0 ) . Then , ∆f > 0 is the key constant to determine the IFO complexity for convergence as ∆f is in the numerator of Equation 11 . However , a continual learning algorithm has already converged to a local optimal point x0 for the previous task f ( x ) and might get a much smaller ∆f than the general SGD . It means that ∆f for nonconvex continual learining in Equation 11 dose not have a large impact on the IFO complexity . To generalize the theoretical result , we define the worst local minimum to explain the upper bound of convergence rate in Equation 2 . This implies that ∆f is not a critical reason for moving away from stationary points of f by catastrophic forgetting , which we will explain in Section 3 . We also define σf and σg for f and g , respectively , as the upper bounds on the variance of the stochastic gradients of a given mini-batch . For brevity , we write only one of them σf , σf = sup x 1 nf nf∑ i=1 ‖∇fi ( x ) −∇f ( x ) ‖2 . ( 3 ) Throughout the paper , we assume the L-smoothness . Assumption 1. fi is L-smooth that there exists a constant L > 0 such that for any x , y ∈ Rd , ‖∇fi ( x ) −∇fi ( y ) ‖ ≤ L‖x− y‖ ( 4 ) where ‖·‖ denotes the Euclidean norm . Then the following inequality directly holds that −L 2 ‖x− y‖2 ≤ fi ( x ) − fi ( y ) − 〈∇fi ( y ) , x− y〉 ≤ L 2 ‖x− y‖2 . ( 5 ) We derive the inequality in Appendix B . With Assumption 1 , we can successfully handle individual noncovex objectives for each data point . In the next section , we investigate nonconvex continual learning with adaptive learning rates to overcome catastrophic forgetting . 3 NONCONVEX CONTINUAL LEARNING . We first present a theoretical convergence analysis of memory-based continual learning in nonconvex setting . We use the convergence rate of stochastic gradient methods , which denotes the IFO complexity to reach an -accurate solution for smooth nonconvex finite-sum problem ( Reddi et al. , 2016a ) . This generic form enables both deep learning and optimization communities to formulate various accelerated gradient methods with theoretical guarantee . We seek to understand why catastrophic forgetting happens in terms of the convergence rate , and propose non-convex continual learning ( NCCL ) algorithms with theoretical convergence analysis . 3.1 PROBLEM FORMULATION . Given two finite sets P and C at the initial time step t = 0 , we let two sets denote the sets of indices for previously learned data points and upcoming data points , respectively . Note that the task description for a continual learner is two separate sets . In this section , we will show a convergence analysis of the model parameter that we have trained on P and starts to learn C. Thus , we simply denote a data stream of continual learning as two consecutive sets P and C. We consider our goal as a smooth nonconvex finite-sum optimization problem with two objectives min x∈Rd F ( x ) = f ( x ) + g ( x ) = 1 nf ∑ i∈P fi ( x ) + 1 ng ∑ j∈C gj ( x ) , ( 6 ) where fi ( x ) and gj ( x ) denote the objectives of data points i ∈ P and j ∈ C , respectively . In addition , nf and ng are the numbers of elements for P and C. To ease exposition , we use a different notation gj ( x ) for a data point j ∈ C , which is usually the same objective function for a data point i ∈ P . To formulate a theoretical convergence analysis of continual learning , we consider a replay memory based method of which memory is a subset of P ∪ C. Let a random variable Mt ⊂ P ∪ C be the replay memory at time step t ∈ [ 0 , T ] , whose union is of the form M : = ∪tMt . We focus both the episodic memory and the replay memory with dropping rule . The episodic memory based methods include GEM ( Aljundi et al. , 2019 ) , A-GEM ( Chaudhry et al. , 2019a ) , and ORTHOG-SUBSPACE ( Chaudhry et al. , 2020 ) . ER-Reservoir ( Chaudhry et al. , 2019b ) is a replay memory based method with dropping rule , which replaces the dropped sample d ∈Mt with a sample in the stream for C. We now define the gradient update of continual learning xt+1 = xt − αHt∇fIt ( xt ) − βHt∇gJt ( xt ) , ( 7 ) where It ⊂Mt and Jt ⊂ C denote the mini-batches from the replay memory and the current data stream , respectively . Here , Ht is the union of It and Jt . The adaptive learning rates of ∇fIt ( xt ) and ∇gJt ( xt ) are denoted by αHt and βHt which are the functions of Ht . Strictly speaking , the mini-batch It from Mt might contain a datapoint d ∈ C for ER-Reservoir . We describe the details of the problem in Appendix B and assume that the notation It indicates a subset of P for convenience . Equation 7 is a generalized version of continual learning algorithms , which is our novelty to prove the convergence rates in the nonconvex setting for the proposed method , A-GEM , and ER-Reservoir later .

Summary: This paper provides a convergence rate characterization for a continual learning problem where the objective functions (for two consecutive tasks) are generally nonconvex and have a finite-sum form over data samples (which belong to current and previous tasks). The analysis uses the tools from nonconvex optimization, with a goal to find a stationary point for current and previous tasks. Under the setting of continual learning, the authors consider the standard replay memory based method, focusing on both the episodic memory and the replay memory with sample dropping. For the analysis, the authors first address two important types of errors: gradient estimation bias at time t and the catastrophic forgetting error. They show that for these two memory based methods, with a good initialization of replay memory, the gradient estimation error vanishes. They further show that in term of the convergence rate of previous task, i.e., $f$ in its context, the catastrophic forgetting error seems to be inevitable and can be even unbounded. To address this error, they further propose some adaptive learning rate schemes, which show some effectiveness in several experiments.