Title: LEARNING WITH LONG-TERM REMEMBERING: FOL-LOWING THE LEAD OF MIXED STOCHASTIC GRADI-ENT

Abstract: Current deep neural networks can achieve remarkable performance on a single task. However, when the deep neural network is continually trained on a sequence of tasks, it seems to gradually forget the previous learned knowledge. This phenomenon is referred to as catastrophic forgetting and motivates the field called lifelong learning. The central question in lifelong learning is how to enable deep neural networks to maintain performance on old tasks while learning a new task. In this paper, we introduce a novel and effective lifelong learning algorithm, called MixEd stochastic GrAdient (MEGA), which allows deep neural networks to acquire the ability of retaining performance on old tasks while learning new tasks. MEGA modulates the balance between old tasks and the new task by integrating the current gradient with the gradient computed on a small reference episodic memory. Extensive experimental results show that the proposed MEGA algorithm significantly advances the state-of-the-art on all four commonly used lifelong learning benchmarks, reducing the error by up to 18%.

1. INTRODUCTION:

A significant step towards artificial general intelligence (AGI) is to enable the learning agent to acquire the ability of remembering past experiences while being trained on a continuum of tasks. Current deep neural networks are capable of achieving remarkable performance on a single task (Goodfellow et al., 2016) . However when the network is retrained on a new task, its performance drops drastically on previously trained tasks, a phenomenon which is referred to as catastrophic forgetting (Ratcliff, 1990; Robins, 1995; French, 1999; Kirkpatrick et al., 2017) . In stark contrast, human cognitive system is capable of acquiring new knowledge without damaging previously learned experiences. It is thus of great importance to develop algorithms to allow deep neural networks to achieve continual learning capability (i.e., avoiding catastrophic forgetting).

The problem of catastrophic forgetting motivates the field called lifelong learning (Thrun & Mitchell, 1995; Kirkpatrick et al., 2017; Parisi et al., 2019) . A central dilemma in lifelong learning is how to achieve a balance between the performance on old tasks and the new task (Robins, 1995; Kirkpatrick et al., 2017) . During the process of learning the new task, the originally learned knowledge will typically be disrupted, which leads to catastrophic forgetting. On the other hand, a learning algorithm biasing towards old tasks will interfere with the learning of the new task. Several lines of methods are proposed recently to address this issue. Examples include regularization based methods (Kirkpatrick et al., 2017; Zenke et al., 2017) , knowledge transfer based methods (Rusu et al., 2016) , episodic memory based methods (Lopez-Paz et al., 2017; Chaudhry et al., 2018b; Riemer et al., 2018) . However, the existing methods require over-parameterized neural networks (Kirkpatrick et al., 2017; Chaudhry et al., 2018a) or are not flexible enough handle the stochastic nature of the learning process (Lopez-Paz et al., 2017; Chaudhry et al., 2018b) since they did not explicitly consider the model's performance on the current task and old tasks in the learning process.

In this paper, we propose a novel and effective lifelong learning algorithm, called MixEd stochastic GrAdient (MEGA), to address the catastrophic forgetting problem. We cast the problem of balancing the performance on old tasks and the new task as an optimization problem with composite objective. Our formulation is general and closely related to several recently proposed lifelong learning algorithms (Lopez-Paz et al., 2017; Chaudhry et al., 2018b; Riemer et al., 2018) . We approximately solve the optimization problem using one-step stochastic gradient descent with the standard gradient replaced by the proposed mixed stochastic gradient. The mixed stochastic gradient is derived from the gradients computed on the data of the current task and an episodic memory which stores a small subset of observed examples from old tasks (Lopez-Paz et al., 2017; Chaudhry et al., 2018b; Riemer et al., 2018) . Based on our derivation, the direction of the mixed stochastic gradient balances the loss on old tasks and the new task in an adaptive manner. Therefore, the proposed MEGA algorithm allows deep neural networks to learn new tasks while avoiding catastrophic forgetting.

Our contributions are as follows.

(1) We propose a novel and effective algorithm, called MEGA, for lifelong learning problems. (2) We extensively evaluate our algorithm using conventional lifelong learning benchmark datasets, and the results show that the proposed MEGA algorithm significantly advances the state-of-the-art performance across all the datasets. MEGA achieves an average accuracy of 91.21±0.10% on Permuted MNIST, which is 2% better than the previous state-of-the-art model. On Split CIFAR, our proposed MEGA achieves an average accuracy of 66.12±1.93%, which is about 5% better than the state-of-the-art method. Specially, on the Split CUB dataset, MEGA achieves an average accuracy of 80.58±1.94%, which surpasses the multi-task baseline which is previously believed as an upper bound performance of lifelong learning algorithms (Chaudhry et al., 2018b) . ( 3) Finally, we also show that the proposed MEGA algorithm can handle increasingly non-stationary settings when the number of tasks becomes significantly larger.

2. RELATED WORK:

Improving the continual learning ability of neural network is a prerequisite to extending it to more practical vision tasks. Several lines of lifelong learning methods are proposed recently, we categorize them into different types based on the methodology, Regularization based approaches: EWC (Kirkpatrick et al., 2017) adopts Fisher information matrix to prevent important weights for old tasks from changing drastically. In PI (Zenke et al., 2017) , the authors introduce intelligent synapses which endows each individual synapse with a local measure of "importance" to avoid old memories from being overwritten. RWALK (Chaudhry et al., 2018a) utilizes a KL-divergence based regularization for preserving knowledge of old tasks. While in MAS (Aljundi et al., 2018) the importance measure for each parameter of the network is computed based on how sensitive the predicted output function is to a change in this parameter.

Knowledge transfer based methods: PROG-NN (Rusu et al., 2016) is a representative knowledge transfer based lifelong method. In PROG-NN, a new "column" with lateral connections to previous hidden layers is added for each new task. The lateral connections allow the new task to leverage the knowledge extracted from the old tasks. Recently, in Lee et al. (2019) , the authors proposed a method to leverage unlabeled data in the wild to avoid catastrophic forgetting using knowledge distillation. The authors introduce global distillation for incorporating unlabeled data in the learning process.

Episodic memory based approaches::

In episodic memory based lifelong learning methods, a small episodic memory is used for storing a subset of the examples from old tasks. Different episodic memory based approaches differ in the way of computing the reference gradients of the episodic memory. GEM (Lopez-Paz et al., 2017) computes the reference gradients using each individual previous tasks while in AGEM (Chaudhry et al., 2018b) the reference gradient is computed on the episodic memory by randomly sampling a batch of examples. MER a (Riemer et al., 2018) is recently proposed lifelong learning algorithm which maintains an experience replay style memory with reservoir sampling and employs a meta-learning style training strategy. Several methods are proposed recently to improve the episodic memory based approaches. In (Aljundi et al., 2019) , a line of methods are proposed to select important samples to store in the memory in order to reduce memory size. Instead of storing samples, in (Farajtabar et al., 2019) the authors proposed Orthogonal Gradient Descent (OGD) which projects the gradients on the new task onto a subspace in which the projected gradient will not affect the model's output on old tasks and is still useful for learning the new task.

There are also some other lifelong learning methods which do not fall in above categories. In (He & Jaeger, 2018) , the authors proposed a variant of the standard back-propagation algorithm called conceptor aided backprop that shields gradients against degradation of performance on old task. Zeng et al. (2019) proposed orthogonal weights modification (OWM) to enables networks to continually learn different mapping rules in a context-dependent way.

Our proposed method belongs to episodic memory based appraoch and is most related to Chaudhry et al. (2018b) . Our work differs from Chaudhry et al. (2018b) in two aspects. First, we introduce a more effective to modify the direction of the current gradient which put equal emphasis on both old tasks and the new task. Second, we explicitly consider the performance of the model on old tasks and the new task in the process of modifying the gradient direction. Our method is also related to several multi-task learning works (Sener & Koltun, 2018; Kendall et al., 2018; Chen et al., 2017) . In (Sener & Koltun, 2018; Kendall et al., 2018) , the authors aim at achieving a good balance between different tasks by learning to weigh the loss on each task . In contrast, our approach directly leverages loss information in the context of lifelong learning for overcoming catastrophic forgetting. Compared with (Chen et al., 2017) , instead of using the gradient norm information, our method and Lopez-Paz et al. (2017); Chaudhry et al. (2018b) focus on modifying the direction of the current gradient.

3. LIFELONG LEARNING:



3.1. PROBLEM STATEMENT:

Lifelong learning (LLL) (Rusu et al., 2016; Kirkpatrick et al., 2017; Lopez-Paz et al., 2017; Chaudhry et al., 2018b) considers the problem of learning a new task without degrading performance on old tasks, i.e., to avoid catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017) . Suppose there are T tasks which are characterized by T datasets: {D 1 , D 2 , .., D T }. Each dataset D t consists of a list of triplets (x i , y i , t), where y i is the label of i-th example x i , and t is a task descriptor that indicates which task the example comes from. Similar to supervised learning, each dataset D t is split into a training set D tr t and a test set D te t . In the learning protocol introduced in Chaudhry et al. (2018b) , the tasks are separated into D CV = {D 1 , D 2 , ..., D T CV } and D EV = {D T CV +1 , D T CV +2 , ..., D T }. D CV is used for cross-validation to search for hyperparameters. D EV is used for actual training and evaluation. As pointed out in Chaudhry et al. (2018b) , some regularization-based lifelong learning algorithms, e.g., Elastic Weight Consolidation (Kirkpatrick et al., 2017) , are sensitive to the choice of the regularization parameters. Introducing D CV can help find the best regularization parameter without exposing the actual training and evaluation data. While searching for the hyperparameters, we can have multiple passes over the examples in D CV , the training is performed on D EV with only a single pass over the examples (Lopez-Paz et al., 2017; Chaudhry et al., 2018b) .

In lifelong learning, a given model f (x; w) is trained sequentially on a series of tasks {D T CV +1 , D T CV +2 , ..., D T }. When the model f (x; w) is trained on task D t , the goal is to predict the labels of the examples in D te t by minimizing the empirical loss t (w) on D tr t in an online fashion without suffering accuracy drop on {D te T CV +1 , D te T CV +2 , ..., D te t }.

3.2. EVALUATION METRICS:

Average Accuracy and Forgetting Measure (Chaudhry et al., 2018a) are common used metrics for evaluating performance of lifelong learning algorithms. In Chaudhry et al. (2018b) , the authors introduce another metric, called Learning Curve Area (LCA), to assess the learning speed of different lifelong learning algorithms. In this paper, we further introduce a new evaluation metric, called Long-term Remembering (LTR), to characterize the ability of lifelong learning algorithms for remembering the performance of tasks trained in the far past.

Suppose there are M k mini-batches in the training set of task D k . Similar to Chaudhry et al. (2018b) , we define a k,i,j as the accuracy on the test set of task D j after the model is trained on the i-th minibatch of task D k . Generally, suppose the model f (x; w) is trained on a sequence of T tasks {D 1 , D 2 , ..., D T }. Average Accuracy and Forgetting Measure after the model is trained on the task D k are defined as

A k = 1 k k j=1 a k,M k ,j F k = 1 k -1 k-1 j=1 f k j (1)

where f k j = max l∈{1,2,..,k-1} a l,M l ,ja k,M k ,j . Clearly, A T is the average test accuracy and F T assesses the degree of accuracy drop on old tasks after the model is trained on all the T tasks. Learning Curve Area (LCA) (Chaudhry et al., 2018b) at β is defined as,

LCA β = 1 β + 1 β b=0 Z b (2)

where

Z b = 1 T T k=1 a k,b,k .

Intuitively, LCA measures the learning speed of different lifelong learning algorithms. A higher value of LCA indicates that the model learns quickly. We refer the readers to Chaudhry et al. (2018b) for more details about LCA.

All the metrics introduced above fail to capture one important aspect of lifelong learning algorithms, that is, the ability to retain performance on the tasks trained in the far past. In this paper we introduce a new metric, called Long-Term Remembering (LTR), which is defined as

EQUATION

After the model is trained on all the T tasks, LTR quantifies the accuracy drop on task D j relative to a j,Mj ,j . The coefficient Tj emphasizes more on the tasks trained earlier. Different algorithms can have the same average accuracy but very different LTR based on their ability to maintain the performance on the past tasks (a.k.a, long-term remembering).

4. MIXED STOCHASTIC GRADIENT:

In this section, we introduce the proposed Mixed Stochastic Gradient (MEGA) algorithm. Following previous works (Lopez-Paz et al., 2017; Chaudhry et al., 2018b) , when the model is trained on the t-th task, an episodic memory M is used for storing a subset of the examples from all the old tasks k < t. The main idea of MEGA is to minimize the loss on the episodic memory and the t-th task by iteratively moving in the direction of the proposed mixed stochastic gradient.

In the lifelong learning setting, the learning of task t is conducted over a single pass of the training examples in an online fashion. To establish the tradeoff between the performance on old tasks and the t-th task, we consider the following optimization problem with composite objective:

EQUATION

where w ∈ R d is the parameter of the model, ξ, ζ are random variables with finite support, t (w) is the expected training loss of the t-th task, ref (w) is the expected loss calculated on the data stored in the episodic memory, α 1 and α 2 are hyperparameters which control the relative importance of t (w) and ref (w) . Intuitively, a larger ref (w) signifies catastrophic forgetting. Note that during the learning process of each task, every data sample is i.i.d., and hence the current example of task t could be viewed as a random sample due to online-to-batch conversion argument (Cesa-Bianchi et al., 2004) . In the lifelong learning setting, t (w; ξ) is the training loss calculated on the current mini-batch (controlled by ξ) of task t, ref (w; ζ) is the loss calculated on a random mini-batch (controlled by ζ) sampled from the episodic memory. In the traditional online learning, the loss is calculated only based on the current received example but not on historical samples, and hence α 1 = 1, α 2 = 0. In this case, the weights are only optimized for the current task while ignoring previous tasks which leads to catastrophic forgetting. If α 2 > 0, this formulation naturally involves examples from old tasks. In lifelong learning, since we need to consider performance on both old tasks and the current task, we typically do not consider the degenerate case when α 2 = 0.

We use w t k to denote the weight when the model is being trained on the k-th mini-batch of task t. Clearly, both t (w) and ref (w) are determined by w t k during training. This implies that the relative value of t (w) and ref (w) is changing between mini-batches. Therefore, α 1 and α 2 should be adjusted adaptively based on w t k in order to achieve a good balance between old tasks and the current task. To this end, with a little abuse of notation, we define two parameter-dependent functions α 1 , α 2 : R d → R + to characterize the relative importance of the current task and old tasks. Mathematically, we propose to use the following update:

EQUATION

Algorithm 1 MEGA, the proposed algorithm for lifelong learning. T is the number of tasks. n t is the number of mini-batches of task t. M is the episodic memory. ξ t k is the k-th mini-batch of task t and y t k is the corresponding label. ζ t k is a random mini-batch from the episodic memory. w t k stands for the parameter after k-th mini-batch during the training of t-th task. 

for k ← 1 to n t do 4: if M = {} then 5: ζ t k ← SAMPLE(M ) 6: θ ← arccos(∇ t (w t k ; ξ t k ), ∇ ref (w t k ; ζ t k )) 7:

Solve the optimization problem in Eq. ( 9) to obtain θ.

8::

Obtain α 1 and α 2 as in Appendix A.1. 9:

else 10:

Set α 1 = 1 and α 2 = 0.

11:

end if 12:

Update w t k using Eq. 6.

13:

M ← M (ξ t k , y t k ) 14:

Discard the samples added initially if M is full.

15:

end for 16: end for where α 1 , α 2 : R d → R + are real-valued mappings.

We employ first-order methods (e.g., stochastic gradient descent) to approximately solve the optimization problem (5 which naturally motivates us to design the MixEd stochastic GrAdient (MEGA) algorithm. MEGA is doing update (6) to approximately solve ( 5), where one-step stochastic gradient descent is performed with the initial point set to be w t k :

EQUATION

)

where η is the learning rate, ξ 6) is to define well-behaved mappings α 1 (•) and α 2 (•) which are consistent with the goal of lifelong learning. To this end, we introduce two approaches, angle-based approach (Section 4.1) and a direct approach (Section 4.2) to address this difficulty. It is worth mentioning that several recent advances on lifelong learning (Lopez-Paz et al., 2017; Chaudhry et al., 2018b; Riemer et al., 2018) have close relationship with the angle-based approach in our MEGA framework which will be illustrated in Section 4.1.

4.1. ANGLE-BASED APPROACH:

Note that the mixed stochastic gradient is a linear combination of ∇ t (w t k ; ξ t k ) and ∇ ref (w t k ; ζ t k ). While keeping the magnitude the same as ∇ t (w t k ; ξ t k ), geometrically the mixed stochastic gradient can be viewed as an appropriate rotation of ∇ t (w t k ; ξ t k ) by a desired angle θ. This perspective leads to the angle-based approach in our framework. The key idea of the angle-based approach is to first appropriately rotate the stochastic gradient calculated on the current task (i.e., ∇ t (w t k ; ξ t k )) by an angle θ t k , and then use the rotated vector as the mixed stochastic gradient to conduct the update (6) in each mini-batch. For simplicity, we omit the subscript k and superscript t later on unless specified.

We use g mix to denote the desired mixed stochastic gradient which has the same magnitude as ∇ t (w; ξ). Specifically, we look for the mixed stochastic gradient g mix which direction aligns well with both ∇ t (w; ξ) and ∇ ref (w; ζ). Mathematically, we want to maximize

EQUATION

which is equivalent to find an angle θ such that

EQUATION

where θ ∈ [0, π] is the angle between ∇ t (w; ξ) and ∇ ref (w; ζ). To capture the relative importance of the current task and old tasks which is crucial for lifelong learning, we introduce t (w; ξ) and ref

EQUATION

Here we give some discussions of several special cases of Eq. ( 9),

• When ref (w; ζ) = 0, then θ = 0, and in this case α 1 (w) = 1, α 2 (w) = 0 in (6), which means the mixed stochastic gradient reduces to ∇ t (w; ξ). In the lifelong learning setting, ref (w; ζ) = 0 implies that there is almost no catastrophic forgetting, and hence we can update the model parameters exclusively for the current task by moving in the direction of ∇ t (w; ξ).

• When t (w; ξ) = 0, then θ = θ, and in this case

α 1 (w) = 0, α 2 (w) = ∇ t (w; ξ) 2 / ∇ ref (w; ζ) 2 , provided that ∇ ref (w; ζ) 2 = 0 (define 0/0=0).

This means the direction of the mixed stochastic gradient is the same as the stochastic gradient calculated on the data in the episodic memory (i.e., ref (w; ζ)). In the lifelong learning setting, this update can help improve the performance on old tasks, i.e., avoid catastrophic forgetting.

In the general case, we assume ref (w; ζ) and t (w; ξ) are both positive (the edge cases are covered in the above discussion). Since the optimization problem ( 9) is possibly nonconvex, we propose to use projected gradient ascent to approximately solve it. Mathematically, we do multiple updates using the following formula,

EQUATION

)

where Π is the projection operator, g(β) =sin(β) + ref (w;ζ) t(w;ξ) sin( θβ), r = ref(w;ζ) t(w;ξ) . It is not difficult to show that the smoothness parameter of the function g (β) = cos(β) + ref (w;ζ) t(w;ξ) cos( θβ) is 1 + r, and hence projected gradient ascent can converge to a stationary point (Nesterov, 1998) . To avoid getting stuck on saddle point or local minima or local maxima, we can use multiple random starting points and select the one which achieves the largest function value. This strategy is proven to be successful in our experiments.

After we find the desired angle θ, it is easy to obtain α 1 (w) and α 2 (w) in Eq. ( 6). For details, please refer to Appendix A.1. Please see Algorithm 1 for the summary of the algorithm. Chaudhry et al. (2018b) in two aspects. First, we explicitly maximize the correlation between the mixed stochastic gradient g mix and the current gradient ∇ t (w; ξ) (calculated on current data), and the correlation between the mixed stochastic gradient g mix and the reference gradient ∇ ref (w; ζ) (calculated on memory data) as in Eq (7). Intuitively, the direction of g mix will not bias towards the current task or old tasks (since it is easy to see that the angle between g mix and ∇ t (w; ξ), and the angle between g mix and ∇ ref (w; ζ) are both acute (except the edge case where the angle between ∇ t (w; ξ) and ∇ ref (w; ζ) is 180 degree). By following g mix , each update tends to degrade the loss on both current task and old tasks. While in previous works (Lopez-Paz et al., 2017; Chaudhry et al., 2018b) , the corresponding g mix is found by projecting the current gradient to be perpendicular to ∇ ref (w; ζ) when the angle between them is obtuse (in this case, g mix • ∇ ref (w; ζ) = 0). Intuitively, both GEM and A-GEM put more emphasis on the current task (when g mix is perpendicular to ∇ ref (w; ζ), the update following the direction of g mix will not directly reduce the loss on old tasks). Second, we introduce the loss on the current batch and on the sampled batch from episodic memory. This allows us to better balance the performance on the current task and old tasks. Since it is preferable to put more emphasis on the tasks (task) which achieve(s) low performance.

Comparison with Existing Works:



4.2. DIRECT APPROACH:

We also introduce MEGA-D which is a direct approach for implementing MEGA. In MEGA-D, instead of rotating the stochastic gradient computed on the data in the current mini-batch as in the angle-based approach, we define α 1 (•) and α 2 (•) in the definition of mixed stochastic gradient in a direct manner. Specifically, in the update of (6), we set

α 1 = 1, α 2 = ref (w; ζ)/ t (w; ξ) if t (w; ξ) > 0, and α 1 = 0, α 2 = 1 if t (w; ξ) = 0.

Intuitively, if t (w; ξ) = 0, then it means that the model performs well on the current task and can focus on improving performance on the data stored in the episodic memory, and hence α 1 = 0, α 2 = 1. Otherwise, we keep the balance of the two terms of mixed stochastic gradient according to t (w; ξ) and ref (w; ζ). MEGA-D is a simple variant of MEGA which balances current task and old tasks by only leveraging loss information.

5. EXPERIMENTS:



5.1. DATASETS:

In the experiments, we consider the following four conventional lifelong learning benchmarks,

• Permuted MNIST (Kirkpatrick et al., 2017) : this is a variant of standard MNIST dataset (LeCun et al., 1998) of handwritten digits with 20 tasks. Each task has a fixed random permutation of the input pixels which is applied to all the images of that task. • Split CIFAR (Zenke et al., 2017) : this dataset consists of 20 disjoint subsets of CIFAR-100 dataset (Krizhevsky et al., 2009) , where each subset is formed by randomly sampling 5 classes without replacement from the original 100 classes. • Split CUB (Chaudhry et al., 2018b) : the CUB dataset (Wah et al., 2011) is split into 20 disjoint subsets by randomly sampling 10 classes without replacement from the original 200 classes. • Split AWA (Chaudhry et al., 2018b) : this dataset consists of 20 subsets of the AWA dataset (Lampert et al., 2009) . Each subset is constructed by sampling 5 classes with replacement from a total of 50 classes. Note that the same class can appear in different subsets. As in Chaudhry et al. (2018b) , in order to guarantee that each training example only appears once in the learning process, based on the occurrences in different subsets the training data of each class is split into disjoint sets.

We also include Many Permutations which is a variant of Permuted MNIST to introduce more non-stationality into the learning process. In Many Permutations, there are a total 100 tasks with 200 examples per task. The way to generate the tasks is the same as in Permuted MNIST, that is, a fixed random permutation of input pixels is applied to all the examples for a particular task.

5.2. NETWORK ARCHITECTURES:

To be consistent with the previous works (Lopez-Paz et al., 2017; Chaudhry et al., 2018b) , for Permuted MNIST we adopt a standard fully-connected network with two hidden layers. Each layer has 256 units with ReLU activation. For Split CIFAR we use a reduced ResNet18. For Split CUB and Split AWA, we use a standard ResNet18 (He et al., 2016) .

5.3. BASELINES AND EXPERIMENTAL SETTINGS:

We compare the proposed MEGA with several state-of-the-art lifelong learning methods,

• VAN: in VAN, a single network is trained continuously on a sequence of tasks in a standard supervised learning manner. • MULTI-TASK: in MULTI-TASK, a single network is trained on the shuffled data from all the tasks with a single pass.

• Episodic memory based approach: GEM (Lopez-Paz et al., 2017) and AGEM (Chaudhry et al., 2018b) are episodic memory based approaches which modify the current gradient when its angle between the gradient computed on the episodic memory is obtuse. MER (Riemer et al., 2018) is another recently proposed episodic memory based approach which maintains an experience replay style memory with reservoir sampling and employs a meta-learning style training strategy.

• Regularization-based approaches: EWC (Kirkpatrick et al., 2017) , PI (Zenke et al., 2017) , RWALK (Chaudhry et al., 2018a) and MAS (Aljundi et al., 2018) are regularization-based approaches which prevent the important weights of the old tasks from changing too much.

• Knowledge transfer based approach: in PROG-NN (Rusu et al., 2016) , a new column with lateral connections with previous hidden layers is added for each new task. This allows knowledge transfer between old tasks and the new task.

To be consistent with Chaudhry et al. (2018b) , for episodic memory based approaches, the episodic memory size for each task is 250, 65, 50, and 100, and the batch size for computing the gradients on the episodic memory (if needed) is 256, 1300, 128 and 128 for MNIST, CIFAR, CUB and AWA, respectively. To fill the episodic memory, the examples are chosen uniformly at random for each task as in Chaudhry et al. (2018b) . For each dataset, 17 tasks are used for training and 3 tasks are used for hyperparameter search. For MEGA and MER Riemer et al. (2018) , we do not conduct exhaustive hyperparameter search and reuse the hyperparameters of AGEM (Chaudhry et al., 2018b) . For other baselines, we use best hyperparameters found by Chaudhry et al. (2018b) . For the detailed hyperparameters, please see Appendix G of Chaudhry et al. (2018b) . In MEGA, we solve Eq.9 three times with different random starting points and the update in Eq.10 is repeated for ten iterations. 

6. RESULTS:



6.1. MEGA VS BASELINES:

In Fig. 1 we show the results across different measures on all the benchmark datasets. We have the following observations based on the results. First, the proposed MEGA outperforms all baselines across the benchmarks, except that PROG-NN achieves a slightly higher accuracy on Permuted MNIST. As we can see from the memory comparison, PROG-NN is very memory inefficient since it allocates a new network for each task, thus the number of parameters grows super-linearly with the number of tasks. This becomes problematic when large networks are being used. For example, PROG-NN runs out of memory on Split CUB and Split AWA which prevents it from scaling up to real-life problems. On other datasets, MEGA consistently performs better than all the baselines, from Fig. 2 we can see that on Split CUB, MEGA even surpasses the multi-task baseline which is previously believed as an upper bound performance of lifelong learning algorithms (Chaudhry et al., 2018b) . Second, MEGA achieves the lowest Forgetting Measure across all the datasets which indicates its ability to overcome catastrophic forgetting. Third, the proposed MEGA also obtains a high LCA across all the datasets which shows that MEGA also learns quickly. The evolution of LCA in the first ten mini-batches across all the datasets is shown in Fig. 3 . Last, compared with AGEM (Chaudhry et al., 2018b) , which is the state-of-the-art method for lifelong learning, MEGA has the same memory cost and similar time complexity. For detailed results, please refer to Table 4 and Table 5 in Appendix A.2.

In Fig. 2 we show the evolution of average accuracy during the lifelong learning process. As more tasks are added, while the average accuracy of the baselines generally drops due to catastrophic forgetting, MEGA can maintain and even improve its performance. This shows MEGA has a clear advantage over the state-of-the-art lifelong learning methods.

6.2. LONG-TERM REMEMBERING:

In Table 1 we show the results of Long-term Remembering (LTR) of some representative lifelong learning methods on different datasets. As stated before, an algorithm with low LTR indicates that it can maintain the performance on the tasks trained initially. From Table 1 we can see that the proposed MEGA algorithm achieves the lowest LTR across all the datasets. This demonstrates that MEGA can learn tasks in succession without forgetting the initial tasks which is crucial for realworld lifelong learning applications. 

We show the comparison of MEGA and MEGA-D in Table 2 . The performance of MEGA-D is on par with MEGA across all the datasets. This shows that it is important to explicitly consider the loss on the episodic memory in order to overcome catastrophic forgetting. 

7. CONCLUSION:

In this paper, we present a lifelong learning algorithm called MEGA which achieves the state-ofthe-art performance across all the benchmark datasets. In MEGA, we cast the lifelong learning problem as an optimization problem with composite objective and solve it with the proposed mixed stochastic gradient. We also propose an important lifelong learning metric called LTR to characterize the ability of lifelong learning algorithms to maintain performance on the tasks trained in the far past.

Extensive experimental results show that the proposed MEGA achieves superior results across all the considered metrics and establishes the new state-of-the-art on all the datasets.

A APPENDIX:

A.1 SOME DERIVATIONS For notation simplicity, we use g, ĝ, a, b to replace ∇ t (w; ξ), ∇ ref (w; ζ), α 1 (w), α 2 (w) respectively. If g = ĝ, then a = 1, b = 0. Otherwise, the goal is to solve

EQUATION

The solution of ( 11) is

EQUATION

A.2 RESULT TABLES 

A.3 ABLATION STUDIES:

In this section, we include ablation studies to why the proposed method can improve upon A-GEM Chaudhry et al. (2018b) . We only consider maximizing the correlation between the mixed stochastic gradient g mix and the current gradient ∇ t (w; ξ) (calculated on current data), and the correlation between the mixed stochastic gradient g mix and the reference gradient ∇ ref (w; ζ) (calculated on memory data) as in Eq (7). We call this method MEGA-C. The only difference between MEGA-C and A-GEM is the way of modifying the gradient direction. We compare MEGA-C to MEGA and A-GEM in the Table 6 . The experimental settings are the same as in Section 5.3. From Table 6 we observe that except on Split CIFAR, MEGA-C outperforms A-GEM on all the datasets. This demonstrate the benefits of the proposed approach for modifying gradient direction. By considering the loss information as in MEGA, we further improve the results on all the datasets. This shows that both of the two components (gradient direction modification and loss information) contribute to the improvement of the proposed approach.

A.4 LEARNING PROCESS

In this section we report the result matrices for MEGA and AGEM (Chaudhry et al., 2018b) A.4.2 SPLIT CIFAR MEGA 0.6472 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6260 0.5824 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6324 0.5700 0.6300 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6236 0.5496 0.5624 0.6452 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6132 0.5612 0.6048 0.6736 0.6960 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6140 0.5628 0.5692 0.6632 0.6792 0.7688 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5780 0.5500 0.5864 0.6364 0.6792 0.7420 0.6868 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5756 0.5308 0.5764 0.6292 0.6500 0.6820 0.6580 0.6580 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6212 0.5704 0.5876 0.6376 0.6600 0.7056 0.6636 0.6916 0.7376 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5992 0.5580 0.5828 0.6212 0.6528 0.6512 0.6496 0.6700 0.7276 0.6732 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 AGEM 0.6772 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5948 0.5764 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6324 0.5828 0.6432 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5980 0.5384 0.5396 0.6456 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5864 0.5404 0.5576 0.6436 0.7004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5728 0.5392 0.5068 0.5940 0.6344 0.7180 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5572 0.5404 0.5308 0.6224 0.6116 0.6520 0.6688 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6064 0.5356 0.5492 0.5872 0.6164 0.6532 0.6296 0.6724 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6060 0.5472 0.5528 0.6236 0.5920 0.6348 0.6076 0.6348 0.6972 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6004 0.5080 0.4960 0.6128 0.5656 0.6356 0.5752 0.6140 0.6580 0.6792 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5992 0.5408 0.5332 0.5964 0.5928 0.6520 0.5928 0.6304 0.6764 0.5916 0.7364 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5748 0.5020 0.5104 0.5936 0.6016 0.6184 0.5772 0.6164 0.6408 0.5688 0.6776 0.6436 0.0000 0.0000 0.0000 0.0000 0.0000 0.6056 0.5100 0.5200 0.5916 0.6012 0.6056 0.5816 0.6060 0.6236 0.5808 0.6288 0.5768 0.7332 0.0000 0.0000 0.0000 0.0000 0.6184 0.5344 0.5308 0.5888 0.6116 0.6188 0.6012 0.6248 0.6136 0.5836 0.6428 0.5688 0.6524 0.7392 0.0000 0.0000 0.0000 0.6012 0.5220 0.5488 0.6008 0.5828 0.6048 0.5728 0.5884 0.6356 0.5740 0.6476 0.5540 0.6520 0.6804 0.6460 0.0000 0.0000 0.5984 0.5360 0.5520 0.5808 0.5704 0.6184 0.6068 0.6108 0.6452 0.5404 0.6520 0.5256 0.6624 0.6512 0.5864 0.7388 0.0000 0.6232 0.5356 0.5412 0.6104 0.6080 0.6248 0.5944 0.5900 0.6492 0.5872 0.6468 0.5352 0.6336 0.6520 0.5908 0.6444 0.7508 A.4.3 SPLIT CUB MEGA 0.4050 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6876 0.5088 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6844 0.7244 0.6263 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7071 0.7445 0.7753 0.6553 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7294 0.7483 0.7941 0.7787 0.6857 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7049 0.7266 0.7712 0.7538 0.7528 0.6366 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7271 0.7488 0.7877 0.7893 0.7861 0.7905 0.6830 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7316 0.7737 0.7920 0.7867 0.7851 0.7956 0.8179 0.6816 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7503 0.7525 0.7860 0.7947 0.7838 0.7818 0.8148 0.7912 0.6934 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7445 0.7475 0.7819 0.7599 0.7680 0.7834 0.7901 0.7858 0.7883 0.7085 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7506 0.7732 0.8053 0.8129 0.8049 0.8099 0.8275 0.7938 0.8041 0.8146 0.7211 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7475 0.7670 0.7860 0.7982 0.7818 0.7892 0.8133 0.7881 0.8014 0.8145 0.7958 0.7144 0.0000 0.0000 0.0000 0.0000 0.0000

:



Figures and Tables:

There are several existing works setting θ in different manners. InLopez-Paz et al. (2017) andChaudhry et al. (2018b), θ = θ if θ ≤ π 2 , and θ = θπ 2 if θ ≥ π 2 . Note that ∇ ref (w; ζ) isdefined differently in Lopez-Paz et al. (2017) and Chaudhry et al. (2018a). In Lopez-Paz et al. (2017), ∇ ref (w; ζ) is calculated on the data of each task separately stored in the episodic memory. While in Chaudhry et al. (2018a), ∇ ref (w; ζ) is computed on a random mini-batch sampled from the episodic memory. Our work differs from Lopez-Paz et al. (2017);

Figure Performance of lifelong learning models across different measures on Permuted MNIST, Split CIFAR, Split CUB and Split AWA.

Figure 2: Evolution of average accuracy during the lifelong learning process.

Figure 3: LCA of first ten mini-batches on different datasets.

t (w t k ; ξ t k ) is the training loss calculated on ξ t k . ref (w t k ; ζ t k ) is the reference loss calculated on ζ t k . α 1 and α 2 are defined in Eq. 5 1: M ← {} 2: for t ← 1 to T do

Table content: <table><tr><td>3:</td></tr></table>

Results of Long-term Remembering (LTR).

Table content: <table><tr><td/><td/><td/><td/><td/><td colspan="6">Methods</td><td>Permuted MNIST</td><td>Split CIFAR</td><td>Split CUB</td><td>Split AWA</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="5">MEGA AGEM EWC VAN</td><td>0.524 ± 0.017 0.716 ± 0.048 3.292 ± 0.135 5.375 ± 0.194</td><td>0.356 ± 0.114 0.643 ± 0.124 2.493 ± 0.427 2.613 ± 0.174</td><td>0.002 ± 0.002 0.456 ± 0.174 1.021 ± 0.210 0.976 ± 0.215</td><td>0.070 ± 0.114 0.178 ± 0.082 0.675 ± 0.214 0.202 ± 0.090</td></tr><tr><td>0</td><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td><td>7</td><td>8</td><td>9</td><td>10</td></tr></table>

Comparison of MEGA and MEGA-D.We show the results on Many Permutation in Table3. Compared with Permuted MNIST, Many Permutation has 5 times more tasks (100) and much fewer examples per task (200). This introduces more non-stationarity into the learning process. Nevertheless, the proposed MEGA achieves competitive results in this setting. Compared with MER which achieves similar results to MEGA, MEGA is much more time efficient since it does not rely on the meta-learning procedure. Update MEGA-D result in the table.

Table content: <table><tr><td>Methods</td><td colspan="2">Permuted MNIST</td><td colspan="2">Split CIFAR</td><td colspan="3">Split CUB</td><td/><td>Split AWA</td></tr><tr><td/><td>AT (%)</td><td>FT</td><td>AT (%)</td><td>FT</td><td>AT (%)</td><td/><td>FT</td><td/><td>AT (%)</td><td>FT</td></tr><tr><td>MEGA MEGA-D</td><td>91.21 ± 0.10 91.14 ± 0.16</td><td>0.05 ± 0.01 0.05 ± 0.02</td><td>66.12 ± 1.93 66.72 ± 1.50</td><td>0.05 ± 0.02 0.04 ± 0.01</td><td colspan="2">80.58 ± 1.94 79.68 ± 2.37</td><td colspan="2">0.01 ± 0.01 0.01 ± 0.02</td><td>54.28 ± 4.84 54.67 ± 4.69</td><td>0.04 ± 0.04 0.04 ± 0.03</td></tr><tr><td colspan="3">6.4 MANY PERMUTATION</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>Methods</td><td>VAN</td><td>EWC</td><td>GEM</td><td>AGEM</td><td/><td>MER</td><td/><td>MEGA-D</td><td>MEGA</td></tr><tr><td colspan="3">Average Accuracy (%) 32.62 ± 0.43</td><td>33.46 ± 0.46</td><td>56.76 ± 0.29</td><td>34.15 ± 0.55</td><td colspan="2">62.52 ± 0.32</td><td colspan="2">56.52± 0.43</td><td>62.48 ± 0.51</td></tr></table>

Results on Many Permutation.

Table content: <table/>

The results of Average Accuracy (A T ), Forgetting Measure (F T ) and LCA of different methods on Permuted MNIST and Split CIFAR. The results are averaged across 5 runs with different random seeds.

Table content: <table><tr><td>Methods</td><td/><td>Permuted MNIST</td><td/><td/><td>Split CIFAR</td><td/></tr><tr><td/><td>A T (%)</td><td>F T</td><td>LCA 10</td><td>A T (%)</td><td>F T</td><td>LCA 10</td></tr><tr><td>VAN</td><td>47.55±2.37</td><td>0.52±0.026</td><td>0.259±0.005</td><td>40.44±1.02</td><td>0.27±0.006</td><td>0.309±0.011</td></tr><tr><td>EWC</td><td>68.68±0.98</td><td>0.28±0.010</td><td>0.276±0.002</td><td>42.67±4.24</td><td>0.26±0.039</td><td>0.336±0.010</td></tr><tr><td>MAS</td><td>70.30±1.67</td><td>0.26±0.018</td><td>0.298±0.006</td><td>42.35±3.52</td><td>0.26±0.030</td><td>0.332±0.010</td></tr><tr><td>RWALK</td><td>85.60±0.71</td><td>0.08±0.007</td><td>0.319±0.003</td><td>42.11±3.69</td><td>0.27±0.032</td><td>0.334±0.012</td></tr><tr><td>MER</td><td>-</td><td>-</td><td>-</td><td>37.27±1.68</td><td>0.03±0.030</td><td>0.051±0.101</td></tr><tr><td>PROG-NN</td><td>93.55±0.06</td><td>0.0±0.000</td><td>0.198±0.006</td><td>59.79±1.23</td><td>0.0±0.000</td><td>0.208±0.002</td></tr><tr><td>GEM</td><td>89.50±0.48</td><td>0.06±0.004</td><td>0.230±0.005</td><td>61.20±0.78</td><td>0.06±0.007</td><td>0.360±0.007</td></tr><tr><td>AGEM MEGA</td><td>89.32±0.46 91.21±0.10</td><td>0.07±0.004 0.05±0.001</td><td>0.277±0.008 0.283±0.004</td><td>61.28±1.88 66.12±1.94</td><td>0.09±0.018 0.06±0.015</td><td>0.350 ±0.013 0.375±0.012</td></tr></table>

The results of Average Accuracy (A T ), Forgetting Measure (F T ) and LCA of different methods on Split CUB and Split AWA. The results are averaged across 10 runs with different random seeds.

Table content: <table><tr><td>Methods</td><td/><td>Split CUB</td><td/><td/><td>Split AWA</td><td/></tr><tr><td/><td>A T (%)</td><td>F T</td><td>LCA 10</td><td>A T (%)</td><td>F T</td><td>LCA 10</td></tr><tr><td>VAN</td><td>53.89±2.00</td><td>0.13±0.020</td><td>0.292±0.008</td><td>30.35±2.81</td><td>0.04±0.013</td><td>0.214±0.008</td></tr><tr><td>EWC</td><td>53.56±1.67</td><td>0.14±0.024</td><td>0.292±0.009</td><td>33.43±3.07</td><td>0.08±0.021</td><td>0.257±0.011</td></tr><tr><td>MAS</td><td>54.12±1.72</td><td>0.13±0.013</td><td>0.293±0.008</td><td>33.83±2.99</td><td>0.08±0.022</td><td>0.257±0.011</td></tr><tr><td>RWALk</td><td>54.11±1.71</td><td>0.13±0.013</td><td>0.293±0.009</td><td>33.63±2.64</td><td>0.08±0.023</td><td>0.258±0.011</td></tr><tr><td>PI</td><td>55.04±3.05</td><td>0.12±0.026</td><td>0.292±0.010</td><td>33.86±2.77</td><td>0.08±0.022</td><td>0.259±0.011</td></tr><tr><td>AGEM</td><td>61.82±3.72</td><td>0.08±0.021</td><td>0.302±0.011</td><td>44.95±2.97</td><td>0.05±0.014</td><td>0.287±0.012</td></tr><tr><td>MEGA</td><td>80.58±1.94</td><td>0.01±0.017</td><td>0.311±0.010</td><td>54.28±4.84</td><td>0.05±0.040</td><td>0.305±0.015</td></tr></table>

Comparison of MEGA-C, MEGA and A-GEM

Table content: <table><tr><td>Methods</td><td>Permuted MNIST</td><td>Split CIFAR</td><td>Split CUB</td><td>Split AWA</td></tr><tr><td/><td>A T (%)</td><td>A T (%)</td><td>A T (%)</td><td>A T (%)</td></tr><tr><td>MEGA MEGA-C A-GEM</td><td>91.21 ± 0.10 91.15 ± 0.12 89.32 ± 0.46</td><td>66.12 ± 1.93 58.04 ± 1.89 61.28 ± 1.88</td><td>80.58 ± 1.94 68.60 ± 1.98 61.82 ± 3.72</td><td>54.28 ± 4.84 47.95 ± 4.54 44.95 ± 2.97</td></tr></table>

on each dataset. The entry (i, j) of the matrix is the test accuracy of the j-th task after the model is trained on the i-th task.

Table content: <table><tr><td>0.8322 0.8700 0.8644 0.8493 0.8765 0.8787 0.8904 0.8848 0.8883 0.8979 0.9110 0.9158 0.9177 0.9299 0.9403 0.9609 0.1179</td></tr><tr><td>0.8438 0.8603 0.8555 0.8488 0.8864 0.8785 0.8798 0.8702 0.8916 0.8968 0.9076 0.9094 0.9092 0.9228 0.9228 0.9463 0.9551</td></tr><tr><td>A.4.1 PERMUTED MNIST</td></tr><tr><td>MEGA</td></tr><tr><td>0.9613 0.1091 0.1229 0.0832 0.1374 0.0708 0.0907 0.1017 0.1165 0.1286 0.0979 0.1182 0.1188 0.0886 0.0968 0.0854 0.0928</td></tr><tr><td>0.9535 0.9645 0.0895 0.0997 0.1191 0.0685 0.0803 0.1022 0.1165 0.1472 0.1054 0.1112 0.1264 0.1027 0.0872 0.0979 0.0993</td></tr><tr><td>0.9391 0.9556 0.9596 0.0996 0.1020 0.0900 0.0807 0.1083 0.0959 0.1400 0.1001 0.1012 0.1096 0.1085 0.0977 0.0716 0.0768</td></tr><tr><td>0.9295 0.9473 0.9527 0.9477 0.1113 0.0725 0.0856 0.1033 0.0884 0.1209 0.0847 0.1149 0.1285 0.0939 0.1193 0.0867 0.0824</td></tr><tr><td>0.9206 0.9405 0.9437 0.9569 0.9611 0.0785 0.0884 0.1113 0.0926 0.1189 0.0936 0.1337 0.1544 0.1154 0.1282 0.1010 0.0994</td></tr><tr><td>0.9119 0.9347 0.9378 0.9481 0.9547 0.9594 0.0916 0.1200 0.1013 0.1042 0.0908 0.1380 0.1415 0.1199 0.1210 0.0908 0.0847</td></tr><tr><td>0.9083 0.9261 0.9348 0.9419 0.9478 0.9556 0.9575 0.1092 0.1040 0.1083 0.0863 0.1202 0.1286 0.1177 0.1250 0.0801 0.0931</td></tr><tr><td>0.9100 0.9192 0.9291 0.9332 0.9419 0.9462 0.9527 0.9598 0.1152 0.1132 0.0945 0.1054 0.1248 0.1228 0.1187 0.0945 0.0934</td></tr><tr><td>0.9022 0.9133 0.9215 0.9271 0.9344 0.9381 0.9430 0.9476 0.9551 0.1187 0.1162 0.1119 0.1364 0.1249 0.1108 0.1012 0.1059</td></tr><tr><td>0.8974 0.9074 0.9147 0.9242 0.9289 0.9348 0.9367 0.9404 0.9519 0.9571 0.1102 0.1032 0.1536 0.1261 0.1122 0.1036 0.1093</td></tr><tr><td>0.8957 0.9042 0.9146 0.9193 0.9229 0.9313 0.9321 0.9318 0.9409 0.9545 0.9591 0.1144 0.1368 0.1373 0.1143 0.1092 0.1169</td></tr><tr><td>0.8863 0.8981 0.9056 0.9127 0.9148 0.9220 0.9270 0.9249 0.9335 0.9444 0.9528 0.9564 0.1517 0.1103 0.1051 0.1002 0.1390</td></tr><tr><td>0.8840 0.8992 0.9054 0.9085 0.9149 0.9179 0.9238 0.9198 0.9304 0.9419 0.9441 0.9523 0.9570 0.1292 0.1083 0.0926 0.1301</td></tr><tr><td>0.8808 0.8901 0.8994 0.8986 0.9084 0.9113 0.9168 0.9185 0.9239 0.9360 0.9382 0.9453 0.9522 0.9589 0.1017 0.0941 0.1277</td></tr><tr><td>0.8770 0.8850 0.8957 0.8926 0.9012 0.9101 0.909 0 0.9132 0.9188 0.9301 0.9358 0.9368 0.9430 0.9508 0.9521 0.0946 0.1334</td></tr><tr><td>0.8752 0.8806 0.8911 0.8854 0.8965 0.9070 0.9062 0.9059 0.9145 0.9265 0.9286 0.9338 0.9374 0.9434 0.9462 0.9601 0.1291</td></tr><tr><td>0.8732 0.8765 0.8824 0.8809 0.8945 0.9024 0.9016 0.9007 0.9088 0.9202 0.9228 0.9276 0.9279 0.9376 0.9408 0.9521 0.9556</td></tr><tr><td>AGEM</td></tr><tr><td>0.9613 0.1091 0.1229 0.0832 0.1374 0.0708 0.0907 0.1017 0.1165 0.1286 0.0979 0.1182 0.1188 0.0886 0.0968 0.0854 0.0928</td></tr><tr><td>0.9509 0.9645 0.0956 0.0991 0.1304 0.0696 0.0840 0.1033 0.1219 0.1454 0.1064 0.1133 0.1314 0.1043 0.0883 0.0979 0.0973</td></tr><tr><td>0.9410 0.9545 0.9615 0.0995 0.0964 0.0921 0.0710 0.1126 0.1176 0.1402 0.1112 0.1026 0.1185 0.1204 0.1077 0.0779 0.0761</td></tr><tr><td>0.9299 0.9450 0.9540 0.9546 0.1046 0.0788 0.0959 0.1033 0.1096 0.1266 0.1015 0.1152 0.1476 0.0885 0.1375 0.0984 0.0831</td></tr><tr><td>0.9151 0.9361 0.9425 0.9551 0.9588 0.0803 0.0809 0.1143 0.1063 0.1227 0.1066 0.1253 0.1436 0.1154 0.1131 0.1079 0.0915</td></tr><tr><td>0.9068 0.9312 0.9401 0.9450 0.9566 0.9590 0.0892 0.1189 0.1285 0.1086 0.1007 0.1433 0.1279 0.1179 0.1097 0.0892 0.0865</td></tr><tr><td>0.9015 0.9228 0.9339 0.9385 0.9473 0.9548 0.9586 0.1063 0.1073 0.1102 0.1048 0.1164 0.1291 0.1284 0.1341 0.0854 0.1024</td></tr><tr><td>0.8980 0.9155 0.9248 0.9280 0.9356 0.9403 0.9539 0.9580 0.1015 0.1231 0.1129 0.1125 0.1267 0.1133 0.1220 0.0921 0.0985</td></tr><tr><td>0.8952 0.9055 0.9201 0.9182 0.9273 0.9310 0.9447 0.9435 0.9512 0.1098 0.1374 0.1166 0.1264 0.1064 0.1183 0.0986 0.1048</td></tr><tr><td>0.8846 0.8996 0.9083 0.9154 0.9189 0.9267 0.9363 0.9339 0.9513 0.9558 0.1243 0.1095 0.1179 0.1137 0.1126 0.0945 0.0979</td></tr><tr><td>0.8764 0.8977 0.9011 0.9073 0.9086 0.9167 0.9292 0.9274 0.9386 0.9481 0.9631 0.1116 0.1099 0.1417 0.1100 0.0975 0.1166</td></tr><tr><td>0.8710 0.8882 0.8937 0.8922 0.9043 0.9077 0.9151 0.9174 0.9279 0.9324 0.9518 0.9572 0.1346 0.1240 0.0964 0.0930 0.1283</td></tr><tr><td>0.8625 0.8822 0.8847 0.8855 0.9013 0.8990 0.9093 0.9088 0.9189 0.9295 0.9411 0.9458 0.9533 0.1309 0.1059 0.0987 0.1139</td></tr><tr><td>0.8581 0.8784 0.8774 0.8817 0.8954 0.8938 0.8986 0.9003 0.9082 0.9225 0.9307 0.9350 0.9435 0.9603 0.1048 0.1023 0.1048</td></tr><tr><td>0.8492 0.8674 0.8732 0.8697 0.8828 0.8826 0.8930 0.8898 0.8962 0.9098 0.9184 0.9267 0.9290 0.9425 0.9542 0.1012 0.1070</td></tr></table>

.6568 0.6412 0.6284 0.7044 0.6304 0.7196 0.6660 0.0000 0.0000 0.0000 0.0000 0.0000 0.6020 0.5628 0.5792 0.6164 0.6444 0.6636 0.6356 0.6536 0.7008 0.6132 0.7000 0.6336 0.7108 0.0000 0.0000 0.0000 0.0000 0.6124 0.5692 0.5924 0.6420 0.6516 0.6912 0.6352 0.6492 0.6848 0.6400 0.6872 0.6312 0.7280 0.7596 0.0000 0.0000 0.0000 0.6012 0.5468 0.5908 0.6128 0.6552 0.6852 0.6288 0.6428 0.6704 0.6176 0.6988 0.6268 0.7088 0.7348 0.6324 0.0000 0.0000 0.6244 0.5588 0.5960 0.6432 0.6448 0.6868 0.6388 0.6540 0.6896 0.6056 0.6920 0.6192 0.7124 0.7344 0.6560 0.7604 0.0000 0.6088 0.5896 0.5840 0.6552 0.6716 0.6904 0.6584 0.6372 0.7032 0.6300 0.6900 0.5864 0.6832 0.7140 0.6348 0.7264 0.7780

Table content: <table><tr><td>0.6104 0.5552 0.5804 0.6396 0.6700 0.6960 0.6568 0.6752 0.7412 0.6752 0.7432 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000</td></tr><tr><td>0.5880 0.5516 0.5816 0.6248 0.6552 0</td></tr></table>

Peer review (iclr2020):
I am very torn about this paper as the experiments are thorough and the results are quite significant. The results are on four popular benchmarks and they also test on Many Permutations, which is an important finding as well. While the angle based approach presented here is interesting and intuitively appealing, I am not really sure whether it is a needed generalization beyond past approaches. I don*t really have a strong intuition for where the gains are coming from. As such, I think the paper would be much stronger if it could intuitively or theoretically get at why it improves on past approaches. Actually, one of my biggest concerns is how good the performance of the direct approach is. The direct approach seems like a straightforward extension of experience replay with a special learning rate for current examples vs. buffer examples that is adaptive to the relative loss. While it is potentially interesting that this approach works so well, it is not positioned as the selling point of the paper as currently written. Additionally, this is kind of an orthogonal contribution to past papers leveraging experience replay, which could potentially be modified and improved based on this approach. I wonder if the authors tried modifying any of the baseline approaches in this way as well. A small note on the experiments: the authors closely follow settings from past work and at times it can be unclear which results were implemented in this paper and which in past papers. I don*t think that you have addressed what hyperparameters you chose for MER. It also seems to perform worse than past reported results on CIFAR and is not reported despite the best past performance on MNIST Permutations.