ATTACKING LIFELONG LEARNING MODELS WITH GRADIENT REVERSION
Year: 2019

Abstract:
Lifelong learning aims at avoiding the catastrophic forgetting problem of traditional supervised learning models. Episodic memory based lifelong learning methods such as A-GEM (Chaudhry et al., 2018b) are shown to achieve the stateof-the-art results across the benchmarks. In A-GEM, a small episodic memory is utilized to store a random subset of the examples from previous tasks. While the model is trained on a new task, a reference gradient is computed on the episodic memory to guide the direction of the current update. While A-GEM has strong continual learning ability, it is not clear that if it can retain the performance in the presence of adversarial attacks. In this paper, we examine the robustness of A-GEM against adversarial attacks to the examples in the episodic memory. We evaluate the effectiveness of traditional attack methods such as FGSM and PGD. The results show that A-GEM still possesses strong continual learning ability in the presence of adversarial examples in the memory and simple defense techniques such as label smoothing can further alleviate the adversarial effects. We presume that traditional attack methods are specially designed for standard supervised learning models rather than lifelong learning models. we therefore propose a principled way for attacking A-GEM called gradient reversion (GREV) which is shown to be more effective. Our results indicate that future lifelong learning research should bear adversarial attacks in mind to develop more robust lifelong learning algorithms.

text: Lifelong learning aims at avoiding the catastrophic forgetting problem of traditional supervised learning models. Episodic memory based lifelong learning methods such as A-GEM (Chaudhry et al., 2018b) are shown to achieve the stateof-the-art results across the benchmarks. In A-GEM, a small episodic memory is utilized to store a random subset of the examples from previous tasks. While the model is trained on a new task, a reference gradient is computed on the episodic memory to guide the direction of the current update. While A-GEM has strong continual learning ability, it is not clear that if it can retain the performance in the presence of adversarial attacks. In this paper, we examine the robustness of A-GEM against adversarial attacks to the examples in the episodic memory. We evaluate the effectiveness of traditional attack methods such as FGSM and PGD. The results show that A-GEM still possesses strong continual learning ability in the presence of adversarial examples in the memory and simple defense techniques such as label smoothing can further alleviate the adversarial effects. We presume that traditional attack methods are specially designed for standard supervised learning models rather than lifelong learning models. we therefore propose a principled way for attacking A-GEM called gradient reversion (GREV) which is shown to be more effective. Our results indicate that future lifelong learning research should bear adversarial attacks in mind to develop more robust lifelong learning algorithms.

heading: 1 INTRODUCTION
text: Lifelong learning (French, 1999; Thrun & Mitchell, 1995; Kirkpatrick et al., 2017) aims at improving the continual learning ability of neural networks. Standard supervised learning methods suffer from the problem of catastrophic forgetting, in which case the models gradually forget previously learned knowledge while learning on a sequence of new tasks. In lifelong learning, neural networks are equipped with the capability to learn new tasks while maintaining the performance on the tasks trained previously. Lifelong learning models with continual learning ability can be deployed in complex environments with the aim to process a continuous stream of information.
Several methodologies are proposed recently to address the catastrophic forgetting problem. In Kirkpatrick et al. (2017), the authors adopt Fisher information matrix to prevent important weights for old tasks from drastic changes while the model is trained on a new task. While in Rusu et al. (2016), a neural network that has lateral connections with old tasks is trained each time for the new task. Recently, lifelong learning methods based on episodic memory (Lopez-Paz et al., 2017; Chaudhry et al., 2018b; d’Autume et al., 2019) such as A-GEM (Chaudhry et al., 2018b) are shown to be able to achieve the state-of-the-art performance across several benchmarks. In A-GEM, a small episodic memory is utilized to store a random subset of the examples from old tasks. While the model is trained on a new task, a reference gradient is computed on a batch of the samples from the episodic memory to guide the current update direction. If the angle between the reference gradient and the current gradient computed on the new task is obtuse, the current gradient is projected to be perpendicular to the reference gradient.
The strong continual learning ability of A-GEM relies on the episodic memory which can give a hint on the performance of the current model on old tasks. It has been known that in the standard super-
vised learning setting, deep neural networks can be easily fooled by adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2014). A natural question then arises:
Can A-GEM retain the continual learning ability in the presence of adversarial examples in the episodic memory?
In this paper, we systematically evaluate the robustness of A-GEM against traditional adversarial attack methods such as FGSM (Goodfellow et al., 2014) and PGD (Madry et al., 2017). The results show that A-GEM is surprisingly robust under traditional adversarial attacks. We therefore propose gradient reversion (GREV) attack, which is a principled way for attacking episodic memory based lifelong learning algorithms such as A-GEM. Essentially, GREV alters the direction of reference gradient computed on the episodic memory by slightly perturbing the examples. Our results show that for future research on lifelong learning, it is important to design algorithms bearing adversarial attacks in mind.
In this paper, we have the following contributions,
• To the best of our knowledge, we are the first to systematically evaluate the robustness of episodic memory based lifelong learning algorithms such as A-GEM.
• We show that simple adversarial attack methods such as fast gradient sign method (FGSM) (Goodfellow et al., 2014) and projected gradient descent (PGD) (Madry et al., 2017) can hardly hurt the performance of A-GEM. Defense techniques such as label smoothing can be used to further alleviate the adversarial effect.
• We propose a principled way called gradient reversion (GREV) for attacking A-GEM. On Permuted MNIST, we show that A-GEM achieves an accuracy which is 40% lower than the original accuracy under the proposed GREV attack. On Split CIFAR, while FGSM and PGD cannot hurt the performance of A-GEM, the proposed GREV degrades the accuracy of A-GEM by as much as 20%.

heading: 2 BACKGROUND

heading: 2.1 LIFELONG LEARNING
text: In a lifelong learning task, suppose there are a sequence ofN datasets denoted as {D1, D2, ..., DN}. Each dataset Di is a collection of pairs {xij , yij}, where xij is the j-th example of task i and yij is the corresponding label. A model f(w;x) with weight w is trained continually on the tasks with a single pass over the examples of each task. We denote the weight as wi while the model is trained on the i-th task and the training loss on the i-th task is denoted as `(wi;Di).
The most commonly used metric for evaluating the performance of lifelong learning models is Average Accuracy (AA), which is the average test accuracy on the test set of each task after the model finishes training on all tasks. In order to achieve a high Average Accuracy, the model should maintain the performance on old tasks while training on a new task.

heading: 2.2 A-GEM
text: In this section, we review the Averaged Gradient Episode Memory (A-GEM) (Chaudhry et al., 2018b), one of the state-of-the-art lifelong learning methods. In A-GEM, a small episodic memory M with fixed size is used to store a subset of the examples from old tasks. The episodic memory is populated by choosing examples uniformly at random for each task. Mk is used to denote the examples in the episodic memory from task k. While training on task i, the loss on the episodic memory M can be computed as `(wi;M), where M = ∪k<iMk. A-GEM ensures that each update on the i-th task will not increase the loss on the episodic memory, that is,
minw`(w;Di) s.t. `(w;M) ≤ `(wi−1;M) where M = ∪k<iMk (1) To inspect the increase of loss on the episodic memory, A-GEM computes the gradient g on the current task and the reference gradient gref on the episodic memory. When the angle between g and
gref is obtuse, A-GEM projects the current gradient g to have a right or acute angle with gref,
mingtrue 1
2 ‖g − gtrue‖22 s.t. g > truegref ≥ 0 (2)
The above optimization problem can be solved in closed form as,
gtrue = g − g>gref g>refgref gref (3)
The current gradient g is then replaced by gtrue for updating the model.
Essentially, A-GEM avoids the catastrophic forgetting problem by altering the direction of the current gradient. When the current gradient is detrimental to old tasks, it is adjusted to be perpendicular or acute with the gradient computed on the episodic memory.

heading: 3 A THREAT MODEL FOR ATTACKING AND DEFENSE OF MEMORY BASED LIFELONG LEARNING ALGORITHMS
text: In this section, we specify the threat model used in the paper for attacking A-GEM. Specially, we consider a white-box adversary which has access to the model architecture and parameters. In addition, the adversary is allowed to perturb the examples in the episodic memory. However, we do not expose the training process to the adversary, that is, the adversary can only perturb the examples in the episodic memory in an offline fashion. Specially, before the model is trained on the i-th task, the adversary can slightly perturb the examples from task i−1 in the episodic memory. In this setting, each example is only perturbed once during the lifelong learning process. We refer to this threat model as Offline Sequential Attack. The defender, on the other hand, has access to the reference gradients but not the perturbed examples in the episodic memory. The defender is further allowed to take advantage of the labels of the perturbed examples in the memory for defending possible attacks. The proposed threat model is a generalized version of white-box adversaries (Goodfellow et al., 2014; Carlini & Wagner, 2017a) for lifelong learning setting. The Average Accuracy of the lifelong learning model with and without adversarial attack is referred to as perturbed accuracy and unperturbed accuracy respectively.

heading: 4 TRADITIONAL ATTACK METHODS
text: The objective of traditional adversaries is to find an adversarial example xadv for x such that they are imperceptibly close and yet the neural network labels them distinctly. We bound the `p distance between an input x and its adversarial counterpart: xadv ∈ Sp(x) := {x′ : ‖x − x′‖p ≤ τp}, where p = 2 or∞. We omit from Sp(x) the argument x and the subscript p when it does not cause ambiguity. We focus on `∞ bounded attack in this paper since `∞ distance has been shown as a natural metric to measure adversarial perturbations.
FGSM Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014) is a simple one-step adversarial attack method. It is featured by efficiency and high performance for generating `∞ bounded adversarial examples. In FGSM, the adversarial counterpart of example x is produced by,
x + sign(∇x`(w;x)) (4)
PGD A more powerful attack technique has been proposed as a multi-step variant of FGSM, which is called projected gradient descent (PGD) (Madry et al., 2017),
xn+1 = ΠS(xn + ηsign(∇xn`(w;xn)) (5)
where S = {x : ‖x− xn‖ ≤ τ} and Π is the projection operator.
Rotation We also consider a simple variant of the rotation attack introduced in Engstrom et al. (2017). If the example x is rotated by an angle θ, the pixel at position (u, v) is translated to (u′, v′) via the following equation, [
u′ v′
] = [ cos θ − sin θ sin θ cos θ ] · [ u v ] (6)

heading: 5 A PRINCIPLED WAY OF ATTACKING MEMORY BASED LIFELONG LEARNING METHODS
text: The traditional attack methods, such as FGSM and PGD, are designed based on the idea that the goal of attacking is to degrade the accuracy of the model by appropriately perturbing the examples. While in lifelong learning, the aim is to perturb the examples in the memory in the way that the model cannot retain performance on old tasks. Dominated by different motivations, traditional attack methods may not be suitable for attacking lifelong learning models.
Note that in the A-GEM framework, as illustrated in Section 2.2, the angle between the stochastic gradient of the current task and the episodic memory is computed in each iteration. If the angle is obtuse, then A-GEM adjusts the current update direction appropriately. The main idea of our attack mechanism is to disseminate misinformation by providing a corrupted reference gradient, which can be realized by manipulating the examples stored in the episodic memory in a specific way. Based on this idea, we propose an attack method which tries to find an appropriate perturbation to the examples in the episodic memory to minimize the correlation (characterized by inner product) between the vanilla reference gradient and the corrupted reference gradient calculated on the perturbed data. Mathematically, the objective function can be written as,
min ‖δ‖≤τ
〈∇w`(w;x),∇w`(w;x + δ)〉 , (7)
where w is the trainable parameter of the model, x is an example in the episodic memory, ` is the loss function, τ > 0 is a hyper-parameter which characterizes the feasible region of the perturbation. If τ is large enough, there exists at least one sub-optimal perturbation δ̃ such that∇w`(w;x+ δ̃) is in the opposite direction of ∇w`(w;x). In Figure 1, g, gref, grev, gtrue, ggrev stand for stochastic gradient calculated on the current task, stochastic gradient calculated on the examples in episodic memory (i.e., the reference gradient∇w`(w;x)), the corrupted reference gradient (i.e.,∇w`(w;x+ δ̃)), the true update direction found by A-GEM on the unperturbed memory, the update direction found by A-GEM under gradient reversion attack. Below we analyze two common cases during the learning process,
• In case 1, the angle between the current gradient g and the reference gradient gref is acute. The true update direction gtrue found by A-GEM is coincident with g. With the proposed gradient reversion attack, the actual update direction ggrev is perpendicular to gref. Although the update direction ggrev will not lead to an increase of loss on the episodic memory, it does not help the learning on old tasks as gtrue does.
• In case 2, the angle between the current gradient g and the reference gradient gref is obtuse. The true update direction gtrue found by A-GEM is perpendicular to gref. Since ggrev is coincident with g in this case, the update direction ggrev will deteriorate the performance of the model on the episodic memory.
In both cases, we can see that ggrev is either perpendicular or negatively correlated to gref, which directly leads to catastrophic forgetting.
To solve (7), we can utilize projected gradient descent update for multiple iterations: δ ← Π‖δ‖≤τ [ δ − η ( (∇δ∇w`(w;x + δ))>∇w`(w;x )] (8)
It is worth mentioning that the update (8) can be implemented efficiently. It requires three backpropagations in the first iteration, and only two back-propagations thereafter. Specifically, in the first iteration, we need to evaluate∇w`(w;x) and∇w`(w;x+ δ) with two back-propagations, and one more back-propagation is needed to calculate the gradient with respect to δ. In the subsequent iterations, ∇w`(w;x) is already available and only two back-propagations are needed. Note that our computational cost per iteration is almost the same as PGD. The reason is that the dimension of δ is much smaller than that of w, so the additional computational overhead is negligible.

heading: 6 EXPERIMENTAL SETTINGS
text: In the experiments, we are interested in investigating the following questions,
• Can traditional attack methods successfully attack A-GEM? • How effective is our proposed attack method compared with traditional attack methods? • How is the perturbed accuracy affected by the size of the episodic memory?

heading: 6.1 DATASETS
text: We use Permuted MNIST and Split CIFAR in the experiments. Permuted MNIST (Kirkpatrick et al., 2017) consists of 20 tasks and each task is constructed by applying the same permutation to the pixel of examples in the MNIST dataset (LeCun et al., 1998). Split CIFAR (Kirkpatrick et al., 2017) is constructed by splitting the original CIFAR100 dataset (Krizhevsky et al., 2009) into 20 disjoint sets. Each set is constructed by randomly sampling 5 categories of the dataset.

heading: 6.2 NETWORK ARCHITECTURES
text: We use the same network architectures as in Chaudhry et al. (2018b). For Permuted MNIST, we adopt a fully-connected neural network with two hidden layers of 256 ReLU units. For Split CIFAR, we use a reduced ResNet18 (He et al., 2016).

heading: 6.3 EVALUATION PROTOCOL
text: We follow the same training settings as in Chaudhry et al. (2018b). For Permuted MNIST, the maximum perturbation τ is selected from {0.05, 0.1, 0.2}. The episodic memory size is set to be {850, 1700, 2550, 4250}, which corresponds to 5, 10, 15, 25 examples per class. For Split CIFAR, the maximum perturbation τ is selected from {0.015, 0.031, 0.055}. The episodic memory size is set to be {425, 850, 1105}, which corresponds to 5, 10, 13 examples per class. Both PGD and GREV are iterated for 40 steps. In the rotation attack, the examples in the episode memory are rotated by 3, 5, 7 degrees. All the experiments are repeated for 5 times with different random seeds and the variance is reported.

heading: 7 RESULTS

heading: 7.1 PERMUTED MNIST
text: We show the results of the perturbed accuracy of different attack methods and the unperturbed accuracy without attack on Permuted MNIST in Table 1. The results show several intriguing properties
of attacking lifelong learning models compared with attacking standard supervised learning models. When τ = 0.05, both FGSM and PGD can hardly hurt the performance of A-GEM which shows the surprising robustness of A-GEM. Although the model performs much worse on the episodic memory after the attack, we presume that the direction of the gradient on the episodic memory remains nearly unchanged which allows A-GEM to derive the correct update direction. Only when the value of τ increases to 0.2, FGSM and PGD can achieve a much lower perturbed accuracy. For the rotation attack, rotating the examples in the memory by only 3 degrees results in a large drop in accuracy. Intuitively, by rotating the examples, the direction of the gradient changes accordingly which fools A-GEM to conduct incorrect projections. Compared with the traditional attack methods, the proposed gradient reversion attack (GREV) achieves a much lower perturbed accuracy even when τ equals to 0.05. This indicates that by directly perturbing the examples in the way to reverse the direction of reference gradient, we can achieve a much more effective attack.

heading: 7.2 SPLIT CIFAR
text: We show the results of perturbed accuracy of different attack methods and the unperturbed accuracy without attack on Split CIFAR in Table 2. The experiments on Split CIFAR allow us to examine how different attack methods behave with convolutional neural networks. Surprisingly, we observe that FGSM and PGD cannot attack A-GEM in this case even with a large τ . And the rotation attack, which is effective on Permuted MNIST, cannot attack A-GEM with convolutional neural networks either. On the other hand, the proposed gradient reversion attack successfully degrades the accuracy of A-GEM by about 20%. We show in Section 7.3 that PGD can hardly alter the direction of the reference gradient which explains the ineffectiveness of PGD in this case. While the proposed GREV attack drastically change the direction of the reference gradient which leads to a large drop in accuracy. Under the proposed GREV attack, we observe that the perturbed accuracy is roughly inversely proportional to the size of the episodic memory. This indicates that although A-GEM can achieve higher unperturbed accuracy with large episodic memory, it also suffers more in the presence of adversarial examples.

heading: 7.3 ANALYSIS OF PGD AND GREV
text: We now compare the angle between the reference gradient gref on the unperturbed data and the corrupted reference gradient on the perturbed data under PGD and GREV attack. This allows us to gain more insights on how different attack methods behave. On Permuted MNIST, the size of the episodic memory is set to be 4250 and τ = 0.05. On Split CIAFR, the size of the episodic memory is set to be 1105 and τ = 0.015. In Figure 2, we show the distribution of the angles between the
reference gradient on 200 random batches of unperturbed examples from task 1 and the corrupted reference gradient on the corresponding perturbed examples under PGD and GREV attack during the training of task 2. We can see that the proposed GREV attack is a more effective way to alter the direction of the reference gradient, especially on Split CIAFR. In Figure 3, we can see that the angle between the reference gradient and the corrupted reference gradient gradually increases under the GREV attack. The results show that the proposed GREV attack enjoys some special properties which make it a more effective way for attacking A-GEM.

heading: 8 RELATED WORK

heading: 8.1 LIFELONG LEARNING
text: Recent lifelong learning works mostly focus on regularization based lifelong learning methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Chaudhry et al., 2018a) and episodic memory based lifelong learning methods (Lopez-Paz et al., 2017; Chaudhry et al., 2018b; d’Autume et al., 2019). In EWC (Kirkpatrick et al., 2017), Fisher information matrix is adopted to prevent important weights for old tasks from drastic change. Zenke et al. (2017) introduced intelligent synapse which has a local measure of “importance” to avoid old memories from being overwritten. RWALK (Chaudhry et al., 2018a) leverages a KL-divergence for retaining knowledge for old tasks. In (Lopez-Paz et al., 2017), the authors introduced gradient episodic memory (GEM) which achieves the state-of-theart results on several benchmarks. In (Chaudhry et al., 2018b). the authors developed an efficient version of GEM called A-GEM which is more computational effective. d’Autume et al. (2019) generalized episodic memory based methods for lifelong language modelling.

heading: 8.2 ADVERSARIAL ATTACK
text: Adversarial attack techniques can be briefly divided into two categories: white-box attack and blackbox attack. Regarding to the defense aspect, we refer readers to Appendix A.2 for some recent works.
White-box attack. Preliminary studies on the robustness of DNNs focus on white-box setting with assuming full access to the targeted DNN. Szegedy et al. (2013) first prove DNN is fragile against adversarial examples and generate adversarial examples x′ similar to original sample x in `2 distance using box-constrained L-BFGS. Then the fast gradient sign (FGS) (Goodfellow et al., 2014) method has been invented for efficiently producing adversarial examples in `∞ distance. Papernot et al. (2016) introduce an attack optimized under l0 distance known as the Jacobian-based Saliency Map Attack (JSMA). DeepFool (Moosavi-Dezfooli et al., 2016) is an untargeted attack algorithm that aims to find the least `2 distortion leading to misclassification by projecting an image to the closest separating hyperplane. Following these works, Carlini & Wagner (2017b) proposed an iter-
ative optimization based attack (C&W attack), and then it seems to become a standard white-box attack approach. Similarly, projected gradient descent (PGD) has been shown strong in attacking DNNs (Madry et al., 2017). Most of the white-box attacks rely on the gradients of the DNNs. When the gradients are “obfuscated” (e.g., by randomization), Athalye et al. (2018) derive various methods to approximate the gradients.
Black-box attack. The black-box attacking techniques do not exert the internal knowledge of DNN, and are more practical in the real applications. Thanks to the transferability property of adversarial examples (Szegedy et al., 2013), Papernot et al. (2017) can train a substitute DNN to imitate the behavior of the unknown DNN to be attacked, produce adversarial examples of the substitute, and then use them to attack. Chen et al. (2017) instead use zero-th order optimization to find adversarial examples. Ilyas et al. (2018) use the evolution strategy (Salimans et al., 2017) to approximate the gradients. More recently, Brendel et al. (2017) introduce Boundary Attack which starts from a large adversarial perturbation and then seeks to reduce the perturbation while staying adversarial. Li et al. (2019) proposed a universal attack for defended DNNs by modelling the distributions of adversarial examples.

heading: 9 CONCLUSION
text: In this paper, we systematically examine the robustness of episodic memory lifelong learning methods such as A-GEM under traditional adversarial attacks. The results show that different from traditional supervised learning model, A-GEM is surprisingly robust to these attacks. We therefore propose an attack named Gradient Reversion (GREV) which makes A-GEM suffer from significant performance degradation. In the future, we plan to design defense mechanisms to mitigate the negative effects caused by the gradient reversion attack and develop more robust lifelong learning algorithms.

heading: A APPENDIX
text: A.1 DEFENSE LIFELONG LEARNING MODELS
In this section, we investigate a possible defense technique called label smoothing (Warde-Farley & Goodfellow, 2016; Müller et al., 2019) for defensing the attacks to A-GEM. Label smoothing converts hard class labels into soft targets as follows,
ykLS = yk(1− β) + β
K (9)
where yk is the one-hot label of class k, K is the number of classes, β is the label smoothing parameter and ykLS is the new one-hot label after label smoothing. Label smoothing has been shown as a simple way for increasing the robustness of neural networks (Warde-Farley & Goodfellow, 2016).
Instead of using labeling smoothing to train the model, we convert the labels of the examples in the memory to soft labels via label smoothing. The results on Permuted MNIST are shown in Table 3. It can be seen that label smoothing can further increase the robustness of A-GEM against PGD attack. However, the proposed GREV attack can still achieve a low perturbed accuracy. Table 4 also shows that the proposed GREV attack can greatly deteriorate the performance of A-GEM on Split CIFAR with label smoothing. Therefore, one future research direction is to develop defense methods for the proposed GREV attack and design more robust lifelong learning algorithms.
A.2 RELATED WORK ABOUT DEFENSE TECHNIQUES
Comparing with attacking DNNs, defending neural network is a more challenging task and a few explorations have been exerted to improve the robustness of DNNs.
Since the powerful attacking approaches take advantage of the gradient of DNNs (Carlini & Wagner, 2017b; Madry et al., 2017) or estimated gradient (Chen et al., 2017; Salimans et al., 2017), several defended approaches have been shown to be robust against these kind of attacks by ‘obfuscating gradient’. Specifically, Buckman et al. (2018) proposed to transform the input by non-differentiable and non-linear thermometer encoding, followed by a slight change to the input layer of conventional DNNs. Dhillon et al. (2018) randomly dropped some neurons of each layer with the probabilities in proportion to their absolute values. Xie et al. (2018) added a randomization layer between inputs and a DNN classifier. Similarly, Liu et al. (2018) combine the ideas of randomness and ensemble using the same underlying neural network. Guo et al. (2018) explored several combinations of input transformations coupled with adversarial training, such as image cropping and rescaling, bit-depth reduction, JPEG compression. Prakash et al. (2018) randomly sample a pixel from an image and then replace it with another pixel randomly sampled from the former’s neighborhood.
However, Athalye et al. (2018) have derived various approaches to estimate the gradients and successfully or partially attacked such defenses caused ‘obfuscated gradient’, and they have honored adversarial training based defenses. Basically, defending DNNs with adversarial training was firstly introduced by (Madry et al., 2017). The training procedure alternates between seeking an “optimal” adversarial example for each input by projected gradient descent (PGD) and minimizing the classification loss under the PGD attack. Furthermore, Na et al. (2018) reduced the computation cost of the adversarial training (Kurakin et al., 2016) in a cascade manner. Liu et al. (2019) proposed to model the randomness added to DNNs in a Bayesian framework coupled with adversarial training. Wang & Yu (2019) proposed to model the adversarial perturbation with a generative network, and they learned it jointly with the defensive DNN as a discriminator. Liao et al. (2018) use a denoising network architecture to estimate the additive adversarial perturbation to an input.

Review summaries:
The paper proposed a novel approach for robust continual learning model from the adversarial attack. The authors start from one of the state-of-the-art episodic memory based continual learning method, A-GEM. The proposed method, Gradient Reversion (GREV), is specialized on A-GEM. The method perturbed the episodic memory examples on A-GEM, thus modifies the direction of reference gradient. While conventional attack like Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) hardly show the their influence on A-GEM, GREV significantly attacks the performance.