Attacking Lifelong Learning Models with Gradient Reversion
Authors: Yunhui Guo
Mingrui Liu
Yandong Li
Liqiang Wang
Tianbao Yang
Tajana Rosing
Venue:  (2020)
Keywords: lifelong learning
adversarial learning

Abstract: Lifelong  learning  aims  at  avoiding  the  catastrophic  forgetting  problem  of  traditional supervised learning  models.   Episodic memory based  lifelong learning methods such as A-GEM (Chaudhry et al., 2018b) are shown to achieve the state-of-the-art results across the benchmarks. In A-GEM, a small episodic memory is utilized to store a random subset of the examples from previous tasks.  While the model is trained on a new task, a reference gradient is computed on the episodic memory to guide the direction of the current update.  While A-GEM has strong continual learning ability,  it is not clear that if it can retain the performance in the presence of adversarial attacks.  In this paper, we examine the robustness ofA-GEM against adversarial attacks to the examples in the episodic memory.  We evaluate the effectiveness of traditional attack methods such as FGSM and PGD.The results show that A-GEM still possesses strong continual learning ability in the  presence  of  adversarial  examples  in  the  memory  and  simple  defense  techniques such as label smoothing can further alleviate the adversarial effects.  We presume that traditional attack methods are specially designed for standard supervised learning models rather than lifelong learning models. we therefore propose a principled way for attacking A-GEM called gradient reversion(GREV) which is shown to be more effective.  Our results indicate that future lifelong learning research should bear adversarial attacks in mind to develop more robust lifelong learning algorithms.

Under review as a conference paper at ICLR 2020
ATTACKING LIFELONG LEARNING MODELS WITH GRADIENT REVERSION
Anonymous authors Paper under double-blind review
ABSTRACT
Lifelong learning aims at avoiding the catastrophic forgetting problem of tra- ditional supervised learning models. Episodic memory based lifelong learning methods such as A-GEM (Chaudhry et al. (0188) are shown to achieve the state- of-the-art results across the benchmarks. In A-GEM, a small episodic memory is utilized to store a random subset of the examples from previous tasks. While the model is trained on a new task, reference gradient computed on the episodic memory to guide the direction of the current update. While A-GEM has strong continual learning ability, it is not clear that if it can retain the performance the presence of adversarial attacks. In this paper, we examine the robustness of A-GEM against adversarial attacks to the examples in the episodic memory. We evaluate the effectiveness of traditional attack methods such as FGSM and PGD. The results show that A-GEM still possesses strong continual learning ability in the presence of adversarial examples in the memory and simple defense tech- niques such as label smoothing can further alleviate the adversarial effects. We presume that traditional attack methods specially designed for standard super- vised learning models rather than lifelong learning models. we therefore propose a principled way for attacking A-GEM called gradient reversion (GREV) which is shown to be more effective. Our results indicate that future lifelong learning research should bear adversarial attacks in mind to develop more robust lifelong learning algorithms.
INTRODUCTION
Lifelong learning [French 1999 Thrun & Mitchell 1995 Kirkpatrick et al. 2017) aims at improv- ing the continual learning ability of neural networks. Standard supervised learning methods suffer from the problem of catastrophic forgetting, in which case the models gradually forget previously learned knowledge while learning sequence of new tasks. In lifelong learning, neural networks are equipped with the capability to learn new tasks while maintaining the performance on the tasks trained previously. Lifelong learning models with continual learning ability can be deployed in complex enviromments with the aim to process a continuous stream of information.
Several methodologies are proposed recently address the catastrophic forgetting problem. In Kirkpatrick et al. (2017), the authors adopt Fisher information matrix to prevent important weights for old tasks from drastic changes while the model is trained on new task. While in Rusu et al. (2016), a neural network that has lateral connections with old tasks is trained each time for the new task. Recently, lifelong learning methods based on episodic memory (Lopez- Paz et al., 2017 Chaudhry et al. 2018b d' Autume et al., 2019) such A-GEM (Chaudhry et al. 20188)) are shown to be able to achieve the state-of-the-an performance across several benchmarks. In A-GEM, a small episodic memory is utilized to store random subset of the examples from old tasks. While the model is trained on a new task, a reference gradient is computed on a batch of the samples from the episodic memory to guide the current update direction. If the angle between the reference gradient and the current gradient computed on the new task is obtuse, the current gradient is projected to be perpendicular to the reference gradient.
The strong continual learning ability of A-GEM relies on the episodic memory which can give a hint on the performance of the current model on old tasks. It has been known that in the standard super-

---

Under review as a conference paper at ICLR 2020
vised learning setting, deep neural networks can be easily fooled by adversarial examples (Szegedy et al. 2013 Goodfellow et al. 2014). A natural question then arises:
Can A-GEM retain the continual learning ability the presence of adversarial examples in the episodic memory?
In this paper, we systematically evaluate the robustness of A-GEM against traditional adversarial attack methods such as FGSM (Goodfellow et al., 2014) and PGD (Madry et al. 2017). The results show that A-GEM is surprisingly robust under traditional adversarial attacks. We therefore propose gradient reversion (GREV) attack, which is a principled way for attacking episodic memory based lifelong learning algorithms such A-GEM. Essentially, GREV alters the direction of reference gradient computed on the episodic memory by slightly perturbing the examples. Our results show that for future research lifelong learning, it is important to design algorithms bearing adversarial attacks mind.
In this paper, we have the following contributions,
To the best of our knowledge, we are the first systematically evaluate the robustness of episodic memory based lifelong learning algorithms such as A-GEM.
We show that simple adversarial attack methods such as fast gradient sign method (FGSM) (Goodfellow et al., 2014) and projected gradient descent (PGD) (Madry et al., 2017) can hardly hurt the performance of A-GEM. Defense techniques such as label smoothing can be used to further alleviate the adversarial effect.
We propose a principled way called gradient reversion (GREV) for attacking A-GEM. On Permuted MNIST, we show that A-GEM achieves an accuracy which is 40% lower than the original accuracy under the proposed GREV attack. On Split CIFAR, while FGSM and PGD cannot hurt the performance of A-GEM, the proposed GREV degrades the accuracy of A-GEM by as much as 20%.
BACKGROUND
2.1 LIFELONG LEARNING
In lifelong learning task, suppose there are sequence of N datasets denoted as {D1, D2, ..., D }. Each dataset D; is a collection of pairs {x), J. where x) is the j-th example of task i and y is the corresponding label. A model f(w; x) with weight w is trained continually on the tasks with a single pass the examples of each task. We denote the weight w, while the model is trained on the i-th task and the training loss on the i-th task is denoted as ((w;D,)
The most commonly used metric for evaluating the performance of lifelong learning models is Av- erage Accuracy (AA), which the average test accuracy on the test set of each task after the model finishes training all tasks. In order to achieve high Average Accuracy, the model should main- tain the performance on old tasks while training on new task.
2.2 A-GEM
In this section, we review the Averaged Gradient Episode Memory (A-GEM) (Chaudhry et al., 2018b), one of the ofate-of-the-ar lifelong learning methods. A-GEM, small episodic memory M with fixed size is used to store subset of the examples from old tasks. The episodic memory populated by choosing examples uniformly at random for each task. MM. is used to denote the examples in the episodic memory from task k. While training on task i, the loss on the episodic memory M can be computed as f(w,; M), where M = Mg. A-GEM ensures that each update on the i-th task will not increase the loss on the episodic memory, that is,
(in Dz) s.t. E(w; M) ((;-1;; where M = Us
To inspect the increase of loss on the episodic memory, A-GEM computes the gradient g on the current task and the reference gradient gier the episodic memory. When the angle between g and
2

---

Under review as a conference paper at ICLR 2020
giet is obtuse, A-GEM projects the current gradient g to have right acute angle with gat
min Kine
ginel
s.t. Brue 2 0
(2)
The above optimization problem be solved in closed form as,
gref
gime g
gref
8 Bref
The current gradient is then replaced by give for updating the model.
Essentially, A-GEM avoids the catastrophic forgetting problem by altering the direction of the cur- rent gradient. When the current gradient is detrimental to old tasks, it adjusted to perpendicular acute with the gradient computed on the episodic memory.
A THREAT MODEL FOR ATTACKING AND DEFENSE OF MEMORY BASED LIFELONG LEARNING ALGORITHMS
In this section, specify the threat model used in the paper for attacking A-GEM. Specially, consider a white-box adversary which has access to the model architecture and parameters. In addition, the adversary allowed to perturb the examples in the episodic memory. However, do not expose the training process to the adversary, that is, the adversary can only perturb the examples in the episodic memory in offline fashion. Specially, before the model is trained the i-th task, the adversary can slightly perturb the examples from task i-1 in the episodic memory. In this setting, each example only perturbed once during the lifelong learning process. We refer to this threat model as Offline Sequential Attack. The defender, on the other hand, has access to the reference gradients but not the perturbed examples in the episodic memory. The defender further allowed to take advantage of the labels of the perturbed examples in the memory for defending possible attacks. The proposed threat model a generalized version white-box adversaries (Goodfellow) et al.) 2014: Carlini & Wagner 2017a) for lifelong learning setting. The Average Accuracy of the lifelong learning model with and without adversarial attack referred to as perturbed accuracy and unperturbed accuracy respectively.
TRADITIONAL ATTACK METHODS
The objective of traditional adversaries is to find an adversarial example Nadv for x such that they imperceptibly close and yet the neural network labels them distinctly. We bound the l, distance between an input x and its adversarial counterpart: Nadv E Sp) == {x' |xx xx'l, I }p}, where p 2 or co. We omit from (x) the argument x and the subscript p when does not cause ambiguity. We focus on to bounded attack in this paper since for distance has been shown as natural metric measure adversarial perturbations.
Fast Gradient Sign Method (FGSM) (Goodfellow et al. 2014) is simple one-step adver- sarial attack method. featured by efficiency and high performance for generating & bounded adversarial examples. In FGSM, the adversarial counterpart of example x is produced by,
FGSM
x + esign( (xlign(V<((;;<))
(4)
PGD more powerful attack technique has been proposed as multi-ster selli--tepppe variant of FGSM, which is called projected gradient descent (PGD) (Madry et al. 2017).
Xa+1 = II (Xa, + rssign( V ((w; x,))
(5)
where S = (x: x, & r) and IT is the projection operator.
3

---

Under review as a conference paper at ICLR 2020
Bref
gref
8 (@true)
Brev
Brev
Btrue
Sgrev
(Bgrev)
Casel: the angle between the current gradient g and the refer- ence gradient get is acute.
(b) Case2: the angle between the current gradient g and the refer- ence gradient get is obtuse.
Figure I: The illustration of our proposed gradient reversion attack.
Rotation We also consider simple variant of the rotation attack introduced in Engstrom et al. (017). If the example x is rotated by an angle 0, the pixel at position (u, v) is translated to (u', v') via the following equation,
cos sin 0 cos 0
sin
(6)
A PRINCIPLED WAY OF ATTACKING MEMORY BASED LIFELONG LEARNING METHODS
The traditional attack methods, such as FGSM and PGD, are designed based on the idea that the goal of attacking is to degrade the accuracy of the model by appropriately perturbing the examples. While in lifelong learning, the aim is to perturb the examples in the memory in the way that the model cannot retain performance on old tasks. Dominated by different motivations, traditional attack methods may not be suitable for attacking lifelong learning models.
Note that in the A-GEM framework, as illustrated in Section 2.2 the angle between the stochastic gradient of the current task and the episodic memory is computed in each iteration. If the angle is obtuse, then A-GEM adjusts the current update direction appropriately. The main idea our attack mechanism is to disseminate misinformation by providing corrupted reference gradient, which can be realized by manipulating the examples stored in the episodic memory in a specific way. Based on this idea, propose attack method which tries to find an appropriate perturbation to the examples in the episodic memory to minimize the correlation (characterized by inner product) between the vanilla reference gradient and the corrupted reference gradient calculated the perturbed data. Mathematically, the objective function can be written as,
min (Vwe(w; x), Vw +6)) [|]]|ST
where is the trainable parameter of the model, is example in the episodic memory, & is the loss function, > 0 is hyper-paramett which characterizes the feasible region of the perturbation. I is large enough, there exists at least one sub-optimal perturbation such that V (w; x +8) 6) is in the opposite direction of Vw(((;:)) x). Figure g, grets giev. guree Egrev stand for stochastic gradient calculated on the current task, stochastic gradient calculated on the examples in episodic memory
(i.e., the reference gradient V (we(w; x)). the corrupted reference gradient (i.e., Vwf(w; + 5)), the true update direction found by A-GEM on the unperturbed memory, the update direction found by A-GEM under gradient reversion attack. Below we analyze two common cases during the learning process,
In case 1, the angle between the current gradient g and the reference gradient gref is acute. The true update direction gine found by A-GEM is coincident with g. With the proposed gradient reversion attack, the actual update direction ggrev is perpendicular to git- Although the update direction Egrev will not lead to an increase loss on the episodic memory. it does not help the learning on old tasks as girue does.
4

---

Under review as a conference paper at ICLR 2020
In case 2, the angle between the current gradient and the reference gradient gie is obtuse. The true update direction Eune found by A-GEM is perpendicular to gie-- Since Egrev is coincident with in this case, the update direction Sgrev will deteriorate the performance of the model on the episodic memory.
In both cases, directly leads to catastrophic forgetting.
that Sgrev is either perpendicular negatively correlated to Biets which
To solve 77. can utilize projected gradient descent update for multiple iterations:
8 5 -- p 6 - ((V& Vw + 5)) V V
(8)
It is worth mentioning that the update (8) can be implemented efficiently. It requires three back- propagations the first iteration, and only two back-propagations thereafter. Specifically, in the first iteration, need to evaluate (w]((w x) and [w<(w; 6) with two back- pock-prppggttios, and one more back-propagation is needed to calculate the gradient with respect to 6. In the subsequent pack-propagations
iterations, (wW( x) is already available and only our computational cost per iteration is almost the same as PGD. The reason is that the dimension of ô is much smaller than that of w, so the additional computational overhead is negligible.
needed. Note that
6 EXPERIMENTAL SETTINGS
In the experiments, we are interested in investigating the following questions,
Can traditional attack methods successfully attack A-GEM?
How effective is our proposed attack method compared with traditional attack methods?
How is the perturbed accuracy affected by the size of the episodic memory?
6.1 DATASETS
We use Permuted MNIST and Split CIFAR in the experiments. Permuted MNIST (Kirkpatrick al. 2017) consists of 20 tasks and each task is constructed by applying the same permutation to the pixel of examples in the MNIST dataset (LeCun et al. 1998). Split CIFAR (Kirkpatrick et al., 2017) is constructed by splitting the original CIFARTO0 dataset (Krizhevsky al., 22009) into 20 disjoint sets. Each set is constructed by randomly sampling 5 categories of the dataset.
6.2 NETWORK ARCHITECTURES
We use the same network architectures as in Chaudhry 20118)). For Permuted MNIST, we adopt a fully-connected neural network with two hidden layers of 256 ReLU units. For Split CIFAR, we use reduced ResNet18 (He et 2016
6.3 EVALUATION PROTOCOL.
We follow the same training settings as in Chaudhry et al. (2018)). For Permuted MNIST, the maximum perturbation is selected from (0.05, 0.1, 0.2). The episodic memory size is set to be (850, 1700, 2550, 4250), which corresponds to 5, 10, 15, 25 examples per class. For Split CIFAR, the maximum perturbation T is selected from {0.015, 0.031, 0.055). The episodic memory size is set to be (425, 850, 1105). which corresponds to 5, 10, 13 examples per class. Both PGD and GREV are iterated for 40 steps. the rotation attack, the examples in the episode memory are rotated by 3, 5, 7 degrees. All the experiments are repeated for times with different random seeds and the variance is reported.
RESULTS
7.1 PERMUTED MNIST
We show the results of the perturbed accuracy of different attack methods and the unperturbed accu- racy without attack Permuted MNIST in Table The results show several intriguing properties
5

---

Under review as a conference paper at ICLR 2020
Memory Size
= 0.05
0.1
=.02
No Attack
4250
0.872 + 0.002
0.785 + 0.0TO
0.510 4 0.02
0.892 + 0.003
FGSM
2550
0.866 + 0.007
0.788 + 0.004
0.543+ 0.006
0.885 + 0.004
1700
0.858 1 0.002
0.787 1 0.00S
0.555 - 0.010
0.875 1 0.003
850
0.831 + 0.004
0.761 = 0.008
0.550 + 0.016
0.861 + 0.001
Memory Size
0.05
0.1
0.2
No Attack
4250
0.868 + 0.004
0.765 = 0.006
0.471 0.013
0.892 = 0.003
PGD
2550
0.863 - 0.004
0.771 = 0.012
0,511
0.011
0.885 = 0.004
1700
0.852 + 0.003 0.003
0.768 + 0.007
0.523
0.009
0.875 + 0.003
850
0.8299 + 0.003
0.749 = 0.007
0.543
0.004
0.861 + 0.001
Memory Size
deg=3
deg =5
deg =7
No Attack
4250
0.805 + 0.009
0.659
0.018
0.597 0.010
0.892 + 0.003
Rotation
2550
0.804 + 0.012
0.651 4.00199
0.591 0.022
0.885 + 0.004
1700
0.789 1 0.008
0.644
10.020
0.589
0.011
0.875 = 0.003
850
0.748 + 0.012
0,602 10.012
0.550 - 0.019
0.861 = 0.001
Memory Size
=.0.05
0.1
7 =.2
No Attack
4250
0,421 0.066
0,450 0.019
0.474 + 0.018
0.892 + 0.003
GREV
2550
0.440 0.015
0,482 = 0.011
0.508 0.017
0.885 + 0.004
1700
0.467 + 0.019
0.469 + 0.022
0.488 0.014
0.875 + 0.003
850
0.451 + 0.012
0.472 + 0.017
0.494 0.005
0.861 + 0.001
Table 1: The perturbed accuracy of different attack methods on Permuted MNIST.
of attacking lifelong learning models compared with attacking standard supervised learning models. When 5 0.05, both FGSM and PGD can hardly hurt the performance of A-GEM which shows the surprising robustness of A-GEM. Although the model performs much worse on the episodic memory after the attack, we presume that the direction of the gradient the episodic memory mains nearly unchanged which allows A-GEM to derive the correct update direction. Only when the value of increases to 0.2, FGSM and PGD achieve much lower perturbed accuracy. For the rotation attack, rotating the examples in the memory by only 3 degrees results in a large drop in accuracy. Intuitively, by rotating the examples, the direction of the gradient changes accordingly which fools A-GEM to conduct incorrect projections. Compared with the traditional attack meth- ods, the proposed gradient reversion attack (GREV) achieves a much lower perturbed accuracy even when 7 equals to 0.05. This indicates that by directly perturbing the examples in the way to reverse the direction of reference gradient, we can achieve a much more effective attack.
7.2 SPLIT CIFAR
We show the results of perturbed accuracy of different attack methods and the unperturbed accuracy without attack Split CIFAR in Table 2 The experiments Split CIFAR allow us to examine how different attack methods behave with convolutional neural networks. Surprisingly, we observe that FGSM and PGD cannot attack A-GEM in this case even with a large And the rotation attack, which is effective on Permuted MNIST, cannot attack A-GEM with convolutional neural networks either. On the other hand, the proposed gradient reversion attack successfully degrades the accuracy of A-GEM by about 20%. We show in Section 7.3 that PGD can hardly alter the direction of the reference gradient which explains the ineffectiveness of PGD in this case. While the proposed GREV attack drastically change the direction of the reference gradient which leads to large drop in accuracy. Under the proposed GREV attack, we observe that the perturbed accuracy is roughly inversely proportional to the size of the episodic memory. This indicates that although A-GEM can achieve higher unperturbed accuracy with large episodic memory, it also suffers more in the presence of adversarial examples.
7.3 ANALYSIS OF PGD AND GREV
We now compare the angle between the reference gradient gief the unperturbed data and the corrupted reference gradient on the perturbed data under PGD and GREV attack. This allows us to gain more insights on how different attack methods behave. On Permuted MNIST, the size of the episodic memory is set to 4250 and 5 = 0.05. On Split CIAFR, the size of the episodic memory set be 1105 and 0.015. In Figure 2 show the distribution of the angles between the

---

Under review as a conference paper at ICLR 2020
Memory Size
7.0.51
0.031
7 0.055
No Attack
1105
0.601 + 0.025
0.587
+ 0.021
0.587
0.020
0.585 + 0.016
FGSM
850
0.595 + 0.017
0.589 + 0.033
0.589
+ 0.034
0.593 + 0.027
425
0.591 1 0.024
0.024
0.591 = 0.024
0.559
1 0.018
0.590 1 0.025
Memory Size
0.0.5
T= 0.031
= 0.055
No Attack
1105
0.387 = 0.019
0.571 = 0.017
0.571 0.017
0.385 E 0.016
PGD
850
0.572 = 0.030
0.587 = 0.018
0.587 0.017
0.593 = 0.027
425
0.588 = 0.021
0.588 = 0.021
0.020 0.588
0.590 = 0.025
Memory Size
d
deg=5
deg=7
No Attack
1105
0,584 + 0.028
0.586 = 0.019
0.592 - 0.012
0,585 = 0.016
Rotation
850
0.570 0.024
0.594 + 0.032
0.578
0.018
0.593 + 0.027
425
0.586 +: 0.020
0.571
0.039
0.575
0.022
0.590 + 0.025
Memory Size
0.0.51
0.031
7 0.055
No Attack
TTOS
0.382 0.046
0.381
1 0.045
0.397
0.027
0.385 1 0.016
GREV
850
0.393 1 0.037
0.406
1 0.039
0,406
0.040
0.593 1 0.027
425
0,413 = 0.031
0.415
= 0.035
0.414 = 0.035
0.390 = 0.025
Table 2: The perturbed accuracy of different attack methods on Split CIFAR.
reference gradient on 200 random batches of unperturbed examples from task and the corrupted reference gradient the corresponding perturbed examples under PGD and GREV attack during the training of task 2. We can see that the proposed GREV attack is a more effective way to alter that the
the direction of the reference gradient, especially Split CIAFR. In Figure angle between the reference gradient and the corrupted reference gradient gradually increases under the GREV attack. The results show that the proposed GREV attack enjoys special properties which make it a more effective way for attacking A-GEM.
8 RELATED WORK
8. LIFELONG LEARNING
Recent lifelong learning works mostly focus on regularization based lifelong learning methods
(Kirkpatrick al. 2017 Zenke al. 2017 CChaddhry al., 2018a) and episodic memory based
lifelong learning methods (Lopez-Paz et al.; 2017] Chauudryy et al. 2018b d' Autume et al. 2019). In EWC (Kirkpatrick al. 2017), Fisher information matrix adopted to prevent important weights for old tasks from drastic change. Zenke et al. (2017) introduced intelligent synapse which has local measure of "importance" to avoid old memories from being overwritten. RWALK (Chaudhry
et al. 2018a) leverages KL-divergence for retaining knowledge for old tasks. In (Lopez-Paz et al.] 2017. the authors introduced gradient episodic memory (GEM) which achieves the state-of-the- art results on several benchmarks. In (Chaudhry et al.) 8018b). the authors developed an efficient version GEM called A-GEM which is more computational effective. d' d'Auttmme et al. (2019) generalized episodic memory based methods for lifelong language modelling.
8.2 ADVERSARIAL ATTACK
Adversarial attack techniques can be briefly divided into two categories: white-box attack and black- box attack. Regarding to the defense aspect, we refer readers to Appendix A.2 for some recent works.
White-box attack. Preliminary studies on the robustness of DNNs focus on white-box setting with assuming full access to the targeted DNN. Szegedy et al. (2013) first prove DNN is fragile against adversarial examples and generate adversarial examples x similar to original sample in & distance using box-constrained L-BFGS. Then the fast gradient sign (FGS) Goodfellow et al. 2014) method has been invented for efficiently producing adversarial examples in loo distance. Papernot Saliency
et al. (2016) introduce an attack optimized under In distance known as the Jacobian- Map Attack (JSMA). DeepFool Aioosavii-ezfood et al. 2016) is an untargeted attack algorithm that aims to find the least & distortion leading to misclassification by projecting an image to the closest separating hyperplane. Following these works, Carlini & Wagner (2077)) proposed iter-
7

---

Under review as a conference paper at ICLR 2020
On Permuted MNIST
On Split CIFAR
30
5
60 dog
60 dog
dog
deg
90 dog
100 dog
20 deg
deg
60 deg
80 clg
100 deg
Angle betwein the refbreenc gradiont Sher and the comrppted grodient
Angle betwoon the reference g g'odient Gier and the coruupoed gradient
(a)
(b)
Figure 2: Comparison of GREV and PGD with respect to the ability of altering the reference gradient direction.
On Permuted INIS
SpiR
18
Ang
54
Itaration
Beralion
(a)
(b)
The angle between the reference gradient and the corrupted reference gradient gradually increases under the proposed GREV attack.
Figure 3:
ative optimization based attack (C&W attack), and then seems become standard white-box attack approach. Similarly, projected gradient descent (PGD) has been shown strong in attacking DNNs (Madry et al. (0177) Most of the white-box attacks rely on the gradients of the DNNs. When the gradients are "obfuscated" (e.g., by randomization), /Athalye et al. (2018) derive various methods to approximate the gradients.
Black-box attack. The black-box attacking techniques do not exert the internal knowledge of DNN, and are more practical the real applications. Thanks to the transferability property of ad- versarial examples (Szegedy al. a 2013), [Paeernot al. (2017) can train substitute DNN imitate the behavior of the unknown DNN to be attacked, produce adversarial examples of the substitute, and then use them attack. Chen al. (2017) instead use zero-th order optimization find adver- sarial examples. Ilyas et al. (2018) use the evolution strategy (Salimans et al., (017) to approximate the gradients. More recently, Brendel al. (2017) introduce Boundary Attack which starts from large adversarial perturbation and then seeks to reduce the perturbation while staying adversarial. Li al. (2019) proposed a universal attack for defended DNNs by modelling the distributions of adversarial examples.
CONCLUSION
In this paper, we systematically examine the robustness of episodic memory lifelong learning meth- ods such as A-GEM under traditional adversarial attacks. The results show that different from traditional supervised learning model, A-GEM is surprisingly robust to these attacks. We therefore propose an attack named Gradient Reversion (GREV) which makes A-GEM suffer from signifi- cant performance degradation. In the future, plan to design defense mechanisms to mitigate the negative effects caused by the gradient reversion attack and develop more robust lifelong learning algorithms.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Cireumventing defenses to adversarial examples. arXiv preprint (ati::8802.004220 2018.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint arXiv: 2017.
Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. 2018.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 3-14. ACM, 2017a.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Security and Privacy (SP), 20/7 IEEE Symposium on, pp. 39-57. IEEE, 2017b.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 532-547, 2018a.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. arXiv preprint arXiv:
2018b.
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order opti- mization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 15-26. ACM, 2017.
Cyprien de Masson d' Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. arXiv preprint arXiv:1906.01076. 2019.
Guneet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar. Stochastic activation pruning for robust adversarial de- fense. arXiv preprint arXiv: 1803.01434 2018
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. A rotation and a translation suffice: Fooling enns with simple transformations. arXiv preprint arXiv:1712.02779, 2017.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 124):128- 1999.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations, 2018. URL URRlihttts:///peereeee
net. forum? httt:/orooeeee--Syy7yc..cc.. wCb
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. arXiv preprint arXiv: 08598, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan. John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):35113322 2017.

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint 1roXiv 005333, 2016.
Yann LeCun, Corinna Cortes, and Christopher JC Burges. The mnist database. URL http://yann. lecun. com/exal//mrist, 1998.
Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Boqing Gong. Nattack: Learning the distri- butions of adversarial examples for an improved black-box attack deep neural networks. arXiv 2019.
preprint
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Jun Zhu, and Xiaolin Hu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1778-1787, 2018.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsich. Towards robust neural networks via random self-ensemble. In European Conference on Computer Vision, pp. 1-8. Springer, 2018.
Xuanqing Liu, Yao Li, Chongruo Wu, and Cho-Jui Hsieh. Ady-BNN: Improved adversarial defense through robust bayesian neural network. In International Conference on Learning Representa- tions, 2019. URL
David Lopez-Paz Paz et al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017.
Seyed-Mohsen Moosavi-Dezfooli: Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 2574-2582, 2016.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint 1rXiv:990..0262, 2019.
Taesik Na, Jong Hwan Ko, and Saibal Mukhopadhyay. Cascade adversarial machine learning reg- ularized with a unified embedding. In International Conference on Learning Representations, 2018. htt.tttss//epeeeveee..ccc wtt.ee/eooommna--/////
Nicolas Papernot Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In Security and Privacy (Eu- roSP)), 2016 IEEE European Symposium on, pp. 372-387. IEEE, 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506-519, ACM, 2017.
Aaditya Prakash, Nick Moran, Solomon Garber, Antonella DiLillo, and James Storer. Deflecting adversarial attacks with pixel deflection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatriek, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1rXiv:1606.04611, 2016.
Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. Evolution strategies as a scalable alterna- tive reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 211.,61999999 2013.
10

---

Under review as a conference paper at ICLR 2020
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Roboties and autonnomous systems, 15(1-2)25-46 46, 1995.
Huaxia Wang and Chun-Nam Yu. A direct approach to robust deep learning using adversar- ial networks. In International Conference on Learning Representations, 2019. URL https:
//openreview. wtw.ee/orrmm.d-111MMM
David Warde-Farley and Ian Goodfellow. 11 adversarial perturbations of deep neural networks Perturbations, Optimization, and Statistics, 311, 2016.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. In International Conference Learning Representations, 2018.
URL /ttp:tttp:::/ppeeeeiiew... Stt.:neettooodiddttttrro.o...e 9yuq10z
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. caarning-Volume Volume 70, pp. 3987-
In Proceedings of the 34th International Conference on Machine Learning- 3995. JMLR. org, 2017.

---

Under review as a conference paper at ICLR 2020
Memory Size
0.0.05
7=0.1
7 0.2
No Attack
4250
0.853 + 0.004
0.823 + 0.00
0.724
0.007
0.892 + 0.003
PGD
2550
0.855 + 0.003
0.825 + 0,003
0.735
+ 0.008
0.885 + 0.004
1700
0.864 1 0.003
0.814 1 0.008
0.705 1 0.006
0.875 1 0.003
850
0.869 1 0.002
0.819 1 0.004
0.772 0.007
0.861 = 0.001
Memory Size
T= 0.05
0.1
0.2
No Attack
4250
0.428 = 0.011
0,445 = 0.015
0.473 0.017
0.892 = 0.003
GREV
2550
0.436 - 0.019
0.463 = 0.025
0,506 0.010
0.885 = 0.004
1700
0.476 + 0.015
0.471 + 0.026
0.483 + 0.022
0.875 + 0.003
850
0.437 + 0.022
0.461 + 0.018
0.468 + 0.020
0.861 + 0.001
Table 3: PGD and GREV on Permuted MNIST with label smoothing parameter ,3 = 0.1.
Memory Size 1105 (13 * 17)
T =
T= 0.031
7= 0.055
No Attack
0.416 + 0.036
0,415 + 0.035
0.383
0.019
0.585 + 0.016
850 (10 * 17)
0,414 0.032
0.413 - 0.031
0,385
0.024
0.593 + 0.027
425 (5 " 5 17)
0.418 + 0.036
0.417 = 0.036
0.401 + 0.047
0,590 = 0.025
Table 4: GREV on Split CIFAR with label smoothing parameter / 0.1.
A APPENDIX
A.1 DEFENSE LIFELONG LEARNING MODELS
In this section, we investigate possible defense technique called label smoothing (Warde-Farley Goodfellow 2016 Müller et al. 2019) for defensing the attacks A-GEM. Label smoothing converts hard class labels into soft targets as follows,
3
vîs yu (1 33 K
(9)
where y& is the one-hot label of class k K is the number of classes, is the label smoothing parameter and vf.s is the new one-hot label after label smoothing. Label smoothing has been shown as a simple way for increasing the robustness of neural networks "Warde-Farlyy & Goodfellow 2016).
Instead of using labeling smoothing to train the model, we convert the labels of the examples in the memory to soft labels via label smoothing. The results on Permuted MNIST are shown in Table 3 can be seen that label smoothing can further increase the robustness of A-GEM against PGD) attack. However, the proposed GREV attack can still achieve low perturbed accuracy. Table 4 also shows that the proposed GREV attack can greatly deteriorate the performance of A-GEM on Split CIFAR with label smoothing. Therefore, one future research direction is to develop defense methods for the proposed GREV attack and design more robust lifelong learning algorithms.
A.2 RELATED WORK ABOUT DEFENSE TECHNIQUES
Comparing with attacking DNNs, defending neural network is a more challenging task and a few explorations have been exerted to improve the robustness of DNNs.
Since the powerful attacking approaches take advantage of the gradient of DNNs (Carlini & Wagnnrr 20176] Madry et al. (017)) or estimated gradient (Chen et al. 2017 Salimans et al. 2017). several defended approaches have been shown to be robust against these kind of attacks by "obfuscating gradient'. Specifically, Buckman et (2018) proposed to transform the input by non-differentiable and non-linear thermometer encoding, followed by a slight change to the input layer of conventional DNNs. Dhillon et al. (2018) randomly dropped some neurons of each layer with the probabilities in proportion to their absolute values. Xie et al. (2018) added a randomization layer between inputs and a DNN classifier. Similarly, [iu et al. (2018) combine the ideas of randomness and ensemble using the same underlying neural network. Guo et al. (2018) explored several combinations of input transformations coupled with adversarial training, such image cropping and rescaling, bit-depth reduction, JPEG compression. Prakash et al.] (2018) randomly sample a pixel from an image and then replace it with another pixel randomly sampled from the former's 's neighborhood.
12

---

Under review as a conference paper at ICLR 2020
However, Athalye et al. (2018) have derived various approaches to estimate the gradients and suc- cessfully or partially attacked such defenses caused "obfuscated gradient', and they have honored adversarial training based defenses. Basically, defending DNNs with adversarial training was firstly introduced by (Madry et al. 2017). The training procedure alternates between seeking an "optimal" adversarial example for each input by projected gradient descent (PGD) and minimizing the clas- sification loss under the PGD attack. Furthermore, Na et al. (2018) reduced the computation cost of the adversarial training (Kurakin et al. 2016) cascade manner. Liu al. (2019) proposed to model the randomness added to DNNs in Bayesian framework coupled with adversarial training. Wang & Yu (2019) proposed model the adversarial perturbation with generative network, and they learned it jointly with the defensive DNN as a discriminator. [iao et al. (2018) use denoising network architecture to estimate the additive adversarial perturbation to an input.
13


Reviews:
[{"review_id": "SJxFkDtf5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "General:\nThe paper first proposes an adversarial attack on the exemplar-based continual learning algorithm, A-GEM. The problem formulation is new and interesting, but I am not sure how practical or realistic the setting is. The attacker assumes to have access not only to the model but also to the episodic memory. I think that is quite a powerful assumption for the attacker and it is not very surprising that the attack would work. So, I am a bit torn with the judgments. \n\nSummary & Pro:\n1. First proposal of adversarial attack of continual learning algorithm. \n2. The conventional attack schemes are shown to not work well, and they devised new one (GREV) tailored for A-GEM. \n3. Experimental results show convincing results that their method works well. \n\nCon & Questions: \n1. It is not clear whether the proposed method will also work well for other exemplar-based methods like iCaRL or GEM (the simpler version than A-GEM), etc. I get that A-GEM can be attacked by their assumption and method, but how general is it?\n2. What exactly is the practical scenario of this method? How can the attacker get access to the model & memory? In the traditional adversarial attack literature, it is shown that white-box attack can also lead to the black-box attack. But, in this case, I am not sure about the practical implication of the proposed methods. \n3. It seems like the entire memory is under attack. What happens when only the memory is partially attacked, e.g., 10% of the data in the memory is attacked? \n4. Table 1/2 only shows the overall average accuracy. Can you also show the per-task average accuracy curves? It would be much better to see such curves to clearly see the effect of the attack rather than the overall average. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxR4mo3FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper does a good job of raising awareness of adversarial attacks in lifelong learning research with deep neural networks. This is the first time I have considered this problem, but not sure whether any prior work exists in the specific subfield.\n\nAt the conceptual level, many issues can arise when a lifelong learner is attacked, since systematic negative bias could be introduced in the training process and may be very difficult to remove, given the tendency to 'remember everything' which dominates current approaches.\n\nThe paper isolates one lifelong learning approach (A-GEM) which is characteristic of one (of many) different approaches to lifelong learning, and investigates its robustness to standard adversarial attacks and a novel attack developed within this paper, which is stronger, but specific to episodic memory approaches.\n\nI cannot recommend acceptance at this point for the following reasons:\n1) I am not sure what I can generalize away from this paper to the immediate subfield and beyond. The paper claims that the investigated method is SOTA, but it's not clear this is the case, even in restricted class of similar episodic memory based models, see [1] for an independent evaluation of many such approaches. Is there any reasons why conclusions about this particular method are indeed representative of its class?\n2) While the paper does not explicitly make this claim, the title suggests that 'gradient reversion' attacks apply to lifelong learning models in general. Why is this class of approaches particularly informative such that conclusions may hold in general? Are other methods in this class more susceptible to these attacks and can the proposed attack be applied to the whole class, or even other types of approaches? This should be clarified!\n\n\nReferences\n[1] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, Tinne Tuytelaars,  Continual learning: A comparative study on how to defy forgetting in classification tasks, https://arxiv.org/abs/1909.08383", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1x9LHd6tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a novel approach for robust continual learning model from the adversarial attack. The authors start from one of the state-of-the-art episodic memory based continual learning method, A-GEM. The proposed method, Gradient Reversion (GREV), is specialized on A-GEM. The method perturbed the episodic memory examples on A-GEM, thus modifies the direction of reference gradient. While conventional attack like Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) hardly show the their influence on A-GEM, GREV significantly attacks the performance.\n\n\nThe paper is well written, and easy to follow. Also, attack technique on episodic memory based continual learning is interesting and would be valuable. \n\nBut, I feel that some of the analysis are obvious which are not much meaningful to analyze the model,  and the overall contributions are suggested under A-GEM model, while not to cover generic other episodic-based continual learning. \n\nSo, I hesitate to give the high score even the approach is interesting.\n\nAdditional one question.\nI didn't get the concrete reasons that A-GEM is already robust for famous attack methods. What’s the reason that A-GEM is robust for them?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates questions around adversarial attacks in a continual learning algorithm, i.e., A-GEM. While reviewers agree that this is a novel topic of great importance, the contributions are quite narrow, since only a single model (A-GEM) is considered and it is not immediately clear whether this method transfers to other lifelong learning models (or even other models that belong to the same family as A-GEM). This is an interesting submission, but at the moment due to its very narrow scope, it seems more appropriate as a workshop submission investigating a very particular question (that of attacking A-GEM). As such, I cannot recommend acceptance.
Decision: Reject