Attacking Lifelong Learning Models with Gradient Reversion

Lifelong learning aims at avoiding the catastrophic forgetting problem of traditional supervised learning models . Episodic memory based lifelong learning methods such as A-GEM ( Chaudhry et al. , 2018b ) are shown to achieve the stateof-the-art results across the benchmarks . In A-GEM , a small episodic memory is utilized to store a random subset of the examples from previous tasks . While the model is trained on a new task , a reference gradient is computed on the episodic memory to guide the direction of the current update . While A-GEM has strong continual learning ability , it is not clear that if it can retain the performance in the presence of adversarial attacks . In this paper , we examine the robustness of A-GEM against adversarial attacks to the examples in the episodic memory . We evaluate the effectiveness of traditional attack methods such as FGSM and PGD . The results show that A-GEM still possesses strong continual learning ability in the presence of adversarial examples in the memory and simple defense techniques such as label smoothing can further alleviate the adversarial effects . We presume that traditional attack methods are specially designed for standard supervised learning models rather than lifelong learning models . we therefore propose a principled way for attacking A-GEM called gradient reversion ( GREV ) which is shown to be more effective . Our results indicate that future lifelong learning research should bear adversarial attacks in mind to develop more robust lifelong learning algorithms . 1 INTRODUCTION . Lifelong learning ( French , 1999 ; Thrun & Mitchell , 1995 ; Kirkpatrick et al. , 2017 ) aims at improving the continual learning ability of neural networks . Standard supervised learning methods suffer from the problem of catastrophic forgetting , in which case the models gradually forget previously learned knowledge while learning on a sequence of new tasks . In lifelong learning , neural networks are equipped with the capability to learn new tasks while maintaining the performance on the tasks trained previously . Lifelong learning models with continual learning ability can be deployed in complex environments with the aim to process a continuous stream of information . Several methodologies are proposed recently to address the catastrophic forgetting problem . In Kirkpatrick et al . ( 2017 ) , the authors adopt Fisher information matrix to prevent important weights for old tasks from drastic changes while the model is trained on a new task . While in Rusu et al . ( 2016 ) , a neural network that has lateral connections with old tasks is trained each time for the new task . Recently , lifelong learning methods based on episodic memory ( Lopez-Paz et al. , 2017 ; Chaudhry et al. , 2018b ; d ’ Autume et al. , 2019 ) such as A-GEM ( Chaudhry et al. , 2018b ) are shown to be able to achieve the state-of-the-art performance across several benchmarks . In A-GEM , a small episodic memory is utilized to store a random subset of the examples from old tasks . While the model is trained on a new task , a reference gradient is computed on a batch of the samples from the episodic memory to guide the current update direction . If the angle between the reference gradient and the current gradient computed on the new task is obtuse , the current gradient is projected to be perpendicular to the reference gradient . The strong continual learning ability of A-GEM relies on the episodic memory which can give a hint on the performance of the current model on old tasks . It has been known that in the standard super- vised learning setting , deep neural networks can be easily fooled by adversarial examples ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) . A natural question then arises : Can A-GEM retain the continual learning ability in the presence of adversarial examples in the episodic memory ? In this paper , we systematically evaluate the robustness of A-GEM against traditional adversarial attack methods such as FGSM ( Goodfellow et al. , 2014 ) and PGD ( Madry et al. , 2017 ) . The results show that A-GEM is surprisingly robust under traditional adversarial attacks . We therefore propose gradient reversion ( GREV ) attack , which is a principled way for attacking episodic memory based lifelong learning algorithms such as A-GEM . Essentially , GREV alters the direction of reference gradient computed on the episodic memory by slightly perturbing the examples . Our results show that for future research on lifelong learning , it is important to design algorithms bearing adversarial attacks in mind . In this paper , we have the following contributions , • To the best of our knowledge , we are the first to systematically evaluate the robustness of episodic memory based lifelong learning algorithms such as A-GEM . • We show that simple adversarial attack methods such as fast gradient sign method ( FGSM ) ( Goodfellow et al. , 2014 ) and projected gradient descent ( PGD ) ( Madry et al. , 2017 ) can hardly hurt the performance of A-GEM . Defense techniques such as label smoothing can be used to further alleviate the adversarial effect . • We propose a principled way called gradient reversion ( GREV ) for attacking A-GEM . On Permuted MNIST , we show that A-GEM achieves an accuracy which is 40 % lower than the original accuracy under the proposed GREV attack . On Split CIFAR , while FGSM and PGD can not hurt the performance of A-GEM , the proposed GREV degrades the accuracy of A-GEM by as much as 20 % . 2 BACKGROUND . 2.1 LIFELONG LEARNING . In a lifelong learning task , suppose there are a sequence ofN datasets denoted as { D1 , D2 , ... , DN } . Each dataset Di is a collection of pairs { xij , yij } , where xij is the j-th example of task i and yij is the corresponding label . A model f ( w ; x ) with weight w is trained continually on the tasks with a single pass over the examples of each task . We denote the weight as wi while the model is trained on the i-th task and the training loss on the i-th task is denoted as ` ( wi ; Di ) . The most commonly used metric for evaluating the performance of lifelong learning models is Average Accuracy ( AA ) , which is the average test accuracy on the test set of each task after the model finishes training on all tasks . In order to achieve a high Average Accuracy , the model should maintain the performance on old tasks while training on a new task . 2.2 A-GEM . In this section , we review the Averaged Gradient Episode Memory ( A-GEM ) ( Chaudhry et al. , 2018b ) , one of the state-of-the-art lifelong learning methods . In A-GEM , a small episodic memory M with fixed size is used to store a subset of the examples from old tasks . The episodic memory is populated by choosing examples uniformly at random for each task . Mk is used to denote the examples in the episodic memory from task k. While training on task i , the loss on the episodic memory M can be computed as ` ( wi ; M ) , where M = ∪k < iMk . A-GEM ensures that each update on the i-th task will not increase the loss on the episodic memory , that is , minw ` ( w ; Di ) s.t . ` ( w ; M ) ≤ ` ( wi−1 ; M ) where M = ∪k < iMk ( 1 ) To inspect the increase of loss on the episodic memory , A-GEM computes the gradient g on the current task and the reference gradient gref on the episodic memory . When the angle between g and gref is obtuse , A-GEM projects the current gradient g to have a right or acute angle with gref , mingtrue 1 2 ‖g − gtrue‖22 s.t . g > truegref ≥ 0 ( 2 ) The above optimization problem can be solved in closed form as , gtrue = g − g > gref g > refgref gref ( 3 ) The current gradient g is then replaced by gtrue for updating the model . Essentially , A-GEM avoids the catastrophic forgetting problem by altering the direction of the current gradient . When the current gradient is detrimental to old tasks , it is adjusted to be perpendicular or acute with the gradient computed on the episodic memory . 3 A THREAT MODEL FOR ATTACKING AND DEFENSE OF MEMORY BASED LIFELONG LEARNING ALGORITHMS . In this section , we specify the threat model used in the paper for attacking A-GEM . Specially , we consider a white-box adversary which has access to the model architecture and parameters . In addition , the adversary is allowed to perturb the examples in the episodic memory . However , we do not expose the training process to the adversary , that is , the adversary can only perturb the examples in the episodic memory in an offline fashion . Specially , before the model is trained on the i-th task , the adversary can slightly perturb the examples from task i−1 in the episodic memory . In this setting , each example is only perturbed once during the lifelong learning process . We refer to this threat model as Offline Sequential Attack . The defender , on the other hand , has access to the reference gradients but not the perturbed examples in the episodic memory . The defender is further allowed to take advantage of the labels of the perturbed examples in the memory for defending possible attacks . The proposed threat model is a generalized version of white-box adversaries ( Goodfellow et al. , 2014 ; Carlini & Wagner , 2017a ) for lifelong learning setting . The Average Accuracy of the lifelong learning model with and without adversarial attack is referred to as perturbed accuracy and unperturbed accuracy respectively . 4 TRADITIONAL ATTACK METHODS . The objective of traditional adversaries is to find an adversarial example xadv for x such that they are imperceptibly close and yet the neural network labels them distinctly . We bound the ` p distance between an input x and its adversarial counterpart : xadv ∈ Sp ( x ) : = { x′ : ‖x − x′‖p ≤ τp } , where p = 2 or∞ . We omit from Sp ( x ) the argument x and the subscript p when it does not cause ambiguity . We focus on ` ∞ bounded attack in this paper since ` ∞ distance has been shown as a natural metric to measure adversarial perturbations . FGSM Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2014 ) is a simple one-step adversarial attack method . It is featured by efficiency and high performance for generating ` ∞ bounded adversarial examples . In FGSM , the adversarial counterpart of example x is produced by , x + sign ( ∇x ` ( w ; x ) ) ( 4 ) PGD A more powerful attack technique has been proposed as a multi-step variant of FGSM , which is called projected gradient descent ( PGD ) ( Madry et al. , 2017 ) , xn+1 = ΠS ( xn + ηsign ( ∇xn ` ( w ; xn ) ) ( 5 ) where S = { x : ‖x− xn‖ ≤ τ } and Π is the projection operator . Rotation We also consider a simple variant of the rotation attack introduced in Engstrom et al . ( 2017 ) . If the example x is rotated by an angle θ , the pixel at position ( u , v ) is translated to ( u′ , v′ ) via the following equation , [ u′ v′ ] = [ cos θ − sin θ sin θ cos θ ] · [ u v ] ( 6 )

Summary: The paper proposed a novel approach for robust continual learning model from the adversarial attack. The authors start from one of the state-of-the-art episodic memory based continual learning method, A-GEM. The proposed method, Gradient Reversion (GREV), is specialized on A-GEM. The method perturbed the episodic memory examples on A-GEM, thus modifies the direction of reference gradient. While conventional attack like Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) hardly show the their influence on A-GEM, GREV significantly attacks the performance.