Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting

1 INTRODUCTION . Humans are capable of continuously learning novel tasks by leveraging their lifetime knowledge and expanding them when they encounter a new experience . They can remember the majority of their prior knowledge despite the never-ending nature of their learning process by simply keeping a running tally of the observations distributed over time or presented in summary form . The field of continual learning or lifelong learning ( Thrun & Mitchell , 1995 ; Silver et al. , 2013 ) aims at maintaining previous performance and avoiding so-called catastrophic forgetting of previous experience ( McCloskey & Cohen , 1989 ; McClelland et al. , 1995 ) when learning new skills . The goal is to develop algorithms that continually update or add parameters to accommodate an online stream of data over time . An active line of research in continual learning explores the effectiveness of using small memory budgets to store data points from the training set ( Castro et al. , 2018 ; Rajasegaran et al. , 2020 ; Rebuffi et al. , 2017 ; Wu et al. , 2019 ) , gradients ( Lopez-Paz et al. , 2017 ) , or storing an online generative model that can fake them later ( Shin et al. , 2017 ) . Memory has been also exploited in the form of accommodating space for architecture growth and storage to fully recover the old performance when needed ( Ebrahimi et al. , 2020b ; Rusu et al. , 2016 ) . Some methods store an old snapshot of the model to distill the features ( Li & Hoiem , 2016 ) or attention maps ( Dhar et al. , 2019 ) between the teacher and student models . The internal reasoning process of deep models is often treated as a black box and remains hidden from the user . However , recent work in explainable artificial intelligence ( XAI ) has developed methods to create human-interpretable explanations for model decisions ( Simonyan et al. , 2013 ; Zhang et al. , 2018 ; Petsiuk et al. , 2018 ; Zhou et al. , 2016 ; Selvaraju et al. , 2017 ) . We posit that the catastrophic forgetting phenomenon is due in part to not being able to rely on the same reasoning as was used for a previously seen observation . Therefore , we hypothesize that forgetting can be mitigated when the model is encouraged to remember the evidence for previously made decisions . In other words , a model which can remember its final decision and can reconstruct the same prior reasoning . Based on this approach , we develop a novel strategy to exploit explainable models for improving performance . Among the various explainability techniques proposed in XAI , saliency methods have emerged as a popular tool to identify the support of a model prediction in terms of relevant features in the input . These methods produce saliency maps , defined as regions of visual evidence upon which a network makes a decision . Our goal is to investigate whether augmenting experience replay with explanation replay reduces forgetting and how enforcing to remember the explanations will affect the explanations themselves . Figure 1 illustrates our proposed method . In this work , we propose RRR , a training strategy guided by model explanations generated by any white-box differentiable explanation method ; RRR adds an explanation loss to continual learning . White-box methods generate an explanation by using some internal state of the model , such as gradients , enabling their use in end-to-end training . We evaluate our approach using various popular explanation methods including vanilla backpropagation ( Zeiler & Fergus , 2014 ) , backpropagation with smoothing gradients ( Smoothgrad ) ( Smilkov et al. , 2017 ) , Guided Backpropagation ( Springenberg et al. , 2014 ) , and Gradient Class Activation Mapping ( Grad-CAM ) ( Selvaraju et al. , 2017 ) and compare their performance versus their computational feasibility . We integrate RRR into several state of the art class incremental learning ( CIL ) methods , including iTAML ( Rajasegaran et al. , 2020 ) , EEIl ( Castro et al. , 2018 ) , BiC ( Wu et al. , 2019 ) , TOPIC ( Tao et al. , 2020 ) , iCaRL ( Rebuffi et al. , 2017 ) , EWC ( Kirkpatrick et al. , 2017 ) , and LwF ( Li & Hoiem , 2016 ) . Note that RRR does not require task IDs at test time . We qualitatively and quantitatively analyze model explanations in the form of saliency maps and demonstrate that RRR remembers its earlier decisions in a sequence of tasks due to the requirement to focus on the the right evidence . We empirically show the effect of RRR in standard and few-shot class incremental learning ( CIL ) scenarios on popular benchmark datasets including CIFAR100 , ImageNet100 , and Caltech-UCSD Birds 200 using different network architectures where RRR improves overall accuracy and forgetting over experience replay and other memory-based method . Our contribution is threefold : we first propose our novel , simple , yet effective memory constraint , which we call Remembering for the Right Reasons ( RRR ) , and show that it reduces catastrophic forgetting by encouraging the model to look at the same explanations it initially found for its decisions . Second , we show how RRR can be readily combined with memory-based and regularization-based CL methods to improve performance . Third , we demonstrate how guiding a continual learner to remember its explanations can improve the quality of the explanations themselves ; i.e. , the model looks at the right region in an image when making correct decisions while it focuses its maximum attention on the background when it misclassifies an object . 2 BACKGROUND : WHITE-BOX EXPLANABILITY TECHNIQUES . Here we briefly review the explainability methods we have evaluated our approach with . The core idea behind RRR is to guide explanations or saliency maps during training to preserve their values . Hence , only gradient-based saliency techniques can be used which are differentiable and hence trainable during training for the mainstream task as opposed to black-box saliency methods which can be used only after training to determine important parts of an image . Vanilla Backpropagation ( Zeiler & Fergus , 2014 ) : The simplest way to understand and visualize which pixels are most salient in an image is to look at the gradients . This is typically done by making a forward pass through the model and taking the gradient of the given output class with respect to the input . Those pixel-wise derivative values can be rendered as a normalized heatmap representing the amount of change in the output probability of a class caused by perturbing that pixel . To store a saliency map for each RGB image of size 3 ×W × H , we need an equivalent memory size of storing W ×H pixel values . Backpropagation with SmoothGrad : Smilkov et al . ( 2017 ) showed that the saliency maps obtained using raw gradients are visually noisy and using them as a proxy for feature importance is sub-optimal . They proposed a simple technique for denoising the gradients that adds pixel-wise Gaussian noise to n copies of the image , and simply averages the resulting gradients . SmoothGrad requires the same amount of memory to store the saliency maps while it takes n times longer to repeat generating each saliency map . We found n = 40 to be large enough to make a noticeable change in the saliencies in our experiments . Gradient-weighted Class Activation Mapping ( Grad-CAM ) ( Selvaraju et al. , 2017 ) : is a whitebox explainability technique which uses gradients to determine the influence of specific feature map activations on a given prediction . Because later layers in a convolutional neural network are known to encode higher-level semantics , taking the gradient of a model output with respect to the activations of these feature maps discovers which high-level semantics are important for the prediction . We refer to this layer as the target layer in our analysis . For example , when using Grad-CAM to visualize explanations for image classification , taking the gradient of the correct class prediction with respect to the last convolutional layer highlights class-discriminative regions in the image ( such as the wings of a bird when identifying bird species ) . Consider the pre-softmax score yc for class c in an image classification output . In general , any differentiable activation can be used . Consider also a single convolutional layer with K feature maps , with a single feature map noted as Ak ∈ Ru×v . Grad-CAM takes the derivative of yc with respect to each feature map Ak . It then performs global average pooling over the height and width dimensions for each of these feature map gradients , getting a vector of length K. Each element in this vector is used as a weight αck , indicating the importance of feature map k for the prediction yc . Next , these importance weights are used in computing a linear combination of the feature maps . Followed by a ReLU ( Jarrett et al. , 2009 ) to zero-out any activations with a negative influence on the prediction of class c , the final Grad-CAM output ( s ) is as below with Akij defined at location ( i , j ) in feature map Ak . αck = 1 uv u∑ i=1 v∑ j=1 ∂yc ∂Akij scGrad-CAM = ReLU ( K∑ k=1 αckA k ) ( 1 ) Unlike the common saliency map techniques of Guided BackProp ( Springenberg et al. , 2014 ) , Guided GradCAM ( Selvaraju et al. , 2016 ) , Integrated Gradients ( Sundararajan et al. , 2017b ) , Gradient Input ( Shrikumar et al. , 2016 ) , Backpropagation with SmoothGrad ( Smilkov et al. , 2017 ) etc. , vanilla Backpropagation and Grad-CAM pass important “ sanity checks ” regarding their sensitivity to data and model parameters ( Adebayo et al. , 2018 ) . We will compare using vanilla Backpropagation , Backpropagation with SmoothGrad , and Grad-CAM in RRR in Section 4 . We will refer to the function that computes the output s of these saliency method as XAI . Algorithm 1 Remembering for the Right Reasons ( RRR ) for Continual Learning 1 : function TRAIN ( fθ , Dtr , Dts ) 2 : T : # of tasks , n : # of samples in task 3 : R← 0 ∈ RT×T 4 : Mrep ← { } 5 : MRRR ← { } 6 : for k = 1 to T do 7 : for i = 1 to n do 8 : Compute cross entropy on task ( Ltask ) 9 : Compute LRRR using Eq . 2 10 : θ′ ← θ − α∇θ ( Ltask+LRRR ) 11 : end for 12 : Mrep , MRRR ← UPDATE MEM ( fkθ , Dtrk , Mrep , MRRR ) 13 : Rk , { 1···k } ← EVAL ( fkθ , Dts { 1···k } ) 14 : end for 15 : return fθ , R 16 : end function function UPDATE MEM ( fkθ , Dtrk , Mrep , MRRR ) ( xi , k , yi ) ∼ Dtrk Mrep ←Mrep ∪ { ( xi , k , yi ) } ŝ← XAI ( fkθ ( xi , k ) ) MRRR ←MRRR ∪ { ŝ } returnMrep , MRRR end function function EVAL ( fkθ , Dts { 1···k } ) for i = 1 to k do Rk , i = Accuracy ( fkθ ( x , i ) , y|∀ ( x , y ) ∈ Dtsi ) end for return R end function 3 REMEMBERING FOR THE RIGHT REASONS ( RRR ) . Memory-based methods in continual learning have achieved high performance on vision benchmarks using a small amount of memory , i.e . storing a few samples from the training data into the replay buffer to directly train with them when learning new tasks . This simple method , known as experience replay , has been explored and shown to be effective ( Rebuffi et al. , 2017 ; Wu et al. , 2019 ; Castro et al. , 2018 ; Rajasegaran et al. , 2020 ; Ebrahimi et al. , 2020b ; Hayes et al. , 2019 ; Riemer et al. , 2018 ) . In this work we aim to go one step further and investigate the role of explanations in continual learning , particularly on mitigating forgetting and change of model explanations . We consider the problem of learning a sequence of T data distributions Dtr = { Dtr1 , · · · , DtrT } , whereDtrk = { ( xki , yki ) nk i=1 } is the data distribution for task k with n sample tuples of input ( xk ⊂ X ) and set of output labels ( yk ⊂ Y ) . The goal is to sequentially learn the model fθ : X × T → Y for each task that can map each input x to its target output , y , while maintaining its performance on all prior tasks . We aim to achieve this by using memory to enhance better knowledge transfer as well as better avoidance of catastrophic forgetting . We assume two limited memory poolsMrep for raw samples andMRRR for model explanations . In particular , Mrep = { ( xji , y j i ) m i=1 ∼ Dtrj=1···k−1 } stores m samples in total from all prior tasks to k. SimilarlyMRRR stores the saliency maps generated based on fkθ by one of the explanation methods ( XAI ) discussed in Section 2 for images in Mrep where fkθ is fθ being trained for task k. We use a single-head architecture where the task ID integer t is not required at test time . Upon finishing the kth task , we randomly select m/ ( k−1 ) samples per task from its training data and update our replay buffer memoryMrep . RRR uses model explanations on memory samples to perform continual learning such that the model preserves its reasoning for previously seen observations . We explore several explanation techniques to compute saliency maps using fkθ for the stored samples in the replay buffer to populate the xai buffer memoryMxai . The stored saliency maps will serve as reference explanations during the learning of future tasks to prevent model parameters from being altered resulting in different reasoning for the same samples . We implement RRR using an L1 loss on the error in saliency maps generated after training a new task with respect to their stored reference evidence . LRRR ( fθ , Mrep , MRRR ) = E ( ( x , y ) , ŝ ) ∼ ( Mrep , MRRR ) ||XAI ( fkθ ( x ) ) − ŝ||1 ( 2 ) where XAI ( · ) denotes the explanation method used to compute saliency maps using the model trained on the last seen example from task k , and ŝ are the reference saliency maps generated by XAI ( fkθ ) upon learning each task prior to T and stored in to the memory . We show below that combining RRR into the objective function of state-of-the-art memory and regularization-based methods results in significant performance improvements . The full algorithm for RRR is given in Alg . 1 .

Summary: This paper tackles the problem of catastrophic forgetting in a continual learning scenario, in which the same classifier is trained incrementally on new classification tasks, each defined on a new set of output classes, and asked to retain performance on all the previous tasks. To tackle this problem, a stream of approaches keep a minimal "replay buffer" of examples and their labels from each task, that can be accessed during sequential task learning. This paper belongs to this stream of approaches. A naive way of using the replay buffer is to mix the buffer to the current task training set. Additionally, this paper proposes to store gradient attribution maps in the replay buffer and use the stored gradient attribution maps as additional targets for the current model (RRR loss): the model should have "explanations" of previous examples that roughly don't change upon seeing new evidence from other classes. The authors show that the proposed RRR loss can be applied to multiple baselines and improves performance on a continual learning and few-shot continual learning scenario.