Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting
Authors: Sayna Ebrahimi
Suzanne Petryk
Akash Gokul
William Gan
Joseph E. Gonzalez
Marcus Rohrbach
trevor darrell
Venue: ICLR 2021 Poster (2021)
Keywords: Continual Learning
Lifelong Learning
Catastrophic Forgetting
XAI
Explainability

Abstract: The goal of continual learning (CL) is to learn a sequence of tasks without suffering from the phenomenon of catastrophic forgetting. Previous work has shown that leveraging memory in the form of a replay buffer can reduce performance degradation on prior tasks. We hypothesize that forgetting can be further reduced when the model is encouraged to remember the \textit{evidence} for previously made decisions. As a first step towards exploring this hypothesis, we propose a simple novel training paradigm, called Remembering for the Right Reasons (RRR), that additionally stores visual model explanations for each example in the buffer and ensures the model has ``the right reasons'' for its predictions by encouraging its explanations to remain consistent with those used to make decisions at training time. Without this constraint, there is a drift in explanations and increase in forgetting as conventional continual learning algorithms learn new tasks. We demonstrate how RRR can be easily added to any memory or regularization-based approach and results in reduced forgetting, and more importantly, improved model explanations. We have evaluated our approach in the standard and few-shot settings and observed a consistent improvement across various CL approaches using different architectures and techniques to generate model explanations and demonstrated our approach showing a promising connection between explainability and continual learning. Our code is available at \url{https://github.com/SaynaEbrahimi/Remembering-for-the-Right-Reasons}.

Published as a conference paper at ICLR 2021
REMEMBERING FOR THE RIGHT REASONS: EXPLANATIONS REDUCE CATASTROPHIC FORGETTING
Sayna Ebrahimi Suzanne Petryk', Akash Gokul', William Gan', Joseph E. Gonzalez', Marcus Rohrbach', Trevor Darrell'
UC Berkeley, Facebook AI Research
{sayna, spetryk, akashgokul, wigan, Jegonzal, httppp//aa...oooor..aaaaaaaaaaa. mr we.ffb.com com
ABSTRACT
The goal of continual learning (CL) is learn sequence of tasks without suf- fering from the phenomenon of catastrophic forgetting. Previous work has shown that leveraging memory in the form of replay bufler can reduce performance degradation on prior tasks. We hypothesize that forgetting can be further reduced when the model is encouraged remember the evidence for previously made decisions. As a first step towards exploring this hypothesis, we propose sim- ple novel training paradigm, called Remembering for the Right Reasons (RRR). that additionally stores visual model explanations for each example in the buffer and ensures the model has "the right reasons" for its predictions by encourag- ing its explanations remain consistent with those used to make decisions at training time. Without this constraint, there is a drift in explanations and in- crease in forgetting conventional continual learning algorithms learn new tasks. We demonstrate how RRR can be easily added to any memory or regularization- based approach and results in reduced forgetting, and more importantly, improved model explanations. We have evaluated our approach in the standard and few-shot settings and observed consistent improvement across various CL approaches using different architectures and techniques to generate model explanations and demonstrated our approach showing a promising connection between explainabil- ity and continual learning. Our code is available at https::/ggthti:i com/ (SaynEBbrahimi www.immiieeeeeeeeeeeeeee iinbbeeringgoooooo for the Ri Right -iwtorrttttttttttttttte-ooe Reasons
INTRODUCTION
Humans are capable of continuously learning novel tasks by leveraging their lifetime knowledge and expanding them when they encounter
experience. They remember the majority of their prior knowledge despite the never-ending nature of their learning process by simply keeping a running tally the observations distributed over time or presented summary form. The field of continual learning or lifelong learning (Thrun & Mitchell 1995 Silver et al., 2013) aims at main- taining previous performance and avoiding so-called catastrophic forgetting previous experience (McCloskey & Cohen 1989 McClelland et al. 1995) when learning new skills. The goal is to develop algorithms that continually update or add parameters accommodate an online stream of data over time.
active line of research continual learning explores the effectiveness of using small memory budgets to store data points from the training set (Castro et al. 20188; Rajasegaran et al. 2020 Re- buffi et al. 2017 Wu et al.) 2019), ), gradients (Lopez-Paz et a 2017). or storing an online generative model that can fake them later (Shin et al. 2017). Memory has been also exploited in the form of accommodating space for architecture growth and storage to fully recover the old performance when needed (Ebrahimi et al.) 2020b Rusu et al. 2016). Some methods store an old snapshot of the model to distill the features (Li & Hoiem 2016) or attention maps (Dhar et al. 2019) between the teacher and student models.

---

Published as a conference paper at ICLR 2021
Lual% Y.y.%.ym myy)
Aane (aam
Figure I: An illustration of applying RRR paradigm. (Left) In a typical experience replay scenario, samples from prior tasks are kept in memory buffer Ma0 and revisited during training. (Right) In our proposed idea (RRR), in addition to MdP we also store model explanations (saliency maps) as .RRR for those samples and encourage the model remember the original reasoning for the prediction. Note that the saliency maps are small masks resulting in a negligible memory overhead (see Section 4.J
The internal reasoning process of deep models often treated as black box and remains hidden from the user. However, recent work in explainable artificial intelligence (XAI) has developed meth- ods to create human-interpretable explanations for model decisions (Simonyan et al., 2013 Zhang et al. 2018 Petsiuk et al.
2018 Zhou et al. 2016 Selvaraju al. 2017). We posit that the catas- trophic forgetting phenomenon is due in part to not being able to rely on the same reasoning as was used for previously when the model encouraged to remember the evidence for previously made decisions. In other words, model which remember its final decision and can reconstruct the same prior reasoning. Based on this approach, we develop a novel strategy to exploit explainable models for improving performance.
observation. Therefore,
hypothesize that forgetting can be mitigated
Among the various explainability techniques proposed in XAI, saliency methods have emerged as a popular tool to identify the support of model prediction in terms of relevant features in the input. These methods produce saliency maps, defined as regions of visual evidence upon which network makes a decision. Our goal to investigate whether augmenting experience replay with explanation replay reduces forgetting and how enforcing to remember the explanations will affect the explanations themselves. Figure illustrates our proposed method.
In this work, we propose RRR, training strategy guided by model explanations generated by any white-box differentiable explanation method; RRR adds an explanation loss to continual learning. White-box methods generate an explanation by using some internal state of the model, such as gra- dients, enabling their use end-to-end training. We evaluate our approach using various popular explanation methods including vanilla backpropagation (Zeiler & Fergus 2014), backpropagation with smoothing gradients (Smootbgrad) (Smilkov et al. (017)) Guided Backpropagation (Sprin- genberg et al. 2014), and Gradient Class Activation Mapping (Grad-CAM) (Selvaraju et al. 2017) and compare their performance versus their computational feasibility. We integrate RRR into sev- eral state of the art class incremental learning (CIL) methods, including iTAML (Rajasegaran al., 2020). EEN (Castro et al. 2018), BiC (Wu et al. 2019). TOPIC (Tao et al., 2020). iCaRL (Rebuffi al. 2017). EWC (Kirkpatrick al., 2017), and LwF LL Hoiem (016).. Note that RRR does not require task IDs at test time. We qualitatively and quantitatively analyze model explanations in the form of saliency maps and demonstrate that RRR remembers its earlier decisions in a sequence of tasks due to the requirement to focus on the the right evidence. We empirically show the effect of RRR standard and few-shot class incremental learning (CIL.) scenarios on popular benchmark datasets including CIFAR 100, ImageNet100, and Caltech-UCSD Birds 200 using different network architectures where RRR improves overall accuracy and forgetting over experience replay and other memory-based method.
Our contribution is threefold: we first propose our novel, simple, yet effective memory constraint, which we call Remembering for the Right Reasons (RRR), and show that it reduces catastrophic for- getting by encouraging the model to look at the same explanations it initially found for its decisions. Second, show how RRR can be readily combined with memory-based and regularization-basel
2

---

Published as a conference paper at ICLR 2021
CL methods to improve performance. Third, we demonstrate how guiding a continual learner to remember its explanations can improve the quality of the explanations themselves; i.c., the model looks at the right region in image when making correct decisions while it focuses its maximum attention on the background when it misclassifies an object.
2 BACKGROUND: WHITE-BOR EXPLANABILITY TECHNIQUES
Here we briefly review the explainability methods we have evaluated our approach with. The core idea behind RRR to guide explanations or saliency maps during training to preserve their values. Hence, only gradient-basee based saliency techniques can be used which are differentiable and hence trainable during training for the mainstream task as opposed to black-box saliency methods which can be used only after training to determine important parts of an image.
Vanilla Backpropagation (Zeiler & Fergus 2014): The simplest way to understand and visualize which pixels are most salient in an image is to look at the gradients. This is typically done by making a forward pass through the model and taking the gradient of the given output class with respect to the input. Those pixel-wise derivative values can be rendered as a normalized heatmap representing class caused by perturbing that pixel. To store
the amount of change in the output probability a saliency map for each RGB image of size 3 x W x H, we need an equivalent memory size of storing W x H pixel values.
Backpropagation with SmoothGrad: [Smilkov et al. (2017) showed that the saliency maps ob- tained using raw gradients are visually noisy and using them as a proxy for feature importance is sub-optimal. They proposed simple technique for denoising the gradients that adds pixel-wise Gaussian noise to n copies of the image, and simply averages the resulting gradients. SmoothGrad requires the same amount of memory to store the saliency maps while it takes times longer to repeat generating each saliency map. We found n = 40 to be large enough to make a noticeable change the saliencies in our experiments.
Gradient-weighter Class Activation Mapping (Grad-CAM) (Selvaraju et al. 2017): is a white- box explainability technique which uses gradients determine the influence of specific feature map activations on a given prediction. Because later layers in a convolutional neural network are known to encode higher-level semantics, taking the gradient of a model output with respect to the activations of these feature maps discovers which high-level semantics are important for the prediction. We refer to this layer as the target layer in our analysis. For example, when using Grad-CAM to visualize explanations for image classification, taking the gradient of the correct class prediction with respect to the last convolutional layer highlights class discriminative regions in the image (such as the wings of bird when identifying bird species).
Consider the pre-softmax score Ve for class c in an image classification output. In general, any differentiable activation can be used. Consider also single convolutional layer with K feature maps, with a single feature map noted as A E Ruxe_ Grad-CAM takes the derivative of ye with respect to each feature map A². It then performs global average pooling over the height and width dimensions for each of these feature map gradients, getting a vector of length K. Each element in this vector is used as a weight ai, indicating the importance of feature map for the prediction ye- Next, these importance weights are used in computing linear combination of the feature maps. Followed by a ReLU (Jarrett et al.. 2009) to zero-out any activations with a negative influence on the prediction of class c, the final Grad-CAM output (s) is as below with A defined at location (i,j) in feature map A*.
Tye DAh
Eradd-CAM ReLU
Σ
Unlike the common saliency map techniques of Guided BackProp (Springenberg et al.) 2014), Guided GradCAM (Selvaraju et al. 2016), Integrated Gradients (Sundararajan et 20. 7b), Gradi- ent O Input (Shrikumar al. 2016 Backpropagation with SmoothGrad (Smilkov et al. 2017) etc., vanilla Backpropagation and Grad pass important "sanity checks" regarding their sensitivity to data and model parameters (Adebayo et al. 2018). We will compare using vanilla Backpropaga- tion. Backpropagation with SmoothGrad, and Grad-CAM in RRR in Section 4 We will refer to the function that computes the output of these saliency method as X'AI.
3

---

Published as a conference paper at ICLR 2021
Algorithm 1 Remembering for the Right Reasons (RRR) for Continual Learning
1: function TRAIN (fo, D"t)
function UPDATE MEM(JÖ; MMP M M
2: T: # of tasks, n:: # of samples in task
(Ti, ki Vi) D
Rt 0 DER?T
Ma + M {(r,, H.)}
4:
M + 0
XJ (L (2i,
5:
.ARRR < 0
M MRRK + -MRRR {88)
for doo
return M M
7:
for i = do
end function
8:
Compute cross entropy on task (Lmat)
9:
Compute LARR using Eq
function EVALU D P(i....)) for = 1 to k do
10:
@' <- 0 aVo (Cus CRRR) end
11:
R = Accuracy (z, u.v y) E D(
12:
MES .MARK RRR +- UPDATE MEM fo- MMM MRRR)
end for return R end function
13:
(1..1...)) 4- EVAL ((i...)))
14:
end for
15:
return fo.
16:
end function
3 REMEMBERING FOR THE RIGHT REASONS (RRR)
Memory-based methods continual learning have achieved high performance on vision bench- marks using a small amount of memory, i.c. storing few samples from the training data into the replay buffer to directly train with them when learning new tasks. This simple method, known as ex-
perience replay, has been explored and shown to be effective (Rebuffi et al. 2017 Wu et al. 2019
Castro et al. 2018 Rajasegaran et al. 2020; Ebrahimi et al. 2020b Hayes et al. 2019: Riemer al. 2018). In this work aim go one step further and investigate the role of explanations in continual learning, particularly on mitigating forgetting and change of model explanations.
We consider the problem of learning sequence of T data distributions D {DI", D4
where Drr = {(*, y Fi=1 is the data distribution for task i with sample tuples of input (x" X) and set of output labels (y* C). The goal is to sequentially learn the model fo X x X y for each task that can map each imput æ to its target output, y. while maintaining its performance on all prior tasks. We aim to achieve this by using memory enhance better knowledge transfer as well as better avoidance of catastrophic forgetting. We assume two limited memory pools M for raw samples and MRRR for model explanations. In particular, M = {(x), y/ y/ Dir DDj -
stores samples in total from all prior tasks to k. Similarly MRRR stores the saliency maps gen- erated based on f by one of the explanation methods (X. discussed in Section 2 for images in M where fö is fe being trained for task k. We use a single-head architecture where the task ID integer t is not required at test time.
Upon finishing the ttth task, we randomly select m/(k-1) samples per task from its training data and update our replay buffer memory M RRR uses model explanations memory samples to perform continual learning such that the model preserves its reasoning for previously seen observa- tions. We explore several explanation techniques to compute saliency maps using f for the stored samples in the replay buffer to populate the xai buffer memory M The stored saliency maps will serve as reference explanations during the learning of future tasks to prevent model parameters from being altered resulting in different reasoning for the same samples. We implement RRR using an L1 loss on the error in saliency maps generated after training new task with respect to their stored reference evidence.
LRRR (fo, TP .RRRR) B(( .MERR |I 81/1
where X-AI(-) denotes the explanation method used to compute saliency maps using the model trained on the last seen example from task k. and are the reference saliency maps generated by X' ( upon learning each task prior to T and stored in to the memory. We show below that com- bining RRR into the objective function of sate-of-the-as memory and regularization baasz methods results in significant performance improvements. The full algorithm for RRR is given in Alg. [
4

---

Published as a conference paper at ICLR 2021
10-way 5-shot on CUB200 in 11 Tasks
10-way S-shot on CUB200 in 11 Tasks
DanNeti-- EI BIR (OOEICAII
20TIC-184
SCLL
TEPIC
20
Roclaall 13+
JOHYL FIRAA
Jute
Atcckieon en IAI (Cru CAND
Atcched-211
Accurscy
No. 30R CAN))
Ne1
Squecer Ne. Partooo
28
30
100
122
132
140
150
163
180
100
133
140
1.0
172
190
203
Number of Classres
Number of Cllusaos
Figure 2: Few-shot CIL learning of CUB200 in 11 tasks where each point shows the classification accuracy on all seen classes so far. (Left) Shows ER with and without LRRR using different back- bone architectures and saliency map techniques. (Right) Performance of the ofate-of-the-ar existing approaches with and without LRRR on CUB200 including TOPIC (Tao et al., 2020). EEIL (Castro et al., (018)) 8), iCaRL (Rebufhi et al.] (017). Joint training serves as the upper bound. Results for baselines are obtained using their original implementation. All results are averaged over 3 runs and mean and standard deviation values are given in the appendix. Best viewed in color.
EXPERIMENTS
In this section, apply RRR in two distinct learning regimes: standard and few-shot class incre- mental learning. These are the most challenging CL scenarios, in which task descriptions are not available at test time. We first explore the effect of backbone architecture and the saliency map technique on RRR performance. We then report our obtained results integrating Lrrr into existing memory-basee based and regularization-basee methods.
4.1 FEW-SHOT CIL PERFORMANCE
We first explore CIL of low-data regimes where preventing overfitting to few-shot new classes is an- other challenge to overcome in addition to avoiding catastrophic forgetting of old classes. We use C classes and K training samples per class as the C-way K K-shot few-shot class incrementally learning setting where we have set of base classes to learn as the first task while the remaining classes are learned with only a few randomly selected samples. In order to provide direct comparison to the of-te-of-the-art work of Tao al. a (2020) we precisely followed their setup and and used the same Caltech-UCSD Birds dataset (Wah al. 2011), divided into 11 disjoint tasks and 10-way 5-shot setting, where the first task contains b 100 base classes resulting in 3000 samples for training and 2834 images for testing. The remaining 100 classes are divided into 10 tasks where 5 samples per class are randomly selected as the training set, while the test set is kept intact containing near 300 images per task. The images in CUB200 224 x 224 for training. We store 4 images per class from base classes in the first task and sample per each few-shot class in the remaining 10 tasks (Tao et al. (2020). We used the RAdam (Liu al.) 2019) optimizer with a learning rate of 0.001 which was reduced by 0.2 at epochs 20, 40, and 60 and trained for total of 70 epochs with a batch size of 128 for the first and 10 for the remaining tasks.
resized to 256 x 256 and then randomly cropped to
Figure 2 (left) shows results for ER with and without LRrr using different backbone architectures and saliency map techniques. Among the tested saliency map methods, Grad-CAM on ResNet18 outperforms Vanilla Backpropagation and SmoothGrad by 2-3% while SmoothGrad and vanilla Backpropagation achieve similar CL performance. To compute the memory overhead of storing the output for saliency method, we assume the memory required to store an image is M, vanilla Backpropagation and SmoothGrad generate a pixel-wise saliency map that occupies M/3 of mem- ory. However, in Grad-CAM the saliency map size is equal to the feature map of the target layer in the architecture. In our study with Grad-CAM layer before the fully- colly-connected layers. For instance using ResNet18 for colored 224 x 224 images results in the Grad-CAM output of 7 occupying 196B. Table shows the target layer name and
chose target layer to be the last convolution
5

---

Published as a conference paper at ICLR 2021
CIPAR350 Tocks
CITAR300 Trake
ImageNeet00 Tarks
Figure 3: Effect of RRR on existing methods for CIL on CIFAR100 in (a) 10 and (b) 20 tasks and (c) ImageNer100 in 10 tasks. Each point shows the classification accuracy all seen classes so far. Results for iTAML, BiC, and EEIL are produced with their original implementation while EWC and LwF are re-implemented by us. All results averaged over runs and mean and standard deviation values are given in the appendix. Best viewed in color.
saliency map size for other network architectures used in this work (AlexNet and SqueezeNetl_.) as well.
Figure (right) shows the effect of adding Crrr on existing recent state-of-the-ar methods such as TOPIC Tao et al., (020)) EEIL (Castro et al., 2018), and iCaRL (Rebuffi et al. (017).. TToo et al. 2020 used a neural gas network (Martinetz et al. 1991; Fritzke et al.) 1995) which can learn and preserve the topology of the feature manifold formed by different classes and we have followed their experimental protocol for our CUB200 experiment by using identical samples drawn in each task which are used across all the baselines for fair comparison. Adding Lrrr improves the performance of all the baselines; TOPIC becomes nearly on-par with joint training which serves as the upper bound and does not adhere to continual learning. The gap between ER and iCaRL is also reduced when ER uses LRRR.
4.2 STANDARD CIL PERFORMANCE
In order to provide direct comparison the recent work of RRaaseegaann et al. (2020) we perform our standard CIL experiment on CIFAR100 (Krizhevsky & Hinton| 2009) and ImageNet100 where we assume a memory budget of 2000 samples which are identical across all the baselines. Following Rajasegaran et al. (2200) we divide CIFAR 100 to 10 and 20 disjoint tasks with 10 and 5 classes at time. Figures Sa and show the classification accuracy upon learning each task on all seen classes. We see a consistent average improvement of 2 - 4% when Lrrr is added as an additional constraint to preserve the model explanations across all methods, from the most naive memory- -baed method, experience replay (ER), more sophisticated approaches which store set of old class exemplars along with meta-learning (iTAML), correct bias for new classes (BiC), and fine tune on the exemplar set (EEIL). We also applied the RRR constraint on regularization-base LWF with no memory used as a replay buffer. The accuracy for both improves despite not benefiting from revisiting the raw data. However, they fall behind all memory-based methods with or without LRRR. The final accuracy on the entire sequence for joint training (multi-task learning) with RAdam optimizer (Liu et al. 2019) is 80.03% which serves as an upper bound it has access to data from all tasks at all time.
methods such EWC and
Figure 3d shows our results on learning ImageNet100 in 10 tasks. The effect of adding CRRR to baselines on the ImageNet100 experiment is more significant (3-66% 6%% compared to CIFARIO0, This mainly due to the larger size and better quality of images in ImageNer100. resulting in generating larger Grad-CAM saliency maps. These experiments clearly reveal the effectiveness of LRRR on model explanations in a continual learning problem at nearly zero cost of memory overhead when a memory buffer is already created and applied as a catastrophic forgetting avoidance strategy. This makes Grad-CAN the ideal approach to generate saliency maps when applying the RRR training strategy, as it achieves the highest accuracy while utilizing the least storage space to store saliencies. Note that we adopt Grad-CL -AM to generate saliency maps in the remaining experiments in this paper. We also keep using only ResNet18 8 for fair comparison with the literature.

---

Published as a conference paper at ICLR 2021
Table 1: PG experiment results on few-shot CIL CUB200 measuring (a) PG-ACC (%) and PG-BWT (%) and (b) precision and recall averaged over all tasks. Pre and Re,, evaluate the pointing game on each task t' directly after the model has been trained on t'. Prr, and Rey, obtained by the evaluation for task t' using the model trained for all I tasks.
(b) Precision and recall using PG experiment
(a) PG localization accuracy and backward transfer
Precision
Recall
Methods
PG-ACC (5)
PG-BWT (%)
Methods
Pri.
Prr.
Re,.,
Rey.
ER
54.0
11..4
ER+RRR
58.5
-15.6
80.0
68.9
64.1
65.1
TOPIC
72.7
.0.9
ER+RRR
82.1
70.3
61.2
06.8
TOPIC+RRR
74.2
-2.1
TOPIC
91.0
88.4
98.1
97.4
TOPIC+RRR
92.8
89.1
99.6
99.2
5 ANALYSIS OF MODEL EXPLANATIONS
In this section we want to answer the question "How often does the model remember its decision for the right reason upon learning a sequence of tasks?". In particular, we want to evaluate how often the model is "pointing"
the right evidence for its predictions, instead of focusing its maximum attention on the background or other objects in the image. We use the Pointing Game experiment (PG) Zhang et al. 2018) for this evaluation, which was introduced to measure the discriminative- ness of a visualization method for target object localization. Here, we use ground truth segmentation annotation labels provided with the CUB-200 dataset to define the true object region.
First, we look into hits and misses defined by the PG experiment. When the location of the maximum in a predicted saliency map falls inside the object, it is considered as a hit and otherwise it is a miss. Figure [ shows example from CUB200 where the segmentation annotation is used to determine whether the peak of the predicted saliency map (marked with red cross) falls inside the object region. This example is regarded as hit as the red cross is inside the segmentation mask for the bird. PG localization accuracy is defined as the number of hits over the total number of predictions. We would like measure both the overall PG performance of a continual learner as well as how much learning new tasks causes it to forget its ability to hit the target object. For these metrics, inspired by (Lopez-Paz et al., 2017), we define PG-ACC = 1/4 E Rr, as the average PG localization accuracy computed over all prior tasks after training for each new task and PG-BWT = - Ei=1 RT, Rt,; (backward transfer) which indicates how much learning new tasks has influenced the PG localization accuracy on previous tasks where Ra.; is the on task i after learning the th task. Results for ER and TOPIC with and without LRrR on CUB200 are shown in Table Ta It shows how constraining different models to remember their initial evidence can lead to better localization of the bird across learning new tasks.
However, PG performance does not capture all of our desired properties for a continual learner. Ideally, we not only want a model to predict the object correctly if it is looking at the right evidence, but also we want it to not predict an object if it is not able to find the right evidence for it. To measure how close our baselines can get to this ideal model when they are combined with CRRR, we measure the precision as 1/(pp+rr, and recall as fn). We evaluate these metrics learning each task, denoted as Pro,; and Reis. respectively, and again at the end of the learning process of final task 7' denoted as Prt, and Ret, where the first subscript refers to the model ID and the second subscript is the test dataset ID on which the model is evaluated. The higher the precision for a model is, the less often it has made the right decision without looking at the right evidence. On the other hand, the higher the recall, the less often it makes a wrong decision despite looking the correct evidence. We show our evaluation on these metrics in Table [b] for ER and TOPIC with and without LRRR on CUB200 where LRRR increases both precision and recall across all methods, demonstrating that our approach continually makes better predictions because finds the right evidence for its decisions.
immediately after
In our final analysis, we would like to visualize the evolution of saliency maps across learning a sequence of tasks. Figure illustrates Jillustaae the evolution of saliency maps for an image from the test-set of the second task, which both ER without LRRR (top row) and with CRRR (bottom row) have seen during training for the second task. We only visualize the generated saliencies after finishing tasks #2, #5, #7, #9, and #11 for simplicity. We indicate the correctness of the prediction made by each model with 'correct' 'incorrect" written top of their corresponding saliency map. Our goal is
7

---

Published as a conference paper at ICLR 2021
Comrec
Cornec
Incernect
Incornet
Incernet
Connect
HRR
Figure 4: An illustration of the progression of saliencies on an image from the test set of the second task, evaluated after the model is trained on tasks #2, #5, #7, #9, and #11 on CUB200. Failure case for ER w.o. CRRR (top row), where saliency drifts from the original and the prediction becomes incorrect. ER+RRR (bottom row) retains close to the original saliency the model trains more tasks, with the exception of Task #5 which is able to correct later on. Its performance is retained as well, for saliencies that close to the original.
to visualize if adding the loss term LRRR prevents the drifting of explanations. Given the same input image, the ER without Leer model makes incorrect prediction after being continually trained for 11 tasks while never recovering from its mistake. On the other hand, when it is combined with CRRR. it is able to recover from an early mistake after task Considering the saliency map obtained after finishing task one as reference evidence, we can see that ER's s evidence drifts further from the reference. On the bottom row, the region of focus of ER+RRR remains nearly identical to its initial evidence, apart from one incorrect prediction. As applying LRRR corrects its saliency back to the original, this prediction corrected as well. This supports the conclusion that retaining the original saliency is important for retaining the original accuracy.
6 RELATED WORK
Continual learning: Past work in CL has generally made use of either memory. model structure, or regularization to prevent catastrophic forgetting. Memory-based methods store some form of past experience into a replay buffer. However, the definition of "experience" varies between meth- ods. Rehearsal-based methods use episodic memories as raw samples [Robins] 1995 Rebuffi et al.)
2017 Riemer et al. 2018) or their gradients (Lopez-Paz al. 2017, Chaudhry al. 2019) for the model to revisit. Incremental Classifier and Representation Learning (iCaRL) (Rebuffi et al. 2017). is a class- inaassiaccemettll learner that uses nearest-exemplar algorithm for classification and prevents catastrophic forgetting by using episodic memory. iTAML (Rajasegaran et al.) 2020) is a task-agnostic meta-learning algorithm that uses a momentum based strategy for meta-update and in addition to the object classification task, it predicts task labels during inference. An end-to-end in- cremental learning framework (EEIL) (Castro et al. 2018)) also uses an exemplar set along with data augmentation and balanced fine-tuning to alleviate the imbalance between the old and new classes. Bias Correction Method (BiC) (Wu et al. 2019) is another class-incremental learning algorithm for large datasets in which linear model is used to correct bias towards classes using fully con- nected layer. In contrast, pseudo-rehearsal methods generate the replay samples using generative model such an autoencoder (Kemker & Kanan (017)) GAN (Kamra al. 2017 Shin al.) 2017]. Regalarization-bussel methods define different metrics to measure importance and limit the changes on parameters accordingly (Kirkpatrick et al. 2017 Zenke et al., 2017 Ebrahimi et al., 2020a Serra et al.) 2018 Li & Hoiem, [2066] Dhar et al. 2019) but in general these methods have limited capacity. Structure-basse
methods control which portions of model are responsible for spe- cific tasks such that the model increases its capacity in controlled fashion as more tasks are added.
Inference for different tasks can be restricted to various neurons (Fernando et al. 2017 Yoon et al.) 2018), columns (Rusu et al. 2016). task-specific modules (Ebrahimi al. 2020b), or parameters selected by a mask (Mallya & Lazebnik 2018;; Serra et al. 2018). In RRR we explored the addi- tion of explanations to replay buffer and showed that saliency- based explanations offer performance upgrade as well as improvement in explanations across all memory-based and regularization-be based baselines we tried.
8

---

Published as a conference paper at ICLR 2021
Visual explanation approaches saliency methods attempt to produce posterior explanation or pseudo-proo prouuddliobbbiittyy map for the detected signals from the target object in the input image. These ap- proaches can be broadly divided into three categories including activation, gradient, and perturbation based methods. Activation-based methods (Zhou et al.) 2016, Selvaraju et al.) 2017] Chattopadhay al. 2018) use weighted linear combination of feature maps whereas gradient-based methods (Bachrens et al. 2010 Sundararajan et al., 2017aa Springenberg et al., 2014:) Shrikumar et al. [0177] Zhang et al., 2018) use the derivative of outputs w.r.t the input image compute pixel-wise impor- lance scores to generate attention maps. Methods in these categories are only applicable to differ- entiable models, including convolutional neural networks (CNNs). In contrast, perturbation- -baed methods are model-agnostic and produce saliency maps by observing the change in prediction when the input perturbed (Petsiuk et al., 2018 Ribeiro et al. 2016 Ross et al. 2017 Zhou et al. 2014 Seo et al. 2018). While these methods attempt to identify if models are right for the wrong reason, Ross et al. (2017) ) took a step further and applied penalties to correct the explanations provided in supervised//usupervvis
fashion during training. [ScIvaraju et al. (2019) used human explanations in the form of question and answering to bring model explanations closer to human answers.
7 CONCLUSION
In this paper, we proposed the use of model explanations with continual learning algorithms to en- hance better knowledge transfer as well as better recall of the previous tasks. The intuition behind method is that encouraging model to remember its evidence will increase the generalisability and rationality of recalled predictions and help retrieving the relevant aspects of each task. We advo- cate for the use of explainable Al tool improve model performance, rather than an artifact or interpretation of the model itself. We demonstrate that models which incorporate a "remember for the right reasons" constraint as part of a continual learning process can both be interpretable and more accurate. We empirically demonstrated the effectiveness of our approach in a variety of settings and provided an analysis of improved performance and explainability.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp. 9505-9515, 2018
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert MÅÄller. How to explain individual classification decisions. Journal of Machine Learning Research, 1 11JJu))180331100 1831, 2010.
Francisco M Castro, Manuel J Marín Jamén,Jiménez, Nicolás Guil, Cordelia Schmid, and Karteck Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp.
2018.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 839-847. IEEE, 2018.
Arslan Chaudhry, Marc' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference on Learning Representations, 2019.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuar Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5138-5146. 2019.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty-guide continual learning with bayesian neural networks. In International Conference on Learning Rep- resentations, 2020a.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adver- sarial continual learning. In European Conference on Computer Vision (ECCV), 2020b.

---

Published as a conference paper at ICLR 2021
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08334 2017.
Bernd Fritzke et al. A growing neural gas network learns topologies. Advances in neural information processing systems, 7:625-632, 1995.
Tyler L Hayes, Nathan Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In 20/9 International Conference Robotics and Automation (ICRA), pp- 9769-9777. IEEE, 2019.
Kevin Jarrett, Koray Kavukcuoglu, Mare' Aurelio Ranzato, and Yann LeCun. What the best multi-stage architecture for object recognition? In 2009 IEEE 12th international conference computer vision, pp. 2146-2153. IEEE, 2009.
Nitin Kamra, Umang Gupta, and Yan Liu. Deep generative dual memory network for continual learning. arXiv preprint arXiv: /710. 10368, 2017.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. arXiv preprint 1rXiv:1711.1055: 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barskk-kaa al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, pp. 201611835, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, 2009.
Zhizhong Li and Derek Hoiem. Learning without forgetting. In European Conference on Computer Vision, pp. 614-629. Springer, 2016.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv: 1908.0326. 2019.
David Lopez-Paz al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
Thomas Martinetz, Klaus Schulten, et al. A" neural-gas" network learns topologies. 1991.
James L McClelland, Bruce L McNaughton, and Randall C O'reelll. Why there complementary learning systems the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Detito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, W, 2017.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference (BMVC), 2018.
Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incremental task-agnostic meta-learning approach. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, 2017.
10

---

Published as a conference paper at ICLR 2021
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interfer- ence. arXiv preprint arXiv: 19/0,, 2018.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2): 123-146, 1995.
Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. Right for the right reasons: Train- ing differentiable models by constraining their explanations. arXiv preprint trXiv:/703.037/1 2017.
Andrei A Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04677 04677, 2016.
Ramprasaath R Selvaraju, Abhishek Das. Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint arXiv:1611.074533 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. Taking a hint: Leveraging explanations to make vision and language models more grounded. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2591-2600, 2019.
Dasom Seo, Kanghan Oh, and I1-Seok Oh. Regional multi-scale approach for visually pleasing 2018
explanations of deep neural networks. arXiv preprint
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Jennifer Dy and Andreas Krause (eds.), Proceedings the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pp. 4548 4557. PMLR, 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713. 2016.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Ma- chine Learning- Volume 70, pp. 3145-3153. JMLR. org. 2017.
Daniel L Silver, Qiang Yang, and Lianghao Li. Lifelong machine learning systems: Beyond learning algorithms. In 2013 AAAI spring symposium series, 2013.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:/312.6034. 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Vićgas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: 070.770000002..5 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 14/2,68066 2014.

---

Published as a conference paper at ICLR 2021
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3319- 3328. JMLR. org, 2017a.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. arXiv preprint 1eXiv:/703.01380 2017b.
Xiaoyu Tao, Xiaopeng Hong. Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few- shot inteemmaaltttlll learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12183-12192, 2020.
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems, 15(1-2):2546, 1995.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD fards-200-200 Dataset. Technical Report CNS-TR-2011 1-001 California Institute of Technology, 2011.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 372, 2019.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference Learning Representations, 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3987-3995. PMLR, 2017.
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126 110):1084-1102, 2018
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. arXiv preprint 14 2014.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2921-2929, 2016.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 GRAD-CAM TARGET LAYERS
Table 2 shows the target layer names used in Grad-CAM for different network architectures accord- ing to their standard PyTorch (Paszke et al. 2017) implementations. Saliency map size is equal to the activation map of the target layers.
Table 2: Target layer names and activation maps size for saliencies generated by different network architectures in Grad-CAM.
Target layer name in PyTorch torchvision package
Saliency map size
SqueezeNetl_1 AlexNet ResNet18
features. 0 12 expand3x3
13 x 13
features 10 features 7. conv2
13
7
B POINTING GAME VISUALIZATION
Figure shows an example from CUB200 in the Pointing Game. We used the segmentation anno- tation to verify whether the peak of the predicted saliency map (marked with red cross) falls inside the object region. It is regarded as hit as the red cross is inside the segmentation mask for the bird.
Figure 5: An example of PG evaluation hit for image in CUB200. Left: image saliency map overlaid on the image. Right: the segmentation label where the red marks the peak saliency.
C TABULAR RESULTS
In this section, we have tabulated results shown in Figure 2 and Figure 3 with means and standard deviations averaged over 3 runs.
Table 3: Classification accuracy of few-shot CIL learning of CUB200 the end of 11 tasks for ER with and without CRRr using different backbone architectures and saliency map techniques. Results averaged over 3 runs. Figure (left) in the main paper is generated using numbers in this Table.
18
RN CCa
07
53.5
= 07
45. 006
39.6
= 07
35.
32.
29.4
209
25.9
08
25
2
06
26.3
2
23.6
1
RN ER
87.8
29.7
09
11.7
108
35.8
07
314
19
28.5
00
25.5
108
22.1
00
21.
108
22.5
11
11
19.8
1.9
RNIS-RRR RiS-etR--mmtthh
07.8008
50.
0.6
135
109
37.0
108
33
107
29.5
106
26.
108
23.
108
23.
108
23.
108
21.
105
RN1S-RRR-BP
50.8
RN15-Fineture
678+00
448
258
67
26+07
25.2 07
288
+66
168
188
17.1
07
220
21.
Alex-El ER
56.100
41.0077
11.3
07
38.7
0.
31.
107
245
0.7
22.
17
19,0
106
19.1
108
18.7
108
19.1 108
Alex-Finetune
70.7007
42.8
08
20.6
08
360
- 08
20.5
07
28.3
06
17.9
07
18.0
- 0.7
17.0
+ 0.5
16.0
RRR- GCam
88.203
36.2
30.1
06
28.3
104
25.1
105
23.4
105
19.3
106
19,0
06
18.5
0.5
18.4 0.5
18.2 182106
SQ ER
16.005
33.2
13.2.05
27.1
106
25.3
166
22.1
105
20.5
105
163
16.0
15.5
106
15.4
15.2 15.2
FO-
06.203
32.0
25.2
23.
0000000
202
08
1044
04
148404
04
138
052
13.
13

---

Published as a conference paper at ICLR 2021
Table Performance of the state-of-the- existing approaches with and without Lrrr on CUB200 including TOPIC (Tao et al. (020), EEIL (Castro et al. 2018), iCaRL (Rebuffi et al. 2017). Results for baselines are obtained using their original implementation. Results are averaged over 3 runs. Figure (right) in the main paper is generated using numbers in this Table.
10
11
EEIL
68.6 CC4
53.6 0.4
47.9 0.3
48.2
0.8
36.3
0.9
27
25.9
0.7
24.7 + 0.5
23.9 +.7
24.1 +
22.1 +0.
DEIL-RRR
68.6 1.04
56.6 0.5
50.9 0.6
16.3
0.5
30.7
1.2
31.4
0.7
28.5
1.2
25.0 0.6
26.5
27.1100 1.6
25.2 09
ICaKL
2.7
44.1
0.5
16.8
0.3
29.5
=
0.3
27
0.4
862 0.5
210
248 0.
2.7
iCIRL-RER
68.6+004
56.6
53.6 +.7
47.1
38.6
0.5
32.5
1055000
0.5
315+
0.4
20-2+006
27.0.005
278
21.++03
TOPRC
262211
29.20:
8222
208
1..7
0..7009
11.7 0.8
31.4
25.+
RRR
67.8.100
0.7
15.8
0.6
30.8
35.3
0.0
3233
11
29.4 0.9
25.9
139.00
08
25.7
0.06
26.3
26.
24.6 1.7.
68.6 0.44 62.410.4
57.2
0.4
52.
49.5
0.9
48.1
428 1.1
40.1
08
38.7
1.7
57.1
103
33.0 1.99
Table 5: Performance of the state-of-the-art existing approaches with and without Lrrr on CI- FAR100 in 10 tasks. Results for iTAML (Rajasegaran al. 2020), BiC (Wu al. 2019), and EEIL (Castro et al.] (2018) are produced with their original implementation while EWC (Kirkpatrick
al. 2017) LwF (Li & Hoiem 2016) are re-implemented by us. Results are averaged over Figure 3a in the main paper is generated using numbers in this Table.
runs.
3
5
8
10
ITAMI
92.3
1.2
87.5
84.1
0.8
43.9
0.3
79.6
79.7
88.9
80.0=00
BIC
90.3 0.7
0.7
70.1 0.4
698
1.2
65.3
61.3 0.3+108
57.4 0.7
64.9
582 0.
50.3 0
BIC+RRR
-0.
819111 1.1
76.1
69.3 0.3
65.1
6.4
58.7
55.4
52.1 36.8
0.
55.1
EEIL-RRR
0..0
53.5 0.3
1.2
74.0
0.3
15.1004
61220.
57.6
54.110.
5..7
LxF
86.1 1.2
09:0 0.7
M.0
45.
0.5
36.7 0.9
30.8 0.7
28.6
0.7
24.2
LRF-RRR
86.1
72.4
1.
4.3
0.8
39.3
34.1000
12.1
27.1 29.8
0.4
31.
26.
21.7
18.
16.6
EWC+RRR
86.1
1.
56.0 0.4
1.2
H.4009
0.5
2.
25.1 11
23.
188
19.0 1.2
ER
86.1
74.5
66.2
0.8
625
06.7 0.7
50.
47.6=04
03
4.
38.1
1.1
Table 7: Performance of the state-of-the-ar existing approaches with and without LRRR Ima-
geNet100 in 10 tasks. Results for iTAMLL. (Rajasegaran et al.] (020), BiC (Wu et al.) 2209)). and Castro al. 2018) are produced with their original implementation while EWC (Kirkpatrick 2017) and LwF (Li & Hoiem 2016) are re-implemented by us. Results are averaged over runs. Figure the main paper is generated using numbers in this Table.
EEIL
et al.
ITAML
98.4 0.8
96.4
94.4
93.0
92.4
1.2
D.3
80.9
90.3
90.3
89.8 0.4
STAML-RRR
99.4
97.3 17.3100
98.6 0.7
96.3
95.3 2 0.5
93.1 0.5
92.1
0.6
92.1 12.1100 0.6
929 0.9
9.
EEIL.
99.5
0.9
83.0
86.7 +
$0.=
78.2
75.4
BIC
98.3 0.7
94.9 0.8
93.5+
90.+1.2 1.2
89.0 1.2
87.3
066
85.2 0.7
83.2 0.4
82.5 0.9
81.1 11.1+111
BICARRR
0.7
86.9 0.3
96.5
03
0.7
89.3
00.3111 1.1
S722
67.2 1111
8.
81.1 SL120.0
1.1
(CaRLARRR
0..
97.9112
0.7
0.7
91.7
0.9
85.7111
82.1
7.4109
7.2
68.1
LwF
98.3=055
0.99
13
0.
518100
50.1 1 0.6
0..
0.3
LwF+RRR
0.5
87.1112
0.6
19.1
69.1
19.4111
57.2
1.
41. 45.1
02.6003
13.9
123
ER
99.3 0.5
95.2
0..2
2.8
78.1109
0.6
00.1 0.8
6T.1
6.8
55.1
50.1 1.1
RRR
98.3 - 0.5
96-5+
03.4 0.
84.8 0.
7..7+0.4
+
73.1 -
084+08
00
55.1 0
14

---

Published as a conference paper at ICLR 2021
Table 6: Performance of the state-of-the-art existing approaches with and without CRRR on CI- FARI00 in 20 tasks. Results for iTAML (Rajasegaran et al. (020), BiC (Wu et al. 2019). and EEIL CCstro al. 2018) are produced with their original implementation while EWC (Kirkpatrick et al. 2017) and LwF (Li & Hoiem (016)) are re- immmleeeteed by us. Results are averaged over 3 runs.
Figure 3b in the main paper generated using numbers this Table.
(a) Tasks 1-10
ITAML
84.7=0.0
85.7 + 0.4
86.5 2 0.3
86.5 0.8
86.3
1.2
85.7 1 0.8
$1.9 =
82.6 1:6-200 0.3
80.8
0.7
82.4
iTAML-RRR
84.7 0.6 0.6
89.9
0.5
89.2
89.2
0.6
89.0 1.1
87.2 0.6
88.0
85.6
86.6
0.3
90.3
BIC+RRR
25.7=0.8
93.3
1.1
77.5
0.9
0.6
74.8 0.6
0..6
67
65.7 0.5
64.9
FRIL
81.9-05
8633 0.3
84.9 0.4
80.7 0.3
71.7 0.66
74.9 14.++03
70.9-01 0.7
67.4
64.9 0.5
62.4
0.9
50.5
0.
37.5+
33.7
304+09
268
24.9
L*F-RRR
$5.1=077
7161
618
0.5
0.92
00.7
36.7 1.2
01410.4
29.
27.2 1.2
EWC
-5.1=07
613103
0.
0.3
31.
23.7
2.4
20.8 0.8
18.9
ER
85.1 0.7
83.1
0.9
81.8
0.7
74.9
0.3
70.4
61.3
1.2
80.
57.0
54.3
0.4
48.2 0.8
RRR
85.1 0.7
$5.1
0.9
83.8
0.4
0.4
77.9 2 0.4
72.4 1.2
64.5
0.7
82.8 0.7
59.0
57.3
0.8
512
(b) 11-20
11
13
13
14
15
16
17
18
19
20
ITAML
80.0
80.6 + 0.5
74.3 0.8
70.7 +
71.3 + 1.1
68.3 4 0.5
70.3
68.3 0.6
69.5
06.0 + 0.6
IYAMLARRR
85.2
70.7
74.3
74.0
73-4 1.1
74.8
7
73.9
71.8
BIC
58.2 0.4
57.0
0.6
M6.1
+ 1.2
55.7 0.6
13.8+0.5
52.4
1.2
48.7
0.6
49.2
47.7
1.1
46.7 1.2
BICHRRR
0.5
59.1
58.2 0.5
878
51.111
56.6 0.00::0
53.9
52.4
49.5
0.9
EEIL-RRR
63.7=00 0.6
63.91 0.4
0.4
0.3
5..6
53.5 0.4
58.5
52.7
49.1
17.5
LxFF
2.
2.7
20.0
19.1
0.8
17.1100
15.6= 0.8
14.7
1.0
13.7 L.1
LwF-RRR
0.7
26.9
20.77..7
24.5
2..6
22.
19.5
1..
19.7 0.8
18.4
EWC+RRR
20.-=03
19
184 0.7
05
16
158 0.5
15.0-077
16.6
14.3+0.4
1.3
FR
45.8 0.6
42.7++077
4..6+00.6
41.2+06
3..
0.6
338 518-04 0.4
32.4+122
31.4+0.7
312
15


Reviews:
[{"review_id": "YuJk6kDCLIz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\n\nThis paper tackles the problem of catastrophic forgetting in a continual learning scenario, in which the same classifier is trained incrementally on new classification tasks, each defined on a new set of output classes, and asked to retain performance on all the previous tasks. To tackle this problem, a stream of approaches keep a minimal \"replay buffer\" of examples and their labels from each task, that can be accessed during sequential task learning. This paper belongs to this stream of approaches. A naive way of using the replay buffer is to mix the buffer to the current task training set. Additionally, this paper proposes to store gradient attribution maps in the replay buffer and use the stored gradient attribution maps as additional targets for the current model (RRR loss): the model should have \"explanations\" of previous examples that roughly don't change upon seeing new evidence from other classes. The authors show that the proposed RRR loss can be applied to multiple baselines and improves performance on a continual learning and few-shot continual learning scenario.\n\nThe main contribution of the paper is to use and store gradient attribution maps along with the examples and use the attribution maps as additional targets for remembering previous tasks. This goes into the direction of extracting \"summaries\" of previous models that can be re-used to inform the current model and avoid forgetting. The paper is well-written and clear. I particularly like the simplicity of approach of the paper and I think the results support its effectiveness. However, in the current form, I cannot quite conclude that it's the \"explanations\" that are really crucial for this to work and not potentially other statistics that can be extracted from the model. Also, it's unclear what are the negative effects of the proposed approach for backward transfer. About this, some questions / suggestion below.\n\nPros\n------\n- Approach is simple, quite general and improves on a variety of baselines\n\nCons\n-------\n- Is it really the \"explanation\" that is important ? A naive baseline is missing (details afterwards)\n- Conceptually, the limits of this approach for backward transfer could be discussed more extensively\n- Discussion on / references to competing approaches based on feature distillation are not present\n\n\nRemarks\n------------\n\n- This approach is similar in spirit to (https://arxiv.org/pdf/1910.10986.pdf, to be cited) where the authors try to align feature maps between past models and the current model. The general idea is still to \"summarize\" the previous model but only use feature maps instead of gradient attribution maps. I think one of the baseline missing in this paper is to use the feature maps themselves as targets for the distillation, would that be feasible ? How would that compare to distilling gradient attribution maps instead ? I feel the result can be stronger / more interesting if the authors include this baseline.\n\n- It seems to me that distilling attributions of the previously trained models could limit backward-transfer, as the previous attributions acts like a sort of prior on the new information that can be learned. The prior seems \"strong\" as it doesn't change with the number of tasks: the model predicts the same attributions even if new information could indicate that the original \"explanations\" were not correct. Do you think backward transfer can be hurt by this ? Instead, could you maybe try to compute the XAI for *all* the examples in the buffer with the model estimated after task t, instead of computing only the XAI for current examples ? That would boil down to \"filtering\" the prior p(XAI_0) of attributions for task 0 with the posterior p(XAI_0 | 1, ..., t).\n\n- I found quite hard to read Figure 2 and Figure 3. I'd suggest the authors to put in a Table the ablations results with respect to the different XAI methods and some of the most important differences in performance for Figure 3.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "X_jV_eXt8oY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for continual learning of a sequence of supervised tasks which is based on memory-replay for remembering evidence from previously made decisions. This evidence relies on explanations of those decisions rather than data. These explanations are incorporated directly into the loss function. The proposed approach is tested against several counterparts using benchmark datasets in continual learning. \n\nI would like to remark as strengths of the paper:\n-\tThe method adopts a novel approach of incorporating explanations along with data for \n-\tThe experiments provide clear insights onto which kinds of explanations would work better, by exploring several strategies like vanilla backpropagation, Grad-CAM, etc. \n-\tThe paper provides a thoughtful analysis of the impact of explanations that demonstrates some advantages in terms of accuracy and backward transfer.\n\nSome weaknesses of the paper are: \n-\tAs the examples and explanations to remember are selected randomly in the experiments, it is perhaps a good idea to perform this process several times to guarantee reliability of these results. Furthermore, does the number of examples and explanations depend on the dataset size, i.e. is there a relation between the number of training examples, the memory replay examples and the performance of the proposed method? \n\nOther comments: \n-\tI suggest to improve clarity of the paper by improving some of the language used, e.g. in page 4 it’s said that “We show below that combining RRR into the objective function of state-of-the-art memory and regularization-based methods results in significant performance improvements.”. In this case, “significant” is clearly not the word to be used as the paper does not include statistical significance tests supporting this claim!\n\nQuestions for authors: \n- Please address questions in the comments above.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HDWvEjMRKi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThis paper delivers an additional training objective for continual learning which regularizes the update of the model parameters by making them not deviate a lot from its temporal snapshots of the visual saliency map. The additional information, *reason*, that the regularization exploits helps to solve the problem of catastrophic forgetting in addition to providing a qualitative explanation.\n\n**Strong points**\n\n- The method, RRR, is simple to implement for most visual continual learning task setting, yet to yield a significant performance boost.\n- The evaluation verifies its efficacy throughout many state-of-the-art continual learning methods.\n- As a byproduct, the method provides a qualitative explanation map, which in most cases can only be acquired by an additional posthoc training process.\n\n**Weak points**\n\n- The method needs input to be spatially meaningful. Although most of the continual learning tasks currently deal only with the visual domain, in general, the constraints they have are only a condition that is a classification problem. Therefore the method presented in this paper will bring additional constraints to the input and can not be directly applicable when the input is not spatially meaningful (e.g., set classification).\n\n**Recommendation**\n\nI vote for acceptance as the paper brings a notable performance boost in continual learning with a simple yet meaningful objective.\n\n**Supporting arguments for the recommendation**\n\nIn addition to the performance improvements described above, the method presented in this paper is expected to be of great help to the XAI community as well. Considering the impact of this paper on the XAI community that saliency maps and other explanation techniques (although they have to be differentiable w.r.t. parameter) can lead to such direct quantitative performance improvement beyond just XAI use, I think it is sufficient to accept this paper.\n\n**Additional feedbacks**\n\nAs saliency maps are densities with fixed support (fixed image size), I think OT distances can be employed in place of L1 loss. Using L1 loss makes the model hard to discriminate the case when the prediction center of mass is far away from the target center versus the case when the prediction center is still wrong but close to the target center. For example, if target saliency is focused on the wings of a bird image, then predictions that one focusing on the bird's torso (which is very close to its wings) and the other focusing on the image corner equally penalized by L1 loss. Optimal transport distances can handle these cases and I think its employment can boost performance.\n\n**Typos**\n\n- The last paragraph of section 2 seems incomplete (~~~ pass what? pass upstream gradient?)\n- $S$* is missing in the equation 2. maybe $\\hat{s}$?", "extra_scores": {}, "extra_text": {}}, {"review_id": "1nH7TkgSy91", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed technique is simple. It helps existing replay methods to boost their accuracy for class-incremental learning. In addition to save some training examples of previous tasks or classes, it also saves the saliency maps of these examples as explanations of classification. The size the saliency maps is small and requires very little memory. For continual learning, it includes additional term in the loss function to take care of the saved saliency maps. Experimental results show that this additional saved information can help improve existing replay methods for class-incremental learning. \n\nCan you give some reasons why the proposed method works? Replaying previous examples may help continual learning in two ways, preserving the previous parameters and/or correcting previous parameters/features if they are insufficient to discriminate old classes in old tasks and the new classes in the new task. The saved saliency maps mainly play/enhance the first role, but that may conflict with the second role? If that is true, the saliency maps may help the continual learning performance in some cases but harm it in some other cases? Can you try CIFAR10 and MNIST with 2 classes in each task? When you have fewer classes in a task, the second role may be more important. Also, since you have tried CIFAR100 with 10 and 20 tasks, can you also try ImageNet100 with 20 tasks? \n\nSince the memory required by the Grad-CAM saliency maps are small, it is possible to save more of them without using any saved training images? \n\nCan your technique help those methods that don’t use replay? \n\nI thought M^{RRR} is the model explanation memory. What is xai buffer memory M^{xai}? Are they the same?\n\nI prefer tables to line plots when showing results as it is hard to see how much the improvement is in line plots. The three sub-figures are also too small to see. \n\nI have difficulty to parse the first sentence in the last paragraph of section 2. Anything wrong there?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper touches upon the problem of catastrophic forgetting in continual learning. The idea is to enhance experience reply by explanations of the decision/predictions made. Technically, this "Remembering for the Right Reasons" loss adds an explanation loss to continual learning. This is an interesting idea as also the reviewers agree on. I would like to encourage the authors to have consider a different abbreviation. RRR also stand for "Right for the Right Reasons" loss due to Ross et al.; the authors should use a different abbreviation and also mention the work of Ross et al. (Andrew Slavin Ross, Michael C. Hughes, Finale Doshi-Velez: Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations. IJCAI 2017: 2662-2670). Moreover, it might actually be interesting in moving towards interactive learning here as well, because continual learning may also suffer from confounders. Moreover, there is also a connection to HINT (Ramprasaath Ramasamy Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry P. Heck, Dhruv Batra, Devi Parikh: Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded. ICCV 2019: 2591-2600) as it also aims at keeping explanations close to each other. Indeed, they use a ranking loss and do not consider continual learning. Overall, a simple method that is shown empirically to help improving existing replay methods for class-incremental learning.
Decision: Accept (Poster)