Overcoming Catastrophic Forgetting with Hard Attention to the Task

Joan Serrà 1 Dı́dac Surı́s 1 2 Marius Miron 1 3 Alexandros Karatzoglou 1

arXiv:1801.01423v2 [cs.LG] 14 Feb 2018

Abstract
Catastrophic forgetting occurs when a neural network loses the information learned in a previous
task after training on subsequent tasks. This problem remains a hurdle for artificial intelligence
systems with sequential learning capabilities. In
this paper, we propose a task-based hard attention
mechanism that preserves previous tasks’ information without affecting the current task’s learning.
A hard attention mask is learned concurrently to
every task, through stochastic gradient descent,
and previous masks are exploited to condition
such learning. We show that the proposed mechanism is effective for reducing catastrophic forgetting, cutting current rates by 45 to 80%. We also
show that it is robust to different hyperparameter
choices, and that it offers a number of monitoring
capabilities. The approach features the possibility
to control both the stability and compactness of
the learned knowledge, which we believe makes
it also attractive for online learning or network
compression applications.

1. Introduction
With the renewed interest in neural networks, old problems
re-emerge, specially if the solution is still open. That is
the case with the so-called catastrophic forgetting or catastrophic interference problem (McCloskey & Cohen, 1989;
Ratcliff, 1990). In essence, catastrophic forgetting corresponds to the tendency of a neural network to forget what
it learned upon learning from new or different information.
For instance, when a network is first trained to convergence
on one task, and then trained on a second task, it forgets
how to perform the first task.
Overcoming catastrophic forgetting is an important step
in the advancement towards more general artificial intelligence systems (Legg & Hutter, 2007). Such systems
1
Telefónica Research, Barcelona, Spain 2 Universitat
Politècnica de Catalunya, Barcelona, Spain 3 Universitat Pompeu Fabra, Barcelona, Spain. Correspondence to: Joan Serrà
<joan.serra@telefonica.com>.

Ongoing work. Preprint.

should be able to seamlessly remember different tasks, and
to learn them sequentially, following a lifelong learning
paradigm (Thrun & Mitchell, 1995). Apart from being more
biologically plausible (Clegg et al., 1998), there are many
practical situations which require a sequential learning system (cf. Thrun & Mitchell, 1995). For instance, it may be
unattainable for a robot to retrain from scratch its underlying model upon encountering a new object/task. After
accumulating a large number of objects/tasks and their corresponding information, performing concurrent or multitask
learning at scale may be too costly.
Storing previous information and using it to retrain the
model was among the earliest attempts to overcome catastrophic forgetting; a strategy named “rehearsal” (Robins,
1995). The use of memory modules in this context has
been a subject of research until today (Rebuffi et al., 2017;
Lopez-Paz & Ranzato, 2017). However, due to efficiency
and capacity constrains, memory-free approaches were also
introduced, starting with what was termed as “pseudorehearsal” (Robins, 1995). This approach has found some
success in transfer learning situations where one needs to
maintain a certain accuracy on the source task after learning
the target task (Jung et al., 2016; Li & Hoiem, 2017). Within
the pseudo-rehearsal category, we could also consider recent
approaches that substitute the memory module by a generative network (Venkatesan et al., 2017; Shin et al., 2017;
Nguyen et al., 2017). Besides the difficulty of training a
generative network for a sequence of tasks or certain types
of data, both rehearsal and pseudo-rehearsal approaches
imply some form of concurrent learning, that is, having to
re-process ‘old’ instances for learning a new task.
The other popular strategy to overcome catastrophic forgetting is to reduce representational overlap (French, 1991).
This can be done at the output, intermediate, and also input
levels (Gutsein & Stump, 2015; He & Jaeger, 2018). A
clean way of doing that in a soft manner is through so-called
“structural regularization” (Zenke et al., 2017), either present
in the loss function (Kirkpatrick et al., 2017; Zenke et al.,
2017) or at a separate merging step (Lee et al., 2017). With
these strategies, one seeks to prevent major changes in the
weights that were important for previous tasks. Dedicating
specific sub-parts of the network for each task is another
way of reducing representational overlap (Rusu et al., 2016;
Fernando et al., 2017; Yoon et al., 2018). The main trade-

Overcoming Catastrophic Forgetting with Hard Attention to the Task

off in representational overlap is to effectively distribute
the capacity of the network across tasks while maintaining
important weights and reusing previous knowledge.
In this paper, we propose a task-based hard attention mechanism that maintains the information from previous tasks
without affecting the learning of a new task. Concurrently
to learning a task, we also learn almost-binary attention
vectors through gated task embeddings, using backpropagation and minibatch stochastic gradient descent (SGD). The
attention vectors of previous tasks are used to define a mask
and constrain the updates of the network’s weights on current tasks. Since masks are almost binary, a portion of the
weights remains static while the rest adapt to the new task.
We call our approach hard attention to the task (HAT). We
evaluate HAT in the context of image classification, using
what we believe is a high-standard evaluation protocol: we
consider random sequences of 8 publicly-available data sets
representing different tasks, and compare with a dozen of
recent competitive approaches. We show favorable results in
4 different experimental setups, cutting current rates by 45
to 80%. We also show robustness with respect to hyperparameters and illustrate a number of monitoring capabilities.
We make the code for all experiments publicly-available1 .

2. Putting Hard Attention to the Task
2.1. Motivation
The primary observation that drives the proposed approach
is that the task definition or, more pragmatically, its identifier, is crucial for the operation of the network. Consider
the task of discriminating between bird and dog images.
When training the network to do so, it may learn some
set of intermediate features. If the second task is to discriminate between brown and black animals using the same
data (assuming it only contained birds and dogs that were
either brown or black), the network may learn a new set
of features, some of them with not much overlap with the
first ones. Thus, if training data is the same in both tasks,
one important difference should be the task description or
identifier. Our intention is to learn to use the task identifier
to condition every layer, and to later exploit this learned
conditioning to prevent forgetting previous tasks.
2.2. Architecture
To condition to the current task t, we employ a layer-wise
attention mechanism (Fig. 1, top). Given the output of the
units2 of layer l, hl , we element-wise multiply h0l = atl hl .
However, an important difference with common attention
1

Code is attached in a ZIP file as Supplementary Material.
In the remaining of the paper, we will use ‘units’ to refer to
both linear units (or fully-connected neurons) and convolutional
filters. HAT can be extended to any parametric layer.
2

Figure 1. Schematic diagram of the proposed approach: forward
(top) and backward (bottom) passes.

mechanisms is that, instead of forming a probability distribution, atl is a gated version of a task embedding vector etl ,

atl = σ setl ,

(1)

where σ(x) ∈ [0, 1] is a gate function and s is a positive
scaling parameter. We use a sigmoid gate in our experiments,
but note that other gating mechanisms could be used. All
layers l = 1, . . . L − 1 operate equally except the last one,
layer L, where atL is binary hard-coded. The operation
of layer L is equivalent to a multi-head output (Bakker &
Heskes, 2003), which is routinely employed in the context
of catastrophic forgetting (for example Rusu et al., 2016; Li
& Hoiem, 2017; Nguyen et al., 2017).
The idea behind the gating mechanism of Eq. 1 is to form
hard, possibly binary attention masks which, act as “inhibitory synapses” (McCulloch & Pitts, 1943), and can thus
activate or deactivate the output of the units of every layer.
In this way, and similar to PathNet (Fernando et al., 2017),
we dynamically create and destroy paths across layers that
can be later preserved when learning a new task. However,
unlike PathNet, the paths in HAT are not based on modules,
but on single units. Therefore, we do not need to pre-assign
a module size nor to set a maximum number of modules
per task. Given some network architecture, HAT learns
and automatically dimensions individual-unit paths, which
ultimately affect individual layer weights. Furthermore, instead of learning paths in a separate stage using genetic
algorithms, HAT learns them together with the rest of the
network, using backpropagation and SGD.

Overcoming Catastrophic Forgetting with Hard Attention to the Task

2.3. Network Training
To preserve the information learned in previous tasks upon
learning a new task, we condition the gradients according
to the cumulative attention from all the previous tasks. To
obtain a cumulative attention vector, after learning task t
and obtaining atl , we recursively compute


t ≤t−1
a≤t
,
l = max al , al
using element-wise maximum and the all-zero vector for
a≤0
l . This preserves the attention values for units that were
important for previous tasks, allowing them to condition the
training of future tasks.
To condition the training of task t + 1, we modify the gradient gl,ij at layer l with the reverse of the minimum of the
cumulative attention in the current and previous layers:
h

i
≤t
0
gl,ij
= 1 − min a≤t
,
a
gl,ij ,
(2)
l,i
l−1,j
where the unit indices i and j correspond to the output
(l) and input (l − 1) layers, respectively. In other words,
≤t
we expand the vectors a≤t
l and al−1 to match the dimensions of the gradient tensor of layer l, and then perform
a element-wise minimum, subtraction, and multiplication
(Fig. 1, bottom). We do not compute any attention over the
input data if this consists of complex signals like images or
audio. However, in the case such data consisted of separate
or independent features, one could also consider them as the
output of some layer and apply the same methodology.
Note that, with Eq. 2, we create masks to prevent large updates to the weights that were important for previous tasks.
This is similar to the approach of PackNet (Mallya & Lazebnik, 2017), which was made public during the development
of HAT. In PackNet, after an heuristic selection and retraining, a binary mask is found and later applied to freeze the
corresponding network weights. In this regard, HAT differs
from PackNet in three important aspects. Firstly, our mask
is unit-based, with weight-based masks automatically derived from those. Therefore, HAT also stores and maintains
a lightweight structure. Secondly, our mask is learned, instead of heuristically- or rule-driven. Therefore, HAT does
not need to pre-assign compression ratios nor to determine
parameter importance through a post-training step. Thirdly,
our mask is not necessarily binary, allowing intermediate
values between 0 and 1. This can be useful if we want to
reuse weights for learning other tasks, at the expense of
some forgetting, or we want to work in a more online mode,
forgetting the oldest tasks to remember new ones.
2.4. Hard Attention Training
To obtain a totally binary attention vector atl , one could use
a unit step function as gate. However, since we want to

train the embeddings etl with backpropagation (Fig. 1), we
prefer a differentiable function. To construct a pseudo-step
function that allows the gradient to flow, we use a sigmoid
with a positive scaling parameter s (Eq. 1). This scaling
is introduced to control the polarization, or ‘hardness’, of
the pseudo-step function and the resulting output atl . Our
strategy is to anneal s during training, inducing a gradient
flow, and set s = smax during testing, using smax  1
such that Eq. 1 approximates a unit step function. Notice
that when s → ∞ we get atl,i → {0, 1}, and that when
s → 0 we get atl,i → 1/2. We will use the latter to start a
training epoch with all network units being equally active,
and progressively polarize them within the epoch.
During a training epoch, we incrementally linearly anneal
the value of s by


1
1
b−1
s=
+ smax −
,
(3)
smax
smax B − 1
where b = 1, . . . B is the batch index and B is the total
number of batches in an epoch. The hyperparameter smax ≥
1 controls the stability of the learned tasks or, in other words
the plasticity of the network’s units. If smax is close to
1, the gating mechanism operates like a regular sigmoid
function, without particularly enforcing the binarization of
atl . This provides plasticity to the units, with the model being
able to forget previous tasks at the backpropagation stage
(Sec. 2.3). If, alternatively, smax is a larger number, the
gating mechanism starts operating as a unit step function.
This provides stability with regard to previously learned
tasks, preventing changes in the corresponding weights at
the backpropagation stage.
2.5. Embedding Gradient Compensation
In preliminary analysis, we empirically observed that embeddings etl were not changing much, and that the magnitude
of the gradient was weak on those weights. After some investigation, we realized that the major part of the problem
was due to the introduced annealing scheme (Eq. 3). To
illustrate the effect of the annealing scheme on the gradients
of etl , consider a uniformly distributed embedding etl,i across
the active range of a standard sigmoid, etl,i ∈ [−6, 6]. If we
do not perform any annealing and set s = 1, we obtain a cumulative gradient after one epoch that has a bell-like shape
and spans the whole sigmoid range (Fig. 2). Contrastingly, if
we set s = smax , we obtain a much larger magnitude, but in
a much lower range (etl,i ∈ [−1, 1] in Fig. 2). The annealed
version of s yields a distribution in-between, with a lower
range than s = 1 and a lower magnitude than s = smax .
A desirable situation would be to have a wide range, ideally spanning the range of s = 1, and a large cumulative
magnitude, ideally proportional to the one in the active region when s = smax . To achieve that, we apply a gradient
compensation before updating etl .

Overcoming Catastrophic Forgetting with Hard Attention to the Task

Cumulative ql, i

Constant s = 1
Constant s = smax
Annealed s
Compensated

−6

−4

−2

0
el,t i

2

4

6

Figure 2. Illustration of the effect that annealing s has on the gradient q of et .

In essence, the idea of the embedding gradient compensation is to remove the effects of the annealed sigmoid and
to artificially impose the desired range and magnitude motivated in the previous paragraph. To do so, we divide the
gradient ql,i by the derivative of the annealed sigmoid, and
multiply by the desired compensation,
 h
 i
smax σ etl,i 1 − σ etl,i
0

h

i ql,i ,
ql,i
=
sσ setl,i 1 − σ setl,i
which, after operating, yields
h


i
smax cosh setl,i + 1
0
h
 
i ql,i .
ql,i
=
s cosh etl,i + 1

The use of L1 regularization to promote network sparsity
in the context of catastrophic forgetting has also been considered by Yoon et al. (2018) with dynamically expandable
networks (DEN), which were introduced while developing
HAT. In DEN, plain L1 regularization is combined with a
considerable set of heuristics such as L2-transfer, thresholding, and a measure of “semantic drift”, and is applied
to all network weights in the so-called “selective retraining”
phase. In HAT, we use an attention-weighted L1 regularization over attention values, which is an independent part of
the single training phase of the approach. Instead of considering network weights, HAT focuses on unit attention.

3. Related Work

|setl,i |

For numerical stability, we clamp
≤ 50 and constrain
etl,i to remain within the active range of the standard sigmoid,
etl,i ∈ [−6, 6]. In any case, however, ql,i → 0 when we hit
those limits. That is, we are in the constant regions of
the pseudo-step function. Notice also that, by Eq. 3, the
minimum s is never equal to 0.
2.6. Promoting Low Capacity Usage
It is important to realize that the hard attention values
atl,i that are ‘active’, that is, atl,i → 1, directly determine the units that will be dedicated to task t. Therefore, in order to have some model capacity reserved for
future tasks, we promote sparsity on the set of attention
vectors At = {at1 , . . . atL−1 }. To do so, we add a regularization term to the loss function L that takes into account
the set of cumulative attention vectors up to task t − 1,
<t
A<t = {a<t
1 , . . . aL−1 }:


L0 y, ŷ, At , A<t = L (y, ŷ) + cR At , A<t , (4)
where c is the regularization constant,

PL−1 PNl t 
al,i 1 − a<t

l=1
i=1
l,i
R At , A<t =
PL−1 PNl
<t
l=1
i=1 1 − al,i

is the regularization term, and Nl corresponds to the number of units in layer l. Notice that Eq. 5 corresponds to a
weighted and normalized L1 regularization over At . Cumulative attentions over the past tasks A<t define a weight for
t
the current task, such that if a<t
l,i → 1 then al,i receives a
weight close to 0 and vice versa. This excludes the units
that were attended in previous tasks from regularization,
unconstraining their reuse in the current task. The hyperparameter c ≥ 0 controls the capacity spent on each task
(Eq. 4). In a sense, it can be thought of as a compressibility
constant, affecting the compactness of the learned models:
the higher the c, the lower the number of active attention
values atl,i and the more sparse the resulting network is. We
set c globally for all tasks and let HAT adapt to the best
compression for each individual task.

(5)

We compare the proposed approach with the conceptually
closest works, some of which appeared concurrently to the
development of HAT. A more general overview of related
work has been done in Sec. 1. A qualitative comparison
with three of the most related strategies has been done along
Sec. 2. A quantitative comparison with these and other
approaches is done in Sec. 4 and Appendix C.
Both elastic weight consolidation (EWC; Kirkpatrick et al.,
2017) and synaptic intelligence (SI; Zenke et al., 2017) approaches add a ‘soft’ structural regularization term to the
loss function in order to discourage changes to weights that
are important for previous tasks. HAT uses a ‘hard’ structural regularization, and does it both at the loss function
and gradient magnitudes explicitly. EWC measures weights’
importance after network training, while SI and HAT compute weights’ importance concurrently to network training.
EWC and SI use specific formulation while HAT learns
attention masks. Incremental moment matching (IMM; Lee
et al., 2017) is an evolution of EWC, performing a separate
model-merging step after learning a new task.
Progressive neural networks (PNNs; Rusu et al., 2016) distribute the network weights in a column-wise fashion, pre-

Overcoming Catastrophic Forgetting with Hard Attention to the Task

assigning a column width per task. They employ so-called
adapters to reuse knowledge from previous columns/tasks,
leading to a progressive increase of the number of weights
assigned to future tasks. Instead of blindly pre-assigning
column widths, HAT learns such ‘widths’ per layer, together
with the network weights, and adapts them to the difficulty
of the current task. PathNet (Fernando et al., 2017) also
pre-assigns some amount of network capacity per task but,
in contrast to PNNs, avoids network columns and adapters.
It uses an evolutionary approach to learn paths between a
constant number of so-called modules (layer subsets) that
interconnect between themselves. HAT does not maintain a
population of solutions, entirely trains with backpropagation
and SGD, and does not rely on a constant set of modules.
Together with PNNs and PathNet, PackNet (Mallya &
Lazebnik, 2017) also employs a binary mask to constrain the
network. However, such constrain is not based on columns
nor layer modules, but on network weights. Therefore, it
allows for a potentially better use of the network’s capacity. PackNet is based on heuristic weight pruning, with
pre-assigned pruning ratios. HAT also focuses on network
weights, but uses unit-based masks to constrain those, which
also results in a lightweight structure. It avoids any absolute
or pre-assigned pruning ratio, although it uses the compressibility parameter c to influence the compactness of the
learned models. Another difference between HAT and the
previous three approaches is that it does not use purely binary masks. Instead, the stability parameter smax controls
the degree of binarization.
Dynamically expandable networks (DEN; Yoon et al., 2018)
also assign network capacity depending on the task at hand.
However, they do so in a separate stage called “selective
retraining”. A complex mixture of heuristics and hyperparameters is used to identify “drifting” units, which are
duplicated and retrained in another stage. L1 regularization and L2-transfer are employed to condition learning,
together with the corresponding regularization constants
and an additional set of thresholds. HAT strives for simplicity, restricting the number of hyperparameters to two that
have a straightforward conceptual interpretation. Instead
of plain L1 regularization over network weights, HAT employs an attention-weighted L1 regularization over attention
masks. Attention masks are a lightweight structure that can
be plugged in without the need of introducing important
changes to a pre-existing network.

4. Experiments
Setups — Common setups to evaluate catastrophic forgetting in a classification context are based on permutations
of the MNIST data (Srivastava et al., 2013), label splits
of the MNIST data (Lee et al., 2017), incrementally learning classes of the CIFAR data sets (Lopez-Paz & Ranzato,

2017), or two-task transfer learning setups where accuracy
is measured on both source and target tasks (Li & Hoiem,
2017). However, there are some limitations with these setups. Firstly, performing permutations of the MNIST data
has been suggested to favor certain approaches, yielding misleading results3 in the context of catastrophic forgetting (Lee
et al., 2017). Secondly, using only the MNIST data may
not be very representative of modern computer vision tasks,
nor particularly challenging (Xiao et al., 2017). Thirdly,
incrementally adding classes or groups of classes implies
the assumption that all data comes from the same joint distribution, which is unrealistic for a real-world setting. Finally,
evaluating catastrophic forgetting with only two tasks biases
the conclusions towards transfer learning setups, and prevents the analysis of truly sequential learning with more than
two tasks. In this paper, we consider the aforementioned
MNIST and CIFAR setups (Sec. 4.2). Nonetheless, we primarily evaluate on a sequence of multiple tasks formed by
different classification data sets (Sec. 4.1).
To obtain a generic estimate, we weigh a number of tasks
and uniformly randomize their order. After training task t,
we compute the accuracies on all testing sets of tasks τ ≤ t.
We repeat 10 times this sequential train/test procedure with
10 different seed numbers, which are also used in the rest of
randomizations and initializations (see below). To compare
between different task accuracies, and in order to obtain
a general measurement of the amount of forgetting, we
introduce the forgetting ratio
ρτ ≤t =

Aτ ≤t − AτR
AτJ ≤t − AτR

− 1,

(6)

where Aτ ≤t is the accuracy measured on task τ after sequentially learning task t, AτR is the accuracy of a random,
frequency-based classifier solely trained on task τ , and AτJ ≤t
is the accuracy measured on task τ after jointly learning t
tasks in a multitask fashion. Note that ρ ≈ −1 and ρ ≈ 0
correspond to performances close to the ones of the random
and multitask classifiers, respectively. To report a single
number after learning t tasks, we take the average
t

ρ≤t =

1 X τ ≤t
ρ .
t τ =1

Data — We consider 8 common image classification data
sets and adapt them, if necessary, to an input size of
32 × 32 × 3 pixels. The number of classes goes from
10 to 100, training set sizes from 16,853 to 73,257, and
test set sizes from 1,873 to 26,032. For each task, we ran3
Essentially, the MNIST data contains many values close to
0 that allow for an easier identification of the important units or
weights which, if permuted, can then be easily frozen without
overlapping with the ones of the other tasks (see Lee et al., 2017).

Overcoming Catastrophic Forgetting with Hard Attention to the Task

Baselines — We consider 2 reference approaches plus
9 recent and competitive ones: standard SGD with
dropout (Goodfellow et al., 2014), SGD freezing all layers except the last one (SGD-F), EWC, IMM (Mean and
Mode variants), learning without forgetting (LWF; Li &
Hoiem, 2017), less-forgetting learning (LFL; Jung et al.,
2016), PathNet, and PNNs. To find the best hyperparameter
combination for each approach, we perform a grid search using a task sequence determined by a single seed. To compute
the forgetting ratio ρ (Eq. 6), we also run the aforementioned
random and multitask classifiers.
Network — Unless stated otherwise, we employ an
AlexNet-like architecture (Krizhevsky et al., 2012) with
3 convolutional layers of 64, 128, and 256 filters with 4 × 4,
3 × 3, and 2 × 2 kernel sizes, respectively, plus two fullyconnected layers of 2048 units each. We use rectified linear
units as activations, and 2 × 2 max-pooling after the convolutional layers. We also use a dropout of 0.2 for the
first two layers and of 0.5 for the rest. A fully-connected
layer with a softmax output is used as a final layer, together
with categorical cross entropy loss. All layers are randomly
initialized with Xavier uniform initialization (Glorot & Bengio, 2010) except the embedding layers, for which we use a
Gaussian distribution N (0, 1). Unless stated otherwise, our
code uses PyTorch’s defaults for version 0.2.0 (Paszke et al.,
2017). We adapt the same base architecture to all baseline
approaches and match their number of parameters to 7.1 M.

0.0
−0.1
−0.2
ρ ≤t

domly split 15% of the training set and keep it for validation purposes. The considered data sets are: CIFAR10 and
CIFAR100 (Krizhevsky, 2009), FaceScrub (Ng & Winkler,
2014), FashionMNIST (Xiao et al., 2017), NotMNIST (Bulatov, 2011), MNIST (LeCun et al., 1998), SVHN (Netzer
et al., 2011), and TrafficSigns (Stallkamp et al., 2011). For
further details on data we refer to Appendix A.

−0.3
−0.4
−0.5
−0.6
1

2

Multitask
SGD
SGD-F

3

4

t

EWC
IMM-Mode
IMM-Mean

5

6

7

LWF
LFL
PathNet

8
PNN
HAT

Figure 3. Average forgetting ratio ρ≤t for the considered approaches (10 runs).
Table 1. Average forgetting ratio after the second (ρ≤2 ) and the
last (ρ≤8 ) task for the considered approaches (10 runs, standard
deviation into parenthesis).
A PPROACH
LFL
LWF
SGD
IMM-M ODE
SGD-F
IMM-M EAN
EWC
PATH N ET
PNN
HAT

ρ≤2

ρ≤8

-0.73 (0.29)
-0.14 (0.13)
-0.20 (0.08)
-0.11 (0.08)
-0.20 (0.15)
-0.12 (0.10)
-0.08 (0.06)
-0.09 (0.16)
-0.11 (0.10)
-0.02 (0.03)

-0.92 (0.08)
-0.80 (0.06)
-0.66 (0.03)
-0.49 (0.05)
-0.44 (0.06)
-0.42 (0.04)
-0.25 (0.03)
-0.17 (0.23)
-0.11 (0.01)
-0.06 (0.01)

4.1. Results

beyond a transfer learning setup. We find LFL extremely
sensitive to the configuration of its hyperparameter, to the
point that what is a good value for one seed, turns out to
be a bad choice for another seed. Hence the poor average
performance for 10 seeds. The highest standard deviations
are obtained by LFL and PathNet (Table 1), which suggests
a high sensitivity with respect to hyperparameters, initializations, or data sets. Another thing to note is that the IMM
approaches only perform similarly or slightly better than
the SGD-F reference. We believe this is due to both the
different nature of the tasks’ data and the consideration of
more than two tasks, which complicates the choice of the
mixing hyperparameter.

We first look at the average forgetting ratio ρ≤t after learning
task t (Fig. 3). A first thing to note is that not all the considered baselines perform better than the SGD references. That
is the case of LWF and LFL. For LWF, we observe it is still
competitive in the two-task setup for which it was designed,
t = 2. However, its performance rapidly degrades for t > 2,
indicating that the approach has difficulties in extending

The best performing baselines are EWC, PathNet, and PNN.
PathNet and PNN present contrasting behaviors. Both, by
construction, never forget; therefore, the important difference is in their learning capability. PathNet starts by correctly learning the first task and progressively exhibits difficulties to do so for t ≥ 2. Contrastingly, PNNs exhibits
difficulty in the first tasks and becomes better as t increases.

Training — We train all models with backpropagation and
plain SGD, using a learning rate of 0.05, and decaying it by
a factor of 3 if there is no improvement in the validation loss
for 5 consecutive epochs. We stop training when we reach
a learning rate lower than 10−4 or we have iterated over
200 epochs (we made sure that all considered approaches
reached a stable solution before 200 epochs). Batch size is
set to 64. All methods use the same task sequence, data split,
batch shuffle, and weight initialization for a given seed.

Overcoming Catastrophic Forgetting with Hard Attention to the Task

We now move to the HAT results. First of all, we observe
that HAT consistently performs better than all considered
baselines for all t ≥ 2 (Fig. 3). For the case of t = 2, it
obtains an average forgetting ratio ρ≤2 = −0.02, while the
best baseline is EWC with ρ≤2 = −0.08 (Table 1). For
the case of t = 8, HAT obtains ρ≤8 = −0.06, while the
best baseline is PNN with ρ≤8 = −0.11. This implies
a reduction in forgetting of 75% for t = 2 and 45% for
t = 8. Notice that the standard deviation of HAT is lower
than the ones obtained by the big majority of the baselines
(Table 1). This denotes a certain stability of HAT with
respect to different task sequences, data sets, data splits, and
network initializations.
Given the slightly increasing tendency of PNN with t
(Fig. 3), one could speculate that PNN would score above
HAT for t > 8. However, our empirical analyzes suggest
that that is not the case (presumably due to the capacity
pre-assignment and parameter increase problems underlined
in Sec. 3 and above). In particular, we observe a gradual
lowering of PathNet and PNN curves with increasing sequences from t = 2 to 8. In addition, we observe PathNet
and PNN obtaining worse performances than EWC in the
case of t = 10 for the incremental class setup (see below
and Appendix C.1). In general, none of the baseline methods consistently outperforms the rest across setups and for
all t, a situation that we do observe with HAT.
4.2. Additional Results
To broaden the strength of our results, we additionally experiment with three common alternative setups. First, we
consider an incremental class learning scenario, similar
to Lopez-Paz & Ranzato (2017), using class subsets of both
CIFAR10 and CIFAR100 data. In this setup, the best baseline after t ≥ 3 is EWC, with ρ≤10 = −0.18. HAT scores
ρ≤10 = −0.09 (55% forgetting reduction). Next, we consider the permuted MNIST sequence of tasks (Srivastava
et al., 2013). In this setup, the best result we could find in the
literature was from SI, with A≤10 = 97.1%. HAT scores
A≤10 = 98.6% (52% error rate reduction). Finally, we also
consider the split MNIST task of Lee et al. (2017). In this
setup, the best result from the literature corresponds to the
conceptor-aided backpropagation approach (He & Jaeger,
2018), with A≤2 = 94.9%. HAT scores A≤2 = 99.0%
(80% error rate reduction). The detail for all these setups
and results can be found in Appendix C.

0.00
−0.05
ρ ≤8

These contrasting behaviors are due to the way the two approaches allocate the network capacity. As mentioned, they
cannot do it dynamically, and therefore need to pre-assign
a number of network weights per task. When having more
tasks but the same network capacity, this pre-assignment
increasingly harms the performance of these baselines, lowering the corresponding curves in Fig. 3.

−0.10
−0.15
−0.20

0

1
smax = 25
smax = 50

2
c
smax = 100
smax = 200

3

4

smax = 400
smax = 800

Figure 4. Effect of hyperparameters smax and c on average forgetting ratio ρ≤8 . Results for seed 0.

4.3. Hyperparameters
In any machine learning algorithm, it is important to assess
the sensitivity with respect to the hyperparameters. HAT
has two: the stability parameter smax and the compressibility parameter c (Secs. 2.4 and 2.6). A low smax provides
plasticity to the units and capacity of adaptation, but the
network may easily forget what it learned. A high smax
prevents forgetting, but the network may have difficulties in
adapting to new tasks. A low c allows to use almost all of
the network’s capacity for a given task, potentially spending
too much in the current task. A high c forces it to learn
a very compact model, at the expense of not reaching the
accuracy that the original network could have reached. We
empirically found good operation ranges smax ∈ [25, 800]
and c ∈ [0.1, 2.5]. As we can see, any variation within these
ranges results in reasonable performance (Fig. 4). Unless
stated otherwise, we use smax = 400 and c = 0.75.
4.4. Monitoring and Network Pruning
It is interesting to note that the hard attention mechanism introduced in Sec. 2 offers a number of possibilities to monitor
the behavior of our models. For instance, by computing the
conditioning mask in Eq. 2 from the hard attention vectors
a≤t
l , we can assess which weights obtain a high attention
value, binarize it, and compute an estimate of the instantaneous network capacity usage (Fig. 5). We may also inform
ourselves of the amount of active weights per layer and task
(Appendix B.2). Another facet we can monitor is the weight
reuse across tasks. By a similar procedure, comparing the
conditioning masks between tasks ti and tj , j > i, we
can asses the percentage of weights of task ti that are later
reused in task tj (Fig. 6).
Another by-product of hard attention masks is that we can

Overcoming Catastrophic Forgetting with Hard Attention to the Task

80

80

60

60

A 1 [%]

100

Used capacity [%]

100

40
20
0

20
0

200

400
Epochs

FaceScrub
MNIST
CIFAR100

0

600

NotMNIST
SVHN
CIFAR10

TrafficSigns
FashionMNIST

Figure 5. Network capacity usage with sequential task learning
(seed 0). Dashed vertical lines correspond to a task switch.

1

5

2

6

5

5

5

5

5

4

3

3

4

4

3

9

8

11

10

9

8

10

9

9

11

11

10

13

12

3
Task

40

4
5
6

0

20
CIFAR10
CIFAR100
FaceScrub

40
60
Compression [%]
FashionMNIST
MNIST
NotMNIST

80

100
SVHN
TrafficSigns

Figure 7. Validation accuracy A1 as a function of compression
percentage. Every dot corresponds to an epoch and triangles match
the accuracy of the SGD approach (no compression).

the task (Appendix B.3). Comparing these numbers with the
compression rates used by PackNet (25 or 50%), we see that
HAT generally uses a much more compact model. Comparing with DEN on the specific MNIST and CIFAR100 tasks
(18 and 52%), we observe that HAT compresses to 1 and
21%, respectively. Interestingly, and in contrast to these and
the majority of network pruning approaches, HAT learns to
prune network weights through backpropagation and SGD,
and at the same time as the network weights themselves.

13

7
2

3

4

5
Task

6

7

8

Figure 6. Percentage of weight reuse across tasks. Seed 0 sequence:
(1) FaceScrub, (2) MNIST, (3) CIFAR100, (4) NotMNIST, (5)
SVHN, (6) CIFAR10, (7) TrafficSigns, and (8) FashionMNIST.

use them to assess which of the network’s weights are important, and then prune the most irrelevant ones (LeCun et al.,
1990). This way, we can compress the network for further
deployment in low-resource devices or time-constrained
environments (cf. Han et al., 2016). If we want to focus
on such compression task, we can set c to a higher value
than the one used for catastrophic forgetting and start with
a positive random initialization of the embeddings el . The
former will promote more compression while the latter will
ensure we start learning the model by putting attention to all
weights in the first epochs (full capacity). We empirically
found that using c = 1.5 and U(0, 2) yields a reasonable
trade-off between accuracy and compression for a single
task (Fig. 7). With that, we can compress the network to
sizes between 1 and 21% of its original size, depending on

5. Conclusion
We introduce HAT, a hard attention mechanism that, by
focusing on a task embedding, is able to protect the information of previous tasks while learning new tasks. This
hard attention mechanism is lightweight, in the sense that
it adds a small fraction of weights to the base network,
and is trained together with the main model, with negligible overhead using backpropagation and vanilla SGD. We
demonstrate the effectiveness of the approach to control
catastrophic forgetting in the image classification context by
running a series of experiments with multiple data sets and
state-of-the-art approaches. HAT has only two hyperparameters, which intuitively refer to the stability and compactness
of the learned knowledge, and whose tuning we demonstrate
is not crucial for obtaining good performance. In addition,
HAT offers the possibility to monitor the used network capacity across tasks and layers, the unit reuse across tasks,
and the compressibility of a model trained for a given task.
We hope that our approach may be also useful in online
learning or network compression contexts, and that the hard
attention mechanism presented here may also find some
applicability beyond the catastrophic forgetting problem.

Overcoming Catastrophic Forgetting with Hard Attention to the Task

References
Bakker, B. and Heskes, T. Task clustering and gating for
bayesian multitask learning. Journal of Machine Learning Research, 4:83–99, 2003.
Bulatov, Y.
NotMNIST dataset.
Technical report,
2011. URL http://yaroslavvb.blogspot.it/
2011/09/notmnist-dataset.html.
Clegg, B. A., DiGirolamo, G. J., and Keele, S. W. Sequence
learning. Trends in Cognitive Sciences, 2(8):275–281,
1998.
Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient
methods for online learning and stochastic optimization.
Journal of Machine Learning Research, 12:2121–2159,
2011.
Evgeniou, T. and Pontil, M. Regularized multi-task learning.
In Proc. of the ACM SIGKDD Int. Conf. on Knowledge
Discovery and Data Mining (KDD), pp. 109–117, 2004.

Kemelmacher-Shlizerman, I., Seitz, S. M., Miller, D., and
Brossard, E. The megaface benchmark: 1 million faces
for recognition at scale. In Proc. of the IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), pp.
4873–4882, 2016.
Kingma, D. P. and Ba, J. L. Adam: a method for stochastic
optimization. In Proc. of the Int. Conf. on Learning
Representations (ICLR), 2015.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T.,
Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. Overcoming catastrophic forgetting in neural networks. Proc. of the National Academy
of Sciences of the USA, 114(13):3521–3526, 2017.
Krizhevsky, A. Learning multiple layers of features from
tiny images. Msc thesis, University of Toronto, Toronto,
Canada, 2009.

Fernando, C., Banarse, D., Blundell, C., Zwols, Y., Ha, D.,
Rusu, A. A., Pritzel, A., and Wierstra, D. PathNet: evolution channels gradient descent in super neural networks.
ArXiv, 1701.08734, 2017.

Krizhevsky, A., Sutskever, I., and Hinton, G. ImageNet
classification with deep convolutional neural networks.
In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger,
K. Q. (eds.), Advances in Neural Information Processing
Systems (NIPS), volume 25, pp. 1097–1105. 2012.

French, R. M. Using semi-distributed representations to
overcome catastrophic forgetting in connectionist networks. In Proc. of the Annual Conf. of the Cognitive
Science Society (CogSci), pp. 173–178, 1991.

LeCun, Y., Denker, J. S., and Solla, S. A. Optimal brain
damage. In Touretzky, D. S. (ed.), Advances in Neural
Information Processing Systems (NIPS), volume 2, pp.
598–605. Morgan Kaufmann, 1990.

Glorot, X. and Bengio, Y. Understanding the difficulty
of training deep feedforward neural networks. In Proc.
of the Int. Conf. on Artificial Intelligence and Statistics
(AISTATS), pp. 249–256, 2010.

LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.

Goodfellow, I., Mizra, M., Da, X., Courville, A., and Bengio, Y. An empirical investigation of catastrophic forgetting in gradient-based neural networks. In Proc. of the
Int. Conf. on Learning Representations (ICLR), 2014.
Gutsein, S. and Stump, E. Reduction of catastrophic forgetting with transfer learning and ternary output codes. In
Proc. of the Int. Joint Conf. on Neural Networks (IJCNN),
pp. 1–8, 2015.
Han, S., Mao, H., and Dally, W. J. Deep compression:
compressing deep neural networks with pruning, trained
quantization and Huffman coding. In Proc. of the Int.
Conf. on Learning Representations (ICLR), 2016.
He, X. and Jaeger, H. Overcoming catastrophic interference
using conceptor-aided backpropagation. In Proc. of the
Int. Conf. on Learning Representations (ICLR), 2018.
Jung, H., Ju, J., Jung, M., and Kim, J. Less-forgetting
learning in deep neural networks. ArXiv, 1607.00122,
2016.

Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., and Zhang, B.-T.
Overcoming catastrophic forgetting by incremental moment matching. In Guyon, I., Luxburg, U. V., Bengio, S.,
Wallach, H., Fergus, R., Vishwanathan, S., and Garnett,
R. (eds.), Advances in Neural Information Processing
Systems (NIPS), volume 30, pp. 4655–4665. Curran Associates Inc., 2017.
Legg, S. and Hutter, M. Universal intelligence: a definition
of machine intelligence. Minds and Machines, 17(4):
391–444, 2007.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE
Trans. on Pattern Analysis and Machine Intelligence, PP
(99):1–1, 2017.
Lopez-Paz, D. and Ranzato, M. A. Gradient episodic memory for continuum learning. In Guyon, I., Luxburg, U. V.,
Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S.,
and Garnett, R. (eds.), Advances in Neural Information
Processing Systems (NIPS), volume 30, pp. 6449–6458.
Curran Associates Inc., 2017.

Overcoming Catastrophic Forgetting with Hard Attention to the Task

Mallya, A. and Lazebnik, S. PackNet: adding multiple
tasks to a single network by iterative pruning. ArXiv,
1711.05769, 2017.
McCloskey, M. and Cohen, N. Catastrophic interference
in connectionist networks: the sequential learning problem. Psychology of Learning and Motivation, 24:109–
165, 1989.
McCulloch, W. S. and Pitts, W. A logical calculus of the
ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 5(4):115–133, 1943.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B.,
and Ng, A. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep
Learning and Unsupervised Feature Learning (NIPSDeepLearning), 2011.
Ng, H.-W. and Winkler, S. A data-driven approach to cleaning large face datasets. In Proc. of the IEEE Int. Conf. on
Image Processing (ICIP), pp. 343–347, 2014.
Nguyen, C., Li, Y., Bui, T. D., and Turner, R. E. Variational
continual learning. ArXiv, 1710.10628, 2017.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E.,
DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer,
A. Automatic differentiation in PyTorch. In NIPS Workshop on The Future of Gradient-based Machine Learning
Software & Techniques (NIPS-Autodiff), 2017.
Ratcliff, R. Connectionist models of recognition memory:
constraints imposed by learning and forgetting functions.
Psychological Review, 97:285–308, 1990.
Rebuffi, S., Kolesnikov, A., Sperl, G., and Lampert, C.
iCaRL: incremental classifier and representation learning. In Proc. of the IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), pp. 2001–2010, 2017.

Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning
with deep generative replay. In Guyon, I., Luxburg, U. V.,
Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S.,
and Garnett, R. (eds.), Advances in Neural Information
Processing Systems (NIPS), volume 30, pp. 2993–3002.
Curran Associates Inc., 2017.
Sprechmann, P., Jayakumar, S., Rae, J., Pritzel, A., Puigdomènech, A., Uria, B., Vinyals, O., Hassabis, D., Pascanu, R., and Blundell, C. Memory-based parameter
adaptation. In Proc. of the Int. Conf. on Learning Representations (ICLR), 2018.
Srivastava, R. K., Masci, J., Kazerounian, S., Gomez, F.,
and Schmidhuber, J. Compete to compute. In Burges,
C. J. C., Bottou, L., Welling, M., Ghahramani, Z., and
Weinberger, K. (eds.), Advances in Neural Information
Processing Systems (NIPS), volume 26, pp. 2310–2318.
Curran Associates Inc., 2013.
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. The
German traffic sign recognition benchmark: a multi-class
classification competition. In Proc. of the Int. Joint Conf.
on Neural Networks (IJCNN), pp. 1453–1460, 2011.
Thrun, S. and Mitchell, T. Lifelong robot learning. Robotics
and Autonomous Systems, 15:25–46, 1995.
Venkatesan, R., Venkateswara, H., Panchanathan, S., and Li,
B. A strategy for an uncompromising incremental learner.
ArXiv, 1705.00744, 2017.
Wang, H. and Raj, B. On the origin of deep learning. ArXiv,
1702.07800, 2017.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-MNIST: a
novel image dataset for benchmarking machine learning
algorithms. ArXiv, 1708.07747, 2017.

Robins, A. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7:123–146, 1995.

Yoon, J., Yang, E., Lee, J., and Hwang, S. J. Lifelong
learning with dynamically expandable networks. In Proc.
of the Int. Conf. on Learning Representations (ICLR),
2018.

Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H.,
Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. Progressive neural networks. ArXiv, 1606.04671,
2016.

Zenke, F., Poole, B., and Ganguli, S. Improved multitask
learning through synaptic intelligence. In Proc. of the
Int. Conf. on Machine Learning (ICML), pp. 3987–3995,
2017.

Overcoming Catastrophic Forgetting with Hard Attention to the Task

APPENDIX
A. Data
The data sets used in our experiments are summarized in Table 2. The MNIST data set (LeCun et al., 1998) comprises
28 × 28 monochromatic images of handwritten digits. Fashion-MNIST (Xiao et al., 2017) comprises gray-scale images of
the same size from Zalando’s articles4 . The German traffic sign data set (TrafficSigns; Stallkamp et al., 2011) contains traffic
sign images. We used the version of the data set from the Udacity self-driving car github repository5 . The NotMNIST data
set (Bulatov, 2011) comprises glyphs extracted from publicly available fonts, making a similar data set to MNIST; we just
need to resize the images6 . The SVHN data set (Netzer et al., 2011) comprises digits cropped from house numbers in Google
Street View images. The FaceScrub data set (Ng & Winkler, 2014) is widely used in face recognition tasks (KemelmacherShlizerman et al., 2016). Because some of the images listed in the original data set were not hosted anymore on the
corresponding Internet domains, we use a version of the data set stored on the MegaFace challenge website7 (KemelmacherShlizerman et al., 2016), from which we select the first 100 people with the most appearances8 . The CIFAR10 and CIFAR100
data sets contain 32 × 32 color images (Krizhevsky, 2009).
To match the image input shape required in our experiments, some of the images in the corresponding data sets need to be
resized (FaceScrub, TrafficSigns, and NotMNIST) or padded with zeros (MNIST and FashionMNIST). In addition, for the
data sets comprising monochromatic images, we replicate the image across all RGB channels. Note that we do not perform
any sort of data augmentation; we just adapt the inputs. We provide the necessary code to perform such adaptations in the
links listed above.
Table 2. Data sets used in the study: name, reference, number of classes, and number of train and test instances.
DATA SET
CIFAR10 (K RIZHEVSKY , 2009)
CIFAR100 (K RIZHEVSKY , 2009)
FACE S CRUB (N G & W INKLER , 2014)
FASHION MNIST (X IAO ET AL ., 2017)
N OT MNIST (B ULATOV , 2011)
MNIST (L E C UN ET AL ., 1998)
SVHN (N ETZER ET AL ., 2011)
T RAFFIC S IGNS (S TALLKAMP ET AL ., 2011)

C LASSES

T RAIN

T EST

10
100
100
10
10
10
100
43

50,000
50,000
20,600
60,000
16,853
60,000
73,257
39,209

10,000
10,000
2,289
10,000
1,873
10,000
26,032
12,630

B. Raw Results
B.1. Task Mixture
We report all forgetting ratios ρ≤t for t = 1 to 8 in Table 3. A total of 10 runs with 10 different seeds are performed and the
averages and standard deviations are taken.
B.2. Layer Use
In Fig. 8 we show an example of layer capacity monitoring as the sequence of tasks evolves. As mentioned in the main
paper, we can compute a percent of active weights for a given layer and task.
B.3. Network Compression
The final results of the network compression experiment reported in the main paper (after reaching convergence) are available
in Table 4. We run HAT on isolated tasks with c = 1.5 and uniform embedding initialization U(0, 2).
4

https://github.com/zalandoresearch/fashion-mnist
https://github.com/georgesung/traffic_sign_classification_german
6
Code is attached in a ZIP file.
7
http://megaface.cs.washington.edu/participate/challenge.html
8
Code is attached in a ZIP file.
5

Overcoming Catastrophic Forgetting with Hard Attention to the Task
Table 3. Average forgetting ratio ρ≤t for the considered approaches (10 runs, standard deviation into parenthesis).

A PPROACH
LFL
LWF
SGD
IMM-M ODE
SGD-F
IMM-M EAN
EWC
PATH N ET
PNN
HAT

ρ≤1

ρ≤2

ρ≤3

ρ≤4

ρ≤5

ρ≤6

ρ≤7

ρ≤8

-0.00 (0.01)
-0.00 (0.01)
-0.00 (0.00)
-0.00 (0.01)
-0.00 (0.00)
-0.00 (0.00)
-0.00 (0.00)
-0.02 (0.03)
-0.10 (0.12)
-0.01 (0.02)

-0.73 (0.29)
-0.14 (0.13)
-0.20 (0.08)
-0.11 (0.08)
-0.20 (0.15)
-0.12 (0.10)
-0.08 (0.06)
-0.09 (0.16)
-0.11 (0.10)
-0.02 (0.03)

-0.88 (0.18)
-0.38 (0.17)
-0.41 (0.09)
-0.27 (0.12)
-0.30 (0.15)
-0.24 (0.11)
-0.15 (0.11)
-0.11 (0.19)
-0.13 (0.09)
-0.03 (0.03)

-0.89 (0.13)
-0.63 (0.11)
-0.49 (0.07)
-0.37 (0.10)
-0.38 (0.11)
-0.32 (0.06)
-0.18 (0.07)
-0.12 (0.21)
-0.14 (0.04)
-0.03 (0.02)

-0.91 (0.11)
-0.68 (0.08)
-0.54 (0.07)
-0.39 (0.07)
-0.42 (0.09)
-0.37 (0.06)
-0.21 (0.07)
-0.14 (0.22)
-0.13 (0.03)
-0.04 (0.02)

-0.90 (0.09)
-0.70 (0.03)
-0.57 (0.06)
-0.45 (0.05)
-0.44 (0.08)
-0.40 (0.06)
-0.23 (0.04)
-0.15 (0.23)
-0.13 (0.02)
-0.05 (0.02)

-0.92 (0.08)
-0.76 (0.06)
-0.62 (0.06)
-0.49 (0.06)
-0.45 (0.07)
-0.42 (0.07)
-0.25 (0.05)
-0.17 (0.23)
-0.12 (0.01)
-0.06 (0.02)

-0.92 (0.08)
-0.80 (0.06)
-0.66 (0.03)
-0.49 (0.05)
-0.44 (0.06)
-0.42 (0.04)
-0.25 (0.03)
-0.17 (0.23)
-0.11 (0.01)
-0.06 (0.01)

Used weights [%]

100
80
60
40
20
0

Conv1 Conv2 Conv3 FC1
Layer
Cumulative
FaceScrub
MNIST

CIFAR100
NotMNIST
SVHN

FC2

FC3

CIFAR10
TrafficSigns
FashionMNIST

Figure 8. Layer-wise weight usage with sequential task learning, including (lines) and excluding (bars) the cumulative attention of past
tasks. Task sequence corresponds to seed 0.

Table 4. Results for the compression experiment reported in the main paper: test accuracy A1 with SGD, test accuracy A1 after
compressing with HAT, and percentage of network weights used after compression.
DATA SET
CIFAR10
CIFAR100
FACE S CRUB
FASHION MNIST
MNIST
N OT MNIST
SVHN
T RAFFIC S IGNS

R AW A1

C OMPRESSED A1

79.9%
52.7%
82.7%
92.4%
99.5%
90.9%
94.2%
97.5%

80.8%
49.1%
82.3%
91.9%
99.4%
91.5%
93.8%
98.1%

S IZE
13.9%
21.4%
21.0%
2.3%
1.2%
5.7%
3.1%
2.9%

B.4. Training Time
To have an idea of the training time for each of the considered approaches, we report some reference values in Table 5. We
see that HAT is also quite competitive in this aspect.

Overcoming Catastrophic Forgetting with Hard Attention to the Task
Table 5. Wall-clock training time measured on a single NVIDIA Pascal Titan X GPU: total (after learning the 8 tasks), per epoch, and
per batch (batches of 64). Batch processing time is measured for a forward pass (Batch-F), and for both a forward and a backward pass
(Batch-FB).
A PPROACH
T OTAL [ H ]
PNN
PATH N ET
EWC
M ULTITASK
IMM-M EAN
IMM-M ODE
LWF
HAT
SGD
LFL
SGD-F

T RAINING TIME
E POCH [ S ] BATCH -F [ MS ]

6.0
4.5
3.9
3.4
3.2
3.1
2.2
2.2
1.4
1.3
0.5

4.1
3.6
3.1
94.8
2.6
2.5
2.2
1.6
0.9
0.9
0.9

BATCH -FB [ MS ]

10.2
10.6
7.9
3.1
6.9
6.7
5.7
4.0
2.5
4.4
2.5

27.5
23.9
19.7
15.7
17.1
16.0
14.2
11.7
6.6
9.2
6.8

C. Additional Results
C.1. Incremental CIFAR
As an additional experiment to complement our evaluation, we consider the incremental CIFAR setup, following a similar
approach as Lopez-Paz & Ranzato (2017). We divide both CIFAR10 and CIFAR100 data sets into consecutive-class subsets
and use them as tasks, presented in random order according to the seed. We take groups of 2 classes for CIFAR10 and
20 classes for CIFAR100, yielding a total of 10 tasks. We decide to take groups of 2 and 20 classes in order to have a similar
number of training instances per task. The rest of the procedure is as in the main paper. The most important results are
summarized there. The complete numbers are depicted in Fig. 9 and reported in Table 6.

0.0
−0.1

ρ ≤t

−0.2
−0.3
−0.4
−0.5
−0.6
1

2

Multitask
SGD
SGD-F

3

4

5

EWC
IMM-Mode
IMM-Mean

t

6

7

8

LWF
LFL
PathNet

9

10
PNN
HAT

Figure 9. Average forgetting ratio ρ≤t for the incremental CIFAR task (average after 10 runs).

C.2. Permuted MNIST
A common experiment is the one proposed by Srivastava et al. (2013), and later employed to evaluate catastrophic forgetting
by Goodfellow et al. (2014). It consists of taking random permutations of the pixels in the MNIST data set as tasks. Typically,
the average accuracy after sequentially training on 10 MNIST permutations is reported. To match the different number of

Overcoming Catastrophic Forgetting with Hard Attention to the Task
Table 6. Average forgetting ratio ρ≤t for the incremental CIFAR task (10 runs, standard deviation into parenthesis).
A PPROACH
LFL
LWF
SGD-F
PATH N ET
SGD
IMM-M EAN
IMM-M ODE
PNN
EWC
HAT

ρ≤1

ρ≤2

ρ≤3

ρ≤4

ρ≤5

ρ≤6

ρ≤7

ρ≤8

ρ≤9

ρ≤10

-0.00 (0.01)
-0.00 (0.02)
-0.00 (0.01)
-0.15 (0.31)
-0.00 (0.01)
-0.00 (0.02)
-0.00 (0.01)
-0.26 (0.16)
-0.00 (0.01)
-0.03 (0.04)

-0.53 (0.31)
-0.10 (0.03)
-0.25 (0.14)
-0.18 (0.20)
-0.19 (0.09)
-0.14 (0.08)
-0.14 (0.10)
-0.26 (0.08)
-0.13 (0.09)
-0.05 (0.02)

-0.63 (0.25)
-0.27 (0.05)
-0.33 (0.16)
-0.21 (0.26)
-0.27 (0.09)
-0.21 (0.10)
-0.21 (0.11)
-0.25 (0.05)
-0.15 (0.08)
-0.05 (0.02)

-0.67 (0.21)
-0.42 (0.05)
-0.35 (0.18)
-0.22 (0.28)
-0.30 (0.04)
-0.22 (0.10)
-0.23 (0.06)
-0.23 (0.04)
-0.16 (0.07)
-0.06 (0.01)

-0.70 (0.20)
-0.50 (0.06)
-0.37 (0.16)
-0.24 (0.29)
-0.30 (0.06)
-0.25 (0.10)
-0.25 (0.09)
-0.22 (0.03)
-0.17 (0.06)
-0.06 (0.01)

-0.74 (0.17)
-0.53 (0.04)
-0.40 (0.18)
-0.27 (0.29)
-0.28 (0.04)
-0.26 (0.08)
-0.23 (0.07)
-0.23 (0.03)
-0.18 (0.06)
-0.07 (0.01)

-0.77 (0.15)
-0.59 (0.06)
-0.41 (0.18)
-0.28 (0.30)
-0.31 (0.03)
-0.27 (0.08)
-0.26 (0.05)
-0.22 (0.03)
-0.19 (0.08)
-0.07 (0.01)

-0.79 (0.14)
-0.64 (0.06)
-0.41 (0.19)
-0.30 (0.30)
-0.32 (0.04)
-0.28 (0.08)
-0.27 (0.04)
-0.21 (0.02)
-0.18 (0.07)
-0.08 (0.01)

-0.79 (0.14)
-0.68 (0.05)
-0.42 (0.19)
-0.32 (0.29)
-0.30 (0.05)
-0.29 (0.07)
-0.25 (0.04)
-0.21 (0.02)
-0.18 (0.06)
-0.08 (0.01)

-0.78 (0.14)
-0.70 (0.05)
-0.43 (0.20)
-0.35 (0.28)
-0.30 (0.04)
-0.30 (0.07)
-0.25 (0.04)
-0.21 (0.02)
-0.18 (0.06)
-0.09 (0.01)

Table 7. Accuracy on the permuted MNIST task (Srivastava et al., 2013), taking the average after training 10 tasks. The only exception is
the generative replay approach, whose performance was assessed after 5 tasks. Superscripts indicate results reported by (1) Nguyen et al.
(2017) and (2) He & Jaeger (2018). An asterisk after parameter count indicates that the approach presents some additional structure not
included in such parameter count (for instance, some memory module or an additional generative network).
A PPROACH
GEM (L OPEZ -PAZ & R ANZATO , 2017)
SI (Z ENKE ET AL ., 2017)1
EWC (K IRKPATRICK ET AL ., 2017)2
M B PA + EWC – 1000 EX . (S PRECHMANN ET AL ., 2018)
VCL (N GUYEN ET AL ., 2017)
HAT – S MALL
G ENERATIVE R EPLAY (S HIN ET AL ., 2017)
CAB (H E & JAEGER , 2018)
EWC (K IRKPATRICK ET AL ., 2017)
SI (Z ENKE ET AL ., 2017)
HAT – M EDIUM
HAT – L ARGE

PARAMETERS

A≤10

0.1 M*
0.1 M
0.1 M

82.8%
86.0%
88.2%
89.7%
90.0%
91.6%
94.9%
95.2%
96.9%
97.1%
97.4%
98.6%

U NKNOWN *
0.1 M*
0.1 M
U NKNOWN *
0.7 M
5.8 M
5.8 M
0.7 M
5.8 M

parameters used in the literature, we consider a small, medium, and a large network based on a two-layer fully-connected
architecture as Zenke et al. (2017), with 100, 500, and 2000 hidden units, respectively. For the large network we set dropout
probabilities as Kirkpatrick et al. (2017). We use smax = 200 and c = 0.5 for the small network, and smax = 400 and
c = 0.5 for the medium and large networks. The results are available in Table 7.
C.3. Split MNIST
Another popular experiment is to split the MNIST data set into tasks and report the average accuracy after learning them
one after the other. We follow Lee et al. (2017) by splitting the data set using labels 0–4 and 5–9 as tasks and running the
experiment 10 times. We also match the base network architecture to the one used by Lee et al. (2017). We train HAT for
50 epochs with c = 0.1. Results are reported in Table 8. In preliminary experiments we observed that dropout could increase
accuracy by some percentage. However, to keep the same configuration as in the cited reference, we finally did not use it.
Table 8. Average accuracy on the split MNIST task, following the setup of Lee et al. (2017) using 10 runs (standard deviation into
parenthesis). Superscript (1) indicates results reported by Lee et al. (2017).
A PPROACH
SGD (G OODFELLOW ET AL ., 2014)1
L2-T RANSFER (E VGENIOU & P ONTIL , 2004)1
IMM-M EAN (L EE ET AL ., 2017)
IMM-M ODE (L EE ET AL ., 2017)
CAB (H E & JAEGER , 2018)
HAT

PARAMETERS

A≤2

1.9 M
1.9 M
1.9 M
1.9 M
1.9 M
1.9 M

71.3% (1.5)
85.8% (0.5)
94.0% (0.2)
94.1% (0.3)
94.9% (0.3)
99.0% (0.0)

Overcoming Catastrophic Forgetting with Hard Attention to the Task

D. Variations to the Proposed Approach
In this section, we want to mention a number of alternatives we experimented with during the development of HAT. The
purpose of the section is not the report a formal set of results, but to inform the reader about potential different choices when
implementing HAT, or variations of it, and to give an intuition on the outcome of some of such choices.
D.1. Embedding Learning
When we realized that the embedding weights etl were not changing much and that their gradients were small compared
to the rest of the network due to the introduced annealing of s, we initially tackled the issue by using a different learning
rate for the embeddings. With that, we empirically found that factors of 10–50 times the original learning rate were able to
tackle the issue, leading to performances that were almost as good as the final ones reported in the main paper. However,
the use of a different learning rate introduced an additional parameter that we could not conceptually relate to catastrophic
forgetting and that could have been tricky to tune for a generic setting.
We also studied the use of an adaptive optimizer such as Adagrad (Duchi et al., 2011) or Adam (Kingma & Ba, 2015) for the
embedding weights. The idea was that an adaptive optimizer would be able to automatically introduce an appropriate scaling
factor. We found that this option was effectively learning suitable values for etl . However, its performance was worse than
the constant-factor SGD boost explained above. Noticeably, introducing an adaptive optimizer also introduces a number of
new hyperparameters: type of optimizer, another learning rate, weight decays, etc.
D.2. Annealing
In our effort to further reduce the number of hyperparameters, we experimented for quite some time with the annealing

 
b−1
π
1+
s = tan
4
B−1
or using variants of

s = α + β tan

π b−1
2 B−1


.

The rationale for the first expression is that one starts with a sigmoid σ(sx) that is equivalent to a straight line of 45 degrees
for b = 1 and x ≈ 0. Then, with b increasing, it linearly increases the angle towards 90 degrees at x = 0. The second
expression is a parametric evolution of the first one.
These annealing schedules have the (sometimes desirable) feature that the maximum s is infinite, yielding a true step
function in inference time. Therefore, we obtain truly binary attention vectors atl and no forgetting. In addition, if we use the
first expression, we are able to remove the smax hyperparameter. Nonetheless, we found the first expression to perform
worse than the solution proposed in the main paper. The introduction of the second expression with α = 1 and β < 1
improved the situation, but results were still not as good as the ones in the main paper and the tuning of β was a bit tricky.
To conclude this subsection, note that if smax is large, for instance smax > 100, one can use
s = smax

b−1
,
B−1

which is a much simpler annealing formula that closely approximates the one in the main paper. However, one needs then to
be careful with the denominator of the embedding gradient compensation when s = 0.
D.3. Gate
We also studied the use of alternatives to the sigmoid gate. Apart from the rescaled tanh, an interesting alternative we
thought of was a clamped version of the linear function,



setl
1
t
al = max 0, min 1,
+
,
r
2
where r defines the ‘valid’ range for the input of the gate. This gate yields a much simpler formulation for the gradient
compensation described in the main paper. However, it implies that we need to set r, which could be considered a

Overcoming Catastrophic Forgetting with Hard Attention to the Task

further hyperparameter. It also implies that embedding values that are far away from 0, the step transition point, receive a
proportionally similar gradient to the ones that are close to it. That is, values of etl that yield atl that are very close to 0 or 1
(in the constant region of the pseudo-step function) are treated equal to the ones that are still undecided (in the transition
region of the pseudo-step function). We did not test this alternative gate quantitatively.
D.4. Cumulative Attention
In the most preliminary stages we used
a≤t
l =1−

h

1 − atl





1 − al≤t−1

i

for accumulating attention across tasks, but it was soon dismissed for the final max-based formula. The previous equation
could be interesting for online learning scenarios with limited model capacity, together with


≤t−1
t
a≤t
,
l = max al , κ al
where κ is a constant slightly lower than 1 (for instance κ = 0.9 or κ = 0.99).
D.5. Embedding Initialization
We ran a set of experiments using uniform initialization U(0, k1 ) for the embeddings etl instead of Gaussian N (0, 1). We
also experimented with N (k2 , 1). The idea behind these alternative initializations was that, for sufficiently large smax , all
or almost all atl start with a value of 1, which has the effect of distributing the attention over all units for more time at the
beginning of training. Using values of k1 ∈ [1, 6] and k2 ∈ [0.5, 2] yielded competitive results, yet worse than the ones
using N (0, 1). Our intuition is that a uniform initialization like U(0, 2) is better for a purely compressive approach, as used
in the last experiment of the main paper.
D.6. Attention Regularization
We initially experimented with a normalized L1 regularization
PL−1 PNl t

l=1
i=1 al,i
t
R A =
.
PL−1
l=1 Nl
Results were a small percentage lower than the ones with the attention-weighted regularization of the main paper. We also
exchanged the previous L1 regularization with the L2-based regularization
PL−1 PNl t 2

l=1
i=1 (al,i )
t
R A =
.
PL−1
l=1 Nl
With that, we observed similar accuracies as the L1 regularization, but under different values for the hyperparameter c.
D.7. Hard Attention to the Input
As mentioned in the main paper, no attention mask is used for the input (that is, there is no at0 ). We find this is a good
strategy for a general image classification problem and for first-layer convolutional filters in particular. However, if the input
consists of independent, isolated features, one may think of putting hard attention to the input as a kind of supervised feature
selection process. We performed a number of experiments using only fully-connected layers and the MNIST data as above,
and introduced additional hard attention vectors at0 that directly multiplied the input of the network. The results suggested
that it could potentially be a viable option for feature selection and data compression (Fig. 10).

E. A Note on Binary Masks
After writing a first version of the paper, we realized that the idea of a binary mask that affects a given unit could be
potentially traced back to the “inhibitory synapses” of McCulloch & Pitts (1943). This idea of inhibitory synapses is quite
unconventional and rarely seen today (Wang & Raj, 2017) and, to the best of our knowledge, no specific way for learning

Overcoming Catastrophic Forgetting with Hard Attention to the Task

Figure 10. Example of an input mask for MNIST data after training to convergence.

such inputs nor a specific function for them have been proposed. Weight-based binary masks are implicitly or explicitly
used by many catastrophic forgetting approaches, at least by Rusu et al. (2016); Fernando et al. (2017); Mallya & Lazebnik
(2017); Nguyen et al. (2017); Yoon et al. (2018). HAT is a bit different, as it learns unit-based hard attention vectors, with
possible (but not necessarily) binary values.

