CONTINUAL LEARNING WITH HYPERNETWORKS
Year: 2020

Abstract:
Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on task-conditioned hypernetworks, i.e., networks that generate the weights of a target model based on task identity. Continual learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using a simple regularizer. Besides achieving state-ofthe-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories. Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable hypernetwork weights is comparable or smaller than target network size. We provide insight into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and show that task-conditioned hypernetworks demonstrate transfer learning. Finally, forward information transfer is further supported by empirical results on a challenging CL benchmark based on the CIFAR-10/100 image datasets.

heading: 1 INTRODUCTION
text: We assume that a neural network f(x,Θ) with trainable weights Θ is given data from a set of tasks {(X(1),Y(1)), . . . , (X(T ),Y(T ))}, with input samples X(t) = {x(t,i)}nti=1 and output samples Y(t) = {y(t,i)}nti=1, where nt ≡ |X(t)|. A standard training approach learns the model using data from all tasks at once. However, this is not always possible in real-world problems, nor desirable in an online learning setting. Continual learning (CL) refers to an online learning setup in which tasks are presented sequentially (see van de Ven & Tolias, 2019, for a recent review on CL). In CL, when learning a new task t, starting with weights Θ(t−1) and observing only (X(t),Y(t)), the goal is to find a new set of parameters Θ(t) that (1) retains (no catastrophic forgetting) or (2) improves (positive backward transfer) performance on previous tasks compared to Θ(t−1) and (3) solves the new task t potentially utilizing previously acquired knowledge (positive forward transfer). Achieving these goals is non-trivial, and a longstanding issue in neural networks research.
Here, we propose addressing catastrophic forgetting at the meta level: instead of directly attempting to retain f(x,Θ) for previous tasks, we fix the outputs of a metamodel fh(e,Θh) termed task-conditioned hypernetwork which maps a task embedding e to weights Θ. Now, a single point has to be memorized per task. To motivate such approach, we perform a thought experiment: we assume that we are allowed to store all inputs {X(1), . . . ,X(T )} seen so far, and to use these data to compute model outputs corresponding to Θ(T−1). In this idealized setting, one can avoid forgetting by simply mixing data from the current task with data from the past, {(X(1), Ŷ(1)), . . . , (X(T−1), Ŷ(T−1)), (X(T ),Y(T ))}, where Ŷ(t) refers to a set of synthetic targets generated using the model itself f( · ,Θ(t−1)). Hence, by training to retain previously acquired input-output mappings, one can obtain a sequential algorithm in principle as powerful as multi-task learning. Multi-task learning, where all tasks are learned
simultaneously, can be seen as a CL upper-bound. The strategy described above has been termed rehearsal (Robins, 1995). However, storing previous task data violates our CL desiderata.
Therefore, we introduce a change in perspective and move from the challenge of maintaining individual input-output data points to the problem of maintaining sets of parameters {Θ(t)}, without explicitly storing them. To achieve this, we train the metamodel parameters Θh analogous to the above outlined learning scheme, where synthetic targets now correspond to weight configurations that are suitable for previous tasks. This exchanges the storage of an entire dataset by a single low-dimensional task descriptor, yielding a massive memory saving in all but the simplest of tasks. Despite relying on regularization, our approach is a conceptual departure from previous algorithms based on regularization in weight (e.g., Kirkpatrick et al., 2017; Zenke et al., 2017) or activation space (e.g., He & Jaeger, 2018).
Our experimental results show that task-conditioned hypernetworks do not suffer from catastrophic forgetting on a set of standard CL benchmarks. Remarkably, they are capable of retaining memories with practically no decrease in performance, when presented with very long sequences of tasks. Thanks to the expressive power of neural networks, task-conditioned hypernetworks exploit task-totask similarities and transfer information forward in time to future tasks. Finally, the task-conditional metamodelling perspective that we put forth is generic, as it does not depend on the specifics of the target network architecture. We exploit this key principle and show that the very same metamodelling framework extends to, and can improve, an important class of CL methods known as generative replay methods, which are current state-of-the-art performers in many practical problems (Shin et al., 2017; Wu et al., 2018; van de Ven & Tolias, 2018).

heading: 2 MODEL

heading: 2.1 TASK-CONDITIONED HYPERNETWORKS
text: Hypernetworks parameterize target models. The centerpiece of our approach to continual learning is the hypernetwork, Fig. 1a. Instead of learning the parameters Θtrgt of a particular function ftrgt directly (the target model), we learn the parameters Θh of a metamodel. The output of such metamodel, the hypernetwork, is Θtrgt. Hypernetworks can therefore be thought of as weight generators, which were originally introduced to dynamically parameterize models in a compressed form (Ha et al., 2017; Schmidhuber, 1992; Bertinetto et al., 2016; Jia et al., 2016).
Continual learning with hypernetwork output regularization. One approach to avoid catastrophic forgetting is to store data from previous tasks and corresponding model outputs, and then fix such outputs. This can be achieved using an output regularizer of the following form, where past outputs play the role of pseudo-targets (Robins, 1995; Li & Hoiem, 2018; Benjamin et al., 2018):
Loutput = T−1∑ t=1 |X(t)|∑ i=1 ‖f(x(t,i),Θ∗)− f(x(t,i),Θ)‖2, (1)
In the equation above, Θ∗ is the set of parameters before attempting to learn task T , and f is the learner. This approach, however, requires storing and iterating over previous data, a process that is known as rehearsing. This is potentially expensive memory-wise and not strictly online learning. A possible workaround is to generate the pseudo-targets by evaluating f on random patterns (Robins, 1995) or on the current task dataset (Li & Hoiem, 2018). However, this does not necessarily fix the behavior of the function f in the regions of interest.
Hypernetworks sidestep this problem naturally. In target network weight space, a single point (i.e., one set of weights) has to be fixed per task. This can be efficiently achieved with task-conditioned hypernetworks, by fixing the hypernetwork output on the appropriate task embedding.
Similar to Benjamin et al. (2018), we use a two-step optimization procedure to introduce memorypreserving hypernetwork output constraints. First, we compute a candidate change ∆Θh which minimizes the current task loss L(T )task = Ltask(Θh, e(T ),X(T ),Y(T )) with respect to Θ. The candidate ∆Θh is obtained with an optimizer of choice (we use Adam throughout; Kingma & Ba, 2015). The actual parameter change is then computed by minimizing the following total loss:
Ltotal = Ltask(Θh, e(T ),X(T ),Y(T )) + Loutput(Θ∗h ,Θh,∆Θh, {e(t)})
= Ltask(Θh, e(T ),X(T ),Y(T )) + βoutput T − 1 T−1∑ t=1 ‖fh(e(t),Θ∗h)− fh(e(t),Θh + ∆Θh))‖2, (2)
where Θ∗h is the set of hypernetwork parameters before attempting to learn task T , ∆Θh is considered fixed and βoutput is a hyperparameter that controls the strength of the regularizer. On Appendix D, we run a sensitivity analysis on βoutput and experiment with a more efficient stochastic regularizer where the averaging is performed over a random subset of past tasks.
More computationally-intensive algorithms that involve a full inner-loop refinement, or use secondorder gradient information by backpropagating through ∆Θh could be applied. However, we found empirically that our one-step correction worked well. Exploratory hyperparameter scans revealed that the inclusion of the lookahead ∆Θh in (2) brought a minor increase in performance, even when computed with a cheap one-step procedure. Note that unlike in Eq. 1, the memory-preserving term Loutput does not depend on past data. Memory of previous tasks enters only through the collection of task embeddings {e(t)}T−1t=1 .
Learned task embeddings. Task embeddings are differentiable deterministic parameters that can be learned, just like Θh. At every learning step of our algorithm, we also update the current task embedding e(T ) to minimize the task loss L(T )task . After learning the task, the final embedding is saved and added to the collection {e(t)}.

heading: 2.2 MODEL COMPRESSION WITH CHUNKED HYPERNETWORKS
text: Chunking. In a straightforward implementation, a hypernetwork produces the entire set of weights of a target neural network. For modern deep neural networks, this is a very high-dimensional output. However, hypernetworks can be invoked iteratively, filling in only part of the target model at each step, in chunks (Ha et al., 2017; Pawlowski et al., 2017). This strategy allows applying smaller hypernetworks that are reusable. Interestingly, with chunked hypernetworks it is possible to solve tasks in a compressive regime, where the number of learned parameters (those of the hypernetwork) is effectively smaller than the number of target network parameters.
Chunk embeddings and network partitioning. Reapplying the same hypernetwork multiple times introduces weight sharing across partitions of the target network, which is usually not desirable.
To allow for a flexible parameterization of the target network, we introduce a set C = {ci}NCi=1 of chunk embeddings, which are used as an additional input to the hypernetwork, Fig. 1b. Thus, the full set of target network parameters Θtrgt = [fh(e, c1), . . . , fh(e, cNC)] is produced by iteration over C, keeping the task embedding e fixed. This way, the hypernetwork can produce distinct weights for each chunk. Furthermore, chunk embeddings, just like task embeddings, are ordinary deterministic parameters that we learn via backpropagation. For simplicity, we use a shared set of chunk embeddings for all tasks and we do not explore special target network partitioning strategies.
How flexible is our approach? Chunked neural networks can in principle approximate any target weight configuration arbitrarily well. For completeness, we state this formally in Appendix E.

heading: 2.3 CONTEXT-FREE INFERENCE: UNKNOWN TASK IDENTITY
text: Determining which task to solve from input data. Our hypernetwork requires a task embedding input to generate target model weights. In certain CL applications, an appropriate embedding can be immediately selected as task identity is unambiguous, or can be readily inferred from contextual clues. In other cases, knowledge of the task at hand is not explicitly available during inference. In the following, we show that our metamodelling framework generalizes to such situations. In particular, we consider the problem of inferring which task to solve from a given input pattern, a noted benchmark challenge (Farquhar & Gal, 2018; van de Ven & Tolias, 2019). Below, we explore two different strategies that leverage task-conditioned hypernetworks in this CL setting.
Task-dependent predictive uncertainty. Neural network models are increasingly reliable in signalling novelty and appropriately handling out-of-distribution data. For categorical target distributions, the network ideally produces a flat, high entropy output for unseen data and, conversely, a peaked, low-entropy response for in-distribution data (Hendrycks & Gimpel, 2016; Liang et al., 2017). This suggests a first, simple method for task inference (HNET+ENT). Given an input pattern for which task identity is unknown, we pick the task embedding which yields lowest predictive uncertainty, as quantified by output distribution entropy. While this method relies on accurate novelty detection, which is in itself a far from solved research problem, it is otherwise straightforward to implement and no additional learning or model is required to infer task identity.
Hypernetwork-protected synthetic replay. When a generative model is available, catastrophic forgetting can be circumvented by mixing current task data with replayed past synthetic data (for recent work see Shin et al., 2017; Wu et al., 2018). Besides protecting the generative model itself, synthetic data can protect another model of interest, for example, another discriminative model. This conceptually simple strategy is in practice often the state-of-the-art solution to CL (van de Ven & Tolias, 2019). Inspired by these successes, we explore augmenting our system with a replay network, here a standard variational autoencoder (VAE; Kingma & Welling, 2014) (but see Appendix F for experiments with a generative adversarial network, Goodfellow et al., 2014).
Synthetic replay is a strong, but not perfect, CL mechanism as the generative model is subject to drift, and errors tend to accumulate and amplify with time. Here, we build upon the following key observation: just like the target network, the generator of the replay model can be specified by a hypernetwork. This allows protecting it with the output regularizer, Eq. 2, rather than with the model’s own replay data, as done in related work. Thus, in this combined approach, both synthetic replay and task-conditional metamodelling act in tandem to reduce forgetting.
We explore hypernetwork-protected replay in two distinct setups. First, we consider a minimalist architecture (HNET+R), where only the replay model, and not the target classifier, is parameterized by a hypernetwork. Here, forgetting in the target network is obviated by mixing current data with synthetic data. Synthetic target output values for previous tasks are generated using a soft targets method, i.e., by simply evaluating the target function before learning the new task on synthetic input data. Second (HNET+TIR), we introduce an auxiliary task inference classifier, protected using synthetic replay data and trained to predict task identity from input patterns. This architecture requires additional modelling, but it is likely to work well when tasks are strongly dissimilar. Furthermore, the task inference subsystem can be readily applied to process more general forms of contextual information, beyond the current input pattern. We provide additional details, including network architectures and the loss functions that are optimized, in Appendices B and C.

heading: 3 RESULTS
text: We evaluate our method on a set of standard image classification benchmarks on the MNIST, CIFAR10 and CIFAR-100 public datasets1. Our main aims are to (1) study the memory retention capabilities of task-conditioned hypernetworks across three continual learning settings, and (2) investigate information transfer across tasks that are learned sequentially.
Continual learning scenarios. In our experiments we consider three different CL scenarios (van de Ven & Tolias, 2019). In CL1, the task identity is given to the system. This is arguably the standard sequential learning scenario, and the one we consider unless noted otherwise. In CL2, task identity is unknown to the system, but it does not need to be explicitly determined. A target network with a fixed head is required to solve multiple tasks. In CL3, task identity has to be explicitly inferred. It has been argued that this scenario is the most natural, and the one that tends to be harder for neural networks (Farquhar & Gal, 2018; van de Ven & Tolias, 2019).
Experimental details. Aiming at comparability, for the experiments on the MNIST dataset we model the target network as a fully-connected network and set all hyperparameters after van de Ven & Tolias (2019), who recently reviewed and compared a large set of CL algorithms. For our CIFAR experiments, we opt for a ResNet-32 target neural network (He et al., 2016) to assess the scalability of our method. A summary description of the architectures and particular hyperparameter choices, as well as additional experimental details, is provided in Appendix C. We emphasize that, on all our experiments, the number of hypernetwork parameters is always smaller or equal than the number of parameters of the models we compare with.
Nonlinear regression toy problem. To illustrate our approach, we first consider a simple nonlinear regression problem, where the function to be approximated is scalar-valued, Fig. 2. Here, a sequence of polynomial functions of increasing degree has to be inferred from noisy data. This motivates the continual learning problem: when learning each task in succession by modifying Θh with the memory-preserving regularizer turned off (βoutput = 0, see Eq. 2) the network learns the last task but forgets previous ones, Fig. 2c. The regularizer protects old solutions, Fig. 2a, and performance is comparable to an offline non-continual learner, Fig. 2b.
Permuted MNIST benchmark. Next, we study the permuted MNIST benchmark. This problem is set as follows. First, the learner is presented with the full MNIST dataset. Subsequently, novel tasks are obtained by applying a random permutation to the input image pixels. This process can be repeated to yield a long task sequence, with a typical length of T = 10 tasks. Given the low similarity of the generated tasks, permuted MNIST is well suited to study the memory capacity of a continual learner. For T = 10, we find that task-conditioned hypernetworks are state-of-the-art on CL1, Table 1. Interestingly, inferring tasks through the predictive distribution entropy (HNET+ENT) works well on the permuted MNIST benchmark. Despite the simplicity of the method, both synaptic intelligence (SI; Zenke et al., 2017) and online elastic weight consolidation (EWC; Schwarz et al., 2018) are overperformed on CL3 by a large margin. When complemented with generative replay
1Source code is available under https://github.com/chrhenning/hypercl.
a
b
|Θtrgt| versus task-averaged test set accuracy
after learning all tasks (labelled ‘final’, in red) and immediately after learning a task (labelled ‘during’, in purple) for the PermutedMNIST-10 benchmark. Hypernetworks allow for model compression and perform well even when the number of target model parameters exceeds their own. Performance decays nonlinearly: accuracies stay approximately constant for a wide range of compression ratios below unity. Hyperparameters were tuned once for compression ratio ≈ 1 and were then used for all compression ratios. Shaded areas denote STD (a) resp. SEM (b) across 5 random seeds.
methods, task-conditioned hypernetworks (HNET+TIR and HNET+R) are the best performers on all three CL scenarios.
Performance differences become larger in the long sequence limit, Fig. 3a. For longer task sequences (T = 100), SI and DGR+distill (Shin et al., 2017; van de Ven & Tolias, 2018) degrade gracefully, while the regularization strength of online EWC prevents the method from achieving high accuracy (see Fig. A6 for a hyperparameter search on related work). Notably, task-conditioned hypernetworks show minimal memory decay and find high performance solutions. Because the hypernetwork operates in a compressive regime (see Fig. 3b and Fig. A7 for an exploration of compression ratios), our results do not naively rely on an increase in the number of parameters. Rather, they suggest that previous methods are not yet capable of making full use of target model capacity in a CL setting. We report a set of extended results on this benchmark on Appendix D, including a study of CL2/3 (T = 100), where HNET+TIR strongly outperforms the related work.
Split MNIST benchmark. Split MNIST is another popular CL benchmark, designed to introduce task overlap. In this problem, the various digits are sequentially paired and used to form five binary classification tasks. Here, we find that task-conditioned hypernetworks are the best overall performers. In particular, HNET+R improves the previous state-of-the-art method DGR+distill on both CL2 and CL3, almost saturating the CL2 upper bound for replay models (Appendix D). Since HNET+R is essentially hypernetwork-protected DGR, these results demonstrate the generality of task-conditioned hypernetworks as effective memory protectors. To further support this, in Appendix F we show that our replay models (we experiment with both a VAE and a GAN) can learn in a class-incremental manner the full MNIST dataset. Finally, HNET+ENT again outperforms both EWC and SI, without any generative modelling.
On the split MNIST problem, tasks overlap and therefore continual learners can transfer information across tasks. To analyze such effects, we study task-conditioned hypernetworks with two-dimensional task embedding spaces, which can be easily visualized. Despite learning happening continually, we
find that the algorithm converges to a hypernetwork configuration that can produce target model parameters that simultaneously solve old and new tasks, Fig. 4, given the appropriate task embedding.
Split CIFAR-10/100 benchmark. Finally, we study a more challenging benchmark, where the learner is first asked to solve the full CIFAR-10 classification task and is then presented with sets of ten classes from the CIFAR-100 dataset. We perform experiments both with a high-performance ResNet-32 target network architecture (Fig. 5) and with a shallower model (Fig. A3) that we exactly reproduced from previous work (Zenke et al., 2017). Remarkably, on the ResNet-32 model, we find that task-conditioned hypernetworks essentially eliminate altogether forgetting. Furthermore, forward information transfer takes place; knowledge from previous tasks allows the network to find better solutions than when learning each task individually from initial conditions. Interestingly, forward transfer is stronger on the shallow model experiments (Fig. A3), where we otherwise find that our method performs comparably to SI.

heading: 4 DISCUSSION
text: Bayesian accounts of continual learning. According to the standard Bayesian CL perspective, a posterior parameter distribution is recursively updated using Bayes’ rule as tasks arrive
CIFAR-10 1 2 3 4 5 Task t
0
25
50
75
100
A cc
ur ac
y [%
]
88 83 79 82 81 82
Split CIFAR-10/100
hnet during hnet from scratch fine-tuning
Figure 5: Split CIFAR-10/100 CL benchmark. Test set accuracies (mean ± STD, n = 5) on the entire CIFAR10 dataset and subsequent CIFAR-100 splits of ten classes. Our hypernetworkprotected ResNet-32 displays virtually no forgetting; final averaged performance (hnet, in red) matches the immediate one (hnet-during, in blue). Furthermore, information is transferred across tasks, as performance is higher than when training each task from scratch (purple). Disabling our regularizer leads to strong forgetting (in yellow).
(Kirkpatrick et al., 2017; Huszár, 2018; Nguyen et al., 2018). While this approach is theoretically sound, in practice, the approximate inference methods that are typically preferred can lead to stiff models, as a compromise solution that suits all tasks has to be found within the mode determined by the first task. Such restriction does not apply to hypernetworks, which can in principle model complex multimodal distributions (Louizos & Welling, 2017; Pawlowski et al., 2017; Henning et al., 2018). Thus, rich, hypernetwork-modelled priors are one avenue of improvement for Bayesian CL methods. Interestingly, task-conditioning offers an alternative possibility: instead of consolidating every task onto a single distribution, a shared task-conditioned hypernetwork could be leveraged to model a set of parameter posterior distributions. This conditional metamodel naturally extends our framework to the Bayesian learning setting. Such approach will likely benefit from additional flexibility, compared to conventional recursive Bayesian updating.
Related approaches that rely on task-conditioning. Our model fits within, and in certain ways generalizes, previous CL methods that condition network computation on task descriptors. Taskconditioning is commonly implemented using multiplicative masks at the level of modules (Rusu et al., 2016; Fernando et al., 2017), neurons (Serra et al., 2018; Masse et al., 2018) or weights (Mallya & Lazebnik, 2018). Such methods work best with large networks and come with a significant storage overhead, which typically scales with the number of tasks. Our approach differs by explicitly modelling the full parameter space using a metamodel, the hypernetwork. Thanks to this metamodel, generalization in parameter and task space is possible, and task-to-task dependencies can be exploited to efficiently represent solutions and transfer present knowledge to future problems. Interestingly, similar arguments have been drawn in work developed concurrently to ours (Lampinen & McClelland, 2019), where task embedding spaces are further explored in the context of few-shot learning. In the same vein, and like the approach developed here, recent work in CL generates last-layer network parameters as part of a pipeline to avoid catastrophic forgetting (Hu et al., 2019) or distills parameters onto a contractive auto-encoding model (Camp et al., 2018).
Positive backwards transfer. In its current form, the hypernetwork output regularizer protects previously learned solutions from changing, such that only weak backwards transfer of information can occur. Given the role of selective forgetting and refinement of past memories in achieving intelligent behavior (Brea et al., 2014; Richards & Frankland, 2017), investigating and improving backwards transfer stands as an important direction for future research.
Relevance to systems neuroscience. Uncovering the mechanisms that support continual learning in both brains and artificial neural networks is a long-standing question (McCloskey & Cohen, 1989; French, 1999; Parisi et al., 2019). We close with a speculative systems interpretation (Kumaran et al., 2016; Hassabis et al., 2017) of our work as a model for modulatory top-down signals in cortex. Task embeddings can be seen as low-dimensional context switches, which determine the behavior of a modulatory system, the hypernetwork in our case. According to our model, the hypernetwork would in turn regulate the activity of a target cortical network.
As it stands, implementing a hypernetwork would entail dynamically changing the entire connectivity of a target network, or cortical area. Such a process seems difficult to conceive in the brain. However, this strict literal interpretation can be relaxed. For example, a hypernetwork can output lowerdimensional modulatory signals (Marder, 2012), instead of a full set of weights. This interpretation
is consistent with a growing body of work which suggests the involvement of modulatory inputs in implementing context- or task-dependent network mode-switching (Mante et al., 2013; Jaeger, 2014; Stroud et al., 2018; Masse et al., 2018).

heading: 5 CONCLUSION
text: We introduced a novel neural network model, the task-conditioned hypernetwork, that is well-suited for CL problems. A task-conditioned hypernetwork is a metamodel that learns to parameterize target functions, that are specified and identified in a compressed form using a task embedding vector. Past tasks are kept in memory using a hypernetwork output regularizer, which penalizes changes in previously found target weight configurations. This approach is scalable and generic, being applicable as a standalone CL method or in combination with generative replay. Our results are state-of-the-art on standard benchmarks and suggest that task-conditioned hypernetworks can achieve long memory lifetimes, as well as transfer information to future tasks, two essential properties of a continual learner.

heading: ACKNOWLEDGMENTS
text: This work was supported by the Swiss National Science Foundation (B.F.G. CRSII5-173721), ETH project funding (B.F.G. ETH-20 19-01) and funding from the Swiss Data Science Center (B.F.G, C17-18, J. v. O. P18-03). Special thanks to Simone Carlo Surace, Adrian Huber, Xu He, Markus Marks, Maria R. Cervera and Jannes Jegminat for discussions, helpful pointers to the CL literature and for feedback on our paper draft.

heading: A TASK-CONDITIONED HYPERNETWORKS: MODEL SUMMARY
text: In our model, a task-conditioned hypernetwork produces the parameters Θtrgt = fh(e,Θh) of a target neural network. Given one such parameterization, the target model then computes predictions ŷ = ftrgt(x,Θtrgt) based on input data. Learning amounts to adapting the parameters Θh of the hypernetwork, including a set of task embeddings {e(t)}Tt=1, as well as a set of chunk embeddings {ci}NCi=1 in case compression is sought or if the full hypernetwork is too large to be handled directly. To avoid castastrophic forgetting, we introduce an output regularizer which fixes the behavior of the hypernetwork by penalizing changes in target model parameters that are produced for previously learned tasks.
Variables that need to be stored while learning new tasks. What are the storage requirements of our model, when learning continually?
1. Memory retention relies on saving one embedding per task. This collection {e(t)}Tt=1 therefore grows linearly with T . Such linear scaling is undesirable asymptotically, but it turns out to be essentially negligible in practice, as each embedding is a single lowdimensional vector (e.g., see Fig. 4 for a run with 2D embeddings).
2. A frozen snapshot of the hypernetwork parameters Θ∗h , taken before learning a new task, needs to be kept, to evaluate the output regularizer in Eq. 2.

heading: B ADDITIONAL DETAILS ON HYPERNETWORK-PROTECTED REPLAY MODELS
text: Variational autoencoders. For all HNET+TIR and HNET+R experiments reported on the main text we use VAEs as our replay models (Fig. A1a, Kingma & Welling, 2014). Briefly, a VAE consists of an encoder-decoder network pair, where the encoder network processes some input pattern x and its outputs fenc(x) = (µ,σ2) comprise the parameters µ and σ2 (encoded in log domain, to enforce nonnegativity) of a diagonal multivariate Gaussian pZ(z;µ,σ2), which governs the distribution of latent samples z. On the other side of the circuit, the decoder network processes a latent sample z and a one-hot-encoded task identity vector and returns an input pattern reconstruction, fdec(z,1t) = x̂.
VAEs can preserve memories using a technique called generative replay: when training task T , input samples are generated from the current replay network for old tasks t < T , by varying 1t and drawing latent space samples z. Generated data can be mixed with the current dataset, yielding an augmented dataset X̃ used to relearn model parameters. When protecting a discriminative model, synthetic ‘soft’ targets can be generated by evaluating the network on X̃ . We use this strategy to protect an auxiliary task inference classifier in HNET+TIR, and to protect the main target model in HNET+R.
Hypernetwork-protected replay. In our HNET+TIR and HNET+R experiments, we parameterize the decoder network through a task-conditioned hypernetwork, fh,dec(e,Θh,dec). In combination with our output regularizer, this allows us to take advantage of the memory retention capacity of hypernetworks, now on a generative model.
The replay model (encoder, decoder and decoder hypernetwork) is a separate subsystem that is optimized independently from the target network. Its parameters Θenc and Θh,dec are learned by minimizing our regularized loss function, Eq. 2, here with the task-specific term set to the standard VAE objective function,
LVAE ( X,Θenc,Θh,dec ) = Lrec(X,Θenc,Θdec) + Lprior(X,Θenc,Θdec), (3)
with Θdec = fh,dec(e,Θh,dec) introducing the dependence on Θh,dec. LVAE balances a reconstruction Lrec and a prior-matching Lprior penalties. For our MNIST experiments, we choose binary crossentropy (in pixel space) as the reconstruction loss, that we write below for a single example x
Lrec(x,Θenc,Θdec) = Lxent ( x, fdec ( z,1t(x),Θdec )) , (4)
where Lxent(t, y) = − ∑ k tk log yk is the cross entropy. For a diagonal Gaussian pZ , the priormatching term can be evaluated analytically,
Lprior = − 1
2 |z|∑ i=1 ( 1 + log σ2i − σ2i − µ2i ) . (5)
Above, z is a sample from pZ(z;µ(x̃),σ2(x̃)) obtained via the reparameterization trick (Kingma & Welling, 2014; Rezende et al., 2014). This introduces the dependency of Lrec on Θenc.
Task inference network (HNET+TIR). In the HNET+TIR setup, we extend our system to include a task inference neural network classifier α(x) parameterized by ΘTI, where tasks are encoded with a T -dimensional softmax output layer. In both CL2 and CL3 scenarios we use a growing single-head setup for α, and increase the dimensionality of the softmax layer as tasks arrive.
This network is prone to catastrophic forgetting when tasks are learned continually. To prevent this from happening we resort to replay data generated from a hypernetwork-protected VAE, described above. More specifically, we introduce a task inference loss,
LTI(x̃,ΘTI) = Lxent(1t(x̃),α(x̃,Θenc)), (6)
where t(x̃) denotes the correct task identity for a sample x̃ from the augmented dataset X̃ = {X̃(1), . . . X̃(T−1), X̃(T )} with X̃(t) being synthetic data fdec(z,1t,Θdec) for t = 1 . . . T −1 and X̃(T ) = X(T ) is the current task data. Importantly, synthetic data is essential to obtain a well defined objective function for task inference; the cross-entropy loss LTI requires at least two groundtruth classes to be optimized. Note that replayed data can be generated online by drawing samples z from the prior.
a
fdec
fenc
z task id
x
x~ (t) fh,dec
(t)
θdec
b
fgen
fdisc
x~ (t) fh,gen
θgen
z task id c
α
e
fTI x, {x }(T) T-1t=1 (t)~
Figure A1: Hypernetwork-protected replay model setups. (a) A hypernetwork-protected VAE, that we used for HNET+R and HNET+TIR main text experiments. (b) A hypernetwork-protected GAN, that we used for our class-incremental learning Appendix F experiments. (c) A task inference classifier protected with synthetic replay data, used on HNET+TIR experiments.
Hypernetwork-protected GANs. Generative adversarial networks (Goodfellow et al., 2014) have become an established method for generative modelling and tend to produce higher quality images compared to VAEs, even at the scale of datasets as complex as ImageNet (Brock et al., 2019; Lučić et al., 2019; Donahue & Simonyan, 2019). This makes GANs perfect candidates for powerful replay models. A suitable GAN instantiation for CL is the conditional GAN (Mirza & Osindero, 2014) as studied by Wu et al. (2018). Recent developments in the GAN literature already allude towards the potential of using hypernetwork-like structures, e.g., when injecting the latent noise (Karras et al., 2019) or when using class-conditional batch-normalization as in (Brock et al., 2019). We propose to go one step further and use a hypernetwork that maps the condition to the full set of generator parameters Θ∗gen. Our framework allows training a conditional GAN one condition at the time. This is potentially of general interest, and goes beyond the scope of replay models, since conditional GANs trained in a mutli-task fashion as in Brock et al. (2019) require very large computational resources.
For our showcase experiment on class-incremental MNIST learning, Fig. A8, we did not aim to compare to related work and therefore did not tune to have less weights in the hypernetwork than on the target network (for the VAE experiments, we use the same compressive setup as in the main text, see Appendix C). The GAN hypernetwork is a fully-connected chunked hypernetwork with 2 hidden layers of size 25 and 25 followed by an output size of 75,000. We used learning rates for both discriminator and the generator hypernetwork of 0.0001, as well as dropout of 0.4 in the discriminator and the system is trained for 10000 iterations per task. We use the Pearson Chi2 Least-Squares GAN loss from Mao et al. (2017) in our experiments.

heading: C ADDITIONAL EXPERIMENTAL DETAILS
text: All experiments are conducted using 16 NVIDIA GeForce RTX 2080 TI graphics cards.
For simplicity, we decided to always keep the previous task embeddings e(t), t = 1, . . . , T − 1, fixed and only learn the current task embedding e(T ). In general, performance should be improved if the
regularizer in Eq. 2 has a separate copy of the task embeddings e(t,∗) from before learning the current task, such that e(t) can be adapted. Hence, the targets become fh(e(t,∗),Θ∗h) and remain constant while learning task T . This would give the hypernetwork the flexibility to adjust the embeddings i.e. the preimage of the targets and therefore represent any function that includes all desired targets in its image.
Nonlinear regression toy problem. The nonlinear toy regression from Fig. 2 is an illustrative example for a continual learning problem where a set of ground-truth functions {g(1), . . . , g(T )} is given from which we collect 100 noisy training samples per task {(x,y) | y = g(t)(x) + with ∼ N (0, σ2I),x ∼ U(X (t))}, where X (t) denotes the input domain of task t. We set σ = 0.05 in this experiment.
We perform 1D regression and choose the following set of tasks:
g(1)(x) = x+ 3 X (1) = [−4,−2] (7) g(2)(x) = 2x2 − 1 X (2) = [−1, 1] (8) g(3)(x) = (x− 3)3 X (3) = [2, 4] (9)
The target network ftrgt consists of two fully-connected hidden layers using 10 neurons each. For illustrative purposes we use a full hypernetwork fh that generates all 141 weights of ftrgt at once, also being a fully-connected network with two hidden-layers of size 10. Hence, this is the only setup where we did not explore the possibility of a chunked hypernetwork. We use sigmoid activation functions in both networks. The task embedding dimension was set to 2.
We train each task for 4000 iterations using the Adam optimizer with a learning rate of 0.01 (and otherwise default PyTorch options) and a batch size of 32.
To test our regularizer in Fig. 2a we set βoutput to 0.005, while it is set to 0 for the fine-tuning experiment in Fig. 2c.
For the multi-task learner in Fig. 2b we trained only the target network (no hypernetwork) for 12000 iterations with a learning rate of 0.05. Comparable performance could be obtained when training the task-conditioned hypernetwork in this multi-task regime (data not shown).
It is worth noting that the multi-task learner from Fig. 2b that uses no hypernetwork is only able to learn the task since we choose the input domains to be non-overlapping.
Permuted MNIST benchmark. For our experiments conducted on MNIST we replicated the experimental setup proposed by van de Ven & Tolias (2019) whenever applicable. We therefore use the same number of training iterations, the same or a lower number of weights in the hypernetwork than in the target network, the same learning rates and the same optimizer. For the replay model, i.e., the hypernetwork-empowered VAE, as well as for the standard classifier we used 5000 training iterations per task and learning rate is set to 0.0001 for the Adam optimizer (otherwise PyTorch default values). The batchsize is set to 128 for the VAE whereas the classifier is simultaneously trained on a batch of 128 samples of replayed data (evenly distributed over all past tasks) and a batch of 128 images from the currently available dataset. MNIST images are padded with zeros, which results in network inputs of size 32× 32, again strictly following the implementation of the compared work. We experienced better performance when we condition our replay model on a specific task input. We therefore construct for every task a specific input namely a sample from a standard multivariate normal of dimension 100. In practice we found the dimension to be not important. This input stays constant throughout the experiment and is not learned. Note that we use the same hyperparameters for all learning scenarios, which is not true for the reported related work since they have tuned special hyperparameters for all scenarios and all methods.
• Details of hypernetwork for the VAE. We use one hypernetwork configuration to generate weights for all variational autoencoders used for our PermutedMNIST-10 experiments namely a fully-connected chunked hypernetwork with 2 hidden layers of size 25 and 25 followed by an output size of 85,000. We use ELU nonlinearities in the hidden layers
a
1 50 100 Task t
50
75
100
A cc
ur ac
y [%
]
PermutedMNIST-100
βoutput = 1 βoutput = 0.5 βoutput = 0.1
βoutput = 0.05 βoutput = 0.01 βoutput = 0.005
βoutput = 0.001 βoutput = 0.0005
b
1 50 100 Task t
80
90
100
A cc
ur ac
y [%
]
96.78% 95.66%
PermutedMNIST-100
(12 + 128)→ 200→ 250→ 350→ 7500, βoutput = 0.01 (12 + 128)→ 200→ 250→ 300→ 6000, βoutput = 0.01
c
0.001 0.01 0.1 1 0.0005 0.005 0.05 0.5
βoutput
80
90
100
A cc
ur ac
y [%
]
PermutedMNIST-100
during final
Figure A2: Additional experiments on the PermutedMNIST-100 benchmark. (a) Final test set classification accuracy on the t-th task after learning one hundred permutations (PermutedMNIST100). All runs use exactly the same hyperparameter configuration except for varying values of βoutput. The final accuracies are robust for a wide range of regularization strengths. If βoutput is too weak, forgetting will occur. However, there is no severe disadvantage of choosing βoutput too high (cmp. (c)). A too high βoutput simply shifts the attention of the optimizer away from the current task, leading to lower baseline accuracies when the training time is not increased. (b) Due to an increased number of output neurons, the target network for PermutedMNIST-100 has more weights than for PermutedMNIST-10 (this is only the case for CL1 and CL3). This plot shows that the performance drop is minor when choosing a hypernetwork with a comparable number of weights as the target network in CL2 (orange) compared to one that has a similar number of weights as the target network for CL1 in PermutedMNIST-100 (red). (c) Task-averaged test set accuracy after learning all tasks (labelled ‘final’, in red) and immediately after learning a task (labelled ‘during’, in purple) for the runs depicted in (a). For low values of βoutput final accuracies are worse than immediate once (forgetting occurs). If βoutput is too high, baseline accuracies decrease since the optimizer puts less emphasis on the current task (note that the training time per task is not increased). Shaded areas in (a) and (b) denote STD, whereas error bars in (c) denote SEM (always across 5 random seeds).
of the hypernetwork. The size of task embeddings e has been set to 24 and the size of chunk embeddings c to 8. The parameter βoutput is 0.05 . The number of weights in this hypernetwork is 2,211,907 (2,211,691 network weights + 216 task embedding weights). The corresponding target network (and therefore output of the chunked hypernetwork), as taken from related work, has 2,227,024 weights. • Details of the VAE for HNET+TIR. For this variational autoencoder, we use two fully-
connected neural networks with layers of size 1000, 1000 for the encoder and 1000, 1000 for the decoder and a latent space of 100. This setup is again copied from work we compare against. • Details of the VAE for HNET+R. For this variational autoencoder, we use two fully-
connected neural networks with layers of size 400, 400 for the encoder and 400, 400 for the decoder (both 1000, 1000 in the related work) and a latent space of dimension 100. Here, we departure from related work by choosing a smaller architecture for the autoencoder. Note that we still use a hypernetwork with less trainable parameters than the target network (in this case the decoder) that is used in related work. • Details of the hypernetwork for the target classifier in PermutedMNIST-10
(HNET+TIR & HNET+ENT). We use the same setup for the hypernetwork as used for the VAEs above, but since the target network is smaller we reduce the output of the hypernetwork to 78,000. We also adjust the parameter βoutput to 0.01, consistent with our PermutedMNIST-100 experiments. The number of weights in this hypernetwork is therefore 2,029,931 parameters (2,029,691 network weights + 240 task embedding weights). The corresponding target network (from related work) would have 2,126,100 weights for CL1 and CL3 and 2,036,010 for CL2 (only one output head). • Details of the hypernetwork for the target classifier for PermutedMNIST-100. For
these experiments we chose an architecture that worked well on the PermutedMNIST-10 benchmark and did not conduct any more search for new architectures. For PermutedMNIST100, the reported results were obtained by using a chunked hypernetwork with 3 hidden layers of size 200, 250 and 350 (300 for CL2) and an output size of 7500 (6000 for CL2) (such that we approximately match the corresponding target network size for CL1/CL2/CL3). Interestingly, Fig. A2b shows that even if we don’t adjust the number of hypernetwork weights to the increased number of target network weights, the superiority of our method is evident. Aside from this, the plots in Fig. 3 have been generated using the PermutedMNIST-10 HNET+TIR setup (note that this includes the conditions set by related work for PermutedMNIST-10, e.g., target network sizes, the number of training iterations, learning rates, etc.). • Details of the VAE and the hypernetwork for the VAE in PermutedMNIST-100 for CL2/CL3. We use a very similar setup for the VAE and it’s hypernetwork used in HNET+TIR for PermutedMNIST-10 as described above. We only applied the following changes: Fully-connected hypernetwork with one hidden layer of size 100; chunk embedding sizes are set to 12; task embedding sizes are set two 128 and the hidden layer sizes of the VAE its generator are 400, 600. Also we increased the regularisation strength βoutput = 0.1 for the VAE its generator hypernetwork. • Details of the target classifier for HNET+TIR & HNET+ENT. For this classifier, we
use the same setup as in the study we compare to (van de Ven & Tolias, 2019), i.e., a fully-connected network with layers of size 1000, 1000. Note that if the classifier is used as a task inference model, it is trained on replay data and the corresponding hard targets, i.e., the argmax of the soft targets.
Below, we report the specifications for our automatic hyperparameter search (if not noted otherwise, these specifications apply for the split MNIST and split CIFAR experiments as well):
• Hidden layer sizes of the hypernetwork: (no hidden layer), "5,5" "10,10", "25,25", "50,50", "100,100", "10", "50", "100" • Output size of the hypernetwork: fitted such that we obtain less parameters then the target
network which we compare against • Embedding sizes (for e and c): 8, 12, 24, 36, 62, 96, 128 • βoutput: 0.0005, 0.001, 0.005, 0.01, 0.005, 0.1, 0.5, 1.0
• Hypernetwork transfer functions: linear, ReLU, ELU, Leaky-ReLU
Note that only a random subset of all possible combinations of hyperparameters has been explored.
After we found a configuration with promising accuracies and a similar number of weights compared to the original target network, we manually fine-tuned the architecture to increase/decrease the number of hypernetwork weights to approximately match the number of target network weights.
The choice of hypernetwork architecture seems to have a strong influence on the performance. It might be worth exploring alternatives, e.g., an architecture inspired by those used in typical generative models. We note that in addition to the above specifications we explored manually some hyperparameter configurations to gain a better understanding of our method.
Split MNIST benchmark. Again, whenever applicable we reproduce the setup from van de Ven & Tolias (2019). Differences to the PermutedMNIST-10 experiments are just the learning rate (0.001) and the number of training iterations (set to 2000).
• Details of hypernetwork for the VAE. We use one hypernetwork configuration to generate weights for all variational autoencoders used for our split MNIST experiments, namely a fully-connected chunked hypernetwork with 2 hidden layers of size 10, 10 followed by an output size of 50,000. We use ELU nonlinearities in the hidden layers of the hypernetwork. The size of task embeddings e has been set to 96 and the size of chunk embeddings c to 96. The parameter βoutput is 0.01 for HNET+R and 0.05 for HNET+TIR . The number of weights in this hypernetwork is 553,576 (553,192 network weights + 384 task embedding weights). The corresponding target network (and therefore output of the chunked hypernetwork), as taken from related work, has 555,184 weights. For a qualitative analyses of the replay data of this VAE (class incrementally learned), see A8.
• Details of the VAE for HNET+TIR. For this variational autoencoder, we use two fullyconnected neural networks with layers of size 400, 400 for the encoder and 50, 150 for the decoder (both 400, 400 in the related work) and a latent space of dimension 100.
• Details of the VAE for HNET+R. For this variational autoencoder, we use two fullyconnected neural networks with layers of size 400, 400 for the encoder and 250, 350 for the decoder (both 400, 400 in the related work) and a latent space of dimension 100.
• Details of the hypernetwork for the target classifier in split MNIST (HNET+TIR & HNET+ENT). We use the same setup for the hypernetwork as used for the VAE above, but since the target network is smaller we reduce the output of the hypernetwork to 42,000. We also adjust the βoutput to 0.01 although this parameter seems to not have a strong effect on the performance. The number of weights in this hypernetwork is therefore 465,672 parameters (465,192 network weights + 480 task embedding weights). The corresponding target network (from related work) would have 478,410 weights for CL1 and CL3 and 475,202 for CL2 (only one output head).
• Details of the target classifier for HNET+TIR & HNET+ENT. For this classifier, we again use the same setup as in the study we compare to (van de Ven & Tolias, 2019), i.e., a fully-connected neural networks with layers of size 400, 400. Note that if the classifier is used as a task inference model, it is trained on replay data and the corresponding hard targets, i.e., the argmax the soft targets.
Split CIFAR-10/100 benchmark. For these experiments, we used as a target network a ResNet-32 network (He et al. (2016)) and again produce the weights of this target network by a hypernetwork in a compressive manner. The hypernetwork in this experiment directly maps from the joint task and chunk embedding space (both dimension 32) to the output space of the hypernetwork, which is of dimension 7,000. This hypernetwork has 457,336 parameters (457,144 network weights + 192 task embedding weights). The corresponding target network, the ResNet-32, has 468.540 weights (including batch-norm weights). We train for 200 epochs per task using the Adam optimizer with an initial learning rate of 0.001 (and otherwise default PyTorch values) and a batch size of 32. In addition, we apply the two learning rate schedules suggested in the Keras CIFAR-10 example2.
2See https://keras.io/examples/cifar10_resnet/.
Due to the use of batch normalization, we have to find an appropriate way to handle the running statistics which are estimated during training. Note, these are not parameters which are trained through backpropagation. There are different ways how the running statistics could be treated:
1. One could ignore the running statistics altogether and simply compute statistics based on the current batch during evaluation.
2. The statistics could be part of the hypernetwork output. Therefore, one would have to manipulate the target hypernetwork output of the previous task, such that the estimated running statistics of the previous task will be distilled into the hypernetwork.
3. The running statistics can simply be checkpointed and stored after every task. Note, this method would lead to a linear memory growth in the number of tasks that scales with the number of units in the target network.
For simplicity, we chose the last option and simply checkpointed the running statistics after every task.
For the fine-tuning results in Fig. 5 we just continually updated the running statistics (thus, we applied no checkpointing).

heading: D ADDITIONAL EXPERIMENTS AND NOTES
text: Split CIFAR-10/100 benchmark using the model of Zenke et al. (2017). We re-run the split CIFAR-10/100 experiment reported on the main text while reproducing the setup from Zenke et al. (2017). Our overall classification performance is comparable to synaptic intelligence, which achieves 73.85% task-averaged test set accuracy, while our method reaches 71.29% ± 0.32%, with initial baseline performance being slightly worse in our approach, Fig. A3.
CIFAR-10 1 2 3 4 5 Task t
0
25
50
75
100
A cc
ur ac
y [%
]
73 69
66
74 69
7577
63 57
69
58 65
35 31
37
50 54
7877 74 70 76 73 7574 72 71 76 70 77
Split CIFAR-10/100
hnet during hnet from scratch fine-tuning SI
Figure A3: Replication of the split CIFAR-10/100 experiment of Zenke et al. (2017). Test set accuracies on the entire CIFAR-10 dataset and subsequent CIFAR-100 splits. Both task-conditioned hypernetworks (hnet, in red) and synaptic intelligence (SI, in green) transfer information forward and are protected from catastrophic forgetting. The performance of the two methods is comparable. For completeness, we report our test set accuracies achieved immediately after training (hnet-during, in blue), when training from scratch (purple), and with our regularizer turned off (fine-tuning, yellow).
To obtain our results, we use a hypernetwork with 3 hidden-layers of sizes 100, 150, 200 and output size 5500. The size of task embeddings e has been set to 48 and the size of chunk embeddings c to 80. The parameter βoutput is 0.01 and the learning rate is set to 0.0001.
The number of weights in this hypernetwork is 1,182,678 (1,182,390 network weights + 288 task embedding weights). The corresponding target network would have 1,276,508 weights.
In addition to the above specified hyperparameter search configuration we also included the following learning rates: 0.0001, 0.0005, 0.001 and manually tuned some architectural parameters.
a
1 50 100 Task t
25
50
75
100
A cc
ur ac
y [%
]
PermutedMNIST-100 - CL2
HNET+TIR SI DGR+distill Online EWC
b
1 50 100 Task t
25
50
75
100
A cc
ur ac
y [%
]
PermutedMNIST-100 - CL3
HNET+TIR SI DGR+distill Online EWC
Figure A4: Context-free inference using hypernetwork-protected replay (HNET+TIR) on long task sequences. Final test set classification accuracy on the t-th task after learning one hundred permutations of the MNIST dataset (PermutedMNIST-100) for the CL2 (a) and CL3 (b) scenarios, where task identity is not explicitly provided to the system. As before, the number of hypernetwork parameters is not larger than that of the related work we compare to. (a) HNET+TIR displays almost perfect memory retention. We used a stochastic regularizer (cf. Appendix D note below) which evaluates the output regularizer in Eq. 2 only for a random subset of previous tasks (here, twenty). (b) HNET+TIR is the only method that is capable of learning PermutedMNIST-100 in this learning scenario. For this benchmark, the input data domains are easily separable and the task inference system achieves virtually perfect (~100%) task inference accuracy throughout, even for this long experiment. HNET+TIR uses a divide-and-conquer strategy: if task inference is done right, CL3 becomes just CL1. Furthermore, once task identity is predicted, the final softmax computation only needs to consider the corresponding task outputs in isolation (here, of size 10). Curiously, for HNET+TIR, CL2 can be harder than CL3 as the single output layer (of size 10, shared by all tasks) introduces a capacity bottleneck. The related methods, on the other hand, have to consider the entire output layer (here, of size 10*100) at once, which is known to be harder to train sequentially. This leads to overwhelming error rates on long problems such as PermutedMNIST-100. Shaded areas in (a) and (b) denote STD (n = 5).
Upper bound for replay models. We obtain an upper bound for the replay-based experiments (Table 2) by sequentially training a classifier, in the same way as for HNET+R and DGR, now using true input data from past tasks and a synthetic, self-generated target. This corresponds to the rehearsal thought experiment delineated in Sect. 1.
Quantification of forgetting in our continual learning experiments. In order to quantify forgetting of our approach, we compare test set accuracies of every single task directly after training with it’s test set accuracy after training on all tasks.
Only CL1 is shown since other scenarios i.e. CL2 and CL3 depend on task inference which only is measurable after training on all tasks.
Robustness of βoutput-choice. In Fig. A2a and Fig. A2c we provide additional experiments for our method on PermutedMNIST-100. We show that our method performs comparable for a wide range of βoutput-values (including the one depicted in Fig. 3a).
a
1 50 100 Task t
25
50
75
100
A cc
ur ac
y [%
]
PermutedMNIST-100
hnet λ = 1 λ = 10
λ = 50 λ = 100 λ = 500
λ = 1000 λ = 5000 λ = 10000
b
1 25 100 500 2500 7500 hnet 10 50 250 1000 500010000
λ
60
70
80
90
100
A cc
ur ac
y [%
]
PermutedMNIST-100
during final
c
1 50 100 Task t
10
40
70
100
A cc
ur ac
y [%
]
PermutedMNIST-100
hnet hnet fine-tuning fine-tuning
d
hnet hnet fine-tuning fine-tuning 10
40
70
100
A cc
ur ac
y [%
] PermutedMNIST-100
during final
Figure A5: Additional experiments with online EWC and fine-tuning on the PermutedMNIST100 benchmark. (a) Final test set classification accuracy on the t-th task after learning one hundred permutations (PermutedMNIST-100) using the online EWC algorithm (Schwarz et al., 2018) to prevent forgetting. All runs use exactly the same hyperparameter configuration except for varying values of the regularization strength λ. Our method (hnet, in red) and the online EWC run (λ = 100, in orange) from Fig. 3a are shown for comparison. It can be seen that even when tuning the regularization strength one cannot attain similar performance as with our approach (cmp. Fig. A2a). Too strong regularization prevents the learning of new tasks whereas too weak regularization doesn’t prevent forgetting. However, a middle ground (e.g., using λ = 100) does not reach acceptable per-task performances. (b) Task-averaged test set accuracy after learning all tasks (labelled ‘final’, in red) and immediately after learning a task (labelled ‘during’, in purple) for a range of regularization strengths λ when using the online EWC algorithm. Results are complementary to those shown in (a). (c) Final test set classification accuracy on the t-th task after learning one hundred permutations (PermutedMNIST-100) when applying fine-tuning to the hypernetwork (labelled ‘hnet fine-tuning’, in blue) or target network (labelled ‘fine-tuning’, in green). Our method (hnet, in red) from Fig. 3a is shown for comparison. It can be seen that without protection the hypernetwork suffers much more severely from catastrophic forgetting as when training a target network only. (d) This plot is complementary to (c). See description of (b) for an explanation of the labels. Shaded areas in (a) and (c) denote STD, whereas error bars in (b) and (d) denote SEM (always across 5 random seeds).
a
0.005 0.05 0.5 5 50 500 5000 0.001 0.01 0.1 1 10 100 1000 10000
SI: c
50
75 100 A cc ur ac y [% ]
PermutedMNIST-100
during final
b
5 50 500 5000 1 10 100 1000 10000
Online EWC: λ
50
75
100
A cc
ur ac
y [%
]
PermutedMNIST-100
γ = 0.5 γ = 0.6
γ = 0.7 γ = 0.8
γ = 0.9 γ = 1
Figure A6: Hyperparameter search for online EWC and SI on the PermutedMNIST-100 benchmark. We conduct the same hyperparameter search as performed in van de Ven & Tolias (2018). We did not compute different random seeds for this search. (a) Hyperparameter search on the regularisation strength c for the SI algorithm. Accuracies during and after the experiment are shown. (b) Hyperparameter search for parameters λ and γ of the online EWC algorithm. Only accuracies after the experiment are shown.
Varying the regularization strength for online EWC. The performance of online EWC in Fig. 3a is closest to our method (labelled hnet, in red) compared to the other methods. Therefore, we take a closer look at this method and show that further adjustments of the regularization strength λ do not lead to better performance. Results for a wide range of regularization strengths can be seen in Fig. A5a and Fig. A5b. As shown, online EWC cannot attain a performance comparable to our method when tuning the regularization strength only.
The impact of catastrophic forgetting on the hypernetwork and target network. We have successfully shown that by shifting the continual learning problem from the target network to the hypernetwork we can successfully overcome forgetting due to the introduction of our regularizer in Eq. 2. We motivated this success by claiming that it is an inherently simpler task to remember a few input-output mappings in the hypernetwork (namely the weight realizations of each task) rather than the massive number of input-output mappings {(x(t,i),y(t,i))}nti=1 associated with the remembering of each task t by the target network.
Further evidence of this claim is provided by fine-tuning experiments in Fig. A5c and Fig. A5d. Fine-tuning refers to sequentially learning a neural network on a set of tasks without any mechanism in place to prevent forgetting. It is shown that fine-tuning a target network (no hypernetwork in this setup) has no catastrophic influence on the performance of previous tasks. Instead there is a graceful decline in performance. On the contrary, catastrophic forgetting has an almost immediate affect when training a hypernetwork without protection (i.e., training our method with βoutput = 0. The performance quickly drops to chance level, suggesting that if we weren’t solving a simpler task then preventing forgetting in the hypernetwork rather than in the target network might not be beneficial.
Chunking and hypernetwork architecture sensitivity. In this note we investigate the performance sensitivity for different (fully-connected) hypernetwork architectures on split MNIST and PermutedMNIST-10, Fig. A7. We trained thousands of randomly drawn architectures from the following grid (the same training hyperparameters as reported for for CL1, see Appendix C, were used throughout): possible number of hidden layers 1, 2, possible layer size 5, 10, 20, . . . , 90, 100, possible chunk embedding size 8, 12, 24, 56, 96 and hypernetwork output size in {10, 50, 100, 200, 300, 400, 500, 750, 1k, 2k, . . . , 9k, 10k, 20k, 30k, 40k}. Since we realize compression through chunking, we sort our hypernetwork architectures by compression ratio, and consider only architectures with small compression ratios.
Performance of split MNIST stays in the high 90 percentages even when reaching compression ratios close to 1% whereas for PermutedMNIST-10 accuracies decline in a non-linear fashion. For both experiments, the choice of the chunked hypernetwork archicture is robust and high performing even in
a
0.0 0.5 1.0 1.5 Compression ratio
0
20
40
60
80
100
A cc
ur ac
y [%
]
splitMNIST compression-performance trade-off for random hnet architectures
b
0.0 0.5 1.0 Compression ratio
0
20
40
60
80
100
A cc
ur ac
y [%
]
permutedMNIST compression-performance trade-off for random hnet architectures
Figure A7: Robustness to hypernetwork architecture choice for a large range of compression ratios. Performance vs. compression for random hypernetwork architecture choices, for split MNIST and PermutedMNIST-10 (mean ± STD, n = 500 architectures per bin). Every model was trained with the same setup (including all hyperparameters) used to obtain results reported in Table 1 (CL1). We considered architectures yielding compression ratios |Θh ∪ {e(t)}|/|Θtrgt| ∈ [0.01, 2.0] (a) split MNIST performance for CL1 stays high even for compression ratios ≈ 1%. (b) PermutedMNIST-10 accuracies degrade gracefully when compression ratios decline to 1%. Notably, for both benchmarks, performance remained stable across a large pool of hypernetwork configurations.
the compressive regime. Note that the discussed compression ratio compares the amount of trainable parameters in the hypernetwork to its output size, i.e. the parameters of the target network.
Small capacity target networks for the permuted MNIST benchmark. Swaroop et al. (2018) argue for using only small capacity target networks for this benchmark. Specifically, they propose to use hidden layer sizes [100, 100]. Again, we replicated the setup of van de Ven & Tolias (2019) wherever applicable, except for the now smaller hidden layer sizes of [100, 100] in the target network. We use a fully-connected chunked hypernetwork with chunk embeddings c having size 12, hidden layers having size 100, 75, 50 and an output size of 2000, resulting in a total number of hypernetwork weights of 122,459 (including 10 × 64 task embedding weights) compared to 122,700 weights that are generated for the target network. βoutput is set to 0.05. The experiments performed here correspond to CL1.
We achieve an average accuracy of 93.91 ± 0.04 for PermutedMNIST-10 after having trained on all tasks. In general, we saw that the hypernetwork training can benefit from noise injection. For instance, when training with soft-targets (i.e., we modified the 1-hot target to be 0.95 for the correct class and 1−0.95# classes−1 for the remaining classes), we could improve the average accuracy to 94.24 ± 0.03.
We also checked the challenging PermutedMNIST-50 benchmark with this small target network as previously investigated by Ritter et al. (2018). Therefore, we slightly adapted the above setup by using a hypernetwork with hidden layer sizes [100, 100] and a regularization strength of βoutput = 0.1. This hypernetwork is slightly bigger than the corresponding target network |Θh∪{e (t)}|
|Θtrgt| = 1.37. With this configuration, we obtain an average accuracy of 90.91 ± 0.07.
Comparison to HAT. Serra et al. (2018) proposed the hard attention to the task (HAT) algorithm, a strong CL1 method which relies on learning a per-task, per-neuron mask. Since the masks are pushed to become binary, HAT can be viewed as an algorithm for allocating subnetworks (or modules) within the target network, which become specialized to solve a given task. Thus, the method is similar to ours in the sense that the computation of the target network is task-dependent, but different in spirit, as it relies on network modularity.
In HAT, task identity is assumed to be provided, so that the appropriate mask can be picked during inference (scenario CL1). HAT requires explicitly storing a neural mask for each task, whose size
scales with the number of neurons in the target network. In contrast, our method allows solving tasks in a compressive regime. Thanks to the hypernetwork, whose input dimension can be freely chosen, only a low-dimensional embedding needs to be stored per task (cf. Fig. 4), and through chunking it is possible to learn to parameterize large target models with a small number of plastic weights (cf. Fig. 3b).
Here, we compare our task-conditioned hypernetworks to HAT on the permuted MNIST benchmarks (T = 10 and T = 100), cf. Table 6. For large target networks, both methods perform strongly, reaching comparable final task-averaged accuracies. For small target network sizes, task-conditioned hypernetworks perform better, the difference becoming more apparent on PermutedMNIST-100.
We note that the two algorithms use different training setups. In particular, HAT uses 200 epochs (batch size set to 64) and applies a learning rate scheduler that acts on a held out validation set. Furthermore, HAT uses differently tuned forgetting hyperparameters when target network sizes change. This is important to control for the target network capacity used per task and assumes knowledge of the (number of) tasks at hand. Using the code freely made available by the authors, we were able to rerun HAT for our target network size and longer task sequences. Here, we used the setup provided by the author’s code for HAT-Large for PermutedMNIST-10 and PermutedMNIST-100. To draw a fairer comparison, when changing our usual target network size to match the ones reported in Serra et al. (2018), we trained for 50 epochs per task (no training loss improvements afterwards observed) and also changed the batch size to 64 but did not changed our training scheme otherwise; in particular, we did not use a learning rate scheduler.
Efficient PermutedMNIST-250 experiments with a stochastic regularizer on subsets of previous tasks. An apparent drawback of Eq. 2 is that the runtime complexity of the regularizer grows linearly with the number of tasks. To overcome this obstacle, we show here that it is sufficient to consider a small random subset of previous tasks.
In particular, we consider the PermutedMNIST-250 benchmark (250 tasks) on CL1 using the hyperparameter setup from our PermutedMNIST-100 experiments except for a hypernetwork output size of 12000 (to adjust to the bigger multi-head target network) and a regularization strength βoutput = 0.1. Per training iteration, we choose maximally 32 random previous tasks to estimate the regularizer from Eq. 2. With this setup, we achieve a final average accuracy of 94.19 ± 0.16 (compared to an average during accuracy (i.e., the accuracies achieved right after training on the corresponding task) of 95.54 ± 0.05). All results are across 5 random seeds. These results indicate that a full evaluation of the regularizer at every training iteration is not necessary such that the linear runtime complexity can be cropped to a constant one.
Combining hypernetwork output regularizers with weight importance. Our hypernetwork regularizer pulls uniformly in every direction, but it is possible to introduce anisotropy using an EWC-like approach (Kirkpatrick et al., 2017). Instead of weighting parameters, hypernetwork outputs can be weighted. This would allow for a more flexible regularizer, at the expense of additional storage.
Task inference through predictive entropy (HNET+ENT). In this setup, we rely on the capability of neural networks to separate in- from out-of-distribution data. Although this is a difficult research problem on its own, for continual learning, we face a potentially simpler problem, namely to detect and distinguish between the tasks our network was trained on. We here take the first minimal step exploiting this insight and compare the predictive uncertainty, as quantified by output distribution entropy, of the different models given an input. Hence, during test time we iterate over all embeddings and therefore the models our metamodel can generate and compare the predictive entropies which results in making a prediction with the model of lowest entropy. For future work, we wish to explore the possibility of improving our predictive uncertainty by taking parameter uncertainty into account through the generation of approximate, task-specific weight posterior distributions.
Learning without task boundaries with hypernetworks. An interesting problem we did not address in this paper is that of learning without task boundaries. For most CL methods, it is crucial to know when learning one task ends and training of a new tasks begins. This is no exception for the methods introduced in this paper. However, this is not necessarily a realistic or desirable assumption; often, one desires to learn in an online fashion without task boundary supervision, which is particularly relevant for reinforcement learning scenarios where incoming data distributions are frequently subject to change (Rolnick et al., 2018). At least for discrete changes, with our hypernetwork setup, this boils down to a detection mechanism that activates the saving of the current model, i.e., the embedding e(T ), and its storage to the collection of embeddings {e(t)}. We leave the integration of our model with such a hypernetwork-specific switching detection mechanism for future work. Interestingly, our task-conditioned hypernetworks would fit very well with methods that rely on fast remembering (a recently proposed approach which appeared in parallel to our paper, He et al., 2019).

heading: E UNIVERSAL FUNCTION APPROXIMATION WITH CHUNKED NEURAL NETWORKS
text: Proposition 1. Given a compact subset K ⊂ Rm and a continuous function on K i.e. f ∈ C(K), more specifically, f : K → Rn with n = r ·NC. Now ∀ > 0, there exists a chunked neural network fch : Rm × C → Rr with parameters Θh, discrete set C = {c1, . . . , cNC} and ci ∈ Rs such that |f̄ch (x)− f(x)| < , ∀x ∈ K and with f̄ch (x) = [fch (x, c1), . . . , fch (x, cNC)].
For the following proof, we assume the existence of one form of the universal approximation theorem (UAT) for neural networks (Leshno & Schocken, 1993; Hanin, 2017). Note that we will not restrict ourselves to a specific architecture, nonlinearity, input or output dimension. Any neural network that is proven to be a universal function approximator is sufficient.
Proof. Given any > 0, we assume the existence of a neural network fh : Rm → Rn that approximates function f on K:
|fh(x)− f(x)| < 2 , ∀x ∈ K. (10)
We will in the following show that we can always find a chunked neural network fch : Rm × C → Rr approximating the neural network fh on K and conclude with the triangle inequality
|f̄ch (x)− f(x)| ≤ |f̄ch (x)− fh(x)|+ |fh(x)− f(x)| < , ∀x ∈ K. (11)
Indeed, given the neural network fh such that (10) holds true, we construct
f̂h(x, c) = { fcih (x) c = ci 0 else
(12)
by splitting the full neural network fh(x) = [fc1h (x), f c2 h (x), . . . , f cNC h (x)] with f̂h : Rm×C → Rr.
Note that f̂h is continuous on Rm × C with the product topology composed of the topology on Rm induced by the metric | ·− · | : Rm×Rm → R and the discrete topology on C. Now we can make use
of the UAT again: Given the compact K ⊂ Rn, the discrete set C = {c1, . . . , cNC} and any 2NC > 0, there exists a neural network function fch : Rm × Rs → Rr such that
|fch (x, c)− f̂h(x, c)| < 2NC , ∀x ∈ K, ∀c ∈ C. (13)
It follows that
∑ i |fch (x, ci)− f̂h(x, ci)| < ∑ i 2NC = 2 , ∀x ∈ K, (14)
which is equivalent to
|  f c h (x, c1)
... f ch(x, cNC) −  f̂h(x, c1)... f̂h(x, cNC)  | = |f̄ch (x)− fh(x)| < 2 , ∀x ∈ K. (15) We have shown (11) which concludes the proof.
Note that we did not specify the number of chunks NC, r or the dimension s of the embeddings ci. Despite this theoretical result, we emphasize that we are not aware of a constructive procedure to define a chunked hypernetwork that comes with a useful bound on the achievable performance and/or compression rate. We evaluate such aspects empirically in our experimental section.
F QUALITATIVE ANALYSES OF HYPERNETWORK-PROTECTED REPLAY MODELS
a b
Figure A8: Image samples from hypernetwork-protected replay models. The left column of both of the subfigures display images directly after training the replay model on the corresponding class, compared to the right column(s) where samples are obtained after training on eights and nines i.e. all classes. (a) Image samples from a class-incrementally trained VAE. Here the exact same training configuration to obtain results for split MNIST with the HNET+R setup are used, see Appendix C. (b) Image samples from a class-incrementally trained GAN. For the training configurations, see Appendix B. In both cases the weights of the generative part, i.e., the decoder or the generator, are produced and protected by a hypernetwork.

Review summaries:
Paper proposes a method for CL. The method is based on hypernetworks. These networks are a metamodel, which produce the parameters (from a task-conditioned embedding) which will be used in the main network. Preventing forgetting in the main network is now, replaced by preventing forgetting in the hypernetwork. This is done by imposing a regularization on the hypernetwork outcome, imposing that the generated weights should be similar for previous tasks (similar to Li & Hoiem who impose this on the network outputs). In addition, the paper proposes chunking, which refers to using an additional set of chunk, embeddings which are shared for all tasks, which allow compressing the hypernetwork. Furthermore, they propose an extension that allows for image replay (this is not an easy extension and an impressive contribution on itself, but maybe confusing for the current paper).

This paper proposes to use hypernetwork to prevent catastrophic forgetting. In deep learning, the information of the samples are converted to parameters during the training process, however, future training process could interfere with the information from the previous tasks. One of the method to prevent forgetting is to use reheasal, which retrains the network with previous data. The mechanism of this work is to store the previous samples as a trained point in the parameter space, so that a set of points in the original space is stored and thus rehearsed as one point in the parameter space, this saves both the memory and computation.