Continual learning with hypernetworks
Authors: Johannes von Oswald
Christian Henning
Benjamin F. Grewe
João Sacramento
Venue:  (2020)
Keywords: Continual Learning
Catastrophic Forgetting
Meta Model
Hypernetwork

Abstract: Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on task-conditioned hypernetworks, i.e., networks that generate the weights of a target model based on task identity. Continual learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using a simple regularizer. Besides achieving state-of-the-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories. Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable hypernetwork weights is comparable or smaller than target network size. We provide insight into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and show that task-conditioned hypernetworks demonstrate transfer learning. Finally, forward information transfer is further supported by empirical results on a challenging CL benchmark based on the CIFAR-10/100 image datasets.

Published as a conference paper at ICLR 2020
CONTINUAL LEARNING WITH HYPERNETWORKS
Johannes von Oswald*, Christian Henning*, Benjamin F. Grewe, João Sacramento *Equal contribution
Institute of Neuroinformatics University of Zürich and ETH Zürich Zürich, Switzerland [voswaldj, henninge, bgrewe, x 2
@ethz. ch
ABSTRACT
Artificial neural networks suffer from catastrophic forgetting when they se- quentially trained on multiple tasks. To overcome this problem, we present novel approach based on task-conding conditiond hypernetworks, i.e., networks that generate the weights of target model based on task identity. Continual learning (CL) is less difficult for this class of models thanks to simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworss only require rehearsing task-specific weight realizations, which can be maintained in memory using simple regularizer. Besides achieving state-of- the-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that cankicondito hypernetorks display very large capacity to retain previous memories. Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable hypernetwork weights is comparable or smaller than target network size. We provide insight into the structure of oow-dimensional task embedding spaces (the input space of the hypernetoor) and show that mosk-conditioned hypernetworks demonstrate transfer learning. Finally, forward information transfer is further supported by empirical results on challenging CL benchmark based on the CIFAR- 10/00000/10 image datasets.
INTRODUCTION
We assume that neural network f(x, Q) with trainable weights θ given data from a set of tasks ((X(), Y ..... (X(T), Y())), with input samples (() = (x((=)))
Ti=1 and output samples (()) (t.0) J where ni =| =[ A standard training approach learns the model using data from all tasks at once. However, this is not always possible in real-world problems, nor desirable in an online learning setting. Continual learning (CL) refers to an online learning setup in which tasks presented sequentially (see van de Ven & Tolias 2019 for recent review on CL). In CL, when learning new task I, starting with weights !! and observing only (X (t), Y the goal is to find a new set of parameters (1) that (1) retains (no catastrophic forgetting) (2) improves (positive backward transfer) performance on previous tasks compared to (f-1) and (3) solves the new task t potentially utilizing previously acquired knowledge (positive forward transfer). Achieving these goals is non-trivial, and longstanding issue in neural networks research.
Here, we propose addressing catastrophic forgetting the meta level: instead directly attempting to retain [(x +) for previous tasks, we fix the outputs of a metamodel fn(e, O,) termed task fask-conditioned hypernetwork which maps a task embedding e to weights O Now, single point has be memorized per task. To motivate such approach, we perform thought experiment: we assume that we are allowed to store all inputs (1) ..... x(T) seen so far, and to use these data to compute model outputs corresponding to ( In this idealized setting, one can avoid forgetting by simply mixing data from the current task with data from the past, (x (I)),..... (T Y (T--L),, (X( (T)),, where (+) refers to a set of synthetic targets generated using the model itself f(-, (t-)).. Hence, by training to retain previously acquired input-output mappings, one can obtain a sequential algorithm in principle powerful as multi-task learning. Multi-task learning, where all tasks learned

---

Published as a conference paper at ICLR 2020
simultaneously, can be seen CL upper-bound The strategy described above has been termed rehearsal (Robins 1995). However, storing previous task data violates our CL desiderata.
Therefore, introduce a change in perspective and move from the challenge of maintaining individual input-output data points to the problem of maintaining sets of parameters (O(+), ), without explicitly storing them. To achieve this, we train the metamodel parameters On analogous to the above outlined learning scheme, where synthetic targets now correspond to weight configurations that are suitable for previous tasks. This exchanges the storage of an entire dataset by a single www-diensional task descriptor, yielding massive memory saving in all but the simplest of tasks. Despite relying on regularization, our approach a conceptual departure from previous algorithms based regularization in weight (e.g., Kirkpatrick al. 2017: Zenke al. 2017) actívation space (e.g., He & Jaeger 2018).
Our experimental results show that lask-conditioned hypernetworks do not suffer from catastrophic forgetting on set of standard CL benchmarks. Remarkably, they are capable of retaining memories with practically no decrease in performance, when presented with very long sequences of tasks. Thanks to the expressive power of neural networks, task-conditioned hypernetworks exploit task-to- task similarities and transfer information forward in time future tasks. Finally, the task-conditional metamodelling perspective that we put forth is generic, as it does not depend on the specifics of the target network architecture. We exploit this key principle and show that the very same metamodelling framework extends to, and can improve, important class of CL methods known generative replay methods, which are current state-of-the-art performers in many practical problems (Shin et al. 2017
Wu et al. 2018 van de Ven & Tolias 2018
MODEL
2.1 TASK-CONDITIONED HYPERNETWORKS
Hypernetworks parameterize target models. The centerpiece of our approach to continual learn- ing is the hypernetwork, Fig. Th. Instead of learning the parameters Ougt of particular function fugt directly (the target model), we learn the parameters On of metamodel. The output of such meta- model, the hypernetwork, is UugI- Hypernetworks can therefore be thought of as weight generators, which were originally introduced to dynamically parameterize models in compressed form (Ha
et al. 2017 Schmidhuber 1992 Bertinetto et al. 2016 Jia et al. 2016
b
f
target
f
network
Ins
40
Irg
11) e
0
(4.2) ()
(1:3)
N.)
lugt
Int
regularized hypernetwork
f
f
f
f..
f,
f.
(3)
(
c(
chunked hypernetwork
Figure 1: Task-conditioned hypernetworks for continual learning. (a) Commonly, the parame- ters of neural network are directly adjusted from data to solve task. Here, weight generator termed hypernetwork is learned instead. Hypernetworks map embedding vectors to weights, which parameterize a target neural network. In a continual learning scenario, set of task-specific em- beddings is learned via backpropagation. Embedding vectors provide task-dependent context and bias the hypernetwork to particular solutions. (b) A smaller, chunked hypernetwork can be used iteratively, producing chunk of target network weights at time (e.g., one layer at a time). Chunked hypernetworks can achieve model compression: the effective number of trainable parameters can be smaller than the number of target network weights.
2

---

Published as a conference paper at ICLR 2020
Continual learning with hypernetwork output regularization. One approach avoid catas- trophic forgetting is to store data from previous tasks and corresponding model outputs, and then fix such outputs. This can be achieved using output regularizer of the following form, where past outputs play the role of pseudo-targets (Robins, 1995: Li & Hoiem
2018 Benjamin et al. 2018):
T-1XX Couputt ΣΣ
[ θ*) -(x θI|r
In the equation above, @* is the set of parameters before attempting to learn task T, and ff is the learner. This approach, however, requires storing and iterating over previous data, process that is known as rehearsing. This potentially expensive memory-wise and not strictly online learning. A possible workaround is generate the beeudo-targets by evaluating f random patterns (Robins) 1995) or on the current task dataset (Li & Hoiem 2018). However, this does not necessarily fix the behavior of the function f in the regions of interest.
Hypernetworks sidestep this problem naturally. In target network weight space, a single point (i.e., one set of weights) has be fixed per task. This can be efficiently achieved with cask-conditioned hypernetworks, by fixing the hypernetwor output on the appropriate task embedding.
Similar to Benjammin et al. (2018), we use a two-step optimization procedure introduce memory- preserving hypernetwork output constraints. First, we compute a candidate change AO, which minimizes the current task loss - Lask Tank(On,, (On, e(T), ( Y(T) with respect O. The candidate A is obtained with an optimizer of choice (we use Adam throughout; Kingma & Ba 2015). The
actual parameter change then computed by minimizing the following total loss:
Liaall (On (T)) (T), Y(T))
Loutpu (0i, Ob, AGn. })(
Boutpur (Oh (T)) ((),, Y(T) T
T-1 () O,)) - (n((ee On 10,))
where 0% is the set of hypernetwork parameters before attempting to learn task T, AO, is considered fixed and Bourpar is hyperparameter that controls the strength of the regularizer. On Appendix D run a sensitivity analysis on Sompue and experiment with a more efficient stochastic regularizer where the averaging performed over a random subset of past tasks.
algorithms that involve a full inner-loop refinement, or use second- order gradient information by backpropagating through A0, could be applied. However, we found empirically that our one-step correction worked well. Exploratory hyperparameter scans revealed that the inclusion of the lookahead AO, in (2) brought minor increase in performance, even when computed with cheap one-step procedure. Note that unlike in Eq. [ the memory ceenorypreseeiing.. term Comput does not depend on past data. Memory of previous tasks enters only through the collection of T-1 task embeddings {e(t)) 1==1
More computationally-intene
Learned task embeddings. Task embeddings are differentiable deterministic parameters that can be learned, just like On. At every learning step of our algorithm, we also update the current task embedding eTT) to minimize the task loss c(T) After learning the task, the final embedding is saved and added the collection (e(0).
2.2 MODEL COMPRESSION WITH CHUNKED HYPERNETWORKS
Chunking. In straightforward implementation, a hypernetwork produces the entire set of weights of target neural network. For modern deep neural networks, this is a very high sigh-dimessional output. However, hypernetworks can be invoked iteratively, filling in only part of the target model at each step, in chunks (Ha al.) 2017 Pawlowski et al. (017). This strategy allows applying smaller hypernetworks that are reusable. Interestingly, with chunked hypernetworks it is possible to solve tasks in a compressive regime, where the number of learned parameters (those of the hypernetwork) is effectively smaller than the number of target network parameters.
Chunk embeddings and network partitioning. Reapplying the same hypernetwor multiple times introduces weight sharing across partitions of the target network, which is usually not desirable.
3

---

Published as a conference paper at ICLR 2020
To allow for flexible parameterization of the target network, we introduce set C { of chunk embeddings, which are used as an additional input to the hypernetwrk, Fig. [b. Thus, the full set of target network parameters irg = (e, ci)..... file, Cve)] is produced by iteration over C, keeping the task embedding e fixed. This way, the hypernetwork can produce distinct weights for each chunk. Furthermore, chunk embeddingg,, just like task embeddings, deterministic parameters that we learn via backpropagation. For simplicity, we use a shared set of chunk embeddings for all tasks and we do not explore special target network partitioning strategies.
ordinary
How flexible is our approach? Chunked neural networks can in principle approximate any target weight configuration arbitrarily well. For completeness, we state this formally in Appendix
2.3 CONTEXT-FREE INFERENCE: UNKNOWN TASK IDENTITY
Determining which task to solve from input data. Our hypernetwork requires task embedding input to generate target model weights. In certain CL applications, an appropriate embedding can be immediately selected as task identity is unambiguous, or can be readily inferred from contextual clues. In other cases, knowledge of the task at hand is not explicitly available during inference. In the following. we show that our metamodelling framework generalizes to such situations. In particular, we consider the problem of inferring which task to solve from a given input pattern, a noted benchmark challenge (Farquhar & Gal 2018 van Ven & Tolias 2019). Below, we explore two different strategies that leverage cask-condttooned hypernetworks in this CL setting.
Task-dependent predictive uncertainty. Neural network models are increasingly reliable in sig- nalling novelty and appropriately handling out-of-distributiin data. For categorical target distributions, the network ideally produces a flat, high entropy output for unseen data and, conversely, peaked, low-entropy response for in-distribution data (Hendrycks & Gimpel 2016 Liang et al. 2017). This suggests a first, simple method for task inference (HNET+ENT). Given an input pattern for which task identity is unknown, we pick the task embedding which yields lowest predictive uncertainty, as quantified by output distribution entropy. While this method relies on accurate novelty detection, which is in itself a far from solved research problem, it is otherwise straightforward to implement and no additional learning or model is required infer task identity.
wewpeeetwwor--ooteeteeee synthetic replay. When a generative model is available, catastrophic forgetting can be circumvented by mixing current task data with replayed past synthetic data (for recent work see (Shin et al., 2017; Wu al., 2018). Besides protecting the generative model itself, synthetic data can protect another model of interest, for example, another discriminative model. This conceptually simple strategy is in practice often the statc-of-the-art solution to CL (van de Ven & Tolias 2019), Inspired by these successes, we explore augmenting our system with replay network, here a standard variational autoencoder (VAE; Kingma & Welling/ 2014) (but see Appendix for experiments with generative adversarial network, Goodfellow et al. 2014).
Synthetic replay is a strong, but not perfect, CL mechanism as the generative model is subject to drift, and errors tend accumulate and amplify with time. Here, we build upon the following key observation: just like the target network, the generator of the replay model can be specified by a hypernetwork. This allows protecting it with the output regularizer, Eq. [2, rather than with the model's own replay data, as done in related work. Thus, in this combined approach, both synthetic replay and task-conditional metamodelling act in tandem reduce forgetting.
We explore hypernetwork protteetworkpproteeteed replay in two distinct setups. First, we consider a minimalist architecture (HNET+R), where only the replay model, and not the target classifier, is parameterized by hypernetwork. Here, forgetting in the target network is obviated by mixing current data with synthetic data. Synthetic target output values for previous tasks are generated using soft targets method, i.e., by simply evaluating the target function before learning the new task on synthetic input data. Second (HNET+TIR), we introduce an auxiliary task inference classifier, protected using synthetic replay data and trained to predict task identity from input patterns. This architecture requires additional modelling, but it is likely to work well when tasks are strongly dissimilar. Furthermore, the task inference subsystem can be readily applied to process more general forms of contextual information, beyond the current input pattern. We provide additional details, including network architectures and the loss functions that are optimized, in Appendices B aand
4

---

Published as a conference paper at ICLR 2020
3 RESULTS
We evaluate method on set of standard image classification benchmarks the MNIST, CIFAR- 10 and CIFAR-100 public datasets" Our main aims are to (1) study the memory retention capabilities of fask-conditioned hypernetworks across three continual learning settings, and (2) investigate information transfer across tasks that are learned sequentially.
Continual learning scenarios. In our experiments we consider three different CL scenarios (van de Ven Tolias 2019). In CL1, the task identity is given to the system. This arguably the standard sequential learning scenario, and the one consider unless noted otherwise. In CL2, task identity is unknown to the system, but it does not need to be explicitly determined. A target network with a fixed head is required to solve multiple tasks. In CL3, task identity has to be explicitly inferred. It has been argued that this scenario is the most natural, and the one that tends to be harder for neural networks (Farquhar & Gal 2018 van de Ven & Tolias 2019).
Aiming at comparability, for the experiments on the MNIST dataset we model the target network fully-connected network and set all hyperparameters after van de Ven & Tolias 2019), who recently reviewed and compared a large set of CL algorithms. For our CIFAR experiments, we opt for ResNet-33 target neural network (He al. (016) to assess the scalability of our method. A summary description of the architectures and particular hyperparameter choices, as well as additional experimental details, is provided in Appendix C experiments, the number of hypernetwork parameters is always smaller or equal than the number of parameters of the models we compare with.
Experimental details.
We emphasize that, all our
a
b
1D regression (hnet. reg.)
1D regression (non-CL)
1D regression tuniig)
0
0
2.5
0.0
2.5
-2.5
0.0
2.5
2255
0.0
2.5
x
x
Figure 2: 1D nonlinear regression. (a) Task-conditioned hypernetworks with output regularization can easily model a sequence of polynomials of increasing degree, while learning in a continual fashion. (b) The solution found by a target network which is trained directly on all tasks simultaneously is similar. (c) Fine-tuning, i.e., learning sequentially, leads to forgetting of past tasks. Dashed lines depict ground truth, markers show model predictions.
Nonlinear regression toy problem. To illustrate our approach, we first consider a simple nonlinear regression problem, where the function to be approximated is scalar-valued, Fig. [ Here, sequence of polynomial functions of increasing degree has to be inferred from noisy data. This motivates the continual learning problem: when learning each task in succession by modifying O with the memory preeoerire regularizer turned off (Souppt = O, see Eq. [
the network learns the last task but forgets previous ones, Fig. 2 The regularizer protects old solutions, Fig. 2h, and performance is comparable to an offline non non-continual learner, Fig. Db.
Permuted MNIST benchmark. Next, we study the permuted MNIST benchmark. This problem is set as follows. First, the learner is presented with the full MNIST dataset. Subsequently, novel tasks are obtained by applying random permutation to the input image pixels. This process can be repeated to yield long task sequence, with a typical length of T = 10 tasks. Given the low similarity of the generated tasks, permuted MNIST is well suited to study the memory capacity of continual learner. For T = 10, we find that task- maskccondttioned hypernetworks are state-of-the of-the-art on CL1, Table [ Interestingly, inferring tasks through the predictive distribution entropy (HNET+ENT) works well on the permuted MNIST benchmark. Despite the simplicity of the method, both synaptic intelligence (SI; Zenke et al. 2017) and online elastic weight consolidation (EWC: SScwwarz et al. 2018) are overperformed on CL3 by a large margin. When complemented with generative replay
Source code available under mnderrmms//gttttamccmmmmmmmmmmme
5

---

Published as a conference paper at ICLR 2020
a
b
PermutedMNNIT--00
Compression-peformaace trade-off
100
100
Accuracy [%)
75
96.78%
Accuracy [%)]
80
95.66%
50
60
25
40
100
0.0
0.5
1.0
Task
Compression ratio
hnett
SI
DGR-distil
Online EWC
during final
Figure 3: Experiments on the permuted MNIST benchmark. (a) Final test set classification accuracy the t-th task after learning hundred permutations (PermueedMNIST-IOO) conditioned hypernetworks (hnet, in red) achieve very large memory lifetimes on the permuted MNIST benchmark. Synaptic intelligence (SI, in blue; Zenke et al.) (017), online EWC (in orange; Schwarz et al. (2018) and deep generative replay
Task-
in green: SSin et al. 2017) methods are shown for comparison. Memory retention in SI and DGR+distill degrade gracefully, whereas EWC suffers from rigidity and can never reach very high accuracy, even though memories persist for the
entire experiment duration. (b) Compression ratio (
10rgl versus task-averaged test set accuracy after learning all tasks (labelled 'final', in red) and immediately after learning task (labelled 'during', purple) for the PermutedMNIST-IIe 10 benchmark. Hypernetworks allow for model compression and perform well even when the number of target model parameters exceeds their own. Performance decays nonlinearly: accuracies stay approximately constant for a wide range of compression ratios below unity. Hyperparameters were tuned once for compression ratio se 1 and were then used for all compression ratios. Shaded areas denote STD (a) resp. SEM (b) across random seeds.
methods, coskiconditi hypemnetorrks (HNET+TIR and HNET+R) are the best performers on all three CL scenarios.
Performance differences become larger in the long sequence limit, Fig, 3p. For longer task sequences
(T = 100), SI and DdR+iistill (Shin al. 2017 van de Ven & Tolaa 2018 degrade gracefully, while the regularization strength of online EWC prevents the method from achieving high accuracy (see Fig. fo a hyperparameter search related work). Notably, task-conditioned hypernetworks show minimal memory decay and find high performance solutions. Because the hypernetwork operates in compressive regime (see Fig. 3 and Fig. A7] exploration of compression ratios), our results do not naively rely on an increase in the number of parameters. Rather, they suggest that previous methods are not yet capable of making full use of target model capacity in CL setting. We report set of extended results on this benchmark on Appendix D including a study of CL2/ 3 (T 100), where HNET+TIR strongly outperforms the related work.
Split MNIST benchmark. Split MNIST is another popular CL benchmark, designed to introduce task overlap. In this problem, the various digits are sequentially paired and used form five binary classification tasks. Here, we find that task- condittion hypemnetorkss are the best overall performers. In particular, HNET+R improves the previous ofate-of-the-an method DGR+distill on both CL2 and CL3, almost saturating the CI.2 upper bound for replay models (Appendix D). Since HNET+R is essentially hypernetwork-prooterrr
DGR, these results demonstrate the generality of task-conditioned hypernetworks effective memory protectors. To further support this, in Appendix [P) we show that our replay models (we experiment with both a VAE and a GAN) can learn in a class-incremental manner the full MNIST dataset. Finally, HNET+ENT again outperforms both EWC and SI, without any generative modelling.
On the split MNIST problem, tasks overlap and therefore continual learners transfer information across tasks. To analyze such effects, we study aask-conditioned hypernetworks with wwo-dimensional task embedding spaces, which be easily visualized. Despite learning happening continually,

---

Published as a conference paper at ICLR 2020
Table I: Task-averaged test accuracy (# SEM, n. 20) on the permuted ('P10') and split ("S") MNIST experiments. In the table. EWC refers to online EWC and DGR refers to DGR+distill (results reproduced from van de Ven & Tolias 2019). We tested three hypernetwork -bseed models: for HNET+ENT (HNET alone for CL1), we inferred task identity based on the entropy of the predictive distribution; for HNET+TIR, we trained hypernetwork prooteetworkppooteeeee protected recognition-replay network (based on VAE, cf. Fig. ATD to infer the task from input patterns; for HNET+R the main classifier was trained by mixing current task data with synthetic data generated from a hypernetwork-prroneee protected VAE.
EWC
SI
DGR
HNET+ENT HNET+TIR HNET+R
P10-C11
95.96
0.06
94.75 0.14
97.51 0.01
97.57 +:57:000 0.02
97.57 + 0.02
97.87 + 0.01
P10-C12
94.42 0.13
35.33+0.11
97.35 + 0,02
92.80 0.15
97.58
0.02
97.60 + 0.01
PI0-CL3
33.88 + 0.49
29.31 + 0.62
96.38 - 0.03
91.75 + 0.21
97.59
97.76 1 0.01
S-CL1
99.12 + 0.11
99.09
0.15
99.61 = 0.02
99.79 0.01
99.79
99.83 + 0.01
S-CL2
64.32 + 1.90
65.36
1.57
96.83 1 0.20
87.01 0.47
94.43
98.00 1 0.03
S-CL3
19.96 1 0.07
19.99 1 0.06
91.79 + 0.32
69.48
0.80
89.59 0.59
95.30 1 0.13
b
T1 after learning
T5 after learning T5
100
T3
TS
[% Kaeemooy
75
8,
T2
50
-5
-5
-5
-5
5
Figure 4: Two-dimensional task embedding space for the split MNIST benchmark. Color- coded test set classification accuracies after learning the five splits, shown as the embedding vector components are varied. Markers denote the position of final task embeddings. (a) High classification performance with virtually no forgetting achieved even when e-space is dinddmmnsional. The model shows information transfer in embedding space: the first task is solved in large volume that includes embeddings for subsequently learned tasks. (b) Competition embedding space: the last task occupies finite high performance region, with graceful degradation away from the embedding vector. Previously learned task embeddings still lead to moderate, above-chance performance.
find that the algorithm converges to hypernetwork configuration that can produce target model parameters that simultancously solve old and new tasks, Fig. given the appropriate task embedding.
Split CIFAR- 11FAR-10/100 benchmark. Finally, we study a more challenging benchmark, where the learner first asked to solve the full CIFAR-10 classification task and is then presented with sets of ten classes from the CIFAR- 100 dataset. We perform experiments both with teghhpeeformance
ResNet-32 target network architecture (Fig. (5 and with a shallower model (Fig. A3) that we exactly reproduced from previous work (Zenke et al. (017)). Remarkably, the ResNet-32 model, we find that task-conditioned hypernetworks essentially eliminate altogether forgetting. Furthermore, forward information transfer takes place; knowledge from previous tasks allows the network to find better solutions than when learning each task individually from initial conditions. Interestingly, forward transfer is stronger the shallow model experiments (Fig. A3 where otherwise find that method performs comparably to SI.
DISCUSSION
Bayesian accounts of continual learning. According to the standard Bayesian CL perspec- tive, posterior parameter distribution is recursively updated using Bayes' nule tasks arrive
7

---

Published as a conference paper at ICLR 2020
Figure 5: Split CIFAR-10/100 CL benchmark. Test set accuracies (mean + STD, n 5) on the entire CIFAR- 10 dataset and subsequent CIFAR- 100 splits of ten classes. Our hypernetwork- protected ResNet-32 displays virtually no forgetting; final averaged perfor- mance (hnet, in red) matches the imme- diate one (hnet-during, in blue). Further- more, information is transferred across tasks, as performance is higher than when training each task from scratch (purple). Disabling our regularizer leads to strong forgetting (in yellow).
Split CIFAR-10/100
100
83
79
82
75
Accuracy [%)
50
25
CIFAR-10
2
Tassk
hnet during
hnet
from scratch
fine-tuning
(Kirkpatrick et al. 2017 Huszár 2018 Nguyen et al. 2018) While this approach is theoretically
sound, in practice, the approximate inference methods that are typically preferred can lead to stiff models, as a compromise solution that suits all tasks has to be found within the mode determined by the first task. Such restriction does not apply to hypernetworks, which can in principle model complex multimodal distributions (Louizos & Welling 2017 Pawlowski et al., 2017| Henning et al., 2018). Thus, rich, ceerretworr-moddelle priors are one avenue of improvement for Bayesian CL methods. Interestingly, task-conditioning offers an alternative possibility: instead of consolidating every task onto a single distribution, a shared fask-conditioned hypernetwork could be leveraged to model set of parameter posterior distributions. This conditional metamodel naturally extends our framework to the Bayesian learning setting. Such approach will likely benefit from additional flexibility, compared to conventional recursive Bayesian updating.
Related approaches that rely task-conditioning. Our model fits within, and in certain ways generalizes, previous CL methods that condition network computation on task descriptors. Task- conditioning is commonly implemented using multiplicative masks at the level of modules (Rusu (et al.] 2016 |ernando et al.] (017). neurons (Serra et al., 2018: Masse et al., 2018) or weights (Mallya & Lazebnik| 2018). Such methods work best with large networks and come with a significant storage overhead, which typically scales with the number of tasks. Our approach differs by explicitly modelling the full parameter space using a metamodel, the hypernetwork. Thanks this metamodel, generalization in parameter and task space is possible, and task-to-tase dependencies can be exploited to efficiently represent solutions and transfer present knowledge to future problems. Interestingly, similar arguments have been drawn in work developed concurrently to ours (Lampinen & McClelland 2019), where task embedding spaces are further explored in the context of few-shot learning. In the same vein, and like the approach developed here, recent work in CL generates last-layer network parameters as part of a pipeline to avoid catastrophic forgetting (Hu et al. 2019) or distills parameters onto contractive auto-encoding model (Camp et al. 2018).
Positive backwards transfer. In its current form, the hypernetwork output regularizer protects previously learned solutions from changing, such that only weak backwards transfer of information can occur. Given the role of selective forgetting and refinement of past memories in achieving intelligent behavior (Brea et al.) 2014 Richards & Frankland] 2017), 17), investigating and improving backwards transfer stands as important direction for future research.
Relevance to systems neuroscience. Uncovering the mechanisms that support continual learning in both brains and artificial neural networks is long- anngsstaddng question (McCloskey Cohen 1989 French 1999 Parisi et al.) 2019). We close with speculative systems interpretation (Kumaran et al. 2016: Hassabis et al. (011)] of our work as a model for modulatory top-down signals in cortex. Task now-dimensional context switches, which determine the behavior of modulatory system, the hypernetwork in our case. According to our model, the hypernetwork would in turn regulate the activity of target cortical network.
embeddings can be
As it stands, implementing a hypernetwork would entail dynamically changing the entire connectivity of target network, cortical area. Such process seems difficult to conceive in the brain. However, this strict literal interpretation can be relaxed. For example, a hypernetwork can output lower- dimensional modulatory signals (Marder 2012). instead of a full set of weights. This interpretation
8

---

Published as a conference paper at ICLR 2020
is consistent with growing body of work which suggests the involvement of modulatory inputs in implementing context- or desendenn network mode-switching (Mante et al. 2013 Jeger 2014
Stroud et al. 2018 Masse et al. 2018).
CONCLUSION
We introduced novel neural network model, the task-conditioned hypernetwork, that well-suited for CL problems. A task-conditioned hypernetwork is metamodel that learns to parameterize target functions, that are specified and identified in a compressed form using task embedding vector. Past tasks are kept in memory using hypernttwok output regularizer, which penalizes changes in previously found target weight configurations. This approach is scalable and generic, being applicable as standalone CL method in combination with generative replay. Our results state-of -tt-hhe-art art on standard benchmarks and suggest that task-conditioned hypernetoorks can achieve long memory lifetimes, well as transfer information to future tasks, essential properties of continual learner.
ACKNOWLEDGMENTS
This work was supported by the Swiss National Science Foundation (B.FG. CRSI15-173721), ETH project funding (B.F.G. ETH-20 19-01) and funding from the Swiss Data Science Center (B.F.G, C17-18, J. V. O. P18-03). Special thanks Simone Carlo Surace, Adrian Huber, Xu He, Markus Marks, Maria R. Cervera and Jannes Jegminat for discussions, helpful pointers to the CL literature and for feedback on our paper draft.
REFERENCES
Ari S. Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. arXiv preprint arXiv:1805.08250 2018.
Luca Bertinetto, João F. Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed- forward one-shot learners. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 523-531. Curran Associates, Inc., 2016.
Johanni Brea, Robert Urbanczik, and Walter Senn. A Normative Theory of Forgetting: Lessons from the Fruit Fly. PLOS Computational Biology, 1006):e 2014.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Blake Camp, Jaya Krishna Mandivarapu, and Rolando Estrada. Self-net: Lifelong learning via continual modelig.. arXiv preprint arXiv: 10354, 2018.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning, arXiv preprint arXiv:1907.02544, 2019.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning, arXiv preprint aoX/v:180..09733. 2018
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:101.008334 2017. 2017.
Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, (4):128- 135, April 1999.
Ian Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances Neural Information Process- ing Systems 27, pp. 2680. Curran Associates, Inc., 2014.

---

Published as a conference paper at ICLR 2020
David Ha, Andrew M. Dai, and Quoc V. Le. HyperNetworks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings,205 2017.
Boris Hanin. Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations. arXiv preprint: vrXiv:1708.026//1 2017.
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-IIspired Artificial Intelligence. Neuron, 95(2):245-258, July 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Xu He and Herbert Jaeger. Overcoming catastrophic interference using conceptor-aidee aided backpropa- gation. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April May 3, 2018, Conference Track Proceedings, 2018.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint #rXiv::906.05200 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribttion examples in neural networks. arXiv preprint arXiv: 2016.
Christian Henning, Johannes von Oswald, João Sacramento, Simone Carlo Surace, Jean-Pascal Pfister, and Benjamin F Grewe. Approximating the predictive distribution via adversarially-trained hypemetworks. In NewrIPS Bayesian Deep Learning Workshop, 2018.
Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting for continual learning via model adaptation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Ferene Huszár. Note on the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences, (15(1))FE246-F220 March 2018.
Herbert Jaeger. Controlling Recurrent Neural Networks by Conceptors. arXiv preprint:
arXiv: 100.333403.336, 2014.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc Gool. Dynamic Filter Networks. In D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 665-675. Curran Associates, Inc., 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd Interna- tional Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Ctrabsa--Bawwing Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-35 -32263566..16 March 2017.
Dharshan Kumaran, Demis Hassabis, and James L. McClelland. What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated. Trends Cognitive Sciences, 20(7) July 2016.
10

---

Published as a conference paper at ICLR 2020
Andrew K Lampinen and James L McClelland. Embedded meta-learning Toward more flexible deep-learning models. arXiv preprint arXiv:1905.09955 2019.
Moshe Leshno and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function approximate any function. Neural Networks, 6:861-867, 1993.
Z. Li and D. Hoiem. Learning without Forgetting. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 40(12):2935-2947 2018.
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690, 2017.
Christos Louizos and Max Welling. Multiplicative Normalizing Flows for Variational Bayesian Neural Networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 2218 11---222222222 JMLR.org, 2017.
Mario Lučié. Michael Tschannen, Marvin Ritter, Xiaohua Zhai Olivier Bachem, and Sylvain Gelly. High-fidelity image generation with fewer labels. In International Conference on Machine Learning, pp. 4183 4192, 2019.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Valerio Mante, David Sussillo, Krishna V. Shenoy, and William T. Newsome. Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature, 503(7474):78-88 84, 2013.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, pp. 2p.3281--88 IEEE, 2017.
Eve Marder. Neuromodulation of Neuronal Circuits: Back to the Future. Neuron, 76(1):1-11, October 2012.
Nicolas Y Masse, Gregory D Grant, and David J Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences, 115(44):510467--100
2018.
Michael McCloskey and Neal J. Cohen. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. volume 24, pp. 109-165. Academic Press, 1989.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv:1411.1784, 2014.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. 2018.
German Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter, Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, May 2019.
Nick Pawlowski, Andrew Brock, Matthew C. H. Lee, Martin Rajchl, and Ben Glocker. Implicit Weight Uncertainty in Neural Networks. arXiv preprint arXiv:1711.01297 2017.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31st International Conference on International Conference on Machine Learning Volume 32, ICML' pp. II-1278- 11-1286. JMLR.org, 2014.
Blake A. Richards and Paul W. Frankland. The Persistence and Transience of Memory. Neuron, 94 (6):1071-1084, June 2017.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Resa-Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 3748-3748. Curran Associates, Inc., 2018

---

Published as a conference paper at ICLR 2020
Anthony Robins. Catastrophic Forgetting, Rehearsal and Pseudorchearsal. Connection Science, 7(2):
123-146, June 1995.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Pillicrra, and Greg Wayne. Experience replay for continual learning, arXiv preprint 1rXiv:1811.11668 2018.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. arXiv preprint 1606. 04671, 2016.
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Compulation, 4(1):111-13, 1992.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Grabska- Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4528 4537, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4548-4557, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. 8. PMLR.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual Learning with Deep Generative Replay. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 2990-2999. Curran Associates, Inc., 2017.
Jake P. Stroud, Mason A. Porter, Guillaume Hennequin, and Tim P. Vogels. Motor primitives in space and time via targeted gain modulation in cortical networks. Nature Neuroscience, 21(12)11774, December 2018.
Siddharth Swaroop, Cuong V Nguyen, Thang D Bui, and Richard E Turner. Improving and under-
standing variational continual learning. Continual Learning Workshop at NeurIPS, 2018.
Gido M. van de Ven and Andreas S. Tolias. Generative replay with feedback connections as general strategy for continual learning. arXiv preprint 1809. 2018
Gido M. van de Ven and Andreas S. Tolias. Three scenarios for continual learning. arXiv preprint arXiv:194..07334 2019.
Chenshen WU, Luis Herranz, Xialei Liu, yaxing wang, Joost van de Weijer, and Bogdan Raducanu. Memory Replay GANs: Learning Generate New Categories without Forgetting. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 5962-5972. Curran Associates, Inc., 2018
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual Learning Through Synaptic Intelligence. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 3987-3995. JMLR.org, 2017.
A TASK-CONDITIONED HYPERNETWORKS. MODEL SUMMARY
In our model, task-conditioned hypernetwork produces the parameters Ougr = fn(e, Qp) of target neural network. Given one such parameterization, the target model then computes predictions ý fugi(x (x, Ougi) based on input data. Learning amounts adapting the parameters O₂ of the hypernetwork, including a set of task embeddings (e as well as set of chunk embeddings J.r, in case compression is sought or if the full hypernetwork is too large to be handled directly. To avoid castastrophic forgetting, we introduce an output regularizer which fixes the behavior of the hypernetwork by penalizing changes in target model parameters that are produced for previously learned tasks.
12

---

Published as a conference paper at ICLR 2020
Variables that need to be stored while learning new tasks. What are the storage requirements of our model, when learning continually?
1. Memory retention relies on saving one embedding per task. This collection [e(")_ therefore grows linearly with Such linear scaling is undesirable asymptotically, but it turns out to be essentially negligible in practice, as each embedding is a single low- dimensional vector (e.g., see Fig. 4 for a run with 2D embeddings).
2. A frozen snapshot of the hypernetwork parameters 0 taken before learning a new task, needs be kept, evaluate the output regularizer in Eq
B ADDITIONAL DETAILS ON HYPERNETWORKKPROTEETTT REPLAY MODELS
Variational autoencoders. For all HNET+TIR and HNET+R experiments reported on the main text we use VAEs as our replay models (Fig. a Kingma Welling 2014). Briefly, a VAE consists of an encoder-decoder network pair, where the encoder network processes some input pattern x and its outputs Vena()) = (μ,, o²) comprise the parameters μ and  (encoded log domain, to enforce nonnegativity) of diagonal multivariate Gaussian pz (2; μt, o²), which governs the distribution of latent samples z. On the other side of the circuit, the decoder network processes latent sample z and ane-hot-encoded task identity vector and returns imput pattern reconstruction, face (z, 1,) X.
VAEs can preserve memories using a technique called generative replay: when training task T', input samples are generated from the current replay network for old tasks f < I', by varying 1, and drawing latent space samples z. Generated data can be mixed with the current dataset, yielding an augmented dataset used to relearn model parameters. When protecting a discriminative model, synthetic 'soft' targets can be generated by evaluating the network on X. We use this strategy to protect an auxiliary task inference classifier in HNET+TIR, and to protect the main target model in HNET+R.
setpnnetwook-ppooeeeeee replay. In HNET+TIR and HNET+R experiments, parameterize the decoder network through a task-conditioned hypernetwork, fhdee On.ee). combination with our output regularizer, this allows us to take advantage of the memory retention capacity of hypernetworks, now on a generative model.
The replay model (encoder, decoder and decoder hypernetwork) is separate subsystem that is optimized independently from the target network. Its parameters Oene and Ob, are learned by minimizing our regularized loss function, Eq. [2] here with the task-specific term set the standard VAE objective function,
V X, Oency On.ios) = Lree (X, Oene; Odec) Eprior X, Oenc, Odec),
(3)
with O = (e, On.tec) introducing the dependence on Oh,ee- LVae balances a reconstruction free and a prior-matching Lariar penalties. For our MNIST experiments, we choose binary cross- entropy (in pixel space) as the reconstruction loss, that we write below for single example x
Liee (x, Oenc, Osec) Lxem (x, faee (z, 1x(x)= Odec)):
where (ten(t,)) = = -x to log yk is the cross entropy. For diagonal Gaussian pz. the prior- matching term can be evaluated analytically.
1 [z] (1 log of of Ht)
Cprirr
(5)
2
Above, z is sample from pz (z; μu ((x) (x), (x)) obtained via the reparameterization trick (Kngmma & Welling 2014 Rezende et al. 2014). This introduces the dependency of Lae on O
Task inference network (HNET+TIR). In the HNET+TIR setup, we extend our system to include task inference neural network classifier a (x) parameterized by On, where tasks encoded with a T T-dimensional softmax output layer. In both CL2 and CL3 scenarios we use a growing single-head setup for a, and increase the dimensionality of the softmax layer as tasks arrive.
13

---

Published as a conference paper at ICLR 2020
This network is prone to catastrophic forgetting when tasks are learned continually. To prevent this from happening we resort to replay data generated from bypermetwork-proteeeet VAE, described above. More specifically, we introduce a task inference loss.
CT((,, Qn) Lxem (x)) (xx,, enne))
where ((x denotes the correct task identity for a sample x from the augmented dataset X (X ..... X(T-1), ( with ( being synthetic data faze (z, 1,, deec for t 1...T and X(T) ( is the current task data. Importantly, synthetic data is essential to obtain well defined objective function for task inference; the cross-entropy loss LTI requires at least two groundtruth classes be optimized. Note that replayed data be generated online by drawing samples from
the prior.
a
b
C
enc
pce
α
hagen
gen
X
fn
0es
disc
(X")
h,dec
dec
x
Figure Al: Hypernetwwrkkproteeeeed replay model setups. (a) A sypernetwor-pprotectt VAE, that we used for HNET+R and HNET+TIR main text experiments. (b) A appernetwork-proeeeee GAN, that used for our class-increment learning Appendix Fexperiments. (c) A task inference classifier protected with synthetic replay data, used on HNET+1TR experiments.
prreerettorr-ppooeeeeedd protected GANs.
Generative adversarial networks (Goodfellow et al. 2014) have
become an established method for generative modelling and tend to produce higher quality images compared to VAEs, even at the scale of datasets complex ImageNet (Brock et al.) 2019 Lucic
cet al. 2019 Donahue & Simonyan 2019). This makes GANs perfect candidates for powerful replay models. A suitable GAN instantiation for CL is the conditional GAN (Mirza & Osindero 2014) as
studied by Wu et al. (2018). Recent developments in the GAN literature already allude towards the potential of using hypernetwork-like structures, e.g., when injecting the latent noise (Karras et al. 2019) or when using cans-conndiional machh-ormallization as in (Brock et al., 2019). We propose to go one step further and use hypernetorr that maps the condition the full set of generator parameters θ gen- Our framework allows training a conditional GAN one condition at the time. This is potentially of general interest, and goes beyond the scope of replay models, since conditional GANs trained in mutli-task fashion as in Brock et al. (2019) require very large computational resources.
For our showease experiment on class-incremental MNIST learning, Fig. A8 we did not aim to compare to related work and therefore did not tune to have less weights in the hypernetwork than on the target network (for the VAE experiments, we use the same compressive setup as in the main text, see Appendix C). The GAN hypernetwork is fully-connected chunked hypernetwork with hidden layers size 25 and 25 followed by an output size of 75,000. We used learning rates for both discriminator and the generator hypernetoork of 0.0001, well as dropout of 0.4 in the discriminator and the system is trained for 10000 iterations per task. We use the Pearson Chi² Least-Squares GAN loss from Mao et al. (2017) in our experiments.
C ADDITIONAL EXPERIMENTAL DETAILS
All experiments are conducted using 16 NVIDIA GeForce RTX 2080 TI graphics cards.
For simplicity, we decided to always keep the previous task embeddings elt), ......T T 1, fixed and only learn the current task embedding e (T), In general, performance should be improved if the
14

---

Published as a conference paper at ICLR 2020
regularizer in Eq. 2 has separate copy of the task embeddings elt:*) from before learning the current task, such that e can be adapted. Hence, the targets become fi (e((e(t)), O)) and remain constant while learning task T. This would give the hypernetwork the flexibility to adjust the embeddings i.e.
the preimage of the targets and therefore represent any function that includes all desired targets in its image.
Nonlinear regression toy problem. The nonlinear toy regression from Fig. 2 is an illustrative example for continual learning problem where set of ground-truth functions (g(1) .....9' g is
given from which we collect 100 noisy training samples per task {xx,y) ) I = =g (x) + with (0, d²1), x U(((()),, where (+) denotes the input domain of task t. We set 0.05 in this experiment.
We perform 1D regression and choose the following set of tasks:
(x) x x+3
21 = |-4, -2]
(7)
00) (x) 2n=² 1
22 1,
(8)
g 3) (z) = (x - 3)
2² = [2, 4]
(9)
The target network Jug consists of two fully-connected hidden layers using 10 neurons each. For illustrative purposes use full hypemnetwrkk that generates all 141 weights of fugt at once, also being a fully-connected network with two hidden-layers of size 10. Hence, this is the only setup where we did not explore the possibility of chunked hypemetwork. We use sigmoid activation functions in both networks. The task embedding dimension was set to 2.
We train each task for 4000 iterations using the Adam optimizer with learning rate of 0.01 (and otherwise default PyTorch options) and a batch size of 32.
To test our regularizer in Fig. we set Bompar 0.005, while it is set 0 for the fine-tuning experiment in Fig. 2
For the multi-task learner in Fig. 2 we trained only the target network (no hypernetwork) for 12000 iterations with a learning rate of 0.05. Comparable performance could be obtained when training the task-conditioned hypernetwork in this multi-task regime (data not shown).
It is worth noting that the multi-task learner from Fig. D that uses no hypernetwork is only able to learn the task since we choose the input domains be non-overlapping.
Permuted MNIST benchmark. For our experiments conducted on MNIST we replicated the experimental setup proposed by van de Ven & Tolias (0119) whenever applicable. We therefore use the same number of training iterations, the same or lower number of weights in the hypernetwork than in the target network, the same learning rates and the same optimizer. For the replay model, i... the hypernetwork-eent -empoweree VAE, well as for the standard classifier we used 5000 training iterations per task and learning rate is set to 0.0001 for the Adam optimizer (otherwise PyTorch default values). The batchsize is set to 128 for the VAE whereas the classifier simultaneously trained on batch of 128 samples of replayed data (evenly distributed over all past tasks) and batch of 128 images from the currently available dataset. MNIST images are padded with zeros, which results in network inputs of size 32 x 32, again strictly following the implementation of the compared work. We experienced better performance when we condition our replay model on a specific task input. We therefore construct for every task specific input namely a sample from a standard multivariate normal of dimension 100. In practice we found the dimension to be not important. This input stays constant throughout the experiment and is not leamed.. Note that the same hyperparameters for all learning scenarios, which is not true for the reported related work since they have tuned special hyperparameters for all scenarios and all methods.
Details of hypernetwork for the VAE. We use one hypernetwork configuration to generate weights for all variational autoencoders used for our PermuteMMNIST-ID 10 experiments namely a fully-connected chunked hypernetwork with 2 hidden layers of size 25 and 25 followed by output size of 85,000. We use ELU nonlinearities in the hidden layers
15

---

Published as a conference paper at ICLR 2020
a
b
PerMutedIMNIST-100
PermutedMNiST-100
100
100
Accuracy [3%]
Accuracy [%6]
75
90
96.78%
95.66%
50
1
50
100
80
Task
50
100
Task f
Poutuut
Pautpor
Suuppr 0.001
Poutuu 0.5
Sropper 0.0005
125)
200
7500, Sauput 0.01
Poutrut
Paupor 0.005
125)
200
Rasput 0.01
C
PermutedMMNIT--10
100
Accuracy [%)]
90
0.0005
0.005
0.05
0.5
0.001
0.01
0.1
Boupur
during
final
Figure A2: Additional experiments on the PermutedMNIST-100 benchmark. (a) Final test set classification accuracy on the i-th task after learning one hundred permutations (PermutedMNISTT- 100). All runs use exactly the same hyperparameter configuration except for varying values of Bousput- The final accuracies are robust for wide range of regularization strengths. If Soupuu is too weak, forgetting will occur. However, there is no severe disadvantage of choosing Soupptt too high (emp. (c)). too high Sautpur simply shifts the attention of the optimizer away from the current task, leading to lower baseline accuracies when the training time is not increased.
(b) Due an increased number of output neurons, the target network for PermutedMNIST-I 100 has more weights than for PermutedMNIST-IIP 10 (this is only the case for CL1 and CL3). This plot shows that the performance drop minor when choosing a hypernetwork with a comparable number of weights as the target network in CL2 (orange) compared one that has similar number of weights the target network for CL1 in PermutedMNIST. 100 (red). (c) Task-averaged test set accuracy after learning all tasks (labelled 'final', in red) and immediately after learning task (labelled 'during', in purple) for the runs depicted in (a). For low values of Buuppt final accuracies are worse than immediate once (forgetting occurs). If Poupur is too high, baseline accuracies decrease since the optimizer puts less emphasis on the current task (note that the training time per task is not increased). Shaded areas in (a) and (b) denote STD, whereas error bars in (c) denote SEM (always across 5 random seeds).
16

---

Published as a conference paper at ICLR 2020
of the hypemnetwokk. The size of task embeddings has been set to 24 and the size of chunk embeddings с to 8. The parameter Beupur is 0.05 The number of weights in this (2,211,691 network weights 216 task embedding weights).
hypemettwrk is The corresponding target network (and therefore output of the chunked hypernetwork), as taken from related work, has 2,227,02 ,024 weights.
Details of the VAE for HNET+TIR. For this variational autoencoder, we use two fully- connected neural networks with layers of size 1000, 1000 for the encoder and 1000, 1000 for the decoder and a latent space of 100. This setup is again copied from work we compare against.
Details of the VAE for HNET+R. For this variational autoencoder, we use two fully- connected neural networks with layers of size 400, 400 for the encoder and 400, 400 for the decoder (both 1000, 1000 in the related work) and a latent space of dimension 100. Here, we departure from related work by choosing smaller architecture for the autoencoder. Note that we still use a hypernetwork with less trainable parameters than the target network (in this case the decoder) that is used in related work.
Details of the hypernetwork for the target classifier in PermutedMINIST-II (HNET+TIR & HNET+ENT), We use the same setup for the hypernetwork as used for the VAFs above, but since the target network is smaller we reduce the output of the hypernetwork to 78,000. We also adjust the parameter Bompus to 0.01, consistent with our PermutedMNIST-IOO 100 experiments. The number of weights this hypernetwork is therefore 2,029,931 parameters (2,029,691 network weights + 240 task embedding weights). The corresponding target network (from related work) would have 2, 2.126,100 weights for CL1 and CL3 and 2,036,010 for CL2 (only one output head).
Details of the hypernetwork for the target classifier for PermutedMNIST--10. For these experiments we chose an architecture that worked well on the PermutedMNIST-IO benchmark and did not conduct any more search for new architectures. For PermutedMNIST. 100, the reported results were obtained by using a chunked hypernetwork with 3 hidden layers of size 200, 250 and 350 (300 for CL2) and an output size of 7500 (6000 for CL2) (such that we approximately match the corresponding target network size for C1.1/C12//1.33. Interest- ingly, Fig. A2 shows that even if we don't 't adjust the number of hypernetwork weights to the increased number of target network weights, the superiority of our method is evident. Aside from this, the plots in Fig. 3 have been generated using the PermutedMNIST-ID HNET+TIR setup (note that this includes the conditions set by related work for PermutedMNIST-II e.g., target network sizes, the number of training iterations, learning rates, etc.).
Details of the VAE and the hypernetwork for the VAE in PermutedMNISTT-10 for CL2/CL3, We use very similar setup for the VAE and it's hypernetwork used in HNET+TIR for PermutedMNIIT-100 as described above. We only applied the follow- ing changes: Fully-connected hypernetwork with one hidden layer of size 100; chunk embedding sizes are set to 12; task embedding sizes are set two 128 and the hidden layer sizes of the VAE its generator are 400, 600. Also we increased the regularisation strength Buuput = 0.1 for the VAE its generator hypernetwork.
Details of the target classifier for HNET+TIR & HNET+ENT, For this classifier, use the same setup as in the study we compare to (van de Ven & Tolias 2019), i.c., a fully-connected network with layers of size 1000, 1000. Note that if the classifier is used as a task inference model, it is trained on replay data and the corresponding hard targets, i.e., the argmax of the soft targets.
Below, we report the specifications for our automatic hy hyperparameter search (if not noted otherwise. these specifications apply for the split MNIST and split CIFAR experiments as well):
Hidden layer sizes of the hypemetwork: (no hidden layer), "5,5" "10,10", "25,25", "50,50", "100, 100", "10", "50", "100"
Output size of the hypernetwork: fitted such that we obtain less parameters then the target network which we compare against
Embedding sizes (for e and c): 8, 12, 24, 36, 62, 96, 128
A 0.0005, 0.001, 0.005, 0.01, 0.005, 0.1, 0.5, 1.0
17

---

Published as a conference paper at ICLR 2020
Hypernetwork transfer functions: linear, ReLU, ELU, Leaky-ReLU
Note that only random subset of all possible combinations of hyperparameters has been explored.
After we found a configuration with promising accuracies and similar number of weights compared to the original target network, we manually fine-tuned the architecture to increase/decrease the number of hypernetwork weights to approximately match the number of target network weights.
The choice of hypernetwork architecture seems to have strong influence the performance. might be worth exploring alternatives, e.g., an architecture inspired by those used in typical generative models. We note that in addition to the above specifications we explored manually some hyperparameter configurations to gain a better understanding of our method.
Split MNIST benchmark. Again, whenever applicable we reproduce the setup from van de Ven & Tolias 2019). Differences to the PermutedMNIST-10 experiments are just the learning rate (0.001) and the number of training iterations (set to 2000).
Details of hypernetwork for the VAE. We use one hypernetwork configuration to generate weights for all variational autoencoders used for our split MNIST experiments, namely a fully-connected chunked hypernetwork with 2 hidden layers of size 10, 10 followed by an output size of 50,000. We use ELU nonlinearities the hidden layers of the hypernetwork. The size of task embeddings e has been set to 96 and the size of chunk embeddings to 96. The parameter Soupat is 0.01 for HNET+R and 0.05 for HNET+TIR. The number of weights in this hypernetwork is 553,576 (553,192 network weights 384 task embedding weights). The corresponding target network (and therefore output of the chunked hypernetwork), as taken from related work, has 555,184 weights. For qualitative analyses of the replay data of this VAE (class incrementally learned), see A8
Details of the VAE for HNET+TIR. For this variational autoencoder, we use two fully- connected neural networks with layers of size 400, 400 for the encoder and 50, 150 for the decoder (both 400, 400 in the related work) and a latent space of dimension 100.
Details of the VAE for HNET+R. For this variational autoencoder, we use two fully- connected neural networks with layers of size 400, 400 for the encoder and 250, 350 for the decoder (both 400, 400 in the related work) and a latent space of dimension 100.
Details of the hypernetwork for the target classifier in split MNIST (HNET+TIR & HNET+ENT), We use the same setup for the hypernetwork as used for the VAE above, but since the target network is smaller we reduce the output of the hypernetwork to 42,000. We also adjust the Boutput to 0.01 although this parameter seems to not have strong effect on the performance. The number of weights in this hypernetwork is therefore 465,672 parameters (465,192 network weights + 480 task embedding weights). The corresponding target network (from related work) would have 478,410 weights for CL1 and CL3 and 475,202 for CL2 (only one output head).
Details of the target classifier for HNET+TIR & HNET+ENT. For this classifier, again use the same setup as in the study we compare to (van de Ven & Tolia 2019). i.c., fully-connected neural networks with layers of size 400, 400. Note that if the classifier is used as a task inference model, it is trained on replay data and the corresponding hard targets, i.e., the argmax the soft targets.
Split CIFAR-10/100 benchmark. For these experiments, we used target network ResNet-32 network (He et al. (2016) and again produce the weights of this target network by a hypernetwork in compressive manner. The hypernetwork in this experiment directly maps from the joint task and chunk embedding space (both dimension 32) to the output space of the hypernetwork, which is of dimension 7,000. This hypernetoork has 457,336 parameters (457, 144 network weights + 192 task embedding weights). The corresponding target network, the ResNet-32, has 468.540 weights (including batch-norm weights). We train for 200 epochs per task using the Adam optimizer with an initial learning rate of 0.001 (and otherwise default PyTorch values) and a batch size of 32. In addition, we apply the two learning rate schedules suggested in the Keras CIFAR- 10 example?
See https://keras. latesss..oeexmmmma o/ weexamplea/cae ea/
far ICeenett
18

---

Published as a conference paper at ICLR 2020
Due to the of batch normalization, have find appropriate way handle the running statistics which are estimated during training. Note, these are not parameters which are trained through backpropagation. There different ways how the running statistics could be treated:
1. One could ignore the running statistics altogether and simply compute statistics based the current batch during evaluation.
2. The statistics could part of the hypernetwork output. Therefore, one would have to manipulate the target hypemettwrk output of the previous task, such that the estimated running statistics of the previous task will be distilled into the hypernetwork.
3. The running statistics can simply be checkpointed and stored after every task. Note, this method would lead to a linear memory growth in the number of tasks that scales with the number of units the target network.
For simplicity, we chose the last option and simply checkpointed the running statistics after every task.
For the fine-tunne fine-tuning results in Fig. we just continually updated the running statistics (thus, we applied checkpointing).
D ADDITIONAL EXPERIMENTS AND NOTES
Split CIFAR-10/100 benchmark using the model of Zenke et al. (2017). We re-run the split CIFAR-10/ 1001000 experiment reported on the main text while reproducing the setup from Zenke et al. (017)) Our overall classification performance is comparable synaptic intelligence, which achieves 73.85% lask-averaged test set accuracy, while our method reaches 71.29% + 0.32%, with initial
baseline performance being slightly worse in our approach, Fig. A3
CIF1R-10/100
100
75
75
78
09
7575
43
Acouracy [9g]
54
50
31
25
CIFAR-10
Task
hnet during
hnet
from scratch
fine-tuning
SI
Figure A3: Replication of the split CIFAR-10/100 experiment of Zenke et al. (2117). Test set accuracies on the entire CIFAR-10 dataset and subsequent CIFAR-100 splits. Both task-conditioned hypernetworks (hnet, in red) and synaptic intelligence (SI, in green) transfer information forward and are protected from catastrophic forgetting. The performance of the two methods comparable. For completeness, we report our test set accuracies achieved immediately after training (hnet-during, in blue), when training from scratch (purple), and with our regularizer turned off (fine-tuning, yellow).
To obtain results, size 5500. The size of task embeddings e has been set to 48 and the size of chunk embeddings c to 80. The parameter Soupput 0.01 and the learning rate is set 0.0001.
hypernetwork with 3 hidden-layers of sizes 100, 150, 200 and output
The number of weights in this hypernetwork is 1,182,678 (1,182,390 network weights + 288 task embedding weights). The corresponding target network would have 1,276,508 weights.
In addition to the above specified byperparameter search configuration we also included the following learning rates: 0.0001, 0.0005, 0.001 and manually tuned some architectural parameters.
19

---

Published as a conference paper at ICLR 2020
b
PermuledMNIST--00- CL2
PermuledMNNST--100 CL3
100
100
Accuracy (%)
75
Accuracy (%)
75
50
25
25
Task t
Task
HNET+TIR SI
DGR-distill
Online EWC
HNET+TIR
DGR-distill
Online EWC
Figure A4: Context-free inference using hypernetwork-pee prrotteee replay (HNET+TIR) on long task sequences. Final test set classification accuracy on the (-th task after learning one hundred permutations of the MNIST dataset [PermutedMNIST--OO where task identity is not explicitly provided to the system. As before, the number of hypernetoork parameters is not larger than that of the related work we compare to. (a) HNET+TIR displays almost perfect memory retention. We used a stochastic regularizer (cf. Appendix D note below) which evaluates the output regularizer Eq-2 only for random subset of previous tasks (here, twenty). (b) HNET+TIR is the only method that is capable of learning PermutedMNIST--UU 100 in this learning scenario. For this benchmark, the input data domains are easily separable and the task inference system achieves virtually perfect (-100%) task inference accuracy throughout, even for this long experiment. HNET+TIR uses a ewvide-and-conquee strategy: if task inference is done right, CL3 becomes just CL1. Furthermore, once task identity is predicted, the final softmax computation only needs to consider the corresponding task outputs in isolation (here, of size 10). Curiously, for HNET+TIR, CL2 can be harder than CL3 as the single output layer (of size 10, shared by all tasks) introduces a capacity bottleneck. The related methods, on the other hand, have to consider the entire output layer (here, of size 10a 100) once, which is known to harder train sequentially. This leads to overwhelming error rates on long problems such as PermutedMNISTT 100. Shaded areas in (a) and (b) denote STD 5).
for the CL2 (a) and CL3 (b) scenarios,
20

---

Published as a conference paper at ICLR 2020
Upper bound for replay models. We obtain upper bound for the replay-based experiments (Table by sequentially training a classifier, in the same way as for HNET+R and DGR, now using true input data from past tasks and synthetic, self-generated target. This corresponds to the rehearsal thought experiment delineated Sect.
Table 2: Task-averaged test accuracy (# SEM, n. 20) on the permuted (PP0') and split ('S' MNIST experiments. For HNET+R and +GR+distill (van de Ven & Tolias, 2019) the classification network is trained sequentially data from the current task and replayed data from all previous tasks. Our + comes close to saturating the corresponding replay upper bound RPL-UB.
DGR
HNET+R
RPL.UBB
PIO-CL1
97.51 1 0.01
97.85 1 0.02
97.89 1 0.02
PI0-CL2
9.00 9.02
97.72 0.01
PI0-CL3
96.38
0.06
97.91 0.01
S-CL1
99.61
0.01
99.83 0.01
S-CL2
0.20
98.96 0.08
S-CL3
9
94.97 0.18
98.38 0.02
Quantification forgetting in our continual learning experiments. In order to quantify forget- ting of our approach, we compare test set accuracies of every single task directly after training with it's test set accuracy after training on all tasks.
Only CL1 is shown since other scenarios i.e. CL2 and CL3 depend on task inference which only is measurable after training on all tasks.
Table 3: Task-averagee test accuracy (# SEM. n = 20) on the permutedMNNIST-10 ('P10') and splitMNIST ('S') experiments during and after training.
HNET+TIR HNET+TIR HNET+R during
HNET+R after
after
during
S-CL1
99.79 + 0.01 99.79 1 0.01 99.82 1 0.01
99.83 0.01
PI0-CL1 97.58 + 0.02 97.57 + 0.02 98.03 + 0.01
97.87 0.01
Table 4: aask-averaged test accuracy (# SEM, 5) the permutedMNIST-1000 ('P100")
experiments during and after training.
HNET+TIR HNET+TIR
during
after
P100-C11
96.12 + 0.08
96. 18 0.09
P100-1122
95.97
P100-CL3
96.00 0.03
Table 5: Task-averaged test accuracy (# SEM, 5) split CIFAR-10/100 CLI on two different target network architectures.
during
after
ZenkeNet
74.75 + 0.09
71.29 + 0.32
ResNet-32
82.36 + 0.44
82.34 = 0.44
Robustness of chopce. In Fig, A2 and Fig. A2 provide additional experiments for method on PermutedMNISTT We show that our method performs comparable for wide range of (ouspu- values (including the one depicted in Fig. 3)).
21

---

Published as a conference paper at ICLR 2020
b
PermutedIMNIST-100
PermutedMNIST-100
100
100
Accuracy [%)
75
Accuracy [%)
90
50
80
70
1
50
100
60
Task
hnet
10
50
250
1000
5000
10000
25
100
500
2500
7500
hnet
A 50
A 1000
A=1
A 100
A 5000
X
10000
during
linsl
d
PermutedMNNITT-00
PermutedMNNSTTIN
100
100
Accuracy [%)
Accuracy [%]]
70
40
10
50
100
hnet
hnet fine-tuning
fine-tuning
Task
during
final
hnet
hnnet line-tuning
line-tuning
Figure A5: Additional experiments with online EWC and fine-tuning the PermutedMNIST. 100 benchmark. (a) Final test set classification accuracy on the f-th task after learning one hundred permutations (PermutedMNIST-IOO 100) using the online EWC algorithm (Schwarz et 2018 to prevent forgetting, All runs use exactly the same hyperparameter configuration except for varying values of the regularization strength X. Our method (hnet, in red) and the online EWC run in orange) from Fig. Bh are shown for comparison. It can be seen that even when tuning the regularization strength one cannot attain similar performance as with our approach (cmp. Fig. (2 Too strong regularization prevents the learning of new tasks whereas too weak regularization doesn't 't. prevent forgetting. However, a middle ground (e.g., using
100,
100) does not reach acceptable per-task performances. (b) Task-averaged test set accuracy after learning all tasks (labelled 'final', in red) and immediately after learning task (labelled 'during', in purple) for a range regularization strengths À when using the online EWC algorithm. Results are complementary those shown in (a). (c) Final test set classification accuracy on the t-th task after learning one hundred permutations (PermutedMNNIST 100) when applying munitunng the hypernetwork (labelled 'hnet fine-tuning" in blue) or target network (labelled 'fine-tuning,, in green). Our method (hnet, in red) from Fig. 33 shown for comparison. It can be that without protection the hypernetwork suffers much more severely from catastrophic forgetting as when training a target network only. (d) This plot is complementary to (c). See description of (b) for explanation of the labels. Shaded areas in (a) and (c) denote STD, whereas error bars in (b) and (d) denote SEM (always across random seeds).
22

---

Published as a conference paper at ICLR 2020
a
b
PormutedMMNIT--100
PormutedMNNST-100
100
100
Accuracy [%6]
Accuracy (%)
75
75
50
50
10
100
1000
10000
0.001
10000
50
500
0.005
0.05
0.5
50
500
5000
Online EWC: A
Sl: C
0.5
0.7
during
final
7 0.8
Figure A6: Hyperparameter search for online EWC and SI on the PermutedMNNITT- bench- mark. We conduct the same hyperparameter search performed in van de Ven & Tolias (2018). We did not compute different random seeds for this search. (a) Hyperparameter search on the regular- isation strength for the SI algorithm. Accuracies during and after the experiment shown. (b) Hyperparameter search for parameters À and y of the online EWC algorithm. Only accuracies after the experiment are shown.
Varying the regularization strength for online EWC. The performance of online EWC in Fig. Ba is closest to our method (labelled hnet, in red) compared to the other methods. Therefore, we take closer look this method and show that further adjustments of the regularization strength À do not lead to better performance. Results for a wide range of regularization strengths can be seen in Fig. A5 and Fig. AS.. As shown, online EWC cannot attain performance comparable to our method when tuning the regularization strength only.
The impact of catastrophic forgetting on the hypernetwork and target network. We have successfully shown that by shifting the continual learning problem from the target network to the hypernetwork we can successfully overcome forgetting due to the introduction of our regularizer in Eq. 2 We motivated this success by claiming that it an inherently simpler task remember few input-output mappings in the hypernetwork (namely the weight realizations of each task) rather than the massive number of input-output mappings (x1( of each task by the target network.
y y(t.))) } associated with the remembering
Further evidence of this claim is provided by fine-tuning experiments in Fig. A3 and Fig. A5 Fine-tuning refers sequentially learning neural network on set of tasks without any mechanism in place to prevent forgetting, It is shown that fine-tuning a target network (no hypernetwork in this setup) has no catastrophic influence on the performance of previous tasks. Instead there is graceful decline in performance. On the contrary, catastrophic forgetting has an almost immediate affect when training a hypernetwork without protection (i.e., training our method with Soupur O. The performance quickly drops to chance level, suggesting that if we weren't 't solving a simpler task then preventing forgetting the hypernetwork rather than in the target network might not be beneficial.
Chunking and hypernetwork architecture sensitivity. In this note we investigate the perfor- mance sensitivity for different conleecconee hypernetwork architectures on split MNIST and PermutedMNIST-100, Fig. A7 We trained thousands of randomly drawn architectures from the following grid (the same training hyperparameters as reported for for CL1, see Ap- pendix C were used throughout): possible number of hidden layers 1, 2, possible layer size 5, 10, ,0, 90, 100, possible chunk embedding size 8, 12, 24, 56, 96 and hypernetwork output size in {10, 50, 100, 200, 300, 400, 500, 750, 1k, 2k: ..... 9k, 10k, 20k, 30k, 40k). Since we realize compression through chunking, sort our hypernetwork architectures by compression ratio, and consider only architectures with small compression ratios.
Performance of split MNIST stays in the high 90 percentages when reaching compression ratios close to 1% whereas for PermutedMNIST-IU accuracies decline in a non-linear fashion. For both experiments, the choice of the chunked hypernetwork archicture is robust and high performing even in
23

---

Published as a conference paper at ICLR 2020
a
b
splttMNISS ceppressioo-perormaane trade-off for random hnet architectures
permutedMNIST compression-perfrrmatte trade-off
for random hnet architectures
100
100
Accuracy [%)
80
Accuracy [%6]
80
60
40
40
20
0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
Compression ratio
Compression ratio
Figure A7: Robustness to hypernetwork architecture choice for large range of compression ratios. Performance vs. compression for random hypernetwork architecture choices, for split MNIST and PermutedMNIST-11 (mean = STD, 500 architectures per bin). Every model was trained with the same setup (including all hyperparameters) used to obtain results reported in Table I (CL1). We considered architectures yielding compression ratios |0, { Ogg [0.01, 2.0) (a) split MNIST performance for CL1 stays high even for compression ratios = 1%. (b) PermutedMNIST-I accuracies degrade gracefully when compression ratios decline to 1 Notably, for both benchmarks, performance remained stable across large pool of hypernetwork configurations.
the compressive regime. Note that the discussed compression ratio compares the amount of trainable parameters the hypernetwork to its output size, i.e. the parameters of the target network.
Small capacity target networks for the permuted MNIST benchmark. Swaroop al. (2018) argue for using only small capacity target networks for this benchmark. Specifically. they propose to use hidden layer sizes [100, 100]. Again, we replicated the setup of van de Ven & Tolias (2019) wherever applicable, except for the now smaller hidden layer sizes of [100, 100] in the target network. We use a fully-comnected chunked hypemeetwrkk with chunk embeddings having size 12, hidden layers having size 100, 75, 50 and output size of 2000. resulting in total number of hypernetorrk weights of 122,459 (including 10 x 64 task emhedding weights) compared to 122,700 weights that generated for the target network. Pouipu is 0.05. The experiments performed here correspond to CL.1.
We achieve an average accuracy of 93.91 + 0.04 for PermutedMNIST-ID after having trained on all tasks. In general, we saw that the hypernetwork training can benefit from noise injection. For instance, when training with soft-targets (i.e., modified the 1-hot target to be 0.95 for the correct class and 1-0.95 0.03.
Fclasses I for the remaining classes), we could improve the average accuracy to 94.24 +
We also checked the challenging PermutedMNIST-S benchmark with this small target network as previously investigated by Ritter et al. (2018). Therefore, we slightly adapted the above setup by using hypernetwork with hidden layer sizes [100, 100) and a regularization strength of Boupur 0.1.
This hypernetwork is slightly bigger than the corresponding target network [0.u(e(")) 1.37. With this configuration, we obtain an average accuracy of 90.91 0.07.
Comparison HAT. Serra al. (2018) proposed the hard attention to the task (HAT) algorithm, strong CL1 method which relies learning per-task, per-neuron mask. Since the masks pushed to become binary, HAT can be viewed as an algorithm for allocating subnetworks (or modules) within the target network, which become specialized solve given task. Thus, the method is similar to ours in the sense that the computation of the target network task-dependent, but different in spirit, it relies network modularity.
In HAT, task identity assumed to be provided, so that the appropriate mask can be picked during inference (scenario CL.1). HAT requires explicitly storing neural mask for each task, whose size
24

---

Published as a conference paper at ICLR 2020
scales with the number of neurons in the target network. In contrast, method allows solving tasks in compressive regime. Thanks to the hypernetoor,, whose input dimension can be freely chosen, only a low-dimensions
embedding needs to be stored per task (cf. Fig. 4. and through chunking is possible to learn to parameterize large target models with a small number of plastic weights (cf. Fp).
Here, we compare our task-conditioned hypernetworks to HAT on the permuted MNIST benchmarks (T' 10 and T = 100), cf. Table 6 For large target networks, both methods perform strongly, reaching comparable final task-averaged accuracies, For small target network sizes, task task-conditioned hypernetworks perform better, the difference becoming more apparent on PermutedMNNIT--100
We note that the two algorithms use different training setups. In particular, HAT uses 200 epochs (batch size set to 64) and applies learning rate scheduler that acts on held out validation set. Furthermore, HAT uses differently tuned forgetting hyperparameters when target network sizes change. This important to control for the target network capacity used per task and assumes knowledge of the (number of) tasks at hand. Using the code freely made available by the authors, we were able to rerun HAT for our target network size and longer task sequences. Here, we used the setup provided by the author's code for HAT-Large for PermutedMNISTT-DD 10 and PermutedMNIST--100 To draw a fairer comparison, when changing our usual target network size to match the ones reported in Serra et al. (2018), we trained for 50 epochs per task (no training loss improvements afterwards observed) and also changed the batch size to 64 but did not changed our training scheme otherwise; in particular, did not learning rate scheduler.
Table 6: Comparison of HNET and HAT, Serra et al. 2018). Task-averaged test accuracy on the PermutedMINIST experiment with T = 10 and T 100 tasks ('P10', 'P100') with three different larget network sizes, i.e., three fully connected neural networks with hidden layer sizes of (100, 100) or (500, 500) or (2000, 2000) are shown. For these architectures, single accuracy was reported by Serra et al. 2018) without statistics provided. We reran HAT for PermutedMNISTT-100 100 with code provided at https: / giw..u......... com/ joans 1/hat| and for PermuteMMNIST-10 with hidden layer size (1000, 1000) to match our setup. HAT and HNET perform similarly on large target networks for PermutedMNIST-I0 10, while HNET is able to achieve larger performances with smaller target networks well as for long task sequences.
HAT
HNET
PI0-100, 100
91.6
95.92 0.02
P10-500, 500
97.4
97.35 + 0.02
PI0-2000,
98.6
98.06 + 0.02
P10-1000, 1000
97.67 1 0.02
97.56 1 0.02
P100-1000, 1000
86.04 1 0.26
94.98 1 0.07
Efficient PermutedMNNIST-255 experiments with a stochastic regularizer on subsets of previ- ous tasks. An apparent drawback of Eq. is that the runtime complexity of the regularizer grows linearly with the number of tasks. To overcome this obstacle, we show here that it is sufficient to consider a small random subset of previous tasks.
In particular, we consider the PermutedMNNIST-225 benchmark (250 tasks) on CL1 using the hyper- parameter setup from our PermutedMNIST-IM 100 experiments except for hypernetoork output size of 12000 (to adjust to the bigger multi-head target network) and regularization strength Poupuu = 0.1. Per training iteration, we choose maximally 32 random previous tasks to estimate the regularizer from Eq. 2 With this setup, we achieve a final average accuracy of 94.19 + 0.16 (compared to an average during accuracy (i.e., the accuracies achieved right after training on the corresponding task) of 95.54 + 0.05). All results are across 5 random seeds. These results indicate that full evaluation of the regularizer at every training iteration is not necessary such that the lincar runtime complexity can be cropped to constant one.
Combining hypernetwork output regularizers with weight importance. Our hypernetwork reg- ularizer pulls uniformly in every direction, but it is possible to introduce anisotropy using EWC-like approach (Kirkpatrick et al. (017). Instead of weighting parameters, hypernetwork outputs can be weighted. This would allow for more flexible regularizer, the expense of additional storage.
25

---

Published as a conference paper at ICLR 2020
Task inference through predictive entropy (HNET+ENT). In this setup, rely the capa- bility of neural networks to separate in- from dittr--distribution data. Although this is difficult research problem on its own, for continual learning, face a potentially simpler problem, namely to detect and distinguish between the tasks our network was trained on. We here take the first minimal step exploiting this insight and compare the predictive uncertainty, as quantified by output distribution entropy, of the different models given an input. Hence, during test time we iterate over all embeddings and therefore the models our metamodel can generate and compare the predictive entropies which results in making a prediction with the model of lowest entropy. For future work, we wish to explore the possibility of improving our predictive uncertainty by taking parameter uncertainty into account through the generation of approximate, task-specific weight posterior distributions.
Learning without task boundaries with hypernetworks. An interesting problem we did not address in this paper is that of learning without task boundaries. For most CL methods, it crucial to know when learning one task ends and training of a new tasks begins. This is no exception for the methods introduced in this paper. However, this is not necessarily realistic or desirable assumption; often, one desires to learn in an online fashion without task boundary supervision, which is particularly relevant for reinforcement learning scenarios where incoming data distributions are frequently subject to change (Rolnick et al. 2018). At least for discrete changes, with our hypernetwork setup, this boils down to a detection mechanism that activates the saving of the current model, i.e., the embedding c(T), and its storage to the collection of embeddings {c( We leave the integration of model with such a hypernetwork-syie -spccffcccc switching detection mechanism for future work. Interestingly, our task-conditioned hypernetworks would fit very well with methods that rely on fast remembering (a recently proposed approach which appeared in parallel to our paper, He et al. 2019).
E UNIVERSAL FUNCTION APPROXIMATION WITH CHUNKED NEURAL NETWORKS
Proposition 1. Given compact subset K R and continuous function on K i.e. f E C(K), more specifically, f: + R" with n = r Ne. Now Ve > O, there exists a chunked neural network h Rm x C I2' with parameters On discrete set C [CI,,..., CNc) and C₁ R such that [f(()) f(x) < E, Vx E K and with f((x) = [f c1)..... (x(x,
For the following proof, we assume the existence of one form of the universal approximation theorem (UAT) for neural networks (Leshno & Schocken 1993. Hanin 2017). Note that we will not restrict ourselves to a specific architecture, nonlinearity, input or output dimension. Any neural network that proven be a universal function approximator is sufficient.
Proof. Given any > O, we assume the existence of a neural network Sn Rm - R that approximates function f on K:
- -xx))=
< Vz E K.
(10)
We will in the following show that we can always find a chunked neural network fy : R x C + R" approximating the neural network fú K and conclude with the triangle inequality
[x)()) [(x) (x) +[n
fxx) <,, Vo E K.
(11)
Indeed, given the neural network fn such that (10) holds true, we construct
(x c) = [ [ (x)
C2
(12)
else
by splitting the full neural network fı(x)) [f (x), f (x)...... 1 (x)) with fu. : R x C + Rr.
Note that fn is continuous on Rm x C with the product topology composed of the topology on R.
induced by the metric J. .-. R x I IR and the discrete topology on C. Now we can make
26

---

Published as a conference paper at ICLR 2020
of the UAT again: Given the compact K R", the discrete set C {c1, CNc} and any =f >
there exists neural network function fh 18 R + R' such that
(x, (f.( (x(x,))
Vx Kx
(13)
2NC
It follows that
Σ (x) fx(,c,)) < Σ
a K,
(14)
2Nc 2'
which is equivalent to
c1)
c1)
- [f((x) fx(()
2 Vx V K.
(15)
[x()
(x..)
We have shown
11 which concludes the proof.
Note that we did not specify the number of chunks Nc, r or the dimension s of the embeddings c;- Despite this theoretical result, emphasize that we not aware of constructive procedure to define chunked hypernetwork that comes with a useful bound on the achievable performance and/or compression rate. We evaluate such aspects empirically in our experimental section.
27

---

Published as a conference paper at ICLR 2020
F
QUALITATIVE ANALYSES OF HYPERNETWORK REPLAY MODELS
a
b
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
1
1
}
1
1
1
1
1
1
1
1
1
/
1
/
1
1
2
2
2
4
2
2
2 2
2
2
2
2
2
2
a 2 2
a
2
2
2
2
2
2
a
2
2
2
2
2
2
2
2
2
>
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3 3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
5
5
5
5
5
4 4
12
4
J. 4
5
5
5
5
4 4 H
4
4
4.
9
5
5
5
5
5 5 5
5
5
$ 9
5 5 5
4
4
4
4
4
4
4
5 5 5
8
8
5
5
4
4
4
4
H
4
6 5 5
0
3 8
3
H 4
4
H
4
h
h
6
@
@
6 6
6
9
6
9
6 6
6
9
7
3
6
4
6
6
6
9
9
3
9
7 7
#
9
9
7
7
7
7
7
$/ 7
7
7
7
7
7
7
7
7
7
1 4
7 7
7
7
7
$
$
8
8
8
8
8
8
8 5 5
5
8
8
8
8
8
8
8
6
g
8
8
9
8
9
9
9
9
4
9
9
9
9 9
7
9
9
Figure A8: Image samples from hypernetwork- protected replay models. The left column of both of the subfigures display images directly after training the replay model on the corresponding class, compared to the right column(s) where samples are obtained after training on eights and nines i.e. all classes. (a) Image samples from class-ines insremmnccemetlllyy trained VAE. Here the exact same training configuration to obtain results for split MNIST with the HNET+R setup used, see Appendix C (b) Image samples from a ilassinncrementallyy trained GAN. For training configurations, see Appendix B In both cases the weights of the generative part, i.c., the decoder the generator, produced and protected by a hypemeework.
28


Reviews:
[{"review_id": "rygAU3qTtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review of “Continual learning with hypernetworks”\n\nThis paper investigates the use of conditioning “Hypernetworks” (networks where weights are compressed via an embedding) for continual learning. They use “chunked” version of the hypernetwork (used in Ha2017, Pawlowski2017) to learn task-specific embeddings to generate (or map) tasks to weights-space of the target network.\n\nThere is a list of noteworthy contributions of this work:\n\n1) They demonstrate that their approach achieves SOTA on various (well-chosen) standard CL benchmarks (notably P-MNIST for CL, Split MNIST) and also does reasonably well on Split CIFAR-10/100 benchmark. The authors have also spent some effort to replicate previous work so that their results can be compared (and more importantly analyzed) fairly to the literature, and I want to see more of this in current ML papers. (one note is that the results for CIFAR-10/100 is in the Appendix, but I think if the paper gets accepted, let's bring it back to the main text and use 9 pages, since the results for CIFAR10/100 are substantial).\n\n2) In addition to demonstrating good results on standard CL benchmarks, they also conduct analysis of task-conditioned hypernetworks with experiments involving long task sequences to show that they have very large capacities to retain large memories. They provide a treatment (also with visualization) into the structure of low-dim task embeddings to show potential for transfer learning.\n\n3) The authors will release code to reproduce all experiments, which I think is important to push the field forward. Future work can not only reproduce this work, but also the cited works.\n\nThe work seems to be well written, and the motivation of using hypernetworks as a natural solution to avoid catastrophic loss is clearly described. Overall, I think this work is worthy of acceptance and should encourage more investigation into hypernetworks for CL and transfer learning going forward in the community.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "rkebr3IhKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper 1872\nPaper proposes a method for CL. The method is based on hypernetworks. These networks are a metamodel, which produce the parameters (from a task-conditioned embedding) which will be used in the main network. Preventing forgetting in the main network is now, replaced by preventing forgetting in the hypernetwork. This is done by imposing a regularization on the hypernetwork outcome, imposing that the generated weights should be similar for previous tasks (similar to Li & Hoiem who impose this on the network outputs). In addition, the paper proposes chunking, which refers to using an additional set of chunk, embeddings which are shared for all tasks, which allow compressing the hypernetwork. Furthermore, they propose an extension that allows for image replay (this is not an easy extension and an impressive contribution on itself, but maybe confusing for the current paper).\n\nCONCLUSION\nOverall, I like the idea of the paper and it is well explained. However, I found that the experiments of the paper where not well designed to verify the main contribution (hypernetworks), nor where they compared to the most relevant methods. I am borderline with this paper, and recommend borderline accept (borderline not being an option).\n\nQUESTIONS\n1. I think the motivation of why hypernetworks are expected to have less forgetting (than addressing forgetting directly in a network) should be discussed early in the paper. \n\n2.I do not understand why the training is performed in two steps. First computing a candidate Delta THETA_H and then o ptimizing Eq 2. Why not directly optimizing Eq 2, replacing the second factor with|| f_h(e^t, THETA^*_h)-f_h(e^t, THETA_h) ||. This is how this regularization is normally applied (e.g. Li & Hoiem). If the authors insist in using Eq 2, I would like to see it compared with the proposed version. \n\n3. The experiments should show that hypernets better address CL then addressing this directly in the network (and preferably provide reasons for this). Comparison with the closest methods like HAT and PackNet should be included. Especially, HAT is interesting since it is also based on an embedding. \n\n4. Also, more experiments on CIFAR would be welcome. The MNIST variations already provide very high accuracies. For CIFAR-10/100 groups of 20 classes are added in 5 steps ? Scenario CL3 would be interesting for CIFAR as well. \n\n5. I would like to see more analysis and results for the chunking. (As said before the replay is also a nice addition, but it seems an add-on of the main-text, shrinking the space to analyze the main contributions of the paper in the experiments.)\nI guess HNET+ENT for CL1 scenario does not use ENT and is just HNET?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1ehZ8mCtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use hypernetwork to prevent catastrophic forgetting. In deep learning, the information of the samples are converted to parameters during the training process, however, future training process could interfere with the information from the previous tasks. One of the method to prevent forgetting is to use reheasal, which retrains the network with previous data. The mechanism of this work is to store the previous samples as a trained point in the parameter space, so that a set of points in the original space is stored and thus rehearsed as one point in the parameter space, this saves both the memory and computation.\n\nI give a weak accept of this paper due to the following reasons:\nPros:\n- The idea of converting a set of data points to one point and rehearse at a meta level is a smart and novel idea.\n- It shows significant improvement compared to baseline methods, especially for split CIFAR experiments.\n- The Appendix contains a fair amount of details and additional experiments on generative models.\n\nCons:\n- This works assumes a task incremental setting, during training process task is received one by one, within each task we could assume i.i.d shuffling of the data. During testing, the task boundary is optional. Although this setting has been taken by many other works in this field, it is also criticised that availability of task boundary is an unrealistic setting. A more realistic setting would be to continually learn with a continuous non-stationary stream of data, which indicates there's no split of train / test phase. Thus a general continual learning method should not require task boundary, which would be problematic for this work as it depends on task conditioning.\n- For the rehearsal objective in 2, L2 penalty is used. This could be a problem as minimizing the L2 distance in the parameter space does not necessarily minimize the task loss.\n\nQuestions I have that needs clarification:\n- Chunking: Are the chunking parameters shared/updated across tasks?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use hypernetwork to prevent catastrophic forgetting. Overall, the paper is well-written, well-motivated, and the idea is novel. Experimentally, the proposed approach achieves SOTA on various (well-chosen) standard CL benchmarks (notably P-MNIST for CL, Split MNIST) and also does reasonably well on Split CIFAR-10/100 benchmark. The authors are suggested to investigate alternative penalties in the rehearsal objective, and also add comparison with methods like HAT and PackNet.
Decision: Accept (Spotlight)