Efficient Continual Learning with Modular Networks and Task-Driven Priors
Authors: Tom Veniat
Ludovic Denoyer
MarcAurelio Ranzato
Venue: ICLR 2021 Poster (2021)
Keywords: Continual learning
Lifelong learning
Benchmark
Modular network
Neural Network

Abstract: Existing literature in Continual Learning (CL) has focused on overcoming catastrophic forgetting, the inability of the learner to recall how to perform tasks observed in the past. 
There are however other desirable properties of a CL system, such as the ability to transfer knowledge from previous tasks and to scale memory and compute sub-linearly with the number of tasks. Since most current benchmarks focus only on forgetting using short streams of tasks, we first propose a new suite of benchmarks to probe CL algorithms across these new axes. 
Finally, we introduce a new modular architecture, whose modules represent atomic skills that can be composed to perform a certain task. Learning a task reduces to figuring out which past modules to re-use, and which new modules to instantiate to solve the current task. Our learning algorithm leverages a task-driven prior over the exponential search space of all possible ways to combine modules, enabling efficient learning on long streams of tasks. 
Our experiments show that this modular architecture and learning algorithm perform competitively on widely used CL benchmarks while yielding superior performance on the more challenging benchmarks we introduce in this work. The Benchmark is publicly available at https://github.com/facebookresearch/CTrLBenchmark.

Published as a conference paper at ICLR 2021
EFFICIENT CONTINUAL LEARNING WITH MODULAR NETWORKS AND TASK-DRIVEN PRIORS
Tom Veniat
Ludovic Denoyer & Mare'A Aurelio Ranzato Facebook Artificial Intelligence Research (denoyer, ranzato) fb. com
LIP6, Sorbonne Université, France tom. wew.itentill 1p.............................. fr
ABSTRACT
Existing literature in Continual Learning (CL) has focused on overcoming catas- trophic forgetting, the inability of the learner recall how perform tasks ob- served in the past. There are however other desirable properties of CL system, such as the ability to transfer knowledge from previous tasks and to scale memory and compute sub-linearky with the number of tasks. Since most current bench- marks focus only on forgetting using short streams of tasks, we first propose a new suite of benchmarks to probe CL. algorithms across these new axes. Finally, we introduce new modular architecture, whose modules represent atomic skills that can be composed to perform certain task. Learning a task reduces to fig- uring out which past modules to re-use, and which new modules to instantiate solve the current task. Our learning algorithm leverages task-driven prior over the exponential search space of all possible ways to combine modules, en- abling efficient learning long streams of tasks. Our experiments show that this modular architecture and learning algorithm perform competitively on widely used CL benchmarks while yielding superior performance on the more challeng- ing benchmarks we introduce in this work. The Benchmark is publicly available at
com hamm//faeboookeeeeeeeee C chnark
INTRODUCTION
Continual Learning (CL) is a learning framework whereby an agent learns through sequence of tasks (Ring) 1994 Thrun 1994 1998), observing each task once and only once. Much of the focus of the CL literature has been avoiding catastrophic forgetting (McClelland et al., [995 McCloskey & Cohen 1989 Goodfellow et al. 2013), the inability of the learner to recall how to perform task learned in the past. In our view, remembering how to perform previous task is particularly important because it promotes knowledge accrual and transfer. CL has then the potential to address one of the major limitations of modern machine learning: its reliance large amounts of labeled data. An agent may learn well a new task even when provided with little labeled data if it can leverage the knowledge accrued while learning previous tasks.
Our first contribution is then to pinpoint general properties that a good CL learner should have, besides avoiding forgetting. In {3 we explain how the learner should be able transfer knowledge from related tasks seen in the past. At the same time, the learner should be able to scale sub-linearly with the number of tasks, both in terms of memory and compute, when these are related.
Our second contribution is to introduce a new benchmark suite, dubbed CTrL, to test the above properties, since current benchmarks only focus on forgetting. For the sake of simplicity and a first step towards more holistic evaluation of CL. models, in this work we restrict our attention to supervised learning tasks and basic transfer learning properties. Our experiments show that while commonly used benchmarks do not discriminate well between different approaches, our newly in- troduced benchmark let us dissect performance across several new dimensions of transfer and scal- ability (see Fig. for instance), helping machine learning developers better understand the strengths and weaknesses of various approaches.
Our last contribution is model that is designed according the above mentioned properties of CL methods. is based on a modular neural network architecture (Eigen et al., 2014 Denoyer
& Gallinari] 2015 Fernando al. 2017 Li et al. 2019) with novel task-driven prior (44 Every

---

Published as a conference paper at ICLR 2021
Average Accuracy
Average Accuracy
MNTDP-D
MNTDP-D
Independent
HAT
EWC
Pasticity
Forgetting
HAT (Wide)
Plasticity
Forgetting
Transfer Output dist
Memory Efficiency
Transfer Output dist
Memory Effidency
Transfer Input dist
Compute Efficiency
Transfor Input dist.
Compute Efficiency
Knowledge Update
Direct
Know edge Update
Transfer
Transfer
Figure Comparison of various CL methods the CTrl. benchmark using Resnet (left) and Alexnet (right) backbones. MNTDP-D is our method. See Tab. of $53 details.
task is solved by the composition of a handful of neural modules which can be either borrowed from past tasks or freshly trained on the new task. In principle, modularization takes care of all the fundamental properties we care about, as i) by design there no forgetting as modules from past tasks are not updated when new tasks arrive, ii) transfer is enabled via sharing modules across tasks, and iii) the overall model scales sublinearly with the number of tasks as long as similar tasks share modules. The key issue is how to efficiently select modules, as the search space grows exponentially in their number. In this work, we overcome this problem by leveraging a data driven prior over the space of possible architectures, which allows only local perturbations around the architecture of the previous task whose features best solve the current task (44.2).
Our experiments in 5 which employ stricter and more realistic evaluation protocol whereby streams are observed only once but data from each task can be played multiple times, show that this model performs least as well state-of-the-art methods on standard benchmarks, and much better on our new and more challenging benchmark, exhibiting better transfer and ability to scale to streams with hundred tasks.
RELATED WORK
CL methods can be categorized into three main families of approaches. Regularization based meth- ods use a single shared predictor across all tasks with the only exception that there can be task- specific classification head depending on the setting. They rely on various regularization methods to prevent forgetting. Kirkpatrick et al. ( [Schwarz et al. (2018) use an approximation of the Fisher Information matrix while Zenke et al. (2017) using the distance of each weight to its initial- ization as a measure of importance. These approaches work well in streams containing a limited number of tasks but will inevitably either forget stop learning as streams grow in size and diver- sity (van de Ven & Tolias 2019). due to their structural rigidity and fixed capacity.
Similarly. rehearsal based methods also share single predictor across all tasks but attack forgetting by using rehearsal on samples from past tasks. For instance, Cestance.LLooez-PPe... Lopez-Paz & Ranzato (anzztto(2017))) (201)::Chaudrrr
et al. (019h);; Rolnick al. (2019) store past samples in replay buffer, while Shin al. (2017)
leamn to generate new samples from the data distribution of previous tasks and /Zhng et al. (201))
computes per-elass prototypes. These methods share the same drawback of regularization methods:
Their capacity is fixed and pre-determined which makes them ineffective at handling long streams.
Finally, approaches based on evolving architectures directly tackle the issue of the limited capacity by enabling the architecture to grow over time. Rusu et al. (2016) introduce new network on each task, with connection to all previous layers, resulting in network that grows super-lineariy with the number of tasks. This issue was later addressed by Schwarz et al. (2018) who propose to distill the new network back to the original one after each task, henceforth yielding fixed capacity predictor which is going to have severe limitations on long streams. Yoon et al. (2018); Hung et al. (2019) propose a heuristic algorithm to automatically add and prune weights Mehta et al. (2020) propose a Bayesian approach adding an adaptive number of weights to each layer. Li et al. (2019) propose to softly select between reusing, adapting, and introducing a new module at every layer. Similarly, Xu & Zhu (2018) propose to add filters once leading to larger and larger networks even at inference time as time goes by. These two works are the most similar to ours, with the major difference that we restrict the search space architectures,
task arrives using REINFORCE (Williams) 1992),
2

---

Published as a conference paper at ICLR 2021
enabling much better scaling to longer streams. While their search space (and RAM consumption) grows over time, ours is constant. Our approach modular, and only a small (and constant) number of modules is employed for any given task both at training and test time. Non-modus
approaches, like those relying on individual neuron gating (Adel et al., 2020; Serrå et al.] 2018 Kessler et al.) 2019), lack such runtime efficiency which limits their app applicability to long streams. Rajasegaran eet al. 2019);; Fernando et al. (2017) propose to learn a modular architecture, where each task is identified by path in graph of modules like we do. However, they lack the prior over the search space. They both use random search which is rather inefficient as the number of modules grows.
There are of course other works that introduce new benchmarks for CL. Most recently, Wortsman et al. (2020) have proposed a stream with 2500 tasks all derived from MNIST permutations. Unfor- tunately, this may provide little insight terms of how well models transfer knowledge across tasks. Other benchmarks like CORe50 (Lomonaco & Maltoni 2017) and CUB-200 (Wah et al. (001) are more realistic but do not enable precise assessment of how well models transfer and scale.
CL is also related to other learning paradigms, such as meta-learning (Finn al. 2017 Nichol et al.
2018 Duan et al. 2016), but these only consider the problem of quickly adapting to a new task while we are also interested in preventing forgetting and learning better over time. For instance, AAet et al. 2018) proposed a modular approach for robotic applications. However, only the performance on the last task was measured. There is also body of literature modular networks for multi-task and multi-domain learning (Ruder et al. 2019; Rebuffi et al., 2017: Zhao et al., (020). The major differences the static nature of the learning problem they consider and the lack of emphasis on scaling to large number of tasks.
EVALUATING CONTINUAL LEARNING MODELS
Let us start with general formalization of the CL framework. We assume that tasks arrive in sequence, and that each task is associated with an integer task descriptor t = 1,2,... which corre- sponds to the order of the tasks. Task descriptors are provided to the learner both during training and test time. Each task is defined by labeled dataset Dr. We denote with S sequence of such tasks. A predictor for a given task is denoted by f X' x Z - "" The predictor has internally some trainable parameters whose values depend on the stream of tasks S seen in the past, therefore the prediction is: ((). Notice that in general, different streams lead to different predictors for the same task: fzz t # fzz,t(S")
Desirable Properties of CL models And Metrics: Since we are focusing on supervised learning tasks, it is natural to evaluate models in terms of accuracy. We denote the prediction accuracy of the predictor f as A (f t)S), y), where x is the input, is the task descriptor of x, S is the stream of tasks seen by the learner and y is the ground truth label.
In this work, we consider four major properties of a CL. algorithm. First, the algorithm has to yield predictors that accurate by the end of the learning experience. This is measured by their average accuracy at the end of the learning experience:
T
A(S) = Σ B( = I,.... v)l 1=1
(I)
Second, the CL algorithm should yield predictors that do not forget, i.e. that are able to perform a task seen in the past without significant loss of accuracy. Forgetting is defined as:
1
F(S) -==--1 Σ ((t))>>=(((((..
1. T). -(f(x, 1....
t). v)l
This measure of forgetting has been called backward transfer (Lopez- Paz & Ranzato, 2017). and it measures the average loss of accuracy on task the end of training compared to when the task was just learned. Negative values indicate the model has been forgetting. Positive values indicate that the model has been improving on past tasks by learning subsequent tasks.
Third, the CL. algorithm should yield predictors that are capable of transferring knowledge from past tasks when solving a new task. Transfer can be measured by:
T(S)= (e [as(; TIS 1,..., v)) (1f(x, TIS' = -))
(3)
3

---

Published as a conference paper at ICLR 2021
which measures the difference of performance between model that has learned through whole sequence of tasks and a model that has learned the last task in isolation. We would expect this quantity to be positive if there exist previous tasks that are related to the last task. Negative values imply the model has suffered some form of interference or even lack of plasticity when the predictor has too little capacity left to learn the new task.
Finally, the CL. algorithm has to yield predictors that scale sublinearly with the number of tasks both in terms of memory and compute. In order to quantify this, we simply report the total memory usage and compute by the end of the learning experience during training. We therefore include in the memory consumption everything learner has to keep around to be able to continually learn (e.g., regularization parameters of EWC or the replay buffer for experience replay).
Streams The metrics introduced above can be applied any stream of tasks. While current bench- marks are constructed to assess forgetting, they fall short at enabling a comprehensive evaluation of transfer and scalability because they do not control for task relatedness and they are composed of too few tasks. Therefore, we propose with and t' a task whose data is sampled from the same distribution as t, but with a much smaller or larger labeled dataset, respectively. Finally, 1 and t" are tasks that are similar to task 4, while we assume no relation between 1 and fj. for all i i j.
suite of streams. If t is a task in the stream, we denote
We consider five axes of transfer and define stream for each of them. While other dimensions certainly exist, here we are focusing on basic properties that any desirable model should possess.
define the stream S (4F l2, la, la, (T) where the last task is the same as the first one but with much less data to learn from. This is useful to assess whether the learner can directly transfer knowledge from the relevant task.
Direct Transfer:
Knowledge Update: we define the stream S* = (fi f2, f3. t4, to. ttt) where the last task has much more data than the first task with intermediate tasks that are unrelated. In this case, there should not be much need to transfer anything from previous tasks, and the system can just use the last task to update its knowledge of the first task.
Transfer to similar Input/Output Distributions: we define two streams where the last task is similar to the first task but the input distribution changes StD = (1,, 12, La, L4, L5, (1), or the output distribution changes S (41, 12, l3, 44, 4s, (7)
Plasticity: this is a stream where all tasks are unrelated, S = (t1, t3, f1, fs, which is useful to measure the "ability to still leam"" and potential interference (erroneous transfer from unrelated tasks) when learning the last task.
All these tasks are evaluated using T in eq. 3 Other dimensions of transfer (e.g., transfer with compositional task descriptors or under noisy conditions) are avenues of future work. Finally, we evaluate scalability using Skmzz stream with 100 tasks of various degrees of relatedness and with varying amounts of training data. See 15.1 for more details.
4 MODULAR NETWORKS WITH TASK DRIVEN PRIOR (MNTDP)
In this section we describe an approach, dubbed MNTDP, designed according to the properties discussed in 3 The class of predictor functions f1x 1|8) we consider in this work is modular, in the sense that predictors are composed of modules that are potentially shared across different (but related) tasks. A module 2015). The only restriction we use in this work is that all modules in layer should differ only in terms of their actual parameter values, while modules across layers can implement different classes of functions. For instance, in the toy illustration of Fig. A (see caption for details), there are two predictors, each composed of three modules (all ResNet blocks), the first one being shared.
be any parametrie function, for instance, ResNet block (He al.)
4.1 TRAINING
Once the new task & arrives, the algorithm follows three steps. First, it temporarily adds new ran- domly initialized modules at every layer (these are denoted by dashed boxes in Fig. B) and it then defines a search space over all possible ways combine modules. Second. it minimizes loss func- tion, which in our case is the cross-entropy loss, over both ways to combine modules and module parameters, Fig. C. Note that only the parameters of the newly added modules are subject to
4

---

Published as a conference paper at ICLR 2021
A
2
MNTDP-S
MNTDP-D
Toy illustration the approach when each predictor is composed of only three modules and only two tasks have already been observed. A): The predictor of the first task uses modules (1.1.1) (listing modules by incrcasing depth in the network) while the predictor of the second task uses modules (1,2,2); the first layer module is shared between the two predictors. B): When a new task arrives, first add one new randomly initialized module at cach layer (the dashed modules). Second, we search for the most similar past task and retain only the corresponding architecture. In this case, the second task most similar and therefore we remove (gray out) the modules used only by the predictor of the first task. C): We train on the current task by learning both the best way to combine modules and their parameters. However, we restrict the search space. In this case, we only consider four possible compositions. all derived by perturbing the predictor of the second task. In the stochastic version (MNTDP-S), for every input path (sequence of modules) selected stochastically, Notice that the same module may contribute to multiple paths (c.g., the top- most layer with id 3). In the deterministic version instead (MNTDP-D), we train in parallel all paths and then select the best. Note that only the parameters of the newly added (dashed) modules are subject to learning. D): Assuming that the best architecture found at. the previous step is (1,2,3), module 3 at the top layer is added the current lihrary of modules.
Figure 2:
training. de facto preventing forgetting of previous tasks by construction but also preventing posi- tive backward transfer. Finally, it takes the resulting predictor for the current task and it adds the parameters of the newly added modules (if any) back to the existing library of module parameters, see Fig.2ID. D.D.
Since predictors are uniquely identified by which modules compose them, they can also be described by the path in the grid of module parameters. We denote the j-th path in the graph by Tz. The parameters of the modules in path πy are denoted by (TT,). Note that in general O(T,)) n (T.) ) + 0. for # j since some modules may be shared across two different paths.
Let II be the set of all possible paths in the graph. This has a size equal to the product of the number of modules at every layer, after adding the new randomly initialized modules. If I' is a distribution over II which is subject to learning (and initialized uniformly), then the loss function is:
r 0* arg min By-T, (( [[s(x, ((, ((x,)), o.r
(4)
where [xx, (xx,) is the predictor using parameters ((T,); L is the loss, and the minimization over the parameters is limited to only the newly introduced modules. The resulting distribution r* a delta distribution, assuming no ties between paths. Once the best path has been found and its parameters have been learned, the corresponding parameters of the new modules in the optimal path are added to the existing set of modules while the other ones are disregarded (Fig. DD). In this work, consider two instances of the learning problem in eq. 4 which differ in a) how they optimize over paths and b) how they share parameters across modules.
Stochastic version: This algorithm alternates between one step of gradient descent over the paths via REINFORCE, (Williams) 1992) as in (Veniat & Denoyer 2018), and one step of gradient descent over the parameters for given path. The distribution I' is modeled by product of multinomial distributions, one for each layer of the model. These seleet one module at each layer, ultimately determining a particular path. Newly added modules may be shared across several paths which yields a model that can support several predictors while retaining a very parsimonious memory footprint thanks to parameter sharing. This version of the model, dubbed MNTDP-S, is outlined in the left part of Fig. C and in Alg. in the Appendix. In order to encourage the model to explore multiple paths, we use entropy regularization on I' during training.
Deterministic version: This algorithm minimizes the objective over paths in eq. 4 via exhaustive search, see Alg. 2 in Appendix and the right part of Fig. C Here, paths do not share any newly added module and we train one independent network per path, and then select the path yielding the lowest loss on the validation set. While this requires much more memory, it may also lead to better
5

---

Published as a conference paper at ICLR 2021
overall performance because each new module is cloned and trained just for a single path. Moreover, training predictors on each path can be trivially and cheaply parallelized on modern GPU devices.
4.2 DATA DRIVVDR PRIOR
Unfortunately, the algorithms as described above do not scale to large number of tasks (and hence- forth modules) because the search space grows exponentially. This is also the case for other evol vong
architecture approaches proposed in the literature (Li al. 2019
Rajasegaran et al. 2019 Yoon
et al. 2018 Xu & Zhu 2018 as discussed in 2
If there were N modules per layer and L layers, the search space would have size N In order to restrict the search space, we only allow paths that branch to the right: A newly added module at layer I can only connect to another newly added module at layer 1 + 1, but it cannot connect to an already trained module at layer I+11 1. The underlying assumption is that for most tasks we expect changes in the output distribution as opposed to the input distribution, and therefore tasks are related, the base trunk is a good candidate for being shared. We will see in {5.3 what happens when this assumption is not satisfied, e.g., when applying this to S
To further restrict the search space employ data-driven prior. The intuition is to limit the search space to perturbations of the path corresponding to the past task (or to the top-k paths) that is the most similar the current task. There are several methods assess which task is the closest to the current task without accessing data from past tasks and also different ways to perturb path. We propose very simple approach, but others could have been used. We take the predictors from all the past tasks and select the path that yields the best nearest neighbor classification accuracy when feeding data from the current task using the features just before the classification head. This process is shown in Fig. BB. The search space is reduced from T to only L, and I' of eq. 4 is allowed to have non-zero support only in this restricted search space, yielding a much lower computational and memory footprint which is constant with respect to the number of tasks. The designer of the model has now direct control (by varying k for instance) over the trade-off between accuracy and computationallmmemoyy budget.
By construction, the model does not forget, because we do not update modules of previous tasks. The model can transfer well because it can re-use modules from related tasks encountered in the past while not being constrained in terms of its capacity. And finally, the model scales sub-linearly in the number of tasks because modules can be shared across similar tasks. We will validate empirically in (5.3 whether the choice of the restricted search space works well in practice.
EXPERIMENTS
In this section we first introduce our benchmark in [5.1 and the modeling details 55.2 and then report results both on standard benchmarks as well as ours in $5.3
5.1 THE CTRL BENCHMARK
The CTrL (Continual Transfer Learning) benchmark is a collection of streams of tasks built over seven popular computer vision datasets, namely: CIFAR10 10 and CIFAR I10AR100 (Krizhevsky) 2009), DTD (Cipoo et al. 2014), SVHN (Netzer et al. 2011), MNIST (LeCun et al. 1998). Rainoow- MNIST (Finn et al. 2019 and Fashion MNIST (Xiao et al. 2017); see Table in Appendix for basic statistics. These datasets are desirable because they are diverse (hence tasks derived from some of these datasets can be considered unrelated), they have a fairly large number of training examples to simulate tasks that do not need to transfer, and they have low spatial resolution enabling fast evaluation. CTrL designed according to the methodology described in { to enable evaluation of various transfer learning properties and the ability of models to scale to a large number of tasks. Each task consists of training, validation, and test datasets corresponding to a 5-way and 10-way classification problem for the transfer streams and the long stream, respectively. The last task of S consists of a shuffling of the output labels of the first task. The last task of S is the same as its first task except that MNIST images have different background color. Shen has 100 tasks, and it is
1CTrl source code available at httpss: / t Pytorch implementation of the experiments available here: ptttts::/giite httpp////ggiittto..oomoo//eeeeee rk

---

Published as a conference paper at ICLR 2021
constructed by first sampling dataset, then 5 classes random, and finally the amount of training data from a distribution that favors small tasks by the end of the learning experience. See Tab. aldd in Appendix for details. Therefore, Sliee tests not only the ability of model to scale to a relatively large number of tasks but also to transfer more efficiently with age.
All images have been rescaled to 32x32 pixels in RGB color format, and per-channel normalized using statistics computed on the training set of each task. During training. we perform data augmen- tation by using random crops (4 pixels padding and 32x32 crops) and random horizontal reflection. Please, refer to Appendix A for further details.
5.2 METHODOLOOGY AND MODELING DETAILS
Models learn over each task in sequence; data from each task can be replayed several times but each stream is observed only once. Since each task has a validation dataset, hyper-parameterr (c.g., learning rate and number of weight updates) are task-specific and they are cross-validated on the validation set of each task. Once the learning experience ends, test the resulting predictor on the test sets of all the tasks. Notice that this is a stricter paradigm than what is usually employed in the literature (Kirkpatrick et al. (016), where hyper- eyper-parameters are set the stream level (by replaying the stream several times). Our model selection criterion is more realistic because it does not assume that the learner has access to future tasks when cross-vait validating on the current task, and this is more consistent with the CL's assumptions of operating on a stream of data.
All models use the same backbone. Unless otherwise specified, this is a small variant of the ResNet- 18 architecture which is divided into 7 modules; please, refer to Table 8 for details of how many layers and parameters each block contains. Each predictor is trained by minimizing the cross-entropy loss with a small L2 weight decay on the parameters. In our experiments, MNTDP adds 7 new randomly initialized modules, one for every block. The search space does not allow connecting old blocks from new blocks, and it considers two scenarios: using old blocks from the past task that is deemed most similar (k- = 1, the default setting) or considering the whole set of old blocks (k- = all) resulting in much larger search space.
We compare to several baselines: Independent Models which instantiates a randomly initialized predictor for every task (as many paths as tasks without any module overlap), 2) Finetuning which trains a single model to solve all the tasks without any regularization and initializes from the model of the previous task (a single path shared across all tasks), 3) Hea- which also shares the trunk of the network across all tasks but not the classification head which is task-specific, 4) New- Leg which shares all layers across tasks except for the very first input layer which task-specific, 5) EWC (Kirkpatrick et al. 2016) which is like "finetuning" but with a regularizer to alleviate forgetting, 6) Experience Replay (Chaudlar et al.) (019b) which is like finetuning except that the model has access to some samples from the past tasks to rehearse and alleviate forgetting (we use 15 samples per class to obtain a memory consumption similar to other baselines), 7) Progressive Neural Networks (PNNs) (Rusu et al.) 2016) which adds both a new module at every layer as well as lateral connections once a new task arrives. 8) HAT (Serrà et al., 2018): which learns an attention mask over the parameters of the backbone network for each task. Since HAT's open- source implementation uses AlexNet (Krizhevsky et al. 2012) as a backbone, we also implemented version of MNTDP using AlexNet for a fair comparison. Moreover, of HAT, the default as provided by the authors and a version, dubbed HAT-wide, that is as wide as our final MNTDP model (or as wide as we can fit into GPU memory). 9) DEN (Yoon et al.) (018) and 10) RCL (Xu & Zhu 2018) which both evolve architectures. For these two models since there no publicly available implementation with CNNs, we only report their performance on Permuted- MNIIST using fully connected nets (see Fig. 3a and Tab. 11 in Appendix E.2
considered two versions
We report performance across different axes as discussed in 3 Average accuracy as in eq. forget- ting as in eq. [ transfer as in eq. 3 and applicable only to the transfer streams, "Mem." [MB] which refers to the average memory consumed by the end of training, and "Flops" [T] which corresponds to the average amount of computation used by the end of training.
5.3 RESULTS
Existing benchmarks: In Fig. 3 we compare MNTDP against several baselines on two standard streams with 10 tasks, Permuted MNIST, and Split CIFAR100 We observe that all models do fairly
7

---

Published as a conference paper at ICLR 2021
<4> 1
Log Nom.
Log Mom.
Independent
Independett
CWCE
EWCs
MATUPO
PUN
2000
MNTUP-O
PAN
RCLE
HAT (WADO1M
HAT (Wire))
NNTIP-O*
(a) Permuted-MNiSS
(b) Split Cifar-100
Figure 3: Results on standard continual learning streams. denotes an Alexnet Backbone. correspond to models cross-validated at the stream-level, setting that favors them over the other methods which are cross- validated at the task-level. Detailed results are presented in Appendix B.2
< <A>
>
<F> >
Mem.
FL.OPs
T(S)
T(s+)
T(S")
T(S"")
T(S
Independent
0.58
14.1
308
0.0
0.0
0.0
Finetune
0.19
-0.3
2.4
284
0.0
-0.1
-0.0
-0.0
-0.1
New-head
0.48
0.0
2.5
307
0.4
-0.3
-0.2
0.3
-0.4
New-leg
0.41
0.0
2.5
366
0.3
-0.3
0.4
-0.1
-0.4
Online i
0.43
-0.1
310
0.3
-0.3
0.3
-0.4
ER
0.44
-0.1
13.1
604
0.0
-0.2
0.0
0.1
-0.2
PNN
0.57
0.0
48.2
1459
0.3
-0.2
0.1
0.2
-0.1
MNTDP-S
0.59
0.0
11.7
363
0.4
-0.1
0.0
0.3
-0.1
MNTDP-D
0.64
1512
0.4
0.3
-0.0
MNTDP-D*
0.62
0.0
140.7
115
0.3
-0.1
0.1
0.3
-0.1
H
0.58
-0.0
26.6
45
0.1
-0,2
0.0
0.1
-0.2
HAT (Wide)*t
0.61
0.0
163.9
274
0.2
-0.1
0.1
0.1
-0.1
Table Aggregated results on the transfer streams over multiple relevant baselines (complete table with more correspond to models using an Alexnet backbone.
baselines provided in Appendix E)
well, with EWC falling bit behind the others in terms of average accuracy. PNNs good average accuracy but requires more memory and compute. Compared to MNTDP, both RCL and HAT have lower average accuracy and require more compute. MNTDP-D yields the best average accuracy, but requires more computation than "independent models"; notice however that its wall-clock training time is actually the same as "independent models" since all candidate paths (seven in our case) can be trained in parallel modern GPU devices. In fact, it turns out that these standard streams MNTDP trivially reduces to "independent models" without any module sharing. since each task is fairly distinct and has relatively large amount of data. It is therefore not possible assess how well models can transfer knowledge across tasks, nor is possible to assess how well models scale. Fortunately, can leverage the CTrL benchmark better study these properties, as described next.
We first evaluate models in terms of their ability transfer by evaluating them the streams S S* S Suu and St introduced 3 Tab. shows that "independent models" again strong baseline, because even the first four streams, all tasks except the last one unrelated and therefore instantiating an independent model is optimal. However, MNTDP yields the best average accuracy overall. MNTDP-D achieves the best transfer streams S St SouI and Stl and indeed discovers the correct path in each of these cases (e.g., it discovers to reuse the path of the first task when learning on S and to just swap the top modules when learning the last task S*). Examples of discovered path are presented in appendix E.4 MNTDP underperforms on St because its prior does match the data distribution, since in this case, it the input distribution that has changed but swapping the first module is out of MNTDP search space. This highlights the importance of the choice of prior for this algorithm. general, MNTDP offers a clear rofe--off between accuracy, i.e. how broad the prior is which determines how many paths can be evaluated, and memory/compute budget. Computationally MNTDP-D is the most demanding, but practice wall clock time comparable to "independent" because GPU devices store in memory all the paths (in our case, seven) and efficiently train them all in parallel. We observe also that MNTDP-S has a clear advantage in terms of compute at the expense of lower overall average accuracy, as sharing modules across paths during training can lead to sub-optimal convergence.
CTrL:
Overall, MNTDP has much higher average accuracy than methods with fixed capacity. It also beats PNNs, which seems to struggle with interference when learning on Stl, as new modules connect all old modules which irrelevant for the last task. Moreover, PNNs uses much more
8

---

Published as a conference paper at ICLR 2021
MNTOP
A > < F
Mem.
PFLOPs
HAT
Independent
0.57
0.0
243
4
Finetune
0.20
-0.4
5
New-head
0.43
0.0
6
On. EWC
0.27
-0.3
4
0.8
MNTDP-S
0.68
0.0
159
5
MNTDP-D
0.75
0.0
102
26
MNTDP-D*
0.75
0.0
1782
3
3.5
1 0.24
-0.1
32
=00
HAT
(WideD.32 0.0
285
1
100
Table 2: Results on the long evaluation stream. correspond models using an Alexnet hackhone. See Tab. 18 for more baselines error bars.
Figure 4: Evolution of A > and Mem. S
memory. "New-leg" and "new-head" models perform very well only when the tasks in the stream match their assumptions, showing the advantage of the adaptive prior of MNTDP. Finally, EWC shows great transfer on , Stm, which probe the ability of the model to retain information. However, it fails St, Spl that require additional capacity allocation to learn new task. Fig. gives holistic view by reporting the normalized performance across all these dimensions.
S
We conclude by reporting results on glone composed of 100 tasks. Tab. reports the results of all the approaches we could train without running into out-of-memory. MNTDP-D yields an absolute 18% improvement over the baseline independent model while using less than half of its memory, thanks to the discovery of many paths with shared modules. Its actual runtime is close to independent model because of GPU parallel computing. To match the capacity of MNTDP, we scale HAT's backbone to the maximal size that can fit in a Titan X GPU Memory (6.5x, wide version). The wider architecture greatly increases inference time later tasks (see also discussion on memory complexity at test time in Appendix D). while modular approach uses the same backbone for every task and yet it achieves better performance. Fig. 4 shows the average accuracy up to the current task over time. MNTDP-D attains the best performance while growing sublinearly in terms of memory usage. Methods that do not evolve their architecture, like EWC, greatly suffer in terms of average accuracy.
Ablation: We first study the importance of the prior. Instead of selecting the nearest neighbor path, we pick one path corresponding to one of the previous tasks at random. In this case, T(S*) decreases from -0.2 and T(S00)) goes from 0 to -0.3. With a random path, MNTDP learns not to share any module, demonstrating that it is indeed important to form a good prior over the search space. Appendix reports additional results demonstrating how on small streams MNTDP is robust to the choice of le in the prior since we attain similar performance using k = 1 and & = all, although only = let us scale to Stong Finally, we explore the robustness to the number modules by splitting each module in two, yielding total of 10 modules per path, and by merging adjacent modules yielding a total of modules for the same overall number of parameters. We find that T(S""") decreases from 0 to 10.1, with 9% decrease on t when the number of modules decreases but stays the same when the number of modules increases, suggesting that the algorithm has to have sufficient number of modules flexibly grow.
CONCLUSIONS
We introduced a new benchmark to enable a more comprehensive evaluation of CL algorithms, not only in terms of average accuracy and forgetting but also knowledge transfer and scaling. We have also proposed modular network that can gracefully scale thanks to adaptive prior over the search space of possible ways to connect modules. Our experiments show that our approach yields a very desirable trade-off between accuracy and compute/memorr usage, suggesting that modularization in a restricted search space is promising avenue of investigation for continual learning and knowledge transfer.

---

Published as a conference paper at ICLR 2021
REFERENCES
Tameem Adel, Han Zhao, and Richard E. Turner. Continual learning with adaptive weights. In International Conference on Learning Representations, 2020.
Ferran Alet, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Modular meta-learning. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pp. 856-868. PMLR, 2018. URLAtttp:/prrcceiiiio.
mlr wttps///vvaaaeeeeeee 1tmmetm88mmm
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019a. URL https: //openreview.no nwt./ for.rro d=Hk 2_sC_FFX_
Arslan Chaudhry, Marcus Rohrbach, Mohamed Ellhoseiy, Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic memories. CoRR, abs/1902. 10486, 2019b. URL http://trxxx...ooo
11///1902000 10486
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
L. Denoyer and P. Gallinari. Deep sequential neural networks. EWRL, 2015.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI$ Fast reinforcement learning via slow reinforcement learning. CoRR, (tb//1611.02799,, 2016. URL org/ abs/ 1611 .07799
D. Eigen, I. Sutskever, and M. Ranzato. Learning factored representations in deep mixture of experts. ICLR, 2014.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, 0bs/1701.08734 2017. URL http://arraiv.orr/aa//////
08734
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, vol- ume 70 of Proceedings of Machine Learning Research, pp. 1135. PMLR, 2017. URL html
http::/proceediin .mlr. www.rrprrss//ee wwt.sp/r////.immmmm.m
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning. 2019.
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio. An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks. arXiv, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/ 2015. URL http::/ttpaa.....//// abtp://b///555 03385
Steven C. Y. Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu- Song Chen. Compacting, picking and growing for unforgetting continual learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bucc Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8- December 2019, Vancouver, BC. Canada, pp. 13647-13657. 2019. URL http: /ttt::/paapezee nips cc/pspex/
951 8 3-compact wwwecciinnppiciiiiiio and anw.riiiiinnnnngnoioooioooiiee for- in cont in lariuaallleerrma learning
Samuel Kessler, Vu Nguyen, Stefan Zohren, and Stephen Roberts. Indian buffet neural networks for
continual learning. CoRR, htts/1912.02299/ 2019.
10

---

Published as a conference paper at ICLR 2021
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations. ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URLhttp: //arxiv.org/aaa /112. 20211..68800 6980
James Kirkpatrick. Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agmieszka Trabsk-BBarwinska Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. CoRR, abs/1612.00796, 2016. URL
1612.00796]
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, technical report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25, pp. 1097-1105. Curran As- sociates, Inc., 2012. URL httpps://pprcoeddgg......... 33986280accch7663333222226666-63
cc httpppper//22///lle... http:///......aoaoooo///////////
pdf
Y. LeCun, L. Bottou, and and P. Haffner Y. Bengio. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (6(11)):227-2333 1998.
Xilai Li. Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In Kamalika Chaudhuri Machine
and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3925-3934- PMLR, 2019. URL/ttpp:/prooceiing mlr. wwwpppeesss... v 1il 9m. m
Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. arv: 700.03550,, 2017.
David Lopez-Paz and Marc Aurelio Ranzato. Gradient episodic memory for continual learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N.
Vishwanathan, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 6467-6476, 2017. URL http::htttp:pr//e/ee ni nips. cc/paper/ 7225
James L McClelland, Bruce L McNaughton, and Randall C O'rrellly
Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of learning and motivation, 1989.
Nikhil Mehta, Kevin J. Liang, and Lawrence Carin. Bayesian nonparametric weight factorization for continual learning. CoRR, abs/2004.10098 2020. URL UtLLitttpss//axx..oogg/bbb//b//p
10098
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learnin algorithms. CoRR. abs/1803.02999 2018 URL (RLlhttp: //atxi org/ www.sa/s......e... 02999
Jathushan Rajasegaran, Munawar Hayat, Salman H. Khan, Fahad Shahbaz Khan, and Ling Shao. Random path selection for incremental learning. CoRR, abs/1906.01120, 2019. URL http:
//arxxtw.org/ab///10000 9006..//966.... 01120
11

---

Published as a conference paper at ICLR 2021
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple vi- sual domains with residual adapters. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Ro- man Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4.9 December 2017, Long Beach, CA. USA, pp. 506-516, 2017. URL http::/ 6654 learning multiple visual domadiinnnn with residual adapters
cct.pperccpppe
Mark B. Ring. Continual Learning in Reinforcement Environments. PhD thesis, University of Texas at Austin, Austin, Texas 78712, 1994.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap. and Gregory Wayne. Experience replay for continual learning- In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Gar- nett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 348 358, 2019. URL http::/paperr: nips cctpppppp/////ccc 6327-experiencee wwweeeeeereeeeeeppppppeeeeeeeeee rewlay- for---oo----- inta--oootnnnaalaaaaalae
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multi-task architecture learning. In The Thirty-Thire AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019,
The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019,
Honolulu, Hawaii, USA, January 27- February 1, 2019, pp. 4822 4829. AAAI Press, 2019. doi: 16.9/ai/ v33101.30014822. URL https: /ttp:s://doiii doi org 10. 1609/ aaaii. v33101.
33:14822
Andrei A. Rusu, Neil Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Ko- ray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, abs/1606.0el 04671, 2016.
abs/ 1606. 04671
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwingg Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10- 15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4535 4544. PMLR, 2018. UtL.
mlr wwwpprressssee...p vv00 http:////vv/cccaaaaaee scwwarz1wa.....mm html
Joan Serrà, Dídac Surís, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. CoRR, abs/1801.0112) 2018. URL/http://arxiv.
org/abs// 1801.111100 01423
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 2017,445 4-9 December 2017, Long Beach, CA, USA pp. 2990 2999, 2017. URL httt:: nips cc/ paper 6892 cont i nual Learn ng- wit t deep generative-
S. Thrun. A lifelong learning perspective for mobile robot control. Proceedings of the IEEE/RSI/I Conference on Intelligent Robots and Systems, 1994.
Sebastian Thrun Lifelong learning algorithms. In Learning to learn. Springer, 1998.
Gido M. van de Ven and Andreas S. Tolias. Three scenarios for continual learning. CoRR. abs/ /e//1904.07734, 2019. URL ltt wttp://wwwwwo/////./..... 07734
Tom Veniat and Ludovic Denoyer. Learning time/memory--fffciien deep architectures with budgeted super networks. Intl. Conference on Compute Vision and Pattern Recognition, 2018.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-ucsd mods-200-2011 dataset. Tech Report: CNS-TR-201 200-TTR--100000 1-001, 2011.
12

---

Published as a conference paper at ICLR 2021
Ronald J. Williams. Simple statistical following algorithms for connecttonistrreifforremmmm learning. Machine Learning, 8:229-255 256, 1992.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. CoRR, abs/2006. 14769, 2020. URL /attps://ariiv org/ 2.s/2006.1476)
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnis a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. CoRR, 1ess1899.,,69, 2018. URL
http://arxiv org abs /10011805101 12369
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynami- cally expandable networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018. Conference Track Proceedings. OpenRe- view.net, 2018. URL htttps::/openreviese net forum?id Sk 7Ks
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intel- ligence. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. ICML 2017, Sydney, NSW, Australia, 6- August 2017, vol- ume 70 of Proceedings of Machine Learning Research, pp. 3987-3995. PMLR, 2017. URL mll.. press / v v70/ /ee/eeeeee 1
httt:://pocced::
Mengmi Zhang, Tao Wang, Joo Hwee Lim, and Jiashi Feng. Prototype reminding for continual learning. CoRR, abs/ 10dp/999559947, 2019. URL Cellttt::: http://wwwww...g/....///// 190.. 09447
Hanbin Zhao, Hao Zeng, Xin Qin, Yongjian Fu, Hui Wang, Bourahla Omar, and Xi Li. What and where: Learn to plug adapters via NAS for multi-domain learning. CoRR, abs/2007. 12415, 2020. org. 12415
URL httpt::///// /tttpps://axiivii
13

---

Published as a conference paper at ICLR 2021
A DATASETS AND STREAMS
The CTrL benchmark is built using the standard datasets listed in Tab. P Some examples from these datasets shown in Fig. 5 The composition of the different tasks given in Tab. 4 and instance of the long stream is presented in Tab. 5 and 6
The tasks in , Sth S and S are all 10-way classification tasks. In S the first task has 4000 training examples while the last one which is the same as the first task, has only 400, The vice versa true for S instead. The last task of S is the same as the first task, except that the background color of the MNIST digit is different. The last task of S is the same the first task, except that label ids have been shuffled, therefore, if "horse" was associated label id the first task, it is now associated to label id 5 in the last task.
The Slong stream is composed of both large and small tasks that have 5000 (or whatever is the maximum available) and 25 training examples, respectively. Each task is built by choosing one of datasets at random, and categories at random this dataset. During task 1-33, the fraction of small tasks is 50%, this increases 75% for tasks 34-66, and to 100% for tasks 57-100. This a challenging setting allowing to assess not only scalability, but also transfer ability and sample efficiency of learner.
Scripts to rebuild the given streams and evaluate models will be released upon publication.
Dataset CIFAR-T00
no. classes
training
validation
testing
100
40000
10000
10000
CIFAR-10
10
40000
10000
10000
D. Textures
47
1880
1880
1880
SVHN
10
47217
26040
26032
MNIST
10
50000
10000
10000
-NIST
10
50000
10000
10000
Rainbow-MNIST
10
50000
10000
10000
Table 3: Datasets used in the CTrL benchmark.
2 277
6
6
6
2 2
125
2 2 7
6
6
6
751
7515
S 5 5
4
6
6
(a) SVHN
(b) MNIST
(c) Fashion-MNIST
(d) Rainbow- -NNIST
Figure 5: Some datasets used the CTrL benchmark.
14

---

Published as a conference paper at ICLR 2021
Stream
71
T2
T3
T4
75
Te
Datasets
Cifar- 10
MNIST
DTD
F-MNIST
SVHN
Cifar-
ST
Train Samples
4000
400
400
400
400
400
# Val Samples
2000
200
200
200
200
200
Datasets
Cifar-10
MNNSTT
DTD
F-MNIST
SVHN
Cifar-10
S+
Train Samples
400
400
400
400
400
4000
Val Samples
200
200
200
200
200
2000
Datasets
R-MNIST
Cifar-T0
DTD
F-MNIST
SVHN
R-MNIST
sin
# Train Samples
4000
400
400
400
400
50
Val Samples
2000
200
200
200
Datasets
Cifar- 10
MNIST
DTD
F-MNIST
SVHI
Gifar-10
gout
# Train Samples
4000
400
400
400
400
400
Val Samples
2000
200
200
200
200
200
Datasets
MNIST
DTD
F-MNIST
SVHN
Cifar-10
Spl
# Train Samples
400
400
400
400
4000
Val Samples
200
200
200
200
2000
Table 4: Details of the streams used to evaluate the transfer properties of the learner. F-MNIST is
Mishion-MNIST and R-MNIST is variant of Rainbow-MNIST, using only different background colors and keepingthe original scale and rotation of the digits
15

---

Published as a conference paper at ICLR 2021
Dataset
Classes
# Train
# Val
# Test
Task id
1
mnist svhn
[6, 3, 7, 0]
25
15
4830
2
[2, 1, 9, 0,
25
15
5000
3
svhn
[2, 0, 6, 1,5]
5000
2500
5000
4
svhn
[1. 5, 7,4]
25
15
5000
5
fashion-mnist
[T-shirt/top. Pullover, Trouser, Sandal, Sneaker] [Shirt, Ankle boot, Sandal, Pullover, T-shirt/...
25
15
5000
6
fashion-mnist
5000
2500
5000
7
svhn
7, 6,9.
15
5000
8
c0f
Ispider, maple), .tree, ttrec, tulip, leopard, lizard]
25
15
500
9
c0
[frog, automobile, airplane, cat, horse] [Ankle boot, Bag, T-shirt/top, Shirt, Pullover] 8, 7, 6,
25
15
5000
10
fashion-mnist
25
15
5000
11
mnist
5000
2500
4914
12
c0
[automobile, truck, dog, horse, deer]
5000
25500
5000
13
10f
[sea, forest, bear, chimpanzee, dinosaur]
25
15
500
14
mnist
[3, 2. 7.
25
15
5000
15
Tashion-mnist
[Bag. Ankle boot. Trouser, Shirt, Dress]
25
15
5000
16
c0far10
[frog, cat, horse, airplane, deer]
25
15
5000
17
c0
[bird, frog, ship, truck, automobile]
5000
2500
5000
18
svhn
[0, 4, 7, 6]
5000
2500
5000
19
mnist
[6, 9, 9,
5000
2500
4806
20
mnist
[8, 6, 9]
5000
2500
4806
cifar100
[sea, pear, house, spider, aquarium', ]ishh] Ikangaroo, ray, tank, crocodile, table]
15
22
cifar100
2250
250
500
23
cifar100
[trout, rose, pear, lizard, baby]
25
15
500
24
svhn
[3, 2, 1]
5000
2500
5000
25
cifar100
[skyscraper, bear, rocket, tank, spider] [telephone, porcupine, flatfish, plate, shrew] [lawn), mower, crocodile, tiger, hed, bear]
25
15
500
c0f
2250
250
27
cifar100
25
15
500
28
svhn
[3, 7. 1, 6]
25
15
5000
29
fashion-mnist
[Ankle boot, Sneaker, T-shirt/top, Coat, Bag]
5000
2500
5000
mnist
[6, 9. 0, 3.
5000
2500
4938
10
Jautomobile, truck, deer, bird, dogl
15
5000
32
c0
[dog, airplane, frog, deer, automobilc]
5000
2500
5000
33
svhn
[1, 5, 3, 61
5000
2500
5000
34
cifar100
[whale, orange, chimpanzee, poppy, sweet) [worm, camel, bus, keyboard, spider] [T-shirt/top. Coat, Ankle boot, Shirt, Dress] [dog, deer, ship, truck, cat]
25
15
500
35
c0f
15
36
fashion-mnist
25
15
5000
37
c0
25
15
5000
38
cifar10
[cat, dog, airplane, ship, deer]
5000
2500
5000
svhn
[7. 6, 9]
15
5000
mnist
[9, 7, 1,
15
5000
41
c0far100
[mushroom, butterfly, bed, hoy, motorcycle] [Shirt, Pullover, Bag, Sandal, T-shirt/opp Jrabbit, bear, aquarium), fish, bee, bowl] [Coat, T-shirt/top. Pullover. Shirt, Sandal] [Pullover, Dress, Coat, Shirt, Sandal]
25
15
500
42
fashion-mnist
25
5000 15
43
100
25
15
500
44
fashion-mnist
15
5000
45
fashion-mnist
25
15
5000
46
mnist
[3, 9. 7, 6.4]
25
15
4940
47
cifar10
[deer, bird, dog, automobile, frog]
25
15
5000
48
svhn
7, 1,
25
15
5000
49
10f
[forest, skunk, poppy, bridge, sweet). [caterpillar, can, motorcycle, rabbit, wardrobe]
2250
250
500
50
cifar100
25
15
500
Table 5: Details of the tasks used in Slong part 1.
16

---

Published as a conference paper at ICLR 2021
Dataset
Classes
# Train
# Val
# Test
Task id
51
100
[trout, mountain, kangaroo, pine), tree, hee]
25
15
500
c0far100
[clock, fox, castle, bus, willow), tree]
25
500
cifar10
[cat, airplane, dog, ship, truck]
25
15
5000
54
mnist
[9. 7, 8, 5]
25
15
4866
55
fashion-mnin
[Bag. T-shirt/top, Sandal, Shirt, Dress]
25
15
5000
fashion-mma
[Sneaker, Ankle boot, Coat, Sandal, Trouser]
5000
2500
5000
mnist
[1, 4, [truck, automobile, frog, ship, dog]
25
4982
cifar10
25
15
5000
59
mnist
[7, 2, 8, 4]
25
15
4848
60
mnist
8, 9. 1,
5000
2500
4974
19.5, 8, 6]
25
5000
mnist
[1, 7.
25
15
4848
63
cifar10
[truck, dog, bird, automobile, airplane]
25
15
5000
64
mnist
3,
25
15
4914
65
svhn
2,,5,
25
5000
cifar100
[otter, camel, hee, road, poppyl
500
67
svhn
14.2, 1, 7]
25
15
5000
68
mnist
3. 8.
25
15
4932
69
fashion-mnist
[Pullover, Sneaker, Trouser, Dress, Sandal]
25
15
5000
svhn
[5, 2, 31
25
5000
svhn
19. 6, 2, 4,
25
5000
72
mnist
1, 0,
25
15
4938
73
cifar10
[dog. automobile, ship, airplane, cat]
25
15
5000
74
mnist
[0, 7, 2, 4]
25
15
4920
cifar10
[bird, deer, airplane, dog, ship]
5000
76
cifar100
[mountain, bicycle, caterpillar, spider, possum]
25
15
500
77
svhn
[8. 4, 4.0.6 6]
25
15
5000
78
svhn
5. 9. 0. [can, dolphin, house, pickup'. crab]
25
15
5000
eifar100
25
15
500
c0
Isquirrel, possum, crocndile, mountain, hamster]
500
81
mnist
[7, 0, 1, 6,
25
15
4938
82
fashion- mnist
[T-shirt/top. Dress, Trouser, Shirt, Sneaker]
25
15
5000
83
cifar10
[eat, frog. automobile, dog, airplane] Jautomobile, cat, dog, ship, horse] [cup, otter, orchid, kangaroo, rose]
25
15
5000
cifar10
25
S000
85
cifar100
25
15
500
86
mnist
[1. 5, 2, 9]
25
15
4892
87
svhn
5. 3. 2. 7]
25
15
5000
c0
[dog, deer, cat, frog. bird]
25
5000
mnist
[6. 2, 9,4]
4832
90
cifar100
[pear, racket, sea, road, orange]
25
15
500
91
svhn
[0, 8, 4. 1]
25
15
5000
92
ci
[truck, horse, ship, deer, dog]
25
15
5000
mnist
[5. 4.
25
4806
94
svhn
[2. 6, 3. 1]
25
15
5000
95
Bashion-mmist
[Bag, Trouser, Sneaker, Ankle boot, Sandal]
25
15
5000
96
svhn
[7, 9, 1, 81
25
15
5000
100
llamp, otter, skyscraper, sea, raccoon] [clock, flatfish, snake, can, man]
25
500
c0f
500
99
svhn
[6. 3, 0. 8, 7]
25
15
5000
100
fashion-mnist
[Shirt, Coat, Dress, Sandal, Pullover]
25
15
5000
Table 6: Details task Sioe, part
17

---

Published as a conference paper at ICLR 2021
B LEARNING ALGORITHM
We provide a more complete description of the two learning algorithms for MNTDP-S and MNTDP- D. In the deterministic case, all the architectures (given how we restrict the search space, we have 7 of them) are trained over the training set, and the best path is retained based on its score on the validation set. To avoid overfitting when using MNDTDP-S, we split the training set into two halves Df and D The first part is used to update the module parameters 9 while the second is used to update the parameters in the distribution over paths, r. If both sets of parameters were trained on the same dataset, I would favor paths prone to overfitting since they will results in large decrease of its training loss and therefore a larger reward. When they are trained on different sets, has to select paths with reasonable amount of free parameters, allowing 0 to learn and generalize enough to decrease the loss on both training sets. Then, the most promising architecture is selected based on arg max I, and fine-tuned over the whole Dt. In this case, the validation set is used only for hyper-paan prram-parameters selection.
Algorithm 1: MNTDP-S algorithm.
Duta: Dataset of lask DD:;
Algorithm 2: MNTDP-D algorithm.
Past predictors: (S) for j f.....t-1. 1;
Find closest task:
Data:
Dataset of task -D',;
j' = arg maxy (i [A(NN( (x, j|S)), w)l.
Past predictors: [(. for -t...,t-1
where NN is the S-neurest neighbor classifier in l'eature
Find closest task:
space;
j arg max, F(z,))= (A(NN( (())) v/]-
Define search space: Take path correspoeding to predictor of task j" and add a new randomly initialized modulc at
where NN S-nearest neighhor classifier feature space; Define search space: Take path corresponding to predictor of task, j" and add a new randomly initialized module at
every layer. It distribution over paths; ze. i-th path:
Split train set in two halves: Dj and Di;
every layer. my; i-th puth, 1,..., N where N the total number of paths; for
while loss in eq. has not converged do
get sample (a., v) D (iteration mod 2]: sample path me.. P:
.N do
while loss in eq Ela not converged do
if odd iteration or maxe > 0.99 then forward/backward and update ((Tr.)) (only newly added modules)
get (at, 3) D=:
forward/hackwand update purameters ((m;)) (only newly added modules)
end compute accurncy A, on validation set.
11
forward/hackward and update I'; end
end
end
Let 1* arg max; As, then set (ar
Let 2" be the path with largest values I. then faz, r|S U) my
C ARCHITECTURE OF THE MODELS AND
The model used on Permuted-MNIST is presented in table On all other tasks, the base architecture for all baselines and MNTDP is the same CNN, ResNet convolutional neural network reported in Table [ or AlexNet as reported in Table 9 modules for MNTDP and PNN.
The table also shows how layers are grouped into
Block
# layers
#params
hidden units
785000
1000
2
1001000
1000
10010
num. classes
Table 7: Permuted MNIST Model
Block
# layers
#params
# out channels
1 1856
64
147968
64
4
152192
64
4
4
152192
64
5
2
78208
64
6
2
73984
64
650
num. classes
Table 8: Resnet architecture used throughout our experiments.
18

---

Published as a conference paper at ICLR 2021
Block
layers
#params
# out channels/hidden units
3136
64
73856
128
131328
256
2099200
2048
4196352
2048
10245
num. classes
Table 9: Details of the Alexnet architecture used Serrà et (2018) and in our experiments.
As presented in section 5.2 the stream be visited only once, preventing stream-level hyper- parameters tuning. Exceptions are made for HAT (Serrà et al.] 2018) because we have been using authors' implementation, and for EWC and Online-EWC since these approaches fail in the proposed setting. The constraint strength hyper-parameter À must be tuned at the stream level since a task- level tuning of À results in little or no constraint all, leading to severe catastrophic forgetting. The stream-level hyper- peeer-errsssesss optimization considers 9 values for À {1. 5, 10, 50, 100, 500, 108, x 10 10"). Note that this gives an unfair advantage to EWC. Online-EWC and HAT, as other methods including MNTDP use task-level cross-validation as described in 5.2
For all methods and experiments, we use the Adam optimizer (Kiggma & Ba 2015) with 31 0.9,
82 = 0.999 and = 10
For each task and each baseline, two learning rates {10-2, 10-3) and weight decay strengths (0, 10-5, 10-1) are considered. Early stopping is performed each task to identify the best step to stop training. When the current task validation accuracy stops increasing for 300 iterations, we restore the learner to its state after the best iteration and stop training the current task.
For MNTDP-S, we consider two additional learning rates for the I' optimization {10-2, 10 10-3). An entropy regularization term I is added to the loss encourage exploration, preventing early convergence towards a sub-optimal path. The weight for this regularization term set to throughout experiments
Finally, since small tasks in Stone have very few examples in the validation sets, we use test-time augmentation to prevent overfitting during the grid search. For each validation sample, we add four augmented copies following the same data augmentation procedure used during training.
D MEMORY COMPLEXITY OF THE MODELS
In the main paper we report the overall memory consumption by the end of training. However, modular architectures like MNTDP use only a sparse subset of modules for given task. In this section, we report the memory complexity both at training and test time. Table 10 shows that all methods but DEN and RCL are as fast evaluate at time as independent network because they all use single path.
E ADDITIONAL RESULTS
E.
METRIC
LCA In addition the metrics used in the main paper, we report the area under the learning curve after 3 optimization steps (Chaudhry et al. 2019a) as metric to assess learning speed:
LCA@3
Σ
(f)
3
Where As,( corresponds to the test set accuracy on task t of learner f after having seen batches from t
19

---

Published as a conference paper at ICLR 2021
Train
Test
Inde, fintune, new-head
N
N
EWC
+
2TN
N
ER
N
TT
MNTDP-D
kbN
N
MNTDP-S
N + 22N
N
HAT
N
N
Wide HAT
SN
SN
DEN
N + PT
N
pT
RCL
N + +T
N +TT
Lean to Grow
N + 2Tp
N
Table 10: Memory complexity of the different baselines at train time and at test time, where N is the size of the backbone, the number of tasks, r the size of the memory buffer per task, k the number of source columns used by MNTDP, the number of blocks in the backbone, S the scale factor used for Wide-HAT and p the average number new parameters introduced per task. Note that while Wide HAT and MNTDP-D D are using similar amount of memory on CTrL (Table [ the inference model used by MNTDP on each task only uses the memory of the narrow backbone, resulting in more than 6 times smaller inference models.
E.2 CLASSICAL STREAMS
Table 11 and report performance across all axes evaluation on the standard Permuted MNIST and Split CIFAR 100 streams.
AA>
F
Mem.
FLOPs
LCA@5
Model
Independent
0.98
0.00
71.8
9.0
0.43
Finetune
0.49
-0.49
12.0
0.13
New-head freeze
0.89
0.00
7.5
19.0
0.14
New-head finetune
0.55
-0.43
7.5
15.0
0.14
New-leg freeze
0.98
0.00
35.4
10.0
0.43
New-leg finetune
0.89
-0.09
35.4
14.0
0.19
EWC
i
0.94
-0.03
143.7
18.0
0.13
Online EWC i
0.95
-0.01
21.6
14.0
0.16
ER (Reservoir)
0.69
-0.29
15.0
12.0
0.29
ER
0.90
-0.08
15.0
15.0
0.29
PNN
0.98
0.00
253.8
123.0
0.15
MNTDP-S
0.97
0.00
71.8
21.0
0.22
MNTDP-S (k=all)
0.98
0.00
71.8
24.0
0.19
MNTDP-D
0.98
0.00
71.8
56.0
0.37
HAT:
0.95
0.00
25.0
0.11
HAT (Wide)t
0.97
0.00
78.5
209.0
0.13
DEN
0.95
0.00
8.1
RCL
0.96
0.00
8.5
148.2
Table 11: Results on the standard permuted-MNIS stream. In this stream, each of the 10 tasks corresponds to random permutation of the input pixels of MNIST digits. For DEN and RCL, since we are using the authors' implementations, do not have access to the LCA measure. t correspond to models using stream-level crossvvalidation (see Section 5.2).
20

---

Published as a conference paper at ICLR 2021
A>>
Mem.
FLOPs
LCA@5
Model
Independent
0.82
0.00
24.4
1327.0
0.11
Finetune
0.18
-0.56
2.4
886.0
0.15
New-head freeze
0.57
0.00
2.5
740.0
0.21
New-head finetune
0.21
-0.52
2.5
687.0
0.15
New-leg freeze
0.50
0.00
2.5
1759.0
0.12
New-leg finetune
0.18
-0.42
2.5
1115.0
0.11
EWC
0.55
0.01
51.0
1151.0
0.11
Online EWC
0.54
-0.02
7.3
1053.0
0.12
ER (Reservoir)
0.32
-0.50
20.9
1742.0
0.17
ER
0.66
-0.15
20.9
1524.0
0.17
PNN
0.78
0.00
133.8
9889.0
0.15
MNTDP-S
0.75
0.00
24.4
1295.0
0.11
MNTDP-S (k=all)
0.75
0.00
24.4
1323.0
0.11
MNTDP-D
0.83
0.00
24.3
6168.0
0.12
MNTDP-D*
0.81
0.00
260.9
488.0
0.17
HAT
0.74
-0.01
27.0
175.0
0.11
HAT (Wide)*;
0.79
0.00
269.3
1830.0
0.11
Table 12: Results the standard Split Cifar 100 stream. Each task is composed of 10 classes. * cor- respond models using an Alexnet backbone, f models using stream-level cross-validation (see Section 5.2).
21

---

Published as a conference paper at ICLR 2021
E.3
THE CTRL BENCHMARK
Figure D
radar plots of the models evaluated on the CTrL benchmark. The companion
tables of these plots in Tab. 13
14
15
16 and 17
lng Adent
Finetune
New-head freeze
Plasticity
-orgetting
Transfer Output dist.
Mem. Efficiency
Transfer Input dist.
Compute Efficiency
Knowledge Direct Transfer
Update
New-head finetune
New-leg freeze
New-leg finetune
EWCK
Online EWC
ER (Reservoir)
ER
PNN
MNTDP-S
MNTDP-S (k=all)
MNTDP-D
MNTDP-D (Alexnet)
(Alexnet)*
HAT (Wide Alexnet)
Figure 6: Comparison of the global performance of all baselines the CTrL Benchmark. MNTDP-D is the most efficient method on multiple the dimensions, but requires more computation than MNTDP-S.
22

---

Published as a conference paper at ICLR 2021
E.4 DISCOVERED PATHS
(0. 'OUT)
(1, 'OTT)
(2. (UTUT)
(3. 'OTT))
(4, 'OUTT)
(S. ''UUT)
(0, 7)
(1,7)
(2, 7)
(3,7)
(4, 7)
(5, 7)
(0.
(1.6)
(2,6)
(3.6)
(4.
(0, 5)
(1,
(2,
(3.
(4.
(0,4) 4)
(1,4)
(2, 4)
(3, 4)
(4, 4)
(0, 3)
(1.3)
(2.3)
(3,
(4.
(0, 2)
(1,2)
(3, 2)
(4, 2)
I)
(0, 0)
(0. INs')
(I. 'INs')
(2. 'INs'
(3, 'N's')
(4. INs')
(S, 'NN'')
Figure 7: Global graph of paths discovered by MNDTP-D the T(S Stream. "INs" (resp. "OUT") nodes are input (resp. output) of the path for each task. Solid edges correspond to parameterized modules while dashed edges are only used to show which block is selected for each task and don't apply any transformation. We observe that significant amount of new parameters are introduced for tasks 2, 3, and 5, which are very different from the first task. The model is however able correctly identify that the last task is very similar the first one, resulting very large reuse of past modules and only introducing adapt the new task.
classification layer
23

---

Published as a conference paper at ICLR 2021
(0, 'OUT))
(1. 'OTU
(2, 'UTU))
(3. (OTUU))
(4, 'OUUUT)
(5. 'OUUT)
(0, 7)
(1,
(2,
(3,7)
(4.
(5, 7)
(0, 6)
(1,6)
(2,6) 6)
(3
(4, 6)
(5, 6)
(0. 5)
().5)
(2,
(3.5)
(4.
4)
(1,4)
(2,
(3,4)
(4, 4)
(0, 3)
(1.3)
(2. 3)
(3.3)
(, 2)
(1,2)
(0. 1)
(1,
(0.0)
(1.0)
(0. 'INs))
(1, TN')
(2, 'INs)
(3, 'N''s
(4, 'T's'
(S, 'INs)
Figure 8: Global graph paths discovered by MNDTP-S on T(S"" Stream. When facing last task, the model correctly identified that modules from the first task should be reused, ultimately introducing 2 new modules to solve it.
24

---

Published as a conference paper at ICLR 2021
E.5
STREAM S"
Acc Ti
Acc Ty
An,T, T(S < <A>
<F> Mem.
FLOPs L
Model
Independent
0.72
0.36
-0.35
0.00
0.56
0.00
14.6
292.0
0.10
Finetune
0.72
0.37
-0.34
0.01
0.18
-0.33
2.4
308.0
0.10
New-head freeze
0.72
0.71
-0.01
0.35
0.54
0.00
2.5
262.0
0.19
New-head finetune
0.72
0.35
-0.36
-0.01
0.15
-0.39
2.5
296.0
0.10
New-leg freeze
0.72
0.65
-0.07
0.29
0.47
0.00
2.5
362.0
0.11
New-leg finetune
0.72
0.33
-0.38
-0.03
0.13
-0.41
2.5
333.0
0.10
EWC
0.72
0.71
-0.01
0.35
0.52
-0.02
31.5
344.0
0.13
Online EWC
0.72
0.69
-0.03
0.33
0.54
-0.01
7.3
309.0
0.11
ER (Reservoir))
0.72
0.30
-0,41
-0.06
0.20
-0.32
13.5
646.0
0.12
ER
0.72
0.36
-0.36
0.00
0.41
-0.13
13.5
551.0
0.11
PNN
0.72
0.65
-0.06
0.29
0.62
0.00
51.1
1099.0
0.13
MNTDP-S
0.72
0.71
0.00
0.35
0.63
0.00
11.0
310.0
0.10
MNTDP-S (k=all)
0.72
0.63
-0.09
0.27
0.61
0.00
10.7
341.0
0.10
MNTDP-D
0.72
0.72
0.00
0.36
0.67
0.00
9.2
1876.0
0.16
MNTDP-D*
0.64
0.64
0.00
0.28
0.63
0.00
130.2
101.0
0.21
H
0.61
0.42
-0.19
0.06
0.57
-0.01
26.6
54.0
0.12
# (Wide)
0.67
0.54
-0.13
0.18
0.60
0.00
164.0
257.0
0.14
Table 13:
Results in the T(S* evaluation stream. In this stream, the last task is the as the
first with an order of magnitude less data. correspond to models using an Alexnet backbone, i to models using stream-level cross-validation.
Independent
A>> 1
<F>
Log Mem. J
FLOPs
Finetune
100
New head freeze
New-ley freece
0.50
finetune
EWC
-02
1d
1000
ER (Reservoir)
Online EWC
PNN
MNTDP-S
MNTDP-S (k-all)
0.0
MNTOP.D
Figure 9: Comparison of all basclines on the stream.
25

---

Published as a conference paper at ICLR 2021
E.6
STREAM S+
Acc Ti
Acc Ty
An,T, T(s+)
< <A>
<F> Mem.
FLOPs LCA@5
Model
Independent
0.37
0.71
0.35
0.00
0.57
0.00
14.6
404.0
0.10
Finetune
0.37
0.58
0.21
-0.13
0.24
-0.23
2.4
262.0
0.12
New-head freeze
0.37
0,43
0.06
-0.28
0.41
0.00
2.5
361.0
0.17
New-head finetune
0.37
0.57
0.20
-0.14
0.19
-0.36
2.5
327.0
0.11
New-leg freeze
0.37
0.37
0.01
-0.34
0.34
0.00
2.5
425.0
0.11
New-leg finetune
0.37
0.56
0.19
-0.15
0.17
-0.29
2.5
357.0
0.10
EWC
0.37
0.42
0.05
-0.29
0.39
-0.02
31.5
337.0
0.11
Online EWC
0.37
0.40
0.03
-0.31
0.37
-0.04
7.3
271.0
0.11
ER (Reservoir)
0.37
0.56
0.20
-0.15
0.20
-0.33
13.5
493.0
0.11
ER
0.37
0.47
0.10
-0.24
0.43
-0.07
13.5
512.0
0.12
PNN
0.37
0.54
0.17
-0.17
0.52
0.00
51.1
1757.0
0.12
MNTDP-S
0.37
0.62
0.25
-0.09
0.56
0.00
14.0
417.0
0.10
MNTDP-S (k=all)
0.37
0.66
0.29
-0.05
0.56
0.00
14.0
595.0
0.10
MNTDP-D
0.37
0.72
0.35
0.01
0.61
0.00
14.0
1659.0
0.11
MNTDP-D#
0.40
0.64
0.24
-0.07
0.61
0.00
156.3
136.0
0.17
H
0,41
0.52
0.12
-0.19
0.57
0.00
26.6
39.0
0.13
# (Wide)
0.41
0.61
0.19
-0.10
0.59
0.00
164.0
324.0
0.14
Table 14: Results the St evaluation stream, this stream, the 5th task is the same as the first with an order of magnitude more data. Tasks 2, 3, and 4 are distractors. correspond to models using an Alexnet backbone, to models using stream-level cross-validation.
Independent
< A>>
<P>
Log Mem.
FLOPs
Finettue
New-head freeze
New-head
New-leg freeze
New-log finetune
EWC
04
02
1000
10:
ER (Reservoir)
PNN
02
MNTDP-S
MNTDP-S (k=all)
0.0
MNTDP-D
Figure 10: Comparison of all baselines on the S stream
26

---

Published as a conference paper at ICLR 2021
E.7
STREAM S
Ace T1
Acc TV
ATIT T(sin)
< A >>
F>
Mem.
FLOPs
LCA@5
Model
Independent
0.98
0.60
-0.39
0.00
0.57
0.00
14.6
265.0
0.10
Finetune
0.98
0.57
-0,41
-0.03
0.18
-0.31
2.4
244.0
0.11
New-head freeze
0.98
0.39
-0.59
-0.21
0.45
0.00
2.5
246.0
0.13
New-head finetune
0.98
0.62
-0.36
0.02
0.19
-0.33
2.5
188.0
0.11
New-leg freeze
0.98
0.95
-0.04
0.35
0.48
0.00
2.5
282.0
0.10
New-leg finetune
0.98
0.70
-0,28
0.10
0.21
-0.34
2.5
238.0
0.10
EWC
0.98
0.87
-0.12
0.27
0.43
-0.14
31.5
269.5
0.11
Online EWC
0.98
0.87
-0.12
0.27
0.43
-0.12
7.3
287.0
0.10
ER (Reservoir)
0.98
0.60
-0.38
0.00
0.30
-0.26
13.5
570.0
0.12
ER
0.98
0.60
-0.38
0.00
0.38
-0.17
13.5
604.0
0.12
PNN
0.98
0.70
-0.29
0.10
0.57
0.00
51.1
899.0
0.10
MNTDP-S
0.98
0.64
-0.34
0.04
0.57
0.00
12.2
333.0
0.10
MNTDP-S (k=all)
0.98
0.68
-0.30
0.08
0.57
0.00
13.4
327.0
0.10
MNTDP-D
0.98
0.62
-0.36
0.02
0.60
0.00
11.6
1225.0
0.11
MNTDP-D*
0.98
0.67
-0.31
0.07
0.59
0.00
156.6
115.0
0.12
HAT*
0.98
0.61
-0.37
0.01
0.58
-0.01
26.6
41.0
0.12
H (Wide) 0.97
0.67
-0.30
0.07
0.62
0.00
164.0
186.0
0.11
Table 15: Results in the transfer evaluation stream with input perturbation. In this stream, the last task the same as the first one with a modification applied to the input space and with an order of magnitude less data. Tasks 2, 3, 4 and 5 are distractors. correspond to models using Alexnet backbone, i to models using stream-level cross-validation.
Independent
<>>>>
<P>
Log Mem.
FLOPs
Finelune
New-head finetune
1000
freeze
New-leg tinetune
EWC
0.4
10
Online EWC
10t
500
ER (Reservoin
PNN
0.2
MNTDP-S
MNTOP-S (keall)
0.0
Figure 11: Comparison of all baselines on the S stream
27

---

Published as a conference paper at ICLR 2021
E.8 STREAM S
Acc Ti
Acc Ti
AT, T(sout) A
F>>
Mem.
FLOPs
LCA@5
Model
Independent
0.70
0.37
-0.32
0.00
0.61
0.00
14.6
349.0
0.10
Finetune
0.70
0.36
-0.33
-0.01
0.15
-0.37
2.4
331.0
0.11
New-head freeze
0.70
0.70
0.01
0.33
0.54
0.00
2.5
374.0
0.19
New-head finetune
0.70
0.31
-0.39
-0.06
0.14
-0.41
2.5
379.0
0.10
New-leg freeze
0.70
0.25
-0.45
-0.12
0.40
0.00
2.5
369.0
0.10
New-leg finetune
0.70
0.33
-0.36
-0.04
0.14
-0.40
2.5
340.0
0.10
EWC
0.70
0.68
-0.01
0.31
0.52
-0.03
31.5
387.0
0.11
Online EWC
0.70
0.66
-0.04
0.29
0.51
-0.03
7.3
399.0
0.10
ER (Reservoir)
0.70
0.39
-0.31
0.02
0.22
-0.35
13.5
732.0
0.11
ER
0.70
0.50
-0.19
0.13
0.54
-0.07
13.5
758.0
0.11
PNN
0.70
0.62
-0.07
0.25
0.62
0.00
51.1
1799.0
0.10
MNTDP-S
0.70
0.64
-0.05
0.27
0.64
0.00
10.1
406.0
0.11
MNTDP-S (k=all)
0.70
0.63
-0.06
0.26
0.63
0.00
11.6
411.0
0.10
MNTDP-D
0.70
0.70
0.01
0.33
0.68
0.00
11.6
1299.0
0.15
MNTDP-D#
0,64
0.64
0.00
0.27
0.65
0.00
130.2
99.0
0.22
H
0.62
0.44
-0.18
0.07
0.60
0.00
26.6
42.0
0.13
H
0.68
0.51
-0.17
0.14
0.64
0.00
164.0
293.0
0.12
Table Results in the transfer evaluation stream with output perturbation. In this stream, the last task uses the same classes as the first task but in different order and with an order of magnitude less data. Tasks 2, 3, 4 and 5 are distractors. * correspond to models using an Alexnet backbone, to models using stream-level cross-validation.
Independent
< <A> 9
<T>
Log Mem.
FLOPS
Finetune
New-head freeze
-04
New-head finetune
New-leg freeze
New-lng finetune
0.50
0.2
10
1000
Online EWC
(Reservoir)
ER
MNTDP-S
MNTDP-S (k-all)
0.0
MNTDP-D
Figure 12: Comparison of all baselines on the Sul stream
28

---

Published as a conference paper at ICLR 2021
E.9 S
Acc T3
ATg.Ts
<A>
< <F>
Mem.
FLOPs
LCA@5
Model
Independent
0.71
0.00
0.59
0.00
12.2
232.0
0.10
Finetune
0.57
-0.14
0.21
-0.30
2.4
274.0
0.12
New-head freeze
0.29
-0,42
0,45
0.00
2.4
294.0
0.13
New-head finetune
0.56
-0.15
0.20
-0.35
2.4
336.0
0.11
New-leg freeze
0.27
-0.44
0.37
0.00
2.5
390.0
0.11
New-leg finetune
0.58
-0.13
0.19
-0.34
2.5
375.0
0.11
EWC
0.28
-0.43
0.27
-0.19
26.7
239.0
0.11
Online EWC
0.28
-0.43
0.30
-0.17
282.0
0.12
ER (Reservoir)
0,48
-0.23
0.20
-0.27
11.7
383.0
0.10
ER
0.51
-0.20
0.45
-0.08
11.7
597.0
0.10
PNN
0.56
-0.15
0.54
0.00
36.5
1742.0
0.13
MNTDP-S
0.58
-0.13
0.55
0.00
11.0
351.0
0.10
MNTDP-S (k=all)
0.60
-0.11
0.56
0.00
11.0
340.0
0.11
MNTDP-D
0.70
-0.01
0.62
0.00
11.6
1503.0
0.10
MNTDP-D*
0.65
-0.06
0.64
0.00
130.2
124.0
0.17
0.50
-0.21
0.58
0.00
26.5
50.0
0.11
# (Wide)
0.61
-0.10
0.61
0.00
163.7
312.0
0.12
Table 17: Results in the plasticity evaluation stream. In this stream, we compare the performance on probe task when the first problem encountered by the model and when already seen distractor tasks.* correspond to models using an Alexnet backbone, to models using stream-level cross-validation (see Section 5.2)
Independent
< A>>
<T>
Log Mem. J
FLOPs
Finetune
New-head freeze
finetune
New-leg freece
New-log finetune
102
000
EWC
10
ER (Reservoir)
PNN
0.2
MNTDP-S
MNTDP-S (k-all))
0.0
MNTDP-D
Figure 13: Comparison of all baselines on the Stl stream
29

---

Published as a conference paper at ICLR 2021
E.10
S
< A
<F>
Mem.
FLOPs
LCA@5
Model
Independent
0.57 0.01
0.0 0.0
243.7 + 0.0
3542.33 139.16
0.2 + 0.0
Finetune
0.0
0.0
2.0
4961.33 112.16 112.16
0.23 = 0.0
New-head freeze
0.43
0.0
0.0
5574.33 = 249.65
0.27 0.0
Online EWC 1
0.20
0005
7.0
3882.67 159.15
0.21 0.0
MNTDP-S
0.0
0.0
1
5437.67 = -100777
0.21 0.0
MNTDP-D
0.0
0.0
102.03 0.8
26066.67 = 662.74
0.34 = 0.0
MNTDP-D*
0.75 0.0
0.0
1803.47 + 16.45
2598.67 70.48
0.46 + 0.0
HAT*1
0.24 0.01
+
31.9 + 0.0
147.0 + 27.39
0.21 0.0
HAT (Wide) *f
0.32 - 0.0
0.0 0.0
285.0 = 0.0
1056.33 = 137.29
0.21 = 0.0
Table 18: Results on the long evaluation stream. We report the mean and standard error using different instances of the stream, all generated following the procedure described in A correspond to models using an Alexnet backhone, to models using stream-level cross-validation (see Section
5.2)
<A> f
<>> 1
Log Mem. 4
FLOPs
0.75
100
102
0.50
002
Independent
Finetune
10
50
New-head freeze
0.25
Online EWC
MNTDP-S
0.0
MNTDP-D
Figure 14: Comparison of all baselines on the Slong stream
30


Reviews:
[{"review_id": "xC8rUX_66Jl", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes modular networks with task driven prior (MNTNP) for continual learning. For each new task, the model will try to add new modules to each layer, and then search for a best path that leads to the best performance for the current task. During this process only the newly added modules are tuned, therefore, the method does not have catastrophic forgetting by construction. The model size is likely to grow through this process. However, due to similarity among tasks, it is likely that some of the modules will be shared between different tasks. Hence, the growth of the size of the model is likely to be sub-linear with respect to the number of tasks.\n\n+ Pros\n+ The idea of exploring different modules and structures for each task is interesting\n\n+ The paper is in general well written and easy to follow\n\n- Cons\n- The novelty of the proposed method is limited. The only difference it has as compared to [1] seems to be the data prior that it used to get the previous path to start the search.  There is no comparison with [1] or the other similar methods such as [2], and [3].  It is hard to see if the proposed change resulted in a significant benefit for the learning. In addition, the comparison with other methods are not complete. Because some of the method does not add additional parameter for each of the task, it is hard to see if the gain from the proposed method or is more of a matter of adding additional parameters to the model.\n\n- The setting to freeze old path does prevent forgetting by construction, however, it also prevents positive backward transfer as a whole. This takes away half of the benefit from continual learning, and the only thing left is forward transfer. This scheme also limits forward transfer. For example, in the $S^+$ setting, in an ideal case the model can only tune the final classification layer according to this method and lose the chance to improve the overall model. The authors argue that the optimal model for permuted MNIST and split CIFAR are independent models because each task is distinct. I would argue that, for permuted MNIST the semantic information is consistent across tasks, so ideally a continual learning model would only need to re-arrange the initial layers to accommodate the change in surface form and keep the rest of the model intact. For CIFAR the sharing would be more significant, because the input were mostly drawn from the same distribution and it is likely to have sharable information across tasks. In fact, I would argue the split to independent model is no longer continual learning, as the models are already independent, we completely lost the continual part.\n\n- The proposed approach also requires a task identifier to work with. I think in classification setting, which is the experiments focused in the paper, providing this identifier alone already solves a large part of the problem. The setting is more realistic in reinforcement learning setting, however, no experiments were carried out under that setting. \n\n- It is not clear what algorithm is used in the experiments for the proposed method. Is the data driven prior always applied?\n\n\nReferences:\n[1] Li, X., Zhou, Y., Wu, T., Socher, R., & Xiong, C. (2019, May). Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting. In International Conference on Machine Learning\n[2] Serra, J., Suris, D., Miron, M., & Karatzoglou, A. (2018, July). Overcoming Catastrophic Forgetting with Hard Attention to the Task. In International Conference on Machine Learning\n[3] Yoon, J., Yang, E., Lee, J., & Hwang, S. J. (2018, February). Lifelong Learning with Dynamically Expandable Networks. In International Conference on Learning Representations.", "extra_scores": {}, "extra_text": {}}, {"review_id": "uP0WI-oHyNj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors investigate modular neural networks in a continual learning (CL) setting, where multiple classification tasks have to be learned in sequence. To avoid the well-known problem of catastrophic forgetting, parameters used by previous tasks are simply frozen, and new tasks are learned by expanding the architecture (as in e.g. PNN, Rusu et al., 2016). The authors focus on efficient forward transfer to new tasks. A many-tasks benchmark is evaluated on the authors' method as well as a number of baselines.\n\nThe paper is very well written and the explanations are clear. The experiments are fair and well-executed. The proposed long-sequence streams provide useful baselines that can be used in future papers, as the authors will share their code.\n\nThe main weaknesses of the paper lie in its originality and applicability. Dynamic architecture expansion for CL is a well-studied approach (as acknowledged by the authors), and while picking an appropriate initial path with a cheap classifier is a good idea, determining how to expand from there remains difficult and largely expensive in the more effective MNTDP-D version.\n\nHere are some suggested improvements for the authors:\n- Baselines: since PNN, improved methods for architecture expansion have been developed. While the provided baselines are interesting and evaluated in a fair manner, I think that the paper would be much stronger if the authors considered as well stronger, more comparable baselines (for example, DEN, Yoon et al., 2018, or CLAW, Adel et al., 2020). \n\n- Other methods have been developed which leverage reinforcement learning to help determine how to expand a network in a CL setting. These should be at least discussed in the light of MNTDP-S (e.g., Xu and Zhu, NeurIPS 2018 and NHuang, Lavet, Rabusseau, arXiv 2019)\n\n- The proposed method cannot achieve backward transfer. This should be noted.\n\n- It should be discussed that the presented method requires knowledge of task identity at test time. This is not always possible.\n\n- \"Finally, the CL algorithm has to yield predictors that scale sublinearly with the number of tasks both in terms of memory and compute.\"\nTo make the statement general, I would note that sublinear memory and time complexity scaling with the number of tasks should be the aim whenever the continual learner is exposed to tasks that are not independent.\n\nEdit: I have raised my score from 6 to 7 after the rebuttal.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "aFqI0FKFN1-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of paper**\n\nThis paper has two main contributions. Firstly, it introduces a new set of benchmarks designed to proble different qualities that may be important for continual learning methods. Secondly, it introduces a new modular method for continual learning, which learns to add modules when learning on a new task. The method reduces the exponential search-space by only considering a few options for how to add the module, leveraging which task the new task is most similar to. Experimental results are provided on the new benchmarks as well as Permuted MNIST and Split CIFAR100.\n\n**Review summary**\n\nI really like the idea behind the benchmark suite for continual learning. Although the algorithm CTrL is neat, I think the experiments/explanation could do with more work (see later in the review). Therefore I am currently leaning towards a weak reject, pending a discussion with the authors.\n\n**Pros of paper**\n\n1. Testing different properties of a continual learning algorithm, one by one, with different benchmarks, is a very nice idea. In fact, it might be nice for the authors to consider expanding the suite of benchmarks to test even more desirable aspects of continual learning / properties of continual learning methods. Some ideas:\n(i) Multiple tasks which are from the same dataset, to test how continual learning systems react. Some will deal very well with this, others will not.\n2. It's great to see memory and compute reported as benchmarks for continual learning.\n3. The proposed modular method is very nicely presented, and is a neat, relatively simple, method. It has nice ideas to it, such as starting with the modular architecture of the most related task only. \n\n**Cons of paper**\n\n4. The paper defines a transfer metric, using only the final task. This works well with the specific benchmarks they propose. This is very similar to a \"Forward Transfer Metric\" from Pan et al., 2020 (\"Deep Continual Learning by Functional Regularisation of Memorable Past\"), which averages the metric over all tasks (except the first task). I was wondering why the authors restricted the transfer metric to be just for the last task.\n5. Although the related work section is long with some nice detail in places, there could be more comparisons in the paper to model-based continual learning methods. The paper could also mention methods based on IBP priors.\n6. I would like more of a discussion relating to Li et al., 2019, which the authors say is the most related work. It would have been nice to use that as a baseline in the experiments. (Although I understand it is difficult if they do not have code available online.) In general, I would have liked to see some stronger / more recent baselines in the experiment section: Independent and PNNs are known to have bad memory growth; ER is the simplest idea in the replay-based methods; Online EWC is also very simple (and old).\n7. I did not understand why the \"Independent\" baseline in Table 1 has non-zero transfer metrics? (I thought that, because it is independent models, the metric should be 0?)\n\n**Update to review**\n\nI have increased the score from 5 to 6. The authors have made the paper stronger with the inclusion of stronger baselines, cleaned-up presentation, and backward transfer. I am particularly excited by the new suite of benchmarks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NyeAOJ6tGua", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides metrics, benchmarks, and algorithms that can evaluate transferability, not just catastrophic forgetting in continual learning. These methods are reasonable, but there seems to be a need for more justification for the five stream cases used in the benchmark for transfer evaluation. This is the main reason for not giving this paper a higher score.\n\nThis paper provides a set of benchmarks for evaluating continual learning algorithms in terms of transfer and scalability, in addition to catastrophic forgetting, which has been mainly used in the past. In particular, five specific task streams were defined, and it is argued that these can be used to assess the transferability, which was not distinguished in the existing benchmarks. Additionally, the authors propose MNTDP, analogous to the modular network, as a new continual learning algorithm to better meet the metrics assessed in the new benchmark.\n\nThis paper is well-organized and well-written. Defining metrics for continual learning in Section 3 is not very new, but I think it is necessary. I personally like the authors' claim that transferability should also be an important measure in continual learning. However, there seems to be something that needs to be clearer in terms of motivation in the part that proposes five task streams for benchmarking. It would have been nice if there was a discussion on whether the 5 cases provided as a measure for transfer overlap each other or whether there is an area that cannot be covered. For example, in the case of (MNIST -> CIFAR-10 -> ... -> F-MNIST), it is unclear where it corresponds to the five stream cases. It is also unclear how much the performance improvement of the T(S^{}) values in Table 1 actually represents. I think that referencing the transfer learning benchmarks can help both justify stream case classification and compare transfer performance results.\n\nAnother question is, in the case of scalability, what does it mean for regularization-based or replay buffer -based rather than modular network-based continual learning?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new benchmark for evaluating continual learning(CL) algorithms on transferability and scalability. It also introduces a data-driven prior to reduce the architecture searching space. Experiments show the new benchmark helps to analyze the properties of CL algorithms and the proposed algorithm performs better than baselines.

The reviewers raised concerns about evaluation metric, weak baselines, and limited experimental cases for evaluating transferability. The authors added more experiments with stronger baselines and revised the paper based on the reviewers' suggestions. However, the authors also admit that how to evaluate transferability is still an open question. 

Despite the concerns, the reviewers generally agreed that the paper is well written,
and the new benchmark is an important contribution for evaluating continual learning algorithms on transferability and scalability.  Hence it makes a worthwhile contribution to ICLR and I'm recommending acceptance of the paper.
Decision: Accept (Poster)