Model Zoo: A Growing Brain That Learns Continually
Authors: Rahul Ramesh
Pratik Chaudhari
Venue: ICLR 2022 Poster (2022)
Keywords: Continual Learning
Learning Theory

Abstract: This paper argues that continual learning methods can benefit by splitting the capacity of the learner across multiple models. We use statistical learning theory and experimental analysis to show how multiple tasks can interact with each other in a non-trivial fashion when a single model is trained on them. The generalization error on a particular task can improve when it is trained with synergistic tasks, but can also deteriorate when trained with competing tasks. This theory motivates our method named Model Zoo which, inspired from the boosting literature, grows an ensemble of small models, each of which is trained during one episode of continual learning. We demonstrate that Model Zoo obtains large gains in accuracy on a wide variety of continual learning benchmark problems.

Published as a conference paper at ICLR 2022
MODEL Zoo: A GROWING "BRAIN" THAT LEARNS CONTINUALLY
Rahul Ramesh & Pratik Chaudhari University of Pennsylvania
(rahulram, pratikac) upenn. edu
ABSTRACT
This paper argues that continual learning methods can benefit by splitting the capacity of the learner across multiple models. We use statistical learning theory and experimental analysis to show how multiple tasks can interact with each other in a non-trivial fashion when a single model is trained on them. The generalization error a particular task can improve when it is trained with synergistic tasks, but can also deteriorate when trained with competing tasks. This theory motivates our method named Model Zoo which, inspired from the boosting literature, grows an ensemble of small models, each of which trained during one episode of continual learning. We demonstrate that Model Zoo obtains large gains in accuracy on a wide variety of continual learning benchmark problems.
INTRODUCTION
continual learner seeks to leverage data from past tasks to learn new tasks shown to it in the future, and in turn, leverage data from these new tasks to improve its accuracy on past tasks. It stands to reason that the performance of such a learner would depend upon the relatedness of these tasks. If the two sets of tasks are dissimilar, learning on past tasks is unlikely to benefit future tasks-it may even be detrimental. And similarly, new tasks may cause the learner to "forget" and result in deterioration of accuracy on past tasks. Our goal in this paper is to model the relatedness between tasks and develop new methods for continual learning that result in good aeward-backwaadd transfer by accounting for similarities and dissimilarities between tasks. Our contributions are as follows.
1. Theoretical analysis. We characterize when multiple tasks can be learned using a single model and, likewise, when doing so is detrimental to the accuracy of a particular task. The key technical idea here is to define a notion of relatedness between tasks. We first show how the inputs of different tasks are "simple" transformations of each other (and likewise for the outputs), then one can learn shared feature generator that generalizes better on every task, compared training that task in isolation. Such tasks are strongly related to each other and therefore it is beneficial to fit a single model on all of them. We show that if tasks not strongly related, in particular if the optimal model for one task predicts poorly on another task, then fitting a single model on such tasks may be worse than training each task in isolation. Such tasks compete with each other for the fixed capacity in the single model. We also empirically study this competition using the CIFAR-100 dataset.
2. Algorithm development. The above analysis suggests that a continual learner could benefit from splitting its learning capacity across sets of synergistic tasks. We develop such a continual learner called Model Zoo. At each episode, small multi-task model that is fitted to the current task and some of the past tasks is added to Model Zoo. This method is loosely inspired from AdaBoost in that it selects tasks that performed poorly in the past rounds and could therefore benefit the most from being trained with the current task. At inference time, given the task, we average predictions from all models in the ensemble that were trained that task.
3. Empirical results. We comprehensively evaluate Model 7o0 existing task-incremental continual learning benchmark problems and show comparisons with existing methods. There a wide variety in the problem settings used by cxisting methods, e.g., some replay data from past tasks (like Model Zoo is designed to do), some replay only subset of data, some train only for one epoch in each episode, some use extremely small architectures. etc. We compare Model Zoo with existing methods in number of these settings. Model Zoo obtains better accuracy than existing methods on the evaluated benchmarks. Improvement in average per-task accuracy is quite large in some cases, e.g., 30% for Split-minilmagentt. We also show that Model Zoo demonstrates strong forward and backward transfer.
4. A critical look at continual learning. We find that even an Isolated learner, i.e., one which trains (small) model tasks from each episode and does not perform any continual learning, significantly

---

Published as a conference paper at ICLR 2022
Comparison of Methods Mink-imagenet
Individual Task Accuracies Spilemiiillmapeet
11
Task Accussy (4)
AGUH 221
35
Tases
Figure 1: Left: How well do existing continual learning methods work? We track the average accuracy (over all tasks seen until the current episode) on the Split-minilmggene dataset and compare our method Model Zoo and its variants (all in bold) to existing continual learning methods (faint lines, see Table I for references). All methods in this plot (unless specified otherwise) use the single epoch setting, i.e. each new task is allowed only I epoch of training. Isolated refers to very simplistic realization of Model Zoo where a separate model is fitted at each episode without any continual learning, or data sharing between tasks; Isolated (aaated-smmll small or Model Zoo-small refer to using very small deep network with 0.12M weights. A number of surprising findings are seen here. (i) Isolated-small (black) outperforms existing methods by more than 10% margin, while having a faster training time, inference time, comparable model size and without performing any data replay. This indicates that existing methods de not sufficiently leverage data from multiple tasks. This also indicates the utility of simple methods like Isolated perform more prosaic, matter-of fact, evaluation of continual learning. (ii) While the larger model with 3.6M weights per round, Isolated-Single Epoch (royal blue), performs poorly, its accuracy is better than all existing methods (Isolated-Multi Epoch) upon heing trained for multiple epochs. This indicates that existing methods may be severely under-trained in the single-epoch setting and this may not be the appropriate setting build continual learning methods: this was also noticed by Lopez-Paz and Ranzato (2017). (iii) Model Zoo and Model Zoo-small which replay all data from past tasks (A-GEM also replays 10% of the data), achieves around 10% improvement over its Isolated counterparts in both the single-epoch and multi-epoch setting; all these 4 methods advocated in this paper are better than existing algorithms. Even Model Zoo-single epoch which replays past data but trains on the new task only for I epoch outperforms existing methods significantly. This indicates that replaying data from past tasks is beneficial (Robins. 1995), even if replay may not conform to certain stylistic formulations of continual learning in the literature (Farquhar and Gal, 2019a; Kaushik et al., 2021). Not doing so significantly hurts forward and backward transfer, and average task accuracy.
Right: Does the single-epoch setting show borward-backward transfer? The evolution of individual task accuracy of Model Zoo (the multi-epocch setting in bold and single-epoch setting in dotted), on the Split- minilmagenct dataset (only tasks are plotted here, Fig. A6 for the full version). The markers denote the accuracy of Isolated. Accuracy of tasks improves with each episode which indicates backward transfer. Also, the X markers are often below the initial accuracy the task during continual learning, which indicates transfer,
forward transfer. While both single-epoch and multi-epoch Model Zoo show good the accuracy tasks for former is about 25% worse than the latter; corresponding plots for other methods are in Appendix B.7. This indicates that we should also pay attention to under- ander-tranning and per-task accuracy in continual learning.
outperforms most existing continual learning methods on the evaluated benchmark problems, e.g., by more than 8% in Fig. 1 and Tables 1 and A1. This strong performance is surprising because it is very simple learner that has better training/inference time, data replay, and comparable number of weights as that of existing methods.
2 A THEORETICAL ANALYSIS OF HOW TO LEARN FROM MULTIPLE TASKS
In this section, we (i) formulate the problem of learning from multiple tasks, (ii) discuss a simple model that highlights when training one model on multiple tasks is beneficial, and (iii) show new results on how the fixed capacity of the model causes competition between tasks.
2.1 PROBLEM FORMULATION
supervised learning task is defined as a joint probability distribution P(x, v) of inputs x E X and labels y E Y. The learner has access to m i.i.d samples S = {xr,, Vs Bil from the task. A hypothesis is a function h : X + Y with h E H being the hypothesis space. The learner may select hypothesis that minimizes the empirical risk (h)
m _ 1 [p(c)))w) with the hope of achieving small population risk (h) (P(()) # y). Classical PAC-learning results ( (Vapnik, 1998) suggest that with probability at least - d over draws of the data S, uniformly for any h E H, we have ep(h) < (s(h) + if
( log /2)
(1)
2

---

Published as a conference paper at ICLR 2022
where D VC(H) is the VC-dimension of the hypothesis space H. We define the "excess risk. of a hypothesis as Ep(h) = ep(h)) - be Ep (h). In the continual learning setting, new task is shown to the learner at each episode (or round). Hence after episodes. the learner is presented with n tasks P := (P1..... ..)). with the corresponding training sets S := (Si...., ,,)), each with m samples, and the learner selects n. hypotheses h = (h1,-.. 1.ly--.-.. hn) E H", each h2 E H. If it seeks small average population risk ep(h) = A, Eim1 1 Ep, (h.), it may do so by minimizing the average empirical risk (h) = E és, (h.). As Baxter (2000) shows, under very general conditions, if
m O(E (n) - 1/ 6)),
(2)
then we have (h( S eg (h) + for any h H The quantity du (n) here is generalized VC-dimension for the family of hypothesis spaces H", which depends on the joint distribution of tasks. Larger the number of tasks n. smaller the d (12) (Ben-David and Borbely, 2008), Whether (2)
an improvement upon training the task in isolation as in (1) depends upon the hypothesis class H and the relatedness of tasks Pı..... Pn through the quantity du (n2) The most important thing to note here is that according to these calculations, if one wishes obtain a small average population risk across tasks, training multiple tasks together cannot be worse: dh (n) S VC(H).
2.2 CONTROLLING THE EXCESS RISK OF A SPECIFIC TASK FOR SYNERGISTIC TASKS
An important goal of continual learning is to have low risk on all tasks. This is a stronger requirement than for (2) which bounds the average population risk on all tasks.
Suppose there exists a family F of functions f; : X - X that map the imputs of one task to those of another, i.e., any task can be written
(A( = f[P]]()) = P.((((()), (z, y) E A})
for some function f E F for any set A. We can assume without loss of generality that F' acts as a group over the hypothesis space and Hl is closed under its action. In simple words, this entails that given E H suitable for task P, we can obtain a new hypothesis o f that is suitable for another task [[Pl. Instead of searching over the entire space H like in 1, we now only need to find a hypothesis h E H. such that its orbit
[h]p- (h' 3f E F with h' = h f}
contains hypotheses that have low empirical risk on each of the tasks. Conceptually, this step learns the inductive bias (Baxter, 2000; Thrun and Pratt, 2012). The sample complexity of doing so is exactly (2). From within this orbit, we can select a hypothesis that has low empirical risk for a chosen task PL. The sample complexity of this second step
1S|==00 -2 (dmax log
(3)
where dmax SU VC((h) p). By uniform convergence, as Ben-David and Schuller (2003) show, this two-step procedure assures low excess risk for every task Pl..... Pa. We have
suPkeи [C(( = dnuax S d + 1) S d S D = VC(H). The total sample complexity favorable to that of learning the task in isolation if both (n) and dmax are small. For instance, if F is finite and n/ log n 2 D. we have dz((n) < 2 log [F] which indicates that we get a statistical benefit of learning with multiple tasks if D > log [F).
(4)
Remark 1 (Data from other tasks may not improve accuracy even if they are synergistic). Let us make a few observations using the above analysis. (i) From (4), number of samples per task m decreases with nı; this is the benefit of the strong relatedness among tasks and as see next, this is not the case in general. (ii) The number of tasks scales essentially linearly with D, which indicates that one should use a small model if we have few tasks. (iii) But we cannot always use a small model. If tasks are diverse and related by complex transformations with a large F... we need large hypothesis space to learn them together. If [F| is large and H is not appropriately so, the VC-dimension dmax is as large as D itself; in this case there is again no statistical benefit of training with multiple tasks together, but there is no deterioration either.
2.3 TASK COMPETITION OCCURS FOR HYPOTHESIS SPACES WITH LIMITED CAPACITY
There could be settings under which fitting one model on multiple tasks may not suffice. To study this, consider weaker notion of relatedness. We say that two tasks P,, P3 are Pty Pty-related if
(5) Here (p) == - and h; argmin ep, (h) is the best hypothesis for task P:: we set c 2 1 to be coefficient independent of j. Smaller the Pij. more useful the samples from P. to learn P... The definition suggests that all hypotheses h which have low excess risk on P. also have low excess risk on Pj up to an additive term ep, (h*) and this effect becomes stronger as Pij + 1.. Note that the definition of relatedness is not symmetric. Hanneke and Kpotufe (2020) call
Ep,
1/p (h) 2 Er, h),, for all h H.
-
3

---

Published as a conference paper at ICLR 2022
this the transfer exponent. To gain some intuition, inequality between the tasks developed by Crammer et al. (2008): in the realizable setting where ep, (h?) = 0, for c, Pij 1, we can write (5) as
connect this definition to certain triangle
ep, (h) + eny (h,) 2 er, (h)
which is akin to triangle with vertices at h. h and h2 with terms like ep, (h) representing the length of the side between /t and 14 This definition therefore models a set of tasks and hypothesis space that is not unduly pathological, ep, (h) cannot be much worse than the sum of the other two sides. We can now show the following theorem bounds the excess risk Er, (h) for a hypothesis h trained using data from multiple tasks. See Appendix C for the proof.
Theorem 2 (Task competition). Say we wish to find good hypothesis for task Pi and have access to n tasks ,l....,A Pn. where each pair P., P3 are Pij Psy-related. Arrange tasks in an increasing order of Pit. i... their relatedness to Pt. Let this ordering be P()) P(2).... P(n)) with P(1) S P(2) .... S P(n)
and P(), P1 and P(1) = 1. Let h² be the hypothesis that minimizes the average empirical risk of the first k I n tasks. Then, with probability at least 5 over draws of the training data,
En, (h+) S E En (1(x) + (es(t)+ cc
(Dxlox
(D-log Amm
1/2
1/Panxx
(6)
where Pmax (k) = max {Pa)..... ((K)) )())}
and c, c' are constants.
Notice that the first term grows with the number of tasks because we pick tasks with lower pa that are more and more dissimilar to P1. The second term typically decreases with k. The empirical risk (h) is typically small; in our experiments with deep networks we achieve essentially zero training error on all. Increasing the number of tasks k; increases the effective number of samples kmm, thereby reducing the second term in totality. At the same time, these new samples are increasingly more inefficient because Pmax (k) increases with k.
Remark 3 (Picking the size of the hypothesis space). The first and second terms characterize synergies and competition between tasks and balancing them is the key to good performance on a given task. Increasing the size of the hypothesis space reduces the first term since it allows a single hypothesis to more easily agree on two distinct distributions P. and P,. However, this comes the cost of increasing the second term which grows with the size of the hypothesis space.
Remark 4 (The set of synergistic tasks can be different for different tasks). The right hand side in (6) is minimized for a choice of k (where S k < n) that balances the first and second terms. The optimal & can vary with the task, e.g., for generic tasks most other tasks will be synergistic and similarly a small optimal l indicates task dissonance where the particular task, say Pi should be trained on with a specific set of other tasks. Even for typical datasets like CIFAR-100, it is highly nontrivial to understand the ideal set of tasks to train with; Fig. 2 studies this experimentally.
Remark 5 (Continual learning is particularly challenging due to task competition). Theorem 2 indicates that not only is the learner shown tasks sequentially, but also may have to work against the competition between the current task and the representation learned on past task. It does not have access synergistic tasks from the future while learning on the current task. And further, in settings where there is no data replay, the learner cannot benefit from past synergistic tasks explicitly, other than the representation that it has already learnt. This suggests that one must be even more careful about how the representation in continual learning should be updated.
3 MODEL Zoo: A CONTINUAL LEARNER THAT GROWS ITS LEARNING CAPACITY
Theorem 2 can be thought of as a "no free lunch theorem". It indicates that ones should not always expect improved excess risk by combining data from different tasks. This theorem also suggests way to work around the problem via Remarks 3 and 4. If we learn small models on synergistic tasks, we can hope to have each task benefit from the synergies without deterioration of accuracy due to task competition with dissonant tasks. Model Zoo is simple method that designed for this purpose. Let us assume that tasks PI...., P.. are shown sequentially to the continual learner. We assume that all tasks have the same input domain X but may have different output domains Yi...., Y.. At each "episode" k. Model Zoo is designed to train using the current task Ps and a subset of the past tasks. For example, at episode k- 2, we train a model with feature generator h and task-specific classifiers to obtain models 91 o h : X + Y1 and g2 o h X + Yz. This model can classify inputs from both tasks and gives out probability vector Po (y x),, Vy E Y; depending upon the task. We assume that the identity of the task is known at the test time.
4

---

Published as a conference paper at ICLR 2022
Accuracy of fbod subset of tasks
Electrical
66.75
69.65
$9.60
70.90
Household Fumiture
65.85
65
60
65.20
66.30
66.25
60.40
66.30
65. 50
655.85
65.25
66. 00
66.65
67.90
68.00
05.95
59.30
68.55
59.15
65.70
69.75
50.45
.0
$9.00
Large Carnivores
24.03
75.00
75.20
73.05
73.50
73.
72.05
73.05
Man-mace Outdoor
70.05
77.03
78.15
70.23
25
75.40
77.45
00
78.70
79.20
Natural Outdoor
Ommi-Herblvone
71:10
67.95
10
69.50
69.60
60 70
69.75
73.00
People
Repties
50. 75
57.70
$7.05
07.
19.25
69.00
12
18
Number trained together
Figure 2: Competition between tasks in continual learning can be non-trivial. In order to demonstrate how some tasks help and some tasks hurt each other, we multi-task learner for a varying number of tasks (X-axis) and track the accuracy on few tasks from CIFARIO0 100 (each task superclass). Each cell represents different experiment, i.e, there is no continual learning being performed here. Cells are colored warm if accuracy is worse than the median accuracy of that row. For instance, multi-task training with 11 tasks is beneficial for "Man-made Outdoor" but accuracy drops drastically upon introducing task #12. it improves upon introducing #14. while task #17 again leads drop. One may study the other rows to reach similar conclusion: there non-trivial competition between tasks, even in commonly used datasets. As we show, tackling this effectively the key obtaining good performance on multi-task learning problems. See Appendix B. more elaborate version.
Let the set of tasks considered at episode k be de- noted by PA= = [Pu]...., Pug.) where S le is hyper- parameter and wi E (1.....ki kj. Training on PR will involve, like the example above, training one model with a feature generator h&g and task-specific classifiers gk.w for each task selected in that round. Such models, one trained in each round, together form the "Model Zoo". After k rounds, data from, say, P. with i < k can be predicted using the average of class probabilities output by all models that were fitted on that task, i.e.,
Model
Model
(y x) x Ei (REAA) It, o ha()))
(7)
Model
Figure 3: Ideally, we want to train synergis- tic tasks together, e.g., Model 1 for P1 using Ps. Po and Model for P3 using P1, Pa, Ps- At test time, all models (1, 2. 3) that were trained on particular task, say P1 would make predictions. Model Zoo is simple, scalable instantiation of this idea. Discov- ering noncompeting tasks is difficult, it selects tasks that have high training loss under the current ensemble.
This expression is also used to predict at test time.
Selecting tasks to train with for each round using boost-
ing. In principle, we could use the transfer exponents Pij to select synergistic tasks, but computing the transfer ex- ponents is essentially difficult as training on all tasks, continual learning does not have access to all tasks a priori. We therefore develop automatic way to select tasks in each round. We draw inspiration from boosting (Schapire and Freund, 2013) for this purpose. Recall the AdaBoost algorithm which builds an ensemble of weak learners (they can be any learner in principle Mason al. (1999)), each of which fitted upon iteratively re- re-weighted training data (Breiman, 1998). We think of the models learned at each episode of continual learning in Model Zoo as the "weak learners" and each round of boosting the equivalent of each episode of continual learning. Let Wj E IR" be a normalized vector of task-specific weights. After episode A
Tkii exp -1/m E(r,)es, log Pt 12)).
(8)
for each task P. with i < k; for i k Wk,i = O. Tasks for the next round P&+1 are drawn from multinomial distribution with weights Wg. Therefore, tasks with low empirical risk under the current Model Zoo get a low weight for the next boosting round. Just like AdaBoost drives down the training error on all samples to zero exponentially (Schapire and Freund, 2013) by iteratively focusing upon difficul-to-classifyy samples, Model Zoo achieves low empirical risk on all tasks as more models are added.
The key feature of Model Zoo is that it automatically splits the capacity across sets of tasks. Even if competing tasks are chosen in one round, which may result in high excess risk on some task, will be chosen again in future rounds if it has large error under the ensemble. Colloquially speaking, the ensemble in Model Zoo represents a "brain" that grows its learning capacity continually as more tasks are shown to it.
Remark 6 (Assumptions in the formulation of Model Zoo). We assume that, both at training time and test time, the identity of the task is known to the continual learner. Data from past tasks is also
5

---

Published as a conference paper at ICLR 2022
stored with the task identity. This is known as the task-incremental setting in the literature (Van de Ven and Tolias, 2019). Recent work in continual learning also studies settings where such task identity is not known, e.g., (Kaushik et al., 2021), Model Zoo is not designed to handle such settings.
4 EMPIRICAL VALIDATION
4.1 SETUP
Datasets. We evaluate on Rotated-MNIST (Lopez-Paz and Ranzato, 2017), Split-MNIST (Zenke et al., 2017). Commuted-NNSST (Kirkpatrick al., 2017), Split-CIARR0 (Zenke et al., 2017), Split-CIFAR100 (Zenke et al., 2017), Coarse-CIFAR 100 (Rosenbaum et al., 2017; Yoon et al., 2019; Shanahan et al., 2021) and Split-minilmagenet (Vinyals al., 2016; Chaudhry et al., 2019b). Split-MNIST, Split-
10, Split-CIFARE 100 and Splitmmiilmaagne use consecutive groups of labels (2, 2, and 10, respectively) form tasks. Coarse-CIFAR CIFAR 100 is variant of CIFARIO0 where each super-class is considered a different task (Yoon et al., 2019; 2021; Shanahan et al., 2021). Our study in Fig. 2 has found that Coarse-CIFARE 100 is a difficult dataset for continual learning, perhaps because of the semantic differences among the different slpss-s.
Neural architectures and training methodology. We use a small wide-residual network of Zagoruyko and Komodakis (2016) (WRN-16-4 with 3.6M weights) with task-specific classi- fiers (one fully-connected layer). We also use an even smaller network 10.12M weights) with convolution layers (kernel size 3 and 80 filters) interleaved with max-pooling, ReLU, batch-norm layers, with task-specific classifier layers. Stochastic gradient descent (SGD) with Nesterov's 's momentum and
learning rate is used to train all models in mixed precision. Ray Tune (Liaw et al., 2018) was used for hyper-parameter tuning using a multi-task learning model on all tasks from Coarse CIFAR- 100. When we do full replay, Model Zoo samples = min(k; 5) tasks at the hth episode: for problems with n = 5 tasks, we set & = 2: note that 6 = 1 indicates no data replay. All hyper-parameetrs are kept fixed for all datasets and all experiments (see $4.2). See Appendix A for more details.
4.2 EVALUATING CONTINUAL LEARNING METHODS
There a wide variety of problem formulations in the continual learning literature (Farquhar and Gal. 2019a; Prabhu et al., 2020; Vogelstein et al., 2020: Lopez-Paz and Ranzato, 2017: Van de Ven and Tolias, 2019). Formulations vary with respect to whether they allow replaying data from past tasks, the number of epochs the learner is allowed to train each task for, and the capacity of the model being fitted. We next explain these different formulations, the rationale behind them, and how we execute Model Zoo to conform to each of these settings.
(i) The strict formulation, e.g., Kirkpatrick et al. (2017); Kaushik et al. (2021), does not allow any replay of data. For the strict formulation of Model Zoo, we simply set Wa, = 0 for all i + k: (8). At each episode, single model is trained on the current task and added to the zoo-we we call this rather simplistic learner Isolated. From a practical standpoint, such a formulation imposes a constraint on the amount of computational resources (compute and/or memory) available during training.
(ii) One can replay data to various degrees, e.g., all of it (Nguyen et al., 2017: Guo al., 2020b), or a subset of it (Chaudhry et al., 201 9a). Just like AdaBoost, Model Zoo is fundamentally can easily execute it with limited replay by
designed to allow full replay of past tasks. However, only using a subset of the data to compute gradient updates and also the accuracy on past tasks in the hth episode. We use the nomenclature Model Zoo (10% replay) to indicate that only 10% of the data from past tasks is used: algorithms like A-GEM (Chaudhry et al., 2019a) also use 10% of past data on CIFAR 100 datasets. See Appendix A.4 for implementation details. Note that Model Zoo without any data replay is simply Isolated. Let us emphasize that across all these problem settings, Model Zoo remains a legitimate continual learner because it gets access to each task sequentially and has a fixed computational budget (6 tasks) at each episode. For a multi-tase learner, the computational complexity scales with the number of tasks.
(iii) To impose strict constraint on the computational complexity of each episode some works, e.g.. Chaudhry et al. (2019a). train each task for a single epoch. We therefore show results using both Model Zoo (single epoch) (where we replay past data for 1 epoch) and Isolated (single epoch) (no replay). Even if the rationale behind using each datum only once is well-taken, one single epoch is quite insufficient to train modern deep networks; if one thinks of biological considerations,
Some works (Rebuffi et al., 2017: Lopez-Paz and Ranzato, 2017: Chaudhay et al., 2019a; Mirzadeh et al., 2020b) evaluate on a split of the CIFARIO6 100 dataset where each task is random subset of 5 classes. We do not evaluate on this variant because it is difficult to exactly reproduce the composition of tasks; as Fig. 2 suggests different compositions can have vastly different task accuracy. This is also highlighted by large differences in the accuracy on Split- Split-CIFAR1dO 100 and
CIFARION in our work.

---

Published as a conference paper at ICLR 2022
Methol
Reslay
Singl-
RouseS
Pernueed
Spli- CIFARIO
qpach
MNIST
MNIST MNIST
CIFARIOO
CIFARIOO
minllmgged
(Kirgganick stal. 20177)
GEM (Lppz Pox and Ranzas, 2017)
M6007
$2.00
*67.8
51.86
A- G.G (Chundley stul. 2019g)
80.1
162.3
61.13
Subl--SCDD
ER Reservoir (Chandrry et al. 2019b)
79.8
108.5
04
MEGA-II (Gio et al., 2020g)
/
91.20
66.12
(Kaunhik
92.73
99.5
APD (Voce al. 2019)
56.51
Our methods
bodaaed-omil
96.88
90.18
69:07
$2.48
Madel Zisssmall
96.65
92.06
73.72
24
Model small (10Fk ceplay)
96.58
77.18
846
bolard Roweell--
85.95
Madel 1is-
91.15
Inslated
N9.73
Model Zoo
99.66
97.71
09.97
98.68
94,99
84,27
90.84
(mn))-ooa))
99.66 98.16
98.11
05.38
83.19
Table 1: Average per-task accuracy (%) at the end of all episodes. MNIST, Permuted- MNISS and Rotated- MNIST are not informative benchmarks for judging forward and backward transfer because even Isolated achieves 99%+ accuracy. Model Zoo outperforms, by significant margins, all existing continual learning methods on all datasets. Accuracy of existing methods is worse than Isolated which suggests little to no forward or backward transfer. Model Zoo-small and Isolated-small have comparable number of weights as that of existing methods, and in some cases, much fewer. Model Zoo-Ro Resnet18-S and Resnet18 8-S architecture from Lopez-Par Paz and Ranzato (2017). Both Model Zoo/Isolated have similar accuracies on Split-CIFARI 100 with 3 different architectures and all of them are better than existing methods. This indicates that the improvement in accuracy is not a result of the specific choice of architecture. For single-epoch numbers refer Fig. I and Table 2. Note: indicates that the evaluation randomly sampled labels and is hence it not directly comparable to other methods. All numbers without marker are from the paper cited in the first column. denotes that the accuracy is not from the original paper but from one of (Nguyen al., 2017; Serra et al., 2018; Chaudhry et al., 2019 Numbers for other methods on Split-Mimil Minilmagenet were computed by us using open-soures source implementations of the original authors.
make use of the
Split- CIFARIOD with each task containing
local-descent algorithms like stochastic gradient descent (SGD) are quite different from recurrent circuits in the biological brain (Kietzmann et al., 2019). We also run single epoch methods using a very small model (0. (0.12M weights); these are Model Zoo/Isolated-small (single epoch).
(iv) Multi-Head trains one single model on all tasks to minimize the average empirical risk with task-specific classifiers; mini-batches contain samples from different tasks. Since Multi-Head is trained on all tasks together, it is not a continual learner, but its accuracy is expected be an upper bound on the accuracy of continual learning methods.
Evaluation criteria. We compare algorithms in terms of the validation accuracy averaged across all tasks at the end of all episodes, average per-task forward transfer (accuracy on new task when it is first seen, larger this number more the forward transfer), average per-task forgetting (gap in the maximal accuracy of task during continual learning and its accuracy at the end, larger this number more the forgetting and worse the backward transfer), training and inference time, and memory. Let us note that forward transfer is also sometimes called "learning accuracy" (Riemer et al., 201 8), and another measure of backward transfer is the gap between the accuracy the end of training and the initial accuracy of the task.
4.3 RESULTS
Table 1 shows the validation accuracy of different continual learning methods on standard benchmark problems. There are many striking observations here.
(i Accuracy of existing methods compared to in Table I (see Table Al as well) is poorer than Isolated. This is surprising because Isolated can be thought of as the simplest possible continual learner-one one that unfreezes new capacity at each episode and does not replay data. This indicates that existing methods may be failing to achieve forward or backward transfer compared to simply training the task in isolation; Table 2 investigates this further.
(ii) In comparison, Model Zoo (all three variants: small, small with 10% data replay and the standard method) has better accuracy compared to both existing methods well as Isolated. This shows the utility of splitting the capacity of the learner across multiple tasks.
(ii) Model Zoo matches the accuracy of the multi-task learner in the last row of Table I which has access to all tasks beforehand. Surprisingly, Model Zoo performs better than Multi-Head in spite of being trained in continual fashion, especially on harder problems like Coarse-Cil Caanse-CIFARIOO and moli-mmiilmmgeent.. This is a direct demonstration of the effectiveness of Model Zoo in mitigating task competition: the capacity splitting mechanism not only avoids catastrophic forgetting, but it can also leverage data from other tasks even if they are shown sequentially.
7

---

Published as a conference paper at ICLR 2022
Table 2 shows comparison of the methods developed in this paper with existing methods on Split-CIFAR 100 in terms of continual-learning specific metries. We find:
(i) There are no significant differences in the forward transfer performance in the single epoch setting; larger variants of Isolated and Model Zoo do not work well here because single epoch is not sufficient to train modern deep networks But Model Zoo and variants show less forgetting, it essentially zero. This indicates that although existing methods are designed avoid forgetting (the single epoch setting aids this directly), say, A-GEM, or EWC, they do forget. Forgetting can be mitigated by the capacity splitting mechanism in Model Zoo. The per-task accuracy of existing methods is also rather low compared to Model Zoo variants.
(ii) If our methods are implemented in the multi-epoch setting, then the forward transfer is exceptionally good and almost as good as the average accuracy of the task. Surprisingly, this does not come at the cost of forgetting, which is again essentially zero.
(iii) Even if Model Zoo and its variants are implemented with very small models (0.12M weights/episode, which is 2.42M weights/20 episodes), the accuracy is better (Table 1). This suggests that Model Zoo is a performant and viable approach to continual learning. In fact, even the larger model used in Model Zoo is a WRN-16-4 with 3.6M weights and therefore we can train multiple models on the same GPU easily; this is why the training time of Model Zoo is about the same as that of Model Zoo-small.
(iv) The simplicity of Model Zoo and its variants results in much smaller training times and comparable inference times as compared to existing methods.
Method
Inference
Training
Storage
Metrics (Muli Epoch) Forgetting Forward (%)
Metrics (Single Epoch)
time
time
Samples
#Weights Accuracy
Accuracy
Forgetiing Forward (9)
(ms/sample)
(min)
(%)
(M)
(%)
(%)
(%)
(%)
EWC
10.34
50
1.6
42.4
17.52
67.76
Prog-NN
82
23.7
59.2
0.0
59,2
GPM
10.34
1048
5-10
1.6
61.2
6.0
67.61
A-GEM
10.34
5-10
62.3
RMN
2712.4
0
11.5
80.01
Our methods
Iselated small
2.34
17.09
0
2.42
90.18
0.0
91.18
71.6
0.0
71.6
Model Zoo-small
11.70
31.71
100
2.42
92.28
0.17
90.0
73.67
0.30
71.91
Model Zoo-small (10% replay)
11.70
22:41
2.42
89.76
0.22
89.8
71.09
0.69
70.5
Iselated
2.34
20.76
0
51.8
91.9
0.0
91.0
50.43
0.0
50.43
Model
31.84
41.86
100
54.8
94.99
0:21
94.02
57.67
088
56.58
Table 2: A comparison of continual learning evaluation metrics on Aplit-CTFAR10N for existing methods
and the methods developed in this paper. Our methods demonstrate strong forward and backward transfer, high per-task accuracy. smaller training times and comparable inference times. Training times of other methods are from Chaudhry et al. (2019a) and it is the total training time in minutes for all tasks. The Inference time is the per sample prediction latency averaged over 50 batchb of size 16. See Appendix A.5 for more details.
Replay
Split-
Split-
Tasks (8) (100%) replay)
Split-
Split-
Method
Ensemble
(%)
CIFARI00
minilmagenct
CIFAR100
minilmagenci
Zoo Isolaied (100x)
71.91
65.80
71.91
65.02
70.48
67.18
72.26
67.33
Split CIFARIOO
73.67
71,46
71.33
70.71
73.67
81.05
Splis-minitmggmmt
81.05
67.26
10
71.97
74.22
73.97
88.76
100
73.67
81.05
74.13
84.9
Figure 4: Ablation studies that show the average per-task accuracy as we vary the size of data replay for Model Zoo (left), the number of past tasks sampled at each episode (middle, 6 = 1 implies no replay), and compare Model Zoo with an ensemble of Isolated models (right). These results are for the single-epoch setting and are therefore directly comparable to those in Table 2 and Table I as far as comparison to other methods is concerned. Accuracy is roughly the same on Split-CIFAR 100 across varying degrees of replay while it improves significantly on replay. Accuracy on Split-CIFAR 1000 is consistent as the number of replay tasks is changed but increases on larger datasets like Split-minilmagnet where there are many more tasks. Finally, the performance of Model Zoo is not merely an artifact of ensembling. Even if Isolated is strong model, a very large ensemble of Isolated compares poorly to Model Zoo with 100% replay: this indicates that Model Zoo effectively leverage data from past tasks without forgetting. See the Appendix for more ahlation studies.
this suggests that Model Zoo also works with very small amounts of data
5 RELATED WORK
Theoretical work on learning from multiple tasks. Works such as Baxter (2000); Maurer (2006), or recent ones like Du et al. (2020); Tripuraneni et al. (2020) study a shared feature generator with task-specific classifiers, and show that the sample complexity of learning a task improves if true task-specific classifiers are diverse enough. It is also appreciated that such shared feature generator may not exist for dissimilar tasks. So a different perspective on the problem can found in Crammer al. (2008); Ben-David et al. (2010); Ben-David and Borbely (2008) who show that learning diverse
8

---

Published as a conference paper at ICLR 2022
tasks requires a larger feature generator and, thereby, more samples; we discuss this in $2.2. We build upon Hanneke and Kpotufe (2019; 2020) to construct the transfer exponent in $2; their work shows that even in very favorable settings, e.g., when all tasks have the same optimal classifier, having access to large number of tasks may not help. Model Zoo is strongly influenced from these results and we think of it as essentially way to circumvent them.
There a number of algorithmic tools to estimate task relatedness, e.g., Evgeniou al. (2005); Cavallanti et al. (2010); Kumar and Daume III (2012), and although such methods are popular in transfer learning (Pentina and Lampert, 2015; Jaakkola and Haussler, 1999). one cannot apply them in continual learning because we do not know the tasks beforehand. As $2 shows, task relatedness is critical to good learning. So, taking inspiration from AdaBoost (Schapire and Freund, 2013), Model Zoo uses simple indicator of which past tasks can benefit from future ones, these are the ones with low accuracy under the current ensemble.
Catastrophic forgetting, has been the focus of number of continual learning techniques, e.g., Chaudhry et al., 2019a; Farajtabar et al., 2020; Guo et al., 2020a), data replay (Robins, 1995; Shin et al., 2017; Lee et al., 2017), new architectures (Serra et al., 2018), generative replay-based (Mocanu et al., 2016; Shin al., 2017: Liu et al., 2020; Ven et al., 2020), ensemble-based (Aljundi et al., 2017: Wen et al., 2020) and methods that select locaaly-redundant directions in the weight space (Kirkpatrick et al., 2017; Aljundi et al., 2018; Mallya et al., 2018; Zenke et al., 2017: Chaudhry et al., 2018). Variational methods, c.g.. (Nguyen et al., 2017; Farquhar and Gal, 2019b), sequentially update posterior over the weights and have an elegant foundation in Bayesian methods but implementing them for large datasets remains a challenge. In spite of intense activity, an effective solution to forgetting remains largely unknown.
episodic memory-baa -assdd ones (Lopez-Pazz and Ranzato, 2017;
Model Zoo embraces the fact that forgetting is fundamental phenomenon of learning multiple tasks and therefore splitting the capacity may be essential; our results indicate that this approach is effectively tackling forgetting. This approach also significantly improves other key metrics, e.g., forward-backwant transfer and computational complexity of training and inference that have received limited attention Dfaz-Rodrígues et al., 2018). Let us note that Model Zoo is designed for the fask-incremental continual learning setting (Van de Ven and Tolias, 2019).
Parameter sharing/isolation. single shared feature generator (i.e., hard parameter sharing) is a popular architecture (Kirkpatrick et al., 2017: Lopez-Paz and Ranzato, 2017: Rebuffi et al., 2017; Nguyen et al., 2017: Mirzadeh et al.. 2020b: Chaudhry et al., 2019b). It has been recognized that this is not sufficient; this has given rise to methods for soft-parameter sharing that either design or learn specialized routing architectures (Rosenbaum al., 2017; Sun et al., 2019; Fernando al., 2017; Devin et al., 2017: Misra et al., 2016; Vandenhende et al., 2019). Model Zoo is a very simplistic instantiation of parameter isolation, or growing (Rusu et al., 2016; Mallya and Lazebnik, 2018; Xu and Zhu, 2018). Model Zoo trains on one episode and never updates the model again but its accuracy does play role in determining whether new model should be used for that past task, not. To extend the analogy, just like soft- port-parameter sharing architectures use, say gradient conflict (Aljundi et al., 2018) or attention (Serra et al., 3018), to determine which synapses to share, Model Zoo uses the training loss of the ensemble decide what task the new model should be trained upon. It is remarkable that such a simple instantiation performs so well; this has not been noticed before.
6 DISCUSSION
Continual learning is an important problem as deep learning systems transition from the traditional paradigm of having a fixed model that makes inferences user queries to settings where we would like to update the model to handle new types of queries. The key desiderata of such a system are clear: it must display high per-task accuracy and strong to develop such a continual learner and investigates the problem using the lens of task relatedness. argues that the learner must split its capacity across sets of tasks to mitigate competition between tasks and benefit from synergies among them. We develop Model Zoo, which is a continual learning algorithm inspired by AdaBoost, that grows an ensemble of models, each of which is trained on data from the current episode along with a subset of past tasks. We show that across a wide variety of datasets, problem formulations, and evaluation criteria, Model Zoo and its variants outperform existing continual learning methods. We also show that a simple baseline method, where a separate, small model is trained independently in each episode, outperforms a number of existing continual methods. Appendix D discusses these results further.
transfer. This paper seeks
9

---

Published as a conference paper at ICLR 2022
7 ETHICS STATEMENT
This work does not raise any ethical concerns.
REPRODUCIBILITY STATEMENT
To ensure the reproducability of our work, the full source code is available at https: //gi thub. com/rahul 1 wwwps/oooooooooooooooooooooooooo
The code contains scripts to download all datasets, implementations of different variants of our method discussed in the paper, scripts for hyper-parameter tuning, and data analysis. We note here that we use the same set of hyyer-parametert for all our experiments; this will help other researchers reproduce our results.
REFERENCES
Aljundi, R., Babiloni. F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. (2018). Memory aware synapses: learning what (not) to forget. In Proceedings of the European Conference Computer Vision (ECCV), pages 139-154.
Aljundi, R., Chakravarty, P., and Tuytelaars, T. (2017). Expert gate: Lifelong learning with network of experts. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3366 3375.
Baxter, J. (2000). A model of inductive bias learning. Journal of artificial intelligence research, 12:149-198.
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, Pereira, F., and Vaughan, W. (2010). theory of learning from different domains. Muchine learning, 19(1-2):151-17.
Ben-David, S. and Borbely, R. S. (2008). A notion of task relatedness yielding provable multiple-task learning guarantees. Machine learning, (3(3):273-287
Ben-David, S. and Schuller, R. (2003). Exploiting task relatedness for learning multiple tasks. Proceedings of the 16th Annual Conference on Learning Theory.
Breiman, (1998). Arcing classifier (with discussion and rejoinder by the author). Annals Statistics, 26(3):0018849.
Cavallanti, G., Cesa-Bianchi, N., and Gentile, C. (2010). Linear algorithms for online multitask classification. The Journal Machine Learning Research, 11:2901 2.244.1-9934.
Chaudhry, A., Dokania, P. K., Ajanthan, T., and Torr, P. H. (2018). Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pages 532-547.
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. (2019a), Efficient lifelong lcarning with a a-gem. In ICLR.
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H., and Ranzato, M. (2019h). On tiny episadic memories in continual learning. arXiv preprint arXiv: a 10486
Crammer, K., Kearns, M., and Wortman, J. (2008). Learning from multiple sources. Journal of Muchine Learning Research, 9(8).
Devin, C.. Gupta, A., Darrell, T., Abbeel, P., and Levine, S. (2017). Learning modular neural network policies for multi-task and multi-rabot transfer. In 2017 IEEE International Conference on Robotics and Automation (ICRA). pages 2169-2176. IEEE.
Rodríguez, N., Lomonaco, V., Filliat, D.,, Maltoni, D. (2018). Don't forget, there more than forgetting: new metrics for continual learning. arXiv preprint arXiv: 1810.13166.
Du. S. S., Hu, W. Kakade, S. M., Lee, J. D., and Lei, Q (2020). Few-Shot Learning via Learning the Representation, Provably. arXiv:2002.c 094002 [cs, math, stat].
Evgeniou, T., Micchelli, C. A., Pontil, M., and Shawe-Taylor, J. (2005). Iearning multiple tasks with kernel methods. Journal of muchine learning research, 6(4).
Farajtabar, M., Azizan, N., Moul, A., Li, A. (2020). Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligncee and Statistics, pages 3762-3777777 3773. PMLR.
Farquhar, S. and Gal, Y. (2019a). Towards Robust Evaluations of Continual Learning. arXiv:1803 1805.0933 [cs. stat].
Farquhar, and Gal, Y. (2019b). A unifying bayesian view of continual learning. urXiv preprint urXiv: 1902.0449
Fernando, C., Banarse, D., Blundell, C., Zwols, Y. Ha, D., Rusu, A. A., Pritzel, A., and Wierstra, D. (2017). PathNet: Evolution Channels Gradient Descent Super Neural Networks. arXiv:1701.08% [cs].
Guo. Y., Liu, M., Yang. T.. and Rosing. T. (2020a). Improved schemes for episodic memory-basee lifelong learning. In Advances in Neural Information Processing Systems.
10

---

Published as a conference paper at ICLR 2022
Guo. Y., Liu, M., Yang. T., and Rosing, T. (2020b). Improved schemes for episodic memory based lifelong learning algorithm. In NeurIPS.
Hanneke, S. and Kpotufe, S. (2019). On the value of target data in transfer learning. In NewrIPS.
Hanneke, and Kpotufe, S. (2020). A no-free-lunch theorem multitask learning. arXiv preprint arXiv:2006. 15785.
Jaakkola, T. and Haussler, D. (1999). Exploiting generative models in discriminative classifiers. In Advances in Neural Information Processing Systems, pages 487-493.
Kaushik, P., Gain, A., Kortylewski, A., and Yuille, A. (2021). Understanding catastrophic forgetting and remembering in continual learning with optimal relevance mapping. arXiv preprint arXiv:2102.11343
Kietzmann, C., Spoerer, C. J.. Sörensen, L. K., Cichy, R. M., Hauk, O., and Kriegeskorte, N. (2019). Recurrence required to capture the representational dynamics of the human visual system. Proceedings of the National Academy of Sciences, 116443)221844-21666
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Bard Grabska-Barwinska, Barwinska, A.. al. (2017). Overcoming catastrophic forgetting neural networks. Proceedings of the national academy of sciences, 114(13)3521-3522
Kumar, A. and Daume III, H. (2012). Learning task grouping and overlap in multi-task lcarning, arXiv preprint arXiv: arXiv:1266.66
Lee. S. Kim, J.HH. Jun, J., Ha, J.-W., and Zhang, B.-T. (2017). Overcoming catastrophic forgetting by incremental moment matching. Advances in Neural Information Processing Systems, 30:4652 4662..
Li, I.,, Jamieson, K., Rostamizadeh, A., Gonina, F.,, Hardt, M., Recht, B., and Talwalkar, A. (2018). A system for massively parallel hyperparameter tuning. arXiv preprint arXiv: 1et..0190003330 05934..
Liaw, R., Liang. E., Nishihara, R., Moritz, P., Gonzalez. J. E., and Stoica, I. (2018). Tune: A research platform for distributed model selection and training. arXiv preprint (00.:v880770511111
Liu. X., Wu. C.. Menta, M., Herranz, L.. Raducanu, B., Bagdanov, A. D., Jui, S., and de Weijer. J. (2020). Generative feature replay for class-incremental learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition Workshops, pages 226-227.
Lopez-Paz, D. and Ranzato, M. (2017). Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6470-6479.
Mallya, A., Davis, D., and Lazebnik, (2018). Piggyback: Adapting single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), pages 67-82.
Mallya, A. and Lazebnik. S. (2018). Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765-7773.
Malviya, P.. Ravindran, B., and Chandar, S. (2021). Tag: Task based accumulated gradients for lifelong learning.
arXiv preprint ar arXtv:2
Mason, L.,, Baxter, J., Bartlett, P., and Frean, M. (1999). Boosting algorithms as gradient descent. Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS'99, pages 512-518, Cambridge, MA, USA. MIT Press.
Maurer, A. (2006). Bounds linear multi multi-task learning. The Journal of Machine Learning Research, 7:117-139.
Mirzadeh, S. I., Farajtahar, M., Gorur, D., Pascanu, R., and Ghasemzzdeh, H. (2020a). Linear mode connectivity in multitask and continual learning- arXiv preprint ar 2010::001.
Mirzadeh, S. I., Farajtabar, M., Pascanu, R., and Ghasemzadeh, H. (2020b). Understanding the role of training regimes in continual learning, arXiv preprint vrXiv:2066.0695x
Misra, L., Shrivastava, A., Gupta, A., and Hebert, M. (2016). Cross-stitch networks multi-task learning. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 3994-4003.
Mocanu, D. C., Vega, M. T., Faton, E., Stone, P., and Liotta, A. (2016). Online contrastive divergence with generative replay: Experience replay without storing data. arXiv preprint 0rX1v:1610.05555.
Nguyen, C. V., Li, Y., Bui, T. D., and Turner, R. E. (2017). Variational continual learning. arXiv preprint 17Xiv:1710.10628
Pan. P. Swaroop. S., Immer, A., Eschenhagen. R. Turner, R., and Khan, E. (2020). Continual deep learning by functional regularisation memorable past. Larochelle, H., Ranzato, M., Hadsell, R., Balcan, E. and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 4453-4464. Curran Associates, Inc.
Pentina, A. and Lampert, H. (2015). Lifelong learning with non-iid tasks. Adv. Neural Inf. Process. Syst.
11

---

Published as a conference paper at ICLR 2022
Prabhu, A., Torr, P. H. and Dokania, P. K. (2020). Gdumb: A simple approach that questions our progress in continual learning. In European conference on computer vision, pages 524-540. Springer.
Rapin, J. and Teytaud, O. (2018). Nevergrad A gradient-free free optimization platiorm. https:/ / FacebookResearch/ Naaerrchhh.eegra....
Rebuffi, S.-A., Bilen, H., and Vedaldi, A. (2017). Learning multiple visual domains with residual adapters. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 506-516.
Riemer, M., Cases, I., Ajemian, R., Liu, M., Rish, I., Tu, Y., and Tesauro, G. (2018). Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.1199
Robins, A. (1995), Catastrophic forgetting, rehearsal and pseudorchearsal. Connection Science, 7(2):133- 146.
Rosenbaum, C., Klinger, T., and Riemer, M. (2017). Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv: 1.000399.
Rusu, A. A.,, Rabinowitz, N. C.,, Desjardins, G., Soyer. H., Kirkpatrick, J., Kavukcuoglu, K.,, Pascanu, R., and Hadsell, R. (2016). Progressive neural networks. arXiv preprint
Schapire, R. E. and Freund, Y. (2013). Boosting: Foundations and Algorithms. Emerald Group Publishing Limited.
Scrra, J., Suris, D., Miron, M., and Karatzoglou, A. (2018), Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pages 4548-4557. PMLR.
Shanahan, M., Kaplanis, C., and Mitrović, J. (2021). Encoders and ensembles for task-free continual learning. arXiv preprint arXiv:2105.13327
Shin, H., Lee. J. K., Kim, J., and Kim, J. (2017). Continual learning with deep generative replay. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 2994-3003.
Sun, X., Panda, R., Feris, R., and Saenko, K. (2019). Adashare: Learning what to share for efficient deep multi-task learning. arXiv preprint arXiv: 1224.2
Thrun, S. and Pratt, I.. (2012). Learning to Learn. Springer Science & Business Media.
Titsias, M. K., Schwarz, J., de G. Matthews, A. G., Pascanu, R., and Teh, Y. W. (2020). Functional regularisation for continual learning with gaussian processes. In International Conference Learning Representations.
Tripuraneni, N., Jordan, M. L., and Jin, C. (2020). On the Theory of Transfer Learning: The Importance of Task Diversity. arXiv:2006. 1650 [cs, stat].
Van de Ven, G. M. and Tolias, A. (2019). Three scenarios for continual lcarning. arXiv preprint 1xXiv:1904.07734
Vandenhende, S., Georgoulis, S., De Brabandere, B., and Van Gool, L. (2019). Branched multi-task networks: Deciding what layers to share. arXiv preprint arXiv: 194.0090...2220.
Vapnik, V. (1998). Statistical Learning Theory. John Wiley & Sons.
Ven, G. M., Siegelmann, H. T., Tolias, A. S., al. (2020). Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, 11(1):1-14.
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al. (2016). Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29:3630-366 -3338.
Vogelstein, J. T., Dey, J., Helm, H. S., LeVine, W., Mehta, R. D., Geisa, A., van de Ven, G. M., Chang, E.,, Gao, C., Yang, W., et al. (2020). Omnidirectional transfer for quasilinear lifelong learning. arXiv preprint N0Ki::2004.29008
Wen, Y., Tran, D., and Ba, (2020). Batchensemble: an alternative approach to efficient ensemble and lifelong learning. arxiv preprint arXiv:2002.0671.
Xu, J. and Zhu, Z. (2018). Reinforced continual learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 907-916.
Yoon, J., Jeong, W., Lee, G., Yang, E., and Hwang, S. (2021). Federated continual lcarning with weighted inter-client transfer. In International Conference on Machine Learning, pages 12073-12086. PMLR.
Yoon, J. Kim, S., Yang, E., and Hwang, S. J. (2019). Scalable and order-robust continual learning with additive parameter decomposition. arXiv preprint arXiv:1902.000 09432.
Zagoruyko, S. and Komodakis, N. (2016). Wide residual networks. arXiv preprint arXiv:1605.07146.
Zonke, F., Poole, B., and Ganguli, S. (2017). Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987-3995, PMLR. PMLR
12

---

Published as a conference paper at ICLR 2022
A DETAILS OF THE EXPERIMENTAL SETUP
A. DATASETS
We performed experiments using the following datasets.
1. Rotated-MNIST (Lopez-Paz and Ranzato, 2017) uses the MNIST dataset to generate different 10-way classification tasks. Each task involves using the entire MNIST dataset rotated by O, 10, 20, 30, and 40 degrees, respectively.
2. Permuted-MNIS (Kirkpatrick et al., 2017) involves 5 different 10-way classification tasks with each task being different permutation of the input pixels. The first task is the original MNIST task as is convention. All other tasks are distinct random permutations of MNIST images.
3. Split-NNIST (Zenke et al., 2017) has tasks with each task consisting of 2 consecutive labels (0-1, 2-3, 4-5, 6-7, 8-9) of MNIST.
4. Split-CIFAR10 (Zenke et al., 2017) has 5 tasks with each task consisting of consecutive labels lairplane-automobile, bird-cat, deer-dog, frog-horse, ship-truck) of CIFARIO.
5. Split-CIFARION (Zenke al., 2017) has 20 tasks with each task consisting of 5 consecutive labels of CIFAR 100. See the original paper for the exact constitution of each task.
6. Coarse-CIFAR 100 (Rosenbaum al., 2017) has 20 tasks with each task consisting of 5 labels. The tasks are based on an existing categorization of classes into super-classes kriz/cifar.html).
https://www.cs.torontooo
7. Split-minimmagentt (Vinyals et al., 2016) is a ariant introduced in Chaudhry et al. (2019b), consisting of 20 tasks, with each task consisting of 10 consecutive labels. We merge the meta-train and meta-test categories to obtain continual learning problem with 20 tasks. Each task containing 10 consecutive labels and 20% of the samples are used as the validation set.
The CIFARIO and LIFAR100-based datasets consist of RGB images of size 32x32 while MNIST-based datasets consist of images of size 28. The Mini dini-imagenet dataset consists of RGB images of size 84 x4.
A.2 ARCHITECTURE
We the Wide-Resned (Zagoruyko and Komodakis, 2016) architecture for some of our experiments. The final pooling layer is replaced with an adaptive pooling layer in order to handle input images of different sizes. Convolutional layers are initialized using the Kaiming-Normn Normal initialization. The bias parameter in batch normalization is set to zero with the affine scaling term set to one. The bias of the final classification layer is also set to zero; this helps keep the logits of the different tasks on a similar scale.
To ensure that the number of weights is similar to those in other methods, we also consider a smaller convolution neural network consisting of 3 convolution layers, with batch-normalization, ReLU and max-pooling present between each layer.
A.3 TraiNING SETUP
Optimization. All models are trained in mixed- precision (32-bit weights, 16-bit gradients) using Stochastic Gradient Descent (SGD) with Nesterov's 's acceleration with momentum coefficient set to 0.9 and cosine annealing of the learning rate schedule for 200 epochs. Training of any model with multiple tasks involves
that contain samples from all tasks.
Hyper-parameter optimization. We used Ray Tune (Liaw et al., 2018) for hyper-parne parameerre opti- mization. The Asyne Successive Halving Algorithm (ASHA) scheduler (Li et al., 2018) was used to prune hyper-parameter choices with the search space determined by Nevergrad (Rapin and Teytaud, 2018). The mini-batch size was varied over [8, 16, 32, 64]; the logarithm (base 10) of the learning rate was sampled from a uniform distribution on [-4, -2]: dropout probability was sampled from a uniform distribution on 10.1, 0.5]: logarithm of the weight decay coefficient was sampled from [-6,- 2]. We used set of experiments for continual learning the Coarse-CIFAR dataset with different samples/class (100 and 500) to perform syper-parameter tuning.
The final values of training hyper-parameters that were chosen are, learning-rate of 0.01, mini-batch size of 16, dropout probability of 0.2 and weight-decay of 10-3
Model Zoo uses 6 = min (k. 5) at each round of continual learning where n is the number of tasks; for tasks with only tasks (MNISS-vvriants) we use 6 2. We did not tune these two hyper-parameters using Ray because it is quite cumbersome to do so. We selected these values manually across few experiments; changing them may result in improved accuracy for Model Zoo.
13

---

Published as a conference paper at ICLR 2022
All hyper-parameters are kept fixed for all datasets, architectures, and experimental settings. We are interested in characterizing the performance of Model Zoo and its variants across a broad spectrum of problems and datasets. While we believe we can get even better numerical accuracy, by tuning hyper-parameters specially for each problem, we do not so for the sake of simplicity. As the main paper discusses, we outperform existing methods quite convincingly across the board in both multi-task and continual learning.
Data augmentation. MNIST and CIFAR 10/100 datasets use padding (4 pixels) with random cropping to an image of size 28x28 or x32 respectively for data augmentation. CIFAR 1AARR00010000 images additionally have random left/right flips for data augmentation. Images are finally normalized to have mean 0.5 and standard deviation 0.25. moit-minilmaggeet uses the same augmentation as CIFAR-10 and 100. We use augmentation even in the single epoch setting.
A.4 MODEL Zoo WITH LIMITED REPLAY
As discussed in $4.2, this work considers Model Zoo (10%) which stores only 10% of the data from the past tasks, in order compare to other methods that make use of limited replay. When the task (say task A) is first seen, Model Zoo is allowed to use all available data. For all future episodes, if Model Zoo picks past task retrain with, such a retraining uses only a fixed subset of the tasks' data (10% of the samples are selected at random for this purpose). We sample each mini- batch to contain an equal number of samples from all past and current task. At inference time, the member of Model Zoo that is trained on all data of task A (this the model that was fitted when task A was first shown to the continual learner) is assigned a proportionately larger weight in Eq. (7). For 10% replay, this will amount to 10 larger weight than other models which used 10% data from task A. Mathematically, both of these training and inference modifications are equivalent to using coefficients that scale up the loss of the past task depending upon the number of samples that it has.
A.5 EVALUATING TRAINING AND INFERENCE TIMeS
In this section, describe the methodology used to estimate training and inference times reported in Table
Inference time. The column titled inference time corresponds to per-sample prediction latency in milli-seconds. All entries for inference time in Table 2 were computed by GPU and therefore they can be compared directly with each other. Note that inference times can be computed using only the architecture built by each method at the end of all continual learning episodes. We obtained the architectures used in each method from open-source implementations of the original authors (htts://gitthu. wtapiituuuuoo/fceeheeee httm//aaeboooreeerreeeeeee continual-learning Inference time is computed by processing 50 mini-batches from CIFAR- 100, each of batch-size 16. The inference time is computed by normalizing the total computation time by (size of For Model-Zoo, we assume that inference time approximately 6 = 5 times the inference time of Isolated, where 6 is number of tasks sampled in every round).
an Nvidia V100
and https://github. http::/ggihubb..oommmiraattt/t/t com/imirzadeh//taalle
x number of mini-batches), which gives the average inference- per sample.
Training time. corresponds to the time (in minutes) required to train all episodes of the Split- CIFARIO0 dataset. Establishing an accurate comparison is difficult because different papers used different hardware but we have strived to be fair. The training time for EWC, Prog-NN, GEM and A-GEM are obtained from Chaudhry et al. (2019a) (we divide the numbers by 5 since this paper reports the sum of training times of 5 different runs). Chaudhry et al. (2019a) also report the training time for naive fine-tuning (21 mins) which in theory. should be very similar to the training time of our Isolated learner (the training time for us is 20.76 mins on one V100 GPU). Since the two numbers are quite similar, we can estimate training time of the other continual learning methods using their computational cost relative to naive fine-tuning. Therefore, the estimate of the training times that we have reported in Table 2 can be compared to each other.
B ADDITIONAL EXPERIMENTS
B.I UNDERSTANDING TASK COMPETITION
To understand which tasks aid each other's 's learning and which compete for capacity and may thereby deteriorate performance, we investigated the Coarse-CIFAR dataset extensively. We first computed the pairwise task competition by comparing the relative gain/drop in classification accuracy of each pair of tasks when the row task is trained in isolated versus training the row and column tasks together using a simple multi- task learner MMulti-Head) Fig. A1 discusses the results.
Fig. A2, is the extended version of Fig. 2. It shows the validation accuracy of each task (along a single row) as more tasks are added to Multi-Head. Each column is a single Multi-Head model trained on subset of tasks from seratch. As more tasks are added, the accuracy of most tasks increases.
14

---

Published as a conference paper at ICLR 2022
2.6
2.6
2.6
Aq. Mammals
Fish
5.2
0.8
0
1.8
2.8
1.4
2.8
4.2
1.6
4.8
2.8
0.6
1.8
6.2
2.6
2.0
Flowers
00.
5
0
1.4
-0.8
-0.2
1.0
0.2
2.8
0.8
1.6
4.2
-0.2
3.0
-1.2
Container
3.6
3.6
Rel change in Task Accuracy from MURHaNk training
Fruits Veggies
3.2
5.8
4.6
2.4
2.6
Electrical Devices
1.2
3.4
2.6
1.8
3.4
2.2
0
3.4
4.G
0.8
3.2
3.8
6.2
0.8
1.2
3.4
1.44
0.8
2.0
0.44
Househald Furiitur-
5.2
4.4
4.6
2.6
10
3.4
9.4
2.4
5.2
7.4
4.8
2.4
5.6
2.6
7.6
1.4
6.2
5.8
4.4
Large Cariiores
3.4
Man made Outdoor
0
-0.4
1.2
-12
0.8
0.6
2.2
1.2
2.4
3.4
-1.4
3..6
Natural Outdoor
.08
2.8
-4
2
-0.4
3.8
0.8
2.2
1
1.8
-0.4
0.8
.1.4
-1.8
0.6
0.2
11.4
2.4
People
5.4
3.8
1.8
3.2
Reptiess
0.4
0.2
2.8
-0.4
1.8
2.2
2.4
0.8
0.8
0.4
1.2
2.22
-1.4
3.4
0.6
-0.4
Small Mammals
1.8
2.6
0.4
2.8
12
2
0.8
2.2
1.4
2.8
3.2
4.9
00.4
5.2
1.6
2.66
-1.4
0.6
3.4
Invertebrates
1.8
0.8
5.4
4.4
6.6
4.4
Vehicles 1.4
3.4
2.2
3
.02
1.4
2
3.2
2.2
3.6
1.6
2.4
2.6
4.8
0.6
0.4
Vehicles 2
1.2
-3.2
2.8
5.4
0.4
1.8
3.6
3.8
4.8
1.8
-2
3.8
0.6
+0.8
-0.2
Medi Manmats
An Hammaas
Füh
Foed Container
Flowers
Fruste and Vegglee
Elecuncal Drsicess
Houneheld Furntura
Large Chmiseres
ireects
Man Man-made Outdinrr
Natural Ouidoor
Omni Her orses
Perple
Smail Mansmale
Repitles
Invertebrata
Trees
Vehicles
Vehicles
Task
training
Figure A1: Pairwise task competition matrix. Cells are colored by the gain(green)lloss( (warm) of accuracy pairwise Multi-Head training as compared to training the row-task isolation; this is good proxy transfer coefficient Pty in (5). Although most pairs benefit each other (green), certain tasks, e.g.. "Food Container" are best trained in isolation while others such as "Aquatic Mammals" are typically detrimental to most other tasks. One can study this matrix and identify many more such properties. In summary, whether tasks aid or hurt each other is quite nuanced even for CIFAR100.
15

---

Published as a conference paper at ICLR 2022
Mod. Mammals
22.29
13.63
14:95
23.23
22:05
75:03
76.05
74..00
73.70
73.75
73.60
73.20
76.22
75.
75.40
Aq. Mammals
33.03
53.55
53.05
$2.83
55.00
25.25
54.65
53:23
54.75
36.60
55.85
37.03
Fish
$4.00
55.15
05.00
05.
57.25
6.7/0
57.80
07709
67:20
57.75
68.00
57.60
69.15
Flowers
03.50
54.95
56.35
55.85
66.45
57.80
65.50
56.65
66.90
65.
69.10
Food Container
53.00
03:21
$5.10
56.40
10.30
08:35
65.05
Fruits and Veggles
75.00
77.55
77:15
77.75
79.60
77.70
78.20
77.00
77.60
77.30
76.60
78.23
Electrical Devices
69.30
66..75
10.25
67.
70.50
Househald Furiitrre
65.05
65./10
00.33
05.20
06.40
55.00
65.85
05.25
07.00
Accurasy of taok
Insects
00.00
68.95
69:15
69:75
60.45
70 40
69.00
Large Carrivores
24.03
75,00
73.05
73.50
75.
Man-made Outdoor
78.55
77.30
78:15
29.23
78.40
27.45
79.00
Natural Outdoor
79.29
70.25
78.
70.40
77.40
76.45
Omri-Hartivvres
P5.10
0.709
70.39
69.30
People
41.25
41.55
Raptliss
57.05
Small Mamma's
56.45
55.15
56. 90
57.05
57.15
Tress
61.10
61.85
Inverisbrates
55.10
65.20
Vehicias 1-
74.05
73,00
Vahicies
04.30
10
12
13
15
18
Number of tasks trained together
tasks hurt each other, run Multi-Head few tasks from Coarse-CIFAR100. The order of tasks the same for rows (top bottom) and the columns (left right). other words, the first cell (the diagonal) indicates the accuracy the task trained by itself isolation (Isolated). Cells are colored warm accuracy is worse than the median accuracy that row. For instance, multi-task training with 11 tasks beneficial for "Man-made Outdoor" but accuracy drops drastically upon introducing task #12, it improves upon introducing #14, while task #17 again leads to a drop. One may study the other rows to reach a similar conclusion: there is non-trivial competition between tasks, even in commonly used datasets. Tackling this issue effectively is the key to obtaining good performance on multi-task learning problems
Figure A2: In order to demonstrate how some tasks help and for varying number of tasks (X-axis) and track the accuracy
However, the increase not monotonic with each added task, and if one follows a particular row, there are non-trivial patterns wherein adding a particular task may deteriorate the performance on the row task and adding some other task later may recover the lost accuracy. This is direct demonstration of the tussle between the task competition term (first) and the concentration term (third) in Theorem 2. This indicates that training the appropriate set of tasks is crucial to learn from multiple tasks.
B.2 COMPETITION BETWEEN TASKS OF TYPICAL BENCHMARK DATASETS
Next, we investigated such task competition on other continual learning datasets, namely, Permuted- MNIST, Rot-MNIST, COIitCIFFRROO, and Split-MNIST. is clear from Fig. A3 that there is very little competition in this case. Either the tasks are quite different from each other (like the case of Permuted-MNISTT, they are synergistic (most cells are green), they do not hurt each other's performance, i... they may correspond to the model in $2.2. Note that Rotated-MNIST exactly corresponds to the multi-view setting discussed in $2.2 were different input images simple transformations of each other.
B.3
VISUALIZING SUCCESSIVE ITERATIONS OF MODEL. Zo0
In order understand how the accuracy of Model Zoo evolves on all tasks as a function of the episodes, we created Fig. A4. This is very insightful picture and we can draw the following conclusions from it.
(i) The accuracy along the diagonal of most tasks increases along the row, i.e., across episodes. Only for few tasks like Food Container the accuracy drops in later episodes. Note that we also see from Fig. Al that Food Container is a task that best trained in isolation because leads to deterioration of accuracy when trained with essentially any other task.
(ii) The is strong backward transfer throughout the dataset, i.c., the accuracy of task shown in earlier rounds increases, sometimes a large number, as later synergistic tasks are shown to the learner.
16

---

Published as a conference paper at ICLR 2022
Texa Tax0
1.37
0:35
2.65
1.92
Task? Takk3 Trakk Tear3
0.23
0.34
0.31
0.07
Del dhange Taxe fecuncy
Bel drange to Taxt Fauurey
0.34
0.58
1.72
0.47
0.43
0.06
0.27
0.13
Texi
1.18
1.25
0.74
0.79
0.75
0.58
0.1
0.7
Tanks
3.44
-5.19
5.13
0.70
0.43
0.58
0.34
4.17
TaKKS
0.00
88
-0.11
-1
PM
Tasts
0.53
0.43
0.41
0.35
Takk3
Tasss
Tasks
brinig
Other
mastittase
Split-CIFAR10
Split MNVIS
Tex-
0.50
3:21
Rel change The necckeainyy
Rel change Twe Cheamey
Texlo TMa0
1.00
1.35
0.30
Tosks Task Taxkd "Taa2
-0.15
.0.10
2.05
1:25
1.70
1.70
2,67
2:35
Tackt
0.25
-9.00
1.45
0.35
0.40
0.20
0.40
0.05
TAxES
6.05
4:80
3.15
7.55
2.72
0.66
2.42
2.67
Figure A3: Each row the relative increase/decrease (green/red) in accuracy of a two task Multi Heatd-HHadd learner compared to Isolated trained on the task corresponding to the particular row; all entries are computed using 100 samples/class. Cells are colored green for accuracy gained, and warm for accuracy dropped; the entries in this matrix are good proxy for the transfer coefficient in the right panel of Fig. 2. Split-CIFARI 10 and Split- MNISM indicate that most tasks mutually benefit cach other. This is also true, but to a lesser extent, for Rotated- than these, perhaps because there is obvious relationship between the tasks and there exist some tasks that lead to large deterioration of accuracy.
in (5). A similar plot for Coarse-CIFAR 100 tasks shown
Permuted-MNISS is qualitatively different problem
(iii) We also see strong forward transfer. Roughly speaking, in the second half of the rows, the initial accuracy of most tasks does not improve much with successive episodes. This suggests that these tasks already have a good initial accuracy, i... there good forward transfer in the learner.
We advocate that such plots should made for different continual learning algorithms to obtain precise picture of the amount of forward and backward transfer.
17

---

Published as a conference paper at ICLR 2022
Aq. Mammals-
Fish
72
74
78
90
96
95
95
96
96
96
96
96
94
93
93
93
9X
91
91
Flowers
74
86
87
94
Food Container
833
83
83
93
93
90
07
Fruits and Veggies
67
25
24
74
24
74
68
68
68
68
68
80
Electrical Devices
Houschold Fumiture-
58
71
71
82
82
Insects
91
87
92
92
92
Large
80
83
92
92
60
made Outdoer
7d
26
76
82
82
82
82
82
82
82
Natural Outcoor
62
Omni-Herbivvres
77
77
77
77
Med. Mammals-
70
70
People
65
65
65
65
66
65
Hepties
69
Mammals
Trees
69
69
Vehicles 1-
Véhicles 2-
Round-0
Round-1
Round-22
Round-3
Round-4
Round
Round- 6-
Round-7-
Roun-- 8-
Round 10-
Round1l 11-
Round-12 12-
Round13
Round
Round 15-
Round-16
Round-17
Round-a 18
Figure A4: The itcrations of Model Zoo visualized for the Split-mini manilmagenett dataset for 20 rounds, with
tasks selected every iteration Model Zoo. Red elements are tasks that were selected by boosting in that particular round. We observe the accuracy most tasks improves over the rounds, which indicates the utility Model like training scheme This plot also indicates that Model Zoo can improve per-task task accuracy on nearly tasks. The model trained for only single-epeh per boosting round.
18

---

extent) unlike Published as a conference paper at ICLR 2022
B.4
BASELINE PERFORMANCE OF ISOLATED TRAINING COARSE-CIFARIIN
Isolated Accuracies (100 samples/label)
100
75
09
76,0
76.6
74.5
78.4
72.4
80.4
65.1
65.6
NO.7
Accuracy
55.8
52.7
50
Mammals
Electrical Devices-
Fish
Flowers
Veggies-
Household Furniture-
Insects
Inventebrates
Large Carnivores-
Man-made Outde
Med. Mommalss
Natural Outdeer-
Omnikcrrbivores
People-
Repiiles-
Small Marnmals-
Trees-
Vehicles 1-
Vehicles 2-
Food Canta
Task
Isolated Accuracies (500 samples/labee)
91.0
87.8
85.4
61.2
83.2
82.2
75.0
74,8
74.6
Accuracy
52.4
50
25
Aa. Mammals-
Fish-
Flowers-
Container-
Fruits Veggles-
Household Furnitur--
Insects-
invertbrotes-
Large Carnivrres
mane Outdoor
Med. Mammals-
Natural Outdoor-
People-
Reptites-
Small Mammalc-
Trees-
Vehicles 1-
Vehicles 2-
Electrical
Omni
Task
Figure A5: Per-task accuracies of Isolated on the Coarse-CIFAR100 dataset for two cases, one with 100 samples/class (top) and another with all 500 samples/class (bottom). Two points are very important to note here. First. there is large improvement in the two accuracies for all tasks when the learner has access to more samples. Second, different tasks have very different accuracies when trained in isolation (using the same WRN- 16-4 model). This indicates that different tasks are very different in terms how hard they are, for some tasks such as People, the base accuracy of the model is quite low and one must have lots of samples in order perform well. A lot of other multi task learning datasets, e.g., derivatives of MNIST (or even CIFARIO CIFAR100 in this respect.
B.5 ADDITIONAL EXPERIMENTS
Table A1 is detailed version of Table 1 in the main paper.
19

---

Published as a conference paper at ICLR 2022
Method
Replay Single
Roiated.
Permuted-
Split-
Split-
Split-
Course
Split-
Epoch
MINST
MINST
MNIST CIFARIO CIFARI00 CIFARIOO Minitmagenet
GEM (Lopez-Paz Ranzato, 2017)
82.60
67.8"
A-GEM (Chandhry et al,, 2019a)
0
89.1
62.3"
61.13
ER- Reservoir (Chandhry et al., 2019h)
79.8
68.5'
64.03
MC-SGD (Mizzdeeh al., 2020s)
/
82.63
85.3
63.30
MEGA-Il (Guo et al., 2020aa
91.20
66.12
OGD (Farajaabur et al., 2020)
X
88.32
86.44
98.84
Stable-GGD (Mizzadch al., 2020b)
70.8
80.1
59.9*
57.79
TAG (Malviya et al. 2021)
/
62.79
57.2
VCI. (Nguyen 2017)
95.5
98.4
FRCE. (Titsias 2020)
94.3
97.8
FROMP (Pan 2020)
99.0
(Kirkpatrick
Prog-Nec Nets (Rusu 2016)
*59.2
(Zenke et al., 2017)
*97.1
*98.9
HAT(Seraa et sl., 2018)
X
98.6
99.0
APD (Yoon al., 2019)
X
56.81
FedWelT (Yoon al., 2021)
X
55.16
RMN (Kaushik et al., 2021)
X
97.73
99.5
80.01
Our methods
Isoluted-small
x
96.88
90.18
69.07
82.48
Model Zoo-small
x
96.85
92.06
73.72
94.27
Model Zoo-small (10% replay)
X
96.58
89.76
77.18
84.6
Beesated-Resntt
X 98.03
88.95
Model 7oo Resnet
X
93.15
Isolated
X
99.64
99.98
97.46
91.90
80.72
86.28
Model Zoo
99.66
97.71
99.97
98.68
94.99
84.27
96.84
Mult-Head (multi-task)
99.66
98.16
99.98
98.11
95.38
83.19
90.83
Table A1: Average per-task accuracy (%) for continual learning at the end of all episodes. MNIST, Permuted- MNIST and RotAted-MNIST not informative henchmarks for judging forward and backward transfer because even Isolated achieves 99%+ accuracies. Model 7oo outperforms, by sigmificant margins, all existing continual learning methods; in fact their accuracy evaluation was on Split-CIFAR
than Isolated which suggests little no forward backward
task containing randomly sampled labels and hence not directly comparable to other methods. All numbers without marker are from paper cited
transfer. Note: indicates
first column. denotes that the accuracy is not from the original paper from one of (Nguyen et al., 2017; Serra et al., 2018; Chaudhry et al., 2019a). Numbers other methods on Split-Minilaggene were computed by us using open-source implementations of the original authors.
B.6 SINGLE EPOCH METRICS
obtain metrics from publicly available implementations of a few different continual learning algorithms, which are shown in Tables A2 and A3. We see that Model Zoo and its variants uniformly have essentially no forgetting and good forward transfer. The average per-task accuracy is also higher than existing methods on these datasets. These tables show results for single-epoch training (to be consistent with the implementation of these existing methods).
Method
Avg. Accuracy
Forgetting
Forward
SGD
34.52
19.88
53.30
EWC
34.71
18.60
52.19
AGEM
37.23
16.96
52.72
ER
41.36
14.29
54.87
Stable-SGD
37.27
12.07
48.43
TAG
43.33
12.39
55.1
Isolated-smal
58.719
0.0
58.71
Model Zoo-small
60.3
0.370
59.13
Isolated-large
41.28
0.0
41.28
Model Zoo-large
46.98
0.38
44.43
Table A2: Single Epoch continual learning metrics on
20

---

Published as a conference paper at ICLR 2022
Method
Avg. Accuracy
Forgetting
Forward
SGD
46.69
16.653
62.35
EWC
47.93
14.26
61.34
AGEM
51.86
10.102
61.13
ER
55.41
9.52
64.03
Stable-SGD
49.28
9.76
57.79
TAG
58.38
5.15
63.00
Isolated-small
65.8
0.0
65.8
Model Zoo-small
81 $1.049
1.278
66.57
Isolated-larg large
40.2
0.0
40.25
Model Zoo-large
64.12
0.27
48.34
Table A3: Single Epoch continual learning metrics Split-Minlmagenet
B.7 TRACKING INDIVIDUAL TASK ACCURACIES
We next study how the individual per-task accuracy evolves on different datasets. The following figures are extended versions the right panel Fig. We that the accuracy of all tasks increases with successive episodes. This is quite uncommon for continual learning methods and indicates that Model Zoo essentially does not suffer from catastrophic forgetting. We have also juxtaposed the corresponding curves of the single-epoch setting with the multi-epoch training in Model Zoo; would like to demonstrate the dramatic gap in the accuracy of these problem settings. Even if single-epoch variant Model Zoo also does forget (its accuracy is much better than existing continual learning methods), the multi-epoch variant has much higher accuracy for every task. This indicates that continual learning algorithms should also focus on per-task accuracy in addition to mitigating forgetting, if they are to be performant. The performance of Model Zoo is evidence that can build effective continual learning methods that do not forget.
Individual Task Accuracies on Coarse-CIFAR100
75
Task Accuracy (%)
25
Multi Epoch
Single Epoch
Isolated
Ag. Mammals Is
Fruits and Veggies
Large Carnivares
Med. Marmals
+
Small Mammals
Man-made
Invortebrates
Flowers
Household Furniture
Naturel Outdoor
People
Vehicles
Food Container
Insects
Omni-Herrbivoees
Reptiles
Vehicles
10
15
20
Number of Tasks
Figure A6: Evolution task accuracy on Coarse-CIFAR CIFARI00
21

---

Published as a conference paper at ICLR 2022
Individual Task Accuracies on Split-CIFAR100
100
90
Task Accuracy (%)
80
Multi Epoch
Single Epoch
Isolated
Task1
Tasks
Task9
Task13
Task17
Task2
Task6
Task10
Task14
Task18
Task3
Task7
Task11
Task15
Task19
Task12
Task2o
10
15
20
Number Tasks
Figure A7: Evolution of task accuracy on Split-CFFAR100
Individual Task Accuracies on Split-minilmagenett
100
x
Task Accuracy (%)
50
Epoch
Single Epoch
Task1
Task5
Task9
Task13
Task17
Task2
Tasks
Task10
Task14
Task18
Task3
Task7
Task11
Task15
Task19
Task4
Task8
Task12
6
trekk20
0
10
15
Number of Tasks
Figure A8: Evolution of task accuracy Split-minilmagenet
22

---

Published as a conference paper at ICLR 2022
B.8 COMPARISON To EXISTING METHODS
Coarse-CIFAR100
Split-CIFAR100
(2R0-10 10Be 003122
Avg. Accuracy (N)
62-
Avg. Accuracy (%)
62-
40-
30-
20
Number of Tasks
Number of Tasks
Figure A9: This figure compares Model Zoo to existing continual learning methods on the Coarse- CIFAR and Split-CIFAR 100 datasets with respect to average task accuracy. Model Zoo and its variants are in bold, similar to the left panel of Fig. (which is for popit-minilmagenet), Isolated small and Model Zoo-small significantly outperform existing methods. All methods in the figure are run in the single-epach setting.
B.9 ADDITIONAL CONTINULL LEARNING EXPERIMENTS ON 100 SAMPLES/LABEL
We also performed continual learning experiments with 100 samples/class in Table A4. We find that Model Zoo-continual obtains an accuracy that lies in between those of Isolated and the approximate upper bound given by Multi-Heat (multi-task learning). Note that have shown that matching or improving upon the performance of Isolated (which trains a model independently for each task) for continual learning is quite difficult because it necessitates effective forward-backwer so indicates strong ability of the learner for both forward and backward transfer. In some cases, the continual learner even outperforms Multi- trained on all tasks together. This table indicates that Model Zoo can be used continual learning and demonstrate nontrivial forward and backward transfer even with few samples from each class.
transfer. Doing
Dataset
Isolated
Multi-Head (multi-task)
Model Zoo-Continual
Rotated-MNISS
98.17 + 0.24
98.47 +: 0.18
98.44 = 0.17
Split-MNISH
97.11 + 1.21
99.47 0.08
98.98 = 0.51
Bermuted-MNBSS
84.59 +:2 1.65
86.36 1.15
86.04 1.68
Split-CIFARIO
82.09 +: 0.76
85.73 0.60
84.17 0.60
Split-CIFAR 80.04
87.93
86.27
Coarse-CIFAR100
65.34 + 0.41
69.05 + 0.38
66.80 = 6.27
Table A4: Average per-task accuracy (%) for continual learning at the end of all episodes using 100 samples/class. bootstrapped across 5 datasets (mean + std. dev.). Model Zoo-continual performs better than Isolated on all problems even if tasks are shown sequentially.
We next visualize the evolution of the per-task test accuracy for various datasets. This is qualitative way to investigate forward and backward transfer the learner. Forward transfer positive if the accuracy of newly introduced task in particular episode is higher than what it would be if the task were trained isolation. Backward transfer is positive if successive episodes and tasks result in an increase in the accuracy of tasks that were introduced earlier in continual learning. Both Appendix B.7 and Fig. A 110 consistently show non-trivial forward and backward transfer.
23

---

Published as a conference paper at ICLR 2022
Rot-MNIST
Split-MNIST
100
100
97.5
Accuracy (%0
Accuracy (96)
95
32.5
92.5
90
90-
Number of Tasks
Hormuted-MNIST
SpFRR-CIFAR10
100
1.00
Accuracy
90
Accuracy (96)
90
80
70
Number Tasks
of Tasks
PMNGST-11E AAMISTG F491212121 AMNITT aoi FRNGSSS
Figure A10: Per-task validation accuracy a function of the number of episodes of continual learning for problems using variants of CIFARIO and MNIST datasets using Model Zoo-continual. Each task has 100 samples/class. X-markers denote accuracy of Isolated the new task. We both forward transfer (Model Zoo often starts with higher accuracy than Isolated) and backward transfer (accuracy of some past tasks improves in later episodes). For problems like Permuted-MMN and Rotated-MNISST there is little forward or backward transfer.
B.10 MODEL Zoo WITH UNIFORM SAMPLING
At each round of boosting, Model Zoo samples tasks according to equation (8) i.e., tasks with high loss under the current ensemble have a higher probability of being selected in the next round. To study the importance of this heuristic, we compare Model Zoo to variant called Model Zoo (uniform). Model Zoo (uniform) samples uniformly over all seen tasks for each round, as opposed to using equation (8).
Table A5 compares the accuracy of Model Zoo and Model Zoo (uniform) on the Coarse-CIFAR 100 dataset. Model Zoo is marginally better than Model Zoo (uniform) indicating that using the training loss a cheap proxy for splitting the capacity amongst related tasks. At the same time, this also indicates that a better measure of task-distances can further improve performance.
Method
Avg. Accuracy
Model Zoo
84.27
Model Zoo (uniform)
83.60
Table A5: Comparison of accuracies on the Coarse-CIFAR100 dataset
C PROOFS
Proof of Theorem 2. From the definition of Pty relatedness for tasks, we have
cl/oa (h) > Ep, (h)
= Epp (h) Ep. (h, MT).
for any i,j < and h E H. Let us denote (()) P.1. We can over i {1...., k} and divide by
l to get
k 1/P0 (h).
1 L
En (h) < k i=1
En (1))) +
P
24

---

Published as a conference paper at ICLR 2022
The first term is a discrepancy term that measures how distinct different tasks are measured by the probability of the disagreement of their individual hypotheses (i() with that of hi under samples drawn from task P1. We need to bound the second term on the right-hand side to prove Theorem We have
k
k
1/p P(3)
(1)
-
h
P(s
Puoox (ht
k
(er, EP,, (11))) / Praxx
k
pmax (l)) S 2 1/pax (l)) ppp
Ep
where the final step involves Jensen's inequality and P 1/k: 1/k P(a. This is the population risk of hypothesis on the mixture distribution P and by uniform convergence, we can bound as Y.maa (h) S (g(h)) cc' ( log 8)
1/2 1/pmax
(D-log
km
for any h E H, in particular ht, with probability - 8. Putting it all together we have:
k
En (h) -
En (m())
c
P
1P (h)
k
==1
1=1
5 Epp (h(a) + C (
(
D -D-logh (D-logd 8)
1/2 1/Pmax
(s((h) +
D FREQUENTLY ASKED QUESTIONS (FAQs)
1. Why do you consider the setting with unlimited replay?
As mentioned in $6, we would like to ground the practice continual learning. Our investigation is inspired by the existing work on continual learning and with this paper we seek to encourage future works focus their investigations on key desiderata of continual learning, namely per-task accuracy and forward-backward transfer.
With this goal, we are motivated by our results in Theorem 2 that fitting a single model on a set of tasks is fundamentally limiting in performance due to competition between tasks, this problem is only exacerbated by introducing the tasks sequentially. We have developed a general method named Model Zoo that, although designed for unlimited replay, can be executed in any of the standard continual learning settings. Our experiments show that Model Zoo significantly outperforms existing methods in all of these settings, including problem settings with no replay.
We allow Model Zoo to revisit past data and grow its capacity iteratively in order to get the heart of the problem of learning multiple tasks sequentially. In our view, if we can demonstrate effective continual learning without forgetting at least in this setting, it will provide good foundation to build methods that conform to the stricter problem formulations.
We believe that such foundation is needed today if we are to advance the practice of continual learning. Let explain why with an example. The simplest "baseline" algorithm named Isolated in our work, surprisingly outperforms all existing continual learning methods, without performing any data replay, or leveraging data from multiple tasks. An upper bound for performance of a continual learner is the accuracy obtained by a multi-task learner that has access to all tasks before training, We argue that good continual learner's performance should lie in between the above two: it should be-at at comparabee to training the task in isolation, and as close to the performance of the multi-task learner as possible. The fact that existing methods perform much poorly than even Isolated indicates that we need to thoroughly investigate the tradeoffs that these methods make, e.g., while the single epoch setting helps mitigate forgetting, it has quite poor accuracy.
25

---

Published as a conference paper at ICLR 2022
In short, we would like to argue that before we design new sophisticated methods for continual learning, we should take a step back and evaluate what simple methods can do and ascertain some level of baseline performance, so that we have sound benchmark to compare the sophisticated method against. This is our rationale for considering the problem setting with unlimited replay. We would also like to emphasize that Model Zoo is a legitimate continual learner because it gets access to each task sequentially, and has fixed computational budget at each episode. For a multi-task learner, the computationa complexity scales with the number of tasks.
2. Why do you call it continual learning, instead of, say, incremental lifelong learning? The current literature is quite inconclusive about the formal distinction between continual, incremental and lifelong learning. We have chosen to call our problem "continual learning" and, by that, we simply mean that the learner gets access to tasks sequentially instead of having access to all tasks before training begins.
3. Why are you not using the same neural architectures as those in the existing literature? Perhaps the methods in this paper work better because you use larger/different neural architecture.
We use a small deep network (WRN-16-4 with 3.6M weights) for all our experiments. In particular, this is smaller than the Resnet- 12 or Resnet- 18 architectures that are used in a number of continual learning experiments (see Kaushik et al. (2021)) and the Model Zoo has a comparable number of weights. The exceptional performance of Model Zoo indicates that these observations indicate that the significant gains in accuracy of Model Zoo are not simply a result of using a larger model. We also demonstrate results on continual learning with a much smaller model, a CNN with 0. 12M weights (which entails that Model Zoo has about 2.42M weights). This is an extremely small model, and even this model, under all problem settings, improves the accuracy of continual learning over existing methods.
4. Why not compare Model Zoo to ensemble versions of other methods?
We compare the performance of Model Zoo with ensemble versions of Isolated in Fig. 4. We observe that Model Zoo performs better than an ensemble of Isolated models. We did not compare against ensemble variants of existing continual learning methods because as our results show in multiple places, Isolated significantly outperforms the state of the art as a continual learner. We therefore expect that Model Zoo will also outperform ensembles of existing methods.
5. Boosting is not novel.
We do not claim any novelty in developing boosting and moreover our method is only loosely inspired by The key property of Model Zoo that makes it effective the ability to split the capacity of the learner across different sets of tasks, the ones that are chosen at each round. This entails that the implementation of Model Zoo is similar to that of coosting-basee algorithms such as AdaBoost, but that is the extent of the similarity between the two. In particular, Model Zoo only uses the models that were trained on a particular task in order to make predictions for it. Unlike AdaBoost which combines all the weak-learners using specific weights, we simply average the predictions of all models trained on each task. To emphasize. boosting is not novel, but the ability of Model Zoo to split learning capacity across multiple models, one from each round, trained on a set of tasks, is novel.
6. Identifying that tasks compete is not novel. See $6 and the references in 12. The fact that tasks compete with each other is broadly appreciated-if not rigorously studied-in the theoretical machine learning literature. It is also appreciated broadly under the name of catastrophic forgetting in continual lcarning. Theorem 2 clucidates this competition and shows, together with Fig. 2, that it can be quite non-trivial, Even if some tasks compete, i.e., hypothesis that is optimal for one performs poorly on the other, they may benefit each other if we have access to lots of samples from each task. An effective way to resolve this competition has been missing. Model Zoo is simple and effective framework to tackle task competition; such mechanism, and certainly its use for continual learning, is novel to our knowledge.
7. Why does the rate of convergence in Theorem 2 depend upon Pmax, this seems quite inefficient.
The convergence rate in Theorem 2 which depends on Pmax indeed seems pessimistic one chooses bad set of tasks to train together. But this may be a fundamental limitation of non-adaptive methods, e.g., that pool data from all tasks together to compute ht If the
26

---

Published as a conference paper at ICLR 2022
learner uses adaptive methods, e.g., it has access to Pty and iteratively restricts the search space at iteration & to only consider hypotheses that achieve a low empirical risk és;,) on all tasks closer than P(k). then as (Hanneke and Kpotufe, 2020) shows, we can get better convergence rates if all tasks have the same optimal hypothesis. Let us note that we have chosen some drastic inequalities in Appendix C in order to elucidate the main point, and it may be possible to improve upon the rate.
8. Can you give some intuition for the transfer exponent?
The transfer exponent discussed in (5) is inspired by the work of Hanneke and Kpotufe (2020) and is defined by the smallest value such that
c. cEP, pis (h) 2 Ep, (h, = Er, (l)) (k, cr,(h])
for all h E H. This should be understood as measure of similarity between tasks that incorporates properties of the hypothesis space. A small value of Piş ~ suggests that minimizing the excess risk task P: (the left-hand side) is good strategy if we want to minimize the excess risk on task P3 (the right-hand side). But there may be instances when we can only reduce the left hand-side up to an additive term
ép, (h₂) ep, (h₂)
that may be non-zero (or large) the optimal hypotheses hj and h perform very differently on samples from Pj. Mathematically, Púj is seen as the rate of convergence the concentration term in Theorem 2 if samples from P. are used to select a hypothesis for Py; larger the transfer exponent, more inefficient these samples, even if this additive term is zero.
27


Reviews:
[{"review_id": "QjBJbmxMl_4", "reviewer": "Reviewer_Cn5V", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides both theoretical and empirical insights to support the claim that splitting the capacity of a model into several small learners can benefit continual learning, which have the potential to guide future continual learning research. While the empirical results to some extent corroborate the theory, the link between the outperformance of Model Zoo and the reasons predicted by the theory were not entirely obvious and could potentially be clarified with some additional experiments. Additionally, the experiments were independently lacking in some ways detailed below. I would consider increasing my score if these issues are largely addressed with revisions. \n\nComments:\n\n- The theoretical analysis appears accurate to me and it demonstrates that if tasks are sufficiently related, then training them together with a shared feature extractor can benefit improve generalisation, but that if they are not, it can harm generalisation. The benefits of a shared feature extractor among some tasks is reflected in the design of ModelZoo, which trains a shared feature extractor along with task-specific classifiers for a subset of tasks at each round. Furthermore, the empirical results that the ensemble of isolated learners outperforms many existing methods and that ModelZoo sometimes outperforms the multi-task model provide evidence to the theoretical result that claims that it is not always beneficial to train all tasks in a single model. What was not entirely clear was why selecting tasks with high empirical risk under the ensemble is a good heuristic for selecting a subset that are synergistic in the way described in the theoretical analysis. Could it not be that, to paraphrase Tolstoy, each unhappy task is unhappy in its own way? The intuition here is was not clear to me. Two possible ways to empirically demonstrate that this method selects tasks that are more likely to be synergistic are:\n    - Perform an analysis to show that groups of tasks selected by ModelZoo exhibit more transfer between each other than randomly selected groups of tasks.\n    - Train an ablation to ModelZoo where the tasks are uniformly selected for training, rather than biased by empirical risk, and compare performance to model zoo.\n- The experiments were lacking in a number of ways:\n    - It is claimed that this paper is the first to perform continual learning on coarse cifar-100 (according to super-classes) but this does not appear to be true (see [1,2,3]).\n    - In Table 1, no accuracy numbers are given for any of the competing methods on Split-CIFAR 10 nor for Coarse CIFAR-100, and only two methods are reported for Split MNIST, which is a very common benchmark in the literature. Split CIFAR-10, for example, was trained on in [7,8,9].\n    - In Table 2, for the multi-epoch runs, no accuracy or forward/backward transfer metrics are reported for the competing methods, so it is difficult to compare them in terms of computation and storage to the methods proposed in the paper.\n    - The definition of forward transfer is unconventional in that is simply the “accuracy on a new task when it is first seen”. Typically, as in [7], you would compare the accuracy on a task after having trained on each of the previous tasks compared to a baseline performance on a randomly initialised network.\n    - None of the competing methods are reimplemented - all results are taken from the original papers or other ones that have reimplemented them. This is okay but it makes it difficult to compare them directly to the methods in the paper, e.g. to compare inference times, the models should ideally be trained on the same hardware or if not the differences should be clarified. Also, it is not clear if the training is equal across approaches in terms of how much replay data is afforded to each of them.\n    - It would have been useful to empirically compare to at least one or two other continual learning methods that use ensembles (e.g. [4,5,6]) or at least to mention that other ensemble-based approaches to continual learning exist in the related work.\n\n[1] Yoon, Jaehong, et al. \"Scalable and Order-robust Continual Learning with Additive Parameter Decomposition.\" International Conference on Learning Representations. 2020.\n\n[2] Yoon, Jaehong, et al. \"Federated continual learning with weighted inter-client transfer.\" International Conference on Machine Learning. PMLR, 2021.\n\n[3] Shanahan, Murray, Christos Kaplanis, and Jovana Mitrović. \"Encoders and Ensembles for Task-Free Continual Learning.\" arXiv preprint arXiv:2105.13327 (2021).\n\n[4] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3366–3375, 2017.\n\n[5] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations (ICLR), 2020.\n\n[6] Germán Kruszewski, Ionut-Teodor Sorodoc, and Tomas Mikolov. Evaluating online continual learning with calm. ArXiv preprint arXiv:2004.03340, 2021.\n\n[7] Lopez-Paz David and Marc'Aurelio Ranzato. \"Gradient Episodic Memory for Continual Learning\". European Conference on Computer Vision. Advances in Neural Information Processing Systems. 2017.\n\n[8] Rebuffi Sylvestre-Alvise, Alexander Kolesnikov and Christoph H. Lampert. \"iCaRL: Incremental classifier and representation learning.\" arXiv preprint arXiv:1611.07725, 2016.\n\n[9] Zenke, Friedemann, Ben Poole, and Surya Ganguli. \"Continual learning through synaptic intelligence\". International Conference on Machine Learning. 2017.\n", "summary_of_the_paper": "This paper provides two main contributions on the topic of continual learning: (i) the first is to propose a definition of relatedness between tasks in the framework of statistical learning theory, and use it to perform a theoretical analysis of when it can be fruitful to train multiple tasks with the same model and when it can be detrimental to do so; (ii) the second is to propose an architecture for continual learning called ModelZoo, which maintains an ensemble of models that grows as each new task is introduced. Each time a new task arrives, a feature generator h and k task-specific classifiers g_k are initialised, and training proceeds on the combination of h and each g_k on the current task and data from k-1 previously seen tasks. At any point, the performance on a previously seen task can be evaluated by averaging over all small models previously trained on that task. ModelZoo takes inspiration from AdaBoost in how it selecting which previous tasks to train concurrently with the current one, which is by preferentially sampling tasks that have a bad performance under the current ensemble. An empirical analysis is performed by training on various standard continual learning image classification tasks, which demonstrate that Model Zoo is on par with and sometimes outperforms the single model multi-task approach (both , which is often used as an “upper bound” on continual learning methods. The results also demonstrate that a simple baseline of an ensemble of small isolated learners outperforms a selection of existing continual learning methods (with results reported reported from other papers).\n", "summary_of_the_review": "This paper provides both theoretical and empirical insights to support the claim that splitting the capacity of a model into several small learners can benefit continual learning, which have the potential to guide future continual learning research. While the empirical results to some extent corroborate the theory, the link between the outperformance of Model Zoo and the reasons predicted by the theory were not entirely obvious and could potentially be clarified with some additional experiments. Additionally, the experiments were independently lacking in some ways detailed below. I would consider increasing my score if these issues are largely addressed with revisions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "J2hB6hjOGUG", "reviewer": "Reviewer_R6VS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-\tThe proposed boosting-style algorithm for task-incremental learning is an original and important contribution to the continual learning literature.\n-\tThe surprisingly large gap in performance between the Isolated learner and the compared existing methods is intriguing.\n-\tThe theoretical analyses and considerations in Section 2 are interesting, and I think they provide important insights and food-for-thought for many continual learning researchers.\n\nAs stated above, I believe this paper makes important and insightful contributions, but there are several issues that should be addressed before this paper could be published at this conference:\n-\tThe paper suggests that it covers virtually all continual learning settings out there, but the paper ignores several important settings. Specifically, a critical assumption that this paper makes for all its experiments is that task identity is always provided to the learner (i.e. task-incremental learning). The paper does not consider settings where task identity is not provided (i.e. domain- and class-incremental learning; https://arxiv.org/abs/1904.07734). In a footnote it is argued that without the assumption of task identity being provided continual learning is impossible in the case of permutation of classes; but it is unclear why this argument is relevant as none of the experiments considered involve such a permutation of classes. It is important that it is made clear that the insights and observations of this paper are specific to the task-incremental learning setting.\n-\tAt several places, the paper makes claims about performing better than all existing methods, on all datasets and on all benchmarks. Based on the reported experiments, these claims are too strong (e.g. it is clear not all existing methods are compared against), and I also think that these claims are unlikely to be true (e.g. see below). These claims should therefore be moderated.\n-\tPerhaps most importantly, I have serious concerns about whether the comparisons against existing continual learning methods (e.g. in Fig 1A, Table 1) are done fairly. Firstly, for some results it is stated that they are taken from other publications. This seems problematic, as the mentioned publications use different architectures and optimization methods. It is therefore unclear whether the reported difference is due to these differences (which are arguable unrelated to continual learning), or due to the continual learning strategy. Secondly, I wonder how the other results (i.e. those for which it is not stated that they are taken from other publications) were obtained. I could not find any description of this in the paper, and also in the submitted code there does not seem to be any code for these other methods.\nI am particularly worried about this because from Fig 1A, it seems to be the case that the main part of the reported improvement over existing methods is due to an improved ability to learn individual task (e.g. compare all accuracies after just the first), rather than due to an improvement in forward/backward transfer. That is, I am afraid that the main improvements reported in this paper might be due to network architecture/optimization choices that are not interesting from a continual learning perspective.\n-\tAn important class of methods that is not considered in this paper is generative replay (Mocanu et al., 2016 https://arxiv.org/abs/1610.05555; Shin et al., 2017 https://arxiv.org/abs/1705.08690), which does not store data and could be interpreted as adhering to the “strict formulation” of continual learning. I think it is important to at least discuss these methods as I am positive that some more recent generative replay variants (e.g. van de Ven et al., 2020 https://www.nature.com/articles/s41467-020-17866-2; Liu et al., 2020 https://arxiv.org/abs/2004.09199) could outperform the Isolated baseline.\n-\tIt is claimed that the “Multi-Head (multi-task)” baseline is not a continual learner because it is jointly trained on all tasks. However, it seems straight-forward to adapt this baseline in such a way that it is, for the purposes of this paper, a “legitimate continual learner”: the tasks are visited sequentially as in the other baselines, and on each new task, the model is trained using the data of all tasks seen so far. (Or in other words, always all previous tasks are replayed fully.)\n-\tIt is unclear to me how Model Zoo with limited replay is performed, and it would be good to provide some more details. In the text it is stated that only a subset of the data is used in equation (7), but I think the authors probably meant to refer to another equation?\n-\tThe claim in the title that the proposed Model Zoo is a “Growing Brain” does not appear to be supported or justified anywhere in the text. I think it would be good to either modify the title, or to justify and discuss the reasons for this comparison in the text.\n", "summary_of_the_paper": "This paper argues and demonstrates that in a task-incremental continual learning setting, in many cases it is beneficial to split up the capacity of a learning algorithm and to learn separate models for different tasks. Building on this insight, and taking inspiration from the boosting literature, the paper then proposes the method Model Zoo, in which a new model is learned for each new task and whereby each new model is trained on multiple tasks. During evaluation, the predictions of all models trained on the task under consideration are averaged. The paper reports surprisingly large improvements over existing continual learning methods on several different task-incremental learning methods, both with and without utilizing replay.", "summary_of_the_review": "Although I believe this paper provides several important insights, there are several critical issues that prevent me from supporting acceptance of this paper. I am happy to engage in a discussion with the authors, to see whether my concerns can be taken away or whether the paper could be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IbftLPc6jq4", "reviewer": "Reviewer_LD4g", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am not an expert in continual learning, so there are details that I might have overlooked. Overall I think this is a strong paper.\n\nOne of the confusions I had as a reader: The authors talk a lot about task synergy and competition (e.g., sections 2.2 and 2.3, theorem 2, figure 3, etc.) However, in the end it seems that the method does not use task synergies/competitions at all to decide which tasks to train on. Instead, tasks are more likely to be chosen if they have high loss under the existing model (equation 8). I would appreciate a bit more discussion about this. Is the current method a proxy for selecting synergistic tasks? If not, how is the preceding discussion relevant?", "summary_of_the_paper": "This paper begins with a clear exposition of continual learning and an analysis from a statistical learning theory perspective.\n\nIt then introduces the Model Zoo: A sequence of small models that are each trained on a subset of the available tasks at that point (with a separate linear classifier for each task). The tasks selected for training are those that have high loss under the models trained so far (similar to boosting).\n\nThe following section contains an exhaustive set of experiments that compares the Model Zoo with a variety of baselines (with and without replay, with and without limited training, a single model with multiple heads, etc.). Surprisingly, the \"Isolated\" model (i.e., a separate model is trained on each task) does better than most baselines that train on multiple tasks. The Model Zoo outperforms all other baselines and even outperforms the multi-head baseline on ImageNet.\n\nThe paper ends with a critical discussion on how the problem of continual learning should be approached.", "summary_of_the_review": "I think this is an excellent paper. It is well-written, the proposed algorithm has strong results, it asks important and critical questions about continual learning research, it validates the proposed model with a large amount of experiments, is well-referenced, and provides source code and 14 pages of appendices to ensure reproducibility of the results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HRoWahcIku0", "reviewer": "Reviewer_KWuM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nThe paper makes a very good case for the difficulties of sharing parameters between different tasks, which are amplified by sequential data access restrictions, especially without task labels. However, the paper doesn’t address many such issues, and simply goes on to assume many of these limitations don’t exist. That is fine, as long as the setup is clear from the start, and the tone of the paper reflects the reality of results, i.e. a simpler problem can be solved better than much harder problems.\n\nThe results provided show that even in favorable settings, e.g. splits of i.i.d. datasets, parameter sharing may not be optimal, even if data access restrictions are confined only to future data in sequential learning. This is an important contribution, but may not support all the claims.\n\n\nCons:\n\nAccording to this paper, a continual learner’s only limitation is not knowing all data from all the tasks not yet seen; essentially, according to this definition, a continual learner’s only limitation is not knowing the future. Standard sequential learning settings used in the literature are borrowed and simplified by removing data access restrictions and giving full information about past tasks. Other implicit assumptions are made but not all are properly discussed: \n- Data imbalance issues are not mentioned explicitly. What is the meaning of learning both MNIST and ImageNet?\n- Task sequences considered are favorable to transfer and no attempt to study adversarial tasks is made;\n- It is assumed past data can be neatly sorted into i.i.d. buckets (separate tasks) with known labels;\n- Task losses aren’t always comparable in general, e.g. in terms of scale or learning dynamics; MNIST and CIFAR-10 have quite different error curves during learning; what is the meaning of the average joint objective?\nIn my humble opinion the paper should give a thorough discussion of limitations and appropriate qualifications for all claims.\n\nWriting and framing issues:\n\nThe paper simply redefines continual learning in contradiction to a majority of recent literature. For example, the first paragraph of the paper gives a non-standard definition of continual learning: a continual learner isn’t widely defined as an algorithm which “leverages past data to learn new tasks”. While proposing a new definition is very interesting and indeed a potentially important contribution, the presentation of this new definition is at best misleading, significantly reducing the accuracy of the writing.\n\nMany terms, e.g. “task relatedness”, task “competition for capacity”, “synergistic data” are offered without precise definitions or associated references. While difficulties in producing such precise definitions may be well understood, they don’t seem to temper the tone of the paper, nor do they seem to inspire the authors towards ideally a more humble, or at least balanced view of existing literature and the plurality of opinions within. The authors describe different viewpoints as an “intellectual gap” in research, which is not very helpful.\n\nThe authors appear to misattribute properties of the learning setting as qualities of the proposed algorithm, which I believe is done in error. The adopted CL setting offers data neatly split into labelled i.i.d. buckets. This does not mean that the data selection heuristic will always be effective, especially when these implicit assumptions are violated. However, the paper doesn't discuss such issues at length.\n\nWhile the comments about focusing on backward and forward transfer are well taken, it is important to note that the proposed solution heavily relies on the definition of multi-task learning given originally by Caruana (1994), i.e. learning from multiple tasks independently as best suits each individual task. There is no guarantee that the proposed algorithm achieves this, however. Empirical results provided do not give good enough reasons to believe so because all experiments rely on splits of single datasets into well balanced bins with similar data, e.g. preprocessed in the same way, similarly centered and scaled, etc. Because the dataset shift is quite low between splits of X, it is really hard to tell how effective the proposed method is in the general case under its own assumed (much more permissive) CL setting. Indeed, at the beginning of section 3 the authors express hope that interference doesn’t happen, but offer no formal guarantee and do not take on an empirical challenge where the algorithm could demonstrate performance under less than ideal conditions.\n\nIt seems that the choice of how much data to use from tasks with higher loss is left as a hyper-parameter, and hence depends strongly on the data stream. If one can expect little interference, then many sources can be combined without much hindrance, as the paper observes; but this is not necessarily true in general. In fact, such hyper-parameters express the worst case expectations, which in general can be very negative. The paper does not discuss this, or report such results.\n", "summary_of_the_paper": "This paper proposes yet another definition of continual learning (CL) and uses it to motivate a divide-and-rule approach. The algorithm exploits all assumptions of the adopted CL setting, including the relaxation of data access rules, in order to perform better compared to algorithms designed for more restrictive settings. The paper concludes that previous CL settings were holding back progress. Empirical evaluations and ablations are given using the new CL setting proposed in this paper, with the help of datasets favorable to this approach, namely splits of existing i.i.d. datasets.", "summary_of_the_review": "This could be a great paper. It is important to understand what works in practice with current algorithms and architectures. Analysis of worst case performance is not the only important case in many practical settings. Hence, I lean towards acceptance despite the gross lack of nuance and generally imprecise writing. As I said, the authors could turn this paper into a great contribution by providing nuance instead of hyperbole, deference to the audacity of other researchers to face challenges which the authors avoid axiomatically, instead of off hand comments. I will (reluctantly) not oppose publication close to current form, but I cannot confidently recommend acceptance before the writing is improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper sparked a very substantial discussion, not just about its scientific content, but also, as the authors will have seen, from the narrative standpoint. I would like to thank the reviewers who devoted time and efforts to discuss the paper’s content. I encourage the authors to polish further their paper prior to camera ready, using the numerous scientific comments made (e.g. R6VS, KWuM, the refs of Cn5V). 

AC.
Decision: Accept (Poster)