A Probabilistic Framework for Modular Continual Learning
Authors: Lazar Valkov
Akash Srivastava
Swarat Chaudhuri
Charles Sutton
Venue: ICLR 2024 poster (2024)
Keywords: continual learning
modular machine learning
modular continual learning
transfer learning
catastrophic forgetting
Bayesian optimization
probabilistic modelling
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We use probabilistic modelling over pre-trained modules and the data to speed up modular continual learning.

Abstract: Modular approaches that use a different composition of modules for each problem are a promising direction in continual learning (CL). However, searching through the large, discrete space of module compositions is challenging, especially because evaluating a composition’s performance requires a round of neural network training. We address this challenge through a modular CL framework, PICLE, that uses a probabilistic model to cheaply compute the fitness of each composition, allowing PICLE to achieve both perceptual, few-shot and latent transfer. The model combines prior knowledge about good module compositions with dataset-specific information. We evaluate PICLE using two benchmark suites designed to assess different desiderata of CL techniques. Comparing to a wide range of approaches, we show that PICLE is the first modular CL algorithm to achieve perceptual, few-shot and latent transfer while scaling well to large search spaces, outperforming previous state-of-the-art modular CL approaches on long problem sequences.

Published as a conference paper at ICLR 2024
A PROBABILISTIC FRAMEWORK FOR MODULAR CONTINUAL LEARNING
Lazar Valkov'; Akash Srivastava', Swarat Chaudhuri', Charles Sutton MIT-IBM Watson AI Lab, UT Austin, 'University of Edinburgh
ABSTRACT
Modular approaches that use different composition of modules for each problem are promising direction in continual learning (CL). However, searching through the large, discrete space of module compositions is challenging, especially be- cause evaluating a composition's 's performance requires a round of neural network training. We address this challenge through a modular CL. framework, PICLE, that uses a probabilistic model to cheaply compute the fitness of cach composition, al- lowing PICLE to achieve both perceptual, few-shot and latent transfer. The model combines prior knowledge about good module compositions with dataset-specific information. We evaluate PICLE using two benchmark suites designed to assess different desiderata of CL techniques. Comparing we show that PICLE is the first modular CL algorithm to achieve perceptual, few- shot and latent transfer while scaling well to large search spaces, outperforming previous state- of-the-art art modular CL approaches on long problem sequences.
wide range of approaches,
INTRODUCTION
Continual learning (CL) (Thrun & Mitchell 1995) demands algorithms that can solve a sequence of learning problems while performing better on every successive problem. A CL algorithm should avoid catastrophic forgetting, i.e., not allow later problems overwrite knowledge learned from earlier ones, and achieve transfer across a large sequence of problems. It should also be plastic, in the sense of being able solve new problems, and exhibit backward transfer, meaning performance on previously encountered problems should increase after solving new ones. Ideally, the algorithm should be capable of transfer across similar input distributions (perceptual transfer), dissimilar input distributions and different input spaces (transfer of latent concepts, i.c., latent transfer), and to problems with few training examples (few-shot transfer).
Recent work (Vakkov et al.) 2018 Veniat et al. 2020 Ostapenko al. 2021) has shown modular algorithms to be promising approach to CL. These methods represent a neural network as a compo- sition of reusable modules. During learning, the algorithms accumulate library of diverse modules by solving the encountered problems in a sequence. Given a new problem, they seek to find the best composition of pre-trained and new modules as measured by the performance
held-out dataset.
Modular methods can achieve plasticity using fresh modules, while using pre-trained modules to avoid catastrophic forgetting. However, scalability remains key challenge in these methods, as the module "library" grows linearly in the number of problems, the set of all module compositions grows polynomially the size of the library, and hence exponentially in the number of problems. Further, the evaluation of the quality of a composition requires the expensive training of its modules.
In this paper, we present modular CL framework, called PICLI that mitigates this challenge through a probabilistic search (Shahriari et al.) (2015). The central insight is that searching over module compositions would be substantially cheaper if we could compute a composition's ''s fitness, proportional to its final performance, without training the new modules in it. Accordingly, PICLE uses probabilistic model to approximate the fitness of each composition.
The benefit of such probabilistic model is that it combines prior knowledge about good module compositions with problem-specife information in a principled way. We introduce two different
"valkovl@mir.cduu
PICLEE pronounced "pickle", stands for Probabilistic, L.brary-based Continual Learing.

---

Published as a conference paper at ICLR 2024
Past solutions:
Set of all paths:
m
112
mi2
mi]
mi]
mi
mi
mi
mi
mg
mn]
mi
m
UPT
Uieer
INNT
Figure 1: The set of all paths that a modular algorithm considers when solving the 4th problem in sequence. The modular architecture has 2 layers. The shaded modules are re-used from previous problems. The library comprises all pre-trained modules: C = {m] m2, m3 mž). Paths Uer (Section 4 select pre-trained module for the first layer, enabling perceptual transfer. Paths in HPT reuse modules in both layers. They can perform few-shot transfer since they only require few examples (to select the correct path). Paths in KT (Section 5) achieve latent transfer by reusing a module in the second layer, allowing applications to new input domains.
probabilistic models, specialized for different types of transfer: one for perceptual and few-shot transfer that is focused on modelling the distribution over input activations to each module (Section 4); one for latent transfer which is based on Gaussian process capture the idea that similar compositions of modules should have similar performance (Section 5). We make use of both models create scalable modular CL algorithm with constant training requirements, i.e., it evaluates constant number of compositions and trains networks with fixed size for each problem.
We evaluate PICLE on the popular CTrL benchmark suite (Veniat al., 2020), as well as new extension of CTrL, which we call BELL, which introduces compositional tasks. Comparing to wide range approaches, our experiments demonstrate that PICLE the first modular CL algo- rithm to achieve perceptual, few-shot and latent transfer while scaling well to large search spaces, outperforming previous state-of-the-art modular CL approaches on long problem sequences.
2 BACKGROUND
A CL algorithm aims to solve sequence of problems, usually provided one at time. We consider the supervised setting, in which each problem is characterized by training set (X", Y")) and validation set (Xval, Y The algorithm's ' goal is to transfer knowledge between the problems. It considered scalable both its memory and computational requirements scale sub-linearly with the number of solved problems (Veniat et al. 2020).
Modular approaches to CL (Valkov al., 2018 Veniat et al. 2020; Ostapenko et al. 202T repre- sent a deep neural network composition of modules, each of which is parameterized nonlinear transformation. Transfer learning is accomplished by reusing modules from previous learning prob- lems. Each composition is represented by a path: sequence of modules function composition m o o m² me. We refer to each element of the path as layer. Af- ter solving / problems, modular algorithms accumulate library frozen previously trained modules. We denote the set of all pre-trained modules in the library for layer as L..
(m')) that defines
Given a new learning problem, one can construct different modular neural networks by searching over the set of paths (Fig. [ For each layer i, path selects either a pre-trained module from the library L or introduces a new one m that needs to be trained from scratch. In order to find the best path, modular algorithm assesses different paths by training the resulting neural networks and evaluating their final validation performance on the validation set, denoted as f. Thus, evaluating each path costly, and furthermore, the number of paths grows rapidly with the size of the library.
A PROBABILISTIC FRAMEWORK FOR MODULAR CL
PICLE is based on a probabilistic search over paths. For each path, we seek to compute, without training the path, proxy for its validation accuracy. We compute this proxy, called fimess, posterior distribution over the choice pre-trained modules specified by a path, given the problem's data. In doing so, leverage both prior and groblem-speciff knowledge.
We distinguish between two types of paths that lead to different types of transfer and require dif- ferent probabilistic models: (i) Paths that enable perceptual transfer (PT) between similar input
2

---

Published as a conference paper at ICLR 2024
distributions, by using pre-existing modules in the first layers and new modules elsewhere (Sec- tion 4); and (ii) paths that allow latent transfer or non nor-percep transfer (NT) between problems with different input distributions different imput spaces, using pre-existing modules in the last 1 layers and new modules before them. The two kinds of paths are illustrated in Fig.
For PT paths, designing a probabilistic model is easier, on data from the new problem. For NT paths, this is impossible because the initial modules in the network are untrained. Hence, define different probabilistic model, which employs distance in function space across compositions of pre-trained modules.
evaluate the pre-trained modules
Algorithm 1: PICLE
L O, - 0// Empty library, no solutions. for each problem t with data X", Y", X Y do
ASA (M ... Muion // Evaluat Mh.nn JSA TRAINANDEVALUMMII SAA X", Y x Y
fully randomly initialized network.
TPT: JPT <- FINDBESTPTPATHIC2 X", Y X Y
ANT JNT f FINDBESTNTPATHCC π X", Y X Y
f TJ where arg maxy y NT) fg/ // select the best performing path UPDATELLIBRARY(C, +"); APPEND(TT", TT)
Our overall framework is shown in Algorithm 1 For each new learning problem, PICLE first eval- uates a fully randomly initialized, i.c. a standalone (SA), model in case no pre-trained module is useful. Afterwards, it searches for the best path, combining the PT and NT search strategies. Finally, the UPDATELIBRARY function adds modules from the best path to the library for future problems.
4 SCALABLE PERCEPTUAL TRANSFER AND FEW-SHOT TRANSFER
Our method searches through paths, which achieve percep- tual transfer (PT paths) by reusing modules in the first layers (Fig. [T]. This includes cases where all layers are reused, po- tentially combining modules from different problems. Such path can serformfew-shee transfer.
m!
m
m²
h
h
The insight here to select each pre-trained module so that its distribution over inputs on the new problem is similar to the input distribution it was trained on, minimizing distribu- tion shift. Accordingly, we define probabilistic model over the choice pre-trained modules and their inputs, and intro- duce search strategy that searches for the PT path with the highest probability.
Figure Our probabilistic model for PT path with three pre-trained modules, m m2 m2 and their re- spective inputs h and h?
The set of PT paths with prefix-length e is the Cartesian prod-
uet of library items L for the first E {1,...,() layers and new modules in the other layers: Hhr L ... L {m ... (m }. This set grows polynomially with the size of the library, making naive search strategy inapplicable.
Instead, we introduce a strategy based on a generative model of the input x, the choice of pre- trained modules m ...m', , and the latent activations h²...h h after each pretrained module, i.e., hi+1 mi (h,). We imagine sampling the highest- level activation h first, and then sequen- tially sampling each lower-level activation h satisfy the constraint h/+1 m This process is illustrated by the graphical model in Figure 2 That is, we model the joint distribution as
R--2 Dh .m p())pp/kk)
p(m ... m. h- h h--1)= =-x))=p((h">> p(x(hh,.nm
p(h (mm²)) Ip(m')
This allows to infer the distribution pre- treined modules, given the observed inputs. Because hi+1 mi+1 (h"), we can marginalize out the activations and express the posterior distribution over pre-trained modules using terms compute, namely, a prior p(m*) over pre-trained modules
3

---

Published as a conference paper at ICLR 2024
Algorithm 2: FIND Searching through perceptual- transfer paths
Input: A library L of pre-trained modules.
Input: The training data (X", Y") and validation data (x* Y for the new problem.
π < -(..f (); evaluated paths and their final validation performances
m - ()// selected pre-trained modules
for l + to do
m + arg max mEct P (m @ (m) x")
π + m (m') @ (i Muiew) // @ denotes sequence concatenation TRAINANDEVALUATTEO, X", Y", x y
APPEND(T, R); APPEND(F, APPEND(m,
i arg max,, f(i')
return (ii) f[i] returning best path and its final validation performance
and an approximation of the input distribution p(h (mm)) for each pre-trained module:
p(m' ..... m² x) x p(m', .... m. .x) p(x|mm)
p (m²)
Io(m'). (2)
mec, p(h 110-11m)pe (m)
We consider that pre-trained module's competence can be measured by the accuracy it achieved on the problem it was originally trained to solve. Accordingly, our prior p(m') uses softmax to order modules by their original accuracy (see Appendix Eb. To approximate the training input distribution p(h" of m', we first choose a low-rank Gaussian distribution. We estimate its parameters using the set of hidden activations h hi-1_ af-1...h" training. We use random projection (Johnson 1984 reduce each hidden activation's 's dimension- ality to & < v. Next, we approximate the distribution in the www-dimesioonl space by a multivariate Gaussian, by computing the sample mean and covariance of the projected data samples. This greatly reduces the number of parameters required to approximate the resulting distribution from v + v² to k2 Surprisingly, our ablation experiments (Appendix 1b suggest that this approximation also leads to a more reliable search algorithm.
I. that were provided as inputs to m during its
Search Strategy and Scalablity. Our overall algorithm (Algorithm 2) uses a greedy search find a PT path of prefix-length L {1...4} L} with the highest posterior probability p(m1 my X"). At each iteration f, the algorithm extends the current prefix with a library module m. E L which leads to the highest posterior probability. This leads to evaluating one path for each prefi--lnggth (1.... L}. and we return the single path with best validation performance. Importantly, our greedy search strategy scales well in the number of problems: it trains L networks each of which contains L modules, so the training requirements are constant in the size of the library.
5 SCALABLE LATENT TRANSFER
In the latent transfer setting, the search space is the set of latent transfer (NT) paths (Fig. D. These are paths where the first L modules are randomly initialised and the final & modules, referred to as the pre-trained suffix, are selected from the library. The set of all NT paths of suffix-length & is: LL. which grows exponentially in l.
IK (m X (m/_))) C1-
For efficient search, we need model that captures how good an NT path is at predicting the correct outputs after training. This is difficult as the distribution over inputs for the pre-trained modules de- pends on the earlier modules that we wish to avoid training. We overcome this challenge by defining a model that directly approximates an NT path's generalization performance after training. We note that the final f layers of an NT path define function from input activations to output activations. This allows us to employ a similarity metric between NT paths in function space. Specifically, we Gaussian process based on the intuition that two paths that define similar functions likely to perform similarly when stacked with new random modules and fine-tuned on a new dataset.
Our model defines distribution over the pre-trained modules of NT path. For NT path क E H& we specify a prior over its pre-trained modules p(m!--ft ...., ..., and approxi- mate the final validation performance that using these modules would achieve after training:
4

---

Published as a conference paper at ICLR 2024
pl (Y X m -
...., m)) Consequently, express the posterior over NT path's pre-
trained modules given the validation dataset as:
p(m4-6+1 m² X Y & p(Y m m---+11 m ",..mmm)))(mtttt
m²)
(3)
Our prior states that pre-trained modules in the path must have all been used together solve previous problem. Therefore, p(m"-~+1 ..... m²) x 1 if and only if the modules m m-++1 ..... m are a suffix of previous solution characterised by some path x². This prior, used by Veniat et al. (2020) for perceptual transfer, reflects our assumption that using novel combination of modules for latent transfer is unnecessary for our sequences.
To approximate the final validation performance of NT path p( Y [ Y m=--+11,- ..... m²), first note that
m²)) since the validation data is fixed for given problem, and denote it as f() p P X A). Next, approximate by putting a Gaussian Process (GP) prior with kernel x on it: f(A) ~ GP(O, n(A, X)). Here, X denotes another pre-trained suffix of the same length. To define a kernel function K, note that the pre-trained modules for an NT path, m m---+1 m. compute a function m m--i+11 o o... o m² We hypothesise that if two NT paths' pre-trained suffixes compute similar funetions, then their final validation performances will be also be similar. Accordingly, we capture this by using the squared exponential kernel function x(A, X' o² exp dd(, X) )2+"), where d the distance between two functions and and are the kernel hyperparameters which are fit to maximize the marginal likelihood of GP's training data (Rasmussen & Williams (006).
function of the pre-trained suffix - (m---(1
We compute the Euclidean distance between two functions (Appendix F)) which we approximate using Monte Carlo integration with set of inputs from the functions' common input space. For pre-trained suffixes of length f, we store few hidden activations from the input distribution of each pre-trained module at the first pre-trained layer L l + Given a new problem, we create a set of function inputs by combining all the stored hidden activations. For other kernels, see Appendix
Algorithm 3: FINDBESINTPATH: Searching through thenntttransfer paths
Input: A library L of pre-trained modules.
Input: A list of paths x² of solutions to previous problems.
Input: The training data (X", Y") and validation data (X Yv for the task.
x < (. (); evaluated paths and their val validation performance
// Perform Bayesian Opt. for NT' paths with Eni pre-trained modules:
// Compute all suffixes of length bnin from previous solutions, n.
+ (T'( mmin + 1: :::]]]] [.] for x²)) X + /) evaluated suffixes
// Find the most relevant previcus solution x'
for iter 4- 1 to min(c, len(x")) do
arg arg (per(er(f)) (f f))
(m ..ny Maew L- frie
@ X // denotes sequence concatenation TRAINANDEVALUATERN X", Y" x" Y
APPEND(T, π. T); APPEND(X A); APPEND(f.
arg max, fli]
10 n' + the path in x²" that has X x'(L Lmin + +1:1111
// Find the suffix of n' that has the best transfer:
11 for - L Emin to 2 do (miow ..... f=1 ① [e:
TRAINANDEVALLUTERN X", Y" X X"",," y
14 APPEND(T, :); APPEND(f, f)
15 i + arg max, f((')
16 return (1), f(i) // return best path and its final validation performance
Search Strategy and Scalability. We can use our probabilistic model to define a scalable search strategy (Algorithm 3) over a set of NT paths. Due to prior, we only consider NT paths that have the same suffix as a previous solution. Denote by π the set of previous solutions, i.e., the paths that were used to solve previous learning problems. Our method first searches for the previous solution n' E π that is most relevant to the current learning task, using Bayesian optimization. Then we search over the suffixes of x' in particular, and choose exactly how many of its modules reuse.
5

---

Published as a conference paper at ICLR 2024
Lines of our algorithm use our probabilistic model to search for the most relevant previous solution n.. We assess the relevance of a solution by trying to transfer its last émin modules, and choosing the solution which leads to the best performance. The value min is hyperparameter, which we choose to be the minimum number of pre-trained modules needed to obtain increased generalisation performance. We evaluate up to e paths, where is hyperparameter. At each step, we have a sequence of already evaluated NT paths æ,, their pre-trained suffixes X' and their final validation performance f. We use a GP and combine its predictive distribution PGp with the Upper Confidence Bound (UCB) Srinivas et al. (2009) acquisition function, to predict each unevaluated path's final validation performance. This search results in the most relevant previous solution n'. Finally, in lines
1 we evaluate NT paths created by transferring a different number of the last layers of nT' to see if re-using more layers leads to further improvement.
Overall, this search through NT paths requires training and evaluating up to constant number of paths, concretely up to e+ I. min a constant number of L modules, allowing it to scale well with the number of problems.
min 1, the size of the library, with each path specifying a with
EXPERIMENTS
We compare algorithm against several competitive modular CL baselines: RS, which randomly selects from the set of all paths (shown to be a competitive baseline in high-dimensional search
spaces and in neural architecture search (Li & Talwalkar| 2020);; HOUDINI (Valkov et al.. 2018).
with a fixed neural architecture to keep the results comparable; MNTDP-D (Veniat et al. 2020
which scalable modular CL algorithm for perceptual transfer; LMC (Ostapenko al. 2021
which can achieve different transfer properties but has limited scalability. For completeness, we also compare to a standalone (SA) baseline which always trains a new network for every problem; a CL algorithm based on parameter regularisation, online EWC (O-EWC) (Schwarz et al. 2018 Chaudhry et al. 2018); and one based on experience replay, ER (Chaudhry et al. 2019)..
Our hyperparameters are listed in Appendix G For each haseline, we assess the performance on held-out test dataset. To measure transfer learning performance, we report amount of forward trans- fer on the last problem. Tr- computed as the difference in accuracy, compared to the standalone baseline (see Eq. 5 We also report the average forgetting F across all sequences, measured by the difference between the accuracy achieved on each problem the end of the sequence and ini- tially (see Eq. (). Additionally, for completeness, we report the average accuracy of the final model across all problems, A (see Eq. 4 this averages over problems earlier in the sequence, for which no transfer is possible by design, so it is less helpful for comparing transfer learning approaches.
Compositional Benchmarks. The CTrL benchmark (Veniat et al. 2020) has been the most popular for evaluating recent modular CL approaches (Veniat et al. 2020 Ostapenko et al. 202T). It defines sequences of learning problems, each sequence being designed to evaluate a particular CL desider- atum. We introduce new, compositional extension of CTrL, which we call BELL. Each problem in BELL is a binary classification task on a pair of images, which composes an image classification task with pattern recognition task; for example, "do both images depict number less than 5" This allows us to sample a larger variety of random problems by choosing different classification and pattern recognition tasks. Most sequences contain problems, except for sequence of length 60 that tests scalability. Our neural architecture has 8 modules, which leads to a large search space (e.g., for the 6th problem, there are ((6)) possible paths). Details on the problems, sequences, neu- ral architecture and training procedure can be found in Appendix H For each sequence, we create 3 versions by sampling different classification tasks, and averaging the metrics over these versions.
Results. Overall, PICLE outperforms the other methods, achieving, average across the short sequences, +2.7 higher final accuracy and +9.02 higher transfer on the last problem, compared to the second-best bestt algorithm MNTDP-D (Tables and 2). PICLE and MNTDP-D achieve similar performances on sequences evaluating their shared CL properties (perceptual transfer, plasticity, stability). but PICLE also demonstrates few-shot transfer and latent transfer. LMC performed worse than standalone on these sequences despite our efforts to adjust it to this setting, so we do not report it. Forgetting impacts O-EWC and ER's 's final accuracy but is avoided by modular algorithms.
First, we evaluate the methods' perceptual transfer capabilities via sequences S S Sour* Sour* Sour* These are listed in increasing order of difficulty; the last problem in the sequence has an input

---

Published as a conference paper at ICLR 2024
Table 1; Results on the compositional benchmarks from BELL which assess perceptual transfer (S S gour, ", few-shot transfer (Ste*), plasticity (SP)) and backward transfer (S*+).
O-EWC
ER
HOUDINT
MNTDP-D
S
75,47
50.92
63.31
78.14
80.82
82.18
88 12
Sru
74.25
56.98
60.58
76.16
74.40
77.95
78.15
S
72.27
56.72
59.5
73.39
72.27
75.48
75.72
A
S
71.51
35.74
59.27
73.85
71.75
73.71
75.73
93.61
58.87
64.27
93.63
93.61
93.72
93.79
S 73.88
58.07
63.75
76.67
79.59
81.67
81 92
S+ 73.61
59.96
67.34
75.08
73.61
74.49
76.37
56.75
62.86
78.13
78.01
79,89
81.13
Avg.
-17.99
0.
0.
S
5.87
4.54
11.42
16
S
1.57
-13.27
5.64
15.41
15.41
S
0.
0.29
-7.91
0.43
0.
12.53
12.53
T
STM
().
4,21
-8.1
4.61
1.46
1.74
12.04
0.
-22.34
-1.09
0.
0.
00.20
00.20
S
0.
4.8
8.92
17.22
34.27
34.29
34.29
-2.63
-0.57
0.
0.
Avg.
-1.4
-2.93
4.82
5.75
10.8
17.22
Table 2: Results on compositional benchmarks from BELL which assess latent transfer.
SA
O.EWC
ER
HOUDINI
MNTDP-D
PT-only
NT-only
PICLE
5
89.01
57.78
62.68
90.85
89.32
90.62
90.26
92.20
92.82
5
87.94
60.04
64.97
92.22
87.94
87.92
91.92
88.48
58.91
63.83
91.54
91.16
89.28
89.09
92.06
92
Avg.
-24.55
-25.73
0.
0.
S
-0.77
181
11.04
7.61
22
Tt
S
0.
-2.59
-3.56
25.68
30.27
23.65
23.65
Avg.
0.
-1.68
1.31
13.75
20.66
4.85
3,805
21.27
22
domain similar to one of the previous problems but is represented by a low number of data points.
We consider the amount of transfer achieved in the last problem, Tr of Table [ HOUDINI fails to scale to the large search space, only achieving transfer on the easiest sequence S As expected, we find that MNTDP-D and PICLE are equally capable of perceptual transfer and both perform significantly better than other methods. However, Sum=* find that PICLE achieves +10.33 higher transfer, made possible by the performance-basee prior we defined in Section PT paths.
Second, sew-shot transfer is evaluated by the Stew sequence (Table in which the last problem is represented by few examples and shares an input domain of one of the past problems, and the two-dimensional pattern of another. This requires knowledge. HOUDINI demonstrates low amount of transfer due to the large search space. MNTDP- D selects the correct modules process the input domain but is unable compose them with modules from another solution, leading to sub-optimal performance, Finally, PICLE successfully achieves few-shot transfer, attaining 34.65 higher transfer (I'r (Sfee)) than MNTDP-D.
algorithm recompose previously acquired
Third, latent transfer is evaluated by two sequences (Table 2): Sin and S In both, the last problem shares the two-dimensional pattern of the first problem, but has a new image input domain (S or new input space (8 MNTDP-D does not achieve latent transfer: it transfers some perceptual knowledge on Sin and fails on S HOUDINI performs similarly to MNTDP-D the largest amount of latent transfer on S9P, In S the different input space necessitates different modular architecture for the first 5 modules, resulting
Sin but achieves
a much smaller search space, O(63 216). This allows non-scalable approaches, namely RS and HOUDINI, also be effective on this sequence. Finally, PICLE demonstrates latent transfer on both sequences, outperforming MNTDP-D and transferring 14.67 (Tr (Tr-I(Sii) and +23.65 ( compared to the PTT-only ablation.
An algorithm's plasticity, iee ability to acquire new knowledge, is evaluated by SPl in which all problems are different. All modular algorithms demonstrate plasticity, they always introduce a new set of modules, achieving similar final accuracies A(ST)) (Table I). The last short sequence S+ evaluates the backward transfer by having the last problem be the same the first one but with
7

---

Published as a conference paper at ICLR 2024
Table 3: Results on CTrL sequences which assess perceptual transfer (ST. ) plasticity (SR)), backward transfer (S+) and latent transfer 5 Standard deviations appear in Appendix K
SA
O-EWC
FR
RS
HOUDINI
LMC
MNTDP-D
PT-only
NT-only
PICLE
Sa
65.43
24.49
46.4
65.41
64.59
69.
66.23
66.86
68.18
68.34
5
63.11
46.93
56.56
63.77
66.54
65.72
66.78
66.78
66.78
St1
63,97
43.64
58.82
63.72
63,89
62.31
64.56
61.67
64.65 67
S-
62.96
48.3
54.87
64.07
67.16
65.77
67.24
67.24 24
67 24
S+ 63.04
45.93
56.8
63.15
62.83
59.68
63.11
62.99
62.99
Avg. 63.70
41.86
54,69
64.02
65.
64.59
65.58
65.51
66.00
F
Avg.
0.
-20.65
-4.78
0.
0.
-0.66
0.
0.
U.
0.
0.
3.79
41.13
-0.53
-5.03
17.41
-0.63
1.36
16.26
16:26
S
6.75
2,88
2.64
17.
16.79
17.06
17.06
17.06
-h-1
5p1
-1388
0.
6.7
2.42
4.09
20.74
20.74
20.74
20.74
S+
0.
-14.57
-7:52
0.
-3.0
0.
0.
Avg.
0.
-2.24
-9.97
1.24
6.63
7.76
7.43
7,83
10.81
FLOP% (T)
Momory Roquired (MB)
2000
3500
9
OFEIC
3000
500
(a) A logarithmic scale of the number of FLOPs required for fixed number
(b) The maximum memory required
by cach algorithm for each problem.
updates, per problem (in trillions)
Figure 3: Resource requirements for CTrL's glong
a larger dataset. All modular algorithms showed inability to attain backward transfer, due to their pre-trained modules being frozen, thus, achieving similar final accuracies A(Stow).
Finally, to measure scalability, we define glong which contains 60 randomly-selected compositional problems. We compare to MNTDP-D as it was the only other method able to do well in the large search space of the short sequences. PT-only achieved +7.37 higher average accuracy than the standalone baseline demonstrating its ability to achieve perceptual transfer on a long sequence of problems. MNTDP-D D achieved +8.83 higher average accuracy than SA which confirmed its scal- ability. PICLE performed the best, attaining +12.25 higher average accuracy than SA. This shows that PICLE successfully attain perceptual and latent transfer across long problem sequence.
CTrL benchmarks. The CTrL benchmark suite defines fewer sequences to evaluate different CL. properties. Namely, they specify Spi, St. S Suml, SuP which are defined similarly to ours. In contrast, the sequences are over multi-class classification tasks of coloured images from different domains. They also use different modular architecture based on ResNet18 (He al. 2016), which is more complex than the architecture used in the previous benchmarks. Our experimental setup, detailed in Appendix K, mirrors the one used in Ostapenko al.] (202I), except that we are averaging over three random seeds instead of single one, all using the same data. Each network composes 5 modules, leading to a smaller search space on the short sequences - O(65).
Our results (Table 3) show similar performance of the different modular CL. algorithms on the short sequences, with PICLE achieving the highest average performance. The smaller search space alle- viates HOUDINI'S and LMC's scalability issues. As result, PICLE, HOUDINI, MNTDP-D and LMC demonstrate perceptual transfer (S-, S plasticity (SP') and stability (S*). However, la- tent transfer (Sii) ) is exhibited by PICLE, HOUDINI and LMC, but not by MNTDP-D. Overall, the demonstrated properties are consistent with the ones observed on the compositional benchmarks. We also find PICLE to be effective on more complex input domains and neural architectures.
8

---

Published as a conference paper at ICLR 2024
CTrL also specifies Slomy, which has 100 problems. Each problem is constructed by first sampling one of 6 possible datasets (including CIFARI00 (Krizhevsky et al. 2009)), ) then 5 classes from that dataset at random. The 100 tasks represent a varying degree of relk-relassdness and distribution shifts, making it representative of the typical CL benchmarks where one introduces new tasks (task- This more
classes (nasemiinremettal)) challenging sequence is designed to evaluate an algorithm's 's scalability, On it, PICLE achieved the highest average accuracy (69.65), (6).. compared SA (55.04), MNTDP-D (65.64) and LMC (64.499), This demonstrates PICLE's effectiveness on large search spaces. Moreover, Figure 3 plots the com- putational and memory demands of modular CL algorithms on this sequence. It can be seen that LMC's s memory and HOUDINI's computational requirements scale poorly with the number of prob- lems, while MNTDP-D and PICLE's requirements grow slowly with the number of solved problems.
incremental);
new domains (domain-incrrmente)
These results are summarised in Table MNTDP-D and LMC are closest to PICIE in performance, but compared to MNTDP-D, PICLE is capable of latent and few-shot transfer and compared to LMC,, PICLE has superior scalability. Our results demonstrate that PICLE is the first modular CL algorithm to achieve perceptual, few-shot and latent transfer while scaling to long problem sequences.
7 RELATED WORK
This work considers the task-aware, data- incremental (De Lange et al., 2021 super- vised setting of continual learning. Other settings involve overlapping problems (Farquhar & Gall 2018) or reinforcement learning (Khetarpal et al. 2020). Our CL
Table 4:
Observed CL properties of modular CL ap-
proaches.
RS
HOUDINI
LMC
MNTDP-D
PICLE
Plasticity
V
desiderata is derived from Valkov et al. (2018) and Veniat al. (2020). Other lists Schwwrz et al. 2018 Hadsell et al. 2020
Stability
Perceptual
/
Few-shot Tr
x
1
Delange et al. (222]) do not distinguish be-
tween different types of forward transfer.
Latent It
/
x
Continual learning methods can be cate- gorized into ones based on regularisation, replay, or a dynamic architecture (Parisi ctt al. 2019). The first two share the same parameters across all problems, which limits their capac- ity and, in turn, their plasticity (Kirkpatrick et al. 2017). Dynamic architecture methods can share different parameters by learning problem-specifie specific parameter masks (Mallya & Lazebnik 2018) or adding more parameters (Rusu et al. 2016). This category includes modular approaches that share and introduce new modules, allowing groups of parameters to be trained and always reused together. Modular approaches mainly differ in their search space and their search strategy. PathNet (Fernando al. 2017) uses evolutionary search search through paths that combine up modules per layer. Rajasegaran et al, (2019) use random search on the set of all paths. HOUDINI (Valkov et al. 2018)) uses type-guided exhaustive search on the set of all possible modular architectures and all paths. This method can attain the three types of forward transfer, but does not scale to large search spaces.
Scalability
X
x
X
V
/
Backward Tr
x
x
MNTDP-D Veeniat et al., 2020) is a scalable approach which uses a k-NN model to find the closest previous solution, and then restricts its search space to perceptual transfer paths derived from that solution. Similarly to our search through PT paths (Algorithm 2). MNTDP-D evaluates only T. + paths per problem, however, the search space does not include novel combinations of pre- praitrd modules which prevents the method from achieving few-shot transfer, in addition to being unable to achieve latent transfer. In contrast, our approach can achieve all three types of forward transfer.
While LMC (Ostaennko et al. 202I) also approximates each module's input distribution, PICLE'S PT uses orders of magnitude fewer parameters and unites the approximations within a probabilistic model, allowing us to define a prior over the choice of modules and incorporate additional assump- tions in principled way. For each layer, LMC computes linear combination of the outputs of all available pre-trained modules, thus, the trained model grows linearly with the number of solved problems, preventing LMC from scaling well. Finally, LMC's performance deteriorates when ap- plied to long problem sequences. See Appendix C for comparison to works in the field of AutoML.

---

Published as a conference paper at ICLR 2024
Reproducibility Statement. We have taken multiple steps to ensure the reproducibility of the presented results. With respect to our claims, we provide derivation the posterior distribution used for PT paths (Eq. 2 in Appendix [ Moreover, concretely specify the prior which for PT paths in Appendix F. along with the derivation we used to select its hyperparameter.
With respect to experiments, used for both our algorithm and the baselines in Appendix [ The compositional benchmarks and the neural architecture used for them described in Appendix [ Similarly, the CTrL benchmarks and the neural architecture used for them are described in Appendix K
detail our experimental setup, along with all hyperparameters
Finally, PICLE'S source code is available hhttps://gitee hwwpa//aooo/aaaaaaaaaaaoo//aoaaa
This research was partially supported by The NSF National AI Institute for Foundations of Machine Learning (IFML), ARO award #W911NF-21-1-000) and DARPA award #HRO01 12320018.
Acknowledgements.
REFERENCES
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference computer vision (ECCV), pp. 532-547, 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint 1rXiv: 2019.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3606-3613, 2014.
Tarin Clanuwat, Mikel Bober- Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:18/2.017/8/ 2018.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. 2017 International Joint Conference on Neural Networks (IJCNN,, 2017. doi: 10. 10.1 11111099cmn.22177766661. 2017.7966217..
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-338 2021.
Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Norman R Draper and Harry Smith. Applied regression analysis, volume 326. John Wiley & Sons, 1998.
Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, al. Neural architecture search: survey, J. Mach. Learn. Res., 20(55):1-21, 2019.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint 1rXiv:1805.09733, 2018
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint (rXIvv1701.088 2017.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning, pp. 1920-1930, PMLR, 2019.
GPy. GPy: A gaussian process framework in python.
ht
wwm/Sheffielddl
GPy since 2012.
10

---

Published as a conference paper at ICLR 2024
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends cognitive sciences, 24(12):1028-100 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-art. Knowledge- Based Systems, 212:10662: 2021.
William B Johnson. Extensions of lipschitz mappings into a hilbert space. Contemp. Math., 26: 189-206, 1984.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric Xing. Neural architecture search with bayesian optimisation and optimal transport. arXiv preprint UrXiv:/802.001 2018.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforce- ment learning: A review and perspectives. arXiv preprint arXiv:2012.13490, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint orXiv:1412.6998 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barnskkkaaa al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)3521-3356 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2322 1998.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: hutp://yyann lew.n. www.u....o//ex///mistt.t. 2. 2010.
Joseph Lee Rodgers and W Alan Nicewander. Thirteen ways to look at the correlation coefficient.
The American Statistician, 42(1):59-66, 1988.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in Artificial Intelligence, pp. 367-377. PMLR, 2020.
Romy Lorenz, Ricardo P Monti, Ines R Violante, Aldo A Faisal, Christoforos Anagnostopoulos, Robert Leech, and Giovanni Montana. Stopping criteria for boosting automatic experimental
design using real-time fmri with bayesian optimization. arXiv preprint arXiv:1511.07827. 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
Anastasia Makarova, Huibin Shen, Valerio Perrone, Aaron Klein, Jean Baptiste Faddoul, Andreas Krause, Matthias Seeger, and Cedric Archambeau. Automatic termination for hyperparameter optimization. In First Conference Automated Machine Learning (Main Track), 2022.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Seyed Iman Mirzadeh and Hassan Ghasemzadeh. Cl-gym: Full-featured pytorch library for con- tinual learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 3621-3627, June 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011. 1.

---

Published as a conference paper at ICLR 2024
Vu Nguyen, Sunil Gupta, Santu Rana, Cheng Li, and Svetha Venkatesh. Regret for expected im- provement over the best-observed value and stopping condition. In Asian conference on machine learning, pp. 10922799999 294. PMLR, 2017.
Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. Advances in Neural Information Processing Systems, 34:30298- 30312, 2021.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buee E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8033 8035. Curran URL
Associates, Inc.,
an
Fabian Pedregosa, Gaäl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-leam: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Jathushan Rajasegaran, Munawar Hayat, Salman Khan, Fahad Shahbaz Khan, and Ling Shao. Ran- dom path selection for incremental learning, Advances in Neural Information Processing Systems, 3, 2019.
Carl Edward Rasmussen and Christopher KI Williams, Gaussian processes for machine learning. 2006.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatriek, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.cm 04671, 2016.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Tch, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. In International Conference on Machine Learning, pp. 4528-4537. PMLR, 2018.
Yee Whye
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 2015.
Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian pro- cess optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995, 2009.
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems, 15(1-2):25 46, 1995.
Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini: Lifelong learning as program synthesis. Advances in Neural Information Processing Systems, 31, 2018.
Tom Veniat. Mntdp.
2021.
Tom Veniat and Marc Aurelio Ranzato. Continual transfer learning benchmark.
https:// //
2021.
Tom Veniat, Ludovic Denoyer, and Marc' Aurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. arXiv preprint arXiv:2012. /2631, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Pashion-mmisse a novel image dataset for benchmark- ing machine learning algorithms, 2017.
12

---

Published as a conference paper at ICLR 2024
Tong Yu and Hong Zhu. Slyper-paramee optimization: A review of algorithms and applications. arXiv preprint arXiv:2003.05683) 2020.
A LIMITATIONS
Our method's limitations reveal directions for future work. First, PICLE does not search over all possible paths. e.g., paths which transfer both the initial and final layers, and learning new modules in between, could accomplish perceptual and latent transfer simultancously. This can be readily addressed within our framework by employing suitable probabilistic model. Second, the prior used when searching over PT paths relies on accuracy. making it difficult to be applied to problem sequences with different tasks, e.g. classification and regression. Third, the prior we used when searching over NT paths precludes from finding novel pre-trained suffixes, which could be ad- dressed by assigning non-zero probability to suffixes which are not derived from previous solutions.
B EXPERIMENTAL MEASUREMENTS
We use the following three measurements to assess the performance of each continual learning (CL) algorithm. First, we compute the average accuracy A across all problems after the last problem solved. An algorithm's average accuracy after is trained on some problem sequence S
1V,, 2...... w(s) is computed as:
A(S)
1 S
Σ
A S1 (v,)
where V, denotes the 1-th problem and (V,,) denotes model's accuracy on this problem after all problems in the sequence have been solved. Second, we compute the forward transfer on the last problem only, Tr -1, for sequences in which the performance on the last problem diagnoses an CL property. We compute it as the difference between the final accuracy on the last problem by CL algorithm, and the accuracy of a standalone baseline, ASa:
Tr- = A [s)) ASA ( (()))
(5)
Finally, we compute the average forgetting experienced by a CL algorithm, as the average difference between the accuracy it achieves problem at the end of the sequence, and the accuracy it achieved on this problem initially:
1s F(S)
-V.
- A(VV))
(6)
S
1=
C RELATED WORK AUTOML
The work presented in this paper automates the process of choosing the best path. As such, the setting is similar to that AutoML. (He et al. (021) in which the algorithms aim to automate dif- ferent aspects of the process of applying machine learning to a problem. In particular, is similar to hyperparameter optimisation (HPO) (Yu & Zhu (020) and neural architecture search (NAS) (Elsken et al. 2019). HPO aims optimise different hyperparameters related the train- ing procedure (e.g. learning rate, choice of optimizer, magnitude of regularisation) and the neural architecture (c.g. number hidden layers, number of hidden units, choice of activation functions). Categorical hyperpamaters are typically encoded as a one-of-k embedding. However, in our setting, embedding all of the selected modules would result in an embedding of size ( which would make the sample complexity of our surrogate model prohibitively high. NAS is sub-setting of HPO which specialises to searching through more elaborate neural architectures. For this purpose, NAS methods also study how to best featurize neural architecture. However, the approaches ex- pressed in literature are specific to the respective neural architectures and aren't applicable to our setting. Despite the differences, there are also similarities between previous work in HPO and NAS, and our approach. For instance, most HPO and NAS approaches make use a surrogate func- tion. Moreover, is common use Bayesian optimisation with Gaussian processes (GP)
setting
13

---

Published as a conference paper at ICLR 2024
m
m
m
h'
h2
Figure 4: Our probabilistic model for PT path with three pre-trained modules, m m', m and their respective inputs x, h' and h²
surrogate function in HPO. Kandasamy et al. (2018) use Bayesian optimisation with GP for NAS. Similarly our work, they define their own kernel function to compute the similarity between two neural architectures. The idea of early stopping with Bayesian optimisation has also been explored before. For this purpose, Nguyen et al. (2017) use El, [orenz et al. (2015) use the probability of improvement (PI) and Makarova et al. (2022) use measure based on the lower confidence bound Makarova et al. (2022) compare all approaches and show that while using El and PI has a higher chance of stopping before finding optimal solution, they also evaluate significantly fewer different configurations.
D PT PATHS POSTERIOR DERIVATION
In this section we show how we approximate the posterior distribution p(m',..., m Xi,... xx). To ease the presentation, and without loss of generality, we the number of pre-trained modules I = 3. The graphical model which captures the joint distribution is presented Figure 4 Next, we express the joint distribution in terms of quantities which we can approximate.
p(m', m2 m2 x, h h²) = ((mm))p(m?)) )(mm)) ((x(h") m )p(h Ih?, m m2 )(h (m3)
Here, p(x(h', m²) can be expressed as:
p(x[h², m²) p(x, h- m²) p(h² x, m (p(x(m=))((n) p(h² (p)))
p(h² |x, m m²) x mmmot)
Em p(ha Im² )m(m²'
Moreover, (hh m²) can be expressed as:
phh2(h2, p(h², h12,,m, m²² p(h? m² p() (m²m p(h?,m m²)
p(h²lh2 m mz²)phh |mm
p(h p(h?)p(m²))
m3 Em v p(h2lm3 )p(m3)
Therefore, the joint distribution can be expressed as:
p(m² m², m² X1, h2t h²2)
p(h² |x, m ) p(h²lh+ m²) mm?)
= =p(m²))((m)))
En,' p(h" )p(²') Emo p(h² mm3 )p(m))))m))) (mh2lm3))
Here, p(h' xx m²) and p(h2jh', m²) define a distribution over the values of hidden activation, given the module which produced it and said module's 's imput. However, this hidden activation value is given by deterministic transformation. h' m (h- 11. Therefore, we can model them using the Dirac delta function, 8: p(h'(h'-1, mm²) 5(h m mh This function has the property
14

---

Published as a conference paper at ICLR 2024
that 5 f(2)6(2 c)dz f(), which next in order to simplify the posterior. We write: p(m²; m², m x) x p(m", mmmm", m², x)
= =ʃ p(m m² m x, h² h² Jah" dh"
/ plmm ((m"))(m")))) )pmm) mm
p(h" x, mmmm)))(( (((x(m*))
p(h² [h" m²))((h" (m²)
Ema' p(h" mm² )p(m²)) Emsr p(h² mm3 )p(m3)) p(h² (m²)) dh?
/ p(m"))((m )p(m
p(x(m²)
p(h² |h', m² )p(h Im²)
mm p(h² (121 |m2 pp(m²)) Emor p(h² Im pmmm)) p(h" dn
(('(m²²) Em p(h" Im? )p(mm") Em p(h² mm }p(m3) p(h? mm²)
p(x(m=))
- -p(m' (mm))p(m" (p(m²)
(8)
where h' and h² are the hidden activations obtained by processing the given x with the selected modules mn and m
E DEFINING THE PRIOR FOR PT PATHS
define a prior distribution over the choice of pre-trained module, p(m'). Assume that two modules m and mj are trained using two different paths on two different problems. Also assume that the model trained on problem V achieved A (Va.) validation accu- racy after training, while the model trained on problem , achieved a higher validation accuracy ( = a))) + A, for A O. We hypothesise that the module, whose model achieved the higher accuracy after training, is likely to compute transformation of its input which is more likely be useful for other problems. Therefore, if m and mi, have similar likelihood for a given set of training data points, we would like give preference using m/. To this end, define the prior distribution in terms of module's 's original accuracy using the softmax function as follows:
Computing Eq. 2 requires
(AP(A(VV)))) p(m))= Emi,ec EC: ( (4)))
Here T' is the temperature hyperparameter which we compute as follows. Suppose that, for a given set of imput x, we have selected the first i - modules and have computed the imput to the ith module hi-!_ Moreover, suppose that the likelihood of module m is & times higher than the likelihood of mi, i.e. that p(h - [mm]) (mm,) = Ep(h* - |m]]) However, because the model of mý was trained to higher accuracy, we would like give equal preference to m and mi. Therefore, we would like to set the hyperparameter T so that the posterior of the path using m and the posterior of the path using mi are equal. Using Eq, 2 we can express this as:
p(m² ...., m p(m² ...., m²(x)
p(m))p(h''
(mm)
p(mm))((h'-1mm)
p(m²))
p(h² mm3}
p(m(t)
2(h
mm)
cxp(AA(W../IT
p(h
m
(A + A)TT}
p(h-
lmm)
(10)
(aa) (²(Wa)
logp(h" 1 (mm)) - log logg(hh 1h" (
T
-A
T
log p (h'
(1mm) log p(h (hiil|mm) mmm))
A
T
We can then use Eq. in order to determine the value of To do this, we need to decide how much difference in log likelihood should advantage in accuracy compensate for.
15

---

Published as a conference paper at ICLR 2024
F FUNCTION DISTANCE FOR NT PATHS
The Gaussian Process which we defined in Section 5 relies distance between two functions. For this purpose, we make use of the standard Euclidean distance in function space. Let the inner product between ʃ S - IK" and g 52 + R be (J g) [ns(() f()) g(z)dz. This allows to define the distance between two functions as:
9) - 1- (1()) g(4)) (f(z) o(z))dz.
G EXPERIMENTAL SETUP
In our experiments, we make the training process deterministic, so that the difference in perfor- mance can be accredited only to the LML. algorithm, and not due to randomness introduced during training. For this purpose, we fix the random initialisation of new parameters to be problem and path-specific. In other words, for given problem, if a model with the same path is instantiated twice, it will have the same initial values for its new randomly initialised parameters. Moreover, fix the sequence of randomly selected mini batches seen during training to be the same for given problem. Finally, as we use PyTorch for our experiments, we fix the random seed and use the command "torch.use deterministic (lgoorthhms(Tree)) The overall results is that, for a given prob- lem and given library, evaluating the same path will always result in the same performance, even across different modular LML. algorithms.
All experiments are implemented using PyTorch 1.11.0 (Paszke et al. 2019). We also use GPy's since Since 2012) implementation of a Gaussian process. We run each LML algorithm on a single sequence, separate GPU. All experiments are on single machine with two Tesla P100 GPUs with 16 GB VRAM, 64-core CPU the following model: "Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz", and 377 GB RAM.
G.1 ALGORITHM HYPERPARAMETERS
We now provide the implementation details and hyperparameters for each of the CL. algorithms, which we evaluate on the benchmark suite with compositional tasks (BELL) and the CTrL. bench- mark suite.
PICLE When searching through PT paths, we use a prior with softmax temperature (T 0.001) for BELL and (T'
for CTrL. When approximating module's input dis- tribution, we project its inputs to = 20 dimensions. When searching through NT paths, we use GPy's GPy (since 2012) GP implementation. We combine its prediction using UCB Srinivas et al. 2009 with 3 = We start from Imin = and at the end of the problem store 40 of the training inputs the (L- 2)th layer. We the number of paths evaluated during the Bayesian optimisation portion of our NT search, to = I Imin, which is constant in the number of solved problems.
MNTDP-D Veniat et al. (220)) When selecting the closest previous solution, use the 5- nearest-neighbours and the KNN classifier provided by sklearn PPedregoaa et al. (011).
LMC Ostapenko et al. 2021 We use the implementation provided by the authors We the task-award version of the algorithm with otherwise the hyperparameters provided by the authors for the CTrL sequence.
HOUDINI Valkov et al. (0018) We fixed the choice of modular neural architecture and use ex- haustive search through the of all paths. We evaluate 2L where / is the number of solved problems, letting its computational requirements to scale linearly with the number of solved prob- lems.
Accessed httpp///gginab.coooolloo////C
16

---

Published as a conference paper at ICLR 2024
RS Random search randomly selects paths from the set of all paths for evaluation. However, many of the selected paths can consist only of pre-trained modules, which are cheap to evaluate since we parameters. To keep the results comparable, instead limit the amount of time which can be taken by RS to solve a single problem to (2T, T)TSA where TSA is the amount of time it takes to train randomly initialised network.
don't ion't need to train
O-EWC (Schwarz al. 2018: Chaudhry et al., 2018) We use the implementation provided by CL-Gym Mirzadeh & Ghasemzadeh (2021 with À 1000 (following Ostapenko et al. (2021) and 256 samples to approximate the diagonal of the Fisher information matrix.
ER (Chaudhry et al. 2019) For experience replay, we use a Ring Buffer with 15 examples per class (following Veniat et al. (020)) for the multi-class classification tasks in CTrL and 40 for the binary classification compositional tasks in BELL. In this way, ER stores at least as many example per problem as PICLE.
H BELL- BENCHMARKS FOR LIFELONG LEARNING
We identify the following CI. desiderata Valkov et al. 2018): Veniat et al. (2020)
1. Stability - The CL. algorithm should he able to "remember" previous problems. As the al- gorithm solves new problems, its generalisation performance on previous problems should not drop drastically, i.c. no catastrophic forgetting.
2. Plasticity The CL algorithm should be able to solve new problems. The algorithm's performance on a new problem should not be worse than that of a standalone baseline.
3. Forward transfer The CL algorithm should be able to reuse previously obtained knowl- new problem should be greater than that of
edge. As result, its performance standalone baseline, whenever possible.
(a) Perceptual Transfer The CL algorithm should be able transfer knowledge across problems with similar input domains.
(b) Latent Transfer- The CL algorithm should be able to transfer knowledge across prob- lems with dissimilar input domains, including disparate input distributions or different input spaces.
(c) Few-shot Transfer The CL algorithm should be able to solve new problems, repre- sented only by a few training data points, if the new problem can be solved using the already accumulated knowledge.
4. Backward Transfer - The CL. algorithm should be able to use its newly obtained knowledge improve its performance previous problems. As result, its performance on a previous problem should be greater after learning a new problem, than the initial performance on the previous problem, whenever possible.
5. Scalability - The CL algorithm should be applicable to large number of problems. There- fore, the memory (RAM) and computational requirements should scale sub-linearly with the number of problems.
We now introduce BELL- a suite of benchmarks for evaluating the aforementioned CL propertics. We assume compositional tasks and then generate various continual learning sequences, each of which evaluates one or two of the desired properties. Running CL algorithm on all sequences then allows to asses which properties are present and which missing. This builds upon the CTrL. benchmark suite (Veniat et al. 2020), which defines different sequences of image classification tasks, namely Sel 5 Stoul Stm S and Slung They evaluate plasticity, perceptual transfer, latent transfer, catastrophic forgetting, backward transfer and scalability. We define these sequences simi- larly but for problems with compositional tasks. This allows to introduce new sequences which evaluate new CL properties (S and Slow). We also introduce new more challenging sequences (Sour" and ".
"Acesssd hhtp:///hhhb..oom//iiiaaaccc..Co
17

---

Published as a conference paper at ICLR 2024
S
Sequence Pattern
CL
SPi
[
2 V
VI v V.
1,, 2.
S Sou
V 22 V
W
1. 12
v
3.
[T V V
3/ V
(D1, h1 96
3.a
Sowa
[ 4. = (121
3: 44 v
4 6
(D1,
3.a
5
S Sowr
V 12 = (D1.11.92) 3. V 5
4
3.a
Stn
V 22 V3 V7 V. V (Do, 1o:hh::0
01
3.b
Stp
V V 33 V 11, 43 6 (Do, ho
91)
3.b
Slew
V W
5 v (D1, (1.94), V. 4
(22 94)l
12,
3.c
S+
V 2 3: 7 V VT]
4.
5long
[ 1
160
5.
Table 5: A list of all of the different CL sequences in BELL, each of which evaluates different CL. properties. The first column contains the sequence's name, the second shows the sequence's s pattern and the third column lists the CL properties evaluated by this sequence.
We assume compositional tasks and represent each problem as a triple v. = (D,. hy. 9k) where D, the distribution of the imputs and h2 and 9k constitute the labelling function, i.c. g h is used to label each input. We refer to hy as the lower labelling sub-function and to Is as the upper labelling sub-function. We use the indices j and lo to indicate whether the corresponding labelling sub-function has occurred before in the sequence (j k < t) of is new and randomly selected (i = f, k = t). For brevity, jf = and k = which means that both labelling sub-functions are new, don't write out the whole triple but only Ve. By repeating previously labelling sub-functions, we can control what knowledge can be transferred in each of the define sequences. In turn, this allows us to evaluate different CL properties. We use t indicate that the dataset generated for this problem is sufficient to learn a well generalising approximation without transferring knowlede snowledge. On the other hand, V indicates that the CL. algorithm cannot achieve good generalisation on this problem without transferring knowledge. Finally, & indicates that the generated training dataset consists of only a few datapoints, e.g. 10.
A complete list of the different sequences in BELL is presented in Table 5 Following Veniat el al. (2020), we set the sequence length of most sequences to 6 which, as we show in the experiments sec- tion. is sufficient for evaluating different CL properties. Next, we separately present each sequence, detailing which CL properties it evaluates.
Plasticity and Stability: The sequence Sa = [ V . Vi Vel consists of 6 distinct problems, each of which has a different input domain and a different task. Moreover, each of the generated datasets has a sufficient number of data points as not to necessitate transfer. Therefore, this sequence evaluates a CL algorithm's 's ability to learn distinct problems, i.e. its plasticity (1.). Moreover, this sequence can be used to evaluate algorithm's 's stability (2.) by assessing its perfor- mance after training on all problems and checking for forgetting.
Forward Transfer: Most of our sequences are dedicated to evaluate different types of forward transfer. To begin with, in the sequence S = [vt, V V2 VT V,, VT] the first and the last datasets represent the same problem, however, the last dataset has fewer data points. Therefore, CL algorithm would need to transfer the knowledge acquired from solving the first problem, thus, demonstrating its ability to perform overall forward transfer (3.).
Perceptual Forward Transfer: We introduce three different sequences for evaluating perceptual transfer (3.a). First, in Sou | Vn V VT V5 V (D1, hy, h1.96)) the last problem has the same input domain and processsn target function h as in problem 1. However, the last problem's dataset small, therefore, a CL algorithm needs to perform perceptual trans- fer from the first problem, which is described by large dataset. Second, Somr = [V V (D1, hu, 92), V. V V Ve (Di, ()))] shares the same input distributions and lower la- belling sub-function hi across problems I. V2 and Ve. Therefore, a CL algorithm needs to decide whether to transfer knowledge obtained from the first or from the second problem. Third, the se- quence So ( / = (D1, h1,92), TT VT V V is similar to the preceding one, with the distinction that the last problem is the same as the first. In this sequence, CL algorithm needs to decide between reusing knowledge acquired from solving the same problem (L,) )), or to transfer perceptual knowledge from more different problem (V2). Overall, these three sequences are de-
18

---

Published as a conference paper at ICLR 2024
signed to be increasingly more challenging in order to distinguish between different CL algorithms which are capable of perceptual transfer to a different extent.
Latent Forward Transfer: Currently, we define two sequences assess algorithm's 's ability to transfer latent knowledge. Firstly, in 5 = [ V2 , V3 VI. W, V = (Ds. her 91) the last problem has the same upper labelling sub-function as the first problem. However, the two problems' input distributions and lower labelling cub-ftnttions are different. Therefore, a CL al- gorithm would need to transfer knowledge across different input domains. Secondly, the sequence S = [ 2 V3 Va V5 D (Ds. has 91)] is simiarly defined, however, the input distribu- tion of the last problem also defined on a different input space from the input space of the first problem. Therefore, an algorithm would need to transfer knowledge across different input spaces. Few-shot Forward Transfer: In order to evaluate this property, we introduce the following se- quence, in which the first two problems are different from the rest of the sequences: SYew = [V]
(D1, h1), J (D2, h2). V V (D1, A1, 94), V5 V 6 (D2, h2, 94)]. The labelling func- tions of the first two problems are simpler, each consisting only of a lower labelling sub-function. This is done in order to provide CL algorithm with more supervision on how to approximate h, and 91 more accurately. The fourth problem v in this sequence then shares the same input domain and lower lableling sub-function as the first problem, but introduces a new upper labelling fub-function g4. The last problem then shares the input domain and the lower labelling sub-function of V2, while also sharing the upper labelling sub-function of problem 44 Moreover, the last problem's 's training dataset consists of only a few data points. Therefore, a CL. algorithm would need to reuse its ap- proximations of h2 and 94 in a novel manner in order to solve the last problem.
WT]]] has the same first and last problem. However, the first dataset has significantly less data points than the last. Ideally, CL algorithm should use the knowledge acquired after solving the last problem in order to improve its performance on the first problem. While this sequence represents a starting point for evaluating backward transfer, it is possible to introduce other sequences, representing more elaborate evalua- tions. For instance, introducing sequences which evaluate perceptual and latent backward transfer separately. However, backward transfer is not the focus of this paper, this is left for future work. Scalability: This property can be evaluated using a long sequence of problems. For this purpose we define glong 1 which consists of 60 problems, each randomly selected with replacement from set of problems. Most problems are represented by small dataset. V,-, Each of the first 50 problems has 1 3 probability of being represented by large dataset, V Each problem also has 10 probability of being represented by an extra small dataset, V
Backward Transfer: The sequence S+ = [ V2 , VT, 45
1
Next, we present a set of problems which can be used together with the aforementioned sequence definitions in order to evaluate CL. algorithms.
H.1 COMPOSITIONAL PROBLEMS
To implement the sequences defined above, one needs to define a set of compositional problems. To this end, we define 9 different pairs of an input domain and lower labelling sub-function, {(D,, h-) Moreover, we define 16 different upper labelling sub-functions 16 be combined into total of 144 different compositional problems.
These can
First, we define 9 image multi-class classification tasks, which all share input and output spaces 228x28 R$, but each have different input distribution D; and domain-specific labelling function hrs Concretely, we start with the following image classification datasets: MNIST (Le-) Cun et al.] 2010). Fashion MNIST (FMNIST) (Xiao et al., 2017). EMNIST (Cohen al. 2017) and Kuzushiji49 (KMNIST) (Clanuwat et al. 2018). Since some of the classes in KMNIST have significantly fewer training data points, we only use the 33 classes with the following indices: [0, 1,2,4 12, 15, 17: 21, 24 28, 30, 34, 35, 37 : 41, 46, 47], as they have a sufficient number of associated data points. We split the image datasets into smaller 8-class classification datasets. We use an index i to denote the different splits of the same original dataset. For instance EMNIST, represents the third split of EMNIST, corresponding to a classification task among the letters form 'i' to 'p'. As another example, MNIST, represents the only split of MNIST, corresponding to classification tasks among the digits from 0 to 7. Using this, we end up with the following 9 image datasets: MNIST, ,, FMNIST, (EMNIST, 1 1= { J For each of these image datasets, we set aside 4800 validation images from the training dataset. We also keep the provided test images separate.
19

---

Published as a conference paper at ICLR 2024
XOR
CIRCLE
X
SQUARE
H
12345678
12345678 12345678 12345678
Figure 5: An illustration of the four two-dimensional patterns which are used by the four g(2) fune- tions to label the input coordinates. Green indicates a positive label, and red indicates a negative label.
Second, we define a set of binary classification tasks, which map R + {0, 1}. Each task's la- belling function Ii receives two concatenated 8-dimensional one-hot encodings and returns binary value, indicating if the given combination of classes, represented by the input, fulfils a certain criteria. We further decompose the labelling function into gi (x) 9k
( [g) (xx: 8)). (x((8 (x((8 :)))-
Here, 9, maps one-hot encoding to integer between 1 and 8. For instance, of maps the first dimension to 1, the second to and so on. As result, we use g to convert the initial input two one-hot encodings to two-dimensional coordinates. We define 4 different g0)) mappings, where gf defined as above, and gi 99 and gi each map the dimensions to a different randomly selected integer between and 8. At the same time, each (122) RR2 + (0, 1} outputs whether a given two-dimensional coordinate is a part of certain pattern or not. We define different g( functions, each corresponding to one of two-dimensional patterns, shown in Fig 5 In total, these functions need label 8 * 64 different two-dimensional coordinates.
We fuse the 4 different g(2) functions with the 4 different (22) functions to define 16 different functions:
(9(k-1)-4t5(x) = =94 (0,"
(1) (x( [x[: 8)), 9j (1 (x(8 :)),, (1, (,,,,,,4)
S
Figure 6: example input for (DMNIST, IMNIST, 9 (xXoR= g ())). These images are classified by hMNIST and then mapped to the coordinates (1, 6) by 9i (1) since they represent the first and sixth classes respectively. Afterwards, gxoR labels this input as 0, using the XOR pattern, shown in 5
Finally, we can combine our 9 image classification datasets {(D;, h,)) with our 16 binary clas- sification tasks, in order to create 144 compositional problems ( (k-1)++++ = (Dx, hi, 9j), k E (1.....)) 9).j (1..... 16]}. J. The input to problem V = (De, h& 9;) are two images sampled from D,. Each image is labelled by h4 each resulting an eight-dimense dimensioil one-hot encoding of the corresponding image's 's class. The two one-hot encodings are then concatenated and labelled by Jj.
which results a binary label. An example for v (DMNISt, MINNIST, = (5xoR 9 ("))) is shown in Fig
Sequence SOP involves transferring across input spaces by having its last problem's 's input domain be defined over different input space. To create this domain flatten any randomly selected domain from 1228 28 to 7884 This loses the images' spacial information and requires that a different neural architecture is applied to process those inputs.
20

---

Published as a conference paper at ICLR 2024
H.2 REALISING THE SEQUENCES
To implement a sequence S of length 1, we need to select concrete compositional problems which fit the pattern specified by said sequence. Let the sequence have (()) different pairs of image domain and lower labelling sub-function, and ((2) different upper labelling sub-functions. For all sequences, apart from glone, we select ((1) pairs of (D;; h4) by sampling from the set of all possible image classi- fication tasks, without replacement. Similarly, we select ((2) different upper labelling sub-functions by sampling without replacement from the set of available binary classification tasks {g] For Slong, we use sampling with replacement.
If a problem's training dataset needs be large, V,T we generate according the triple mir (30000, Alli, All). The first value indicates that we generate 30000 data points in total. The second value indicates how many unique images from the ones set aside for training, are used when generating the imputs. In this case, we use all the available training images. The third value indicates how many out of the 64 unique two-dimensional coordinates, used by the upper labelling sub-function, are represented by the input images. In this case, we use all two-dimensional coordi- nates.
Some of the problems' training datasets are required be small and to necessitate transfer. For sequences S S Souu" Sout* ,Stow , we generate the training datasets of each problem Nr- using the triple "hr = (10000, 100, All). This way, only 100 unique images are used to gener- ate the training dataset, so solving the problem is likely be difficult without perceptual trans- fer. The subset of unique images is randomly sampled and can be different between two problems which share an input domain. For sequences Stn and S which evaluate latent transfer, we use the triple Fir = (10000, ,ll,,, As a result, the generated datasets will only represent 30 out of 64 of the two-dimensional coordinates, which is not sufficient for learning the underlying two- dimensional pattern. Therefore, these problems will necessitate latent transfer. When generating a dataset for problem Vr in the sequence Slong, we randomly choose between the two, namely between (10000, 100, All) and (10000, Allu. 30).
For the problems in which the training dataset needs contain only a few data points, -- we use the triple fu = (10, 20, 10). This creates only 10 data points, representing 20 different images and 10 different two-dimensional patterns.
For problems with - we use the triple nvai (10, 20, 10) for generating the validation dataset. For the rest of the problems, we use the triple n (5000, All All). Finally, we generate all test datasets using the triple Reest (5000, Allest, All).
H.3 NEURAL ARCHITECTURE AND TRAINING
Here, we present the neural architecture which we have found be suitable for solving the afore- mentioned compositional problems.
We first define convolutional neural network ÂCNN: 288x28 + R suitable for processing im- ages from the image classification datasets. We use a 5-layer architecture with ReLU hidden ac- tivations and a softmax output activation. The layers are as follows: Conv2ddinpul chammeelsl out-
mutput_channels=de kernelssize=5, stride=2, padding=0). put_chamnel==64, kernelsiize=5, stride=2, padding=O), flatten, FC(4*4*64, 64), FC(64, 64), FC(64, 10). Here, Com-2d specified two-dimensional convolutional layer and FC specifies fully-connected layer.
Second, define fully- colly-connected neural network for processing concatenation of two 8- dimensional one-hot embeddings, SMIP : I + R.. consists of FC hidden layers with 64 hidden units and RELU hidden activations, followed by output FC layer with sigmoid activa- tion.
For compositional problem Ve (Dr, h,,g,, 9j) the input is 2-tuple of images, (xl, and the expected output is a binary classification. We solve it using the architecture Coomp SCNN (x)))). This architecture processes each of the input images with the same ÇCNN model. Then the 2 outputs are concatenated and processed by a SMLP model.
SMPP concaaeenate
21

---

Published as a conference paper at ICLR 2024
We represent this as a modular neural architecture by considering each of the 8 parameterised non- linear transformations to be a separate module. This increases the number of possible paths for each problem. As a result, for the 6th problem in sequence, the number of possible paths is upper bounded by Oi6 = 1679616). Therefore, in this setting, even sequences of length 6 are challenging for modular CL. approaches.
The input space of the last problem of sequence S is given by an 8-dimensional vector. Therefore, only for this problem, we replace ÇCNN with a different architecture, (FC. which consists of two fully connected layers, with a hidden size of 64, and uses ReLU as a hidden activation and softmax as its output activation.
We train new parameters to increase the log likelihood of the labels using the AdamW optimiser (Loshchllov & Hutter 2017) with 0.00016 learning rate, and 0.97 weight decay. The training is done with a mini batch size of 32 and across 1200 epochs. We apply early stopping, based on the validation loss. We stop after 6000 updates without improvement and return the parameters which were logged to have had the best validation accuracy during training.
I ABLATION: GAUSSIAN APPROXIMATION FOR PT SEARCH
The probabilistic model which we use to search through PT paths (Eq. D relies on an approxima- tion of the input distribution which a pre-trained module has been trained with. We proposed to first project samples from this distribution to & dimensions using random projection, and then fit mul- tivariate Gaussian on the resulting samples. In this section we would like to evaluate three aspects of this approach. First, would like to assess the usefulness of the resulting approximations for the purposes of selecting the correct input distribution. Second, we would like to assess the sensitiv- ity of our approximations to the hyperparameter k. Third, we would like compare our approach to Gaussian approximation of the original input space in order to determine whether we sacrifice performance.
To this end, we evaluate whether our approach is useful for distinguishing between a set of input distributions. We compare the approximations resulting from different choices of = {10, 20, 40). The resulting methods are referred to as rp_10, rp_20 and rp-40 respectively. Moreover, we compare to the method of fitting a Gaussian on the original samples, without a random projection. Since this can lead to a singular covariance matrix, we make use of diagonal loading (Draper & Smith 1998) in which we add a small constant (10-8) to the diagonal of the computed sample covariance matrix in order to make it positive definite. We refer to the resulting method flag_lo We compare how well these approaches can distinguish between the image datasets used in BELL. We chose to use the input images for our comparison since they have the highest dimension and their distributions should be the most difficult to approximate.
To evaluate one of the methods. first use it to approximate all 9 input distributions using N data points, resulting in 9 approximations, denoted as {4:)9- Second, for each input distribution Pý. we sample 100 different data points and use them order the approximations in descending order of their likelihood. Ideally. if the data points are sampled from the j-th distribution py. the corresponding approximation qj should have the highest likelihood, and thus should be the first in the list, i.c. should have an index equal to O. We compute the index of gj in the ordered list and use it as an indication of how successfully the method has approximated Py- We compute this index for each of the 9 distributions and report the average index, also referred to as the average position.
We evaluate each method for different choices of N, N = (50, 100, 500, 1000, 5000, 10000, 20000, 30000, 44000]. Moreover, we repeat all evaluations times using different random seeds and report the mean and standard error of the average position. The results are reported in Fig. 7
Our results show that directly modelling the original distribution with a Gaussian leads to sub- optimal performance. On the other hand, we observe that for N Z 500, the methods which use random projection can always match the given data points with the correct distribution which they were sampled for. Surprisingly, for N = 50 and N = 100, diag loading outperforms the other methods and can successfully identify the correct distribution of the given data points. Furthermore, we observe that for these values of N, decreasing the dimension & that the data points are projected to leads to better performance of the methods that are based on random projection.
22

---

Published as a conference paper at ICLR 2024
3.0
12 40
dagg lcading
25
loading
12_ 2_20
2.
19_10
average position
2.-
rp 100
15
15-
1.0
0.-
00
1000
10000
20000
30000
40000
training data points
# training data points
(a)
(b)
Figure 7: Comparison of different methods for approximating a module's input distribution. The axis represents the number N of data points used to compute approximation. The y axis represents the average position, as defined in the main text, which indicates how well a method can approximate the distributions. The lower the average position is, the better the model performs. Figure a) presents a plot across all choices of N. Figure b) focuses on the first few values of N.
Overall, our results suggest that the approximations which we use for out probabilistic model for PT paths are effective when the new modules are trained on more than 100 data points. This seems like reasonable requirement, as fewer points are likely to result in sub-optimal performance.
J ABLATIONS: SEARCH THROUGH NT PATHS
Our search through NT paths (Algorithm 3b involves performing Bayesian optimisation (BO) over NT paths with suffix-length Lmnn (lines 8). In this section evaluate different properties of our BO sub-routine. First, we assess its ability to accelerate the search for the optimal NT path. Second, we assess its early stopping capabilities. Third, we compare our kernel function different alternatives.
For this purpose, we create a new sequence of compositional problems:
sin+ [VT V ... V 15+ V 16 (De, ha, g
(11)
which involves all 16 upper labelling sub-functions g of BELL. The last problem's dataset is gener- ated according to the triple nur = (10000, All,,, 30), which states that only 30 out of the 64 possible wwo-dimensional patterns are represented in the dataset. As result, non-perceptual transfer is nec- essary in order to maximise the performance on the final problem. We create 5 realisations of g with different randomly selected problems.
For each of the 5 realisations of the sequence Sin+, we run each competing method 10 times with 10 different random seeds. This results in 10 over when reporting its performance, For each method, we plot its maximum accuracy achieved per number of paths evaluated. We refer to our realisation of Bayesian optimisation as "BO 12" as uses the 12 norm to calculate a distance between functions (see Appendix F for details).
50 evaluations per method which we average
J.1 SEARCH ACCELERATION
The first two paths which we select for evaluation are selected deterministically as follows. We calculate each suffi''s distance to other suffixes, and choose the 2 suffixes whos average distance to others is the lowest. We do this because these 2 suffixes are likely to more informative about the rest, allowing us to better approximate the rest's performance. In this section, we compare this decision to the altermative of randomly selecting the first two suffixes: referred to as "BO 12.r. Moreover, compare using Bayesian optimisation just using random search, referred to as "RS". The results are shown in Fig. Sa It can be observed that our BO implementation, BO 12, greatly reduces the number of paths required to be evaluated before finding near-optimal performance. Note
23

---

Published as a conference paper at ICLR 2024
Comparison w random searcc
Kernel comparison with randomly selected Initial cardidates
0.95
0.90
0.90
0.85
1007
0.75
0.70
80 80_o0p_PC_
_0_oMp_
0.65
80.0
12
15
8
30
14
@vvolaated pathss
* wvallatted paths
(a) Comparing to randomly selected initial points mm__intt, and random scarch (RS).
(b) Comparing between different choices of GP kernels.
NT. Figure Comparing different design choices for our Bayesian optimisation algorithm $B0
that it does not eventually reach the very best performance due early stopping (see next sub- section). Overall, these results indicate that using Bayesian optimisation significantly accelerates the search, compared to random search.
J.2 EARLY STOPPING
Using Gaussian process (GP) makes it possible to detect when further improvement is unlikely, allowing us to perform early stopping. For this purpose, whenever we select a new pre-trained suffix during our Bayesian optimisation (fine 3lgorithm3). we can compute its Expected Improvement (EI): EI(por (f f)). This tells us how much we expect the selected suffix's performance to improve the current best observed performance. If it is lower than a certain threshold, can stop the Bayesian optimisation. El-based early stopping has been previously suggested in Nguyen et al. (2017) and, similarly to Makaaaoov al. (2022), preliminary experiments showed that it leads to fewer path evaluations, compared to using UCB for early stopping.
When evaluated our ablation sequence gin+, early stopping allowed approach BO 80_22 to eval- uate 11.5 paths on average. In contrast, random search always had to evaluate all 17.
J.3 ALTERNATIVE KERNELS
Next, we investigate different choices for a kernel function used by the Gaussian process for Bayesian optimisation. Our implementation BOJ12 uses the RBF kernel in order to convert distance between functions into a similarity between functions. Instead, we can directly calculate different similarity measures sim, which can then be used with the following kernel:
NT. ( T
Z) = oî + (f
(12)
where dg and 0i are scalar hyperparameters that are optimised on the GP's training dataset, and Z set of points from the functions' input space. We define the following similarity measurements for two scalar functions.
First, we can compute the covariance between the function's outputs which computes the linear relationship between the functions' outputs but also reflects the magnitude the outputs. This leads to the following similarity:
simee (/i, f2; : (OV.
(13)
Second, we can compute the sample Pearson correlation coefficient (Lee Rodgers & Nicewander 1988), denoted PC, between the functions' outputs. This captures the linear correlation of the functions' outputs while ignoring their magnitude. As result, the similarity ranges between [-1, 1]. The similarity defined as:
sim simpe (21, = PC(U.f(ss.
(14)
24

---

Published as a conference paper at ICLR 2024
Third, note that the aforementioned similarities computed based on the functions' outputs which only reflect points in their output space. Instead, we can compare the functions' curvatures around each evaluation imput 2,. For each input, compute the gradient of each function's output with respect to its input and normalise it to have a unit norm. The dot product between the two resulting normalised gradients is then computed in order to capture the alignment between the two curvatures. This leads to the following similarity:
1 (f. = V Σ
V
Va(e)) (IAA ( V(2((,)) (1V
(15)
The 3 similarities defined above lead to 3 kernels, which in turn result in the following 3 BO algo- rithms: BO cov r, BO pe r, BO gpd r. Moreover, one can sum kernel functions in order to result in new kernel function, which uses combination of similarities. We combine different kemnlss which leads to the following algorithms: B0-gpd_pe.r, BO 50_gpd_12_r and BO 30_gpd_12_pc_ For all algorithms, we use randomly selected initial paths, because the paths selected deterministically per- form too well which makes it harder to compare algorithms. The resulting comparison is presented in Fig. [b Overall, BO_12.r achieves the best anytime performance by finding well performmngn paths more quickly. We also observe that combining the RBF kernel of BO J2. with other kernels does not result in an improvement.
K CTRL BENCHMARKS
The CTrL benchmark suite was introduced in Veniat et al. (2020).
They define a number of se-
quences, based on seven image classfication tasks, namely: CIFAR10 10 and CIFARI00 (Krizhevsky al. 2009), DTD (Cimpoi et al., 2014), SVHN (Netzer et al.
2011), MNIST (LeCun et al.) Xiao et al. 2017). All images are rescaled to 32x32 pixels in the RGB color format. CTrL was first to introduce the following sequences: St, , Stn, Soul, SPI and glong, which are defined similarly to our definitions. How- ever, the difference is that they are defined for and implemented by image classification tasks. The last task in Sin which evaluates non-perceptual transfer, is given by MNIST images with different background color than the first task. The last task S is given by shuffling the output labels of the first task. çlong has 100 tasks. For each task, they sample random image dataset and a random subset of 5 classes to classify. The number training data points is sampled according to a distri- bution that makes it more likely for later tasks have small training datasets. In contrast us, they use only 1 selection of tasks for each sequence. i.c. realisation of each sequence. To generate the sequences, we use the code provided by the authors (Veniat & Ranzato
1998. RanbowMNIST (Finn et al. 2019), and Fashion MNIST
202T.
Our experimental setup mirrors that used in (Ostaennko et al. 2021 in order for us to be able to compare to LMC using the official implementation. The modular neural architecture consists of modules with hidden size of 64. The last module is a linear transformation. The first 4 modules are identical and are sequence of: 2-dimensional convolution (kernel size 3, stride padding 2), batch normalisation (momentum 0.1), ReLU activation and 2-dimensional max-pool (kernel size 2, no stride and 0 padding).
A single network is trained with Adam Kingma & Ba (014)) with learning rate of le 3 and weight decay of le - 3. The batch size is 64 and we train for 100 epochs. After each epoch, the current validation performance is evaluated. The best weights which resulted in the best validation performance, are restored at the end of the training.
K.1 RESULTS WITH STANDARD DEVIATIONS
Table 6 shows our results on the short CTrL sequences, with included standard deviations.
L RESULTS ORIGINAL CTRL SETUP
We also experiments the original CTrL experimental setup (Veniat al. 2020), which is outlined next. The neural architecture used is a small variant of ResNet18 architecture which is divided into 6 modules, each representing different ResNet block (He et al. 2016). While the paper
25

---

Published as a conference paper at ICLR 2024
O-BWC
ER
HOUDINI
LMC
MNTDR-D
PICLB
65.4 0.4
24.5 = 1.3
46.4 + 1.2
65.4 1.5
6.
69.5 = 0.5
6622 #
68.3
63.1 0.3
46.9 10.7
56.6
63.8 1.
66.5
65.7
66.8
66.8
0.3
43.6
63.7
63.9
62.3
6.6
64.7
A
54,9
6.
0.
63.2
6.8
59.7 - 0.8
0.11
Avg.
63.7
41.9 + 9.9
54.7 + 4.8
64.0
65.
64.6
65.6 +
66. 2.2
Avg.
0
20.6 +: 12.0
+4.8 +: 2.0
0:0
0. 0
-0.7 0.8
=
8
0
3.8 1 1.7
41.1
0.5
5.0 1.
17.4
.0.6 5.
16.3
8
0. 0
6.8 4: 1.8
2,9
2.6 2: 5.9
17.5 d:
16.8 0.6
17.1 1.1
17.1 di
8
0. 0
-139
4.6
0.
0.
-10.8 +
0
0. 0.
Ty
0. 0
6.7 2.6
2.
4.1
20.7
184
20.7 2.9
20.7
0.1
1146 1.4
-7.5
0. 0.0
0. 0
.3.0 1.0.11
0. 0:
0
Avg.
0
-22 11.0
-10, 18.1
1.2
6.6 11.6
13.7
10.6
10.8 10.0
Table 6:
Results on CTrL sequences which assess perceptual transfer (S SS0), plasticity (SP)), backward transfer (S*) ) and latent transfer 5tn
SA
MNTDP-D
PT-only
NT-only
PICLE
S
58.77
61.36
61.78
63.41
63.10
S
74.25
77.95
78.15
78.15
SPI
58.25
93.72
93.79
93.79
A
ST
56.28
81.67
81.92
81.92
S+ 73.61
74.54
74.49
74.49
Avg.
64.23
77.85
78.03
78.29
S
0.
22.12
24.67
32.57
32.57
S
0.
15.41
15.41
15.41
Th-1
SPI
O.
00.20
00.20
00.20
S-
0.
34.29
34.29
34.29
S+
O.
0.
0.
Avg.
0.
14.40
14.91
16.49
Table 7: The evaluations on the short CTrL sequences using the original experimental setup pro- posed by Veniat et al. (2020).
presenting the CTrL benchmark states that 7 modules are used, we used the authors' code (Veniat 202T) for this method which specifies only 6 modules with the same total number of parameters. The difference from the architecture stated in the paper is that the output layer is placed in the last module, instead of a separate module.
All parameters are trained to reduce the cross-entropy loss with Adam optimiser Kingma & Ba (0014) with 31 = 0.9, 31 = 0.999 and = 10 For each task, each path evaluated times with different combinations of values for the hyperparameters of the learning rate (f10-3 10-33) and of the weight decay strength (0, 10-5, 10-4]. The hyperameters which lead to the best validation per- formance selected. Early stopping employed during training. If no improvement is achieved in 300 training iterations, the parameters with the the best logged validation performance are selected. Data augmentation is also used during training, namely random crops (4 pixels padding and 32x32 crops) and random horizontal reflection.
The results, presented in Table show that PICLE able to outperform MNTDP-D on these quences as well.
26


Reviews:
[{"review_id": "DEDRYTNzE5", "reviewer": "Reviewer_gkN4", "summary": "This paper addresses the modular continual learning setting.  They introduce a probabilistic model to determine a distribution over paths for perceptual transfer.  This can be interpreted as selecting modules based on how similar the current features at that layer are to the modules' input distribution.  For latent transfer, they have a probabalistic model based on the idea that suffixes similar in L2 distance should have similar performance.  This modeling allows predicting the validation performance of a path without training.  Due to the modeling, they only need to evaluate a number of paths which grows linearly with the depth L.  Their method outperforms the other methods marginally on average, while in some cases demonstrating significant boosts, e.g. few-shot transfer.", "questions": "**Table 1:** Why are the numbers for MNTDP-D and PICL forward transfer so similar?\n\nTypo: “The CL algorithm should be able to ”remember” previous problems“ (backwards quotation) \n\n**Algorithm 3:** Bold lambda on right side of line 5\n\n“This search results in the most relevant previous solution $\\pi'$. Finally, in lines 11-14, we evaluate NT paths created by transferring a different number of the last $\\ell \\in \\\\{\\ell_{min} + 1, ..., L − 1\\\\}$ layers of $\\pi'$, to see if re-using more layers leads to further improvement”. (do you mean $\\ell \\in \\\\{2, \\ldots L - 1\\\\}$?  That's what the code appears to be searching over", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The probabalistic models introduced are simple and make sense.  Using such modeling to avoid the expensive counterfactual of evaluating a path is a logical approach.\n* The paper is well written and I found the appendix helpful.\n* The evaluations seem consistent with prior works", "weaknesses": "* The quantitative advantage over MNTDP-D in accuracy is marginal on average ~1-3%\n* The method requires slightly more FLOPs than MNTDP-D (Figure 3 (a))\n\nOverall I am voting to accept this paper due to the conceptual contribution, however the quantitative results seem marginal to me, and thus I am not voting for a stronger accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IKJUSZOqiu", "reviewer": "Reviewer_g7xb", "summary": "This paper introduces PICLE, a scalable modular continual learning algorithm that addresses key challenges in the field. PICLE is task-aware and optimizes for different types of transfer learning, such as perceptual, few-shot, and latent transfer. Utilizing a probabilistic search method, it efficiently approximates the fitness of different module compositions, significantly reducing training requirements. The algorithm outperforms existing state-of-the-art solutions, demonstrating its efficacy on the popular CTrL benchmark suite and a new extension called BELL.", "questions": "None", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Modularity in continual learning is highly promising, offering a good balance between scalability and the ability to transfer knowledge across tasks without undue interference.\n  \n2. The key challenge in modular approaches is the exponentially increasing search space for module combinations, particularly as the number of layers grows. PICLE addresses this scalability issue effectively, allowing for both perceptual and latent transfer, which makes it a standout in the field.\n  \n3. The paper's empirical study is both comprehensive and well-executed, adding robustness to its claims.", "weaknesses": "Minor weaknesses, if addressed during the rebuttal, I will keep my score at an 8:\n\n1. The table in the paper lacks uncertainty metrics such as standard deviation. This omission should be addressed to enhance the study's reliability. Additionally, for readability purposes, it would be better to show the percentages only up to three digits instead of four (e.g., XX.XX% should be changed to XX.X%).\n\n2. The paper should clearly state that the PICLE method is task-aware, which is an important limitation. Ideally, there would be a column in Table 4 that discusses task-agnosticism, a feature that another algorithm, LMC, is capable of.\n\n3. The concept of using a generative model to approximate latent activations and assume local independence from one layer to the next, as introduced in Section 4, was initially proposed in the LMC paper. The authors should give credit to this paper and clarify how their methods differ from those originally proposed in the LMC paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o5XbRQa3Zp", "reviewer": "Reviewer_K341", "summary": "The work proposes a probabilistic modeling approach to efficiently search for the best-fit module composition out of the large discrete space of possible compositions of the modules in continual learning. Depending upon the similarity of the input distributions with that of a previous problem, two variants of the probabilistic model have been proposed: one for perceptual transfer where the prior uses the original accuracy of pre-trained modules to order these, and the other for latent transfer where the prior specifies that pre-trained modules in a path have been used together to solve a previous problem.", "questions": "- On page 3, the authors mention their strategy is based on a generative model of the input x. How is the generative quality of the proposed method quantitatively? Some further evaluation of the proposed method using metrics like ECE can thus be more insightful.\n\n- While I am not very familiar with the up-to-date modular continual learning literature, the baselines in Tables 1-2 look classic to me. Can the authors comment on comparing with more recent works?\n\n- Can the authors compare the computational overhead of their method against the baselines?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Good motivation, presentation, and writing. The equations have been explained well.\n- The idea of using the validation accuracy for a path as the proxy for its fitness is simple and elegant.\n- The limitations of the proposed method have been elaborated well.\n- The reported evaluation metrics are rigorous.", "weaknesses": "Please see the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sRhh0qsglO", "reviewer": "Reviewer_REkX", "summary": "The authors propose PICLE a probabilistic approach to spawn, compose and train a series of neural modules for solving continual and transfer learning tasks with compositionality. The proposed method considers two separate scenarios:  perceptual transfer, where the input space is the same for all tasks and the output function has to be adapted; and latent transfer, where the output functions have to be reused on new input spaces. For the first scenario, the authors approximate the input distribution of each module with the combination of a low-rank gaussian distribution and the accuracy as the prior. The final path is found by greedy search. For the second scenario, the authors use Bayesian optimization and a Gaussian process to approximate the accuracy of unseen module combinations. One of the main advantages of the proposed approach is that it scales with respect to the number of tasks sinc it evaluates a constant number of compositions and trains a network with a fixed size for each task. The proposed approach achieves state of the art perfomance on CTrL as well as on a new compositional version of CTrL named BELL by the authors.", "questions": "* Would it be possible to run PICLE on compositional color MNIST to be able to compare with LMC (see LMC paper)? (It is ok if you do not have enough time / compute resources to do it). \n* Could you include some more motivation on why BELL is needed and why researchers should use it rather than other benchmarks or tasks.\n* Why $\\{\\pi}^*$ is not used in Algorithm 2?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Originality\n=======\n* The probabilistic modular framework proposed in this work is novel to the best of my knowledge.\n* The authors introduce BELL, a new compositional version of CTrL along with this work.\n\nQuality\n=====\n* The proposed approach is sound.\n* The low-rank approximation of the input as a proxy for the likelihood is simple yet effective.\n* The appendix contains details to ensure reproducibility and the authors promise to release the code.\n* The authors provide Algorithms.\n* The authors discuss the limitations of their approach.\n* Ablations can be found in the Appendix.\n\nClarity\n=====\n* The text is well-written.\n* I found Figure 3 and Table 4 very useful in order to understand the difference between PICLE and other algorithms.\n* The authors provide details about hyperparameters in the Appendix.\n\nSignificance\n=========\n* Continual learning is a challenging problem and I believe this work is an interesting step towards a modular solution\n* PICLE achieves a slight improvement on CTrL and greater improvement on some of the BELL tasks.\n* BELL complements CTrL with few-shot and compositional tasks.", "weaknesses": "Originality\n=======\n* LMC also introduced a benchmark for compositional generalization based on colored-mnist that is not mentioned when introducing BELL.\n* Moreover the authors claim not to be able to obtain acceptable results with LMC on BELL despite their best efforts, while this is possible, they could have tried to run PICLE on compositional color MNIST task introduced in LMC.\n\nClarity\n=====\n* This work introduces a method and a benchmark, however most information about the benchmark is left in the appendix. I suggest providing simpler versions of the Algorithms in the main text, and to use the space to make the paper more self-contained (less dependent on the Appendix).\n\nSignificance\n=========\n* There exists a vast number of continual learning benchmarks. Although they enrich the field, they also dilute the efforts of the research community. Thus I suggest the authors to include some more motivation on why BELL is needed and why researchers should use it rather than other benchmarks or tasks.\n\nMinor\n====\n* Page 4: Accordignly\n* Appendix I: pre-traiend", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)